From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 01:08:25 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6A6510657CD for ; Sun, 29 Jun 2008 01:08:25 +0000 (UTC) (envelope-from oberman@es.net) Received: from postal1.es.net (postal1.es.net [198.128.3.205]) by mx1.freebsd.org (Postfix) with ESMTP id BD6BF8FC16 for ; Sun, 29 Jun 2008 01:08:25 +0000 (UTC) (envelope-from oberman@es.net) Received: from ptavv.es.net (ptavv.es.net [198.128.4.29]) by postal1.es.net (Postal Node 1) with ESMTP (SSL) id JGI49656; Sat, 28 Jun 2008 17:57:56 -0700 Received: from ptavv.es.net (ptavv.es.net [127.0.0.1]) by ptavv.es.net (Tachyon Server) with ESMTP id 4BA1B4500E; Sat, 28 Jun 2008 17:57:56 -0700 (PDT) To: VANHULLEBUS Yvan In-Reply-To: Your message of "Sat, 28 Jun 2008 23:13:00 +0200." <20080628211300.GA3310@zen.inc> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="==_Exmh_1214701076_59533P"; micalg=pgp-sha1; protocol="application/pgp-signature" Content-Transfer-Encoding: 7bit Date: Sat, 28 Jun 2008 17:57:56 -0700 From: "Kevin Oberman" Message-Id: <20080629005756.4BA1B4500E@ptavv.es.net> X-Sender-IP: 198.128.4.29 X-Sender-Domain: es.net X-Recipent: ;; X-Sender: X-To_Name: VANHULLEBUS Yvan X-To_Domain: zeninc.net X-To: VANHULLEBUS Yvan X-To_Email: vanhu_bsd@zeninc.net X-To_Alias: vanhu_bsd Cc: freebsd-net@freebsd.org Subject: Re: FreeBSD NAT-T patch integration X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 01:08:25 -0000 --==_Exmh_1214701076_59533P Content-Type: text/plain; charset=us-ascii Content-Disposition: inline > Date: Sat, 28 Jun 2008 23:13:00 +0200 > From: VANHULLEBUS Yvan > Sender: owner-freebsd-net@freebsd.org > > On Fri, Jun 27, 2008 at 11:06:19AM -0400, George V. Neville-Neil wrote: > > At Thu, 26 Jun 2008 12:56:41 -0700, > > julian wrote: > > > > > > I'm planning on committing it unless someone can provide a reason not > > > to, as I've seen it working, needed it, and have not seen any bad > > > byproducts. > > > > > > > I'd be interested to know how you tested it. > > I can answer that question: > > - We do have "non regression tests", which test every night that "it > works" in a "good scenario", with a peers who runs quite the same > kernel, with the same patch and the same userland. > > We do also some "bad scenarios" tests, but not as much as I would like > actually (it's much more complex to test). > > > - We do have THOUSANDS of customers who use it for years now. And > customers are really not well nown to always generate "good > scenarios".... > > Customers can do some mistakes, can use some very strange stacks for > peers, can use a lots of other IPSec stacks for peers, can have *a > lot* of peers (of course with various implementations, configurations, > NAT or not, etc...) for the same gate, etc... > > And how do I know that it works ? > Well, when it doesn't work, I do know it, quite quickly most of the > time ! > > As Bjoern said in another mail, we're talking about security. > Security is my job, security is what we provide to our customers, and > even if some of our customers just don't understant what they do, some > others do *really* understand things, and do *really* test devices, > check if it works, and quickly reports us problems (and ask for quick > solutions). > > > > > NAT-T and IPsec are > > non-trivial protocols/subsystems that can have far reaching impacts on > > the network stack. > > Yes. > > > > Also, are you planning to maintain it after > > committing it? > > I plan to do that. > > > > The biggest problem with NAT-T hasn't been the code, > > it's been that the author, > > Really ? > > > > who is doing a great job on the code, > > Thank you.... > > > > has > > been too busy to maintain it anywhere but at work. > > That is not exact, and I'm really sorry if you understood that after > our discussion at the last EuroBSDCon. > > First, you can check that I'm sending this mail on saturday evening, > which is out of my work hours :-) > > Then I can for example confirm that I did the whole migration from > RELENG6 to RELENG7 on my free time, just because I needed a working > FreeBSD7 + NAT-T patch at home before I've been asked to do it at > work. > > Yes, I'm doing most of IPSec / NAT-T stuff at work, but it is mainly > because I'm *paid* for that !!! > Let me tell this again: working on FreeBSD's IPSec stack (and on > NAT-T, of course, adn also on ipsec-tools) is a part of my job. > > What I told you some months ago is that there are some things that I > cannot really do at work, because it takes lot of time and because > those things are more "code cleanups" than "new features" (we were > talking about PFKey V2 cleanups related to NAT-T ports). > > > And if I did not spend that time to do that cleanup, that's not > because I don't have such time, that's because I was starting to > fear that the patch may be never commited, so I wasn't sure to want to > continue spending lots of time on a patch "for nothing". > > If Bjoern or you told me some months ago "do that cleanup and we'll > commit the patch", it would already have been done, and we would have > *all* saved some time ! > > > And yes, I do also spend some parts of my free time on things that are > not related to FreeBSD's IPSec (and, to be completely honest, on > things which are not related at all to computers....). > > > > > That is not a slam > > on the person or the code, I have the highest respect for both, but it > > reflects and important reality of the situation. > > I feel that "the reality of the situation" is that FreeBSD's IPSec > stack needs more people *working on it*, not more people wasting their > time about why some work should or shouldn't be reported. > > That is not a slam on the persons who are still working on the IPsec > stack, I have the highest respect for them, but it reflects an > important reality of the situation... > > > > > Unless you're > > stepping up to maintain it as well as commit it I think it should not > > be committed. I know the Bjoern has been working hard to pick up the > > IPsec stuff in his free time, and I value his input on this subject > > quite a bit. > > You said it yourself: actually, FreeBSD's IPSec stack is just > maintained by one person, which worked hard on it, which does a great > job on it, but which can spend only some parts of his free time on it. > > That is a problem. > Of course, the solution can NOT be to ask Bjoern to spend more time on > it (well, Bjoern, do that if you want :-). > > The only other solution I see is to find a way to help people who want > to help you.... Thanks so much to folks like Bjorn and Yvan who spend the time to do some tough jobs like dealing with IPsec and being stubborn about committing things to security tools without very careful audit. In case you missed it, IPsec is about security, not features. And, in case you have never been involved in real security or crypto, security is really, really hard. No, no, it's much harder than that! While I'd love to see NAT-T, until someone who knows EXACTLY what the IPsec and NAT-T code is doing and what it is required to do can do the line by line review, it should NOT be committed. -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751 --==_Exmh_1214701076_59533P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) Comment: Exmh version 2.5 06/03/2002 iD8DBQFIZt4Ukn3rs5h7N1ERApJOAKCRHsr7B6qizoPgXAoj9CTX01xFNgCfQ+Y8 nfj46S+au7AH71btVOXOYx8= =BONd -----END PGP SIGNATURE----- --==_Exmh_1214701076_59533P-- From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 03:24:51 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0CB0106567F for ; Sun, 29 Jun 2008 03:24:51 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from ape.monkeybrains.net (ape.monkeybrains.net [208.69.40.11]) by mx1.freebsd.org (Postfix) with ESMTP id B7F618FC2E for ; Sun, 29 Jun 2008 03:24:51 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from monchichi.monkeybrains.net (adsl-76-211-243-65.dsl.pltn13.sbcglobal.net [76.211.243.65]) (authenticated bits=0) by ape.monkeybrains.net (8.14.1/8.14.1) with ESMTP id m5T3OwMR062911 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Sat, 28 Jun 2008 20:24:59 -0700 (PDT) (envelope-from crapsh@monkeybrains.net) Message-ID: <4867007E.90409@monkeybrains.net> Date: Sat, 28 Jun 2008 20:24:46 -0700 From: Rudy User-Agent: Thunderbird 2.0.0.12 (X11/20080310) MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <20080516185813.H866@logos.sky.od.ua> <2a41acea0805160904g7dcf9f58rf69ca5d0612945cc@mail.gmail.com> <4853055C.2030303@MonkeyBrains.NET> <48535A11.4020003@monkeybrains.net> <48582C29.8030307@monkeybrains.net> <2a41acea0806191055w5e112b8bsa57a8db2b177adbe@mail.gmail.com> <485C0F07.7000408@monkeybrains.net> <2a41acea0806201355y3b123462wc37280f28a9f4216@mail.gmail.com> <485C41C5.4090706@monkeybrains.net> <2a41acea0806201713m7a083b9dr10894c8a0845df8e@mail.gmail.com> <485D5B9E.8020908@monkeybrains.net> <485ECF20.6000607@monkeybrains.net> <48669F6F.1050005@monkeybrains.net> In-Reply-To: <48669F6F.1050005@monkeybrains.net> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93.1, clamav-milter version 0.93.1 on pita.monkeybrains.net X-Virus-Status: Clean Subject: Re: Seeking help understanding my "emX: watchdog timeout" messages X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 03:24:51 -0000 Rudy wrote: > I just upped values in my loader.conf: > hw.em.rxd=1024 > hw.em.txd=1024 Still getting watchdog timer events. :p Load is low: # ps axw | grep -v 0:0 PID TT STAT TIME COMMAND 10 ?? RL 325:37.05 [idle: cpu1] 11 ?? RL 344:33.83 [idle: cpu0] 13 ?? WL 1:12.35 [swi4: clock sio] 20 ?? DL 14:30.03 [em0 taskq] 21 ?? DL 6:20.68 [em1 taskq] 22 ?? DL 5:29.90 [em2 taskq] 24 ?? DL 12:57.23 [em4 taskq] 31 ?? DL 0:43.60 [dummynet] 1297 ?? Ss 0:11.18 /usr/local/sbin/zebra --daemon --retain -A 10.77.0.6 1303 ?? Ss 3:19.46 /usr/local/sbin/bgpd --daemon --retain -A 10.77.0.6 Trying a kernel rebuild without dummynet (as I am not using that).... - Rudy From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 05:34:15 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5EB7A1065670 for ; Sun, 29 Jun 2008 05:34:15 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.30]) by mx1.freebsd.org (Postfix) with ESMTP id 1B0D78FC1D for ; Sun, 29 Jun 2008 05:34:14 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by yw-out-2324.google.com with SMTP id 9so454760ywe.13 for ; Sat, 28 Jun 2008 22:34:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=G4VBdXvhd/JJQ0GVpHOQ3dVLmghdJx5nqUWV8+f66Ok=; b=cdMpzVnRmUvOXyUh/zPDiBEZYWKGF9H7YYTl+tDmN32CHxhgSjRFhJEpb7tQlAj0tU u41FuAEhT0456haZ1sCwzrFENrHmmebj9EpV9/JDfTOt3cQskTW+ImDN/M0tgP0ttULT vVgURBNdRMmBFdXEX0x8Eoms+47nZg7vsBn4o= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=R22IbB3B8wYgc5Ezg5ywlzvvqxy4kpilToK5uF/Ki5Yl/FwJDXERG4WV9kKt5flISO QilNvZz7vjdeTlJ7J7dMmTUAmIAY2/Awd2Z3rR+iG8RrUvcvifsEYCa+LRWhw6dy+Zq1 Bs9CWHvhE5cl8waEh5PjPRTAsj9WxNokmZAVc= Received: by 10.150.149.19 with SMTP id w19mr5724573ybd.19.1214717654256; Sat, 28 Jun 2008 22:34:14 -0700 (PDT) Received: by 10.151.154.17 with HTTP; Sat, 28 Jun 2008 22:34:14 -0700 (PDT) Message-ID: Date: Sat, 28 Jun 2008 22:34:14 -0700 From: "Freddie Cash" To: freebsd-net@freebsd.org In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: Subject: Re: Understanding where dummynet fits into an ipfw ruleset X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 05:34:15 -0000 On Fri, Jun 27, 2008 at 11:14 PM, Ian Smith wrote: > On Fri, 27 Jun 2008, Chuck Swiger wrote: > > On Jun 27, 2008, at 3:01 PM, Freddie Cash wrote: > > [ ... ] > > >> If net.inet.ip.fw.one_pass is true, then you definitely want to > > >> apply your deny rules first, as once something matches a pipe > > >> rule, it's going to be passed. The tradeoff is that the > > >> accounting/fairness of traffic is less accurate but the firewall > > >> ruleset runs faster... > > > > > > So, in this situation, the "allow" rules would be the queue rules? > > > > > > To add traffic shaping to the following, using one_pass=1: > > > 100 allow ip from 1.1.1.1 to 2.2.2.2 in recv em0 > > > 200 allow ip from 1.1.1.1 to 2.2.2.2 out xmit em1 > > > 300 deny ip from any to 2.2.2.2 in recv em0 > > > > > > Would be: > > > 100 queue 1 ip from 1.1.1.1 to 2.2.2.2 in recv em0 > > > 200 allow ip from 1.1.1.1 to 2.2.2.2 out xmit em1 > > > 300 deny ip from any to 2.2.2.2 in recv em0 > > > > > > Or am I way off here? :) > > > > Hmm. If you have one_pass set, I believe that rule 200 would become > > superfluous. If it was off, rule 200 would be needed to permit > > traffic through. No, the rule is needed. Packets passing through the firewall go through the ruleset twice: once entering the external interface, and again leaving the internal interface (and vice versa). The first rule allows the packet in on the external interface, the second rule allows it out the internal interface, the third rule denies all other incoming traffic. > I'm keen to be certain about this myself. I think that one_pass only > affects the packet on the current pass, in this case on the incoming > pass from ip_input at rule 100 above. So if one_pass is set, the packet > is queued (and accepted in) and the search terminates, but with one_pass > off, the packet is reinjected at rule 200 when it leaves the pipe after > its limitation and/or delay, and would need to be specifically allowed. > > However I don't think that affects the outgoing pass, ie after routing > the packet still has to go through the firewall again (from ip_output), > so in this case it will be allowed if it's going out to 2.2.2.2 via em1, > else is denied at rule 300. > > Another option might be to only apply pipe/queue actions to 'out' rules, > but mixing that with both 1:many and 1:1 NAT will complicate matters .. > > However, queue rulesets are used to classify traffic > > into different bins; then then get pulled out of the bins with packets > > waiting is proportion to the weights configured via something like: > > > > ipfw queue 1 config pipe 1 weight 10 > > > > ie, you have to attach queue(s) to a pipe for this classification or > > sorting to be meaningful. Yes, I know that. I was just trying to use very simplified examples to understand how the traffic shaping works in conjunction with the packet filtering. I understand how the queueing works, and have used it successfully on routers that don't do packet filtering. It's adding it to existing packet filter rulesets that is making my head spin. :) > Yes I suspect Freddie might want to use pipe rather than queue here too, > if just for bandwidth limitation rather than weighted queueing by type > of traffic? And is it only wanted for managing the inbound traffic? No, I want to use queue. I want to create rules to "reserve" bandwidth for connections to important servers, as we're moving to more web-based applications, and I want to make sure students surfing the web don't impact office staff. There will be a single pipe, with two queues, one weighted at twice the value of the other. That way, if there is no staff traffic, the students get the whole pipe. If there is no student traffic, staff get the whole pipe. And if there's a mix, then staff traffic is prioritised ahead of student traffic. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 05:36:12 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CC180106564A for ; Sun, 29 Jun 2008 05:36:12 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.28]) by mx1.freebsd.org (Postfix) with ESMTP id 899508FC1E for ; Sun, 29 Jun 2008 05:36:12 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by yw-out-2324.google.com with SMTP id 9so454883ywe.13 for ; Sat, 28 Jun 2008 22:36:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=KDFa9HrnMnEQd9fwEt2oMqMA8oN/SXI2Qi83/ebd70Q=; b=MS2cqd9VJoYNkse1apKdVn5pMRm/kKleMF+s3sAXn2neIM3QFhFk9ygFsp+4AqBfoK PBzD94VR73AIjsCn8I6UEtq+rLwZaoGq3dQ5pN32PJM2HkcfWBrtSFbJgoLY0/5pAqeZ e3jo28Z9dMnc703fy8Yo6K8sMrZq0YfhZOAfs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=SWs/OjBPLa6V6ObfHkew59+pA5ZzGTdXItuBpenZ1nkSmqKVBLyG1Ll9eaEyOUW6Et MU1XvsbmJeF51HpuXMPnKJl+ESrX19WxNsa64mGGkRHEIIQu7BF/IvOvDpMBfGZXe1aT sLR83/2GrRrGYMYbJUeVpziEkH1Eh36qbVTDE= Received: by 10.150.217.6 with SMTP id p6mr2041312ybg.72.1214717771911; Sat, 28 Jun 2008 22:36:11 -0700 (PDT) Received: by 10.151.154.17 with HTTP; Sat, 28 Jun 2008 22:36:11 -0700 (PDT) Message-ID: Date: Sat, 28 Jun 2008 22:36:11 -0700 From: "Freddie Cash" To: freebsd-net@freebsd.org In-Reply-To: <1214609672.15425.25.camel@devstation> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <1214609672.15425.25.camel@devstation> Subject: Re: Understanding where dummynet fits into an ipfw ruleset X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 05:36:12 -0000 On Fri, Jun 27, 2008 at 4:34 PM, Martes Wigglesworth wrote: > It really does not matter, because the que has nothing to do with denial > of service unless you set it up that way. You just divert to the > dummynet pipe, or que and then the rest is handled by the mechanism. > The ipfw rule only works to augment the flow of traffic into the pipe, > or que. Just make sure you are not dropping any of the intended traffic > prior to a dummynet rule, because then you would be hendering the rule's > ability to shoot traffic to the que, or pipe. > > The dummynet manpage has some examples, and there are some good > tutorials on onlamp.com. > > Otherwise, I hate to say it but, "google it." Yes, I've read the dummynet and ipfw man pages. Yes, I've read articles on the 'Net. Yes, I've done google searches. And no, it still doesn't make sense how queue rules work with packet filter rules. Hence, why I'm asking here. > I have not done much with bandwidth shaping in about a year, so I am a > bit rusty as to the more complex setups however, it is one of the more > easy to remember methods so I think that I am correct, or at least not > far off of base. -- Freddie Cash fjwcash@gmail.com From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 06:22:41 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8D3291065672 for ; Sun, 29 Jun 2008 06:22:41 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from gaia.nimnet.asn.au (nimbin.lnk.telstra.net [139.130.45.143]) by mx1.freebsd.org (Postfix) with ESMTP id B1A6B8FC0C for ; Sun, 29 Jun 2008 06:22:39 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (smithi@localhost) by gaia.nimnet.asn.au (8.8.8/8.8.8R1.5) with SMTP id QAA15258; Sun, 29 Jun 2008 16:22:30 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Sun, 29 Jun 2008 16:22:29 +1000 (EST) From: Ian Smith To: Freddie Cash In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Cc: freebsd-net@freebsd.org Subject: Re: Understanding where dummynet fits into an ipfw ruleset X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 06:22:41 -0000 On Sat, 28 Jun 2008, Freddie Cash wrote: > On Fri, Jun 27, 2008 at 11:14 PM, Ian Smith wrote: > > On Fri, 27 Jun 2008, Chuck Swiger wrote: > > > On Jun 27, 2008, at 3:01 PM, Freddie Cash wrote: > > > [ ... ] > > > >> If net.inet.ip.fw.one_pass is true, then you definitely want to > > > >> apply your deny rules first, as once something matches a pipe > > > >> rule, it's going to be passed. The tradeoff is that the > > > >> accounting/fairness of traffic is less accurate but the firewall > > > >> ruleset runs faster... > > > > > > > > So, in this situation, the "allow" rules would be the queue rules? > > > > > > > > To add traffic shaping to the following, using one_pass=1: > > > > 100 allow ip from 1.1.1.1 to 2.2.2.2 in recv em0 > > > > 200 allow ip from 1.1.1.1 to 2.2.2.2 out xmit em1 > > > > 300 deny ip from any to 2.2.2.2 in recv em0 > > > > > > > > Would be: > > > > 100 queue 1 ip from 1.1.1.1 to 2.2.2.2 in recv em0 > > > > 200 allow ip from 1.1.1.1 to 2.2.2.2 out xmit em1 > > > > 300 deny ip from any to 2.2.2.2 in recv em0 > > > > > > > > Or am I way off here? :) > > > > > > Hmm. If you have one_pass set, I believe that rule 200 would become > > > superfluous. If it was off, rule 200 would be needed to permit > > > traffic through. > > No, the rule is needed. Packets passing through the firewall go > through the ruleset twice: once entering the external interface, and > again leaving the internal interface (and vice versa). > > The first rule allows the packet in on the external interface, the > second rule allows it out the internal interface, the third rule > denies all other incoming traffic. Yes. So isn't that working? ipfw queue show? [ cut my longwinded explanation of what you just said above :] > > > However, queue rulesets are used to classify traffic > > > into different bins; then then get pulled out of the bins with packets > > > waiting is proportion to the weights configured via something like: > > > > > > ipfw queue 1 config pipe 1 weight 10 > > > > > > ie, you have to attach queue(s) to a pipe for this classification or > > > sorting to be meaningful. > > Yes, I know that. I was just trying to use very simplified examples > to understand how the traffic shaping works in conjunction with the > packet filtering. I understand how the queueing works, and have used > it successfully on routers that don't do packet filtering. It's > adding it to existing packet filter rulesets that is making my head > spin. :) And in your subsequent reply to martes@mgwigglesworth.com you said: > Yes, I've read the dummynet and ipfw man pages. Yes, I've read > articles on the 'Net. Yes, I've done google searches. And no, it > still doesn't make sense how queue rules work with packet filter > rules. Hence, why I'm asking here. It's not clear to me what's not working from your example rules above? Given using one_pass=1, that should go. And using one_pass=0, you should only need to also add as say rule 150: 150 allow ip from 1.1.1.1 to 2.2.2.2 in recv em0 > > Yes I suspect Freddie might want to use pipe rather than queue here too, > > if just for bandwidth limitation rather than weighted queueing by type > > of traffic? And is it only wanted for managing the inbound traffic? > > No, I want to use queue. I want to create rules to "reserve" > bandwidth for connections to important servers, as we're moving to > more web-based applications, and I want to make sure students surfing > the web don't impact office staff. There will be a single pipe, with > two queues, one weighted at twice the value of the other. That way, > if there is no staff traffic, the students get the whole pipe. If > there is no student traffic, staff get the whole pipe. And if there's > a mix, then staff traffic is prioritised ahead of student traffic. Ok; on rereading your original, I should have realised that. So with a similar set of rules for the other of staff/students that your above example deals with, and the right pipe and queue configs, what remains to do? Sorry to be thick, but I don't see why that wouldn't work .. cheers, Ian From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 06:42:25 2008 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 83CB71065689 for ; Sun, 29 Jun 2008 06:42:25 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 5B1DE8FC15 for ; Sun, 29 Jun 2008 06:42:25 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id E218446C32; Sun, 29 Jun 2008 02:42:24 -0400 (EDT) Date: Sun, 29 Jun 2008 07:42:24 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Ali Niknam In-Reply-To: <20080627090939.M78484@fledge.watson.org> Message-ID: <20080629073519.D10134@fledge.watson.org> References: <486283B0.3060805@transip.nl> <20080625195523.N29013@fledge.watson.org> <4862BCF5.4070900@transip.nl> <20080626081831.V96707@fledge.watson.org> <20080627090939.M78484@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: net@freebsd.org Subject: Probably not a kernel bug (was: Re: FreeBSD 7.0: sockets stuck in CLOSED state...) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 06:42:25 -0000 On Fri, 27 Jun 2008, Robert Watson wrote: > I've asked Ali to do a bit more debugging and tracing of the application to > see if we can reach any conclusions about this. In particular, if he traces > to a file all file descriptor numbers returned by accept(2), then we can > later compare that file with the leaked descriptors present in > netstat/sockstat and decide whether the application *should* have known they > were open or not. Another public follow-up: Ali has been sending me debugging information privately due to the inclusion of application source code and IP addresses. Tracing of the application suggests that there is an application concurrency bug leading to one socket to be closed twice and another socket to be left open. The bug might be triggering in 7.x but not earlier releases because of the change to libthr, which can lead to more parallelism/asynchrony in the application. In conclusion: we currently believe that this report of sockets stuck in the CLOSED state is not the result of a kernel bug. If any further information comes to light, I will send a followup. Thanks, Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 07:30:05 2008 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A5B521065684 for ; Sun, 29 Jun 2008 07:30:05 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8C1CD8FC15 for ; Sun, 29 Jun 2008 07:30:05 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m5T7U5a0007975 for ; Sun, 29 Jun 2008 07:30:05 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m5T7U57d007972; Sun, 29 Jun 2008 07:30:05 GMT (envelope-from gnats) Date: Sun, 29 Jun 2008 07:30:05 GMT Message-Id: <200806290730.m5T7U57d007972@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: Shunsuke SHINOMIYA Cc: Subject: Re[2]: kern/125003: incorrect EtherIP header format. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Shunsuke SHINOMIYA List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 07:30:05 -0000 The following reply was made to PR kern/125003; it has been noted by GNATS. From: Shunsuke SHINOMIYA To: Andrew Thompson , Hiroki Sato Cc: freebsd-bugs@FreeBSD.org, bug-followup@FreeBSD.org Subject: Re[2]: kern/125003: incorrect EtherIP header format. Date: Sun, 29 Jun 2008 16:27:33 +0900 Hi, > It is unclear where the interoperability problem comes in. I'm sorry. A fix I submitted was a mistake. > Which would conform to the requirement. Can you describe the problem you > are seeing. FreeBSD's current implementation expects 0x03, 0x00 as EtherIP header, but another implementation(UNIVERGE IX2015, products by NEC, Japan) transmits 0x30, 0x00. Then FreeBSD box discards EtherIP packets. I read RFC3378 and thought 0x30, 0x00 is correct. The result of 'tcpdump -np -x proto etherip' at FreeBSD box is as follows. 192.168.2.37: FreeBSD box 192.168.2.128: IX2015 MAC addresses were replaced with ****. 16:02:40.952832 IP 192.168.2.128 > 192.168.2.37: etherip 344 0x0000: 4500 016c 0098 0000 4061 f2a3 c0a8 0280 0x0010: c0a8 0225 3000 **** **** **** **** **** ~~~~ EtherIP header by IX2015 snip 16:02:48.080826 IP 192.168.2.37 > 192.168.2.128: etherip 108 0x0000: 4500 0080 01f3 0000 1e61 1435 c0a8 0225 0x0010: c0a8 0280 0300 **** **** **** **** **** ~~~~ EtherIP header by FreeBSD snip -- Shunsuke SHINOMIYA From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 07:31:47 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C80CD1065690 for ; Sun, 29 Jun 2008 07:31:47 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from ti-out-0910.google.com (ti-out-0910.google.com [209.85.142.188]) by mx1.freebsd.org (Postfix) with ESMTP id 4CACD8FC0C for ; Sun, 29 Jun 2008 07:31:46 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: by ti-out-0910.google.com with SMTP id d27so524643tid.3 for ; Sun, 29 Jun 2008 00:31:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:received:date:from :to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=EAS6exBJc7SrzIVdv9FAltgziDNSG926yL9EqfU7niU=; b=T/zoN94GBPuSxumaU6/P3pJnhWNqcUmxMxKuyHZOJ4vR9AklXWzeJa09dw9gbkGKli uEit4kJLhar3H7Vb7uMgQEOWQPzHGgRby4ed6A8tLZOlTRcS0ED/80wKNlEyh+uKoUWh 4N1O4wYHB5Uw8lfoX9b/zewv5AYLYD6PbsEJI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=jIkwfhDO/CPkljXMVvhyOUiaY/kLSfVR+NPF1IzXYIE84JsLzlPVIsTR+pcMA/MCyC wjmjv81r62LiXIFNm0FJoIVnvaAlZyOiD86FpHsuCCkUmrsHkBEtSIPxxgBdQcazii+j 330MLSOeJj36dAbL484YFr/nhjb1dFESri0ak= Received: by 10.110.68.10 with SMTP id q10mr3445629tia.37.1214724706043; Sun, 29 Jun 2008 00:31:46 -0700 (PDT) Received: from michelle.cdnetworks.co.kr ( [211.53.35.84]) by mx.google.com with ESMTPS id 22sm7621590tim.10.2008.06.29.00.31.43 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 29 Jun 2008 00:31:44 -0700 (PDT) Received: from michelle.cdnetworks.co.kr (localhost.cdnetworks.co.kr [127.0.0.1]) by michelle.cdnetworks.co.kr (8.13.5/8.13.5) with ESMTP id m5T7TZJp076829 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 29 Jun 2008 16:29:35 +0900 (KST) (envelope-from pyunyh@gmail.com) Received: (from yongari@localhost) by michelle.cdnetworks.co.kr (8.13.5/8.13.5/Submit) id m5T7TWEO076828; Sun, 29 Jun 2008 16:29:32 +0900 (KST) (envelope-from pyunyh@gmail.com) Date: Sun, 29 Jun 2008 16:29:32 +0900 From: Pyun YongHyeon To: "Eugene M. Kim" <20080111.freebsd.org@ab.ote.we.lv> Message-ID: <20080629072932.GA76469@cdnetworks.co.kr> References: <200806270345.m5R3j1BT036253@freefall.freebsd.org> <48649776.9040509@ab.ote.we.lv> <20080627074948.GC67753@cdnetworks.co.kr> <4864A217.3040309@ab.ote.we.lv> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4864A217.3040309@ab.ote.we.lv> User-Agent: Mutt/1.4.2.1i Cc: freebsd-net@FreeBSD.org, yongari@FreeBSD.org Subject: Re: kern/125024: vr(4) does not see incoming multicast packets in non-promiscuous mode (broadcast is fine); breaks IPv6 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 07:31:47 -0000 On Fri, Jun 27, 2008 at 01:17:27AM -0700, Eugene M. Kim wrote: > Pyun YongHyeon wrote: > >I've updated patch again. There was a bug that disabled > >multicasting filter. Back out previous patch and try again. > >The URL is the same as before. > > > > > Regards, > > > Eugene > > > > rtsol still doesn't work with vr0 being in non-promiscuous mode. > However, apparently vr0 picked up router solicitations during system > boot-up, as ifconfig shows the correct prefixes autoconfigured. It > seems something goes wrong in between. 'o 'a > Oops, I was accessing CAM mask register as 8bit register which should be 32bit! Try the patch at the following URL. http://people.freebsd.org/~yongari/vr/vr.cam.patch2 > Eugene -- Regards, Pyun YongHyeon From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 07:43:20 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 255D7106566B for ; Sun, 29 Jun 2008 07:43:20 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from yw-out-2324.google.com (yw-out-2324.google.com [74.125.46.29]) by mx1.freebsd.org (Postfix) with ESMTP id D2EFF8FC0C for ; Sun, 29 Jun 2008 07:43:19 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by yw-out-2324.google.com with SMTP id 9so462979ywe.13 for ; Sun, 29 Jun 2008 00:43:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=KXOPcJMMfP912j3AjkGlaftnNG0KoVpYoyZYi+cyhyU=; b=dZHa99RKggt+dzbaE5RD/O5RAMft58rcs3KodLQq/NPqRA5vD+XAW/LA0hu3FT1EpB sd9V5N9hqAjz1z3eCS77ZclDbiu2JS8CB89vTjJBVHYpdpEyZec+7CdeIGA/w5+DLN1B FYaUqxtriRkN11Pdz7PWvi0XsTsE9C+PJadzI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=wKoDF15wX778tZUdiF9GNojWMpPwKchoDKy44ntrPj60592rzk4Tei1GE0wEet/+j3 3ZXJlGO/qVIuwUe2KD15VKu1Smn3AZ/AMpKXDlfm5GogDzwyQLQciSVKgn2lRARZ2XtA UPiPFhCALgKWkrzqfoOszpAWYNAxAGuYmweAA= Received: by 10.150.137.9 with SMTP id k9mr5807614ybd.216.1214725398742; Sun, 29 Jun 2008 00:43:18 -0700 (PDT) Received: by 10.151.154.17 with HTTP; Sun, 29 Jun 2008 00:43:18 -0700 (PDT) Message-ID: Date: Sun, 29 Jun 2008 00:43:18 -0700 From: "Freddie Cash" To: freebsd-net@freebsd.org In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: Subject: Re: Understanding where dummynet fits into an ipfw ruleset X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 07:43:20 -0000 On Sat, Jun 28, 2008 at 11:22 PM, Ian Smith wrote: > It's not clear to me what's not working from your example rules above? I never said anything wasn't working. I was just looking for information to better understand how things work together, and to get a general feeling of where the queue rules would have to go. > Given using one_pass=1, that should go. And using one_pass=0, you > should only need to also add as say rule 150: > > 150 allow ip from 1.1.1.1 to 2.2.2.2 in recv em0 I'm starting to better understand how one_pass affects things. And I think I get, now, where to put the queue rules. I won't be doing any of the actual testing or implementation until July. I was just looking for more info on how to set things up. > > > Yes I suspect Freddie might want to use pipe rather than queue here too, > > > if just for bandwidth limitation rather than weighted queueing by type > > > of traffic? And is it only wanted for managing the inbound traffic? > > > > No, I want to use queue. I want to create rules to "reserve" > > bandwidth for connections to important servers, as we're moving to > > more web-based applications, and I want to make sure students surfing > > the web don't impact office staff. There will be a single pipe, with > > two queues, one weighted at twice the value of the other. That way, > > if there is no staff traffic, the students get the whole pipe. If > > there is no student traffic, staff get the whole pipe. And if there's > > a mix, then staff traffic is prioritised ahead of student traffic. > > Ok; on rereading your original, I should have realised that. So with a > similar set of rules for the other of staff/students that your above > example deals with, and the right pipe and queue configs, what remains > to do? Sorry to be thick, but I don't see why that wouldn't work .. I never said it wouldn't (or didn't) work. :) -- Freddie Cash fjwcash@gmail.com From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 08:02:28 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 35E481065671 for ; Sun, 29 Jun 2008 08:02:28 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id F09F18FC15 for ; Sun, 29 Jun 2008 08:02:27 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KCroS-0005pw-Gg for freebsd-net@freebsd.org; Sun, 29 Jun 2008 03:59:08 -0400 Message-ID: <4867420D.7090406@gtcomm.net> Date: Sun, 29 Jun 2008 04:04:29 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: FreeBSD Net Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 08:02:28 -0000 This is just a question but who can get more than 400k pps forwarding performance ? I have tested fbsd 6/7/8 so far with many different configs. (all using intel pci-ex nic and SMP) fbsd 7-stable/8(current) seem to be the fastest and always hit this ceiling of 400k pps. Soon as it hits that I get errors galore. Received no buffers, missed packets, rx overruns.. It's because 'em0 taskq' is 90% cpu or so.. Now, while this is happening I have two CPU's 100% idle, and the other two CPUs are about 60%/20% .. So why in the world can't it use more cpus? Simple test setup: packet generator on em0 destination out em1 have to have ip forwarding and fastforwarding on (fastforward definitely makes a big difference, another 100kpps or so, without it can barely hit 300k) Packets are TCP, randomized sources, randomized ports for src and dst, single destination ip. I even tried the yandex driver in FBSD6 but it could barely even get 200k pps and it had a lot of weird issues, and fbsd6 couldn't hit 400k pps by itself. I am not using polling, that seems to make no difference, i tried that too. So question. What can I do for more performance (SMP)? Are there any good kernel options? If I disable ip forwarding i can do 750kpps with no errors because it's not going anywhere..em0 taskq cpu usage is less than half of what it is when it's forwarding. so obviously the issue is somewhere in the forwarding path and fastforwarding greatly helps!! see below. forwarding off: input (em0) output packets errs bytes packets errs bytes colls 757223 0 46947830 1 0 226 0 753551 0 46720166 1 0 178 0 756359 0 46894262 1 0 178 0 757570 0 46969344 1 0 178 0 753724 0 46730830 1 0 178 0 745372 0 46213130 1 0 178 0 (I had to slow down the packet generation to about 420-430kpps) forwarding on: input (em0) output packets errs bytes packets errs bytes colls 285918 151029 17726936 460 0 25410 0 284929 146151 17665602 417 0 22642 0 284253 147000 17623690 442 0 23884 0 285438 147765 17697160 448 0 24316 0 286582 147171 17768088 456 0 24748 0 287194 147088 17806032 422 0 22912 0 285812 141713 17720348 440 0 23884 0 284958 137579 17667412 457 0 25104 0 fastforwarding on: input (em0) output packets errs bytes packets errs bytes colls 399795 22790 24787310 459 0 25130 0 397425 25254 24640354 434 0 23560 0 403223 26937 24999830 431 0 23452 0 396587 21431 24588398 467 0 25288 0 400970 25776 24860144 459 0 24910 0 397819 23657 24664782 432 0 23452 0 406222 27418 25185768 432 0 23506 0 406718 12407 25216520 461 0 25018 0 PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 171 ki31 0K 64K CPU1 1 29:24 100.00% {idle: cpu1} 11 root 171 ki31 0K 64K RUN 0 28:46 100.00% {idle: cpu0} 11 root 171 ki31 0K 64K CPU3 3 24:32 84.62% {idle: cpu3} 0 root -68 0 0K 128K CPU2 2 12:59 84.13% {em0 taskq} 0 root -68 0 0K 128K - 3 2:12 19.92% {em1 taskq} 11 root 171 ki31 0K 64K RUN 2 19:46 19.63% {idle: cpu2} Well if anything.. at least it's a good show of the difference fastforwarding makes!! :) I have options NO_ADAPTIVE_MUTEXES ## Improve routing performance? options STOP_NMI # Stop CPUS using NMI instead of IPI no IPV6 no firewall loaded no netgraph HZ is 4000 em driver is 4096 on receive buffers using VLAN devices (em1 output) Tested on Xeon and Opteron processor Don't have exact results. Above results are dual opteron 2212 with freebsd current FreeBSD 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Sat Jun 28 23:37:39 CDT 2008 Well I'm curious of the results of others.. Thanks for reading!! :) From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 08:22:35 2008 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 45D16106566C; Sun, 29 Jun 2008 08:22:35 +0000 (UTC) (envelope-from hrs@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1E15B8FC15; Sun, 29 Jun 2008 08:22:35 +0000 (UTC) (envelope-from hrs@FreeBSD.org) Received: from freefall.freebsd.org (hrs@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m5T8MYUd019535; Sun, 29 Jun 2008 08:22:35 GMT (envelope-from hrs@freefall.freebsd.org) Received: (from hrs@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m5T8MYdw019531; Sun, 29 Jun 2008 08:22:34 GMT (envelope-from hrs) Date: Sun, 29 Jun 2008 08:22:34 GMT Message-Id: <200806290822.m5T8MYdw019531@freefall.freebsd.org> To: shino@fornext.org, hrs@FreeBSD.org, freebsd-net@FreeBSD.org From: hrs@FreeBSD.org Cc: Subject: Re: kern/125003: [gif] incorrect EtherIP header format. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 08:22:35 -0000 Synopsis: [gif] incorrect EtherIP header format. State-Changed-From-To: feedback->open State-Changed-By: hrs State-Changed-When: Sun Jun 29 08:21:04 UTC 2008 State-Changed-Why: We have a concrete case now. http://www.freebsd.org/cgi/query-pr.cgi?pr=125003 From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 09:07:32 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 28B511065675 for ; Sun, 29 Jun 2008 09:07:32 +0000 (UTC) (envelope-from andrew@modulus.org) Received: from email.octopus.com.au (host-122-100-2-232.octopus.com.au [122.100.2.232]) by mx1.freebsd.org (Postfix) with ESMTP id DEB238FC13 for ; Sun, 29 Jun 2008 09:07:31 +0000 (UTC) (envelope-from andrew@modulus.org) Received: by email.octopus.com.au (Postfix, from userid 1002) id 251A717214; Sun, 29 Jun 2008 19:07:29 +1000 (EST) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on email.octopus.com.au X-Spam-Level: X-Spam-Status: No, score=-1.4 required=10.0 tests=ALL_TRUSTED autolearn=failed version=3.2.3 Received: from [10.1.50.60] (138.21.96.58.exetel.com.au [58.96.21.138]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: admin@email.octopus.com.au) by email.octopus.com.au (Postfix) with ESMTP id E4FD2171BF; Sun, 29 Jun 2008 19:07:24 +1000 (EST) Message-ID: <48675095.7020008@modulus.org> Date: Sun, 29 Jun 2008 19:06:29 +1000 From: Andrew Snow User-Agent: Thunderbird 2.0.0.14 (X11/20080523) MIME-Version: 1.0 To: Paul References: <4867420D.7090406@gtcomm.net> In-Reply-To: <4867420D.7090406@gtcomm.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 09:07:32 -0000 The "em" driver currently only has a single worker/queue so will only use one CPU to process packets. However I remember reading that multi-threaded version of the driver is being worked on and is "coming soon", but there is no known ETA yet. I see you mentioned that you played with the receive descriptors and set that to 4096, which is good if your chipset supports it. But I can't see that you mentioned playing around with hw.em.rx_int_delay or tx_int_delay - have you tried this? By default rx_int_delay is disabled but it has the capability to lower CPU consumption if you enable it and it works on your chipset. By my reckoning, if you are aiming for a million pps, thats microsecond per packet, so you can afford to increase the delay quite alot, try 50 or 100 or even more? - Andrew From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 10:00:17 2008 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0E1D510656D6 for ; Sun, 29 Jun 2008 10:00:17 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id A00588FC13 for ; Sun, 29 Jun 2008 10:00:16 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m5TA0GUJ027437 for ; Sun, 29 Jun 2008 10:00:16 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m5TA0GGr027434; Sun, 29 Jun 2008 10:00:16 GMT (envelope-from gnats) Date: Sun, 29 Jun 2008 10:00:16 GMT Message-Id: <200806291000.m5TA0GGr027434@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: Hiroki Sato Cc: Subject: Re: kern/125003: incorrect EtherIP header format. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Hiroki Sato List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 10:00:17 -0000 The following reply was made to PR kern/125003; it has been noted by GNATS. From: Hiroki Sato To: shino@fornext.org Cc: thompsa@FreeBSD.org, freebsd-bugs@FreeBSD.org, bug-followup@FreeBSD.org, hrs@FreeBSD.org Subject: Re: kern/125003: incorrect EtherIP header format. Date: Sun, 29 Jun 2008 18:57:11 +0900 (JST) ----Security_Multipart(Sun_Jun_29_18_57_11_2008_878)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Shunsuke SHINOMIYA wrote in <20080629150108.6783.A2D40D1E@fornext.org>: sh> FreeBSD's current implementation expects 0x03, 0x00 as EtherIP header, sh> but another implementation(UNIVERGE IX2015, products by NEC, Japan) sh> transmits 0x30, 0x00. Then FreeBSD box discards EtherIP packets. Could you let me know if the following patch solves your symptom or not? It can be applied to 8.x and 7.x: http://people.allbsd.org/~hrs/FreeBSD/gif.diff -- | Hiroki SATO ----Security_Multipart(Sun_Jun_29_18_57_11_2008_878)-- Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (FreeBSD) iEYEABECAAYFAkhnXHcACgkQTyzT2CeTzy1/SgCgueGgx06kNVVrgyR5Ryf2K+0I XJ0AnRbtkaW72joKYjyqGF/jVkKeRWAm =IsCy -----END PGP SIGNATURE----- ----Security_Multipart(Sun_Jun_29_18_57_11_2008_878)---- From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 10:56:25 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 785741065688 for ; Sun, 29 Jun 2008 10:56:25 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id D604B8FC20 for ; Sun, 29 Jun 2008 10:56:24 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 22248 invoked from network); 29 Jun 2008 12:56:22 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 29 Jun 2008 12:56:22 +0200 Date: Sun, 29 Jun 2008 12:56:22 +0200 (CEST) From: Ingo Flaschberger To: Paul In-Reply-To: <4867420D.7090406@gtcomm.net> Message-ID: References: <4867420D.7090406@gtcomm.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 10:56:25 -0000 Dear Paul, tried interface polling? what hardware system? how are the nic's connected? Kind regards, ingo flaschberger geschaeftsleitung --------------------------- netstorage-crossip-flat:fee powered by crossip communications gmbh --------------------------- sebastian kneipp gasse 1 a-1020 wien fix: +43-1-726 15 22-217 fax: +43-1-726 15 22-111 --------------------------- On Sun, 29 Jun 2008, Paul wrote: > This is just a question but who can get more than 400k pps forwarding > performance ? > I have tested fbsd 6/7/8 so far with many different configs. (all using intel > pci-ex nic and SMP) > fbsd 7-stable/8(current) seem to be the fastest and always hit this ceiling > of 400k pps. Soon as it hits that I get errors galore. > Received no buffers, missed packets, rx overruns.. It's because 'em0 taskq' > is 90% cpu or so.. > Now, while this is happening I have two CPU's 100% idle, and the other two > CPUs are about 60%/20% .. > So why in the world can't it use more cpus? Simple test setup: > packet generator on em0 > destination out em1 > have to have ip forwarding and fastforwarding on (fastforward definitely > makes a big difference, another 100kpps or so, without it can barely hit > 300k) > Packets are TCP, randomized sources, randomized ports for src and dst, single > destination ip. > I even tried the yandex driver in FBSD6 but it could barely even get 200k pps > and it had a lot of weird issues, and fbsd6 couldn't hit 400k pps by itself. > I am not using polling, that seems to make no difference, i tried that too. > So question. What can I do for more performance (SMP)? Are there any good > kernel options? > If I disable ip forwarding i can do 750kpps with no errors because it's not > going anywhere..em0 taskq cpu usage is less than half of what it is when it's > forwarding. so obviously the issue is somewhere in the forwarding path and > fastforwarding greatly helps!! see below. > forwarding off: > input (em0) output > packets errs bytes packets errs bytes colls > 757223 0 46947830 1 0 226 0 > 753551 0 46720166 1 0 178 0 > 756359 0 46894262 1 0 178 0 > 757570 0 46969344 1 0 178 0 > 753724 0 46730830 1 0 178 0 > 745372 0 46213130 1 0 178 0 > > > (I had to slow down the packet generation to about 420-430kpps) > forwarding on: > input (em0) output > packets errs bytes packets errs bytes colls > 285918 151029 17726936 460 0 25410 0 > 284929 146151 17665602 417 0 22642 0 > 284253 147000 17623690 442 0 23884 0 > 285438 147765 17697160 448 0 24316 0 > 286582 147171 17768088 456 0 24748 0 > 287194 147088 17806032 422 0 22912 0 > 285812 141713 17720348 440 0 23884 0 > 284958 137579 17667412 457 0 25104 0 > > fastforwarding on: > > input (em0) output > packets errs bytes packets errs bytes colls > 399795 22790 24787310 459 0 25130 0 > 397425 25254 24640354 434 0 23560 0 > 403223 26937 24999830 431 0 23452 0 > 396587 21431 24588398 467 0 25288 0 > 400970 25776 24860144 459 0 24910 0 > 397819 23657 24664782 432 0 23452 0 > 406222 27418 25185768 432 0 23506 0 > 406718 12407 25216520 461 0 25018 0 > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 11 root 171 ki31 0K 64K CPU1 1 29:24 100.00% {idle: cpu1} > 11 root 171 ki31 0K 64K RUN 0 28:46 100.00% {idle: cpu0} > 11 root 171 ki31 0K 64K CPU3 3 24:32 84.62% {idle: cpu3} > 0 root -68 0 0K 128K CPU2 2 12:59 84.13% {em0 taskq} > 0 root -68 0 0K 128K - 3 2:12 19.92% {em1 taskq} > 11 root 171 ki31 0K 64K RUN 2 19:46 19.63% {idle: cpu2} > > > > Well if anything.. at least it's a good show of the difference fastforwarding > makes!! :) > I have > options NO_ADAPTIVE_MUTEXES ## Improve routing performance? > options STOP_NMI # Stop CPUS using NMI instead of IPI > no IPV6 > no firewall loaded > no netgraph > HZ is 4000 > em driver is 4096 on receive buffers > using VLAN devices (em1 output) > Tested on Xeon and Opteron processor > Don't have exact results. > Above results are dual opteron 2212 with freebsd current > FreeBSD 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Sat Jun 28 23:37:39 CDT 2008 > Well I'm curious of the results of others.. > > Thanks for reading!! :) > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 11:30:30 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A485B1065681 for ; Sun, 29 Jun 2008 11:30:30 +0000 (UTC) (envelope-from jbut@swin.edu.au) Received: from swin.edu.au (gpo4.cc.swin.edu.au [136.186.1.224]) by mx1.freebsd.org (Postfix) with ESMTP id 4140C8FC17 for ; Sun, 29 Jun 2008 11:30:29 +0000 (UTC) (envelope-from jbut@swin.edu.au) Received: from [192.168.101.17] (jbut.caia.swin.edu.au [136.186.228.20]) by swin.edu.au (8.14.1/8.13.1) with ESMTP id m5TAb1HD027496 for ; Sun, 29 Jun 2008 20:37:01 +1000 Message-ID: <486765C7.1010409@swin.edu.au> Date: Sun, 29 Jun 2008 20:36:55 +1000 From: Jason But User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.4 required=5.0 tests=ALL_TRUSTED,DRUGS_SLEEP autolearn=disabled version=3.1.9 X-Spam-Checker-Version: SpamAssassin 3.1.9 (2007-02-13) on gpo4.cc.swin.edu.au Subject: Code release of ipfw NAT support for SCTP in FreeBSD-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 11:30:30 -0000 The Centre for Advanced Internet Architectures (CAIA - http://caia.swin.edu.au) is proud to announce the release of alias_sctp version 0.1, a SCTP NAT patch to FreeBSD 8.x. Alias_sctp provides SCTP NAT functionality to the ipfw/ipfw_nat/libalias suite. It is part of the CAIA SONATA project (http://caia.swin.edu.au/urp/sonata). The code has been intentionally kept as separate as possible from the base modules to aid testing and debugging, and make it easier to port to other systems. This project has been made possible in part by a grant from the Cisco University Research Program Fund at Community Foundation Silicon Valley. We welcome and value feedback and comments. Please forward feedback to dahayes@swin.edu.au and jbut@swin.edu.au Download patch from http://caia.swin.edu.au/urp/sonata/downloads.html Features of alias_sctp version 0.1: - Basic configuration through "ipfw nat ... config" commands. - Forwarding of incoming SCTP associations through "ipfw nat ... redirect_addr ..." commands. - A variety of log levels (currently #define, but sysctl in version 0.2). - Stateful SCTP association management. 12345678901234567890123456789012345678901234567890123456789012345678901234567890 - Tested on single-homed hosts, but should work when the multi-homed host is on the global side of the NAT (same mechanism for address translation). - Dynamic hash table size allocation (currently #define, but sysctl in version 0.2). - Initial testing has been for up to 10000 concurrent flows arriving and leaving at about 2000/second. Tested for periods of up to 72 hours. Features in the pipline for further releases: - Sysctl interface for logging, timeouts, hash table size. Status - mostly complete. - Port forwarding and load sharing. Status - mostly complete. - Support for, soon to be specified, enhancements of SCTP to aid interworking with NATs. - New AddIP ASCONF chunks. Status - preliminary coding and investigation. (Requires finalised standards to be completed) - AbortM and ErrorM NAT originated messages. Status - preliminary coding, with work starting on the ipfw send interface - IPv6 support. Status - preliminary investigation. - Global IP address tracing. Status - preliminary investigation. Other tasks: - Exaustive testing of the various configurations and scenarios. - Stress and load testing. - Performance analysis. Jason -- ---------- Dr. Jason But Lecturer Telecommunications Engineering Academic Group Faculty of Information and Communication Technologies Swinburne University of Technology http://www.swinburne.edu.au/ict/telecommshome.htm From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 11:40:04 2008 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 10665106566C for ; Sun, 29 Jun 2008 11:40:04 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id F3B748FC1B for ; Sun, 29 Jun 2008 11:40:03 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m5TBe3jX040357 for ; Sun, 29 Jun 2008 11:40:03 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m5TBe3QL040356; Sun, 29 Jun 2008 11:40:03 GMT (envelope-from gnats) Date: Sun, 29 Jun 2008 11:40:03 GMT Message-Id: <200806291140.m5TBe3QL040356@freefall.freebsd.org> To: freebsd-net@FreeBSD.org From: Shunsuke SHINOMIYA Cc: Subject: Re[2]: kern/125003: incorrect EtherIP header format. X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Shunsuke SHINOMIYA List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 11:40:04 -0000 The following reply was made to PR kern/125003; it has been noted by GNATS. From: Shunsuke SHINOMIYA To: Hiroki Sato Cc: thompsa@FreeBSD.org, freebsd-bugs@FreeBSD.org, bug-followup@FreeBSD.org Subject: Re[2]: kern/125003: incorrect EtherIP header format. Date: Sun, 29 Jun 2008 20:38:51 +0900 --------_4867660B00000000A846_MULTIPART_MIXED_ Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: 7bit Hi, > Could you let me know if the following patch solves your symptom or > not? It can be applied to 8.x and 7.x: > > http://people.allbsd.org/~hrs/FreeBSD/gif.diff Thank you. I applied your patch to RELENG_6_3 and modified netinet6/in6_gif.c manually(because of tab expansion?). I had 2 problems. One is syntax error around `eip_ver:4,' when BYTE_ORDER is LITTELE_ENDIAN. Another one is 2 octet padding for struct etherip_header. To solve this, I specified __packed attribute for this structure. Attached patch based on yours is for RELENG_6_3. Patched implementation works well with IX2015. 192.168.2.37: FreeBSD box 192.168.2.128: IX2015 20:22:55.540804 IP 192.168.2.128 > 192.168.2.37: etherip 76 0x0000: 4500 0060 076b 0000 4061 ecdc c0a8 0280 0x0010: c0a8 0225 3000 **** **** **** **** **** 0x0020: **** 0800 4500 003c 0cc0 0000 8001 a82e 0x0030: c0a8 0281 c0a8 0201 0800 4753 0001 0608 0x0040: 6162 6364 6566 6768 696a 6b6c 6d6e 6f70 0x0050: 7172 20:22:55.541815 IP 192.168.2.37 > 192.168.2.128: etherip 76 0x0000: 4500 0060 3511 0000 1e61 e136 c0a8 0225 0x0010: c0a8 0280 3000 **** **** **** **** **** 0x0020: **** 0800 4500 003c 319b 0000 4001 c353 0x0030: c0a8 0201 c0a8 0281 0000 4f53 0001 0608 0x0040: 6162 6364 6566 6768 696a 6b6c 6d6e 6f70 0x0050: 7172 -- Shunsuke SHINOMIYA --------_4867660B00000000A846_MULTIPART_MIXED_ Content-Type: application/octet-stream; name="gif6.diff" Content-Disposition: attachment; filename="gif6.diff" Content-Transfer-Encoding: base64 LS0tIG5ldC9pZl9naWYuaC5vcmlnCTIwMDYtMDItMDEgMDA6NTY6NDYuMDAwMDAwMDAwICswOTAw CisrKyBuZXQvaWZfZ2lmLmgJMjAwOC0wNi0yOSAxOTozNjo0MC4wMDAwMDAwMDAgKzA5MDAKQEAg LTkzLDEyICs5MywxNyBAQAogI2RlZmluZQlNVEFHX0dJRl9DQUxMRUQJMAogCiBzdHJ1Y3QgZXRo ZXJpcF9oZWFkZXIgewotCXVfaW50OF90IGVpcF92ZXI7CS8qIHZlcnNpb24vcmVzZXJ2ZWQgKi8K LQl1X2ludDhfdCBlaXBfcGFkOwkvKiByZXF1aXJlZCBwYWRkaW5nIGJ5dGUgKi8KLX07Ci0jZGVm aW5lIEVUSEVSSVBfVkVSX1ZFUlNfTUFTSyAgIDB4MGYKLSNkZWZpbmUgRVRIRVJJUF9WRVJfUlNW RF9NQVNLICAgMHhmMAotI2RlZmluZSBFVEhFUklQX1ZFUlNJT04gICAgICAgICAweDAzCisjaWYg QllURV9PUkRFUiA9PSBMSVRUTEVfRU5ESUFOCisJdV9pbnQJZWlwX3Jlc3ZsOjQsCS8qIHJlc2Vy dmVkICAqLworCQllaXBfdmVyOjQ7CS8qIHZlcnNpb24gKi8KKyNlbmRpZgorI2lmIEJZVEVfT1JE RVIgPT0gQklHX0VORElBTgorCXVfaW50CWVpcF92ZXI6NCwJLyogdmVyc2lvbiAqLworCQllaXBf cmVzdmw6NDsJLyogcmVzZXJ2ZWQgICovCisjZW5kaWYKKwl1X2ludDhfdCBlaXBfcmVzdmg7CS8q IHJlc2VydmVkICAqLworfSBfX3BhY2tlZDsKKyNkZWZpbmUgRVRIRVJJUF9WRVJTSU9OICAgICAg ICAgMHgzCiAKIC8qIFByb3RvdHlwZXMgKi8KIHZvaWQgZ2lmX2lucHV0KHN0cnVjdCBtYnVmICos IGludCwgc3RydWN0IGlmbmV0ICopOwotLS0gbmV0aW5ldC9pbl9naWYuYy5vcmlnCTIwMDYtMDIt MDEgMDA6NTY6NDYuMDAwMDAwMDAwICswOTAwCisrKyBuZXRpbmV0L2luX2dpZi5jCTIwMDgtMDYt MjkgMTk6MjI6NDguMDAwMDAwMDAwICswOTAwCkBAIC0xNDcsOCArMTQ3LDkgQEAKICNlbmRpZiAv KiBJTkVUNiAqLwogCWNhc2UgQUZfTElOSzoKICAJCXByb3RvID0gSVBQUk9UT19FVEhFUklQOwot IAkJZWlwaGRyLmVpcF92ZXIgPSBFVEhFUklQX1ZFUlNJT04gJiBFVEhFUklQX1ZFUl9WRVJTX01B U0s7Ci0gCQllaXBoZHIuZWlwX3BhZCA9IDA7CisgCQllaXBoZHIuZWlwX3ZlciA9IEVUSEVSSVBf VkVSU0lPTjsKKyAJCWVpcGhkci5laXBfcmVzdmwgPSAwOworIAkJZWlwaGRyLmVpcF9yZXN2aCA9 IDA7CiAgCQkvKiBwcmVwZW5kIEV0aGVybmV0LWluLUlQIGhlYWRlciAqLwogIAkJTV9QUkVQRU5E KG0sIHNpemVvZihzdHJ1Y3QgZXRoZXJpcF9oZWFkZXIpLCBNX0RPTlRXQUlUKTsKICAJCWlmICht ICYmIG0tPm1fbGVuIDwgc2l6ZW9mKHN0cnVjdCBldGhlcmlwX2hlYWRlcikpCi0tLSBuZXRpbmV0 Ni9pbjZfZ2lmLmMub3JpZwkyMDA2LTAyLTAxIDAwOjU2OjQ3LjAwMDAwMDAwMCArMDkwMAorKysg bmV0aW5ldDYvaW42X2dpZi5jCTIwMDgtMDYtMjkgMTk6MjQ6NDkuMDAwMDAwMDAwICswOTAwCkBA IC0xNDAsOCArMTQwLDkgQEAKICNlbmRpZgogCWNhc2UgQUZfTElOSzoKICAJCXByb3RvID0gSVBQ Uk9UT19FVEhFUklQOwotIAkJZWlwaGRyLmVpcF92ZXIgPSBFVEhFUklQX1ZFUlNJT04gJiBFVEhF UklQX1ZFUl9WRVJTX01BU0s7Ci0gCQllaXBoZHIuZWlwX3BhZCA9IDA7CisgCQllaXBoZHIuZWlw X3ZlciA9IEVUSEVSSVBfVkVSU0lPTjsKKyAJCWVpcGhkci5laXBfcmVzdmwgPSAwOworIAkJZWlw aGRyLmVpcF9yZXN2aCA9IDA7CiAgCQkvKiBwcmVwZW5kIEV0aGVybmV0LWluLUlQIGhlYWRlciAq LwogIAkJTV9QUkVQRU5EKG0sIHNpemVvZihzdHJ1Y3QgZXRoZXJpcF9oZWFkZXIpLCBNX0RPTlRX QUlUKTsKICAJCWlmIChtICYmIG0tPm1fbGVuIDwgc2l6ZW9mKHN0cnVjdCBldGhlcmlwX2hlYWRl cikpCg== --------_4867660B00000000A846_MULTIPART_MIXED_-- From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 11:58:27 2008 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 03492106567A; Sun, 29 Jun 2008 11:58:27 +0000 (UTC) (envelope-from ali@transip.nl) Received: from relay0.transip.nl (relay0.transip.nl [80.69.67.21]) by mx1.freebsd.org (Postfix) with ESMTP id BCAB48FC0C; Sun, 29 Jun 2008 11:58:26 +0000 (UTC) (envelope-from ali@transip.nl) Received: from [192.168.0.3] (ip86-50-212-87.adsl2.static.versatel.nl [87.212.50.86]) by relay0.transip.nl (Postfix) with ESMTP id 1D70310310A; Sun, 29 Jun 2008 13:58:24 +0200 (CEST) Message-ID: <486778CE.4000103@transip.nl> Date: Sun, 29 Jun 2008 13:58:06 +0200 From: Ali Niknam Organization: Transip BV User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Robert Watson References: <486283B0.3060805@transip.nl> <20080625195523.N29013@fledge.watson.org> <4862BCF5.4070900@transip.nl> <20080626081831.V96707@fledge.watson.org> <20080627090939.M78484@fledge.watson.org> <20080629073519.D10134@fledge.watson.org> In-Reply-To: <20080629073519.D10134@fledge.watson.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: net@freebsd.org Subject: Re: Probably not a kernel bug (was: Re: FreeBSD 7.0: sockets stuck in CLOSED state...) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 11:58:27 -0000 Hi Guys, > Another public follow-up: Ali has been sending me debugging information > privately due to the inclusion of application source code and IP > addresses. Tracing of the application suggests that there is an > application concurrency bug leading to one socket to be closed twice and > another socket to be left open. The bug might be triggering in 7.x but > not earlier releases because of the change to libthr, which can lead to > more parallelism/asynchrony in the application. > After more testing I can with almost absolute certainty conclude that it is indeed an application bug. Reasons that it did not manifest itself earlier most probably are due to the much increased performance of BSD 7, making a certain race condition likely where it once was improbable. I want to thank all who've helped track this issue down! Kind Regards, Ali From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 15:24:24 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 941F91065671 for ; Sun, 29 Jun 2008 15:24:24 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 553608FC15 for ; Sun, 29 Jun 2008 15:24:24 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KCyi6-00023Y-EP; Sun, 29 Jun 2008 11:21:02 -0400 Message-ID: <4867A9A1.9070507@gtcomm.net> Date: Sun, 29 Jun 2008 11:26:25 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , andrew@modulus.org Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 15:24:24 -0000 Polling makes no difference.. It uses the cpus in a slightly different way but the pps rate is similar.. I tried different HZ settings, I edited kern_poll so i could have a burst max of 8000.. Polling doesn't do anything any more. The only thing I noticed it does is lower the latency on packets when the cpu is idle (i.e. not many pps going through) Hardware system is dual opteron 2212 with intel pci express server NIC dual port. em0: Flow control watermarks high = 30720 low = 29220 em0: tx_int_delay = 66, tx_abs_int_delay = 66 em0: rx_int_delay = 64, rx_abs_int_delay = 98 em0: fifo workaround = 0, fifo_reset_count = 0 input (em0) output packets errs bytes packets errs bytes colls 384700 32541 23851406 457 0 24876 0 385543 25403 23903670 459 0 24910 0 383906 24245 23802188 435 0 23738 0 383656 25182 23786676 427 0 23128 0 em0: tx_int_delay = 66, tx_abs_int_delay = 66 em0: rx_int_delay = 0, rx_abs_int_delay = 98 em0: fifo workaround = 0, fifo_reset_count = 0 input (em0) output packets errs bytes packets errs bytes colls 393787 11217 24414800 461 0 25012 0 390227 16909 24194076 439 0 23776 0 389938 15321 24176158 433 0 23506 0 388685 19562 24098474 449 0 24370 0 392908 11242 24360300 465 0 25234 0 387329 19426 24014402 440 0 23938 0 Ingo Flaschberger wrote: > Dear Paul, > > tried interface polling? > > what hardware system? how are the nic's connected? > > Kind regards, > ingo flaschberger > > geschaeftsleitung > --------------------------- > netstorage-crossip-flat:fee > powered by > crossip communications gmbh > --------------------------- > sebastian kneipp gasse 1 > a-1020 wien > fix: +43-1-726 15 22-217 > fax: +43-1-726 15 22-111 > --------------------------- > On Sun, 29 Jun 2008, Paul wrote: > >> This is just a question but who can get more than 400k pps forwarding >> performance ? >> I have tested fbsd 6/7/8 so far with many different configs. (all >> using intel pci-ex nic and SMP) >> fbsd 7-stable/8(current) seem to be the fastest and always hit this >> ceiling of 400k pps. Soon as it hits that I get errors galore. >> Received no buffers, missed packets, rx overruns.. It's because 'em0 >> taskq' is 90% cpu or so.. >> Now, while this is happening I have two CPU's 100% idle, and the >> other two CPUs are about 60%/20% .. >> So why in the world can't it use more cpus? Simple test setup: >> packet generator on em0 >> destination out em1 >> have to have ip forwarding and fastforwarding on (fastforward >> definitely makes a big difference, another 100kpps or so, without it >> can barely hit 300k) >> Packets are TCP, randomized sources, randomized ports for src and >> dst, single destination ip. >> I even tried the yandex driver in FBSD6 but it could barely even get >> 200k pps and it had a lot of weird issues, and fbsd6 couldn't hit >> 400k pps by itself. >> I am not using polling, that seems to make no difference, i tried >> that too. >> So question. What can I do for more performance (SMP)? Are there any >> good kernel options? >> If I disable ip forwarding i can do 750kpps with no errors because >> it's not going anywhere..em0 taskq cpu usage is less than half of >> what it is when it's forwarding. so obviously the issue is somewhere >> in the forwarding path and fastforwarding greatly helps!! see below. >> forwarding off: >> input (em0) output >> packets errs bytes packets errs bytes colls >> 757223 0 46947830 1 0 226 0 >> 753551 0 46720166 1 0 178 0 >> 756359 0 46894262 1 0 178 0 >> 757570 0 46969344 1 0 178 0 >> 753724 0 46730830 1 0 178 0 >> 745372 0 46213130 1 0 178 0 >> >> >> (I had to slow down the packet generation to about 420-430kpps) >> forwarding on: >> input (em0) output >> packets errs bytes packets errs bytes colls >> 285918 151029 17726936 460 0 25410 0 >> 284929 146151 17665602 417 0 22642 0 >> 284253 147000 17623690 442 0 23884 0 >> 285438 147765 17697160 448 0 24316 0 >> 286582 147171 17768088 456 0 24748 0 >> 287194 147088 17806032 422 0 22912 0 >> 285812 141713 17720348 440 0 23884 0 >> 284958 137579 17667412 457 0 25104 0 >> >> fastforwarding on: >> >> input (em0) output >> packets errs bytes packets errs bytes colls >> 399795 22790 24787310 459 0 25130 0 >> 397425 25254 24640354 434 0 23560 0 >> 403223 26937 24999830 431 0 23452 0 >> 396587 21431 24588398 467 0 25288 0 >> 400970 25776 24860144 459 0 24910 0 >> 397819 23657 24664782 432 0 23452 0 >> 406222 27418 25185768 432 0 23506 0 >> 406718 12407 25216520 461 0 25018 0 >> >> PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND >> 11 root 171 ki31 0K 64K CPU1 1 29:24 100.00% {idle: cpu1} >> 11 root 171 ki31 0K 64K RUN 0 28:46 100.00% {idle: cpu0} >> 11 root 171 ki31 0K 64K CPU3 3 24:32 84.62% {idle: cpu3} >> 0 root -68 0 0K 128K CPU2 2 12:59 84.13% {em0 taskq} >> 0 root -68 0 0K 128K - 3 2:12 19.92% {em1 taskq} >> 11 root 171 ki31 0K 64K RUN 2 19:46 19.63% {idle: cpu2} >> >> >> >> Well if anything.. at least it's a good show of the difference >> fastforwarding makes!! :) >> I have >> options NO_ADAPTIVE_MUTEXES ## Improve routing performance? >> options STOP_NMI # Stop CPUS using NMI instead >> of IPI >> no IPV6 >> no firewall loaded >> no netgraph >> HZ is 4000 >> em driver is 4096 on receive buffers >> using VLAN devices (em1 output) >> Tested on Xeon and Opteron processor >> Don't have exact results. >> Above results are dual opteron 2212 with freebsd current >> FreeBSD 8.0-CURRENT FreeBSD 8.0-CURRENT #0: Sat Jun 28 23:37:39 CDT >> 2008 Well I'm curious of the results of others.. >> >> Thanks for reading!! :) >> >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 15:40:00 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A95821065670 for ; Sun, 29 Jun 2008 15:40:00 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id 726ED8FC1A for ; Sun, 29 Jun 2008 15:40:00 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.2/8.14.2) with ESMTP id m5TFdwUW033747; Sun, 29 Jun 2008 11:39:58 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m5TFdvCO075285 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 29 Jun 2008 11:39:57 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200806291539.m5TFdvCO075285@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Sun, 29 Jun 2008 11:40:03 -0400 To: Paul , FreeBSD Net From: Mike Tancsa In-Reply-To: <4867420D.7090406@gtcomm.net> References: <4867420D.7090406@gtcomm.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 15:40:00 -0000 At 04:04 AM 6/29/2008, Paul wrote: >This is just a question but who can get more than 400k pps >forwarding performance ? >I have tested fbsd 6/7/8 so far with many different configs. (all >using intel pci-ex nic and SMP) >fbsd 7-stable/8(current) seem to be the fastest and always hit this >ceiling of 400k pps. Soon as it hits that I get errors galore. In your testing of current, or even RELENG_7, did you ever solve the problem of http://www.freebsd.org/cgi/query-pr.cgi?pr=123621&cat=kern that you also ran into in a previous thread ? I wonder if the generation of all those seemingly bogus RTM_MISS messages is hampering performance. ---Mike From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 15:45:15 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 66A851065673 for ; Sun, 29 Jun 2008 15:45:15 +0000 (UTC) (envelope-from max@love2party.net) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.187]) by mx1.freebsd.org (Postfix) with ESMTP id EC6528FC19 for ; Sun, 29 Jun 2008 15:45:14 +0000 (UTC) (envelope-from max@love2party.net) Received: from vampire.homelinux.org (dslb-088-066-056-045.pools.arcor-ip.net [88.66.56.45]) by mrelayeu.kundenserver.de (node=mrelayeu8) with ESMTP (Nemesis) id 0ML31I-1KCz5V2JEy-0002Fw; Sun, 29 Jun 2008 17:45:13 +0200 Received: (qmail 19638 invoked from network); 29 Jun 2008 15:42:53 -0000 Received: from myhost.laiers.local (192.168.4.151) by ns1.laiers.local with SMTP; 29 Jun 2008 15:42:53 -0000 From: Max Laier Organization: FreeBSD To: .@babolo.ru Date: Sun, 29 Jun 2008 17:43:14 +0200 User-Agent: KMail/1.9.9 References: <1214651667.267043.71931.nullmailer@cicuta.babolo.ru> In-Reply-To: <1214651667.267043.71931.nullmailer@cicuta.babolo.ru> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200806291743.15021.max@love2party.net> X-Provags-ID: V01U2FsdGVkX1+D0T2TdD9LCzS+FUZXSxLcfOxojFyKBoYXoAO PPpv/Em/hfNN4MvykzoI1gXut9pi12zGtfbGXjfgr79zJ6WVGr Sser+kMLZbKt0OTU4jktw== Cc: freebsd-net@freebsd.org, Giulio Ferro , Alexandre Biancalana Subject: Re: altq on vlan X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 15:45:15 -0000 On Saturday 28 June 2008 13:14:27 .@babolo.ru wrote: > [ Charset ISO-8859-1 unsupported, converting... ] > > > On Friday 27 June 2008 18:57:59 Alexandre Biancalana wrote: > > > On 6/27/08, Max Laier wrote: > > > > You don't need a patch at all. What you do is: Queue on the > > > > physical interface, classify on the vlan interface. It is broken > > > > to allow ALTQ on a virtual interface if you can do it otherwise. > > > > > > > > in pf.conf speak: > > > > > > > > If you have "ifconfig vlanX vlandev bge0 ..." > > > > > > > > altq on bge0 .... queue { vlan0, vlan1, ... } > > > > queue vlan0 ... { vlan0_foo, vlan0_bar, ... } > > > > queue vlan0_foo > > > > queue vlan0_bar > > > > ... > > > > > > > > pass on vlanX ... queue vlanX_foobar > > > > > > > > And there you go. No patch - whatsoever - required here. > > > > > > But the patch simplify the cases where you need one queue per vlan. > > > > NO! It is just wrong! There is no relation between vlan queues on > > the same physical interface and thus you can't guarantee anything! > > Can we please stop with this nonsense and not bring up the patch > > every other month. > > Remember vlan anoter end. > > Vlan queues on the same physical interface has sense. > > Let see typical vlan use: > +--------+ 100M untagged vlan1 > | |--------------.. > +---------+ | | 100M untagged vlan2 > 1G | | 1G tagged | |---------------- > --------+ FreeBSD +------------+ switch | 100M untagged vlan3 > | | | |--------------.. > +---------+ | | 100M untagged vlanN > | |--------------- > +--------+ > > There is noting interesting in common queue on 1G physical interface, > the only right queues are that on vlans when number of > vlans < 10. > > More of that, sum traffic on 1G tagged intervace is limited > by incoming traffic from 1G external interface and > so common queue on 1G tagged interface is not > interesting even when number of vlans > 10. Sorry, but you are completely off track here. If you use one queue per vlan one vlan can easily DoS the rest, because once a packet has passed the queue in the vlan it falls into a common queue with all the others and - as you correctly point out - there is no guarantee that a 1G interface can really sent at 1G all the time. The vlan queues, however, will not get any feedback from the parent about it's real send speed. E.g. a vlan sending *a lot* of tiny packets will dominate the 1G link and thus DoS any other vlan that sends big packets. This you can prevent with a common queue. Now please ... let this die, it's stupid! -- /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 16:46:33 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E655A106567A for ; Sun, 29 Jun 2008 16:46:33 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 98CD98FC1C for ; Sun, 29 Jun 2008 16:46:33 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KCzzc-0003Gz-D1; Sun, 29 Jun 2008 12:43:12 -0400 Message-ID: <4867BCE3.2000909@gtcomm.net> Date: Sun, 29 Jun 2008 12:48:35 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Mike Tancsa References: <4867420D.7090406@gtcomm.net> <200806291539.m5TFdvCO075285@lava.sentex.ca> In-Reply-To: <200806291539.m5TFdvCO075285@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 16:46:34 -0000 [RTM_MISS information added] I have noticed something weird.. It doesn't generate the RTM_MISS with all traffic... Check this out.. flooding packets through the router 11:29:56.147177 IP (tos 0x0, ttl 255, id 51487, offset 0, flags [none], proto TCP (6), length 40) 87.42.160.8.8195 > 10.3.9.50.623: ., cksum 0x999e (incorrect (-> 0xb7e6), 2714784062:2714784062(0) ack 1638282847 win 16384 11:29:56.147182 IP (tos 0x0, ttl 255, id 22197, offset 0, flags [none], proto TCP (6), length 40) 43.253.49.61.13857 > 10.3.9.50.6232: ., cksum 0x51b4 (incorrect (-> 0xeb52), 2685378913:2685378913(0) ack 3576016642 win 16384 11:29:56.147186 IP (tos 0x0, ttl 255, id 19160, offset 0, flags [none], proto TCP (6), length 40) 148.198.237.1.31784 > 10.3.9.50.63: ., cksum 0x7d2b (incorrect (-> 0xcedf), 2790811715:2790811715(0) ack 3733955172 win 16384 11:29:56.147190 IP (tos 0x0, ttl 255, id 45483, offset 0, flags [none], proto TCP (6), length 40) 50.140.153.13.23035 > 10.3.9.50.324: ., cksum 0x5e56 (incorrect (-> 0xdb81), 2403102209:2403102209(0) ack 923149825 win 16384 11:29:56.147194 IP (tos 0x0, ttl 255, id 53653, offset 0, flags [none], proto TCP (6), length 40) 215.20.25.100.18066 > 10.3.9.50.6232: ., cksum 0x592b (incorrect (-> 0x9f26), 3678810149:3678810149(0) ack 916597768 win 16384 11:29:56.147198 IP (tos 0x0, ttl 255, id 25330, offset 0, flags [none], proto TCP (6), length 40) 30.216.125.43.23715 > 10.3.9.50.5325: ., cksum 0x7d62 (incorrect (-> 0xdbb8), 4058239037:4058239037(0) ack 1225812033 win 16384 11:29:56.147210 IP (tos 0x0, ttl 255, id 56031, offset 0, flags [none], proto TCP (6), length 40) 92.202.56.54.1180 > 10.3.9.50.623: ., cksum 0x45fb (incorrect (-> 0xc35d), 1051146817:1051146817(0) ack 1914442290 win 16384 11:29:56.147214 IP (tos 0x0, ttl 255, id 17414, offset 0, flags [none], proto TCP (6), length 40) 243.139.107.120.22763 > 10.3.9.50.63: ., cksum 0x3f12 (incorrect (-> 0x983d), 343363109:343363109(0) ack 2790458180 win 16384 11:29:56.147219 IP (tos 0x0, ttl 255, id 15785, offset 0, flags [none], proto TCP (6), length 40) 193.30.160.14.19071 > 10.3.9.50.324: ., cksum 0x0021 (incorrect (-> 0x3f33), 3909194617:3909194617(0) ack 1335272554 win 16384 11:29:56.147223 IP (tos 0x0, ttl 255, id 36379, offset 0, flags [none], proto TCP (6), length 40) 93.46.193.34.22779 > 10.3.9.50.5325: ., cksum 0x2f95 (incorrect (-> 0x2fb6), 1506935083:1506935083(0) ack 2882181640 win 16384 11:29:56.147228 IP (tos 0x0, ttl 255, id 43639, offset 0, flags [none], proto TCP (6), length 40) 50.78.186.1.1437 > 10.3.9.50.623: ., cksum 0xba44 (incorrect (-> 0xe9d9), 3585077271:3585077271(0) ack 1754157076 win 16384 11:29:56.147232 IP (tos 0x0, ttl 255, id 31282, offset 0, flags [none], proto TCP (6), length 40) 230.61.232.98.9799 > 10.3.9.50.6232: ., cksum 0xe26a (incorrect (-> 0x9caf), 2184338509:2184338509(0) ack 1881236495 win 16384 11:29:56.147242 IP (tos 0x0, ttl 255, id 18266, offset 0, flags [none], proto TCP (6), length 40) 185.133.224.35.18694 > 10.3.9.50.63: ., cksum 0x5bb9 (incorrect (-> 0x3e24), 1265308989:1265308989(0) ack 898540886 win 16384 11:29:56.147247 IP (tos 0x0, ttl 255, id 63295, offset 0, flags [none], proto TCP (6), length 40) 60.149.107.97.14907 > 10.3.9.50.324: ., cksum 0x75ac (incorrect (-> 0xd165), 680418413:680418413(0) ack 1399770970 win 16384 11:29:56.147264 IP (tos 0x0, ttl 255, id 44265, offset 0, flags [none], proto TCP (6), length 40) 226.37.11.125.16380 > 10.3.9.50.5325: ., cksum 0xb763 (incorrect (-> 0x2d10), 1640245563:1640245563(0) ack 3534655349 win 16384 11:29:56.147276 IP (tos 0x0, ttl 255, id 62142, offset 0, flags [none], proto TCP (6), length 40) 76.66.176.108.8173 > 10.3.9.50.623: ., cksum 0xf66b (incorrect (-> 0xadcf), 1200243821:1200243821(0) ack 398846984 win 16384 11:29:56.147281 IP (tos 0x0, ttl 255, id 60663, offset 0, flags [none], proto TCP (6), length 40) 241.35.125.100.4600 > 10.3.9.50.324: ., cksum 0x7299 (incorrect (-> 0x63e8), 4145462333:4145462333(0) ack 1113096518 win 16384 11:29:56.147286 IP (tos 0x0, ttl 255, id 55417, offset 0, flags [none], proto TCP (6), length 40) 152.7.87.5.14990 > 10.3.9.50.6232: ., cksum 0xd955 (incorrect (-> 0xcfc1), 892720934:892720934(0) ack 1334374150 win 16384 ^C11:29:56.147290 IP (tos 0x0, ttl 255, id 29468, offset 0, flags [none], proto TCP (6), length 40) 246.253.188.115.23425 > 10.3.9.50.63: ., cksum 0xf14e (incorrect (-> 0xcaa4), 1030196069:1030196069(0) ack 2661620567 win 16384 these generate RTM_MISS for what looks like every packet : got message of size 160 on Sun Jun 29 11:31:04 2008 RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 160 on Sun Jun 29 11:31:04 2008 RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 160 on Sun Jun 29 11:31:04 2008 RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 160 on Sun Jun 29 11:31:04 2008 RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 160 on Sun Jun 29 11:31:04 2008 RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 160 on Sun Jun 29 11:31:04 2008 RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 160 on Sun Jun 29 11:31:04 2008 RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default And then these packets: 11:32:50.540714 IP (tos 0x0, ttl 128, id 60717, offset 0, flags [DF], proto TCP (6), length 48) 199.253.132.217.1145 > 10.3.9.50.1230: S, cksum 0xe22a (correct), 2254729531:2254729531(0) win 16384 11:32:50.540719 IP (tos 0x0, ttl 128, id 29798, offset 0, flags [DF], proto TCP (6), length 48) 128.201.66.139.1074 > 10.3.9.50.1076: S, cksum 0xbc56 (correct), 699104812:699104812(0) win 16384 11:32:50.540737 IP (tos 0x0, ttl 128, id 24548, offset 0, flags [DF], proto TCP (6), length 48) 64.6.56.150.1137 > 10.3.9.50.1065: S, cksum 0x3d9f (correct), 3231494262:3231494262(0) win 16384 11:32:50.540742 IP (tos 0x0, ttl 128, id 24872, offset 0, flags [DF], proto TCP (6), length 48) 209.236.188.202.1119 > 10.3.9.50.1137: S, cksum 0xb4bb (correct), 3403159757:3403159757(0) win 16384 11:32:50.540746 IP (tos 0x0, ttl 128, id 7860, offset 0, flags [DF], proto TCP (6), length 48) 193.42.23.202.1147 > 10.3.9.50.1047: S, cksum 0x7900 (correct), 2067225130:2067225130(0) win 16384 11:32:50.540751 IP (tos 0x0, ttl 128, id 6369, offset 0, flags [DF], proto TCP (6), length 48) 198.222.98.165.1070 > 10.3.9.50.1197: S, cksum 0x8245 (correct), 1566580196:1566580196(0) win 16384 11:32:50.540759 IP (tos 0x0, ttl 128, id 29494, offset 0, flags [DF], proto TCP (6), length 48) 194.32.233.216.1217 > 10.3.9.50.1035: S, cksum 0x0036 (correct), 608655014:608655014(0) win 16384 11:32:50.540772 IP (tos 0x0, ttl 128, id 57975, offset 0, flags [DF], proto TCP (6), length 48) 205.206.20.208.1220 > 10.3.9.50.1232: S, cksum 0x52fc (correct), 1339728095:1339728095(0) win 16384 11:32:50.540777 IP (tos 0x0, ttl 128, id 5551, offset 0, flags [DF], proto TCP (6), length 48) 4.42.66.89.1142 > 10.3.9.50.1178: S, cksum 0xaa57 (correct), 1309337587:1309337587(0) win 16384 11:32:50.540781 IP (tos 0x0, ttl 128, id 64726, offset 0, flags [DF], proto TCP (6), length 48) 208.166.197.140.1052 > 10.3.9.50.1084: S, cksum 0x4dfd (correct), 3514659298:3514659298(0) win 16384 11:32:50.540786 IP (tos 0x0, ttl 128, id 51126, offset 0, flags [DF], proto TCP (6), length 48) 206.67.34.143.1071 > 10.3.9.50.1100: S, cksum 0xbf84 (correct), 3369643581:3369643581(0) win 16384 11:32:50.540790 IP (tos 0x0, ttl 128, id 59088, offset 0, flags [DF], proto TCP (6), length 48) 210.246.25.177.1277 > 10.3.9.50.1205: S, cksum 0x7080 (correct), 1667720615:1667720615(0) win 16384 11:32:50.540795 IP (tos 0x0, ttl 128, id 2326, offset 0, flags [DF], proto TCP (6), length 48) 129.251.33.231.1154 > 10.3.9.50.1195: S, cksum 0x28e5 (correct), 312297303:312297303(0) win 16384 11:32:50.540799 IP (tos 0x0, ttl 128, id 8268, offset 0, flags [DF], proto TCP (6), length 48) 216.84.213.152.1115 > 10.3.9.50.1027: S, cksum 0x7a81 (correct), 2232908292:2232908292(0) win 16384 11:32:50.540803 IP (tos 0x0, ttl 128, id 15857, offset 0, flags [DF], proto TCP (6), length 48) 199.228.246.182.1053 > 10.3.9.50.1053: S, cksum 0xe187 (correct), 376795414:376795414(0) win 16384 11:32:50.540808 IP (tos 0x0, ttl 128, id 33924, offset 0, flags [DF], proto TCP (6), length 48) 128.92.244.80.1060 > 10.3.9.50.1148: S, cksum 0xc4e1 (correct), 2038527032:2038527032(0) win 16384 11:32:50.540812 IP (tos 0x0, ttl 128, id 12114, offset 0, flags [DF], proto TCP (6), length 48) 64.118.145.169.1252 > 10.3.9.50.1148: S, cksum 0xb270 (correct), 3489780214:3489780214(0) win 16384 11:32:50.540816 IP (tos 0x0, ttl 128, id 23385, offset 0, flags [DF], proto TCP (6), length 48) 211.212.238.186.1217 > 10.3.9.50.1083: S, cksum 0xc082 (correct), 2887120836:2887120836(0) win 16384 11:32:50.540821 IP (tos 0x0, ttl 128, id 10979, offset 0, flags [DF], proto TCP (6), length 48) 24.119.38.75.1088 > 10.3.9.50.1027: S, cksum 0xee10 (correct), 3079816000:3079816000(0) win 16384 11:32:50.540830 IP (tos 0x0, ttl 128, id 44562, offset 0, flags [DF], proto TCP (6), length 48) 193.99.52.25.1249 > 10.3.9.50.1047: S, cksum 0x9ef4 (correct), 1263552303:1263552303(0) win 16384 11:32:50.540835 IP (tos 0x0, ttl 128, id 45332, offset 0, flags [DF], proto TCP (6), length 48) 209.168.140.136.1275 > 10.3.9.50.1053: S, cksum 0x03b5 (correct), 1521904180:1521904180(0) win 16384 11:32:50.540840 IP (tos 0x0, ttl 128, id 10244, offset 0, flags [DF], proto TCP (6), length 48) 198.114.145.88.1214 > 10.3.9.50.1083: S, cksum 0x51a9 (correct), 2907689003:2907689003(0) win 16384 I get no RTM MISS. So this absoutely makes no sense?? It should not depend on the TYPE of packet at ALL. It's just forwarding IP regardless of what's in the TCP header so why on earth would the tcp header many any difference to how it routes.. It should not even be looking at the tcp header unless I have a firewall turned on or any type of rules that would filter based on tcp options. Forwarding a packet doesn't require looking at TCP option bits or even TCP port numbers. I suppose it could be checking the mss or trying to calculate checksums but I would prefer it didn't even do that :/ Raw performance is all I care about at this point, I don't even need to use any firewall at all. It's also funny to notice, that tcpdump does not see those packets as TCP.. tcpdump -n -i em0 tcp does not show either of my 'floods' tcpdump -n -i em1 tcp shows my floods tcpdump -n -i em0 shows them all.. and they say TCP ..lol :) Weird I tell ya! So the bottom line is, why do one type of packet generate RTM_MISS and another doesn't? I haven't confirmed this, but I don't think it does it on 7.0-RELEASE, only 7.0-STABLE and CURRENT. I will try and confirm though. Paul Mike Tancsa wrote: > At 04:04 AM 6/29/2008, Paul wrote: >> This is just a question but who can get more than 400k pps forwarding >> performance ? >> I have tested fbsd 6/7/8 so far with many different configs. (all >> using intel pci-ex nic and SMP) >> fbsd 7-stable/8(current) seem to be the fastest and always hit this >> ceiling of 400k pps. Soon as it hits that I get errors galore. > > In your testing of current, or even RELENG_7, did you ever solve the > problem of > > http://www.freebsd.org/cgi/query-pr.cgi?pr=123621&cat=kern > that you also ran into in a previous thread ? I wonder if the > generation of all those seemingly bogus RTM_MISS messages is hampering > performance. > > ---Mike > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 17:02:50 2008 Return-Path: Delivered-To: net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E0241065678; Sun, 29 Jun 2008 17:02:50 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 29C128FC0C; Sun, 29 Jun 2008 17:02:49 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 5FF2146C27; Sun, 29 Jun 2008 13:02:49 -0400 (EDT) Date: Sun, 29 Jun 2008 18:02:49 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: net@FreeBSD.org, current@FreeBSD.org In-Reply-To: <20080524111715.T64552@fledge.watson.org> Message-ID: <20080629180126.F90836@fledge.watson.org> References: <20080524111715.T64552@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@FreeBSD.org Subject: Re: HEAD UP: non-MPSAFE network drivers to be disabled (was: 8.0 network stack MPsafety goals (fwd)) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 17:02:50 -0000 On Sat, 24 May 2008, Robert Watson wrote: > Just as a reminder, we've just about reached the one month date before > IFF_NEEDSGIANT drivers are disabled in the build. You can find a > description of the general problem and list of specific drivers below. > > As USB work is on-going, I will *not* disable the USB drivers at this time, > but all other drivers in the list below will be disabled on 26 June. They > will remain in the tree, easily accessible for patch distribution and > re-enabling, until October, when any remaining non-MPSAFE drivers will be > deleted in 8.x. FreeBSD 8.0 will not ship with compatibility shims to > support non-MPSAFE network device drivers. An FYI on the state of things here: in the last month, John has updated a number of device drivers to be MPSAFE, and the USB work remains in-flight. I'm holding fire a bit on disabling IFF_NEEDSGIANT while things settle and I catch up on driver state, and will likely send out an update next week regarding which device drivers remain on the kill list, and generally what the status of this project is. Robert N M Watson Computer Laboratory University of Cambridge > > Robert N M Watson Computer Laboratory University of Cambridge > > ---------- Forwarded message ---------- > Date: Sun, 3 Feb 2008 20:59:05 +0000 (GMT) > From: Robert Watson > To: arch@FreeBSD.org > Subject: 8.0 network stack MPsafety goals (fwd) > > > Only a few days after predicted, this is a reminder that IFF_NEEDSGIANT > network drivers are going to stop working in the forseeable future. Please > review the attached driver list, and if you depend on or care about a > Giant-dependent device driver, take action to make sure it doesn't remain on > the list in a month's time! > > (As far as I'm aware, the list has not changed since my December posting.) > > Robert N M Watson > Computer Laboratory > University of Cambridge > > ---------- Forwarded message ---------- > Date: Mon, 24 Dec 2007 10:43:28 +0000 (GMT) > From: Robert Watson > To: arch@FreeBSD.org > Subject: 8.0 network stack MPsafety goals > > > Dear all: > > With the 7.0 release around the corner, many developers are starting to think > about (and in quite a few cases, work on) their goals for 8.0. One of our > on-going kernel projects has been the elimination of the Giant lock, and that > project has transformed into one of optimizating behavior on increasing > numbers of processors. > > In 7.0, despite the noteworth accomplishment of eliminating debug.mpsasfenet > and conditional network stack Gian acquisition, we were unable to fully > eliminate the IFF_NEEDSGIANT flag, which controls the conditional acquisition > of the Giant lock around non-MPSAFE network device drivers. Primarily these > drivers are aging ISA network device drivers, although there are some > exceptions, such as the USB stack. > > This e-mail proposes the elimination of the IFF_NEEDSGIANT flag and > associated infrastructure in FreeBSD 8.0, meaning that all network device > drivers must be able to operate without the Giant lock (largely the case > already). Remaining drivers using the IFF_NEEDSGIANT flag must either be > updated, or less ideally, removed. I propose the following schedule: > > Date Goals > ---- ----- > 26 Dec 2007 Post proposed schedule for flag and infrastructure removal > Post affected driver list > > 26 Jan 2008 Repost proposed schedule for flag and infrastructure removal > Post updated affected driver list > > 26 Feb 2008 Adjust boot-time printf for affect drivers to generate a loud > warning. > Post updated affected driver list > > 26 May 2008 Post HEADS UP of impending driver disabling > Post updated affected driver list > > 26 Jun 2008 Disable build of all drivers requiring IFF_NEEDSGIANT > Post updated affected driver list > > 26 Sep 2008 Post HEADS up of impending driver removal > Post updated affected driver list > > 26 Oct 2008 Delete source of all drivers requiring IFF_NEEDSGIANT > Remove flag and infrastructure > > Here is a list of potentially affected drivers: > > Name Bus Man page description > --- --- -------------------- > ar ISA/PCI synchronous Digi/Arnet device driver > arl ISA Aironet Arlan 655 wireless network adapter driver > awi PCCARD AMD PCnetMobile IEEE 802.11 PCMCIA wireless network > driver > axe USB ASIX Electronics AX88172 USB Ethernet driver > cdce USB USB Communication Device Class Ethernet driver > cnw PCCARD Netwave AirSurfer wireless network driver > cs ISA/PCCARD Ethernet device driver > cue USB CATC USB-EL1210A USB Ethernet driver > ex ISA/PCCARD Ethernet device driver for the Intel EtherExpress > Pro/10 and Pro/10+ > fe CBUS/ISA/PCCARD Fujitsu MB86960A/MB86965A based Ethernet adapters > ic I2C I2C bus system > ie ISA Ethernet device driver > kue USB Kawasaki LSI KL5KUSB101B USB Ethernet driver > oltr ISA/PCI Olicom Token Ring device driver > plip PPBUS printer port Internet Protocol driver > ppp TTY point to point protocol network interface > ray PCCARD Raytheon Raylink/Webgear Aviator PCCard driver > rue USB RealTek RTL8150 USB to Fast Ethernet controller > driver > rum USB Ralink Technology USB IEEE 802.11a/b/g wireless > network device > sbni ISA/PCI Granch SBNI12 leased line modem driver > sbsh PCI Granch SBNI16 SHDSL modem device driver > sl TTY slip network interface > snc ISA/PCCARD National Semiconductor DP8393X SONIC Ethernet adapter > driver > sr ISA/PCI synchronous RISCom/N2 / WANic 400/405 device driver > udav USB Davicom DM9601 USB Ethernet driver > ural USB Ralink Technology RT2500USB IEEE 802.11 driver > xe PCCARD Xircom PCMCIA Ethernet device driver > zyd USB ZyDAS ZD1211/ZD1211B USB IEEE 802.11b/g wireless > network device > > In some cases, the requirement for Giant is a property of a subsystem the > driver depends on as the driver itself; for example, the tty subsystem for > SLIP and PPP, and the USB subsystem for a number of USB ethernet and wireless > drivers. With most of a year before to go on the proposed schedule, my hope > is that we will have lots of time to address these issues, but wanted to get > a roadmap out from a network protocol stack architecture perspective so that > device driver and subsystem authors could have a schedule in mind. > > FYI, the following drivers also reference IFF_NEEDSGIANT, but only in order > to provide their own conditional MPSAFEty, which can be removed without > affecting device driver functionality (I believe): > > Name Bus Man page description > --- --- -------------------- > ce PCI driver for synchronous Cronyx Tau-PCI/32 WAN adapters > cp PCI driver for synchronous Cronyx Tau-PCI WAN adapters > ctau ISA driver for synchronous Cronyx Tau WAN adapters > cx ISA driver for synchronous/asynchronous Cronyx Sigma WAN > adapters > > Developers and users of the above drivers are heavily encouraged to update > the drivers to remove dependence on Giant, and/or make other contingency > plans. > > Robert N M Watson > Computer Laboratory > University of Cambridge > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 18:29:30 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 459C5106566B for ; Sun, 29 Jun 2008 18:29:30 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id E7DAD8FC1C for ; Sun, 29 Jun 2008 18:29:29 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KD1bE-00083T-5C; Sun, 29 Jun 2008 14:26:08 -0400 Message-ID: <4867D503.6080101@gtcomm.net> Date: Sun, 29 Jun 2008 14:31:31 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Mike Tancsa References: <4867420D.7090406@gtcomm.net> <200806291539.m5TFdvCO075285@lava.sentex.ca> <4867BCE3.2000909@gtcomm.net> In-Reply-To: <4867BCE3.2000909@gtcomm.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 18:29:30 -0000 Other interesting behavior: netstat -rs hitting up arrow and enter to get repeated views of it: [root@irc-router1 /usr/src/sys/net]# netstat -rs routing: 0 bad routing redirects 0 dynamically created routes 0 new gateways due to redirects 25691 destinations found unreachable 0 uses of a wildcard route 0 routes not in table but not freed [root@router1 /usr/src/sys/net]# netstat -rs routing: 0 bad routing redirects 0 dynamically created routes 0 new gateways due to redirects 27879 destinations found unreachable 0 uses of a wildcard route 0 routes not in table but not freed [root@router1 /usr/src/sys/net]# netstat -rs routing: 0 bad routing redirects 0 dynamically created routes 0 new gateways due to redirects 30256 destinations found unreachable 0 uses of a wildcard route 0 routes not in table but not freed [root@router1 /usr/src/sys/net]# netstat -rs routing: 0 bad routing redirects 0 dynamically created routes 0 new gateways due to redirects 32699 destinations found unreachable 0 uses of a wildcard route 0 routes not in table but not freed [root@router1 /usr/src/sys/net]# netstat -rs routing: 0 bad routing redirects 0 dynamically created routes 0 new gateways due to redirects 4294936832 destinations found unreachable 0 uses of a wildcard route 0 routes not in table but not freed [root@router1 /usr/src/sys/net]# netstat -rs routing: 0 bad routing redirects 0 dynamically created routes 0 new gateways due to redirects 4294941139 destinations found unreachable 0 uses of a wildcard route 0 routes not in table but not freed [root@router1 /usr/src/sys/net]# after 32768 it jumps to close to a 32 bit number limit and then after adding a few more it goes back to zero.. This doesn't happen on 7.0-RELEASE but it happens on 7.0-STABLE and -CURRENT Whatever is generating the RTM_MISS is incrementing this also. There's a lot of differences in the /net dir from release to stable but most are in bridging/gif/lagg etc. Route.c the only difference is -RELEASE +STABLE - rtfree(rt); + RTFREE_LOCKED(rt); Must be generating it from somewhere else? Paul wrote: > > [RTM_MISS information added] > > I have noticed something weird.. It doesn't generate the RTM_MISS with > all traffic... > Check this out.. > flooding packets through the router > 11:29:56.147177 IP (tos 0x0, ttl 255, id 51487, offset 0, flags > [none], proto TCP (6), length 40) 87.42.160.8.8195 > 10.3.9.50.623: ., > cksum 0x999e (incorrect (-> 0xb7e6), 2714784062:2714784062(0) ack > 1638282847 win 16384 > 11:29:56.147182 IP (tos 0x0, ttl 255, id 22197, offset 0, flags > [none], proto TCP (6), length 40) 43.253.49.61.13857 > 10.3.9.50.6232: > ., cksum 0x51b4 (incorrect (-> 0xeb52), 2685378913:2685378913(0) ack > 3576016642 win 16384 > 11:29:56.147186 IP (tos 0x0, ttl 255, id 19160, offset 0, flags > [none], proto TCP (6), length 40) 148.198.237.1.31784 > 10.3.9.50.63: > ., cksum 0x7d2b (incorrect (-> 0xcedf), 2790811715:2790811715(0) ack > 3733955172 win 16384 > 11:29:56.147190 IP (tos 0x0, ttl 255, id 45483, offset 0, flags > [none], proto TCP (6), length 40) 50.140.153.13.23035 > 10.3.9.50.324: > ., cksum 0x5e56 (incorrect (-> 0xdb81), 2403102209:2403102209(0) ack > 923149825 win 16384 > 11:29:56.147194 IP (tos 0x0, ttl 255, id 53653, offset 0, flags > [none], proto TCP (6), length 40) 215.20.25.100.18066 > > 10.3.9.50.6232: ., cksum 0x592b (incorrect (-> 0x9f26), > 3678810149:3678810149(0) ack 916597768 win 16384 > 11:29:56.147198 IP (tos 0x0, ttl 255, id 25330, offset 0, flags > [none], proto TCP (6), length 40) 30.216.125.43.23715 > > 10.3.9.50.5325: ., cksum 0x7d62 (incorrect (-> 0xdbb8), > 4058239037:4058239037(0) ack 1225812033 win 16384 > 11:29:56.147210 IP (tos 0x0, ttl 255, id 56031, offset 0, flags > [none], proto TCP (6), length 40) 92.202.56.54.1180 > 10.3.9.50.623: > ., cksum 0x45fb (incorrect (-> 0xc35d), 1051146817:1051146817(0) ack > 1914442290 win 16384 > 11:29:56.147214 IP (tos 0x0, ttl 255, id 17414, offset 0, flags > [none], proto TCP (6), length 40) 243.139.107.120.22763 > > 10.3.9.50.63: ., cksum 0x3f12 (incorrect (-> 0x983d), > 343363109:343363109(0) ack 2790458180 win 16384 > 11:29:56.147219 IP (tos 0x0, ttl 255, id 15785, offset 0, flags > [none], proto TCP (6), length 40) 193.30.160.14.19071 > 10.3.9.50.324: > ., cksum 0x0021 (incorrect (-> 0x3f33), 3909194617:3909194617(0) ack > 1335272554 win 16384 > 11:29:56.147223 IP (tos 0x0, ttl 255, id 36379, offset 0, flags > [none], proto TCP (6), length 40) 93.46.193.34.22779 > 10.3.9.50.5325: > ., cksum 0x2f95 (incorrect (-> 0x2fb6), 1506935083:1506935083(0) ack > 2882181640 win 16384 > 11:29:56.147228 IP (tos 0x0, ttl 255, id 43639, offset 0, flags > [none], proto TCP (6), length 40) 50.78.186.1.1437 > 10.3.9.50.623: ., > cksum 0xba44 (incorrect (-> 0xe9d9), 3585077271:3585077271(0) ack > 1754157076 win 16384 > 11:29:56.147232 IP (tos 0x0, ttl 255, id 31282, offset 0, flags > [none], proto TCP (6), length 40) 230.61.232.98.9799 > 10.3.9.50.6232: > ., cksum 0xe26a (incorrect (-> 0x9caf), 2184338509:2184338509(0) ack > 1881236495 win 16384 > 11:29:56.147242 IP (tos 0x0, ttl 255, id 18266, offset 0, flags > [none], proto TCP (6), length 40) 185.133.224.35.18694 > 10.3.9.50.63: > ., cksum 0x5bb9 (incorrect (-> 0x3e24), 1265308989:1265308989(0) ack > 898540886 win 16384 > 11:29:56.147247 IP (tos 0x0, ttl 255, id 63295, offset 0, flags > [none], proto TCP (6), length 40) 60.149.107.97.14907 > 10.3.9.50.324: > ., cksum 0x75ac (incorrect (-> 0xd165), 680418413:680418413(0) ack > 1399770970 win 16384 > 11:29:56.147264 IP (tos 0x0, ttl 255, id 44265, offset 0, flags > [none], proto TCP (6), length 40) 226.37.11.125.16380 > > 10.3.9.50.5325: ., cksum 0xb763 (incorrect (-> 0x2d10), > 1640245563:1640245563(0) ack 3534655349 win 16384 > 11:29:56.147276 IP (tos 0x0, ttl 255, id 62142, offset 0, flags > [none], proto TCP (6), length 40) 76.66.176.108.8173 > 10.3.9.50.623: > ., cksum 0xf66b (incorrect (-> 0xadcf), 1200243821:1200243821(0) ack > 398846984 win 16384 > 11:29:56.147281 IP (tos 0x0, ttl 255, id 60663, offset 0, flags > [none], proto TCP (6), length 40) 241.35.125.100.4600 > 10.3.9.50.324: > ., cksum 0x7299 (incorrect (-> 0x63e8), 4145462333:4145462333(0) ack > 1113096518 win 16384 > 11:29:56.147286 IP (tos 0x0, ttl 255, id 55417, offset 0, flags > [none], proto TCP (6), length 40) 152.7.87.5.14990 > 10.3.9.50.6232: > ., cksum 0xd955 (incorrect (-> 0xcfc1), 892720934:892720934(0) ack > 1334374150 win 16384 > ^C11:29:56.147290 IP (tos 0x0, ttl 255, id 29468, offset 0, flags > [none], proto TCP (6), length 40) 246.253.188.115.23425 > > 10.3.9.50.63: ., cksum 0xf14e (incorrect (-> 0xcaa4), > 1030196069:1030196069(0) ack 2661620567 win 16384 > > these generate RTM_MISS for what looks like every packet : > > got message of size 160 on Sun Jun 29 11:31:04 2008 > RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > default > > got message of size 160 on Sun Jun 29 11:31:04 2008 > RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > default > > got message of size 160 on Sun Jun 29 11:31:04 2008 > RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > default > > got message of size 160 on Sun Jun 29 11:31:04 2008 > RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > default > > got message of size 160 on Sun Jun 29 11:31:04 2008 > RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > default > > got message of size 160 on Sun Jun 29 11:31:04 2008 > RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > default > > got message of size 160 on Sun Jun 29 11:31:04 2008 > RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > default > > > And then these packets: > > 11:32:50.540714 IP (tos 0x0, ttl 128, id 60717, offset 0, flags [DF], > proto TCP (6), length 48) 199.253.132.217.1145 > 10.3.9.50.1230: S, > cksum 0xe22a (correct), 2254729531:2254729531(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540719 IP (tos 0x0, ttl 128, id 29798, offset 0, flags [DF], > proto TCP (6), length 48) 128.201.66.139.1074 > 10.3.9.50.1076: S, > cksum 0xbc56 (correct), 699104812:699104812(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540737 IP (tos 0x0, ttl 128, id 24548, offset 0, flags [DF], > proto TCP (6), length 48) 64.6.56.150.1137 > 10.3.9.50.1065: S, cksum > 0x3d9f (correct), 3231494262:3231494262(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540742 IP (tos 0x0, ttl 128, id 24872, offset 0, flags [DF], > proto TCP (6), length 48) 209.236.188.202.1119 > 10.3.9.50.1137: S, > cksum 0xb4bb (correct), 3403159757:3403159757(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540746 IP (tos 0x0, ttl 128, id 7860, offset 0, flags [DF], > proto TCP (6), length 48) 193.42.23.202.1147 > 10.3.9.50.1047: S, > cksum 0x7900 (correct), 2067225130:2067225130(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540751 IP (tos 0x0, ttl 128, id 6369, offset 0, flags [DF], > proto TCP (6), length 48) 198.222.98.165.1070 > 10.3.9.50.1197: S, > cksum 0x8245 (correct), 1566580196:1566580196(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540759 IP (tos 0x0, ttl 128, id 29494, offset 0, flags [DF], > proto TCP (6), length 48) 194.32.233.216.1217 > 10.3.9.50.1035: S, > cksum 0x0036 (correct), 608655014:608655014(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540772 IP (tos 0x0, ttl 128, id 57975, offset 0, flags [DF], > proto TCP (6), length 48) 205.206.20.208.1220 > 10.3.9.50.1232: S, > cksum 0x52fc (correct), 1339728095:1339728095(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540777 IP (tos 0x0, ttl 128, id 5551, offset 0, flags [DF], > proto TCP (6), length 48) 4.42.66.89.1142 > 10.3.9.50.1178: S, cksum > 0xaa57 (correct), 1309337587:1309337587(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540781 IP (tos 0x0, ttl 128, id 64726, offset 0, flags [DF], > proto TCP (6), length 48) 208.166.197.140.1052 > 10.3.9.50.1084: S, > cksum 0x4dfd (correct), 3514659298:3514659298(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540786 IP (tos 0x0, ttl 128, id 51126, offset 0, flags [DF], > proto TCP (6), length 48) 206.67.34.143.1071 > 10.3.9.50.1100: S, > cksum 0xbf84 (correct), 3369643581:3369643581(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540790 IP (tos 0x0, ttl 128, id 59088, offset 0, flags [DF], > proto TCP (6), length 48) 210.246.25.177.1277 > 10.3.9.50.1205: S, > cksum 0x7080 (correct), 1667720615:1667720615(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540795 IP (tos 0x0, ttl 128, id 2326, offset 0, flags [DF], > proto TCP (6), length 48) 129.251.33.231.1154 > 10.3.9.50.1195: S, > cksum 0x28e5 (correct), 312297303:312297303(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540799 IP (tos 0x0, ttl 128, id 8268, offset 0, flags [DF], > proto TCP (6), length 48) 216.84.213.152.1115 > 10.3.9.50.1027: S, > cksum 0x7a81 (correct), 2232908292:2232908292(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540803 IP (tos 0x0, ttl 128, id 15857, offset 0, flags [DF], > proto TCP (6), length 48) 199.228.246.182.1053 > 10.3.9.50.1053: S, > cksum 0xe187 (correct), 376795414:376795414(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540808 IP (tos 0x0, ttl 128, id 33924, offset 0, flags [DF], > proto TCP (6), length 48) 128.92.244.80.1060 > 10.3.9.50.1148: S, > cksum 0xc4e1 (correct), 2038527032:2038527032(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540812 IP (tos 0x0, ttl 128, id 12114, offset 0, flags [DF], > proto TCP (6), length 48) 64.118.145.169.1252 > 10.3.9.50.1148: S, > cksum 0xb270 (correct), 3489780214:3489780214(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540816 IP (tos 0x0, ttl 128, id 23385, offset 0, flags [DF], > proto TCP (6), length 48) 211.212.238.186.1217 > 10.3.9.50.1083: S, > cksum 0xc082 (correct), 2887120836:2887120836(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540821 IP (tos 0x0, ttl 128, id 10979, offset 0, flags [DF], > proto TCP (6), length 48) 24.119.38.75.1088 > 10.3.9.50.1027: S, cksum > 0xee10 (correct), 3079816000:3079816000(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540830 IP (tos 0x0, ttl 128, id 44562, offset 0, flags [DF], > proto TCP (6), length 48) 193.99.52.25.1249 > 10.3.9.50.1047: S, cksum > 0x9ef4 (correct), 1263552303:1263552303(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540835 IP (tos 0x0, ttl 128, id 45332, offset 0, flags [DF], > proto TCP (6), length 48) 209.168.140.136.1275 > 10.3.9.50.1053: S, > cksum 0x03b5 (correct), 1521904180:1521904180(0) win 16384 1460,nop,[bad opt]> > 11:32:50.540840 IP (tos 0x0, ttl 128, id 10244, offset 0, flags [DF], > proto TCP (6), length 48) 198.114.145.88.1214 > 10.3.9.50.1083: S, > cksum 0x51a9 (correct), 2907689003:2907689003(0) win 16384 1460,nop,[bad opt]> > > I get no RTM MISS. So this absoutely makes no sense?? It should not > depend on the TYPE of packet at ALL. It's just forwarding IP > regardless of what's in the TCP header so why on earth would the tcp > header many any difference to how it routes.. It should not even be > looking at the tcp header unless I have a firewall turned on or any > type of rules that would filter based on tcp options. Forwarding a > packet doesn't require looking at TCP option bits or even TCP port > numbers. I suppose it could be checking the mss or trying to > calculate checksums but I would prefer it didn't even do that :/ Raw > performance is all I care about at this point, I don't even need to > use any firewall at all. > It's also funny to notice, that tcpdump does not see those packets as > TCP.. > tcpdump -n -i em0 tcp > does not show either of my 'floods' > tcpdump -n -i em1 tcp > shows my floods > tcpdump -n -i em0 > shows them all.. and they say TCP ..lol :) Weird I tell ya! > > So the bottom line is, why do one type of packet generate RTM_MISS and > another doesn't? > I haven't confirmed this, but I don't think it does it on 7.0-RELEASE, > only 7.0-STABLE and CURRENT. > I will try and confirm though. > > Paul > > > > Mike Tancsa wrote: >> At 04:04 AM 6/29/2008, Paul wrote: >>> This is just a question but who can get more than 400k pps >>> forwarding performance ? >>> I have tested fbsd 6/7/8 so far with many different configs. (all >>> using intel pci-ex nic and SMP) >>> fbsd 7-stable/8(current) seem to be the fastest and always hit this >>> ceiling of 400k pps. Soon as it hits that I get errors galore. >> >> In your testing of current, or even RELENG_7, did you ever solve the >> problem of >> >> http://www.freebsd.org/cgi/query-pr.cgi?pr=123621&cat=kern >> that you also ran into in a previous thread ? I wonder if the >> generation of all those seemingly bogus RTM_MISS messages is >> hampering performance. >> >> ---Mike >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 18:39:56 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4925E1065671 for ; Sun, 29 Jun 2008 18:39:56 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id EABD98FC19 for ; Sun, 29 Jun 2008 18:39:55 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.2/8.14.2) with ESMTP id m5TIdr8h042211; Sun, 29 Jun 2008 14:39:53 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m5TIdrFs075883 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 29 Jun 2008 14:39:53 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200806291839.m5TIdrFs075883@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Sun, 29 Jun 2008 14:39:59 -0400 To: Paul From: Mike Tancsa In-Reply-To: <4867BCE3.2000909@gtcomm.net> References: <4867420D.7090406@gtcomm.net> <200806291539.m5TFdvCO075285@lava.sentex.ca> <4867BCE3.2000909@gtcomm.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 18:39:56 -0000 At 12:48 PM 6/29/2008, Paul wrote: >[RTM_MISS information added] > >I have noticed something weird.. It doesn't generate the RTM_MISS >with all traffic... Does turning off tso and or chksum offload make a difference ? ---Mike From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 20:06:46 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C5C8D106567A for ; Sun, 29 Jun 2008 20:06:46 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outF.internet-mail-service.net (outf.internet-mail-service.net [216.240.47.229]) by mx1.freebsd.org (Postfix) with ESMTP id AC0938FC0C for ; Sun, 29 Jun 2008 20:06:46 +0000 (UTC) (envelope-from julian@elischer.org) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id B244423F1; Sun, 29 Jun 2008 13:06:53 -0700 (PDT) Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id E5F9B2D6004; Sun, 29 Jun 2008 13:06:45 -0700 (PDT) Message-ID: <4867EB67.1020900@elischer.org> Date: Sun, 29 Jun 2008 13:07:03 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Kevin Oberman References: <20080629005756.4BA1B4500E@ptavv.es.net> In-Reply-To: <20080629005756.4BA1B4500E@ptavv.es.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, VANHULLEBUS Yvan Subject: Re: FreeBSD NAT-T patch integration X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 20:06:46 -0000 Kevin Oberman wrote: >> Date: Sat, 28 Jun 2008 23:13:00 +0200 >> From: VANHULLEBUS Yvan >> Sender: owner-freebsd-net@freebsd.org >> >> On Fri, Jun 27, 2008 at 11:06:19AM -0400, George V. Neville-Neil wrote: >>> At Thu, 26 Jun 2008 12:56:41 -0700, >>> julian wrote: >>>> I'm planning on committing it unless someone can provide a reason not >>>> to, as I've seen it working, needed it, and have not seen any bad >>>> byproducts. >>>> >>> I'd be interested to know how you tested it. >> I can answer that question: >> >> - We do have "non regression tests", which test every night that "it >> works" in a "good scenario", with a peers who runs quite the same >> kernel, with the same patch and the same userland. >> >> We do also some "bad scenarios" tests, but not as much as I would like >> actually (it's much more complex to test). >> >> >> - We do have THOUSANDS of customers who use it for years now. And >> customers are really not well nown to always generate "good >> scenarios".... >> >> Customers can do some mistakes, can use some very strange stacks for >> peers, can use a lots of other IPSec stacks for peers, can have *a >> lot* of peers (of course with various implementations, configurations, >> NAT or not, etc...) for the same gate, etc... >> >> And how do I know that it works ? >> Well, when it doesn't work, I do know it, quite quickly most of the >> time ! >> >> As Bjoern said in another mail, we're talking about security. >> Security is my job, security is what we provide to our customers, and >> even if some of our customers just don't understant what they do, some >> others do *really* understand things, and do *really* test devices, >> check if it works, and quickly reports us problems (and ask for quick >> solutions). >> >> >> >>> NAT-T and IPsec are >>> non-trivial protocols/subsystems that can have far reaching impacts on >>> the network stack. >> Yes. >> >> >>> Also, are you planning to maintain it after >>> committing it? >> I plan to do that. >> >> >>> The biggest problem with NAT-T hasn't been the code, >>> it's been that the author, >> Really ? >> >> >>> who is doing a great job on the code, >> Thank you.... >> >> >>> has >>> been too busy to maintain it anywhere but at work. >> That is not exact, and I'm really sorry if you understood that after >> our discussion at the last EuroBSDCon. >> >> First, you can check that I'm sending this mail on saturday evening, >> which is out of my work hours :-) >> >> Then I can for example confirm that I did the whole migration from >> RELENG6 to RELENG7 on my free time, just because I needed a working >> FreeBSD7 + NAT-T patch at home before I've been asked to do it at >> work. >> >> Yes, I'm doing most of IPSec / NAT-T stuff at work, but it is mainly >> because I'm *paid* for that !!! >> Let me tell this again: working on FreeBSD's IPSec stack (and on >> NAT-T, of course, adn also on ipsec-tools) is a part of my job. >> >> What I told you some months ago is that there are some things that I >> cannot really do at work, because it takes lot of time and because >> those things are more "code cleanups" than "new features" (we were >> talking about PFKey V2 cleanups related to NAT-T ports). >> >> >> And if I did not spend that time to do that cleanup, that's not >> because I don't have such time, that's because I was starting to >> fear that the patch may be never commited, so I wasn't sure to want to >> continue spending lots of time on a patch "for nothing". >> >> If Bjoern or you told me some months ago "do that cleanup and we'll >> commit the patch", it would already have been done, and we would have >> *all* saved some time ! >> >> >> And yes, I do also spend some parts of my free time on things that are >> not related to FreeBSD's IPSec (and, to be completely honest, on >> things which are not related at all to computers....). >> >> >> >>> That is not a slam >>> on the person or the code, I have the highest respect for both, but it >>> reflects and important reality of the situation. >> I feel that "the reality of the situation" is that FreeBSD's IPSec >> stack needs more people *working on it*, not more people wasting their >> time about why some work should or shouldn't be reported. >> >> That is not a slam on the persons who are still working on the IPsec >> stack, I have the highest respect for them, but it reflects an >> important reality of the situation... >> >> >> >>> Unless you're >>> stepping up to maintain it as well as commit it I think it should not >>> be committed. I know the Bjoern has been working hard to pick up the >>> IPsec stuff in his free time, and I value his input on this subject >>> quite a bit. >> You said it yourself: actually, FreeBSD's IPSec stack is just >> maintained by one person, which worked hard on it, which does a great >> job on it, but which can spend only some parts of his free time on it. >> >> That is a problem. >> Of course, the solution can NOT be to ask Bjoern to spend more time on >> it (well, Bjoern, do that if you want :-). >> >> The only other solution I see is to find a way to help people who want >> to help you.... > > Thanks so much to folks like Bjorn and Yvan who spend the time to do > some tough jobs like dealing with IPsec and being stubborn about > committing things to security tools without very careful audit. > > In case you missed it, IPsec is about security, not features. And, in > case you have never been involved in real security or crypto, security > is really, really hard. > > No, no, it's much harder than that! > > While I'd love to see NAT-T, until someone who knows EXACTLY what the > IPsec and NAT-T code is doing and what it is required to do can do the > line by line review, it should NOT be committed. so, you don't think that this is rather a slap in teh face of the person who wrote this, and who's job is to do security related work in a proffesional context? "This requires a someone who knows what they are doing, and despite the fact that you do this for a living, I arbitrarlily exclude you from that category"? From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 20:32:39 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6622F106566C for ; Sun, 29 Jun 2008 20:32:39 +0000 (UTC) (envelope-from oberman@es.net) Received: from postal1.es.net (postal1.es.net [198.128.3.205]) by mx1.freebsd.org (Postfix) with ESMTP id 487688FC0A for ; Sun, 29 Jun 2008 20:32:39 +0000 (UTC) (envelope-from oberman@es.net) Received: from ptavv.es.net (ptavv.es.net [198.128.4.29]) by postal1.es.net (Postal Node 1) with ESMTP (SSL) id JAJ06637; Sun, 29 Jun 2008 13:32:37 -0700 Received: from ptavv.es.net (ptavv.es.net [127.0.0.1]) by ptavv.es.net (Tachyon Server) with ESMTP id 78F6245047; Sun, 29 Jun 2008 13:32:36 -0700 (PDT) To: Julian Elischer In-Reply-To: Your message of "Sun, 29 Jun 2008 13:07:03 PDT." <4867EB67.1020900@elischer.org> Mime-Version: 1.0 Content-Type: multipart/signed; boundary="==_Exmh_1214771556_77486P"; micalg=pgp-sha1; protocol="application/pgp-signature" Content-Transfer-Encoding: 7bit Date: Sun, 29 Jun 2008 13:32:36 -0700 From: "Kevin Oberman" Message-Id: <20080629203236.78F6245047@ptavv.es.net> X-Sender-IP: 198.128.4.29 X-Sender-Domain: es.net X-Recipent: ; ; ; X-Sender: X-To_Name: Julian Elischer X-To_Domain: elischer.org X-To: Julian Elischer X-To_Email: julian@elischer.org X-To_Alias: julian Cc: freebsd-net@freebsd.org, VANHULLEBUS Yvan Subject: Re: FreeBSD NAT-T patch integration X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 20:32:39 -0000 --==_Exmh_1214771556_77486P Content-Type: text/plain; charset=us-ascii Content-Disposition: inline > Date: Sun, 29 Jun 2008 13:07:03 -0700 > From: Julian Elischer > Sender: owner-freebsd-net@freebsd.org > > Kevin Oberman wrote: > >> Date: Sat, 28 Jun 2008 23:13:00 +0200 > >> From: VANHULLEBUS Yvan > >> Sender: owner-freebsd-net@freebsd.org > >> > >> On Fri, Jun 27, 2008 at 11:06:19AM -0400, George V. Neville-Neil wrote: > >>> At Thu, 26 Jun 2008 12:56:41 -0700, > >>> julian wrote: > >>>> I'm planning on committing it unless someone can provide a reason not > >>>> to, as I've seen it working, needed it, and have not seen any bad > >>>> byproducts. > >>>> > >>> I'd be interested to know how you tested it. > >> I can answer that question: > >> > >> - We do have "non regression tests", which test every night that "it > >> works" in a "good scenario", with a peers who runs quite the same > >> kernel, with the same patch and the same userland. > >> > >> We do also some "bad scenarios" tests, but not as much as I would like > >> actually (it's much more complex to test). > >> > >> > >> - We do have THOUSANDS of customers who use it for years now. And > >> customers are really not well nown to always generate "good > >> scenarios".... > >> > >> Customers can do some mistakes, can use some very strange stacks for > >> peers, can use a lots of other IPSec stacks for peers, can have *a > >> lot* of peers (of course with various implementations, configurations, > >> NAT or not, etc...) for the same gate, etc... > >> > >> And how do I know that it works ? > >> Well, when it doesn't work, I do know it, quite quickly most of the > >> time ! > >> > >> As Bjoern said in another mail, we're talking about security. > >> Security is my job, security is what we provide to our customers, and > >> even if some of our customers just don't understant what they do, some > >> others do *really* understand things, and do *really* test devices, > >> check if it works, and quickly reports us problems (and ask for quick > >> solutions). > >> > >> > >> > >>> NAT-T and IPsec are > >>> non-trivial protocols/subsystems that can have far reaching impacts on > >>> the network stack. > >> Yes. > >> > >> > >>> Also, are you planning to maintain it after > >>> committing it? > >> I plan to do that. > >> > >> > >>> The biggest problem with NAT-T hasn't been the code, > >>> it's been that the author, > >> Really ? > >> > >> > >>> who is doing a great job on the code, > >> Thank you.... > >> > >> > >>> has > >>> been too busy to maintain it anywhere but at work. > >> That is not exact, and I'm really sorry if you understood that after > >> our discussion at the last EuroBSDCon. > >> > >> First, you can check that I'm sending this mail on saturday evening, > >> which is out of my work hours :-) > >> > >> Then I can for example confirm that I did the whole migration from > >> RELENG6 to RELENG7 on my free time, just because I needed a working > >> FreeBSD7 + NAT-T patch at home before I've been asked to do it at > >> work. > >> > >> Yes, I'm doing most of IPSec / NAT-T stuff at work, but it is mainly > >> because I'm *paid* for that !!! > >> Let me tell this again: working on FreeBSD's IPSec stack (and on > >> NAT-T, of course, adn also on ipsec-tools) is a part of my job. > >> > >> What I told you some months ago is that there are some things that I > >> cannot really do at work, because it takes lot of time and because > >> those things are more "code cleanups" than "new features" (we were > >> talking about PFKey V2 cleanups related to NAT-T ports). > >> > >> > >> And if I did not spend that time to do that cleanup, that's not > >> because I don't have such time, that's because I was starting to > >> fear that the patch may be never commited, so I wasn't sure to want to > >> continue spending lots of time on a patch "for nothing". > >> > >> If Bjoern or you told me some months ago "do that cleanup and we'll > >> commit the patch", it would already have been done, and we would have > >> *all* saved some time ! > >> > >> > >> And yes, I do also spend some parts of my free time on things that are > >> not related to FreeBSD's IPSec (and, to be completely honest, on > >> things which are not related at all to computers....). > >> > >> > >> > >>> That is not a slam > >>> on the person or the code, I have the highest respect for both, but it > >>> reflects and important reality of the situation. > >> I feel that "the reality of the situation" is that FreeBSD's IPSec > >> stack needs more people *working on it*, not more people wasting their > >> time about why some work should or shouldn't be reported. > >> > >> That is not a slam on the persons who are still working on the IPsec > >> stack, I have the highest respect for them, but it reflects an > >> important reality of the situation... > >> > >> > >> > >>> Unless you're > >>> stepping up to maintain it as well as commit it I think it should not > >>> be committed. I know the Bjoern has been working hard to pick up the > >>> IPsec stuff in his free time, and I value his input on this subject > >>> quite a bit. > >> You said it yourself: actually, FreeBSD's IPSec stack is just > >> maintained by one person, which worked hard on it, which does a great > >> job on it, but which can spend only some parts of his free time on it. > >> > >> That is a problem. > >> Of course, the solution can NOT be to ask Bjoern to spend more time on > >> it (well, Bjoern, do that if you want :-). > >> > >> The only other solution I see is to find a way to help people who want > >> to help you.... > > > > Thanks so much to folks like Bjorn and Yvan who spend the time to do > > some tough jobs like dealing with IPsec and being stubborn about > > committing things to security tools without very careful audit. > > > > In case you missed it, IPsec is about security, not features. And, in > > case you have never been involved in real security or crypto, security > > is really, really hard. > > > > No, no, it's much harder than that! > > > > While I'd love to see NAT-T, until someone who knows EXACTLY what the > > IPsec and NAT-T code is doing and what it is required to do can do the > > line by line review, it should NOT be committed. > > so, you don't think that this is rather a slap in teh face of the > person who wrote this, and who's job is to do security related work in > a proffesional context? > > "This requires a someone who knows what they are doing, and despite > the fact that you do this for a living, I arbitrarlily exclude you > from that category"? Not at all. No one should both write and vet security code. It takes two pairs of eyes connected to two brains to write and check code. I don't care how competent the programmer. This is a good rule for all code, but is too "expensive" for most code. Can you imagine how few updates would ever be checked in if every one needed a line-by-line vetting before a commit was allowed? Security code really an exception. It simply has to be right and I heartily approve of the care being taken by both Bjorn and Yvan. I feel both are doing exactly the right thing, much as I'd with they could do it faster. This is how you avoid potential disasters like the recent Debian openssh screw-up. (Just a small change that looked fine, but broke security done by a single programmer and never vetted). -- R. Kevin Oberman, Network Engineer Energy Sciences Network (ESnet) Ernest O. Lawrence Berkeley National Laboratory (Berkeley Lab) E-mail: oberman@es.net Phone: +1 510 486-8634 Key fingerprint:059B 2DDF 031C 9BA3 14A4 EADA 927D EBB3 987B 3751 --==_Exmh_1214771556_77486P Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) Comment: Exmh version 2.5 06/03/2002 iD8DBQFIZ/Fkkn3rs5h7N1ERAjS2AKCPhIDpbHgm75abb+mnPSJgm8TBAACghovp pizJSf/Tmn+Ez2t3Ly62r7w= =sPsb -----END PGP SIGNATURE----- --==_Exmh_1214771556_77486P-- From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 22:04:22 2008 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 340051065677; Sun, 29 Jun 2008 22:04:22 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 0B01F8FC17; Sun, 29 Jun 2008 22:04:22 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: from freefall.freebsd.org (gavin@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m5TM4LdZ094640; Sun, 29 Jun 2008 22:04:21 GMT (envelope-from gavin@freefall.freebsd.org) Received: (from gavin@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m5TM4LSe094636; Sun, 29 Jun 2008 22:04:21 GMT (envelope-from gavin) Date: Sun, 29 Jun 2008 22:04:21 GMT Message-Id: <200806292204.m5TM4LSe094636@freefall.freebsd.org> To: gavin@FreeBSD.org, freebsd-i386@FreeBSD.org, freebsd-net@FreeBSD.org From: gavin@FreeBSD.org Cc: Subject: Re: kern/125010: [vr] ripd: multicast join failed, interface vr0 not running X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 22:04:22 -0000 Old Synopsis: vr driver: ripd: multicast join failed, interface vr0 not running New Synopsis: [vr] ripd: multicast join failed, interface vr0 not running Responsible-Changed-From-To: freebsd-i386->freebsd-net Responsible-Changed-By: gavin Responsible-Changed-When: Sun Jun 29 22:03:30 UTC 2008 Responsible-Changed-Why: Over to maintainers http://www.freebsd.org/cgi/query-pr.cgi?pr=125010 From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 22:33:13 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 063C11065678 for ; Sun, 29 Jun 2008 22:33:13 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from shrew.net (206-223-169-85.beanfield.net [206.223.169.85]) by mx1.freebsd.org (Postfix) with ESMTP id CC3C18FC1D for ; Sun, 29 Jun 2008 22:33:12 +0000 (UTC) (envelope-from mgrooms@shrew.net) Received: from localhost (wm-ca.hub.org [206.223.169.82]) by shrew.net (Postfix) with ESMTP id 28B2B79E2CA for ; Sun, 29 Jun 2008 17:33:12 -0500 (CDT) Received: from shrew.net ([206.223.169.85]) by localhost (mx1.hub.org [206.223.169.82]) (amavisd-new, port 10024) with ESMTP id 54215-05 for ; Sun, 29 Jun 2008 22:33:11 +0000 (UTC) Received: from hole.shrew.net (cpe-70-113-206-103.austin.res.rr.com [70.113.206.103]) by shrew.net (Postfix) with ESMTP id F094B79E26A for ; Sun, 29 Jun 2008 17:33:10 -0500 (CDT) Received: from [10.22.200.30] ([10.22.200.30]) by hole.shrew.net (8.14.2/8.14.2) with ESMTP id m5TMX8wr081493 for ; Sun, 29 Jun 2008 17:33:08 -0500 (CDT) (envelope-from mgrooms@shrew.net) Message-ID: <48680DB8.708@shrew.net> Date: Sun, 29 Jun 2008 17:33:28 -0500 From: Matthew Grooms User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <4867B2B3.3090208@shrew.net> In-Reply-To: <4867B2B3.3090208@shrew.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: FreeBSD NAT-T patch integration X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 22:33:13 -0000 > Thanks so much to folks like Bjorn and Yvan who spend the time to do > some tough jobs like dealing with IPsec and being stubborn about > committing things to security tools without very careful audit. > Seconded. > In case you missed it, IPsec is about security, not features. And, in > case you have never been involved in real security or crypto, security > is really, really hard. > > No, no, it's much harder than that! > > While I'd love to see NAT-T, until someone who knows EXACTLY what the > IPsec and NAT-T code is doing and what it is required to do can do the > line by line review, it should NOT be committed. Well, none of my emails advocated applying the patch without a complete review. I asked what the state of the review and testing process was and, in the opinion of my betters, if the patch could kindly be applied. Additionally, if the holdup was related to a lack of time available to the IPsec maintainers, what could be done to assist in getting this patch integrated. As for Yvans commitment, I would think that 3 years spent maintaining a patch on a moving target more than demonstrates it. I'm also a bit surprised to see that the request I made was categorized as impatient crying. But programmer hide tends to be thicker than most and opinions always vary. I won't deny that there has been a few misconceptions about what NAT-T is good for and what is involved in NAT-T packet processing. I'm not a BSD kernel hacker, but I have written a full IPv4 IPsec implementation as well as a key daemon from scratch that includes NAT-T support. For portability reasons, this involved emulating the PF_KEY interface on platforms that don't provide one natively so I am quite familiar with that component as well. No, NAT-T isn't required to interoperate with any particular vendor IPsec in the vanilla sense. It is required to work around issues that arise when at least one peer is behind a NAT device. Its particularly agonizing when you are trying to act as a IPsec remote access client. Without NAT-T, you will quickly experience why its been adopted almost universally. As the author of a port that attempts to offer this style of connectivity, I have a vested interest in seeing this functionality present in FreeBSD which explains my persistence :) How complex is IPsec NAT-T? The concept behind it is pretty simple. You wrap ESP in UDP so it traverses NAT more easily. The IKE traffic is multiplexed on the same port so you only issue keep-alives from userland to refresh a single NAT state. Most of the complexity related to negotiating NAT-T support, detecting the presence of a NAT and setting up the SA options are handled by the IKE daemon. Kernel side changes for a tunnel-only implementation, like Yvans, can be more or less summed up like so ... 1) A key daemon needs to be able to set a socket option that signifies how packets will be multiplexed on a bound port. This helps the IPsec layer qualify which packets need to be dropped, processed as ESP or passed on to IKE via the socket layer. 2) The PF_KEY system needs to be extended so that IPsec knows which SAs are NAT-T enabled and what UDP port values are to be used. 3) UDP headers, and non-isakmp markers for old versions of NAT-T, need to be added or removed from ESP packets when they are processed using a NAT-T enabled SA. After which, they are processed normally. Yes, the devil is in the detail but the details in the patch should be self evident. To coin a popular phrase on this list, IPsec is about security. If you look at Yvans patch, it doesn't do anything to fundamentally change the way ESP payloads are constructed or interpreted. It mostly just inserts or removes a UDP header where appropriate. If your not familiar with PF_KEY, the other parts of the patch may take a bit longer to evaluate. I'm not trying to downplay the expertise or experience that both gnn and bjorn bring to FreeBSD. I just don't think it would take a world class IPsec kernel hacker to review Yvans work competently. RFC3948 is literally 10 pages if you skip the end credits. The draft versions are considerably shorter. Again, opinions may vary. So the horse seems dead so Ill stop flogging it. If it takes another 6 months or more to get this reviewed, I won't complain. If/when the time comes, I have a half dozen commercial gateways and lots of Linix *BSD on the far side of a NAT if anyone would like additional testing. One last thing. I admit that I can't qualify how feasible this is, but it may be beneficial to split out the portion of the patch required to set the UDP socket options and commit that first. If this can be done in benign way, it would probably make a lot of people happier. Applying a patch that only requires rebuilding the kernel is a lot less annoying than having to perform a full buildworld. -Matthew From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 23:06:17 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 79281106564A for ; Sun, 29 Jun 2008 23:06:17 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 0E9E68FC1A for ; Sun, 29 Jun 2008 23:06:16 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 24371 invoked from network); 30 Jun 2008 01:06:14 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 30 Jun 2008 01:06:14 +0200 Date: Mon, 30 Jun 2008 01:06:14 +0200 (CEST) From: Ingo Flaschberger To: Paul In-Reply-To: <4867A9A1.9070507@gtcomm.net> Message-ID: References: <4867420D.7090406@gtcomm.net> <4867A9A1.9070507@gtcomm.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 23:06:17 -0000 Dear Paul, does the em-task jump from cpu to cpu? (mp-systems are not really better for forwarding performance). try once with only 1 cpu. bye, Ingo From owner-freebsd-net@FreeBSD.ORG Sun Jun 29 23:24:52 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 28BEE106564A for ; Sun, 29 Jun 2008 23:24:52 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 009868FC12 for ; Sun, 29 Jun 2008 23:24:51 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KD6D3-00037a-6D; Sun, 29 Jun 2008 19:21:29 -0400 Message-ID: <48681A3D.9040509@gtcomm.net> Date: Sun, 29 Jun 2008 19:26:53 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <4867A9A1.9070507@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Jun 2008 23:24:52 -0000 Yes it does but it seems to use a lot more of one cpu than the others so It's really not SMP.. Can I stop it from doing this with some setting? Why can't there be 4 taskq's? Also with full internet table I can't even do 100kpps without errors.. I don't get it :/ I could do 300kpps on a p3 and now I have a 3ghz xeon and 2.2ghz opteron brand new hardware and can barely get more than that.. Doesn't make sense to me. Ingo Flaschberger wrote: > Dear Paul, > > does the em-task jump from cpu to cpu? > (mp-systems are not really better for forwarding performance). > > try once with only 1 cpu. > > bye, > Ingo > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 00:16:37 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 33B9A106564A for ; Mon, 30 Jun 2008 00:16:37 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 78FCB8FC12 for ; Mon, 30 Jun 2008 00:16:35 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 29824 invoked from network); 30 Jun 2008 02:16:34 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 30 Jun 2008 02:16:34 +0200 Date: Mon, 30 Jun 2008 02:16:33 +0200 (CEST) From: Ingo Flaschberger To: Paul In-Reply-To: <48681A3D.9040509@gtcomm.net> Message-ID: References: <4867420D.7090406@gtcomm.net> <4867A9A1.9070507@gtcomm.net> <48681A3D.9040509@gtcomm.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 00:16:37 -0000 Dear Paul, > Yes it does but it seems to use a lot more of one cpu than the others so It's > really not SMP.. Can I stop it from doing this with some setting? > Why can't there be 4 taskq's? it is possible, but it need to be coded. hz 4000 is also too high, use 1000-2000 http://www.tancsa.com/blast.html > Also with full internet table I can't even do 100kpps without errors.. I > don't get it :/ I could do 300kpps on a p3 and now I have a 3ghz xeon and > 2.2ghz opteron brand new hardware and can barely get more than that.. > Doesn't make sense to me. have you tested with freebsd 6? or try dragonfly? Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 00:25:29 2008 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B86B61065682; Mon, 30 Jun 2008 00:25:29 +0000 (UTC) (envelope-from yongari@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 8E8C08FC0A; Mon, 30 Jun 2008 00:25:29 +0000 (UTC) (envelope-from yongari@FreeBSD.org) Received: from freefall.freebsd.org (yongari@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m5U0PTQT007673; Mon, 30 Jun 2008 00:25:29 GMT (envelope-from yongari@freefall.freebsd.org) Received: (from yongari@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m5U0PTu4007669; Mon, 30 Jun 2008 00:25:29 GMT (envelope-from yongari) Date: Mon, 30 Jun 2008 00:25:29 GMT Message-Id: <200806300025.m5U0PTu4007669@freefall.freebsd.org> To: goran.lowkrantz@ismobile.com, yongari@FreeBSD.org, freebsd-net@FreeBSD.org, yongari@FreeBSD.org From: yongari@FreeBSD.org Cc: Subject: Re: kern/125010: [vr] ripd: multicast join failed, interface vr0 not running X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 00:25:29 -0000 Synopsis: [vr] ripd: multicast join failed, interface vr0 not running State-Changed-From-To: open->feedback State-Changed-By: yongari State-Changed-When: Mon Jun 30 00:24:17 UTC 2008 State-Changed-Why: Would you show me the dmesg output related with vr(4)? There is a bug in multicasting filter handing on VT6105M(VIA Rhine III). If the hardware is VT6105M, please try the patch at the following URL. http://people.freebsd.org/~yongari/vr/vr.cam.patch2 Responsible-Changed-From-To: freebsd-net->yongari Responsible-Changed-By: yongari Responsible-Changed-When: Mon Jun 30 00:24:17 UTC 2008 Responsible-Changed-Why: Grab. http://www.freebsd.org/cgi/query-pr.cgi?pr=125010 From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 00:34:45 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D053E1065673 for ; Mon, 30 Jun 2008 00:34:45 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id 94AD78FC22 for ; Mon, 30 Jun 2008 00:34:45 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.2/8.14.2) with ESMTP id m5U0YgTq064712; Sun, 29 Jun 2008 20:34:42 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m5U0YfsF077111 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 29 Jun 2008 20:34:42 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200806300034.m5U0YfsF077111@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Sun, 29 Jun 2008 20:34:47 -0400 To: Ingo Flaschberger , Paul From: Mike Tancsa In-Reply-To: References: <4867420D.7090406@gtcomm.net> <4867A9A1.9070507@gtcomm.net> <48681A3D.9040509@gtcomm.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 00:34:45 -0000 At 08:16 PM 6/29/2008, Ingo Flaschberger wrote: >Dear Paul, > >>Yes it does but it seems to use a lot more of one cpu than the >>others so It's really not SMP.. Can I stop it from doing this with >>some setting? >>Why can't there be 4 taskq's? > >it is possible, but it need to be coded. > >hz 4000 is also too high, use 1000-2000 >http://www.tancsa.com/blast.html Those are very old test results using a fairly different em driver than whats in the tree. You might look at http://people.yandex-team.ru/~wawa/ on RELENG_6. I havent tested it, but supposedly its optimized for forwarding on SMP hardware ---Mike From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 00:50:37 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 38F271065672 for ; Mon, 30 Jun 2008 00:50:37 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 083448FC0A for ; Mon, 30 Jun 2008 00:50:36 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KD7Y2-0007yz-43; Sun, 29 Jun 2008 20:47:14 -0400 Message-ID: <48682E57.8040509@gtcomm.net> Date: Sun, 29 Jun 2008 20:52:39 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <4867A9A1.9070507@gtcomm.net> <48681A3D.9040509@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 00:50:37 -0000 Well who wants to code it ? I would gladly pay someone to make it work the way I want. :) It needs to be able to do line rate gig-e with 64 byte packets and 250k routes. FBSD6 is definitely slower. Haven't tried dragonfly. Thanks Ingo Flaschberger wrote: > Dear Paul, > >> Yes it does but it seems to use a lot more of one cpu than the others >> so It's really not SMP.. Can I stop it from doing this with some >> setting? >> Why can't there be 4 taskq's? > > it is possible, but it need to be coded. > > hz 4000 is also too high, use 1000-2000 > http://www.tancsa.com/blast.html > >> Also with full internet table I can't even do 100kpps without >> errors.. I don't get it :/ I could do 300kpps on a p3 and now I have >> a 3ghz xeon and 2.2ghz opteron brand new hardware and can barely get >> more than that.. Doesn't make sense to me. > > have you tested with freebsd 6? > or try dragonfly? > > Kind regards, > Ingo Flaschberger > > From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 00:53:48 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C28CC106566C for ; Mon, 30 Jun 2008 00:53:48 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 983E18FC0A for ; Mon, 30 Jun 2008 00:53:48 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KD7b6-0008CR-4H; Sun, 29 Jun 2008 20:50:24 -0400 Message-ID: <48682F15.6070707@gtcomm.net> Date: Sun, 29 Jun 2008 20:55:49 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Mike Tancsa References: <4867420D.7090406@gtcomm.net> <4867A9A1.9070507@gtcomm.net> <48681A3D.9040509@gtcomm.net> <200806300034.m5U0YfsF077111@lava.sentex.ca> In-Reply-To: <200806300034.m5U0YfsF077111@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Ingo Flaschberger Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 00:53:48 -0000 I tried this.. I put 6-STABLE (6.3), using default driver was slower than FBSD7 I tried yandex driver 1.36 and it was even worse.. It would make the machine unresponsive and drop more packets than the default driver. It would make 'top' show cpu's were 300% idle and a command such as 'w' would take 5 minutes to respond so obviously something is wrong with this driver :/ In the em taskq does it do all the routing also? Can the received packets be accepted with em0 taskq and dumped into another multi threaded thing which will do the routing and send it to the output taskq? Mike Tancsa wrote: > At 08:16 PM 6/29/2008, Ingo Flaschberger wrote: >> Dear Paul, >> >>> Yes it does but it seems to use a lot more of one cpu than the >>> others so It's really not SMP.. Can I stop it from doing this with >>> some setting? >>> Why can't there be 4 taskq's? >> >> it is possible, but it need to be coded. >> >> hz 4000 is also too high, use 1000-2000 >> http://www.tancsa.com/blast.html > > > Those are very old test results using a fairly different em driver > than whats in the tree. > > You might look at > http://people.yandex-team.ru/~wawa/ > on RELENG_6. I havent tested it, but supposedly its optimized for > forwarding on SMP hardware > > ---Mike > From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 01:21:17 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B4AD91065673 for ; Mon, 30 Jun 2008 01:21:17 +0000 (UTC) (envelope-from andrew@modulus.org) Received: from email.octopus.com.au (host-122-100-2-232.octopus.com.au [122.100.2.232]) by mx1.freebsd.org (Postfix) with ESMTP id 76FC78FC1A for ; Mon, 30 Jun 2008 01:21:16 +0000 (UTC) (envelope-from andrew@modulus.org) Received: by email.octopus.com.au (Postfix, from userid 1002) id 4AF2A173B5; Mon, 30 Jun 2008 11:21:14 +1000 (EST) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on email.octopus.com.au X-Spam-Level: X-Spam-Status: No, score=-1.4 required=10.0 tests=ALL_TRUSTED autolearn=failed version=3.2.3 Received: from [10.20.30.101] (60.218.233.220.exetel.com.au [220.233.218.60]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: admin@email.octopus.com.au) by email.octopus.com.au (Postfix) with ESMTP id AEFD1173AD for ; Mon, 30 Jun 2008 11:21:09 +1000 (EST) Message-ID: <486834F5.8080307@modulus.org> Date: Mon, 30 Jun 2008 11:20:53 +1000 From: Andrew Snow User-Agent: Thunderbird 1.5.0.9 (Windows/20061207) MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <4867B2B3.3090208@shrew.net> <48680DB8.708@shrew.net> In-Reply-To: <48680DB8.708@shrew.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: FreeBSD NAT-T patch integration X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 01:21:17 -0000 I've just started moving a medium IPSEC+gif VPN to one based on OpenVPN. OpenVPN solved all my problems with IPSEC: * does not require kernel modules or recompiles * works over UDP by default (and optionally TCP) + only requires a single IP port at each end * supports compression out of the box * supports bridging as well as tunneling Despite that, I didn't have to give up features or performance: * fast and secure enough (authentication, replay prevention) * very easy to configure & manage via either CLI/config files * supports both preshared keys or standard TLS+certs * also works on linux and windows. * supports hardware acceleration via openssl engines FWIW, I will probably never go back to IPSEC after this. - Andrew From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 01:44:19 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1858C1065674 for ; Mon, 30 Jun 2008 01:44:19 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 58DE28FC26 for ; Mon, 30 Jun 2008 01:44:17 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 31963 invoked from network); 30 Jun 2008 03:44:16 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 30 Jun 2008 03:44:16 +0200 Date: Mon, 30 Jun 2008 03:44:15 +0200 (CEST) From: Ingo Flaschberger To: Paul In-Reply-To: <48682F15.6070707@gtcomm.net> Message-ID: References: <4867420D.7090406@gtcomm.net> <4867A9A1.9070507@gtcomm.net> <48681A3D.9040509@gtcomm.net> <200806300034.m5U0YfsF077111@lava.sentex.ca> <48682F15.6070707@gtcomm.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net , Mike Tancsa Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 01:44:19 -0000 Dear Paul, > I tried this.. I put 6-STABLE (6.3), using default driver was slower than > FBSD7 have you set the rx/tx buffers? /boot/loader.conf hw.em.rxd=4096 hw.em.txd=4096 bye, Ingo From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 02:48:18 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C9BE106566C for ; Mon, 30 Jun 2008 02:48:18 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id D55348FC17 for ; Mon, 30 Jun 2008 02:48:17 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KD9Ns-0008Ka-Gr; Sun, 29 Jun 2008 22:44:52 -0400 Message-ID: <486849E9.6010405@gtcomm.net> Date: Sun, 29 Jun 2008 22:50:17 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <4867A9A1.9070507@gtcomm.net> <48681A3D.9040509@gtcomm.net> <200806300034.m5U0YfsF077111@lava.sentex.ca> <48682F15.6070707@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Mike Tancsa Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 02:48:18 -0000 The higher I set the buffer the worse it is.. 256 and 512 I get about 50-60k more pps than i do with 2048 or 4096.. You would think it would be the other way around but obviously there is some contention going on. :/ I'm sticking with 512 for now, as it seems to make it worse with anything higher. Keep in mind, i'm using random source ips, random source and destination ports.. Although that should have zero impact on the amount of PPS it can route but for some reason it seems to.. ? Any ideas on that one? A single stream one source ip/port to one destination ip/port seems to use less cpu, although I haven't generated the same pps with that yet.. I am going to test it soon Ingo Flaschberger wrote: > Dear Paul, > >> I tried this.. I put 6-STABLE (6.3), using default driver was slower >> than FBSD7 > > have you set the rx/tx buffers? > > /boot/loader.conf > hw.em.rxd=4096 > hw.em.txd=4096 > > bye, > Ingo > From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 04:27:46 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C099D1065678 for ; Mon, 30 Jun 2008 04:27:46 +0000 (UTC) (envelope-from lab@gta.com) Received: from mailgate.gta.com (mailgate.gta.com [199.120.225.20]) by mx1.freebsd.org (Postfix) with SMTP id 551A98FC18 for ; Mon, 30 Jun 2008 04:27:46 +0000 (UTC) (envelope-from lab@gta.com) Received: (qmail 94731 invoked by uid 1000); 30 Jun 2008 04:01:03 -0000 Date: 30 Jun 2008 04:01:03 -0000 Message-ID: <20080630040103.94730.qmail@mailgate.gta.com> From: Larry Baird To: freebsd-net@freebsd.org In-Reply-To: <108528.171316.50257@localhost> User-Agent: tin/1.8.3-20070201 ("Scotasay") (UNIX) (FreeBSD/6.3-PRERELEASE (i386)) Cc: vanhu_bsd@zeninc.net Subject: Re: FreeBSD NAT-T patch integration X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 04:27:46 -0000 > And how do I know that it works ? > Well, when it doesn't work, I do know it, quite quickly most of the > time ! I have to chime in here. I did most of the initial porting of the NAT-T patches from Kame IPSec to FAST_IPSEC. I did look at every line of code during this process. I found no security problems during the port. Like Yvan, my company uses the NAT-T patches commercially. Like he says, if it had problems, we would hear about it. If the patches don't get commited, I highly suspect Yvan or myself would try to keep the patches up todate. So far I have done FAST_IPSEC pacthes for FreeBSD 4,5,6. Yvan did 7 and 8 by himself. Keeping up gets to be a pain after a while. I do plan to look at the FreeBSD 7 patches soon, but it sure would be nice to see it commited. Larry -- ------------------------------------------------------------------------ Larry Baird | http://www.gta.com Global Technology Associates, Inc. | Orlando, FL Email: lab@gta.com | TEL 407-380-0220, FAX 407-380-6080 From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 08:27:12 2008 Return-Path: Delivered-To: net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BF745106566C; Mon, 30 Jun 2008 08:27:12 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 9087B8FC21; Mon, 30 Jun 2008 08:27:12 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 133FA46C0D; Mon, 30 Jun 2008 04:27:12 -0400 (EDT) Date: Mon, 30 Jun 2008 09:27:11 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: net@FreeBSD.org, current@FreeBSD.org In-Reply-To: <20080629180126.F90836@fledge.watson.org> Message-ID: <20080630091033.P3968@fledge.watson.org> References: <20080524111715.T64552@fledge.watson.org> <20080629180126.F90836@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: arch@FreeBSD.org Subject: Re: HEAD UP: non-MPSAFE network drivers to be disabled (was: 8.0 network stack MPsafety goals (fwd)) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 08:27:12 -0000 On Sun, 29 Jun 2008, Robert Watson wrote: > An FYI on the state of things here: in the last month, John has updated a > number of device drivers to be MPSAFE, and the USB work remains in-flight. > I'm holding fire a bit on disabling IFF_NEEDSGIANT while things settle and I > catch up on driver state, and will likely send out an update next week > regarding which device drivers remain on the kill list, and generally what > the status of this project is. Here's the revised list of drivers that will have their build disabled in the next week (subject to an appropriate block of time for me): Name Bus Man page description ---- --- -------------------- ar ISA/PCI synchronous Digi/Arnet device driver arl ISA Aironet Arlan 655 wireless network adapter driver cnw ISA Netwave AirSurfer wireless network driver ic I2C I2C bus system oltr ISA/PCI Olicom Token Ring device driver plip PPBUS printer port Internet Protocol driver ppp TTY point to point protocol network interface ray PCCARD Raytheon Raylink/Webgear Aviator PCCard driver sbni ISA/PCI Granch SBNI12 leased line modem driver sbsh PCI Granch SBNI16 SHDSL modem device driver sl TTY slip network interface snc ISA/PCCARD National Semiconductor DP8393X SONIC Ethernet adapter driver sppp TTY point to point protocol network layer for synchronous lines sr ISA/PCI synchronous RISCom/N2 / WANic 400/405 device driver Obviously, if necessary work is done to remove the IFF_NEEDSGIANT requirement from a driver, it will be pulled from the list, and I'll do an IFF_NEEDSGIANT scan before pulling the plug. Drivers will remain in the tree but disconnected for about a month before being removed from HEAD. Thanks greatly to John and others who have worked hard to reduce the size of the list in the last year! The following USB drivers will remain enabled due to on-going USB work that should eliminate IFF_NEEDSGIANT: Name Bus Man page description ---- --- -------------------- axe USB ASIX Electronics AX88172 USB Ethernet driver cdce USB USB Communication Device Class Ethernet driver cue USB CATC USB-EL1210A USB Ethernet driver kue USB Kawasaki LSI KL5KUSB101B USB Ethernet driver rue USB RealTek RTL8150 USB to Fast Ethernet controller rum USB Ralink Technology USB IEEE 802.11a/b/g wireless network device udav USB Davicom DM9601 USB Ethernet driver ural USB Ralink Technology RT2500USB IEEE 802.11 driver zyd USB ZyDAS ZD1211/ZD1211B USB IEEE 802.11b/g wireless network device The following drivers reference IFF_NEEDSGIANT but only when running in an optional non-MPSAFE mode; that optional mode will be removed but the drivers will remain: Name Bus Man page description ---- --- -------------------- ce PCI driver for synchronous Cronyx Tau-PCI/32 WAN adapters cp PCI driver for synchronous Cronyx Tau-PCI WAN adapters ctau ISA driver for synchronous Cronyx Tau WAN adapters cx ISA driver for synchronous/asynchronous Cronyx Sigma WAN adapters Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 09:11:45 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4C6F01065681 for ; Mon, 30 Jun 2008 09:11:45 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 0365E8FC0C for ; Mon, 30 Jun 2008 09:11:44 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id BEE6C1B10E4E; Mon, 30 Jun 2008 11:11:43 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on malcho.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.4 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 1385C1B10CAA for ; Mon, 30 Jun 2008 11:11:40 +0200 (CEST) Message-ID: <4868A34C.6030304@moneybookers.com> Date: Mon, 30 Jun 2008 12:11:40 +0300 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.14 (X11/20080616) MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on blah.cmotd.com X-Virus-Status: Clean Subject: if_bridge turns off checksum offload of members? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 09:11:45 -0000 Greetings, I just noticed, that when I add em network card to bridge the checksum offload is turned off. I even put in my rc.conf: ifconfig_em0="rxcsum up" ifconfig_em1="rxcsum up" but after reboot both em0 and em1 have this feature disabled. Is this expected behavior? Should I care about csum in bridge mode? I noticed that enabling checksum offload manually improve things little btw. Also I'm experimenting with bridge performance and with today's 7-stable I can't reach the results from my previous test with 7-current (before few months) The best that bridge can do today is just 720kpps (just incoming) vs 1000kpps with sources from few months ago. I'm using the same hardware and same configuration so I'm not sure why -stable is slower. -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 10:18:42 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CE3F81065673 for ; Mon, 30 Jun 2008 10:18:42 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.235]) by mx1.freebsd.org (Postfix) with ESMTP id 97F018FC12 for ; Mon, 30 Jun 2008 10:18:42 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so1476659rvf.43 for ; Mon, 30 Jun 2008 03:18:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:received:date:from :to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=UL5lbijdQ/Q0nSk0vsqQZbdHYdw/wTzOf0RX9FPvfoo=; b=aPQrLRJCqaOOm+fuhMBzyOEUKiYVWd/9WHMJFs9wFtob4/TO++itxVHSOlRMDVfGKs OOkouUnXMrBxs19fDa2PpW7gqWuswxVBoXpbOKeyQMN6ONnZbnmvyyEN2h9cFSyHO7eO DNvGEr/+0EIuTsBYirv17bdRwhOGeubwR/5Jk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=RzPU3moybttclHZ2A7mjC5jv93l6CRPMjrWwcobMHiTIFnw7udUX4uRSHroWuXW1sL 7mRTJcJXEkaTeJ/aYjok0nrGkiy3jHn3bXZsAwmgUwO2EAKXqALapOeoIfFvK68F3ywQ 4KlEWVmE3787wmQLUqn7VbdPJzn/7ghPQMUZg= Received: by 10.140.250.14 with SMTP id x14mr2526778rvh.79.1214821122332; Mon, 30 Jun 2008 03:18:42 -0700 (PDT) Received: from michelle.cdnetworks.co.kr ( [211.53.35.84]) by mx.google.com with ESMTPS id g31sm7477149rvb.2.2008.06.30.03.18.39 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 30 Jun 2008 03:18:41 -0700 (PDT) Received: from michelle.cdnetworks.co.kr (localhost.cdnetworks.co.kr [127.0.0.1]) by michelle.cdnetworks.co.kr (8.13.5/8.13.5) with ESMTP id m5UAGWCa081297 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 30 Jun 2008 19:16:32 +0900 (KST) (envelope-from pyunyh@gmail.com) Received: (from yongari@localhost) by michelle.cdnetworks.co.kr (8.13.5/8.13.5/Submit) id m5UAGTAP081296; Mon, 30 Jun 2008 19:16:29 +0900 (KST) (envelope-from pyunyh@gmail.com) Date: Mon, 30 Jun 2008 19:16:29 +0900 From: Pyun YongHyeon To: Stefan Lambrev Message-ID: <20080630101629.GD79537@cdnetworks.co.kr> References: <4868A34C.6030304@moneybookers.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4868A34C.6030304@moneybookers.com> User-Agent: Mutt/1.4.2.1i Cc: freebsd-net@freebsd.org Subject: Re: if_bridge turns off checksum offload of members? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 10:18:43 -0000 On Mon, Jun 30, 2008 at 12:11:40PM +0300, Stefan Lambrev wrote: > Greetings, > > I just noticed, that when I add em network card to bridge the checksum > offload is turned off. > I even put in my rc.conf: > ifconfig_em0="rxcsum up" > ifconfig_em1="rxcsum up" > but after reboot both em0 and em1 have this feature disabled. > > Is this expected behavior? Should I care about csum in bridge mode? > I noticed that enabling checksum offload manually improve things little btw. > AFAIK this is intended one, bridge(4) turns off Tx side checksum offload by default. I think disabling Tx checksum offload is required as not all members of a bridge may be able to do checksum offload. The same is true for TSO but it seems that bridge(4) doesn't disable it. If all members of bridge have the same hardware capability I think bridge(4) may not need to disable Tx side hardware assistance. I guess bridge(4) can scan every interface capabilities in a member and can decide what hardware assistance can be activated instead of blindly turning off Tx side hardware assistance. > Also I'm experimenting with bridge performance and with today's 7-stable > I can't reach > the results from my previous test with 7-current (before few months) > > The best that bridge can do today is just 720kpps (just incoming) vs > 1000kpps with sources from few months ago. > I'm using the same hardware and same configuration so I'm not sure why > -stable is slower. > > -- > > Best Wishes, > Stefan Lambrev > ICQ# 24134177 > -- Regards, Pyun YongHyeon From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 11:07:01 2008 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 540B610656D2 for ; Mon, 30 Jun 2008 11:07:01 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1FBE58FC0A for ; Mon, 30 Jun 2008 11:07:01 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m5UB71kK095822 for ; Mon, 30 Jun 2008 11:07:01 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m5UB703A095818 for freebsd-net@FreeBSD.org; Mon, 30 Jun 2008 11:07:00 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 30 Jun 2008 11:07:00 GMT Message-Id: <200806301107.m5UB703A095818@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-net@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-net@FreeBSD.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 11:07:01 -0000 Current FreeBSD problem reports Critical problems Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/27474 net [ipf] [ppp] Interactive use of user PPP and ipfilter c o kern/35442 net [sis] [patch] Problem transmitting runts in if_sis dri a kern/38554 net changing interface ipaddress doesn't seem to work s kern/39937 net ipstealth issue s kern/77195 net [ipf] [patch] ipfilter ioctl SIOCGNATL does not match o kern/79895 net [ipf] 5.4-RC2 breaks ipfilter NAT when using netgraph s kern/81147 net [net] [patch] em0 reinitialization while adding aliase o kern/86103 net [ipf] Illegal NAT Traversal in IPFilter s kern/86920 net [ndis] ifconfig: SIOCS80211: Invalid argument [regress o kern/87521 net [ipf] [panic] using ipfilter "auth" keyword leads to k o kern/92090 net [bge] bge0: watchdog timeout -- resetting f kern/92552 net A serious bug in most network drivers from 5.X to 6.X o kern/95288 net [pppd] [tty] [panic] if_ppp panic in sys/kern/tty_subr o kern/98978 net [ipf] [patch] ipfilter drops OOW packets under 6.1-Rel o kern/101948 net [ipf] [panic] Kernel Panic Trap No 12 Page Fault - cau f kern/102344 net [ipf] Some packets do not pass through network interfa o bin/105925 net problems with ifconfig(8) and vlan(4) [regression] s kern/105943 net Network stack may modify read-only mbuf chain copies o kern/106316 net [dummynet] dummynet with multipass ipfw drops packets o kern/106438 net [ipf] ipfilter: keep state does not seem to allow repl o kern/108542 net [bce]: Huge network latencies with 6.2-RELEASE / STABL o bin/108895 net pppd(8): PPPoE dead connections on 6.2 [regression] o kern/109308 net [pppd] [panic] Multiple panics kernel ppp suspected [r o kern/109733 net [bge] bge link state issues [regression] o kern/112528 net [nfs] NFS over TCP under load hangs with "impossible p o kern/112686 net [patm] patm driver freezes System (FreeBSD 6.2-p4) i38 o kern/112722 net [udp] IP v4 udp fragmented packet reject o kern/113842 net [ip6] PF_INET6 proto domain state can't be cleared wit o kern/114714 net [gre][patch] gre(4) is not MPSAFE and does not support o kern/114839 net [fxp] fxp looses ability to speak with traffic o kern/115239 net [ipnat] panic with 'kmem_map too small' using ipnat o kern/116077 net [ip] [patch] 6.2-STABLE panic during use of multi-cast o kern/116185 net [iwi] if_iwi driver leads system to reboot o kern/116328 net [bge]: Solid hang with bge interface o kern/116747 net [ndis] FreeBSD 7.0-CURRENT crash with Dell TrueMobile o kern/116837 net [tun] [panic] [patch] ifconfig tunX destroy: panic o kern/117043 net [em] Intel PWLA8492MT Dual-Port Network adapter EEPROM o kern/117271 net [tap] OpenVPN TAP uses 99% CPU on releng_6 when if_tap o kern/117423 net [vlan] Duplicate IP on different interfaces o kern/117448 net [carp] 6.2 kernel crash [regression] o kern/118880 net [ip6] IP_RECVDSTADDR & IP_SENDSRCADDR not implemented o kern/119225 net [wi] 7.0-RC1 no carrier with Prism 2.5 wifi card [regr o kern/119345 net [ath] Unsuported Atheros 5424/2424 and CPU speedstep n o kern/119361 net [bge] bge(4) transmit performance problem o kern/119945 net [rum] [panic] rum device in hostap mode, cause kernel o kern/120130 net [carp] [panic] carp causes kernel panics in any conste o kern/120266 net [panic] gnugk causes kernel panic when closing UDP soc o kern/120304 net [netgraph] [patch] netgraph source assumes 32-bit time o kern/120966 net [rum] kernel panic with if_rum and WPA encryption o kern/121080 net [bge] IPv6 NUD problem on multi address config on bge0 o kern/121181 net [panic] Fatal trap 3: breakpoint instruction fault whi o kern/121298 net [em] [panic] Fatal trap 12: page fault while in kernel o kern/121437 net [vlan] Routing to layer-2 address does not work on VLA o kern/121555 net [panic] Fatal trap 12: current process = 12 (swi1: net o kern/121624 net [em] [regression] Intel em WOL fails after upgrade to o kern/121872 net [wpi] driver fails to attach on a fujitsu-siemens s711 o kern/121983 net [fxp] fxp0 MBUF and PAE o kern/122033 net [ral] [lor] Lock order reversal in ral0 at bootup [reg o kern/122058 net [em] [panic] Panic on em1: taskq o kern/122082 net [in_pcb] NULL pointer dereference in in_pcbdrop o kern/122195 net [ed] Alignment problems in if_ed f kern/122252 net [ipmi] [bge] IPMI problem with BCM5704 (does not work o kern/122290 net [netgraph] [panic] Netgraph related "kmem_map too smal o kern/122427 net [apm] [panic] apm and mDNSResponder cause panic during o kern/122551 net [bge] Broadcom 5715S no carrier on HP BL460c blade usi o kern/122743 net [panic] vm_page_unwire: invalid wire count: 0 o kern/122772 net [em] em0 taskq panic, tcp reassembly bug causes radix f kern/122794 net [lagg] Kernel panic after brings lagg(8) up if NICs ar f conf/122858 net [nsswitch.conf] nsswitch in 7.0 is f*cked up o kern/122954 net [lagg] IPv6 EUI64 incorrectly chosen for lagg devices o kern/122989 net [swi] [panic] 6.3 kernel panic in swi1: net o kern/123066 net [ipsec] [panic] kernel trap with ipsec o kern/123160 net [ip] Panic and reboot at sysctl kern.polling.enable=0 f kern/123172 net [bce] Watchdog timeout problems with if_bce o conf/123330 net [nsswitch.conf] Enabling samba wins in nsswitch.conf c o kern/123347 net [bge] bge1: watchdog timeout -- linkstate changed to D o kern/123429 net [nfe] [hang] "ifconfig nfe up" causes a hard system lo o kern/123463 net [ipsec] [panic] repeatable crash related to ipsec-tool o bin/123465 net [ip6] route(8): route add -inet6 -interfac o kern/123559 net [iwi] iwi periodically disassociates/associates [regre o kern/123603 net [tcp] tcp_do_segment and Received duplicate SYN o kern/123617 net [tcp] breaking connection when client downloading file o bin/123633 net ifconfig(8) doesn't set inet and ether address in one o kern/123796 net [ipf] FreeBSD 6.1+VPN+ipnat+ipf: port mapping does not o kern/123881 net [tcp] Turning on TCP blackholing causes slow localhost o kern/123968 net [rum] [panic] rum driver causes kernel panic with WPA. o kern/124021 net [ip6] [panic] page fault in nd6_output() o kern/124127 net [msk] watchdog timeout (missed Tx interrupts) -- recov o kern/124540 net [tcp] RTM_MISS with the transit packets o kern/124753 net net80211 discards power-save queue packets early o kern/124904 net [fxp] EEPROM corruption with Compaq NC3163 NIC o kern/125079 net [ppp] host routes added by ppp with gateway flag (regr 92 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- o conf/23063 net [PATCH] for static ARP tables in rc.network o kern/34665 net [ipf] [hang] ipfilter rcmd proxy "hangs". s bin/41647 net ifconfig(8) doesn't accept lladdr along with inet addr o kern/54383 net [nfs] [patch] NFS root configurations without dynamic s kern/60293 net FreeBSD arp poison patch o kern/64556 net [sis] if_sis short cable fix problems with NetGear FA3 o kern/70904 net [ipf] ipfilter ipnat problem with h323 proxy support o kern/77273 net [ipf] ipfilter breaks ipv6 statefull filtering on 5.3 o kern/77913 net [wi] [patch] Add the APDL-325 WLAN pccard to wi(4) o kern/78090 net [ipf] ipf filtering on bridged packets doesn't work if o bin/79228 net [patch] extend arp(8) to be able to create blackhole r o kern/91594 net [em] FreeBSD > 5.4 w/ACPI fails to detect Intel Pro/10 s kern/91777 net [ipf] [patch] wrong behaviour with skip rule inside an o kern/93378 net [tcp] Slow data transfer in Postfix and Cyrus IMAP (wo o kern/95267 net packet drops periodically appear o kern/95277 net [netinet] [patch] IP Encapsulation mask_match() return o kern/100519 net [netisr] suggestion to fix suboptimal network polling o kern/102035 net [plip] plip networking disables parallel port printing o conf/102502 net [patch] ifconfig name does't rename netgraph node in n o conf/107035 net [patch] bridge interface given in rc.conf not taking a o kern/109470 net [wi] Orinoco Classic Gold PC Card Can't Channel Hop o kern/112179 net [sis] [patch] sis driver for natsemi DP83815D autonego o bin/112557 net [patch] ppp(8) lock file should not use symlink name o kern/114915 net [patch] [pcn] pcn (sys/pci/if_pcn.c) ethernet driver f o bin/116643 net [patch] [request] fstat(1): add INET/INET6 socket deta o bin/117339 net [patch] route(8): loading routing management commands o kern/118727 net [netgraph] [patch] [request] add new ng_pf module a kern/118879 net [bge] [patch] bge has checksum problems on the 5703 ch o bin/118987 net ifconfig(8): ifconfig -l (address_family) does not wor o kern/119432 net [arp] route add -host -iface causes arp e f kern/119516 net [ip6] [panic] _mtx_lock_sleep: recursed on non-recursi o kern/119617 net [nfs] nfs error on wpa network when reseting/shutdown o kern/119791 net [nfs] UDP NFS mount of aliased IP addresses from a Sol o kern/120232 net [nfe] [patch] Bring in nfe(4) to RELENG_6 o kern/120566 net [request]: ifconfig(8) make order of arguments more fr o kern/121257 net [tcp] TSO + natd -> slow outgoing tcp traffic o kern/121443 net [gif] LOR icmp6_input/nd6_lookup o kern/121706 net [netinet] [patch] "rtfree: 0xc4383870 has 1 refs" emit s kern/121774 net [swi] [panic] 6.3 kernel panic in swi1: net o kern/122068 net [ppp] ppp can not set the correct interface with pptpd p kern/122145 net error while compiling with device ath_rate_amrr o kern/122295 net [bge] bge Ierr rate increase (since 6.0R) [regression] o kern/122319 net [wi] imposible to enable ad-hoc demo mode with Orinoco o kern/122697 net [ath] Atheros card is not well supported o kern/122780 net [lagg] tcpdump on lagg interface during high pps wedge f kern/122839 net [multicast] FreeBSD 7 multicast routing problem o kern/122928 net [em] interface watchdog timeouts and stops receiving p o kern/123892 net [tap] [patch] No buffer space available p kern/123961 net [vr] [patch] Allow vr interface to handle vlans o bin/124004 net ifconfig(8): Cannot assign both an IP and a MAC addres o kern/124225 net [ndis] [patch] ndis network driver sometimes loses net o kern/124341 net [ral] promiscuous mode for wireless device ral0 looses o kern/124609 net [ipsec] [panic] ipsec 'remainder too big' panic with p o kern/124767 net [iwi] Wireless connection using iwi0 driver (Intel 220 o kern/125003 net [gif] incorrect EtherIP header format. 55 problems total. From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 11:32:23 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23E10106564A for ; Mon, 30 Jun 2008 11:32:23 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id C53518FC2A for ; Mon, 30 Jun 2008 11:32:22 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 4030A1B10EBB; Mon, 30 Jun 2008 13:32:21 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on malcho.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.4 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id EA2A31B10E4E; Mon, 30 Jun 2008 13:32:18 +0200 (CEST) Message-ID: <4868C442.7030406@moneybookers.com> Date: Mon, 30 Jun 2008 14:32:18 +0300 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.14 (X11/20080616) MIME-Version: 1.0 To: Paul References: <4867420D.7090406@gtcomm.net> <4867A9A1.9070507@gtcomm.net> <48681A3D.9040509@gtcomm.net> <200806300034.m5U0YfsF077111@lava.sentex.ca> <48682F15.6070707@gtcomm.net> <486849E9.6010405@gtcomm.net> In-Reply-To: <486849E9.6010405@gtcomm.net> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on blah.cmotd.com X-Virus-Status: Clean Cc: FreeBSD Net , Ingo Flaschberger , Mike Tancsa Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 11:32:23 -0000 Paul wrote: > The higher I set the buffer the worse it is.. 256 and 512 I get about > 50-60k more pps than i do with 2048 or 4096.. You > would think it would be the other way around but obviously there is > some contention going on. :/ Looks like in bridge mode hw.em.rxd=512 and hw.em.txd=512 yields best results also. reducing or increasing those leads to worse performance. btw is there any news with hwpmc for new CPUs ? last time I checked was real pain to get it working with core2 CPUs :( > I'm sticking with 512 for now, as it seems to make it worse with > anything higher. > Keep in mind, i'm using random source ips, random source and > destination ports.. Although that should have zero impact on the > amount of PPS it can route but for some reason it seems to.. ? Any > ideas on that one? A single stream one source ip/port to one > destination ip/port seems to use less cpu, although I haven't > generated the same pps with that yet.. I am going to test it soon > > Ingo Flaschberger wrote: >> Dear Paul, >> >>> I tried this.. I put 6-STABLE (6.3), using default driver was >>> slower than FBSD7 >> >> have you set the rx/tx buffers? >> >> /boot/loader.conf >> hw.em.rxd=4096 >> hw.em.txd=4096 >> >> bye, >> Ingo >> > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 11:37:51 2008 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5D0AC106566C; Mon, 30 Jun 2008 11:37:51 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 0117D8FC23; Mon, 30 Jun 2008 11:37:51 +0000 (UTC) (envelope-from gavin@FreeBSD.org) Received: from freefall.freebsd.org (gavin@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m5UBbo7k003434; Mon, 30 Jun 2008 11:37:50 GMT (envelope-from gavin@freefall.freebsd.org) Received: (from gavin@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m5UBboEp003430; Mon, 30 Jun 2008 11:37:50 GMT (envelope-from gavin) Date: Mon, 30 Jun 2008 11:37:50 GMT Message-Id: <200806301137.m5UBboEp003430@freefall.freebsd.org> To: gavin@FreeBSD.org, gavin@FreeBSD.org, freebsd-net@FreeBSD.org From: gavin@FreeBSD.org Cc: Subject: Re: kern/124160: connect() function loops indefinitely X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 11:37:51 -0000 Synopsis: connect() function loops indefinitely Responsible-Changed-From-To: gavin->freebsd-net Responsible-Changed-By: gavin Responsible-Changed-When: Mon Jun 30 11:36:12 UTC 2008 Responsible-Changed-Why: Over to maintainers. http://www.freebsd.org/cgi/query-pr.cgi?pr=124160 From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 13:37:09 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5385F106564A for ; Mon, 30 Jun 2008 13:37:09 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: from ibctech.ca (v6.ibctech.ca [IPv6:2607:f118::b6]) by mx1.freebsd.org (Postfix) with SMTP id E123A8FC20 for ; Mon, 30 Jun 2008 13:37:08 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: (qmail 20933 invoked by uid 89); 30 Jun 2008 13:38:41 -0000 Received: from unknown (HELO ?IPv6:2607:f118::5?) (steve@ibctech.ca@2607:f118::5) by 2607:f118::b6 with ESMTPA; 30 Jun 2008 13:38:41 -0000 Message-ID: <4868E183.2070504@ibctech.ca> Date: Mon, 30 Jun 2008 09:37:07 -0400 From: Steve Bertrand User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: "Tuc at T-B-O-H.NET" References: <200806270430.m5R4U5Ib025336@himinbjorg.tucs-beachin-obx-house.com> In-Reply-To: <200806270430.m5R4U5Ib025336@himinbjorg.tucs-beachin-obx-house.com> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: IPV6 problem : nd6_lookup: failed to add route for a neighbor X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 13:37:09 -0000 Tuc at T-B-O-H.NET wrote: > But once I brought it all up, I got : > > kernel: nd6_lookup: failed to add route for a neighbor(2001:0470:0007:0028::0001), errno=17 With your exact configuration between two 7.0 boxes, I see no indication of this error whatsoever, with the /128 prefix. > The tunnel came up, was passing traffic, but those messages were > getting out of hand. I tried a prefixlen of 64, and I got: > > ifconfig: ioctl (SIOCAIFADDR): Invalid argument When I attempted to recreate the tunnel with any prefix length other than 128, the same ifconfig error appeared as above. It appears as though when creating a gif tunnel, the only IPv6 prefix length that is valid as far as ifconfig is concerned is /128 even though the link's prefix is /64 even in 7. Steve From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 15:20:12 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5913C1065671 for ; Mon, 30 Jun 2008 15:20:12 +0000 (UTC) (envelope-from ml@t-b-o-h.net) Received: from vjofn.tucs-beachin-obx-house.com (unknown [IPv6:2001:470:1f00:ffff::5e5]) by mx1.freebsd.org (Postfix) with ESMTP id EBBD88FC15 for ; Mon, 30 Jun 2008 15:20:11 +0000 (UTC) (envelope-from ml@t-b-o-h.net) Received: from himinbjorg.tucs-beachin-obx-house.com (cpe-24-161-6-139.hvc.res.rr.com [24.161.6.139]) (authenticated bits=0) by vjofn.tucs-beachin-obx-house.com (8.14.2/8.14.2) with ESMTP id m5UFJRe5065828; Mon, 30 Jun 2008 11:19:37 -0400 (EDT) Received: from himinbjorg.tucs-beachin-obx-house.com (localhost.tucs-beachin-obx-house.com [127.0.0.1]) by himinbjorg.tucs-beachin-obx-house.com (8.13.8/8.13.6) with ESMTP id m5UFJtMP072578; Mon, 30 Jun 2008 11:19:55 -0400 (EDT) (envelope-from ml@t-b-o-h.net) Received: (from tbohml@localhost) by himinbjorg.tucs-beachin-obx-house.com (8.13.8/8.13.6/Submit) id m5UFJsOe072577; Mon, 30 Jun 2008 11:19:54 -0400 (EDT) (envelope-from tbohml) From: "Tuc at T-B-O-H.NET" Message-Id: <200806301519.m5UFJsOe072577@himinbjorg.tucs-beachin-obx-house.com> To: steve@ibctech.ca (Steve Bertrand) Date: Mon, 30 Jun 2008 11:19:54 -0400 (EDT) In-Reply-To: <4868E183.2070504@ibctech.ca> X-Mailer: ELM [version 2.5 PL8] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, "Tuc at T-B-O-H.NET" Subject: Re: [freebsd-net] Re: IPV6 problem : nd6_lookup: failed to add route for a neighbor X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 15:20:12 -0000 > > Tuc at T-B-O-H.NET wrote: > > > But once I brought it all up, I got : > > > > kernel: nd6_lookup: failed to add route for a neighbor(2001:0470:0007:0028::0001), errno=17 > > With your exact configuration between two 7.0 boxes, I see no indication > of this error whatsoever, with the /128 prefix. > Possibly because of : http://lists.freebsd.org/pipermail/freebsd-net/2006-May/010718.html http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/93220 which wouldn't apply to me due to my version. > > > The tunnel came up, was passing traffic, but those messages were > > getting out of hand. I tried a prefixlen of 64, and I got: > > > > ifconfig: ioctl (SIOCAIFADDR): Invalid argument > > When I attempted to recreate the tunnel with any prefix length other > than 128, the same ifconfig error appeared as above. It appears as > though when creating a gif tunnel, the only IPv6 prefix length that is > valid as far as ifconfig is concerned is /128 even though the link's > prefix is /64 even in 7. > It seems if I just did : ifconfig gif0 inet6 2001:470:7:28::2/64 instead of : ifconfig gif0 inet6 2001:470:7:28::2 2001:470:7:28::1 prefixlen 128 It "got happy". I haven't seen any of the messages yet, and so far no issues I can readily see. Tuc From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 19:18:55 2008 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2875B1065679; Mon, 30 Jun 2008 19:18:55 +0000 (UTC) (envelope-from 20080111.freebsd.org@ab.ote.we.lv) Received: from purple.the-7.net (purple.the-7.net [IPv6:2001:470:1f01:622:230:48ff:fe23:4c67]) by mx1.freebsd.org (Postfix) with ESMTP id 0D0948FC0A; Mon, 30 Jun 2008 19:18:55 +0000 (UTC) (envelope-from 20080111.freebsd.org@ab.ote.we.lv) Received: from redion.astralblue.net (c-76-21-113-189.hsd1.ca.comcast.net [76.21.113.189]) by purple.the-7.net (8.14.2/8.14.2) with ESMTP id m5UJIgUE024313; Mon, 30 Jun 2008 12:18:42 -0700 (PDT) (envelope-from 20080111.freebsd.org@ab.ote.we.lv) Message-ID: <48693193.1020404@ab.ote.we.lv> Date: Mon, 30 Jun 2008 12:18:43 -0700 From: "Eugene M. Kim" <20080111.freebsd.org@ab.ote.we.lv> User-Agent: Thunderbird 2.0.0.14 (X11/20080620) MIME-Version: 1.0 To: pyunyh@gmail.com References: <200806270345.m5R3j1BT036253@freefall.freebsd.org> <48649776.9040509@ab.ote.we.lv> <20080627074948.GC67753@cdnetworks.co.kr> <4864A217.3040309@ab.ote.we.lv> <20080629072932.GA76469@cdnetworks.co.kr> In-Reply-To: <20080629072932.GA76469@cdnetworks.co.kr> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=2.9 required=10.0 tests=FROM_STARTS_WITH_NUMS, RCVD_IN_PBL, RCVD_IN_SORBS_DUL, RDNS_DYNAMIC autolearn=disabled version=3.2.3 X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on purple.the-7.net Cc: freebsd-net@FreeBSD.org, yongari@FreeBSD.org Subject: Re: kern/125024: vr(4) does not see incoming multicast packets in non-promiscuous mode (broadcast is fine); breaks IPv6 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 19:18:55 -0000 Than you! The new patch fixed the problem. I'll put it under test for a few more days and let you know if any regression is seen. Cheers, Eugene Pyun YongHyeon wrote: > On Fri, Jun 27, 2008 at 01:17:27AM -0700, Eugene M. Kim wrote: > > Pyun YongHyeon wrote: > > >I've updated patch again. There was a bug that disabled > > >multicasting filter. Back out previous patch and try again. > > >The URL is the same as before. > > > > > > > Regards, > > > > Eugene > > > > > > > rtsol still doesn't work with vr0 being in non-promiscuous mode. > > However, apparently vr0 picked up router solicitations during system > > boot-up, as ifconfig shows the correct prefixes autoconfigured. It > > seems something goes wrong in between. 'o 'a > > > > Oops, I was accessing CAM mask register as 8bit register which > should be 32bit! Try the patch at the following URL. > > http://people.freebsd.org/~yongari/vr/vr.cam.patch2 > > > Eugene > From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 19:44:43 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C02581065677 for ; Mon, 30 Jun 2008 19:44:43 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id 9A04E8FC1E for ; Mon, 30 Jun 2008 19:44:43 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.2/8.14.2) with ESMTP id m5UJifQE005284; Mon, 30 Jun 2008 15:44:41 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m5UJifJD081781 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 30 Jun 2008 15:44:41 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200806301944.m5UJifJD081781@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Mon, 30 Jun 2008 15:44:48 -0400 To: Paul , FreeBSD Net From: Mike Tancsa In-Reply-To: <4867420D.7090406@gtcomm.net> References: <4867420D.7090406@gtcomm.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 19:44:43 -0000 At 04:04 AM 6/29/2008, Paul wrote: >This is just a question but who can get more than 400k pps >forwarding performance ? OK, I setup 2 boxes on either end of a RELENG_7 box from about May 7th just now, to see with 2 boxes blasting across it how it would work. *However*, this is with no firewall loaded and, I must enable ip fast forwarding. Without that enabled, the box just falls over. even at 20Kpps, I start seeing all sorts of messages spewing to route -n monitor got message of size 96 on Mon Jun 30 15:39:10 2008 RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default I am starting to wonder if those messages are the results of corrupted packets the machine just cant keep up with ? CPU is CPU: Intel(R) Xeon(R) CPU 3070 @ 2.66GHz (2660.01-MHz 686-class CPU) input (Total) output packets errs bytes packets errs bytes colls 611945 0 77892098 611955 0 77013002 0 616727 0 78215508 616742 0 77303454 0 617066 0 78162130 617082 0 77238434 0 618238 0 78302314 618225 0 77377582 0 617035 0 78141000 617038 0 77215672 0 617625 0 78225600 617588 0 77301734 0 616190 0 78017320 616165 0 77091774 0 615583 0 78064130 615628 0 77152800 0 617662 0 78254388 617658 0 77332340 0 618000 0 78269912 617950 0 77344554 0 617248 0 78183136 617315 0 77259588 0 617325 0 78204566 617289 0 77282094 0 618391 0 78337734 618357 0 77413756 0 616025 0 78116070 616082 0 77203116 0 To generate the packets, I am just using /usr/src/tools/tools/netblast on 2 endpoints starting at about the same time # ./netblast 10.10.1.2 500 100 40 start: 1214854131.083679919 finish: 1214854171.084668592 send calls: 20139141 send errors: 0 approx send rate: 503478 approx error rate: 0 # ./netblast 10.10.1.3 500 10 40 start: 1214854273.882202815 finish: 1214854313.882319031 send calls: 23354971 send errors: 18757223 approx send rate: 114943 approx error rate: 0 The box in the middle doing the forwarding 1[spare-r7]# ifconfig -u em0: flags=8843 metric 0 mtu 1500 options=19b ether 00:1b:21:08:32:a8 inet 10.20.1.1 netmask 0xffffff00 broadcast 10.20.1.255 media: Ethernet autoselect (1000baseTX ) status: active em1: flags=8843 metric 0 mtu 1500 options=9b ether 00:1b:21:08:32:a9 inet 192.168.43.193 netmask 0xffffff00 broadcast 192.168.43.255 media: Ethernet autoselect (100baseTX ) status: active em3: flags=8843 metric 0 mtu 1500 options=19b ether 00:30:48:90:4c:ff inet 10.10.1.1 netmask 0xffffff00 broadcast 10.10.1.255 media: Ethernet autoselect (1000baseTX ) status: active lo0: flags=8049 metric 0 mtu 16384 inet 127.0.0.1 netmask 0xff000000 I am going to try a few more tests with and without, firewall rules etc as well as an updated kernel to RELENG_7 as of today and see how that goes. ---Mike From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 20:12:51 2008 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0BDAF10656C3 for ; Mon, 30 Jun 2008 20:12:51 +0000 (UTC) (envelope-from 20080111.freebsd.org@ab.ote.we.lv) Received: from purple.the-7.net (purple.the-7.net [IPv6:2001:470:1f01:622:230:48ff:fe23:4c67]) by mx1.freebsd.org (Postfix) with ESMTP id EA8FA8FC16 for ; Mon, 30 Jun 2008 20:12:50 +0000 (UTC) (envelope-from 20080111.freebsd.org@ab.ote.we.lv) Received: from redion.astralblue.net (redion.astralblue.net [IPv6:2001:470:1f05:155:216:17ff:fed4:71f0]) by purple.the-7.net (8.14.2/8.14.2) with ESMTP id m5UKCeZ4026054 for ; Mon, 30 Jun 2008 13:12:40 -0700 (PDT) (envelope-from 20080111.freebsd.org@ab.ote.we.lv) Message-ID: <48693E39.4080104@ab.ote.we.lv> Date: Mon, 30 Jun 2008 13:12:41 -0700 From: "Eugene M. Kim" <20080111.freebsd.org@ab.ote.we.lv> User-Agent: Thunderbird 2.0.0.14 (X11/20080620) MIME-Version: 1.0 To: freebsd-net@FreeBSD.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=0.7 required=10.0 tests=FROM_STARTS_WITH_NUMS, NO_RELAYS autolearn=disabled version=3.2.3 X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on purple.the-7.net Cc: Subject: bridge(4) and IPv6 link-local address X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 20:12:51 -0000 Hello, A quick question: Is bridge(4) supposed /not/ to automatically configure an IPv6 link-local address? I'm trying to use it to bridge a wired segment and a wireless segment, and router advertisement over bridge0 wouldn't work because, with bridge0 lacking a LL address, the router uses a non-LL address as the source address for RA packets, which then is ignored as invalid by other IPv6 nodes. Cheers, Eugene From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 21:03:37 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B4F68106567B for ; Mon, 30 Jun 2008 21:03:37 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 799F18FC18 for ; Mon, 30 Jun 2008 21:03:37 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDQTq-0003I6-98; Mon, 30 Jun 2008 17:00:10 -0400 Message-ID: <48694A9D.1030001@gtcomm.net> Date: Mon, 30 Jun 2008 17:05:33 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Mike Tancsa References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> In-Reply-To: <200806301944.m5UJifJD081781@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 21:03:37 -0000 With hours and days of tweaking i can't even get 500k pps :/ no firewall no anything else.. What is your kernel config? Sysctl configs? My machine i'm testing on is dual opteron 2212 , with intel 2 port 82571 nic.. Using 7-STABLE and I tried 6-stable and -current I get the RTM_MISS with 7 and current but only with certain types of packets at a certain rate.. :/ I can not get more than 500kpps.. i tried everything I could think of... lowering the rx descriptors on EM to 512 instead of 2048 gave me some more.. I was stuck at 400kpps until i changed those and i lowered the rx processing limit. My tests are going incoming em0 and outgoing em1 in one direction only and it has major errors when em0 taskq gets close to 80% cpu.. I am pretty disappointed that it maxes out a little over 400kpps and even then it gets some errors here and there , mainly missed packets due to no buffer and rx overruns (dev.em.0.stats=1) Mike Tancsa wrote: > At 04:04 AM 6/29/2008, Paul wrote: >> This is just a question but who can get more than 400k pps forwarding >> performance ? > > > OK, I setup 2 boxes on either end of a RELENG_7 box from about May 7th > just now, to see with 2 boxes blasting across it how it would work. > *However*, this is with no firewall loaded and, I must enable ip fast > forwarding. Without that enabled, the box just falls over. > > even at 20Kpps, I start seeing all sorts of messages spewing to route > -n monitor > > > got message of size 96 on Mon Jun 30 15:39:10 2008 > RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > default > > I am starting to wonder if those messages are the results of corrupted > packets the machine just cant keep up with ? > > > CPU is > > CPU: Intel(R) Xeon(R) CPU 3070 @ 2.66GHz (2660.01-MHz > 686-class CPU) > > > input (Total) output > packets errs bytes packets errs bytes colls > 611945 0 77892098 611955 0 77013002 0 > 616727 0 78215508 616742 0 77303454 0 > 617066 0 78162130 617082 0 77238434 0 > 618238 0 78302314 618225 0 77377582 0 > 617035 0 78141000 617038 0 77215672 0 > 617625 0 78225600 617588 0 77301734 0 > 616190 0 78017320 616165 0 77091774 0 > 615583 0 78064130 615628 0 77152800 0 > 617662 0 78254388 617658 0 77332340 0 > 618000 0 78269912 617950 0 77344554 0 > 617248 0 78183136 617315 0 77259588 0 > 617325 0 78204566 617289 0 77282094 0 > 618391 0 78337734 618357 0 77413756 0 > 616025 0 78116070 616082 0 77203116 0 > > > To generate the packets, I am just using > /usr/src/tools/tools/netblast on 2 endpoints starting at about the > same time > > # ./netblast 10.10.1.2 500 100 40 > > start: 1214854131.083679919 > finish: 1214854171.084668592 > send calls: 20139141 > send errors: 0 > approx send rate: 503478 > approx error rate: 0 > > > # ./netblast 10.10.1.3 500 10 40 > > start: 1214854273.882202815 > finish: 1214854313.882319031 > send calls: 23354971 > send errors: 18757223 > approx send rate: 114943 > approx error rate: 0 > > The box in the middle doing the forwarding > > 1[spare-r7]# ifconfig -u > em0: flags=8843 metric 0 mtu 1500 > > options=19b > ether 00:1b:21:08:32:a8 > inet 10.20.1.1 netmask 0xffffff00 broadcast 10.20.1.255 > media: Ethernet autoselect (1000baseTX ) > status: active > em1: flags=8843 metric 0 mtu 1500 > options=9b > ether 00:1b:21:08:32:a9 > inet 192.168.43.193 netmask 0xffffff00 broadcast 192.168.43.255 > media: Ethernet autoselect (100baseTX ) > status: active > em3: flags=8843 metric 0 mtu 1500 > > options=19b > ether 00:30:48:90:4c:ff > inet 10.10.1.1 netmask 0xffffff00 broadcast 10.10.1.255 > media: Ethernet autoselect (1000baseTX ) > status: active > lo0: flags=8049 metric 0 mtu 16384 > inet 127.0.0.1 netmask 0xff000000 > > > I am going to try a few more tests with and without, firewall rules > etc as well as an updated kernel to RELENG_7 as of today and see how > that goes. > > ---Mike > > From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 22:15:08 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E901E1065676 for ; Mon, 30 Jun 2008 22:15:08 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.cksoft.de (mail.cksoft.de [62.111.66.27]) by mx1.freebsd.org (Postfix) with ESMTP id A2E9A8FC20 for ; Mon, 30 Jun 2008 22:15:08 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from localhost (amavis.str.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id 2972341C7AB; Tue, 1 Jul 2008 00:15:06 +0200 (CEST) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([62.111.66.27]) by localhost (amavis.str.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id y1cuYKo-yXk7; Tue, 1 Jul 2008 00:15:05 +0200 (CEST) Received: by mail.cksoft.de (Postfix, from userid 66) id C39D941C7B8; Tue, 1 Jul 2008 00:15:05 +0200 (CEST) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id 78FEC44487F; Mon, 30 Jun 2008 22:12:29 +0000 (UTC) Date: Mon, 30 Jun 2008 22:12:29 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: "Eugene M. Kim" <20080111.freebsd.org@ab.ote.we.lv> In-Reply-To: <48693E39.4080104@ab.ote.we.lv> Message-ID: <20080630220842.X83875@maildrop.int.zabbadoz.net> References: <48693E39.4080104@ab.ote.we.lv> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@FreeBSD.org Subject: Re: bridge(4) and IPv6 link-local address X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 22:15:09 -0000 On Mon, 30 Jun 2008, Eugene M. Kim wrote: Hi, > A quick question: Is bridge(4) supposed /not/ to automatically configure an > IPv6 link-local address? yes there is a check for this in the code and if remoed (tried that lately) more things go wrong. > I'm trying to use it to bridge a wired segment and a wireless segment, and > router advertisement over bridge0 wouldn't work because, with bridge0 lacking > a LL address, the router uses a non-LL address as the source address for RA > packets, which then is ignored as invalid by other IPv6 nodes. yes, seem something similar lately but ETIMEOUT on debugging. The problem basically was: lan bridge ath --- wlan client the LL address was on the "lan" interface. ping6 LL on lan from wlan client did not work. I could see the packets being bridged and visible on all interfaces and even the router on lan noticed them but there was no reply going to the client. ping6 from the bridge ``box'' to the wlan client and everything was fine as nd was seeded. Removing the check we ended up with the same LL address on both bridge and the lan interface if I can remember correctly and you do not want that... it's a bit tricky and there is something that does not work as expected, right. If you find the time to debug it I'll happily test patches;-) -- Bjoern A. Zeeb Stop bit received. Insert coin for new game. From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 22:18:25 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C907F1065671 for ; Mon, 30 Jun 2008 22:18:25 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: from ibctech.ca (v6.ibctech.ca [IPv6:2607:f118::b6]) by mx1.freebsd.org (Postfix) with SMTP id 4D2A78FC12 for ; Mon, 30 Jun 2008 22:18:25 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: (qmail 47472 invoked by uid 89); 30 Jun 2008 22:20:00 -0000 Received: from unknown (HELO ?IPv6:2607:f118::5?) (steve@ibctech.ca@2607:f118::5) by 2607:f118::b6 with ESMTPA; 30 Jun 2008 22:19:50 -0000 Message-ID: <48695BA6.7060207@ibctech.ca> Date: Mon, 30 Jun 2008 18:18:14 -0400 From: Steve Bertrand User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Mike Tancsa References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> In-Reply-To: <200806301944.m5UJifJD081781@lava.sentex.ca> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 22:18:25 -0000 Mike Tancsa wrote: > At 04:04 AM 6/29/2008, Paul wrote: >> This is just a question but who can get more than 400k pps forwarding >> performance ? > > > OK, I setup 2 boxes on either end of a RELENG_7 box from about May 7th > just now, to see with 2 boxes blasting across it how it would work. > *However*, this is with no firewall loaded and, I must enable ip fast > forwarding. Without that enabled, the box just falls over. > > even at 20Kpps, I start seeing all sorts of messages spewing to route -n > monitor > > > got message of size 96 on Mon Jun 30 15:39:10 2008 > RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 0, > flags: > locks: inits: > sockaddrs: > default Mike, Is the monitor running on the 7.0 box in the middle you are testing? I set up the same configuration, and even with almost no load (< 1Kpps) can replicate these error messages by making the remote IP address (in your case 'default', disappear (ie: unplug the cable, DDoS etc). ...to further, I can even replicate the problem at a single packet per second by trying to ping an IP address that I know for fact that the router can not get to. Do you see these error messages if you set up a loopback address with an IP on the router, and effectively chop your test environment in half? In your case, can the router in the middle actually get to a default gateway for external addresses (when I perform the test, your 'default' is substituted with the IP I am trying to reach, so I am only assuming that 'default' is implying default gateway). Steve From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 23:03:04 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E36EA106564A for ; Mon, 30 Jun 2008 23:03:04 +0000 (UTC) (envelope-from thompsa@FreeBSD.org) Received: from pele.citylink.co.nz (pele.citylink.co.nz [202.8.44.226]) by mx1.freebsd.org (Postfix) with ESMTP id A103E8FC15 for ; Mon, 30 Jun 2008 23:03:04 +0000 (UTC) (envelope-from thompsa@FreeBSD.org) Received: from localhost (localhost [127.0.0.1]) by pele.citylink.co.nz (Postfix) with ESMTP id 437D82BE2E; Tue, 1 Jul 2008 11:03:03 +1200 (NZST) X-Virus-Scanned: Debian amavisd-new at citylink.co.nz Received: from pele.citylink.co.nz ([127.0.0.1]) by localhost (pele.citylink.co.nz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zTuJ65JG3MIi; Tue, 1 Jul 2008 11:02:59 +1200 (NZST) Received: from citylink.fud.org.nz (unknown [202.8.44.45]) by pele.citylink.co.nz (Postfix) with ESMTP; Tue, 1 Jul 2008 11:02:59 +1200 (NZST) Received: by citylink.fud.org.nz (Postfix, from userid 1001) id F285C11430; Tue, 1 Jul 2008 11:04:26 +1200 (NZST) Date: Mon, 30 Jun 2008 16:04:26 -0700 From: Andrew Thompson To: Pyun YongHyeon Message-ID: <20080630230426.GI60479@citylink.fud.org.nz> References: <4868A34C.6030304@moneybookers.com> <20080630101629.GD79537@cdnetworks.co.kr> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080630101629.GD79537@cdnetworks.co.kr> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: freebsd-net@freebsd.org, Stefan Lambrev Subject: Re: if_bridge turns off checksum offload of members? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 23:03:05 -0000 On Mon, Jun 30, 2008 at 07:16:29PM +0900, Pyun YongHyeon wrote: > On Mon, Jun 30, 2008 at 12:11:40PM +0300, Stefan Lambrev wrote: > > Greetings, > > > > I just noticed, that when I add em network card to bridge the checksum > > offload is turned off. > > I even put in my rc.conf: > > ifconfig_em0="rxcsum up" > > ifconfig_em1="rxcsum up" > > but after reboot both em0 and em1 have this feature disabled. > > > > Is this expected behavior? Should I care about csum in bridge mode? > > I noticed that enabling checksum offload manually improve things little btw. > > > > AFAIK this is intended one, bridge(4) turns off Tx side checksum > offload by default. I think disabling Tx checksum offload is > required as not all members of a bridge may be able to do checksum > offload. The same is true for TSO but it seems that bridge(4) > doesn't disable it. It should be added. > If all members of bridge have the same hardware capability I think > bridge(4) may not need to disable Tx side hardware assistance. I > guess bridge(4) can scan every interface capabilities in a member > and can decide what hardware assistance can be activated instead of > blindly turning off Tx side hardware assistance. Yes, it could me smarter in this regard. The other thing to note is that unless you are manipulating the packets as they pass the bridge the checksums are unaffected and do not get recalculated anyway. Andrew From owner-freebsd-net@FreeBSD.ORG Mon Jun 30 23:10:23 2008 Return-Path: Delivered-To: net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AC5431065670; Mon, 30 Jun 2008 23:10:23 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from server.baldwin.cx (bigknife-pt.tunnel.tserv9.chi1.ipv6.he.net [IPv6:2001:470:1f10:75::2]) by mx1.freebsd.org (Postfix) with ESMTP id 1143B8FC1E; Mon, 30 Jun 2008 23:10:22 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from localhost.corp.yahoo.com (john@localhost [IPv6:::1]) (authenticated bits=0) by server.baldwin.cx (8.14.2/8.14.2) with ESMTP id m5UNA9vo022818; Mon, 30 Jun 2008 19:10:16 -0400 (EDT) (envelope-from jhb@freebsd.org) From: John Baldwin To: freebsd-arch@freebsd.org Date: Mon, 30 Jun 2008 15:17:21 -0400 User-Agent: KMail/1.9.7 References: <20080524111715.T64552@fledge.watson.org> <20080629180126.F90836@fledge.watson.org> In-Reply-To: <20080629180126.F90836@fledge.watson.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200806301517.22173.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH authentication, not delayed by milter-greylist-2.0.2 (server.baldwin.cx [IPv6:::1]); Mon, 30 Jun 2008 19:10:16 -0400 (EDT) X-Virus-Scanned: ClamAV 0.93.1/7596/Mon Jun 30 18:21:48 2008 on server.baldwin.cx X-Virus-Status: Clean X-Spam-Status: No, score=-2.3 required=4.2 tests=AWL,BAYES_00, DATE_IN_PAST_03_06,NO_RELAYS autolearn=ham version=3.1.3 X-Spam-Checker-Version: SpamAssassin 3.1.3 (2006-06-01) on server.baldwin.cx Cc: arch@freebsd.org, Robert Watson , current@freebsd.org, net@freebsd.org Subject: Re: HEAD UP: non-MPSAFE network drivers to be disabled (was: 8.0 network stack MPsafety goals (fwd)) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Jun 2008 23:10:23 -0000 On Sunday 29 June 2008 01:02:49 pm Robert Watson wrote: > > On Sat, 24 May 2008, Robert Watson wrote: > > > Just as a reminder, we've just about reached the one month date before > > IFF_NEEDSGIANT drivers are disabled in the build. You can find a > > description of the general problem and list of specific drivers below. > > > > As USB work is on-going, I will *not* disable the USB drivers at this time, > > but all other drivers in the list below will be disabled on 26 June. They > > will remain in the tree, easily accessible for patch distribution and > > re-enabling, until October, when any remaining non-MPSAFE drivers will be > > deleted in 8.x. FreeBSD 8.0 will not ship with compatibility shims to > > support non-MPSAFE network device drivers. > > An FYI on the state of things here: in the last month, John has updated a > number of device drivers to be MPSAFE, and the USB work remains in-flight. > I'm holding fire a bit on disabling IFF_NEEDSGIANT while things settle and I > catch up on driver state, and will likely send out an update next week > regarding which device drivers remain on the kill list, and generally what the > status of this project is. Pending someone testing them real soon, I will be axeing at least sbsh, sbni, snc, and cnw this week (I had someone e-mail who said they would test the arl(4) pages). I still want to work on if_plip (which means locking ppbus) and if_ic (iicbus). I started working on locking iicbus recently FWIW. Aside from USB that leaves ray(4) (which does tsleep() all over the place including in interrupt handlers.. this couldn't have worked on 4.x all that well) and the various Cronyx drivers which interface with netgraph. Someone who knows netgraph can probably step up to lock those. Could also ask rik@ directly about the Cronyx drivers (cx, ctau, and cp). -- John Baldwin From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 00:05:55 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B93391065680 for ; Tue, 1 Jul 2008 00:05:55 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 7C03E8FC20 for ; Tue, 1 Jul 2008 00:05:55 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDTKC-0004a8-Gs; Mon, 30 Jun 2008 20:02:24 -0400 Message-ID: <4869755A.1020903@gtcomm.net> Date: Mon, 30 Jun 2008 20:07:54 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Steve Bertrand References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <48695BA6.7060207@ibctech.ca> In-Reply-To: <48695BA6.7060207@ibctech.ca> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Mike Tancsa Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 00:05:55 -0000 I am getting this message with normal routing. say... em0 10.1.1.1/24 em1 10.2.2.1/24 using a box 10.1.1.2 on em0 and having another box on 10.2.2.2 on em1 I send packet from 10.1.1.2 which goes through em0 and has a route to 10.2.2.2 out em1 of course and I get MASSIVE RTM_MISS messages but ONLY with this certain packets.. I don't get it? I posted the tcpdump of the types of packets that generate them and the ones that don't. RTM_MISS is normal if the box can't get to a route, it's the 'destination unreachable' message. I would prefer a kernel option to disable this message to save CPU cycles though as it is completely unnecessary to generate. I even set the default gateway to loopback interface and I STILL get the message.. Something is wrong in the code somewhere. Does anyone have any idea how to disable this message? It's causing major cpu usage on my zebra daemon which is watching the route messages and most likely severely limiting pps throughput :/ It generates the messages with only ip on em1 and em0 with nothing else in the routing table and a default gateway set. So it has nothing to do with zebra. It happens in 7-STABLE and (8) -CURRENT, I tested both. There are no RTM_MISS message in 7-RELEASE so someone changed something to -STABLE :/ Paul Steve Bertrand wrote: > Mike Tancsa wrote: >> At 04:04 AM 6/29/2008, Paul wrote: >>> This is just a question but who can get more than 400k pps >>> forwarding performance ? >> >> >> OK, I setup 2 boxes on either end of a RELENG_7 box from about May >> 7th just now, to see with 2 boxes blasting across it how it would >> work. *However*, this is with no firewall loaded and, I must enable >> ip fast forwarding. Without that enabled, the box just falls over. >> >> even at 20Kpps, I start seeing all sorts of messages spewing to route >> -n monitor >> >> >> got message of size 96 on Mon Jun 30 15:39:10 2008 >> RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno >> 0, flags: >> locks: inits: >> sockaddrs: >> default > > Mike, > > Is the monitor running on the 7.0 box in the middle you are testing? > > I set up the same configuration, and even with almost no load (< > 1Kpps) can replicate these error messages by making the remote IP > address (in your case 'default', disappear (ie: unplug the cable, DDoS > etc). > > ...to further, I can even replicate the problem at a single packet per > second by trying to ping an IP address that I know for fact that the > router can not get to. > > Do you see these error messages if you set up a loopback address with > an IP on the router, and effectively chop your test environment in > half? In your case, can the router in the middle actually get to a > default gateway for external addresses (when I perform the test, your > 'default' is substituted with the IP I am trying to reach, so I am > only assuming that 'default' is implying default gateway). > > Steve > From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 00:29:11 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 73AC7106567A for ; Tue, 1 Jul 2008 00:29:11 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.228]) by mx1.freebsd.org (Postfix) with ESMTP id 380388FC1A for ; Tue, 1 Jul 2008 00:29:11 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so1875557rvf.43 for ; Mon, 30 Jun 2008 17:29:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:received:date:from :to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=CHP88dbAgDv3VBpbYTjqXE3zk90ENoydFnXqYguuh/0=; b=TPyxuL4AKM9zmy9Lkn56tPS46Y9jNHrLCzcndHxgs+gj0muJ1p50YIauyquD0R0Rcl rh3dwQPIT7VHJRgldqsDnJrTiEyPh4JmIDhUYD9aNR98zP2G4usWAO1zeUYcv5Humzkv cS7DOvkXaicDfGQvuhVuy3CBucLMBuLdnqWP0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=J22G4BkqJWnHDlI4Lh5+VY2H58u+649wwM47SWAwQ97Rb/W1Sgno8pYHlYzJthKpKE dnT+M4e4xEPaw+j1jIHlhBeIuIz7qzCPnirZ2GH5pvP495DAO4YfKbhhbOs0TU2ded4F RJGy97f3PBU/j5CR3G0gwWSsvYWJfqW90fsLc= Received: by 10.141.51.15 with SMTP id d15mr3082729rvk.106.1214872150856; Mon, 30 Jun 2008 17:29:10 -0700 (PDT) Received: from michelle.cdnetworks.co.kr ( [211.53.35.84]) by mx.google.com with ESMTPS id l31sm8714988rvb.6.2008.06.30.17.29.07 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 30 Jun 2008 17:29:09 -0700 (PDT) Received: from michelle.cdnetworks.co.kr (localhost.cdnetworks.co.kr [127.0.0.1]) by michelle.cdnetworks.co.kr (8.13.5/8.13.5) with ESMTP id m610R0Pb083839 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 1 Jul 2008 09:27:00 +0900 (KST) (envelope-from pyunyh@gmail.com) Received: (from yongari@localhost) by michelle.cdnetworks.co.kr (8.13.5/8.13.5/Submit) id m610QxM3083838; Tue, 1 Jul 2008 09:26:59 +0900 (KST) (envelope-from pyunyh@gmail.com) Date: Tue, 1 Jul 2008 09:26:59 +0900 From: Pyun YongHyeon To: "Eugene M. Kim" <20080111.freebsd.org@ab.ote.we.lv> Message-ID: <20080701002659.GC83626@cdnetworks.co.kr> References: <200806270345.m5R3j1BT036253@freefall.freebsd.org> <48649776.9040509@ab.ote.we.lv> <20080627074948.GC67753@cdnetworks.co.kr> <4864A217.3040309@ab.ote.we.lv> <20080629072932.GA76469@cdnetworks.co.kr> <48693193.1020404@ab.ote.we.lv> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <48693193.1020404@ab.ote.we.lv> User-Agent: Mutt/1.4.2.1i Cc: freebsd-net@FreeBSD.org, yongari@FreeBSD.org Subject: Re: kern/125024: vr(4) does not see incoming multicast packets in non-promiscuous mode (broadcast is fine); breaks IPv6 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 00:29:11 -0000 On Mon, Jun 30, 2008 at 12:18:43PM -0700, Eugene M. Kim wrote: > Than you! The new patch fixed the problem. I'll put it under test for > a few more days and let you know if any regression is seen. > Cool, thanks for testing! > Cheers, > Eugene > > Pyun YongHyeon wrote: > >On Fri, Jun 27, 2008 at 01:17:27AM -0700, Eugene M. Kim wrote: > > > Pyun YongHyeon wrote: > > > >I've updated patch again. There was a bug that disabled > > > >multicasting filter. Back out previous patch and try again. > > > >The URL is the same as before. > > > > > > > > > Regards, > > > > > Eugene > > > > > > > > > > rtsol still doesn't work with vr0 being in non-promiscuous mode. > > > However, apparently vr0 picked up router solicitations during system > > > boot-up, as ifconfig shows the correct prefixes autoconfigured. It > > > seems something goes wrong in between. 'o 'a > > > > > > >Oops, I was accessing CAM mask register as 8bit register which > >should be 32bit! Try the patch at the following URL. > > > >http://people.freebsd.org/~yongari/vr/vr.cam.patch2 > > > > > Eugene > > > -- Regards, Pyun YongHyeon From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 00:35:57 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3D0D61065678 for ; Tue, 1 Jul 2008 00:35:57 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 809438FC12 for ; Tue, 1 Jul 2008 00:35:55 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 25745 invoked from network); 1 Jul 2008 02:35:49 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 1 Jul 2008 02:35:49 +0200 Date: Tue, 1 Jul 2008 02:35:48 +0200 (CEST) From: Ingo Flaschberger To: Paul In-Reply-To: <4869755A.1020903@gtcomm.net> Message-ID: References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <48695BA6.7060207@ibctech.ca> <4869755A.1020903@gtcomm.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net , Steve Bertrand , Mike Tancsa Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 00:35:57 -0000 Dear Paul, > I am getting this message with normal routing. > > say... > > em0 10.1.1.1/24 > > em1 10.2.2.1/24 > > using a box 10.1.1.2 on em0 > and having another box on 10.2.2.2 on em1 > > I send packet from 10.1.1.2 which goes through em0 and has a route to > 10.2.2.2 out em1 of course and I get MASSIVE RTM_MISS messages but ONLY with > this certain packets.. I don't get it? I posted the tcpdump of the types of There is a open bug report: http://www.freebsd.org/cgi/query-pr.cgi?pr=124540 perhaps it has something todo with the multiple fip-stuff? kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 00:54:53 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3CEED106566C for ; Tue, 1 Jul 2008 00:54:53 +0000 (UTC) (envelope-from wilkinsa@obelix.dsto.defence.gov.au) Received: from digger1.defence.gov.au (digger1.defence.gov.au [203.5.217.4]) by mx1.freebsd.org (Postfix) with ESMTP id 9585B8FC25 for ; Tue, 1 Jul 2008 00:54:52 +0000 (UTC) (envelope-from wilkinsa@obelix.dsto.defence.gov.au) Received: from ednmsw510.dsto.defence.gov.au (ednmsw510.dsto.defence.gov.au [131.185.68.11]) by digger1.defence.gov.au (DSTO/DSTO) with ESMTP id m610hhOl016079 for ; Tue, 1 Jul 2008 10:13:43 +0930 (CST) Received: from fmbex510.dsto.defence.gov.au (fmbex510.dsto.defence.gov.au) by ednmsw510.dsto.defence.gov.au (Clearswift SMTPRS 5.2.9) with ESMTP id for ; Tue, 1 Jul 2008 10:14:13 +0930 Received: from stlex510.dsto.defence.gov.au ([203.6.60.184]) by fmbex510.dsto.defence.gov.au with Microsoft SMTPSVC(6.0.3790.1830); Tue, 1 Jul 2008 10:44:13 +1000 Received: from obelix.dsto.defence.gov.au ([203.6.60.208]) by stlex510.dsto.defence.gov.au with Microsoft SMTPSVC(6.0.3790.1830); Tue, 1 Jul 2008 08:44:12 +0800 Received: from obelix.dsto.defence.gov.au (localhost [127.0.0.1]) by obelix.dsto.defence.gov.au (8.14.2/8.14.2) with ESMTP id m610hk4A007398 for ; Tue, 1 Jul 2008 08:43:46 +0800 (WST) (envelope-from wilkinsa@obelix.dsto.defence.gov.au) Received: (from wilkinsa@localhost) by obelix.dsto.defence.gov.au (8.14.2/8.14.2/Submit) id m610hkcc007397 for freebsd-net@freebsd.org; Tue, 1 Jul 2008 08:43:46 +0800 (WST) (envelope-from wilkinsa) Date: Tue, 1 Jul 2008 08:43:46 +0800 From: "Wilkinson, Alex" To: freebsd-net@freebsd.org Message-ID: <20080701004346.GA3898@stlux503.dsto.defence.gov.au> Mail-Followup-To: freebsd-net@freebsd.org References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <200806301944.m5UJifJD081781@lava.sentex.ca> Organisation: Defence Science Technology Organisation User-Agent: Mutt/1.5.18 (2008-05-17) X-OriginalArrivalTime: 01 Jul 2008 00:44:12.0697 (UTC) FILETIME=[94C6B090:01C8DB13] X-TM-AS-Product-Ver: SMEX-7.0.0.1526-5.5.1027-16002.007 X-TM-AS-Result: No--0.066000-0.000000-31 Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 00:54:53 -0000 0n Mon, Jun 30, 2008 at 03:44:48PM -0400, Mike Tancsa wrote: >OK, I setup 2 boxes on either end of a RELENG_7 box from about May >7th just now, to see with 2 boxes blasting across it how it would >work. *However*, this is with no firewall loaded and, I must enable >ip fast forwarding. Without that enabled, the box just falls over. What is "ip fast forwarding" ? -aW IMPORTANT: This email remains the property of the Australian Defence Organisation and is subject to the jurisdiction of section 70 of the CRIMES ACT 1914. If you have received this email in error, you are requested to contact the sender and delete the email. From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 01:00:33 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E7DC1106567C for ; Tue, 1 Jul 2008 01:00:33 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 1A5778FC0A for ; Tue, 1 Jul 2008 01:00:32 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 3649 invoked from network); 1 Jul 2008 03:00:31 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 1 Jul 2008 03:00:31 +0200 Date: Tue, 1 Jul 2008 03:00:31 +0200 (CEST) From: Ingo Flaschberger To: "Wilkinson, Alex" In-Reply-To: <20080701004346.GA3898@stlux503.dsto.defence.gov.au> Message-ID: References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 01:00:34 -0000 Dear Alex, > >OK, I setup 2 boxes on either end of a RELENG_7 box from about May > >7th just now, to see with 2 boxes blasting across it how it would > >work. *However*, this is with no firewall loaded and, I must enable > >ip fast forwarding. Without that enabled, the box just falls over. > > What is "ip fast forwarding" ? instead of copying the while ip packet into system memory, only the ip header is copyied and then in a "fast" path determined if it could be fast forwarded. if possible, a ned header is created at the other network-cards-buffer and the ip-data is copied from network-card-buffer to network-card-buffer directly. Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 01:07:48 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E58FD1065778 for ; Tue, 1 Jul 2008 01:07:48 +0000 (UTC) (envelope-from wilkinsa@obelix.dsto.defence.gov.au) Received: from digger1.defence.gov.au (digger1.defence.gov.au [203.5.217.4]) by mx1.freebsd.org (Postfix) with ESMTP id C1AD48FC1E for ; Tue, 1 Jul 2008 01:07:45 +0000 (UTC) (envelope-from wilkinsa@obelix.dsto.defence.gov.au) Received: from ednmsw510.dsto.defence.gov.au (ednmsw510.dsto.defence.gov.au [131.185.68.11]) by digger1.defence.gov.au (DSTO/DSTO) with ESMTP id m6117DmO021690 for ; Tue, 1 Jul 2008 10:37:13 +0930 (CST) Received: from fmbex510.dsto.defence.gov.au (fmbex510.dsto.defence.gov.au) by ednmsw510.dsto.defence.gov.au (Clearswift SMTPRS 5.2.9) with ESMTP id for ; Tue, 1 Jul 2008 10:37:44 +0930 Received: from stlex510.dsto.defence.gov.au ([203.6.60.184]) by fmbex510.dsto.defence.gov.au with Microsoft SMTPSVC(6.0.3790.1830); Tue, 1 Jul 2008 11:07:44 +1000 Received: from obelix.dsto.defence.gov.au ([203.6.60.208]) by stlex510.dsto.defence.gov.au with Microsoft SMTPSVC(6.0.3790.1830); Tue, 1 Jul 2008 09:07:43 +0800 Received: from obelix.dsto.defence.gov.au (localhost [127.0.0.1]) by obelix.dsto.defence.gov.au (8.14.2/8.14.2) with ESMTP id m6117HM4007592 for ; Tue, 1 Jul 2008 09:07:17 +0800 (WST) (envelope-from wilkinsa@obelix.dsto.defence.gov.au) Received: (from wilkinsa@localhost) by obelix.dsto.defence.gov.au (8.14.2/8.14.2/Submit) id m6117HLf007591 for freebsd-net@freebsd.org; Tue, 1 Jul 2008 09:07:17 +0800 (WST) (envelope-from wilkinsa) Date: Tue, 1 Jul 2008 09:07:17 +0800 From: "Wilkinson, Alex" To: freebsd-net@freebsd.org Message-ID: <20080701010716.GF3898@stlux503.dsto.defence.gov.au> Mail-Followup-To: freebsd-net@freebsd.org References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: Organisation: Defence Science Technology Organisation User-Agent: Mutt/1.5.18 (2008-05-17) X-OriginalArrivalTime: 01 Jul 2008 01:07:43.0407 (UTC) FILETIME=[DD9FF3F0:01C8DB16] X-TM-AS-Product-Ver: SMEX-7.0.0.1526-5.5.1027-16002.007 X-TM-AS-Result: No--6.981000-0.000000-31 Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 01:07:49 -0000 0n Tue, Jul 01, 2008 at 03:00:31AM +0200, Ingo Flaschberger wrote: >Dear Alex, > >> >OK, I setup 2 boxes on either end of a RELENG_7 box from about May >> >7th just now, to see with 2 boxes blasting across it how it would >> >work. *However*, this is with no firewall loaded and, I must enable >> >ip fast forwarding. Without that enabled, the box just falls over. >> >> What is "ip fast forwarding" ? > >instead of copying the while ip packet into system memory, only the ip >header is copyied and then in a "fast" path determined if it could be fast >forwarded. >if possible, a ned header is created at the other network-cards-buffer >and the ip-data is copied from network-card-buffer to network-card-buffer >directly. So how does one enable "ip fast forwarding" on FreeBSD ? -aW IMPORTANT: This email remains the property of the Australian Defence Organisation and is subject to the jurisdiction of section 70 of the CRIMES ACT 1914. If you have received this email in error, you are requested to contact the sender and delete the email. From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 01:10:53 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C20C41065686 for ; Tue, 1 Jul 2008 01:10:53 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 123DF8FC22 for ; Tue, 1 Jul 2008 01:10:52 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 9514 invoked from network); 1 Jul 2008 03:10:51 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 1 Jul 2008 03:10:51 +0200 Date: Tue, 1 Jul 2008 03:10:51 +0200 (CEST) From: Ingo Flaschberger To: "Wilkinson, Alex" In-Reply-To: <20080701010716.GF3898@stlux503.dsto.defence.gov.au> Message-ID: References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 01:10:53 -0000 Dear Alex, > >if possible, a ned header is created at the other network-cards-buffer > >and the ip-data is copied from network-card-buffer to network-card-buffer > >directly. > > So how does one enable "ip fast forwarding" on FreeBSD ? sysctl -w net.inet.ip.fastforwarding=1 usually interface polling is also chosen to prevent "lock-ups". man polling kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 01:23:05 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B038D1065670 for ; Tue, 1 Jul 2008 01:23:05 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from ape.monkeybrains.net (ape.monkeybrains.net [208.69.40.11]) by mx1.freebsd.org (Postfix) with ESMTP id 934A88FC0A for ; Tue, 1 Jul 2008 01:23:05 +0000 (UTC) (envelope-from crapsh@monkeybrains.net) Received: from [192.168.0.101] (busymonkey.monkeybrains.net [66.92.187.117]) (authenticated bits=0) by ape.monkeybrains.net (8.14.1/8.14.1) with ESMTP id m611MiAb042049 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 30 Jun 2008 18:22:45 -0700 (PDT) (envelope-from crapsh@monkeybrains.net) Message-ID: <486986D9.3000607@monkeybrains.net> Date: Mon, 30 Jun 2008 18:22:33 -0700 From: "Support (Rudy)" User-Agent: Thunderbird 1.5.0.14ubu (X11/20080306) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93.1, clamav-milter version 0.93.1 on pita.monkeybrains.net X-Virus-Status: Clean Cc: freebsd-net@freebsd.org, "Wilkinson, Alex" Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 01:23:05 -0000 Ingo Flaschberger wrote: > usually interface polling is also chosen to prevent "lock-ups". > man polling I used polling in FreeBSD 5.x and it helped a bunch. I set up a new router with 7.0 and MSI was recommended to me. (I noticed no difference when moving from polling -> MSI, however, on 5.4 polling seemed to help a lot. What are people using in 7.0? polling or MSI? Rudy From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 01:24:10 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 53730106567F for ; Tue, 1 Jul 2008 01:24:10 +0000 (UTC) (envelope-from thompsa@FreeBSD.org) Received: from pele.citylink.co.nz (pele.citylink.co.nz [202.8.44.226]) by mx1.freebsd.org (Postfix) with ESMTP id C97568FC1A for ; Tue, 1 Jul 2008 01:24:09 +0000 (UTC) (envelope-from thompsa@FreeBSD.org) Received: from localhost (localhost [127.0.0.1]) by pele.citylink.co.nz (Postfix) with ESMTP id 983CA2BE31; Tue, 1 Jul 2008 13:24:08 +1200 (NZST) X-Virus-Scanned: Debian amavisd-new at citylink.co.nz Received: from pele.citylink.co.nz ([127.0.0.1]) by localhost (pele.citylink.co.nz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ZaWKx4DGUtRG; Tue, 1 Jul 2008 13:24:04 +1200 (NZST) Received: from citylink.fud.org.nz (unknown [202.8.44.45]) by pele.citylink.co.nz (Postfix) with ESMTP; Tue, 1 Jul 2008 13:24:04 +1200 (NZST) Received: by citylink.fud.org.nz (Postfix, from userid 1001) id A60DC1142C; Tue, 1 Jul 2008 13:25:31 +1200 (NZST) Date: Mon, 30 Jun 2008 18:25:31 -0700 From: Andrew Thompson To: Pyun YongHyeon Message-ID: <20080701012531.GA92392@citylink.fud.org.nz> References: <4868A34C.6030304@moneybookers.com> <20080630101629.GD79537@cdnetworks.co.kr> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="mYCpIKhGyMATD0i+" Content-Disposition: inline In-Reply-To: <20080630101629.GD79537@cdnetworks.co.kr> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: freebsd-net@freebsd.org, Stefan Lambrev Subject: Re: if_bridge turns off checksum offload of members? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 01:24:10 -0000 --mYCpIKhGyMATD0i+ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Mon, Jun 30, 2008 at 07:16:29PM +0900, Pyun YongHyeon wrote: > On Mon, Jun 30, 2008 at 12:11:40PM +0300, Stefan Lambrev wrote: > > Greetings, > > > > I just noticed, that when I add em network card to bridge the checksum > > offload is turned off. > > I even put in my rc.conf: > > ifconfig_em0="rxcsum up" > > ifconfig_em1="rxcsum up" > > but after reboot both em0 and em1 have this feature disabled. > > > > Is this expected behavior? Should I care about csum in bridge mode? > > I noticed that enabling checksum offload manually improve things little btw. > > > > AFAIK this is intended one, bridge(4) turns off Tx side checksum > offload by default. I think disabling Tx checksum offload is > required as not all members of a bridge may be able to do checksum > offload. The same is true for TSO but it seems that bridge(4) > doesn't disable it. > If all members of bridge have the same hardware capability I think > bridge(4) may not need to disable Tx side hardware assistance. I > guess bridge(4) can scan every interface capabilities in a member > and can decide what hardware assistance can be activated instead of > blindly turning off Tx side hardware assistance. This patch should do that, are you able to test it Stefan? cheers, Andrew --mYCpIKhGyMATD0i+ Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="bridge_caps.diff" === if_bridge.c ================================================================== --- if_bridge.c (revision 180136) +++ if_bridge.c (local) @@ -163,9 +163,10 @@ #endif /* - * List of capabilities to mask on the member interface. + * List of capabilities to possibly mask on the member interface. */ -#define BRIDGE_IFCAPS_MASK IFCAP_TXCSUM +#define BRIDGE_IFCAPS_MASK (IFCAP_TOE|IFCAP_TSO|IFCAP_TXCSUM) +#define BRIDGE_IFCAPS_STRIP IFCAP_LRO /* * Bridge interface list entry. @@ -175,7 +176,7 @@ struct ifnet *bif_ifp; /* member if */ struct bstp_port bif_stp; /* STP state */ uint32_t bif_flags; /* member if flags */ - int bif_mutecap; /* member muted caps */ + int bif_savedcaps; /* saved capabilities */ uint32_t bif_addrmax; /* max # of addresses */ uint32_t bif_addrcnt; /* cur. # of addresses */ uint32_t bif_addrexceeded;/* # of address violations */ @@ -229,7 +230,9 @@ static void bridge_clone_destroy(struct ifnet *); static int bridge_ioctl(struct ifnet *, u_long, caddr_t); -static void bridge_mutecaps(struct bridge_iflist *, int); +static void bridge_capabilities(struct bridge_softc *); +static void bridge_set_ifcap(struct bridge_softc *, struct bridge_iflist *, + int); static void bridge_ifdetach(void *arg __unused, struct ifnet *); static void bridge_init(void *); static void bridge_dummynet(struct mbuf *, struct ifnet *); @@ -771,37 +774,52 @@ } /* - * bridge_mutecaps: + * bridge_capabilities: * * Clear or restore unwanted capabilities on the member interface */ static void -bridge_mutecaps(struct bridge_iflist *bif, int mute) +bridge_capabilities(struct bridge_softc *sc) { + struct bridge_iflist *bif; + int enabled, mask; + + mask = BRIDGE_IFCAPS_MASK; + + /* Get the set of capabilities supported */ + LIST_FOREACH(bif, &sc->sc_iflist, bif_next) { + mask &= bif->bif_ifp->if_capenable; + } + + /* Update all member interfaces to the new caps set */ + LIST_FOREACH(bif, &sc->sc_iflist, bif_next) { + enabled = bif->bif_ifp->if_capenable; + /* strip off bits and enable them again if allowed */ + enabled &= ~(BRIDGE_IFCAPS_STRIP|BRIDGE_IFCAPS_MASK); + enabled |= mask; + bridge_set_ifcap(sc, bif, enabled); + } + +} + +static void +bridge_set_ifcap(struct bridge_softc *sc, struct bridge_iflist *bif, int set) +{ struct ifnet *ifp = bif->bif_ifp; struct ifreq ifr; int error; - if (ifp->if_ioctl == NULL) - return; - bzero(&ifr, sizeof(ifr)); - ifr.ifr_reqcap = ifp->if_capenable; + ifr.ifr_reqcap = set; - if (mute) { - /* mask off and save capabilities */ - bif->bif_mutecap = ifr.ifr_reqcap & BRIDGE_IFCAPS_MASK; - if (bif->bif_mutecap != 0) - ifr.ifr_reqcap &= ~BRIDGE_IFCAPS_MASK; - } else - /* restore muted capabilities */ - ifr.ifr_reqcap |= bif->bif_mutecap; - - - if (bif->bif_mutecap != 0) { + if (ifp->if_capenable != set) { IFF_LOCKGIANT(ifp); - error = (*ifp->if_ioctl)(ifp, SIOCSIFCAP, (caddr_t)&ifr); + error = ifp->if_ioctl(ifp, SIOCSIFCAP, (caddr_t)&ifr); IFF_UNLOCKGIANT(ifp); + if (error) + if_printf(sc->sc_ifp, + "error setting interface capabilities on %s\n", + ifp->if_xname); } } @@ -868,7 +886,6 @@ * Take the interface out of promiscuous mode. */ (void) ifpromisc(ifs, 0); - bridge_mutecaps(bif, 0); break; case IFT_GIF: @@ -880,6 +897,8 @@ #endif break; } + /* reneable any interface capabilities */ + bridge_set_ifcap(sc, bif, bif->bif_savedcaps); } if (bif->bif_flags & IFBIF_STP) @@ -928,6 +947,8 @@ ifs = ifunit(req->ifbr_ifsname); if (ifs == NULL) return (ENOENT); + if (ifs->if_ioctl == NULL) /* must be supported */ + return (EINVAL); /* If it's in the span list, it can't be a member. */ LIST_FOREACH(bif, &sc->sc_spanlist, bif_next) @@ -957,6 +978,7 @@ bif->bif_ifp = ifs; bif->bif_flags = IFBIF_LEARNING | IFBIF_DISCOVER; + bif->bif_savedcaps = ifs->if_capabilities; switch (ifs->if_type) { case IFT_ETHER: @@ -967,8 +989,6 @@ error = ifpromisc(ifs, 1); if (error) goto out; - - bridge_mutecaps(bif, 1); break; case IFT_GIF: @@ -988,6 +1008,8 @@ */ LIST_INSERT_HEAD(&sc->sc_iflist, bif, bif_next); + /* Set interface capabilities to the intersection set of all members */ + bridge_capabilities(sc); out: if (error) { if (bif != NULL) --mYCpIKhGyMATD0i+-- From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 01:25:17 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 21F9E1065674 for ; Tue, 1 Jul 2008 01:25:17 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: from ibctech.ca (v6.ibctech.ca [IPv6:2607:f118::b6]) by mx1.freebsd.org (Postfix) with SMTP id 8C07F8FC12 for ; Tue, 1 Jul 2008 01:25:16 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: (qmail 56341 invoked by uid 89); 1 Jul 2008 01:26:52 -0000 Received: from unknown (HELO ?IPv6:2607:f118::5?) (steve@ibctech.ca@2607:f118::5) by 2607:f118::b6 with ESMTPA; 1 Jul 2008 01:26:52 -0000 Message-ID: <4869877C.20007@ibctech.ca> Date: Mon, 30 Jun 2008 21:25:16 -0400 From: Steve Bertrand User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> In-Reply-To: <20080701010716.GF3898@stlux503.dsto.defence.gov.au> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 01:25:17 -0000 Wilkinson, Alex wrote: > So how does one enable "ip fast forwarding" on FreeBSD ? Not to take anything away from Ingo's response, but to inform how to add the functionality to span across reboots, add the following line to /etc/sysctl.conf net.inet.ip.fastforwarding=1 Steve From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 01:27:41 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C12BD106564A for ; Tue, 1 Jul 2008 01:27:41 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: from ibctech.ca (v6.ibctech.ca [IPv6:2607:f118::b6]) by mx1.freebsd.org (Postfix) with SMTP id 31B578FC0A for ; Tue, 1 Jul 2008 01:27:41 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: (qmail 56367 invoked by uid 89); 1 Jul 2008 01:29:17 -0000 Received: from unknown (HELO ?IPv6:2607:f118::5?) (steve@ibctech.ca@2607:f118::5) by 2607:f118::b6 with ESMTPA; 1 Jul 2008 01:29:17 -0000 Message-ID: <4869880D.8040901@ibctech.ca> Date: Mon, 30 Jun 2008 21:27:41 -0400 From: Steve Bertrand User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: "Support (Rudy)" References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> In-Reply-To: <486986D9.3000607@monkeybrains.net> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, "Wilkinson, Alex" , Ingo Flaschberger Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 01:27:41 -0000 Support (Rudy) wrote: > Ingo Flaschberger wrote: >> usually interface polling is also chosen to prevent "lock-ups". >> man polling > > > I used polling in FreeBSD 5.x and it helped a bunch. I set up a new > router with 7.0 and MSI was recommended to me. (I noticed no difference > when moving from polling -> MSI, however, on 5.4 polling seemed to help > a lot. I'm curious now... how do you change individual device polling via sysctl? Steve From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 01:29:34 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49840106564A for ; Tue, 1 Jul 2008 01:29:34 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id 071CE8FC0C for ; Tue, 1 Jul 2008 01:29:33 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.2/8.14.2) with ESMTP id m611TVev042493; Mon, 30 Jun 2008 21:29:31 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m611TUV5083067 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 30 Jun 2008 21:29:31 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200807010129.m611TUV5083067@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Mon, 30 Jun 2008 21:29:38 -0400 To: Paul From: Mike Tancsa In-Reply-To: <48694A9D.1030001@gtcomm.net> References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <48694A9D.1030001@gtcomm.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 01:29:34 -0000 At 05:05 PM 6/30/2008, Paul wrote: >With hours and days of tweaking i can't even get 500k pps :/ no >firewall no anything else.. >What is your kernel config? Sysctl configs? The only thing that makes a difference is net.inet.ip.fastforwarding=1 >My machine i'm testing on is dual opteron 2212 , with intel 2 port >82571 nic.. xeon dual core on a supermicro MB. I am using one NIC on the MB and one on the dual port. em0@pci0:10:1:0: class=0x020000 card=0x11798086 chip=0x10798086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82546EB Dual Port Gigabit Ethernet Controller' class = network subclass = ethernet cap 01[dc] = powerspec 2 supports D0 D3 current D0 cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction cap 05[f0] = MSI supports 1 message, 64 bit em1@pci0:10:1:1: class=0x020000 card=0x11798086 chip=0x10798086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82546EB Dual Port Gigabit Ethernet Controller' class = network subclass = ethernet cap 01[dc] = powerspec 2 supports D0 D3 current D0 cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split transaction cap 05[f0] = MSI supports 1 message, 64 bit em2@pci0:13:0:0: class=0x020000 card=0x108c15d9 chip=0x108c8086 rev=0x03 hdr=0x00 vendor = 'Intel Corporation' device = '82573E Intel Corporation 82573E Gigabit Ethernet Controller (Copper)' class = network subclass = ethernet cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 10[e0] = PCI-Express 1 endpoint em3@pci0:14:0:0: class=0x020000 card=0x109a15d9 chip=0x109a8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '82573L Intel PRO/1000 PL Network Adaptor' class = network subclass = ethernet cap 01[c8] = powerspec 2 supports D0 D3 current D0 cap 05[d0] = MSI supports 1 message, 64 bit enabled with 1 message cap 10[e0] = PCI-Express 1 endpoint >Using 7-STABLE and I tried 6-stable and -current >I get the RTM_MISS with 7 and current but only with certain types of >packets at a certain rate.. :/ I wonder if its a bug with the em driver ? I dont have any other dual port cards handy right now to test with >I can not get more than 500kpps.. i tried everything I could think >of... lowering the rx descriptors on EM to 512 instead of 2048 gave >me some more.. I was stuck at 400kpps until i changed those and i >lowered the rx processing limit. >My tests are going incoming em0 and outgoing em1 in one direction >only and it has major errors when em0 taskq gets close to 80% cpu.. I now have 3 boxes now generating traffic through the box acting as a router. I will try some other operating systems as well to see how they compare when back at the office on Wednesday >I am pretty disappointed that it maxes out a little over 400kpps and >even then it gets some errors here and there , mainly missed packets >due to no buffer and rx overruns (dev.em.0.stats=1) Something about the MB you are using perhaps ? Just for rough comparison, how long does # time make -j4 buildkernel > /var/log/build.out.k 670.485u 66.061s 8:29.54 144.5% 5962+1087k 9185+7419io 380pf+0w take on your machine ? The above value is with inet6 and sctp commented out from the kernel. ---Mike >Mike Tancsa wrote: >>At 04:04 AM 6/29/2008, Paul wrote: >>>This is just a question but who can get more than 400k pps >>>forwarding performance ? >> >> >>OK, I setup 2 boxes on either end of a RELENG_7 box from about May >>7th just now, to see with 2 boxes blasting across it how it would work. >>*However*, this is with no firewall loaded and, I must enable ip >>fast forwarding. Without that enabled, the box just falls over. >> >>even at 20Kpps, I start seeing all sorts of messages spewing to >>route -n monitor >> >> >>got message of size 96 on Mon Jun 30 15:39:10 2008 >>RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, >>errno 0, flags: >>locks: inits: >>sockaddrs: >> default >> >>I am starting to wonder if those messages are the results of >>corrupted packets the machine just cant keep up with ? >> >> >>CPU is >> >>CPU: Intel(R) Xeon(R) CPU 3070 @ 2.66GHz (2660.01-MHz >>686-class CPU) >> >> >> input (Total) output >> packets errs bytes packets errs bytes colls >> 611945 0 77892098 611955 0 77013002 0 >> 616727 0 78215508 616742 0 77303454 0 >> 617066 0 78162130 617082 0 77238434 0 >> 618238 0 78302314 618225 0 77377582 0 >> 617035 0 78141000 617038 0 77215672 0 >> 617625 0 78225600 617588 0 77301734 0 >> 616190 0 78017320 616165 0 77091774 0 >> 615583 0 78064130 615628 0 77152800 0 >> 617662 0 78254388 617658 0 77332340 0 >> 618000 0 78269912 617950 0 77344554 0 >> 617248 0 78183136 617315 0 77259588 0 >> 617325 0 78204566 617289 0 77282094 0 >> 618391 0 78337734 618357 0 77413756 0 >> 616025 0 78116070 616082 0 77203116 0 >> >> >>To generate the packets, I am just using >>/usr/src/tools/tools/netblast on 2 endpoints starting at about the same time >> >># ./netblast 10.10.1.2 500 100 40 >> >>start: 1214854131.083679919 >>finish: 1214854171.084668592 >>send calls: 20139141 >>send errors: 0 >>approx send rate: 503478 >>approx error rate: 0 >> >> >># ./netblast 10.10.1.3 500 10 40 >> >>start: 1214854273.882202815 >>finish: 1214854313.882319031 >>send calls: 23354971 >>send errors: 18757223 >>approx send rate: 114943 >>approx error rate: 0 >> >>The box in the middle doing the forwarding >> >>1[spare-r7]# ifconfig -u >>em0: flags=8843 metric 0 mtu 1500 >> >>options=19b >> ether 00:1b:21:08:32:a8 >> inet 10.20.1.1 netmask 0xffffff00 broadcast 10.20.1.255 >> media: Ethernet autoselect (1000baseTX ) >> status: active >>em1: flags=8843 metric 0 mtu 1500 >> options=9b >> ether 00:1b:21:08:32:a9 >> inet 192.168.43.193 netmask 0xffffff00 broadcast 192.168.43.255 >> media: Ethernet autoselect (100baseTX ) >> status: active >>em3: flags=8843 metric 0 mtu 1500 >> >>options=19b >> ether 00:30:48:90:4c:ff >> inet 10.10.1.1 netmask 0xffffff00 broadcast 10.10.1.255 >> media: Ethernet autoselect (1000baseTX ) >> status: active >>lo0: flags=8049 metric 0 mtu 16384 >> inet 127.0.0.1 netmask 0xff000000 >> >> >>I am going to try a few more tests with and without, firewall rules >>etc as well as an updated kernel to RELENG_7 as of today and see how that goes. >> >> ---Mike >> From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 01:36:07 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D2B7B106567D for ; Tue, 1 Jul 2008 01:36:07 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 1A9598FC15 for ; Tue, 1 Jul 2008 01:36:06 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 19677 invoked from network); 1 Jul 2008 03:36:05 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 1 Jul 2008 03:36:05 +0200 Date: Tue, 1 Jul 2008 03:36:04 +0200 (CEST) From: Ingo Flaschberger To: "Support (Rudy)" In-Reply-To: <486986D9.3000607@monkeybrains.net> Message-ID: References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org, "Wilkinson, Alex" Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 01:36:07 -0000 Dear Rudy, > I used polling in FreeBSD 5.x and it helped a bunch. I set up a new router > with 7.0 and MSI was recommended to me. (I noticed no difference when moving > from polling -> MSI, however, on 5.4 polling seemed to help a lot. What are > people using in 7.0? > polling or MSI? if you have a inet-router with gige-uplinks, it is possible that there will be (d)dos attacks. only polling helps you then to keep the router manageable (but dropping packets). Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 01:39:02 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 96181106567C for ; Tue, 1 Jul 2008 01:39:02 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id D14C48FC14 for ; Tue, 1 Jul 2008 01:39:01 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 20196 invoked from network); 1 Jul 2008 03:39:00 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 1 Jul 2008 03:39:00 +0200 Date: Tue, 1 Jul 2008 03:39:00 +0200 (CEST) From: Ingo Flaschberger To: Steve Bertrand In-Reply-To: <4869880D.8040901@ibctech.ca> Message-ID: References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <4869880D.8040901@ibctech.ca> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org, "Wilkinson, Alex" , "Support \(Rudy\)" Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 01:39:02 -0000 Dear Steve, > I'm curious now... how do you change individual device polling via sysctl? not via sysctl, via ifconfig: # enable interface polling /sbin/ifconfig em0 polling /sbin/ifconfig em1 polling /sbin/ifconfig em2 polling /sbin/ifconfig em3 polling (and via /etc/rc.local also across reboots) kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 01:57:13 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78F2B1065672 for ; Tue, 1 Jul 2008 01:57:13 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id 42FF38FC14 for ; Tue, 1 Jul 2008 01:57:13 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.2/8.14.2) with ESMTP id m611vBqg044889; Mon, 30 Jun 2008 21:57:11 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m611vAOt083163 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 30 Jun 2008 21:57:10 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200807010157.m611vAOt083163@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Mon, 30 Jun 2008 21:57:18 -0400 To: Steve Bertrand From: Mike Tancsa In-Reply-To: <48695BA6.7060207@ibctech.ca> References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <48695BA6.7060207@ibctech.ca> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: FreeBSD Net , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 01:57:13 -0000 At 06:18 PM 6/30/2008, Steve Bertrand wrote: >Mike Tancsa wrote: >>At 04:04 AM 6/29/2008, Paul wrote: >>>This is just a question but who can get more than 400k pps >>>forwarding performance ? >> >>OK, I setup 2 boxes on either end of a RELENG_7 box from about May >>7th just now, to see with 2 boxes blasting across it how it would work. >>*However*, this is with no firewall loaded and, I must enable ip >>fast forwarding. Without that enabled, the box just falls over. >>even at 20Kpps, I start seeing all sorts of messages spewing to >>route -n monitor >> >>got message of size 96 on Mon Jun 30 15:39:10 2008 >>RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, >>errno 0, flags: >>locks: inits: >>sockaddrs: >> default > >Mike, > >Is the monitor running on the 7.0 box in the middle you are testing? In the middle On the box that has 0[r7-router]# ifconfig -u em0: flags=8843 metric 0 mtu 1500 options=9b ether 00:1b:21:08:32:a8 inet 10.20.1.1 netmask 0xffffff00 broadcast 10.20.1.255 media: Ethernet autoselect (1000baseTX ) status: active em1: flags=8843 metric 0 mtu 1500 options=9b ether 00:1b:21:08:32:a9 inet 192.168.43.193 netmask 0xffffff00 broadcast 192.168.43.255 media: Ethernet autoselect (100baseTX ) status: active em3: flags=8843 metric 0 mtu 1500 options=19b ether 00:30:48:90:4c:ff inet 10.10.1.1 netmask 0xffffff00 broadcast 10.10.1.255 media: Ethernet autoselect (1000baseTX ) status: active lo0: flags=8049 metric 0 mtu 16384 inet 127.0.0.1 netmask 0xff000000 0[r7-router]# If I dont have fast forwarding on, yes it seems one packet generates one error message per packet forwarded. This sure does feel like a driver bug, but I just tried on another machine with rl0 acting as a forwarding interface and it too generates a message per packet forwarded! From one end point that sends packets across r7-router # ping -c3 -S 10.10.1.2 10.20.1.3 PING 10.20.1.3 (10.20.1.3) from 10.10.1.2: 56 data bytes 64 bytes from 10.20.1.3: icmp_seq=0 ttl=63 time=0.349 ms 64 bytes from 10.20.1.3: icmp_seq=1 ttl=63 time=0.319 ms 64 bytes from 10.20.1.3: icmp_seq=2 ttl=63 time=0.343 ms --- 10.20.1.3 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.319/0.337/0.349/0.013 ms Here are all the messages generated on the intermediary router 1[r7-router]# route -n monitor got message of size 96 on Mon Jun 30 21:36:41 2008 RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 96 on Mon Jun 30 21:36:41 2008 RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 96 on Mon Jun 30 21:36:42 2008 RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 96 on Mon Jun 30 21:36:42 2008 RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 96 on Mon Jun 30 21:36:42 2008 RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 96 on Mon Jun 30 21:36:43 2008 RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 96 on Mon Jun 30 21:36:43 2008 RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 96 on Mon Jun 30 21:36:44 2008 RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default If I turn on fast forwarding on r7-router, route -n monitor doesnt spew out messages >I set up the same configuration, and even with almost no load (< >1Kpps) can replicate these error messages by making the remote IP >address (in your case 'default', disappear (ie: unplug the cable, DDoS etc). > >...to further, I can even replicate the problem at a single packet >per second by trying to ping an IP address that I know for fact that >the router can not get to. I get it when I can ping a valid host on the other side, which responds >Do you see these error messages if you set up a loopback address >with an IP on the router, and effectively chop your test environment >in half? In your case, can the router in the middle actually get to >a default gateway for external addresses (when I perform the test, >your 'default' is substituted with the IP I am trying to reach, so I >am only assuming that 'default' is implying default gateway). No, its only when the box is forwarding packets. If I do the following on the router, 1[r7-router]# sysctl -w net.inet.ip.fastforwarding=0 net.inet.ip.fastforwarding: 1 -> 0 0[r7-router]# 0[r7-router]# ifconfig lo0 10.40.1.1/32 alias 0[r7-router]# route -n monitor ...all is quiet on the router if I do the following # ping -c3 -S 10.10.1.2 10.40.1.1 PING 10.40.1.1 (10.40.1.1) from 10.10.1.2: 56 data bytes 64 bytes from 10.40.1.1: icmp_seq=0 ttl=64 time=0.221 ms 64 bytes from 10.40.1.1: icmp_seq=1 ttl=64 time=0.177 ms 64 bytes from 10.40.1.1: icmp_seq=2 ttl=64 time=0.213 ms --- 10.40.1.1 ping statistics --- 3 packets transmitted, 3 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.177/0.204/0.221/0.019 ms ---Mike From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 01:58:52 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7865B106566C for ; Tue, 1 Jul 2008 01:58:52 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: from ibctech.ca (v6.ibctech.ca [IPv6:2607:f118::b6]) by mx1.freebsd.org (Postfix) with SMTP id 93A938FC15 for ; Tue, 1 Jul 2008 01:58:51 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: (qmail 57698 invoked by uid 89); 1 Jul 2008 02:00:27 -0000 Received: from unknown (HELO ?IPv6:2607:f118::5?) (steve@ibctech.ca@2607:f118::5) by 2607:f118::b6 with ESMTPA; 1 Jul 2008 02:00:27 -0000 Message-ID: <48698F5B.2070605@ibctech.ca> Date: Mon, 30 Jun 2008 21:58:51 -0400 From: Steve Bertrand User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Mike Tancsa References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <48694A9D.1030001@gtcomm.net> <200807010129.m611TUV5083067@lava.sentex.ca> In-Reply-To: <200807010129.m611TUV5083067@lava.sentex.ca> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 01:58:52 -0000 Mike Tancsa wrote: >>> The box in the middle doing the forwarding If I can help in any way, a topo map of the setup that you are facing would be good. What do you have at either end. In the interest of pushing 500kpps, I have this, if it helps with troubleshooting: em0@pci1:0:0: class=0x020000 card=0x00008086 chip=0x109a8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet em1@pci2:0:0: class=0x020000 card=0x00008086 chip=0x109a8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet em2@pci3:0:0: class=0x020000 card=0x00008086 chip=0x109a8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet em3@pci4:0:0: class=0x020000 card=0x00008086 chip=0x109a8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet em4@pci5:0:0: class=0x020000 card=0x00008086 chip=0x109a8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet em5@pci6:0:0: class=0x020000 card=0x00008086 chip=0x109a8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet em6@pci7:0:0: class=0x020000 card=0x00008086 chip=0x109a8086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' class = network subclass = ethernet Steve From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 02:05:06 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F982106567D for ; Tue, 1 Jul 2008 02:05:06 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id 41E948FC18 for ; Tue, 1 Jul 2008 02:05:06 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.2/8.14.2) with ESMTP id m61251ge045460; Mon, 30 Jun 2008 22:05:01 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m61250IT083199 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Mon, 30 Jun 2008 22:05:00 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200807010205.m61250IT083199@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Mon, 30 Jun 2008 22:05:08 -0400 To: "Wilkinson, Alex" , freebsd-net@freebsd.org From: Mike Tancsa In-Reply-To: <20080701004346.GA3898@stlux503.dsto.defence.gov.au> References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 02:05:06 -0000 At 08:43 PM 6/30/2008, Wilkinson, Alex wrote: > 0n Mon, Jun 30, 2008 at 03:44:48PM -0400, Mike Tancsa wrote: > > >OK, I setup 2 boxes on either end of a RELENG_7 box from about May > >7th just now, to see with 2 boxes blasting across it how it would > >work. *However*, this is with no firewall loaded and, I must enable > >ip fast forwarding. Without that enabled, the box just falls over. > >What is "ip fast forwarding" ? From /usr/src/sys/netinet/ip_fastfwd.c /* * ip_fastforward gets its speed from processing the forwarded packet to * completion (if_output on the other side) without any queues or netisr's. * The receiving interface DMAs the packet into memory, the upper half of * driver calls ip_fastforward, we do our routing table lookup and directly * send it off to the outgoing interface, which DMAs the packet to the * network card. The only part of the packet we touch with the CPU is the * IP header (unless there are complex firewall rules touching other parts * of the packet, but that is up to you). We are essentially limited by bus * bandwidth and how fast the network card/driver can set up receives and * transmits. * * We handle basic errors, IP header errors, checksum errors, * destination unreachable, fragmentation and fragmentation needed and * report them via ICMP to the sender. * * Else if something is not pure IPv4 unicast forwarding we fall back to * the normal ip_input processing path. We should only be called from * interfaces connected to the outside world. * * Firewalling is fully supported including divert, ipfw fwd and ipfilter * ipnat and address rewrite. * * IPSEC is not supported if this host is a tunnel broker. IPSEC is * supported for connections to/from local host. * * We try to do the least expensive (in CPU ops) checks and operations * first to catch junk with as little overhead as possible. * * We take full advantage of hardware support for IP checksum and * fragmentation offloading. * * We don't do ICMP redirect in the fast forwarding path. I have had my own * cases where two core routers with Zebra routing suite would send millions * ICMP redirects to connected hosts if the destination router was not the * default gateway. In one case it was filling the routing table of a host * with approximately 300.000 cloned redirect entries until it ran out of * kernel memory. However the networking code proved very robust and it didn't * crash or fail in other ways. */ > -aW > >IMPORTANT: This email remains the property of the Australian Defence >Organisation and is subject to the jurisdiction of section 70 of the >CRIMES ACT 1914. If you have received this email in error, you are >requested to contact the sender and delete the email. > > >_______________________________________________ >freebsd-net@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-net >To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 02:38:21 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D75521065674 for ; Tue, 1 Jul 2008 02:38:21 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id ABFB98FC0A for ; Tue, 1 Jul 2008 02:38:21 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDVhi-0005NJ-E0; Mon, 30 Jun 2008 22:34:50 -0400 Message-ID: <48699915.4020804@gtcomm.net> Date: Mon, 30 Jun 2008 22:40:21 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, "Wilkinson, Alex" Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 02:38:21 -0000 Well it's supposed to, but it doesn't seem to do it as well as it should :> How about copying header direct DMA from NIC into cache, then copy from cache into output NIC after applying whatever filters/changes/etc? Ingo Flaschberger wrote: > Dear Alex, > >> >OK, I setup 2 boxes on either end of a RELENG_7 box from about May >> >7th just now, to see with 2 boxes blasting across it how it would >> >work. *However*, this is with no firewall loaded and, I must enable >> >ip fast forwarding. Without that enabled, the box just falls over. >> >> What is "ip fast forwarding" ? > > instead of copying the while ip packet into system memory, only the ip > header is copyied and then in a "fast" path determined if it could be > fast forwarded. > if possible, a ned header is created at the other network-cards-buffer > and the ip-data is copied from network-card-buffer to > network-card-buffer directly. > > Kind regards, > Ingo Flaschberger > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 02:39:35 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 92B6F106564A for ; Tue, 1 Jul 2008 02:39:35 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 659A28FC13 for ; Tue, 1 Jul 2008 02:39:35 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDViv-0005W4-Kl; Mon, 30 Jun 2008 22:36:05 -0400 Message-ID: <48699960.9070100@gtcomm.net> Date: Mon, 30 Jun 2008 22:41:36 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: "Support (Rudy)" References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> In-Reply-To: <486986D9.3000607@monkeybrains.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, "Wilkinson, Alex" , Ingo Flaschberger Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 02:39:35 -0000 All the NIC drivers in 7 pretty much use interrupt moderation so it can never lock the machine anyway.. This effectively kills polling and it really no longer has any use except to be able to have a fraction of the cpu set aside for user space but you can do that anyway with SMP Support (Rudy) wrote: > Ingo Flaschberger wrote: >> usually interface polling is also chosen to prevent "lock-ups". >> man polling > > > I used polling in FreeBSD 5.x and it helped a bunch. I set up a new > router with 7.0 and MSI was recommended to me. (I noticed no > difference when moving from polling -> MSI, however, on 5.4 polling > seemed to help a lot. What are people using in 7.0? > polling or MSI? > > Rudy > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 03:02:46 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7BC881065673 for ; Tue, 1 Jul 2008 03:02:46 +0000 (UTC) (envelope-from wilkinsa@obelix.dsto.defence.gov.au) Received: from digger1.defence.gov.au (digger1.defence.gov.au [203.5.217.4]) by mx1.freebsd.org (Postfix) with ESMTP id EA5608FC2E for ; Tue, 1 Jul 2008 03:02:45 +0000 (UTC) (envelope-from wilkinsa@obelix.dsto.defence.gov.au) Received: from ednmsw510.dsto.defence.gov.au (ednmsw510.dsto.defence.gov.au [131.185.68.11]) by digger1.defence.gov.au (DSTO/DSTO) with ESMTP id m6132DcZ021396 for ; Tue, 1 Jul 2008 12:32:13 +0930 (CST) Received: from fmbex510.dsto.defence.gov.au (fmbex510.dsto.defence.gov.au) by ednmsw510.dsto.defence.gov.au (Clearswift SMTPRS 5.2.9) with ESMTP id for ; Tue, 1 Jul 2008 12:32:44 +0930 Received: from stlex510.dsto.defence.gov.au ([203.6.60.184]) by fmbex510.dsto.defence.gov.au with Microsoft SMTPSVC(6.0.3790.1830); Tue, 1 Jul 2008 13:02:43 +1000 Received: from obelix.dsto.defence.gov.au ([203.6.60.208]) by stlex510.dsto.defence.gov.au with Microsoft SMTPSVC(6.0.3790.1830); Tue, 1 Jul 2008 11:02:43 +0800 Received: from obelix.dsto.defence.gov.au (localhost [127.0.0.1]) by obelix.dsto.defence.gov.au (8.14.2/8.14.2) with ESMTP id m6132Goi008366 for ; Tue, 1 Jul 2008 11:02:16 +0800 (WST) (envelope-from wilkinsa@obelix.dsto.defence.gov.au) Received: (from wilkinsa@localhost) by obelix.dsto.defence.gov.au (8.14.2/8.14.2/Submit) id m6132G2P008365 for freebsd-net@freebsd.org; Tue, 1 Jul 2008 11:02:16 +0800 (WST) (envelope-from wilkinsa) Date: Tue, 1 Jul 2008 11:02:16 +0800 From: "Wilkinson, Alex" To: freebsd-net@freebsd.org Message-ID: <20080701030216.GP3898@stlux503.dsto.defence.gov.au> Mail-Followup-To: freebsd-net@freebsd.org References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Disposition: inline In-Reply-To: <48699960.9070100@gtcomm.net> Organisation: Defence Science Technology Organisation User-Agent: Mutt/1.5.18 (2008-05-17) X-OriginalArrivalTime: 01 Jul 2008 03:02:43.0217 (UTC) FILETIME=[EE3B6810:01C8DB26] X-TM-AS-Product-Ver: SMEX-7.0.0.1526-5.5.1027-16002.007 X-TM-AS-Result: No--0.740000-0.000000-31 Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 03:02:46 -0000 0n Mon, Jun 30, 2008 at 10:41:36PM -0400, Paul wrote: >All the NIC drivers in 7 pretty much use interrupt moderation so it can >never lock the machine anyway.. This effectively kills polling and it >really no longer has any use except to be able to have a fraction of the >cpu set aside for user space but you can do that anyway with SMP what is "interrupt moderation" ? -aW IMPORTANT: This email remains the property of the Australian Defence Organisation and is subject to the jurisdiction of section 70 of the CRIMES ACT 1914. If you have received this email in error, you are requested to contact the sender and delete the email. From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 03:05:04 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E65F4106567D for ; Tue, 1 Jul 2008 03:05:04 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.251]) by mx1.freebsd.org (Postfix) with ESMTP id 992408FC0C for ; Tue, 1 Jul 2008 03:05:04 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: by an-out-0708.google.com with SMTP id b33so385993ana.13 for ; Mon, 30 Jun 2008 20:05:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=PL+8lZ0zGKyaatfkZrSjVMInwCzhLUxLyP8jIT/B5tk=; b=hsH9O76O7tnFqjCf0wkuCte/T2ccW8iZ+WZ2yNfSskbqAgpqXxUKGULpjG8rhEp49Q 4fzteawWj5ysxsE6sco7/JOpt7KSwAWdTXE0SYph1sfQ/ZLrBrOGrbhQrA98VKXQLo6h vpmrTyaRiBD+uizfZ8eNMULLzuPKONop8amlU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=r8JPPpQWbXOAoprqGIRYVbZnZsN8N7qVQKhzfc78M344esV/MIObD1wrjwRzuDp2ne 4LqGm/IaD+xE6e6QA/pWA9OEMhJlah4flIm74TL6JAhBlWNa85hK+iqrqLNAD3VaSQ1E Fd2L8fudUAaREcOkfvo7O98q/snKyk4/+OkPY= Received: by 10.100.194.5 with SMTP id r5mr5111215anf.2.1214881503109; Mon, 30 Jun 2008 20:05:03 -0700 (PDT) Received: by 10.100.107.6 with HTTP; Mon, 30 Jun 2008 20:05:03 -0700 (PDT) Message-ID: Date: Tue, 1 Jul 2008 11:05:03 +0800 From: "Sepherosa Ziehau" To: Paul In-Reply-To: <48699960.9070100@gtcomm.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> Cc: freebsd-net@freebsd.org, Ingo Flaschberger , "Wilkinson, Alex" , "Support \(Rudy\)" Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 03:05:05 -0000 On 7/1/08, Paul wrote: > All the NIC drivers in 7 pretty much use interrupt moderation so it can I am not quite sure whether em(4)'s RX interrupt moderation works as expected or not. But, AFAIK, nfe(4) and re(4) does not have RX interrupt moderation. Their TX interrupt moderation could be mimiced by using their hardware timer and disabling their TX interrupt. The lacking of RX im is difficult to handle, I could imagine following way: - During init, enable RX intr - When RX intr comes, disable RX intr and set up hardware timer intr - When timer intr comes and no RX happens, disable timer intr and enable RX intr Properly configured #RX desc and timer intr interval will be required to make sure that the RX desc collection could keep up with the hardware speed. I used pure timer intr (8000Hz) on nfe(4) in dfly w/ good result, i.e. TX/RX @linespeed without livelocking the system. The drawback of pure timer intr is that you waste extra cpu power, when there is nothing to process. > never lock the machine anyway.. This effectively kills polling and it really > no longer has any use except to be able to have a fraction of the cpu set Oh? Really? :] Best Regards, sephe -- Live Free or Die From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 03:10:25 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 872901065672 for ; Tue, 1 Jul 2008 03:10:25 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 409B48FC1C for ; Tue, 1 Jul 2008 03:10:25 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDWCl-0000mb-7x; Mon, 30 Jun 2008 23:06:55 -0400 Message-ID: <4869A099.5070206@gtcomm.net> Date: Mon, 30 Jun 2008 23:12:25 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Sepherosa Ziehau References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Ingo Flaschberger , "Wilkinson, Alex" , "Support \(Rudy\)" Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 03:10:25 -0000 I have been unable to even come close to livelocking the machine with the em driver interrupt moderation. So that to me throws polling out the window. I tried 8000hz with polling modified to allow 10000 burst and it makes no difference in the amount of pps I can jam through.. It' seems to be limited by the routing path in the kernel more than anything else. If a driver/hardware didn't support interrupt mitigation then it would definitely lock the machine. Sepherosa Ziehau wrote: > On 7/1/08, Paul wrote: > >> All the NIC drivers in 7 pretty much use interrupt moderation so it can >> > > I am not quite sure whether em(4)'s RX interrupt moderation works as > expected or not. But, AFAIK, nfe(4) and re(4) does not have RX > interrupt moderation. Their TX interrupt moderation could be mimiced > by using their hardware timer and disabling their TX interrupt. > > The lacking of RX im is difficult to handle, I could imagine following way: > - During init, enable RX intr > - When RX intr comes, disable RX intr and set up hardware timer intr > - When timer intr comes and no RX happens, disable timer intr and enable RX intr > > Properly configured #RX desc and timer intr interval will be required > to make sure that the RX desc collection could keep up with the > hardware speed. I used pure timer intr (8000Hz) on nfe(4) in dfly w/ > good result, i.e. TX/RX @linespeed without livelocking the system. > The drawback of pure timer intr is that you waste extra cpu power, > when there is nothing to process. > > >> never lock the machine anyway.. This effectively kills polling and it really >> no longer has any use except to be able to have a fraction of the cpu set >> > > Oh? Really? :] > > Best Regards, > sephe > > From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 03:33:37 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 397E1106567C for ; Tue, 1 Jul 2008 03:33:37 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.230]) by mx1.freebsd.org (Postfix) with ESMTP id ECEC38FC1A for ; Tue, 1 Jul 2008 03:33:36 +0000 (UTC) (envelope-from pyunyh@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so1951026rvf.43 for ; Mon, 30 Jun 2008 20:33:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:received:date:from :to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; bh=0tpsrP5xDdsJ9/FXj8fjtU75eqLOZrdVOLQhuuIRAX8=; b=uhZe3q6xMVMDkf9qhyFailwf2SsGvVbDNpcSF5GXBpbaWGHHbpuz2CIbqWssj9DCpD Egj0j0n2oShZH0284JcnuQsU//Lk+yzhVQwLtbs/TLuhIDVgG9VHg2b5hJv7DGBtndq2 kxyH5BsqA7hj5zCl3aesPke078MxYIb7fawnU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:reply-to:references:mime-version :content-type:content-disposition:in-reply-to:user-agent; b=Oib6eNV2HIuoWbs4XlSSVhLSh+76utn54K8WDlrBW3uaZZZINm/Xi1+AgxrO4QJFIx SbGMYwTjZuWRdc7O/n2eFOSu9IDJDN/KqJL14xanVgy55zZhahzkEAdpZCVh3QKgGWxy Vh4rDdewAYdUQk0ZbOZTYBUxi0vJ+sutJXo4E= Received: by 10.141.5.17 with SMTP id h17mr3194316rvi.8.1214883216528; Mon, 30 Jun 2008 20:33:36 -0700 (PDT) Received: from michelle.cdnetworks.co.kr ( [211.53.35.84]) by mx.google.com with ESMTPS id b8sm8977089rvf.9.2008.06.30.20.33.33 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 30 Jun 2008 20:33:35 -0700 (PDT) Received: from michelle.cdnetworks.co.kr (localhost.cdnetworks.co.kr [127.0.0.1]) by michelle.cdnetworks.co.kr (8.13.5/8.13.5) with ESMTP id m613VQlv084406 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 1 Jul 2008 12:31:26 +0900 (KST) (envelope-from pyunyh@gmail.com) Received: (from yongari@localhost) by michelle.cdnetworks.co.kr (8.13.5/8.13.5/Submit) id m613VI6Y084405; Tue, 1 Jul 2008 12:31:18 +0900 (KST) (envelope-from pyunyh@gmail.com) Date: Tue, 1 Jul 2008 12:31:18 +0900 From: Pyun YongHyeon To: Sepherosa Ziehau Message-ID: <20080701033117.GH83626@cdnetworks.co.kr> References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.1i Cc: freebsd-net@freebsd.org, "Support \(Rudy\)" , "Wilkinson, Alex" , Ingo Flaschberger , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: pyunyh@gmail.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 03:33:37 -0000 On Tue, Jul 01, 2008 at 11:05:03AM +0800, Sepherosa Ziehau wrote: > On 7/1/08, Paul wrote: > > All the NIC drivers in 7 pretty much use interrupt moderation so it can > > I am not quite sure whether em(4)'s RX interrupt moderation works as > expected or not. But, AFAIK, nfe(4) and re(4) does not have RX > interrupt moderation. Their TX interrupt moderation could be mimiced > by using their hardware timer and disabling their TX interrupt. > > The lacking of RX im is difficult to handle, I could imagine following way: > - During init, enable RX intr > - When RX intr comes, disable RX intr and set up hardware timer intr > - When timer intr comes and no RX happens, disable timer intr and enable RX intr > I guess adaptive polling would give the same effect withtout sacrificing CPU cycles. > Properly configured #RX desc and timer intr interval will be required > to make sure that the RX desc collection could keep up with the > hardware speed. I used pure timer intr (8000Hz) on nfe(4) in dfly w/ > good result, i.e. TX/RX @linespeed without livelocking the system. I thought that too for a while but I prefer to hardware intertrrupt moderation feature. Of course I still have no clue how to enable that interrupt feature on nvidia controllers. :-( > The drawback of pure timer intr is that you waste extra cpu power, > when there is nothing to process. > > > never lock the machine anyway.. This effectively kills polling and it really > > no longer has any use except to be able to have a fraction of the cpu set > > Oh? Really? :] > > Best Regards, > sephe > > -- > Live Free or Die -- Regards, Pyun YongHyeon From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 03:34:53 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DEC35106566C for ; Tue, 1 Jul 2008 03:34:53 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.247]) by mx1.freebsd.org (Postfix) with ESMTP id 8F82C8FC29 for ; Tue, 1 Jul 2008 03:34:53 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: by an-out-0708.google.com with SMTP id b33so388054ana.13 for ; Mon, 30 Jun 2008 20:34:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=F1hv5LS2zU0lUMOVJO+Ny44Wk9VfvAQaiIQ9f1r+aW8=; b=Pwi4f9BMjDBKdZSQerJvpsygE9/bbOvgFIcJYgENyoKCsrPPK0Jyc7ck5PKXlZSI/s IbFwV9IM2qgsG0iJxywH3r7sI10AfuvknfoQeqj0baqcKkCXjhO/1lYGKv6xlRcL4k6Q cfEs4rLQEYSpArlPX6z7qazmLSqmfjnRvHvV0= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=kO1JsrSsB/n8O09hOgSqEpZ4dO78RTR8qPKQgXFrcd2BxmHbfYQ+MWe6z3jmrewPoD l9Oc57QnilYxqPOSIOQWLpma6N0qgReK9q/9W/sU5CVk87txhnMqJ2GNpk3cHycNUAkP CFLA0eeJMeLAprIna6hStG/WHrV/z7sImHodE= Received: by 10.100.154.9 with SMTP id b9mr5062681ane.78.1214883292462; Mon, 30 Jun 2008 20:34:52 -0700 (PDT) Received: by 10.100.107.6 with HTTP; Mon, 30 Jun 2008 20:34:52 -0700 (PDT) Message-ID: Date: Tue, 1 Jul 2008 11:34:52 +0800 From: "Sepherosa Ziehau" To: Paul In-Reply-To: <4869A099.5070206@gtcomm.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <4869A099.5070206@gtcomm.net> Cc: freebsd-net@freebsd.org, Ingo Flaschberger , "Wilkinson, Alex" , "Support \(Rudy\)" Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 03:34:53 -0000 On 7/1/08, Paul wrote: > I have been unable to even come close to livelocking the machine with the em > driver interrupt moderation. Yeah, system will not be livelocked. But even setting its imtimer to 4000, the overall system response is still worse than using polling @4000 with a 9402PT. > So that to me throws polling out the window. I tried 8000hz with polling I don't believe high polling rate will improve forwarding performance. I used to set polling rate to 2000hz, burst max to 750 and each burst to 60. > modified to allow 10000 burst and it makes no difference > in the amount of pps I can jam through.. It' seems to be limited by the > routing path in the kernel more than anything else. > > If a driver/hardware didn't support interrupt mitigation then it would > definitely lock the machine. So polling(4) still has its place. Best Regards, sephe -- Live Free or Die From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 03:50:25 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7DD6E106568D for ; Tue, 1 Jul 2008 03:50:25 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.246]) by mx1.freebsd.org (Postfix) with ESMTP id 2799E8FC17 for ; Tue, 1 Jul 2008 03:50:22 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: by an-out-0708.google.com with SMTP id b33so388804ana.13 for ; Mon, 30 Jun 2008 20:50:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=Z+oEB9RKD3pSstDnS0x+7VVqNcbLj2ViI8GSIWXRmz8=; b=W/YKWQBGAIkWxc4th2DAl9abyAZCxquadxL0O4FFpR1TXLJPFKGFsq9R3yYBHIcIwO dzKFNWW0wIS/H8mUqxivxDz6ZwQwQSSU5p8RSwHSvvF9SsZdKgAmvzdPGpkP+dZ4CEaI bfrf9FZh7Jw9PC0UpdgstslcP+FFZCPN0XnhA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=B56qE+mfioAMMmKqn9fR+BEtD/Pp5zixIt1XGs/EZnwUW/sh8WYlQHlF1kMqx7/5vj Ekp6uGxckSTcBFJ0Rh+q6/6v8seppDQ0jaJuymgiprgXjxw8Othos/0B41FA/1Z8MRJe +PPSgqA+asiS93P8oypZv5HtyA/CA7PWgFlOI= Received: by 10.100.136.15 with SMTP id j15mr5111544and.11.1214884222342; Mon, 30 Jun 2008 20:50:22 -0700 (PDT) Received: by 10.100.107.6 with HTTP; Mon, 30 Jun 2008 20:50:22 -0700 (PDT) Message-ID: Date: Tue, 1 Jul 2008 11:50:22 +0800 From: "Sepherosa Ziehau" To: pyunyh@gmail.com In-Reply-To: <20080701033117.GH83626@cdnetworks.co.kr> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> Cc: freebsd-net@freebsd.org, "Support \(Rudy\)" , "Wilkinson, Alex" , Ingo Flaschberger , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 03:50:25 -0000 On 7/1/08, Pyun YongHyeon wrote: > On Tue, Jul 01, 2008 at 11:05:03AM +0800, Sepherosa Ziehau wrote: > > On 7/1/08, Paul wrote: > > > All the NIC drivers in 7 pretty much use interrupt moderation so it can > > > > I am not quite sure whether em(4)'s RX interrupt moderation works as > > expected or not. But, AFAIK, nfe(4) and re(4) does not have RX > > interrupt moderation. Their TX interrupt moderation could be mimiced > > by using their hardware timer and disabling their TX interrupt. > > > > The lacking of RX im is difficult to handle, I could imagine following way: > > - During init, enable RX intr > > - When RX intr comes, disable RX intr and set up hardware timer intr > > - When timer intr comes and no RX happens, disable timer intr and enable RX intr > > > > > I guess adaptive polling would give the same effect withtout > sacrificing CPU cycles. The possible wasting is one extra timer intr if there is nothing to processing at all. But would it be counted as wasting, if the system was that idle? :) We will see the result, when I could find some free time to implement it :] > > > > Properly configured #RX desc and timer intr interval will be required > > to make sure that the RX desc collection could keep up with the > > hardware speed. I used pure timer intr (8000Hz) on nfe(4) in dfly w/ > > good result, i.e. TX/RX @linespeed without livelocking the system. > > > I thought that too for a while but I prefer to hardware intertrrupt > moderation feature. Of course I still have no clue how to enable > that interrupt feature on nvidia controllers. :-( RX/TX intr is not affected at all by the so called IMTIMER register. The IMTIMER register is actually only a hardware timer counter register. I took a look at Linux's forcedeth several days ago, but I didn't see anything improved in that area. Best Regards, sephe -- Live Free or Die From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 04:03:13 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7F3F31065680 for ; Tue, 1 Jul 2008 04:03:13 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 3C16C8FC18 for ; Tue, 1 Jul 2008 04:03:13 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDX1t-0007Sf-0q for freebsd-net@freebsd.org; Mon, 30 Jun 2008 23:59:45 -0400 Message-ID: <4869ACFC.5020205@gtcomm.net> Date: Tue, 01 Jul 2008 00:05:16 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 CC: FreeBSD Net References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 04:03:13 -0000 Dual opteron 270 32 bit GENERIC KERNEL Nothing changed in sysctl except forwarding and ip forwarding Broadcom interfaces on board NIC last pid: 11557; load averages: 1.13, 0.83, 0.48 up 0+03:24:26 21:58:38 70 processes: 6 running, 46 sleeping, 18 waiting CPU states: 0.0% user, 0.0% nice, 0.2% system, 28.8% interrupt, 71.1% idle Mem: 9124K Active, 6844K Inact, 26M Wired, 9776K Buf, 1957M Free Swap: 4096M Total, 4096M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 13 root 171 ki31 0K 8K CPU1 1 203:39 98.93% idle: cpu1 14 root 171 ki31 0K 8K RUN 0 203:26 98.05% idle: cpu0 33 root -68 - 0K 8K CPU2 2 13:06 93.99% irq24: bge0 11 root 171 ki31 0K 8K CPU3 3 201:59 79.10% idle: cpu3 34 root -68 - 0K 8K WAIT 3 1:49 18.90% irq25: bge1 12 root 171 ki31 0K 8K RUN 2 190:42 4.00% idle: cpu2 input (bge0) output packets errs bytes packets errs bytes colls 410143 65619 24608586 1 0 226 0 410350 66749 24621006 1 0 178 0 409452 65382 24567126 1 0 178 0 410338 63157 24620286 1 0 178 0 410125 65777 24607446 1 0 178 0 409794 63018 24587706 1 0 178 0 408208 67566 24492486 1 0 178 0 408416 70305 24504906 1 0 178 0 407919 68339 24475206 1 0 178 0 sysctl -a | grep bge.0.stats dev.bge.0.stats.FramesDroppedDueToFilters: 0 dev.bge.0.stats.DmaWriteQueueFull: 310574781 dev.bge.0.stats.DmaWriteHighPriQueueFull: 0 dev.bge.0.stats.NoMoreRxBDs: 0 dev.bge.0.stats.InputDiscards: 20942213 dev.bge.0.stats.InputErrors: 15 dev.bge.0.stats.RecvThresholdHit: 51440366 dev.bge.0.stats.DmaReadQueueFull: 0 dev.bge.0.stats.DmaReadHighPriQueueFull: 0 dev.bge.0.stats.SendDataCompQueueFull: 0 dev.bge.0.stats.RingSetSendProdIndex: 25738223 dev.bge.0.stats.RingStatusUpdate: 51719806 dev.bge.0.stats.Interrupts: 51719806 dev.bge.0.stats.AvoidedInterrupts: 0 dev.bge.0.stats.SendThresholdHit: 0 dev.bge.0.stats.rx.Octets: 3477501149 dev.bge.0.stats.rx.Fragments: 116 dev.bge.0.stats.rx.UcastPkts: 519429709 dev.bge.0.stats.rx.MulticastPkts: 0 dev.bge.0.stats.rx.FCSErrors: 3 dev.bge.0.stats.rx.AlignmentErrors: 0 dev.bge.0.stats.rx.xonPauseFramesReceived: 0 dev.bge.0.stats.rx.xoffPauseFramesReceived: 0 dev.bge.0.stats.rx.ControlFramesReceived: 0 dev.bge.0.stats.rx.xoffStateEntered: 0 dev.bge.0.stats.rx.FramesTooLong: 0 dev.bge.0.stats.rx.Jabbers: 0 dev.bge.0.stats.rx.UndersizePkts: 0 dev.bge.0.stats.rx.inRangeLengthError: 0 dev.bge.0.stats.rx.outRangeLengthError: 0 dev.bge.0.stats.tx.Octets: 2215096864 dev.bge.0.stats.tx.Collisions: 0 dev.bge.0.stats.tx.XonSent: 0 dev.bge.0.stats.tx.XoffSent: 0 dev.bge.0.stats.tx.flowControlDone: 0 dev.bge.0.stats.tx.InternalMacTransmitErrors: 0 dev.bge.0.stats.tx.SingleCollisionFrames: 0 dev.bge.0.stats.tx.MultipleCollisionFrames: 0 dev.bge.0.stats.tx.DeferredTransmissions: 0 dev.bge.0.stats.tx.ExcessiveCollisions: 0 dev.bge.0.stats.tx.LateCollisions: 0 dev.bge.0.stats.tx.UcastPkts: 25738364 dev.bge.0.stats.tx.MulticastPkts: 0 dev.bge.0.stats.tx.BroadcastPkts: 15 dev.bge.0.stats.tx.CarrierSenseErrors: 0 dev.bge.0.stats.tx.Discards: 0 dev.bge.0.stats.tx.Errors: 0 errors..ERRORS!)(@!*() Hitting that 400k pps limit again, and this is a slower machine than my dual 2212. Going to compile 7-STABLE with options for the cpu and will report back From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 04:16:43 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 24D40106564A for ; Tue, 1 Jul 2008 04:16:43 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id D60598FC17 for ; Tue, 1 Jul 2008 04:16:42 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDXEw-0000tx-MQ for freebsd-net@freebsd.org; Tue, 01 Jul 2008 00:13:14 -0400 Message-ID: <4869B025.9080006@gtcomm.net> Date: Tue, 01 Jul 2008 00:18:45 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: FreeBSD Net References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> In-Reply-To: <4869ACFC.5020205@gtcomm.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 04:16:43 -0000 Dual opteron 2212, amd64 kernel, GENERIC (same setup as my 270 I just posted except this is 64 bit and different NIC) Intel dual port 82571 , nothing changed in sysctl except fw and fastfw input (em0) output packets errs bytes packets errs bytes colls 455729 69094 27343822 9 0 1412 0 455582 67566 27334938 3 0 566 0 455525 66033 27331518 3 0 566 0 456632 68384 27397938 3 0 566 0 457144 70006 27428604 4 0 776 0 453551 71307 27213226 7 0 1038 0 455282 74669 27317022 6 0 1068 0 452268 75878 27136104 4 0 744 0 455447 76942 27326838 4 0 744 0 455408 68166 27324522 5 0 922 0 456113 74600 27366798 4 0 744 0 456316 68522 27378996 5 0 922 0 455888 71665 27353322 5 0 1034 0 453448 76407 27206904 4 0 744 0 453907 79158 27234444 4 0 744 0 452921 66532 27175278 4 0 744 0 456049 60947 27363048 8 0 1216 0 em0: Excessive collisions = 0 em0: Sequence errors = 0 em0: Defer count = 0 em0: Missed Packets = 131239807 em0: Receive No Buffers = 154088217 em0: Receive Length Errors = 0 em0: Receive errors = 0 em0: Crc errors = 0 em0: Alignment errors = 0 em0: Collision/Carrier extension errors = 0 em0: RX overruns = 4986006 em0: watchdog timeouts = 0 em0: XON Rcvd = 0 em0: XON Xmtd = 0 em0: XOFF Rcvd = 0 em0: XOFF Xmtd = 0 em0: Good Packets Rcvd = 969917996 em0: Good Packets Xmtd = 98120420 em0: TSO Contexts Xmtd = 2062 em0: TSO Contexts Failed = 0 errrrrrrrrrrorss..... 79 processes: 6 running, 52 sleeping, 3 stopped, 18 waiting CPU states: 0.0% user, 0.0% nice, 28.2% system, 0.0% interrupt, 71.8% idle Mem: 18M Active, 787M Inact, 241M Wired, 196K Cache, 213M Buf, 928M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root 171 ki31 0K 16K CPU2 2 348:19 99.02% idle: cpu2 13 root 171 ki31 0K 16K CPU1 1 347:02 99.02% idle: cpu1 37 root -68 - 0K 16K CPU3 3 37:48 99.02% em0 taskq 14 root 171 ki31 0K 16K RUN 0 340:08 85.94% idle: cpu0 38 root -68 - 0K 16K - 1 8:46 12.06% em1 taskq How do you like those IDLE CPUS :) except 3 of course.. :> Paul wrote: > Dual opteron 270 > 32 bit GENERIC KERNEL Nothing changed in sysctl except forwarding and > ip forwarding > Broadcom interfaces on board NIC > > last pid: 11557; load averages: 1.13, 0.83, > 0.48 > up 0+03:24:26 21:58:38 > 70 processes: 6 running, 46 sleeping, 18 waiting > CPU states: 0.0% user, 0.0% nice, 0.2% system, 28.8% interrupt, > 71.1% idle > Mem: 9124K Active, 6844K Inact, 26M Wired, 9776K Buf, 1957M Free > Swap: 4096M Total, 4096M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 13 root 171 ki31 0K 8K CPU1 1 203:39 98.93% idle: cpu1 > 14 root 171 ki31 0K 8K RUN 0 203:26 98.05% idle: cpu0 > 33 root -68 - 0K 8K CPU2 2 13:06 93.99% irq24: bge0 > 11 root 171 ki31 0K 8K CPU3 3 201:59 79.10% idle: cpu3 > 34 root -68 - 0K 8K WAIT 3 1:49 18.90% irq25: bge1 > 12 root 171 ki31 0K 8K RUN 2 190:42 4.00% idle: cpu2 > > input (bge0) output > packets errs bytes packets errs bytes colls > 410143 65619 24608586 1 0 226 0 > 410350 66749 24621006 1 0 178 0 > 409452 65382 24567126 1 0 178 0 > 410338 63157 24620286 1 0 178 0 > 410125 65777 24607446 1 0 178 0 > 409794 63018 24587706 1 0 178 0 > 408208 67566 24492486 1 0 178 0 > 408416 70305 24504906 1 0 178 0 > 407919 68339 24475206 1 0 178 0 > > > sysctl -a | grep bge.0.stats > dev.bge.0.stats.FramesDroppedDueToFilters: 0 > dev.bge.0.stats.DmaWriteQueueFull: 310574781 > dev.bge.0.stats.DmaWriteHighPriQueueFull: 0 > dev.bge.0.stats.NoMoreRxBDs: 0 > dev.bge.0.stats.InputDiscards: 20942213 > dev.bge.0.stats.InputErrors: 15 > dev.bge.0.stats.RecvThresholdHit: 51440366 > dev.bge.0.stats.DmaReadQueueFull: 0 > dev.bge.0.stats.DmaReadHighPriQueueFull: 0 > dev.bge.0.stats.SendDataCompQueueFull: 0 > dev.bge.0.stats.RingSetSendProdIndex: 25738223 > dev.bge.0.stats.RingStatusUpdate: 51719806 > dev.bge.0.stats.Interrupts: 51719806 > dev.bge.0.stats.AvoidedInterrupts: 0 > dev.bge.0.stats.SendThresholdHit: 0 > dev.bge.0.stats.rx.Octets: 3477501149 > dev.bge.0.stats.rx.Fragments: 116 > dev.bge.0.stats.rx.UcastPkts: 519429709 > dev.bge.0.stats.rx.MulticastPkts: 0 > dev.bge.0.stats.rx.FCSErrors: 3 > dev.bge.0.stats.rx.AlignmentErrors: 0 > dev.bge.0.stats.rx.xonPauseFramesReceived: 0 > dev.bge.0.stats.rx.xoffPauseFramesReceived: 0 > dev.bge.0.stats.rx.ControlFramesReceived: 0 > dev.bge.0.stats.rx.xoffStateEntered: 0 > dev.bge.0.stats.rx.FramesTooLong: 0 > dev.bge.0.stats.rx.Jabbers: 0 > dev.bge.0.stats.rx.UndersizePkts: 0 > dev.bge.0.stats.rx.inRangeLengthError: 0 > dev.bge.0.stats.rx.outRangeLengthError: 0 > dev.bge.0.stats.tx.Octets: 2215096864 > dev.bge.0.stats.tx.Collisions: 0 > dev.bge.0.stats.tx.XonSent: 0 > dev.bge.0.stats.tx.XoffSent: 0 > dev.bge.0.stats.tx.flowControlDone: 0 > dev.bge.0.stats.tx.InternalMacTransmitErrors: 0 > dev.bge.0.stats.tx.SingleCollisionFrames: 0 > dev.bge.0.stats.tx.MultipleCollisionFrames: 0 > dev.bge.0.stats.tx.DeferredTransmissions: 0 > dev.bge.0.stats.tx.ExcessiveCollisions: 0 > dev.bge.0.stats.tx.LateCollisions: 0 > dev.bge.0.stats.tx.UcastPkts: 25738364 > dev.bge.0.stats.tx.MulticastPkts: 0 > dev.bge.0.stats.tx.BroadcastPkts: 15 > dev.bge.0.stats.tx.CarrierSenseErrors: 0 > dev.bge.0.stats.tx.Discards: 0 > dev.bge.0.stats.tx.Errors: 0 > > errors..ERRORS!)(@!*() > > Hitting that 400k pps limit again, and this is a slower machine than > my dual 2212. > Going to compile 7-STABLE with options for the cpu and will report back > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 04:36:28 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 01DF71065670 for ; Tue, 1 Jul 2008 04:36:28 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.228]) by mx1.freebsd.org (Postfix) with ESMTP id BB0A98FC0C for ; Tue, 1 Jul 2008 04:36:22 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so1966502rvf.43 for ; Mon, 30 Jun 2008 21:36:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender :to:subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references :x-google-sender-auth; bh=ZC8nBK0E6fKH0VqnUhaRO7i6NZe7pyMbAcl3FOTELTo=; b=Szr46mDHH6G+hxXUAoxRHR09HIKNM7NSc0MBGyuOutj2iYFYeDd0aZgoVg7LFNf4ek x+xvabMn7qIUv6JB5W6rSoyrOdw8GTVzovDTmYeDmKONth9zFqy6C3wIcND5Qs3EEa8M IO2VoxE9+Or9zd2cI66nUXwXU9Ya4BsSYSgwU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references:x-google-sender-auth; b=VTgM+0DZ9F/gQw+bLchuqteGl0X+QAId0MYr6Suel32Z1eHwWgQFGdD9kyX42ZgxiO /w6+kRoMmD8CPn4VTSrmWErNd0pNRnfIHQfh4y+kRmQxVf3x3/F4sg5FQ4vzoiMRBpII AUcaT8bv3m8k7KHksB9JAbl+x0/Rzf/iHq4dM= Received: by 10.141.206.13 with SMTP id i13mr3164800rvq.211.1214885260317; Mon, 30 Jun 2008 21:07:40 -0700 (PDT) Received: by 10.141.212.9 with HTTP; Mon, 30 Jun 2008 21:07:40 -0700 (PDT) Message-ID: Date: Tue, 1 Jul 2008 12:07:40 +0800 From: "Adrian Chadd" Sender: adrian.chadd@gmail.com To: "Sepherosa Ziehau" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> X-Google-Sender-Auth: 0452eabfa753112e Cc: freebsd-net@freebsd.org, Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 04:36:28 -0000 2008/7/1 Sepherosa Ziehau : > Properly configured #RX desc and timer intr interval will be required > to make sure that the RX desc collection could keep up with the > hardware speed. I used pure timer intr (8000Hz) on nfe(4) in dfly w/ > good result, i.e. TX/RX @linespeed without livelocking the system. > The drawback of pure timer intr is that you waste extra cpu power, > when there is nothing to process. What packet rate is "linespeed" ? With what size packets? arian From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 05:05:38 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AD4BC1065A15 for ; Tue, 1 Jul 2008 05:05:38 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.243]) by mx1.freebsd.org (Postfix) with ESMTP id 5C57A8FC29 for ; Tue, 1 Jul 2008 05:05:38 +0000 (UTC) (envelope-from sepherosa@gmail.com) Received: by an-out-0708.google.com with SMTP id b33so392408ana.13 for ; Mon, 30 Jun 2008 22:05:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=rnD+ZOZPLgq1cdpuF7zPW76vlwPFueoh0GgLQcRlMBU=; b=TWRCEXwKCa/Cl4Y5oY3R8k7xwqlMXnb5g1Wg7JvhJDV8K/vl7P7VfJuLRzr1oue6Ea 4Y63ZcM5ymA0plrO/OclDaC9faDjfOithMmn/6B8EV5I3IFySe3xw5uhE/hcVwCLNDdb BwI+M2PMkBikgyElL3J43iTE3arKqf1lR6fiQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=UZks7VXWMifQk0G5e03HpBul7Tl3YwJFS6phhuhb29xSpDz8NqhHHHzM7+mQCwcna4 +g5sZdI0WIUdLjmWvEtsyiOvRGD/SxtEHai1NXZcmBhjVKUq/wKdX2DA2TGPJneRvDhN z/kL9kXjpjaRmLfEGN5QjyzT+LhXciTjGVHes= Received: by 10.100.141.5 with SMTP id o5mr5142787and.33.1214888737512; Mon, 30 Jun 2008 22:05:37 -0700 (PDT) Received: by 10.100.107.6 with HTTP; Mon, 30 Jun 2008 22:05:37 -0700 (PDT) Message-ID: Date: Tue, 1 Jul 2008 13:05:37 +0800 From: "Sepherosa Ziehau" To: "Adrian Chadd" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> Cc: freebsd-net@freebsd.org, Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 05:05:38 -0000 On 7/1/08, Adrian Chadd wrote: > 2008/7/1 Sepherosa Ziehau : > > > > > Properly configured #RX desc and timer intr interval will be required > > to make sure that the RX desc collection could keep up with the > > hardware speed. I used pure timer intr (8000Hz) on nfe(4) in dfly w/ > > good result, i.e. TX/RX @linespeed without livelocking the system. > > The drawback of pure timer intr is that you waste extra cpu power, > > when there is nothing to process. > > > What packet rate is "linespeed" ? With what size packets? I did not mean pps w/ 64bytes packet. Didn't have time to measure it yet. I only saw that msk(4) and em(4) could accept 64bytes packets @1.4Mpps. I meant netperf -t TCP_STREAM with -s/-S/-m 65536 on an AMDX2 3600+ with 1GBytes ram in i386 mode. Best Regards, sephe -- Live Free or Die From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 06:06:49 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B002F106567B; Tue, 1 Jul 2008 06:06:49 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id 847668FC1D; Tue, 1 Jul 2008 06:06:49 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.2/8.14.2) with ESMTP id m6166k6i062688; Tue, 1 Jul 2008 02:06:46 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m6166jFe084204 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 1 Jul 2008 02:06:46 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200807010606.m6166jFe084204@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 01 Jul 2008 02:06:53 -0400 To: "Bruce M. Simpson" From: Mike Tancsa In-Reply-To: References: <4852E23E.2040505@gtcomm.net> <4854EBF1.7020708@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: freebsd-net@freebsd.org, bz@freebsd.org, Paul Subject: Re: Route messages X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 06:06:49 -0000 At 10:34 PM 6/27/2008, mike@sentex.net wrote: >On Sun, 15 Jun 2008 11:16:17 +0100, in sentex.lists.freebsd.net you >wrote: > > >Paul wrote: > >> Get these with GRE tunnel on > >> FreeBSD 7.0-STABLE FreeBSD 7.0-STABLE #5: Sun May 11 19:00:57 EDT > >> 2008 :/usr/obj/usr/src/sys/ROUTER amd64 > >> But do not get them with 7.0-RELEASE > >> > >> Any ideas what changed? :) Wish there was some sort of changelog.. > >> # of messages per second seems consistent with packets per second on > >> GRE interface.. > >> No impact in routing, but definitely impact in cpu usage for all > >> processes monitoring the route messages. > > > >RTM_MISS is actually fairly common when you don't have a default route. > > > >Hi, > I am seeing this issue as well on a pair of recently deployed >boxes, one running MPD and one acting as an area router in front of >it. The MPD box has a default route and only has 400 routes or so. > >A steady stream of those messages, upwards of 500 per second. > >got message of size 96 on Fri Jun 27 22:25:42 2008 >RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno >0, flags: >locks: inits: >sockaddrs: > default > >got message of size 96 on Fri Jun 27 22:25:42 2008 >RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno >0, flags: >locks: inits: >sockaddrs: > default > >Is there a way to try and track down what is generating those messages >? Its eating up a fair bit of cpu with quagga (the zebra process >specifically) I narrowed down where the change to RELENG_7 happened. It looks like a commit around April 22nd caused the behaviour to change. When a box acting as a router has a packet transit it, an RTM_MISS is generated for *each packet*... Given a setup of H1 ---- R1 ----- H2 where H1 is 10.10.1.2/24 H2 is 10.20.1.2/24 and R1 has 2 interfaces, 10.10.1.1/24 and 10.20.1.1/24 Pinging H2 from H1 makes R1 generate a RTM_MISS for each packet! For routing daemons such as zebra, this eats up a *lot* of CPU. Turning on ip_fast_forwarding stops this behaviour on R1. However, if the interface routing the packet is an netgraph interface (e.g. mpd) fast_forwarding doesnt seem to have an effect and the RTM_MISS messages are generated again for each packet. The ping packet below is a valid icmp echo request and reply. e.g 0[releng7]# ping -c 2 -S 10.20.1.2 10.10.1.2 PING 10.10.1.2 (10.10.1.2) from 10.20.1.2: 56 data bytes 64 bytes from 10.10.1.2: icmp_seq=0 ttl=63 time=0.302 ms 64 bytes from 10.10.1.2: icmp_seq=1 ttl=63 time=0.337 ms --- 10.10.1.2 ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.302/0.320/0.337/0.018 ms 0[releng7]# generates 4 messages on the router [r7-router]# route -n monitor got message of size 96 on Tue Jul 1 00:42:35 2008 RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 96 on Tue Jul 1 00:42:35 2008 RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 96 on Tue Jul 1 00:42:36 2008 RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default got message of size 96 on Tue Jul 1 00:42:36 2008 RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno 0, flags: locks: inits: sockaddrs: default I am thinking http://lists.freebsd.org/pipermail/cvs-src/2008-April/090303.html is the commit ? If I revert to the prev version, the issue goes away. kernel is just 0[r7-router]% diff router GENERIC 24,27c24 < ident router < < makeoptions MODULES_OVERRIDE="ipfw acpi" < --- > ident GENERIC 37,38c34,35 < #options INET6 # IPv6 communications protocols < #options SCTP # Stream Control Transmission Protocol --- > options INET6 # IPv6 communications protocols > options SCTP # Stream Control Transmission Protocol 47c44 < #options NFSLOCKD # Network Lock Manager --- > options NFSLOCKD # Network Lock Manager 61c58 < #options STACK # stack(9) support --- > options STACK # stack(9) support 303c300 < #device uslcom # SI Labs CP2101/CP2102 serial adapters --- > device uslcom # SI Labs CP2101/CP2102 serial adapters ---Mike From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 06:24:00 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 98424106564A; Tue, 1 Jul 2008 06:24:00 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 58CCF8FC24; Tue, 1 Jul 2008 06:24:00 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDZE7-0006ye-Ir; Tue, 01 Jul 2008 02:20:31 -0400 Message-ID: <4869CDFA.3090800@gtcomm.net> Date: Tue, 01 Jul 2008 02:26:02 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Mike Tancsa References: <4852E23E.2040505@gtcomm.net> <4854EBF1.7020708@FreeBSD.org> <200807010606.m6166jFe084204@lava.sentex.ca> In-Reply-To: <200807010606.m6166jFe084204@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, bz@freebsd.org, "Bruce M. Simpson" Subject: Re: Route messages X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 06:24:00 -0000 Turning on / off fastforwarding has no effect for me. I still get the messages. I also get major ticks of 'destinations found unreachable' in netstat -rs Mike Tancsa wrote: > At 10:34 PM 6/27/2008, mike@sentex.net wrote: >> On Sun, 15 Jun 2008 11:16:17 +0100, in sentex.lists.freebsd.net you >> wrote: >> >> >Paul wrote: >> >> Get these with GRE tunnel on >> >> FreeBSD 7.0-STABLE FreeBSD 7.0-STABLE #5: Sun May 11 19:00:57 EDT >> >> 2008 :/usr/obj/usr/src/sys/ROUTER amd64 >> >> But do not get them with 7.0-RELEASE >> >> >> >> Any ideas what changed? :) Wish there was some sort of changelog.. >> >> # of messages per second seems consistent with packets per second on >> >> GRE interface.. >> >> No impact in routing, but definitely impact in cpu usage for all >> >> processes monitoring the route messages. >> > >> >RTM_MISS is actually fairly common when you don't have a default route. >> > >> >> Hi, >> I am seeing this issue as well on a pair of recently deployed >> boxes, one running MPD and one acting as an area router in front of >> it. The MPD box has a default route and only has 400 routes or so. >> >> A steady stream of those messages, upwards of 500 per second. >> >> got message of size 96 on Fri Jun 27 22:25:42 2008 >> RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno >> 0, flags: >> locks: inits: >> sockaddrs: >> default >> >> got message of size 96 on Fri Jun 27 22:25:42 2008 >> RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno >> 0, flags: >> locks: inits: >> sockaddrs: >> default >> >> Is there a way to try and track down what is generating those messages >> ? Its eating up a fair bit of cpu with quagga (the zebra process >> specifically) > > I narrowed down where the change to RELENG_7 happened. It looks like > a commit around April 22nd caused the behaviour to change. > > When a box acting as a router has a packet transit it, an RTM_MISS is > generated for *each packet*... > > > Given a setup of > > H1 ---- R1 ----- H2 > > where > H1 is 10.10.1.2/24 > H2 is 10.20.1.2/24 > and > R1 has 2 interfaces, 10.10.1.1/24 and 10.20.1.1/24 > > Pinging H2 from H1 makes R1 generate a RTM_MISS for each packet! For > routing daemons such as zebra, this eats up a *lot* of CPU. Turning > on ip_fast_forwarding stops this behaviour on R1. However, if the > interface routing the packet is an netgraph interface (e.g. mpd) > fast_forwarding doesnt seem to have an effect and the RTM_MISS > messages are generated again for each packet. > > > The ping packet below is a valid icmp echo request and reply. > > e.g > 0[releng7]# ping -c 2 -S 10.20.1.2 10.10.1.2 > PING 10.10.1.2 (10.10.1.2) from 10.20.1.2: 56 data bytes > 64 bytes from 10.10.1.2: icmp_seq=0 ttl=63 time=0.302 ms > 64 bytes from 10.10.1.2: icmp_seq=1 ttl=63 time=0.337 ms > > --- 10.10.1.2 ping statistics --- > 2 packets transmitted, 2 packets received, 0.0% packet loss > round-trip min/avg/max/stddev = 0.302/0.320/0.337/0.018 ms > 0[releng7]# > > generates 4 messages on the router > > [r7-router]# route -n monitor > > got message of size 96 on Tue Jul 1 00:42:35 2008 > RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > default > > got message of size 96 on Tue Jul 1 00:42:35 2008 > RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > default > > got message of size 96 on Tue Jul 1 00:42:36 2008 > RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > default > > got message of size 96 on Tue Jul 1 00:42:36 2008 > RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno > 0, flags: > locks: inits: > sockaddrs: > default > > > > I am thinking > > http://lists.freebsd.org/pipermail/cvs-src/2008-April/090303.html > is the commit ? If I revert to the prev version, the issue goes away. > > > kernel is just > > 0[r7-router]% diff router GENERIC > 24,27c24 > < ident router > < > < makeoptions MODULES_OVERRIDE="ipfw acpi" > < > --- > > ident GENERIC > 37,38c34,35 > < #options INET6 # IPv6 communications protocols > < #options SCTP # Stream Control Transmission > Protocol > --- > > options INET6 # IPv6 communications protocols > > options SCTP # Stream Control Transmission > Protocol > 47c44 > < #options NFSLOCKD # Network Lock Manager > --- > > options NFSLOCKD # Network Lock Manager > 61c58 > < #options STACK # stack(9) support > --- > > options STACK # stack(9) support > 303c300 > < #device uslcom # SI Labs CP2101/CP2102 serial > adapters > --- > > device uslcom # SI Labs CP2101/CP2102 serial > adapters > > > ---Mike > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 06:32:10 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D1B4B1065681; Tue, 1 Jul 2008 06:32:10 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id 999578FC12; Tue, 1 Jul 2008 06:32:10 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.2/8.14.2) with ESMTP id m616W8II064089; Tue, 1 Jul 2008 02:32:08 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m616W7i2084311 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 1 Jul 2008 02:32:07 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200807010632.m616W7i2084311@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 01 Jul 2008 02:32:15 -0400 To: Paul From: Mike Tancsa In-Reply-To: <4869CDFA.3090800@gtcomm.net> References: <4852E23E.2040505@gtcomm.net> <4854EBF1.7020708@FreeBSD.org> <200807010606.m6166jFe084204@lava.sentex.ca> <4869CDFA.3090800@gtcomm.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: freebsd-net@freebsd.org, bz@freebsd.org, "Bruce M. Simpson" Subject: Re: Route messages X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 06:32:10 -0000 At 02:26 AM 7/1/2008, Paul wrote: >Turning on / off fastforwarding has no effect for me. I still get >the messages. >I also get major ticks of 'destinations found unreachable' in netstat -rs if you use http://www.freebsd.org/cgi/cvsweb.cgi/~checkout~/src/sys/netinet/ip_input.c?rev=1.332.2.1;content-type=text%2Fplain does it fix it for you ? I just cvsup'd to a RELENG_7 as of today, but used the older version of ip_input.c and I no longer get the blast of RTM_MISS messages ---Mike >Mike Tancsa wrote: >>At 10:34 PM 6/27/2008, mike@sentex.net wrote: >>>On Sun, 15 Jun 2008 11:16:17 +0100, in sentex.lists.freebsd.net you >>>wrote: >>> >>> >Paul wrote: >>> >> Get these with GRE tunnel on >>> >> FreeBSD 7.0-STABLE FreeBSD 7.0-STABLE #5: Sun May 11 19:00:57 EDT >>> >> 2008 :/usr/obj/usr/src/sys/ROUTER amd64 >>> >> But do not get them with 7.0-RELEASE >>> >> >>> >> Any ideas what changed? :) Wish there was some sort of changelog.. >>> >> # of messages per second seems consistent with packets per second on >>> >> GRE interface.. >>> >> No impact in routing, but definitely impact in cpu usage for all >>> >> processes monitoring the route messages. >>> > >>> >RTM_MISS is actually fairly common when you don't have a default route. >>> > >>> >>>Hi, >>> I am seeing this issue as well on a pair of recently deployed >>>boxes, one running MPD and one acting as an area router in front of >>>it. The MPD box has a default route and only has 400 routes or so. >>> >>>A steady stream of those messages, upwards of 500 per second. >>> >>>got message of size 96 on Fri Jun 27 22:25:42 2008 >>>RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno >>>0, flags: >>>locks: inits: >>>sockaddrs: >>> default >>> >>>got message of size 96 on Fri Jun 27 22:25:42 2008 >>>RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, errno >>>0, flags: >>>locks: inits: >>>sockaddrs: >>> default >>> >>>Is there a way to try and track down what is generating those messages >>>? Its eating up a fair bit of cpu with quagga (the zebra process >>>specifically) >> >>I narrowed down where the change to RELENG_7 happened. It looks >>like a commit around April 22nd caused the behaviour to change. >> >>When a box acting as a router has a packet transit it, an RTM_MISS >>is generated for *each packet*... >> >> >>Given a setup of >> >>H1 ---- R1 ----- H2 >> >>where >>H1 is 10.10.1.2/24 >>H2 is 10.20.1.2/24 >>and >>R1 has 2 interfaces, 10.10.1.1/24 and 10.20.1.1/24 >> >>Pinging H2 from H1 makes R1 generate a RTM_MISS for each >>packet! For routing daemons such as zebra, this eats up a *lot* of >>CPU. Turning on ip_fast_forwarding stops this behaviour on >>R1. However, if the interface routing the packet is an netgraph >>interface (e.g. mpd) fast_forwarding doesnt seem to have an effect >>and the RTM_MISS messages are generated again for each packet. >> >> >>The ping packet below is a valid icmp echo request and reply. >> >>e.g >>0[releng7]# ping -c 2 -S 10.20.1.2 10.10.1.2 >>PING 10.10.1.2 (10.10.1.2) from 10.20.1.2: 56 data bytes >>64 bytes from 10.10.1.2: icmp_seq=0 ttl=63 time=0.302 ms >>64 bytes from 10.10.1.2: icmp_seq=1 ttl=63 time=0.337 ms >> >>--- 10.10.1.2 ping statistics --- >>2 packets transmitted, 2 packets received, 0.0% packet loss >>round-trip min/avg/max/stddev = 0.302/0.320/0.337/0.018 ms >>0[releng7]# >> >>generates 4 messages on the router >> >>[r7-router]# route -n monitor >> >>got message of size 96 on Tue Jul 1 00:42:35 2008 >>RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, >>errno 0, flags: >>locks: inits: >>sockaddrs: >> default >> >>got message of size 96 on Tue Jul 1 00:42:35 2008 >>RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, >>errno 0, flags: >>locks: inits: >>sockaddrs: >> default >> >>got message of size 96 on Tue Jul 1 00:42:36 2008 >>RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, >>errno 0, flags: >>locks: inits: >>sockaddrs: >> default >> >>got message of size 96 on Tue Jul 1 00:42:36 2008 >>RTM_MISS: Lookup failed on this address: len 96, pid: 0, seq 0, >>errno 0, flags: >>locks: inits: >>sockaddrs: >> default >> >> >> >>I am thinking >> >>http://lists.freebsd.org/pipermail/cvs-src/2008-April/090303.html >>is the commit ? If I revert to the prev version, the issue goes away. >> >> >>kernel is just >> >>0[r7-router]% diff router GENERIC >>24,27c24 >>< ident router >>< >>< makeoptions MODULES_OVERRIDE="ipfw acpi" >>< >>--- >> > ident GENERIC >>37,38c34,35 >>< #options INET6 # IPv6 communications protocols >>< #options SCTP # Stream Control >>Transmission Protocol >>--- >> > options INET6 # IPv6 communications protocols >> > options SCTP # Stream Control >> Transmission Protocol >>47c44 >>< #options NFSLOCKD # Network Lock Manager >>--- >> > options NFSLOCKD # Network Lock Manager >>61c58 >>< #options STACK # stack(9) support >>--- >> > options STACK # stack(9) support >>303c300 >>< #device uslcom # SI Labs CP2101/CP2102 >>serial adapters >>--- >> > device uslcom # SI Labs CP2101/CP2102 >> serial adapters >> >> >> ---Mike >>_______________________________________________ >>freebsd-net@freebsd.org mailing list >>http://lists.freebsd.org/mailman/listinfo/freebsd-net >>To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > >_______________________________________________ >freebsd-net@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-net >To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 08:34:38 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B7F061065679 for ; Tue, 1 Jul 2008 08:34:38 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 27D868FC0A for ; Tue, 1 Jul 2008 08:34:37 +0000 (UTC) (envelope-from andre@freebsd.org) Received: (qmail 54037 invoked from network); 1 Jul 2008 07:26:14 -0000 Received: from localhost (HELO [127.0.0.1]) ([127.0.0.1]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 1 Jul 2008 07:26:14 -0000 Message-ID: <4869EC1E.8060009@freebsd.org> Date: Tue, 01 Jul 2008 10:34:38 +0200 From: Andre Oppermann User-Agent: Thunderbird 1.5.0.14 (Windows/20071210) MIME-Version: 1.0 To: Mike Tancsa References: <4852E23E.2040505@gtcomm.net> <4854EBF1.7020708@FreeBSD.org> <200807010606.m6166jFe084204@lava.sentex.ca> In-Reply-To: <200807010606.m6166jFe084204@lava.sentex.ca> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, bz@freebsd.org, "Bruce M. Simpson" , Paul Subject: Re: Route messages X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 08:34:38 -0000 Mike Tancsa wrote: > I am thinking > > http://lists.freebsd.org/pipermail/cvs-src/2008-April/090303.html > is the commit ? If I revert to the prev version, the issue goes away. Yes, this change doesn't look right. It should only do the route lookup in ip_input.c when there was an EMSGSIZE error returned by ip_output(). The rtalloc_ign() call causes the message to be sent because it always sets report to one. The default message is RTM_MISS. I'll try to prep an updated patch which doesn't have these issues later today. -- Andre From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 08:40:51 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 232C61065670 for ; Tue, 1 Jul 2008 08:40:51 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 937498FC14 for ; Tue, 1 Jul 2008 08:40:50 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id AA1461B10EDC; Tue, 1 Jul 2008 10:40:49 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on malcho.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.4 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 404061B10EF2; Tue, 1 Jul 2008 10:40:44 +0200 (CEST) Message-ID: <4869ED8B.80508@moneybookers.com> Date: Tue, 01 Jul 2008 11:40:43 +0300 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.14 (X11/20080616) MIME-Version: 1.0 To: Andrew Thompson References: <4868A34C.6030304@moneybookers.com> <20080630101629.GD79537@cdnetworks.co.kr> <20080701012531.GA92392@citylink.fud.org.nz> In-Reply-To: <20080701012531.GA92392@citylink.fud.org.nz> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on blah.cmotd.com X-Virus-Status: Clean Cc: Pyun YongHyeon , freebsd-net@freebsd.org Subject: Re: if_bridge turns off checksum offload of members? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 08:40:51 -0000 Andrew Thompson wrote: > On Mon, Jun 30, 2008 at 07:16:29PM +0900, Pyun YongHyeon wrote: > >> On Mon, Jun 30, 2008 at 12:11:40PM +0300, Stefan Lambrev wrote: >> > Greetings, >> > >> > I just noticed, that when I add em network card to bridge the checksum >> > offload is turned off. >> > I even put in my rc.conf: >> > ifconfig_em0="rxcsum up" >> > ifconfig_em1="rxcsum up" >> > but after reboot both em0 and em1 have this feature disabled. >> > >> > Is this expected behavior? Should I care about csum in bridge mode? >> > I noticed that enabling checksum offload manually improve things little btw. >> > >> >> AFAIK this is intended one, bridge(4) turns off Tx side checksum >> offload by default. I think disabling Tx checksum offload is >> required as not all members of a bridge may be able to do checksum >> offload. The same is true for TSO but it seems that bridge(4) >> doesn't disable it. >> If all members of bridge have the same hardware capability I think >> bridge(4) may not need to disable Tx side hardware assistance. I >> guess bridge(4) can scan every interface capabilities in a member >> and can decide what hardware assistance can be activated instead of >> blindly turning off Tx side hardware assistance. >> > > This patch should do that, are you able to test it Stefan? > ===> if_bridge (all) cc -O2 -fno-strict-aliasing -pipe -march=nocona -D_KERNEL -DKLD_MODULE -std=c99 -nostdinc -DHAVE_KERNEL_OPTION_HEADERS -include /usr/obj/usr/src/sys/CORE/opt_global.h -I. -I@ -I@/contrib/altq -finline-limit=8000 --param inline-unit-growth=100 --param large-function-growth=1000 -fno-common -g -fno-omit-frame-pointer -I/usr/obj/usr/src/sys/CORE -mcmodel=kernel -mno-red-zone -mfpmath=387 -mno-sse -mno-sse2 -mno-mmx -mno-3dnow -msoft-float -fno-asynchronous-unwind-tables -ffreestanding -Wall -Wredundant-decls -Wnested-externs -Wstrict-prototypes -Wmissing-prototypes -Wpointer-arith -Winline -Wcast-qual -Wundef -Wno-pointer-sign -fformat-extensions -c /usr/src/sys/modules/if_bridge/../../net/if_bridge.c /usr/src/sys/modules/if_bridge/../../net/if_bridge.c: In function 'bridge_capabilities': /usr/src/sys/modules/if_bridge/../../net/if_bridge.c:787: error: 'IFCAP_TOE' undeclared (first use in this function) /usr/src/sys/modules/if_bridge/../../net/if_bridge.c:787: error: (Each undeclared identifier is reported only once /usr/src/sys/modules/if_bridge/../../net/if_bridge.c:787: error: for each function it appears in.) *** Error code 1 1 error *** Error code 2 1 error *** Error code 2 1 error *** Error code 2 1 error *** Error code 2 1 error I'm building without "-j5" to see if the error message will change :) I'm using 7-STABLE from Jun 27 > > cheers, > Andrew > > ------------------------------------------------------------------------ > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 08:49:00 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B4361065671 for ; Tue, 1 Jul 2008 08:49:00 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 070398FC1C for ; Tue, 1 Jul 2008 08:49:00 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 417F01B10EE7; Tue, 1 Jul 2008 10:48:59 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on malcho.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.4 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id DFAFD1B10EA4; Tue, 1 Jul 2008 10:48:56 +0200 (CEST) Message-ID: <4869EF78.5060306@moneybookers.com> Date: Tue, 01 Jul 2008 11:48:56 +0300 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.14 (X11/20080616) MIME-Version: 1.0 To: Steve Bertrand References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <4869880D.8040901@ibctech.ca> In-Reply-To: <4869880D.8040901@ibctech.ca> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-net@freebsd.org, Ingo Flaschberger , "Wilkinson, Alex" , "Support \(Rudy\)" Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 08:49:00 -0000 Steve Bertrand wrote: > Support (Rudy) wrote: >> Ingo Flaschberger wrote: >>> usually interface polling is also chosen to prevent "lock-ups". >>> man polling >> >> >> I used polling in FreeBSD 5.x and it helped a bunch. I set up a new >> router with 7.0 and MSI was recommended to me. (I noticed no >> difference when moving from polling -> MSI, however, on 5.4 polling >> seemed to help a lot. > > I'm curious now... how do you change individual device polling via > sysctl? Using sysctl for polling is deprecated I think. You can do it with ifconfig ifX polling (-polling) you can add polling in rc.conf options also: ifconfig_em0="polling up" #bridged interface in my conf > > Steve > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 08:53:30 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1DF8F1065689 for ; Tue, 1 Jul 2008 08:53:30 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id C05C78FC25 for ; Tue, 1 Jul 2008 08:53:29 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id F3B471B10EE0; Tue, 1 Jul 2008 10:53:28 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on malcho.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.4 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 746071B10E4E; Tue, 1 Jul 2008 10:53:23 +0200 (CEST) Message-ID: <4869F082.9010606@moneybookers.com> Date: Tue, 01 Jul 2008 11:53:22 +0300 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.14 (X11/20080616) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> In-Reply-To: Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-net@freebsd.org, "Wilkinson, Alex" , "Support \(Rudy\)" Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 08:53:30 -0000 Hi, Ingo Flaschberger wrote: > Dear Rudy, > >> I used polling in FreeBSD 5.x and it helped a bunch. I set up a new >> router with 7.0 and MSI was recommended to me. (I noticed no >> difference when moving from polling -> MSI, however, on 5.4 polling >> seemed to help a lot. What are people using in 7.0? >> polling or MSI? > > if you have a inet-router with gige-uplinks, it is possible that there > will be (d)dos attacks. > only polling helps you then to keep the router manageable (but > dropping packets). Let me disagree :) I'm experimenting with bridge and Intel 82571EB Gigabit Ethernet Controller. On quad core system I have no problems with the stability of the bridge without polling. taskq em0 takes 100% CPU, but I have another three (cpus/cores) that are free and the router is very very stable, no lag on other interfaces and the average load is not very high too. > > Kind regards, > Ingo Flaschberger > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 09:06:59 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 97BA2106568A for ; Tue, 1 Jul 2008 09:06:59 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 5622A8FC22 for ; Tue, 1 Jul 2008 09:06:59 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDblq-0001PR-0B for freebsd-net@freebsd.org; Tue, 01 Jul 2008 05:03:30 -0400 Message-ID: <4869F42E.8040904@gtcomm.net> Date: Tue, 01 Jul 2008 05:09:02 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: FreeBSD Net References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> In-Reply-To: <4869B025.9080006@gtcomm.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 09:06:59 -0000 [Big list of testing , rebuilding kernel follows] Dual Opteron 2212, Recompiled kernel with 7-STABLE and removed a lot of junk in the config, added options NO_ADAPTIVE_MUTEXES not sure if that makes any difference or not, will test without. Used ULE scheduler, used preemption, CPUTYPE=opteron in /etc/make.conf 7.0-STABLE FreeBSD 7.0-STABLE #4: Tue Jul 1 01:22:18 CDT 2008 amd64 Max input rate .. 587kpps? Take into consideration that these packets are being forwarded out em1 interface which causes a great impact on cpu usage. If I set up a firewall rule to block the packets it can do over 1mpps on em0 input. input (em0) output packets errs bytes packets errs bytes colls 587425 67677 35435456 466 0 25616 0 587412 26629 35434766 453 0 24866 0 587043 26874 35412442 410 0 22544 0 536117 30264 32347300 440 0 24164 0 546240 61521 32951060 459 0 25350 0 563568 66881 33998676 435 0 23894 0 572766 43243 34550840 440 0 24164 0 572336 44411 34525836 445 0 24558 0 572539 37013 34536222 457 0 25136 0 571340 39512 34459008 440 0 24110 0 572673 55137 34540576 438 0 24056 0 555506 49918 33505764 457 0 25330 0 545744 69010 32916908 461 0 25298 0 559472 75650 33745636 429 0 23694 0 564358 60130 34039104 433 0 23786 0 last pid: 1134; load averages: 1.04, 0.94, 0.59 up 0+00:14:13 01:49:59 70 processes: 6 running, 46 sleeping, 17 waiting, 1 lock CPU: 0.0% user, 0.0% nice, 25.6% system, 0.0% interrupt, 74.4% idle Mem: 11M Active, 6596K Inact, 45M Wired, 156K Cache, 9072K Buf, 1917M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 12 root 171 ki31 0K 16K RUN 1 12:40 97.56% idle: cpu1 36 root -68 - 0K 16K *em1 2 9:44 85.06% em0 taskq 10 root 171 ki31 0K 16K CPU3 3 11:10 82.47% idle: cpu3 13 root 171 ki31 0K 16K CPU0 0 12:25 73.88% idle: cpu0 11 root 171 ki31 0K 16K RUN 2 6:43 50.10% idle: cpu2 37 root -68 - 0K 16K CPU3 3 1:58 16.46% em1 taskq I noticed.. em0 taskq isn't using 100% cpu like it was on the generic kernel.. What's up with that? Why do I still have all 4 CPUs pretty idle and em0 taskq isn't near 100%? I'm going to try 4bsd and see if that makes it go back to the other way. em0: Excessive collisions = 0 em0: Sequence errors = 0 em0: Defer count = 0 em0: Missed Packets = 45395545 em0: Receive No Buffers = 95916690 em0: Receive Length Errors = 0 em0: Receive errors = 0 em0: Crc errors = 0 em0: Alignment errors = 0 em0: Collision/Carrier extension errors = 0 em0: RX overruns = 2740181 em0: watchdog timeouts = 0 em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0 em0: XON Rcvd = 0 em0: XON Xmtd = 0 em0: XOFF Rcvd = 0 em0: XOFF Xmtd = 0 em0: Good Packets Rcvd = 450913688 em0: Good Packets Xmtd = 304777 em0: TSO Contexts Xmtd = 94 em0: TSO Contexts Failed = 0 -----Rebooting with: kern.hz=2000 hw.em.rxd=512 hw.em.txd=512 Seems maybe a little bit slower but it's hard to tell since i'm generating random packets the pps varies about 50k +/- probably depending on the randomness.. About the same PPS/errors.. here's a vmstat 1 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us sy id 0 0 1 52276K 1922M 286 0 1 0 277 0 0 0 7686 838 19436 0 15 85 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13431 127 33430 0 27 73 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13406 115 33222 0 27 73 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13430 115 33393 0 26 74 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13411 115 33322 0 26 74 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13576 123 33415 0 25 75 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13842 115 33354 0 26 74 ------Trying kern.kz=250 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us sy id 0 0 1 52288K 1923M 607 1 2 0 582 0 0 0 4885 789 12073 0 8 92 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13793 119 33552 0 27 73 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13959 115 33446 0 26 74 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13861 115 33707 0 30 70 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13784 115 33602 0 26 74 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13886 123 33843 0 26 74 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13913 115 33711 0 26 74 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13920 115 33766 0 27 73 pps still no major difference.. jumps between 530k-580k -----Putting HZ back to 1000, recompiling kernel with 4BSD SCHED.. many minutes later.. (can't do make -j with the kernel or it errors) Well, I have to say.. 4BSD is less pps, it will not go over 530k however it seems much, more consistent and not jumping around as much it stays between 520-530 most of the time and i see some ticks at 480's in netstat.. em0 taskq still not using 100%, max around 75-80 -----Building same as above but with preemption off procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us sy id 0 0 0 52288K 1922M 563 1 2 0 540 0 0 0 6724 725 22195 0 12 88 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13200 119 48075 0 27 73 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13243 123 49137 0 24 76 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13260 115 48633 0 26 74 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13247 115 48625 0 25 75 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13248 115 48687 0 24 76 hmm more context switches.. pps same, maybe a shade lower.. PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 171 ki31 0K 16K RUN 2 3:39 97.12% idle: cpu2 12 root 171 ki31 0K 16K CPU1 1 3:45 95.70% idle: cpu1 36 root -68 - 0K 16K CPU0 0 2:18 82.67% em0 taskq 10 root 171 ki31 0K 16K CPU3 3 3:24 82.57% idle: cpu3 13 root 171 ki31 0K 16K RUN 0 2:01 20.07% idle: cpu0 37 root -68 - 0K 16K - 3 0:31 15.58% em1 taskq -------rebuilding with ULE, keeping preemption off Hmm.. what the? 450-480kpps seems to be max here. That's.. weird.. I'm going to have to rebuild with Preemption on again just to double check this.. input (em0) output packets errs bytes packets errs bytes colls 464020 95690 28009004 434 0 23728 0 455318 90105 27484456 469 0 25778 0 455720 99914 27511970 462 0 25384 0 465019 86021 28071946 428 0 23392 0 456024 78336 27528862 440 0 24040 0 455018 93526 27468908 440 0 24040 0 461235 91218 27841604 464 0 25336 0 454345 89812 27427262 424 0 23176 0 452661 96937 27327392 441 0 24094 0 456584 90393 27561138 459 0 25222 0 455021 97441 27470158 450 0 24736 0 procs memory page disks faults cpu r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us sy id 0 0 1 52276K 1655M 456 1 1 0 441 0 0 0 9775 3598 26256 0 20 80 0 0 0 52276K 1655M 0 0 0 0 0 0 0 0 12817 119 33056 0 25 75 0 0 0 52276K 1655M 0 0 0 0 0 0 0 0 12700 123 32975 0 27 73 0 0 0 52276K 1655M 0 0 0 0 0 0 0 0 12659 115 32897 0 27 73 ------OK I'm stumped now.. Rebuilt with preemption and ULE and preemption again and it's not doing what it did before.. How could that be? Now about 500kpps.. That kind of inconsistency almost invalidates all my testing.. why would it be so much different after trying a bunch of kernel options and rebooting a bunch of times and then going back to the original config doesn't get you what it did in the beginning.. I'll have to dig into this further.. never seen anything like it :) Hopefully the ip_input fix will help free up a few cpu cycles. From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 09:09:48 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7259C1065673 for ; Tue, 1 Jul 2008 09:09:48 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.233]) by mx1.freebsd.org (Postfix) with ESMTP id 3E1238FC23 for ; Tue, 1 Jul 2008 09:09:48 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so2061251rvf.43 for ; Tue, 01 Jul 2008 02:09:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender :to:subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references :x-google-sender-auth; bh=GynRNili3n2tC6lMrAWNdIUuQbY2RKyDMsTPkoEqCXA=; b=TXNQfFUZuxwCjs79LzXrw9qEw67EGLnyjT9tIdyCWNH8zixqRkpGB4J7/MAXQsr5cF QS9cCxAJzN0IxtiiZpiPIVaQF/J54AdtIhqiW02h6A4RrjFrBHboCEWPzH9ZXuKs+BZ6 qohb1KBByQVFwYzVm1o82cHGOxlUbW0llYo0Q= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references:x-google-sender-auth; b=OOtg9cGdvInJWaKI8T9U67ivlTc/WrSq0IR2Vkw1W/EwPjVGb7DpnCScRy76JfAw1e lvjWj2POIBbI4B6y2dUDSF7IslsCDBhW5e1LjEe6HNGurj2aKe4NpTLe0NrznsJZ2xnY e+6CAsL728ucvkf9Mivko6TDS6oXwtQzfUntA= Received: by 10.141.198.2 with SMTP id a2mr3302532rvq.219.1214903387892; Tue, 01 Jul 2008 02:09:47 -0700 (PDT) Received: by 10.141.212.9 with HTTP; Tue, 1 Jul 2008 02:09:47 -0700 (PDT) Message-ID: Date: Tue, 1 Jul 2008 17:09:47 +0800 From: "Adrian Chadd" Sender: adrian.chadd@gmail.com To: Paul In-Reply-To: <4869F42E.8040904@gtcomm.net> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4867420D.7090406@gtcomm.net> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <4869F42E.8040904@gtcomm.net> X-Google-Sender-Auth: 8626adbe26cff990 Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 09:09:48 -0000 There's an option to control how many packets it'll process each pass through the isr thread, isn't there? It'd be nicer if this stuff were able to be dynamically tuned. Adrian 2008/7/1 Paul : > [Big list of testing , rebuilding kernel follows] > > Dual Opteron 2212, Recompiled kernel with 7-STABLE and removed a lot of junk > in the config, added > options NO_ADAPTIVE_MUTEXES not sure if that makes any difference > or not, will test without. > Used ULE scheduler, used preemption, CPUTYPE=opteron in /etc/make.conf > 7.0-STABLE FreeBSD 7.0-STABLE #4: Tue Jul 1 01:22:18 CDT 2008 amd64 > Max input rate .. 587kpps? Take into consideration that these packets are > being forwarded out em1 interface which > causes a great impact on cpu usage. If I set up a firewall rule to block > the packets it can do over 1mpps on em0 input. > > input (em0) output > packets errs bytes packets errs bytes colls > 587425 67677 35435456 466 0 25616 0 > 587412 26629 35434766 453 0 24866 0 > 587043 26874 35412442 410 0 22544 0 > 536117 30264 32347300 440 0 24164 0 > 546240 61521 32951060 459 0 25350 0 > 563568 66881 33998676 435 0 23894 0 > 572766 43243 34550840 440 0 24164 0 > 572336 44411 34525836 445 0 24558 0 > 572539 37013 34536222 457 0 25136 0 > 571340 39512 34459008 440 0 24110 0 > 572673 55137 34540576 438 0 24056 0 > 555506 49918 33505764 457 0 25330 0 > 545744 69010 32916908 461 0 25298 0 > 559472 75650 33745636 429 0 23694 0 > 564358 60130 34039104 433 0 23786 0 > > last pid: 1134; load averages: 1.04, 0.94, 0.59 > up 0+00:14:13 01:49:59 > 70 processes: 6 running, 46 sleeping, 17 waiting, 1 lock > CPU: 0.0% user, 0.0% nice, 25.6% system, 0.0% interrupt, 74.4% idle > Mem: 11M Active, 6596K Inact, 45M Wired, 156K Cache, 9072K Buf, 1917M Free > Swap: 8192M Total, 8192M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 12 root 171 ki31 0K 16K RUN 1 12:40 97.56% idle: cpu1 > 36 root -68 - 0K 16K *em1 2 9:44 85.06% em0 taskq > 10 root 171 ki31 0K 16K CPU3 3 11:10 82.47% idle: cpu3 > 13 root 171 ki31 0K 16K CPU0 0 12:25 73.88% idle: cpu0 > 11 root 171 ki31 0K 16K RUN 2 6:43 50.10% idle: cpu2 > 37 root -68 - 0K 16K CPU3 3 1:58 16.46% em1 taskq > > > I noticed.. em0 taskq isn't using 100% cpu like it was on the generic > kernel.. What's up with that? Why do I still have all 4 CPUs pretty idle and > em0 taskq isn't near 100%? I'm going to try 4bsd and see > if that makes it go back to the other way. > > em0: Excessive collisions = 0 > em0: Sequence errors = 0 > em0: Defer count = 0 > em0: Missed Packets = 45395545 > em0: Receive No Buffers = 95916690 > em0: Receive Length Errors = 0 > em0: Receive errors = 0 > em0: Crc errors = 0 > em0: Alignment errors = 0 > em0: Collision/Carrier extension errors = 0 > em0: RX overruns = 2740181 > em0: watchdog timeouts = 0 > em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0 > em0: XON Rcvd = 0 > em0: XON Xmtd = 0 > em0: XOFF Rcvd = 0 > em0: XOFF Xmtd = 0 > em0: Good Packets Rcvd = 450913688 > em0: Good Packets Xmtd = 304777 > em0: TSO Contexts Xmtd = 94 > em0: TSO Contexts Failed = 0 > > -----Rebooting with: > kern.hz=2000 > hw.em.rxd=512 > hw.em.txd=512 > > Seems maybe a little bit slower but it's hard to tell since i'm generating > random packets the pps varies about 50k +/- probably depending > on the randomness.. About the same PPS/errors.. here's a vmstat 1 > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us > sy id > 0 0 1 52276K 1922M 286 0 1 0 277 0 0 0 7686 838 19436 0 > 15 85 > 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13431 127 33430 0 > 27 73 > 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13406 115 33222 0 > 27 73 > 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13430 115 33393 0 > 26 74 > 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13411 115 33322 0 > 26 74 > 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13576 123 33415 0 > 25 75 > 0 0 0 52276K 1922M 0 0 0 0 0 0 0 0 13842 115 33354 0 > 26 74 > > ------Trying kern.kz=250 > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us > sy id > 0 0 1 52288K 1923M 607 1 2 0 582 0 0 0 4885 789 12073 0 > 8 92 > 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13793 119 33552 0 > 27 73 > 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13959 115 33446 0 > 26 74 > 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13861 115 33707 0 > 30 70 > 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13784 115 33602 0 > 26 74 > 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13886 123 33843 0 > 26 74 > 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13913 115 33711 0 > 26 74 > 0 0 0 52288K 1923M 0 0 0 0 0 0 0 0 13920 115 33766 0 > 27 73 > > pps still no major difference.. > jumps between 530k-580k > > -----Putting HZ back to 1000, > recompiling kernel with 4BSD SCHED.. > many minutes later.. (can't do make -j with the kernel or it errors) > Well, I have to say.. 4BSD is less pps, it will not go over 530k however it > seems much, > more consistent and not jumping around as much it stays between 520-530 most > of the time and i see some ticks > at 480's in netstat.. > em0 taskq still not using 100%, max around 75-80 > > -----Building same as above but with preemption off > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us > sy id > 0 0 0 52288K 1922M 563 1 2 0 540 0 0 0 6724 725 22195 0 > 12 88 > 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13200 119 48075 0 > 27 73 > 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13243 123 49137 0 > 24 76 > 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13260 115 48633 0 > 26 74 > 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13247 115 48625 0 > 25 75 > 0 0 0 52288K 1922M 0 0 0 0 0 0 0 0 13248 115 48687 0 > 24 76 > > hmm more context switches.. > pps same, maybe a shade lower.. > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 11 root 171 ki31 0K 16K RUN 2 3:39 97.12% idle: cpu2 > 12 root 171 ki31 0K 16K CPU1 1 3:45 95.70% idle: cpu1 > 36 root -68 - 0K 16K CPU0 0 2:18 82.67% em0 taskq > 10 root 171 ki31 0K 16K CPU3 3 3:24 82.57% idle: cpu3 > 13 root 171 ki31 0K 16K RUN 0 2:01 20.07% idle: cpu0 > 37 root -68 - 0K 16K - 3 0:31 15.58% em1 taskq > > > -------rebuilding with ULE, keeping preemption off > Hmm.. what the? > 450-480kpps seems to be max here. That's.. weird.. > I'm going to have to rebuild with Preemption on again just to double check > this.. > input (em0) output > packets errs bytes packets errs bytes colls > 464020 95690 28009004 434 0 23728 0 > 455318 90105 27484456 469 0 25778 0 > 455720 99914 27511970 462 0 25384 0 > 465019 86021 28071946 428 0 23392 0 > 456024 78336 27528862 440 0 24040 0 > 455018 93526 27468908 440 0 24040 0 > 461235 91218 27841604 464 0 25336 0 > 454345 89812 27427262 424 0 23176 0 > 452661 96937 27327392 441 0 24094 0 > 456584 90393 27561138 459 0 25222 0 > 455021 97441 27470158 450 0 24736 0 > > procs memory page disks faults cpu > r b w avm fre flt re pi po fr sr ad4 ad6 in sy cs us > sy id > 0 0 1 52276K 1655M 456 1 1 0 441 0 0 0 9775 3598 26256 0 > 20 80 > 0 0 0 52276K 1655M 0 0 0 0 0 0 0 0 12817 119 33056 0 > 25 75 > 0 0 0 52276K 1655M 0 0 0 0 0 0 0 0 12700 123 32975 0 > 27 73 > 0 0 0 52276K 1655M 0 0 0 0 0 0 0 0 12659 115 32897 0 > 27 73 > > > ------OK I'm stumped now.. Rebuilt with preemption and ULE and preemption > again and it's not doing what it did before.. > How could that be? Now about 500kpps.. > > That kind of inconsistency almost invalidates all my testing.. why would it > be so much different after trying a bunch of kernel options and rebooting a > bunch of times and then going back to the original config doesn't get you > what it did in the beginning.. > > I'll have to dig into this further.. never seen anything like it :) > > Hopefully the ip_input fix will help free up a few cpu cycles. > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 09:14:01 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CAA3E1065671; Tue, 1 Jul 2008 09:14:00 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from mail.cksoft.de (mail.cksoft.de [62.111.66.27]) by mx1.freebsd.org (Postfix) with ESMTP id 120438FC23; Tue, 1 Jul 2008 09:13:59 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from localhost (amavis.str.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id 6C72141C7B1; Tue, 1 Jul 2008 10:55:05 +0200 (CEST) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([62.111.66.27]) by localhost (amavis.str.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id W0-Pz+bA67jl; Tue, 1 Jul 2008 10:55:05 +0200 (CEST) Received: by mail.cksoft.de (Postfix, from userid 66) id 174EB41C7B0; Tue, 1 Jul 2008 10:55:05 +0200 (CEST) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id 3C7EF44487F; Tue, 1 Jul 2008 08:51:30 +0000 (UTC) Date: Tue, 1 Jul 2008 08:51:28 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Andre Oppermann In-Reply-To: <4869EC1E.8060009@freebsd.org> Message-ID: <20080701084933.W57089@maildrop.int.zabbadoz.net> References: <4852E23E.2040505@gtcomm.net> <4854EBF1.7020708@FreeBSD.org> <200807010606.m6166jFe084204@lava.sentex.ca> <4869EC1E.8060009@freebsd.org> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org, Paul , "Bruce M. Simpson" , Mike Tancsa Subject: Re: Route messages X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 09:14:01 -0000 On Tue, 1 Jul 2008, Andre Oppermann wrote: Hi, > Mike Tancsa wrote: >> I am thinking >> >> http://lists.freebsd.org/pipermail/cvs-src/2008-April/090303.html >> is the commit ? If I revert to the prev version, the issue goes away. Ha, I finally know why I ended up on Cc: of a thread I had no idea about. Someone could have told me instead of blindly adding me;-) > Yes, this change doesn't look right. It should only do the route > lookup in ip_input.c when there was an EMSGSIZE error returned by > ip_output(). The rtalloc_ign() call causes the message to be sent > because it always sets report to one. The default message is RTM_MISS. > > I'll try to prep an updated patch which doesn't have these issues later > today. Yeah my bad. Sorry. If you do that, do not do an extra route lookup if possible, correct the rtalloc call. Thanks. Bjoern -- Bjoern A. Zeeb Stop bit received. Insert coin for new game. From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 09:14:48 2008 Return-Path: Delivered-To: net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 241F6106567F; Tue, 1 Jul 2008 09:14:48 +0000 (UTC) (envelope-from bms@FreeBSD.org) Received: from out3.smtp.messagingengine.com (out3.smtp.messagingengine.com [66.111.4.27]) by mx1.freebsd.org (Postfix) with ESMTP id E74E58FC27; Tue, 1 Jul 2008 09:14:47 +0000 (UTC) (envelope-from bms@FreeBSD.org) Received: from compute1.internal (compute1.internal [10.202.2.41]) by out1.messagingengine.com (Postfix) with ESMTP id 37EF31325BE; Tue, 1 Jul 2008 05:14:47 -0400 (EDT) Received: from heartbeat1.messagingengine.com ([10.202.2.160]) by compute1.internal (MEProxy); Tue, 01 Jul 2008 05:14:47 -0400 X-Sasl-enc: pg1O+wpjmR4ds5E47tjB60Xr0CQ1z9taMYSboiCqX0zU 1214903686 Received: from empiric.lon.incunabulum.net (82-35-112-254.cable.ubr07.dals.blueyonder.co.uk [82.35.112.254]) by mail.messagingengine.com (Postfix) with ESMTPSA id 767C713356; Tue, 1 Jul 2008 05:14:46 -0400 (EDT) Message-ID: <4869F586.7010708@FreeBSD.org> Date: Tue, 01 Jul 2008 10:14:46 +0100 From: "Bruce M. Simpson" User-Agent: Thunderbird 2.0.0.14 (X11/20080514) MIME-Version: 1.0 To: Robert Watson References: <20080524111715.T64552@fledge.watson.org> <20080629180126.F90836@fledge.watson.org> In-Reply-To: <20080629180126.F90836@fledge.watson.org> X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: arch@FreeBSD.org, current@FreeBSD.org, net@FreeBSD.org Subject: Re: HEAD UP: non-MPSAFE network drivers to be disabled (was: 8.0 network stack MPsafety goals (fwd)) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 09:14:48 -0000 Robert Watson wrote: > > An FYI on the state of things here: in the last month, John has > updated a number of device drivers to be MPSAFE, and the USB work > remains in-flight. I'm holding fire a bit on disabling IFF_NEEDSGIANT > while things settle and I catch up on driver state, and will likely > send out an update next week regarding which device drivers remain on > the kill list, and generally what the status of this project is. Goliath needs to get stoned, it's been a major hurdle in doing IGMPv3/SSM because of the locking fandango. I look forward to it. [For those who ask, what the hell? IGMPv3 potentially makes your wireless multicast better with or without little things like SSM, because of protocol robustness, compact state-changes, and the use of a single link-local IPv4 group for state-change reports, making it easier for your switches to actually do their job.] From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 09:17:28 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EC39B1065672 for ; Tue, 1 Jul 2008 09:17:28 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 9A2CC8FC16 for ; Tue, 1 Jul 2008 09:17:28 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 639471B10EE0; Tue, 1 Jul 2008 11:17:27 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on malcho.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.4 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id E99FC1B10CB6; Tue, 1 Jul 2008 11:17:21 +0200 (CEST) Message-ID: <4869F621.4000508@moneybookers.com> Date: Tue, 01 Jul 2008 12:17:21 +0300 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.14 (X11/20080616) MIME-Version: 1.0 To: Paul References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <4869F42E.8040904@gtcomm.net> In-Reply-To: <4869F42E.8040904@gtcomm.net> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on blah.cmotd.com X-Virus-Status: Clean Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 09:17:29 -0000 Greetings Paul, > > > ------OK I'm stumped now.. Rebuilt with preemption and ULE and > preemption again and it's not doing what it did before.. I saw this in my configuration too :) Just leave your test running for longer time and you will see this strange inconsistency in action. In my configuration I almost always have better throughput after reboot, which drops latter (5-10min under flood) with 50-60kpps and after another 10-15min the number of correctly passed packet increase again. Looks like "auto tuning" of which I'm not aware :) > How could that be? Now about 500kpps.. > > That kind of inconsistency almost invalidates all my testing.. why > would it be so much different after trying a bunch of kernel options > and rebooting a bunch of times and then going back to the original > config doesn't get you what it did in the beginning.. > > I'll have to dig into this further.. never seen anything like it :) > > Hopefully the ip_input fix will help free up a few cpu cycles. > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 09:25:06 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DE733106567F; Tue, 1 Jul 2008 09:25:06 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from mail.cksoft.de (mail.cksoft.de [62.111.66.27]) by mx1.freebsd.org (Postfix) with ESMTP id 946F48FC1C; Tue, 1 Jul 2008 09:25:06 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from localhost (amavis.str.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id BB33841C795; Tue, 1 Jul 2008 11:25:05 +0200 (CEST) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([62.111.66.27]) by localhost (amavis.str.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id oWqrVf1XDQB2; Tue, 1 Jul 2008 11:25:05 +0200 (CEST) Received: by mail.cksoft.de (Postfix, from userid 66) id 5619A41C758; Tue, 1 Jul 2008 11:25:05 +0200 (CEST) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id 60BB744487F; Tue, 1 Jul 2008 09:24:59 +0000 (UTC) Date: Tue, 1 Jul 2008 09:24:59 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Andre Oppermann In-Reply-To: <20080701084933.W57089@maildrop.int.zabbadoz.net> Message-ID: <20080701092254.T57089@maildrop.int.zabbadoz.net> References: <4852E23E.2040505@gtcomm.net> <4854EBF1.7020708@FreeBSD.org> <200807010606.m6166jFe084204@lava.sentex.ca> <4869EC1E.8060009@freebsd.org> <20080701084933.W57089@maildrop.int.zabbadoz.net> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org, Mike Tancsa , "Bruce M. Simpson" , Paul Subject: Re: Route messages X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 09:25:07 -0000 On Tue, 1 Jul 2008, Bjoern A. Zeeb wrote: Hi, > On Tue, 1 Jul 2008, Andre Oppermann wrote: > > Hi, > >> Mike Tancsa wrote: >>> I am thinking >>> >>> http://lists.freebsd.org/pipermail/cvs-src/2008-April/090303.html >>> is the commit ? If I revert to the prev version, the issue goes away. > > Ha, I finally know why I ended up on Cc: of a thread I had no idea > about. Someone could have told me instead of blindly adding me;-) > > >> Yes, this change doesn't look right. It should only do the route >> lookup in ip_input.c when there was an EMSGSIZE error returned by >> ip_output(). The rtalloc_ign() call causes the message to be sent >> because it always sets report to one. The default message is RTM_MISS. >> >> I'll try to prep an updated patch which doesn't have these issues later >> today. > > Yeah my bad. Sorry. > > If you do that, do not do an extra route lookup if possible, correct > the rtalloc call. Thanks. So I had a very quick look at the code between doing something else. I think the only change needed is this if I am not mistaken but my head is far away nowhere close enough in this code. Andre, could you review this? Index: sys/netinet/ip_input.c =================================================================== RCS file: /shared/mirror/FreeBSD/r/ncvs/src/sys/netinet/ip_input.c,v retrieving revision 1.332.2.2 diff -u -p -r1.332.2.2 ip_input.c --- sys/netinet/ip_input.c 22 Apr 2008 12:02:55 -0000 1.332.2.2 +++ sys/netinet/ip_input.c 1 Jul 2008 09:23:08 -0000 @@ -1363,7 +1363,6 @@ ip_forward(struct mbuf *m, int srcrt) * the ICMP_UNREACH_NEEDFRAG "Next-Hop MTU" field described in RFC1191. */ bzero(&ro, sizeof(ro)); - rtalloc_ign(&ro, RTF_CLONING); error = ip_output(m, NULL, &ro, IP_FORWARDING, NULL, NULL); -- Bjoern A. Zeeb Stop bit received. Insert coin for new game. From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 09:51:49 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E5106106567F; Tue, 1 Jul 2008 09:51:49 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 97C4F8FC22; Tue, 1 Jul 2008 09:51:49 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 8E31B1B10EE7; Tue, 1 Jul 2008 11:51:48 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on malcho.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.4 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id DF2F21B10ED2; Tue, 1 Jul 2008 11:51:42 +0200 (CEST) Message-ID: <4869FE2E.4070805@moneybookers.com> Date: Tue, 01 Jul 2008 12:51:42 +0300 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.14 (X11/20080616) MIME-Version: 1.0 To: Andrew Thompson References: <4868A34C.6030304@moneybookers.com> <20080630101629.GD79537@cdnetworks.co.kr> <20080701012531.GA92392@citylink.fud.org.nz> In-Reply-To: <20080701012531.GA92392@citylink.fud.org.nz> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on blah.cmotd.com X-Virus-Status: Clean Cc: Pyun YongHyeon , freebsd-net@freebsd.org Subject: Re: if_bridge turns off checksum offload of members? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 09:51:50 -0000 Hi, May be a stupid questions, but: 1) There are zero matches of IFCAP_TOE in kernel sources .. there is not support for TOE in 7.0, but may be this is work in progress for 8-current? 2) In #define BRIDGE_IFCAPS_MASK (IFCAP_TOE|IFCAP_TSO|IFCAP_TXCSUM) - TOE should be repleaced with RXCSUM or just removed? 3) Why RX is never checked? In my case this doesn't matter because em turn off both TX and RX if only one is disabled, but probably there is a hardware, that can separate them e.g. RX disabled while TX enabled? 4) I'm not sure why bridge should not work with two interfaces one of which support TX and the other does not? At least if I turn on checksum offload only on one of the interfaces the bridge is still working ... Andrew Thompson wrote: - cut - > > > This patch should do that, are you able to test it Stefan? > > > cheers, > Andrew > P.S. I saw very good results with netisr2 on a kernel from p4 before few months .. are there any patches flying around so I can test them with 7-STABLE? :) -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 10:10:16 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBD321065673; Tue, 1 Jul 2008 10:10:16 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 999A08FC15; Tue, 1 Jul 2008 10:10:16 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id AEF6D1B10EE0; Tue, 1 Jul 2008 12:10:14 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on malcho.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.4 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 577F91B10EBB; Tue, 1 Jul 2008 12:10:09 +0200 (CEST) Message-ID: <486A0281.208@moneybookers.com> Date: Tue, 01 Jul 2008 13:10:09 +0300 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.14 (X11/20080616) MIME-Version: 1.0 To: Andrew Thompson References: <4868A34C.6030304@moneybookers.com> <20080630101629.GD79537@cdnetworks.co.kr> <20080701012531.GA92392@citylink.fud.org.nz> <4869FE2E.4070805@moneybookers.com> In-Reply-To: <4869FE2E.4070805@moneybookers.com> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-net@freebsd.org Subject: Re: if_bridge turns off checksum offload of members? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 10:10:17 -0000 Hi, Sorry to reply to myself. Stefan Lambrev wrote: > Hi, > > May be a stupid questions, but: > > 1) There are zero matches of IFCAP_TOE in kernel sources .. there is > not support for TOE in 7.0, but may be this is work in progress for > 8-current? > 2) In #define BRIDGE_IFCAPS_MASK (IFCAP_TOE|IFCAP_TSO|IFCAP_TXCSUM) - > TOE should be repleaced with RXCSUM or just removed? Your patch plus this small change (replacing TOE with RXCSUM) seems to work fine for me - kernel compiles without a problem and checksum offload is enabled after reboot. > 3) Why RX is never checked? In my case this doesn't matter because em > turn off both TX and RX if only one is disabled, but probably there is > a hardware, > that can separate them e.g. RX disabled while TX enabled? > 4) I'm not sure why bridge should not work with two interfaces one of > which support TX and the other does not? At least if I turn on > checksum offload > only on one of the interfaces the bridge is still working ... > > Andrew Thompson wrote: > > - cut - >> >> >> This patch should do that, are you able to test it Stefan? >> >> >> cheers, >> Andrew >> > P.S. I saw very good results with netisr2 on a kernel from p4 before > few months .. are there any patches flying around so I can test them > with 7-STABLE? :) > -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 12:27:18 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 36F19106568F for ; Tue, 1 Jul 2008 12:27:18 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 763D98FC1B for ; Tue, 1 Jul 2008 12:27:16 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 5868 invoked from network); 1 Jul 2008 14:27:15 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 1 Jul 2008 14:27:15 +0200 Date: Tue, 1 Jul 2008 14:27:15 +0200 (CEST) From: Ingo Flaschberger To: Paul In-Reply-To: <4869A099.5070206@gtcomm.net> Message-ID: References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <4869A099.5070206@gtcomm.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 12:27:18 -0000 Dear Paul, > I have been unable to even come close to livelocking the machine with the em > driver interrupt moderation. > So that to me throws polling out the window. I tried 8000hz with polling > modified to allow 10000 burst and it makes no difference higher hz-values gives you better latenca but less overall speed. 2000hz should be enough. Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 12:43:17 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 29A07106566C for ; Tue, 1 Jul 2008 12:43:17 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 6D05D8FC16 for ; Tue, 1 Jul 2008 12:43:15 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 15393 invoked from network); 1 Jul 2008 14:43:14 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 1 Jul 2008 14:43:14 +0200 Date: Tue, 1 Jul 2008 14:43:14 +0200 (CEST) From: Ingo Flaschberger To: Paul In-Reply-To: <4869F42E.8040904@gtcomm.net> Message-ID: References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <4869F42E.8040904@gtcomm.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 12:43:17 -0000 Dear Paul, > > Dual Opteron 2212, Recompiled kernel with 7-STABLE and removed a lot of junk > in the config, added > options NO_ADAPTIVE_MUTEXES not sure if that makes any difference > or not, will test without. > Used ULE scheduler, used preemption, CPUTYPE=opteron in /etc/make.conf > 7.0-STABLE FreeBSD 7.0-STABLE #4: Tue Jul 1 01:22:18 CDT 2008 amd64 > Max input rate .. 587kpps? Take into consideration that these packets are > being forwarded out em1 interface which > causes a great impact on cpu usage. If I set up a firewall rule to block the > packets it can do over 1mpps on em0 input. would be great if you can also test with 32bit. what value do you have at net.inet.ip.intr_queue_maxlen? kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 13:06:57 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 435EB1065671 for ; Tue, 1 Jul 2008 13:06:57 +0000 (UTC) (envelope-from Matthias.Apitz@oclc.org) Received: from mail.pica.nl (mail.pica.nl [192.87.44.30]) by mx1.freebsd.org (Postfix) with ESMTP id CF62D8FC1F for ; Tue, 1 Jul 2008 13:06:56 +0000 (UTC) (envelope-from Matthias.Apitz@oclc.org) Received: from rebelion.Sisis.de ([193.31.10.34]) by mail.pica.nl with Microsoft SMTPSVC(6.0.3790.3959); Tue, 1 Jul 2008 14:54:52 +0200 Received: (from guru@localhost) by rebelion.Sisis.de (8.14.2/8.13.8/Submit) id m61Csqh2010941; Tue, 1 Jul 2008 14:54:52 +0200 (CEST) (envelope-from matthias.apitz@oclc.org) X-Authentication-Warning: rebelion.Sisis.de: guru set sender to matthias.apitz@oclc.org using -f Date: Tue, 1 Jul 2008 14:54:52 +0200 From: Matthias Apitz To: freebsd-net@freebsd.org Message-ID: <20080701125452.GA10729@rebelion.Sisis.de> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.0-RELEASE (i386) X-OriginalArrivalTime: 01 Jul 2008 12:54:53.0019 (UTC) FILETIME=[A7A4F2B0:01C8DB79] Cc: freebsd-mobile@freebsd.org Subject: RELENG_7 && ath && WPA && stuck when bgscan is active on interface X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Matthias Apitz List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 13:06:57 -0000 Hello, I'm running the above configuration, RELENG_7 kernel and WPA, on an Asus laptop eeePC 900 for which one must patch the HAL with: http://snapshots.madwifi.org/special/madwifi-ng-r2756+ar5007.tar.gz ) all is fine, mostly, but when 'bgscan' is activated on the interface ath0 it get stuck reproduce-able after some time without any traffic through the interface; setting 'ifconfig ath0 -bgscan' makes the problem going away; could it be related to the bug I'm facing on another laptop with bgscan/WPA/iwi0, see: http://www.freebsd.org/cgi/query-pr.cgi?pr=122331 thx matthias -- Matthias Apitz Manager Technical Support - OCLC GmbH Gruenwalder Weg 28g - 82041 Oberhaching - Germany t +49-89-61308 351 - f +49-89-61308 399 - m +49-170-4527211 e - w http://www.oclc.org/ http://www.UnixArea.de/ b http://gurucubano.blogspot.com/ «...una sola vez, que es cuanto basta si se trata de verdades definitivas.» «...only once, which is enough if it has todo with definite truth.» José Saramago, Historia del Cerca de Lisboa From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 13:34:47 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 23D721065681 for ; Tue, 1 Jul 2008 13:34:47 +0000 (UTC) (envelope-from max@love2party.net) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.183]) by mx1.freebsd.org (Postfix) with ESMTP id ABDB08FC19 for ; Tue, 1 Jul 2008 13:34:46 +0000 (UTC) (envelope-from max@love2party.net) Received: from vampire.homelinux.org (dslb-088-066-057-020.pools.arcor-ip.net [88.66.57.20]) by mrelayeu.kundenserver.de (node=mrelayeu5) with ESMTP (Nemesis) id 0ML25U-1KDg0L0yAW-0006Dj; Tue, 01 Jul 2008 15:34:45 +0200 Received: (qmail 56383 invoked from network); 1 Jul 2008 13:32:23 -0000 Received: from myhost.laiers.local (192.168.4.151) by laiers.local with SMTP; 1 Jul 2008 13:32:23 -0000 From: Max Laier Organization: FreeBSD To: Sergey Matveychuk Date: Tue, 1 Jul 2008 15:32:39 +0200 User-Agent: KMail/1.9.9 References: <1214651667.267043.71931.nullmailer@cicuta.babolo.ru> <200806291743.15021.max@love2party.net> <486A2F5F.6070408@FreeBSD.org> In-Reply-To: <486A2F5F.6070408@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200807011532.39508.max@love2party.net> X-Provags-ID: V01U2FsdGVkX1/r/sOKut9LM0VulmOLnX6QUZvkV6/U2HSG9+K yj+E31cRKlauJVynfNLfctPZjkvVzEQU4PMvgcWFtmziGv3qZp CaJmztivUI++RJj3np+5w== Cc: freebsd-net@freebsd.org, Giulio Ferro , Alexandre Biancalana Subject: Re: altq on vlan X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 13:34:47 -0000 On Tuesday 01 July 2008 15:21:35 Sergey Matveychuk wrote: > Max Laier wrote: > > Now please ... let this die, it's stupid! > > I wrote the patch for *very* specific purpose. I've never want to ask > commit it and I did not think it'll be use someone seriously. > > Sorry for touching your religious sense :) Sorry for the harsh language. It's just that this comes up every other month and (unexperienced) users might be using the patch without clue - hence I wanted to have some kind of "DON'T DO THIS UNLESS YOU KNOW WHAT YOU ARE DOING" recorded in this thread - I hope that google will help people to actually find it. Would you mind adding some words to that effect to your patch? And yes, the patch has some value in some situations, but most certainly not in the general case. -- /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 13:41:50 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E59A7106564A for ; Tue, 1 Jul 2008 13:41:50 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from mail.ciam.ru (ns.ciam.ru [213.247.195.75]) by mx1.freebsd.org (Postfix) with ESMTP id A10F68FC13 for ; Tue, 1 Jul 2008 13:41:50 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from dhcp250-210.yandex.ru ([87.250.250.210]) by mail.ciam.ru with esmtpa (Exim 4.x) id 1KDfnd-000KE0-NP; Tue, 01 Jul 2008 17:21:37 +0400 Message-ID: <486A2F5F.6070408@FreeBSD.org> Date: Tue, 01 Jul 2008 17:21:35 +0400 From: Sergey Matveychuk User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Max Laier References: <1214651667.267043.71931.nullmailer@cicuta.babolo.ru> <200806291743.15021.max@love2party.net> In-Reply-To: <200806291743.15021.max@love2party.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Giulio Ferro , Alexandre Biancalana Subject: Re: altq on vlan X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 13:41:51 -0000 Max Laier wrote: > > Now please ... let this die, it's stupid! > I wrote the patch for *very* specific purpose. I've never want to ask commit it and I did not think it'll be use someone seriously. Sorry for touching your religious sense :) -- Dixi. Sem. From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 13:47:20 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3678B1065674 for ; Tue, 1 Jul 2008 13:47:20 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from mail.ciam.ru (ns.ciam.ru [213.247.195.75]) by mx1.freebsd.org (Postfix) with ESMTP id E57DA8FC1C for ; Tue, 1 Jul 2008 13:47:19 +0000 (UTC) (envelope-from sem@FreeBSD.org) Received: from dhcp250-210.yandex.ru ([87.250.250.210]) by mail.ciam.ru with esmtpa (Exim 4.x) id 1KDgCV-000LsO-4E; Tue, 01 Jul 2008 17:47:19 +0400 Message-ID: <486A356E.5000307@FreeBSD.org> Date: Tue, 01 Jul 2008 17:47:26 +0400 From: Sergey Matveychuk User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Max Laier References: <1214651667.267043.71931.nullmailer@cicuta.babolo.ru> <200806291743.15021.max@love2party.net> <486A2F5F.6070408@FreeBSD.org> <200807011532.39508.max@love2party.net> In-Reply-To: <200807011532.39508.max@love2party.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Giulio Ferro , Alexandre Biancalana Subject: Re: altq on vlan X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 13:47:20 -0000 Max Laier wrote: > > Would you mind adding some words to that effect to your patch? > I think I'll hide it from public access instead. Looks like some people prefer to patch kernel instead of learning how to make a queue on parent interface. -- Dixi. Sem. From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 14:04:28 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 03D3E106566C for ; Tue, 1 Jul 2008 14:04:28 +0000 (UTC) (envelope-from thompsa@FreeBSD.org) Received: from pele.citylink.co.nz (pele.citylink.co.nz [202.8.44.226]) by mx1.freebsd.org (Postfix) with ESMTP id 8F4C68FC0C for ; Tue, 1 Jul 2008 14:04:27 +0000 (UTC) (envelope-from thompsa@FreeBSD.org) Received: from localhost (localhost [127.0.0.1]) by pele.citylink.co.nz (Postfix) with ESMTP id B374F2BE4B; Wed, 2 Jul 2008 02:04:26 +1200 (NZST) X-Virus-Scanned: Debian amavisd-new at citylink.co.nz Received: from pele.citylink.co.nz ([127.0.0.1]) by localhost (pele.citylink.co.nz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id pAy1MBr69iBu; Wed, 2 Jul 2008 02:04:23 +1200 (NZST) Received: from citylink.fud.org.nz (unknown [202.8.44.45]) by pele.citylink.co.nz (Postfix) with ESMTP; Wed, 2 Jul 2008 02:04:23 +1200 (NZST) Received: by citylink.fud.org.nz (Postfix, from userid 1001) id 94B5F11430; Wed, 2 Jul 2008 02:05:50 +1200 (NZST) Date: Tue, 1 Jul 2008 07:05:50 -0700 From: Andrew Thompson To: Stefan Lambrev Message-ID: <20080701140550.GA379@citylink.fud.org.nz> References: <4868A34C.6030304@moneybookers.com> <20080630101629.GD79537@cdnetworks.co.kr> <20080701012531.GA92392@citylink.fud.org.nz> <4869FE2E.4070805@moneybookers.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4869FE2E.4070805@moneybookers.com> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: Pyun YongHyeon , freebsd-net@freebsd.org Subject: Re: if_bridge turns off checksum offload of members? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 14:04:28 -0000 On Tue, Jul 01, 2008 at 12:51:42PM +0300, Stefan Lambrev wrote: > Hi, > > May be a stupid questions, but: > > 1) There are zero matches of IFCAP_TOE in kernel sources .. there is not > support for TOE in 7.0, but may be this is work in progress for 8-current? Yes, its in current only. Just remove IFCAP_TOE. > 2) In #define BRIDGE_IFCAPS_MASK (IFCAP_TOE|IFCAP_TSO|IFCAP_TXCSUM) - TOE > should be repleaced with RXCSUM or just removed? > 3) Why RX is never checked? In my case this doesn't matter because em turn > off both TX and RX if only one is disabled, but probably there is a > hardware, > that can separate them e.g. RX disabled while TX enabled? Rx does not matter, whatever isnt offloaded in hardware is just computed locally such as checking the cksum. Its Tx that messes up the bridge, if a outgoing packet is generated locally on an interface that has Tx offloading, it may actaully be sent out a different bridge member that does not have that capability. This would cause it to be sent with an invalid checksum for instance. The bridge used to just disable Tx offloading but this patch you are testing makes sure each feature is supported by all members. > 4) I'm not sure why bridge should not work with two interfaces one of which > support TX and the other does not? At least if I turn on checksum offload > only on one of the interfaces the bridge is still working ... > > Andrew Thompson wrote: > > - cut - >> >> >> This patch should do that, are you able to test it Stefan? >> >> >> cheers, >> Andrew >> > P.S. I saw very good results with netisr2 on a kernel from p4 before few > months .. are there any patches flying around so I can test them with > 7-STABLE? :) > > -- > > Best Wishes, > Stefan Lambrev > ICQ# 24134177 > From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 14:20:25 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6112C1065677; Tue, 1 Jul 2008 14:20:25 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 11FC98FC18; Tue, 1 Jul 2008 14:20:24 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id CC4C31B10EDC; Tue, 1 Jul 2008 16:20:22 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on malcho.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.4 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id D61421B10EA4; Tue, 1 Jul 2008 16:20:19 +0200 (CEST) Message-ID: <486A3D23.2020100@moneybookers.com> Date: Tue, 01 Jul 2008 17:20:19 +0300 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.14 (X11/20080616) MIME-Version: 1.0 To: Andrew Thompson References: <4868A34C.6030304@moneybookers.com> <20080630101629.GD79537@cdnetworks.co.kr> <20080701012531.GA92392@citylink.fud.org.nz> <4869FE2E.4070805@moneybookers.com> <20080701140550.GA379@citylink.fud.org.nz> In-Reply-To: <20080701140550.GA379@citylink.fud.org.nz> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on blah.cmotd.com X-Virus-Status: Clean Cc: Pyun YongHyeon , freebsd-net@freebsd.org Subject: Re: if_bridge turns off checksum offload of members? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 14:20:25 -0000 Greetings Andrew, The patch compiles and works as expected. I noticed something strange btw - swi1: net was consuming 100% WCPU (shown on top -S) but I'm not sure this have something to do with your patch, as I can't reproduce it right now .. Andrew Thompson wrote: > On Tue, Jul 01, 2008 at 12:51:42PM +0300, Stefan Lambrev wrote: > >> Hi, >> >> May be a stupid questions, but: >> >> 1) There are zero matches of IFCAP_TOE in kernel sources .. there is not >> support for TOE in 7.0, but may be this is work in progress for 8-current? >> > > Yes, its in current only. Just remove IFCAP_TOE. > > >> 2) In #define BRIDGE_IFCAPS_MASK (IFCAP_TOE|IFCAP_TSO|IFCAP_TXCSUM) - TOE >> should be repleaced with RXCSUM or just removed? >> 3) Why RX is never checked? In my case this doesn't matter because em turn >> off both TX and RX if only one is disabled, but probably there is a >> hardware, >> that can separate them e.g. RX disabled while TX enabled? >> > > Rx does not matter, whatever isnt offloaded in hardware is just computed > locally such as checking the cksum. Its Tx that messes up the bridge, if > a outgoing packet is generated locally on an interface that has Tx > offloading, it may actaully be sent out a different bridge member that > does not have that capability. This would cause it to be sent with an > invalid checksum for instance. > > The bridge used to just disable Tx offloading but this patch you are > testing makes sure each feature is supported by all members. > > >> 4) I'm not sure why bridge should not work with two interfaces one of which >> support TX and the other does not? At least if I turn on checksum offload >> only on one of the interfaces the bridge is still working ... >> >> Andrew Thompson wrote: >> >> - cut - >> >>> This patch should do that, are you able to test it Stefan? >>> >>> >>> cheers, >>> Andrew >>> >>> >> P.S. I saw very good results with netisr2 on a kernel from p4 before few >> months .. are there any patches flying around so I can test them with >> 7-STABLE? :) >> >> -- >> >> Best Wishes, >> Stefan Lambrev >> ICQ# 24134177 >> >> > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 14:56:46 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A561D1065672 for ; Tue, 1 Jul 2008 14:56:46 +0000 (UTC) (envelope-from sam@freebsd.org) Received: from ebb.errno.com (ebb.errno.com [69.12.149.25]) by mx1.freebsd.org (Postfix) with ESMTP id 80A538FC20 for ; Tue, 1 Jul 2008 14:56:46 +0000 (UTC) (envelope-from sam@freebsd.org) Received: from trouble.errno.com (trouble.errno.com [10.0.0.248]) (authenticated bits=0) by ebb.errno.com (8.13.6/8.12.6) with ESMTP id m61EuhNP060632 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 1 Jul 2008 07:56:44 -0700 (PDT) (envelope-from sam@freebsd.org) Message-ID: <486A45AB.2080609@freebsd.org> Date: Tue, 01 Jul 2008 07:56:43 -0700 From: Sam Leffler Organization: FreeBSD Project User-Agent: Thunderbird 2.0.0.9 (X11/20071125) MIME-Version: 1.0 To: Larry Baird References: <20080630040103.94730.qmail@mailgate.gta.com> In-Reply-To: <20080630040103.94730.qmail@mailgate.gta.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DCC--Metrics: ebb.errno.com; whitelist Cc: freebsd-net@freebsd.org, vanhu_bsd@zeninc.net Subject: Re: FreeBSD NAT-T patch integration X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 14:56:46 -0000 Larry Baird wrote: >> And how do I know that it works ? >> Well, when it doesn't work, I do know it, quite quickly most of the >> time ! >> > I have to chime in here. I did most of the initial porting of the > NAT-T patches from Kame IPSec to FAST_IPSEC. I did look at every > line of code during this process. I found no security problems during > the port. Like Yvan, my company uses the NAT-T patches commercially. > Like he says, if it had problems, we would hear about it. If the patches > don't get commited, I highly suspect Yvan or myself would try to keep the > patches up todate. So far I have done FAST_IPSEC pacthes for FreeBSD 4,5,6. > Yvan did 7 and 8 by himself. Keeping up gets to be a pain after a while. > I do plan to look at the FreeBSD 7 patches soon, but it sure would be nice > to see it commited. > > This whole issue seems ridiculous. I've been trying to get the NAT-T patches committed for a while but since I'm not setup to do any IPSEC testing have deferred to others. If we need to break a logjam I'll pitch in. Sam From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 15:12:00 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EF1271065675 for ; Tue, 1 Jul 2008 15:12:00 +0000 (UTC) (envelope-from sam@freebsd.org) Received: from ebb.errno.com (ebb.errno.com [69.12.149.25]) by mx1.freebsd.org (Postfix) with ESMTP id 870068FC19 for ; Tue, 1 Jul 2008 15:12:00 +0000 (UTC) (envelope-from sam@freebsd.org) Received: from trouble.errno.com (trouble.errno.com [10.0.0.248]) (authenticated bits=0) by ebb.errno.com (8.13.6/8.12.6) with ESMTP id m61EZdoG060471 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 1 Jul 2008 07:35:39 -0700 (PDT) (envelope-from sam@freebsd.org) Message-ID: <486A40BB.70006@freebsd.org> Date: Tue, 01 Jul 2008 07:35:39 -0700 From: Sam Leffler Organization: FreeBSD Project User-Agent: Thunderbird 2.0.0.9 (X11/20071125) MIME-Version: 1.0 To: Andrew Thompson References: <4868A34C.6030304@moneybookers.com> <20080630101629.GD79537@cdnetworks.co.kr> <20080701012531.GA92392@citylink.fud.org.nz> <4869FE2E.4070805@moneybookers.com> <20080701140550.GA379@citylink.fud.org.nz> In-Reply-To: <20080701140550.GA379@citylink.fud.org.nz> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DCC--Metrics: ebb.errno.com; whitelist Cc: Pyun YongHyeon , freebsd-net@freebsd.org, Stefan Lambrev Subject: Re: if_bridge turns off checksum offload of members? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 15:12:01 -0000 Andrew Thompson wrote: > On Tue, Jul 01, 2008 at 12:51:42PM +0300, Stefan Lambrev wrote: > >> Hi, >> >> May be a stupid questions, but: >> >> 1) There are zero matches of IFCAP_TOE in kernel sources .. there is not >> support for TOE in 7.0, but may be this is work in progress for 8-current? >> > > Yes, its in current only. Just remove IFCAP_TOE. > > >> 2) In #define BRIDGE_IFCAPS_MASK (IFCAP_TOE|IFCAP_TSO|IFCAP_TXCSUM) - TOE >> should be repleaced with RXCSUM or just removed? >> 3) Why RX is never checked? In my case this doesn't matter because em turn >> off both TX and RX if only one is disabled, but probably there is a >> hardware, >> that can separate them e.g. RX disabled while TX enabled? >> > > Rx does not matter, whatever isnt offloaded in hardware is just computed > locally such as checking the cksum. Its Tx that messes up the bridge, if > a outgoing packet is generated locally on an interface that has Tx > offloading, it may actaully be sent out a different bridge member that > does not have that capability. This would cause it to be sent with an > invalid checksum for instance. > > The bridge used to just disable Tx offloading but this patch you are > testing makes sure each feature is supported by all members. > > >> 4) I'm not sure why bridge should not work with two interfaces one of which >> support TX and the other does not? At least if I turn on checksum offload >> only on one of the interfaces the bridge is still working ... >> >> Andrew Thompson wrote: >> >> - cut - >> >>> This patch should do that, are you able to test it Stefan? >>> >>> >>> cheers, >>> Andrew >>> >>> >> P.S. I saw very good results with netisr2 on a kernel from p4 before few >> months .. are there any patches flying around so I can test them with >> 7-STABLE? :) >> >> This issue has come up before. Handling checksum offload in the bridge for devices that are not capable is not a big deal and is important for performance. TSO likewise should be done but we're missing a generic TSO support routine to do that (I believe, netbsd has one and linux has a GSO mechanism). Sam From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 16:04:02 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EAD5C1065673 for ; Tue, 1 Jul 2008 16:04:02 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id AACA68FC1A for ; Tue, 1 Jul 2008 16:04:02 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDiHP-00012S-R0; Tue, 01 Jul 2008 12:00:31 -0400 Message-ID: <486A55ED.2030304@gtcomm.net> Date: Tue, 01 Jul 2008 12:06:05 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Stefan Lambrev References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <4869F42E.8040904@gtcomm.net> <4869F621.4000508@moneybookers.com> In-Reply-To: <4869F621.4000508@moneybookers.com> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 16:04:03 -0000 Thanks.. I was hoping I wasn't seeing things :> I do not like inconsistencies.. :/ Stefan Lambrev wrote: > > > Greetings Paul, >> >> >> ------OK I'm stumped now.. Rebuilt with preemption and ULE and >> preemption again and it's not doing what it did before.. > I saw this in my configuration too :) Just leave your test running for > longer time and you will see this strange inconsistency in action. > In my configuration I almost always have better throughput after > reboot, which drops latter (5-10min under flood) with 50-60kpps and > after another 10-15min the number of correctly passed packet increase > again. Looks like "auto tuning" of which I'm not aware :) > >> How could that be? Now about 500kpps.. >> >> That kind of inconsistency almost invalidates all my testing.. why >> would it be so much different after trying a bunch of kernel options >> and rebooting a bunch of times and then going back to the original >> config doesn't get you what it did in the beginning.. >> >> I'll have to dig into this further.. never seen anything like it :) >> >> Hopefully the ip_input fix will help free up a few cpu cycles. >> >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 16:08:23 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A5C41106567D for ; Tue, 1 Jul 2008 16:08:23 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 7888A8FC19 for ; Tue, 1 Jul 2008 16:08:23 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDiLc-0001d5-Km; Tue, 01 Jul 2008 12:04:52 -0400 Message-ID: <486A56F2.2000400@gtcomm.net> Date: Tue, 01 Jul 2008 12:10:26 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <4869F42E.8040904@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 16:08:23 -0000 I am going to.. I have an opteron 270 dual set up on 32 bit and the 2212 is set up on 64 bit :) Today should bring some 32 bit results as well as etherchannel results. Ingo Flaschberger wrote: > Dear Paul, > >> >> Dual Opteron 2212, Recompiled kernel with 7-STABLE and removed a lot >> of junk in the config, added >> options NO_ADAPTIVE_MUTEXES not sure if that makes any >> difference or not, will test without. >> Used ULE scheduler, used preemption, CPUTYPE=opteron in /etc/make.conf >> 7.0-STABLE FreeBSD 7.0-STABLE #4: Tue Jul 1 01:22:18 CDT 2008 amd64 >> Max input rate .. 587kpps? Take into consideration that these >> packets are being forwarded out em1 interface which >> causes a great impact on cpu usage. If I set up a firewall rule to >> block the packets it can do over 1mpps on em0 input. > > would be great if you can also test with 32bit. > > what value do you have at net.inet.ip.intr_queue_maxlen? > > kind regards, > Ingo Flaschberger > > From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 17:23:07 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 94A8F1065703 for ; Tue, 1 Jul 2008 17:23:07 +0000 (UTC) (envelope-from petar@smokva.net) Received: from morrison.andev.ch (morrison.andev.ch [78.47.142.202]) by mx1.freebsd.org (Postfix) with ESMTP id 5CB998FC25 for ; Tue, 1 Jul 2008 17:23:07 +0000 (UTC) (envelope-from petar@smokva.net) Received: from pintail.smokva.net (84-74-146-124.dclient.hispeed.ch [84.74.146.124]) by morrison.andev.ch (Postfix) with ESMTP id 543925DADC for ; Tue, 1 Jul 2008 19:22:28 +0200 (CEST) Date: Tue, 1 Jul 2008 19:23:04 +0200 From: Petar Bogdanovic To: freebsd-net@freebsd.org Message-ID: <20080701172304.GA17817@pintail.smokva.net> Mail-Followup-To: freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.18 (2008-05-17) Subject: dhclient.c: script_go() vs. priv_script_go() X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 17:23:07 -0000 Hi, it's probably because I don't understand the code but may I ask what script_go() is supposed to do? The only function in dhclient.c using execve() is priv_script_go() and this gets executed only once in main() with $reason = PREINIT. That's why I looked at the code in the first place: I can't make dhclient run dhclient-script with anything else than PREINIT. I would expect at least one additional run with i.e. $reason = BOUND. Thanks for the enlightenment, Petar From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 18:56:11 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 486811065684 for ; Tue, 1 Jul 2008 18:56:11 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 06BB18FC1C for ; Tue, 1 Jul 2008 18:56:10 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDkxz-0004Hl-MJ for freebsd-net@freebsd.org; Tue, 01 Jul 2008 14:52:39 -0400 Message-ID: <486A7E45.3030902@gtcomm.net> Date: Tue, 01 Jul 2008 14:58:13 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: FreeBSD Net References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> In-Reply-To: <4869B025.9080006@gtcomm.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 18:56:11 -0000 I can't reproduce the 580kpps maximum that I saw when I first compiled for some reason, I don't understand, the max I get even with ULE and preemption is now about 530 and it dips to 480 a lot.. The first time I tried it it was at 580 and dipped to 520...what the?.. (kernel config attached at end) * noticed that SOMETIMES the em0 taskq jumps around cpus and doesn't use 100% of any one cpu * noticed that the netstat packets per second rate varies explicitly with the CPU usage of em0 taskq (top output with ULE/PREEMPTION compiled in): PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 10 root 171 ki31 0K 16K RUN 3 64:12 94.09% idle: cpu3 36 root -68 - 0K 16K CPU1 1 5:43 89.75% em0 taskq 13 root 171 ki31 0K 16K CPU0 0 63:21 87.30% idle: cpu0 12 root 171 ki31 0K 16K RUN 1 62:44 66.75% idle: cpu1 11 root 171 ki31 0K 16K CPU2 2 62:17 56.49% idle: cpu2 39 root -68 - 0K 16K - 0 0:54 10.64% em3 taskq this is about 480-500kpps rate......... now I wait a minute and PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 10 root 171 ki31 0K 16K CPU3 3 64:56 100.00% idle: cpu3 36 root -68 - 0K 16K CPU2 2 6:21 94.14% em0 taskq 13 root 171 ki31 0K 16K RUN 0 63:55 80.18% idle: cpu0 11 root 171 ki31 0K 16K RUN 2 62:48 67.38% idle: cpu2 12 root 171 ki31 0K 16K CPU1 1 63:04 58.40% idle: cpu1 39 root -68 - 0K 16K - 1 1:00 10.21% em3 taskq 530kpps rate....... drops to 85%.. 480kpps rate goes back up to 95% 530kpps it keeps flopping like this........... none of the CPUs are 100% use and none of the cpus add up , like the cpu time of em0 taskq is 94% so one of the cpus should be 6% idle but it's not. This is with ULE/PREEMPTION.. I see different behavior without preemption and with 4bsd.. and I also see different behavior depending on the time of day lol :) Figure that one out I'll post back without preemption and with 4bsd in a min then i'll move on to the 32 bit platform tests From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 20:02:33 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0D4E01065685 for ; Tue, 1 Jul 2008 20:02:33 +0000 (UTC) (envelope-from david.kwan@isilon.com) Received: from seaxch07.isilon.com (seaxch07.isilon.com [74.85.160.23]) by mx1.freebsd.org (Postfix) with ESMTP id E0F9E8FC13 for ; Tue, 1 Jul 2008 20:02:32 +0000 (UTC) (envelope-from david.kwan@isilon.com) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Date: Tue, 1 Jul 2008 12:50:35 -0700 Message-ID: X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Poor network performance for clients in 100MB to Gigabit environment Thread-Index: Acjbs7pC0SIg6mLESgGl4Dur+pKt8Q== From: "David Kwan" To: Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Poor network performance for clients in 100MB to Gigabit environment X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 20:02:33 -0000 I have a couple of questions regarding the TCP Stack: =20 I have a situation with clients on a 100MB network connecting to servers on a Gigabit network where the client read speeds are very slow from the FreeBSD server and fast from the Linux server; Write speeds from the clients to both servers are fast. (Clients on the gigabit network work fine with blazing read and write speeds). The network traces shows congestion packets for both servers when doing reads from the clients (dup acks and retransmissions), but the Linux server seem to handle the congestion better. ECN is not enabled on the network and I don't see any congestion windowing or clients window changing. The 100MB/1G switch is dropping packets. I double checked the network configuration and also swapped swithports for the servers to use the others to make sure the switch configuration are the same, and the Linux always does better than FreeBSD. Assuming that the network configuration is a constant for all clients and servers (speed, duplex, and etc...), the only variable is the servers themselves (Linux and FreeBSD). I have tried a couple of FreeBSD machines with 6.1 and 7.0 and they exhibit the same problem, with no luck matching the speed and network utilization of Linux (2 years old). The read speed test I'm referring is doing transferring of a 100MB file (cifs, nfs, and ftp), and the Linux server does it consistently in around 10 sec (line speed) with a constant network utilization chart, while the FreeBSD servers are magnitudes slower with erratic network utilization chart. I've attempted to tweak some network sysctl options on the FreeBSD, and the only ones that helped were disabling TSO and inflight; which leads me to think that the inter-packet gap was slightly increased to partially relieve congestion on the switch; not a long term solution. =20 My questions are:=20 1. Have you heard of this problem before with 100MB clients to Gigabit servers? 2. Are you aware of any Linux fix/patch in the TCP stack to better handling congestion than FreeBSD? I'm looking to address this issue in the FreeBSD, but wondering if the Linux stack did something special that can help with the FreeBSD performance. =20 David K. =20 From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 20:08:10 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 46B2F1065688 for ; Tue, 1 Jul 2008 20:08:10 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 01AFB8FC1C for ; Tue, 1 Jul 2008 20:08:09 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDm5e-0004Vb-DO for freebsd-net@freebsd.org; Tue, 01 Jul 2008 16:04:38 -0400 Message-ID: <486A8F24.5010000@gtcomm.net> Date: Tue, 01 Jul 2008 16:10:12 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: FreeBSD Net References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> In-Reply-To: <486A7E45.3030902@gtcomm.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 20:08:10 -0000 ULE without PREEMPTION is now yeilding better results. input (em0) output packets errs bytes packets errs bytes colls 571595 40639 34564108 1 0 226 0 577892 48865 34941908 1 0 178 0 545240 84744 32966404 1 0 178 0 587661 44691 35534512 1 0 178 0 587839 38073 35544904 1 0 178 0 587787 43556 35540360 1 0 178 0 540786 39492 32712746 1 0 178 0 572071 55797 34595650 1 0 178 0 *OUCH, IPFW HURTS.. loading ipfw, and adding one ipfw rule allow ip from any to any drops 100Kpps off :/ what's up with THAT? unloaded ipfw module and back 100kpps more again, that's not right with ONE rule.. :/ em0 taskq is still jumping cpus.. is there any way to lock it to one cpu or is this just a function of ULE running a tar czpvf all.tgz * and seeing if pps changes.. negligible.. guess scheduler is doing it's job at least.. Hmm. even when it's getting 50-60k errors per second on the interface I can still SCP a file through that interface although it's not fast.. 3-4MB/s.. You know, I wouldn't care if it added 5ms latency to the packets when it was doing 1mpps as long as it didn't drop any.. Why can't it do that? Queue them up and do them in bigggg chunks so none are dropped........hmm? 32 bit system is compiling now.. won't do > 400kpps with GENERIC kernel, as with 64 bit did 450k with GENERIC, although that could be the difference between opteron 270 and opteron 2212.. Paul From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 20:19:02 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2AE741065673 for ; Tue, 1 Jul 2008 20:19:02 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id E117B8FC19 for ; Tue, 1 Jul 2008 20:19:01 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDmGA-0005tI-DM; Tue, 01 Jul 2008 16:15:30 -0400 Message-ID: <486A91B0.6040505@gtcomm.net> Date: Tue, 01 Jul 2008 16:21:04 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: David Kwan References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: Poor network performance for clients in 100MB to Gigabit environment X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 20:19:02 -0000 What options do you have enabled on the linux server? sysctl -a | grep net.ipv4.tcp and on the bsd sysctl -a net.inet.tcp It sounds like a problem with BSD not handing the dropped data or ack packets so what happens is it pushes a burst of data out > 100mbit and the switch drops the packets and then BSD waits too long to recover and doesn't scale the transmission back. TCP is supposed to scale down the transmission speed until packets are not dropped to a point even without ECN. Options such as 'reno' and 'sack' etc. are congestion control algorithms that use congestion windows. David Kwan wrote: > I have a couple of questions regarding the TCP Stack: > > > > I have a situation with clients on a 100MB network connecting to servers > on a Gigabit network where the client read speeds are very slow from the > FreeBSD server and fast from the Linux server; Write speeds from the > clients to both servers are fast. (Clients on the gigabit network work > fine with blazing read and write speeds). The network traces shows > congestion packets for both servers when doing reads from the clients > (dup acks and retransmissions), but the Linux server seem to handle the > congestion better. ECN is not enabled on the network and I don't see any > congestion windowing or clients window changing. The 100MB/1G switch > > is dropping packets. I double checked the network configuration and > also swapped swithports for the servers to use the others to make sure > the switch configuration are the same, and the Linux always does better > than FreeBSD. Assuming that the network configuration is a constant for > all clients and servers (speed, duplex, and etc...), the only variable > is the servers themselves (Linux and FreeBSD). I have tried a couple of > FreeBSD machines with 6.1 and 7.0 and they exhibit the same problem, > with no luck matching the speed and network utilization of Linux (2 > years old). The read speed test I'm referring is doing transferring of > a 100MB file (cifs, nfs, and ftp), and the Linux server does it > consistently in around 10 sec (line speed) with a constant network > utilization chart, while the FreeBSD servers are magnitudes slower with > erratic network utilization chart. I've attempted to tweak some network > sysctl options on the FreeBSD, and the only ones that helped were > disabling TSO and inflight; which leads me to think that the > inter-packet gap was slightly increased to partially relieve congestion > on the switch; not a long term solution. > > > > My questions are: > > 1. Have you heard of this problem before with 100MB clients to Gigabit > servers? > > 2. Are you aware of any Linux fix/patch in the TCP stack to better > handling congestion than FreeBSD? I'm looking to address this issue in > the FreeBSD, but wondering if the Linux stack did something special that > can help with the FreeBSD performance. > > > > David K. > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 20:25:27 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B0FE106567F for ; Tue, 1 Jul 2008 20:25:27 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: from py-out-1112.google.com (py-out-1112.google.com [64.233.166.180]) by mx1.freebsd.org (Postfix) with ESMTP id B4A9D8FC24 for ; Tue, 1 Jul 2008 20:25:26 +0000 (UTC) (envelope-from jfvogel@gmail.com) Received: by py-out-1112.google.com with SMTP id p76so43322pyb.10 for ; Tue, 01 Jul 2008 13:25:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=cSky9T449eUuyF7fz1ws2pr64CJKPgU45jpQ+p7hjvY=; b=yBRxWigvnBQC1giOCuHFFzplWqNFrWtGpF/RGh0aCC6i9pQ7gBvQsnv1gqCsjUtO/t L+ouILDgh1DulGOxXhif2WkpWla5Wn2kUFU3mKfdhM8ucZ/+ssvEWQkkj6qJLGGQd6ne QcA9RMuh0Q/mBYy7y2qdhyXmmmHNvy9mwD+TA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=VJAhT5X0VEQys8n6cHxvnndo208JF2ssWAD36LWh/wE4frRXSAPuwjVvOSHne0Uhkl 9LbmhbgVPYbAfruRK4IZwmYy2u1CTYebCKti2f1fGWDy/XERO+Zb50lJL8a2odB+ATYF 1jPZCPbiNMVZ+6uc3Gn5MzAZLVMJnhB2fLDdM= Received: by 10.114.127.1 with SMTP id z1mr6280685wac.94.1214943924062; Tue, 01 Jul 2008 13:25:24 -0700 (PDT) Received: by 10.114.174.13 with HTTP; Tue, 1 Jul 2008 13:25:24 -0700 (PDT) Message-ID: <2a41acea0807011325n782b1ca7hfe78f9da67ba0462@mail.gmail.com> Date: Tue, 1 Jul 2008 13:25:24 -0700 From: "Jack Vogel" To: "David Kwan" In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: Cc: freebsd-net@freebsd.org Subject: Re: Poor network performance for clients in 100MB to Gigabit environment X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 20:25:27 -0000 Take it from someone who has spent a couple weeks beating his head against a wall over this... system tuning is essential. If your driver is going to the kernel looking for a resource and having to wait, its gonna hurt... Look into kern.ipc, and as Paul said net.inet. Off the shelf config is more than likely going to be inadequate. Good luck, Jack On Tue, Jul 1, 2008 at 12:50 PM, David Kwan wrote: > I have a couple of questions regarding the TCP Stack: > > > > I have a situation with clients on a 100MB network connecting to servers > on a Gigabit network where the client read speeds are very slow from the > FreeBSD server and fast from the Linux server; Write speeds from the > clients to both servers are fast. (Clients on the gigabit network work > fine with blazing read and write speeds). The network traces shows > congestion packets for both servers when doing reads from the clients > (dup acks and retransmissions), but the Linux server seem to handle the > congestion better. ECN is not enabled on the network and I don't see any > congestion windowing or clients window changing. The 100MB/1G switch > > is dropping packets. I double checked the network configuration and > also swapped swithports for the servers to use the others to make sure > the switch configuration are the same, and the Linux always does better > than FreeBSD. Assuming that the network configuration is a constant for > all clients and servers (speed, duplex, and etc...), the only variable > is the servers themselves (Linux and FreeBSD). I have tried a couple of > FreeBSD machines with 6.1 and 7.0 and they exhibit the same problem, > with no luck matching the speed and network utilization of Linux (2 > years old). The read speed test I'm referring is doing transferring of > a 100MB file (cifs, nfs, and ftp), and the Linux server does it > consistently in around 10 sec (line speed) with a constant network > utilization chart, while the FreeBSD servers are magnitudes slower with > erratic network utilization chart. I've attempted to tweak some network > sysctl options on the FreeBSD, and the only ones that helped were > disabling TSO and inflight; which leads me to think that the > inter-packet gap was slightly increased to partially relieve congestion > on the switch; not a long term solution. > > > > My questions are: > > 1. Have you heard of this problem before with 100MB clients to Gigabit > servers? > > 2. Are you aware of any Linux fix/patch in the TCP stack to better > handling congestion than FreeBSD? I'm looking to address this issue in > the FreeBSD, but wondering if the Linux stack did something special that > can help with the FreeBSD performance. > > > > David K. > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 20:34:58 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5935E1065677 for ; Tue, 1 Jul 2008 20:34:58 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 32A898FC18 for ; Tue, 1 Jul 2008 20:34:58 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDmVa-0007tb-PW for freebsd-net@freebsd.org; Tue, 01 Jul 2008 16:31:26 -0400 Message-ID: <486A956D.3030001@gtcomm.net> Date: Tue, 01 Jul 2008 16:37:01 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: FreeBSD Net Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Maximum ARP Entries X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 20:34:58 -0000 Does anyone know if there is a maximum number of ARP entries/ adjacencies that FBSD can handle before recycling? I want to route several thousand ips direct to some interfaces so it will have 3-4k ARP entries.. I'm curious because in Linux I have to set the sysctl net.ipv4.neigh threshholds a lot higher or it bombs with 'too many neighbors'... I don't see a setting like this in BSD sysctl . Thanks! Paul From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 20:56:43 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 279BF1065726 for ; Tue, 1 Jul 2008 20:56:40 +0000 (UTC) (envelope-from julian@elischer.org) Received: from outO.internet-mail-service.net (outo.internet-mail-service.net [216.240.47.238]) by mx1.freebsd.org (Postfix) with ESMTP id DF7FD8FC17 for ; Tue, 1 Jul 2008 20:56:39 +0000 (UTC) (envelope-from julian@elischer.org) Received: from idiom.com (mx0.idiom.com [216.240.32.160]) by out.internet-mail-service.net (Postfix) with ESMTP id 762102370; Tue, 1 Jul 2008 13:56:27 -0700 (PDT) Received: from julian-mac.elischer.org (localhost [127.0.0.1]) by idiom.com (Postfix) with ESMTP id 85BF02D600E; Tue, 1 Jul 2008 13:56:26 -0700 (PDT) Message-ID: <486A9A0E.6060308@elischer.org> Date: Tue, 01 Jul 2008 13:56:46 -0700 From: Julian Elischer User-Agent: Thunderbird 2.0.0.14 (Macintosh/20080421) MIME-Version: 1.0 To: Paul References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> In-Reply-To: <486A8F24.5010000@gtcomm.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 20:56:43 -0000 Paul wrote: > ULE without PREEMPTION is now yeilding better results. > input (em0) output > packets errs bytes packets errs bytes colls > 571595 40639 34564108 1 0 226 0 > 577892 48865 34941908 1 0 178 0 > 545240 84744 32966404 1 0 178 0 > 587661 44691 35534512 1 0 178 0 > 587839 38073 35544904 1 0 178 0 > 587787 43556 35540360 1 0 178 0 > 540786 39492 32712746 1 0 178 0 > 572071 55797 34595650 1 0 178 0 > > *OUCH, IPFW HURTS.. > loading ipfw, and adding one ipfw rule allow ip from any to any drops > 100Kpps off :/ what's up with THAT? > unloaded ipfw module and back 100kpps more again, that's not right with > ONE rule.. :/ ipfw need sto gain a lock on hte firewall before running, and is quite complex.. I can believe it.. in FreeBSD 4.8 I was able to use ipfw and filter 1Gb between two interfaces (bridged) but I think it has slowed down since then due to the SMP locking. > > em0 taskq is still jumping cpus.. is there any way to lock it to one cpu > or is this just a function of ULE > > running a tar czpvf all.tgz * and seeing if pps changes.. > negligible.. guess scheduler is doing it's job at least.. > > Hmm. even when it's getting 50-60k errors per second on the interface I > can still SCP a file through that interface although it's not fast.. > 3-4MB/s.. > > You know, I wouldn't care if it added 5ms latency to the packets when it > was doing 1mpps as long as it didn't drop any.. Why can't it do that? > Queue them up and do them in bigggg chunks so none are dropped........hmm? > > 32 bit system is compiling now.. won't do > 400kpps with GENERIC > kernel, as with 64 bit did 450k with GENERIC, although that could be > the difference between opteron 270 and opteron 2212.. > > Paul > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 22:47:29 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B2FC1065684 for ; Tue, 1 Jul 2008 22:47:29 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 21FAD8FC25 for ; Tue, 1 Jul 2008 22:47:28 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDoZo-0005e4-JW; Tue, 01 Jul 2008 18:43:56 -0400 Message-ID: <486AB47B.2050200@gtcomm.net> Date: Tue, 01 Jul 2008 18:49:31 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Julian Elischer References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> In-Reply-To: <486A9A0E.6060308@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 22:47:29 -0000 Ok, now THIS is absoultely a whole bunch of ridiculousness.. I set up etherchannel, and I'm evenly distributing packets over em0 em1 and em2 to lagg0 and i get WORSE performance than with a single interface.. Can anyone explain this one? This is horrible. I got em0-em2 taskq's using 80% cpu EACH and they are only doing 100kpps EACH looks: packets errs bytes packets errs bytes colls 105050 11066 6303000 0 0 0 0 104952 13969 6297120 0 0 0 0 104331 12121 6259860 0 0 0 0 input (em1) output packets errs bytes packets errs bytes colls 103734 70658 6223998 0 0 0 0 103483 75703 6209046 0 0 0 0 103848 76195 6230886 0 0 0 0 input (em2) output packets errs bytes packets errs bytes colls 103299 62957 6197940 1 0 226 0 106388 73071 6383280 1 0 178 0 104503 70573 6270180 4 0 712 0 last pid: 1378; load averages: 2.31, 1.28, 0.57 up 0+00:06:27 17:42:32 68 processes: 8 running, 42 sleeping, 18 waiting CPU: 0.0% user, 0.0% nice, 58.9% system, 0.0% interrupt, 41.1% idle Mem: 7980K Active, 5932K Inact, 47M Wired, 16K Cache, 8512K Buf, 1920M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 171 ki31 0K 16K RUN 2 5:18 80.47% idle: cpu2 38 root -68 - 0K 16K CPU3 3 2:30 80.18% em2 taskq 37 root -68 - 0K 16K CPU1 1 2:28 76.90% em1 taskq 36 root -68 - 0K 16K CPU2 2 2:28 72.56% em0 taskq 13 root 171 ki31 0K 16K RUN 0 3:32 29.20% idle: cpu0 12 root 171 ki31 0K 16K RUN 1 3:29 27.88% idle: cpu1 10 root 171 ki31 0K 16K RUN 3 3:21 25.63% idle: cpu3 39 root -68 - 0K 16K - 3 0:32 17.68% em3 taskq See that's total wrongness.. something is very wrong here. Does anyone have any ideas? I really need to get this working. I figured if I evenly distributed the packets over 3 interfaces it simulates having 3 rx queues because it has a separate process for each interface and the result is WAY more CPU usage and a little over half the pps throughput with a single port .. If anyone is interested in tackling some these issues please e-mail me. It would be greatly appreciated. Paul Julian Elischer wrote: > Paul wrote: >> ULE without PREEMPTION is now yeilding better results. >> input (em0) output >> packets errs bytes packets errs bytes colls >> 571595 40639 34564108 1 0 226 0 >> 577892 48865 34941908 1 0 178 0 >> 545240 84744 32966404 1 0 178 0 >> 587661 44691 35534512 1 0 178 0 >> 587839 38073 35544904 1 0 178 0 >> 587787 43556 35540360 1 0 178 0 >> 540786 39492 32712746 1 0 178 0 >> 572071 55797 34595650 1 0 178 0 >> >> *OUCH, IPFW HURTS.. >> loading ipfw, and adding one ipfw rule allow ip from any to any drops >> 100Kpps off :/ what's up with THAT? >> unloaded ipfw module and back 100kpps more again, that's not right >> with ONE rule.. :/ > > ipfw need sto gain a lock on hte firewall before running, > and is quite complex.. I can believe it.. > > in FreeBSD 4.8 I was able to use ipfw and filter 1Gb between two > interfaces (bridged) but I think it has slowed down since then due to > the SMP locking. > > >> >> em0 taskq is still jumping cpus.. is there any way to lock it to one >> cpu or is this just a function of ULE >> >> running a tar czpvf all.tgz * and seeing if pps changes.. >> negligible.. guess scheduler is doing it's job at least.. >> >> Hmm. even when it's getting 50-60k errors per second on the interface >> I can still SCP a file through that interface although it's not >> fast.. 3-4MB/s.. >> >> You know, I wouldn't care if it added 5ms latency to the packets when >> it was doing 1mpps as long as it didn't drop any.. Why can't it do >> that? Queue them up and do them in bigggg chunks so none are >> dropped........hmm? >> >> 32 bit system is compiling now.. won't do > 400kpps with GENERIC >> kernel, as with 64 bit did 450k with GENERIC, although that could be >> the difference between opteron 270 and opteron 2212.. >> >> Paul >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 22:54:07 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B9AC106566C; Tue, 1 Jul 2008 22:54:07 +0000 (UTC) (envelope-from sam@errno.com) Received: from ebb.errno.com (ebb.errno.com [69.12.149.25]) by mx1.freebsd.org (Postfix) with ESMTP id AEFAA8FC2C; Tue, 1 Jul 2008 22:54:06 +0000 (UTC) (envelope-from sam@errno.com) Received: from trouble.errno.com (trouble.errno.com [10.0.0.248]) (authenticated bits=0) by ebb.errno.com (8.13.6/8.12.6) with ESMTP id m61Ms5X4063837 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 1 Jul 2008 15:54:06 -0700 (PDT) (envelope-from sam@errno.com) Message-ID: <486AB58D.7050005@errno.com> Date: Tue, 01 Jul 2008 15:54:05 -0700 From: Sam Leffler User-Agent: Thunderbird 2.0.0.9 (X11/20071125) MIME-Version: 1.0 To: Sepherosa Ziehau References: <200806191030.m5JAU36i027140@freefall.freebsd.org> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DCC-Misty-Metrics: ebb.errno.com; whitelist Cc: freebsd-net , Andrew Thompson , Joseph Lee Subject: Re: kern/124753: net80211 discards power-save queue packets early X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 22:54:07 -0000 Sepherosa Ziehau wrote: > On Thu, Jun 19, 2008 at 6:30 PM, wrote: > >> Synopsis: net80211 discards power-save queue packets early >> >> Responsible-Changed-From-To: freebsd-i386->freebsd-net >> Responsible-Changed-By: remko >> Responsible-Changed-When: Thu Jun 19 10:29:47 UTC 2008 >> Responsible-Changed-Why: >> reassign to networking team. >> >> http://www.freebsd.org/cgi/query-pr.cgi?pr=124753 >> > > In How-To-Repeat, you said: > "Then associate a recent Windows Mobile 6.1 device to the FreeBSD box > running hostapd ..." > > In Description, you said: > "The WM6.1 device recv ps-poll's for packets every 20 seconds ..." > > AFAIK, STA sends ps-poll to AP; AP does not send ps-poll to STA. Why > did your windows STA receive ps-poll from freebsd AP? Did you capture > it by using 802.11 tap? > > And which freebsd driver were you using? > > Your problem looks like: > - Either freebsd AP did not properly configure TIM in beacons, which > could be easily found out by using 802.11 tap. But I highly suspect > if you were using ath(4), TIM would be misconfigured. > - Or your windows STA didn't process TIM according to 802.11 standard. > > The PR states the listen interval sent by the station is 3 (beacons) and the beacon interval is 100TU. This means the AP is required to buffer unicast frames for only 300TU which is ~300 ms. But according to the report the Windows device is polling every 20 seconds so there's no guarantee any packets will be present (even with the net80211 code arbitrarily using 4x the list interval specified by the sta). I find it really hard to believe a device would poll every 20 secs so something seems wrong in what's reported/observed. Given that defeating the aging logic just pushed the problem elsewhere it sounds like there's something else wrong which (as you note) probably requires a packet capture to understand. I'm pretty sure TIM is handled correctly in RELENG_7 but a packet capture would help us verify that. Sam From owner-freebsd-net@FreeBSD.ORG Tue Jul 1 23:00:19 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 730E91065671 for ; Tue, 1 Jul 2008 23:00:19 +0000 (UTC) (envelope-from ru@FreeBSD.org) Received: from mail.vega.ru (infra.dev.vega.ru [90.156.167.14]) by mx1.freebsd.org (Postfix) with ESMTP id 2496D8FC25 for ; Tue, 1 Jul 2008 23:00:18 +0000 (UTC) (envelope-from ru@FreeBSD.org) Received: from [87.242.97.68] (port=61742 helo=edoofus.dev.vega.ru) by mail.vega.ru with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.68 (FreeBSD)) (envelope-from ) id 1KDoPL-000FfK-Kh; Tue, 01 Jul 2008 22:33:07 +0000 Date: Wed, 2 Jul 2008 02:32:47 +0400 From: Ruslan Ermilov To: Paul Message-ID: <20080701223247.GB8518@edoofus.dev.vega.ru> References: <486A956D.3030001@gtcomm.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <486A956D.3030001@gtcomm.net> Cc: FreeBSD Net Subject: Re: Maximum ARP Entries X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 01 Jul 2008 23:00:19 -0000 On Tue, Jul 01, 2008 at 04:37:01PM -0400, Paul wrote: > Does anyone know if there is a maximum number of ARP entries/ > adjacencies that FBSD can handle before recycling? > In FreeBSD, ARP still uses routing table as its storage, and as such limits on the routing table memory applies, and the latter currently has no limit. Cheers, -- Ruslan Ermilov ru@FreeBSD.org FreeBSD committer From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 00:06:59 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8A4531065678 for ; Wed, 2 Jul 2008 00:06:59 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 0B7E58FC12 for ; Wed, 2 Jul 2008 00:06:58 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDpok-0004XK-AO for freebsd-net@freebsd.org; Tue, 01 Jul 2008 20:03:26 -0400 Message-ID: <486AC71D.2080804@gtcomm.net> Date: Tue, 01 Jul 2008 20:09:01 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: FreeBSD Net References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486A9BBF.1060308@gtcomm.net> <486AA299.7090904@elischer.org> In-Reply-To: <486AA299.7090904@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 00:06:59 -0000 Apparently lagg hasn't been giant fixed :/ Can we do something about this quickly? with adaptive giant i get more performance on lagg but the cpu usage is smashed 100% I get about 50k more pps per interface (so 150kpps total which STILL is less than a single gigabit port) Check it out 68 processes: 9 running, 41 sleeping, 18 waiting CPU: 0.0% user, 0.0% nice, 89.5% system, 0.0% interrupt, 10.5% idle Mem: 8016K Active, 6192K Inact, 47M Wired, 108K Cache, 9056K Buf, 1919M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 38 root -68 - 0K 16K CPU1 1 3:29 100.00% em2 taskq 37 root -68 - 0K 16K CPU0 0 3:31 98.78% em1 taskq 36 root -68 - 0K 16K CPU3 3 2:53 82.42% em0 taskq 11 root 171 ki31 0K 16K RUN 2 22:48 79.00% idle: cpu2 10 root 171 ki31 0K 16K RUN 3 20:51 22.90% idle: cpu3 39 root -68 - 0K 16K RUN 2 0:32 16.60% em3 taskq 12 root 171 ki31 0K 16K RUN 1 20:16 2.05% idle: cpu1 13 root 171 ki31 0K 16K RUN 0 20:25 1.90% idle: cpu0 input (em0) output packets errs bytes packets errs bytes colls 122588 0 7355280 0 0 0 0 123057 0 7383420 0 0 0 0 input (em1) output packets errs bytes packets errs bytes colls 174917 11899 10495032 2 0 178 0 173967 11697 10438038 2 0 356 0 174630 10603 10477806 2 0 268 0 input (em2) output packets errs bytes packets errs bytes colls 175843 3928 10550580 0 0 0 0 175952 5750 10557120 0 0 0 0 Still less performance than single gig-e.. that giant lock really sucks , and why on earth would LAGG require that.. It seems so simple to fix :/ Anyone up for it:) I wish I was a programmer sometimes, but network engineering will have to do. :D Julian Elischer wrote: > Paul wrote: >> Is PF better than ipfw? iptables almost has no impact on routing >> performance unless I add a swath of rules to it and then it bombs >> I need maybe 10 rules max and I don't want 20% performance drop for >> that.. :P > > well lots of people have wanted to fix it, and I've investigated > quite a lot but it takes someone with 2 weeks of free time and > all the right clue. It's not inherrent in ipfw but it needs some > TLC from someone who cares :-). > > > >> Ouch! :) Is this going to be fixed any time soon? We have some >> money that can be used for development costs to fix things like this >> because >> we use linux and freebsd machines as firewalls for a lot of customers >> and with the increasing bandwidth and pps the customers are demanding >> more and I >> can't give them better performance with a brand new dual xeon or >> opteron machine vs the old p4 machines I have them running on now :/ >> The only difference >> in the new machine vs old machine is that the new one can take in >> more pps and drop it but it can't route a whole lot more. >> Routing/firewalling must still not be lock free, ugh.. :P >> >> Thanks >> >> >> >> Julian Elischer wrote: >>> Paul wrote: >>>> ULE without PREEMPTION is now yeilding better results. >>>> input (em0) output >>>> packets errs bytes packets errs bytes colls >>>> 571595 40639 34564108 1 0 226 0 >>>> 577892 48865 34941908 1 0 178 0 >>>> 545240 84744 32966404 1 0 178 0 >>>> 587661 44691 35534512 1 0 178 0 >>>> 587839 38073 35544904 1 0 178 0 >>>> 587787 43556 35540360 1 0 178 0 >>>> 540786 39492 32712746 1 0 178 0 >>>> 572071 55797 34595650 1 0 178 0 >>>> >>>> *OUCH, IPFW HURTS.. >>>> loading ipfw, and adding one ipfw rule allow ip from any to any >>>> drops 100Kpps off :/ what's up with THAT? >>>> unloaded ipfw module and back 100kpps more again, that's not right >>>> with ONE rule.. :/ >>> >>> ipfw need sto gain a lock on hte firewall before running, >>> and is quite complex.. I can believe it.. >>> >>> in FreeBSD 4.8 I was able to use ipfw and filter 1Gb between two >>> interfaces (bridged) but I think it has slowed down since then due >>> to the SMP locking. >>> >>> >>>> >>>> em0 taskq is still jumping cpus.. is there any way to lock it to >>>> one cpu or is this just a function of ULE >>>> >>>> running a tar czpvf all.tgz * and seeing if pps changes.. >>>> negligible.. guess scheduler is doing it's job at least.. >>>> >>>> Hmm. even when it's getting 50-60k errors per second on the >>>> interface I can still SCP a file through that interface although >>>> it's not fast.. 3-4MB/s.. >>>> >>>> You know, I wouldn't care if it added 5ms latency to the packets >>>> when it was doing 1mpps as long as it didn't drop any.. Why can't >>>> it do that? Queue them up and do them in bigggg chunks so none are >>>> dropped........hmm? >>>> >>>> 32 bit system is compiling now.. won't do > 400kpps with GENERIC >>>> kernel, as with 64 bit did 450k with GENERIC, although that could be >>>> the difference between opteron 270 and opteron 2212.. >>>> >>>> Paul >>>> >>>> _______________________________________________ >>>> freebsd-net@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-net >>>> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >>> >>> > > From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 00:30:38 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 63489106567D for ; Wed, 2 Jul 2008 00:30:38 +0000 (UTC) (envelope-from david.kwan@isilon.com) Received: from seaxch07.isilon.com (seaxch07.isilon.com [74.85.160.23]) by mx1.freebsd.org (Postfix) with ESMTP id 3EEFD8FC19 for ; Wed, 2 Jul 2008 00:30:38 +0000 (UTC) (envelope-from david.kwan@isilon.com) X-MimeOLE: Produced By Microsoft Exchange V6.5 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Tue, 1 Jul 2008 17:30:35 -0700 Message-ID: In-Reply-To: <486A91B0.6040505@gtcomm.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: Poor network performance for clients in 100MB toGigabit environment Thread-Index: Acjbt8AtX75FGKThS8Wo8FWT8IAt2AAIdmIA References: <486A91B0.6040505@gtcomm.net> From: "David Kwan" To: Subject: RE: Poor network performance for clients in 100MB toGigabit environment X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 00:30:38 -0000 I've attempt many standard and non-standard permutations of the tcp tuning parameters without much successful via sysctl. It feels like FreeBSD is not handling the congestion very well and is beyond tuning sysctl. It's just clients on the 100MB networks has slow/erratic reads; Clients on the Gigabit network are fine and screams, so the original tcp parameters are just fine for them. For the record, these are the sysctl options for the Linux and FreeBSD.=20 Linux: net.ipv4.conf.eth0.force_igmp_version =3D 0 net.ipv4.conf.eth0.disable_policy =3D 0 net.ipv4.conf.eth0.disable_xfrm =3D 0 net.ipv4.conf.eth0.arp_ignore =3D 0 net.ipv4.conf.eth0.arp_announce =3D 0 net.ipv4.conf.eth0.arp_filter =3D 0 net.ipv4.conf.eth0.tag =3D 0 net.ipv4.conf.eth0.log_martians =3D 0 net.ipv4.conf.eth0.bootp_relay =3D 0 net.ipv4.conf.eth0.medium_id =3D 0 net.ipv4.conf.eth0.proxy_arp =3D 0 net.ipv4.conf.eth0.accept_source_route =3D 0 net.ipv4.conf.eth0.send_redirects =3D 1 net.ipv4.conf.eth0.rp_filter =3D 1 net.ipv4.conf.eth0.shared_media =3D 1 net.ipv4.conf.eth0.secure_redirects =3D 1 net.ipv4.conf.eth0.accept_redirects =3D 1 net.ipv4.conf.eth0.mc_forwarding =3D 0 net.ipv4.conf.eth0.forwarding =3D 0 net.ipv4.conf.lo.force_igmp_version =3D 0 net.ipv4.conf.lo.disable_policy =3D 1 net.ipv4.conf.lo.disable_xfrm =3D 1 net.ipv4.conf.lo.arp_ignore =3D 0 net.ipv4.conf.lo.arp_announce =3D 0 net.ipv4.conf.lo.arp_filter =3D 0 net.ipv4.conf.lo.tag =3D 0 net.ipv4.conf.lo.log_martians =3D 0 net.ipv4.conf.lo.bootp_relay =3D 0 net.ipv4.conf.lo.medium_id =3D 0 net.ipv4.conf.lo.proxy_arp =3D 0 net.ipv4.conf.lo.accept_source_route =3D 1 net.ipv4.conf.lo.send_redirects =3D 1 net.ipv4.conf.lo.rp_filter =3D 0 net.ipv4.conf.lo.shared_media =3D 1 net.ipv4.conf.lo.secure_redirects =3D 1 net.ipv4.conf.lo.accept_redirects =3D 1 net.ipv4.conf.lo.mc_forwarding =3D 0 net.ipv4.conf.lo.forwarding =3D 0 net.ipv4.conf.default.force_igmp_version =3D 0 net.ipv4.conf.default.disable_policy =3D 0 net.ipv4.conf.default.disable_xfrm =3D 0 net.ipv4.conf.default.arp_ignore =3D 0 net.ipv4.conf.default.arp_announce =3D 0 net.ipv4.conf.default.arp_filter =3D 0 net.ipv4.conf.default.tag =3D 0 net.ipv4.conf.default.log_martians =3D 0 net.ipv4.conf.default.bootp_relay =3D 0 net.ipv4.conf.default.medium_id =3D 0 net.ipv4.conf.default.proxy_arp =3D 0 net.ipv4.conf.default.accept_source_route =3D 0 net.ipv4.conf.default.send_redirects =3D 1 net.ipv4.conf.default.rp_filter =3D 1 net.ipv4.conf.default.shared_media =3D 1 net.ipv4.conf.default.secure_redirects =3D 1 net.ipv4.conf.default.accept_redirects =3D 1 net.ipv4.conf.default.mc_forwarding =3D 0 net.ipv4.conf.default.forwarding =3D 0 net.ipv4.conf.all.force_igmp_version =3D 0 net.ipv4.conf.all.disable_policy =3D 0 net.ipv4.conf.all.disable_xfrm =3D 0 net.ipv4.conf.all.arp_ignore =3D 0 net.ipv4.conf.all.arp_announce =3D 0 net.ipv4.conf.all.arp_filter =3D 0 net.ipv4.conf.all.tag =3D 0 net.ipv4.conf.all.log_martians =3D 0 net.ipv4.conf.all.bootp_relay =3D 0 net.ipv4.conf.all.medium_id =3D 0 net.ipv4.conf.all.proxy_arp =3D 0 net.ipv4.conf.all.accept_source_route =3D 0 net.ipv4.conf.all.send_redirects =3D 1 net.ipv4.conf.all.rp_filter =3D 0 net.ipv4.conf.all.shared_media =3D 1 net.ipv4.conf.all.secure_redirects =3D 1 net.ipv4.conf.all.accept_redirects =3D 1 net.ipv4.conf.all.mc_forwarding =3D 0 net.ipv4.conf.all.forwarding =3D 0 net.ipv4.neigh.eth0.locktime =3D 99 net.ipv4.neigh.eth0.proxy_delay =3D 79 net.ipv4.neigh.eth0.anycast_delay =3D 99 net.ipv4.neigh.eth0.proxy_qlen =3D 64 net.ipv4.neigh.eth0.unres_qlen =3D 3 net.ipv4.neigh.eth0.gc_stale_time =3D 60 net.ipv4.neigh.eth0.delay_first_probe_time =3D 5 net.ipv4.neigh.eth0.base_reachable_time =3D 30 net.ipv4.neigh.eth0.retrans_time =3D 99 net.ipv4.neigh.eth0.app_solicit =3D 0 net.ipv4.neigh.eth0.ucast_solicit =3D 3 net.ipv4.neigh.eth0.mcast_solicit =3D 3 net.ipv4.neigh.lo.locktime =3D 99 net.ipv4.neigh.lo.proxy_delay =3D 79 net.ipv4.neigh.lo.anycast_delay =3D 99 net.ipv4.neigh.lo.proxy_qlen =3D 64 net.ipv4.neigh.lo.unres_qlen =3D 3 net.ipv4.neigh.lo.gc_stale_time =3D 60 net.ipv4.neigh.lo.delay_first_probe_time =3D 5 net.ipv4.neigh.lo.base_reachable_time =3D 30 net.ipv4.neigh.lo.retrans_time =3D 99 net.ipv4.neigh.lo.app_solicit =3D 0 net.ipv4.neigh.lo.ucast_solicit =3D 3 net.ipv4.neigh.lo.mcast_solicit =3D 3 net.ipv4.neigh.default.gc_thresh3 =3D 1024 net.ipv4.neigh.default.gc_thresh2 =3D 512 net.ipv4.neigh.default.gc_thresh1 =3D 128 net.ipv4.neigh.default.gc_interval =3D 30 net.ipv4.neigh.default.locktime =3D 99 net.ipv4.neigh.default.proxy_delay =3D 79 net.ipv4.neigh.default.anycast_delay =3D 99 net.ipv4.neigh.default.proxy_qlen =3D 64 net.ipv4.neigh.default.unres_qlen =3D 3 net.ipv4.neigh.default.gc_stale_time =3D 60 net.ipv4.neigh.default.delay_first_probe_time =3D 5 net.ipv4.neigh.default.base_reachable_time =3D 30 net.ipv4.neigh.default.retrans_time =3D 99 net.ipv4.neigh.default.app_solicit =3D 0 net.ipv4.neigh.default.ucast_solicit =3D 3 net.ipv4.neigh.default.mcast_solicit =3D 3 net.ipv4.tcp_slow_start_after_idle =3D 1 net.ipv4.tcp_workaround_signed_windows =3D 1 net.ipv4.tcp_bic_beta =3D 819 net.ipv4.tcp_tso_win_divisor =3D 8 net.ipv4.tcp_moderate_rcvbuf =3D 1 net.ipv4.tcp_bic_low_window =3D 14 net.ipv4.tcp_bic_fast_convergence =3D 1 net.ipv4.tcp_bic =3D 1 net.ipv4.tcp_vegas_gamma =3D 2 net.ipv4.tcp_vegas_beta =3D 6 net.ipv4.tcp_vegas_alpha =3D 2 net.ipv4.tcp_vegas_cong_avoid =3D 0 net.ipv4.tcp_westwood =3D 0 net.ipv4.tcp_no_metrics_save =3D 0 net.ipv4.ipfrag_secret_interval =3D 600 net.ipv4.tcp_low_latency =3D 0 net.ipv4.tcp_frto =3D 0 net.ipv4.tcp_tw_reuse =3D 0 net.ipv4.icmp_ratemask =3D 6168 net.ipv4.icmp_ratelimit =3D 1000 net.ipv4.tcp_adv_win_scale =3D 2 net.ipv4.tcp_app_win =3D 31 net.ipv4.tcp_rmem =3D 4096 87380 174760 net.ipv4.tcp_wmem =3D 4096 16384 131072 net.ipv4.tcp_mem =3D 786432 1048576 1572864 net.ipv4.tcp_dsack =3D 1 net.ipv4.tcp_ecn =3D 0 net.ipv4.tcp_reordering =3D 3 net.ipv4.tcp_fack =3D 1 net.ipv4.tcp_orphan_retries =3D 0 net.ipv4.inet_peer_gc_maxtime =3D 120 net.ipv4.inet_peer_gc_mintime =3D 10 net.ipv4.inet_peer_maxttl =3D 600 net.ipv4.inet_peer_minttl =3D 120 net.ipv4.inet_peer_threshold =3D 65664 net.ipv4.igmp_max_msf =3D 10 net.ipv4.igmp_max_memberships =3D 20 net.ipv4.route.secret_interval =3D 600 net.ipv4.route.min_adv_mss =3D 256 net.ipv4.route.min_pmtu =3D 552 net.ipv4.route.mtu_expires =3D 600 net.ipv4.route.gc_elasticity =3D 8 net.ipv4.route.error_burst =3D 5000 net.ipv4.route.error_cost =3D 1000 net.ipv4.route.redirect_silence =3D 20480 net.ipv4.route.redirect_number =3D 9 net.ipv4.route.redirect_load =3D 20 net.ipv4.route.gc_interval =3D 60 net.ipv4.route.gc_timeout =3D 300 net.ipv4.route.gc_min_interval =3D 0 net.ipv4.route.max_size =3D 1048576 net.ipv4.route.gc_thresh =3D 65536 net.ipv4.route.max_delay =3D 10 net.ipv4.route.min_delay =3D 2 net.ipv4.icmp_errors_use_inbound_ifaddr =3D 0 net.ipv4.icmp_ignore_bogus_error_responses =3D 0 net.ipv4.icmp_echo_ignore_broadcasts =3D 0 net.ipv4.icmp_echo_ignore_all =3D 0 net.ipv4.ip_local_port_range =3D 32768 61000 net.ipv4.tcp_max_syn_backlog =3D 1024 net.ipv4.tcp_rfc1337 =3D 0 net.ipv4.tcp_stdurg =3D 0 net.ipv4.tcp_abort_on_overflow =3D 0 net.ipv4.tcp_tw_recycle =3D 0 net.ipv4.tcp_syncookies =3D 0 net.ipv4.tcp_fin_timeout =3D 60 net.ipv4.tcp_retries2 =3D 15 net.ipv4.tcp_retries1 =3D 3 net.ipv4.tcp_keepalive_intvl =3D 75 net.ipv4.tcp_keepalive_probes =3D 9 net.ipv4.tcp_keepalive_time =3D 7200 net.ipv4.ipfrag_time =3D 30 net.ipv4.ip_dynaddr =3D 0 net.ipv4.ipfrag_low_thresh =3D 196608 net.ipv4.ipfrag_high_thresh =3D 262144 net.ipv4.tcp_max_tw_buckets =3D 180000 net.ipv4.tcp_max_orphans =3D 262144 net.ipv4.tcp_synack_retries =3D 5 net.ipv4.tcp_syn_retries =3D 5 net.ipv4.ip_nonlocal_bind =3D 0 net.ipv4.ip_no_pmtu_disc =3D 0 net.ipv4.ip_autoconfig =3D 0 net.ipv4.ip_default_ttl =3D 64 net.ipv4.ip_forward =3D 0 net.ipv4.tcp_retrans_collapse =3D 1 net.ipv4.tcp_sack =3D 1 net.ipv4.tcp_window_scaling =3D 1 net.ipv4.tcp_timestamps =3D 1 FreeBSD: net.inet.ip.portrange.lowfirst: 1023 net.inet.ip.portrange.lowlast: 600 net.inet.ip.portrange.first: 49152 net.inet.ip.portrange.last: 65535 net.inet.ip.portrange.hifirst: 49152 net.inet.ip.portrange.hilast: 65535 net.inet.ip.portrange.reservedhigh: 1023 net.inet.ip.portrange.reservedlow: 0 net.inet.ip.portrange.randomized: 1 net.inet.ip.portrange.randomcps: 10 net.inet.ip.portrange.randomtime: 45 net.inet.ip.forwarding: 1 net.inet.ip.redirect: 1 net.inet.ip.ttl: 64 net.inet.ip.rtexpire: 3600 net.inet.ip.rtminexpire: 10 net.inet.ip.rtmaxcache: 128 net.inet.ip.sourceroute: 0 net.inet.ip.intr_queue_maxlen: 5000 net.inet.ip.intr_queue_drops: 0 net.inet.ip.accept_sourceroute: 0 net.inet.ip.keepfaith: 0 net.inet.ip.subnets_are_local: 0 net.inet.ip.same_prefix_carp_only: 0 net.inet.ip.fastforwarding: 0 net.inet.ip.process_options: 1 net.inet.ip.sendsourcequench: 0 net.inet.ip.random_id: 0 net.inet.ip.check_interface: 0 net.inet.ip.fragpackets: 0 net.inet.ip.maxfragsperpacket: 32 net.inet.ip.maxfragpackets: 1024 net.inet.icmp.maskrepl: 0 net.inet.icmp.icmplim: 1000 net.inet.icmp.maskfake: 0 net.inet.icmp.drop_redirect: 0 net.inet.icmp.log_redirect: 0 net.inet.icmp.icmplim_output: 1 net.inet.icmp.reply_src:=20 net.inet.icmp.reply_from_interface: 0 net.inet.icmp.quotelen: 8 net.inet.icmp.bmcastecho: 0 net.inet.tcp.rfc1323: 1 net.inet.tcp.mssdflt: 512 net.inet.tcp.keepidle: 7200000 net.inet.tcp.keepintvl: 75000 net.inet.tcp.sendspace: 131072 net.inet.tcp.recvspace: 131072 net.inet.tcp.keepinit: 75000 net.inet.tcp.delacktime: 100 net.inet.tcp.hostcache.cachelimit: 15360 net.inet.tcp.hostcache.hashsize: 512 net.inet.tcp.hostcache.bucketlimit: 30 net.inet.tcp.hostcache.count: 4 net.inet.tcp.hostcache.expire: 3600 net.inet.tcp.hostcache.purge: 0 net.inet.tcp.log_in_vain: 0 net.inet.tcp.blackhole: 0 net.inet.tcp.delayed_ack: 1 net.inet.tcp.rfc3042: 1 net.inet.tcp.rfc3390: 1 net.inet.tcp.insecure_rst: 0 net.inet.tcp.reass.maxsegments: 8256 net.inet.tcp.reass.cursegments: 0 net.inet.tcp.reass.maxqlen: 48 net.inet.tcp.reass.overflows: 0 net.inet.tcp.path_mtu_discovery: 1 net.inet.tcp.slowstart_flightsize: 1 net.inet.tcp.local_slowstart_flightsize: 4 net.inet.tcp.newreno: 1 net.inet.tcp.sndrexmitpack: 0 net.inet.tcp.sndrexmitbyte: 0 net.inet.tcp.do_tso: 1 net.inet.tcp.effective_maxseg_limit: 65535 net.inet.tcp.min_tso_factor: 2 net.inet.tcp.sack.enable: 1 net.inet.tcp.sack.maxholes: 128 net.inet.tcp.sack.globalmaxholes: 65536 net.inet.tcp.sack.globalholes: 0 net.inet.tcp.minmss: 216 net.inet.tcp.minmssoverload: 0 net.inet.tcp.tcbhashsize: 512 net.inet.tcp.do_tcpdrain: 1 net.inet.tcp.pcbcount: 199 net.inet.tcp.icmp_may_rst: 1 net.inet.tcp.isn_reseed_interval: 0 net.inet.tcp.inflight.enable: 1 net.inet.tcp.inflight.debug: 0 net.inet.tcp.inflight.rttthresh: 10 net.inet.tcp.inflight.min: 6144 net.inet.tcp.inflight.max: 1073725440 net.inet.tcp.inflight.stab: 20 net.inet.tcp.min_rtt: 3 net.inet.tcp.max_rexmt_time: 6400 net.inet.tcp.rexmt_dupacks: 3 net.inet.tcp.syncookies: 1 net.inet.tcp.syncache.bucketlimit: 30 net.inet.tcp.syncache.cachelimit: 15359 net.inet.tcp.syncache.count: 0 net.inet.tcp.syncache.hashsize: 512 net.inet.tcp.syncache.rexmtlimit: 3 net.inet.tcp.msl: 30000 net.inet.tcp.rexmit_min: 30 net.inet.tcp.rexmit_slop: 200 net.inet.tcp.always_keepalive: 1 net.inet.udp.checksum: 1 net.inet.udp.maxdgram: 9216 net.inet.udp.recvspace: 512000 net.inet.udp.log_in_vain: 0 net.inet.udp.blackhole: 0 net.inet.udp.strict_mcast_mship: 0 net.inet.raw.maxdgram: 8192 net.inet.raw.recvspace: 411648 net.inet.accf.unloadable: 0 David K. -----Original Message----- From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd-net@freebsd.org] On Behalf Of Paul Sent: Tuesday, July 01, 2008 1:21 PM To: David Kwan Cc: freebsd-net@freebsd.org Subject: Re: Poor network performance for clients in 100MB toGigabit environment What options do you have enabled on the linux server? sysctl -a | grep net.ipv4.tcp and on the bsd sysctl -a net.inet.tcp It sounds like a problem with BSD not handing the dropped data or ack=20 packets so what happens is it pushes a burst of data out > 100mbit and the switch drops the packets and then BSD waits=20 too long to recover and doesn't scale the transmission back. TCP is supposed to scale down the transmission speed until=20 packets are not dropped to a point even without ECN. Options such as 'reno' and 'sack' etc. are congestion control algorithms that use congestion windows. David Kwan wrote: > I have a couple of questions regarding the TCP Stack: > > =20 > > I have a situation with clients on a 100MB network connecting to servers > on a Gigabit network where the client read speeds are very slow from the > FreeBSD server and fast from the Linux server; Write speeds from the > clients to both servers are fast. (Clients on the gigabit network work > fine with blazing read and write speeds). The network traces shows > congestion packets for both servers when doing reads from the clients > (dup acks and retransmissions), but the Linux server seem to handle the > congestion better. ECN is not enabled on the network and I don't see any > congestion windowing or clients window changing. The 100MB/1G switch > > is dropping packets. I double checked the network configuration and > also swapped swithports for the servers to use the others to make sure > the switch configuration are the same, and the Linux always does better > than FreeBSD. Assuming that the network configuration is a constant for > all clients and servers (speed, duplex, and etc...), the only variable > is the servers themselves (Linux and FreeBSD). I have tried a couple of > FreeBSD machines with 6.1 and 7.0 and they exhibit the same problem, > with no luck matching the speed and network utilization of Linux (2 > years old). The read speed test I'm referring is doing transferring of > a 100MB file (cifs, nfs, and ftp), and the Linux server does it > consistently in around 10 sec (line speed) with a constant network > utilization chart, while the FreeBSD servers are magnitudes slower with > erratic network utilization chart. I've attempted to tweak some network > sysctl options on the FreeBSD, and the only ones that helped were > disabling TSO and inflight; which leads me to think that the > inter-packet gap was slightly increased to partially relieve congestion > on the switch; not a long term solution. > > =20 > > My questions are:=20 > > 1. Have you heard of this problem before with 100MB clients to Gigabit > servers? > > 2. Are you aware of any Linux fix/patch in the TCP stack to better > handling congestion than FreeBSD? I'm looking to address this issue in > the FreeBSD, but wondering if the Linux stack did something special that > can help with the FreeBSD performance. > > =20 > > David K. > > =20 > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > =20 _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 02:22:02 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 766B91065677; Wed, 2 Jul 2008 02:22:02 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id 33A378FC17; Wed, 2 Jul 2008 02:22:02 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.2/8.14.2) with ESMTP id m622Lx1k087476; Tue, 1 Jul 2008 22:21:59 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m622Lxnf088882 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 1 Jul 2008 22:21:59 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200807020221.m622Lxnf088882@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Tue, 01 Jul 2008 22:21:53 -0400 To: "Bjoern A. Zeeb" , Andre Oppermann From: Mike Tancsa In-Reply-To: <20080701092254.T57089@maildrop.int.zabbadoz.net> References: <4852E23E.2040505@gtcomm.net> <4854EBF1.7020708@FreeBSD.org> <200807010606.m6166jFe084204@lava.sentex.ca> <4869EC1E.8060009@freebsd.org> <20080701084933.W57089@maildrop.int.zabbadoz.net> <20080701092254.T57089@maildrop.int.zabbadoz.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: freebsd-net@freebsd.org, "Bruce M. Simpson" , Paul Subject: Re: Route messages X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 02:22:02 -0000 At 05:24 AM 7/1/2008, Bjoern A. Zeeb wrote: >So I had a very quick look at the code between doing something else. >I think the only change needed is this if I am not mistaken but my >head is far away nowhere close enough in this code. Hi, The patch seems to work in that there is not an RTM_MISS message generated per packet forwarded on my test box. Is it the "final" / correct version ? ---Mike From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 05:10:09 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 279D41065680 for ; Wed, 2 Jul 2008 05:10:09 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mx.egr.msu.edu (surfnturf.egr.msu.edu [35.9.37.164]) by mx1.freebsd.org (Postfix) with ESMTP id DD9928FC22 for ; Wed, 2 Jul 2008 05:10:08 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from localhost (localhost [127.0.0.1]) by mx.egr.msu.edu (Postfix) with ESMTP id C173D71F092; Wed, 2 Jul 2008 00:53:29 -0400 (EDT) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mx.egr.msu.edu ([127.0.0.1]) by localhost (surfnturf.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id WTLFQJi+xn1s; Wed, 2 Jul 2008 00:53:29 -0400 (EDT) Received: from localhost (daemon.egr.msu.edu [35.9.44.65]) by mx.egr.msu.edu (Postfix) with ESMTP id 8BD7D71F08A; Wed, 2 Jul 2008 00:53:29 -0400 (EDT) Received: by localhost (Postfix, from userid 21281) id 896DF34A; Wed, 2 Jul 2008 00:53:29 -0400 (EDT) Date: Wed, 2 Jul 2008 00:53:29 -0400 From: Adam McDougall To: David Kwan Message-ID: <20080702045329.GT23350@egr.msu.edu> References: <486A91B0.6040505@gtcomm.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-net@freebsd.org Subject: Re: Poor network performance for clients in 100MB toGigabit environment X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 05:10:09 -0000 Are the NFS mounts UDP or TCP on Linux and FreeBSD? I believe FreeBSD still defaults to UDP which can act differently especially for NFS. On Tue, Jul 01, 2008 at 05:30:35PM -0700, David Kwan wrote: I've attempt many standard and non-standard permutations of the tcp tuning parameters without much successful via sysctl. It feels like FreeBSD is not handling the congestion very well and is beyond tuning sysctl. It's just clients on the 100MB networks has slow/erratic reads; Clients on the Gigabit network are fine and screams, so the original tcp parameters are just fine for them. David K. -----Original Message----- From: owner-freebsd-net@freebsd.org [mailto:owner-freebsd-net@freebsd.org] On Behalf Of Paul Sent: Tuesday, July 01, 2008 1:21 PM To: David Kwan Cc: freebsd-net@freebsd.org Subject: Re: Poor network performance for clients in 100MB toGigabit environment What options do you have enabled on the linux server? sysctl -a | grep net.ipv4.tcp and on the bsd sysctl -a net.inet.tcp It sounds like a problem with BSD not handing the dropped data or ack packets so what happens is it pushes a burst of data out > 100mbit and the switch drops the packets and then BSD waits too long to recover and doesn't scale the transmission back. TCP is supposed to scale down the transmission speed until packets are not dropped to a point even without ECN. Options such as 'reno' and 'sack' etc. are congestion control algorithms that use congestion windows. David Kwan wrote: > I have a couple of questions regarding the TCP Stack: > > > > I have a situation with clients on a 100MB network connecting to servers > on a Gigabit network where the client read speeds are very slow from the > FreeBSD server and fast from the Linux server; Write speeds from the > clients to both servers are fast. (Clients on the gigabit network work > fine with blazing read and write speeds). The network traces shows > congestion packets for both servers when doing reads from the clients > (dup acks and retransmissions), but the Linux server seem to handle the > congestion better. ECN is not enabled on the network and I don't see any > congestion windowing or clients window changing. The 100MB/1G switch > > is dropping packets. I double checked the network configuration and > also swapped swithports for the servers to use the others to make sure > the switch configuration are the same, and the Linux always does better > than FreeBSD. Assuming that the network configuration is a constant for > all clients and servers (speed, duplex, and etc...), the only variable > is the servers themselves (Linux and FreeBSD). I have tried a couple of > FreeBSD machines with 6.1 and 7.0 and they exhibit the same problem, > with no luck matching the speed and network utilization of Linux (2 > years old). The read speed test I'm referring is doing transferring of > a 100MB file (cifs, nfs, and ftp), and the Linux server does it > consistently in around 10 sec (line speed) with a constant network > utilization chart, while the FreeBSD servers are magnitudes slower with > erratic network utilization chart. I've attempted to tweak some network > sysctl options on the FreeBSD, and the only ones that helped were > disabling TSO and inflight; which leads me to think that the > inter-packet gap was slightly increased to partially relieve congestion > on the switch; not a long term solution. > > > > My questions are: > > 1. Have you heard of this problem before with 100MB clients to Gigabit > servers? > > 2. Are you aware of any Linux fix/patch in the TCP stack to better > handling congestion than FreeBSD? I'm looking to address this issue in > the FreeBSD, but wondering if the Linux stack did something special that > can help with the FreeBSD performance. > > > > David K. > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" _______________________________________________ freebsd-net@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 08:48:35 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E69DD1065678 for ; Wed, 2 Jul 2008 08:48:35 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 53CB78FC19 for ; Wed, 2 Jul 2008 08:48:34 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 27995 invoked from network); 2 Jul 2008 10:48:32 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 2 Jul 2008 10:48:32 +0200 Date: Wed, 2 Jul 2008 10:48:32 +0200 (CEST) From: Ingo Flaschberger To: David Kwan In-Reply-To: Message-ID: References: User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org Subject: Re: Poor network performance for clients in 100MB to Gigabit environment X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 08:48:36 -0000 Dear David, try to enable flow-control at the gig-e switch and freebsd network card. Kind regards, ingo flaschberger geschaeftsleitung --------------------------- netstorage-crossip-flat:fee powered by crossip communications gmbh --------------------------- sebastian kneipp gasse 1 a-1020 wien fix: +43-1-726 15 22-217 fax: +43-1-726 15 22-111 --------------------------- On Tue, 1 Jul 2008, David Kwan wrote: > I have a couple of questions regarding the TCP Stack: > > > > I have a situation with clients on a 100MB network connecting to servers > on a Gigabit network where the client read speeds are very slow from the > FreeBSD server and fast from the Linux server; Write speeds from the > clients to both servers are fast. (Clients on the gigabit network work > fine with blazing read and write speeds). The network traces shows > congestion packets for both servers when doing reads from the clients > (dup acks and retransmissions), but the Linux server seem to handle the > congestion better. ECN is not enabled on the network and I don't see any > congestion windowing or clients window changing. The 100MB/1G switch > > is dropping packets. I double checked the network configuration and > also swapped swithports for the servers to use the others to make sure > the switch configuration are the same, and the Linux always does better > than FreeBSD. Assuming that the network configuration is a constant for > all clients and servers (speed, duplex, and etc...), the only variable > is the servers themselves (Linux and FreeBSD). I have tried a couple of > FreeBSD machines with 6.1 and 7.0 and they exhibit the same problem, > with no luck matching the speed and network utilization of Linux (2 > years old). The read speed test I'm referring is doing transferring of > a 100MB file (cifs, nfs, and ftp), and the Linux server does it > consistently in around 10 sec (line speed) with a constant network > utilization chart, while the FreeBSD servers are magnitudes slower with > erratic network utilization chart. I've attempted to tweak some network > sysctl options on the FreeBSD, and the only ones that helped were > disabling TSO and inflight; which leads me to think that the > inter-packet gap was slightly increased to partially relieve congestion > on the switch; not a long term solution. > > > > My questions are: > > 1. Have you heard of this problem before with 100MB clients to Gigabit > servers? > > 2. Are you aware of any Linux fix/patch in the TCP stack to better > handling congestion than FreeBSD? I'm looking to address this issue in > the FreeBSD, but wondering if the Linux stack did something special that > can help with the FreeBSD performance. > > > > David K. > > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 08:50:35 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1F2901065682 for ; Wed, 2 Jul 2008 08:50:35 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id CFC1D8FC26 for ; Wed, 2 Jul 2008 08:50:34 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDxzQ-0007sm-LX for freebsd-net@freebsd.org; Wed, 02 Jul 2008 04:47:00 -0400 Message-ID: <486B41D5.3060609@gtcomm.net> Date: Wed, 02 Jul 2008 04:52:37 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: FreeBSD Net References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> In-Reply-To: <486A9A0E.6060308@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 08:50:35 -0000 SMP DISABLED on my Opteron 2212 (ULE, Preemption on) Yields ~750kpps in em0 and out em1 (one direction) I am miffed why this yields more pps than a) with all 4 cpus running and b) 4 cpus with lagg load balanced over 3 incoming connections so 3 taskq threads I would be willing to set up test equipment (several servers plugged into a switch) with ipkvm and power port access if someone or a group of people want to figure out ways to improve the routing process, ipfw, and lagg. Maximum PPS with one ipfw rule on UP: tops out about 570Kpps.. almost 200kpps lower ? (frown) I'm going to drop in a 3ghz opteron instead of the 2ghz 2212 that's in here and see how that scales, using UP same kernel etc I have now. Julian Elischer wrote: > Paul wrote: >> ULE without PREEMPTION is now yeilding better results. >> input (em0) output >> packets errs bytes packets errs bytes colls >> 571595 40639 34564108 1 0 226 0 >> 577892 48865 34941908 1 0 178 0 >> 545240 84744 32966404 1 0 178 0 >> 587661 44691 35534512 1 0 178 0 >> 587839 38073 35544904 1 0 178 0 >> 587787 43556 35540360 1 0 178 0 >> 540786 39492 32712746 1 0 178 0 >> 572071 55797 34595650 1 0 178 0 >> >> *OUCH, IPFW HURTS.. >> loading ipfw, and adding one ipfw rule allow ip from any to any drops >> 100Kpps off :/ what's up with THAT? >> unloaded ipfw module and back 100kpps more again, that's not right >> with ONE rule.. :/ > > ipfw need sto gain a lock on hte firewall before running, > and is quite complex.. I can believe it.. > > in FreeBSD 4.8 I was able to use ipfw and filter 1Gb between two > interfaces (bridged) but I think it has slowed down since then due to > the SMP locking. > > >> >> em0 taskq is still jumping cpus.. is there any way to lock it to one >> cpu or is this just a function of ULE >> >> running a tar czpvf all.tgz * and seeing if pps changes.. >> negligible.. guess scheduler is doing it's job at least.. >> >> Hmm. even when it's getting 50-60k errors per second on the interface >> I can still SCP a file through that interface although it's not >> fast.. 3-4MB/s.. >> >> You know, I wouldn't care if it added 5ms latency to the packets when >> it was doing 1mpps as long as it didn't drop any.. Why can't it do >> that? Queue them up and do them in bigggg chunks so none are >> dropped........hmm? >> >> 32 bit system is compiling now.. won't do > 400kpps with GENERIC >> kernel, as with 64 bit did 450k with GENERIC, although that could be >> the difference between opteron 270 and opteron 2212.. >> >> Paul >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > > From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 08:54:01 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EA6D31065672 for ; Wed, 2 Jul 2008 08:54:01 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 33F8A8FC17 for ; Wed, 2 Jul 2008 08:54:00 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 30532 invoked from network); 2 Jul 2008 10:54:00 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 2 Jul 2008 10:54:00 +0200 Date: Wed, 2 Jul 2008 10:54:00 +0200 (CEST) From: Ingo Flaschberger To: Paul In-Reply-To: <486B41D5.3060609@gtcomm.net> Message-ID: References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 08:54:02 -0000 Dear Paul, > SMP DISABLED on my Opteron 2212 (ULE, Preemption on) > Yields ~750kpps in em0 and out em1 (one direction) > I am miffed why this yields more pps than > a) with all 4 cpus running and b) 4 cpus with lagg load balanced over 3 > incoming connections so 3 taskq threads because less locking, less synchronisation, .... > I would be willing to set up test equipment (several servers plugged into a > switch) with ipkvm and power port access > if someone or a group of people want to figure out ways to improve the > routing process, ipfw, and lagg. > > Maximum PPS with one ipfw rule on UP: > tops out about 570Kpps.. almost 200kpps lower ? (frown) can you post the rule here? > I'm going to drop in a 3ghz opteron instead of the 2ghz 2212 that's in here > and see how that scales, using UP same kernel etc I have now. really, please try 32bit and 1 cpu. Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 09:47:07 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 673FE1065674 for ; Wed, 2 Jul 2008 09:47:07 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 214EA8FC14 for ; Wed, 2 Jul 2008 09:47:07 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KDys3-0003rf-Q1; Wed, 02 Jul 2008 05:43:27 -0400 Message-ID: <486B4F11.6040906@gtcomm.net> Date: Wed, 02 Jul 2008 05:49:05 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 09:47:07 -0000 Ipfw rule was simply allow ip from any to any :) This is 64bit i'm testing now.. I have a 32 bit install I tested on another machine but it only has bge NIC and wasn't performing as well so I'll reinstall 32 bit on this 2212 and test then drop in the 2222 (3ghz) and test. I still don't like the huge hit ipfw and lagg take :/ ** I tried polling in UP mode and I got some VERY interesting results.. CPU is 44% idle (idle polling isn't on) but I'm getting errors! It's doing 530kpps with ipfw loaded, which without polling uses 100% cpu but now it says my cpu is 44% idle? that makes no sense.. If it was idle why am I getting errors? I only get errors when em taskq was eating 100% cpu.. Idle polling on/off makes no difference. user_frac is set to 5 .. last pid: 1598; load averages: 0.01, 0.16, 0.43 up 0+00:34:41 04:04:43 66 processes: 2 running, 46 sleeping, 18 waiting CPU: 0.0% user, 0.0% nice, 7.3% system, 46.5% interrupt, 46.2% idle Mem: 8064K Active, 6808K Inact, 43M Wired, 92K Cache, 9264K Buf, 1923M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 10 root 171 ki31 0K 16K RUN 10:10 88.87% idle 1598 root 45 0 8084K 2052K RUN 0:00 1.12% top 11 root -32 - 0K 16K WAIT 0:02 0.24% swi4: clock sio 13 root -44 - 0K 16K WAIT 14:13 0.15% swi1: net 1329 root 44 0 33732K 4572K select 0:00 0.05% sshd input (em0) output packets errs bytes packets errs bytes colls 541186 68741 33107504 1 0 0 0 540036 70611 33044632 1 0 178 0 540470 66493 33043148 1 0 178 0 541903 67981 33125414 1 0 178 0 541238 84979 33105898 1 0 178 0 541338 74067 33115984 2 0 356 0 539116 49286 32991516 2 0 220 0 kldunload ipfw....... input (em0) output packets errs bytes packets errs bytes colls 600589 0 36751064 1 0 226 0 606294 0 37102868 2 0 220 0 616802 0 37733866 1 0 178 0 623017 0 38117436 1 0 178 0 624800 0 38225470 1 0 178 0 626791 0 38347426 1 0 178 0 last pid: 1605; load averages: 0.00, 0.13, 0.40 up 0+00:35:30 04:05:32 66 processes: 2 running, 46 sleeping, 18 waiting CPU: 0.0% user, 0.0% nice, 7.1% system, 36.0% interrupt, 56.9% idle Mem: 8064K Active, 6812K Inact, 43M Wired, 92K Cache, 9264K Buf, 1923M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 10 root 171 ki31 0K 16K RUN 10:16 95.36% idle 13 root -44 - 0K 16K WAIT 14:53 0.24% swi1: net 36 root -68 - 0K 16K - 1:03 0.10% em3 taskq 1605 root 44 0 8084K 2052K RUN 0:00 0.10% top 11 root -32 - 0K 16K WAIT 0:02 0.05% swi4: clock sio add some more PPS...... input (em0) output packets errs bytes packets errs bytes colls 749015 169684 46438936 1 0 42 0 749176 184574 46448916 1 0 178 0 759576 188462 47093716 1 0 178 0 762904 182854 47300052 1 0 178 0 798039 147509 49478422 1 0 178 0 759528 194297 47090740 1 0 178 0 746849 195935 46304642 1 0 178 0 747566 186703 46349096 1 0 178 0 750011 181630 46500702 2 last pid: 1607; load averages: 0.19, 0.17, 0.40 up 0+00:36:18 04:06:20 66 processes: 2 running, 46 sleeping, 18 waiting CPU: 0.0% user, 0.0% nice, 12.5% system, 45.4% interrupt, 42.1% idle Mem: 8068K Active, 6808K Inact, 43M Wired, 92K Cache, 9264K Buf, 1923M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 10 root 171 ki31 0K 16K RUN 10:21 85.64% idle 36 root -68 - 0K 16K - 1:07 3.61% em3 taskq 1607 root 44 0 8084K 2052K RUN 0:00 0.93% top 13 root -44 - 0K 16K WAIT 15:32 0.20% swi1: net 11 root -32 - 0K 16K WAIT 0:02 0.05% swi4: clock sio So my maximum without polling is close to 800kpps but if I push that it starts locking me from doing things, or my maximum is 750kpps with polling and the console is very responsive? How on EARTH can my CPU be 42% idle with polling and i'm getting all these errors.. The whole thing makes no sense, something is bugged somewheres.. HZ=2000 for this test (512/512 descriptors) If i lower HZ to 100, I can get a little over 800kpps without polling.. --------Going to reboot with 4000hz and 1024k rx/tx descriptors .......... about the same.. input (em0) output packets errs bytes packets errs bytes colls 720833 244835 44691662 1 0 178 0 744746 215689 46174256 1 0 178 0 744943 194252 46186470 1 0 178 0 743685 199487 46108486 2 0 356 0 743715 209263 46110346 2 0 356 0 last pid: 1426; load averages: 0.22, 0.65, 0.40 up 0+00:07:17 04:16:43 66 processes: 2 running, 46 sleeping, 18 waiting CPU: 0.4% user, 0.0% nice, 12.8% system, 44.2% interrupt, 42.6% idle Mem: 8052K Active, 6192K Inact, 46M Wired, 96K Cache, 8944K Buf, 1921M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 10 root 171 ki31 0K 16K RUN 0:49 82.52% idle 36 root -68 - 0K 16K - 0:31 6.84% em3 taskq 1426 root 45 0 8084K 2052K RUN 0:00 1.32% top 13 root -44 - 0K 16K WAIT 3:07 0.59% swi1: net 11 root -32 - 0K 16K WAIT 0:00 0.05% swi4: clock sio ------reboot with 2048/2048 descriptors NOTE: without polling, 128,256,512 give best performance for some strange reason, maybe cache hits this is worse.. input (em0) output packets errs bytes packets errs bytes colls 646290 269912 40080528 0 0 0 0 672548 250198 41687440 1 0 178 0 674856 247162 41841076 1 0 178 0 665062 248851 41233848 1 0 178 0 671764 253300 41649372 bah.. ------- 10000HZ, 512/512 CPU still will not go below 42% idle 700-720 kpps.. actualyl got 40% cpu idle lol Oh well.. Tomorrow hopefully 2222 test and 32 bit test.. then i'm done for while.. :P Paul Ingo Flaschberger wrote: > Dear Paul, > >> SMP DISABLED on my Opteron 2212 (ULE, Preemption on) >> Yields ~750kpps in em0 and out em1 (one direction) >> I am miffed why this yields more pps than >> a) with all 4 cpus running and b) 4 cpus with lagg load balanced over >> 3 incoming connections so 3 taskq threads > > because less locking, less synchronisation, .... > >> I would be willing to set up test equipment (several servers plugged >> into a switch) with ipkvm and power port access >> if someone or a group of people want to figure out ways to improve >> the routing process, ipfw, and lagg. >> >> Maximum PPS with one ipfw rule on UP: >> tops out about 570Kpps.. almost 200kpps lower ? (frown) > > can you post the rule here? > >> I'm going to drop in a 3ghz opteron instead of the 2ghz 2212 that's >> in here and see how that scales, using UP same kernel etc I have now. > > really, please try 32bit and 1 cpu. > > Kind regards, > Ingo Flaschberger > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 10:05:52 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9CB361065672 for ; Wed, 2 Jul 2008 10:05:52 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id ECBA68FC12 for ; Wed, 2 Jul 2008 10:05:51 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 7835 invoked from network); 2 Jul 2008 12:05:49 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 2 Jul 2008 12:05:49 +0200 Date: Wed, 2 Jul 2008 12:05:49 +0200 (CEST) From: Ingo Flaschberger To: Paul In-Reply-To: <486B4F11.6040906@gtcomm.net> Message-ID: References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 10:05:52 -0000 Dear Paul, > I still don't like the huge hit ipfw and lagg take :/ I think, you can't use fastforward with with lagg. > ** I tried polling in UP mode and I got some VERY interesting results.. > CPU is 44% idle (idle polling isn't on) but I'm getting errors! It's doing > 530kpps with ipfw loaded, which without polling uses 100% cpu but now it says > my cpu is 44% idle? that makes no sense.. If it was idle why am I getting > errors? I only get errors when em taskq was eating 100% cpu.. > Idle polling on/off makes no difference. > user_frac is set to 5 .. what are your values: kern.polling.reg_frac= kern.polling.user_frac= kern.polling.burst_max= I use: kern.polling.reg_frac=20 kern.polling.user_frac=20 kern.polling.burst_max=512 if you need more than 1000, you need to change the code: src/sys/kern/kern_poll.c #define MAX_POLL_BURST_MAX 1000 > So my maximum without polling is close to 800kpps but if I push that it > starts locking me from doing things, or how many kpps do you want to achieve? > HZ=2000 for this test (512/512 descriptors) you mean: hw.em.rxd=512 hw.em.txd=512 ? can you try with polling: hw.em.rxd=4096 hw.em.txd=4096 Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 13:02:47 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 35D101065674 for ; Wed, 2 Jul 2008 13:02:47 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id D13A08FC14 for ; Wed, 2 Jul 2008 13:02:46 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id EBB721B10EE7; Wed, 2 Jul 2008 15:02:44 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on malcho.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.4 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 0E3F61B10CB6; Wed, 2 Jul 2008 15:02:34 +0200 (CEST) Message-ID: <486B7C69.1010304@moneybookers.com> Date: Wed, 02 Jul 2008 16:02:33 +0300 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.14 (X11/20080616) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on blah.cmotd.com X-Virus-Status: Clean Cc: FreeBSD Net , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 13:02:47 -0000 Hi Ingo Flaschberger wrote: > Dear Paul, > >> I still don't like the huge hit ipfw and lagg take :/ You have to try PF, then you will respect IPFW again ;) -cut- > >> So my maximum without polling is close to 800kpps but if I push that >> it starts locking me from doing things, or > > how many kpps do you want to achieve? Do not know for Paul but, I want to be able to route (and/or bridge to handle) 600-700mbps syn flood, which is something like 1500kpps in every direction. Is it unrealistic? If the code is optimized to fully utilize MP I do not see a reason why quad core processor should not be able to do this. After all single core seems to handle 500kpps, if we utilize four, eight or even more cores we should be able to route 1500kpps + ? I hope TOE once MFCed to 7-STABLE will help too? -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 13:06:04 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ED21B1065671 for ; Wed, 2 Jul 2008 13:06:04 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from rv-out-0506.google.com (rv-out-0506.google.com [209.85.198.233]) by mx1.freebsd.org (Postfix) with ESMTP id B89038FC1B for ; Wed, 2 Jul 2008 13:06:04 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by rv-out-0506.google.com with SMTP id b25so480989rvf.43 for ; Wed, 02 Jul 2008 06:06:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:sender :to:subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references :x-google-sender-auth; bh=3RLlvKJcSOVpuMxDtDi+RJbe2Xf6ze94jBil7DBaFWA=; b=EKNDNPWuCRUJDvDyAZHmPA/swWDKD9++lwVXwSusljISYxQZUVE1lnNJgAeyJGysrp Bwm6fzvTTwFrqBXHC+wmzRcM0TcRgyanmin5gFMAhfMkXInzHXdC/e6LPg/T2UC/MieT pu2x8Hb84onjWcdoYG1PzUAQOLVkXAD89HDYk= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references:x-google-sender-auth; b=Fl7nqVNO1u9/MPMcfYmWeFbXgoW2lUtY5wiUPyH6DdtQETCy04LdkIutFjI6jt1cpp 5kS69gOHM164pOvDjEY7LGGd6fAh6YB0cgFppSSPGjia4eWGQn0YuWyKfmMtN9KeLf6G 1brRIoy0DtlJMmQhD/a/3ZMCVADvTIgy5N5SQ= Received: by 10.141.23.7 with SMTP id a7mr4392275rvj.58.1215003964228; Wed, 02 Jul 2008 06:06:04 -0700 (PDT) Received: by 10.141.212.9 with HTTP; Wed, 2 Jul 2008 06:06:04 -0700 (PDT) Message-ID: Date: Wed, 2 Jul 2008 21:06:04 +0800 From: "Adrian Chadd" Sender: adrian.chadd@gmail.com To: "Stefan Lambrev" In-Reply-To: <486B7C69.1010304@moneybookers.com> MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4867420D.7090406@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486B7C69.1010304@moneybookers.com> X-Google-Sender-Auth: bc25d5e11823fda7 Cc: FreeBSD Net , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 13:06:05 -0000 2008/7/2 Stefan Lambrev : > Do not know for Paul but, I want to be able to route (and/or bridge to > handle) 600-700mbps syn flood, > which is something like 1500kpps in every direction. Is it unrealistic? > If the code is optimized to fully utilize MP I do not see a reason why quad > core processor should not be able to do this. > After all single core seems to handle 500kpps, if we utilize four, eight or > even more cores we should be able to route 1500kpps + ? > I hope TOE once MFCed to 7-STABLE will help too? But its not just about CPU use, its about your NIC, your IO bus path, your memory interface, your caches .. things get screwy. Especially if you're holding a full internet routing table. If you're interested in participating in a group funding project to make this happen then let me know. The more the merrier (read: the more that can be achieved :) Adrian From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 13:46:42 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C99D1065672; Wed, 2 Jul 2008 13:46:42 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id A54268FC24; Wed, 2 Jul 2008 13:46:41 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.2/8.14.2) with ESMTP id m62Dkd46052322; Wed, 2 Jul 2008 09:46:39 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m62DkdHx091961 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 2 Jul 2008 09:46:39 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200807021346.m62DkdHx091961@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Wed, 02 Jul 2008 09:46:29 -0400 To: "Bjoern A. Zeeb" , Andre Oppermann From: Mike Tancsa In-Reply-To: <20080701092254.T57089@maildrop.int.zabbadoz.net> References: <4852E23E.2040505@gtcomm.net> <4854EBF1.7020708@FreeBSD.org> <200807010606.m6166jFe084204@lava.sentex.ca> <4869EC1E.8060009@freebsd.org> <20080701084933.W57089@maildrop.int.zabbadoz.net> <20080701092254.T57089@maildrop.int.zabbadoz.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: freebsd-net@freebsd.org, Paul Subject: Re: Route messages X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 13:46:42 -0000 At 05:24 AM 7/1/2008, Bjoern A. Zeeb wrote: >On Tue, 1 Jul 2008, Bjoern A. Zeeb wrote: > >Hi, > >>On Tue, 1 Jul 2008, Andre Oppermann wrote: >> >>Hi, >> >>>Mike Tancsa wrote: >>>>I am thinking >>>>http://lists.freebsd.org/pipermail/cvs-src/2008-April/090303.html >>>>is the commit ? If I revert to the prev version, the issue goes away. >> >>Ha, I finally know why I ended up on Cc: of a thread I had no idea >>about. Someone could have told me instead of blindly adding me;-) >> >> >>>Yes, this change doesn't look right. It should only do the route >>>lookup in ip_input.c when there was an EMSGSIZE error returned by >>>ip_output(). The rtalloc_ign() call causes the message to be sent >>>because it always sets report to one. The default message is RTM_MISS. >>>I'll try to prep an updated patch which doesn't have these issues later >>>today. >> >>Yeah my bad. Sorry. >> >>If you do that, do not do an extra route lookup if possible, correct >>the rtalloc call. Thanks. > >So I had a very quick look at the code between doing something else. >I think the only change needed is this if I am not mistaken but my >head is far away nowhere close enough in this code. > >Andre, could you review this? > >Index: sys/netinet/ip_input.c >=================================================================== >RCS file: /shared/mirror/FreeBSD/r/ncvs/src/sys/netinet/ip_input.c,v >retrieving revision 1.332.2.2 >diff -u -p -r1.332.2.2 ip_input.c >--- sys/netinet/ip_input.c 22 Apr 2008 12:02:55 -0000 1.332.2.2 >+++ sys/netinet/ip_input.c 1 Jul 2008 09:23:08 -0000 >@@ -1363,7 +1363,6 @@ ip_forward(struct mbuf *m, int srcrt) > * the ICMP_UNREACH_NEEDFRAG "Next-Hop MTU" field described > in RFC1191. > */ > bzero(&ro, sizeof(ro)); >- rtalloc_ign(&ro, RTF_CLONING); > > error = ip_output(m, NULL, &ro, IP_FORWARDING, NULL, NULL); > This could also potentially close http://www.freebsd.org/cgi/query-pr.cgi?pr=124540 http://www.freebsd.org/cgi/query-pr.cgi?pr=123621 ---Mike From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 13:51:24 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 78A5A106567E for ; Wed, 2 Jul 2008 13:51:24 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 9DF408FC17 for ; Wed, 2 Jul 2008 13:51:23 +0000 (UTC) (envelope-from andre@freebsd.org) Received: (qmail 14361 invoked from network); 2 Jul 2008 12:42:46 -0000 Received: from localhost (HELO [127.0.0.1]) ([127.0.0.1]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 2 Jul 2008 12:42:46 -0000 Message-ID: <486B87DB.3080007@freebsd.org> Date: Wed, 02 Jul 2008 15:51:23 +0200 From: Andre Oppermann User-Agent: Thunderbird 1.5.0.14 (Windows/20071210) MIME-Version: 1.0 To: "Bjoern A. Zeeb" References: <4852E23E.2040505@gtcomm.net> <4854EBF1.7020708@FreeBSD.org> <200807010606.m6166jFe084204@lava.sentex.ca> <4869EC1E.8060009@freebsd.org> <20080701084933.W57089@maildrop.int.zabbadoz.net> <20080701092254.T57089@maildrop.int.zabbadoz.net> In-Reply-To: <20080701092254.T57089@maildrop.int.zabbadoz.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Mike Tancsa , "Bruce M. Simpson" , Paul Subject: Re: Route messages X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 13:51:24 -0000 Bjoern A. Zeeb wrote: > On Tue, 1 Jul 2008, Bjoern A. Zeeb wrote: > > Hi, > >> On Tue, 1 Jul 2008, Andre Oppermann wrote: >> >> Hi, >> >>> Mike Tancsa wrote: >>>> I am thinking >>>> >>>> http://lists.freebsd.org/pipermail/cvs-src/2008-April/090303.html >>>> is the commit ? If I revert to the prev version, the issue goes away. >> >> Ha, I finally know why I ended up on Cc: of a thread I had no idea >> about. Someone could have told me instead of blindly adding me;-) >> >> >>> Yes, this change doesn't look right. It should only do the route >>> lookup in ip_input.c when there was an EMSGSIZE error returned by >>> ip_output(). The rtalloc_ign() call causes the message to be sent >>> because it always sets report to one. The default message is RTM_MISS. >>> >>> I'll try to prep an updated patch which doesn't have these issues later >>> today. >> >> Yeah my bad. Sorry. >> >> If you do that, do not do an extra route lookup if possible, correct >> the rtalloc call. Thanks. > > So I had a very quick look at the code between doing something else. > I think the only change needed is this if I am not mistaken but my > head is far away nowhere close enough in this code. > > Andre, could you review this? Yes, this should fix the problem. I haven't tested the patch though. -- Andre > Index: sys/netinet/ip_input.c > =================================================================== > RCS file: /shared/mirror/FreeBSD/r/ncvs/src/sys/netinet/ip_input.c,v > retrieving revision 1.332.2.2 > diff -u -p -r1.332.2.2 ip_input.c > --- sys/netinet/ip_input.c 22 Apr 2008 12:02:55 -0000 1.332.2.2 > +++ sys/netinet/ip_input.c 1 Jul 2008 09:23:08 -0000 > @@ -1363,7 +1363,6 @@ ip_forward(struct mbuf *m, int srcrt) > * the ICMP_UNREACH_NEEDFRAG "Next-Hop MTU" field described in > RFC1191. > */ > bzero(&ro, sizeof(ro)); > - rtalloc_ign(&ro, RTF_CLONING); > > error = ip_output(m, NULL, &ro, IP_FORWARDING, NULL, NULL); > > From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 13:53:22 2008 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 85AD9106567E; Wed, 2 Jul 2008 13:53:22 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 5D6D08FC12; Wed, 2 Jul 2008 13:53:22 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from freefall.freebsd.org (bz@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m62DrMsY048490; Wed, 2 Jul 2008 13:53:22 GMT (envelope-from bz@freefall.freebsd.org) Received: (from bz@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m62DrMnx048486; Wed, 2 Jul 2008 13:53:22 GMT (envelope-from bz) Date: Wed, 2 Jul 2008 13:53:22 GMT Message-Id: <200807021353.m62DrMnx048486@freefall.freebsd.org> To: bz@FreeBSD.org, freebsd-net@FreeBSD.org, bz@FreeBSD.org From: bz@FreeBSD.org Cc: Subject: Re: kern/124540: [tcp] RTM_MISS with the transit packets X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 13:53:22 -0000 Synopsis: [tcp] RTM_MISS with the transit packets Responsible-Changed-From-To: freebsd-net->bz Responsible-Changed-By: bz Responsible-Changed-When: Wed Jul 2 13:52:56 UTC 2008 Responsible-Changed-Why: My fault most likely, patch out for review already. http://www.freebsd.org/cgi/query-pr.cgi?pr=124540 From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 13:55:07 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4ACC81065675; Wed, 2 Jul 2008 13:55:07 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from mail.cksoft.de (mail.cksoft.de [62.111.66.27]) by mx1.freebsd.org (Postfix) with ESMTP id 009778FC25; Wed, 2 Jul 2008 13:55:06 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from localhost (amavis.str.cksoft.de [192.168.74.71]) by mail.cksoft.de (Postfix) with ESMTP id 4EFBF41C83F; Wed, 2 Jul 2008 15:55:05 +0200 (CEST) X-Virus-Scanned: amavisd-new at cksoft.de Received: from mail.cksoft.de ([62.111.66.27]) by localhost (amavis.str.cksoft.de [192.168.74.71]) (amavisd-new, port 10024) with ESMTP id 5o0akjclbU8U; Wed, 2 Jul 2008 15:55:05 +0200 (CEST) Received: by mail.cksoft.de (Postfix, from userid 66) id F128541C83C; Wed, 2 Jul 2008 15:55:04 +0200 (CEST) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id 02CF444487F; Wed, 2 Jul 2008 13:54:55 +0000 (UTC) Date: Wed, 2 Jul 2008 13:54:55 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: Mike Tancsa In-Reply-To: <200807021346.m62DkdHx091961@lava.sentex.ca> Message-ID: <20080702135402.E57089@maildrop.int.zabbadoz.net> References: <4852E23E.2040505@gtcomm.net> <4854EBF1.7020708@FreeBSD.org> <200807010606.m6166jFe084204@lava.sentex.ca> <4869EC1E.8060009@freebsd.org> <20080701084933.W57089@maildrop.int.zabbadoz.net> <20080701092254.T57089@maildrop.int.zabbadoz.net> <200807021346.m62DkdHx091961@lava.sentex.ca> X-OpenPGP-Key: 0x14003F198FEFA3E77207EE8D2B58B8F83CCF1842 MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org, Andre Oppermann , Paul Subject: Re: Route messages X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 13:55:07 -0000 On Wed, 2 Jul 2008, Mike Tancsa wrote: Hi, >> Index: sys/netinet/ip_input.c >> =================================================================== >> RCS file: /shared/mirror/FreeBSD/r/ncvs/src/sys/netinet/ip_input.c,v >> retrieving revision 1.332.2.2 >> diff -u -p -r1.332.2.2 ip_input.c >> --- sys/netinet/ip_input.c 22 Apr 2008 12:02:55 -0000 1.332.2.2 >> +++ sys/netinet/ip_input.c 1 Jul 2008 09:23:08 -0000 >> @@ -1363,7 +1363,6 @@ ip_forward(struct mbuf *m, int srcrt) >> * the ICMP_UNREACH_NEEDFRAG "Next-Hop MTU" field described in >> RFC1191. >> */ >> bzero(&ro, sizeof(ro)); >> - rtalloc_ign(&ro, RTF_CLONING); >> >> error = ip_output(m, NULL, &ro, IP_FORWARDING, NULL, NULL); >> Still waiting on any second pairs of eyes to review this. > This could also potentially close > > http://www.freebsd.org/cgi/query-pr.cgi?pr=124540 > http://www.freebsd.org/cgi/query-pr.cgi?pr=123621 taken, will handle them. -- Bjoern A. Zeeb Stop bit received. Insert coin for new game. From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 13:55:54 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BEA631065687; Wed, 2 Jul 2008 13:55:54 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id 825F18FC3F; Wed, 2 Jul 2008 13:55:54 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.2/8.14.2) with ESMTP id m62DtqfC055482; Wed, 2 Jul 2008 09:55:52 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m62Dtpli092002 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Wed, 2 Jul 2008 09:55:51 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200807021355.m62Dtpli092002@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Wed, 02 Jul 2008 09:55:42 -0400 To: Andre Oppermann , "Bjoern A. Zeeb" From: Mike Tancsa In-Reply-To: <486B87DB.3080007@freebsd.org> References: <4852E23E.2040505@gtcomm.net> <4854EBF1.7020708@FreeBSD.org> <200807010606.m6166jFe084204@lava.sentex.ca> <4869EC1E.8060009@freebsd.org> <20080701084933.W57089@maildrop.int.zabbadoz.net> <20080701092254.T57089@maildrop.int.zabbadoz.net> <486B87DB.3080007@freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: freebsd-net@freebsd.org Subject: Re: Route messages X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 13:55:54 -0000 At 09:51 AM 7/2/2008, Andre Oppermann wrote: >>Andre, could you review this? > >Yes, this should fix the problem. I haven't tested the patch though. It works for me in the lab and on one production machine I patched early this morning. ---Mike >-- >Andre > >>Index: sys/netinet/ip_input.c >>=================================================================== >>RCS file: /shared/mirror/FreeBSD/r/ncvs/src/sys/netinet/ip_input.c,v >>retrieving revision 1.332.2.2 >>diff -u -p -r1.332.2.2 ip_input.c >>--- sys/netinet/ip_input.c 22 Apr 2008 12:02:55 -0000 1.332.2.2 >>+++ sys/netinet/ip_input.c 1 Jul 2008 09:23:08 -0000 >>@@ -1363,7 +1363,6 @@ ip_forward(struct mbuf *m, int srcrt) >> * the ICMP_UNREACH_NEEDFRAG "Next-Hop MTU" field >> described in RFC1191. >> */ >> bzero(&ro, sizeof(ro)); >>- rtalloc_ign(&ro, RTF_CLONING); >> error = ip_output(m, NULL, &ro, IP_FORWARDING, NULL, NULL); > >_______________________________________________ >freebsd-net@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-net >To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 14:59:09 2008 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 51BB6106564A; Wed, 2 Jul 2008 14:59:09 +0000 (UTC) (envelope-from cokane@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 288898FC2C; Wed, 2 Jul 2008 14:59:09 +0000 (UTC) (envelope-from cokane@FreeBSD.org) Received: from freefall.freebsd.org (cokane@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m62Ex9np054519; Wed, 2 Jul 2008 14:59:09 GMT (envelope-from cokane@freefall.freebsd.org) Received: (from cokane@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m62Ex92w054515; Wed, 2 Jul 2008 14:59:09 GMT (envelope-from cokane) Date: Wed, 2 Jul 2008 14:59:09 GMT Message-Id: <200807021459.m62Ex92w054515@freefall.freebsd.org> To: cokane@FreeBSD.org, freebsd-net@FreeBSD.org, cokane@FreeBSD.org From: cokane@FreeBSD.org Cc: Subject: Re: kern/124225: [ndis] [patch] ndis network driver sometimes loses network connection X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 14:59:09 -0000 Synopsis: [ndis] [patch] ndis network driver sometimes loses network connection Responsible-Changed-From-To: freebsd-net->cokane Responsible-Changed-By: cokane Responsible-Changed-When: Wed Jul 2 14:56:51 UTC 2008 Responsible-Changed-Why: PR refers to a recent commit of changes that I made, I will look into solving this problem in my development branch. http://www.freebsd.org/cgi/query-pr.cgi?pr=124225 From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 18:22:55 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0CE691065674 for ; Wed, 2 Jul 2008 18:22:55 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id B93358FC0C for ; Wed, 2 Jul 2008 18:22:54 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KE6vB-0004AV-TS; Wed, 02 Jul 2008 14:19:14 -0400 Message-ID: <486BC7F5.5070604@gtcomm.net> Date: Wed, 02 Jul 2008 14:24:53 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 18:22:55 -0000 Fastforward works with lagg , lagg just has some issues that need to be fixed, even on UP system. It has the same issue as IPFW. kern.polling.idlepoll_sleeping: 1 kern.polling.stalled: 806 kern.polling.suspect: 97861 kern.polling.phase: 0 kern.polling.enable: 0 kern.polling.handlers: 2 kern.polling.residual_burst: 0 kern.polling.pending_polls: 0 kern.polling.lost_polls: 128535 kern.polling.short_ticks: 1455 kern.polling.reg_frac: 1200 kern.polling.user_frac: 0 kern.polling.idle_poll: 0 kern.polling.each_burst: 50 kern.polling.burst_max: 1440 kern.polling.burst: 377 It's doing 720kpps right now, and it's having a lot of errors, but the cpu is 40% idle! I don't understand? Is it reporting the wrong cpu usage in TOP? Is this a bug? input (em0) output packets errs bytes packets errs bytes colls 722012 42861 44764748 1 0 178 0 704432 52679 43674800 2 0 580 0 693297 53536 42984418 1 0 178 0 704046 42525 43650854 2 0 220 0 714959 37876 44327462 1 0 178 0 744923 24202 46185230 1 0 178 0 726069 34699 45016282 1 0 178 0 681837 78581 42273898 1 0 178 0 663106 85699 41112576 1 0 178 0 708274 55414 43912992 1 0 178 0 659659 94430 40898862 1 0 178 0 669235 100248 41492574 1 0 178 0 676510 100102 41943624 1 0 178 0 679847 98972 42150518 1 0 178 0 677700 92586 42017416 2 0 356 0 672639 86454 41703622 1 0 178 0 675841 72821 41902146 1 0 178 0 679522 86423 42130368 1 0 178 0 660737 72883 40965698 1 0 178 0 637085 81303 39499274 1 0 178 0 655463 98183 40638710 1 0 178 0 input (em0) output packets errs bytes packets errs bytes colls 683650 66140 42386304 1 0 178 0 654910 110089 40604424 1 0 290 0 647969 120709 40174082 1 0 178 0 666260 67037 41308124 1 0 178 0 671570 68276 41637344 1 0 178 0 691683 60819 42884350 1 0 178 0 663656 79528 41146728 2 0 244 0 703917 47860 43642870 2 0 356 0 710988 55792 44081258 2 0 220 0 697062 77661 43217848 1 0 178 0 65 processes: 2 running, 46 sleeping, 17 waiting CPU: 0.0% user, 0.0% nice, 10.8% system, 45.7% interrupt, 43.5% idle Mem: 7968K Active, 6028K Inact, 42M Wired, 84K Cache, 8768K Buf, 1925M Free Swap: 8192M Total, 8192M Free PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND 10 root 171 ki31 0K 16K RUN 4:06 86.43% idle 36 root -68 - 0K 16K - 0:57 3.81% em3 taskq 13 root -44 - 0K 16K WAIT 9:22 1.42% swi1: net 1429 root 44 0 8084K 2052K RUN 0:00 0.29% top 11 root -32 - 0K 16K WAIT 0:01 0.05% swi4: clock sio net.isr.swi_count: 39442306 net.isr.drop: 0 net.isr.queued: 8 net.isr.deferred: 0 net.isr.directed: 2189 net.isr.count: 2189 net.isr.direct: 1 net.route.netisr_maxqlen: 16384 net.inet.ip.intr_queue_maxlen: 16384 net.inet.ip.intr_queue_drops: 0 em0: Excessive collisions = 0 em0: Sequence errors = 0 em0: Defer count = 0 em0: Missed Packets = 22574958 em0: Receive No Buffers = 65713041 em0: Receive Length Errors = 0 em0: Receive errors = 0 em0: Crc errors = 0 em0: Alignment errors = 0 em0: Collision/Carrier extension errors = 0 em0: RX overruns = 52679 em0: watchdog timeouts = 0 em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0 em0: XON Rcvd = 0 em0: XON Xmtd = 0 em0: XOFF Rcvd = 0 em0: XOFF Xmtd = 0 em0: Good Packets Rcvd = 547984791 em0: Good Packets Xmtd = 5000 em0: TSO Contexts Xmtd = 18 em0: TSO Contexts Failed = 0 kern.hz=2000 hw.em.rxd=512 hw.em.txd=512 -----------Reboot with 4096/4096........(my guess is that it will be a lot worse, more errors..) ........ Without polling, 4096 is horrible, about 200kpps less ... :/ Turning on polling.. polling on, 4096 is bad, input (em0) output packets errs bytes packets errs bytes colls 622379 307753 38587506 1 0 178 0 635689 277303 39412718 1 0 178 0 625552 291235 38784244 2 0 580 0 630143 287872 39068870 1 0 178 0 620225 292071 38453954 1 0 178 0 627499 295329 38904942 1 0 178 0 623854 288086 38678952 1 0 178 0 632433 267698 39210850 1 0 178 0 619177 279541 38388978 1 0 178 0 618049 265926 38319038 2 0 356 0 627026 263882 38875616 1 0 178 0 em0: Missed Packets = 16570461 em0: Receive No Buffers = 9220592 em0: Receive Length Errors = 0 em0: Receive errors = 0 em0: Crc errors = 0 em0: Alignment errors = 0 em0: Collision/Carrier extension errors = 0 em0: RX overruns = 40539 ------Rebooting with 256/256 descriptors.......... .......... No polling: 843762 25337 52313248 1 0 178 0 763555 0 47340414 1 0 178 0 830189 0 51471722 1 0 178 0 838724 0 52000892 1 0 178 0 813594 939 50442832 1 0 178 0 807303 763 50052790 1 0 178 0 791024 0 49043492 1 0 178 0 768316 1106 47635596 1 0 178 0 Machine is maxed and is unresponsive.. Polling ON: input (em0) output packets errs bytes packets errs bytes colls 784138 179079 48616564 1 0 226 0 788815 129608 48906530 2 0 356 0 755555 142997 46844426 2 0 468 0 803670 144459 49827544 1 0 178 0 777649 147120 48214242 1 0 178 0 779539 146820 48331422 1 0 178 0 786201 148215 48744478 2 0 356 0 776013 101660 48112810 1 0 178 0 774239 145041 48002834 2 0 356 0 771774 102969 47850004 1 0 178 0 Machine is responsive and has 40% idle cpu.. Why ALWAYS 40% ? I'm really mistified by this.. Every time it maxes out and gets errors, top reports: CPU: 0.0% user, 0.0% nice, 10.1% system, 45.3% interrupt, 44.6% idle pretty much the same line every time 256/256 blows away 4096 , probably fits the descriptors into the cache lines on the cpu and 4096 has too many cache misses and causes worse performance. This is probably just some nasty programming and they could optimize it for 4096 by taking larger chunks, we don't need the latency to be 0.08ms for each packet, i don't care if it's 0.3ms as long as it doesn't drop any. Setting HZ=100 and 256/256 gets a maximum higher than 2000 polling with 256/256 but the box is unresponsive around 850kpps If this only worked with SMP and was optimized it could do millions of pps :/ Ingo Flaschberger wrote: > Dear Paul, > >> I still don't like the huge hit ipfw and lagg take :/ > > I think, you can't use fastforward with with lagg. > >> ** I tried polling in UP mode and I got some VERY interesting results.. >> CPU is 44% idle (idle polling isn't on) but I'm getting errors! >> It's doing 530kpps with ipfw loaded, which without polling uses 100% >> cpu but now it says my cpu is 44% idle? that makes no sense.. If it >> was idle why am I getting errors? I only get errors when em taskq >> was eating 100% cpu.. >> Idle polling on/off makes no difference. >> user_frac is set to 5 .. > > what are your values: > kern.polling.reg_frac= > kern.polling.user_frac= > kern.polling.burst_max= > > I use: > kern.polling.reg_frac=20 > kern.polling.user_frac=20 > kern.polling.burst_max=512 > > if you need more than 1000, you need to change the code: > src/sys/kern/kern_poll.c > #define MAX_POLL_BURST_MAX 1000 > >> So my maximum without polling is close to 800kpps but if I push that >> it starts locking me from doing things, or > > how many kpps do you want to achieve? > >> HZ=2000 for this test (512/512 descriptors) > > you mean: > hw.em.rxd=512 > hw.em.txd=512 > ? > > can you try with polling: > hw.em.rxd=4096 > hw.em.txd=4096 > > Kind regards, > Ingo Flaschberger > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 18:25:54 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B0231065671; Wed, 2 Jul 2008 18:25:54 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 3CBE78FC1A; Wed, 2 Jul 2008 18:25:54 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KE6y8-0004Z2-NQ; Wed, 02 Jul 2008 14:22:16 -0400 Message-ID: <486BC8AB.60708@gtcomm.net> Date: Wed, 02 Jul 2008 14:27:55 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: "Bjoern A. Zeeb" References: <4852E23E.2040505@gtcomm.net> <4854EBF1.7020708@FreeBSD.org> <200807010606.m6166jFe084204@lava.sentex.ca> <4869EC1E.8060009@freebsd.org> <20080701084933.W57089@maildrop.int.zabbadoz.net> <20080701092254.T57089@maildrop.int.zabbadoz.net> <200807021346.m62DkdHx091961@lava.sentex.ca> <20080702135402.E57089@maildrop.int.zabbadoz.net> In-Reply-To: <20080702135402.E57089@maildrop.int.zabbadoz.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Andre Oppermann , Mike Tancsa Subject: Re: Route messages X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 18:25:54 -0000 Works for me on test machine.. I was expecting a performance increase, but nothing changed.. Just no more route messages, zebra will be happy. Bjoern A. Zeeb wrote: > On Wed, 2 Jul 2008, Mike Tancsa wrote: > > Hi, > >>> Index: sys/netinet/ip_input.c >>> =================================================================== >>> RCS file: /shared/mirror/FreeBSD/r/ncvs/src/sys/netinet/ip_input.c,v >>> retrieving revision 1.332.2.2 >>> diff -u -p -r1.332.2.2 ip_input.c >>> --- sys/netinet/ip_input.c 22 Apr 2008 12:02:55 -0000 1.332.2.2 >>> +++ sys/netinet/ip_input.c 1 Jul 2008 09:23:08 -0000 >>> @@ -1363,7 +1363,6 @@ ip_forward(struct mbuf *m, int srcrt) >>> * the ICMP_UNREACH_NEEDFRAG "Next-Hop MTU" field described >>> in RFC1191. >>> */ >>> bzero(&ro, sizeof(ro)); >>> - rtalloc_ign(&ro, RTF_CLONING); >>> >>> error = ip_output(m, NULL, &ro, IP_FORWARDING, NULL, NULL); >>> > > Still waiting on any second pairs of eyes to review this. > > > >> This could also potentially close >> >> http://www.freebsd.org/cgi/query-pr.cgi?pr=124540 >> http://www.freebsd.org/cgi/query-pr.cgi?pr=123621 > > taken, will handle them. > From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 18:37:12 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5FAD51065671 for ; Wed, 2 Jul 2008 18:37:12 +0000 (UTC) (envelope-from rahman.sazzadur@gmail.com) Received: from yx-out-2324.google.com (yx-out-2324.google.com [74.125.44.30]) by mx1.freebsd.org (Postfix) with ESMTP id 6D0A28FC1F for ; Wed, 2 Jul 2008 18:37:11 +0000 (UTC) (envelope-from rahman.sazzadur@gmail.com) Received: by yx-out-2324.google.com with SMTP id 8so137273yxb.13 for ; Wed, 02 Jul 2008 11:37:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type:references; bh=l01/Pck+V7odk7ZGQy89fQwzeCDfZGfM6i2gvydlI4c=; b=kkxm/BD1XddAcAtUcBcW7ehORhXWsZ35W1QB1Ujtbg0UtIaz3fjFzFeaGHoz/6L8DM I/wR54c1vkdgK7pA9+H9OiD7cuiJxpfO8szydTVQSPFltuLs+L+E48FDpgb95FgjD3st uANK0rS+PRn3FdP7/qtjfw4y4sL7JdemDs6hQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:references; b=LGcmMSMqEDv/Wz7uTM3Zk04p9kVhKHxpwU6xXBjOdn81NQ5iYcZI6xQe6eOd1CwzJ2 Qvt9U5jFSilBnGz7ueaLJzQP5aZRjFXGBXxqcaXvVt5+n4KnmEdCw8AAKLchoyJCjlyX mlgGiUCLP8ScnvI2lpLnMsM+x+EDxtSJTxUao= Received: by 10.142.252.11 with SMTP id z11mr3178166wfh.232.1215023829401; Wed, 02 Jul 2008 11:37:09 -0700 (PDT) Received: by 10.143.173.10 with HTTP; Wed, 2 Jul 2008 11:37:09 -0700 (PDT) Message-ID: <82bdb5ec0807021137m7819153rbc0631ab6f310d0e@mail.gmail.com> Date: Wed, 2 Jul 2008 13:37:09 -0500 From: "sazzadur rahman" To: "Randall Stewart" , freebsd-net@freebsd.org In-Reply-To: <48060748.1090807@cisco.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_Part_6989_27538040.1215023829381" References: <7059EA19D7837E44A3BA7DAB464944B37FDA715193@XMAIL5.sooner.net.ou.edu> <48060748.1090807@cisco.com> X-Mailman-Approved-At: Wed, 02 Jul 2008 19:30:35 +0000 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: atiq@ou.edu, "Rahman, Md Sazzadur" Subject: Re: A query regarding SCTP congestion control X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 18:37:12 -0000 ------=_Part_6989_27538040.1215023829381 Content-Type: text/plain; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hello, I need to get SCTP congestion window data for research purpose. I collected cwnd data from SCTP sender running on FreeBSD 7.0 machine by using KTR kernel log. After that, I tried to plot cwnd vs. time and generated graph. But I am unable to explain the graph and it is very different compared to the graph as shown in the book "Stream Control Transmission Protocol (SCTP)", a reference guide by Randall R. Stewart, page 187 and TCP congestion window. An typical entry from the log looks like: 749199232185105 Net:0xc7703000 at cwnd_event (SACK) cwnd:25140 flight:0 pq:= 0 atpc:72 needpc:235 (tsn:0,sendcnt:191,strcnt:191) I have used 749199232185105 in x axis as time and cwnd:25140 in y axis. I have attached the image file of the graph herewith this mail. >From the log, I found that cwnd varies very frequently accross time. Does anyone have any idea regarding this issue? Please let me know if you have any questions further. Thanks in advance. Best regards, Md Sazzadur Rahman Graduate Student, School of Computer Science, University of Oklahoma, Norman, Oklahoma, USA Steps for getting kernel log ------------------------------------------ 1. Add options: options KTR options KTR_ENTRIES=3D65536 options KTR_MASK=3DKTR_SUBSYS 2. Recompile kernel config CUSTOM_KERNEL_9_6 cd ../compile/ CUSTOM_KERNEL_9_6 make cleandepend;make depend; make all install 3. Tried to enable trace point by: Sysctl -w "net.inet.sctp.log_level=3D0x00000004" 4. run SCTP sender. 5. pull out data: Ktrdump =96q =96t =96o file_name Prtcwndlog =96l filename > cwnd.txt --------------------------------------------------- On Wed, Apr 16, 2008 at 9:03 AM, Randall Stewart wrote: > Rahman, Md Sazzadur wrote: > >> Hi, I would like to get the values of SCTP congestion control >> algorithm variables (cwnd, ssthresh, flightsize and pba) from any >> SCTP based application in runtime for research purpose. Does any API >> exist in SCTP for that? Do I need to dig the SCTP code in kernel to >> get the values? >> > > There is a socket option to get the cwnd. > > However, I think what you really want is some of the researchish > tracing stuff that SCTP provides. > > You can actually get a real time trace of the cwnd/flight etc via the > various logging functions. > > You basically must compile this as an option.. have to go look > at the options.. > > And then you can either use ktrace (which I don't recommend since > it turns on to much overhead in the kernel) or you can > use SCTP_LOCAL_TRACE_BUF > > This will put it into a piece of memory only for SCTP and > not turn on all the other ktrace points. > > After you enable the logging in your compile you must turn > on the logging level.. > > SCTP_CWND_LOGGING_ENABLE > > woudl be my recommendation. > > It gives you a real time up/down growth of the cwnd/flight/rwnd > > I think I wrote a "how to" somewhere.. let me go look.. > > R > > > >> I will appreciate any help in this regard. >> >> Best Regards, Md Sazzadur Rahman Graduate Student, School of Computer >> Science, University of Oklahoma, Norman, Oklahoma, USA >> >> _______________________________________________ freebsd-net@freebsd.orgm= ailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, >> send any mail to "freebsd-net-unsubscribe@freebsd.org" >> >> > > -- > Randall Stewart > NSSTG - Cisco Systems Inc. > 803-345-0369 803-317-4952 (cell) > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > ------=_Part_6989_27538040.1215023829381-- From owner-freebsd-net@FreeBSD.ORG Wed Jul 2 19:48:30 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1B53C1065673 for ; Wed, 2 Jul 2008 19:48:30 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.238]) by mx1.freebsd.org (Postfix) with ESMTP id B6CF18FC16 for ; Wed, 2 Jul 2008 19:48:29 +0000 (UTC) (envelope-from fjwcash@gmail.com) Received: by wx-out-0506.google.com with SMTP id h27so257935wxd.7 for ; Wed, 02 Jul 2008 12:48:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=LU2IkWfoOmi6oYVxFa9SZbmk7v4LF7l4M0SFWkPbZNI=; b=InCit8ylaYbBl4iBiurUf8PVbQ0w5+V7m7toDnJtNglinkrLx0G6ghYiwZJqDCaZJ8 BGyqxN6cOBrTQjVZH1P0Rglc0OoB0sbkcWxDUPwlOkt7sZg0H9f3mFUWyTx1+lTdvBoM 7Crnpef7P9+ODwgaWod5MBbBhimHwAg5crjgI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=QXH6HA5PtvFdzQM0pbeYCQNiOQSxec5geODGnUYgt59+fndxrn0pCNgVF+83lAqgiG HvYyOOD9p8vX7zTZgAxFdBE56kkyWW9F5enuES8ugqvbhkUOseXID/d9fdwH33PyDxEv Ma56xL2MSx0kieVFVkLkN0GnXB9T7gLViHSy0= Received: by 10.151.142.2 with SMTP id u2mr843108ybn.184.1215028108571; Wed, 02 Jul 2008 12:48:28 -0700 (PDT) Received: by 10.151.154.17 with HTTP; Wed, 2 Jul 2008 12:48:28 -0700 (PDT) Message-ID: Date: Wed, 2 Jul 2008 12:48:28 -0700 From: "Freddie Cash" To: freebsd-net@freebsd.org In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <4869880D.8040901@ibctech.ca> Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Jul 2008 19:48:30 -0000 On Mon, Jun 30, 2008 at 6:39 PM, Ingo Flaschberger wrote: >> I'm curious now... how do you change individual device polling via sysctl? > > not via sysctl, via ifconfig: > > # enable interface polling > /sbin/ifconfig em0 polling > /sbin/ifconfig em1 polling > /sbin/ifconfig em2 polling > /sbin/ifconfig em3 polling > > (and via /etc/rc.local also across reboots) No, you put it into the ifconfig_X lines in /etc/rc.conf as the last option. Or -polling to disable it. ifconfig_em0='inet 1.2.3.4/24 polling" ifconfig_em2='inet 1.2.3.5/24 -polling" -- Freddie Cash fjwcash@gmail.com From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 01:08:33 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4867A106567A; Thu, 3 Jul 2008 01:08:33 +0000 (UTC) (envelope-from stef-list@memberwebs.com) Received: from mx.npubs.com (mail.writemehere.com [209.66.100.224]) by mx1.freebsd.org (Postfix) with ESMTP id 208508FC23; Thu, 3 Jul 2008 01:08:32 +0000 (UTC) (envelope-from stef-list@memberwebs.com) Received: from mx.npubs.com (avhost [209.66.100.194]) by mx.npubs.com (Postfix) with ESMTP id 1F032F1816B; Thu, 3 Jul 2008 00:39:57 +0000 (UTC) Received: from northstar-srv2 (unknown [172.27.2.11]) by mx.npubs.com (Postfix) with ESMTP id 859BCF180C0; Thu, 3 Jul 2008 00:39:55 +0000 (UTC) From: Stef User-Agent: Thunderbird 2.0.0.14 (X11/20080505) MIME-Version: 1.0 To: Kian Mohageri References: <678A03F5-5E8A-4CF6-90DF-AA9A4F30FBE1@stromnet.se> <1211037564.6326.27.camel@porksoda> <679DB462-75D6-45CC-949C-1BE8E12C22CD@stromnet.se> <482FD877.6050707@infracaninophile.co.uk> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit Message-Id: <20080703003955.859BCF180C0@mx.npubs.com> X-Virus-Scanned: ClamAV using ClamSMTP Date: Thu, 3 Jul 2008 00:39:57 +0000 (UTC) Cc: freebsd-stable , freebsd-net@freebsd.org, =?ISO-8859-1?Q?Johan_Str=F6m?= , Matthew Seaman , freebsd-pf@freebsd.org, Alex Trull Subject: Re: connect(): Operation not permitted X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: stef@memberwebs.com List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 01:08:33 -0000 Kian Mohageri wrote: > On Sun, May 18, 2008 at 3:33 AM, Johan Ström wrote: >> On May 18, 2008, at 9:19 AM, Matthew Seaman wrote: >> >>> Johan Ström wrote: >>> >>>> drop all traffic)? A check with pfctl -vsr reveals that the actual rule >>>> inserted is "pass on lo0 inet from 123.123.123.123 to 123.123.123.123 flags >>>> S/SA keep state". Where did that "keep state" come from? >>> 'flags S/SA keep state' is the default now for tcp filter rules -- that >>> was new in 7.0 reflecting the upstream changes made between the 4.0 and >>> 4.1 >>> releases of OpenBSD. If you want a stateless rule, append 'no state'. >>> >>> http://www.openbsd.org/faq/pf/filter.html#state >> Thanks! I was actually looking around in the pf.conf manpage but failed to >> find it yesterday, but looking closer today I now saw it. >> Applied the no state (and quick) to the rule, and now no state is created. >> And the problem I had in the first place seems to have been resolved too >> now, even though it didn't look like a state problem.. (started to deny new >> connections much earlier than the states was full, altough maybee i wasnt >> looking for updates fast enough or something). >> > > I'd be willing to bet it's because you're reusing the source port on a > new connection before the old state expires. > > You'll know if you check the state-mismatch counter. > > Anyway, glad you found a resolution. I've been experiencing this "Operation not permitted" too. I've been trying to track down the problem for many months, but due to the complexity of my firewalls (scores of jails each with scores of rules), I wasn't brave enough to ask for help :) As a work around we started creating rules without state, whenever we would run into the problem. Thanks for the pointer about state-mismatch. The state-mismatch counter does is in fact high in my case (see below). How would I go about getting the pf state timeout and the reuse of ports for outbound connections to match? Or is this an intractable problem, that just needs to be worked around? Cheers, Stef Walter Status: Enabled for 13 days 23:55:25 Debug: Urgent Hostid: 0x38ae6776 State Table Total Rate current entries 65 searches 819507771 677.7/s inserts 1136670 0.9/s removals 1136605 0.9/s Counters match 787482855 651.2/s bad-offset 0 0.0/s fragment 0 0.0/s short 0 0.0/s normalize 0 0.0/s memory 0 0.0/s bad-timestamp 0 0.0/s congestion 0 0.0/s ip-option 0 0.0/s proto-cksum 0 0.0/s state-mismatch 748 0.0/s state-insert 0 0.0/s state-limit 0 0.0/s src-limit 0 0.0/s synproxy 0 0.0/s From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 02:58:25 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E0B631065672 for ; Thu, 3 Jul 2008 02:58:25 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail17.syd.optusnet.com.au (mail17.syd.optusnet.com.au [211.29.132.198]) by mx1.freebsd.org (Postfix) with ESMTP id 794B38FC0A for ; Thu, 3 Jul 2008 02:58:25 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from server.vk2pj.dyndns.org (c122-106-215-175.belrs3.nsw.optusnet.com.au [122.106.215.175]) by mail17.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m632wM5a014893 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 3 Jul 2008 12:58:23 +1000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.2/8.14.2) with ESMTP id m632wMIO024817 for ; Thu, 3 Jul 2008 12:58:22 +1000 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.2/8.14.2/Submit) id m632wMoH024816 for freebsd-net@freebsd.org; Thu, 3 Jul 2008 12:58:22 +1000 (EST) (envelope-from peter) Date: Thu, 3 Jul 2008 12:58:22 +1000 From: Peter Jeremy To: freebsd-net@freebsd.org Message-ID: <20080703025822.GA24765@server.vk2pj.dyndns.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="IJpNTDwzlM2Ie8A6" Content-Disposition: inline X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.18 (2008-05-17) Subject: arplookup x.x.x.x failed: host is not on local network X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 02:58:26 -0000 --IJpNTDwzlM2Ie8A6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable I'm occasionally seeing pairs of messages like the following on my NAT host: arplookup 192.168.181.114 failed: host is not on local network arpresolve: can't allocate route for 192.168.181.114 In my particular configuration, there are dual subnets between the NAT and target host. My initial assumption was that the request was arriving on the other subnet and I added if_xname to the arplookup printf() - but that shows that interface matches the IP address. I've looked back through the mailing lists but the previous reports of this problem don't match my scenario. I've seen this with FreeBSD 5.3, 6.2 and 7.0. The (in)frequency of the problem makes me wonder if it's actually a resource exhaustion problem. Has anyone got any suggestions? --=20 Peter Jeremy --IJpNTDwzlM2Ie8A6 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkhsQE4ACgkQ/opHv/APuIcCegCcDcqGAwRWE8pIAYxx3QZrPXSD PaEAn2OXzWrqM6pBaYDgIY6VnL1SDOzJ =12nD -----END PGP SIGNATURE----- --IJpNTDwzlM2Ie8A6-- From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 06:45:59 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 800771065672 for ; Thu, 3 Jul 2008 06:45:59 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 3B3378FC14 for ; Thu, 3 Jul 2008 06:45:59 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KEIWA-0002nV-H2; Thu, 03 Jul 2008 02:42:10 -0400 Message-ID: <486C7611.9030905@gtcomm.net> Date: Thu, 03 Jul 2008 02:47:45 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 06:45:59 -0000 Preliminary 32 bit results... When I started out it looked like 32 bit was worse than 64 bit, but it's just the timers are different. For instance, 4000 hz in 64 bit gives better results than 4000hz in 32 bit. Low HZ gives better result with polling on in 32 bit Bottom line, so far I'm not able to get any better performance out of 32 bit at all. In fact I think it might be even a tad slower. I didn't see as high of bursts like I did on 64 bit so far but I'm still testing. Tomorrow comes opteron 2222 so it's 1ghz faster than this one, and I can see if it scales directly with cpu speed or what happens. I did another SMP test with an interesting results. I took one of the cpus out of the machine, so it was just left with a single 2212 (dual core) and it performed better. Less contention I suppose? some results: kern.hz=4000 hw.em.rxd=512 hw.em.txd=512 polling on, idle polling on (only way I can get a reliable netstat output) input (em0) output packets errs bytes packets errs bytes colls 681961 117612 42281586 1 0 226 0 655095 83418 40615892 2 0 220 0 683881 93559 42400626 1 0 178 0 683637 90452 42385498 1 0 178 0 683345 87471 42367394 1 0 178 0 682737 81483 42329696 2 0 220 0 683154 95413 42355552 1 0 178 0 684556 111013 42442476 1 0 178 0 684365 110960 42430634 1 0 178 0 679089 116440 42103518 3 0 534 0 684328 122713 42428340 1 0 178 0 684852 121387 42460828 1 0 178 0 685358 113256 42492200 1 0 178 0 685060 123110 42473724 1 0 178 0 684463 118335 42436710 1 0 178 0 677182 127788 41985300 2 0 356 0 685920 126144 42527044 1 0 178 0 684946 107034 42466656 1 0 178 0 (reboot) kern.hz=1000 input (em0) output packets errs bytes packets errs bytes colls 679611 97394 42136046 5 0 762 0 663939 104714 41164254 5 0 1322 0 685538 91102 42503412 4 0 536 0 676704 94629 41955668 2 0 404 0 685323 115060 42490030 1 0 178 0 675954 105506 41909164 2 0 356 0 655321 92118 40629906 1 0 178 0 686826 85674 42583228 2 0 356 0 686378 89983 42555440 1 0 178 0 685539 80180 42503422 1 0 178 0 686704 88626 42575652 1 0 178 0 686567 88596 42567158 1 0 178 0 687031 82640 42595936 3 0 398 0 sysctl -w kern.polling.each_burst=50 kern.polling.each_burst: 256 -> 50 [root@ircrouter ~]# netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 693036 39992 42968315 3 0 400 0 695538 58189 43123360 1 0 178 0 692670 62765 42945544 1 0 178 0 693219 60755 42979580 2 0 220 0 692637 64761 42943498 1 sysctl -w kern.polling.each_burst=33 kern.polling.each_burst: 50 -> 33 [root@ircrouter ~]# netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 690530 63359 42812868 1 0 226 0 689748 57670 42764380 1 0 178 0 690489 57874 42810322 1 0 178 0 689655 60606 42758614 1 0 178 0 ^C [root@ircrouter ~]# sysctl -w kern.polling.each_burst=3 kern.polling.each_burst: 33 -> 3 [root@ircrouter ~]# netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 612234 110896 37958512 1 0 226 0 614391 112506 38092246 1 0 178 0 ^C [root@ircrouter ~]# sysctl -w kern.polling.each_burst=800 kern.polling.each_burst: 3 -> 800 [root@ircrouter ~]# netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 668057 76496 41419538 1 0 226 0 667689 88674 41396720 2 0 220 0 670526 106654 41572616 1 0 178 0 667326 97832 41374216 1 0 178 0 ^C [root@ircrouter ~]# sysctl -w kern.polling.each_burst=66 kern.polling.each_burst: 800 -> 66 [root@ircrouter ~]# netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 690164 89290 42790172 1 0 226 0 688886 74360 42710936 1 0 178 0 674079 77027 41792902 1 0 178 0 kern.hz=2000 input (em0) output packets errs bytes packets errs bytes colls 699116 238016 43345196 1 0 178 0 698263 225244 43292310 1 0 290 0 697246 222395 43229256 1 0 178 0 696749 207766 43198442 1 0 178 0 697304 217384 43232852 1 0 178 0 696401 209901 43176866 1 0 178 0 696508 207757 43183500 1 0 178 0 ^C hz=2000 with 1024/1024 descriptors input (em0) output packets errs bytes packets errs bytes colls 670315 235780 41559534 1 0 226 0 683218 225838 42359520 1 0 178 0 682998 242551 42345880 1 0 178 0 681777 239649 42270178 1 0 178 0 hz=1000 with 256/256 descriptors netstat -w1 -I em0 input (em0) output packets errs bytes packets errs bytes colls 740584 160355 45916212 2 0 0 0 746027 165198 46253678 1 0 178 0 746068 165921 46256220 1 0 178 0 746505 167527 46283314 1 0 178 0 743902 175019 46121928 1 0 178 0 746130 179795 46260064 1 0 178 0 744457 166448 46156338 1 0 178 0 746169 176137 46262482 1 0 178 0 hz=667 with 256/256 input (em0) output packets errs bytes packets errs bytes colls 742614 91687 46042072 1 0 226 0 739746 85695 45864256 1 0 178 0 733723 85162 45490840 3 0 398 0 737561 102207 45728786 1 0 178 0 739618 127597 45856320 1 0 178 0 ^C Hrm finally same pps as 64 bit..... Now I wonder what happens if I go back to the 64 bit and try 1000 256/256 ?? I don't think I tried that one.. Guess another reinstall :> Installing 64 bit.. (again) Just to be sure.. Paul Ingo Flaschberger wrote: > Dear Paul, > >> SMP DISABLED on my Opteron 2212 (ULE, Preemption on) >> Yields ~750kpps in em0 and out em1 (one direction) >> I am miffed why this yields more pps than >> a) with all 4 cpus running and b) 4 cpus with lagg load balanced over >> 3 incoming connections so 3 taskq threads > > because less locking, less synchronisation, .... > >> I would be willing to set up test equipment (several servers plugged >> into a switch) with ipkvm and power port access >> if someone or a group of people want to figure out ways to improve >> the routing process, ipfw, and lagg. >> >> Maximum PPS with one ipfw rule on UP: >> tops out about 570Kpps.. almost 200kpps lower ? (frown) > > can you post the rule here? > >> I'm going to drop in a 3ghz opteron instead of the 2ghz 2212 that's >> in here and see how that scales, using UP same kernel etc I have now. > > really, please try 32bit and 1 cpu. > > Kind regards, > Ingo Flaschberger > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 07:05:16 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF9EF1065675 for ; Thu, 3 Jul 2008 07:05:16 +0000 (UTC) (envelope-from daniel@skytek.it) Received: from mail.skytek.it (mail.skytek.it [217.194.176.18]) by mx1.freebsd.org (Postfix) with ESMTP id 326058FC1F for ; Thu, 3 Jul 2008 07:05:15 +0000 (UTC) (envelope-from daniel@skytek.it) Received: from [192.168.30.100] ([192.168.30.100]) by mail.skytek.it (Skytek Mail Server v.11.47-p9) with ASMTP id JOI50421; Thu, 03 Jul 2008 09:05:21 +0200 Message-ID: <486C7A2B.1050902@skytek.it> Date: Thu, 03 Jul 2008 09:05:15 +0200 From: Daniel Ponticello User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Peter Jeremy References: <20080703025822.GA24765@server.vk2pj.dyndns.org> In-Reply-To: <20080703025822.GA24765@server.vk2pj.dyndns.org> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org Subject: Re: arplookup x.x.x.x failed: host is not on local network X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 07:05:16 -0000 Hi Peter, i'm having exactly the same problem, but without NAT configuration. Just a simple host on network 192.168.170.xxx that when tries to reach an host on 192.168.181.xxx: it gives the same error arplookup 192.168.181.253 failed: host is not on local network The funny thing is that IP connection works anyway. The problem is present only when trying to reach network 192.168.181.xxx, which is absolutely not on local network. The problem started with freebsd 5.3 and today, with 7, it is still present. Daniel Peter Jeremy ha scritto: > I'm occasionally seeing pairs of messages like the following on > my NAT host: > arplookup 192.168.181.114 failed: host is not on local network > arpresolve: can't allocate route for 192.168.181.114 > > In my particular configuration, there are dual subnets between the NAT > and target host. My initial assumption was that the request was > arriving on the other subnet and I added if_xname to the arplookup > printf() - but that shows that interface matches the IP address. > I've looked back through the mailing lists but the previous reports > of this problem don't match my scenario. > > I've seen this with FreeBSD 5.3, 6.2 and 7.0. > > The (in)frequency of the problem makes me wonder if it's actually a > resource exhaustion problem. > > Has anyone got any suggestions? > > -- WBR, Cordiali Saluti, Daniel Ponticello, VP of Engineering Network Coordination Centre of Skytek --- - For further information about our services: - Please visit our website at http://www.Skytek.it --- From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 07:07:32 2008 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8873E106567E for ; Thu, 3 Jul 2008 07:07:32 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail06.syd.optusnet.com.au (mail06.syd.optusnet.com.au [211.29.132.187]) by mx1.freebsd.org (Postfix) with ESMTP id 0E9B28FC1A for ; Thu, 3 Jul 2008 07:07:31 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c220-239-252-11.carlnfd3.nsw.optusnet.com.au (c220-239-252-11.carlnfd3.nsw.optusnet.com.au [220.239.252.11]) by mail06.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m6377Pwl020704 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 3 Jul 2008 17:07:26 +1000 Date: Thu, 3 Jul 2008 17:07:23 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Paul In-Reply-To: <486BC7F5.5070604@gtcomm.net> Message-ID: <20080703160540.W6369@delplex.bde.org> References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net , Ingo Flaschberger Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 07:07:32 -0000 On Wed, 2 Jul 2008, Paul wrote: >... > -----------Reboot with 4096/4096........(my guess is that it will be a lot > worse, more errors..) > ........ > Without polling, 4096 is horrible, about 200kpps less ... :/ > Turning on polling.. > polling on, 4096 is bad, > input (em0) output > packets errs bytes packets errs bytes colls > 622379 307753 38587506 1 0 178 0 > 635689 277303 39412718 1 0 178 0 > ... > ------Rebooting with 256/256 descriptors.......... > .......... > No polling: > 843762 25337 52313248 1 0 178 0 > 763555 0 47340414 1 0 178 0 > 830189 0 51471722 1 0 178 0 > 838724 0 52000892 1 0 178 0 > 813594 939 50442832 1 0 178 0 > 807303 763 50052790 1 0 178 0 > 791024 0 49043492 1 0 178 0 > 768316 1106 47635596 1 0 178 0 > Machine is maxed and is unresponsive.. That's the most interesting one. Even 1% packet loss would probably destroy performance, so the benchmarks that give 10-50% packet loss are uninteresting. All indications are that you are running out of CPU and memory (DMA and/or cache fills) throughput. The above apparently hits both limits at the same time, while with more descriptors memory throughput runs out first. 1 CPU is apparently barely enough for 800 kpps (is this all with UP now?), and I think more CPUs could only be slower, as you saw with SMP, especially using multiple em taskqs, since memory traffic would be higher. I wouldn't expect this to be fixed soon (except by throwing better/different hardware at it). The CPU/DMA balance can probably be investigated by slowing down the CPU/ memory system. You may remember my previous mail about getting higher pps on bge. Again, all indications are that I'm running out of CPU, memory, and bus throughput too since the bus is only PCI 33MHz. These interact in a complicated way which I haven't been able to untangle. -current is fairly consistently slower than my ~5.2 by about 10%, apparently due to code bloat (extra CPU and related extra cache misses). OTOH, like you I've seen huge variations for changes that should be null (e.g., disturbing the alignment of the text section without changing anything else). My ~5.2 is very consistent since I rarely change it, while -current changes a lot and shows more variation, but with no sign of getting near the ~5.2 plateau or even its old peaks. > Polling ON: > input (em0) output > packets errs bytes packets errs bytes colls > 784138 179079 48616564 1 0 226 0 > 788815 129608 48906530 2 0 356 0 > 755555 142997 46844426 2 0 468 0 > 803670 144459 49827544 1 0 178 0 > 777649 147120 48214242 1 0 178 0 > 779539 146820 48331422 1 0 178 0 > 786201 148215 48744478 2 0 356 0 > 776013 101660 48112810 1 0 178 0 > 774239 145041 48002834 2 0 356 0 > 771774 102969 47850004 1 0 178 0 > > Machine is responsive and has 40% idle cpu.. Why ALWAYS 40% ? I'm really > mistified by this.. Is this with hz=2000 and 256/256 and no polling in idle? 40% is easy to explain (perhaps incorrectly). Polling can then read at most 256 descriptors every 1/2000 second, giving a max throughput of 512 kpps. Packets < descriptors in general but might be equal here (for small packets). You seem to actually get 784 kpps, which is too high even in descriptors unless, but matches exactly if the errors are counted twice (784 - 179 - 505 ~= 512). CPU is getting short too, but 40% still happens to be left over after giving up at 512 kpps. Most of the errors are probably handled by the hardware at low cost in CPU by dropping packets. There are other types of errors but none except dropped packets is likely. > Every time it maxes out and gets errors, top reports: > CPU: 0.0% user, 0.0% nice, 10.1% system, 45.3% interrupt, 44.6% idle > pretty much the same line every time > > 256/256 blows away 4096 , probably fits the descriptors into the cache lines > on the cpu and 4096 has too many cache misses and causes worse performance. Quite likely. Maybe your systems have memory systems that are weak relative to other resources, so that they this limit sooner than expected. I should look at my "fixes" for bge, one than changes rxd from 256 to 512, and one that increases the ifq tx length from txd = 512 to about 20000. Both of these might thrash caches. The former makes little difference except for polling at < 4000 Hz, but I don't believe in or use polling. The latter works around select() for write descriptors not working on sockets, so that high frequency polling from userland is not needed to determine a good time to retry after ENOBUFs errors. This is probably only important in pps benchmarks. txd = 512 gives good efficiency in my version of bge, but might be too high for good throughput and is mostly wasted in distribution versions of FreeBSD. Bruce From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 07:26:21 2008 Return-Path: Delivered-To: freebsd-net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 793691065673 for ; Thu, 3 Jul 2008 07:26:21 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 300948FC23 for ; Thu, 3 Jul 2008 07:26:21 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KEJ9J-0007nD-It; Thu, 03 Jul 2008 03:22:37 -0400 Message-ID: <486C7F93.7010308@gtcomm.net> Date: Thu, 03 Jul 2008 03:28:19 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Bruce Evans References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> In-Reply-To: <20080703160540.W6369@delplex.bde.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Ingo Flaschberger Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 07:26:21 -0000 Bruce Evans wrote: > On Wed, 2 Jul 2008, Paul wrote: > >> ... >> -----------Reboot with 4096/4096........(my guess is that it will be >> a lot worse, more errors..) >> ........ >> Without polling, 4096 is horrible, about 200kpps less ... :/ >> Turning on polling.. >> polling on, 4096 is bad, >> input (em0) output >> packets errs bytes packets errs bytes colls >> 622379 307753 38587506 1 0 178 0 >> 635689 277303 39412718 1 0 178 0 >> ... >> ------Rebooting with 256/256 descriptors.......... >> .......... >> No polling: >> 843762 25337 52313248 1 0 178 0 >> 763555 0 47340414 1 0 178 0 >> 830189 0 51471722 1 0 178 0 >> 838724 0 52000892 1 0 178 0 >> 813594 939 50442832 1 0 178 0 >> 807303 763 50052790 1 0 178 0 >> 791024 0 49043492 1 0 178 0 >> 768316 1106 47635596 1 0 178 0 >> Machine is maxed and is unresponsive.. > > That's the most interesting one. Even 1% packet loss would probably > destroy performance, so the benchmarks that give 10-50% packet loss > are uninteresting. > But you realize that it's outputting all of these packets on em3 and I'm watching them coming out and they are consistent with the packets received on em0 that netstat shows are 'good' packets. > All indications are that you are running out of CPU and memory (DMA > and/or cache fills) throughput. The above apparently hits both limits > at the same time, while with more descriptors memory throughput runs > out first. 1 CPU is apparently barely enough for 800 kpps (is this > all with UP now?), and I think more CPUs could only be slower, as you > saw with SMP, especially using multiple em taskqs, since memory traffic > would be higher. I wouldn't expect this to be fixed soon (except by > throwing better/different hardware at it). > > The CPU/DMA balance can probably be investigated by slowing down the CPU/ > memory system. > I'm using a server opteron which supposedly has the best memory performance out of any CPU right now. Plus opterons have the biggest l1 cache, but small l2 cache. Do you think larger l2 cache on the Xeon (6mb for 2 core) would be better? I have a 2222 opteron coming which is 1ghz faster so we will see what happens :> My NIC is PCI-E 4x so there's no bottleneck there. > You may remember my previous mail about getting higher pps on bge. > Again, all indications are that I'm running out of CPU, memory, and > bus throughput too since the bus is only PCI 33MHz. These interact > in a complicated way which I haven't been able to untangle. -current > is fairly consistently slower than my ~5.2 by about 10%, apparently > due to code bloat (extra CPU and related extra cache misses). OTOH, > like you I've seen huge variations for changes that should be null > (e.g., disturbing the alignment of the text section without changing > anything else). My ~5.2 is very consistent since I rarely change it, > while -current changes a lot and shows more variation, but with no > sign of getting near the ~5.2 plateau or even its old peaks. > >> Polling ON: >> input (em0) output >> packets errs bytes packets errs bytes colls >> 784138 179079 48616564 1 0 226 0 >> 788815 129608 48906530 2 0 356 0 >> 755555 142997 46844426 2 0 468 0 >> 803670 144459 49827544 1 0 178 0 >> 777649 147120 48214242 1 0 178 0 >> 779539 146820 48331422 1 0 178 0 >> 786201 148215 48744478 2 0 356 0 >> 776013 101660 48112810 1 0 178 0 >> 774239 145041 48002834 2 0 356 0 >> 771774 102969 47850004 1 0 178 0 >> >> Machine is responsive and has 40% idle cpu.. Why ALWAYS 40% ? I'm >> really mistified by this.. > > Is this with hz=2000 and 256/256 and no polling in idle? 40% is easy > to explain (perhaps incorrectly). Polling can then read at most 256 > descriptors every 1/2000 second, giving a max throughput of 512 kpps. > Packets < descriptors in general but might be equal here (for small > packets). You seem to actually get 784 kpps, which is too high even > in descriptors unless, but matches exactly if the errors are counted > twice (784 - 179 - 505 ~= 512). CPU is getting short too, but 40% > still happens to be left over after giving up at 512 kpps. Most of > the errors are probably handled by the hardware at low cost in CPU by > dropping packets. There are other types of errors but none except > dropped packets is likely. > Read above, it's actually transmitting 770kpps out of em3 so it can't just be 512kpps. I suppose multiple packets can fit in 1 descriptor? I am using VERY small tcp packets.. >> Every time it maxes out and gets errors, top reports: >> CPU: 0.0% user, 0.0% nice, 10.1% system, 45.3% interrupt, 44.6% idle >> pretty much the same line every time >> >> 256/256 blows away 4096 , probably fits the descriptors into the >> cache lines on the cpu and 4096 has too many cache misses and causes >> worse performance. > > Quite likely. Maybe your systems have memory systems that are weak > relative > to other resources, so that they this limit sooner than expected. > > I should look at my "fixes" for bge, one than changes rxd from 256 to > 512, > and one that increases the ifq tx length from txd = 512 to about 20000. > Both of these might thrash caches. The former makes little difference > except for polling at < 4000 Hz, but I don't believe in or use polling. > The latter works around select() for write descriptors not working on > sockets, so that high frequency polling from userland is not needed to > determine a good time to retry after ENOBUFs errors. This is probably > only important in pps benchmarks. txd = 512 gives good efficiency in > my version of bge, but might be too high for good throughput and is > mostly > wasted in distribution versions of FreeBSD. > I was thinking of trying 4 or 5.. but how would that work with this new hardware? Thanks Paul From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 07:40:41 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 124C4106567E for ; Thu, 3 Jul 2008 07:40:41 +0000 (UTC) (envelope-from ddesimone@verio.net) Received: from relay2-bcrtfl2.verio.net (relay2-bcrtfl2.verio.net [131.103.218.177]) by mx1.freebsd.org (Postfix) with ESMTP id B14A48FC13 for ; Thu, 3 Jul 2008 07:40:40 +0000 (UTC) (envelope-from ddesimone@verio.net) Received: from iad-wprd-xchw01.corp.verio.net (iad-wprd-xchw01.corp.verio.net [198.87.7.164]) by relay2-bcrtfl2.verio.net (Postfix) with ESMTP id 6ECD71FF019D for ; Thu, 3 Jul 2008 03:16:31 -0400 (EDT) thread-index: Acjc3LeHOhlcGsVJT0qgI82aN7X/Ow== Received: from limbo.int.dllstx01.us.it.verio.net ([10.10.10.11]) by iad-wprd-xchw01.corp.verio.net with Microsoft SMTPSVC(6.0.3790.1830); Thu, 3 Jul 2008 03:16:30 -0400 Received: by limbo.int.dllstx01.us.it.verio.net (Postfix, from userid 1000) id 8F4F68E29B; Thu, 3 Jul 2008 02:16:30 -0500 (CDT) Date: Thu, 3 Jul 2008 02:16:30 -0500 From: "David DeSimone" Content-Transfer-Encoding: 7bit To: Message-ID: <20080703071629.GA29305@verio.net> Mail-Followup-To: freebsd-net@freebsd.org Content-Class: urn:content-classes:message Importance: normal References: <20080703025822.GA24765@server.vk2pj.dyndns.org> Priority: normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.2992 MIME-Version: 1.0 Content-Type: text/plain; x-action=pgp-signed; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20080703025822.GA24765@server.vk2pj.dyndns.org> Precedence: bulk User-Agent: Mutt/1.5.9i X-OriginalArrivalTime: 03 Jul 2008 07:16:30.0929 (UTC) FILETIME=[B77B5C10:01C8DCDC] Subject: Re: arplookup x.x.x.x failed: host is not on local network X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 07:40:41 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Peter Jeremy wrote: > > I'm occasionally seeing pairs of messages like the following on > my NAT host: > arplookup 192.168.181.114 failed: host is not on local network > arpresolve: can't allocate route for 192.168.181.114 We see these too on a 7.0 box (amd64). My theory is that this is a response to ARP requests. ARP requests are broadcast, so the BSD box hears someone asking for this IP, but cannot find it on any local interfaces, and so complains that it cannot construct a proper reply. Have not tested it though. - -- David DeSimone == Network Admin == fox@verio.net "This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, dis- tribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that this email is error or virus free. Thank you." --Lawyer Bot 6000 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFIbHzNFSrKRjX5eCoRAlfXAKCDLSbKzl2aNF9rPFpLQuyknm6dGgCeNAK0 DYsnpm+5EED36G2D461JQR8= =VP12 -----END PGP SIGNATURE----- This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that this email is error or virus free. Thank you. From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 07:48:28 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0C7D61065684 for ; Thu, 3 Jul 2008 07:48:28 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id B13938FC20 for ; Thu, 3 Jul 2008 07:48:27 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 8A91B1B10E4E; Thu, 3 Jul 2008 09:48:25 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.2.4 (2008-01-01) on malcho.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-9.6 required=5.0 tests=ALL_TRUSTED,BAYES_00, GAPPY_SUBJECT autolearn=no version=3.2.4 Received: from hater.haters.org (hater.cmotd.com [192.168.3.125]) by blah.sun-fish.com (Postfix) with ESMTP id 34B0F1B10CAA; Thu, 3 Jul 2008 09:48:23 +0200 (CEST) Message-ID: <486C8446.9060302@moneybookers.com> Date: Thu, 03 Jul 2008 10:48:22 +0300 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.14 (X11/20080616) MIME-Version: 1.0 To: Peter Jeremy References: <20080703025822.GA24765@server.vk2pj.dyndns.org> In-Reply-To: <20080703025822.GA24765@server.vk2pj.dyndns.org> Content-Type: text/plain; charset=windows-1251; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV version 0.93, clamav-milter version 0.93 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-net@freebsd.org Subject: Re: arplookup x.x.x.x failed: host is not on local network X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 07:48:28 -0000 Hi, Peter Jeremy wrote: > I'm occasionally seeing pairs of messages like the following on > my NAT host: > arplookup 192.168.181.114 failed: host is not on local network > arpresolve: can't allocate route for 192.168.181.114 > Normally this happens in badly configured LAN. Lets say we have two hosts in the same physical network (same switch for example) Host1 is configured 192.168.1.33/24 and Hosts2 have 192.168.1.1/30 Now when a broadcast or other packet is sent from Host1 it can reach Host2 without a problem. But when Host2 try reach directly Host1 it doesn't know how and from here - can't allocate route ... I bet 192.168.181.114 have a wrong network mask ;) > In my particular configuration, there are dual subnets between the NAT > and target host. My initial assumption was that the request was > arriving on the other subnet and I added if_xname to the arplookup > printf() - but that shows that interface matches the IP address. > I've looked back through the mailing lists but the previous reports > of this problem don't match my scenario. > > I've seen this with FreeBSD 5.3, 6.2 and 7.0. > > The (in)frequency of the problem makes me wonder if it's actually a > resource exhaustion problem. > > Has anyone got any suggestions? > > -- Best Wishes, Stefan Lambrev ICQ# 24134177 From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 10:42:02 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 757531065683 for ; Thu, 3 Jul 2008 10:42:02 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail06.syd.optusnet.com.au (mail06.syd.optusnet.com.au [211.29.132.187]) by mx1.freebsd.org (Postfix) with ESMTP id C2B388FC32 for ; Thu, 3 Jul 2008 10:42:01 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c220-239-252-11.carlnfd3.nsw.optusnet.com.au (c220-239-252-11.carlnfd3.nsw.optusnet.com.au [220.239.252.11]) by mail06.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m63AftAB010598 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 3 Jul 2008 20:41:56 +1000 Date: Thu, 3 Jul 2008 20:41:54 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Paul In-Reply-To: <486C7F93.7010308@gtcomm.net> Message-ID: <20080703195521.O6973@delplex.bde.org> References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> <486C7F93.7010308@gtcomm.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net , Ingo Flaschberger Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 10:42:02 -0000 On Thu, 3 Jul 2008, Paul wrote: > Bruce Evans wrote: >>> No polling: >>> 843762 25337 52313248 1 0 178 0 >>> 763555 0 47340414 1 0 178 0 >>> 830189 0 51471722 1 0 178 0 >>> 838724 0 52000892 1 0 178 0 >>> 813594 939 50442832 1 0 178 0 >>> 807303 763 50052790 1 0 178 0 >>> 791024 0 49043492 1 0 178 0 >>> 768316 1106 47635596 1 0 178 0 >>> Machine is maxed and is unresponsive.. >> >> That's the most interesting one. Even 1% packet loss would probably >> destroy performance, so the benchmarks that give 10-50% packet loss >> are uninteresting. >> > But you realize that it's outputting all of these packets on em3 and I'm > watching them coming out > and they are consistent with the packets received on em0 that netstat shows > are 'good' packets. Well, output is easier. I don't remember seeing the load on a taskq for em3. If there is a memory bottleneck, it might to might not be more related to running only 1 taskq per interrupt, depending on how independent the memory system is for different CPU. I think Opterons have more indenpendence here than most x86's. > I'm using a server opteron which supposedly has the best memory performance > out of any CPU right now. > Plus opterons have the biggest l1 cache, but small l2 cache. Do you think > larger l2 cache on the Xeon (6mb for 2 core) would be better? > I have a 2222 opteron coming which is 1ghz faster so we will see what happens I suspect lower latency memory would help more. Big memory systems have inherently higher latency. My little old A64 workstation and laptop have main memory latencies 3 times smaller than freebsd.org's new Core2 servers according to lmbench2 (42 nsec for the overclocked DDR PC3200 one and 55 for the DDR2 PC5400 (?) one, vs 145-155 nsec). If there are a lot of cache misses, then the extra 100 nsec can be important. Profiling of sendto() using hwpmc or perfmon shows a significant number of cache misses per packet (2 or 10?). >>> Polling ON: >>> input (em0) output >>> packets errs bytes packets errs bytes colls >>> 784138 179079 48616564 1 0 226 0 >>> 788815 129608 48906530 2 0 356 0 >>> Machine is responsive and has 40% idle cpu.. Why ALWAYS 40% ? I'm really >>> mistified by this.. >> >> Is this with hz=2000 and 256/256 and no polling in idle? 40% is easy >> to explain (perhaps incorrectly). Polling can then read at most 256 >> descriptors every 1/2000 second, giving a max throughput of 512 kpps. >> Packets < descriptors in general but might be equal here (for small >> packets). You seem to actually get 784 kpps, which is too high even >> in descriptors unless, but matches exactly if the errors are counted >> twice (784 - 179 - 505 ~= 512). CPU is getting short too, but 40% >> still happens to be left over after giving up at 512 kpps. Most of >> the errors are probably handled by the hardware at low cost in CPU by >> dropping packets. There are other types of errors but none except >> dropped packets is likely. >> > Read above, it's actually transmitting 770kpps out of em3 so it can't just be > 512kpps. Transmitting is easier, but with polling its even harder to send faster than hz * queue_length than to receive. This is without polling in idle. > I was thinking of trying 4 or 5.. but how would that work with this new > hardware? Poorly, except possibly with polling in FreeBSD-4. FreeBSD-4 generally has lower overheads and latency, but is missing important improvements (mainly tcp optimizations in upper layers, better DMA and/or mbuf handling, and support for newer NICs). FreeBSD-5 is also missing the overhead+latency advantage. Here are some benchmarks. (ttcp mainly tests sendto(). 4.10 em needed a 2-line change to support a not-so-new PCI em NIC. Summary: - my bge NIC can handle about 600 kpps on my faster machine, but only achieves 300 in 4.10 unpatched. - my em NIC can handle about 400 kpps on my slower machine, except in later versions it can receive at about 600 kpps. - only 6.x and later can achieve near wire throughput for 1500-MTU packets (81 kpps vs 76 kpps). This depends on better DMA or mbuf handling... I now remember the details -- it is mainly better mbuf handling: old versions split the 1500-MTU packets into 2 mbufs and this causes 2 descriptors per packet, which causes extra software overheads and even larger overheads for the hardware. %%% Results of benchmarks run on 23 Feb 2007: my~5.2 bge --> ~4.10 em tx rx kpps load% ips kpps load% ips ttcp -l5 -u -t 639 98 1660 398* 77 8k ttcp -l5 -t 6.0 100 3960 6.0 6 5900 ttcp -l1472 -u -t 76 27 395 76 40 8k ttcp -l1472 -t 51 40 11k 51 26 8k (*) Same as sender according to netstat -I, but systat -ip shows that almost half aren't delivered to upper layers. my~5.2 bge --> 4.11 em tx rx kpps load% ips kpps load% ips ttcp -l5 -u -t 635 98 1650 399* 74 8k ttcp -l5 -t 5.8 100 3900 5.8 6 5800 ttcp -l1472 -u -t 76 27 395 76 32 8k ttcp -l1472 -t 51 40 11k 51 25 8k (*) Same as sender according to netstat -I, but systat -ip shows that almost half aren't delivered to upper layers. my~5.2 bge --> my~5.2 em tx rx kpps load% ips kpps load% ips ttcp -l5 -u -t 638 98 1660 394* 100- 8k ttcp -l5 -t 5.8 100 3900 5.8 9 6000 ttcp -l1472 -u -t 76 27 395 76 46 8k ttcp -l1472 -t 51 40 11k 51 35 8k (*) Same as sender according to netstat -I, but systat -ip shows that almost half aren't delivered to upper layers. With the em rate limit on ips changed from 8k to 80k, about 95% are delivered up. my~5.2 bge --> 6.2 em tx rx kpps load% ips kpps load% ips ttcp -l5 -u -t 637 98 1660 637 100- 15k ttcp -l5 -t 5.8 100 3900 5.8 8 12k ttcp -l1472 -u -t 76 27 395 76 36 16k ttcp -l1472 -t 51 40 11k 51 37 16k my~5.2 bge --> ~current em-fastintr tx rx kpps load% ips kpps load% ips ttcp -l5 -u -t 641 98 1670 641 99 8k ttcp -l5 -t 5.9 100 2670 5.9 7 6k ttcp -l1472 -u -t 76 27 395 76 35 8k ttcp -l1472 -t 52 43 11k 52 30 8k ~6.2 bge --> ~current em-fastintr tx rx kpps load% ips kpps load% ips ttcp -l5 -u -t 309 62 1600 309 64 8k ttcp -l5 -t 4.9 100 3000 4.9 6 7k ttcp -l1472 -u -t 76 27 395 76 34 8k ttcp -l1472 -t 54 28 6800 54 30 8k ~current bge --> ~current em-fastintr tx rx kpps load% ips kpps load% ips ttcp -l5 -u -t 602 100 1570 602 99 8k ttcp -l5 -t 5.3 100 2660 5.3 5 5300 ttcp -l1472 -u -t 81# 19 212 81# 38 8k ttcp -l1472 -t 53 34 11k 53 30 8k (#) Wire speed to within 0.5%. This is the only kppps in this set of benchmarks that is close to wire speed. Older kernels apparently lose relative to -current because mbufs for mtu-sized packets are not contiguous in older kernels. Old results: ~4.10 bge --> my~5.2 em tx rx kpps load% ips kpps load% ips ttcp -l5 -u -t n/a n/a n/a 346 79 8k ttcp -l5 -t n/a n/a n/a 5.4 10 6800 ttcp -l1472 -u -t n/a n/a n/a 67 40 8k ttcp -l1472 -t n/a n/a n/a 51 36 8k ~4.10 kernel, =4 bge --> ~current em tx rx kpps load% ips kpps load% ips ttcp -l5 -u -t n/a n/a n/a 347 96 14k ttcp -l5 -t n/a n/a n/a 5.8 10 14k ttcp -l1472 -u -t n/a n/a n/a 67 62 14K ttcp -l1472 -t n/a n/a n/a 52 40 16k ~4.10 kernel, =4+ bge --> ~current em tx rx kpps load% ips kpps load% ips ttcp -l5 -u -t n/a n/a n/a 627 100 9k ttcp -l5 -t n/a n/a n/a 5.6 9 13k ttcp -l1472 -u -t n/a n/a n/a 68 63 14k ttcp -l1472 -t n/a n/a n/a 54 44 16k %%% %%% Results of benchmarks run on 28 Dec 2007: ~5.2 epsplex (em) ttcp: Csw Trp Sys Int Sof Sys Intr User Idle local no sink: 825k 3 206k 229 412k 52.1 45.1 2.8 local with sink: 659k 3 263k 231 131k 66.5 27.3 6.2 tx remote no sink: 35k 3 273k 8237 266k 42.0 52.1 2.3 3.6 tx remote with sink: 26k 3 394k 8224 100 60.0 5.41 3.4 11.2 rx remote no sink: 25k 4 26 8237 373k 20.6 79.4 0.0 0.0 rx remote with sink: 30k 3 203k 8237 398k 36.5 60.7 2.8 0.0 6.3-PR besplex (em) ttcp: Csw Trp Sys Int Sof Sys Intr User Idle local no sink: 417k 1 208k 418k 2 49.5 48.5 2.0 local with sink: 420k 1 276k 145k 2 70.0 23.6 6.4 tx remote no sink: 19k 2 250k 8144 2 58.5 38.7 2.8 0.0 tx remote with sink: 16k 2 361k 8336 2 72.9 24.0 3.1 4.4 rx remote no sink: 429 3 49 888 2 0.3 99.33 0.0 0.4 tx remote with sink: 13k 2 316k 5385 2 31.7 63.8 3.6 0.8 8.0-C epsplex (em-fast) ttcp: Csw Trp Sys Int Sof Sys Intr User Idle local no sink: 442k 3 221k 230 442k 47.2 49.6 2.7 local with sink: 394k 3 262k 228 131k 72.1 22.6 5.3 tx remote no sink: 17k 3 226k 7832 100 94.1 0.2 3.0 0.0 tx remote with sink: 17k 3 360k 7962 100 91.7 0.2 3.7 4.4 rx remote no sink: saturated -- cannot update systat display rx remote with sink: 15k 6 358k 8224 100 97.0 0.0 2.5 0.5 ~4.10 besplex (bge) ttcp: Csw Trp Sys Int Sof Sys Intr User Idle local no sink: 15 0 425k 228 11 96.3 0.0 3.7 local with sink: ** 0 622k 229 ** 94.7 0.3 5.0 tx remote no sink: 29 1 490k 7024 11 47.9 29.8 4.4 17.9 tx remote with sink: 26 1 635k 1883 11 65.7 11.4 5.6 17.3 rx remote no sink: 5 1 68 7025 1 0.0 47.3 0.0 52.7 rx remote with sink: 6679 2 365k 6899 12 19.7 29.2 2.5 48.7 ~5.2-C besplex (bge) ttcp: Csw Trp Sys Int Sof Sys Intr User Idle local no sink: 1M 3 271k 229 543k 50.7 46.8 2.5 local with sink: 1M 3 406k 229 203k 67.4 28.2 4.4 tx remote no sink: 49k 3 474k 11k 167k 52.3 42.7 5.0 0.0 tx remote with sink: 6371 3 641k 1900 100 76.0 16.8 6.2 0.9 rx remote no sink: 34k 3 25 11k 270k 0.8 65.4 0.0 33.8 rx remote with sink: 41k 3 365k 10k 370k 31.5 47.1 2.3 19.0 6.3-PR besplex (bge) ttcp (hz = 1000 else stathz broken): Csw Trp Sys Int Sof Sys Intr User Idle local no sink: 540k 0 270k 540k 0 50.5 46.0 3.5 local with sink: 628k 0 417k 210k 0 68.8 27.9 3.3 tx remote no sink: 15k 1 222k 7190 1 28.4 29.3 1.7 40.6 tx remote with sink: 5947 1 315k 2825 1 39.9 14.7 2.6 42.8 rx remote no sink: 13k 1 23 6943 0 0.3 49.5 0.2 50.0 rx remote with sink: 20k 1 371k 6819 0 29.5 30.1 3.9 36.5 8.0-C besplex (bge) ttcp: Csw Trp Sys Int Sof Sys Intr User Idle local no sink: 649k 3 324k 100 649k 53.9 42.9 3.2 local with sink: 649k 3 433k 100 216k 75.2 18.8 6.0 tx remote no sink: 24k 3 432k 10k 100 49.7 41.3 2.4 6.6 tx remote with sink: 3199 3 568k 1580 100 64.3 19.6 4.0 12.2 rx remote no sink: 20k 3 27 10k 100 0.0 46.1 0.0 53.9 rx remote with sink: 31k 3 370k 10k 100 30.7 30.9 4.8 33.5 %%% Bruce From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 10:46:04 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CE4E41065678 for ; Thu, 3 Jul 2008 10:46:04 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 292FB8FC1D for ; Thu, 3 Jul 2008 10:46:03 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 16240 invoked from network); 3 Jul 2008 12:46:01 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 3 Jul 2008 12:46:01 +0200 Date: Thu, 3 Jul 2008 12:46:01 +0200 (CEST) From: Ingo Flaschberger To: Stefan Lambrev In-Reply-To: <486B7C69.1010304@moneybookers.com> Message-ID: References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486B7C69.1010304@moneybookers.com> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Cc: FreeBSD Net , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 10:46:04 -0000 Dear Stefan, >>> So my maximum without polling is close to 800kpps but if I push that it >>> starts locking me from doing things, or >> >> how many kpps do you want to achieve? > Do not know for Paul but, I want to be able to route (and/or bridge to > handle) 600-700mbps syn flood, > which is something like 1500kpps in every direction. Is it unrealistic? yes, I think so. look at this project: http://yuba.stanford.edu/NetFPGA/ This card(s) could do that. Maximum count of routes seems to be limited, but with lpf it should work. A freebsd-kernel interface is missing. > If the code is optimized to fully utilize MP I do not see a reason why quad > core processor should not be able to do this. > After all single core seems to handle 500kpps, if we utilize four, eight or > even more cores we should be able to route 1500kpps + ? Theres a "sun" used at quagga dev as bgp-route-server. http://quagga.net/route-server.php (but they don't answered my question regarding fw-performance). > I hope TOE once MFCed to 7-STABLE will help too? I don't think toe will help. Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 10:52:19 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 393BB1065679 for ; Thu, 3 Jul 2008 10:52:19 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 6A77A8FC1D for ; Thu, 3 Jul 2008 10:52:18 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 20075 invoked from network); 3 Jul 2008 12:52:17 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 3 Jul 2008 12:52:17 +0200 Date: Thu, 3 Jul 2008 12:52:17 +0200 (CEST) From: Ingo Flaschberger To: Paul In-Reply-To: <486C7611.9030905@gtcomm.net> Message-ID: References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486C7611.9030905@gtcomm.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 10:52:19 -0000 Dear Paul, > Tomorrow comes opteron 2222 so it's 1ghz faster than this one, and I can see > if it scales directly with cpu speed or what happens. can you send me a lspci -v? > I did another SMP test with an interesting results. I took one of the cpus > out of the machine, so it was just left with a single 2212 (dual core) > and it performed better. Less contention I suppose? in smp locking is a performance killer. My next "router" appliance will be: http://www.axiomtek.com.tw/products/ViewProduct.asp?view=429 Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 10:57:42 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 75B531065671 for ; Thu, 3 Jul 2008 10:57:42 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id A3D118FC24 for ; Thu, 3 Jul 2008 10:57:41 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 23822 invoked from network); 3 Jul 2008 12:57:40 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 3 Jul 2008 12:57:40 +0200 Date: Thu, 3 Jul 2008 12:57:39 +0200 (CEST) From: Ingo Flaschberger To: Stefan Lambrev In-Reply-To: Message-ID: References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486B7C69.1010304@moneybookers.com> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 10:57:42 -0000 Dear Stefan, >>>> So my maximum without polling is close to 800kpps but if I push that it >>>> starts locking me from doing things, or >>> >>> how many kpps do you want to achieve? >> Do not know for Paul but, I want to be able to route (and/or bridge to >> handle) 600-700mbps syn flood, >> which is something like 1500kpps in every direction. Is it unrealistic? I would also give Dragonfly bsd a try, as Mike had the best results with it. Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 11:52:47 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C5D0D1065671 for ; Thu, 3 Jul 2008 11:52:47 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from mail36.syd.optusnet.com.au (mail36.syd.optusnet.com.au [211.29.133.76]) by mx1.freebsd.org (Postfix) with ESMTP id 4F4088FC1A for ; Thu, 3 Jul 2008 11:52:46 +0000 (UTC) (envelope-from peterjeremy@optushome.com.au) Received: from server.vk2pj.dyndns.org (c122-106-215-175.belrs3.nsw.optusnet.com.au [122.106.215.175]) by mail36.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m63BqiFF007563 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Thu, 3 Jul 2008 21:52:45 +1000 Received: from server.vk2pj.dyndns.org (localhost.vk2pj.dyndns.org [127.0.0.1]) by server.vk2pj.dyndns.org (8.14.2/8.14.2) with ESMTP id m63Bqitj026549 for ; Thu, 3 Jul 2008 21:52:44 +1000 (EST) (envelope-from peter@server.vk2pj.dyndns.org) Received: (from peter@localhost) by server.vk2pj.dyndns.org (8.14.2/8.14.2/Submit) id m63Bqh61026548 for freebsd-net@freebsd.org; Thu, 3 Jul 2008 21:52:44 +1000 (EST) (envelope-from peter) Date: Thu, 3 Jul 2008 21:52:43 +1000 From: Peter Jeremy To: freebsd-net@freebsd.org Message-ID: <20080703115243.GR29380@server.vk2pj.dyndns.org> References: <20080703025822.GA24765@server.vk2pj.dyndns.org> <486C8446.9060302@moneybookers.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="5CUMAwwhRxlRszMD" Content-Disposition: inline In-Reply-To: <486C8446.9060302@moneybookers.com> X-PGP-Key: http://members.optusnet.com.au/peterjeremy/pubkey.asc User-Agent: Mutt/1.5.18 (2008-05-17) Subject: Re: arplookup x.x.x.x failed: host is not on local network X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 11:52:48 -0000 --5CUMAwwhRxlRszMD Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable OK, my responses to the replies so far. One off-line reply requested a topology and netstat output. Since the toplogy may be relevant, below is an extremely simplified approximation (the real network has about 60 subnets and about 70 hosts, each appearing in between two and four subnets). Corp Network 192.168.10.0/24 | 192.168.12.0/24 +------+-------------+----------| | |----------+-------------+-----+ .1| .2| .254| | |.254 .3| .4| +-------+ +-------+ +-------+ +-------+ +-------+ | | | | | | | | | | | host1 | | host2 | | NAT | | host3 | | host4 | | | | | | | | | | | +-------+ +-------+ +-------+ +-------+ +-------+ .1| .2| .254| |.254 .3| .4| +------+-------------+----------| |----------+-------------+-----+ 192.168.11.0/24 192.168.13.0/24 The errors appear to be randomly spread across hosts and subnets. It does not appear consistently and seems to correlate with load (I am getting significant numbers at present and the NAT host is routing about 90Kpps and 100MBps if netstat can be believed). The problem also shows up on another interior routing host that has visibility across the internal networks so it isn't related to NAT or directly related to host load (that host is only seeing about 3.5Kpps - but is also a much slower host). I have managed to capture a tcpdump across the error. syslog reported: Jul 3 21:28:30 xxxx kernel: arplookup 192.168.169.26 failed: host is not o= n local network and the packets for that host during that second are: 21:28:30.320340 00:0b:cd:d6:66:26 > 00:03:ba:ab:6f:ef, ethertype 802.1Q (0x= 8100), length 64: vlan 169, p 0, ethertype IPv4, IP (tos 0x0, ttl 64, id 2= 9304, offset 0, flags [none], length: 28) 192.168.169.26 > 192.168.169.111:= icmp 8: echo request seq 35079 21:28:30.320429 00:d0:b7:20:8f:ee > 00:03:ba:ab:6f:ef, ethertype 802.1Q (0x= 8100), length 46: vlan 168, p 0, ethertype IPv4, IP (tos 0x0, ttl 63, id 2= 9304, offset 0, flags [none], length: 28) 192.168.169.26 > 192.168.169.111:= icmp 8: echo request seq 35079 21:28:30.320445 00:0b:cd:d6:66:26 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x= 8100), length 64: vlan 169, p 0, ethertype ARP, arp who-has 192.168.169.250= tell 192.168.169.26 21:28:30.320459 00:0b:cd:d6:66:26 > 00:d0:b7:20:8f:ee, ethertype 802.1Q (0x= 8100), length 64: vlan 169, p 0, ethertype IPv4, IP (tos 0x0, ttl 64, id 2= 9307, offset 0, flags [none], length: 28) 192.168.169.26 > 192.168.169.250:= icmp 8: echo request seq 35079 21:28:30.320493 00:d0:b7:20:8f:ee > 00:0b:cd:d6:66:e4, ethertype 802.1Q (0x= 8100), length 46: vlan 168, p 0, ethertype IPv4, IP (tos 0x0, ttl 64, id 1= 5305, offset 0, flags [none], length: 28) 192.168.169.250 > 192.168.169.26:= icmp 8: echo reply seq 35079 21:28:30.320531 00:d0:b7:20:8f:ee > 00:0b:cd:d6:66:26, ethertype 802.1Q (0x= 8100), length 46: vlan 169, p 0, ethertype ARP, arp reply 192.168.169.250 i= s-at 00:d0:b7:20:8f:ee (this was captured MAC 00:d0:b7:20:8f:ee). Possibly, I'm seeing packet leakage from the switches and that is confusing FreeBSD - definitely the first packet above should not be visible. On 2008-Jul-03 09:05:15 +0200, Daniel Ponticello wrote: >i'm having exactly the same problem, but without NAT configuration. Just= =20 >a simple host on network 192.168.170.xxx >that when tries to reach an host on 192.168.181.xxx: it gives the same err= or Except that in my case, the addresses _are_ local. On 2008-Jul-03 02:16:30 -0500, David DeSimone wrote: >My theory is that this is a response to ARP requests. ARP requests are >broadcast, so the BSD box hears someone asking for this IP, but cannot >find it on any local interfaces, and so complains that it cannot >construct a proper reply. Except that the address it's complaining about is on a local subnet. Interestingly, in the above case, the host is spuriously seeing a packet and has re-routed it via vlan168 - which is the wrong subnet, though the destination host will still see it there. On 2008-Jul-03 10:48:22 +0300, Stefan Lambrev wrote: >I bet 192.168.181.114 have a wrong network mask ;) You lose. --=20 Peter Jeremy Please excuse any delays as the result of my ISP's inability to implement an MTA that is either RFC2821-compliant or matches their claimed behaviour. --5CUMAwwhRxlRszMD Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.9 (FreeBSD) iEYEARECAAYFAkhsvYsACgkQ/opHv/APuIed6QCeJ+STyhbqADxqD8AS4Wr9hbAy rFIAoIDQYODT2p6Zae0xFic7S4zSFI1B =rEx7 -----END PGP SIGNATURE----- --5CUMAwwhRxlRszMD-- From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 12:48:52 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 21D1A10656CB for ; Thu, 3 Jul 2008 12:48:52 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id BA5D78FC1B for ; Thu, 3 Jul 2008 12:48:51 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KEOBO-0004BE-Ke; Thu, 03 Jul 2008 08:45:06 -0400 Message-ID: <486CCB29.3080308@gtcomm.net> Date: Thu, 03 Jul 2008 08:50:49 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Bruce Evans References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> <486C7F93.7010308@gtcomm.net> <20080703195521.O6973@delplex.bde. org> In-Reply-To: <20080703195521.O6973@delplex.bde.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Ingo Flaschberger Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 12:48:52 -0000 Bruce Evans wrote: > On Thu, 3 Jul 2008, Paul wrote: > >> Bruce Evans wrote: >>>> No polling: >>>> 843762 25337 52313248 1 0 178 0 >>>> 763555 0 47340414 1 0 178 0 >>>> 830189 0 51471722 1 0 178 0 >>>> 838724 0 52000892 1 0 178 0 >>>> 813594 939 50442832 1 0 178 0 >>>> 807303 763 50052790 1 0 178 0 >>>> 791024 0 49043492 1 0 178 0 >>>> 768316 1106 47635596 1 0 178 0 >>>> Machine is maxed and is unresponsive.. >>> >>> That's the most interesting one. Even 1% packet loss would probably >>> destroy performance, so the benchmarks that give 10-50% packet loss >>> are uninteresting. >>> >> But you realize that it's outputting all of these packets on em3 and >> I'm watching them coming out >> and they are consistent with the packets received on em0 that netstat >> shows are 'good' packets. > > Well, output is easier. I don't remember seeing the load on a taskq for > em3. If there is a memory bottleneck, it might to might not be more > related > to running only 1 taskq per interrupt, depending on how independent the > memory system is for different CPU. I think Opterons have more > indenpendence > here than most x86's. > Opterons have on cpu memory controller.. That should help a little. :P But I must be getting more than 1 packet per descriptor because I can do HZ=100 and still get it without polling.. idle polling helps in all cases of polling that I have tested it with, seems moreso on 32 bit >> I'm using a server opteron which supposedly has the best memory >> performance out of any CPU right now. >> Plus opterons have the biggest l1 cache, but small l2 cache. Do you >> think larger l2 cache on the Xeon (6mb for 2 core) would be better? >> I have a 2222 opteron coming which is 1ghz faster so we will see what >> happens > > I suspect lower latency memory would help more. Big memory systems > have inherently higher latency. My little old A64 workstation and > laptop have main memory latencies 3 times smaller than freebsd.org's > new Core2 servers according to lmbench2 (42 nsec for the overclocked > DDR PC3200 one and 55 for the DDR2 PC5400 (?) one, vs 145-155 nsec). > If there are a lot of cache misses, then the extra 100 nsec can be > important. Profiling of sendto() using hwpmc or perfmon shows a > significant number of cache misses per packet (2 or 10?). > The opterons are 667mhz DDR2 [registered], I have a Xeon that is ddr3 but i think the latency is higher than ddr2. I'll look up those programs you mentioned and see If I can run some tests. >>>> Polling ON: >>>> input (em0) output >>>> packets errs bytes packets errs bytes colls >>>> 784138 179079 48616564 1 0 226 0 >>>> 788815 129608 48906530 2 0 356 0 >>>> Machine is responsive and has 40% idle cpu.. Why ALWAYS 40% ? I'm >>>> really mistified by this.. >>> >>> Is this with hz=2000 and 256/256 and no polling in idle? 40% is easy >>> to explain (perhaps incorrectly). Polling can then read at most 256 >>> descriptors every 1/2000 second, giving a max throughput of 512 kpps. >>> Packets < descriptors in general but might be equal here (for small >>> packets). You seem to actually get 784 kpps, which is too high even >>> in descriptors unless, but matches exactly if the errors are counted >>> twice (784 - 179 - 505 ~= 512). CPU is getting short too, but 40% >>> still happens to be left over after giving up at 512 kpps. Most of >>> the errors are probably handled by the hardware at low cost in CPU by >>> dropping packets. There are other types of errors but none except >>> dropped packets is likely. >>> >> Read above, it's actually transmitting 770kpps out of em3 so it can't >> just be 512kpps. > > Transmitting is easier, but with polling its even harder to send > faster than > hz * queue_length than to receive. This is without polling in idle. > What i'm saying though, it that it's not giving up at 512kpps because 784kpps is coming in em0 and going out em3 so obviously it's reading more than 256 every 1/2000th of a second (packets). What would be the best settings (theoretical) for 1mpps processing? I actually don't have a problem 'receiving' more than 800kpps with much lower CPU usage if it's going to blackhole . so obviously it can receive a lot more, maybe even line rate pps but i can't generate that much. >> I was thinking of trying 4 or 5.. but how would that work with this >> new hardware? > > Poorly, except possibly with polling in FreeBSD-4. FreeBSD-4 generally > has lower overheads and latency, but is missing important improvements > (mainly tcp optimizations in upper layers, better DMA and/or mbuf > handling, and support for newer NICs). FreeBSD-5 is also missing the > overhead+latency advantage. > > Here are some benchmarks. (ttcp mainly tests sendto(). 4.10 em needed a > 2-line change to support a not-so-new PCI em NIC. Summary: > - my bge NIC can handle about 600 kpps on my faster machine, but only > achieves 300 in 4.10 unpatched. > - my em NIC can handle about 400 kpps on my slower machine, except in > later versions it can receive at about 600 kpps. > - only 6.x and later can achieve near wire throughput for 1500-MTU > packets (81 kpps vs 76 kpps). This depends on better DMA or mbuf > handling... I now remember the details -- it is mainly better mbuf > handling: old versions split the 1500-MTU packets into 2 mbufs and > this causes 2 descriptors per packet, which causes extra software > overheads and even larger overheads for the hardware. > > %%% > Results of benchmarks run on 23 Feb 2007: > > my~5.2 bge --> ~4.10 em > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t 639 98 1660 398* 77 8k > ttcp -l5 -t 6.0 100 3960 6.0 6 5900 > ttcp -l1472 -u -t 76 27 395 76 40 8k > ttcp -l1472 -t 51 40 11k 51 26 8k > > (*) Same as sender according to netstat -I, but systat -ip shows that > almost half aren't delivered to upper layers. > > my~5.2 bge --> 4.11 em > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t 635 98 1650 399* 74 8k > ttcp -l5 -t 5.8 100 3900 5.8 6 5800 > ttcp -l1472 -u -t 76 27 395 76 32 8k > ttcp -l1472 -t 51 40 11k 51 25 8k > > (*) Same as sender according to netstat -I, but systat -ip shows that > almost half aren't delivered to upper layers. > > my~5.2 bge --> my~5.2 em > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t 638 98 1660 394* 100- 8k > ttcp -l5 -t 5.8 100 3900 5.8 9 6000 > ttcp -l1472 -u -t 76 27 395 76 46 8k > ttcp -l1472 -t 51 40 11k 51 35 8k > > (*) Same as sender according to netstat -I, but systat -ip shows that > almost half aren't delivered to upper layers. With the em rate > limit on ips changed from 8k to 80k, about 95% are delivered up. > > my~5.2 bge --> 6.2 em > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t 637 98 1660 637 100- 15k > ttcp -l5 -t 5.8 100 3900 5.8 8 12k > ttcp -l1472 -u -t 76 27 395 76 36 16k > ttcp -l1472 -t 51 40 11k 51 37 16k > > my~5.2 bge --> ~current em-fastintr > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t 641 98 1670 641 99 8k > ttcp -l5 -t 5.9 100 2670 5.9 7 6k > ttcp -l1472 -u -t 76 27 395 76 35 8k > ttcp -l1472 -t 52 43 11k 52 30 8k > > ~6.2 bge --> ~current em-fastintr > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t 309 62 1600 309 64 8k > ttcp -l5 -t 4.9 100 3000 4.9 6 7k > ttcp -l1472 -u -t 76 27 395 76 34 8k > ttcp -l1472 -t 54 28 6800 54 30 8k > > ~current bge --> ~current em-fastintr > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t 602 100 1570 602 99 8k > ttcp -l5 -t 5.3 100 2660 5.3 5 5300 > ttcp -l1472 -u -t 81# 19 212 81# 38 8k > ttcp -l1472 -t 53 34 11k 53 30 8k > > (#) Wire speed to within 0.5%. This is the only kppps in this set of > benchmarks that is close to wire speed. Older kernels apparently > lose relative to -current because mbufs for mtu-sized packets are > not contiguous in older kernels. > > Old results: > > ~4.10 bge --> my~5.2 em > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t n/a n/a n/a 346 79 8k > ttcp -l5 -t n/a n/a n/a 5.4 10 6800 > ttcp -l1472 -u -t n/a n/a n/a 67 40 8k > ttcp -l1472 -t n/a n/a n/a 51 36 8k > > ~4.10 kernel, =4 bge --> ~current em > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t n/a n/a n/a 347 96 14k > ttcp -l5 -t n/a n/a n/a 5.8 10 14k > ttcp -l1472 -u -t n/a n/a n/a 67 62 14K > ttcp -l1472 -t n/a n/a n/a 52 40 16k > > ~4.10 kernel, =4+ bge --> ~current em > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t n/a n/a n/a 627 100 9k > ttcp -l5 -t n/a n/a n/a 5.6 9 13k > ttcp -l1472 -u -t n/a n/a n/a 68 63 14k > ttcp -l1472 -t n/a n/a n/a 54 44 16k > %%% > > %%% > Results of benchmarks run on 28 Dec 2007: > > ~5.2 epsplex (em) ttcp: > Csw Trp Sys Int Sof Sys Intr User Idle > local no sink: 825k 3 206k 229 412k 52.1 45.1 2.8 > local with sink: 659k 3 263k 231 131k 66.5 27.3 6.2 > tx remote no sink: 35k 3 273k 8237 266k 42.0 52.1 2.3 3.6 > tx remote with sink: 26k 3 394k 8224 100 60.0 5.41 3.4 11.2 > rx remote no sink: 25k 4 26 8237 373k 20.6 79.4 0.0 0.0 > rx remote with sink: 30k 3 203k 8237 398k 36.5 60.7 2.8 0.0 > > 6.3-PR besplex (em) ttcp: > Csw Trp Sys Int Sof Sys Intr User Idle > local no sink: 417k 1 208k 418k 2 49.5 48.5 2.0 > local with sink: 420k 1 276k 145k 2 70.0 23.6 6.4 > tx remote no sink: 19k 2 250k 8144 2 58.5 38.7 2.8 0.0 > tx remote with sink: 16k 2 361k 8336 2 72.9 24.0 3.1 4.4 > rx remote no sink: 429 3 49 888 2 0.3 99.33 0.0 0.4 > tx remote with sink: 13k 2 316k 5385 2 31.7 63.8 3.6 0.8 > > 8.0-C epsplex (em-fast) ttcp: > Csw Trp Sys Int Sof Sys Intr User Idle > local no sink: 442k 3 221k 230 442k 47.2 49.6 2.7 > local with sink: 394k 3 262k 228 131k 72.1 22.6 5.3 > tx remote no sink: 17k 3 226k 7832 100 94.1 0.2 3.0 0.0 > tx remote with sink: 17k 3 360k 7962 100 91.7 0.2 3.7 4.4 > rx remote no sink: saturated -- cannot update systat display > rx remote with sink: 15k 6 358k 8224 100 97.0 0.0 2.5 0.5 > > ~4.10 besplex (bge) ttcp: > Csw Trp Sys Int Sof Sys Intr User Idle > local no sink: 15 0 425k 228 11 96.3 0.0 3.7 > local with sink: ** 0 622k 229 ** 94.7 0.3 5.0 > tx remote no sink: 29 1 490k 7024 11 47.9 29.8 4.4 17.9 > tx remote with sink: 26 1 635k 1883 11 65.7 11.4 5.6 17.3 > rx remote no sink: 5 1 68 7025 1 0.0 47.3 0.0 52.7 > rx remote with sink: 6679 2 365k 6899 12 19.7 29.2 2.5 48.7 > > ~5.2-C besplex (bge) ttcp: > Csw Trp Sys Int Sof Sys Intr User Idle > local no sink: 1M 3 271k 229 543k 50.7 46.8 2.5 > local with sink: 1M 3 406k 229 203k 67.4 28.2 4.4 > tx remote no sink: 49k 3 474k 11k 167k 52.3 42.7 5.0 0.0 > tx remote with sink: 6371 3 641k 1900 100 76.0 16.8 6.2 0.9 > rx remote no sink: 34k 3 25 11k 270k 0.8 65.4 0.0 33.8 > rx remote with sink: 41k 3 365k 10k 370k 31.5 47.1 2.3 19.0 > > 6.3-PR besplex (bge) ttcp (hz = 1000 else stathz broken): > Csw Trp Sys Int Sof Sys Intr User Idle > local no sink: 540k 0 270k 540k 0 50.5 46.0 3.5 > local with sink: 628k 0 417k 210k 0 68.8 27.9 3.3 > tx remote no sink: 15k 1 222k 7190 1 28.4 29.3 1.7 40.6 > tx remote with sink: 5947 1 315k 2825 1 39.9 14.7 2.6 42.8 > rx remote no sink: 13k 1 23 6943 0 0.3 49.5 0.2 50.0 > rx remote with sink: 20k 1 371k 6819 0 29.5 30.1 3.9 36.5 > > 8.0-C besplex (bge) ttcp: > Csw Trp Sys Int Sof Sys Intr User Idle > local no sink: 649k 3 324k 100 649k 53.9 42.9 3.2 > local with sink: 649k 3 433k 100 216k 75.2 18.8 6.0 > tx remote no sink: 24k 3 432k 10k 100 49.7 41.3 2.4 6.6 > tx remote with sink: 3199 3 568k 1580 100 64.3 19.6 4.0 12.2 > rx remote no sink: 20k 3 27 10k 100 0.0 46.1 0.0 53.9 > rx remote with sink: 31k 3 370k 10k 100 30.7 30.9 4.8 33.5 > %%% > > Bruce > From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 12:50:37 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D96C51065684 for ; Thu, 3 Jul 2008 12:50:37 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 91D948FC31 for ; Thu, 3 Jul 2008 12:50:37 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KEODB-0004Oq-C8; Thu, 03 Jul 2008 08:46:57 -0400 Message-ID: <486CCB98.5000805@gtcomm.net> Date: Thu, 03 Jul 2008 08:52:40 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486C7611.9030905@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 12:50:38 -0000 Err.. pciconf -lv ? none0@pci0:0:0:0: class=0x050000 card=0x151115d9 chip=0x036910de rev=0xa2 hdr=0x00 vendor = 'Nvidia Corp' device = 'MCP55 Memory Controller' class = memory subclass = RAM isab0@pci0:0:1:0: class=0x060100 card=0x151115d9 chip=0x036410de rev=0xa3 hdr=0x00 vendor = 'Nvidia Corp' device = 'MCP55 LPC Bridge' class = bridge subclass = PCI-ISA none1@pci0:0:1:1: class=0x0c0500 card=0x151115d9 chip=0x036810de rev=0xa3 hdr=0x00 vendor = 'Nvidia Corp' device = 'MCP55 SMBus' class = serial bus subclass = SMBus ohci0@pci0:0:2:0: class=0x0c0310 card=0x151115d9 chip=0x036c10de rev=0xa1 hdr=0x00 vendor = 'Nvidia Corp' device = 'MCP55 USB Controller' class = serial bus subclass = USB ehci0@pci0:0:2:1: class=0x0c0320 card=0x151115d9 chip=0x036d10de rev=0xa2 hdr=0x00 vendor = 'Nvidia Corp' device = 'MCP55 USB Controller' class = serial bus subclass = USB atapci0@pci0:0:4:0: class=0x01018a card=0x151115d9 chip=0x036e10de rev=0xa1 hdr=0x00 vendor = 'Nvidia Corp' device = 'MCP55 IDE' class = mass storage subclass = ATA atapci1@pci0:0:5:0: class=0x010185 card=0x151115d9 chip=0x037f10de rev=0xa3 hdr=0x00 vendor = 'Nvidia Corp' device = 'MCP55 SATA Controller' class = mass storage subclass = ATA atapci2@pci0:0:5:1: class=0x010185 card=0x151115d9 chip=0x037f10de rev=0xa3 hdr=0x00 vendor = 'Nvidia Corp' device = 'MCP55 SATA Controller' class = mass storage subclass = ATA atapci3@pci0:0:5:2: class=0x010185 card=0x151115d9 chip=0x037f10de rev=0xa3 hdr=0x00 vendor = 'Nvidia Corp' device = 'MCP55 SATA Controller' class = mass storage subclass = ATA pcib1@pci0:0:6:0: class=0x060401 card=0x151115d9 chip=0x037010de rev=0xa2 hdr=0x01 vendor = 'Nvidia Corp' device = 'MCP55 PCI bridge' class = bridge subclass = PCI-PCI nfe0@pci0:0:8:0: class=0x020000 card=0x151115d9 chip=0x037210de rev=0xa3 hdr=0x00 vendor = 'Nvidia Corp' device = 'MCP55 Ethernet' class = network subclass = ethernet nfe1@pci0:0:9:0: class=0x020000 card=0x151115d9 chip=0x037210de rev=0xa3 hdr=0x00 vendor = 'Nvidia Corp' device = 'MCP55 Ethernet' class = network subclass = ethernet pcib2@pci0:0:10:0: class=0x060400 card=0x000010de chip=0x037610de rev=0xa3 hdr=0x01 vendor = 'Nvidia Corp' device = 'MCP55 PCIe bridge' class = bridge subclass = PCI-PCI pcib5@pci0:0:13:0: class=0x060400 card=0x000010de chip=0x037810de rev=0xa3 hdr=0x01 vendor = 'Nvidia Corp' device = 'MCP55 PCIe bridge' class = bridge subclass = PCI-PCI pcib6@pci0:0:14:0: class=0x060400 card=0x000010de chip=0x037510de rev=0xa3 hdr=0x01 vendor = 'Nvidia Corp' device = 'MCP55 PCIe bridge' class = bridge subclass = PCI-PCI pcib7@pci0:0:15:0: class=0x060400 card=0x000010de chip=0x037710de rev=0xa3 hdr=0x01 vendor = 'Nvidia Corp' device = 'MCP55 PCIe bridge' class = bridge subclass = PCI-PCI hostb0@pci0:0:24:0: class=0x060000 card=0x00000000 chip=0x11001022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron HyperTransport Technology Configuration' class = bridge subclass = HOST-PCI hostb1@pci0:0:24:1: class=0x060000 card=0x00000000 chip=0x11011022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron Address Map' class = bridge subclass = HOST-PCI hostb2@pci0:0:24:2: class=0x060000 card=0x00000000 chip=0x11021022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron DRAM Controller' class = bridge subclass = HOST-PCI hostb3@pci0:0:24:3: class=0x060000 card=0x00000000 chip=0x11031022 rev=0x00 hdr=0x00 vendor = 'Advanced Micro Devices (AMD)' device = '(K8) Athlon 64/Opteron Miscellaneous Control' class = bridge subclass = HOST-PCI vgapci0@pci0:1:6:0: class=0x030000 card=0x151115d9 chip=0x515e1002 rev=0x02 hdr=0x00 vendor = 'ATI Technologies Inc' device = 'Radeon ES1000 Radeon ES1000' class = display subclass = VGA pcib3@pci0:2:0:0: class=0x060400 card=0x00000000 chip=0x01251033 rev=0x07 hdr=0x01 vendor = 'NEC Electronics Hong Kong' class = bridge subclass = PCI-PCI pcib4@pci0:2:0:1: class=0x060400 card=0x00000000 chip=0x01251033 rev=0x07 hdr=0x01 vendor = 'NEC Electronics Hong Kong' class = bridge subclass = PCI-PCI pcib8@pci0:7:0:0: class=0x060400 card=0x00000000 chip=0x8018111d rev=0x04 hdr=0x01 vendor = 'Integrated Device Technology Inc.' class = bridge subclass = PCI-PCI pcib9@pci0:8:0:0: class=0x060400 card=0x00000000 chip=0x8018111d rev=0x04 hdr=0x01 vendor = 'Integrated Device Technology Inc.' class = bridge subclass = PCI-PCI pcib10@pci0:8:1:0: class=0x060400 card=0x00000000 chip=0x8018111d rev=0x04 hdr=0x01 vendor = 'Integrated Device Technology Inc.' class = bridge subclass = PCI-PCI em0@pci0:9:0:0: class=0x020000 card=0x10a48086 chip=0x10a48086 rev=0x06 hdr=0x00 vendor = 'Intel Corporation' device = '82571EB Gigabit Ethernet Controller' class = network subclass = ethernet em1@pci0:9:0:1: class=0x020000 card=0x10a48086 chip=0x10a48086 rev=0x06 hdr=0x00 vendor = 'Intel Corporation' device = '82571EB Gigabit Ethernet Controller' class = network subclass = ethernet em2@pci0:10:0:0: class=0x020000 card=0x10a48086 chip=0x10a48086 rev=0x06 hdr=0x00 vendor = 'Intel Corporation' device = '82571EB Gigabit Ethernet Controller' class = network subclass = ethernet em3@pci0:10:0:1: class=0x020000 card=0x10a48086 chip=0x10a48086 rev=0x06 hdr=0x00 vendor = 'Intel Corporation' device = '82571EB Gigabit Ethernet Controller' class = network subclass = ethernet Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-RELEASE #0: Sun Feb 24 10:35:36 UTC 2008 root@driscoll.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Dual-Core AMD Opteron(tm) Processor 2212 (2010.32-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x40f12 Stepping = 2 Features=0x178bfbff Features2=0x2001 AMD Features=0xea500800 AMD Features2=0x1f Cores per package: 2 usable memory = 1060921344 (1011 MB) avail memory = 1022271488 (974 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) hptrr: HPT RocketRAID controller driver v1.1 (Feb 24 2008 10:34:18) acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of fec00000, 1000 (3) failed acpi0: reservation of fee00000, 1000 (3) failed acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, 3ff00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x2008-0x200b on acpi0 cpu0: on acpi0 powernow0: on cpu0 device_attach: powernow0 attach returned 6 cpu1: on acpi0 powernow1: on cpu1 device_attach: powernow1 attach returned 6 Ingo Flaschberger wrote: > Dear Paul, > >> Tomorrow comes opteron 2222 so it's 1ghz faster than this one, and I >> can see if it scales directly with cpu speed or what happens. > > can you send me a lspci -v? > >> I did another SMP test with an interesting results. I took one of the >> cpus out of the machine, so it was just left with a single 2212 (dual >> core) >> and it performed better. Less contention I suppose? > > in smp locking is a performance killer. > > My next "router" appliance will be: > http://www.axiomtek.com.tw/products/ViewProduct.asp?view=429 > > Kind regards, > Ingo Flaschberger > From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 12:51:49 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 974B3106567D for ; Thu, 3 Jul 2008 12:51:49 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: from ibctech.ca (v6.ibctech.ca [IPv6:2607:f118::b6]) by mx1.freebsd.org (Postfix) with SMTP id 18FFC8FC1B for ; Thu, 3 Jul 2008 12:51:49 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: (qmail 3893 invoked by uid 89); 3 Jul 2008 12:53:41 -0000 Received: from unknown (HELO ?IPv6:2607:f118::5?) (steve@ibctech.ca@2607:f118::5) by 2607:f118::b6 with ESMTPA; 3 Jul 2008 12:53:41 -0000 Message-ID: <486CCB6A.6070104@ibctech.ca> Date: Thu, 03 Jul 2008 08:51:54 -0400 From: Steve Bertrand User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486C7611.9030905@gtcomm.net> In-Reply-To: X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 12:51:49 -0000 Ingo Flaschberger wrote: > My next "router" appliance will be: > http://www.axiomtek.com.tw/products/ViewProduct.asp?view=429 This is exactly the device that I have been testing with (just rebranded). Steve From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 12:55:37 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACC261065674 for ; Thu, 3 Jul 2008 12:55:37 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id DB3CC8FC18 for ; Thu, 3 Jul 2008 12:55:36 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 3434 invoked from network); 3 Jul 2008 14:55:35 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 3 Jul 2008 14:55:35 +0200 Date: Thu, 3 Jul 2008 14:55:34 +0200 (CEST) From: Ingo Flaschberger To: Steve Bertrand In-Reply-To: <486CCB6A.6070104@ibctech.ca> Message-ID: References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486C7611.9030905@gtcomm.net> <486CCB6A.6070104@ibctech.ca> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 12:55:37 -0000 Dear Steve, >> My next "router" appliance will be: >> http://www.axiomtek.com.tw/products/ViewProduct.asp?view=429 > > This is exactly the device that I have been testing with (just rebranded). cool. what performace do you reach? Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 13:02:19 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5D1DB1065672 for ; Thu, 3 Jul 2008 13:02:19 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: from ibctech.ca (v6.ibctech.ca [IPv6:2607:f118::b6]) by mx1.freebsd.org (Postfix) with SMTP id E92A18FC1B for ; Thu, 3 Jul 2008 13:02:18 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: (qmail 4272 invoked by uid 89); 3 Jul 2008 13:04:11 -0000 Received: from unknown (HELO ?IPv6:2607:f118::5?) (steve@ibctech.ca@2607:f118::5) by 2607:f118::b6 with ESMTPA; 3 Jul 2008 13:04:11 -0000 Message-ID: <486CCDE0.9030901@ibctech.ca> Date: Thu, 03 Jul 2008 09:02:24 -0400 From: Steve Bertrand User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486C7611.9030905@gtcomm.net> <486CCB6A.6070104@ibctech.ca> In-Reply-To: X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 13:02:19 -0000 Ingo Flaschberger wrote: > Dear Steve, > >>> My next "router" appliance will be: >>> http://www.axiomtek.com.tw/products/ViewProduct.asp?view=429 >> >> This is exactly the device that I have been testing with (just >> rebranded). > > cool. > what performace do you reach? It's hard to say right now as I've really only been testing it with BGP and mpd. The only pps testing I've done have been with 100Mbps hosts as the only cards I have for hosts on either side are 're' cards which I have bad luck with. I should be getting more Intel GigE cards today, so I'll be testing proper pps throughput tomorrow. On a side, do you know where I can keep track of driver progress for 're'? I have a boatload of these cards and can't get them to operate properly under any version of FreeBSD. If anyone has any specific tests that they want run on this box, including any tweaking, let me know. Also, I run this box by booting it from USB thumbdrive, so it's trivial for me to simply dd the thumbdrive to another, and have multiple configurations. Steve From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 15:36:17 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 359D11065675 for ; Thu, 3 Jul 2008 15:36:17 +0000 (UTC) (envelope-from fabien.thomas@netasq.com) Received: from netasq.netasq.com (netasq.netasq.com [213.30.137.178]) by mx1.freebsd.org (Postfix) with ESMTP id 6238E8FC17 for ; Thu, 3 Jul 2008 15:36:16 +0000 (UTC) (envelope-from fabien.thomas@netasq.com) Received: from [10.20.1.5] (unknown [10.0.0.126]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (Client did not present a certificate) by netasq.netasq.com (Postfix) with ESMTP id 47C352C162; Thu, 3 Jul 2008 17:18:07 +0200 (CEST) Message-Id: <07AF62F2-E35F-4C2B-8C59-9F4E0249BD2A@netasq.com> From: Fabien Thomas To: Paul In-Reply-To: Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v926) Date: Thu, 3 Jul 2008 17:18:06 +0200 References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486B7C69.1010304@moneybookers.com> X-Mailer: Apple Mail (2.926) Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 15:36:17 -0000 For your information we have mesured 730Kpps using pollng and fastforwarding with 64bits frame without loss (<0.001% packet loss) on a Spirent Smarbits (Pentium D 2.8GHZ + 8xGig em) You can find the code / and some performance report at : http://www.netasq.com/opensource/pollng-rev1-freebsd.tgz The best performance / CPU cost ratio is to use 1 core only and the others core are free to do application processing. Fabien From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 16:01:24 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E9A5106566C for ; Thu, 3 Jul 2008 16:01:24 +0000 (UTC) (envelope-from thompsa@FreeBSD.org) Received: from pele.citylink.co.nz (pele.citylink.co.nz [202.8.44.226]) by mx1.freebsd.org (Postfix) with ESMTP id 4BF658FC1B for ; Thu, 3 Jul 2008 16:01:24 +0000 (UTC) (envelope-from thompsa@FreeBSD.org) Received: from localhost (localhost [127.0.0.1]) by pele.citylink.co.nz (Postfix) with ESMTP id B9B692BE8E; Fri, 4 Jul 2008 04:01:22 +1200 (NZST) X-Virus-Scanned: Debian amavisd-new at citylink.co.nz Received: from pele.citylink.co.nz ([127.0.0.1]) by localhost (pele.citylink.co.nz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 367+z-Upqick; Fri, 4 Jul 2008 04:01:19 +1200 (NZST) Received: from citylink.fud.org.nz (unknown [202.8.44.45]) by pele.citylink.co.nz (Postfix) with ESMTP; Fri, 4 Jul 2008 04:01:19 +1200 (NZST) Received: by citylink.fud.org.nz (Postfix, from userid 1001) id BDAFB1142E; Fri, 4 Jul 2008 04:02:46 +1200 (NZST) Date: Thu, 3 Jul 2008 09:02:46 -0700 From: Andrew Thompson To: Stefan Lambrev Message-ID: <20080703160246.GA45363@citylink.fud.org.nz> References: <4868A34C.6030304@moneybookers.com> <20080630101629.GD79537@cdnetworks.co.kr> <20080701012531.GA92392@citylink.fud.org.nz> <4869FE2E.4070805@moneybookers.com> <486A0281.208@moneybookers.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <486A0281.208@moneybookers.com> User-Agent: Mutt/1.5.17 (2007-11-01) Cc: freebsd-net@freebsd.org Subject: Re: if_bridge turns off checksum offload of members? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 16:01:24 -0000 On Tue, Jul 01, 2008 at 01:10:09PM +0300, Stefan Lambrev wrote: > Hi, > > Sorry to reply to myself. > > Stefan Lambrev wrote: >> Hi, >> >> May be a stupid questions, but: >> >> 1) There are zero matches of IFCAP_TOE in kernel sources .. there is not >> support for TOE in 7.0, but may be this is work in progress for 8-current? >> 2) In #define BRIDGE_IFCAPS_MASK (IFCAP_TOE|IFCAP_TSO|IFCAP_TXCSUM) - TOE >> should be repleaced with RXCSUM or just removed? > Your patch plus this small change (replacing TOE with RXCSUM) seems to work > fine for me - kernel compiles without a problem and checksum offload is > enabled after reboot. I have committed an updated version of this patch, thanks for testing. Andrew From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 16:20:41 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 70EAA1065676 for ; Thu, 3 Jul 2008 16:20:41 +0000 (UTC) (envelope-from kian.mohageri@gmail.com) Received: from hs-out-0708.google.com (hs-out-0708.google.com [64.233.178.250]) by mx1.freebsd.org (Postfix) with ESMTP id 0B7698FC1A for ; Thu, 3 Jul 2008 16:20:40 +0000 (UTC) (envelope-from kian.mohageri@gmail.com) Received: by hs-out-0708.google.com with SMTP id h53so242496hsh.11 for ; Thu, 03 Jul 2008 09:20:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=kfeGtfjs08OEStUdSly5zk0y4nyRGWlIwufnWTZLZv4=; b=kIFYrNDOneF70sAVHJCvOv1+meGjLUZ/nT6vEtpI8VoPfv6f+UzQLfThZu2JfeWzF0 yZ5/uH0gBN3/ndvingpF1fRaiPgBzpFgYAuXCQvQnL/pQ1ND6tm2r4akGl50qNAXn8E/ cpYyzah+LZAzuM9r1SMmmjYXuLenKp1JA/EKQ= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=too0DqNKz9dPsh10Ysygzua+h8ZCOheEcqQKuq/gdB3b54JC6PYber/tEspVWfrP9U dIs4eQAs/nfcZA6hf0/F79VlEFoNYmHWJoOXkUzSUnKpfDopUCdk/uA1cGAvmtvkNhn6 pZHjN/ofFHJw64s2HkgxJziLmG3Qoj4a2+Las= Received: by 10.151.108.10 with SMTP id k10mr636595ybm.6.1215100521747; Thu, 03 Jul 2008 08:55:21 -0700 (PDT) Received: by 10.151.101.9 with HTTP; Thu, 3 Jul 2008 08:55:21 -0700 (PDT) Message-ID: Date: Thu, 3 Jul 2008 08:55:21 -0700 From: "Kian Mohageri" To: stef@memberwebs.com In-Reply-To: <20080703003955.859BCF180C0@mx.npubs.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 Content-Disposition: inline References: <678A03F5-5E8A-4CF6-90DF-AA9A4F30FBE1@stromnet.se> <1211037564.6326.27.camel@porksoda> <679DB462-75D6-45CC-949C-1BE8E12C22CD@stromnet.se> <482FD877.6050707@infracaninophile.co.uk> <20080703003955.859BCF180C0@mx.npubs.com> Cc: freebsd-stable , freebsd-net@freebsd.org, =?UTF-8?Q?Johan_Str=C3=B6m?= , Matthew Seaman , freebsd-pf@freebsd.org, Alex Trull Subject: Re: connect(): Operation not permitted X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 16:20:41 -0000 T24gV2VkLCBKdWwgMiwgMjAwOCBhdCA1OjM5IFBNLCBTdGVmIDxzdGVmLWxpc3RAbWVtYmVyd2Vi cy5jb20+IHdyb3RlOgo+IEtpYW4gTW9oYWdlcmkgd3JvdGU6Cj4+IE9uIFN1biwgTWF5IDE4LCAy MDA4IGF0IDM6MzMgQU0sIEpvaGFuIFN0csO2bSA8am9oYW5Ac3Ryb21uZXQuc2U+IHdyb3RlOgo+ Pj4gT24gTWF5IDE4LCAyMDA4LCBhdCA5OjE5IEFNLCBNYXR0aGV3IFNlYW1hbiB3cm90ZToKPj4+ Cj4+Pj4gSm9oYW4gU3Ryw7ZtIHdyb3RlOgo+Pj4+Cj4+Pj4+IGRyb3AgYWxsIHRyYWZmaWMpPyBB IGNoZWNrIHdpdGggcGZjdGwgLXZzciByZXZlYWxzIHRoYXQgdGhlIGFjdHVhbCBydWxlCj4+Pj4+ IGluc2VydGVkIGlzICJwYXNzIG9uIGxvMCBpbmV0IGZyb20gMTIzLjEyMy4xMjMuMTIzIHRvIDEy My4xMjMuMTIzLjEyMyBmbGFncwo+Pj4+PiBTL1NBIGtlZXAgc3RhdGUiLiBXaGVyZSBkaWQgdGhh dCAia2VlcCBzdGF0ZSIgY29tZSBmcm9tPwo+Pj4+ICdmbGFncyBTL1NBIGtlZXAgc3RhdGUnIGlz IHRoZSBkZWZhdWx0IG5vdyBmb3IgdGNwIGZpbHRlciBydWxlcyAtLSB0aGF0Cj4+Pj4gd2FzIG5l dyBpbiA3LjAgcmVmbGVjdGluZyB0aGUgdXBzdHJlYW0gY2hhbmdlcyBtYWRlIGJldHdlZW4gdGhl IDQuMCBhbmQKPj4+PiA0LjEKPj4+PiByZWxlYXNlcyBvZiBPcGVuQlNELiAgSWYgeW91IHdhbnQg YSBzdGF0ZWxlc3MgcnVsZSwgYXBwZW5kICdubyBzdGF0ZScuCj4+Pj4KPj4+PiBodHRwOi8vd3d3 Lm9wZW5ic2Qub3JnL2ZhcS9wZi9maWx0ZXIuaHRtbCNzdGF0ZQo+Pj4gVGhhbmtzISBJIHdhcyBh Y3R1YWxseSBsb29raW5nIGFyb3VuZCBpbiB0aGUgcGYuY29uZiBtYW5wYWdlIGJ1dCBmYWlsZWQg dG8KPj4+IGZpbmQgaXQgeWVzdGVyZGF5LCBidXQgbG9va2luZyBjbG9zZXIgdG9kYXkgSSBub3cg c2F3IGl0Lgo+Pj4gQXBwbGllZCB0aGUgbm8gc3RhdGUgKGFuZCBxdWljaykgdG8gdGhlIHJ1bGUs IGFuZCBub3cgbm8gc3RhdGUgaXMgY3JlYXRlZC4KPj4+IEFuZCB0aGUgcHJvYmxlbSBJIGhhZCBp biB0aGUgZmlyc3QgcGxhY2Ugc2VlbXMgdG8gaGF2ZSBiZWVuIHJlc29sdmVkIHRvbwo+Pj4gbm93 LCBldmVuIHRob3VnaCBpdCBkaWRuJ3QgbG9vayBsaWtlIGEgc3RhdGUgcHJvYmxlbS4uIChzdGFy dGVkIHRvIGRlbnkgbmV3Cj4+PiBjb25uZWN0aW9ucyBtdWNoIGVhcmxpZXIgdGhhbiB0aGUgc3Rh dGVzIHdhcyBmdWxsLCBhbHRvdWdoIG1heWJlZSBpIHdhc250Cj4+PiBsb29raW5nIGZvciB1cGRh dGVzIGZhc3QgZW5vdWdoIG9yIHNvbWV0aGluZykuCj4+Pgo+Pgo+PiBJJ2QgYmUgd2lsbGluZyB0 byBiZXQgaXQncyBiZWNhdXNlIHlvdSdyZSByZXVzaW5nIHRoZSBzb3VyY2UgcG9ydCBvbiBhCj4+ IG5ldyBjb25uZWN0aW9uIGJlZm9yZSB0aGUgb2xkIHN0YXRlIGV4cGlyZXMuCj4+Cj4+IFlvdSds bCBrbm93IGlmIHlvdSBjaGVjayB0aGUgc3RhdGUtbWlzbWF0Y2ggY291bnRlci4KPj4KPj4gQW55 d2F5LCBnbGFkIHlvdSBmb3VuZCBhIHJlc29sdXRpb24uCj4KPiBJJ3ZlIGJlZW4gZXhwZXJpZW5j aW5nIHRoaXMgIk9wZXJhdGlvbiBub3QgcGVybWl0dGVkIiB0b28uIEkndmUgYmVlbgo+IHRyeWlu ZyB0byB0cmFjayBkb3duIHRoZSBwcm9ibGVtIGZvciBtYW55IG1vbnRocywgYnV0IGR1ZSB0byB0 aGUKPiBjb21wbGV4aXR5IG9mIG15IGZpcmV3YWxscyAoc2NvcmVzIG9mIGphaWxzIGVhY2ggd2l0 aCBzY29yZXMgb2YgcnVsZXMpLAo+IEkgd2Fzbid0IGJyYXZlIGVub3VnaCB0byBhc2sgZm9yIGhl bHAgOikKPgo+IEFzIGEgd29yayBhcm91bmQgd2Ugc3RhcnRlZCBjcmVhdGluZyBydWxlcyB3aXRo b3V0IHN0YXRlLCB3aGVuZXZlciB3ZQo+IHdvdWxkIHJ1biBpbnRvIHRoZSBwcm9ibGVtLgo+Cj4g VGhhbmtzIGZvciB0aGUgcG9pbnRlciBhYm91dCBzdGF0ZS1taXNtYXRjaC4gVGhlIHN0YXRlLW1p c21hdGNoIGNvdW50ZXIKPiBkb2VzIGlzIGluIGZhY3QgaGlnaCBpbiBteSBjYXNlIChzZWUgYmVs b3cpLiBIb3cgd291bGQgSSBnbyBhYm91dAo+IGdldHRpbmcgdGhlIHBmIHN0YXRlIHRpbWVvdXQg YW5kIHRoZSByZXVzZSBvZiBwb3J0cyBmb3Igb3V0Ym91bmQKPiBjb25uZWN0aW9ucyB0byBtYXRj aD8gT3IgaXMgdGhpcyBhbiBpbnRyYWN0YWJsZSBwcm9ibGVtLCB0aGF0IGp1c3QgbmVlZHMKPiB0 byBiZSB3b3JrZWQgYXJvdW5kPwo+CgpNYWtlIHN1cmUgeW91ciBzdGF0ZS1taXNtYXRjaCBjb3Vu dGVyIGlzIGluY3JlYXNpbmcgYXQgdGhlIHNhbWUgdGltZXMKeW91IGV4cGVyaWVuY2UgdGhlIHBy b2JsZW0gKGFuZCBpc24ndCBqdXN0IGhpZ2ggZnJvbSBzb21lIHVucmVsYXRlZAppc3N1ZSkuCgpB IHNpbWlsYXIvcmVsYXRlZCBwcm9ibGVtIHdhcyBhZGRyZXNzZWQgaW4gT3BlbkJTRCA0LjMKKGh0 dHA6Ly93d3cub3BlbmJzZC5vcmcvcGx1czQzLmh0bWwpLgoKICAqIEluIHBmKDQpLCBhbGxvdyBz dGF0ZSByZXVzZSBpZiBib3RoIHNpZGVzIGFyZSBpbiBGSU5fV0FJVF8yIGFuZCBhCm5ldyBTWU4g YXJyaXZlcy4KCkknbSBub3Qgc3VyZSBpZiBpdCdzIGJlZW4gaW1wb3J0ZWQgeWV0LiAgSWYgbm90 LCB5b3UgY291bGQgdHJ5IHR1bmluZwp5b3VyIHRpbWVvdXQgdmFsdWVzIChzZWUgcGYuY29uZig1 KSkuCgpUaGUgc3BlY2lmaWMgaXNzdWUgSSB3YXMgZXhwZXJpZW5jZWQgd2FzIHNvbHZlZCBieSBz aG9ydGVuaW5nCnRjcC5jbG9zZWQsIElJUkMuICBJdCdzIGJlZW4gYSB3aGlsZSB0aG91Z2guCgot S2lhbgo= From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 16:40:52 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AFFB61065684 for ; Thu, 3 Jul 2008 16:40:52 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: from ibctech.ca (v6.ibctech.ca [IPv6:2607:f118::b6]) by mx1.freebsd.org (Postfix) with SMTP id 547E68FC25 for ; Thu, 3 Jul 2008 16:40:52 +0000 (UTC) (envelope-from steve@ibctech.ca) Received: (qmail 11540 invoked by uid 89); 3 Jul 2008 16:42:46 -0000 Received: from unknown (HELO ?IPv6:2607:f118::5?) (steve@ibctech.ca@2607:f118::5) by 2607:f118::b6 with ESMTPA; 3 Jul 2008 16:42:46 -0000 Message-ID: <486D011A.7080406@ibctech.ca> Date: Thu, 03 Jul 2008 12:40:58 -0400 From: Steve Bertrand User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486C7611.9030905@gtcomm.net> <486CCB6A.6070104@ibctech.ca> In-Reply-To: X-Enigmail-Version: 0.95.6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 16:40:52 -0000 Ingo Flaschberger wrote: > Dear Steve, > >>> My next "router" appliance will be: >>> http://www.axiomtek.com.tw/products/ViewProduct.asp?view=429 >> >> This is exactly the device that I have been testing with (just >> rebranded). > > cool. > what performace do you reach? After some very quick testing with everything default, I am witnessing results that are far below what I would have expected. I have a few questions: - how do I identify if polling on an interface is enabled? I see no difference with ifconfig output - do I need to compile a new kernel to be able to enable/disable polling? - without moving some hardware around, I only have a single box connected to a router, and I've been testing from that box to a different interface within the router. Will the test results be optimal if I ping all the way through the router to a second device connected to it? - how are the results affected when generating and receiving the test packets within the router itself (as opposed to using outside devices)? Steve From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 17:17:04 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 51CD910656D4 for ; Thu, 3 Jul 2008 17:17:04 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id A49B48FC24 for ; Thu, 3 Jul 2008 17:17:02 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 1725 invoked from network); 3 Jul 2008 19:17:01 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 3 Jul 2008 19:17:01 +0200 Date: Thu, 3 Jul 2008 19:17:01 +0200 (CEST) From: Ingo Flaschberger To: Steve Bertrand In-Reply-To: <486D011A.7080406@ibctech.ca> Message-ID: References: <4867420D.7090406@gtcomm.net> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486C7611.9030905@gtcomm.net> <486CCB6A.6070104@ibctech.ca> <486D011A.7080406@ibctech.ca> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net , Paul Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 17:17:04 -0000 Dear Steve, >>>> My next "router" appliance will be: >>>> http://www.axiomtek.com.tw/products/ViewProduct.asp?view=429 >>> >>> This is exactly the device that I have been testing with (just rebranded). >> >> cool. >> what performace do you reach? > > After some very quick testing with everything default, I am witnessing > results that are far below what I would have expected. I have a few > questions: > > - how do I identify if polling on an interface is enabled? I see no > difference with ifconfig output em0: flags=8843 mtu 1500 options=5b <--- ether 00:90:0b:08:d7:90 media: Ethernet autoselect (1000baseTX ) status: active kern.polling.reg_frac=20 kern.polling.user_frac=20 kern.polling.burst_max=512 man polling polling does not help to get more pps, but prevent locks and preserve some %cpu for other tasks (routing daemons,..) > - do I need to compile a new kernel to be able to enable/disable polling? options DEVICE_POLLING you need this in kern-conf. > - without moving some hardware around, I only have a single box connected to > a router, and I've been testing from that box to a different interface within > the router. Will the test results be optimal if I ping all the way through > the router to a second device connected to it? use any other packet generator. linux has one in kernel, and there are moch more. (iperf,...) ping uses a lot of cpu. > - how are the results affected when generating and receiving the test packets > within the router itself (as opposed to using outside devices)? thats no real "pps" forwarding performance over the network cards. Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 19:23:31 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 94DAA1065688 for ; Thu, 3 Jul 2008 19:23:31 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from mail.bitblocks.com (mail.bitblocks.com [64.142.15.60]) by mx1.freebsd.org (Postfix) with ESMTP id 6546A8FC16 for ; Thu, 3 Jul 2008 19:23:31 +0000 (UTC) (envelope-from bakul@bitblocks.com) Received: from bitblocks.com (localhost.bitblocks.com [127.0.0.1]) by mail.bitblocks.com (Postfix) with ESMTP id 5CD5D5B4C; Thu, 3 Jul 2008 12:05:13 -0700 (PDT) To: Peter Jeremy In-reply-to: Your message of "Thu, 03 Jul 2008 21:52:43 +1000." <20080703115243.GR29380@server.vk2pj.dyndns.org> Date: Thu, 03 Jul 2008 12:05:13 -0700 From: Bakul Shah Message-Id: <20080703190513.5CD5D5B4C@mail.bitblocks.com> Cc: freebsd-net@freebsd.org Subject: Re: arplookup x.x.x.x failed: host is not on local network X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 19:23:31 -0000 > Possibly, I'm seeing packet leakage from the switches and that is > confusing FreeBSD - definitely the first packet above should not be > visible. Even if the switch broadcasts on all ports (effectively becoming a hub) that should not cause the symptom you are seeing. If the switch sent arp response to the wrong port, you would've seen this ARP request at least on the sending machine. There is no such packet (for .26) in your tcpdump output. That either means there was no such packet or you've failed to capture it! You said you see the problem with different freebsd versions. Did you boot diff. versions on the same hardware or do you mean different versions are running on diff. hosts? If the latter, specific freebsd versions are not ruled out. You might want to capture many more arp failed messages to see if there is a pattern. Earlier you had wondered if resource exhaustion was to blame. That is ruled out by the arp failed message since the reason indicates the route goes to a gateway. We don't see any ARP request for .26 so this likely means .26 is not the one doing arp lookup (on receiving a request) & the arplookup failed message is on .111, right? We see packets flowing from .26 to .111 but not the other way around. What does netstat -nr look like on .111? If all the clocks are synchronized, you might want to capture tcpdump on *all* the machines! Since syslog timestamp has a granuality of 1 sec, you want to look at packets within a second before and a second after. BTW, your picture is nice but it doesn't jive with anything in the tcpdump output you attached! > Corp Network > 192.168.10.0/24 | 192.168.12.0/24 > +------+-------------+----------| | |----------+-------------+-----+ > .1| .2| .254| | |.254 .3| .4| > +-------+ +-------+ +-------+ +-------+ +-------+ > | | | | | | | | | | > | host1 | | host2 | | NAT | | host3 | | host4 | > | | | | | | | | | | > +-------+ +-------+ +-------+ +-------+ +-------+ > .1| .2| .254| |.254 .3| .4| > +------+-------------+----------| |----------+-------------+-----+ > 192.168.11.0/24 192.168.13.0/24 > > The errors appear to be randomly spread across hosts and subnets. It > does not appear consistently and seems to correlate with load (I am > getting significant numbers at present and the NAT host is routing > about 90Kpps and 100MBps if netstat can be believed). The problem > also shows up on another interior routing host that has visibility > across the internal networks so it isn't related to NAT or directly > related to host load (that host is only seeing about 3.5Kpps - but is > also a much slower host). > > I have managed to capture a tcpdump across the error. syslog reported: > Jul 3 21:28:30 xxxx kernel: arplookup 192.168.169.26 failed: host is not o= > n local network > and the packets for that host during that second are: > 21:28:30.320340 00:0b:cd:d6:66:26 > 00:03:ba:ab:6f:ef, ethertype 802.1Q (0x= > 8100), length 64: vlan 169, p 0, ethertype IPv4, IP (tos 0x0, ttl 64, id 2= > 9304, offset 0, flags [none], length: 28) 192.168.169.26 > 192.168.169.111:= > icmp 8: echo request seq 35079 > 21:28:30.320429 00:d0:b7:20:8f:ee > 00:03:ba:ab:6f:ef, ethertype 802.1Q (0x= > 8100), length 46: vlan 168, p 0, ethertype IPv4, IP (tos 0x0, ttl 63, id 2= > 9304, offset 0, flags [none], length: 28) 192.168.169.26 > 192.168.169.111:= > icmp 8: echo request seq 35079 > 21:28:30.320445 00:0b:cd:d6:66:26 > ff:ff:ff:ff:ff:ff, ethertype 802.1Q (0x= > 8100), length 64: vlan 169, p 0, ethertype ARP, arp who-has 192.168.169.250= > tell 192.168.169.26 > 21:28:30.320459 00:0b:cd:d6:66:26 > 00:d0:b7:20:8f:ee, ethertype 802.1Q (0x= > 8100), length 64: vlan 169, p 0, ethertype IPv4, IP (tos 0x0, ttl 64, id 2= > 9307, offset 0, flags [none], length: 28) 192.168.169.26 > 192.168.169.250:= > icmp 8: echo request seq 35079 > 21:28:30.320493 00:d0:b7:20:8f:ee > 00:0b:cd:d6:66:e4, ethertype 802.1Q (0x= > 8100), length 46: vlan 168, p 0, ethertype IPv4, IP (tos 0x0, ttl 64, id 1= > 5305, offset 0, flags [none], length: 28) 192.168.169.250 > 192.168.169.26:= > icmp 8: echo reply seq 35079 > 21:28:30.320531 00:d0:b7:20:8f:ee > 00:0b:cd:d6:66:26, ethertype 802.1Q (0x= > 8100), length 46: vlan 169, p 0, ethertype ARP, arp reply 192.168.169.250 i= > s-at 00:d0:b7:20:8f:ee > (this was captured MAC 00:d0:b7:20:8f:ee). From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 19:40:36 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 36A76106568A for ; Thu, 3 Jul 2008 19:40:36 +0000 (UTC) (envelope-from zaphod@fsklaw.com) Received: from thor-new.fsklaw.com (thor-new.fsklaw.com [64.174.116.34]) by mx1.freebsd.org (Postfix) with ESMTP id 1AC598FC14 for ; Thu, 3 Jul 2008 19:40:35 +0000 (UTC) (envelope-from zaphod@fsklaw.com) Received: from localhost (localhost [127.0.0.1]) by thor-new.fsklaw.com (Postfix) with ESMTP id 50D5B166307C for ; Thu, 3 Jul 2008 12:15:58 -0700 (PDT) Received: from thor-new.fsklaw.com ([127.0.0.1]) by localhost (localhost [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 11160-06 for ; Thu, 3 Jul 2008 12:15:54 -0700 (PDT) Received: from cor (unknown [192.168.61.119]) by thor-new.fsklaw.com (Postfix) with ESMTP id DA1071663055 for ; Thu, 3 Jul 2008 12:15:54 -0700 (PDT) Received: from 192.168.62.153 (SquirrelMail authenticated user zaphod) by cor with HTTP; Thu, 3 Jul 2008 12:15:09 -0700 (PDT) Message-ID: <8f7879db41dbaecc479a017110e8f32f.squirrel@cor> Date: Thu, 3 Jul 2008 12:15:09 -0700 (PDT) From: zaphod@fsklaw.com To: freebsd-net@freebsd.org User-Agent: SquirrelMail/1.4.15 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal X-Virus-Scanned: by amavisd-new at fsklaw.com Subject: Tunneling issues X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 19:40:36 -0000 I have a real poser, and I ccan't solve it. Currently I have a ipsec vpn tunneling 14 servers through a central server. Like this: ________________ | | |_______________| | | _________________ | | |________________| | | _________________ | | |________________| I would like to restructure this so that each server talks to each other directly, rather than passing everything through a single server. However, on every other machine I cannot get a second tunnel to come up. Not a gre or gif tunnel. And yet I have 14 on the central machine. The central machine is FreeBSD5.3, the rest are 6.1 or greater. I also fear that I won't be able to update the central server, because I fear not being able to get the tunnels up. I have been just trying to tunnel. IPSEC isn't the issue as I'm not binding an ipsec policy to the tunnel. I've been googling for days, and can't find anything on this. (Can't find anyone creating more than one tunnel). Any ideas would be appreciated as I'm totally stumped here. TIA Cheers, Zaphod From owner-freebsd-net@FreeBSD.ORG Thu Jul 3 20:23:04 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B35871065680 for ; Thu, 3 Jul 2008 20:23:04 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 6C0028FC1A for ; Thu, 3 Jul 2008 20:23:04 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KEVGx-0000yI-Ow; Thu, 03 Jul 2008 16:19:20 -0400 Message-ID: <486D35A0.4000302@gtcomm.net> Date: Thu, 03 Jul 2008 16:25:04 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Bruce Evans References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> <486C7F93.7010308@gtcomm.net> <20080703195521.O6973@delplex.bde. org> In-Reply-To: <20080703195521.O6973@delplex.bde.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Ingo Flaschberger Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Jul 2008 20:23:04 -0000 Opteron 2222 UP mode, no polling input (em0) output packets errs bytes packets errs bytes colls 1071020 0 66403248 2 0 404 0 1049793 0 65087174 2 0 356 0 1040320 0 64499848 2 0 356 0 1049712 0 65082152 2 0 356 0 1039504 0 64449256 2 0 356 0 933118 0 57853324 2 0 356 0 still has some cpu left and i can't generate any more packets Polling turned on provided better performance on 32 bit, but it gets strange errors on 64 bit.. Even at low pps I get small amounts of errors, and high pps same thing.. you would think that if it got errors at low pps it would get more errors at high pps but that isn't the case.. Polling on: packets errs bytes packets errs bytes colls 979736 963 60743636 1 0 226 0 991838 496 61493960 1 0 178 0 996125 460 61759754 1 0 178 0 979381 326 60721626 1 0 178 0 1022249 379 63379442 1 0 178 0 991468 557 61471020 1 0 178 0 lowering pps a little....... input (em0) output packets errs bytes packets errs bytes colls 818688 151 50758660 1 0 226 0 837920 179 51951044 1 0 178 0 826217 168 51225458 1 0 178 0 801017 100 49663058 1 0 178 0 761857 287 47235138 1 0 178 0 what could cause this? If i'm going to use a uniprocessor mode system I NEED polling to work because I have to have cpu cycles left over for userspace processes and I can't afford to have it lock those out. SMP is no big deal if it actually worked.. I'm going to do a SMP test with this cpu now with polling off/on and then I'm going to apply the polling patch and try that. Bruce Evans wrote: > On Thu, 3 Jul 2008, Paul wrote: > >> Bruce Evans wrote: >>>> No polling: >>>> 843762 25337 52313248 1 0 178 0 >>>> 763555 0 47340414 1 0 178 0 >>>> 830189 0 51471722 1 0 178 0 >>>> 838724 0 52000892 1 0 178 0 >>>> 813594 939 50442832 1 0 178 0 >>>> 807303 763 50052790 1 0 178 0 >>>> 791024 0 49043492 1 0 178 0 >>>> 768316 1106 47635596 1 0 178 0 >>>> Machine is maxed and is unresponsive.. >>> >>> That's the most interesting one. Even 1% packet loss would probably >>> destroy performance, so the benchmarks that give 10-50% packet loss >>> are uninteresting. >>> >> But you realize that it's outputting all of these packets on em3 and >> I'm watching them coming out >> and they are consistent with the packets received on em0 that netstat >> shows are 'good' packets. > > Well, output is easier. I don't remember seeing the load on a taskq for > em3. If there is a memory bottleneck, it might to might not be more > related > to running only 1 taskq per interrupt, depending on how independent the > memory system is for different CPU. I think Opterons have more > indenpendence > here than most x86's. > >> I'm using a server opteron which supposedly has the best memory >> performance out of any CPU right now. >> Plus opterons have the biggest l1 cache, but small l2 cache. Do you >> think larger l2 cache on the Xeon (6mb for 2 core) would be better? >> I have a 2222 opteron coming which is 1ghz faster so we will see what >> happens > > I suspect lower latency memory would help more. Big memory systems > have inherently higher latency. My little old A64 workstation and > laptop have main memory latencies 3 times smaller than freebsd.org's > new Core2 servers according to lmbench2 (42 nsec for the overclocked > DDR PC3200 one and 55 for the DDR2 PC5400 (?) one, vs 145-155 nsec). > If there are a lot of cache misses, then the extra 100 nsec can be > important. Profiling of sendto() using hwpmc or perfmon shows a > significant number of cache misses per packet (2 or 10?). > >>>> Polling ON: >>>> input (em0) output >>>> packets errs bytes packets errs bytes colls >>>> 784138 179079 48616564 1 0 226 0 >>>> 788815 129608 48906530 2 0 356 0 >>>> Machine is responsive and has 40% idle cpu.. Why ALWAYS 40% ? I'm >>>> really mistified by this.. >>> >>> Is this with hz=2000 and 256/256 and no polling in idle? 40% is easy >>> to explain (perhaps incorrectly). Polling can then read at most 256 >>> descriptors every 1/2000 second, giving a max throughput of 512 kpps. >>> Packets < descriptors in general but might be equal here (for small >>> packets). You seem to actually get 784 kpps, which is too high even >>> in descriptors unless, but matches exactly if the errors are counted >>> twice (784 - 179 - 505 ~= 512). CPU is getting short too, but 40% >>> still happens to be left over after giving up at 512 kpps. Most of >>> the errors are probably handled by the hardware at low cost in CPU by >>> dropping packets. There are other types of errors but none except >>> dropped packets is likely. >>> >> Read above, it's actually transmitting 770kpps out of em3 so it can't >> just be 512kpps. > > Transmitting is easier, but with polling its even harder to send > faster than > hz * queue_length than to receive. This is without polling in idle. > >> I was thinking of trying 4 or 5.. but how would that work with this >> new hardware? > > Poorly, except possibly with polling in FreeBSD-4. FreeBSD-4 generally > has lower overheads and latency, but is missing important improvements > (mainly tcp optimizations in upper layers, better DMA and/or mbuf > handling, and support for newer NICs). FreeBSD-5 is also missing the > overhead+latency advantage. > > Here are some benchmarks. (ttcp mainly tests sendto(). 4.10 em needed a > 2-line change to support a not-so-new PCI em NIC. Summary: > - my bge NIC can handle about 600 kpps on my faster machine, but only > achieves 300 in 4.10 unpatched. > - my em NIC can handle about 400 kpps on my slower machine, except in > later versions it can receive at about 600 kpps. > - only 6.x and later can achieve near wire throughput for 1500-MTU > packets (81 kpps vs 76 kpps). This depends on better DMA or mbuf > handling... I now remember the details -- it is mainly better mbuf > handling: old versions split the 1500-MTU packets into 2 mbufs and > this causes 2 descriptors per packet, which causes extra software > overheads and even larger overheads for the hardware. > > %%% > Results of benchmarks run on 23 Feb 2007: > > my~5.2 bge --> ~4.10 em > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t 639 98 1660 398* 77 8k > ttcp -l5 -t 6.0 100 3960 6.0 6 5900 > ttcp -l1472 -u -t 76 27 395 76 40 8k > ttcp -l1472 -t 51 40 11k 51 26 8k > > (*) Same as sender according to netstat -I, but systat -ip shows that > almost half aren't delivered to upper layers. > > my~5.2 bge --> 4.11 em > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t 635 98 1650 399* 74 8k > ttcp -l5 -t 5.8 100 3900 5.8 6 5800 > ttcp -l1472 -u -t 76 27 395 76 32 8k > ttcp -l1472 -t 51 40 11k 51 25 8k > > (*) Same as sender according to netstat -I, but systat -ip shows that > almost half aren't delivered to upper layers. > > my~5.2 bge --> my~5.2 em > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t 638 98 1660 394* 100- 8k > ttcp -l5 -t 5.8 100 3900 5.8 9 6000 > ttcp -l1472 -u -t 76 27 395 76 46 8k > ttcp -l1472 -t 51 40 11k 51 35 8k > > (*) Same as sender according to netstat -I, but systat -ip shows that > almost half aren't delivered to upper layers. With the em rate > limit on ips changed from 8k to 80k, about 95% are delivered up. > > my~5.2 bge --> 6.2 em > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t 637 98 1660 637 100- 15k > ttcp -l5 -t 5.8 100 3900 5.8 8 12k > ttcp -l1472 -u -t 76 27 395 76 36 16k > ttcp -l1472 -t 51 40 11k 51 37 16k > > my~5.2 bge --> ~current em-fastintr > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t 641 98 1670 641 99 8k > ttcp -l5 -t 5.9 100 2670 5.9 7 6k > ttcp -l1472 -u -t 76 27 395 76 35 8k > ttcp -l1472 -t 52 43 11k 52 30 8k > > ~6.2 bge --> ~current em-fastintr > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t 309 62 1600 309 64 8k > ttcp -l5 -t 4.9 100 3000 4.9 6 7k > ttcp -l1472 -u -t 76 27 395 76 34 8k > ttcp -l1472 -t 54 28 6800 54 30 8k > > ~current bge --> ~current em-fastintr > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t 602 100 1570 602 99 8k > ttcp -l5 -t 5.3 100 2660 5.3 5 5300 > ttcp -l1472 -u -t 81# 19 212 81# 38 8k > ttcp -l1472 -t 53 34 11k 53 30 8k > > (#) Wire speed to within 0.5%. This is the only kppps in this set of > benchmarks that is close to wire speed. Older kernels apparently > lose relative to -current because mbufs for mtu-sized packets are > not contiguous in older kernels. > > Old results: > > ~4.10 bge --> my~5.2 em > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t n/a n/a n/a 346 79 8k > ttcp -l5 -t n/a n/a n/a 5.4 10 6800 > ttcp -l1472 -u -t n/a n/a n/a 67 40 8k > ttcp -l1472 -t n/a n/a n/a 51 36 8k > > ~4.10 kernel, =4 bge --> ~current em > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t n/a n/a n/a 347 96 14k > ttcp -l5 -t n/a n/a n/a 5.8 10 14k > ttcp -l1472 -u -t n/a n/a n/a 67 62 14K > ttcp -l1472 -t n/a n/a n/a 52 40 16k > > ~4.10 kernel, =4+ bge --> ~current em > tx rx > kpps load% ips kpps load% ips > ttcp -l5 -u -t n/a n/a n/a 627 100 9k > ttcp -l5 -t n/a n/a n/a 5.6 9 13k > ttcp -l1472 -u -t n/a n/a n/a 68 63 14k > ttcp -l1472 -t n/a n/a n/a 54 44 16k > %%% > > %%% > Results of benchmarks run on 28 Dec 2007: > > ~5.2 epsplex (em) ttcp: > Csw Trp Sys Int Sof Sys Intr User Idle > local no sink: 825k 3 206k 229 412k 52.1 45.1 2.8 > local with sink: 659k 3 263k 231 131k 66.5 27.3 6.2 > tx remote no sink: 35k 3 273k 8237 266k 42.0 52.1 2.3 3.6 > tx remote with sink: 26k 3 394k 8224 100 60.0 5.41 3.4 11.2 > rx remote no sink: 25k 4 26 8237 373k 20.6 79.4 0.0 0.0 > rx remote with sink: 30k 3 203k 8237 398k 36.5 60.7 2.8 0.0 > > 6.3-PR besplex (em) ttcp: > Csw Trp Sys Int Sof Sys Intr User Idle > local no sink: 417k 1 208k 418k 2 49.5 48.5 2.0 > local with sink: 420k 1 276k 145k 2 70.0 23.6 6.4 > tx remote no sink: 19k 2 250k 8144 2 58.5 38.7 2.8 0.0 > tx remote with sink: 16k 2 361k 8336 2 72.9 24.0 3.1 4.4 > rx remote no sink: 429 3 49 888 2 0.3 99.33 0.0 0.4 > tx remote with sink: 13k 2 316k 5385 2 31.7 63.8 3.6 0.8 > > 8.0-C epsplex (em-fast) ttcp: > Csw Trp Sys Int Sof Sys Intr User Idle > local no sink: 442k 3 221k 230 442k 47.2 49.6 2.7 > local with sink: 394k 3 262k 228 131k 72.1 22.6 5.3 > tx remote no sink: 17k 3 226k 7832 100 94.1 0.2 3.0 0.0 > tx remote with sink: 17k 3 360k 7962 100 91.7 0.2 3.7 4.4 > rx remote no sink: saturated -- cannot update systat display > rx remote with sink: 15k 6 358k 8224 100 97.0 0.0 2.5 0.5 > > ~4.10 besplex (bge) ttcp: > Csw Trp Sys Int Sof Sys Intr User Idle > local no sink: 15 0 425k 228 11 96.3 0.0 3.7 > local with sink: ** 0 622k 229 ** 94.7 0.3 5.0 > tx remote no sink: 29 1 490k 7024 11 47.9 29.8 4.4 17.9 > tx remote with sink: 26 1 635k 1883 11 65.7 11.4 5.6 17.3 > rx remote no sink: 5 1 68 7025 1 0.0 47.3 0.0 52.7 > rx remote with sink: 6679 2 365k 6899 12 19.7 29.2 2.5 48.7 > > ~5.2-C besplex (bge) ttcp: > Csw Trp Sys Int Sof Sys Intr User Idle > local no sink: 1M 3 271k 229 543k 50.7 46.8 2.5 > local with sink: 1M 3 406k 229 203k 67.4 28.2 4.4 > tx remote no sink: 49k 3 474k 11k 167k 52.3 42.7 5.0 0.0 > tx remote with sink: 6371 3 641k 1900 100 76.0 16.8 6.2 0.9 > rx remote no sink: 34k 3 25 11k 270k 0.8 65.4 0.0 33.8 > rx remote with sink: 41k 3 365k 10k 370k 31.5 47.1 2.3 19.0 > > 6.3-PR besplex (bge) ttcp (hz = 1000 else stathz broken): > Csw Trp Sys Int Sof Sys Intr User Idle > local no sink: 540k 0 270k 540k 0 50.5 46.0 3.5 > local with sink: 628k 0 417k 210k 0 68.8 27.9 3.3 > tx remote no sink: 15k 1 222k 7190 1 28.4 29.3 1.7 40.6 > tx remote with sink: 5947 1 315k 2825 1 39.9 14.7 2.6 42.8 > rx remote no sink: 13k 1 23 6943 0 0.3 49.5 0.2 50.0 > rx remote with sink: 20k 1 371k 6819 0 29.5 30.1 3.9 36.5 > > 8.0-C besplex (bge) ttcp: > Csw Trp Sys Int Sof Sys Intr User Idle > local no sink: 649k 3 324k 100 649k 53.9 42.9 3.2 > local with sink: 649k 3 433k 100 216k 75.2 18.8 6.0 > tx remote no sink: 24k 3 432k 10k 100 49.7 41.3 2.4 6.6 > tx remote with sink: 3199 3 568k 1580 100 64.3 19.6 4.0 12.2 > rx remote no sink: 20k 3 27 10k 100 0.0 46.1 0.0 53.9 > rx remote with sink: 31k 3 370k 10k 100 30.7 30.9 4.8 33.5 > %%% > > Bruce > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 00:30:19 2008 Return-Path: Delivered-To: net@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CF4D2106566C; Fri, 4 Jul 2008 00:30:19 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 4A03B8FC20; Fri, 4 Jul 2008 00:30:19 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id A7DAF46C1A; Thu, 3 Jul 2008 20:30:18 -0400 (EDT) Date: Fri, 4 Jul 2008 01:30:18 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: arch@FreeBSD.org, net@FreeBSD.org In-Reply-To: <20080526102345.G26343@fledge.watson.org> Message-ID: <20080704012901.U90881@fledge.watson.org> References: <20080526102345.G26343@fledge.watson.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Subject: Re: Remaining non-MPSAFE netisr handlers X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 00:30:19 -0000 On Mon, 26 May 2008, Robert Watson wrote: > In the continuing campaign to eliminate the Giant lock from the dregs of the > network stack, I thought I'd send out a list of non-MPSAFE netisr handlers: > > Location Handler Removed with IFF_NEEDSGIANT > dev/usb/usb_ethersubr.c:120 usbintr Yes > net/if_ppp.c:277 pppintr Yes > netinet6/ip6_input.c ip6_input No > > The plan for 8.0 is to remove the NETISR_MPSAFE flag -- all netisr handlers > will be executed without the Giant lock. This doesn't prohibit acquiring > Giant in the handler if required, although that's undesirable for the > obvious reasons (potentially stalling interrupt handling, etc). Obviously, > what would be most desirable is eliminating the remaining requirement for > Giant in the IPv6 input path, primarily consisting of mld6 and nd6. > > With this in mind, my current plan is to remove the flag and add explicit > Giant acquisition for any remaining handlers in June when IFF_NEEDSGIANT > device drivers are disabled. I've now removed the NETISR_MPSAFE flag -- all netisr handlers are now assumed to DTRT with respect to locking. At least until usb and ppp are sorted out, I've introduced NETISR_FORCEQUEUE as an interim measure, which allows protocols to request that they always operate the deferred dispatch, meaning they can acquire Giant if they need to, and modified those two to do so. That should go away by 8.0 also. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 01:55:50 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D90481065675 for ; Fri, 4 Jul 2008 01:55:50 +0000 (UTC) (envelope-from mike@sentex.net) Received: from smarthost1.sentex.ca (smarthost1.sentex.ca [64.7.153.18]) by mx1.freebsd.org (Postfix) with ESMTP id A353A8FC0A for ; Fri, 4 Jul 2008 01:55:50 +0000 (UTC) (envelope-from mike@sentex.net) Received: from lava.sentex.ca (pyroxene.sentex.ca [199.212.134.18]) by smarthost1.sentex.ca (8.14.2/8.14.2) with ESMTP id m641tmAq090072; Thu, 3 Jul 2008 21:55:48 -0400 (EDT) (envelope-from mike@sentex.net) Received: from mdt-xp.sentex.net (simeon.sentex.ca [192.168.43.27]) by lava.sentex.ca (8.13.8/8.13.3) with ESMTP id m641tl8s000607 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 3 Jul 2008 21:55:47 -0400 (EDT) (envelope-from mike@sentex.net) Message-Id: <200807040155.m641tl8s000607@lava.sentex.ca> X-Mailer: QUALCOMM Windows Eudora Version 7.1.0.9 Date: Thu, 03 Jul 2008 21:55:41 -0400 To: zaphod@fsklaw.com, freebsd-net@freebsd.org From: Mike Tancsa In-Reply-To: <8f7879db41dbaecc479a017110e8f32f.squirrel@cor> References: <8f7879db41dbaecc479a017110e8f32f.squirrel@cor> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed X-Scanned-By: MIMEDefang 2.64 on 64.7.153.18 Cc: Subject: Re: Tunneling issues X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 01:55:50 -0000 At 03:15 PM 7/3/2008, zaphod@fsklaw.com wrote: >I have a real poser, and I ccan't solve it. > >Currently I have a ipsec vpn tunneling 14 servers through a central server. > >I would like to restructure this so that each server talks to each other >directly, rather than passing everything through a single server. > >However, on every other machine I cannot get a second tunnel to come up. >Not a gre or gif tunnel. And yet I have 14 on the central machine. You would need a lot of policies on each of the boxes (14) but there is no reason it should not work. Do each of the sites have a unique subnet ? Do they have static IP addresses ? An easier solution might be to use something like OpenVPN which allows all the boxes to auth and route through a single server, but they can also talk to each other with a single config option. ---Mike From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 02:32:46 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E6CD7106567E for ; Fri, 4 Jul 2008 02:32:46 +0000 (UTC) (envelope-from ddesimone@verio.net) Received: from relay2-bcrtfl2.verio.net (relay2-bcrtfl2.verio.net [131.103.218.177]) by mx1.freebsd.org (Postfix) with ESMTP id 9124A8FC0A for ; Fri, 4 Jul 2008 02:32:46 +0000 (UTC) (envelope-from ddesimone@verio.net) Received: from iad-wprd-xchw01.corp.verio.net (iad-wprd-xchw01.corp.verio.net [198.87.7.164]) by relay2-bcrtfl2.verio.net (Postfix) with ESMTP id 47BA91FF0184 for ; Thu, 3 Jul 2008 22:32:45 -0400 (EDT) thread-index: Acjdfj2LNyxq4Q0bQtyNHkJOtaHBZQ== Received: from limbo.int.dllstx01.us.it.verio.net ([10.10.10.11]) by iad-wprd-xchw01.corp.verio.net with Microsoft SMTPSVC(6.0.3790.1830); Thu, 3 Jul 2008 22:32:44 -0400 Received: by limbo.int.dllstx01.us.it.verio.net (Postfix, from userid 1000) id 6D94C8E29B; Thu, 3 Jul 2008 21:32:44 -0500 (CDT) Date: Thu, 3 Jul 2008 21:32:44 -0500 From: "David DeSimone" Content-Transfer-Encoding: 7bit To: Message-ID: <20080704023244.GH29305@verio.net> Mail-Followup-To: freebsd-net@freebsd.org Content-Class: urn:content-classes:message Importance: normal References: <20080703115243.GR29380@server.vk2pj.dyndns.org> <20080703190513.5CD5D5B4C@mail.bitblocks.com> Priority: normal X-MimeOLE: Produced By Microsoft MimeOLE V6.00.3790.2992 MIME-Version: 1.0 Content-Type: text/plain; x-action=pgp-signed; charset="us-ascii" Content-Disposition: inline In-Reply-To: <20080703190513.5CD5D5B4C@mail.bitblocks.com> Precedence: bulk User-Agent: Mutt/1.5.9i X-OriginalArrivalTime: 04 Jul 2008 02:32:44.0742 (UTC) FILETIME=[3D7F2660:01C8DD7E] Subject: Re: arplookup x.x.x.x failed: host is not on local network X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 02:32:47 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Bakul Shah wrote: > > There is no such packet (for .26) in your tcpdump output. I spotted this: > > 21:28:30.320445 00:0b:cd:d6:66:26 > ff:ff:ff:ff:ff:ff, ethertype > > 802.1Q (0x8100), length 64: vlan 169, p 0, ethertype ARP, arp > > who-has 192.168.169.250 tell 192.168.169.26 which matches with: > > Jul 3 21:28:30 xxxx kernel: arplookup 192.168.169.26 failed: host > > is not on local network So BSD is complaining about the IP of the source, not the host being looked up. Again, I did see these messages in my environment, but in my case, the error was correct: The IP *was not* on the local network. The reason being that we had multiple subnets configured on the same broadcast domain, so the BSD box could indeed hear ARP for subnets it did not know about. I don't know why the box feels moved to complain about this, however. I would think it should not care. In this case, however, the user claims that the box is indeed a member of the 192.168.169 subnet, and therefore it should not be complaining. - -- David DeSimone == Network Admin == fox@verio.net "This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, dis- tribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that this email is error or virus free. Thank you." --Lawyer Bot 6000 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) iD8DBQFIbYvMFSrKRjX5eCoRAr87AJ9Sr0CBdOazW4SYu3uRbylu1bwz4wCghEcT VvpeTB5KK57fgxuSViz6tb0= =XEIH -----END PGP SIGNATURE----- This email message is intended for the use of the person to whom it has been sent, and may contain information that is confidential or legally protected. If you are not the intended recipient or have received this message in error, you are not authorized to copy, distribute, or otherwise use this message or its attachments. Please notify the sender immediately by return e-mail and permanently delete this message and any attachments. Verio, Inc. makes no warranty that this email is error or virus free. Thank you. From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 02:57:45 2008 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 051EE1065673; Fri, 4 Jul 2008 02:57:45 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id CED758FC1C; Fri, 4 Jul 2008 02:57:44 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m642viTB085949; Fri, 4 Jul 2008 02:57:44 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m642viGB085945; Fri, 4 Jul 2008 02:57:44 GMT (envelope-from linimon) Date: Fri, 4 Jul 2008 02:57:44 GMT Message-Id: <200807040257.m642viGB085945@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-net@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/125239: [gre] kernel crash when using gre X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 02:57:45 -0000 Synopsis: [gre] kernel crash when using gre Responsible-Changed-From-To: freebsd-bugs->freebsd-net Responsible-Changed-By: linimon Responsible-Changed-When: Fri Jul 4 02:57:25 UTC 2008 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=125239 From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 03:27:18 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 370E11065670 for ; Fri, 4 Jul 2008 03:27:18 +0000 (UTC) (envelope-from gcshekhar@sbcglobal.net) Received: from web81002.mail.mud.yahoo.com (web81002.mail.mud.yahoo.com [68.142.199.82]) by mx1.freebsd.org (Postfix) with SMTP id F0C6C8FC1C for ; Fri, 4 Jul 2008 03:27:17 +0000 (UTC) (envelope-from gcshekhar@sbcglobal.net) Received: (qmail 55190 invoked by uid 60001); 4 Jul 2008 03:00:37 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=sbcglobal.net; h=Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID; b=MrlQ4oncn9KHKnjm64Vi8DX+njdIPgq/Xs/hhPfNCp9SCYpxGUEf3+zHQES+3qH3mtzarkqNs+L5mtohxPvvxiTN3+uAyocf4fcy0JzWb9mQhWEi7zWybJACV+1bHH+ow+Cw6T6ue+axqRdrdUHe1NyAiCF333+1tr66g4tUfZQ=; Received: from [65.113.40.1] by web81002.mail.mud.yahoo.com via HTTP; Thu, 03 Jul 2008 20:00:37 PDT X-Mailer: YahooMailRC/975.45 YahooMailWebService/0.7.199 Date: Thu, 3 Jul 2008 20:00:37 -0700 (PDT) From: Shekhar Chandrashekhar To: freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <370399.55106.qm@web81002.mail.mud.yahoo.com> Subject: Incorrect ipv6 prefix detaching behavior? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 03:27:18 -0000 I'm running into an issue where nd6_rtr.c:pfxlist_onlink_check() is possibly not doing the right thing by marking a prefix as not ONLINK - I've noticed this behavior in both FreeBSD 6 and 7. I have an interface (say fxp0) which has an router-advertised address for access outside the local subnet (fc00:10:1:2::/64 prefix) and a static address (fc00:10:1:1::/64) for connecting to servers on the same subnet. The def router only advertises the fc00:10:1:2:: prefix. However, as you see from the "flags=LD" below, freebsd seems to mark the fc00:10:1:1::/64 prefix as detached and always forwards to the def router even for a dest in the fc00:10:1:1:: subnet. # ndp -p ;# abbreviated for interesting prefixes fc00:10:1:2::/64 if=fxp0 flags=LAO vltime=2592000, pltime-604800, expire=29d23h59m50s, ref=1 advertised by fe80::214:f604:65f0:93f0%fxp0 (reachable) fc00:10:1:1::/64 if=fxp0 flags=LD vltime=0, pltime=0, expired, ref=1 No advertising router The code in question (around line 1396 of n6_rtr.c) seems to mark any non-advertised prefix as detached - the comment in front of this segment (around line 1376) indicates this is done to take care of a move to a different network: if ((pr->ndpr_stateflags & NDPRF_DETACHED) == 0 && find_pfxlist_reachable_router(pr) == NULL) ---> pr->ndpr_stateflags |= NDPRF_DETACHED; If this is still the current thinking, it looks like my usage scenario is incorrect and I would like to understand why that is so. And what is the workaround? If not and this is a bug, then would suggest a addition to the code to allow only "non-static" prefixes to be detached... if ((pr->ndpr_stateflags & NDPRF_DETACHED) == 0 && find_pfxlist_reachable_router(pr) == NULL && pr->ndpr_pltime != ND6_INFINITE_LIFETIME) pr->ndpr_stateflags |= NDPRF_DETACHED; Thanks in advance for your help, --shekhar ------------------------------------------------------------------------------------------------- (gcshekhar AT sbc NOSPACE global DOT net) Confidence is the feeling you have before you understand the situation From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 04:52:36 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 49FC51065673 for ; Fri, 4 Jul 2008 04:52:36 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id EA7FA8FC0A for ; Fri, 4 Jul 2008 04:52:35 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KEdE3-0004Lb-6y; Fri, 04 Jul 2008 00:48:51 -0400 Message-ID: <486DAD0D.8090604@gtcomm.net> Date: Fri, 04 Jul 2008 00:54:37 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Bruce Evans References: <4867420D.7090406@gtcomm.net> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> <486C7F93.7010308@gtcomm.net> <20080703195521.O6973@delplex.bde. org> <486D35A0.4000302@gtcomm.net> In-Reply-To: <486D35A0.4000302@gtcomm.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Ingo Flaschberger Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 04:52:36 -0000 Numbers are maximum with near 100% cpu usage and some errors occuring, just for testing. FreeBSD 7.0-STABLE FreeBSD 7.0-STABLE #6: Thu Jul 3 19:32:38 CDT 2008 root@foo:/usr/obj/usr/src/sys/ROUTER amd64 CPU: Dual-Core AMD Opteron(tm) Processor 2222 (3015.47-MHz K8-class CPU) NON-SMP KERNEL em driver, intel 82571EB NICs fastforwarding on, isr.direct on, ULE, Preemption (NOTE: Interesting thing, without preemption gets errors similar to polling) 64 bit.. 1.1mpps max with opteron 2222 one direction no routing table, no firewall -> em0 --> em3 -> 64 bit.. 700k max with opteron 2222 one direction no routing table, one ipfw rule -> em0 --> em3 -> 64 bit.. 500kpps max with opteron 2222 one direction no routing table, 20 ipfw rule -> em0 --> em3 -> 64 bit.. 750kpps max with opteron 2222 one direction Full BGP (260k route) table -> em0 --> em3 -> 64 bit.. 400kpps max with opteron 2222 one direction no routing table, 2 pf rules no state -> em0 --> em3 -> using lagg driver in etherchannel with 2 ports (em0,em1) reduces the performance by about 8% which is strange as it shouldn't. In SMP mode lagg driver reduces it substantially more, and this is where it should increase performance greatly because incoming packets are load balanced over multiple NICs.. :/ 32 bit test coming next, then I'm going with a high mhz Xeon or c2d proc 45nm and post those results (using same source tree/kernel/etc) I tried polling, and I tried the polling patch that was posted to the list and both work but generate too many errors (missed packets). Without polling the packet errors ONLY occur when the cpu is near 100% usage Paul wrote: > Opteron 2222 UP mode, no polling > > input (em0) output > packets errs bytes packets errs bytes colls > 1071020 0 66403248 2 0 404 0 > 1049793 0 65087174 2 0 356 0 > 1040320 0 64499848 2 0 356 0 > 1049712 0 65082152 2 0 356 0 > 1039504 0 64449256 2 0 356 0 > 933118 0 57853324 2 0 356 0 > > still has some cpu left and i can't generate any more packets > > Polling turned on provided better performance on 32 bit, but it gets > strange errors on 64 bit.. > Even at low pps I get small amounts of errors, and high pps same > thing.. you would think that if > it got errors at low pps it would get more errors at high pps but that > isn't the case.. > Polling on: > packets errs bytes packets errs bytes colls > 979736 963 60743636 1 0 226 0 > 991838 496 61493960 1 0 178 0 > 996125 460 61759754 1 0 178 0 > 979381 326 60721626 1 0 178 0 > 1022249 379 63379442 1 0 178 0 > 991468 557 61471020 1 0 178 0 > > lowering pps a little....... > input (em0) output > packets errs bytes packets errs bytes colls > 818688 151 50758660 1 0 226 0 > 837920 179 51951044 1 0 178 0 > 826217 168 51225458 1 0 178 0 > 801017 100 49663058 1 0 178 0 > 761857 287 47235138 1 0 178 0 > > > what could cause this? > > If i'm going to use a uniprocessor mode system I NEED polling to work > because I have to have > cpu cycles left over for userspace processes and I can't afford to > have it lock those out. > SMP is no big deal if it actually worked.. > > I'm going to do a SMP test with this cpu now with polling off/on and > then I'm going to apply the polling patch and try that. > > > > Bruce Evans wrote: >> On Thu, 3 Jul 2008, Paul wrote: >> >>> Bruce Evans wrote: >>>>> No polling: >>>>> 843762 25337 52313248 1 0 178 0 >>>>> 763555 0 47340414 1 0 178 0 >>>>> 830189 0 51471722 1 0 178 0 >>>>> 838724 0 52000892 1 0 178 0 >>>>> 813594 939 50442832 1 0 178 0 >>>>> 807303 763 50052790 1 0 178 0 >>>>> 791024 0 49043492 1 0 178 0 >>>>> 768316 1106 47635596 1 0 178 0 >>>>> Machine is maxed and is unresponsive.. >>>> >>>> That's the most interesting one. Even 1% packet loss would probably >>>> destroy performance, so the benchmarks that give 10-50% packet loss >>>> are uninteresting. >>>> >>> But you realize that it's outputting all of these packets on em3 >>> and I'm watching them coming out >>> and they are consistent with the packets received on em0 that >>> netstat shows are 'good' packets. >> >> Well, output is easier. I don't remember seeing the load on a taskq for >> em3. If there is a memory bottleneck, it might to might not be more >> related >> to running only 1 taskq per interrupt, depending on how independent the >> memory system is for different CPU. I think Opterons have more >> indenpendence >> here than most x86's. >> >>> I'm using a server opteron which supposedly has the best memory >>> performance out of any CPU right now. >>> Plus opterons have the biggest l1 cache, but small l2 cache. Do you >>> think larger l2 cache on the Xeon (6mb for 2 core) would be better? >>> I have a 2222 opteron coming which is 1ghz faster so we will see >>> what happens >> >> I suspect lower latency memory would help more. Big memory systems >> have inherently higher latency. My little old A64 workstation and >> laptop have main memory latencies 3 times smaller than freebsd.org's >> new Core2 servers according to lmbench2 (42 nsec for the overclocked >> DDR PC3200 one and 55 for the DDR2 PC5400 (?) one, vs 145-155 nsec). >> If there are a lot of cache misses, then the extra 100 nsec can be >> important. Profiling of sendto() using hwpmc or perfmon shows a >> significant number of cache misses per packet (2 or 10?). >> >>>>> Polling ON: >>>>> input (em0) output >>>>> packets errs bytes packets errs bytes colls >>>>> 784138 179079 48616564 1 0 226 0 >>>>> 788815 129608 48906530 2 0 356 0 >>>>> Machine is responsive and has 40% idle cpu.. Why ALWAYS 40% ? I'm >>>>> really mistified by this.. >>>> >>>> Is this with hz=2000 and 256/256 and no polling in idle? 40% is easy >>>> to explain (perhaps incorrectly). Polling can then read at most 256 >>>> descriptors every 1/2000 second, giving a max throughput of 512 kpps. >>>> Packets < descriptors in general but might be equal here (for small >>>> packets). You seem to actually get 784 kpps, which is too high even >>>> in descriptors unless, but matches exactly if the errors are counted >>>> twice (784 - 179 - 505 ~= 512). CPU is getting short too, but 40% >>>> still happens to be left over after giving up at 512 kpps. Most of >>>> the errors are probably handled by the hardware at low cost in CPU by >>>> dropping packets. There are other types of errors but none except >>>> dropped packets is likely. >>>> >>> Read above, it's actually transmitting 770kpps out of em3 so it >>> can't just be 512kpps. >> >> Transmitting is easier, but with polling its even harder to send >> faster than >> hz * queue_length than to receive. This is without polling in idle. >> >>> I was thinking of trying 4 or 5.. but how would that work with this >>> new hardware? >> >> Poorly, except possibly with polling in FreeBSD-4. FreeBSD-4 generally >> has lower overheads and latency, but is missing important improvements >> (mainly tcp optimizations in upper layers, better DMA and/or mbuf >> handling, and support for newer NICs). FreeBSD-5 is also missing the >> overhead+latency advantage. >> >> Here are some benchmarks. (ttcp mainly tests sendto(). 4.10 em needed a >> 2-line change to support a not-so-new PCI em NIC. Summary: >> - my bge NIC can handle about 600 kpps on my faster machine, but only >> achieves 300 in 4.10 unpatched. >> - my em NIC can handle about 400 kpps on my slower machine, except in >> later versions it can receive at about 600 kpps. >> - only 6.x and later can achieve near wire throughput for 1500-MTU >> packets (81 kpps vs 76 kpps). This depends on better DMA or mbuf >> handling... I now remember the details -- it is mainly better mbuf >> handling: old versions split the 1500-MTU packets into 2 mbufs and >> this causes 2 descriptors per packet, which causes extra software >> overheads and even larger overheads for the hardware. >> >> %%% >> Results of benchmarks run on 23 Feb 2007: >> >> my~5.2 bge --> ~4.10 em >> tx rx >> kpps load% ips kpps load% ips >> ttcp -l5 -u -t 639 98 1660 398* 77 8k >> ttcp -l5 -t 6.0 100 3960 6.0 6 5900 >> ttcp -l1472 -u -t 76 27 395 76 40 8k >> ttcp -l1472 -t 51 40 11k 51 26 8k >> >> (*) Same as sender according to netstat -I, but systat -ip shows that >> almost half aren't delivered to upper layers. >> >> my~5.2 bge --> 4.11 em >> tx rx >> kpps load% ips kpps load% ips >> ttcp -l5 -u -t 635 98 1650 399* 74 8k >> ttcp -l5 -t 5.8 100 3900 5.8 6 5800 >> ttcp -l1472 -u -t 76 27 395 76 32 8k >> ttcp -l1472 -t 51 40 11k 51 25 8k >> >> (*) Same as sender according to netstat -I, but systat -ip shows that >> almost half aren't delivered to upper layers. >> >> my~5.2 bge --> my~5.2 em >> tx rx >> kpps load% ips kpps load% ips >> ttcp -l5 -u -t 638 98 1660 394* 100- 8k >> ttcp -l5 -t 5.8 100 3900 5.8 9 6000 >> ttcp -l1472 -u -t 76 27 395 76 46 8k >> ttcp -l1472 -t 51 40 11k 51 35 8k >> >> (*) Same as sender according to netstat -I, but systat -ip shows that >> almost half aren't delivered to upper layers. With the em rate >> limit on ips changed from 8k to 80k, about 95% are delivered up. >> >> my~5.2 bge --> 6.2 em >> tx rx >> kpps load% ips kpps load% ips >> ttcp -l5 -u -t 637 98 1660 637 100- 15k >> ttcp -l5 -t 5.8 100 3900 5.8 8 12k >> ttcp -l1472 -u -t 76 27 395 76 36 16k >> ttcp -l1472 -t 51 40 11k 51 37 16k >> >> my~5.2 bge --> ~current em-fastintr >> tx rx >> kpps load% ips kpps load% ips >> ttcp -l5 -u -t 641 98 1670 641 99 8k >> ttcp -l5 -t 5.9 100 2670 5.9 7 6k >> ttcp -l1472 -u -t 76 27 395 76 35 8k >> ttcp -l1472 -t 52 43 11k 52 30 8k >> >> ~6.2 bge --> ~current em-fastintr >> tx rx >> kpps load% ips kpps load% ips >> ttcp -l5 -u -t 309 62 1600 309 64 8k >> ttcp -l5 -t 4.9 100 3000 4.9 6 7k >> ttcp -l1472 -u -t 76 27 395 76 34 8k >> ttcp -l1472 -t 54 28 6800 54 30 8k >> >> ~current bge --> ~current em-fastintr >> tx rx >> kpps load% ips kpps load% ips >> ttcp -l5 -u -t 602 100 1570 602 99 8k >> ttcp -l5 -t 5.3 100 2660 5.3 5 5300 >> ttcp -l1472 -u -t 81# 19 212 81# 38 8k >> ttcp -l1472 -t 53 34 11k 53 30 8k >> >> (#) Wire speed to within 0.5%. This is the only kppps in this set of >> benchmarks that is close to wire speed. Older kernels apparently >> lose relative to -current because mbufs for mtu-sized packets are >> not contiguous in older kernels. >> >> Old results: >> >> ~4.10 bge --> my~5.2 em >> tx rx >> kpps load% ips kpps load% ips >> ttcp -l5 -u -t n/a n/a n/a 346 79 8k >> ttcp -l5 -t n/a n/a n/a 5.4 10 6800 >> ttcp -l1472 -u -t n/a n/a n/a 67 40 8k >> ttcp -l1472 -t n/a n/a n/a 51 36 8k >> >> ~4.10 kernel, =4 bge --> ~current em >> tx rx >> kpps load% ips kpps load% ips >> ttcp -l5 -u -t n/a n/a n/a 347 96 14k >> ttcp -l5 -t n/a n/a n/a 5.8 10 14k >> ttcp -l1472 -u -t n/a n/a n/a 67 62 14K >> ttcp -l1472 -t n/a n/a n/a 52 40 16k >> >> ~4.10 kernel, =4+ bge --> ~current em >> tx rx >> kpps load% ips kpps load% ips >> ttcp -l5 -u -t n/a n/a n/a 627 100 9k >> ttcp -l5 -t n/a n/a n/a 5.6 9 13k >> ttcp -l1472 -u -t n/a n/a n/a 68 63 14k >> ttcp -l1472 -t n/a n/a n/a 54 44 16k >> %%% >> >> %%% >> Results of benchmarks run on 28 Dec 2007: >> >> ~5.2 epsplex (em) ttcp: >> Csw Trp Sys Int Sof Sys Intr User >> Idle >> local no sink: 825k 3 206k 229 412k 52.1 45.1 2.8 >> local with sink: 659k 3 263k 231 131k 66.5 27.3 6.2 >> tx remote no sink: 35k 3 273k 8237 266k 42.0 52.1 2.3 >> 3.6 >> tx remote with sink: 26k 3 394k 8224 100 60.0 5.41 3.4 >> 11.2 >> rx remote no sink: 25k 4 26 8237 373k 20.6 79.4 0.0 >> 0.0 >> rx remote with sink: 30k 3 203k 8237 398k 36.5 60.7 2.8 >> 0.0 >> >> 6.3-PR besplex (em) ttcp: >> Csw Trp Sys Int Sof Sys Intr User >> Idle >> local no sink: 417k 1 208k 418k 2 49.5 48.5 2.0 >> local with sink: 420k 1 276k 145k 2 70.0 23.6 6.4 >> tx remote no sink: 19k 2 250k 8144 2 58.5 38.7 2.8 >> 0.0 >> tx remote with sink: 16k 2 361k 8336 2 72.9 24.0 3.1 >> 4.4 >> rx remote no sink: 429 3 49 888 2 0.3 99.33 0.0 >> 0.4 >> tx remote with sink: 13k 2 316k 5385 2 31.7 63.8 3.6 >> 0.8 >> >> 8.0-C epsplex (em-fast) ttcp: >> Csw Trp Sys Int Sof Sys Intr User >> Idle >> local no sink: 442k 3 221k 230 442k 47.2 49.6 2.7 >> local with sink: 394k 3 262k 228 131k 72.1 22.6 5.3 >> tx remote no sink: 17k 3 226k 7832 100 94.1 0.2 3.0 >> 0.0 >> tx remote with sink: 17k 3 360k 7962 100 91.7 0.2 3.7 >> 4.4 >> rx remote no sink: saturated -- cannot update systat display >> rx remote with sink: 15k 6 358k 8224 100 97.0 0.0 2.5 >> 0.5 >> >> ~4.10 besplex (bge) ttcp: >> Csw Trp Sys Int Sof Sys Intr User >> Idle >> local no sink: 15 0 425k 228 11 96.3 0.0 3.7 >> local with sink: ** 0 622k 229 ** 94.7 0.3 5.0 >> tx remote no sink: 29 1 490k 7024 11 47.9 29.8 4.4 >> 17.9 >> tx remote with sink: 26 1 635k 1883 11 65.7 11.4 5.6 >> 17.3 >> rx remote no sink: 5 1 68 7025 1 0.0 47.3 0.0 >> 52.7 >> rx remote with sink: 6679 2 365k 6899 12 19.7 29.2 2.5 >> 48.7 >> >> ~5.2-C besplex (bge) ttcp: >> Csw Trp Sys Int Sof Sys Intr User >> Idle >> local no sink: 1M 3 271k 229 543k 50.7 46.8 2.5 >> local with sink: 1M 3 406k 229 203k 67.4 28.2 4.4 >> tx remote no sink: 49k 3 474k 11k 167k 52.3 42.7 5.0 >> 0.0 >> tx remote with sink: 6371 3 641k 1900 100 76.0 16.8 6.2 >> 0.9 >> rx remote no sink: 34k 3 25 11k 270k 0.8 65.4 0.0 >> 33.8 >> rx remote with sink: 41k 3 365k 10k 370k 31.5 47.1 2.3 >> 19.0 >> >> 6.3-PR besplex (bge) ttcp (hz = 1000 else stathz broken): >> Csw Trp Sys Int Sof Sys Intr User >> Idle >> local no sink: 540k 0 270k 540k 0 50.5 46.0 3.5 >> local with sink: 628k 0 417k 210k 0 68.8 27.9 3.3 >> tx remote no sink: 15k 1 222k 7190 1 28.4 29.3 1.7 >> 40.6 >> tx remote with sink: 5947 1 315k 2825 1 39.9 14.7 2.6 >> 42.8 >> rx remote no sink: 13k 1 23 6943 0 0.3 49.5 0.2 >> 50.0 >> rx remote with sink: 20k 1 371k 6819 0 29.5 30.1 3.9 >> 36.5 >> >> 8.0-C besplex (bge) ttcp: >> Csw Trp Sys Int Sof Sys Intr User >> Idle >> local no sink: 649k 3 324k 100 649k 53.9 42.9 3.2 >> local with sink: 649k 3 433k 100 216k 75.2 18.8 6.0 >> tx remote no sink: 24k 3 432k 10k 100 49.7 41.3 2.4 >> 6.6 >> tx remote with sink: 3199 3 568k 1580 100 64.3 19.6 4.0 >> 12.2 >> rx remote no sink: 20k 3 27 10k 100 0.0 46.1 0.0 >> 53.9 >> rx remote with sink: 31k 3 370k 10k 100 30.7 30.9 4.8 >> 33.5 >> %%% >> >> Bruce >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" > From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 06:28:56 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EAE44106564A for ; Fri, 4 Jul 2008 06:28:56 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail05.syd.optusnet.com.au (mail05.syd.optusnet.com.au [211.29.132.186]) by mx1.freebsd.org (Postfix) with ESMTP id 7949B8FC1B for ; Fri, 4 Jul 2008 06:28:56 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from c220-239-252-11.carlnfd3.nsw.optusnet.com.au (c220-239-252-11.carlnfd3.nsw.optusnet.com.au [220.239.252.11]) by mail05.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id m646SltI019351 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 4 Jul 2008 16:28:51 +1000 Date: Fri, 4 Jul 2008 16:28:47 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Paul In-Reply-To: <486DAD0D.8090604@gtcomm.net> Message-ID: <20080704160950.Y9864@delplex.bde.org> References: <4867420D.7090406@gtcomm.net> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> <486C7F93.7010308@gtcomm.net> <20080703195521.O6973@delplex.bde. org> <486D35A0.4000302@gtcomm.net> <486DAD0D.8090604@gtcomm.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net , Ingo Flaschberger Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 06:28:57 -0000 On Fri, 4 Jul 2008, Paul wrote: > Numbers are maximum with near 100% cpu usage and some errors occuring, just > for testing. > FreeBSD 7.0-STABLE FreeBSD 7.0-STABLE #6: Thu Jul 3 19:32:38 CDT 2008 > root@foo:/usr/obj/usr/src/sys/ROUTER amd64 > CPU: Dual-Core AMD Opteron(tm) Processor 2222 (3015.47-MHz K8-class CPU) > NON-SMP KERNEL em driver, intel 82571EB NICs > fastforwarding on, isr.direct on, ULE, Preemption (NOTE: Interesting thing, > without preemption gets errors similar to polling) PREEMPTION is certainly needed with UP. Without it, interrupts don't actually work (to work, they need to preempt the running thread, but they often (usually?) don't do that). Then with UP, there is a good chance that the interrupt thread doesn't get scheduled to run for a long time, but with SMP (especially with lots of CPUs) there is a good chance that another CPU gets scheduled to run the interrupt thread. em (unless misconfigured) doesn't have an interrupt thread; it uses a taskq which might take even longer to be scheduled than an interrupt thread. I use PREEMPTION with UP and !PREEMPTION with SMP. With polling, missed polls cause the same packet loss as not preempting. > I tried polling, and I tried the polling patch that was posted to the list > and both work but generate too many errors (missed packets). > Without polling the packet errors ONLY occur when the cpu is near 100% usage Polling should also only cause packet loss when the CPU is near 100% usage, but now transients of near 100% usually cause packet loss, while with interrupts it takes a transient of > 100% on the competing interrupt- driven resources to cause packet loss. Pleas trim quotes. Bruce From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 09:10:20 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 510EA1065676 for ; Fri, 4 Jul 2008 09:10:20 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id A4F9D8FC17 for ; Fri, 4 Jul 2008 09:10:18 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 27787 invoked from network); 4 Jul 2008 11:10:17 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 4 Jul 2008 11:10:17 +0200 Date: Fri, 4 Jul 2008 11:10:17 +0200 (CEST) From: Ingo Flaschberger To: Paul In-Reply-To: <486D35A0.4000302@gtcomm.net> Message-ID: References: <4867420D.7090406@gtcomm.net> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> <486C7F93.7010308@gtcomm.net> <20080703195521.O6973@delplex.bde.org> <486D35A0.4000302@gtcomm.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; format=flowed; charset=US-ASCII Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 09:10:20 -0000 Dear Paul, > Opteron 2222 UP mode, no polling > > input (em0) output > packets errs bytes packets errs bytes colls > 1071020 0 66403248 2 0 404 0 that looks good. (but seems to be near the limit). > Polling turned on provided better performance on 32 bit, but it gets strange > errors on 64 bit.. > Even at low pps I get small amounts of errors, and high pps same thing.. you > would think that if > it got errors at low pps it would get more errors at high pps but that isn't > the case.. > Polling on: > packets errs bytes packets errs bytes colls > 979736 963 60743636 1 0 226 0 > 991838 496 61493960 1 0 178 0 > 996125 460 61759754 1 0 178 0 > 979381 326 60721626 1 0 178 0 > 1022249 379 63379442 1 0 178 0 > 991468 557 61471020 1 0 178 0 > > lowering pps a little....... > input (em0) output > packets errs bytes packets errs bytes colls > 818688 151 50758660 1 0 226 0 > 837920 179 51951044 1 0 178 0 > 826217 168 51225458 1 0 178 0 > 801017 100 49663058 1 0 178 0 > 761857 287 47235138 1 0 178 0 > > > what could cause this? *) kern.polling.idle_poll enabled? *) kern.polling.user_frac ? *) kern.polling.reg_frac ? *) kern.polling.burst_max ? *) kern.polling.each_burst ? Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 09:45:22 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74D591065674 for ; Fri, 4 Jul 2008 09:45:22 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 2B8F18FC21 for ; Fri, 4 Jul 2008 09:45:21 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KEhnD-0002fY-N9; Fri, 04 Jul 2008 05:41:27 -0400 Message-ID: <486DF1A3.9000409@gtcomm.net> Date: Fri, 04 Jul 2008 05:47:15 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> <486C7F93.7010308@gtcomm.net> <20080703195521.O6973@delplex.bde.org> <486D35A0.4000302@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 09:45:22 -0000 ngo Flaschberger wrote: > Dear Paul, > >> Opteron 2222 UP mode, no polling >> >> input (em0) output >> packets errs bytes packets errs bytes colls >> 1071020 0 66403248 2 0 404 0 > > that looks good. (but seems to be near the limit). > Yes it is , any more and errors start. >> Polling turned on provided better performance on 32 bit, but it gets >> strange errors on 64 bit.. >> Even at low pps I get small amounts of errors, and high pps same >> thing.. you would think that if >> it got errors at low pps it would get more errors at high pps but >> that isn't the case.. >> Polling on: >> packets errs bytes packets errs bytes colls >> 979736 963 60743636 1 0 226 0 >> 991838 496 61493960 1 0 178 0 >> 996125 460 61759754 1 0 178 0 >> >> >> what could cause this? > > *) kern.polling.idle_poll enabled? > *) kern.polling.user_frac ? > *) kern.polling.reg_frac ? > *) kern.polling.burst_max ? > *) kern.polling.each_burst ? I tried tons of different values for these and nothing made any significant difference. Idle polling makes a difference, allows more pps, but still errors. Without idle polling it seems PPS is limited to HZ * descriptors, or 1000 * 256 or 512 but 1000 * 1024 is the same as 512.. 4000 * 256 or 2000 * 512 works but starts erroring 600kpps (SMP right now but it happens in UP too) If anyone wants to log into the box and play with settings, recompile the kernel, etc. Let me know. Paul From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 11:16:16 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 901521065676 for ; Fri, 4 Jul 2008 11:16:16 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id DE5C28FC18 for ; Fri, 4 Jul 2008 11:16:15 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 1868 invoked from network); 4 Jul 2008 13:16:13 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 4 Jul 2008 13:16:13 +0200 Date: Fri, 4 Jul 2008 13:16:13 +0200 (CEST) From: Ingo Flaschberger To: Paul In-Reply-To: <486DF1A3.9000409@gtcomm.net> Message-ID: References: <4867420D.7090406@gtcomm.net> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> <486C7F93.7010308@gtcomm.net> <20080703195521.O6973@delplex.bde.org> <486D35A0.4000302@gtcomm.net> <486DF1A3.9000409@gtcomm.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 11:16:16 -0000 Dear Paul, >>> what could cause this? >> >> *) kern.polling.idle_poll enabled? >> *) kern.polling.user_frac ? >> *) kern.polling.reg_frac ? >> *) kern.polling.burst_max ? >> *) kern.polling.each_burst ? > > I tried tons of different values for these and nothing made any significant > difference. > Idle polling makes a difference, allows more pps, but still errors. > Without idle polling it seems PPS is limited to HZ * descriptors, or 1000 * > 256 or 512 > but 1000 * 1024 is the same as 512.. 4000 * 256 or 2000 * 512 works but > starts erroring 600kpps (SMP right now but it happens in UP too) I have patched src/sys/kern/kern_poll.c to support higher burst_max values: #define MAX_POLL_BURST_MAX 10000 When setting kern.polling.burst_max to higher values, the server reach a point, where cpu-usage goes up without load, so try to keep below this values. I also have set the network card to 4096 rx-"ram", to have more room for late polls. Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 11:32:14 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 74D13106567E; Fri, 4 Jul 2008 11:32:14 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 597A98FC1C; Fri, 4 Jul 2008 11:32:14 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id EAE391CC073; Fri, 4 Jul 2008 04:32:13 -0700 (PDT) Date: Fri, 4 Jul 2008 04:32:13 -0700 From: Jeremy Chadwick To: Kian Mohageri Message-ID: <20080704113213.GA13586@eos.sc1.parodius.com> References: <678A03F5-5E8A-4CF6-90DF-AA9A4F30FBE1@stromnet.se> <1211037564.6326.27.camel@porksoda> <679DB462-75D6-45CC-949C-1BE8E12C22CD@stromnet.se> <482FD877.6050707@infracaninophile.co.uk> <20080703003955.859BCF180C0@mx.npubs.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-stable , stef@memberwebs.com, freebsd-net@freebsd.org, Matthew Seaman , freebsd-pf@freebsd.org, Alex Trull Subject: Re: connect(): Operation not permitted X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 11:32:14 -0000 On Thu, Jul 03, 2008 at 08:55:21AM -0700, Kian Mohageri wrote: > On Wed, Jul 2, 2008 at 5:39 PM, Stef wrote: > > Kian Mohageri wrote: > >> On Sun, May 18, 2008 at 3:33 AM, Johan Ström wrote: > >>> On May 18, 2008, at 9:19 AM, Matthew Seaman wrote: > >>> > >>>> Johan Ström wrote: > >>>> > >>>>> drop all traffic)? A check with pfctl -vsr reveals that the actual rule > >>>>> inserted is "pass on lo0 inet from 123.123.123.123 to 123.123.123.123 flags > >>>>> S/SA keep state". Where did that "keep state" come from? > >>>> 'flags S/SA keep state' is the default now for tcp filter rules -- that > >>>> was new in 7.0 reflecting the upstream changes made between the 4.0 and > >>>> 4.1 > >>>> releases of OpenBSD. If you want a stateless rule, append 'no state'. > >>>> > >>>> http://www.openbsd.org/faq/pf/filter.html#state > >>> Thanks! I was actually looking around in the pf.conf manpage but failed to > >>> find it yesterday, but looking closer today I now saw it. > >>> Applied the no state (and quick) to the rule, and now no state is created. > >>> And the problem I had in the first place seems to have been resolved too > >>> now, even though it didn't look like a state problem.. (started to deny new > >>> connections much earlier than the states was full, altough maybee i wasnt > >>> looking for updates fast enough or something). > >>> > >> > >> I'd be willing to bet it's because you're reusing the source port on a > >> new connection before the old state expires. > >> > >> You'll know if you check the state-mismatch counter. > >> > >> Anyway, glad you found a resolution. > > > > I've been experiencing this "Operation not permitted" too. I've been > > trying to track down the problem for many months, but due to the > > complexity of my firewalls (scores of jails each with scores of rules), > > I wasn't brave enough to ask for help :) > > > > As a work around we started creating rules without state, whenever we > > would run into the problem. > > > > Thanks for the pointer about state-mismatch. The state-mismatch counter > > does is in fact high in my case (see below). How would I go about > > getting the pf state timeout and the reuse of ports for outbound > > connections to match? Or is this an intractable problem, that just needs > > to be worked around? > > Make sure your state-mismatch counter is increasing at the same times > you experience the problem (and isn't just high from some unrelated > issue). > > A similar/related problem was addressed in OpenBSD 4.3 > (http://www.openbsd.org/plus43.html). > > * In pf(4), allow state reuse if both sides are in FIN_WAIT_2 and a > new SYN arrives. > > I'm not sure if it's been imported yet. If not, you could try tuning > your timeout values (see pf.conf(5)). > > The specific issue I was experienced was solved by shortening > tcp.closed, IIRC. It's been a while though. When administrators see state-mismatch increasing, they get concerned. The common scapegoat is tcp.closed, which people don't even bother to describe (pf has an internal value of 10 seconds applied to that value, e.g. tcp.closed=5 means 15 seconds). You can set tcp.closed as low as you want, but chances are random Internet users will have equipment with IP stacks that re-use outbound sockets which haven't fully closed down within the aforementioned interval. pf cannot fix this. For example, on our production/hosting systems, we see state-mismatch increase fairly often. I just pfctl -F info'd our main webserver, and within about 15 minutes, state-mismatch was up to 22. We use tcp.closed of 5 (which means 15 seconds). Workarounds such as "no state" suffice, but if you use rdr rules, you MUST track state, which means there's no way of winning in that case. For sake of example, OpenBSD spamd requires the use of rdr rules. Administrators then ask 3 questions: 1) How do I determine whether or not state-mismatch increasing is a sign of bad things, or due to peoples' broken IP stacks, 2) What happens to packets which cause state-mismatch to increment, e.g. are they blocked, passed, or what? 3) Why isn't state-mismatch described in detail in the documentation? Finally, the fix in OpenBSD 4.3 should really be backported to FreeBSD ASAP. -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 12:10:50 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AEA6B106567F; Fri, 4 Jul 2008 12:10:50 +0000 (UTC) (envelope-from jdc@parodius.com) Received: from mx01.sc1.parodius.com (mx01.sc1.parodius.com [72.20.106.3]) by mx1.freebsd.org (Postfix) with ESMTP id 896208FC1B; Fri, 4 Jul 2008 12:10:50 +0000 (UTC) (envelope-from jdc@parodius.com) Received: by mx01.sc1.parodius.com (Postfix, from userid 1000) id 4D7EC1CC081; Fri, 4 Jul 2008 05:10:50 -0700 (PDT) Date: Fri, 4 Jul 2008 05:10:50 -0700 From: Jeremy Chadwick To: Kian Mohageri Message-ID: <20080704121050.GA14604@eos.sc1.parodius.com> References: <678A03F5-5E8A-4CF6-90DF-AA9A4F30FBE1@stromnet.se> <1211037564.6326.27.camel@porksoda> <679DB462-75D6-45CC-949C-1BE8E12C22CD@stromnet.se> <482FD877.6050707@infracaninophile.co.uk> <20080703003955.859BCF180C0@mx.npubs.com> <20080704113213.GA13586@eos.sc1.parodius.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080704113213.GA13586@eos.sc1.parodius.com> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-stable , stef@memberwebs.com, freebsd-net@freebsd.org, Matthew Seaman , freebsd-pf@freebsd.org, Alex Trull Subject: Re: connect(): Operation not permitted X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 12:10:50 -0000 On Fri, Jul 04, 2008 at 04:32:13AM -0700, Jeremy Chadwick wrote: > On Thu, Jul 03, 2008 at 08:55:21AM -0700, Kian Mohageri wrote: > > A similar/related problem was addressed in OpenBSD 4.3 > > (http://www.openbsd.org/plus43.html). > > > > * In pf(4), allow state reuse if both sides are in FIN_WAIT_2 and a > > new SYN arrives. The OpenBSD diff: http://www.openbsd.org/cgi-bin/cvsweb/src/sys/net/pf.c.diff?r2=1.559&r1=1.558&f=H I've submit a FreeBSD PR to get the above backported into RELENG_7 and RELENG_6: http://www.freebsd.org/cgi/query-pr.cgi?pr=125261 -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 14:50:33 2008 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 00ACB1065672; Fri, 4 Jul 2008 14:50:33 +0000 (UTC) (envelope-from remko@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id CA22C8FC1B; Fri, 4 Jul 2008 14:50:32 +0000 (UTC) (envelope-from remko@FreeBSD.org) Received: from freefall.freebsd.org (remko@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m64EoWMj080209; Fri, 4 Jul 2008 14:50:32 GMT (envelope-from remko@freefall.freebsd.org) Received: (from remko@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m64EoWMa080205; Fri, 4 Jul 2008 14:50:32 GMT (envelope-from remko) Date: Fri, 4 Jul 2008 14:50:32 GMT Message-Id: <200807041450.m64EoWMa080205@freefall.freebsd.org> To: remko@FreeBSD.org, freebsd-i386@FreeBSD.org, freebsd-net@FreeBSD.org From: remko@FreeBSD.org Cc: Subject: Re: kern/125195: fxp(4) driver failed to initialize device Intel 82801DB X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 14:50:33 -0000 Synopsis: fxp(4) driver failed to initialize device Intel 82801DB Responsible-Changed-From-To: freebsd-i386->freebsd-net Responsible-Changed-By: remko Responsible-Changed-When: Fri Jul 4 14:50:18 UTC 2008 Responsible-Changed-Why: Reassign to -NET http://www.freebsd.org/cgi/query-pr.cgi?pr=125195 From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 14:51:07 2008 Return-Path: Delivered-To: freebsd-net@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 90CB11065675; Fri, 4 Jul 2008 14:51:07 +0000 (UTC) (envelope-from remko@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 66A768FC14; Fri, 4 Jul 2008 14:51:07 +0000 (UTC) (envelope-from remko@FreeBSD.org) Received: from freefall.freebsd.org (remko@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.2/8.14.2) with ESMTP id m64Ep79e080296; Fri, 4 Jul 2008 14:51:07 GMT (envelope-from remko@freefall.freebsd.org) Received: (from remko@localhost) by freefall.freebsd.org (8.14.2/8.14.1/Submit) id m64Ep7Yk080292; Fri, 4 Jul 2008 14:51:07 GMT (envelope-from remko) Date: Fri, 4 Jul 2008 14:51:07 GMT Message-Id: <200807041451.m64Ep7Yk080292@freefall.freebsd.org> To: remko@FreeBSD.org, freebsd-i386@FreeBSD.org, freebsd-net@FreeBSD.org From: remko@FreeBSD.org Cc: Subject: Re: kern/125258: socket's SO_REUSEADDR option does not work X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 14:51:07 -0000 Synopsis: socket's SO_REUSEADDR option does not work Responsible-Changed-From-To: freebsd-i386->freebsd-net Responsible-Changed-By: remko Responsible-Changed-When: Fri Jul 4 14:50:44 UTC 2008 Responsible-Changed-Why: reassign to -net http://www.freebsd.org/cgi/query-pr.cgi?pr=125258 From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 18:01:23 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 641EA106564A for ; Fri, 4 Jul 2008 18:01:23 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 1B74E8FC0A for ; Fri, 4 Jul 2008 18:01:22 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KEpXE-0003H3-JC; Fri, 04 Jul 2008 13:57:28 -0400 Message-ID: <486E65E6.3060301@gtcomm.net> Date: Fri, 04 Jul 2008 14:03:18 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Ingo Flaschberger References: <4867420D.7090406@gtcomm.net> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> <486C7F93.7010308@gtcomm.net> <20080703195521.O6973@delplex.bde.org> <486D35A0.4000302@gtcomm.net> <486DF1A3.9000409@gtcomm.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 18:01:23 -0000 I tried all of this :/ still, 256/512 descriptors seem to work the best. Happy to let you log into the machine and fiddle around if you want :) Paul Ingo Flaschberger wrote: > Dear Paul, > >>>> what could cause this? >>> >>> *) kern.polling.idle_poll enabled? >>> *) kern.polling.user_frac ? >>> *) kern.polling.reg_frac ? >>> *) kern.polling.burst_max ? >>> *) kern.polling.each_burst ? >> >> I tried tons of different values for these and nothing made any >> significant difference. >> Idle polling makes a difference, allows more pps, but still errors. >> Without idle polling it seems PPS is limited to HZ * descriptors, or >> 1000 * 256 or 512 >> but 1000 * 1024 is the same as 512.. 4000 * 256 or 2000 * 512 works >> but starts erroring 600kpps (SMP right now but it happens in UP too) > > I have patched src/sys/kern/kern_poll.c to support higher burst_max > values: > #define MAX_POLL_BURST_MAX 10000 > > When setting kern.polling.burst_max to higher values, the server reach > a point, where cpu-usage goes up without load, so try to keep below > this values. I also have set the network card to 4096 rx-"ram", to > have more room for late polls. > > Kind regards, > Ingo Flaschberger > > From owner-freebsd-net@FreeBSD.ORG Fri Jul 4 21:30:50 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 878A81065671 for ; Fri, 4 Jul 2008 21:30:50 +0000 (UTC) (envelope-from kian.mohageri@gmail.com) Received: from yx-out-2324.google.com (yx-out-2324.google.com [74.125.44.30]) by mx1.freebsd.org (Postfix) with ESMTP id 2677B8FC16 for ; Fri, 4 Jul 2008 21:30:49 +0000 (UTC) (envelope-from kian.mohageri@gmail.com) Received: by yx-out-2324.google.com with SMTP id 8so336044yxb.13 for ; Fri, 04 Jul 2008 14:30:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=6pXem0cLxaszFpgTAhj7jAyaMMKuIqyurj4g7QOQ6pg=; b=gX0akhlXbmduCIy8Zi5jUuSlb5E7/zzjtYnRQCljoh0ISXTk3EXL3/O7x0+WywiArv KBch6xH2nPku0sB6uv5Nh6LWzQS9HxDVAHJm6jd731r4oBXTje7fgVnfJWSohIjcr/iW rdd0syFIPLeO22/6UO+s1q2Ab68x1rltPhEXA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=YdFJY7oeCyzYRRB+wq/zgWKyyLqvIDVlo2GXRhctegXpXI2+gLs72DGUcNiq1E556T FAONduwyjX1+QgVzUMPpozsTMFBSe5wJMlXW75HAzVRl5+DpqVzQjhakr0mQfTpfyGur R9ermhvzPpOTgeHNT0GRzgFDglHHyPQjc9B8w= Received: by 10.150.153.3 with SMTP id a3mr140529ybe.84.1215207048951; Fri, 04 Jul 2008 14:30:48 -0700 (PDT) Received: by 10.151.101.9 with HTTP; Fri, 4 Jul 2008 14:30:48 -0700 (PDT) Message-ID: Date: Fri, 4 Jul 2008 14:30:48 -0700 From: "Kian Mohageri" To: "Jeremy Chadwick" In-Reply-To: <20080704113213.GA13586@eos.sc1.parodius.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: base64 Content-Disposition: inline References: <678A03F5-5E8A-4CF6-90DF-AA9A4F30FBE1@stromnet.se> <1211037564.6326.27.camel@porksoda> <679DB462-75D6-45CC-949C-1BE8E12C22CD@stromnet.se> <482FD877.6050707@infracaninophile.co.uk> <20080703003955.859BCF180C0@mx.npubs.com> <20080704113213.GA13586@eos.sc1.parodius.com> Cc: freebsd-stable , stef@memberwebs.com, freebsd-net@freebsd.org, Matthew Seaman , freebsd-pf@freebsd.org, Alex Trull Subject: Re: connect(): Operation not permitted X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 04 Jul 2008 21:30:50 -0000 T24gRnJpLCBKdWwgNCwgMjAwOCBhdCA0OjMyIEFNLCBKZXJlbXkgQ2hhZHdpY2sgPGtvaXRzdUBm cmVlYnNkLm9yZz4gd3JvdGU6Cj4gT24gVGh1LCBKdWwgMDMsIDIwMDggYXQgMDg6NTU6MjFBTSAt MDcwMCwgS2lhbiBNb2hhZ2VyaSB3cm90ZToKPj4gT24gV2VkLCBKdWwgMiwgMjAwOCBhdCA1OjM5 IFBNLCBTdGVmIDxzdGVmLWxpc3RAbWVtYmVyd2Vicy5jb20+IHdyb3RlOgo+PiA+IEtpYW4gTW9o YWdlcmkgd3JvdGU6Cj4+ID4+IE9uIFN1biwgTWF5IDE4LCAyMDA4IGF0IDM6MzMgQU0sIEpvaGFu IFN0csO2bSA8am9oYW5Ac3Ryb21uZXQuc2U+IHdyb3RlOgo+PiA+Pj4gT24gTWF5IDE4LCAyMDA4 LCBhdCA5OjE5IEFNLCBNYXR0aGV3IFNlYW1hbiB3cm90ZToKPj4gPj4+Cj4+ID4+Pj4gSm9oYW4g U3Ryw7ZtIHdyb3RlOgo+PiA+Pj4+Cj4+ID4+Pj4+IGRyb3AgYWxsIHRyYWZmaWMpPyBBIGNoZWNr IHdpdGggcGZjdGwgLXZzciByZXZlYWxzIHRoYXQgdGhlIGFjdHVhbCBydWxlCj4+ID4+Pj4+IGlu c2VydGVkIGlzICJwYXNzIG9uIGxvMCBpbmV0IGZyb20gMTIzLjEyMy4xMjMuMTIzIHRvIDEyMy4x MjMuMTIzLjEyMyBmbGFncwo+PiA+Pj4+PiBTL1NBIGtlZXAgc3RhdGUiLiBXaGVyZSBkaWQgdGhh dCAia2VlcCBzdGF0ZSIgY29tZSBmcm9tPwo+PiA+Pj4+ICdmbGFncyBTL1NBIGtlZXAgc3RhdGUn IGlzIHRoZSBkZWZhdWx0IG5vdyBmb3IgdGNwIGZpbHRlciBydWxlcyAtLSB0aGF0Cj4+ID4+Pj4g d2FzIG5ldyBpbiA3LjAgcmVmbGVjdGluZyB0aGUgdXBzdHJlYW0gY2hhbmdlcyBtYWRlIGJldHdl ZW4gdGhlIDQuMCBhbmQKPj4gPj4+PiA0LjEKPj4gPj4+PiByZWxlYXNlcyBvZiBPcGVuQlNELiAg SWYgeW91IHdhbnQgYSBzdGF0ZWxlc3MgcnVsZSwgYXBwZW5kICdubyBzdGF0ZScuCj4+ID4+Pj4K Pj4gPj4+PiBodHRwOi8vd3d3Lm9wZW5ic2Qub3JnL2ZhcS9wZi9maWx0ZXIuaHRtbCNzdGF0ZQo+ PiA+Pj4gVGhhbmtzISBJIHdhcyBhY3R1YWxseSBsb29raW5nIGFyb3VuZCBpbiB0aGUgcGYuY29u ZiBtYW5wYWdlIGJ1dCBmYWlsZWQgdG8KPj4gPj4+IGZpbmQgaXQgeWVzdGVyZGF5LCBidXQgbG9v a2luZyBjbG9zZXIgdG9kYXkgSSBub3cgc2F3IGl0Lgo+PiA+Pj4gQXBwbGllZCB0aGUgbm8gc3Rh dGUgKGFuZCBxdWljaykgdG8gdGhlIHJ1bGUsIGFuZCBub3cgbm8gc3RhdGUgaXMgY3JlYXRlZC4K Pj4gPj4+IEFuZCB0aGUgcHJvYmxlbSBJIGhhZCBpbiB0aGUgZmlyc3QgcGxhY2Ugc2VlbXMgdG8g aGF2ZSBiZWVuIHJlc29sdmVkIHRvbwo+PiA+Pj4gbm93LCBldmVuIHRob3VnaCBpdCBkaWRuJ3Qg bG9vayBsaWtlIGEgc3RhdGUgcHJvYmxlbS4uIChzdGFydGVkIHRvIGRlbnkgbmV3Cj4+ID4+PiBj b25uZWN0aW9ucyBtdWNoIGVhcmxpZXIgdGhhbiB0aGUgc3RhdGVzIHdhcyBmdWxsLCBhbHRvdWdo IG1heWJlZSBpIHdhc250Cj4+ID4+PiBsb29raW5nIGZvciB1cGRhdGVzIGZhc3QgZW5vdWdoIG9y IHNvbWV0aGluZykuCj4+ID4+Pgo+PiA+Pgo+PiA+PiBJJ2QgYmUgd2lsbGluZyB0byBiZXQgaXQn cyBiZWNhdXNlIHlvdSdyZSByZXVzaW5nIHRoZSBzb3VyY2UgcG9ydCBvbiBhCj4+ID4+IG5ldyBj b25uZWN0aW9uIGJlZm9yZSB0aGUgb2xkIHN0YXRlIGV4cGlyZXMuCj4+ID4+Cj4+ID4+IFlvdSds bCBrbm93IGlmIHlvdSBjaGVjayB0aGUgc3RhdGUtbWlzbWF0Y2ggY291bnRlci4KPj4gPj4KPj4g Pj4gQW55d2F5LCBnbGFkIHlvdSBmb3VuZCBhIHJlc29sdXRpb24uCj4+ID4KPj4gPiBJJ3ZlIGJl ZW4gZXhwZXJpZW5jaW5nIHRoaXMgIk9wZXJhdGlvbiBub3QgcGVybWl0dGVkIiB0b28uIEkndmUg YmVlbgo+PiA+IHRyeWluZyB0byB0cmFjayBkb3duIHRoZSBwcm9ibGVtIGZvciBtYW55IG1vbnRo cywgYnV0IGR1ZSB0byB0aGUKPj4gPiBjb21wbGV4aXR5IG9mIG15IGZpcmV3YWxscyAoc2NvcmVz IG9mIGphaWxzIGVhY2ggd2l0aCBzY29yZXMgb2YgcnVsZXMpLAo+PiA+IEkgd2Fzbid0IGJyYXZl IGVub3VnaCB0byBhc2sgZm9yIGhlbHAgOikKPj4gPgo+PiA+IEFzIGEgd29yayBhcm91bmQgd2Ug c3RhcnRlZCBjcmVhdGluZyBydWxlcyB3aXRob3V0IHN0YXRlLCB3aGVuZXZlciB3ZQo+PiA+IHdv dWxkIHJ1biBpbnRvIHRoZSBwcm9ibGVtLgo+PiA+Cj4+ID4gVGhhbmtzIGZvciB0aGUgcG9pbnRl ciBhYm91dCBzdGF0ZS1taXNtYXRjaC4gVGhlIHN0YXRlLW1pc21hdGNoIGNvdW50ZXIKPj4gPiBk b2VzIGlzIGluIGZhY3QgaGlnaCBpbiBteSBjYXNlIChzZWUgYmVsb3cpLiBIb3cgd291bGQgSSBn byBhYm91dAo+PiA+IGdldHRpbmcgdGhlIHBmIHN0YXRlIHRpbWVvdXQgYW5kIHRoZSByZXVzZSBv ZiBwb3J0cyBmb3Igb3V0Ym91bmQKPj4gPiBjb25uZWN0aW9ucyB0byBtYXRjaD8gT3IgaXMgdGhp cyBhbiBpbnRyYWN0YWJsZSBwcm9ibGVtLCB0aGF0IGp1c3QgbmVlZHMKPj4gPiB0byBiZSB3b3Jr ZWQgYXJvdW5kPwo+Pgo+PiBNYWtlIHN1cmUgeW91ciBzdGF0ZS1taXNtYXRjaCBjb3VudGVyIGlz IGluY3JlYXNpbmcgYXQgdGhlIHNhbWUgdGltZXMKPj4geW91IGV4cGVyaWVuY2UgdGhlIHByb2Js ZW0gKGFuZCBpc24ndCBqdXN0IGhpZ2ggZnJvbSBzb21lIHVucmVsYXRlZAo+PiBpc3N1ZSkuCj4+ Cj4+IEEgc2ltaWxhci9yZWxhdGVkIHByb2JsZW0gd2FzIGFkZHJlc3NlZCBpbiBPcGVuQlNEIDQu Mwo+PiAoaHR0cDovL3d3dy5vcGVuYnNkLm9yZy9wbHVzNDMuaHRtbCkuCj4+Cj4+ICAgKiBJbiBw Zig0KSwgYWxsb3cgc3RhdGUgcmV1c2UgaWYgYm90aCBzaWRlcyBhcmUgaW4gRklOX1dBSVRfMiBh bmQgYQo+PiBuZXcgU1lOIGFycml2ZXMuCj4+Cj4+IEknbSBub3Qgc3VyZSBpZiBpdCdzIGJlZW4g aW1wb3J0ZWQgeWV0LiAgSWYgbm90LCB5b3UgY291bGQgdHJ5IHR1bmluZwo+PiB5b3VyIHRpbWVv dXQgdmFsdWVzIChzZWUgcGYuY29uZig1KSkuCj4+Cj4+IFRoZSBzcGVjaWZpYyBpc3N1ZSBJIHdh cyBleHBlcmllbmNlZCB3YXMgc29sdmVkIGJ5IHNob3J0ZW5pbmcKPj4gdGNwLmNsb3NlZCwgSUlS Qy4gIEl0J3MgYmVlbiBhIHdoaWxlIHRob3VnaC4KPgo+IFdoZW4gYWRtaW5pc3RyYXRvcnMgc2Vl IHN0YXRlLW1pc21hdGNoIGluY3JlYXNpbmcsIHRoZXkgZ2V0IGNvbmNlcm5lZC4KPiBUaGUgY29t bW9uIHNjYXBlZ29hdCBpcyB0Y3AuY2xvc2VkLCB3aGljaCBwZW9wbGUgZG9uJ3QgZXZlbiBib3Ro ZXIgdG8KPiBkZXNjcmliZSAocGYgaGFzIGFuIGludGVybmFsIHZhbHVlIG9mIDEwIHNlY29uZHMg YXBwbGllZCB0byB0aGF0IHZhbHVlLAo+IGUuZy4gdGNwLmNsb3NlZD01IG1lYW5zIDE1IHNlY29u ZHMpLgo+Cj4gWW91IGNhbiBzZXQgdGNwLmNsb3NlZCBhcyBsb3cgYXMgeW91IHdhbnQsIGJ1dCBj aGFuY2VzIGFyZSByYW5kb20KPiBJbnRlcm5ldCB1c2VycyB3aWxsIGhhdmUgZXF1aXBtZW50IHdp dGggSVAgc3RhY2tzIHRoYXQgcmUtdXNlIG91dGJvdW5kCj4gc29ja2V0cyB3aGljaCBoYXZlbid0 IGZ1bGx5IGNsb3NlZCBkb3duIHdpdGhpbiB0aGUgYWZvcmVtZW50aW9uZWQKPiBpbnRlcnZhbC4g IHBmIGNhbm5vdCBmaXggdGhpcy4KPgo+IEZvciBleGFtcGxlLCBvbiBvdXIgcHJvZHVjdGlvbi9o b3N0aW5nIHN5c3RlbXMsIHdlIHNlZSBzdGF0ZS1taXNtYXRjaAo+IGluY3JlYXNlIGZhaXJseSBv ZnRlbi4gIEkganVzdCBwZmN0bCAtRiBpbmZvJ2Qgb3VyIG1haW4gd2Vic2VydmVyLCBhbmQKPiB3 aXRoaW4gYWJvdXQgMTUgbWludXRlcywgc3RhdGUtbWlzbWF0Y2ggd2FzIHVwIHRvIDIyLiAgV2Ug dXNlIHRjcC5jbG9zZWQKPiBvZiA1ICh3aGljaCBtZWFucyAxNSBzZWNvbmRzKS4KPgo+IFdvcmth cm91bmRzIHN1Y2ggYXMgIm5vIHN0YXRlIiBzdWZmaWNlLCBidXQgaWYgeW91IHVzZSByZHIgcnVs ZXMsIHlvdQo+IE1VU1QgdHJhY2sgc3RhdGUsIHdoaWNoIG1lYW5zIHRoZXJlJ3Mgbm8gd2F5IG9m IHdpbm5pbmcgaW4gdGhhdCBjYXNlLgo+IEZvciBzYWtlIG9mIGV4YW1wbGUsIE9wZW5CU0Qgc3Bh bWQgcmVxdWlyZXMgdGhlIHVzZSBvZiByZHIgcnVsZXMuCj4KPiBBZG1pbmlzdHJhdG9ycyB0aGVu IGFzayAzIHF1ZXN0aW9uczoKPgoKRm9yIHRoZSBzYWtlIG9mIGEgaGVscGZ1bCBhcmNoaXZlLi4u Cgo+IDEpIEhvdyBkbyBJIGRldGVybWluZSB3aGV0aGVyIG9yIG5vdCBzdGF0ZS1taXNtYXRjaCBp bmNyZWFzaW5nIGlzIGEKPiAgIHNpZ24gb2YgYmFkIHRoaW5ncywgb3IgZHVlIHRvIHBlb3BsZXMn IGJyb2tlbiBJUCBzdGFja3MsCgpZb3UgY2FuJ3QuICBPbmx5IHdheSB5b3Uga25vdyBpcyBwcm9i YWJseSB3aGVuIHBlb3BsZSBjb21wbGFpbiwgb3IgeW91Cm5vdGljZSBzY3JpcHRzL3BhZ2UgbG9h ZHMgZmFpbGluZy4KCj4gMikgV2hhdCBoYXBwZW5zIHRvIHBhY2tldHMgd2hpY2ggY2F1c2Ugc3Rh dGUtbWlzbWF0Y2ggdG8gaW5jcmVtZW50LAo+ICAgZS5nLiBhcmUgdGhleSBibG9ja2VkLCBwYXNz ZWQsIG9yIHdoYXQ/CgpEcm9wcGVkLiAgSW4gdGhlIGNhc2Ugb2YgYSBzdGF0ZS1taXNtYXRjaCBk dXJpbmcgVENQIGhhbmRzaGFrZSwgYW4gUlNUCmlzIHNlbnQuICBUaGF0J3Mgd2h5IHRoZSBmYWls dXJlIGhhcHBlbnMgaW1tZWRpYXRlbHkuCgo+IDMpIFdoeSBpc24ndCBzdGF0ZS1taXNtYXRjaCBk ZXNjcmliZWQgaW4gZGV0YWlsIGluIHRoZSBkb2N1bWVudGF0aW9uPwo+CgpHb29kIHF1ZXN0aW9u LiAgSSBndWVzcyBiZWNhdXNlIGl0IHdvdWxkIGJlIGRpZmZpY3VsdCB0byBkb2N1bWVudCBhbGwK b2YgdGhlIHJlYXNvbnMgYSBzdGF0ZSB3b3VsZG4ndCBtYXRjaC4gIEl0IHdvdWxkIGJlIGVhc2ll ciB0byBzaW1wbHkKZG9jdW1lbnQgd2hhdCBhIHN0YXRlIF9pc18sIGJ1dCB0aGF0J3MgYWxyZWFk eSBpbiB0aGUgYXJjaGl2ZXMuCgotS2lhbgo= From owner-freebsd-net@FreeBSD.ORG Sat Jul 5 16:00:18 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2E0F7106564A for ; Sat, 5 Jul 2008 16:00:18 +0000 (UTC) (envelope-from mnd999@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.187]) by mx1.freebsd.org (Postfix) with ESMTP id ACD0D8FC1D for ; Sat, 5 Jul 2008 16:00:17 +0000 (UTC) (envelope-from mnd999@gmail.com) Received: by nf-out-0910.google.com with SMTP id h3so547036nfh.33 for ; Sat, 05 Jul 2008 09:00:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:from:to :content-type:content-transfer-encoding:mime-version:subject:date :x-mailer; bh=p/1YdnCE+FN84e6kPZSHRoBxN2EtErjaLJalNp+b6EA=; b=j+ctbzAOjnVz34DgyHVfIY19f62qswXEltdOG4ks7xaI7HO2Z833UYOLjyBp0VXIut I2XXxqT+jRHmttnpg8+QWZQNjvNwGGhCFafAaPKhlmk2vbpnepWyY6ufYio32I63C8Vx FvKirpte6Zi8+Q4AmSuUrpXclQjt6x1Hm5EMw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:from:to:content-type:content-transfer-encoding :mime-version:subject:date:x-mailer; b=Uj83v0HFhY4/Js00gaOJbWu34NCdayBMao0cWahAC43zxnBJEl2o2InbLL3J+DQVA4 7aUKZhKafQGh2n7TZ4vtnlo9hmgBMTB3JZSb9fsV/sU4tCbnD4jSQd55gvUr+jWPsCP/ FOy+NNKkXUhyWhkxUDdPW56g8XVaFInx+oVvk= Received: by 10.210.90.8 with SMTP id n8mr1317516ebb.191.1215271936489; Sat, 05 Jul 2008 08:32:16 -0700 (PDT) Received: from ?192.168.0.222? ( [90.193.89.181]) by mx.google.com with ESMTPS id t2sm2642225gve.9.2008.07.05.08.32.13 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sat, 05 Jul 2008 08:32:14 -0700 (PDT) Message-Id: <6C1A742B-AB45-4E72-8BA9-F333F607A55E@gmail.com> From: Mark Dixon To: freebsd-net@freebsd.org, freebsd-stable@freebsd.org Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v926) Date: Sat, 5 Jul 2008 16:32:10 +0100 X-Mailer: Apple Mail (2.926) Cc: Subject: Net crash in ath on FREEBSD-7 STABLE X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Jul 2008 16:00:18 -0000 Hi, I get the following when I run portupgrade and it tries to hit the network for a package on stable. It looks like something has been broken in ath? Seeing as I don't update very often it could have been around for a while. Kernel is GENERIC. Modules are kernel, snd_ich, sound, aio and linux. If anyone wants to look at this and needs more info, let me know. Mark GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 2; apic id = 02 fault virtual address = 0x0 fault code = supervisor read data, page not present instruction pointer = 0x8:0xffffffff8025e2f2 stack pointer = 0x10:0xffffffffaec9e630 frame pointer = 0x10:0xffffff00018fa100 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 16516 (fetch) trap number = 12 panic: page fault cpuid = 2 Uptime: 2m56s Physical memory: 2034 MB Dumping 251 MB: (CTRL-C to abort) 236 220 (CTRL-C to abort) 204 188 172 156 140 124 108 92 76 60 (CTRL-C to abort) 44 28 12 Reading symbols from /boot/kernel/snd_ich.ko...Reading symbols from / boot/kernel/snd_ich.ko.symbols...done. done. Loaded symbols for /boot/kernel/snd_ich.ko Reading symbols from /boot/kernel/sound.ko...Reading symbols from / boot/kernel/sound.ko.symbols...done. done. Loaded symbols for /boot/kernel/sound.ko Reading symbols from /boot/kernel/aio.ko...Reading symbols from /boot/ kernel/aio.ko.symbols...done. done. Loaded symbols for /boot/kernel/aio.ko Reading symbols from /boot/kernel/linux.ko...Reading symbols from / boot/kernel/linux.ko.symbols...done. done. Loaded symbols for /boot/kernel/linux.ko #0 doadump () at pcpu.h:194 194 pcpu.h: No such file or directory. in pcpu.h (kgdb) where #0 doadump () at pcpu.h:194 #1 0x0000000000000004 in ?? () #2 0xffffffff804964a9 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 #3 0xffffffff804968ad in panic (fmt=0x104
) at /usr/src/sys/kern/kern_shutdown.c:572 #4 0xffffffff8075ff74 in trap_fatal (frame=0xffffff00044d66a0, eva=18446742974267905128) at /usr/src/sys/amd64/amd64/trap.c:724 #5 0xffffffff80760345 in trap_pfault (frame=0xffffffffaec9e580, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:641 #6 0xffffffff80760c88 in trap (frame=0xffffffffaec9e580) at /usr/src/sys/amd64/amd64/trap.c:410 #7 0xffffffff8074664e in calltrap () at /usr/src/sys/amd64/amd64/exception.S:169 #8 0xffffffff8025e2f2 in ath_start (ifp=0xffffff00011cd800) at /usr/src/sys/dev/ath/if_ath.c:1747 #9 0xffffffff8052fab6 in ether_output_frame (ifp=0xffffff00011cd800, m=0xffffff000421eb00) at /usr/src/sys/net/if_ethersubr.c:405 #10 0xffffffff805300e2 in ether_output (ifp=0xffffff00011cd800, m=0xffffff000421eb00, dst=Variable "dst" is not available. ) at /usr/src/sys/net/if_ethersubr.c:374 #11 0xffffffff805755df in ip_output (m=0xffffff000421eb00, opt=Variable "opt" is not available. ) at /usr/src/sys/netinet/ip_output.c:551 #12 0xffffffff805ce48c in tcp_output (tp=0xffffff0004216b60) at /usr/src/sys/netinet/tcp_output.c:1135 #13 0xffffffff805d880d in tcp_usr_rcvd (so=Variable "so" is not available. ) at /usr/src/sys/netinet/tcp_usrreq.c:738 #14 0xffffffff804ef5cf in soreceive_generic (so=0xffffff001040c570, psa=0x0, uio=0xffffffffaec9eb00, mp0=Variable "mp0" is not available. ) at /usr/src/sys/kern/uipc_socket.c:1825 #15 0xffffffff804cee1d in dofileread (td=0xffffff00044d66a0, fd=3, fp=0xffffff00047e4168, auio=0xffffffffaec9eb00, offset=Variable "offset" is not available. ) at file.h:242 #16 0xffffffff804cf18e in kern_readv (td=0xffffff00044d66a0, fd=3, auio=0xffffffffaec9eb00) at /usr/src/sys/kern/sys_generic.c:192 #17 0xffffffff804cf27c in read (td=0xffffffff80e62000, uap=0xffffff00044d66a0) at /usr/src/sys/kern/sys_generic.c:108 #18 0xffffffff807605c7 in syscall (frame=0xffffffffaec9ec70) at /usr/src/sys/amd64/amd64/trap.c:852 #19 0xffffffff8074685b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:290 #20 0x0000000800c0297c in ?? () Previous frame inner to this frame (corrupt stack?) From owner-freebsd-net@FreeBSD.ORG Sat Jul 5 16:57:17 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DA7BD1065674 for ; Sat, 5 Jul 2008 16:57:17 +0000 (UTC) (envelope-from cswiger@mac.com) Received: from pi.codefab.com (pi.codefab.com [199.103.21.227]) by mx1.freebsd.org (Postfix) with ESMTP id ABE358FC0A for ; Sat, 5 Jul 2008 16:57:17 +0000 (UTC) (envelope-from cswiger@mac.com) Received: from localhost (localhost [127.0.0.1]) by pi.codefab.com (Postfix) with ESMTP id 1E7485CDB for ; Sat, 5 Jul 2008 12:39:28 -0400 (EDT) X-Virus-Scanned: amavisd-new+ClamAV at codefab.com Received: from [192.168.1.3] (pool-96-224-166-247.nycmny.east.verizon.net [96.224.166.247]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by pi.codefab.com (Postfix) with ESMTPSA id 8A05C5C5F for ; Sat, 5 Jul 2008 12:39:26 -0400 (EDT) Message-ID: <486FA3B1.6040806@mac.com> Date: Sat, 05 Jul 2008 12:39:13 -0400 From: Chuck Swiger User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: freebsd-net@freebsd.org References: <20080703115243.GR29380@server.vk2pj.dyndns.org> <20080703190513.5CD5D5B4C@mail.bitblocks.com> <20080704023244.GH29305@verio.net> In-Reply-To: <20080704023244.GH29305@verio.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: arplookup x.x.x.x failed: host is not on local network X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Jul 2008 16:57:17 -0000 David DeSimone wrote: [ ... ] > Again, I did see these messages in my environment, but in my case, the > error was correct: The IP *was not* on the local network. The reason > being that we had multiple subnets configured on the same broadcast > domain, so the BSD box could indeed hear ARP for subnets it did not know > about. I don't know why the box feels moved to complain about this, > however. I would think it should not care. It's good practice for machines intended to be on different subnets to be in different collision domains. Seeing traffic to or from the wrong network should be considered a potential "red flag", warning that there might be a network misconfiguration that could compromise security. In particular, if you want to securely host a bunch of client machines, setting them up on individual /30 subnets using a multiport firewall or a BSD box with a couple of 4-port NIC cards, rather than a switch, is a good idea. While this situation is something which is supposedly well-suited for VLANs, in practice most switches cannot be relied upon to actually prevent traffic from leaking outside of the specified VLAN. This is more common for ARP traffic, which is sent to the all-ones MAC and may well get forwarded to all ports regardless of VLAN tagging, particularly if the switch is under load and has switched to some kind of "fast forwarding" mode or if it tends to consider all ports trunk ports by default or via dubious autolearning algorithms.... Regards, -- -Chuck From owner-freebsd-net@FreeBSD.ORG Sat Jul 5 19:02:53 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 076171065680 for ; Sat, 5 Jul 2008 19:02:53 +0000 (UTC) (envelope-from spawk@acm.poly.edu) Received: from acm.poly.edu (acm.poly.edu [128.238.9.200]) by mx1.freebsd.org (Postfix) with ESMTP id 9B8208FC14 for ; Sat, 5 Jul 2008 19:02:51 +0000 (UTC) (envelope-from spawk@acm.poly.edu) Received: (qmail 47721 invoked from network); 5 Jul 2008 19:02:51 -0000 Received: from unknown (HELO ?192.168.0.2?) (spawk@69.123.45.64) by acm.poly.edu with AES256-SHA encrypted SMTP; 5 Jul 2008 19:02:51 -0000 Message-ID: <486FC54B.3060706@acm.poly.edu> Date: Sat, 05 Jul 2008 15:02:35 -0400 From: Boris Kochergin User-Agent: Thunderbird 2.0.0.9 (X11/20071208) MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: One-liner for setting IPv6 address and IPv4 endpoints on gif interface? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Jul 2008 19:02:53 -0000 Hi, list. I set up an IPv6-over-IPv4 tunnel from Hurricane Electric using a gif(4) interface using the commands: ifconfig gif0 tunnel [source IPv4] [destination IPv4] ifconfig gif0 inet6 [source IPv6] [destination IPv6] prefixlen 128 route -n add -inet6 default [destination IPv6] I'm wondering whether there's a one-liner for executing the first two commands, or some non-one-liner way of making it happen through /etc/rc.conf. Thanks. -Boris From owner-freebsd-net@FreeBSD.ORG Sat Jul 5 19:42:45 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 82E411065677 for ; Sat, 5 Jul 2008 19:42:45 +0000 (UTC) (envelope-from yuri.pankov@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.188]) by mx1.freebsd.org (Postfix) with ESMTP id 013CF8FC0C for ; Sat, 5 Jul 2008 19:42:44 +0000 (UTC) (envelope-from yuri.pankov@gmail.com) Received: by nf-out-0910.google.com with SMTP id h3so560473nfh.33 for ; Sat, 05 Jul 2008 12:42:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:received:received :x-authentication-warning:date:from:to:cc:subject:message-id :references:mime-version:content-type:content-disposition :in-reply-to:user-agent; bh=Jta9sAzwUOrsaUg4MNawvL11/bM+tzfcvLUtx392lL4=; b=nz0dDKLCdU7JQxn5KLT5THJoBSgr2JdT58HfX1iIEK1hCH4pKV0+e75BwA1vUA8kyR MiLdo+Yl5mP2JgwtJtdkCSXr81dqzHkSc9hlniNR2LCPOCwuSnJnAbBIAnpHiswCUzuO TkElcGKOTph/CBCHB6+bCCeGL5eE1mBQByQLc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=x-authentication-warning:date:from:to:cc:subject:message-id :references:mime-version:content-type:content-disposition :in-reply-to:user-agent; b=LzsVFt4k18zDwfaO+JD5jyg3WZsKeZT5wh2cS4raAZ7y4WcYVwYrN6Q384o2P4gdrw cv0rOGjLWd0NhPOdWdP/B7OsHu2k2naDwZ1xavskhEszbgQmg3H44xtlAK9ZTpggm+7x 6Mt6GU01W0xAYh6GjYaj97r0Uxmbbzg2p/brc= Received: by 10.210.20.17 with SMTP id 17mr1494516ebt.21.1215285497923; Sat, 05 Jul 2008 12:18:17 -0700 (PDT) Received: from darklight.homeunix.org ( [85.172.52.13]) by mx.google.com with ESMTPS id p10sm3009781gvf.7.2008.07.05.12.18.15 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sat, 05 Jul 2008 12:18:16 -0700 (PDT) Received: from darklight.homeunix.org (darklight.homeunix.org [127.0.0.1]) by darklight.homeunix.org (8.14.2/8.14.2) with ESMTP id m65JIClK058450; Sat, 5 Jul 2008 23:18:12 +0400 (MSD) (envelope-from yuri.pankov@gmail.com) Received: (from yuri@localhost) by darklight.homeunix.org (8.14.2/8.14.2/Submit) id m65JIBok058449; Sat, 5 Jul 2008 23:18:11 +0400 (MSD) (envelope-from yuri.pankov@gmail.com) X-Authentication-Warning: darklight.homeunix.org: yuri set sender to yuri.pankov@gmail.com using -f Date: Sat, 5 Jul 2008 23:18:11 +0400 From: Yuri Pankov To: Boris Kochergin Message-ID: <20080705191811.GA58433@darklight.homeunix.org> References: <486FC54B.3060706@acm.poly.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <486FC54B.3060706@acm.poly.edu> User-Agent: Mutt/1.5.18 (2008-05-17) Cc: freebsd-net@freebsd.org Subject: Re: One-liner for setting IPv6 address and IPv4 endpoints on gif interface? X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Jul 2008 19:42:45 -0000 On Sat, Jul 05, 2008 at 03:02:35PM -0400, Boris Kochergin wrote: > Hi, list. I set up an IPv6-over-IPv4 tunnel from Hurricane Electric > using a gif(4) interface using the commands: > > ifconfig gif0 tunnel [source IPv4] [destination IPv4] > ifconfig gif0 inet6 [source IPv6] [destination IPv6] prefixlen 128 > route -n add -inet6 default [destination IPv6] > > I'm wondering whether there's a one-liner for executing the first two > commands, or some non-one-liner way of making it happen through > /etc/rc.conf. Thanks. > > -Boris Not sure about one-liner, but that's what I'm using in rc.conf (Hurricane Electric's tunnelbroker.net tunnel): gif_interfaces="gif0" gifconfig_gif0="src_ipv4 dst_ipv4" ipv6_enable="YES" ipv6_ifconfig_gif0="src_ipv6 dst_ipv6 prefixlen 128" ipv6_defaultrouter="dst_ipv6" HTH, Yuri From owner-freebsd-net@FreeBSD.ORG Sat Jul 5 22:10:09 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D98BB1065673 for ; Sat, 5 Jul 2008 22:10:09 +0000 (UTC) (envelope-from if@xip.at) Received: from chile.gbit.at (ns1.xip.at [193.239.188.99]) by mx1.freebsd.org (Postfix) with ESMTP id 0134D8FC12 for ; Sat, 5 Jul 2008 22:10:08 +0000 (UTC) (envelope-from if@xip.at) Received: (qmail 27957 invoked from network); 6 Jul 2008 00:10:06 +0200 Received: from unknown (HELO filebunker.xip.at) (86.59.10.180) by chile.gbit.at with (DHE-RSA-AES256-SHA encrypted) SMTP; 6 Jul 2008 00:10:06 +0200 Date: Sun, 6 Jul 2008 00:10:05 +0200 (CEST) From: Ingo Flaschberger To: Paul In-Reply-To: <486E65E6.3060301@gtcomm.net> Message-ID: References: <4867420D.7090406@gtcomm.net> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <20080701033117.GH83626@cdnetworks.co.kr> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <486B4F11.6040906@gtcomm.net> <486BC7F5.5070604@gtcomm.net> <20080703160540.W6369@delplex.bde.org> <486C7F93.7010308@gtcomm.net> <20080703195521.O6973@delplex.bde.org> <486D35A0.4000302@gtcomm.net> <486DF1A3.9000409@gtcomm.net> <486E65E6.3060301@gtcomm.net> User-Agent: Alpine 1.10 (LFD 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Net Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Jul 2008 22:10:10 -0000 Dear Paul, > I tried all of this :/ still, 256/512 descriptors seem to work the best. > Happy to let you log into the machine and fiddle around if you want :) yes, but I'm shure I will also not be able to achieve much more pps. As it seems that you hit hardware-software-level-barriers, my only idea is to test dragonfly bsd, which seems to have less software overhead. I don't think you will be able to route 64byte packets at 1gbit wirespeed (2Mpps) with a current x86 platform. I hoped to reach 1Mpps with the hardware I mentioned some mails before, but 2Mpps is far far away. Currently I get 160kpps via pci-32mbit-33mhz-1,2ghz mobile pentium. Perhaps you have some better luck at some different hardware systems (ppc, mips, ..?) or use freebsd only for routing-table-updates and special network-cards (netfpga) for real routing. Kind regards, Ingo Flaschberger From owner-freebsd-net@FreeBSD.ORG Sat Jul 5 23:08:50 2008 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 58DA1106564A for ; Sat, 5 Jul 2008 23:08:50 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from atlas.gtcomm.net (atlas.gtcomm.net [67.215.15.242]) by mx1.freebsd.org (Postfix) with ESMTP id 0CEFC8FC19 for ; Sat, 5 Jul 2008 23:08:49 +0000 (UTC) (envelope-from paul@gtcomm.net) Received: from c-76-108-179-28.hsd1.fl.comcast.net ([76.108.179.28] helo=[192.168.1.6]) by atlas.gtcomm.net with esmtpsa (TLSv1:AES256-SHA:256) (Exim 4.67) (envelope-from ) id 1KFGo8-0004cT-T1; Sat, 05 Jul 2008 19:04:45 -0400 Message-ID: <486FFF70.3090402@gtcomm.net> Date: Sat, 05 Jul 2008 19:10:40 -0400 From: Paul User-Agent: Thunderbird 2.0.0.14 (Windows/20080421) MIME-Version: 1.0 To: Bart Van Kerckhove References: <4867420D.7090406@gtcomm.net> <48699960.9070100@gtcomm.net><20080701033117.GH83626@cdnetworks.co.kr><4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net><486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net><486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net><486B4F11.6040906@gtcomm.net><486BC7F5.5070604@gtcomm.net><20080703160540.W6369@delplex.bde.org><486C7F93.7010308@gtcomm.net><20080703195521.O6973@delplex.bde.org><486D35A0.4000302@gtcomm.net><486DF1A3.9000409@gtcomm.net><486E65E6.3060301@gtcomm.net> <2d3001c8def1$f4309b90$020b000a@bartwrkstxp> In-Reply-To: <2d3001c8def1$f4309b90$020b000a@bartwrkstxp> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Net , Ingo Flaschberger Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 05 Jul 2008 23:08:50 -0000 ULE + PREEMPTION for non SMP no major differences with SMP with ULE/4BSD and preemption ON/OFF 32 bit UP test coming up with new cpu and I'm installing dragonfly sometime this weekend :] UP: 1mpps in one direction with no firewall/no routing table is not too bad, but 1mpps both directions is the goal here 700kpps with full bgp table in one direction is not too bad Ipfw needs a lot of work, barely gets 500kpps with no routing table with a few ipfw rules loaded.. that's horrible Linux barely takes a hit when you start loading iptables rules , but then again linux has a HUGE problem with routing random packet sources/ports .. grr My problem Is I need some box to do fast routing and some to do firewall.. :/ I'll have 32 bit 7-stable UP test with ipfw/routing table and then move on to dragonfly. I'll post the dragonfly results here as well as sign up for their mailing list. Bart Van Kerckhove wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Paul / Ingo, > >>> I tried all of this :/ still, 256/512 descriptors seem to work the >>> best. Happy to let you log into the machine and fiddle around if you >>> want :) >>> > I've been watching this thread closely, since I'm in a very similair > situation. > A few questions/remarks: > > Does ULE provide better performance than 4BSD for forwarding? > Did you try freebsd4 as well? This thread had a report about that quite > opposite to my own experiences, -4 seemed to be a lot faster at forwarding > than anything else I 've tried so far. > Obviously the thing I'm interested in is IMIX - and 64byte packets. > Does anyone have any benchmarks for DragonFly? I asked around on IRC, but > that nor google turned up any useful results. > > > >> I don't think you will be able to route 64byte packets at 1gbit >> wirespeed (2Mpps) with a current x86 platform. >> > Are there actual hardware related reasons this should not be possible, or > is this purely lack of dedicated work towards this goal? > > > >> Theres a "sun" used at quagga dev as bgp-route-server. >> http://quagga.net/route-server.php >> (but they don't answered my question regarding fw-performance). >> > > > the Quagga guys are running a sun T1000 (niagara 1) route server - I happen > to have the machine in my racks, > please let me know if you want to run some tests on it, I'm sure they won't > mind ;-) > It should also make a great testbed for SMP performance testing imho (and > they're pretty cheap these days) > Also, feel free to use me as a relay for your questions, they're not always > very reachable. > > > >> Perhaps you have some better luck at some different hardware systems >> (ppc, mips, ..?) or use freebsd only for routing-table-updates and >> special network-cards (netfpga) for real routing. >> > The netfpga site seems more or less dead - is this project still alive? > It does look like a very interesting idea, even though it's currently quite > linux-centric (and according to docs doesn't have VLAN nor ip6 support, the > former being quite a dealbreaker) > > Paul: I'm looking forward to the C2D 32bit benchmarks (maybe throw in a > freebsd4 and/or dragonfly bench if you can..) - appreciate the lots of > information you are providing us :) > > Met vriendelijke groet / With kind regards, > > Bart Van Kerckhove > http://friet.net/pgp.txt > > -----BEGIN PGP SIGNATURE----- > > iQA/AwUBSG/tMgoIFchBM0BKEQKUSQCcCJqsw2wtUX7HQi050HEDYX3WPuMAnjmi > eca31f7WQ/oXq9tJ8TEDN3CA > =YGYq > -----END PGP SIGNATURE----- > > >