From owner-freebsd-net@FreeBSD.ORG Sat Nov 11 10:06:48 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 30AFF16A403 for ; Sat, 11 Nov 2006 10:06:48 +0000 (UTC) (envelope-from misho@interbgc.com) Received: from mail.interbgc.com (mx01.cablebg.net [217.9.224.226]) by mx1.FreeBSD.org (Postfix) with SMTP id 69BFF43D4C for ; Sat, 11 Nov 2006 10:06:46 +0000 (GMT) (envelope-from misho@interbgc.com) Received: (qmail 31298 invoked from network); 11 Nov 2006 10:06:45 -0000 Received: from misho@interbgc.com by keeper.interbgc.com by uid 1002 with qmail-scanner-1.14 (uvscan: v4.2.40/v4374. spamassassin: 2.63. Clear:SA:0(0.0/8.0):. Processed in 1.908091 secs); 11 Nov 2006 10:06:45 -0000 X-Spam-Status: No, hits=0.0 required=8.0 Received: from 85-130-33-242.1698539.ddns.cablebg.net (HELO misho) (85.130.33.242) by mx01.interbgc.com with SMTP; 11 Nov 2006 10:06:43 -0000 Message-ID: <001701c70579$16b152d0$f2218255@misho> From: "Mihail Balikov" To: "Scott Long" References: <2a41acea0611081719h31be096eu614d2f2325aff511@mail.gmail.com><200611091536.kA9FaltD018819@lava.sentex.ca><45534E76.6020906@samsco.org><200611092200.kA9M0q1E020473@lava.sentex.ca><200611102004.kAAK4iO9027778@lava.sentex.ca><2a41acea0611101400w5b8cef40ob84ed6de181f3e2c@mail.gmail.com><200611102221.kAAML6ol028630@lava.sentex.ca> <455570D8.6070000@samsco.org> Date: Sat, 11 Nov 2006 12:06:43 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1807 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1807 Cc: freebsd-net , Jack Vogel Subject: Re: Proposed 6.2 em RELEASE patch X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Mihail Balikov List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Nov 2006 10:06:48 -0000 Our routers are with 2 em NICs, doing about 100kkps. Without kernel polling system become unstable, it seems that default interrupts moderation on em is 10000 intr/sec. I have made some modification in kernel to decrease packet drops when polling is enabled - modify clock routines to allow high clock rate only for polling (5000 polls/sec) - remove netisr_poll_more - loop in ether_poll() untill there's ready packets Small optimization in em(): - add CSUM_IP checksum offloading on transmit - call em_txeof if there's more than 32 busy packet - remove E1000_TXD_CMD_RS (Report Status) and check (TDH == adapter->oldest_used_tx_desc). This should reduce PCI overhead, but adds one more PCI read on every em_txoef() call. OS: FreeBSD 4.11, em() is almost up to date with HEAD. ----- Original Message ----- From: "Scott Long" To: "Mike Tancsa" Cc: "freebsd-net" ; ; "Jack Vogel" Sent: Saturday, November 11, 2006 8:42 AM Subject: Re: Proposed 6.2 em RELEASE patch > Mike Tancsa wrote: > > At 05:00 PM 11/10/2006, Jack Vogel wrote: > >> On 11/10/06, Mike Tancsa wrote: > >>> > >>> Some more tests. I tried again with what was committed to today's > >>> RELENG_6. I am guessing its pretty well the same patch. Polling is > >>> the only way to avoid livelock at a high pps rate. Does anyone know > >>> of any simple tools to measure end to end packet loss ? Polling will > >>> end up dropping some packets and I want to be able to compare. Same > >>> hardware from the previous post. > >> > >> The commit WAS the last patch I posted. SO, making sure I understood you, > >> you are saying that POLLING is doing better than FAST_INTR, or only > >> better than the legacy code that went in with my merge? > > > > Hi, > > The last set of tests I posted are ONLY with what is in today's > > RELENG_6-- i.e. the latest commit. I did a few variations on the > > driver-- first with > > #define EM_FAST_INTR 1 > > in if_em.c > > > > one without > > > > and one with polling in the kernel. > > > > With a decent packet rate passing through, the box will lockup. Not > > sure if I am just hitting the limits of the PCIe bus, or interrupt > > moderation is not kicking in, or this is a case of "Doctor, it hurts > > when I send a lot of packets through"... "Well, dont do that" > > > > Using polling prevents the lockup, but it will of course drop packets. > > This is for firewalls with a fairly high bandwidth rate, as well as I > > need it to be able to survive a decent DDoS attack. I am not looking > > for 1Mpps, but something more than 100Kpps > > > > ---Mike > > Hi, > > Thanks for all of the data. I know that a good amount of testing was > done with single stream stress tests, but it's not clear how much was > done with multiple streams prior to your efforts. So, I'm not terribly > surprised by your results. I'm still a bit unclear on the exact > topology of your setup, so if could explain it some more in private > email, I'd appreciate it. > > For the short term, I don't think that there is anything that can be > magically tweaked that will safely give better results. I know that > Gleb has some ideas on a fairly simple change for the non-INTR_FAST, > non-POLLING case, but I and several others worry that it's not robust > in the face of real-world network problems. > > For the long term, I have a number of ideas for improving both the RX > and TX paths in the driver. Some of it is specific to the if_em driver, > some involve improvements in the FFWD and PFIL_HOOKS code as well as the > driver. What will help me is if you can hook up a serial console to > your machine and see if it can be made to drop to the debugger while it > is under load and otherwise unresponsive. If you can, getting a process > dump might help confirm where each CPU is spending its time. > > Scott > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >