From owner-freebsd-net@FreeBSD.ORG Fri Jun 22 08:55:51 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E38E1065897; Fri, 22 Jun 2012 08:55:51 +0000 (UTC) (envelope-from venkatduvvuru.ml@gmail.com) Received: from mail-yx0-f182.google.com (mail-yx0-f182.google.com [209.85.213.182]) by mx1.freebsd.org (Postfix) with ESMTP id 31D698FC08; Fri, 22 Jun 2012 08:55:51 +0000 (UTC) Received: by yenl8 with SMTP id l8so1564993yen.13 for ; Fri, 22 Jun 2012 01:55:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=TiG4MmM/wdVnTqTGsU9r+M2skdKXosprbme5u8CyBus=; b=cJbPGfdHHBge6qD68PATfnFWLAtwv8g1NSJ5lOozAbfcm9VPs3DXCufAvIwuVj/lx5 KXBsHtvvY7oRJ/be1fSG7uleiNxuzIhFFZMMc9fnKl9XlmeHhcsVgfjdRWVBbN5BpS8j KwFFuO3+is7dR48qcT6pmKxW5o7xq47PA70luZt+qcS/2ZgwcCWBpwF33/6Umqehfcey 3hfpK65EweJMvUD0CucCgosT9suFAr6Pe4onMBORvDHNwJ3zr3GGpb6gaeiot77smcq9 /ZX2wvSfToq2CzO62uy6+mdkCZuY2ikxCqYF9xLnKSloiP0jlr2E7b3Ic6ZGpi+n2p1P 7e1g== MIME-Version: 1.0 Received: by 10.50.183.230 with SMTP id ep6mr752424igc.52.1340355350060; Fri, 22 Jun 2012 01:55:50 -0700 (PDT) Received: by 10.64.77.72 with HTTP; Fri, 22 Jun 2012 01:55:50 -0700 (PDT) In-Reply-To: <201206210947.30171.jhb@freebsd.org> References: <201206210816.22774.jhb@freebsd.org> <201206210947.30171.jhb@freebsd.org> Date: Fri, 22 Jun 2012 14:25:50 +0530 Message-ID: From: Venkat Duvvuru To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-net@freebsd.org Subject: Re: MSI-X limitation in freebsd 8.2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jun 2012 08:55:51 -0000 This setup has variety of NICs, Intel's, Emulex's and all are stock drivers. /Venkat On Thu, Jun 21, 2012 at 7:17 PM, John Baldwin wrote: > On Thursday, June 21, 2012 8:50:06 am Venkat Duvvuru wrote: > > While I'm able to ping to the machine..every command on the current > active > > session is sluggish infact doesn't complete, unable to open another ssh > > session..an already opened ssh session doesn't react which is running > > "systat -vmstat"... > > > > The only thing I could do is to successfully complete "top -P" command > > once..even top hung for the second time. > > Hmm, is this a stock FreeBSD driver or an out-of-tree NIC driver? > > > /Venkat > > > > On Thu, Jun 21, 2012 at 5:46 PM, John Baldwin wrote: > > > > > On Thursday, June 21, 2012 5:47:48 am Venkat Duvvuru wrote: > > > > John - Thanks for the reply. > > > > > > > > All the CPUs are ~100% idle. I don't see any interrupt storm on any > of > > > the > > > > irqs (vmstat -i). > > > > > > > > One observation I made is that I see messages like these in dmesg > > > > > > > > ===> mem > > > 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf1c000-0xfaf1ffff > > > > irq 40 at device 0.1 on pci6 > > > > > > > > Looking at the irq value I think it is the INTx irq range which > shouldn't > > > > have probably got allocated as the device is msix capable and there > are > > > > vectors allocated for these devices in the range (256-380). > > > > > > > > Could this be a problem? > > > > > > No, that line is output before the driver's attach routine is run, so > it > > > will > > > always show INTx IRQ value even if it isn't used. > > > > > > > The scenario where I am hitting this problem is a setup with 4 NICs, > each > > > > NIC with two ports and each port using up 4 msix vectors. The system > is > > > > fine till some ports are up but once I ifup the 5th port, the system > > > > becomes sluggish. > > > > > > > > I'm not sure whether all the 30 vectors are from a single cpu..I > don't > > > know > > > > how to get that information. > > > > > > Unfortunately there isn't an easy way. I have this gdb script which > can > > > display it from kgdb on x86: > > > > > > define irqs > > > set $e = event_list->tqh_first > > > while ($e != 0) > > > if ($e->ie_source != 0 && $e->ie_handlers.tqh_first != 0) > > > set $src = (struct intsrc *)$e->ie_source > > > if ($src->is_pic->pic_enable_source == > &ioapic_enable_source) > > > set $_cpu = ((struct ioapic_intsrc *)$src)->io_cpu > > > else > > > if ($src->is_pic->pic_enable_source == > &msi_enable_source) > > > set $_cpu = ((struct msi_intsrc *)$src)->msi_cpu > > > else > > > set $_cpu = 0 > > > end > > > end > > > printf "CPU %d: %s\n", $_cpu, $e->ie_fullname > > > end > > > set $e = $e->ie_list.tqe_next > > > end > > > end > > > > > > document irqs > > > Dump list of IRQs with associated CPU. > > > end > > > > > > However, unless the driver is using BUS_BIND_IRQ() or you are using > cpuset > > > -x, > > > the interrupts should be round-robin assigned among CPUs. > > > > > > What exactly do you mean by sluggish? Trying to interact with the box > over > > > SSH is sluggish? Is there a change in RTT if you are pinging the box, > is > > > there a change in performance of TCP or UDP streams to/from the box? > > > > > > -- > > > John Baldwin > > > > > > > -- > John Baldwin >