Date: Fri, 22 Jun 2012 15:55:23 +0530 From: Venkat Duvvuru <venkatduvvuru.ml@gmail.com> To: John Baldwin <jhb@freebsd.org> Cc: freebsd-net@freebsd.org Subject: Re: MSI-X limitation in freebsd 8.2 Message-ID: <CAGdae7Y7KN_P-noqct0H0JXB-9mkXnkvaEPQcr13x1_Be_Vk=Q@mail.gmail.com> In-Reply-To: <CAGdae7YSKf=vqCO0EvsFcg9nysmnRFMbv%2BXj8chBW62uoKHFFQ@mail.gmail.com> References: <CAGdae7YW0jt4gTmHufHFTrcdtFGhngu_Wi%2BMDg1nY_T0J1C_SQ@mail.gmail.com> <201206210816.22774.jhb@freebsd.org> <CAGdae7Yum_MP99kSK1Xi8z02APnZPoXFKR22fTuE5tiuorYMhw@mail.gmail.com> <201206210947.30171.jhb@freebsd.org> <CAGdae7YSKf=vqCO0EvsFcg9nysmnRFMbv%2BXj8chBW62uoKHFFQ@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
and no matter in which order I bring up the interfaces, the system becomes sluggish after the 5th interface is brought up. So doesn't look like a specific driver issue. On Fri, Jun 22, 2012 at 2:25 PM, Venkat Duvvuru <venkatduvvuru.ml@gmail.com>wrote: > This setup has variety of NICs, Intel's, Emulex's and all are stock > drivers. > > /Venkat > > On Thu, Jun 21, 2012 at 7:17 PM, John Baldwin <jhb@freebsd.org> wrote: > >> On Thursday, June 21, 2012 8:50:06 am Venkat Duvvuru wrote: >> > While I'm able to ping to the machine..every command on the current >> active >> > session is sluggish infact doesn't complete, unable to open another ssh >> > session..an already opened ssh session doesn't react which is running >> > "systat -vmstat"... >> > >> > The only thing I could do is to successfully complete "top -P" command >> > once..even top hung for the second time. >> >> Hmm, is this a stock FreeBSD driver or an out-of-tree NIC driver? >> >> > /Venkat >> > >> > On Thu, Jun 21, 2012 at 5:46 PM, John Baldwin <jhb@freebsd.org> wrote: >> > >> > > On Thursday, June 21, 2012 5:47:48 am Venkat Duvvuru wrote: >> > > > John - Thanks for the reply. >> > > > >> > > > All the CPUs are ~100% idle. I don't see any interrupt storm on any >> of >> > > the >> > > > irqs (vmstat -i). >> > > > >> > > > One observation I made is that I see messages like these in dmesg >> > > > >> > > > ===> mem >> > > 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf1c000-0xfaf1ffff >> > > > irq 40 at device 0.1 on pci6 >> > > > >> > > > Looking at the irq value I think it is the INTx irq range which >> shouldn't >> > > > have probably got allocated as the device is msix capable and there >> are >> > > > vectors allocated for these devices in the range (256-380). >> > > > >> > > > Could this be a problem? >> > > >> > > No, that line is output before the driver's attach routine is run, so >> it >> > > will >> > > always show INTx IRQ value even if it isn't used. >> > > >> > > > The scenario where I am hitting this problem is a setup with 4 >> NICs, each >> > > > NIC with two ports and each port using up 4 msix vectors. The >> system is >> > > > fine till some ports are up but once I ifup the 5th port, the system >> > > > becomes sluggish. >> > > > >> > > > I'm not sure whether all the 30 vectors are from a single cpu..I >> don't >> > > know >> > > > how to get that information. >> > > >> > > Unfortunately there isn't an easy way. I have this gdb script which >> can >> > > display it from kgdb on x86: >> > > >> > > define irqs >> > > set $e = event_list->tqh_first >> > > while ($e != 0) >> > > if ($e->ie_source != 0 && $e->ie_handlers.tqh_first != 0) >> > > set $src = (struct intsrc *)$e->ie_source >> > > if ($src->is_pic->pic_enable_source == >> &ioapic_enable_source) >> > > set $_cpu = ((struct ioapic_intsrc *)$src)->io_cpu >> > > else >> > > if ($src->is_pic->pic_enable_source == >> &msi_enable_source) >> > > set $_cpu = ((struct msi_intsrc *)$src)->msi_cpu >> > > else >> > > set $_cpu = 0 >> > > end >> > > end >> > > printf "CPU %d: %s\n", $_cpu, $e->ie_fullname >> > > end >> > > set $e = $e->ie_list.tqe_next >> > > end >> > > end >> > > >> > > document irqs >> > > Dump list of IRQs with associated CPU. >> > > end >> > > >> > > However, unless the driver is using BUS_BIND_IRQ() or you are using >> cpuset >> > > -x, >> > > the interrupts should be round-robin assigned among CPUs. >> > > >> > > What exactly do you mean by sluggish? Trying to interact with the >> box over >> > > SSH is sluggish? Is there a change in RTT if you are pinging the >> box, is >> > > there a change in performance of TCP or UDP streams to/from the box? >> > > >> > > -- >> > > John Baldwin >> > > >> > >> >> -- >> John Baldwin >> > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAGdae7Y7KN_P-noqct0H0JXB-9mkXnkvaEPQcr13x1_Be_Vk=Q>