Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 21 Jun 2012 08:16:22 -0400
From:      John Baldwin <jhb@freebsd.org>
To:        Venkat Duvvuru <venkatduvvuru.ml@gmail.com>
Cc:        freebsd-net@freebsd.org
Subject:   Re: MSI-X limitation in freebsd 8.2
Message-ID:  <201206210816.22774.jhb@freebsd.org>
In-Reply-To: <CAGdae7anhqdzacVt-BMv3e7GvKFO4QbYEVxd9k-agGjLfPi_iQ@mail.gmail.com>
References:  <CAGdae7YW0jt4gTmHufHFTrcdtFGhngu_Wi%2BMDg1nY_T0J1C_SQ@mail.gmail.com> <201206201054.40824.jhb@freebsd.org> <CAGdae7anhqdzacVt-BMv3e7GvKFO4QbYEVxd9k-agGjLfPi_iQ@mail.gmail.com>

next in thread | previous in thread | raw e-mail | index | archive | help
On Thursday, June 21, 2012 5:47:48 am Venkat Duvvuru wrote:
> John - Thanks for the reply.
> 
> All the CPUs are ~100% idle. I don't see any interrupt storm on any of the
> irqs (vmstat -i).
> 
> One observation I made is that I see messages like these in dmesg
> 
> ===> mem 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf1c000-0xfaf1ffff
> irq 40 at device 0.1 on pci6
> 
> Looking at the irq value I think it is the INTx irq range which shouldn't
> have probably got allocated as the device is msix capable and there are
> vectors allocated for these devices in the range (256-380).
> 
> Could this be a problem?

No, that line is output before the driver's attach routine is run, so it will 
always show INTx IRQ value even if it isn't used.

> The scenario where I am hitting this problem is a setup with 4 NICs, each
> NIC with two ports and each port using up 4 msix vectors. The system is
> fine till some ports are up but once I ifup the 5th port, the system
> becomes sluggish.
> 
> I'm not sure whether all the 30 vectors are from a single cpu..I don't know
> how to get that information.

Unfortunately there isn't an easy way.  I have this gdb script which can 
display it from kgdb on x86:

define irqs
    set $e = event_list->tqh_first
    while ($e != 0)
	if ($e->ie_source != 0 && $e->ie_handlers.tqh_first != 0)
	    set $src = (struct intsrc *)$e->ie_source
	    if ($src->is_pic->pic_enable_source == &ioapic_enable_source)
		set $_cpu = ((struct ioapic_intsrc *)$src)->io_cpu
	    else
		if ($src->is_pic->pic_enable_source == &msi_enable_source)
		    set $_cpu = ((struct msi_intsrc *)$src)->msi_cpu
		else
		    set $_cpu = 0
		end
	    end
	    printf "CPU %d: %s\n", $_cpu, $e->ie_fullname
	end
	set $e = $e->ie_list.tqe_next
    end
end

document irqs
Dump list of IRQs with associated CPU.
end

However, unless the driver is using BUS_BIND_IRQ() or you are using cpuset -x, 
the interrupts should be round-robin assigned among CPUs.

What exactly do you mean by sluggish?  Trying to interact with the box over 
SSH is sluggish?  Is there a change in RTT if you are pinging the box, is 
there a change in performance of TCP or UDP streams to/from the box?

-- 
John Baldwin



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201206210816.22774.jhb>