Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 24 Jun 2003 10:46:37 +1000
From:      Paul Koch <paul.koch@statscout.com>
To:        "Cagle, John (ISS-Houston)" <john.cagle@hp.com>
Cc:        freebsd-smp@freebsd.org
Subject:   Re: FreeBSD 4.8 SMP performance
Message-ID:  <200306241046.37229.paul.koch@statscout.com>
In-Reply-To: <C50AB9511EE59B49B2A503CB7AE1ABD10476C694@cceexc19.americas.cpqcorp.net>
References:  <C50AB9511EE59B49B2A503CB7AE1ABD10476C694@cceexc19.americas.cpqcorp.net>

next in thread | previous in thread | raw e-mail | index | archive | help
Ah! thanks, I did a bit of reading on this and it sounds like
we need to keep clear of these types of cards for the moment.
We didn't plan on using the multiport NICs, it's just what came
in the box.  Is this only an issue when running SMP ? because
the NICs appeared to work fine when initially running a 
non-SMP kernel.

OS selection of "Linux": hmm, didn't know about that. More
reading to be done here. I assume this does is some vendor
specific trickery setting the hardware up.

	Paul.

On Mon, 23 Jun 2003 11:55 pm, Cagle, John (ISS-Houston) wrote:
> Paul,
>
> There's a known issue with APIC IRQ routing when using Multi-Port
> NICs in PCI slots.  The existing code doesn't know how to swizzle the
> IRQs through the PCI-to-PCI bridges that are on the NIC.  If you want
> to use Multi-port NICs, you'll need to make sure you're not running
> in APIC Full Table mode.  Double check that your OS Selection (in ROM
> Based Setup) is "Linux".
>
> Of course, that may not be the problem you're experiencing, but it's
> my best guess.
>
> Regards,
> John
> --------------------------------
> John Cagle     john.cagle@hp.com
> Principal Member Technical Staff
>    Industry Standard Servers
>     Hewlett-Packard Company
>
> -----Original Message-----
> From: Paul Koch [mailto:paul.koch@statscout.com]
> Sent: Monday, June 23, 2003 1:13 AM
> To: freebsd-smp@freebsd.org
> Subject: FreeBSD 4.8 SMP performance
>
>
> Last week we did a fairly major installation of our network
> monitoring software at a customers' site on a couple of
> identical quad processor machines running FreeBSD-4.8.
> One as a production machine, the other as a warm spare.
>
> The machines were unfortunately "hand me downs" from the
> weenies server group:
>  - Compaq server of some description
>  - quad 700Mhz Intel Xeon 2M cache CPUs
>  - 4 gigabytes of ram
>  - usual arrangement of Compaq/Adaptec SCSI and raid
>     controllers
>  - 1 * raid 1 disk setup
>  - 1 * raid 5 disk setup
>  - single and dual port Intel NICs
>
> We originally intended to benchmark a 4.8 and 5.1 box side
> by side to see what the SMP performance gains were, but ran
> out of time due to SMP kernel and dual port Intel NIC problem. When
> running 5.1 with SMP, the dual port NICs would lock up with "fxp0
> device timeout" messages. Thinking this was a 5.x
> related problem, we fell back to 4.8 SMP, eventually finding
> that the same problem also existed in it.  A quick change of NIC
> fixed our immediate issue.  Time ran short so we unfortunately never
> got back to 5.1 SMP or to investigate the dual port NIC problems.
>
> Having never run a SMP box before, we were surprised when
> running 'top' on the 5.1 machine to see a process state of "Giant".
> Thought this was suppose to be fine grain SMP :)
>
> Our product is a real-time network performance monitoring
> application, only available on FreeBSD of course. Statscout
> NPM is typically deployed as a "blanket monitoring" solution.
> That is, monitor every port on every switch/router/repeater each
> minute (ie. ping/snmp) for things like bytes, frames, errors,
> discards and status.
>
> This particular customer has are rather large Cisco network,
> consisting of over 5000 switches/routers and ~100,000 network
> interfaces. Our software was configured to poll some devices at 60
> second intervals and others at 120 seconds, collecting 10
> SNMP OIDs per interface per poll. That's approximately one
> million SNMP OIDs to collect.
>
> The customer provided a list of device names/IP addresses and then it
> took just under one hour to SNMP walk all 5000 devices and populate
> our configuration. Once running and with a bit of
> tuning to our software, the machine was cruising along collecting,
> processing and saving over 500,000 SNMP OIDs per minute
> (8333/sec), within ~20,000 SNMP requests, and ~4000 pings
> per minute, using less than 50% of available CPU. We were very
> impressed with how smoothly and responsive the SMP machine ran doing
> lots of simultaneous disk/network activity, while also servicing user
> requested real-time reports.
>
> It was a pity that the network wasn't big enough to push the
> machine to its limits !
>
>
> We noticed several things on 4.8:
>
>  - Some of our busy processes jumped around 'a lot' between
>    the different processors. It would be nice if processor affinity
>    was implemented to get the best out of those 2M CPU caches.
>
>  - "systat -vm" would chew up ~10 percent on one CPU when
>    running a SMP kernel. Don't know what was going on there.
>
>  - The dual port Intel fxp lockup mentioned above.
>
> 	Paul.



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200306241046.37229.paul.koch>