Date: Tue, 11 Aug 2015 19:05:15 -0700 From: Maxim Sobolev <sobomax@FreeBSD.org> To: Luigi Rizzo <rizzo@iet.unipi.it>, FreeBSD Net <freebsd-net@freebsd.org>, =?UTF-8?Q?Jev_Bj=C3=B6rsell?= <jev@sippysoft.com> Subject: Re: Poor high-PPS performance of the 10G ixgbe(9) NIC/driver in FreeBSD 10.1 Message-ID: <CAH7qZfvFZ1ne%2Bg%2B3JzU92b1goNiTgYjudtgn%2BDiUPNoc-t97yg@mail.gmail.com> In-Reply-To: <CA%2BhQ2%2BiOB6Hq3x9R2ckNx98a3ssgPMK=YC_=DQV1wBtoe6ZdjA@mail.gmail.com> References: <CAH7qZftMB34PM1CqNhdg7AWhsq6YknUDgc60ASfT2Z0L1z8XCQ@mail.gmail.com> <CA%2BhQ2%2BgGnq3WJkwaBnSgy-LXn0229b4iNntXq%2BNN7enfV36CRQ@mail.gmail.com> <CAH7qZft4Vcuc2KopqnJNZWqO4jj3-B3GSy6TendQxHrZDA8FmQ@mail.gmail.com> <CA%2BhQ2%2BiOB6Hq3x9R2ckNx98a3ssgPMK=YC_=DQV1wBtoe6ZdjA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
OK, so following Luigi's suggestion we've re-enabled AIM, set max_interrupt_rate to 8000 (matching igb), and reduced number of queues to 6. We'll have next peak in about 14 hours, I'll try to capture and record history of the per-queue interrupt rate. It still remains somewhat puzzling why something that worked with old-gen hardware/driver almost out of the box now needs some extra tuning. Thanks everyone for useful suggestions in any case! On Tue, Aug 11, 2015 at 4:04 PM, Luigi Rizzo <rizzo@iet.unipi.it> wrote: > On Wed, Aug 12, 2015 at 12:46 AM, Maxim Sobolev <sobomax@sippysoft.com> > wrote: > > Hi Luigi, thanks for the input. Are you sure about the number of queues? > > Keep in mind we are not using netmap or anything as advanced, so we need > lot > > of parallel pipes just to push packet through the whole UDP/IP kernel > stack. > > To make things worse, our app cannot use connected sockets at the > moment, so > > we are having worst case scenario wrt kernel overhead. Single thread > sender > > on that hardware is limited to only 225K PPS until it hits CPU limit. We > use > > 8 queues on other older servers with I350 NICs. My send-only testing > shows > > that the hardware peaks at about 5 concurrent threads for connected > sockets > > and 7 concurrent threads for unconnected. This is with default 8 hardware > > queues. The test is somewhat crippled since it only tests outgoing > traffic. > > > > thing is, you are using 8 queues per NIC so the total number of different > interrupts is huge and will likely eat all of your cores. > This is why i'd try to start low and ramp up later. > > > http://bit.ly/1MiLFHg > > > > With regards to the moderation, what effect it has on the traffic? Is NIC > > going to drop packets once it starts moderating interrupts? > > moderation only means you get larger batches on each interrupt, > which overall makes the system more efficient. The NIC will only drop > packets when the CPU is not fast enough to drain them,. > > (this said, it might be a good idea to try netmap, passing packets > that you do not know how to handle to the host stack, and instead > intercepting those that you can deal with in netmap. those > unconnected udp sockets are the ideal case for netmap.) > > cheers > luigi >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAH7qZfvFZ1ne%2Bg%2B3JzU92b1goNiTgYjudtgn%2BDiUPNoc-t97yg>