Date: Tue, 23 Sep 2014 19:17:08 +0400 From: "Alexander V. Chernikov" <melifaro@FreeBSD.org> To: Luigi Rizzo <rizzo@iet.unipi.it>, Adrian Chadd <adrian@freebsd.org> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Elof Ofel <elofu17@hotmail.com> Subject: Re: How do I balance bandwidth over several virtual NICs? Message-ID: <54218EF4.6090102@FreeBSD.org> In-Reply-To: <CA%2BhQ2%2BhU6XwaeyqdcseL9uusmKfyG4Ko7TwvsZwU=r1cUyCw1w@mail.gmail.com> References: <DUB125-W13FDC584F5DF9881CF5FDEBCB30@phx.gbl> <CA%2BP_MZGA_uz_H_QsB%2BdgXEgbXNCjv7w-OToKby=ww%2BvKgnU4_Q@mail.gmail.com> <DUB125-W851F972702452D9809C8E5BCB30@phx.gbl> <CAJ-Vmo=NGGkOkPWQKZ=3gA3vYYyM2kcjd3m85ymdJY3q4ixxLw@mail.gmail.com> <5421310C.5010406@FreeBSD.org> <CAJ-VmomamPVwja2Tgfi46EBu8E5Hohr6FcshsOkDeycEvGV51Q@mail.gmail.com> <CA%2BhQ2%2BhU6XwaeyqdcseL9uusmKfyG4Ko7TwvsZwU=r1cUyCw1w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 23.09.2014 18:44, Luigi Rizzo wrote: > > > On Tue, Sep 23, 2014 at 4:36 PM, Adrian Chadd <adrian@freebsd.org > <mailto:adrian@freebsd.org>> wrote: > > On 23 September 2014 01:36, Alexander V. Chernikov > <melifaro@freebsd.org <mailto:melifaro@freebsd.org>> wrote: > > On 22.09.2014 23:46, Adrian Chadd wrote: > >> Hi, > >> > >> Yes. > >> > >> * grab an ixgbe NIC and the -HEAD driver; (or cxgbe - I haven't > gone > >> and written RSS programming code for that just yet); > >> * patch it to use a symmetric RSS key; > >> * configure up N queues; > >> * run an instance of snort on each TX/RX ring from the NIC. > > Oh, wow. > > I have a low priority task to do that. > > Nice to see this in stock fbsd! > > > >> > >> The last step requires that you have snort use netmap rather > than just > >> straight bpf - or maybe somehow there's a way to glue bpf into a > >> single netmap ring. > > I've wrote snort netmap DAG once, but it does not play well w/o > > symmetric rss. > > I've see if I can share it. > > That'd be great! > > I'll see if I can get -HEAD enabled with an optional symmetric RSS > key. > > It shouldn't be too difficult. The problem is the current RSS setup > uses the same key for all NICs. > I _guess_ that isn't going to /really/ be a problem here - unless you > really want your server to serve lots of traffic /and/ snort :) > > Then we just need a netmap enabled snort :) > > > from my (not first-hand) knowledge with IDSes, > i > believe > that the bottleneck is > > mostly the processing > done in the IDS, rather than > > the network I/O (provided > it is > reasonably fast > ). True. > > As a result, just running IDS instances on top > of a netmap-enabled libpcap (i.e. no source code > modifications) should do the job. The problem with snort is that is single-threaded, so you have to (somehow) split traffic (preserving sessions) and run multiple snort instances on each. Linux guys do that with pf_ring. I've created snort netmap DAG to be able to open each NIC queue with different snort process. However, in addition to non-symmetric RSS (which is hopefully being addressed), there is another usual "producer - multuple consumers" problem: one snort process can start process packets very slowly, or hang, or crash. In that case host RX ring is getting full, NIC fails to push packets to given queue and start storing them inside its skid buffer (512k for Niantic afair). After that buffer becomes full traffic and all processing stops. > > I know the Bro developers (in Bcc so they can pitch > in if they like) have been playing with some > external traffic demultiplexer that reads from the > NIC (in netmap mode) and passes traffic to IDS > instances using VALE ports or netmap pipes, > all of which are compatible with the netmap-libpcap. > > In other words, even if the hardware cannot do rss > in a useful way, you should be able to do the > demux in software. > > Of course, if you can put the hardware at work, > you should go for that. > > cheers > luigi > >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?54218EF4.6090102>