Date: Tue, 23 Sep 2014 20:35:04 +0400 From: "Alexander V. Chernikov" <melifaro@FreeBSD.org> To: Luigi Rizzo <rizzo@iet.unipi.it> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, Adrian Chadd <adrian@freebsd.org>, Elof Ofel <elofu17@hotmail.com> Subject: Re: How do I balance bandwidth over several virtual NICs? Message-ID: <5421A138.90803@FreeBSD.org> In-Reply-To: <20140923154610.GD84074@onelab2.iet.unipi.it> References: <DUB125-W13FDC584F5DF9881CF5FDEBCB30@phx.gbl> <CA%2BP_MZGA_uz_H_QsB%2BdgXEgbXNCjv7w-OToKby=ww%2BvKgnU4_Q@mail.gmail.com> <DUB125-W851F972702452D9809C8E5BCB30@phx.gbl> <CAJ-Vmo=NGGkOkPWQKZ=3gA3vYYyM2kcjd3m85ymdJY3q4ixxLw@mail.gmail.com> <5421310C.5010406@FreeBSD.org> <CAJ-VmomamPVwja2Tgfi46EBu8E5Hohr6FcshsOkDeycEvGV51Q@mail.gmail.com> <CA%2BhQ2%2BhU6XwaeyqdcseL9uusmKfyG4Ko7TwvsZwU=r1cUyCw1w@mail.gmail.com> <54218EF4.6090102@FreeBSD.org> <20140923154610.GD84074@onelab2.iet.unipi.it>
next in thread | previous in thread | raw e-mail | index | archive | help
On 23.09.2014 19:46, Luigi Rizzo wrote: > On Tue, Sep 23, 2014 at 07:17:08PM +0400, Alexander V. Chernikov wrote: >> On 23.09.2014 18:44, Luigi Rizzo wrote: > ... >> However, in addition to non-symmetric RSS (which is hopefully being >> addressed), there is another >> usual "producer - multuple consumers" problem: one snort process can >> start process packets very slowly, or hang, or crash. >> In that case host RX ring is getting full, NIC fails to push packets to >> given queue and start storing them inside >> its skid buffer (512k for Niantic afair). After that buffer becomes full >> traffic and all processing stops. > interesting. Actually, scary! > Do you have any reference to the data sheets documenting > that behaviour ? I have indeed received reports saying something Quoting latest 82599 datasheet: Flow control: 3.7.7.3.4 Packet Buffer Size .. Suggested buffer size (FC on): 512K Suggested buffer size (FC off, jumbo): 40Kb 7.1 Receive Functionality Packet reception consists of: • Recognizing the presence of a packet on the wire • Performing address filtering • DMA queue assignment • Storing the packet in the receive data FIFO ----------------------------------- ^^^^^^^^^^^ • Transferring the data to assigned receive queues in host memory • Updating the state of a receive descriptor. 7.1.9.1 Low Receive Descriptors Threshold The number of usable (free) descriptors for the hardware is the distance between the Tail and Head registers. When the tail reaches the head, there are no free descriptors and further packets might be either dropped or block the receive FIFO. In order to avoid this situation, the 82599 might generate a low latency interrupt (associated to the relevant > similar but always suspected user errors. > The fact that a starved queue can consume the entire internal > buffer seems a really bad bug. > > At least you can overcome this one by having the demux done > in software. Well, yes. Despite the fact that 82599 can handle huge (65k) RX queues, this does not help much unless there is some knob to prevent HOL. > > cheers > luigi >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5421A138.90803>