Date: Wed, 10 Feb 2016 16:53:18 +0300 From: Slawa Olhovchenkov <slw@zxy.spb.ru> To: Giuseppe Lettieri <g.lettieri@iet.unipi.it> Cc: Luigi Rizzo <rizzo@iet.unipi.it>, Adrian Chadd <adrian.chadd@gmail.com>, "stable@freebsd.org" <stable@freebsd.org> Subject: Re: 82576 + NETMAP + VLAN Message-ID: <20160210135318.GL68298@zxy.spb.ru> In-Reply-To: <56BB3C20.600@iet.unipi.it> References: <CA%2BhQ2%2Bg0ggpS%2BE5nOpON66efs7cwsed=NvaKa=mzsg6FycGhiQ@mail.gmail.com> <20160202204446.GQ88527@zxy.spb.ru> <20160204130029.GC88527@zxy.spb.ru> <CAJ-Vmok%2B7Vt4ww4iWkQY505eapxVQF4MBtnb%2BwGg-TNSmJTLGw@mail.gmail.com> <CAJ-VmomzvoZZZPUveTZUJ5zAhHQkJ5M9%2B7gfN8gGSGp05JpOWw@mail.gmail.com> <20160208173935.GK68298@zxy.spb.ru> <CA%2BhQ2%2BiD3X9wR8exw2p-9G8pPNHCQtLdMdJJXU78PDrQaWBH7w@mail.gmail.com> <56B9E398.1060105@iet.unipi.it> <20160210115937.GA37895@zxy.spb.ru> <56BB3C20.600@iet.unipi.it>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Feb 10, 2016 at 02:33:20PM +0100, Giuseppe Lettieri wrote: > Il 10/02/2016 12:59, Slawa Olhovchenkov ha scritto: > > Can you look also on second issue? > > > > PS: What need from me? May be open PR? > > May you provide some example code that triggers the issue? This is about 700 lines of code (not very clear), may be I can describe it? First: this is thread code. I think it impotant point. All sync with netmap go throw kevent() interface. Main thread got information about two NIC (determine number of rings) and spawn 1 balancer thread and NCPU worker thread. All thread pinned to cores. Balancer thread open all rings on both NIC separate (i.e. open 6+6 NETMAP fds for hardware rings) and allocate on first NIC and open NCPU*2 master NETMAP pipes. All fds register in kevent for read event as EV_SET(kp, fd, EVFILT_READ, EV_ADD | EV_ENABLE | EV_CLEAR, 0, 0, kidx); and for future write event as EV_SET(kp, fd, EVFILT_WRITE, EV_ADD | EV_DISABLE, 0, 0, 0); after write occur next event will be added: EV_SET(kp, fd, EVFILT_WRITE, EV_ADD | EV_DISPATCH, 0, 0, 0); Each worker thread open two slave NETMAP pipes in same maner as above. Balancer got packet from inside hardware ring, dispatch (zero-copy) to inside master pipe. Worker got packet from inside slave pipe and zero-copy to outside slave pipe. Balancer got packet from outside master pipe and dispatch (zero-copy) to outside hardware ring. And vise versa. After program start I have deadlock instantly (with WITNESS enabled kernel). W/o WITNESS deadloc occur only under load. W/o EV_DISABLE/EV_DISPATCH NETAMP don't see output packets. NETMAP don't forward incoming packet to program when open and attach to NETMAP do in main thread.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160210135318.GL68298>