Date: Wed, 20 Jul 2016 08:47:46 +0300 From: Slawa Olhovchenkov <slw@zxy.spb.ru> To: Konstantin Belousov <kostikbel@gmail.com> Cc: Hartmut Brandt <hartmut.brandt@dlr.de>, current@freebsd.org Subject: Re: (boost::)asio and kqueue problem Message-ID: <20160720054746.GM20831@zxy.spb.ru> In-Reply-To: <20160720043809.GP38613@kib.kiev.ua> References: <alpine.BSF.2.00.1607191024380.30581@KNOP-BEAGLE.kn.op.dlr.de> <20160720043809.GP38613@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, Jul 20, 2016 at 07:38:09AM +0300, Konstantin Belousov wrote:
> On Tue, Jul 19, 2016 at 05:35:59PM +0200, Hartmut Brandt wrote:
> > Hi,
> >
> > I'm trying to use asio (that's boost::asio without boost) to handle
> > listening sockets asynchronuosly. This appears not to work. There are also
> > some reports on the net about this problem. I was able to reproduce the
> > problem with a small C-programm that does the same steps as asio. The
> > relevant sequence of system calls is:
> >
> > kqueue() = 3 (0x3)
> > socket(PF_INET,SOCK_STREAM,6) = 4 (0x4)
> > setsockopt(0x4,0xffff,0x800,0x7fffffffea2c,0x4) = 0 (0x0)
> > kevent(3,{ 4,EVFILT_READ,EV_ADD|EV_CLEAR,0x0,0x0,0x0 4,EVFILT_WRITE,EV_ADD|EV_CLEAR,0x0,0x0,0x0 },2,0x0,0,0x0) = 0 (0x0)
> > setsockopt(0x4,0xffff,0x4,0x7fffffffea2c,0x4) = 0 (0x0)
> > bind(4,{ AF_INET 0.0.0.0:8080 },16) = 0 (0x0)
> > listen(0x4,0x80) = 0 (0x0)
> > ioctl(4,FIONBIO,0xffffea2c) = 0 (0x0)
> > kevent(3,{ 4,EVFILT_READ,EV_ADD|EV_CLEAR,0x0,0x0,0x0 4,EVFILT_WRITE,EV_ADD|EV_CLEAR,0x0,0x0,0x0 },2,0x0,0,0x0) = 0 (0x0)
> > kevent(3,0x0,0,0x7fffffffe5a0,32,0x0) ERR#4 'Interrupted system call'
> >
> > The problem here is that asio registers each file descriptor with
> > EVFILT_READ and EVFILT_WRITE as soon as it is opened (first kevent call).
> > After bringing the socket into the listening state and when async_accept()
> > is called it registers the socket a second time. According to the man page
> > this is perfectly legal and can be used to modify the registration.
> >
> > With this sequence of calls kevent() does not return when a connection is
> > established successfully.
> >
> > I tracked down the problem and the reason is in soo_kqfilter(). This is
> > called for the first EVFILT_READ registration and decides based on the
> > SO_ACCEPTCONN flag which filter operations to use solisten_filtops or
> > soread_filtops. In this case it chooses soread_filtops.
> >
> > The second EVFILT_READ registration does not call soo_kqfilter() again,
> > but just updates the filter from the data and fflags field so the
> > listening socket ends up with the wrong filter operations.
> >
> > The attached patch fixes this (kind of) by using the f_touch
> > operation (currently used only by the user filter). The filt_sotouch()
> > function changes the operation pointer in the knote when the socket is now
> > in the listening state. I suppose that the required locking is already
> > done in kqueue_register(), but I'm not sure. Asynchronous accepting now works.
> >
> > A better fix would probably be to change the operation vector on all
> > knotes attached to the socket in solisten(), but I fear I don't have the
> > necessary understanding of the locking that is required for this.
> >
> > Could somebody with enough kqueue() knowledge look whether the patch is
> > correct lock-wise?
> I find it weird that the fix still requires re-registration of the socket
> event to get it working after socket is marked as listen. In other words,
> until the re-registration is done, the events for the registered filter
> are lost.
>
> IMO more correct solution would be to merge the filt_solisten and
> filt_soread, deciding which path to take by testing the SO_ACCEPTCON
> flag in the f_event() op.
>
> This should also eliminate the concerns with the patching of pointer
> which is not supposed to be changed.
may be this is related to netmap problem also?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160720054746.GM20831>