Date: Thu, 11 Jul 2013 12:36:30 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Adrian Chadd <adrian@freebsd.org> Cc: freebsd-current <freebsd-current@freebsd.org> Subject: Re: hacking - aio_sendfile() Message-ID: <20130711093630.GL91021@kib.kiev.ua> In-Reply-To: <CAJ-VmomFcZv6_uhpy6ShBzL21=Fqw7_y2xE2oYGod2T=Z5Qgpg@mail.gmail.com> References: <CAJ-Vmo=icr6bda%2BWMNUarc3WbdqJ%2BMdauX6kByxxdTx8oSovBg@mail.gmail.com> <20130711061753.GK91021@kib.kiev.ua> <CAJ-VmomFcZv6_uhpy6ShBzL21=Fqw7_y2xE2oYGod2T=Z5Qgpg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--4oFiy8uZhEhZB7V2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jul 11, 2013 at 01:37:19AM -0700, Adrian Chadd wrote: > Hiya, >=20 > I'm more interested in the API than the implementation at the moment. >=20 > Yes, you're right - it should eventually be driven using disk io > completion upcalls which triggers the push of data into the socket > buffer. I totally agree. >=20 > I'm hacking up some libevent-ish looking thing that uses kqueue and > wraps aio, read, write, and other event types into something I can > easily shoehorn this stuff into. I'll then throughly test it (and > other options) out. You're right, it's likely going to end up with a > whole lot of aio threads sitting there waiting for disk IO to complete > - and at that point, I'll start hacking at sendfile() to split it into > two halves and have it driven by a completion call from g_up or > wherever, triggering the socket write side of things. >=20 > There are some other questions too - like whether the IO completion > should just queue socket IO (and have it potentially block in the TCP > code) or whether it should funnel completions into a per-CPU aio > completion thread which does the socket write bit. That way disk IO > completion isn't going to be blocked by longer-held locks in the > networking stack. No, it is not disk I/O which is problematic there. It is socket I/O e.g. wait for the socket buffers lomark in the kern_sendfile() which causes unbounded sleep. Look for the sbwait() call, both in the kern_sendfile() itself, and in the pru_send methods of the protocols, e.g. in sosend_generic(). The wait scope controlled by the other side of connection and allow it to completely block the aio subsystem. Disk I/O is supposed to finish in the finite time. --4oFiy8uZhEhZB7V2 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJR3nydAAoJEJDCuSvBvK1B4GkP/32Hyjeqm6RYGgcKW1fBYuaA mo0Eh2zwkYxAr+c3NPj0Tcd/c06cc+bwjbTbtlL1uiyNZNvK1Hb8LnlatfxkubGs 9TZ+T1NzyPJae4uPGMZMcegOVJoeDh9liAcbkTGj6gwoMklAbTGwVYQc9Xs1FhiY hP/D8/ITrnRiltFi7OwAtpqJGjRvz34iSl9viroZcv9+K8uGNobCXuiE2J35eh/p a2F13Lt6kiPSlynkaL6xwabchTgtsAOjNNvtFAeR5D4125nWGijFU9cpU/iFobpH 3BZZ6WbgMTNhxXEqjEbtTKx8YyUEjrRE+p0MoJ2RpgjKXtnhxTtneLyjHiC2Xmam Uk/1IdIhyUnexqBFcN/wGAq36Bd95/+VpwzrIDeWM3E8UoMrtn6oLnV5lbC+oFRA J6FceZIuk6d7RYFLo5tQGzZ+Nuwi95VpO6gT8hCfWxbGVJ3WX72aWlq5qC2tmIT5 TyCDLYAFmFE6sna3xgwLVwnUlGdyV10kSWJLY3VITv3oFn9H9I335+sQBdujpcd+ HflR111vRUabZ3H0FJ7c0wO+jNBPhxYw97K7ckqZsN9ww+vaHPPObImi/oEGVafL W0PYcMFZ1mMfh4xq5GPjrAwz2o4uVe0ub3opadWfx/s2DvYhr82uMgZRtDLlF+s8 BczIcfB67jsn8SxETBRW =xjg4 -----END PGP SIGNATURE----- --4oFiy8uZhEhZB7V2--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130711093630.GL91021>