From owner-freebsd-current@FreeBSD.ORG Thu Jul 11 09:36:42 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 0A5F3C78; Thu, 11 Jul 2013 09:36:42 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 75DE01DD7; Thu, 11 Jul 2013 09:36:41 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r6B9aUQO080417; Thu, 11 Jul 2013 12:36:30 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r6B9aUQO080417 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r6B9aUEw080416; Thu, 11 Jul 2013 12:36:30 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 11 Jul 2013 12:36:30 +0300 From: Konstantin Belousov To: Adrian Chadd Subject: Re: hacking - aio_sendfile() Message-ID: <20130711093630.GL91021@kib.kiev.ua> References: <20130711061753.GK91021@kib.kiev.ua> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="4oFiy8uZhEhZB7V2" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-current X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Jul 2013 09:36:42 -0000 --4oFiy8uZhEhZB7V2 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jul 11, 2013 at 01:37:19AM -0700, Adrian Chadd wrote: > Hiya, >=20 > I'm more interested in the API than the implementation at the moment. >=20 > Yes, you're right - it should eventually be driven using disk io > completion upcalls which triggers the push of data into the socket > buffer. I totally agree. >=20 > I'm hacking up some libevent-ish looking thing that uses kqueue and > wraps aio, read, write, and other event types into something I can > easily shoehorn this stuff into. I'll then throughly test it (and > other options) out. You're right, it's likely going to end up with a > whole lot of aio threads sitting there waiting for disk IO to complete > - and at that point, I'll start hacking at sendfile() to split it into > two halves and have it driven by a completion call from g_up or > wherever, triggering the socket write side of things. >=20 > There are some other questions too - like whether the IO completion > should just queue socket IO (and have it potentially block in the TCP > code) or whether it should funnel completions into a per-CPU aio > completion thread which does the socket write bit. That way disk IO > completion isn't going to be blocked by longer-held locks in the > networking stack. No, it is not disk I/O which is problematic there. It is socket I/O e.g. wait for the socket buffers lomark in the kern_sendfile() which causes unbounded sleep. Look for the sbwait() call, both in the kern_sendfile() itself, and in the pru_send methods of the protocols, e.g. in sosend_generic(). The wait scope controlled by the other side of connection and allow it to completely block the aio subsystem. Disk I/O is supposed to finish in the finite time. --4oFiy8uZhEhZB7V2 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJR3nydAAoJEJDCuSvBvK1B4GkP/32Hyjeqm6RYGgcKW1fBYuaA mo0Eh2zwkYxAr+c3NPj0Tcd/c06cc+bwjbTbtlL1uiyNZNvK1Hb8LnlatfxkubGs 9TZ+T1NzyPJae4uPGMZMcegOVJoeDh9liAcbkTGj6gwoMklAbTGwVYQc9Xs1FhiY hP/D8/ITrnRiltFi7OwAtpqJGjRvz34iSl9viroZcv9+K8uGNobCXuiE2J35eh/p a2F13Lt6kiPSlynkaL6xwabchTgtsAOjNNvtFAeR5D4125nWGijFU9cpU/iFobpH 3BZZ6WbgMTNhxXEqjEbtTKx8YyUEjrRE+p0MoJ2RpgjKXtnhxTtneLyjHiC2Xmam Uk/1IdIhyUnexqBFcN/wGAq36Bd95/+VpwzrIDeWM3E8UoMrtn6oLnV5lbC+oFRA J6FceZIuk6d7RYFLo5tQGzZ+Nuwi95VpO6gT8hCfWxbGVJ3WX72aWlq5qC2tmIT5 TyCDLYAFmFE6sna3xgwLVwnUlGdyV10kSWJLY3VITv3oFn9H9I335+sQBdujpcd+ HflR111vRUabZ3H0FJ7c0wO+jNBPhxYw97K7ckqZsN9ww+vaHPPObImi/oEGVafL W0PYcMFZ1mMfh4xq5GPjrAwz2o4uVe0ub3opadWfx/s2DvYhr82uMgZRtDLlF+s8 BczIcfB67jsn8SxETBRW =xjg4 -----END PGP SIGNATURE----- --4oFiy8uZhEhZB7V2--