Date: Thu, 11 Jul 2013 22:50:08 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Scott Long <scottl@samsco.org> Cc: Adrian Chadd <adrian@freebsd.org>, freebsd-current <freebsd-current@freebsd.org> Subject: Re: hacking - aio_sendfile() Message-ID: <20130711195008.GU91021@kib.kiev.ua> In-Reply-To: <292A629B-7650-4845-83A2-E2C985D9C346@samsco.org> References: <CAJ-Vmo=icr6bda%2BWMNUarc3WbdqJ%2BMdauX6kByxxdTx8oSovBg@mail.gmail.com> <20130711061753.GK91021@kib.kiev.ua> <D722FEF8-158D-44C1-A701-87095750CF07@samsco.org> <20130711184835.GS91021@kib.kiev.ua> <292A629B-7650-4845-83A2-E2C985D9C346@samsco.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--eyadjJPLjITSyz6C Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Jul 11, 2013 at 12:04:57PM -0700, Scott Long wrote: >=20 > On Jul 11, 2013, at 11:48 AM, Konstantin Belousov <kostikbel@gmail.com> w= rote: >=20 > > On Thu, Jul 11, 2013 at 11:44:32AM -0700, Scott Long wrote: > >>=20 > >> On Jul 10, 2013, at 11:17 PM, Konstantin Belousov <kostikbel@gmail.com= > wrote: > >>=20 > >>> On Wed, Jul 10, 2013 at 04:36:23PM -0700, Adrian Chadd wrote: > >>>> Hiya, > >>>>=20 > >>>> I've started writing an aio_sendfile() syscall. > >>>>=20 > >>>> http://people.freebsd.org/~adrian/ath/20130710-aio-sendfile-3.diff > >>>>=20 > >>>> Yes, the diff is against -HEAD and not stable/9. > >>>>=20 > >>>> It's totally horrible, hackish and likely bad. I've only done some > >>>> very, very basic testing to ensure it actually works; i haven't at a= ll > >>>> stress tested it out yet. It's also very naive - I'm not at all doing > >>>> any checks to see whether I can short-cut to do the aio there and > >>>> then; I'm always queuing the sendfile() op through the worker thread= s. > >>>> That's likely stupid and inefficient in a lot of cases, but it at > >>>> least gets the syscall up and working. > >>> Yes, it is naive, but for different reason. > >>>=20 > >>> The kern_sendfile() is synchronous function, it only completes after > >>> the other end of the network communication allows it. This means > >>> that calling kern_sendfile() from the aio thread blocks the thread > >>> indefinitely by unbounded sleep. > >>=20 > >>=20 > >> No, kern_sendfile is async unless you specify the SF_SYNC hack flag. > >> Otherwise, it'll fill the socket buffer and then return immediately, u= nless > >> the socket buffer is full and the socket is set to blocking mode. Tha= t's > >> outside the scope, as I said in my previous email. > >=20 > > You do not understand what I said, please re-read both my mail and code > > before replying. Implementing aio_sendfile() as proposed would create > > yet another possibility of indefinitely block all processes using aio. >=20 > I'm lost, maybe I missed some emails? I see a set of emails where you in= correctly > state that kern_sendfile() will always call sbwait(), and then you backtr= ack on that > and claim that it's unacceptable to enforce that SS_NBIO be used for aio = operations. > I apologize if I'm missing something here. Can you cite my exact text where I claimed that kern_sendfile() always calls sbwait ? I wrote about this explicitely, stating that it is very easy to make kern_sendfile() sleep for the socket buffer space, and the duration of the sleep is user-controllable. As result, it allows to hang all processes doing aio calls, since aio thread pool is finite. I am sorry for retyping this and stealing your time by repeating. Making the kern_sendfile() to behave from the aio context as if the SS_NBIO was set on the socket contradicts the behaviour of other aio operations. E.g. aio_read and aio_write do not perform short reads and writes to not block the aio daemon threads (which is the cause of buggy behaviour of existing aio syscalls on sockets). More, I do not think that setting SS_NBIO is enough to prevent the blocking of aio threads in kern_sendfile(). The send socket buffer is locked exclusively by kern_sendfile(). Other thread which entered sendfile(2) and was deliberately put to sleep on the low watermark, still owns the so_snd sx. This means that aio threads trying to do kern_sendfile() on this socket would be also blocked, for duration controlled by other end. That said, even assuming SS_NBIO is always enforced and other sleep points are identified and worked around, the only benefit of such implementation comparing with the direct sendfile(2) call would be preventing the use of the calling thread context for disk i/o. FreeBSD recently gained aio_mlock(2) which allows to get the same result in non-hackish way. --eyadjJPLjITSyz6C Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJR3wxvAAoJEJDCuSvBvK1BRaUP/Rcn/GYBzl0LPFiM21JxC4Fg Yy9as3pbFc2YycwBXboCYMVZh7zZzRdAkT2yHkc9fifZG+vkvmGMtw9uQXlUaEEq 5A4fUz7PY7WNusJM/GLZOR0dGiOWh+63XMzZ6zqTGi/89GyEU4gscvJs3SZZvpAS dMGKP5LXZyCz8xiRrbOJOvj0VdpQNs4L2+nxyhr1lirQKNhU9+t5/GThPAK0WwQQ oCQgAkBfOLD0SEgU2A+ca3oSlZ3pT6sKVolVSNnOtuRuUsrF5nU3jGrH5cs+Vqi+ E8ncvCvXTklGEfnY1Fk0M6xHZmSGigPZZsSNY3dKc/muFiT5O9h78P01Dr7WAwnl mQ8AQAqNdyTij+d+kjN3Sitkqa0cdOQUvent4E7YIM/ljafEvvsJEVgjFA6SLXI7 eUCVzyfaWT1xth1ozh556lpDLdcE7gltCsfVZvjuy2ZTwmWV/jNEzQIxaDRH/flC 13VA7vghAcHQDJWAP+UAJSY595Tw2ljYMkG92E6EoEiKT4z4KqKg/HT0m5XKs8s7 jSjRaG+XAZWyrDMNgzeGvNOVQ+cmLXkH42WSIwZozliYBkjRmcPk8m3HomvnEzRR G+KwdW21tS/oMYIeQNGOkZBIGzPbJEAP5QutnCfe8vwBETIiCSTaaKnwzuMfDvIr iI+zCIrCWLU+dO1rcZrp =jP1Y -----END PGP SIGNATURE----- --eyadjJPLjITSyz6C--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130711195008.GU91021>