From owner-freebsd-arch@freebsd.org Tue Jan 26 22:28:49 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2739AA46DD1; Tue, 26 Jan 2016 22:28:49 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from smtp.vangyzen.net (hotblack.vangyzen.net [IPv6:2607:fc50:1000:7400:216:3eff:fe72:314f]) by mx1.freebsd.org (Postfix) with ESMTP id 0F7DC3B6; Tue, 26 Jan 2016 22:28:49 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from sweettea.beer.town (unknown [76.164.8.130]) by smtp.vangyzen.net (Postfix) with ESMTPSA id 668945648E; Tue, 26 Jan 2016 16:28:48 -0600 (CST) From: Eric van Gyzen Subject: Hot-Plug PCIe Support X-Enigmail-Draft-Status: N1110 To: freebsd-hackers@freebsd.org, freebsd-current@freebsd.org, freebsd-arch@freebsd.org Reply-To: freebsd-hackers@freebsd.org Message-ID: <56A7F31C.3030209@FreeBSD.org> Date: Tue, 26 Jan 2016 16:28:44 -0600 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jan 2016 22:28:49 -0000 FreeBSD Folks: I am currently scoping the effort to add hot-plug PCIe support to FreeBSD. Is anyone else currently working on this or aware of any design, code, or other effort available outside the tree? FYI, here are perhaps the most interesting references I could find: https://wiki.freebsd.org/PCIHotplug https://wiki.freebsd.org/IdeasPage#Implementing_PCI-Hotplug_and_ExpressCard_support https://lists.freebsd.org/pipermail/freebsd-current/2015-April/055290.html https://lists.freebsd.org/pipermail/freebsd-ia32/2010-February/date.html Please reply on freebsd-hackers@FreeBSD.org to minimize cross-posting. Thanks in advance. Eric From owner-freebsd-arch@freebsd.org Tue Jan 26 22:31:56 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A6C41A6E09B for ; Tue, 26 Jan 2016 22:31:56 +0000 (UTC) (envelope-from oliver.pinter@hardenedbsd.org) Received: from mail-wm0-x22b.google.com (mail-wm0-x22b.google.com [IPv6:2a00:1450:400c:c09::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 43DA1A49 for ; Tue, 26 Jan 2016 22:31:56 +0000 (UTC) (envelope-from oliver.pinter@hardenedbsd.org) Received: by mail-wm0-x22b.google.com with SMTP id n5so154730711wmn.0 for ; Tue, 26 Jan 2016 14:31:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hardenedbsd-org.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=cU3Ht0C5f6v3qQt5Z2LLH7BXA49Ea6D5VS/5ITDqqdU=; b=fwtswwk4KlO2Qql8en+AreqfB/XRXAg5hJl/JoYSjXcslJFP18NXrSnRZAJ3pVNxcc 8uXx650/ea66RNOU78zIfJ3leHVMqhB/DP2xjgzKjqhusilu5UyG+XAvBG3qbgLehyaz xQ1RKlIKaEPE1KINJBX8sjys/J8LVlNHl9O1G1DzZ32bjKhMjEbsYyB9tLxY2kWkpm7W r/5SMgV6Ixiw1GsX0LrQIw320xrb0F60e9OYpwUdYdCmvxP7cnNRMOkTjEz1+lT31GFe arEolzvHALwp5sZ3fKEOqux6fjNXXcfQ7ijlcjYF8OorQn7nI5wMQ89Hu0JXdAodUtGk a5hA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=cU3Ht0C5f6v3qQt5Z2LLH7BXA49Ea6D5VS/5ITDqqdU=; b=W1cfKWMivOmTZeONxc8YOJN+oCvOThrmpUMHhKZsy0tprMknaa9kQPJKdIFL57nVx7 5WIztC6bSkRZR0IPgvdM7x9gEn8UaGxtllQ+E9VVdMEOPVpCAB4GzOWjDi57v55G2b+K CBKc01OWmwnub5AELRcNBjbotmtw2DfzEQFVRWVgh6JOwIJyF2+0lx6992DQd+wc0NF9 wwrYtlOLr+46zcrA3z8PgjixUzlTKddymiSMLYGbLfNaQrAeku7pOYz0lfvq7J+6CerM HpnUGe7qFpNAZeK3PVCxpduc1otM84LtukMbmtV/TrCn1rH8/jJDIh/L8UJw+lSrrazD U97A== X-Gm-Message-State: AG10YOSXS/tY3JVOeHgB/+8kNa2B2hBgd7vxP4r6rnPJl4FwO/LjZdmI8vdMZkJYCTZb+0WRJO0l9MLns2VgF+Me MIME-Version: 1.0 X-Received: by 10.28.228.87 with SMTP id b84mr25883753wmh.81.1453847514708; Tue, 26 Jan 2016 14:31:54 -0800 (PST) Received: by 10.194.82.6 with HTTP; Tue, 26 Jan 2016 14:31:54 -0800 (PST) In-Reply-To: <56A7F31C.3030209@FreeBSD.org> References: <56A7F31C.3030209@FreeBSD.org> Date: Tue, 26 Jan 2016 23:31:54 +0100 Message-ID: Subject: Re: Hot-Plug PCIe Support From: Oliver Pinter To: freebsd-hackers@freebsd.org Cc: freebsd-current@freebsd.org, freebsd-arch@freebsd.org, jmg@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Jan 2016 22:31:56 -0000 On 1/26/16, Eric van Gyzen wrote: > FreeBSD Folks: > > I am currently scoping the effort to add hot-plug PCIe support to > FreeBSD. Is anyone else currently working on this or aware of any > design, code, or other effort available outside the tree? > > FYI, here are perhaps the most interesting references I could find: > > https://wiki.freebsd.org/PCIHotplug > > https://wiki.freebsd.org/IdeasPage#Implementing_PCI-Hotplug_and_ExpressCard_support > > https://lists.freebsd.org/pipermail/freebsd-current/2015-April/055290.html > > https://lists.freebsd.org/pipermail/freebsd-ia32/2010-February/date.html > > Please reply on freebsd-hackers@FreeBSD.org to minimize cross-posting. > > Thanks in advance. Hi! Added John-Mark to the CC, if I'm not wrong, I stared to play with them. > > Eric > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > From owner-freebsd-arch@freebsd.org Wed Jan 27 01:39:10 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5D9F5A6FA07 for ; Wed, 27 Jan 2016 01:39:10 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4E466153F for ; Wed, 27 Jan 2016 01:39:10 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 4A933A6FA06; Wed, 27 Jan 2016 01:39:10 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A1ECA6FA05 for ; Wed, 27 Jan 2016 01:39:10 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0B7F4153D for ; Wed, 27 Jan 2016 01:39:10 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 4D232B917 for ; Tue, 26 Jan 2016 20:39:08 -0500 (EST) From: John Baldwin To: arch@freebsd.org Subject: Refactoring asynchronous I/O Date: Tue, 26 Jan 2016 17:39:03 -0800 Message-ID: <2793494.0Z1kBV82mT@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 26 Jan 2016 20:39:08 -0500 (EST) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 01:39:10 -0000 You may have noticed some small cleanups and fixes going into the AIO code recently. I have been working on a minor overhaul of the AIO code recently and the recent changes have been trying to reduce the diff down to the truly meaty changes so they are easier to review. I think things are far enough along to start on the meaty bits. The current AIO code is a bit creaky and not very extensible. It forces all requests down via the existing fo_read/fo_write file ops so that all requests are inherently synchronous even if the underlying file descriptor could support async operation. This also makes cancellation more fragile as you can't cancel a job that is stuck sleeping in one of the AIO daemons. The original motivation for my changes is to support efficient zero-copy receive for TOE using Chelsio T4/T5 adapters. However, read() is ill suited to this type of workflow. Various efforts in the past have tried using page flipping (the old ZERO_COPY_SOCKETS code which required custom ti(4) firmware) which only works if you can get things lined up "just right" (e.g. page-aligned and sized buffers, custom firmware on your NIC, etc.) or introducing new APIs that replace read/write (such as IO-Lite). The primary issue with read() of course is that the NIC DMAs data to one place and later userland comes along and tells the kernel where it wants the data. The issue with introducing a new API like IO-Lite is convincing software to use it. However, aio_read() is an existing API that can be used to queue user buffers in advance. In particular, you can use two buffers to ping-pong similar to the BPF zero-copy code where you queue two buffers at the start and requeue each completed buffer after consuming its data. In theory the Chelsio driver can "peek" into the request queue for a socket and schedule the pending requests for zero copy receive. However, doing that requires a way for allowing the driver to "claim" certain requests and support cancelling them, completing them, etc. To facilitate this use case I decided to rework the AIO code to use a model closer to the I/O Request Packets (Irps) that Windows drivers use. In particular, when a Windows driver decides to queue a request so that it can be completed later, it has to install a cancel routine that is responsible for cancelling a queued request. To this end, I have reworked the AIO code as such: 1) File descriptor types are now the "drivers" for AIO requests rather than the AIO code. When an AIO request for an fd is queued (via aio_read/write, etc.) a new file op (fo_aio_queue()) is called to queue or handle the request. This method is responsible for completeing the request or queueing it to be completed later. Currently, a default implementation of this method which queues the job to the existing AIO daemon pool for fo_read/fo_write is provided, but file types can override that with more specific behavior if desired. 2) The AIO code is now a library of calls for manipulating AIO requests. Functions to manage cancel routines, mark AIO requests as cancelled or completed, and schedule handler functions to run in an AIO daemon context are provided. 3) Operations that choose to queue an AIO request while waiting for a suitable resource to service it (CPU time, data to arrive on a socket, etc.) are required to install a cancel routine to handle cancellation of a request due to aio_cancel() or the exit or exec of the owning process. This allows the type-specific queueing logic to be self-contained without the AIO code having to know about all the possible queue states of an AIO request. In my current implementation I use the "default" fo_aio_queue method for most file types. However, sockets now use a private pool of AIO kprocs, and they also service sockets (rather than jobs). This means that when a socket becomes ready for either read or write, it queues a task for that socket buffer to the socket AIO daemon pool. That task will complete as many requests as possible for that socket buffer (ensuring that there are no concurrent AIO operations on a given socket). It is also able to use MSG_NOWAIT to avoid blocking even for blocking sockets. One thing I have not yet done is move the physio fast-path out of vfs_aio.c and into the devfs-specific fileops, but that can easily be done with a custom fo_aio_queue op for the devfs file ops. I believe that this approach also permits other file types to provide more suitable AIO handling when suitable. For the Chelsio use case I have added a protocol hook to allow a given protocol to claim AIO requests instead of letting them be handled by the generic socket AIO fileop. This ends up being a very small change, and the Chelsio-specific logic can live in the TOM module using the AIO library calls to service the AIO requests. My current WIP (not including the Chelsio bits, they need to be forward ported from an earlier prototype) is available on the 'aio_rework' branch of freebsd in my github space: https://github.com/freebsd/freebsd/compare/master...bsdjhb:aio_rework Note that binding the AIO support to a new fileop does mean that the AIO code now becomes mandatory (rather than optional). We could perhaps make the system calls continue to be optional if people really need that, but the guts of the code will now need to always be in the kernel. I'd like to hear what people think of this design. It needs some additional cleanup before it is a commit candidate (and I'll see if I can't split it up some more if we go this route). -- John Baldwin From owner-freebsd-arch@freebsd.org Wed Jan 27 01:58:40 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 39F25A6E132; Wed, 27 Jan 2016 01:58:40 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-io0-x235.google.com (mail-io0-x235.google.com [IPv6:2607:f8b0:4001:c06::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 07CA91E9E; Wed, 27 Jan 2016 01:58:40 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-io0-x235.google.com with SMTP id f81so4189157iof.0; Tue, 26 Jan 2016 17:58:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=7SX96HW3bKGb9Rngz1txKYWjJA1r98yiO/njSAL+Lwc=; b=rORGBnJEE0zkaeL6/TkzJnTQivMRso7fikBQO00Ec8tCf8WHHYQA6uXibuT5grivLH CK5niSZOLhvdSaKwMXpPcWDfDOuPGhrS2YmzVBSsAm7z3R7611JBObecozlQnYj1jRUs jcJpLvJERC9CjPhTnzXJ8SpvZBxmmPIj4AeflIMtZin/oXXpvVCNT1vweqXOqbOstMTz WKqRIM5NNtpAGOA7O4jHRaNztJPppFKZyVctIab8B3uN2b9WEeqMaO13wN6iaFgprpG6 oLpGwXeFGgsQC6zzZLaxx6m1a0PU/p9xy+e+XMm8RI5Diwt12omu3QvE8nYiCh31Za8n 9Vvw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=7SX96HW3bKGb9Rngz1txKYWjJA1r98yiO/njSAL+Lwc=; b=giCv5VtDc0L7wIdbADT3uhBOWcwx2Xt+eqAYKCQjfmtNVPKbTrlIxQNRl7w2GPd/uu LdD+cRs3XijjPy72Q7VdbGn0g5W/oNjPUcxQmPPfjJ+o+t/rZYLgiXVqdgclpSTdoBnw af2ULw/IvkbWXUkog9Qvd7fcwduZgVoc3f+9uqi14KaYgS+MhqTNq1TV7y6QJ7LaZs5m nyH4Y+8rooWDDBRk5k/3Bf1JL+nk/igXWhLah30tI3mqHbmrrCEWsB2Fr/FBgo8zZ+6b tU+ycfKi/kzNX/GZxbTvrBFLRsLP71ctRr1cC3QUW9paX+ulS7K29ukFIijfAH4V0Ifz AwJg== X-Gm-Message-State: AG10YOTXA9q4Fotwva7auxPVsK0ZfYPMV07Yr/7LKlAclYmRODYCwBW5ReTw0vj7fHhqkZHAOC1HCNvc0iAQBw== MIME-Version: 1.0 X-Received: by 10.107.10.217 with SMTP id 86mr25091254iok.75.1453859919407; Tue, 26 Jan 2016 17:58:39 -0800 (PST) Received: by 10.36.121.16 with HTTP; Tue, 26 Jan 2016 17:58:39 -0800 (PST) In-Reply-To: <56A7F31C.3030209@FreeBSD.org> References: <56A7F31C.3030209@FreeBSD.org> Date: Tue, 26 Jan 2016 17:58:39 -0800 Message-ID: Subject: Re: Hot-Plug PCIe Support From: Adrian Chadd To: "freebsd-hackers@freebsd.org" Cc: freebsd-current , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 01:58:40 -0000 please grab the pciehp work that jmg has done and push it along to completion. Pretty please in fact. -a On 26 January 2016 at 14:28, Eric van Gyzen wrote: > FreeBSD Folks: > > I am currently scoping the effort to add hot-plug PCIe support to > FreeBSD. Is anyone else currently working on this or aware of any > design, code, or other effort available outside the tree? > > FYI, here are perhaps the most interesting references I could find: > > https://wiki.freebsd.org/PCIHotplug > > https://wiki.freebsd.org/IdeasPage#Implementing_PCI-Hotplug_and_ExpressCard_support > > https://lists.freebsd.org/pipermail/freebsd-current/2015-April/055290.html > > https://lists.freebsd.org/pipermail/freebsd-ia32/2010-February/date.html > > Please reply on freebsd-hackers@FreeBSD.org to minimize cross-posting. > > Thanks in advance. > > Eric > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" From owner-freebsd-arch@freebsd.org Wed Jan 27 03:17:34 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D3539A6FC55 for ; Wed, 27 Jan 2016 03:17:34 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B6827129B for ; Wed, 27 Jan 2016 03:17:34 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id B48BEA6FC54; Wed, 27 Jan 2016 03:17:34 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B42A5A6FC53 for ; Wed, 27 Jan 2016 03:17:34 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-ig0-x236.google.com (mail-ig0-x236.google.com [IPv6:2607:f8b0:4001:c05::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 816B3129A; Wed, 27 Jan 2016 03:17:34 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: by mail-ig0-x236.google.com with SMTP id z14so74408750igp.1; Tue, 26 Jan 2016 19:17:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=1UHAjXEnM0GBuHjIWYfLMvHXybPoVj8/61NS+/55vIY=; b=jwLXKOR+f8fCYJEOKx8cLSppEG6f5lHyZIMuNL2rPYUAE29Kv6Q8MTZpay3DZqoQJr frze92FccjOi0l74UiyqPxip+Kxm+VHrMtQp5O1fMsrJEd1qutaRCt1Iut5nOBM5r9fc R6JpJROJhDcSd2OmdAvvzRF2DjB2S503NTLRhCgGigXX8tN1uDDsbwX+OuX74ab0V0ms +iKzwsPHAwcMInQ+MJjkMUzzxJnw1YRU+wy9LmJD8q58FkXrdHdoz/32iu37Bn7F2IUI jYvEF7nVKvatbzOD4Lnhr+HI5NN3d8XiksCOQ3yiI+PoN6gSyMRTtTJUHqtEcqdSEhMT 1Jug== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=1UHAjXEnM0GBuHjIWYfLMvHXybPoVj8/61NS+/55vIY=; b=G01QkbCHpFWiIm39mL6J4Q3CdqMlcDIG7a/ifkKN0Phji3UemYd4hmc04EYEsxfJdL 98aJ2Wij3R2yZBh3Lo/Qz4FMjW6PMC5Rtg1QqGvlUbgn+UA6ndr6Mmu/HKU2Bx56t7z2 SKH7wfZnyLejoICrGR6R5TDz6mAz2e37o3F8Q/j8F1+ZzVZnSmovenWu5+t1v1KV6jgY l/2tbc+pf9VCoTUF/zN5IGOQ/TFMiKyyjgMH0dUWFRcl/T4UINxWucBPYlu26eYb6BlL Wf7+191bZThAglnqr2SZ11Zif4OIpZXWvvApnhJ4lfZ8WefLJtF86zjuv5do5wHS2oFW yCdw== X-Gm-Message-State: AG10YOQ35Uv/oqCu6TOJdtHkCYT4lO+Qo10VDGnymfQXZCwMXRPNRxHK9m9FCI/eUBr8E4NXhyJatGihLdB0vQ== MIME-Version: 1.0 X-Received: by 10.50.137.41 with SMTP id qf9mr25255801igb.22.1453864653946; Tue, 26 Jan 2016 19:17:33 -0800 (PST) Received: by 10.36.121.16 with HTTP; Tue, 26 Jan 2016 19:17:33 -0800 (PST) In-Reply-To: <2793494.0Z1kBV82mT@ralph.baldwin.cx> References: <2793494.0Z1kBV82mT@ralph.baldwin.cx> Date: Tue, 26 Jan 2016 19:17:33 -0800 Message-ID: Subject: Re: Refactoring asynchronous I/O From: Adrian Chadd To: John Baldwin Cc: "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 03:17:35 -0000 On 26 January 2016 at 17:39, John Baldwin wrote: [snip] > > Note that binding the AIO support to a new fileop does mean that the AIO code > now becomes mandatory (rather than optional). We could perhaps make the > system calls continue to be optional if people really need that, but the guts > of the code will now need to always be in the kernel. > > I'd like to hear what people think of this design. It needs some additional > cleanup before it is a commit candidate (and I'll see if I can't split it up > some more if we go this route). So this doesn't change the direct dispatch read/write to a block device, right? That strategy path is pretty damned cheap. Also, why's it become mandatory? I thought we had support for optional fileops... -a From owner-freebsd-arch@freebsd.org Wed Jan 27 06:33:10 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 08EC2A6FDBB for ; Wed, 27 Jan 2016 06:33:10 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qg0-x232.google.com (mail-qg0-x232.google.com [IPv6:2607:f8b0:400d:c04::232]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BE7AA1D91 for ; Wed, 27 Jan 2016 06:33:09 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qg0-x232.google.com with SMTP id b35so157755218qge.0 for ; Tue, 26 Jan 2016 22:33:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=2tLKvgc5Lqk/nnUVNMtpszlr+5Hkfi6JQw6Fgperrqs=; b=n8G30lpPK7j7ZonD7r+nh3xt9mWdJYR/H3mAD1WVVq6sUGOtR+BsJ171vrBzM+Mnra ixmjQQnem40sc5I3PAase/JuRjGcbKjrvtt3XW6bEdLvdzXixTR+S34hAHW177n8De/Y Em83hNpO6xE6qeOa/xuh90bFiuRWzFBk8Vu0+oBk1QVwG2awLkgRvosEwMm72bX0/8EG u6R5RgkzIYYDK2oc/KO52T05mA/tlGqimu81PiZ3ydTpT568kdXp/nbJEO7ITTLx2wBy PP2F3EmSPDK6qa4ZQb5PRmbfInF4zqV7nf/g0R3/QPsiteHGByOYZxeNIjzdJzLGHdOw zPgA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=2tLKvgc5Lqk/nnUVNMtpszlr+5Hkfi6JQw6Fgperrqs=; b=DPH10Ddq0q56vFU+zyYuilzILHGX4pGN80XkDQYZa8LhrZWendE19pzN/S1WQIoE6y RNMBkrEpxlq38zvwb4KXPdKsW+NoUf7poKf7xp7gJtfnoVHZ8HNEc/Y+eupAgsER3PB/ LSe2JbPDKZvszoqQIPvq/7NuKlTuceHhD5xev27J80XFBbJiR3vB0p1YxO8IvFKpXhXs FP4zTt33DEKKjrUvN8ci+fFU24UMEyyZOs1ZRrcmaOx31L0W6IxEz0C5TYYmIDA/pQy4 rfpmXSx7aUWD6aeYRHWx8f6zf0SmYmqBoOdgQQu5XP0LsFrB7Bh/o/smK2KDKxme3xFK Gfrg== X-Gm-Message-State: AG10YOQgSyJrEqBJWIrCest46rGUA2jOr28RGReeSjo0aKRmF9AXeH9BlA8619y1LOolGu0cMeR8ftJwvhj+KA== MIME-Version: 1.0 X-Received: by 10.140.109.247 with SMTP id l110mr33079979qgf.52.1453876388814; Tue, 26 Jan 2016 22:33:08 -0800 (PST) Sender: wlosh@bsdimp.com Received: by 10.140.30.166 with HTTP; Tue, 26 Jan 2016 22:33:08 -0800 (PST) X-Originating-IP: [50.253.99.174] In-Reply-To: References: <56A7F31C.3030209@FreeBSD.org> Date: Tue, 26 Jan 2016 23:33:08 -0700 X-Google-Sender-Auth: 7-XXkam3y4UwcxutlMyEoBYDEAk Message-ID: Subject: Re: Hot-Plug PCIe Support From: Warner Losh To: Adrian Chadd Cc: "freebsd-hackers@freebsd.org" , freebsd-current , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 06:33:10 -0000 It's mostly done, but needs a careful review by PCI domain experts. I've been doing it a little at a time, but have been crunched for time. Since $DAYJOB doesn't care about hot plug, it's a lower priority than all the things related or semi-related to it. Warner On Tue, Jan 26, 2016 at 6:58 PM, Adrian Chadd wrote: > please grab the pciehp work that jmg has done and push it along to > completion. Pretty please in fact. > > > -a > > > On 26 January 2016 at 14:28, Eric van Gyzen wrote: > > FreeBSD Folks: > > > > I am currently scoping the effort to add hot-plug PCIe support to > > FreeBSD. Is anyone else currently working on this or aware of any > > design, code, or other effort available outside the tree? > > > > FYI, here are perhaps the most interesting references I could find: > > > > https://wiki.freebsd.org/PCIHotplug > > > > > https://wiki.freebsd.org/IdeasPage#Implementing_PCI-Hotplug_and_ExpressCard_support > > > > > https://lists.freebsd.org/pipermail/freebsd-current/2015-April/055290.html > > > > https://lists.freebsd.org/pipermail/freebsd-ia32/2010-February/date.html > > > > Please reply on freebsd-hackers@FreeBSD.org to minimize cross-posting. > > > > Thanks in advance. > > > > Eric > > _______________________________________________ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to " > freebsd-current-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-arch@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > From owner-freebsd-arch@freebsd.org Wed Jan 27 10:52:15 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 601ACA6FFA5 for ; Wed, 27 Jan 2016 10:52:15 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 51A5F13BB for ; Wed, 27 Jan 2016 10:52:15 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: by mailman.ysv.freebsd.org (Postfix) id 4F4ACA6FFA4; Wed, 27 Jan 2016 10:52:15 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4EE84A6FFA3 for ; Wed, 27 Jan 2016 10:52:15 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 14FB013B8; Wed, 27 Jan 2016 10:52:15 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1aONhl-000OI6-Hg; Wed, 27 Jan 2016 13:52:05 +0300 Date: Wed, 27 Jan 2016 13:52:05 +0300 From: Slawa Olhovchenkov To: John Baldwin Cc: arch@freebsd.org Subject: Re: Refactoring asynchronous I/O Message-ID: <20160127105205.GP37895@zxy.spb.ru> References: <2793494.0Z1kBV82mT@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2793494.0Z1kBV82mT@ralph.baldwin.cx> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 10:52:15 -0000 On Tue, Jan 26, 2016 at 05:39:03PM -0800, John Baldwin wrote: > The original motivation for my changes is to support efficient zero-copy > receive for TOE using Chelsio T4/T5 adapters. However, read() is ill I undertuns that not you work, but: what about (teoretical) async open/close/unlink/etc? From owner-freebsd-arch@freebsd.org Wed Jan 27 16:47:32 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5DE31A707FE; Wed, 27 Jan 2016 16:47:32 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3CE4B12D2; Wed, 27 Jan 2016 16:47:32 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id C0D97B93C; Wed, 27 Jan 2016 11:47:30 -0500 (EST) From: John Baldwin To: freebsd-arch@freebsd.org Cc: Warner Losh , Adrian Chadd , "freebsd-hackers@freebsd.org" , freebsd-current Subject: Re: Hot-Plug PCIe Support Date: Wed, 27 Jan 2016 08:47:27 -0800 Message-ID: <3676083.ysX5rbmR19@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) In-Reply-To: References: <56A7F31C.3030209@FreeBSD.org> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 27 Jan 2016 11:47:30 -0500 (EST) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 16:47:32 -0000 On Tuesday, January 26, 2016 11:33:08 PM Warner Losh wrote: > It's mostly done, but needs a careful review by PCI domain experts. I've > been doing it a little at a time, but have been crunched for time. Since > $DAYJOB doesn't care about hot plug, it's a lower priority than all the > things related or semi-related to it. As I noted in the review on phabricator, it needs to be rearranged. It currently puts a bunch of code in the PCI bus that instead belongs in the PCI-PCI bridge (all the interrupt handling, MSI, etc. are properties of the bridge and should be handled in their rather than magic fields in the pci_dinfo of the bridge in the parent PCI bus). It is not a lot of code and probably wouldn't take long to finish. I had been waiting to let jmg@ finish it. https://reviews.freebsd.org/D3932 > Warner > > > On Tue, Jan 26, 2016 at 6:58 PM, Adrian Chadd > wrote: > > > please grab the pciehp work that jmg has done and push it along to > > completion. Pretty please in fact. > > > > > > -a > > > > > > On 26 January 2016 at 14:28, Eric van Gyzen wrote: > > > FreeBSD Folks: > > > > > > I am currently scoping the effort to add hot-plug PCIe support to > > > FreeBSD. Is anyone else currently working on this or aware of any > > > design, code, or other effort available outside the tree? > > > > > > FYI, here are perhaps the most interesting references I could find: > > > > > > https://wiki.freebsd.org/PCIHotplug > > > > > > > > https://wiki.freebsd.org/IdeasPage#Implementing_PCI-Hotplug_and_ExpressCard_support > > > > > > > > https://lists.freebsd.org/pipermail/freebsd-current/2015-April/055290.html > > > > > > https://lists.freebsd.org/pipermail/freebsd-ia32/2010-February/date.html > > > > > > Please reply on freebsd-hackers@FreeBSD.org to minimize cross-posting. > > > > > > Thanks in advance. > > > > > > Eric > > > _______________________________________________ > > > freebsd-current@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > > To unsubscribe, send any mail to " > > freebsd-current-unsubscribe@freebsd.org" > > _______________________________________________ > > freebsd-arch@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" -- John Baldwin From owner-freebsd-arch@freebsd.org Wed Jan 27 17:52:34 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6877A70218 for ; Wed, 27 Jan 2016 17:52:34 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C5B361183 for ; Wed, 27 Jan 2016 17:52:34 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id C4AD9A70217; Wed, 27 Jan 2016 17:52:34 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C445FA70216 for ; Wed, 27 Jan 2016 17:52:34 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A61841182 for ; Wed, 27 Jan 2016 17:52:34 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id C375DB93C; Wed, 27 Jan 2016 12:52:32 -0500 (EST) From: John Baldwin To: Slawa Olhovchenkov Cc: arch@freebsd.org Subject: Re: Refactoring asynchronous I/O Date: Wed, 27 Jan 2016 09:52:12 -0800 Message-ID: <1723457.HAUy43H1XN@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) In-Reply-To: <20160127105205.GP37895@zxy.spb.ru> References: <2793494.0Z1kBV82mT@ralph.baldwin.cx> <20160127105205.GP37895@zxy.spb.ru> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 27 Jan 2016 12:52:32 -0500 (EST) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 17:52:34 -0000 On Wednesday, January 27, 2016 01:52:05 PM Slawa Olhovchenkov wrote: > On Tue, Jan 26, 2016 at 05:39:03PM -0800, John Baldwin wrote: > > > The original motivation for my changes is to support efficient zero-copy > > receive for TOE using Chelsio T4/T5 adapters. However, read() is ill > > I undertuns that not you work, but: what about (teoretical) async > open/close/unlink/etc? Implementing more asynchronous operations is orthogonal to this. It would perhaps be a bit simpler to implement these in the new model since most of the logic would live in a vnode-specific aio_queue method in vfs_vnops.c. However, the current AIO approach is to add a new system call for each async system call (e.g. aio_open()). You would then create an internal LIO opcode (e.g. LIO_OPEN). The vnode aio hook would then have to support LIO_OPEN requests and return the opened fd via aio_complete(). Async stat / open might be nice for network filesystems in particular. I've known of programs forking separate threads just to do open/fstat of NFS files to achieve the equivalent of aio_open() / aio_stat(). -- John Baldwin From owner-freebsd-arch@freebsd.org Wed Jan 27 17:52:36 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AC997A70223 for ; Wed, 27 Jan 2016 17:52:36 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 9AE60118A for ; Wed, 27 Jan 2016 17:52:36 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 99EF4A70222; Wed, 27 Jan 2016 17:52:36 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 99929A70221 for ; Wed, 27 Jan 2016 17:52:36 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7B9411188 for ; Wed, 27 Jan 2016 17:52:36 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 90060B989; Wed, 27 Jan 2016 12:52:35 -0500 (EST) From: John Baldwin To: Adrian Chadd Cc: "freebsd-arch@freebsd.org" Subject: Re: Refactoring asynchronous I/O Date: Wed, 27 Jan 2016 09:23:56 -0800 Message-ID: <4442858.kLAljyq4HJ@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) In-Reply-To: References: <2793494.0Z1kBV82mT@ralph.baldwin.cx> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 27 Jan 2016 12:52:35 -0500 (EST) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 17:52:36 -0000 On Tuesday, January 26, 2016 07:17:33 PM Adrian Chadd wrote: > On 26 January 2016 at 17:39, John Baldwin wrote: > > [snip] > > > > > Note that binding the AIO support to a new fileop does mean that the AIO code > > now becomes mandatory (rather than optional). We could perhaps make the > > system calls continue to be optional if people really need that, but the guts > > of the code will now need to always be in the kernel. > > > > I'd like to hear what people think of this design. It needs some additional > > cleanup before it is a commit candidate (and I'll see if I can't split it up > > some more if we go this route). > > So this doesn't change the direct dispatch read/write to a block device, right? > That strategy path is pretty damned cheap. No, that is the physio code that could be moved to be devfs specific. However, by exposing the aio queueing point to the fileops directly it allows for other fileops to implement direct dispatch without having to expose internals to the AIO code. > Also, why's it become mandatory? I thought we had support for optional > fileops... Some fileops are optional in that not all file descriptors implement them (e.g. not all fileops can be monitored by kevent() or mmap()ed), however we do not support kldloading a fileop. If you leave soo_aio_queue() in sys_socket.c then the kernel would not link without vfs_aio.c since the socket aio routine needs things like aio_complete, etc. You could perhaps split the system calls themselves out of vfs_aio.c into a separate sys_aio.c and make that loadable, but that would be fairly small. It also seems fairly pointless. The module is loadable because it was experimental and unfinished when it was first implemented. The current version is certainly not perfect, but it is much further along than the original code from 1997. I know folks have used the current code in production. -- John Baldwin From owner-freebsd-arch@freebsd.org Wed Jan 27 19:27:29 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D222A6F554 for ; Wed, 27 Jan 2016 19:27:29 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 36A771E7D for ; Wed, 27 Jan 2016 19:27:29 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 33B48A6F552; Wed, 27 Jan 2016 19:27:29 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 333E6A6F551 for ; Wed, 27 Jan 2016 19:27:29 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 220B91E7C for ; Wed, 27 Jan 2016 19:27:29 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [IPv6:::1]) by freefall.freebsd.org (Postfix) with ESMTP id 163DE189F for ; Wed, 27 Jan 2016 19:27:29 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id C916914640 for ; Wed, 27 Jan 2016 19:27:28 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id yp97_-UQ1MZX for ; Wed, 27 Jan 2016 19:27:25 +0000 (UTC) To: arch@FreeBSD.org DKIM-Filter: OpenDKIM Filter v2.9.2 mail.xzibition.com 9F6451463A From: Bryan Drewery Subject: build: FAST_DEPEND default (kernel first, then world) Openpgp: id=F9173CB2C3AAEA7A5C8A1F0935D771BB6E4697CF; url=http://www.shatow.net/bryan/bryan2.asc X-Enigmail-Draft-Status: N1110 Organization: FreeBSD Message-ID: <56A91A1E.4060304@FreeBSD.org> Date: Wed, 27 Jan 2016 11:27:26 -0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="KhfkIiPlw1xAunSpK83XREa2G5lOqUEHn" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 19:27:29 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --KhfkIiPlw1xAunSpK83XREa2G5lOqUEHn Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Hi, WITH_FAST_DEPEND makes 'make depend' mostly a NOP and pushes the dependency generation to the compilation phase which saves around 35% time in the kernel build. Various bugs have been found with the implementation and all but 1 are now fixed. The last is not including the dependency files via .depend (really .MAKE.DEPENDFILE) which make considers special when finding missing targets (it would be nice if make had something like a .dinclude to flag any file with this magic). I have a fix in testing that I plan to commit today. I would like to enable FAST_DEPEND by default in the kernel build. I have been asked to also remove the old mkdep version of 'make depend'. I am unsure on this as I think it may be useful for some people for the sake of debugging. However it does entail running the preprocessor and given you can compile the entire kernel in minutes usually it may not be worth keeping this as opposed to just compiling to generate them. Keeping a behavior of generating the depend files pre-compile or without-compile is my only real question here. I could likely work this without-compile behavior into the FAST_DEPEND method to avoid needing mkdep entirely. The old code is horrendous, slow, and easily wrong and hard to maintain. Some of the code that would be removed: sys/conf/kern.post.mk > .depend: .PRECIOUS ${SRCS} > .if ${MK_FAST_DEPEND} =3D=3D "no" > rm -f ${.TARGET}.tmp > # C files > ${MAKE} -V CFILES_NORMAL -V SYSTEM_CFILES -V GEN_CFILES | \ > CC=3D"${_MKDEPCC}" xargs mkdep -a -f ${.TARGET}.tmp ${CFLAG= S} > ${MAKE} -V CFILES_CDDL | \ > CC=3D"${_MKDEPCC}" xargs mkdep -a -f ${.TARGET}.tmp ${ZFS_C= FLAGS} \ > ${FBT_CFLAGS} ${DTRACE_CFLAGS} > ${MAKE} -V CFILES_LINUXKPI | \ > CC=3D"${_MKDEPCC}" xargs mkdep -a -f ${.TARGET}.tmp \ > ${CFLAGS} ${LINUXKPI_INCLUDES} > ${MAKE} -V CFILES_OFED -V CFILES_MLX5 | \ > CC=3D"${_MKDEPCC}" xargs mkdep -a -f ${.TARGET}.tmp \ > ${CFLAGS} ${OFEDINCLUDES} > # Assembly files > ${MAKE} -V SFILES_NORMAL | \ > CC=3D"${_MKDEPCC}" xargs mkdep -a -f ${.TARGET}.tmp ${ASM_C= FLAGS} > ${MAKE} -V SFILES_CDDL | \ > CC=3D"${_MKDEPCC}" xargs mkdep -a -f ${.TARGET}.tmp ${ZFS_A= SM_CFLAGS} > mv ${.TARGET}.tmp ${.TARGET} > .else > { \ > echo '.for __dependfile in $${DEPENDFILES_OBJS}'; \ > echo '.sinclude "$${__dependfile}"'; \ > echo '.endfor'; \ > } > ${.TARGET} > .endif (the for loop is a pending fix for the .depend issue) sys/conf/kern.pre.mk > .if ${MK_FAST_DEPEND} =3D=3D "no" && (make(depend) || make(kernel-depen= d)) =20 >=20 > # This hack lets us use the ipfilter code without spamming a new > # include path into contrib'ed source files. > INCLUDES+=3D -I$S/contrib/ipfilter >=20 > # ... and the same for ath > INCLUDES+=3D -I$S/dev/ath -I$S/dev/ath/ath_hal -I$S/contrib/dev/ath/ath= _hal >=20 > # ... and the same for the NgATM stuff > INCLUDES+=3D -I$S/contrib/ngatm >=20 > # ... and the same for vchiq > INCLUDES+=3D -I$S/contrib/vchiq >=20 > # ... and the same for twa > INCLUDES+=3D -I$S/dev/twa >=20 > # ... and the same for cxgb and cxgbe > INCLUDES+=3D -I$S/dev/cxgb -I$S/dev/cxgbe >=20 > .endif As for world, it is far larger and impacts ports. So I am still taking it slow there. I do want to enable it by default and make the same decisions there in the future. --=20 Regards, Bryan Drewery --KhfkIiPlw1xAunSpK83XREa2G5lOqUEHn Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJWqRoeAAoJEDXXcbtuRpfP+fcH/3meIRYw8XDg90LHLR7/kobr u0P+PBH1r4Ux9kR8PebwGu1k8sB3q9YZWyQBL8Izy8FwGqO+v+toGYwX1R9EiSA2 7cgy/Q1uzs5vxnXSVl2gkmWIHBFEExWFy84rJPC6gSLS1nLAf2KSEmpKo7WvZKx5 Q6hif3yeh7UFPqAh5lZ1Fp8w+CK3bUm2EsfbjV5O53oyX0GNuewJ0WTXE7G9wOfO XijzZXdVms+yBCNpHKh7WCjuEDKdi8QzVnaOO2SP1zUSM7rBMRvZPbMG/9JDcIg2 2bGZ/dDpEYIdH5nBpx/ixR3njF/2Q1E7JbHxy3iYxDhjJPEqvBfdvpCsHf63MlQ= =cSh6 -----END PGP SIGNATURE----- --KhfkIiPlw1xAunSpK83XREa2G5lOqUEHn-- From owner-freebsd-arch@freebsd.org Wed Jan 27 21:04:24 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D7E5DA6FBF7 for ; Wed, 27 Jan 2016 21:04:24 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C75C215E1 for ; Wed, 27 Jan 2016 21:04:24 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: by mailman.ysv.freebsd.org (Postfix) id C4D01A6FBF6; Wed, 27 Jan 2016 21:04:24 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C37D2A6FBF5 for ; Wed, 27 Jan 2016 21:04:24 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7F5AA15DF; Wed, 27 Jan 2016 21:04:24 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1aOXGG-000Ci2-UJ; Thu, 28 Jan 2016 00:04:20 +0300 Date: Thu, 28 Jan 2016 00:04:20 +0300 From: Slawa Olhovchenkov To: John Baldwin Cc: arch@freebsd.org Subject: Re: Refactoring asynchronous I/O Message-ID: <20160127210420.GZ88527@zxy.spb.ru> References: <2793494.0Z1kBV82mT@ralph.baldwin.cx> <20160127105205.GP37895@zxy.spb.ru> <1723457.HAUy43H1XN@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1723457.HAUy43H1XN@ralph.baldwin.cx> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 21:04:24 -0000 On Wed, Jan 27, 2016 at 09:52:12AM -0800, John Baldwin wrote: > On Wednesday, January 27, 2016 01:52:05 PM Slawa Olhovchenkov wrote: > > On Tue, Jan 26, 2016 at 05:39:03PM -0800, John Baldwin wrote: > > > > > The original motivation for my changes is to support efficient zero-copy > > > receive for TOE using Chelsio T4/T5 adapters. However, read() is ill > > > > I undertuns that not you work, but: what about (teoretical) async > > open/close/unlink/etc? > > Implementing more asynchronous operations is orthogonal to this. It > would perhaps be a bit simpler to implement these in the new model > since most of the logic would live in a vnode-specific aio_queue > method in vfs_vnops.c. However, the current AIO approach is to add a > new system call for each async system call (e.g. aio_open()). You > would then create an internal LIO opcode (e.g. LIO_OPEN). The vnode > aio hook would then have to support LIO_OPEN requests and return the > opened fd via aio_complete(). Async stat / open might be nice for > network filesystems in particular. I've known of programs forking > separate threads just to do open/fstat of NFS files to achieve the > equivalent of aio_open() / aio_stat(). Some problem exist for open()/unlink/rename/etc -- you can't use fd-related semantic. From owner-freebsd-arch@freebsd.org Wed Jan 27 22:56:43 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A013AA70446 for ; Wed, 27 Jan 2016 22:56:43 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8F370D8F for ; Wed, 27 Jan 2016 22:56:43 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 8BE5BA70445; Wed, 27 Jan 2016 22:56:43 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8B8A3A70444 for ; Wed, 27 Jan 2016 22:56:43 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6E27FD8D for ; Wed, 27 Jan 2016 22:56:43 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id DF5DDB990; Wed, 27 Jan 2016 17:56:41 -0500 (EST) From: John Baldwin To: Slawa Olhovchenkov Cc: arch@freebsd.org Subject: Re: Refactoring asynchronous I/O Date: Wed, 27 Jan 2016 13:16:37 -0800 Message-ID: <3640314.Du7Q0QmH0W@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) In-Reply-To: <20160127210420.GZ88527@zxy.spb.ru> References: <2793494.0Z1kBV82mT@ralph.baldwin.cx> <1723457.HAUy43H1XN@ralph.baldwin.cx> <20160127210420.GZ88527@zxy.spb.ru> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 27 Jan 2016 17:56:42 -0500 (EST) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 22:56:43 -0000 On Thursday, January 28, 2016 12:04:20 AM Slawa Olhovchenkov wrote: > On Wed, Jan 27, 2016 at 09:52:12AM -0800, John Baldwin wrote: > > > On Wednesday, January 27, 2016 01:52:05 PM Slawa Olhovchenkov wrote: > > > On Tue, Jan 26, 2016 at 05:39:03PM -0800, John Baldwin wrote: > > > > > > > The original motivation for my changes is to support efficient zero-copy > > > > receive for TOE using Chelsio T4/T5 adapters. However, read() is ill > > > > > > I undertuns that not you work, but: what about (teoretical) async > > > open/close/unlink/etc? > > > > Implementing more asynchronous operations is orthogonal to this. It > > would perhaps be a bit simpler to implement these in the new model > > since most of the logic would live in a vnode-specific aio_queue > > method in vfs_vnops.c. However, the current AIO approach is to add a > > new system call for each async system call (e.g. aio_open()). You > > would then create an internal LIO opcode (e.g. LIO_OPEN). The vnode > > aio hook would then have to support LIO_OPEN requests and return the > > opened fd via aio_complete(). Async stat / open might be nice for > > network filesystems in particular. I've known of programs forking > > separate threads just to do open/fstat of NFS files to achieve the > > equivalent of aio_open() / aio_stat(). > > Some problem exist for open()/unlink/rename/etc -- you can't use > fd-related semantic. Mmmm. We have an aio_mlock(). aio_open() would require more of a special case like aio_mlock(). It's still doable, but it would not go via the fileop, yes. fstat could go via the fileop, but a path-based stat would be akin to aio_open(). -- John Baldwin From owner-freebsd-arch@freebsd.org Wed Jan 27 23:18:22 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C3C26A709CE for ; Wed, 27 Jan 2016 23:18:22 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B35F41469 for ; Wed, 27 Jan 2016 23:18:22 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: by mailman.ysv.freebsd.org (Postfix) id AFF00A709CD; Wed, 27 Jan 2016 23:18:22 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AF86EA709CC for ; Wed, 27 Jan 2016 23:18:22 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7353C1468; Wed, 27 Jan 2016 23:18:22 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1aOZLu-000G0N-1O; Thu, 28 Jan 2016 02:18:18 +0300 Date: Thu, 28 Jan 2016 02:18:17 +0300 From: Slawa Olhovchenkov To: John Baldwin Cc: arch@freebsd.org Subject: Re: Refactoring asynchronous I/O Message-ID: <20160127231817.GA88527@zxy.spb.ru> References: <2793494.0Z1kBV82mT@ralph.baldwin.cx> <1723457.HAUy43H1XN@ralph.baldwin.cx> <20160127210420.GZ88527@zxy.spb.ru> <3640314.Du7Q0QmH0W@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <3640314.Du7Q0QmH0W@ralph.baldwin.cx> User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 23:18:23 -0000 On Wed, Jan 27, 2016 at 01:16:37PM -0800, John Baldwin wrote: > On Thursday, January 28, 2016 12:04:20 AM Slawa Olhovchenkov wrote: > > On Wed, Jan 27, 2016 at 09:52:12AM -0800, John Baldwin wrote: > > > > > On Wednesday, January 27, 2016 01:52:05 PM Slawa Olhovchenkov wrote: > > > > On Tue, Jan 26, 2016 at 05:39:03PM -0800, John Baldwin wrote: > > > > > > > > > The original motivation for my changes is to support efficient zero-copy > > > > > receive for TOE using Chelsio T4/T5 adapters. However, read() is ill > > > > > > > > I undertuns that not you work, but: what about (teoretical) async > > > > open/close/unlink/etc? > > > > > > Implementing more asynchronous operations is orthogonal to this. It > > > would perhaps be a bit simpler to implement these in the new model > > > since most of the logic would live in a vnode-specific aio_queue > > > method in vfs_vnops.c. However, the current AIO approach is to add a > > > new system call for each async system call (e.g. aio_open()). You > > > would then create an internal LIO opcode (e.g. LIO_OPEN). The vnode > > > aio hook would then have to support LIO_OPEN requests and return the > > > opened fd via aio_complete(). Async stat / open might be nice for > > > network filesystems in particular. I've known of programs forking > > > separate threads just to do open/fstat of NFS files to achieve the > > > equivalent of aio_open() / aio_stat(). > > > > Some problem exist for open()/unlink/rename/etc -- you can't use > > fd-related semantic. > > Mmmm. We have an aio_mlock(). aio_open() would require more of a special > case like aio_mlock(). It's still doable, but it would not go via the > fileop, yes. fstat could go via the fileop, but a path-based stat would > be akin to aio_open(). aio_rename require yet more of special handling. As I see this is can't be packed in current structures (aiocb and perhaps sigevent). I am don't see space for multiple paths. I am don't see space for fd return. Need to change some semantics (dissalow some notifications, for examples, only SIGEV_THREAD will be allowed? How pass information about called aio operation?). Also, may be some problems inside kernel for fd-less operations? From owner-freebsd-arch@freebsd.org Thu Jan 28 00:44:51 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F42CA6F727 for ; Thu, 28 Jan 2016 00:44:51 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 44E261ADD for ; Thu, 28 Jan 2016 00:44:51 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: by mailman.ysv.freebsd.org (Postfix) id 43503A6F726; Thu, 28 Jan 2016 00:44:51 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 42E06A6F725 for ; Thu, 28 Jan 2016 00:44:51 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 24E681ADC for ; Thu, 28 Jan 2016 00:44:51 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id BE1F3B917; Wed, 27 Jan 2016 19:44:49 -0500 (EST) From: John Baldwin To: Slawa Olhovchenkov Cc: arch@freebsd.org Subject: Re: Refactoring asynchronous I/O Date: Wed, 27 Jan 2016 16:44:28 -0800 Message-ID: <5889488.CCOMJGym34@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) In-Reply-To: <20160127231817.GA88527@zxy.spb.ru> References: <2793494.0Z1kBV82mT@ralph.baldwin.cx> <3640314.Du7Q0QmH0W@ralph.baldwin.cx> <20160127231817.GA88527@zxy.spb.ru> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 27 Jan 2016 19:44:49 -0500 (EST) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jan 2016 00:44:51 -0000 On Thursday, January 28, 2016 02:18:17 AM Slawa Olhovchenkov wrote: > On Wed, Jan 27, 2016 at 01:16:37PM -0800, John Baldwin wrote: > > > On Thursday, January 28, 2016 12:04:20 AM Slawa Olhovchenkov wrote: > > > On Wed, Jan 27, 2016 at 09:52:12AM -0800, John Baldwin wrote: > > > > > > > On Wednesday, January 27, 2016 01:52:05 PM Slawa Olhovchenkov wrote: > > > > > On Tue, Jan 26, 2016 at 05:39:03PM -0800, John Baldwin wrote: > > > > > > > > > > > The original motivation for my changes is to support efficient zero-copy > > > > > > receive for TOE using Chelsio T4/T5 adapters. However, read() is ill > > > > > > > > > > I undertuns that not you work, but: what about (teoretical) async > > > > > open/close/unlink/etc? > > > > > > > > Implementing more asynchronous operations is orthogonal to this. It > > > > would perhaps be a bit simpler to implement these in the new model > > > > since most of the logic would live in a vnode-specific aio_queue > > > > method in vfs_vnops.c. However, the current AIO approach is to add a > > > > new system call for each async system call (e.g. aio_open()). You > > > > would then create an internal LIO opcode (e.g. LIO_OPEN). The vnode > > > > aio hook would then have to support LIO_OPEN requests and return the > > > > opened fd via aio_complete(). Async stat / open might be nice for > > > > network filesystems in particular. I've known of programs forking > > > > separate threads just to do open/fstat of NFS files to achieve the > > > > equivalent of aio_open() / aio_stat(). > > > > > > Some problem exist for open()/unlink/rename/etc -- you can't use > > > fd-related semantic. > > > > Mmmm. We have an aio_mlock(). aio_open() would require more of a special > > case like aio_mlock(). It's still doable, but it would not go via the > > fileop, yes. fstat could go via the fileop, but a path-based stat would > > be akin to aio_open(). > > aio_rename require yet more of special handling. > As I see this is can't be packed in current structures (aiocb and > perhaps sigevent). I am don't see space for multiple paths. I am don't > see space for fd return. > > Need to change some semantics (dissalow some notifications, for > examples, only SIGEV_THREAD will be allowed? How pass information > about called aio operation?). > > Also, may be some problems inside kernel for fd-less operations? The kernel side of aiocb is free to hold additional information, and you could always pass additional information via arguments, e.g. aio_rename(aiocb, from, to) (Alternatively, you could define aio_buf as pointing to a structure that holds arguments.) -- John Baldwin From owner-freebsd-arch@freebsd.org Thu Jan 28 18:41:32 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5FEB1A705A7 for ; Thu, 28 Jan 2016 18:41:32 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4BC141A04 for ; Thu, 28 Jan 2016 18:41:32 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: by mailman.ysv.freebsd.org (Postfix) id 48FC9A705A6; Thu, 28 Jan 2016 18:41:32 +0000 (UTC) Delivered-To: arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48772A705A5 for ; Thu, 28 Jan 2016 18:41:32 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 39E461A03 for ; Thu, 28 Jan 2016 18:41:32 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [IPv6:::1]) by freefall.freebsd.org (Postfix) with ESMTP id 33C5C1419 for ; Thu, 28 Jan 2016 18:41:32 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id 048831B633 for ; Thu, 28 Jan 2016 18:41:32 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id A9ByRRJ9YzhD for ; Thu, 28 Jan 2016 18:41:29 +0000 (UTC) Subject: Re: build: FAST_DEPEND default (kernel first, then world) DKIM-Filter: OpenDKIM Filter v2.9.2 mail.xzibition.com 8672E1B62D To: arch@FreeBSD.org References: <56A91A1E.4060304@FreeBSD.org> From: Bryan Drewery X-Enigmail-Draft-Status: N1110 Organization: FreeBSD Message-ID: <56AA60D5.7090306@FreeBSD.org> Date: Thu, 28 Jan 2016 10:41:25 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:38.0) Gecko/20100101 Thunderbird/38.5.1 MIME-Version: 1.0 In-Reply-To: <56A91A1E.4060304@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Jan 2016 18:41:32 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 On 1/27/16 11:27 AM, Bryan Drewery wrote: > I have been asked to also remove the old mkdep version of 'make > depend'. I should note some of the bugs with the old 'mkdep' method that are fixed by the FAST_DEPEND method, for world. 1. CXXFLAGS/CFLAGS are not passed properly from Makefile.inc1 for external toolchain with gcc. A DEPFLAGS hack was added to address this. The problem being that we pass CFLAGS via CC=3D"${CC} ${CFLAGS}" from Makefile.inc1 rather than a CFLAGS_APPEND that mkdep can pick up. 2. CFLAGS=3D-include was ignored in mkdep until r294370. 3. Avoiding ccache for mkdep involves annoying hacks that broke someone's build (unfixed) 4. ccache can't benefit from mkdep 5. Similar to -include, we only pass certain flags to mkdep which is a maintenance problem. 6. It hides dependency problems due to 'requiring' running 'make depend' before build. A goal with FAST_DEPEND is to not need this, which has proven to be fine. DPSRCS was widely abused and fixed. It should not even exist really, it's not needed. - --=20 Regards, Bryan Drewery -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJWqmDVAAoJEDXXcbtuRpfPd6gH/1EqBsKQGn8qApzVUHJnrzGL lJGbD+zZVY2RjmpChMLYA4cJu0lA9edKB2oj5pIQWXDZK6rO2Q7ekAElVE64gnDw UsMeAi6v0m5/UwSGvLYHW1Nm7q/j0JTcIKXRpbseI3a40Gervdxmswrw0vGGWBEp Hg+nfeiiEFjR84lBvfwynoHjtGb3ovakPo6lGVVNHD5LIpEwuSxaldWb99uoPbDt h5iqWICvVbR4ugxEstGdPnx/6bMof+WngiQssc+8ZEiDQ0uuXneI7fyFaEd3MYCZ +LdswwPA1cJPpDmCif/fRxDWoCAQum0HGPGB2bkXRdwERMOVltRsiDEWr/kM6Ko=3D =3DmCSt -----END PGP SIGNATURE-----