Date: Wed, 8 Oct 2014 16:12:57 -0700 From: Adrian Chadd <adrian@freebsd.org> To: Warner Losh <imp@bsdimp.com> Cc: "freebsd-arch@freebsd.org" <freebsd-arch@freebsd.org> Subject: Re: [rfc] enumerating device / bus domain information Message-ID: <CAJ-VmonbGW1JbEiKXJ0sQCFr0%2BCRphVrSuBhFnh1gq6-X1CFdQ@mail.gmail.com> In-Reply-To: <2975E3D3-0335-4739-9242-5733CCEE726C@bsdimp.com> References: <CAJ-VmokF7Ey0fxaQ7EMBJpCbgFnyOteiL2497Z4AFovc%2BQRkTA@mail.gmail.com> <2975E3D3-0335-4739-9242-5733CCEE726C@bsdimp.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On 8 October 2014 12:07, Warner Losh <imp@bsdimp.com> wrote: > > On Oct 7, 2014, at 7:37 PM, Adrian Chadd <adrian@FreeBSD.org> wrote: > >> Hi, >> >> Right now we're not enumerating any NUMA domain information about device= s. >> >> The more recent intel NUMA stuff has some extra affinity information >> for devices that (eventually) will allow us to bind kernel/user >> threads and/or memory allocation to devices to keep access local. >> There's a penalty for DMAing in/out of remote memory, so we'll want to >> figure out what counts as "Local" for memory allocation and perhaps >> constrain the CPU set that worker threads for a device run on. >> >> This patch adds a few things: >> >> * it adds a bus_if.m method for fetching the VM domain ID of a given >> device; or ENOENT if it's not in a VM domain; > > Maybe a default VM domain. All devices are in VM domains :) By default > today, we have only one VM domain, and that=E2=80=99s the model that most= of the > code expects=E2=80=A6 Right, and that doesn't change until you compile in with num domains > 1. Then, CPUs and memory have VM domains, but devices may or may not have a VM domain. There's no "default" VM domain defined if num domains > 1. The devices themselves don't know about VM domains right now, so there's nothing constraining things like IRQ routing, CPU set, memory allocation, etc. The isilon team is working on extending the cpuset and allocators to "know" about numa and I'm sure this stuff will fall out of whatever they're working on. So when I go to add sysctl and other tree knowledge for device -> vm domain mapping I'm going to make them return -1 for "no domain." (Things will get pretty hilarious later on if we have devices that are "local" to two or more VM domains ..) -a
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAJ-VmonbGW1JbEiKXJ0sQCFr0%2BCRphVrSuBhFnh1gq6-X1CFdQ>