From owner-freebsd-hackers@FreeBSD.ORG Sun Dec 21 20:40:04 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B08B6288 for ; Sun, 21 Dec 2014 20:40:04 +0000 (UTC) Received: from mail-ob0-x231.google.com (mail-ob0-x231.google.com [IPv6:2607:f8b0:4003:c01::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 757242A94 for ; Sun, 21 Dec 2014 20:40:04 +0000 (UTC) Received: by mail-ob0-f177.google.com with SMTP id va2so18445270obc.8 for ; Sun, 21 Dec 2014 12:40:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=57XWaflaYAOa6xyvS21Qr6pbz0JPsjUS2DSMvEEHbo4=; b=U5RkrBHr/MKndyHtVBWuiqfKLTY1TFykhfKWi2Wu7huWAk5OvkWko8Ia9gO4bqnEmw 1lS22nKIz15QEqc6rkGL6EXsUwIXyB94AMZ+1mSaLkAAuUbp6YAcyi0bizuugJ97zWjk WNE5N3XhizHt95UVe3Hd7G5xoq0m8sQfaxvtJ+KXSBjvGgmuUtpjC6AY/M050YdNo1LL vQhu9o8SJhN6K4+QheZR8yw6YL1eEjh3ItOFy2YwKBJYYIdMEEteznUx6rHTKm3Fz7OB EnZfq5zqlCIGWHnm/bME+QX7XeUAn8YC8GRxXuasHiYGONgfWDI3cDxdU7ZNy0btDqZh Ppyw== MIME-Version: 1.0 X-Received: by 10.202.50.84 with SMTP id y81mr10513863oiy.122.1419194403794; Sun, 21 Dec 2014 12:40:03 -0800 (PST) Received: by 10.76.132.65 with HTTP; Sun, 21 Dec 2014 12:40:03 -0800 (PST) Date: Mon, 22 Dec 2014 07:40:03 +1100 Message-ID: Subject: Fun with PF & redirection From: Stephen Hocking To: hackers@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Dec 2014 20:40:04 -0000 Hi all, I'm using PF on a 10.1 box, and am trying to redirect a range of ports to a single port, with a rule like this: rdr on $ext_if proto tcp from any to any port 65334:5044 -> $spoof_host port $spoof_port spoof_host has been set to 127.0.0.1. This does not seem to work. Any ideas? Stephen From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 22 12:48:10 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EEFC6DC6 for ; Mon, 22 Dec 2014 12:48:09 +0000 (UTC) Received: from mail-yk0-x233.google.com (mail-yk0-x233.google.com [IPv6:2607:f8b0:4002:c07::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AACBE32C1 for ; Mon, 22 Dec 2014 12:48:09 +0000 (UTC) Received: by mail-yk0-f179.google.com with SMTP id 19so2208651ykq.10 for ; Mon, 22 Dec 2014 04:48:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=IRjPxkaLojIP4XItc1fLvhFzU5jLyoVOdzBrvYqKFEk=; b=rhzavli74t5ntWNUdlaT8HxvbAUBvGb49aoMcD6L57eCkelV8pTOayMu2kupT9yZr8 pOv7Pg+ar4D6m610u4m+oxBsjzxy2Bi+Hs2mYN7GVFpZazWQuICNvaOnSc3an6qE+Ywq WWk9AkKU/lR7KL4h6RabAiuggehKzVORkkvECNC1rc7qOLbcAnZdgcbQYdEZ7lXARk43 I6tmCddPJf2bi8X1/hzdJueWatkV8ncy0DPfavNotzrKVlUf7btRbCQ08ZdT/f13we9g l8xU9dp8CEddbn9X7diPYLRawQ4mINFfq1GCj5rwF2rvlWIVpqVde5nad5Eam3yxGv+i NzhQ== MIME-Version: 1.0 X-Received: by 10.236.70.168 with SMTP id p28mr17465650yhd.86.1419252488727; Mon, 22 Dec 2014 04:48:08 -0800 (PST) Received: by 10.170.188.144 with HTTP; Mon, 22 Dec 2014 04:48:08 -0800 (PST) In-Reply-To: References: Date: Mon, 22 Dec 2014 12:48:08 +0000 Message-ID: Subject: Re: Fun with PF & redirection From: krad To: Stephen Hocking X-Mailman-Approved-At: Mon, 22 Dec 2014 14:15:52 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2014 12:48:10 -0000 should that be 5044:65334 rather than 65334:5044? also make sure you are not filtering ports 5044-65334 and that the $spoof_port isnt filtered On 21 December 2014 at 20:40, Stephen Hocking wrote: > Hi all, > > I'm using PF on a 10.1 box, and am trying to redirect a range of ports to a > single port, with a rule like this: > > rdr on $ext_if proto tcp from any to any port 65334:5044 -> $spoof_host > port $spoof_port > > spoof_host has been set to 127.0.0.1. > > This does not seem to work. Any ideas? > > > Stephen > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 22 15:51:51 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 8B3D6744; Mon, 22 Dec 2014 15:51:51 +0000 (UTC) Received: from mail-pd0-x234.google.com (mail-pd0-x234.google.com [IPv6:2607:f8b0:400e:c02::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B8E5363D; Mon, 22 Dec 2014 15:51:51 +0000 (UTC) Received: by mail-pd0-f180.google.com with SMTP id w10so6006838pde.25; Mon, 22 Dec 2014 07:51:51 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:content-transfer-encoding:mime-version:date :subject:message-id:to; bh=eInr2dUm//9TdNzhzPtPT/3lJoO8jQc31KTKJruWziA=; b=ohxhSg+wE12WauznbBr3KyasRmZrsy2g//V72gh8UuaMCw2qb0OWOGkZQeCb8QHGN2 Rz7oOmZ825W8uanFAqZ813e7oDMM7t8R8w6OadeaKjSvZcKjUHCoFjZyyoeVh0BiA+WB pQIKeUB0GNQ+pTvMcv/8EnsSSsLxL4xr9X2KjEXGztF7SZ7IuveYVDQf74fZvXtZmeoW ywERxQchwKCi2ytOyYimTWII77dUq3DXIuitXhyl1uC57A6aPY9DbTCcsPqNHjz765f+ s0QJ7LYpM5vBATWynRdn694nFmN0YGJB9pQr3Pf+AvLrMF2XPk4FMEVoiLmbcnY/zESf VAmw== X-Received: by 10.66.102.103 with SMTP id fn7mr18817505pab.113.1419263510771; Mon, 22 Dec 2014 07:51:50 -0800 (PST) Received: from [192.168.1.200] ([111.199.201.113]) by mx.google.com with ESMTPSA id z5sm17514179pbt.89.2014.12.22.07.51.48 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 22 Dec 2014 07:51:49 -0800 (PST) From: Gavin Mu Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Date: Mon, 22 Dec 2014 23:51:48 +0800 Subject: status of projects/numa? Message-Id: To: "attilio@freebsd.org" , freebsd-hackers@freebsd.org X-Mailer: iPad Mail (12B440) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2014 15:51:51 -0000 Hi, Do you know what is the status of projects/numa? It seems it has been there f= or more than half a year without any update. Is there any plan to merge to H= EAD? or is it not ready yet? The new Intel Haswell Xeon supports a feature called Cluster on Die, and the= re will be two NUMA domains in one CPU socket. I am wondering if NUMA suppor= t is being more and more important. Regards, Gavin Mu From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 22 17:11:10 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 87CECE7F; Mon, 22 Dec 2014 17:11:10 +0000 (UTC) Received: from mail-wg0-x22a.google.com (mail-wg0-x22a.google.com [IPv6:2a00:1450:400c:c00::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 198CD64E02; Mon, 22 Dec 2014 17:11:10 +0000 (UTC) Received: by mail-wg0-f42.google.com with SMTP id k14so7243110wgh.15; Mon, 22 Dec 2014 09:11:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=ue0A9yf1Gpb/m4/lAQdXLu0oo4pdobOt2y4Veg5KG8I=; b=TfwstBYIni1puBu3smDs1bM5nDG0GCCqMuvnbHmy1zxPqskimTFvGmIo28TWSRP/G4 bcgMU4INFRQYTjFmhndKONh7WRb2cMMFQhePsSKnda5rbfrwHlSn+fTSzTPjRes44bRw T+2sHVyOXsiiNRbBMEV1V6EzvF99Zigze5o2NqmBf1ESOBACFHLhgJ9gLs24mNrtu1Wn wy6H3dT1tysb70qNZt4Jl7ONcE+rMGyNFJIxVy26F+lBqw5NZpmNBQsyCby5O9Yjdd/o 2X/vFqKnQ32JoKpNcNEpOqxRxiWbn3G4Vj9D5DDzdnu9UdmSdpPxKRnHdVRJFSj5O+Gw oqSA== MIME-Version: 1.0 X-Received: by 10.194.24.103 with SMTP id t7mr43968144wjf.15.1419268268263; Mon, 22 Dec 2014 09:11:08 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.106.195 with HTTP; Mon, 22 Dec 2014 09:11:08 -0800 (PST) In-Reply-To: References: Date: Mon, 22 Dec 2014 09:11:08 -0800 X-Google-Sender-Auth: oK00ElcVTCCzokjE4mtxUXfkiBI Message-ID: Subject: Re: status of projects/numa? From: Adrian Chadd To: Gavin Mu Content-Type: text/plain; charset=UTF-8 Cc: "attilio@freebsd.org" , "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2014 17:11:10 -0000 There's been a whole bunch of discussion going on between the various parties finishing this off. I'll see if I can get some further updates about it. (I've been chasing it up so I can teach device drivers and the network stack about NUMA..) -adrian On 22 December 2014 at 07:51, Gavin Mu wrote: > Hi, > > Do you know what is the status of projects/numa? It seems it has been there for more than half a year without any update. Is there any plan to merge to HEAD? or is it not ready yet? > > The new Intel Haswell Xeon supports a feature called Cluster on Die, and there will be two NUMA domains in one CPU socket. I am wondering if NUMA support is being more and more important. > > Regards, > Gavin Mu > > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 22 20:18:08 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9AC86547 for ; Mon, 22 Dec 2014 20:18:08 +0000 (UTC) Received: from mail-oi0-x231.google.com (mail-oi0-x231.google.com [IPv6:2607:f8b0:4003:c06::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5BED2232 for ; Mon, 22 Dec 2014 20:18:08 +0000 (UTC) Received: by mail-oi0-f49.google.com with SMTP id a141so10706569oig.8 for ; Mon, 22 Dec 2014 12:18:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=h/QXNOYQ9mTIwl6584P9ctAQIyFebK7A38XSxqyEshg=; b=RffeP1NUHnB3Li7I9yjV8W/6B01vdsveGSRtabpkg5rz3/Vq2sgKSOOUvwVQxcU+pW 3is5xmxpUcwrwCb5g1EwEdsGGAKWRp8S4kSCn/LNrLnOT06miB+hGoIqBk8WJNSZgJYT vop/7K/wAXW39uzaPHttDXjkgs8R3MDtvYQEg+f24VnqIlCA3J0sRDtOBcThyXDa5C5S 7g4IElWcOnQDlEDRnKGOhZa7Tsv1rERN0Zl5OttRqsYv+tAOeWv1CpGUDVTjKNWk8pBO hKXCk3gLuIwP6iDhZYNUC+PkFRUndZo4wueuU6Oi7OjyAF2Wj3A/kmll//JIdI8TjnJY OpSg== MIME-Version: 1.0 X-Received: by 10.202.201.23 with SMTP id z23mr13459564oif.32.1419279487811; Mon, 22 Dec 2014 12:18:07 -0800 (PST) Received: by 10.76.132.65 with HTTP; Mon, 22 Dec 2014 12:18:07 -0800 (PST) In-Reply-To: References: Date: Tue, 23 Dec 2014 07:18:07 +1100 Message-ID: Subject: Re: Fun with PF & redirection From: Stephen Hocking To: krad Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2014 20:18:08 -0000 Spot on! It turns out the issue was that the port ranges need to be in ascending order, as you suggested. I now have a small box which is capable of driving Nessus to tears. On Mon, Dec 22, 2014 at 11:48 PM, krad wrote: > should that be 5044:65334 rather than 65334:5044? > also make sure you are not filtering ports 5044-65334 and that the $spoof_port > isnt filtered > > On 21 December 2014 at 20:40, Stephen Hocking > wrote: > >> Hi all, >> >> I'm using PF on a 10.1 box, and am trying to redirect a range of ports to >> a >> single port, with a rule like this: >> >> rdr on $ext_if proto tcp from any to any port 65334:5044 -> $spoof_host >> port $spoof_port >> >> spoof_host has been set to 127.0.0.1. >> >> This does not seem to work. Any ideas? >> >> >> Stephen >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org >> " >> > > From owner-freebsd-hackers@FreeBSD.ORG Mon Dec 22 22:09:29 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9C6B34A0 for ; Mon, 22 Dec 2014 22:09:29 +0000 (UTC) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E46B3647 for ; Mon, 22 Dec 2014 22:09:28 +0000 (UTC) Received: from Julian-MBP3.local (ppp121-45-252-117.lns20.per2.internode.on.net [121.45.252.117]) (authenticated bits=0) by vps1.elischer.org (8.14.9/8.14.9) with ESMTP id sBMM9I4r037684 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Mon, 22 Dec 2014 14:09:21 -0800 (PST) (envelope-from julian@freebsd.org) Message-ID: <54989688.9050601@freebsd.org> Date: Tue, 23 Dec 2014 06:09:12 +0800 From: Julian Elischer User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: krad , Stephen Hocking Subject: Re: Fun with PF & redirection References: In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Dec 2014 22:09:29 -0000 On 12/22/14 8:48 PM, krad wrote: > should that be 5044:65334 rather than 65334:5044? > also make sure you are not filtering ports 5044-65334 and that the $spoof_port > isnt filtered > > On 21 December 2014 at 20:40, Stephen Hocking > wrote: > >> Hi all, >> >> I'm using PF on a 10.1 box, and am trying to redirect a range of ports to a >> single port, with a rule like this: >> >> rdr on $ext_if proto tcp from any to any port 65334:5044 -> $spoof_host >> port $spoof_port >> >> spoof_host has been set to 127.0.0.1. >> >> This does not seem to work. Any ideas? use ipfw? fwd 127.0.0.1:$spoof_port tcp from any to me 5044-65334 >> >> Stephen >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >> > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 23 01:11:29 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1AD18C2D for ; Tue, 23 Dec 2014 01:11:29 +0000 (UTC) Received: from mail-wi0-x22d.google.com (mail-wi0-x22d.google.com [IPv6:2a00:1450:400c:c05::22d]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9FB41D91 for ; Tue, 23 Dec 2014 01:11:28 +0000 (UTC) Received: by mail-wi0-f173.google.com with SMTP id r20so9563523wiv.0 for ; Mon, 22 Dec 2014 17:11:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:sender:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=wLA1+mkTcyxYWKL9zfzw58V96b/GYVKsrQCfiUuq5dM=; b=Uep8y+8eMN5MGiIOZ94xKBzJkyFffdgbVWdS6SEDKkxxB3+Q9ooeupZSL4OITRmyu2 u+hAVAbajtm9DXNX4td1ixqkVK/1E8ec5+X/kYUU9H0FxFt5xlKUbD19i3huIkGUrOWG uMwN9Dvg/X5M+kQL4+RgWXx/5j6+CY2HU8rfV+8WT7Wlb9iH+PFpxF37eeTjIFRYsDq4 zr0rrQmXS2o8dmkNxgEVmado2jgBtGcFCd0LkYzW3mUgvFiWfqbp5AbQ+wvczEe697mb jrpiPafZcqvfR0+IbnllLvvnm5UFCU8b+WsD720jmzH8CHDVglfdFPWclkdzZ/UCRPqV GGqA== MIME-Version: 1.0 X-Received: by 10.180.82.98 with SMTP id h2mr36476198wiy.7.1419297087015; Mon, 22 Dec 2014 17:11:27 -0800 (PST) Reply-To: attilio@FreeBSD.org Sender: asmrookie@gmail.com Received: by 10.216.30.145 with HTTP; Mon, 22 Dec 2014 17:11:26 -0800 (PST) In-Reply-To: References: Date: Tue, 23 Dec 2014 02:11:26 +0100 X-Google-Sender-Auth: 4jBT4B8_7_gj6d3zbsWEWjzaAtQ Message-ID: Subject: Re: status of projects/numa? From: Attilio Rao To: Gavin Mu Content-Type: text/plain; charset=UTF-8 Cc: FreeBSD Hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Dec 2014 01:11:29 -0000 I'm not working on FreeBSD anymore, so I have no idea who is going to finish it. I think you should discuss this with Alan or Jeff. Attilio On Mon, Dec 22, 2014 at 4:51 PM, Gavin Mu wrote: > Hi, > > Do you know what is the status of projects/numa? It seems it has been there for more than half a year without any update. Is there any plan to merge to HEAD? or is it not ready yet? > > The new Intel Haswell Xeon supports a feature called Cluster on Die, and there will be two NUMA domains in one CPU socket. I am wondering if NUMA support is being more and more important. > > Regards, > Gavin Mu > -- Peace can only be achieved by understanding - A. Einstein From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 23 10:19:34 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 19245F1C; Tue, 23 Dec 2014 10:19:34 +0000 (UTC) Received: from mail-pd0-x236.google.com (mail-pd0-x236.google.com [IPv6:2607:f8b0:400e:c02::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D9AC82D73; Tue, 23 Dec 2014 10:19:33 +0000 (UTC) Received: by mail-pd0-f182.google.com with SMTP id p10so7569492pdj.27; Tue, 23 Dec 2014 02:19:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=LyRmD9mF+B0spoHA5t+yi2PIhUTltKLSpBIGTvSBByw=; b=LulXRyrylL4YC0juvqW8/pIWTkQCWvw3VqaXNSBGL8Crizx5Hw53GjzQvaMDDg6s0u mDe1/L3s6w9jgFvM2xBTlMUTx40tTzDRJRowN9n6lvO3Z5hhBDCdl7HJGpbF8LSFXvu7 VxHx6d4Zry6p8D4ACELGIecq3vgjVhmFfzXQdhDI60LCNXnpbnrewcKpCIkIXVugCgOo dziqKrirK36thTEESK8Bfmk5yyE10BEeiodMeNzY+E5SXMGCSxV3upms0j3OwCCKD0ub Bq7dBOrbOGEWxCs3NuG/TlVcHrPr2RdsVpQqNz6Hmc+LwRDUuaxEqfX9oykGI1t/ckeE 1YXA== X-Received: by 10.69.26.130 with SMTP id iy2mr42737577pbd.93.1419329973427; Tue, 23 Dec 2014 02:19:33 -0800 (PST) Received: from [10.28.193.15] ([114.242.249.201]) by mx.google.com with ESMTPSA id fs1sm19613003pdb.16.2014.12.23.02.19.31 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 23 Dec 2014 02:19:32 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: status of projects/numa? From: Gavin Mu X-Mailer: iPad Mail (12B440) In-Reply-To: Date: Tue, 23 Dec 2014 18:19:31 +0800 Content-Transfer-Encoding: quoted-printable Message-Id: <35AF9553-9536-41B0-8C2C-D907ED8E8531@gmail.com> References: To: Adrian Chadd Cc: "attilio@freebsd.org" , "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Dec 2014 10:19:34 -0000 Hi, Adrian, Thanks, I have checked out your ACPI related change for NUMA support. Lookin= g forward to your further updates. Is the discussion public? it will be very helpful if there are any records. W= e have had a hacked and limited impl of NUMA, and wondering if we can help o= n a general design for FreeBSD. Regards, Gavin Mu > On Dec 23, 2014, at 01:11, Adrian Chadd wrote: >=20 > There's been a whole bunch of discussion going on between the various > parties finishing this off. I'll see if I can get some further updates > about it. >=20 > (I've been chasing it up so I can teach device drivers and the network > stack about NUMA..) >=20 >=20 >=20 > -adrian >=20 >=20 >> On 22 December 2014 at 07:51, Gavin Mu wrote: >> Hi, >>=20 >> Do you know what is the status of projects/numa? It seems it has been the= re for more than half a year without any update. Is there any plan to merge t= o HEAD? or is it not ready yet? >>=20 >> The new Intel Haswell Xeon supports a feature called Cluster on Die, and t= here will be two NUMA domains in one CPU socket. I am wondering if NUMA supp= ort is being more and more important. >>=20 >> Regards, >> Gavin Mu >>=20 >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 23 18:12:48 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 817B6C59; Tue, 23 Dec 2014 18:12:48 +0000 (UTC) Received: from mail-wi0-x235.google.com (mail-wi0-x235.google.com [IPv6:2a00:1450:400c:c05::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 12A3564425; Tue, 23 Dec 2014 18:12:48 +0000 (UTC) Received: by mail-wi0-f181.google.com with SMTP id r20so11587097wiv.14; Tue, 23 Dec 2014 10:12:46 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=e4/W0kjATqs8bGPS7U+yI3QCW6Dp04uPYr95KsVvanM=; b=BjpRNg14Xkh/4CkTWfUqKQu4xVvKU6o71TroIXtzpFKTUJODd8jR0itmA0qSITN33P RPUi8zEYnQMsOeipSWnFCOfUz4fcX30l2PfqDSxb/SCJGJQCrZ5u7aQmsc5HeTSy8xGa RDbN0cQmtAYjpw57dVCvOIs7AxP8RA+4I/BYpns9k//eSOZ7WDapT+UbgYd1xrgpFowp GuQk+vlWncsxpMUBrCBvX720rOL7Ge0NF+KSJ+uUl+1gk/1CvlkeyrG7S38l7zF2/iDr o1GLv0XAuhRuIsUhHkLaSNyTwxxFl8Gy/LEP7C2hSrdVu0DO2cLuKW5R4iAzr9RIlv7l JlZA== MIME-Version: 1.0 X-Received: by 10.180.20.6 with SMTP id j6mr42586539wie.59.1419358366489; Tue, 23 Dec 2014 10:12:46 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.106.195 with HTTP; Tue, 23 Dec 2014 10:12:46 -0800 (PST) In-Reply-To: <35AF9553-9536-41B0-8C2C-D907ED8E8531@gmail.com> References: <35AF9553-9536-41B0-8C2C-D907ED8E8531@gmail.com> Date: Tue, 23 Dec 2014 10:12:46 -0800 X-Google-Sender-Auth: KSKrzjwCsi55noAnq3s8X119cR8 Message-ID: Subject: Re: status of projects/numa? From: Adrian Chadd To: Gavin Mu Content-Type: text/plain; charset=UTF-8 Cc: "attilio@freebsd.org" , "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Dec 2014 18:12:48 -0000 On 23 December 2014 at 02:19, Gavin Mu wrote: > Hi, Adrian, > > Thanks, I have checked out your ACPI related change for NUMA support. Looking forward to your further updates. > > Is the discussion public? it will be very helpful if there are any records. We have had a hacked and limited impl of NUMA, and wondering if we can help on a general design for FreeBSD. I'm 100% for having a public discussion list and discussion about adding more NUMA support. So please kick-start this a little - what can you share about your private NUMA support? -adrian From owner-freebsd-hackers@FreeBSD.ORG Tue Dec 23 23:40:32 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 581B89DB; Tue, 23 Dec 2014 23:40:32 +0000 (UTC) Received: from mail-yk0-x22f.google.com (mail-yk0-x22f.google.com [IPv6:2607:f8b0:4002:c07::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1218F21E4; Tue, 23 Dec 2014 23:40:32 +0000 (UTC) Received: by mail-yk0-f175.google.com with SMTP id 200so3498464ykr.6; Tue, 23 Dec 2014 15:40:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=Emcx8/aVyCDUfvTQIaVciNh/j83PVTbLV3D9geKYWTU=; b=pn9Ud/7BUgaHyKmg2eddDoDB5brZafGHYLhwXF/Xmsfz3BMbmMw/rKUuB7Q5hgBFJb XY1GEQFOUbWyBeFoI8TbYKWse0QCZfF9ubZnh7pd2YIjtE6547HJP8WdouLK2bNEPHya Fxvft3pcZws5sPfEIgFrmcmF+mY+Ir0DxAGdUAh5AZW5vowxJH8dHsG6ob7WPhYH3E6J 2xuQ5yAWudOWpDCv+8sWQTVpmr4HXeE/QXSqoOjAsHFCnJkoXflrtSneV4oAfqkdG0fh GfldWo1Gm9Zu0tYJfGpov813mLeTQOWS9+6NTR5xsS48e/KOsrvjQwJZuJuc3PXksn4H IVuQ== MIME-Version: 1.0 X-Received: by 10.170.204.23 with SMTP id v23mr27747498yke.115.1419378031100; Tue, 23 Dec 2014 15:40:31 -0800 (PST) Sender: kmacybsd@gmail.com Received: by 10.170.70.132 with HTTP; Tue, 23 Dec 2014 15:40:31 -0800 (PST) In-Reply-To: References: <35AF9553-9536-41B0-8C2C-D907ED8E8531@gmail.com> Date: Tue, 23 Dec 2014 15:40:31 -0800 X-Google-Sender-Auth: FVgymbwfs6qD7v6J8onTsTvLpJQ Message-ID: Subject: Re: status of projects/numa? From: "K. Macy" To: Adrian Chadd Content-Type: text/plain; charset=UTF-8 Cc: "attilio@freebsd.org" , "freebsd-hackers@freebsd.org" , Gavin Mu X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Dec 2014 23:40:32 -0000 What's wrong with freebsd-arch? Do we really need a dedicated discussion list? Or have I mis-parsed your suggestions? Cheers. -K On Tue, Dec 23, 2014 at 10:12 AM, Adrian Chadd wrote: > On 23 December 2014 at 02:19, Gavin Mu wrote: >> Hi, Adrian, >> >> Thanks, I have checked out your ACPI related change for NUMA support. Looking forward to your further updates. >> >> Is the discussion public? it will be very helpful if there are any records. We have had a hacked and limited impl of NUMA, and wondering if we can help on a general design for FreeBSD. > > I'm 100% for having a public discussion list and discussion about > adding more NUMA support. > > So please kick-start this a little - what can you share about your > private NUMA support? > > > > > -adrian > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Wed Dec 24 01:16:48 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id A2AF5173; Wed, 24 Dec 2014 01:16:48 +0000 (UTC) Received: from mail-pa0-x234.google.com (mail-pa0-x234.google.com [IPv6:2607:f8b0:400e:c03::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6E83316B0; Wed, 24 Dec 2014 01:16:48 +0000 (UTC) Received: by mail-pa0-f52.google.com with SMTP id eu11so8921253pac.25; Tue, 23 Dec 2014 17:16:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:content-transfer-encoding:mime-version:subject:from :in-reply-to:date:cc:message-id:references:to; bh=iNf5xInP47aoY40zJR9acP+JFFBSR0cW9gVra3fBgcE=; b=Rv5CyCtlOdasTpJ1vP2dzcDRrIBx1u2+x3AOUF9Bf6bjWo8MWBio1ERJCWmcsdCSSk 4jTMnvCVcijZhEuun5QPJt6+6oLTMkaF0J/oivh9HZa9HB4IQkYMbflnrNz9/2Q0pWZZ v9UX8hLfeNwkNpw1kBkH4PLl/RAo+o9v8wNkh3LcmuL8+uFa9U3GUIGORgACz+nrvtgl 1kavUr5EDh27n4UxEqbb7tzlE1xqlOHAJLZM7LnH524CPkhr0fEaR3ez7/He/k8jTZOC 4k+x7NGlHRwVkQ6hHkqSRIAtcX8RmfTsAxwNVinMxA6f7nB8pCE0NgtbUPuWyG4Npzqx XYjw== X-Received: by 10.66.253.197 with SMTP id ac5mr48150285pad.152.1419383808058; Tue, 23 Dec 2014 17:16:48 -0800 (PST) Received: from [10.30.5.32] ([114.242.249.212]) by mx.google.com with ESMTPSA id nl14sm10446863pdb.81.2014.12.23.17.16.46 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 23 Dec 2014 17:16:47 -0800 (PST) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Subject: Re: status of projects/numa? From: Gavin Mu In-Reply-To: Date: Wed, 24 Dec 2014 09:11:33 +0800 Message-Id: References: <35AF9553-9536-41B0-8C2C-D907ED8E8531@gmail.com> To: Adrian Chadd X-Mailer: iPad Mail (12B440) Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Dec 2014 01:16:48 -0000 Hi, Adrian, Our impl is based on FreeBSD 7.x, and full of tricky, and we are moving to n= ewer FreeBSD version. This is why I am asking for this, we would like to re-= write and keep sync with FreeBSD community. We want to know the latest statu= s and where we can start from. I can see that projects/numa and also Jeff's repo have more functions implem= ented, and one userland API proposal on github. There is no update for a lon= g time for all. There is also the wiki page at https://wiki.freebsd.org/NUMA. I am not sure i= f it is outdated or not. Regards, Gavin Mu > On Dec 24, 2014, at 02:12, Adrian Chadd wrote: >=20 >> On 23 December 2014 at 02:19, Gavin Mu wrote: >> Hi, Adrian, >>=20 >> Thanks, I have checked out your ACPI related change for NUMA support. Loo= king forward to your further updates. >>=20 >> Is the discussion public? it will be very helpful if there are any record= s. We have had a hacked and limited impl of NUMA, and wondering if we can he= lp on a general design for FreeBSD. >=20 > I'm 100% for having a public discussion list and discussion about > adding more NUMA support. >=20 > So please kick-start this a little - what can you share about your > private NUMA support? >=20 >=20 >=20 >=20 > -adrian From owner-freebsd-hackers@FreeBSD.ORG Wed Dec 24 02:46:51 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D56A23A9; Wed, 24 Dec 2014 02:46:51 +0000 (UTC) Received: from mail-wg0-x230.google.com (mail-wg0-x230.google.com [IPv6:2a00:1450:400c:c00::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6480364D89; Wed, 24 Dec 2014 02:46:51 +0000 (UTC) Received: by mail-wg0-f48.google.com with SMTP id y19so10439169wgg.35; Tue, 23 Dec 2014 18:46:49 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=D94g2qIE8PiPzihfZMB/0Bv1BqEU8374Jp8I660WOsA=; b=nZSGG7qB0UBUERfqGqCb7QzyjCTj1oD4GcGbdqG5/LBl10hwwc1Otvc+jPurDp1sj/ 3W14RCzfnSH+mmkhcCRq19roWreEKhH3d4iAW4CKpFLKv6jEjPILxHA/5x4JoJYOYaS8 zrdJFGpTjYVaM3AdLHLqM5YWGyul7e9M68UbR1bxQdYXENBsYP70UwbI/IMdM31sI6Ii cFxIi357VNCh36ZmliLtjzs4wqWmns/980/+9Kf7tS5I+e5cBUg4l+F6ielghckgSXYt WSGwGUW5C1wGV1T8biSGEoAwBNaYZ/kNDZgkOWfVmvOKXYG6yY/4bz0FNOj3HWhG9Dxd OM1g== MIME-Version: 1.0 X-Received: by 10.180.14.136 with SMTP id p8mr11525778wic.20.1419389209795; Tue, 23 Dec 2014 18:46:49 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.106.195 with HTTP; Tue, 23 Dec 2014 18:46:49 -0800 (PST) In-Reply-To: References: <35AF9553-9536-41B0-8C2C-D907ED8E8531@gmail.com> Date: Tue, 23 Dec 2014 18:46:49 -0800 X-Google-Sender-Auth: qbvPo15QkFHMBiojjpPMe17v84g Message-ID: Subject: Re: status of projects/numa? From: Adrian Chadd To: "K. Macy" Content-Type: text/plain; charset=UTF-8 Cc: "attilio@freebsd.org" , "freebsd-hackers@freebsd.org" , Gavin Mu X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Dec 2014 02:46:51 -0000 On 23 December 2014 at 15:40, K. Macy wrote: > What's wrong with freebsd-arch? Do we really need a dedicated > discussion list? Or have I mis-parsed your suggestions? I was unclear. I don't care where the discussion is; I'm happy for it to be public. freebsd-arch@ is as good a place as any. -adrian > Cheers. > > -K > > On Tue, Dec 23, 2014 at 10:12 AM, Adrian Chadd wrote: >> On 23 December 2014 at 02:19, Gavin Mu wrote: >>> Hi, Adrian, >>> >>> Thanks, I have checked out your ACPI related change for NUMA support. Looking forward to your further updates. >>> >>> Is the discussion public? it will be very helpful if there are any records. We have had a hacked and limited impl of NUMA, and wondering if we can help on a general design for FreeBSD. >> >> I'm 100% for having a public discussion list and discussion about >> adding more NUMA support. >> >> So please kick-start this a little - what can you share about your >> private NUMA support? >> >> >> >> >> -adrian >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" From owner-freebsd-hackers@FreeBSD.ORG Wed Dec 24 02:47:55 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DB05E584 for ; Wed, 24 Dec 2014 02:47:55 +0000 (UTC) Received: from mail-wg0-x235.google.com (mail-wg0-x235.google.com [IPv6:2a00:1450:400c:c00::235]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6A3B464DB6 for ; Wed, 24 Dec 2014 02:47:55 +0000 (UTC) Received: by mail-wg0-f53.google.com with SMTP id l18so10362273wgh.40 for ; Tue, 23 Dec 2014 18:47:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=zotGfQ6SEFS+/EGUgmsbQGrhTA0bakT6+jOtS4M62JI=; b=a1RGXYuEqjUxfBS7nhgdUWb0Xmr4kCcMcRM1VetrBpGvQ+wiWl64acYsTWbfgweBb1 L5xrDyG01TzIQ9qWSdy0RYk56APAk1cCkW67egRIXZgGXWGsJSAE8nAuH+tkU1Hf67FI eg3cWpwKiYq38ayoeOBVzcyD6Tl05VcxuQbbPAR2cCwVJffRe44IOI6qAU1BtlbrdFh2 RbZwyTIrFaTnCO6hyJ88qjQjBxIn4N22syPsn+nLIHi9EZJX7j1nAbcpWQrBmeOtKv+N kfi/TH91AnFXXYerW041QNrvCDvM+3ohaTP63XTp3QKvER0GhQNMGW/7aVpDToWfxPA9 GuhA== MIME-Version: 1.0 X-Received: by 10.194.108.9 with SMTP id hg9mr57745791wjb.68.1419389273843; Tue, 23 Dec 2014 18:47:53 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.106.195 with HTTP; Tue, 23 Dec 2014 18:47:53 -0800 (PST) In-Reply-To: References: <35AF9553-9536-41B0-8C2C-D907ED8E8531@gmail.com> Date: Tue, 23 Dec 2014 18:47:53 -0800 X-Google-Sender-Auth: ptYFZVY74WcBrVyxWWvJXNCQucg Message-ID: Subject: Re: status of projects/numa? From: Adrian Chadd To: Gavin Mu Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Dec 2014 02:47:55 -0000 Hi, Well can you at least outline what your implementation did? I've had a lot of people say "We have a local set of hacks for NUMA" but never really any public-ish discussion about them. -adrian On 23 December 2014 at 17:11, Gavin Mu wrote: > Hi, Adrian, > > Our impl is based on FreeBSD 7.x, and full of tricky, and we are moving t= o newer FreeBSD version. This is why I am asking for this, we would like to= re-write and keep sync with FreeBSD community. We want to know the latest = status and where we can start from. > > I can see that projects/numa and also Jeff's repo have more functions imp= lemented, and one userland API proposal on github. There is no update for a= long time for all. > > There is also the wiki page at https://wiki.freebsd.org/NUMA. I am not su= re if it is outdated or not. > > Regards, > Gavin Mu > > >> On Dec 24, 2014, at 02:12, Adrian Chadd wrote: >> >>> On 23 December 2014 at 02:19, Gavin Mu wrote: >>> Hi, Adrian, >>> >>> Thanks, I have checked out your ACPI related change for NUMA support. L= ooking forward to your further updates. >>> >>> Is the discussion public? it will be very helpful if there are any reco= rds. We have had a hacked and limited impl of NUMA, and wondering if we can= help on a general design for FreeBSD. >> >> I'm 100% for having a public discussion list and discussion about >> adding more NUMA support. >> >> So please kick-start this a little - what can you share about your >> private NUMA support? >> >> >> >> >> -adrian From owner-freebsd-hackers@FreeBSD.ORG Wed Dec 24 03:09:13 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 70801D18 for ; Wed, 24 Dec 2014 03:09:13 +0000 (UTC) Received: from mail-wg0-x236.google.com (mail-wg0-x236.google.com [IPv6:2a00:1450:400c:c00::236]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E9613F0E for ; Wed, 24 Dec 2014 03:09:12 +0000 (UTC) Received: by mail-wg0-f54.google.com with SMTP id l2so10453292wgh.41 for ; Tue, 23 Dec 2014 19:09:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=RNOX4GKWANmwwbSkTUrbeLm4fnmKL8wbtpBMHrY2Bec=; b=LYmMDmiv6M1LoGhSGvdvNMtJVzwpL6EIkLBbuwhzmULvXxWmAv+uHxwSIOJLiAZc21 NN3Y4uec13KKwM+SLgwaH06FvopfeiU3LBwRV88DMwrrxsxKSAKQ8tRCA/0Ia3UYUGyz xz9SgKPevxjEwo7R3jxiI+ZOlyVKMDHbiU2cx/qBprB4pzkGxl/SAw+06q/EoqPkbIQ0 88ZIZ91X4ZCsJtKseVygojYU0siy3hdvVFxxzy0ykMpfJEJg6ze9E5uDSl6g/lcGkh6U FnYDNXr7mM462/13LztiH5NTRZFWDxeooDZZUSBBIsNnm44Si+/orwwyeci0i2BVABng ACJA== MIME-Version: 1.0 X-Received: by 10.180.91.193 with SMTP id cg1mr47638049wib.26.1419390550573; Tue, 23 Dec 2014 19:09:10 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.106.195 with HTTP; Tue, 23 Dec 2014 19:09:10 -0800 (PST) In-Reply-To: References: Date: Tue, 23 Dec 2014 19:09:10 -0800 X-Google-Sender-Auth: OrTBEhDvfzrRVfRfr1uq25Ca9f0 Message-ID: Subject: Re: status of projects/numa? From: Adrian Chadd To: Gavin Mu Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Dec 2014 03:09:13 -0000 Ok, so to summarise the various bits/pieces I hear from around the grapevine and speaking with numa people like jhb: The projects/numa stuff has a lot of good stuff in it. Jeff/Isilon's idea seems to be to create an allocator policy framework that can be specified global, per-process and/or per-thread. That's pretty good stuff right there. What's missing however before it can be useful by drivers is the rest of the UMA stuff and I think there's some VM pagetable work that needs doing. I haven't dug into the latter because I was trusting that those working on projects/numa will get to it like they've said, but .. well, it's still not done. >From a UMA API perspective, it isn't trying very hard to return memory back to the domain that it was allocated from. I'm worried that on a real NUMA box with no policy configured or used by userland threads (ie, everything is being scheduled everywhere), we'll end up with threads allocating from a CPU-local pool, being migrated to another CPU, then freeing it there. So over time it'll end up with the cache spiked with non-local pages. I talked to jhb about it but I don't think we have a consensus about it at all. I think we should go the extra mile of ensuring that when we return pages to UMA it goes back into the right NUMA domain - I don't want to have to debug systems that start really fast and then end up getting slower over time as stuff ends up on the wrong CPU. But that's just me. >From the physmem perspective, I'll likely implement an SRAT-aware allocator option that allocates local-first, then tries to allocate from locall-er domains until it just round-robins. We have that cost matrix that says how expensive things are from a given NUMA domain, so that shouldn't be too difficult to pre-compute. >From a driver perspective, I've added the basic "which domain am I in" call. John's been playing with a couple of drivers (igb/ixgbe I think) in his local repository for teaching them about what is a local interrupt and local CPU set to start threads on. Ideally all the drivers would use the same API for querying what their local cpuset is and assigning worker threads / interrupts appropriately. I should poke him again to find the status of that work and at least get that into -HEAD so it can be evaluated and used by people. What I'm hoping to do with that in the short term is to make it generic enough so that a generic, consistent set of hints can be configured for a given driver to setup its worker thread and cpuset map. Ie, instead of each driver having its own "how many queues" sysctl and probe logic, it'll have some API to say "give me my number of threads and local cpuset(s)" so if someone wants to override it, it's done by something like code in the bus layer rather than above individual driver hacks. We can also add options for things like "pin threads" or not. >From the driver /allocation/ perspective, there's a few things to think about: * how we allocate busdma memory for things like descriptors; * how we allocate DMA memory for things like mbufs, bufs, etc - what we're dma'ing into and out of. None of that is currently specified, although there's a couple of tidbits in the projects/numa branch that haven't been fleshed out. In my vague drawing-on-paper sense of this, I think we also should then extend busdma a little to be aware of a numaset for allocating memory for descriptors. Same with calling malloc and contigmalloc. Ideally we'd keep descriptor accesses in memory local to the device. Now for things like bufs/mbufs/etc - the DMA bits - that's where it gets a little tricky. Sometimes we're handed memory to send from/to. The NIC RX path allocates memory itself. storage controllers get handed memory to read storage data into. So, the higher layer question of "how do we tell something where to allocate memory from" gets a little complicated, because we (a) may have to allocate memory for a non-local device, knowing we're going to hand it to some device on another core, and (b) we then have to return it to the right NUMA aware pool (UMA, vm_page stuff, etc.) The other tricksy bit is when drivers want to allocate memory for local ephemeral work (eg M_TEMP) versus DMA/descriptor/busdma stuff. Here's about where I say "don't worry about this - do all the above bits first then worry about worrying about this": say we have a NIC driver on an 8-core CPU, and it's in a 4-socket box (so 4 sockets * 8 cores each socket.) The receive threads can be told to run in a specific cpuset local to the CPU the NIC is plugged into. But the transmit threads, taskqueue bits, etc may run on any CPU - remember, we don't queue traffic and wakeup a thread now; we call if_transmit() from whichever CPU is running the transmit code. So if it's local then it'll come from NUMA memory domain local memory and stuff is fine. But if it's transmitting from some /remote/ thread: * the mbuf that's been allocated to send data is likely allocated from the wrong NUMA domain; * the local memory used by the driver to do work should be coming from memory local to that CPU, but it may call malloc/etc to get temporary working memory - and that hopefully is also coming from a local memory domain; * if it has to allocate descriptor memory or something then it may need to allocate memory from the remote memory domain; * .. then touch the NIC hardware, which is remote. This is where I say "screw it, don't get bogged down in the details." It gets hairier when we think about what's local versus remote for vm_page entries, because hey, who knows what the right thing to do there is. But I think the right thing to do here is to not worry about it and get the above stuff done. For NUMA aware network applications they're likely going to be CPU set to threads local to the same domain as the NIC, and if that isn't good enough, we'll have to come up with some way to mark a socket as local to a NUMA domain so things like mbufs, TCP timers, etc get scheduled appropriately. (This is where I say "And this is where the whole RSS framework and NUMA need to know about each other", but I'm not really ready for that yet.) So, that's my braindump of what I remember from discussing with others. I think the stages to go here are: * get a firmer idea of what's missing from projects/numa for UMA, physmem and VM allocator / pagedaemon/etc stuff; * get jhb's numaset code into -HEAD; * get jhb's driver NUMA awareness bits out, add more generic hooks for expressing this stuff via hints, and dump that into -HEAD; * evaluate what we're going to do about mbufs, malloc/contigmalloc for busdma, bounce buffers, descriptors, etc and get that into -HEAD; * make sure the intel-pcm tools (and then hwpmc!) work for the uncore CPU interconnects work so we can measure how well we are/aren't doing things; * come back to the drawing board once the above is done and we've got some experience with it. I haven't even started talking about what's needed for /userland/ memory pages. I think that's part of what needs to be fleshed out / discussed with the VM bits. -adrian From owner-freebsd-hackers@FreeBSD.ORG Wed Dec 24 12:23:19 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 671D3C20 for ; Wed, 24 Dec 2014 12:23:19 +0000 (UTC) Received: from mail-lb0-x233.google.com (mail-lb0-x233.google.com [IPv6:2a00:1450:4010:c04::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D6B9D66E95 for ; Wed, 24 Dec 2014 12:23:18 +0000 (UTC) Received: by mail-lb0-f179.google.com with SMTP id z11so6602341lbi.24 for ; Wed, 24 Dec 2014 04:23:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=I24td0hcsw3wcxxxU8BKRSsmTFtRpeqCKbxOut9tvg8=; b=YAr+mwcgTRAPxWrv5k1XnynLMAkQScZIscB3Ezsy82CYYHQUZx8YIxTReM3duxEKFC ZPTBRf9TwpX/QGCImHuw1hjxJI4qlBb6izhI6obtLEhAc7rKmgHv0yRuGSLjHCDvSGTE 5CKDeuR8Rzkl5YdoGwqIdUsKg8Yih7Rpc9K6Orinz2JMfoPdZ8gQ0eLwQLQzbxWwB1z4 Gqy33c0hFsOo3LNuWd+wsbBrWUJCzYGn33f/7ymUzaefLYJcnDczzXIgoV6dveJE+PGQ bS8MW1hICuLHmM8K4ctwQLRNVLpW7u5QCqMLAYgAUyEdK4DVlbPqbCs2mQN3kYTqF29b tYcQ== X-Received: by 10.152.207.37 with SMTP id lt5mr33668983lac.66.1419423797063; Wed, 24 Dec 2014 04:23:17 -0800 (PST) Received: from ?IPv6:2a02:6b8::408:bcc5:e03b:4a46:f865? ([2a02:6b8:0:408:bcc5:e03b:4a46:f865]) by mx.google.com with ESMTPSA id fb6sm6611578lbc.46.2014.12.24.04.23.15 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 24 Dec 2014 04:23:15 -0800 (PST) Content-Type: text/plain; charset=koi8-r Mime-Version: 1.0 (Mac OS X Mail 8.1 \(1993\)) Subject: Re: mmap() question From: Dmitry Sivachenko In-Reply-To: <20141008170152.GG2153@kib.kiev.ua> Date: Wed, 24 Dec 2014 15:23:14 +0300 Content-Transfer-Encoding: quoted-printable Message-Id: <48AF6092-DD43-470B-A89A-93EA8A38CDC1@gmail.com> References: <95E0B821-BF9B-4EBF-A1E5-1DDCBB1C3D1B@gmail.com> <20131011051702.GE41229@kib.kiev.ua> <20131012095919.GI41229@kib.kiev.ua> <5C10922E-7030-4C89-9FD3-DA770E462067@gmail.com> <20141008170152.GG2153@kib.kiev.ua> To: Konstantin Belousov X-Mailer: Apple Mail (2.1993) Cc: "hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Dec 2014 12:23:19 -0000 > On 8 =CF=CB=D4. 2014 =C7., at 21:01, Konstantin Belousov = wrote: >>=20 >=20 > I wrote something different after discussion with Alan, it instead = measures > the pagedaemon progress. Unfortunately, I have some issues with = auto-tuning > the detection. Could you try the patch below. Possibly, you would = need > to change vm.pageout_oom_miss (either lower or raise, depending on the > load) to get the tuning. Bigger the value, more the time pagedaemon = tries > to make any progress before declaring OOM. Sorry for the delay. I tried your patch and I was unable to reproduce = test program crash with OOM error. Thanks!= From owner-freebsd-hackers@FreeBSD.ORG Wed Dec 24 20:57:35 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C8E88B6B; Wed, 24 Dec 2014 20:57:35 +0000 (UTC) Received: from dmz-mailsec-scanner-5.mit.edu (dmz-mailsec-scanner-5.mit.edu [18.7.68.34]) by mx1.freebsd.org (Postfix) with ESMTP id 58A331502; Wed, 24 Dec 2014 20:57:35 +0000 (UTC) X-AuditID: 12074422-f79476d000000d9e-ae-549b278ba112 Received: from mailhub-auth-1.mit.edu ( [18.9.21.35]) (using TLS with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by dmz-mailsec-scanner-5.mit.edu (Symantec Messaging Gateway) with SMTP id AE.88.03486.B872B945; Wed, 24 Dec 2014 15:52:27 -0500 (EST) Received: from outgoing.mit.edu (outgoing-auth-1.mit.edu [18.9.28.11]) by mailhub-auth-1.mit.edu (8.13.8/8.9.2) with ESMTP id sBOKqRxr032704; Wed, 24 Dec 2014 15:52:27 -0500 Received: from multics.mit.edu (system-low-sipb.mit.edu [18.187.2.37]) (authenticated bits=56) (User authenticated as kaduk@ATHENA.MIT.EDU) by outgoing.mit.edu (8.13.8/8.12.4) with ESMTP id sBOKqPAk027006 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Wed, 24 Dec 2014 15:52:26 -0500 Received: (from kaduk@localhost) by multics.mit.edu (8.12.9.20060308) id sBOKqO4g028501; Wed, 24 Dec 2014 15:52:24 -0500 (EST) Date: Wed, 24 Dec 2014 15:52:24 -0500 (EST) From: Benjamin Kaduk X-X-Sender: kaduk@multics.mit.edu To: freebsd-hackers@freebsd.org Subject: Re: Call for FreeBSD 2014Q4 (October-December) Status Reports In-Reply-To: Message-ID: References: User-Agent: Alpine 1.10 (GSO 962 2008-03-14) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-Brightmail-Tracker: H4sIAAAAAAAAA+NgFvrKIsWRmVeSWpSXmKPExsUixCmqrNutPjvEYM5ta4td106zW8x584HJ Yvvmf4wOzB4zPs1nCWCM4rJJSc3JLEst0rdL4MrYtSC14B1PxaobOQ2Me7m6GDk5JARMJJ7M vMAGYYtJXLi3Hsjm4hASWMwksb79KhOEs5FRYvmONSwQziEmiQlL7rNCOA2MEmd27QTrZxHQ lrj+4S0riM0moCbxeG8zK8RcRYnNpyYxdzFycIgIyEssOG8PEmYGMv9fucwEEhYWcJO49U8O xOQUcJKYeYQXpIJXwFFi5sb1zCC2EJDd/XkZmC0qoCOxev8UFogaQYmTM5+wQEzUklg+fRvL BEahWUhSs5CkFjAyrWKUTcmt0s1NzMwpTk3WLU5OzMtLLdI11cvNLNFLTSndxAgKXXYXpR2M Pw8qHWIU4GBU4uE9ED8rRIg1say4MvcQoyQHk5Io7zf22SFCfEn5KZUZicUZ8UWlOanFhxgl OJiVRHjfqwDleFMSK6tSi/JhUtIcLErivJt+8IUICaQnlqRmp6YWpBbBZGU4OJQkeKvVgBoF i1LTUyvSMnNKENJMHJwgw3mAhqeA1PAWFyTmFmemQ+RPMSpKifM2giQEQBIZpXlwvbDU8opR HOgVYd5WkCoeYFqC634FNJgJaLDUrekgg0sSEVJSDYxRX7ZMrCr2dTmi1CSU6eCzP81uTtvu n/Ofd/XNCxBYc1lRTzGhdd2uXdbRnyyXpNX0HX7P+Hybf1eTRMqf9Nw2mavnWp6qcwsUyB6M 373ch/mMXID22w1nP7DVbTpwbe1fjWsX4n/sWSS2nIWrQCjldPqhJLZ7z9p2SDmZTN9io23+ eeMC11lKLMUZiYZazEXFiQCVRqR9CAMAAA== Cc: freebsd-current@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Dec 2014 20:57:36 -0000 Reminder: the submission deadline for 2014Q4 status reports is just two weeks away! Send submissions to monthly@ for inclusion in the report. -Ben, for the monthly@ team On Mon, 8 Dec 2014, Benjamin Kaduk wrote: > Dear FreeBSD Community, > > The deadline for the next FreeBSD Quarterly Status update is January 7, > 2015, for work done in October through December. > > Status report submissions do not have to be very long. They may be > about anything happening in the FreeBSD project and community, and > provide a great way to inform FreeBSD users and developers about what > you're working on. Submission of reports is not restricted to > committers. Anyone doing anything interesting and FreeBSD-related > can -- and should -- write one! > > The preferred and easiest submission method is to use the XML > generator [1] with the results emailed to the status report team at > monthly at freebsd.org . There is also an XML template [2] which can be > filled out manually and attached if preferred. For the expected > content and style, please study our guidelines on how to write a good > status report [3]. You can also review previous issues [4][5] for > ideas on the style and format. > > We are looking forward to all of your 2014Q4 reports! > > Thanks, > Ben (on behalf of monthly@) > > > [1] http://www.freebsd.org/cgi/monthly.cgi > [2] http://www.freebsd.org/news/status/report-sample.xml > [3] http://www.freebsd.org/news/status/howto.html > [4] http://www.freebsd.org/news/status/report-2014-04-2014-06.html > [5] http://www.freebsd.org/news/status/report-2014-07-2014-09.html > > From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 25 00:28:25 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 402666E7; Thu, 25 Dec 2014 00:28:25 +0000 (UTC) Received: from mail-pd0-x233.google.com (mail-pd0-x233.google.com [IPv6:2607:f8b0:400e:c02::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0C40819BC; Thu, 25 Dec 2014 00:28:25 +0000 (UTC) Received: by mail-pd0-f179.google.com with SMTP id fp1so10739904pdb.38; Wed, 24 Dec 2014 16:28:24 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=exMtCxRjUJI4+4ndAK79GKX3nGTQx8AN4SK5Lorj7NA=; b=G+/QkhO9c/ndnXuWWQRMEmHQ4SI2pg2XA7sIZzVQjrdygCxq7WvU5yKUHSALf6SdfS yGOuyUWU/iLynCjUbmHZvdaiQ5H8NK0w2bWuxs4ffUTpdQnZtgZ4WBgS9GfYA5iaSlJK JSUw4vYJVg8hGI8AmXYYzEFzcUZEgtIdhU456KluBXGQt+bqd+3L75tDgUF5Rb0rwvwa HmF4QQSl+k7a7c/oX0OkOKTPcEIvSVh0akN2GWKjFHZ3j/nlrv4cGtPaSo2lSaEhm/vN XhwT7xQyJA7qExFIYAFommfSnF06R6bYB/lZFI72Wzt+ZV2nuQn1U5LsGWisR2Ohyr6u SXzw== X-Received: by 10.68.226.69 with SMTP id rq5mr57810450pbc.116.1419467304597; Wed, 24 Dec 2014 16:28:24 -0800 (PST) Received: from [172.30.255.118] ([61.148.244.82]) by mx.google.com with ESMTPSA id o10sm23815518pdr.96.2014.12.24.16.28.22 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 24 Dec 2014 16:28:23 -0800 (PST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (1.0) Subject: Re: status of projects/numa? From: Gavin Mu X-Mailer: iPad Mail (12B440) In-Reply-To: Date: Thu, 25 Dec 2014 08:28:19 +0800 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Adrian Chadd Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Dec 2014 00:28:25 -0000 Hi, Adrian, Thanks for such detail information. I think now I can and will: 1. start from projects/numa and have a summary first. 2. post the summary and thoughts out for discussion. 3. merge latest HEAD code. 4. implement the idea and post the patch for review. Regards, Gavin Mu > On Dec 24, 2014, at 11:09, Adrian Chadd wrote: >=20 > Ok, so to summarise the various bits/pieces I hear from around the > grapevine and speaking with numa people like jhb: >=20 > The projects/numa stuff has a lot of good stuff in it. Jeff/Isilon's > idea seems to be to create an allocator policy framework that can be > specified global, per-process and/or per-thread. That's pretty good > stuff right there. >=20 > What's missing however before it can be useful by drivers is the rest > of the UMA stuff and I think there's some VM pagetable work that needs > doing. I haven't dug into the latter because I was trusting that those > working on projects/numa will get to it like they've said, but .. > well, it's still not done. >=20 > =46rom a UMA API perspective, it isn't trying very hard to return memory > back to the domain that it was allocated from. I'm worried that on a > real NUMA box with no policy configured or used by userland threads > (ie, everything is being scheduled everywhere), we'll end up with > threads allocating from a CPU-local pool, being migrated to another > CPU, then freeing it there. So over time it'll end up with the cache > spiked with non-local pages. I talked to jhb about it but I don't > think we have a consensus about it at all. I think we should go the > extra mile of ensuring that when we return pages to UMA it goes back > into the right NUMA domain - I don't want to have to debug systems > that start really fast and then end up getting slower over time as > stuff ends up on the wrong CPU. But that's just me. >=20 > =46rom the physmem perspective, I'll likely implement an SRAT-aware > allocator option that allocates local-first, then tries to allocate > from locall-er domains until it just round-robins. We have that cost > matrix that says how expensive things are from a given NUMA domain, so > that shouldn't be too difficult to pre-compute. >=20 > =46rom a driver perspective, I've added the basic "which domain am I in" > call. John's been playing with a couple of drivers (igb/ixgbe I > think) in his local repository for teaching them about what is a local > interrupt and local CPU set to start threads on. Ideally all the > drivers would use the same API for querying what their local cpuset is > and assigning worker threads / interrupts appropriately. I should poke > him again to find the status of that work and at least get that into > -HEAD so it can be evaluated and used by people. >=20 > What I'm hoping to do with that in the short term is to make it > generic enough so that a generic, consistent set of hints can be > configured for a given driver to setup its worker thread and cpuset > map. Ie, instead of each driver having its own "how many queues" > sysctl and probe logic, it'll have some API to say "give me my number > of threads and local cpuset(s)" so if someone wants to override it, > it's done by something like code in the bus layer rather than above > individual driver hacks. We can also add options for things like "pin > threads" or not. >=20 > =46rom the driver /allocation/ perspective, there's a few things to think a= bout: >=20 > * how we allocate busdma memory for things like descriptors; > * how we allocate DMA memory for things like mbufs, bufs, etc - what > we're dma'ing into and out of. >=20 > None of that is currently specified, although there's a couple of > tidbits in the projects/numa branch that haven't been fleshed out. >=20 > In my vague drawing-on-paper sense of this, I think we also should > then extend busdma a little to be aware of a numaset for allocating > memory for descriptors. Same with calling malloc and contigmalloc. > Ideally we'd keep descriptor accesses in memory local to the device. >=20 > Now for things like bufs/mbufs/etc - the DMA bits - that's where it > gets a little tricky. Sometimes we're handed memory to send from/to. > The NIC RX path allocates memory itself. storage controllers get > handed memory to read storage data into. So, the higher layer question > of "how do we tell something where to allocate memory from" gets a > little complicated, because we (a) may have to allocate memory for a > non-local device, knowing we're going to hand it to some device on > another core, and (b) we then have to return it to the right NUMA > aware pool (UMA, vm_page stuff, etc.) >=20 > The other tricksy bit is when drivers want to allocate memory for > local ephemeral work (eg M_TEMP) versus DMA/descriptor/busdma stuff. > Here's about where I say "don't worry about this - do all the above > bits first then worry about worrying about this": >=20 > say we have a NIC driver on an 8-core CPU, and it's in a 4-socket box > (so 4 sockets * 8 cores each socket.) The receive threads can be told > to run in a specific cpuset local to the CPU the NIC is plugged into. > But the transmit threads, taskqueue bits, etc may run on any CPU - > remember, we don't queue traffic and wakeup a thread now; we call > if_transmit() from whichever CPU is running the transmit code. So if > it's local then it'll come from NUMA memory domain local memory and > stuff is fine. But if it's transmitting from some /remote/ thread: >=20 > * the mbuf that's been allocated to send data is likely allocated from > the wrong NUMA domain; > * the local memory used by the driver to do work should be coming from > memory local to that CPU, but it may call malloc/etc to get temporary > working memory - and that hopefully is also coming from a local memory > domain; > * if it has to allocate descriptor memory or something then it may > need to allocate memory from the remote memory domain; > * .. then touch the NIC hardware, which is remote. >=20 > This is where I say "screw it, don't get bogged down in the details." > It gets hairier when we think about what's local versus remote for > vm_page entries, because hey, who knows what the right thing to do > there is. But I think the right thing to do here is to not worry about > it and get the above stuff done. For NUMA aware network applications > they're likely going to be CPU set to threads local to the same domain > as the NIC, and if that isn't good enough, we'll have to come up with > some way to mark a socket as local to a NUMA domain so things like > mbufs, TCP timers, etc get scheduled appropriately. >=20 > (This is where I say "And this is where the whole RSS framework and > NUMA need to know about each other", but I'm not really ready for that > yet.) >=20 > So, that's my braindump of what I remember from discussing with others. >=20 > I think the stages to go here are: >=20 > * get a firmer idea of what's missing from projects/numa for UMA, > physmem and VM allocator / pagedaemon/etc stuff; > * get jhb's numaset code into -HEAD; > * get jhb's driver NUMA awareness bits out, add more generic hooks for > expressing this stuff via hints, and dump that into -HEAD; > * evaluate what we're going to do about mbufs, malloc/contigmalloc for > busdma, bounce buffers, descriptors, etc and get that into -HEAD; > * make sure the intel-pcm tools (and then hwpmc!) work for the uncore > CPU interconnects work so we can measure how well we are/aren't doing > things; > * come back to the drawing board once the above is done and we've got > some experience with it. >=20 > I haven't even started talking about what's needed for /userland/ > memory pages. I think that's part of what needs to be fleshed out / > discussed with the VM bits. >=20 >=20 >=20 > -adrian From owner-freebsd-hackers@FreeBSD.ORG Sat Dec 27 16:44:25 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1EE95E41 for ; Sat, 27 Dec 2014 16:44:25 +0000 (UTC) Received: from mail.metricspace.net (207-172-209-89.c3-0.arl-ubr1.sbo-arl.ma.static.cable.rcn.com [207.172.209.89]) by mx1.freebsd.org (Postfix) with ESMTP id E8CD866AFE for ; Sat, 27 Dec 2014 16:44:24 +0000 (UTC) Received: from [172.16.0.68] (unknown [172.16.0.68]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) (Authenticated sender: eric) by mail.metricspace.net (Postfix) with ESMTPSA id 492851FCB for ; Sat, 27 Dec 2014 16:44:23 +0000 (UTC) Subject: Dual-boot with common ZFS From: Eric McCorkle Content-Type: text/plain; charset=us-ascii X-Mailer: iPad Mail (11B511) Message-Id: <4A40BFDA-ABE9-4AB5-947F-C11343B9630B@metricspace.net> Date: Sat, 27 Dec 2014 11:44:24 -0500 To: "freebsd-hackers@freebsd.org" Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Dec 2014 16:44:25 -0000 It looks like I'm going to have to install Linux on my laptop for a while be= cause of lack of driver support (haswell graphics). However, I thought of a= n idea and wanted to see if anyone has done something similar. I have a pure-ZFS setup, with a GPT. I was wondering if it might be possibl= e to use the same ZFS volume to hold both a Linux and a FreeBSD installation= . At least on the surface, it seems like you ought to be able to create sep= arate file systems for each OS. The real trick, I think, would be in findin= g the right boot/loader.conf arguments to point the kernels at the correct r= oot, and the the right flags for ZFS mount to mount the other file systems c= orrectly. I can think of several things you could do with this scheme, among them bein= g Linux-to-FreeBSD driver porting. There's probably things you could do wit= h virtualization layers as well. Has anyone attempted such a setup, and if so, what were your experiences?= From owner-freebsd-hackers@FreeBSD.ORG Sat Dec 27 17:54:17 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 03F064F6 for ; Sat, 27 Dec 2014 17:54:17 +0000 (UTC) Received: from mx1.scaleengine.net (beauharnois2.bhs1.scaleengine.net [142.4.218.15]) by mx1.freebsd.org (Postfix) with ESMTP id B8DE26487F for ; Sat, 27 Dec 2014 17:54:16 +0000 (UTC) Received: from [192.168.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id BF68F89EF8 for ; Sat, 27 Dec 2014 17:54:15 +0000 (UTC) Message-ID: <549EF25D.8010303@freebsd.org> Date: Sat, 27 Dec 2014 12:54:37 -0500 From: Allan Jude User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: Dual-boot with common ZFS References: <4A40BFDA-ABE9-4AB5-947F-C11343B9630B@metricspace.net> In-Reply-To: <4A40BFDA-ABE9-4AB5-947F-C11343B9630B@metricspace.net> Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="tVc7g4BcfuIuDpmGDdihbpjxSqO9IW3qQ" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Dec 2014 17:54:17 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --tVc7g4BcfuIuDpmGDdihbpjxSqO9IW3qQ Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2014-12-27 11:44, Eric McCorkle wrote: > It looks like I'm going to have to install Linux on my laptop for a whi= le because of lack of driver support (haswell graphics). However, I thou= ght of an idea and wanted to see if anyone has done something similar. >=20 > I have a pure-ZFS setup, with a GPT. I was wondering if it might be po= ssible to use the same ZFS volume to hold both a Linux and a FreeBSD inst= allation. At least on the surface, it seems like you ought to be able to= create separate file systems for each OS. The real trick, I think, woul= d be in finding the right boot/loader.conf arguments to point the kernels= at the correct root, and the the right flags for ZFS mount to mount the = other file systems correctly. >=20 > I can think of several things you could do with this scheme, among them= being Linux-to-FreeBSD driver porting. There's probably things you coul= d do with virtualization layers as well. >=20 > Has anyone attempted such a setup, and if so, what were your experience= s? > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.o= rg" >=20 I have not attempted such a setup, but if I was going to, I'd probably use the GRUB boot loader, like PCBSD does. Then you can use their ZFS drivers, and specify the location of the kernel (on different ZFS datasets), and what dataset to mount as root etc= =2E Basically, ZFS boot environments, where you have 1 that is FreeBSD and 1 that is Linux. Your biggest issue with doing this with an existing ZFS install, is that you may have used ZFS features that are too new for Linux to support them. You might have to start over with a pool created with some of the feature flags turned off (or create the pool under Linux, and don't upgrade it on FreeBSD). --=20 Allan Jude --tVc7g4BcfuIuDpmGDdihbpjxSqO9IW3qQ Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJUnvJgAAoJEJrBFpNRJZKfY5oP/345uXwPm205ecM3WN7J9rWD NdoptHDMuxoICScN66m6eMsum72t6i/hzB7MI1vWRwwzplZ8kqyVy1Jx3jfJIgTg JjmtE/aaiZol8gxYfi/d6SLSjSZyqYyyUVelydkiSzag+hMwLP/R9lQ++sKtiw9x AjYPaZmvyA+6Emri08hEoL1elrsqyBOBGTCk1BW5F00UE16LbRYLo1Ok9/KRAafr xSTf/IqGEwuGHDnhAh7zBw72b9vAtMzjUK4UpHHjzrzSfuiARUa28nYQrt3tVHB6 qg71cOybn2lhlZtW9NudHARNF8uePD8Yc9AjGNMvDvfJLIvWOKZT2aK8x5MqeD4A /AsrePjSFv35joJ3NGsiewXmqj7Ke2xG0PVXoGLX0UcoEY/jgG8yeodGlfeRM+AB GrhgIWx5P+zPmeIjI7K+wafZ5yfe0HzOQEs5Z9wwjD4hAcxO8D0h2yS7x7K04xFg XJdCgO9wK6op7vjaCgsu4XIfGLPpBBJhvZn8MpoldjuAE9T3a14i4InEzFmmMASx 5ayBdGGdXRccYWnQPbSJBEP2fXXCpEc9Ralj0R0QfBP9QyjVDIbZJNXeCHdY2An7 2PW4HDZgCbs1Jn5//XSl3f57aRzdjWGW3/7UptaPwLx5Zi6rXwY8pF+wnYcJMNAq ku877FBJKsmhh4SM1/8J =O7Xn -----END PGP SIGNATURE----- --tVc7g4BcfuIuDpmGDdihbpjxSqO9IW3qQ-- From owner-freebsd-hackers@FreeBSD.ORG Sat Dec 27 18:53:39 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 966A4FF4 for ; Sat, 27 Dec 2014 18:53:39 +0000 (UTC) Received: from mail-qg0-x234.google.com (mail-qg0-x234.google.com [IPv6:2607:f8b0:400d:c04::234]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B71A66F50 for ; Sat, 27 Dec 2014 18:53:39 +0000 (UTC) Received: by mail-qg0-f52.google.com with SMTP id a108so7924956qge.25 for ; Sat, 27 Dec 2014 10:53:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:content-type:content-transfer-encoding; bh=gOw3I7wCttMgzWCuxjzQ//PcmoWermqBT4urdabSotM=; b=TAuEtINSYhdbm3a5M0Lx11NPcq8k99AJcBv6wtYbqByP1v9OXQV6fEGCDqUvsyycwH AsKKpBemzJEHYT88y9VSpY327x6tpt8v0f/QIJnQAHo0fYRbD1PE5sAlhSQhKGf/69Mw 0pOks8igKk5jTOcdp5EBaqPI8pAnRf4KopANg06vlTF7tEbAFs0otwU2GKwR2989bjJ0 odtl6TrWe6abm2Iw1kdQ8aJkAnuyd7fAY9p/mR0J7+lGKVQ18yhix/DOGURLpISDGGem 9buKMxW9/aoRXnz1CnNQJItiYQ3htGrnzkcgxcfhXlXQJwTnKhMHd+ShXukpv0v2Ex8x PaMA== X-Received: by 10.224.63.198 with SMTP id c6mr36816457qai.41.1419706418487; Sat, 27 Dec 2014 10:53:38 -0800 (PST) Received: from [192.168.1.1] (adsl-98-66-33-141.mem.bellsouth.net. [98.66.33.141]) by mx.google.com with ESMTPSA id t67sm29146125qgt.11.2014.12.27.10.53.37 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 27 Dec 2014 10:53:38 -0800 (PST) Message-ID: <549F0030.8030206@gmail.com> Date: Sat, 27 Dec 2014 12:53:36 -0600 From: Garry Roseman User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.3.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: Dual-boot with common ZFS References: <4A40BFDA-ABE9-4AB5-947F-C11343B9630B@metricspace.net> <549EF25D.8010303@freebsd.org> In-Reply-To: <549EF25D.8010303@freebsd.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Dec 2014 18:53:39 -0000 On 12/27/2014 11:54 AM, Allan Jude wrote: > Your biggest issue with doing this with an existing ZFS install, is > that you may have used ZFS features that are too new for Linux to > support them. You might have to start over with a pool created with > some of the feature flags turned off (or create the pool under Linux, > and don't upgrade it on FreeBSD). I dual boot Gentoo (drive 0) and FreeBSD (drive 1) and share a zfs pool (mirror drives 2 and 3) between them. It was indeed necessary to create the pool under Gentoo and not allow FreeBSD to upgrade it. Luckily I have an "old" Sandybridge cpu and FreeBSD Intel graphics is now very good for me.