From owner-svn-src-head@freebsd.org Tue Dec 12 19:32:29 2017 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AFCA2E819CC; Tue, 12 Dec 2017 19:32:29 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from mail.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 87D406EA15; Tue, 12 Dec 2017 19:32:29 +0000 (UTC) (envelope-from jhb@FreeBSD.org) Received: from John-Baldwins-MacBook-Pro-3.local (unknown [128.57.46.190]) by mail.baldwin.cx (Postfix) with ESMTPSA id DB2A910A8BA; Tue, 12 Dec 2017 14:32:27 -0500 (EST) Subject: Re: svn commit: r326758 - in head/sys/i386: conf include To: Eugene Grosbein , Konstantin Belousov , Conrad Meyer References: <201712110432.vBB4WbnE021090@repo.freebsd.org> <20171211091943.GF2272@kib.kiev.ua> <5A2E5D44.9030904@grosbein.net> Cc: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org From: John Baldwin Message-ID: <4a9c76c9-8063-9420-b198-14487b089840@FreeBSD.org> Date: Tue, 12 Dec 2017 14:32:26 -0500 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: <5A2E5D44.9030904@grosbein.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (mail.baldwin.cx); Tue, 12 Dec 2017 14:32:28 -0500 (EST) X-Virus-Scanned: clamav-milter 0.99.2 at mail.baldwin.cx X-Virus-Status: Clean X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Dec 2017 19:32:29 -0000 On 12/11/17 5:26 AM, Eugene Grosbein wrote: > On 11.12.2017 16:19, Konstantin Belousov wrote: >> On Mon, Dec 11, 2017 at 04:32:37AM +0000, Conrad Meyer wrote: >>> Author: cem >>> Date: Mon Dec 11 04:32:37 2017 >>> New Revision: 326758 >>> URL: https://svnweb.freebsd.org/changeset/base/326758 >>> >>> Log: >>> i386: Bump KSTACK_PAGES default to match amd64 >> i386 is not amd64, the change is wrong. >> >> i386 has the word size two times smaller than amd64, which makes typical >> frame smaller by 30-40% over same code on amd64. Also i386 has much >> smaller available KVA size (tens of MB) and KVA fragmentation is both >> more severe and more fatal due to this. I expect that your change will >> make any non-trivial load which creates enough threads to either fail >> randomly or deadlock. >> >> If somebody tries to fit large load onto i386 machine, he must know what to >> do and how to configure the kernel to adapt to the load (which does not >> require the recompilation). > > Its very easy to get kernel stack overflow with 11-STABLE/i386 > without any significant load due to abuse of kernel stack in many kernel subsystems > as shown in the https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219476 > > Contrary, "enough threads" seems to be very non-trivial number of threads > and pretty unusual load pattern for i386 as I run several such systems > with kern.kstack_pages=4 for quite long time and have no problems. > No random fails, no deadlocks. And with 2 pages only 11-STABLE/i386 is just unusable > if one utilizes SCTP, IPv6, some WiFi connectivity, IPSEC or even very small ZFS pool. > > I still wonder if there is really such load pattern that creates "enough threads" > for i386 to make 4-pages stack troublesome. I suspect two things are at play in running out of stack in 10.x and later. 1) Usage of XSAVE for AVX, etc. state uses more of the kstack (as kib@ alludes to), and 2) clang. Certainly for MIPS I have found that compiling with clang instead of gcc for mips64 gives a kernel that panics for stack overflow for any use of NFS. It might be that this is due to something MIPS-specific, but it might be worthwhile retesting with kstack_pages=2 and building the kernel with CROSS_TOOLCHAIN=i386-gcc after installing the appropriate package. For people wanting to use 32-bit binaries on a modern x86 box, we should probably add support for the x32 ABI permitting you to use 32-bit pointers but with the full complement of 64-bit registers (and still using the 64-bit kernel so there is not KVA pressure for the kernel itself). -- John Baldwin