Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 12 Dec 2017 14:32:26 -0500
From:      John Baldwin <jhb@FreeBSD.org>
To:        Eugene Grosbein <eugen@grosbein.net>, Konstantin Belousov <kostikbel@gmail.com>, Conrad Meyer <cem@FreeBSD.org>
Cc:        src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org
Subject:   Re: svn commit: r326758 - in head/sys/i386: conf include
Message-ID:  <4a9c76c9-8063-9420-b198-14487b089840@FreeBSD.org>
In-Reply-To: <5A2E5D44.9030904@grosbein.net>
References:  <201712110432.vBB4WbnE021090@repo.freebsd.org> <20171211091943.GF2272@kib.kiev.ua> <5A2E5D44.9030904@grosbein.net>

next in thread | previous in thread | raw e-mail | index | archive | help
On 12/11/17 5:26 AM, Eugene Grosbein wrote:
> On 11.12.2017 16:19, Konstantin Belousov wrote:
>> On Mon, Dec 11, 2017 at 04:32:37AM +0000, Conrad Meyer wrote:
>>> Author: cem
>>> Date: Mon Dec 11 04:32:37 2017
>>> New Revision: 326758
>>> URL: https://svnweb.freebsd.org/changeset/base/326758
>>>
>>> Log:
>>>   i386: Bump KSTACK_PAGES default to match amd64
>> i386 is not amd64, the change is wrong.
>>
>> i386 has the word size two times smaller than amd64, which makes typical
>> frame smaller by 30-40% over same code on amd64. Also i386 has much
>> smaller available KVA size (tens of MB) and KVA fragmentation is both
>> more severe and more fatal due to this. I expect that your change will
>> make any non-trivial load which creates enough threads to either fail
>> randomly or deadlock.
>>
>> If somebody tries to fit large load onto i386 machine, he must know what to
>> do and how to configure the kernel to adapt to the load (which does not
>> require the recompilation).
> 
> Its very easy to get kernel stack overflow with 11-STABLE/i386
> without any significant load due to abuse of kernel stack in many kernel subsystems
> as shown in the https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=219476
> 
> Contrary, "enough threads" seems to be very non-trivial number of threads
> and pretty unusual load pattern for i386 as I run several such systems
> with kern.kstack_pages=4 for quite long time and have no problems.
> No random fails, no deadlocks. And with 2 pages only 11-STABLE/i386 is just unusable
> if one utilizes SCTP, IPv6, some WiFi connectivity, IPSEC or even very small ZFS pool.
> 
> I still wonder if there is really such load pattern that creates "enough threads"
> for i386 to make 4-pages stack troublesome.

I suspect two things are at play in running out of stack in 10.x and later.
1) Usage of XSAVE for AVX, etc. state uses more of the kstack (as kib@ alludes
to), and 2) clang.  Certainly for MIPS I have found that compiling with clang
instead of gcc for mips64 gives a kernel that panics for stack overflow for any
use of NFS.  It might be that this is due to something MIPS-specific, but it
might be worthwhile retesting with kstack_pages=2 and building the kernel
with CROSS_TOOLCHAIN=i386-gcc after installing the appropriate package.

For people wanting to use 32-bit binaries on a modern x86 box, we should probably
add support for the x32 ABI permitting you to use 32-bit pointers but with the
full complement of 64-bit registers (and still using the 64-bit kernel so
there is not KVA pressure for the kernel itself).

-- 
John Baldwin



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4a9c76c9-8063-9420-b198-14487b089840>