Date: Sat, 20 Sep 2014 20:06:58 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: John Baldwin <jhb@freebsd.org> Cc: "Justin T. Gibbs" <gibbs@scsiguy.com>, "Ivan A. Kosarev" <ivan@ivan-labs.com>, freebsd-current@freebsd.org, doc@freebsd.org Subject: Re: libthr and main thread stack size Message-ID: <20140920170658.GE2210@kib.kiev.ua> In-Reply-To: <5242716.s4iaScq0Bu@ralph.baldwin.cx> References: <53E36E84.4060806@ivan-labs.com> <FEB60EB5-546D-454D-AE62-B2483246E42C@scsiguy.com> <20140916081324.GQ2737@kib.kiev.ua> <5242716.s4iaScq0Bu@ralph.baldwin.cx>
next in thread | previous in thread | raw e-mail | index | archive | help
--pAwQNkOnpTn9IO2O
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Fri, Sep 19, 2014 at 03:27:25PM -0400, John Baldwin wrote:
> I suspect it was done out of reasons of being overly conservative in
> interpreting RLIMIT_STACK. I think it is quite surprising behavior
> though and would rather we make your option the default and implement
> what the Open Group says above.
Ok, below is the patch. I felt bad about adding yet another magic and
undocumented tunable to our libthr. Since there seems to be no
alternative than a tunable to enforce old behaviour, I documented
the quirks I am aware of.
Doc people, please review the man page in the patch.
diff --git a/lib/libthr/thread/thr_init.c b/lib/libthr/thread/thr_init.c
index 9bf0e29..72a067a 100644
--- a/lib/libthr/thread/thr_init.c
+++ b/lib/libthr/thread/thr_init.c
@@ -445,7 +445,7 @@ init_private(void)
struct rlimit rlim;
size_t len;
int mib[2];
- char *env;
+ char *env, *env_bigstack, *env_splitstack;
=20
_thr_umutex_init(&_mutex_static_lock);
_thr_umutex_init(&_cond_static_lock);
@@ -473,8 +473,9 @@ init_private(void)
len =3D sizeof (_usrstack);
if (sysctl(mib, 2, &_usrstack, &len, NULL, 0) =3D=3D -1)
PANIC("Cannot get kern.usrstack from sysctl");
- env =3D getenv("LIBPTHREAD_BIGSTACK_MAIN");
- if (env !=3D NULL) {
+ env_bigstack =3D getenv("LIBPTHREAD_BIGSTACK_MAIN");
+ env_splitstack =3D getenv("LIBPTHREAD_SPLITSTACK_MAIN");
+ if (bigstack !=3D NULL || env_splitstack =3D=3D NULL) {
if (getrlimit(RLIMIT_STACK, &rlim) =3D=3D -1)
PANIC("Cannot get stack rlimit");
_thr_stack_initial =3D rlim.rlim_cur;
diff --git a/share/man/man7/libthr.7 b/share/man/man7/libthr.7
new file mode 100644
index 0000000..16d916f
--- /dev/null
+++ b/share/man/man7/libthr.7
@@ -0,0 +1,254 @@
+.\" Copyright (c) 2014 The FreeBSD Foundation, Inc.
+.\" All rights reserved.
+.\"
+.\" This documentation was written by Konstantin Belousov <kib@FreeBSD.org>
+.\" under sponsorship from the FreeBSD Foundation.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\" notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\" notice, this list of conditions and the following disclaimer in the
+.\" documentation and/or other materials provided with the distribution.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
+.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PUR=
POSE
+.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIAB=
LE
+.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUEN=
TIAL
+.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, ST=
RICT
+.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY =
WAY
+.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+.\" SUCH DAMAGE.
+.\"
+.\" $FreeBSD$
+.\"
+.Dd September 20, 2014
+.Dt libthr
+.Os
+.Sh NAME
+.Nm libthr
+.Nd FreeBSD implementation of the Posix threading library
+.Sh LIBRARY
+.Lb libpthread
+.Sh DESCRIPTION
+The man page documents the quirks and tunables of the
+.Fx
+implementation for the
+.Lb libpthread .
+When linking with the
+.Li -lpthread ,
+the run-time dependency
+.Dv libthr.so.3
+library is recorded in the produced object.
+.Pp
+The library is tigthly integrated with the Run-time Link-editor
+.Xr ld-elf.so.1 1
+and
+.Lb libc ,
+all three components must be built from the same source tree.
+Mixing
+.Dv libc.so
+and
+.Nm
+libraries from different versions of
+.Fx
+is not supported.
+The run-time linker
+.Li ld-elf.so.1
+has some code to ensure backward-compatibility with older
+.Nm .
+.Sh MUTEX ACQUISITION
+The locked mutex (see
+.Xr pthread_mutex_lock 3 )
+is represented by a volatile variable of type
+.Dv lwpid_t ,
+which records the global system identifier of the thread
+owning the lock.
+The
+.Nm
+performs a congested mutex acquisition in three stages, each of which
+is more resource-consuming than the previous.
+.Pp
+First, the
+.Li spin loop
+is performed, where the library attempts to acquire the lock by
+.Xr atomic 9
+operations.
+The loop count is controlled by the
+.Ev LIBPTHREAD_SPINLOOPS
+environment variable.
+.Pp
+If the
+.Li spin loop
+was unable to acquire the mutex, the
+.Li yield loop
+is executed, performing the same
+.Xr atomic 9
+acquisition attempts as
+.Li spin loop ,
+but each attempt is followed by yield of the CPU time of the thread by
+.Xr sched_yield 2
+syscall.
+By default, the
+.Li yield loop
+is not executed.
+This is controlled by
+.Ev LIBPTHREAD_YIELDLOOPS
+environment variable.
+.Pp
+If both
+.Li spin
+and
+.Li yield loops
+failed to acquire the lock, the thread is taken off the CPU and
+put to sleep in kernel with the
+.Xr umtx 2
+syscall.
+Kernel wakes up a thread and hands the ownership of the lock to
+the woken thread.
+.Sh THREADS STACKS
+Each thread is provided with the private stack area used by C runtime.
+The size of the main (initial) thread stack is set by kernel, and is
+controlled by the
+.Dv RLIMIT_STACK
+process resource limit (see
+.Xr getrlimit 2 ) .
+.Pp
+By default, the main thread size is equal to the value of resource
+.Dv RLIMIT_STACK
+for the process.
+If the
+.Dv LIBPTHREAD_SPLITSTACK_MAIN
+environment variable is present (its value does not matter),
+the main thread size if chomped to 4MB on 64bit architectures, and to
+2MB on 32bit architectures, on the threading library initialization.
+The rest of the address space area reserved by the kernel for initial
+process stack, is used for non-initial threads stack in this case.
+The presence of the
+.Dv LIBPTHREAD_BIGSTACK_MAIN
+environment variable overrides the
+.Dv LIBPTHREAD_SPLITSTACK_MAIN ,
+it is kept for backward-compatibility.
+.Pp
+The size of the stacks for threads created by the process at run-time
+with the
+.Xr pthread_create 3
+call, is controlled by thread attributes, see
+.Xr pthread_attr 3 ,
+in particular, the
+.Xr pthread_attr_setstacksize 3 ,
+.Xr pthread_attr_setguardsize 3
+and
+.Xr pthread_attr_setstackaddr 3 .
+If no attributes for the thread stack size are specified, the default
+non-initial thread stack size is 2MB for 64bit architectures, and 1MB
+for 32bit architectures.
+.Sh RUN-TIME SETTINGS
+The following environment variables are recognized by
+.Dv libthr
+and adjust the operation of the library at run-time:
+.Bl -tag -width LIBPTHREAD_SPLITSTACK_MAIN
+.It Ev LIBPTHREAD_BIGSTACK_MAIN
+Disables the chomp of the initial thread stack, enabled by
+.Ev LIBPTHREAD_SPLITSTACK_MAIN .
+.It Ev LIBPTHREAD_SPLITSTACK_MAIN
+Causes the chomp of the initial thread stack, as described in the
+section
+.Li THREAD_STACKS .
+This was the default behaviour of the
+.Nm
+before
+.Fx 11.0 .
+.It Ev LIBPTHREAD_SPINLOOPS
+The integer value of the variable overrides the default count of
+iterations in the
+.Li spin loop
+of the mutex acquisition.
+The default count is 2000, set by the
+.Dv MUTEX_ADAPTIVE_SPINS
+define in the
+.Nm
+sources.
+.It Ev LIBPTHREAD_YIELDLOOPS
+The non-zero integer value of the variable allows the
+.Li yield loop
+in the process of the mutex acquisition.
+The value is the counter of loop operations.
+.It Ev LIBPTHREAD_QUEUE_FIFO
+The integer value of the variable specifies how often the blocked
+threads are put into the head of the sleep queue, instead of it tail.
+The bigger value reduces the frequency of the FIFO discipline.
+The value must be between 0 and 255.
+.El
+.Sh INTERACTION WITH RUN-TIME LINKER.
+The
+.Nm
+library must appear before
+.Dv libc
+in the global order of depended objects.
+.Pp
+.Pp
+Loading the
+.Nm
+library with the
+.Xr dlopen 3
+call in the process after the program binary is activated,
+is not supported, and causes miscellaneous and hard to diagnose misbehavio=
ur.
+This is due to
+.Nm
+interposing several important
+.Dv libc
+symbols to provide thread-safe services.
+In particular,
+.Dv errno
+and locking stubs from
+.Dv libc
+are affected.
+This requirement is currently not enforced.
+.Pp
+If the program loads the modules at run-time, and modules may require
+the threading services, the main program binary must be linked with
+.Dv libpthread ,
+even if it does not require any service from the library.
+.Pp
+The library cannot be unloaded, the
+.Xr dlclose 3
+function does not perform any action when called with a handle for
+.Dv libthr .
+One of the reason is that interposing of
+.Dv libc
+functions cannot be undone.
+.Sh SIGNALS
+The implementation also interposes the user-installed
+.Xr signal 3
+handlers.
+The interposing is done to postpone signal delivery to threads which
+entered the (libthr-internal) critical sections, where the calling
+of the signal handler is unsafe.
+Example of such situation is owning the internal library lock.
+When the signal is delivered while signal handler cannot be safely
+called, call is postponed and performed after the critical section
+is left.
+This should be taken into account when interpreting the
+.Xr ktrace 1
+logs.
+.Sh SEE ALSO
+.Xr atomic 9 ,
+.Xr dlclose 3 ,
+.Xr dlopen 3 ,
+.Xr errno 3 ,
+.Xr getenv 3 ,
+.Xr getrlimit 2 ,
+.Xr ktrace 1 ,
+.Xr ld-elf.so.1 1 ,
+.Xr libc 3 ,
+.Xr pthread_attr 3 ,
+.Xr pthread_attr_setstacksize 3 ,
+.Xr pthread_create 3 ,
+.Xr signal 3 ,
+.Xr umtx 2 .
--pAwQNkOnpTn9IO2O
Content-Type: application/pgp-signature
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
iQIcBAEBAgAGBQJUHbQxAAoJEJDCuSvBvK1BJowQAIkGc0gokVCcaGUUXT+WPkuE
2u49WokeTewPizBT0040afss1FKeyeDKI5Gn+mogNl0IIh7XCOLNpL0B3qFtWf44
Ie3hkuuSN+p47e1auaN7KBuRAyWVHCVdJsXsEEO/MiJT8tqVt+fZiLfvSmNAWrse
9hgYupxmQRsNRRTz6m22zdRxAWZVhvM56pt/oFaHbM5DGXw2C9nNqg3EKZjnpqtc
A7uXJelz95k5T/mXK0yxr/aoG/rHv4ON57slvp1cRD+6PiSlsOsR3TWYRPmUgFZO
KEjBknEhSsuaXlh02fjMbY/Ht0r9VqTdijmYffnwxmF3cn4flCewn86zO+6mGo9B
162AS5XpxNMJsudCK/sP9+4VyxICB5A/V7LXESBNYVt67Et6jJBNBZXu50xbMTU+
l5wrx2mMc4XlDQZZM+KRnynzxF3B8FoAUBOvQV2Rw58hLqooJ1HR5rf1HJIdZ2yw
nJr7GH6hIsLYOTDGLHiZpMAYJMwgsAlm++ZPqQAf6a8P3Fqs9A0X40BC07JHS+G5
dCSJgtFd9nzuo75V7qyFpCZVPM4+q9jayWILihcrdjFi6ZuDDpMLjuqaxRHVVP/G
dw9FiV/W8u+Gx2cFdwUPcYQTbxgjgb5DFuTnmln9pRqiWTndwqk7hM6foMEG8qT9
f3QBuQLeUJrKqWpNAzfB
=z6GS
-----END PGP SIGNATURE-----
--pAwQNkOnpTn9IO2O--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140920170658.GE2210>