Date: Sat, 14 Apr 2012 00:45:36 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Sushanth Rai <sushanth_rai@yahoo.com> Cc: alc@freebsd.org, freebsd-hackers@freebsd.org Subject: Re: mlockall() on freebsd 7.2 + amd64 returns EAGAIN Message-ID: <20120413214536.GL2358@deviant.kiev.zoral.com.ua> In-Reply-To: <1334342264.8672.YahooMailClassic@web180006.mail.gq1.yahoo.com> References: <1334342264.8672.YahooMailClassic@web180006.mail.gq1.yahoo.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--p9yXrYA3ANr0wrJ4
Content-Type: text/plain; charset=koi8-r
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
On Fri, Apr 13, 2012 at 11:37:44AM -0700, Sushanth Rai wrote:
> I've attached the simple program that creates 5 threads. Following is the=
o/p of /proc/<pid>/map when this program is running. Note that I modified=
=20
> sys/fs/procfs/procfs_map.c to print whether a region is wired. As you can=
see from this o/p, none of stack areas get wired.=20
>=20
> 0x400000 0x401000 1 0 0xffffff002d943bd0 r-x 1 0 0x1000 COW NC wired vnod=
e /var/tmp/thread1
> 0x500000 0x501000 1 0 0xffffff002dd13e58 rw- 2 0 0x3100 NCOW NNC wired de=
fault -
> 0x501000 0x600000 255 0 0xffffff002dd13e58 rwx 2 0 0x3100 NCOW NNC wired =
default -
> 0x800500000 0x800526000 38 0 0xffffff0025574000 r-x 192 46 0x1004 COW NC =
wired vnode /libexec/ld-elf.so.1
> 0x800526000 0x800537000 17 0 0xffffff002d9f81b0 rw- 1 0 0x3100 NCOW NNC w=
ired default -
> 0x800626000 0x80062d000 7 0 0xffffff002dd13bd0 rw- 1 0 0x3100 COW NNC wir=
ed vnode /libexec/ld-elf.so.1
> 0x80062d000 0x800633000 6 0 0xffffff002dd145e8 rw- 1 0 0x3100 NCOW NNC wi=
red default -
> 0x800633000 0x800645000 18 0 0xffffff00256d71b0 r-x 63 42 0x4 COW NC wire=
d vnode /lib/libthr.so.3
> 0x800645000 0x800646000 1 0 0xffffff002d975510 r-x 1 0 0x3100 COW NNC wir=
ed vnode /lib/libthr.so.3
> 0x800646000 0x800746000 0 0 0xffffff002dc5cca8 --- 4 0 0x3100 NCOW NNC no=
t-wired default -
> 0x800746000 0x80074a000 4 0 0xffffff002572a288 rw- 1 0 0x3100 COW NNC wir=
ed vnode /lib/libthr.so.3
> 0x80074a000 0x80074c000 2 0 0xffffff002dc5cca8 rw- 4 0 0x3100 NCOW NNC wi=
red default -
> 0x80074c000 0x80083e000 242 0 0xffffff001cd226c0 r-x 238 92 0x1004 COW NC=
wired vnode /lib/libc.so.7
> 0x80083e000 0x80083f000 1 0 0xffffff002dd12000 r-x 1 0 0x3100 COW NNC wir=
ed vnode /lib/libc.so.7
> 0x80083f000 0x80093e000 0 0 0xffffff002dc5cca8 --- 4 0 0x3100 NCOW NNC no=
t-wired default -
> 0x80093e000 0x80095d000 31 0 0xffffff002dddc360 rw- 1 0 0x3100 COW NNC wi=
red vnode /lib/libc.so.7
> 0x80095d000 0x800974000 23 0 0xffffff002dc5cca8 rw- 4 0 0x3100 NCOW NNC w=
ired default -
> 0x800a00000 0x800b00000 256 0 0xffffff002dbd1798 rw- 1 0 0x3100 NCOW NNC =
wired default -
> 0x800b00000 0x800c00000 256 0 0xffffff002dd14948 rw- 1 0 0x3100 NCOW NNC =
wired default -
> 0x7fffff3db000 0x7fffff3fb000 1 0 0xffffff002dbb4360 rw- 1 0 0x3100 NCOW =
NNC not-wired default -
> 0x7fffff5dc000 0x7fffff5fc000 1 0 0xffffff002dc66af8 rw- 1 0 0x3100 NCOW =
NNC not-wired default -
> 0x7fffff7dd000 0x7fffff7fd000 1 0 0xffffff002dbea438 rw- 1 0 0x3100 NCOW =
NNC not-wired default -
> 0x7fffff9de000 0x7fffff9fe000 1 0 0xffffff002dd7fd80 rw- 1 0 0x3100 NCOW =
NNC not-wired default -
> 0x7fffffbdf000 0x7fffffbff000 1 0 0xffffff002dbe9438 rw- 1 0 0x3100 NCOW =
NNC not-wired default -
> 0x7fffffbff000 0x7fffffc00000 0 0 0 --- 0 0 0x0 NCOW NNC not-wired none -
> 0x7ffffffe0000 0x800000000000 32 0 0xffffff002dd125e8 rwx 1 0 0x3100 NCOW=
NNC wired default -
>=20
> --- On Fri, 4/13/12, Konstantin Belousov <kostikbel@gmail.com> wrote:
>=20
> > From: Konstantin Belousov <kostikbel@gmail.com>
> > Subject: Re: mlockall() on freebsd 7.2 + amd64 returns EAGAIN
> > To: "Sushanth Rai" <sushanth_rai@yahoo.com>
> > Cc: freebsd-hackers@freebsd.org
> > Date: Friday, April 13, 2012, 1:11 AM
> > On Thu, Apr 12, 2012 at 08:10:26PM
> > -0700, Sushanth Rai wrote:
> > > >=20
> > > > Then it should be fixed in r190885.
> > > >=20
> > >=20
> > > Thanks. That works like a charm.=20
> > >=20
> > > mlockall() mostly works now. There is still a, issue in
> > wiring the stacks of multithreaded program when the program
> > uses default stack allocation scheme. Thread library
> > allocates stack for each thread by calling mmap() and
> > sending address and size to be mapped. The kernel adjusts
> > the start address to sgrowsz in=9A vm_map_stack() and
> > maps at the adjusted address. But the subsequent wiring is
> > done using the original address, which fails.=20
Oh, I see. The problem is the VM_MAP_WIRE_NOHOLES flag. Since we
map only the initial stack fragment even for the MCL_WIREFUTURE maps,
there is a hole in the stack region.
In fact, for MCL_WIREFUTURE, we probably should map the whole
stack at once, prefaulting all pages.
Below are two patches. The change for vm_mmap.c would fix your immediate
problem by allowing holes in wired region.
The change for vm_map.c prefaults the whole stack instead of the
initial fragment. The single-threaded programs still get a fault
on stack growth.
diff --git a/sys/vm/vm_map.c b/sys/vm/vm_map.c
index 6198629..2fd18d1 100644
--- a/sys/vm/vm_map.c
+++ b/sys/vm/vm_map.c
@@ -3259,7 +3259,10 @@ vm_map_stack(vm_map_t map, vm_offset_t addrbos, vm_s=
ize_t max_ssize,
addrbos + max_ssize < addrbos)
return (KERN_NO_SPACE);
=20
- init_ssize =3D (max_ssize < sgrowsiz) ? max_ssize : sgrowsiz;
+ if (map->flags & MAP_WIREFUTURE)
+ init_ssize =3D max_ssize;
+ else
+ init_ssize =3D (max_ssize < sgrowsiz) ? max_ssize : sgrowsiz;
=20
PROC_LOCK(curthread->td_proc);
vmemlim =3D lim_cur(curthread->td_proc, RLIMIT_VMEM);
diff --git a/sys/vm/vm_mmap.c b/sys/vm/vm_mmap.c
index 2588c85..3fccd9e 100644
--- a/sys/vm/vm_mmap.c
+++ b/sys/vm/vm_mmap.c
@@ -1561,9 +1561,11 @@ vm_mmap(vm_map_t map, vm_offset_t *addr, vm_size_t s=
ize, vm_prot_t prot,
* If the process has requested that all future mappings
* be wired, then heed this.
*/
- if (map->flags & MAP_WIREFUTURE)
+ if (map->flags & MAP_WIREFUTURE) {
vm_map_wire(map, *addr, *addr + size,
- VM_MAP_WIRE_USER | VM_MAP_WIRE_NOHOLES);
+ VM_MAP_WIRE_USER | ((flags & MAP_STACK) ?
+ VM_MAP_WIRE_HOLESOK : VM_MAP_WIRE_NOHOLES));
+ }
} else {
/*
* If this mapping was accounted for in the vnode's
--p9yXrYA3ANr0wrJ4
Content-Type: application/pgp-signature
Content-Disposition: inline
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.12 (FreeBSD)
iEYEARECAAYFAk+Inn8ACgkQC3+MBN1Mb4iolQCcCiKLilWM5Ok8fuf5rSOAyQlu
h/IAoOL9B0Pn/HYRABYAN0uKsOXUy+bP
=QwC2
-----END PGP SIGNATURE-----
--p9yXrYA3ANr0wrJ4--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120413214536.GL2358>