Date: Sat, 21 Apr 2012 13:22:31 -0500 From: Alan Cox <alc@rice.edu> To: Konstantin Belousov <kostikbel@gmail.com> Cc: alc@freebsd.org, freebsd-hackers@freebsd.org, Sushanth Rai <sushanth_rai@yahoo.com> Subject: Re: mlockall() on freebsd 7.2 + amd64 returns EAGAIN Message-ID: <4F92FAE7.3090702@rice.edu> In-Reply-To: <20120413214536.GL2358@deviant.kiev.zoral.com.ua> References: <1334342264.8672.YahooMailClassic@web180006.mail.gq1.yahoo.com> <20120413214536.GL2358@deviant.kiev.zoral.com.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On 04/13/2012 16:45, Konstantin Belousov wrote: > On Fri, Apr 13, 2012 at 11:37:44AM -0700, Sushanth Rai wrote: >> I've attached the simple program that creates 5 threads. Following is the o/p of /proc/<pid>/map when this program is running. Note that I modified >> sys/fs/procfs/procfs_map.c to print whether a region is wired. As you can see from this o/p, none of stack areas get wired. >> >> 0x400000 0x401000 1 0 0xffffff002d943bd0 r-x 1 0 0x1000 COW NC wired vnode /var/tmp/thread1 >> 0x500000 0x501000 1 0 0xffffff002dd13e58 rw- 2 0 0x3100 NCOW NNC wired default - >> 0x501000 0x600000 255 0 0xffffff002dd13e58 rwx 2 0 0x3100 NCOW NNC wired default - >> 0x800500000 0x800526000 38 0 0xffffff0025574000 r-x 192 46 0x1004 COW NC wired vnode /libexec/ld-elf.so.1 >> 0x800526000 0x800537000 17 0 0xffffff002d9f81b0 rw- 1 0 0x3100 NCOW NNC wired default - >> 0x800626000 0x80062d000 7 0 0xffffff002dd13bd0 rw- 1 0 0x3100 COW NNC wired vnode /libexec/ld-elf.so.1 >> 0x80062d000 0x800633000 6 0 0xffffff002dd145e8 rw- 1 0 0x3100 NCOW NNC wired default - >> 0x800633000 0x800645000 18 0 0xffffff00256d71b0 r-x 63 42 0x4 COW NC wired vnode /lib/libthr.so.3 >> 0x800645000 0x800646000 1 0 0xffffff002d975510 r-x 1 0 0x3100 COW NNC wired vnode /lib/libthr.so.3 >> 0x800646000 0x800746000 0 0 0xffffff002dc5cca8 --- 4 0 0x3100 NCOW NNC not-wired default - >> 0x800746000 0x80074a000 4 0 0xffffff002572a288 rw- 1 0 0x3100 COW NNC wired vnode /lib/libthr.so.3 >> 0x80074a000 0x80074c000 2 0 0xffffff002dc5cca8 rw- 4 0 0x3100 NCOW NNC wired default - >> 0x80074c000 0x80083e000 242 0 0xffffff001cd226c0 r-x 238 92 0x1004 COW NC wired vnode /lib/libc.so.7 >> 0x80083e000 0x80083f000 1 0 0xffffff002dd12000 r-x 1 0 0x3100 COW NNC wired vnode /lib/libc.so.7 >> 0x80083f000 0x80093e000 0 0 0xffffff002dc5cca8 --- 4 0 0x3100 NCOW NNC not-wired default - >> 0x80093e000 0x80095d000 31 0 0xffffff002dddc360 rw- 1 0 0x3100 COW NNC wired vnode /lib/libc.so.7 >> 0x80095d000 0x800974000 23 0 0xffffff002dc5cca8 rw- 4 0 0x3100 NCOW NNC wired default - >> 0x800a00000 0x800b00000 256 0 0xffffff002dbd1798 rw- 1 0 0x3100 NCOW NNC wired default - >> 0x800b00000 0x800c00000 256 0 0xffffff002dd14948 rw- 1 0 0x3100 NCOW NNC wired default - >> 0x7fffff3db000 0x7fffff3fb000 1 0 0xffffff002dbb4360 rw- 1 0 0x3100 NCOW NNC not-wired default - >> 0x7fffff5dc000 0x7fffff5fc000 1 0 0xffffff002dc66af8 rw- 1 0 0x3100 NCOW NNC not-wired default - >> 0x7fffff7dd000 0x7fffff7fd000 1 0 0xffffff002dbea438 rw- 1 0 0x3100 NCOW NNC not-wired default - >> 0x7fffff9de000 0x7fffff9fe000 1 0 0xffffff002dd7fd80 rw- 1 0 0x3100 NCOW NNC not-wired default - >> 0x7fffffbdf000 0x7fffffbff000 1 0 0xffffff002dbe9438 rw- 1 0 0x3100 NCOW NNC not-wired default - >> 0x7fffffbff000 0x7fffffc00000 0 0 0 --- 0 0 0x0 NCOW NNC not-wired none - >> 0x7ffffffe0000 0x800000000000 32 0 0xffffff002dd125e8 rwx 1 0 0x3100 NCOW NNC wired default - >> >> --- On Fri, 4/13/12, Konstantin Belousov<kostikbel@gmail.com> wrote: >> >>> From: Konstantin Belousov<kostikbel@gmail.com> >>> Subject: Re: mlockall() on freebsd 7.2 + amd64 returns EAGAIN >>> To: "Sushanth Rai"<sushanth_rai@yahoo.com> >>> Cc: freebsd-hackers@freebsd.org >>> Date: Friday, April 13, 2012, 1:11 AM >>> On Thu, Apr 12, 2012 at 08:10:26PM >>> -0700, Sushanth Rai wrote: >>>>> Then it should be fixed in r190885. >>>>> >>>> Thanks. That works like a charm. >>>> >>>> mlockall() mostly works now. There is still a, issue in >>> wiring the stacks of multithreaded program when the program >>> uses default stack allocation scheme. Thread library >>> allocates stack for each thread by calling mmap() and >>> sending address and size to be mapped. The kernel adjusts >>> the start address to sgrowsz in vm_map_stack() and >>> maps at the adjusted address. But the subsequent wiring is >>> done using the original address, which fails. > Oh, I see. The problem is the VM_MAP_WIRE_NOHOLES flag. Since we > map only the initial stack fragment even for the MCL_WIREFUTURE maps, > there is a hole in the stack region. > > In fact, for MCL_WIREFUTURE, we probably should map the whole > stack at once, prefaulting all pages. > > Below are two patches. The change for vm_mmap.c would fix your immediate > problem by allowing holes in wired region. > > The change for vm_map.c prefaults the whole stack instead of the > initial fragment. The single-threaded programs still get a fault > on stack growth. The vm_mmap.c change looks ok to me. Please commit it. I haven't yet had a chance to think about the other change. Alan > diff --git a/sys/vm/vm_map.c b/sys/vm/vm_map.c > index 6198629..2fd18d1 100644 > --- a/sys/vm/vm_map.c > +++ b/sys/vm/vm_map.c > @@ -3259,7 +3259,10 @@ vm_map_stack(vm_map_t map, vm_offset_t addrbos, vm_size_t max_ssize, > addrbos + max_ssize< addrbos) > return (KERN_NO_SPACE); > > - init_ssize = (max_ssize< sgrowsiz) ? max_ssize : sgrowsiz; > + if (map->flags& MAP_WIREFUTURE) > + init_ssize = max_ssize; > + else > + init_ssize = (max_ssize< sgrowsiz) ? max_ssize : sgrowsiz; > > PROC_LOCK(curthread->td_proc); > vmemlim = lim_cur(curthread->td_proc, RLIMIT_VMEM); > diff --git a/sys/vm/vm_mmap.c b/sys/vm/vm_mmap.c > index 2588c85..3fccd9e 100644 > --- a/sys/vm/vm_mmap.c > +++ b/sys/vm/vm_mmap.c > @@ -1561,9 +1561,11 @@ vm_mmap(vm_map_t map, vm_offset_t *addr, vm_size_t size, vm_prot_t prot, > * If the process has requested that all future mappings > * be wired, then heed this. > */ > - if (map->flags& MAP_WIREFUTURE) > + if (map->flags& MAP_WIREFUTURE) { > vm_map_wire(map, *addr, *addr + size, > - VM_MAP_WIRE_USER | VM_MAP_WIRE_NOHOLES); > + VM_MAP_WIRE_USER | ((flags& MAP_STACK) ? > + VM_MAP_WIRE_HOLESOK : VM_MAP_WIRE_NOHOLES)); > + } > } else { > /* > * If this mapping was accounted for in the vnode's
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F92FAE7.3090702>