From owner-freebsd-arm@FreeBSD.ORG Fri Jun 21 00:03:29 2013 Return-Path: Delivered-To: freebsd-arm@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C16BE26C for ; Fri, 21 Jun 2013 00:03:29 +0000 (UTC) (envelope-from jroberson@jroberson.net) Received: from mail-pa0-x22c.google.com (mail-pa0-x22c.google.com [IPv6:2607:f8b0:400e:c03::22c]) by mx1.freebsd.org (Postfix) with ESMTP id 9CC3B1DB4 for ; Fri, 21 Jun 2013 00:03:29 +0000 (UTC) Received: by mail-pa0-f44.google.com with SMTP id lj1so6951912pab.17 for ; Thu, 20 Jun 2013 17:03:29 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=date:from:x-x-sender:to:cc:subject:in-reply-to:message-id :references:user-agent:mime-version:content-type:x-gm-message-state; bh=uHRFq4tPH1oiN9uQQeMLwktKB0uusMHn6ky66SCDCi4=; b=Q0QBeNEnsodEMAELOCjc7GAWG5ecLcjPpcyJwuPl/QLUpFa2Ndv+QVsOAb8elAFUZc YJqR5rxaSSp64D+q3iqgRd01cpbfL73wVeaXKKXALQQlGnZftW60YVvkg+aBW75o/0Mn 6DyoonTMVpXdAmsUtet+cNiCxyN4poJYMmJfhK8e7mqXXEbZ42QPjHfKRtDpNLzGV1JX cY9pDJu/cRGUA+C3YRD82IGcpFN05tkrkxzDO4iEJwmto7yXwQ+Gvxn7wpTLytQr767F lc0P6AG0NB7xVcvvh2vRb+ZR+AUu5nO+Iy1Z7t9bCedri4AAdrytQ/ocngWXzuhEVbDx g2EQ== X-Received: by 10.68.228.201 with SMTP id sk9mr10077179pbc.4.1371773009310; Thu, 20 Jun 2013 17:03:29 -0700 (PDT) Received: from rrcs-66-91-135-210.west.biz.rr.com (rrcs-66-91-135-210.west.biz.rr.com. [66.91.135.210]) by mx.google.com with ESMTPSA id xe9sm2161347pbc.21.2013.06.20.17.03.27 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Thu, 20 Jun 2013 17:03:28 -0700 (PDT) Date: Thu, 20 Jun 2013 13:56:10 -1000 (HST) From: Jeff Roberson X-X-Sender: jroberson@desktop To: Zbyszek Bodek Subject: Re: Kernel build fails on ARM: Cannot fork: Cannot allocate memory In-Reply-To: Message-ID: References: <51C1F53B.2080502@semihalf.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Gm-Message-State: ALoCoQkKdxYcYlEoRahqjMA24kpNYDuLI+Wx5IRfMZ9YNgOSYV9KJ4r7msc607brO0DfKEoCj6R7 Cc: freebsd-arm@FreeBSD.org, freebsd-current@freebsd.org, jeff@FreeBSD.org X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to the StrongARM Processor List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Jun 2013 00:03:29 -0000 On Thu, 20 Jun 2013, Jeff Roberson wrote: > On Wed, 19 Jun 2013, Zbyszek Bodek wrote: > >> Hello, >> >> I've been trying to compile the kernel on my ARMv7 platform using the >> sources from the current FreeBSD HEAD. >> >> make buildkernel <.....> -j5 >> >> 1/2 builds fails in the way described below: >> -------------------------------------------------------------------------- >> ing-include-dirs -fdiagnostics-show-option -nostdinc -I. >> -I/root/src/freebsd-arm-superpages/sys >> -I/root/src/freebsd-arm-superpages/sys/contrib/altq >> -I/root/src/freebsd-arm-superpages/sys/contrib/libfdt -D_KERNEL >> -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common >> -finline-limit=8000 --param inline-unit-growth=100 --param >> large-function-growth=1000 -mno-thumb-interwork -ffreestanding -Werror >> /root/src/freebsd-arm-superpages/sys/ufs/ffs/ffs_snapshot.c >> Cannot fork: Cannot allocate memory >> *** [ffs_snapshot.o] Error code 2 >> 1 error >> *** [buildkernel] Error code 2 >> 1 error >> *** [buildkernel] Error code 2 >> 1 error >> 5487.888u 481.569s 7:35.65 1310.0% 1443+167k 1741+5388io 221pf+0w >> -------------------------------------------------------------------------- >> >> The warning from std err is: >> -------------------------------------------------------------------------- >> vm_thread_new: kstack allocation failed >> vm_thread_new: kstack allocation failed >> -------------------------------------------------------------------------- >> >> I was trying to find out which commit is causing this (because I was >> previously working on some older revision) and using bisect I got to: >> >> -------------------------------------------------------------------------- >> Author: jeff >> Date: Tue Jun 18 04:50:20 2013 +0000 >> >> Refine UMA bucket allocation to reduce space consumption and improve >> performance. >> >> - Always free to the alloc bucket if there is space. This gives LIFO >> allocation order to improve hot-cache performance. This also allows >> for zones with a single bucket per-cpu rather than a pair if the >> entire >> working set fits in one bucket. >> - Enable per-cpu caches of buckets. To prevent recursive bucket >> allocation one bucket zone still has per-cpu caches disabled. >> - Pick the initial bucket size based on a table driven maximum size >> per-bucket rather than the number of items per-page. This gives >> more sane initial sizes. >> - Only grow the bucket size when we face contention on the zone >> lock, this >> causes bucket sizes to grow more slowly. >> - Adjust the number of items per-bucket to account for the header >> space. >> This packs the buckets more efficiently per-page while making them >> not quite powers of two. >> - Eliminate the per-zone free bucket list. Always return buckets back >> to the bucket zone. This ensures that as zones grow into larger >> bucket sizes they eventually discard the smaller sizes. It persists >> fewer buckets in the system. The locking is slightly trickier. >> - Only switch buckets in zalloc, not zfree, this eliminates >> pathological >> cases where we ping-pong between two buckets. >> - Ensure that the thread that fills a new bucket gets to allocate from >> it to give a better upper bound on allocation time. >> >> Sponsored by: EMC / Isilon Storage Division >> -------------------------------------------------------------------------- >> >> I checked this several times and this commits seems to be causing this. > > Can you tell me how many cores and how much memory you have? And paste the > output of vmstat -z when you see this error. > > You can try changing bucket_select() at line 339 in uma_core.c to read: > > static int > bucket_select(int size) > { > return (MAX(PAGE_SIZE / size, 1)); > } > > This will approximate the old bucket sizing behavior. Just to add some more information; On my machine with 16GB of ram the handful of recent UMA commits save about 20MB of kmem on boot. There are 30% fewer buckets allocated. And all of the malloc zones have similar amounts of cached space. Actually the page size malloc bucket is taking up much less space. I don't know if the problem is unique to arm but I have tested x86 limited to 512MB of ram without trouble. I will need the stats I mentioned before to understand what has happened. Jeff > > Thanks, > Jeff > >> >> Does anyone observe similar behavior or have a solution? >> >> Best regards >> Zbyszek Bodek >> >