Skip site navigation (1)Skip section navigation (2)
Date:      Sun, 23 Jun 2013 09:57:06 +0300
From:      Konstantin Belousov <kostikbel@gmail.com>
To:        Zbyszek Bodek <zbb@semihalf.com>
Cc:        freebsd-arm@FreeBSD.org, freebsd-current@freebsd.org, Jeff Roberson <jroberson@jroberson.net>
Subject:   Re: Kernel build fails on ARM: Cannot fork: Cannot allocate memory
Message-ID:  <20130623065706.GV91021@kib.kiev.ua>
In-Reply-To: <alpine.BSF.2.00.1306221636360.43796@desktop>
References:  <51C1F53B.2080502@semihalf.com> <alpine.BSF.2.00.1306200755210.2005@desktop> <alpine.BSF.2.00.1306201350330.2005@desktop> <51C4A067.7010203@semihalf.com> <alpine.BSF.2.00.1306221636360.43796@desktop>

next in thread | previous in thread | raw e-mail | index | archive | help

--WZjgKYXV2jnEACro
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Sat, Jun 22, 2013 at 04:37:24PM -1000, Jeff Roberson wrote:
> On Fri, 21 Jun 2013, Zbyszek Bodek wrote:
>=20
> > On 21.06.2013 01:56, Jeff Roberson wrote:
> >> On Thu, 20 Jun 2013, Jeff Roberson wrote:
> >>
> >>> On Wed, 19 Jun 2013, Zbyszek Bodek wrote:
> >>>
> >>>> Hello,
> >>>>
> >>>> I've been trying to compile the kernel on my ARMv7 platform using the
> >>>> sources from the current FreeBSD HEAD.
> >>>>
> >>>> make buildkernel <.....> -j5
> >>>>
> >>>> 1/2 builds fails in the way described below:
> >>>> --------------------------------------------------------------------=
------
> >>>>
> >>>> ing-include-dirs -fdiagnostics-show-option   -nostdinc  -I.
> >>>> -I/root/src/freebsd-arm-superpages/sys
> >>>> -I/root/src/freebsd-arm-superpages/sys/contrib/altq
> >>>> -I/root/src/freebsd-arm-superpages/sys/contrib/libfdt -D_KERNEL
> >>>> -DHAVE_KERNEL_OPTION_HEADERS -include opt_global.h -fno-common
> >>>> -finline-limit=3D8000 --param inline-unit-growth=3D100 --param
> >>>> large-function-growth=3D1000  -mno-thumb-interwork -ffreestanding -W=
error
> >>>> /root/src/freebsd-arm-superpages/sys/ufs/ffs/ffs_snapshot.c
> >>>> Cannot fork: Cannot allocate memory
> >>>> *** [ffs_snapshot.o] Error code 2
> >>>> 1 error
> >>>> *** [buildkernel] Error code 2
> >>>> 1 error
> >>>> *** [buildkernel] Error code 2
> >>>> 1 error
> >>>> 5487.888u 481.569s 7:35.65 1310.0%      1443+167k 1741+5388io 221pf+=
0w
> >>>> --------------------------------------------------------------------=
------
> >>>>
> >>>>
> >>>> The warning from std err is:
> >>>> --------------------------------------------------------------------=
------
> >>>>
> >>>> vm_thread_new: kstack allocation failed
> >>>> vm_thread_new: kstack allocation failed
> >>>> --------------------------------------------------------------------=
------
> >>>>
> >>>>
> >>>> I was trying to find out which commit is causing this (because I was
> >>>> previously working on some older revision) and using bisect I got to:
> >>>>
> >>>> --------------------------------------------------------------------=
------
> >>>>
> >>>> Author: jeff <jeff@FreeBSD.org>
> >>>> Date:   Tue Jun 18 04:50:20 2013 +0000
> >>>>
> >>>>    Refine UMA bucket allocation to reduce space consumption and impr=
ove
> >>>>    performance.
> >>>>
> >>>>     - Always free to the alloc bucket if there is space.  This gives
> >>>> LIFO
> >>>>       allocation order to improve hot-cache performance.  This also
> >>>> allows
> >>>>       for zones with a single bucket per-cpu rather than a pair if t=
he
> >>>> entire
> >>>>       working set fits in one bucket.
> >>>>     - Enable per-cpu caches of buckets.  To prevent recursive bucket
> >>>>       allocation one bucket zone still has per-cpu caches disabled.
> >>>>     - Pick the initial bucket size based on a table driven maximum s=
ize
> >>>>       per-bucket rather than the number of items per-page.  This giv=
es
> >>>>       more sane initial sizes.
> >>>>     - Only grow the bucket size when we face contention on the zone
> >>>> lock, this
> >>>>       causes bucket sizes to grow more slowly.
> >>>>     - Adjust the number of items per-bucket to account for the header
> >>>> space.
> >>>>       This packs the buckets more efficiently per-page while making =
them
> >>>>       not quite powers of two.
> >>>>     - Eliminate the per-zone free bucket list.  Always return buckets
> >>>> back
> >>>>       to the bucket zone.  This ensures that as zones grow into larg=
er
> >>>>       bucket sizes they eventually discard the smaller sizes.  It
> >>>> persists
> >>>>       fewer buckets in the system.  The locking is slightly trickier.
> >>>>     - Only switch buckets in zalloc, not zfree, this eliminates
> >>>> pathological
> >>>>       cases where we ping-pong between two buckets.
> >>>>     - Ensure that the thread that fills a new bucket gets to allocate
> >>>> from
> >>>>       it to give a better upper bound on allocation time.
> >>>>
> >>>>    Sponsored by:    EMC / Isilon Storage Division
> >>>> --------------------------------------------------------------------=
------
> >>>>
> >>>>
> >>>> I checked this several times and this commits seems to be causing th=
is.
> >>>
> >>> Can you tell me how many cores and how much memory you have?  And
> >>> paste the output of vmstat -z when you see this error.
> >>>
> >>> You can try changing bucket_select() at line 339 in uma_core.c to rea=
d:
> >>>
> >>> static int
> >>> bucket_select(int size)
> >>> {
> >>>     return (MAX(PAGE_SIZE / size, 1));
> >>> }
> >>>
> >>> This will approximate the old bucket sizing behavior.
> >>
> >> Just to add some more information;  On my machine with 16GB of ram the
> >> handful of recent UMA commits save about 20MB of kmem on boot.  There
> >> are 30% fewer buckets allocated.  And all of the malloc zones have
> >> similar amounts of cached space.  Actually the page size malloc bucket
> >> is taking up much less space.
> >>
> >> I don't know if the problem is unique to arm but I have tested x86
> >> limited to 512MB of ram without trouble.  I will need the stats I
> >> mentioned before to understand what has happened.
> >>
> >
> > Hello Jeff,
> >
> > Thank you for your interest in my problem.
> >
> > My system is a quad-core ARMv7 with 2048 MB of RAM on board.
> > Please see attachment for the output from vmstat -z when the error occu=
rs.
> >
> > Changing bucket_select() to
> >
> > static int
> > bucket_select(int size)
> > {
> >    return (MAX(PAGE_SIZE / size, 1));
> > }
> >
> > as you suggested helps for the problem. I've performed numerous attempts
> > to build the kernel and none of them failed.
> >
>=20
> I don't really see a lot of wasted memory in the zones.  There is=20
> certainly some.  Can you give me sysctl vm from both a working and=20
> non-working kernel after the build is done or fails?

Try this:
http://people.freebsd.org/~kib/misc/arm_bcache.1.patch

Please _do_ notify me whether it compiled and helped with your problem.

--WZjgKYXV2jnEACro
Content-Type: application/pgp-signature

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.20 (FreeBSD)

iQIcBAEBAgAGBQJRxpxBAAoJEJDCuSvBvK1BaV0P/0dcOgHNcxsL0rv7T2YJXi2Z
9ie3nlIHBQvpA4mfbC2KAGOTX//3CFX+ZHotdRCdypN768h6QuZXfPawohy4Qjem
q0lajL5rtzDjBgR+zmhmtnMGkqsew0o3UENEBzsbRmspDQIEccoxxQgkbzX8BtsF
hi7EPVsoKw4CNkP/4puQmmTmWFS63AxCLrDYeqyNp/FMJBsffq+GLmRT4VhAiREr
5QbWm4OiXgONg8SDkcp6akbsn8Tpybfszaoj6N3RsGq6tE3gWO4UFVLqT8Wmrwhm
gb90Q/N48gFJHCSyde62u4TEfgVozcSa3G2y3Jw9tjSHPPSqICyrwT/azTYn8TgM
ypwaZ1W0NAEchAh8IXcc7YVYMeSoMaG0yY53X3gbWZLHVjdgJdoP3dwpAE65meoj
jWJW0CXAQKSqJFd5DeCOcqIh0dhMUfTIa18NrUIUi7Uwm3MStvGLwjbYFXZVkdQL
tJIHaWTm+XRvdPMpyYji/FBuIQomkcPS9KMlCXypvYjlCNRhj6HzlXw7dDnfYs44
WOGY81btBHdfw2vMwapoRqa+EWmucO29V9+GBlUpSF48mQiLyk0a5XkEXIg+Xo/T
dKGTpgDjjBh6J5mqBpQCLrXf5wfvYRflK2BSNY8vQnRuybbCgb9jK0YzDxcPZ0zu
FMQZnqqgZhk6XUsYBIvy
=ig/L
-----END PGP SIGNATURE-----

--WZjgKYXV2jnEACro--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130623065706.GV91021>