Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 31 Aug 2002 14:06:19 -0700 (PDT)
From:      Matthew Dillon <dillon@apollo.backplane.com>
To:        Arnvid Karstad <arnvid@karstad.org>
Cc:        bmah@FreeBSD.ORG, freebsd-stable@FreeBSD.ORG
Subject:   Re: Problems with FreeBSD - causing zalloc to return 0 ?!
Message-ID:  <200208312106.g7VL6JEU002376@apollo.backplane.com>
References:  <20020830094151.41DC.ARNVID@karstad.org> <200208301652.g7UGq3Ud059184@intruder.bmah.org> <20020830190849.8B8A.ARNVID@karstad.org>

next in thread | previous in thread | raw e-mail | index | archive | help

:> almost identical.
:
:Just for an intresting side note....
:
:With option INVARIANTS we get no problems and vmstat's shows new highs'
:
:root@irc:/usr# vmstat -z | grep VNODE
:VNODE:           192,        0, 170782,     90,   170782
:
:With out.. it dies horribly when the number reaches around 44000-45000.
:
:Arnvid

    Ok, I've examined the kernel core dump.  I'm still not sure why
    INVARIANTS made any difference, but the kernel is definitely running
    out of KVM and it looks like the main culprit is the number of 
    mbufs and mbuf clusters configured.  It looks like they were 
    manually configured up.

    (kgdb) printf "%08x\n", kernel_vm_end
    ffc00000				<<<< indicates kernel ran out of KVM
    (kgdb) print nmbclusters
    $9 = 129536				<<<< this is huge.  autoconfigure
					does not do this, you must be 
					overriding it.
    (kgdb) print nmbufs
    $10 = 518144
    (kgdb) print nmbufs * 256 + nmbclusters * 2048
    $11 = 397934592			<<<< too much.  397MB reserved!

    (kgdb) print clean_map->header.end - clean_map->header.start
    $21 = 186744832			<<<< (mainly buffer cache)
    (kgdb) print mb_map->header.end - mb_map->header.start
    $22 = 397934592			<<<< KVM reservation for MBUFs
    (kgdb) print maxswzone
    $4 = 73400320			<<<< maxswzone (used to manage
					     swap)
    (kgdb) printf "%d\n", zone_kmem_kvaspace
    214933504				<<<< zones eating 214MB

define zlist
    set $zp = zlist
    while ($zp != 0)
        set $initmem = $zp->zmax * $zp->zsize
        set $addmem = $zp->ztotal * $zp->zsize
        printf "%p\t%-15s\t%8d init + %8d dyn = %8d\n", $zp, $zp->zname, $initmem, $addmem, $initmem + $addmem
        set $zp = $zp->znext
    end
    set $initmem = zone_kmem_kvaspace
    set $addmem = (zone_kmem_pages + zone_kern_pages ) * 0x1000
    printf "TOTAL ZONE KMEM RESERVED: %d init + %d dynamic = %d\n", $initmem, $addmem, $initmem + $addmem
end

(kgdb) zlist
0xda1c4e80      PIPE                   0 init +    16320 dyn =    16320
0xda15e780      SWAPMETA        51381120 init +        0 dyn = 51381120
0xda0d3100      ripcb           24870912 init +     4032 dyn = 24874944
0xda0d3180      syncache         2457440 init +     4000 dyn =  2461440
0xda0d3200      tcpcb           70467584 init +     8160 dyn = 70475744
0xda0d3280      udpcb           24870912 init +     8064 dyn = 24878976
0xda0d3300      unpcb                  0 init +     8000 dyn =     8000
0xda0d3380      socket          24870912 init +     8064 dyn = 24878976
0xda0d3400      DIRHASH                0 init +   729088 dyn =   729088
0xda0d3480      KNOTE                  0 init +     8192 dyn =     8192
0xda011e80      VNODE                  0 init +  8120448 dyn =  8120448
0xda011f00      NAMEI                  0 init +    16384 dyn =    16384
0xc2436900      VMSPACE                0 init +    12288 dyn =    12288
0xc2436a00      PROC                   0 init +    20384 dyn =    20384
0xc02b7a40      DP fakepg              0 init +        0 dyn =        0
0xc02c8700      PV ENTRY        21327600 init +  9174200 dyn = 30501800
0xc02b7be0      MAP ENTRY              0 init +    22464 dyn =    22464
0xc02b7b80      KMAP ENTRY       3859728 init +    10224 dyn =  3869952
0xc02b7c40      MAP                    0 init +     1080 dyn =     1080
0xc02bb320      VM OBJECT              0 init +  4080768 dyn =  4080768
TOTAL ZONE KMEM RESERVED: 214933504 init + 13156352 dynamic = 228089856

    You've run out of KVM, it looks mainly due to increasing the number
    of mbufs in the system beyond the autoconfig and you've also
    massively increased maxsockets, so much so that the zone allocator
    is reserving over 110 MB just to hold tcpcb and udpcb allocations.
    The tcpcb and udpcb zmemory reservations are huge!

    There are a couple of things you can do.  I recommend setting the 
    following kernel boot variables in /boot/loader.conf:

	kern.maxswzone="32m"
	kern.ipc.maxsockets="30000"

	(how many active sockets do you actually normally have?  Either you
	set your maxsockets to 129536 or the system autoconfig did it)

    In your kernel config:

	NSWAPDEV="2"

    Additionally I strongly recommend reducing the number of mbufs in
    the system.  You almost certainly have an NMBCLUSTERS thing in your
    kernel config or a kern.ipc.nmbclusters in your /boot/loader.conf
    to get a number so high (your is set to 129536).  I recommend:

	kern.ipc.nmbclusters="70000"

    If you are running out of buffer space I recommend reducing 
    net.inet.tcp.recvspace and net.inet.tcp.sendspace in /etc/sysctl.conf.
    Currently you have them set at:

    (kgdb) print tcp_recvspace
    $3 = 57344
    (kgdb) print tcp_sendspace
    $4 = 32768

    Try reducing sendspace to 24576 and tcp_recvspace to 32768.

    --

    I think that for large-memory machines I am still reserving too much
    KVM space for swap meta structures.  I am going to cut that down even
    more for this release.  It's obviously been party responsible for a lot
    of the KVM exhaustion problems people have reported on large-memory
    machines.

    However, it looks like the primary issue here is that you made
    the resource settings so high there was no room left for anything else
    in KVM.

					-Matt
					Matthew Dillon 
					<dillon@backplane.com>

To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-stable" in the body of the message




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200208312106.g7VL6JEU002376>