Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 29 Sep 2005 00:31:02 -0700 (PDT)
From:      Don Lewis <truckman@FreeBSD.org>
To:        phk@phk.freebsd.dk
Cc:        current@FreeBSD.org
Subject:   Re: stress test deadlock involving vm and/or geom 
Message-ID:  <200509290731.j8T7V2OR007019@gw.catspoiler.org>
In-Reply-To: <6785.1127975825@critter.freebsd.dk>

next in thread | previous in thread | raw e-mail | index | archive | help
On 29 Sep, Poul-Henning Kamp wrote:
> In message <200509290214.j8T2EogR006609@gw.catspoiler.org>, Don Lewis writes:
> 
>>Also, g_io_check() can only return ENOMEM if that is the provider error
>>status, but the provider seems to be happy:
> 
> ENOMEM can also be returned by the code in the GEOM class.

Ok, that was obscure enough.

It looks like the
	uma_zalloc(biozone, M_NOWAIT | M_ZERO)
call in g_clone_bio() is the likely culprit.  Lots of allocation
failures for this zone:

(kgdb) print *biozone
$2 = {uz_name = 0xc086938c "g_bio", uz_lock = 0xc10295a8, uz_keg = 0xc10295a0, 
  uz_link = {le_next = 0x0, le_prev = 0xc10295d8}, uz_full_bucket = {
    lh_first = 0x0}, uz_free_bucket = {lh_first = 0x0}, uz_ctor = 0, 
  uz_dtor = 0, uz_init = 0, uz_fini = 0, uz_allocs = 22457718, 
  uz_frees = 22456210, uz_fails = 6603729, uz_fills = 1, uz_count = 128, 
  uz_cpu = {{uc_freebucket = 0x0, uc_allocbucket = 0x0, uc_allocs = 0, 
      uc_frees = 0}}}

Any easy way to figure out why the memory allocation is repeatedly
failing?

Here's the output of vmstat -z:

ITEM            SIZE     LIMIT     USED    FREE  REQUESTS

UMA Kegs:        140,        0,      62,     10,       62
UMA Zones:       120,        0,      62,     28,       62
UMA Slabs:        64,        0,    2169,    132,    51890
UMA RCntSlabs:   104,        0,     104,    266,    37549
UMA Hash:        128,        0,       1,     29,        3
16 Bucket:        76,        0,      12,     38,       45
32 Bucket:       140,        0,      30,     26,       69
64 Bucket:       268,        0,      14,     14,       72
128 Bucket:      524,        0,    2409,    244,    53217
VM OBJECT:       132,        0,    5249,  17255,  2323529
MAP:             192,        0,       7,     33,        7
KMAP ENTRY:       68,    56616,     867,   1317,   797654
MAP ENTRY:        68,        0,    1881,   3271,   615413
PV ENTRY:         24,  1494950,  186575,  68335, 77309488
DP fakepg:        72,        0,       0,      0,        0
mt_zone:          64,        0,     194,    101,      194
16:               16,        0,    2357,    282,  2618883
32:               32,        0,   36199,  19962,  9905089
64:               64,        0,    5020,   1883,  3448783
128:             128,        0,   29552,   4078,  5969903
256:             256,        0,    2556,    309,  6529179
512:             512,        0,      71,     33,     5456
1024:           1024,        0,     139,     65,    67843
2048:           2048,        0,    3123,     57,   775663
4096:           4096,        0,     216,     97,    22551
Files:            72,        0,     151,    326,  2846128
PROC:            524,        0,     143,    130,    14242
THREAD:          372,        0,     273,     77,     5308
KSEGRP:           88,        0,     273,     47,      707
UPCALL:           44,        0,       0,    156,      866
VMSPACE:         300,        0,      92,    142,    14191
mbuf_packet:     256,        0,      66,     51, 82560936
mbuf:            256,        0,       3,    540, 231234746
mbuf_cluster:   2048,    25600,     117,     91, 12760461
ACL UMA zone:    388,        0,       0,      0,        0
g_bio:           132,        0,    1508,      0, 22457718
ata_request:     200,        0,       0,      0,        0
ata_composite:   192,        0,       0,      0,        0
VNODE:           272,        0,   29373,   6523,  4183368
VNODEPOLL:        76,        0,       0,      0,        0
S VFS Cache:      68,        0,   11060,   2828,   844011
L VFS Cache:     291,        0,       0,      0,        0
NAMEI:          1024,        0,      24,     52,  7276418
NFSMOUNT:        480,        0,       2,     14,        2
NFSNODE:         424,        0,      25,    902,     7316
DIRHASH:        1024,        0,     491,     97,    56851
PIPE:            408,        0,       4,     59,     4381
KNOTE:            68,        0,       0,    112,       80
socket:          356,    25608,      66,     55,   807098
unpcb:           140,    25620,      17,     39,      217
udpcb:           180,    25608,      25,     41,   805728
inpcb:           180,    25608,      24,     64,     1152
tcpcb:           460,    25600,      24,     32,     1152
tcptw:            48,     5148,       0,    156,      237
syncache:        100,    15366,       0,     78,      348
hostcache:        76,    15400,       1,     99,        2
tcpreass:         20,     1690,       0,      0,        0
sackhole:         20,        0,       0,      0,        0
ripcb:           180,    25608,       0,      0,        0
rtentry:         132,        0,      14,     44,       14
SWAPMETA:        276,   121576,    2620,   5542,   354141
FFS inode:       132,        0,   29314,   4123,  4175811
FFS1 dinode:     128,        0,     183,   1497,    13155
FFS2 dinode:     256,        0,   29124,   3441,  4162649


It looks like the problem might be caused by having too much I/O in
progress.

(kgdb) print runningbufspace
$3 = 1585152
(kgdb) print lorunningspace
$4 = 524288
print hirunningspace
$5 = 1048576

so runningbufspace >> hirunningspace

The bufdaemon and the syncer both ignore the runningbufspace limit.
Maybe they should obey the limit, but each be guaranteed a minimum
quota.




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200509290731.j8T7V2OR007019>