Date: Sat, 26 Jul 2003 21:48:25 +0100 From: Mark Blackman <mark@exonetric.com> To: freebsd-current@freebsd.org Subject: Re: device driver memory leak in 5.1-20030726? Message-ID: <80B1C75A-BFAA-11D7-A23D-00039315D3FE@exonetric.com> In-Reply-To: <95013A6C-BFA4-11D7-A23D-00039315D3FE@exonetric.com>
next in thread | previous in thread | raw e-mail | index | archive | help
A backtrace: (where and where full) for those who can decipher them
uma_core.c seems to have been the trigger.
(kgdb) where
#0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240
#1 0xc032cc4c in boot (howto=3D260) at =20
/usr/src/sys/kern/kern_shutdown.c:372
#2 0xc032cfd7 in panic () at /usr/src/sys/kern/kern_shutdown.c:550
#3 0xc0163e22 in db_panic () at /usr/src/sys/ddb/db_command.c:449
#4 0xc0163da2 in db_command (last_cmdp=3D0xc05c6b40, cmd_table=3D0x0,
aux_cmd_tablep=3D0xc054de7c, aux_cmd_tablep_end=3D0xc054de94)
at /usr/src/sys/ddb/db_command.c:346
#5 0xc0163ec5 in db_command_loop () at =20
/usr/src/sys/ddb/db_command.c:471
#6 0xc0166dc5 in db_trap (type=3D3, code=3D0) at =20
/usr/src/sys/ddb/db_trap.c:73
#7 0xc04b454c in kdb_trap (type=3D3, code=3D0, regs=3D0xcc464aa4)
at /usr/src/sys/i386/i386/db_interface.c:172
#8 0xc04c5e1d in trap (frame=3D
{tf_fs =3D -1047855080, tf_es =3D -867827696, tf_ds =3D 16, =
tf_edi =3D 1, =20
tf_esi =3D -1068224493, tf_ebp =3D -867808528, tf_isp =3D -867808560, =
tf_ebx =20
=3D 0, tf_edx =3D 0, tf_ecx =3D -1067232032, tf_eax =3D 18, tf_trapno =3D =
3, =20
tf_err =3D 0, tf_eip =3D -1068808188, tf_cs =3D 8, tf_eflags =3D 646, =
tf_esp =3D =20
-1068208597, tf_ss =3D -1068312245})
at /usr/src/sys/i386/i386/trap.c:580
#9 0xc04b5f38 in calltrap () at {standard input}:102
#10 0xc032cf65 in panic (
fmt=3D0xc0543013 "kmem_malloc(%ld): kmem_map too small: %ld total =20=
allocated")
at /usr/src/sys/kern/kern_shutdown.c:534
#11 0xc047dee0 in kmem_malloc (map=3D0xc082f0b0, size=3D4096, flags=3D2)
at /usr/src/sys/vm/vm_kern.c:339
#12 0xc048ee87 in page_alloc (zone=3D0xc083aee0, bytes=3D0, pflag=3D0x0, =
=20
wait=3D0)
---Type <return> to continue, or q <return> to quit---
at /usr/src/sys/vm/uma_core.c:806
#13 0xc048ebbf in slab_zalloc (zone=3D0xc083aee0, wait=3D2)
at /usr/src/sys/vm/uma_core.c:711
#14 0xc048fd58 in uma_zone_slab (zone=3D0xc083aee0, flags=3D258)
at /usr/src/sys/vm/uma_core.c:1503
#15 0xc048ff96 in uma_zalloc_bucket (zone=3D0xc083aee0, flags=3D258)
at /usr/src/sys/vm/uma_core.c:1606
#16 0xc048fbf9 in uma_zalloc_arg (zone=3D0xc083aee0, udata=3D0x0, =
flags=3D258)
at /usr/src/sys/vm/uma_core.c:1434
#17 0xc0321543 in malloc (size=3D3229855456, type=3D0xc0583a80, =
flags=3D258)
at /usr/src/sys/vm/uma.h:229
#18 0xc03325f5 in sigacts_alloc () at /usr/src/sys/kern/kern_sig.c:2719
#19 0xc03173ce in fork1 (td=3D0xc18bce40, flags=3D20, pages=3D0, =20
procp=3D0xcc464cd8)
at /usr/src/sys/kern/kern_fork.c:414
#20 0xc0316c2b in fork (td=3D0xc18bce40, uap=3D0xcc464d10)
at /usr/src/sys/kern/kern_fork.c:102
#21 0xc04c6753 in syscall (frame=3D
{tf_fs =3D 134938671, tf_es =3D 134873135, tf_ds =3D -1078001617, =
=20
tf_edi =3D 6, tf_esi =3D 135030952, tf_ebp =3D -1077937480, tf_isp =3D =20=
-867807884, tf_ebx =3D 135016448, tf_edx =3D 3, tf_ecx =3D -1077937680, =20=
tf_eax =3D 2, tf_trapno =3D 12, tf_err =3D 2, tf_eip =3D 673679423, =
tf_cs =3D 31, =20
tf_eflags =3D 531, tf_esp =3D -1077937732, tf_ss =3D 47})
at /usr/src/sys/i386/i386/trap.c:1008
#22 0xc04b5f8d in Xint0x80_syscall () at {standard input}:144
---Can't read userspace from dump, or kernel process---
(kgdb) where full
#0 doadump () at /usr/src/sys/kern/kern_shutdown.c:240
No locals.
#1 0xc032cc4c in boot (howto=3D260) at =20
/usr/src/sys/kern/kern_shutdown.c:372
No locals.
#2 0xc032cfd7 in panic () at /usr/src/sys/kern/kern_shutdown.c:550
td =3D (struct thread *) 0xc18bce40
bootopt =3D 260
newpanic =3D 0
ap =3D 0xcc464924 "=E2=80=B9IF=C3=83=C2=A2=3D\026=C2=BF\004HK=C2=BF=
"
buf =3D "kmem_malloc(4096): kmem_map too small: 112951296 total =20=
allocated", '\0' <repeats 191 times>
#3 0xc0163e22 in db_panic () at /usr/src/sys/ddb/db_command.c:449
No locals.
#4 0xc0163da2 in db_command (last_cmdp=3D0xc05c6b40, cmd_table=3D0x0,
aux_cmd_tablep=3D0xc054de7c, aux_cmd_tablep_end=3D0xc054de94)
at /usr/src/sys/ddb/db_command.c:346
cmd =3D (struct command *) 0xc04dedfc
t =3D 0
modif =3D =20
=
"\0t\\=C2=BFhid=C2=BFlIF=C3=83\r\0\0\0=E2=80=A1Tc=C2=BF\r\0\0\0\001\0\0\0\=
214IF=C3=83F=C2=A3J=C2=BF=E2=80=A1:b=C2=BF\aK\0 =20
=
`Uc=C2=BF=E2=80=A1]a=C2=BF=E2=80=A0t\\=C2=BFx\0\0\0=E2=80=A0t\\=C2=BFhid=C2=
=BF=E2=88=9EIF=C3=83a[\026=C2=BF\222ZP=C2=BFPZ\026=C2=BF\0\0\0\0\020\0\0\0=
=20
hid=C2=BF=E2=80=A0t\\=C2=BF=E2=88=82S\026=C2=BF=E2=80=A0t\\=C2=BF=E2=80=93=
l\\=C2=BFx\0\0\0\003\0\0"
addr =3D -1068808188
count =3D -1
have_addr =3D 0
---Type <return> to continue, or q <return> to quit---
result =3D 0
#5 0xc0163ec5 in db_command_loop () at =20
/usr/src/sys/ddb/db_command.c:471
No locals.
#6 0xc0166dc5 in db_trap (type=3D3, code=3D0) at =20
/usr/src/sys/ddb/db_trap.c:73
bkpt =3D 0
#7 0xc04b454c in kdb_trap (type=3D3, code=3D0, regs=3D0xcc464aa4)
at /usr/src/sys/i386/i386/db_interface.c:172
ef =3D 70
ddb_mode =3D 1
#8 0xc04c5e1d in trap (frame=3D
{tf_fs =3D -1047855080, tf_es =3D -867827696, tf_ds =3D 16, =
tf_edi =3D 1, =20
tf_esi =3D -1068224493, tf_ebp =3D -867808528, tf_isp =3D -867808560, =
tf_ebx =20
=3D 0, tf_edx =3D 0, tf_ecx =3D -1067232032, tf_eax =3D 18, tf_trapno =3D =
3, =20
tf_err =3D 0, tf_eip =3D -1068808188, tf_cs =3D 8, tf_eflags =3D 646, =
tf_esp =3D =20
-1068208597, tf_ss =3D -1068312245})
at /usr/src/sys/i386/i386/trap.c:580
td =3D (struct thread *) 0xc18bce40
p =3D (struct proc *) 0xc19c7d3c
sticks =3D 3224514865
i =3D 0
ucode =3D 0
type =3D 3
code =3D 0
eva =3D 0
#9 0xc04b5f38 in calltrap () at {standard input}:102
---Type <return> to continue, or q <return> to quit---
No locals.
#10 0xc032cf65 in panic (
fmt=3D0xc0543013 "kmem_malloc(%ld): kmem_map too small: %ld total =20=
allocated")
at /usr/src/sys/kern/kern_shutdown.c:534
td =3D (struct thread *) 0xc18bce40
bootopt =3D 256
newpanic =3D 1
ap =3D 0x0
buf =3D "kmem_malloc(4096): kmem_map too small: 112951296 total =20=
allocated", '\0' <repeats 191 times>
#11 0xc047dee0 in kmem_malloc (map=3D0xc082f0b0, size=3D4096, flags=3D2)
at /usr/src/sys/vm/vm_kern.c:339
offset =3D 710
i =3D 3229855456
entry =3D 0xcc464b7c
addr =3D 3233144832
m =3D 0x2
pflags =3D -1065111820
#12 0xc048ee87 in page_alloc (zone=3D0xc083aee0, bytes=3D0, pflag=3D0x0, =
=20
wait=3D0)
at /usr/src/sys/vm/uma_core.c:806
p =3D (void *) 0x0
#13 0xc048ebbf in slab_zalloc (zone=3D0xc083aee0, wait=3D2)
at /usr/src/sys/vm/uma_core.c:711
slab =3D 0xc76f24c8
---Type <return> to continue, or q <return> to quit---
mem =3D (u_int8_t *) 0xc083aef4 "=C2=A87X=C2=BF\227uO=C2=BF\235IT=C2=
=BF"
flags =3D 2 '\002'
i =3D 2
#14 0xc048fd58 in uma_zone_slab (zone=3D0xc083aee0, flags=3D258)
at /usr/src/sys/vm/uma_core.c:1503
slab =3D 0x0
#15 0xc048ff96 in uma_zalloc_bucket (zone=3D0xc083aee0, flags=3D258)
at /usr/src/sys/vm/uma_core.c:1606
bucket =3D 0xc192d400
slab =3D 0xc083aef4
#16 0xc048fbf9 in uma_zalloc_arg (zone=3D0xc083aee0, udata=3D0x0, =
flags=3D258)
at /usr/src/sys/vm/uma_core.c:1434
item =3D (void *) 0xc18bce40
cache =3D 0xc083afa8
bucket =3D 0x0
cpu =3D 0
#17 0xc0321543 in malloc (size=3D3229855456, type=3D0xc0583a80, =
flags=3D258)
at /usr/src/sys/vm/uma.h:229
indx =3D 8
va =3D 0xc05eff60 "LHX=C2=BF=C2=B6=E2=80=9CR=C2=BF=C2=B6=E2=80=9CR=
=C2=BF"
zone =3D 0xc083aee0
ksp =3D (struct malloc_type *) 0xc0583a80
#18 0xc03325f5 in sigacts_alloc () at /usr/src/sys/kern/kern_sig.c:2719
No locals.
---Type <return> to continue, or q <return> to quit---
#19 0xc03173ce in fork1 (td=3D0xc18bce40, flags=3D20, pages=3D0, =20
procp=3D0xcc464cd8)
at /usr/src/sys/kern/kern_fork.c:414
p2 =3D (struct proc *) 0xc1920974
pptr =3D (struct proc *) 0x0
uid =3D 3247573364
newproc =3D (struct proc *) 0xc1920974
trypid =3D 669
ok =3D 669
curfail =3D 0
pidchecked =3D 99999
lastfail =3D {tv_sec =3D 0, tv_usec =3D 0}
fd =3D (struct filedesc *) 0xc19c7da8
fdtol =3D (struct filedesc_to_leader *) 0x165
p1 =3D (struct proc *) 0xc19c7d3c
td2 =3D (struct thread *) 0x246
ke2 =3D (struct kse *) 0x29d
kg2 =3D (struct ksegrp *) 0x23
newsigacts =3D (struct sigacts *) 0x0
error =3D 35
#20 0xc0316c2b in fork (td=3D0xc18bce40, uap=3D0xcc464d10)
at /usr/src/sys/kern/kern_fork.c:102
error =3D 0
p2 =3D (struct proc *) 0xc18bce40
#21 0xc04c6753 in syscall (frame=3D
---Type <return> to continue, or q <return> to quit---
{tf_fs =3D 134938671, tf_es =3D 134873135, tf_ds =3D -1078001617, =
=20
tf_edi =3D 6, tf_esi =3D 135030952, tf_ebp =3D -1077937480, tf_isp =3D =20=
-867807884, tf_ebx =3D 135016448, tf_edx =3D 3, tf_ecx =3D -1077937680, =20=
tf_eax =3D 2, tf_trapno =3D 12, tf_err =3D 2, tf_eip =3D 673679423, =
tf_cs =3D 31, =20
tf_eflags =3D 531, tf_esp =3D -1077937732, tf_ss =3D 47})
at /usr/src/sys/i386/i386/trap.c:1008
params =3D 0xbfbff9c0---Can't read userspace from dump, or =
kernel =20
process---
(kgdb)
(kgdb) quit
On Saturday, July 26, 2003, at 09:06 PM, Mark Blackman wrote:
> Hi all,
>
> I'm seeing the same 'kmem_malloc(4096): kmem_map too small: XXXXX =20
> total allocated'
> messages that a few other have reported.
>
> Now, I understand that setting kern.vm.kmem.size larger is supposed to
> help, but I'm using a 128M Celeron-650 i386 system with no unusual
> devices (expect perhaps a Speedtouch ADSL modem) and I've =
progressively
> set the kern.vm.kmem.size to larger and larger values, starting at
> 64MB, then 96MB and finally 128MB.
>
> As I approached the physical memory size of the machine (128MB),
> the panic problem failed to reappear, but I got another problem =20
> whereby the kernel
> appeared to take over all of memory (i.e. processes were gradually
> all getting swapped out, but no other process seemed to be taking
> the memory) within about 30 minutes of boot-up.
>
> I noticed in the final minutes of the case where kmem.size=3D128MB =
(i.e. =20
> all
> of physical RAM), that kern.malloc was reporting 100M of 'devbuf' =20
> memory
> allocations and that it was gradually increasing at about 25k per
> second. I can't believe this is normal behaviour, but I'm no
> expert. I believe the devbuf allocations are specifically for
> device drivers.
>
> =46rom these symptoms, I'm speculating that one or more device drivers
> are producing kernel memory leaks and either triggering the
> 'kmem_map too small' messages or pushing all of the userland processes
> out of the way. Is this a reasonable interpretation?
>
> Does anyone else see symptoms that might lead to this conclusion?
>
> As a side note, I also briefly witnessed scrolling
> errors like 'ad0: out of memory in start'.
>
> I have no idea if this implies the 'ad' driver is an issue.
>
> Regards,
> Mark Blackman
> Exonetric Consulting
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?80B1C75A-BFAA-11D7-A23D-00039315D3FE>