Date: Sat, 23 Aug 2003 02:50:46 +0200 (CEST) From: Pawel Malachowski <pawmal-posting@freebsd.lublin.pl> To: FreeBSD-gnats-submit@FreeBSD.org Subject: kern/55886: mbuf exhaustion can cause panic Message-ID: <20030823005046.5FBDE5F103@shellma.zin.lublin.pl> Resent-Message-ID: <200308230050.h7N0oE0n024413@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 55886
>Category: kern
>Synopsis: mbuf exhaustion can cause panic
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-bugs
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Fri Aug 22 17:50:13 PDT 2003
>Closed-Date:
>Last-Modified:
>Originator: Pawe³ Ma³achowski
>Release: FreeBSD 4.7-RELEASE-p11 i386
>Organization:
ZiN
>Environment:
System: FreeBSD 4.8, 5.1
>Description:
After mbuf exhaustion it is possible, that kernel will panic because of page
fault. This was observed by me on two different machines:
1. Celeron/333 with 128MB RAM, running latest GENERIC 4.8-STABLE or 5.1-RELEASE
2. PIII/750 with 768MB RAM, runnig latest customized 4.8-STABLE
and by Adam Liberacki from polish FreeBSD users group on his:
3. PIII/600 with 256MB RAM, running 5.1-CURRENT
After pingflooding localhost, machine crashes (1. within one minute after
mbuf exhaustion, 2. within few minutes, 3. after 4 hours) in exactly the
same way (from 4.8-STABLE):
Fatal trap 12: page fault while in kernel mode
fault virtual address = 0x8
fault code = supervisor read, page not present
instruction pointer = 0x8:0xc0299fb7
stack pointer = 0x10:0xe3b8ddfc
frame pointer = 0x10:0xe3b8de10
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 24445 (ping)
interrupt mask =
trap number = 12
panic: page fault
syncing disks...
done
Uptime: 10d0h19m26s
dumping to dev #da/0x20009, offset 2621464
[cut]
#0 dumpsys () at ../../kern/kern_shutdown.c:487
487 if (dumping++) {
(kgdb) bt
#0 dumpsys () at ../../kern/kern_shutdown.c:487
#1 0xc023911f in boot (howto=256) at ../../kern/kern_shutdown.c:316
#2 0xc0239544 in poweroff_wait (junk=0xc0437c2c, howto=-1069320401)
at ../../kern/kern_shutdown.c:595
#3 0xc03b304a in trap_fatal (frame=0xe3b8ddbc, eva=8)
at ../../i386/i386/trap.c:974
#4 0xc03b2d1d in trap_pfault (frame=0xe3b8ddbc, usermode=0, eva=8)
at ../../i386/i386/trap.c:867
#5 0xc03b28db in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = -474480624,
tf_edi = -526853952, tf_esi = 20, tf_ebp = -474423792, tf_isp = -474423832,
tf_ebx = -1027443712, tf_edx = 0, tf_ecx = 0, tf_eax = 0, tf_trapno = 12,
tf_err = 0, tf_eip = -1071013961, tf_cs = 8, tf_eflags = 66118,
tf_esp = -1027443712, tf_ss = -1044738560}) at ../../i386/i386/trap.c:466
#6 0xc0299fb7 in rip_output (m=0x0, so=0xdf9c28c0, dst=16777343)
at ../../netinet/raw_ip.c:260
#7 0xc029a503 in rip_send (so=0xdf9c28c0, flags=0, m=0xc2c27400, nam=0xc409f410,
control=0x0, p=0xe3ad9d00) at ../../netinet/raw_ip.c:612
#8 0xc025833b in sosend (so=0xdf9c28c0, addr=0xc409f410, uio=0xe3b8decc,
top=0xc2c27400, control=0x0, flags=0, p=0xe3ad9d00)
at ../../kern/uipc_socket.c:609
#9 0xc025b7bb in sendit (p=0xe3ad9d00, s=3, mp=0xe3b8df0c, flags=0)
at ../../kern/uipc_syscalls.c:590
#10 0xc025b8be in sendto (p=0xe3ad9d00, uap=0xe3b8df80)
at ../../kern/uipc_syscalls.c:643
#11 0xc03b32f9 in syscall2 (frame={tf_fs = -1078067153, tf_es = -1078067153,
tf_ds = -1078067153, tf_edi = 500, tf_esi = 134753312, tf_ebp = -1078003760,
tf_isp = -474423340, tf_ebx = 65445, tf_edx = 16409, tf_ecx = 134818756,
tf_eax = 133, tf_trapno = 7, tf_err = 2, tf_eip = 134530472, tf_cs = 31,
tf_eflags = 663, tf_esp = -1078003836, tf_ss = 47})
at ../../i386/i386/trap.c:1175
#12 0xc03a44c5 in Xint0x80_syscall ()
#13 0x80492e1 in ?? ()
#14 0x804813e in ?? ()
(kgdb) up 6
#6 0xc0299fb7 in rip_output (m=0x0, so=0xdf9c28c0, dst=16777343)
at ../../netinet/raw_ip.c:260
260 ip = mtod(m, struct ip *);
(kgdb) list
255 if (m->m_pkthdr.len + sizeof(struct ip) > IP_MAXPACKET) {
256 m_freem(m);
257 return(EMSGSIZE);
258 }
259 M_PREPEND(m, sizeof(struct ip), M_WAIT);
260 ip = mtod(m, struct ip *);
261 ip->ip_tos = inp->inp_ip_tos;
262 ip->ip_off = 0;
263 ip->ip_p = inp->inp_ip_p;
264 ip->ip_len = m->m_pkthdr.len;
(kgdb) p m
$1 = (struct mbuf *) 0x0
(kgdb) up
#7 0xc029a503 in rip_send (so=0xdf9c28c0, flags=0, m=0xc2c27400, nam=0xc409f410,
control=0x0, p=0xe3ad9d00) at ../../netinet/raw_ip.c:612
612 return rip_output(m, so, dst);
(kgdb) p m
$2 = (struct mbuf *) 0xc2c27400
(kgdb) p *m
$3 = {m_hdr = {mh_next = 0x0, mh_nextpkt = 0x0, mh_data = 0xc2c2742c "\b",
mh_len = 212, mh_type = 0, mh_flags = 2}, M_dat = {MH = {MH_pkthdr = {
rcvif = 0x0, len = 65445, header = 0x2, csum_flags = 0, csum_data = 5,
tags = {slh_first = 0x0}}, MH_dat = {MH_ext = {
ext_buf = 0x95010008 <Address 0x95010008 out of bounds>,
ext_free = 0x5f45f7d, ext_size = 1061458863, ext_ref = 0xbe5e8},
MH_databuf = "\b\000\001\225}_ô\005¯\223D?èå\013\000\b\t\n\013\f\r\016\017\020\021\022\023\024\025\026\027\030\031\032\e\034\035\036\037 !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_bcdefghijklmnopqrstuvwxyz{|}~\177\200\201\202\203\204\205\206\207\210\211\212\213\214\215\216\217\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237 ¡¢£¤¥¦§¨©ª«¬®¯°±²³´µ¶·¸¹º»¼½¾¿"...}},
M_databuf = "\000\000\000\000¥ÿ\000\000\002\000\000\000\000\000\000\000\005\000\000\000\000\000\000\000\b\000\001\225}_ô\005¯\223D?èå\013\000\b\t\n\013\f\r\016\017\020\021\022\023\024\025\026\027\030\031\032\e\034\035\036\037 !\"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_bcdefghijklmnopqrstuvwxyz{|}~\177\200\201\202\203\204\205\206\207\210\211\212\213\214\215\216\217\220\221\222\223\224\225\226\227\230\231\232\233\234\235\236\237 ¡¢£¤¥¦§"...}}
It also seems, that before M_PREPEND at line 259, *m is still valid.
On 5.1, it panics earlier, at rip_send().
2nd machine:
% netstat -m -M vmcore.2
26391/26432/26432 mbufs in use (current/peak/max):
25035 mbufs allocated to data
678 mbufs allocated to ancillary data
678 mbufs allocated to socket names and addresses
6576/6608/6608 mbuf clusters in use (current/peak/max)
19824 Kbytes allocated to network (100% of mb_map in use)
5808737 requests for memory denied
155238 requests for memory delayed
143324 calls to protocol drain routines
1st machine (with kern.ipc.nmbclusters="8192" on loader.conf):
% netstat -m -M vmcore.78
32767/32768/32768 mbufs in use (current/peak/max):
10968 mbufs allocated to data
10897 mbufs allocated to ancillary data
10902 mbufs allocated to socket names and addresses
4023/4024/8192 mbuf clusters in use (current/peak/max)
16240 Kbytes allocated to network (66% of mb_map in use)
323769 requests for memory denied
178 requests for memory delayed
228 calls to protocol drain routines
>How-To-Repeat:
Try running this on a machine with 128MB memory and GENERIC autosized
mbuf-related values. Run as root and without login.conf limits.
It is very easy to reproduce for me, however, as discussed at polish FreeBSD
usenet group, some people claim, they are not able to reproduce that.
#!/bin/sh
while [ 1 ]
do
ping -q -i 0 -s 65437 localhost >/dev/null 2>&1 &
done
Wait patiently. :)
>Fix:
Workaround: increase kern.ipc.nmbclusters="32768" for example, after that
I can't reproduce panic on 1. machine.
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20030823005046.5FBDE5F103>