Date: Fri, 4 Mar 2016 14:40:54 +0200 From: Vitalij Satanivskij <satan@ukr.net> To: current@freebsd.org Subject: CURRENT r296381 panic in vn_sendfile (/usr/src/sys/kern/kern_sendfile.c:833) Message-ID: <20160304124053.GA25071@hell.ukr.net>
next in thread | raw e-mail | index | archive | help
Hello.
I get kernel panic on high loaded server with messages
savecore: reboot after panic:
vn_sendfile: mlen 326 space -20 hdrlen 326
# kgdb kernel.debug /var/crash/vmcore.0
Unread portion of the kernel message buffer:
panic: vn_sendfile: mlen 326 space -20 hdrlen 326
cpuid = 5
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe20206314f0
vpanic() at vpanic+0x182/frame 0xfffffe2020631570
kassert_panic() at kassert_panic+0x126/frame 0xfffffe20206315e0
vn_sendfile() at vn_sendfile+0x14ca/frame 0xfffffe2020631900
sys_sendfile() at sys_sendfile+0x11e/frame 0xfffffe20206319a0
amd64_syscall() at amd64_syscall+0x2db/frame 0xfffffe2020631ab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe2020631ab0
--- syscall (393, FreeBSD ELF64, sys_sendfile), rip = 0x801ef062a, rsp = 0x7fffffffd8d8, rbp = 0x7fffffffe1d0 ---
KDB: enter: panic
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
Loaded symbols for /boot/kernel/opensolaris.ko
Reading symbols from /boot/kernel/carp.ko...Reading symbols from /usr/lib/debug//boot/kernel/carp.ko.debug...done.
done.
Loaded symbols for /boot/kernel/carp.ko
Reading symbols from /boot/kernel/ums.ko...Reading symbols from /usr/lib/debug//boot/kernel/ums.ko.debug...done.
done.
Loaded symbols for /boot/kernel/ums.ko
Reading symbols from /boot/kernel/tmpfs.ko...Reading symbols from /usr/lib/debug//boot/kernel/tmpfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/tmpfs.ko
#0 doadump (textdump=0) at pcpu.h:221
221 __asm("movq %%gs:%1,%0" : "=r" (td)
(kgdb) bt
#0 doadump (textdump=0) at pcpu.h:221
#1 0xffffffff80384a0b in db_dump (dummy=<value optimized out>, dummy2=false, dummy3=0, dummy4=0x0) at /usr/src/sys/ddb/db_command.c:533
#2 0xffffffff803847fe in db_command (cmd_table=0x0) at /usr/src/sys/ddb/db_command.c:440
#3 0xffffffff80384594 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493
#4 0xffffffff8038702b in db_trap (type=<value optimized out>, code=0) at /usr/src/sys/ddb/db_main.c:251
#5 0xffffffff80a656e3 in kdb_trap (type=3, code=0, tf=<value optimized out>) at /usr/src/sys/kern/subr_kdb.c:654
#6 0xffffffff80ea1298 in trap (frame=0xfffffe2020631420) at /usr/src/sys/amd64/amd64/trap.c:556
#7 0xffffffff80e81a77 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:234
#8 0xffffffff80a64dcb in kdb_enter (why=0xffffffff813b6c2f "panic", msg=0x80 <Address 0x80 out of bounds>) at cpufunc.h:63
#9 0xffffffff80a27b5f in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:750
#10 0xffffffff80a279b6 in kassert_panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:647
#11 0xffffffff80a25efa in vn_sendfile (fp=<value optimized out>, sockfd=1619, hdr_uio=<value optimized out>, trl_uio=0x0, offset=0,
nbytes=<value optimized out>, sent=<value optimized out>, flags=<value optimized out>, kflags=<value optimized out>, td=0xa8)
at /usr/src/sys/kern/kern_sendfile.c:833
#12 0xffffffff80a2641e in sys_sendfile (td=0xfffff80253593000, uap=0xfffffe2020631a40) at file.h:382
#13 0xffffffff80ea214b in amd64_syscall (td=0xfffff80253593000, traced=0) at subr_syscall.c:135
#14 0xffffffff80e81d5b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:394
#15 0x0000000801ef062a in ?? ()
Previous frame inner to this frame (corrupt stack?)
Current language: auto; currently minimal
(kgdb) list *0xffffffff80a25efa
0xffffffff80a25efa is in vn_sendfile (/usr/src/sys/kern/kern_sendfile.c:833).
828 free(sfio, M_TEMP);
829 goto done;
830 }
831
832 /* Add the buffer chain to the socket buffer. */
833 KASSERT(m_length(m, NULL) == space + hdrlen,
834 ("%s: mlen %u space %d hdrlen %d",
835 __func__, m_length(m, NULL), space, hdrlen));
836
837 CURVNET_SET(so->so_vnet);
System have 128Gb memory
zfs as FS
DB's worked on it and web pages served by this server.
core saved.
panic periodicaly repeted (few hours -- up to few days)
Before this, old current (about two year old CURRENT ) work on this server without crashes.
Can anybody point me to way of more complex problem diagnostic or any other useful things
Thank you.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20160304124053.GA25071>