Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 13 Nov 2007 23:50:26 +0200
From:      "Vlad GALU" <dudu@dudu.ro>
To:        freebsd-stable@freebsd.org
Subject:   Re: reboot after panic: vm_page_unwire: invalid wire count: 0
Message-ID:  <ad79ad6b0711131350u7c00fb0ek16a23f5dc12089cb@mail.gmail.com>
In-Reply-To: <0ECA0F43-497A-4823-8731-D517889D5822@khera.org>
References:  <0ECA0F43-497A-4823-8731-D517889D5822@khera.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On 11/13/07, Vivek Khera <vivek@khera.org> wrote:
> I've got a Dell 1750 box that was rock-solid stable running 4.11 for a
> couple of years now operating a pretty busy website backend.  A month
> or so ago we wiped it clean and repurposed it to run a different
> website running Drupal with a Varnish front-end cache using FreeBSD
> 6.2-RELEASE-p5.  The system is i386 and has 1Gb of RAM.
>
> Uname output: FreeBSD mb.kcilink.com 6.2-RELEASE-p5 FreeBSD 6.2-
> RELEASE-p5 #0: Wed Jun 27 10:47:15 EDT 2007
> vivek@dbarchive.m1e.net:/n/lorax1/usr6/obj.i386/n/lorax1/usr6/src/sys/
> KCI32SMP  i386
>
>
> The last week or so, it has been crashing regularly.  Sometimes twice
> per day, and sometimes it runs for two days without a problem.  I
> finally managed to make it dump a crashlog and core, and discovered
> that the panic was:
>
>   reboot after panic: vm_page_unwire: invalid wire count: 0
>
> I google around and found one old PR #33637 which had a patch but that
> was for FreeBSD 4.5.  I have also found two other mentions of this
> panic, one on the mailing lists with no responses, and another for a
> PR from 6.1-PRERELEASE, PR #94578, which has no comments on it.
>
> According to the http and varnish logs, we're not being particularly
> hit very hard when the panic happens, but I don't know if we lose some
> log data during the panic.
>
> I have the core and the kernel.debug.  I'm not sure what info to
> extract from it beyond the backtrace.  The watchdog timer fired and
> dropped me to DDB, so I just typed "watchdog" and "c" and let it
> finish dumping.
>
> Here's the backtrace, and "bt full" output.
>
>
> # kgdb kernel.debug /var/crash/vmcore.0
> [GDB will not be able to debug user-mode threads: /usr/lib/
> libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and
> you are
> welcome to change it and/or distribute copies of it under certain
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for
> details.
> This GDB was configured as "i386-marcel-freebsd".
>
> Unread portion of the kernel message buffer:
> panic: vm_page_unwire: invalid wire count: 0
> cpuid = 1
> KDB: stack backtrace:
> kdb_backtrace(100,c5a76000,c0e88ab0,0,d90d82c8,...) at kdb_backtrace
> +0x29
> panic(c06b011f,0,c0e88ab0,efe80900,c057b96a,...) at panic+0x114
> vm_page_unwire(c0e88ab0,0) at vm_page_unwire+0x68
> vfs_vmio_release(d90d82c8) at vfs_vmio_release+0xa2
> getnewbuf(0,0,4000,4000) at getnewbuf+0x2bc
> getblk(c6f81550,4f5,0,4000,0,...) at getblk+0x360
> ffs_balloc_ufs2(c6f81550,13d4000,0,fa,c4f32780,...) at
> ffs_balloc_ufs2+0x1606
> ffs_write(efe80bec) at ffs_write+0x2ec
> VOP_WRITE_APV(c06e06a0,efe80bec) at VOP_WRITE_APV+0xce
> vn_write(c59c8000,efe80cbc,c51cf400,0,c5a76000) at vn_write+0x1ee
> dofilewrite(c5a76000,c,c59c8000,efe80cbc,ffffffff,...) at dofilewrite
> +0x77
> kern_writev(c5a76000,c,efe80cbc,821bba3,fa,...) at kern_writev+0x3b
> write(c5a76000,efe80d04) at write+0x45
> syscall(3b,809003b,bfbf003b,0,bfbfeaa4,...) at syscall+0x2bf
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (4, FreeBSD ELF32, write), eip = 0x483d732f, esp =
> 0xbfbfe9dc, ebp = 0xbfbfea08 ---
> Uptime: 1d20h51m58s
> Dumping 1023 MB (2 chunks)
>    chunk 0: 1MB (159 pages) ... ok
>    chunk 1: 1023MB (261872 pages) 1007 991 975 959 943 927 911 895 879
> 863 847 831 815 799 783 767 751 735 719 703 687 671 655 639 623 607
> 591 575 559 543 527 511 495 479 463 447 431 415 399 383 367 351 335
> 319 303 287 271 255 239 223 207 191 175 159 143 127 111
> 95interrupt                   total
> irq4: sio0                         21758
> irq15: ata1                            1
> irq16: bge0                      4544565
> irq17: bge1                     17684238
> irq18: amr0                       588223
> cpu0: timer                    323148326
> cpu2: timer                    323148294
> cpu1: timer                    323148331
> cpu3: timer                    323148344
> Total                  1315432158
> KDB: stack backtrace:
> kdb_backtrace(c069ec5d,4e67e6de,0,c06ea170,c06e9818,...) at
> kdb_backtrace+0x29
> watchdog_fire(c07120e0,c8,efe80634,c065c821,efe8063c,...) at
> watchdog_fire+0x9d
> hardclock(efe8063c) at hardclock+0x115
> lapic_handle_timer(0) at lapic_handle_timer+0x51
> Xtimerint(c4fe6000,1,efe806a8,c066d57b,c4fe6000,...) at Xtimerint+0x30
> getit(c4fe6000,c4fe6000,4,efe806c0,c0496f97,...) at getit+0x88
> DELAY(1) at DELAY+0x3b
> amr_quartz_poll_command1(c4fe6000,c51fbff0,0,0,1000,...) at
> amr_quartz_poll_command1+0x1af
> amr_setup_polled_dmamap(c51fbff0,c4fef800,1,0) at
> amr_setup_polled_dmamap+0x94
> bus_dmamap_load(c4ffe380,0,c0c22000,10000,c0496cd4,c51fbff0,1) at
> bus_dmamap_load+0x4b5
> amr_quartz_poll_command(c51fbff0) at amr_quartz_poll_command+0x51
> amr_dump_blocks(c4fe6000,0,4cb25e,c0c22000,80) at amr_dump_blocks+0x5f
> amrd_dump(c515b700,c0c22000,0,9964bc00,0,10000) at amrd_dump+0x7c
> cb_dumpdata(c0711a48,1,c06f44a0) at cb_dumpdata+0x100
> foreach_chunk(c0655a78,c06f44a0,c06f44a0) at foreach_chunk+0x23
> dumpsys(c06f44a0,efe808b0,c05319be,0,c515b88c,...) at dumpsys+0x1ec
> doadump(0,c515b88c,483d732f,c4f8998c,c5a76000,...) at doadump+0x42
> boot(104,104,c5a76000,c0e88ab0,0,...) at boot+0x4ea
> panic(c06b011f,0,c0e88ab0,efe80900,c057b96a,...) at panic+0x1b9
> vm_page_unwire(c0e88ab0,0) at vm_page_unwire+0x68
> vfs_vmio_release(d90d82c8) at vfs_vmio_release+0xa2
> getnewbuf(0,0,4000,4000) at getnewbuf+0x2bc
> getblk(c6f81550,4f5,0,4000,0,...) at getblk+0x360
> ffs_balloc_ufs2(c6f81550,13d4000,0,fa,c4f32780,...) at
> ffs_balloc_ufs2+0x1606
> ffs_write(efe80bec) at ffs_write+0x2ec
> VOP_WRITE_APV(c06e06a0,efe80bec) at VOP_WRITE_APV+0xce
> vn_write(c59c8000,efe80cbc,c51cf400,0,c5a76000) at vn_write+0x1ee
> dofilewrite(c5a76000,c,c59c8000,efe80cbc,ffffffff,...) at dofilewrite
> +0x77
> kern_writev(c5a76000,c,efe80cbc,821bba3,fa,...) at kern_writev+0x3b
> write(c5a76000,efe80d04) at write+0x45
> syscall(3b,809003b,bfbf003b,0,bfbfeaa4,...) at syscall+0x2bf
> Xint0x80_syscall() at Xint0x80_syscall+0x1f
> --- syscall (4, FreeBSD ELF32, write), eip = 0x483d732f, esp =
> 0xbfbfe9dc, ebp = 0xbfbfea08 ---
> KDB: enter: watchdog timeout
>   79 63 47 31 15
>
> #0  doadump () at pcpu.h:165
> 165             __asm __volatile("movl %%fs:0,%0" : "=r" (td));
> (kgdb) bt full
> #0  doadump () at pcpu.h:165
> No locals.
> #1  0xc05319be in boot (howto=260)
>      at /n/lorax1/usr6/src/sys/kern/kern_shutdown.c:409
>         first_buf_printf = 1
> #2  0xc0531d15 in panic (
>      fmt=0xc06b011f "vm_page_unwire: invalid wire count: %d")
>      at /n/lorax1/usr6/src/sys/kern/kern_shutdown.c:565
>         td = (struct thread *) 0xc5a76000
>         bootopt = 260
>         newpanic = 1
>         ap = 0xc5a76000 "0D???T;?"
>         buf = "vm_page_unwire: invalid wire count: 0", '\0' <repeats 218 times>
> #3  0xc0633948 in vm_page_unwire (m=0xc0e88ab0, activate=0)
>      at /n/lorax1/usr6/src/sys/vm/vm_page.c:1275
> No locals.
> #4  0xc057b96a in vfs_vmio_release (bp=0xd90d82c8)
>      at /n/lorax1/usr6/src/sys/kern/vfs_bio.c:1470
>         i = 0
>         m = 0xc0e88ab0
> #5  0xc057c108 in getnewbuf (slpflag=0, slptimeo=0, size=16384,
> maxsize=16384)
>      at /n/lorax1/usr6/src/sys/kern/vfs_bio.c:1779
>         qindex = 1
>         bp = (struct buf *) 0xd90d82c8
>         nbp = (struct buf *) 0xd907ae20
>         defrag = 0
>         nqindex = 1
>         flushingbufs = 0
> #6  0xc057d6ac in getblk (vp=0xc6f81550, blkno=1269, size=16384,
> slpflag=0,
> ---Type <return> to continue, or q <return> to quit---
>      slptimeo=0, flags=0) at /n/lorax1/usr6/src/sys/kern/vfs_bio.c:2497
>         bsize = 16384
>         maxsize = 0
>         vmio = 1
>         offset = Unhandled dwarf expression opcode 0x93
> (kgdb)
>

    Do you happen to have ZERO_COPY_SOCKETS in your kernel config?

> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"
>


-- 
Mahnahmahnah!



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ad79ad6b0711131350u7c00fb0ek16a23f5dc12089cb>