Date: Fri, 12 Jul 2013 08:05:27 +0200 From: Andre Albsmeier <Andre.Albsmeier@siemens.com> To: Konstantin Belousov <kostikbel@gmail.com> Cc: "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org>, John Baldwin <jhb@freebsd.org> Subject: Re: FreeBSD-9.1: machine reboots during snapshot creation, LORs found Message-ID: <20130712060527.GA483@bali> In-Reply-To: <20130712060112.GY91021@kib.kiev.ua> References: <20130616063942.GA72803@bali> <201306171530.31208.jhb@freebsd.org> <20130704051409.GA22021@bali> <20130704052440.GG91021@kib.kiev.ua> <20130704052659.GA23398@bali> <20130704061550.GI91021@kib.kiev.ua> <20130704142919.GA1798@bali> <20130704172528.GL91021@kib.kiev.ua> <20130712052440.GA97779@bali> <20130712060112.GY91021@kib.kiev.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, 12-Jul-2013 at 08:01:12 +0200, Konstantin Belousov wrote: > On Fri, Jul 12, 2013 at 07:24:40AM +0200, Andre Albsmeier wrote: > > On Thu, 04-Jul-2013 at 19:25:28 +0200, Konstantin Belousov wrote: > > > On Thu, Jul 04, 2013 at 04:29:19PM +0200, Andre Albsmeier wrote: > > > > OK, patch is applied. I will reboot the machine later > > > > and see what happens tomorrow in the morning. However, > > > > it might take a few days since the last 2 weeks all was > > > > fine. > > > > > > > > BTW, should this patch be used in general or is it just > > > > for debugging? My understanding is that it is something > > > > which could stay in the code... > > > > > > Patch is to improve debugging. > > > > > > I probably commit it after the issue is closed. Arguments against > > > the commit is that the change imposes small performance penalty > > > due to save and restore of the %ebp (I doubt that this is measureable > > > by any means). Also, arguably, such change should be done for all > > > functions in support.s, but bcopy() is the hot spot. > > > > Got a new one, 2 hours old ;-) > > > > GNU gdb 6.1.1 [FreeBSD] > > Copyright 2004 Free Software Foundation, Inc. > > GDB is free software, covered by the GNU General Public License, and you are > > welcome to change it and/or distribute copies of it under certain conditions. > > Type "show copying" to see the conditions. > > There is absolutely no warranty for GDB. Type "show warranty" for details. > > This GDB was configured as "i386-marcel-freebsd"... > > > > Unread portion of the kernel message buffer: > > > > > > Fatal trap 12: page fault while in kernel mode > > fault virtual address = 0xcd5ec000 > > fault code = supervisor write, page not present > > instruction pointer = 0x20:0xc07cb2fe > > stack pointer = 0x28:0xd82e45cc > > frame pointer = 0x28:0xd82e45d4 > > code segment = base 0x0, limit 0xfffff, type 0x1b > > = DPL 0, pres 1, def32 1, gran 1 > > processor eflags = interrupt enabled, resume, IOPL = 0 > > current process = 18714 (mksnap_ffs) > > trap number = 12 > > panic: page fault > > KDB: stack backtrace: > > db_trace_self_wrapper(c08207eb,d82e4418,c05fdfc9,c081df13,c08a82e0,...) at db_trace_self_wrapper+0x26/frame 0xd82e43e8 > > kdb_backtrace(c081df13,c08a82e0,c0801bfa,d82e4424,d82e4424,...) at kdb_backtrace+0x29/frame 0xd82e43f4 > > panic(c0801bfa,c0845a01,c2b067d4,1,1,...) at panic+0xc9/frame 0xd82e4418 > > trap_fatal(c0ff6000,cd5ec000,2,0,c08b6bf4,...) at trap_fatal+0x353/frame 0xd82e4458 > > trap_pfault(baa8454b,21510,0,c2b06620,c08b6bf0,...) at trap_pfault+0x2d7/frame 0xd82e44a0 > > trap(d82e458c) at trap+0x41a/frame 0xd82e4580 > > calltrap() at calltrap+0x6/frame 0xd82e4580 > > --- trap 0xc, eip = 0xc07cb2fe, esp = 0xd82e45cc, ebp = 0xd82e45d4 --- > > bcopy(c36ed000,cd5e6000,8000,8000,c281b980,...) at bcopy+0x1a/frame 0xd82e45d4 > > ffs_snapshot(c2b35a90,c2ed0400,0,0,0,...) at ffs_snapshot+0x2933/frame 0xd82e490c > > ffs_mount(c2b35a90,c322e200,ff,d82e4c08,c2ccbc8c,...) at ffs_mount+0x15ee/frame 0xd82e4a3c > > vfs_donmount(c2b06620,10313108,0,c2b74d80,c2b74d80,...) at vfs_donmount+0x196b/frame 0xd82e4c2c > > sys_nmount(c2b06620,d82e4ccc,c2b06908,d82e4c6c,c0605015,...) at sys_nmount+0x63/frame 0xd82e4c50 > > syscall(d82e4d08) at syscall+0x2ce/frame 0xd82e4cfc > > Xint0x80_syscall() at Xint0x80_syscall+0x21/frame 0xd82e4cfc > > --- syscall (378, FreeBSD ELF32, sys_nmount), eip = 0x180bdf37, esp = 0xbfbfd65c, ebp = 0xbfbfddd8 --- > > Uptime: 4d20h0m44s > > Physical memory: 503 MB > > Dumping 104 MB: 89 73 57 41 25 9 > > > > No symbol "stopped_cpus" in current context. > > No symbol "stoppcbs" in current context. > > #0 doadump (textdump=1) at pcpu.h:249 > > 249 pcpu.h: No such file or directory. > > in pcpu.h > > (kgdb) where > > #0 doadump (textdump=1) at pcpu.h:249 > > #1 0xc05fdddd in kern_reboot (howto=260) at /src/src-9/sys/kern/kern_shutdown.c:449 > > #2 0xc05fe028 in panic (fmt=<value optimized out>) at /src/src-9/sys/kern/kern_shutdown.c:637 > > #3 0xc07cd1d3 in trap_fatal (frame=0xd82e458c, eva=3445538816) > > at /src/src-9/sys/i386/i386/trap.c:1044 > > #4 0xc07cd4b7 in trap_pfault (frame=0xd82e458c, usermode=0, eva=3445538816) > > at /src/src-9/sys/i386/i386/trap.c:957 > > #5 0xc07ce05a in trap (frame=0xd82e458c) at /src/src-9/sys/i386/i386/trap.c:555 > > #6 0xc07ba88c in calltrap () at /src/src-9/sys/i386/i386/exception.s:170 > > #7 0xc07cb2fe in bcopy () at /src/src-9/sys/i386/i386/support.s:198 > > #8 0xc072be13 in ffs_snapshot (mp=0xc2b35a90, snapfile=0xc2ed0400 "s5-2013.07.12-03.15.01") > > at /src/src-9/sys/ufs/ffs/ffs_snapshot.c:793 > > #9 0xc0748e8e in ffs_mount (mp=0xc2b35a90) at /src/src-9/sys/ufs/ffs/ffs_vfsops.c:483 > > #10 0xc068a72b in vfs_donmount (td=0xc2b06620, fsflags=271659272, fsoptions=0xc2b74d80) > > at /src/src-9/sys/kern/vfs_mount.c:948 > > #11 0xc068a8e3 in sys_nmount (td=0xc2b06620, uap=0xd82e4ccc) at /src/src-9/sys/kern/vfs_mount.c:417 > > #12 0xc07cd7ae in syscall (frame=0xd82e4d08) at subr_syscall.c:135 > > #13 0xc07ba8f1 in Xint0x80_syscall () at /src/src-9/sys/i386/i386/exception.s:270 > > #14 0x00000033 in ?? () > > Previous frame inner to this frame (corrupt stack?) > > Please show me the first 100 lines of the output of dumpfs(8) on the > filesystem where snapshot creation caused the panic. OK, dumpfs /dev/stripe/p | head -100: magic 11954 (UFS1) time Fri Jul 12 08:02:40 2013 id [ 517fa356 4ecc9335 ] ncg 82 size 17774144 blocks 17737399 bsize 32768 shift 15 mask 0xffff8000 fsize 4096 shift 12 mask 0xfffff000 frag 8 shift 3 fsbtodb 3 minfree 8% optim time symlinklen 60 maxbpg 4096 maxcontig 4 contigsumsize 4 nbfree 1958555 ndir 695 nifree 1123668 nffree 5395 cpg 1 bpg 27415 fpg 219320 ipg 13824 nindir 8192 inopb 256 nspf 8 maxfilesize 18016597801566207 sbsize 4096 cgsize 32768 cgoffset 0 cgmask 0xffffffff csaddr 456 cssize 4096 rotdelay 0ms rps 60 trackskew 0 interleave 1 nsect 1754560 npsect 1754560 spc 1754560 sblkno 8 cblkno 16 iblkno 24 dblkno 456 cgrotor 50 fmod 0 ronly 0 clean 0 metaspace 0 avgfpdir 64 avgfilesize 16384 flags soft-updates fsmnt /palveli volname swuid 0 providersize 17774144 cs[].cs_(nbfree,ndir,nifree,nffree): (6,43,12636,563) (0,82,12780,534) (0,21,13512,313) (10299,81,12257,612) (18212,74,13467,297) (9782,206,10086,2351) (0,115,12467,419) (0,0,13824,2) (25472,73,13487,275) (0,0,13824,13) (23074,0,13824,0) (27335,0,13824,0) (27359,0,13824,0) (0,0,13824,9) (27268,0,13824,0) (27353,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (10758,0,13824,7) (27353,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (27321,0,13824,0) (27359,0,13824,0) (27341,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27341,0,13824,0) (27357,0,13824,0) (27353,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (27353,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27353,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27353,0,13824,0) (27353,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (27353,0,13824,0) (27359,0,13824,0) (27353,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27353,0,13824,0) (27359,0,13824,0) (27353,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27353,0,13824,0) (27353,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (27359,0,13824,0) (27347,0,13824,0) (1097,0,13824,0) cylinders in last group 1 blocks in last group 1153 cg 0: magic 90255 tell 10000 time Fri Jul 12 08:02:20 2013 cgx 0 ncyl 1 niblk 13824 ndblk 219320 nbfree 6 ndir 43 nifree 12636 nffree 563 rotor 204848 irotor 1187 frotor 204848 frsum 13 20 18 20 22 14 26 sum of frsum: 563 clusters 1-3: 4 1 0 clusters size 4 and over: 0 clusters free: 26198, 26219-26220, 26226, 26775, 26798 inodes used: 0-1186, 1188 blks free: 457-458, 1028-1030, 1094-1095, 1590-1591, 4077-4079, 8258-8262, 9241-9247, 13710-13711, 13763-13767, 14241-14247, 14385-14391, 14433-14439, 20761-20767, 20857-20863, 20907-20911, 21051-21055, 21098-21103, 21193-21199, 21262-21263, 28404-28406, 28497-28503, 28545-28551, 28882-28884, 28933-28935, 28982-28983, 29028-29029, 29073-29077, 30021-30023, 30036-30038, 30067-30070, 36569-36573, 36617-36623, 36713-36719, 36761-36767, 42941-42943, 43124-43126, 43169-43175, 43313-43319, 43490-43495, 43537-43543, 54337-54343, 54386-54391, 54481-54486, 54529-54535, 54577-54583, 54817-54821, 56186-56191, 56549-56551, 56957-56958, 57373-57375, 57733-57734, 58045-58046, 58535, 59865-59871, 59913-59919, 59961-59967, 64762-64767, 65330-65335, 65379-65383, 66450-66454, 66810-66814, 67874-67878, 72902-72903, 73622-73623, 73982-73983, 78009-78015, 78057-78063, 78107-78111, 78428-78431, 92620-92623, 94311, 95071, 95823, 96207, 97343, 98103, 99281-99286, 100242-100246, 100986-100991, 101730-101735, 102474-102479, 103220-103223, 103982-103983, 105502-105503, 106849-106855, 107596-107599, 107964-107967, 108708-108711, 109444-109447, 109820-109823, 110188-110191, 110556-110559, 110924-110927, 111660-111663, 112345-112351, 112393-112399, 113570-113575, 114662-114663, 114685-114687, 114699-114703, 114747-114751, 114797-114799, 114810-114815, 114870-114871, 114954-114959, 115291-115295, 115307-115311, 115348-115351, 115357-115359, 116293-116295, 116342, 119981-119983, 120013-120015, 122603-122607, 122628-122631, 122748-122751, 122805-122807, 124235-124239, 189915-189919, 189956-189959, 189999, 190052-190055, 190067-190071, 190611-190615, 190671, 190678, 190687, 193116-193119, 193135, 193254-193255, 193324-193327, 193350-193351, 193390-193391, 209584-209591, 209752-209767, 209808-209815, 214200-214207, 214384-214391 cg 1: magic 90255 tell 358c8000 time Tue Apr 30 13:01:34 2013 cgx 1 ncyl 1 niblk 13824 ndblk 219320 nbfree 0 ndir 82 nifree 12780 nffree 534 rotor 0 irotor 1043 frotor 4624 frsum 9 4 25 32 35 22 1 sum of frsum: 534 clusters 1-3: 0 0 0 clusters size 4 and over: 0 clusters free: inodes used: 0-1043 blks free: 459-463, 2804-2807, 3674-3679, 3755-3759, 4436-4439, 4466-4468, 4490-4494, 4523-4526, 4629, 4667-4671, 4682-4684, 4722-4726, 4738-4742, 4779-4783, 4796-4798, 4827-4830, 4842-4846, 4858-4863, 4876-4879, 4908-4911, 4940-4941, 4954-4959, 5002-5006, 5019-5023, 5038-5039, 5082-5087, 5133-5135, 5244-5247, 5259-5263, 5274-5279, 5290-5293, 5309-5311, 5335, 5348-5351, 5364-5367, 5397-5399,
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130712060527.GA483>