From owner-freebsd-bugs@FreeBSD.ORG Thu Apr 19 19:30:11 2012 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 1102D106566B for ; Thu, 19 Apr 2012 19:30:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D8EC98FC17 for ; Thu, 19 Apr 2012 19:30:10 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q3JJUA42054873 for ; Thu, 19 Apr 2012 19:30:10 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q3JJUA52054872; Thu, 19 Apr 2012 19:30:10 GMT (envelope-from gnats) Resent-Date: Thu, 19 Apr 2012 19:30:10 GMT Resent-Message-Id: <201204191930.q3JJUA52054872@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Claudius Herder Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 772AC106564A for ; Thu, 19 Apr 2012 19:22:03 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id 584BF8FC14 for ; Thu, 19 Apr 2012 19:22:03 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q3JJM3GR080723 for ; Thu, 19 Apr 2012 19:22:03 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id q3JJM330080722; Thu, 19 Apr 2012 19:22:03 GMT (envelope-from nobody) Message-Id: <201204191922.q3JJM330080722@red.freebsd.org> Date: Thu, 19 Apr 2012 19:22:03 GMT From: Claudius Herder To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: kern/167109: zfs diff kernel panic Fatal trap 9: general protection fault X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Apr 2012 19:30:11 -0000 >Number: 167109 >Category: kern >Synopsis: zfs diff kernel panic Fatal trap 9: general protection fault >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Thu Apr 19 19:30:10 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Claudius Herder >Release: FreeBSD 9.0-RELEASE >Organization: >Environment: FreeBSD server.ambtec.de 9.0-RELEASE FreeBSD 9.0-RELEASE #2: Thu Apr 19 19:09:14 CEST 2012 claudius@server.ambtec.de:/usr/obj/usr/src/sys/CUSTOM amd64 >Description: I already posted to freebsd-fs@freebsd.org but got no response: I received a kernel panic after running zfs diff on my system/rootfs filesystem, other filesystems work without problems, but i have not tested them all. Apr 15 21:11:17 server kernel: Fatal trap 9: general protection fault while in kernel mode Apr 15 21:11:17 server kernel: cpuid = 3; apic id = 03 Apr 15 21:11:17 server kernel: instruction pointer = 0x20:0xffffffff80b0ff91 Apr 15 21:11:17 server kernel: stack pointer = 0x28:0xffffff845fa1b430 Apr 15 21:11:17 server kernel: frame pointer = 0x28:0xffffff845fa1b470 Apr 15 21:11:17 server kernel: code segment = base 0x0, limit 0xfffff, type 0x1b Apr 15 21:11:17 server kernel: = DPL 0, pres 1, long 1, def32 0, gran 1 Apr 15 21:11:17 server kernel: processor eflags = interrupt enabled, resume, IOPL = 0 Apr 15 21:11:17 server kernel: current process = 1797 (zfs) Apr 15 21:11:17 server kernel: trap number = 9 Apr 15 21:11:17 server kernel: panic: general protection fault Apr 15 21:11:17 server kernel: cpuid = 3 Apr 15 21:11:17 server kernel: KDB: stack backtrace: Apr 15 21:11:17 server kernel: #0 0xffffffff803fcb5e at kdb_backtrace+0x5e Apr 15 21:11:17 server kernel: #1 0xffffffff803c9ce3 at panic+0x183 Apr 15 21:11:17 server kernel: #2 0xffffffff805c49a0 at trap_fatal+0x290 Apr 15 21:11:17 server kernel: #3 0xffffffff805c4eda at trap+0x10a Apr 15 21:11:17 server kernel: #4 0xffffffff805af7ef at calltrap+0x8 Apr 15 21:11:17 server kernel: #5 0xffffffff80b0f0e0 at fzap_cursor_retrieve+0x110 Apr 15 21:11:17 server kernel: #6 0xffffffff80b13bc5 at zap_cursor_retrieve+0x155 Apr 15 21:11:17 server kernel: #7 0xffffffff80b0e12f at zap_value_search+0x7f Apr 15 21:11:17 server kernel: #8 0xffffffff80b18912 at zfs_obj_to_path_impl+0x292 Apr 15 21:11:17 server kernel: #9 0xffffffff80b18b45 at zfs_obj_to_stats+0x175 Apr 15 21:11:17 server kernel: #10 0xffffffff80b3075e at zfs_ioc_obj_to_stats+0x7e Apr 15 21:11:17 server kernel: #11 0xffffffff80b33a46 at zfsdev_ioctl+0xe6 Apr 15 21:11:17 server kernel: #12 0xffffffff8034e4db at devfs_ioctl_f+0x7b Apr 15 21:11:17 server kernel: #13 0xffffffff8040e185 at kern_ioctl+0x115 Apr 15 21:11:17 server kernel: #14 0xffffffff8040e3b0 at sys_ioctl+0xf0 Apr 15 21:11:17 server kernel: #15 0xffffffff805c4290 at amd64_syscall+0x450 Apr 15 21:11:17 server kernel: #16 0xffffffff805afad7 at Xfast_syscall+0xf7 I tried different snapshots on rootfs to narrow down then problem, but after a few forced reboots and zpool scrub after each crash zfs diff hangs only. procstat -k -k 8428 PID TID COMM TDNAME KSTACK 8428 100719 zfs - mi_switch+0x174 sleepq_catch_signals+0x2f4 sleepq_wait_sig+0xc _sleep+0x279 pipe_write+0x125e write_record+0x79 report_free_dnode_range+0x2c diff_cb+0x1bf traverse_visitbp+0x21c traverse_visitbp+0x309 traverse_visitbp+0x309 traverse_visitbp+0x309 traverse_visitbp+0x309 traverse_visitbp+0x309 traverse_visitbp+0x309 traverse_dnode+0x7c traverse_visitbp+0x47f traverse_impl+0x188 8428 101237 zfs - mi_switch+0x174 sleepq_wait+0x42 _sx_slock_hard+0x1f2 _sx_slock+0x45 zap_get_leaf_byblk+0xbd zap_deref_leaf+0x68 fzap_cursor_retrieve+0xe7 zap_cursor_retrieve+0x155 zap_value_search+0x7f zfs_obj_to_path_impl+0x292 zfs_obj_to_stats+0x175 zfs_ioc_obj_to_stats+0x7e zfsdev_ioctl+0xe6 devfs_ioctl_f+0x7b kern_ioctl+0x115 sys_ioctl+0xf0 amd64_syscall+0x450 Xfast_syscall+0xf7 zdb: system: version: 28 name: 'system' state: 0 txg: 744 pool_guid: 10799670079652932983 hostid: 3191383858 hostname: 'tmpserver.ambtec.de' vdev_children: 1 vdev_tree: type: 'root' id: 0 guid: 10799670079652932983 children[0]: type: 'mirror' id: 0 guid: 2010203102961720602 metaslab_array: 30 metaslab_shift: 34 ashift: 12 asize: 2966227255296 is_log: 0 create_txg: 4 children[0]: type: 'disk' id: 0 guid: 4310542124404815932 path: '/dev/gpt/system0' phys_path: '/dev/gpt/system0' whole_disk: 1 create_txg: 4 children[1]: type: 'disk' id: 1 guid: 14038163402933743119 path: '/dev/gpt/system1' phys_path: '/dev/gpt/system1' whole_disk: 1 create_txg: 4 dmesg: Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-RELEASE #2: Thu Apr 19 19:09:14 CEST 2012 claudius@server.ambtec.de:/usr/obj/usr/src/sys/CUSTOM amd64 CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz (3411.56-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206a7 Family = 6 Model = 2a Stepping = 7 Features=0xbfebfbff Features2=0x17bae3ff AMD Features=0x28100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 17179869184 (16384 MB) avail memory = 16436305920 (15674 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 cryptosoft0: on motherboard aesni0: on motherboard acpi0: on motherboard ACPI Error: [RAMB] Namespace lookup failure, AE_NOT_FOUND (20110527/psargs-392) ACPI Exception: AE_NOT_FOUND, Could not execute arguments for [RAMW] (Region) (20110527/nsinit-380) acpi0: Power Button (fixed) acpi0: reservation of 67, 1 (4) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xf000-0xf03f mem 0xf7800000-0xf7bfffff,0xe0000000-0xefffffff irq 16 at device 2.0 on pci0 pci0: at device 22.0 (no driver attached) ehci0: mem 0xf7d04000-0xf7d043ff irq 23 at device 26.0 on pci0 usbus0: EHCI version 1.0 usbus0: on ehci0 pcib2: irq 16 at device 28.0 on pci0 pci2: on pcib2 pcib3: irq 17 at device 28.5 on pci0 pci3: on pcib3 xhci0: mem 0xf7c00000-0xf7c07fff irq 17 at device 0.0 on pci3 xhci0: 32 byte context size. usbus1 on xhci0 pcib4: irq 18 at device 28.6 on pci0 pci4: on pcib4 re0: port 0xe000-0xe0ff mem 0xf0004000-0xf0004fff,0xf0000000-0xf0003fff irq 18 at device 0.0 on pci4 re0: Using 1 MSI-X message re0: Chip rev. 0x2c000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Ethernet address: 54:04:a6:7e:eb:34 pcib5: irq 19 at device 28.7 on pci0 pci5: on pcib5 pcib6: irq 19 at device 0.0 on pci5 pci6: on pcib6 ehci1: mem 0xf7d03000-0xf7d033ff irq 23 at device 29.0 on pci0 usbus2: EHCI version 1.0 usbus2: on ehci1 isab0: at device 31.0 on pci0 isa0: on isab0 ahci0: port 0xf0b0-0xf0b7,0xf0a0-0xf0a3,0xf090-0xf097,0xf080-0xf083,0xf060-0xf07f mem 0xf7d02000-0xf7d027ff irq 19 at device 31.2 on pci0 ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 pci0: at device 31.3 (no driver attached) acpi_button0: on acpi0 acpi_tz0: on acpi0 acpi_tz1: on acpi0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 550 atrtc0: port 0x70-0x77 irq 8 on acpi0 atrtc0: Warning: Couldn't map I/O. Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 coretemp0: on cpu0 est0: on cpu0 p4tcc0: on cpu0 coretemp1: on cpu1 est1: on cpu1 p4tcc1: on cpu1 coretemp2: on cpu2 est2: on cpu2 p4tcc2: on cpu2 coretemp3: on cpu3 est3: on cpu3 p4tcc3: on cpu3 coretemp4: on cpu4 est4: on cpu4 p4tcc4: on cpu4 coretemp5: on cpu5 est5: on cpu5 p4tcc5: on cpu5 coretemp6: on cpu6 est6: on cpu6 p4tcc6: on cpu6 coretemp7: on cpu7 est7: on cpu7 p4tcc7: on cpu7 ZFS filesystem version 5 ZFS storage pool version 28 Timecounters tick every 1.000 msec usbus0: 480Mbps High Speed USB v2.0 usbus1: 5.0Gbps Super Speed USB v3.0 usbus2: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: <0x1b21> at usbus1 uhub1: <0x1b21 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0: ATA-8 SATA 3.x device ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C) ada1 at ahcich1 bus 0 scbus1 target 0 lun 0 ada1: ATA-8 SATA 3.x device ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C) SMP: AP CPU #1 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #7 Launched! Timecounter "TSC-low" frequency 13326397 Hz quality 1000 GEOM_MIRROR: Device mirror/swap launched (2/2). uhub1: 4 ports with 4 removable, self powered Trying to mount root from zfs:system/rootfs [rw,noatime]... uhub0: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered ugen0.2: at usbus0 uhub3: on usbus0 ugen2.2: at usbus2 uhub4: on usbus2 uhub3: 6 ports with 6 removable, self powered uhub4: 8 ports with 8 removable, self powered GEOM_ELI: Device mirror/swap.eli created. GEOM_ELI: Encryption: AES-XTS 256 GEOM_ELI: Crypto: hardware >How-To-Repeat: I tried to analyze the size of a few snapshot size with zfs diff >Fix: >Release-Note: >Audit-Trail: >Unformatted: