From owner-freebsd-questions@FreeBSD.ORG Mon Jan 19 16:31:14 2015 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BEBD3B53 for ; Mon, 19 Jan 2015 16:31:14 +0000 (UTC) Received: from mail-oi0-x231.google.com (mail-oi0-x231.google.com [IPv6:2607:f8b0:4003:c06::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8261D60F for ; Mon, 19 Jan 2015 16:31:14 +0000 (UTC) Received: by mail-oi0-f49.google.com with SMTP id a3so3755478oib.8 for ; Mon, 19 Jan 2015 08:31:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=khera.org; s=google11; h=mime-version:date:message-id:subject:from:to:content-type; bh=Z5zK42oXpNR+ugxs1DF4AmBDLSNdu1VUBoaNdkuZ6z4=; b=PzkCm/xeO16oFwjYoIsmXvJzMPpb3XyUHTch2xw6EOy59k+JB1zr6zzlSY930nfReL 4qB5fTBP4GTxkhbxwGa+khZPpeLfizOYaHnPO4uh89lg83QtIe8w7UdPIu1b8W2/VU4k biyFhp4YETdQKkCznxyHLAmmXd4mwNtmTM/2M= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=Z5zK42oXpNR+ugxs1DF4AmBDLSNdu1VUBoaNdkuZ6z4=; b=OIG/yA4u8OyDdRL4LqCb+VsO6kcpZZtzZR4gf6lZqGk++/gPvXA0SCvJHtImQEXdWM kW7ojCXwa31ER+f6ysUVIjVSaEdbrmbdix4lPi1wxT6LzT1yOc41MAmJBVxrfFJU10sN JW9W/+qjfjbhrUrS223Q1GZ2L8qym63ZqYEUaKZz6HH4/70QVomFK/8Tu+/YrniEJjk9 Oe8E848FmxDfQEKFFEOmDTNfB6rTOGu2HMHmfBNWVdoRhq8TdhopxgvB6kEZjM62kAmc 4Ao+iOjNGiZm41FnEMOZF9gST11lZEQE3OnuSmVoNI2v6N6yShwX+Q1JKaJBYDrB3lV/ YUmg== X-Gm-Message-State: ALoCoQlbwV4V43ea7vbVwRDr96qarAf8k03vCcgAOxeZ8o13gzAT8boPczt2tjHAy9+/tCBa241G MIME-Version: 1.0 X-Received: by 10.202.169.200 with SMTP id s191mr17833216oie.7.1421685073554; Mon, 19 Jan 2015 08:31:13 -0800 (PST) Received: by 10.76.28.103 with HTTP; Mon, 19 Jan 2015 08:31:13 -0800 (PST) Date: Mon, 19 Jan 2015 11:31:13 -0500 Message-ID: Subject: kernel panic twice in two days. how to tell if it is hardware or software? From: Vick Khera To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 19 Jan 2015 16:31:14 -0000 Two days in a row last week, one of my FreeBSD/amd64 systems running 10.0 failed with the following panic: [panic: vm_radix_insert: key 46f is already present cpuid = 9 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe3fcf1b1820 ] [kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe3fcf1b18d0 panic() at panic+0x155/frame 0xfffffe3fcf1b1950 ] [vm_radix_insert() at vm_radix_insert+0x2ed/frame 0xfffffe3fcf1b19b0 vm_page_cache() at vm_page_cache+0x121/frame 0xfffffe3fcf1b19f0 ] [vm_pageout() at vm_pageout+0x8f7/frame 0xfffffe3fcf1b1a70 fork_exit() at fork_exit+0x9a/frame 0xfffffe3fcf1b1ab0 ] [fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe3fcf1b1ab0 --- trap 0, rip = 0, rsp = 0xfffffe3fcf1b1b70, rbp = 0 --- ] At this point the serial console was unresponsive, network unresponsive. After about 35 minutes, I just power cycled it. I figured if it was dumping the RAM I should let it finish, but it didn't seem to be doing so. The above was captured by my serial port terminal server. Obviously nothing made it to the log files on the server itself. After this panic, I disabled crash dumps on a hunch that it may have been stuck doing that. Almost exactly 25 hours later, the same panic occurred: [panic: vm_radix_insert: key 46f is already present cpuid = 0 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe3fcf1b1820 ] [kdb_backtrace() at kdb_backtrace+0x39/frame 0xfffffe3fcf1b18d0 panic() at panic+0x155/frame 0xfffffe3fcf1b1950 ] [vm_radix_insert() at vm_radix_insert+0x2ed/frame 0xfffffe3fcf1b19b0 vm_page_cache() at vm_page_cache+0x121/frame 0xfffffe3fcf1b19f0 ] [vm_pageout() at vm_pageout+0x8f7/frame 0xfffffe3fcf1b1a70 fork_exit() at fork_exit+0x9a/frame 0xfffffe3fcf1b1ab0 ] [fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe3fcf1b1ab0 --- trap 0, rip = 0, rsp = 0xfffffe3fcf1b1b70, rbp = 0 --- ] Again, the serial console was unresponsive. Since I knew there was not going to be a crash dump, I remote power-cyced it within a couple of minutes and it recovered quite nicely. I *LOVE* ZFS. :) This machine has 256GB of ECC RAM, and 2 6-core Intel Xeon CPUs with hyper-threading turned off. The uptime was 300 days when it crashed first on Thursday. The sole purpose of this machine is running Postgres 9.2 server. The other things it does is run NRPE for nagios monitoring and slony1 for database replication. The boot messages are below. This is running FreeBSD 10.0. I wanted to try the tips in Section 17.6 of the handbook, but the panic did not emit an instruction pointer for me. I run a custom kernel. I have plans to update this system to FreeBSD 10.1, but I wanted to see if there was any other debugging I could do here to figure out if it is a kernel bug or if it is a hardware fault starting to happen. The system in question is just about 1 year old. The rest of this email is just the dmesg.boot: Copyright (c) 1992-2014 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 10.0-RELEASE #1 r261210: Tue Jan 28 10:36:57 EST 2014 vivek@devbox.int.kcilink.com:/n/yertle1/sources/usr10/obj/n/yertle1/sources/usr10/src/sys/KCI64 amd64 FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610 CPU: Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz (2400.05-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x306e4 Family = 0x6 Model = 0x3e Stepping = 4 Features=0xbfebfbff Features2=0x7fbee3ff AMD Features=0x2c100800 AMD Features2=0x1 Standard Extended Features=0x281 TSC: P-state invariant, performance statistics real memory = 274894684160 (262160 MB) avail memory = 267173515264 (254796 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: < > FreeBSD/SMP: Multiprocessor System Detected: 12 CPUs FreeBSD/SMP: 2 package(s) x 6 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 2 cpu2 (AP): APIC ID: 4 cpu3 (AP): APIC ID: 6 cpu4 (AP): APIC ID: 8 cpu5 (AP): APIC ID: 10 cpu6 (AP): APIC ID: 32 cpu7 (AP): APIC ID: 34 cpu8 (AP): APIC ID: 36 cpu9 (AP): APIC ID: 38 cpu10 (AP): APIC ID: 40 cpu11 (AP): APIC ID: 42 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-47 on motherboard ioapic2 irqs 48-71 on motherboard random: initialized kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 cpu8: on acpi0 cpu9: on acpi0 cpu10: on acpi0 cpu11: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 26 at device 1.0 on pci0 pci1: on pcib1 pcib2: mem 0xdfc00000-0xdfc03fff irq 16 at device 0.0 on pci1 pci2: on pcib2 pcib3: irq 16 at device 8.0 on pci2 pci3: on pcib3 pcib4: irq 26 at device 1.1 on pci0 pci4: on pcib4 igb0: port 0x8020-0x803f mem 0xdfb20000-0xdfb3ffff,0xdfb44000-0xdfb47fff irq 27 at device 0.0 on pci4 igb0: Using MSIX interrupts with 9 vectors igb0: Ethernet address: 00:25:90:c3:be:14 igb0: Bound queue 0 to cpu 0 igb0: Bound queue 1 to cpu 1 igb0: Bound queue 2 to cpu 2 igb0: Bound queue 3 to cpu 3 igb0: Bound queue 4 to cpu 4 igb0: Bound queue 5 to cpu 5 igb0: Bound queue 6 to cpu 6 igb0: Bound queue 7 to cpu 7 igb1: port 0x8000-0x801f mem 0xdfb00000-0xdfb1ffff,0xdfb40000-0xdfb43fff irq 30 at device 0.1 on pci4 igb1: Using MSIX interrupts with 9 vectors igb1: Ethernet address: 00:25:90:c3:be:15 igb1: Bound queue 0 to cpu 8 igb1: Bound queue 1 to cpu 9 igb1: Bound queue 2 to cpu 10 igb1: Bound queue 3 to cpu 11 igb1: Bound queue 4 to cpu 0 igb1: Bound queue 5 to cpu 1 igb1: Bound queue 6 to cpu 2 igb1: Bound queue 7 to cpu 3 pcib5: irq 33 at device 2.0 on pci0 pci6: on pcib5 pcib6: irq 41 at device 3.0 on pci0 pci7: on pcib6 mps0: port 0x7000-0x70ff mem 0xdfa40000-0xdfa4ffff,0xdfa00000-0xdfa3ffff irq 40 at device 0.0 on pci7 mps0: Firmware: 17.00.01.00, Driver: 16.00.00.00-fbsd mps0: IOCCapabilities: 1285c pci0: at device 4.0 (no driver attached) pci0: at device 4.1 (no driver attached) pci0: at device 4.2 (no driver attached) pci0: at device 4.3 (no driver attached) pci0: at device 4.4 (no driver attached) pci0: at device 4.5 (no driver attached) pci0: at device 4.6 (no driver attached) pci0: at device 4.7 (no driver attached) pci0: at device 5.0 (no driver attached) pci0: at device 5.2 (no driver attached) pci0: at device 22.0 (no driver attached) pci0: at device 22.1 (no driver attached) ehci0: mem 0xdfd01000-0xdfd013ff irq 16 at device 26.0 on pci0 usbus0: EHCI version 1.0 usbus0 on ehci0 pcib7: irq 17 at device 28.0 on pci0 pci8: on pcib7 pcib8: irq 19 at device 28.7 on pci0 pci9: on pcib8 pcib9: at device 0.0 on pci9 pci10: on pcib9 pcib10: at device 0.0 on pci10 pci11: on pcib10 pcib11: at device 0.0 on pci11 pci12: on pcib11 vgapci0: mem 0xde000000-0xdeffffff,0xdf800000-0xdf803fff,0xdf000000-0xdf7fffff irq 19 at device 0.0 on pci12 vgapci0: Boot video device pcib12: at device 1.0 on pci10 pci13: on pcib12 ehci1: mem 0xdfd00000-0xdfd003ff irq 23 at device 29.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci1 pcib13: at device 30.0 on pci0 pci14: on pcib13 isab0: at device 31.0 on pci0 isa0: on isab0 pci0: at device 31.3 (no driver attached) pci0: at device 31.6 (no driver attached) pcib14: on acpi0 pci127: on pcib14 pci127: at device 8.0 (no driver attached) pci127: at device 9.0 (no driver attached) pci127: at device 10.0 (no driver attached) pci127: at device 10.1 (no driver attached) pci127: at device 10.2 (no driver attached) pci127: at device 10.3 (no driver attached) pci127: at device 11.0 (no driver attached) pci127: at device 11.3 (no driver attached) pci127: at device 12.0 (no driver attached) pci127: at device 12.1 (no driver attached) pci127: at device 12.2 (no driver attached) pci127: at device 13.0 (no driver attached) pci127: at device 13.1 (no driver attached) pci127: at device 13.2 (no driver attached) pci127: at device 14.0 (no driver attached) pci127: at device 14.1 (no driver attached) pci127: at device 15.0 (no driver attached) pci127: at device 15.1 (no driver attached) pci127: at device 15.2 (no driver attached) pci127: at device 15.3 (no driver attached) pci127: at device 15.4 (no driver attached) pci127: at device 15.5 (no driver attached) pci127: at device 16.0 (no driver attached) pci127: at device 16.1 (no driver attached) pci127: at device 16.2 (no driver attached) pci127: at device 16.3 (no driver attached) pci127: at device 16.4 (no driver attached) pci127: at device 16.5 (no driver attached) pci127: at device 16.6 (no driver attached) pci127: at device 16.7 (no driver attached) pci127: at device 19.0 (no driver attached) pci127: at device 19.1 (no driver attached) pci127: at device 19.4 (no driver attached) pci127: at device 19.5 (no driver attached) pci127: at device 22.0 (no driver attached) pci127: at device 22.1 (no driver attached) pci127: at device 22.2 (no driver attached) pcib15: on acpi0 pci128: on pcib15 pcib16: irq 50 at device 1.0 on pci128 pci129: on pcib16 pcib17: irq 57 at device 2.0 on pci128 pci130: on pcib17 pci128: at device 4.0 (no driver attached) pci128: at device 4.1 (no driver attached) pci128: at device 4.2 (no driver attached) pci128: at device 4.3 (no driver attached) pci128: at device 4.4 (no driver attached) pci128: at device 4.5 (no driver attached) pci128: at device 4.6 (no driver attached) pci128: at device 4.7 (no driver attached) pci128: at device 5.0 (no driver attached) pci128: at device 5.2 (no driver attached) pcib18: on acpi0 pci255: on pcib18 pci255: at device 8.0 (no driver attached) pci255: at device 9.0 (no driver attached) pci255: at device 10.0 (no driver attached) pci255: at device 10.1 (no driver attached) pci255: at device 10.2 (no driver attached) pci255: at device 10.3 (no driver attached) pci255: at device 11.0 (no driver attached) pci255: at device 11.3 (no driver attached) pci255: at device 12.0 (no driver attached) pci255: at device 12.1 (no driver attached) pci255: at device 12.2 (no driver attached) pci255: at device 13.0 (no driver attached) pci255: at device 13.1 (no driver attached) pci255: at device 13.2 (no driver attached) pci255: at device 14.0 (no driver attached) pci255: at device 14.1 (no driver attached) pci255: at device 15.0 (no driver attached) pci255: at device 15.1 (no driver attached) pci255: at device 15.2 (no driver attached) pci255: at device 15.3 (no driver attached) pci255: at device 15.4 (no driver attached) pci255: at device 15.5 (no driver attached) pci255: at device 16.0 (no driver attached) pci255: at device 16.1 (no driver attached) pci255: at device 16.2 (no driver attached) pci255: at device 16.3 (no driver attached) pci255: at device 16.4 (no driver attached) pci255: at device 16.5 (no driver attached) pci255: at device 16.6 (no driver attached) pci255: at device 16.7 (no driver attached) pci255: at device 19.0 (no driver attached) pci255: at device 19.1 (no driver attached) pci255: at device 19.4 (no driver attached) pci255: at device 19.5 (no driver attached) pci255: at device 22.0 (no driver attached) pci255: at device 22.1 (no driver attached) pci255: at device 22.2 (no driver attached) acpi_button0: on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: console (115200,n,8,1) ipmi0: port 0xca2,0xca3 on acpi0 ipmi0: KCS mode found at io 0xca2 on acpi orm0: at iomem 0xc0000-0xc7fff,0xce000-0xcefff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 acpi_throttle0: on cpu0 coretemp0: on cpu0 acpi_throttle1: on cpu1 acpi_throttle1: failed to attach P_CNT device_attach: acpi_throttle1 attach returned 6 coretemp1: on cpu1 acpi_throttle2: on cpu2 acpi_throttle2: failed to attach P_CNT device_attach: acpi_throttle2 attach returned 6 coretemp2: on cpu2 acpi_throttle3: on cpu3 acpi_throttle3: failed to attach P_CNT device_attach: acpi_throttle3 attach returned 6 coretemp3: on cpu3 acpi_throttle4: on cpu4 acpi_throttle4: failed to attach P_CNT device_attach: acpi_throttle4 attach returned 6 coretemp4: on cpu4 acpi_throttle5: on cpu5 acpi_throttle5: failed to attach P_CNT device_attach: acpi_throttle5 attach returned 6 coretemp5: on cpu5 acpi_throttle6: on cpu6 acpi_throttle6: failed to attach P_CNT device_attach: acpi_throttle6 attach returned 6 coretemp6: on cpu6 acpi_throttle7: on cpu7 acpi_throttle7: failed to attach P_CNT device_attach: acpi_throttle7 attach returned 6 coretemp7: on cpu7 acpi_throttle8: on cpu8 acpi_throttle8: failed to attach P_CNT device_attach: acpi_throttle8 attach returned 6 coretemp8: on cpu8 acpi_throttle9: on cpu9 acpi_throttle9: failed to attach P_CNT device_attach: acpi_throttle9 attach returned 6 coretemp9: on cpu9 acpi_throttle10: on cpu10 acpi_throttle10: failed to attach P_CNT device_attach: acpi_throttle10 attach returned 6 coretemp10: on cpu10 acpi_throttle11: on cpu11 acpi_throttle11: failed to attach P_CNT device_attach: acpi_throttle11 attach returned 6 coretemp11: on cpu11 Timecounters tick every 1.000 msec random: unblocking device. usbus0: 480Mbps High Speed USB v2.0 usbus1: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ipmi0: IPMI device rev. 1, firmware rev. 2.19, version 2.0 ipmi0: Number of channels 3 ipmi0: Attached watchdog da0 at mps0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-6 device da0: Serial Number BTTV335200Z3100FGN da0: 600.000MB/s transfers da0: Command Queueing enabled da0: 95396MB (195371568 512 byte sectors: 255H 63S/T 12161C) da1 at mps0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-6 device da1: Serial Number BTTV335106X6100FGN da1: 600.000MB/s transfers da1: Command Queueing enabled da1: 95396MB (195371568 512 byte sectors: 255H 63S/T 12161C) da2 at mps0 bus 0 scbus0 target 4 lun 0 da2: Fixed Direct Access SCSI-6 device da2: Serial Number Z0M0032V00009328Z4K4 da2: 600.000MB/s transfers da2: Command Queueing enabled da2: 572325MB (1172123568 512 byte sectors: 255H 63S/T 72961C) da3 at mps0 bus 0 scbus0 target 5 lun 0 da3: Fixed Direct Access SCSI-6 device da3: Serial Number Z0M0006B00009329D7CW da3: 600.000MB/s transfers da3: Command Queueing enabled da3: 572325MB (1172123568 512 byte sectors: 255H 63S/T 72961C) da4 at mps0 bus 0 scbus0 target 6 lun 0 da4: Fixed Direct Access SCSI-6 device da4: Serial Number Z0M002MT00009329D793 da4: 600.000MB/s transfers da4: Command Queueing enabled da4: 572325MB (1172123568 512 byte sectors: 255H 63S/T 72961C) da5 at mps0 bus 0 scbus0 target 7 lun 0 da5: Fixed Direct Access SCSI-6 device da5: Serial Number Z0M003L200009338Y1P0 da5: 600.000MB/s transfers da5: Command Queueing enabled da5: 572325MB (1172123568 512 byte sectors: 255H 63S/T 72961C) da6 at mps0 bus 0 scbus0 target 8 lun 0 da6: Fixed Direct Access SCSI-6 device da6: Serial Number BTWL33900172080KGN da6: 600.000MB/s transfers da6: Command Queueing enabled da6: 76319MB (156301488 512 byte sectors: 255H 63S/T 9729C) da7 at mps0 bus 0 scbus0 target 9 lun 0 da7: Fixed Direct Access SCSI-6 device da7: Serial Number BTWL339000A5080KGN da7: 600.000MB/s transfers da7: Command Queueing enabled da7: 76319MB (156301488 512 byte sectors: 255H 63S/T 9729C) GEOM_MIRROR: Device mirror/gmroot launched (2/2). GEOM_MIRROR: Device mirror/gmusr launched (2/2). GEOM_MIRROR: Device mirror/gmvar launched (1/2). GEOM_MIRROR: Device gmvar: rebuilding provider da6p5. SMP: AP CPU #10 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #8 Launched! Timecounter "TSC-low" frequency 1200025156 Hz quality 1000 Root mount waiting for: usbus1 usbus0 uhub1: 2 ports with 2 removable, self powered uhub0: 2 ports with 2 removable, self powered Root mount waiting for: usbus1 usbus0 ugen0.2: at usbus0 uhub2: on usbus0 ugen1.2: at usbus1 uhub3: on usbus1 Root mount waiting for: usbus1 usbus0 uhub2: 6 ports with 6 removable, self powered uhub3: 8 ports with 8 removable, self powered ugen0.3: at usbus0 ukbd0: on usbus0 kbd2 at ukbd0 Trying to mount root from ufs:/dev/mirror/gmroot [rw]... WARNING: / was not properly dismounted ZFS filesystem version: 5 ZFS storage pool version: features support (5000) igb0: link state changed to UP