Date: Wed, 15 Oct 2008 01:23:43 +0300 From: Panagiotis Christias <p.christias@noc.ntua.gr> To: freebsd-scsi@freebsd.org, freebsd-stable@freebsd.org Subject: FreeBSD 7-STABLE, isp(4), QLE2462: panic & deadlocks Message-ID: <20081014222343.GA8706@noc.ntua.gr>
next in thread | raw e-mail | index | archive | help
Hello everybody, we recently got three Dell PowerEdge servers equipped with Qlogic QLE2462 cards (dual FC 4Gbps ports) and an EMC CLARiiON CX3-40 SAN. The servers with the FC cards were successfully and extensively tested with an older SUN StorEdge T3 SAN. However, when we connect them to the CX3-40, create and mount a new partition and then do something as simple as "tar -C /san -xf ports.tgz" the system panics and deadlocks. We have tried several FreeBSD versions (6.3 i386/adm64, 7.0 i386/adm64, 7.1 i386/adm64 and lastly 7-STABLE i386 - we also tried the latest 8-CURRENT snapshot but it panicked too soon). The result is always the same; panic and deadlock. In case it helps, we have also tried all available connection speeds (4, 2 and 1Gbps), the QLE2462 cards have installed the latest firwmare version (4.03.10) and we have verified our hardware setup by testing it with Linux (CentOS 5.2), although we would try anything in order to keep the whole of our server base to FreeBSD (we have succeeded till now). Currently, we have installed the latest version of 7-STABLE (csuped today) along with a debug-enabled kernel and collected as much output as we can. See below. We are open to any suggestion and any kind of tests in order to help resolving the problem. Thanks! COLLECTED DATA: 1. uname -a: FreeBSD 7x86.noc.ntua.gr 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #0: Tue Oct 14 21:12:57 EEST 2008 root@7x86.noc.ntua.gr:/usr/obj/usr/src/sys/GENERIC-DEBUG i386 --------------------------------------------------------------- 2. Kernel configuration file /sys/i386/conf/GENERIC-DEBUG: include GENERIC ident GENERIC-DEBUG # force isp firmware to fully loaded # device ispfw options KDB options DDB options INVARIANTS options INVARIANT_SUPPORT options WITNESS options DEBUG_LOCKS options DEBUG_VFS_LOCKS options DIAGNOSTIC options CAMDEBUG options CAM_DEBUG_BUS=-1 options CAM_DEBUG_TARGET=-1 options CAM_DEBUG_LUN=-1 options CAM_DEBUG_FLAGS=CAM_DEBUG_INFO --------------------------------------------------------------- 3. dmesg output, also available at http://noc.ntua.gr/~christia/san/7-STABLE.dmesg.out : Copyright (c) 1992-2008 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.1-PRERELEASE #0: Tue Oct 14 21:12:57 EEST 2008 root@7x86.noc.ntua.gr:/usr/obj/usr/src/sys/GENERIC-DEBUG WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5405 @ 2.00GHz (1997.03-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x10676 Stepping = 6 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0xce33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,PDCM,DCA,<b19>> AMD Features=0x20100000<NX,LM> AMD Features2=0x1<LAHF> Cores per package: 4 real memory = 3484745728 (3323 MB) avail memory = 3405070336 (3247 MB) ACPI APIC Table: <DELL PE_SC3 > FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 WITNESS: spin lock cpuset not in order list WITNESS: spin lock intrcnt not in order list ioapic0: Changing APIC ID to 8 ioapic1: Changing APIC ID to 9 ioapic2: Changing APIC ID to 10 ioapic0 <Version 2.0> irqs 0-23 on motherboard ioapic1 <Version 2.0> irqs 160-183 on motherboard ioapic2 <Version 2.0> irqs 224-247 on motherboard kbd1 at kbdmux0 ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413) acpi0: <DELL PE_SC3> on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 pci0: <ACPI PCI bus> on pcib0 pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0 pci4: <ACPI PCI bus> on pcib1 pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci4 pci5: <ACPI PCI bus> on pcib2 pcib3: <ACPI PCI-PCI bridge> at device 0.0 on pci5 pci6: <ACPI PCI bus> on pcib3 pcib4: <PCI-PCI bridge> at device 0.0 on pci6 pci7: <PCI bus> on pcib4 bce0: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem 0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci7 miibus0: <MII bus> on bce0 brgphy0: <BCM5708C 10/100/1000baseTX PHY> PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bce0: Ethernet address: 00:1e:c9:e8:13:e9 bce0: [ITHREAD] bce0: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W (0x04000305); Flags( MFW MSI ) pcib5: <ACPI PCI-PCI bridge> at device 1.0 on pci5 pci8: <ACPI PCI bus> on pcib5 pcib6: <PCI-PCI bridge> at device 0.3 on pci4 pci9: <PCI bus> on pcib6 pcib7: <ACPI PCI-PCI bridge> at device 3.0 on pci0 pci1: <ACPI PCI bus> on pcib7 mfi0: <Dell PERC 6> port 0xec00-0xecff mem 0xfc780000-0xfc7bffff,0xfc740000-0xfc77ffff irq 16 at device 0.0 on pci1 mfi0: Megaraid SAS driver Ver 2.00 mfi0: 3299 (277344669s/0x0020/info) - Shutdown command received from host mfi0: 3300 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID 0060/1000/1f0c/1028) mfi0: 3301 (boot + 3s/0x0020/info) - Firmware version 1.11.82-0473 mfi0: 3302 (boot + 3s/0x0008/info) - Battery Present mfi0: 3303 (boot + 3s/0x0020/info) - Package version 6.0.3-0002 mfi0: 3304 (boot + 23s/0x0004/info) - Enclosure (SES) discovered on PD 20(c None/p0) mfi0: 3305 (boot + 23s/0x0002/info) - Unexpected sense: Encl PD 20, CDB: 12 00 00 00 00 00, Sense: 70 00 06 00 00 00 00 0a 00 00 00 00 29 00 00 00 00 00 mfi0: 3306 (boot + 24s/0x0002/info) - Inserted: Encl PD 20 mfi0: 3307 (boot + 24s/0x0002/info) - Inserted: PD 20(c None/p0) Info: enclPd=20, scsiType=d, portMap=09, sasAddr=5001e090d62bfb00,0000000000000000 mfi0: 3308 (boot + 24s/0x0002/info) - Inserted: PD 00(e0x20/s0) mfi0: 3309 (boot + 24s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=5000cca005b1bf89,0000000000000000 mfi0: 3310 (boot + 24s/0x0002/info) - Inserted: PD 01(e0x20/s1) mfi0: 3311 (boot + 24s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info: enclPd=20, scsiType=0, portMap=01, sasAddr=5000cca005b1427d,0000000000000000 mfi0: 3312 (boot + 24s/0x0002/info) - Inserted: PD 02(e0x20/s2) mfi0: 3313 (boot + 24s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info: enclPd=20, scsiType=0, portMap=02, sasAddr=5000cca005b13c61,0000000000000000 mfi0: 3314 (boot + 24s/0x0042/info) - Global Hot Spare created on PD 01(e0x20/s1) (global) mfi0: 3315 (boot + 24s/0x0002/info) - State change on PD 01(e0x20/s1) from UNCONFIGURED_GOOD(0) to HOT SPARE(2) mfi0: 3316 (boot + 68s/0x0008/info) - Battery temperature is normal mfi0: 3317 (boot + 68s/0x0008/info) - Battery started charging mfi0: 3318 (boot + 68s/0x0008/info) - Current capacity of the battery is above threshold mfi0: [ITHREAD] pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0 pci10: <ACPI PCI bus> on pcib8 isp0: <Qlogic ISP 2432 PCI FC-AL Adapter> port 0xdc00-0xdcff mem 0xfc5bc000-0xfc5bffff irq 16 at device 0.0 on pci10 firmware_get: failed to load firmware image isp_2400 isp0: [ITHREAD] isp0: Board Type 2422, Chip Revision 0x3, resident F/W Revision 4.3.1 isp1: <Qlogic ISP 2432 PCI FC-AL Adapter> port 0xd800-0xd8ff mem 0xfc5b8000-0xfc5bbfff irq 17 at device 0.1 on pci10 firmware_get: failed to load firmware image isp_2400 isp1: [ITHREAD] isp1: Board Type 2422, Chip Revision 0x3, resident F/W Revision 4.3.1 pcib9: <PCI-PCI bridge> at device 5.0 on pci0 pci11: <PCI bus> on pcib9 pcib10: <ACPI PCI-PCI bridge> at device 6.0 on pci0 pci12: <ACPI PCI bus> on pcib10 pcib11: <ACPI PCI-PCI bridge> at device 0.0 on pci12 pci13: <ACPI PCI bus> on pcib11 pcib12: <ACPI PCI-PCI bridge> at device 0.2 on pci12 pci14: <ACPI PCI bus> on pcib12 pcib13: <PCI-PCI bridge> at device 7.0 on pci0 pci15: <PCI bus> on pcib13 pcib14: <ACPI PCI-PCI bridge> at device 28.0 on pci0 pci2: <ACPI PCI bus> on pcib14 pcib15: <PCI-PCI bridge> at device 0.0 on pci2 pci3: <PCI bus> on pcib15 bce1: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem 0xf8000000-0xf9ffffff irq 16 at device 0.0 on pci3 miibus1: <MII bus> on bce1 brgphy1: <BCM5708C 10/100/1000baseTX PHY> PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bce1: Ethernet address: 00:1e:c9:e8:13:e7 bce1: [ITHREAD] bce1: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W (0x04000305); Flags( MFW MSI ) uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port 0xbce0-0xbcff irq 21 at device 29.0 on pci0 uhci0: [GIANT-LOCKED] uhci0: [ITHREAD] usb0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0 usb0: USB revision 1.0 uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0 uhub0: 2 ports with 2 removable, self powered uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port 0xbcc0-0xbcdf irq 20 at device 29.1 on pci0 uhci1: [GIANT-LOCKED] uhci1: [ITHREAD] usb1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1 usb1: USB revision 1.0 uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1 uhub1: 2 ports with 2 removable, self powered uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port 0xbca0-0xbcbf irq 21 at device 29.2 on pci0 uhci2: [GIANT-LOCKED] uhci2: [ITHREAD] usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2 usb2: USB revision 1.0 uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2 uhub2: 2 ports with 2 removable, self powered uhci3: <Intel 631XESB/632XESB/3100 USB controller USB-4> port 0xbc80-0xbc9f irq 20 at device 29.3 on pci0 uhci3: [GIANT-LOCKED] uhci3: [ITHREAD] usb3: <Intel 631XESB/632XESB/3100 USB controller USB-4> on uhci3 usb3: USB revision 1.0 uhub3: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3 uhub3: 2 ports with 2 removable, self powered ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xfc800000-0xfc8003ff irq 21 at device 29.7 on pci0 ehci0: [GIANT-LOCKED] ehci0: [ITHREAD] usb4: EHCI version 1.0 usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3 usb4: <Intel 63XXESB USB 2.0 controller> on ehci0 usb4: USB revision 2.0 uhub4: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4 uhub4: 8 ports with 8 removable, self powered uhub5: <vendor 0x413c product 0xa001, class 9/0, rev 2.00/0.00, addr 2> on uhub4 uhub5: multiple transaction translators uhub5: 2 ports with 2 removable, self powered ehci_device_intr_done: not done, ex=0xc699e800 ukbd0: <Dell DRAC5, class 0/0, rev 1.10/0.00, addr 3> on uhub5 ehci_device_intr_done: not done, ex=0xc699e000 kbd2 at ukbd0 ums0: <Dell DRAC5, class 0/0, rev 1.10/0.00, addr 3> on uhub5 ums0: X report 0x0002 not supported device_attach: ums0 attach returned 6 umass0: <DELL INC. DRAC5 VIRTUAL MEDIA, class 0/0, rev 2.00/0.00, addr 4> on uhub5 umass1: <DELL INC. DRAC5 VIRTUAL MEDIA, class 0/0, rev 2.00/0.00, addr 4> on uhub5 uhub6: <vendor 0x04b4 product 0x6560, class 9/0, rev 2.00/0.0b, addr 5> on uhub4 uhub6: multiple transaction translators uhub6: 4 ports with 4 removable, self powered ehci_device_intr_done: not done, ex=0xc69a9400 pcib16: <ACPI PCI-PCI bridge> at device 30.0 on pci0 pci16: <ACPI PCI bus> on pcib16 vgapci0: <VGA-compatible display> port 0xcc00-0xccff mem 0xd8000000-0xdfffffff,0xfc2d0000-0xfc2dffff irq 19 at device 13.0 on pci16 isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Intel 63XXESB2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0 ata0: <ATA channel 0> on atapci0 ata0: [ITHREAD] ata1: <ATA channel 1> on atapci0 ata1: [ITHREAD] fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A, console sio0: [FILTER] sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0 sio1: type 16550A sio1: [FILTER] cpu0: <ACPI CPU> on acpi0 p4tcc0: <CPU Frequency Thermal Control> on cpu0 cpu1: <ACPI CPU> on acpi0 p4tcc1: <CPU Frequency Thermal Control> on cpu1 cpu2: <ACPI CPU> on acpi0 p4tcc2: <CPU Frequency Thermal Control> on cpu2 cpu3: <ACPI CPU> on acpi0 p4tcc3: <CPU Frequency Thermal Control> on cpu3 cpu4: <ACPI CPU> on acpi0 p4tcc4: <CPU Frequency Thermal Control> on cpu4 cpu5: <ACPI CPU> on acpi0 p4tcc5: <CPU Frequency Thermal Control> on cpu5 cpu6: <ACPI CPU> on acpi0 p4tcc6: <CPU Frequency Thermal Control> on cpu6 cpu7: <ACPI CPU> on acpi0 p4tcc7: <CPU Frequency Thermal Control> on cpu7 fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 pmtimer0 on isa0 orm0: <ISA Option ROMs> at iomem 0xc0000-0xc8fff,0xc9000-0xc9fff,0xd4800-0xd57ff,0xec000-0xeffff pnpid ORM0000 on isa0 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] ppc0: parallel port not found. sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x100> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec bce0: link state changed to DOWN bce1: link state changed to DOWN acd0: DVDROM <TEAC DVD-ROM DV28EV/D.AE> at ata0-master UDMA33 mfi0: 3319 (277344766s/0x0020/info) - Time established as 10/15/08 0:12:46; (71 seconds since power on) Expensive timeout(9) function: 0xc06f9de0(0xc0c52520) 0.113017715 s mfid0: <MFI Logical Disk> on mfi0 mfid0: 69376MB (142082048 sectors) RAID volume '' is optimal (probe376:isp0:0:376:0): cam completion status 0x4a (probe376:isp0:0:376:0): cam completion status 0x4a (probe395:isp0:0:395:0): cam completion status 0x4a (probe395:isp0:0:395:0): cam completion status 0x4a (probe400:isp0:0:400:0): cam completion status 0x4a (probe400:isp0:0:400:0): cam completion status 0x4a (probe411:isp0:0:411:0): cam completion status 0x4a (probe411:isp0:0:411:0): cam completion status 0x4a (probe423:isp0:0:423:0): cam completion status 0x4a (probe423:isp0:0:423:0): cam completion status 0x4a (probe437:isp0:0:437:0): cam completion status 0x4a (probe437:isp0:0:437:0): cam completion status 0x4a (probe449:isp0:0:449:0): cam completion status 0x4a (probe449:isp0:0:449:0): cam completion status 0x4a (probe459:isp0:0:459:0): cam completion status 0x4a (probe459:isp0:0:459:0): cam completion status 0x4a (probe464:isp0:0:464:0): cam completion status 0x4a (probe464:isp0:0:464:0): cam completion status 0x4a (probe471:isp0:0:471:0): cam completion status 0x4a (probe471:isp0:0:471:0): cam completion status 0x4a (probe478:isp0:0:478:0): cam completion status 0x4a (probe478:isp0:0:478:0): cam completion status 0x4a (probe488:isp0:0:488:0): cam completion status 0x4a (probe488:isp0:0:488:0): cam completion status 0x4a (probe504:isp0:0:504:0): cam completion status 0x4a (probe504:isp0:0:504:0): cam completion status 0x4a (probe505:isp0:0:505:0): cam completion status 0x4a (probe505:isp0:0:505:0): cam completion status 0x4a (probe506:isp0:0:506:0): cam completion status 0x4a (probe506:isp0:0:506:0): cam completion status 0x4a (probe507:isp0:0:507:0): cam completion status 0x4a (probe507:isp0:0:507:0): cam completion status 0x4a (probe508:isp0:0:508:0): cam completion status 0x4a (probe508:isp0:0:508:0): cam completion status 0x4a (probe509:isp0:0:509:0): cam completion status 0x4a (probe509:isp0:0:509:0): cam completion status 0x4a (probe510:isp0:0:510:0): cam completion status 0x4a (probe510:isp0:0:510:0): cam completion status 0x4a (probe511:isp0:0:511:0): cam completion status 0x4a (probe511:isp0:0:511:0): cam completion status 0x4a (probe1:isp0:0:1:0): cam completion status 0x4a (probe1:isp0:0:1:0): cam completion status 0x4a (probe2:isp0:0:2:0): cam completion status 0x4a (probe2:isp0:0:2:0): cam completion status 0x4a (probe3:isp0:0:3:0): cam completion status 0x4a (probe0:isp0:0:0:0): cam completion status 0xcc (probe3:isp0:0:3:0): cam completion status 0x4a (probe4:isp0:0:4:0): cam completion status 0x4a (probe4:isp0:0:4:0): cam completion status 0x4a (probe5:isp0:0:5:0): cam completion status 0x4a (probe5:isp0:0:5:0): cam completion status 0x4a (probe7:isp0:0:7:0): cam completion status 0x4a (probe7:isp0:0:7:0): cam completion status 0x4a (probe8:isp0:0:8:0): cam completion status 0x4a (probe8:isp0:0:8:0): cam completion status 0x4a (probe9:isp0:0:9:0): cam completion status 0x4a (probe9:isp0:0:9:0): cam completion status 0x4a (probe10:isp0:0:10:0): cam completion status 0x4a (probe10:isp0:0:10:0): cam completion status 0x4a (probe11:isp0:0:11:0): cam completion status 0x4a (probe11:isp0:0:11:0): cam completion status 0x4a (probe12:isp0:0:12:0): cam completion status 0x4a (probe12:isp0:0:12:0): cam completion status 0x4a (probe13:isp0:0:13:0): cam completion status 0x4a (probe13:isp0:0:13:0): cam completion status 0x4a (probe14:isp0:0:14:0): cam completion status 0x4a (probe14:isp0:0:14:0): cam completion status 0x4a (probe15:isp0:0:15:0): cam completion status 0x4a (probe15:isp0:0:15:0): cam completion status 0x4a (probe16:isp0:0:16:0): cam completion status 0x4a (probe16:isp0:0:16:0): cam completion status 0x4a (probe17:isp0:0:17:0): cam completion status 0x4a (probe17:isp0:0:17:0): cam completion status 0x4a (probe18:isp0:0:18:0): cam completion status 0x4a (probe18:isp0:0:18:0): cam completion status 0x4a (probe19:isp0:0:19:0): cam completion status 0x4a (probe19:isp0:0:19:0): cam completion status 0x4a (probe20:isp0:0:20:0): cam completion status 0x4a (probe20:isp0:0:20:0): cam completion status 0x4a (probe21:isp0:0:21:0): cam completion status 0x4a (probe21:isp0:0:21:0): cam completion status 0x4a (probe22:isp0:0:22:0): cam completion status 0x4a (probe22:isp0:0:22:0): cam completion status 0x4a (probe23:isp0:0:23:0): cam completion status 0x4a (probe23:isp0:0:23:0): cam completion status 0x4a (probe24:isp0:0:24:0): cam completion status 0x4a (probe24:isp0:0:24:0): cam completion status 0x4a (probe26:isp0:0:26:0): cam completion status 0x4a (probe26:isp0:0:26:0): cam completion status 0x4a (probe27:isp0:0:27:0): cam completion status 0x4a (probe27:isp0:0:27:0): cam completion status 0x4a (probe28:isp0:0:28:0): cam completion status 0x4a (probe28:isp0:0:28:0): cam completion status 0x4a (probe29:isp0:0:29:0): cam completion status 0x4a (probe29:isp0:0:29:0): cam completion status 0x4a (probe30:isp0:0:30:0): cam completion status 0x4a (probe30:isp0:0:30:0): cam completion status 0x4a (probe31:isp0:0:31:0): cam completion status 0x4a (probe31:isp0:0:31:0): cam completion status 0x4a (probe32:isp0:0:32:0): cam completion status 0x4a (probe32:isp0:0:32:0): cam completion status 0x4a (probe33:isp0:0:33:0): cam completion status 0x4a (probe33:isp0:0:33:0): cam completion status 0x4a (probe34:isp0:0:34:0): cam completion status 0x4a (probe34:isp0:0:34:0): cam completion status 0x4a (probe35:isp0:0:35:0): cam completion status 0x4a (probe35:isp0:0:35:0): cam completion status 0x4a (probe36:isp0:0:36:0): cam completion status 0x4a (probe36:isp0:0:36:0): cam completion status 0x4a (probe37:isp0:0:37:0): cam completion status 0x4a (probe37:isp0:0:37:0): cam completion status 0x4a (probe38:isp0:0:38:0): cam completion status 0x4a (probe38:isp0:0:38:0): cam completion status 0x4a (probe39:isp0:0:39:0): cam completion status 0x4a (probe39:isp0:0:39:0): cam completion status 0x4a (probe40:isp0:0:40:0): cam completion status 0x4a (probe40:isp0:0:40:0): cam completion status 0x4a (probe41:isp0:0:41:0): cam completion status 0x4a (probe41:isp0:0:41:0): cam completion status 0x4a (probe42:isp0:0:42:0): cam completion status 0x4a (probe42:isp0:0:42:0): cam completion status 0x4a (probe43:isp0:0:43:0): cam completion status 0x4a (probe43:isp0:0:43:0): cam completion status 0x4a (probe44:isp0:0:44:0): cam completion status 0x4a (probe44:isp0:0:44:0): cam completion status 0x4a (probe45:isp0:0:45:0): cam completion status 0x4a (probe45:isp0:0:45:0): cam completion status 0x4a (probe46:isp0:0:46:0): cam completion status 0x4a (probe46:isp0:0:46:0): cam completion status 0x4a (probe47:isp0:0:47:0): cam completion status 0x4a (probe47:isp0:0:47:0): cam completion status 0x4a (probe48:isp0:0:48:0): cam completion status 0x4a (probe48:isp0:0:48:0): cam completion status 0x4a (probe49:isp0:0:49:0): cam completion status 0x4a (probe49:isp0:0:49:0): cam completion status 0x4a (probe50:isp0:0:50:0): cam completion status 0x4a (probe50:isp0:0:50:0): cam completion status 0x4a (probe51:isp0:0:51:0): cam completion status 0x4a (probe51:isp0:0:51:0): cam completion status 0x4a (probe52:isp0:0:52:0): cam completion status 0x4a (probe52:isp0:0:52:0): cam completion status 0x4a (probe53:isp0:0:53:0): cam completion status 0x4a (probe53:isp0:0:53:0): cam completion status 0x4a (probe54:isp0:0:54:0): cam completion status 0x4a (probe54:isp0:0:54:0): cam completion status 0x4a (probe55:isp0:0:55:0): cam completion status 0x4a (probe55:isp0:0:55:0): cam completion status 0x4a (probe56:isp0:0:56:0): cam completion status 0x4a (probe56:isp0:0:56:0): cam completion status 0x4a (probe57:isp0:0:57:0): cam completion status 0x4a (probe57:isp0:0:57:0): cam completion status 0x4a (probe58:isp0:0:58:0): cam completion status 0x4a (probe58:isp0:0:58:0): cam completion status 0x4a (probe59:isp0:0:59:0): cam completion status 0x4a (probe59:isp0:0:59:0): cam completion status 0x4a (probe60:isp0:0:60:0): cam completion status 0x4a (probe60:isp0:0:60:0): cam completion status 0x4a (probe61:isp0:0:61:0): cam completion status 0x4a (probe61:isp0:0:61:0): cam completion status 0x4a (probe63:isp0:0:63:0): cam completion status 0x4a (probe63:isp0:0:63:0): cam completion status 0x4a (probe64:isp0:0:64:0): cam completion status 0x4a (probe64:isp0:0:64:0): cam completion status 0x4a (probe65:isp0:0:65:0): cam completion status 0x4a (probe65:isp0:0:65:0): cam completion status 0x4a (probe66:isp0:0:66:0): cam completion status 0x4a (probe66:isp0:0:66:0): cam completion status 0x4a (probe67:isp0:0:67:0): cam completion status 0x4a (probe67:isp0:0:67:0): cam completion status 0x4a (probe68:isp0:0:68:0): cam completion status 0x4a (probe68:isp0:0:68:0): cam completion status 0x4a (probe69:isp0:0:69:0): cam completion status 0x4a (probe69:isp0:0:69:0): cam completion status 0x4a (probe70:isp0:0:70:0): cam completion status 0x4a (probe70:isp0:0:70:0): cam completion status 0x4a (probe71:isp0:0:71:0): cam completion status 0x4a (probe71:isp0:0:71:0): cam completion status 0x4a (probe72:isp0:0:72:0): cam completion status 0x4a (probe72:isp0:0:72:0): cam completion status 0x4a (probe73:isp0:0:73:0): cam completion status 0x4a (probe73:isp0:0:73:0): cam completion status 0x4a (probe75:isp0:0:75:0): cam completion status 0x4a (probe75:isp0:0:75:0): cam completion status 0x4a (probe76:isp0:0:76:0): cam completion status 0x4a (probe76:isp0:0:76:0): cam completion status 0x4a (probe77:isp0:0:77:0): cam completion status 0x4a (probe77:isp0:0:77:0): cam completion status 0x4a (probe78:isp0:0:78:0): cam completion status 0x4a (probe78:isp0:0:78:0): cam completion status 0x4a (probe79:isp0:0:79:0): cam completion status 0x4a (probe79:isp0:0:79:0): cam completion status 0x4a (probe81:isp0:0:81:0): cam completion status 0x4a (probe81:isp0:0:81:0): cam completion status 0x4a (probe82:isp0:0:82:0): cam completion status 0x4a (probe82:isp0:0:82:0): cam completion status 0x4a (probe83:isp0:0:83:0): cam completion status 0x4a (probe83:isp0:0:83:0): cam completion status 0x4a (probe84:isp0:0:84:0): cam completion status 0x4a (probe84:isp0:0:84:0): cam completion status 0x4a (probe85:isp0:0:85:0): cam completion status 0x4a (probe85:isp0:0:85:0): cam completion status 0x4a (probe86:isp0:0:86:0): cam completion status 0x4a (probe86:isp0:0:86:0): cam completion status 0x4a (probe87:isp0:0:87:0): cam completion status 0x4a (probe87:isp0:0:87:0): cam completion status 0x4a (probe88:isp0:0:88:0): cam completion status 0x4a (probe88:isp0:0:88:0): cam completion status 0x4a (probe89:isp0:0:89:0): cam completion status 0x4a (probe89:isp0:0:89:0): cam completion status 0x4a (probe90:isp0:0:90:0): cam completion status 0x4a (probe90:isp0:0:90:0): cam completion status 0x4a (probe91:isp0:0:91:0): cam completion status 0x4a (probe91:isp0:0:91:0): cam completion status 0x4a (probe92:isp0:0:92:0): cam completion status 0x4a (probe92:isp0:0:92:0): cam completion status 0x4a (probe93:isp0:0:93:0): cam completion status 0x4a (probe93:isp0:0:93:0): cam completion status 0x4a (probe94:isp0:0:94:0): cam completion status 0x4a (probe94:isp0:0:94:0): cam completion status 0x4a (probe95:isp0:0:95:0): cam completion status 0x4a (probe95:isp0:0:95:0): cam completion status 0x4a (probe96:isp0:0:96:0): cam completion status 0x4a (probe96:isp0:0:96:0): cam completion status 0x4a (probe97:isp0:0:97:0): cam completion status 0x4a (probe97:isp0:0:97:0): cam completion status 0x4a (probe98:isp0:0:98:0): cam completion status 0x4a (probe98:isp0:0:98:0): cam completion status 0x4a (probe99:isp0:0:99:0): cam completion status 0x4a (probe99:isp0:0:99:0): cam completion status 0x4a (probe100:isp0:0:100:0): cam completion status 0x4a (probe100:isp0:0:100:0): cam completion status 0x4a (probe101:isp0:0:101:0): cam completion status 0x4a (probe101:isp0:0:101:0): cam completion status 0x4a (probe102:isp0:0:102:0): cam completion status 0x4a (probe102:isp0:0:102:0): cam completion status 0x4a (probe103:isp0:0:103:0): cam completion status 0x4a (probe103:isp0:0:103:0): cam completion status 0x4a (probe104:isp0:0:104:0): cam completion status 0x4a (probe104:isp0:0:104:0): cam completion status 0x4a (probe105:isp0:0:105:0): cam completion status 0x4a (probe105:isp0:0:105:0): cam completion status 0x4a (probe106:isp0:0:106:0): cam completion status 0x4a (probe106:isp0:0:106:0): cam completion status 0x4a (probe107:isp0:0:107:0): cam completion status 0x4a (probe107:isp0:0:107:0): cam completion status 0x4a (probe108:isp0:0:108:0): cam completion status 0x4a (probe108:isp0:0:108:0): cam completion status 0x4a (probe110:isp0:0:110:0): cam completion status 0x4a (probe110:isp0:0:110:0): cam completion status 0x4a (probe111:isp0:0:111:0): cam completion status 0x4a (probe111:isp0:0:111:0): cam completion status 0x4a (probe112:isp0:0:112:0): cam completion status 0x4a (probe112:isp0:0:112:0): cam completion status 0x4a (probe113:isp0:0:113:0): cam completion status 0x4a (probe113:isp0:0:113:0): cam completion status 0x4a (probe114:isp0:0:114:0): cam completion status 0x4a (probe114:isp0:0:114:0): cam completion status 0x4a (probe115:isp0:0:115:0): cam completion status 0x4a (probe115:isp0:0:115:0): cam completion status 0x4a (probe116:isp0:0:116:0): cam completion status 0x4a (probe116:isp0:0:116:0): cam completion status 0x4a (probe117:isp0:0:117:0): cam completion status 0x4a (probe117:isp0:0:117:0): cam completion status 0x4a (probe118:isp0:0:118:0): cam completion status 0x4a (probe118:isp0:0:118:0): cam completion status 0x4a (probe119:isp0:0:119:0): cam completion status 0x4a (probe119:isp0:0:119:0): cam completion status 0x4a (probe120:isp0:0:120:0): cam completion status 0x4a (probe120:isp0:0:120:0): cam completion status 0x4a (probe121:isp0:0:121:0): cam completion status 0x4a (probe121:isp0:0:121:0): cam completion status 0x4a (probe122:isp0:0:122:0): cam completion status 0x4a (probe122:isp0:0:122:0): cam completion status 0x4a (probe123:isp0:0:123:0): cam completion status 0x4a (probe123:isp0:0:123:0): cam completion status 0x4a (probe124:isp0:0:124:0): cam completion status 0x4a (probe124:isp0:0:124:0): cam completion status 0x4a (probe125:isp0:0:125:0): cam completion status 0x4a (probe125:isp0:0:125:0): cam completion status 0x4a (probe126:isp0:0:126:0): cam completion status 0x4a (probe126:isp0:0:126:0): cam completion status 0x4a (probe127:isp0:0:127:0): cam completion status 0x4a (probe127:isp0:0:127:0): cam completion status 0x4a (probe128:isp0:0:128:0): cam completion status 0x4a (probe128:isp0:0:128:0): cam completion status 0x4a (probe129:isp0:0:129:0): cam completion status 0x4a (probe129:isp0:0:129:0): cam completion status 0x4a (probe130:isp0:0:130:0): cam completion status 0x4a (probe130:isp0:0:130:0): cam completion status 0x4a (probe132:isp0:0:132:0): cam completion status 0x4a (probe132:isp0:0:132:0): cam completion status 0x4a (probe133:isp0:0:133:0): cam completion status 0x4a (probe133:isp0:0:133:0): cam completion status 0x4a (probe134:isp0:0:134:0): cam completion status 0x4a (probe134:isp0:0:134:0): cam completion status 0x4a (probe135:isp0:0:135:0): cam completion status 0x4a (probe135:isp0:0:135:0): cam completion status 0x4a (probe136:isp0:0:136:0): cam completion status 0x4a (probe136:isp0:0:136:0): cam completion status 0x4a (probe137:isp0:0:137:0): cam completion status 0x4a (probe137:isp0:0:137:0): cam completion status 0x4a (probe138:isp0:0:138:0): cam completion status 0x4a (probe138:isp0:0:138:0): cam completion status 0x4a (probe139:isp0:0:139:0): cam completion status 0x4a (probe139:isp0:0:139:0): cam completion status 0x4a (probe140:isp0:0:140:0): cam completion status 0x4a (probe140:isp0:0:140:0): cam completion status 0x4a (probe141:isp0:0:141:0): cam completion status 0x4a (probe141:isp0:0:141:0): cam completion status 0x4a (probe142:isp0:0:142:0): cam completion status 0x4a (probe142:isp0:0:142:0): cam completion status 0x4a (probe143:isp0:0:143:0): cam completion status 0x4a (probe143:isp0:0:143:0): cam completion status 0x4a (probe144:isp0:0:144:0): cam completion status 0x4a (probe144:isp0:0:144:0): cam completion status 0x4a (probe145:isp0:0:145:0): cam completion status 0x4a (probe145:isp0:0:145:0): cam completion status 0x4a (probe146:isp0:0:146:0): cam completion status 0x4a (probe146:isp0:0:146:0): cam completion status 0x4a (probe148:isp0:0:148:0): cam completion status 0x4a (probe148:isp0:0:148:0): cam completion status 0x4a (probe149:isp0:0:149:0): cam completion status 0x4a (probe149:isp0:0:149:0): cam completion status 0x4a (probe150:isp0:0:150:0): cam completion status 0x4a (probe150:isp0:0:150:0): cam completion status 0x4a (probe151:isp0:0:151:0): cam completion status 0x4a (probe151:isp0:0:151:0): cam completion status 0x4a (probe152:isp0:0:152:0): cam completion status 0x4a (probe152:isp0:0:152:0): cam completion status 0x4a (probe153:isp0:0:153:0): cam completion status 0x4a (probe153:isp0:0:153:0): cam completion status 0x4a (probe154:isp0:0:154:0): cam completion status 0x4a (probe154:isp0:0:154:0): cam completion status 0x4a (probe155:isp0:0:155:0): cam completion status 0x4a (probe155:isp0:0:155:0): cam completion status 0x4a (probe156:isp0:0:156:0): cam completion status 0x4a (probe156:isp0:0:156:0): cam completion status 0x4a (probe157:isp0:0:157:0): cam completion status 0x4a (probe157:isp0:0:157:0): cam completion status 0x4a (probe158:isp0:0:158:0): cam completion status 0x4a (probe158:isp0:0:158:0): cam completion status 0x4a (probe159:isp0:0:159:0): cam completion status 0x4a (probe159:isp0:0:159:0): cam completion status 0x4a (probe160:isp0:0:160:0): cam completion status 0x4a (probe160:isp0:0:160:0): cam completion status 0x4a (probe161:isp0:0:161:0): cam completion status 0x4a (probe161:isp0:0:161:0): cam completion status 0x4a (probe162:isp0:0:162:0): cam completion status 0x4a (probe162:isp0:0:162:0): cam completion status 0x4a (probe163:isp0:0:163:0): cam completion status 0x4a (probe163:isp0:0:163:0): cam completion status 0x4a (probe164:isp0:0:164:0): cam completion status 0x4a (probe164:isp0:0:164:0): cam completion status 0x4a (probe165:isp0:0:165:0): cam completion status 0x4a (probe165:isp0:0:165:0): cam completion status 0x4a (probe166:isp0:0:166:0): cam completion status 0x4a (probe166:isp0:0:166:0): cam completion status 0x4a (probe167:isp0:0:167:0): cam completion status 0x4a (probe167:isp0:0:167:0): cam completion status 0x4a (probe168:isp0:0:168:0): cam completion status 0x4a (probe168:isp0:0:168:0): cam completion status 0x4a (probe169:isp0:0:169:0): cam completion status 0x4a (probe169:isp0:0:169:0): cam completion status 0x4a (probe170:isp0:0:170:0): cam completion status 0x4a (probe170:isp0:0:170:0): cam completion status 0x4a (probe171:isp0:0:171:0): cam completion status 0x4a (probe171:isp0:0:171:0): cam completion status 0x4a (probe172:isp0:0:172:0): cam completion status 0x4a (probe172:isp0:0:172:0): cam completion status 0x4a (probe173:isp0:0:173:0): cam completion status 0x4a (probe173:isp0:0:173:0): cam completion status 0x4a (probe174:isp0:0:174:0): cam completion status 0x4a (probe174:isp0:0:174:0): cam completion status 0x4a (probe175:isp0:0:175:0): cam completion status 0x4a (probe175:isp0:0:175:0): cam completion status 0x4a (probe176:isp0:0:176:0): cam completion status 0x4a (probe176:isp0:0:176:0): cam completion status 0x4a (probe177:isp0:0:177:0): cam completion status 0x4a (probe177:isp0:0:177:0): cam completion status 0x4a (probe178:isp0:0:178:0): cam completion status 0x4a (probe178:isp0:0:178:0): cam completion status 0x4a (probe179:isp0:0:179:0): cam completion status 0x4a (probe179:isp0:0:179:0): cam completion status 0x4a (probe180:isp0:0:180:0): cam completion status 0x4a (probe180:isp0:0:180:0): cam completion status 0x4a (probe181:isp0:0:181:0): cam completion status 0x4a (probe181:isp0:0:181:0): cam completion status 0x4a (probe182:isp0:0:182:0): cam completion status 0x4a (probe182:isp0:0:182:0): cam completion status 0x4a (probe183:isp0:0:183:0): cam completion status 0x4a (probe183:isp0:0:183:0): cam completion status 0x4a (probe184:isp0:0:184:0): cam completion status 0x4a (probe184:isp0:0:184:0): cam completion status 0x4a (probe185:isp0:0:185:0): cam completion status 0x4a (probe185:isp0:0:185:0): cam completion status 0x4a (probe186:isp0:0:186:0): cam completion status 0x4a (probe186:isp0:0:186:0): cam completion status 0x4a (probe187:isp0:0:187:0): cam completion status 0x4a (probe187:isp0:0:187:0): cam completion status 0x4a (probe188:isp0:0:188:0): cam completion status 0x4a (probe188:isp0:0:188:0): cam completion status 0x4a (probe189:isp0:0:189:0): cam completion status 0x4a (probe189:isp0:0:189:0): cam completion status 0x4a (probe190:isp0:0:190:0): cam completion status 0x4a (probe190:isp0:0:190:0): cam completion status 0x4a (probe191:isp0:0:191:0): cam completion status 0x4a (probe191:isp0:0:191:0): cam completion status 0x4a (probe192:isp0:0:192:0): cam completion status 0x4a (probe192:isp0:0:192:0): cam completion status 0x4a (probe193:isp0:0:193:0): cam completion status 0x4a (probe193:isp0:0:193:0): cam completion status 0x4a (probe194:isp0:0:194:0): cam completion status 0x4a (probe194:isp0:0:194:0): cam completion status 0x4a (probe195:isp0:0:195:0): cam completion status 0x4a (probe195:isp0:0:195:0): cam completion status 0x4a (probe196:isp0:0:196:0): cam completion status 0x4a (probe196:isp0:0:196:0): cam completion status 0x4a (probe197:isp0:0:197:0): cam completion status 0x4a (probe197:isp0:0:197:0): cam completion status 0x4a (probe198:isp0:0:198:0): cam completion status 0x4a (probe198:isp0:0:198:0): cam completion status 0x4a (probe199:isp0:0:199:0): cam completion status 0x4a (probe199:isp0:0:199:0): cam completion status 0x4a (probe200:isp0:0:200:0): cam completion status 0x4a (probe200:isp0:0:200:0): cam completion status 0x4a (probe201:isp0:0:201:0): cam completion status 0x4a (probe201:isp0:0:201:0): cam completion status 0x4a (probe202:isp0:0:202:0): cam completion status 0x4a (probe202:isp0:0:202:0): cam completion status 0x4a (probe203:isp0:0:203:0): cam completion status 0x4a (probe203:isp0:0:203:0): cam completion status 0x4a (probe204:isp0:0:204:0): cam completion status 0x4a (probe204:isp0:0:204:0): cam completion status 0x4a (probe205:isp0:0:205:0): cam completion status 0x4a (probe205:isp0:0:205:0): cam completion status 0x4a (probe207:isp0:0:207:0): cam completion status 0x4a (probe207:isp0:0:207:0): cam completion status 0x4a (probe208:isp0:0:208:0): cam completion status 0x4a (probe208:isp0:0:208:0): cam completion status 0x4a (probe209:isp0:0:209:0): cam completion status 0x4a (probe209:isp0:0:209:0): cam completion status 0x4a (probe210:isp0:0:210:0): cam completion status 0x4a (probe210:isp0:0:210:0): cam completion status 0x4a (probe211:isp0:0:211:0): cam completion status 0x4a (probe211:isp0:0:211:0): cam completion status 0x4a (probe212:isp0:0:212:0): cam completion status 0x4a (probe212:isp0:0:212:0): cam completion status 0x4a (probe213:isp0:0:213:0): cam completion status 0x4a (probe213:isp0:0:213:0): cam completion status 0x4a (probe214:isp0:0:214:0): cam completion status 0x4a (probe214:isp0:0:214:0): cam completion status 0x4a (probe215:isp0:0:215:0): cam completion status 0x4a (probe215:isp0:0:215:0): cam completion status 0x4a (probe216:isp0:0:216:0): cam completion status 0x4a (probe216:isp0:0:216:0): cam completion status 0x4a (probe217:isp0:0:217:0): cam completion status 0x4a (probe217:isp0:0:217:0): cam completion status 0x4a (probe218:isp0:0:218:0): cam completion status 0x4a (probe218:isp0:0:218:0): cam completion status 0x4a (probe219:isp0:0:219:0): cam completion status 0x4a (probe219:isp0:0:219:0): cam completion status 0x4a (probe220:isp0:0:220:0): cam completion status 0x4a (probe220:isp0:0:220:0): cam completion status 0x4a (probe221:isp0:0:221:0): cam completion status 0x4a (probe221:isp0:0:221:0): cam completion status 0x4a (probe222:isp0:0:222:0): cam completion status 0x4a (probe222:isp0:0:222:0): cam completion status 0x4a (probe223:isp0:0:223:0): cam completion status 0x4a (probe223:isp0:0:223:0): cam completion status 0x4a (probe224:isp0:0:224:0): cam completion status 0x4a (probe224:isp0:0:224:0): cam completion status 0x4a (probe225:isp0:0:225:0): cam completion status 0x4a (probe225:isp0:0:225:0): cam completion status 0x4a (probe226:isp0:0:226:0): cam completion status 0x4a (probe226:isp0:0:226:0): cam completion status 0x4a (probe227:isp0:0:227:0): cam completion status 0x4a (probe227:isp0:0:227:0): cam completion status 0x4a (probe228:isp0:0:228:0): cam completion status 0x4a (probe228:isp0:0:228:0): cam completion status 0x4a (probe229:isp0:0:229:0): cam completion status 0x4a (probe229:isp0:0:229:0): cam completion status 0x4a (probe230:isp0:0:230:0): cam completion status 0x4a (probe230:isp0:0:230:0): cam completion status 0x4a (probe231:isp0:0:231:0): cam completion status 0x4a (probe231:isp0:0:231:0): cam completion status 0x4a (probe232:isp0:0:232:0): cam completion status 0x4a (probe232:isp0:0:232:0): cam completion status 0x4a (probe233:isp0:0:233:0): cam completion status 0x4a (probe233:isp0:0:233:0): cam completion status 0x4a (probe234:isp0:0:234:0): cam completion status 0x4a (probe234:isp0:0:234:0): cam completion status 0x4a (probe235:isp0:0:235:0): cam completion status 0x4a (probe235:isp0:0:235:0): cam completion status 0x4a (probe236:isp0:0:236:0): cam completion status 0x4a (probe236:isp0:0:236:0): cam completion status 0x4a (probe237:isp0:0:237:0): cam completion status 0x4a (probe237:isp0:0:237:0): cam completion status 0x4a (probe238:isp0:0:238:0): cam completion status 0x4a (probe238:isp0:0:238:0): cam completion status 0x4a (probe239:isp0:0:239:0): cam completion status 0x4a (probe239:isp0:0:239:0): cam completion status 0x4a (probe240:isp0:0:240:0): cam completion status 0x4a (probe240:isp0:0:240:0): cam completion status 0x4a (probe241:isp0:0:241:0): cam completion status 0x4a (probe241:isp0:0:241:0): cam completion status 0x4a (probe242:isp0:0:242:0): cam completion status 0x4a (probe242:isp0:0:242:0): cam completion status 0x4a (probe243:isp0:0:243:0): cam completion status 0x4a (probe243:isp0:0:243:0): cam completion status 0x4a (probe244:isp0:0:244:0): cam completion status 0x4a (probe244:isp0:0:244:0): cam completion status 0x4a (probe245:isp0:0:245:0): cam completion status 0x4a (probe245:isp0:0:245:0): cam completion status 0x4a (probe246:isp0:0:246:0): cam completion status 0x4a (probe246:isp0:0:246:0): cam completion status 0x4a (probe247:isp0:0:247:0): cam completion status 0x4a (probe247:isp0:0:247:0): cam completion status 0x4a (probe248:isp0:0:248:0): cam completion status 0x4a (probe248:isp0:0:248:0): cam completion status 0x4a (probe249:isp0:0:249:0): cam completion status 0x4a (probe249:isp0:0:249:0): cam completion status 0x4a (probe250:isp0:0:250:0): cam completion status 0x4a (probe250:isp0:0:250:0): cam completion status 0x4a (probe251:isp0:0:251:0): cam completion status 0x4a (probe251:isp0:0:251:0): cam completion status 0x4a (probe252:isp0:0:252:0): cam completion status 0x4a (probe252:isp0:0:252:0): cam completion status 0x4a (probe253:isp0:0:253:0): cam completion status 0x4a (probe253:isp0:0:253:0): cam completion status 0x4a (probe254:isp0:0:254:0): cam completion status 0x4a (probe254:isp0:0:254:0): cam completion status 0x4a (probe255:isp0:0:255:0): cam completion status 0x4a (probe255:isp0:0:255:0): cam completion status 0x4a (probe256:isp0:0:256:0): cam completion status 0x4a (probe256:isp0:0:256:0): cam completion status 0x4a (probe257:isp0:0:257:0): cam completion status 0x4a (probe257:isp0:0:257:0): cam completion status 0x4a (probe258:isp0:0:258:0): cam completion status 0x4a (probe258:isp0:0:258:0): cam completion status 0x4a (probe259:isp0:0:259:0): cam completion status 0x4a (probe259:isp0:0:259:0): cam completion status 0x4a (probe260:isp0:0:260:0): cam completion status 0x4a (probe260:isp0:0:260:0): cam completion status 0x4a (probe261:isp0:0:261:0): cam completion status 0x4a (probe261:isp0:0:261:0): cam completion status 0x4a (probe262:isp0:0:262:0): cam completion status 0x4a (probe262:isp0:0:262:0): cam completion status 0x4a (probe263:isp0:0:263:0): cam completion status 0x4a (probe263:isp0:0:263:0): cam completion status 0x4a (probe264:isp0:0:264:0): cam completion status 0x4a (probe264:isp0:0:264:0): cam completion status 0x4a (probe265:isp0:0:265:0): cam completion status 0x4a (probe265:isp0:0:265:0): cam completion status 0x4a (probe266:isp0:0:266:0): cam completion status 0x4a (probe266:isp0:0:266:0): cam completion status 0x4a (probe267:isp0:0:267:0): cam completion status 0x4a (probe267:isp0:0:267:0): cam completion status 0x4a (probe268:isp0:0:268:0): cam completion status 0x4a (probe268:isp0:0:268:0): cam completion status 0x4a (probe269:isp0:0:269:0): cam completion status 0x4a (probe269:isp0:0:269:0): cam completion status 0x4a (probe270:isp0:0:270:0): cam completion status 0x4a (probe270:isp0:0:270:0): cam completion status 0x4a (probe271:isp0:0:271:0): cam completion status 0x4a (probe271:isp0:0:271:0): cam completion status 0x4a (probe272:isp0:0:272:0): cam completion status 0x4a (probe272:isp0:0:272:0): cam completion status 0x4a (probe273:isp0:0:273:0): cam completion status 0x4a (probe273:isp0:0:273:0): cam completion status 0x4a (probe274:isp0:0:274:0): cam completion status 0x4a (probe274:isp0:0:274:0): cam completion status 0x4a (probe275:isp0:0:275:0): cam completion status 0x4a (probe275:isp0:0:275:0): cam completion status 0x4a (probe276:isp0:0:276:0): cam completion status 0x4a (probe276:isp0:0:276:0): cam completion status 0x4a (probe277:isp0:0:277:0): cam completion status 0x4a (probe277:isp0:0:277:0): cam completion status 0x4a (probe278:isp0:0:278:0): cam completion status 0x4a (probe278:isp0:0:278:0): cam completion status 0x4a (probe279:isp0:0:279:0): cam completion status 0x4a (probe279:isp0:0:279:0): cam completion status 0x4a (probe280:isp0:0:280:0): cam completion status 0x4a (probe280:isp0:0:280:0): cam completion status 0x4a (probe281:isp0:0:281:0): cam completion status 0x4a (probe281:isp0:0:281:0): cam completion status 0x4a (probe282:isp0:0:282:0): cam completion status 0x4a (probe282:isp0:0:282:0): cam completion status 0x4a (probe283:isp0:0:283:0): cam completion status 0x4a (probe283:isp0:0:283:0): cam completion status 0x4a (probe284:isp0:0:284:0): cam completion status 0x4a (probe284:isp0:0:284:0): cam completion status 0x4a (probe285:isp0:0:285:0): cam completion status 0x4a (probe285:isp0:0:285:0): cam completion status 0x4a (probe286:isp0:0:286:0): cam completion status 0x4a (probe286:isp0:0:286:0): cam completion status 0x4a (probe287:isp0:0:287:0): cam completion status 0x4a (probe287:isp0:0:287:0): cam completion status 0x4a (probe288:isp0:0:288:0): cam completion status 0x4a (probe288:isp0:0:288:0): cam completion status 0x4a (probe289:isp0:0:289:0): cam completion status 0x4a (probe289:isp0:0:289:0): cam completion status 0x4a (probe290:isp0:0:290:0): cam completion status 0x4a (probe290:isp0:0:290:0): cam completion status 0x4a (probe291:isp0:0:291:0): cam completion status 0x4a (probe291:isp0:0:291:0): cam completion status 0x4a (probe292:isp0:0:292:0): cam completion status 0x4a (probe292:isp0:0:292:0): cam completion status 0x4a (probe293:isp0:0:293:0): cam completion status 0x4a (probe293:isp0:0:293:0): cam completion status 0x4a (probe294:isp0:0:294:0): cam completion status 0x4a (probe294:isp0:0:294:0): cam completion status 0x4a (probe295:isp0:0:295:0): cam completion status 0x4a (probe295:isp0:0:295:0): cam completion status 0x4a (probe296:isp0:0:296:0): cam completion status 0x4a (probe296:isp0:0:296:0): cam completion status 0x4a (probe297:isp0:0:297:0): cam completion status 0x4a (probe297:isp0:0:297:0): cam completion status 0x4a (probe298:isp0:0:298:0): cam completion status 0x4a (probe298:isp0:0:298:0): cam completion status 0x4a (probe299:isp0:0:299:0): cam completion status 0x4a (probe299:isp0:0:299:0): cam completion status 0x4a (probe300:isp0:0:300:0): cam completion status 0x4a (probe300:isp0:0:300:0): cam completion status 0x4a (probe301:isp0:0:301:0): cam completion status 0x4a (probe301:isp0:0:301:0): cam completion status 0x4a (probe302:isp0:0:302:0): cam completion status 0x4a (probe302:isp0:0:302:0): cam completion status 0x4a (probe303:isp0:0:303:0): cam completion status 0x4a (probe303:isp0:0:303:0): cam completion status 0x4a (probe304:isp0:0:304:0): cam completion status 0x4a (probe304:isp0:0:304:0): cam completion status 0x4a (probe305:isp0:0:305:0): cam completion status 0x4a (probe305:isp0:0:305:0): cam completion status 0x4a (probe306:isp0:0:306:0): cam completion status 0x4a (probe306:isp0:0:306:0): cam completion status 0x4a (probe307:isp0:0:307:0): cam completion status 0x4a (probe307:isp0:0:307:0): cam completion status 0x4a (probe308:isp0:0:308:0): cam completion status 0x4a (probe308:isp0:0:308:0): cam completion status 0x4a (probe309:isp0:0:309:0): cam completion status 0x4a (probe309:isp0:0:309:0): cam completion status 0x4a (probe310:isp0:0:310:0): cam completion status 0x4a (probe310:isp0:0:310:0): cam completion status 0x4a (probe311:isp0:0:311:0): cam completion status 0x4a (probe311:isp0:0:311:0): cam completion status 0x4a (probe312:isp0:0:312:0): cam completion status 0x4a (probe312:isp0:0:312:0): cam completion status 0x4a (probe313:isp0:0:313:0): cam completion status 0x4a (probe313:isp0:0:313:0): cam completion status 0x4a (probe314:isp0:0:314:0): cam completion status 0x4a (probe314:isp0:0:314:0): cam completion status 0x4a (probe315:isp0:0:315:0): cam completion status 0x4a (probe315:isp0:0:315:0): cam completion status 0x4a (probe316:isp0:0:316:0): cam completion status 0x4a (probe316:isp0:0:316:0): cam completion status 0x4a (probe317:isp0:0:317:0): cam completion status 0x4a (probe317:isp0:0:317:0): cam completion status 0x4a (probe318:isp0:0:318:0): cam completion status 0x4a (probe318:isp0:0:318:0): cam completion status 0x4a (probe319:isp0:0:319:0): cam completion status 0x4a (probe319:isp0:0:319:0): cam completion status 0x4a (probe320:isp0:0:320:0): cam completion status 0x4a (probe320:isp0:0:320:0): cam completion status 0x4a (probe321:isp0:0:321:0): cam completion status 0x4a (probe321:isp0:0:321:0): cam completion status 0x4a (probe322:isp0:0:322:0): cam completion status 0x4a (probe322:isp0:0:322:0): cam completion status 0x4a (probe323:isp0:0:323:0): cam completion status 0x4a (probe323:isp0:0:323:0): cam completion status 0x4a (probe324:isp0:0:324:0): cam completion status 0x4a (probe324:isp0:0:324:0): cam completion status 0x4a (probe325:isp0:0:325:0): cam completion status 0x4a (probe325:isp0:0:325:0): cam completion status 0x4a (probe326:isp0:0:326:0): cam completion status 0x4a (probe326:isp0:0:326:0): cam completion status 0x4a (probe327:isp0:0:327:0): cam completion status 0x4a (probe327:isp0:0:327:0): cam completion status 0x4a (probe328:isp0:0:328:0): cam completion status 0x4a (probe328:isp0:0:328:0): cam completion status 0x4a (probe329:isp0:0:329:0): cam completion status 0x4a (probe329:isp0:0:329:0): cam completion status 0x4a (probe330:isp0:0:330:0): cam completion status 0x4a (probe330:isp0:0:330:0): cam completion status 0x4a (probe331:isp0:0:331:0): cam completion status 0x4a (probe331:isp0:0:331:0): cam completion status 0x4a (probe332:isp0:0:332:0): cam completion status 0x4a (probe332:isp0:0:332:0): cam completion status 0x4a (probe333:isp0:0:333:0): cam completion status 0x4a (probe333:isp0:0:333:0): cam completion status 0x4a (probe334:isp0:0:334:0): cam completion status 0x4a (probe334:isp0:0:334:0): cam completion status 0x4a (probe335:isp0:0:335:0): cam completion status 0x4a (probe335:isp0:0:335:0): cam completion status 0x4a (probe336:isp0:0:336:0): cam completion status 0x4a (probe336:isp0:0:336:0): cam completion status 0x4a (probe337:isp0:0:337:0): cam completion status 0x4a (probe337:isp0:0:337:0): cam completion status 0x4a (probe338:isp0:0:338:0): cam completion status 0x4a (probe338:isp0:0:338:0): cam completion status 0x4a (probe339:isp0:0:339:0): cam completion status 0x4a (probe339:isp0:0:339:0): cam completion status 0x4a (probe340:isp0:0:340:0): cam completion status 0x4a (probe340:isp0:0:340:0): cam completion status 0x4a (probe341:isp0:0:341:0): cam completion status 0x4a (probe341:isp0:0:341:0): cam completion status 0x4a (probe342:isp0:0:342:0): cam completion status 0x4a (probe342:isp0:0:342:0): cam completion status 0x4a (probe343:isp0:0:343:0): cam completion status 0x4a (probe343:isp0:0:343:0): cam completion status 0x4a (probe344:isp0:0:344:0): cam completion status 0x4a (probe344:isp0:0:344:0): cam completion status 0x4a (probe345:isp0:0:345:0): cam completion status 0x4a (probe345:isp0:0:345:0): cam completion status 0x4a (probe346:isp0:0:346:0): cam completion status 0x4a (probe346:isp0:0:346:0): cam completion status 0x4a (probe347:isp0:0:347:0): cam completion status 0x4a (probe347:isp0:0:347:0): cam completion status 0x4a (probe348:isp0:0:348:0): cam completion status 0x4a (probe348:isp0:0:348:0): cam completion status 0x4a (probe349:isp0:0:349:0): cam completion status 0x4a (probe349:isp0:0:349:0): cam completion status 0x4a (probe350:isp0:0:350:0): cam completion status 0x4a (probe350:isp0:0:350:0): cam completion status 0x4a (probe351:isp0:0:351:0): cam completion status 0x4a (probe351:isp0:0:351:0): cam completion status 0x4a (probe352:isp0:0:352:0): cam completion status 0x4a (probe352:isp0:0:352:0): cam completion status 0x4a (probe353:isp0:0:353:0): cam completion status 0x4a (probe353:isp0:0:353:0): cam completion status 0x4a (probe354:isp0:0:354:0): cam completion status 0x4a (probe354:isp0:0:354:0): cam completion status 0x4a (probe355:isp0:0:355:0): cam completion status 0x4a (probe355:isp0:0:355:0): cam completion status 0x4a (probe356:isp0:0:356:0): cam completion status 0x4a (probe356:isp0:0:356:0): cam completion status 0x4a (probe357:isp0:0:357:0): cam completion status 0x4a (probe357:isp0:0:357:0): cam completion status 0x4a (probe358:isp0:0:358:0): cam completion status 0x4a (probe358:isp0:0:358:0): cam completion status 0x4a (probe359:isp0:0:359:0): cam completion status 0x4a (probe359:isp0:0:359:0): cam completion status 0x4a (probe360:isp0:0:360:0): cam completion status 0x4a (probe360:isp0:0:360:0): cam completion status 0x4a (probe361:isp0:0:361:0): cam completion status 0x4a (probe361:isp0:0:361:0): cam completion status 0x4a (probe362:isp0:0:362:0): cam completion status 0x4a (probe362:isp0:0:362:0): cam completion status 0x4a (probe363:isp0:0:363:0): cam completion status 0x4a (probe363:isp0:0:363:0): cam completion status 0x4a (probe364:isp0:0:364:0): cam completion status 0x4a (probe364:isp0:0:364:0): cam completion status 0x4a (probe365:isp0:0:365:0): cam completion status 0x4a (probe365:isp0:0:365:0): cam completion status 0x4a (probe366:isp0:0:366:0): cam completion status 0x4a (probe366:isp0:0:366:0): cam completion status 0x4a (probe367:isp0:0:367:0): cam completion status 0x4a (probe367:isp0:0:367:0): cam completion status 0x4a (probe368:isp0:0:368:0): cam completion status 0x4a (probe368:isp0:0:368:0): cam completion status 0x4a (probe369:isp0:0:369:0): cam completion status 0x4a (probe369:isp0:0:369:0): cam completion status 0x4a (probe370:isp0:0:370:0): cam completion status 0x4a (probe370:isp0:0:370:0): cam completion status 0x4a (probe371:isp0:0:371:0): cam completion status 0x4a (probe371:isp0:0:371:0): cam completion status 0x4a (probe372:isp0:0:372:0): cam completion status 0x4a (probe372:isp0:0:372:0): cam completion status 0x4a (probe373:isp0:0:373:0): cam completion status 0x4a (probe373:isp0:0:373:0): cam completion status 0x4a (probe374:isp0:0:374:0): cam completion status 0x4a (probe374:isp0:0:374:0): cam completion status 0x4a (probe375:isp0:0:375:0): cam completion status 0x4a (probe375:isp0:0:375:0): cam completion status 0x4a (probe377:isp0:0:377:0): cam completion status 0x4a (probe377:isp0:0:377:0): cam completion status 0x4a (probe378:isp0:0:378:0): cam completion status 0x4a (probe378:isp0:0:378:0): cam completion status 0x4a (probe379:isp0:0:379:0): cam completion status 0x4a (probe379:isp0:0:379:0): cam completion status 0x4a (probe380:isp0:0:380:0): cam completion status 0x4a (probe380:isp0:0:380:0): cam completion status 0x4a (probe381:isp0:0:381:0): cam completion status 0x4a (probe381:isp0:0:381:0): cam completion status 0x4a (probe382:isp0:0:382:0): cam completion status 0x4a (probe382:isp0:0:382:0): cam completion status 0x4a (probe383:isp0:0:383:0): cam completion status 0x4a (probe383:isp0:0:383:0): cam completion status 0x4a (probe384:isp0:0:384:0): cam completion status 0x4a (probe384:isp0:0:384:0): cam completion status 0x4a (probe385:isp0:0:385:0): cam completion status 0x4a (probe385:isp0:0:385:0): cam completion status 0x4a (probe386:isp0:0:386:0): cam completion status 0x4a (probe386:isp0:0:386:0): cam completion status 0x4a (probe387:isp0:0:387:0): cam completion status 0x4a (probe387:isp0:0:387:0): cam completion status 0x4a (probe388:isp0:0:388:0): cam completion status 0x4a (probe388:isp0:0:388:0): cam completion status 0x4a (probe389:isp0:0:389:0): cam completion status 0x4a (probe389:isp0:0:389:0): cam completion status 0x4a (probe390:isp0:0:390:0): cam completion status 0x4a (probe390:isp0:0:390:0): cam completion status 0x4a (probe391:isp0:0:391:0): cam completion status 0x4a (probe391:isp0:0:391:0): cam completion status 0x4a (probe392:isp0:0:392:0): cam completion status 0x4a (probe392:isp0:0:392:0): cam completion status 0x4a (probe393:isp0:0:393:0): cam completion status 0x4a (probe393:isp0:0:393:0): cam completion status 0x4a (probe394:isp0:0:394:0): cam completion status 0x4a (probe394:isp0:0:394:0): cam completion status 0x4a (probe396:isp0:0:396:0): cam completion status 0x4a (probe396:isp0:0:396:0): cam completion status 0x4a (probe397:isp0:0:397:0): cam completion status 0x4a (probe397:isp0:0:397:0): cam completion status 0x4a (probe398:isp0:0:398:0): cam completion status 0x4a (probe398:isp0:0:398:0): cam completion status 0x4a (probe399:isp0:0:399:0): cam completion status 0x4a (probe399:isp0:0:399:0): cam completion status 0x4a (probe401:isp0:0:401:0): cam completion status 0x4a (probe401:isp0:0:401:0): cam completion status 0x4a (probe402:isp0:0:402:0): cam completion status 0x4a (probe402:isp0:0:402:0): cam completion status 0x4a (probe403:isp0:0:403:0): cam completion status 0x4a (probe403:isp0:0:403:0): cam completion status 0x4a (probe404:isp0:0:404:0): cam completion status 0x4a (probe404:isp0:0:404:0): cam completion status 0x4a (probe405:isp0:0:405:0): cam completion status 0x4a (probe405:isp0:0:405:0): cam completion status 0x4a (probe406:isp0:0:406:0): cam completion status 0x4a (probe406:isp0:0:406:0): cam completion status 0x4a (probe407:isp0:0:407:0): cam completion status 0x4a (probe407:isp0:0:407:0): cam completion status 0x4a (probe408:isp0:0:408:0): cam completion status 0x4a (probe408:isp0:0:408:0): cam completion status 0x4a (probe409:isp0:0:409:0): cam completion status 0x4a (probe409:isp0:0:409:0): cam completion status 0x4a (probe410:isp0:0:410:0): cam completion status 0x4a (probe410:isp0:0:410:0): cam completion status 0x4a (probe412:isp0:0:412:0): cam completion status 0x4a (probe412:isp0:0:412:0): cam completion status 0x4a (probe413:isp0:0:413:0): cam completion status 0x4a (probe413:isp0:0:413:0): cam completion status 0x4a (probe414:isp0:0:414:0): cam completion status 0x4a (probe414:isp0:0:414:0): cam completion status 0x4a (probe415:isp0:0:415:0): cam completion status 0x4a (probe415:isp0:0:415:0): cam completion status 0x4a (probe416:isp0:0:416:0): cam completion status 0x4a (probe416:isp0:0:416:0): cam completion status 0x4a (probe417:isp0:0:417:0): cam completion status 0x4a (probe417:isp0:0:417:0): cam completion status 0x4a (probe418:isp0:0:418:0): cam completion status 0x4a (probe418:isp0:0:418:0): cam completion status 0x4a (probe419:isp0:0:419:0): cam completion status 0x4a (probe419:isp0:0:419:0): cam completion status 0x4a (probe420:isp0:0:420:0): cam completion status 0x4a (probe420:isp0:0:420:0): cam completion status 0x4a (probe421:isp0:0:421:0): cam completion status 0x4a (probe421:isp0:0:421:0): cam completion status 0x4a (probe422:isp0:0:422:0): cam completion status 0x4a (probe422:isp0:0:422:0): cam completion status 0x4a (probe424:isp0:0:424:0): cam completion status 0x4a (probe424:isp0:0:424:0): cam completion status 0x4a (probe425:isp0:0:425:0): cam completion status 0x4a (probe425:isp0:0:425:0): cam completion status 0x4a (probe426:isp0:0:426:0): cam completion status 0x4a (probe426:isp0:0:426:0): cam completion status 0x4a (probe427:isp0:0:427:0): cam completion status 0x4a (probe427:isp0:0:427:0): cam completion status 0x4a (probe428:isp0:0:428:0): cam completion status 0x4a (probe428:isp0:0:428:0): cam completion status 0x4a (probe429:isp0:0:429:0): cam completion status 0x4a (probe429:isp0:0:429:0): cam completion status 0x4a (probe430:isp0:0:430:0): cam completion status 0x4a (probe430:isp0:0:430:0): cam completion status 0x4a (probe431:isp0:0:431:0): cam completion status 0x4a (probe431:isp0:0:431:0): cam completion status 0x4a (probe432:isp0:0:432:0): cam completion status 0x4a (probe432:isp0:0:432:0): cam completion status 0x4a (probe433:isp0:0:433:0): cam completion status 0x4a (probe433:isp0:0:433:0): cam completion status 0x4a (probe434:isp0:0:434:0): cam completion status 0x4a (probe434:isp0:0:434:0): cam completion status 0x4a (probe435:isp0:0:435:0): cam completion status 0x4a (probe435:isp0:0:435:0): cam completion status 0x4a (probe436:isp0:0:436:0): cam completion status 0x4a (probe436:isp0:0:436:0): cam completion status 0x4a (probe438:isp0:0:438:0): cam completion status 0x4a (probe438:isp0:0:438:0): cam completion status 0x4a (probe439:isp0:0:439:0): cam completion status 0x4a (probe439:isp0:0:439:0): cam completion status 0x4a (probe440:isp0:0:440:0): cam completion status 0x4a (probe440:isp0:0:440:0): cam completion status 0x4a (probe441:isp0:0:441:0): cam completion status 0x4a (probe441:isp0:0:441:0): cam completion status 0x4a (probe442:isp0:0:442:0): cam completion status 0x4a (probe442:isp0:0:442:0): cam completion status 0x4a (probe443:isp0:0:443:0): cam completion status 0x4a (probe443:isp0:0:443:0): cam completion status 0x4a (probe444:isp0:0:444:0): cam completion status 0x4a (probe444:isp0:0:444:0): cam completion status 0x4a (probe445:isp0:0:445:0): cam completion status 0x4a (probe445:isp0:0:445:0): cam completion status 0x4a (probe446:isp0:0:446:0): cam completion status 0x4a (probe446:isp0:0:446:0): cam completion status 0x4a (probe447:isp0:0:447:0): cam completion status 0x4a (probe447:isp0:0:447:0): cam completion status 0x4a (probe448:isp0:0:448:0): cam completion status 0x4a (probe448:isp0:0:448:0): cam completion status 0x4a (probe450:isp0:0:450:0): cam completion status 0x4a (probe450:isp0:0:450:0): cam completion status 0x4a (probe451:isp0:0:451:0): cam completion status 0x4a (probe451:isp0:0:451:0): cam completion status 0x4a (probe452:isp0:0:452:0): cam completion status 0x4a (probe452:isp0:0:452:0): cam completion status 0x4a (probe453:isp0:0:453:0): cam completion status 0x4a (probe453:isp0:0:453:0): cam completion status 0x4a (probe454:isp0:0:454:0): cam completion status 0x4a (probe454:isp0:0:454:0): cam completion status 0x4a (probe455:isp0:0:455:0): cam completion status 0x4a (probe455:isp0:0:455:0): cam completion status 0x4a (probe456:isp0:0:456:0): cam completion status 0x4a (probe456:isp0:0:456:0): cam completion status 0x4a (probe457:isp0:0:457:0): cam completion status 0x4a (probe457:isp0:0:457:0): cam completion status 0x4a (probe458:isp0:0:458:0): cam completion status 0x4a (probe458:isp0:0:458:0): cam completion status 0x4a (probe460:isp0:0:460:0): cam completion status 0x4a (probe460:isp0:0:460:0): cam completion status 0x4a (probe461:isp0:0:461:0): cam completion status 0x4a (probe461:isp0:0:461:0): cam completion status 0x4a (probe462:isp0:0:462:0): cam completion status 0x4a (probe462:isp0:0:462:0): cam completion status 0x4a (probe463:isp0:0:463:0): cam completion status 0x4a (probe463:isp0:0:463:0): cam completion status 0x4a (probe465:isp0:0:465:0): cam completion status 0x4a (probe465:isp0:0:465:0): cam completion status 0x4a (probe466:isp0:0:466:0): cam completion status 0x4a (probe466:isp0:0:466:0): cam completion status 0x4a (probe467:isp0:0:467:0): cam completion status 0x4a (probe467:isp0:0:467:0): cam completion status 0x4a (probe468:isp0:0:468:0): cam completion status 0x4a (probe468:isp0:0:468:0): cam completion status 0x4a (probe469:isp0:0:469:0): cam completion status 0x4a (probe469:isp0:0:469:0): cam completion status 0x4a (probe470:isp0:0:470:0): cam completion status 0x4a (probe470:isp0:0:470:0): cam completion status 0x4a (probe472:isp0:0:472:0): cam completion status 0x4a (probe472:isp0:0:472:0): cam completion status 0x4a (probe473:isp0:0:473:0): cam completion status 0x4a (probe473:isp0:0:473:0): cam completion status 0x4a (probe474:isp0:0:474:0): cam completion status 0x4a (probe474:isp0:0:474:0): cam completion status 0x4a (probe475:isp0:0:475:0): cam completion status 0x4a (probe475:isp0:0:475:0): cam completion status 0x4a (probe476:isp0:0:476:0): cam completion status 0x4a (probe476:isp0:0:476:0): cam completion status 0x4a (probe477:isp0:0:477:0): cam completion status 0x4a (probe477:isp0:0:477:0): cam completion status 0x4a (probe479:isp0:0:479:0): cam completion status 0x4a (probe479:isp0:0:479:0): cam completion status 0x4a (probe480:isp0:0:480:0): cam completion status 0x4a (probe480:isp0:0:480:0): cam completion status 0x4a (probe481:isp0:0:481:0): cam completion status 0x4a (probe481:isp0:0:481:0): cam completion status 0x4a (probe482:isp0:0:482:0): cam completion status 0x4a (probe482:isp0:0:482:0): cam completion status 0x4a (probe483:isp0:0:483:0): cam completion status 0x4a (probe483:isp0:0:483:0): cam completion status 0x4a (probe484:isp0:0:484:0): cam completion status 0x4a (probe484:isp0:0:484:0): cam completion status 0x4a (probe485:isp0:0:485:0): cam completion status 0x4a (probe485:isp0:0:485:0): cam completion status 0x4a (probe486:isp0:0:486:0): cam completion status 0x4a (probe486:isp0:0:486:0): cam completion status 0x4a (probe487:isp0:0:487:0): cam completion status 0x4a (probe487:isp0:0:487:0): cam completion status 0x4a (probe489:isp0:0:489:0): cam completion status 0x4a (probe489:isp0:0:489:0): cam completion status 0x4a (probe490:isp0:0:490:0): cam completion status 0x4a (probe490:isp0:0:490:0): cam completion status 0x4a (probe491:isp0:0:491:0): cam completion status 0x4a (probe491:isp0:0:491:0): cam completion status 0x4a (probe492:isp0:0:492:0): cam completion status 0x4a (probe492:isp0:0:492:0): cam completion status 0x4a (probe493:isp0:0:493:0): cam completion status 0x4a (probe493:isp0:0:493:0): cam completion status 0x4a (probe494:isp0:0:494:0): cam completion status 0x4a (probe494:isp0:0:494:0): cam completion status 0x4a (probe495:isp0:0:495:0): cam completion status 0x4a (probe495:isp0:0:495:0): cam completion status 0x4a (probe496:isp0:0:496:0): cam completion status 0x4a (probe496:isp0:0:496:0): cam completion status 0x4a (probe497:isp0:0:497:0): cam completion status 0x4a (probe497:isp0:0:497:0): cam completion status 0x4a (probe498:isp0:0:498:0): cam completion status 0x4a (probe498:isp0:0:498:0): cam completion status 0x4a (probe499:isp0:0:499:0): cam completion status 0x4a (probe499:isp0:0:499:0): cam completion status 0x4a (probe500:isp0:0:500:0): cam completion status 0x4a (probe500:isp0:0:500:0): cam completion status 0x4a (probe501:isp0:0:501:0): cam completion status 0x4a (probe501:isp0:0:501:0): cam completion status 0x4a (probe502:isp0:0:502:0): cam completion status 0x4a (probe502:isp0:0:502:0): cam completion status 0x4a (probe503:isp0:0:503:0): cam completion status 0x4a (probe503:isp0:0:503:0): cam completion status 0x4a (probe6:isp0:0:6:0): cam completion status 0x4a (probe6:isp0:0:6:0): cam completion status 0x4a (probe25:isp0:0:25:0): cam completion status 0x4a (probe25:isp0:0:25:0): cam completion status 0x4a (probe62:isp0:0:62:0): cam completion status 0x4a (probe62:isp0:0:62:0): cam completion status 0x4a (probe74:isp0:0:74:0): cam completion status 0x4a (probe74:isp0:0:74:0): cam completion status 0x4a (probe80:isp0:0:80:0): cam completion status 0x4a (probe80:isp0:0:80:0): cam completion status 0x4a (probe109:isp0:0:109:0): cam completion status 0x4a (probe109:isp0:0:109:0): cam completion status 0x4a (probe131:isp0:0:131:0): cam completion status 0x4a (probe131:isp0:0:131:0): cam completion status 0x4a (probe147:isp0:0:147:0): cam completion status 0x4a (probe147:isp0:0:147:0): cam completion status 0x4a (probe206:isp0:0:206:0): cam completion status 0x4a (probe206:isp0:0:206:0): cam completion status 0x4a da0 at isp0 bus 0 target 0 lun 0 da0: <DGC RAID 5 0326> Fixed Direct Access SCSI-4 device da0: 100.000MB/s transfers da0: Command Queueing Enabled da0: 512000MB (1048576000 512 byte sectors: 255H 63S/T 65270C) cd0 at umass-sim0 bus 0 target 0 lun 0 cd0: <Dell Virtual CDROM 123> Removable CD-ROM SCSI-0 device cd0: 40.000MB/s transfers cd0: Attempt to query device size failed: NOT READY, Medium not present SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #4 Launched! WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. da1 at umass-sim1 bus 1 target 0 lun 0 da1: <Dell Virtual Floppy 123> Removable Direct Access SCSI-0 device da1: 40.000MB/s transfers da1: Attempt to query device size failed: NOT READY, Medium not present Trying to mount root from ufs:/dev/mfid0s1a bce0: link state changed to UP mfi0: 3320 (277345023s/0x0008/info) - Battery charge complete --------------------------------------------------------------- 4. panic data and ddb output, also available at http://noc.ntua.gr/~christia/san/7-STABLE.deadlock-ddb.out : KKDDpanic: ufs_dirbad: /a: bad dir ino 3370734 at ofBBfset 0: mangled entry cpuid = 1 KDB: enter: panic [thread pid 768 tid 100104 ] Stopped at kdb_enter_why+0x3a: movl $0,kdb_why db> trace Tracing pid 768 tid 100104 td 0xc74c6690 kdb_enter_why(c0b3b0a1,c0b3b0a1,c0b5d0ec,ed2dd85c,1,...) at kdb_enter_why+0x3a panic(c0b5d0ec,c70365c0,336eee,0,c0b5d164,...) at panic+0x136 ufs_dirbad(c9c40180,0,c0b5d164,0,ed2dd904,...) at ufs_dirbad+0x49 ufs_lookup(ed2dd94c,c0b6f08d,cb090160,cb090160,ed2ddba4,...) at ufs_lookup+0x377 VOP_CACHEDLOOKUP_APV(c0c0c1a0,ed2dd94c,ed2ddba4,ed2ddb90,c75dae00,...) at VOP_CACHEDLOOKUP_APV+0xc5 vfs_cache_lookup(ed2dd9d0,c0b471e4,cb090160,2,cb090160,...) at vfs_cache_lookup+0xd3 VOP_LOOKUP_APV(c0c0c1a0,ed2dd9d0,c74c6690,c0b46172,19b,...) at VOP_LOOKUP_APV+0xe5 lookup(ed2ddb7c,c0b46172,c6,bf,c703a72c,...) at lookup+0x59e namei(ed2ddb7c,c74c6690,ed2dda74,246,c0c61ab8,...) at namei+0x35e vn_open_cred(ed2ddb7c,ed2ddc78,1a4,c75dae00,c74871c8,...) at vn_open_cred+0x62 vn_open(ed2ddb7c,ed2ddc78,1a4,c74871c8,0,...) at vn_open+0x33 kern_open(c74c6690,8164100,0,a02,1a4,...) at kern_open+0xe7 open(c74c6690,ed2ddcfc,c,c0b40d50,c0beb278,...) at open+0x30 syscall(ed2ddd38) at syscall+0x2d3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (5, FreeBSD ELF32, open), eip = 0x281a161b, esp = 0xbfbfe8cc, ebp = 0xbfbfe8f8 --- db> ps pid ppid pgrp uid state wmesg wchan cmd 768 728 768 0 R+ CPU 1 bsdtar 766 728 766 0 R+ CPU 4 bsdtar 728 724 728 0 Ss+ ttyin 0xc68ce810 csh 724 645 724 0 Ss select 0xc0ca821c sshd 720 711 720 0 S+ ttyin 0xc69a7810 csh 719 1 719 0 Ss+ ttyin 0xc69db410 getty 718 1 718 0 Ss+ ttyin 0xc69dc810 getty 717 1 717 0 Ss+ ttyin 0xc69de410 getty 716 1 716 0 Ss+ ttyin 0xc69d3810 getty 715 1 715 0 Ss+ ttyin 0xc69d5810 getty 714 1 714 0 Ss+ ttyin 0xc69d5c10 getty 713 1 713 0 Ss+ ttyin 0xc69dc010 getty 712 1 712 0 Ss+ ttyin 0xc69db010 getty 711 1 711 0 Ss+ wait 0xc6feb2b8 login 672 1 672 0 Ss nanslp 0xc0c56844 cron 666 1 666 25 Ss pause 0xc7049b40 sendmail 660 1 660 0 Ss select 0xc0ca821c sendmail 645 1 645 0 Ss select 0xc0ca821c sshd 527 1 527 0 Ss select 0xc0ca821c syslogd 473 1 473 0 Ss select 0xc0ca821c devd 143 1 143 0 Ss pause 0xc704ab40 adjkerntz 56 0 0 0 SL sdflush 0xc0cb4668 [softdepflush] 55 0 0 0 SL syncer 0xc0c5666c [syncer] 54 0 0 0 SL vlruwt 0xc6fec000 [vnlru] 53 0 0 0 RL CPU 5 [bufdaemon] 52 0 0 0 SL pgzero 0xc0cb5220 [pagezero] 51 0 0 0 SL psleep 0xc0cb4e38 [vmdaemon] 50 0 0 0 SL psleep 0xc0cb4e00 [pagedaemon] 49 0 0 0 SL waiting_ 0xc0caa434 [sctp_iterator] 48 0 0 0 WL [irq1: atkbd0] 47 0 0 0 WL [swi0: sio] 46 0 0 0 WL [irq15: ata1] 45 0 0 0 WL [irq14: ata0] 44 0 0 0 SL usbevt 0xc68a8210 [usb4] 43 0 0 0 SL usbevt 0xc6994210 [usb3] 42 0 0 0 SL usbevt 0xc6977210 [usb2] 41 0 0 0 SL usbevt 0xc697c210 [usb1] 40 0 0 0 WL [irq20: uhci1 uhci3] 39 0 0 0 SL usbtsk 0xc0c53e14 [usbtask-dr] 38 0 0 0 SL usbtsk 0xc0c53e00 [usbtask-hc] 37 0 0 0 SL usbevt 0xc692d210 [usb0] 36 0 0 0 WL [irq21: uhci0 uhci+] 35 0 0 0 WL [irq259: bce1] 34 0 0 0 SL ispf 0xc67e14d4 [isp1: fc_thrd] 33 0 0 0 WL [irq258: isp1] 32 0 0 0 SL ispf 0xc67e06d4 [isp0: fc_thrd] 31 0 0 0 RL [irq257: isp0] 30 0 0 0 WL [irq16: mfi0] 29 0 0 0 WL [irq256: bce0] 28 0 0 0 WL [irq9: acpi0] 27 0 0 0 SL - 0xc685e900 [thread taskq] 26 0 0 0 WL [swi5: +] 9 0 0 0 SL - 0xc685eb00 [kqueue taskq] 8 0 0 0 SL - 0xc685eb80 [acpi_task_2] 7 0 0 0 SL - 0xc685eb80 [acpi_task_1] 6 0 0 0 SL - 0xc685eb80 [acpi_task_0] 25 0 0 0 WL [swi2: cambio] 5 0 0 0 SL ccb_scan 0xc0c24394 [xpt_thrd] 24 0 0 0 WL [swi6: task queue] 23 0 0 0 WL [swi6: Giant taskq] 22 0 0 0 SL - 0xc0c56674 [yarrow] 4 0 0 0 SL - 0xc0c5420c [g_down] 3 0 0 0 RL CPU 3 [g_up] 2 0 0 0 SL - 0xc0c54200 [g_event] 21 0 0 0 WL [swi3: vm] 20 0 0 0 WL [swi4: clock sio] 19 0 0 0 WL [swi1: net] 18 0 0 0 RL CPU 0 [idle: cpu0] 17 0 0 0 RL [idle: cpu1] 16 0 0 0 RL CPU 2 [idle: cpu2] 15 0 0 0 RL [idle: cpu3] 14 0 0 0 RL [idle: cpu4] 13 0 0 0 RL [idle: cpu5] 12 0 0 0 RL CPU 6 [idle: cpu6] 11 0 0 0 RL CPU 7 [idle: cpu7] 1 0 1 0 SLs wait 0xc6793ae0 [init] 10 0 0 0 SL audit_wo 0xc0cb40a4 [audit] 0 0 0 0 SLs sched 0xc0c542c0 [swapper] db> show pcpu cpuid = 1 curthread = 0xc74c6690: pid 768 "bsdtar" curpcb = 0xed2ddd90 fpcurthread = none idlethread = 0xc67a7460: pid 17 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: db> show allpcpu Current CPU: 1 cpuid = 0 curthread = 0xc67a7230: pid 18 "idle: cpu0" curpcb = 0xe5424d90 fpcurthread = none idlethread = 0xc67a7230: pid 18 "idle: cpu0" APIC ID = 0 currentldt = 0x50 spin locks held: cpuid = 1 curthread = 0xc74c6690: pid 768 "bsdtar" curpcb = 0xed2ddd90 fpcurthread = none idlethread = 0xc67a7460: pid 17 "idle: cpu1" APIC ID = 1 currentldt = 0x50 spin locks held: cpuid = 2 curthread = 0xc6795000: pid 16 "idle: cpu2" curpcb = 0xe541ed90 fpcurthread = none idlethread = 0xc6795000: pid 16 "idle: cpu2" APIC ID = 2 currentldt = 0x50 spin locks held: cpuid = 3 curthread = 0xc6797690: pid 3 "g_up" curpcb = 0xe6e36d90 fpcurthread = none idlethread = 0xc6795230: pid 15 "idle: cpu3" APIC ID = 3 currentldt = 0x50 spin locks held: cpuid = 4 curthread = 0xc75c3690: pid 766 "bsdtar" curpcb = 0xed348d90 fpcurthread = none idlethread = 0xc6795460: pid 14 "idle: cpu4" APIC ID = 4 currentldt = 0x50 spin locks held: cpuid = 5 curthread = 0xc6fea8c0: pid 53 "bufdaemon" curpcb = 0xeafead90 fpcurthread = none idlethread = 0xc6795690: pid 13 "idle: cpu5" APIC ID = 5 currentldt = 0x50 spin locks held: cpuid = 6 curthread = 0xc67958c0: pid 12 "idle: cpu6" curpcb = 0xe5412d90 fpcurthread = none idlethread = 0xc67958c0: pid 12 "idle: cpu6" APIC ID = 6 currentldt = 0x50 spin locks held: cpuid = 7 curthread = 0xc6795af0: pid 11 "idle: cpu7" curpcb = 0xe540fd90 fpcurthread = none idlethread = 0xc6795af0: pid 11 "idle: cpu7" APIC ID = 7 currentldt = 0x50 spin locks held: db> show locks db> show alllocks Process 4 (g_down) thread 0xc6797460 (100015) exclusive sleep mutex isp r = 0 (0xc67e0690) locked @ /usr/src/sys/cam/cam_periph.h:182 db> show lockedvnods Locked vnodes 0xcb090160: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags () v_object 0xcaa70c1c ref 0 pages 1 lock type ufs: EXCL (count 1) by thread 0xc74c6690 (pid 768)#0 0xc077f6c0 at _lockmgr+0x560 #1 0xc099cb51 at ffs_lock+0x91 #2 0xc0aafcc5 at VOP_LOCK1_APV+0xa5 #3 0xc081f7b8 at _vn_lock+0xf8 #4 0xc0813c20 at vget+0x110 #5 0xc08029b7 at cache_lookup+0x4e7 #6 0xc0802aea at vfs_cache_lookup+0xaa #7 0xc0ab0e65 at VOP_LOOKUP_APV+0xe5 #8 0xc0808d2e at lookup+0x59e #9 0xc08098de at namei+0x35e #10 0xc081f022 at vn_open_cred+0x62 #11 0xc081f513 at vn_open+0x33 #12 0xc081e177 at kern_open+0xe7 #13 0xc081e660 at open+0x30 #14 0xc0a98853 at syscall+0x2d3 #15 0xc0a7ea80 at Xint0x80_syscall+0x20 ino 3370734, on dev da0s1d 0xcb5906e0: tag ufs, type VDIR usecount 0, writecount 0, refcount 2 mountedhere 0 flags () v_object 0xc9b65a2c ref 0 pages 0 lock type ufs: EXCL (count 1) by thread 0xc6fea8c0 (pid 53)#0 0xc077f6c0 at _lockmgr+0x560 #1 0xc099cb51 at ffs_lock+0x91 #2 0xc0aafcc5 at VOP_LOCK1_APV+0xa5 #3 0xc081f7b8 at _vn_lock+0xf8 #4 0xc07ff5d8 at flushbufqueues+0x568 #5 0xc07ff8f0 at buf_daemon+0xc0 #6 0xc0770ea8 at fork_exit+0xb8 #7 0xc0a7ea90 at fork_trampoline+0x8 ino 24168915, on dev da0s1d db> trace 768 Tracing pid 768 tid 100104 td 0xc74c6690 kdb_enter_why(c0b3b0a1,c0b3b0a1,c0b5d0ec,ed2dd85c,1,...) at kdb_enter_why+0x3a panic(c0b5d0ec,c70365c0,336eee,0,c0b5d164,...) at panic+0x136 ufs_dirbad(c9c40180,0,c0b5d164,0,ed2dd904,...) at ufs_dirbad+0x49 ufs_lookup(ed2dd94c,c0b6f08d,cb090160,cb090160,ed2ddba4,...) at ufs_lookup+0x377 VOP_CACHEDLOOKUP_APV(c0c0c1a0,ed2dd94c,ed2ddba4,ed2ddb90,c75dae00,...) at VOP_CACHEDLOOKUP_APV+0xc5 vfs_cache_lookup(ed2dd9d0,c0b471e4,cb090160,2,cb090160,...) at vfs_cache_lookup+0xd3 VOP_LOOKUP_APV(c0c0c1a0,ed2dd9d0,c74c6690,c0b46172,19b,...) at VOP_LOOKUP_APV+0xe5 lookup(ed2ddb7c,c0b46172,c6,bf,c703a72c,...) at lookup+0x59e namei(ed2ddb7c,c74c6690,ed2dda74,246,c0c61ab8,...) at namei+0x35e vn_open_cred(ed2ddb7c,ed2ddc78,1a4,c75dae00,c74871c8,...) at vn_open_cred+0x62 vn_open(ed2ddb7c,ed2ddc78,1a4,c74871c8,0,...) at vn_open+0x33 kern_open(c74c6690,8164100,0,a02,1a4,...) at kern_open+0xe7 open(c74c6690,ed2ddcfc,c,c0b40d50,c0beb278,...) at open+0x30 syscall(ed2ddd38) at syscall+0x2d3 Xint0x80_syscall() at Xint0x80_syscall+0x20 --- syscall (5, FreeBSD ELF32, open), eip = 0x281a161b, esp = 0xbfbfe8cc, ebp = 0xbfbfe8f8 --- db> trace 55 Tracing pid 55 tid 100054 td 0xc6fea460 sched_switch(c6fea460,0,1,17c,29bc06da,...) at sched_switch+0x49b mi_switch(1,0,c0b3ec35,1cb,c6fea460,...) at mi_switch+0x223 sleepq_switch(c6fea460,0,c0b3ec35,243,68,...) at sleepq_switch+0x182 sleepq_wait(c0c5666c,0,c0b3ba36,d8,0,...) at sleepq_wait+0x60 _sleep(c0c5666c,c0ca8988,68,c0b47461,0,...) at _sleep+0x369 sched_sync(0,eaff0d38,c0b37584,31c,c6febae0,...) at sched_sync+0x752 fork_exit(c0814220,0,eaff0d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xeaff0d70, ebp = 0 --- db> trace 4 Tracing pid 4 tid 100015 td 0xc6797460 sched_switch(c6797460,0,1,17c,5512f4a0,...) at sched_switch+0x49b mi_switch(1,0,c0b3ec35,1cb,4c,...) at mi_switch+0x223 sleepq_switch(c6797460,0,c0b3ec35,266,4c,...) at sleepq_switch+0x182 sleepq_timedwait(c0c5420c,64,c0b32247,0,0,...) at sleepq_timedwait+0x68 _sleep(c0c5420c,c0c54128,24c,c0b32247,64,...) at _sleep+0x345 g_io_schedule_down(c6797460,0,c0b339ab,74,0,...) at g_io_schedule_down+0x6b g_down_procbody(0,e6e39d38,c0b37584,31c,c67eb828,...) at g_down_procbody+0x8d fork_exit(c073d540,0,e6e39d38) at fork_exit+0xb8 fork_trampoline() at fork_trampoline+0x8 --- trap 0, eip = 0, esp = 0xe6e39d70, ebp = 0 --- db> continue ::Uptime: 7m21s ssttPhysical memory: 3310 MB Dumping 225 MB:aacckk bbaacckkttrraaccee:: ddbb__ttrraaccee__sseellff__wwrrappppeerr((cc00bb33ee003388,,eeadf3e4a8ab400c,,cc00881100333333,,00,,eeadf3e4a8ab400c,,......)) aatt ddbb__ttrraaccee__sseellff__wwrraappppeerr++00xx2266 kkddbb__bbaacckkttrraaccee((00,,ee 4a8ab400c,,cc00aaaacc99bb55,,eeadf3e4a8ab510c,, c b4623806dec00,,......)) aatt kkddbb__bbaacckkttrraaccee++00xx2299 vv f bbaaddlloocckk((c00cc00cc11aa00,,eeadf3e4a8ab510c,,cc00cc223388aa00,,ccbb4623806dc00,,00)) aatt vvffss__bbaaddlloocckk++00xx2233 aasssseerrtt__vvoopp__lloocckkeedd((ccbb4623806dec00,,cc ) aatt aasssseerrtt__vvoopp__lloocckkeedd++00xx5500 vvoopp__lloocckk__ppoosstt((eeda3f4e8aba7a8c,,00,,11020,21,21,0c0b242c8b66e300,d.c.0.,). .a.t) vaotp _vloopc_kl_opcoks_tp+o0sxt3+60 x P O _VLOOPC_KL1O_CAKP1V_(AcP0Vc(0cc01ca00c,1eaa0f,eeada3a48,bc708b,4c709bb427,93b222,,3c2b24,2c8b76b340,e.9.4.,). .a.t) VaOtP _VLOOPC_KL1O_CAKP1V_x0PxVc+10 nc_1v __lvonc_kl(occbk4(2c8b66e300,d1c20,,c160f0e2a, c 05,cc306b9404,7cc04b,487a93b,2.,.2.4)2 ,a.t. ._)v na_tl o_ + lxofc8k h0fxlfu8s 0bvunf_qwureiutees((cc760f5e2c62d48,,e0d,3c40bc46407,cc47,58d0abe,0604,,0.,.c.7)5 ca3t6 9f0l,u.s.h.b)u faqtu uvens_+w0rxi5t6e8+ dxb1u6fa_ adeomfoinl(e0w,reiatfee(aedd3384,8cc06b03,7f5f8f4f,f3f1fcf,,cf6fffefcf2bf8f,,.0.,.c)7 0a5t2 6bdu4f,_.d.a.e)m oant+ 0dxocf0 i efworrikt_ee+x0ixfr843_0w,r0i,teeavf(eca7d53c83)6 9a0t, 4f,oerdk3_4e8xci6t0+,08x1b281 o0f0o,r1k9_at,r.a.m.p)o laitn ek(e)r na_tw rfiotrekv_+t0rxa5m8p 5lwirniet+e0(xc87 4c-3-6-9 0t,readp3 408,c feci,pc ,=c 00b,4 0edscp5 ,=c 00bxeeba2f6e0a,d.7.0.,) eabtp w=r i0t e-+-0-x OfV tPs_yLsOcCaKl:l (0exdc3b4482d8368e)0 aits snyostc allolc+k0exd2 db3u y Xsihnotu0lxd8 0b_es sKc + aBl:l (e)n taetr :X ilnotc0kx 8v0i_oslyastciaolnl [thread pid 53 tid 100052 ] Stopped at kdb_enter_why+0x3a: movl $0,kdb_why db> <Enter> xD2B0: s-t-a-c ks ybsaccakltlr a(c4:, Fdrbe_etBrSaDc eE_LsFe3l2f,_ wwrraiptpee)r,( c0ibp3 e=0 308x,2e8a1fbe1a793e3c,, ce0s8p1 0=3 303x,b0f,befaef9e2ac9,e ce,b.p. .=) 0axtb fdbbf_etlr7a8ce-_-s-e 6fV_OwPr_aLpOpCeKr:+ 00xx2c6b ,3k0ddbc_0baicsk tnroatc el(o0c,keeadf ebau9te cs,hco0ualadc 9bbe5 deKaDfBe:a 9efnct,ecrb:4 2l8o6cek ,v.i.o.l)a taito nk [thread pid 766 tid 100137 ] Stopped at kdb_enter_why+0x3a: movl $0,kdb_why db> <Enter> r_DbBa:c ksttraaccke +b0axc2k9t bavcfes:_ addbl_otcrka(cce0_cs0ecl1f c _w,reaapfpeear9(fcc0,bc30ec023388,ae0d,3c4b84a2d8c6,ec00,801)0 3a3t3 ,vcf7s5_cb3a6d9l0o,cekd+304x82a3d ,,a.s.s.e)r ta_tv odpb__ltorcakceed_(scebl4f2_8w6rea0p,pce0rb+60fx22465 cck0dbb3_9bca4c9k,tdraa9c3e3(0c1705,c43,6.9.0.,)e da3t4 8aasdsce,rct0_avaocp9_bl5o,cekde3d4+80axe5c0, bV6O3P0_dScT0R,A.T.E.G)Y _aAtP Vk(dcb0_c 0 , c1kat0r,aecaef+e0axa239c cvbf4 ) 2_b7abd4l,occ0kc(2c30cca00c,1cab04,2e8d63e408,a.e.c.,)c 0act2 3V8OaP0_,ScTbR6A3T0EdGcY0_,AcP7V5+c03x699b0 bautf svtfrsa_tbeag , yl(occbk4+20x72e3c daas9s3e3r0t1_0v , c 0_0e,lco0cckae8d6(3c0b,6d3a09d3c300,7c00,b.6.f.4)8 0a,te db3u4f8sbt2rca,ce0g8y1+20dx6601, e0bcu0fcw1rai0t,e.(.d.a)9 3a3t0 1a0s,seearfte_avao9p0_,ecl0oacakfecde+10,xe5a3f aV OaPc_,W0R,I.T.E._)A PaVt( c 0ucf0wcr1iat0e,+e0dx314981b 4v,fcs7_5bci3o6_9a0w,rci0tbe4(7d9ab923,320420,,.0.,.c)0 ba4t4 7VcO4P,_8WaR4I,T8Ea3A,P.V.+.0)x 9a8t 3vvfns__wbriiot_ea(wcr7i0t5e2+60dx43,2e8d 48lcu6s0h,bcu7f5qduaeeu0e0s,(0c,6cf7e5cc23b689,00,,. . 4a7tc 4v,n8_0wbr, t 6t4e,+.0.x.2)3 1a f dfolfuislhebwurfiqtuee(ueeds3+408xc5690c, fbfufff_fd f b fonf(f0f,fefaff,e0a,dc3780,5c206bd347,5.8.4.,)3 1act, cd6offeicl2ebw8r,i.t.e.+)0 xa9t5 5ukfe_rdna_ewmroint+e0vx(c c07 02cf3o6r9k0_,e4x,ietd(3c4087cf6f08,3801,201,4e0a0f,e1a9da3,8.). .a)t afto rkke_renx_iwtr+i0txebv8+ kxf5o8r 0_wtrriatmep(ocl7i5nce3(6)9 0a,te df3o4r8kc_ftcr,acm,pco0lbi4n0ed+c05x,8c =b-e-b-2 6t0r,a.p. .0), aeti pw r=i t0e,+ 0exs4pf c s0yxsecaaflela(d1e7d03,04 8edb3p8 )= a0t -s-y-s EaVlOlP+_0SxT2RdA3T 0GXYi:n t00xxc8b04_2s8y6sec0a lils( )n oatt lXoicnkte0dx 8b0u_ts ysshcoaullld+ 0bxe2 K LD-B-:- esnytsecra:l ll o(c4k, vFiroeleaBtSiDo nE [thread pid 53 tid 100052 ] Stopped at kdb_enter_why+0x3a: movl $0,kdb_why db> panic [no answer... let's power cycle the server once more] -- Panagiotis J. Christias Network Management Center P.Christias@noc.ntua.gr National Technical Univ. of Athens, GREECE
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20081014222343.GA8706>