Date: Thu, 26 Jan 2012 22:43:52 -0500 From: Gary Palmer <gpalmer@freebsd.org> To: Jeremy Chadwick <freebsd@jdc.parodius.com> Cc: freebsd-stable@freebsd.org Subject: Re: Panic on 7.4-RELEASE-p5 Message-ID: <20120127034352.GG17973@in-addr.com> In-Reply-To: <20120127031351.GA67596@icarus.home.lan> References: <20120127024815.GD17973@in-addr.com> <20120127030906.GA67449@icarus.home.lan> <20120127031351.GA67596@icarus.home.lan>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thu, Jan 26, 2012 at 07:13:51PM -0800, Jeremy Chadwick wrote: > On Thu, Jan 26, 2012 at 07:09:06PM -0800, Jeremy Chadwick wrote: > > On Thu, Jan 26, 2012 at 09:48:15PM -0500, Gary Palmer wrote: > > > Hi, > > > > > > My Soekris firewall just panic'd > > > > > > Fatal trap 12: page fault while in kernel mode > > > fault virtual address = 0x14001d > > > fault code = supervisor write, page not present > > > instruction pointer = 0x20:0xc06fd2d3 > > > stack pointer = 0x28:0xd63557bc > > > frame pointer = 0x28:0xd63558a4 > > > code segment = base 0x0, limit 0xfffff, type 0x1b > > > = DPL 0, pres 1, def32 1, gran 1 > > > processor eflags = interrupt enabled, resume, IOPL = 0 > > > current process = 1869 (tcpdump) > > > trap number = 12 > > > panic: page fault > > > KDB: stack backtrace: > > > db_trace_self_wrapper(c079bbfc,c07fb800,c078d680,d6355660,d6355660,...) at db_tr > > > ace_self_wrapper+0x26 > > > panic(c078d680,c07b829c,c36d6d24,1,1,...) at panic+0xed > > > trap_fatal(c2c352d0,140000,2,8,d63556c8,...) at trap_fatal+0x234 > > > trap_pfault(c2b13620,0,c368f460,4,c36d6b00,...) at trap_pfault+0x27a > > > trap(d635577c) at trap+0x34e > > > calltrap() at calltrap+0x6 > > > --- trap 0xc, eip = 0xc06fd2d3, esp = 0xd63557bc, ebp = 0xd63558a4 --- > > > softdep_disk_io_initiation(c2ade474,1000,c3003738,1,c2aab9b8,...) at softdep_dis > > > k_io_initiation+0xb3 > > > ffs_geom_strategy(c3003738,c2aab9b8,1000,edb000,0,...) at ffs_geom_strategy+0x10 > > > c > > > ufs_strategy(d6355910,d6355910,c07f3980,c378bbdc,c2aab9b8,...) at ufs_strategy+0 > > > x64 > > > bufstrategy(c378bc9c,c2aab9b8,c368f460,c2aab9b8,c2bd46b4,...) at bufstrategy+0x2 > > > e > > > bufwrite(c2aab9b8,c2aabb04,20,c368f460,0,...) at bufwrite+0xf4 > > > cluster_wbuild(c378bbdc,4000,3b7,0,8,...) at cluster_wbuild+0x6c9 > > > cluster_write(c378bbdc,c2bd46b4,edc000,0,7f,...) at cluster_write+0x715 > > > ffs_write(d6355bc0,c07091e4,c378bc34,0,c378bc64,...) at ffs_write+0x837 > > > VOP_WRITE_APV(c07e9500,d6355bc0,c368f460,c07a2aec,252,...) at VOP_WRITE_APV+0xa0 > > > vn_write(c2e2adf4,d6355c54,c36d7500,0,c368f460,...) at vn_write+0x26f > > > dofilewrite(d6355c54,ffffffff,ffffffff,0,c2e2adf4,...) at dofilewrite+0x84 > > > kern_writev(c368f460,4,d6355c54,d6355c74,1,...) at kern_writev+0x58 > > > write(c368f460,d6355cf8,c,c0798e1b,39bfbd8f,...) at write+0x50 > > > syscall(d6355d38) at syscall+0x1b9 > > > Xint0x80_syscall() at Xint0x80_syscall+0x20 > > > --- syscall (4, FreeBSD ELF32, write), eip = 0x28354663, esp = 0xbfbfe85c, ebp = > > > 0xbfbfe878 --- > > > Uptime: 11d5h5m49s > > > Physical memory: 503 MB > > > Dumping 112 MB: 97 81 65 49 33 17 1 > > > Dump complete > > > Automatic reboot in 1 seconds - press a key on the console to abort > > > > > > fsck prevented the box from coming up automatically > > > > > > /dev/ufs/varlog: PARTIALLY TRUNCATED INODE I=1496123 > > > /dev/ufs/varlog: UNEXPECTED SOFT UPDATE INCONSISTENCY; RUN fsck MANUALLY. > > > THE FOLLOWING FILE SYSTEM HAD AN UNEXPECTED INCONSISTENCY: > > > ufs: /dev/ufs/varlog (/var/log) > > > > > > Anyone seen this before? Any clues? > > > > Please shed some light as to your filesystem and disk setup. > > Partitioning details, any GEOM layers you use, you get the idea. > > The more verbose the better. Please be sure to state which filesystems > > use softupdates and which do not (this matters a lot in this situation). > > I forgot to mention -- if the storage disk is physical and isn't > something like a CF card or similar, if you could provide smartctl -a > output from any attached disks (please be sure to state which output > correlates with which disk), I can review that to ensure there isn't an > underlying disk issue. kryten# smartctl -a /dev/ad1 smartctl 5.42 2011-10-20 r3458 [FreeBSD 7.4-RELEASE-p5 i386] (local build) Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Model Family: Hitachi Travelstar 5K250 Device Model: Hitachi HTS542512K9SA00 Serial Number: 071202BB0200WBHGW9XC LU WWN Device Id: 5 000cca 530d4db41 Firmware Version: BB2OC31P User Capacity: 120,034,123,776 bytes [120 GB] Sector Size: 512 bytes logical/physical Device is: In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 3f Local Time is: Fri Jan 27 03:39:16 2012 GMT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 645) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 56) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 100 100 062 Pre-fail Always - 0 2 Throughput_Performance 0x0005 100 100 040 Pre-fail Offline - 0 3 Spin_Up_Time 0x0007 253 253 033 Pre-fail Always - 0 4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 28 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 0 7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0 8 Seek_Time_Performance 0x0005 100 100 040 Pre-fail Offline - 0 9 Power_On_Hours 0x0012 023 023 000 Old_age Always - 33748 10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 28 191 G-Sense_Error_Rate 0x000a 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 4 193 Load_Cycle_Count 0x0012 001 001 000 Old_age Always - 7286353 194 Temperature_Celsius 0x0002 103 103 000 Old_age Always - 53 (Min/Max 17/58) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0 223 Load_Retry_Count 0x000a 100 100 000 Old_age Always - 0 SMART Error Log Version: 1 ATA Error Count: 239 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 239 occurred at disk power-on lifetime: 25188 hours (1049 days + 12 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 01 eb bb 12 e0 Error: UNC 1 sectors at LBA = 0x0012bbeb = 1227755 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 01 eb bb 12 e0 08 22d+07:23:48.580 READ DMA ca 00 20 a0 b3 a6 ed 08 22d+07:23:48.580 WRITE DMA ca 00 20 60 b3 a6 ed 08 22d+07:23:48.580 WRITE DMA ca 00 14 df ef 12 e0 08 22d+07:23:48.580 WRITE DMA c8 00 01 ea bb 12 e0 08 22d+07:23:44.380 READ DMA Error 238 occurred at disk power-on lifetime: 25188 hours (1049 days + 12 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 01 ea bb 12 e0 Error: UNC 1 sectors at LBA = 0x0012bbea = 1227754 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 01 ea bb 12 e0 08 22d+07:23:44.380 READ DMA ca 00 20 1f c1 12 e0 08 22d+07:23:44.380 WRITE DMA c8 00 01 ea bc 12 e0 08 22d+07:23:40.280 READ DMA ca 00 80 ff 8c 88 e4 08 22d+07:23:40.280 WRITE DMA ca 00 80 7f 8c 88 e4 08 22d+07:23:40.280 WRITE DMA Error 237 occurred at disk power-on lifetime: 25188 hours (1049 days + 12 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 01 ea bc 12 e0 Error: UNC 1 sectors at LBA = 0x0012bcea = 1228010 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 01 ea bc 12 e0 08 22d+07:23:40.280 READ DMA ca 00 80 ff 8c 88 e4 08 22d+07:23:40.280 WRITE DMA ca 00 80 7f 8c 88 e4 08 22d+07:23:40.280 WRITE DMA ca 00 20 5f 4b 87 e4 08 22d+07:23:40.280 WRITE DMA ca 00 14 df ef 12 e0 08 22d+07:23:40.280 WRITE DMA Error 236 occurred at disk power-on lifetime: 25188 hours (1049 days + 12 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 01 eb bb 12 e0 Error: UNC 1 sectors at LBA = 0x0012bbeb = 1227755 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 01 eb bb 12 e0 08 22d+07:23:36.280 READ DMA ca 00 20 7f 8c 88 e4 08 22d+07:23:36.280 WRITE DMA ca 00 40 3f 8c 88 e4 08 22d+07:23:36.280 WRITE DMA ca 00 20 1f 22 16 e0 08 22d+07:23:36.280 WRITE DMA ca 00 18 3f e9 12 e0 08 22d+07:23:36.280 WRITE DMA Error 235 occurred at disk power-on lifetime: 25188 hours (1049 days + 12 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 01 ea bb 12 e0 Error: UNC 1 sectors at LBA = 0x0012bbea = 1227754 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 01 ea bb 12 e0 08 22d+07:23:32.180 READ DMA ca 00 20 bf 54 69 eb 08 22d+07:23:32.180 WRITE DMA ca 00 20 3f 28 24 e3 08 22d+07:23:32.180 WRITE DMA ca 00 14 df ef 12 e0 08 22d+07:23:32.180 WRITE DMA ca 00 20 ff c1 12 e0 08 22d+07:23:32.180 WRITE DMA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed without error 00% 33078 - # 2 Short offline Completed without error 00% 33077 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. I noticed a while ago that there were some "bad" sectors on the disk, and at the time they were under the swap partition if my math was correct, and the box never swaps so it wasn't a problem. I don't know if the errors above are the same ones I saw earlier or not. There were no read or write errors on the console prior to the panic earlier today. In fact the previos output on the console relates to the last reboot for a software upgrade (fixing some packages) 11 days prior. The only thing in logs going back to November relating to ad1 are boot messages. Thanks, Gary
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120127034352.GG17973>