Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 08 Feb 2011 23:07:21 -0800
From:      Greg Bonett <greg@bonett.org>
To:        Jeremy Chadwick <freebsd@jdc.parodius.com>
Cc:        freebsd-stable@freebsd.org
Subject:   Re: 8.1 amd64 lockup (maybe zfs or disk related)
Message-ID:  <1297235241.4729.35.camel@ubuntu>
In-Reply-To: <20110208064633.GA3367@icarus.home.lan>
References:  <1297026074.23922.8.camel@ubuntu> <20110207045501.GA15568@icarus.home.lan> <1297065041.754.12.camel@ubuntu> <20110207085537.GA20545@icarus.home.lan> <1297143276.9417.400.camel@ubuntu> <20110208055239.GA2557@icarus.home.lan> <1297145806.9417.413.camel@ubuntu> <20110208064633.GA3367@icarus.home.lan>

next in thread | previous in thread | raw e-mail | index | archive | help

--=-tk2Zc8zBx+cz/jsOhEgH
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: 7bit

ok, I think you're right - there is more than one problem with this
system, but I think I'm starting to isolate them and make some
progress.  

> # Debugging options
> options         BREAK_TO_DEBUGGER       # Sending a serial BREAK drops to DDB
> options         KDB                     # Enable kernel debugger support
> options         KDB_TRACE               # Print stack trace automatically on panic
> options         DDB                     # Support DDB
> options         GDB                     # Support remote GDB
> 
> Documented here:
> http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-options.html


rebuilt my kernel with debug options, but thankfully I think I've
learned how to avoid lockup for the time being.  I think I am asking too
much of my 650 watt power supply.  I unplugged one hard drive and
disabled another CPU core (now running 4 of 6).  I'm sad to lose the
horsepower, but I was able to complete an entire zpool scrub and other
high load tasks without a lockup. 


> Let's look at your storage controller setup:
> 
> atapci0: <JMicron JMB361 UDMA133 controller> irq 18
> atapci1: <AHCI SATA controller> on atapci0
>    ata2: <ATA channel 0> on atapci1
>    ata3: <ATA channel 1> on atapci1
>    ata4: <ATA channel 0> on atapci0
> atapci2: <ATI IXP700/800 SATA300 controller> irq 19
> atapci2: AHCI v1.20 controller with 4 6Gbps ports, PM supported
>    ata5: <ATA channel 0> on atapci2
>    ata6: <ATA channel 1> on atapci2
>    ata7: <ATA channel 2> on atapci2
>    ata8: <ATA channel 3> on atapci2
> atapci3: <ATI IXP700/800 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on pci0
>    ata0: <ATA channel 0> on atapci3
>    ata1: <ATA channel 1> on atapci3
> 
> There have been recent discussions about "problems" on the ATI
> IXP700/800 controllers.  I do not buy AMD systems, so I can't comment on
> this controllers' reliability.  Just a FYI point.  Here's the thread:
> 
> http://lists.freebsd.org/pipermail/freebsd-stable/2011-February/thread.html#61348
> 
> I also tend to avoid JMicron controllers like the plague.  I've seen too
> many problem reports with them over the years, regardless of OS.

I'll look into this.  I think the controller is the source of the
"FAILURE - READ_LMA48" errors.  I switched the disk/sata port pairing
and the error stayed with the sata port, not the disk.


> Now for the disk layout (I'm excluding da0, which is a USB flash disk of
> some kind).
> 
>  ad0: 953869MB <WDC WD10EARS-00Y5B1 80.00A80> at ata0-master UDMA133 SATA
>  ad1: 953869MB <Seagate ST31000333AS CC1H> at ata0-slave UDMA133 SATA
>  ad4: 1430799MB <WL1500GSA6472 05.00F.1> at ata2-master UDMA100 SATA 3Gb/s
>  ad8: 15279MB <TRANSCEND 20091215> at ata4-master UDMA66 
> acd0: CDRW <NEC CD-RW NR-7900A/1.08> at ata4-slave UDMA33 
> ad10: 953869MB <WL1000GSA1672 05.00J05> at ata5-master UDMA100 SATA 3Gb/s
> ad12: 953869MB <Seagate ST31000333AS CC1H> at ata6-master UDMA100 SATA 3Gb/s
> ad14: 953869MB <SAMSUNG HD103UJ 1AA01118> at ata7-master UDMA100 SATA 3Gb/s
> ad16: 953869MB <WL1000GSA1672 HA.00CHA> at ata8-master UDMA100 SATA 3Gb/s
> 
> You have a very large number of hard disks in this machine, so I sure
> hope you do have a decent enough PSU to handle it all.
> 
> If I had to make a recommendation, it would be to decrease the number of
> hard disks in the machine.  You have 8 of them -- one of which may be a
> RAM drive or something similar -- and that isn't including your CDRW
> drive.

Yes, I think this is the problem.  Though, for clarification, there are
only 6 spindle disks in the machine.  ad4 is an external drive over
esata (with it's own power), and ad8 is a CF drive.


> I would also try getting rid of the JMicron controller; I would
> recommend investing in a Silicon Image controller to replace it,
> specifically one driven by the 3124, 3132, or 3531 chips.  Avoid the
> 3112, 3114, and 3512 chips:
> http://en.wikipedia.org/wiki/Silicon_Image#Product_alerts

Thanks for the recommendation.  I'll probably pick one of these up along
with a new power supply.


> Next we have this:
> 
> > ad1: TIMEOUT - READ_DMA retrying (1 retry left) LBA=1
> > GEOM: ad1: partition 1 does not start on a track boundary.
> > GEOM: ad1: partition 1 does not end on a track boundary.
> > GEOM: label/1TBdisk5: partition 1 does not start on a track boundary.
> > GEOM: label/1TBdisk5: partition 1 does not end on a track boundary.
> 
> This doesn't look good, especially the READ_DMA timeout on ad1.  That's
> a different disk than the one you told me about before.  LBA 1 is
> literally the 2nd block on the disk, which is a little too close to
> block 0 for comfort.  I'd love to see "smartctl -a /dev/ad1" output
> here.

I've attached the output of smartctl -a /dev/ad1.  I don't think this
error is being caused by the disk though.  As I said above, I changed
the sata port / drive pairing and this error stays with the sata port,
not the drive.  (so, as you said, time for a new controller)

> > calcru: runtime went backwards from 82 usec to 70 usec for pid 20 (flowcleaner)
> > calcru: runtime went backwards from 363 usec to 317 usec for pid 8 (pagedaemon)
> > calcru: runtime went backwards from 111 usec to 95 usec for pid 7 (xpt_thrd)
> > calcru: runtime went backwards from 1892 usec to 1629 usec for pid 1 (init)
> > calcru: runtime went backwards from 6786 usec to 6591 usec for pid 0 (kernel)
> 
> This is a problem that has plagued FreeBSD for some time.  It's usually
> caused by EIST (est) being used, but that's on Intel platforms.  AMD has
> something similar called Cool'n'Quiet (see cpufreq(4) man page).  Are
> you running powerd(8) on this system?  If so, try disabling that and see
> if these go away.

sadly, I don't know if I'm running powerd. 
ps aux | grep power gives nothing, so no I guess...
as far as I can tell, this error is the least of my problems right now,
but i would like to fix it.

> > GEOM_ELI: Device label/1tbgreendisk.eli created.
> > GEOM_ELI: Encryption: AES-CBC 256
> > GEOM_ELI:     Crypto: software
> > {...}
> 
> There was no mention of geli(8) being used on this system until now.
> There may be other complexities as a result of this; I don't know.

yeah, geli is being used on this system, sorry i forgot to mention that 

> Good luck.
> 

Thanks for the help, I'm at least able to keep the machine online now.

--=-tk2Zc8zBx+cz/jsOhEgH
Content-Disposition: attachment; filename="ad1.smart"
Content-Type: text/plain; name="ad1.smart"; charset="UTF-8"
Content-Transfer-Encoding: 7bit

smartctl 5.40 2010-10-16 r3189 [FreeBSD 8.1-RELEASE-p2 amd64] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 7200.11 family
Device Model:     ST31000333AS
Serial Number:    9TE1MB10
Firmware Version: CC1H
User Capacity:    1,000,204,886,016 bytes
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 4
Local Time is:    Tue Feb  8 07:41:31 2011 PST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82)	Offline data collection activity
					was completed without error.
					Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		 ( 617) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   1) minutes.
Extended self-test routine
recommended polling time: 	 ( 208) minutes.
Conveyance self-test routine
recommended polling time: 	 (   2) minutes.
SCT capabilities: 	       (0x103f)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   120   099   006    Pre-fail  Always       -       243069120
  3 Spin_Up_Time            0x0003   100   100   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       84
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   079   060   030    Pre-fail  Always       -       83902794
  9 Power_On_Hours          0x0032   084   084   000    Old_age   Always       -       14308
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       84
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   093   000    Old_age   Always       -       56
189 High_Fly_Writes         0x003a   017   017   000    Old_age   Always       -       83
190 Airflow_Temperature_Cel 0x0022   076   051   045    Old_age   Always       -       24 (Min/Max 24/24)
194 Temperature_Celsius     0x0022   024   049   000    Old_age   Always       -       24 (0 17 0 0)
195 Hardware_ECC_Recovered  0x001a   050   019   000    Old_age   Always       -       243069120
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       96619584305016
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       412576321
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       2438661969

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     13221         -
# 2  Extended offline    Interrupted (host reset)      90%     13216         -
# 3  Short offline       Completed without error       00%     13207         -
# 4  Extended offline    Interrupted (host reset)      50%     13199         -
# 5  Extended offline    Completed without error       00%     13134         -
# 6  Conveyance offline  Completed without error       00%     13131         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.


--=-tk2Zc8zBx+cz/jsOhEgH--




Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1297235241.4729.35.camel>