Skip site navigation (1)Skip section navigation (2)
Date:      Fri, 25 Jan 2013 00:36:19 -0800
From:      Jeremy Chadwick <jdc@koitsu.org>
To:        Alexander Motin <mav@FreeBSD.org>
Cc:        freebsd-fs@freebsd.org, avg@freebsd.org
Subject:   Re: disk "flipped" - a known problem?
Message-ID:  <20130125083619.GA51096@icarus.home.lan>
In-Reply-To: <50FED818.7070704@FreeBSD.org>
References:  <20130121221617.GA23909@icarus.home.lan> <50FED818.7070704@FreeBSD.org>

next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, Jan 22, 2013 at 08:19:04PM +0200, Alexander Motin wrote:
> On 22.01.2013 00:16, Jeremy Chadwick wrote:
> > (Please keep me CC'd as I am not subscribed)
> > 
> > WRT this:
> > 
> > http://lists.freebsd.org/pipermail/freebsd-fs/2013-January/016197.html
> > 
> > I can reproduce the first problem 100% of the time on my home system
> > here.  I can provide hardware specs if needed, but the important part is
> > that I'm using RELENG_9 / r245697, the controller is an ICH9R in AHCI
> > mode (and does not share an IRQ), hot-swap bays are in use, and I'm
> > using ahci.ko.
> > 
> > I also want to make this clear to Andriy: I'm not saying "there's a
> > problem with your disk".  In my case, I KNOW there's a problem with the
> > disk (that's the entire point to my tests! :-) ).
> > 
> > In my case the disk is a WD Raptor (150GB, circa 2006) that has a very
> > badly-designed firmware that goes completely catatonic when encountering
> > certain sector-level conditions.  That's not the problem though -- the
> > problem is with FreeBSD apparently getting confused as to the internal
> > state of its devices after a device falls off the bus and comes back.
> > Explanation:
> > 
> > 1. System powered off; disk is attached; system powered on, shows up as
> > ada5.  Can communicate with device in every way (the way I tend to test
> > simple I/O is to use "smartctl -a /dev/ada5").  This disk has no
> > filesystems or other "stuff" on it -- it's just a raw disk, so I believe
> > the g_wither_washer oddity does not apply in this situation.
> > 
> > 2. "dd if=/dev/zero of=/dev/ada5 bs=64k"
> > 
> > 3. Drive hits a bad sector which it cannot remap/deal with.  Drive
> > firmware design flaw results in drive becoming 100% stuck trying to
> > re-read the sector and work out internal decisions to do remapping or
> > not.  Drive audibly clicking during this time (not actuator arm being
> > reset to track 0 noise; some other mechanical issue).  Due to firmware
> > issue, drive remains in this state indefinitely.
> > 
> > 4. FreeBSD CAM reports repeated WRITE_FPDMA_QUEUED (i.e. writes using NCQ)
> > errors every 30 seconds (kern.cam.ada.default_timeout), for a total of 5
> > times (kern.cam.da.retry_count+1).
> > 
> > 5. FreeBSD spits out similar messages you see; retries exhausted,
> > cam_periph_alloc error, and devfs claims device removal.
> > 
> > 6. Drive is still catatonic of course.  Only way to reset the drive is
> > to power-cycle it.  Drive removed from hot-swap bay, let sit for 20
> > seconds, then is reinserted.
> > 
> > 7. FreeBSD sees the disk reappear, shows up much like it did during #1,
> > except...
> > 
> > 8. "smartctl -a /dev/ada5" claims no such device or unknown device type
> > (I forget which).  "ls -l /dev/ada5" shows an entry.  "camcontrol
> > devlist" shows the disk on the bus, yet I/O does not work.  If I
> > remember right, re-attempting the dd command returns some error (I
> > forget which).
> > 
> > 9. "camcontrol rescan all" stalls for quite some time when trying to
> > communicate with entry 5, but eventually does return (I think with some
> > error).  camcontrol reset all" works without a hitch.  "camcontrol
> > devlist" during this time shows the same disk on ada5 (which to me means
> > ATA IDENTIFY, i.e. vendor strings, etc. are reobtained somehow, meaning
> > I/O works at some level).
> > 
> > 10. System otherwise works fine, but the only way to bring back
> > usability of ada5 is to reboot ("shutdown -r now").
> > 
> > To me, this looks like FreeBSD at some layer within the kernel (or some
> > driver (I don't know which)) is internally confused about the true state
> > of things.
> > 
> > Alexander, do you have any ideas?
> > 
> > I can enable CAM debugging (I do use options CAMDEBUG so I can toggle
> > this with camcontrol) as well as take notes and do a full step-by-step
> > diagnosis (along with relevant kernel output seen during each phase) if
> > that would help you.  And I can test patches but not against -CURRENT
> > (will be a cold day in hell before I run that, sorry).
> 
> Command timeout itself is not a reason for AHCI driver to drop the disk,
> neither it is for CAM in case of payload requests. Disk can be dropped
> if controller report device absence detected by SATA PHY, or by errors
> during device reinitialization after reset by CAM SATA XPT.

I have some theories as to why this is happening and it relates to the
underlying design of the drive firmware and the drive controller used.
I could write some pseudo-code showing how I believe the drive behaves,
but it's really besides the point, as you point out below.

> What is interesting, is what exactly goes on after disk got stuck and
> you have removed it. In normal case controller should immediately report
> PHY status change, driver should run PHY reset and see that link is
> lost. It should trigger bus rescan for CAM, that should invalidate
> device. That should make dd abort with error. After dd gone, device
> should be destroyed and ready for reattachment.

Yup, that sounds exactly like what should happen.  I know that in
userland (dd) the command eventually does abort/fail with an error (I
believe I/O error or some other message), and that's good.  The device
disappearing can also be confirmed.  It's after the drive is
power-cycled (to bring it back online) where its re-tasted and I/O (at
the kernel level) works, but now userland utilities interfacing with
/dev/ada5 insist "unknown device" or "no such device".  It's easier to
show than it is to explain.  My theory is that there is some kind of
internal (kernel-level) "state" that is not being reset correctly when a
device is lost and then brought back.

> So it should be great if you start with the full verbose dmesg from the
> boot up to the moment when system becomes stable after disk removal. If
> it won't be enough, we can enable some more debugging with `camcontrol
> debug -IPXp BUS`, where BUS is the bus number from `camcontrol devlist`.

This is exactly what I needed; thank you!

I'll spend some time tomorrow collecting the data + documenting and will
provide the results once I've compiled them.  This will be more useful
than speculation on my part.

-- 
| Jeremy Chadwick                                   jdc@koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20130125083619.GA51096>