Date: Fri, 19 Aug 2011 20:24:38 -0700 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: Dan Langille <dan@langille.org> Cc: freebsd-stable@freebsd.org Subject: Re: bad sector in gmirror HDD Message-ID: <20110820032438.GA21925@icarus.home.lan> In-Reply-To: <B6B0AD0F-A74C-4F2C-88B0-101443D7831A@langille.org> References: <1B4FC0D8-60E6-49DA-BC52-688052C4DA51@langille.org> <20110819232125.GA4965@icarus.home.lan> <B6B0AD0F-A74C-4F2C-88B0-101443D7831A@langille.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Aug 19, 2011 at 09:39:17PM -0400, Dan Langille wrote: > > On Aug 19, 2011, at 7:21 PM, Jeremy Chadwick wrote: > > > On Fri, Aug 19, 2011 at 04:50:01PM -0400, Dan Langille wrote: > >> System in question: FreeBSD 8.2-STABLE #3: Thu Mar 3 04:52:04 GMT 2011 > >> > >> After a recent power failure, I'm seeing this in my logs: > >> > >> Aug 19 20:36:34 bast smartd[1575]: Device: /dev/ad2, 2 Currently unreadable (pending) sectors > > > > I doubt this is related to a power failure. > > > >> Searching on that error message, I was led to believe that identifying the bad sector and > >> running dd to read it would cause the HDD to reallocate that bad block. > >> > >> http://smartmontools.sourceforge.net/badblockhowto.html > > > > This is incorrect (meaning you've misunderstood what's written there). > > > > Unreadable LBAs can be a result of the LBA being actually bad (as in > > uncorrectable), or the LBA being marked "suspect". In either case the > > LBA will return an I/O error when read. > > > > If the LBAs are marked "suspect", the drive will perform re-analysis of > > the LBA (to determine if the LBA can be read and the data re-mapped, or > > if it cannot then the LBA is marked uncorrectable) when you **write** to > > the LBA. > > > > The above smartd output doesn't tell me much. Providing actual SMART > > attribute data (smartctl -a) for the drive would help. The brand of the > > drive, the firmware version, and the model all matter -- every drive > > behaves a little differently. > > Information such as this? http://beta.freebsddiary.org/smart-fixing-bad-sector.php Yes, perfect. Thank you. First thing first: upgrade smartmontools to 5.41. Your attributes will be the same after you do this (the drive is already in smartmontools' internal drive DB), but I often have to remind people that they really need to keep smartmontools updated as often as possible. The changes between versions are vast; this is especially important for people with SSDs (I'm responsible for submitting some recent improvements for Intel 320 and 510 SSDs). Anyway, the drive (albeit an old PATA Maxtor) appears to have three anomalies: 1) One confirmed reallocated LBA (SMART attribute 5) 2) One "suspect" LBA (SMART attribute 197) 3) A very high temperature of 51C (SMART attribute 194). If this drive is in an enclosure or in a system with no fans this would be understandable, otherwise this is a bit high. My home workstation which has only one case fan has a drive with more platters than your Maxtor, and it idles at ~38C. Possibly this drive has been undergoing constant I/O recently (which does greatly increase drive temperature)? Not sure. I'm not going to focus too much on this one. The SMART error log also indicates an LBA failure at the 26000 hour mark (which is 16 hours prior to when you did smartctl -a /dev/ad2). Whether that LBA is the remapped one or the suspect one is unknown. The LBA was 5566440. The SMART tests you did didn't really amount to anything; no surprise. short and long tests usually do not test the surface of the disk. There are some drives which do it on a long test, but as I said before, everything varies from drive to drive. Furthermore, on this model of drive, you cannot do a surface scans via SMART. Bummer. That's indicated in the "Offline data collection capabilities" section at the top, where it reads: No Selective Self-test supported. So you'll have to use the dd method. This takes longer than if surface scanning was supported by the drive, but is acceptable. I'll get to how to go about that in a moment. The reallocated LBA cannot be dealt with aside from re-creating the filesystem and telling it not to use the LBA. I see no flags in newfs(8) that indicate a way to specify LBAs to avoid. And we don't know what LBA it is so we can't refer to it right now anyway. As I said previously, I have no idea how UFS/FFS deals with this. Using fsck(8) is not sufficient; fsck does not attempt reading every LBA on the disk or every LBA that makes up the data portions of an inode. It only examines the "structure" of the filesystem. Is it possible the remapped LBA lived within a structure region and not data? Yes. Is it likely? Given the size of the disk, probably not. As mentioned previously too, there's badsect(8) but I don't know if it works correctly on present-day FreeBSD, if it works with larger drives, on 64-bit, etc... You get the idea. Plus as I said I don't know what LBA to tell it to avoid. You also need to keep something in mind: the terms "sector" and "LBA" are in some ways interchangeable and in other ways aren't. I use the term LBA because nobody in their right mind uses CHS addressing any more. badsect(8) claims it wants sectors, which I want to assume are LBAs. I hope someone familiar with UFS/FFS can explain how to go about this process for UFS/FFS. As for ZFS (because I know someone will ask) -- AFAIK there is no mechanism to deal with excluding certain LBAs from use. The attitude is that disks are cheap, if you see errors replace the disk. I agree with this attitude. You can "deal with" the error with ZFS if the pool consists of a mirror or raidzN, but you'll never be able to rid yourself of seeing R/W/CKSUM errors or possible I/O timeouts when accessing those LBAs. That's just how it goes. Anyway -- as for the "suspect" LBA -- we can absolutely determine what this one is and submit a write request to it to see if it turns out to be bad (uncorrectable) or if it's remappable. If remapped, see above explanation. Below is a script I wrote for scanning disks with dd. See script comments for how to use it. Quite simple. Things to note about the script because I'm 100% certain people will get all spun up about it: 1) It assumes 512-byte LBAs. Using this on an SSD or a 4KB-sector drive is probably not wise. 2) It's slow ("unintelligent"). This is by choice -- I wanted to keep it simple. It reads 512 bytes at a time, rather than larger chunks (e.g. 64k) and then "work down" to a smaller size when it encounters a read error to determine what LBA is responsible. I wanted something that "just worked" and wasn't fancy. There may be alternate utilities out there which do this (dd_rescue?). 3) I needed something that worked on Solaris and FreeBSD regardless of disk type. We use PATA, SATA, and SCSI disks at my workplace, and smartmontools really needs a rehaul for SATA on Solaris; so, shell scripting for the win. 4) I needed something that didn't depend on third-party tools I had to compile or deal with (see #5 though). 5) The hashbang refers to bash, though there aren't "bash-isms" in the script. The reason for this is Solaris; /bin/sh there is a non-evolved travesty that I loathe, so I write everything using /usr/local/bin/bash. You could, on FreeBSD, change this to /bin/sh and it should just work. That said: http://jdc.parodius.com/freebsd/bad_block_scan If you run this on your ad2 drive, I'm hoping what you'll find are two LBAs which can't be read -- one will be the remapped LBA and one will be the "suspect" LBA. If you only get one LBA error then that's fine too, and will be the "suspect" LBA. Once you have the LBA(s), you can submit writes to them to get the drive to re-analyse them (assuming they're "suspect"): dd if=/dev/zero of=/dev/XXX bs=512 count=1 seek=NNNNN Where XXX is the device and NNNNN is the LBA number. If this works properly, the dd command should sit there for a little bit (as the drive does its re-analysis magic) and then should complete. You'll want to check SMART stats after that; you should see Current_Pending_Sector drop to 0. If Offline_Uncorrectable incremented then the LBA could not be re-read/remapped. If Reallocated_Sector_Ct incremented then you now have a total of 2 LBAs which are remapped. In the case of remapping, you get to deal with the UFS/FFS thing above. To get the stats to update in this situation you *might* (but probably not) have to run "smartctl -t offline /dev/XXX". You might also be wondering "that dd command writes 512 bytes of zero to that LBA; what about the old data that was there, in the case that the drive remaps the LBA?" This is a great question, and one I've never actually taken the time to answer because at this present time I have absolutely *no* bad disks in my possession. I'm under the impression that the write does in fact write zeros if the LBA is remapped, but that might not be true at all. I've been waiting to test this for quite some time and document it/write about it. I still suggest you replace the drive, although given its age I doubt you'll be able to find a suitable replacement. I tend to keep disks like this around for testing/experimental purposes and not for actual use. Good luck! -- | Jeremy Chadwick jdc at parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, US | | Making life hard for others since 1977. PGP 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110820032438.GA21925>