Date: Mon, 27 Apr 2009 13:53:17 GMT From: Peter Steele <psteele@maxiscale.com> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/134044: Gmirror overwrites fs with stale data from returning member Message-ID: <200904271353.n3RDrH9D060353@www.freebsd.org> Resent-Message-ID: <200904271400.n3RE04Eg036236@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 134044 >Category: kern >Synopsis: Gmirror overwrites fs with stale data from returning member >Confidential: no >Severity: critical >Priority: high >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Apr 27 14:00:04 UTC 2009 >Closed-Date: >Last-Modified: >Originator: Peter Steele >Release: 7.0-RELEASE-p9 >Organization: MaxiScale Inc. >Environment: FreeBSD r02s17 7.0-RELEASE-p9 FreeBSD 7.0-RELEASE-p9 #2: Thu Apr 2 22:09:33 UTC 2009 root@r04s25:/usr/obj/usr/src/sys/GENERIC amd64 >Description: We had a four member gmirror'ed slice and one of the members died. This member happened to be drive 0 on our system (which mapped to ad4). On a subsequent reboot the system came back up with a reduced 3/4 mirror but otherwise everything worked as expected. Some additional software was installed on the system and then we shut it down again to deal with the faulty drive. We discovered it wasn't pushed in snuggly enough so the fix was easy. However, when the system came back up, instead of the gmirror driver reinserting that old drive into the existing mirror, the driver decided to use that drive as the primary member and instead tried to insert the other three drives into that mirror. This failed leaving a mirror consisting of a single drive ad4. When the system finished booting, we have a service that runs automatically to check the status of the mirror. This service automatically reinserts any drive that isn't participating in the mirror, to make sure the mirror is always fully populated. When this happened though we lost the new data that had been installed on the mirrored file system while ad4 was absent. It got overwritten by the stale data from ad4. Not good. >How-To-Repeat: This problem is easy to reproduce. Create a mirrored slice with > 2 members. (In fact we have only observed the problem with a four member mirror and cannot say for sure if it would happen on a three member mirror.) Shut the system down and remove drive 0 (the first drive) from the system. Reboot the system. This should result in a degraded mirror with drive 0 missing. Create some new files on the file system, then shut the system down again. Reinsert drive 0 and reboot the system. As the gmirror driver starts up, instead of it reinserting drive 0 into the existing mirror, the driver will instead assume drive 0 is the "good drive" and try to add the other drives to that mirror, even though they have the more recent data. This will fail during the boot phase, leaving a mirror consisting only of drive 0 and its stale data. This problem only happens with drive 0. If the same steps are repeated with say drive 1, when the drive is reinserted and the system rebooted, it is added back to the existing mirror instead of becoming the sole member of the mirror with its stale data. >Fix: None has been discovered, beyond making sure that a drive 0 is fully formatted before being reinserted into an existing cluster. This is not a practical solution in the field; you can never be sure what a customer will do. >Release-Note: >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200904271353.n3RDrH9D060353>