Date: Thu, 22 Oct 1998 17:13:09 -0700 From: Don Lewis <Don.Lewis@tsc.tdk.com> To: freebsd-fs@FreeBSD.ORG, freebsd-scsi@FreeBSD.ORG Subject: Re: filesystem safety and SCSI disk write caching Message-ID: <199810230013.RAA19305@salsa.gv.tsc.tdk.com>
next in thread | raw e-mail | index | archive | help
On Oct 14, 9:01am, Nate Williams wrote: } Subject: Re: filesystem safety and SCSI disk write caching } Bottom line is that by default FreeBSD w/SoftUpdates is more *unstable* } now with CAM than it was w/out CAM for 99% of the users. That's certainly not my experience. Once the initiate_write_filepage and newdirrem panics were fixed, my system has been completely stable. Since then only filesystem damage that has happened is when I had write caching enabled and I was playing with the reset button. I'm very anxious to upgrade our pre-CAM systems here because of stability problems I've been having with them. On Oct 14, 9:02am, "Justin T. Gibbs" wrote: } Subject: Re: filesystem safety and SCSI disk write caching } Neither the old SCSI code nor the } new CAM code has ever modified the caching behavior of the devices attached } on the bus. My position on this is that we should *never* modify device } mode parameters unless instructed to do so by the end user. I agree. } If the user } community insists that we add a warning about the cache being enabled, now, } after years of silently ignoring the effects of the cache on filesystem } integrity, so be it. My opinion is that if this was a problem in practice, } we'd have heard about it by now from our user community. I think much of the user community is uneducated on some of the finer points and also has low expectations. On Oct 14, 9:18am, Nate Williams wrote: } Subject: Re: filesystem safety and SCSI disk write caching } Up till this point, we never had any FS code that attempted to deal with } 'crashing' robustly. Not true. Unless you were using async mounts, the filesystem code was careful to order certain writes so that there would not be any unrecoverable damage. } I for one knew that if my machine crashed, an fsck } was expected and it was going to repair my disk. It still is necessary to recover lost resources, though in theory it could be done after the filesystem is mounted. With softupdates, it isn't strictly necessary to do it before mount time because blocks are no longer marked free in the bitmaps before they have been deallocated. The traditional code wasn't quite so careful in the interest of performance, since fsck could easily fix the bitmaps. } This is no longer the } case with SoftUpdates, so what was previously attributed to 'the crash' } can now be more fine-tuned to 'write caching screwed up' or 'I have a } bad power supply which breaks my drive when I hit the reset button' or } even simply 'when I lose power, write-caching spams my disk'. These problems can damage the filesystem in ways that fsck can't recover without data loss. On May 13, 5:54pm, "Christopher R. Bowman" wrote: } Subject: Re: filesystem safety and SCSI disk write caching } At 11:08 AM 10/14/98 -0700, Julian Elischer wrote: } >7/ to allow for this to be achieved easily, there should be an easy way to } >ensure that the write cache is turned off. Possibly as simple as } >a good example in camctl.8 . ... and a note in the handbook reminding folks of the importance of doing this when building a system or adding new disks. } Could we make this a mount time option, say if -wc to turn write caching on, } -nowc to turn it off, and if neither flag is present use whatever the drive is } already set for. My initial request was for the opposite, a warning if write caching was on, to keep folks from silently shooting themselves in the foot. This avoids the problems that Justin mentioned in his reply. This could even be tweaked to warn you if write caching was not in the state you desired. However, I now think this isn't the way to go. See below. On Oct 14, 4:54pm, "Justin T. Gibbs" wrote: } Subject: Re: filesystem safety and SCSI disk write caching } The moral of this story is that everyone should decide what kinds of } performance/safety tradeoffs they are willing to make and design their } systems accordingly. Yes, this sounds like an issue that should be discussed in the handbook. On Oct 14, 7:15pm, David Kelly wrote: } Subject: Re: filesystem safety and SCSI disk write caching } Before you fight it too much more, replace the power supply. I've cured } a number of "impossible" problems with a new power supply. One } spectacular example was a Power Mac 7200/120. Crash, crash, crash. } Sometimes it would run for 30 minutes. Sometimes overnight. Technician } replaced everything several times over a couple of weeks. Everything } but the plastic case and the power supply. I insisted on a new PS the } last time back. And it worked like a charm. In normal operation, the system is absolutely stable. The only problem occurs when I hit reset while the system is busy. If I turn off write caching, I haven't gotten any filesystem damage even then. } Power supply filter capacitors age with heat. And lose their ability to } be good capacitors. No telling what kind of noise is on your DC power } wires inside the case. Your PS could be generating a spike of its own } on RESET when/if something suddenly demands a lot of current. Or if } something suddenly quits demanding the current it was using. The capacitors "should" be good since the system is fairly new and has generally only been lightly used. If the problem was load regulation, then I'd expect the problem to also occur during heavy use, but I haven't seen any problems like that. One can never be sure though. On Oct 14, 11:19pm, Dan Nelson wrote: } Subject: Re: filesystem safety and SCSI disk write caching } I humbly submit the following script, to be added to /etc/security, or } periodic/weekly, /etc/rc, or wherever. It's dependant on the exact } output of "camcontrol inquiry" and "camcontrol modepage", but does the } job. Ah, a positive contribution to this thread! I was coming to the conclusion that a script to check this was probably the way to go when I saw your message. You can also use something like this to check some of the other SCSI control bits to make sure things like error recovery are also properly configured. I'd also combine this with a scan for new grown defects. I'd recommend running the script both at boot time and from cron to detect any potential problems. Using a script also avoids kernel bloat. On Oct 16, 6:09am, David Kelly wrote: } Subject: Re: filesystem safety and SCSI disk write caching } Why not? It might be interesting to put a recording voltmeter such as a } digital storage oscilloscope on the HD power leads when Don is punching } the reset. No telling what kind of voltage surges are generated when } the load on the power supply is altered. Ok, so the chapter in the handbook about SCSI write caching will recommend connecting a recording voltmeter to the power supply and monitoring it under varying load (including when the reset button is hit) to make sure there are no problems before enabling write caching? Repeat this procedure after adding new hardware and periodically as the power supply capacitors age. And you still can't prove that you don't have a power supply problem lurking, since you can't prove a negative. All you can state is that the power looks clean under the conditions that you tested. Problems still might occur when the machine is placed in service. On Oct 16, 7:26pm, Ollivier Robert wrote: } Subject: Re: filesystem safety and SCSI disk write caching } I agree. HP-UX has a kernel option to enable write caching and it is off by } default. Not that I'd advocate to do things like HP-SUX but I think it is } better to be safe. I think you are thinking of NFS write caching. An NFS server isn't supposed to tell the NFS client that the write has completed until the data is on stable storage (so the client can retry the write if the server crashes and reboots before the write is done). This badly hurts performance unless the client is able to handle a lot of outstanding write requests. } A lot if not all of the modern drives are shipped with WCE == 1 though. That didn't use to be the case. I think this changed a few years ago. Benchmarkitis no doubt. I didn't even think to check this since I haven't been putting too many new drives in service lately. On Oct 17, 1:48pm, Bill Vermillion wrote: } Subject: Re: filesystem safety and SCSI disk write caching } Guido van Rooij recently said: } > I always thought a drive will always be able to flush its write cache } > to disk, even when power fails. } } Not all do. The high-end IBM's do/did. They used the inertia of } the spinnging platters to generate enough current to flush the } buffers to disk. There aren't a lot of drives that do it. That could be pretty hard to do with the sizeable caches on drives these days. Quite a few seeks could be necessary which are expensive in terms of power. I also suspect that the necessary circuitry to recover the energy from the platters to power the rest of the drive adds a significant amount of cost to the drive, and the drive manufacturers are under quite a lot of cost pressure these days. This would be a disincentive to include a seldom used feature. On Oct 19, 12:04pm, "Justin T. Gibbs" wrote: } Subject: Re: filesystem safety and SCSI disk write caching } >, *after* you explain why } >Don Lewis is seeing the empirical behaviour he is seeing, in } >contradiction to your claims of what's possible and not. } } I've already given my opinion on this. I believe the Hawk is seeing } a power glitch or temporary power loss when the reset switch is hit and } so the contents of the cache are lost. I have never said that the } behavior that Don Lewis is seeing is 'not possible', only that, for } the drive in question, the reset causing cache corruption is not likely. If this problem caused by a load related power glitch, then it is possible to get silent filesystem corruption in normal operation if write caching is enabled, since cached writes could get lost and the driver might never notice. Without write caching, the driver would see transactions timing out and it would have an opportunity to retry the writes, preventing filesystem damage and data loss, and the driver would no doubt complain verbosely. This would alert the operator to the hardware problem. To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-fs" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199810230013.RAA19305>