Date: Thu, 11 Feb 1999 13:33:49 -0700 (MST) From: "Kenneth D. Merry" <ken@plutotech.com> To: andreas@klemm.gtn.com (Andreas Klemm) Cc: gibbs@pluto.plutotech.com, scsi@FreeBSD.ORG Subject: Re: 2.2.7/3.0-STABLE: tagged command queueing && poor write performance Message-ID: <199902112033.NAA30231@panzer.plutotech.com> In-Reply-To: <19990211203455.A19293@titan.klemm.gtn.com> from Andreas Klemm at "Feb 11, 1999 8:34:56 pm"
next in thread | previous in thread | raw e-mail | index | archive | help
Andreas: > I installed today a self made FreeBSD 3.0 SNAPSHOT from 990203 > on a HP Pentium II, 450 MHz machine, 128 MB RAM. > > da0 at ahc0 bus 0 target 0 lun 0 > da0: <SEAGATE ST39140W 1444> Fixed Direct Access SCSI-2 device > da0: 40.0MB/s transfers (20.0MHz, offset 8, 16bit), Tagged Queueing Enabled > da0: 8683MB (17783240 512 byte sectors: 255H 63S/T 1106C) > Everything went fine and I was glad until measuring the SCSI > disk performance using bonnie, which showed very poor values > for write performance: I think I know what the problem is. See below... > -------Sequential Output-------- ---Sequential Input-- --Random-- > -Per Char- --Block--- -Rewrite-- -Per Char- --Block--- --Seeks--- > Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU /sec %CPU > 200 2323 10.2 2504 3.2 2477 3.8 12545 54.7 12654 10.4 211.2 2.1 > 200 2377 10.4 1672 2.1 2502 3.9 11887 51.8 12003 10.0 211.9 2.1 > > First I thought it's due to the NCR controller. HP uses the > Symbios Logic Conbi controller, which has Fast Ethernet and > NCR SCSI on the card. But using an AHA 2940UW didn't solve > the problem. I doubt the controller would make a huge difference. The NCR controllers, especially the newer ones, have enough performance to handle your disk drive. > After installing FreeBSD 2.2.7 everything was fine, until I > build a custom kernel with Tagged Command Queuing. > Using Tagged Command Queuing the write performance decreased > from 12MB/sec to about 4 MB/sec !!! Turning off tagged queueing probably disabled most of the effects of the disk's write cache. See below for an explanation... > Stefan Esser told me I should use "iostat 1" to see, how many > MB/sec and transactions are reached. You see very little of > both when using tagged command queuing: > > These are the values when bonnie is doing buffered (intelligent) > write (200 MB). > > tty fd0 da0 sa0 cpu > tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id > 1 51 0.00 0 0.00 14.19 104 1.45 0.00 0 0.00 2 0 1 0 96 > 0 231 0.00 0 0.00 54.09 79 4.16 0.00 0 0.00 0 0 10 2 88 > 0 76 0.00 0 0.00 8.49 113 0.94 0.00 0 0.00 0 0 0 2 98 > 0 76 0.00 0 0.00 9.02 109 0.96 0.00 0 0.00 0 0 0 0100 > 0 76 0.00 0 0.00 8.50 111 0.92 0.00 0 0.00 0 0 1 2 98 > 0 76 0.00 0 0.00 18.45 106 1.91 0.00 0 0.00 1 0 11 0 88 > 0 0 0.00 0 0.00 44.89 84 3.69 0.00 0 0.00 0 0 0 0100 > 0 0 0.00 0 0.00 8.00 113 0.88 0.00 0 0.00 0 0 0 0100 > 0 0 0.00 0 0.00 8.00 111 0.87 0.00 0 0.00 0 0 0 0100 > 0 0 0.00 0 0.00 8.00 111 0.87 0.00 0 0.00 0 0 0 0100 > 0 0 0.00 0 0.00 9.00 111 0.97 0.00 0 0.00 0 0 0 0100 > 0 455 0.00 0 0.00 29.04 92 2.61 0.00 0 0.00 0 0 9 0 91 > 0 76 0.00 0 0.00 24.00 97 2.27 0.00 0 0.00 0 0 0 0100 > 0 76 0.00 0 0.00 8.00 113 0.88 0.00 0 0.00 0 0 0 0100 > 0 76 0.00 0 0.00 9.02 109 0.96 0.00 0 0.00 0 0 0 1 99 > 0 76 0.00 0 0.00 23.12 110 2.48 0.00 0 0.00 0 0 12 0 88 Yep, I've seen I/O patterns like that before... > We have the theory, that 64 tags are too many for the Seagate Disk. I don't think that's really the problem. > Is there a possibility to reduce the number of tags / disk using > syscontrol or camcontrol ? Or where can I reduce the maximum number > of tags for this specific harddisk ? The only way to reduce the number of tags for a given hard disk is to put a quirk entry in the transport layer. (sys/cam/cam_xpt.c) > Stefan told me, that it might have a negative impact on performance, > if so many tags are in use, because the harddisk hardly will service > other requests, if bonnie writes to it like hell ... > > Other observations are, that if tagged command queuing is active > and I start bonnie, I nearly can't login to a second vty. And > commands are delayed for about 10-15 seconds !!! That is, I think, more of a problem with CAM's queueing algorithms. Justin has talked of tuning them, or perhaps the way the upper layer code queues things. > If this theory should be right, that too many tags are useful > or only for special devices ... how can we handle this ??? > > Could we create a kernel option like > CAM_TAGS_MAX 16 > Or could it be tuneable via syscontrol or camcontrol ??? > > For now I'm only looking for a possibility to turn of tagged > command queuing on a 3.0-STABLE system or to reduce the number > of tags .... where is the hook for "rogue" entries .... and > how would an entry be, to reduce tags to 16 or so .... You can do it in cam_xpt.c. > In the attachement more infos .... kernel config file, messages > file, iostat output during the various phases of bonnie, and > an dumpfs of the /home filesys, where I made the bonnie bench > on. I did it on a 5 GB large virgin fs, the last 5 GB of the 9 GB > harddisk ... I think you're probably barking up the wrong tree here. The problem, I think, has to do with write caching, both in FreeBSD and on the disk itself. I have found that with or without softupdates, FreeBSD's write caching policy, when combined with write caching on a disk, can sometimes produce erratic performance for sequential writes. The solution is to disable write caching for that disk, and see if that helps you any. To disable write caching, edit mode page 8: camcontrol modepage -v -n da -u 0 -m 0x8 -e Change the "WCE" bit from 1 to 0, and save it. In my experience, this will definitely increase the throughput you get to the disk for sequential writes. For many of my machines, though, I leave write caching turned on, since I think that in the general case write caching on the disk will help random I/O performance. I am assuming here that your disk has write caching turned on. Many disk drive vendors ship with it turned on. (Seagate does, I think. IBM may not, but I think Quantum does.) If your disk doesn't have write caching enabled, then you've probably got some other problem. Ken -- Kenneth Merry ken@plutotech.com To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-scsi" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199902112033.NAA30231>