Date: Tue, 10 Nov 1998 14:47:35 -0800 (PST) From: Matthew Dillon <dillon@apollo.backplane.com> To: Peter Jeremy <peter.jeremy@auss2.alcatel.com.au> Cc: hackers@FreeBSD.ORG Subject: Re: SCSI tagged queueing and softupdates Message-ID: <199811102247.OAA16696@apollo.backplane.com> References: <98Nov11.084659est.40344@border.alcanet.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
:AFAIK, SCSI tagged queueing is the default in 3.0 (via cam or inside
:the ahc driver). (This was optionally enabled via AHC_TAGENABLE in 2.x
:and the 3.0 man page still describes this option, although the option
:is gone).
:
:According to the description, tagged queueing allows a SCSI device to
:re-order pending I/O requests in order to improve throughput. Soft
:updates, on the other hand, rely on the correct sequencing of physical
:writes to ensure that the disk metadata is consistent.
:
:Is this a real issue? If so, it would seem that tagged queueing must
:be disabled (at least for writes) when softupdates are enabled. It's
:not obvious to me that this is currently done (or even easy to do).
:
:Comments please.
:
:Peter
No, this is not an issue. The SCSI command set allows you to flag
tags for ordered sequencing. Reads, of course, can be done out of
order and that is the main advantage. Write ordering is a more
complex issue and I think (?) they are all simply flagged to be ordered.
Tags also mean that you avoid much of the per-transaction latencies
involved with SCSI command sequencing by 'absorbing it' in parallel with
the drive running the previousw N requested transactions.
The advantages, especially with CAM and softupdates, are phenominal.
I am running a production -current news reader box with three striped
18G drives for the spool (128 sector = 64 KByte stripe). Even with the
drive LEDs solid-on most of the time, the responsiveness of the news
system is incredible.
da1-da3, all:
da3: <SEAGATE ST118273W 5764> Fixed Direct Access SCSI2 device
da3: 40.0MB/s transfers (20.0MHz, offset 8, 16bit), Tagged Queueing Enabled
da3: 17366MB (35566480 512 byte sectors: 255H 63S/T 2213C)
The OS reports on the order of 50-60 tags per device later on, but I
don't have those logs any more.
nntp1:/news> iostat -oK da1 da2 da3 da4 5
tty da1 da2 da3 cpu
tin tout sps tps msps sps tps msps sps tps msps us ni sy in id
0 202859 69 14.5 2529 55 18.2 2419 55 18.2 6 0 18 4 72
0 71670 73 13.7 1509 71 14.1 1563 75 13.3 7 0 37 5 51
0 72358 109 9.2 2211 102 9.8 2110 105 9.5 4 0 18 4 73
0 72262 104 9.6 1992 96 10.4 2171 102 9.8 12 0 78 7 3
0 71625 73 13.7 1491 66 15.2 1511 69 14.5 8 0 36 4 52
0 71667 77 12.9 1830 83 12.1 1779 83 12.1 6 0 28 4 61
0 71861 89 11.2 1861 84 11.9 2091 96 10.4 4 0 16 3 77
0 72088 96 10.4 2017 97 10.4 2035 103 9.7 6 0 16 4 74
The drives are still doing over a megabyte/sec each with 16KByte
transfers and almost totally random seeks. This is with 366 online
users and taking a full feed 'burst' at the same time. And this
hasn't even reached saturation. At saturation the msps drops to
a very consistant 9ms.
Now watch what happens when, with 366 online users and an incoming feed
burst, I also copy my history file with dd. Oops, looks like the
feed burst ended about half way through my test. In anycase, *THIS*
is full saturation.
nntp1:/news> dd if=dhistory of=xx bs=128k
1771+1 records in
1771+1 records out
232208400 bytes transferred in 100.238914 secs (2316549 bytes/sec)
iostat -oK da1 da2 da3 da4 5
tty da1 da2 da3 cpu
tin tout sps tps msps sps tps msps sps tps msps us ni sy in id
0 72551 103 9.7 2560 106 9.5 2759 107 9.4 6 0 31 6 57
0 72681 100 10.0 2698 100 10.0 2738 100 10.0 11 0 59 7 22
0 72854 105 9.5 2866 110 9.1 2870 105 9.5 6 0 30 6 58
0 72954 109 9.1 2768 103 9.7 2754 97 10.4 6 0 42 4 48
0 73059 109 9.2 3004 104 9.6 3129 104 9.6 7 0 25 5 63
(feed burst into news box ends, dd still going)
0 74777 107 9.3 4709 104 9.6 4667 101 9.9 6 0 33 4 57
0 75155 123 8.1 5111 115 8.7 5151 120 8.3 5 0 31 5 59
0 74663 108 9.3 4812 115 8.7 4709 111 9.0 6 0 35 5 55
0 74818 111 9.0 5031 121 8.3 4879 114 8.8 8 0 34 6 52
0 75381 124 8.0 5393 124 8.0 5482 125 8.0 5 0 34 6 54
When the burst ends, and doing no manual copying, leaving only the online
readers accessing the disk, I get (not even close to saturation now):
nntp1:/news> iostat -oK da1 da2 da3 da4 5
tty da1 da2 da3 cpu
tin tout sps tps msps sps tps msps sps tps msps us ni sy in id
0 20 134 17 59.8 803 50 19.9 1204 67 14.9 6 0 18 4 72
0 71345 67 15.0 1172 58 17.2 1228 63 16.0 7 0 11 3 78
0 71389 80 12.5 1369 84 11.9 1473 83 12.0 4 0 12 3 80
0 71606 94 10.6 1629 90 11.1 1551 91 11.0 6 0 11 2 80
0 71187 61 16.3 1187 58 17.2 1100 60 16.7 5 0 12 3 80
-Matt
:--
:Peter Jeremy (VK2PJ) peter.jeremy@alcatel.com.au
:Alcatel Australia Limited
:41 Mandible St Phone: +61 2 9690 5019
:ALEXANDRIA NSW 2015 Fax: +61 2 9690 5247
:
:To Unsubscribe: send mail to majordomo@FreeBSD.org
:with "unsubscribe freebsd-hackers" in the body of the message
:
Matthew Dillon Engineering, HiWay Technologies, Inc. & BEST Internet
Communications & God knows what else.
<dillon@backplane.com> (Please include original email in any response)
To Unsubscribe: send mail to majordomo@FreeBSD.org
with "unsubscribe freebsd-hackers" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?199811102247.OAA16696>