Date: Mon, 20 Feb 2017 15:56:22 -0500 From: "Kenneth D. Merry" <ken@FreeBSD.ORG> To: Dirk-Willem van Gulik <dirkx@webweaving.org> Cc: freebsd-hackers@freebsd.org Subject: Re: sa and mpt in FreeBSD 11.0-RELEASE-p2 Message-ID: <20170220205621.GA14597@mithlond.kdm.org> In-Reply-To: <C2E846C2-D96B-4429-8DC9-5612EF529F3E@webweaving.org> References: <C2E846C2-D96B-4429-8DC9-5612EF529F3E@webweaving.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Feb 20, 2017 at 21:04:11 +0100, Dirk-Willem van Gulik wrote: > A machine, recently upgraded from 8.4 to FreeBSD 11.0-RELEASE-p2 seems to have lost pretty much any and all performance on mpt(4) with its attached tape drives > > Read performance is around 50Mbyte/second - and write a paltry 100-200kbyte/second (and occasionally hitting 800kbyte/second) (from/to memory disk&/dev/null; no risk of shoe shining, compression off). > > SCSI bus looks happy; with no kernel messages. Output of the DLT script below. > > Normal LTO drives in a tape robot, G9 server; pretty much all SAS *except* for a U320 to tape drive with the performance issue. Active terminator. > > Does this ring a bell with anyone? Was anything changed in either the sa or mpt driver since 8.x ? > > One odd thing is that a 'dd(1)' -without- a block size (compression is off, data is a prepared urandom file on md disk) writes much faster -- while on 8.x one -had- to use a sane blockwise to get the 'normal' speed of around 120Mbyte/second. Could it be that one needs to fiddle with MAXPHYS (which AFAIK is a readonly sysctl). > What blocksize are you using? What blocksize were you using before? What application do you normally use to talk to the tape drive? One thing that did change is that I disabled I/O splitting by default in the sa(4) driver. That means that previously, you could write a 10MB block to the sa(4) character device, and it would get split up by physio(9) into a bunch of MAXPHYS-size chunks. The problem with that behavior is that you can get an error back, and not have a good indication of which block caused the problem, and how many blocks made it out to tape if a write fails. You can also have applications that think they're writing large blocks, but in reality aren't actually doing that. The application doesn't have control of what size block is going to tape. Now, you can't write more than the lowest common denominator supported by MAXPHYS, the controller and the tape drive. This lets the application see on a block by block basis what succeeded and what failed. It also gives the application control of the blocksize, up to the limit supported by the configuration. You can see what is possible with your current configuration by doing a 'mt status -v'. For example: # mt status -v Drive: sa0: <IBM ULTRIUM-HH5 G9N1> Serial Number: 101300520A --------------------------------- Mode Density Blocksize bpi Compression Current: 0x58:LTO-5 variable 384607 enabled (0x1) --------------------------------- Current Driver State: at rest. --------------------------------- Partition: 0 Calc File Number: 0 Calc Record Number: 0 Residual: 0 Reported File Number: 0 Reported Record Number: 0 Flags: BOP --------------------------------- Tape I/O parameters: Maximum I/O size allowed by driver and controller (maxio): 1081344 bytes Maximum I/O size reported by controller (cpi_maxio): 5197824 bytes Maximum block size supported by tape drive and media (max_blk): 8388608 bytes Minimum block size supported by tape drive and media (min_blk): 1 bytes Block granularity supported by tape drive and media (blk_gran): 0 bytes Maximum possible I/O size (max_effective_iosize): 1081344 bytes This particular machine has an LTO-5 attached via a Qlogic FC controller, and MAXPHYS is set to 1024*1056 (just over 1MB) in the kernel config file. If you want to re-enable I/O splitting (which I should have disabled by now), you can set the following loader tunable in /boot/loader.conf: kern.cam.sa.allow_io_split=1 Then reboot and re-run your test and see if the performance is any different. If so, you probably want to look at the blocksize you're using, and bump up MAXPHYS in your kernel configuration to a larger value. > Dw. > > DLT / http://www.freebsddiary.org/tape-testing.php > READING > Corrected errors with substantial delay: 0 > Corrected errors with possible delay : 0 > Total errors : 0 > Total errors corrected : 0 > Total times correction algorithm used : 0 > Total bytes processed : 8590352388 > Total corrected errors / GB : 0 > Total uncorrected errors : 0 > Read compression ratio : 101% > On tape Mbytes read : 2 > On tape kbytes read residual : 329320 > WRITING > Corrected errors with substantial delay: 154 > Corrected errors with possible delay : 0 > Total errors : 0 > Total errors corrected : 0 > Total times correction algorithm used : 168 > Total bytes processed : 4909148037124 > Total corrected errors / GB : 0 > Total uncorrected errors : 0 > Write compression ratio : 99% > Host requested Mbytes written : 1442 > Host requested kbytes written residual : 115996 > On tape Mbytes written : 1443 > On tape kbytes written residual : 0 > > camcontrol devlist > > <QUANTUM ULTRIUM 4 W52T> at scbus3 target 5 lun 0 (sa0,pass0) > <QUANTUM UHDL 0091> at scbus3 target 5 lun 1 (ch0,pass1) > <HP RAID 0 OK> at scbus4 target 0 lun 0 (pass2,da0) > <HP RAID 1(1+0) OK> at scbus6 target 0 lun 0 (pass3,da1) > <HP RAID 0 OK> at scbus6 target 1 lun 0 (pass4,da2) > <QUANTUM ULTRIUM 4 W52T> at scbus7 target 5 lun 0 (sa1,pass5) > <QUANTUM UHDL 0091> at scbus7 target 5 lun 1 (ch1,pass6) > > mt rblim > /dev/nsa0: > min blocksize 1 byte > max blocksize 16777215 bytes > granularity 1 byte > > mt status, ostatus and errstatus > > Mode Density Blocksize bpi Compression > Current: 0x46:LTO-4 variable 323215 disabled > ---------available modes--------- > 0: 0x46:LTO-4 variable 323215 0x1 > 1: 0x46:LTO-4 variable 323215 0x1 > 2: 0x46:LTO-4 variable 323215 0x1 > 3: 0x46:LTO-4 variable 323215 0x1 > --------------------------------- > Current Driver State: at rest. > --------------------------------- > File Number: 0 Record Number: 0 Residual Count 0 > > > Drive: sa0: <QUANTUM ULTRIUM 4 W52T> Serial Number: HU1027B53Y > --------------------------------- > Mode Density Blocksize bpi Compression > Current: 0x46:LTO-4 variable 323215 disabled > --------------------------------- > Current Driver State: at rest. > --------------------------------- > Partition: 0 Calc File Number: 0 Calc Record Number: 0 > Residual: 0 Reported File Number: 0 Reported Record Number: 0 > Flags: BOP What does 'mt status -v' show? That will show you the maximum blocksize that you can effectively send to the device. Ken -- Kenneth Merry ken@FreeBSD.ORG
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20170220205621.GA14597>