From owner-freebsd-current@FreeBSD.ORG Mon Mar 2 00:41:09 2015 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C728BB2; Mon, 2 Mar 2015 00:41:09 +0000 (UTC) Received: from clavin1.langille.org (clavin.langille.org [162.208.116.86]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "clavin.langille.org", Issuer "StartCom Class 2 Primary Intermediate Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9C8A9D17; Mon, 2 Mar 2015 00:41:09 +0000 (UTC) Received: from (clavin1.int.langille.org (clavin1.int.unixathome.org [10.4.7.7]) (Authenticated sender: hidden) with ESMTPSA id 8B385F6F ; Mon, 2 Mar 2015 00:41:08 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: sa(4) driver changes available for test From: Dan Langille In-Reply-To: <20150302003150.GB71528@mithlond.kdm.org> Date: Sun, 1 Mar 2015 19:41:07 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20150214003232.GA63990@mithlond.kdm.org> <7CA52DF3-E073-4F50-BE4E-01C51CCDF2C7@langille.org> <20150217183645.GA30947@mithlond.kdm.org> <303C48AC-0DA8-447C-8DE7-0A1DD5DD7351@langille.org> <20150302003150.GB71528@mithlond.kdm.org> To: "Kenneth D. Merry" X-Mailer: Apple Mail (2.2070.6) Cc: current@freebsd.org, scsi@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Mar 2015 00:41:09 -0000 > On Mar 1, 2015, at 7:31 PM, Kenneth D. Merry wrote: >=20 > On Sun, Mar 01, 2015 at 19:15:05 -0500, Dan Langille wrote: >>=20 >>> On Feb 17, 2015, at 1:36 PM, Kenneth D. Merry = wrote: >>>=20 >>> On Sat, Feb 14, 2015 at 18:22:43 -0500, Dan Langille wrote: >>>>=20 >>>>> On Feb 13, 2015, at 7:32 PM, Kenneth D. Merry = wrote: >>>>>=20 >>>>>=20 >>>>> I have a fairly large set of changes to the sa(4) driver and mt(1) = driver >>>>> that I'm planning to commit in the near future. >>>>>=20 >>>>> A description of the changes is here and below in this message. >>>>>=20 >>>>> If you have tape hardware and the inclination, I'd appreciate = testing and >>>>> feedback. >>>>=20 >>>> I have a DLT 8000 and an SDLT 220. >>>>=20 >>>> I don't have anything running current, but I have a spare machine = which I could use for testing. >>>>=20 >>>> Do you see any value is tests with that hardware? I'd be testing it = via Bacula. >>>>=20 >>>> disclosure: I'm the sysutils/bacula-* maintainer and a Bacula = committer. >>>>=20 >>>=20 >>> Actually, yes. Bacula is a bit tricky to configure, so your trying = it out >>> would be helpful if you have the time. >>>=20 >>> In looking at the manuals for both the SDLT 220 and the DLT 8000, = they both >>> claim to support long position information for the SCSI READ = POSITION >>> command. >>>=20 >>> You can see what I'm talking about by doing: >>>=20 >>> mt eod >>> mt status >>>=20 >>> On my DDS-4 tape drive, this shows: >>>=20 >>> # mt -f /dev/nsa3 status >>> Drive: sa3: Serial Number: HJ00YWY >>> --------------------------------- >>> Mode Density Blocksize bpi Compression >>> Current: 0x26:DDS-4 1024 bytes 97000 enabled = (DCLZ) >>> --------------------------------- >>> Current Driver State: at rest. >>> --------------------------------- >>> Partition: 0 Calc File Number: -1 Calc Record Number: -1 >>> Residual: 0 Reported File Number: -1 Reported Record Number: -1 >>> Flags: None >>>=20 >>> But on an LTO-5, which will give long position information, I get: >>>=20 >>> [root@doc ~]# mt status >>> Drive: sa0: >>> --------------------------------- >>> Mode Density Blocksize bpi Compression >>> Current: 0x58:LTO-5 variable 384607 enabled (0x1) >>> --------------------------------- >>> Current Driver State: at rest. >>> --------------------------------- >>> Partition: 0 Calc File Number: 2 Calc Record Number: -1 >>> Residual: 0 Reported File Number: 2 Reported Record Number: = 32373 >>> Flags: None >>>=20 >>> That, in combination with the changes I made to the position = information >>> code in the driver, mean that even the old MTIOCGET ioctl should = return an >>> accurate file number at end of data. e.g., on the LTO-5: >>>=20 >>> [root@doc ~]# mt ostatus >>> Mode Density Blocksize bpi Compression >>> Current: 0x58:LTO-5 variable 384607 0x1 >>> ---------available modes--------- >>> 0: 0x58:LTO-5 variable 384607 0x1 >>> 1: 0x58:LTO-5 variable 384607 0x1 >>> 2: 0x58:LTO-5 variable 384607 0x1 >>> 3: 0x58:LTO-5 variable 384607 0x1 >>> --------------------------------- >>> Current Driver State: at rest. >>> --------------------------------- >>> File Number: 2 Record Number: -1 Residual Count -1 >>>=20 >>> So the thing to try, in addition to just making sure that Bacula = continues >>> to work properly, is to try setting this for the tape drive in >>> bacula-sd.conf: >>>=20 >>> Hardware End of Medium =3D yes >>>=20 >>> It looks like the Bacula tape program (btape) has a test mode, and = it would >>> be good to run through the tests on one of the tape drives and see = whether >>> they work, and whether the results are different before and after = the >>> changes. I'm not sure how to enable the test mode. >>=20 >> I have this in /usr/local/etc/bacula/bacula-sd.conf >>=20 >> Device { >> Name =3D DLT >> Description =3D "QUANTUM DLT7000 1624" >> Media Type =3D DLT >> Archive Device =3D /dev/nsa1 >>=20 >> Autochanger =3D YES >> Drive Index =3D 0 >>=20 >> Offline On Unmount =3D no >> Hardware End of Medium =3D yes >> BSF at EOM =3D yes >> Backward Space Record =3D no >> Fast Forward Space File =3D no >> TWO EOF =3D yes >> } >>=20 >> FYI, http://www.freebsddiary.org/digital-tl891.php (from 2006) has a = btape test on this same model. >>=20 >> Here's the test I ran tonight: >>=20 >> [root@cuppy:/usr/home/dan] # btape -c = /usr/local/etc/bacula/bacula-sd.conf /dev/nsa1 = =20 >> Tape block granularity is 1024 bytes. >> btape: butil.c:287-0 Using device: "/dev/nsa1" for writing. >> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK >> *test >>=20 >> =3D=3D=3D Write, rewind, and re-read test =3D=3D=3D >>=20 >> I'm going to write 10000 records and an EOF >> then write 10000 records and an EOF, then rewind, >> and re-read the data to verify that it is correct. >>=20 >> This is an *essential* feature ... >>=20 >> btape: btape.c:1152-0 Wrote 10000 blocks of 64412 bytes. >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:1168-0 Wrote 10000 blocks of 64412 bytes. >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:1210-0 Rewind OK. >> 10000 blocks re-read correctly. >> Got EOF on tape. >> 10000 blocks re-read correctly. >> =3D=3D=3D Test Succeeded. End Write, rewind, and re-read test =3D=3D=3D= >>=20 >> btape: btape.c:1277-0 Block position test >> btape: btape.c:1289-0 Rewind OK. >> Reposition to file:block 0:4 >> Block 5 re-read correctly. >> Reposition to file:block 0:200 >> Block 201 re-read correctly. >> Reposition to file:block 0:9999 >> Block 10000 re-read correctly. >> Reposition to file:block 1:0 >> Block 10001 re-read correctly. >> Reposition to file:block 1:600 >> Block 10601 re-read correctly. >> Reposition to file:block 1:9999 >> Block 20000 re-read correctly. >> =3D=3D=3D Test Succeeded. End Write, rewind, and re-read test =3D=3D=3D= >>=20 >>=20 >>=20 >> =3D=3D=3D Append files test =3D=3D=3D >>=20 >> This test is essential to Bacula. >>=20 >> I'm going to write one record in file 0, >> two records in file 1, >> and three records in file 2 >>=20 >> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK >> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >> btape: btape.c:1420-0 Now moving to end of medium. >=20 > This is the critical piece. The test moves the tape to the end of the > medium. With hardware position information, you can tell what = filemark > you're on. Without it, you can't. >=20 >> btape: btape.c:622-0 tape_dev.c:345 ioctl MTIOCGET error on "DLT" = (/dev/nsa1). ERR=3DNo error: 0. >> We should be in file 3. I am at file 0. This is NOT correct!!!! >>=20 >> Append test failed. Attempting again. >> Setting "Hardware End of Medium =3D no >> and "Fast Forward Space File =3D no >> and retrying append test. >=20 > This is not surprsing, given that the drive doesn't support long read > position data. (It's a SCSI-2 device.) So that means that Bacula = will > need to do it manually. Yes, I have nothing newer than SCSI-2. Even my SDLT is SCSI-2 but that tape library is hooked up to a different computer and was doing backups = today. > =3D=3D=3D Append files test =3D=3D=3D >>=20 >> This test is essential to Bacula. >>=20 >> I'm going to write one record in file 0, >> two records in file 1, >> and three records in file 2 >>=20 >> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK >> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >> btape: btape.c:1420-0 Now moving to end of medium. >> btape: btape.c:625-0 Moved to end of medium. >> We should be in file 3. I am at file 3. This is correct! >>=20 >> Now the important part, I am going to attempt to append to the tape. >>=20 >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >> Done appending, there should be no I/O errors >>=20 >> Doing Bacula scan of blocks: >> 1 block of 64448 bytes in file 1 >> End of File mark. >> 2 blocks of 64448 bytes in file 2 >> End of File mark. >> 3 blocks of 64448 bytes in file 3 >> End of File mark. >> 1 block of 64448 bytes in file 4 >> End of File mark. >> Total files=3D4, blocks=3D7, bytes =3D 451,136 >> End scanning the tape. >> We should be in file 4. I am at file 4. This is correct! >>=20 >>=20 >> It looks like the test worked this time, please add: >>=20 >> Hardware End of Medium =3D No >>=20 >> Fast Forward Space File =3D No >> to your Device resource in the Storage conf file. >>=20 >> The above Bacula scan should have output identical to what follows. >> Please double check it ... >> =3D=3D=3D Sample correct output =3D=3D=3D >> 1 block of 64448 bytes in file 1 >> End of File mark. >> 2 blocks of 64448 bytes in file 2 >> End of File mark. >> 3 blocks of 64448 bytes in file 3 >> End of File mark. >> 1 block of 64448 bytes in file 4 >> End of File mark. >> Total files=3D4, blocks=3D7, bytes =3D 451,136 >> =3D=3D=3D End sample correct output =3D=3D=3D >>=20 >> If the above scan output is not identical to the >> sample output, you MUST correct the problem >> or Bacula will not be able to write multiple Jobs to=20 >> the tape. >>=20 >> Skipping read backwards test because BSR turned off. >>=20 >>=20 >> =3D=3D=3D Forward space files test =3D=3D=3D >>=20 >> This test is essential to Bacula. >>=20 >> I'm going to write five files then test forward spacing >>=20 >> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >> btape: btape.c:1909-0 Wrote block to device. >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >> btape: btape.c:1634-0 Now forward spacing 1 file. >> We should be in file 1. I am at file 1. This is correct! >> btape: btape.c:1646-0 Now forward spacing 2 files. >> We should be in file 3. I am at file 3. This is correct! >> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >> btape: btape.c:1659-0 Now forward spacing 4 files. >> We should be in file 4. I am at file 4. This is correct! >>=20 >> btape: btape.c:1677-0 Now forward spacing 1 more file. >> We should be in file 5. I am at file 5. This is correct! >>=20 >> =3D=3D=3D End Forward space files test =3D=3D=3D >>=20 >>=20 >> Ah, I see you have an autochanger configured. >> To test the autochanger you must have a blank tape >> that I can write on in Slot 1. >>=20 >> Do you wish to continue with the Autochanger test? (y/n): y >>=20 >>=20 >> =3D=3D=3D Autochanger test =3D=3D=3D >>=20 >> 3301 Issuing autochanger "loaded" command. >> Nothing loaded in the drive. OK. >> 3303 Issuing autochanger "load 1 0" command. >> 3303 Autochanger "load 1 0" status is OK. >> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK >> btape: btape.c:1564-0 Rewound "DLT" (/dev/nsa1) >> btape: btape.c:1571-0 Wrote EOF to "DLT" (/dev/nsa1) >>=20 >> The test autochanger worked!! >=20 > Great, thanks for running the test! Looks like things are working as = well > as they were before. >=20 >>>=20 >>>> I'll let the other Bacula devs know about this. They deal with the = hardware. I work on PostgreSQL. >>>>=20 >>>=20 >>> Thanks! If there are additional features they would like out of the = tape >>> driver, I'm happy to talk about it. (Or help if they'd like to use = the new >>> status reporting ioctl, MTIOCEXTGET or any of the other new ioctls.) >>=20 >> Errors are interesting to me. Especially corrected errors. They are = a good indicator of tape quality. >>=20 >=20 > Yes. At least on modern drives, there is a good bit available in the = log > pages. You can try seeing what logs your drive supports by installing = the > sg3_utils package and running 'sg_logs sa1 --all'. >=20 > Given the large amount of data available on some drives, and the = difficulty > distilling it down to a clear good/bad, I probably won't stick that in = the > 'mt status' output. >=20 > But you can certainly take a look at it and have an idea of whether = your > particular tape/drive are experiencing issues. That's a lot of output: = https://gist.github.com/dlangille/0e15a7fbf7acab56fd32 I will run some Bacula jobs soon. I'm still setting up config files. =E2=80=94=20 Dan Langille http://langille.org/