Date: Sat, 27 Dec 2003 12:05:43 -0500 (EST) From: John Linwood Griffin <griffin2+freebsd-scsi@ece.cmu.edu> To: freebsd-scsi@freebsd.org Subject: scsi_target experiences and questions Message-ID: <Pine.LNX.4.53L-ECE.CMU.EDU.0312271201350.21079@glee.pdl.cmu.edu>
next in thread | raw e-mail | index | archive | help
Greetings to all, In regards the scsi_target code, I'd like to share some experiences and ask a few questions. I've been working with scsi_target with FreeBSD 4.8-R and 5.1-R targets and mostly Linux 2.4.[18,20] initiators, using both Fibre Channel (Qlogic QLA2100) and Parallel SCSI (Adaptec 29160) connections. First, the experiences: 1) Chuck Tuffli wrote to this list in August about having trouble using anything but LUN "0" when configuring a Qlogic card in target mode. (The specific error was "enable lun CCB rejected, status 0x39".) I had this problem also, and it seemed to disappear after I enabled the card's BIOS in the Alt-Q setup utility. Also in the BIOS, I set the execution throttle to 255, because 256 was incorrectly read as "0" during system boot. 2) I had a problem that the cam/scsi/scsi_target kernel code wouldn't shut down cleanly: The TARGIOCDISABLE call would never return, and the system had to be reboot between emulator invocations. I tracked this down to the "tsleep()" (msleep() in 5.1?) that immediately follows the comment "If we aborted at least one pending CCB ok, wait for it." I commented out the tsleep() call, which appears to allow it to be cleanly shut down and restarted multiple times. (Perhaps that's causing some kernel state to not get cleaned up?) This was using the fibre channel card -- I haven't tested whether the problem exists using parallel SCSI. 3) I had to disable the "pending unit attention" that is set to indicate the device's "powering on". (I.e., I commented out the line "istate->pending_ua = UA_POWER_ON" in scsi_cmds.c.) This was because a FreeBSD initiator wasn't correctly handling the CHECK CONDITION status received during the READ CAPACITY command, and instead tried reading an invalid geometry. (This was also only tested with fibre channel. Also, it was before I tried enabling autosense, which may fix the problem.) Without this comment, the device was seen as: da1: <FreeBSD Emulated Disk 0.1> Fixed Direct Access SCSI-3 device da1: 100.000MB/s transfers da1: 0MB (268784067 0 byte sectors: 0H 0S/T 0C) Once the comment was in place, it was correctly read as: da1: <FreeBSD Emulated Disk 0.1> Fixed Direct Access SCSI-3 device da1: 100.000MB/s transfers da1: 20MB (40960 512 byte sectors: 64H 32S/T 20C) 4) This isn't specifically related to the scsi_target code, but I was surprised to find that the AIO code has a race condition: if you aio_write block X, then immediately synchronously read block X before the AIO code commits the written block to disk, you'll get the old version of X during the read. I worked around this by always doing synchronous writes. 5) Kudos to Nate Lawson and Justin Gibbs for all their good work. I recommend that Nate add a "FAQ" section to his scsi_target page with some of the other relevant goodies that have been posted to this list (for example, Kenneth Merry posted some gems about enabling autosense and increasing MAX_INITIATORS); that would aid in us users getting working systems more quickly. 6) I think I've finally figured out the execution path for READ and WRITE requests through the scsi_target.c and scsi_cmds.c files. Nate is either an unparalleled genius (for getting it to work in the first place) or a diabolical masochist (for knowing that the rest of us would tear our hair out following his multiple re-entries into the tcmd_handle() function). Going off the top of my head, here are a few pointers for anyone else digging through the code. (Note that I may be wrong in places.) "ATIO" stands for "Accept Target I/O". At startup, the user code allocates space for a bunch of ATIOs and gives them to the kernel. When new requests arrive on the SCSI bus, the kernel sends one of these ATIOs back to the user. "CTIO" stands for "Continue Target I/O". CTIOs are used to transfer data between the user and kernel processes. This is necessary because each CTIO can only handle DFLTPHYS (or maybe MAXPHYS) amount of data, but SCSI requests can be larger than that. Once the request is done processing, the user code sends the ATIO back to the kernel for reuse. The user signals that a READ request is complete [i.e., then the kernel code is permitted to finish the bus transaction] by sending the final CTIO, and it signals that a WRITE request is complete by sending the final ATIO. I found it very useful to print out function names (as well as ATIO and CTIO pointers) as functions were entered; this greatly helped my understanding of how READ and WRITE requests were processed. Now, for a few questions: A) How difficult will it be to add tagged queuing support to the Adaptec driver? (Or, is the missing piece simply support in the scsi_target user code?) I would find this capability extremely useful. B) I've been having a maddening problem using tagged queuing with fibre channel. When the system is under high write load (e.g., if I am emulating a large disk and I run mkfs over the disk) the scsi_target will crash in a seemingly nondeterministic way. (Note, I am using a modified version of scsi_target, but I haven't changed its core behavior.) Sometimes an ATIO is received with its flags field set to zero (causing work_atio() to abort) -- this would seem to indicate that the ATIO were sent before the data were received from the initiator, but this seems impossible based on my cursory look over the Qlogic driver code. Other times I start getting a series of error messages on the console [it may be the "no ATIO2s for lun 0 from initiator 3" message, but I'm not certain -- I can reproduce the exact message if it will help this debugging] but I don't think it's run out of ATIOs, because I increased the number to 1024 and there were only about ten outstanding requests at the time. My best guesses are that because of the high write load some kernel structures are getting overwritten, or linked lists of free/allocated ATIOs in the kernel are being improperly managed, but the answer is so far eluding me. A colleague suggested it may be mishandling of tagged requests by the XPT code, but that seems unlikely. Does anyone have any idea what might be causing this, or what a solution might be (other than rate-limiting the initiator)?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?Pine.LNX.4.53L-ECE.CMU.EDU.0312271201350.21079>