From owner-freebsd-hackers@FreeBSD.ORG Thu Dec 8 04:45:34 2005 Return-Path: X-Original-To: hackers@freebsd.org Delivered-To: freebsd-hackers@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D7C2216A41F for ; Thu, 8 Dec 2005 04:45:34 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from mh2.centtech.com (moat3.centtech.com [207.200.51.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6BF4143D6D for ; Thu, 8 Dec 2005 04:45:34 +0000 (GMT) (envelope-from anderson@centtech.com) Received: from [192.168.42.22] (andersonbox2.centtech.com [192.168.42.22]) by mh2.centtech.com (8.13.1/8.13.1) with ESMTP id jB84jXe5007221; Wed, 7 Dec 2005 22:45:33 -0600 (CST) (envelope-from anderson@centtech.com) Message-ID: <4397BA4F.8060708@centtech.com> Date: Wed, 07 Dec 2005 22:45:03 -0600 From: Eric Anderson User-Agent: Mozilla Thunderbird 1.0.7 (X11/20051204) X-Accept-Language: en-us, en MIME-Version: 1.0 To: Scott Long References: <4395BF04.50101@centtech.com> <43960F55.3010508@root.org> <43975926.1010302@centtech.com> <43975F5F.5080901@samsco.org> <439782AA.6000408@root.org> <4397B731.6010308@centtech.com> <4397B82C.5020004@samsco.org> In-Reply-To: <4397B82C.5020004@samsco.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.82/1205/Wed Dec 7 08:00:48 2005 on mh2.centtech.com X-Virus-Status: Clean Cc: hackers@freebsd.org, Nate Lawson Subject: Re: scsi-target and the buffer cache X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Dec 2005 04:45:35 -0000 Scott Long wrote: > Eric Anderson wrote: > >> Nate Lawson wrote: >> >>> Scott Long wrote: >>> >>>> Eric Anderson wrote: >>>> >>>>> Nate Lawson wrote: >>>>> >>>>>> Eric Anderson wrote: >>>>>> >>>>>>> I'm curious about whether a target mode device would use the >>>>>>> buffer cache or not. Here's a scenario: >>>>>>> >>>>>>> Host A: has fibre channel host adapter, in target mode, large >>>>>>> memory pool, and another fiber channel host adapter connecting >>>>>>> to fibre channel block device. >>>>>>> Host B: Fibre channel host adapter, connecting to Host A. >>>>>>> 'sees' the target mode block device created by Host A. >>>>>>> >>>>>>> Will Host A use the buffer cache to cache blocks between the >>>>>>> real block device, and the shared target mode device? >>>>>>> What about if Host A put a filesystem on the block device, >>>>>>> created a single file the size of the filesystem, and shared >>>>>>> that filesystem via a target mode device to Host B? >>>>>>> What I'm wanting is a box (FreeBSD?) that can be placed between >>>>>>> a fibre channel block device (like a RAID array), and a fibre >>>>>>> channel host using that block device, and act as a block cache >>>>>>> for that device, using the FreeBSD's memory. If it had a >>>>>>> significant amount of memory, this could be very useful. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> If you use the example scsi_target usermode >>>>>> (usr/share/examples/scsi_target), then the buffer cache will be >>>>>> used since its reads/writes are from usermode like normal. If >>>>>> you don't want that behavior, you can set O_DIRECT in the open() >>>>>> call of the backing store file. >>>>>> >>>>>> If you chose to modify the kernel side, you'd have to make sure >>>>>> your accesses were through the VOP layer and then it would be >>>>>> cached. >>>>>> >>>>>> You should check to be sure the target mode performance meets >>>>>> your expectations also. >>>>>> >>>>> >>>>> I guess I would be using the user mode tool, unless there's >>>>> another way? Your comment on performance also makes me a little >>>>> worried about that now - do you think I would see a large >>>>> performance hit? >>>>> Thanks! >>>>> Eric >>>>> >>>>> >>>> >>>> The way the target mode stack works in FreeBSD is that the kernel >>>> provides some of the basic services, but the actual target emulator >>>> is meant to live in userland. The userland program responds to >>>> events from the kernel via the select interface. This generally >>>> works pretty well. However, it does mean that control has to >>>> cross the kernel-userland boundary at least once for every event. >>>> >>>> What I'd suggest doing is prototyping your target emulator in userland >>>> and evaluating the performance there, and then moving it to the kernel >>>> if you _really_ need more performance. >>> >>> >>> >>> >>> Agree 100%. While having it in usermode means there are boundary >>> crossings that increase per-transaction latency, the actual bulk >>> data transfer is via zero-copy IO and you should be able to exceed >>> the data transfer rates of several 10K RPM drives on decent hardware. >>> >> >> >> Ok, great.. Now, will scsi_target work ok with raw devices, or only >> files? (although I'm not sure theres all that much difference really). >> >> Thanks!! >> Eric >> >> > > You can write your userland code to use whatever files or devices you > want. Are you talking about the scs_target.c code in > /usr/share/examples? That's just a skeletal example that you can use > as a starting point for your own work. Alright.. I was indeed talking about the example code, but I suppose it wouldn't be too hard to make it work with raw devices. Thanks for the help! Eric -- ------------------------------------------------------------------------ Eric Anderson Sr. Systems Administrator Centaur Technology Anything that works is better than anything that doesn't. ------------------------------------------------------------------------