From owner-freebsd-fs@FreeBSD.ORG Thu Aug 2 15:45:26 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6102D16A468 for ; Thu, 2 Aug 2007 15:45:26 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from ns.trinitel.com (186.161.36.72.static.reverse.ltdomains.com [72.36.161.186]) by mx1.freebsd.org (Postfix) with ESMTP id 39A7F13C47E for ; Thu, 2 Aug 2007 15:45:26 +0000 (UTC) (envelope-from anderson@freebsd.org) Received: from proton.local (209-163-168-124.static.twtelecom.net [209.163.168.124]) (authenticated bits=0) by ns.trinitel.com (8.14.1/8.14.1) with ESMTP id l72FjIXF080394; Thu, 2 Aug 2007 10:45:23 -0500 (CDT) (envelope-from anderson@freebsd.org) Message-ID: <46B1FC0E.7040702@freebsd.org> Date: Thu, 02 Aug 2007 10:45:18 -0500 From: Eric Anderson User-Agent: Thunderbird 2.0.0.6 (Macintosh/20070728) MIME-Version: 1.0 To: Patrick Tracanelli References: <46B0F505.8090102@freebsdbrasil.com.br> <46B10798.5050504@freebsdbrasil.com.br> <200708011536.37926.matt@ixsystems.com> <46B12D0C.20808@freebsd.org> <46B1D167.4030206@freebsdbrasil.com.br> <46B1E234.7010005@freebsd.org> <46B1ECDF.10407@freebsdbrasil.com.br> In-Reply-To: <46B1ECDF.10407@freebsdbrasil.com.br> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-1.5 required=5.0 tests=AWL,BAYES_00 autolearn=ham version=3.1.8 X-Spam-Checker-Version: SpamAssassin 3.1.8 (2007-02-13) on ns.trinitel.com Cc: freebsd-fs@freebsd.org Subject: Re: Xsan (Apple) on FreeBSD X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Aug 2007 15:45:26 -0000 Patrick Tracanelli wrote: > Eric Anderson wrote: >> Patrick Tracanelli wrote: >>> Eric Anderson wrote: >>>> Matt Olander wrote: >>>>> On Wednesday 01 August 2007 3:22 pm, Patrick Tracanelli wrote: >>>>>> Hello Jeff, >>>>>> >>>>>> Jeff Mohler wrote: >>>>>>> Im yet to hear of a large Xsan install that stayed Xsan once it >>>>>>> grew. >>>>>>> >>>>>>> Most, if not all, have gone to netapp or umm..Isilon (spelling) >>>>>>> that ive >>>>>>> been close to. Latest large dump of Xsan that I know of was >>>>>>> Current TV >>>>>>> in San Francisco, for Isilon. >>>>>> Hmm, good to know. I have tested XServe RAID only, which has shown >>>>>> to be >>>>>> a good solution as storage system for the usage profile I need, but >>>>>> Xsan, never saw it working. Believed it to be the usual path to >>>>>> follow, >>>>>> but have hear of people running Stornext instead of xsan. >>>>>> >>>>>> So, now my question goes on a different path. Will Isilon OneFS run >>>>>> FreeBSD? I head from people at Zoic Studios it is based on >>>>>> FreeBSD, but >>>>>> I am not sure how true this information is. Anyway, "based on" wont >>>>>> always mean fully compatible. >>>>>> >>>>>> Are you aware of OneFS running on FreeBSD? >>>>> >>>>> It is indeed true. OneFS is FreeBSD. The Isilon product is really >>>>> neat and you can buy it modularly, starting with just one unit. >>>> >>>> To be clear, OneFS isn't a solution to add to FreeBSD - it is an >>>> all-in-one solution, that happens to be built with FreeBSD (or parts >>>> of it anyway). It's like a NetApp. You don't run NetApp on >>>> FreeBSD, you use whatever clients you want, and they connect *to* >>>> Isilon or NetApp. As far as I understand, they are all just >>>> NFS/CIFS/iSCSI servers/targets. There's really no solution for >>>> sharing a SAN block device safely using FreeBSD (using the same >>>> blocks with the same fs). That would require a clustered filesystem, >>>> and there is no such beast for FreeBSD at this point. Simultaneous >>>> read-write activity from different hosts to the same file system >>>> leads to Bad Stuff. >>> >>> Hello Eric, >>> >>> Thank you for clearing up some things. I believed OneFS to be an >>> extension the the file system which would allow shared access. So, >>> OneFS seem to be exclusively used by the appliance itself, as you >>> mention. >>> >>> FreeBSD unfortunately don't have a shared file system (it would be a >>> solution to this matter, combining a shared FS with ggate and >>> gmirror, or using a central gvirstor/zfs storage solution, 100% >>> FreeBSD-only). It really is a missing piece of feature which would >>> boost usage/combination of many other ones. >> >> >> Agreed - I've been beating that drum for some time. It's a *lot* of >> work, and not enough developers/money to do it right now. > > I can imagine how much work would be needed for a whole new FS. Just > curious, no chance for a geom module to do the trick? Or "tricky" would > also be writing a geom to do this? =) Well, GEOM is block level, so that doesn't really fix the file system locking issues. However, maybe it's possible to abuse geom_journal and make it a distributed journal to achieve it. It's still really hard. :) >> >> >>> I dont know about iSCSI support on FreeBSD. A quick research on the >>> archieves seem to show that there is no iSCSI support at this time. So >> >> There is iSCSI support, and -CURRENT recently got an iSCSI initiator >> in the base system. The iSCSI target is in the ports collection. >> This doesn't fix the issue, as it's still a block device transport. > > Well, so, on 7.0 I can have a FreeBSD as target sharing some data and > two 7.0 boxes as initiator acessing the same data? If so, that would be > an approach to test. Yes, you can share the blocks, but that doesn't solve the file system clustering issues. >>> NFS/CIFS and something like that would the option? In this case, a >>> FreeBSD solution seem a lot more flexible than a storage appliance. >>> In fact I run NFS today, but performance is becoming a problem as the >>> usage increases. I have never used CIFS on Unix-to-Unix enviroment, >>> and I dont believe it to be better than NFS anyway. But maybe I >>> should give it a try. Is there any other CIFS implementation other >>> than Samba? Samba just happen to have so many features Ill never need >>> in this enviroment. Is there any chance it will perform better than NFS? >> >> NFS will beat CIFS in performance almost always. NFS is a commonly >> used protocol for shared file access, and should perform fairly well. >> If you are hitting NFS performance issues, you might want to dig there >> first, since there are things that you can do to improve your >> performance, depending on your usage. It may in fact be that NFS >> itself is not the bottleneck for you. > > Right. The situation I have is among many 5.5-STABLE systems. Sometimes, > under high load, the client systems get high load averages, but userland > apps are sort of sitting idle. I can see however, that "system" starts > using ~ 80% of CPU (from top). Clients are quad-processed servers. > > On server, gstat shows me that the disk is on 100% usage, but not 100% > throughput (since they are many, but small disk operations). nfsstat > also shows we are on limit of what I could find to be the limit via dd > parallely and massively writing and read tests. And thus, nfs server > "stops responding" for a while, and later becomes responsive again. > > I didnt find out something I could tune up to make it better. Well, it sounds like you are IO bound on your server. If the system is waiting on disks to get you data, then you probably need to adjust your storage to allow more IO. This usually means more disks, or a different RAID set up, more cache, etc. You might also bump up the number of nfsd processes on your server (try doubling it first). > Side question: is 5.5-STABLE to 6.2-STABLE NFS code too different? Would > the behavior on 6.2 be different? There are many file system tweaks, and few NFS tweaks. I don't think any of them will give you massive performance increase. I think you are throttling on disk IO. Eric