Date: Mon, 06 Feb 2012 19:05:37 +0100 From: Peter Maloney <peter.maloney@brockmann-consult.de> To: freebsd-fs@freebsd.org Subject: Re: HPC and zfs. Message-ID: <4F301671.3060507@brockmann-consult.de> In-Reply-To: <4F2FF72B.6000509@pean.org> References: <4F2FF72B.6000509@pean.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Am 06.02.2012 16:52, schrieb Peter Ankerstål: > Hi, > > I want to investigate if it is possible to create your own usable HPC > storage using zfs and some > network filesystem like nfs. > > Just a thought experiment.. > A machine with 2 6 core XEON, 3.46Ghz 12MB and 192GB of ram (or more) > I addition the machine will use 3-6 SSD drives for ZIL FYI: With ESXi's NFS client (**the most evil of anything doing synchronous writes), striping my ZIL made no difference. Either way, it used 100% load on the SSDs and went the same pathetic speed. (gstripe worked to some extent) Using "-o sync" in the Linux NFS client does not use the ZIL at max load*. If you can suggest another way to test (without ESXi), I will try it on 3 SSDs (4 if the one that I RMAed comes back in time) and let you know the results. * With a pool that was created in 8.2-STABLE and upgraded to v28, it did use max load. Now it doesn't. But is recreating that situation the right way to test? It is not what you will be using. ** For more info about ESXi's terrible NFS performance, see this graph http://doub.home.xs4all.nl/bench/sync.png and find the mail I got it from in this mailing list fromjoh.hendriks@gmail.com with the title "Re: ZFS sync / ZIL clarification". > and 3-6 SSD deives for cache. > Preferrably in mirror where applicable. > > Connected to this machine we will have about 410 3TB drives to give > approx > 1PB of usable storage in a 8+2 raidz configuration. > > Connected to this will be a ~800 nodes big HPC cluster that will > access the storage in parallell > is this even possible or do we need to distribute the meta data load > over many servers? If that is the case, > does it exist any software for FreeBSD that could accomplish this > distribution (pNFS dosent seem to be > anywhere close to usable in FreeBSD) or do I need to call NetApp or > Panasas right away? It would be > really nice if I could build my own storage solution. > > Other possible solutions to this problem is extremley welcome. > > Best Regards > Peter Ankerstål > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4F301671.3060507>