From owner-freebsd-questions@freebsd.org Sat Sep 2 16:11:15 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4F62BE1A461 for ; Sat, 2 Sep 2017 16:11:15 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from sola.nimnet.asn.au (paqi.nimnet.asn.au [115.70.110.159]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C1B2267734 for ; Sat, 2 Sep 2017 16:11:10 +0000 (UTC) (envelope-from smithi@nimnet.asn.au) Received: from localhost (localhost [127.0.0.1]) by sola.nimnet.asn.au (8.14.2/8.14.2) with ESMTP id v82GApvT030480; Sun, 3 Sep 2017 02:10:54 +1000 (EST) (envelope-from smithi@nimnet.asn.au) Date: Sun, 3 Sep 2017 02:10:51 +1000 (EST) From: Ian Smith To: Frank Leonhardt cc: hps@freebsd.org, freebsd-questions@freebsd.org Subject: Re: Fun with HAST and inter-host connections In-Reply-To: Message-ID: <20170903015737.H23641@sola.nimnet.asn.au> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Sep 2017 16:11:15 -0000 In freebsd-questions Digest, Vol 691, Issue 4, Message: 10 On Wed, 30 Aug 2017 22:40:03 +0100 Frank Leonhardt wrote: > Please note - I'm pushing what can be done with commodity hardware for > amusement here, and trying to do interesting things with FreeBSD. I *do* > want to do it this way. > > Okay, so the game is this: I've got some identical hosts with disks and > I've been working on ways of clustering them for a long time. It's what > I do for fun, right? > > I'm currently playing with failover storage. Not necessarily with HAST; > iSCSI is fun and I'm messing with geom stuff in general. But let's stick > with HAST as it illustrates the dilemma. > > When I started this game, 1Gb Ethernet was blistering. Now it's not so > hot. How do I "network" the hosts with as much throughput as possible > (with IP sockets, preferably)? Options: > > 1) 10Gb Ethernet is expensive. 10Gb switches even more so, but I can do > point-to-point. > > 2) LAG is more about failover than speed. And anyway, you end up needing > a lot of Ethernet ports on each host and it soon gets crazy. > > 3) Fibrechannel - if I bought a few old (cheap) fiberchannel cards, I'm > not sure how I could use them point-to-point. Is this possible and has > anyone done it? > > 4) USB 3.1. 10Gb. PCIe cards cost about $30. Now this sounds fun. IP > over USB anyone? > > Now please don't advise me to get a pair of fabric switches and do the > job properly. That's not what this is about. I want to see if it's > possible to make a fast(er) storage solution using cheap components. A > sort of Redundant Array of Inexpensive NAS. > > Thoughts anyone? In particular, is the USB 3.1 idea crazy? And is anyone > else crazy enough to be trying the same thing? > > Thanks, Frank. Been following this thread with idle curiousity through all the options. It's all out of my experience, but if anyone knows what's what with USB 3.1 and whether crazy or not, it'd be Hans Petter (cc'd / dobbed in :) cheers, Ian