Date: Tue, 17 May 2016 11:56:28 +0200 From: Jan Bramkamp <crest@rlwinm.de> To: freebsd-fs@freebsd.org Subject: Re: Best practice for high availability ZFS pool Message-ID: <84e3b485-d8bd-0f2f-47a4-85a64678d286@rlwinm.de> In-Reply-To: <5E69742D-D2E0-437F-B4A9-A71508C370F9@FreeBSD.org> References: <5E69742D-D2E0-437F-B4A9-A71508C370F9@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 16/05/16 12:08, Palle Girgensohn wrote: > Hi, > > We need to set up a ZFS pool with redundance. The main goal is high availability - uptime. > > I can see a few of paths to follow. > > 1. HAST + ZFS > > 2. Some sort of shared storage, two machines sharing a JBOD box. If you're willing to put your disks into JBODs you can use JBODs with two upstream ports per SAS expander and hook up one port to each head node. Now you can access the all disks on both head nodes. The next step you require is reliable master election. Two nodes alone can't form the required consensus. In theory you could use SCSI persistent reservations, but afaik FreeBSD lacks the tooling unless you want send raw SCSI commands through camcontrol. The easier solution is to run some master election using one or three additional nodes for a total of three or five. Both consul and etcd are available as ports and are designed for reliable master election without special hardware. If you go done this way you still need some kind of fencing (maybe via IPMI or PDUs). Now the JBOD is your SPoF so get yourself at least two or better three JBODs. For optimal performance and reliability use three JBODs with 3-way mirrors spread over all JOBDs. In this setup no hardware protects your disks from the hot standby. If it falls out of sync you have to keep it from writing to the shared directed attached storage. One way to achieve this would be to load the SAS HBA kernel module only after the role (primary, backup) has been elected and disable the HBA option ROM in the UEFI/BIOS. I tried this once out of curiosity and it performed well, but good look finding any support for such a setup. The same kind of setup should be possible with iSCSI instead of SAS disks connected to dual ported expanders, but I can't say anything about the performance you can expect from the FreeBSD iSCSI target and initiator. At least it would simply fencing a lot because the fencing could be moved from the SCSI initiators into the SCSI targets. Jan Bramkamp
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?84e3b485-d8bd-0f2f-47a4-85a64678d286>