From owner-freebsd-cluster@FreeBSD.ORG Fri Dec 16 15:03:14 2011 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1BF91106566C for ; Fri, 16 Dec 2011 15:03:14 +0000 (UTC) (envelope-from filip@valder.cz) Received: from smtp.vsb.cz (smtp.vsb.cz [IPv6:2001:718:1001:149::155]) by mx1.freebsd.org (Postfix) with ESMTP id 394948FC16 for ; Fri, 16 Dec 2011 15:03:12 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.vsb.cz (Postfix) with ESMTP id 881B31C1F22 for ; Fri, 16 Dec 2011 16:03:11 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at smtp.vsb.cz Received: from smtp.vsb.cz ([127.0.0.1]) by localhost (smtp-ha1.vsb.cz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id QNHEFsHBX5-s for ; Fri, 16 Dec 2011 16:03:08 +0100 (CET) Received: from [IPv6:2001:718:1001:158:224:1dff:fe85:f221] (unknown [IPv6:2001:718:1001:158:224:1dff:fe85:f221]) by smtp.vsb.cz (Postfix) with ESMTPSA id E04421C1F12 for ; Fri, 16 Dec 2011 16:03:08 +0100 (CET) Message-ID: <4EEB5DAC.9050704@valder.cz> Date: Fri, 16 Dec 2011 16:03:08 +0100 From: Filip Valder User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111003 Thunderbird/7.0.1 MIME-Version: 1.0 To: freebsd-cluster@freebsd.org Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms010305050207020608090500" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: HA storage, diskless booting, clustering X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 15:03:14 -0000 Toto je elektronicky podepsaná zpráva ve formátu MIME. --------------ms010305050207020608090500 Content-Type: multipart/mixed; boundary="------------060206000908040806050505" This is a multi-part message in MIME format. --------------060206000908040806050505 Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: quoted-printable Hi everyone! It seems that I brought diskless booting back to life in FreeBSD 7.2 and now I'm about to try it in 8.2 together with HAST. It seems that nothing relevant has changed in diskless since 7.2. The official docs also seem outdated, so it's a good playground for my experiments and I'm very happy about my quick and dirty hacks. Or clean...? Who knows? :-) First of all prepare your HAST environment with this guide: http://www.freebsd.org/doc/handbook/disks-hast.html - SSD disks and gigE are fast enough, so don't be afraid of it! Just use attached ifstated.conf instead of the devd.conf in order to avoid split-brain situations. - Configure all that HAST, pfsync, CARP and other stuff to get this "Gemini" system working. When your HA storage is alive, go ahead and: - prepare your PXE environment (incl. DHCP) and tftpboot, like: /hast/pxe/tftpboot/ /hast/pxe/tftpboot/boot /hast/pxe/tftpboot/boot/defaults =2E.. /hast/pxe/tftpboot/boot/firmware =2E.. /hast/pxe/tftpboot/boot/kernel =2E.. /hast/pxe/tftpboot/boot/modules =2E.. /hast/pxe/tftpboot/boot/zfs =2E.. /hast/pxe/tftpboot/boot/loader.conf /hast/pxe/tftpboot/boot/mbr /hast/pxe/tftpboot/boot/boot0sio /hast/pxe/tftpboot/boot/boot2 /hast/pxe/tftpboot/boot/boot0 /hast/pxe/tftpboot/boot/boot1 /hast/pxe/tftpboot/boot/cdboot /hast/pxe/tftpboot/boot/loader.4th /hast/pxe/tftpboot/boot/loader.help /hast/pxe/tftpboot/boot/loader.rc /hast/pxe/tftpboot/boot/support.4th /hast/pxe/tftpboot/boot/gptboot /hast/pxe/tftpboot/boot/boot /hast/pxe/tftpboot/boot/pmbr /hast/pxe/tftpboot/boot/screen.4th /hast/pxe/tftpboot/boot/frames.4th /hast/pxe/tftpboot/boot/device.hints /hast/pxe/tftpboot/boot/loader /hast/pxe/tftpboot/boot/beastie.4th /hast/pxe/tftpboot/boot/pxeboot /hast/pxe/tftpboot/pxeboot -> /hast/pxe/tftpboot/boot/pxeboot - sysinstall your diskless machine into /hast/pxe/diskless-box, and separate /, /usr and /var according to the fstab (below) - configure NFS in order to export that HAST wonder - pay attention that CARP and UDP are not friends. You have to configure rpcbind, nfsd a mountd to listen on the CARP interface instead of a wildcard address. - hack some files in the /hast/pxe/diskless-box/root/etc: Add to rc.conf: background_fsck=3D"NO" dumpdev=3D"NO" netfs_types=3D"nfs4:NFS4 smbfs:SMB portalfs:PORTAL nwfs:NWFS" Comment out theese lines in rc: dlv=3D`/sbin/sysctl -n vfs.nfs.diskless_valid 2> /dev/null` if [ ${dlv:=3D0} -ne 0 -o -f /etc/diskless ]; then sh /etc/rc.initdiskless fi Comment out rc.d/mountcritremote from this line (incl.) until the end of file: =2E /etc/rc.subr fstab: my-hast:/hast/pxe/diskless-box/root / nfs -L,noauto,rw = 0 0 my-hast:/hast/pxe/diskless-box/usr /usr nfs -L,rw 0 = 0 my-hast:/hast/pxe/diskless-box/var /var nfs -L,rw 0 = 0 When it all seems ready, launch your diskless box and see what happens! :-) TODOs: I will try it with 8.2, will introduce a template system, so that diskless-boxes will have / and /usr in common, with separated /etc/rc.conf and /var. If it all works you will get some low- or mid-end clustering solution. Hint: export PATH_FSTAB via /etc/rc could help with individual fstabs according to hostnames e.g. ******** NEWS: I've tried it already. It works great! ******** Some detailed docs should come from me sooner or later. This is just a nightly draft... Cheers! Filip --=20 filip@valder.cz | +420 608 123 966 | http://www.valder.cz/ --------------060206000908040806050505 Content-Type: text/plain; name="ifstated.conf" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="ifstated.conf" # This is the only difference between the primary and secondary # HAST node. Set it accordingly. init-state primary #init-state secondary # Leave it or substitute "carp0" for the name of your CARP interface # if it varies. carp_up =3D "carp0.link.up" carp_down =3D "!carp0.link.up" # The "net" address is an address which can be used to determine # whether we have connectivity. Make sure the host is always up, or # test multiple ip's, 'or'-ing the tests. net =3D '( "ping -q -c 1 -W 1000 10.11.12.13 > /dev/null" every 10)' state primary { init { # Leave it or substitute "carp0" for the name of # your CARP interface if it varies. run "ifconfig carp0 advskew 10" # Where your carp-hast-switch resides... run "/root/bin/carp-hast-switch master" } if $carp_down || ! $net set-state secondary } state secondary { init { # Leave it or substitute "carp0" for the name of # your CARP interface if it varies. run "ifconfig carp0 advskew 100" # Where your carp-hast-switch resides... run "/root/bin/carp-hast-switch slave" } if $carp_up && $net set-state primary } --------------060206000908040806050505-- --------------ms010305050207020608090500-- From owner-freebsd-cluster@FreeBSD.ORG Fri Dec 16 15:05:04 2011 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 79389106575A for ; Fri, 16 Dec 2011 15:05:04 +0000 (UTC) (envelope-from filip@valder.cz) Received: from smtp.vsb.cz (smtp.vsb.cz [IPv6:2001:718:1001:149::155]) by mx1.freebsd.org (Postfix) with ESMTP id 2178C8FC15 for ; Fri, 16 Dec 2011 15:05:01 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.vsb.cz (Postfix) with ESMTP id 58BDE1C1F53 for ; Fri, 16 Dec 2011 16:05:00 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at smtp.vsb.cz Received: from smtp.vsb.cz ([127.0.0.1]) by localhost (smtp-ha1.vsb.cz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3UoggU79hjdu for ; Fri, 16 Dec 2011 16:04:57 +0100 (CET) Received: from [IPv6:2001:718:1001:158:224:1dff:fe85:f221] (unknown [IPv6:2001:718:1001:158:224:1dff:fe85:f221]) by smtp.vsb.cz (Postfix) with ESMTPSA id C8FC31C1F12 for ; Fri, 16 Dec 2011 16:04:57 +0100 (CET) Message-ID: <4EEB5E19.5020105@valder.cz> Date: Fri, 16 Dec 2011 16:04:57 +0100 From: Filip Valder User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111003 Thunderbird/7.0.1 MIME-Version: 1.0 To: freebsd-cluster@freebsd.org Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms040201080505040002080807" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: HA storage, diskless booting, clustering X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 15:05:04 -0000 Toto je elektronicky podepsaná zpráva ve formátu MIME. --------------ms040201080505040002080807 Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: quoted-printable Hi everyone! It seems that I brought diskless booting back to life in FreeBSD 7.2 and now I'm about to try it in 8.2 together with HAST. It seems that nothing relevant has changed in diskless since 7.2. The official docs also seem outdated, so it's a good playground for my experiments and I'm very happy about my quick and dirty hacks. Or clean...? Who knows? :-) First of all prepare your HAST environment with this guide: http://www.freebsd.org/doc/handbook/disks-hast.html - SSD disks and gigE are fast enough, so don't be afraid of it! Just use attached ifstated.conf instead of the devd.conf in order to avoid split-brain situations. - Configure all that HAST, pfsync, CARP and other stuff to get this "Gemini" system working. When your HA storage is alive, go ahead and: - prepare your PXE environment (incl. DHCP) and tftpboot, like: /hast/pxe/tftpboot/ /hast/pxe/tftpboot/boot /hast/pxe/tftpboot/boot/defaults =2E.. /hast/pxe/tftpboot/boot/firmware =2E.. /hast/pxe/tftpboot/boot/kernel =2E.. /hast/pxe/tftpboot/boot/modules =2E.. /hast/pxe/tftpboot/boot/zfs =2E.. /hast/pxe/tftpboot/boot/loader.conf /hast/pxe/tftpboot/boot/mbr /hast/pxe/tftpboot/boot/boot0sio /hast/pxe/tftpboot/boot/boot2 /hast/pxe/tftpboot/boot/boot0 /hast/pxe/tftpboot/boot/boot1 /hast/pxe/tftpboot/boot/cdboot /hast/pxe/tftpboot/boot/loader.4th /hast/pxe/tftpboot/boot/loader.help /hast/pxe/tftpboot/boot/loader.rc /hast/pxe/tftpboot/boot/support.4th /hast/pxe/tftpboot/boot/gptboot /hast/pxe/tftpboot/boot/boot /hast/pxe/tftpboot/boot/pmbr /hast/pxe/tftpboot/boot/screen.4th /hast/pxe/tftpboot/boot/frames.4th /hast/pxe/tftpboot/boot/device.hints /hast/pxe/tftpboot/boot/loader /hast/pxe/tftpboot/boot/beastie.4th /hast/pxe/tftpboot/boot/pxeboot /hast/pxe/tftpboot/pxeboot -> /hast/pxe/tftpboot/boot/pxeboot - sysinstall your diskless machine into /hast/pxe/diskless-box, and separate /, /usr and /var according to the fstab (below) - configure NFS in order to export that HAST wonder - pay attention that CARP and UDP are not friends. You have to configure rpcbind, nfsd a mountd to listen on the CARP interface instead of a wildcard address. - hack some files in the /hast/pxe/diskless-box/root/etc: Add to rc.conf: background_fsck=3D"NO" dumpdev=3D"NO" netfs_types=3D"nfs4:NFS4 smbfs:SMB portalfs:PORTAL nwfs:NWFS" Comment out theese lines in rc: dlv=3D`/sbin/sysctl -n vfs.nfs.diskless_valid 2> /dev/null` if [ ${dlv:=3D0} -ne 0 -o -f /etc/diskless ]; then sh /etc/rc.initdiskless fi Comment out rc.d/mountcritremote from this line (incl.) until the end of file: =2E /etc/rc.subr fstab: my-hast:/hast/pxe/diskless-box/root / nfs -L,noauto,rw = 0 0 my-hast:/hast/pxe/diskless-box/usr /usr nfs -L,rw 0 = 0 my-hast:/hast/pxe/diskless-box/var /var nfs -L,rw 0 = 0 When it all seems ready, launch your diskless box and see what happens! :-) TODOs: I will try it with 8.2, will introduce a template system, so that diskless-boxes will have / and /usr in common, with separated /etc/rc.conf and /var. If it all works you will get some low- or mid-end clustering solution. Hint: export PATH_FSTAB via /etc/rc could help with individual fstabs according to hostnames e.g. ******** NEWS: I've tried it already. It works great! ******** Some detailed docs should come from me sooner or later. This is just a nightly draft... Cheers! Filip --=20 filip@valder.cz | +420 608 123 966 | http://www.valder.cz/ --------------ms040201080505040002080807-- From owner-freebsd-cluster@FreeBSD.ORG Fri Dec 16 15:07:24 2011 Return-Path: Delivered-To: freebsd-cluster@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2356A106566C for ; Fri, 16 Dec 2011 15:07:24 +0000 (UTC) (envelope-from filip@valder.cz) Received: from smtp.vsb.cz (smtp.vsb.cz [IPv6:2001:718:1001:149::155]) by mx1.freebsd.org (Postfix) with ESMTP id B68C98FC0C for ; Fri, 16 Dec 2011 15:07:23 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.vsb.cz (Postfix) with ESMTP id 00D151C1E8B for ; Fri, 16 Dec 2011 16:07:23 +0100 (CET) X-Virus-Scanned: Debian amavisd-new at smtp.vsb.cz Received: from smtp.vsb.cz ([127.0.0.1]) by localhost (smtp-ha1.vsb.cz [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id JnMlaundZVqF for ; Fri, 16 Dec 2011 16:07:13 +0100 (CET) Received: from [IPv6:2001:718:1001:158:224:1dff:fe85:f221] (unknown [IPv6:2001:718:1001:158:224:1dff:fe85:f221]) by smtp.vsb.cz (Postfix) with ESMTPSA id 1DE3C1C1F73 for ; Fri, 16 Dec 2011 16:07:13 +0100 (CET) Message-ID: <4EEB5EA0.1070104@valder.cz> Date: Fri, 16 Dec 2011 16:07:12 +0100 From: Filip Valder User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:7.0.1) Gecko/20111003 Thunderbird/7.0.1 MIME-Version: 1.0 To: freebsd-cluster@freebsd.org Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha1; boundary="------------ms040201090106090703090908" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: HA storage, diskless booting, clustering - ifstated.conf X-BeenThere: freebsd-cluster@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Clustering FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Dec 2011 15:07:24 -0000 Toto je elektronicky podepsaná zpráva ve formátu MIME. --------------ms040201090106090703090908 Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: quoted-printable # This is the only difference between the primary and secondary # HAST node. Set it accordingly. init-state primary #init-state secondary # Leave it or substitute "carp0" for the name of your CARP interface # if it varies. carp_up =3D "carp0.link.up" carp_down =3D "!carp0.link.up" # The "net" address is an address which can be used to determine # whether we have connectivity. Make sure the host is always up, or # test multiple ip's, 'or'-ing the tests. net =3D '( "ping -q -c 1 -W 1000 10.11.12.13 > /dev/null" every 10)' state primary { init { # Leave it or substitute "carp0" for the name of # your CARP interface if it varies. run "ifconfig carp0 advskew 10" # Where your carp-hast-switch resides... run "/root/bin/carp-hast-switch master" } if $carp_down || ! $net set-state secondary } state secondary { init { # Leave it or substitute "carp0" for the name of # your CARP interface if it varies. run "ifconfig carp0 advskew 100" # Where your carp-hast-switch resides... run "/root/bin/carp-hast-switch slave" } if $carp_up && $net set-state primary } --------------ms040201090106090703090908--