Date: Wed, 3 Mar 2004 04:30:26 -0800 (PST) From: Jean-Francois Dockes <jean-francois.dockes@wanadoo.fr> To: freebsd-doc@FreeBSD.org Subject: Re: docs/62834: Diskless operation handbook section update Message-ID: <200403031230.i23CUQAl038646@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR docs/62834; it has been noted by GNATS. From: Jean-Francois Dockes <jean-francois.dockes@wanadoo.fr> To: freebsd-gnats-submit@FreeBSD.org Cc: Subject: Re: docs/62834: Diskless operation handbook section update Date: Wed, 3 Mar 2004 13:28:57 +0100 I have integrated initial remarks to the handbook advanced networking diskless section update. The revised text can be found at: http://perso.wanadoo.fr/dockes/diskless/network-diskless.html The unified diffs follow, or can be found at: http://perso.wanadoo.fr/dockes/diskless/udiffs.txt I really believe that the new version is better than the previous one (which I wrote). It corrects some errors and contains updated material for FreeBSD 5.x I would appreciate it if someone could find the time to review and possibly commit the new text. Regards, J.F. Dockes --- chapter.sgml.old Wed Mar 3 12:48:33 2004 +++ chapter.sgml Wed Mar 3 12:58:57 2004 @@ -2402,10 +2402,10 @@ <indexterm><primary>diskless operation</primary></indexterm> <para>A FreeBSD machine can boot over the network and operate without a - local disk, using filesystems mounted from an NFS server. No system - modification is necessary, beyond standard configuration files. - Such a system is easy to set up because all the necessary elements - are readily available:</para> + local disk, using filesystems mounted from an NFS server. No + system modification is necessary, beyond standard configuration + files. Such a system is relatively easy to set up because all + the necessary elements are readily available:</para> <itemizedlist> <listitem> <para>There are at least two possible methods to load the kernel over @@ -2451,10 +2451,10 @@ <para>There are many ways to set up diskless workstations. Many elements are involved, and most can be customized to suit local - taste. The following will describe the setup of a complete system, - emphasizing simplicity and compatibility with the - standard FreeBSD startup scripts. The system described has the - following characteristics:</para> + taste. The following will describe variations on the setup of a + complete system, emphasizing simplicity and compatibility with + the standard FreeBSD startup scripts. The system described has + the following characteristics:</para> <itemizedlist> <listitem> @@ -2466,12 +2466,14 @@ configuration files overridden by ones specific to diskless operation or, possibly, to the workstation they belong to.</para> <para>The parts of the <filename>root</filename> which have to be - writable are overlaid with &man.mfs.8; filesystems. Any changes - will be lost when the system reboots.</para> + writable are overlaid with &man.mfs.8; (FreeBSD 4.X) or + &man.md.4; (FreeBSD 5.X) filesystems. Any changes will be lost + when the system reboots.</para> </listitem> <listitem> - <para>The kernel is loaded by <application>etherboot - </application>, using DHCP (or BOOTP) and TFTP.</para> + <para>The kernel is transferred and loaded either with + <application>etherboot</application>, or <acronym>PXE</acronym> + as some situations may mandate the use of either method.</para> </listitem> </itemizedlist> @@ -2480,6 +2482,111 @@ other hosts.</para> </caution> + <para>All the information in this section has been tested + using FreeBSD releases 4.9-RELEASE-p2 and 5.2-RELEASE. The text is + primarily structured for 4.X usage. Notes have been inserted where + appropriate to indicate 5.X changes.</para> + + + + <sect2> + <title>Background information</title> + + <para>Setting up diskless workstations is both relatively + straightforward and prone to errors. These are sometimes + difficult to diagnose for a number of reasons. For example: + <itemizedlist> + <listitem> + <para>Compile time options may determine the possible values + or syntax of runtime parameters.</para> + </listitem> + <listitem> + <para>Error messages are often cryptic or totally + absent.</para> + </listitem> + </itemizedlist> + + <para>In this context, having some knowledge of the background + mechanisms involved is very useful to solve the problems that + may arise.</para> + + <para>Several operations need to be performed for a successful + bootstrap:</para> + + <itemizedlist> + <listitem> + <para>The machine needs to obtain initial + parameters such as its IP address, executable file name, + server name, root path. This is done using the DHCP or + BOOTP protocols. DHCP is a compatible extension of BOOTP, + and uses the same port numbers and basic packet + format.</para> + <para>It is possible to configure a system to use only BOOTP. + The &man.bootpd.8; server program is included in the + base FreeBSD system.</para> + + <para>However, DHCP has a number of advantages over BOOTP (nicer + configuration files, possibility of using PXE, plus many + others not directly related to diskless operation), and + we shall describe mainly a DHCP configuration, with + equivalent exemples using &man.bootpd.8; when possible. + The sample configuration will use the ISC DHCP software + package (release 3.0.1.r11 was installed on the test + server: ).</para> + </listitem> + + <listitem> + <para>The machine needs to transfer one or several + programs to local memory. Either TFTP or NFS are used. The + choice between TFTP and NFS is a compile time option in + several places. A common source of error is to specify + file names for the wrong protocol: TFTP typically + transfers all files from a single directory on the server, + and would expect file names relative to this + directory. NFS needs absolute file paths.</para> + </listitem> + + <listitem> + <para>The possible intermediate bootstrap programs and the + kernel need to be initialized and executed. There are + several important variations in this area:</para> + <itemizedlist> + <listitem> + <para>PXE will load &man.pxeboot.8;, + which is a modified version of the FreeBSD third stage + loader. The loader will obtain most parameters necessary + to system startup, and leave them in the kernel + environment before transferring control. It is possible to + use a GENERIC kernel in this case.</para> + </listitem> + + <listitem> + <para><application>etherboot</application>, will directly + load the kernel, with less preparation. A kernel with + specific options will need to be built.</para> + </listitem> + </itemizedlist> + + <para>PXE and <application>etherboot</application> work + equally well with 4.X systems. Because 5.X kernels + normally let the loader do more work for them, PXE is + preferred for 5.X systems.</para> + <para>If your BIOS and network + cards support PXE, you should probably use it. However, + it is still possible to start a 5.2 system with + <application>etherboot</application></para> + + </listitem> + + <listitem> + <para>Finally, the machine needs to access its filesystems. NFS + is used in all cases.</para> + </listitem> + </itemizedlist> + + <para>See also &man.diskless.8;</para> + + </sect2> <sect2> <title>Setup Instructions</title> @@ -2491,31 +2598,7 @@ <secondary>booting</secondary> </indexterm> - <para>There are two protocols that are commonly used to boot a - workstation that retrieves its configuration over the network: BOOTP - and DHCP. They are used at several points in the workstation - bootstrap:</para> - <itemizedlist> - <listitem><para><application>etherboot</application> uses - DHCP (by default) or BOOTP (needs a configuration option) to - find the kernel. (PXE uses DHCP).</para> - </listitem> - <listitem><para>The kernel uses BOOTP to locate the NFS - root.</para> - </listitem> - </itemizedlist> - - <para>It is possible to configure a system to use only BOOTP. - The &man.bootpd.8; server program is included in the - base FreeBSD system.</para> - - <para>However, DHCP has a number of advantages over BOOTP (nicer - configuration files, possibility of using PXE, plus many others - not directly related to diskless operation), and we shall describe - both a pure BOOTP, and a BOOTP+DHCP configuration, with an - emphasis on the latter, which will use the ISC DHCP software - package.</para> - + <sect4> <title>Configuration Using ISC DHCP</title> <indexterm> @@ -2526,7 +2609,7 @@ <para>The <application>isc-dhcp</application> server can answer both BOOTP and DHCP requests.</para> - <para>As of release 4.4, <application>isc-dhcp + <para>As of release 4.9, <application>isc-dhcp 3.0</application> is not part of the base system. You will first need to install the <filename role="package">net/isc-dhcp3-server</filename> port or the @@ -2536,7 +2619,9 @@ <para>Once <application>isc-dhcp</application> is installed, it needs a configuration file to run, (normally named <filename>/usr/local/etc/dhcpd.conf</filename>). Here follows - a commented example:</para> + a commented example, where host <literal>margaux</literal> + uses <application>etherboot</application> and host + <literal>corbieres</literal> uses PXE.</para> <programlisting> default-lease-time 600; @@ -2556,9 +2641,16 @@ hardware ethernet 01:23:45:67:89:ab; fixed-address margaux.example.com; next-server 192.168.4.4;<co id="co-dhcp-next-server"> - filename "/tftpboot/kernel.diskless";<co id="co-dhcp-filename"> + filename "/data/misc/kernel.diskless";<co id="co-dhcp-filename"> option root-path "192.168.4.4:/data/misc/diskless";<co id="co-dhcp-root-path"> } + host corbieres { + hardware ethernet 00:02:b3:27:62:df; + fixed-address corbieres.example.com; + filename "pxeboot";<co id="co-dhcp-filenamePxe"> + next-server 192.168.4.4; + option root-path "192.168.4.4:/data/misc/diskless"; + } } </programlisting> @@ -2573,26 +2665,37 @@ </callout> <callout arearefs="co-dhcp-next-server"><para>The - <literal>next-server</literal> directive designates - the TFTP server (the default is to use the same host as the - DHCP server).</para> + <literal>next-server</literal> directive designates the TFTP + or NFS server to use for loading loader or kernel file (the + default is to use the same host as the DHCP server).</para> </callout> - <callout arearefs="co-dhcp-filename"><para>The - <literal>filename</literal> directive defines the file that - <application>etherboot</application> will load as a - kernel. - <note><para>PXE appears to prefer a relative file - name, and it loads <command>pxeboot</command>, not the - kernel (<literal>option filename - "pxeboot"</literal>).</para> - </note> - </para> + <callout arearefs="co-dhcp-filename"><para>The + <literal>filename</literal> directive defines the file that + <application>etherboot</application> or PXE will load for the + next execution step. It must be specified according to the + transfer method used. <application>etherboot</application> can + be compiled to use NFS or TFTP. The FreeBSD port + configures NFS by default.</para> + </callout> + + <callout arearefs="co-dhcp-filenamePxe"><para>PXE uses TFTP, + which is why a relative file name is used here (this may + depend on the TFTP server configuration, but would be fairly + typical). Also, PXE loads <filename>pxeboot</filename>, not + the kernel. There are other interesting possibilities, like + loading <filename>pxeboot</filename> from a FreeBSD cdrom + <filename>/boot</filename> directory (as &man.pxeboot.8; can + load a GENERIC kernel, this makes it possible to use PXE to + boot from a remote cdrom).</para> </callout> <callout arearefs="co-dhcp-root-path"><para>The - <literal>root-path</literal> option defines the path to - the root filesystem, in usual NFS notation.</para> + <literal>root-path</literal> option defines the path to the + root filesystem, in usual NFS notation. When using PXE, it is + possible to leave off the host's IP as long as you do not + enable the kernel option BOOTP. The NFS server will then be + the same as the TFTP one.</para> </callout> </calloutlist> @@ -2605,7 +2708,7 @@ </indexterm> <para>Here follows an equivalent <command>bootpd</command> - configuration. This would be found in + configuration (reduced to one client). This would be found in <filename>/etc/bootptab</filename>.</para> <para>Please note that <application>etherboot</application> @@ -2656,6 +2759,10 @@ of everything. Else refer to <xref linkend="ports"> for information about ports and packages.</para> + <para>You can change the <application>etherboot</application> + configuration (ie: to use TFTP instead of NFS) by editing the + Config file in the etherboot source directory.</para> + <para>For our setup, we shall use a boot floppy. For other methods (PROM, or dos program), please refer to the <application>etherboot</application> documentation.</para> @@ -2677,6 +2784,30 @@ </sect3> + <sect3> + <title>Booting with PXE</title> + + <para>By default, the &man.pxeboot.8; loader loads the kernel via NFS. + It can be compiled to use TFTP instead by specifying the + LOADER_TFTP_SUPPORT option in + <filename>/etc/make.conf</filename> . See the comments in + <filename>/etc/defaults/make.conf</filename> (or + <filename>/usr/share/examples/etc/make.conf</filename> for 5.x + systems) for instructions.</para> + + <para>There are two other undocumented + <filename>make.conf</filename> options which may be useful for + setting up a serial console diskless machine: + BOOT_PXELDR_PROBE_KEYBOARD, and BOOT_PXELDR_ALWAYS_SERIAL (the + latter only exists on FreeBSD 5.x).</para> + + + <para>To use PXE when the machine starts, you will usually + need to select the <literal>Boot from network</literal> option + in your BIOS setup, or type a function key during the PC + initialization.</para> + + </sect3> <sect3> <title>Configuring the TFTP and NFS Servers</title> @@ -2690,8 +2821,10 @@ <secondary>diskless operation</secondary> </indexterm> - <para>You need to enable <command>tftpd</command> on the TFTP - server:</para> + <para>If any of the boostrap programs (PXE, pxeboot, or etherboot) + are configured to use TFTP, you need to enable + <command>tftpd</command> on the file server:</para> + <procedure> <step> <para>Create a directory from which <command>tftpd</command> @@ -2702,7 +2835,7 @@ <para>Add this line to your <filename>/etc/inetd.conf</filename>:</para> - <programlisting>tftp dgram udp wait root /usr/libexec/tftpd tftpd -s /tftpboot</programlisting> + <programlisting>tftp dgram udp wait root /usr/libexec/tftpd tftpd -s /tftpboot</programlisting> <note><para>It appears that at least some PXE versions want the TCP version of TFTP. In this case, add a second line, @@ -2722,7 +2855,7 @@ location is set in both <filename>inetd.conf</filename> and <filename>dhcpd.conf</filename>.</para> - <para>You also need to enable NFS and export the + <para>In all cases, you also need to enable NFS and export the appropriate filesystem on the NFS server.</para> <procedure> @@ -2738,7 +2871,7 @@ point and replace <replaceable>margaux</replaceable> with the name of the diskless workstation):</para> - <programlisting><replaceable>/data/misc</replaceable> -alldirs -ro <replaceable>margaux</replaceable></programlisting> + <programlisting><replaceable>/data/misc</replaceable> -alldirs -ro <replaceable>margaux corbieres</replaceable></programlisting> </step> <step> <para>Tell <command>mountd</command> to reread its configuration @@ -2759,23 +2892,45 @@ <secondary>kernel configuration</secondary> </indexterm> - <para>Create a kernel configuration file for the diskless client - with the following options (in addition to the usual - ones):</para> + <para>If using <application>etherboot</application>, you need to + create a kernel configuration file for the diskless client + with the following options (in addition to the usual ones):</para> <programlisting> - options BOOTP # Use BOOTP to obtain IP address/hostname - options BOOTP_NFSROOT # NFS mount root filesystem using BOOTP info - options BOOTP_COMPAT # Workaround for broken bootp daemons. + options BOOTP # Use BOOTP to obtain IP address/hostname + options BOOTP_NFSROOT # NFS mount root filesystem using BOOTP info </programlisting> - <para>You may also want to use <literal>BOOTP_NFSV3</literal> and - <literal>BOOTP_WIRED_TO</literal> (refer to <filename>LINT</filename>).</para> - + <para>You may also want to use <literal>BOOTP_NFSV3</literal>, + <literal>BOOT_COMPAT</literal> and + <literal>BOOTP_WIRED_TO</literal> (refer to + <filename>LINT</filename>).</para> + + <para>These option names are historical and slightly + misleading as they actually enable indifferent use of DHCP or BOOTP + inside the kernel (it is also possible to force strict BOOTP or + DHCP use).</para> + <para>Build the kernel (See <xref linkend="kernelconfig">), - and copy it to the tftp directory, under the name listed - in <filename>dhcpd.conf</filename>.</para> + and copy it to the place specified in + <filename>dhcpd.conf</filename>.</para> + <note><para>When using PXE, building a kernel with the above + options is not strictly necessary. Enabling them will cause more + DHCP requests to be issued during kernel startup, with a small + risk of inconsistency between the new values and those retrieved + by pxeboot in some special cases. The advantage of using them is + that the host name will be set as a side effect. Else you will + need to set the host name by another method, for example in a + client-specific <filename>rc.conf</filename> file.</para> </note> + + <note><para>In order to be loadable with + <application>etherboot</application>, a 5.x kernel needs to have + the device hints compiled in. You would typically set the + following option in the configuration file (see the NOTES + configuration comments file). + <programlisting>hints "GENERIC.hints" </programlisting> + </para></note> </sect3> @@ -2798,6 +2953,13 @@ the place where the filesystem will be created (the <literal>DEST</literal> variable).</para> + <note><para>The <filename>clone_root</filename> script distributed + with FreeBSD 5.2 is out of date relative to the 4.x version, and + also needs a few specific 5.x adjustments. Please refer to PR + <ulink url="http://www.freebsd.org/cgi/query-pr.cgi?pr=misc/62417"> + misc/62417</ulink>.</para> + </note> + <para>Refer to the comments at the top of the script for instructions. They explain how the base filesystem is built, and how files may be selectively overridden by versions specific @@ -2812,39 +2974,68 @@ other examples in the <filename>diskless</filename> directory, they actually document a configuration method which is distinct from the one used by <filename>clone_root</filename> and - <filename>/etc/rc.diskless[12]</filename>, which is a little - confusing. Use them for reference only, except if you prefer - the method that they describe, in which case you will need + the system startup scripts in <filename>/etc</filename>, which + is a little confusing. Use them for reference only, except if + you prefer the method that they describe, in which case you will need customized <filename>rc</filename> scripts.</para> + + <note><para>When using PXE, it may happen that the loader will + get a wrong idea of the root filesystem to be used by the + kernel (this will be the case if the cloned root + <filename>/etc/fstab</filename> is a copy of the server's, + as the client-specific version will not be visible by the + loader, which looks for hints there).</para> + <para>This will cause + error messages at the end of the kernel initialization, + when the kernel tries to mount the wrong root and fails, + or even actual errors if it succeeds because a local disk + happens to be present. </para> + + <para>It is possible to prevent this problem from happening + by inserting the following line in + <filename>/boot/loader.conf</filename>:</para> + <programlisting> + vfs.root.mountfrom="" + </programlisting> + </note> + + </sect3> <sect3> <title>Configuring Swap</title> <para>If needed, a swap file located on the server can be - accessed via NFS. The exact <filename>bootptab</filename> - or <filename>dhcpd.conf</filename> options are not clearly - documented at this time. The following configuration - suggestions have been reported to work in some installations - using isc-dhcp 3.0rc11.</para> + accessed via NFS. One of the methods commonly used to do this + has been discontinued in release 5.X.</para> + + <sect4> + <title>NFS swap with FreeBSD 4.x</title> + + <para>The swap file location and size can be specified with + BOOTP/DHCP FreeBSD-specific options 128 and 129. Examples of + configuration files for isc-dhcp 3 or bootpd follow.</para> + <procedure> <step><para>Add the following lines to <filename>dhcpd.conf</filename>:</para> <programlisting> - # Global section - option swap-path code 128 = string; - option swap-size code 129 = integer 32; - - host margaux { - ... # Standard lines, see above - option swap-path <replaceable>"192.168.4.4:/netswapvolume/netswap"</replaceable>; - option swap-size <replaceable>64000</replaceable>; - } + # Global section + option swap-path code 128 = string; + option swap-size code 129 = integer 32; + + host margaux { + ... # Standard lines, see above + option swap-path <replaceable>"192.168.4.4:/netswapvolume/netswap"</replaceable>; + option swap-size <replaceable>64000</replaceable>; + } </programlisting> - <para>The idea is that, at least for a FreeBSD client, - DHCP/BOOTP option code 128 is the path to the NFS swap file, - and option code 129 is the swap size in kilobytes. Older - versions of <command>dhcpd</command> allowed a syntax of + + <para><literal>swap-path</literal> is the path to a directory + where swap files will be located. Each file will be named + <filename>swap.<replaceable>clientIp</replaceable></filename>.</para> + + <para>Older versions of <command>dhcpd</command> used a syntax of <literal>option option-128 "...</literal>, which is no longer supported.</para> <para><filename>/etc/bootptab</filename> would use the @@ -2860,12 +3051,12 @@ <step> <para>On the NFS swap file server, create the swap file(s)</para> - <screen> - &prompt.root; <userinput>mkdir <replaceable>/netswapvolume/netswap</replaceable></userinput> - &prompt.root; <userinput>cd <replaceable>/netswapvolume/netswap</replaceable></userinput> - &prompt.root; <userinput>dd if=/dev/zero bs=1024 count=<replaceable>64000</replaceable> of=swap.<replaceable>192.168.4.6</replaceable></userinput> - &prompt.root; <userinput>chmod 0600 swap.<replaceable>192.168.4.6</replaceable></userinput> - </screen> + <screen> + &prompt.root; <userinput>mkdir <replaceable>/netswapvolume/netswap</replaceable></userinput> + &prompt.root; <userinput>cd <replaceable>/netswapvolume/netswap</replaceable></userinput> + &prompt.root; <userinput>dd if=/dev/zero bs=1024 count=<replaceable>64000</replaceable> of=swap.<replaceable>192.168.4.6</replaceable></userinput> + &prompt.root; <userinput>chmod 0600 swap.<replaceable>192.168.4.6</replaceable></userinput> + </screen> <para><replaceable>192.168.4.6</replaceable> is the IP address for the diskless client.</para> </step> @@ -2874,13 +3065,31 @@ <para>On the NFS swap file server, add the following line to <filename>/etc/exports</filename>:</para> <programlisting> - <replaceable>/netswapvolume</replaceable> -maproot=0:10 -alldirs <replaceable>margaux</replaceable> +<replaceable>/netswapvolume</replaceable> -maproot=0:10 -alldirs <replaceable>margaux</replaceable> </programlisting> <para>Then tell <application>mountd</application> to reread the exports file, as above.</para> </step> </procedure> + </sect4> + + <sect4> + <title>NFS swap with FreeBSD 5.x</title> + + <para>The kernel does not support enabling NFS swap at boot + time. Swap must be enabled by the startup scripts, by + mounting a writeable file system and creating and enabling a + swap file. This can be done, for example, in + <filename>/etc/rc.local</filename>. Example:</para> + + <screen> + swapfile=/writableVolume/swap/swap.<replaceable>MyIP</replaceable> + dd if=/dev/zero of=$swapfile bs=1k count=1 oseek=100000 + swapon $swapfile + rm -rf $swapfile + </screen> + </sect4> </sect3> <sect3> @@ -2915,7 +3124,7 @@ to create the correct device entries (FreeBSD 5.0 and later use &man.devfs.5; to allocate device nodes transparently for the user, running <command>MAKEDEV</command> on these - versions is useless).</para> + versions is pointless).</para> </sect4>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200403031230.i23CUQAl038646>