From owner-freebsd-stable@FreeBSD.ORG Wed Nov 5 15:56:24 2008 Return-Path: Delivered-To: stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9E5ED106564A for ; Wed, 5 Nov 2008 15:56:24 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from bunrab.catwhisker.org (adsl-63-193-123-122.dsl.snfc21.pacbell.net [63.193.123.122]) by mx1.freebsd.org (Postfix) with ESMTP id D9D3B8FC17 for ; Wed, 5 Nov 2008 15:56:21 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from bunrab.catwhisker.org (localhost [127.0.0.1]) by bunrab.catwhisker.org (8.13.3/8.13.3) with ESMTP id mA5FVE4v038130 for ; Wed, 5 Nov 2008 07:31:14 -0800 (PST) (envelope-from david@bunrab.catwhisker.org) Received: (from david@localhost) by bunrab.catwhisker.org (8.13.3/8.13.1/Submit) id mA5FVEoI038129 for stable@freebsd.org; Wed, 5 Nov 2008 07:31:14 -0800 (PST) (envelope-from david) Date: Wed, 5 Nov 2008 07:31:14 -0800 From: David Wolfskill To: stable@freebsd.org Message-ID: <20081105153114.GA37748@bunrab.catwhisker.org> Mail-Followup-To: David Wolfskill , stable@freebsd.org Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="fdj2RfSjLxBAspz7" Content-Disposition: inline User-Agent: Mutt/1.4.2.1i Cc: Subject: Using r/o root with amd(8)-mounted file systems X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Nov 2008 15:56:24 -0000 --fdj2RfSjLxBAspz7 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In networks that I control and which are "sufficiently small" while having enough resources to make it practical -- such as at home -- I like to do a few things to split up the workload and make the "common case" (of merely quietly doing their jobs) easier for the bulk of the machines ... at the expense of needing to tweak things a bit initially to get there, and needing to do a bit more work for upgrades. For example, one of the things I like to do is set up "production" machines (e.g., my firewall box and the central mail server) so they: * Each have 2 separate bootable slices, each of which contains a fully-functional root on the "a" partition and /usr on the "d" partition, and a 3rd slice to contain "everything else" (that is used regardless of which slice is the current boot slice): swap space, /var, and a file system that contains the directories where the /home and /usr/local symlinks point. (Yes, I make /usr/local a symlink.) Because I can easily control which slice is the default boot slice via boot0cfg(8), I use the FreeBSD boot loader. * Use NIS for "installation-wide" notions of users & groups. (Hey; one of the machines at home is a SPARCstation 5/170, after all.) * Use NFS for making certain file systems & directories have an "appearance" on the local machine. (Home directories & a few others are presently served by the above-cited SS5/170, though I've started lobbying the "family CFO" to free up funds to migrate that job to a ReadyNAS. /usr/{obj,ports,src} are hosted on the build machine.) * Avoid "hard" NFS mounts. I use amd(8) to manage the NFS mounts, and it's been working well for me for around a decade or so. * Do not have their own /usr/src, /usr/obj, or /usr/ports directory hierarchies. Rather, these are NFS-mounted from a dedicated "build machine" that has no role in the usual day-to-day "production" activities. the build machine has a local private mirror of the FreeBSD CVS repository which I update in 2 stages overnight (via cron(8), of course), and I track branches of interest on it, usually daily, as well as update ports on it daily. At present, I'm tracking RELENG_6, RELENG_7, and HEAD. Thus, the build machine, in addition to building the "world" (userland) and its own kernel, also builds kernels for the other machines. * Mount /usr read-only. Yes, this becomes a slight nuisance when it's time to upgrade, but that nearly vanishes inside a few csh(1) aliases. It's slightly more annoying when it's time to upgrade ports on production machines, but I still find it useful: it provides a degree of assurance that things aren't likely changing without my knowledge. And should there be a reboot, that's one more file system that need not be checked. (And there have been cases where the UPS batteries haven't lasted as long as an electrical supply outage.) The above all have been working well for me -- as long as I've had a working build machine, anyway. I had tried mounting the root file system read-only (back in 3.x days); while it mostly worked, sshd(8) threw a bit of a hissy-fit because it couldn't chown(1) a pty entry in /dev. And since my normal mode of operation is to access everything from my laptop (running FreeBSD, of course) vis ssh(1), I wasn't too keen on risking running afoul of sshd(8). :-} Now that /dev is merely a figment of the kernel's imagination :-}, I thought I'd re-try mounting root as read-only. As expected, sshd(8) didn't complain -- at least, not about ownership of a pty. What I did encounter -- at least sometimes -- is that If I specify that / is read-only in /etc/fstab, on reboot: * sometimes everything work nicely. * other times, the interaction between the read-only root and amd(8) is such that amd(8) is started, but doesn't actually work. In such cases, a workaround is to mount root read-write, restart amd(8), then mount root read-only. I'm a bit bothered by the nuisance of the latter, but even more concerned about the apparent lack of determinism in the process. Any ideas on how to track this down? The most recent occurrence was on a machine I'm in the process of setting up to replace our internal mail server: albert(7.1-P)[1] uname -a FreeBSD albert.catwhisker.org 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #1: Wed= Nov 5 05:31:00 PST 2008 root@freebeast.catwhisker.org:/common/S3/obj/= usr/src/sys/ALBERT i386 albert(7.1-P)[2]=20 I rebooted it about 5 times in succession with amd(8) failing to do its job; on the first & the last of these, I performed the above-cited "workaround", after which a reboot came up "normally": albert(7.1-P)[2] mount /dev/ad0s2a on / (ufs, local, read-only, soft-updates) devfs on /dev (devfs, local) /dev/ad0s2d on /usr (ufs, NFS exported, local, read-only) /dev/ad0s3d on /var (ufs, local, soft-updates) /dev/ad0s3e on /bkp (ufs, local, soft-updates) /dev/ad1s1d on /common (ufs, local, soft-updates) /dev/md0 on /tmp (ufs, asynchronous, local) pid660@albert:/host on /host (nfs) pid660@albert:/net on /net (nfs) pogo:/cdrom on /.amd_mnt/pogo/host/cdrom (nfs, nosuid) pogo:/export on /.amd_mnt/pogo/host/export (nfs, nosuid) pogo:/export/bd1 on /.amd_mnt/pogo/host/export/bd1 (nfs, nosuid) pogo:/export/bd2 on /.amd_mnt/pogo/host/export/bd2 (nfs, nosuid) pogo:/export/home on /.amd_mnt/pogo/host/export/home (nfs, nosuid) pogo:/export/local on /.amd_mnt/pogo/host/export/local (nfs, nosuid) albert(7.1-P)[3] uptime 7:29AM up 17 mins, 1 user, load averages: 0.00, 0.00, 0.01 albert(7.1-P)[4]=20 Deploying the machine in production is neither urgent nor critical at this point, so I have some time to work on it. Here's where rcorder(8) has to say: albert(7.1-P)[3] rcorder /etc/rc.d/* /usr/local/etc/rc.d/* /etc/rc.d/dumpon /etc/rc.d/ddb /etc/rc.d/initrandom /etc/rc.d/geli /etc/rc.d/gbde /etc/rc.d/encswap /etc/rc.d/ccd /etc/rc.d/swap1 /etc/rc.d/early.sh /etc/rc.d/fsck /etc/rc.d/root /etc/rc.d/hostid /etc/rc.d/mdconfig /etc/rc.d/mountcritlocal /etc/rc.d/zfs /etc/rc.d/FILESYSTEMS /etc/rc.d/var /etc/rc.d/cleanvar /etc/rc.d/random /etc/rc.d/adjkerntz /etc/rc.d/atm1 /etc/rc.d/hostname /etc/rc.d/ipfilter /etc/rc.d/ipnat /etc/rc.d/ipfs /etc/rc.d/kldxref /etc/rc.d/sppp /etc/rc.d/addswap /etc/rc.d/auto_linklocal /etc/rc.d/sysctl /etc/rc.d/serial /etc/rc.d/netif /etc/rc.d/ip6addrctl /etc/rc.d/atm2 /etc/rc.d/pfsync /etc/rc.d/pflog /etc/rc.d/pf /etc/rc.d/isdnd /etc/rc.d/ppp /etc/rc.d/routing /etc/rc.d/ip6fw /etc/rc.d/network_ipv6 /etc/rc.d/devd /etc/rc.d/ipsec /etc/rc.d/ipfw /etc/rc.d/nsswitch /etc/rc.d/resolv /etc/rc.d/mroute6d /etc/rc.d/route6d /etc/rc.d/mrouted /etc/rc.d/routed /etc/rc.d/netoptions /etc/rc.d/NETWORKING /etc/rc.d/mountcritremote /etc/rc.d/ldconfig /etc/rc.d/tmp /etc/rc.d/cleartmp /usr/local/etc/rc.d/xfs /usr/local/etc/rc.d/xdm.sh.noauto /usr/local/etc/rc.d/rplayd.sh.sample /etc/rc.d/accounting /etc/rc.d/devfs /etc/rc.d/ipmon /etc/rc.d/mdconfig2 /etc/rc.d/newsyslog /etc/rc.d/syslogd /etc/rc.d/savecore /etc/rc.d/archdep /etc/rc.d/abi /etc/rc.d/SERVERS /etc/rc.d/named /etc/rc.d/ntpdate /etc/rc.d/rpcbind /etc/rc.d/nfsclient /etc/rc.d/nisdomain /etc/rc.d/ypserv /etc/rc.d/ypbind /etc/rc.d/amd /etc/rc.d/atm3 /etc/rc.d/auditd /etc/rc.d/dmesg /etc/rc.d/ipxrouted /etc/rc.d/kerberos /etc/rc.d/kadmind /etc/rc.d/keyserv /etc/rc.d/kpasswdd /etc/rc.d/quota /etc/rc.d/nfsserver /etc/rc.d/mountd /etc/rc.d/nfsd /etc/rc.d/statd /etc/rc.d/lockd /etc/rc.d/pppoed /etc/rc.d/pwcheck /etc/rc.d/virecover /etc/rc.d/DAEMON /etc/rc.d/apm /etc/rc.d/apmd /etc/rc.d/bootparams /etc/rc.d/hcsecd /etc/rc.d/bthidd /etc/rc.d/local /etc/rc.d/lpd /etc/rc.d/motd /etc/rc.d/mountlate /etc/rc.d/nscd /etc/rc.d/ntpd /etc/rc.d/powerd /etc/rc.d/rarpd /etc/rc.d/sdpd /etc/rc.d/rfcomm_pppd_server /etc/rc.d/rtadvd /etc/rc.d/rwho /etc/rc.d/timed /etc/rc.d/ugidfw /etc/rc.d/yppasswdd /etc/rc.d/LOGIN /usr/local/etc/rc.d/mysql-server /usr/local/etc/rc.d/htcacheclean /usr/local/etc/rc.d/dbus rcorder: requirement `usbd' in file `/usr/local/etc/rc.d/hald' has no provi= ders. /usr/local/etc/rc.d/hald /usr/local/etc/rc.d/ffserver /usr/local/etc/rc.d/apache22 /etc/rc.d/ypxfrd /etc/rc.d/ypupdated /etc/rc.d/ypset /etc/rc.d/wpa_supplicant /etc/rc.d/watchdogd /etc/rc.d/syscons /etc/rc.d/sshd /etc/rc.d/sendmail /etc/rc.d/cron /etc/rc.d/jail /etc/rc.d/localpkg /etc/rc.d/securelevel /etc/rc.d/power_profile /etc/rc.d/othermta /etc/rc.d/natd /etc/rc.d/msgs /etc/rc.d/moused /etc/rc.d/mixer /etc/rc.d/inetd /etc/rc.d/idmapd /etc/rc.d/hostapd /etc/rc.d/geli2 /etc/rc.d/ftpd /etc/rc.d/ftp-proxy /etc/rc.d/dhclient /etc/rc.d/bsnmpd /etc/rc.d/bridge /etc/rc.d/bluetooth /etc/rc.d/bgfsck albert(7.1-P)[4]=20 Peace, david --=20 David H. Wolfskill david@catwhisker.org Depriving a girl or boy of an opportunity for education is evil. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --fdj2RfSjLxBAspz7 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (FreeBSD) iEYEARECAAYFAkkRvEIACgkQmprOCmdXAD33wQCdF1+fmdTD2KAh9mELNmqZ5AqV YtoAn0EXCI3UZmXgFUIBdGDp6STLBNWF =MHwe -----END PGP SIGNATURE----- --fdj2RfSjLxBAspz7--