From owner-freebsd-jail@freebsd.org Wed Jul 6 12:13:38 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A3B94B75437 for ; Wed, 6 Jul 2016 12:13:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 92F131517 for ; Wed, 6 Jul 2016 12:13:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u66CDbYR011899 for ; Wed, 6 Jul 2016 12:13:38 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-jail@FreeBSD.org Subject: [Bug 209112] /usr/sbin/jail jails fail to launch with possible race when jails mount common dir with nullfs Date: Wed, 06 Jul 2016 12:13:37 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: djn@araxis.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2016 12:13:38 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209112 djn@araxis.com changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |djn@araxis.com --- Comment #4 from djn@araxis.com --- I am also seeing this issue with FreeBSD 10.3-Release-p4 =E2=80=93 I have s= ix jails, and only two or three start on boot (two fail due to a dependency on another jail that fails to start). I encounter no problems when starting the jails post-boot. I added logging to /etc/rc.d/jail in a similar manner to the original poste= r, and see this output: ---- mount_nullfs: /jails/pgsql/data: mount_nullfs: /jails/hg/data: Operation not supported by device Operation not supported by device jail: pgsql: /sbin/mount -t nullfs -o rw /data/pgsql /jails/pgsql/data: fai= led jail: hg: /sbin/mount -t nullfs -o rw /data/hg /jails/hg/data: failed jail: support: skipped jail: logic: skipped www-bhs-0: created backup-bhs-0: created ... ---- Here is my jail.conf: ---- # START BLOCK: Araxis jail defaults -- DO NOT EDIT exec.start =3D "/bin/sh /etc/rc"; exec.stop =3D "/bin/sh /etc/rc.shutdown"; exec.clean; mount.devfs; mount.fdescfs; allow.noset_hostname; host =3D new; # temporarily allow during development/testing allow.raw_sockets; # END BLOCK: Araxis jail defaults # START BLOCK: Araxis jail settings for jail www-bhs-0.araxis.net -- DO NOT EDIT www-bhs-0 { mount.fstab =3D "/jails/.fstabs/www-bhs-0"; path =3D "/jails/www-bhs-0"; host.hostname =3D www-bhs-0.araxis.net; ip4.addr =3D "lo1|10.11.11.2/32"; ip6.addr =3D "ix0|2607:5300:60:9b9c::2/64"; } # END BLOCK: Araxis jail settings for jail www-bhs-0.araxis.net # START BLOCK: Araxis jail settings for jail backup-bhs-0.araxis.net -- DO = NOT EDIT backup-bhs-0 { mount.fstab =3D "/jails/.fstabs/backup-bhs-0"; path =3D "/jails/backup-bhs-0"; host.hostname =3D backup-bhs-0.araxis.net; ip4.addr =3D "lo2|10.12.12.8/32"; enforce_statfs =3D 1; allow.mount; allow.mount.zfs; exec.poststart =3D "/sbin/zfs set jailed=3Don zroot/bkup && /sbin/zfs jail ${name} zroot/bkup && jexec ${name} /sbin/zfs mount -a"; exec.prestop =3D "/sbin/zfs unjail ${name} zroot/bkup && /sbin/zfs set jailed=3Doff zroot/bkup"; } # END BLOCK: Araxis jail settings for jail backup-bhs-0.araxis.net # START BLOCK: Araxis jail settings for jail pgsql.araxis.net -- DO NOT EDIT pgsql { mount.fstab =3D "/jails/.fstabs/pgsql"; path =3D "/jails/pgsql"; host.hostname =3D pgsql.araxis.net; ip4.addr =3D "lo2|10.12.12.4/32"; allow.sysvipc; } # END BLOCK: Araxis jail settings for jail pgsql.araxis.net # START BLOCK: Araxis jail settings for jail support.araxis.net -- DO NOT E= DIT support { mount.fstab =3D "/jails/.fstabs/support"; path =3D "/jails/support"; host.hostname =3D support.araxis.net; ip4.addr =3D "lo2|10.12.12.5/32"; depend =3D pgsql; } # END BLOCK: Araxis jail settings for jail support.araxis.net # START BLOCK: Araxis jail settings for jail logic.araxis.com -- DO NOT EDIT logic { mount.fstab =3D "/jails/.fstabs/logic"; path =3D "/jails/logic"; host.hostname =3D logic.araxis.com; ip4.addr =3D "lo2|10.12.12.6/32"; depend =3D pgsql; } # END BLOCK: Araxis jail settings for jail logic.araxis.com # START BLOCK: Araxis jail settings for jail hg.araxis.net -- DO NOT EDIT hg { mount.fstab =3D "/jails/.fstabs/hg"; path =3D "/jails/hg"; host.hostname =3D hg.araxis.net; ip4.addr =3D "lo2|10.12.12.3/32"; } # END BLOCK: Araxis jail settings for jail hg.araxis.net ---- The various fstab files are nearly identical. Here=E2=80=99s the one (/jails/.fstabs/pgsql) for the pgsql jail, which always fails to start on b= oot: ---- # Device Mountpoint FStype Options Dump Pass# /data/pgsql /jails/pgsql/data nullfs rw 0 0 ---- --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-jail@freebsd.org Wed Jul 6 12:20:03 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B1272B7565B for ; Wed, 6 Jul 2016 12:20:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A025A17F5 for ; Wed, 6 Jul 2016 12:20:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u66CK3Ve093049 for ; Wed, 6 Jul 2016 12:20:03 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-jail@FreeBSD.org Subject: [Bug 209112] /usr/sbin/jail jails fail to launch with possible race when jails mount common dir with nullfs Date: Wed, 06 Jul 2016 12:20:03 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: djn@araxis.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2016 12:20:03 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209112 --- Comment #5 from djn@araxis.com --- I should also like to add my voice to those requesting logging of boot-time jail startup problems. The lack of any diagnostics cost me several hours of time in tracking down this problem. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-jail@freebsd.org Wed Jul 6 12:57:08 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A4B2BB75DC0 for ; Wed, 6 Jul 2016 12:57:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 865851885 for ; Wed, 6 Jul 2016 12:57:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id u66Cv8RJ070501 for ; Wed, 6 Jul 2016 12:57:08 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-jail@FreeBSD.org Subject: [Bug 209112] /usr/sbin/jail jails fail to launch with possible race when jails mount common dir with nullfs Date: Wed, 06 Jul 2016 12:57:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: bin X-Bugzilla-Version: 10.3-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: djn@araxis.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-jail@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2016 12:57:08 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D209112 --- Comment #6 from djn@araxis.com --- I have what seems (on first try, at least) to be a viable alternative workaround that is (somewhat) less icky than adding artificial dependencies between jails. Simply add the following two lines to /etc/rc.conf (or /etc/rc.conf.d/jail): jail_parallel_start=3D"NO" jail_list=3D"list of all jails to start" Specifying the jail_list explicitly means that the jail_parallel_start sett= ing can take effect, since the default _ALL case in jail_start() (which ignores jail_parallel_start) is then bypassed. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-jail@freebsd.org Wed Jul 6 16:50:02 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7031B75287 for ; Wed, 6 Jul 2016 16:50:02 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 792FC1AA2 for ; Wed, 6 Jul 2016 16:50:01 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u66Gnqkg030411 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Wed, 6 Jul 2016 16:49:52 GMT (envelope-from list1@gjunka.com) To: freebsd-jail@freebsd.org From: Grzegorz Junka Subject: Effective rule sets in a jail? Message-ID: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com> Date: Wed, 6 Jul 2016 16:49:52 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Jul 2016 16:50:03 -0000 I have the following in my jail.conf: devfs_ruleset = 4; vpn1 { ip4.addr = 10.70.5.254; ip4.addr += "tun0|10.70.5.1 10.70.5.254 mtu 1500 netmask 255.255.255.255"; interface = lagg0; devfs_ruleset = 5; } I expect that in the jail both rules 4 and 5 are active. How can I check that? Grzegorz From owner-freebsd-jail@freebsd.org Thu Jul 7 04:04:44 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BF55DB758DD for ; Thu, 7 Jul 2016 04:04:44 +0000 (UTC) (envelope-from ultima1252@gmail.com) Received: from mail-yw0-x22d.google.com (mail-yw0-x22d.google.com [IPv6:2607:f8b0:4002:c05::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6C4A013AD for ; Thu, 7 Jul 2016 04:04:44 +0000 (UTC) (envelope-from ultima1252@gmail.com) Received: by mail-yw0-x22d.google.com with SMTP id i12so5050559ywa.1 for ; Wed, 06 Jul 2016 21:04:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=zS1OQyP+Fho3y4s2969wxqZTa2NPIBuspy0iaKe9Ffs=; b=AZUF9GFtyGCH0W52gG9AI9ph3xK2JnRwa0ndbZ+AOEFPU7ROis0m/1a6GXy/qrNzEA 9KB09LFBTeTBqlih4HoqjtGNqLrle4Ev4pbfs3NfEwWHY2a/0bLzgg6O7evhrPeMP3dy SZ0JJqvl/9aoEVRDxuoMx31y5pnNP5s7RZ7S2jpgPtQMNNn2fP6ol16zHlrkZ7IgMCeV qmQr2EEEKz1qXif363hdVZZe4xwW2gr44yL/+P5vKifpGVyjto7vH47BM5dgG+qnsRgH HKRVCU9BRY/AU2St74T0up5JH/8jTklaplQwK41ZB4J/mx9eS8FFpe3LSm5xHNKKs0Gp FfMw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=zS1OQyP+Fho3y4s2969wxqZTa2NPIBuspy0iaKe9Ffs=; b=VlQSh/Cvq0Hwsi+WQePCUIzAeKcEyJZEUtomhGTRR2kwucoStI0K0xOf83bIBJWHAx ArCmHgJxhvBDgPKP7hrPTNlrgch8C+lxJHgKTqZHnQHr7I+8QzXZ3bO6V7bmTev0hQUJ mKh3iU3OyjZ59syQ4lqBXc1nUosV4eCevHY6HDoND49GD4Hdy10/0EGLlkASMm1wmmtA V8pDDNYM+8JTeSmf6dz5lO41fFzEI89fpzh/w0EWDQPdYBmGWykf17D+IjL1bqlH7Fcy pPuQmjTxmj/HRte0iEIVIP6tnLqeFj8qWA7/46id6QELN75M1yf2CGMsFQ/b+lMeZmnJ tfew== X-Gm-Message-State: ALyK8tKDd9vO5t0FLoXkiKDqouM29ncEZBAW00JoxPv9BjPBsJ5xcokDaifFed306KpjTLJZLXUssXLodZCngA== X-Received: by 10.37.11.129 with SMTP id 123mr16568486ybl.55.1467864283535; Wed, 06 Jul 2016 21:04:43 -0700 (PDT) MIME-Version: 1.0 Received: by 10.129.5.216 with HTTP; Wed, 6 Jul 2016 21:04:43 -0700 (PDT) In-Reply-To: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com> References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com> From: Ultima Date: Thu, 7 Jul 2016 00:04:43 -0400 Message-ID: Subject: Re: Effective rule sets in a jail? To: Grzegorz Junka Cc: freebsd-jail@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2016 04:04:44 -0000 Not so. The top variable, devfs_ruleset = 4 is being set as the default for all jails. The devfs_ruleset = 5 inside the brackets is changing the default value. How to check what ruleset is mounted? That is a great question. I'm not sure of an easy way to check other than verifying the /dev directory inside the jail. Ultima From owner-freebsd-jail@freebsd.org Thu Jul 7 07:53:39 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 45D4DB21F54 for ; Thu, 7 Jul 2016 07:53:39 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 09D5D1845 for ; Thu, 7 Jul 2016 07:53:38 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id B1A9C284E7; Thu, 7 Jul 2016 09:53:29 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 00B5328454; Thu, 7 Jul 2016 09:53:28 +0200 (CEST) Message-ID: <577E0A78.1040600@quip.cz> Date: Thu, 07 Jul 2016 09:53:28 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: Ultima , Grzegorz Junka CC: freebsd-jail@freebsd.org Subject: Re: Effective rule sets in a jail? References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2016 07:53:39 -0000 Ultima wrote on 07/07/2016 06:04: > Not so. The top variable, devfs_ruleset = 4 is being set as the default for > all jails. The devfs_ruleset = 5 inside the brackets is changing the > default value. > > How to check what ruleset is mounted? That is a great question. I'm not > sure of an easy way to check other than verifying the /dev directory inside > the jail. There is no way to set more than one devfs rule to jail AFAIK. You can see the rule number in output of jls -s or jls -n. Miroslav Lachman From owner-freebsd-jail@freebsd.org Thu Jul 7 08:41:36 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5FCDEB769FA for ; Thu, 7 Jul 2016 08:41:36 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 092331D08 for ; Thu, 7 Jul 2016 08:41:35 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u678fW6v053455 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 7 Jul 2016 08:41:33 GMT (envelope-from list1@gjunka.com) Subject: Re: Effective rule sets in a jail? References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com> <577E0A78.1040600@quip.cz> To: freebsd-jail@freebsd.org From: Grzegorz Junka Message-ID: <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com> Date: Thu, 7 Jul 2016 08:41:32 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: <577E0A78.1040600@quip.cz> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2016 08:41:36 -0000 On 07/07/2016 07:53, Miroslav Lachman wrote: > Ultima wrote on 07/07/2016 06:04: >> Not so. The top variable, devfs_ruleset = 4 is being set as the >> default for >> all jails. The devfs_ruleset = 5 inside the brackets is changing the >> default value. >> >> How to check what ruleset is mounted? That is a great question. I'm not >> sure of an easy way to check other than verifying the /dev directory >> inside >> the jail. > > There is no way to set more than one devfs rule to jail AFAIK. > You can see the rule number in output of jls -s or jls -n. > > Miroslav Lachman > I was referring to this clause in the man document: Descendant jails inherit the parent jail's devfs ruleset enforcement. I thought that the outside rule is combined with the inside rule in the jail definition. But thanks for the hint about jls -s, it does shows the (single) active rule set (however without referring to the specific rules defined in devfs.rules or a combination of it). Grzegorz From owner-freebsd-jail@freebsd.org Thu Jul 7 09:04:01 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8CA9CB74164 for ; Thu, 7 Jul 2016 09:04:01 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 50FD81CF2 for ; Thu, 7 Jul 2016 09:04:00 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id E4936284AE; Thu, 7 Jul 2016 11:03:56 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id C875D2848E; Thu, 7 Jul 2016 11:03:55 +0200 (CEST) Message-ID: <577E1AFB.90100@quip.cz> Date: Thu, 07 Jul 2016 11:03:55 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: Grzegorz Junka , freebsd-jail@freebsd.org Subject: Re: Effective rule sets in a jail? References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com> <577E0A78.1040600@quip.cz> <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com> In-Reply-To: <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2016 09:04:01 -0000 Grzegorz Junka wrote on 07/07/2016 10:41: > I was referring to this clause in the man document: > > Descendant jails inherit the parent jail's devfs ruleset enforcement. This is true for hierarchical "nested" jails = jail inside jail. And inheriting doesn't mean merging. You can't allow devices in descendant jail which are not allowed on parent. > I thought that the outside rule is combined with the inside rule in the > jail definition. But thanks for the hint about jls -s, it does shows the > (single) active rule set (however without referring to the specific > rules defined in devfs.rules or a combination of it). You are mixing nested jails context with jail.conf context where "outside" definitions are the defaults for all jails which are not overriding those values with own values. Miroslav Lachman From owner-freebsd-jail@freebsd.org Thu Jul 7 09:42:07 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7733AB74C5B for ; Thu, 7 Jul 2016 09:42:07 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1E83D1236 for ; Thu, 7 Jul 2016 09:42:06 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u679g4Ff054596 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 7 Jul 2016 09:42:05 GMT (envelope-from list1@gjunka.com) Subject: Re: Effective rule sets in a jail? To: freebsd-jail@freebsd.org References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com> <577E0A78.1040600@quip.cz> <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com> <577E1AFB.90100@quip.cz> From: Grzegorz Junka Message-ID: <6ccead58-a38a-80a4-b5b8-a509c4271b8f@gjunka.com> Date: Thu, 7 Jul 2016 09:42:04 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: <577E1AFB.90100@quip.cz> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2016 09:42:07 -0000 On 07/07/2016 09:03, Miroslav Lachman wrote: > Grzegorz Junka wrote on 07/07/2016 10:41: > > >> I was referring to this clause in the man document: >> >> Descendant jails inherit the parent jail's devfs ruleset enforcement. > > This is true for hierarchical "nested" jails = jail inside jail. > And inheriting doesn't mean merging. > You can't allow devices in descendant jail which are not allowed on > parent. > >> I thought that the outside rule is combined with the inside rule in the >> jail definition. But thanks for the hint about jls -s, it does shows the >> (single) active rule set (however without referring to the specific >> rules defined in devfs.rules or a combination of it). > > You are mixing nested jails context with jail.conf context where > "outside" definitions are the defaults for all jails which are not > overriding those values with own values. > > Miroslav Lachman OK, I am just an user, not very familiar with the terminology. For me (as a programmer) inheriting means overriding, so merging the more specific to the less specific declarations. Does it mean that the "inheriting" works in nested declarations but doesn't take into account the default value? In other words, the default is just default unless it re-defined in a jail declaration. If that's the case then wouldn't be more clear to name the "outside" default declaration as default, e.g. "default_devfs_ruleset"? Then it would be more difficult to confuse the default with the one that can be inherited. Grzegorz From owner-freebsd-jail@freebsd.org Thu Jul 7 10:06:43 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4EEC8B75193 for ; Thu, 7 Jul 2016 10:06:43 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AF4231AD3 for ; Thu, 7 Jul 2016 10:06:42 +0000 (UTC) (envelope-from 000.fbsd@quip.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 361322848C; Thu, 7 Jul 2016 12:06:34 +0200 (CEST) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 30B3B28412; Thu, 7 Jul 2016 12:06:33 +0200 (CEST) Message-ID: <577E29A8.5000504@quip.cz> Date: Thu, 07 Jul 2016 12:06:32 +0200 From: Miroslav Lachman <000.fbsd@quip.cz> User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:35.0) Gecko/20100101 Firefox/35.0 SeaMonkey/2.32 MIME-Version: 1.0 To: Grzegorz Junka , freebsd-jail@freebsd.org Subject: Re: Effective rule sets in a jail? References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com> <577E0A78.1040600@quip.cz> <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com> <577E1AFB.90100@quip.cz> <6ccead58-a38a-80a4-b5b8-a509c4271b8f@gjunka.com> In-Reply-To: <6ccead58-a38a-80a4-b5b8-a509c4271b8f@gjunka.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2016 10:06:43 -0000 Grzegorz Junka wrote on 07/07/2016 11:42: > OK, I am just an user, not very familiar with the terminology. For me > (as a programmer) inheriting means overriding, so merging the more > specific to the less specific declarations. > > Does it mean that the "inheriting" works in nested declarations but > doesn't take into account the default value? In other words, the default > is just default unless it re-defined in a jail declaration. If that's > the case then wouldn't be more clear to name the "outside" default > declaration as default, e.g. "default_devfs_ruleset"? Then it would be > more difficult to confuse the default with the one that can be inherited. I think it is simple in current form. (And I am not sys developer, I was web application programmer before I became sysadmin) I started with jails long time before jail2 with jail.conf. Current jail.conf is soooo simpler in comparision with rc.conf style variables. Naming each default variable with different name will be harder to code, harder to write in jail.conf, harder to document in manpages. Almost all programming languages works the same in this context - later variable definition wins. So you can easily define all variables needed to run jails and then set just those specific to one jail - IPs and hostname: ## Typical static defaults: ## Use the rc scripts to start and stop jails. Mount jail's /dev. exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; exec.clean; exec.system_user = "root"; exec.jail_user = "root"; mount.devfs; devfs_ruleset = 4; enforce_statfs = 1; #allow.set_hostname = false; #allow.mount; allow.set_hostname = 0; allow.sysvipc = 0; allow.raw_sockets = 0; ## Dynamic wildcard parameter: path = "/vol1/jail/$name"; exec.consolelog = "/var/log/jail/$name.console"; mount.fstab = "/etc/fstab.$name"; ## Jail myjail0 myjail0 { host.hostname = "myjail0.example.conf"; ip4.addr = 10.20.30.40; } ## Jail myjail1 myjail1 { host.hostname = "myjail1.example.conf"; ip4.addr = 10.20.30.41; } devfs_ruleset is the same as the other variables - you can't (and I hope nobody expect) to merge global default value of e.g. exec.system_user or allow.sysvipc with variables defined in specific jail context. Those variables can have only one value (bool, or string, or number; not an array). It is the same for devfs_rules. Can't have more than one numeric value, can't combine two together. I think you will be familiar with this very soon. Miroslav Lachman From owner-freebsd-jail@freebsd.org Thu Jul 7 11:17:45 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 950FEB76142 for ; Thu, 7 Jul 2016 11:17:45 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B24A17B0 for ; Thu, 7 Jul 2016 11:17:44 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u67BHfSP056496 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 7 Jul 2016 11:17:41 GMT (envelope-from list1@gjunka.com) Subject: Re: Effective rule sets in a jail? To: freebsd-jail@freebsd.org References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com> <577E0A78.1040600@quip.cz> <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com> <577E1AFB.90100@quip.cz> <6ccead58-a38a-80a4-b5b8-a509c4271b8f@gjunka.com> <577E29A8.5000504@quip.cz> From: Grzegorz Junka Message-ID: <4d3f5584-7dd1-e6fc-540f-9ed3f1fb63f4@gjunka.com> Date: Thu, 7 Jul 2016 11:17:41 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: <577E29A8.5000504@quip.cz> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2016 11:17:45 -0000 On 07/07/2016 10:06, Miroslav Lachman wrote: > Grzegorz Junka wrote on 07/07/2016 11:42: > >> OK, I am just an user, not very familiar with the terminology. For me >> (as a programmer) inheriting means overriding, so merging the more >> specific to the less specific declarations. >> >> Does it mean that the "inheriting" works in nested declarations but >> doesn't take into account the default value? In other words, the default >> is just default unless it re-defined in a jail declaration. If that's >> the case then wouldn't be more clear to name the "outside" default >> declaration as default, e.g. "default_devfs_ruleset"? Then it would be >> more difficult to confuse the default with the one that can be >> inherited. > > I think it is simple in current form. (And I am not sys developer, I > was web application programmer before I became sysadmin) > I started with jails long time before jail2 with jail.conf. Current > jail.conf is soooo simpler in comparision with rc.conf style variables. > > Naming each default variable with different name will be harder to > code, harder to write in jail.conf, harder to document in manpages. > > Almost all programming languages works the same in this context - > later variable definition wins. > > So you can easily define all variables needed to run jails and then > set just those specific to one jail - IPs and hostname: > > ## Typical static defaults: > ## Use the rc scripts to start and stop jails. Mount jail's /dev. > exec.start = "/bin/sh /etc/rc"; > exec.stop = "/bin/sh /etc/rc.shutdown"; > exec.clean; > exec.system_user = "root"; > exec.jail_user = "root"; > mount.devfs; > devfs_ruleset = 4; > enforce_statfs = 1; > #allow.set_hostname = false; > #allow.mount; > allow.set_hostname = 0; > allow.sysvipc = 0; > allow.raw_sockets = 0; > > ## Dynamic wildcard parameter: > path = "/vol1/jail/$name"; > exec.consolelog = "/var/log/jail/$name.console"; > mount.fstab = "/etc/fstab.$name"; > > ## Jail myjail0 > myjail0 { > host.hostname = "myjail0.example.conf"; > ip4.addr = 10.20.30.40; > } > > ## Jail myjail1 > myjail1 { > host.hostname = "myjail1.example.conf"; > ip4.addr = 10.20.30.41; > } > > > devfs_ruleset is the same as the other variables - you can't (and I > hope nobody expect) to merge global default value of e.g. > exec.system_user or allow.sysvipc with variables defined in specific > jail context. Those variables can have only one value (bool, or > string, or number; not an array). It is the same for devfs_rules. > Can't have more than one numeric value, can't combine two together. > > I think you will be familiar with this very soon. > > Miroslav Lachman OK, so my confusion steams from the fact that the devfs rules are defined somewhere else and the jail.conf is simply taking into account the rule number, not its content. In that context it indeed makes sense. It could be simply a matter of adding a clarification that each jail can have only one devfs ruleset assigned to it (which then would be calculated according to the standard rules defined in jail.conf), for example: Descendant jails inherit the parent jail's devfs ruleset. Devfs rules enforced in the jail are defined by the single calculated ruleset. What do you think? Grzegorz From owner-freebsd-jail@freebsd.org Thu Jul 7 14:03:17 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 82A43B747EB for ; Thu, 7 Jul 2016 14:03:17 +0000 (UTC) (envelope-from eto.freebsd@ethome.sk) Received: from smtpout6.dnsserver.eu (smtpout6.dnsserver.eu [92.240.253.144]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 438CF1A08 for ; Thu, 7 Jul 2016 14:03:16 +0000 (UTC) (envelope-from eto.freebsd@ethome.sk) Received: from [92.240.253.67] (helo=smtp3s109.dnsserver.eu) by smtpout6.dnsserver.eu with esmtp (Exim 4.84 (FreeBSD)) (envelope-from ) id 1bL9WO-00070o-20 for freebsd-jail@freebsd.org; Thu, 07 Jul 2016 15:39:16 +0200 Received: from [80.242.44.220] (helo=eto-mona.office.smartweb.sk) by smtp3s109.dnsserver.eu with esmtpsa (TLSv1.2:AES128-GCM-SHA256:128) (Exim 4.83 (FreeBSD)) (envelope-from ) id 1bL9WP-000Ak0-DQ for freebsd-jail@freebsd.org; Thu, 07 Jul 2016 15:39:17 +0200 Date: Thu, 7 Jul 2016 15:31:48 +0200 From: "Martin \"eto\" Misuth" To: freebsd-jail@freebsd.org Subject: Re: Effective rule sets in a jail? Message-ID: <20160707153148.7da31609@eto-mona.office.smartweb.sk> In-Reply-To: <4d3f5584-7dd1-e6fc-540f-9ed3f1fb63f4@gjunka.com> References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com> <577E0A78.1040600@quip.cz> <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com> <577E1AFB.90100@quip.cz> <6ccead58-a38a-80a4-b5b8-a509c4271b8f@gjunka.com> <577E29A8.5000504@quip.cz> <4d3f5584-7dd1-e6fc-540f-9ed3f1fb63f4@gjunka.com> Organization: ethome.sk MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 80.242.44.220 X-SA-Exim-Mail-From: eto.freebsd@ethome.sk X-SA-Exim-Scanned: No (on smtp3s109.dnsserver.eu); SAEximRunCond expanded to false X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2016 14:03:17 -0000 On Thu, 7 Jul 2016 11:17:41 +0000 Grzegorz Junka wrote: > > Descendant jails inherit the parent jail's devfs ruleset. Devfs rules > enforced in the jail are defined by the single calculated ruleset. > > What do you think? > IMHO, regarding jails, better mental model would be like this: - any single jail can have one and only one devfs ruleset number assigned - however, different standalone jails can have different devfs ruleset number assigned - nested jails inherit ruleset number from their parent jail Regarding rulesets "inheritance"/"merging" you are probably looking into the wrong place. devfs ruleset system is completely orthogonal to jails, as it is used for other things as well. You can "merge" devfs rulesets in devfs /etc/devfs.rules. Look into /etc/defaults/devfs.rules how initial rulesets are built. First everything is hidden by ruleset 1 aka "devfsrules_hide_all". This is "by default deny" policy, which should, according to me, used whenever one can. Then, new rulesets are created by unhiding various groups of devices. Like for example you have minimal sub-ruleset 2 aka "devfsrules_unhide_basic". That one is required to get minimal working /dev. Otherwise most programs break. Finally ruleset 4 aka "devfsrules_jail" is built, which can be used by jails. I personally "classify" jail types into groups. Let's call such group a jail "class" (for the purpose of classification). Thus to get what you want, you should create custom ruleset per jail "class" and assign it to your jails based on their "class". [devfsrules_jail_class_no_zfs=16] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login Class might be not good word for this, as it is quite "loaded" by now, but I am using it that way. Some jails might end up so special, they require completely fine tuned ruleset. Those cannot be completely "classified" at all like this for example: [devfsrules_jail_proxy=333] add include $devfsrules_hide_all add include $devfsrules_unhide_basic add include $devfsrules_unhide_login add include $devfsrules_unhide_jail_proxy_tuns "devfsrules_unhide_jail_proxy_tuns" sub-rule in this case unhides several tun interfaces used solely by this jail only. devfs.conf files are "parsed" by /etc/rc.d/devfs rc script which is run quite early after boot. If you look at it you will see it is using /etc/rc.subr devfs_* subroutines of rc.d framework which invoke /sbin/devfs helper program. Theoretically if /etc/rc.d/devfs and /etc/rc.subr are not enough for you, you could write helper script to invoke /sbin/devfs to setup most convoluted rule ids directly by hand. eto From owner-freebsd-jail@freebsd.org Thu Jul 7 18:27:00 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5401BB75577 for ; Thu, 7 Jul 2016 18:27:00 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from msa1.earth.yoonka.com (yoonka.com [185.24.122.233]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "msa1.earth.yoonka.com", Issuer "msa1.earth.yoonka.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 0AC761F2B for ; Thu, 7 Jul 2016 18:26:59 +0000 (UTC) (envelope-from list1@gjunka.com) Received: from crayon2.yoonka.com (crayon2.yoonka.com [10.70.7.20]) (authenticated bits=0) by msa1.earth.yoonka.com (8.15.2/8.15.2) with ESMTPSA id u67IQoDA064384 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO) for ; Thu, 7 Jul 2016 18:26:50 GMT (envelope-from list1@gjunka.com) Subject: Re: Effective rule sets in a jail? To: freebsd-jail@freebsd.org References: <2aeb6798-11ee-27c0-610a-d745aa322f97@gjunka.com> <577E0A78.1040600@quip.cz> <2c9d10fd-35ba-5470-026d-a1483e47fcf2@gjunka.com> <577E1AFB.90100@quip.cz> <6ccead58-a38a-80a4-b5b8-a509c4271b8f@gjunka.com> <577E29A8.5000504@quip.cz> <4d3f5584-7dd1-e6fc-540f-9ed3f1fb63f4@gjunka.com> <20160707153148.7da31609@eto-mona.office.smartweb.sk> From: Grzegorz Junka Message-ID: <8d24b89f-3a83-ac6b-5051-c50accb24e9d@gjunka.com> Date: Thu, 7 Jul 2016 18:26:50 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: <20160707153148.7da31609@eto-mona.office.smartweb.sk> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Jul 2016 18:27:00 -0000 On 07/07/2016 13:31, Martin "eto" Misuth wrote: > IMHO, regarding jails, better mental model would be like this: > - any single jail can have one and only one devfs ruleset number assigned > - however, different standalone jails can have different devfs ruleset > number assigned > - nested jails inherit ruleset number from their parent jail > > Regarding rulesets "inheritance"/"merging" you are probably looking into the > wrong place. devfs ruleset system is completely orthogonal to jails, as it > is used for other things as well. > > You can "merge" devfs rulesets in devfs /etc/devfs.rules. > > Look into /etc/defaults/devfs.rules how initial rulesets are built. > > First everything is hidden by ruleset 1 aka "devfsrules_hide_all". This is "by > default deny" policy, which should, according to me, used whenever one can. > > Then, new rulesets are created by unhiding various groups of devices. > Like for example you have minimal sub-ruleset 2 aka "devfsrules_unhide_basic". > That one is required to get minimal working /dev. Otherwise most programs break. > > Finally ruleset 4 aka "devfsrules_jail" is built, which can be used by jails. > > I personally "classify" jail types into groups. Let's call such group a jail > "class" (for the purpose of classification). > > Thus to get what you want, you should create custom ruleset per jail "class" and > assign it to your jails based on their "class". > > [devfsrules_jail_class_no_zfs=16] > add include $devfsrules_hide_all > add include $devfsrules_unhide_basic > add include $devfsrules_unhide_login > > Class might be not good word for this, as it is quite "loaded" by now, but I am > using it that way. > > Some jails might end up so special, they require completely fine tuned ruleset. > Those cannot be completely "classified" at all like this for example: > > [devfsrules_jail_proxy=333] > add include $devfsrules_hide_all > add include $devfsrules_unhide_basic > add include $devfsrules_unhide_login > add include $devfsrules_unhide_jail_proxy_tuns > > "devfsrules_unhide_jail_proxy_tuns" sub-rule in this case unhides > several tun interfaces used solely by this jail only. > > devfs.conf files are "parsed" by /etc/rc.d/devfs rc script which is run quite > early after boot. If you look at it you will see it is using /etc/rc.subr > devfs_* subroutines of rc.d framework which invoke /sbin/devfs helper program. > > Theoretically if /etc/rc.d/devfs and /etc/rc.subr are not enough for > you, you could write helper script to invoke /sbin/devfs to setup most > convoluted rule ids directly by hand. > > eto > _______________________________________________ > freebsd-jail@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-jail > To unsubscribe, send any mail to "freebsd-jail-unsubscribe@freebsd.org" That's great! Thanks for the comprehensive explanation, I wish it was in the man already so I wouldn't need to enquiry additionally here. It makes sense, as I mentioned in my previous email, I got confused and messed jail inheritance with the inheritance of devfs rule sets, they are orthogonal as you stated. I amended my rules to include the basic ones from rule 4 to the more specific one for one particular jail and it works. Thanks again! Grzegorz From owner-freebsd-jail@freebsd.org Fri Jul 8 18:28:55 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 586FFB85B9D for ; Fri, 8 Jul 2016 18:28:55 +0000 (UTC) (envelope-from tommyj27@gmail.com) Received: from mail-vk0-x22a.google.com (mail-vk0-x22a.google.com [IPv6:2607:f8b0:400c:c05::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1508E1744 for ; Fri, 8 Jul 2016 18:28:55 +0000 (UTC) (envelope-from tommyj27@gmail.com) Received: by mail-vk0-x22a.google.com with SMTP id v6so65154868vkb.2 for ; Fri, 08 Jul 2016 11:28:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=RUaLTzswdADaJlJvtoai3ksxkQWFKe1tXTg1tLJ36a8=; b=shgWxQWCSFP38Djlp11KReuVANggaa9rRT+qro7f8A5A6HbicPpuhyncN8gRUbyjfU 5sGOUwYOqwH/bvRrc/mLgos2IqevN8nueZa9JW6h3Og5rDhhyqurZ+4o5TqYk9fQeoEH onO/5hP7Ynozn2GOjoDFcZMRcjXY6iFo/gsHXKMcUuaBClWXnVLugkyahRwrawJ28TY4 T6eicFFmJg08429vmfUfAC4HaDUp2F61+1bM8fnA3Q32tXv5l3sTU3OGEkjXl3cewo2d u8AKcpSVA1kQv9tnVBAz4KSgN0v5y9cqwXUoSx2RGDv4vUpRSVTbzidaudahtHuFnzoV eNyA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=RUaLTzswdADaJlJvtoai3ksxkQWFKe1tXTg1tLJ36a8=; b=jMyNxE78EjScGMSAELfMtEB1yOpoRh+Vsoi5hOfIpLNg/ADLzlOByXFgLY5vqgY0QS 9x9WsDjrnaNuhdm/ve2A6FH73tQWqI2O1wxVqLzm1kCmoR7ic09wo49i2wgkM0QbStCK 7GT0cdnbvHn4cCgaVtCVo95Ecg0W5yzb7jCB35SsyFnbjra3vzuYWLOoqh6e2lUgug3X Y1au1CLferi/kDnBVl6hVEjsWwMetqnGKpLSYkLbDI3m7zNg1RIFutJ7dFkVB++03PXf IFS7H4F/j54of5VzYCG2nD9zOR8Shzd88BVD4a3t02lOy+6YbuutIwrQz8lHM9kshlyk 9dfw== X-Gm-Message-State: ALyK8tLx2iDCWp1L4H3U4mCvfDyQxUvk21R2z9EWlI0pZs+rZu6Ia0q4SRs7b4wU40CfIoL8QOOZ72wALlgyRA== X-Received: by 10.159.41.164 with SMTP id s33mr3324906uas.146.1468002533977; Fri, 08 Jul 2016 11:28:53 -0700 (PDT) MIME-Version: 1.0 Received: by 10.159.33.136 with HTTP; Fri, 8 Jul 2016 11:28:53 -0700 (PDT) From: Thomas Johnson Date: Fri, 8 Jul 2016 13:28:53 -0500 Message-ID: Subject: NFS + nullfs + jail = zombies? To: freebsd-jail@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 18:28:55 -0000 I am working on developing a clustered application utilizing jails and running into problems that seem to be NFS-related. I'm hoping that someone can point out my error. The jail images and my application data are served via NFS. The host mounts NFS at boot, and then uses nullfs mounts to assemble the jail tree when the jail is created (fstab files and jail.conf are below). This seems to work fine, the jail starts and is usable. The problem comes when I remove/restart the jail. Frequently (but not consistently), the jail gets stuck in a dying state, causing the unmount of the jail root (nullfs) to fail with a "device busy" error. # jail -f /var/local/jail.conf -r wds1-1a Stopping cron. Waiting for PIDS: 1361. . Terminated wds1-1a: removed umount: unmount of /var/jail/wds1-1a failed: Device busy # jls -av JID Hostname Path Name State CPUSetID IP Address(es) 1 wds1-1a /var/jail/wds1-1a wds1-1a DYING 2 2620:1:1:1:1a::1 Through trial-and-error I have determined that forcing an unmount of the root works, but subsequent mounts to that mount point will fail to unmount with the same error. Deleting and recreating the mountpoint fixes the mounting issue, but the dying jail remains permanently. I have also found that if I copy the jail root to local storage and update the jail's fstab to nullfs mount this, the problem seems to go away. This leads me to believe that the issue is related to the NFS source for the nullfs mount. statd and lockd are both running on the host. My relevant configurations are below. I can provide any other information desired. # Host fstab line for jail root. # 10.219.212.1:/vol/dev/wds/jail_base /jail/base nfs ro 0 0 # Jail fstab file (mount.fstab) # /jail/base /var/jail/wds1-1a nullfs ro 0 0 # writable (UFS-backed) /var /var/jail-vars/wds1-1a /var/jail/wds1-1a/var nullfs rw 0 0 # jail.conf file # * { devfs_ruleset = "4"; mount.devfs; exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; interface = "vmx1"; allow.dying = 1; exec.prestart = "/usr/local/bin/rsync -avC --delete /jail/${image}/var/ /var/jail-vars/${host.hostname}/"; } # JMANAGE wds1-1a wds1-1a { path = "/var/jail/wds1-1a"; ip6.addr = "2620:1:1:1:1a::1"; host.hostname = "wds1-1a"; host.domainname = "dev"; mount.fstab = "/var/local/fstab.wds1-1a"; $image = "base"; } From owner-freebsd-jail@freebsd.org Fri Jul 8 21:07:30 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E44D5B858AE for ; Fri, 8 Jul 2016 21:07:30 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-it0-x229.google.com (mail-it0-x229.google.com [IPv6:2607:f8b0:4001:c0b::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id AC58719C3 for ; Fri, 8 Jul 2016 21:07:30 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-it0-x229.google.com with SMTP id h190so19286989ith.1 for ; Fri, 08 Jul 2016 14:07:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=PNA7aC7MBpWej+rTAtpn9mPYY+hWCrZS2ZXk4Oj2gqA=; b=Ycd0hldEqvDt9bOzq9nnrHxZ+EEoI9Y1jO6vPlBxumTZbPJ7BUVKhb2t66ADuC8/Ro mKDD/vqktCtOxDnpGww5je4zJxoxCPEqv9dNnwvXBGlFENdeR/Ej77emfq3TI2PCTyTL R1U6IpzieBBQ983AIzcFM+hAovq4wp+ESk6YqUT9Bi6r98HGNCavHd+6rZAJNPSAk/T2 csTa90sohbODx2a16sZpPND38/X7+uzyy+J3msX5UjkHPJ3KH3BLpgEMN5EXxpCgecBY w2Yck0Ocd3cjNiGH4uvodtalJNNWxiz5/BlVAMrEWuhfIi6G/Hd5rfJ2RNG/NcgpRbue xvPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=PNA7aC7MBpWej+rTAtpn9mPYY+hWCrZS2ZXk4Oj2gqA=; b=jr2bk9+gkXDdEpLU3rO/tT+KcaR3Xe8ovX0M7YY5WMUfnnUDII0zDZ+WuvMFcn5v+j cw8szxhL0oHbatcQ/af18N4zi8xXP78CAHrzfe6rfsBYd0BLIoz3ukFruOtBUxcDI63D wI+Al44l9GLRDUvapm5Bw6+Hua3AQvXBg3/KsN1LLesVcnDlA/IG+p0WOzcS3rER/WEx DaBDijVY330Llo+mEuo6jhIxnVEbvNuCJYtx3qeDbtqyF0xXdWymaejCqWr/MkPmRxdi /o5lrCQ+rPEdbX7Adyv3pnbjiqTJRLh4iVth/OBnQlFKVvcEIgojZf0hK8fgSU488rVV nLzg== X-Gm-Message-State: ALyK8tLBxDwahnbGIv/RjjOu6huDZxCh4MP+pYbhnUv4JfLDtazzUT7J+avmCu+0t/Mkzw== X-Received: by 10.36.117.200 with SMTP id y191mr526187itc.35.1468012050040; Fri, 08 Jul 2016 14:07:30 -0700 (PDT) Received: from [10.0.10.3] (cpe-184-56-210-236.neo.res.rr.com. [184.56.210.236]) by smtp.googlemail.com with ESMTPSA id v129sm1998920itg.10.2016.07.08.14.07.29 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Fri, 08 Jul 2016 14:07:29 -0700 (PDT) Message-ID: <57801614.3000205@gmail.com> Date: Fri, 08 Jul 2016 17:07:32 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Thomas Johnson CC: freebsd-jail@freebsd.org Subject: Re: NFS + nullfs + jail = zombies? References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 Jul 2016 21:07:31 -0000 Thomas Johnson wrote: > I am working on developing a clustered application utilizing jails and > running into problems that seem to be NFS-related. I'm hoping that > someone can point out my error. > > The jail images and my application data are served via NFS. The host > mounts NFS at boot, and then uses nullfs mounts to assemble the jail > tree when the jail is created (fstab files and jail.conf are below). > This seems to work fine, the jail starts and is usable. The problem > comes when I remove/restart the jail. Frequently (but not > consistently), the jail gets stuck in a dying state, causing the > unmount of the jail root (nullfs) to fail with a "device busy" error. > > # jail -f /var/local/jail.conf -r wds1-1a > Stopping cron. > Waiting for PIDS: 1361. > . > Terminated > wds1-1a: removed > umount: unmount of /var/jail/wds1-1a failed: Device busy > # jls -av > JID Hostname Path > Name State > CPUSetID > IP Address(es) > 1 wds1-1a /var/jail/wds1-1a > wds1-1a DYING > 2 > 2620:1:1:1:1a::1 > > Through trial-and-error I have determined that forcing an unmount of > the root works, but subsequent mounts to that mount point will fail to > unmount with the same error. Deleting and recreating the mountpoint > fixes the mounting issue, but the dying jail remains permanently. > > I have also found that if I copy the jail root to local storage and > update the jail's fstab to nullfs mount this, the problem seems to go > away. This leads me to believe that the issue is related to the NFS > source for the nullfs mount. statd and lockd are both running on the > host. > > My relevant configurations are below. I can provide any other > information desired. > > # Host fstab line for jail root. > # > 10.219.212.1:/vol/dev/wds/jail_base /jail/base nfs ro 0 0 > > > # Jail fstab file (mount.fstab) > # > /jail/base /var/jail/wds1-1a nullfs ro 0 0 > # writable (UFS-backed) /var > /var/jail-vars/wds1-1a /var/jail/wds1-1a/var nullfs rw 0 0 > > > # jail.conf file > # > * { > devfs_ruleset = "4"; > mount.devfs; > exec.start = "/bin/sh /etc/rc"; > exec.stop = "/bin/sh /etc/rc.shutdown"; > interface = "vmx1"; > allow.dying = 1; > exec.prestart = "/usr/local/bin/rsync -avC --delete > /jail/${image}/var/ /var/jail-vars/${host.hostname}/"; > } > > # JMANAGE wds1-1a > wds1-1a { > path = "/var/jail/wds1-1a"; > ip6.addr = "2620:1:1:1:1a::1"; > host.hostname = "wds1-1a"; > host.domainname = "dev"; > mount.fstab = "/var/local/fstab.wds1-1a"; > $image = "base"; > } If I remember correctly from past posts about NFS and jails, NFS does not play well with jails. The jails directory tree has to be on the host running the jail. Remove NFS from the picture and it should just work. Your beating a dead horse. From owner-freebsd-jail@freebsd.org Sat Jul 9 14:23:08 2016 Return-Path: Delivered-To: freebsd-jail@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8EEEBB8367B for ; Sat, 9 Jul 2016 14:23:08 +0000 (UTC) (envelope-from jamie@gritton.org) Received: from gritton.org (gritton.org [162.220.209.3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "www.gritton.org", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 778AE18A5 for ; Sat, 9 Jul 2016 14:23:08 +0000 (UTC) (envelope-from jamie@gritton.org) Received: from gritton.org (gritton.org [162.220.209.3]) by gritton.org (8.15.2/8.15.2) with ESMTPS id u69EMPDx005353 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 9 Jul 2016 08:22:25 -0600 (MDT) (envelope-from jamie@gritton.org) Received: (from www@localhost) by gritton.org (8.15.2/8.15.2/Submit) id u69EMPPF005352; Sat, 9 Jul 2016 08:22:25 -0600 (MDT) (envelope-from jamie@gritton.org) X-Authentication-Warning: gritton.org: www set sender to jamie@gritton.org using -f To: freebsd-jail@freebsd.org Subject: Re: NFS + nullfs + jail = zombies? X-PHP-Originating-Script: 0:rcube.php MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 09 Jul 2016 08:22:25 -0600 From: James Gritton In-Reply-To: References: Message-ID: <596319b8dce811ea6e332c48d3542451@gritton.org> X-Sender: jamie@gritton.org User-Agent: Roundcube Webmail/1.2.0 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Jul 2016 14:23:08 -0000 On 2016-07-08 12:28, Thomas Johnson wrote: > I am working on developing a clustered application utilizing jails and > running into problems that seem to be NFS-related. I'm hoping that > someone can point out my error. > > The jail images and my application data are served via NFS. The host > mounts NFS at boot, and then uses nullfs mounts to assemble the jail > tree when the jail is created (fstab files and jail.conf are below). > This seems to work fine, the jail starts and is usable. The problem > comes when I remove/restart the jail. Frequently (but not > consistently), the jail gets stuck in a dying state, causing the > unmount of the jail root (nullfs) to fail with a "device busy" error. > > # jail -f /var/local/jail.conf -r wds1-1a > Stopping cron. > Waiting for PIDS: 1361. > . > Terminated > wds1-1a: removed > umount: unmount of /var/jail/wds1-1a failed: Device busy > # jls -av > JID Hostname Path > Name State > CPUSetID > IP Address(es) > 1 wds1-1a /var/jail/wds1-1a > wds1-1a DYING > 2 > 2620:1:1:1:1a::1 > > Through trial-and-error I have determined that forcing an unmount of > the root works, but subsequent mounts to that mount point will fail to > unmount with the same error. Deleting and recreating the mountpoint > fixes the mounting issue, but the dying jail remains permanently. > > I have also found that if I copy the jail root to local storage and > update the jail's fstab to nullfs mount this, the problem seems to go > away. This leads me to believe that the issue is related to the NFS > source for the nullfs mount. statd and lockd are both running on the > host. > > My relevant configurations are below. I can provide any other > information desired. > > # Host fstab line for jail root. > # > 10.219.212.1:/vol/dev/wds/jail_base /jail/base nfs ro 0 0 > > > # Jail fstab file (mount.fstab) > # > /jail/base /var/jail/wds1-1a nullfs ro 0 0 > # writable (UFS-backed) /var > /var/jail-vars/wds1-1a /var/jail/wds1-1a/var nullfs rw 0 0 > > > # jail.conf file > # > * { > devfs_ruleset = "4"; > mount.devfs; > exec.start = "/bin/sh /etc/rc"; > exec.stop = "/bin/sh /etc/rc.shutdown"; > interface = "vmx1"; > allow.dying = 1; > exec.prestart = "/usr/local/bin/rsync -avC --delete > /jail/${image}/var/ /var/jail-vars/${host.hostname}/"; > } > > # JMANAGE wds1-1a > wds1-1a { > path = "/var/jail/wds1-1a"; > ip6.addr = "2620:1:1:1:1a::1"; > host.hostname = "wds1-1a"; > host.domainname = "dev"; > mount.fstab = "/var/local/fstab.wds1-1a"; > $image = "base"; > } What happens if you take jails out of the equation? I know this isn't entirely a non-jail issue, but I wonder if a jail is required for the mount point to be un-re-mountable. I've heard before of NFS-related problems where a jail remains dying forever, but this has been more of an annoyance than a real problem. It's not so much that I want to absolve jails, as I want to see where the main fight exists. It's tricky enough fixing an interface between two systems, but we've got three here. - Jamie