From owner-freebsd-jail@FreeBSD.ORG Mon Mar 30 15:52:14 2015 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5F7D22E5 for ; Mon, 30 Mar 2015 15:52:14 +0000 (UTC) Received: from internal.electricembers.net (internal.electricembers.net [208.90.215.71]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.electricembers.net", Issuer "DigiCert High Assurance CA-3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 413A369D for ; Mon, 30 Mar 2015 15:52:13 +0000 (UTC) Received: from [192.168.1.4] (cpe-66-66-190-62.rochester.res.rr.com [66.66.190.62]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: ben) by internal.electricembers.net (Postfix) with ESMTPSA id BBB6F27333 for ; Mon, 30 Mar 2015 08:42:12 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 8.2 \(2070.6\)) Subject: Re: ftasv and ScoreBoardFile on FreeBSD 10 with jails From: Benjamin Connelly In-Reply-To: <88a082c0bbf3a1bae7e5a6864f73884d@gritton.org> Date: Mon, 30 Mar 2015 08:42:11 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <16C8A490-722C-464D-AFA5-6E3CED4B2EDD@electricembers.coop> References: <88a082c0bbf3a1bae7e5a6864f73884d@gritton.org> To: freebsd-jail@freebsd.org X-Mailer: Apple Mail (2.2070.6) X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Mar 2015 15:52:14 -0000 >> We recently upgraded some FreeBSD 9.1 servers to FreeBSD 10.1 and >> found it broke the scoreboard viewing utility we were using, the >> "ftasv" port (ftss). >> For that tool to work apache is supposed to be configured to use 'a >> "name based" shared memory segment' (from their README) by the >> directive >> ScoreBoardFile /var/run/apache_status >> That used to (on FreeBSD 9.1) create that "file". Then we could >> execute 'ftasv /var/run/apache_status' to interpret it and see what >> requests apache was working to serve. >> This even worked with many different apache instances running each in >> their own jail, where all the jails actually share the same basejail >> /usr/local/sbin/httpd binary. Inside each jail we could see just the >> requests that instance of apache was working on. >> But after the FreeBSD upgrade to 10.1 we no longer see the >> apache_status file in the filesystem, and ftasv seems to actually >> report the most recent hits from the most recently restarted instance >> of apache, even if that's in another jail!? (On a system with no = jails >> and just the one instance of apache, it's not actually a problem!) >> Can anybody point me toward the right dials to turn if it's still >> possible to do this scoreboard viewing of each independent apache >> instance? (Like I think I may need = security.jail.param.allow.sysvipc=3D1 >> in the jails, but I'm also finding with ezjail I'm not actually able >> to get that set because it's creating the /var/run/jail.JAILNAME.conf >> file with both these lines in it: >> allow.sysvipc =3D 0; >> allow.sysvipc=3D1; >> Ben >=20 > You definitely don't want to try setting anything under = security.jail.param.* - those are just informational, used by jail(8) to = know the identities and formats of the currently available parameters. = One of the two lines that is ending up in /var/run/jail.JAILNAME.conf is = correct, though it's not immediately obvious which one. >=20 > ftss claims you need name-based shared memory, i.e. memory-mapped = files. This has nothing to do with SYSV-style shared memory, except = that it's the modern (i.e. right) way to do shared memory and SYSV IPC = is the old (i.e. wrong) way. So that would make me think it doesn't = matter what you do with allow.sysvipc. Maybe ftss first tries SYSV, and = if that works it goes with that, and if it doesn't then it tries the = memory-mapped file (which isn't what it says it does, but that's neither = here nor there). Jails that allow SYSV IPC don't segregate it into = per-jail namespaces, which is IMHO a bug and which would explain it = seeing some other jail's status. Memory-mapped files on the other hand = depend on the file being the same (and not just the same name), so a = typical jail will not be able to share another jail's memory-mapped = files because it can't see another jail's filesystem namespace. >=20 > This is making me think you want allow.syscipc=3D0. I'm not sure how = you would set that in ezjail, but I would assume it's ... well ... easy. >=20 > - Jamie Well I have heard back from the developers of ftss that it uses = =E2=80=9Cwhatever apr uses=E2=80=9D for shared memory.=20 It sure seems like something changed here and that jails should each = have their own apache scoreboard shared memory and not be able to see = the others, like it was in the FreeBSD 9=E2=80=99s. . .=20 Does anybody know anything about this? Benjamin= From owner-freebsd-jail@FreeBSD.ORG Thu Apr 2 22:39:31 2015 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5FF2BFA1 for ; Thu, 2 Apr 2015 22:39:31 +0000 (UTC) Received: from mail-lb0-x234.google.com (mail-lb0-x234.google.com [IPv6:2a00:1450:4010:c04::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D5A8082 for ; Thu, 2 Apr 2015 22:39:30 +0000 (UTC) Received: by lboc7 with SMTP id c7so69098431lbo.1 for ; Thu, 02 Apr 2015 15:39:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=hYpPt/m2LQCKv0h2oOUCHNsx9UZtsORS/3Cf05mZ218=; b=ky0bzJUYRTIG43Q3ZtJWqLVh1OPtd7RkgWBIxeQNcGMwu+nFalNkf9RAPnsl8mjFMJ LOBOCxllzw2SDoMYflrpN/AuSMEsI2H/NUFAiC3s7Q7wNDkPkY4fN1rx5vGMUW3mk0Ai KdcNLM9Mq+0S7kHzz81FdFATntWY1tMXTDdiw/L5q2DuhhbwC7XNs9hJeIFljXQE2YIP y5KPJa19QuwsB5yli5KDAZENxUleZOXCh5LvO4wYctJREd2k2xpOBoT9vkvqGTn5July mt6DTFvbhcZaUyX/e/Q3u0bq2cxGE90zYyFdXQZv6si42T1QEFJjowUbPmxu/D8RuFXk 2Tvw== MIME-Version: 1.0 X-Received: by 10.152.170.199 with SMTP id ao7mr41957699lac.27.1428014368634; Thu, 02 Apr 2015 15:39:28 -0700 (PDT) Received: by 10.114.24.72 with HTTP; Thu, 2 Apr 2015 15:39:28 -0700 (PDT) In-Reply-To: References: Date: Thu, 2 Apr 2015 15:39:28 -0700 Message-ID: Subject: Fwd: What to do about RPC errors in an old 8.3 jail From: javocado To: freebsd-jail@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2015 22:39:31 -0000 We have an older jail running on 8.3-RELEASE, and when we attempt to run a certain linux binary, it successfully runs, but bombs out with RPC errors: [03.03.2015 21:03:56] < 49156> cli| Thread started. Thread id: 49156, parent id: 16384, role: VRPC server thread [03.03.2015 21:03:56] < 49156> net| Veeam RPC server started. [03.03.2015 21:03:56] < 49156> net| Selected vRPC port: '2500'. [03.03.2015 21:03:56] < 49156> net| Listening vRPC port '2500'. [03.03.2015 21:03:56] < 16384> cli| Client works in standalone mode.[03.03.2015 21:03:57] < 49156> net| ERR |Veeam RPC server broken. [03.03.2015 21:03:57] < 49156> net| >> |WIN: Unable to update socket keep-alive settings. Error code: [92]. [03.03.2015 21:03:57] < 49156> net| >> |An exception was thrown from thread [49156]. [03.03.2015 21:03:57] < 49156> cli| Thread finished. Role: 'VRPC server thread'. As you can see, an RPC server is started, and is immediately broken with the error: [03.03.2015 21:03:57] < 49156> net| ERR |Veeam RPC server broken. We think this is an RPC problem and wonder if it is possible to solve it, perhaps with entries in the jails /etc/rc.conf ? Currently we have no rpc or lockd/statd entries in the jails rc.conf. When you see rpc errors in a jail, what does it mean, and how can it be fixed ? Thank you. From owner-freebsd-jail@FreeBSD.ORG Fri Apr 3 00:28:52 2015 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B9109D04 for ; Fri, 3 Apr 2015 00:28:52 +0000 (UTC) Received: from m2.gritton.org (gritton.org [45.56.208.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6713BE0E for ; Fri, 3 Apr 2015 00:28:51 +0000 (UTC) Received: from m2.gritton.org (gritton.org [45.56.208.41]) by m2.gritton.org (8.14.9/8.14.9) with ESMTP id t330MRvr008661; Thu, 2 Apr 2015 18:22:27 -0600 (MDT) (envelope-from jamie@freebsd.org) Received: (from www@localhost) by m2.gritton.org (8.14.9/8.14.9/Submit) id t330MR80008660; Thu, 2 Apr 2015 18:22:27 -0600 (MDT) (envelope-from jamie@freebsd.org) X-Authentication-Warning: gritton.org: www set sender to jamie@freebsd.org using -f To: freebsd-jail@freebsd.org Subject: Re: Fwd: What to do about RPC errors in an old 8.3 jail X-PHP-Originating-Script: 0:rcube.php MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 02 Apr 2015 18:22:27 -0600 From: James Gritton In-Reply-To: References: Message-ID: X-Sender: jamie@freebsd.org User-Agent: Roundcube Webmail/1.1.0 Cc: javocado X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2015 00:28:52 -0000 On 2015-04-02 16:39, javocado wrote: > We have an older jail running on 8.3-RELEASE, and when we attempt to > run a > certain linux binary, it successfully runs, but bombs out with RPC > errors: > > > [03.03.2015 21:03:56] < 49156> cli| Thread started. Thread id: 49156, > parent id: 16384, role: VRPC server thread > [03.03.2015 21:03:56] < 49156> net| Veeam RPC server started. > [03.03.2015 21:03:56] < 49156> net| Selected vRPC port: '2500'. > [03.03.2015 21:03:56] < 49156> net| Listening vRPC port '2500'. > [03.03.2015 21:03:56] < 16384> cli| Client works in standalone > mode.[03.03.2015 21:03:57] < 49156> net| ERR |Veeam RPC server broken. > [03.03.2015 21:03:57] < 49156> net| >> |WIN: Unable to update socket > keep-alive settings. Error code: [92]. > [03.03.2015 21:03:57] < 49156> net| >> |An exception was thrown from > thread [49156]. > [03.03.2015 21:03:57] < 49156> cli| Thread finished. Role: 'VRPC server > thread'. > > > As you can see, an RPC server is started, and is immediately broken > with > the error: > [03.03.2015 21:03:57] < 49156> net| ERR |Veeam RPC server broken. > > We think this is an RPC problem and wonder if it is possible to solve > it, > perhaps with entries in the jails /etc/rc.conf ? Currently we have no > rpc > or lockd/statd entries in the jails rc.conf. > > When you see rpc errors in a jail, what does it mean, and how can it be > fixed ? > > Thank you. The error code 92 may be the key. errno(2) tells me it's EPROTO ("A device or socket encountered an unrecoverable protocol error."). It's possible that updating the socket keepalive settings requires some of the socket permissions that jails don't always have. FreeBSD 8 ... so I suppose that's old-style jails. It might do to set one of these sysctls: security.jail.param.allow.raw_sockets=1 security.jail.socket_unixiproute_only=0 Or maybe you're actually using jail parameters? I think those went in 8 (but I'm not sure). If so, and if you're actually using a jail.conf, try "allow.socket_af" and/or "allow.raw_sockets". One of those might help. Or neither. But it's at least worth a try. - Jamie