From owner-freebsd-virtualization@freebsd.org Mon Nov 27 16:28:57 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 484A6DB899B for ; Mon, 27 Nov 2017 16:28:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 35ADE79DA3 for ; Mon, 27 Nov 2017 16:28:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vARGSuLg003103 for ; Mon, 27 Nov 2017 16:28:57 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 184046] bhyve(4) manpage references non-existant manpages bhyvectl(8), vmm(4) Date: Mon, 27 Nov 2017 16:28:57 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Documentation X-Bugzilla-Component: Documentation X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: commit-hook@freebsd.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Nov 2017 16:28:57 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D184046 --- Comment #3 from commit-hook@freebsd.org --- A commit references this bug: Author: sbruno Date: Mon Nov 27 16:28:28 UTC 2017 New revision: 326281 URL: https://svnweb.freebsd.org/changeset/base/326281 Log: Add vmm(4) man page PR: 205705 184046 Submitted by: Matt Macy Reviewed by: grehan Differential Revision: https://reviews.freebsd.org/D13062 Changes: head/share/man/man4/vmm.4 --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Mon Nov 27 19:30:34 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 964EDDE3A37 for ; Mon, 27 Nov 2017 19:30:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 83DC4182B for ; Mon, 27 Nov 2017 19:30:34 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vARJUXRu087769 for ; Mon, 27 Nov 2017 19:30:34 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 184046] bhyve(4) manpage references non-existant manpages bhyvectl(8), vmm(4) Date: Mon, 27 Nov 2017 19:30:33 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Documentation X-Bugzilla-Component: Documentation X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: sbruno@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: cc Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Nov 2017 19:30:34 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D184046 Sean Bruno changed: What |Removed |Added ---------------------------------------------------------------------------- CC| |sbruno@FreeBSD.org --- Comment #4 from Sean Bruno --- man bhyvectl(8) exists in -current, it appears to be in usr.sbin/bhyvectl/bhyvectl8 and is installed. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Tue Nov 28 19:30:32 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CAE20DEC253 for ; Tue, 28 Nov 2017 19:30:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2001:1900:2254:206a::16:76]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B83C574B70 for ; Tue, 28 Nov 2017 19:30:32 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from bugs.freebsd.org ([127.0.1.118]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id vASJUWqV014072 for ; Tue, 28 Nov 2017 19:30:32 GMT (envelope-from bugzilla-noreply@freebsd.org) From: bugzilla-noreply@freebsd.org To: freebsd-virtualization@FreeBSD.org Subject: [Bug 223531] Instant kernel panic on 'kldload virtio_console' on oVirt 3.5 VM Date: Tue, 28 Nov 2017 19:30:32 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: CURRENT X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Many People X-Bugzilla-Who: bryanv@FreeBSD.org X-Bugzilla-Status: In Progress X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: freebsd-virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Nov 2017 19:30:32 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D223531 --- Comment #18 from Bryan Venteicher --- I'll try to look into this in the next week if @jceel does not. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Wed Nov 29 01:54:24 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1058FDF5C52 for ; Wed, 29 Nov 2017 01:54:24 +0000 (UTC) (envelope-from danieljewell@gmail.com) Received: from mail-lf0-x234.google.com (mail-lf0-x234.google.com [IPv6:2a00:1450:4010:c07::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8B75015ED for ; Wed, 29 Nov 2017 01:54:23 +0000 (UTC) (envelope-from danieljewell@gmail.com) Received: by mail-lf0-x234.google.com with SMTP id 94so2034827lfy.10 for ; Tue, 28 Nov 2017 17:54:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=xCDCN6PoCqVucoKOGzkG+JQ27PtLTBg83Z1/jArS0I0=; b=YXiY2K2NtdeyV1DauQ/lQ064Bfoe661kFvyCBW2PEUoMsRAD7HhTweUxC6ZeIe4nLq QNu0N1mZW90bDmSJ8YMl2F+i2Q33ORMYFtlUs+iyslOdXX1XdGYrygBmrhyulWeUHYa4 99P28KfyJKpjVBXtxnMmAL4yaieyxvgPQnQG74RnGx+CTOoPaeviR/tOVVfppcJDMI4W 9HQHJRoERC5ygOejjGH/NTted5ux2A7uky5S+A05Q4pmJLqGbZ2NhHNRPLfiyrYRuSRi nEVcZGBXYq9CorAVuISfKOe2Nf006+Aj/sFNdhS4F7x5yBgJPDQz/3J3FKRCOG5jhQjR yfDw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=xCDCN6PoCqVucoKOGzkG+JQ27PtLTBg83Z1/jArS0I0=; b=I1ht41tC07b57ovck2BW1WtuXb3vnYoepv1/54x6+EgAEztJanl7KnkAim+HSUuNf1 Hlk7zm2bjra1M6hp3A2zpdgR7jtSuNf2egMxsCp5A2C8ztgARiqTZC7pMTujIsYTuvoT Q04keCicveKBv6JHYvswlHPNmHzAjEBPeO1HZ65YhBc3hSiRiU3t/OigWfMBoWrLIqjp Ub/XyNKz77S7WfSlFfq5FDLg2iYyX7OYgYkR/q5M1qS1lIghhm93ae3Vu+VRFLMYizKI 2IKtDGa7n4QVcX3OHUsBRQV9peykZv0e4U47hf1VZHeGOdoYzeUgLqgflnJLWRoFmH/2 XUSg== X-Gm-Message-State: AJaThX5WCEYLMv4wnIjqmM4SlPFz0e06KGBYdvduHNaFNMTbkcS9lxYc 45xtccp6vcRINU/LuqeEOZWaykBPG/wBN7F1u9dS460R X-Google-Smtp-Source: AGs4zMYyYOB8b6yP0aN250gkpla7i63zr8Iz4rz7eULV6WQl2EvxuspNm99QlPl0BF1cplLRDDM20fPNDk+Jc8CuTMk= X-Received: by 10.25.125.195 with SMTP id y186mr395358lfc.137.1511920461045; Tue, 28 Nov 2017 17:54:21 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.20.209 with HTTP; Tue, 28 Nov 2017 17:54:20 -0800 (PST) From: Daniel Jewell Date: Tue, 28 Nov 2017 18:54:20 -0700 Message-ID: Subject: Fresh Install of 11.1-RELEASE-p4 (fully updated) on Hyper-V 2012 Failover Cluster To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Nov 2017 01:54:24 -0000 Hello All, I just did an install of 11.1 and fully updated. Ever since the install, I get a kernel message from the Hyper-V VSS module: hvvss0: Unknown opt from host: 4 The message repeats regularly (30 seconds? 60 seconds?)... The VSS module is loaded at boot and seems to initialize OK: hvvss0: on vmbus0 I did the requisite searches for any possible answers and turned up nothing. I haven't attempted a backup with VSS yet. Environment is running a Dell EqualLogic iSCSI SAN as the cluster storage and the cluster hosts do have the Dell EqualLogic MPIO drivers on and active as well as the Dell HIT kit - which does include a Hardware VSS provider to integrate snapshot functionality on the SAN. However, since I haven't called any backup routines, it shouldn't be affecting this VM instance. I also reviewed the Windows Event Log(s) - nothing stands out that I can see in the Hyper-V logs, Application, or System. Any suggestions? :) Best, -Daniel Jewell From owner-freebsd-virtualization@freebsd.org Wed Nov 29 22:58:33 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D6737DEC318 for ; Wed, 29 Nov 2017 22:58:33 +0000 (UTC) (envelope-from rajil.s@gmail.com) Received: from mail-pl0-x229.google.com (mail-pl0-x229.google.com [IPv6:2607:f8b0:400e:c01::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9E84B681E1 for ; Wed, 29 Nov 2017 22:58:33 +0000 (UTC) (envelope-from rajil.s@gmail.com) Received: by mail-pl0-x229.google.com with SMTP id g2so3012137pli.8 for ; Wed, 29 Nov 2017 14:58:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:from:subject:message-id:date:user-agent:mime-version :content-transfer-encoding:content-language; bh=IksHVpMv6nydnGl9C8jwO3brhViYIPLqE0vlgSFL2u8=; b=NcWLVRvdVpLIhKHoDPqmw/lvzivU60e48AjajZHqY57B3LEnua7RPj0av5c922wSqF 2cfqF10pScIwZ5h++FQ6/tkXElOen0mrQZOPgk12/dRPtWcrR8zJxnOpI+380dYJUHtZ kT+pCOVnDiUPMK2hOCmm59EE2nwftNlUve4/dEM0WNqe+HiAydVJFzPUvCHoFOxNiQPh BLFcldq39UCtnFSBTp+BgSo3JJAWcNLyrtHFJcRRGGdNjDO18WqxHHolM0nJbtRJHAwh s4h8O+bvssWi3t/dFvc2MS/gv//7xOA2YVGK1hb7/oElzWtz4QfusEzhC/FH/OpezqIB Up2g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-transfer-encoding:content-language; bh=IksHVpMv6nydnGl9C8jwO3brhViYIPLqE0vlgSFL2u8=; b=b1xf1CV1pltnb1C/1RdbijTfmYWgzo7FrrUIyx8qQTpO+Oyvu9ZctN9KfVplMYpNVp T2P9cItHf/zw1l/9vLSg1j6kNXxB8Hy2ZvBFMVLv8+WZSUj9mm/CC19BZKXnNxFh4ids L39N+Gqj2VmdIgxOUFzGGJ9SR9s0bp90xZKsabhkMqJW8GrMLMEeIXY82eyLBTqGzb4P YWm0T53lltFPvNgVAbwx9fURrlDLUHkXUluekUW87yiVEr60VqrJUkvOfRaJA2Z2wV3I 7EsXOpMQJdM6V6Y5B7azhwhUuVEOLT45MEoJIZpcX3uGs2g9SbZVPzXBG1tp1JDFj6h5 ATAQ== X-Gm-Message-State: AJaThX5yo4C9hcqNM7p38vvqaSu0rqQxB5k6ZTnGLmuATBjG+mcq6sC/ NrAkizwHvoDHwE8796pZ52BqH4ak X-Google-Smtp-Source: AGs4zMaG0rGbb3cZkvITWegIEjzoEp/J/+pAPQzKsdbuy+W1a8GaoXMdAZgSlGgE08zt7miWk0Is8Q== X-Received: by 10.84.235.65 with SMTP id g1mr455398plt.13.1511996312711; Wed, 29 Nov 2017 14:58:32 -0800 (PST) Received: from ?IPv6:2601:2c2:880:6cc:f151:cc3:5b51:8a24? ([2601:2c2:880:6cc:f151:cc3:5b51:8a24]) by smtp.gmail.com with ESMTPSA id q12sm4144008pgn.56.2017.11.29.14.58.31 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 29 Nov 2017 14:58:32 -0800 (PST) To: freebsd-virtualization From: Rajil Saraswat Subject: Bhyve and Chromecast Message-ID: Date: Wed, 29 Nov 2017 16:58:28 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Content-Language: en-US X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Nov 2017 22:58:33 -0000 Hi, I would like to cast a Youtube video to chromecast sitting on the network. The VM is running Gentoo linux. Unfortunately, google chrome doesnt find the chromecast. Any idea how to make it to work? Thanks From owner-freebsd-virtualization@freebsd.org Wed Nov 29 23:06:07 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A18DFDEC675 for ; Wed, 29 Nov 2017 23:06:07 +0000 (UTC) (envelope-from paul.g.webster@googlemail.com) Received: from mail-yw0-x22c.google.com (mail-yw0-x22c.google.com [IPv6:2607:f8b0:4002:c05::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5D343686A4 for ; Wed, 29 Nov 2017 23:06:07 +0000 (UTC) (envelope-from paul.g.webster@googlemail.com) Received: by mail-yw0-x22c.google.com with SMTP id x199so2022869ywg.5 for ; Wed, 29 Nov 2017 15:06:07 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=fhyyGKB55Hymr8jzFqK0MsEFEclmhXF+2qqwUlpXgGE=; b=l4snG7wDxwvRAVcff51Vo7Ubcgn6/JE+ptRnFFcwiFmFLD7/YqxhKYp0gRUe3jhKYq lYALsg54fT96wuIQCjROQbNH67cwfVICWi0NuNSXD8sUvUdTgcBbBvH0BwdNAj4Dh0Hz HpE9hUeGwljQ/7Jw13EHRLVPdMOK8zbkGS6Qgpi31Uuk3FCkVaoeVD5akDCVYpW5K9xH 6G3Hi8MvRcocoIMFZrh80tSQZ2q2hsdgvLzDQ4tq+9+KbYRSRD12Q4bwo61eeJ+XzuSf 1n51K/EPnoIHThyRiW9AR1SIisU+C3cMbI672JNLecxhZwQ7xLH4wdIRmixvBpwPgfcB gK8A== X-Gm-Message-State: AJaThX4jZLSNgsPU2KZnb8xVQaqAGpSMjeEmz9wb2H0exjor5uMO5f3v hRHKjulJBylvqpPZtAZaIGahYwoOPwzezCpi13hOwA== X-Google-Smtp-Source: AGs4zMbEfnVToFNfog9sECqLJfVDwuA6++OKJL1CE4DcVMlGVFZZx2W9agyMxjYWmdqEsCXlfikcvCd9DYS2DcI6ybY= X-Received: by 10.129.44.69 with SMTP id s66mr336177yws.227.1511996766187; Wed, 29 Nov 2017 15:06:06 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.165.8 with HTTP; Wed, 29 Nov 2017 15:06:05 -0800 (PST) In-Reply-To: References: From: Paul Webster Date: Wed, 29 Nov 2017 23:06:05 +0000 Message-ID: Subject: Re: Fresh Install of 11.1-RELEASE-p4 (fully updated) on Hyper-V 2012 Failover Cluster To: freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Nov 2017 23:06:07 -0000 Not sure if helpful but im running an entire network gateway under Hyper-V/Windows Server 2012 R2, with no IOMMU: FreeBSD gateway.ourlan.net 11.1-STABLE FreeBSD 11.1-STABLE #0 r324941: Tue Oct 24 20:19:02 BST 2017 root@gateway.ourlan.net:/usr/obj/usr/src/sys/GATEWAY amd64 Copyright (c) 1992-2017 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.1-STABLE #0 r324941: Tue Oct 24 20:19:02 BST 2017 root@gateway.ourlan.net:/usr/obj/usr/src/sys/GATEWAY amd64 FreeBSD clang version 5.0.0 (tags/RELEASE_500/final 312559) (based on LLVM 5.0.0svn) VT(vga): text 80x25 Hyper-V Version: 6.3.9600 [SP17] Features=0xc7f PM Features=0x0 [C2] Features3=0x7b2 Timecounter "Hyper-V" frequency 10000000 Hz quality 2000 CPU: Intel(R) Xeon(R) CPU E5430 @ 2.66GHz (2133.42-MHz K8-class CPU) Origin="GenuineIntel" Id=0x10676 Family=0x6 Model=0x17 Stepping=6 Features=0x1f83fbff Features2=0x80082201 AMD Features=0x20100800 AMD Features2=0x1 Hypervisor: Origin = "Microsoft Hv" real memory = 8589934592 (8192 MB) avail memory = 8277839872 (7894 MB) Event timer "LAPIC" quality 100 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 8 core(s) random: unblocking device. ioapic0: Changing APIC ID to 0 ioapic0 irqs 0-23 on motherboard SMP: AP CPU #3 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #2 Launched! random: entropy device external interface kbd1 at kbdmux0 netmap: loaded module module_register_init: MOD_LOAD (vesa, 0xffffffff80fbbb30, 0) error 19 nexus0 vtvga0: on motherboard cryptosoft0: on motherboard acpi0: on motherboard acpi0: Power Button (fixed) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 attimer0: port 0x40-0x43 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 atrtc0: port 0x70-0x71 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 vmbus0: on pcib0 pci0: on pcib0 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 7.1 on pci0 ata0: at channel 0 on atapci0 ata1: at channel 1 on atapci0 pci0: at device 7.3 (no driver attached) vgapci0: mem 0xf8000000-0xfbffffff irq 11 at device 8.0 on pci0 vgapci0: Boot video device atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model IntelliMouse Explorer, device ID 4 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fd0: <1440-KB 3.5" drive> on fdc0 drive 0 vmbus_res0: irq 5,7 on acpi0 orm0: at iomem 0xc0000-0xcbfff on isa0 vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec usb_needs_explore_all: no devclass nvme cam probe device init vmbus0: version 3.0 hvet0: on vmbus0 Event timer "Hyper-V" frequency 10000000 Hz quality 1000 storvsc0: on vmbus0 hvkbd0: on vmbus0 hvheartbeat0: on vmbus0 hvkvp0: on vmbus0 hvshutdown0: on vmbus0 hvtimesync0: on vmbus0 hn0: on vmbus0 da0 at blkvsc0 bus 0 scbus2 target 0 lun 0 da0: Fixed Direct Access SPC-3 SCSI device da0: 300.000MB/s transfers da0: Command Queueing enabled da0: 25600MB (52428800 512 byte sectors) cd0 at ata1 bus 0 scbus1 target 0 lun 0 cd0: Removable CD-ROM SPC-3 SCSI device cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes) cd0: Attempt to query device size failed: NOT READY, Medium not present hn0: Ethernet address: 00:15:5d:00:25:0b hn0: link state changed to UP hn1: on vmbus0 hn1: Ethernet address: 00:15:5d:00:25:0c hn1: link state changed to UP hn2: on vmbus0 hn2: Ethernet address: 00:15:5d:00:25:0d hn2: link state changed to UP hn3: on vmbus0 hn3: Ethernet address: 00:15:5d:00:25:0e hn3: link state changed to UP hn4: on vmbus0 hn4: Ethernet address: 00:15:5d:00:25:0f hn4: link state changed to UP Trying to mount root from zfs:zroot []... bridge0: Ethernet address: 02:98:0d:24:71:00 bridge0: link state changed to UP hn0: promiscuous mode enabled hn1: promiscuous mode enabled hn2: promiscuous mode enabled pflog0: promiscuous mode enabled tun0: link state changed to UP WARNING: attempt to domain_add(netgraph) after domainfinalize() ipfw2 (+ipv6) initialized, divert loadable, nat loadable, default to deny, logging disabled On 29 November 2017 at 01:54, Daniel Jewell wrote: > Hello All, > > I just did an install of 11.1 and fully updated. Ever since the > install, I get a kernel message from the Hyper-V VSS module: > > hvvss0: Unknown opt from host: 4 > > The message repeats regularly (30 seconds? 60 seconds?)... The VSS > module is loaded at boot and seems to initialize OK: > > hvvss0: on vmbus0 > > I did the requisite searches for any possible answers and turned up > nothing. I haven't attempted a backup with VSS yet. > > Environment is running a Dell EqualLogic iSCSI SAN as the cluster > storage and the cluster hosts do have the Dell EqualLogic MPIO drivers > on and active as well as the Dell HIT kit - which does include a > Hardware VSS provider to integrate snapshot functionality on the SAN. > However, since I haven't called any backup routines, it shouldn't be > affecting this VM instance. > > I also reviewed the Windows Event Log(s) - nothing stands out that I > can see in the Hyper-V logs, Application, or System. > > Any suggestions? :) > > Best, > > -Daniel Jewell > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscribe@freebsd.org" > From owner-freebsd-virtualization@freebsd.org Wed Nov 29 23:09:57 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1F60ADEC754 for ; Wed, 29 Nov 2017 23:09:57 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: from mail-it0-x241.google.com (mail-it0-x241.google.com [IPv6:2607:f8b0:4001:c0b::241]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D9710687DC for ; Wed, 29 Nov 2017 23:09:56 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: by mail-it0-x241.google.com with SMTP id u62so6101462ita.2 for ; Wed, 29 Nov 2017 15:09:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=Dsx6D+tXgIn5BZmsLWaCYREPyQZoazMSc/u/xjMnMM8=; b=bxEyzXioVKpFecDfC4vtjqoI1Be6CQo2XSqYVRb/Hl0hfGSQNet9L+oMOO6MI231OK OmSbPGl2jO+8hjhY3RRt7MefrcRHNIYnC3AZPOVQ52SfOcbK/R0MIKQ8cMUgcqyPSoiy WLC6atLCF80DNOpqty7WpJ4Dth78x4SOYX4aHQF4QnzJpypCYumX3iQMTdWEeXhueiG0 spf2cld76bR/kbbr32UGtSlngO1EibFWhI15RSbdycyIdnHYlgmK37tsqoheSo5s97Lm ZhFJLDxqlkzqlZl46bcYo05C3lLbuZL+r8xXXcEKxaHlpxWPoMtTl96d5iiLeOSbK3EZ GBIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=Dsx6D+tXgIn5BZmsLWaCYREPyQZoazMSc/u/xjMnMM8=; b=FD5WU0L2V2GN9w+Tc9MlY9fgez83j2eN32RSRPXqTSoyC9FhvY7CIIWVurhKZlZuqh Yg78bGlzbt/XXPC6rndoL3mIZnLE+l4IZyuqGvwcQwDymk5ZNfhpasvUQfkC6Uhc+1hr snXJ9n8DcHD4q0nOv9hGRENSavz6l2e5mq5TU4fU2USCj/ivg5ZqQW56ZsBKSFfIwSfb pfcmLfaIYyY46jSfxTldOhoA35+M8kcOUO0hs4vpsET8AGWC+8ARN0nhp6dme6rZeXqv LOzcS2Pt67m5pm2EOo566w6NdRmLwYTG4UCB896wuKZxytKtAPc7/aM3LZ+PetOVkGEd rxSg== X-Gm-Message-State: AJaThX4V86CKjPrmfEtOpO4EKaJ21f8EvgyTRz/HtUJmEnUWEff6v2Dz BTaqjWsginGzeQkccfGppftV6IB9SznSkVLqT3E= X-Google-Smtp-Source: AGs4zMZuwYi0uucDwnLfotCZcWXFCfJoo+D7OK0Poz142yymiJL/acpsXzuHqwd2S5JSITSWAbpT8+5kcRFKXrKr/jU= X-Received: by 10.36.175.19 with SMTP id t19mr610290ite.67.1511996996268; Wed, 29 Nov 2017 15:09:56 -0800 (PST) MIME-Version: 1.0 Sender: jtubnor@gmail.com Received: by 10.2.69.213 with HTTP; Wed, 29 Nov 2017 15:09:35 -0800 (PST) In-Reply-To: References: From: Jason Tubnor Date: Thu, 30 Nov 2017 10:09:35 +1100 X-Google-Sender-Auth: -N9c6ZUYENfSq63vhEjiE3N27Lw Message-ID: Subject: Re: Bhyve and Chromecast To: Rajil Saraswat Cc: freebsd-virtualization Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 Nov 2017 23:09:57 -0000 On 30 November 2017 at 09:58, Rajil Saraswat wrote: > Hi, > > I would like to cast a Youtube video to chromecast sitting on the > network. The VM is running Gentoo linux. Unfortunately, google chrome > doesnt find the chromecast. Any idea how to make it to work? > > I don't think this is specific to bhyve. I assume your bhyve instance has a tap onto the same subnet as the chromecast and you can move other TCP/IP traffic around your network from your Gentoo instance? I have seen chromecasts complain and not work correctly when the controlling device and the chromecast aren't on the same frequency network (SSID is the same on the 2.4 and 5Ghz spectrum). From owner-freebsd-virtualization@freebsd.org Thu Nov 30 19:47:45 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 105CAE6A220 for ; Thu, 30 Nov 2017 19:47:45 +0000 (UTC) (envelope-from notification+kxwm5b4x@facebookmail.com) Received: from mx-out.facebook.com (66-220-155-149.outmail.facebook.com [66.220.155.149]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 84D7872077 for ; Thu, 30 Nov 2017 19:47:43 +0000 (UTC) (envelope-from notification+kxwm5b4x@facebookmail.com) Received: from facebook.com (i5m3Mc/p4hg89VOWvoqbivinhQJDXgTQKAzp8O2obt/jCRLisqMyYfdDVpphiPMv 2401:db00:3020:b009:face:0000:0019:0000) by facebook.com with Thrift id 53423c62d60711e7a35b000af79d742e-3fbfa058; Thu, 30 Nov 2017 11:47:40 -0800 X-Facebook: from 2401:db00:3020:9103:face:0:32:0 ([MTI3LjAuMC4x]) by async.twshared0377.22.lla1.facebook.com with HTTP (ZuckMail); Date: Thu, 30 Nov 2017 11:47:40 -0800 To: Najat Lamini Subject: =?UTF-8?B?Vm9pciBsZSBtZXNzYWdl?= =?UTF-8?B?IGRlIFNhbWlhIExhbWkg?= =?UTF-8?B?ZXQgbGVzIGF1dHJlcyBu?= =?UTF-8?B?b3RpZmljYXRpb25zIHF1?= =?UTF-8?B?ZSB2b3VzIGF2ZXogbWFu?= =?UTF-8?B?cXXDqWVz?= X-Priority: 3 X-Mailer: ZuckMail [version 1.00] From: "Facebook" Reply-to: noreply Errors-To: notification+kxwm5b4x@facebookmail.com X-Facebook-Notify: stale_notifications; mailid=55f383cb37d18G45580756G55f3886497feaG32b Feedback-ID: 3171:stale_notifications:Facebook X-FACEBOOK-PRIORITY: 1 X-Auto-Response-Suppress: All Require-Recipient-Valid-Since: freebsd-virtualization@freebsd.org; Thursday, 26 Mar 2009 21:50:33 +0000 Message-ID: <95ab8cf95789336764f9c98b3f2101ef@async.twshared0377.22.lla1.facebook.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Nov 2017 19:47:45 -0000 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Acc=C3=A9dez =C3=A0 Facebook https://www.facebook.com/n/?aref=3D1512071259324394&medium=3Demail&mid=3D5= 5f383cb37d18G45580756G55f3886497feaG32b&bcode=3D2.1512071260.AbzrDG_4sAJcU= xTVlWY&n_m=3Dfreebsd-virtualization%40freebsd.org&lloc=3D2nd_cta Afficher les notifications https://www.facebook.com/n/?notifications&aref=3D1512071259324394&medium= =3Demail&mid=3D55f383cb37d18G45580756G55f3886497feaG32b&bcode=3D2.15120712= 60.AbzrDG_4sAJcUxTVlWY&n_m=3Dfreebsd-virtualization%40freebsd.org&lloc=3D1= st_cta =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Bonjour Najat, Pas mal de choses se sont pass=C3=A9es sur Facebook depuis votre = derni=C3=A8re connexion. Voici quelques notifications que vous auriez pu = rater. "=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02 messages =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A09 invitations d=E2=80=99ajout =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A026 nouvelles notifications" Merci, L=E2=80=99=C3=A9quipe Facebook =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Ce message a =C3=A9t=C3=A9 envoy=C3=A9 =C3=A0 = freebsd-virtualization@freebsd.org. Si vous ne souhaitez plus recevoir ces = messages de la part de Facebook, veuillez suivre le lien ci-dessous pour = annuler votre abonnement. https://www.facebook.com/o.php?k=3DAS1GRSsE5W4kX35x&u=3D1163396950&mid=3D5= 5f383cb37d18G45580756G55f3886497feaG32b Facebook, Inc., Attention: Community Support, 1 Hacker Way, Menlo Park, CA = 94025 From owner-freebsd-virtualization@freebsd.org Thu Nov 30 20:46:02 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 83F91DB9D17 for ; Thu, 30 Nov 2017 20:46:02 +0000 (UTC) (envelope-from update+kxwm5b4x@facebookmail.com) Received: from mx-out.facebook.com (66-220-155-139.outmail.facebook.com [66.220.155.139]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D402E74943 for ; Thu, 30 Nov 2017 20:46:01 +0000 (UTC) (envelope-from update+kxwm5b4x@facebookmail.com) Received: from facebook.com (Z6JzQ9dpNbB7leNWk1rgITgMl8RcEaOfe/N4+KcoFtsm/14nKrZ34HZW4PjU4rVZ 10.224.41.47) by facebook.com with Thrift id 744ef03cd60f11e79e090002c9e0da90-f47fa058; Thu, 30 Nov 2017 12:45:52 -0800 X-Facebook: from 2401:db00:2111:b0ae:face:0:8d:0 ([MTI3LjAuMC4x]) by www.facebook.com with HTTP (ZuckMail); Date: Thu, 30 Nov 2017 12:45:52 -0800 To: Najat Lamini Subject: Nous sommes heureux de vous revoir sur Facebook X-Priority: 3 X-Mailer: ZuckMail [version 1.00] From: "Facebook" Reply-to: noreply Errors-To: update+kxwm5b4x@facebookmail.com X-Facebook-Notify: rewelcome; mailid=55f390ce17b6aG45580756G0G87 Feedback-ID: 3171:rewelcome:Facebook X-FACEBOOK-PRIORITY: 1 X-Auto-Response-Suppress: All Require-Recipient-Valid-Since: freebsd-virtualization@freebsd.org; Thursday, 26 Mar 2009 21:50:33 +0000 Message-ID: <4c79190acb92da7eefc0824c9c4b3afa@www.facebook.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Nov 2017 20:46:02 -0000 Bonjour Najat, Facebook est un bon moyen de garder le contact avec vos amis, surtout si = vous ne les avez pas vus depuis quelque temps. Facebook est un bon moyen de garder le contact avec vos amis, surtout si = vous ne les avez pas vus depuis quelque temps.Retrouvez vos amis =C3=A0 = l=E2=80=99aide de l=E2=80=99outil automatique de recherche d=E2=80=99amis = de Facebook=C2=A0:Retrouver des amis Retrouvez vos amis =C3=A0 l=E2=80=99aide de l=E2=80=99outil automatique de = recherche d=E2=80=99amis de Facebook=C2=A0: Merci, L=E2=80=99=C3=A9quipe Facebook =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Ce message a =C3=A9t=C3=A9 envoy=C3=A9 =C3=A0 = freebsd-virtualization@freebsd.org. Si vous ne souhaitez plus recevoir ces = messages de la part de Facebook, veuillez suivre le lien ci-dessous pour = annuler votre abonnement. https://www.facebook.com/o.php?k=3DAS2nxm8EX3GwQe87&u=3D1163396950&mid=3D5= 5f390ce17b6aG45580756G0G87 Facebook, Inc., Attention: Community Support, 1 Hacker Way, Menlo Park, CA = 94025 From owner-freebsd-virtualization@freebsd.org Thu Nov 30 21:19:39 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6DBE3DBB08E for ; Thu, 30 Nov 2017 21:19:39 +0000 (UTC) (envelope-from security@facebookmail.com) Received: from mx-out.facebook.com (66-220-155-151.outmail.facebook.com [66.220.155.151]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3348C766AA for ; Thu, 30 Nov 2017 21:19:38 +0000 (UTC) (envelope-from security@facebookmail.com) Received: from facebook.com (gUfGRxpjr7TKorYNSdSPIeUZFRuEtq6/gcC7ker6dZhBm3qwAkYxm5mx4gkOUuVt 10.224.57.55) by facebook.com with Thrift id 2a1825a6d61411e791f30002c9e17dd4-e07fa058; Thu, 30 Nov 2017 13:19:35 -0800 X-Facebook: from 2401:db00:2111:b0aa:face:0:8d:0 ([MTI3LjAuMC4x]) by graph.facebook.com with HTTP (ZuckMail); Date: Thu, 30 Nov 2017 13:19:35 -0800 To: Najat Lamini Subject: Najat, revenez sur Facebook en un clic X-Priority: 3 X-Mailer: ZuckMail [version 1.00] From: "Facebook" Reply-to: noreply Errors-To: security@facebookmail.com X-Facebook-Notify: password_reset:first_send; mailid=55f3985789837G45580756G55f39cf0e9b09G178 Feedback-ID: 3171:password_reset:Facebook X-FACEBOOK-PRIORITY: 1 X-Auto-Response-Suppress: All Require-Recipient-Valid-Since: freebsd-virtualization@freebsd.org; Thursday, 26 Mar 2009 21:50:33 +0000 Message-ID: <59c36c6f2f4d2dd8815430f6ed4ad59a@graph.facebook.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Nov 2017 21:19:39 -0000 Bonjour Najat, Bonjour Najat,Il semble que vous ayez des difficult=C3=A9s =C3=A0 vous = connecter =C3=A0 Facebook. Cliquez sur le bouton ci-dessous et nous vous = connecterons.=C2=A0Revenir=C2=A0sur=C2=A0Facebook=C2=A0Si vous = n=E2=80=99avez pas tent=C3=A9 de vous connecter, dites-le nous. Merci, L=E2=80=99=C3=A9quipe Facebook =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Ce message a =C3=A9t=C3=A9 envoy=C3=A9 =C3=A0 = freebsd-virtualization@freebsd.org sur votre demande. Facebook, Inc., Attention: Community Support, 1 Hacker Way, Menlo Park, CA = 94025 From owner-freebsd-virtualization@freebsd.org Thu Nov 30 21:29:39 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 668D9DBB529 for ; Thu, 30 Nov 2017 21:29:39 +0000 (UTC) (envelope-from k@free.de) Received: from smtp.free.de (smtp.free.de [91.204.6.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 131FF76D9A for ; Thu, 30 Nov 2017 21:29:38 +0000 (UTC) (envelope-from k@free.de) Received: from [127.0.0.1] (localhost [127.0.0.1]) by smtp.free.de (Postfix) with ESMTPSA id 1953D153FC for ; Thu, 30 Nov 2017 22:29:36 +0100 (CET) From: Kai Gallasch To: freebsd-virtualization@freebsd.org Subject: Linux lockups inside bhyve VM on FreeBSD 11.1 Message-ID: <5b1166b2-0b8c-248c-9a3b-b5c9685e4f37@free.de> Date: Thu, 30 Nov 2017 22:29:35 +0100 Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Nov 2017 21:29:39 -0000 Hello. Last day an apache 2.4 running inside a Debian9 linux bhyve VM locked up on one of my servers (FreeBSD 11.1-RELENG, GENERIC kernel) overloading the VM. The VM uses a ZFS zvol blockdevice on top of a zpool, consisting of two mirrored SSDs. I was able to enter the VM through the bhyve console, kill and restart the stuck apache process and regain stability inside the VM. I found below output in the Linux dmesg and suspect the ext4 journaling to be the culprit. Has anyone experienced similar lockups running Linux inside a bhyve VM? At the time when this happened there was no high I/O on the VM zpool. Cheers, K. [1594985.015199] INFO: task jbd2/vda1-8:161 blocked for more than 120 seconds. [1594985.015841] Not tainted 4.9.0-4-amd64 #1 Debian 4.9.51-1 [1594985.016375] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1594985.017074] jbd2/vda1-8 D 0 161 2 0x00000000 [1594985.017078] ffff885477ec5400 0000000000000000 ffff88547620e280 ffff88547fc18240 [1594985.017080] ffffffff8e00e500 ffffb056c0957ca0 ffffffff8da038e3 ffff8854765bd088 [1594985.017081] 0000000000000246 ffff88547fc18240 ffffb056c0957d80 ffff88547620e280 [1594985.017082] Call Trace: [1594985.017116] [] ? __schedule+0x233/0x6d0 [1594985.017131] [] ? prepare_to_wait_event+0xf0/0xf0 [1594985.017132] [] ? schedule+0x32/0x80 [1594985.017165] [] ? jbd2_journal_commit_transaction+0x25f/0x17a0 [jbd2] [1594985.017171] [] ? update_curr+0xe1/0x160 [1594985.017172] [] ? account_entity_dequeue+0xa4/0xc0 [1594985.017173] [] ? prepare_to_wait_event+0xf0/0xf0 [1594985.017176] [] ? kjournald2+0xc2/0x260 [jbd2] [1594985.017177] [] ? prepare_to_wait_event+0xf0/0xf0 [1594985.017180] [] ? commit_timeout+0x10/0x10 [jbd2] [1594985.017186] [] ? do_group_exit+0x3a/0xa0 [1594985.017191] [] ? kthread+0xd7/0xf0 [1594985.017192] [] ? kthread_park+0x60/0x60 [1594985.017198] [] ? ret_from_fork+0x25/0x30 [1594985.017202] INFO: task rs:main Q:Reg:407 blocked for more than 120 seconds. [1594985.017841] Not tainted 4.9.0-4-amd64 #1 Debian 4.9.51-1 [1594985.018373] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1594985.019116] rs:main Q:Reg D 0 407 1 0x00000000 [1594985.019118] ffff885476928000 0000000000000000 ffff885479283140 ffff88547fc18240 [1594985.019119] ffffffff8e00e500 ffffb056c0c1fb48 ffffffff8da038e3 2406f83c84b44b0f [1594985.019121] 00ff8853c8784380 ffff88547fc18240 ffffb056c0c1fb68 ffff885479283140 [1594985.019122] Call Trace: [1594985.019124] [] ? __schedule+0x233/0x6d0 [1594985.019125] [] ? schedule+0x32/0x80 [1594985.019129] [] ? wait_transaction_locked+0x86/0xc0 [jbd2] [1594985.019130] [] ? prepare_to_wait_event+0xf[1599459.680158] serial8250: too much work for irq4 0/0xf0 [1594985.019132] [] ? add_transaction_credits+0x1b8/0x290 [jbd2] [1594985.019142] [] ? __switch_to+0x2c1/0x6d0 [1594985.019145] [] ? start_this_handle+0x105/0x400 [jbd2] [1594985.019146] [] ? __schedule+0x23b/0x6d0 [1594985.019147] [] ? check_preempt_wakeup+0x103/0x210 [1594985.019150] [] ? jbd2__journal_start+0xd9/0x1e0 [jbd2] [1594985.019238] [] ? ext4_dirty_inode+0x2d/0x60 [ext4] [1594985.019253] [] ? __mark_inode_dirty+0x165/0x350 [1594985.019258] [] ? generic_update_time+0x79/0xd0 [1594985.019259] [] ? current_time+0x36/0x70 [1594985.019260] [] ? file_update_time+0xbc/0x110 [1594985.019271] [] ? __generic_file_write_iter+0x99/0x1b0 [1594985.019278] [] ? ext4_file_write_iter+0x90/0x370 [ext4] [1594985.019288] [] ? do_futex+0x2c9/0xb00 [1594985.019294] [] ? fsnotify+0x381/0x4e0 [1594985.019299] [] ? new_sync_write+0xda/0x130 [1594985.019305] [] ? vfs_write+0xb0/0x190 [1594985.019307] [] ? SyS_write+0x52/0xc0 [1594985.019309] [] ? system_call_fast_compare_end+0xc/0x9b [1594985.019344] INFO: task kworker/u8:2:19882 blocked for more than 120 seconds. [1594985.019985] Not tainted 4.9.0-4-amd64 #1 Debian 4.9.51-1 [1594985.020512] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [1594985.021215] kworker/u8:2 D 0 19882 2 0x00000000 [1594985.021220] Workqueue: writeback wb_workfn (flush-254:0) From owner-freebsd-virtualization@freebsd.org Thu Nov 30 23:21:49 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E92B1DEC7B7 for ; Thu, 30 Nov 2017 23:21:49 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id C3D2C7B04E for ; Thu, 30 Nov 2017 23:21:49 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id 3C2F48432DEC for ; Thu, 30 Nov 2017 17:15:20 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id NpukMehXGUEc for ; Thu, 30 Nov 2017 17:15:19 -0600 (CST) Received: from square.office.ebureau.com (unknown [10.10.21.22]) by internet06.ebureau.com (Postfix) with ESMTPSA id CEABF8432DDF for ; Thu, 30 Nov 2017 17:15:19 -0600 (CST) From: Dustin Wenz Content-Type: multipart/signed; boundary="Apple-Mail=_B95D81D8-FCDE-491A-8C07-07FF98971E35"; protocol="application/pkcs7-signature"; micalg=sha1 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: bhyve uses all available memory during IO-intensive operations Message-Id: Date: Thu, 30 Nov 2017 17:15:19 -0600 To: freebsd-virtualization@freebsd.org X-Mailer: Apple Mail (2.3445.4.7) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Nov 2017 23:21:50 -0000 --Apple-Mail=_B95D81D8-FCDE-491A-8C07-07FF98971E35 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS = is also FreeBSD 11.1). Their sole purpose is to house some medium-sized = Postgres databases (100-200GB). The host system has 64GB of real memory = and 112GB of swap. I have configured each guest to only use 16GB of = memory, yet while doing my initial database imports in the VMs, bhyve = will quickly grow to use all available system memory and then be killed = by the kernel: kernel: swap_pager: I/O error - pageout failed; blkno 1735,size = 4096, error 12 kernel: swap_pager: I/O error - pageout failed; blkno 1610,size = 4096, error 12 kernel: swap_pager: I/O error - pageout failed; blkno 1763,size = 4096, error 12 kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space The OOM condition seems related to doing moderate IO within the VM, = though nothing within the VM itself shows high memory usage. This is the = chyves config for one of them: bargs -A -H -P -S bhyve_disk_type virtio-blk bhyve_net_type virtio-net bhyveload_flags chyves_guest_version 0300 cpu 4 creation Created on Mon Oct 23 16:17:04 CDT = 2017 by chyves v0.2.0 2016/09/11 using __create() loader bhyveload net_ifaces tap51 os default ram 16G rcboot 0 revert_to_snapshot revert_to_snapshot_method off serial nmdm51 template no uuid 8495a130-b837-11e7-b092-0025909a8b56 I've also tried using different bhyve_disk_types, with no improvement. = How is it that bhyve can use far more memory that I'm specifying? - .Dustin= --Apple-Mail=_B95D81D8-FCDE-491A-8C07-07FF98971E35 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIEzDCCBMgw ggOwoAMCAQICAUEwDQYJKoZIhvcNAQELBQAwgZkxCzAJBgNVBAYTAlVTMRIwEAYDVQQIEwlNaW5u ZXNvdGExFDASBgNVBAcTC1NhaW50IENsb3VkMRAwDgYDVQQKEwdlQnVyZWF1MRQwEgYDVQQLEwtJ bnRlZ3JhdGlvbjEUMBIGA1UEAxMLZWJ1cmVhdS5jb20xIjAgBgkqhkiG9w0BCQEWE3N1cHBvcnRA ZWJ1cmVhdS5jb20wHhcNMTcwNTA1MTYxNjE1WhcNMjcwNTAzMTYxNjE1WjBKMQswCQYDVQQGEwJV UzEUMBIGA1UEAwwLRHVzdGluIFdlbnoxJTAjBgkqhkiG9w0BCQEWFmR1c3RpbndlbnpAZWJ1cmVh dS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCQ/HJXe7JhUaexqEaxCNVifsue sUMgohgXLmi5YIcbAWhdxTr1PUzKYkeTkL9sYNjXU6uaI2tZMF3hA9gcFjxQIfkKSo31MrYOgMdU xQe0Q+t6Vd4pGAmtDQDwhAsrNGccADp3Yjy4eBtVfkDGdzz1Y8Lbc684TPFcW7i9+U/dDaXlcxeq fyDqiHZ5y8Lp/1M2Ot/Rz7eikJZTAuHOWKs/PEiJIM2JHuhPyNy+mL2oqEWeOcEsKMNzgn7HVt4k Xz2irBAG+cj4WAxWs418l46EEXgur4PvhBXZMl0LJg0TyaxOHbsUam4R4tbKnaZ3HhRkg79k2Had sb6DKbnCw9/1AgMBAAGjggFnMIIBYzAJBgNVHRMEAjAAMAsGA1UdDwQEAwIE8DAnBgNVHSUEIDAe BggrBgEFBQcDAQYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBTLi/8HUHpbBEt9OtPqQoax AmpaNDCBzgYDVR0jBIHGMIHDgBRnpZeXB5rQYLgsUKqiiBcLIHyu6aGBn6SBnDCBmTELMAkGA1UE BhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoT B2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAG CSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbYIJAMwZcjAWAsWXMDAGCWCGSAGG+EIBBAQj FiFodHRwOi8vd3d3LmVidXJlYXUuY29tL2NhLWNybC5wZW0wDQYJKoZIhvcNAQELBQADggEBAHbO qVdB9raUKXCgZRA/nES5a60dlIaGnIlpgz+Y3SjFt0bcJxoUYhIzumBHk9yjyP4M1DubOphkQpJ4 LNZbAS01cjCxjnC0ZUq5V3FCeaDwrn1qPY+QJGoZPLlhWdJUNu17OpnR7ZfBWlp3/pRhvNU5PCbJ nmF7rnvsqxUFq9oeiV3SmqBux5lwJ7p2Uss5SHSW6g17K/KdTMK1roQr/+rWpxp2233qddDrLpOE xGRlvhEqSa/IZbGC9oiYmsiaG1PefQkadoob5IMIS5/MDpWHUgSHqAj1V/LwcCx0rbt73SazGMND EzHVWhsj+khepB/MG5QGfWP23IGFmvQYWWcxggOQMIIDjAIBATCBnzCBmTELMAkGA1UEBhMCVVMx EjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoTB2VCdXJl YXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAGCSqGSIb3 DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTAJBgUrDgMCGgUAoIIBxTAYBgkqhkiG9w0BCQMx CwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNzExMzAyMzE1MTlaMCMGCSqGSIb3DQEJBDEW BBTyU3crdrxhunlwSLEDBcmGeEtHvTCBsAYJKwYBBAGCNxAEMYGiMIGfMIGZMQswCQYDVQQGEwJV UzESMBAGA1UECBMJTWlubmVzb3RhMRQwEgYDVQQHEwtTYWludCBDbG91ZDEQMA4GA1UEChMHZUJ1 cmVhdTEUMBIGA1UECxMLSW50ZWdyYXRpb24xFDASBgNVBAMTC2VidXJlYXUuY29tMSIwIAYJKoZI hvcNAQkBFhNzdXBwb3J0QGVidXJlYXUuY29tAgFBMIGyBgsqhkiG9w0BCRACCzGBoqCBnzCBmTEL MAkGA1UEBhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAO BgNVBAoTB2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNv bTEiMCAGCSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTANBgkqhkiG9w0BAQEFAASC AQAIga0Y/e+v/2M+VVbdLrxM+3syJ2sLEliNv0+MmGBI0tdnUSve745seRmnkqCvE0GMQwx/iaEo uFua/veh8A501m1kJuOU/wlgcsrfcYeFAkq0tW/DpWTEuNJtTdnYyUJkzpTpcqx6oQPOSI6FU9e+ Zcm6XcvIFHHz2cLP3+QMLEIK5b3LPcAySCjTvojyDmlfF8OL1M3oLN8ULUpdwIvd8QcjeeLrez/L bpOiBw3yzHb3Yp+FyeGrxWLS+gMD+oHOTcbzANU9vmNNIoVA6mD8qHpghna/jrutZPVrhkvUOp2s /i0TCpRRzFbZ3niFoV5EYc84Z6m5V8Jw1WZkjUw5AAAAAAAA --Apple-Mail=_B95D81D8-FCDE-491A-8C07-07FF98971E35-- From owner-freebsd-virtualization@freebsd.org Thu Nov 30 23:28:42 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AB91BDECA84 for ; Thu, 30 Nov 2017 23:28:42 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8D5327B325 for ; Thu, 30 Nov 2017 23:28:42 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from Ticonderoga.HML3.ScaleEngine.net (senat1-01.HML3.ScaleEngine.net [209.51.186.5]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id D93151440A for ; Thu, 30 Nov 2017 23:28:40 +0000 (UTC) Subject: Re: bhyve uses all available memory during IO-intensive operations To: freebsd-virtualization@freebsd.org References: From: Allan Jude Message-ID: Date: Thu, 30 Nov 2017 18:28:39 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 Nov 2017 23:28:42 -0000 On 11/30/2017 18:15, Dustin Wenz wrote: > I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS is also FreeBSD 11.1). Their sole purpose is to house some medium-sized Postgres databases (100-200GB). The host system has 64GB of real memory and 112GB of swap. I have configured each guest to only use 16GB of memory, yet while doing my initial database imports in the VMs, bhyve will quickly grow to use all available system memory and then be killed by the kernel: > > kernel: swap_pager: I/O error - pageout failed; blkno 1735,size 4096, error 12 > kernel: swap_pager: I/O error - pageout failed; blkno 1610,size 4096, error 12 > kernel: swap_pager: I/O error - pageout failed; blkno 1763,size 4096, error 12 > kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space > > The OOM condition seems related to doing moderate IO within the VM, though nothing within the VM itself shows high memory usage. This is the chyves config for one of them: > > bargs -A -H -P -S > bhyve_disk_type virtio-blk > bhyve_net_type virtio-net > bhyveload_flags > chyves_guest_version 0300 > cpu 4 > creation Created on Mon Oct 23 16:17:04 CDT 2017 by chyves v0.2.0 2016/09/11 using __create() > loader bhyveload > net_ifaces tap51 > os default > ram 16G > rcboot 0 > revert_to_snapshot > revert_to_snapshot_method off > serial nmdm51 > template no > uuid 8495a130-b837-11e7-b092-0025909a8b56 > > > I've also tried using different bhyve_disk_types, with no improvement. How is it that bhyve can use far more memory that I'm specifying? > > - .Dustin > Can you show 'top' output. What makes you think bhyve is using the memory? Are you using ZFS? Have you limited the vfs.zfs.arc_max to leave some free RAM for the bhyve instances? -- Allan Jude From owner-freebsd-virtualization@freebsd.org Fri Dec 1 02:41:46 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0D514DF13FD for ; Fri, 1 Dec 2017 02:41:46 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: from mail-it0-x231.google.com (mail-it0-x231.google.com [IPv6:2607:f8b0:4001:c0b::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C5FF7804BA for ; Fri, 1 Dec 2017 02:41:45 +0000 (UTC) (envelope-from jtubnor@gmail.com) Received: by mail-it0-x231.google.com with SMTP id x28so915146ita.0 for ; Thu, 30 Nov 2017 18:41:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=dJ1vNPcMWjDKuZa505uDckNwZjnY3dR1XQUvMeoow0w=; b=gTnapkK49jBrif0mvG+5GcctvcBWAaESQX3u2FtTSZwRzH8OWTa0tTQFKYk2bg/YyC vQhb3HxMF60T5vsfS29tkFpcgFemn0mYpPwhYZwsTts62z1JHfACy8iLT98WXs98GWHK jj09xxi/uXa/gLXz+Q5/eoh7FMLytNr17hptBsLrghxhSA/Jz893t0/Oy0jJ8Fzxe2bY pk5jLfB2f1AQzvhN3fy9QE1C3w3IzlYaXUHpE4LNlDCzqSRy1xE+5lqwAJCQQU6Jw0JD kN0jRWEay2D/aZWFJ+6T7kmGCuJsuRy6eB3K2K5orREF76S7fHT4oIMc4uXnHu84gvid 5ZrA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=dJ1vNPcMWjDKuZa505uDckNwZjnY3dR1XQUvMeoow0w=; b=NTAlzCBiMu0GWQIBNe39ge2YJjTps4EC1leDl+IOvU26JTtHu/HmmrIv9Xfu9GxCbq abumIK/Yed6DUYO9igFYgYj5AVGYi1DYoH5eqC0NhLZPWNwI22W7WfCJhQxeuL6mtmvy lOjg3P02ELYdix8rddi/ti/Q/CCKxn96exNxq3L4wxM0xgA/ZSURJNK4GEVasgJzE4sr HV/MxvEhsf3T12Jc2hVmFtV8RDJ1u+79zt8dU66daXo3mt14MGpRs3vWuVPTdXrcj0RV zA5jZ7SeTqdfuqYUXELL02yuYN7f1R2vZlFrJu9GB25cRBL9szTU1jlAroP85H23EdGB 3DoA== X-Gm-Message-State: AKGB3mKXpZ4E+Z0neImQu3A8c6NFhwhIR8Udwaow1wTD1Jv9elvsal6U 3J6ggEqId/WwyIhV9pT85tjrLj22a++GJzs/kH4= X-Google-Smtp-Source: AGs4zMaWU1pX7P3SsL2yybx5r/H2mspWm4HY1sQEo2fRK65Y21FfLW1sy97LHCHUK38tu88jtiuBcLfOHnSy4iCRtxU= X-Received: by 10.36.122.17 with SMTP id a17mr36851itc.30.1512096105120; Thu, 30 Nov 2017 18:41:45 -0800 (PST) MIME-Version: 1.0 Sender: jtubnor@gmail.com Received: by 10.2.69.213 with HTTP; Thu, 30 Nov 2017 18:41:24 -0800 (PST) In-Reply-To: <5b1166b2-0b8c-248c-9a3b-b5c9685e4f37@free.de> References: <5b1166b2-0b8c-248c-9a3b-b5c9685e4f37@free.de> From: Jason Tubnor Date: Fri, 1 Dec 2017 13:41:24 +1100 X-Google-Sender-Auth: RDC_Ihk1EBI9Com-qlQXdVl_hCY Message-ID: Subject: Re: Linux lockups inside bhyve VM on FreeBSD 11.1 To: Kai Gallasch Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 02:41:46 -0000 On 1 December 2017 at 08:29, Kai Gallasch wrote: > Hello. > > Last day an apache 2.4 running inside a Debian9 linux bhyve VM locked up > on one of my servers (FreeBSD 11.1-RELENG, GENERIC kernel) overloading > the VM. > > The VM uses a ZFS zvol blockdevice on top of a zpool, consisting of two > mirrored SSDs. > > I was able to enter the VM through the bhyve console, kill and restart > the stuck apache process and regain stability inside the VM. > > I found below output in the Linux dmesg and suspect the ext4 journaling > to be the culprit. > > Has anyone experienced similar lockups running Linux inside a bhyve VM? > At the time when this happened there was no high I/O on the VM zpool. Have you set vfs.zfs.arc_max to a lower value to allow for bhyve head room? How was the host system swap, did the host start to eat into it? I run a few guests with Ubuntu 16.04 but mainly use XFS for areas that aren't system related and haven't come across this issue. Cheers, Jason. From owner-freebsd-virtualization@freebsd.org Fri Dec 1 03:11:57 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41034DF213C for ; Fri, 1 Dec 2017 03:11:57 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 1A97D168F; Fri, 1 Dec 2017 03:11:56 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id E320A84345D8; Thu, 30 Nov 2017 21:11:54 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id LbnUmqXrebcE; Thu, 30 Nov 2017 21:11:54 -0600 (CST) Received: from [10.124.49.90] (mobile-166-175-190-37.mycingular.net [166.175.190.37]) by internet06.ebureau.com (Postfix) with ESMTPSA id 8F20684345C3; Thu, 30 Nov 2017 21:11:53 -0600 (CST) Mime-Version: 1.0 (1.0) Subject: Re: bhyve uses all available memory during IO-intensive operations From: Dustin Wenz X-Mailer: iPhone Mail (15B202) In-Reply-To: Date: Thu, 30 Nov 2017 21:10:26 -0600 Cc: freebsd-virtualization@freebsd.org Message-Id: References: To: Allan Jude Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 03:11:57 -0000 I am using a zvol as the storage for the VM, and I do not have any ARC limit= s set. However, the bhyve process itself ends up grabbing the vast majority o= f memory.=20 I=E2=80=99ll run a test tomorrow to get the exact output from top. - .Dustin > On Nov 30, 2017, at 5:28 PM, Allan Jude wrote: >=20 >> On 11/30/2017 18:15, Dustin Wenz wrote: >> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS is= also FreeBSD 11.1). Their sole purpose is to house some medium-sized Postgr= es databases (100-200GB). The host system has 64GB of real memory and 112GB o= f swap. I have configured each guest to only use 16GB of memory, yet while d= oing my initial database imports in the VMs, bhyve will quickly grow to use a= ll available system memory and then be killed by the kernel: >>=20 >> kernel: swap_pager: I/O error - pageout failed; blkno 1735,size 4096, e= rror 12 >> kernel: swap_pager: I/O error - pageout failed; blkno 1610,size 4096, e= rror 12 >> kernel: swap_pager: I/O error - pageout failed; blkno 1763,size 4096, e= rror 12 >> kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space >>=20 >> The OOM condition seems related to doing moderate IO within the VM, thoug= h nothing within the VM itself shows high memory usage. This is the chyves c= onfig for one of them: >>=20 >> bargs -A -H -P -S >> bhyve_disk_type virtio-blk >> bhyve_net_type virtio-net >> bhyveload_flags >> chyves_guest_version 0300 >> cpu 4 >> creation Created on Mon Oct 23 16:17:04 CDT 2017 by c= hyves v0.2.0 2016/09/11 using __create() >> loader bhyveload >> net_ifaces tap51 >> os default >> ram 16G >> rcboot 0 >> revert_to_snapshot >> revert_to_snapshot_method off >> serial nmdm51 >> template no >> uuid 8495a130-b837-11e7-b092-0025909a8b56 >>=20 >>=20 >> I've also tried using different bhyve_disk_types, with no improvement. Ho= w is it that bhyve can use far more memory that I'm specifying? >>=20 >> - .Dustin >>=20 >=20 > Can you show 'top' output. What makes you think bhyve is using the > memory? Are you using ZFS? Have you limited the vfs.zfs.arc_max to leave > some free RAM for the bhyve instances? >=20 > --=20 > Allan Jude > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization-unsubscribe@freeb= sd.org" From owner-freebsd-virtualization@freebsd.org Fri Dec 1 03:13:52 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BAC91DF21E1 for ; Fri, 1 Dec 2017 03:13:52 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 98C6417C7 for ; Fri, 1 Dec 2017 03:13:52 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (Seawolf.HML3.ScaleEngine.net [209.51.186.28]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id DA15B14817; Fri, 1 Dec 2017 03:13:44 +0000 (UTC) Subject: Re: bhyve uses all available memory during IO-intensive operations To: Dustin Wenz Cc: freebsd-virtualization@freebsd.org References: From: Allan Jude Message-ID: Date: Thu, 30 Nov 2017 22:13:44 -0500 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 03:13:52 -0000 On 2017-11-30 22:10, Dustin Wenz wrote: > I am using a zvol as the storage for the VM, and I do not have any ARC > limits set. However, the bhyve process itself ends up grabbing the vast > majority of memory.  > > I’ll run a test tomorrow to get the exact output from top. > >    - .Dustin > > On Nov 30, 2017, at 5:28 PM, Allan Jude > wrote: > >> On 11/30/2017 18:15, Dustin Wenz wrote: >>> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest >>> OS is also FreeBSD 11.1). Their sole purpose is to house some >>> medium-sized Postgres databases (100-200GB). The host system has 64GB >>> of real memory and 112GB of swap. I have configured each guest to >>> only use 16GB of memory, yet while doing my initial database imports >>> in the VMs, bhyve will quickly grow to use all available system >>> memory and then be killed by the kernel: >>> >>>    kernel: swap_pager: I/O error - pageout failed; blkno 1735,size >>> 4096, error 12 >>>    kernel: swap_pager: I/O error - pageout failed; blkno 1610,size >>> 4096, error 12 >>>    kernel: swap_pager: I/O error - pageout failed; blkno 1763,size >>> 4096, error 12 >>>    kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space >>> >>> The OOM condition seems related to doing moderate IO within the VM, >>> though nothing within the VM itself shows high memory usage. This is >>> the chyves config for one of them: >>> >>>    bargs                      -A -H -P -S >>>    bhyve_disk_type            virtio-blk >>>    bhyve_net_type             virtio-net >>>    bhyveload_flags >>>    chyves_guest_version       0300 >>>    cpu                        4 >>>    creation                   Created on Mon Oct 23 16:17:04 CDT 2017 >>> by chyves v0.2.0 2016/09/11 using __create() >>>    loader                     bhyveload >>>    net_ifaces                 tap51 >>>    os                         default >>>    ram                        16G >>>    rcboot                     0 >>>    revert_to_snapshot >>>    revert_to_snapshot_method  off >>>    serial                     nmdm51 >>>    template                   no >>>    uuid                       8495a130-b837-11e7-b092-0025909a8b56 >>> >>> >>> I've also tried using different bhyve_disk_types, with no >>> improvement. How is it that bhyve can use far more memory that I'm >>> specifying? >>> >>>    - .Dustin >>> >> >> Can you show 'top' output. What makes you think bhyve is using the >> memory? Are you using ZFS? Have you limited the vfs.zfs.arc_max to leave >> some free RAM for the bhyve instances? >> >> -- >> Allan Jude >> _______________________________________________ >> freebsd-virtualization@freebsd.org >> mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization >> To unsubscribe, send any mail to >> "freebsd-virtualization-unsubscribe@freebsd.org >> " The default limit for the ARC is 99% of your ram, so you are definitely going to want to reduce that to something like: 90% of your 64GB of ram, less the total amount of RAM you have given to all VMs. -- Allan Jude From owner-freebsd-virtualization@freebsd.org Fri Dec 1 09:39:27 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 40BBDDF8EFA for ; Fri, 1 Dec 2017 09:39:27 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from forward101o.mail.yandex.net (forward101o.mail.yandex.net [IPv6:2a02:6b8:0:1a2d::601]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Yandex CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EB33E78573 for ; Fri, 1 Dec 2017 09:39:26 +0000 (UTC) (envelope-from bsam@passap.ru) Received: from mxback10j.mail.yandex.net (mxback10j.mail.yandex.net [IPv6:2a02:6b8:0:1619::113]) by forward101o.mail.yandex.net (Yandex) with ESMTP id 214761345A8F for ; Fri, 1 Dec 2017 12:39:23 +0300 (MSK) Received: from smtp3o.mail.yandex.net (smtp3o.mail.yandex.net [2a02:6b8:0:1a2d::27]) by mxback10j.mail.yandex.net (nwsmtp/Yandex) with ESMTP id 28lZuTZf9z-dNgKCrj0; Fri, 01 Dec 2017 12:39:23 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1512121163; bh=BOwZkwJCM35St2QaVYIChhbGzouiKkz1g64RPmT6Aa8=; h=Subject:To:References:From:Message-ID:Date:In-Reply-To; b=WT68vBm09lXcaWiPGkthlHqDYLGdxIh0xKdA+KIm7uuUrTz6EyJUMhkPYeUZPp7cP jOViV/SU1mBVMD2+Jb22zz0K/1sLQKYApTKgCzrDoWrUHdGxsu/M9zF37wyTdrOORg sXyaOzsbd8AGxhqUwItJ7s0/Fp04MXiXTQ0PY0JQ= Received: by smtp3o.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id jkX3X6C5Un-dMS0f9Gm; Fri, 01 Dec 2017 12:39:22 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=passap.ru; s=mail; t=1512121162; bh=BOwZkwJCM35St2QaVYIChhbGzouiKkz1g64RPmT6Aa8=; h=Subject:To:References:From:Message-ID:Date:In-Reply-To; b=jVz3etsB19u5tlVRhC/HyKWPIQF6yNJdINuw/HSOIV1rYtUVWHtjFYvJVyhi21P9p Chpo9cQqb9duVTZ6kZtdXs5k1BDzf5e5k/vt0v5XMXgS86av75vJyb59Ykg7BksO1L 8f4MNw9tcow4WgD2vJh28ST4/kO8OXyoizfpeVXw= Authentication-Results: smtp3o.mail.yandex.net; dkim=pass header.i=@passap.ru Subject: Re: bhyve uses all available memory during IO-intensive operations To: freebsd-virtualization@freebsd.org References: From: Boris Samorodov Message-ID: Date: Fri, 1 Dec 2017 12:39:22 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 09:39:27 -0000 Hi Dustin, All, 01.12.2017 02:15, Dustin Wenz пишет: > bhyve will quickly grow to use all available system memory I'd say that some logs/stats/values should help here. JFYI: I've just got a success importing of an earth base OSM/nominatim/postgis at bhyve guest (CentOS-7.3, 14 CPU, 36GB RAM, 1TB HDD). The process lasted for about 14 days. It was really IO and CPU intensive. No other bhyve guests and the host as well have been disturbed. The host OS is FreeBSD-HEAD though. -- WBR, Boris Samorodov (bsam) FreeBSD Committer, http://www.FreeBSD.org The Power To Serve From owner-freebsd-virtualization@freebsd.org Fri Dec 1 09:54:03 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3A373DFA36F for ; Fri, 1 Dec 2017 09:54:03 +0000 (UTC) (envelope-from FreeBSD@shaneware.biz) Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by mx1.freebsd.org (Postfix) with ESMTP id 8F29C791A9; Fri, 1 Dec 2017 09:54:01 +0000 (UTC) (envelope-from FreeBSD@shaneware.biz) Received: from ppp121-45-11-215.bras1.adl4.internode.on.net (HELO leader.local) ([121.45.11.215]) by ipmail07.adl2.internode.on.net with ESMTP; 01 Dec 2017 20:23:53 +1030 Subject: Re: bhyve uses all available memory during IO-intensive operations To: Allan Jude , Dustin Wenz Cc: freebsd-virtualization@freebsd.org References: From: Shane Ambler Message-ID: Date: Fri, 1 Dec 2017 20:23:51 +1030 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-AU Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 09:54:03 -0000 On 01/12/2017 13:43, Allan Jude wrote: > On 2017-11-30 22:10, Dustin Wenz wrote: >> I am using a zvol as the storage for the VM, and I do not have any ARC >> limits set. However, the bhyve process itself ends up grabbing the vast >> majority of memory.  >> >> I’ll run a test tomorrow to get the exact output from top. >> >>    - .Dustin >> >> On Nov 30, 2017, at 5:28 PM, Allan Jude > > wrote: >> >>> On 11/30/2017 18:15, Dustin Wenz wrote: >>>> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest >>>> OS is also FreeBSD 11.1). Their sole purpose is to house some >>>> medium-sized Postgres databases (100-200GB). The host system has 64GB >>>> of real memory and 112GB of swap. I have configured each guest to >>>> only use 16GB of memory, yet while doing my initial database imports >>>> in the VMs, bhyve will quickly grow to use all available system >>>> memory and then be killed by the kernel: >>>> >>>>    kernel: swap_pager: I/O error - pageout failed; blkno 1735,size >>>> 4096, error 12 >>>>    kernel: swap_pager: I/O error - pageout failed; blkno 1610,size >>>> 4096, error 12 >>>>    kernel: swap_pager: I/O error - pageout failed; blkno 1763,size >>>> 4096, error 12 >>>>    kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space That's the type of errors I see when wired jumps high. I'm not seeing this from bhyve but when your watching top, keep an eye on the wired amount. -- FreeBSD - the place to B...Sharing Devices Shane Ambler From owner-freebsd-virtualization@freebsd.org Fri Dec 1 11:23:19 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 09B8ADFCCB9 for ; Fri, 1 Dec 2017 11:23:19 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: from mail-lf0-x231.google.com (mail-lf0-x231.google.com [IPv6:2a00:1450:4010:c07::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7E5397D46A; Fri, 1 Dec 2017 11:23:18 +0000 (UTC) (envelope-from araujobsdport@gmail.com) Received: by mail-lf0-x231.google.com with SMTP id t197so11323075lfe.7; Fri, 01 Dec 2017 03:23:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:reply-to:in-reply-to:references:from:date:message-id :subject:to:cc; bh=oobhfC0mhEzq3bPOHrPgc3CODSRtORePKAlM3n9DeGY=; b=NjpdjgJApQByaHsu71HAcl6N3+4yxLJhil6INLOJglMrXcGbd/mjwCNRYQxV0slYX+ UbTuCm5vyVAhnETrJbNzfE+c9lPyDG6wejJ2QxnZ2mqgYrIB3diFuY9uuRaVvyr+pisp HiCn8oFZwlRhU+KcBET43F//FIdKidx6hiRce6naV0MdZAoHbhC9706WFGG9BWSym6pD 31gmMoHleoBkyZ8wHlhspNElc0YojKTuGGpMGuiFUM/szMigAzb4PbKU5xyOKH7zBdLx xxT1tzznWRgh/8V/7Uar6/Sr6Ji2+r/wTQNzzc4II447clo9/Rzg/Y5crQzBox/jhWd1 dl+A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:reply-to:in-reply-to:references :from:date:message-id:subject:to:cc; bh=oobhfC0mhEzq3bPOHrPgc3CODSRtORePKAlM3n9DeGY=; b=JMKUvyRjD7I9nJOR7dfuu2iCOk7I6bsWGgEBjQz3AcSHGyr1ghJ2ClntGkYkkXNZWw mcE0hcSVv+OZRw8CmW7YITMBkUwMSZHi7tTi90vu+WhsURWp0MfHKv+rIB2bpouxOffj 2g5wpELAuevSG2zdj24TO4nPAikb4+Gt6cv4HM0lKO0kKNTZ7KlilMsgpm/wBBzo2cZc b9EJEoXs0D2g46urrkoeVTf+3GQb17hDUVQXlM+u9xcvijoFcsUZEXDTbbb+U5k5ea7J 08VDXr/1EFzDaUWOj+Xie9fCf8gkUV6yFa6crkam35LosC0qthxXCxecoz+IjpTLNTA+ dt3g== X-Gm-Message-State: AJaThX5uomYR8Ae5WYrbvJq1uB+P0P1/lXJ3CCFXtxcAmiYzPV/qpVux qg9GZIRFwvn/wbPvfoAogn9TBzswYOW9TkXk8xcv8Q== X-Google-Smtp-Source: AGs4zMbt4d11wu2hNjM3mSu2Tnqo5PYT8BplR/POrs8Cbb3vFNQOvvdZ4pCndq1J/BCKMS/j5kk0s15eac6oSfGWDdI= X-Received: by 10.25.79.67 with SMTP id a3mr3810174lfk.162.1512127396300; Fri, 01 Dec 2017 03:23:16 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.168.197 with HTTP; Fri, 1 Dec 2017 03:23:15 -0800 (PST) Reply-To: araujo@freebsd.org In-Reply-To: References: From: Marcelo Araujo Date: Fri, 1 Dec 2017 19:23:15 +0800 Message-ID: Subject: Re: bhyve uses all available memory during IO-intensive operations To: Shane Ambler Cc: Allan Jude , Dustin Wenz , freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 11:23:19 -0000 2017-12-01 17:53 GMT+08:00 Shane Ambler : > On 01/12/2017 13:43, Allan Jude wrote: > > On 2017-11-30 22:10, Dustin Wenz wrote: > >> I am using a zvol as the storage for the VM, and I do not have any ARC > >> limits set. However, the bhyve process itself ends up grabbing the vas= t > >> majority of memory.=C3=82 > >> > >> I=C3=A2=E2=82=AC=E2=84=A2ll run a test tomorrow to get the exact outpu= t from top. > >> > >> =C3=82 =C3=82 - .Dustin > >> > >> On Nov 30, 2017, at 5:28 PM, Allan Jude >> > wrote: > >> > >>> On 11/30/2017 18:15, Dustin Wenz wrote: > >>>> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest > >>>> OS is also FreeBSD 11.1). Their sole purpose is to house some > >>>> medium-sized Postgres databases (100-200GB). The host system has 64G= B > >>>> of real memory and 112GB of swap. I have configured each guest to > >>>> only use 16GB of memory, yet while doing my initial database imports > >>>> in the VMs, bhyve will quickly grow to use all available system > >>>> memory and then be killed by the kernel: > >>>> > >>>> =C3=82 =C3=82 kernel: swap_pager: I/O error - pageout failed; blkno= 1735,size > >>>> 4096, error 12 > >>>> =C3=82 =C3=82 kernel: swap_pager: I/O error - pageout failed; blkno= 1610,size > >>>> 4096, error 12 > >>>> =C3=82 =C3=82 kernel: swap_pager: I/O error - pageout failed; blkno= 1763,size > >>>> 4096, error 12 > >>>> =C3=82 =C3=82 kernel: pid 41123 (bhyve), uid 0, was killed: out of = swap space > > That's the type of errors I see when wired jumps high. I'm not seeing > this from bhyve but when your watching top, keep an eye on the wired > amount. > > -- > FreeBSD - the place to B...Sharing Devices > > Shane Ambler > > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscribe@freebsd.org" > As Allan said, it is likely to be your ARC cache holding memory and unfortunately right now all these swap out doesn't play well with the combination of zfs + bhyve. Try to tune your vfs.zfs.arc_max to a minimum where you give memory space enough to your VM. What I'm doing now is, launch a VM get the amount of memory and remove it from vfs.zfs.arc_max, as soon as the VM stops, I give the memory back to vfs.zfs.arc_max. Best, --=20 --=20 Marcelo Araujo (__)araujo@FreeBSD.org \\\'',)http://www.FreeBSD.org \/ \ ^ Power To Server. .\. /_) From owner-freebsd-virtualization@freebsd.org Fri Dec 1 17:43:02 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 01BD5E65F26 for ; Fri, 1 Dec 2017 17:43:02 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id CB2C569D69 for ; Fri, 1 Dec 2017 17:43:01 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id 799C4843D18E for ; Fri, 1 Dec 2017 11:42:59 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CRaDdGSJZPUu for ; Fri, 1 Dec 2017 11:42:58 -0600 (CST) Received: from square.office.ebureau.com (unknown [10.10.21.22]) by internet06.ebureau.com (Postfix) with ESMTPSA id EC1F0843D181 for ; Fri, 1 Dec 2017 11:42:58 -0600 (CST) From: Dustin Wenz Content-Type: multipart/signed; boundary="Apple-Mail=_6D89BD5E-2557-48AD-BD16-688CF9BE4761"; protocol="application/pkcs7-signature"; micalg=sha1 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: bhyve uses all available memory during IO-intensive operations Date: Fri, 1 Dec 2017 11:42:58 -0600 References: To: freebsd-virtualization@freebsd.org In-Reply-To: Message-Id: <9C202C72-EF74-4DB7-9B2C-736C049A9F7A@ebureau.com> X-Mailer: Apple Mail (2.3445.4.7) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 17:43:02 -0000 --Apple-Mail=_6D89BD5E-2557-48AD-BD16-688CF9BE4761 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Here's the top -uS output from a test this morning: last pid: 57375; load averages: 8.29, 7.02, 4.05 = = = = up 38+22:19:14 = 11:28:25 68 processes: 2 running, 65 sleeping, 1 waiting CPU: 0.1% user, 0.0% nice, 40.4% system, 0.4% interrupt, 59.1% idle Mem: 2188K Active, 4K Inact, 62G Wired, 449M Free ARC: 7947M Total, 58M MFU, 3364M MRU, 1000M Anon, 2620M Header, 904M = Other 4070M Compressed, 4658M Uncompressed, 1.14:1 Ratio Swap: 112G Total, 78M Used, 112G Free, 4K In, 12K Out PID UID THR PRI NICE SIZE RES STATE C TIME WCPU = COMMAND 11 0 24 155 ki31 0K 384K RUN 0 ??? 1446.82% = idle 0 0 644 -16 - 0K 10304K swapin 21 554:59 492.45% = kernel 57333 0 30 20 0 17445M 1325M kqread 9 16:38 357.42% = bhyve 15 0 10 -8 - 0K 192K arc_re 20 80:54 81.55% = zfskern 5 0 6 -16 - 0K 96K - 5 12:35 11.50% cam 12 0 53 -60 - 0K 848K WAIT 21 74:35 9.40% intr 41094 0 30 20 0 17445M 14587M kqread 17 301:29 0.39% = bhyve Dec 1 11:29:31 service014 kernel: pid 57333 (bhyve), uid 0, = was killed: out of swap space Dec 1 11:29:31 service014 kernel: pid 69549 (bhyve), uid 0, = was killed: out of swap space Dec 1 11:29:31 service014 kernel: pid 41094 (bhyve), uid 0, = was killed: out of swap space This was with three VMs running, but only one of them was doing any IO. = Note that the whole machine hung for about about 60 seconds before the = VMs were shut down and memory recovered. That's why the top output is = over a minute older than the kill messages (top had stopped refreshing). What I'm suspicious of is that almost all of the physical memory is = wired. If that is bhyve memory, why did it not page out? - .Dustin > On Nov 30, 2017, at 5:15 PM, Dustin Wenz = wrote: >=20 > I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS = is also FreeBSD 11.1). Their sole purpose is to house some medium-sized = Postgres databases (100-200GB). The host system has 64GB of real memory = and 112GB of swap. I have configured each guest to only use 16GB of = memory, yet while doing my initial database imports in the VMs, bhyve = will quickly grow to use all available system memory and then be killed = by the kernel: >=20 > kernel: swap_pager: I/O error - pageout failed; blkno 1735,size = 4096, error 12 > kernel: swap_pager: I/O error - pageout failed; blkno 1610,size = 4096, error 12 > kernel: swap_pager: I/O error - pageout failed; blkno 1763,size = 4096, error 12 > kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space >=20 > The OOM condition seems related to doing moderate IO within the VM, = though nothing within the VM itself shows high memory usage. This is the = chyves config for one of them: >=20 > bargs -A -H -P -S > bhyve_disk_type virtio-blk > bhyve_net_type virtio-net > bhyveload_flags > chyves_guest_version 0300 > cpu 4 > creation Created on Mon Oct 23 16:17:04 CDT = 2017 by chyves v0.2.0 2016/09/11 using __create() > loader bhyveload > net_ifaces tap51 > os default > ram 16G > rcboot 0 > revert_to_snapshot > revert_to_snapshot_method off > serial nmdm51 > template no > uuid 8495a130-b837-11e7-b092-0025909a8b56 >=20 >=20 > I've also tried using different bhyve_disk_types, with no improvement. = How is it that bhyve can use far more memory that I'm specifying? >=20 > - .Dustin --Apple-Mail=_6D89BD5E-2557-48AD-BD16-688CF9BE4761 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIEzDCCBMgw ggOwoAMCAQICAUEwDQYJKoZIhvcNAQELBQAwgZkxCzAJBgNVBAYTAlVTMRIwEAYDVQQIEwlNaW5u ZXNvdGExFDASBgNVBAcTC1NhaW50IENsb3VkMRAwDgYDVQQKEwdlQnVyZWF1MRQwEgYDVQQLEwtJ bnRlZ3JhdGlvbjEUMBIGA1UEAxMLZWJ1cmVhdS5jb20xIjAgBgkqhkiG9w0BCQEWE3N1cHBvcnRA ZWJ1cmVhdS5jb20wHhcNMTcwNTA1MTYxNjE1WhcNMjcwNTAzMTYxNjE1WjBKMQswCQYDVQQGEwJV UzEUMBIGA1UEAwwLRHVzdGluIFdlbnoxJTAjBgkqhkiG9w0BCQEWFmR1c3RpbndlbnpAZWJ1cmVh dS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCQ/HJXe7JhUaexqEaxCNVifsue sUMgohgXLmi5YIcbAWhdxTr1PUzKYkeTkL9sYNjXU6uaI2tZMF3hA9gcFjxQIfkKSo31MrYOgMdU xQe0Q+t6Vd4pGAmtDQDwhAsrNGccADp3Yjy4eBtVfkDGdzz1Y8Lbc684TPFcW7i9+U/dDaXlcxeq fyDqiHZ5y8Lp/1M2Ot/Rz7eikJZTAuHOWKs/PEiJIM2JHuhPyNy+mL2oqEWeOcEsKMNzgn7HVt4k Xz2irBAG+cj4WAxWs418l46EEXgur4PvhBXZMl0LJg0TyaxOHbsUam4R4tbKnaZ3HhRkg79k2Had sb6DKbnCw9/1AgMBAAGjggFnMIIBYzAJBgNVHRMEAjAAMAsGA1UdDwQEAwIE8DAnBgNVHSUEIDAe BggrBgEFBQcDAQYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBTLi/8HUHpbBEt9OtPqQoax AmpaNDCBzgYDVR0jBIHGMIHDgBRnpZeXB5rQYLgsUKqiiBcLIHyu6aGBn6SBnDCBmTELMAkGA1UE BhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoT B2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAG CSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbYIJAMwZcjAWAsWXMDAGCWCGSAGG+EIBBAQj FiFodHRwOi8vd3d3LmVidXJlYXUuY29tL2NhLWNybC5wZW0wDQYJKoZIhvcNAQELBQADggEBAHbO qVdB9raUKXCgZRA/nES5a60dlIaGnIlpgz+Y3SjFt0bcJxoUYhIzumBHk9yjyP4M1DubOphkQpJ4 LNZbAS01cjCxjnC0ZUq5V3FCeaDwrn1qPY+QJGoZPLlhWdJUNu17OpnR7ZfBWlp3/pRhvNU5PCbJ nmF7rnvsqxUFq9oeiV3SmqBux5lwJ7p2Uss5SHSW6g17K/KdTMK1roQr/+rWpxp2233qddDrLpOE xGRlvhEqSa/IZbGC9oiYmsiaG1PefQkadoob5IMIS5/MDpWHUgSHqAj1V/LwcCx0rbt73SazGMND EzHVWhsj+khepB/MG5QGfWP23IGFmvQYWWcxggOQMIIDjAIBATCBnzCBmTELMAkGA1UEBhMCVVMx EjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoTB2VCdXJl YXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAGCSqGSIb3 DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTAJBgUrDgMCGgUAoIIBxTAYBgkqhkiG9w0BCQMx CwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNzEyMDExNzQyNThaMCMGCSqGSIb3DQEJBDEW BBSQmDVwsaLi/1A4nK4vJezoM3o3/zCBsAYJKwYBBAGCNxAEMYGiMIGfMIGZMQswCQYDVQQGEwJV UzESMBAGA1UECBMJTWlubmVzb3RhMRQwEgYDVQQHEwtTYWludCBDbG91ZDEQMA4GA1UEChMHZUJ1 cmVhdTEUMBIGA1UECxMLSW50ZWdyYXRpb24xFDASBgNVBAMTC2VidXJlYXUuY29tMSIwIAYJKoZI hvcNAQkBFhNzdXBwb3J0QGVidXJlYXUuY29tAgFBMIGyBgsqhkiG9w0BCRACCzGBoqCBnzCBmTEL MAkGA1UEBhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAO BgNVBAoTB2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNv bTEiMCAGCSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTANBgkqhkiG9w0BAQEFAASC AQCC585LUU2Fuhug7DjxhEWllg0NzDYIUP7o2gEPckgOiT5o0kc3peE6cgiMMZu+Is/wIE1PqzCn lyhOLOje49R7YADBFG+YgFLI5MlHn4P+iEVigja0Ksyu3idyQlPjtN/Fhv/LR6s4I1VTOX2HOhZQ UE5j/yPLKLs9bps6OAg5u+jk1G9bROsqAOFXeSgwZTSfYOGGgS+ijNQ868RT39Mea0RcOMYEQwJY qZUHPWIL+8dSUrrMgkgeCSs9fwiNVwAk35sLiz4YnYdnRyQPqbiA0J9g9aAHtAhoyBDn1UpOQo/X PE/ePATSAOyn1OMwmiQ+5OKoSnohnWQ0DfFvdDhTAAAAAAAA --Apple-Mail=_6D89BD5E-2557-48AD-BD16-688CF9BE4761-- From owner-freebsd-virtualization@freebsd.org Fri Dec 1 17:54:48 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7CD5DE66531 for ; Fri, 1 Dec 2017 17:54:48 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 426166A482 for ; Fri, 1 Dec 2017 17:54:47 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from Ticonderoga.HML3.ScaleEngine.net (senat1-01.HML3.ScaleEngine.net [209.51.186.5]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id A1C66139BF for ; Fri, 1 Dec 2017 17:54:46 +0000 (UTC) Subject: Re: bhyve uses all available memory during IO-intensive operations To: freebsd-virtualization@freebsd.org References: <9C202C72-EF74-4DB7-9B2C-736C049A9F7A@ebureau.com> From: Allan Jude Message-ID: <8626f53b-6296-b694-c29f-fde4329f3118@freebsd.org> Date: Fri, 1 Dec 2017 12:54:46 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.3.0 MIME-Version: 1.0 In-Reply-To: <9C202C72-EF74-4DB7-9B2C-736C049A9F7A@ebureau.com> Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 17:54:48 -0000 On 12/01/2017 12:42, Dustin Wenz wrote: > Here's the top -uS output from a test this morning: > > last pid: 57375; load averages: 8.29, 7.02, 4.05 up 38+22:19:14 11:28:25 > 68 processes: 2 running, 65 sleeping, 1 waiting > CPU: 0.1% user, 0.0% nice, 40.4% system, 0.4% interrupt, 59.1% idle > Mem: 2188K Active, 4K Inact, 62G Wired, 449M Free > ARC: 7947M Total, 58M MFU, 3364M MRU, 1000M Anon, 2620M Header, 904M Other > 4070M Compressed, 4658M Uncompressed, 1.14:1 Ratio > Swap: 112G Total, 78M Used, 112G Free, 4K In, 12K Out > > PID UID THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND > 11 0 24 155 ki31 0K 384K RUN 0 ??? 1446.82% idle > 0 0 644 -16 - 0K 10304K swapin 21 554:59 492.45% kernel > 57333 0 30 20 0 17445M 1325M kqread 9 16:38 357.42% bhyve > 15 0 10 -8 - 0K 192K arc_re 20 80:54 81.55% zfskern > 5 0 6 -16 - 0K 96K - 5 12:35 11.50% cam > 12 0 53 -60 - 0K 848K WAIT 21 74:35 9.40% intr > 41094 0 30 20 0 17445M 14587M kqread 17 301:29 0.39% bhyve > > Dec 1 11:29:31 service014 kernel: pid 57333 (bhyve), uid 0, was killed: out of swap space > Dec 1 11:29:31 service014 kernel: pid 69549 (bhyve), uid 0, was killed: out of swap space > Dec 1 11:29:31 service014 kernel: pid 41094 (bhyve), uid 0, was killed: out of swap space > > > This was with three VMs running, but only one of them was doing any IO. Note that the whole machine hung for about about 60 seconds before the VMs were shut down and memory recovered. That's why the top output is over a minute older than the kill messages (top had stopped refreshing). > > What I'm suspicious of is that almost all of the physical memory is wired. If that is bhyve memory, why did it not page out? > > > - .Dustin > > >> On Nov 30, 2017, at 5:15 PM, Dustin Wenz wrote: >> >> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS is also FreeBSD 11.1). Their sole purpose is to house some medium-sized Postgres databases (100-200GB). The host system has 64GB of real memory and 112GB of swap. I have configured each guest to only use 16GB of memory, yet while doing my initial database imports in the VMs, bhyve will quickly grow to use all available system memory and then be killed by the kernel: >> >> kernel: swap_pager: I/O error - pageout failed; blkno 1735,size 4096, error 12 >> kernel: swap_pager: I/O error - pageout failed; blkno 1610,size 4096, error 12 >> kernel: swap_pager: I/O error - pageout failed; blkno 1763,size 4096, error 12 >> kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space >> >> The OOM condition seems related to doing moderate IO within the VM, though nothing within the VM itself shows high memory usage. This is the chyves config for one of them: >> >> bargs -A -H -P -S >> bhyve_disk_type virtio-blk >> bhyve_net_type virtio-net >> bhyveload_flags >> chyves_guest_version 0300 >> cpu 4 >> creation Created on Mon Oct 23 16:17:04 CDT 2017 by chyves v0.2.0 2016/09/11 using __create() >> loader bhyveload >> net_ifaces tap51 >> os default >> ram 16G >> rcboot 0 >> revert_to_snapshot >> revert_to_snapshot_method off >> serial nmdm51 >> template no >> uuid 8495a130-b837-11e7-b092-0025909a8b56 >> >> >> I've also tried using different bhyve_disk_types, with no improvement. How is it that bhyve can use far more memory that I'm specifying? >> >> - .Dustin > 'Wired' memory, specifically means that it cannot be paged out. It is not bhyve, it is ZFS. Please lower your vfs.zfs.arc_max to about 10-12 GB instead of 60+ GB it is at now. And you might also want to double your: vfs.zfs.arc_free_target to make ZFS give up memory more easily. -- Allan Jude From owner-freebsd-virtualization@freebsd.org Fri Dec 1 17:56:50 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 53A40E6663A for ; Fri, 1 Dec 2017 17:56:50 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto.OntheNet.com.au [203.13.68.12]) by mx1.freebsd.org (Postfix) with ESMTP id 0BB796A573 for ; Fri, 1 Dec 2017 17:56:49 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id AF453219438A for ; Sat, 2 Dec 2017 03:56:41 +1000 (AEST) Received: from localhost (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id A9BBC2820A8 for ; Sat, 2 Dec 2017 03:56:41 +1000 (AEST) X-Amavis-Modified: Mail body modified (using disclaimer) - iredmail.onthenet.com.au Received: from iredmail.onthenet.com.au ([127.0.0.1]) by localhost (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 3I5J8hvCgJIN for ; Sat, 2 Dec 2017 03:56:41 +1000 (AEST) Received: from Peters-MacBook-Pro-2.local (c-67-180-92-13.hsd1.ca.comcast.net [67.180.92.13]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 9DAC72809F4; Sat, 2 Dec 2017 03:56:39 +1000 (AEST) Subject: Re: bhyve uses all available memory during IO-intensive operations To: Dustin Wenz References: <9C202C72-EF74-4DB7-9B2C-736C049A9F7A@ebureau.com> Cc: freebsd-virtualization@freebsd.org From: Peter Grehan Message-ID: Date: Fri, 1 Dec 2017 09:56:36 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.10; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <9C202C72-EF74-4DB7-9B2C-736C049A9F7A@ebureau.com> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.2 cv=XtWKARN9 c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=5eVCmCvhg37cu/pjidAGzw==:17 a=N659UExz7-8A:10 a=ocR9PWop10UA:10 a=9l57FugCJxmoHp07MWYA:9 a=pILNOxqGKmIA:10 wl=host:3 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 17:56:50 -0000 >> bargs -A -H -P -S The -S flag to bhyve wires guest memory so it won't be swapped out. later, Peter. From owner-freebsd-virtualization@freebsd.org Fri Dec 1 18:09:11 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 72DF8E66AAE for ; Fri, 1 Dec 2017 18:09:11 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3434F6AD3E; Fri, 1 Dec 2017 18:09:11 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id BDAEF843D963; Fri, 1 Dec 2017 12:09:09 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id o4uhnPRGImW2; Fri, 1 Dec 2017 12:09:09 -0600 (CST) Received: from square.office.ebureau.com (unknown [10.10.21.22]) by internet06.ebureau.com (Postfix) with ESMTPSA id 84318843D956; Fri, 1 Dec 2017 12:09:09 -0600 (CST) From: Dustin Wenz Message-Id: <22BC8832-924A-480B-A9A0-DB717D37BF08@ebureau.com> Content-Type: multipart/signed; boundary="Apple-Mail=_795FD86B-D743-431E-89A2-87C7103A9392"; protocol="application/pkcs7-signature"; micalg=sha1 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: bhyve uses all available memory during IO-intensive operations Date: Fri, 1 Dec 2017 12:09:08 -0600 In-Reply-To: Cc: freebsd-virtualization@freebsd.org To: Peter Grehan References: <9C202C72-EF74-4DB7-9B2C-736C049A9F7A@ebureau.com> X-Mailer: Apple Mail (2.3445.4.7) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 18:09:11 -0000 --Apple-Mail=_795FD86B-D743-431E-89A2-87C7103A9392 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Yep, and that's also why bhyve is getting killed instead of paging out. = For some inexplicable reason, chyves defaulted to setting -S on new VMs. = That has the effect of wiring in the max amount of memory for each guest = at startup. I changed the bargs option to "-A -H -P" instead of "-A -H -P -S". = Memory pressure is greatly alleviated upon restart. I'm going to do more = testing, but I suspect this will fix my problem. Take this as a PSA for = chyves users. - .Dustin > On Dec 1, 2017, at 11:56 AM, Peter Grehan wrote: >=20 > The -S flag to bhyve wires guest memory so it won't be swapped out. >=20 > later, >=20 > Peter. --Apple-Mail=_795FD86B-D743-431E-89A2-87C7103A9392 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIEzDCCBMgw ggOwoAMCAQICAUEwDQYJKoZIhvcNAQELBQAwgZkxCzAJBgNVBAYTAlVTMRIwEAYDVQQIEwlNaW5u ZXNvdGExFDASBgNVBAcTC1NhaW50IENsb3VkMRAwDgYDVQQKEwdlQnVyZWF1MRQwEgYDVQQLEwtJ bnRlZ3JhdGlvbjEUMBIGA1UEAxMLZWJ1cmVhdS5jb20xIjAgBgkqhkiG9w0BCQEWE3N1cHBvcnRA ZWJ1cmVhdS5jb20wHhcNMTcwNTA1MTYxNjE1WhcNMjcwNTAzMTYxNjE1WjBKMQswCQYDVQQGEwJV UzEUMBIGA1UEAwwLRHVzdGluIFdlbnoxJTAjBgkqhkiG9w0BCQEWFmR1c3RpbndlbnpAZWJ1cmVh dS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCQ/HJXe7JhUaexqEaxCNVifsue sUMgohgXLmi5YIcbAWhdxTr1PUzKYkeTkL9sYNjXU6uaI2tZMF3hA9gcFjxQIfkKSo31MrYOgMdU xQe0Q+t6Vd4pGAmtDQDwhAsrNGccADp3Yjy4eBtVfkDGdzz1Y8Lbc684TPFcW7i9+U/dDaXlcxeq fyDqiHZ5y8Lp/1M2Ot/Rz7eikJZTAuHOWKs/PEiJIM2JHuhPyNy+mL2oqEWeOcEsKMNzgn7HVt4k Xz2irBAG+cj4WAxWs418l46EEXgur4PvhBXZMl0LJg0TyaxOHbsUam4R4tbKnaZ3HhRkg79k2Had sb6DKbnCw9/1AgMBAAGjggFnMIIBYzAJBgNVHRMEAjAAMAsGA1UdDwQEAwIE8DAnBgNVHSUEIDAe BggrBgEFBQcDAQYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBTLi/8HUHpbBEt9OtPqQoax AmpaNDCBzgYDVR0jBIHGMIHDgBRnpZeXB5rQYLgsUKqiiBcLIHyu6aGBn6SBnDCBmTELMAkGA1UE BhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoT B2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAG CSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbYIJAMwZcjAWAsWXMDAGCWCGSAGG+EIBBAQj FiFodHRwOi8vd3d3LmVidXJlYXUuY29tL2NhLWNybC5wZW0wDQYJKoZIhvcNAQELBQADggEBAHbO qVdB9raUKXCgZRA/nES5a60dlIaGnIlpgz+Y3SjFt0bcJxoUYhIzumBHk9yjyP4M1DubOphkQpJ4 LNZbAS01cjCxjnC0ZUq5V3FCeaDwrn1qPY+QJGoZPLlhWdJUNu17OpnR7ZfBWlp3/pRhvNU5PCbJ nmF7rnvsqxUFq9oeiV3SmqBux5lwJ7p2Uss5SHSW6g17K/KdTMK1roQr/+rWpxp2233qddDrLpOE xGRlvhEqSa/IZbGC9oiYmsiaG1PefQkadoob5IMIS5/MDpWHUgSHqAj1V/LwcCx0rbt73SazGMND EzHVWhsj+khepB/MG5QGfWP23IGFmvQYWWcxggOQMIIDjAIBATCBnzCBmTELMAkGA1UEBhMCVVMx EjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoTB2VCdXJl YXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAGCSqGSIb3 DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTAJBgUrDgMCGgUAoIIBxTAYBgkqhkiG9w0BCQMx CwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNzEyMDExODA5MDlaMCMGCSqGSIb3DQEJBDEW BBTMExP8X1N53zljBQuG/a+Ux9IsvTCBsAYJKwYBBAGCNxAEMYGiMIGfMIGZMQswCQYDVQQGEwJV UzESMBAGA1UECBMJTWlubmVzb3RhMRQwEgYDVQQHEwtTYWludCBDbG91ZDEQMA4GA1UEChMHZUJ1 cmVhdTEUMBIGA1UECxMLSW50ZWdyYXRpb24xFDASBgNVBAMTC2VidXJlYXUuY29tMSIwIAYJKoZI hvcNAQkBFhNzdXBwb3J0QGVidXJlYXUuY29tAgFBMIGyBgsqhkiG9w0BCRACCzGBoqCBnzCBmTEL MAkGA1UEBhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAO BgNVBAoTB2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNv bTEiMCAGCSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTANBgkqhkiG9w0BAQEFAASC AQBe4sGYoqFXj7bcrIvsRzNWRjKNrknntyj4mfepSW2OiKvEa0XQjix84sHtrk39MEFMvFWYMBHP IQZPuR4653abSp49qnOwhoruJEJLKg+qasaVRf2mFJdr4+JPKZ0V7ZRHcGJkd7el8bhgt80XyJC2 ECyUbstJCT/HU1Ji3jMPct1TvlbOFORDjLrZ9uIBfNd/U8+GgxXjs0FZ9EONUWXLfsTYUx7b2ojg Drfb/lyCN3JpWNuE5eHquHirVlBm/fvxXoCvHnfjIWyk2efNIwFmMq34KJWhkXTQ7gNTakYajiaI PajYmVmdt5HnkSSC9iNyNlpiuzYo9bgXvtI2uEk8AAAAAAAA --Apple-Mail=_795FD86B-D743-431E-89A2-87C7103A9392-- From owner-freebsd-virtualization@freebsd.org Fri Dec 1 21:41:54 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BFB98E6BC22 for ; Fri, 1 Dec 2017 21:41:54 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 96C39720E8; Fri, 1 Dec 2017 21:41:53 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id 3A12B8440D34; Fri, 1 Dec 2017 15:41:52 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Lo6y8p65e7FA; Fri, 1 Dec 2017 15:41:51 -0600 (CST) Received: from square.office.ebureau.com (unknown [10.10.21.22]) by internet06.ebureau.com (Postfix) with ESMTPSA id 81A728440D21; Fri, 1 Dec 2017 15:41:51 -0600 (CST) From: Dustin Wenz Message-Id: <9897C115-4212-4161-811D-E71B0CAE911A@ebureau.com> Content-Type: multipart/signed; boundary="Apple-Mail=_A006E473-222E-40BF-B0DF-417C3F9B9DE7"; protocol="application/pkcs7-signature"; micalg=sha1 Mime-Version: 1.0 (Mac OS X Mail 11.1 \(3445.4.7\)) Subject: Re: bhyve uses all available memory during IO-intensive operations Date: Fri, 1 Dec 2017 15:41:50 -0600 In-Reply-To: <22BC8832-924A-480B-A9A0-DB717D37BF08@ebureau.com> Cc: freebsd-virtualization@freebsd.org To: Peter Grehan References: <9C202C72-EF74-4DB7-9B2C-736C049A9F7A@ebureau.com> <22BC8832-924A-480B-A9A0-DB717D37BF08@ebureau.com> X-Mailer: Apple Mail (2.3445.4.7) X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Dec 2017 21:41:54 -0000 --Apple-Mail=_A006E473-222E-40BF-B0DF-417C3F9B9DE7 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii I've been running a database stress test on my VMs for the last few = hours without issue, and I've noticed no unexpected memory usage. Prior = to changing the wired option, this would never have run as long. I = haven't limited the ARC size yet, but I probably will since it sounds = like best practice for a bhyve host. The commit history shows that chyves defaults to -S if you are hosting = from FreeBSD 10.3 or later. I'm sure they had a reason for doing that, = but I don't know what that would be. It seems to an inefficient use of = main memory if you need to run a lot of VMs. Thanks everyone for helping to nail this down! - .Dustin > On Dec 1, 2017, at 12:09 PM, Dustin Wenz = wrote: >=20 > Yep, and that's also why bhyve is getting killed instead of paging = out. For some inexplicable reason, chyves defaulted to setting -S on new = VMs. That has the effect of wiring in the max amount of memory for each = guest at startup. >=20 > I changed the bargs option to "-A -H -P" instead of "-A -H -P -S". = Memory pressure is greatly alleviated upon restart. I'm going to do more = testing, but I suspect this will fix my problem. Take this as a PSA for = chyves users. >=20 > - .Dustin >=20 >> On Dec 1, 2017, at 11:56 AM, Peter Grehan wrote: >>=20 >> The -S flag to bhyve wires guest memory so it won't be swapped out. >>=20 >> later, >>=20 >> Peter. >=20 --Apple-Mail=_A006E473-222E-40BF-B0DF-417C3F9B9DE7 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIEzDCCBMgw ggOwoAMCAQICAUEwDQYJKoZIhvcNAQELBQAwgZkxCzAJBgNVBAYTAlVTMRIwEAYDVQQIEwlNaW5u ZXNvdGExFDASBgNVBAcTC1NhaW50IENsb3VkMRAwDgYDVQQKEwdlQnVyZWF1MRQwEgYDVQQLEwtJ bnRlZ3JhdGlvbjEUMBIGA1UEAxMLZWJ1cmVhdS5jb20xIjAgBgkqhkiG9w0BCQEWE3N1cHBvcnRA ZWJ1cmVhdS5jb20wHhcNMTcwNTA1MTYxNjE1WhcNMjcwNTAzMTYxNjE1WjBKMQswCQYDVQQGEwJV UzEUMBIGA1UEAwwLRHVzdGluIFdlbnoxJTAjBgkqhkiG9w0BCQEWFmR1c3RpbndlbnpAZWJ1cmVh dS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCQ/HJXe7JhUaexqEaxCNVifsue sUMgohgXLmi5YIcbAWhdxTr1PUzKYkeTkL9sYNjXU6uaI2tZMF3hA9gcFjxQIfkKSo31MrYOgMdU xQe0Q+t6Vd4pGAmtDQDwhAsrNGccADp3Yjy4eBtVfkDGdzz1Y8Lbc684TPFcW7i9+U/dDaXlcxeq fyDqiHZ5y8Lp/1M2Ot/Rz7eikJZTAuHOWKs/PEiJIM2JHuhPyNy+mL2oqEWeOcEsKMNzgn7HVt4k Xz2irBAG+cj4WAxWs418l46EEXgur4PvhBXZMl0LJg0TyaxOHbsUam4R4tbKnaZ3HhRkg79k2Had sb6DKbnCw9/1AgMBAAGjggFnMIIBYzAJBgNVHRMEAjAAMAsGA1UdDwQEAwIE8DAnBgNVHSUEIDAe BggrBgEFBQcDAQYIKwYBBQUHAwIGCCsGAQUFBwMEMB0GA1UdDgQWBBTLi/8HUHpbBEt9OtPqQoax AmpaNDCBzgYDVR0jBIHGMIHDgBRnpZeXB5rQYLgsUKqiiBcLIHyu6aGBn6SBnDCBmTELMAkGA1UE BhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoT B2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAG CSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbYIJAMwZcjAWAsWXMDAGCWCGSAGG+EIBBAQj FiFodHRwOi8vd3d3LmVidXJlYXUuY29tL2NhLWNybC5wZW0wDQYJKoZIhvcNAQELBQADggEBAHbO qVdB9raUKXCgZRA/nES5a60dlIaGnIlpgz+Y3SjFt0bcJxoUYhIzumBHk9yjyP4M1DubOphkQpJ4 LNZbAS01cjCxjnC0ZUq5V3FCeaDwrn1qPY+QJGoZPLlhWdJUNu17OpnR7ZfBWlp3/pRhvNU5PCbJ nmF7rnvsqxUFq9oeiV3SmqBux5lwJ7p2Uss5SHSW6g17K/KdTMK1roQr/+rWpxp2233qddDrLpOE xGRlvhEqSa/IZbGC9oiYmsiaG1PefQkadoob5IMIS5/MDpWHUgSHqAj1V/LwcCx0rbt73SazGMND EzHVWhsj+khepB/MG5QGfWP23IGFmvQYWWcxggOQMIIDjAIBATCBnzCBmTELMAkGA1UEBhMCVVMx EjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAOBgNVBAoTB2VCdXJl YXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNvbTEiMCAGCSqGSIb3 DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTAJBgUrDgMCGgUAoIIBxTAYBgkqhkiG9w0BCQMx CwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0xNzEyMDEyMTQxNTFaMCMGCSqGSIb3DQEJBDEW BBS7mGFJFlVztlpS1h8ZB1BaUVSzJTCBsAYJKwYBBAGCNxAEMYGiMIGfMIGZMQswCQYDVQQGEwJV UzESMBAGA1UECBMJTWlubmVzb3RhMRQwEgYDVQQHEwtTYWludCBDbG91ZDEQMA4GA1UEChMHZUJ1 cmVhdTEUMBIGA1UECxMLSW50ZWdyYXRpb24xFDASBgNVBAMTC2VidXJlYXUuY29tMSIwIAYJKoZI hvcNAQkBFhNzdXBwb3J0QGVidXJlYXUuY29tAgFBMIGyBgsqhkiG9w0BCRACCzGBoqCBnzCBmTEL MAkGA1UEBhMCVVMxEjAQBgNVBAgTCU1pbm5lc290YTEUMBIGA1UEBxMLU2FpbnQgQ2xvdWQxEDAO BgNVBAoTB2VCdXJlYXUxFDASBgNVBAsTC0ludGVncmF0aW9uMRQwEgYDVQQDEwtlYnVyZWF1LmNv bTEiMCAGCSqGSIb3DQEJARYTc3VwcG9ydEBlYnVyZWF1LmNvbQIBQTANBgkqhkiG9w0BAQEFAASC AQBmWSFE50gh4QkV0/TvmopgVz4EydlcFXFjWXapwX9caeSn0T8Y/nbhs7uB70xABBbvZZ6/nHCE nlSZc9qPpEoYfe0O8tn+H6cw2r5U1r7j/jc0DlK4pzzpb/fc6yb4RU/I8ke9Jn+cKY6Lhu7BaXzU 8rizyFhYvYrE1HlPLH9krCrPb0GJgZ5kASvaufC1ObKwSjt2KJkUkQE0xioyy0whM7oTNQmCkC4S NI2JvrzaWQWZRxY2KISED2PNKbIn+rEIamWPvnS4Gu4wTLHd3p/8QlKUC0nJvYETf1wAQwdRrMSt J/HYTG08nIxPMK1Dmeyi1/Mws8A9b+p+5JbRfO29AAAAAAAA --Apple-Mail=_A006E473-222E-40BF-B0DF-417C3F9B9DE7-- From owner-freebsd-virtualization@freebsd.org Sat Dec 2 02:50:46 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C90FADEED0F for ; Sat, 2 Dec 2017 02:50:46 +0000 (UTC) (envelope-from FreeBSD@shaneware.biz) Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by mx1.freebsd.org (Postfix) with ESMTP id 655B27D81B for ; Sat, 2 Dec 2017 02:50:45 +0000 (UTC) (envelope-from FreeBSD@shaneware.biz) Received: from ppp121-45-11-215.bras1.adl4.internode.on.net (HELO leader.local) ([121.45.11.215]) by ipmail07.adl2.internode.on.net with ESMTP; 02 Dec 2017 13:20:43 +1030 Subject: Re: bhyve uses all available memory during IO-intensive operations To: Dustin Wenz Cc: freebsd-virtualization@freebsd.org References: <9C202C72-EF74-4DB7-9B2C-736C049A9F7A@ebureau.com> <22BC8832-924A-480B-A9A0-DB717D37BF08@ebureau.com> <9897C115-4212-4161-811D-E71B0CAE911A@ebureau.com> From: Shane Ambler Message-ID: <5bb5649f-5785-7baf-6871-625d1f63bd8b@ShaneWare.Biz> Date: Sat, 2 Dec 2017 13:20:41 +1030 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: <9897C115-4212-4161-811D-E71B0CAE911A@ebureau.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-AU Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 02:50:46 -0000 On 02/12/2017 08:11, Dustin Wenz wrote: > > The commit history shows that chyves defaults to -S if you are > hosting from FreeBSD 10.3 or later. I'm sure they had a reason for > doing that, but I don't know what that would be. It seems to an > inefficient use of main memory if you need to run a lot of VMs. It sounds like a reasonable solution to a problem. If host memory is full it swaps some out, so a bhyve might have free mem but some could be swapped out by the host. If the bhyve is out of mem, it's system swaps to it's disk, so the host swaps it back in so that the bhyve can then swap it to its disk... Wiring bhyve ram might be a reasonable solution as long as the hosts physical ram isn't over allocated by bhyve guests. The best solution would involve a host and guest talking to each other about used mem, but that would break the whole virtual machine illusion. At the least it would involve a system telling the hardware what memory is used and what is not, which just isn't something any system does. Maybe that is an idea for the vm guest aware systems of the future. -- FreeBSD - the place to B...Sharing Devices Shane Ambler From owner-freebsd-virtualization@freebsd.org Sat Dec 2 03:18:39 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18F76DF02F7 for ; Sat, 2 Dec 2017 03:18:39 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-oi0-x22a.google.com (mail-oi0-x22a.google.com [IPv6:2607:f8b0:4003:c06::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D8C3C7E86F for ; Sat, 2 Dec 2017 03:18:38 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-oi0-x22a.google.com with SMTP id r63so8469177oia.6 for ; Fri, 01 Dec 2017 19:18:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc:content-transfer-encoding; bh=Cj0OC29vWntwxTcp2tw1Fio7AenCU1B+BsjxoBeUe2I=; b=Up89FjTFFT9tG+X7ZHULswAQxsdoL16fya9DPl7y6h87JcuOoUTfH8cYIHTvdxfgxM iqLJ+ittxxuNCVuGJXD88EiRmRDZh4OGG4JMpeyEubQscdibyfM0AUaQDSAvTqgQbyfN X+DB4CM8798d/0lfRUGaxR1KIk3/osuTmwM2aWeu6YS9/mOc7F2TmxU1X/2XuIMA/l41 y8SI0Z9/9HqThWZMVVeV6UOZ3ScXEMNhSxjZmVCps/9SgXm58sKGDF3yydhN5xkW9GO9 bSn70vsiFcyTU7lhpOuTdmIEdE56n1xuO9L9hq0QqFRuwZZVwoexGiUpIrW7bflVdv1j aHCQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc:content-transfer-encoding; bh=Cj0OC29vWntwxTcp2tw1Fio7AenCU1B+BsjxoBeUe2I=; b=ZStUJmZDlafypn8+YHPHQ95HDOjraCeorFiWxFFyUP1iobHurXiLfosflUm4VocBe7 5lIAw4Bm66KnjT3i8D4/mYw3ZOBG/g0xygXj/JD6CL4EPGUDcWSaJNNx69aP6Epbvbnt 9HstIkgUPdEYyujeAM6SbAFE0EkyNemNsspYwiHIdrtsgLSR02VrCeYRPt+uVcKch04K Hgmxo1yg0jTseRSsG2Djn1euGAnoZGhMwbvuH8a2x2OBF9ewHt5/PvTd3Fuyruoo7V0i 0f3u/cEmI68NYRcTzaXgtu6UEJgajzRPz7h7//o4k2TC20Fk+JtwJqDIbqDof9V6+V8k XBVw== X-Gm-Message-State: AJaThX6vNs0FBU8FoJUDH9VK0bK3GYiC2tRiXR5GA5IKnv0QTz8h3YVa IfPijzqySLxifMtjfiqR3jXqGAgck6eleCqqOCNFSw== X-Google-Smtp-Source: AGs4zMZsubcc0tcSnvYd3XWq0bnqTLsr7SiF2U7DffKWTK2P/lVW+AqLSgsn7wAXpknM5SNIhml2SPd2a+dOpnrDzXo= X-Received: by 10.202.231.197 with SMTP id e188mr9073377oih.131.1512184717581; Fri, 01 Dec 2017 19:18:37 -0800 (PST) MIME-Version: 1.0 Sender: kmacybsd@gmail.com Received: by 10.157.66.238 with HTTP; Fri, 1 Dec 2017 19:18:37 -0800 (PST) In-Reply-To: References: From: "K. Macy" Date: Fri, 1 Dec 2017 19:18:37 -0800 X-Google-Sender-Auth: 6UPeOrhOc8K8r7_zjfS9yjVqm8k Message-ID: Subject: Re: bhyve uses all available memory during IO-intensive operations To: Dustin Wenz Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 03:18:39 -0000 One thing to watch out for with chyves if your virtual disk is more than 20G is the fact that it uses 512 byte blocks for the zvols it creates. I ended up using up 1.4TB only half filling up a 250G zvol. Chyves is quick and easy, but it's not exactly production ready. -M On Thu, Nov 30, 2017 at 3:15 PM, Dustin Wenz wrote= : > I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS is= also FreeBSD 11.1). Their sole purpose is to house some medium-sized Postg= res databases (100-200GB). The host system has 64GB of real memory and 112G= B of swap. I have configured each guest to only use 16GB of memory, yet whi= le doing my initial database imports in the VMs, bhyve will quickly grow to= use all available system memory and then be killed by the kernel: > > kernel: swap_pager: I/O error - pageout failed; blkno 1735,size 4= 096, error 12 > kernel: swap_pager: I/O error - pageout failed; blkno 1610,size 4= 096, error 12 > kernel: swap_pager: I/O error - pageout failed; blkno 1763,size 4= 096, error 12 > kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space > > The OOM condition seems related to doing moderate IO within the VM, thoug= h nothing within the VM itself shows high memory usage. This is the chyves = config for one of them: > > bargs -A -H -P -S > bhyve_disk_type virtio-blk > bhyve_net_type virtio-net > bhyveload_flags > chyves_guest_version 0300 > cpu 4 > creation Created on Mon Oct 23 16:17:04 CDT 201= 7 by chyves v0.2.0 2016/09/11 using __create() > loader bhyveload > net_ifaces tap51 > os default > ram 16G > rcboot 0 > revert_to_snapshot > revert_to_snapshot_method off > serial nmdm51 > template no > uuid 8495a130-b837-11e7-b092-0025909a8b56 > > > I've also tried using different bhyve_disk_types, with no improvement. Ho= w is it that bhyve can use far more memory that I'm specifying? > > - .Dustin From owner-freebsd-virtualization@freebsd.org Sat Dec 2 04:02:34 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7F075DF1992 for ; Sat, 2 Dec 2017 04:02:34 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 643F77FCCB for ; Sat, 2 Dec 2017 04:02:33 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id vB242G4e032506; Fri, 1 Dec 2017 20:02:16 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id vB242DaJ032505; Fri, 1 Dec 2017 20:02:13 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201712020402.vB242DaJ032505@pdx.rh.CN85.dnsmgr.net> Subject: Re: bhyve uses all available memory during IO-intensive operations In-Reply-To: <5bb5649f-5785-7baf-6871-625d1f63bd8b@ShaneWare.Biz> To: Shane Ambler Date: Fri, 1 Dec 2017 20:02:13 -0800 (PST) CC: Dustin Wenz , freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 04:02:34 -0000 > On 02/12/2017 08:11, Dustin Wenz wrote: > > > > The commit history shows that chyves defaults to -S if you are > > hosting from FreeBSD 10.3 or later. I'm sure they had a reason for > > doing that, but I don't know what that would be. It seems to an > > inefficient use of main memory if you need to run a lot of VMs. > > It sounds like a reasonable solution to a problem. If host memory is > full it swaps some out, so a bhyve might have free mem but some could be > swapped out by the host. If the bhyve is out of mem, it's system swaps > to it's disk, so the host swaps it back in so that the bhyve can then > swap it to its disk... > > Wiring bhyve ram might be a reasonable solution as long as the hosts > physical ram isn't over allocated by bhyve guests. > > The best solution would involve a host and guest talking to each other > about used mem, but that would break the whole virtual machine illusion. > At the least it would involve a system telling the hardware what memory > is used and what is not, which just isn't something any system does. > Maybe that is an idea for the vm guest aware systems of the future. Its actually old technology, its called the memory balloon driver, but bhyve does not have that functionality, yet. -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Sat Dec 2 05:10:23 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3FF7BDF3438 for ; Sat, 2 Dec 2017 05:10:23 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-oi0-f54.google.com (mail-oi0-f54.google.com [209.85.218.54]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 109491ABC for ; Sat, 2 Dec 2017 05:10:22 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-oi0-f54.google.com with SMTP id t81so8531208oih.13 for ; Fri, 01 Dec 2017 21:10:22 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=Gqy4fuOVf6a9tYoqEELBndGiioGEknKXz38KZOnjBAM=; b=XMkA2hJfgT0fgZ7B7U+mUJwF+qQsWfgn4P+KFE0snNyUmR1Bh2ORm/1vVUNUkmSXS0 LymQdhpmUYQCOkwy1fn1MOLc6wCGbrheNRnvLHaI/s9P4bb68tBxGP97TOwWqq8T9II6 lFTarrCzyjwN3ELIFnMu4Z2LrboLqYKVoqedMHB0m6WdRWNGaX7mCxZ8NLgJVlqMgfTc wFUSq6wZaflvXu75A7eavSOOGSQl6pvysv5BhnlBVLEKi8cImfQ9hUMeI8U19gMNqh6R eyoG2/bAoa0IhjAPIiWBglKvSOMiK14SGz1XUfgDnFLyXGowgubp7/RaBZZYPAYgg4NO YCmA== X-Gm-Message-State: AJaThX775RaOz4job9GXYumqfJQxosnrhX2RoDROfZG3fIz9ly0uh9ek D3F+Z+jYy8VJiDKyUGdN9b+c+gnV72sLgXDFcos= X-Google-Smtp-Source: AGs4zMbZGv5kbBziM9cNT7WlP9tu9TAKigb2VnGYcnmzc+nwgRA5fu/avvP8RBt0Pdxs4Yv1RZYlTFNj4ox+9VjHPJI= X-Received: by 10.202.191.6 with SMTP id p6mr5356365oif.311.1512191416124; Fri, 01 Dec 2017 21:10:16 -0800 (PST) MIME-Version: 1.0 References: <5bb5649f-5785-7baf-6871-625d1f63bd8b@ShaneWare.Biz> <201712020402.vB242DaJ032505@pdx.rh.CN85.dnsmgr.net> In-Reply-To: <201712020402.vB242DaJ032505@pdx.rh.CN85.dnsmgr.net> From: "K. Macy" Date: Sat, 02 Dec 2017 05:10:05 +0000 Message-ID: Subject: Re: bhyve uses all available memory during IO-intensive operations To: "Rodney W. Grimes" Cc: Shane Ambler , freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 05:10:23 -0000 On Fri, Dec 1, 2017 at 20:02 Rodney W. Grimes < freebsd-rwg@pdx.rh.cn85.dnsmgr.net> wrote: > > On 02/12/2017 08:11, Dustin Wenz wrote: > > > > > > The commit history shows that chyves defaults to -S if you are > > > hosting from FreeBSD 10.3 or later. I'm sure they had a reason for > > > doing that, but I don't know what that would be. It seems to an > > > inefficient use of main memory if you need to run a lot of VMs. > > > > It sounds like a reasonable solution to a problem. If host memory is > > full it swaps some out, so a bhyve might have free mem but some could be > > swapped out by the host. If the bhyve is out of mem, it's system swaps > > to it's disk, so the host swaps it back in so that the bhyve can then > > swap it to its disk... > > > > Wiring bhyve ram might be a reasonable solution as long as the hosts > > physical ram isn't over allocated by bhyve guests. > > > > The best solution would involve a host and guest talking to each other > > about used mem, but that would break the whole virtual machine illusion. > > At the least it would involve a system telling the hardware what memory > > is used and what is not, which just isn't something any system does. > > Maybe that is an idea for the vm guest aware systems of the future. > > Its actually old technology, its called the memory balloon driver, > but bhyve does not have that functionality, yet. > > The virtio ballon driver is already there. Implementing a kernel backend for it would be trivial. In kernel virtio-net and virtio-p9fs backends are already well underway. -M > -- > Rod Grimes > rgrimes@freebsd.org > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to " > freebsd-virtualization-unsubscribe@freebsd.org" > From owner-freebsd-virtualization@freebsd.org Sat Dec 2 05:23:39 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABCC2DF3BEF for ; Sat, 2 Dec 2017 05:23:39 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from internet06.ebureau.com (internet06.ebureau.com [65.127.24.25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "internet06.ebureau.com", Issuer "internet06.ebureau.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 86AEA21AF; Sat, 2 Dec 2017 05:23:39 +0000 (UTC) (envelope-from dustinwenz@ebureau.com) Received: from localhost (localhost [127.0.0.1]) by internet06.ebureau.com (Postfix) with ESMTP id 08FB98444930; Fri, 1 Dec 2017 23:23:32 -0600 (CST) X-Virus-Scanned: amavisd-new at mydomain = ebureau.com Received: from internet06.ebureau.com ([127.0.0.1]) by localhost (internet06.ebureau.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id KVwM3CeZqom9; Fri, 1 Dec 2017 23:23:31 -0600 (CST) Received: from [172.16.3.163] (unknown [67.148.93.60]) by internet06.ebureau.com (Postfix) with ESMTPSA id 04C0C844491A; Fri, 1 Dec 2017 23:23:30 -0600 (CST) Mime-Version: 1.0 (1.0) Subject: Re: bhyve uses all available memory during IO-intensive operations From: Dustin Wenz X-Mailer: iPhone Mail (15B202) In-Reply-To: Date: Fri, 1 Dec 2017 23:23:28 -0600 Cc: "freebsd-virtualization@freebsd.org" Message-Id: <59DFCE5F-029F-4585-B0BA-8FABC43357F2@ebureau.com> References: To: "K. Macy" Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 05:23:39 -0000 I have noticed significant storage amplification for my zvols; that could ve= ry well be the reason. I would like to know more about why it happens.=20 Since the volblocksize is 512 bytes, I certainly expect extra cpu overhead (= and maybe an extra 1k or so worth of checksums for each 128k block in the vm= ), but how do you get a 10X expansion in stored data? What is the recommended zvol block size for a FreeBSD/ZFS guest? Perhaps 4k,= to match the most common mass storage sector size? - .Dustin > On Dec 1, 2017, at 9:18 PM, K. Macy wrote: >=20 > One thing to watch out for with chyves if your virtual disk is more > than 20G is the fact that it uses 512 byte blocks for the zvols it > creates. I ended up using up 1.4TB only half filling up a 250G zvol. > Chyves is quick and easy, but it's not exactly production ready. >=20 > -M >=20 >=20 >=20 >> On Thu, Nov 30, 2017 at 3:15 PM, Dustin Wenz wro= te: >> I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS is= also FreeBSD 11.1). Their sole purpose is to house some medium-sized Postgr= es databases (100-200GB). The host system has 64GB of real memory and 112GB o= f swap. I have configured each guest to only use 16GB of memory, yet while d= oing my initial database imports in the VMs, bhyve will quickly grow to use a= ll available system memory and then be killed by the kernel: >>=20 >> kernel: swap_pager: I/O error - pageout failed; blkno 1735,size 40= 96, error 12 >> kernel: swap_pager: I/O error - pageout failed; blkno 1610,size 40= 96, error 12 >> kernel: swap_pager: I/O error - pageout failed; blkno 1763,size 40= 96, error 12 >> kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space >>=20 >> The OOM condition seems related to doing moderate IO within the VM, thoug= h nothing within the VM itself shows high memory usage. This is the chyves c= onfig for one of them: >>=20 >> bargs -A -H -P -S >> bhyve_disk_type virtio-blk >> bhyve_net_type virtio-net >> bhyveload_flags >> chyves_guest_version 0300 >> cpu 4 >> creation Created on Mon Oct 23 16:17:04 CDT 2017= by chyves v0.2.0 2016/09/11 using __create() >> loader bhyveload >> net_ifaces tap51 >> os default >> ram 16G >> rcboot 0 >> revert_to_snapshot >> revert_to_snapshot_method off >> serial nmdm51 >> template no >> uuid 8495a130-b837-11e7-b092-0025909a8b56 >>=20 >>=20 >> I've also tried using different bhyve_disk_types, with no improvement. Ho= w is it that bhyve can use far more memory that I'm specifying? >>=20 >> - .Dustin From owner-freebsd-virtualization@freebsd.org Sat Dec 2 06:47:26 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 512D4DF5B59 for ; Sat, 2 Dec 2017 06:47:26 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-oi0-x22f.google.com (mail-oi0-x22f.google.com [IPv6:2607:f8b0:4003:c06::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1986563A94 for ; Sat, 2 Dec 2017 06:47:26 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-oi0-x22f.google.com with SMTP id j17so8637628oih.3 for ; Fri, 01 Dec 2017 22:47:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=N4KRZpl1zRKfYTO04ldgATXSrE+XVPVH2jpjVAQarGg=; b=amdRoOacebgzeDXYSeFlfNnvMgI/ZyfYRuAGvIuXugXqs0+KV/F8nGLYBxzKlL4osa 99LvFomMItN8osbillKatWeKPmYvHf5wJ25Q6bn4fQQmkrZre1ym4nDJpXAHu0Xb15dL azj2gYQt0Z4u/BWhOpl9TdHMWyOt1T4cViqG/hQwyGxhQRNEMJZ+z4jz3RxD9WmEPX/M fss9Vc2I8HtiGr3YyPWfju+2nBGOjyYwYn0txl4vaJnsJqPiHFzwuQV/utzsArIRutkP 9lX1GKoil7lcire7aNGe8SLddT4TQzecAyQ8tADg/IeCZfhBoeJ43AuahXR7hLcsDyPs NuKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=N4KRZpl1zRKfYTO04ldgATXSrE+XVPVH2jpjVAQarGg=; b=bGj2CyGqjIcWR2EvUye6Knrzr0RlfIfQ/eUrx3WejUdsCToqPUSlH9Bi+UWzfbPK7L y1MnJZkQ5hVzmvRlp2a7NLR47oHNJIjqiEoaXYH1Sps5htWWTSghyPfTydBlWy7AnTVV FAgp2YmgH5bDDKalBOVcbDd3VcLPRQ0F2EGZmK4l4bCjMi16WGpjnXOZPb7Q0eRQF+7T DJYgd+JZbfpp/Cp+qF8f6TOHP5PngFZwg0FRgug5SZukP/FBKXNe33HpbTjm949VIxYt +af8BXJkiU4otG6ZBgN+ZEFjGFTSpCGb/g4qQsrNqXCjIfpHAPPUPvcqCuOOG/XqZqlD PH4Q== X-Gm-Message-State: AJaThX7daUHauMVzNZxRCSzzM5xOyNWzhX1WQssnkFcCdDoKVSY10DVj 6df6hga2wa56SKOnLhN0OmGRWIzOKWi4WeWaff9vbA== X-Google-Smtp-Source: AGs4zMbxm38YTK2u+CyUhIfOCdDCQsBJtl4Y7lmM0NRQNyLTlqoWwoA76cYlXSTsyoV8IBZj3kw4hn9S9qhCmihA7jE= X-Received: by 10.202.199.133 with SMTP id x127mr8857313oif.17.1512197245165; Fri, 01 Dec 2017 22:47:25 -0800 (PST) MIME-Version: 1.0 Sender: kmacybsd@gmail.com Received: by 10.157.66.238 with HTTP; Fri, 1 Dec 2017 22:47:24 -0800 (PST) In-Reply-To: <59DFCE5F-029F-4585-B0BA-8FABC43357F2@ebureau.com> References: <59DFCE5F-029F-4585-B0BA-8FABC43357F2@ebureau.com> From: "K. Macy" Date: Fri, 1 Dec 2017 22:47:24 -0800 X-Google-Sender-Auth: 2Rjh3-wG_jYcM6PfLLVEW2vXcQU Message-ID: Subject: Re: bhyve uses all available memory during IO-intensive operations To: Dustin Wenz Cc: "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 06:47:26 -0000 On Fri, Dec 1, 2017 at 9:23 PM, Dustin Wenz wrote: > I have noticed significant storage amplification for my zvols; that could > very well be the reason. I would like to know more about why it happens. > > Since the volblocksize is 512 bytes, I certainly expect extra cpu overhead > (and maybe an extra 1k or so worth of checksums for each 128k block in the > vm), but how do you get a 10X expansion in stored data? > > What is the recommended zvol block size for a FreeBSD/ZFS guest? Perhaps 4k, > to match the most common mass storage sector size? I would err somewhat larger, the benefits of shallower indirect block chains will outweigh the cost of RMW I would guess. And I think it should be your guest file system block size. I don't know what ext4 is, but ext2/3 was 16k by default IIRC. -M > > - .Dustin > > On Dec 1, 2017, at 9:18 PM, K. Macy wrote: > > One thing to watch out for with chyves if your virtual disk is more > than 20G is the fact that it uses 512 byte blocks for the zvols it > creates. I ended up using up 1.4TB only half filling up a 250G zvol. > Chyves is quick and easy, but it's not exactly production ready. > > -M > > > > On Thu, Nov 30, 2017 at 3:15 PM, Dustin Wenz wrote: > > I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS is > also FreeBSD 11.1). Their sole purpose is to house some medium-sized > Postgres databases (100-200GB). The host system has 64GB of real memory and > 112GB of swap. I have configured each guest to only use 16GB of memory, yet > while doing my initial database imports in the VMs, bhyve will quickly grow > to use all available system memory and then be killed by the kernel: > > > kernel: swap_pager: I/O error - pageout failed; blkno 1735,size 4096, > error 12 > > kernel: swap_pager: I/O error - pageout failed; blkno 1610,size 4096, > error 12 > > kernel: swap_pager: I/O error - pageout failed; blkno 1763,size 4096, > error 12 > > kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space > > > The OOM condition seems related to doing moderate IO within the VM, though > nothing within the VM itself shows high memory usage. This is the chyves > config for one of them: > > > bargs -A -H -P -S > > bhyve_disk_type virtio-blk > > bhyve_net_type virtio-net > > bhyveload_flags > > chyves_guest_version 0300 > > cpu 4 > > creation Created on Mon Oct 23 16:17:04 CDT 2017 by > chyves v0.2.0 2016/09/11 using __create() > > loader bhyveload > > net_ifaces tap51 > > os default > > ram 16G > > rcboot 0 > > revert_to_snapshot > > revert_to_snapshot_method off > > serial nmdm51 > > template no > > uuid 8495a130-b837-11e7-b092-0025909a8b56 > > > > I've also tried using different bhyve_disk_types, with no improvement. How > is it that bhyve can use far more memory that I'm specifying? > > > - .Dustin From owner-freebsd-virtualization@freebsd.org Sat Dec 2 09:23:10 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 23547DF9006 for ; Sat, 2 Dec 2017 09:23:10 +0000 (UTC) (envelope-from k@free.de) Received: from smtp.free.de (smtp.free.de [91.204.6.103]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E155D6784E for ; Sat, 2 Dec 2017 09:23:09 +0000 (UTC) (envelope-from k@free.de) Received: from [91.204.5.154] (unknown [91.204.5.154]) by smtp.free.de (Postfix) with ESMTPSA id A5755160D8; Sat, 2 Dec 2017 10:23:00 +0100 (CET) Subject: Re: Linux lockups inside bhyve VM on FreeBSD 11.1 To: Jason Tubnor Cc: freebsd-virtualization@freebsd.org References: <5b1166b2-0b8c-248c-9a3b-b5c9685e4f37@free.de> From: Kai Gallasch Message-ID: <61029a6b-84fb-6eca-c5c6-73e517dd9bce@free.de> Date: Sat, 2 Dec 2017 10:22:59 +0100 Mime-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 09:23:10 -0000 Am 01.12.2017 um 03:41 schrieb Jason Tubnor: > > On 1 December 2017 at 08:29, Kai Gallasch > > wrote: > > Hello. > > Last day an apache 2.4 running inside a Debian9 linux bhyve VM locked up > on one of my servers (FreeBSD 11.1-RELENG, GENERIC kernel) overloading > the VM. > > The VM uses a ZFS zvol blockdevice on top of a zpool, consisting of two > mirrored SSDs. > > I was able to enter the VM through the bhyve console, kill and restart > the stuck apache process and regain stability inside the VM. > > I found below output in the Linux dmesg and suspect the ext4 journaling > to be the culprit. > > Has anyone experienced similar lockups running Linux inside a bhyve VM? > At the time when this happened there was no high I/O on the VM zpool. > > > Have you set vfs.zfs.arc_max to a lower value to allow for bhyve head > room?  How was the host system swap, did the host start to eat into it? > > I run a few guests with Ubuntu 16.04 but mainly use XFS for areas that > aren't system related and haven't come across this issue. Hello Jason. My bhyve host server has 96GB RAM and all Linux VMs together are allocated 20GB. I now have set vfs.zfs.arc_max to 64G to see if other lockups occur. K. From owner-freebsd-virtualization@freebsd.org Sat Dec 2 11:11:24 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15B08DFB1AB for ; Sat, 2 Dec 2017 11:11:24 +0000 (UTC) (envelope-from paul.g.webster@googlemail.com) Received: from mail-yw0-x233.google.com (mail-yw0-x233.google.com [IPv6:2607:f8b0:4002:c05::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 96EB16A3EB; Sat, 2 Dec 2017 11:11:23 +0000 (UTC) (envelope-from paul.g.webster@googlemail.com) Received: by mail-yw0-x233.google.com with SMTP id m81so5041807ywd.2; Sat, 02 Dec 2017 03:11:23 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=U0ZrOmVegDeJatiBeioqiWaeGJiCE3sqth0UJJqk7hQ=; b=Q7hrOwFt3T5gy9q8n8W2Pqt1gZDM2IW2lBVyF8HWPJ+gUrkxNsgXM8ibW6TA67HFqO biFTA5xqxA4D+qZBp2k+7C77blDIRmBGjDw7Aydgj26c8ySFlyCTr9IOhNS9+CZWX1qf 9IaBoh8as101+UcLE4l3jMnEZcYVTUQGCaFSn/z7rqZUnE7om4Ye4VHaDO4k2K6GoHFq g0ivtGanaskcIvhOmlDy8LsYCRSSWzq1l84dj8nC9JC8JWp7UnK2neiCIA2cJb+PotJ9 +sCx2zJwzR3jr52rKrHhKGkY3W52X053MqxmKUg8LjlcX6utxyyAjqo4Etrii4Gbo3Ro 44BA== X-Gm-Message-State: AJaThX65I3QiA+HUGHhwdZ1ZtXxJg1rgIIlOKoHyOMP/daqmqztJMOF4 Tcyt9tfOr9IMjHVeX7ytqAsbTp3Re+WtpREuGj6Tng== X-Google-Smtp-Source: AGs4zMZs/wdrE8xPQSqyGWdQosks/nmC+xNGM7+1KzI7ghRsmdvjFeVTtjjzqp6Eh7vxziBmlW39mhOGSuN+tKVEmVU= X-Received: by 10.129.118.74 with SMTP id j10mr5847239ywk.152.1512213082197; Sat, 02 Dec 2017 03:11:22 -0800 (PST) MIME-Version: 1.0 Received: by 10.37.165.8 with HTTP; Sat, 2 Dec 2017 03:11:21 -0800 (PST) In-Reply-To: References: <59DFCE5F-029F-4585-B0BA-8FABC43357F2@ebureau.com> From: Paul Webster Date: Sat, 2 Dec 2017 11:11:21 +0000 Message-ID: Subject: Re: bhyve uses all available memory during IO-intensive operations To: "K. Macy" Cc: Dustin Wenz , "freebsd-virtualization@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 11:11:24 -0000 Just as I was near one at the time, apparently ext4 is 4096 default sudo tune2fs -l /dev/sda tune2fs 1.43.4 (31-Jan-2017) Filesystem volume name: xdock Last mounted on: /var/lib/docker Filesystem UUID: b1dd0790-970d-4596-9192-49c704337015 Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype needs_recovery extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize metadata_csum Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 14655488 Block count: 58607766 Reserved block count: 2930388 Free blocks: 44314753 Free inodes: 13960548 First block: 0 Block size: 4096 Fragment size: 4096 Group descriptor size: 64 Reserved GDT blocks: 1024 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 512 Flex block group size: 16 Filesystem created: Thu Nov 9 10:32:16 2017 Last mount time: Wed Nov 29 17:08:30 2017 Last write time: Wed Nov 29 17:08:30 2017 Mount count: 21 Maximum mount count: -1 Last checked: Thu Nov 9 10:32:16 2017 Check interval: 0 () Lifetime writes: 147 GB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 32 Desired extra isize: 32 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: e943c6b0-9b5c-402a-a2ca-5f7dd094712d Journal backup: inode blocks Checksum type: crc32c Checksum: 0x04f644e2 On 2 December 2017 at 06:47, K. Macy wrote: > On Fri, Dec 1, 2017 at 9:23 PM, Dustin Wenz > wrote: > > I have noticed significant storage amplification for my zvols; that could > > very well be the reason. I would like to know more about why it happens. > > > > Since the volblocksize is 512 bytes, I certainly expect extra cpu > overhead > > (and maybe an extra 1k or so worth of checksums for each 128k block in > the > > vm), but how do you get a 10X expansion in stored data? > > > > What is the recommended zvol block size for a FreeBSD/ZFS guest? Perhaps > 4k, > > to match the most common mass storage sector size? > > I would err somewhat larger, the benefits of shallower indirect block > chains will outweigh the cost of RMW I would guess. And I think it > should be your guest file system block size. I don't know what ext4 > is, but ext2/3 was 16k by default IIRC. > > -M > > > > > - .Dustin > > > > On Dec 1, 2017, at 9:18 PM, K. Macy wrote: > > > > One thing to watch out for with chyves if your virtual disk is more > > than 20G is the fact that it uses 512 byte blocks for the zvols it > > creates. I ended up using up 1.4TB only half filling up a 250G zvol. > > Chyves is quick and easy, but it's not exactly production ready. > > > > -M > > > > > > > > On Thu, Nov 30, 2017 at 3:15 PM, Dustin Wenz > wrote: > > > > I'm using chyves on FreeBSD 11.1 RELEASE to manage a few VMs (guest OS is > > also FreeBSD 11.1). Their sole purpose is to house some medium-sized > > Postgres databases (100-200GB). The host system has 64GB of real memory > and > > 112GB of swap. I have configured each guest to only use 16GB of memory, > yet > > while doing my initial database imports in the VMs, bhyve will quickly > grow > > to use all available system memory and then be killed by the kernel: > > > > > > kernel: swap_pager: I/O error - pageout failed; blkno 1735,size > 4096, > > error 12 > > > > kernel: swap_pager: I/O error - pageout failed; blkno 1610,size > 4096, > > error 12 > > > > kernel: swap_pager: I/O error - pageout failed; blkno 1763,size > 4096, > > error 12 > > > > kernel: pid 41123 (bhyve), uid 0, was killed: out of swap space > > > > > > The OOM condition seems related to doing moderate IO within the VM, > though > > nothing within the VM itself shows high memory usage. This is the chyves > > config for one of them: > > > > > > bargs -A -H -P -S > > > > bhyve_disk_type virtio-blk > > > > bhyve_net_type virtio-net > > > > bhyveload_flags > > > > chyves_guest_version 0300 > > > > cpu 4 > > > > creation Created on Mon Oct 23 16:17:04 CDT > 2017 by > > chyves v0.2.0 2016/09/11 using __create() > > > > loader bhyveload > > > > net_ifaces tap51 > > > > os default > > > > ram 16G > > > > rcboot 0 > > > > revert_to_snapshot > > > > revert_to_snapshot_method off > > > > serial nmdm51 > > > > template no > > > > uuid 8495a130-b837-11e7-b092-0025909a8b56 > > > > > > > > I've also tried using different bhyve_disk_types, with no improvement. > How > > is it that bhyve can use far more memory that I'm specifying? > > > > > > - .Dustin > _______________________________________________ > freebsd-virtualization@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > To unsubscribe, send any mail to "freebsd-virtualization- > unsubscribe@freebsd.org" > From owner-freebsd-virtualization@freebsd.org Sat Dec 2 19:57:36 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C39EDE68566 for ; Sat, 2 Dec 2017 19:57:36 +0000 (UTC) (envelope-from notification+kxwm5b4x@facebookmail.com) Received: from mx-out.facebook.com (66-220-144-144.outmail.facebook.com [66.220.144.144]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 883597B9E2 for ; Sat, 2 Dec 2017 19:57:35 +0000 (UTC) (envelope-from notification+kxwm5b4x@facebookmail.com) Received: from facebook.com (t7tktz67oAn1pESJ2eRhXpbibOGtZcln3fsaWTCIOQINs5uCWqRfn8K20PfUtSrv 2401:db00:1030:51cf:face:0000:000b:0000) by facebook.com with Thrift id 05fb9472d79b11e7b38b000af7a31f42-9d9fa058; Sat, 02 Dec 2017 11:57:27 -0800 X-Facebook: from 2401:db00:1030:9091:face:0:79:0 ([MTI3LjAuMC4x]) by async.twshared1686.06.atn2.facebook.com with HTTP (ZuckMail); Date: Sat, 2 Dec 2017 11:57:27 -0800 To: Najat Lamini Subject: =?UTF-8?B?Vm9pciBsZSBtZXNzYWdl?= =?UTF-8?B?IGRlIFNhbWlhIExhbWkg?= =?UTF-8?B?ZXQgbGVzIGF1dHJlcyBu?= =?UTF-8?B?b3RpZmljYXRpb25zIHF1?= =?UTF-8?B?ZSB2b3VzIGF2ZXogbWFu?= =?UTF-8?B?cXXDqWVz?= X-Priority: 3 X-Mailer: ZuckMail [version 1.00] From: "Facebook" Reply-to: noreply Errors-To: notification+kxwm5b4x@facebookmail.com X-Facebook-Notify: stale_notifications; mailid=55f609b517e6bG45580756G55f60e4e7813dG32b Feedback-ID: 1:stale_notifications:Facebook X-FACEBOOK-PRIORITY: 1 X-Auto-Response-Suppress: All Require-Recipient-Valid-Since: freebsd-virtualization@freebsd.org; Thursday, 26 Mar 2009 21:50:33 +0000 Message-ID: <51221ca77fe0a34102a47b27166a4aa6@async.twshared1686.06.atn2.facebook.com> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 19:57:36 -0000 =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Acc=C3=A9dez =C3=A0 Facebook https://www.facebook.com/n/?aref=3D1512244645429565&medium=3Demail&mid=3D5= 5f609b517e6bG45580756G55f60e4e7813dG32b&bcode=3D2.1512244647.AbzZiyeOP0ek9= 15if3Q&n_m=3Dfreebsd-virtualization%40freebsd.org&lloc=3D2nd_cta Afficher les notifications https://www.facebook.com/n/?notifications&aref=3D1512244645429565&medium= =3Demail&mid=3D55f609b517e6bG45580756G55f60e4e7813dG32b&bcode=3D2.15122446= 47.AbzZiyeOP0ek915if3Q&n_m=3Dfreebsd-virtualization%40freebsd.org&lloc=3D1= st_cta =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Bonjour Najat, Pas mal de choses se sont pass=C3=A9es sur Facebook depuis votre = derni=C3=A8re connexion. Voici quelques notifications que vous auriez pu = rater. "=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A02 messages =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A026 nouvelles notifications" Merci, L=E2=80=99=C3=A9quipe Facebook =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D Ce message a =C3=A9t=C3=A9 envoy=C3=A9 =C3=A0 = freebsd-virtualization@freebsd.org. Si vous ne souhaitez plus recevoir ces = messages de la part de Facebook, veuillez suivre le lien ci-dessous pour = annuler votre abonnement. https://www.facebook.com/o.php?k=3DAS3tJmWNWfJVslvz&u=3D1163396950&mid=3D5= 5f609b517e6bG45580756G55f60e4e7813dG32b Facebook, Inc., Attention: Community Support, 1 Hacker Way, Menlo Park, CA = 94025 From owner-freebsd-virtualization@freebsd.org Sat Dec 2 20:03:42 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5C9D2E68855 for ; Sat, 2 Dec 2017 20:03:42 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (br1.CN84in.dnsmgr.net [69.59.192.140]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1D0E97BF07; Sat, 2 Dec 2017 20:03:41 +0000 (UTC) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: from pdx.rh.CN85.dnsmgr.net (localhost [127.0.0.1]) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3) with ESMTP id vB2K3daS036624; Sat, 2 Dec 2017 12:03:39 -0800 (PST) (envelope-from freebsd-rwg@pdx.rh.CN85.dnsmgr.net) Received: (from freebsd-rwg@localhost) by pdx.rh.CN85.dnsmgr.net (8.13.3/8.13.3/Submit) id vB2K3cok036623; Sat, 2 Dec 2017 12:03:38 -0800 (PST) (envelope-from freebsd-rwg) From: "Rodney W. Grimes" Message-Id: <201712022003.vB2K3cok036623@pdx.rh.CN85.dnsmgr.net> Subject: Re: bhyve uses all available memory during IO-intensive operations In-Reply-To: To: "K. Macy" Date: Sat, 2 Dec 2017 12:03:38 -0800 (PST) CC: Shane Ambler , freebsd-virtualization@freebsd.org X-Mailer: ELM [version 2.4ME+ PL121h (25)] MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=US-ASCII X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 20:03:42 -0000 > On Fri, Dec 1, 2017 at 20:02 Rodney W. Grimes < > freebsd-rwg@pdx.rh.cn85.dnsmgr.net> wrote: > > > > On 02/12/2017 08:11, Dustin Wenz wrote: > > > > > > > > The commit history shows that chyves defaults to -S if you are > > > > hosting from FreeBSD 10.3 or later. I'm sure they had a reason for > > > > doing that, but I don't know what that would be. It seems to an > > > > inefficient use of main memory if you need to run a lot of VMs. > > > > > > It sounds like a reasonable solution to a problem. If host memory is > > > full it swaps some out, so a bhyve might have free mem but some could be > > > swapped out by the host. If the bhyve is out of mem, it's system swaps > > > to it's disk, so the host swaps it back in so that the bhyve can then > > > swap it to its disk... > > > > > > Wiring bhyve ram might be a reasonable solution as long as the hosts > > > physical ram isn't over allocated by bhyve guests. > > > > > > The best solution would involve a host and guest talking to each other > > > about used mem, but that would break the whole virtual machine illusion. > > > At the least it would involve a system telling the hardware what memory > > > is used and what is not, which just isn't something any system does. > > > Maybe that is an idea for the vm guest aware systems of the future. > > > > Its actually old technology, its called the memory balloon driver, > > but bhyve does not have that functionality, yet. > > > > > > The virtio ballon driver is already there. Implementing a kernel backend > for it would be trivial. In kernel virtio-net and virtio-p9fs backends are > already well underway. Don't you also need guest front ends for each OS? That is the hard part from what I have seen of all the memory over commit stuff. Especially when you get to the part of "this page of memory in this guest is the exact same thing as that page of memory in that guest". But if we had the backend working, and just the FreeBSD guest frontend it would be a big win for many of us using bhyve with quantities of FreeBSD guests. Are we compativle enough we can use the KVM guest balloning extensions? -- Rod Grimes rgrimes@freebsd.org From owner-freebsd-virtualization@freebsd.org Sat Dec 2 23:46:36 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1F1EAE6CA96 for ; Sat, 2 Dec 2017 23:46:36 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-ot0-f179.google.com (mail-ot0-f179.google.com [74.125.82.179]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E4463191D for ; Sat, 2 Dec 2017 23:46:35 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: by mail-ot0-f179.google.com with SMTP id j2so11945528ota.13 for ; Sat, 02 Dec 2017 15:46:35 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=/dyrasKuMg848gFOLbiIxOIfeNEEAjW7+kgdcWN6K0g=; b=sgzW58tHoR9zWjhgPtBhm4vHfSo+xf6g8Kz6+gr8RfdeZvcWjMrVfitPeDpV3xqqeN bSEXd+FARTaffSc2N5cuZYtE2MnUz/fw2hygPuMl3a6L5/9UjMMzYtsTwV6T+Rth4+8B HJLGuuOK+CUMdzrgQPAo5rQ2nrDdy1Qu3wL+MaL0cgixIyS6nVXyhzILcOmNqHQ6Q3DS ItQaeQx0y7iiYIeIg7z+q3n+MzQ/aJZP3SvpeozuwRq8kSEnsvvuX5su8JVWSdGTbq0U 36gVscjpYWHNvdqSNU9uVlAyFffzMiQuojLn/rKRYsacYPh2jM9Nn64Fx+IdhuxtVhXH lN/g== X-Gm-Message-State: AJaThX7JlGQ1tJmn7iTUoxx+SzLkyOWiPCp5+yfZS0jMMTzxzlBkWEDo SCZEjC13QbPMOzFhmWfu+BM78aVfb6oNsAkvLmg= X-Google-Smtp-Source: AGs4zMYovQxbSAMyHB4zzZxnezgNHnNYf5o4S7CSlWoiUFwFHhhAPuUnk6ixfBG9SyGWEBGTyXCaiKvwbyIbXWGKsWo= X-Received: by 10.157.45.161 with SMTP id g30mr11263443otb.122.1512257901448; Sat, 02 Dec 2017 15:38:21 -0800 (PST) MIME-Version: 1.0 References: <201712022003.vB2K3cok036623@pdx.rh.CN85.dnsmgr.net> In-Reply-To: <201712022003.vB2K3cok036623@pdx.rh.CN85.dnsmgr.net> From: "K. Macy" Date: Sat, 02 Dec 2017 23:38:11 +0000 Message-ID: Subject: Re: bhyve uses all available memory during IO-intensive operations To: "Rodney W. Grimes" Cc: Shane Ambler , freebsd-virtualization@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 02 Dec 2017 23:46:36 -0000 There was a standards group but now the interfaces used buy the Linux virtio drivers define the de facto standard. As virtual interfaces go they're fairly decent. So all we need is a backend. The one thing FreeBSD doesn't have that I miss is CPU hot plug when running as a guest - or at least a mechanism to be told to stop running on the APs. That would make live migration much simpler. -M On Sat, Dec 2, 2017 at 12:03 Rodney W. Grimes < freebsd-rwg@pdx.rh.cn85.dnsmgr.net> wrote: > > On Fri, Dec 1, 2017 at 20:02 Rodney W. Grimes < > > freebsd-rwg@pdx.rh.cn85.dnsmgr.net> wrote: > > > > > > On 02/12/2017 08:11, Dustin Wenz wrote: > > > > > > > > > > The commit history shows that chyves defaults to -S if you are > > > > > hosting from FreeBSD 10.3 or later. I'm sure they had a reason for > > > > > doing that, but I don't know what that would be. It seems to an > > > > > inefficient use of main memory if you need to run a lot of VMs. > > > > > > > > It sounds like a reasonable solution to a problem. If host memory is > > > > full it swaps some out, so a bhyve might have free mem but some > could be > > > > swapped out by the host. If the bhyve is out of mem, it's system > swaps > > > > to it's disk, so the host swaps it back in so that the bhyve can then > > > > swap it to its disk... > > > > > > > > Wiring bhyve ram might be a reasonable solution as long as the hosts > > > > physical ram isn't over allocated by bhyve guests. > > > > > > > > The best solution would involve a host and guest talking to each > other > > > > about used mem, but that would break the whole virtual machine > illusion. > > > > At the least it would involve a system telling the hardware what > memory > > > > is used and what is not, which just isn't something any system does. > > > > Maybe that is an idea for the vm guest aware systems of the future. > > > > > > Its actually old technology, its called the memory balloon driver, > > > but bhyve does not have that functionality, yet. > > > > > > > > > > The virtio ballon driver is already there. Implementing a kernel backend > > for it would be trivial. In kernel virtio-net and virtio-p9fs backends > are > > already well underway. > > Don't you also need guest front ends for each OS? That is the hard part > from what I have seen of all the memory over commit stuff. Especially when > you get to the part of "this page of memory in this guest is the exact > same thing as that page of memory in that guest". > > But if we had the backend working, and just the FreeBSD guest frontend > it would be a big win for many of us using bhyve with quantities of > FreeBSD guests. > > Are we compativle enough we can use the KVM guest balloning extensions? > > -- > Rod Grimes > rgrimes@freebsd.org >