From owner-freebsd-jail@FreeBSD.ORG Mon Sep 6 07:18:37 2010 Return-Path: Delivered-To: freebsd-jail@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 946BE10656AD; Mon, 6 Sep 2010 07:18:37 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 6A8948FC19; Mon, 6 Sep 2010 07:18:37 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o867IbTU045693; Mon, 6 Sep 2010 07:18:37 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o867IaxD045689; Mon, 6 Sep 2010 07:18:36 GMT (envelope-from linimon) Date: Mon, 6 Sep 2010 07:18:36 GMT Message-Id: <201009060718.o867IaxD045689@freefall.freebsd.org> To: tom.dewaele@abvv.be, linimon@FreeBSD.org, freebsd-jail@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/147162: [jail] [panic] Page Fault / Kernel panic when jail starts on boot X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 07:18:37 -0000 Synopsis: [jail] [panic] Page Fault / Kernel panic when jail starts on boot State-Changed-From-To: open->closed State-Changed-By: linimon State-Changed-When: Mon Sep 6 07:18:03 UTC 2010 State-Changed-Why: Submitter notes that this seems to be fixed on 8.1. http://www.freebsd.org/cgi/query-pr.cgi?pr=147162 From owner-freebsd-jail@FreeBSD.ORG Mon Sep 6 11:06:58 2010 Return-Path: Delivered-To: freebsd-jail@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D023F10656E2 for ; Mon, 6 Sep 2010 11:06:58 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id BE6AF8FC23 for ; Mon, 6 Sep 2010 11:06:58 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o86B6wqp011813 for ; Mon, 6 Sep 2010 11:06:58 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o86B6woN011811 for freebsd-jail@FreeBSD.org; Mon, 6 Sep 2010 11:06:58 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 6 Sep 2010 11:06:58 GMT Message-Id: <201009061106.o86B6woN011811@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-jail@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-jail@FreeBSD.org X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Sep 2010 11:06:58 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o conf/149050 jail [jail] rcorder ``nojail'' too coarse for Jail+VNET s conf/142972 jail [jail] [patch] Support JAILv2 and vnet in rc.d/jail o conf/141317 jail [patch] uncorrect jail stop in /etc/rc.d/jail o kern/133265 jail [jail] is there a solution how to run nfs client in ja o kern/119842 jail [smbfs] [jail] "Bad address" with smbfs inside a jail o bin/99566 jail [jail] [patch] fstat(1) according to specified jid o bin/32828 jail [jail] w(1) incorrectly handles stale utmp slots with 7 problems total. From owner-freebsd-jail@FreeBSD.ORG Fri Sep 10 04:59:01 2010 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2E3331065674 for ; Fri, 10 Sep 2010 04:59:01 +0000 (UTC) (envelope-from freebsd@snap.net.nz) Received: from snapmx1.ironport.snap.net.nz (snapmx1.ironport.snap.net.nz [202.37.100.100]) by mx1.freebsd.org (Postfix) with ESMTP id B4E668FC08 for ; Fri, 10 Sep 2010 04:59:00 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AoIFACdRiUzKfG1Y/2dsb2JhbACUAoVuh1DBI4U9BIog X-IronPort-AV: E=Sophos;i="4.56,344,1280664000"; d="scan'208";a="14319166" Received: from rupert.snap.net.nz ([202.37.100.140]) by smtp1.ironport.snap.net.nz with ESMTP; 10 Sep 2010 16:29:35 +1200 X-Sender-IP: 202.124.109.88 Received: from voyager.local (88.109.124.202.static.snap.net.nz [202.124.109.88]) by rupert.snap.net.nz (Postfix) with ESMTP id A8E2E20382 for ; Fri, 10 Sep 2010 16:26:47 +1200 (NZST) Message-ID: <4C89B3DF.7050004@snap.net.nz> Date: Fri, 10 Sep 2010 16:28:15 +1200 From: Peter Toth User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.11) Gecko/20100805 Thunderbird/3.0.6 MIME-Version: 1.0 To: freebsd-jail@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Jail hot migration X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 04:59:01 -0000 Hi guys, I was lately thinking around jail hot-migration feature where one jail could be moved from one host to another without shutting it down, something like vmotion in VMware world. The storage layer should be easy with zfs send and receive or some form of shared storage. The tricky part would be a memory copy from one node to another and also the CPU handling. Anyone has an idea how this could be achieved? I guess it would require a kernel module which could take care of memory reservations and a daemon to copy and incrementally sync the jails memory across. Then also there is the CPU problem.. Sounds like a fair amount of work and development. All comments are welcomed! Cheers, Peter From owner-freebsd-jail@FreeBSD.ORG Fri Sep 10 12:48:45 2010 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9F26C106566B for ; Fri, 10 Sep 2010 12:48:45 +0000 (UTC) (envelope-from andrew.hotlab@hotmail.com) Received: from snt0-omc4-s32.snt0.hotmail.com (snt0-omc4-s32.snt0.hotmail.com [65.55.90.235]) by mx1.freebsd.org (Postfix) with ESMTP id 793D88FC17 for ; Fri, 10 Sep 2010 12:48:45 +0000 (UTC) Received: from SNT139-W26 ([65.55.90.201]) by snt0-omc4-s32.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Fri, 10 Sep 2010 05:36:42 -0700 Message-ID: X-Originating-IP: [81.174.54.98] From: Andrew Hotlab To: , Date: Fri, 10 Sep 2010 12:36:42 +0000 Importance: Normal Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginalArrivalTime: 10 Sep 2010 12:36:42.0555 (UTC) FILETIME=[D28F18B0:01CB50E4] Cc: Subject: RE: Jail hot migration X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 12:48:45 -0000 > Date: Fri=2C 10 Sep 2010 16:28:15 +1200 > From: freebsd@snap.net.nz > To: freebsd-jail@freebsd.org > Subject: Jail hot migration > > Hi guys=2C > > I was lately thinking around jail hot-migration feature where one jail > could be moved from one host to another without > shutting it down=2C something like vmotion in VMware world. > > The storage layer should be easy with zfs send and receive or some form > of shared storage. The tricky part would > be a memory copy from one node to another and also the CPU handling. > > Anyone has an idea how this could be achieved? I guess it would require > a kernel module which could take care of memory > reservations and a daemon to copy and incrementally sync the jails > memory across. > > Then also there is the CPU problem.. > > Sounds like a fair amount of work and development. > > All comments are welcomed! > Well=2C I'm not a developer=2C but I think that the jail framework surely d= eservestobe evolved in something more "friendly" from a sysadmin's point of= view. The architecture is great=2C and that's just enough to consolidate a lot of= workloads=2Cbut for some applications there are features (resource contain= ers=2C offline andonline migration of jails=2C etc.) that need to be improv= ed to be affordable in aproduction environment. I think that a lot of work is getting done (resource container and virtuali= zationstack projects have ben just sponsored by the FreeBSD Foundation)=2C = I do not knowhow much time will take to reach a complete "business-ready" v= irtualizationframework=2C neither if they are on the right path=2C but I'm = quite confident=2C because I'msure that the simplicity of the FreeBSD solut= ion will rule any other virtualizationframework out there. As the feature you are asking for=2C I think that resource containers and o= fflinemigration should be considered first=2C because overall they'll have = more impact onbusiness environments. Further=2C I believe that building a h= ot migration procedureworking with third-party running applications will be= a very complex task to achieve=2Cmaybe too expensive at this time=2C compa= red to the amount of work required. That's an interesting argument=2C I'll be glad if more member of the Commun= ity willcontribute with their considerations. Sincerely. Andrew = From owner-freebsd-jail@FreeBSD.ORG Fri Sep 10 15:14:26 2010 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1211B106564A for ; Fri, 10 Sep 2010 15:14:26 +0000 (UTC) (envelope-from gnrp@gnrp.in-berlin.de) Received: from einhorn.in-berlin.de (einhorn.in-berlin.de [192.109.42.8]) by mx1.freebsd.org (Postfix) with ESMTP id 9065A8FC0A for ; Fri, 10 Sep 2010 15:14:25 +0000 (UTC) X-Envelope-From: gnrp@gnrp.in-berlin.de X-Envelope-To: Received: from adolfputzen (c-68-42-215-192.hsd1.mi.comcast.net [68.42.215.192]) (authenticated bits=0) by einhorn.in-berlin.de (8.13.6/8.13.6/Debian-1) with ESMTP id o8AEgR7P020360 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT) for ; Fri, 10 Sep 2010 16:42:29 +0200 Date: Fri, 10 Sep 2010 16:42:22 +0200 From: Julian Fagir To: freebsd-jail@freebsd.org Message-ID: <20100910164222.31ae35d8@adolfputzen> In-Reply-To: <4C89B3DF.7050004@snap.net.nz> References: <4C89B3DF.7050004@snap.net.nz> X-Mailer: Claws Mail 3.7.5 (GTK+ 2.14.7; x86_64-unknown-linux-gnu) Mime-Version: 1.0 Content-Type: multipart/signed; boundary="Sig_/kjJ.tJZvPz=tWh/XGF0rPBP"; protocol="application/pgp-signature"; micalg=PGP-SHA1 X-Scanned-By: MIMEDefang_at_IN-Berlin_e.V. on 192.109.42.8 Subject: Re: Jail hot migration X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: freebsd-jail@freebsd.org List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 15:14:26 -0000 --Sig_/kjJ.tJZvPz=tWh/XGF0rPBP Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: quoted-printable Hi, correct me if I'm wrong, I never actually worked with VMware (only a bit with Xen) but read a bit about the theory behind it. I don't think this can be as easilly achieved as it is with VMware (or Xen). > I was lately thinking around jail hot-migration feature where one jail > could be moved from one host to another without > shutting it down, something like vmotion in VMware world. With paravirtualisation, you have the machine in a 'container', the host system does not know except for the interfaces what the guest system is doi= ng with the ram it has reserved. So moving a machine is 'not more' than copying a bunch of memory to another machine and transferring the interfaces. Jails are no virtualisation, in the end, what makes up a jail, are only a f= ew data structures. Moving a jail would mean to transfer all these single fragments of data reasonably to another machine. I.e. you would have to transfer all the kernel-things a process does to another machine. How sould locks or access to system ressources be managed when transferring to another machine? I don't even know if the kernel does know all the stuff about a process on a central place to do this transfer. > The storage layer should be easy with zfs send and receive or some form > of shared storage. The tricky part would > be a memory copy from one node to another and also the CPU handling. Afaik you have containers for the machines with VMware/Xen, as the host system does not have to know the file system underneath. Again, transferring/sharing those is simple as it is only one file to the host system. With jails, you only have mount points. What happens to locks and other fs-specific stuff, ACLs etc (ok, zfs perhaps handles that)? But you can't r= ely on the other system having the same filesystem underneath. Also, sharing fi= le systems is not as easy as sharing a single file (without caring for permissions). You see the problem when you just look at all the different ways for locking over the network... > Anyone has an idea how this could be achieved? I guess it would require > a kernel module which could take care of memory > reservations and a daemon to copy and incrementally sync the jails > memory across. In case you could build a jail-transportation, a framework for transferring= a single process to another machine would be more appropriate imho as more generic and simpler. Then just adding the possibility of jailing such a process is not much work. A single example why the whole target cannot be achieved easily: Imagine a program which writes its pid into a lockfile. As jails and host share the pids, this pid could be already in use by the other host you want to transf= er the jail to. So what to do know? Just change the pid? Then the process would be confused and possibly die or malfunction. Not changing the pid would not be possible as then there would be two processes with the same pid. Sure, remapping pids would be possible but not within the current jails and the complications could be great (if the process sees another pid than the host-system, what will rc-files do which check lockfiles?). And there are other cases where processes do things the kernel cannot predict and so cannot transfer that easily. Also, the nic-transfer would make problems. When you usually have a bridge = or so with Xen/VMware, you have real interfaces or aliases on a BSD-machine, h= ow should they be taken out of service in time and back to service without problems (I often have cases when just taking up an alias does not work)? Jails are no virtualisation. They are an 'extended chroot'-environment and thus moving a jail is like moving a working machine except for the system drivers. I hope this was not all too wrong. Regards, Julian --Sig_/kjJ.tJZvPz=tWh/XGF0rPBP Content-Type: application/pgp-signature; name=signature.asc Content-Disposition: attachment; filename=signature.asc -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) iEYEARECAAYFAkyKQ84ACgkQc7h7cu1Hpp5blwCcC0mn5684xDjIW2prE/Ex8JwA 5GQAnA2ivRnmVyEsL5Ccd7v26ZDsXlqo =ugw9 -----END PGP SIGNATURE----- --Sig_/kjJ.tJZvPz=tWh/XGF0rPBP-- From owner-freebsd-jail@FreeBSD.ORG Fri Sep 10 15:17:15 2010 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 016D110656C5 for ; Fri, 10 Sep 2010 15:17:15 +0000 (UTC) (envelope-from freebsd@deman.com) Received: from cp11.openaccess.org (cp11.openaccess.org [66.114.41.130]) by mx1.freebsd.org (Postfix) with ESMTP id D55068FC12 for ; Fri, 10 Sep 2010 15:17:14 +0000 (UTC) Received: from mono-sis1.s.bli.openaccess.org ([66.114.32.149] helo=[192.168.2.226]) by cp11.openaccess.org with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69) (envelope-from ) id 1Ou52k-0002Os-T0 for freebsd-jail@freebsd.org; Fri, 10 Sep 2010 07:57:34 -0700 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Apple Message framework v1081) From: Michael DeMan In-Reply-To: <4C89B3DF.7050004@snap.net.nz> Date: Fri, 10 Sep 2010 07:57:31 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <1FD7CB89-8F7F-4E8B-A507-16E72784D906@deman.com> References: <4C89B3DF.7050004@snap.net.nz> To: freebsd-jail@freebsd.org X-Mailer: Apple Mail (2.1081) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cp11.openaccess.org X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - deman.com X-Source: X-Source-Args: X-Source-Dir: Subject: Re: Jail hot migration X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 15:17:15 -0000 There are other issues too - like network I/O, particularly dealing with = established TCP connections? Basically, you have to retain the in-state = 'memory' of the machine being migrated from, while also letting all = those 3rd party sockets know that everything is okay Thus far, I have yet to see any vendor that delivers 'zero downtime' = virtual migrations. There is always downtime - question is whether it = is costed in terms of milliseconds, seconds, minutes, hours, or worse? Definitely worth doing though. Personally, I have found value in = running OSPF/BGP on the jail host itself, and putting the actual jails = on the loopback interface. It certainly does not solve the problem in = terms of 'minute' by any means, but having the Iayer-3 component work = automatically definitely helps. I think this idea is a worthwhile goal, but I would much, much rather = see NFSv4 and ZFS wrapped up first. On Sep 9, 2010, at 9:28 PM, Peter Toth wrote: > Hi guys, >=20 > I was lately thinking around jail hot-migration feature where one jail > could be moved from one host to another without > shutting it down, something like vmotion in VMware world. >=20 > The storage layer should be easy with zfs send and receive or some = form > of shared storage. The tricky part would > be a memory copy from one node to another and also the CPU handling. >=20 > Anyone has an idea how this could be achieved? I guess it would = require > a kernel module which could take care of memory > reservations and a daemon to copy and incrementally sync the jails > memory across. >=20 > Then also there is the CPU problem.. >=20 > Sounds like a fair amount of work and development. >=20 > All comments are welcomed! >=20 > Cheers, > Peter > _______________________________________________ > freebsd-jail@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-jail > To unsubscribe, send any mail to = "freebsd-jail-unsubscribe@freebsd.org" From owner-freebsd-jail@FreeBSD.ORG Fri Sep 10 15:17:15 2010 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A5F2B10656C8 for ; Fri, 10 Sep 2010 15:17:15 +0000 (UTC) (envelope-from freebsd@deman.com) Received: from cp11.openaccess.org (cp11.openaccess.org [66.114.41.130]) by mx1.freebsd.org (Postfix) with ESMTP id 8A1098FC16 for ; Fri, 10 Sep 2010 15:17:15 +0000 (UTC) Received: from mono-sis1.s.bli.openaccess.org ([66.114.32.149] helo=[192.168.2.226]) by cp11.openaccess.org with esmtpsa (TLSv1:AES128-SHA:128) (Exim 4.69) (envelope-from ) id 1Ou577-0002fJ-6m; Fri, 10 Sep 2010 08:02:05 -0700 Mime-Version: 1.0 (Apple Message framework v1081) Content-Type: text/plain; charset=us-ascii From: Michael DeMan In-Reply-To: Date: Fri, 10 Sep 2010 08:02:01 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: <1C266502-EAF4-4211-93E1-83BA1090C94E@deman.com> References: To: Andrew Hotlab X-Mailer: Apple Mail (2.1081) X-AntiAbuse: This header was added to track abuse, please include it with any abuse report X-AntiAbuse: Primary Hostname - cp11.openaccess.org X-AntiAbuse: Original Domain - freebsd.org X-AntiAbuse: Originator/Caller UID/GID - [47 12] / [47 12] X-AntiAbuse: Sender Address Domain - deman.com X-Source: X-Source-Args: X-Source-Dir: Cc: freebsd-jail@freebsd.org, freebsd@snap.net.nz Subject: Re: Jail hot migration X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 15:17:15 -0000 See inline. On Sep 10, 2010, at 5:36 AM, Andrew Hotlab wrote: >=20 >> Date: Fri, 10 Sep 2010 16:28:15 +1200 >> From: freebsd@snap.net.nz >> To: freebsd-jail@freebsd.org >> Subject: Jail hot migration >>=20 >> Hi guys, >>=20 >> I was lately thinking around jail hot-migration feature where one = jail >> could be moved from one host to another without >> shutting it down, something like vmotion in VMware world. >>=20 >> The storage layer should be easy with zfs send and receive or some = form >> of shared storage. The tricky part would >> be a memory copy from one node to another and also the CPU handling. >>=20 >> Anyone has an idea how this could be achieved? I guess it would = require >> a kernel module which could take care of memory >> reservations and a daemon to copy and incrementally sync the jails >> memory across. >>=20 >> Then also there is the CPU problem.. >>=20 >> Sounds like a fair amount of work and development. >>=20 >> All comments are welcomed! >>=20 > Well, I'm not a developer, but I think that the jail framework surely = deservestobe evolved in something more "friendly" from a sysadmin's = point of view. > The architecture is great, and that's just enough to consolidate a lot = of workloads,but for some applications there are features (resource = containers, offline andonline migration of jails, etc.) that need to be = improved to be affordable in aproduction environment. > I think that a lot of work is getting done (resource container and = virtualizationstack projects have ben just sponsored by the FreeBSD = Foundation), I do not knowhow much time will take to reach a complete = "business-ready" virtualizationframework, neither if they are on the = right path, but I'm quite confident, because I'msure that the simplicity = of the FreeBSD solution will rule any other virtualizationframework out = there. > As the feature you are asking for, I think that resource containers = and offlinemigration should be considered first, because overall they'll = have more impact onbusiness environments. Further, I believe that = building a hot migration procedureworking with third-party running = applications will be a very complex task to achieve,maybe too expensive = at this time, compared to the amount of work required. > That's an interesting argument, I'll be glad if more member of the = Community willcontribute with their considerations. > Sincerely. >=20 Sorry, I'm accustomed to 'top positing' - LIFO - polish notation. Anyway, yes, is it no the plan that ezjail-admin *MIGHT* be integrated = into the FreeBSD core release? Once that is done, anybody could feel = comfortable writing a webmin module to manage jails? From owner-freebsd-jail@FreeBSD.ORG Fri Sep 10 20:03:28 2010 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5A3F1106564A for ; Fri, 10 Sep 2010 20:03:28 +0000 (UTC) (envelope-from klaus@mx.7he.at) Received: from smtp-01.sil.at (smtp-01.sil.at [78.142.186.24]) by mx1.freebsd.org (Postfix) with ESMTP id 112718FC1F for ; Fri, 10 Sep 2010 20:03:28 +0000 (UTC) Received: from mx.7he.at ([86.59.13.138]) by smtp-01.sil.at with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.63) (envelope-from ) id 1Ou9ok-0003Ys-KI for freebsd-jail@freebsd.org; Fri, 10 Sep 2010 22:03:26 +0200 Received: from mx.7he.at (mx.7he.at [86.59.13.138]) by mx.7he.at (8.14.3/8.14.3) with ESMTP id o8AK2pbJ007484 for ; Fri, 10 Sep 2010 20:02:51 GMT (envelope-from klaus@mx.7he.at) Received: (from klaus@localhost) by mx.7he.at (8.14.3/8.14.3/Submit) id o8AK2oXq007483 for freebsd-jail@freebsd.org; Fri, 10 Sep 2010 20:02:50 GMT (envelope-from klaus) Date: Fri, 10 Sep 2010 20:02:50 +0000 From: "Klaus P. Ohrhallinger" To: freebsd-jail@freebsd.org Message-ID: <20100910200250.GS2965@mx.7he.at> References: <4C89B3DF.7050004@snap.net.nz> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C89B3DF.7050004@snap.net.nz> User-Agent: Mutt/1.4.2.3i X-Political-Attitude: Anarchistic X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.0 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on mx.7he.at X-Scan-Signature: 2eb3c689388ef5a45072784d1e8a9ce2 Subject: Re: Jail hot migration X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 20:03:28 -0000 On Fri, Sep 10, 2010 at 04:28:15PM +1200, Peter Toth wrote: > Hi guys, > > I was lately thinking around jail hot-migration feature where one jail > could be moved from one host to another without > shutting it down, something like vmotion in VMware world. > [snip] Hello; I implemented a working prototype of a container-style virtualization for FreeBSD, capable of live/hot migration. It has nothing to do with jail, but it makes use of vimage/vnet. I am going to do a presentation at the EuroBSDCon in October, but a paper and a demo video (no sources yet, sorry) are already online at: http://www.7he.at/freebsd/vps/ Best regards, klaus -- Alle sagten ''es geht nicht''. Dann kam einer, der wusste das nicht. Der hat's einfach gemacht. From owner-freebsd-jail@FreeBSD.ORG Fri Sep 10 20:10:28 2010 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 000421065695 for ; Fri, 10 Sep 2010 20:10:27 +0000 (UTC) (envelope-from klaus@mx.7he.at) Received: from smtp-07.sil.at (smtp-07.sil.at [78.142.186.51]) by mx1.freebsd.org (Postfix) with ESMTP id A9CC88FC18 for ; Fri, 10 Sep 2010 20:10:26 +0000 (UTC) Received: from mx.7he.at ([86.59.13.138]) by smtp-07.sil.at with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.63) (envelope-from ) id 1Ou9Xk-00056F-5j for freebsd-jail@freebsd.org; Fri, 10 Sep 2010 21:45:52 +0200 Received: from mx.7he.at (mx.7he.at [86.59.13.138]) by mx.7he.at (8.14.3/8.14.3) with ESMTP id o8AJjKYx006380 for ; Fri, 10 Sep 2010 19:45:20 GMT (envelope-from klaus@mx.7he.at) Received: (from klaus@localhost) by mx.7he.at (8.14.3/8.14.3/Submit) id o8AJjK9f006379 for freebsd-jail@freebsd.org; Fri, 10 Sep 2010 19:45:20 GMT (envelope-from klaus) Date: Fri, 10 Sep 2010 19:45:20 +0000 From: "Klaus P. Ohrhallinger" To: freebsd-jail@freebsd.org Message-ID: <20100910194520.GP2965@mx.7he.at> References: <4C89B3DF.7050004@snap.net.nz> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C89B3DF.7050004@snap.net.nz> User-Agent: Mutt/1.4.2.3i X-Political-Attitude: Anarchistic X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.0 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on mx.7he.at X-Scan-Signature: 2eb3c689388ef5a45072784d1e8a9ce2 Subject: Re: Jail hot migration X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 20:10:28 -0000 On Fri, Sep 10, 2010 at 04:28:15PM +1200, Peter Toth wrote: > Hi guys, > > I was lately thinking around jail hot-migration feature where one jail > could be moved from one host to another without > shutting it down, something like vmotion in VMware world. > [snip] Hello; I implemented a working prototype of a container-style virtualization for FreeBSD, capable of live/hot migration. It has nothing to do with jail, but it makes use of vimage/vnet. I am going to do a presentation at the EuroBSDCon in October, but a paper and a demo video (no sources yet, sorry) are already online at: http://www.7he.at/freebsd/vps/ Best regards, klaus -- Alle sagten ''es geht nicht''. Dann kam einer, der wusste das nicht. Der hat's einfach gemacht. From owner-freebsd-jail@FreeBSD.ORG Fri Sep 10 22:28:10 2010 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C6C1E106566C for ; Fri, 10 Sep 2010 22:28:10 +0000 (UTC) (envelope-from andrew.hotlab@hotmail.com) Received: from snt0-omc4-s11.snt0.hotmail.com (snt0-omc4-s11.snt0.hotmail.com [65.55.90.214]) by mx1.freebsd.org (Postfix) with ESMTP id 9D3118FC12 for ; Fri, 10 Sep 2010 22:28:10 +0000 (UTC) Received: from SNT139-W19 ([65.55.90.200]) by snt0-omc4-s11.snt0.hotmail.com with Microsoft SMTPSVC(6.0.3790.4675); Fri, 10 Sep 2010 15:28:10 -0700 Message-ID: X-Originating-IP: [81.174.54.98] From: Andrew Hotlab To: , Date: Fri, 10 Sep 2010 22:28:09 +0000 Importance: Normal MIME-Version: 1.0 X-OriginalArrivalTime: 10 Sep 2010 22:28:10.0271 (UTC) FILETIME=[72E2D2F0:01CB5137] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: RE: Jail hot migration X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 10 Sep 2010 22:28:10 -0000 > Date: Fri=2C 10 Sep 2010 19:45:20 +0000 > From: k@7he.at > To: freebsd-jail@freebsd.org > Subject: Re: Jail hot migration >=20 > On Fri=2C Sep 10=2C 2010 at 04:28:15PM +1200=2C Peter Toth wrote: > > Hi guys=2C > >=20 > > I was lately thinking around jail hot-migration feature where one jail > > could be moved from one host to another without > > shutting it down=2C something like vmotion in VMware world. > >=20 >=20 > I implemented a working prototype of a container-style virtualization > for FreeBSD=2C capable of live/hot migration. It has nothing to do with > jail=2C but it makes use of vimage/vnet. >=20 > I am going to do a presentation at the EuroBSDCon in October=2C > but a paper and a demo video (no sources yet=2C sorry) are already > online at: >=20 > http://www.7he.at/freebsd/vps/ >=20 I was really impressed by the live migration demo video and I wish to ask y= ou a few questions about your solution... At this time VPS technology lacks resource limiting capability=2C do you th= ink it could be compatible with the resource container project sponsored by= the FreeBSD Foundation? Multiple virtual instances of FreeBSD running as VPS are required to share = the same kernel=2C as in the Jail paradigm=2C or they may run different ker= nels=2C as in NetBSD (http://www.NetBSD.org/docs/rump/)? How many resources do you estimate in order to complete your project? It seems that it will be far more difficult to implement the live migration= feature on the Jail framework than in VPS=2C but how about offline migrati= on and resource control? Some work has been done to achieve that with the c= urrent Jail paradigm=2C do you think that VPS will be able to do a better j= ob in these areas too? Sorry if any question sounds stupid: I do not know much on the internals of= the technologies I mentioned. Thank you very much for your work. Andrew P.S. I'm sorry for the format of the e-mail=2C I'm using this awful Hotmail= web interface. :( = From owner-freebsd-jail@FreeBSD.ORG Sat Sep 11 09:18:23 2010 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3DEE3106566B for ; Sat, 11 Sep 2010 09:18:23 +0000 (UTC) (envelope-from klaus@mx.7he.at) Received: from smtp-07.sil.at (smtp-07.sil.at [78.142.186.51]) by mx1.freebsd.org (Postfix) with ESMTP id BA9718FC08 for ; Sat, 11 Sep 2010 09:18:22 +0000 (UTC) Received: from mx.7he.at ([86.59.13.138]) by smtp-07.sil.at with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.63) (envelope-from ) id 1OuME1-0004d8-B7 for freebsd-jail@freebsd.org; Sat, 11 Sep 2010 11:18:21 +0200 Received: from mx.7he.at (mx.7he.at [86.59.13.138]) by mx.7he.at (8.14.3/8.14.3) with ESMTP id o8B9Hgjx061667 for ; Sat, 11 Sep 2010 09:17:42 GMT (envelope-from klaus@mx.7he.at) Received: (from klaus@localhost) by mx.7he.at (8.14.3/8.14.3/Submit) id o8B9HgN8061666 for freebsd-jail@freebsd.org; Sat, 11 Sep 2010 09:17:42 GMT (envelope-from klaus) Date: Sat, 11 Sep 2010 09:17:42 +0000 From: "Klaus P. Ohrhallinger" To: freebsd-jail@freebsd.org Message-ID: <20100911091742.GA14592@mx.7he.at> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Political-Attitude: Anarchistic X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.0 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on mx.7he.at X-Scan-Signature: dd53a031cb8b38c4127844b8be3aa059 Subject: Re: Jail hot migration X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Sep 2010 09:18:23 -0000 On Fri, Sep 10, 2010 at 10:28:09PM +0000, Andrew Hotlab wrote: Hello; > > I was really impressed by the live migration demo video and I wish to ask you a few questions about your solution... > At this time VPS technology lacks resource limiting capability, do you think it could be compatible with the resource container project sponsored by the FreeBSD Foundation? I heard about that resource container project, but don't know any details. But it sounds like it could be adapted or integrated to my work. I didn't put any effort in resource accounting/limiting yet because it's the most boring part of the whole story, IMHO. > Multiple virtual instances of FreeBSD running as VPS are required to share the same kernel, as in the Jail paradigm, or they may run different kernels, as in NetBSD (http://www.NetBSD.org/docs/rump/)? They share the same kernel, altough it works different from jail. Jail restricts processes from seeing and accessing certain resources, while VPS duplicates almost any resource. Please see my paper for details. > How many resources do you estimate in order to complete your project? Well, if I get positive feedback at the EuroBSDCon and people want to have it I can put quite a large amount of my time in it, but then it's still a long way to go until production quality. Anyways I plan to publish patches and binaries for testing in october. > It seems that it will be far more difficult to implement the live migration feature on the Jail framework than in VPS, but how about offline migration and resource control? Some work has been done to achieve that with the current Jail paradigm, do you think that VPS will be able to do a better job in these areas too? Live migration of jails is asking for some really nasty hacks. For example imagine what to do if a process has a PID number that is already assigned on the host you are migrating the jail to. > Sorry if any question sounds stupid: I do not know much on the internals of the technologies I mentioned. > Thank you very much for your work. > Andrew > P.S. I'm sorry for the format of the e-mail, I'm using this awful Hotmail web interface. :( > > Best regards, Klaus -- Alle sagten ''es geht nicht''. Dann kam einer, der wusste das nicht. Der hat's einfach gemacht. From owner-freebsd-jail@FreeBSD.ORG Sat Sep 11 10:08:07 2010 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0AE23106564A for ; Sat, 11 Sep 2010 10:08:07 +0000 (UTC) (envelope-from peter.toth@snap.net.nz) Received: from snapmx1.ironport.snap.net.nz (snapmx1.ironport.snap.net.nz [202.37.100.100]) by mx1.freebsd.org (Postfix) with ESMTP id 945EA8FC0A for ; Sat, 11 Sep 2010 10:08:06 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AoIFAE7rikzKfG1m/2dsb2JhbACUB4Vuh1O9S4U9BIog X-IronPort-AV: E=Sophos;i="4.56,351,1280664000"; d="scan'208";a="14586501" Received: from rupert.snap.net.nz ([202.37.100.140]) by smtp1.ironport.snap.net.nz with ESMTP; 11 Sep 2010 21:38:25 +1200 X-Sender-IP: 202.124.109.102 Received: from voyager.local (102.109.124.202.static.snap.net.nz [202.124.109.102]) by rupert.snap.net.nz (Postfix) with SMTP id 35DC020308 for ; Sat, 11 Sep 2010 21:35:36 +1200 (NZST) Message-ID: <4C8B4DF3.9050209@snap.net.nz> Date: Sat, 11 Sep 2010 21:37:55 +1200 From: Peter Toth User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.11) Gecko/20100805 Thunderbird/3.0.6 MIME-Version: 1.0 To: freebsd-jail@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sat, 11 Sep 2010 11:17:55 +0000 Subject: RE: Jail hot migration X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Sep 2010 10:08:07 -0000 That VPS solution is exactly what I was thinking of. Klaus, will this be licensed under BSD? How far is the source release? Regards From owner-freebsd-jail@FreeBSD.ORG Sat Sep 11 21:16:28 2010 Return-Path: Delivered-To: freebsd-jail@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 173751065672 for ; Sat, 11 Sep 2010 21:16:28 +0000 (UTC) (envelope-from klaus@mx.7he.at) Received: from smtp-03.sil.at (smtp-03.sil.at [78.142.186.22]) by mx1.freebsd.org (Postfix) with ESMTP id C37D68FC12 for ; Sat, 11 Sep 2010 21:16:27 +0000 (UTC) Received: from mx.7he.at ([86.59.13.138]) by smtp-03.sil.at with esmtps (TLS-1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.63) (envelope-from ) id 1OuXQv-0008MX-Rg for freebsd-jail@freebsd.org; Sat, 11 Sep 2010 23:16:25 +0200 Received: from mx.7he.at (mx.7he.at [86.59.13.138]) by mx.7he.at (8.14.3/8.14.3) with ESMTP id o8BLFt1Z006036 for ; Sat, 11 Sep 2010 21:15:55 GMT (envelope-from klaus@mx.7he.at) Received: (from klaus@localhost) by mx.7he.at (8.14.3/8.14.3/Submit) id o8BLFtkH006035 for freebsd-jail@freebsd.org; Sat, 11 Sep 2010 21:15:55 GMT (envelope-from klaus) Date: Sat, 11 Sep 2010 21:15:55 +0000 From: "Klaus P. Ohrhallinger" To: freebsd-jail@freebsd.org Message-ID: <20100911211555.GB14592@mx.7he.at> References: <4C8B4DF3.9050209@snap.net.nz> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4C8B4DF3.9050209@snap.net.nz> User-Agent: Mutt/1.4.2.3i X-Political-Attitude: Anarchistic X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED, T_RP_MATCHES_RCVD autolearn=unavailable version=3.3.0 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on mx.7he.at X-Scan-Signature: db536feaa0166abc2d1db13d7d51d3a9 Subject: Re: Jail hot migration X-BeenThere: freebsd-jail@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion about FreeBSD jail\(8\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Sep 2010 21:16:28 -0000 On Sat, Sep 11, 2010 at 09:37:55PM +1200, Peter Toth wrote: > That VPS solution is exactly what I was thinking of. > > Klaus, will this be licensed under BSD? How far is the source release? > Hello; It will be BSD licensed. I am going to release it in october, but it still has to be considered highly experimental, and there are still many points where virtualization doesn't take effect yet. Regards, Klaus -- Alle sagten ''es geht nicht''. Dann kam einer, der wusste das nicht. Der hat's einfach gemacht.