From owner-freebsd-virtualization@FreeBSD.ORG Sun Feb 8 11:43:18 2009 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E2A4106564A for ; Sun, 8 Feb 2009 11:43:18 +0000 (UTC) (envelope-from ragnar@gatorhole.com) Received: from svosch.gatorhole.com (lonn.org [213.136.43.225]) by mx1.freebsd.org (Postfix) with ESMTP id 2D20F8FC1A for ; Sun, 8 Feb 2009 11:43:17 +0000 (UTC) (envelope-from ragnar@gatorhole.com) Received: from 90-227-60-174-no53.tbcn.telia.com (90-227-60-174-no53.tbcn.telia.com [90.227.60.174]) (Authenticated sender: ragnar@gatorhole.com) by svosch.gatorhole.com (Postfix) with ESMTPSA id D68EDBE578; Sun, 8 Feb 2009 12:47:18 +0100 (CET) Message-ID: <498EC554.4020905@gatorhole.com> Date: Sun, 08 Feb 2009 12:43:16 +0100 From: Ragnar Lonn User-Agent: Thunderbird 2.0.0.19 (Macintosh/20081209) MIME-Version: 1.0 To: Julian Elischer References: <498DF945.3000702@gatorhole.com> <498E0797.4040002@elischer.org> In-Reply-To: <498E0797.4040002@elischer.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-virtualization@freebsd.org Subject: Re: More open sockets with vimages? X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Feb 2009 11:43:18 -0000 Julian Elischer wrote: > sockets are a global resource that are assigned to vimages. > However the amount of sockets available are tunable. > how many are we talking about here? 100,000+ sockets. It seems to me like there is a need to be able to handle *many* open network connections as servers get more and more CPU cores, memory, and higher-speed network interfaces, but most people claim that it is very hard to get 100k open sockets working nicely on a single machine, even on a modern OS (though I've found a couple of people that say they can, also, on Linux systems). Ok if 65k sockets is the normal limit per process and per IP address, but for the whole OS, it just seems strange to limit things to 65k (or less). /Ragnar From owner-freebsd-virtualization@FreeBSD.ORG Sun Feb 8 13:29:13 2009 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 68B34106566B for ; Sun, 8 Feb 2009 13:29:13 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id CD3F98FC0A for ; Sun, 8 Feb 2009 13:29:12 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from cicely5.cicely.de ([10.1.1.7]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id n18D4er0045359 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sun, 8 Feb 2009 14:04:40 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (cicely7.cicely.de [10.1.1.9]) by cicely5.cicely.de (8.14.2/8.14.2) with ESMTP id n18D4bC2015646 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 8 Feb 2009 14:04:37 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (localhost [127.0.0.1]) by cicely7.cicely.de (8.14.2/8.14.2) with ESMTP id n18D4btZ038125; Sun, 8 Feb 2009 14:04:37 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) Received: (from ticso@localhost) by cicely7.cicely.de (8.14.2/8.14.2/Submit) id n18D4a8l038124; Sun, 8 Feb 2009 14:04:36 +0100 (CET) (envelope-from ticso) Date: Sun, 8 Feb 2009 14:04:36 +0100 From: Bernd Walter To: Ragnar Lonn Message-ID: <20090208130435.GL32126@cicely7.cicely.de> References: <498DF945.3000702@gatorhole.com> <498E0797.4040002@elischer.org> <498EC554.4020905@gatorhole.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <498EC554.4020905@gatorhole.com> X-Operating-System: FreeBSD cicely7.cicely.de 7.0-STABLE i386 User-Agent: Mutt/1.5.11 X-Spam-Status: No, score=-4.3 required=5.0 tests=ALL_TRUSTED=-1.8, AWL=0.052, BAYES_00=-2.599 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on spamd.cicely.de X-Mailman-Approved-At: Sun, 08 Feb 2009 13:34:47 +0000 Cc: Julian Elischer , freebsd-virtualization@freebsd.org Subject: Re: More open sockets with vimages? X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ticso@cicely.de List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Feb 2009 13:29:13 -0000 On Sun, Feb 08, 2009 at 12:43:16PM +0100, Ragnar Lonn wrote: > Julian Elischer wrote: > >sockets are a global resource that are assigned to vimages. > >However the amount of sockets available are tunable. > >how many are we talking about here? > > 100,000+ sockets. It seems to me like there is a need to be able to > handle *many* open network connections as servers get more and more CPU > cores, memory, and higher-speed network interfaces, but most people > claim that it is very hard to get 100k open sockets working nicely on a > single machine, even on a modern OS (though I've found a couple of > people that say they can, also, on Linux systems). Ok if 65k sockets is > the normal limit per process and per IP address, but for the whole OS, > it just seems strange to limit things to 65k (or less). This is simple maths: 100k Sockets with 32k TX and 64k RX buffer take 9G Memory. Just buffer space, not to mention socket state, ... On i386 this is limited by kmem, which defaults to IIRC 512MB and is limited by 32bit virtual address space on i386. On amd64 depending on the OS version you can have a kmem of slighty less than 2G max or several GB. Nevertheless you are still limited with physical RAM. Smaller buffers are possible, but usually people want larger buffers to keep up with recent line speeds. Today buffer sizes can be dynamic - don't know the exact details, but you should keep in mind that 32k/96k is already quite small for many purposes. -- B.Walter http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm. From owner-freebsd-virtualization@FreeBSD.ORG Sun Feb 8 13:46:26 2009 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E56D1065696 for ; Sun, 8 Feb 2009 13:46:26 +0000 (UTC) (envelope-from ragnar@gatorhole.com) Received: from svosch.gatorhole.com (lonn.org [213.136.43.225]) by mx1.freebsd.org (Postfix) with ESMTP id 4A7828FC0A for ; Sun, 8 Feb 2009 13:46:26 +0000 (UTC) (envelope-from ragnar@gatorhole.com) Received: from 90-227-60-174-no53.tbcn.telia.com (90-227-60-174-no53.tbcn.telia.com [90.227.60.174]) (Authenticated sender: ragnar@gatorhole.com) by svosch.gatorhole.com (Postfix) with ESMTPSA id 23422BE577; Sun, 8 Feb 2009 14:50:28 +0100 (CET) Message-ID: <498EE22E.7020005@gatorhole.com> Date: Sun, 08 Feb 2009 14:46:22 +0100 From: Ragnar Lonn User-Agent: Thunderbird 2.0.0.19 (Macintosh/20081209) MIME-Version: 1.0 To: ticso@cicely.de References: <498DF945.3000702@gatorhole.com> <498E0797.4040002@elischer.org> <498EC554.4020905@gatorhole.com> <20090208130435.GL32126@cicely7.cicely.de> In-Reply-To: <20090208130435.GL32126@cicely7.cicely.de> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Julian Elischer , freebsd-virtualization@freebsd.org Subject: Re: More open sockets with vimages? X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Feb 2009 13:46:27 -0000 Bernd Walter wrote: > This is simple maths: > 100k Sockets with 32k TX and 64k RX buffer take 9G Memory. > Just buffer space, not to mention socket state, ... > On i386 this is limited by kmem, which defaults to IIRC 512MB and > is limited by 32bit virtual address space on i386. > On amd64 depending on the OS version you can have a kmem of slighty > less than 2G max or several GB. > Nevertheless you are still limited with physical RAM. > Smaller buffers are possible, but usually people want larger buffers > to keep up with recent line speeds. > Today buffer sizes can be dynamic - don't know the exact details, but > you should keep in mind that 32k/96k is already quite small for > many purposes. > But physical memory is cheap, and most low-end machines can have 16G or more today. Is it just a matter of having enough RAM and a 64-bit OS then? How much is "several GB [kmem]" that you mention above? /Ragnar From owner-freebsd-virtualization@FreeBSD.ORG Sun Feb 8 14:42:03 2009 Return-Path: Delivered-To: freebsd-virtualization@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 09C6E1065673 for ; Sun, 8 Feb 2009 14:42:03 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from raven.bwct.de (raven.bwct.de [85.159.14.73]) by mx1.freebsd.org (Postfix) with ESMTP id 6B4408FC12 for ; Sun, 8 Feb 2009 14:42:02 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from cicely5.cicely.de ([10.1.1.7]) by raven.bwct.de (8.13.4/8.13.4) with ESMTP id n18Efwta047095 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sun, 8 Feb 2009 15:41:59 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (cicely7.cicely.de [10.1.1.9]) by cicely5.cicely.de (8.14.2/8.14.2) with ESMTP id n18Efueu018202 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sun, 8 Feb 2009 15:41:56 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (localhost [127.0.0.1]) by cicely7.cicely.de (8.14.2/8.14.2) with ESMTP id n18Efuxl038391; Sun, 8 Feb 2009 15:41:56 +0100 (CET) (envelope-from ticso@cicely7.cicely.de) Received: (from ticso@localhost) by cicely7.cicely.de (8.14.2/8.14.2/Submit) id n18EftUk038390; Sun, 8 Feb 2009 15:41:55 +0100 (CET) (envelope-from ticso) Date: Sun, 8 Feb 2009 15:41:55 +0100 From: Bernd Walter To: Ragnar Lonn Message-ID: <20090208144155.GN32126@cicely7.cicely.de> References: <498DF945.3000702@gatorhole.com> <498E0797.4040002@elischer.org> <498EC554.4020905@gatorhole.com> <20090208130435.GL32126@cicely7.cicely.de> <498EE22E.7020005@gatorhole.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <498EE22E.7020005@gatorhole.com> X-Operating-System: FreeBSD cicely7.cicely.de 7.0-STABLE i386 User-Agent: Mutt/1.5.11 X-Spam-Status: No, score=-4.3 required=5.0 tests=ALL_TRUSTED=-1.8, AWL=0.052, BAYES_00=-2.599 autolearn=ham version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on spamd.cicely.de Cc: ticso@cicely.de, freebsd-virtualization@freebsd.org, Julian Elischer Subject: Re: More open sockets with vimages? X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: ticso@cicely.de List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 08 Feb 2009 14:42:03 -0000 On Sun, Feb 08, 2009 at 02:46:22PM +0100, Ragnar Lonn wrote: > Bernd Walter wrote: > >This is simple maths: > >100k Sockets with 32k TX and 64k RX buffer take 9G Memory. > >Just buffer space, not to mention socket state, ... > >On i386 this is limited by kmem, which defaults to IIRC 512MB and > >is limited by 32bit virtual address space on i386. > >On amd64 depending on the OS version you can have a kmem of slighty > >less than 2G max or several GB. > >Nevertheless you are still limited with physical RAM. > >Smaller buffers are possible, but usually people want larger buffers > >to keep up with recent line speeds. > >Today buffer sizes can be dynamic - don't know the exact details, but > >you should keep in mind that 32k/96k is already quite small for > >many purposes. > > > > But physical memory is cheap, and most low-end machines can have 16G or > more today. Is it just a matter of having enough RAM and a 64-bit OS > then? How much is "several GB [kmem]" that you mention above? AFAIK it is the only limitation - people are using 100k+ sockets since at least FreeBSD-4, but with several restrictions because of memory. It mostly depends on your application and network topology to your peers. Don't know where the current kmem limits exactly are - AFAIK kmem is hold within KVA and KVA is limited by a static map size. It has been widely discussed recently, because ZFS loves a large kmem. -- B.Walter http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.