Date: Mon, 4 Nov 2013 18:33:58 +0200 From: Konstantin Belousov <kostikbel@gmail.com> To: David Wolfskill <david@catwhisker.org> Cc: freebsd-performance@freebsd.org Subject: Re: Reality check? compat.ia32.maxdsiz in jailed "32-bit" environment Message-ID: <20131104163358.GR59496@kib.kiev.ua> In-Reply-To: <20131104154908.GD1711@albert.catwhisker.org> References: <20131104154908.GD1711@albert.catwhisker.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--jsBaDDfOofhewO/R Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Nov 04, 2013 at 07:49:08AM -0800, David Wolfskill wrote: > I am supporting a performance-sensitive software development > environment that: > * Is presently contrained to operate in a FreeBSD/i386 (32-bit) > environment. > * Primarily uses svn, gcc (with various target architectures), and bmake. > * Has more main memory than FreeBSD appears to be able to make > constructive use of for this workload, but not enough to just put the > entire set of files involved in a malloc-based memory disk. >=20 > Our current implementation involves running the processes involved > in a "full system image" jail on FreeBSD/amd64 8.x hosts.(We presently = =20 > have but one jail on each host; the developers are, in general, only = =20 > permitted to login to the jailed environment, not the host itself. > Several develeopers are usually logged in for each jail, and concurrent > builds are not uncommon.) >=20 > My "test" environment (which I intend to provide for deployment ... > soon) is similar, but switches the host to FreeBSD/amd64 9.2-S while > leaving the jailed image unchanged; the performance is about 18% > better (using elapsed time of the software build as the salient > metric), and system CPU is significantly reduced as well (which is > a bug issue when we have multiple such builds running concurrently). >=20 > I have (also) started experimenting with increasing compat.ia32.maxdsiz > beyond its default of 512MB: Initially, I kicked it to 2GB; more > recently, I tried setting it to 3GB. >=20 > While I have not noticed any negative issues from either of the above > non-default settings, I have done but limited testing with concurrent > software builds in my test environment. Our swap usage is neglible, and > given that there is a more-than-adequate amount of RAM on the machines, > I wouldn't expect that increasing the compat.ia32.maxdsiz value to > be a constraint even with concurrent builds (as they are in separate > processes, and thus, separate address spaces), but one of the reasons > for this note is to ask for a reality check on that perception. >=20 > Oddly enough, I have not noted any statistically significant > performance change from setting compat.ia32.maxdsiz. On the other > hand, I have noticed one significant behavioral change: some svn > merge operations that had been failing (for alck of available memory) > with the default (512MB) compat.ia32.maxdsiz no longer fail when > it is set to 2GB. (I have not actually tested that for 3GB, though > I expect that it would also work.) Does this make sense to anyone > else? (I have a vague memory of gcc altering its behavior to take > advantage of additional memory if it finds "enough" memory available > -- but it's been a couple of decades since I poked around in the > internals of gcc, and that's not actually an experience I'd prefer > to repeat.) >=20 > Thanks for any insights you're willing to share! For 32 bit kernel, the user VA layout is empty - text - data - bss ->|<- mmap area, including dso -->|<- stack ->| 0 maxdsiz 3G-68M 3G-4M For the 64bit kernel executing 32bit process, the layout is the same, but the top of the user VA is at 4G, and bottom of stack is at 4G-64M. By increasing maxdsiz to 3G, you allow for the larger heap in exchange for the limiting the space for mmap(2) to use, and shared libraries to load. In your load, where toolchain binaries are linked statically, the 'dso' is put out of the scope for memory-hungry processes. This would not work on the 32bit kernel, because 3G is the total address space available to the user process. --jsBaDDfOofhewO/R Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJSd8x1AAoJEJDCuSvBvK1BQ9YP/0sqWjn5H/iG7LUor1FdiyUi qrr6HuFUqrQtZko7I4gow9cmbaqC9tV3lTI9eXSleBbItH1Y3JW/jV4/1UDS5xpD OwlbD21jypvA5lzHT6y9SLTedv611qf6nhHTI1nyzfFHEQDXxRbJwk3Wv5Gg9IJY NgEuXSDw8LuuCUCBICiprlAywgiNd9jOI6JvIz2ALyVodw3vW6tatW/Q06Az1my0 p2Vh6y2ifyxg/HWNxetnMwhO4sKhCdxl1jywcwd+l4CKqTQJsfLYlWaOTz8resT7 DDPnfmLtSc9xwCVp43BxnWzXtN4/irjS9QP5GOZ63QGf3eOmEXp9K6WVwEtcYzYK GxP2ybHOGhh04yopmiwl3gmfLNNhSZ+41K/1LuDFrl/6TYQHSZ5FoYPOwSz+c84l AWzss0HRjEkue3vQZFNRiSsA+x8KhC/02Mf/6rO0F5/dKCGcJcpx275sHZhH4Lzu 0gj9c9f7opEPRX4H/5IXOqjsxacQJB9MBMmO5pkMAU7EwbsPVCuLdJzMfEyPQzxd tO/6jqzQ7l/WUgAC1STL2UEHgBY1l9oSDtUqUoF89vTFeRbLuA9LdKfUuCC66nKY g8uXV7QtqyRLDJsvYG1LTKOxtCi3wALqbdUrL8XwNQHK6VqYNEEjLcefCJogEyvX WyC0mG/fFqybbP0bxQ29 =0kx0 -----END PGP SIGNATURE----- --jsBaDDfOofhewO/R--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20131104163358.GR59496>