Date: Fri, 12 Apr 2019 10:10:12 +0300 From: Alexander Zagrebin <alex@zagrebin.ru> To: freebsd-ports@freebsd.org Cc: Jan Bramkamp <crest@rlwinm.de>, Jan Bramkamp <crest@rlwinm.de> Subject: Re: python 3 subprocess performance Message-ID: <20190412101012.4142854f@vm2.home.zagrebin.ru> In-Reply-To: <9729db47-12c4-caf4-cdcf-1913dab73c8e@rlwinm.de> References: <20190411161649.1b740d21@vm2.home.zagrebin.ru> <8f3f8413-60f2-bb03-a6b4-4f6364cdc3df@rlwinm.de> <20190411143926.5rg4jskmodt4shhi@laparbeit> <9729db47-12c4-caf4-cdcf-1913dab73c8e@rlwinm.de>
next in thread | previous in thread | raw e-mail | index | archive | help
В Thu, 11 Apr 2019 17:32:42 +0200 Jan Bramkamp <crest@rlwinm.de> пишет: > The reason is that that python does something stupid (tm). It tries to > close all file descriptors (except a few whitelisted ones) up to the > maximum file descriptor number. It does this by asking the kernel for > the maximum possible number and closing everything it doesn't want to > keep. Some time later someone came up with an optimization (read the > open file descriptors from /dev/fd). All of this pain and suffering is > caused by good old Ulrich Drepper braindamage: > https://sourceware.org/bugzilla/show_bug.cgi?id=10353. > > Most Linux distros have lower default file descriptor limits than > FreeBSD making this workaround less painful. The correct solution > would be to teach python3 about closefrom(2). Thank you for hint and testing! Indeed the problem is in closing more than 400,000 file descriptors in loop. It seems that all current versions of Python are affected. Python2 uses False as default value for the close_fds parameter of the Popen constructor, so this issue is mostly not visible. Python3 has changed this default to True. As Jan Bramkamp suggested, I've wrote simple patch to fix an issue (see attached file). It seems the problem has gone. -- Alexander Zagrebin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20190412101012.4142854f>