Date: Fri, 12 Apr 2019 09:36:13 +0200 From: Dima Pasechnik <dimpase+freebsd@gmail.com> To: Alexander Zagrebin <alex@zagrebin.ru> Cc: FreeBSD Ports <freebsd-ports@freebsd.org> Subject: Re: python 3 subprocess performance Message-ID: <CAAWYfq2P5QsSbc2bzQauApF6=ijQYpKCb2HQ1jy-4dNZCOB_=g@mail.gmail.com> In-Reply-To: <20190412101012.4142854f@vm2.home.zagrebin.ru> References: <20190411161649.1b740d21@vm2.home.zagrebin.ru> <8f3f8413-60f2-bb03-a6b4-4f6364cdc3df@rlwinm.de> <20190411143926.5rg4jskmodt4shhi@laparbeit> <9729db47-12c4-caf4-cdcf-1913dab73c8e@rlwinm.de> <20190412101012.4142854f@vm2.home.zagrebin.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, Apr 12, 2019 at 9:11 AM Alexander Zagrebin <alex@zagrebin.ru> wrote= : > > =D0=92 Thu, 11 Apr 2019 17:32:42 +0200 > Jan Bramkamp <crest@rlwinm.de> =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > > > The reason is that that python does something stupid (tm). It tries to > > close all file descriptors (except a few whitelisted ones) up to the > > maximum file descriptor number. It does this by asking the kernel for > > the maximum possible number and closing everything it doesn't want to > > keep. Some time later someone came up with an optimization (read the > > open file descriptors from /dev/fd). All of this pain and suffering is > > caused by good old Ulrich Drepper braindamage: > > https://sourceware.org/bugzilla/show_bug.cgi?id=3D10353. > > > > Most Linux distros have lower default file descriptor limits than > > FreeBSD making this workaround less painful. The correct solution > > would be to teach python3 about closefrom(2). > > Thank you for hint and testing! > > Indeed the problem is in closing more than 400,000 file descriptors in > loop. It seems that all current versions of Python are affected. > Python2 uses False as default value for the close_fds parameter of > the Popen constructor, so this issue is mostly not visible. > Python3 has changed this default to True. > > As Jan Bramkamp suggested, I've wrote simple patch to fix an issue (see > attached file). It seems the problem has gone. The attachment has been stripped out. Could you paste the diff into the mes= sage? If this is a Python issue, shouldn't this be reported upstream, on https://bugs.python.org ? Dima > > -- > Alexander Zagrebin > _______________________________________________ > freebsd-ports@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-ports > To unsubscribe, send any mail to "freebsd-ports-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAAWYfq2P5QsSbc2bzQauApF6=ijQYpKCb2HQ1jy-4dNZCOB_=g>