From owner-freebsd-ports@freebsd.org Sun Apr 14 07:25:59 2019 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B6E8F156B468 for ; Sun, 14 Apr 2019 07:25:59 +0000 (UTC) (envelope-from jbeich@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 533A3852B7; Sun, 14 Apr 2019 07:25:59 +0000 (UTC) (envelope-from jbeich@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1354) id 423E811523; Sun, 14 Apr 2019 07:25:59 +0000 (UTC) From: Jan Beich To: Kubilay Kocak Cc: FreeBSD Ports Subject: Re: python 3 subprocess performance References: <20190411161649.1b740d21@vm2.home.zagrebin.ru> <8f3f8413-60f2-bb03-a6b4-4f6364cdc3df@rlwinm.de> <20190411143926.5rg4jskmodt4shhi@laparbeit> <9729db47-12c4-caf4-cdcf-1913dab73c8e@rlwinm.de> <20190412101012.4142854f@vm2.home.zagrebin.ru> <20190412104531.7b492a3c@vm2.home.zagrebin.ru> <800de545-0310-fcda-ebae-a4dffabe47e9@FreeBSD.org> Date: Sun, 14 Apr 2019 09:25:54 +0200 In-Reply-To: <800de545-0310-fcda-ebae-a4dffabe47e9@FreeBSD.org> (Kubilay Kocak's message of "Sun, 14 Apr 2019 14:59:53 +1000") Message-ID: <36ml-oxod-wny@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 533A3852B7 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.94 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-0.99)[-0.992,0]; NEURAL_HAM_SHORT(-0.95)[-0.951,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 14 Apr 2019 07:26:00 -0000 Kubilay Kocak writes: > On 12/04/2019 8:41 pm, Dima Pasechnik wrote: > >> On Fri, Apr 12, 2019 at 9:46 AM Alexander Zagrebin wr= ote: >>> >>> =D0=92 Fri, 12 Apr 2019 09:36:13 +0200 >>> Dima Pasechnik =D0=BF=D0=B8=D1=88=D0=B5=D1= =82: >>> >>>> On Fri, Apr 12, 2019 at 9:11 AM Alexander Zagrebin >>>> wrote: >>>>> >>>>> =D0=92 Thu, 11 Apr 2019 17:32:42 +0200 >>>>> Jan Bramkamp =D0=BF=D0=B8=D1=88=D0=B5=D1=82: >>>>> >>>>>> The reason is that that python does something stupid (tm). It >>>>>> tries to close all file descriptors (except a few whitelisted >>>>>> ones) up to the maximum file descriptor number. It does this by >>>>>> asking the kernel for the maximum possible number and closing >>>>>> everything it doesn't want to keep. Some time later someone came >>>>>> up with an optimization (read the open file descriptors >>>>>> from /dev/fd). All of this pain and suffering is caused by good >>>>>> old Ulrich Drepper braindamage: >>>>>> https://sourceware.org/bugzilla/show_bug.cgi?id=3D10353. >>>>>> >>>>>> Most Linux distros have lower default file descriptor limits than >>>>>> FreeBSD making this workaround less painful. The correct solution >>>>>> would be to teach python3 about closefrom(2). >>>>> >>>>> Thank you for hint and testing! >>>>> >>>>> Indeed the problem is in closing more than 400,000 file descriptors >>>>> in loop. It seems that all current versions of Python are affected. >>>>> Python2 uses False as default value for the close_fds parameter of >>>>> the Popen constructor, so this issue is mostly not visible. >>>>> Python3 has changed this default to True. >>>>> >>>>> As Jan Bramkamp suggested, I've wrote simple patch to fix an issue >>>>> (see attached file). It seems the problem has gone. >>>> >>>> The attachment has been stripped out. Could you paste the diff into >>>> the message? >>> >>> Yes, sure. >>> >>> --- Modules/_posixsubprocess.c.orig 2018-12-24 00:37:14.000000000 >>> +0300 +++ Modules/_posixsubprocess.c 2019-04-12 >>> 09:25:21.549389000 +0300 @@ -235,11 +235,15 @@ >>> _close_fds_by_brute_force(long start_fd, } >>> start_fd =3D keep_fd + 1; >>> } >>> +#if defined(__FreeBSD__) >>> + closefrom(start_fd); >>> +#else >>> if (start_fd <=3D end_fd) { >>> for (fd_num =3D start_fd; fd_num < end_fd; ++fd_num) { >>> close(fd_num); >>> } >>> } >>> +#endif >>> } >>> >>>> If this is a Python issue, shouldn't this be reported upstream, on >>>> https://bugs.python.org ? >>> >>> May be. Rather, it is a FreeBSD-specific optimization. >> >> Well, closefrom() is also available in Darwin (a.k.a. MacOSX :-)), >> OpenBSD and NetBSD. (It's not documented in current MacOSX, but it is >> there, I just checked) FreeBSD was late to the party. Even DragonFly got closefrom() before. ;) Note, closefrom() looks non-atomic on Solaris. http://src.illumos.org/source/xref/illumos-gate/usr/src/lib/libc/port/gen/c= losefrom.c > Issue exists for this: > > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D221700 Oh, nice. I've recently it hit via www/firefox build. https://bugzilla.mozilla.org/show_bug.cgi?id=3D1507655