From owner-freebsd-ports@freebsd.org Fri Apr 12 07:45:34 2019 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4BCEF15756B3 for ; Fri, 12 Apr 2019 07:45:34 +0000 (UTC) (envelope-from alex@zagrebin.ru) Received: from mail.zagrebin.ru (mail.zagrebin.ru [IPv6:2001:470:1f15:30e::1:2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8B55A8FC90 for ; Fri, 12 Apr 2019 07:45:33 +0000 (UTC) (envelope-from alex@zagrebin.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zagrebin.ru ; s=mail; h=Content-Transfer-Encoding:Content-Type:MIME-Version:References: In-Reply-To:Message-ID:Subject:To:From:Date:Sender:Reply-To:Cc:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=iCA78iufQOz3/haVqp7eFUxMu59djXcBM3oqKnwSqbU=; b=bUkaNAm8XkdbQh0hr0CzWTnuLt oN9Y9HDAXC7bUIypIIzbBOG6R+lQKbC+RKFS/RR2N2QzHKC+1s6IPgPcGfim7tuVLHyPJh//qRXlA ZrwjVyNiCYO5lfXn7AsT0DQdDNqTmmkjO7bCtVLam8k2cnjt1xA4yKOqGgoLGnzuTCy87PqsXs1Hm OluB2p/R/sWy7lIj3v9oPuPMwLi+NGXBhgEirXEV3rtE3vWK1M4ngUaaBW65E1rZOQN+iwu8gY+VG TBlCIPPGWexvp64JVWKR2hfp4Mup7z3f1NN6w5NhDEFDLOYZGSFvsy4tTbl7qUyVhqYRfdsp9qBeo 8caURbwA==; Received: from [2001:470:1f15:30e::2] (helo=vm2.home.zagrebin.ru) by mail.zagrebin.ru with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.92 (FreeBSD)) (envelope-from ) id 1hEqsN-000Djh-Q0 for freebsd-ports@freebsd.org; Fri, 12 Apr 2019 10:45:31 +0300 Date: Fri, 12 Apr 2019 10:45:31 +0300 From: Alexander Zagrebin To: freebsd-ports@freebsd.org Subject: Re: python 3 subprocess performance Message-ID: <20190412104531.7b492a3c@vm2.home.zagrebin.ru> In-Reply-To: References: <20190411161649.1b740d21@vm2.home.zagrebin.ru> <8f3f8413-60f2-bb03-a6b4-4f6364cdc3df@rlwinm.de> <20190411143926.5rg4jskmodt4shhi@laparbeit> <9729db47-12c4-caf4-cdcf-1913dab73c8e@rlwinm.de> <20190412101012.4142854f@vm2.home.zagrebin.ru> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; amd64-portbld-freebsd11.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 07:45:34 -0000 В Fri, 12 Apr 2019 09:36:13 +0200 Dima Pasechnik пишет: > On Fri, Apr 12, 2019 at 9:11 AM Alexander Zagrebin > wrote: > > > > В Thu, 11 Apr 2019 17:32:42 +0200 > > Jan Bramkamp пишет: > > > > > The reason is that that python does something stupid (tm). It > > > tries to close all file descriptors (except a few whitelisted > > > ones) up to the maximum file descriptor number. It does this by > > > asking the kernel for the maximum possible number and closing > > > everything it doesn't want to keep. Some time later someone came > > > up with an optimization (read the open file descriptors > > > from /dev/fd). All of this pain and suffering is caused by good > > > old Ulrich Drepper braindamage: > > > https://sourceware.org/bugzilla/show_bug.cgi?id=10353. > > > > > > Most Linux distros have lower default file descriptor limits than > > > FreeBSD making this workaround less painful. The correct solution > > > would be to teach python3 about closefrom(2). > > > > Thank you for hint and testing! > > > > Indeed the problem is in closing more than 400,000 file descriptors > > in loop. It seems that all current versions of Python are affected. > > Python2 uses False as default value for the close_fds parameter of > > the Popen constructor, so this issue is mostly not visible. > > Python3 has changed this default to True. > > > > As Jan Bramkamp suggested, I've wrote simple patch to fix an issue > > (see attached file). It seems the problem has gone. > > The attachment has been stripped out. Could you paste the diff into > the message? Yes, sure. --- Modules/_posixsubprocess.c.orig 2018-12-24 00:37:14.000000000 +0300 +++ Modules/_posixsubprocess.c 2019-04-12 09:25:21.549389000 +0300 @@ -235,11 +235,15 @@ _close_fds_by_brute_force(long start_fd, } start_fd = keep_fd + 1; } +#if defined(__FreeBSD__) + closefrom(start_fd); +#else if (start_fd <= end_fd) { for (fd_num = start_fd; fd_num < end_fd; ++fd_num) { close(fd_num); } } +#endif } > If this is a Python issue, shouldn't this be reported upstream, on > https://bugs.python.org ? May be. Rather, it is a FreeBSD-specific optimization. -- Alexander Zagrebin