From owner-freebsd-ports@freebsd.org Fri Apr 12 07:10:18 2019 Return-Path: Delivered-To: freebsd-ports@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5F48D1574B98 for ; Fri, 12 Apr 2019 07:10:18 +0000 (UTC) (envelope-from alex@zagrebin.ru) Received: from mail.zagrebin.ru (mail.zagrebin.ru [IPv6:2001:470:1f15:30e::1:2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 60BB08EA70 for ; Fri, 12 Apr 2019 07:10:17 +0000 (UTC) (envelope-from alex@zagrebin.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zagrebin.ru ; s=mail; h=Content-Type:MIME-Version:References:In-Reply-To:Message-ID: Subject:Cc:To:From:Date:Sender:Reply-To:Content-Transfer-Encoding:Content-ID: Content-Description:Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc :Resent-Message-ID:List-Id:List-Help:List-Unsubscribe:List-Subscribe: List-Post:List-Owner:List-Archive; bh=/I4NuHxuxL7gC9Bg/+8MtYch/RNZx1QEmJBzjmcan5c=; b=XCVs0VYwvcKKCJclG85d9DJWgr MmCZOeql2w4SvPYhBDBAzJm/fMHHzz0VmN01S/6Xk5UNkoLom5NW6k+pjJ5li9F6B/klJ2e4EhZz6 RVuMvxDPsdsX8MMhiHc7CRvOlnD0uHQ5FQ1pTX1PJjo97LZtEH/dQ2gRjMjB5J5inhcymE5IuNXBz nDzAfVXhUjnQDjAcvLuq72jagIeWsnZ9o7WpTNhuqW8zxwc3gE3o5lVE85sa+TFJUCRhaxRiF/AHM V7l39EkGQY4tm3LW02Rq3Hpp5O2RyLpRDYuTFHloKELyCHBcVu5SrjZpFWRpiWgccy6fSrpPOqH2Y xwyzt9dw==; Received: from [2001:470:1f15:30e::2] (helo=vm2.home.zagrebin.ru) by mail.zagrebin.ru with esmtpsa (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256) (Exim 4.92 (FreeBSD)) (envelope-from ) id 1hEqKD-000Lef-Au; Fri, 12 Apr 2019 10:10:13 +0300 Date: Fri, 12 Apr 2019 10:10:12 +0300 From: Alexander Zagrebin To: freebsd-ports@freebsd.org Cc: Jan Bramkamp , Jan Bramkamp Subject: Re: python 3 subprocess performance Message-ID: <20190412101012.4142854f@vm2.home.zagrebin.ru> In-Reply-To: <9729db47-12c4-caf4-cdcf-1913dab73c8e@rlwinm.de> References: <20190411161649.1b740d21@vm2.home.zagrebin.ru> <8f3f8413-60f2-bb03-a6b4-4f6364cdc3df@rlwinm.de> <20190411143926.5rg4jskmodt4shhi@laparbeit> <9729db47-12c4-caf4-cdcf-1913dab73c8e@rlwinm.de> X-Mailer: Claws Mail 3.17.3 (GTK+ 2.24.32; amd64-portbld-freebsd11.2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-ports@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Porting software to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 07:10:18 -0000 В Thu, 11 Apr 2019 17:32:42 +0200 Jan Bramkamp пишет: > The reason is that that python does something stupid (tm). It tries to > close all file descriptors (except a few whitelisted ones) up to the > maximum file descriptor number. It does this by asking the kernel for > the maximum possible number and closing everything it doesn't want to > keep. Some time later someone came up with an optimization (read the > open file descriptors from /dev/fd). All of this pain and suffering is > caused by good old Ulrich Drepper braindamage: > https://sourceware.org/bugzilla/show_bug.cgi?id=10353. > > Most Linux distros have lower default file descriptor limits than > FreeBSD making this workaround less painful. The correct solution > would be to teach python3 about closefrom(2). Thank you for hint and testing! Indeed the problem is in closing more than 400,000 file descriptors in loop. It seems that all current versions of Python are affected. Python2 uses False as default value for the close_fds parameter of the Popen constructor, so this issue is mostly not visible. Python3 has changed this default to True. As Jan Bramkamp suggested, I've wrote simple patch to fix an issue (see attached file). It seems the problem has gone. -- Alexander Zagrebin