From owner-freebsd-performance@FreeBSD.ORG Tue Aug 12 19:37:14 2014 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 30344A8; Tue, 12 Aug 2014 19:37:14 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 07FE125D0; Tue, 12 Aug 2014 19:37:14 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id EB40BB949; Tue, 12 Aug 2014 15:37:12 -0400 (EDT) From: John Baldwin To: freebsd-current@freebsd.org Subject: Re: PostgreSQL performance on FreeBSD Date: Tue, 12 Aug 2014 14:09:40 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <20140627125613.GT93733@kib.kiev.ua> <20140716132938.GB93733@kib.kiev.ua> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201408121409.40653.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 12 Aug 2014 15:37:13 -0400 (EDT) X-Mailman-Approved-At: Tue, 12 Aug 2014 19:38:04 +0000 Cc: Konstantin Belousov , Adrian Chadd , performance@freebsd.org, "current@freebsd.org" X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Aug 2014 19:37:14 -0000 On Wednesday, July 16, 2014 1:52:45 pm Adrian Chadd wrote: > Hi! > > > On 16 July 2014 06:29, Konstantin Belousov wrote: > > On Fri, Jun 27, 2014 at 03:56:13PM +0300, Konstantin Belousov wrote: > >> Hi, > >> I did some measurements and hacks to see about the performance and > >> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD > >> Foundation. > >> > >> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf. > >> The uncommitted patches, referenced in the article, are available as > >> https://kib.kiev.ua/kib/pig1.patch.txt > >> https://kib.kiev.ua/kib/patch-2 > > > > A followup to the original paper. > > > > Most importantly, I identified the cause for the drop on the graph > > after the 30 clients, which appeared to be the debugging version > > of malloc(3) in libc. > > > > Also there are some updates on the patches. > > > > New version of the paper is available at > > https://www.kib.kiev.ua/kib/pgsql_perf_v2.0.pdf > > The changes are marked as 'update for version 2.0'. > > Would you mind trying a default (non-PRODUCTION) build, but with junk > filling turned off? > > adrian@adrian-hackbox:~ % ls -l /etc/malloc.conf > > lrwxr-xr-x 1 root wheel 10 Jun 24 04:37 /etc/malloc.conf -> junk:false > > That fixes almost all of the malloc debug performance issues that I > see without having to recompile. > > I'd like to know if you see any after that. OTOH, I have actually seen junk profiling _improve_ performance in certain cases as it forces promotion of allocated pages to superpages since all pages are dirtied. (I have a local hack that adds a new malloc option to explicitly memset() new pages allocated via mmap() that gives the same benefit without the junking overheadon each malloc() / free(), but it does increase physical RAM usage.) -- John Baldwin From owner-freebsd-performance@FreeBSD.ORG Tue Aug 12 21:36:27 2014 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6D52995; Tue, 12 Aug 2014 21:36:27 +0000 (UTC) Received: from mail-qa0-x233.google.com (mail-qa0-x233.google.com [IPv6:2607:f8b0:400d:c00::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EE673262D; Tue, 12 Aug 2014 21:36:26 +0000 (UTC) Received: by mail-qa0-f51.google.com with SMTP id k15so9140923qaq.38 for ; Tue, 12 Aug 2014 14:36:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=P6HwJjuC1lBpaSdeq0qKfWJgobtnaZR9SENeJE8kAaY=; b=gYYfTM/jbnHY+TGEIP0z2Jd8EBz5JOT5Y3wWGegdYHJunP2FKshJdOqsYPbyEQtnHM XpjfT147RCUI2gCjvfsEJhwU69L075TBRh+ty+6Pb0K48UlmOg3A3imcgCJF3zEM7/M3 vHJ+/4GkHfyLvLB8oCmEOWWKuhiWIzBO/QRwfDd7UTq4I0QGRugFS4bcIqnz/HdPAIYS 519UQkdj94wTkKhz4jCCO9E5LGb+qcBOF1/cr4A4X/C23tiqc5Ks3FeAWKKv5S5dfMsj Tuhse0EJ2D49xB56c3ZZ8wHsk5Uqw7rTlsNqurvMe1bpGkoGoVK+lZOUPg4i4vxCbDtG fecg== MIME-Version: 1.0 X-Received: by 10.140.27.144 with SMTP id 16mr748440qgx.18.1407879386162; Tue, 12 Aug 2014 14:36:26 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.39.139 with HTTP; Tue, 12 Aug 2014 14:36:26 -0700 (PDT) In-Reply-To: <201408121409.40653.jhb@freebsd.org> References: <20140627125613.GT93733@kib.kiev.ua> <20140716132938.GB93733@kib.kiev.ua> <201408121409.40653.jhb@freebsd.org> Date: Tue, 12 Aug 2014 14:36:26 -0700 X-Google-Sender-Auth: Ul21HWjPHQrdSrY658-LIWJGqFw Message-ID: Subject: Re: PostgreSQL performance on FreeBSD From: Adrian Chadd To: John Baldwin Content-Type: text/plain; charset=UTF-8 Cc: Konstantin Belousov , freebsd-current , performance@freebsd.org, "current@freebsd.org" X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Aug 2014 21:36:27 -0000 On 12 August 2014 11:09, John Baldwin wrote: > On Wednesday, July 16, 2014 1:52:45 pm Adrian Chadd wrote: >> Hi! >> >> >> On 16 July 2014 06:29, Konstantin Belousov wrote: >> > On Fri, Jun 27, 2014 at 03:56:13PM +0300, Konstantin Belousov wrote: >> >> Hi, >> >> I did some measurements and hacks to see about the performance and >> >> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD >> >> Foundation. >> >> >> >> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf. >> >> The uncommitted patches, referenced in the article, are available as >> >> https://kib.kiev.ua/kib/pig1.patch.txt >> >> https://kib.kiev.ua/kib/patch-2 >> > >> > A followup to the original paper. >> > >> > Most importantly, I identified the cause for the drop on the graph >> > after the 30 clients, which appeared to be the debugging version >> > of malloc(3) in libc. >> > >> > Also there are some updates on the patches. >> > >> > New version of the paper is available at >> > https://www.kib.kiev.ua/kib/pgsql_perf_v2.0.pdf >> > The changes are marked as 'update for version 2.0'. >> >> Would you mind trying a default (non-PRODUCTION) build, but with junk >> filling turned off? >> >> adrian@adrian-hackbox:~ % ls -l /etc/malloc.conf >> >> lrwxr-xr-x 1 root wheel 10 Jun 24 04:37 /etc/malloc.conf -> junk:false >> >> That fixes almost all of the malloc debug performance issues that I >> see without having to recompile. >> >> I'd like to know if you see any after that. > > OTOH, I have actually seen junk profiling _improve_ performance in certain > cases as it forces promotion of allocated pages to superpages since all pages > are dirtied. (I have a local hack that adds a new malloc option to explicitly > memset() new pages allocated via mmap() that gives the same benefit without > the junking overheadon each malloc() / free(), but it does increase physical > RAM usage.) Hm. this isn't a jemalloc config option? -a From owner-freebsd-performance@FreeBSD.ORG Wed Aug 13 09:59:00 2014 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9C4D21DC; Wed, 13 Aug 2014 09:59:00 +0000 (UTC) Received: from theravensnest.org (theraven.freebsd.your.org [216.14.102.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "cloud.theravensnest.org", Issuer "StartCom Class 1 Primary Intermediate Server CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 61D1F2D5B; Wed, 13 Aug 2014 09:58:59 +0000 (UTC) Received: from [192.168.0.7] (cpc14-cmbg15-2-0-cust307.5-4.cable.virginm.net [82.26.1.52]) (authenticated bits=0) by theravensnest.org (8.14.7/8.14.7) with ESMTP id s7D9wliD006016 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 13 Aug 2014 09:58:49 GMT (envelope-from theraven@FreeBSD.org) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: PostgreSQL performance on FreeBSD From: David Chisnall In-Reply-To: <201408121409.40653.jhb@freebsd.org> Date: Wed, 13 Aug 2014 10:58:42 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20140627125613.GT93733@kib.kiev.ua> <20140716132938.GB93733@kib.kiev.ua> <201408121409.40653.jhb@freebsd.org> To: John Baldwin X-Mailer: Apple Mail (2.1878.6) X-Mailman-Approved-At: Wed, 13 Aug 2014 11:33:26 +0000 Cc: Konstantin Belousov , Adrian Chadd , freebsd-current@freebsd.org, performance@freebsd.org, "current@freebsd.org" X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Aug 2014 09:59:00 -0000 On 12 Aug 2014, at 19:09, John Baldwin wrote: > OTOH, I have actually seen junk profiling _improve_ performance in = certain=20 > cases as it forces promotion of allocated pages to superpages since = all pages=20 > are dirtied. (I have a local hack that adds a new malloc option to = explicitly=20 > memset() new pages allocated via mmap() that gives the same benefit = without=20 > the junking overheadon each malloc() / free(), but it does increase = physical=20 > RAM usage.) Do you get the same effect by adding MAP_ALIGNED_SUPER | = MAP_PREFAULT_READ to the mmap() call in jemalloc? I've been meaning to = try the latter on BERI, as we spend a lot of time bouncing back and = forth between user code and the TLB miss handlers. Given that jemalloc = asks for memory in 8MB chunks (I think via a single mmap call, although = I'm not 100% certain), MAP_ALIGNED_SUPER should have little impact on = any platform. MAP_PREFAULT_READ may cause problems on machines with = limited RAM and no swap (I don't know if the VM subsystem knows that it = can safely discard a zero'd page that has been read but not written - = I'd hope so, but it's been a while since I read that code). It might be that we can make jemalloc autotune whether to use = MAP_PREFAULT_READ depending on some heuristic. I wonder if something as = simple as 'turn it on after the first mmap call' would be enough: = programs that don't use more than 8MB of RAM won't prefault, but after = that the wasted physical memory becomes an increasingly small = percentage. David From owner-freebsd-performance@FreeBSD.ORG Wed Aug 13 16:23:45 2014 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 404B84E2; Wed, 13 Aug 2014 16:23:45 +0000 (UTC) Received: from mail-ig0-x22a.google.com (mail-ig0-x22a.google.com [IPv6:2607:f8b0:4001:c05::22a]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D095D2CE7; Wed, 13 Aug 2014 16:23:44 +0000 (UTC) Received: by mail-ig0-f170.google.com with SMTP id h3so11740309igd.1 for ; Wed, 13 Aug 2014 09:23:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=/LHRks6tSEzobWSKZkHFWORM4xAAJreA0l3Sw73vwxM=; b=t/cahxnlf03Hdk0gjdKYSvU75CBvUYRubYmSDSbZ0G27O3yzt1PvFuYpOL/a+aRjsf ztNttTTnDvXsCUafYKrkLFrvj1ZJgbPgpLG7FHqab/O/UMhuGFM9/wUu1D5WRNJvetSO v4f4+74pbJoQGtemvlshefx4O2YZzl3Y0VY0ULVY/IsRa4nDug0NTTWrHBy8wD/iaMUc e7eoQE4QcTYvef3FGl02Q3/cT/1u2AvTRZdNQtns7M7Jt3pDdoUxWEErv5nY2P85A4De JhjT3coeOCg4h2WVOIs4RrJ8r2m4XJMEOLI0bEaACznD13GfgKmGGfIqhxAY8HFPxpJW lwzQ== MIME-Version: 1.0 X-Received: by 10.50.66.197 with SMTP id h5mr51099519igt.34.1407947024349; Wed, 13 Aug 2014 09:23:44 -0700 (PDT) Received: by 10.43.17.196 with HTTP; Wed, 13 Aug 2014 09:23:44 -0700 (PDT) Reply-To: alc@freebsd.org In-Reply-To: References: <20140627125613.GT93733@kib.kiev.ua> <20140716132938.GB93733@kib.kiev.ua> <201408121409.40653.jhb@freebsd.org> Date: Wed, 13 Aug 2014 11:23:44 -0500 Message-ID: Subject: Re: PostgreSQL performance on FreeBSD From: Alan Cox To: David Chisnall X-Mailman-Approved-At: Wed, 13 Aug 2014 17:05:58 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Adrian Chadd , performance@freebsd.org, "current@freebsd.org" , John Baldwin , freebsd-current , Konstantin Belousov X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Aug 2014 16:23:45 -0000 On Wed, Aug 13, 2014 at 4:58 AM, David Chisnall wrote: > On 12 Aug 2014, at 19:09, John Baldwin wrote: > > > OTOH, I have actually seen junk profiling _improve_ performance in > certain > > cases as it forces promotion of allocated pages to superpages since all > pages > > are dirtied. (I have a local hack that adds a new malloc option to > explicitly > > memset() new pages allocated via mmap() that gives the same benefit > without > > the junking overheadon each malloc() / free(), but it does increase > physical > > RAM usage.) > > Do you get the same effect by adding MAP_ALIGNED_SUPER | MAP_PREFAULT_READ > to the mmap() call in jemalloc? No. MAP_PREFAULT_READ does not allocate physical pages. It establishes mappings to pages that are already allocated. > I've been meaning to try the latter on BERI, as we spend a lot of time > bouncing back and forth between user code and the TLB miss handlers. Given > that jemalloc asks for memory in 8MB chunks (I think via a single mmap > call, although I'm not 100% certain), MAP_ALIGNED_SUPER should have little > impact on any platform. MAP_PREFAULT_READ may cause problems on machines > with limited RAM and no swap (I don't know if the VM subsystem knows that > it can safely discard a zero'd page that has been read but not written - > I'd hope so, but it's been a while since I read that code). > > It might be that we can make jemalloc autotune whether to use > MAP_PREFAULT_READ depending on some heuristic. I wonder if something as > simple as 'turn it on after the first mmap call' would be enough: programs > that don't use more than 8MB of RAM won't prefault, but after that the > wasted physical memory becomes an increasingly small percentage. > > David > > _______________________________________________ > freebsd-current@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Wed Aug 13 17:00:23 2014 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 17BF144F; Wed, 13 Aug 2014 17:00:23 +0000 (UTC) Received: from mail-ig0-x232.google.com (mail-ig0-x232.google.com [IPv6:2607:f8b0:4001:c05::232]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B16C2218B; Wed, 13 Aug 2014 17:00:22 +0000 (UTC) Received: by mail-ig0-f178.google.com with SMTP id uq10so2360817igb.11 for ; Wed, 13 Aug 2014 10:00:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:reply-to:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=9yTd9xmdHRCwOCj9Kajn99AcXNp1hjx1kt3gpWeA27Q=; b=VbDiFa6Nh6+yz6mMJ0qWKqKFYVqohM0KGNDav54JehEkFkKmP4bly/X//J0/69r682 FoqDv9P1n55KmX29HaMqpUX+1T/Fl2KyfV5zETgU4TIYNjKVxcfRHrzrV/5CAGrGUjmT s80OX9pi4DXDnGHUbCbQFOT+OeTOw1QGCDRLtTbA9Zw3+CD7s8s8K3aR7YI6R5T3Vcjt MEJlzGEzUI5RzA15LhTdmCWFODhQk/z5bdiGDHB2Unh8cKQ5whnkU8j3fN0a2LIUuyWf jIL8o+C8efarT3MRmjMHuKVkHhBzc3iwG+6E5k3eSwN0O8BpHQE7KaR4ish4ksT6+tKA 2tww== MIME-Version: 1.0 X-Received: by 10.42.212.146 with SMTP id gs18mr4622270icb.96.1407949222191; Wed, 13 Aug 2014 10:00:22 -0700 (PDT) Received: by 10.43.17.196 with HTTP; Wed, 13 Aug 2014 10:00:22 -0700 (PDT) Reply-To: alc@freebsd.org In-Reply-To: <201408121409.40653.jhb@freebsd.org> References: <20140627125613.GT93733@kib.kiev.ua> <20140716132938.GB93733@kib.kiev.ua> <201408121409.40653.jhb@freebsd.org> Date: Wed, 13 Aug 2014 12:00:22 -0500 Message-ID: Subject: Re: PostgreSQL performance on FreeBSD From: Alan Cox To: John Baldwin X-Mailman-Approved-At: Wed, 13 Aug 2014 19:15:35 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: Konstantin Belousov , Adrian Chadd , freebsd-current , performance@freebsd.org, "current@freebsd.org" X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Aug 2014 17:00:23 -0000 On Tue, Aug 12, 2014 at 1:09 PM, John Baldwin wrote: > On Wednesday, July 16, 2014 1:52:45 pm Adrian Chadd wrote: > > Hi! > > > > > > On 16 July 2014 06:29, Konstantin Belousov wrote: > > > On Fri, Jun 27, 2014 at 03:56:13PM +0300, Konstantin Belousov wrote: > > >> Hi, > > >> I did some measurements and hacks to see about the performance and > > >> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD > > >> Foundation. > > >> > > >> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf. > > >> The uncommitted patches, referenced in the article, are available as > > >> https://kib.kiev.ua/kib/pig1.patch.txt > > >> https://kib.kiev.ua/kib/patch-2 > > > > > > A followup to the original paper. > > > > > > Most importantly, I identified the cause for the drop on the graph > > > after the 30 clients, which appeared to be the debugging version > > > of malloc(3) in libc. > > > > > > Also there are some updates on the patches. > > > > > > New version of the paper is available at > > > https://www.kib.kiev.ua/kib/pgsql_perf_v2.0.pdf > > > The changes are marked as 'update for version 2.0'. > > > > Would you mind trying a default (non-PRODUCTION) build, but with junk > > filling turned off? > > > > adrian@adrian-hackbox:~ % ls -l /etc/malloc.conf > > > > lrwxr-xr-x 1 root wheel 10 Jun 24 04:37 /etc/malloc.conf -> junk:false > > > > That fixes almost all of the malloc debug performance issues that I > > see without having to recompile. > > > > I'd like to know if you see any after that. > > OTOH, I have actually seen junk profiling _improve_ performance in certain > cases as it forces promotion of allocated pages to superpages since all > pages > are dirtied. (I have a local hack that adds a new malloc option to > explicitly > memset() new pages allocated via mmap() that gives the same benefit without > the junking overheadon each malloc() / free(), but it does increase > physical > RAM usage.) > > John, A couple small steps have been taken toward eliminating the need for this hack: the addition of the "page size index" field to struct vm_page and the addition of a similarly named parameter to pmap_enter(). However, at the moment, the only tangible effect is in the automatic prefaulting by mmap(2). Instead of establishing 96 4KB page mappings, the automatic prefaulting establishes 96 page mappings whose size is determined by the size of the physical pages that it finds in the vm object. So, the prefaulting overhead remains constant, but the coverage provided by the automatic prefaulting will vary with the underlying page size. From owner-freebsd-performance@FreeBSD.ORG Thu Aug 14 16:07:21 2014 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id AA4384A5; Thu, 14 Aug 2014 16:07:21 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7FAF729AC; Thu, 14 Aug 2014 16:07:21 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 60873B9B9; Thu, 14 Aug 2014 12:07:20 -0400 (EDT) From: John Baldwin To: Adrian Chadd Subject: Re: PostgreSQL performance on FreeBSD Date: Thu, 14 Aug 2014 11:49:02 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <20140627125613.GT93733@kib.kiev.ua> <201408121409.40653.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="utf-8" Content-Transfer-Encoding: 7bit Message-Id: <201408141149.02488.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 14 Aug 2014 12:07:20 -0400 (EDT) Cc: Konstantin Belousov , freebsd-current , performance@freebsd.org, "current@freebsd.org" X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 16:07:21 -0000 On Tuesday, August 12, 2014 5:36:26 pm Adrian Chadd wrote: > On 12 August 2014 11:09, John Baldwin wrote: > > On Wednesday, July 16, 2014 1:52:45 pm Adrian Chadd wrote: > >> Hi! > >> > >> > >> On 16 July 2014 06:29, Konstantin Belousov wrote: > >> > On Fri, Jun 27, 2014 at 03:56:13PM +0300, Konstantin Belousov wrote: > >> >> Hi, > >> >> I did some measurements and hacks to see about the performance and > >> >> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD > >> >> Foundation. > >> >> > >> >> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf. > >> >> The uncommitted patches, referenced in the article, are available as > >> >> https://kib.kiev.ua/kib/pig1.patch.txt > >> >> https://kib.kiev.ua/kib/patch-2 > >> > > >> > A followup to the original paper. > >> > > >> > Most importantly, I identified the cause for the drop on the graph > >> > after the 30 clients, which appeared to be the debugging version > >> > of malloc(3) in libc. > >> > > >> > Also there are some updates on the patches. > >> > > >> > New version of the paper is available at > >> > https://www.kib.kiev.ua/kib/pgsql_perf_v2.0.pdf > >> > The changes are marked as 'update for version 2.0'. > >> > >> Would you mind trying a default (non-PRODUCTION) build, but with junk > >> filling turned off? > >> > >> adrian@adrian-hackbox:~ % ls -l /etc/malloc.conf > >> > >> lrwxr-xr-x 1 root wheel 10 Jun 24 04:37 /etc/malloc.conf -> junk:false > >> > >> That fixes almost all of the malloc debug performance issues that I > >> see without having to recompile. > >> > >> I'd like to know if you see any after that. > > > > OTOH, I have actually seen junk profiling _improve_ performance in certain > > cases as it forces promotion of allocated pages to superpages since all pages > > are dirtied. (I have a local hack that adds a new malloc option to explicitly > > memset() new pages allocated via mmap() that gives the same benefit without > > the junking overheadon each malloc() / free(), but it does increase physical > > RAM usage.) > > Hm. this isn't a jemalloc config option? No, jemalloc does have a 'zero fill' option, but that runs on every malloc so it is just as expensive as junking. What my hack does is only zero the pages when they are first mmap'd, so subsequent free() / malloc() cycles that reuse the pages do not do any zeroing. -- John Baldwin From owner-freebsd-performance@FreeBSD.ORG Thu Aug 14 16:07:21 2014 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F2F2F415; Thu, 14 Aug 2014 16:07:20 +0000 (UTC) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C711129AB; Thu, 14 Aug 2014 16:07:20 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id ADA57B987; Thu, 14 Aug 2014 12:07:19 -0400 (EDT) From: John Baldwin To: alc@freebsd.org Subject: Re: PostgreSQL performance on FreeBSD Date: Thu, 14 Aug 2014 11:47:45 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20140415; KDE/4.5.5; amd64; ; ) References: <20140627125613.GT93733@kib.kiev.ua> <201408121409.40653.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-15" Content-Transfer-Encoding: 7bit Message-Id: <201408141147.45698.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Thu, 14 Aug 2014 12:07:19 -0400 (EDT) Cc: Konstantin Belousov , Adrian Chadd , freebsd-current , performance@freebsd.org, "current@freebsd.org" X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 16:07:21 -0000 On Wednesday, August 13, 2014 1:00:22 pm Alan Cox wrote: > On Tue, Aug 12, 2014 at 1:09 PM, John Baldwin wrote: > > > On Wednesday, July 16, 2014 1:52:45 pm Adrian Chadd wrote: > > > Hi! > > > > > > > > > On 16 July 2014 06:29, Konstantin Belousov wrote: > > > > On Fri, Jun 27, 2014 at 03:56:13PM +0300, Konstantin Belousov wrote: > > > >> Hi, > > > >> I did some measurements and hacks to see about the performance and > > > >> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD > > > >> Foundation. > > > >> > > > >> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf. > > > >> The uncommitted patches, referenced in the article, are available as > > > >> https://kib.kiev.ua/kib/pig1.patch.txt > > > >> https://kib.kiev.ua/kib/patch-2 > > > > > > > > A followup to the original paper. > > > > > > > > Most importantly, I identified the cause for the drop on the graph > > > > after the 30 clients, which appeared to be the debugging version > > > > of malloc(3) in libc. > > > > > > > > Also there are some updates on the patches. > > > > > > > > New version of the paper is available at > > > > https://www.kib.kiev.ua/kib/pgsql_perf_v2.0.pdf > > > > The changes are marked as 'update for version 2.0'. > > > > > > Would you mind trying a default (non-PRODUCTION) build, but with junk > > > filling turned off? > > > > > > adrian@adrian-hackbox:~ % ls -l /etc/malloc.conf > > > > > > lrwxr-xr-x 1 root wheel 10 Jun 24 04:37 /etc/malloc.conf -> junk:false > > > > > > That fixes almost all of the malloc debug performance issues that I > > > see without having to recompile. > > > > > > I'd like to know if you see any after that. > > > > OTOH, I have actually seen junk profiling _improve_ performance in certain > > cases as it forces promotion of allocated pages to superpages since all > > pages > > are dirtied. (I have a local hack that adds a new malloc option to > > explicitly > > memset() new pages allocated via mmap() that gives the same benefit without > > the junking overheadon each malloc() / free(), but it does increase > > physical > > RAM usage.) > > > > > > John, > > A couple small steps have been taken toward eliminating the need for this > hack: the addition of the "page size index" field to struct vm_page and the > addition of a similarly named parameter to pmap_enter(). However, at the > moment, the only tangible effect is in the automatic prefaulting by > mmap(2). Instead of establishing 96 4KB page mappings, the automatic > prefaulting establishes 96 page mappings whose size is determined by the > size of the physical pages that it finds in the vm object. So, the > prefaulting overhead remains constant, but the coverage provided by the > automatic prefaulting will vary with the underlying page size. Yes, I think what we might actually want is what I mentioned in person at BSDCan: some sort of flag to mmap() that malloc() could use to assume that any reservations are fully used when they are reserved. This would avoid the need to wait for all pages to be dirtied before promotion provides a superpage mapping and would avoid demotions while still allowing the kernel to gracefully fall back to regular pages if a reservation can't be made. -- John Baldwin From owner-freebsd-performance@FreeBSD.ORG Thu Aug 14 18:19:55 2014 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0292CAB1; Thu, 14 Aug 2014 18:19:55 +0000 (UTC) Received: from pp1.rice.edu (proofpoint1.mail.rice.edu [128.42.201.100]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id B8A6A2A8A; Thu, 14 Aug 2014 18:19:54 +0000 (UTC) Received: from pps.filterd (pp1.rice.edu [127.0.0.1]) by pp1.rice.edu (8.14.5/8.14.5) with SMTP id s7EIIL63031150; Thu, 14 Aug 2014 13:19:53 -0500 Received: from mh2.mail.rice.edu (mh2.mail.rice.edu [128.42.201.21]) by pp1.rice.edu with ESMTP id 1nqt2wgy21-1; Thu, 14 Aug 2014 13:19:53 -0500 X-Virus-Scanned: by amavis-2.7.0 at mh2.mail.rice.edu, auth channel Received: from 108-254-203-201.lightspeed.hstntx.sbcglobal.net (108-254-203-201.lightspeed.hstntx.sbcglobal.net [108.254.203.201]) (using TLSv1 with cipher RC4-MD5 (128/128 bits)) (No client certificate requested) (Authenticated sender: alc) by mh2.mail.rice.edu (Postfix) with ESMTPSA id E1A3B5000E5; Thu, 14 Aug 2014 13:19:52 -0500 (CDT) Message-ID: <53ECFDC8.1070200@rice.edu> Date: Thu, 14 Aug 2014 13:19:52 -0500 From: Alan Cox User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:24.0) Gecko/20100101 Thunderbird/24.2.0 MIME-Version: 1.0 To: John Baldwin , alc@freebsd.org Subject: Re: PostgreSQL performance on FreeBSD References: <20140627125613.GT93733@kib.kiev.ua> <201408121409.40653.jhb@freebsd.org> <201408141147.45698.jhb@freebsd.org> In-Reply-To: <201408141147.45698.jhb@freebsd.org> X-Enigmail-Version: 1.6 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 kscore.is_bulkscore=0 kscore.compositescore=0 circleOfTrustscore=0 compositescore=0.629899992726084 urlsuspect_oldscore=0.0298999927260837 suspectscore=10 recipient_domain_to_sender_totalscore=0 phishscore=0 bulkscore=0 kscore.is_spamscore=7.54939022407086e-09 recipient_to_sender_totalscore=0 recipient_domain_to_sender_domain_totalscore=498 rbsscore=0.629899992726084 spamscore=0 recipient_to_sender_domain_totalscore=0 urlsuspectscore=0.9 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=7.0.1-1402240000 definitions=main-1408140213 Cc: Konstantin Belousov , Adrian Chadd , freebsd-current , performance@freebsd.org, "current@freebsd.org" X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Aug 2014 18:19:55 -0000 On 08/14/2014 10:47, John Baldwin wrote: > On Wednesday, August 13, 2014 1:00:22 pm Alan Cox wrote: >> On Tue, Aug 12, 2014 at 1:09 PM, John Baldwin wrote: >> >>> On Wednesday, July 16, 2014 1:52:45 pm Adrian Chadd wrote: >>>> Hi! >>>> >>>> >>>> On 16 July 2014 06:29, Konstantin Belousov wrote: >>>>> On Fri, Jun 27, 2014 at 03:56:13PM +0300, Konstantin Belousov wrote: >>>>>> Hi, >>>>>> I did some measurements and hacks to see about the performance and >>>>>> scalability of PostgreSQL 9.3 on FreeBSD, sponsored by The FreeBSD >>>>>> Foundation. >>>>>> >>>>>> The results are described in https://kib.kiev.ua/kib/pgsql_perf.pdf. >>>>>> The uncommitted patches, referenced in the article, are available as >>>>>> https://kib.kiev.ua/kib/pig1.patch.txt >>>>>> https://kib.kiev.ua/kib/patch-2 >>>>> A followup to the original paper. >>>>> >>>>> Most importantly, I identified the cause for the drop on the graph >>>>> after the 30 clients, which appeared to be the debugging version >>>>> of malloc(3) in libc. >>>>> >>>>> Also there are some updates on the patches. >>>>> >>>>> New version of the paper is available at >>>>> https://www.kib.kiev.ua/kib/pgsql_perf_v2.0.pdf >>>>> The changes are marked as 'update for version 2.0'. >>>> Would you mind trying a default (non-PRODUCTION) build, but with junk >>>> filling turned off? >>>> >>>> adrian@adrian-hackbox:~ % ls -l /etc/malloc.conf >>>> >>>> lrwxr-xr-x 1 root wheel 10 Jun 24 04:37 /etc/malloc.conf -> junk:false >>>> >>>> That fixes almost all of the malloc debug performance issues that I >>>> see without having to recompile. >>>> >>>> I'd like to know if you see any after that. >>> OTOH, I have actually seen junk profiling _improve_ performance in certain >>> cases as it forces promotion of allocated pages to superpages since all >>> pages >>> are dirtied. (I have a local hack that adds a new malloc option to >>> explicitly >>> memset() new pages allocated via mmap() that gives the same benefit without >>> the junking overheadon each malloc() / free(), but it does increase >>> physical >>> RAM usage.) >>> >>> >> John, >> >> A couple small steps have been taken toward eliminating the need for this >> hack: the addition of the "page size index" field to struct vm_page and the >> addition of a similarly named parameter to pmap_enter(). However, at the >> moment, the only tangible effect is in the automatic prefaulting by >> mmap(2). Instead of establishing 96 4KB page mappings, the automatic >> prefaulting establishes 96 page mappings whose size is determined by the >> size of the physical pages that it finds in the vm object. So, the >> prefaulting overhead remains constant, but the coverage provided by the >> automatic prefaulting will vary with the underlying page size. > Yes, I think what we might actually want is what I mentioned in person at > BSDCan: some sort of flag to mmap() that malloc() could use to assume that any > reservations are fully used when they are reserved. This would avoid the need > to wait for all pages to be dirtied before promotion provides a superpage > mapping and would avoid demotions while still allowing the kernel to gracefully > fall back to regular pages if a reservation can't be made. > I agree.