From owner-freebsd-performance@freebsd.org Mon Jun 20 06:58:23 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1847EA7B9D8; Mon, 20 Jun 2016 06:58:23 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id EEC7C22B2; Mon, 20 Jun 2016 06:58:22 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (ppp121-45-242-176.lns20.per4.internode.on.net [121.45.242.176]) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id u5K6w3mT086534 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Sun, 19 Jun 2016 23:58:06 -0700 (PDT) (envelope-from julian@freebsd.org) Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i To: Jason Zhang , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, freebsd-hardware@freebsd.org References: From: Julian Elischer Message-ID: <72fcce2c-2296-a9fe-bce2-d3f92caeb1d0@freebsd.org> Date: Mon, 20 Jun 2016 14:57:58 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.11; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jun 2016 06:58:23 -0000 On 17/06/2016 3:16 PM, Jason Zhang wrote: > Hi, > > I am working on storage service based on FreeBSD. I look forward to a good result because many professional storage company use FreeBSD as its OS. But I am disappointed with the Bad performance. I tested the the performance of LSI MegaRAID 9260-8i and had the following bad result: > > 1. Test environment: > (1) OS: FreeBSD 10.0 release > (2) Memory: 16G > (3) RAID adapter: LSI MegaRAID 9260-8i > (4) Disks: 9 SAS hard drives (10000 rpm), performance is expected for each hard drive > (5) Test tools: fio with io-depth=1, thread num is 32 and block size is 64k or 1M > (6) RAID configuration: RAID 5, stripe size is 1M > > 2. Test result: > (1) write performance too bad: 20Mbytes/s throughput and 200 random write IOPS > (2) read performance is expected: 700Mbytes/s throughput and 1500 random read IOPS > > > I tested the same hardware configuration with CentOS linux and Linux's write performance is 5 times better than FreeBSD. > > > Anyone encountered the same performance problem? Does the mfi driver have performance issue or I should give up on FreeBSD? > > Unfortunatley issues related to performance can often be very specific. We use the LSI cards with great success under FreeBSD 8 in our product at work but it is impossible to say what is specifically wrong in your setup. Some years ago I did discover that fio needed to have completely different arguments to get good performance under FreeBSD, so please check that first. What does performance look like with a single large write stream? Also look at the handling of interrupts (systat -vmstat) to ensure that interrupts are being handled correctly. that can vary greatly from motherboard to motherboard and bios to bios. (even between revisions). Sometimes Linux will cope differently with these issues as they have better support from the motherboard makers themselves. (sometimes we cope better too). One final thought.. make sure you have partitioned your drives and filesyste,s so that all the block boundaries agree and line up. At on place I worked we found we had accidentally partitioned all our drives starting 63 sectors into the drive. That did NOT work well. :-) 8k raid stripe writes were always 2 writes (and sometimes a read) > > > 张京城 Jason > > 赛凡信息科技（厦门）有限公司 > Cyphy Technology (Xiamen) Co.Ltd. > 公司总部：厦门市软件园望海路55号A座901-904单元 > 研发总部：北京市东城区美术馆后街大取灯胡同2号 > 热线：4008798066 > 总机：0592-2936100 > 邮箱：jasonzhang@cyphytech.com > 公司网址：Http://www.cyphytech.com > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > From owner-freebsd-performance@freebsd.org Tue Jun 21 16:36:43 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A1EF1AC5FE2; Tue, 21 Jun 2016 16:36:43 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7CA582AC8; Tue, 21 Jun 2016 16:36:43 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 22E4B2092B; Tue, 21 Jun 2016 12:36:42 -0400 (EDT) Received: from web4 ([10.202.2.214]) by compute1.internal (MEProxy); Tue, 21 Jun 2016 12:36:42 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=QYn+6/JOvuakMzb KpvdFmEmnZuE=; b=p50kHpsAPgbhZ615Yfu56Y9RdrQQrzPOEdD+RnkCa/Ob1eM MhnWtyYJIaie7+J/+0u43NaISSP4Os/l6cn0uHw9fuvV490A7h0pQxC3DAJfrahA 8FUrnTc3Pqz6qxtIqLEWLFo4nGizeUcpUXDZISF33mqZjjttSn4uqltb5Sp0= Received: by mailuser.nyi.internal (Postfix, from userid 99) id F1955CC672; Tue, 21 Jun 2016 12:36:41 -0400 (EDT) Message-Id: <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> X-Sasl-Enc: k4/kYWm4awCS1xnjnT4ajGNVz/VSF8t/f/bie76/NtQk 1466527001 From: Mark Felder To: Jason Zhang , freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, freebsd-hardware@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Mailer: MessagingEngine.com Webmail Interface - ajax-eb8588c4 Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i Date: Tue, 21 Jun 2016 11:36:41 -0500 In-Reply-To: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Jun 2016 16:36:43 -0000 On Fri, Jun 17, 2016, at 02:17, Jason Zhang wrote: > Hi, > > I am working on storage service based on FreeBSD. I look forward to a > good result because many professional storage company use FreeBSD as its > OS. But I am disappointed with the Bad performance. I tested the the > performance of LSI MegaRAID 9260-8i and had the following bad result: > > 1. Test environment: > (1) OS: FreeBSD 10.0 release 10.0-RELEASE is no longer supported. Can you test this on 10.3-RELEASE? Have you confirmed that both servers are using identical RAID controller settings? It's possible the CentOS install has enabled write caching but it's disabled on your FreeBSD server. Are you using UFS or ZFS on FreeBSD? Do you have atime enabled? I believe CentOS is going to have "relatime" or "nodiratime" by default to mitigate the write penalty on each read access. We need more data :-) -- Mark Felder ports-secteam member feld@FreeBSD.org From owner-freebsd-performance@freebsd.org Tue Jun 21 19:48:01 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D98DFAC532F for ; Tue, 21 Jun 2016 19:48:01 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id B45D32001 for ; Tue, 21 Jun 2016 19:48:01 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mailman.ysv.freebsd.org (Postfix) id B02CAAC532B; Tue, 21 Jun 2016 19:48:01 +0000 (UTC) Delivered-To: performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AFD58AC532A for ; Tue, 21 Jun 2016 19:48:01 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-oi0-x234.google.com (mail-oi0-x234.google.com [IPv6:2607:f8b0:4003:c06::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 617911FDC for ; Tue, 21 Jun 2016 19:48:01 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mail-oi0-x234.google.com with SMTP id r2so940742oih.2 for ; Tue, 21 Jun 2016 12:48:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=BWu424J08XNTOZZDYk8pJct/4YcWXKZL5/IlcVjBLww=; b=iJPKziUxHO0tsRC/zzIVAVu38nJ8yRoDGgld62ptXQbu6JV7h5Are7VbL5UctnPF5v oKdHL616cNCASpZU10m7vElATRjPg601xJ2qxxDuYk5aB1qvpDw/NY/Dg/J7Xxez/CO8 Vh9cO/9Q6wEJH1jCiJh3yQHVx5mlBvHCrg/R9ThvqwFVRZSdHCO0ehTAjFNeAFphPQRy FsFGQpjP8FZVU6IPT3Hdg71Kmke9nQom4A0NrovudgYvTbA0NzT3Y++/xcGvq3eMpOAU DXY1mtV+JO8PLBigpRccrC8I9Xt4s9BKN0x+/ZdBjgc1FEOaYfEBuM3DJDxvt+SAxHtm DnTg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=BWu424J08XNTOZZDYk8pJct/4YcWXKZL5/IlcVjBLww=; b=btuXcFJTVuuJjaSplhdJ59uqd895bLANJHeCbg6jsJnhOtyAUvxBngeDHbLCa2TGw1 Wx/rnLRVEJQTodPsl0FNt/VAPpSoMxw1KWkKNrRGMlQaX9rYTIh91Ak2/tvMmLkYDPZi VUqRPKEOc8+96Ys9w6Pf2fwWr2+xC0yl/ULToKP/sMQWBYfi8lqSG/zYkY4DFHuZdFjt wlk7i4K5YF56Gs2/YmKPN1LPKHZqPcDTXCz/6X1W3Oa6A31ymprcIjiiUAnAZZQDW46v sgWtUV/mKVBmAGwl+I5sc34HgbycOtGiObi/16Rp5A/zLb6bOlEPgY655wZJB82xA7++ t7dQ== X-Gm-Message-State: ALyK8tKoSPyzfHcotc4nUu2ypfe9ZpJNCzBcfMfmJxgyOtcceJUE/hzqTLyDEKcZ/fQKuQrUgNZka6D/PvmMbFFQ X-Received: by 10.202.240.11 with SMTP id o11mr67831oih.45.1466538480657; Tue, 21 Jun 2016 12:48:00 -0700 (PDT) MIME-Version: 1.0 Sender: sobomax@sippysoft.com Received: by 10.157.41.209 with HTTP; Tue, 21 Jun 2016 12:48:00 -0700 (PDT) In-Reply-To: <1603235.2ShtoCfSqO@ralph.baldwin.cx> References: <20140627125613.GT93733@kib.kiev.ua> <1603235.2ShtoCfSqO@ralph.baldwin.cx> From: Maxim Sobolev Date: Tue, 21 Jun 2016 12:48:00 -0700 X-Google-Sender-Auth: TGqbUJWsBeRDE3jS2pTvxu0J558 Message-ID: Subject: Re: PostgreSQL performance on FreeBSD To: John Baldwin Cc: Adrian Chadd , Konstantin Belousov , Alan Somers , Alan Cox , Alan Cox , freebsd-current , performance@freebsd.org, "current@freebsd.org" X-Mailman-Approved-At: Tue, 21 Jun 2016 20:20:31 +0000 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Jun 2016 19:48:02 -0000 Thanks, Konstantin for the great work, we are definitely looking forward to get all those improvements to be part of the default FreeBSD kernel/port. Would be nice if you can post an update some day later as to what's integrated and what's not. Just in case, I've opened #14206 with PG to switch us to using POSIX semaphores by default. Apart from the mentioned performance benefits, SYSV semaphores are PITA to deal with as they come in very limited quantities by default. Also they might stay around if PG dies/gets nuked and prevent it from starting again due to overflow. We've got some quite ugly code to clean up those using ipcrm(1) in our build scripts to deal with just that. I am happy that code could be retired now. -Maxim On Fri, Jun 3, 2016 at 11:53 AM, John Baldwin wrote: > On Friday, June 03, 2016 11:29:03 AM Adrian Chadd wrote: > > On 3 June 2016 at 11:27, Adrian Chadd wrote: > > > > > That and the other NUMA stuff is something to address in -12. > > > > And, I completely welcome continued development in NUMA scaling in > > combination with discussion. The iterator changes I committed are a > > more generic version of a patch people were applying on top of -10 and > > -head for at least what, three years now? Maybe more if -9 also just > > did round-robin and not first-touch? > > 8 and 9 did first-touch. Only 10 did round-robin. > > -- > John Baldwin > _______________________________________________ > freebsd-current@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe@freebsd.org" > > From owner-freebsd-performance@freebsd.org Wed Jun 22 02:14:22 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 379B3AC524F; Wed, 22 Jun 2016 02:14:22 +0000 (UTC) (envelope-from jasonzhang@cyphytech.com) Received: from m50-210.qiye.163.com (m50-210.qiye.163.com [123.125.50.210]) by mx1.freebsd.org (Postfix) with ESMTP id DED3F16A7; Wed, 22 Jun 2016 02:14:19 +0000 (UTC) (envelope-from jasonzhang@cyphytech.com) Received: from [192.168.120.81] (unknown [125.39.30.134]) by smtp6 (Coremail) with SMTP id RNOowEDp80Mt82lXOrMOBQ--.15S3; Wed, 22 Jun 2016 10:08:45 +0800 (CST) Content-Type: text/plain; charset=gb2312 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i From: Jason Zhang In-Reply-To: <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> Date: Wed, 22 Jun 2016 10:08:49 +0800 Cc: freebsd-stable@freebsd.org, freebsd-performance@freebsd.org, freebsd-hardware@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> To: Mark Felder X-Mailer: Apple Mail (2.3124) X-CM-TRANSID: RNOowEDp80Mt82lXOrMOBQ--.15S3 X-Coremail-Antispam: 1Uf129KBjDUn29KB7ZKAUJUUUUU529EdanIXcx71UUUUU7v73 VFW2AGmfu7bjvjm3AaLaJ3UbIYCTnIWIevJa73UjIFyTuYvjxUrXo7UUUUU X-Originating-IP: [125.39.30.134] X-CM-SenderInfo: pmdv00x2kd0wg6f11xp1whuxoofrz/1tbiQA2QHFbdGLa1zwAAse X-Mailman-Approved-At: Wed, 22 Jun 2016 05:19:24 +0000 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jun 2016 02:14:22 -0000 Mark, Thanks We have same RAID setting both on FreeBSD and CentOS including cache = setting. In FreeBSD, I enabled the write cache but the performance is = the same. =20 We don=A1=AFt use ZFS or UFS, and test the performance on the RAW GEOM = disk =A1=B0mfidx=A1=B1 exported by mfi driver. We observed the = =A1=B0gstat=A1=B1 result and found that the write latency is too high. When we =A1=B0dd" the disk with 8k, it is lower than 1ms, = but it is 6ms on 64kb write. It seems that each single write operation = is very slow. But I don=A1=AFt know whether it is a driver problem or not. Jason > =D4=DA 2016=C4=EA6=D4=C222=C8=D5=A3=AC=C9=CF=CE=E712:36=A3=ACMark = Felder =D0=B4=B5=C0=A3=BA >=20 >=20 >=20 > On Fri, Jun 17, 2016, at 02:17, Jason Zhang wrote: >> Hi, >>=20 >> I am working on storage service based on FreeBSD. I look forward to = a >> good result because many professional storage company use FreeBSD as = its >> OS. But I am disappointed with the Bad performance. I tested the = the >> performance of LSI MegaRAID 9260-8i and had the following bad result: >>=20 >> 1. Test environment: >> (1) OS: FreeBSD 10.0 release >=20 > 10.0-RELEASE is no longer supported. Can you test this on = 10.3-RELEASE? >=20 > Have you confirmed that both servers are using identical RAID = controller > settings? It's possible the CentOS install has enabled write caching = but > it's disabled on your FreeBSD server. Are you using UFS or ZFS on > FreeBSD? Do you have atime enabled? I believe CentOS is going to have > "relatime" or "nodiratime" by default to mitigate the write penalty on > each read access. >=20 > We need more data :-) >=20 >=20 > --=20 > Mark Felder > ports-secteam member > feld@FreeBSD.org From owner-freebsd-performance@freebsd.org Wed Jun 22 05:44:20 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C364DAC5365; Wed, 22 Jun 2016 05:44:20 +0000 (UTC) (envelope-from d.eracledes@albourne.com) Received: from despam-hpn-0.albourne.com (ns-hpn-0.albourne.com [64.47.157.36]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.albourne.com", Issuer "SwissSign Server Silver CA 2014 - G22" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 865B0221E; Wed, 22 Jun 2016 05:44:20 +0000 (UTC) (envelope-from d.eracledes@albourne.com) X-Virus-Scanned: amavisd-new at despam-hpn-0.albourne.com Received: from mail-hpn-0.intern.albourne.com (mail-hpn-0.intern.albourne.com [192.168.96.24]) by despam-hpn-0.albourne.com (8.15.2/8.15.2/Albourne/DNSBL/grey) with ESMTP id u5M5SFGY027800; Wed, 22 Jun 2016 05:28:16 GMT Received: from localhost (localhost [127.0.0.1]) by mail-hpn-0.intern.albourne.com (Postfix) with ESMTP id 63916357DD5A5; Wed, 22 Jun 2016 05:28:15 +0000 (UTC) Received: from mail-hpn-0.intern.albourne.com ([127.0.0.1]) by localhost (mail-hpn-0.intern.albourne.com [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id BI1zOc298Tgf; Wed, 22 Jun 2016 05:28:15 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mail-hpn-0.intern.albourne.com (Postfix) with ESMTP id 27A6F357DD5A3; Wed, 22 Jun 2016 05:28:15 +0000 (UTC) X-Quarantine-ID: Received: from mail-hpn-0.intern.albourne.com ([127.0.0.1]) by localhost (mail-hpn-0.intern.albourne.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id AlXNKQbooAs8; Wed, 22 Jun 2016 05:28:15 +0000 (UTC) Received: from mail-hpn-0.intern.albourne.com (mail-hpn-0.intern.albourne.com [192.168.96.24]) by mail-hpn-0.intern.albourne.com (Postfix) with ESMTP id 0B3D5357DD3F6; Wed, 22 Jun 2016 05:28:15 +0000 (UTC) Content-Type: text/plain; charset="iso-8859-1" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable X-Client-ID: 172709 X-Mailer: BlackBerry Email (10.3.2.2876) X-Mailer: Zimbra 8.0.5_GA_5839 (MobileSync - RIM-Passport-SQW100-1/10.3.2.2876) Message-ID: <20160622052814.5779540.99036.172709@albourne.com> Date: Wed, 22 Jun 2016 05:28:15 +0000 (UTC) Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i From: Doros Eracledes In-Reply-To: <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> Cc: freebsd-performance@freebsd.org, freebsd-stable@freebsd.org, freebsd-hardware@freebsd.org Thread-Topic: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i Thread-Index: gZYY2ta0yZK6SLr/gqj9E82KQYMYEA== X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jun 2016 05:44:20 -0000 As a side note, we also use this controller with FreeBSD 10.1 but configure= d each drive as a JBOD and then created raidz zfs pools and that was much f= aster than to let the LSI do raid5.=A0 Best Doros From owner-freebsd-performance@freebsd.org Wed Jun 22 07:05:03 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DF723AC678A; Wed, 22 Jun 2016 07:05:03 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from cu1176c.smtpx.saremail.com (cu1176c.smtpx.saremail.com [195.16.148.151]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id A1F722730; Wed, 22 Jun 2016 07:05:03 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from [172.16.8.36] (izaro.sarenet.es [192.148.167.11]) by proxypop02.sare.net (Postfix) with ESMTPSA id 48E909DC9D4; Wed, 22 Jun 2016 08:58:08 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: mfi driver performance too bad on LSI MegaRAID SAS 9260-8i From: Borja Marcos In-Reply-To: <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> Date: Wed, 22 Jun 2016 08:58:08 +0200 Cc: Mark Felder , freebsd-performance@freebsd.org, freebsd-stable@freebsd.org, freebsd-hardware@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <16CD100A-3BD0-47BA-A91E-F445E5DF6DBC@cyphytech.com> <1466527001.2694442.644278905.18E236CD@webmail.messagingengine.com> <1790833A-9292-4A46-B43C-BF41C7C801BE@cyphytech.com> To: Jason Zhang X-Mailer: Apple Mail (2.3124) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jun 2016 07:05:04 -0000 > On 22 Jun 2016, at 04:08, Jason Zhang = wrote: >=20 > Mark, >=20 > Thanks >=20 > We have same RAID setting both on FreeBSD and CentOS including cache = setting. In FreeBSD, I enabled the write cache but the performance is = the same. =20 >=20 > We don=E2=80=99t use ZFS or UFS, and test the performance on the RAW = GEOM disk =E2=80=9Cmfidx=E2=80=9D exported by mfi driver. We observed = the =E2=80=9Cgstat=E2=80=9D result and found that the write latency > is too high. When we =E2=80=9Cdd" the disk with 8k, it is lower than = 1ms, but it is 6ms on 64kb write. It seems that each single write = operation is very slow. But I don=E2=80=99t know > whether it is a driver problem or not. There is an option you can use (I do it all the time!) to make the card = behave as a plain HBA so that the disks are handled by the =E2=80=9Cda=E2=80= =9D driver.=20 Add this to /boot/loader.conf hw.mfi.allow_cam_disk_passthrough=3D1 mfip_load=3D=E2=80=9CYES" And do the tests accessing the disks as =E2=80=9Cda=E2=80=9D. To avoid = confusions, it=E2=80=99s better to make sure the disks are not part of a = =E2=80=9Cjbod=E2=80=9D or logical volume configuration. Borja. From owner-freebsd-performance@freebsd.org Wed Jun 22 10:02:54 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E24AB3B0F1 for ; Wed, 22 Jun 2016 10:02:54 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 4E7C324F4 for ; Wed, 22 Jun 2016 10:02:54 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 48B9BB3B0EE; Wed, 22 Jun 2016 10:02:54 +0000 (UTC) Delivered-To: performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 435E9B3B0EB; Wed, 22 Jun 2016 10:02:54 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AA01324EE; Wed, 22 Jun 2016 10:02:53 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id u5MA2ggC018605 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Wed, 22 Jun 2016 13:02:42 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua u5MA2ggC018605 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id u5MA2fPD018604; Wed, 22 Jun 2016 13:02:41 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 22 Jun 2016 13:02:41 +0300 From: Konstantin Belousov To: Maxim Sobolev Cc: John Baldwin , Adrian Chadd , Alan Somers , Alan Cox , Alan Cox , freebsd-current , performance@freebsd.org, "current@freebsd.org" Subject: Re: PostgreSQL performance on FreeBSD Message-ID: <20160622100241.GM38613@kib.kiev.ua> References: <20140627125613.GT93733@kib.kiev.ua> <1603235.2ShtoCfSqO@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.6.1 (2016-04-27) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jun 2016 10:02:54 -0000 On Tue, Jun 21, 2016 at 12:48:00PM -0700, Maxim Sobolev wrote: > Thanks, Konstantin for the great work, we are definitely looking forward to > get all those improvements to be part of the default FreeBSD kernel/port. > Would be nice if you can post an update some day later as to what's > integrated and what's not. I did posted the update several days earlier. Since you replying to this thread, it would be not unreasonable to read recent messages that were sent. > > Just in case, I've opened #14206 with PG to switch us to using POSIX > semaphores by default. Apart from the mentioned performance benefits, SYSV > semaphores are PITA to deal with as they come in very limited quantities by > default. Also they might stay around if PG dies/gets nuked and prevent it > from starting again due to overflow. We've got some quite ugly code to > clean up those using ipcrm(1) in our build scripts to deal with just that. > I am happy that code could be retired now. Named semaphores also stuck around if processes are killed without cleanup. From owner-freebsd-performance@freebsd.org Wed Jun 22 14:26:54 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 99D15B69642 for ; Wed, 22 Jun 2016 14:26:54 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 73DE519DC for ; Wed, 22 Jun 2016 14:26:54 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mailman.ysv.freebsd.org (Postfix) id 6FD69B6963D; Wed, 22 Jun 2016 14:26:54 +0000 (UTC) Delivered-To: performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6D2B8B6963C for ; Wed, 22 Jun 2016 14:26:54 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-oi0-x22e.google.com (mail-oi0-x22e.google.com [IPv6:2607:f8b0:4003:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2D0BF19D6 for ; Wed, 22 Jun 2016 14:26:54 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mail-oi0-x22e.google.com with SMTP id f189so24334284oig.3 for ; Wed, 22 Jun 2016 07:26:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=r/NmRzpGThk1aVRAcTUeMQLapWNn/JvOvb9Ir6Ec4Lg=; b=uCdt91Dec+M7FIguu2hYXxZJ9WFe11ak9weMMYHgr09dYXXTHLRjgBW6VAMpDk6d93 YixkRSoTpNoQuc9TR/kMS/eZlWrmdKhv3XHkUT3315CN730RuILXkuV7x8UHyrRLrS+9 Xbsa5rXmqfXI8DCJhfLaDARPnFUFeMHEhb7jp0sCusgPRxyviGpHD1Y0cZ9DSs7zz2kL uJkTc8GzXT49UHJARxPE9Gi6lSjVtIfKQJ8QvwxKoDQAaNIzbmQ33J2/NJZjdhTLnwIS LTGV3uqngQl7MJPaMlF5HFHewitJAlkyVUOHHJ8Uosy7zMgF6LUvZA3aZ6qzh651U/e5 gHtg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=r/NmRzpGThk1aVRAcTUeMQLapWNn/JvOvb9Ir6Ec4Lg=; b=mWjAgha9xuoMhoYs2v0mxsMAKIWSxrBZa5ZlVHk/vM5zbIDjjHoT+RR3v1ZxBIYAPg 5H8PUFFQEgPZYaXed2WkSjj/JUwJ1SPMSEpZp0XxW6T9NZ0nczv7T201kxCFITUeZI0S fi0oRokfYq305lWA+Kfaqm+bImhOK1mxoVkwF97fbhex/1lHt4K2bHQKQbLdZ5Ywa3jL M0QUc8n5Q2xug8mnjDm8Ib4tRiuWOeAmXf0EvV5iGp9s+DgrdvIkVrdf1jUnQ9ftyxNF PhNe6OKiAJlcmbZjt4P8zXwawB08QPgHIknCzFvQfSikqHkQUEVBnGTAI8MvIF1175Nb LwwQ== X-Gm-Message-State: ALyK8tLMfS5qFvvdU3MLuyV18UmdLlXKuO+dsfGjZ+7C3IFs+eHmQF3VIz2pzh15QOHdu1FBAeo6/tlhWwTLXHom X-Received: by 10.202.231.198 with SMTP id e189mr1715841oih.3.1466605613413; Wed, 22 Jun 2016 07:26:53 -0700 (PDT) MIME-Version: 1.0 Sender: sobomax@sippysoft.com Received: by 10.157.41.209 with HTTP; Wed, 22 Jun 2016 07:26:52 -0700 (PDT) In-Reply-To: <20160622100241.GM38613@kib.kiev.ua> References: <20140627125613.GT93733@kib.kiev.ua> <1603235.2ShtoCfSqO@ralph.baldwin.cx> <20160622100241.GM38613@kib.kiev.ua> From: Maxim Sobolev Date: Wed, 22 Jun 2016 07:26:52 -0700 X-Google-Sender-Auth: GEhz5HWzcNo5tUqTMHS36sfjGuc Message-ID: Subject: Re: PostgreSQL performance on FreeBSD To: Konstantin Belousov Cc: John Baldwin , Adrian Chadd , Alan Somers , Alan Cox , Alan Cox , freebsd-current , performance@freebsd.org, "current@freebsd.org" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Jun 2016 14:26:54 -0000 Konstantin, Not if you do sem_unlink() immediately, AFAIK. And that's what PG does. So the window of opportunity for the leakage is quite small, much smaller than for SYSV primitives. Sorry for missing your status update message, I've missed it somehow. ---- mySem = sem_open(semname, O_CREAT | O_EXCL, (mode_t) IPCProtection, (unsigned) 1); #ifdef SEM_FAILED if (mySem != (sem_t *) SEM_FAILED) break; #else if (mySem != (sem_t *) (-1)) break; #endif /* Loop if error indicates a collision */ if (errno == EEXIST || errno == EACCES || errno == EINTR) continue; /* * Else complain and abort */ elog(FATAL, "sem_open(\"%s\") failed: %m", semname); } /* * Unlink the semaphore immediately, so it can't be accessed externally. * This also ensures that it will go away if we crash. */ sem_unlink(semname); return mySem; ---- -Max On Wed, Jun 22, 2016 at 3:02 AM, Konstantin Belousov wrote: > On Tue, Jun 21, 2016 at 12:48:00PM -0700, Maxim Sobolev wrote: > > Thanks, Konstantin for the great work, we are definitely looking forward > to > > get all those improvements to be part of the default FreeBSD kernel/port. > > Would be nice if you can post an update some day later as to what's > > integrated and what's not. > I did posted the update several days earlier. Since you replying to this > thread, it would be not unreasonable to read recent messages that were > sent. > > > > > Just in case, I've opened #14206 with PG to switch us to using POSIX > > semaphores by default. Apart from the mentioned performance benefits, > SYSV > > semaphores are PITA to deal with as they come in very limited quantities > by > > default. Also they might stay around if PG dies/gets nuked and prevent it > > from starting again due to overflow. We've got some quite ugly code to > > clean up those using ipcrm(1) in our build scripts to deal with just > that. > > I am happy that code could be retired now. > > Named semaphores also stuck around if processes are killed without cleanup. > > From owner-freebsd-performance@freebsd.org Thu Jun 23 22:31:47 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1F9E0B74D45 for ; Thu, 23 Jun 2016 22:31:47 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 071252BB6 for ; Thu, 23 Jun 2016 22:31:47 +0000 (UTC) (envelope-from jilles@stack.nl) Received: by mailman.ysv.freebsd.org (Postfix) id 023E0B74D42; Thu, 23 Jun 2016 22:31:47 +0000 (UTC) Delivered-To: performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0193BB74D40; Thu, 23 Jun 2016 22:31:47 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client CN "mailhost.stack.nl", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id BC4CE2BB3; Thu, 23 Jun 2016 22:31:46 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id DE525358C5A; Fri, 24 Jun 2016 00:31:43 +0200 (CEST) Received: by snail.stack.nl (Postfix, from userid 1677) id CC6BE28494; Fri, 24 Jun 2016 00:31:43 +0200 (CEST) Date: Fri, 24 Jun 2016 00:31:43 +0200 From: Jilles Tjoelker To: Maxim Sobolev Cc: Konstantin Belousov , John Baldwin , Adrian Chadd , Alan Somers , Alan Cox , Alan Cox , freebsd-current , performance@freebsd.org, "current@freebsd.org" Subject: Re: PostgreSQL performance on FreeBSD Message-ID: <20160623223143.GC5381@stack.nl> References: <20140627125613.GT93733@kib.kiev.ua> <1603235.2ShtoCfSqO@ralph.baldwin.cx> <20160622100241.GM38613@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) X-Mailman-Approved-At: Thu, 23 Jun 2016 22:34:42 +0000 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Jun 2016 22:31:47 -0000 On Wed, Jun 22, 2016 at 07:26:52AM -0700, Maxim Sobolev wrote: > Konstantin, > Not if you do sem_unlink() immediately, AFAIK. And that's what PG does. So > the window of opportunity for the leakage is quite small, much smaller than > for SYSV primitives. Sorry for missing your status update message, I've > missed it somehow. True, but if your process architecture supports that, you can also put unnamed process-shared semaphores into a piece of shared memory (pad sem_t to a cache line if contention is expected). This is more natural API use (fully avoiding the leak) and uses less memory. It has been supported for a long time, at least since FreeBSD 9.0. Process-shared mutexes, condition variables, reader/writer locks, etc. are available in FreeBSD 11 but use more memory (a 1-page object per synchronization object), somewhat like named semaphores. -- Jilles Tjoelker From owner-freebsd-performance@freebsd.org Fri Jun 24 00:27:29 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4292DAC6E7C for ; Fri, 24 Jun 2016 00:27:29 +0000 (UTC) (envelope-from sean@chittenden.org) Received: from mailman.ysv.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 252562F33 for ; Fri, 24 Jun 2016 00:27:29 +0000 (UTC) (envelope-from sean@chittenden.org) Received: by mailman.ysv.freebsd.org (Postfix) id 1E449AC6E79; Fri, 24 Jun 2016 00:27:29 +0000 (UTC) Delivered-To: performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 18C63AC6E77; Fri, 24 Jun 2016 00:27:29 +0000 (UTC) (envelope-from sean@chittenden.org) Received: from mail01.lax1.stackjet.com (mon01.lax1.stackjet.com [174.136.104.178]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DE5432F2C; Fri, 24 Jun 2016 00:27:28 +0000 (UTC) (envelope-from sean@chittenden.org) Received: from [172.20.10.9] (unknown [172.56.39.215]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: sean@chittenden.org) by mail01.lax1.stackjet.com (Postfix) with ESMTPSA id 1E1BD2F88F5; Thu, 23 Jun 2016 17:19:15 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: PostgreSQL performance on FreeBSD From: Sean Chittenden In-Reply-To: Date: Thu, 23 Jun 2016 15:42:32 -0700 Cc: Konstantin Belousov , Adrian Chadd , performance@freebsd.org, John Baldwin , Alan Somers , Alan Cox , Alan Cox , freebsd-current , "current@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: References: <20140627125613.GT93733@kib.kiev.ua> <1603235.2ShtoCfSqO@ralph.baldwin.cx> <20160622100241.GM38613@kib.kiev.ua> To: Maxim Sobolev X-Mailer: Apple Mail (2.3124) X-Mailman-Approved-At: Fri, 24 Jun 2016 01:26:02 +0000 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Jun 2016 00:27:29 -0000 Small nit: PostgreSQL used SYSV because it allowed for the detection of dead = processes. If you `kill -9`=E2=80=99ed a process, PostgreSQL can detect = that and then shut down and perform an automatic recovery. In this = regard, sysv is pretty clever. The move to POSIX shared mem was done = for a host of reasons, but it means that you don=E2=80=99t have to = adjust your SYSV limits. My understanding from a few years ago is that = there is still a ~64KB SYSV memory segment that is still used to act as = the latch to signal if a process was killed, but all of the shared = buffers are stored in posix mmap=E2=80=99ed regions. At this point in time this could be replaced with kqueue(2) EVFILT_PROC, = but no one has done that yet. -sc -- Sean Chittenden sean@chittenden.org > On Jun 22, 2016, at 07:26 , Maxim Sobolev wrote: >=20 > Konstantin, >=20 > Not if you do sem_unlink() immediately, AFAIK. And that's what PG = does. So > the window of opportunity for the leakage is quite small, much smaller = than > for SYSV primitives. Sorry for missing your status update message, = I've > missed it somehow. >=20 > ---- > mySem =3D sem_open(semname, O_CREAT | O_EXCL, > (mode_t) = IPCProtection, > (unsigned) 1); >=20 > #ifdef SEM_FAILED > if (mySem !=3D (sem_t *) SEM_FAILED) > break; > #else > if (mySem !=3D (sem_t *) (-1)) > break; > #endif >=20 > /* Loop if error indicates a collision */ > if (errno =3D=3D EEXIST || errno =3D=3D EACCES || errno = =3D=3D EINTR) > continue; >=20 > /* > * Else complain and abort > */ > elog(FATAL, "sem_open(\"%s\") failed: %m", semname); > } >=20 > /* > * Unlink the semaphore immediately, so it can't be accessed > externally. > * This also ensures that it will go away if we crash. > */ > sem_unlink(semname); >=20 > return mySem; > ---- >=20 > -Max >=20 > On Wed, Jun 22, 2016 at 3:02 AM, Konstantin Belousov = > wrote: >=20 >> On Tue, Jun 21, 2016 at 12:48:00PM -0700, Maxim Sobolev wrote: >>> Thanks, Konstantin for the great work, we are definitely looking = forward >> to >>> get all those improvements to be part of the default FreeBSD = kernel/port. >>> Would be nice if you can post an update some day later as to what's >>> integrated and what's not. >> I did posted the update several days earlier. Since you replying to = this >> thread, it would be not unreasonable to read recent messages that = were >> sent. >>=20 >>>=20 >>> Just in case, I've opened #14206 with PG to switch us to using POSIX >>> semaphores by default. Apart from the mentioned performance = benefits, >> SYSV >>> semaphores are PITA to deal with as they come in very limited = quantities >> by >>> default. Also they might stay around if PG dies/gets nuked and = prevent it >>> from starting again due to overflow. We've got some quite ugly code = to >>> clean up those using ipcrm(1) in our build scripts to deal with just >> that. >>> I am happy that code could be retired now. >>=20 >> Named semaphores also stuck around if processes are killed without = cleanup. >>=20 >>=20 > _______________________________________________ > freebsd-performance@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to = "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@freebsd.org Sat Jun 25 16:16:03 2016 Return-Path: Delivered-To: freebsd-performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EB0E1B82F4D for ; Sat, 25 Jun 2016 16:16:03 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id C2A6D1E9C for ; Sat, 25 Jun 2016 16:16:03 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mailman.ysv.freebsd.org (Postfix) id BC526B82F47; Sat, 25 Jun 2016 16:16:03 +0000 (UTC) Delivered-To: performance@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B9A03B82F44 for ; Sat, 25 Jun 2016 16:16:03 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: from mail-it0-x22f.google.com (mail-it0-x22f.google.com [IPv6:2607:f8b0:4001:c0b::22f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 77B9C1E90 for ; Sat, 25 Jun 2016 16:16:03 +0000 (UTC) (envelope-from sobomax@sippysoft.com) Received: by mail-it0-x22f.google.com with SMTP id g127so36674747ith.0 for ; Sat, 25 Jun 2016 09:16:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=sippysoft-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=KPGvviuXJCPPNzg8N0itYWnVzqmvsMZ6wlYso68tcds=; b=Xv7KMMSGluuD2Bkm3wYQR4t04pJCNJ7hQKuQwjd+ONTKflWo3qE5vpWORMqSE9sKYg CuBbXmlNtJYW4C4k1gaIWAd9VxPsu3j+PsxSGBsYPYOlmwFK8dcKLHmWYPZc87Zpp8YZ lEOSirf3Y2sgmfZmOSHqLANHM7A4xtmQiEoMuCcbVr84WGV7vr90CoCaMNI+SCTE0kC3 LKHJL119d6IfVZ11YwW6d6834G2gaLA7o8fristayQP4ybNmpzzMKmU8acYT5tki7pRm LR/CzDb62QNnSREwNmL6d4/dz75gzEAmvOB/N69RBhojxjPixz9K7fDOvAOovtMXTbdA fSZw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=KPGvviuXJCPPNzg8N0itYWnVzqmvsMZ6wlYso68tcds=; b=g6bRu3SsIwDhZX/cVfgJhEhhGiDFseuCCp7tIlnzpiur36Qbxvc0vhqmL+xNy/D01r EXiT5N0T4A3aE1SQ8Z/uNVCmnDC2Nk8nm9bl3jcTauTEpU2AKWtIJ1KHkamLIiS5Ft7X 0wJckYWuByy5NZMf++GBqk6zHkHBi/813vpx/ALc97QYAB/yhuTs4X1MrxT7+vKewYzI +hRGtBYLY8OF396KVE1SQ1Cfe7oBC7bEu/E+2zDMKkq93MGLKEcR6qow4oH5gE9rdCv8 8HOfW+M5CVEiNA77TrUQvq6olmpf1fM1gm7VXXW8mEXHefmvfLT318/GkVodvy6n6VBD Bf2g== X-Gm-Message-State: ALyK8tKDmwNQEVaKtWVj6znTS6BFGBVqxA6WhYyTaS3plHaqOtJRNw2RLYjyLuZmEztPsgcSzjQHZotc53WIB2Xi X-Received: by 10.36.123.75 with SMTP id q72mr2486183itc.44.1466871362552; Sat, 25 Jun 2016 09:16:02 -0700 (PDT) MIME-Version: 1.0 Sender: sobomax@sippysoft.com Received: by 10.36.125.197 with HTTP; Sat, 25 Jun 2016 09:16:01 -0700 (PDT) In-Reply-To: References: <20140627125613.GT93733@kib.kiev.ua> <1603235.2ShtoCfSqO@ralph.baldwin.cx> <20160622100241.GM38613@kib.kiev.ua> From: Maxim Sobolev Date: Sat, 25 Jun 2016 09:16:01 -0700 X-Google-Sender-Auth: OyT3s5nJ4OaYhoaFUlp5IrO8VdU Message-ID: Subject: Re: PostgreSQL performance on FreeBSD To: Sean Chittenden Cc: Konstantin Belousov , Adrian Chadd , performance@freebsd.org, John Baldwin , Alan Somers , Alan Cox , Alan Cox , freebsd-current , "current@freebsd.org" X-Mailman-Approved-At: Sat, 25 Jun 2016 17:38:47 +0000 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Jun 2016 16:16:04 -0000 Sean, to the issue that you are describing it is also might be possible to do it some other way around. One, perhaps more portable, is to share a connected socketpair between two communicating processes, so that you can do non-blocking read on one of its ends from time to time and check if it returns EOF. Which would be the case if whatever process holds the other end of it is no longer there. So instead of shared memory segment, you can have pool of descriptors, one for each worker that you care about. Polling on those would be trivial with just regular poll(2). The only issue might be that postgres forks a lot, so we would probably need to implement FD_CLOFORK to avoid copying those extra fds into every child. Something akin to a solution that I recently posted to work around problem that you cannot really waitpid() on a grand-child see PG BUG #14199 for details & patch. But yes, it would be really nice to get rid of SYSV shared memory use in PG completely as some point one way or another. -Max On Thu, Jun 23, 2016 at 3:42 PM, Sean Chittenden wrote: > Small nit: > > PostgreSQL used SYSV because it allowed for the detection of dead > processes. If you `kill -9`=E2=80=99ed a process, PostgreSQL can detect = that and > then shut down and perform an automatic recovery. In this regard, sysv i= s > pretty clever. The move to POSIX shared mem was done for a host of > reasons, but it means that you don=E2=80=99t have to adjust your SYSV lim= its. My > understanding from a few years ago is that there is still a ~64KB SYSV > memory segment that is still used to act as the latch to signal if a > process was killed, but all of the shared buffers are stored in posix > mmap=E2=80=99ed regions. > > At this point in time this could be replaced with kqueue(2) EVFILT_PROC, > but no one has done that yet. > > -sc > > > > -- > Sean Chittenden > sean@chittenden.org > > > On Jun 22, 2016, at 07:26 , Maxim Sobolev wrote: > > > > Konstantin, > > > > Not if you do sem_unlink() immediately, AFAIK. And that's what PG does. > So > > the window of opportunity for the leakage is quite small, much smaller > than > > for SYSV primitives. Sorry for missing your status update message, I've > > missed it somehow. > > > > ---- > > mySem =3D sem_open(semname, O_CREAT | O_EXCL, > > (mode_t) IPCProtection, > > (unsigned) 1); > > > > #ifdef SEM_FAILED > > if (mySem !=3D (sem_t *) SEM_FAILED) > > break; > > #else > > if (mySem !=3D (sem_t *) (-1)) > > break; > > #endif > > > > /* Loop if error indicates a collision */ > > if (errno =3D=3D EEXIST || errno =3D=3D EACCES || errno = =3D=3D EINTR) > > continue; > > > > /* > > * Else complain and abort > > */ > > elog(FATAL, "sem_open(\"%s\") failed: %m", semname); > > } > > > > /* > > * Unlink the semaphore immediately, so it can't be accessed > > externally. > > * This also ensures that it will go away if we crash. > > */ > > sem_unlink(semname); > > > > return mySem; > > ---- > > > > -Max > > > > On Wed, Jun 22, 2016 at 3:02 AM, Konstantin Belousov < > kostikbel@gmail.com> > > wrote: > > > >> On Tue, Jun 21, 2016 at 12:48:00PM -0700, Maxim Sobolev wrote: > >>> Thanks, Konstantin for the great work, we are definitely looking > forward > >> to > >>> get all those improvements to be part of the default FreeBSD > kernel/port. > >>> Would be nice if you can post an update some day later as to what's > >>> integrated and what's not. > >> I did posted the update several days earlier. Since you replying to > this > >> thread, it would be not unreasonable to read recent messages that were > >> sent. > >> > >>> > >>> Just in case, I've opened #14206 with PG to switch us to using POSIX > >>> semaphores by default. Apart from the mentioned performance benefits, > >> SYSV > >>> semaphores are PITA to deal with as they come in very limited > quantities > >> by > >>> default. Also they might stay around if PG dies/gets nuked and preven= t > it > >>> from starting again due to overflow. We've got some quite ugly code t= o > >>> clean up those using ipcrm(1) in our build scripts to deal with just > >> that. > >>> I am happy that code could be retired now. > >> > >> Named semaphores also stuck around if processes are killed without > cleanup. > >> > >> > > _______________________________________________ > > freebsd-performance@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-performance > > To unsubscribe, send any mail to " > freebsd-performance-unsubscribe@freebsd.org" > >