From owner-freebsd-performance@FreeBSD.ORG Mon Mar 18 18:32:09 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 88B757B3 for ; Mon, 18 Mar 2013 18:32:09 +0000 (UTC) (envelope-from krjeschke@omniti.com) Received: from mail-qa0-f48.google.com (mail-qa0-f48.google.com [209.85.216.48]) by mx1.freebsd.org (Postfix) with ESMTP id 51F5AFAD for ; Mon, 18 Mar 2013 18:32:09 +0000 (UTC) Received: by mail-qa0-f48.google.com with SMTP id j8so1860437qah.0 for ; Mon, 18 Mar 2013 11:32:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-received:date:message-id:subject:from:to :content-type:x-gm-message-state; bh=CnY3w0YgW4j53xuJU+rscsIV6OIciRDNtooLa35/f84=; b=GM/5nlgkDmS5CieFHVGXvZS8hpg2Nysbb1SWBgsoK16mRxVuU0f2LUcbOrgBKaIJkd lxnj923w0HzW+Kc+iHpBLOSXUcFoPVcAEzDFLBiH9nveRouU07BfizLBGCiH60oCUR5h 1HryvzrVz3A5G4P4/ISQGhnmWOsV6lghNeCWweRiVMqT18BpUUrd90d2lqYyQ58vZQz1 pcxjBeSC6rZlLjVAezX9tHOtuuHgBrX6iFJuHDbFm+JJMRxAMGuHkK+w6rYbc5nrvaSP EOM5YRGOLV/eOsEdDNGXWhinUXE1pgY66RudKPb7+tc7Gu58Ftrr0H7ek+gjCVBiAkUd MfOg== MIME-Version: 1.0 X-Received: by 10.49.94.238 with SMTP id df14mr24202434qeb.44.1363631523368; Mon, 18 Mar 2013 11:32:03 -0700 (PDT) Received: by 10.229.17.134 with HTTP; Mon, 18 Mar 2013 11:32:03 -0700 (PDT) Date: Mon, 18 Mar 2013 14:32:03 -0400 Message-ID: Subject: Surge 2013 CFP Open From: Katherine Jeschke To: freebsd-performance@freebsd.org X-Gm-Message-State: ALoCoQl2uZFKwxJIjgHpeojKk1R6dEfwXHV7fTuhOgw4bpYiew4vro2IUyqo+e5FVv0QeliLN8SX X-Mailman-Approved-At: Mon, 18 Mar 2013 19:55:55 +0000 Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Mar 2013 18:32:09 -0000 The Surge 2013 CFP is open. For details or to submit a paper, please visit http://surge.omniti.com/2013 -- Katherine Jeschke Director of Marketing and Creative Services OmniTI Computer Consulting, Inc. 11830 West Market Place, Suite F Fulton, MD 20759 O: 240-646-0770, 222 F: 301-497-2001 C: 443/643-6140 omniti.com Surge 2013 The information contained in this electronic message and any attached documents is privileged, confidential, and protected from disclosure. If you are not the intended recipient, note that any review, disclosure, copying, distribution, or use of the contents of this electronic message or any attached documents is prohibited. If you have received this communication in error, please destroy it and notify us immediately by telephone (1-443-325-1360) or by electronic mail (info@omniti.com). Thank you. From owner-freebsd-performance@FreeBSD.ORG Tue Mar 19 11:00:08 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 44A047B3 for ; Tue, 19 Mar 2013 11:00:08 +0000 (UTC) (envelope-from rkearsley@blueyonder.co.uk) Received: from mtaout02-winn.ispmail.ntl.com (mtaout02-winn.ispmail.ntl.com [81.103.221.48]) by mx1.freebsd.org (Postfix) with ESMTP id 983E5997 for ; Tue, 19 Mar 2013 11:00:07 +0000 (UTC) Received: from know-smtpout-1.server.virginmedia.net ([62.254.123.3]) by mtaout02-winn.ispmail.ntl.com (InterMail vM.7.08.04.00 201-2186-134-20080326) with ESMTP id <20130319110000.HBXI23282.mtaout02-winn.ispmail.ntl.com@know-smtpout-1.server.virginmedia.net>; Tue, 19 Mar 2013 11:00:00 +0000 Received: from [87.194.108.92] (helo=[192.168.1.168]) by know-smtpout-1.server.virginmedia.net with esmtpa (Exim 4.63) (envelope-from ) id 1UHuEN-0000tl-Er; Tue, 19 Mar 2013 10:57:23 +0000 Message-ID: <51484494.5090604@blueyonder.co.uk> Date: Tue, 19 Mar 2013 10:57:24 +0000 From: Richard Kearsley User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130106 Thunderbird/17.0.2 MIME-Version: 1.0 To: freebsd-performance@freebsd.org Subject: buffer cache usage Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Cloudmark-Analysis: v=1.1 cv=GaEGOwq9FwezmTggA+b6yC6zDZF2HYaK6RN/tSqdnVA= c=1 sm=0 a=D6wcrpBcUpcA:10 a=3NElcqgl2aoA:10 a=8nJEP1OIZ-IA:10 a=9rJ3xXJf5Y35wX6ri6YA:9 a=wPNLvfGTeEIA:10 a=HpAAvcLHHh0Zw7uRqdWCyQ==:117 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Mar 2013 11:00:08 -0000 Hi I'm using FreeBSD and nginx as a web server serving static files. I always like the "buffer cache" which lets frequently used files stay in memory rather than read from disk again. I have a new server with 32GB of ram and I notice a lot of memory is listed as "Inact": Mem: 688M Active, 26G Inact, 3758M Wired, 823M Cache, 3281M Buf, 173M Free What's preventing this ram from being used as buffer cache for files on the hard disks? Is there a limit or a tunable stopping it? I never had a server with this amount of ram before. It's a very busy server, I'm sure a lot of files are accessed regularly as the disks are always busy. Using 9.1 amd64: FreeBSD richard 9.1-RELEASE FreeBSD 9.1-RELEASE #0 r243825: Tue Dec 4 09:23:10 UTC 2012 root@farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 Many thanks From owner-freebsd-performance@FreeBSD.ORG Tue Mar 19 12:11:29 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8B7DABB2 for ; Tue, 19 Mar 2013 12:11:29 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail06.syd.optusnet.com.au (mail06.syd.optusnet.com.au [211.29.132.187]) by mx1.freebsd.org (Postfix) with ESMTP id 28AB9E3D for ; Tue, 19 Mar 2013 12:11:28 +0000 (UTC) Received: from c211-30-173-106.carlnfd1.nsw.optusnet.com.au (c211-30-173-106.carlnfd1.nsw.optusnet.com.au [211.30.173.106]) by mail06.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id r2JCBEMA026540 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 19 Mar 2013 23:11:18 +1100 Date: Tue, 19 Mar 2013 23:11:14 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Richard Kearsley Subject: Re: buffer cache usage In-Reply-To: <51484494.5090604@blueyonder.co.uk> Message-ID: <20130319225406.S1332@besplex.bde.org> References: <51484494.5090604@blueyonder.co.uk> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Optus-CM-Score: 0 X-Optus-CM-Analysis: v=2.0 cv=UMKFh1jy c=1 sm=1 a=kj9zAlcOel0A:10 a=PO7r1zJSAAAA:8 a=JzwRw_2MAAAA:8 a=aNkou1BObYEA:10 a=b-5j8YVxNRDxf2V7eiYA:9 a=CjuIK1q_8ugA:10 a=TEtd8y5WR3g2ypngnwZWYw==:117 Cc: freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 19 Mar 2013 12:11:29 -0000 On Tue, 19 Mar 2013, Richard Kearsley wrote: > I'm using FreeBSD and nginx as a web server serving static files. I always > like the "buffer cache" which lets frequently used files stay in memory > rather than read from disk again. > I have a new server with 32GB of ram and I notice a lot of memory is listed > as "Inact": > > Mem: 688M Active, 26G Inact, 3758M Wired, 823M Cache, 3281M Buf, 173M Free Inact more or less means "buffer cache" (actually VMIO memory). Buf more or less means nothing (it is the amount of virtual memory reserved but not necessarily used for mapping physical VMIO memory into virtual memory (the classical buffer cache) so that it can be accessed easily). If your "buffer cache" is working well, then it will have almost 26GB of disks mapped into VMIO and almost 3281MB of this at a time mapped into virtual memory. The vm part of the mapping is a bit slow but not nearly as slow as disks. Except maybe with SSDs and/or huge arrays. > What's preventing this ram from being used as buffer cache for files on the > hard disks? Is there a limit or a tunable stopping it? I never had a server > with this amount of ram before. With less memory, the ratio of Inact to Buf would be smaller. On i386, Buf is excessively restricted even with physical memory limited to 4G, since virtual address space is also limited to 4G and it is hard to fit enough Buf in it. Inact and VMIO are only limited by physical memory, so the too-small Buf is not a large pessimization. Bruce From owner-freebsd-performance@FreeBSD.ORG Fri Mar 22 09:03:37 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 78BDE8DD for ; Fri, 22 Mar 2013 09:03:37 +0000 (UTC) (envelope-from davide.damico@contactlab.com) Received: from mail2.shared.smtp.contactlab.it (mail2.shared.smtp.contactlab.it [93.94.37.7]) by mx1.freebsd.org (Postfix) with ESMTP id E92C9F95 for ; Fri, 22 Mar 2013 09:03:36 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; d=contactlab.it; s=clab1; c=relaxed/relaxed; q=dns/txt; i=@contactlab.it; t=1363943008; h=From:Subject:Date:To:MIME-Version:Content-Type; bh=33Ec4PVtCfSjIWVe2ofzodNqu7ovrjNMp/qt0dsf1qU=; b=pWaZHDRriYgDrEaQhhdUJlgz2o77KIpeWbBBl5779e04bBH/VT2Nuxhzy2GV7Dk3 JL+YCe+KYmnOkU0gaary2LQaEAmQ9poYnGuu0wv2lrE+BgvL7BGk/1Kx17jEQgHJ IT1lpXVYD+gyrIlbNDOn7UUemMS+BS35HkYyneVIbvw=; Received: from [213.92.90.12] ([213.92.90.12:53725] helo=mail3.tomato.it) by t.contactlab.it (envelope-from ) (ecelerity 3.5.1.37854 r(Momo-dev:3.5.1.0)) with ESMTP id D3/F9-24145-06E1C415; Fri, 22 Mar 2013 10:03:28 +0100 Received: from mx3-master.housing.tomato.lan ([172.16.7.55]) by mail3.tomato.it with smtp (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UIxsm-000NYc-Kx for freebsd-performance@freebsd.org; Fri, 22 Mar 2013 10:03:28 +0100 Received: (qmail 90556 invoked by uid 89); 22 Mar 2013 09:03:28 -0000 Received: from localhost (HELO davepro.local) (127.0.0.1) by mx3-master.housing.tomato.lan with SMTP; 22 Mar 2013 09:03:28 -0000 Message-ID: <514C1E5F.8040504@contactlab.com> Date: Fri, 22 Mar 2013 10:03:27 +0100 From: Davide D'Amico User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130307 Thunderbird/17.0.4 MIME-Version: 1.0 To: freebsd-performance@freebsd.org Subject: FreeBSD 9.1 vs CentOS 6.3 Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2013 09:03:37 -0000 Hi, I'm doing performance tests on a DELL R720, follows dmesg: Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.1-RELEASE #0: Tue Mar 19 10:24:21 CET 2013 root:/usr/obj/usr/src/sys/R720 amd64 CPU: Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz (2500.05-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206d7 Family = 6 Model = 2d Stepping = 7 Features=0xbfebfbff Features2=0x1fbee3ff AMD Features=0x2c100800 AMD Features2=0x1 TSC: P-state invariant, performance statistics real memory = 34359738368 (32768 MB) avail memory = 33027436544 (31497 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 2 package(s) x 2 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 10 cpu3 (AP): APIC ID: 11 cpu4 (AP): APIC ID: 32 cpu5 (AP): APIC ID: 33 cpu6 (AP): APIC ID: 42 cpu7 (AP): APIC ID: 43 ioapic1: Changing APIC ID to 1 ioapic2: Changing APIC ID to 2 ioapic0 irqs 0-23 on motherboard ioapic1 irqs 32-55 on motherboard ioapic2 irqs 64-87 on motherboard kbd1 at kbdmux0 ctl: CAM Target Layer loaded acpi0: on motherboard acpi0: Power Button (fixed) acpi0: reservation of 0, 2000 (3) failed cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 atrtc0: port 0x70-0x7f irq 8 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 attimer0: port 0x40-0x5f irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 550 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 53 at device 1.0 on pci0 pci1: on pcib1 igb0: port 0xfc80-0xfc9f mem 0xdcb00000-0xdcbfffff,0xdcff0000-0xdcff3fff irq 38 at device 0.0 on pci1 igb0: Using MSIX interrupts with 9 vectors igb0: Ethernet address: bc:30:5b:ef:27:b8 [...] igb1: port 0xfca0-0xfcbf mem 0xdcc00000-0xdccfffff,0xdcff4000-0xdcff7fff irq 37 at device 0.1 on pci1 igb1: Using MSIX interrupts with 9 vectors igb1: Ethernet address: bc:30:5b:ef:27:b9 [...] igb2: port 0xfcc0-0xfcdf mem 0xdcd00000-0xdcdfffff,0xdcff8000-0xdcffbfff irq 36 at device 0.2 on pci1 igb2: Using MSIX interrupts with 9 vectors igb2: Ethernet address: bc:30:5b:ef:27:ba [...] igb3: port 0xfce0-0xfcff mem 0xdce00000-0xdcefffff,0xdcffc000-0xdcffffff irq 34 at device 0.3 on pci1 igb3: Using MSIX interrupts with 9 vectors igb3: Ethernet address: bc:30:5b:ef:27:bb [...] pcib2: irq 53 at device 2.0 on pci0 pci3: on pcib2 pcib3: irq 53 at device 2.2 on pci0 pci2: on pcib3 mfi0: port 0xec00-0xecff mem 0xddffc000-0xddffffff,0xddf80000-0xddfbffff irq 42 at device 0.0 on pci2 mfi0: Using MSI mfi0: Megaraid SAS driver Ver 4.23 mfi0: MaxCmd = 3f0 MaxSgl = 46 state = b73c03f0 pcib4: irq 53 at device 3.0 on pci0 pci4: on pcib4 pcib5: irq 53 at device 3.2 on pci0 pci5: on pcib5 pci0: at device 5.0 (no driver attached) pci0: at device 5.2 (no driver attached) pcib6: irq 16 at device 17.0 on pci0 pci6: on pcib6 pci0: at device 22.0 (no driver attached) pci0: at device 22.1 (no driver attached) ehci0: mem 0xdf8fe000-0xdf8fe3ff irq 23 at device 26.0 on pci0 usbus0: EHCI version 1.0 usbus0 on ehci0 pcib7: at device 28.0 on pci0 device_attach: pcib7 attach returned 6 pcib7: irq 19 at device 28.7 on pci0 pci8: on pcib7 pcib8: at device 0.0 on pci8 pci9: on pcib8 pcib9: at device 0.0 on pci9 pci10: on pcib9 pcib10: at device 0.0 on pci10 pci11: on pcib10 vgapci0: mem 0xd8000000-0xd8ffffff,0xdeffc000-0xdeffffff,0xde000000-0xde7fffff irq 19 at device 0.0 on pci11 pcib11: at device 1.0 on pci9 pci12: on pcib11 ehci1: mem 0xdf8ff000-0xdf8ff3ff irq 22 at device 29.0 on pci0 usbus1: EHCI version 1.0 usbus1 on ehci1 pcib12: at device 30.0 on pci0 pci13: on pcib12 isab0: at device 31.0 on pci0 isa0: on isab0 pcib13: on acpi0 pci64: on pcib13 pcib14: irq 85 at device 1.0 on pci64 pci65: on pcib14 pcib15: irq 85 at device 2.0 on pci64 pci66: on pcib15 pcib16: irq 85 at device 3.0 on pci64 pci67: on pcib16 pcib17: irq 85 at device 3.2 on pci64 pci68: on pcib17 pci64: at device 5.0 (no driver attached) pci64: at device 5.2 (no driver attached) pcib18: on acpi0 pci63: on pcib18 pci63: at device 8.0 (no driver attached) [...] pci63: at device 14.1 (no driver attached) [...] pci63: at device 19.1 (no driver attached) pci63: at device 19.4 (no driver attached) pci63: at device 19.5 (no driver attached) pci63: at device 19.6 (no driver attached) pcib19: on acpi0 pci127: on pcib19 [...] pci127: at device 14.1 (no driver attached) [...] pci127: at device 19.1 (no driver attached) pci127: at device 19.4 (no driver attached) pci127: at device 19.5 (no driver attached) pci127: at device 19.6 (no driver attached) uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 orm0: at iomem 0xc0000-0xc7fff,0xec000-0xeffff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 226d00001e00 device_attach: est0 attach returned 6 p4tcc0: on cpu0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 226d00001e00 device_attach: est1 attach returned 6 p4tcc1: on cpu1 est2: on cpu2 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 226d00001e00 device_attach: est2 attach returned 6 p4tcc2: on cpu2 est3: on cpu3 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 224400001e00 device_attach: est3 attach returned 6 p4tcc3: on cpu3 est4: on cpu4 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 226d00001e00 device_attach: est4 attach returned 6 p4tcc4: on cpu4 est5: on cpu5 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 224400001e00 device_attach: est5 attach returned 6 p4tcc5: on cpu5 est6: on cpu6 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 224400001e00 device_attach: est6 attach returned 6 p4tcc6: on cpu6 est7: on cpu7 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 224400001e00 device_attach: est7 attach returned 6 p4tcc7: on cpu7 mfi0: 6040 (417120269s/0x0020/info) - Shutdown command received from host mfi0: 6041 (boot + 4s/0x0020/info) - Firmware initialization started (PCI ID 005b/1000/1f34/1028) mfi0: 6042 (boot + 4s/0x0020/info) - Firmware version 3.130.05-1796 mfi0: 6043 (boot + 5s/0x0008/info) - Battery Present mfi0: 6044 (boot + 5s/0x0020/info) - Package version 21.1.0-0007 mfi0: 6045 (boot + 5s/0x0020/info) - Board Revision A01 mfi0: 6046 (boot + 6s/0x0008/info) - Battery temperature is normal mfi0: 6047 (boot + 6s/0x0008/info) - Current capacity of the battery is above threshold mfi0: 6048 (boot + 19s/0x0004/info) - Enclosure PD 20(c None/p1) communication restored mfi0: 6049 (boot + 19s/0x0002/info) - Inserted: Encl PD 20 mfi0: 6050 (boot + 19s/0x0002/info) - Inserted: PD 20(c None/p1) Info: enclPd=20, scsiType=d, portMap=00, sasAddr=5e4ae020a26cb500,0000000000000000 mfi0: 6051 (boot + 19s/0x0002/info) - Inserted: PD 00(e0x20/s0) mfi0: 6052 (boot + 19s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: enclPd=20, scsiType=0, portMap=04, sasAddr=5000c50054982339,0000000000000000 mfi0: 6053 (boot + 19s/0x0002/info) - Inserted: PD 01(e0x20/s1) mfi0: 6054 (boot + 19s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info: enclPd=20, scsiType=0, portMap=05, sasAddr=5000c50054986c91,0000000000000000 mfi0: 6055 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2) mfi0: 6056 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info: enclPd=20, scsiType=0, portMap=06, sasAddr=5000c50053da8f75,0000000000000000 mfi0: 6057 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s3) mfi0: 6058 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s3) Info: enclPd=20, scsiType=0, portMap=07, sasAddr=5000c50053c57eb5,0000000000000000 mfi0: 6059 (boot + 19s/0x0002/info) - Inserted: PD 04(e0x20/s4) mfi0: 6060 (boot + 19s/0x0002/info) - Inserted: PD 04(e0x20/s4) Info: enclPd=20, scsiType=0, portMap=00, sasAddr=5000c50053d364c5,0000000000000000 mfi0: 6061 (boot + 19s/0x0002/info) - Inserted: PD 05(e0x20/s5) mfi0: 6062 (boot + 19s/0x0002/info) - Inserted: PD 05(e0x20/s5) Info: enclPd=20, scsiType=0, portMap=03, sasAddr=5000c50053da2b41,0000000000000000 mfi0: 6063 (boot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6) mfi0: 6064 (boot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6) Info: enclPd=20, scsiType=0, portMap=02, sasAddr=4433221102000000,0000000000000000 mfi0: 6065 (boot + 19s/0x0002/WARN) - PD 06(e0x20/s6) is not a certified drive mfi0: 6066 (boot + 19s/0x0002/info) - Inserted: PD 07(e0x20/s7) mfi0: 6067 (boot + 19s/0x0002/info) - Inserted: PD 07(e0x20/s7) Info: enclPd=20, scsiType=0, portMap=01, sasAddr=4433221103000000,0000000000000000 mfi0: 6068 (boot + 19s/0x0002/WARN) - PD 07(e0x20/s7) is not a certified drive mfi0: 6069 (417120312s/0x0020/info) - Time established as 03/20/13 18:45:12; (32 seconds since power on) mfi0: 6070 (417120445s/0x0020/info) - Host driver is loaded and operational Timecounters tick every 1.000 msec usbus0: 480Mbps High Speed USB v2.0 usbus1: 480Mbps High Speed USB v2.0 mfid0 on mfi0 mfid0: 285568MB (584843264 sectors) RAID volume 'BASE' is optimal mfid1 on mfi0 mfid1: 121472MB (248774656 sectors) RAID volume 'L2ARC0' is optimal mfid2 on mfi0 mfid2: 121472MB (248774656 sectors) RAID volume 'ZIL0' is optimal mfid3 on mfi0 mfid3: 1143552MB (2341994496 sectors) RAID volume 'DATASTRIPE1' is optimal mfid4 on mfi0 mfid4: 1143552MB (2341994496 sectors) RAID volume 'DATASTRIPE2' is optimal ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered ugen0.2: at usbus0 uhub2: on usbus0 ugen1.2: at usbus1 uhub3: on usbus1 uhub2: 6 ports with 6 removable, self powered uhub3: 8 ports with 8 removable, self powered ugen0.3: at usbus0 uhub4: on usbus0 SMP: AP CPU #1 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #4 Launched! Timecounter "TSC-low" frequency 9765827 Hz quality 1000 Root mount waiting for: usbus0 uhub4: 6 ports with 6 removable, self powered ugen0.4: at usbus0 ukbd0: on usbus0 kbd0 at ukbd0 Root mount waiting for: usbus0 Root mount waiting for: usbus0 ugen0.5: at usbus0 umass0: on usbus0 Trying to mount root from ufs:/dev/mfid0p3 [rw]... cd0 at umass-sim0 bus 0 scbus1 target 0 lun 0 cd0: Removable CD-ROM SCSI-0 device cd0: 40.000MB/s transfers cd0: Attempt to query device size failed: NOT READY, Medium not present da0 at umass-sim0 bus 0 scbus1 target 0 lun 1 da0: Removable Direct Access SCSI-0 device da0: 40.000MB/s transfers da0: Attempt to query device size failed: NOT READY, Medium not present ZFS filesystem version 5 ZFS storage pool version 28 ums0: on usbus0 ums0: 3 buttons and [Z] coordinates ID=0 ums1: on usbus0 ums1: 3 buttons and [XYZ] coordinates ID=0 igb0: link state changed to UP With this loader.conf: vm.kmem_size_max="64G" vm.kmem_size="48G" vfs.zfs.arc_min="4096M" vfs.zfs.arc_max="15872M" vfs.zfs.txg.synctime="1" vfs.zfs.txg.timeout="5" vfs.zfs.write_limit_override="0" vfs.zfs.txg.write_limit_override="0" vfs.zfs.zil_disable="0" And with this sysctl.conf: security.bsd.see_other_uids=0 net.inet.tcp.blackhole=2 net.inet.udp.blackhole=1 net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=65536 net.link.ether.inet.log_arp_wrong_iface=0 kern.ipc.somaxconn=32768 kern.threads.max_threads_per_proc=16384 kern.maxfiles=262144 kern.maxfilesperproc=131072 kern.ipc.nmbclusters=65536 kern.corefile="/var/coredumps/%U.%N.%P.core" vfs.zfs.txg.timeout=5 vfs.zfs.l2arc_headroom=16 vfs.zfs.l2arc_write_max=26843546 vfs.zfs.l2arc_write_boost=26843546 vfs.zfs.vdev.min_pending=4 vfs.zfs.vdev.max_pending=64 vfs.zfs.l2arc_noprefetch=1 vfs.zfs.dedup.prefetch=0 vfs.zfs.prefetch_disable=1 I will use this server as a mysql-5.6 dbserver so I have a root partition using a hw raid1 and a /DATAZFS partition, follows configuration: root# mfiutil show drives mfi0 Physical Drives: 0 ( 279G) ONLINE SAS E1:S0 1 ( 279G) ONLINE SAS E1:S1 2 ( 558G) ONLINE SAS E1:S2 3 ( 558G) ONLINE SAS E1:S3 4 ( 558G) ONLINE SAS E1:S4 5 ( 558G) ONLINE SAS E1:S5 6 ( 119G) ONLINE SATA E1:S6 7 ( 119G) ONLINE SATA E1:S7 root# mfiutil show volumes mfi0 Volumes: Id Size Level Stripe State Cache Name mfid0 ( 278G) RAID-1 64k OPTIMAL Disabled mfid1 ( 118G) RAID-0 64k OPTIMAL Disabled mfid2 ( 118G) RAID-0 64k OPTIMAL Disabled mfid3 ( 1116G) RAID-1 64k OPTIMAL Disabled I have a /DATAZFS partition, too: root# zfs get all DATAZFS NAME PROPERTY VALUE SOURCE DATAZFS type filesystem - DATAZFS creation Thu Mar 21 10:45 2013 - DATAZFS used 54.2G - DATAZFS available 1.02T - DATAZFS referenced 54.2G - DATAZFS compressratio 1.00x - DATAZFS mounted yes - DATAZFS quota none default DATAZFS reservation none default DATAZFS recordsize 16K local DATAZFS mountpoint /DATAZFS default DATAZFS sharenfs off default DATAZFS checksum on default DATAZFS compression off default DATAZFS atime off local DATAZFS devices on default DATAZFS exec on default DATAZFS setuid on default DATAZFS readonly off default DATAZFS jailed off default DATAZFS snapdir hidden default DATAZFS aclmode discard default DATAZFS aclinherit restricted default DATAZFS canmount on default DATAZFS xattr off temporary DATAZFS copies 1 default DATAZFS version 5 - DATAZFS utf8only off - DATAZFS normalization none - DATAZFS casesensitivity sensitive - DATAZFS vscan off default DATAZFS nbmand off default DATAZFS sharesmb off default DATAZFS refquota none default DATAZFS refreservation none default DATAZFS primarycache all default DATAZFS secondarycache all default DATAZFS usedbysnapshots 0 - DATAZFS usedbydataset 54.2G - DATAZFS usedbychildren 1.41M - DATAZFS usedbyrefreservation 0 - DATAZFS logbias latency default DATAZFS dedup off local DATAZFS mlslabel - DATAZFS sync standard default DATAZFS refcompressratio 1.00x - DATAZFS written 54.2G - root# zpool status -v pool: DATAZFS state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM DATAZFS ONLINE 0 0 0 mfid3 ONLINE 0 0 0 logs mfid2 ONLINE 0 0 0 cache mfid1 ONLINE 0 0 0 errors: No known data errors root# mfiutil show firmware mfi0 Firmware Package Version: 21.1.0-0007 mfi0 Firmware Images: Name Version Date Time Status BIOS 5.31.01_4.12.05.00_0x05110000 6/20/2012 6/20/2012 active CTLR 4.00-0014 Aug 04 2011 12:49:17 active PCLI 05.00-03:#%00008 Feb 17 2011 14:03:12 active APP 3.130.05-1796 Jul 27 2012 15:58:31 active NVDT 2.1108.03-0094 Jun 22 2012 12:40:55 active BTBL 2.03.00.00-0004 Jul 27 2012 16:39:29 active BOOT 06.253.57.219 9/9/2010 15:32:25 active root# Then, I installed CentOS on the same hardware, same version of mysql (5.6.10-enterprise and 5.6.10-GA), and I tried an OLTP test (included in sysbench-0.5) executing 2000000 queries on 10000000 rows, with these results (transactions per seconds): BSD 9.1 amd64 ZFS (without tweaks): 1 thread: 2780 2 threads: 4371 4 threads: 6876 8 threads: 8202 12 threads: 8077 16 threads: 7780 24 threads: 7563 32 threads: 7632 48 threads: 7960 64 threads: 8062 BSD 9.1 amd64 ZFS (with tweaks): 1 thread: 2589 2 threads: 4679 4 threads: 6438 8 threads: 7073 12 threads: 7121 16 threads: 7227 24 threads: 7132 32 threads: 7273 48 threads: 7623 64 threads: 7672 CentOS 6.3 (ext4, without tweaks): 1 thread: 4163 2 threads: 7653 4 threads: 10905 8 threads: 12511 12 threads: 13556 16 threads: 14832 24 threads: 16270 32 threads: 16733 48 threads: 16925 64 threads: 16895 Well, any of you could ask "why zfs, please try ufs that is better": you are right, but changing ZFS with UFS doesn't change so much: http://lists.freebsd.org/pipermail/freebsd-fs/2013-March/016747.html So I hope I'm missing something because the performances of a standard CentOS installation on the same hardware performs much better than a 9.1 amd64 installation (I'm using the standard ULE scheduler included in the 9.1 release). Could anyone help me? Thanks, d. From owner-freebsd-performance@FreeBSD.ORG Fri Mar 22 10:01:01 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id DD166F1 for ; Fri, 22 Mar 2013 10:01:01 +0000 (UTC) (envelope-from mike@thesandbenders.com) Received: from mail-ia0-x234.google.com (mail-ia0-x234.google.com [IPv6:2607:f8b0:4001:c02::234]) by mx1.freebsd.org (Postfix) with ESMTP id AFF85710 for ; Fri, 22 Mar 2013 10:01:01 +0000 (UTC) Received: by mail-ia0-f180.google.com with SMTP id f27so3404949iae.11 for ; Fri, 22 Mar 2013 03:01:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=thesandbenders.com; s=google; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type; bh=K/ZwZV3eflbZexh6s2wkm9no1kCCnubMC4RnijJG9CE=; b=bVnO8czEslKPsVaJ8CTbRl2UjhPJM/32Z8NC1hX+8hQ9/8ncf98T4qZy6uSGOUTPeO UyxDjvFmCh2F7kb6hiURAqM9WA6ZyYr2wrylJQsSTp7eZ49K8bLxYKt9rBGr+UA2VY0s W9c5C+AE6wngsFvfft1mtMHKSgVHWBFjauc7Y= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:mime-version:in-reply-to:references:from:date:message-id :subject:to:cc:content-type:x-gm-message-state; bh=K/ZwZV3eflbZexh6s2wkm9no1kCCnubMC4RnijJG9CE=; b=YAfFqZqyy6MKTCBGq/daFyvZpUhb4mlNsSI3XsidHbjZg6oiVKyEWLX7xY/oAIrcGk PbFdfmSSAEiVwMW0LdoFxKkWtGqat6tuRvXlmq4tw4YT4wj6MnzhhNuINuZOsQcGYT0f EAqDYd0iUTafwO29kDWA6n4RuS7PbEJJlkVE5ZnsEzCmVc67DU5R53jtO+Ad8U1hsMff w93JH0hcgR0Y8zvjHUHaH0Sn2XAfaozCWCyg9sjBPXOLAWQM+pCHhp8ZEGvOFBACMp57 p5nYjE+FspWkf2CYo+mlzCIJPobJIST5VCOBCRWmXoiD3BUTcd48N76vG3I0n9e0aLTW F6MA== X-Received: by 10.50.46.133 with SMTP id v5mr4232103igm.94.1363946461253; Fri, 22 Mar 2013 03:01:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.170.198 with HTTP; Fri, 22 Mar 2013 03:00:40 -0700 (PDT) In-Reply-To: <514C1E5F.8040504@contactlab.com> References: <514C1E5F.8040504@contactlab.com> From: "Traffanstead, Mike" Date: Fri, 22 Mar 2013 06:00:40 -0400 Message-ID: Subject: Re: FreeBSD 9.1 vs CentOS 6.3 To: "Davide D'Amico" Content-Type: text/plain; charset=ISO-8859-1 X-Gm-Message-State: ALoCoQlNjcWcQT32Ds25Lj4ZZDyM/r20z0q4/mfRqywKCcr9Ex23vbLqMeZc3+XNitncfxwoCOuY Cc: freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2013 10:01:01 -0000 May I ask why you're running ZFS on top of a RAID array? That's not recommended. One of the advantages of ZFS is that it balance disk activity across devices but when put it on top drives that at are already raided it loses that insight and may end up scheduling reads/writes that all land on the same device. The only case where it's okay to do this is if you mirroring individual disks (e.g. several RAID-1 devices) and even that's arguable. Mike On Fri, Mar 22, 2013 at 5:03 AM, Davide D'Amico wrote: > Hi, I'm doing performance tests on a DELL R720, follows dmesg: > > Copyright (c) 1992-2012 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 9.1-RELEASE #0: Tue Mar 19 10:24:21 CET 2013 > root:/usr/obj/usr/src/sys/R720 amd64 > CPU: Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz (2500.05-MHz K8-class CPU) > Origin = "GenuineIntel" Id = 0x206d7 Family = 6 Model = 2d Stepping = 7 > Features=0xbfebfbff > Features2=0x1fbee3ff > AMD Features=0x2c100800 > AMD Features2=0x1 > TSC: P-state invariant, performance statistics > real memory = 34359738368 (32768 MB) > avail memory = 33027436544 (31497 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs > FreeBSD/SMP: 2 package(s) x 2 core(s) x 2 SMT threads > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 10 > cpu3 (AP): APIC ID: 11 > cpu4 (AP): APIC ID: 32 > cpu5 (AP): APIC ID: 33 > cpu6 (AP): APIC ID: 42 > cpu7 (AP): APIC ID: 43 > ioapic1: Changing APIC ID to 1 > ioapic2: Changing APIC ID to 2 > ioapic0 irqs 0-23 on motherboard > ioapic1 irqs 32-55 on motherboard > ioapic2 irqs 64-87 on motherboard > kbd1 at kbdmux0 > ctl: CAM Target Layer loaded > acpi0: on motherboard > acpi0: Power Button (fixed) > acpi0: reservation of 0, 2000 (3) failed > cpu0: on acpi0 > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > cpu4: on acpi0 > cpu5: on acpi0 > cpu6: on acpi0 > cpu7: on acpi0 > atrtc0: port 0x70-0x7f irq 8 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > attimer0: port 0x40-0x5f irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > hpet0: iomem 0xfed00000-0xfed003ff on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 950 > Event timer "HPET" frequency 14318180 Hz quality 550 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: irq 53 at device 1.0 on pci0 > pci1: on pcib1 > igb0: port > 0xfc80-0xfc9f mem 0xdcb00000-0xdcbfffff,0xdcff0000-0xdcff3fff irq 38 at > device 0.0 on pci1 > igb0: Using MSIX interrupts with 9 vectors > igb0: Ethernet address: bc:30:5b:ef:27:b8 > [...] > igb1: port > 0xfca0-0xfcbf mem 0xdcc00000-0xdccfffff,0xdcff4000-0xdcff7fff irq 37 at > device 0.1 on pci1 > igb1: Using MSIX interrupts with 9 vectors > igb1: Ethernet address: bc:30:5b:ef:27:b9 > [...] > igb2: port > 0xfcc0-0xfcdf mem 0xdcd00000-0xdcdfffff,0xdcff8000-0xdcffbfff irq 36 at > device 0.2 on pci1 > igb2: Using MSIX interrupts with 9 vectors > igb2: Ethernet address: bc:30:5b:ef:27:ba > [...] > igb3: port > 0xfce0-0xfcff mem 0xdce00000-0xdcefffff,0xdcffc000-0xdcffffff irq 34 at > device 0.3 on pci1 > igb3: Using MSIX interrupts with 9 vectors > igb3: Ethernet address: bc:30:5b:ef:27:bb > [...] > pcib2: irq 53 at device 2.0 on pci0 > pci3: on pcib2 > pcib3: irq 53 at device 2.2 on pci0 > pci2: on pcib3 > mfi0: port 0xec00-0xecff mem > 0xddffc000-0xddffffff,0xddf80000-0xddfbffff irq 42 at device 0.0 on pci2 > mfi0: Using MSI > mfi0: Megaraid SAS driver Ver 4.23 > mfi0: MaxCmd = 3f0 MaxSgl = 46 state = b73c03f0 > pcib4: irq 53 at device 3.0 on pci0 > pci4: on pcib4 > pcib5: irq 53 at device 3.2 on pci0 > pci5: on pcib5 > pci0: at device 5.0 (no driver attached) > pci0: at device 5.2 (no driver attached) > pcib6: irq 16 at device 17.0 on pci0 > pci6: on pcib6 > pci0: at device 22.0 (no driver attached) > pci0: at device 22.1 (no driver attached) > ehci0: mem 0xdf8fe000-0xdf8fe3ff irq 23 > at device 26.0 on pci0 > usbus0: EHCI version 1.0 > usbus0 on ehci0 > pcib7: at device 28.0 on pci0 > device_attach: pcib7 attach returned 6 > pcib7: irq 19 at device 28.7 on pci0 > pci8: on pcib7 > pcib8: at device 0.0 on pci8 > pci9: on pcib8 > pcib9: at device 0.0 on pci9 > pci10: on pcib9 > pcib10: at device 0.0 on pci10 > pci11: on pcib10 > vgapci0: mem > 0xd8000000-0xd8ffffff,0xdeffc000-0xdeffffff,0xde000000-0xde7fffff irq 19 at > device 0.0 on pci11 > pcib11: at device 1.0 on pci9 > pci12: on pcib11 > ehci1: mem 0xdf8ff000-0xdf8ff3ff irq 22 > at device 29.0 on pci0 > usbus1: EHCI version 1.0 > usbus1 on ehci1 > pcib12: at device 30.0 on pci0 > pci13: on pcib12 > isab0: at device 31.0 on pci0 > isa0: on isab0 > pcib13: on acpi0 > pci64: on pcib13 > pcib14: irq 85 at device 1.0 on pci64 > pci65: on pcib14 > pcib15: irq 85 at device 2.0 on pci64 > pci66: on pcib15 > pcib16: irq 85 at device 3.0 on pci64 > pci67: on pcib16 > pcib17: irq 85 at device 3.2 on pci64 > pci68: on pcib17 > pci64: at device 5.0 (no driver attached) > pci64: at device 5.2 (no driver attached) > pcib18: on acpi0 > pci63: on pcib18 > pci63: at device 8.0 (no driver attached) > [...] > pci63: at device 14.1 (no driver attached) > [...] > pci63: at device 19.1 (no driver attached) > pci63: at device 19.4 (no driver attached) > pci63: at device 19.5 (no driver attached) > pci63: at device 19.6 (no driver attached) > pcib19: on acpi0 > pci127: on pcib19 > [...] > pci127: at device 14.1 (no driver attached) > [...] > pci127: at device 19.1 (no driver attached) > pci127: at device 19.4 (no driver attached) > pci127: at device 19.5 (no driver attached) > pci127: at device 19.6 (no driver attached) > uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > orm0: at iomem 0xc0000-0xc7fff,0xec000-0xeffff on isa0 > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > est0: on cpu0 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 226d00001e00 > device_attach: est0 attach returned 6 > p4tcc0: on cpu0 > est1: on cpu1 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 226d00001e00 > device_attach: est1 attach returned 6 > p4tcc1: on cpu1 > est2: on cpu2 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 226d00001e00 > device_attach: est2 attach returned 6 > p4tcc2: on cpu2 > est3: on cpu3 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 224400001e00 > device_attach: est3 attach returned 6 > p4tcc3: on cpu3 > est4: on cpu4 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 226d00001e00 > device_attach: est4 attach returned 6 > p4tcc4: on cpu4 > est5: on cpu5 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 224400001e00 > device_attach: est5 attach returned 6 > p4tcc5: on cpu5 > est6: on cpu6 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 224400001e00 > device_attach: est6 attach returned 6 > p4tcc6: on cpu6 > est7: on cpu7 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 224400001e00 > device_attach: est7 attach returned 6 > p4tcc7: on cpu7 > mfi0: 6040 (417120269s/0x0020/info) - Shutdown command received from host > mfi0: 6041 (boot + 4s/0x0020/info) - Firmware initialization started (PCI ID > 005b/1000/1f34/1028) > mfi0: 6042 (boot + 4s/0x0020/info) - Firmware version 3.130.05-1796 > mfi0: 6043 (boot + 5s/0x0008/info) - Battery Present > mfi0: 6044 (boot + 5s/0x0020/info) - Package version 21.1.0-0007 > mfi0: 6045 (boot + 5s/0x0020/info) - Board Revision A01 > mfi0: 6046 (boot + 6s/0x0008/info) - Battery temperature is normal > mfi0: 6047 (boot + 6s/0x0008/info) - Current capacity of the battery is > above threshold > mfi0: 6048 (boot + 19s/0x0004/info) - Enclosure PD 20(c None/p1) > communication restored > mfi0: 6049 (boot + 19s/0x0002/info) - Inserted: Encl PD 20 > mfi0: 6050 (boot + 19s/0x0002/info) - Inserted: PD 20(c None/p1) Info: > enclPd=20, scsiType=d, portMap=00, sasAddr=5e4ae020a26cb500,0000000000000000 > mfi0: 6051 (boot + 19s/0x0002/info) - Inserted: PD 00(e0x20/s0) > mfi0: 6052 (boot + 19s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: > enclPd=20, scsiType=0, portMap=04, sasAddr=5000c50054982339,0000000000000000 > mfi0: 6053 (boot + 19s/0x0002/info) - Inserted: PD 01(e0x20/s1) > mfi0: 6054 (boot + 19s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info: > enclPd=20, scsiType=0, portMap=05, sasAddr=5000c50054986c91,0000000000000000 > mfi0: 6055 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2) > mfi0: 6056 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info: > enclPd=20, scsiType=0, portMap=06, sasAddr=5000c50053da8f75,0000000000000000 > mfi0: 6057 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s3) > mfi0: 6058 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s3) Info: > enclPd=20, scsiType=0, portMap=07, sasAddr=5000c50053c57eb5,0000000000000000 > mfi0: 6059 (boot + 19s/0x0002/info) - Inserted: PD 04(e0x20/s4) > mfi0: 6060 (boot + 19s/0x0002/info) - Inserted: PD 04(e0x20/s4) Info: > enclPd=20, scsiType=0, portMap=00, sasAddr=5000c50053d364c5,0000000000000000 > mfi0: 6061 (boot + 19s/0x0002/info) - Inserted: PD 05(e0x20/s5) > mfi0: 6062 (boot + 19s/0x0002/info) - Inserted: PD 05(e0x20/s5) Info: > enclPd=20, scsiType=0, portMap=03, sasAddr=5000c50053da2b41,0000000000000000 > mfi0: 6063 (boot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6) > mfi0: 6064 (boot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6) Info: > enclPd=20, scsiType=0, portMap=02, sasAddr=4433221102000000,0000000000000000 > mfi0: 6065 (boot + 19s/0x0002/WARN) - PD 06(e0x20/s6) is not a certified > drive > mfi0: 6066 (boot + 19s/0x0002/info) - Inserted: PD 07(e0x20/s7) > mfi0: 6067 (boot + 19s/0x0002/info) - Inserted: PD 07(e0x20/s7) Info: > enclPd=20, scsiType=0, portMap=01, sasAddr=4433221103000000,0000000000000000 > mfi0: 6068 (boot + 19s/0x0002/WARN) - PD 07(e0x20/s7) is not a certified > drive > mfi0: 6069 (417120312s/0x0020/info) - Time established as 03/20/13 18:45:12; > (32 seconds since power on) > mfi0: 6070 (417120445s/0x0020/info) - Host driver is loaded and operational > Timecounters tick every 1.000 msec > usbus0: 480Mbps High Speed USB v2.0 > usbus1: 480Mbps High Speed USB v2.0 > mfid0 on mfi0 > mfid0: 285568MB (584843264 sectors) RAID volume 'BASE' is optimal > mfid1 on mfi0 > mfid1: 121472MB (248774656 sectors) RAID volume 'L2ARC0' is optimal > mfid2 on mfi0 > mfid2: 121472MB (248774656 sectors) RAID volume 'ZIL0' is optimal > mfid3 on mfi0 > mfid3: 1143552MB (2341994496 sectors) RAID volume 'DATASTRIPE1' is optimal > mfid4 on mfi0 > mfid4: 1143552MB (2341994496 sectors) RAID volume 'DATASTRIPE2' is optimal > ugen0.1: at usbus0 > uhub0: on usbus0 > ugen1.1: at usbus1 > uhub1: on usbus1 > uhub0: 2 ports with 2 removable, self powered > uhub1: 2 ports with 2 removable, self powered > ugen0.2: at usbus0 > uhub2: on > usbus0 > ugen1.2: at usbus1 > uhub3: on > usbus1 > uhub2: 6 ports with 6 removable, self powered > uhub3: 8 ports with 8 removable, self powered > ugen0.3: at usbus0 > uhub4: on > usbus0 > SMP: AP CPU #1 Launched! > SMP: AP CPU #7 Launched! > SMP: AP CPU #3 Launched! > SMP: AP CPU #6 Launched! > SMP: AP CPU #2 Launched! > SMP: AP CPU #5 Launched! > SMP: AP CPU #4 Launched! > Timecounter "TSC-low" frequency 9765827 Hz quality 1000 > Root mount waiting for: usbus0 > uhub4: 6 ports with 6 removable, self powered > ugen0.4: at usbus0 > ukbd0: on usbus0 > kbd0 at ukbd0 > Root mount waiting for: usbus0 > Root mount waiting for: usbus0 > ugen0.5: at usbus0 > umass0: on usbus0 > Trying to mount root from ufs:/dev/mfid0p3 [rw]... > cd0 at umass-sim0 bus 0 scbus1 target 0 lun 0 > cd0: Removable CD-ROM SCSI-0 device > cd0: 40.000MB/s transfers > cd0: Attempt to query device size failed: NOT READY, Medium not present > da0 at umass-sim0 bus 0 scbus1 target 0 lun 1 > da0: Removable Direct Access SCSI-0 device > da0: 40.000MB/s transfers > da0: Attempt to query device size failed: NOT READY, Medium not present > ZFS filesystem version 5 > ZFS storage pool version 28 > ums0: on usbus0 > ums0: 3 buttons and [Z] coordinates ID=0 > ums1: on usbus0 > ums1: 3 buttons and [XYZ] coordinates ID=0 > igb0: link state changed to UP > > With this loader.conf: > vm.kmem_size_max="64G" > vm.kmem_size="48G" > vfs.zfs.arc_min="4096M" > vfs.zfs.arc_max="15872M" > vfs.zfs.txg.synctime="1" > vfs.zfs.txg.timeout="5" > vfs.zfs.write_limit_override="0" > vfs.zfs.txg.write_limit_override="0" > vfs.zfs.zil_disable="0" > > And with this sysctl.conf: > security.bsd.see_other_uids=0 > net.inet.tcp.blackhole=2 > net.inet.udp.blackhole=1 > net.inet.tcp.sendspace=65536 > net.inet.tcp.recvspace=65536 > net.link.ether.inet.log_arp_wrong_iface=0 > kern.ipc.somaxconn=32768 > kern.threads.max_threads_per_proc=16384 > kern.maxfiles=262144 > kern.maxfilesperproc=131072 > kern.ipc.nmbclusters=65536 > kern.corefile="/var/coredumps/%U.%N.%P.core" > > vfs.zfs.txg.timeout=5 > vfs.zfs.l2arc_headroom=16 > vfs.zfs.l2arc_write_max=26843546 > vfs.zfs.l2arc_write_boost=26843546 > vfs.zfs.vdev.min_pending=4 > vfs.zfs.vdev.max_pending=64 > vfs.zfs.l2arc_noprefetch=1 > vfs.zfs.dedup.prefetch=0 > vfs.zfs.prefetch_disable=1 > > I will use this server as a mysql-5.6 dbserver so I have a root partition > using a hw raid1 and a /DATAZFS partition, follows configuration: > > root# mfiutil show drives > mfi0 Physical Drives: > 0 ( 279G) ONLINE SAS E1:S0 > 1 ( 279G) ONLINE SAS E1:S1 > 2 ( 558G) ONLINE SAS E1:S2 > 3 ( 558G) ONLINE SAS E1:S3 > 4 ( 558G) ONLINE SAS E1:S4 > 5 ( 558G) ONLINE SAS E1:S5 > 6 ( 119G) ONLINE SATA E1:S6 > 7 ( 119G) ONLINE SATA E1:S7 > root# mfiutil show volumes > mfi0 Volumes: > Id Size Level Stripe State Cache Name > mfid0 ( 278G) RAID-1 64k OPTIMAL Disabled > mfid1 ( 118G) RAID-0 64k OPTIMAL Disabled > mfid2 ( 118G) RAID-0 64k OPTIMAL Disabled > mfid3 ( 1116G) RAID-1 64k OPTIMAL Disabled > > I have a /DATAZFS partition, too: > root# zfs get all DATAZFS > NAME PROPERTY VALUE SOURCE > DATAZFS type filesystem - > DATAZFS creation Thu Mar 21 10:45 2013 - > DATAZFS used 54.2G - > DATAZFS available 1.02T - > DATAZFS referenced 54.2G - > DATAZFS compressratio 1.00x - > DATAZFS mounted yes - > DATAZFS quota none default > DATAZFS reservation none default > DATAZFS recordsize 16K local > DATAZFS mountpoint /DATAZFS default > DATAZFS sharenfs off default > DATAZFS checksum on default > DATAZFS compression off default > DATAZFS atime off local > DATAZFS devices on default > DATAZFS exec on default > DATAZFS setuid on default > DATAZFS readonly off default > DATAZFS jailed off default > DATAZFS snapdir hidden default > DATAZFS aclmode discard default > DATAZFS aclinherit restricted default > DATAZFS canmount on default > DATAZFS xattr off temporary > DATAZFS copies 1 default > DATAZFS version 5 - > DATAZFS utf8only off - > DATAZFS normalization none - > DATAZFS casesensitivity sensitive - > DATAZFS vscan off default > DATAZFS nbmand off default > DATAZFS sharesmb off default > DATAZFS refquota none default > DATAZFS refreservation none default > DATAZFS primarycache all default > DATAZFS secondarycache all default > DATAZFS usedbysnapshots 0 - > DATAZFS usedbydataset 54.2G - > DATAZFS usedbychildren 1.41M - > DATAZFS usedbyrefreservation 0 - > DATAZFS logbias latency default > DATAZFS dedup off local > DATAZFS mlslabel - > DATAZFS sync standard default > DATAZFS refcompressratio 1.00x - > DATAZFS written 54.2G - > > root# zpool status -v > pool: DATAZFS > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > DATAZFS ONLINE 0 0 0 > mfid3 ONLINE 0 0 0 > logs > mfid2 ONLINE 0 0 0 > cache > mfid1 ONLINE 0 0 0 > > errors: No known data errors > > root# mfiutil show firmware > mfi0 Firmware Package Version: 21.1.0-0007 > mfi0 Firmware Images: > Name Version Date Time Status > BIOS 5.31.01_4.12.05.00_0x05110000 6/20/2012 > 6/20/2012 > active > CTLR 4.00-0014 Aug 04 2011 12:49:17 active > PCLI 05.00-03:#%00008 Feb 17 2011 14:03:12 active > APP 3.130.05-1796 Jul 27 2012 15:58:31 active > NVDT 2.1108.03-0094 Jun 22 2012 12:40:55 active > BTBL 2.03.00.00-0004 Jul 27 2012 16:39:29 active > BOOT 06.253.57.219 9/9/2010 15:32:25 active > root# > > Then, I installed CentOS on the same hardware, same version of mysql > (5.6.10-enterprise and 5.6.10-GA), and I tried an OLTP test (included in > sysbench-0.5) executing 2000000 queries on 10000000 rows, with these results > (transactions per seconds): > > BSD 9.1 amd64 ZFS (without tweaks): > 1 thread: 2780 > 2 threads: 4371 > 4 threads: 6876 > 8 threads: 8202 > 12 threads: 8077 > 16 threads: 7780 > 24 threads: 7563 > 32 threads: 7632 > 48 threads: 7960 > 64 threads: 8062 > > BSD 9.1 amd64 ZFS (with tweaks): > 1 thread: 2589 > 2 threads: 4679 > 4 threads: 6438 > 8 threads: 7073 > 12 threads: 7121 > 16 threads: 7227 > 24 threads: 7132 > 32 threads: 7273 > 48 threads: 7623 > 64 threads: 7672 > > CentOS 6.3 (ext4, without tweaks): > 1 thread: 4163 > 2 threads: 7653 > 4 threads: 10905 > 8 threads: 12511 > 12 threads: 13556 > 16 threads: 14832 > 24 threads: 16270 > 32 threads: 16733 > 48 threads: 16925 > 64 threads: 16895 > > > Well, any of you could ask "why zfs, please try ufs that is better": you are > right, but changing ZFS with UFS doesn't change so much: > http://lists.freebsd.org/pipermail/freebsd-fs/2013-March/016747.html > > So I hope I'm missing something because the performances of a standard > CentOS installation on the same hardware performs much better than a 9.1 > amd64 installation (I'm using the standard ULE scheduler included in the 9.1 > release). > > Could anyone help me? > > Thanks, > d. > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Fri Mar 22 10:06:50 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id ABE663FC for ; Fri, 22 Mar 2013 10:06:50 +0000 (UTC) (envelope-from davide.damico@contactlab.com) Received: from mail2.shared.smtp.contactlab.it (mail2.shared.smtp.contactlab.it [93.94.37.7]) by mx1.freebsd.org (Postfix) with ESMTP id 30F667B4 for ; Fri, 22 Mar 2013 10:06:49 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; d=contactlab.it; s=clab1; c=relaxed/relaxed; q=dns/txt; i=@contactlab.it; t=1363946808; h=From:Subject:Date:To:MIME-Version:Content-Type; bh=IK225Aw1QisgIObIFAeAM4dtM40ditgubwW0b+Sh7a0=; b=Yw+4SXj+ilLeIao+/dBJyaG/q/daS86Dj6HOs9nsASr7ra6JLR4c7A4MFZ+ELdUG JBFukmMuSwQog0FmBQhbePNuc8SvMcKnNzNdBysABi913kkwv3YRCOBqFIXD3IGb w5uIUWzajgcdaYsvsnBfJ6ZdZr/NDy9+tHnkx4RhfWA=; Received: from [213.92.90.12] ([213.92.90.12:28718] helo=mail3.tomato.it) by t.contactlab.it (envelope-from ) (ecelerity 3.5.1.37854 r(Momo-dev:3.5.1.0)) with ESMTP id D3/6E-24145-83D2C415; Fri, 22 Mar 2013 11:06:48 +0100 Received: from mx3-master.housing.tomato.lan ([172.16.7.55]) by mail3.tomato.it with smtp (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UIys4-0003Dl-3U for freebsd-performance@freebsd.org; Fri, 22 Mar 2013 11:06:48 +0100 Received: (qmail 12381 invoked by uid 89); 22 Mar 2013 10:06:48 -0000 Received: from localhost (HELO davepro.local) (127.0.0.1) by mx3-master.housing.tomato.lan with SMTP; 22 Mar 2013 10:06:48 -0000 Message-ID: <514C2D36.8090505@contactlab.com> Date: Fri, 22 Mar 2013 11:06:46 +0100 From: Davide D'Amico User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130307 Thunderbird/17.0.4 MIME-Version: 1.0 To: "Traffanstead, Mike" Subject: Re: FreeBSD 9.1 vs CentOS 6.3 References: <514C1E5F.8040504@contactlab.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2013 10:06:50 -0000 Il 22/03/13 11:00, Traffanstead, Mike ha scritto: > May I ask why you're running ZFS on top of a RAID array? That's not > recommended. One of the advantages of ZFS is that it balance disk > activity across devices but when put it on top drives that at are > already raided it loses that insight and may end up scheduling > reads/writes that all land on the same device. The only case where > it's okay to do this is if you mirroring individual disks (e.g. > several RAID-1 devices) and even that's arguable. > Hi, we tried different approaches to a /DATA partition (before trying using a ZFS /DATAZFS partition): - an UFS partition (/DATA) on hardware raid10; - a ZFS on hardware raid10; - a ZFS mirror on two hardware stripes; The UFS filesystems performed at 400MBps without any tweak while ZFS performed at 400MBps after tweaks. So I don't think that these levels of performaces are related to file system. Thanks, d. From owner-freebsd-performance@FreeBSD.ORG Fri Mar 22 13:16:18 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B8E23C15 for ; Fri, 22 Mar 2013 13:16:18 +0000 (UTC) (envelope-from nonesuch@longcount.org) Received: from mail-vb0-x22b.google.com (mail-vb0-x22b.google.com [IPv6:2607:f8b0:400c:c02::22b]) by mx1.freebsd.org (Postfix) with ESMTP id 7ACC2949 for ; Fri, 22 Mar 2013 13:16:18 +0000 (UTC) Received: by mail-vb0-f43.google.com with SMTP id fs19so2586509vbb.2 for ; Fri, 22 Mar 2013 06:16:18 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=x-received:subject:references:from:content-type:x-mailer :in-reply-to:message-id:date:to:content-transfer-encoding :mime-version:x-gm-message-state; bh=FKyVlTwGZW+q5MFD3NlACGRnQuJC2T5SC0KRuJ7c7Vg=; b=MqVGV8ZR+nz+6tHrati/29rNPJvAU8EgueZQ1XLYP+h5kuI3IU5QrFQTVu17Eg4DOA Sk6nyhdURd0NGAj5LZXMNRh9KwhrZM6LVEiGMTvBgDZXsW72rgY9719/Xv4SUUUIqlNV cqU+YSi9QeKTJod0kOj3uwM9/svrfHHxkjhdhjS8qRFcX9dwYzEromW5M3qtR5d/XTNr qAfSa6AYsRm3/Wgp42y4hjVjW7TYh+jJBlKL3iKnVF3n0jMTBPl1F4vgSeEoppx/FePB eUdG2CXvs7Fhy/GVlGSp0k4z6r6L5gy4XyCJBpU7X9DRu8SLup/qiAL0Og3Hb2m3ShCc F5fw== X-Received: by 10.52.32.230 with SMTP id m6mr1780237vdi.83.1363958177954; Fri, 22 Mar 2013 06:16:17 -0700 (PDT) Received: from [97.250.185.66] (66.sub-97-250-185.myvzw.com. [97.250.185.66]) by mx.google.com with ESMTPS id p7sm3013792vdt.2.2013.03.22.06.16.15 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Fri, 22 Mar 2013 06:16:16 -0700 (PDT) Subject: Re: FreeBSD 9.1 vs CentOS 6.3 References: <514C1E5F.8040504@contactlab.com> <514C2D36.8090505@contactlab.com> From: Mark Saad Content-Type: text/plain; charset=us-ascii X-Mailer: iPhone Mail (10B146) In-Reply-To: <514C2D36.8090505@contactlab.com> Message-Id: Date: Fri, 22 Mar 2013 09:16:13 -0400 To: "freebsd-performance@freebsd.org" Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) X-Gm-Message-State: ALoCoQl+dtpMoClII0Avqa6Bd2WunNigF6D7EhQ9W1TKF9N24bokQd2PZk8xaq/gaAVKx9xNkvWA X-Mailman-Approved-At: Fri, 22 Mar 2013 13:44:51 +0000 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2013 13:16:18 -0000 On Mar 22, 2013, at 6:06 AM, Davide D'Amico w= rote: > Il 22/03/13 11:00, Traffanstead, Mike ha scritto: >> May I ask why you're running ZFS on top of a RAID array? That's not >> recommended. One of the advantages of ZFS is that it balance disk >> activity across devices but when put it on top drives that at are >> already raided it loses that insight and may end up scheduling >> reads/writes that all land on the same device. The only case where >> it's okay to do this is if you mirroring individual disks (e.g. >> several RAID-1 devices) and even that's arguable. > Hi, > we tried different approaches to a /DATA partition (before trying using a Z= FS /DATAZFS partition): >=20 > - an UFS partition (/DATA) on hardware raid10; What are details of the ufs setup? Version, softupdate , softupdate journaled, gjournal . What mount options ? > - a ZFS on hardware raid10; > - a ZFS mirror on two hardware stripes; >=20 > The UFS filesystems performed at 400MBps without any tweak while ZFS perfo= rmed at 400MBps after tweaks. >=20 > So I don't think that these levels of performaces are related to file syst= em. >=20 > Thanks, > d. >=20 > _____________________________________________ What can you tell us about the raid card. And disks ? --- Mark saad | mark.saad@longcount.org From owner-freebsd-performance@FreeBSD.ORG Fri Mar 22 15:56:11 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A200AF9F for ; Fri, 22 Mar 2013 15:56:11 +0000 (UTC) (envelope-from worsediablo@gmail.com) Received: from mail-qa0-f52.google.com (mail-qa0-f52.google.com [209.85.216.52]) by mx1.freebsd.org (Postfix) with ESMTP id 5B46B8D8 for ; Fri, 22 Mar 2013 15:56:11 +0000 (UTC) Received: by mail-qa0-f52.google.com with SMTP id bs12so456491qab.11 for ; Fri, 22 Mar 2013 08:56:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=NbB0qkZZIAMW8exCz3+uFcILIe0gB2KvDL8PmQMsLFU=; b=yGKi6ORFoe6MVYsgDqk61ekYwSLAgfpmV5SA+mmhoZ/zjphKK3LMOez4Xt3rzgi1eD v9NY41eHe1NmEYVvpcXTzAdT2YxoUfBxu5wtnr2+xQPfWLdPaN7+cesbJ1wSxlJy+dVC 7a8tE7EYU4oDUudJ+HaxVVmg+88mTfoEziuIVNsLhBmH7Q7aSABYpezZn/R3zt1U5ROt 6DTw9e8I7g9kWLPZiK5MY3LgiOcBOT8/O2593Y5yIVL9esb5vRBHalUddh02st4eNgCD ZTN+cOchO+cyCXlRIJnb6nubl8DIaR0cATcIL+ctSU3svMNZ48JKYDfebyKmLzxUHJyU fY2w== MIME-Version: 1.0 X-Received: by 10.229.179.197 with SMTP id br5mr364190qcb.107.1363967770417; Fri, 22 Mar 2013 08:56:10 -0700 (PDT) Sender: worsediablo@gmail.com Received: by 10.229.84.73 with HTTP; Fri, 22 Mar 2013 08:56:09 -0700 (PDT) In-Reply-To: <514C1E5F.8040504@contactlab.com> References: <514C1E5F.8040504@contactlab.com> Date: Fri, 22 Mar 2013 19:56:09 +0400 X-Google-Sender-Auth: 96-h5BxSvtCdjqFNV4E7d4Q-ZGI Message-ID: Subject: Re: FreeBSD 9.1 vs CentOS 6.3 From: =?UTF-8?B?0JXQstCz0LXQvdC40Lkg0KXQvtGA0LrQuNC9?= To: "Davide D'Amico" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2013 15:56:11 -0000 Hi Davide! Are you sure that disk is the bottleneck in your test? Does systat -vm 1 show 100% busy for disk ? Evgeny. 2013/3/22 Davide D'Amico > Hi, I'm doing performance tests on a DELL R720, follows dmesg: > > Copyright (c) 1992-2012 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 9.1-RELEASE #0: Tue Mar 19 10:24:21 CET 2013 > root:/usr/obj/usr/src/sys/R720 amd64 > CPU: Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz (2500.05-MHz K8-class CPU) > Origin = "GenuineIntel" Id = 0x206d7 Family = 6 Model = 2d Stepping = > 7 > Features=0xbfebfbff APIC,SEP,MTRR,PGE,MCA,CMOV,**PAT,PSE36,CLFLUSH,DTS,ACPI,** > MMX,FXSR,SSE,SSE2,SS,HTT,TM,**PBE> > Features2=0x1fbee3ff VMX,SMX,EST,TM2,SSSE3,CX16,**xTPR,PDCM,PCID,DCA,SSE4.1,** > SSE4.2,x2APIC,POPCNT,TSCDLT,**AESNI,XSAVE,OSXSAVE,AVX> > AMD Features=0x2c100800 > AMD Features2=0x1 > TSC: P-state invariant, performance statistics > real memory = 34359738368 (32768 MB) > avail memory = 33027436544 (31497 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs > FreeBSD/SMP: 2 package(s) x 2 core(s) x 2 SMT threads > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 10 > cpu3 (AP): APIC ID: 11 > cpu4 (AP): APIC ID: 32 > cpu5 (AP): APIC ID: 33 > cpu6 (AP): APIC ID: 42 > cpu7 (AP): APIC ID: 43 > ioapic1: Changing APIC ID to 1 > ioapic2: Changing APIC ID to 2 > ioapic0 irqs 0-23 on motherboard > ioapic1 irqs 32-55 on motherboard > ioapic2 irqs 64-87 on motherboard > kbd1 at kbdmux0 > ctl: CAM Target Layer loaded > acpi0: on motherboard > acpi0: Power Button (fixed) > acpi0: reservation of 0, 2000 (3) failed > cpu0: on acpi0 > cpu1: on acpi0 > cpu2: on acpi0 > cpu3: on acpi0 > cpu4: on acpi0 > cpu5: on acpi0 > cpu6: on acpi0 > cpu7: on acpi0 > atrtc0: port 0x70-0x7f irq 8 on acpi0 > Event timer "RTC" frequency 32768 Hz quality 0 > attimer0: port 0x40-0x5f irq 0 on acpi0 > Timecounter "i8254" frequency 1193182 Hz quality 0 > Event timer "i8254" frequency 1193182 Hz quality 100 > Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > hpet0: iomem 0xfed00000-0xfed003ff on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 950 > Event timer "HPET" frequency 14318180 Hz quality 550 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > pcib1: irq 53 at device 1.0 on pci0 > pci1: on pcib1 > igb0: port > 0xfc80-0xfc9f mem 0xdcb00000-0xdcbfffff,**0xdcff0000-0xdcff3fff irq 38 at > device 0.0 on pci1 > igb0: Using MSIX interrupts with 9 vectors > igb0: Ethernet address: bc:30:5b:ef:27:b8 > [...] > igb1: port > 0xfca0-0xfcbf mem 0xdcc00000-0xdccfffff,**0xdcff4000-0xdcff7fff irq 37 at > device 0.1 on pci1 > igb1: Using MSIX interrupts with 9 vectors > igb1: Ethernet address: bc:30:5b:ef:27:b9 > [...] > igb2: port > 0xfcc0-0xfcdf mem 0xdcd00000-0xdcdfffff,**0xdcff8000-0xdcffbfff irq 36 at > device 0.2 on pci1 > igb2: Using MSIX interrupts with 9 vectors > igb2: Ethernet address: bc:30:5b:ef:27:ba > [...] > igb3: port > 0xfce0-0xfcff mem 0xdce00000-0xdcefffff,**0xdcffc000-0xdcffffff irq 34 at > device 0.3 on pci1 > igb3: Using MSIX interrupts with 9 vectors > igb3: Ethernet address: bc:30:5b:ef:27:bb > [...] > pcib2: irq 53 at device 2.0 on pci0 > pci3: on pcib2 > pcib3: irq 53 at device 2.2 on pci0 > pci2: on pcib3 > mfi0: port 0xec00-0xecff mem > 0xddffc000-0xddffffff,**0xddf80000-0xddfbffff irq 42 at device 0.0 on pci2 > mfi0: Using MSI > mfi0: Megaraid SAS driver Ver 4.23 > mfi0: MaxCmd = 3f0 MaxSgl = 46 state = b73c03f0 > pcib4: irq 53 at device 3.0 on pci0 > pci4: on pcib4 > pcib5: irq 53 at device 3.2 on pci0 > pci5: on pcib5 > pci0: at device 5.0 (no driver attached) > pci0: at device 5.2 (no driver attached) > pcib6: irq 16 at device 17.0 on pci0 > pci6: on pcib6 > pci0: at device 22.0 (no driver attached) > pci0: at device 22.1 (no driver attached) > ehci0: mem 0xdf8fe000-0xdf8fe3ff irq > 23 at device 26.0 on pci0 > usbus0: EHCI version 1.0 > usbus0 on ehci0 > pcib7: at device 28.0 on pci0 > device_attach: pcib7 attach returned 6 > pcib7: irq 19 at device 28.7 on pci0 > pci8: on pcib7 > pcib8: at device 0.0 on pci8 > pci9: on pcib8 > pcib9: at device 0.0 on pci9 > pci10: on pcib9 > pcib10: at device 0.0 on pci10 > pci11: on pcib10 > vgapci0: mem 0xd8000000-0xd8ffffff,** > 0xdeffc000-0xdeffffff,**0xde000000-0xde7fffff irq 19 at device 0.0 on > pci11 > pcib11: at device 1.0 on pci9 > pci12: on pcib11 > ehci1: mem 0xdf8ff000-0xdf8ff3ff irq > 22 at device 29.0 on pci0 > usbus1: EHCI version 1.0 > usbus1 on ehci1 > pcib12: at device 30.0 on pci0 > pci13: on pcib12 > isab0: at device 31.0 on pci0 > isa0: on isab0 > pcib13: on acpi0 > pci64: on pcib13 > pcib14: irq 85 at device 1.0 on pci64 > pci65: on pcib14 > pcib15: irq 85 at device 2.0 on pci64 > pci66: on pcib15 > pcib16: irq 85 at device 3.0 on pci64 > pci67: on pcib16 > pcib17: irq 85 at device 3.2 on pci64 > pci68: on pcib17 > pci64: at device 5.0 (no driver attached) > pci64: at device 5.2 (no driver attached) > pcib18: on acpi0 > pci63: on pcib18 > pci63: at device 8.0 (no driver attached) > [...] > pci63: at device 14.1 (no driver attached) > [...] > pci63: at device 19.1 (no driver attached) > pci63: at device 19.4 (no driver attached) > pci63: at device 19.5 (no driver attached) > pci63: at device 19.6 (no driver attached) > pcib19: on acpi0 > pci127: on pcib19 > [...] > pci127: at device 14.1 (no driver attached) > [...] > pci127: at device 19.1 (no driver attached) > pci127: at device 19.4 (no driver attached) > pci127: at device 19.5 (no driver attached) > pci127: at device 19.6 (no driver attached) > uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 > orm0: at iomem 0xc0000-0xc7fff,0xec000-**0xeffff on isa0 > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 > est0: on cpu0 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 226d00001e00 > device_attach: est0 attach returned 6 > p4tcc0: on cpu0 > est1: on cpu1 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 226d00001e00 > device_attach: est1 attach returned 6 > p4tcc1: on cpu1 > est2: on cpu2 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 226d00001e00 > device_attach: est2 attach returned 6 > p4tcc2: on cpu2 > est3: on cpu3 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 224400001e00 > device_attach: est3 attach returned 6 > p4tcc3: on cpu3 > est4: on cpu4 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 226d00001e00 > device_attach: est4 attach returned 6 > p4tcc4: on cpu4 > est5: on cpu5 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 224400001e00 > device_attach: est5 attach returned 6 > p4tcc5: on cpu5 > est6: on cpu6 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 224400001e00 > device_attach: est6 attach returned 6 > p4tcc6: on cpu6 > est7: on cpu7 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 224400001e00 > device_attach: est7 attach returned 6 > p4tcc7: on cpu7 > mfi0: 6040 (417120269s/0x0020/info) - Shutdown command received from host > mfi0: 6041 (boot + 4s/0x0020/info) - Firmware initialization started (PCI > ID 005b/1000/1f34/1028) > mfi0: 6042 (boot + 4s/0x0020/info) - Firmware version 3.130.05-1796 > mfi0: 6043 (boot + 5s/0x0008/info) - Battery Present > mfi0: 6044 (boot + 5s/0x0020/info) - Package version 21.1.0-0007 > mfi0: 6045 (boot + 5s/0x0020/info) - Board Revision A01 > mfi0: 6046 (boot + 6s/0x0008/info) - Battery temperature is normal > mfi0: 6047 (boot + 6s/0x0008/info) - Current capacity of the battery is > above threshold > mfi0: 6048 (boot + 19s/0x0004/info) - Enclosure PD 20(c None/p1) > communication restored > mfi0: 6049 (boot + 19s/0x0002/info) - Inserted: Encl PD 20 > mfi0: 6050 (boot + 19s/0x0002/info) - Inserted: PD 20(c None/p1) Info: > enclPd=20, scsiType=d, portMap=00, sasAddr=5e4ae020a26cb500,** > 0000000000000000 > mfi0: 6051 (boot + 19s/0x0002/info) - Inserted: PD 00(e0x20/s0) > mfi0: 6052 (boot + 19s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: > enclPd=20, scsiType=0, portMap=04, sasAddr=5000c50054982339,** > 0000000000000000 > mfi0: 6053 (boot + 19s/0x0002/info) - Inserted: PD 01(e0x20/s1) > mfi0: 6054 (boot + 19s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info: > enclPd=20, scsiType=0, portMap=05, sasAddr=5000c50054986c91,** > 0000000000000000 > mfi0: 6055 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2) > mfi0: 6056 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info: > enclPd=20, scsiType=0, portMap=06, sasAddr=5000c50053da8f75,** > 0000000000000000 > mfi0: 6057 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s3) > mfi0: 6058 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s3) Info: > enclPd=20, scsiType=0, portMap=07, sasAddr=5000c50053c57eb5,** > 0000000000000000 > mfi0: 6059 (boot + 19s/0x0002/info) - Inserted: PD 04(e0x20/s4) > mfi0: 6060 (boot + 19s/0x0002/info) - Inserted: PD 04(e0x20/s4) Info: > enclPd=20, scsiType=0, portMap=00, sasAddr=5000c50053d364c5,** > 0000000000000000 > mfi0: 6061 (boot + 19s/0x0002/info) - Inserted: PD 05(e0x20/s5) > mfi0: 6062 (boot + 19s/0x0002/info) - Inserted: PD 05(e0x20/s5) Info: > enclPd=20, scsiType=0, portMap=03, sasAddr=5000c50053da2b41,** > 0000000000000000 > mfi0: 6063 (boot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6) > mfi0: 6064 (boot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6) Info: > enclPd=20, scsiType=0, portMap=02, sasAddr=4433221102000000,** > 0000000000000000 > mfi0: 6065 (boot + 19s/0x0002/WARN) - PD 06(e0x20/s6) is not a certified > drive > mfi0: 6066 (boot + 19s/0x0002/info) - Inserted: PD 07(e0x20/s7) > mfi0: 6067 (boot + 19s/0x0002/info) - Inserted: PD 07(e0x20/s7) Info: > enclPd=20, scsiType=0, portMap=01, sasAddr=4433221103000000,** > 0000000000000000 > mfi0: 6068 (boot + 19s/0x0002/WARN) - PD 07(e0x20/s7) is not a certified > drive > mfi0: 6069 (417120312s/0x0020/info) - Time established as 03/20/13 > 18:45:12; (32 seconds since power on) > mfi0: 6070 (417120445s/0x0020/info) - Host driver is loaded and operational > Timecounters tick every 1.000 msec > usbus0: 480Mbps High Speed USB v2.0 > usbus1: 480Mbps High Speed USB v2.0 > mfid0 on mfi0 > mfid0: 285568MB (584843264 sectors) RAID volume 'BASE' is optimal > mfid1 on mfi0 > mfid1: 121472MB (248774656 sectors) RAID volume 'L2ARC0' is optimal > mfid2 on mfi0 > mfid2: 121472MB (248774656 sectors) RAID volume 'ZIL0' is optimal > mfid3 on mfi0 > mfid3: 1143552MB (2341994496 sectors) RAID volume 'DATASTRIPE1' is optimal > mfid4 on mfi0 > mfid4: 1143552MB (2341994496 sectors) RAID volume 'DATASTRIPE2' is optimal > ugen0.1: at usbus0 > uhub0: on usbus0 > ugen1.1: at usbus1 > uhub1: on usbus1 > uhub0: 2 ports with 2 removable, self powered > uhub1: 2 ports with 2 removable, self powered > ugen0.2: at usbus0 > uhub2: on > usbus0 > ugen1.2: at usbus1 > uhub3: on > usbus1 > uhub2: 6 ports with 6 removable, self powered > uhub3: 8 ports with 8 removable, self powered > ugen0.3: at usbus0 > uhub4: > on usbus0 > SMP: AP CPU #1 Launched! > SMP: AP CPU #7 Launched! > SMP: AP CPU #3 Launched! > SMP: AP CPU #6 Launched! > SMP: AP CPU #2 Launched! > SMP: AP CPU #5 Launched! > SMP: AP CPU #4 Launched! > Timecounter "TSC-low" frequency 9765827 Hz quality 1000 > Root mount waiting for: usbus0 > uhub4: 6 ports with 6 removable, self powered > ugen0.4: at usbus0 > ukbd0: on usbus0 > kbd0 at ukbd0 > Root mount waiting for: usbus0 > Root mount waiting for: usbus0 > ugen0.5: at usbus0 > umass0: on usbus0 > Trying to mount root from ufs:/dev/mfid0p3 [rw]... > cd0 at umass-sim0 bus 0 scbus1 target 0 lun 0 > cd0: Removable CD-ROM SCSI-0 device > cd0: 40.000MB/s transfers > cd0: Attempt to query device size failed: NOT READY, Medium not present > da0 at umass-sim0 bus 0 scbus1 target 0 lun 1 > da0: Removable Direct Access SCSI-0 device > da0: 40.000MB/s transfers > da0: Attempt to query device size failed: NOT READY, Medium not present > ZFS filesystem version 5 > ZFS storage pool version 28 > ums0: on usbus0 > ums0: 3 buttons and [Z] coordinates ID=0 > ums1: on usbus0 > ums1: 3 buttons and [XYZ] coordinates ID=0 > igb0: link state changed to UP > > With this loader.conf: > vm.kmem_size_max="64G" > vm.kmem_size="48G" > vfs.zfs.arc_min="4096M" > vfs.zfs.arc_max="15872M" > vfs.zfs.txg.synctime="1" > vfs.zfs.txg.timeout="5" > vfs.zfs.write_limit_override="**0" > vfs.zfs.txg.write_limit_**override="0" > vfs.zfs.zil_disable="0" > > And with this sysctl.conf: > security.bsd.see_other_uids=0 > net.inet.tcp.blackhole=2 > net.inet.udp.blackhole=1 > net.inet.tcp.sendspace=65536 > net.inet.tcp.recvspace=65536 > net.link.ether.inet.log_arp_**wrong_iface=0 > kern.ipc.somaxconn=32768 > kern.threads.max_threads_per_**proc=16384 > kern.maxfiles=262144 > kern.maxfilesperproc=131072 > kern.ipc.nmbclusters=65536 > kern.corefile="/var/coredumps/**%U.%N.%P.core" > > vfs.zfs.txg.timeout=5 > vfs.zfs.l2arc_headroom=16 > vfs.zfs.l2arc_write_max=**26843546 > vfs.zfs.l2arc_write_boost=**26843546 > vfs.zfs.vdev.min_pending=4 > vfs.zfs.vdev.max_pending=64 > vfs.zfs.l2arc_noprefetch=1 > vfs.zfs.dedup.prefetch=0 > vfs.zfs.prefetch_disable=1 > > I will use this server as a mysql-5.6 dbserver so I have a root partition > using a hw raid1 and a /DATAZFS partition, follows configuration: > > root# mfiutil show drives > mfi0 Physical Drives: > 0 ( 279G) ONLINE SAS E1:S0 > 1 ( 279G) ONLINE SAS E1:S1 > 2 ( 558G) ONLINE SAS E1:S2 > 3 ( 558G) ONLINE SAS E1:S3 > 4 ( 558G) ONLINE SAS E1:S4 > 5 ( 558G) ONLINE SAS E1:S5 > 6 ( 119G) ONLINE SATA E1:S6 > 7 ( 119G) ONLINE SATA E1:S7 > root# mfiutil show volumes > mfi0 Volumes: > Id Size Level Stripe State Cache Name > mfid0 ( 278G) RAID-1 64k OPTIMAL Disabled > mfid1 ( 118G) RAID-0 64k OPTIMAL Disabled > mfid2 ( 118G) RAID-0 64k OPTIMAL Disabled > mfid3 ( 1116G) RAID-1 64k OPTIMAL Disabled > > I have a /DATAZFS partition, too: > root# zfs get all DATAZFS > NAME PROPERTY VALUE SOURCE > DATAZFS type filesystem - > DATAZFS creation Thu Mar 21 10:45 2013 - > DATAZFS used 54.2G - > DATAZFS available 1.02T - > DATAZFS referenced 54.2G - > DATAZFS compressratio 1.00x - > DATAZFS mounted yes - > DATAZFS quota none default > DATAZFS reservation none default > DATAZFS recordsize 16K local > DATAZFS mountpoint /DATAZFS default > DATAZFS sharenfs off default > DATAZFS checksum on default > DATAZFS compression off default > DATAZFS atime off local > DATAZFS devices on default > DATAZFS exec on default > DATAZFS setuid on default > DATAZFS readonly off default > DATAZFS jailed off default > DATAZFS snapdir hidden default > DATAZFS aclmode discard default > DATAZFS aclinherit restricted default > DATAZFS canmount on default > DATAZFS xattr off temporary > DATAZFS copies 1 default > DATAZFS version 5 - > DATAZFS utf8only off - > DATAZFS normalization none - > DATAZFS casesensitivity sensitive - > DATAZFS vscan off default > DATAZFS nbmand off default > DATAZFS sharesmb off default > DATAZFS refquota none default > DATAZFS refreservation none default > DATAZFS primarycache all default > DATAZFS secondarycache all default > DATAZFS usedbysnapshots 0 - > DATAZFS usedbydataset 54.2G - > DATAZFS usedbychildren 1.41M - > DATAZFS usedbyrefreservation 0 - > DATAZFS logbias latency default > DATAZFS dedup off local > DATAZFS mlslabel - > DATAZFS sync standard default > DATAZFS refcompressratio 1.00x - > DATAZFS written 54.2G - > > root# zpool status -v > pool: DATAZFS > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > DATAZFS ONLINE 0 0 0 > mfid3 ONLINE 0 0 0 > logs > mfid2 ONLINE 0 0 0 > cache > mfid1 ONLINE 0 0 0 > > errors: No known data errors > > root# mfiutil show firmware > mfi0 Firmware Package Version: 21.1.0-0007 > mfi0 Firmware Images: > Name Version Date Time Status > BIOS 5.31.01_4.12.05.00_0x05110000 6/20/2012 > 6/20/2012 > active > CTLR 4.00-0014 Aug 04 2011 12:49:17 active > PCLI 05.00-03:#%00008 Feb 17 2011 14:03:12 active > APP 3.130.05-1796 Jul 27 2012 15:58:31 active > NVDT 2.1108.03-0094 Jun 22 2012 12:40:55 active > BTBL 2.03.00.00-0004 Jul 27 2012 16:39:29 active > BOOT 06.253.57.219 9/9/2010 15:32:25 active > root# > > Then, I installed CentOS on the same hardware, same version of mysql > (5.6.10-enterprise and 5.6.10-GA), and I tried an OLTP test (included in > sysbench-0.5) executing 2000000 queries on 10000000 rows, with these > results (transactions per seconds): > > BSD 9.1 amd64 ZFS (without tweaks): > 1 thread: 2780 > 2 threads: 4371 > 4 threads: 6876 > 8 threads: 8202 > 12 threads: 8077 > 16 threads: 7780 > 24 threads: 7563 > 32 threads: 7632 > 48 threads: 7960 > 64 threads: 8062 > > BSD 9.1 amd64 ZFS (with tweaks): > 1 thread: 2589 > 2 threads: 4679 > 4 threads: 6438 > 8 threads: 7073 > 12 threads: 7121 > 16 threads: 7227 > 24 threads: 7132 > 32 threads: 7273 > 48 threads: 7623 > 64 threads: 7672 > > CentOS 6.3 (ext4, without tweaks): > 1 thread: 4163 > 2 threads: 7653 > 4 threads: 10905 > 8 threads: 12511 > 12 threads: 13556 > 16 threads: 14832 > 24 threads: 16270 > 32 threads: 16733 > 48 threads: 16925 > 64 threads: 16895 > > > Well, any of you could ask "why zfs, please try ufs that is better": you > are right, but changing ZFS with UFS doesn't change so much: > http://lists.freebsd.org/**pipermail/freebsd-fs/2013-**March/016747.html > > So I hope I'm missing something because the performances of a standard > CentOS installation on the same hardware performs much better than a 9.1 > amd64 installation (I'm using the standard ULE scheduler included in the > 9.1 release). > > Could anyone help me? > > Thanks, > d. > ______________________________**_________________ > freebsd-performance@freebsd.**org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-**performance > To unsubscribe, send any mail to "freebsd-performance-** > unsubscribe@freebsd.org " > From owner-freebsd-performance@FreeBSD.ORG Fri Mar 22 16:15:12 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 028B17D0 for ; Fri, 22 Mar 2013 16:15:12 +0000 (UTC) (envelope-from davide.damico@contactlab.com) Received: from mail2.shared.smtp.contactlab.it (mail2.shared.smtp.contactlab.it [93.94.37.7]) by mx1.freebsd.org (Postfix) with ESMTP id 6ECB1A3B for ; Fri, 22 Mar 2013 16:15:10 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; d=contactlab.it; s=clab1; c=relaxed/relaxed; q=dns/txt; i=@contactlab.it; t=1363968909; h=From:Subject:Date:To:MIME-Version:Content-Type; bh=Cd9SJ64SI+x0aTB2YefwG1Y40MtCxLvzBK75NTueN3c=; b=a/bnww8aQSXKWp1YqjHHZqJCdWvue7fQNh/prQQJW8m0fvZ6ZeYC7NlYKhI+z9Bh IRKLx9Y2e+ldQ+Bv5h1plrC4l4SV8GucfJujyAAXhdMpbBAKfCvuju8tuSn4UcmM 1zPLQXXgsViLlIllvJnqzBRsWRHY/3aKFdqqCsw70Yw=; Received: from [213.92.90.12] ([213.92.90.12:19101] helo=mail3.tomato.it) by t.contactlab.it (envelope-from ) (ecelerity 3.5.1.37854 r(Momo-dev:3.5.1.0)) with ESMTP id 36/A3-24145-D838C415; Fri, 22 Mar 2013 17:15:09 +0100 Received: from mx3-master.housing.tomato.lan ([172.16.7.55]) by mail3.tomato.it with smtp (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UJ4cX-000HMF-2V for freebsd-performance@freebsd.org; Fri, 22 Mar 2013 17:15:09 +0100 Received: (qmail 66692 invoked by uid 89); 22 Mar 2013 16:15:08 -0000 Received: from localhost (HELO ?172.16.17.33?) (127.0.0.1) by mx3-master.housing.tomato.lan with SMTP; 22 Mar 2013 16:15:08 -0000 User-Agent: K-9 Mail for Android In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Subject: Re: FreeBSD 9.1 vs CentOS 6.3 From: Davide D'Amico Date: Fri, 22 Mar 2013 17:15:08 +0100 To: =?UTF-8?B?0JXQstCz0LXQvdC40Lkg0KXQvtGA0LrQuNC9?= Message-ID: Cc: freebsd-performance@freebsd.org X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2013 16:15:12 -0000 Well, the I/O isn't the bottleneck (if you follow the link to freebsd-fs, you'll see iostats values) but it seems something related to cpu/scheduler or something else. Now I am trying vmware 5 on the same server and a vm with centos6: the vm outperforms freebsd with every concurrency from 1 to 48 threads in sysbench oltp tests. Thanks, d. "Евгений Хоркин" ha scritto: >Hi Davide! > >Are you sure that disk is the bottleneck in your test? >Does systat -vm 1 show 100% busy for disk ? > >Evgeny. > > >2013/3/22 Davide D'Amico > >> Hi, I'm doing performance tests on a DELL R720, follows dmesg: >> >> Copyright (c) 1992-2012 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >1994 >> The Regents of the University of California. All rights >reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 9.1-RELEASE #0: Tue Mar 19 10:24:21 CET 2013 >> root:/usr/obj/usr/src/sys/R720 amd64 >> CPU: Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz (2500.05-MHz K8-class >CPU) >> Origin = "GenuineIntel" Id = 0x206d7 Family = 6 Model = 2d >Stepping = >> 7 >> Features=0xbfebfbff> APIC,SEP,MTRR,PGE,MCA,CMOV,**PAT,PSE36,CLFLUSH,DTS,ACPI,** >> MMX,FXSR,SSE,SSE2,SS,HTT,TM,**PBE> >> Features2=0x1fbee3ff> VMX,SMX,EST,TM2,SSSE3,CX16,**xTPR,PDCM,PCID,DCA,SSE4.1,** >> SSE4.2,x2APIC,POPCNT,TSCDLT,**AESNI,XSAVE,OSXSAVE,AVX> >> AMD Features=0x2c100800 >> AMD Features2=0x1 >> TSC: P-state invariant, performance statistics >> real memory = 34359738368 (32768 MB) >> avail memory = 33027436544 (31497 MB) >> Event timer "LAPIC" quality 600 >> ACPI APIC Table: >> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs >> FreeBSD/SMP: 2 package(s) x 2 core(s) x 2 SMT threads >> cpu0 (BSP): APIC ID: 0 >> cpu1 (AP): APIC ID: 1 >> cpu2 (AP): APIC ID: 10 >> cpu3 (AP): APIC ID: 11 >> cpu4 (AP): APIC ID: 32 >> cpu5 (AP): APIC ID: 33 >> cpu6 (AP): APIC ID: 42 >> cpu7 (AP): APIC ID: 43 >> ioapic1: Changing APIC ID to 1 >> ioapic2: Changing APIC ID to 2 >> ioapic0 irqs 0-23 on motherboard >> ioapic1 irqs 32-55 on motherboard >> ioapic2 irqs 64-87 on motherboard >> kbd1 at kbdmux0 >> ctl: CAM Target Layer loaded >> acpi0: on motherboard >> acpi0: Power Button (fixed) >> acpi0: reservation of 0, 2000 (3) failed >> cpu0: on acpi0 >> cpu1: on acpi0 >> cpu2: on acpi0 >> cpu3: on acpi0 >> cpu4: on acpi0 >> cpu5: on acpi0 >> cpu6: on acpi0 >> cpu7: on acpi0 >> atrtc0: port 0x70-0x7f irq 8 on acpi0 >> Event timer "RTC" frequency 32768 Hz quality 0 >> attimer0: port 0x40-0x5f irq 0 on acpi0 >> Timecounter "i8254" frequency 1193182 Hz quality 0 >> Event timer "i8254" frequency 1193182 Hz quality 100 >> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 >> hpet0: iomem 0xfed00000-0xfed003ff on >acpi0 >> Timecounter "HPET" frequency 14318180 Hz quality 950 >> Event timer "HPET" frequency 14318180 Hz quality 550 >> pcib0: port 0xcf8-0xcff on acpi0 >> pci0: on pcib0 >> pcib1: irq 53 at device 1.0 on pci0 >> pci1: on pcib1 >> igb0: port >> 0xfc80-0xfc9f mem 0xdcb00000-0xdcbfffff,**0xdcff0000-0xdcff3fff irq >38 at >> device 0.0 on pci1 >> igb0: Using MSIX interrupts with 9 vectors >> igb0: Ethernet address: bc:30:5b:ef:27:b8 >> [...] >> igb1: port >> 0xfca0-0xfcbf mem 0xdcc00000-0xdccfffff,**0xdcff4000-0xdcff7fff irq >37 at >> device 0.1 on pci1 >> igb1: Using MSIX interrupts with 9 vectors >> igb1: Ethernet address: bc:30:5b:ef:27:b9 >> [...] >> igb2: port >> 0xfcc0-0xfcdf mem 0xdcd00000-0xdcdfffff,**0xdcff8000-0xdcffbfff irq >36 at >> device 0.2 on pci1 >> igb2: Using MSIX interrupts with 9 vectors >> igb2: Ethernet address: bc:30:5b:ef:27:ba >> [...] >> igb3: port >> 0xfce0-0xfcff mem 0xdce00000-0xdcefffff,**0xdcffc000-0xdcffffff irq >34 at >> device 0.3 on pci1 >> igb3: Using MSIX interrupts with 9 vectors >> igb3: Ethernet address: bc:30:5b:ef:27:bb >> [...] >> pcib2: irq 53 at device 2.0 on pci0 >> pci3: on pcib2 >> pcib3: irq 53 at device 2.2 on pci0 >> pci2: on pcib3 >> mfi0: port 0xec00-0xecff mem >> 0xddffc000-0xddffffff,**0xddf80000-0xddfbffff irq 42 at device 0.0 on >pci2 >> mfi0: Using MSI >> mfi0: Megaraid SAS driver Ver 4.23 >> mfi0: MaxCmd = 3f0 MaxSgl = 46 state = b73c03f0 >> pcib4: irq 53 at device 3.0 on pci0 >> pci4: on pcib4 >> pcib5: irq 53 at device 3.2 on pci0 >> pci5: on pcib5 >> pci0: at device 5.0 (no driver attached) >> pci0: at device 5.2 (no driver attached) >> pcib6: irq 16 at device 17.0 on pci0 >> pci6: on pcib6 >> pci0: at device 22.0 (no driver attached) >> pci0: at device 22.1 (no driver attached) >> ehci0: mem 0xdf8fe000-0xdf8fe3ff >irq >> 23 at device 26.0 on pci0 >> usbus0: EHCI version 1.0 >> usbus0 on ehci0 >> pcib7: at device 28.0 on pci0 >> device_attach: pcib7 attach returned 6 >> pcib7: irq 19 at device 28.7 on pci0 >> pci8: on pcib7 >> pcib8: at device 0.0 on pci8 >> pci9: on pcib8 >> pcib9: at device 0.0 on pci9 >> pci10: on pcib9 >> pcib10: at device 0.0 on pci10 >> pci11: on pcib10 >> vgapci0: mem 0xd8000000-0xd8ffffff,** >> 0xdeffc000-0xdeffffff,**0xde000000-0xde7fffff irq 19 at device 0.0 on >> pci11 >> pcib11: at device 1.0 on pci9 >> pci12: on pcib11 >> ehci1: mem 0xdf8ff000-0xdf8ff3ff >irq >> 22 at device 29.0 on pci0 >> usbus1: EHCI version 1.0 >> usbus1 on ehci1 >> pcib12: at device 30.0 on pci0 >> pci13: on pcib12 >> isab0: at device 31.0 on pci0 >> isa0: on isab0 >> pcib13: on acpi0 >> pci64: on pcib13 >> pcib14: irq 85 at device 1.0 on pci64 >> pci65: on pcib14 >> pcib15: irq 85 at device 2.0 on pci64 >> pci66: on pcib15 >> pcib16: irq 85 at device 3.0 on pci64 >> pci67: on pcib16 >> pcib17: irq 85 at device 3.2 on pci64 >> pci68: on pcib17 >> pci64: at device 5.0 (no driver attached) >> pci64: at device 5.2 (no driver attached) >> pcib18: on acpi0 >> pci63: on pcib18 >> pci63: at device 8.0 (no driver attached) >> [...] >> pci63: at device 14.1 (no driver attached) >> [...] >> pci63: at device 19.1 (no driver attached) >> pci63: at device 19.4 (no driver attached) >> pci63: at device 19.5 (no driver attached) >> pci63: at device 19.6 (no driver attached) >> pcib19: on acpi0 >> pci127: on pcib19 >> [...] >> pci127: at device 14.1 (no driver attached) >> [...] >> pci127: at device 19.1 (no driver attached) >> pci127: at device 19.4 (no driver attached) >> pci127: at device 19.5 (no driver attached) >> pci127: at device 19.6 (no driver attached) >> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 >> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on >acpi0 >> orm0: at iomem 0xc0000-0xc7fff,0xec000-**0xeffff on >isa0 >> sc0: at flags 0x100 on isa0 >> sc0: VGA <16 virtual consoles, flags=0x300> >> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on >isa0 >> est0: on cpu0 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 226d00001e00 >> device_attach: est0 attach returned 6 >> p4tcc0: on cpu0 >> est1: on cpu1 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 226d00001e00 >> device_attach: est1 attach returned 6 >> p4tcc1: on cpu1 >> est2: on cpu2 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 226d00001e00 >> device_attach: est2 attach returned 6 >> p4tcc2: on cpu2 >> est3: on cpu3 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 224400001e00 >> device_attach: est3 attach returned 6 >> p4tcc3: on cpu3 >> est4: on cpu4 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 226d00001e00 >> device_attach: est4 attach returned 6 >> p4tcc4: on cpu4 >> est5: on cpu5 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 224400001e00 >> device_attach: est5 attach returned 6 >> p4tcc5: on cpu5 >> est6: on cpu6 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 224400001e00 >> device_attach: est6 attach returned 6 >> p4tcc6: on cpu6 >> est7: on cpu7 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 224400001e00 >> device_attach: est7 attach returned 6 >> p4tcc7: on cpu7 >> mfi0: 6040 (417120269s/0x0020/info) - Shutdown command received from >host >> mfi0: 6041 (boot + 4s/0x0020/info) - Firmware initialization started >(PCI >> ID 005b/1000/1f34/1028) >> mfi0: 6042 (boot + 4s/0x0020/info) - Firmware version 3.130.05-1796 >> mfi0: 6043 (boot + 5s/0x0008/info) - Battery Present >> mfi0: 6044 (boot + 5s/0x0020/info) - Package version 21.1.0-0007 >> mfi0: 6045 (boot + 5s/0x0020/info) - Board Revision A01 >> mfi0: 6046 (boot + 6s/0x0008/info) - Battery temperature is normal >> mfi0: 6047 (boot + 6s/0x0008/info) - Current capacity of the battery >is >> above threshold >> mfi0: 6048 (boot + 19s/0x0004/info) - Enclosure PD 20(c None/p1) >> communication restored >> mfi0: 6049 (boot + 19s/0x0002/info) - Inserted: Encl PD 20 >> mfi0: 6050 (boot + 19s/0x0002/info) - Inserted: PD 20(c None/p1) >Info: >> enclPd=20, scsiType=d, portMap=00, sasAddr=5e4ae020a26cb500,** >> 0000000000000000 >> mfi0: 6051 (boot + 19s/0x0002/info) - Inserted: PD 00(e0x20/s0) >> mfi0: 6052 (boot + 19s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: >> enclPd=20, scsiType=0, portMap=04, sasAddr=5000c50054982339,** >> 0000000000000000 >> mfi0: 6053 (boot + 19s/0x0002/info) - Inserted: PD 01(e0x20/s1) >> mfi0: 6054 (boot + 19s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info: >> enclPd=20, scsiType=0, portMap=05, sasAddr=5000c50054986c91,** >> 0000000000000000 >> mfi0: 6055 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2) >> mfi0: 6056 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info: >> enclPd=20, scsiType=0, portMap=06, sasAddr=5000c50053da8f75,** >> 0000000000000000 >> mfi0: 6057 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s3) >> mfi0: 6058 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s3) Info: >> enclPd=20, scsiType=0, portMap=07, sasAddr=5000c50053c57eb5,** >> 0000000000000000 >> mfi0: 6059 (boot + 19s/0x0002/info) - Inserted: PD 04(e0x20/s4) >> mfi0: 6060 (boot + 19s/0x0002/info) - Inserted: PD 04(e0x20/s4) Info: >> enclPd=20, scsiType=0, portMap=00, sasAddr=5000c50053d364c5,** >> 0000000000000000 >> mfi0: 6061 (boot + 19s/0x0002/info) - Inserted: PD 05(e0x20/s5) >> mfi0: 6062 (boot + 19s/0x0002/info) - Inserted: PD 05(e0x20/s5) Info: >> enclPd=20, scsiType=0, portMap=03, sasAddr=5000c50053da2b41,** >> 0000000000000000 >> mfi0: 6063 (boot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6) >> mfi0: 6064 (boot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6) Info: >> enclPd=20, scsiType=0, portMap=02, sasAddr=4433221102000000,** >> 0000000000000000 >> mfi0: 6065 (boot + 19s/0x0002/WARN) - PD 06(e0x20/s6) is not a >certified >> drive >> mfi0: 6066 (boot + 19s/0x0002/info) - Inserted: PD 07(e0x20/s7) >> mfi0: 6067 (boot + 19s/0x0002/info) - Inserted: PD 07(e0x20/s7) Info: >> enclPd=20, scsiType=0, portMap=01, sasAddr=4433221103000000,** >> 0000000000000000 >> mfi0: 6068 (boot + 19s/0x0002/WARN) - PD 07(e0x20/s7) is not a >certified >> drive >> mfi0: 6069 (417120312s/0x0020/info) - Time established as 03/20/13 >> 18:45:12; (32 seconds since power on) >> mfi0: 6070 (417120445s/0x0020/info) - Host driver is loaded and >operational >> Timecounters tick every 1.000 msec >> usbus0: 480Mbps High Speed USB v2.0 >> usbus1: 480Mbps High Speed USB v2.0 >> mfid0 on mfi0 >> mfid0: 285568MB (584843264 sectors) RAID volume 'BASE' is optimal >> mfid1 on mfi0 >> mfid1: 121472MB (248774656 sectors) RAID volume 'L2ARC0' is optimal >> mfid2 on mfi0 >> mfid2: 121472MB (248774656 sectors) RAID volume 'ZIL0' is optimal >> mfid3 on mfi0 >> mfid3: 1143552MB (2341994496 sectors) RAID volume 'DATASTRIPE1' is >optimal >> mfid4 on mfi0 >> mfid4: 1143552MB (2341994496 sectors) RAID volume 'DATASTRIPE2' is >optimal >> ugen0.1: at usbus0 >> uhub0: on >usbus0 >> ugen1.1: at usbus1 >> uhub1: on >usbus1 >> uhub0: 2 ports with 2 removable, self powered >> uhub1: 2 ports with 2 removable, self powered >> ugen0.2: at usbus0 >> uhub2: 2> on >> usbus0 >> ugen1.2: at usbus1 >> uhub3: 2> on >> usbus1 >> uhub2: 6 ports with 6 removable, self powered >> uhub3: 8 ports with 8 removable, self powered >> ugen0.3: at usbus0 >> uhub4: addr 3> >> on usbus0 >> SMP: AP CPU #1 Launched! >> SMP: AP CPU #7 Launched! >> SMP: AP CPU #3 Launched! >> SMP: AP CPU #6 Launched! >> SMP: AP CPU #2 Launched! >> SMP: AP CPU #5 Launched! >> SMP: AP CPU #4 Launched! >> Timecounter "TSC-low" frequency 9765827 Hz quality 1000 >> Root mount waiting for: usbus0 >> uhub4: 6 ports with 6 removable, self powered >> ugen0.4: at usbus0 >> ukbd0: on usbus0 >> kbd0 at ukbd0 >> Root mount waiting for: usbus0 >> Root mount waiting for: usbus0 >> ugen0.5: at usbus0 >> umass0: on usbus0 >> Trying to mount root from ufs:/dev/mfid0p3 [rw]... >> cd0 at umass-sim0 bus 0 scbus1 target 0 lun 0 >> cd0: Removable CD-ROM SCSI-0 device >> cd0: 40.000MB/s transfers >> cd0: Attempt to query device size failed: NOT READY, Medium not >present >> da0 at umass-sim0 bus 0 scbus1 target 0 lun 1 >> da0: Removable Direct Access SCSI-0 >device >> da0: 40.000MB/s transfers >> da0: Attempt to query device size failed: NOT READY, Medium not >present >> ZFS filesystem version 5 >> ZFS storage pool version 28 >> ums0: on usbus0 >> ums0: 3 buttons and [Z] coordinates ID=0 >> ums1: on usbus0 >> ums1: 3 buttons and [XYZ] coordinates ID=0 >> igb0: link state changed to UP >> >> With this loader.conf: >> vm.kmem_size_max="64G" >> vm.kmem_size="48G" >> vfs.zfs.arc_min="4096M" >> vfs.zfs.arc_max="15872M" >> vfs.zfs.txg.synctime="1" >> vfs.zfs.txg.timeout="5" >> vfs.zfs.write_limit_override="**0" >> vfs.zfs.txg.write_limit_**override="0" >> vfs.zfs.zil_disable="0" >> >> And with this sysctl.conf: >> security.bsd.see_other_uids=0 >> net.inet.tcp.blackhole=2 >> net.inet.udp.blackhole=1 >> net.inet.tcp.sendspace=65536 >> net.inet.tcp.recvspace=65536 >> net.link.ether.inet.log_arp_**wrong_iface=0 >> kern.ipc.somaxconn=32768 >> kern.threads.max_threads_per_**proc=16384 >> kern.maxfiles=262144 >> kern.maxfilesperproc=131072 >> kern.ipc.nmbclusters=65536 >> kern.corefile="/var/coredumps/**%U.%N.%P.core" >> >> vfs.zfs.txg.timeout=5 >> vfs.zfs.l2arc_headroom=16 >> vfs.zfs.l2arc_write_max=**26843546 >> vfs.zfs.l2arc_write_boost=**26843546 >> vfs.zfs.vdev.min_pending=4 >> vfs.zfs.vdev.max_pending=64 >> vfs.zfs.l2arc_noprefetch=1 >> vfs.zfs.dedup.prefetch=0 >> vfs.zfs.prefetch_disable=1 >> >> I will use this server as a mysql-5.6 dbserver so I have a root >partition >> using a hw raid1 and a /DATAZFS partition, follows configuration: >> >> root# mfiutil show drives >> mfi0 Physical Drives: >> 0 ( 279G) ONLINE SAS E1:S0 >> 1 ( 279G) ONLINE SAS E1:S1 >> 2 ( 558G) ONLINE SAS E1:S2 >> 3 ( 558G) ONLINE SAS E1:S3 >> 4 ( 558G) ONLINE SAS E1:S4 >> 5 ( 558G) ONLINE SAS E1:S5 >> 6 ( 119G) ONLINE SATA E1:S6 >> 7 ( 119G) ONLINE SATA E1:S7 >> root# mfiutil show volumes >> mfi0 Volumes: >> Id Size Level Stripe State Cache Name >> mfid0 ( 278G) RAID-1 64k OPTIMAL Disabled >> mfid1 ( 118G) RAID-0 64k OPTIMAL Disabled >> mfid2 ( 118G) RAID-0 64k OPTIMAL Disabled >> mfid3 ( 1116G) RAID-1 64k OPTIMAL Disabled >> >> I have a /DATAZFS partition, too: >> root# zfs get all DATAZFS >> NAME PROPERTY VALUE SOURCE >> DATAZFS type filesystem - >> DATAZFS creation Thu Mar 21 10:45 2013 - >> DATAZFS used 54.2G - >> DATAZFS available 1.02T - >> DATAZFS referenced 54.2G - >> DATAZFS compressratio 1.00x - >> DATAZFS mounted yes - >> DATAZFS quota none default >> DATAZFS reservation none default >> DATAZFS recordsize 16K local >> DATAZFS mountpoint /DATAZFS default >> DATAZFS sharenfs off default >> DATAZFS checksum on default >> DATAZFS compression off default >> DATAZFS atime off local >> DATAZFS devices on default >> DATAZFS exec on default >> DATAZFS setuid on default >> DATAZFS readonly off default >> DATAZFS jailed off default >> DATAZFS snapdir hidden default >> DATAZFS aclmode discard default >> DATAZFS aclinherit restricted default >> DATAZFS canmount on default >> DATAZFS xattr off temporary >> DATAZFS copies 1 default >> DATAZFS version 5 - >> DATAZFS utf8only off - >> DATAZFS normalization none - >> DATAZFS casesensitivity sensitive - >> DATAZFS vscan off default >> DATAZFS nbmand off default >> DATAZFS sharesmb off default >> DATAZFS refquota none default >> DATAZFS refreservation none default >> DATAZFS primarycache all default >> DATAZFS secondarycache all default >> DATAZFS usedbysnapshots 0 - >> DATAZFS usedbydataset 54.2G - >> DATAZFS usedbychildren 1.41M - >> DATAZFS usedbyrefreservation 0 - >> DATAZFS logbias latency default >> DATAZFS dedup off local >> DATAZFS mlslabel - >> DATAZFS sync standard default >> DATAZFS refcompressratio 1.00x - >> DATAZFS written 54.2G - >> >> root# zpool status -v >> pool: DATAZFS >> state: ONLINE >> scan: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> DATAZFS ONLINE 0 0 0 >> mfid3 ONLINE 0 0 0 >> logs >> mfid2 ONLINE 0 0 0 >> cache >> mfid1 ONLINE 0 0 0 >> >> errors: No known data errors >> >> root# mfiutil show firmware >> mfi0 Firmware Package Version: 21.1.0-0007 >> mfi0 Firmware Images: >> Name Version Date Time Status >> BIOS 5.31.01_4.12.05.00_0x05110000 6/20/2012 >> 6/20/2012 >> active >> CTLR 4.00-0014 Aug 04 2011 12:49:17 active >> PCLI 05.00-03:#%00008 Feb 17 2011 14:03:12 active >> APP 3.130.05-1796 Jul 27 2012 15:58:31 active >> NVDT 2.1108.03-0094 Jun 22 2012 12:40:55 active >> BTBL 2.03.00.00-0004 Jul 27 2012 16:39:29 active >> BOOT 06.253.57.219 9/9/2010 15:32:25 active >> root# >> >> Then, I installed CentOS on the same hardware, same version of mysql >> (5.6.10-enterprise and 5.6.10-GA), and I tried an OLTP test (included >in >> sysbench-0.5) executing 2000000 queries on 10000000 rows, with these >> results (transactions per seconds): >> >> BSD 9.1 amd64 ZFS (without tweaks): >> 1 thread: 2780 >> 2 threads: 4371 >> 4 threads: 6876 >> 8 threads: 8202 >> 12 threads: 8077 >> 16 threads: 7780 >> 24 threads: 7563 >> 32 threads: 7632 >> 48 threads: 7960 >> 64 threads: 8062 >> >> BSD 9.1 amd64 ZFS (with tweaks): >> 1 thread: 2589 >> 2 threads: 4679 >> 4 threads: 6438 >> 8 threads: 7073 >> 12 threads: 7121 >> 16 threads: 7227 >> 24 threads: 7132 >> 32 threads: 7273 >> 48 threads: 7623 >> 64 threads: 7672 >> >> CentOS 6.3 (ext4, without tweaks): >> 1 thread: 4163 >> 2 threads: 7653 >> 4 threads: 10905 >> 8 threads: 12511 >> 12 threads: 13556 >> 16 threads: 14832 >> 24 threads: 16270 >> 32 threads: 16733 >> 48 threads: 16925 >> 64 threads: 16895 >> >> >> Well, any of you could ask "why zfs, please try ufs that is better": >you >> are right, but changing ZFS with UFS doesn't change so much: >> >http://lists.freebsd.org/**pipermail/freebsd-fs/2013-**March/016747.html >> >> So I hope I'm missing something because the performances of a >standard >> CentOS installation on the same hardware performs much better than a >9.1 >> amd64 installation (I'm using the standard ULE scheduler included in >the >> 9.1 release). >> >> Could anyone help me? >> >> Thanks, >> d. >> ______________________________**_________________ >> freebsd-performance@freebsd.**org >mailing list >> >http://lists.freebsd.org/**mailman/listinfo/freebsd-**performance >> To unsubscribe, send any mail to "freebsd-performance-** >> unsubscribe@freebsd.org >" >> -- d. From owner-freebsd-performance@FreeBSD.ORG Fri Mar 22 22:07:24 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 44C99E05 for ; Fri, 22 Mar 2013 22:07:24 +0000 (UTC) (envelope-from pheriko.support@gmail.com) Received: from mail-vb0-x234.google.com (mail-vb0-x234.google.com [IPv6:2607:f8b0:400c:c02::234]) by mx1.freebsd.org (Postfix) with ESMTP id 0A3D4EFD for ; Fri, 22 Mar 2013 22:07:23 +0000 (UTC) Received: by mail-vb0-f52.google.com with SMTP id fa15so2823285vbb.39 for ; Fri, 22 Mar 2013 15:07:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:content-type:content-transfer-encoding; bh=3tjEwrSthVm89VDRtUN1t0i0cTgAujT9FahNMOdKdnY=; b=ImqGna1NQcvYsiOGpIAW3h+YKpDH+v7kC+uVWhlAjDZyD6EY2ZjTDskYkU1Z9APK6b HQ5PMxXp1U1yUVNQXRfkSoT4Xx4tLevzCWB26cgNBmO84wudGDcsvu68RrPSECLNITJY 194T1Kj1Dy/cfcRLrn4Y/WfB40eKjcMdqRnwX709RmLVJfscbjpRRw6mrl4JjXB7ecKq IiZ2F9ZAZwEZ6BhUZBm7uzH8yoTxwA6SIc0QMQLz4m9nxtXK/oE42agMLI+H1kyzNrjV N2of55blOk2HT+i34qlY/vINRLwQgUMT0ZPbuUzN/DzcYoJCxQVAdly7jNjQI4lCFx+0 T/yQ== MIME-Version: 1.0 X-Received: by 10.52.28.101 with SMTP id a5mr3781248vdh.92.1363990043502; Fri, 22 Mar 2013 15:07:23 -0700 (PDT) Received: by 10.58.39.231 with HTTP; Fri, 22 Mar 2013 15:07:23 -0700 (PDT) In-Reply-To: References: Date: Fri, 22 Mar 2013 15:07:23 -0700 Message-ID: Subject: Re: FreeBSD 9.1 vs CentOS 6.3 From: Periko Support To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Mar 2013 22:07:24 -0000 On Fri, Mar 22, 2013 at 9:15 AM, Davide D'Amico wrote: > Well, the I/O isn't the bottleneck (if you follow the link to freebsd-fs,= you'll see iostats values) but it seems something related to cpu/scheduler= or something else. > Now I am trying vmware 5 on the same server and a vm with centos6: the vm= outperforms freebsd with every concurrency from 1 to 48 threads in sysbenc= h oltp tests. > > Thanks, > d. > > "=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9 =D0=A5=D0=BE=D1=80=D0=BA=D0= =B8=D0=BD" ha scritto: > >>Hi Davide! >> >>Are you sure that disk is the bottleneck in your test? >>Does systat -vm 1 show 100% busy for disk ? >> >>Evgeny. >> >> >>2013/3/22 Davide D'Amico >> >>> Hi, I'm doing performance tests on a DELL R720, follows dmesg: >>> >>> Copyright (c) 1992-2012 The FreeBSD Project. >>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >>1994 >>> The Regents of the University of California. All rights >>reserved. >>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>> FreeBSD 9.1-RELEASE #0: Tue Mar 19 10:24:21 CET 2013 >>> root:/usr/obj/usr/src/sys/R720 amd64 >>> CPU: Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz (2500.05-MHz K8-class >>CPU) >>> Origin =3D "GenuineIntel" Id =3D 0x206d7 Family =3D 6 Model =3D 2d >>Stepping =3D >>> 7 >>> Features=3D0xbfebfbff>> APIC,SEP,MTRR,PGE,MCA,CMOV,**PAT,PSE36,CLFLUSH,DTS,ACPI,** >>> MMX,FXSR,SSE,SSE2,SS,HTT,TM,**PBE> >>> Features2=3D0x1fbee3ff>> VMX,SMX,EST,TM2,SSSE3,CX16,**xTPR,PDCM,PCID,DCA,SSE4.1,** >>> SSE4.2,x2APIC,POPCNT,TSCDLT,**AESNI,XSAVE,OSXSAVE,AVX> >>> AMD Features=3D0x2c100800 >>> AMD Features2=3D0x1 >>> TSC: P-state invariant, performance statistics >>> real memory =3D 34359738368 (32768 MB) >>> avail memory =3D 33027436544 (31497 MB) >>> Event timer "LAPIC" quality 600 >>> ACPI APIC Table: >>> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs >>> FreeBSD/SMP: 2 package(s) x 2 core(s) x 2 SMT threads >>> cpu0 (BSP): APIC ID: 0 >>> cpu1 (AP): APIC ID: 1 >>> cpu2 (AP): APIC ID: 10 >>> cpu3 (AP): APIC ID: 11 >>> cpu4 (AP): APIC ID: 32 >>> cpu5 (AP): APIC ID: 33 >>> cpu6 (AP): APIC ID: 42 >>> cpu7 (AP): APIC ID: 43 >>> ioapic1: Changing APIC ID to 1 >>> ioapic2: Changing APIC ID to 2 >>> ioapic0 irqs 0-23 on motherboard >>> ioapic1 irqs 32-55 on motherboard >>> ioapic2 irqs 64-87 on motherboard >>> kbd1 at kbdmux0 >>> ctl: CAM Target Layer loaded >>> acpi0: on motherboard >>> acpi0: Power Button (fixed) >>> acpi0: reservation of 0, 2000 (3) failed >>> cpu0: on acpi0 >>> cpu1: on acpi0 >>> cpu2: on acpi0 >>> cpu3: on acpi0 >>> cpu4: on acpi0 >>> cpu5: on acpi0 >>> cpu6: on acpi0 >>> cpu7: on acpi0 >>> atrtc0: port 0x70-0x7f irq 8 on acpi0 >>> Event timer "RTC" frequency 32768 Hz quality 0 >>> attimer0: port 0x40-0x5f irq 0 on acpi0 >>> Timecounter "i8254" frequency 1193182 Hz quality 0 >>> Event timer "i8254" frequency 1193182 Hz quality 100 >>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 >>> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 >>> hpet0: iomem 0xfed00000-0xfed003ff on >>acpi0 >>> Timecounter "HPET" frequency 14318180 Hz quality 950 >>> Event timer "HPET" frequency 14318180 Hz quality 550 >>> pcib0: port 0xcf8-0xcff on acpi0 >>> pci0: on pcib0 >>> pcib1: irq 53 at device 1.0 on pci0 >>> pci1: on pcib1 >>> igb0: port >>> 0xfc80-0xfc9f mem 0xdcb00000-0xdcbfffff,**0xdcff0000-0xdcff3fff irq >>38 at >>> device 0.0 on pci1 >>> igb0: Using MSIX interrupts with 9 vectors >>> igb0: Ethernet address: bc:30:5b:ef:27:b8 >>> [...] >>> igb1: port >>> 0xfca0-0xfcbf mem 0xdcc00000-0xdccfffff,**0xdcff4000-0xdcff7fff irq >>37 at >>> device 0.1 on pci1 >>> igb1: Using MSIX interrupts with 9 vectors >>> igb1: Ethernet address: bc:30:5b:ef:27:b9 >>> [...] >>> igb2: port >>> 0xfcc0-0xfcdf mem 0xdcd00000-0xdcdfffff,**0xdcff8000-0xdcffbfff irq >>36 at >>> device 0.2 on pci1 >>> igb2: Using MSIX interrupts with 9 vectors >>> igb2: Ethernet address: bc:30:5b:ef:27:ba >>> [...] >>> igb3: port >>> 0xfce0-0xfcff mem 0xdce00000-0xdcefffff,**0xdcffc000-0xdcffffff irq >>34 at >>> device 0.3 on pci1 >>> igb3: Using MSIX interrupts with 9 vectors >>> igb3: Ethernet address: bc:30:5b:ef:27:bb >>> [...] >>> pcib2: irq 53 at device 2.0 on pci0 >>> pci3: on pcib2 >>> pcib3: irq 53 at device 2.2 on pci0 >>> pci2: on pcib3 >>> mfi0: port 0xec00-0xecff mem >>> 0xddffc000-0xddffffff,**0xddf80000-0xddfbffff irq 42 at device 0.0 on >>pci2 >>> mfi0: Using MSI >>> mfi0: Megaraid SAS driver Ver 4.23 >>> mfi0: MaxCmd =3D 3f0 MaxSgl =3D 46 state =3D b73c03f0 >>> pcib4: irq 53 at device 3.0 on pci0 >>> pci4: on pcib4 >>> pcib5: irq 53 at device 3.2 on pci0 >>> pci5: on pcib5 >>> pci0: at device 5.0 (no driver attached) >>> pci0: at device 5.2 (no driver attached) >>> pcib6: irq 16 at device 17.0 on pci0 >>> pci6: on pcib6 >>> pci0: at device 22.0 (no driver attached) >>> pci0: at device 22.1 (no driver attached) >>> ehci0: mem 0xdf8fe000-0xdf8fe3ff >>irq >>> 23 at device 26.0 on pci0 >>> usbus0: EHCI version 1.0 >>> usbus0 on ehci0 >>> pcib7: at device 28.0 on pci0 >>> device_attach: pcib7 attach returned 6 >>> pcib7: irq 19 at device 28.7 on pci0 >>> pci8: on pcib7 >>> pcib8: at device 0.0 on pci8 >>> pci9: on pcib8 >>> pcib9: at device 0.0 on pci9 >>> pci10: on pcib9 >>> pcib10: at device 0.0 on pci10 >>> pci11: on pcib10 >>> vgapci0: mem 0xd8000000-0xd8ffffff,** >>> 0xdeffc000-0xdeffffff,**0xde000000-0xde7fffff irq 19 at device 0.0 on >>> pci11 >>> pcib11: at device 1.0 on pci9 >>> pci12: on pcib11 >>> ehci1: mem 0xdf8ff000-0xdf8ff3ff >>irq >>> 22 at device 29.0 on pci0 >>> usbus1: EHCI version 1.0 >>> usbus1 on ehci1 >>> pcib12: at device 30.0 on pci0 >>> pci13: on pcib12 >>> isab0: at device 31.0 on pci0 >>> isa0: on isab0 >>> pcib13: on acpi0 >>> pci64: on pcib13 >>> pcib14: irq 85 at device 1.0 on pci64 >>> pci65: on pcib14 >>> pcib15: irq 85 at device 2.0 on pci64 >>> pci66: on pcib15 >>> pcib16: irq 85 at device 3.0 on pci64 >>> pci67: on pcib16 >>> pcib17: irq 85 at device 3.2 on pci64 >>> pci68: on pcib17 >>> pci64: at device 5.0 (no driver attached) >>> pci64: at device 5.2 (no driver attached) >>> pcib18: on acpi0 >>> pci63: on pcib18 >>> pci63: at device 8.0 (no driver attached) >>> [...] >>> pci63: at device 14.1 (no driver attached) >>> [...] >>> pci63: at device 19.1 (no driver attached) >>> pci63: at device 19.4 (no driver attached) >>> pci63: at device 19.5 (no driver attached) >>> pci63: at device 19.6 (no driver attached) >>> pcib19: on acpi0 >>> pci127: on pcib19 >>> [...] >>> pci127: at device 14.1 (no driver attached) >>> [...] >>> pci127: at device 19.1 (no driver attached) >>> pci127: at device 19.4 (no driver attached) >>> pci127: at device 19.5 (no driver attached) >>> pci127: at device 19.6 (no driver attached) >>> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 >>> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on >>acpi0 >>> orm0: at iomem 0xc0000-0xc7fff,0xec000-**0xeffff on >>isa0 >>> sc0: at flags 0x100 on isa0 >>> sc0: VGA <16 virtual consoles, flags=3D0x300> >>> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on >>isa0 >>> est0: on cpu0 >>> est: CPU supports Enhanced Speedstep, but is not recognized. >>> est: cpu_vendor GenuineIntel, msr 226d00001e00 >>> device_attach: est0 attach returned 6 >>> p4tcc0: on cpu0 >>> est1: on cpu1 >>> est: CPU supports Enhanced Speedstep, but is not recognized. >>> est: cpu_vendor GenuineIntel, msr 226d00001e00 >>> device_attach: est1 attach returned 6 >>> p4tcc1: on cpu1 >>> est2: on cpu2 >>> est: CPU supports Enhanced Speedstep, but is not recognized. >>> est: cpu_vendor GenuineIntel, msr 226d00001e00 >>> device_attach: est2 attach returned 6 >>> p4tcc2: on cpu2 >>> est3: on cpu3 >>> est: CPU supports Enhanced Speedstep, but is not recognized. >>> est: cpu_vendor GenuineIntel, msr 224400001e00 >>> device_attach: est3 attach returned 6 >>> p4tcc3: on cpu3 >>> est4: on cpu4 >>> est: CPU supports Enhanced Speedstep, but is not recognized. >>> est: cpu_vendor GenuineIntel, msr 226d00001e00 >>> device_attach: est4 attach returned 6 >>> p4tcc4: on cpu4 >>> est5: on cpu5 >>> est: CPU supports Enhanced Speedstep, but is not recognized. >>> est: cpu_vendor GenuineIntel, msr 224400001e00 >>> device_attach: est5 attach returned 6 >>> p4tcc5: on cpu5 >>> est6: on cpu6 >>> est: CPU supports Enhanced Speedstep, but is not recognized. >>> est: cpu_vendor GenuineIntel, msr 224400001e00 >>> device_attach: est6 attach returned 6 >>> p4tcc6: on cpu6 >>> est7: on cpu7 >>> est: CPU supports Enhanced Speedstep, but is not recognized. >>> est: cpu_vendor GenuineIntel, msr 224400001e00 >>> device_attach: est7 attach returned 6 >>> p4tcc7: on cpu7 >>> mfi0: 6040 (417120269s/0x0020/info) - Shutdown command received from >>host >>> mfi0: 6041 (boot + 4s/0x0020/info) - Firmware initialization started >>(PCI >>> ID 005b/1000/1f34/1028) >>> mfi0: 6042 (boot + 4s/0x0020/info) - Firmware version 3.130.05-1796 >>> mfi0: 6043 (boot + 5s/0x0008/info) - Battery Present >>> mfi0: 6044 (boot + 5s/0x0020/info) - Package version 21.1.0-0007 >>> mfi0: 6045 (boot + 5s/0x0020/info) - Board Revision A01 >>> mfi0: 6046 (boot + 6s/0x0008/info) - Battery temperature is normal >>> mfi0: 6047 (boot + 6s/0x0008/info) - Current capacity of the battery >>is >>> above threshold >>> mfi0: 6048 (boot + 19s/0x0004/info) - Enclosure PD 20(c None/p1) >>> communication restored >>> mfi0: 6049 (boot + 19s/0x0002/info) - Inserted: Encl PD 20 >>> mfi0: 6050 (boot + 19s/0x0002/info) - Inserted: PD 20(c None/p1) >>Info: >>> enclPd=3D20, scsiType=3Dd, portMap=3D00, sasAddr=3D5e4ae020a26cb500,** >>> 0000000000000000 >>> mfi0: 6051 (boot + 19s/0x0002/info) - Inserted: PD 00(e0x20/s0) >>> mfi0: 6052 (boot + 19s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: >>> enclPd=3D20, scsiType=3D0, portMap=3D04, sasAddr=3D5000c50054982339,** >>> 0000000000000000 >>> mfi0: 6053 (boot + 19s/0x0002/info) - Inserted: PD 01(e0x20/s1) >>> mfi0: 6054 (boot + 19s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info: >>> enclPd=3D20, scsiType=3D0, portMap=3D05, sasAddr=3D5000c50054986c91,** >>> 0000000000000000 >>> mfi0: 6055 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2) >>> mfi0: 6056 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info: >>> enclPd=3D20, scsiType=3D0, portMap=3D06, sasAddr=3D5000c50053da8f75,** >>> 0000000000000000 >>> mfi0: 6057 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s3) >>> mfi0: 6058 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s3) Info: >>> enclPd=3D20, scsiType=3D0, portMap=3D07, sasAddr=3D5000c50053c57eb5,** >>> 0000000000000000 >>> mfi0: 6059 (boot + 19s/0x0002/info) - Inserted: PD 04(e0x20/s4) >>> mfi0: 6060 (boot + 19s/0x0002/info) - Inserted: PD 04(e0x20/s4) Info: >>> enclPd=3D20, scsiType=3D0, portMap=3D00, sasAddr=3D5000c50053d364c5,** >>> 0000000000000000 >>> mfi0: 6061 (boot + 19s/0x0002/info) - Inserted: PD 05(e0x20/s5) >>> mfi0: 6062 (boot + 19s/0x0002/info) - Inserted: PD 05(e0x20/s5) Info: >>> enclPd=3D20, scsiType=3D0, portMap=3D03, sasAddr=3D5000c50053da2b41,** >>> 0000000000000000 >>> mfi0: 6063 (boot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6) >>> mfi0: 6064 (boot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6) Info: >>> enclPd=3D20, scsiType=3D0, portMap=3D02, sasAddr=3D4433221102000000,** >>> 0000000000000000 >>> mfi0: 6065 (boot + 19s/0x0002/WARN) - PD 06(e0x20/s6) is not a >>certified >>> drive >>> mfi0: 6066 (boot + 19s/0x0002/info) - Inserted: PD 07(e0x20/s7) >>> mfi0: 6067 (boot + 19s/0x0002/info) - Inserted: PD 07(e0x20/s7) Info: >>> enclPd=3D20, scsiType=3D0, portMap=3D01, sasAddr=3D4433221103000000,** >>> 0000000000000000 >>> mfi0: 6068 (boot + 19s/0x0002/WARN) - PD 07(e0x20/s7) is not a >>certified >>> drive >>> mfi0: 6069 (417120312s/0x0020/info) - Time established as 03/20/13 >>> 18:45:12; (32 seconds since power on) >>> mfi0: 6070 (417120445s/0x0020/info) - Host driver is loaded and >>operational >>> Timecounters tick every 1.000 msec >>> usbus0: 480Mbps High Speed USB v2.0 >>> usbus1: 480Mbps High Speed USB v2.0 >>> mfid0 on mfi0 >>> mfid0: 285568MB (584843264 sectors) RAID volume 'BASE' is optimal >>> mfid1 on mfi0 >>> mfid1: 121472MB (248774656 sectors) RAID volume 'L2ARC0' is optimal >>> mfid2 on mfi0 >>> mfid2: 121472MB (248774656 sectors) RAID volume 'ZIL0' is optimal >>> mfid3 on mfi0 >>> mfid3: 1143552MB (2341994496 sectors) RAID volume 'DATASTRIPE1' is >>optimal >>> mfid4 on mfi0 >>> mfid4: 1143552MB (2341994496 sectors) RAID volume 'DATASTRIPE2' is >>optimal >>> ugen0.1: at usbus0 >>> uhub0: on >>usbus0 >>> ugen1.1: at usbus1 >>> uhub1: on >>usbus1 >>> uhub0: 2 ports with 2 removable, self powered >>> uhub1: 2 ports with 2 removable, self powered >>> ugen0.2: at usbus0 >>> uhub2: >2> on >>> usbus0 >>> ugen1.2: at usbus1 >>> uhub3: >2> on >>> usbus1 >>> uhub2: 6 ports with 6 removable, self powered >>> uhub3: 8 ports with 8 removable, self powered >>> ugen0.3: at usbus0 >>> uhub4: >addr 3> >>> on usbus0 >>> SMP: AP CPU #1 Launched! >>> SMP: AP CPU #7 Launched! >>> SMP: AP CPU #3 Launched! >>> SMP: AP CPU #6 Launched! >>> SMP: AP CPU #2 Launched! >>> SMP: AP CPU #5 Launched! >>> SMP: AP CPU #4 Launched! >>> Timecounter "TSC-low" frequency 9765827 Hz quality 1000 >>> Root mount waiting for: usbus0 >>> uhub4: 6 ports with 6 removable, self powered >>> ugen0.4: at usbus0 >>> ukbd0: on usbus0 >>> kbd0 at ukbd0 >>> Root mount waiting for: usbus0 >>> Root mount waiting for: usbus0 >>> ugen0.5: at usbus0 >>> umass0: on usbus0 >>> Trying to mount root from ufs:/dev/mfid0p3 [rw]... >>> cd0 at umass-sim0 bus 0 scbus1 target 0 lun 0 >>> cd0: Removable CD-ROM SCSI-0 device >>> cd0: 40.000MB/s transfers >>> cd0: Attempt to query device size failed: NOT READY, Medium not >>present >>> da0 at umass-sim0 bus 0 scbus1 target 0 lun 1 >>> da0: Removable Direct Access SCSI-0 >>device >>> da0: 40.000MB/s transfers >>> da0: Attempt to query device size failed: NOT READY, Medium not >>present >>> ZFS filesystem version 5 >>> ZFS storage pool version 28 >>> ums0: on usbus0 >>> ums0: 3 buttons and [Z] coordinates ID=3D0 >>> ums1: on usbus0 >>> ums1: 3 buttons and [XYZ] coordinates ID=3D0 >>> igb0: link state changed to UP >>> >>> With this loader.conf: >>> vm.kmem_size_max=3D"64G" >>> vm.kmem_size=3D"48G" >>> vfs.zfs.arc_min=3D"4096M" >>> vfs.zfs.arc_max=3D"15872M" >>> vfs.zfs.txg.synctime=3D"1" >>> vfs.zfs.txg.timeout=3D"5" >>> vfs.zfs.write_limit_override=3D"**0" >>> vfs.zfs.txg.write_limit_**override=3D"0" >>> vfs.zfs.zil_disable=3D"0" >>> >>> And with this sysctl.conf: >>> security.bsd.see_other_uids=3D0 >>> net.inet.tcp.blackhole=3D2 >>> net.inet.udp.blackhole=3D1 >>> net.inet.tcp.sendspace=3D65536 >>> net.inet.tcp.recvspace=3D65536 >>> net.link.ether.inet.log_arp_**wrong_iface=3D0 >>> kern.ipc.somaxconn=3D32768 >>> kern.threads.max_threads_per_**proc=3D16384 >>> kern.maxfiles=3D262144 >>> kern.maxfilesperproc=3D131072 >>> kern.ipc.nmbclusters=3D65536 >>> kern.corefile=3D"/var/coredumps/**%U.%N.%P.core" >>> >>> vfs.zfs.txg.timeout=3D5 >>> vfs.zfs.l2arc_headroom=3D16 >>> vfs.zfs.l2arc_write_max=3D**26843546 >>> vfs.zfs.l2arc_write_boost=3D**26843546 >>> vfs.zfs.vdev.min_pending=3D4 >>> vfs.zfs.vdev.max_pending=3D64 >>> vfs.zfs.l2arc_noprefetch=3D1 >>> vfs.zfs.dedup.prefetch=3D0 >>> vfs.zfs.prefetch_disable=3D1 >>> >>> I will use this server as a mysql-5.6 dbserver so I have a root >>partition >>> using a hw raid1 and a /DATAZFS partition, follows configuration: >>> >>> root# mfiutil show drives >>> mfi0 Physical Drives: >>> 0 ( 279G) ONLINE SAS E1:S0 >>> 1 ( 279G) ONLINE SAS E1:S1 >>> 2 ( 558G) ONLINE SAS E1:S2 >>> 3 ( 558G) ONLINE SAS E1:S3 >>> 4 ( 558G) ONLINE SAS E1:S4 >>> 5 ( 558G) ONLINE SAS E1:S5 >>> 6 ( 119G) ONLINE SATA E1:S6 >>> 7 ( 119G) ONLINE SATA E1:S7 >>> root# mfiutil show volumes >>> mfi0 Volumes: >>> Id Size Level Stripe State Cache Name >>> mfid0 ( 278G) RAID-1 64k OPTIMAL Disabled >>> mfid1 ( 118G) RAID-0 64k OPTIMAL Disabled >>> mfid2 ( 118G) RAID-0 64k OPTIMAL Disabled >>> mfid3 ( 1116G) RAID-1 64k OPTIMAL Disabled >>> >>> I have a /DATAZFS partition, too: >>> root# zfs get all DATAZFS >>> NAME PROPERTY VALUE SOURCE >>> DATAZFS type filesystem - >>> DATAZFS creation Thu Mar 21 10:45 2013 - >>> DATAZFS used 54.2G - >>> DATAZFS available 1.02T - >>> DATAZFS referenced 54.2G - >>> DATAZFS compressratio 1.00x - >>> DATAZFS mounted yes - >>> DATAZFS quota none default >>> DATAZFS reservation none default >>> DATAZFS recordsize 16K local >>> DATAZFS mountpoint /DATAZFS default >>> DATAZFS sharenfs off default >>> DATAZFS checksum on default >>> DATAZFS compression off default >>> DATAZFS atime off local >>> DATAZFS devices on default >>> DATAZFS exec on default >>> DATAZFS setuid on default >>> DATAZFS readonly off default >>> DATAZFS jailed off default >>> DATAZFS snapdir hidden default >>> DATAZFS aclmode discard default >>> DATAZFS aclinherit restricted default >>> DATAZFS canmount on default >>> DATAZFS xattr off temporary >>> DATAZFS copies 1 default >>> DATAZFS version 5 - >>> DATAZFS utf8only off - >>> DATAZFS normalization none - >>> DATAZFS casesensitivity sensitive - >>> DATAZFS vscan off default >>> DATAZFS nbmand off default >>> DATAZFS sharesmb off default >>> DATAZFS refquota none default >>> DATAZFS refreservation none default >>> DATAZFS primarycache all default >>> DATAZFS secondarycache all default >>> DATAZFS usedbysnapshots 0 - >>> DATAZFS usedbydataset 54.2G - >>> DATAZFS usedbychildren 1.41M - >>> DATAZFS usedbyrefreservation 0 - >>> DATAZFS logbias latency default >>> DATAZFS dedup off local >>> DATAZFS mlslabel - >>> DATAZFS sync standard default >>> DATAZFS refcompressratio 1.00x - >>> DATAZFS written 54.2G - >>> >>> root# zpool status -v >>> pool: DATAZFS >>> state: ONLINE >>> scan: none requested >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> DATAZFS ONLINE 0 0 0 >>> mfid3 ONLINE 0 0 0 >>> logs >>> mfid2 ONLINE 0 0 0 >>> cache >>> mfid1 ONLINE 0 0 0 >>> >>> errors: No known data errors >>> >>> root# mfiutil show firmware >>> mfi0 Firmware Package Version: 21.1.0-0007 >>> mfi0 Firmware Images: >>> Name Version Date Time Status >>> BIOS 5.31.01_4.12.05.00_0x05110000 6/20/2012 >>> 6/20/2012 >>> active >>> CTLR 4.00-0014 Aug 04 2011 12:49:17 active >>> PCLI 05.00-03:#%00008 Feb 17 2011 14:03:12 active >>> APP 3.130.05-1796 Jul 27 2012 15:58:31 active >>> NVDT 2.1108.03-0094 Jun 22 2012 12:40:55 active >>> BTBL 2.03.00.00-0004 Jul 27 2012 16:39:29 active >>> BOOT 06.253.57.219 9/9/2010 15:32:25 active >>> root# >>> >>> Then, I installed CentOS on the same hardware, same version of mysql >>> (5.6.10-enterprise and 5.6.10-GA), and I tried an OLTP test (included >>in >>> sysbench-0.5) executing 2000000 queries on 10000000 rows, with these >>> results (transactions per seconds): >>> >>> BSD 9.1 amd64 ZFS (without tweaks): >>> 1 thread: 2780 >>> 2 threads: 4371 >>> 4 threads: 6876 >>> 8 threads: 8202 >>> 12 threads: 8077 >>> 16 threads: 7780 >>> 24 threads: 7563 >>> 32 threads: 7632 >>> 48 threads: 7960 >>> 64 threads: 8062 >>> >>> BSD 9.1 amd64 ZFS (with tweaks): >>> 1 thread: 2589 >>> 2 threads: 4679 >>> 4 threads: 6438 >>> 8 threads: 7073 >>> 12 threads: 7121 >>> 16 threads: 7227 >>> 24 threads: 7132 >>> 32 threads: 7273 >>> 48 threads: 7623 >>> 64 threads: 7672 >>> >>> CentOS 6.3 (ext4, without tweaks): >>> 1 thread: 4163 >>> 2 threads: 7653 >>> 4 threads: 10905 >>> 8 threads: 12511 >>> 12 threads: 13556 >>> 16 threads: 14832 >>> 24 threads: 16270 >>> 32 threads: 16733 >>> 48 threads: 16925 >>> 64 threads: 16895 >>> >>> >>> Well, any of you could ask "why zfs, please try ufs that is better": >>you >>> are right, but changing ZFS with UFS doesn't change so much: >>> >>http://lists.freebsd.org/**pipermail/freebsd-fs/2013-**March/016747.html<= http://lists.freebsd.org/pipermail/freebsd-fs/2013-March/016747.html> >>> >>> So I hope I'm missing something because the performances of a >>standard >>> CentOS installation on the same hardware performs much better than a >>9.1 >>> amd64 installation (I'm using the standard ULE scheduler included in >>the >>> 9.1 release). >>> >>> Could anyone help me? >>> >>> Thanks, >>> d. >>> ______________________________**_________________ >>> freebsd-performance@freebsd.**org >>mailing list >>> >>http://lists.freebsd.org/**mailman/listinfo/freebsd-**performance >>> To unsubscribe, send any mail to "freebsd-performance-** >>> unsubscribe@freebsd.org >>" >>> > > -- > d. > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd= .org" What are the mysql settings? my.cnf. From owner-freebsd-performance@FreeBSD.ORG Sat Mar 23 00:37:25 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B9D3D91C for ; Sat, 23 Mar 2013 00:37:25 +0000 (UTC) (envelope-from pathiaki2@yahoo.com) Received: from nm37-vm5.bullet.mail.bf1.yahoo.com (nm37-vm5.bullet.mail.bf1.yahoo.com [72.30.238.205]) by mx1.freebsd.org (Postfix) with ESMTP id E9072F3E for ; Sat, 23 Mar 2013 00:37:24 +0000 (UTC) Received: from [98.139.212.149] by nm37.bullet.mail.bf1.yahoo.com with NNFMP; 23 Mar 2013 00:34:44 -0000 Received: from [98.139.212.217] by tm6.bullet.mail.bf1.yahoo.com with NNFMP; 23 Mar 2013 00:34:44 -0000 Received: from [127.0.0.1] by omp1026.mail.bf1.yahoo.com with NNFMP; 23 Mar 2013 00:34:44 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 84157.53494.bm@omp1026.mail.bf1.yahoo.com Received: (qmail 27256 invoked by uid 60001); 23 Mar 2013 00:34:43 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1363998883; bh=4n6L3m6BidfPCbuRZ2sfKN1rwqBfyr07Z64ev/tfR6M=; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=zxdLBJIOi8NJp478SWP9N9K8DGv0jMaRz+XnTXFQ3J+HihMQST7bdFA8wQvqwEtfNcVmWofyuAGOSE8yuO2jnTn0zQjfMiSpfFdVQ0K5VXfKhSiH1r1lJnP6xteO3WhaMyFPoWqXSAT9OKXStfvz1m6k8b8/lHcg3857IVcwVWk= DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Rocket-MIMEInfo:X-Mailer:References:Message-ID:Date:From:Reply-To:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type; b=Ux74imsg9XJUy+HyKXM0wMl4o2CUqVdRzihUGijfALeyRQ7FIPsyzhzmo8+oJZnOgLE+m3t2grkZ73ttMk2zekotNpy4QA5zv+DUzuHmHE+iD5i5inw4pmWprn8jiaRhV2N4Guqhpk4bjh77Av2qgl/iOswZb+Wv3TrR02ZxDSc=; X-YMail-OSG: 3XZagrkVM1k3FzwNwsMgtOtWoMoSXcA0fPjX_icdbYV1evb w62vm_Lg_ZnfCQU5ZgPrljNxSoDcmQ5F88_6xFYNy2AlxAs6E1_FQh0AL9_X l6Kln1hKxIv.iGAiZYqe9SKkjHcxziImbHWtMv2zyNtLszUiyZE310j0Rz8J dJWbHF1GR0lL9IWnmrdyRu9SR_tXDgs.6eUmYvNRnIJN3vHggaMrujPaWhbB w.eWmL58vn6ztwqYFCWnYnVeGRK8GeBRcTIYHOSjloHgU4KccAgRp3pgSj_4 _PLFltYZmQdIAn3IY0xEZNxpwKzulQaRSJcSL.pfe5y8Cm4gAImy7Qr3kGF0 2dvhlu52ejOoYGuP7z0NPKdGnWTrNgp3ZZ0m.mx4hnQqiKRtwD.4kq1ZZWjE 318o5Ych4CynVRy5cU8m3tU1PAMQ3lee5oCSuJ2Me6YV23nq1S01TOUVH7VQ 29fUkrBQjx72Qoe9Gc2W5VNY.qlUWqqanhOA_kWhsECxr2HSRRZ4tsLhHVKy 9uZR0VoSawKQFp41l3Q-- Received: from [66.189.28.218] by web141401.mail.bf1.yahoo.com via HTTP; Fri, 22 Mar 2013 17:34:43 PDT X-Rocket-MIMEInfo: 002.001, SGksCgpUaGVyZSBhcmUgc2V2ZXJhbCB0aGluZ3MgYWJvdXQgdGhpcyB0aGF0IGFyZSBoaWdobHkgc3VzcGVjdC7CoCAKCgpGaXJzdCwgd2lwZSBvdXQgdGhlIGhhcmR3YXJlIFJBSUQuwqAgVGhlIHByb2Nlc3NvciBkb2luZyBSQUlEIGNvbXB1dGF0aW9uIGlzLCBwcm9iYWJseSwgTVVDSCBzbG93ZXIgdGhhbiBhIGNvcmUgb24gdGhlIENQVS7CoCBFdmVuIGlmIGl0J3MgUkFJRC0xIChTaW1wbGUgTWlycm9yKSB0aGlzIFJBSUQgY2FyZCBpcyBwZXJmb3JtaW5nIHRhc2tzIHRoYXQgaXMgZG9lcyBub3QgbmVlZCABMAEBAQE- X-Mailer: YahooMailWebService/0.8.138.524 References: Message-ID: <1363998883.22604.YahooMailNeo@web141401.mail.bf1.yahoo.com> Date: Fri, 22 Mar 2013 17:34:43 -0700 (PDT) From: Paul Pathiakis Subject: Re: FreeBSD 9.1 vs CentOS 6.3 To: Davide D'Amico , =?utf-8?B?0JXQstCz0LXQvdC40Lkg0KXQvtGA0LrQuNC9?= In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "freebsd-performance@freebsd.org" X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Paul Pathiakis List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Mar 2013 00:37:25 -0000 Hi,=0A=0AThere are several things about this that are highly suspect.=C2=A0= =0A=0A=0AFirst, wipe out the hardware RAID.=C2=A0 The processor doing RAID= computation is, probably, MUCH slower than a core on the CPU.=C2=A0 Even i= f it's RAID-1 (Simple Mirror) this RAID card is performing tasks that is do= es not need to do including replicating writes to two targets from the cont= roller or checking it's cache, battery, etc.=C2=A0 If it's possible to disa= ble the onboard cache, do it.=0A=0A=0A(This controller seems to use the mfi= driver so that's probably a re-badged LSI with an LSI200x chipset.=C2=A0 (= Lately, I have not been too thrilled with the mfi driver performance.=C2=A0= The mpt seems much more stable and robust - my opinion not bashing to whoe= ver wrote it.)=0A=0A=0ATake the card to full JBOD mode before testing your = "throughput".=0A=0AAlso, make sure that all on-disk caches are "off".=C2=A0= You do not want to get into a double-buffer flush corruption issue.=0A=0AI= see you are using two SSDs as ARC2 caches....=C2=A0 Quite frankly, junk th= em as well, for now.=C2=A0 You may want to deal with increasing throughput = once you find out what the issue is with the base throughput.=0A=0A=0ACreat= e the appropriate pools/mirrors as necessary, all on simple JBOD.=0A=0AFrom= there, run a simple test to see what your throughput is on single spindle.= =C2=A0 (I'm talking SIMPLE.=C2=A0 dd if=3D/dev/zero of=3D//zeroes.out= =C2=A0 bs=3D128k count=3D - Thi= s should be over 100 MB/sec throughput.=C2=A0 (with those drives it may be = 150 MB/sec or more)=0A=0AAfter that, check the throughput of the card/chann= el.=C2=A0 Write a very simple script script that performs the same throughp= ut test on half the available spindles (something as ludicrous as the same = dd command to half your spindles throwing it in the background as each kick= s off - Then, do it for all of your drives)=C2=A0 =0A=0A=0AThis is an imper= fect test but it's down and dirty.=C2=A0 You can use the performance/benchm= ark tools later.=0A=0ALet's just remember that the throughput of 6 Gb/sec i= s gigaBITS not gigaBYTES.=C2=A0 There's a factor of 8 difference.... or 750= MB/sec... theoretical.=0A=0AYou should exceed channel speed with just 5 of= these drives/spindles... use 6 to be sure (I would think you won't see agg= regate throughput beyond 725 MB/sec....=C2=A0 If any tool is showing you th= at on any OS with this configuration, they aren't doing proper calculations= and/or they are doing MB vs MiB.=C2=A0 =0A=0A=0AIf you can't get up to tha= t speed, you may have a driver issue or channel issue.=0A=0A(I was getting = as high as 700 MB/sec with 6 SATA drives doing it this way.... No SSD, ASUS= MB 990FX with 1100T 6 core proc, 16 GB RAM... )=0A=0AP.=0A=0A=0A=0A_______= _________________________=0A From: Davide D'Amico =0ATo: =D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9 =D0=A5=D0=BE=D1=80= =D0=BA=D0=B8=D0=BD =0ACc: freebsd-performance@f= reebsd.org =0ASent: Friday, March 22, 2013 12:15 PM=0ASubject: Re: FreeBSD = 9.1 vs CentOS 6.3=0A =0AWell, the I/O isn't the bottleneck (if you follow t= he link to freebsd-fs, you'll see iostats values) but it seems something re= lated to cpu/scheduler or something else.=0ANow I am trying vmware 5 on the= same server and a vm with centos6: the vm outperforms freebsd with every c= oncurrency from 1 to 48 threads in sysbench oltp tests.=0A=0AThanks,=0Ad.= =0A=0A"=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9 =D0=A5=D0=BE=D1=80=D0=BA= =D0=B8=D0=BD" ha scritto:=0A=0A>Hi Davide!=0A>= =0A>Are you sure that disk is the bottleneck in your test?=0A>Does systat -= vm 1 show 100% busy for disk ?=0A>=0A>Evgeny.=0A>=0A>=0A>2013/3/22 Davide D= 'Amico =0A>=0A>> Hi, I'm doing performance te= sts on a DELL R720, follows dmesg:=0A>>=0A>> Copyright (c) 1992-2012 The Fr= eeBSD Project.=0A>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991,= 1992, 1993,=0A>1994=0A>>=C2=A0 =C2=A0 =C2=A0 =C2=A0 The Regents of the Un= iversity of California. All rights=0A>reserved.=0A>> FreeBSD is a registere= d trademark of The FreeBSD Foundation.=0A>> FreeBSD 9.1-RELEASE #0: Tue Mar= 19 10:24:21 CET 2013=0A>>=C2=A0 =C2=A0 root:/usr/obj/usr/src/sys/R720 amd= 64=0A>> CPU: Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz (2500.05-MHz K8-class= =0A>CPU)=0A>>=C2=A0 Origin =3D "GenuineIntel"=C2=A0 Id =3D 0x206d7=C2=A0 F= amily =3D 6=C2=A0 Model =3D 2d=0A>Stepping =3D=0A>> 7=0A>> Features=3D0xbfe= bfbff> APIC,SEP,MTRR,PGE,MCA,CM= OV,**PAT,PSE36,CLFLUSH,DTS,ACPI,**=0A>> MMX,FXSR,SSE,SSE2,SS,HTT,TM,**PBE>= =0A>> Features2=3D0x1fbee3ff> VMX= ,SMX,EST,TM2,SSSE3,CX16,**xTPR,PDCM,PCID,DCA,SSE4.1,**=0A>> SSE4.2,x2APIC,P= OPCNT,TSCDLT,**AESNI,XSAVE,OSXSAVE,AVX>=0A>>=C2=A0 AMD Features=3D0x2c1008= 00=0A>>=C2=A0 AMD Features2=3D0x1=0A= >>=C2=A0 TSC: P-state invariant, performance statistics=0A>> real memory= =C2=A0 =3D 34359738368 (32768 MB)=0A>> avail memory =3D 33027436544 (31497 = MB)=0A>> Event timer "LAPIC" quality 600=0A>> ACPI APIC Table: =0A>> FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs=0A= >> FreeBSD/SMP: 2 package(s) x 2 core(s) x 2 SMT threads=0A>>=C2=A0 cpu0 (B= SP): APIC ID:=C2=A0 0=0A>>=C2=A0 cpu1 (AP): APIC ID:=C2=A0 1=0A>>=C2=A0 cpu= 2 (AP): APIC ID: 10=0A>>=C2=A0 cpu3 (AP): APIC ID: 11=0A>>=C2=A0 cpu4 (AP):= APIC ID: 32=0A>>=C2=A0 cpu5 (AP): APIC ID: 33=0A>>=C2=A0 cpu6 (AP): APIC I= D: 42=0A>>=C2=A0 cpu7 (AP): APIC ID: 43=0A>> ioapic1: Changing APIC ID to 1= =0A>> ioapic2: Changing APIC ID to 2=0A>> ioapic0 irqs 0-23 o= n motherboard=0A>> ioapic1 irqs 32-55 on motherboard=0A>> ioa= pic2 irqs 64-87 on motherboard=0A>> kbd1 at kbdmux0=0A>> ctl:= CAM Target Layer loaded=0A>> acpi0: on motherboard=0A>> acpi= 0: Power Button (fixed)=0A>> acpi0: reservation of 0, 2000 (3) failed=0A>> = cpu0: on acpi0=0A>> cpu1: on acpi0=0A>> cpu2: on acpi0=0A>> cpu3: on acpi0=0A>> cpu4: on acpi0= =0A>> cpu5: on acpi0=0A>> cpu6: on acpi0=0A>> cpu7: <= ACPI CPU> on acpi0=0A>> atrtc0: port 0x70-0x7f irq 8 on= acpi0=0A>> Event timer "RTC" frequency 32768 Hz quality 0=0A>> attimer0: <= AT timer> port 0x40-0x5f irq 0 on acpi0=0A>> Timecounter "i8254" frequency = 1193182 Hz quality 0=0A>> Event timer "i8254" frequency 1193182 Hz quality = 100=0A>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 900=0A>> acpi= _timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0=0A>> hpet0= : iomem 0xfed00000-0xfed003ff on=0A>acpi0=0A>>= Timecounter "HPET" frequency 14318180 Hz quality 950=0A>> Event timer "HPE= T" frequency 14318180 Hz quality 550=0A>> pcib0: por= t 0xcf8-0xcff on acpi0=0A>> pci0: on pcib0=0A>> pcib1: irq 53 at device 1.0 on pci0=0A>> pci1: on = pcib1=0A>> igb0: por= t=0A>> 0xfc80-0xfc9f mem 0xdcb00000-0xdcbfffff,**0xdcff0000-0xdcff3fff irq= =0A>38 at=0A>> device 0.0 on pci1=0A>> igb0: Using MSIX interrupts with 9 v= ectors=0A>> igb0: Ethernet address: bc:30:5b:ef:27:b8=0A>> [...]=0A>> igb1:= port=0A>> 0xfca0-0x= fcbf mem 0xdcc00000-0xdccfffff,**0xdcff4000-0xdcff7fff irq=0A>37 at=0A>> de= vice 0.1 on pci1=0A>> igb1: Using MSIX interrupts with 9 vectors=0A>> igb1:= Ethernet address: bc:30:5b:ef:27:b9=0A>> [...]=0A>> igb2: port=0A>> 0xfcc0-0xfcdf mem 0xdcd000= 00-0xdcdfffff,**0xdcff8000-0xdcffbfff irq=0A>36 at=0A>> device 0.2 on pci1= =0A>> igb2: Using MSIX interrupts with 9 vectors=0A>> igb2: Ethernet addres= s: bc:30:5b:ef:27:ba=0A>> [...]=0A>> igb3: port=0A>> 0xfce0-0xfcff mem 0xdce00000-0xdcefffff,**= 0xdcffc000-0xdcffffff irq=0A>34 at=0A>> device 0.3 on pci1=0A>> igb3: Using= MSIX interrupts with 9 vectors=0A>> igb3: Ethernet address: bc:30:5b:ef:27= :bb=0A>> [...]=0A>> pcib2: irq 53 at device 2.0 on pc= i0=0A>> pci3: on pcib2=0A>> pcib3: irq= 53 at device 2.2 on pci0=0A>> pci2: on pcib3=0A>> mfi0: port 0xec00-0xecff mem=0A>> 0xddffc000-0x= ddffffff,**0xddf80000-0xddfbffff irq 42 at device 0.0 on=0A>pci2=0A>> mfi0:= Using MSI=0A>> mfi0: Megaraid SAS driver Ver 4.23=0A>> mfi0: MaxCmd =3D 3f= 0 MaxSgl =3D 46 state =3D b73c03f0=0A>> pcib4: irq 53= at device 3.0 on pci0=0A>> pci4: on pcib4=0A>> pcib5: irq 53 at device 3.2 on pci0=0A>> pci5: on = pcib5=0A>> pci0: at device 5.0 (no driver attached)=0A>> = pci0: at device 5.2 (no driver attached)=0A>> pcib6: irq 16 at device 17.0 on pci0=0A>> pci6: on pcib6=0A= >> pci0: at device 22.0 (no driver attached)=0A>> pci0: at device 22.1 (no driver attached)=0A>> ehci0: mem 0xdf8fe000-0xdf8fe3ff=0A>irq=0A>> 23 at device 26.0= on pci0=0A>> usbus0: EHCI version 1.0=0A>> usbus0 on ehci0=0A>> pcib7: at device 28.0 on pci0=0A>> device_attach: pcib7 attach = returned 6=0A>> pcib7: irq 19 at device 28.7 on pci0= =0A>> pci8: on pcib7=0A>> pcib8: at device = 0.0 on pci8=0A>> pci9: on pcib8=0A>> pcib9: at d= evice 0.0 on pci9=0A>> pci10: on pcib9=0A>> pcib10: at device 0.0 on pci10=0A>> pci11: on pcib10=0A>> vgapci0: mem 0xd8000000-0xd8ffffff,**=0A>> 0xdeffc000-0xdefff= fff,**0xde000000-0xde7fffff irq 19 at device 0.0 on=0A>> pci11=0A>> pcib11:= at device 1.0 on pci9=0A>> pci12: on pcib11=0A>= > ehci1: mem 0xdf8ff000-0xdf8ff3ff=0A>i= rq=0A>> 22 at device 29.0 on pci0=0A>> usbus1: EHCI version 1.0=0A>> usbus1= on ehci1=0A>> pcib12: at device 30.0 on pci0=0A>> pci13: = on pcib12=0A>> isab0: at device 31.0 on pci0=0A>= > isa0: on isab0=0A>> pcib13: on acpi0=0A>= > pci64: on pcib13=0A>> pcib14: irq 85= at device 1.0 on pci64=0A>> pci65: on pcib14=0A>> pcib15: <= ACPI PCI-PCI bridge> irq 85 at device 2.0 on pci64=0A>> pci66: on pcib15=0A>> pcib16: irq 85 at device 3.0 on pci= 64=0A>> pci67: on pcib16=0A>> pcib17: = irq 85 at device 3.2 on pci64=0A>> pci68: on pcib17=0A>> pci= 64: at device 5.0 (no driver attached)=0A>> pci64: at device 5.2 (no driver attached)=0A>> pcib18: on acpi0=0A>> pci63: on pcib18=0A>> pci63: at device 8.0 (no driver attached)=0A>> [...]=0A>> pci63: a= t device 14.1 (no driver attached)=0A>> [...]=0A>> pci63: at device = 19.1 (no driver attached)=0A>> pci63: at device 19.4 (no driver atta= ched)=0A>> pci63: at device 19.5 (no driver attached)=0A>> pci63: at device 19.6 (no driver attached)=0A>> pcib19: on acpi0=0A>> pci127: on pcib19=0A>> [...]=0A>>= pci127: at device 14.1 (no driver attached)=0A>> [...]=0A>> pci127:= at device 19.1 (no driver attached)=0A>> pci127: at device 1= 9.4 (no driver attached)=0A>> pci127: at device 19.5 (no driver atta= ched)=0A>> pci127: at device 19.6 (no driver attached)=0A= >> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0=0A>> uart0:= <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on=0A>acpi0=0A>> o= rm0: at iomem 0xc0000-0xc7fff,0xec000-**0xeffff on=0A>isa= 0=0A>> sc0: at flags 0x100 on isa0=0A>> sc0: VGA <16 virtu= al consoles, flags=3D0x300>=0A>> vga0: at port 0x3c0-0x3d= f iomem 0xa0000-0xbffff on=0A>isa0=0A>> est0: on cpu0=0A>> est: CPU supports Enhanced Speedstep, but is not rec= ognized.=0A>> est: cpu_vendor GenuineIntel, msr 226d00001e00=0A>> device_at= tach: est0 attach returned 6=0A>> p4tcc0: o= n cpu0=0A>> est1: on cpu1=0A>> est: = CPU supports Enhanced Speedstep, but is not recognized.=0A>> est: cpu_vendo= r GenuineIntel, msr 226d00001e00=0A>> device_attach: est1 attach returned 6= =0A>> p4tcc1: on cpu1=0A>> est2: on cpu2=0A>> est: CPU supports Enhanced Speeds= tep, but is not recognized.=0A>> est: cpu_vendor GenuineIntel, msr 226d0000= 1e00=0A>> device_attach: est2 attach returned 6=0A>> p4tcc2: on cpu2=0A>> est3: = on cpu3=0A>> est: CPU supports Enhanced Speedstep, but is not recognized.= =0A>> est: cpu_vendor GenuineIntel, msr 224400001e00=0A>> device_attach: es= t3 attach returned 6=0A>> p4tcc3: on cpu3= =0A>> est4: on cpu4=0A>> est: CPU su= pports Enhanced Speedstep, but is not recognized.=0A>> est: cpu_vendor Genu= ineIntel, msr 226d00001e00=0A>> device_attach: est4 attach returned 6=0A>> = p4tcc4: on cpu4=0A>> est5: on cpu5=0A>> est: CPU supports Enhanced Speedstep, b= ut is not recognized.=0A>> est: cpu_vendor GenuineIntel, msr 224400001e00= =0A>> device_attach: est5 attach returned 6=0A>> p4tcc5: on cpu5=0A>> est6: on = cpu6=0A>> est: CPU supports Enhanced Speedstep, but is not recognized.=0A>>= est: cpu_vendor GenuineIntel, msr 224400001e00=0A>> device_attach: est6 at= tach returned 6=0A>> p4tcc6: on cpu6=0A>> e= st7: on cpu7=0A>> est: CPU supports = Enhanced Speedstep, but is not recognized.=0A>> est: cpu_vendor GenuineInte= l, msr 224400001e00=0A>> device_attach: est7 attach returned 6=0A>> p4tcc7:= on cpu7=0A>> mfi0: 6040 (417120269s/0x0020= /info) - Shutdown command received from=0A>host=0A>> mfi0: 6041 (boot + 4s/= 0x0020/info) - Firmware initialization started=0A>(PCI=0A>> ID 005b/1000/1f= 34/1028)=0A>> mfi0: 6042 (boot + 4s/0x0020/info) - Firmware version 3.130.0= 5-1796=0A>> mfi0: 6043 (boot + 5s/0x0008/info) - Battery Present=0A>> mfi0:= 6044 (boot + 5s/0x0020/info) - Package version 21.1.0-0007=0A>> mfi0: 6045= (boot + 5s/0x0020/info) - Board Revision A01=0A>> mfi0: 6046 (boot + 6s/0x= 0008/info) - Battery temperature is normal=0A>> mfi0: 6047 (boot + 6s/0x000= 8/info) - Current capacity of the battery=0A>is=0A>> above threshold=0A>> m= fi0: 6048 (boot + 19s/0x0004/info) - Enclosure PD 20(c None/p1)=0A>> commun= ication restored=0A>> mfi0: 6049 (boot + 19s/0x0002/info) - Inserted: Encl = PD 20=0A>> mfi0: 6050 (boot + 19s/0x0002/info) - Inserted: PD 20(c None/p1)= =0A>Info:=0A>> enclPd=3D20, scsiType=3Dd, portMap=3D00, sasAddr=3D5e4ae020a= 26cb500,**=0A>> 0000000000000000=0A>> mfi0: 6051 (boot + 19s/0x0002/info) -= Inserted: PD 00(e0x20/s0)=0A>> mfi0: 6052 (boot + 19s/0x0002/info) - Inser= ted: PD 00(e0x20/s0) Info:=0A>> enclPd=3D20, scsiType=3D0, portMap=3D04, sa= sAddr=3D5000c50054982339,**=0A>> 0000000000000000=0A>> mfi0: 6053 (boot + 1= 9s/0x0002/info) - Inserted: PD 01(e0x20/s1)=0A>> mfi0: 6054 (boot + 19s/0x0= 002/info) - Inserted: PD 01(e0x20/s1) Info:=0A>> enclPd=3D20, scsiType=3D0,= portMap=3D05, sasAddr=3D5000c50054986c91,**=0A>> 0000000000000000=0A>> mfi= 0: 6055 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2)=0A>> mfi0: 605= 6 (boot + 19s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info:=0A>> enclPd=3D= 20, scsiType=3D0, portMap=3D06, sasAddr=3D5000c50053da8f75,**=0A>> 00000000= 00000000=0A>> mfi0: 6057 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s= 3)=0A>> mfi0: 6058 (boot + 19s/0x0002/info) - Inserted: PD 03(e0x20/s3) Inf= o:=0A>> enclPd=3D20, scsiType=3D0, portMap=3D07, sasAddr=3D5000c50053c57eb5= ,**=0A>> 0000000000000000=0A>> mfi0: 6059 (boot + 19s/0x0002/info) - Insert= ed: PD 04(e0x20/s4)=0A>> mfi0: 6060 (boot + 19s/0x0002/info) - Inserted: PD= 04(e0x20/s4) Info:=0A>> enclPd=3D20, scsiType=3D0, portMap=3D00, sasAddr= =3D5000c50053d364c5,**=0A>> 0000000000000000=0A>> mfi0: 6061 (boot + 19s/0x= 0002/info) - Inserted: PD 05(e0x20/s5)=0A>> mfi0: 6062 (boot + 19s/0x0002/i= nfo) - Inserted: PD 05(e0x20/s5) Info:=0A>> enclPd=3D20, scsiType=3D0, port= Map=3D03, sasAddr=3D5000c50053da2b41,**=0A>> 0000000000000000=0A>> mfi0: 60= 63 (boot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6)=0A>> mfi0: 6064 (bo= ot + 19s/0x0002/info) - Inserted: PD 06(e0x20/s6) Info:=0A>> enclPd=3D20, s= csiType=3D0, portMap=3D02, sasAddr=3D4433221102000000,**=0A>> 0000000000000= 000=0A>> mfi0: 6065 (boot + 19s/0x0002/WARN) - PD 06(e0x20/s6) is not a=0A>= certified=0A>> drive=0A>> mfi0: 6066 (boot + 19s/0x0002/info) - Inserted: P= D 07(e0x20/s7)=0A>> mfi0: 6067 (boot + 19s/0x0002/info) - Inserted: PD 07(e= 0x20/s7) Info:=0A>> enclPd=3D20, scsiType=3D0, portMap=3D01, sasAddr=3D4433= 221103000000,**=0A>> 0000000000000000=0A>> mfi0: 6068 (boot + 19s/0x0002/WA= RN) - PD 07(e0x20/s7) is not a=0A>certified=0A>> drive=0A>> mfi0: 6069 (417= 120312s/0x0020/info) - Time established as 03/20/13=0A>> 18:45:12; (32 seco= nds since power on)=0A>> mfi0: 6070 (417120445s/0x0020/info) - Host driver = is loaded and=0A>operational=0A>> Timecounters tick every 1.000 msec=0A>> u= sbus0: 480Mbps High Speed USB v2.0=0A>> usbus1: 480Mbps High Speed USB v2.0= =0A>> mfid0 on mfi0=0A>> mfid0: 285568MB (584843264 sectors) RAID volume 'B= ASE' is optimal=0A>> mfid1 on mfi0=0A>> mfid1: 121472MB (248774656 sectors)= RAID volume 'L2ARC0' is optimal=0A>> mfid2 on mfi0=0A>> mfid2: 121472MB (2= 48774656 sectors) RAID volume 'ZIL0' is optimal=0A>> mfid3 on mfi0=0A>> mfi= d3: 1143552MB (2341994496 sectors) RAID volume 'DATASTRIPE1' is=0A>optimal= =0A>> mfid4 on mfi0=0A>> mfid4: 1143552MB (2341994496 sectors) RAID volume = 'DATASTRIPE2' is=0A>optimal=0A>> ugen0.1: at usbus0=0A>> uhub0: on=0A>usbus0=0A>> ugen= 1.1: at usbus1=0A>> uhub1: on=0A>usbus1=0A>> uhub0: 2 ports with 2 removable, self pow= ered=0A>> uhub1: 2 ports with 2 removable, self powered=0A>> ugen0.2: at usbus0=0A>> uhub2: 2> on=0A>> usbus0=0A>> ugen1.2: at u= sbus1=0A>> uhub3: 2> on=0A>> usbus1=0A>> uhub2: 6 ports with 6 removable, self powere= d=0A>> uhub3: 8 ports with 8 removable, self powered=0A>> ugen0.3: at usbus0=0A>> uhub4: addr 3>=0A>> on usbus0=0A>> SMP: AP CPU #1 Launched!=0A>= > SMP: AP CPU #7 Launched!=0A>> SMP: AP CPU #3 Launched!=0A>> SMP: AP CPU #= 6 Launched!=0A>> SMP: AP CPU #2 Launched!=0A>> SMP: AP CPU #5 Launched!=0A>= > SMP: AP CPU #4 Launched!=0A>> Timecounter "TSC-low" frequency 9765827 Hz = quality 1000=0A>> Root mount waiting for: usbus0=0A>> uhub4: 6 ports with 6= removable, self powered=0A>> ugen0.4: at usbus0=0A>> ukbd0: on usbus0=0A>> kbd0 at ukbd0=0A>> Root mount waiting for: usbus0=0A>= > Root mount waiting for: usbus0=0A>> ugen0.5: at usbus0=0A>> uma= ss0: on usbus0=0A>> Trying to mount root fro= m ufs:/dev/mfid0p3 [rw]...=0A>> cd0 at umass-sim0 bus 0 scbus1 target 0 lun= 0=0A>> cd0: Removable CD-ROM SCSI-0 device=0A>> cd= 0: 40.000MB/s transfers=0A>> cd0: Attempt to query device size failed: NOT = READY, Medium not=0A>present=0A>> da0 at umass-sim0 bus 0 scbus1 target 0 l= un 1=0A>> da0: Removable Direct Access SCSI-0= =0A>device=0A>> da0: 40.000MB/s transfers=0A>> da0: Attempt to query device= size failed: NOT READY, Medium not=0A>present=0A>> ZFS filesystem version = 5=0A>> ZFS storage pool version 28=0A>> ums0: on usbus0=0A>> ums0: = 3 buttons and [Z] coordinates ID=3D0=0A>> ums1: on usbus0=0A>> = ums1: 3 buttons and [XYZ] coordinates ID=3D0=0A>> igb0: link state changed = to UP=0A>>=0A>> With this loader.conf:=0A>> vm.kmem_size_max=3D"64G"=0A>> v= m.kmem_size=3D"48G"=0A>> vfs.zfs.arc_min=3D"4096M"=0A>> vfs.zfs.arc_max=3D"= 15872M"=0A>> vfs.zfs.txg.synctime=3D"1"=0A>> vfs.zfs.txg.timeout=3D"5"=0A>>= vfs.zfs.write_limit_override=3D"**0"=0A>> vfs.zfs.txg.write_limit_**overri= de=3D"0"=0A>> vfs.zfs.zil_disable=3D"0"=0A>>=0A>> And with this sysctl.conf= :=0A>> security.bsd.see_other_uids=3D0=0A>> net.inet.tcp.blackhole=3D2=0A>>= net.inet.udp.blackhole=3D1=0A>> net.inet.tcp.sendspace=3D65536=0A>> net.in= et.tcp.recvspace=3D65536=0A>> net.link.ether.inet.log_arp_**wrong_iface=3D0= =0A>> kern.ipc.somaxconn=3D32768=0A>> kern.threads.max_threads_per_**proc= =3D16384=0A>> kern.maxfiles=3D262144=0A>> kern.maxfilesperproc=3D131072=0A>= > kern.ipc.nmbclusters=3D65536=0A>> kern.corefile=3D"/var/coredumps/**%U.%N= .%P.core"=0A>>=0A>> vfs.zfs.txg.timeout=3D5=0A>> vfs.zfs.l2arc_headroom=3D1= 6=0A>> vfs.zfs.l2arc_write_max=3D**26843546=0A>> vfs.zfs.l2arc_write_boost= =3D**26843546=0A>> vfs.zfs.vdev.min_pending=3D4=0A>> vfs.zfs.vdev.max_pendi= ng=3D64=0A>> vfs.zfs.l2arc_noprefetch=3D1=0A>> vfs.zfs.dedup.prefetch=3D0= =0A>> vfs.zfs.prefetch_disable=3D1=0A>>=0A>> I will use this server as a my= sql-5.6 dbserver so I have a root=0A>partition=0A>> using a hw raid1 and a = /DATAZFS partition, follows configuration:=0A>>=0A>> root# mfiutil show dri= ves=0A>> mfi0 Physical Drives:=0A>>=C2=A0 0 (=C2=A0 279G) ONLINE SAS E1:S0=0A>>=C2=A0 1 (=C2=A0 279G) ONLINE SAS E1:S1=0A>>=C2=A0 2 (=C2=A0 558G) ONLINE SAS E1:S2=0A>>=C2=A0 3 (=C2=A0 558G) ONLINE SAS E1:S3=0A>>=C2=A0 4 (=C2=A0 558G) ONLINE = SAS E1:S4=0A>>=C2=A0 5 (=C2=A0 558G) ONLINE SAS= E1:S5=0A>>=C2=A0 6 (=C2=A0 119G) ONLINE SATA E1:S6=0A>>= =C2=A0 7 (=C2=A0 119G) ONLINE SATA E1:S7=0A>> root# mfiut= il show volumes=0A>> mfi0 Volumes:=0A>>=C2=A0 Id=C2=A0 =C2=A0 Size=C2=A0 = =C2=A0 Level=C2=A0 Stripe=C2=A0 State=C2=A0 Cache=C2=A0 Name=0A>>=C2=A0 = mfid0 (=C2=A0 278G) RAID-1=C2=A0 =C2=A0 =C2=A0 64k OPTIMAL Disabled = =0A>>=C2=A0 mfid1 (=C2=A0 118G) RAID-0=C2=A0 =C2=A0 =C2=A0 64k OPTIMAL Disa= bled =0A>>=C2=A0 mfid2 (=C2=A0 118G) RAID-0=C2=A0 =C2=A0 =C2=A0 64k= OPTIMAL Disabled =0A>>=C2=A0 mfid3 ( 1116G) RAID-1=C2=A0 =C2=A0 =C2= =A0 64k OPTIMAL Disabled =0A>>=0A>> I have a /DATAZFS partition, too:= =0A>> root# zfs get all DATAZFS=0A>> NAME=C2=A0 =C2=A0 PROPERTY=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 VALUE=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 SOURCE=0A>> DATAZFS=C2=A0 type=C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 filesystem=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS=C2=A0 creation=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Thu Mar 21 10:45 2013=C2=A0 -=0A>> DATAZFS= =C2=A0 used=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 5= 4.2G=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> D= ATAZFS=C2=A0 available=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1.02T=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS= =C2=A0 referenced=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 54.2G=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS=C2=A0 c= ompressratio=C2=A0 =C2=A0 =C2=A0 =C2=A0 1.00x=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS=C2=A0 mounted=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 yes=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS=C2=A0 quota=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 none=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 reservat= ion=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 none=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 recordsize=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 16K=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 local=0A>> DATAZFS=C2=A0 mountpoint= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 /DATAZFS=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 sharenfs=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 checksum=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on=C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 comp= ression=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 atime= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 local=0A>> DATAZFS= =C2=A0 devices=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>= > DATAZFS=C2=A0 exec=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 on=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 default=0A>> DATAZFS=C2=A0 setuid=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 on=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 readonly=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 jailed=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 sn= apdir=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 hidden=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 aclmo= de=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 discard=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 aclinh= erit=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 restricted=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 canmount=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 on=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 xattr=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 temporary=0A>> DATAZFS=C2=A0 = copies=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 1=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default= =0A>> DATAZFS=C2=A0 version=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= 5=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 -=0A>> DATAZFS=C2=A0 utf8only=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 off=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 -=0A>> DATAZFS=C2=A0 normalization=C2=A0 =C2=A0 =C2=A0 =C2=A0 n= one=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> D= ATAZFS=C2=A0 casesensitivity=C2=A0 =C2=A0 =C2=A0 sensitive=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS=C2=A0 vscan=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0 nbmand= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZF= S=C2=A0 sharesmb=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>>= DATAZFS=C2=A0 refquota=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 non= e=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A= >> DATAZFS=C2=A0 refreservation=C2=A0 =C2=A0 =C2=A0 =C2=A0 none=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS= =C2=A0 primarycache=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 all=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS= =C2=A0 secondarycache=C2=A0 =C2=A0 =C2=A0 =C2=A0 all=C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS=C2=A0= usedbysnapshots=C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS=C2=A0 usedbydatase= t=C2=A0 =C2=A0 =C2=A0 =C2=A0 54.2G=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS=C2=A0 usedbychildren=C2=A0 =C2=A0 = =C2=A0 =C2=A0 1.41M=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 -=0A>> DATAZFS=C2=A0 usedbyrefreservation=C2=A0 0=C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS= =C2=A0 logbias=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 latency=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 default=0A>> DATAZFS= =C2=A0 dedup=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 off=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 local=0A= >> DATAZFS=C2=A0 mlslabel=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 -=0A>> DATAZFS=C2=A0 sync=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 standard=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = default=0A>> DATAZFS=C2=A0 refcompressratio=C2=A0 =C2=A0 =C2=A0 1.00x=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>> DATAZFS= =C2=A0 written=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 54.2G=C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 -=0A>>=0A>> root# = zpool status -v=0A>>=C2=A0 pool: DATAZFS=0A>>=C2=A0 state: ONLINE=0A>>=C2= =A0 scan: none requested=0A>> config:=0A>>=0A>> NAME=C2=A0 =C2=A0 =C2=A0 = =C2=A0 STATE=C2=A0 =C2=A0 READ WRITE CKSUM=0A>> DATAZFS=C2=A0 =C2=A0 ONLI= NE=C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 0=C2=A0 =C2=A0 0=0A>>=C2=A0 mfid3= =C2=A0 =C2=A0 ONLINE=C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 0=C2=A0 =C2=A0 = 0=0A>> logs=0A>>=C2=A0 mfid2=C2=A0 =C2=A0 ONLINE=C2=A0 =C2=A0 =C2=A0 0= =C2=A0 =C2=A0 0=C2=A0 =C2=A0 0=0A>> cache=0A>>=C2=A0 mfid1=C2=A0 =C2=A0 = ONLINE=C2=A0 =C2=A0 =C2=A0 0=C2=A0 =C2=A0 0=C2=A0 =C2=A0 0=0A>>=0A>> er= rors: No known data errors=0A>>=0A>> root# mfiutil show firmware=0A>> mfi0 = Firmware Package Version: 21.1.0-0007=0A>> mfi0 Firmware Images:=0A>> Name= =C2=A0 Version=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 Date=C2=A0 =C2=A0 =C2=A0 =C2=A0 Time=C2=A0 =C2=A0= =C2=A0 =C2=A0 Status=0A>> BIOS=C2=A0 5.31.01_4.12.05.00_0x05110000=C2=A0 = 6/20/2012=0A>>=C2=A0 =C2=A0 6/20/2012=0A>>=C2=A0 active=0A>> CTLR=C2=A0 4= .00-0014=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 Aug 04 2011=C2=A0 12:49:17=C2=A0 =C2=A0 active=0A>> PCLI=C2=A0 = 05.00-03:#%00008=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Feb 17 20= 11=C2=A0 14:03:12=C2=A0 =C2=A0 active=0A>> APP=C2=A0 3.130.05-1796=C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Jul 27 2012=C2=A0 1= 5:58:31=C2=A0 =C2=A0 active=0A>> NVDT=C2=A0 2.1108.03-0094=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Jun 22 2012=C2=A0 12:40:55=C2=A0= =C2=A0 active=0A>> BTBL=C2=A0 2.03.00.00-0004=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 Jul 27 2012=C2=A0 16:39:29=C2=A0 =C2=A0 active= =0A>> BOOT=C2=A0 06.253.57.219=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 9/9/2010=C2=A0 =C2=A0 15:32:25=C2=A0 =C2=A0 active=0A>>= root#=0A>>=0A>> Then, I installed CentOS on the same hardware, same versio= n of mysql=0A>> (5.6.10-enterprise and 5.6.10-GA), and I tried an OLTP test= (included=0A>in=0A>> sysbench-0.5) executing 2000000 queries on 10000000 r= ows, with these=0A>> results (transactions per seconds):=0A>>=0A>> BSD 9.1 = amd64 ZFS (without tweaks):=0A>> 1 thread: 2780=0A>> 2 threads: 4371=0A>> 4= threads: 6876=0A>> 8 threads: 8202=0A>> 12 threads: 8077=0A>> 16 threads: = 7780=0A>> 24 threads: 7563=0A>> 32 threads: 7632=0A>> 48 threads: 7960=0A>>= 64 threads: 8062=0A>>=0A>> BSD 9.1 amd64 ZFS (with tweaks):=0A>> 1 thread:= 2589=0A>> 2 threads: 4679=0A>> 4 threads: 6438=0A>> 8 threads: 7073=0A>> 1= 2 threads: 7121=0A>> 16 threads: 7227=0A>> 24 threads: 7132=0A>> 32 threads= : 7273=0A>> 48 threads: 7623=0A>> 64 threads: 7672=0A>>=0A>> CentOS 6.3 (ex= t4, without tweaks):=0A>> 1 thread: 4163=0A>> 2 threads: 7653=0A>> 4 thread= s: 10905=0A>> 8 threads: 12511=0A>> 12 threads: 13556=0A>> 16 threads: 1483= 2=0A>> 24 threads: 16270=0A>> 32 threads: 16733=0A>> 48 threads: 16925=0A>>= 64 threads: 16895=0A>>=0A>>=0A>> Well, any of you could ask "why zfs, plea= se try ufs that is better":=0A>you=0A>> are right, but changing ZFS with UF= S doesn't change so much:=0A>>=0A>http://lists.freebsd.org/**pipermail/free= bsd-fs/2013-**March/016747.html=0A>>=0A>> So I hope I'm missing something becaus= e the performances of a=0A>standard=0A>> CentOS installation on the same ha= rdware performs much better than a=0A>9.1=0A>> amd64 installation (I'm usin= g the standard ULE scheduler included in=0A>the=0A>> 9.1 release).=0A>>=0A>= > Could anyone help me?=0A>>=0A>> Thanks,=0A>> d.=0A>> ____________________= __________**_________________=0A>> freebsd-performance@freebsd.**org=0A>mailing list=0A>>=0A>http://lists.freebsd.org= /**mailman/listinfo/freebsd-**performance=0A>> To unsubscribe, send any mail to "freebs= d-performance-**=0A>> unsubscribe@freebsd.org=0A>"=0A>>=0A=0A--=0Ad.=0A__________________________________= _____________=0Afreebsd-performance@freebsd.org mailing list=0Ahttp://lists= .freebsd.org/mailman/listinfo/freebsd-performance=0ATo unsubscribe, send an= y mail to "freebsd-performance-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Sat Mar 23 07:44:34 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8632261A for ; Sat, 23 Mar 2013 07:44:34 +0000 (UTC) (envelope-from davide.damico@contactlab.com) Received: from mail2.shared.smtp.contactlab.it (mail2.shared.smtp.contactlab.it [93.94.37.7]) by mx1.freebsd.org (Postfix) with ESMTP id 14A42979 for ; Sat, 23 Mar 2013 07:44:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; d=contactlab.it; s=clab1; c=relaxed/relaxed; q=dns/txt; i=@contactlab.it; t=1364024671; h=From:Subject:Date:To:MIME-Version:Content-Type; bh=sVP3v1AW07tk+cpo0iIlPaC3P7T3DOUwyUlqA589++k=; b=VHxXWhW9/BZ619E3u+b6/OD4A5E8O2kPCaU1KKKVQT3jY5LxuovJ3eH4cFQ6kAbO Ina/P4HYUs7OmJlTl0tBWAgaTU6TqFO6+aUOZ3arVqlGrqGNv8pA5ZsX/SDPEGnc OsGhAzw078HrPOFG+i7ptFGYWzEzQ4vWwt5yWoFyj+E=; Received: from [213.92.90.12] ([213.92.90.12:26194] helo=mail3.tomato.it) by t.contactlab.it (envelope-from ) (ecelerity 3.5.1.37854 r(Momo-dev:3.5.1.0)) with ESMTP id FB/91-24145-F5D5D415; Sat, 23 Mar 2013 08:44:31 +0100 Received: from mx3-master.housing.tomato.lan ([172.16.7.55]) by mail3.tomato.it with smtp (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1UJJ7v-000Pn0-CO for freebsd-performance@freebsd.org; Sat, 23 Mar 2013 08:44:31 +0100 Received: (qmail 99132 invoked by uid 80); 23 Mar 2013 07:44:31 -0000 To: Paul Pathiakis Subject: Re: FreeBSD 9.1 vs CentOS 6.3 X-PHP-Script: uebmeil.sys.tomatointeractive.it/index.php for 172.16.16.227 X-PHP-Originating-Script: 0:main.inc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 23 Mar 2013 08:44:31 +0100 From: Davide D'Amico Organization: ContactLab Mail-Reply-To: In-Reply-To: <1363998883.22604.YahooMailNeo@web141401.mail.bf1.yahoo.com> References: <1363998883.22604.YahooMailNeo@web141401.mail.bf1.yahoo.com> Message-ID: <5d10e2a0b0f6477293459a26df1fc272@sys.tomatointeractive.it> X-Sender: davide.damico@contactlab.com User-Agent: Roundcube Webmail/0.8.5 Cc: freebsd-performance@freebsd.org, =?UTF-8?Q?=D0=95=D0=B2=D0=B3=D0=B5=D0=BD=D0=B8=D0=B9_=D0=A5=D0=BE?= =?UTF-8?Q?=D1=80=D0=BA=D0=B8=D0=BD?= X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: davide.damico@contactlab.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Mar 2013 07:44:34 -0000 Il 23.03.2013 01:34 Paul Pathiakis ha scritto: > Hi, > > There are several things about this that are highly suspect. > > First, wipe out the hardware RAID. The processor doing RAID > computation is, probably, MUCH slower than a core on the CPU. Even if > it's RAID-1 (Simple Mirror) this RAID card is performing tasks that is > does not need to do including replicating writes to two targets from > the controller or checking it's cache, battery, etc. If it's possible > to disable the onboard cache, do it. Hi Paul, thanks for your suggestions (some of them I've applied before starting any consideration, like disabling all on-disk caches or controller buffers) I'll try next monday. Anyway, the fact is that using the same hardware configuration (raid1+raid10) I saw that a centos 6.x outperformed freebsd 9.1. Another test I made yesterday was: on the same hardware I installed vmware esx 5.x and created a vm with centos inside it. The result was really impressive: the centos vm outperformed the 'real' freebsd 9.1 too and checking vmware performances graphs I didn't see any huge need for a massive throughput (I saw values from KBps to 10MBps), instead I saw a big use of CPU (using OLTP tests with a concurrency of 32 threads it's performaces began to slow down). I don't know is using some magic value for HZ or setting some trick with scheduler, I could gain something: I hope so, because I don't want to "pinguinate" my farm :) Thanks, d. From owner-freebsd-performance@FreeBSD.ORG Sat Mar 23 20:35:20 2013 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6090FAEA for ; Sat, 23 Mar 2013 20:35:20 +0000 (UTC) (envelope-from daniel.bilik@neosystem.cz) Received: from mail.neosystem.cz (mail.neosystem.cz [IPv6:2001:41d0:2:5ab8::10:15]) by mx1.freebsd.org (Postfix) with ESMTP id 28D086A3 for ; Sat, 23 Mar 2013 20:35:19 +0000 (UTC) Received: from mail.neosystem.cz (unknown [127.0.10.15]) by mail.neosystem.cz (Postfix) with ESMTP id 0A8F0C9F7 for ; Sat, 23 Mar 2013 21:35:11 +0100 (CET) X-Virus-Scanned: amavisd-new at mail.neosystem.cz Received: from neon.sn.neosystem.cz (unknown [172.19.9.2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.neosystem.cz (Postfix) with ESMTPSA id 26DB0C9EF for ; Sat, 23 Mar 2013 21:35:02 +0100 (CET) Date: Sat, 23 Mar 2013 21:34:06 +0100 From: Daniel Bilik To: freebsd-performance@freebsd.org Subject: Re: FreeBSD 9.1 vs CentOS 6.3 Message-Id: <20130323213406.93cc3baddf69d5d71f10365e@neosystem.cz> In-Reply-To: <514C1E5F.8040504@contactlab.com> References: <514C1E5F.8040504@contactlab.com> Organization: neosystem.cz X-Mailer: Sylpheed 3.3.0 (GTK+ 2.24.17; amd64-portbld-freebsd9.1) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sat, 23 Mar 2013 21:28:26 +0000 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 23 Mar 2013 20:35:20 -0000 On Fri, 22 Mar 2013 10:03:27 +0100 Davide D'Amico wrote: > Hi, I'm doing performance tests on a DELL R720, follows dmesg: > ... > I will use this server as a mysql-5.6 dbserver so I have a root > partition using a hw raid1 and a /DATAZFS partition, follows > configuration: > ... Well, it seems to be interesting coincidence... We've just finished benchmarking MySQL with various (m)allocators. The goal was to test tcmalloc, but when the system was up and running, we've taken the opportunity to benchmark also other alternatives... including jemalloc. All tests were performed on default MySQL 5.5.28 running on Debian Wheezy. Between the tests nothing was touched on the machine or the system, just allocators were changed (ie. mysqld restarted). Results for different test modes are available here... http://neosystem.cz/benchmark/mysql/ It seems there is notable performance penalty for read-only transactions when MySQL is using jemalloc. The more concurrent threads are running, the more is jemalloc losing to other allocators. The penalty is also there for read-write transactions, but not that significant (error bars in the histograms also show that results for read-write tests tend to be very unstable). OTOH in non-transactional tests, jemalloc seems to be in par with others, and under specific load can even outperform some of them. In your original post, there is not mentioned in what mode you've performed OLTP test, but according to numbers I suspect it was "complex", ie. transactional. Can you repeat tests (both on CentOS and FreeBSD) with --oltp-test-mode=nontrx and/or simple? -- Daniel Bilik neosystem.cz