From owner-freebsd-performance@FreeBSD.ORG Mon Oct 5 09:02:53 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6C7A61065692 for ; Mon, 5 Oct 2009 09:02:53 +0000 (UTC) (envelope-from serenity@exscape.org) Received: from ch-smtp01.sth.basefarm.net (ch-smtp01.sth.basefarm.net [80.76.149.212]) by mx1.freebsd.org (Postfix) with ESMTP id C20B78FC2D for ; Mon, 5 Oct 2009 09:02:52 +0000 (UTC) Received: from c83-253-248-99.bredband.comhem.se ([83.253.248.99]:49706 helo=mx.exscape.org) by ch-smtp01.sth.basefarm.net with esmtp (Exim 4.68) (envelope-from ) id 1MujCb-0006lh-3X for freebsd-performance@freebsd.org; Mon, 05 Oct 2009 10:45:56 +0200 Received: from [192.168.1.5] (macbookpro [192.168.1.5]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx.exscape.org (Postfix) with ESMTPSA id 1E0AEFDA2A for ; Mon, 5 Oct 2009 10:45:51 +0200 (CEST) From: Thomas Backman Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Date: Mon, 5 Oct 2009 10:45:48 +0200 Message-Id: To: freebsd-performance@freebsd.org Mime-Version: 1.0 (Apple Message framework v1076) X-Mailer: Apple Mail (2.1076) X-Originating-IP: 83.253.248.99 X-Scan-Result: No virus found in message 1MujCb-0006lh-3X. X-Scan-Signature: ch-smtp01.sth.basefarm.net 1MujCb-0006lh-3X d280975608bf822686d4e8d84ca40cb6 Subject: Re: A specific example of a disk i/o problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 09:02:53 -0000 (Note: I hope this reply shows up correctly. I subscribed to the mailing list after the fact and had to "forge" the subject line.) Hey everyone, I'm having serious trouble with the same thing, and just found this thread while looking for the correct place to post. Looks like I found it. (I wrote most of the post before finding the thread, so some of it will seem a bit odd.) I run 8.0-RC1/amd64 with ZFS on an A64 3200+ with 2GB RAM and an old 80GB 7200rpm disk. My problem is that I get completely unacceptable latency on console IO (both via SSH and serial console) when the system is performing disk IO. The worst case I've noticed yet was when I tried copying a core dump from a lzjb compressed ZFS file system to a gzip-9 compressed one, to compare the file size/compression ratio. screen(1) took at LEAST ten seconds - probably a bit more - I'm not exaggerating here - to switch to another window, and an "ls" in an empty directory also about 5-10 seconds. Doing some silly CPU load with two instances of "yes >/dev/null" (on a single core, remember) doesn't change anything, the system remains very responsive. "cat /dev/zero > /uncompressed_fs/..." however produces the extreme slowdown. (On a gzip-1 FS it doesn't, since the file ends up extremely small - a kilobyte or so - even after a while, thus performing minimal IO). I'm thinking about switching to FreeBSD on my beefier "production" system (dual-core amd64, 4GB RAM, 4x1TB disks, compared to this one, single-core, 2GB RAM, 80GB disk), but unless I feel assured this won't happen there as well, I'm not so sure anymore. I can do any kind of heavy IO/compilation/whatever on that box, currently running Linux, and it's always unnoticable. In this case it's impossible *not* to notice that your key input is lagging behind 5-10 seconds... I thought multiple times that the box must have panicked. I do realize that the hardware isn't the best, especially the disks, but this is far worse than it should be! Here's some of the testing done in this thread (or at least something like it): [root@chaos ~]# time file /etc/* >/dev/null real 0m1.725s user 0m0.993s sys 0m0.021s [root@chaos ~]# time file /etc/* >/dev/null real 0m1.008s user 0m0.990s sys 0m0.015s [root@chaos ~]# time file /etc/* >/dev/null real 0m1.008s user 0m0.967s sys 0m0.038s [root@chaos ~]# time file /etc/* >/dev/null real 0m1.015s user 0m0.998s sys 0m0.008s So, pretty much exactly 1 second every time once the cache is warmed up. Now, let's try it 10 seconds after starting heavy disk writing... [root@chaos ~]# cat /dev/zero > /DELETE_ME & (wait for 10 seconds) [root@chaos ~]# time file /etc/* >/dev/null real 0m13.217s user 0m1.004s sys 0m0.023s [root@chaos ~]# time file /etc/* >/dev/null real 0m24.012s user 0m1.020s sys 0m0.008s ... the numbers speak for themselves. FWIW I tried the same on the Linux box: "file" took 0.8 seconds the first time, 0.125s subsequent times. While running cat, 0.13-0.21 seconds (0.21 being the first, subsequent runs took 0.13s). The system remained completely responsive, which cannot be said for the FreeBSD one! Any advice? Info below - please ask if you need more! Regards, Thomas ----------------------------------------------------------------------------- Basic info: [root@chaos ~]# uname -a FreeBSD chaos.exscape.org 8.0-RC1 FreeBSD 8.0-RC1 #1 r197613M: Tue Sep 29 15:04:44 CEST 2009 root@chaos.exscape.org:/usr/obj/usr/src/sys/ DTRACE amd64 (KDB/DDB enabled, WITNESS/INVARIANTS *disabled*) [root@chaos ~]# mount tank/root on / (zfs, local, noatime) devfs on /dev (devfs, local, multilabel) /dev/ad0s1a on /bootdir (ufs, local, soft-updates) tank/export on /export (zfs, NFS exported, local, noatime) tank/tmp on /tmp (zfs, local, noatime) tank/usr on /usr (zfs, local, noatime) tank/usr/obj on /usr/obj (zfs, local, noatime) tank/usr/ports on /usr/ports (zfs, local, noatime) tank/usr/ports/distfiles on /usr/ports/distfiles (zfs, local, noatime) tank/usr/src on /usr/src (zfs, local, noatime) tank/usr/src_r196905 on /usr/src_r196905 (zfs, local, noatime, read- only) tank/var on /var (zfs, local, noatime) tank/var/crash on /var/crash (zfs, local, noatime) tank/var/log on /var/log (zfs, local, noatime) tank/var/tmp on /var/tmp (zfs, local, noatime) dmesg: ----------------------------------------------------------------------------- Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-RC1 #1 r197613M: Tue Sep 29 15:04:44 CEST 2009 root@chaos.exscape.org:/usr/obj/usr/src/sys/DTRACE Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Athlon(tm) 64 Processor 3200+ (2009.27-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x10ff0 Stepping = 0 Features = 0x78bfbff < FPU ,VME ,DE ,PSE ,TSC ,MSR ,PAE ,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2> AMD Features=0xe2500800 AMD Features2=0x1 real memory = 2147483648 (2048 MB) avail memory = 2052362240 (1957 MB) ACPI APIC Table: ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, 7fef0000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pci0: at device 0.0 (no driver attached) isab0: at device 1.0 on pci0 isa0: on isab0 pci0: at device 1.1 (no driver attached) ohci0: mem 0xfe02f000-0xfe02ffff irq 21 at device 2.0 on pci0 ohci0: [ITHREAD] usbus0: on ohci0 ehci0: mem 0xfe02e000-0xfe02e0ff irq 22 at device 2.1 on pci0 ehci0: [ITHREAD] usbus1: EHCI version 1.0 usbus1: on ehci0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfb00-0xfb0f at device 6.0 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] atapci1: port 0x9f0-0x9f7,0xbf0-0xbf3,0x970-0x977,0xb70-0xb73,0xf600-0xf60f mem 0xfe02b000-0xfe02bfff irq 23 at device 7.0 on pci0 atapci1: [ITHREAD] ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] atapci2: port 0x9e0-0x9e7,0xbe0-0xbe3,0x960-0x967,0xb60-0xb63,0xf100-0xf10f mem 0xfe02a000-0xfe02afff irq 21 at device 8.0 on pci0 atapci2: [ITHREAD] ata4: on atapci2 ata4: [ITHREAD] ata5: on atapci2 ata5: [ITHREAD] pcib1: at device 9.0 on pci0 pci1: on pcib1 vgapci0: mem 0xfcff8000-0xfcffbfff, 0xfd000000-0xfd7fffff,0xfc000000-0xfc7fffff irq 17 at device 7.0 on pci1 xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xdf00-0xdf7f mem 0xfcfff000-0xfcfff07f irq 18 at device 9.0 on pci1 miibus0: on xl0 xlphy0: <3c905C 10/100 internal PHY> PHY 24 on miibus0 xlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto xl0: Ethernet address: 00:50:da:44:c0:4a xl0: [ITHREAD] nfe0: port 0xf000-0xf007 mem 0xfe029000-0xfe029fff irq 22 at device 10.0 on pci0 miibus1: on nfe0 e1000phy0: PHY 1 on miibus1 e1000phy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto nfe0: Ethernet address: 00:13:d3:a2:aa:0f nfe0: [FILTER] pcib2: at device 11.0 on pci0 pci2: on pcib2 pcib3: at device 12.0 on pci0 pci3: on pcib3 pcib4: at device 13.0 on pci0 pci4: on pcib4 pcib5: at device 14.0 on pci0 pci5: on pcib5 amdtemp0: on hostb3 acpi_tz0: on acpi0 atrtc0: port 0x70-0x73 irq 8 on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart0: console (115200,n,8,1) atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] cpu0: on acpi0 powernow0: on cpu0 device_attach: powernow0 attach returned 6 orm0: at iomem 0xc0000-0xc7fff,0xc8000-0xcbfff, 0xcc000-0xcc7ff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ZFS NOTICE: system has less than 4GB and prefetch enable is not set... disabling. ZFS filesystem version 13 ZFS storage pool version 13 Timecounter "TSC" frequency 2009269513 Hz quality 800 Timecounters tick every 1.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 480Mbps High Speed USB v2.0 ad0: 76318MB at ata0-master UDMA100 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ad2: 9768MB at ata1-master UDMA100 GEOM: ad2s1: geometry does not match label (255h,63s != 16h,63s). Root mount waiting for: usbus1 usbus0 uhub0: 10 ports with 10 removable, self powered Root mount waiting for: usbus1 Root mount waiting for: usbus1 Root mount waiting for: usbus1 uhub1: 10 ports with 10 removable, self powered Trying to mount root from zfs:tank/root netsmb_dev: loaded ----------------------------------------------------------------------------- The 80GB disk is used for everything except swap (aka. dump device) for which the 10GB disk is used, exclusively. loader.conf has nothing of value (just loading a few modules and a vfs.root.mountfrom, plus serial console settings). From owner-freebsd-performance@FreeBSD.ORG Mon Oct 5 10:45:53 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 178961065670 for ; Mon, 5 Oct 2009 10:45:53 +0000 (UTC) (envelope-from vilmos.gyorgy@gmail.com) Received: from mail-ew0-f209.google.com (mail-ew0-f209.google.com [209.85.219.209]) by mx1.freebsd.org (Postfix) with ESMTP id A32928FC18 for ; Mon, 5 Oct 2009 10:45:51 +0000 (UTC) Received: by ewy5 with SMTP id 5so2018859ewy.36 for ; Mon, 05 Oct 2009 03:45:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=ALqxerBQXSQkLWI7PFDIVWWiXDiB2k1124V1A+kTE6Y=; b=CoTnJj1mHeyQwC2jrKGiXYZIeN07ZGhWx8Gerc0RO0O4OQxw4cXdZ76v/llOLsKfDu 9yNwN4bvOnywk5/gSdt58tNoVQFaWZazwa0ucTtJSWxXl72hraN1/C48iLiSzn/I29RN JVapOZQBRwIJdwKSbn1B9wKPNUEAEhj3g6l/s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=sk1Usqsdi2Wd+0w355c+2ZalJGUZkQjxnL2lc/1BcaqxHqoyyi8gujr4LGg9QtD5jB kQNjekbL8L+xBP9E8VegfzzU9ELK7fHmHG/lJSVhQv6cckaZZU7xvQMPXssJqChdY/+d Kyg52BS9YerPVRglvqgZAvz6mRDBZZo83XZwI= MIME-Version: 1.0 Received: by 10.216.17.145 with SMTP id j17mr1020192wej.2.1254739550666; Mon, 05 Oct 2009 03:45:50 -0700 (PDT) Date: Mon, 5 Oct 2009 12:45:50 +0200 Message-ID: From: =?ISO-8859-1?Q?Gy=F6rgy_Vilmos?= To: freebsd-performance X-Mailman-Approved-At: Mon, 05 Oct 2009 11:41:48 +0000 Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: MySQL and PostgreSQL benchmarks, FreeBSD 7 vs. 8 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 10:45:53 -0000 Hello, I have two new articles: First, MySQL history, which takes some older (major) versions from MySQL and shows their performance on FreeBSD 8: http://suckit.blog.hu/2009/10/03/mysql_history and an article, which compares PostgreSQL and MySQL performance between FreeBSD 7 and 8: http://suckit.blog.hu/2009/10/05/freebsd_8_is_it_worth_to_upgrade -- http://suckit.blog.hu/ From owner-freebsd-performance@FreeBSD.ORG Mon Oct 5 14:48:55 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1A0751065786 for ; Mon, 5 Oct 2009 14:48:55 +0000 (UTC) (envelope-from pieter@degoeje.nl) Received: from mx.utwente.nl (mx1.utsp.utwente.nl [130.89.2.12]) by mx1.freebsd.org (Postfix) with ESMTP id 9111C8FC12 for ; Mon, 5 Oct 2009 14:48:54 +0000 (UTC) Received: from nox-laptop.student.utwente.nl (nox-laptop.student.utwente.nl [130.89.170.109]) by mx.utwente.nl (8.12.10/SuSE Linux 0.7) with ESMTP id n95EF0eR008000 for ; Mon, 5 Oct 2009 16:15:00 +0200 From: Pieter de Goeje To: freebsd-performance@freebsd.org Date: Mon, 5 Oct 2009 16:14:59 +0200 User-Agent: KMail/1.9.10 References: In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Message-Id: <200910051615.00313.pieter@degoeje.nl> X-UTwente-MailScanner-Information: Scanned by MailScanner. Contact icts.servicedesk@utwente.nl for more information. X-UTwente-MailScanner: Found to be clean X-UTwente-MailScanner-From: pieter@degoeje.nl X-Spam-Status: No Subject: Re: MySQL and PostgreSQL benchmarks, FreeBSD 7 vs. 8 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Oct 2009 14:48:55 -0000 On Monday 05 October 2009 12:45:50 Gy=F6rgy Vilmos wrote: > Hello, > > I have two new articles: > First, MySQL history, which takes some older (major) versions from MySQL > and shows their performance on FreeBSD 8: > http://suckit.blog.hu/2009/10/03/mysql_history > > and an article, which compares PostgreSQL and MySQL performance between > FreeBSD 7 and 8: > http://suckit.blog.hu/2009/10/05/freebsd_8_is_it_worth_to_upgrade Very interesting. Good to see 8 is performing better than 7 :-) Did you see any performance change after implementing the correct cpu=20 topology? =2D-=20 Pieter de Goeje From owner-freebsd-performance@FreeBSD.ORG Tue Oct 6 04:15:35 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 92757106568F for ; Tue, 6 Oct 2009 04:15:35 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from sopwith.solgatos.com (pool-173-50-131-36.ptldor.fios.verizon.net [173.50.131.36]) by mx1.freebsd.org (Postfix) with ESMTP id 550568FC20 for ; Tue, 6 Oct 2009 04:15:34 +0000 (UTC) Received: by sopwith.solgatos.com (Postfix, from userid 66) id 726C8B64F; Mon, 5 Oct 2009 21:11:21 -0700 (PDT) Received: from localhost by sopwith.solgatos.com (8.8.8/6.24) id QAA06722; Mon, 5 Oct 2009 16:05:49 GMT Message-Id: <200910051605.QAA06722@sopwith.solgatos.com> To: freebsd-performance@freebsd.org In-reply-to: Your message of "Mon, 05 Oct 2009 10:45:48 +0200." Date: Mon, 05 Oct 2009 09:05:49 PDT From: Dieter Subject: ZFS Re: A specific example of a disk i/o problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 04:15:35 -0000 In message , Thomas Backman writes: > I run 8.0-RC1/amd64 with ZFS on an A64 3200+ with 2GB RAM and an old > 80GB 7200rpm disk. > > My problem is that I get completely unacceptable latency on console IO > (both via SSH and serial console) when the system is performing disk > IO. The worst case I've noticed yet was when I tried copying a core > dump from a lzjb compressed ZFS file system to a gzip-9 compressed > one, to compare the file size/compression ratio. screen(1) took at > LEAST ten seconds - probably a bit more - I'm not exaggerating here - > to switch to another window, and an "ls" in an empty directory also > about 5-10 seconds. You might find the "RELENG_7 heavy disk = system crawls" thread interesting: } I didn't actually solve it or do anything. } I just upgraded to RELENG_8. } } Now it's behaving more like FreeBSD should. } I can do sequential reads/writes and still } use kbd/mouse/X11/buildworld and so on. From owner-freebsd-performance@FreeBSD.ORG Tue Oct 6 04:15:35 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 93B6C1065694 for ; Tue, 6 Oct 2009 04:15:35 +0000 (UTC) (envelope-from freebsd@sopwith.solgatos.com) Received: from sopwith.solgatos.com (pool-173-50-131-36.ptldor.fios.verizon.net [173.50.131.36]) by mx1.freebsd.org (Postfix) with ESMTP id 554658FC21 for ; Tue, 6 Oct 2009 04:15:34 +0000 (UTC) Received: by sopwith.solgatos.com (Postfix, from userid 66) id 7D06BB650; Mon, 5 Oct 2009 21:11:23 -0700 (PDT) Received: from localhost by sopwith.solgatos.com (8.8.8/6.24) id RAA11047; Mon, 5 Oct 2009 17:55:45 GMT Message-Id: <200910051755.RAA11047@sopwith.solgatos.com> To: freebsd-performance@freebsd.org In-reply-to: Your message of "Thu, 01 Oct 2009 09:10:43 EDT." <20091001091043.477f4b9b.wmoran@collaborativefusion.com> Date: Mon, 05 Oct 2009 10:55:45 PDT From: Dieter Subject: tuning FFS for large files Re: A specific example of a disk i/o problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 04:15:35 -0000 I found a clue! The problem occurs with my big data partitions, which are newfs-ed with options intended to improve things. Reading a large file from the normal ad4s5b partition only delays other commands slightly, as expected. Reading a large file from the tuned ad4s11 partition yields the delay of minutes for other i/o. # tunefs -p /dev/ad4s5b tunefs: ACLs: (-a) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: gjournal: (-J) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 1024 tunefs: average file size: (-f) 16384 tunefs: average number of files in a directory: (-s) 64 tunefs: minimum percentage of free space: (-m) 8% tunefs: optimization preference: (-o) time tunefs: volume label: (-L) # tunefs -p /dev/ad4s11 tunefs: ACLs: (-a) disabled tunefs: MAC multilabel: (-l) disabled tunefs: soft updates: (-n) enabled tunefs: gjournal: (-J) disabled tunefs: maximum blocks per file in a cylinder group: (-e) 57984 tunefs: average file size: (-f) 67108864 tunefs: average number of files in a directory: (-s) 16 tunefs: minimum percentage of free space: (-m) 8% tunefs: optimization preference: (-o) time tunefs: volume label: (-L) Here is the newfs command used for creating large data partitions: newfs -e 57984 -b 65536 -f 8192 -g 67108864 -h 16 -i 67108864 -U -o time $partition Even this isn't tuned the way I wanted to. -g * -h must be less than 4 G due to 32 bit problem (system panics). Note the 32 bit problem is in the filesystem code, I'm running amd64. IIRC there is a PR about this. (I'm assuming the bug hasn't been fixed yet) Result is that I must specify -g and -h smaller than they should be. And they have way more inodes than needed. (IIRC it doesn't actually use -i 67108864) Partition sizes range from ~200 GB to 1.5 TB. A small number of directories, perhaps a dozen. About 1/2 the files are small, a few KB, the other half are large, 500MB-25GB. Goals are to minimize seeking and use space efficiently. Files are written sequentially, read mostly (99.99%) sequentially, no editing. I still think it may have something to do with a common resource such as disk cache, as it can affect other disks, including swap. Hmmm, does swap use the disk cache? But now it appears that FFS tuning is the cause of the bottleneck. Do these tuning parameters increase the number of some data structure needed? Can I crank that up? Do I have to change the FFS tuning? Suggestions? From owner-freebsd-performance@FreeBSD.ORG Tue Oct 6 06:56:56 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B1DC7106568B for ; Tue, 6 Oct 2009 06:56:56 +0000 (UTC) (envelope-from serenity@exscape.org) Received: from ch-smtp01.sth.basefarm.net (ch-smtp01.sth.basefarm.net [80.76.149.212]) by mx1.freebsd.org (Postfix) with ESMTP id 6C57A8FC13 for ; Tue, 6 Oct 2009 06:56:56 +0000 (UTC) Received: from c83-253-248-99.bredband.comhem.se ([83.253.248.99]:42699 helo=mx.exscape.org) by ch-smtp01.sth.basefarm.net with esmtp (Exim 4.68) (envelope-from ) id 1Mv3yA-000307-4e; Tue, 06 Oct 2009 08:56:24 +0200 Received: from [192.168.1.5] (macbookpro [192.168.1.5]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx.exscape.org (Postfix) with ESMTPSA id 382D3123492; Tue, 6 Oct 2009 08:55:50 +0200 (CEST) Mime-Version: 1.0 (Apple Message framework v1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes From: Thomas Backman In-Reply-To: <200910051605.QAA06722@sopwith.solgatos.com> Date: Tue, 6 Oct 2009 08:55:48 +0200 Content-Transfer-Encoding: 7bit Message-Id: <88D64C60-D8ED-4622-959C-9946DBB99198@exscape.org> References: <200910051605.QAA06722@sopwith.solgatos.com> To: Dieter X-Mailer: Apple Mail (2.1076) X-Originating-IP: 83.253.248.99 X-Scan-Result: No virus found in message 1Mv3yA-000307-4e. X-Scan-Signature: ch-smtp01.sth.basefarm.net 1Mv3yA-000307-4e 56e6360244b6c40794a6187881e2ff62 Cc: freebsd-performance@freebsd.org Subject: Re: ZFS Re: A specific example of a disk i/o problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 06:56:56 -0000 On Oct 5, 2009, at 6:05 PM, Dieter wrote: > In message , > Thomas Backman writes: > >> I run 8.0-RC1/amd64 with ZFS on an A64 3200+ with 2GB RAM and an old >> 80GB 7200rpm disk. >> >> My problem is that I get completely unacceptable latency on console >> IO >> (both via SSH and serial console) when the system is performing disk >> IO. The worst case I've noticed yet was when I tried copying a core >> dump from a lzjb compressed ZFS file system to a gzip-9 compressed >> one, to compare the file size/compression ratio. screen(1) took at >> LEAST ten seconds - probably a bit more - I'm not exaggerating here - >> to switch to another window, and an "ls" in an empty directory also >> about 5-10 seconds. > > You might find the "RELENG_7 heavy disk = system crawls" thread > interesting: > > } I didn't actually solve it or do anything. > } I just upgraded to RELENG_8. > } > } Now it's behaving more like FreeBSD should. > } I can do sequential reads/writes and still > } use kbd/mouse/X11/buildworld and so on. Doesn't really help much I'm afraid, since 8.0-RC1 = RELENG_8 = stable/ 8. Been running current/stable-8 since May. Regards, Thomas From owner-freebsd-performance@FreeBSD.ORG Tue Oct 6 09:29:18 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 04BB41065672 for ; Tue, 6 Oct 2009 09:29:18 +0000 (UTC) (envelope-from ohartman@zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 842C08FC1D for ; Tue, 6 Oct 2009 09:29:17 +0000 (UTC) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.69) with esmtp (envelope-from ) id <1Mv5rW-0004Qa-MB>; Tue, 06 Oct 2009 10:57:38 +0200 Received: from telesto.geoinf.fu-berlin.de ([130.133.86.198]) by inpost2.zedat.fu-berlin.de (Exim 4.69) with esmtpsa (envelope-from ) id <1Mv5rW-0004Gn-Ke>; Tue, 06 Oct 2009 10:57:38 +0200 Message-ID: <4ACB068E.2070004@zedat.fu-berlin.de> Date: Tue, 06 Oct 2009 08:57:50 +0000 From: "O. Hartmann" Organization: Freie =?ISO-8859-15?Q?Universit=E4t_Berlin?= User-Agent: Thunderbird 2.0.0.23 (X11/20090824) MIME-Version: 1.0 To: Thomas Backman References: <200910051605.QAA06722@sopwith.solgatos.com> <88D64C60-D8ED-4622-959C-9946DBB99198@exscape.org> In-Reply-To: <88D64C60-D8ED-4622-959C-9946DBB99198@exscape.org> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: 130.133.86.198 Cc: Dieter , freebsd-performance@freebsd.org Subject: Re: ZFS Re: A specific example of a disk i/o problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 09:29:18 -0000 Thomas Backman wrote: > On Oct 5, 2009, at 6:05 PM, Dieter wrote: > >> In message , Thomas >> Backman writes: >> >>> I run 8.0-RC1/amd64 with ZFS on an A64 3200+ with 2GB RAM and an old >>> 80GB 7200rpm disk. >>> >>> My problem is that I get completely unacceptable latency on console IO >>> (both via SSH and serial console) when the system is performing disk >>> IO. The worst case I've noticed yet was when I tried copying a core >>> dump from a lzjb compressed ZFS file system to a gzip-9 compressed >>> one, to compare the file size/compression ratio. screen(1) took at >>> LEAST ten seconds - probably a bit more - I'm not exaggerating here - >>> to switch to another window, and an "ls" in an empty directory also >>> about 5-10 seconds. >> >> You might find the "RELENG_7 heavy disk = system crawls" thread >> interesting: >> >> } I didn't actually solve it or do anything. >> } I just upgraded to RELENG_8. >> } >> } Now it's behaving more like FreeBSD should. >> } I can do sequential reads/writes and still >> } use kbd/mouse/X11/buildworld and so on. > > Doesn't really help much I'm afraid, since 8.0-RC1 = RELENG_8 = > stable/8. Been running current/stable-8 since May. > > Regards, > Thomas > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" Is this really ZFS-bound? I also saw (and still see!) massive performance impacts when a lot of disk activities, especially when compiling large packages (done on UFS2 disks). Copying data from one ZFS drive to (on different harddrive) another ZFS drive (which is compressed) doesn't impact performance that much. When the box hit this performance issue, console, X11 and other activities tend to be stuck for minutes! This happens on an UP box (Athlon64 3500+, 2 GB ECC RAM, 2x 250 GB WD SATA-II disks and 1 TB WD SATA-II disk (1x 250 GB UFS2 for system, 1x 250 GB ZFS compressed for backup, 1x 1 TB ZFS holding /home). But this impact is also noticable on my lab's machine: Quad core Intel Q6600 CPU at 3,0 GHZ, 8 GB RAM, 1x 320 GB SATA-II UFS2 disk for system, 2x SATA-II 500 ZFS and 1x 750 GB disks ZFS and even on a 8 core, two socket Dell PowerEdge 1950-III box with two XEON CPUs running at 2,5 GHz, 16 GB RAM and two SATA-II harddrives attached to the built in SAS controller. Maybe this is of interest: http://www.phoronix.com/scan.php?page=article&item=freebsd8_ubuntu910&num=1 Watch the threaded I/O impact ... Regards, Oliver From owner-freebsd-performance@FreeBSD.ORG Tue Oct 6 09:47:05 2009 Return-Path: Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DB9DD106568B for ; Tue, 6 Oct 2009 09:47:05 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx06.syd.optusnet.com.au (fallbackmx06.syd.optusnet.com.au [211.29.132.8]) by mx1.freebsd.org (Postfix) with ESMTP id 47C2D8FC0C for ; Tue, 6 Oct 2009 09:47:04 +0000 (UTC) Received: from mail08.syd.optusnet.com.au (mail08.syd.optusnet.com.au [211.29.132.189]) by fallbackmx06.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id n9673KM1018267 for ; Tue, 6 Oct 2009 18:03:20 +1100 Received: from c122-107-125-150.carlnfd1.nsw.optusnet.com.au (c122-107-125-150.carlnfd1.nsw.optusnet.com.au [122.107.125.150]) by mail08.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id n9673Gwd004442 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Tue, 6 Oct 2009 18:03:18 +1100 Date: Tue, 6 Oct 2009 18:03:16 +1100 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Dieter In-Reply-To: <200910051755.RAA11047@sopwith.solgatos.com> Message-ID: <20091006174121.V25604@delplex.bde.org> References: <200910051755.RAA11047@sopwith.solgatos.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@FreeBSD.org Subject: Re: tuning FFS for large files Re: A specific example of a disk i/o problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 09:47:05 -0000 On Mon, 5 Oct 2009, Dieter wrote: > I found a clue! The problem occurs with my big data partitions, > which are newfs-ed with options intended to improve things. > > Reading a large file from the normal ad4s5b partition only delays other > commands slightly, as expected. Reading a large file from the tuned > ad4s11 partition yields the delay of minutes for other i/o. > ... > Here is the newfs command used for creating large data partitions: > newfs -e 57984 -b 65536 -f 8192 -g 67108864 -h 16 -i 67108864 -U -o time $partition Any block size above the default (16K) tends to thrash and fragment buffer cache virtual memory. This is obviously a good pessimization with lots of small files, and apparently, as you have found, it is a good pessimization with a few large files too. I think severe fragmentation can easily take several seconds to recover from. The worst case for causing fragmentaion is probably a mixture of small and large files. Some users fear fs consistency bugs with block sizes >= 16K, but I've never seen them cause any bugs ecept performance ones. > Even this isn't tuned the way I wanted to. > -g * -h must be less than 4 G due to 32 bit problem (system panics). The panic is now avoided in some versions of FreeBSD (-8 and -current at least). > Note the 32 bit problem is in the filesystem code, I'm running amd64. > IIRC there is a PR about this. (I'm assuming the bug hasn't been fixed yet) > Result is that I must specify -g and -h smaller than they should be. I bet you can't see any difference (except the panic) from enlarging -g and -h. > And they have way more inodes than needed. (IIRC it doesn't actually > use -i 67108864) It has to have at least 1 inode per cg, and may as well have a full block of them, which gives a fairly large number of inodes especially if the block size is too large (64K), so the -i ratio is limited. Bruce From owner-freebsd-performance@FreeBSD.ORG Tue Oct 6 20:08:06 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B1A1E1065670 for ; Tue, 6 Oct 2009 20:08:06 +0000 (UTC) (envelope-from serenity@exscape.org) Received: from ch-smtp01.sth.basefarm.net (ch-smtp01.sth.basefarm.net [80.76.149.212]) by mx1.freebsd.org (Postfix) with ESMTP id 42AC18FC0C for ; Tue, 6 Oct 2009 20:08:06 +0000 (UTC) Received: from c83-253-248-99.bredband.comhem.se ([83.253.248.99]:41136 helo=mx.exscape.org) by ch-smtp01.sth.basefarm.net with esmtp (Exim 4.68) (envelope-from ) id 1MvGKD-0002pY-3O; Tue, 06 Oct 2009 22:07:59 +0200 Received: from [192.168.1.5] (macbookpro [192.168.1.5]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mx.exscape.org (Postfix) with ESMTPSA id F0DB517D725; Tue, 6 Oct 2009 22:07:27 +0200 (CEST) Mime-Version: 1.0 (Apple Message framework v1076) Content-Type: text/plain; charset=us-ascii; format=flowed; delsp=yes From: Thomas Backman In-Reply-To: <4ACB068E.2070004@zedat.fu-berlin.de> Date: Tue, 6 Oct 2009 22:07:24 +0200 Content-Transfer-Encoding: 7bit Message-Id: <74A4B916-9090-40B1-958F-8FF772BD94AB@exscape.org> References: <200910051605.QAA06722@sopwith.solgatos.com> <88D64C60-D8ED-4622-959C-9946DBB99198@exscape.org> <4ACB068E.2070004@zedat.fu-berlin.de> To: O. Hartmann X-Mailer: Apple Mail (2.1076) X-Originating-IP: 83.253.248.99 X-Scan-Result: No virus found in message 1MvGKD-0002pY-3O. X-Scan-Signature: ch-smtp01.sth.basefarm.net 1MvGKD-0002pY-3O 5aad38e66d71c681732852611066b173 Cc: Dieter , freebsd-performance@freebsd.org Subject: Re: ZFS Re: A specific example of a disk i/o problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Oct 2009 20:08:06 -0000 On Oct 6, 2009, at 10:57 AM, O. Hartmann wrote: > Thomas Backman wrote: >> On Oct 5, 2009, at 6:05 PM, Dieter wrote: >>> In message , >>> Thomas Backman writes: >>> >>>> I run 8.0-RC1/amd64 with ZFS on an A64 3200+ with 2GB RAM and an >>>> old >>>> 80GB 7200rpm disk. >>>> >>>> My problem is that I get completely unacceptable latency on >>>> console IO >>>> (both via SSH and serial console) when the system is performing >>>> disk >>>> IO. The worst case I've noticed yet was when I tried copying a core >>>> dump from a lzjb compressed ZFS file system to a gzip-9 compressed >>>> one, to compare the file size/compression ratio. screen(1) took at >>>> LEAST ten seconds - probably a bit more - I'm not exaggerating >>>> here - >>>> to switch to another window, and an "ls" in an empty directory also >>>> about 5-10 seconds. >>> >>> You might find the "RELENG_7 heavy disk = system crawls" thread >>> interesting: >>> >>> } I didn't actually solve it or do anything. >>> } I just upgraded to RELENG_8. >>> } >>> } Now it's behaving more like FreeBSD should. >>> } I can do sequential reads/writes and still >>> } use kbd/mouse/X11/buildworld and so on. >> Doesn't really help much I'm afraid, since 8.0-RC1 = RELENG_8 = >> stable/8. Been running current/stable-8 since May. >> Regards, >> Thomas >> _______________________________________________ >> freebsd-performance@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-performance >> To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org >> " > > Is this really ZFS-bound? I also saw (and still see!) massive > performance impacts when a lot of disk activities, especially when > compiling large packages (done on UFS2 disks). Copying data from one > ZFS drive to (on different harddrive) another ZFS drive (which is > compressed) doesn't impact performance that much. > > When the box hit this performance issue, console, X11 and other > activities tend to be stuck for minutes! This happens on an UP box > (Athlon64 3500+, 2 GB ECC RAM, 2x 250 GB WD SATA-II disks and 1 TB > WD SATA-II disk (1x 250 GB UFS2 for system, 1x 250 GB ZFS compressed > for backup, 1x 1 TB ZFS holding /home). > > But this impact is also noticable on my lab's machine: Quad core > Intel Q6600 CPU at 3,0 GHZ, 8 GB RAM, 1x 320 GB SATA-II UFS2 disk > for system, 2x SATA-II 500 ZFS and 1x 750 GB disks ZFS and even on a > 8 core, two socket Dell PowerEdge 1950-III box with two XEON CPUs > running at 2,5 GHz, 16 GB RAM and two SATA-II harddrives attached to > the built in SAS controller. > > Maybe this is of interest: > http://www.phoronix.com/scan.php?page=article&item=freebsd8_ubuntu910&num=1 > > Watch the threaded I/O impact ... > > Regards, > Oliver It doesn't appear to be an ZFS issue, no. (Note that I wasn't the one who added it to the subject line :) I just tried it to my one UFS filesystem, and while screen(1) DID remain 100% responsive, everything didn't. vim worked OK most of the time, but other FS ops on the same disk... oh no. I aborted the "file / etc/*" after 57 seconds. Tried it again after stopping the disk IO (cat /dev/zero > /bootdir/filename) - 1.52 seconds. This raises the question: is this some kind of hardware specific issue? If so, what hardware? I can't think of anything my computer would have in common with the ones you've listed! I mean, come on, FreeBSD's IO performance can't be this abysmal for everybody or nobody would use it for serious applications. Something must be wrong for a few of us, but where and why? Regards, Thomas From owner-freebsd-performance@FreeBSD.ORG Sat Oct 10 16:23:24 2009 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 253E8106566B for ; Sat, 10 Oct 2009 16:23:24 +0000 (UTC) (envelope-from net.help@m2k.com.tw) Received: from py3.mail2000.com.tw (py3.mail2000.com.tw [203.69.82.67]) by mx1.freebsd.org (Postfix) with SMTP id 9F7448FC13 for ; Sat, 10 Oct 2009 16:23:23 +0000 (UTC) Received: from 10.0.0.81 by mx6.mail2000.com.tw with Mail2000 ESMTP Server V3.20M(54103:0:AUTH_RELAY) (envelope-from ); Sat, 10 Oct 2009 23:51:42 +0800 (CST) Received: from 10.0.0.50 by bak1.mail2000.com.tw with Mail2000 ESMTP Server V4.50S(8017:0:AUTH_LOGIN) (envelope-from ); Sat, 10 Oct 2009 23:46:56 +0800 (CST) Received: By OpenMail Mailer;Sat, 10 Oct 2009 23:46:52 +0800 (CST) From: "Ckcheng" Message-ID: <1255189612.42461.net.help@m2k.com.tw> To: "freebsd-performance" Date: Sat, 10 Oct 2009 23:46:52 +0800 (CST) MIME-Version: 1.0 X-Priority: 1 X-MSMail-Priority: High Content-Type: text/plain; charset=big5 Content-Transfer-Encoding: quoted-printable X-Mailman-Approved-At: Sat, 10 Oct 2009 16:43:50 +0000 Subject: Migrate large amount of small files X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: net.help@m2k.com.tw List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Oct 2009 16:23:24 -0000 Hi=20all, Currently,=20I=20have=20a=20directory=20with=20over=205M=20small=20files=20(= 1~32K).=20Now, I=20want=20to=20transfer=20this=20directory=20to=20another=20machine=20and=20= found that=20it's=20extremely=20slow=20and=20painful=20process.=20I=20tried=20the=20= following method: 1.=20rsync 2.=20tar=20via=20ssh 3.=20tar=20via=20nc (all=20take=20hours=20and=20hours=20to=20finish) None=20of=20them=20is=20able=20to=20give=20me=20a=20reasonable=20migration=20= time.=20So,=20I'm here=20for=20asking=20help.=20Any=20suggestion=20is=20extremely=20welcomed.=20= Thank=20you. Btw,=20here=20is=20brief=20information=20of=20my=20server.=20(both=20machine= s=20are=20the=20same) OS:=20FreeBSD=206.4-Stable=2064Bit CPU:=202=20x=20Xeon=20L5420=202.50GHz RAM:=202=20x=202G=20ECC=20DDR2-667=20(full=20buffered) DISK:=20Seagate=20Barracuda=20ES=2016MB=20(SATA=20300) Network:=201Gbps=20(Broadcom=20BCM5708) Filesystem:=20UFS2 Regards,