From owner-freebsd-performance@FreeBSD.ORG Sun Mar 4 11:22:31 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 38C2A16A400 for ; Sun, 4 Mar 2007 11:22:31 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 0DFA913C46B for ; Sun, 4 Mar 2007 11:22:30 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 1214546EA6; Sun, 4 Mar 2007 06:22:30 -0500 (EST) Date: Sun, 4 Mar 2007 11:22:29 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: "O. Hartmann" In-Reply-To: <45E7F09B.7070005@zedat.fu-berlin.de> Message-ID: <20070304111317.A60688@fledge.watson.org> References: <45E7F09B.7070005@zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Subject: Re: (S)ATA performance in FBSD 6.2/7.0 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Mar 2007 11:22:31 -0000 On Fri, 2 Mar 2007, O. Hartmann wrote: > On my lab's FreeBSD 6.2/i386 box (ASUS P4P800, ICH5 with two SATA 150 ports, > two SATA 300 drives attached) I copied big files (~ 5GB) from one drive to > another while the box didn't do anything else than copying. I watched the > copy process via 'systat -vmstat 1' and realized, that the value of 'KB/t' > never go byond 128 (128kb buffer limit?). But more frustrating, I never got > beyond 33 MB/s transfer rate although bonni/bonni++ told me both drives are > capable doing much more (~75 MB/s each). At home, I use a FreeBSD > 7.0-CURRENT box on an ASUS A8N32-SLI/nForce4-SLI based box, amd64 (no 32Bit > compatibility). Two Hitachi T7K250 250 GB/SATA II drives build up a RAID 0 > (nVidia MediaShield), and additionally there is a SAMSUNG Spinpoitn SP2004C > attached to the controller. bonni results in 55 MB/s for the SP2004C alone > and gives ~ 65 - 70 MB/s for the Hitachis, each and roughly 115 MB/s for the > RAID 0. But copying from the single drive to the RAID 0 or from the RAID 0 > to the single drive also reaches this oscure 33 MB/s boundary! It would be helpful if you could provide the dmesg output for the ATA controllers and devices. Sometimes, issues such as this come down to negotiated speed and bus topology. For example, if you have two devices on the same bus, then each byte of data copied has to pass over the bus twice, which may run into performance limits on the bus where moving the data over the bus once won't. Another area to investigate is the difference between raw device performance and file system performance. Does it make a difference if you copy using dd to a file that is already allocated? What performance does dd get when just reading from the file, or using dd to copy the file? What if the file already exists and you are simply overwriting it (avoiding the overhead of allocating)? Is your disk fragmented, leading to more seeks in placing new files, or is it relatively empty, which should lead to more contiguous storage? Finally, disk performance varies enourmously by location where the I/O is taking place on the disk. Try running diskinfo -t on your various disk devices, and see if there's a correlation between the maximum I/O rate you're experiencing and the location on the source or target drive? On the ATA disk on my workstation, I see 53M/s on the outside of the disk platter, but only 31M/s on the inside of the disk platter. Especially if your drive is relatively full, files may be ending up in less preferred locations on the disk resulting in reduced performance. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-performance@FreeBSD.ORG Sun Mar 4 11:55:01 2007 Return-Path: X-Original-To: freebsd-performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4D84916A400 for ; Sun, 4 Mar 2007 11:55:01 +0000 (UTC) (envelope-from bde@zeta.org.au) Received: from mailout2.pacific.net.au (mailout2-3.pacific.net.au [61.8.2.226]) by mx1.freebsd.org (Postfix) with ESMTP id 0F74E13C481 for ; Sun, 4 Mar 2007 11:55:01 +0000 (UTC) (envelope-from bde@zeta.org.au) Received: from mailproxy2.pacific.net.au (mailproxy2.pacific.net.au [61.8.2.163]) by mailout2.pacific.net.au (Postfix) with ESMTP id CCD746E1B1; Sun, 4 Mar 2007 22:54:55 +1100 (EST) Received: from besplex.bde.org (katana.zip.com.au [61.8.7.246]) by mailproxy2.pacific.net.au (Postfix) with ESMTP id 8C44F2740A; Sun, 4 Mar 2007 22:54:58 +1100 (EST) Date: Sun, 4 Mar 2007 22:54:56 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Alexander Leidinger In-Reply-To: <20070303110940.32893388@Magellan.Leidinger.net> Message-ID: <20070304225448.R7151@besplex.bde.org> References: <20070302120018.5830716A4A9@hub.freebsd.org> <45E8B994.2010100@umn.edu> <20070303110940.32893388@Magellan.Leidinger.net> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@FreeBSD.org, Alan Amesbury Subject: Re: (S)ATA performance in FBSD 6.2/7.0 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Mar 2007 11:55:01 -0000 On Sat, 3 Mar 2007, Alexander Leidinger wrote: > In FreeBSD we have 3 types, not 2. We have "sync", "noasync" (default) > and "async". Are you sure? 2 independent flags give 4 states. I think the flags are not completely independent, but sync doesn't cancel async in all cases (partly because of layering violations in ffs_update()) so sync together with async gives sync data with an undocumented combination of sync and async metadata. > For async the complete IO is done asynchronous (the meta-data too). Exccept possibly for (small) bugs. > And for sync the complete IO is synchronous. Except certainly for (larger) bugs. ffs_inode() normally refuses to do sync writes unless !DOINGASYNC(vp). When DOINGASYNC(vp), it mostly ignores its waitfor arg (this is its layering violation) and does a delayed write (not an async write, but an async write after a delay). This breaks cases where its caller wants to force a non-async write. Callers want to do this in the following cases: - sync together with async on the mount. sync should have precedence. I think the sync code knows to set waitfor = 1 for the call, but doesn't know that ffs_inode() will ignore the setting. - async on the mount with explicit fsync(2) or O_[F]SYNC. fsync(2) sets waitfor = 1, so the inode should get synced even if DOINGASYNC(vp), but it doesn't due to the layering violation. fsync(2) doesn't know how to sync all the directories above the file, and their metadata. O_[F]SYNC for the file together with async for the mount should work for the file like sync together with async for the mount work for all files, but has much the same bugs as fsync(2). > mount(8) tells this with some more words. It has related bugs. It literally says that sync and async have precedence over each other, but this is impossible. Bruce From owner-freebsd-performance@FreeBSD.ORG Sun Mar 4 16:51:02 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 5639C16A401 for ; Sun, 4 Mar 2007 16:51:02 +0000 (UTC) (envelope-from ohartman@mail.zedat.fu-berlin.de) Received: from outpost1.zedat.fu-berlin.de (outpost1.zedat.fu-berlin.de [130.133.4.66]) by mx1.freebsd.org (Postfix) with ESMTP id 6A98213C4B2 for ; Sun, 4 Mar 2007 16:51:01 +0000 (UTC) (envelope-from ohartman@mail.zedat.fu-berlin.de) Received: from inpost2.zedat.fu-berlin.de ([130.133.4.69]) by outpost1.zedat.fu-berlin.de (Exim 4.66) with esmtp (envelope-from ) id <1HNtdW-0006Bc-RO>; Sun, 04 Mar 2007 17:32:38 +0100 Received: from e178015054.adsl.alicedsl.de ([85.178.15.54] helo=[192.168.1.128]) by inpost2.zedat.fu-berlin.de (Exim 4.66) with esmtpsa (envelope-from ) id <1HNtdW-0004sL-DE>; Sun, 04 Mar 2007 17:32:38 +0100 Message-ID: <45EAF4E7.6040708@mail.zedat.fu-berlin.de> Date: Sun, 04 Mar 2007 17:33:43 +0100 From: "O. Hartmann" User-Agent: Thunderbird 1.5.0.9 (X11/20070211) MIME-Version: 1.0 References: <45E7F09B.7070005@zedat.fu-berlin.de> <200703021929.l22JTdrU091774@lava.sentex.ca> In-Reply-To: <200703021929.l22JTdrU091774@lava.sentex.ca> X-Enigmail-Version: 0.94.1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Originating-IP: 85.178.15.54 X-Mailman-Approved-At: Sun, 04 Mar 2007 20:40:58 +0000 Cc: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org Subject: Re: (S)ATA performance in FBSD 6.2/7.0 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Mar 2007 16:51:02 -0000 Mike Tancsa wrote: > At 04:38 AM 3/2/2007, O. Hartmann wrote: >> The last days I tried to figure out why some of my lab's FreeBSD >> boxes and also mine at home seem to be outperformed by some Linux >> setups around here and I saw something interesting. >> >> On my lab's FreeBSD 6.2/i386 box (ASUS P4P800, ICH5 with two SATA 150 >> ports, two SATA 300 drives attached) I copied big files (~ 5GB) from >> one drive to > > > Something strange about your setup I would say. I just tried on a > Segate SATA drive off an ICH5 chipset (plain old P IV 2.4Ghz). Do you > have an option in your BIOS for "native mode" or compatibility mode > for the SATA controller ? If so, try toggling that to native SATA mode > > [ns4]% iostat -c 1000 > tty ad4 twed0 cpu > tin tout KB/t tps MB/s KB/t tps MB/s us ni sy in id > 2 447 4.91 0 0.00 23.77 40 0.92 20 0 6 0 74 > 4 307 0.00 0 0.00 12.61 14 0.17 0 0 0 0 100 > 1 183 0.00 0 0.00 14.50 4 0.06 0 0 0 0 100 > 1 63 128.00 47 5.82 0.00 0 0.00 7 0 7 0 86 > 0 182 128.00 534 66.70 15.25 8 0.12 0 0 15 8 77 > 0 60 128.00 553 69.13 2.00 2 0.00 0 0 8 8 85 > 0 182 128.00 537 67.14 14.50 4 0.06 15 0 31 15 38 > 0 60 128.00 553 69.06 0.00 0 0.00 54 0 0 8 38 > 0 60 128.00 538 67.21 0.00 0 0.00 23 0 0 8 69 > 1 301 128.00 495 61.88 12.18 22 0.26 0 0 8 0 92 > > > [ns4]# dd if=/dev/ad4 of=/dev/null bs=1024k > ^C410+0 records in > 410+0 records out > 429916160 bytes transferred in 6.089321 secs (70601659 bytes/sec) > [ns4]# > > > [ns4]# atacontrol cap ad4 > > Protocol Serial ATA II > device model ST3400833NS > serial number 5NF25DTG > firmware revision 3.AEH > cylinders 16383 > heads 16 > sectors/track 63 > lba supported 268435455 sectors > lba48 supported 781422768 sectors > dma supported > overlap not supported > > Feature Support Enable Value Vendor > write cache yes yes > read ahead yes yes > Native Command Queuing (NCQ) yes - 31/0x1F > Tagged Command Queuing (TCQ) no no 31/0x1F > SMART yes yes > microcode download yes yes > security yes no > power management yes yes > advanced power management no no 65278/0xFEFE > automatic acoustic management no no 0/0x00 254/0xFE > [ns4]# > Sorry for waiting so long, but weekend and a lot of work prevented me from answering earlier. All right, I checked on three boxes the BIOS settings and ensured no legacy settings on any SATA device. In general, I realized using 'dd' results in much better drive-to-drive bandwith than doing just a 'cp'. As many times recommended herein, I used bs=128k, but look for yourself on the results. Box A, FreeBSD 7.0-CURRENT/AMD64: FreeBSD foo.org 7.0-CURRENT FreeBSD 7.0-CURRENT #38: Sun Mar 4 12:45:42 CET 2007 root@foor.org:/usr/obj/usr/src/sys/FOO amd64 Relevant harware: Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 7.0-CURRENT #38: Sun Mar 4 12:45:42 CET 2007 root@thor.walstatt.dyndns.org:/usr/obj/usr/src/sys/THOR ACPI APIC Table: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Athlon(tm) 64 Processor 3500+ (2210.07-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x10ff0 Stepping = 0 Features=0x78bfbff AMD Features=0xe2500800 AMD Features2=0x1 usable memory = 2136956928 (2037 MB) avail memory = 2062622720 (1967 MB) ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x508-0x50b on acpi0 cpu0: on acpi0 [...] atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xffa0-0xffaf at device 15.0 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] atapci1: port 0xb480-0xb487,0xb400-0xb403,0xb080-0xb087,0xb000-0xb003,0xac00-0xac0f mem 0xddcf9000-0xddcf9fff irq 23 at device 16.0 on pci0 atapci1: [ITHREAD] ata2: on atapci1 ata2: [ITHREAD] ata3: on atapci1 ata3: [ITHREAD] atapci2: port 0xa880-0xa887,0xa800-0xa803,0xa480-0xa487,0xa400-0xa403,0xa080-0xa08f mem 0xddcf8000-0xddcf8fff irq 20 at device 17.0 on pci0 atapci2: [ITHREAD] ata4: on atapci2 ata4: [ITHREAD] ata5: on atapci2 ata5: [ITHREAD] [...] thor# atacontrol cap ad4 Protocol Serial ATA II device model HDT722525DLA380 serial number VDK41BT4EJJW3K firmware revision V44OA96A cylinders 16383 heads 16 sectors/track 63 lba supported 268435455 sectors lba48 supported 488397168 sectors dma supported overlap not supported Feature Support Enable Value Vendor write cache yes yes read ahead yes yes Native Command Queuing (NCQ) yes - 31/0x1F Tagged Command Queuing (TCQ) no no 31/0x1F SMART yes yes microcode download yes yes security yes no power management yes yes advanced power management yes no 0/0x00 automatic acoustic management yes no 254/0xFE 128/0x80 thor# atacontrol cap ad6 Protocol Serial ATA II device model HDT722525DLA380 serial number VDK41BT4EJ58WK firmware revision V44OA96A cylinders 16383 heads 16 sectors/track 63 lba supported 268435455 sectors lba48 supported 488397168 sectors dma supported overlap not supported Feature Support Enable Value Vendor write cache yes yes read ahead yes yes Native Command Queuing (NCQ) yes - 31/0x1F Tagged Command Queuing (TCQ) no no 31/0x1F SMART yes yes microcode download yes yes security yes no power management yes yes advanced power management yes no 0/0x00 automatic acoustic management yes no 254/0xFE 128/0x80 thor# atacontrol cap ad8 Protocol Serial ATA II device model SAMSUNG SP2004C serial number S07GJ10Y304470 firmware revision VM100-31 cylinders 16383 heads 16 sectors/track 63 lba supported 268435455 sectors lba48 supported 390721968 sectors dma supported overlap not supported Feature Support Enable Value Vendor write cache yes yes read ahead yes yes Native Command Queuing (NCQ) yes - 31/0x1F Tagged Command Queuing (TCQ) no no 31/0x1F SMART yes yes microcode download yes yes security yes no power management yes yes advanced power management no no 0/0x00 automatic acoustic management yes no 0/0x00 254/0xFE The dmesg portion reflecting the drive configuration on the above mentioned box: acd0: DVDR at ata0-master UDMA33 ad4: 238475MB at ata2-master SATA300 ad6: 238475MB at ata3-master SATA300 ad8: 190782MB at ata4-master SATA300 ar0: 476950MB status: READY ar0: disk0 READY using ad6 at ata3-master ar0: disk1 READY using ad4 at ata2-master Below some test results using dd: 1) dd if=/dev/ad8 of=/dev/null (/dev/null is on ar0) thor# iostat -c 25 tty ad4 ad6 ad8 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 71 35 2.57 0 0.00 2.61 0 0.00 22.55 249 5.47 3 0 1 0 96 1 231 0.00 0 0.00 0.00 0 0.00 0.50 7535 3.68 2 0 30 6 62 287 78 0.00 0 0.00 0.00 0 0.00 0.50 7474 3.65 0 0 30 4 65 351 78 0.00 0 0.00 0.00 0 0.00 0.50 7475 3.65 0 0 20 6 74 519 78 0.00 0 0.00 0.00 0 0.00 0.50 7484 3.65 2 0 20 6 72 671 78 0.00 0 0.00 0.00 0 0.00 0.50 7535 3.68 0 0 17 4 79 559 78 0.00 0 0.00 0.00 0 0.00 0.50 7470 3.65 0 0 24 2 74 495 78 0.00 0 0.00 0.00 0 0.00 0.50 7469 3.65 2 0 25 2 72 511 78 0.00 0 0.00 0.00 0 0.00 0.50 7504 3.66 0 0 20 3 77 144 78 0.00 0 0.00 0.00 0 0.00 0.50 7521 3.67 0 0 27 4 69 0 78 0.00 0 0.00 0.00 0 0.00 0.50 7474 3.65 0 0 26 6 68 0 78 0.00 0 0.00 0.00 0 0.00 0.50 7476 3.65 0 0 24 8 68 136 78 0.00 0 0.00 0.00 0 0.00 0.50 7506 3.66 0 0 27 5 68 40 78 0.00 0 0.00 0.00 0 0.00 0.50 7513 3.67 0 0 24 4 71 567 78 0.00 0 0.00 0.00 0 0.00 0.50 7465 3.64 2 0 27 4 68 455 78 0.00 0 0.00 0.00 0 0.00 0.50 7482 3.65 1 0 28 5 65 655 78 0.00 0 0.00 0.00 0 0.00 0.50 7543 3.68 2 0 28 8 61 208 78 0.00 0 0.00 0.00 0 0.00 0.50 7484 3.65 1 0 27 4 68 152 78 0.00 0 0.00 0.00 0 0.00 0.50 7471 3.65 0 0 29 6 64 287 78 0.00 0 0.00 0.00 0 0.00 0.50 7476 3.65 0 0 28 4 68 2) dd if=/dev/ad8 of=/dev/null bs=128k (/dev/null is on ar0) thor# iostat -c 25 tty ad4 ad6 ad8 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 75 35 2.57 0 0.00 2.61 0 0.00 13.20 424 5.46 3 0 2 0 95 1 231 0.00 0 0.00 0.00 0 0.00 64.00 913 57.09 1 0 3 0 95 415 77 0.00 0 0.00 0.00 0 0.00 64.00 912 57.01 0 0 2 0 98 64 77 0.00 0 0.00 0.00 0 0.00 64.00 914 57.14 0 0 2 0 98 16 77 0.00 0 0.00 0.00 0 0.00 64.00 914 57.14 0 0 4 0 96 0 77 0.00 0 0.00 0.00 0 0.00 64.00 912 57.01 0 0 2 0 97 0 77 0.00 0 0.00 0.00 0 0.00 64.00 914 57.14 0 0 3 0 97 96 77 0.00 0 0.00 0.00 0 0.00 64.00 906 56.64 0 0 2 2 96 8 77 0.00 0 0.00 0.00 0 0.00 64.00 905 56.57 0 0 3 0 96 48 77 0.00 0 0.00 0.00 0 0.00 64.00 914 57.14 0 0 2 0 98 431 77 0.00 0 0.00 0.00 0 0.00 64.00 913 57.07 0 0 4 0 96 144 77 0.00 0 0.00 0.00 0 0.00 64.00 914 57.14 0 0 2 0 98 104 77 0.00 0 0.00 0.00 0 0.00 64.00 913 57.07 0 0 2 1 96 16 77 0.00 0 0.00 0.00 0 0.00 64.00 913 57.07 0 0 2 1 97 0 77 0.00 0 0.00 0.00 0 0.00 64.00 914 57.14 0 0 2 0 97 0 77 0.00 0 0.00 0.00 0 0.00 64.00 913 57.07 0 0 3 1 96 1 77 0.00 0 0.00 0.00 0 0.00 64.00 913 57.07 0 0 2 0 98 15 77 0.00 0 0.00 0.00 0 0.00 64.00 914 57.14 0 0 2 2 96 As we can see in this case, bandwith hits the maximum what could be delivered by the SAMSUNG SP2004C drive. 3) dd if=/dev/ad4 of=/dev/null thor# iostat -c 25 tty ad4 ad6 ad8 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 79 34 32.72 31 1.00 2.61 0 0.00 15.83 425 6.57 3 0 2 0 95 1 230 0.50 14370 7.02 0.00 0 0.00 0.00 0 0.00 1 0 42 8 50 0 79 0.50 14371 7.02 0.00 0 0.00 0.00 0 0.00 2 0 44 8 46 0 79 0.50 14277 6.97 0.00 0 0.00 0.00 0 0.00 2 0 45 8 45 0 79 0.50 14263 6.96 0.00 0 0.00 0.00 0 0.00 1 0 36 8 55 0 79 0.50 14387 7.03 0.00 0 0.00 0.00 0 0.00 2 0 43 10 44 0 79 0.50 14383 7.02 0.00 0 0.00 0.00 0 0.00 1 0 38 9 52 0 79 0.50 14395 7.03 0.00 0 0.00 0.00 0 0.00 0 0 36 11 53 431 79 0.50 14358 7.01 0.00 0 0.00 0.00 0 0.00 2 0 48 8 42 399 79 0.50 14330 7.00 0.00 0 0.00 0.00 0 0.00 1 0 44 6 50 0 79 0.50 14380 7.02 0.00 0 0.00 0.00 0 0.00 1 0 40 6 52 33 79 0.50 14363 7.01 0.00 0 0.00 0.00 0 0.00 0 0 44 6 50 17 79 0.50 14378 7.02 0.00 0 0.00 0.00 0 0.00 0 0 39 12 49 4) dd if=/dev/ad4 of=/dev/null bs=128k thor# iostat -c 25 tty ad4 ad6 ad8 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 78 35 63.44 1 0.05 2.61 0 0.00 15.83 432 6.67 3 0 2 0 95 1 231 64.00 1074 67.14 0.00 0 0.00 0.00 0 0.00 0 0 3 1 95 0 78 64.00 1076 67.24 0.00 0 0.00 0.00 0 0.00 0 0 1 1 98 0 78 64.00 1076 67.24 0.00 0 0.00 0.00 0 0.00 0 0 2 0 97 0 78 64.00 1076 67.24 0.00 0 0.00 0.00 0 0.00 0 0 4 0 95 0 78 64.00 1076 67.24 0.00 0 0.00 0.00 0 0.00 0 0 0 0 99 0 78 64.00 1075 67.18 0.00 0 0.00 0.00 0 0.00 0 0 4 0 95 0 78 64.00 1076 67.24 0.00 0 0.00 0.00 0 0.00 0 0 6 1 93 0 78 64.00 1076 67.24 0.00 0 0.00 0.00 0 0.00 0 0 0 1 99 0 78 64.00 1075 67.18 0.00 0 0.00 0.00 0 0.00 1 0 5 0 93 0 78 64.00 1068 66.74 0.00 0 0.00 0.00 0 0.00 0 0 2 0 98 0 78 64.00 1074 67.12 0.00 0 0.00 0.00 0 0.00 0 0 4 1 94 0 78 64.00 1076 67.24 0.00 0 0.00 0.00 0 0.00 0 0 4 0 96 0 78 64.00 1076 67.24 0.00 0 0.00 0.00 0 0.00 0 0 1 0 99 407 78 64.00 1076 67.24 0.00 0 0.00 0.00 0 0.00 0 0 4 0 96 260 78 64.00 1077 67.30 0.00 0 0.00 0.00 0 0.00 0 0 2 2 96 ==> for /dev/ad6 it is exactly the same! 5) dd if=/dev/ar0 of=/dev/null bs=128k 4 users Load 0.06 0.11 0.06 Mar 4 16:50 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 153036 46624 424148 57400 1731088 count All 196176 47268 4685068 69708 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 5176 total 2 57 10k 27 2307 2166 2330 zfod 5 atkbd0 1 ozfod fdc0 irq6 5.0%Sys 1.5%Intr 0.0%User 0.0%Nice 93.5%Idle %ozfod ata0 irq14 | | | | | | | | | | | daefr atapci2 20 === prcfr 1 nfe0 ohci0 5 dtbuf totfr ehci0 22 Namei Name-cache Dir-cache 100000 desvn react 2160 atapci1 23 Calls hits % hits % 2523 numvn pdwak 3010 cpu0: time 49 49 100 187 frevn pdpgs intrn Disks ad4 ad6 ad8 ar0 cd0 pass0 119912 wire KB/t 0.00 0.00 0.00 64.00 0.00 0.00 109532 act tps 0 0 0 2152 0 0 51188 inact MB/s 0.00 0.00 0.00 135 0.00 0.00 4660 cache <<<<=========!!! %busy 0 0 0 97 0 0 1726428 free 93360 buf Sorry for the messed up cut-and-paste tty output, but as you can see, pretty fast 135 MB/s with 'dd'. This is the systat-output copying an 1G in size QEMU disk image via 'cp' from /dev/ad8 (/dev/ufs/BACKUP) to my home, located on /dev/ar0s3d: 4 users Load 0.11 0.08 0.06 Mar 4 16:57 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 130784 28360 413124 57056 90404 count 6 All 173300 29004 4674044 69364 pages 22 Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt 37 cow 4549 total 2 55 6637 607 2004 1539 1244 110 50 zfod atkbd0 1 50 ozfod fdc0 irq6 14.0%Sys 3.0%Intr 0.0%User 0.0%Nice 83.0%Idle 100%ozfod ata0 irq14 | | | | | | | | | | | daefr 495 atapci2 20 =======++ 57 prcfr 2 nfe0 ohci0 42 dtbuf 16105 totfr ehci0 22 Namei Name-cache Dir-cache 100000 desvn react 1042 atapci1 23 Calls hits % hits % 2626 numvn 1 pdwak 3010 cpu0: time 64 64 100 1804 frevn 12961 pdpgs intrn Disks ad4 ad6 ad8 ar0 cd0 pass0 264168 wire KB/t 0.00 0.00 63.74 58.47 0.00 0.00 134896 act tps 0 0 492 551 0 0 1514576 inact MB/s 0.00 0.00 30.65 31.48 0.00 0.00 87564 cache %busy 0 0 70 53 0 0 2840 free 218608 buf The transfer rate is jumping between 28 MB/s and sometimes it reaches 38 MB/s, but most time it is around 32/33 MB/s. The next tests were taken on the mentioned FreeBSD 6.2/i386 box at my lab. FreeBSD iota.fu-berlin.de 6.2-STABLE FreeBSD 6.2-STABLE #5: Fri Mar 2 10:09:06 CET 2007 root@iota.fu-berlin.de:/usr/obj/usr/src/sys/IOTA i386 Portions of interest taken from dmesg: Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-STABLE #5: Fri Mar 2 10:09:06 CET 2007 root@telesto.geoinf.fu-berlin.de:/usr/obj/usr/src/sys/TELESTO Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (2998.57-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0xf29 Stepping = 9 Features=0xbfebfbff Features2=0x4400> Logical CPUs per core: 2 real memory = 2146631680 (2047 MB) avail memory = 2095501312 (1998 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 netsmb_dev: loaded acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 cpu1: on acpi0 [...] atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0 ata0: on atapci0 ata1: on atapci0 atapci1: port 0xefe0-0xefe7,0xefac-0xefaf,0xefa0-0xefa7,0xefa8-0xefab,0xef90-0xef9f irq 18 at device 31.2 on pci0 ata2: on atapci1 ata3: on atapci1 [...] acd0: CDRW at ata1-master UDMA33 ad4: 238475MB at ata2-master SATA150 ad6: 381554MB at ata3-master SATA150 [...] atacontrol cap ad4 Protocol Serial ATA II device model Hitachi HDT725025VLA380 serial number VFA100R1CD9VBA firmware revision V5DOA52A cylinders 16383 heads 16 sectors/track 63 lba supported 268435455 sectors lba48 supported 488397168 sectors dma supported overlap not supported Feature Support Enable Value Vendor write cache yes yes read ahead yes yes Native Command Queuing (NCQ) yes - 31/0x1F Tagged Command Queuing (TCQ) no no 31/0x1F SMART yes yes microcode download yes yes security yes no power management yes yes advanced power management yes no 0/0x00 automatic acoustic management yes no 254/0xFE 128/0x80 atacontrol cap ad6 Protocol Serial ATA II device model SAMSUNG HD401LJ serial number S0HVJ1CLB07119 firmware revision ZZ100-15 cylinders 16383 heads 16 sectors/track 63 lba supported 268435455 sectors lba48 supported 781422768 sectors dma supported overlap not supported Feature Support Enable Value Vendor write cache yes yes read ahead yes yes Native Command Queuing (NCQ) yes - 31/0x1F Tagged Command Queuing (TCQ) no no 31/0x1F SMART yes yes microcode download yes yes security yes no power management yes yes advanced power management no no 0/0x00 automatic acoustic management yes no 0/0x00 254/0xFE 'dd' test, shotened, because they reflect the same situation as shown above for the AMD64 box: dd if=/dev/ad4 of=/dev/null bs=128k hartmann: iostat -c 10 tty ad4 ad6 cd0 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 2 31.12 2 0.06 124.95 1 0.13 0.00 0 0.00 2 0 0 0 98 0 232 128.00 489 61.08 0.00 0 0.00 0.00 0 0.00 0 0 3 1 96 0 78 128.00 505 63.19 0.00 0 0.00 0.00 0 0.00 0 0 5 0 95 0 78 128.00 505 63.19 0.00 0 0.00 0.00 0 0.00 0 0 1 4 95 0 78 128.00 505 63.19 0.00 0 0.00 0.00 0 0.00 0 0 2 1 97 0 78 128.00 499 62.44 0.00 0 0.00 0.00 0 0.00 0 0 2 1 97 0 78 128.00 502 62.81 0.00 0 0.00 0.00 0 0.00 0 0 0 3 97 0 78 128.00 505 63.19 0.00 0 0.00 0.00 0 0.00 0 0 0 3 97 0 78 128.00 505 63.18 0.00 0 0.00 0.00 0 0.00 0 0 2 1 97 0 78 127.78 501 62.45 0.00 0 0.00 0.00 0 0.00 0 0 3 1 97 and dd if=/dev/ad6 of=/dev/null bs=128 hartmann: iostat -c 10 tty ad4 ad6 cd0 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 2 37.21 2 0.08 124.97 1 0.13 0.00 0 0.00 2 0 0 0 98 0 232 0.00 0 0.00 128.00 551 68.94 0.00 0 0.00 0 0 2 0 98 0 78 0.00 0 0.00 128.00 549 68.68 0.00 0 0.00 0 0 1 1 98 0 78 0.00 0 0.00 128.00 563 70.43 0.00 0 0.00 0 0 3 1 96 0 78 0.00 0 0.00 128.00 559 69.93 0.00 0 0.00 0 0 2 2 96 0 78 0.00 0 0.00 128.00 558 69.81 0.00 0 0.00 0 0 2 2 97 0 78 0.00 0 0.00 128.00 574 71.80 0.00 0 0.00 0 0 5 1 94 0 78 0.00 0 0.00 128.00 547 68.43 0.00 0 0.00 0 0 3 1 96 0 78 0.00 0 0.00 128.00 564 70.55 0.00 0 0.00 0 0 2 2 97 0 78 0.00 0 0.00 128.00 574 71.80 0.00 0 0.00 0 0 3 2 95 And this is the result copying copying a 6.1 G in size qemu drive image from ad4 to ad6: cp qemu/c.img /mnt/ext/s1/qemu/c.img.test hartmann: iostat -c 10 tty ad4 ad6 cd0 cpu tin tout KB/t tps MB/s KB/t tps MB/s KB/t tps MB/s us ni sy in id 0 2 37.49 2 0.08 126.30 2 0.24 0.00 0 0.00 2 0 0 0 98 0 232 127.66 329 40.99 127.66 329 40.99 0.00 0 0.00 0 0 12 2 86 0 79 126.93 322 39.87 127.30 320 39.74 0.00 0 0.00 0 0 14 2 83 0 79 127.66 329 40.97 127.66 329 40.97 0.00 0 0.00 0 0 12 1 87 0 79 127.66 326 40.60 127.66 326 40.60 0.00 0 0.00 0 0 13 2 85 0 79 127.66 327 40.71 127.66 327 40.71 0.00 0 0.00 0 0 13 3 84 0 79 127.30 320 39.75 126.53 322 39.76 0.00 0 0.00 0 0 11 1 88 0 79 124.90 289 35.21 109.45 337 35.99 0.00 0 0.00 0 0 12 3 85 0 79 127.66 327 40.73 127.66 326 40.60 0.00 0 0.00 0 0 15 1 84 0 79 127.64 312 38.85 127.64 312 38.85 0.00 0 0.00 0 0 11 1 88 It is very interesting to see that FreeBSD 6.2-STABLE/i386 (SMP by HTT) shows up the wrong KB/t size of 128 KB, while FreeBSD 7.0-CURRENT/AMD64 (UP) shows the correct KB/t size of 64 KB/t as mentioned before in this thread. I'm a little bit suprised by the 'good' results of 'cp', because when I did these tests with 'cp' and copying the big files (5 and 6 G qemu drive images) systat hits consequently the 33 MB/s boundary - but I must confess that I had loaded the "aio.ko" kernel module and ran before qemu, kldunloaded kqemu.ko but left aio.ko kldloaded. I will do the tests tomorrow when I'm back at my lab. Nevertheless, on the RAID 0 box I would expect a much better result than on the single-drive boxes. 'dd' seems to do a well job, but it runs as 'root' and I guess a lot of us doing a lot of file copying via 'cp', so there is still a big gap left I can not explain. Sorry for the huge output ... Regards, Oliver From owner-freebsd-performance@FreeBSD.ORG Sun Mar 4 22:16:51 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EB36216A400 for ; Sun, 4 Mar 2007 22:16:51 +0000 (UTC) (envelope-from _pppp@mail.ru) Received: from mx28.mail.ru (mx28.mail.ru [194.67.23.67]) by mx1.freebsd.org (Postfix) with ESMTP id A95E213C441 for ; Sun, 4 Mar 2007 22:16:51 +0000 (UTC) (envelope-from _pppp@mail.ru) Received: from f31.mail.ru (f31.mail.ru [194.67.57.70]) by mx28.mail.ru (mPOP.Fallback_MX) with ESMTP id 274276E97F8 for ; Mon, 5 Mar 2007 00:53:27 +0300 (MSK) Received: from mail by f31.mail.ru with local id 1HNydx-000MLw-00; Mon, 05 Mar 2007 00:53:25 +0300 Received: from [91.76.23.231] by koi.mail.ru with HTTP; Mon, 05 Mar 2007 00:53:25 +0300 From: dima <_pppp@mail.ru> To: O. Hartmann Mime-Version: 1.0 X-Mailer: mPOP Web-Mail 2.19 X-Originating-IP: [91.76.23.231] Date: Mon, 05 Mar 2007 00:53:25 +0300 In-Reply-To: <45E7F09B.7070005@zedat.fu-berlin.de> References: <45E7F09B.7070005@zedat.fu-berlin.de> Content-Type: text/plain; charset=koi8-r Content-Transfer-Encoding: 8bit Message-Id: Cc: freebsd-performance@freebsd.org Subject: Re: (S)ATA performance in FBSD 6.2/7.0 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: dima <_pppp@mail.ru> List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 04 Mar 2007 22:16:52 -0000 > The last days I tried to figure out why some of my lab's FreeBSD boxes > and also mine at home seem to be outperformed by some Linux setups > around here and I saw something interesting. > > On my lab's FreeBSD 6.2/i386 box (ASUS P4P800, ICH5 with two SATA 150 > ports, two SATA 300 drives attached) I copied big files (~ 5GB) from one > drive to another while the box didn't do anything else than copying. I > watched the copy process via 'systat -vmstat 1' and realized, that the > value of 'KB/t' never go byond 128 (128kb buffer limit?). But more > frustrating, I never got beyond 33 MB/s transfer rate although > bonni/bonni++ told me both drives are capable doing much more (~75 MB/s > each). > At home, I use a FreeBSD 7.0-CURRENT box on an ASUS > A8N32-SLI/nForce4-SLI based box, amd64 (no 32Bit compatibility). Two > Hitachi T7K250 250 GB/SATA II drives build up a RAID 0 (nVidia > MediaShield), and additionally there is a SAMSUNG Spinpoitn SP2004C > attached to the controller. bonni results in 55 MB/s for the SP2004C > alone and gives ~ 65 - 70 MB/s for the Hitachis, each and roughly 115 > MB/s for the RAID 0. But copying from the single drive to the RAID 0 or > from the RAID 0 to the single drive also reaches this oscure 33 MB/s > boundary! > > In the first place I thought the older i386 hardware has some > hard-limits, but we have several boxes of the exact same hardware around > here and a wide spread Linux and Windows utilization and on those boxes > equipted with more than one harddrive (PATA or SATA) the effective > transfer rate shown up is about 50 - 65 MB/s as expected with copying a > big 5G file from one drive to another. > > The hardwrae limit is completely nonsense when it comes to the AMD64 box > with newer hardware. > > Before digging into this problem deeper with benchmarks, could anyone > explain why FreeBSD reaches this 33 MB/s limit (sounds like UDMA 33 > defaults, but on both boxes nForce4 and ICH5 controller are recognized > and show up with SATA300 or SATA150 capabilities, respective)? May I > have some knobs I'm not aware of to tune disk performance? > > I would appreciate any coments on that and if someone has some good > ideas how to benchmark those subjects, please let me know. I think you're quite OK with dd. I do believe you'll get a comparable results for cp setting "noatime" option in mount(8)s. As you were told, the default mount mode is noasync which is "synchronous metadata + asynchronous data". But FreeBSD still updates metadata for atime on every file read. 2all: Are there any reasons not to make noasync the default mount option? It greatly improves FS performance and I don't think many modern administrators do care about access times of their files (assuming how many files a modern box has). Regards, Dmitriy. From owner-freebsd-performance@FreeBSD.ORG Mon Mar 5 03:19:23 2007 Return-Path: X-Original-To: freebsd-performance@FreeBSD.org Delivered-To: freebsd-performance@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 028A716A402 for ; Mon, 5 Mar 2007 03:19:23 +0000 (UTC) (envelope-from bde@zeta.org.au) Received: from mailout1.pacific.net.au (mailout1-3.pacific.net.au [61.8.2.210]) by mx1.freebsd.org (Postfix) with ESMTP id 95A5A13C467 for ; Mon, 5 Mar 2007 03:19:22 +0000 (UTC) (envelope-from bde@zeta.org.au) Received: from mailproxy1.pacific.net.au (mailproxy1.pacific.net.au [61.8.2.162]) by mailout1.pacific.net.au (Postfix) with ESMTP id 0A56E5A7F48; Mon, 5 Mar 2007 14:19:21 +1100 (EST) Received: from besplex.bde.org (katana.zip.com.au [61.8.7.246]) by mailproxy1.pacific.net.au (Postfix) with ESMTP id E25A78C23; Mon, 5 Mar 2007 14:19:19 +1100 (EST) Date: Mon, 5 Mar 2007 14:19:18 +1100 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: dima <_pppp@mail.ru> In-Reply-To: Message-ID: <20070305133954.D2598@besplex.bde.org> References: <45E7F09B.7070005@zedat.fu-berlin.de> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@FreeBSD.org, "O. Hartmann" Subject: Re: (S)ATA performance in FBSD 6.2/7.0 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Mar 2007 03:19:23 -0000 On Mon, 5 Mar 2007, dima wrote: > I think you're quite OK with dd. I do believe you'll get a comparable results for cp setting "noatime" option in mount(8)s. As you were told, the default mount mode is noasync which is "synchronous metadata + asynchronous data". But FreeBSD still updates metadata for atime on every file read. No. Atime updates are always async (strictly, delayed). It is another bug in mount.8 and in many mails in this thread to claim that !noasync gives sync for all metadata or that sync gives sync for metadata. The async and sync flags have no effect on syncing atime updates. Similarly for updates of all times set by utimes(2). This might be a bug in the sync mount and O_FSYNC cases. fsync(2) on the file after the utimes(2) will update the metadata (to the disk), but O_FSYNC doesn't make this happen automatically. 4.4BSD-Lite1 gets this wrong in the opposite way, by always doing a sync update for times set by utimes(2). This was changed in 4.4BSD-Lite2 and misimported into FreeBSD in rev.1.79 of ufs_vnops.c by splatting Lite2 changes onto the non-vendor branch and not documenting them. Similarly except for history for the more critical updates of metadata changed by chown(2), chmod(2), chflags(2) and perhaps others. These updates were always done only to the in-core inode and any syncing to the disk was accidental. The syncing for utimes(2) is just an accident that always happened immediately. It was silly to sync relatively unimportant changes to times while not syncing changes to permissions. > 2all: Are there any reasons not to make noasync the default mount option? It greatly improves FS performance and I don't think many modern administrators do care about access times of their files (assuming how many files a modern box has). I think you mean noatime. It is required by POSIX and POLA. noatime has an especially tiny effect on large copies. The atime updates get written to disk every 30 seconds or so, one for every GB or so of data. noatime makes a difference mainly for tree walks where looking in lots of directories clobbers their atimes. Bruce From owner-freebsd-performance@FreeBSD.ORG Mon Mar 5 11:23:10 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2A27A16A402 for ; Mon, 5 Mar 2007 11:23:10 +0000 (UTC) (envelope-from dinesh@alphaque.com) Received: from ns2.alphaque.com (mail.qubeconnect.com [202.190.74.25]) by mx1.freebsd.org (Postfix) with SMTP id 3EF1913C4B3 for ; Mon, 5 Mar 2007 11:23:08 +0000 (UTC) (envelope-from dinesh@alphaque.com) Received: (qmail 47627 invoked by uid 0); 5 Mar 2007 11:22:16 -0000 Received: from lucifer.net-gw.com (HELO prophet.alphaque.com) (202.190.74.25) by lucifer.net-gw.com with SMTP; 5 Mar 2007 11:22:16 -0000 Received: from prophet.alphaque.com (localhost [127.0.0.1]) by prophet.alphaque.com (8.13.8/8.13.4) with ESMTP id l25Aqus0015244 for ; Mon, 5 Mar 2007 18:52:56 +0800 (MYT) (envelope-from dinesh@alphaque.com) Date: Mon, 5 Mar 2007 18:52:56 +0800 From: Dinesh Nair To: freebsd-performance@freebsd.org Message-ID: <20070305185256.647b4541@prophet.alphaque.com> In-Reply-To: References: <20070224215508.GA41968@xor.obsecurity.org> <45E13410.7020505@he.iki.fi> <20070225071946.GA48242@xor.obsecurity.org> <45E14BAD.80909@he.iki.fi> <20070225084737.GA49231@xor.obsecurity.org> <5a0a9d6f0702260936u3408f8d8rd4cde9234b2f7776@mail.gmail.com> <45E54619.7000503@isc.org> Organization: Alphaque. Anytime. Anywhere X-Mailer: Sylpheed-Claws 2.3.1 (GTK+ 2.8.12; i386-portbld-freebsd6.2) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: Re: UDP performance. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Mar 2007 11:23:10 -0000 On Thu, 1 Mar 2007 07:16:29 -0800, Barrett Lyon wrote: > I also have a few 10GbE interfaces and Ixia chassis that I can run > tests with if someone wants to send me a spec. and i'd be willing to test UDP RTP performance on 6.2 with asterisk for this. RTP traffic can generate thousands of small (60-70 byte) packets per second. -- Regards, /\_/\ "All dogs go to heaven." dinesh@alphaque.com (0 0) http://www.openmalaysiablog.com/ +==========================----oOO--(_)--OOo----==========================+ | for a in past present future; do | | for b in clients employers associates relatives neighbours pets; do | | echo "The opinions here in no way reflect the opinions of my $a $b." | | done; done | +=========================================================================+ From owner-freebsd-performance@FreeBSD.ORG Mon Mar 5 18:36:51 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 435A716A400 for ; Mon, 5 Mar 2007 18:36:51 +0000 (UTC) (envelope-from dgilbert@daveg.ca) Received: from ox.eicat.ca (ox.eicat.ca [66.96.30.35]) by mx1.freebsd.org (Postfix) with ESMTP id 1418B13C4B7 for ; Mon, 5 Mar 2007 18:36:51 +0000 (UTC) (envelope-from dgilbert@daveg.ca) Received: by ox.eicat.ca (Postfix, from userid 66) id A1251D651; Mon, 5 Mar 2007 13:36:50 -0500 (EST) Received: by canoe.dclg.ca (Postfix, from userid 101) id 6264561C8A; Mon, 5 Mar 2007 13:36:56 -0500 (EST) From: David Gilbert MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17900.25416.355835.2354@canoe.dclg.ca> Date: Mon, 5 Mar 2007 13:36:56 -0500 To: Eric Anderson In-Reply-To: <45E57EB5.4070703@freebsd.org> References: <20070224215508.GA41968@xor.obsecurity.org> <45E13410.7020505@he.iki.fi> <20070225071946.GA48242@xor.obsecurity.org> <45E14BAD.80909@he.iki.fi> <20070225084737.GA49231@xor.obsecurity.org> <5a0a9d6f0702260936u3408f8d8rd4cde9234b2f7776@mail.gmail.com> <45E54619.7000503@isc.org> <45E57EB5.4070703@freebsd.org> X-Mailer: VM 7.17 under 21.4 (patch 20) "Double Solitaire" XEmacs Lucid Cc: freebsd-performance@freebsd.org, Peter Losher Subject: Re: UDP performance. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Mar 2007 18:36:51 -0000 >>>>> "Eric" == Eric Anderson writes: Eric> On 02/28/07 03:06, Peter Losher wrote: >> Ivan Voras wrote: >> We recently put a stock Fedora Core 6 and a stock FreeBSD 6.2 on >> the same HW (HP ProLiant DL320 G5 Dual Core Xeons w/ 16GB RAM) and >> running BIND 9.4.0 and a well known ccTLD zone that we slammed a >> query stream to. On a single threaded BIND, there was a 20% >> advantage to Linux, on a multi threaded build, Linux trounced >> FreeBSD (39k to 89k queries/sec) I don't believe this is specifically a FreeBSD UDP performance issue. Using NSD as the nameserver and fast core-2 duos, FreeBSD was able to old 140k queries/sec while linux (on the same machine) ran 90k queries per second. This was using two server threads. I believe the numbers were 120k and 80k for one server thread. But: the whole reason I'm testing NSD for a client is because BIND has huge performance bottlenecks with large zones (bind reportedly takes 20 minutes from when it run to when it's ready to serve the first query). Dave. -- ============================================================================ |David Gilbert, Independent Contractor. | Two things can be | |Mail: dave@daveg.ca | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================ From owner-freebsd-performance@FreeBSD.ORG Mon Mar 5 18:38:20 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 3D55616A402 for ; Mon, 5 Mar 2007 18:38:20 +0000 (UTC) (envelope-from dgilbert@daveg.ca) Received: from ox.eicat.ca (ox.eicat.ca [66.96.30.35]) by mx1.freebsd.org (Postfix) with ESMTP id 1025213C481 for ; Mon, 5 Mar 2007 18:38:20 +0000 (UTC) (envelope-from dgilbert@daveg.ca) Received: by ox.eicat.ca (Postfix, from userid 66) id A1977D67E; Mon, 5 Mar 2007 13:38:19 -0500 (EST) Received: by canoe.dclg.ca (Postfix, from userid 101) id 4732161C8A; Mon, 5 Mar 2007 13:38:25 -0500 (EST) From: David Gilbert MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-ID: <17900.25505.245946.235433@canoe.dclg.ca> Date: Mon, 5 Mar 2007 13:38:25 -0500 To: Dinesh Nair In-Reply-To: <20070228214200.60cc9b7a@prophet.alphaque.com> References: <20070224215508.GA41968@xor.obsecurity.org> <45E13410.7020505@he.iki.fi> <20070225071946.GA48242@xor.obsecurity.org> <45E14BAD.80909@he.iki.fi> <20070225084737.GA49231@xor.obsecurity.org> <5a0a9d6f0702260936u3408f8d8rd4cde9234b2f7776@mail.gmail.com> <45E54619.7000503@isc.org> <20070228214200.60cc9b7a@prophet.alphaque.com> X-Mailer: VM 7.17 under 21.4 (patch 20) "Double Solitaire" XEmacs Lucid Cc: freebsd-performance@freebsd.org Subject: Re: UDP performance. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Mar 2007 18:38:20 -0000 >>>>> "Dinesh" == Dinesh Nair writes: Dinesh> On Wed, 28 Feb 2007 01:06:33 -0800, Peter Losher Dinesh> wrote: >> Ivan Voras wrote: >> >> > I agree in general, but MySQL performance is very exposed as an > >> advocacy issue - it has traditionally been the source of statements >> > like "FreeBSD's threading implementation is weak/bad/broken". >> >> And these days ISC can't consciously recommend FreeBSD for use on >> high-traffic DNS servers because UDP performance has (frankly) gone >> downhill since 5.x. [..snipped..] >> http://lists.freebsd.org/pipermail/freebsd-net/2006-September/011748.html Dinesh> if UDP performance in 6.x and 7.x has dropped, this could even Dinesh> affect voip applications/servers such as asterisk when run on Dinesh> FreeBSD. most all use RTP for media traffic and RTP is nearly Dinesh> always UDP generating up to 50 packets per second per call per Dinesh> direction. Dinesh> 14,000+ packets per second is only about 140 calls. Well... again, BIND is not a good indicator of UDP performance. A non-trivial application can send and receive about 250k pps on moderate hardware. Dave. -- ============================================================================ |David Gilbert, Independent Contractor. | Two things can be | |Mail: dave@daveg.ca | equal if and only if they | |http://daveg.ca | are precisely opposite. | =========================================================GLO================ From owner-freebsd-performance@FreeBSD.ORG Tue Mar 6 12:47:39 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 30BEB16A407 for ; Tue, 6 Mar 2007 12:47:39 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id F3E1613C442 for ; Tue, 6 Mar 2007 12:47:38 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 6C50C47025; Tue, 6 Mar 2007 07:47:38 -0500 (EST) Date: Tue, 6 Mar 2007 12:47:37 +0000 (GMT) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: David Gilbert In-Reply-To: <17900.25505.245946.235433@canoe.dclg.ca> Message-ID: <20070306124411.H51876@fledge.watson.org> References: <20070224215508.GA41968@xor.obsecurity.org> <45E13410.7020505@he.iki.fi> <20070225071946.GA48242@xor.obsecurity.org> <45E14BAD.80909@he.iki.fi> <20070225084737.GA49231@xor.obsecurity.org> <5a0a9d6f0702260936u3408f8d8rd4cde9234b2f7776@mail.gmail.com> <45E54619.7000503@isc.org> <20070228214200.60cc9b7a@prophet.alphaque.com> <17900.25505.245946.235433@canoe.dclg.ca> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Dinesh Nair , freebsd-performance@freebsd.org Subject: Re: UDP performance. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Mar 2007 12:47:39 -0000 On Mon, 5 Mar 2007, David Gilbert wrote: >>>>>> "Dinesh" == Dinesh Nair writes: > > Dinesh> On Wed, 28 Feb 2007 01:06:33 -0800, Peter Losher > Dinesh> wrote: > >>> Ivan Voras wrote: >>> >>>> I agree in general, but MySQL performance is very exposed as an advocacy >>>> issue - it has traditionally been the source of statements like >>>> "FreeBSD's threading implementation is weak/bad/broken". >>> >>> And these days ISC can't consciously recommend FreeBSD for use on >>> high-traffic DNS servers because UDP performance has (frankly) gone >>> downhill since 5.x. [..snipped..] >>> http://lists.freebsd.org/pipermail/freebsd-net/2006-September/011748.html > > Dinesh> if UDP performance in 6.x and 7.x has dropped, this could even > Dinesh> affect voip applications/servers such as asterisk when run on > Dinesh> FreeBSD. most all use RTP for media traffic and RTP is nearly > Dinesh> always UDP generating up to 50 packets per second per call per > Dinesh> direction. > > Dinesh> 14,000+ packets per second is only about 140 calls. > > Well... again, BIND is not a good indicator of UDP performance. A > non-trivial application can send and receive about 250k pps on moderate > hardware. One of the problems ISC diagnosed had to do with the highly unusual workload pattern of UDP: many different threads simultaneously sending using a single socket leading to unnecessary socket buffer contention. The previously posted patches may help with this problem. So far I've not had any feedback on the RELENG_6 patches, which I'm happy to commit once I've had reports of both good stability and a performance improvement. This patch won't help more traditional UDP workloads, in which aplications use different sockets to serve different instances of the service, so access each socket from only one thread. Portions of the patch would also support future MFC's of other socket-related performance optimizations and cleanups going on in FreeBSD 7.x, such as Andre's recent work on optimized TCP send. Robert N M Watson Computer Laboratory University of Cambridge From owner-freebsd-performance@FreeBSD.ORG Wed Mar 7 13:59:34 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 35B7816A406 for ; Wed, 7 Mar 2007 13:59:34 +0000 (UTC) (envelope-from cryx-freebsd@h3q.com) Received: from mail.h3q.com (mail.h3q.com [217.13.206.148]) by mx1.freebsd.org (Postfix) with ESMTP id C6C4213C494 for ; Wed, 7 Mar 2007 13:59:32 +0000 (UTC) (envelope-from cryx-freebsd@h3q.com) Received: (qmail 80039 invoked from network); 7 Mar 2007 13:32:50 -0000 Received: from unknown (HELO ?10.1.1.245?) (smtpsend@62.220.7.20) by mail.h3q.com with AES256-SHA encrypted SMTP; 7 Mar 2007 13:32:50 -0000 Message-ID: <45EEBF01.8030109@h3q.com> Date: Wed, 07 Mar 2007 14:32:49 +0100 From: Philipp Wuensche User-Agent: Thunderbird 1.5.0.9 (Macintosh/20061207) MIME-Version: 1.0 To: Francisco Reyes References: <0e2001c7574c$26bc15a0$b3db87d4@multiplay.co.uk> <20070224005748.GA25295@xor.obsecurity.org> <45E1C3FC.6040905@he.iki.fi> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: FreeBSD Scaling on 6.2-RELEASE? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2007 13:59:34 -0000 Francisco Reyes wrote: > Petri Helenius writes: > >> The point in threading is that different threads can execute >> simultaneously on multiple CPU's. > > What combination of FreeBSD+Mysql will have multiple threads run by > different CPUs? > > In the few SMP FreeBSD + Mysql setups (mysql 4.X) that I have at work I > only see mysql in one cpu as reported by top. Maybe you should use Strg+H (-H as option) to show threads seperatly in top or take a look at the THR column. greetings, philipp From owner-freebsd-performance@FreeBSD.ORG Wed Mar 7 14:06:08 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 6E1C116A400 for ; Wed, 7 Mar 2007 14:06:08 +0000 (UTC) (envelope-from garcol@postino.it) Received: from abraham.elitel.it (vdisk2.elitel.it [212.34.224.151]) by mx1.freebsd.org (Postfix) with SMTP id B28B613C474 for ; Wed, 7 Mar 2007 14:06:07 +0000 (UTC) (envelope-from garcol@postino.it) Received: (qmail 23265 invoked by uid 65534); 7 Mar 2007 14:06:04 -0000 X-Spam-Checker-Version: SpamAssassin on abraham.elitel.it X-Spam-Level: X-Spam-Status: No, score=-1.6 required=5.2 tests=BAYES_00,NO_REAL_NAME autolearn=no Received: from 212.63.96.97 ([212.63.96.97]) by www.postino.punto.it (IMP) with HTTP for ; Wed, 7 Mar 2007 15:06:04 +0100 Message-ID: <1173276364.45eec6ccb23ff@www.postino.punto.it> Date: Wed, 7 Mar 2007 15:06:04 +0100 From: garcol@postino.it To: "freebsd-performance@freebsd.org" MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.2 X-Originating-IP: 212.63.96.97 Cc: rwatson@abraham.elitel.it, FreeBSD.org@abraham.elitel.it, at@abraham.elitel.it Subject: Re: MFC of UDP socket enhancement for BIND? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2007 14:06:08 -0000 Hi, >For reference, the sosend_copyin.diff applies these changes: > > src/sys/kern/uipc_socket.c:1.253, 1.254, 1.255 Sorry, It's seem doesn't work ... . The function sosend_copyin() where Have I to declare it? Regards Alessandro >The sosend_dgram.diff patch incrementally also applies these changes on top of >sosend_copyin.diff: > > src/sys/kern/uipc_socket.c:1.256 > src/sys/netinet/udp_usrreq.c:1.188 > From owner-freebsd-performance@FreeBSD.ORG Wed Mar 7 22:04:46 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 4BA0116A46C for ; Wed, 7 Mar 2007 22:04:46 +0000 (UTC) (envelope-from cswiger@mac.com) Received: from mail-out4.apple.com (mail-out4.apple.com [17.254.13.23]) by mx1.freebsd.org (Postfix) with ESMTP id D5D9013C441 for ; Wed, 7 Mar 2007 22:04:45 +0000 (UTC) (envelope-from cswiger@mac.com) Received: from relay8.apple.com (relay8.apple.com [17.128.113.38]) by mail-out4.apple.com (8.13.8/8.13.8) with ESMTP id l27LOIao006602; Wed, 7 Mar 2007 13:24:18 -0800 (PST) Received: from relay8.apple.com (unknown [127.0.0.1]) by relay8.apple.com (Symantec Mail Security) with ESMTP id 943C4404EE; Wed, 7 Mar 2007 13:24:18 -0800 (PST) X-AuditID: 11807126-9f797bb0000007dd-5d-45ef2d824546 Received: from [17.214.13.96] (cswiger1.apple.com [17.214.13.96]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by relay8.apple.com (Apple SCV relay) with ESMTP id 828FF40024; Wed, 7 Mar 2007 13:24:18 -0800 (PST) In-Reply-To: References: Mime-Version: 1.0 (Apple Message framework v752.2) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Chuck Swiger Date: Wed, 7 Mar 2007 13:24:17 -0800 To: vivek.prasannan@gmail.com X-Mailer: Apple Mail (2.752.2) X-Brightmail-Tracker: AAAAAA== Cc: freebsd-performance@freebsd.org, User Questions Subject: Re: w hangs before loading X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: User Questions List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2007 22:04:46 -0000 On Mar 7, 2007, at 12:41 PM, Vivek Prasannan wrote: [ ...cross-posting between freebsd-questions and other FreeBSD lists is generally not encouraged; Reply-To: set... ] > When I type the command 'w' it holds for a while before printing > the output. > There is no firewall in the system, load average is below 0.8 and I > suspect > it is something else. Any thoughts on this. The 'w' command attempts to look up the hostnames for any users which are remotely logged in. If you encounter a significant delay, that suggests that reverse DNS (aka PTR record lookups) is not properly configured or is performing slowly in your situation. -- -Chuck From owner-freebsd-performance@FreeBSD.ORG Wed Mar 7 21:09:43 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0DA9216A419 for ; Wed, 7 Mar 2007 21:09:43 +0000 (UTC) (envelope-from vivek.prasannan@gmail.com) Received: from nf-out-0910.google.com (nf-out-0910.google.com [64.233.182.185]) by mx1.freebsd.org (Postfix) with ESMTP id 8C5C013C4BA for ; Wed, 7 Mar 2007 21:09:41 +0000 (UTC) (envelope-from vivek.prasannan@gmail.com) Received: by nf-out-0910.google.com with SMTP id k27so321750nfc for ; Wed, 07 Mar 2007 13:09:41 -0800 (PST) DKIM-Signature: a=rsa-sha1; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:reply-to:to:subject:mime-version:content-type; b=RMZCxTRFkIcs1J/2d3dergnvUvXSmKcwr3G6XgKQLS2mktCjhA2GfemxeabvZqiDL5WBzo31eBs52hkPa3dKPbgIsDmHw2gBHrzNbkvPCxR7cRF3Am0VC94VWMtgKTVnoZExZicwxC3Te+t+/Puzc7+Opaf5mpiwZ0hYYzVty/c= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=received:message-id:date:from:reply-to:to:subject:mime-version:content-type; b=tLWAuKVirifzgEU3VAlbDGNZP/WhYqyMJutJtgGes5m2r/EJa7OIudshCPqcO41ttHH+jO0sgaoDWbQlr2n7MQMIUPrDdCB0Pr1E26lB84bPmOKPskO1a6ucWMm139k4sE2a70P5JvyxmGP21Yz6VrwHAfyX7/1bE/9JOLAc+N4= Received: by 10.82.111.8 with SMTP id j8mr10048563buc.1173300063677; Wed, 07 Mar 2007 12:41:03 -0800 (PST) Received: by 10.82.179.19 with HTTP; Wed, 7 Mar 2007 12:41:03 -0800 (PST) Message-ID: Date: Thu, 8 Mar 2007 02:11:03 +0530 From: "Vivek Prasannan" To: freebsd-performance@freebsd.org, freebsd-questions@freebsd.org MIME-Version: 1.0 X-Mailman-Approved-At: Wed, 07 Mar 2007 22:56:48 +0000 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: w hangs before loading X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: vivek.prasannan@gmail.com List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 07 Mar 2007 21:09:43 -0000 Hi, When I type the command 'w' it holds for a while before printing the output. There is no firewall in the system, load average is below 0.8 and I suspect it is something else. Any thoughts on this. -Vivek From owner-freebsd-performance@FreeBSD.ORG Fri Mar 9 14:04:30 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BEF1916A400 for ; Fri, 9 Mar 2007 14:04:30 +0000 (UTC) (envelope-from cryx-freebsd@h3q.com) Received: from mail.h3q.com (mail.h3q.com [217.13.206.148]) by mx1.freebsd.org (Postfix) with ESMTP id 0B3BA13C4A5 for ; Fri, 9 Mar 2007 14:04:29 +0000 (UTC) (envelope-from cryx-freebsd@h3q.com) Received: (qmail 77990 invoked from network); 9 Mar 2007 14:04:28 -0000 Received: from unknown (HELO ?192.168.23.150?) (smtpsend@85.179.43.128) by mail.h3q.com with AES256-SHA encrypted SMTP; 9 Mar 2007 14:04:28 -0000 Message-ID: <45F1696C.30401@h3q.com> Date: Fri, 09 Mar 2007 15:04:28 +0100 From: Philipp Wuensche User-Agent: Thunderbird 1.5.0.9 (Macintosh/20061207) MIME-Version: 1.0 To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: 2000req/sec with RTL8169 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Mar 2007 14:04:30 -0000 Hi, I administrate a FreeBSD 6-STABLE server with a RTL8169 network card which has 64k buffer for incoming packets. We get about 2000req/sec on our service which results in about 12-15k packets/sec in and out. Packets are about 100-600 bytes. After not using an SMP kernel (it locked up with interrupt) I get about 9-10k interrupts on the re0 interface which works okay until the point where the CPU will do something else (monitoring getting triggered for example) resulting in input errors etc. The service producing the traffic is not very CPU hungry. Seems like the system will not do much more using interrupt. I tried using polling enabled with HZ=1000 set in the kernel configuration but this leads to the system hanging after a while with pings coming through sometimes, so it seems it gets a lot of packetloss. Did the same with SMP enabled. Are there any sysctl setting or any hints which I could use to get polling to work? Unfortunally I don't have the choice to use a better network card and the server only has remote access :-/. greetings, philipp From owner-freebsd-performance@FreeBSD.ORG Fri Mar 9 14:52:21 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id BEC6016A46D for ; Fri, 9 Mar 2007 14:52:21 +0000 (UTC) (envelope-from garcol@postino.it) Received: from abraham.elitel.it (vdisk2.elitel.it [212.34.224.151]) by mx1.freebsd.org (Postfix) with SMTP id 0B21B13C474 for ; Fri, 9 Mar 2007 14:52:18 +0000 (UTC) (envelope-from garcol@postino.it) Received: (qmail 5445 invoked by uid 65534); 9 Mar 2007 14:52:16 -0000 X-Spam-Checker-Version: SpamAssassin on abraham.elitel.it X-Spam-Level: X-Spam-Status: No, score=-1.6 required=5.2 tests=BAYES_00,NO_REAL_NAME autolearn=no Received: from 212.63.96.97 ([212.63.96.97]) by www.postino.punto.it (IMP) with HTTP for ; Fri, 9 Mar 2007 15:52:15 +0100 Message-ID: <1173451935.45f1749febf61@www.postino.punto.it> Date: Fri, 9 Mar 2007 15:52:15 +0100 From: garcol@postino.it To: "freebsd-performance@freebsd.org" MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.2 X-Originating-IP: 212.63.96.97 Subject: Re: 2000req/sec with RTL8169 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Mar 2007 14:52:21 -0000 Hi, you can look at man 4 re and retrive various informations and BUG. You can see TCP/IP checksum offload and descriptor-based DMA mechanism. You must check if the negotation is OK. (No input error, deferred ...) Es. Force to 100Full. Regards Alessandro From owner-freebsd-performance@FreeBSD.ORG Fri Mar 9 15:32:15 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id C458016A407 for ; Fri, 9 Mar 2007 15:32:15 +0000 (UTC) (envelope-from cryx-freebsd@h3q.com) Received: from mail.h3q.com (mail.h3q.com [217.13.206.148]) by mx1.freebsd.org (Postfix) with ESMTP id 1138C13C4B5 for ; Fri, 9 Mar 2007 15:32:14 +0000 (UTC) (envelope-from cryx-freebsd@h3q.com) Received: (qmail 96001 invoked from network); 9 Mar 2007 15:32:12 -0000 Received: from unknown (HELO ?192.168.23.150?) (smtpsend@85.179.43.128) by mail.h3q.com with AES256-SHA encrypted SMTP; 9 Mar 2007 15:32:12 -0000 Message-ID: <45F17DFB.5050201@h3q.com> Date: Fri, 09 Mar 2007 16:32:11 +0100 From: Philipp Wuensche User-Agent: Thunderbird 1.5.0.9 (Macintosh/20061207) MIME-Version: 1.0 To: "freebsd-performance@freebsd.org" References: <1173451935.45f1749febf61@www.postino.punto.it> In-Reply-To: <1173451935.45f1749febf61@www.postino.punto.it> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: 2000req/sec with RTL8169 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 09 Mar 2007 15:32:15 -0000 garcol@postino.it wrote: > Hi, > you can look at > man 4 re > and retrive various informations and BUG. > You can see TCP/IP checksum offload and descriptor-based DMA mechanism. > You must check if the negotation is OK. (No input error, deferred ...) Es. > Force to 100Full. The negotation is okay, 100baseTX with full-duplex. Or do you mean I have to force it because it sometimes loses negotiation? Checksum offloading is enabled too: options=1b. The DMA corruption mentioned in BUGS is interesting but its an onboard NIC anyway, so I don't know if it is connected via 32 or 64bit PCI bus, dmesg doesn't show an error message: re0: port 0xe000-0xe0ff mem 0xf8121000-0xf81210ff irq 16 at device 13.0 on pci0 miibus0: on re0 rgephy0: on miibus0 rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto re0: Ethernet address: 00:16:17:9f:fb:36 re0: [FAST] greetings, philipp From owner-freebsd-performance@FreeBSD.ORG Sat Mar 10 12:18:51 2007 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id E713516A400 for ; Sat, 10 Mar 2007 12:18:51 +0000 (UTC) (envelope-from garcol@postino.it) Received: from abraham.elitel.it (vdisk2.elitel.it [212.34.224.151]) by mx1.freebsd.org (Postfix) with SMTP id 3A8ED13C442 for ; Sat, 10 Mar 2007 12:18:50 +0000 (UTC) (envelope-from garcol@postino.it) Received: (qmail 2179 invoked by uid 65534); 10 Mar 2007 12:18:48 -0000 X-Spam-Checker-Version: SpamAssassin on abraham.elitel.it X-Spam-Level: X-Spam-Status: No, score=-1.6 required=5.2 tests=BAYES_00,NO_REAL_NAME autolearn=no Received: from 212.34.237.250 ([212.34.237.250]) by www.postino.punto.it (IMP) with HTTP for ; Sat, 10 Mar 2007 13:18:48 +0100 Message-ID: <1173529128.45f2a22865021@www.postino.punto.it> Date: Sat, 10 Mar 2007 13:18:48 +0100 From: garcol@postino.it To: "freebsd-performance@freebsd.org" MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit User-Agent: Internet Messaging Program (IMP) 3.2.2 X-Originating-IP: 212.34.237.250 Subject: Re: 2000req/sec with RTL8169 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 10 Mar 2007 12:18:52 -0000 Hi, >The negotation is okay, 100baseTX with full-duplex. Or do you mean I >have to force it because it sometimes loses negotiation? Yes, first of all you'll not have error report by netstat -in or by counters on switch Try to force on freebsd and switch the negotiation to 100full if-needed. If the problem persist, try to disable via ifconfig TXCSUM and RXCSUM. Regards Alessandro