From owner-freebsd-stable@FreeBSD.ORG Fri Nov 18 03:47:01 2011 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0A4EC106566B for ; Fri, 18 Nov 2011 03:47:01 +0000 (UTC) (envelope-from daryl@ci.com.au) Received: from mippet.ci.com.au (mippet.ci.com.au [192.65.182.30]) by mx1.freebsd.org (Postfix) with ESMTP id 45ACB8FC0C for ; Fri, 18 Nov 2011 03:46:59 +0000 (UTC) Received: from mippet.ci.com.au (localhost [127.0.0.1]) by mippet.ci.com.au (8.14.4/8.14.4/CE101231/cmlga) with ESMTP id pAI3ARLH075118 for ; Fri, 18 Nov 2011 14:10:28 +1100 (EST) (envelope-from daryl@mippet.ci.com.au) Received: (from daryl@localhost) by mippet.ci.com.au (8.14.4/8.14.4/Submit) id pAI3ARbZ075115; Fri, 18 Nov 2011 14:10:27 +1100 (EST) (envelope-from daryl) Date: Fri, 18 Nov 2011 14:10:27 +1100 (EST) From: Daryl Sayers Message-Id: <201111180310.pAI3ARbZ075115@mippet.ci.com.au> To: freebsd-stable@freebsd.org X-Scanned-By: MIMEDefang 2.68 on 192.65.182.30 Subject: Low nfs write throughput X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Nov 2011 03:47:01 -0000 Can anyone suggest why I am getting poor write performance from my nfs setup. I have 2 x FreeBSD 8.2-STABLE i386 machines with ASUS P5B-plus mother boards, 4G mem and Dual core 3g processor using 147G 15k Seagate SAS drives with onboard Gb network cards connected to an idle network. The results below show that I get nearly 100Mb/s with a dd over rsh but only 15Mbs using nfs. It improves if I use async but a smbfs mount still beats it. I am using the same file, source and destinations for all tests. I have tried alternate Network cards with no resulting benefit. oguido# dd if=/u0/tmp/D2 | rsh castor dd of=/dsk/ufs/D2 1950511+1 records in 1950511+1 records out 998661755 bytes transferred in 10.402483 secs (96002246 bytes/sec) 1950477+74 records in 1950511+1 records out 998661755 bytes transferred in 10.115458 secs (98726301 bytes/sec) (98Mb/s) oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp gemini:/dsk/ufs /mnt oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k 7619+1 records in 7619+1 records out 998661755 bytes transferred in 62.570260 secs (15960646 bytes/sec) (15Mb/s) oguido# mount -t nfs -o wsize=65536,rsize=65536,tcp,async gemini:/dsk/ufs /mnt oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k 7619+1 records in 7619+1 records out 998661755 bytes transferred in 50.697024 secs (19698627 bytes/sec) (19Mb/s) oguido# mount -t smbfs //gemini/ufs /mnt oguido# dd if=/u0/tmp/D2 of=/mnt/tmp/D2 bs=128k 7619+1 records in 7619+1 records out 998661755 bytes transferred in 29.787616 secs (33526072 bytes/sec) (33Mb/s) Looking at a systat -v on the destination I see that the nfs test does not exceed 16KB/t with 100% busy where the other tests reach up to 128KB/t. For the record I get reads of 22Mb/s without and 77Mb/s with async turned on for the nfs mount. A copy of dmesg: ================ Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-STABLE #0: Tue Jul 26 02:49:49 UTC 2011 root@fm32-8-1106:/usr/obj/usr/src/sys/LOCAL i386 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU E6850 @ 3.00GHz (2995.21-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6fb Family = 6 Model = f Stepping = 11 Features=0xbfebfbff Features2=0xe3fd AMD Features=0x20100000 AMD Features2=0x1 TSC: P-state invariant real memory = 4294967296 (4096 MB) avail memory = 3141234688 (2995 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 cryptosoft0: on motherboard acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, bff00000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 cpu0: on acpi0 ACPI Warning: Incorrect checksum in table [OEMB] - 0xBE, should be 0xB1 (20101013/tbutils-354) cpu1: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: irq 16 at device 1.0 on pci0 pci1: on pcib1 mpt0: port 0x7800-0x78ff mem 0xfd4fc000-0xfd4fffff,0xfd4e0000-0xfd4effff irq 16 at device 0.0 on pci1 mpt0: [ITHREAD] mpt0: MPI Version=1.5.18.0 mpt0: Capabilities: ( RAID-0 RAID-1E RAID-1 ) mpt0: 0 Active Volumes (2 Max) mpt0: 0 Hidden Drive Members (14 Max) uhci0: port 0xdc00-0xdc1f irq 16 at device 26.0 on pci0 uhci0: [ITHREAD] uhci0: LegSup = 0x2f00 usbus0: on uhci0 uhci1: port 0xe000-0xe01f irq 17 at device 26.1 on pci0 uhci1: [ITHREAD] uhci1: LegSup = 0x2f00 usbus1: on uhci1 ehci0: mem 0xfebffc00-0xfebfffff irq 18 at device 26.7 on pci0 ehci0: [ITHREAD] usbus2: EHCI version 1.0 usbus2: on ehci0 pci0: at device 27.0 (no driver attached) pcib2: irq 16 at device 28.0 on pci0 pci5: on pcib2 atapci0: port 0xac00-0xac7f mem 0xfd9ffc00-0xfd9ffc7f,0xfd9f8000-0xfd9fbfff irq 16 at device 0.0 on pci5 atapci0: [ITHREAD] ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] pcib3: irq 17 at device 28.1 on pci0 pci4: on pcib3 em0: port 0x9c00-0x9c1f mem 0xfd7e0000-0xfd7fffff,0xfd7c0000-0xfd7dffff irq 17 at device 0.0 on pci4 em0: Using an MSI interrupt em0: [FILTER] em0: Ethernet address: 00:1b:21:04:ac:11 pcib4: irq 19 at device 28.3 on pci0 pci3: on pcib4 age0: mem 0xfd6c0000-0xfd6fffff irq 19 at device 0.0 on pci3 age0: 1280 Tx FIFO, 2364 Rx FIFO age0: Using 1 MSI messages. miibus0: on age0 atphy0: PHY 0 on miibus0 atphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT-FDX, 1000baseT-FDX-master, auto age0: Ethernet address: 00:1a:92:d2:de:cc age0: [FILTER] pcib5: irq 16 at device 28.4 on pci0 pci2: on pcib5 atapci1: port 0x8c00-0x8c07,0x8880-0x8883,0x8800-0x8807,0x8480-0x8483,0x8400-0x840f mem 0xfd5fe000-0xfd5fffff irq 16 at device 0.0 on pci2 atapci1: [ITHREAD] atapci2: on atapci1 atapci2: [ITHREAD] atapci2: AHCI v1.00 controller with 2 3Gbps ports, PM supported ata4: on atapci2 ata4: [ITHREAD] ata5: on atapci2 ata5: [ITHREAD] ata6: on atapci1 ata6: [ITHREAD] uhci2: port 0xd480-0xd49f irq 23 at device 29.0 on pci0 uhci2: [ITHREAD] uhci2: LegSup = 0x2f00 usbus3: on uhci2 uhci3: port 0xd800-0xd81f irq 19 at device 29.1 on pci0 uhci3: [ITHREAD] uhci3: LegSup = 0x2f00 usbus4: on uhci3 uhci4: port 0xd880-0xd89f irq 18 at device 29.2 on pci0 uhci4: [ITHREAD] uhci4: LegSup = 0x2f00 usbus5: on uhci4 ehci1: mem 0xfebff800-0xfebffbff irq 23 at device 29.7 on pci0 ehci1: [ITHREAD] usbus6: EHCI version 1.0 usbus6: on ehci1 pcib6: at device 30.0 on pci0 pci6: on pcib6 vgapci0: mem 0xfb000000-0xfbffffff,0xfeafc000-0xfeafffff,0xfe000000-0xfe7fffff irq 22 at device 1.0 on pci6 fwohci0: port 0xbc00-0xbc7f mem 0xfeafb800-0xfeafbfff irq 21 at device 3.0 on pci6 fwohci0: [ITHREAD] fwohci0: OHCI version 1.10 (ROM=1) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 00:11:d8:00:01:39:52:d3 fwohci0: Phy 1394a available S400, 2 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: on fwohci0 dcons_crom0: on firewire0 dcons_crom0: bus_addr 0x1494000 fwe0: on firewire0 if_fwe0: Fake Ethernet address: 02:11:d8:39:52:d3 fwe0: Ethernet address: 02:11:d8:39:52:d3 fwip0: on firewire0 fwip0: Firewire address: 00:11:d8:00:01:39:52:d3 @ 0xfffe00000000, S400, maxrec 2048 fwohci0: Initiate bus reset fwohci0: fwohci_intr_core: BUS reset fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode isab0: at device 31.0 on pci0 isa0: on isab0 atapci3: port 0xec00-0xec07,0xe880-0xe883,0xe800-0xe807,0xe480-0xe483,0xe400-0xe40f,0xe080-0xe08f irq 19 at device 31.2 on pci0 atapci3: [ITHREAD] ata7: on atapci3 ata7: [ITHREAD] ata8: on atapci3 ata8: [ITHREAD] pci0: at device 31.3 (no driver attached) atapci4: port 0xd400-0xd407,0xd080-0xd083,0xd000-0xd007,0xcc00-0xcc03,0xc880-0xc88f,0xc800-0xc80f irq 19 at device 31.5 on pci0 atapci4: [ITHREAD] ata9: on atapci4 ata9: [ITHREAD] ata10: on atapci4 ata10: [ITHREAD] acpi_button0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 ppc0: port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/9 bytes threshold ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xc7fff pnpid ORM0000 on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ata0 at port 0x1f0-0x1f7,0x3f6 irq 14 on isa0 ata0: [ITHREAD] ata1 at port 0x170-0x177,0x376 irq 15 on isa0 ata1: [ITHREAD] atkbdc0: at port 0x60,0x64 on isa0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] est0: on cpu0 p4tcc0: on cpu0 est1: on cpu1 p4tcc1: on cpu1 Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing. firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) firewire0: bus manager 0 usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 480Mbps High Speed USB v2.0 usbus3: 12Mbps Full Speed USB v1.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 acd0: DVDR at ata7-slave UDMA100 SATA 1.5Gb/s uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub2: 4 ports with 4 removable, self powered uhub6: 6 ports with 6 removable, self powered da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 300.000MB/s transfers da1: Command Queueing enabled da1: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) da2 at mpt0 bus 0 scbus0 target 2 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 300.000MB/s transfers da2: Command Queueing enabled da2: 140014MB (286749488 512 byte sectors: 255H 63S/T 17849C) da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 300.000MB/s transfers da0: Command Queueing enabled da0: 70136MB (143638992 512 byte sectors: 255H 63S/T 8941C) SMP: AP CPU #1 Launched! ugen1.2: at usbus1 ugen3.2: at usbus3 ukbd1: on usbus1 ums0: on usbus3 ums0: 3 buttons and [XYZ] coordinates ID=0 kbd2 at ukbd1 uhid0: on usbus1 Trying to mount root from ufs:/dev/da0s1a -- Daryl Sayers Direct: +612 95525510 Corinthian Engineering Office: +612 95525500 Suite 54, Jones Bay Wharf Fax: +612 95525549 26-32 Pirrama Rd email: daryl@ci.com.au Pyrmont NSW 2009 Australia www: http://www.ci.com.au