From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 00:48:01 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 693D116A400 for ; Sun, 19 Mar 2006 00:48:01 +0000 (UTC) (envelope-from g_jin@lbl.gov) Received: from smtp110.sbc.mail.mud.yahoo.com (smtp110.sbc.mail.mud.yahoo.com [68.142.198.209]) by mx1.FreeBSD.org (Postfix) with SMTP id 09C6543D46 for ; Sun, 19 Mar 2006 00:48:00 +0000 (GMT) (envelope-from g_jin@lbl.gov) Received: (qmail 62692 invoked from network); 19 Mar 2006 00:48:00 -0000 Received: from unknown (HELO ?192.168.2.10?) (jinmtb@sbcglobal.net@68.127.172.58 with plain) by smtp110.sbc.mail.mud.yahoo.com with SMTP; 19 Mar 2006 00:47:59 -0000 Message-ID: <441CAA8D.3020308@lbl.gov> Date: Sat, 18 Mar 2006 16:49:17 -0800 From: "Jin Guojun [VFFS]" User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050108 X-Accept-Language: zh, zh-CN, en MIME-Version: 1.0 To: OxY , Chuck Swiger X-Priority: 3) References: <000a01c64a81$45eb6850$0201a8c0@oxy> <441BF838.1080600@mac.com><000601c64a87$51d7dee0$0201a8c0@oxy> <441BFF26.90807@mac.com> <000e01c64a8f$1b2bec80$0201a8c0@oxy> In-Reply-To: <000e01c64a8f$1b2bec80$0201a8c0@oxy> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sun, 19 Mar 2006 01:23:40 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 00:48:01 -0000 It is still not clear how you did measurement. Did FTP show such % drop? or Did you measure it by other tools? How did you measured incoming traffic? http://field.hu/netstat.txt shows 0 tcp packet drop. Anyway, the first thing first is to have CPU utilization when you see packet drop. This can be get from running "top" or "vmstat 1". As well as run netstat -i -p tcp | grep -i drop If CPU utilization is approaching 100%, either the traffic is no 2 MBps, or some process is taking CPU time. For this reason, "top" is a better tool to use. At this point, if you run netstat command multiple times, you would see drop counter increasing. Once you find out what process takes CPU time, then further tuning can be determined. If CPU utilization is well below 70-80%, then you need to use tcpdump and tcptrace to visualize what cause packet drop, then perform a solution. Jin ----- Original Message ----- From: "OxY" To: "Chuck Swiger" Cc: Sent: Saturday, March 18, 2006 2:23 PM Subject: Re: packet drop with intel gigabit / marwell gigabit > currently i use HZ=2000 > here's the output of netstat -i, -s, and vmstat -i : > (currently i am uploading on the gigabit with ftp, 3 threads) > > Field root# vmstat -i > interrupt total rate > irq0: clk 27503959 1993 > irq1: atkbd0 1 0 > irq3: fxp0 2 0 > irq7: 146 0 > stray irq7 146 0 > irq8: rtc 1765569 127 > irq10: atapci1 2807786 203 > irq11: atapci0 475039 34 > irq13: npx0 1 0 > irq14: ata0 99 0 > Total 32552748 2359 > > Field root# netstat -i > Name Mtu Network Address Ipkts Ierrs Opkts Oerrs > Coll > fxp0 1500 00:a0:c9:8d:79:68 13163545 0 21899372 1 > 0 > fxp0 1500 195.38.96.64/ field 141 - > 6 - - > em0 1500 00:0e:0c:a2:ac:42 68644181 4 66793904 0 > 0 > em0 1500 195.38.96.64/ field 211255811 - > - - > lo0 16384 129622061 0 129622061 > 0 0 > > netstat -s is here: > http://field.hu/netstat.txt > > ----- Original Message ----- > From: "Chuck Swiger" > To: "OxY" > Cc: > Sent: Saturday, March 18, 2006 1:37 PM > Subject: Re: packet drop with intel gigabit / marwell gigabit > > >> OxY wrote: >>> yeah, i googled these settings, but i put them back to default then! >>> i measured iperf performance, and it showed that the packet drop is >>> depending on the system load.. >> >> If you are using the normal interrupt-driven configuration, you should >> look at >> netstat -i, -s, and vmstat -i. If you're turning on device polling, you >> ought >> to retry your testing at higher HZ (try 2000 or 5000): >> >> echo 'kern.hz="2000"' >> /boot/loader.conf >> >> -- >> -Chuck From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 08:59:46 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D2C8116A400 for ; Sun, 19 Mar 2006 08:59:46 +0000 (UTC) (envelope-from oxy@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id 21E5A43D70 for ; Sun, 19 Mar 2006 08:59:41 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (green.field.hu [217.20.130.28]) by green.field.hu (Postfix) with ESMTP id D7AE5119D2E; Sun, 19 Mar 2006 09:59:01 +0100 (CET) Received: from green.field.hu ([217.20.130.28]) by localhost (green.field.hu [217.20.130.28]) (amavisd-new, port 10024) with ESMTP id 13062-02; Sun, 19 Mar 2006 09:59:01 +0100 (CET) Received: from oxy (dsl217-197-187-71.pool.tvnet.hu [217.197.187.71]) by green.field.hu (Postfix) with ESMTP id 52D91119CC4; Sun, 19 Mar 2006 09:59:01 +0100 (CET) Message-ID: <000401c64b33$7561d940$0201a8c0@oxy> From: "OxY" To: "Jin Guojun [VFFS]" , "Chuck Swiger" References: <000a01c64a81$45eb6850$0201a8c0@oxy> <441BF838.1080600@mac.com><000601c64a87$51d7dee0$0201a8c0@oxy> <441BFF26.90807@mac.com> <000e01c64a8f$1b2bec80$0201a8c0@oxy> <441CAA8D.3020308@lbl.gov> Date: Sun, 19 Mar 2006 09:59:40 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2670 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670 X-Virus-Scanned: by Amavisd-new (Spamassassin+Razor2+Pyzor+DCC+Bayes db, Clamd Antivirus) at field.hu Cc: freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 08:59:46 -0000 CPU utilization is 0% if apache is not running and 10-20%, when running and serving 30-40 concurrent downloads (traffic is 3-4MB/s on fxp0 interface) i measured the network performance with 'iperf' util, started the server on my box and benchmarked with a client on the other gigabit machine. it showed 0% packet drop, when apache was not running and 4-7%, when running.. then i checked how it behave when i shut down apache and init a local file copy from one (not system!) disk to other (not system disk either). packet drop was 5-10%, due to the higher load. so i think interrupts or just the load takes the network performance, but i have no clue how to fix it. is it possible that the 2000+ xp amd is just weak to serve such a traffic? (em0 traffic's maximum is 18-23MB/s) i think it might be around 30MB/s without packet drop. I did FTP measurement, because what i want is to copy files with high speed from the other gigabit machine. However FTP needs resources (CPU, I/O, etc), but iperf not! iperf shows 20% CPU utilization when apache not running and when there's no packet drop. ps: Now apache says: 14 requests currently being processed traffic is 1MB/s on fxp0, and em0 benchmark with iperf says (64k udp window size): [ 3] 0.0-10.0 sec 235 MBytes 197 Mbits/sec [ 3] Sent 167375 datagrams [ 3] Server Report: [ 3] 0.0-10.0 sec 229 MBytes 192 Mbits/sec 0.066 ms 4115/167375 (2.5%) the other gigabit machine is OK, because i have 0% packet drop, when my machine is totally idle. ----- Original Message ----- From: "Jin Guojun [VFFS]" To: "OxY" ; "Chuck Swiger" Cc: Sent: Sunday, March 19, 2006 1:49 AM Subject: Re: packet drop with intel gigabit / marwell gigabit > It is still not clear how you did measurement. > Did FTP show such % drop? or Did you measure it by other tools? > How did you measured incoming traffic? > > http://field.hu/netstat.txt shows 0 tcp packet drop. > > Anyway, the first thing first is to have CPU utilization when you see > packet drop. > This can be get from running "top" or "vmstat 1". As well as run > netstat -i -p tcp | grep -i drop > If CPU utilization is approaching 100%, either the traffic is no 2 MBps, > or some process is taking CPU time. For this reason, "top" is a better > tool to use. At this point, if you run netstat command multiple times, > you would see drop counter increasing. > Once you find out what process takes CPU time, then further tuning can be > determined. > > If CPU utilization is well below 70-80%, then you need to use tcpdump and > tcptrace to visualize what cause packet drop, then perform a solution. > > Jin > > ----- Original Message ----- > From: "OxY" > To: "Chuck Swiger" > Cc: > Sent: Saturday, March 18, 2006 2:23 PM > Subject: Re: packet drop with intel gigabit / marwell gigabit > > >> currently i use HZ=2000 >> here's the output of netstat -i, -s, and vmstat -i : >> (currently i am uploading on the gigabit with ftp, 3 threads) >> >> Field root# vmstat -i >> interrupt total rate >> irq0: clk 27503959 1993 >> irq1: atkbd0 1 0 >> irq3: fxp0 2 0 >> irq7: 146 0 >> stray irq7 146 0 >> irq8: rtc 1765569 127 >> irq10: atapci1 2807786 203 >> irq11: atapci0 475039 34 >> irq13: npx0 1 0 >> irq14: ata0 99 0 >> Total 32552748 2359 >> >> Field root# netstat -i >> Name Mtu Network Address Ipkts Ierrs Opkts Oerrs >> Coll >> fxp0 1500 00:a0:c9:8d:79:68 13163545 0 21899372 1 >> 0 >> fxp0 1500 195.38.96.64/ field 141 - 6 - - >> em0 1500 00:0e:0c:a2:ac:42 68644181 4 66793904 0 >> 0 >> em0 1500 195.38.96.64/ field 211255811 - - - >> lo0 16384 129622061 0 129622061 0 0 >> >> netstat -s is here: >> http://field.hu/netstat.txt >> >> ----- Original Message ----- >> From: "Chuck Swiger" >> To: "OxY" >> Cc: >> Sent: Saturday, March 18, 2006 1:37 PM >> Subject: Re: packet drop with intel gigabit / marwell gigabit >> >> >>> OxY wrote: >>>> yeah, i googled these settings, but i put them back to default then! >>>> i measured iperf performance, and it showed that the packet drop is >>>> depending on the system load.. >>> >>> If you are using the normal interrupt-driven configuration, you should >>> look at >>> netstat -i, -s, and vmstat -i. If you're turning on device polling, you >>> ought >>> to retry your testing at higher HZ (try 2000 or 5000): >>> >>> echo 'kern.hz="2000"' >> /boot/loader.conf >>> >>> -- >>> -Chuck From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 11:04:15 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 278FC16A400 for ; Sun, 19 Mar 2006 11:04:15 +0000 (UTC) (envelope-from oxy@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id 95FB643D45 for ; Sun, 19 Mar 2006 11:04:14 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (green.field.hu [217.20.130.28]) by green.field.hu (Postfix) with ESMTP id D7E5A119CD7; Sun, 19 Mar 2006 12:03:33 +0100 (CET) Received: from green.field.hu ([217.20.130.28]) by localhost (green.field.hu [217.20.130.28]) (amavisd-new, port 10024) with ESMTP id 20210-02; Sun, 19 Mar 2006 12:03:33 +0100 (CET) Received: from oxy (dsl217-197-187-71.pool.tvnet.hu [217.197.187.71]) by green.field.hu (Postfix) with ESMTP id 80B91119CC4; Sun, 19 Mar 2006 12:03:33 +0100 (CET) Message-ID: <000601c64b44$db8dcb00$0201a8c0@oxy> From: "OxY" To: "Jin Guojun \(VFFS\)" References: <000a01c64a81$45eb6850$0201a8c0@oxy> <441BF838.1080600@mac.com><000601c64a87$51d7dee0$0201a8c0@oxy> <441BFF26.90807@mac.com> <000e01c64a8f$1b2bec80$0201a8c0@oxy> <441CAA8D.3020308@lbl.gov> <000401c64b33$7561d940$0201a8c0@oxy> <441D3698.10300@lbl.gov> Date: Sun, 19 Mar 2006 12:04:13 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="ISO-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2670 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670 X-Virus-Scanned: by Amavisd-new (Spamassassin+Razor2+Pyzor+DCC+Bayes db, Clamd Antivirus) at field.hu Cc: freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 11:04:15 -0000 ----- Original Message ----- From: "Jin Guojun (VFFS)" To: "OxY" Cc: "Chuck Swiger" ; Sent: Sunday, March 19, 2006 11:46 AM Subject: Re: packet drop with intel gigabit / marwell gigabit > OxY wrote: > >> CPU utilization is 0% if apache is not running and 10-20%, when running >> and >> serving 30-40 concurrent downloads (traffic is 3-4MB/s on fxp0 interface) > > Is the number 3-4MB/s for per stream or the total for all 30-40 streams? > Are these downloads sent to a disk? it's a total, 30-40 streams get the files from two sata disks in raid1 > >> i measured the network performance with 'iperf' util, started the server >> on my box >> and benchmarked with a client on the other gigabit machine. >> it showed 0% packet drop, when apache was not running and 4-7%, when >> running.. >> then i checked how it behave when i shut down apache and init a local >> file copy from one >> (not system!) disk to other (not system disk either). packet drop was >> 5-10%, due to the higher load. >> so i think interrupts or just the load takes the network performance, but >> i have no clue how to fix it. >> is it possible that the 2000+ xp amd is just weak to serve such a >> traffic? (em0 traffic's maximum is 18-23MB/s) >> i think it might be around 30MB/s without packet drop. > > First let's clear the notation -- Is 30MB/s (MBytes/s) = 240Mb/s (Mbit/s) > or MB/s means Mbits/s > If MB/s is MBytes/s and you also write this amount data to a disk, plus > other traffic on fxp0 to disk too, > then your problem may be bonded by memory bandwidth because CPU > utilization is low: > (240 + 24~32) x 2 is about 535 Mbit/s (some chipset/motherboard has low > memory BW for AMD) > If this is true, then this no thing you can tune. What does the chipset > (Motherboard) this machine have? 30MB/s is Megabytes/sec, currently i have 18-20MB/s peak and 15MB/s avg. it's not 535Mbit/s, because i only download it to my machine, no upload. disks are different from apache disks, these disks have own controller in one pci slot. the packet drop is 5-7% at 200Mbit iperf test, 100Mbit drop is around zero. i have on motherboard which has VIA KT400 northbridge http://uk.asus.com/products4.aspx?modelmenu=2&model=226&l1=3&l2=13&l3=62 i have an ABIT BE7 (http://www.abit.com.tw/page/uk/motherboard/motherboard_detail.php?pMODEL_NAME=BE7&fMTYPE=Socket%20478&pPRODINFO=Specifications) resting somewhere, could it improve the network performance with a P4-2.4GHZ(533FSB)? (i don't want Intel-AMD flame :) ) dmesg if available here: http://field.hu/dmesg.txt system disks are ad4 and ad6 in raid 1, these have the files for apache users (first i thought system disks & apache are the problem, tested on other disk, have the same result) From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 12:25:57 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 731BF16A422 for ; Sun, 19 Mar 2006 12:25:57 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: from web30304.mail.mud.yahoo.com (web30304.mail.mud.yahoo.com [68.142.200.97]) by mx1.FreeBSD.org (Postfix) with SMTP id AC0B043D49 for ; Sun, 19 Mar 2006 12:25:56 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 52327 invoked by uid 60001); 19 Mar 2006 12:25:56 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=EktICLtI8OOfBJqNx/Tb0KEw5n9dE5fZ777O3JAxr2gMLxd4sh13TMK8vEcA04pf3lSK49ZT98p1/XA/jNzF0I+2dqOyFw+dGevI1qYXy2xJOwUjGUrkaIyk6fJI8LU4oouRB7nT0sLMTqtL87XwfRtURlAYu9ozHT9gSZU3OcI= ; Message-ID: <20060319122556.52325.qmail@web30304.mail.mud.yahoo.com> Received: from [213.54.75.154] by web30304.mail.mud.yahoo.com via HTTP; Sun, 19 Mar 2006 04:25:56 PST Date: Sun, 19 Mar 2006 04:25:56 -0800 (PST) From: Arne Woerner To: OxY In-Reply-To: <000601c64b44$db8dcb00$0201a8c0@oxy> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Cc: freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 12:25:57 -0000 --- OxY wrote: > i have an ABIT BE7 > (http://www.abit.com.tw/page/uk/motherboard/motherboard_detail.php?pMODEL_NAME=BE7&fMTYPE=Socket%20478&pPRODINFO=Specifications) > resting somewhere, could it improve the network performance > with a P4-2.4GHZ(533FSB)? > Maybe some udp packet drop is normal when there is concurrent network load? I mean: Maybe some timeout makes the UDP packet drop, when there r other (tcp) packets in the network queue... Just my 2 pence... ;-) -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 12:37:42 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CCDAA16A401 for ; Sun, 19 Mar 2006 12:37:42 +0000 (UTC) (envelope-from oxy@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1CC7043D46 for ; Sun, 19 Mar 2006 12:37:42 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (green.field.hu [217.20.130.28]) by green.field.hu (Postfix) with ESMTP id B39E0119D2E; Sun, 19 Mar 2006 13:37:01 +0100 (CET) Received: from green.field.hu ([217.20.130.28]) by localhost (green.field.hu [217.20.130.28]) (amavisd-new, port 10024) with ESMTP id 23782-06; Sun, 19 Mar 2006 13:37:01 +0100 (CET) Received: from oxy (dsl217-197-187-71.pool.tvnet.hu [217.197.187.71]) by green.field.hu (Postfix) with ESMTP id 71907119CC4; Sun, 19 Mar 2006 13:37:01 +0100 (CET) Message-ID: <000601c64b51$eabda890$0201a8c0@oxy> From: "OxY" To: "Arne Woerner" References: <20060319122556.52325.qmail@web30304.mail.mud.yahoo.com> Date: Sun, 19 Mar 2006 13:37:42 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2670 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670 X-Virus-Scanned: by Amavisd-new (Spamassassin+Razor2+Pyzor+DCC+Bayes db, Clamd Antivirus) at field.hu Cc: freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 12:37:42 -0000 but the udp drop came out (10-15%) when i stopped apache (all tcp traffic) and initiated a local disk-to-disk file copy to make some load. ----- Original Message ----- From: "Arne Woerner" To: "OxY" Cc: Sent: Sunday, March 19, 2006 1:25 PM Subject: Re: packet drop with intel gigabit / marwell gigabit > --- OxY wrote: >> i have an ABIT BE7 >> > (http://www.abit.com.tw/page/uk/motherboard/motherboard_detail.php?pMODEL_NAME=BE7&fMTYPE=Socket%20478&pPRODINFO=Specifications) >> resting somewhere, could it improve the network performance >> with a P4-2.4GHZ(533FSB)? >> > Maybe some udp packet drop is normal when there is concurrent > network load? > I mean: Maybe some timeout makes the UDP packet drop, when there r > other (tcp) packets in the network queue... > > Just my 2 pence... ;-) > > -Arne > > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 12:48:02 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DC00216A400 for ; Sun, 19 Mar 2006 12:48:02 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: from web30308.mail.mud.yahoo.com (web30308.mail.mud.yahoo.com [68.142.200.101]) by mx1.FreeBSD.org (Postfix) with SMTP id 63EDF43D45 for ; Sun, 19 Mar 2006 12:48:02 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 1720 invoked by uid 60001); 19 Mar 2006 12:48:01 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=D/blNJrNaxtMTDgFX8amio6pmLRjCaXKGIreLVLexfNiWAa75xIIBFaTJgeZJpIpeIOnR0ewkNvGZEzILfKK8kM+MMYWZCv6zEINFuhy1X5FWSuQY28AprSpytQENHW2p/LzHgkA29z2iQ9gDWuQMpmhRscLD0bjOs8HVBHy2DY= ; Message-ID: <20060319124801.1717.qmail@web30308.mail.mud.yahoo.com> Received: from [213.54.75.154] by web30308.mail.mud.yahoo.com via HTTP; Sun, 19 Mar 2006 04:48:01 PST Date: Sun, 19 Mar 2006 04:48:01 -0800 (PST) From: Arne Woerner To: OxY In-Reply-To: <000601c64b51$eabda890$0201a8c0@oxy> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Cc: freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 12:48:02 -0000 --- OxY wrote: > but the udp drop came out (10-15%) when i stopped apache (all > tcp traffic) and initiated a local disk-to-disk file copy to make some > load. > Ok... That lets my idea look wrong... :-)) Then it might be the main board like somebody else wrote some minutes ago (maybe ur main board cannot move so much data so quickly)? My Athlon XP 2400+ can do more than 2000Mbit/sec, when I do dd if=/dev/zero of=/dev/null bs=1m count=1000 Maybe u want to try the same on ur "patient"? -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 12:53:13 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0C9DC16A401 for ; Sun, 19 Mar 2006 12:53:13 +0000 (UTC) (envelope-from oxy@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7048A43D4C for ; Sun, 19 Mar 2006 12:53:12 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (green.field.hu [217.20.130.28]) by green.field.hu (Postfix) with ESMTP id C180C119CD7; Sun, 19 Mar 2006 13:52:32 +0100 (CET) Received: from green.field.hu ([217.20.130.28]) by localhost (green.field.hu [217.20.130.28]) (amavisd-new, port 10024) with ESMTP id 26105-01; Sun, 19 Mar 2006 13:52:32 +0100 (CET) Received: from oxy (dsl217-197-187-71.pool.tvnet.hu [217.197.187.71]) by green.field.hu (Postfix) with ESMTP id 73EA7119CC4; Sun, 19 Mar 2006 13:52:32 +0100 (CET) Message-ID: <000e01c64b54$15b99de0$0201a8c0@oxy> From: "OxY" To: "Arne Woerner" References: <20060319124801.1717.qmail@web30308.mail.mud.yahoo.com> Date: Sun, 19 Mar 2006 13:53:13 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="ISO-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2670 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670 X-Virus-Scanned: by Amavisd-new (Spamassassin+Razor2+Pyzor+DCC+Bayes db, Clamd Antivirus) at field.hu Cc: freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 12:53:13 -0000 # with apache running: Field root# dd if=/dev/zero of=/dev/null bs=1m count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 2.626162 secs (399280768 bytes/sec) #without apache: Field root# dd if=/dev/zero of=/dev/null bs=1m count=1000 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 2.193038 secs (478138497 bytes/sec) ----- Original Message ----- From: "Arne Woerner" To: "OxY" Cc: Sent: Sunday, March 19, 2006 1:48 PM Subject: Re: packet drop with intel gigabit / marwell gigabit > --- OxY wrote: >> but the udp drop came out (10-15%) when i stopped apache (all >> tcp traffic) and initiated a local disk-to-disk file copy to > make some >> load. >> > Ok... That lets my idea look wrong... :-)) > > Then it might be the main board like somebody else wrote some > minutes ago (maybe ur main board cannot move so much data so > quickly)? > My Athlon XP 2400+ can do more than 2000Mbit/sec, when I do > dd if=/dev/zero of=/dev/null bs=1m count=1000 > Maybe u want to try the same on ur "patient"? > > -Arne > > > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 12:56:35 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 05D0C16A401 for ; Sun, 19 Mar 2006 12:56:35 +0000 (UTC) (envelope-from sten@blinkenlights.nl) Received: from ford.blinkenlights.nl (ford.blinkenlights.nl [213.204.211.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8452443D46 for ; Sun, 19 Mar 2006 12:56:34 +0000 (GMT) (envelope-from sten@blinkenlights.nl) Received: from tea.blinkenlights.nl (tea.blinkenlights.nl [192.168.1.21]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by ford.blinkenlights.nl (Postfix) with ESMTP id 9638CBDBC; Sun, 19 Mar 2006 13:56:31 +0100 (CET) Received: by tea.blinkenlights.nl (Postfix, from userid 101) id 2A178276; Sun, 19 Mar 2006 13:56:31 +0100 (CET) Received: from localhost (localhost [127.0.0.1]) by tea.blinkenlights.nl (Postfix) with ESMTP id 0FDD918F; Sun, 19 Mar 2006 13:56:31 +0100 (CET) Date: Sun, 19 Mar 2006 13:56:30 +0100 (CET) From: Sten Spans To: Arne Woerner In-Reply-To: <20060319124801.1717.qmail@web30308.mail.mud.yahoo.com> Message-ID: References: <20060319124801.1717.qmail@web30308.mail.mud.yahoo.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-performance@freebsd.org, OxY Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 12:56:35 -0000 On Sun, 19 Mar 2006, Arne Woerner wrote: > --- OxY wrote: >> but the udp drop came out (10-15%) when i stopped apache (all >> tcp traffic) and initiated a local disk-to-disk file copy to > make some >> load. >> > Ok... That lets my idea look wrong... :-)) > > Then it might be the main board like somebody else wrote some > minutes ago (maybe ur main board cannot move so much data so > quickly)? > My Athlon XP 2400+ can do more than 2000Mbit/sec, when I do > dd if=/dev/zero of=/dev/null bs=1m count=1000 > Maybe u want to try the same on ur "patient"? The main thing to look out for is the pci-bus. A normal 33mhz/64bits pci bus can only do about 100 megabyte, if storage and and network are both connected to this bus contention will happen. (70mb ide, and 500mbit network just won't fit) Which is why most newer motherboards have dedicated paths for network and sata. This is also why serverboards have faster/wider and multiple pci buses, and why pci-e has dedicated paths for each slot. -- Sten Spans "There is a crack in everything, that's how the light gets in." Leonard Cohen - Anthem From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 13:09:14 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9AE4516A401 for ; Sun, 19 Mar 2006 13:09:14 +0000 (UTC) (envelope-from oxy@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1DD0D43D48 for ; Sun, 19 Mar 2006 13:09:13 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (green.field.hu [217.20.130.28]) by green.field.hu (Postfix) with ESMTP id 49C1D119D2E; Sun, 19 Mar 2006 14:08:33 +0100 (CET) Received: from green.field.hu ([217.20.130.28]) by localhost (green.field.hu [217.20.130.28]) (amavisd-new, port 10024) with ESMTP id 26048-06; Sun, 19 Mar 2006 14:08:33 +0100 (CET) Received: from oxy (dsl217-197-187-71.pool.tvnet.hu [217.197.187.71]) by green.field.hu (Postfix) with ESMTP id E7353119CC4; Sun, 19 Mar 2006 14:08:32 +0100 (CET) Message-ID: <000c01c64b56$52465710$0201a8c0@oxy> From: "OxY" To: "Sten Spans" , "Arne Woerner" References: <20060319124801.1717.qmail@web30308.mail.mud.yahoo.com> Date: Sun, 19 Mar 2006 14:09:14 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="ISO-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2670 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670 X-Virus-Scanned: by Amavisd-new (Spamassassin+Razor2+Pyzor+DCC+Bayes db, Clamd Antivirus) at field.hu Cc: freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 13:09:14 -0000 ----- Original Message ----- From: "Sten Spans" To: "Arne Woerner" Cc: "OxY" ; Sent: Sunday, March 19, 2006 1:56 PM Subject: Re: packet drop with intel gigabit / marwell gigabit > On Sun, 19 Mar 2006, Arne Woerner wrote: > >> --- OxY wrote: >> Then it might be the main board like somebody else wrote some >> minutes ago (maybe ur main board cannot move so much data so >> quickly)? >> My Athlon XP 2400+ can do more than 2000Mbit/sec, when I do >> dd if=/dev/zero of=/dev/null bs=1m count=1000 >> Maybe u want to try the same on ur "patient"? > > The main thing to look out for is the pci-bus. > A normal 33mhz/64bits pci bus can only do about > 100 megabyte, if storage and and network are both > connected to this bus contention will happen. > > (70mb ide, and 500mbit network just won't fit) i have ~50mbit traffic on fxp0 x2 disk read = 100Mbit and currently have ~200-250mbit on em0 x 2 disk write = 400-500Mbit so 500Mbit total with disk r/w.. both network cards and all disks are in pci slot (33mhz/32bit) > -- > Sten Spans > > "There is a crack in everything, that's how the light gets in." > Leonard Cohen - Anthem From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 10:45:41 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9407B16A420 for ; Sun, 19 Mar 2006 10:45:41 +0000 (UTC) (envelope-from g_jin@lbl.gov) Received: from smtp107.sbc.mail.mud.yahoo.com (smtp107.sbc.mail.mud.yahoo.com [68.142.198.206]) by mx1.FreeBSD.org (Postfix) with SMTP id 2841143D49 for ; Sun, 19 Mar 2006 10:45:38 +0000 (GMT) (envelope-from g_jin@lbl.gov) Received: (qmail 304 invoked from network); 19 Mar 2006 10:45:38 -0000 Received: from unknown (HELO ?192.168.2.9?) (jinmtb@sbcglobal.net@68.127.172.58 with plain) by smtp107.sbc.mail.mud.yahoo.com with SMTP; 19 Mar 2006 10:45:38 -0000 Message-ID: <441D3698.10300@lbl.gov> Date: Sun, 19 Mar 2006 02:46:48 -0800 From: "Jin Guojun (VFFS)" User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.2) Gecko/20040804 Netscape/7.2 (ax) X-Accept-Language: en-us, en MIME-Version: 1.0 To: OxY References: <000a01c64a81$45eb6850$0201a8c0@oxy> <441BF838.1080600@mac.com><000601c64a87$51d7dee0$0201a8c0@oxy> <441BFF26.90807@mac.com> <000e01c64a8f$1b2bec80$0201a8c0@oxy> <441CAA8D.3020308@lbl.gov> <000401c64b33$7561d940$0201a8c0@oxy> In-Reply-To: <000401c64b33$7561d940$0201a8c0@oxy> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Sun, 19 Mar 2006 13:22:02 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 10:45:41 -0000 OxY wrote: > CPU utilization is 0% if apache is not running and 10-20%, when > running and > serving 30-40 concurrent downloads (traffic is 3-4MB/s on fxp0 interface) Is the number 3-4MB/s for per stream or the total for all 30-40 streams? Are these downloads sent to a disk? > i measured the network performance with 'iperf' util, started the > server on my box > and benchmarked with a client on the other gigabit machine. > it showed 0% packet drop, when apache was not running and 4-7%, when > running.. > then i checked how it behave when i shut down apache and init a local > file copy from one > (not system!) disk to other (not system disk either). packet drop was > 5-10%, due to the higher load. > so i think interrupts or just the load takes the network performance, > but i have no clue how to fix it. > is it possible that the 2000+ xp amd is just weak to serve such a > traffic? (em0 traffic's maximum is 18-23MB/s) > i think it might be around 30MB/s without packet drop. First let's clear the notation -- Is 30MB/s (MBytes/s) = 240Mb/s (Mbit/s) or MB/s means Mbits/s If MB/s is MBytes/s and you also write this amount data to a disk, plus other traffic on fxp0 to disk too, then your problem may be bonded by memory bandwidth because CPU utilization is low: (240 + 24~32) x 2 is about 535 Mbit/s (some chipset/motherboard has low memory BW for AMD) If this is true, then this no thing you can tune. What does the chipset (Motherboard) this machine have? > I did FTP measurement, because what i want is to copy files with high > speed from the > other gigabit machine. However FTP needs resources (CPU, I/O, etc), > but iperf not! > iperf shows 20% CPU utilization when apache not running and when > there's no packet drop. > > ps: Now apache says: 14 requests currently being processed > traffic is 1MB/s on fxp0, and em0 benchmark with iperf says (64k udp > window size): > > [ 3] 0.0-10.0 sec 235 MBytes 197 Mbits/sec > [ 3] Sent 167375 datagrams > [ 3] Server Report: > [ 3] 0.0-10.0 sec 229 MBytes 192 Mbits/sec 0.066 ms > 4115/167375 (2.5%) > > the other gigabit machine is OK, because i have 0% packet drop, when > my machine is totally idle. From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 13:22:29 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3C38C16A401 for ; Sun, 19 Mar 2006 13:22:29 +0000 (UTC) (envelope-from sthalik@tehran.lain.pl) Received: from tehran.lain.pl (tehran.lain.pl [85.221.230.102]) by mx1.FreeBSD.org (Postfix) with ESMTP id A5A2A43D45 for ; Sun, 19 Mar 2006 13:22:28 +0000 (GMT) (envelope-from sthalik@tehran.lain.pl) Received: from sthalik by tehran.lain.pl with local (envelope-from ) id 1FKxrU-000H4S-EU for freebsd-performance@freebsd.org; Sun, 19 Mar 2006 14:22:24 +0100 Date: Sun, 19 Mar 2006 14:22:24 +0100 From: Stanislaw Halik To: freebsd-performance@freebsd.org Message-ID: <20060319132224.GA57147@tehran.lain.pl> Mail-Followup-To: freebsd-performance@freebsd.org References: <000a01c64a81$45eb6850$0201a8c0@oxy> <441BFF26.90807@mac.com> <000e01c64a8f$1b2bec80$0201a8c0@oxy> <000401c64a91$c3962c30$0201a8c0@oxy> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="G4iJoqBmSsgzjUCe" Content-Disposition: inline In-Reply-To: <000401c64a91$c3962c30$0201a8c0@oxy> X-PGP-Key: http://tehran.lain.pl/public.key User-Agent: Mutt/1.5.11 Sender: Stanislaw Halik Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 13:22:29 -0000 --G4iJoqBmSsgzjUCe Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Mar 18, 2006, OxY wrote: > i increased hz from 2000 to 5000, now the packet loss is decreased > from 5-6% to 0.6-0,8% !!! > huge improve! > should i increase hz more? won't increasing HZ over 1000 break TCP support? from <20051216134759.795206f3.dokas@oitsec.umn.edu> on freebsd-pf@: #v+ > So it's not that far off, the server seems to increment timestamps at > 0.5 ms per tick (2 kHz), instead of the RFC mandated 1 ms (1 kHz). [...] Bingo (I think). I found the following in the firewall's kernel config: =20 options HZ=3D2000 #v- regards, --=20 w. --G4iJoqBmSsgzjUCe Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (FreeBSD) iD8DBQFEHVsQadU+vjT62TERArLqAJsGL82w09FgGuBBusLj8F9GayqhYQCfXXP+ mXWt0ua0UKlN+kDByCvF9+I= =fBGq -----END PGP SIGNATURE----- --G4iJoqBmSsgzjUCe-- From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 17:02:47 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id AE33716A400; Sun, 19 Mar 2006 17:02:47 +0000 (UTC) (envelope-from oxy@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id 787AF43D45; Sun, 19 Mar 2006 17:02:46 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (green.field.hu [217.20.130.28]) by green.field.hu (Postfix) with ESMTP id 10211119D54; Sun, 19 Mar 2006 18:02:04 +0100 (CET) Received: from green.field.hu ([217.20.130.28]) by localhost (green.field.hu [217.20.130.28]) (amavisd-new, port 10024) with ESMTP id 42013-07; Sun, 19 Mar 2006 18:02:03 +0100 (CET) Received: from oxy (dsl217-197-187-71.pool.tvnet.hu [217.197.187.71]) by green.field.hu (Postfix) with ESMTP id C783C119CF5; Sun, 19 Mar 2006 18:02:03 +0100 (CET) Message-ID: <000e01c64b76$f2247ee0$0201a8c0@oxy> From: "OxY" To: "Kevin Day" References: <000a01c64a81$45eb6850$0201a8c0@oxy> <9A8985D2-816C-4AF8-9E4A-116EA6BAEEE7@dragondata.com> Date: Sun, 19 Mar 2006 18:02:46 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2670 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670 X-Virus-Scanned: by Amavisd-new (Spamassassin+Razor2+Pyzor+DCC+Bayes db, Clamd Antivirus) at field.hu Cc: freebsd-net@freebsd.org, freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 17:02:47 -0000 okay, i will try it in a couple days ----- Original Message ----- From: "Kevin Day" To: "OxY" Cc: ; Sent: Sunday, March 19, 2006 5:52 PM Subject: Re: packet drop with intel gigabit / marwell gigabit > > > On Mar 18, 2006, at 5:44 AM, OxY wrote: > >> hi! >> >> i had the packet drop problem with the marwell yukon gigabitcard: >> (system is an amd 2000+xp, 512mb ram, fbsd 6.0-p5) >> >> when the apache ran, with no http, just used to share files and the >> traffic was >> 2-2,5MB/S i had 14-17% packet drop on the gigabit interface.. >> with the sysctl i succesfully pulled it down to 12-14%, but it was >> terrible, >> so i bought an intel pro/1000 gt. >> with this i have 3-6% drop with same traffic load on the other >> interface.. >> when i stop the apache packet drop falls down to 0-0.1%, which is >> great. >> but with apache it's terrible.. > > > Just on a hunch, can you try putting the card in a different PCI > slot? There may be interrupt routing issues. > > > From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 16:52:56 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BFAD616A400; Sun, 19 Mar 2006 16:52:56 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from tokyo01.jp.mail.your.org (tokyo01.jp.mail.your.org [204.9.54.5]) by mx1.FreeBSD.org (Postfix) with ESMTP id 648DC43D46; Sun, 19 Mar 2006 16:52:56 +0000 (GMT) (envelope-from toasty@dragondata.com) Received: from mail.your.org (server3-a.your.org [64.202.112.67]) by tokyo01.jp.mail.your.org (Postfix) with ESMTP id F0EE52AD576D; Sun, 19 Mar 2006 16:52:54 +0000 (UTC) Received: from [69.31.99.38] (pool038.dhcp.your.org [69.31.99.38]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) by mail.your.org (Postfix) with ESMTP id C0D2EA0A44E; Sun, 19 Mar 2006 16:52:53 +0000 (UTC) In-Reply-To: <000a01c64a81$45eb6850$0201a8c0@oxy> References: <000a01c64a81$45eb6850$0201a8c0@oxy> Mime-Version: 1.0 (Apple Message framework v746.2) X-Priority: 3 Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed Message-Id: <9A8985D2-816C-4AF8-9E4A-116EA6BAEEE7@dragondata.com> Content-Transfer-Encoding: 7bit From: Kevin Day Date: Sun, 19 Mar 2006 10:52:35 -0600 To: "OxY" X-Mailer: Apple Mail (2.746.2) X-Mailman-Approved-At: Sun, 19 Mar 2006 17:05:28 +0000 Cc: freebsd-net@freebsd.org, freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 16:52:56 -0000 On Mar 18, 2006, at 5:44 AM, OxY wrote: > hi! > > i had the packet drop problem with the marwell yukon gigabitcard: > (system is an amd 2000+xp, 512mb ram, fbsd 6.0-p5) > > when the apache ran, with no http, just used to share files and the > traffic was > 2-2,5MB/S i had 14-17% packet drop on the gigabit interface.. > with the sysctl i succesfully pulled it down to 12-14%, but it was > terrible, > so i bought an intel pro/1000 gt. > with this i have 3-6% drop with same traffic load on the other > interface.. > when i stop the apache packet drop falls down to 0-0.1%, which is > great. > but with apache it's terrible.. Just on a hunch, can you try putting the card in a different PCI slot? There may be interrupt routing issues. From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 18:31:58 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 239DF16A400; Sun, 19 Mar 2006 18:31:58 +0000 (UTC) (envelope-from oxy@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id D2C7543D7C; Sun, 19 Mar 2006 18:31:48 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (green.field.hu [217.20.130.28]) by green.field.hu (Postfix) with ESMTP id 84482119D2E; Sun, 19 Mar 2006 19:31:06 +0100 (CET) Received: from green.field.hu ([217.20.130.28]) by localhost (green.field.hu [217.20.130.28]) (amavisd-new, port 10024) with ESMTP id 48858-02; Sun, 19 Mar 2006 19:31:06 +0100 (CET) Received: from oxy (dsl217-197-187-71.pool.tvnet.hu [217.197.187.71]) by green.field.hu (Postfix) with ESMTP id 42267119CC4; Sun, 19 Mar 2006 19:31:06 +0100 (CET) Message-ID: <000b01c64b83$62ccdd70$0201a8c0@oxy> From: "OxY" To: "Bjoern A. Zeeb" References: <000a01c64a81$45eb6850$0201a8c0@oxy> <9A8985D2-816C-4AF8-9E4A-116EA6BAEEE7@dragondata.com> <000e01c64b76$f2247ee0$0201a8c0@oxy> <20060319182105.H2181@maildrop.int.zabbadoz.net> Date: Sun, 19 Mar 2006 19:31:49 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="ISO-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2670 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670 X-Virus-Scanned: by Amavisd-new (Spamassassin+Razor2+Pyzor+DCC+Bayes db, Clamd Antivirus) at field.hu Cc: freebsd-net@freebsd.org, freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 18:31:58 -0000 i changed sk to em. how could i measure speed or benchmark the network performance? ----- Original Message ----- From: "Bjoern A. Zeeb" To: "OxY" Cc: ; Sent: Sunday, March 19, 2006 7:26 PM Subject: Re: packet drop with intel gigabit / marwell gigabit > On Sun, 19 Mar 2006, OxY wrote: > > Hi, > >>> Just on a hunch, can you try putting the card in a different PCI slot? >>> There may be interrupt routing issues. >>> >> okay, i will try it in a couple days > > the card also has a sysctl for intr moderation. See man 4 sk. The > default changed with Pyun's updated driver, I think, but you could > play with that too. > > Further I still have the feeling that your measurings are not > comparable. > > -- > Bjoern A. Zeeb bzeeb at Zabbadoz dot NeT From owner-freebsd-performance@FreeBSD.ORG Sun Mar 19 18:30:12 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 466A416A41F; Sun, 19 Mar 2006 18:30:12 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from transport.cksoft.de (transport.cksoft.de [62.111.66.27]) by mx1.FreeBSD.org (Postfix) with ESMTP id D118743D60; Sun, 19 Mar 2006 18:30:10 +0000 (GMT) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from transport.cksoft.de (localhost [127.0.0.1]) by transport.cksoft.de (Postfix) with ESMTP id 4962D2000AB; Sun, 19 Mar 2006 19:30:08 +0100 (CET) Received: by transport.cksoft.de (Postfix, from userid 66) id 69167200005; Sun, 19 Mar 2006 19:30:05 +0100 (CET) Received: from maildrop.int.zabbadoz.net (maildrop.int.zabbadoz.net [10.111.66.10]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.int.zabbadoz.net (Postfix) with ESMTP id 3CE7B444F45; Sun, 19 Mar 2006 18:26:01 +0000 (UTC) Date: Sun, 19 Mar 2006 18:26:01 +0000 (UTC) From: "Bjoern A. Zeeb" X-X-Sender: bz@maildrop.int.zabbadoz.net To: OxY In-Reply-To: <000e01c64b76$f2247ee0$0201a8c0@oxy> Message-ID: <20060319182105.H2181@maildrop.int.zabbadoz.net> References: <000a01c64a81$45eb6850$0201a8c0@oxy> <9A8985D2-816C-4AF8-9E4A-116EA6BAEEE7@dragondata.com> <000e01c64b76$f2247ee0$0201a8c0@oxy> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Virus-Scanned: by AMaViS cksoft-s20020300-20031204bz on transport.cksoft.de X-Mailman-Approved-At: Sun, 19 Mar 2006 18:40:39 +0000 Cc: freebsd-net@freebsd.org, freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Mar 2006 18:30:12 -0000 On Sun, 19 Mar 2006, OxY wrote: Hi, >> Just on a hunch, can you try putting the card in a different PCI slot? >> There may be interrupt routing issues. >> > okay, i will try it in a couple days the card also has a sysctl for intr moderation. See man 4 sk. The default changed with Pyun's updated driver, I think, but you could play with that too. Further I still have the feeling that your measurings are not comparable. -- Bjoern A. Zeeb bzeeb at Zabbadoz dot NeT From owner-freebsd-performance@FreeBSD.ORG Mon Mar 20 03:04:03 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5498616A401 for ; Mon, 20 Mar 2006 03:04:03 +0000 (UTC) (envelope-from g_jin@lbl.gov) Received: from smtp114.sbc.mail.mud.yahoo.com (smtp114.sbc.mail.mud.yahoo.com [68.142.198.213]) by mx1.FreeBSD.org (Postfix) with SMTP id BB09743D48 for ; Mon, 20 Mar 2006 03:04:02 +0000 (GMT) (envelope-from g_jin@lbl.gov) Received: (qmail 2986 invoked from network); 20 Mar 2006 03:04:02 -0000 Received: from unknown (HELO ?192.168.2.10?) (jinmtb@sbcglobal.net@68.127.178.44 with plain) by smtp114.sbc.mail.mud.yahoo.com with SMTP; 20 Mar 2006 03:04:01 -0000 Message-ID: <441E1BF1.6050205@lbl.gov> Date: Sun, 19 Mar 2006 19:05:21 -0800 From: "Jin Guojun [VFFS]" User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050108 X-Accept-Language: zh, zh-CN, en MIME-Version: 1.0 To: OxY References: <000a01c64a81$45eb6850$0201a8c0@oxy> <441BF838.1080600@mac.com><000601c64a87$51d7dee0$0201a8c0@oxy> <441BFF26.90807@mac.com> <000e01c64a8f$1b2bec80$0201a8c0@oxy> <441CAA8D.3020308@lbl.gov> <000401c64b33$7561d940$0201a8c0@oxy> <441D3698.10300@lbl.gov> <000601c64b44$db8dcb00$0201a8c0@oxy> In-Reply-To: <000601c64b44$db8dcb00$0201a8c0@oxy> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 20 Mar 2006 03:36:25 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Mar 2006 03:04:03 -0000 OxY wrote: > > ----- Original Message ----- From: "Jin Guojun (VFFS)" > To: "OxY" > Cc: "Chuck Swiger" ; > Sent: Sunday, March 19, 2006 11:46 AM > Subject: Re: packet drop with intel gigabit / marwell gigabit > > >> OxY wrote: >> >>> CPU utilization is 0% if apache is not running and 10-20%, when >>> running and >>> serving 30-40 concurrent downloads (traffic is 3-4MB/s on fxp0 >>> interface) >> >> >> Is the number 3-4MB/s for per stream or the total for all 30-40 >> streams? >> Are these downloads sent to a disk? > > > it's a total, 30-40 streams get the files from two sata disks in raid1 > >> .... >> First let's clear the notation -- Is 30MB/s (MBytes/s) = 240Mb/s >> (Mbit/s) or MB/s means Mbits/s >> If MB/s is MBytes/s and you also write this amount data to a disk, >> plus other traffic on fxp0 to disk too, >> then your problem may be bonded by memory bandwidth because CPU >> utilization is low: >> (240 + 24~32) x 2 is about 535 Mbit/s (some chipset/motherboard >> has low memory BW for AMD) >> If this is true, then this no thing you can tune. What does the >> chipset (Motherboard) this machine have? > > > 30MB/s is Megabytes/sec, currently i have 18-20MB/s peak and 15MB/s avg. > it's not 535Mbit/s, because i only download it to my machine, no upload. > disks are different from apache disks, these disks have own controller > in one pci slot. > the packet drop is 5-7% at 200Mbit iperf test, 100Mbit drop is around > zero. > i have on motherboard which has VIA KT400 northbridge > http://uk.asus.com/products4.aspx?modelmenu=2&model=226&l1=3&l2=13&l3=62 Yes, this is one of problem chipset. I bought one about 3 years ago. After one day testing, I returned it for changing a A7V600 (VIA KT600 chipset), which is 30% more memory bandwidth than KT400. A7V600 can only receive max 604 Mb/s TCP, so You can imagine what the KT400 can do :-) I do not have a record (because it is too bad), but taking minimum 25% off, it probably about 420-430 Mb/s (50MB/s). Now you can do the math when the machine also writing data to a disk (assume disk a fast enough). I would expect 2/3 of 430 Mb/s, which is about 280~290 Mb/s (35 MB/s). If you experiment these numbers, you are at there. No improvement you can make further. > > i have an ABIT BE7 > (http://www.abit.com.tw/page/uk/motherboard/motherboard_detail.php?pMODEL_NAME=BE7&fMTYPE=Socket%20478&pPRODINFO=Specifications) > > resting somewhere, could it improve the network performance > with a P4-2.4GHZ(533FSB)? > (i don't want Intel-AMD flame :) ) AMD is good. As I mentioned earlier, it is motherboard shipset make's problem. I like AMD CPU, and I amd writing from A7V600 with AMD XP 2100+ :-) The ABIT motherboard has an Intel 845 chipset, which has very good memory bandwidth. You should be able to saturate your GigE interface on this motherboard. If you have problem to obtain the performance, we can tune it . Good luck! -Jin From owner-freebsd-performance@FreeBSD.ORG Mon Mar 20 05:44:36 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 197DA16A401; Mon, 20 Mar 2006 05:44:36 +0000 (UTC) (envelope-from mv@roq.com) Received: from p4.roq.com (ns1.ecoms.com [207.44.130.137]) by mx1.FreeBSD.org (Postfix) with ESMTP id B54FC43D46; Mon, 20 Mar 2006 05:44:35 +0000 (GMT) (envelope-from mv@roq.com) Received: from p4.roq.com (localhost.roq.com [127.0.0.1]) by p4.roq.com (Postfix) with ESMTP id D84E14CD1F; Mon, 20 Mar 2006 05:45:15 +0000 (GMT) Received: from [192.168.46.101] (ppp166-27.static.internode.on.net [150.101.166.27]) by p4.roq.com (Postfix) with ESMTP id AE0474CCD9; Mon, 20 Mar 2006 05:45:13 +0000 (GMT) Message-ID: <441E413E.7090006@roq.com> Date: Mon, 20 Mar 2006 16:44:30 +1100 From: Michael Vince User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.7.12) Gecko/20060216 X-Accept-Language: en-us, en MIME-Version: 1.0 To: OxY References: <000a01c64a81$45eb6850$0201a8c0@oxy> <9A8985D2-816C-4AF8-9E4A-116EA6BAEEE7@dragondata.com> <000e01c64b76$f2247ee0$0201a8c0@oxy> <20060319182105.H2181@maildrop.int.zabbadoz.net> <000b01c64b83$62ccdd70$0201a8c0@oxy> In-Reply-To: <000b01c64b83$62ccdd70$0201a8c0@oxy> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV using ClamSMTP Cc: freebsd-net@freebsd.org, freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Mar 2006 05:44:36 -0000 I use netperf which is a pure network traffic tester I also just use basic 'ab/apache' tests which would also test HD/IO if getting large files. For the 'em' driver I have seen some posts/cvs commit updates to the driver saying it now works better without polling then with polling. I think this is in -stable but it might just be in current. I haven't done any testing for a while. OxY wrote: > i changed sk to em. > how could i measure speed or benchmark the network performance? > > ----- Original Message ----- From: "Bjoern A. Zeeb" > > To: "OxY" > Cc: ; > Sent: Sunday, March 19, 2006 7:26 PM > Subject: Re: packet drop with intel gigabit / marwell gigabit > > >> On Sun, 19 Mar 2006, OxY wrote: >> >> Hi, >> >>>> Just on a hunch, can you try putting the card in a different PCI >>>> slot? There may be interrupt routing issues. >>>> >>> okay, i will try it in a couple days >> >> >> the card also has a sysctl for intr moderation. See man 4 sk. The >> default changed with Pyun's updated driver, I think, but you could >> play with that too. >> >> Further I still have the feeling that your measurings are not >> comparable. >> >> -- >> Bjoern A. Zeeb bzeeb at Zabbadoz dot NeT > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" From owner-freebsd-performance@FreeBSD.ORG Mon Mar 20 10:22:18 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 107A816A41F for ; Mon, 20 Mar 2006 10:22:18 +0000 (UTC) (envelope-from oxy@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id 750EE43D46 for ; Mon, 20 Mar 2006 10:22:17 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (green.field.hu [217.20.130.28]) by green.field.hu (Postfix) with ESMTP id 1F8DE119CD7; Mon, 20 Mar 2006 11:22:10 +0100 (CET) Received: from green.field.hu ([217.20.130.28]) by localhost (green.field.hu [217.20.130.28]) (amavisd-new, port 10024) with ESMTP id 87700-02; Mon, 20 Mar 2006 11:22:09 +0100 (CET) Received: from oxy (dsl217-197-187-71.pool.tvnet.hu [217.197.187.71]) by green.field.hu (Postfix) with ESMTP id B2F4C119CC4; Mon, 20 Mar 2006 11:22:09 +0100 (CET) Message-ID: <000601c64c08$2a7b4990$0201a8c0@oxy> From: "OxY" To: "Jin Guojun [VFFS]" References: <000a01c64a81$45eb6850$0201a8c0@oxy> <441BF838.1080600@mac.com><000601c64a87$51d7dee0$0201a8c0@oxy> <441BFF26.90807@mac.com> <000e01c64a8f$1b2bec80$0201a8c0@oxy> <441CAA8D.3020308@lbl.gov> <000401c64b33$7561d940$0201a8c0@oxy> <441D3698.10300@lbl.gov> <000601c64b44$db8dcb00$0201a8c0@oxy> <441E1BF1.6050205@lbl.gov> Date: Mon, 20 Mar 2006 11:22:17 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="ISO-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2670 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670 X-Virus-Scanned: by Amavisd-new (Spamassassin+Razor2+Pyzor+DCC+Bayes db, Clamd Antivirus) at field.hu Cc: freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Mar 2006 10:22:18 -0000 ----- Original Message ----- From: "Jin Guojun [VFFS]" To: "OxY" Cc: Sent: Monday, March 20, 2006 4:05 AM Subject: Re: packet drop with intel gigabit / marwell gigabit > OxY wrote: > >> >> ----- Original Message ----- From: "Jin Guojun (VFFS)" >> To: "OxY" >> Cc: "Chuck Swiger" ; >> Sent: Sunday, March 19, 2006 11:46 AM >> Subject: Re: packet drop with intel gigabit / marwell gigabit >> >> >>> OxY wrote: >>> >>>> CPU utilization is 0% if apache is not running and 10-20%, when running >>>> and >>>> serving 30-40 concurrent downloads (traffic is 3-4MB/s on fxp0 >>>> interface) >>> >>> >>> Is the number 3-4MB/s for per stream or the total for all 30-40 >>> streams? >>> Are these downloads sent to a disk? >> >> >> it's a total, 30-40 streams get the files from two sata disks in raid1 >> >>> .... >>> First let's clear the notation -- Is 30MB/s (MBytes/s) = 240Mb/s >>> (Mbit/s) or MB/s means Mbits/s >>> If MB/s is MBytes/s and you also write this amount data to a disk, plus >>> other traffic on fxp0 to disk too, >>> then your problem may be bonded by memory bandwidth because CPU >>> utilization is low: >>> (240 + 24~32) x 2 is about 535 Mbit/s (some chipset/motherboard has >>> low memory BW for AMD) >>> If this is true, then this no thing you can tune. What does the chipset >>> (Motherboard) this machine have? >> >> >> 30MB/s is Megabytes/sec, currently i have 18-20MB/s peak and 15MB/s avg. >> it's not 535Mbit/s, because i only download it to my machine, no upload. >> disks are different from apache disks, these disks have own controller in >> one pci slot. >> the packet drop is 5-7% at 200Mbit iperf test, 100Mbit drop is around >> zero. >> i have on motherboard which has VIA KT400 northbridge >> http://uk.asus.com/products4.aspx?modelmenu=2&model=226&l1=3&l2=13&l3=62 > > Yes, this is one of problem chipset. I bought one about 3 years ago. > After one day testing, I returned it for changing a A7V600 (VIA KT600 > chipset), > which is 30% more memory bandwidth than KT400. A7V600 can only receive max > 604 Mb/s TCP, so You can imagine what the KT400 can do :-) > I do not have a record (because it is too bad), but taking minimum 25% > off, > it probably about 420-430 Mb/s (50MB/s). Now you can do the math when the > machine also writing data to a disk (assume disk a fast enough). I would > expect > 2/3 of 430 Mb/s, which is about 280~290 Mb/s (35 MB/s). > If you experiment these numbers, you are at there. No improvement you can > make > further. i have doubts, because when i have 3-4MB/s traffic on fxp0 then em0 peak is 18MB/s, but when fxp0 is almost idle, have 500kB/s traffic, then em0 can only do 20MB/s.. > AMD is good. As I mentioned earlier, it is motherboard shipset make's > problem. > I like AMD CPU, and I amd writing from A7V600 with AMD XP 2100+ :-) > > The ABIT motherboard has an Intel 845 chipset, which has very good memory > bandwidth. > You should be able to saturate your GigE interface on this motherboard. > If you have problem to obtain the performance, we can tune it . > Good luck! okay, i check with intel soon :) > > -Jin > From owner-freebsd-performance@FreeBSD.ORG Mon Mar 20 19:48:50 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4D0D116A423 for ; Mon, 20 Mar 2006 19:48:50 +0000 (UTC) (envelope-from g_jin@lbl.gov) Received: from smtp108.sbc.mail.mud.yahoo.com (smtp108.sbc.mail.mud.yahoo.com [68.142.198.207]) by mx1.FreeBSD.org (Postfix) with SMTP id E5C7143D45 for ; Mon, 20 Mar 2006 19:48:49 +0000 (GMT) (envelope-from g_jin@lbl.gov) Received: (qmail 49947 invoked from network); 20 Mar 2006 19:48:49 -0000 Received: from unknown (HELO ?192.168.2.10?) (jinmtb@sbcglobal.net@68.127.178.44 with plain) by smtp108.sbc.mail.mud.yahoo.com with SMTP; 20 Mar 2006 19:48:48 -0000 Message-ID: <441F0771.6030807@lbl.gov> Date: Mon, 20 Mar 2006 11:50:09 -0800 From: "Jin Guojun [VFFS]" User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050108 X-Accept-Language: zh, zh-CN, en MIME-Version: 1.0 To: OxY References: <000a01c64a81$45eb6850$0201a8c0@oxy> <441BF838.1080600@mac.com><000601c64a87$51d7dee0$0201a8c0@oxy> <441BFF26.90807@mac.com> <000e01c64a8f$1b2bec80$0201a8c0@oxy> <441CAA8D.3020308@lbl.gov> <000401c64b33$7561d940$0201a8c0@oxy> <441D3698.10300@lbl.gov> <000601c64b44$db8dcb00$0201a8c0@oxy> <441E1BF1.6050205@lbl.gov> <000601c64c08$2a7b4990$0201a8c0@oxy> In-Reply-To: <000601c64c08$2a7b4990$0201a8c0@oxy> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 20 Mar 2006 19:55:39 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Mar 2006 19:48:50 -0000 OxY wrote: > > ----- Original Message ----- From: "Jin Guojun [VFFS]" > To: "OxY" > Cc: > Sent: Monday, March 20, 2006 4:05 AM > Subject: Re: packet drop with intel gigabit / marwell gigabit > >>>> .... >>>> First let's clear the notation -- Is 30MB/s (MBytes/s) = 240Mb/s >>>> (Mbit/s) or MB/s means Mbits/s >>>> If MB/s is MBytes/s and you also write this amount data to a disk, >>>> plus other traffic on fxp0 to disk too, >>>> then your problem may be bonded by memory bandwidth because CPU >>>> utilization is low: >>>> (240 + 24~32) x 2 is about 535 Mbit/s (some chipset/motherboard >>>> has low memory BW for AMD) >>>> If this is true, then this no thing you can tune. What does the >>>> chipset (Motherboard) this machine have? >>> >>> >>> >>> 30MB/s is Megabytes/sec, currently i have 18-20MB/s peak and 15MB/s >>> avg. >>> it's not 535Mbit/s, because i only download it to my machine, no >>> upload. >>> disks are different from apache disks, these disks have own >>> controller in one pci slot. >>> the packet drop is 5-7% at 200Mbit iperf test, 100Mbit drop is >>> around zero. >>> i have on motherboard which has VIA KT400 northbridge >>> http://uk.asus.com/products4.aspx?modelmenu=2&model=226&l1=3&l2=13&l3=62 >>> >> >> >> Yes, this is one of problem chipset. I bought one about 3 years ago. >> After one day testing, I returned it for changing a A7V600 (VIA KT600 >> chipset), >> which is 30% more memory bandwidth than KT400. A7V600 can only >> receive max >> 604 Mb/s TCP, so You can imagine what the KT400 can do :-) >> I do not have a record (because it is too bad), but taking minimum >> 25% off, >> it probably about 420-430 Mb/s (50MB/s). Now you can do the math when >> the >> machine also writing data to a disk (assume disk a fast enough). I >> would expect >> 2/3 of 430 Mb/s, which is about 280~290 Mb/s (35 MB/s). >> If you experiment these numbers, you are at there. No improvement you >> can make >> further. > > > i have doubts, because when i have 3-4MB/s traffic on fxp0 then em0 peak > is 18MB/s, but when fxp0 is almost idle, have 500kB/s traffic, then > em0 can only > do 20MB/s.. Since you did not get anything better than 35MB/s, then, what is your doubt -- the maximum I/O A7V8X can do? The 35 MB/s is the theoretical ceiling based on 2100+ CPU. 2000+ will be slower. In previous email, you mentioned you had 240 Mb/s (30 MB/s) on em0 with some traffic on fxp0, it is pretty much close to your hardware physical limitation. Forget drop in this figure, because this demonstrated how much hardware can do, rather than lossless transmission. Once you have determined the ceiling, you need to keep a margin for lossless Tx. for other overhead, such as context switch, etc. 20 MB/s is not good enough for this board, you may expect 28-30 MB/s with fine tuning. Unless you will be happy with 28 MB/s, it does not make sense to waste time to try to bump I/O above 30 MB/s for your application if you have another motherboard. Again, this motherboard is designed for entertainment boxes not for network I/O based applications. -Jin From owner-freebsd-performance@FreeBSD.ORG Tue Mar 21 18:42:48 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3458F16A41F for ; Tue, 21 Mar 2006 18:42:48 +0000 (UTC) (envelope-from gthorpe@myrealbox.com) Received: from smtp-send.myrealbox.com (smtp-send.myrealbox.com [151.155.5.143]) by mx1.FreeBSD.org (Postfix) with ESMTP id B4F6543D49 for ; Tue, 21 Mar 2006 18:42:47 +0000 (GMT) (envelope-from gthorpe@myrealbox.com) Received: from gthorpe [149.99.114.61] by myrealbox.com with NetMail ModWeb Module; Tue, 21 Mar 2006 13:42:35 -0500 From: "Gary Thorpe" To: g_jin@ibl.gov Date: Tue, 21 Mar 2006 13:42:35 -0500 X-Mailer: NetMail ModWeb Module X-Sender: gthorpe MIME-Version: 1.0 Message-ID: <1142966555.c7f9603cgthorpe@myrealbox.com> Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Cc: freebsd-performance@freebsd.org, oxy@field.hu Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Mar 2006 18:42:48 -0000 Jin Guojun [VFFS] wrote: > OxY wrote: >=20 >> >> ----- Original Message ----- From: "Jin Guojun [VFFS]" >> To: "OxY" >> Cc: >> Sent: Monday, March 20, 2006 4:05 AM >> Subject: Re: packet drop with intel gigabit / marwell gigabit >> >>>>> .... >>>>> First let's clear the notation -- Is 30MB/s (MBytes/s) =3D 240Mb/s=20 >>>>> (Mbit/s) or MB/s means Mbits/s >>>>> If MB/s is MBytes/s and you also write this amount data to a disk,= =20 >>>>> plus other traffic on fxp0 to disk too, >>>>> then your problem may be bonded by memory bandwidth because CPU=20 >>>>> utilization is low: >>>>> (240 + 24~32) x 2 is about 535 Mbit/s (some chipset/motherboard=20 >>>>> has low memory BW for AMD) >>>>> If this is true, then this no thing you can tune. What does the=20 >>>>> chipset (Motherboard) this machine have? >>>> >>>> >>>> >>>> >>>> 30MB/s is Megabytes/sec, currently i have 18-20MB/s peak and 15MB/s=20 >>>> avg. >>>> it's not 535Mbit/s, because i only download it to my machine, no=20 >>>> upload. >>>> disks are different from apache disks, these disks have own=20 >>>> controller in one pci slot. >>>> the packet drop is 5-7% at 200Mbit iperf test, 100Mbit drop is=20 >>>> around zero. >>>> i have on motherboard which has VIA KT400 northbridge >>>> http://uk.asus.com/products4.aspx?modelmenu=3D2&model=3D226&l1=3D3&l2= =3D13&l3=3D62=20 >>>> >>> >>> >>> >>> Yes, this is one of problem chipset. I bought one about 3 years ago. >>> After one day testing, I returned it for changing a A7V600 (VIA KT600= =20 >>> chipset), >>> which is 30% more memory bandwidth than KT400. A7V600 can only=20 >>> receive max >>> 604 Mb/s TCP, so You can imagine what the KT400 can do :-) >>> I do not have a record (because it is too bad), but taking minimum=20 >>> 25% off, >>> it probably about 420-430 Mb/s (50MB/s). Now you can do the math when= =20 >>> the >>> machine also writing data to a disk (assume disk a fast enough). I=20 >>> would expect >>> 2/3 of 430 Mb/s, which is about 280~290 Mb/s (35 MB/s). >>> If you experiment these numbers, you are at there. No improvement you= =20 >>> can make >>> further. >> >> >> >> i have doubts, because when i have 3-4MB/s traffic on fxp0 then em0 peak >> is 18MB/s, but when fxp0 is almost idle, have 500kB/s traffic, then=20 >> em0 can only >> do 20MB/s.. >=20 >=20 > Since you did not get anything better than 35MB/s, then, what is your=20 > doubt -- > the maximum I/O A7V8X can do? >=20 > The 35 MB/s is the theoretical ceiling based on 2100+ CPU. 2000+ will be= =20 > slower. > In previous email, you mentioned you had 240 Mb/s (30 MB/s) on em0 with= =20 > some > traffic on fxp0, it is pretty much close to your hardware physical=20 > limitation. I thought all modern NICs used bus mastering DMA i.e. not dependent on CPU = for data transfers? In addition, the available memory bandwidth for moder= n CPU's/systems is well over 100 MB/s. DDR400 is 400 MB/s (megabytes per = second). Bus mastering DMA will be limited by the memory or IO bus bandwi= dth primarily. The system bus bandwidth cannot be the problem either: his= motherboard's lowest front side bus speed is 200 MHz * 64-bit width =3D = 1.6 GB/s (gigabytes per second) of peak system bus bandwidth. The limitation of 32-bit/33 MHz PCI is 133 MB/s (again, megabytes not bits)= maximum. Gigabit ethernet requires 125 MB/s (not Mb/s) maximum bandwidth= : 32/33 PCI has enough for bursts but bus contention with disk bandwidth = will reduce the sustained bandwidth. The motherboard in question has an o= ption for integrated gigabit LAN which may bypass the shared PCI bus alto= gether (or it might not). Anyway, the original problem was packet loss and not bandwidth. His CPU is = mostly idle, so that cannot be the reason for packet loss. If 32/33 PCI c= an sustain 133 MB/s then it cannot be a problem because he needs=20 less than this. If it cannot, then packets will arrive too fast from the ne= twork before they can be moved from the board into memory and would cause= the packet loss. Otherwise, his system is capable of achieving what he w= ants in theory and the suboptimal behavior may be due to hardware (e.g. P= CI bus bandwidth not being able to reach 133 MB/s sustained) or software = limitations (e.g inefficient operating system). > Forget drop in this figure, because this demonstrated how much hardware= =20 > can do, > rather than lossless transmission. > Once you have determined the ceiling, you need to keep a margin for=20 > lossless Tx. > for other overhead, such as context switch, etc. > 20 MB/s is not good enough for this board, you may expect 28-30 MB/s with > fine tuning. Unless you will be happy with 28 MB/s, it does not make=20 > sense to > waste time to try to bump I/O above 30 MB/s for your application if you= =20 > have > another motherboard. > Again, this motherboard is designed for entertainment boxes not for netwo= rk > I/O based applications. >=20 > -Jin > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" > > From owner-freebsd-performance@FreeBSD.ORG Wed Mar 22 04:26:57 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 96A3116A401 for ; Wed, 22 Mar 2006 04:26:57 +0000 (UTC) (envelope-from g_jin@lbl.gov) Received: from smtp107.sbc.mail.mud.yahoo.com (smtp107.sbc.mail.mud.yahoo.com [68.142.198.206]) by mx1.FreeBSD.org (Postfix) with SMTP id 333B243D4C for ; Wed, 22 Mar 2006 04:26:57 +0000 (GMT) (envelope-from g_jin@lbl.gov) Received: (qmail 22472 invoked from network); 22 Mar 2006 04:26:56 -0000 Received: from unknown (HELO ?192.168.2.10?) (jinmtb@sbcglobal.net@68.127.178.44 with plain) by smtp107.sbc.mail.mud.yahoo.com with SMTP; 22 Mar 2006 04:26:56 -0000 Message-ID: <4420D25F.6050203@lbl.gov> Date: Tue, 21 Mar 2006 20:28:15 -0800 From: "Jin Guojun [VFFS]" User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050108 X-Accept-Language: zh, zh-CN, en MIME-Version: 1.0 To: Gary Thorpe References: <1142966555.c7f9603cgthorpe@myrealbox.com> In-Reply-To: <1142966555.c7f9603cgthorpe@myrealbox.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 22 Mar 2006 04:45:06 +0000 Cc: freebsd-performance@freebsd.org, oxy@field.hu Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2006 04:26:57 -0000 You are fast away from the real world. This has been explained million times, just like I teach intern student every summer :-) First of all, DDR400 and 200 MHz bus mean nothing -- A DDR 266 + 500MHz CPU system can over perform a DDR 400 + 1.7 GHz CPU system. Another example: Ixxxx 2 CPU was designed with 3 level caches. Supposedly Level 1 to level2 takes 5 cycles Level 2 to level 3 takes 11 cycles What you expect CPU to memory time (cycles) -- CPU to level-1 is one cycle ? you would expect 17 cycles to 20 cycles of total. But it actually takes 210 cycles due to some design issues. Now your 1.6 GB/s reduced to 16MB/s or even worse just based on this factor. Number of other factors affect memory bandwidth, such as bus arbitration. Have you done any memory benchmark on a system before doing such simple calculation? Secondly, DMA moves data from NIC to mbuf, then who moves data from mbuf to user buffer? Not human. It is CPU. When DMA moving data, can CPU moves data simultaneously? DMA takes both I/O bandwidth and memory bandwidth. If your system has only 16 MB/s memory bandwidth, your network throughput is less 8 MB/s, typically below 6.4 MB/s. If you cannot move data fast enough away from NIC, what happens? packet loss! That is why his CPU utilization was low because there was no much data cross CPU. So, that is why I asked him what is the CPU utilization first, then the chipset. This is the basic steps to diagnose network performance. If you know a CPU and chipset for a system, you will know the network performance ceiling for that system, guaranteed. But it does not guarantee you can get that ceiling performance, especially over OC-12 (622 Mb/s) high-speed networks. That requires intensive tuning knowledge for current TCP stack, which is well explained on the Internet by searching for "TCP tuning". -Jin Gary Thorpe wrote: > I thought all modern NICs used bus mastering DMA i.e. not dependent on > CPU for data transfers? In addition, the available memory bandwidth > for modern CPU's/systems is well over 100 MB/s. DDR400 is 400 MB/s > (megabytes per second). Bus mastering DMA will be limited by the > memory or IO bus bandwidth primarily. The system bus bandwidth cannot > be the problem either: his motherboard's lowest front side bus speed > is 200 MHz * 64-bit width = 1.6 GB/s (gigabytes per second) of peak > system bus bandwidth. > >The limitation of 32-bit/33 MHz PCI is 133 MB/s (again, megabytes not bits) maximum. Gigabit ethernet requires 125 MB/s (not Mb/s) maximum bandwidth: 32/33 PCI has enough for bursts but bus contention with disk bandwidth will reduce the sustained bandwidth. The motherboard in question has an option for integrated gigabit LAN which may bypass the shared PCI bus altogether (or it might not). > >Anyway, the original problem was packet loss and not bandwidth. His CPU is mostly idle, so that cannot be the reason for packet loss. If 32/33 PCI can sustain 133 MB/s then it cannot be a problem because he needs >less than this. If it cannot, then packets will arrive too fast from the network before they can be moved from the board into memory and would cause the packet loss. Otherwise, his system is capable of achieving what he wants in theory and the suboptimal behavior may be due to hardware (e.g. PCI bus bandwidth not being able to reach 133 MB/s sustained) or software limitations (e.g inefficient operating system). > > From owner-freebsd-performance@FreeBSD.ORG Wed Mar 22 07:10:24 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 68B1516A41F for ; Wed, 22 Mar 2006 07:10:24 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: from web30305.mail.mud.yahoo.com (web30305.mail.mud.yahoo.com [68.142.200.98]) by mx1.FreeBSD.org (Postfix) with SMTP id E132943D46 for ; Wed, 22 Mar 2006 07:10:23 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 70810 invoked by uid 60001); 22 Mar 2006 07:10:23 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=eQDdyUR2aW7czXTq8YhUe9xX5Z6hsEMlp7T+HD3CvZHLr5edFsVqJLzbsrdLsEILqdMfNDE90xsLjrZOmpIYdQstvMpNc3oLGo+wZPT9j5bFZGsw/FkJTUMTEAAMw4Ti4OzQ1JGSD/7pJ/VW0t/omNd72vREdV/zCMVMpG3x9dg= ; Message-ID: <20060322071023.70808.qmail@web30305.mail.mud.yahoo.com> Received: from [213.54.73.238] by web30305.mail.mud.yahoo.com via HTTP; Tue, 21 Mar 2006 23:10:23 PST Date: Tue, 21 Mar 2006 23:10:23 -0800 (PST) From: Arne Woerner To: "Jin Guojun \[VFFS\]" , Gary Thorpe In-Reply-To: <4420D25F.6050203@lbl.gov> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Cc: freebsd-performance@freebsd.org, oxy@field.hu Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2006 07:10:24 -0000 --- "Jin Guojun [VFFS]" wrote: In you example: > Now your 1.6 GB/s reduced to 16MB/s or even worse just based > on this factor. > What did we show by this <
> test? I thought that would prove the memory bandwidth is about 8Gbit/sec (1GByte/sec; 2 * /2^30). But I can see what u mean: On my Athlon XP 2400+ (this should be faster than 2100+) I just get about 5Gbit/sec with the same test (I use DDR RAM and this 266(133*2?)FSB, I think...)... I dont know so much about hardware... But it sounds plaubsible that the architecture around the CPU influences performence... But it is very interesting for me to learn where the bottleneck is... -Arne P.S.: Arne liked "school TV" until it was shutdown... *sob* __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Wed Mar 22 17:53:41 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 02FB816A424 for ; Wed, 22 Mar 2006 17:53:41 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: from web30309.mail.mud.yahoo.com (web30309.mail.mud.yahoo.com [68.142.200.102]) by mx1.FreeBSD.org (Postfix) with SMTP id 6A14843D55 for ; Wed, 22 Mar 2006 17:53:37 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 59010 invoked by uid 60001); 22 Mar 2006 17:53:36 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=mHKGYrYKuqG8RWpP4cS8a9yI/gz36PQITnyRw3K1Bp5JlZOHsDCYJG9THL/qaGymorEwYex76GN0DtJSXIL1CcfrJSCNPDfLuIkPB/HKdnkdTFtz0PqtOmgImGJ5woJq9KSEe3G+YCXBMKZULRffvkJyaN8AEA2pNENgFgbhJ7Q= ; Message-ID: <20060322175336.59008.qmail@web30309.mail.mud.yahoo.com> Received: from [213.54.73.238] by web30309.mail.mud.yahoo.com via HTTP; Wed, 22 Mar 2006 09:53:36 PST Date: Wed, 22 Mar 2006 09:53:36 -0800 (PST) From: Arne Woerner To: "Jin Guojun \[VFFS\]" In-Reply-To: <442187FE.3060300@lbl.gov> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Cc: freebsd-performance@freebsd.org, Gary Thorpe , oxy@field.hu Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2006 17:53:41 -0000 --- "Jin Guojun [VFFS]" wrote: >Arne Woerner wrote: >> What did we show by this <
> test? I thoughtthat >> would prove the memory bandwidth is about 8Gbit/sec(1GByte/sec; >> 2 * >number>/2^30). >> > It depends on how you use /dev/zero. > dd of=/dev/null if=/dev/zero bs=4k count=100k > tests cache speed % dd of=/dev/null if=/dev/zero bs=4k count=100k 102400+0 records in 102400+0 records out 419430400 bytes transferred in 0.204511 secs (2050894814 bytes/sec) about 32Gbit/sec? > dd of=/dev/null if=/dev/zero bs=4m count=100 > tests memory bandwidth if your cache is less than 2 MB > % dd of=/dev/null if=/dev/zero bs=4m count=100 100+0 records in 100+0 records out 419430400 bytes transferred in 2.587341 secs (162108677 bytes/sec) about 2.4Gbit/sec? I had an mpeg encoder in the background, when i did those benchmarks... :-) > Now you may give me the real memory bandwidth on your > system :-) > I would expect something around 500. > Hmm... 500Mbit/sec? even if i divide 2.4Gbit by 4, i still get 600Mbit/sec on a quite busy (50%) system... Oh... The docu TV series episode is over now and I re-ran the tests: % dd of=/dev/null if=/dev/zero bs=4m count=100 100+0 records in 100+0 records out 419430400 bytes transferred in 1.513610 secs (277106012 bytes/sec) % dd of=/dev/null if=/dev/zero bs=4k count=500k 512000+0 records in 512000+0 records out 2097152000 bytes transferred in 0.945430 secs (2218199591 bytes/sec) > Notice that your memory copy speed will be one half of it. > Why "half"? dd causes two copies but counts each byte just once... Maybe "dd" in combination with /dev/zero is not the right way to measure memory bandwidth? -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Wed Mar 22 18:52:42 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8061E16A401 for ; Wed, 22 Mar 2006 18:52:42 +0000 (UTC) (envelope-from arne_woerner@yahoo.com) Received: from web30312.mail.mud.yahoo.com (web30312.mail.mud.yahoo.com [68.142.201.230]) by mx1.FreeBSD.org (Postfix) with SMTP id 1C5B643D68 for ; Wed, 22 Mar 2006 18:52:42 +0000 (GMT) (envelope-from arne_woerner@yahoo.com) Received: (qmail 98218 invoked by uid 60001); 22 Mar 2006 18:52:41 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=Message-ID:Received:Date:From:Subject:To:Cc:In-Reply-To:MIME-Version:Content-Type:Content-Transfer-Encoding; b=eFFUSnEDnkB7G1mjgRRjZwewpyzJ81BDNUAeAP62Qil35AQNw9lND4r1zQQtJTiPrb4Ud9LCT/OvlTR6KmJzZYnYfI/mYEOi/SNLepSCIEMlaOiMvBLS+TbRr3dfo8VLchHnwHp7QImv6npQqJmdWzvzjLPb9hg8ImFdGCbOZOQ= ; Message-ID: <20060322185241.98216.qmail@web30312.mail.mud.yahoo.com> Received: from [213.54.73.238] by web30312.mail.mud.yahoo.com via HTTP; Wed, 22 Mar 2006 10:52:41 PST Date: Wed, 22 Mar 2006 10:52:41 -0800 (PST) From: Arne Woerner To: "Jin Guojun \[VFFS\]" In-Reply-To: <44219619.7020900@lbl.gov> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 8bit Cc: freebsd-performance@freebsd.org, oxy@field.hu Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2006 18:52:42 -0000 --- "Jin Guojun [VFFS]" wrote: > Even after your program finished, you had only 277 MB/s (DDR memory?), > which is far below a good motherboard. Good motherboards should > have 500 - 900 MB/s memory bandwidth, while expensive motherboards > can have 1-3 GB/s memory bandwidth, which are suitable for 10 Gb/s NIC. > Hmm... Ok... Yes, DDR and 266FSB... So you meant, I would have about 500MByte/sec... Then I am far below that... My formula was: 8*277MByte/sec = 2.16...Gbit/sec -- Since dd reads and writes memory I multiplied that with 2, which results in 4.328...Gbit/sec (50%read, 50%write) throughput... Or does a write(2)-request to /dev/null just return without reading the buffer? If yes, it would be just 2.16Gbit/sec for filling the buffer with zeroes... Then we should look again at the bandwidths in oxy's(?) setting... I thought he just needed 500Mbit/sec alltogether (disc io, NIC io)... > It sounds like you have a A7V8X or similar motherboard, Do you? > It is an ECS K7VMM or K7VMM+ if I recall it correctly... Bought in 2003... Is it easy to explain, why the 266FSB cannot do 8Gbit/sec without problem? I mean: 2*133MHz*32bit=8.3125Gbit/sec... Is the MMU too slow (e. g. due to "cheap" implementation of cache strategies) to utilize the FSB to the maximum? -Arne __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From owner-freebsd-performance@FreeBSD.ORG Wed Mar 22 17:21:59 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B30DA16A400 for ; Wed, 22 Mar 2006 17:21:59 +0000 (UTC) (envelope-from g_jin@lbl.gov) Received: from smtp105.sbc.mail.mud.yahoo.com (smtp105.sbc.mail.mud.yahoo.com [68.142.198.204]) by mx1.FreeBSD.org (Postfix) with SMTP id 7355843D94 for ; Wed, 22 Mar 2006 17:21:51 +0000 (GMT) (envelope-from g_jin@lbl.gov) Received: (qmail 38412 invoked from network); 22 Mar 2006 17:21:49 -0000 Received: from unknown (HELO ?192.168.2.10?) (jinmtb@sbcglobal.net@68.127.178.44 with plain) by smtp105.sbc.mail.mud.yahoo.com with SMTP; 22 Mar 2006 17:21:48 -0000 Message-ID: <442187FE.3060300@lbl.gov> Date: Wed, 22 Mar 2006 09:23:10 -0800 From: "Jin Guojun [VFFS]" User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050108 X-Accept-Language: zh, zh-CN, en To: Arne Woerner References: <20060322071023.70808.qmail@web30305.mail.mud.yahoo.com> In-Reply-To: <20060322071023.70808.qmail@web30305.mail.mud.yahoo.com> Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 22 Mar 2006 21:22:11 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org, Gary Thorpe , oxy@field.hu Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2006 17:21:59 -0000 Arne Woerner wrote: --- "Jin Guojun [VFFS]" [1] wrote: In you example: Now your 1.6 GB/s reduced to 16MB/s or even worse just based on this factor. What did we show by this <
> test? I thought that would prove the memory bandwidth is about 8Gbit/sec (1GByte/sec; 2 * /2^30). It depends on how you use /dev/zero. dd of=/dev/null if=/dev/zero bs=4k count=100k tests cache speed dd of=/dev/null if=/dev/zero bs=4m count=100 tests memory bandwidth if your cache is less than 2 MB Now you may give me the real memory bandwidth on your system :-) I would expect something around 500. Notice that your memory copy speed will be one half of it. /dev/null device really does nothing beside throwing away data. That is, it can be counted as a cost for system call. -Jin References 1. mailto:g_jin@lbl.gov From owner-freebsd-performance@FreeBSD.ORG Wed Mar 22 18:21:59 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 77B2216A426 for ; Wed, 22 Mar 2006 18:21:59 +0000 (UTC) (envelope-from g_jin@lbl.gov) Received: from smtp101.sbc.mail.mud.yahoo.com (smtp101.sbc.mail.mud.yahoo.com [68.142.198.200]) by mx1.FreeBSD.org (Postfix) with SMTP id 5CE0143D62 for ; Wed, 22 Mar 2006 18:21:56 +0000 (GMT) (envelope-from g_jin@lbl.gov) Received: (qmail 62627 invoked from network); 22 Mar 2006 18:21:55 -0000 Received: from unknown (HELO ?192.168.2.10?) (jinmtb@sbcglobal.net@68.127.178.44 with plain) by smtp101.sbc.mail.mud.yahoo.com with SMTP; 22 Mar 2006 18:21:55 -0000 Message-ID: <44219619.7020900@lbl.gov> Date: Wed, 22 Mar 2006 10:23:21 -0800 From: "Jin Guojun [VFFS]" User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050108 X-Accept-Language: zh, zh-CN, en To: Arne Woerner References: <20060322175336.59008.qmail@web30309.mail.mud.yahoo.com> In-Reply-To: <20060322175336.59008.qmail@web30309.mail.mud.yahoo.com> Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 22 Mar 2006 21:22:24 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org, Gary Thorpe , oxy@field.hu Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2006 18:21:59 -0000 Arne Woerner wrote: It depends on how you use /dev/zero. dd of=/dev/null if=/dev/zero bs=4k count=100k tests cache speed % dd of=/dev/null if=/dev/zero bs=4k count=100k 102400+0 records in 102400+0 records out 419430400 bytes transferred in 0.204511 secs (2050894814 bytes/sec) about 32Gbit/sec? If you have 1.8-19.GHz 32-bit CPU with 2 level caches, 16 Gb/s cache speed is about right, not 32 (2x8=16). dd of=/dev/null if=/dev/zero bs=4m count=100 tests memory bandwidth if your cache is less than 2 MB % dd of=/dev/null if=/dev/zero bs=4m count=100 100+0 records in 100+0 records out 419430400 bytes transferred in 2.587341 secs (162108677 bytes/sec) about 2.4Gbit/sec? I had an mpeg encoder in the background, when i did those benchmarks... :-) Now you may give me the real memory bandwidth on your system :-) I would expect something around 500. Hmm... 500Mbit/sec? even if i divide 2.4Gbit by 4, i still get 600Mbit/sec on a quite busy (50%) system... All previous notations are MB, not Mb. Also, 1 Byte is 8 bits not 4 bits. :-) Even after your program finished, you had only 277 MB/s (DDR memory?), which is far below a good motherboard. Good motherboards should have 500 - 900 MB/s memory bandwidth, while expensive motherboards can have 1-3 GB/s memory bandwidth, which are suitable for 10 Gb/s NIC. It sounds like you have a A7V8X or similar motherboard, Do you? -Jin From owner-freebsd-performance@FreeBSD.ORG Wed Mar 22 18:43:44 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4077E16A48E for ; Wed, 22 Mar 2006 18:43:44 +0000 (UTC) (envelope-from g_jin@lbl.gov) Received: from smtp111.sbc.mail.mud.yahoo.com (smtp111.sbc.mail.mud.yahoo.com [68.142.198.210]) by mx1.FreeBSD.org (Postfix) with SMTP id 7A32143D96 for ; Wed, 22 Mar 2006 18:43:27 +0000 (GMT) (envelope-from g_jin@lbl.gov) Received: (qmail 34168 invoked from network); 22 Mar 2006 18:43:27 -0000 Received: from unknown (HELO ?192.168.2.10?) (jinmtb@sbcglobal.net@68.127.178.44 with plain) by smtp111.sbc.mail.mud.yahoo.com with SMTP; 22 Mar 2006 18:43:27 -0000 Message-ID: <44219B25.9000700@lbl.gov> Date: Wed, 22 Mar 2006 10:44:53 -0800 From: "Jin Guojun [VFFS]" User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050108 X-Accept-Language: zh, zh-CN, en MIME-Version: 1.0 To: Arne Woerner References: <20060322175336.59008.qmail@web30309.mail.mud.yahoo.com> In-Reply-To: <20060322175336.59008.qmail@web30309.mail.mud.yahoo.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Wed, 22 Mar 2006 21:22:43 +0000 Cc: freebsd-performance@freebsd.org, Gary Thorpe , oxy@field.hu Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2006 18:43:44 -0000 Arne Woerner wrote: >>Notice that your memory copy speed will be one half of it. >> >> >> >Why "half"? dd causes two copies but counts each byte just once... > >Maybe "dd" in combination with /dev/zero is not the right way to >measure memory bandwidth? > > It depends on how /dev/null implemented. It may just throw data in water. If so, dd only does one CPU to memory access. It is not accurate, but it is a fast way to estimate memory bandwidth. You may write a simple memory copy program to get a better number. If you use 64-bit register (like FP register -- double) to do the copy, on some CPUs, you may double memory copy speed. There are number of tricks to improve system performance. Also, memory copy speed is slightly higher one half of the memory bandwidth. This is because only memory read is 100 % cache missing, while memory write is 100% cache hit. So the memory write speed is based on the last level cache to memory bandwidth + a few cycles overhead for CPU signal the last level cache. For example, 500MB/s memory bandwidth can give you roughly 300 MB/s memory copy speed, not 250 MB/s. -Jin From owner-freebsd-performance@FreeBSD.ORG Wed Mar 22 19:34:44 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4386D16A42D for ; Wed, 22 Mar 2006 19:34:44 +0000 (UTC) (envelope-from howells@kde.org) Received: from mail.devrandom.org.uk (host-84-9-223-82.bulldogdsl.com [84.9.223.82]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0B95D43D80 for ; Wed, 22 Mar 2006 19:34:09 +0000 (GMT) (envelope-from howells@kde.org) Received: from localhost (localhost [127.0.0.1]) by mail.devrandom.org.uk (Postfix) with ESMTP id 453F8FD04B for ; Wed, 22 Mar 2006 19:34:08 +0000 (GMT) Received: from mail.devrandom.org.uk ([127.0.0.1]) by localhost (mail.devrandom.org.uk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 25697-02 for ; Wed, 22 Mar 2006 19:34:04 +0000 (GMT) Received: from [192.168.1.177] (unknown [192.168.1.177]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.devrandom.org.uk (Postfix) with ESMTP id E0EA6FD01D for ; Wed, 22 Mar 2006 19:34:04 +0000 (GMT) From: Chris Howells Organization: K Desktop Environment To: freebsd-performance@freebsd.org Date: Wed, 22 Mar 2006 19:32:44 +0000 User-Agent: KMail/1.9.1 References: <20060322185241.98216.qmail@web30312.mail.mud.yahoo.com> In-Reply-To: <20060322185241.98216.qmail@web30312.mail.mud.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline Message-Id: <200603221932.44334.howells@kde.org> X-Virus-Scanned: amavisd-new at devrandom.org.uk X-Mailman-Approved-At: Wed, 22 Mar 2006 21:22:59 +0000 Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2006 19:34:44 -0000 On Wednesday 22 March 2006 18:52, Arne Woerner wrote: > It is an ECS K7VMM or K7VMM+ if I recall it correctly... Bought in > 2003... > > Is it easy to explain, why the 266FSB cannot do 8Gbit/sec without > problem? I mean: 2*133MHz*32bit=8.3125Gbit/sec... Is the MMU too > slow (e. g. due to "cheap" implementation of cache strategies) to > utilize the FSB to the maximum? I'd be tempted to blame the Via chipset. -- Cheers, Chris Howells -- chris@chrishowells.co.uk, howells@kde.org Web: http://chrishowells.co.uk, PGP ID: 0x33795A2C KDE/Qt/C++/PHP Developer: http://www.kde.org From owner-freebsd-performance@FreeBSD.ORG Wed Mar 22 21:36:01 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2E5AB16A401 for ; Wed, 22 Mar 2006 21:36:01 +0000 (UTC) (envelope-from g_jin@lbl.gov) Received: from smtp114.sbc.mail.mud.yahoo.com (smtp114.sbc.mail.mud.yahoo.com [68.142.198.213]) by mx1.FreeBSD.org (Postfix) with SMTP id 9807743D5F for ; Wed, 22 Mar 2006 21:36:00 +0000 (GMT) (envelope-from g_jin@lbl.gov) Received: (qmail 5482 invoked from network); 22 Mar 2006 21:36:00 -0000 Received: from unknown (HELO ?192.168.2.10?) (jinmtb@sbcglobal.net@68.127.178.44 with plain) by smtp114.sbc.mail.mud.yahoo.com with SMTP; 22 Mar 2006 21:35:59 -0000 Message-ID: <4421C392.1080306@lbl.gov> Date: Wed, 22 Mar 2006 13:37:22 -0800 From: "Jin Guojun [VFFS]" User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050108 X-Accept-Language: zh, zh-CN, en MIME-Version: 1.0 To: Chris Howells References: <20060322185241.98216.qmail@web30312.mail.mud.yahoo.com> <200603221932.44334.howells@kde.org> In-Reply-To: <200603221932.44334.howells@kde.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Thu, 23 Mar 2006 03:01:10 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 22 Mar 2006 21:36:01 -0000 Chris Howells wrote: >On Wednesday 22 March 2006 18:52, Arne Woerner wrote: > > > >>It is an ECS K7VMM or K7VMM+ if I recall it correctly... Bought in >>2003... >> >>Is it easy to explain, why the 266FSB cannot do 8Gbit/sec without >>problem? I mean: 2*133MHz*32bit=8.3125Gbit/sec... Is the MMU too >>slow (e. g. due to "cheap" implementation of cache strategies) to >>utilize the FSB to the maximum? >> Can be everywhere, but it is most likely in north bridge design. >>I'd be tempted to blame the Via chipset. >> >> K7VMM is an older motherboard than A7V8X, and it has similar chipset as A7V8X. From Arne's test, it seems to have better memory bandwidth than A7V8X. In past, VIA designed some good chipset. It is behind others now, but it may have other features we do not know / use. -Jin From owner-freebsd-performance@FreeBSD.ORG Thu Mar 23 05:32:39 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D10F016A400 for ; Thu, 23 Mar 2006 05:32:39 +0000 (UTC) (envelope-from gthorpe@myrealbox.com) Received: from smtp-send.myrealbox.com (smtp-send.myrealbox.com [151.155.5.143]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6A04F43D48 for ; Thu, 23 Mar 2006 05:32:39 +0000 (GMT) (envelope-from gthorpe@myrealbox.com) Received: from gthorpe [149.99.116.204] by myrealbox.com with NetMail ModWeb Module; Thu, 23 Mar 2006 00:32:34 -0500 From: "Gary Thorpe" To: g_jin@lbl.gov Date: Thu, 23 Mar 2006 00:32:34 -0500 X-Mailer: NetMail ModWeb Module X-Sender: gthorpe MIME-Version: 1.0 Message-ID: <1143091954.c7e8491cgthorpe@myrealbox.com> Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Cc: freebsd-performance@freebsd.org, oxy@field.hu Subject: (no subject) X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Mar 2006 05:32:40 -0000 Jin Guojun [VFFS] wrote: > You are fast away from the real world. This has been explained million=20 > times, just like > I teach intern student every summer :-) >=20 > First of all, DDR400 and 200 MHz bus mean nothing -- A DDR 266 + 500MHz= =20 > CPU system > can over perform a DDR 400 + 1.7 GHz CPU system.=20 Given the same chipset+motherboard, no. DDR400 has more bandwidth and a sma= ller latency. Given different chipsets/motherboards, this may be true. Ho= wever, one could also say with accuracy that a 500 Mhz processor can outp= erform the same family running at 1.7 GHz under some conditons but few pe= ople will run to buy 500 Mhz over 1.7 GHz for performance alone. Another example: > Ixxxx 2 CPU was designed with 3 level caches. Supposedly > Level 1 to level2 takes 5 cycles > Level 2 to level 3 takes 11 cycles > What you expect CPU to memory time (cycles) -- CPU to level-1 is one= =20 > cycle ? > you would expect 17 cycles to 20 cycles of total. But it actually=20 > takes 210 cycles > due to some design issues. > Now your 1.6 GB/s reduced to 16MB/s or even worse just based on this=20 > factor. 1.6 Gb/s =3D system bus bandwidth. Cache won't affect this bandwidth. DDR40= 0 has 400 MB/s: only attainable for long sequential accesses of either re= ad or write but not a mix of both. DMA should be able to get near this li= mit (long and sequential, read or write only per transfer). A NIC with bu= s mastering DMA should be able to effectively use the memory bandwidth. > Number of other factors affect memory bandwidth, such as bus arbitration. > Have you done any memory benchmark on a system before doing such simple= =20 > calculation? No, they are just theoretical values telling you the limits of performance.= I asume that a decent implementation can get 75% of the theoretical limi= t at least some of the time under good conditions (like DMA). >=20 > Secondly, DMA moves data from NIC to mbuf, then who moves data from mbuf= =20 > to user buffer? > Not human. It is CPU. When DMA moving data, can CPU moves data=20 > simultaneously? > DMA takes both I/O bandwidth and memory bandwidth. If your system has=20 > only 16 MB/s > memory bandwidth, your network throughput is less 8 MB/s, typically=20 > below 6.4 MB/s. > If you cannot move data fast enough away from NIC, what happens? =20 > packet loss! True, but would this type of packet loss even be measured by the OS? Packet= loss to the OS means some packets were dropped from the software portion= of the network stack right? This means that the NIC has no problems deli= vering it to the OS and the OS has problems delivering it to the user pro= cess. You are arguing that the bandwidth is not sufficient for the processor to d= o this copy out (or page loan out =3D zero copy, only memory management t= ricks) and the software has to drop packets from mbufs when more packets = arrive for UDP. Enough bandwidth is theoretically available for this (muc= h more than required), it may or may not be true that the actual sustaine= d bandwidth is insufficient. I don't think that 1/4 of the bandwidth is = actually available for any reasonable (i.e. not junk) system. > > That is why his CPU utilization was low because there was no much data > cross CPU. > So, that is why I asked him what is the CPU utilization first, then the > chipset. This is > the basic steps to diagnose network performance. > If you know a CPU and chipset for a system, you will know the network > performance > ceiling for that system, guaranteed. But it does not guarantee you can > get that ceiling > performance, especially over OC-12 (622 Mb/s) high-speed networks. That > requires > intensive tuning knowledge for current TCP stack, which is well > explained on the Internet > by searching for "TCP tuning". In this case, bandwidth should not factor in (16 MB/s is low, disks can reg= ularly double this easily). The 1 Gb/s NIC is not being fully used in thi= s case (< 40 MB/s) and the processor is mostly idle. From owner-freebsd-performance@FreeBSD.ORG Thu Mar 23 05:40:22 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A127916A400 for ; Thu, 23 Mar 2006 05:40:22 +0000 (UTC) (envelope-from gthorpe@myrealbox.com) Received: from smtp-send.myrealbox.com (smtp-send.myrealbox.com [151.155.5.143]) by mx1.FreeBSD.org (Postfix) with ESMTP id EB10843D53 for ; Thu, 23 Mar 2006 05:40:21 +0000 (GMT) (envelope-from gthorpe@myrealbox.com) Received: from gthorpe [149.99.116.204] by myrealbox.com with NetMail ModWeb Module; Thu, 23 Mar 2006 00:40:17 -0500 From: "Gary Thorpe" To: g_jin@lbl.gov Date: Thu, 23 Mar 2006 00:40:17 -0500 X-Mailer: NetMail ModWeb Module X-Sender: gthorpe MIME-Version: 1.0 Message-ID: <1143092417.c7f62afcgthorpe@myrealbox.com> Content-Type: text/plain; charset="ISO-8859-1" Content-Transfer-Encoding: quoted-printable Cc: freebsd-performance@freebsd.org, oxy@field.hu Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Mar 2006 05:40:22 -0000 [No subject in first one, sorry for repost] =20 Jin Guojun [VFFS] wrote: > You are fast away from the real world. This has been explained million > times, just like > I teach intern student every summer :-) > > First of all, DDR400 and 200 MHz bus mean nothing -- A DDR 266 + 500MHz > CPU system > can over perform a DDR 400 + 1.7 GHz CPU system. Given the same chipset+motherboard, no. DDR400 has more bandwidth and a sma= ller latency. Given different chipsets/motherboards, this may be true. Ho= wever, one could also say with accuracy that a 500 Mhz processor can outp= erform the same family running at 1.7 GHz under some conditons but few pe= ople will run to buy 500 Mhz over 1.7 GHz for performance alone. Another example: > Ixxxx 2 CPU was designed with 3 level caches. Supposedly > Level 1 to level2 takes 5 cycles > Level 2 to level 3 takes 11 cycles > What you expect CPU to memory time (cycles) -- CPU to level-1 is one > cycle ? > you would expect 17 cycles to 20 cycles of total. But it actually > takes 210 cycles > due to some design issues. > Now your 1.6 GB/s reduced to 16MB/s or even worse just based on this > factor. 1.6 Gb/s =3D system bus bandwidth. Cache won't affect this bandwidth. DDR40= 0 has 400 MB/s: only attainable for long sequential accesses of either re= ad or write but not a mix of both. DMA should be able to get near this li= mit (long and sequential, read or write only per transfer). A NIC with bu= s mastering DMA should be able to effectively use the memory bandwidth. > Number of other factors affect memory bandwidth, such as bus arbitration. > Have you done any memory benchmark on a system before doing such simple > calculation? No, they are just theoretical values telling you the limits of performance.= I asume that a decent implementation can get 75% of the theoretical limi= t at least some of the time under good conditions (like DMA). > > Secondly, DMA moves data from NIC to mbuf, then who moves data from mbuf > to user buffer? > Not human. It is CPU. When DMA moving data, can CPU moves data > simultaneously? > DMA takes both I/O bandwidth and memory bandwidth. If your system has > only 16 MB/s > memory bandwidth, your network throughput is less 8 MB/s, typically > below 6.4 MB/s. > If you cannot move data fast enough away from NIC, what happens? > packet loss! True, but would this type of packet loss even be measured by the OS? Packet= loss to the OS means some packets were dropped from the software portion= of the network stack right? This means that the NIC has no problems deli= vering it to the OS and the OS has problems delivering it to the user pro= cess. You are arguing that the bandwidth is not sufficient for the processor to d= o this copy out (or page loan out =3D zero copy, only memory management t= ricks) and the software has to drop packets from mbufs when more packets = arrive for UDP. Enough bandwidth is theoretically available for this (muc= h more than required), it may or may not be true that the actual sustaine= d bandwidth is insufficient. I don't think that 1/4 of the bandwidth is a= ctually available for any reasonable (i.e. not junk) system. > > That is why his CPU utilization was low because there was no much data > cross CPU. > So, that is why I asked him what is the CPU utilization first, then the > chipset. This is > the basic steps to diagnose network performance. > If you know a CPU and chipset for a system, you will know the network > performance > ceiling for that system, guaranteed. But it does not guarantee you can > get that ceiling > performance, especially over OC-12 (622 Mb/s) high-speed networks. That > requires > intensive tuning knowledge for current TCP stack, which is well > explained on the Internet > by searching for "TCP tuning". In this case, bandwidth should not factor in (16 MB/s is low, disks can reg= ularly double this easily). The 1 Gb/s NIC is not being fully used in thi= s case (< 40 MB/s) and the processor is mostly idle. From owner-freebsd-performance@FreeBSD.ORG Thu Mar 23 07:20:13 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DF28A16A400 for ; Thu, 23 Mar 2006 07:20:13 +0000 (UTC) (envelope-from g_jin@lbl.gov) Received: from smtp108.sbc.mail.mud.yahoo.com (smtp108.sbc.mail.mud.yahoo.com [68.142.198.207]) by mx1.FreeBSD.org (Postfix) with SMTP id 6787F43D46 for ; Thu, 23 Mar 2006 07:20:13 +0000 (GMT) (envelope-from g_jin@lbl.gov) Received: (qmail 60010 invoked from network); 23 Mar 2006 07:20:12 -0000 Received: from unknown (HELO ?192.168.2.10?) (jinmtb@sbcglobal.net@68.127.178.44 with plain) by smtp108.sbc.mail.mud.yahoo.com with SMTP; 23 Mar 2006 07:20:12 -0000 Message-ID: <44224C80.5080706@lbl.gov> Date: Wed, 22 Mar 2006 23:21:36 -0800 From: "Jin Guojun [VFFS]" User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050108 X-Accept-Language: zh, zh-CN, en MIME-Version: 1.0 To: Gary Thorpe References: <1143092417.c7f62afcgthorpe@myrealbox.com> In-Reply-To: <1143092417.c7f62afcgthorpe@myrealbox.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Thu, 23 Mar 2006 07:28:39 +0000 Cc: freebsd-performance@freebsd.org, oxy@field.hu Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Mar 2006 07:20:14 -0000 Gary Thorpe wrote: >[No subject in first one, sorry for repost] >... >1.6 Gb/s = system bus bandwidth. Cache won't affect this bandwidth. DDR400 has 400 MB/s: only attainable for long sequential accesses of either read or write but not a mix of both. DMA should be able to get near this limit (long and sequential, read or write only per transfer). A NIC with bus mastering DMA should be able to effectively use the memory bandwidth. > > This is not such simple thing and it is hard to explain in email. Two things to study: (1) DMA does not directly affect memory bandwidth. It directly sticks with I/O bandwidth, which is indirectly occupy the memory bandwidth. Slower I/O bus takes more memory bandwidth. DMA burst size also affects bandwidth efficiency. Smaller size is good for CPU, while large burst size is good for I/O. Does this make sense? (2) Try to analyze Intel 860 chipset -- 82806 AA PCI Hub (P64H) and 82860 memory controller Hub (MCH). This is well know problem, so you may find discussion on the Internet. This chipset had only 90 MB/s system bandwidth. When changing DT registers setting in P64H (see below), you may get 105MB/s in average (or 117 MB/s maximum) bandwidth by setting Dev31, Fun0, Reg50h[2] = 1 Dev31, Fun0, Reg80h[1:0]=2 It uses DDR400. Why is its system bandwidth not 1.6GB/s, not even 0.16GB, but 90MB/s? or 117 MB/s after modifying register setting? (This has nothing to do with cache or CPU. Any speed CPU has the similar system bandwidth when used with motherboard having such chipset.) (That is) The system bandwidth is not equal to memory speed or bus_speed times bus_width in the real world. (Done) -Jin From owner-freebsd-performance@FreeBSD.ORG Fri Mar 24 13:12:05 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6011E16A400 for ; Fri, 24 Mar 2006 13:12:05 +0000 (UTC) (envelope-from oxy@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id C4D9B43D45 for ; Fri, 24 Mar 2006 13:12:04 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (green.field.hu [217.20.130.28]) by green.field.hu (Postfix) with ESMTP id C21A6119CD4; Fri, 24 Mar 2006 14:11:24 +0100 (CET) Received: from green.field.hu ([217.20.130.28]) by localhost (green.field.hu [217.20.130.28]) (amavisd-new, port 10024) with ESMTP id 65051-02; Fri, 24 Mar 2006 14:11:24 +0100 (CET) Received: from oxy (dsl217-197-187-71.pool.tvnet.hu [217.197.187.71]) by green.field.hu (Postfix) with ESMTP id 41CD5119CC4; Fri, 24 Mar 2006 14:11:24 +0100 (CET) Message-ID: <003301c64f44$89fdcd40$0201a8c0@oxy> From: "OxY" To: "Jin Guojun [VFFS]" , "Arne Woerner" References: <20060322071023.70808.qmail@web30305.mail.mud.yahoo.com> <442187FE.3060300@lbl.gov> Date: Fri, 24 Mar 2006 14:12:01 +0100 MIME-Version: 1.0 X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2670 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670 X-Virus-Scanned: by Amavisd-new (Spamassassin+Razor2+Pyzor+DCC+Bayes db, Clamd Antivirus) at field.hu Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org, Gary Thorpe Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Mar 2006 13:12:05 -0000 hi guys! well, i changed my motherboard and CPU from the asus a7v8x+amd 2000+ xp to=20 the abit be7 + p4 2.4 (533fsb) and the packet loss fell down from 8% to = 2%, but still have loss... loss coming when i have load.. i guess it decreased because of the = bigger resources. still waiting for tipps, hints, everything :) ----- Original Message -----=20 From: Jin Guojun [VFFS]=20 To: Arne Woerner=20 Cc: Gary Thorpe ; freebsd-performance@freebsd.org ; oxy@field.hu=20 Sent: Wednesday, March 22, 2006 6:23 PM Subject: Re: packet drop with intel gigabit / marwell gigabit Arne Woerner wrote:=20 --- "Jin Guojun [VFFS]" wrote: In you example: Now your 1.6 GB/s reduced to 16MB/s or even worse just based on this factor. What did we show by this <
> test? I thought that would prove the memory bandwidth is about 8Gbit/sec (1GByte/sec; 2 * /2^30). It depends on how you use /dev/zero.=20 dd of=3D/dev/null if=3D/dev/zero bs=3D4k count=3D100k tests cache speed dd of=3D/dev/null if=3D/dev/zero bs=3D4m count=3D100 tests memory bandwidth if your cache is less than 2 MB Now you may give me the real memory bandwidth on your system :-) I would expect something around 500. Notice that your memory copy speed will be one half of it.=20 /dev/null device really does nothing beside throwing away data. That is, it can be counted as a cost for system call. -Jin From owner-freebsd-performance@FreeBSD.ORG Fri Mar 24 16:03:33 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6033F16A41F for ; Fri, 24 Mar 2006 16:03:33 +0000 (UTC) (envelope-from luke@foolishgames.com) Received: from mail.foolishgames.com (mail.foolishgames.com [206.222.28.162]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0C1AB43D55 for ; Fri, 24 Mar 2006 16:03:31 +0000 (GMT) (envelope-from luke@foolishgames.com) Received: from [192.168.0.151] (24-176-2-209.dhcp.klmz.mi.charter.com [24.176.2.209]) (authenticated bits=0) by mail.foolishgames.com (8.13.5/8.13.3) with ESMTP id k2OG3Hmp050176 (version=TLSv1/SSLv3 cipher=RC4-SHA bits=128 verify=NO); Fri, 24 Mar 2006 11:03:20 -0500 (EST) (envelope-from luke@foolishgames.com) X-Habeas-Swe-9: mark in spam to . X-Habeas-Swe-6: email in exchange for a license for this Habeas Content-Transfer-Encoding: 7bit X-Habeas-Swe-3: like Habeas SWE (tm) Date: Fri, 24 Mar 2006 11:06:11 -0500 X-Priority: 3 X-Habeas-Swe-8: Message (HCM) and not spam. Please report use of this X-Habeas-Swe-5: Sender Warranted Email (SWE) (tm). The sender of this From: Lucas Holt X-Habeas-Swe-2: brightly anticipated In-Reply-To: <003301c64f44$89fdcd40$0201a8c0@oxy> References: <20060322071023.70808.qmail@web30305.mail.mud.yahoo.com> <442187FE.3060300@lbl.gov> <003301c64f44$89fdcd40$0201a8c0@oxy> To: OxY X-Habeas-Swe-7: warrant mark warrants that this is a Habeas Compliant Mime-Version: 1.0 (Apple Message framework v746.3) X-Habeas-Swe-4: Copyright 2002 Habeas (tm) Content-Type: text/plain; charset=US-ASCII; delsp=yes; format=flowed X-Habeas-Swe-1: winter into spring Message-Id: <820F5FD6-C31F-4C28-9E66-64643C03086B@foolishgames.com> X-Mailer: Apple Mail (2.746.3) X-Virus-Scanned: ClamAV version 0.88, clamav-milter version 0.87 on mail.foolishgames.com X-Virus-Status: Clean Cc: "Jin Guojun \[VFFS\]" , FreeBSD Mailing Lists , Gary Thorpe , Arne Woerner Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Mar 2006 16:03:33 -0000 On Mar 24, 2006, at 8:12 AM, OxY wrote: > hi guys! > > well, i changed my motherboard and CPU from the > asus a7v8x+amd 2000+ xp to > the abit be7 + p4 2.4 (533fsb) and the packet loss fell down from > 8% to 2%, but > still have loss... > loss coming when i have load.. i guess it decreased because of the > bigger resources. > still waiting for tipps, hints, everything :) > > I don't think you'll ever get down to 0% in your situation. I noticed in the initial post that you have net.inet.tcp.inflight.enable=1 set. On my home network, turning that off helped a great deal with samba traffic to my freebsd file server/ router. It didn't seem to affect traffic to my webserver much, but its very low traffic. The problem with tuning on other people's settings is that each workload is different though. There might not be a miracle hack to get this working how you want. I'm sure the new box is a bit better as I attempted some of the steps outlined by Jin on my two machines. (amd 2300+ w/ msi nforce2 512mb ram and P4 2.4ghz 1gb ram 533mhz fsb) The P4 system was faster on all my tests by quite a large margin. I can't remember what version of FreeBSD you are using, but I do know they've done work on the em and fxp drivers during the 6.x series. I noticed a big improvement from 5.4 to 6.0 release and to 6.1 betas from 6.0 release. You might have better luck when 6.1 release comes out. I must admit, I didn't follow all of Jin's calculations. I do think he's right about some motherboard chipsets having limitations that limit real world traffic on the bus though. It follows what I learned in college during electrical engineering courses I was required to take. Hardware and Software are a lot alike. Just because something claims to support specific performance characteristics, does not mean that it does. Windows is a great example. I would guess that your problem is a combination of several factors including hardware, software, and network conditions. Almost every time I've had a problem like this at work or home its been a wiring problem or a switch limiting the throughput. Lucas Holt Luke@FoolishGames.com ________________________________________________________ FoolishGames.com (Jewel Fan Site) JustJournal.com (Free blogging) FoolishGames.net (Enemy Territory site) From owner-freebsd-performance@FreeBSD.ORG Fri Mar 24 20:08:26 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 955BA16A401 for ; Fri, 24 Mar 2006 20:08:26 +0000 (UTC) (envelope-from g_jin@lbl.gov) Received: from smtp108.sbc.mail.mud.yahoo.com (smtp108.sbc.mail.mud.yahoo.com [68.142.198.207]) by mx1.FreeBSD.org (Postfix) with SMTP id 28FDB43D49 for ; Fri, 24 Mar 2006 20:08:25 +0000 (GMT) (envelope-from g_jin@lbl.gov) Received: (qmail 46990 invoked from network); 24 Mar 2006 20:08:25 -0000 Received: from unknown (HELO ?192.168.2.10?) (jinmtb@sbcglobal.net@68.127.178.61 with plain) by smtp108.sbc.mail.mud.yahoo.com with SMTP; 24 Mar 2006 20:08:24 -0000 Message-ID: <4424520D.9000504@lbl.gov> Date: Fri, 24 Mar 2006 12:09:49 -0800 From: "Jin Guojun [VFFS]" User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050108 X-Accept-Language: zh, zh-CN, en MIME-Version: 1.0 To: Lucas Holt References: <20060322071023.70808.qmail@web30305.mail.mud.yahoo.com> <442187FE.3060300@lbl.gov> <003301c64f44$89fdcd40$0201a8c0@oxy> <820F5FD6-C31F-4C28-9E66-64643C03086B@foolishgames.com> In-Reply-To: <820F5FD6-C31F-4C28-9E66-64643C03086B@foolishgames.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Mailing Lists , OxY , Arne Woerner Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Mar 2006 20:08:26 -0000 Lucas Holt wrote: > > On Mar 24, 2006, at 8:12 AM, OxY wrote: > >> hi guys! >> >> well, i changed my motherboard and CPU from the >> asus a7v8x+amd 2000+ xp to >> the abit be7 + p4 2.4 (533fsb) and the packet loss fell down from >> 8% to 2%, but >> still have loss... >> loss coming when i have load.. i guess it decreased because of the >> bigger resources. >> still waiting for tipps, hints, everything :) >> >> > > I don't think you'll ever get down to 0% in your situation. I > noticed in the initial post that you have > net.inet.tcp.inflight.enable=1 set. On my home network, turning that > off helped a great deal with samba traffic to my freebsd file server/ > router. It didn't seem to affect traffic to my webserver much, but > its very low traffic. The problem with tuning on other people's > settings is that each workload is different though. Especially, when a user did not mention what network traffic condition and system load cause packet loss, it is difficult to get insight of the problem. So, the other thing in getting help in troubleshooting and performance tuning is to provide systematic and more detailed information. > There might not be a miracle hack to get this working how you want. > I'm sure the new box is a bit better as I attempted some of the steps > outlined by Jin on my two machines. (amd 2300+ w/ msi nforce2 512mb > ram and P4 2.4ghz 1gb ram 533mhz fsb) The P4 system was faster on > all my tests by quite a large margin. Just curious, were all your tests I/O related? 2300+ should over perform P4 2.4GHz in some computation tasks. > I must admit, I didn't follow all of Jin's calculations. I had quite sloppy email since I did not intend to involve detailed hardware discussion, but... For example, when I said that "cache design affects memory bandwidth [x1]" is very vague. It really means: "cache design affects memory copy speed (except DMA)." Generally, if we talk access data between CPU and main memory, then technically [x1] is right. If we talk to entire system design, theoretically, [x1] is wrong. I stand corrected for all such writing. -Jin From owner-freebsd-performance@FreeBSD.ORG Sat Mar 25 17:21:44 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B27BB16A400 for ; Sat, 25 Mar 2006 17:21:44 +0000 (UTC) (envelope-from oxy@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id 29E4543D45 for ; Sat, 25 Mar 2006 17:21:43 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (green.field.hu [217.20.130.28]) by green.field.hu (Postfix) with ESMTP id DDA0D119D2E; Sat, 25 Mar 2006 18:20:56 +0100 (CET) Received: from green.field.hu ([217.20.130.28]) by localhost (green.field.hu [217.20.130.28]) (amavisd-new, port 10024) with ESMTP id 54067-05; Sat, 25 Mar 2006 18:20:56 +0100 (CET) Received: from oxy (dsl217-197-187-71.pool.tvnet.hu [217.197.187.71]) by green.field.hu (Postfix) with ESMTP id 7787D119CA5; Sat, 25 Mar 2006 18:20:56 +0100 (CET) Message-ID: <008501c65030$96726710$0201a8c0@oxy> From: "OxY" To: "Jin Guojun [VFFS]" , "Lucas Holt" References: <20060322071023.70808.qmail@web30305.mail.mud.yahoo.com> <442187FE.3060300@lbl.gov> <003301c64f44$89fdcd40$0201a8c0@oxy> <820F5FD6-C31F-4C28-9E66-64643C03086B@foolishgames.com> <4424520D.9000504@lbl.gov> Date: Sat, 25 Mar 2006 18:21:43 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=response Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2670 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670 X-Virus-Scanned: by Amavisd-new (Spamassassin+Razor2+Pyzor+DCC+Bayes db, Clamd Antivirus) at field.hu Cc: FreeBSD Mailing Lists , Arne Woerner Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 17:21:44 -0000 what kind of details should i attach? to analyze the problem? ----- Original Message ----- From: "Jin Guojun [VFFS]" To: "Lucas Holt" Cc: "OxY" ; "FreeBSD Mailing Lists" ; "Arne Woerner" Sent: Friday, March 24, 2006 9:09 PM Subject: Re: packet drop with intel gigabit / marwell gigabit > Lucas Holt wrote: > >> >> On Mar 24, 2006, at 8:12 AM, OxY wrote: >> >>> hi guys! >>> >>> well, i changed my motherboard and CPU from the >>> asus a7v8x+amd 2000+ xp to >>> the abit be7 + p4 2.4 (533fsb) and the packet loss fell down from 8% >>> to 2%, but >>> still have loss... >>> loss coming when i have load.. i guess it decreased because of the >>> bigger resources. >>> still waiting for tipps, hints, everything :) >>> >>> >> >> I don't think you'll ever get down to 0% in your situation. I noticed >> in the initial post that you have net.inet.tcp.inflight.enable=1 set. >> On my home network, turning that off helped a great deal with samba >> traffic to my freebsd file server/ router. It didn't seem to affect >> traffic to my webserver much, but its very low traffic. The problem >> with tuning on other people's settings is that each workload is >> different though. > > Especially, when a user did not mention what network traffic condition and > system load > cause packet loss, it is difficult to get insight of the problem. So, the > other thing in getting > help in troubleshooting and performance tuning is to provide systematic > and more detailed > information. > >> There might not be a miracle hack to get this working how you want. I'm >> sure the new box is a bit better as I attempted some of the steps >> outlined by Jin on my two machines. (amd 2300+ w/ msi nforce2 512mb ram >> and P4 2.4ghz 1gb ram 533mhz fsb) The P4 system was faster on all my >> tests by quite a large margin. > > Just curious, were all your tests I/O related? 2300+ should over perform > P4 2.4GHz in some > computation tasks. > >> I must admit, I didn't follow all of Jin's calculations. > > I had quite sloppy email since I did not intend to involve detailed > hardware discussion, but... > For example, when I said that "cache design affects memory bandwidth [x1]" > is very vague. > It really means: "cache design affects memory copy speed (except DMA)." > Generally, if we talk access data between CPU and main memory, then > technically [x1] is right. > If we talk to entire system design, theoretically, [x1] is wrong. > I stand corrected for all such writing. > > -Jin > From owner-freebsd-performance@FreeBSD.ORG Sat Mar 25 19:23:44 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1CA0916A401 for ; Sat, 25 Mar 2006 19:23:44 +0000 (UTC) (envelope-from lists@wm-access.no) Received: from lakepoint.domeneshop.no (lakepoint.domeneshop.no [194.63.248.54]) by mx1.FreeBSD.org (Postfix) with ESMTP id 86DBA43D49 for ; Sat, 25 Mar 2006 19:23:42 +0000 (GMT) (envelope-from lists@wm-access.no) Received: from [192.168.5.8] (host-81-191-3-170.bluecom.no [81.191.3.170]) (authenticated bits=0) by lakepoint.domeneshop.no (8.13.6/8.13.6) with ESMTP id k2PJNUhw030988 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sat, 25 Mar 2006 20:23:31 +0100 Message-ID: <442598C6.8000802@wm-access.no> Date: Sat, 25 Mar 2006 20:23:50 +0100 From: =?ISO-8859-1?Q?Sten_Daniel_S=F8rsdal?= User-Agent: Thunderbird 1.5 (Windows/20051201) MIME-Version: 1.0 To: OxY References: <20060322071023.70808.qmail@web30305.mail.mud.yahoo.com> <442187FE.3060300@lbl.gov> <003301c64f44$89fdcd40$0201a8c0@oxy> <820F5FD6-C31F-4C28-9E66-64643C03086B@foolishgames.com> <4424520D.9000504@lbl.gov> <008501c65030$96726710$0201a8c0@oxy> In-Reply-To: <008501c65030$96726710$0201a8c0@oxy> X-Enigmail-Version: 0.94.0.0 OpenPGP: id=D6F56A9B Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig0B5CF7AB7030B604F0588AE6" Cc: FreeBSD Mailing Lists Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 19:23:44 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig0B5CF7AB7030B604F0588AE6 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: quoted-printable OxY wrote: > what kind of details should i attach? to analyze the problem? >=20 Host and peer packet statistics, cabling specs. What measures have you taken to exclude cable, peer or duplex mismatch=20 problems? --=20 Sten Daniel S=F8rsdal --------------enig0B5CF7AB7030B604F0588AE6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFEJZjNMvOF8Nb1apsRAsrVAJ9RjTdSYUfuLeOvX3PCzz11XIB65ACbB8OR jTgtjO8TTFH18BZX/NNXlrY= =61yK -----END PGP SIGNATURE----- --------------enig0B5CF7AB7030B604F0588AE6-- From owner-freebsd-performance@FreeBSD.ORG Sat Mar 25 21:03:20 2006 Return-Path: X-Original-To: freebsd-performance@freebsd.org Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DA2F116A401 for ; Sat, 25 Mar 2006 21:03:20 +0000 (UTC) (envelope-from oxy@field.hu) Received: from green.field.hu (green.field.hu [217.20.130.28]) by mx1.FreeBSD.org (Postfix) with ESMTP id 3F7C143D48 for ; Sat, 25 Mar 2006 21:03:18 +0000 (GMT) (envelope-from oxy@field.hu) Received: from localhost (green.field.hu [217.20.130.28]) by green.field.hu (Postfix) with ESMTP id D6ACD119D2E; Sat, 25 Mar 2006 22:02:29 +0100 (CET) Received: from green.field.hu ([217.20.130.28]) by localhost (green.field.hu [217.20.130.28]) (amavisd-new, port 10024) with ESMTP id 67610-09; Sat, 25 Mar 2006 22:02:29 +0100 (CET) Received: from oxy (dsl217-197-187-71.pool.tvnet.hu [217.197.187.71]) by green.field.hu (Postfix) with ESMTP id 8F4BB119CA5; Sat, 25 Mar 2006 22:02:29 +0100 (CET) Message-ID: <000a01c6504f$8a809980$0201a8c0@oxy> From: "OxY" To: =?ISO-8859-1?Q?Sten_Daniel_S=F8rsdal?= References: <20060322071023.70808.qmail@web30305.mail.mud.yahoo.com> <442187FE.3060300@lbl.gov> <003301c64f44$89fdcd40$0201a8c0@oxy> <820F5FD6-C31F-4C28-9E66-64643C03086B@foolishgames.com> <4424520D.9000504@lbl.gov> <008501c65030$96726710$0201a8c0@oxy> <442598C6.8000802@wm-access.no> Date: Sat, 25 Mar 2006 22:03:17 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="ISO-8859-1"; reply-type=original Content-Transfer-Encoding: 8bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.2670 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.2670 X-Virus-Scanned: by Amavisd-new (Spamassassin+Razor2+Pyzor+DCC+Bayes db, Clamd Antivirus) at field.hu Cc: freebsd-performance@freebsd.org Subject: Re: packet drop with intel gigabit / marwell gigabit X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Mar 2006 21:03:20 -0000 OxY wrote: > what kind of details should i attach? to analyze the problem? > Host and peer packet statistics, cabling specs. What measures have you taken to exclude cable, peer or duplex mismatch problems? cable, peer and duplex problems are excluded, because i measure 0% drop, when apache is not running and the system if full idle.. host is excluded too, because of previous.. please tell me which peer statistics should i send, thanks! -- Sten Daniel Srrsdal ----- Original Message ----- From: "Sten Daniel Sørsdal" To: "OxY" Cc: "FreeBSD Mailing Lists" Sent: Saturday, March 25, 2006 8:23 PM Subject: Re: packet drop with intel gigabit / marwell gigabit