From owner-freebsd-performance@FreeBSD.ORG Sun Mar 23 22:48:36 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4A5B91065674 for ; Sun, 23 Mar 2008 22:48:36 +0000 (UTC) (envelope-from engywook@gmail.com) Received: from an-out-0708.google.com (an-out-0708.google.com [209.85.132.246]) by mx1.freebsd.org (Postfix) with ESMTP id 017468FC23 for ; Sun, 23 Mar 2008 22:48:35 +0000 (UTC) (envelope-from engywook@gmail.com) Received: by an-out-0708.google.com with SMTP id c14so865375anc.13 for ; Sun, 23 Mar 2008 15:48:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:mime-version:content-type; bh=HxdgJplPUuKTTAS8jPkAwp7yALGUCqUd5lbd7RjbFPk=; b=GrgK9pXqtb0wNK4tyJwqZA+6cqKQKRxvKpAPNnguowIahpjDAUWbhsFbdEN/KdDSOhO3TAzWe+W3kIE67slqqVVUO7V8XgkQ3OA651YdPeyX0g3m8uQaso2/Fx4Qtcz4MdlnYtEgL4slSwv+Xc3T8208IhXjXmcB5dKy7y8mxw8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=message-id:date:from:to:subject:mime-version:content-type; b=MAR7kwtR82ptxaiT9uu7ddHutRwGoNHYscO/rm8nNcu4Q7wzk0XAOWR+btRVsmO/Rz0UlbikdKrjgoxaV9BM3lkFXNNkRqDfkni+YO0i9dyMo9K6MmEMOYokubQI08uYseC/9BQnMFfaHJeWe74H2fVgOD0b+YNpfWn7e8G71eo= Received: by 10.100.110.16 with SMTP id i16mr16177536anc.8.1206310908907; Sun, 23 Mar 2008 15:21:48 -0700 (PDT) Received: by 10.70.35.20 with HTTP; Sun, 23 Mar 2008 15:21:48 -0700 (PDT) Message-ID: <24adbbc00803231521h78844f26q77c48573f82408b9@mail.gmail.com> Date: Sun, 23 Mar 2008 23:21:48 +0100 From: "Daniel Andersson" To: freebsd-performance@freebsd.org MIME-Version: 1.0 X-Mailman-Approved-At: Sun, 23 Mar 2008 22:58:07 +0000 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Tuning: 100mbit faster, gbit slower. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 23 Mar 2008 22:48:36 -0000 Hey! I was trying to milk the most out of my 100/100. What I ended up with was something, to me, quite odd. When I hadn't done anything I could ftp things from my server box at 50mb/s and run rtorrent at about 9-10 mb/s at most. After my "tuning" I can only ftp at a very "choppy" 30-40mb/s, but rtorrent runs at about 11mb/s. This is what I did: kern.ipc.maxsockbuf=16777216 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.recvbuf_max=16777216 according to http://dsd.lbl.gov/TCP-tuning/FreeBSD.html every other setting there was default I believe. I also set these: net.inet.tcp.recvspace: 262144 net.inet.tcp.sendspace: 262144 dmesg: http://pastebin.org/24780 Am I just imagining that rtorrent runs faster? Can't ftp handle high buffers or did I mess something up? Is there something else I could do to make it faster? Setting up polling perhaps? Cheers, Daniel Andersson From owner-freebsd-performance@FreeBSD.ORG Mon Mar 24 08:44:39 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EF96B1065673 for ; Mon, 24 Mar 2008 08:44:39 +0000 (UTC) (envelope-from josh@tcbug.org) Received: from conn-smtp.mc.mpls.visi.com (conn.mc.mpls.visi.com [208.42.156.2]) by mx1.freebsd.org (Postfix) with ESMTP id D0D348FC23 for ; Mon, 24 Mar 2008 08:44:39 +0000 (UTC) (envelope-from josh@tcbug.org) Received: from mail.tcbug.org (mail.tcbug.org [208.42.70.163]) by conn-smtp.mc.mpls.visi.com (Postfix) with ESMTP id A862D7873; Mon, 24 Mar 2008 03:27:13 -0500 (CDT) Received: from build64.tcbug.org (unknown [208.42.70.167]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.tcbug.org (Postfix) with ESMTPSA id 52B646D9FBB; Mon, 24 Mar 2008 03:27:13 -0500 (CDT) From: Josh Paetzel To: freebsd-performance@freebsd.org Date: Mon, 24 Mar 2008 03:26:56 -0500 User-Agent: KMail/1.9.7 References: <24adbbc00803231521h78844f26q77c48573f82408b9@mail.gmail.com> In-Reply-To: <24adbbc00803231521h78844f26q77c48573f82408b9@mail.gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart3306768.7CppRB3lX1"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200803240327.01211.josh@tcbug.org> Cc: Daniel Andersson Subject: Re: Tuning: 100mbit faster, gbit slower. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 08:44:40 -0000 --nextPart3306768.7CppRB3lX1 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Sunday 23 March 2008 05:21:48 pm Daniel Andersson wrote: > Hey! > > I was trying to milk the most out of my 100/100. What I > ended up with was something, to me, quite odd. When I > hadn't done anything I could ftp things from my server > box at 50mb/s and run rtorrent at about 9-10 mb/s at most. > After my "tuning" I can only ftp at a very "choppy" > 30-40mb/s, but rtorrent runs at about 11mb/s. > This is what I did: > > kern.ipc.maxsockbuf=3D16777216 > net.inet.tcp.sendbuf_max=3D16777216 > net.inet.tcp.recvbuf_max=3D16777216 > > according to > http://dsd.lbl.gov/TCP-tuning/FreeBSD.html > every other setting there was default > I believe. > > I also set these: > > net.inet.tcp.recvspace: 262144 > net.inet.tcp.sendspace: 262144 > > dmesg: > http://pastebin.org/24780 > > Am I just imagining that rtorrent runs faster? > Can't ftp handle high buffers or did I mess > something up? Is there something else I > could do to make it faster? Setting up > polling perhaps? > > Cheers, > Daniel Andersson The stock settings are more than enough to saturate 100TX with even relativ= ely=20 ancient hardware. And by ancient I mean Pentium 2 class machines. The biggest tuning you can do is use intel (fxp) or 3com (xl) NICS and a=20 halfway decent switch. If your server box can't saturate 100TX ethernet with the defaults then=20 something is amiss. Perhaps provide a dmesg from the server and a client a= nd=20 a tcpdump from an FTP session between them? =2D-=20 Thanks, Josh Paetzel PGP: 8A48 EF36 5E9F 4EDA 5A8C 11B4 26F9 01F1 27AF AECB --nextPart3306768.7CppRB3lX1 Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.8 (FreeBSD) iEYEABECAAYFAkfnZdUACgkQJvkB8SevrsuegQCdHo32RtuNpTgpf6LCH4ZcSLFs 7SsAnAq+yfO9pzTX5Uepmy+tiFpvsStX =I8t4 -----END PGP SIGNATURE----- --nextPart3306768.7CppRB3lX1-- From owner-freebsd-performance@FreeBSD.ORG Mon Mar 24 10:54:39 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3F403106566B for ; Mon, 24 Mar 2008 10:54:39 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: from blah.sun-fish.com (blah.sun-fish.com [217.18.249.150]) by mx1.freebsd.org (Postfix) with ESMTP id 076788FC16 for ; Mon, 24 Mar 2008 10:54:38 +0000 (UTC) (envelope-from stefan.lambrev@moneybookers.com) Received: by blah.sun-fish.com (Postfix, from userid 1002) id 36E061B10EF7; Mon, 24 Mar 2008 11:54:37 +0100 (CET) X-Spam-Checker-Version: SpamAssassin 3.2.3 (2007-08-08) on blah.cmotd.com X-Spam-Level: X-Spam-Status: No, score=-10.6 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.2.3 Received: from [10.1.1.2] (unknown [192.168.25.10]) by blah.sun-fish.com (Postfix) with ESMTP id A22771B10E4E; Mon, 24 Mar 2008 11:54:32 +0100 (CET) Message-ID: <47E7885D.6080507@moneybookers.com> Date: Mon, 24 Mar 2008 12:54:21 +0200 From: Stefan Lambrev User-Agent: Thunderbird 2.0.0.12 (Windows/20080213) MIME-Version: 1.0 To: Daniel Andersson References: <24adbbc00803231521h78844f26q77c48573f82408b9@mail.gmail.com> In-Reply-To: <24adbbc00803231521h78844f26q77c48573f82408b9@mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: ClamAV 0.91.2/6364/Mon Mar 24 10:25:38 2008 on blah.cmotd.com X-Virus-Status: Clean Cc: freebsd-performance@freebsd.org Subject: Re: Tuning: 100mbit faster, gbit slower. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 10:54:39 -0000 Daniel Andersson wrote: > Hey! > > I was trying to milk the most out of my 100/100. What I > ended up with was something, to me, quite odd. When I > hadn't done anything I could ftp things from my server > box at 50mb/s and run rtorrent at about 9-10 mb/s at most. > After my "tuning" I can only ftp at a very "choppy" > 30-40mb/s, but rtorrent runs at about 11mb/s. > Are you sure the problem is in the network ? Sounds like the bottleneck is your HDD. You can run netperf to check this :) > This is what I did: > > kern.ipc.maxsockbuf=16777216 > net.inet.tcp.sendbuf_max=16777216 > net.inet.tcp.recvbuf_max=16777216 > > according to > http://dsd.lbl.gov/TCP-tuning/FreeBSD.html > every other setting there was default > I believe. > > I also set these: > > net.inet.tcp.recvspace: 262144 > net.inet.tcp.sendspace: 262144 > > dmesg: > http://pastebin.org/24780 > > Am I just imagining that rtorrent runs faster? > Can't ftp handle high buffers or did I mess > something up? Is there something else I > could do to make it faster? Setting up > polling perhaps? > > Cheers, > Daniel Andersson > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > From owner-freebsd-performance@FreeBSD.ORG Mon Mar 24 05:40:37 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBF9C106564A for ; Mon, 24 Mar 2008 05:40:37 +0000 (UTC) (envelope-from j_guojun@lbl.gov) Received: from smtp122.sbc.mail.sp1.yahoo.com (smtp122.sbc.mail.sp1.yahoo.com [69.147.64.95]) by mx1.freebsd.org (Postfix) with SMTP id C5A4B8FC14 for ; Mon, 24 Mar 2008 05:40:37 +0000 (UTC) (envelope-from j_guojun@lbl.gov) Received: (qmail 97617 invoked from network); 24 Mar 2008 05:13:57 -0000 Received: from unknown (HELO ?192.168.2.15?) (jinmtb@sbcglobal.net@75.37.3.8 with plain) by smtp122.sbc.mail.sp1.yahoo.com with SMTP; 24 Mar 2008 05:13:57 -0000 X-YMail-OSG: Ni.tp0AVM1nYIWeVU_KV6rH1Np7K91ZxGrXA5zE1F2ax8ViTcRREUXqWxuMwGWkZcZjhfXBcMZZHQgUR7qO8GKlWZhA8kIqB.gwGq1dxwhljOcC8AQ-- X-Yahoo-Newman-Property: ymail-3 Message-ID: <47E73891.8000803@lbl.gov> Date: Sun, 23 Mar 2008 22:13:53 -0700 From: "Jin Guojun [VFFS]" User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.13) Gecko/20061027 X-Accept-Language: en, zh, zh-CN MIME-Version: 1.0 To: Daniel Andersson References: <24adbbc00803231521h78844f26q77c48573f82408b9@mail.gmail.com> In-Reply-To: <24adbbc00803231521h78844f26q77c48573f82408b9@mail.gmail.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 24 Mar 2008 11:13:20 +0000 Cc: freebsd-performance@freebsd.org Subject: Re: Tuning: 100mbit faster, gbit slower. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 05:40:38 -0000 You cannot do blind tuning according those numbers. They are for 1-10Gbps pipe. The proper number is "pipe diamter" x "pipe length" = capacity In your case, the maximum number = 100Mbps x "delay from your machine to server" -Jin Daniel Andersson wrote: >Hey! > >I was trying to milk the most out of my 100/100. What I >ended up with was something, to me, quite odd. When I >hadn't done anything I could ftp things from my server >box at 50mb/s and run rtorrent at about 9-10 mb/s at most. >After my "tuning" I can only ftp at a very "choppy" >30-40mb/s, but rtorrent runs at about 11mb/s. >This is what I did: > >kern.ipc.maxsockbuf=16777216 >net.inet.tcp.sendbuf_max=16777216 >net.inet.tcp.recvbuf_max=16777216 > >according to >http://dsd.lbl.gov/TCP-tuning/FreeBSD.html >every other setting there was default >I believe. > >I also set these: > >net.inet.tcp.recvspace: 262144 >net.inet.tcp.sendspace: 262144 > >dmesg: >http://pastebin.org/24780 > >Am I just imagining that rtorrent runs faster? >Can't ftp handle high buffers or did I mess >something up? Is there something else I >could do to make it faster? Setting up >polling perhaps? > >Cheers, >Daniel Andersson > > From owner-freebsd-performance@FreeBSD.ORG Mon Mar 24 11:22:42 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 042741065671 for ; Mon, 24 Mar 2008 11:22:42 +0000 (UTC) (envelope-from engywook@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.191]) by mx1.freebsd.org (Postfix) with ESMTP id A89EE8FC18 for ; Mon, 24 Mar 2008 11:22:41 +0000 (UTC) (envelope-from engywook@gmail.com) Received: by rv-out-0910.google.com with SMTP id g13so1650681rvb.43 for ; Mon, 24 Mar 2008 04:22:12 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; bh=QGJRaxX356IEvpiJ1Lqscx/V6ebKk/ia5sIYM/xwoJs=; b=IeS8JO/NpbbEfPglrCEFG4ot3NBqkhrbfNY137imFL7nRfpHmQgreRLmmC+P2HHnGtg/1TSq7fnGNKPJy3+PaaMUxXSrj9IGrg5sS+A9P5wUJSs6iOQD2FT80uoq5m/u6dxjFUPNvL6xT95fStanE/SXxZEJ+aXpegWtwTGWteU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=Y2daHrXFakIL9Bz05fSzc6zQegxHPaVFdbcRYuvb9jTKVkmzF/ARfKw+wea9094A3JL7sT9ONBp+867pvsfX7oHL3nb0KAVw0TlpMXZfj0u23sRaKjYFolfYBqwQqaiOZnf7MKk/5CXyHE8CY52R4eAmwtKBL/ZyYCKXuZf6Vek= Received: by 10.141.211.13 with SMTP id n13mr2017965rvq.184.1206357732880; Mon, 24 Mar 2008 04:22:12 -0700 (PDT) Received: by 10.70.35.20 with HTTP; Mon, 24 Mar 2008 04:22:12 -0700 (PDT) Message-ID: <24adbbc00803240422m5b04b485s5df2f406aa89dc2b@mail.gmail.com> Date: Mon, 24 Mar 2008 12:22:12 +0100 From: "Daniel Andersson" To: "Josh Paetzel" In-Reply-To: <200803240327.01211.josh@tcbug.org> MIME-Version: 1.0 References: <24adbbc00803231521h78844f26q77c48573f82408b9@mail.gmail.com> <200803240327.01211.josh@tcbug.org> X-Mailman-Approved-At: Mon, 24 Mar 2008 11:36:36 +0000 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org Subject: Re: Tuning: 100mbit faster, gbit slower. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 11:22:42 -0000 Thanks for the reply! Maybe I should have been more clear. My setup looks like this: internet - em1(server box) em0 - windoze desktop. The internet(em1) 100mbit seems to do fine, even better than before, I get about 11mb/s with rtorrent(uploading). It's the internal gbit connection that's weird with ftp. It not as fast nor as smooth as it was before I did the "tuning". I doesn't have any trouble running ftping at 30mb/s after the tuning so it is definately capabel of delivering 100mbit? $ifconfig -a em0: flags=8843 metric 0 mtu 1500 options=9b ether 00:1b:21:0a:1d:87 inet 192.168.0.10 netmask 0xffffff00 broadcast 192.168.0.255 media: Ethernet autoselect (1000baseTX ) status: active em1: flags=8843 metric 0 mtu 1500 options=9b ether 00:1b:21:0c:d1:b3 inet external.ip.goes.here netmask 0xfffffc00 broadcast xx.xxx.147.255 media: Ethernet autoselect (100baseTX ) status: active plip0: flags=108810 metric 0 mtu 1500 lo0: flags=8049 metric 0 mtu 16384 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4 inet6 ::1 prefixlen 128 inet 127.0.0.1 netmask 0xff000000 pflog0: flags=141 metric 0 mtu 33204 tcpdump as requested: engy# tcpdump -vv -i em0 portrange 40000-42000 or portrange 20-21 | cat > /usr/home/engy/tcpdump http://pastebin.org/25081 If you had something else in mind, let me know. When I transferred large files I sometimes got about 600 packages dropped by kernel. That can't be good. Also it seemed that I got a lot of ack packages, don't know if that's normal. Cheers, Daniel >The stock settings are more than enough to saturate 100TX with even > relatively > >ancient hardware. And by ancient I mean Pentium 2 class machines. > > > >The biggest tuning you can do is use intel (fxp) or 3com (xl) NICS and a > >halfway decent switch. > > > >If your server box can't saturate 100TX ethernet with the defaults then > >something is amiss. Perhaps provide a dmesg from the server and a client > and > >a tcpdump from an FTP session between them? > > > >-- > >Thanks, > > > >Josh Paetzel > > >PGP: 8A48 EF36 5E9F 4EDA 5A8C 11B4 26F9 01F1 27AF AECB > > From owner-freebsd-performance@FreeBSD.ORG Mon Mar 24 11:43:52 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0055C1065672 for ; Mon, 24 Mar 2008 11:43:51 +0000 (UTC) (envelope-from engywook@gmail.com) Received: from wx-out-0506.google.com (wx-out-0506.google.com [66.249.82.236]) by mx1.freebsd.org (Postfix) with ESMTP id BD4F38FC28 for ; Mon, 24 Mar 2008 11:43:51 +0000 (UTC) (envelope-from engywook@gmail.com) Received: by wx-out-0506.google.com with SMTP id i29so3272287wxd.7 for ; Mon, 24 Mar 2008 04:43:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; bh=CSBDZAfRLTUHSMP178RUmy8oaHSSZrM/DvIhEW5aajc=; b=KZDRB3dwh1VSAcmc9cVPBGOjxrum7oj9OVTGE+8dM5foLrxKBs0dQ/hv8+GCVo3WZLqZLeOOHDfHteuZi6kouKTklOTyooJVcE2qBQKBbS/1OOxJNXI+HeUwpgpd5cy7LLGrFpnqF4AC3TpbqItaRFAHNWzrqnI+iqaJdEQL3MY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version:content-type:references; b=oxo4T89liE/CkLsd0yF9GYVKAEn42qhaN21kC0lZ1EibdHVOj2gQxEqbtylNqmbEKcdRUVWex57Jn2SoUA4IKCpdOxuTAaQCuTDBULLlJTYBElZg5+4nbcMYUjAxoedOFXvD/WTEQQKyZpnJS3Dc/aSUHPtpE2TN/gGeZIAdaZA= Received: by 10.70.50.3 with SMTP id x3mr8585104wxx.72.1206359030738; Mon, 24 Mar 2008 04:43:50 -0700 (PDT) Received: by 10.70.35.20 with HTTP; Mon, 24 Mar 2008 04:43:50 -0700 (PDT) Message-ID: <24adbbc00803240443p3fffc741tb80dfda257eb29f@mail.gmail.com> Date: Mon, 24 Mar 2008 12:43:50 +0100 From: "Daniel Andersson" To: "Stefan Lambrev" In-Reply-To: <47E7885D.6080507@moneybookers.com> MIME-Version: 1.0 References: <24adbbc00803231521h78844f26q77c48573f82408b9@mail.gmail.com> <47E7885D.6080507@moneybookers.com> X-Mailman-Approved-At: Mon, 24 Mar 2008 11:51:32 +0000 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-performance@freebsd.org Subject: Re: Tuning: 100mbit faster, gbit slower. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 11:43:52 -0000 Bloody hell! You were right. I transferred a big file to the system disk and ftped it from there to my desktop and it topped at 59mb/s, Sorry guys! It's quite odd though. It's my newest disk: ad10: 476940MB at ata5-master SATA150 It's connected to: atapci2: port 0xd200-0xd27f,0xd300-0xd3ff mem 0xee0c0000-0xee0c0fff,0xee080000-0xee09ffff irq 18 at device 11.0 on pci2 atapci2: [ITHREAD]atapci2: [ITHREAD] I recall having trouble installing on that controller, it would just reboot. I just checked the 7.0 Hardware notes and can't find it there. That sucks. Guess I'll have to buy a PCI controller then. Any good ones you can recommend? Cheers, Daniel >Are you sure the problem is in the network ? > >Sounds like the bottleneck is your HDD. > >You can run netperf to check this :) > > From owner-freebsd-performance@FreeBSD.ORG Mon Mar 24 12:08:19 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3AD8E1065672 for ; Mon, 24 Mar 2008 12:08:19 +0000 (UTC) (envelope-from engywook@gmail.com) Received: from ti-out-0910.google.com (ti-out-0910.google.com [209.85.142.189]) by mx1.freebsd.org (Postfix) with ESMTP id CDDE68FC3E for ; Mon, 24 Mar 2008 12:08:18 +0000 (UTC) (envelope-from engywook@gmail.com) Received: by ti-out-0910.google.com with SMTP id j2so606009tid.3 for ; Mon, 24 Mar 2008 05:08:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; bh=rvr6NSziuT/qiIiTNRwufy+sX50A4NStOzAck54Q9Qc=; b=gjyXrkMlYEEUmi/ahi2sOCSPxu8drLB4T5czbCSxDgL3LEVOYsQnPp7mgEX5NqFedPx0Maw2Dey+Q5qEavIFdZXE9b5DpH+ZJroqMTRgYMAjRB4PUnz+EpBbmlxkxoilUb+PcdMoJu9SYSJ7WOWMz3dtrQgRJ3BIjPOBGCuI9EY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=ZcSrxPt1630cxdIIO8Vy4MCwdpXbZRbdJhy2zMy3oPxsvS4qPvY3Ez6i8RE4IfZo/xQVkIXSpq+t4Qj02IcLt/SvFw6RLsH12Z4lv2mdgRfyOqjdLRxTJRoCrElaME824e1jh3fHVaMLWybRRcWCdSbaTKQDjVA8+oR0i2xdBQs= Received: by 10.110.31.5 with SMTP id e5mr1105003tie.37.1206360497523; Mon, 24 Mar 2008 05:08:17 -0700 (PDT) Received: by 10.70.35.20 with HTTP; Mon, 24 Mar 2008 05:08:17 -0700 (PDT) Message-ID: <24adbbc00803240508q224003e1j77f586d4ef6a8bf@mail.gmail.com> Date: Mon, 24 Mar 2008 13:08:17 +0100 From: "Daniel Andersson" To: freebsd-performance@freebsd.org In-Reply-To: <24adbbc00803231521h78844f26q77c48573f82408b9@mail.gmail.com> MIME-Version: 1.0 References: <24adbbc00803231521h78844f26q77c48573f82408b9@mail.gmail.com> X-Mailman-Approved-At: Mon, 24 Mar 2008 12:13:51 +0000 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: Tuning: 100mbit faster, gbit slower. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 12:08:19 -0000 I might have spoken too soon. I ran the transfer a few times and got this once: 215653 packets captured 379235 packets received by filter 97650 packets dropped by kernel The transfer wouldn't even start. Buggy windoze client perhaps? Other times it would jump anywhere from 26-61mb/s. That could be disk bottleneck I guess. From owner-freebsd-performance@FreeBSD.ORG Mon Mar 24 14:59:43 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2AA181065670 for ; Mon, 24 Mar 2008 14:59:43 +0000 (UTC) (envelope-from archwndas@yahoo.com) Received: from web56509.mail.re3.yahoo.com (web56509.mail.re3.yahoo.com [66.196.97.38]) by mx1.freebsd.org (Postfix) with SMTP id C643E8FC2D for ; Mon, 24 Mar 2008 14:59:42 +0000 (UTC) (envelope-from archwndas@yahoo.com) Received: (qmail 97265 invoked by uid 60001); 24 Mar 2008 14:33:02 -0000 DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=s1024; d=yahoo.com; h=X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:MIME-Version:Content-Type:Message-ID; b=T9BerKNbVKjo6KXrpZjYv4n3PiBwaXKmNmNWMG9sIIS3RW+LYPJKTKSLzBh3EZ+7Iz9WVoiWJJ5IhJ/mfRZWEVsOaxomWUB0Z7aeKj5YJO4g7B0qerAcwWeGW8Ogk/l60RezqO6hvAhS6op4W7GZIwfkv48ID4O3f+Wtgq2OVT8=; X-YMail-OSG: FnWZuBIVM1le3NG11lWKjQ3DdiaYHe0ypf2ykrtBBZQkEEExjMr5WoAWr9Igg75eH8sElXHE7r27p1ebuahT1UJ5VQm64CZiVuhMvaL9PscET.vVMi8P.3Q06__XMggBQSYAhuBrRI_Oe.U- Received: from [79.130.152.193] by web56509.mail.re3.yahoo.com via HTTP; Mon, 24 Mar 2008 07:33:01 PDT X-Mailer: YahooMailRC/902.40 YahooMailWebService/0.7.185 Date: Mon, 24 Mar 2008 07:33:01 -0700 (PDT) From: Simeon Nifos To: freebsd-performance@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Message-ID: <147812.96521.qm@web56509.mail.re3.yahoo.com> Subject: run-time performance of regression of sparse matrix vector multiplication X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 14:59:43 -0000 I have found a problem with FreeBSD AMD64 (maybe i386 too). Performance decrease related to Linux. I am attaching the results and the piece of code I used. You have to install g++42 on FreeBSD first. here are the results of the benchmark: =============== ==== LINUX ==== =============== Intel Core 2 ============ number of threads: 1/ 2 Sun CC create : 808/443 multiply: 5063/4488 g++-4.2.2 create : 881/479 multiply: 5245/4691 intel icpc create : 724/404 multiply: 4903/4594 we see that although the allocation of can be safely parallelized the multiplication has a really hard time to do so. Are there any problems with this approach I cannot see? sysctl dev.cpu.0.freq [archwn@home /usr/home/archwn/sparsematrixvector]$ sysctl dev.cpu.0.freq dev.cpu.0.freq: 1654 ===================== ==== FreeBSD 7.0 ==== ===================== Intel Core 2 ============ number of threads: 1/ 2 g++-4.2.2 create : 1750/1288 multiply: 7098/5271 Same optimization flags in both cases with g++-4.2.2. I have also written a pthreads version of the above code which doesn't need OpenMP capable compiler at all. This allows us to try gcc-3.4.6 compiler which is unlikely to have problems of its own. Is there anything you would like me to try out? Is anybody interested in having the code in order to perform his own tests? Thanks in advance, Archwn. ____________________________________________________________________________________ Never miss a thing. Make Yahoo your home page. http://www.yahoo.com/r/hs From owner-freebsd-performance@FreeBSD.ORG Mon Mar 24 15:41:06 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 21973106566C for ; Mon, 24 Mar 2008 15:41:06 +0000 (UTC) (envelope-from josh@tcbug.org) Received: from conn-smtp.mc.mpls.visi.com (conn.mc.mpls.visi.com [208.42.156.2]) by mx1.freebsd.org (Postfix) with ESMTP id D97888FC1E for ; Mon, 24 Mar 2008 15:41:05 +0000 (UTC) (envelope-from josh@tcbug.org) Received: from mail.tcbug.org (mail.tcbug.org [208.42.70.163]) by conn-smtp.mc.mpls.visi.com (Postfix) with ESMTP id 07BBB78F7; Mon, 24 Mar 2008 10:41:05 -0500 (CDT) Received: from build64.tcbug.org (unknown [208.42.70.167]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.tcbug.org (Postfix) with ESMTPSA id CD74A6D9E2B; Mon, 24 Mar 2008 10:41:04 -0500 (CDT) From: Josh Paetzel To: "Daniel Andersson" Date: Mon, 24 Mar 2008 10:40:47 -0500 User-Agent: KMail/1.9.7 References: <24adbbc00803231521h78844f26q77c48573f82408b9@mail.gmail.com> <200803240327.01211.josh@tcbug.org> <24adbbc00803240422m5b04b485s5df2f406aa89dc2b@mail.gmail.com> In-Reply-To: <24adbbc00803240422m5b04b485s5df2f406aa89dc2b@mail.gmail.com> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1574323.vpPRRNzLlh"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200803241040.53346.josh@tcbug.org> Cc: freebsd-performance@freebsd.org Subject: Re: Tuning: 100mbit faster, gbit slower. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 15:41:06 -0000 --nextPart1574323.vpPRRNzLlh Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Monday 24 March 2008 06:22:12 am Daniel Andersson wrote: > Thanks for the reply! Maybe I should have been more clear. > My setup looks like this: > internet - em1(server box) em0 - windoze desktop. > > The internet(em1) 100mbit seems to do fine, even better than > before, I get about 11mb/s with rtorrent(uploading). It's the > internal gbit connection that's weird with ftp. It not as fast nor > as smooth as it was before I did the "tuning". I doesn't have > any trouble running ftping at 30mb/s after the tuning so it is > definately capabel of delivering 100mbit? > I think we are having a terminology problem here. Are you meaning 30mb/s a= s=20 in 30 megabits per second (30% of 100TX speed) or do you mean 30 MB/s as in= =20 30 Megabytes/sec? I think in rereading you post you are meaning Megabytes= =20 but using the terminology for megabits. So here's what I've done to nearly saturate gig-e. Keep in mind that I hav= e=20 15k SAS drives and intel gig-e adapters that aren't sitting in 33mhz 32bit= =20 PCI slots, single IDE/SATA drives are going to be a bottleneck as are 33mhz= =20 32bit PCI NICs. This is on RELENG_6_3 net.inet.tcp.sendspace=3D262144 net.inet.tcp.recvspace=3D262144 kern.ipc.maxsockbuf=3D1048576 ifconfig em0 mtu 9014 (You'll need a switch that supports jumbo frames to d= o=20 this) iperf shows wire traffic around 969 mbps and FTP runs at 110 Megs/sec scp/sftp appears to be cpu bound at 45 Megs/sec, and NFS with TCP mounts an= d=20 send/receive packets set to 16384 manages about 90 Megs/sec. =2D-=20 Thanks, Josh Paetzel PGP: 8A48 EF36 5E9F 4EDA 5A8C 11B4 26F9 01F1 27AF AECB --nextPart1574323.vpPRRNzLlh Content-Type: application/pgp-signature; name=signature.asc Content-Description: This is a digitally signed message part. -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.8 (FreeBSD) iEYEABECAAYFAkfny4UACgkQJvkB8SevrsuUmgCgjarINiVQ0Qi8FB0LGsTSPPNO SiMAmQFJP+ygvk69VZsY3KiDGI7Z885J =UBlt -----END PGP SIGNATURE----- --nextPart1574323.vpPRRNzLlh-- From owner-freebsd-performance@FreeBSD.ORG Mon Mar 24 15:59:56 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 246A21065673 for ; Mon, 24 Mar 2008 15:59:56 +0000 (UTC) (envelope-from engywook@gmail.com) Received: from ti-out-0910.google.com (ti-out-0910.google.com [209.85.142.190]) by mx1.freebsd.org (Postfix) with ESMTP id 8B1FD8FC1E for ; Mon, 24 Mar 2008 15:59:55 +0000 (UTC) (envelope-from engywook@gmail.com) Received: by ti-out-0910.google.com with SMTP id j2so675876tid.3 for ; Mon, 24 Mar 2008 08:59:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; bh=x87LVRs3q+FavuP5m72YT7x8GMS1HibGIx6gbq9ecIM=; b=Pxeg5xorweHj7zq/i8xaaiHQjy+AY8KNzbnAsr3uU/co1rslNYeIQBTMW3PZNJpvjvy6pOfDyND6W3yTWP1d2a5sJJKb1FIxW6cdz64shogYVWbb1CiIbtCc8wdiqsKk1wKK44oDOnF784N1h6kpWMQ453MHqnjD4vAjvF0kPWc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=message-id:date:from:to:subject:in-reply-to:mime-version:content-type:references; b=jN6R8xzVT77pcdcZW6t/VBL2qcfZlidCFvpbv1L6v0HaRPODDZw1j73pO12OWsxiCeSr95IxiWQNW7GDcor+hMdDD6e+ziI+3SsPbKsidaFuSGY2yf+sz5ml3JcdRil/i8xGN0FZ8PHVyFUR44Wo3B4LXgTqMFtDdiCziEHHKJE= Received: by 10.110.68.10 with SMTP id q10mr1575663tia.22.1206374393818; Mon, 24 Mar 2008 08:59:53 -0700 (PDT) Received: by 10.70.35.20 with HTTP; Mon, 24 Mar 2008 08:59:53 -0700 (PDT) Message-ID: <24adbbc00803240859w1e1cc372xae8c97a29d474b92@mail.gmail.com> Date: Mon, 24 Mar 2008 16:59:53 +0100 From: "Daniel Andersson" To: "Josh Paetzel" In-Reply-To: <200803241040.53346.josh@tcbug.org> MIME-Version: 1.0 References: <24adbbc00803231521h78844f26q77c48573f82408b9@mail.gmail.com> <200803240327.01211.josh@tcbug.org> <24adbbc00803240422m5b04b485s5df2f406aa89dc2b@mail.gmail.com> <200803241040.53346.josh@tcbug.org> X-Mailman-Approved-At: Mon, 24 Mar 2008 16:15:09 +0000 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: Subject: Re: Tuning: 100mbit faster, gbit slower. X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 24 Mar 2008 15:59:56 -0000 > >I think we are having a terminology problem here. Are you meaning 30mb/s > as > >in 30 megabits per second (30% of 100TX speed) or do you mean 30 MB/s as > in > >30 Megabytes/sec? I think in rereading you post you are meaning > Megabytes > >but using the terminology for megabits. Ah, yes, sorry! I ment 30MB/s. >So here's what I've done to nearly saturate gig-e. Keep in mind that I > have > >15k SAS drives and intel gig-e adapters that aren't sitting in 33mhz > 32bit > >PCI slots, single IDE/SATA drives are going to be a bottleneck as are > 33mhz > >32bit PCI NICs. > > > >This is on RELENG_6_3 > > > > > >net.inet.tcp.sendspace=262144 > >net.inet.tcp.recvspace=262144 > > > >kern.ipc.maxsockbuf=1048576 > >ifconfig em0 mtu 9014 (You'll need a switch that supports jumbo frames to > do > >this) > > > >iperf shows wire traffic around 969 mbps and FTP runs at 110 Megs/sec > >scp/sftp appears to be cpu bound at 45 Megs/sec, and NFS with TCP mounts > and > >send/receive packets set to 16384 manages about 90 Megs/sec. > >-- > >Thanks, > >Josh Paetzel > >PGP: 8A48 EF36 5E9F 4EDA 5A8C 11B4 26F9 01F1 27AF AECB > > I don't really need the extra speed on the internal network. I just want the steady 50MB/s back, but also the increased performance of rtorrent I got from my "tuning". I'm also planning on getting more disks and setting up either zfs or softraid to lessen the hdd bottleneck. One thing at a time though. Maybe setting kern.ipc.maxsockbuf=1M+ and leaving everything else default will solve it. Since rtorrent has it's own setting for buffer sizes. Thanks for the replies and sorry for the mixup! Daniel From owner-freebsd-performance@FreeBSD.ORG Tue Mar 25 02:27:23 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A2E3F106566C for ; Tue, 25 Mar 2008 02:27:23 +0000 (UTC) (envelope-from bmeekhof@umich.edu) Received: from skycaptain.mr.itd.umich.edu (smtp.mail.umich.edu [141.211.93.160]) by mx1.freebsd.org (Postfix) with ESMTP id 45CD18FC12 for ; Tue, 25 Mar 2008 02:27:23 +0000 (UTC) (envelope-from bmeekhof@umich.edu) Received: FROM atom.heart.mother (c-68-40-199-244.hsd1.mi.comcast.net [68.40.199.244]) BY skycaptain.mr.itd.umich.edu ID 47E85C00.F18F7.12794 ; 24 Mar 2008 21:57:21 -0400 Message-ID: <47E85C00.4010601@umich.edu> Date: Mon, 24 Mar 2008 21:57:20 -0400 From: "Benjeman J. Meekhof" User-Agent: Thunderbird 2.0.0.9 (X11/20071031) MIME-Version: 1.0 To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: performance tuning on perc6 (LSI) controller X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2008 02:27:23 -0000 Hello, I think this might be useful information, and am also hoping for a little input. We've been doing some FreeBSD benchmarking on Dell PE2950 systems with Perc6 controllers (dual-quad Xeon, 16GB, Perc6=LSI card, mfi driver, 7.0-RELEASE). There are two controllers in each system, and each has two MD1000 disk shelves attached via the 2 4x SAS interfaces. (so 30PD available to each controller, 60 PD on the system). My baseline was this - on linux 2.6.20 we're doing 800MB/s write and greater read with this configuration: 2 raid6 volumes volumes striped into a raid0 volume using linux software raid, XFS filesystem. Each raid6 is a volume on one controller using 30 PD. We've spent time tuning this, more than I have with FreeBSD so far. Initially I was getting strangely poor read results. Here is one example (before launching into quicker dd tests, i already had similarly bad results from some more complete iozone tests): time dd if=/dev/zero of=/test/deletafile bs=1M count=10240 10737418240 bytes transferred in 26.473629 secs (405589209 bytes/sec) time dd if=/test/deletafile of=/dev/null bs=1M count=10240 10737418240 bytes transferred in 157.700367 secs (68087465 bytes/sec) To make a very long story short, much better results achieved in the end by simply by increasing the filesystem blocksize to the maximum (same dd commands). I'm running a more thorough test on this setup using iozone: #gstripe label -v -s 128k test /dev/mfid0 /dev/mfid2 #newfs -U -b 65536 /dev/stripe/test #write: 19.240875 secs (558052492 bytes/sec) #read: 20.000606 secs (536854644 bytes/sec) Also did this in /boot/loader.conf - it effected nothing very much in any test but the settings seemed reasonable so I kept them: kern.geom.stripe.fast=1 vfs.hirunningspace=5242880 vfs.read_max=32 Any other suggestions to get best throughput? There is also HW RAID stripe size to adjust larger or smaller. ZFS is also on the list for testing. Should I perhaps be running -CURRENT or -STABLE to be get best results with ZFS? -Ben -- Benjeman Meekhof - UM ATLAS/AGLT2 Computing bmeekhof@umich.edu From owner-freebsd-performance@FreeBSD.ORG Tue Mar 25 02:27:24 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 307D71065671 for ; Tue, 25 Mar 2008 02:27:24 +0000 (UTC) (envelope-from bmeekhof@umich.edu) Received: from skycaptain.mr.itd.umich.edu (smtp.mail.umich.edu [141.211.93.160]) by mx1.freebsd.org (Postfix) with ESMTP id C77868FC21 for ; Tue, 25 Mar 2008 02:27:23 +0000 (UTC) (envelope-from bmeekhof@umich.edu) Received: FROM atom.heart.mother (c-68-40-199-244.hsd1.mi.comcast.net [68.40.199.244]) BY skycaptain.mr.itd.umich.edu ID 47E85D55.CE4DF.13836 ; 24 Mar 2008 22:03:01 -0400 Message-ID: <47E85D54.4070905@umich.edu> Date: Mon, 24 Mar 2008 22:03:00 -0400 From: "Benjeman J. Meekhof" User-Agent: Thunderbird 2.0.0.9 (X11/20071031) MIME-Version: 1.0 To: freebsd-performance@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: performance tuning on perc6 (LSI) controller X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2008 02:27:24 -0000 Should clarify that the first test mentioned below used the same gstripe setup as the latter one but did not specify any newfs blocksize: #gstripe label -v -s 128k test /dev/mfid0 /dev/mfid2 #newfs -U /dev/stripe/test sorry, Ben ---------------------------------------- Hello, I think this might be useful information, and am also hoping for a little input. We've been doing some FreeBSD benchmarking on Dell PE2950 systems with Perc6 controllers (dual-quad Xeon, 16GB, Perc6=LSI card, mfi driver, 7.0-RELEASE). There are two controllers in each system, and each has two MD1000 disk shelves attached via the 2 4x SAS interfaces. (so 30PD available to each controller, 60 PD on the system). My baseline was this - on linux 2.6.20 we're doing 800MB/s write and greater read with this configuration: 2 raid6 volumes volumes striped into a raid0 volume using linux software raid, XFS filesystem. Each raid6 is a volume on one controller using 30 PD. We've spent time tuning this, more than I have with FreeBSD so far. Initially I was getting strangely poor read results. Here is one example (before launching into quicker dd tests, i already had similarly bad results from some more complete iozone tests): time dd if=/dev/zero of=/test/deletafile bs=1M count=10240 10737418240 bytes transferred in 26.473629 secs (405589209 bytes/sec) time dd if=/test/deletafile of=/dev/null bs=1M count=10240 10737418240 bytes transferred in 157.700367 secs (68087465 bytes/sec) To make a very long story short, much better results achieved in the end by simply by increasing the filesystem blocksize to the maximum (same dd commands). I'm running a more thorough test on this setup using iozone: #gstripe label -v -s 128k test /dev/mfid0 /dev/mfid2 #newfs -U -b 65536 /dev/stripe/test #write: 19.240875 secs (558052492 bytes/sec) #read: 20.000606 secs (536854644 bytes/sec) Also did this in /boot/loader.conf - it effected nothing very much in any test but the settings seemed reasonable so I kept them: kern.geom.stripe.fast=1 vfs.hirunningspace=5242880 vfs.read_max=32 Any other suggestions to get best throughput? There is also HW RAID stripe size to adjust larger or smaller. ZFS is also on the list for testing. Should I perhaps be running -CURRENT or -STABLE to be get best results with ZFS? -Ben -- Benjeman Meekhof - UM ATLAS/AGLT2 Computing bmeekhof@umich.edu From owner-freebsd-performance@FreeBSD.ORG Tue Mar 25 09:52:46 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E389F1065673 for ; Tue, 25 Mar 2008 09:52:46 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from ciao.gmane.org (main.gmane.org [80.91.229.2]) by mx1.freebsd.org (Postfix) with ESMTP id 5F49E8FC31 for ; Tue, 25 Mar 2008 09:52:46 +0000 (UTC) (envelope-from gofp-freebsd-performance@m.gmane.org) Received: from list by ciao.gmane.org with local (Exim 4.43) id 1Je5pk-0007yR-EX for freebsd-performance@freebsd.org; Tue, 25 Mar 2008 09:52:44 +0000 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 25 Mar 2008 09:52:44 +0000 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 25 Mar 2008 09:52:44 +0000 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-performance@freebsd.org From: Ivan Voras Date: Tue, 25 Mar 2008 10:58:07 +0100 Lines: 50 Message-ID: References: <47E85C00.4010601@umich.edu> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enig15D4D51D86A039EFE88DFDD8" X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.6 (X11/20071022) In-Reply-To: <47E85C00.4010601@umich.edu> X-Enigmail-Version: 0.95.0 Sender: news Subject: Re: performance tuning on perc6 (LSI) controller X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 25 Mar 2008 09:52:47 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig15D4D51D86A039EFE88DFDD8 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Benjeman J. Meekhof wrote: > My baseline was this - on linux 2.6.20 we're doing 800MB/s write and > greater read with this configuration: 2 raid6 volumes volumes striped > into a raid0 volume using linux software raid, XFS filesystem. Each > raid6 is a volume on one controller using 30 PD. We've spent time > tuning this, more than I have with FreeBSD so far. > time dd if=3D/dev/zero of=3D/test/deletafile bs=3D1M count=3D10240 > 10737418240 bytes transferred in 26.473629 secs (405589209 bytes/sec) > time dd if=3D/test/deletafile of=3D/dev/null bs=3D1M count=3D10240 > 10737418240 bytes transferred in 157.700367 secs (68087465 bytes/sec) I had similar ratio of results when comparing FreeBSD+UFS to most high-performance Linux file systems (XFS is really great!), so I'd guess it's about as fast as you can get with this combination. > Any other suggestions to get best throughput? There is also HW RAID > stripe size to adjust larger or smaller. ZFS is also on the list for > testing. Should I perhaps be running -CURRENT or -STABLE to be get bes= t > results with ZFS? ZFS will be up to 50% faster on tests such as yours, so you should definitely try it. Unfortunately it's not stable and you probably don't want to use it in production. AFAIK there are no significant differences between ZFS in -current and -stable. --------------enig15D4D51D86A039EFE88DFDD8 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) iD8DBQFH6My1ldnAQVacBcgRAqn2AJwISFhr2Pp6AnjW+UEuqJJ7e6TI9QCg8Yzn 4BhFANWhF+5RMgVTQ17zR94= =SVSi -----END PGP SIGNATURE----- --------------enig15D4D51D86A039EFE88DFDD8-- From owner-freebsd-performance@FreeBSD.ORG Wed Mar 26 06:00:05 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B13501065670 for ; Wed, 26 Mar 2008 06:00:05 +0000 (UTC) (envelope-from bmeekhof@umich.edu) Received: from hellskitchen.mr.itd.umich.edu (smtp.mail.umich.edu [141.211.14.82]) by mx1.freebsd.org (Postfix) with ESMTP id 6296A8FC1C for ; Wed, 26 Mar 2008 06:00:04 +0000 (UTC) (envelope-from bmeekhof@umich.edu) Received: FROM atom.heart.mother (c-68-40-199-244.hsd1.mi.comcast.net [68.40.199.244]) BY hellskitchen.mr.itd.umich.edu ID 47E9E661.9A4C0.27008 ; 26 Mar 2008 02:00:01 -0400 Message-ID: <47E9E660.6090101@umich.edu> Date: Wed, 26 Mar 2008 02:00:00 -0400 From: "Benjeman J. Meekhof" User-Agent: Thunderbird 2.0.0.9 (X11/20071031) MIME-Version: 1.0 To: Ivan Voras References: <47E85C00.4010601@umich.edu> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-performance@freebsd.org Subject: Re: performance tuning on perc6 (LSI) controller X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Mar 2008 06:00:05 -0000 Hi Ivan, Thanks for the response. Your response quotes my initial uneven results, but are you also implying that I most likely cannot achieve results better than the later results which use a larger filesystem blocksize? gstripe label -v -s 128k test /dev/mfid0 /dev/mfid2 #newfs -U -b 65536 /dev/stripe/test #write: 19.240875 secs (558052492 bytes/sec) #read: 20.000606 secs (536854644 bytes/sec) (iozone showed reasonably similar results - depending on recordsize would mostly be writing/reading around 500MB/s, though lows of 300MB/s were recorded in some read situations). I suppose my real question is whether there is some inherent limit in UFS2 or FreeBSD or geom that would prevent going higher than this. Maybe that's really not possible to answer, but certainly I plan to explore a few more configurations. Most of my tuning so far has been trial and error to get to this point, and all I ended up doing to finally get good results was changing filesystem blocksize to the max possible (I wanted to go to 128k but it doesn't let you do that). Apparently UFS2 and/or geom interact differently with the controller than Linux/XFS. This is no great surprise. thanks, Ben Ivan Voras wrote: > Benjeman J. Meekhof wrote: > >> My baseline was this - on linux 2.6.20 we're doing 800MB/s write and >> greater read with this configuration: 2 raid6 volumes volumes striped >> into a raid0 volume using linux software raid, XFS filesystem. Each >> raid6 is a volume on one controller using 30 PD. We've spent time >> tuning this, more than I have with FreeBSD so far. > >> time dd if=/dev/zero of=/test/deletafile bs=1M count=10240 >> 10737418240 bytes transferred in 26.473629 secs (405589209 bytes/sec) >> time dd if=/test/deletafile of=/dev/null bs=1M count=10240 >> 10737418240 bytes transferred in 157.700367 secs (68087465 bytes/sec) > > I had similar ratio of results when comparing FreeBSD+UFS to most > high-performance Linux file systems (XFS is really great!), so I'd guess > it's about as fast as you can get with this combination. > >> Any other suggestions to get best throughput? There is also HW RAID >> stripe size to adjust larger or smaller. ZFS is also on the list for >> testing. Should I perhaps be running -CURRENT or -STABLE to be get best >> results with ZFS? > > ZFS will be up to 50% faster on tests such as yours, so you should > definitely try it. Unfortunately it's not stable and you probably don't > want to use it in production. AFAIK there are no significant differences > between ZFS in -current and -stable. > > > -- Benjeman Meekhof - UM ATLAS/AGLT2 Computing office: 734-764-3450 cell: 734-417-6312 From owner-freebsd-performance@FreeBSD.ORG Wed Mar 26 09:51:33 2008 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D0D22106564A for ; Wed, 26 Mar 2008 09:51:33 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from rv-out-0910.google.com (rv-out-0910.google.com [209.85.198.184]) by mx1.freebsd.org (Postfix) with ESMTP id A3AEC8FC2F for ; Wed, 26 Mar 2008 09:51:33 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: by rv-out-0910.google.com with SMTP id g13so2285281rvb.43 for ; Wed, 26 Mar 2008 02:51:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=beta; h=domainkey-signature:received:received:message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; bh=KoD0eC2FBiLaAFj4a22+Qnv87G13wwPdRNdxAjF5r3E=; b=K0Ckh6L81B3caNvX+ea8vzN3o289NqT/41Wuh2Xij8TA9OSFI0JfJqb9aa9uw1zGkqf7HeF5LEr7xKi2La6y/DTPkD6THOIKVIQCoiCiRl/M9hW7o/2lujoD1Bqc9FRT/QNm4ugX9hGPBbTrm58pasown09z+7wePVpRdaPRXAM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=beta; h=message-id:date:from:sender:to:subject:cc:in-reply-to:mime-version:content-type:content-transfer-encoding:content-disposition:references:x-google-sender-auth; b=Z6V7ThvC48pCnYJuyoQK28304fcVBCddOmeqZl6S74O0r0A9jHo0fww9stgEdTSaoke/d9o9O8+lfxu2kcdZ0atTNKWlR3Ae3HXniopDPV2ad1pGdElMNE/le3+hvHuWFevzPvU4a2hc3z9ej3Ve4T/kg0sI5vk77nyYG3SRw4M= Received: by 10.141.197.8 with SMTP id z8mr4239071rvp.157.1206525092898; Wed, 26 Mar 2008 02:51:32 -0700 (PDT) Received: by 10.141.212.1 with HTTP; Wed, 26 Mar 2008 02:51:32 -0700 (PDT) Message-ID: <9bbcef730803260251g43583da6pff1c291db0cc9246@mail.gmail.com> Date: Wed, 26 Mar 2008 10:51:32 +0100 From: "Ivan Voras" Sender: ivoras@gmail.com To: "Benjeman J. Meekhof" In-Reply-To: <47E9E660.6090101@umich.edu> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <47E85C00.4010601@umich.edu> <47E9E660.6090101@umich.edu> X-Google-Sender-Auth: 58f4d0d2dd91f3c5 Cc: freebsd-performance@freebsd.org Subject: Re: performance tuning on perc6 (LSI) controller X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Mar 2008 09:51:33 -0000 On 26/03/2008, Benjeman J. Meekhof wrote: > Hi Ivan, > > Thanks for the response. Your response quotes my initial uneven > results, but are you also implying that I most likely cannot achieve > results better than the later results which use a larger filesystem > blocksize? > > gstripe label -v -s 128k test /dev/mfid0 /dev/mfid2 > #newfs -U -b 65536 /dev/stripe/test > #write: 19.240875 secs (558052492 bytes/sec) > #read: 20.000606 secs (536854644 bytes/sec) > > (iozone showed reasonably similar results - depending on recordsize > would mostly be writing/reading around 500MB/s, though lows of 300MB/s > were recorded in some read situations). Yes, that was my meaning. If I understood you correctly, Linux manages ~~ 800 MB/s on the array, right? > I suppose my real question is whether there is some inherent limit in > UFS2 or FreeBSD or geom that would prevent going higher than this. > Maybe that's really not possible to answer, but certainly I plan to > explore a few more configurations. I'd guess it's UFS(2), but I don't really know. My own benchmarking was on a different controller (IBM ServeRAID 8) and I got a similar ratio between Linux and FreeBSD, so I don't think it's the drivers' fault. ZFS achieves noticeably better results so it's probably not GEOM's. From owner-freebsd-performance@FreeBSD.ORG Wed Mar 26 20:45:00 2008 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 564E51065670 for ; Wed, 26 Mar 2008 20:45:00 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from bunrab.catwhisker.org (adsl-63-193-123-122.dsl.snfc21.pacbell.net [63.193.123.122]) by mx1.freebsd.org (Postfix) with ESMTP id 19FE08FC25 for ; Wed, 26 Mar 2008 20:45:00 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from bunrab.catwhisker.org (localhost [127.0.0.1]) by bunrab.catwhisker.org (8.13.3/8.13.3) with ESMTP id m2QKVwUH006395 for ; Wed, 26 Mar 2008 13:31:58 -0700 (PDT) (envelope-from david@bunrab.catwhisker.org) Received: (from david@localhost) by bunrab.catwhisker.org (8.13.3/8.13.1/Submit) id m2QKVwtB006394 for performance@freebsd.org; Wed, 26 Mar 2008 13:31:58 -0700 (PDT) (envelope-from david) Date: Wed, 26 Mar 2008 13:31:58 -0700 From: David Wolfskill To: performance@freebsd.org Message-ID: <20080326203158.GA6302@bunrab.catwhisker.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="8t9RHnE3ZwKMSgU+" Content-Disposition: inline User-Agent: Mutt/1.4.2.1i X-Mailman-Approved-At: Wed, 26 Mar 2008 20:50:06 +0000 Cc: Subject: Are sysctl(8) values useful for measuring system resource consumption? X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Mar 2008 20:45:00 -0000 --8t9RHnE3ZwKMSgU+ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable At ${work}, one of my projects is to help obtain information regarding the "developer experience," what resources are thus consumed, and figure out ways to mitigate the pain -- the objective, of course, being to help the developers be more productive within a FreeBSD environment. A couple of the perceived "pain points" are the time it takes to perform a CVS checkout and the time it takes to perform a build of the system (ours, at work -- not FreeBSD itself). As a step toward obtaining the information, I've cobbled up a Perl script that essentailly acts as a bit of "scaffolding" around time(1); the script sets things up to invoke time(1) with the "-l" flag (so we get the rusage structure information) and use "-o" to direct the output of time(1) to a file in /tmp, which the script then reads. The script then spits out a bunch of information as a single record in a CSV (Comma-Separated Variable) file (as that's the format my colleague wanted): start- and stop-timestamps, the hostname where the processes ran, the current working directory, real- and effective UIDs & GIDs; the exit code for the invoked command, the output from time(1), and (finally) the invoked command itself. (I then use a different script to read the CSV and update an RRD, then use rrdcgi(1) to generate graphs.) This has proved to be interesting, and quite possibly useful, but it merely provides a view as to the resources used by the processes being invoked from within the "scaffolding." I believe we would be well-served by also collecting information as to the resources being consumed by the system as a whole, as well -- for instance, if there's a lot of other activity on the machine in question, it might be nice to know that (and it might be even better if we had a way to characterize the rest of the workload as a whole). It would be handy if I could arrange to run vmstat(8), iostat(8), or netstat(1) in such a way that I got counters for the values immediately prior to starting the command being tested, then got a similar set of counters just after the test command completed, so I could store the "counter differences" some place handy. But that doesn't seem too readily feasible at this time. I had been trying to think of a decent way to get the overall system information for precisely the interval that I'm running the test, and then the thought occurred to me that perhaps I could invoke sysctl(8) with a suitable set of arguments both before & after invoking the process being tested; perhaps that would be a reasonable way to get information of the desired quality. I do not expect to necessarily be able to install random ports on the development machines, so there's a significant benefit to using an approach that doesn't require doing that. (I can use scripts that I write, as they are being invoked by the "test" user, from that test user's environment.) So this is a reality check: does that approach make sense? If not, what shortcomings does it have with respect to other alternatives? Please recall that the intent is to be able to place the rusage data from time(1) in a relevant context. I'm reasonably open to suggestions & alternatives. Thanks! Peace, david --=20 David H. Wolfskill david@catwhisker.org I submit that "conspiracy" would be an appropriate collective noun for cats. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --8t9RHnE3ZwKMSgU+ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.0 (FreeBSD) iEYEARECAAYFAkfqsr0ACgkQmprOCmdXAD2nugCcDWrN/826zWt3PiBNai2wnCdu p5wAniGqHHJAJIyCt0G7pOg/2G7UzUho =BwZ1 -----END PGP SIGNATURE----- --8t9RHnE3ZwKMSgU+-- From owner-freebsd-performance@FreeBSD.ORG Fri Mar 28 17:31:55 2008 Return-Path: Delivered-To: performance@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E8800106564A; Fri, 28 Mar 2008 17:31:55 +0000 (UTC) (envelope-from ap00@mail.ru) Received: from mx0.awanti.com (mx0.awanti.com [91.190.112.18]) by mx1.freebsd.org (Postfix) with ESMTP id A2DE38FC15; Fri, 28 Mar 2008 17:31:55 +0000 (UTC) (envelope-from ap00@mail.ru) Received: by mx0.awanti.com (Postfix, from userid 100) id F1AF74C022; Fri, 28 Mar 2008 20:13:18 +0300 (MSK) X-Spam-Flag: NO X-Spam-Checker-Version: SpamAssassin 3.1.9 on mx0.awanti.com X-Spam-Status: No, score=-2.3 required=6.5 tests=AWL,BAYES_00 autolearn=ham version=3.1.9 Received: from pstation (unknown [10.28.4.14]) by mx0.awanti.com (Postfix) with ESMTP id 467204C019; Fri, 28 Mar 2008 20:13:17 +0300 (MSK) Date: Fri, 28 Mar 2008 20:14:58 +0300 From: Anthony Pankov X-Mailer: The Bat! (v1.51) Personal X-Priority: 3 (Normal) Message-ID: <1333421734.20080328201458@mail.ru> To: freebsd-net@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: performance@freebsd.org Subject: packet delay because of blackhole X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Anthony Pankov List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 28 Mar 2008 17:31:56 -0000 Just for somebody convince. While analyzing client<->server HTTPS conversation one second delay in packet exchange was discovered (strongly reproducible): Sample: N time 6 0.002303 10.28.4.14 10.28.4.50 SSL Client Hello 7 0.106710 10.28.4.50 10.28.4.14 TCP 443 > 1447 [ACK] Seq=1 Ack=103 Win=65535 Len=0 8 1.045712 10.28.4.50 10.28.4.14 TLSv1 Server Hello, Certificate, Server Hello Done Another sample: 10 0.011722 10.28.4.14 10.28.4.50 TLSv1 Application Data 11 0.115933 10.28.4.50 10.28.4.14 TCP 443 > 1442 [ACK] Seq=839 Ack=519 Win=65466 Len=0 12 1.054037 10.28.4.50 10.28.4.14 TLSv1 Application Data The reason for delay is sysctl tcp.blackhole value grater than 0, much to surprise. So, turning tcp.blackhole to 0 eliminate any delay (strongly reproducible). System: FreeBSD 6_2_stable -- Best regards, Anthony mailto:ap00@mail.ru