From owner-freebsd-stable@FreeBSD.ORG Wed Aug 11 00:50:33 2004 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4E2AD16A4CE for ; Wed, 11 Aug 2004 00:50:33 +0000 (GMT) Received: from ktts.kharkov.ua (ktts.kharkov.ua [193.124.57.65]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5DE0F43D31 for ; Wed, 11 Aug 2004 00:50:32 +0000 (GMT) (envelope-from igor@ktts.kharkov.ua) Received: from ktts.kharkov.ua (smmsp@localhost [127.0.0.1]) by ktts.kharkov.ua (8.12.10/8.12.9) with ESMTP id i7B0oSTY026294 for ; Wed, 11 Aug 2004 03:50:29 +0300 (EEST) Received: (from igor@localhost) by ktts.kharkov.ua (8.12.10/8.12.9/Submit) id i7B0oSbD026292 for freebsd-stable@freebsd.org; Wed, 11 Aug 2004 03:50:28 +0300 (EEST) Message-Id: <200408110050.i7B0oSbD026292@ktts.kharkov.ua> To: freebsd-stable@freebsd.org Date: Wed, 11 Aug 2004 03:50:27 +0300 (EEST) From: igor@ktts.kharkov.ua X-Flames-To: /dev/null X-Header: /usr/include gives great headers X-Mailer: ELM [version 2.5 PL3] MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Subject: 5.2.1 performance issue X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: igor@ktts.kharkov.ua List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Aug 2004 00:50:33 -0000 Folks, My box is Dell PowerEdge 4600 with: aac0 Dell PERC 3/Di with 10x 70Gb HDDs, em0 82544EI, bge0, fxp0, 6GB RAM of which only 4GB being used, This is a Usenet server running INN 2.4.1 (release) connected to Cisco GSR 12416 with Gigabit Ethernet em0; physical connection is 1000SX Full-Duplex. Yesterday I upgraded my news server from 5.1 to 5.2.1 and ran into performance problem. Want to say from the beginning that during upgrade I transferred all my sysctl and other settings from 5.1 into 5.2.1 The box has about 40 NNTP peers (more that 300 TCP connections) and traffic flow must be at least 160mbit/100mbit in/out repectively. However after upgrade these number dropped down t/80mbit (input/output respectively), which is not normal for a Usenet server. I tried to do some FTP with nearby servers - along with news traffic it gave something like 380kbytes/sec transfer rate, so total output traffic doesn't exceed 100mbit/sec. I noticed that box started getting input errors on em0. In order to fix that I increased EM_MAX_TXD from 256 to 1024 and EM_MAX_RXD from 256 to 2048. After that Input errors disappeared but I continued getting intr_queue_drops counter increasing. I also observe full Send-Q for many TCP connections. I increased intr_queue_maxlen several times, from 400 up to 1200 with 200 increments, but it's still increasing. I changed kern.hz from 100 to 1000; also played with hw.em0.{tx,rx}_int_delay - neither one helped. Here are my modified sysctl values: kern.ipc.maxsockbuf=524288 net.inet.ip.intr_queue_maxlen=1200 net.inet.tcp.sendspace=65536 net.inet.tcp.recvspace=131072 net.inet.tcp.tcbhashsize=16384 net.inet.tcp.rfc1323=1 net.isr.enable=1 net.inet.tcp.path_mtu_discovery=0 net.inet.tcp.delayed_ack=0 net.inet.tcp.newreno=0 net.inet6.ip6.accept_rtadv=1 Also from loader.conf: kern.ipc.nmbclusters=64512 kern.ipc.nmbufs=129024 net.inet.tcp.tcbhashsize=16384 kern.maxdsiz=1342177280 # deprecated in 5.2.1 if_em_load="YES" # made this driver as a module to play with tunables in if_em.h kern.hz="1000" My friend from OpenBSD core team advised to try to increase IFQ_MAXLEN from 50 to 500, but I have a feeling it's not gonna help. If anybody has any ideas or suggestions, all are welcome and very appreciated. -- Thanks and Regards, Igor. If TCP/IP handshaking was less formal, perhaps SYN/ACK would be YO!/SUP! instead...