From owner-freebsd-performance@FreeBSD.ORG Sun Dec 19 19:17:36 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CC2DC16A4CE for ; Sun, 19 Dec 2004 19:17:36 +0000 (GMT) Received: from rproxy.gmail.com (rproxy.gmail.com [64.233.170.205]) by mx1.FreeBSD.org (Postfix) with ESMTP id 78BE743D2D for ; Sun, 19 Dec 2004 19:17:36 +0000 (GMT) (envelope-from joseph.koshy@gmail.com) Received: by rproxy.gmail.com with SMTP id z35so90503rne for ; Sun, 19 Dec 2004 11:17:35 -0800 (PST) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=beta; d=gmail.com; h=received:message-id:date:from:reply-to:to:subject:mime-version:content-type:content-transfer-encoding; b=M4bmWcZXO80Ux59bKA9ynhPF1AUip+G3fvTxnFwW4yIXKqV3V5nHTGS30UGcDEvY2JMRQBAHz28yG597dA0afRzX8279rOIl479Sj75QiKaLLV78IoCdytfzdBNdZVlPdjEfjE6eyE+9iUDPM2aesC5w+6Ig9QAG4JoW2dIChfA= Received: by 10.38.96.35 with SMTP id t35mr167849rnb; Sun, 19 Dec 2004 11:17:35 -0800 (PST) Received: by 10.38.209.11 with HTTP; Sun, 19 Dec 2004 11:17:35 -0800 (PST) Message-ID: <84dead720412191117a03ac67@mail.gmail.com> Date: Sun, 19 Dec 2004 19:17:35 +0000 From: Joseph Koshy To: freebsd-performance@freebsd.org Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Subject: [Request for Comments] PMC(3) API X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: Joseph Koshy List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Dec 2004 19:17:36 -0000 I would like to solicit comments on the following API for using CPU performance monitoring counters in FreeBSD. The intent is to use this API to support PAPI and other 'portable' performance monitoring libraries. The manual page is available here: http://people.freebsd.org/~jkoshy/download/pmc.3.txt A small tutorial on the API is available here: http://people.freebsd.org/~jkoshy/projects/perf-measurement/api-tutorial.html Code is being developed in our Perforce repository: http://perforce.freebsd.org/changeList.cgi?FSPC=//depot/user/jkoshy/projects/pmc/... From owner-freebsd-performance@FreeBSD.ORG Mon Dec 20 18:24:51 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3866216A4CE for ; Mon, 20 Dec 2004 18:24:51 +0000 (GMT) Received: from fury.noc.clara.net (fury.noc.clara.net [195.8.70.97]) by mx1.FreeBSD.org (Postfix) with ESMTP id 80E5D43D39 for ; Mon, 20 Dec 2004 18:24:50 +0000 (GMT) (envelope-from dave@uk.clara.net) Received: from dave (helo=fury.noc.clara.net) by fury.noc.clara.net with local-esmtp (Exim 4.34) id 1CgSDB-000CXI-Gq for freebsd-performance@freebsd.org; Mon, 20 Dec 2004 18:24:49 +0000 To: freebsd-performance@freebsd.org Date: Mon, 20 Dec 2004 18:24:49 +0000 From: Dave Williams Message-Id: Sender: Dave Williams Subject: Odd news transit performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Dec 2004 18:24:51 -0000 Folks, we're seeing a somewhat puzzling scenario with a FreeBSD 4.9RC box of an October '03 vintage kernel. The box is currently running innd for news transit, and in circumstances of heavy traffic load performance on the machine drops through the floor. Various reporting tools (systat, vmstat, top, etc) report that the box is idle - there's no significant contention for memory, disk, network, etc. that we can see and actually bouncing the box seems to bring performance back up to speed again for a period - restarting innd doesn't have the same effect. The machine is a 2.8GHz Xeon, Intel 10/100 and 1000 NICs, is running with two onboard LSI 53C{mumble} scsi controllers, device polling enabled, HZ=1000, and MAXDSIZ set to 1GB and MAXSSIZ/DFLDSIZ. We're a bit stumped as to where to look at the moment. :( Thanks Dave -- Dave Williams dave@uk.clara.net From owner-freebsd-performance@FreeBSD.ORG Tue Dec 21 12:58:30 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 558B716A4CF for ; Tue, 21 Dec 2004 12:58:30 +0000 (GMT) Received: from mutare.noc.clara.net (mutare.noc.clara.net [195.8.70.95]) by mx1.FreeBSD.org (Postfix) with ESMTP id 98C1D43D45 for ; Tue, 21 Dec 2004 12:58:29 +0000 (GMT) (envelope-from ollie@mutare.noc.clara.net) Received: from ollie by mutare.noc.clara.net with local (Exim 4.43) id 1Cgjau-000BWA-EY for freebsd-performance@freebsd.org; Tue, 21 Dec 2004 12:58:28 +0000 Date: Tue, 21 Dec 2004 12:58:28 +0000 From: Ollie Cook To: freebsd-performance@freebsd.org Message-ID: <20041221125828.GE42562@mutare.noc.clara.net> References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.1i X-Operating-System: FreeBSD 4.10-STABLE i386 X-NCC-RegID: uk.claranet Sender: Ollie Cook X-Mailman-Approved-At: Tue, 21 Dec 2004 13:33:07 +0000 Subject: Re: Odd news transit performance problem X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Dec 2004 12:58:30 -0000 On Mon, Dec 20, 2004 at 06:24:49PM +0000, Dave Williams wrote: > Various reporting tools (systat, vmstat, top, etc) report that the > box is idle - there's no significant contention for memory, disk, > network, etc. that we can see and actually bouncing the box seems > to bring performance back up to speed again for a period - restarting > innd doesn't have the same effect. I'd like to flesh out this thread with some more detail. I've put my interpretation of the detail in too, but if I'm not correct, clarification would be appreciated! The host is built as follows: - Intel Xeon 2.8GHz - 3GB RAM - 1x fxp NIC - 2x em NICs - 2x sym SCSI controllers <1010-66> - 14x 18GB U160 SCSI disks (not quite split equally across the two SCSI controllers) - vinum is used to stripe volumes across multiple spindles. In particular the history database is striped over 10 devices. The host is fed news from only two sources which on average equates to 25 streams. The total volume this host is handling per day is of the order of 1.4TB inbound. At present it's handling ~50 articles per second (average article size is 350KB). News is then fed out to a number of other hosts which equates to a further 40 streams. We are seeing this host not keep up with all the news being offered to it, and consequently the hosts feeding it are keeping a backlog of articles to offer it later. The host that is backlogging is behind the first em interface. 'top' shows the CPU to be on average 25% idle, and that little swap is being used: CPU states: 11.3% user, 1.6% nice, 47.1% system, 15.2% interrupt, 24.9% idle Mem: 438M Active, 1815M Inact, 658M Wired, 102M Cache, 199M Buf, 4348K Free Swap: 4096M Total, 156K Used, 4096M Free 'vmstat 1' shows a number of pages being paged out per second, but few ever being paged in? How does this tally with the fact that < 1MB of swap is in use? I think my understanding of paging could be at fault here. :) procs memory page disks faults cpu r b w avm fre flt re pi po fr sr da0 da1 in sy cs us sy id 3 0 0 576472 88672 4854 0 0 139 9465 0 0 1 1474 10821 3655 18 48 33 2 0 0 577492 148356 4796 1 0 94 9566 19789 0 0 1675 12267 4471 17 53 29 1 6 0 576088 129444 5655 1 0 139 10133 0 0 1 1511 11115 4033 20 51 29 1 6 0 576172 106752 5110 0 0 146 10618 0 0 0 1462 12521 4411 24 50 26 2 0 0 577168 167000 4753 0 1 142 8541 19797 0 4 1664 11800 4307 20 52 28 3 0 0 580476 143228 6905 2 1 93 11867 0 0 1 1426 10999 3798 19 51 30 2 6 0 579984 126212 4678 0 0 209 8677 0 0 0 1492 14254 5125 15 44 41 4 0 0 575016 112616 6526 0 0 94 11638 0 0 1 1736 11860 4002 21 44 35 2 5 0 576932 93844 5016 0 0 93 8178 0 0 0 1477 8863 2898 16 45 39 0 7 0 579076 164308 2546 1 7 3553 4244 19793 13 1 3921 10493 2388 9 43 47 Both 'systat -vmstat' and 'iostat' show the disks are not busy. They are transferring approximately 2.0MB/s on average. 'systat -vmstat' indicates the devices are <20% busy. Indeed, writing from /dev/zero to a vinum volume striped over ten disks I can achieve a further 5MB/s per disk over and above what the system is usually generating. This still only pushes the disks to 50% busy. Network-wise the fxp interface does ~45mbit/s (majority outbound), em0 does 20mbit (majority inbound) does and em1 does ~150mbit/s (majority outbound). Duplex settings are all correct and 'netstat' shows very few errors on host interfaces: Name Mtu Network Address Ipkts Ierrs Opkts Oerrs Coll fxp0 1500 00:e0:18:a4:d4:4d 96710493 0 103018264 2 0 em0 1500 00:e0:18:a4:d4:4c 393932814 0 429947240 0 0 em1 1500 00:07:e9:0f:9a:34 465099760 0 559720416 0 0 There appears not to be a shortage of mbufs: # netstat -m 1307/5936/262144 mbufs in use (current/peak/max): 1307 mbufs allocated to data 1301/4970/65536 mbuf clusters in use (current/peak/max) 11424 Kbytes allocated to network (5% of mb_map in use) 0 requests for memory denied 0 requests for memory delayed 0 calls to protocol drain routines There doesn't appear to be another bottleneck in the network subsystem because an scp from one host to the other over em0 generates a further 30mbit/s of traffic. Receive queues are about 20k on each inbound news stream. Does this figure indicate that data that has not yet been transferred from the kernel to user space? The following sysctl's have been set over the years: net.inet.tcp.inflight_enable=1 vfs.vmiodirenable=1 vfs.lorunningspace=2097152 vfs.hirunningspace=4194304 kern.maxfiles=262144 kern.maxfilesperproc=32768 net.inet.tcp.rfc1323=1 net.inet.tcp.delayed_ack=0 net.inet.tcp.sendspace=131070 net.inet.tcp.recvspace=131070 net.inet.udp.recvspace=65535 net.inet.udp.maxdgram=57344 net.local.stream.recvspace=65535 net.local.stream.sendspace=65535 kern.polling.enable=1 as well as the following kernel configuration options: options NMBCLUSTERS=65536 options MAXDSIZ="(1024*1024*1024)" options MAXSSIZ="(256*1024*1024)" options DFLDSIZ="(256*1024*1024)" options DEVICE_POLLING options HZ=1000 Given that the CPU is not wedged at 100%, that there is free memory, that the disks have plenty of bandwidth left, and likewise the network interfaces, I'm convinced that this host ought to be keeping up and not causing other hosts to keep a backlog for it. Does anyone have any suggestions for where else we might look to see why this host doesn't appear to be performing as well as one might expect it do? Yours, Ollie -- Ollie Cook Systems Architect, Claranet UK ollie@uk.clara.net +44 20 7685 8065 From owner-freebsd-performance@FreeBSD.ORG Thu Dec 23 18:09:16 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1750316A4CE; Thu, 23 Dec 2004 18:09:16 +0000 (GMT) Received: from Ehost067.exch005intermedia.net (ehost067.exch005intermedia.net [64.78.61.105]) by mx1.FreeBSD.org (Postfix) with ESMTP id D2EA343D41; Thu, 23 Dec 2004 18:09:15 +0000 (GMT) (envelope-from jbehl@fastclick.com) X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Thu, 23 Dec 2004 10:09:14 -0800 Message-ID: <7632915A8F000C4FAEFCF272A880344165164F@Ehost067.exch005intermedia.net> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: %cpu in system - squid performance in FreeBSD 5.3 thread-index: AcTcgGtwqE5TcaCWT+CNhckTP6RbvAMmJgnA From: "Jeff Behl" To: , Subject: RE: %cpu in system - squid performance in FreeBSD 5.3 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Dec 2004 18:09:16 -0000 =20 As a follow up to the below (original message at the very bottom), I installed a load balancer in front of the machines which terminates the tcp connections from clients and opens up a few, persistent connections to each server over which requests are pipelined. In this scenario everything is copasetic: last pid: 3377; load averages: 0.12, 0.09, 0.08 up 0+17:24:53 10:02:13 31 processes: 1 running, 30 sleeping CPU states: 5.1% user, 0.0% nice, 1.8% system, 1.2% interrupt, 92.0% idle Mem: 75M Active, 187M Inact, 168M Wired, 40K Cache, 214M Buf, 1482M Free Swap: 4069M Total, 4069M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMAND 474 squid 96 0 68276K 62480K select 0 53:38 16.80% 16.80% squid 311 bind 20 0 10628K 6016K kserel 0 12:28 0.00% 0.00% named It's actually so good that one machine can now handle all traffic (around 180 Mb/s) at < %50 cpu utilization. Seems like something in the network stack is responsible for the high %system cpu util... jeff -----Original Message----- From: owner-freebsd-performance@freebsd.org [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Jeff Behl Sent: Tuesday, December 07, 2004 9:17 AM To: Sean Chittenden Cc: freebsd-performance@freebsd.org Subject: Re: %cpu in system - squid performance in FreeBSD 5.3 I upgraded to STABLE but most cpu time is still being spent in system. This system is doing ~20Mb/s total with all content being grabbed out of memory. I see similar results when running MySQL (a lot of time being spent in system) Any ideas on what updates to be on the lookout for that might help with this? Am I right in guessing that this is a SMP issue and doesn't have anything to do with AMD architecture? thx FreeBSD www2 5.3-STABLE FreeBSD 5.3-STABLE #2: Sun Dec 5 21:06:14 PST=20 2004 root@www2.cdn.sjc:/usr/obj/usr/src/sys/SMP amd64 last pid: 15702; load averages: 0.15, 0.31, 0.31 up=20 0+19:55:14 09:09:28 38 processes: 2 running, 36 sleeping CPU states: 5.4% user, 0.0% nice, 12.7% system, 3.4% interrupt, 78.4% idle Mem: 163M Active, 284M Inact, 193M Wired, 72K Cache, 214M Buf, 1245M Free Swap: 4069M Total, 4069M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMAND 486 squid 96 0 79820K 73996K CPU1 1 110:00 15.04% 15.04% squid 480 squid 96 0 75804K 70012K select 0 105:56 14.89% 14.89% squid Sean Chittenden wrote: >> but the % system time can fluctuate up to 60 at times. My question=20 >> is if this is about the type of performance I could expect, or if=20 >> people have seen better. > > > I don't know about other people, but I suspect you're running into=20 > lock contention. Try using a post 5.3 snapshot (something from > RELENG_5) since alc@ has set debug.mpsafevm=3D1, which lets many calls = > to the VM run without GIANT, which I suspect is your problem and why=20 > the system usage is all over the place. -sc > _______________________________________________ freebsd-performance@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-performance To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" howdy, I've got a dual proc AMD64 (2gHz) FreeBSD 5.3R system running two squid processes (to take advantage of both CPUs). Each process is doing around 195 req/s, and the total bandwidth is ~40Mb/s (gig nic via bge driver). Squid is being used exclusively as a reverse proxy, with all content being served out of memory (very little disk activity). Top shows: CPU states: 16.0% user, 0.0% nice, 42.7% system, 7.6% interrupt, 33.6% idle Mem: 898M Active, 569M Inact, 179M Wired, 214M Buf, 171M Free Swap: 4069M Total, 4069M Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMAND 14598 squid 108 0 463M 459M select 0 39.2H 59.96% 59.96% squid 14605 squid 105 0 421M 416M CPU0 1 38.4H 49.95% 49.95% squid but the % system time can fluctuate up to 60 at times. My question is if this is about the type of performance I could expect, or if people have seen better. I was expecting to see much better performance, seeing how everything is being served out of memory, but maybe I'm asking too much? 400 reqs/s from RAM doesn't seem like much. Is this a FreeBSD issue (anybody else with similar experience)? A majority of the cpu time being spent in system would seem to indictate such. What is all the system load? How can i tell? Any help/pointers/remarks appreciated thanks, jeff From owner-freebsd-performance@FreeBSD.ORG Thu Dec 23 21:11:22 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 78B7716A4CE for ; Thu, 23 Dec 2004 21:11:22 +0000 (GMT) Received: from mail.trippynames.com (mail.trippynames.com [38.113.223.19]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5904E43D45 for ; Thu, 23 Dec 2004 21:11:22 +0000 (GMT) (envelope-from sean@chittenden.org) Received: from localhost (localhost [127.0.0.1]) by mail.trippynames.com (Postfix) with ESMTP id 0F080A6C7E; Thu, 23 Dec 2004 13:10:56 -0800 (PST) Received: from mail.trippynames.com ([127.0.0.1]) by localhost (rand.nxad.com [127.0.0.1]) (amavisd-new, port 10024) with LMTP id 20313-02; Thu, 23 Dec 2004 13:10:45 -0800 (PST) Received: from [192.168.102.100] (dsl231-047-005.sea1.dsl.speakeasy.net [216.231.47.5]) by mail.trippynames.com (Postfix) with ESMTP id 0CE23A6C27; Thu, 23 Dec 2004 13:10:44 -0800 (PST) In-Reply-To: <7632915A8F000C4FAEFCF272A880344165164F@Ehost067.exch005intermedia.net> References: <7632915A8F000C4FAEFCF272A880344165164F@Ehost067.exch005intermedia.net> Mime-Version: 1.0 (Apple Message framework v619) Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: <298151B6-5527-11D9-B830-000A95C705DC@chittenden.org> Content-Transfer-Encoding: 7bit From: Sean Chittenden Date: Thu, 23 Dec 2004 13:11:07 -0800 To: "Jeff Behl" X-Mailer: Apple Mail (2.619) cc: freebsd-performance@freebsd.org Subject: Re: %cpu in system - squid performance in FreeBSD 5.3 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 23 Dec 2004 21:11:22 -0000 > As a follow up to the below (original message at the very bottom), I > installed a load balancer in front of the machines which terminates the > tcp connections from clients and opens up a few, persistent connections > to each server over which requests are pipelined. In this scenario > everything is copasetic: Interesting... I wonder what the lock contention path is between the VM and network stack. Has anyone else noticed this in their testing? Did you try a post-5.3 release or not? rwatson@ just MFC'ed a bunch of network locking fixes to RELENG_5, but none-stand out in my mind as being something that'd potentially fix your issue. Actually, he probably knows better than anyone what this could be. Some of the post RELENG_5_3 commits by alc@ could explain this, however, and is why I ask what the specific version/build time is for your release. -sc -- Sean Chittenden From owner-freebsd-performance@FreeBSD.ORG Sat Dec 25 12:40:18 2004 Return-Path: Delivered-To: freebsd-performance@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3A9A016A4CE; Sat, 25 Dec 2004 12:40:18 +0000 (GMT) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by mx1.FreeBSD.org (Postfix) with ESMTP id A893C43D2F; Sat, 25 Dec 2004 12:40:17 +0000 (GMT) (envelope-from robert@fledge.watson.org) Received: from fledge.watson.org (localhost [127.0.0.1]) by fledge.watson.org (8.13.1/8.13.1) with ESMTP id iBPCb4wv038775; Sat, 25 Dec 2004 07:37:04 -0500 (EST) (envelope-from robert@fledge.watson.org) Received: from localhost (robert@localhost)iBPCb3Ah038772; Sat, 25 Dec 2004 12:37:04 GMT (envelope-from robert@fledge.watson.org) Date: Sat, 25 Dec 2004 12:37:03 +0000 (GMT) From: Robert Watson X-Sender: robert@fledge.watson.org To: Jeff Behl In-Reply-To: <7632915A8F000C4FAEFCF272A880344165164F@Ehost067.exch005intermedia.net> Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: freebsd-net@freebsd.org cc: freebsd-performance@freebsd.org Subject: RE: %cpu in system - squid performance in FreeBSD 5.3 X-BeenThere: freebsd-performance@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Performance/tuning List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Dec 2004 12:40:18 -0000 On Thu, 23 Dec 2004, Jeff Behl wrote: > As a follow up to the below (original message at the very bottom), I > installed a load balancer in front of the machines which terminates the > tcp connections from clients and opens up a few, persistent connections > to each server over which requests are pipelined. In this scenario > everything is copasetic: I'm not very familiar with Squid's architecture, but I would anticipate that what you're seeing is that the cost of additional connections served in parallel is pretty high due to the use of processes. Specifically: if each TCP connection being served gets its own process, and there are a lot of TCP connections, you'll be doing a lot of process forking, context switching, exceeding cache sizes, etc. With just a couple of connections, even if they're doing the same "work", the overhead is much lower. Depending on how much time you're willing to invest in this, we can probably do quite a bit to diagnose where the cost is coming from and look for any specific problems or areas we could optimize. I might start by turning on kernel profiling and doing a profile dump under load. Be aware that turning on profiling uses up a lot of CPU itself, so will reduce the capacity of the system. There's probably documentation elsewhere, but the process I use to set up profiling is here: http://www.watson.org/~robert/freebsd/netperf/profile/ Note that it warns the some results may be incorrect on SMP. I think it would be useful to give it a try anyway just to see if we get something useful. The next thing that would be interesting is using mutex profiling to measure contention on mutexes. The instructions in MUTEX_PROFILING(9) are pretty decent for this purpose. On an SMP system, time spent contending a mutex in active use will be spent spinning, which means wasted CPU. You can cause the kernel to block threads instead using options NO_ADAPTIVE_MUTEXES, but measurement in the past has shown that the overhead of blocking and restarting a thread is generally higher than just spinning. It would be useful to see the output of dmesg at boot to see if any performance options are obviously out of place. Likewise, the output of a couple of stats commands while the system is active would be useful -- for example, a couple of snapshots of "systat -vmstat 1", "netstat -mb", "vmstat -i", "top -S", and "iostat". As a final question: other than CPU consumption, do you have a reliable way to measure how efficiently the system is operating -- in particular, how fast it is able to serve data? Having some sort of metric for performance can be quite useful in optimizing, as it can tell us whether we're accomplishing incremental improvements prior to performance improving to a point where the system isn't saturated. Typical forms might be some sort of web benchmark, etc. If so, it might be interesting to compare the performance of the following configurations: - UP kernel (no SMP compiled in) - SMP kernel but SMP disabled using the appropriate tunable - SMP kernel with SMP enabled Finally, I'm not sure if the box has HTT on it, and if so, if HTT is enabled, but you might want to try disabling it, as it has proven to be relatively ineffective in improving performance in the application tests I've run, while at the same time increasing operating overhead. Another variable that might be interesting to look at is net.isr.enable. To do this, you want to be running 5-STABLE rather than 5.3-RELEASE, as I merged at least one significant bug fix that affects its operation. By default, net.isr.enable is 0, meaning that all inbound network traffic is processed in the netisr thread. When this variable is set to 1, inbound network traffic will be, where possible, directly dispatched in the device driver ithread. This has a couple of impacts, but the main ones are that there are substantially fewer context switches being done, and that parallelism is possible between the netisr and each interface card. This is an experimental feature, so be on the lookout for any resulting nits. Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Principal Research Scientist, McAfee Research > > last pid: 3377; load averages: 0.12, 0.09, 0.08 > up 0+17:24:53 10:02:13 > 31 processes: 1 running, 30 sleeping > CPU states: 5.1% user, 0.0% nice, 1.8% system, 1.2% interrupt, 92.0% > idle > Mem: 75M Active, 187M Inact, 168M Wired, 40K Cache, 214M Buf, 1482M Free > Swap: 4069M Total, 4069M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU > COMMAND > 474 squid 96 0 68276K 62480K select 0 53:38 16.80% 16.80% > squid > 311 bind 20 0 10628K 6016K kserel 0 12:28 0.00% 0.00% > named > > > > It's actually so good that one machine can now handle all traffic > (around 180 Mb/s) at < %50 cpu utilization. Seems like something in the > network stack is responsible for the high %system cpu util... > > jeff > > > -----Original Message----- > From: owner-freebsd-performance@freebsd.org > [mailto:owner-freebsd-performance@freebsd.org] On Behalf Of Jeff Behl > Sent: Tuesday, December 07, 2004 9:17 AM > To: Sean Chittenden > Cc: freebsd-performance@freebsd.org > Subject: Re: %cpu in system - squid performance in FreeBSD 5.3 > > I upgraded to STABLE but most cpu time is still being spent in system. > > This system is doing ~20Mb/s total with all content being grabbed out of > memory. I see similar results when running MySQL (a lot of time being > spent in system) > > Any ideas on what updates to be on the lookout for that might help with > this? Am I right in guessing that this is a SMP issue and doesn't have > anything to do with AMD architecture? > > thx > > > > FreeBSD www2 5.3-STABLE FreeBSD 5.3-STABLE #2: Sun Dec 5 21:06:14 PST > 2004 root@www2.cdn.sjc:/usr/obj/usr/src/sys/SMP amd64 > > > last pid: 15702; load averages: 0.15, 0.31, 0.31 up > 0+19:55:14 09:09:28 > 38 processes: 2 running, 36 sleeping > CPU states: 5.4% user, 0.0% nice, 12.7% system, 3.4% interrupt, 78.4% > idle > Mem: 163M Active, 284M Inact, 193M Wired, 72K Cache, 214M Buf, 1245M > Free > Swap: 4069M Total, 4069M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU > COMMAND > 486 squid 96 0 79820K 73996K CPU1 1 110:00 15.04% 15.04% > squid > 480 squid 96 0 75804K 70012K select 0 105:56 14.89% 14.89% > squid > > > > > > Sean Chittenden wrote: > > >> but the % system time can fluctuate up to 60 at times. My question > >> is if this is about the type of performance I could expect, or if > >> people have seen better. > > > > > > I don't know about other people, but I suspect you're running into > > lock contention. Try using a post 5.3 snapshot (something from > > RELENG_5) since alc@ has set debug.mpsafevm=1, which lets many calls > > to the VM run without GIANT, which I suspect is your problem and why > > the system usage is all over the place. -sc > > > > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to > "freebsd-performance-unsubscribe@freebsd.org" > > > > howdy, > > I've got a dual proc AMD64 (2gHz) FreeBSD 5.3R system running two squid > processes (to take advantage of both CPUs). Each process is doing > around 195 req/s, and the total bandwidth is ~40Mb/s (gig nic via bge > driver). Squid is being used exclusively as a reverse proxy, with all > content being served out of memory (very little disk activity). > > Top shows: > > CPU states: 16.0% user, 0.0% nice, 42.7% system, 7.6% interrupt, 33.6% > idle > Mem: 898M Active, 569M Inact, 179M Wired, 214M Buf, 171M Free > Swap: 4069M Total, 4069M Free > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU CPU COMMAND > 14598 squid 108 0 463M 459M select 0 39.2H 59.96% 59.96% squid > 14605 squid 105 0 421M 416M CPU0 1 38.4H 49.95% 49.95% squid > > but the % system time can fluctuate up to 60 at times. My question is > if this is about the type of performance I could expect, or if people > have seen better. I was expecting to see much better performance, > seeing how everything is being served out of memory, but maybe I'm > asking too much? 400 reqs/s from RAM doesn't seem like much. Is this a > FreeBSD issue (anybody else with similar experience)? A majority of the > cpu time being spent in system would seem to indictate such. What is > all the system load? How can i tell? > > Any help/pointers/remarks appreciated > > thanks, > jeff > _______________________________________________ > freebsd-performance@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-performance > To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" >