From owner-freebsd-net@FreeBSD.ORG Wed May 26 05:59:15 2004 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BC6C916A4CE for ; Wed, 26 May 2004 05:59:15 -0700 (PDT) Received: from mail.dragondata.com (server2-b.your.org [64.202.113.66]) by mx1.FreeBSD.org (Postfix) with ESMTP id 5E57F43D58 for ; Wed, 26 May 2004 05:59:15 -0700 (PDT) (envelope-from toasty@dragondata.com) Received: (qmail 57223 invoked by uid 1092); 26 May 2004 10:44:13 -0000 Received: from toasty@dragondata.com by server2.dragondata.com by uid 82 with qmail-scanner-1.20rc3 (uvscan: v4.2.40/v4296. spamassassin: 2.60-cvs. Clear:RC:1:. Processed in 1.202854 secs); 26 May 2004 10:44:13 -0000 Received: from ppp045.dhcp.your.org (HELO ?199.165.179.45?) (199.165.179.45) by mail.dragondata.com with RC4-SHA encrypted SMTP; 26 May 2004 10:44:11 -0000 Mime-Version: 1.0 (Apple Message framework v613) In-Reply-To: References: Content-Type: text/plain; charset=US-ASCII; format=flowed Message-Id: Content-Transfer-Encoding: 7bit From: Kevin Day Date: Wed, 26 May 2004 05:44:32 -0500 To: freebsd-net@freebsd.org X-Mailer: Apple Mail (2.613) Subject: Re: Sendfile performance regression from 4.x to 5.x X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 May 2004 12:59:15 -0000 For those who asked for more info... The issue here seems to have been SCHED_ULE. Switching Server 2 back to SCHED_4BSD puts its performance back on par with 4.8. No idea why yet, but I'm still poking around. It seems to have something to do with how long thttpd's process has been running though, the longer it runs the worse it gets. On May 16, 2004, at 12:27 AM, Kevin Day wrote: > > > We've got two nearly identically configured servers, serving the exact > same load. Quick config: > > Dual P4 2.8Ghz with HTT enabled, 2GB RAM > ahc SCSI adapter, with 4 drives in a vinum RAID5 config > Dual bge gigabit network cards > > Server 1 is running 4.8 > Server 2 is running 5.2.1 > > Other than OS version, everything is identical for all practical > purposes. > I'm running thttpd with USE_SENDFILE enabled. Both servers are > receiving approximately equal numbers of requests to download some > large files. (5MB to 200MB) > > Server 1 can easily push 300-500mbps. I'm pretty sure it could go > higher, we just run into bandwidth limits at that point. With the caps > removed, we've hit 800mbps for a short while. > Server 2 seems to cap out around 80mbps, with thttpd spending most of > it's time in "Giant" or "kqueue" according to top, the CPU around 90% > idle. > > What's really strange is that right when thttpd starts up on 5.2, it > will start sending 200mbps or higher for the first 10-15 minutes, and > then start tapering off back down to 80mbps. It doesn't look like a > thttpd bug, since it seems to work flawlessly on 4.x. > > I know a lot of work is being done on locking, sendfile and kqueue > fun, so I'm not complaining. I just wanted someone to know this was > happening, and hopefully have someone point out something obvious I'm > missing that could be causing this... Anyone run into this before? > > -- Kevin