From owner-freebsd-stable@FreeBSD.ORG Mon Nov 2 11:55:58 2009 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 43807106566B for ; Mon, 2 Nov 2009 11:55:58 +0000 (UTC) (envelope-from freebsd-stable@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id C65198FC0A for ; Mon, 2 Nov 2009 11:55:57 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.50) id 1N4vVp-0006qQ-Ey for freebsd-stable@freebsd.org; Mon, 02 Nov 2009 12:55:53 +0100 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 02 Nov 2009 12:55:53 +0100 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Mon, 02 Nov 2009 12:55:53 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-stable@freebsd.org From: Ivan Voras Date: Mon, 02 Nov 2009 12:55:34 +0100 Lines: 35 Message-ID: References: <772532900-1257123963-cardhu_decombobulator_blackberry.rim.net-1402739480-@bda715.bisx.prod.on.blackberry> <4AEEBD4B.1050407@quip.cz> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@ger.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Thunderbird 2.0.0.23 (X11/20090928) In-Reply-To: <4AEEBD4B.1050407@quip.cz> Sender: news Subject: Re: Performance issues with 8.0 ZFS and sendfile/lighttpd X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Nov 2009 11:55:58 -0000 Miroslav Lachman wrote: > Ivan Voras wrote: >> gnukix@alltel.blackberry.com wrote: >>> I can send in more documentation later but I am seeing severe zfs >>> performance issues with lighttpd. Same machine using UFS will push >>> 1gbit or more but same content and traffic load can not hit 200mbit. >>> Ufs does around 3 megabytes/sec IO at 800mbit network but zfs pushes >>> the disks into the ground with 50+ megabytes/sec dusk i/o. No >>> compression no atime no checksums on zfs and still same IO levels. Ufs >>> with soft updates and atime on. Orders of magnitude more disk IO... >>> Like zfs isn't using cache or isn't coalescing disk reads or both. >>> Has anyone else seen this or have any recommendations? Lighttpd config >>> remains exactly the same as well FYI. Only difference is ufs vs zfs. >> >> AFAIK, ZFS is incompatible (currently) with some advanced VM operations >> (like mmap, and I think sendfile relies on the same mechanism as mmap), >> so that could be a cause of the slowdown. Though I'm surprised you can >> only get 200 MBit/s - that's 25 MB/s and I think that even with multiple >> memcpy-ing data around the kernel you should be able to get hundreds of >> MB/s on newer hardware (which normally really can achieve tens of >> gigabytes/s of sustained memory access). > > I have more strange issue with Lighttpd in jail on top of ZFS. Lighttpd > is serving static content (mp3 downloads thru flash player). Is runs > fine for relatively small number of parallel clients with bandwidth > about 30 Mbps, but after some number of clients is reached (about 50-60 > parallel clients) the throughput drops down to 6 Mbps. > > I can server hundereds of clients on same HW using Lighttpd not in jail > and UFS2 with gjournal instead of ZFS reaching 100 Mbps (maybe more). > > I don't know if it is ZFS or Jail issue. Do you have actual disk IO or is the vast majority of your data served from the caches? (actually - the same question to the OP)