From owner-freebsd-stable@FreeBSD.ORG Sun Jan 10 00:41:00 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1C4A41065692 for ; Sun, 10 Jan 2010 00:41:00 +0000 (UTC) (envelope-from garrettmoore@gmail.com) Received: from mail-iw0-f198.google.com (mail-iw0-f198.google.com [209.85.223.198]) by mx1.freebsd.org (Postfix) with ESMTP id D423A8FC15 for ; Sun, 10 Jan 2010 00:40:59 +0000 (UTC) Received: by iwn36 with SMTP id 36so13875396iwn.3 for ; Sat, 09 Jan 2010 16:40:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:content-type; bh=zI+TGMH9gVOwQ99ZtcICBxHtU8iAZ0A9//RtGccG3Ic=; b=nOk03s5jfUY8Uk/snu7YAlLSUbxZ89yW2JaejqpMayW7mWKE0UGrqVqNcRNMzd25g/ 4XVNyy8rmnFF7AD8zOSMe7quBvfeOsyEMWHYrsX+vzJexw9Ea/MoP3/gtNKafChF02b5 GMR5jI1Z/H38fFhvZTD8HuwrlRamdrK1p6jYE= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=tadttgQiNLNNpRdAbXiAZbf7/WX7ApgQlNEzWsbBUb/ipDkr3ms5Nyr1hRo/kFUGIy 6GiTNnOnUtUJq2cGIhk0YVSxZ1mRkHDC2DXO7nS36+XJ0TvSkaZP0A1Q5lBwhIE+n9nW AIXbIuOA/prdBfmeR0XNa/pvjGHpVYdQv1OEs= MIME-Version: 1.0 Received: by 10.231.153.69 with SMTP id j5mr2290046ibw.33.1263084053372; Sat, 09 Jan 2010 16:40:53 -0800 (PST) In-Reply-To: <2ae8edf31001091051l156cb57alf549cfe06f1c7197@mail.gmail.com> References: <7346c5c61001030842r7dc76199y51e4c1c90a3eea6e@mail.gmail.com> <7346c5c61001080831w375d158fu5b1996ee58cb0f8d@mail.gmail.com> <2ae8edf31001091051l156cb57alf549cfe06f1c7197@mail.gmail.com> Date: Sat, 9 Jan 2010 19:40:53 -0500 Message-ID: <7346c5c61001091640w1a0ffb5y3ffe97cb88c76436@mail.gmail.com> From: Garrett Moore To: freebsd-stable Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: ZFS performance degradation over time X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Jan 2010 00:41:00 -0000 After being up for a few days and having good performance, I turned on rTorrent and downloaded two 700MB files. After these torrents completed, my performance has tanked again. My combined read/write speed to the array won't exceed about 25MB/s. It seems that after having downloaded a few torrents, resources are grabbed and then not released. From top: Mem: 2326M Active, 962M Inact, 484M Wired, 82M Cache, 418M Buf, 87M Free Nothing in userland is using a significant amount of memory. eg rTorrent is using 41MB according to top. Killing rTorrent does not alleviate the performance problems. arcstats.size is hovering around 30-40MB. [root@leviathan ~]# sysctl kstat.zfs.misc.arcstats.size kstat.zfs.misc.arcstats.size: 28953448 [root@leviathan ~]# sysctl vm.stats.vm.v_inactive_count vm.stats.vm.v_inactive_count: 237831 [root@leviathan ~]# sysctl vm.stats.vm.v_active_count vm.stats.vm.v_active_count: 595762 [root@leviathan ~]# sysctl vm.stats.vm.v_cache_count vm.stats.vm.v_cache_count: 21472 If no-one has any questions, I'll try Artem's suggestion of wasting a bunch of memory in Perl/Python and forcing some memory to be swapped out. (I don't want to do it yet in case someone wants a specific number before I do that).