From owner-freebsd-bugs@FreeBSD.ORG Sun Jan 17 18:00:07 2010 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C168A1065693 for ; Sun, 17 Jan 2010 18:00:07 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 95DB38FC1D for ; Sun, 17 Jan 2010 18:00:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.3/8.14.3) with ESMTP id o0HI07XZ016479 for ; Sun, 17 Jan 2010 18:00:07 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.3/8.14.3/Submit) id o0HI07H0016478; Sun, 17 Jan 2010 18:00:07 GMT (envelope-from gnats) Resent-Date: Sun, 17 Jan 2010 18:00:07 GMT Resent-Message-Id: <201001171800.o0HI07H0016478@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Miks Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CEC3D1065693 for ; Sun, 17 Jan 2010 17:50:59 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from www.freebsd.org (www.freebsd.org [IPv6:2001:4f8:fff6::21]) by mx1.freebsd.org (Postfix) with ESMTP id B5D4C8FC14 for ; Sun, 17 Jan 2010 17:50:59 +0000 (UTC) Received: from www.freebsd.org (localhost [127.0.0.1]) by www.freebsd.org (8.14.3/8.14.3) with ESMTP id o0HHoxIH034873 for ; Sun, 17 Jan 2010 17:50:59 GMT (envelope-from nobody@www.freebsd.org) Received: (from nobody@localhost) by www.freebsd.org (8.14.3/8.14.3/Submit) id o0HHoxaM034872; Sun, 17 Jan 2010 17:50:59 GMT (envelope-from nobody) Message-Id: <201001171750.o0HHoxaM034872@www.freebsd.org> Date: Sun, 17 Jan 2010 17:50:59 GMT From: Miks To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: misc/142914: ZFS performance degratation over time X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jan 2010 18:00:07 -0000 >Number: 142914 >Category: misc >Synopsis: ZFS performance degratation over time >Confidential: no >Severity: serious >Priority: low >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sun Jan 17 18:00:07 UTC 2010 >Closed-Date: >Last-Modified: >Originator: Miks >Release: Freebsd 8.0 Release >Organization: >Environment: FreeBSD host1 8.0-RELEASE FreeBSD 8.0-RELEASE #0: Sat Nov 21 15:02:08 UTC 2009 root@mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64 >Description: After migrating servers to from UFS to ZFS, we got constant file system performance degradation in couple time by week. System load has not been changed since migration to ZFS, so only reason of this problems can be ZFS itself. We had this problems on Intel server systems with 8GB and more RAM. We also tried to disable prefetch, but problem still existed. Even with latest FreeBSD 8.0 Stable we had same problem. One solution to this problem is - stop all httpd, php-cgi and mysql processes, wait 1-2 minutes and then start it again. On other servers we just migrated back to UFS and problem was gone. One strange thing we noticed is - when server have performance problem, "zpool iostat" is showing like this " pool used avail read write read write ---------- ----- ----- ----- ----- ----- ----- zroot 113G 26.1G 53 392 2.80M 20.3M " after processes restart: " pool used avail read write read write ---------- ----- ----- ----- ----- ----- ----- zroot 99.9G 39.1G 6 40 367K 1.12M " How can "used" value differ? >How-To-Repeat: >Fix: >Release-Note: >Audit-Trail: >Unformatted: