From owner-freebsd-fs@FreeBSD.ORG Tue May 31 16:09:12 2011 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F37E6106564A; Tue, 31 May 2011 16:09:11 +0000 (UTC) (envelope-from prvs=11320824c3=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 07F9C8FC08; Tue, 31 May 2011 16:09:10 +0000 (UTC) X-MDAV-Processed: mail1.multiplay.co.uk, Tue, 31 May 2011 16:57:51 +0100 X-Spam-Processed: mail1.multiplay.co.uk, Tue, 31 May 2011 16:57:47 +0100 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on mail1.multiplay.co.uk X-Spam-Level: X-Spam-Status: No, score=-5.0 required=6.0 tests=USER_IN_WHITELIST shortcircuit=ham autolearn=disabled version=3.2.5 Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50013430624.msg; Tue, 31 May 2011 16:57:45 +0100 X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=11320824c3=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <7F79B120F4ED415F8BB9EB7A4483AF8D@multiplay.co.uk> From: "Steven Hartland" To: "Andriy Gapon" , "Artem Belevich" , "David P Discher" References: <0EFD28CD-F2E9-4AE2-B927-1D327EC99DB9@bitgravity.com> <4DE50811.5060606@FreeBSD.org> Date: Tue, 31 May 2011 16:57:05 +0100 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6090 Cc: freebsd-fs@FreeBSD.org Subject: Re: ZFS: arc_reclaim_thread running 100%, 8.1-RELEASE, LBOLT related X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2011 16:09:12 -0000 ----- Original Message ----- > However, the arc_reclaim_thread does not have a ~24 day rollover - it > does not use clock_t. I think this rollover in the integer results > in LBOLT going negative, after about 106-107 days. We haven't noticed > this until actually 112-115 days of uptime. I think it is also related > to L1 ARC sizing, and load. Our systems with arc set to min-max of > 512M/2G ARC haven't developed the issue - at least the CPU hogging thread > - but the systems with 12G+ of ARC, and lots of rsync and du activity > along side of random reads from the zpool develop the issue. Looks like we had this on machine today which had only been up 66 days. A reboot cleared it, but 66 days up time is nearly half previously reported making it a bit more serious. last pid: 9562; load averages: 1.35, 2.12, 1.93 up 66+20:01:35 10:48:19 172 processes: 10 running, 143 sleeping, 19 waiting CPU: 0.0% user, 0.0% nice, 12.2% system, 0.0% interrupt, 87.8% idle Mem: 14M Active, 2812K Inact, 6394M Wired, 1596K Cache, 9448M Free Swap: 24G Total, 122M Used, 24G Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 8 171 ki31 0K 128K CPU0 0 ??? 704.35% idle 6 root 4 -8 - 0K 60K tx->tx 0 1031.9 100.00% zfskern 18 root 1 44 - 0K 16K syncer 0 167:04 0.10% syncer Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-RELEASE #1: Fri Mar 25 14:23:21 UTC 2011 root@... amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU X5355 @ 2.66GHz (2666.68-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x6f7 Family = 6 Model = f Stepping = 7 Features=0xbfebfbff Features2=0x4e3bd AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant real memory = 18253611008 (17408 MB) avail memory = 16509063168 (15744 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) > zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 9.06T 4.55T 4.51T 50% ONLINE - > zpool status pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 da0p3 ONLINE 0 0 0 cache ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 errors: No known data errors ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.