Date: Tue, 31 May 2011 16:57:05 +0100 From: "Steven Hartland" <killing@multiplay.co.uk> To: "Andriy Gapon" <avg@FreeBSD.org>, "Artem Belevich" <art@FreeBSD.org>, "David P Discher" <dpd@bitgravity.com> Cc: freebsd-fs@FreeBSD.org Subject: Re: ZFS: arc_reclaim_thread running 100%, 8.1-RELEASE, LBOLT related Message-ID: <7F79B120F4ED415F8BB9EB7A4483AF8D@multiplay.co.uk> References: <0EFD28CD-F2E9-4AE2-B927-1D327EC99DB9@bitgravity.com><BANLkTikVq0-En7=4Dy_dTf=tM55Cqou_mw@mail.gmail.com> <4DE50811.5060606@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
----- Original Message -----
> However, the arc_reclaim_thread does not have a ~24 day rollover - it
> does not use clock_t. I think this rollover in the integer results
> in LBOLT going negative, after about 106-107 days. We haven't noticed
> this until actually 112-115 days of uptime. I think it is also related
> to L1 ARC sizing, and load. Our systems with arc set to min-max of
> 512M/2G ARC haven't developed the issue - at least the CPU hogging thread
> - but the systems with 12G+ of ARC, and lots of rsync and du activity
> along side of random reads from the zpool develop the issue.
Looks like we had this on machine today which had only been up 66 days.
A reboot cleared it, but 66 days up time is nearly half previously reported
making it a bit more serious.
last pid: 9562; load averages: 1.35, 2.12, 1.93 up 66+20:01:35 10:48:19
172 processes: 10 running, 143 sleeping, 19 waiting
CPU: 0.0% user, 0.0% nice, 12.2% system, 0.0% interrupt, 87.8% idle
Mem: 14M Active, 2812K Inact, 6394M Wired, 1596K Cache, 9448M Free
Swap: 24G Total, 122M Used, 24G Free
PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
11 root 8 171 ki31 0K 128K CPU0 0 ??? 704.35% idle
6 root 4 -8 - 0K 60K tx->tx 0 1031.9 100.00% zfskern
18 root 1 44 - 0K 16K syncer 0 167:04 0.10% syncer
Copyright (c) 1992-2011 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.2-RELEASE #1: Fri Mar 25 14:23:21 UTC 2011
root@... amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU X5355 @ 2.66GHz (2666.68-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x6f7 Family = 6 Model = f Stepping = 7
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0x4e3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
TSC: P-state invariant
real memory = 18253611008 (17408 MB)
avail memory = 16509063168 (15744 MB)
ACPI APIC Table: <PTLTD APIC >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s)
> zpool list
NAME SIZE USED AVAIL CAP HEALTH ALTROOT
tank 9.06T 4.55T 4.51T 50% ONLINE -
> zpool status
pool: tank
state: ONLINE
scrub: none requested
config:
NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
da0p3 ONLINE 0 0 0
cache
ada0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
errors: No known data errors
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?7F79B120F4ED415F8BB9EB7A4483AF8D>