Date: Tue, 11 Aug 2015 16:42:27 +0200 From: Daniel Genis <daniel@byte.nl> To: freebsd-stable@freebsd.org Subject: Continuously increasing L2ARC header size Message-ID: <55CA09D3.5020007@byte.nl>
next in thread | raw e-mail | index | archive | help
Dear FreeBSD community, We're facing a somewhat odd issue, perhaps similar to what is discussed here: https://forums.freebsd.org/threads/l2arc-degraded.47540/ The issue is that the L2ARC header seems to grow without limit, similar to a memory leak, pressuring more and more memory over time out of the ARC. For example, the output of "zpool iostat -v 1" capacity operations bandwidth pool alloc free read write read write ------------ ----- ----- ----- ----- ----- ----- syspool 1.15G 275G 0 0 0 0 mirror 1.15G 275G 0 0 0 0 gpt/zfs0 - - 0 0 0 0 gpt/zfs1 - - 0 0 0 0 ------------ ----- ----- ----- ----- ----- ----- tank 1.21T 1.51T 229 1.99K 3.67M 9.48M mirror 124G 154G 67 125 787K 503K da0 - - 20 27 440K 503K da1 - - 45 28 379K 503K [...] mirror 124G 154G 34 164 454K 612K da18 - - 26 12 417K 612K da19 - - 6 13 58.8K 612K logs - - - - - - mirror 117M 74.4G 0 109 0 1.75M da21 - - 0 109 0 1.75M da22 - - 0 109 0 1.75M cache - - - - - - da23 1.67T 16.0E 302 7 2.85M 223K ------------ ----- ----- ----- ----- ----- ----- Here the cache shows 1.67T, in use and 16.0E free. The cache is a 373GB Intel SSD. # diskinfo -v da23 da23 512 # sectorsize 400088457216 # mediasize in bytes (373G) 781422768 # mediasize in sectors 4096 # stripesize 0 # stripeoffset 48641 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. BTTV4234089C400HGN # Disk ident. id1,enc@n500e004aaaaaaa3e/type@0/slot@18 # Physical path The L2ARC stats section from "zfs-stats -a": L2 ARC Summary: (DEGRADED) Passed Headroom: 133.33m Tried Lock Failures: 4.90b IO In Progress: 313.63k Low Memory Aborts: 1.52k Free on Write: 589.79k Writes While Full: 34.57k R/W Clashes: 46.95k Bad Checksums: 408.40m IO Errors: 151.99m SPA Mismatch: 632.00m L2 ARC Size: (Adaptive) 1.89 TiB Header Size: 0.88% 16.98 GiB L2 ARC Evicts: Lock Retries: 1.27k Upon Reading: 2 L2 ARC Breakdown: 2.10b Hit Ratio: 32.89% 691.15m Miss Ratio: 67.11% 1.41b Feeds: 3.70m L2 ARC Buffer: Bytes Scanned: 10.70 PiB Buffer Iterations: 3.70m List Iterations: 236.30m NULL List Iterations: 24.86m L2 ARC Writes: Writes Sent: 100.00% 3.38m Here we can see that currently the Header Size is almost 17gb. This header size grows continuously without (apparent) limit. Also zfs appears to think it's holding 1.89 TiB inside the L2ARC, which seems very very unlikely. # freebsd-version 10.1-RELEASE-p13 # uname -a FreeBSD servername 10.1-RELEASE-p10 FreeBSD 10.1-RELEASE-p10 #0: Wed May 13 06:54:13 UTC 2015 root@amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 # uptime 4:35PM up 42 days, 15:24, 1 user, load averages: 1.35, 0.96, 0.84 Does anyone know how we can alleviate the issue? We originally thought the issue was caused by https://www.freebsd.org/security/advisories/FreeBSD-EN-15:07.zfs.asc We have updated our Servers since but the header size seems to keep growing still. For reference, we have multiple bsd fileservers which are used mostly over NFS, all with identical configuration (but varying workload). They all still show these symptoms. Any tips/hints/pointers are appreciated! With kind regards, Daniel
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?55CA09D3.5020007>