From owner-freebsd-questions@FreeBSD.ORG Wed Mar 26 15:52:47 2014 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id E964070C for ; Wed, 26 Mar 2014 15:52:47 +0000 (UTC) Received: from mail-ob0-x230.google.com (mail-ob0-x230.google.com [IPv6:2607:f8b0:4003:c01::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id B23CBF42 for ; Wed, 26 Mar 2014 15:52:47 +0000 (UTC) Received: by mail-ob0-f176.google.com with SMTP id wp18so2726319obc.35 for ; Wed, 26 Mar 2014 08:52:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=khera.org; s=google11; h=mime-version:date:message-id:subject:from:to:content-type; bh=rPYCbGOFRTppTklpN5seOhvLGtQPTGZh41jM2Js6S+w=; b=ZfCQDeAv44sGWYoOURSF/n9rfviseRY3kgDt6LlufMdQozGj4EPoa4ymVG27MN3VeJ fyQoxAikHCxbUQFEYjPcL+D81EsmYIxvgos+rSZC3idwJM4RP+t1Kk2jtD7mFejBtWVn zMQeUJOChsgpvxDi9FBhvgk10fVGpGyd9UbMg= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=rPYCbGOFRTppTklpN5seOhvLGtQPTGZh41jM2Js6S+w=; b=MCmMVFNUn3Aqub65cNeqlc6maj1cmRjFMUYBzt07MzgwUsotTToBrsSmp4xMCbRoOx 6fTe5NKDC9QqK/wZJOR5qrJZYeWjz7ZyO+GrlmfFeMeCk5TgwDgSskvDfV8z6hcaQ41k Jd6M45SFESxM2jNu86xwjKCts4Zn0jWkUA8FRF4RfflS0ljuHvZzjo1P+UdvORCzEeVk 4UZOQs80B/HYNaCTlkR1kg6w43IY1wIHFko9WYyL7bnEFJNCotwPKCDRFj+HrGIxzLV+ DIAxGgQg63Lx9t45actINWF8gklFCc3V88WTAT7Kl+tS5T9PKbujVmawzdGYYuHItdVL Hr+w== X-Gm-Message-State: ALoCoQkUhWlg8jLxNIvF5vhn0OkAu43xjjhP0+qme2XH2n3s9UYyXEzyY+hckPl7I2geAcpB5UBc MIME-Version: 1.0 X-Received: by 10.182.40.37 with SMTP id u5mr18107977obk.41.1395849167010; Wed, 26 Mar 2014 08:52:47 -0700 (PDT) Received: by 10.76.151.134 with HTTP; Wed, 26 Mar 2014 08:52:46 -0700 (PDT) Date: Wed, 26 Mar 2014 11:52:46 -0400 Message-ID: Subject: ZFS l2arc stats not updating From: Vick Khera To: freebsd-questions@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 26 Mar 2014 15:52:48 -0000 I have two identical FreeBSD 10.0/amd64 systems I recently put into production running ZFS. They have 256GB of RAM each. I installed the zfs-stats utility to monitor their stats. The zfs-mon program on one of the boxes does not report any L2ARC stats, while the other one does. Checking the stats via sysctl shows that kstat.zfs.misc.arcstats.l2_misses and kstat.zfs.misc.arcstats.l2_hits are not updating on the host for which zfs-mon shows no L2 data. They are both configured identically via script-generated files. The ZFS pool was created identically as well. The hardware was manufactured at the same time. I can find nothing in sysctl output that would indicate that the L2ARC is somehow disabled, and it does indeed seem to be getting updated by the system when watching iostat output. Any clues where to look to see why my L2ARC stats are not updating on just the one host? Both are doing a fair amount of read/write activity. Host A (the one not reporting L2 stats): % zpool iostat -v capacity operations bandwidth pool alloc free read write read write ---------------- ----- ----- ----- ----- ----- ----- zfspool 57.6G 1.03T 7 246 112K 1.99M mirror 28.8G 527G 3 123 56.0K 1018K label/zdisk0 - - 0 59 31.0K 1019K label/zdisk1 - - 0 59 31.0K 1019K mirror 28.7G 527G 3 123 56.0K 1018K label/zdisk2 - - 0 57 30.9K 1019K label/zdisk3 - - 0 57 31.1K 1019K cache - - - - - - label/zcache0 93.1G 16.0M 0 8 0 689K label/zcache1 93.1G 16M 0 8 0 688K ---------------- ----- ----- ----- ----- ----- ----- % zfs-mon -ALZ -b ZFS real-time cache activity monitor Cache efficiency percentage: 10s 60s tot ARC: 100.00 100.00 100.00 ZFETCH: 48.44 48.44 48.44 % sysctl kstat.zfs.misc.arcstats.l2_misses kstat.zfs.misc.arcstats.l2_misses: 4073823 % sysctl kstat.zfs.misc.arcstats.l2_hits kstat.zfs.misc.arcstats.l2_hits: 0 these above values are not changing. However, watching the zpool iostat values, the cache drives do get read/write activity and the capacity numbers change. Host B: % zpool iostat -v capacity operations bandwidth pool alloc free read write read write ---------------- ----- ----- ----- ----- ----- ----- zfspool 130G 982G 101 313 1.23M 2.83M mirror 64.8G 491G 50 156 632K 1.41M label/zdisk0 - - 14 45 431K 1.41M label/zdisk1 - - 14 45 431K 1.41M mirror 64.8G 491G 50 156 632K 1.41M label/zdisk2 - - 14 45 432K 1.41M label/zdisk3 - - 14 45 433K 1.41M cache - - - - - - label/zcache0 93.1G 8M 1 4 18.4K 264K label/zcache1 93.1G 7.98M 1 4 18.3K 265K ---------------- ----- ----- ----- ----- ----- ----- % zfs-mon -ALZ -b ZFS real-time cache activity monitor Cache efficiency percentage: 10s 60s tot ARC: 100.00 100.00 100.00 L2ARC: 0.00 0.00 0.00 ZFETCH: 32.88 32.88 32.88 % sysctl kstat.zfs.misc.arcstats.l2_misses kstat.zfs.misc.arcstats.l2_misses: 42635721 % sysctl kstat.zfs.misc.arcstats.l2_hits kstat.zfs.misc.arcstats.l2_hits: 1605684