Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 16 Jul 2013 15:16:38 +0300
From:      Daniel Kalchev <daniel@digsys.bg>
To:        freebsd-fs <freebsd-fs@freebsd.org>
Subject:   Re: ZFS vdev I/O questions
Message-ID:  <51E539A6.9090109@digsys.bg>
In-Reply-To: <3472068604314C9887FE3BD4CD42B7C8@multiplay.co.uk>
References:  <51E5316B.9070201@digsys.bg> <3472068604314C9887FE3BD4CD42B7C8@multiplay.co.uk>

next in thread | previous in thread | raw e-mail | index | archive | help

On 16.07.13 14:53, Mark Felder wrote:
>> The other observation I have is with the first vdev: the 512b drives do
>> a lot of I/O fast, complete first and then sit idle, while da0 continues
>> to write for many more seconds. They consistently show many more IOPS
>> than the other drives for this type of activity -- on streaming writes
>> all drives behave more or less the same. It is only on this un-dedup
>> scenario where the difference is so much pronounced.
>>
>> All the vdevs in the pool are with ashift=12 so the theory that ZFS
>> actually issues 512b writes to these drives can't be true, can it?
>>
>> Another worry is this Seagate Barracuda 7200.14 (AF)
>> (ST3000DM001-1CH166, firmware CC24) drive. It seems constantly
>> under-performing. Does anyone know if it is so different from the
>> ST3000DM001-9YN166 drives? Might be, I should just replace it?
>>
> A lot of information here.
>
> Those Hitachis are great drives. The addition of the Barracuda with
> different performance characteristics could be part of the problem. I'm
> glad you pointed out that the pool ashift=12 so we can try to rule that
> out. I'd be quite interested in knowing if some or perhaps even all of
> your issues go away simply by replacing that drive with another Hitachi.

I don't have any more of these available and the vendor unfortunately 
could not supply more (but I am considering looking elsewhere)
At the moment, I could only replace that Barracuda with one of the spare 
drives, which are Seagate SV35 (ST3000VX000-9YW166, firmware CV13)
I have observed that these drives are not particularly good at random 
I/O however....


On 16.07.13 14:58, Steven Hartland wrote:
> One thing to check with this is to add -d to your gstat to see
> if your waiting deletes IO? I doubt it but worth checking.

Here is with gstat -d. Actually, running this script makes more sense:

while true ;do
gstat -f 'da[0-9]*$' -d -b
sleep 1
done


(small random reads, heavy writes, then some reads)
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0     11      1      4   20.8     10     44    0.4      0 0    
0.0    2.2  da0
     0     15      5     20    7.1     10     40    5.8      0 0    
0.0    5.1  da1
     0      7      0      0    0.0      7     28    0.4      0 0    
0.0    0.2  da2
     1      7      0      0    0.0      7     28    4.4      0 0    
0.0    1.1  da3
     1     11      0      0    0.0     11     44    0.5      0 0    
0.0    0.2  da4
     0     11      0      0    0.0     11     44    0.4      0 0    
0.0    0.2  da5
     0     25     16     68    7.8      9     36    0.4      0 0    
0.0    5.4  da6
     0     21     12     52   14.2      9     36    0.3      0 0    
0.0    6.0  da7
     0      9      0      0    0.0      9     36    0.3      0 0    
0.0    0.1  da8
     0     35     23    176   12.5     12     48    0.4      0 0    
0.0    9.4  da9
     0     26     17    140   13.7      9     36    0.3      0 0    
0.0    7.3  da10
     0      8      0      0    0.0      8     32    0.3      0 0    
0.0    0.1  da11
     0    158     29    212   17.4    129   4083   81.9      0 0    
0.0   57.8  da12
     0    162     25    192   16.7    137   4423   84.8      0 0    
0.0   61.9  da13
     0    118      0      0    0.0    118   3844   95.5      0 0    
0.0   45.6  da14
     5    152     20    112   10.8    132   4127   66.9      0 0    
0.0   45.8  da15
     0    171     14     88   17.4    157   4990   89.2      0 0    
0.0   69.6  da16
     0    124      1      4    6.6    123   3840   71.5      0 0    
0.0   37.6  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0   4558   4557   6736    0.5      1      1    0.2      0 0    
0.0   14.7  da20
     0    120      0      0    0.0    119   8083   35.2      0 0    
0.0   17.4  ada0
     0    120      0      0    0.0    119   8083   34.0      0 0    
0.0   16.8  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      1      1      4   14.0      0      0    0.0      0 0    
0.0    1.4  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      2      2      8   18.8      0      0    0.0      0 0    
0.0    3.7  da2
     0      1      1      4   35.1      0      0    0.0      0 0    
0.0    3.5  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     1      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
     0      3      3     12   12.8      0      0    0.0      0 0    
0.0    3.8  da6
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da7
     0      3      3     12   14.3      0      0    0.0      0 0    
0.0    4.3  da8
     0      1      1      4    9.8      0      0    0.0      0 0    
0.0    1.0  da9
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da10
     0      7      7     28    9.8      0      0    0.0      0 0    
0.0    6.8  da11
     0      1      1      4    7.4      0      0    0.0      0 0    
0.0    0.7  da12
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da13
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da14
     0      3      3     12   34.0      0      0    0.0      0 0    
0.0   10.2  da15
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da16
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0   2656    236    455    0.2   2420  12284    0.3      0 0    
0.0   21.3  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      4      4     16   22.8      0      0    0.0      0 0    
0.0    9.1  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     1      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      1      1      4   17.4      0      0    0.0      0 0    
0.0    1.7  da3
     0      3      3     12   13.2      0      0    0.0      0 0    
0.0    4.0  da4
     0      2      2      8   28.2      0      0    0.0      0 0    
0.0    5.6  da5
     0      1      1      4    9.7      0      0    0.0      0 0    
0.0    1.0  da6
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da7
     0      3      3     12    8.8      0      0    0.0      0 0    
0.0    2.6  da8
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da9
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da10
     0      5      5     20   10.1      0      0    0.0      0 0    
0.0    5.0  da11
     0      4      4     16   10.4      0      0    0.0      0 0    
0.0    4.1  da12
     0      1      1      4   16.0      0      0    0.0      0 0    
0.0    1.6  da13
     0      1      1      4   13.8      0      0    0.0      0 0    
0.0    1.4  da14
     0      4      4     16    7.3      0      0    0.0      0 0    
0.0    2.9  da15
     0      1      1      4   13.1      0      0    0.0      0 0    
0.0    1.3  da16
     0      1      1      4   21.3      0      0    0.0      0 0    
0.0    2.1  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0    515    515    979    0.2      0      0    0.0      0 0    
0.0    9.2  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      1      1      4   32.3      0      0    0.0      0 0    
0.0    3.2  da0
     0      1      1      4   35.0      0      0    0.0      0 0    
0.0    3.5  da1
     0      4      4     16    4.3      0      0    0.0      0 0    
0.0    1.7  da2
     0      3      3     12   31.8      0      0    0.0      0 0    
0.0    9.5  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      4      4     16    6.9      0      0    0.0      0 0    
0.0    2.8  da5
     0      2      2      8   13.6      0      0    0.0      0 0    
0.0    2.7  da6
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da7
     0      4      4     16    7.4      0      0    0.0      0 0    
0.0    3.0  da8
     0      1      1      4   12.6      0      0    0.0      0 0    
0.0    1.3  da9
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da10
     0      6      6     24   10.0      0      0    0.0      0 0    
0.0    6.0  da11
     0      1      1      4   21.6      0      0    0.0      0 0    
0.0    2.2  da12
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da13
     1      1      1      4   10.8      0      0    0.0      0 0    
0.0    1.1  da14
     0      1      1      4   18.3      0      0    0.0      0 0    
0.0    1.8  da15
     0      1      1      4    9.9      0      0    0.0      0 0    
0.0    1.0  da16
     0      1      1      4   21.6      0      0    0.0      0 0    
0.0    2.2  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0    588    587   1130    0.2      1      2    0.6      0 0    
0.0   10.5  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
    23   1412      0      0    0.0   1412  24423   15.0      0 0    
0.0   99.6  da0
    23   4699      0      0    0.0   4699  72429    4.1      0 0    
0.0   97.2  da1
    24   4294      0      0    0.0   4294  67749    4.6      0 0    
0.0   99.0  da2
    23   4923      0      0    0.0   4923  75746    3.8      0 0    
0.0   96.2  da3
    24   4452      0      0    0.0   4452  67729    4.3      0 0    
0.0   97.5  da4
    23   4558      0      0    0.0   4558  68992    4.2      0 0    
0.0   97.3  da5
    24    813      0      0    0.0    813  10675   27.2      0 0    
0.0   99.6  da6
    24    866      0      0    0.0    866  11338   25.5      0 0    
0.0   99.5  da7
    24    845      0      0    0.0    845  11058   25.8      0 0    
0.0   99.5  da8
    24    866      0      0    0.0    866  11174   25.3      0 0    
0.0   98.8  da9
    24    928      0      0    0.0    928  12061   23.6      0 0    
0.0   99.2  da10
    24    864      0      0    0.0    864  11150   25.6      0 0    
0.0   99.2  da11
    24    674      0      0    0.0    674  14587   32.4      0 0    
0.0   99.9  da12
    24    705      0      0    0.0    705  15043   31.7      0 0    0.0  
100.1  da13
    24    670      0      0    0.0    670  14527   33.0      0 0    
0.0   99.5  da14
    24    648      0      0    0.0    648  13360   34.3      0 0    
0.0   99.5  da15
    24    716      0      0    0.0    716  14835   30.9      0 0    
0.0   99.5  da16
    24    730      0      0    0.0    730  15227   30.0      0 0    
0.0   98.7  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
    24   1169      0      0    0.0   1169  15954   20.6      0 0    0.0  
100.3  da0
     0   1811      0      0    0.0   1811  20961    3.9      0 0    
0.0   29.6  da1
     0   2392      0      0    0.0   2392  27444    3.7      0 0    
0.0   37.1  da2
     0   1199      0      0    0.0   1199  14571    3.7      0 0    
0.0   18.5  da3
     0   2305      0      0    0.0   2305  26924    4.0      0 0    
0.0   38.1  da4
     0   1820      0      0    0.0   1820  21573    3.8      0 0    
0.0   28.8  da5
    24    744      0      0    0.0    744   9248   32.6      0 0    
0.0   99.5  da6
    24    771      0      0    0.0    771  10722   31.1      0 0    0.0  
100.0  da7
    24    716      0      0    0.0    716   9268   33.5      0 0    0.0  
100.2  da8
    24    754      0      0    0.0    754   9280   31.8      0 0    
0.0   99.9  da9
    24    769      0      0    0.0    769   9384   31.4      0 0    
0.0   99.9  da10
    24    742      0      0    0.0    742   9508   32.1      0 0    
0.0   98.5  da11
    24    665      0      0    0.0    665   9967   36.5      0 0    
0.0   99.3  da12
    24    740      0      0    0.0    740  10958   32.2      0 0    
0.0   98.7  da13
    24    639      0      0    0.0    639   9619   37.4      0 0    
0.0   99.7  da14
    24    695      0      0    0.0    695  10367   35.1      0 0    
0.0   99.8  da15
    24    583      0      0    0.0    583   8429   41.5      0 0    0.0  
100.2  da16
    24    697      0      0    0.0    697  10287   34.3      0 0    0.0  
100.0  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
    24    895      0      0    0.0    895  14773   26.9      0 0    0.0  
101.6  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
    24    702      0      0    0.0    702   9327   33.8      0 0    
0.0   97.6  da6
    24    675      0      0    0.0    675   9191   35.3      0 0    0.0  
100.1  da7
    24    668      0      0    0.0    668   8575   36.1      0 0    0.0  
100.7  da8
    24    744      0      0    0.0    744   9179   31.4      0 0    
0.0   97.1  da9
    24    750      0      0    0.0    750   9003   32.3      0 0    0.0  
100.1  da10
    24    586      0      0    0.0    586   6909   40.9      0 0    0.0  
100.0  da11
    24    711      0      0    0.0    711  13990   33.6      0 0    
0.0   99.4  da12
    24    653      0      0    0.0    653  10074   36.5      0 0    0.0  
100.0  da13
    24    645      0      0    0.0    645  13111   37.0      0 0    0.0  
100.0  da14
    24    687      0      0    0.0    687  12795   34.8      0 0    
0.0   99.9  da15
    24    678      0      0    0.0    678  12855   35.6      0 0    0.0  
100.6  da16
    24    696      0      0    0.0    696  13091   34.4      0 0    
0.0   99.2  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
    24   1177      0      0    0.0   1177  18586   20.6      0 0    0.0  
100.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
    24    780      0      0    0.0    780  11733   30.6      0 0    
0.0   99.3  da6
    23    776      0      0    0.0    776  12636   31.1      0 0    0.0  
100.2  da7
    24    636      0      0    0.0    636   9387   37.8      0 0    
0.0   99.7  da8
    24    778      0      0    0.0    778  13343   31.2      0 0    0.0  
100.6  da9
    24    821      0      0    0.0    821  14202   29.3      0 0    0.0  
101.0  da10
    24    786      0      0    0.0    786  12024   30.6      0 0    0.0  
100.0  da11
    24    666      0      0    0.0    666   9351   35.9      0 0    0.0  
100.0  da12
    24    720      0      0    0.0    720   9954   33.1      0 0    
0.0   99.7  da13
    24    706      0      0    0.0    706  10014   34.0      0 0    
0.0   99.8  da14
    24    801      0      0    0.0    801  11105   30.3      0 0    0.0  
101.1  da15
    24    738      0      0    0.0    738  10126   32.5      0 0    
0.0   99.9  da16
    24    670      0      0    0.0    670   9203   36.5      0 0    0.0  
101.4  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
    23   1650      0      0    0.0   1650  18674   14.4      0 0    0.0  
100.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
    24    669      0      0    0.0    669  13234   35.9      0 0    0.0  
100.1  da6
    24    573      0      0    0.0    573  12223   42.4      0 0    
0.0   99.7  da7
    23    519      0      0    0.0    519  10584   46.3      0 0    0.0  
100.1  da8
    24    659      0      0    0.0    659  15552   36.9      0 0    0.0  
100.5  da9
    24    687      0      0    0.0    687  18410   35.1      0 0    
0.0   99.3  da10
    23    671      0      0    0.0    671  13190   36.0      0 0    0.0  
100.2  da11
    24    807      0      0    0.0    807   9425   29.7      0 0    
0.0   99.9  da12
    24    665      0      0    0.0    665   7902   36.4      0 0    0.0  
100.3  da13
    24    861      0      0    0.0    861  10140   28.0      0 0    
0.0   99.6  da14
    24    714      0      0    0.0    714   8797   33.4      0 0    
0.0   99.7  da15
    24    753      0      0    0.0    753   9225   32.5      0 0    0.0  
100.0  da16
    24    760      0      0    0.0    760   9337   31.5      0 0    0.0  
100.0  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
    24    648      0      0    0.0    648   9954   37.0      0 0    0.0  
100.0  da6
    24    664      0      0    0.0    664  10194   36.1      0 0    0.0  
100.0  da7
    24    653      0      0    0.0    653  13163   36.7      0 0    
0.0   99.9  da8
    24    749      0      0    0.0    749  12716   31.8      0 0    0.0  
100.4  da9
    24    769      0      0    0.0    769  13239   31.4      0 0    0.0  
100.3  da10
    24    742      0      0    0.0    742  12017   32.4      0 0    0.0  
100.1  da11
    24    584      0      0    0.0    584   7393   40.8      0 0    0.0  
101.1  da12
    24    648      0      0    0.0    648   8336   37.7      0 0    
0.0   99.7  da13
    24    679      0      0    0.0    679   8756   35.3      0 0    0.0  
100.8  da14
    24    646      0      0    0.0    646   8160   37.0      0 0    0.0  
100.5  da15
    24    692      0      0    0.0    692   8900   35.5      0 0    
0.0   99.9  da16
    24    684      0      0    0.0    684   8692   35.6      0 0    0.0  
101.7  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
    24    662      0      0    0.0    662  14641   35.8      0 0    0.0  
100.0  da6
    24    705      0      0    0.0    705  16535   34.2      0 0    
0.0   99.1  da7
    24    644      0      0    0.0    644  10120   37.4      0 0    0.0  
101.8  da8
    24    560      0      0    0.0    560  14721   43.4      0 0    
0.0   99.5  da9
    24    677      0      0    0.0    677  20864   35.2      0 0    0.0  
100.0  da10
    24    685      0      0    0.0    685  19009   35.6      0 0    0.0  
100.3  da11
    24    714      0      0    0.0    714  10328   34.0      0 0    0.0  
101.7  da12
    24    738      0      0    0.0    738  10476   32.2      0 0    0.0  
100.0  da13
    24    587      0      0    0.0    587   8465   40.6      0 0    0.0  
100.1  da14
    24    720      0      0    0.0    720   9888   33.8      0 0    0.0  
100.1  da15
    24    603      0      0    0.0    603   8258   39.1      0 0    0.0  
100.1  da16
    24    623      0      0    0.0    623   9025   38.3      0 0    
0.0   99.9  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
     0     76      0      0    0.0     76   1702   39.6      0 0    
0.0   10.9  da6
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da7
     0    298      0      0    0.0    298   8704   33.8      0 0    
0.0   40.5  da8
     0    112      0      0    0.0    112   3065   48.0      0 0    
0.0   22.1  da9
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da10
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da11
    24    645      0      0    0.0    645   9942   37.3      0 0    0.0  
100.9  da12
    24    581      0      0    0.0    581   8660   41.1      0 0    
0.0   99.8  da13
    24    717      0      0    0.0    717  10910   33.6      0 0    0.0  
101.4  da14
    24    737      0      0    0.0    737  11737   32.6      0 0    
0.0   99.9  da15
    24    556      0      0    0.0    556   8796   42.3      0 0    
0.0   99.5  da16
    24    538      0      0    0.0    538   8444   44.1      0 0    0.0  
100.1  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da6
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da7
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da8
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da9
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da10
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da11
     0    642      0      0    0.0    642   8875   37.2      0 0    0.0  
100.0  da12
    24    672      0      0    0.0    672   9323   36.0      0 0    0.0  
100.0  da13
    24    765      0      0    0.0    765  10602   31.5      0 0    0.0  
101.6  da14
     0    684      0      0    0.0    684   9902   31.0      0 0    
0.0   87.1  da15
    23    683      0      0    0.0    683  10210   35.6      0 0    0.0  
100.4  da16
    24    654      0      0    0.0    654   9603   37.1      0 0    
0.0   99.9  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.000s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0    114    109   3811    1.9      4     16    0.5      0 0    
0.0   12.6  da0
     0    349    343  12129    0.7      4     16    0.5      0 0    
0.0   24.5  da1
     0    232    226   8322    1.0      4     16    0.5      0 0    
0.0   23.6  da2
     0    230    224   8338    2.7      4     16    0.5      0 0    
0.0   28.0  da3
     0    349    343  12129    0.7      4     16    0.5      0 0    
0.0   23.9  da4
     0    122    116   3775    1.1      4     16    0.5      0 0    
0.0   21.5  da5
     0      5      0      0    0.0      4     16    0.4      0 0    
0.0    4.8  da6
     0      5      0      0    0.0      4     16    0.3      0 0    
0.0    4.2  da7
     0    356    351  12577    0.8      4     16    0.4      0 0    
0.0   14.4  da8
     0    349    344  12573    0.7      4     16    0.3      0 0    
0.0   15.5  da9
     0    359    354  12573    0.6      4     16    0.4      0 0    
0.0    9.5  da10
     0    360    355  12589    1.0      4     16    0.3      0 0    
0.0   14.0  da11
     0      6      0      0    0.0      4     16    0.3      0 0    
0.0   16.9  da12
     0      5      0      0    0.0      4     16    0.3      0 0    
0.0    3.6  da13
     0      5      0      0    0.0      4     16    0.3      0 0    
0.0    5.3  da14
     0      5      0      0    0.0      4     16    0.3      0 0    
0.0    3.8  da15
     0      5      0      0    0.0      4     16    0.3      0 0    
0.0    4.9  da16
     0      9      4     16   15.7      4     16    0.4      0 0    
0.0    5.8  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      1      0      0    0.0      0      0    0.0      1 3839    
0.4    0.0  ada0
     0      2      0      0    0.0      0      0    0.0      2 3839    
0.5    0.0  ada1
^C

There were some heavy writes to da20 (the L2ARC) but no delete requests. 
While deletes on ada0 and ada1 do happen as expected. The L2ARC should 
delete, as it is nearly full.

zpool iostat -v storage 10
                    capacity     operations    bandwidth
pool            alloc   free   read  write   read  write
--------------  -----  -----  -----  -----  -----  -----
storage         44.8T  3.95T     77  9.34K  5.03M  43.1M
   raidz2        15.1T  1.13T     27  3.09K  2.37M  14.1M
     gpt/disk00      -      -      8    485   303K  6.66M
     gpt/disk01      -      -     16    477   603K  6.56M
     gpt/disk02      -      -      9    476   320K  6.56M
     gpt/disk03      -      -      8    476   322K  6.56M
     gpt/disk04      -      -     16    477   603K  6.56M
     gpt/disk05      -      -      8    476   301K  6.55M
   raidz2        14.7T  1.53T     32  3.13K  2.57M  14.5M
     gpt/disk06      -      -     14    444   502K  6.74M
     gpt/disk07      -      -     11    444   385K  6.74M
     gpt/disk08      -      -     14    444   552K  6.74M
     gpt/disk09      -      -     15    445   582K  6.73M
     gpt/disk10      -      -      7    444   280K  6.72M
     gpt/disk11      -      -      9    444   369K  6.72M
   raidz2        15.0T  1.29T     17  3.10K  87.0K  14.1M
     gpt/disk12      -      -      4    434  40.8K  6.65M
     gpt/disk13      -      -      2    434  16.3K  6.65M
     gpt/disk14      -      -      0    433  8.75K  6.62M
     gpt/disk15      -      -      4    435  40.6K  6.66M
     gpt/disk16      -      -      2    433  15.3K  6.61M
     gpt/disk17      -      -      0    434  8.54K  6.65M
logs                -      -      -      -      -      -
   ada0p2        1.01M  7.94G      0      2      0   185K
   ada1p2        1.01M  7.94G      0      2      0   185K
cache               -      -      -      -      -      -
   da20p2         215G    80M    439    674   703K  2.84M
--------------  -----  -----  -----  -----  -----  -----


Daniel



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?51E539A6.9090109>