Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 16 Jul 2013 16:16:09 +0300
From:      Daniel Kalchev <daniel@digsys.bg>
To:        freebsd-fs@freebsd.org
Subject:   Re: ZFS vdev I/O questions
Message-ID:  <51E54799.8070700@digsys.bg>
In-Reply-To: <20130716115305.GA40918@mwi1.coffeenet.org>
References:  <51E5316B.9070201@digsys.bg> <20130716115305.GA40918@mwi1.coffeenet.org>

next in thread | previous in thread | raw e-mail | index | archive | help

On 16.07.13 14:53, Mark Felder wrote:
> On Tue, Jul 16, 2013 at 02:41:31PM +0300, Daniel Kalchev wrote:
>> I am observing some "strange" behaviour with I/O spread on ZFS vdevs and
>> thought I might ask if someone has observed it too.
>>
> --SNIP--
>
>> Drives da0-da5 were Hitachi Deskstar 7K3000 (Hitachi HDS723030ALA640,
>> firmware MKAOA3B0) -- these are 512 byte sector drives, but da0 has been
>> replaced by Seagate Barracuda 7200.14 (AF) (ST3000DM001-1CH166, firmware
>> CC24) -- this is an 4k sector drive of a new generation (notice the
>> relatively 'old' firmware, that can't be upgraded).
> --SNIP--
>
>> The other observation I have is with the first vdev: the 512b drives do
>> a lot of I/O fast, complete first and then sit idle, while da0 continues
>> to write for many more seconds. They consistently show many more IOPS
>> than the other drives for this type of activity -- on streaming writes
>> all drives behave more or less the same. It is only on this un-dedup
>> scenario where the difference is so much pronounced.
>>
>> All the vdevs in the pool are with ashift=12 so the theory that ZFS
>> actually issues 512b writes to these drives can't be true, can it?
>>
>> Another worry is this Seagate Barracuda 7200.14 (AF)
>> (ST3000DM001-1CH166, firmware CC24) drive. It seems constantly
>> under-performing. Does anyone know if it is so different from the
>> ST3000DM001-9YN166 drives? Might be, I should just replace it?
>>
> A lot of information here.
>
> Those Hitachis are great drives. The addition of the Barracuda with
> different performance characteristics could be part of the problem. I'm
> glad you pointed out that the pool ashift=12 so we can try to rule that
> out. I'd be quite interested in knowing if some or perhaps even all of
> your issues go away simply by replacing that drive with another Hitachi.
> _______________________________________________
> freebsd-fs@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"

I wanted to commend further on this. The Hitachi drives are only in the 
first vdev (da0-da5) together with that new Seagate Barracuda drive. 
However, I observe very irregular writing to all the vdevs, not just 
within the same vdev.

Here is output of gstat -d with interval 1 second:

dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0     42     42    600    1.1      0      0    0.0      0 0    
0.0    2.5  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0     33     33    460    4.3      0      0    0.0      0 0    
0.0    3.5  da5
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da6
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da7
     0     30     30    656    2.1      0      0    0.0      0 0    
0.0    3.0  da8
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da9
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da10
     0     34     34    748    1.5      0      0    0.0      0 0    
0.0    2.4  da11
     0     43     43   1299    1.7      0      0    0.0      0 0    
0.0    4.2  da12
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da13
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da14
     0     41     41   1395    1.5      0      0    0.0      0 0    
0.0    3.3  da15
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da16
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0   1081      0      0    0.0   1081  14551    0.7      0 0    
0.0   10.3  da20
     0    124      0      0    0.0     97    286    0.5     25 273    
3.7    1.2  ada0
     0    119      0      0    0.0     92    286    0.4     25 273    
3.5    1.1  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
    24    501      0      0    0.0    501  18421   46.8      0 0    
0.0   98.8  da0
    24    690      0      0    0.0    690  34208   34.6      0 0    
0.0   99.8  da1
    24    691      0      0    0.0    691  33317   33.6      0 0    0.0  
100.2  da2
    24    750      0      0    0.0    750  37752   30.9      0 0    
0.0   99.9  da3
    24    672      0      0    0.0    672  32694   34.9      0 0    0.0  
100.1  da4
    24    722      0      0    0.0    722  36178   32.5      0 0    0.0  
100.0  da5
    24    633      0      0    0.0    633   9046   37.6      0 0    0.0  
100.1  da6
    24    601      0      0    0.0    601   8727   39.2      0 0    0.0  
100.0  da7
    24    620      0      0    0.0    620   9198   38.1      0 0    0.0  
100.0  da8
    24    619      0      0    0.0    619   8915   38.3      0 0    0.0  
100.3  da9
    24    539      0      0    0.0    539   7692   43.3      0 0    0.0  
100.0  da10
    24    715      0      0    0.0    715  10221   33.0      0 0    0.0  
100.5  da11
    24    584      0      0    0.0    584  44525   39.8      0 0    
0.0   99.4  da12
    24    543      0      0    0.0    543  41081   43.2      0 0    0.0  
100.6  da13
    24    523      0      0    0.0    523  40641   44.2      0 0    0.0  
100.0  da14
    24    521      0      0    0.0    521  40509   44.9      0 0    
0.0   99.9  da15
    24    505      0      0    0.0    505  40206   46.1      0 0    
0.0   99.8  da16
    24    524      0      0    0.0    524  40677   43.9      0 0    
0.0   99.9  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0   1082      0      0    0.0   1082   2941    0.2      0 0    
0.0    6.5  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.000s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
    24    507      0      0    0.0    507  30284   47.7      0 0    
0.0   99.9  da0
    24    625      0      0    0.0    625  36929   38.7      0 0    0.0  
100.6  da1
    24    724      0      0    0.0    724  44142   33.2      0 0    0.0  
100.5  da2
    24    775      0      0    0.0    775  53063   30.6      0 0    
0.0   98.2  da3
    24    630      0      0    0.0    630  40891   37.8      0 0    0.0  
100.2  da4
    24    698      0      0    0.0    698  38149   35.4      0 0    0.0  
102.5  da5
    24    784      0      0    0.0    784  11787   30.7      0 0    
0.0   99.9  da6
    24    707      0      0    0.0    707  10840   34.3      0 0    
0.0   99.1  da7
    24    689      0      0    0.0    689  10668   34.9      0 0    
0.0   99.6  da8
    24    635      0      0    0.0    635   9528   37.8      0 0    0.0  
100.1  da9
    24    669      0      0    0.0    669  10268   35.6      0 0    
0.0   99.7  da10
    24    675      0      0    0.0    675  10304   35.2      0 0    0.0  
100.3  da11
    24    507      0      0    0.0    507  23746   47.4      0 0    0.0  
100.0  da12
    24    476      0      0    0.0    476  24454   48.9      0 0    0.0  
100.0  da13
    24    495      0      0    0.0    495  31043   48.2      0 0    0.0  
100.8  da14
    24    582      0      0    0.0    582  34710   41.3      0 0    0.0  
100.1  da15
    24    592      0      0    0.0    592  34022   41.1      0 0    0.0  
100.4  da16
    24    559      0      0    0.0    559  34854   42.5      0 0    
0.0   99.6  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.000s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
    24    719      0      0    0.0    719  23063   33.0      0 0    
0.0   99.2  da0
     0     94      0      0    0.0     94   8274   43.6      0 0    
0.0   16.0  da1
     0     46      0      0    0.0     46   3839   37.0      0 0    
0.0    7.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0    135      0      0    0.0    135   8966   39.3      0 0    
0.0   21.9  da4
     0     11      0      0    0.0     11    896   38.8      0 0    
0.0    1.4  da5
    24    648      0      0    0.0    648   9070   36.6      0 0    
0.0   99.9  da6
    24    679      0      0    0.0    679   9750   35.7      0 0    0.0  
100.1  da7
    24    686      0      0    0.0    686   9922   35.0      0 0    
0.0   99.9  da8
    24    666      0      0    0.0    666   9654   35.8      0 0    0.0  
100.6  da9
    24    682      0      0    0.0    682   9450   35.1      0 0    0.0  
100.4  da10
    24    700      0      0    0.0    700   9346   34.1      0 0    0.0  
100.0  da11
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da12
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da13
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da14
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da15
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da16
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.000s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
    24    428      0      0    0.0    428   4207   55.6      0 0    0.0  
100.6  da6
    24    447      0      0    0.0    447   4279   52.9      0 0    0.0  
100.4  da7
    24    432      0      0    0.0    432   4087   55.6      0 0    0.0  
100.5  da8
    24    524      0      0    0.0    524   6243   45.3      0 0    
0.0   99.6  da9
    24    554      0      0    0.0    554   6379   43.2      0 0    0.0  
100.0  da10
    24    439      0      0    0.0    439   4611   54.0      0 0    
0.0   97.9  da11
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da12
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da13
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da14
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da15
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da16
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.000s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
    24    350      0      0    0.0    350   3263   69.9      0 0    0.0  
102.4  da6
    24    326      0      0    0.0    326   3611   74.2      0 0    0.0  
100.1  da7
    24    335      0      0    0.0    335   3367   72.4      0 0    0.0  
100.0  da8
    24    329      0      0    0.0    329   2943   73.5      0 0    0.0  
100.3  da9
    24    326      0      0    0.0    326   2883   75.1      0 0    
0.0   99.8  da10
    24    369      0      0    0.0    369   2995   65.2      0 0    0.0  
100.3  da11
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da12
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da13
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da14
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da15
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da16
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
    24    525      0      0    0.0    525   8507   45.3      0 0    0.0  
100.1  da6
    24    430      0      0    0.0    430   6761   55.6      0 0    0.0  
101.7  da7
    24    479      0      0    0.0    479   7548   50.1      0 0    0.0  
100.6  da8
    24    542      0      0    0.0    542   9463   44.3      0 0    0.0  
100.2  da9
    23    593      0      0    0.0    593  10386   40.6      0 0    0.0  
100.1  da10
    24    555      0      0    0.0    555   9678   42.9      0 0    
0.0   98.0  da11
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da12
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da13
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da14
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da15
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da16
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
    24    566      0      0    0.0    566   9800   41.3      0 0    
0.0   98.4  da6
    24    526      0      0    0.0    526  12370   46.5      0 0    
0.0   99.9  da7
    24    577      0      0    0.0    577  13166   41.5      0 0    
0.0   99.9  da8
    24    538      0      0    0.0    538  11990   44.7      0 0    
0.0   99.9  da9
    24    631      0      0    0.0    631  12666   37.8      0 0    
0.0   99.6  da10
    24    650      0      0    0.0    650  12894   36.4      0 0    0.0  
101.2  da11
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da12
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da13
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da14
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da15
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da16
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
    24    365      0      0    0.0    365   3604   66.3      0 0    0.0  
100.0  da6
    24    361      0      0    0.0    361   3724   65.8      0 0    0.0  
100.2  da7
    24    363      0      0    0.0    363   3680   65.4      0 0    
0.0   99.5  da8
    24    342      0      0    0.0    342   3500   69.4      0 0    0.0  
100.8  da9
    24    355      0      0    0.0    355   3460   70.1      0 0    0.0  
101.1  da10
    24    373      0      0    0.0    373   3616   65.0      0 0    
0.0   99.5  da11
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da12
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da13
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da14
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da15
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da16
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.000s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
    24    539      0      0    0.0    539   4947   44.2      0 0    
0.0   99.8  da6
    24    468      0      0    0.0    468  12565   52.0      0 0    0.0  
100.2  da7
    24    493      0      0    0.0    493  10950   49.5      0 0    0.0  
100.0  da8
    24    450      0      0    0.0    450  12665   52.8      0 0    0.0  
100.2  da9
    24    528      0      0    0.0    528  11070   45.7      0 0    0.0  
100.0  da10
    24    542      0      0    0.0    542  10750   43.9      0 0    
0.0   98.0  da11
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da12
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da13
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da14
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da15
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da16
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0     39      0      0    0.0     38   2583   14.6      0 0    
0.0    4.9  ada0
     0     39      0      0    0.0     38   2583   14.5      0 0    
0.0    4.9  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
    24    367      0      0    0.0    367   7972   65.1      0 0    0.0  
100.0  da6
    24    360      0      0    0.0    360   5055   67.4      0 0    0.0  
100.1  da7
    24    345      0      0    0.0    345   6233   69.4      0 0    0.0  
100.2  da8
    24    359      0      0    0.0    359   5191   65.9      0 0    0.0  
101.3  da9
    24    383      0      0    0.0    383   7452   62.2      0 0    0.0  
100.2  da10
    24    368      0      0    0.0    368   7528   64.7      0 0    0.0  
100.9  da11
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da12
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da13
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da14
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da15
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da16
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da1
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da2
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da3
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da4
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da5
     0    181      0      0    0.0    181   6534   68.1      0 0    
0.0   50.3  da6
     0    379      0      0    0.0    379  14071   57.0      0 0    
0.0   90.5  da7
     0    229      0      0    0.0    229   8680   65.1      0 0    
0.0   64.1  da8
    24    400      0      0    0.0    400  13871   60.5      0 0    
0.0   99.8  da9
     0    193      0      0    0.0    193   6874   57.7      0 0    
0.0   45.4  da10
     0    222      0      0    0.0    222   7753   68.2      0 0    
0.0   65.0  da11
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da12
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da13
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da14
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da15
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da16
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1
dT: 1.001s  w: 1.000s  filter: da[0-9]*$
  L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w    d/s kBps   
ms/d   %busy Name
     0    332    202  10093    8.8    130  11620   13.3      0 0    
0.0   51.2  da0
     0    253    121   4381    2.1    132  11620   10.9      0 0    
0.0   20.0  da1
     0    280    152   4976    4.6    128  11612   11.2      0 0    
0.0   28.9  da2
     0    376    247   9705    3.9    129  11612   11.3      0 0    
0.0   34.6  da3
     0    244    115   5540    5.4    129  11628   11.0      0 0    
0.0   27.7  da4
     0    268    138   5896    7.6    130  11620   10.9      0 0    
0.0   38.7  da5
     1    467    273  10988    7.9    194  12627   12.2      0 0    
0.0   60.4  da6
     0    349    147   6583    8.7    202  12659    9.4      0 0    
0.0   49.6  da7
     5    368    169   6803    7.0    199  12647   10.7      0 0    
0.0   48.7  da8
     0    451    253  11020    8.4    198  12651    9.7      0 0    
0.0   61.1  da9
     0    306    104   4493    6.2    202  12647   10.4      0 0    
0.0   31.7  da10
     0    350    151   4765    5.6    199  12627   10.4      0 0    
0.0   40.3  da11
     9    366    258  11652    6.9    108  12455   10.3      0 0    
0.0   47.1  da12
     0    302    194   8126    5.2    108  12455   13.2      0 0    
0.0   36.8  da13
     0    292    186   8162    3.1    106  12447   12.9      0 0    
0.0   30.0  da14
     0    370    264  12627    9.7    106  12447   13.3      0 0    
0.0   54.1  da15
     0    182     72   3110   10.2    110  12459    9.8      0 0    
0.0   28.1  da16
     0    171     62   3206    8.5    109  12455    9.8      0 0    
0.0   25.5  da17
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da18
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  da19
     4    744    216    394    0.4    529   1540    0.1      0 0    
0.0    4.0  da20
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada0
     0      0      0      0    0.0      0      0    0.0      0 0    
0.0    0.0  ada1

As you can see, the initial burst is to all vdevs, saturating drives at 
100%. Then vdev 3 completes, then the Hitachi drives of vdev 1 complete 
with the Seagate drive writing some more and then for few more seconds, 
only vdev 2 drives are writing. It seems the amount of data is the same, 
just vdev 2 writes the data slower. However, drives in vdev 2 and vdev 3 
are the same. They should have the same performance characteristics (and 
as long as the drives are not 100% saturated, all vdevs complete more or 
less at the same time). At other times, some other vdev would complete 
last -- it is never the same vdev that is 'slow'.

Could this be DDT/metadata specific issue? Is the DDT/metadata 
vdev-specific? The pool initially had only two vdevs and after vdev 3 
was added, most of the written data had no dedup enabled. Also, the ZIL 
was added later and initial metadata could be fragmented. But.. why 
should this affect writing? The zpool is indeed pretty full, but 
performance should degrade for all vdevs (which are more or less equally 
full).

Daniel



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?51E54799.8070700>