Date: Fri, 1 Oct 2010 14:56:41 -0700 From: Artem Belevich <fbsdlist@src.cx> To: Dan Langille <dan@langille.org> Cc: freebsd-stable@freebsd.org Subject: Re: zfs send/receive: is this slow? Message-ID: <AANLkTik0aTDDSNRUBvfX5sMfhW%2B-nfSV9Q89v%2BeJo0ov@mail.gmail.com> In-Reply-To: <45cfd27021fb93f9b0877a1596089776.squirrel@nyi.unixathome.org> References: <a263c3beaeb0fa3acd82650775e31ee3.squirrel@nyi.unixathome.org> <45cfd27021fb93f9b0877a1596089776.squirrel@nyi.unixathome.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hmm. It did help me a lot when I was replicating ~2TB worth of data over GigE. Without mbuffer things were roughly in the ballpark of your numbers. With mbuffer I've got around 100MB/s. Assuming that you have two boxes connected via ethernet, it would be good to check that nobody generates PAUSE frames. Some time back I've discovered that el-cheapo switch I've been using for some reason could not keep up with traffic bursts and generated tons of PAUSE frames that severely limited throughput. If you're using Intel adapters, check xon/xoff counters in "sysctl dev.em.0.mac_stats". If you see them increasing, that may explain slow speed. If you have a switch between your boxes, try bypassing it and connect boxes directly. --Artem On Fri, Oct 1, 2010 at 11:51 AM, Dan Langille <dan@langille.org> wrote: > > On Wed, September 29, 2010 2:04 pm, Dan Langille wrote: >> $ zpool iostat 10 >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0capacity =A0 =A0 operations =A0 =A0bandwi= dth >> pool =A0 =A0 =A0 =A0 used =A0avail =A0 read =A0write =A0 read =A0write >> ---------- =A0----- =A0----- =A0----- =A0----- =A0----- =A0----- >> storage =A0 =A0 7.67T =A05.02T =A0 =A0358 =A0 =A0 38 =A043.1M =A01.96M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0317 =A0 =A0475 =A039.4M =A030.9M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0357 =A0 =A0533 =A044.3M =A034.4M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0371 =A0 =A0556 =A046.0M =A035.8M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0313 =A0 =A0521 =A038.9M =A028.7M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0309 =A0 =A0457 =A038.4M =A030.4M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0388 =A0 =A0589 =A048.2M =A037.8M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0377 =A0 =A0581 =A046.8M =A036.5M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0310 =A0 =A0559 =A038.4M =A030.4M >> storage =A0 =A0 7.67T =A05.02T =A0 =A0430 =A0 =A0611 =A053.4M =A041.3M > > Now that I'm using mbuffer: > > $ zpool iostat 10 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 capacity =A0 =A0 operations =A0 =A0bandwidth > pool =A0 =A0 =A0 =A0 used =A0avail =A0 read =A0write =A0 read =A0write > ---------- =A0----- =A0----- =A0----- =A0----- =A0----- =A0----- > storage =A0 =A0 9.96T =A02.73T =A02.01K =A0 =A0131 =A0 151M =A06.72M > storage =A0 =A0 9.96T =A02.73T =A0 =A0615 =A0 =A0515 =A076.3M =A033.5M > storage =A0 =A0 9.96T =A02.73T =A0 =A0360 =A0 =A0492 =A044.7M =A033.7M > storage =A0 =A0 9.96T =A02.73T =A0 =A0388 =A0 =A0554 =A048.3M =A038.4M > storage =A0 =A0 9.96T =A02.73T =A0 =A0403 =A0 =A0562 =A050.1M =A039.6M > storage =A0 =A0 9.96T =A02.73T =A0 =A0313 =A0 =A0468 =A038.9M =A028.0M > storage =A0 =A0 9.96T =A02.73T =A0 =A0462 =A0 =A0677 =A057.3M =A022.4M > storage =A0 =A0 9.96T =A02.73T =A0 =A0383 =A0 =A0581 =A047.5M =A021.6M > storage =A0 =A0 9.96T =A02.72T =A0 =A0142 =A0 =A0571 =A017.7M =A015.4M > storage =A0 =A0 9.96T =A02.72T =A0 =A0 80 =A0 =A0598 =A010.0M =A018.8M > storage =A0 =A0 9.96T =A02.72T =A0 =A0718 =A0 =A0503 =A089.1M =A013.6M > storage =A0 =A0 9.96T =A02.72T =A0 =A0594 =A0 =A0517 =A073.8M =A014.1M > storage =A0 =A0 9.96T =A02.72T =A0 =A0367 =A0 =A0528 =A045.6M =A015.1M > storage =A0 =A0 9.96T =A02.72T =A0 =A0338 =A0 =A0520 =A041.9M =A016.4M > storage =A0 =A0 9.96T =A02.72T =A0 =A0348 =A0 =A0499 =A043.3M =A021.5M > storage =A0 =A0 9.96T =A02.72T =A0 =A0398 =A0 =A0553 =A049.4M =A014.4M > storage =A0 =A0 9.96T =A02.72T =A0 =A0346 =A0 =A0481 =A043.0M =A06.78M > > If anything, it's slower. > > The above was without -s 128. =A0The following used that setting: > > =A0$ zpool iostat 10 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 capacity =A0 =A0 operations =A0 =A0bandwidth > pool =A0 =A0 =A0 =A0 used =A0avail =A0 read =A0write =A0 read =A0write > ---------- =A0----- =A0----- =A0----- =A0----- =A0----- =A0----- > storage =A0 =A0 9.78T =A02.91T =A01.98K =A0 =A0137 =A0 149M =A06.92M > storage =A0 =A0 9.78T =A02.91T =A0 =A0761 =A0 =A0577 =A094.4M =A042.6M > storage =A0 =A0 9.78T =A02.91T =A0 =A0462 =A0 =A0411 =A057.4M =A024.6M > storage =A0 =A0 9.78T =A02.91T =A0 =A0492 =A0 =A0497 =A061.1M =A027.6M > storage =A0 =A0 9.78T =A02.91T =A0 =A0632 =A0 =A0446 =A078.5M =A022.5M > storage =A0 =A0 9.78T =A02.91T =A0 =A0554 =A0 =A0414 =A068.7M =A021.8M > storage =A0 =A0 9.78T =A02.91T =A0 =A0459 =A0 =A0434 =A057.0M =A031.4M > storage =A0 =A0 9.78T =A02.91T =A0 =A0398 =A0 =A0570 =A049.4M =A032.7M > storage =A0 =A0 9.78T =A02.91T =A0 =A0338 =A0 =A0495 =A041.9M =A026.5M > storage =A0 =A0 9.78T =A02.91T =A0 =A0358 =A0 =A0526 =A044.5M =A033.3M > storage =A0 =A0 9.78T =A02.91T =A0 =A0385 =A0 =A0555 =A047.8M =A039.8M > storage =A0 =A0 9.78T =A02.91T =A0 =A0271 =A0 =A0453 =A033.6M =A023.3M > storage =A0 =A0 9.78T =A02.91T =A0 =A0270 =A0 =A0456 =A033.5M =A028.8M > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTik0aTDDSNRUBvfX5sMfhW%2B-nfSV9Q89v%2BeJo0ov>