From owner-freebsd-stable@FreeBSD.ORG Mon Oct 4 02:06:21 2010 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7DE90106566C for ; Mon, 4 Oct 2010 02:06:21 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-qw0-f54.google.com (mail-qw0-f54.google.com [209.85.216.54]) by mx1.freebsd.org (Postfix) with ESMTP id 2F3C68FC14 for ; Mon, 4 Oct 2010 02:06:20 +0000 (UTC) Received: by qwd6 with SMTP id 6so3390217qwd.13 for ; Sun, 03 Oct 2010 19:06:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:sender:received :in-reply-to:references:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=JUumkBnMyRUv2GiNO62HWu+KH5js2PmzNWgbKUiq6IY=; b=wgqIvdyu2CRMmzCpJSOOr91T1kQZXmsl9ONCCx2PX06/oq8CFUYJC/HxbknlLijwHE IzlY1kY1r1Nuv00UdXATNdoOqxSSCejsfRv76t1hD04AzZDkyxBxi/fEs97BvoSTmnc9 sAP0mkVLDfOVlVc1mqWIFTzCu4GMQWvl6eZHc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=uLxA6MRtHBT+PyBqTyNN1FtI5jtwOuf/ZAMZf6ICQcbzamXYv7edWzMRdhGgAs/tPg Vbjmpt2oV+axXZPMlVaJWLwMshosrHzgfFoBB2A2Uc0mO9yjLmYuLgnnFdKK6zax/UCB qduOD3tiaJogNGURhe1v2DEr+HtvBoY3NnDhI= MIME-Version: 1.0 Received: by 10.220.63.5 with SMTP id z5mr2256722vch.245.1286157980258; Sun, 03 Oct 2010 19:06:20 -0700 (PDT) Sender: artemb@gmail.com Received: by 10.220.176.77 with HTTP; Sun, 3 Oct 2010 19:06:20 -0700 (PDT) In-Reply-To: <4CA929A8.6000708@langille.org> References: <45cfd27021fb93f9b0877a1596089776.squirrel@nyi.unixathome.org> <4C511EF8-591C-4BB9-B7AA-30D5C3DDC0FF@langille.org> <4CA68BBD.6060601@langille.org> <4CA929A8.6000708@langille.org> Date: Sun, 3 Oct 2010 19:06:20 -0700 X-Google-Sender-Auth: nI87NwFUIWOQtNXWo4pPd8KuMdY Message-ID: From: Artem Belevich To: Dan Langille Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable Subject: Re: zfs send/receive: is this slow? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2010 02:06:21 -0000 On Sun, Oct 3, 2010 at 6:11 PM, Dan Langille wrote: > I'm rerunning my test after I had a drive go offline[1]. =A0But I'm not > getting anything like the previous test: > > time zfs send storage/bacula@transfer | mbuffer | zfs receive > storage/compressed/bacula-buffer > > $ zpool iostat 10 10 > =A0 =A0 =A0 =A0 =A0 =A0 =A0 capacity =A0 =A0 operations =A0 =A0bandwidth > pool =A0 =A0 =A0 =A0 used =A0avail =A0 read =A0write =A0 read =A0write > ---------- =A0----- =A0----- =A0----- =A0----- =A0----- =A0----- > storage =A0 =A0 6.83T =A05.86T =A0 =A0 =A08 =A0 =A0 31 =A01.00M =A02.11M > storage =A0 =A0 6.83T =A05.86T =A0 =A0207 =A0 =A0481 =A025.7M =A017.8M It may be worth checking individual disk activity using gstat -f 'da.$' Some time back I had one drive that was noticeably slower than the rest of the drives in RAID-Z2 vdev and was holding everything back. SMART looked OK, there were no obvious errors and yet performance was much worse than what I'd expect. gstat clearly showed that one drive was almost constantly busy with much lower number of reads and writes per second than its peers. Perhaps previously fast transfer rates were due to caching effects. I.e. if all metadata already made it into ARC, subsequent "zfs send" commands would avoid a lot of random seeks and would show much better throughput. --Artem