From owner-freebsd-fs@FreeBSD.ORG Wed Jan 16 10:20:25 2013 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 7C89BC2D; Wed, 16 Jan 2013 10:20:25 +0000 (UTC) (envelope-from prvs=1728d5906c=killing@multiplay.co.uk) Received: from mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) by mx1.freebsd.org (Postfix) with ESMTP id 7EBD1AAD; Wed, 16 Jan 2013 10:20:24 +0000 (UTC) Received: from r2d2 ([188.220.16.49]) by mail1.multiplay.co.uk (mail1.multiplay.co.uk [85.236.96.23]) (MDaemon PRO v10.0.4) with ESMTP id md50001723941.msg; Wed, 16 Jan 2013 10:20:22 +0000 X-Spam-Processed: mail1.multiplay.co.uk, Wed, 16 Jan 2013 10:20:22 +0000 (not processed: message from valid local sender) X-MDRemoteIP: 188.220.16.49 X-Return-Path: prvs=1728d5906c=killing@multiplay.co.uk X-Envelope-From: killing@multiplay.co.uk Message-ID: <98723B7F45F643F3A96FDB6B9285E935@multiplay.co.uk> From: "Steven Hartland" To: "Nicolas Rachinsky" References: <20130114094010.GA75529@mid.pc5.i.0x5.de> <20130114195148.GA20540@mid.pc5.i.0x5.de> <20130114214652.GA76779@mid.pc5.i.0x5.de> <20130115224556.GA41774@mid.pc5.i.0x5.de> <00F86FD0E85D4EEEA1A01E115497F022@multiplay.co.uk> <20130116074245.GB47781@mid.pc5.i.0x5.de> Subject: Re: slowdown of zfs (tx->tx) Date: Wed, 16 Jan 2013 10:20:46 -0000 MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="iso-8859-1"; reply-type=original Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2900.5931 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.6157 Cc: freebsd-fs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jan 2013 10:20:25 -0000 ----- Original Message ----- From: "Nicolas Rachinsky" To: "Steven Hartland" Cc: "Artem Belevich" ; "freebsd-fs" Sent: Wednesday, January 16, 2013 7:42 AM Subject: Re: slowdown of zfs (tx->tx) >* Steven Hartland [2013-01-16 00:57 -0000]: >> >> ----- Original Message ----- From: "Artem Belevich" >> >> To: "Nicolas Rachinsky" >> Cc: "freebsd-fs" >> Sent: Wednesday, January 16, 2013 12:16 AM >> Subject: Re: slowdown of zfs (tx->tx) >> >> >> >It appears that lots of threads are stuck in >> >metaslab_activate->space_map_load_wait path. This sounds like CR# >> >6876962 in Solaris: "degraded write performance with threads held up >> >by space_map_load_wait(). This bug is fixed in patch 147440-05, -06 or >> >-07, which is current and contains the fix." Alas, I could not find >> >specifics on how the issue got fixed and whether the same fix is >> >present in illumos and FreeBSD. >> >> That would tend to indicate its blocking on write. If this is the case >> yet the rsync is copying from this box, with little else doing writes >> it could be atime which is causing the issue. > > I was probably misformulating my mail. The rsync writes to the local > zpool. > >> A test for this would be to use the following to disable atime and see >> if that helps: >> zfs set atime=off [filesystem] If you don't need atime I would still recommend setting atime=off. >> Also out of interest does the pool have many snapshots? > > There are 115 filesystems. 84 of these have between 10 and 20 > snapshots. Hmm so over 1000 snapshots, that's not going to help. If there's are something that's built up over time + increased disk usage, that could well explain the slowdown your seeing and would also explain why your seeing threads taking time in metaslab_activate->space_map_load_wait. Are the snapshots something you can clear down and test to see if that improves things? Out of interested what type of data are you working with and is it compressible? If it is, it might be worth testing with compression enabled. Regards Steve ================================================ This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmaster@multiplay.co.uk.