Date: Wed, 16 Jan 2013 10:20:46 -0000 From: "Steven Hartland" <killing@multiplay.co.uk> To: "Nicolas Rachinsky" <fbsd-mas-0@ml.turing-complete.org> Cc: freebsd-fs <freebsd-fs@freebsd.org> Subject: Re: slowdown of zfs (tx->tx) Message-ID: <98723B7F45F643F3A96FDB6B9285E935@multiplay.co.uk> References: <CAFqOu6gWpMsWN0pTBiv10WfwyGWMfO9GzMLWTtcVxHixr-_i3Q@mail.gmail.com> <20130114094010.GA75529@mid.pc5.i.0x5.de> <CAFqOu6hxfGt_M6Jo9qWeifDz9YnNc_Bd9H-GEe4RYtutaPvH5w@mail.gmail.com> <20130114195148.GA20540@mid.pc5.i.0x5.de> <CAFqOu6jwJ4qhbOovN_NhzusdQJvrbvUC3g93sziR=Uw99SGenw@mail.gmail.com> <20130114214652.GA76779@mid.pc5.i.0x5.de> <CAFqOu6jKX-Ks6C1RK5GwZ51ZVUSnGSe7S99_EfK%2BfwLPjAFFYw@mail.gmail.com> <20130115224556.GA41774@mid.pc5.i.0x5.de> <CAFqOu6jJnWdbikPmE1-UML5i_x7meF%2BiyY=9WBRyv2j7AeOaSg@mail.gmail.com> <00F86FD0E85D4EEEA1A01E115497F022@multiplay.co.uk> <20130116074245.GB47781@mid.pc5.i.0x5.de>
next in thread | previous in thread | raw e-mail | index | archive | help
----- Original Message -----
From: "Nicolas Rachinsky" <fbsd-mas-0@ml.turing-complete.org>
To: "Steven Hartland" <killing@multiplay.co.uk>
Cc: "Artem Belevich" <art@freebsd.org>; "freebsd-fs" <freebsd-fs@freebsd.org>
Sent: Wednesday, January 16, 2013 7:42 AM
Subject: Re: slowdown of zfs (tx->tx)
>* Steven Hartland <killing@multiplay.co.uk> [2013-01-16 00:57 -0000]:
>>
>> ----- Original Message ----- From: "Artem Belevich"
>> <art@freebsd.org>
>> To: "Nicolas Rachinsky" <fbsd-mas-0@ml.turing-complete.org>
>> Cc: "freebsd-fs" <freebsd-fs@freebsd.org>
>> Sent: Wednesday, January 16, 2013 12:16 AM
>> Subject: Re: slowdown of zfs (tx->tx)
>>
>>
>> >It appears that lots of threads are stuck in
>> >metaslab_activate->space_map_load_wait path. This sounds like CR#
>> >6876962 in Solaris: "degraded write performance with threads held up
>> >by space_map_load_wait(). This bug is fixed in patch 147440-05, -06 or
>> >-07, which is current and contains the fix." Alas, I could not find
>> >specifics on how the issue got fixed and whether the same fix is
>> >present in illumos and FreeBSD.
>>
>> That would tend to indicate its blocking on write. If this is the case
>> yet the rsync is copying from this box, with little else doing writes
>> it could be atime which is causing the issue.
>
> I was probably misformulating my mail. The rsync writes to the local
> zpool.
>
>> A test for this would be to use the following to disable atime and see
>> if that helps:
>> zfs set atime=off [filesystem]
If you don't need atime I would still recommend setting atime=off.
>> Also out of interest does the pool have many snapshots?
>
> There are 115 filesystems. 84 of these have between 10 and 20
> snapshots.
Hmm so over 1000 snapshots, that's not going to help. If there's are something
that's built up over time + increased disk usage, that could well explain
the slowdown your seeing and would also explain why your seeing threads
taking time in metaslab_activate->space_map_load_wait.
Are the snapshots something you can clear down and test to see if that
improves things?
Out of interested what type of data are you working with and is it
compressible? If it is, it might be worth testing with compression
enabled.
Regards
Steve
================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.
In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337
or return the E.mail to postmaster@multiplay.co.uk.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?98723B7F45F643F3A96FDB6B9285E935>