Date: Tue, 14 Apr 2009 21:35:10 -0700 From: Artem Belevich <fbsdlist@src.cx> To: Ben Kelly <ben@wanderview.com> Cc: freebsd-current@freebsd.org Subject: Re: [patch] zfs livelock and thread priorities Message-ID: <ed91d4a80904142135n429dea52o672abf51116fa707@mail.gmail.com> In-Reply-To: <AC3C4C3F-40C6-4AF9-BAF3-2C4D1E444839@wanderview.com> References: <DC9F2088-A0AF-467D-8574-F24A045ABD81@wanderview.com> <49C2CFF6.8070608@egr.msu.edu> <BDABA909-C2AE-4A55-869B-CA01BE778A82@wanderview.com> <ed91d4a80904131636u18c90474w7cdaa57bc7000e02@mail.gmail.com> <08D7DC2A-68BE-47B6-8D5D-5DE6B48F87E5@wanderview.com> <AC3C4C3F-40C6-4AF9-BAF3-2C4D1E444839@wanderview.com>
next in thread | previous in thread | raw e-mail | index | archive | help
I'll give it a try in a few days. I'll let you know how it went.
BTW, now that you're tinkering with ZFS threads and priorities, whould
you by any chance have any idea why zfs scrub is so painfully slow on
-current?
When I start scrub on my -stable box, it pretty much runs full speed
-- I can see disks under load all the time.
However on -current scrub seems to run in small bursts. Disks get busy
for a second or so and then things get quiet for about five seconds or
so and this pattern repeats over and over.
--Artem
On Tue, Apr 14, 2009 at 7:32 PM, Ben Kelly <ben@wanderview.com> wrote:
> On Apr 14, 2009, at 11:50 AM, Ben Kelly wrote:
>>
>> On Apr 13, 2009, at 7:36 PM, Artem Belevich wrote:
>>>
>>> Tried your patch that used PRIBIO+{1,2} for priorities with -current
>>> r191008 and the kernel died with "spinlock held too long" panic.
>>> Actually, there apparently were two instances of panic on different
>>> cores..
>>>
>>> Here's output of "alltrace" and "ps" after the crash:
>>> http://pastebin.com/f140f4596
>>>
>>> I've reverted the change and kernel booted just fine.
>>>
>>> The box is quad-core with two ZFS pools -- one single-disk and another
>>> one is a two-disk mirror. Freebsd is installed on UFS partitions, ZFS
>>> is used for user stuff only.
>>
>> Thanks for the report!
>>
>> I don't have a lot of time to look at this today, but it appears that
>> there is a race condition on SMP machines when setting the priority
>> immediately after the kproc is spawned. =A0As a quick hack I tried addin=
g a
>> pause between the kproc_create() and the sched_prio(). =A0Can you try th=
is
>> patch?
>>
>>
>> =A0http://www.wanderview.com/svn/public/misc/zfs_livelock/zfs_thread_pri=
ority.diff
>>
>> I'll try to take a closer look at this later in the week.
>
> Sorry for replying to my own e-mail, but I've updated the patch again wit=
h a
> less hackish approach. =A0(At the same URL above.) =A0I added a new
> kproc_create_priority() function to set the priority of the new thread
> before its first scheduled. =A0This should avoid any SMP races with setti=
ng
> the priority from an external thread.
>
> If you would be willing to try the test again with this new patch I would
> appreciate it.
>
> Thanks!
>
> - Ben
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?ed91d4a80904142135n429dea52o672abf51116fa707>