Date: Sun, 8 Sep 2019 16:18:19 -0400 From: Mark Johnston <markj@freebsd.org> To: Slawa Olhovchenkov <slw@zxy.spb.ru> Cc: Andriy Gapon <avg@freebsd.org>, src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: Re: svn commit: r351673 - in head: lib/libmemstat share/man/man9 sys/cddl/compat/opensolaris/kern sys/kern sys/vm Message-ID: <20190908201819.GA49837@raichu> In-Reply-To: <20190907153110.GG3953@zxy.spb.ru> References: <201909012222.x81MMh0F022462@repo.freebsd.org> <79c74018-1329-ee69-3480-e2f99821fa93@FreeBSD.org> <20190903161427.GA38096@zxy.spb.ru> <20190903220106.GB26733@raichu> <20190904144524.GD3953@zxy.spb.ru> <20190907145034.GB6523@spy> <20190907153110.GG3953@zxy.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Sep 07, 2019 at 06:31:10PM +0300, Slawa Olhovchenkov wrote:
> On Sat, Sep 07, 2019 at 10:50:34AM -0400, Mark Johnston wrote:
>
> > On Wed, Sep 04, 2019 at 05:45:24PM +0300, Slawa Olhovchenkov wrote:
> > > On Tue, Sep 03, 2019 at 06:01:06PM -0400, Mark Johnston wrote:
> > > > > Mostly problem I am see at this
> > > > > work -- very slowly vm_page_free(). May be currenly this is more
> > > > > speedy...
> > > >
> > > > How did you determine this?
> > >
> > > This is you guess:
> >
> > So was the guess correct?
>
> I am just trust to you.
> How to check this guess?
You can try to measure time spent in pmap_remove() relative to the rest
of _kmem_unback().
> > If so, IMO the real solution is to avoid kmem_*
> > for data buffers and use ABD instead.
>
> What problem resolve this?
To allocate buffers larger than PAGE_SIZE the kernel must allocate a
number of physical pages and map them using the page tables. The step
of creating and destroying mappings is expensive and doesn't scale well
to many CPUs. With ABD, ZFS avoids this expense when its caches are
shrunk.
> ABD any way is slowly vs kmem_*.
Can we solve this problem instead?
> > > ======
> > > > while ((slab = SLIST_FIRST(&freeslabs)) != NULL) {
> > > > SLIST_REMOVE(&freeslabs, slab, uma_slab, us_hlink);
> > > > keg_free_slab(keg, slab, keg->uk_ipers);
> > > > }
> > > > 2019 Feb 2 19:49:54.800524364 zio_data_buf_1048576 1032605 cache_reclaim limit 100 dom 0 nitems 1672 imin 298
> > > > 2019 Feb 2 19:49:54.800524364 zio_data_buf_1048576 1033736 cache_reclaim recla 149 dom 0 nitems 1672 imin 298
> > > > 2019 Feb 2 19:49:54.802524468 zio_data_buf_1048576 3119710 cache_reclaim limit 100 dom 1 nitems 1 imin 0
> > > > 2019 Feb 2 19:49:54.802524468 zio_data_buf_1048576 3127550 keg_drain2
> > > > 2019 Feb 2 19:49:54.803524487 zio_data_buf_1048576 4444219 keg_drain3
> > > > 2019 Feb 2 19:49:54.838524634 zio_data_buf_1048576 39553705 keg_drain4
> > > > 2019 Feb 2 19:49:54.838524634 zio_data_buf_1048576 39565323 zone_reclaim:return
> > > >
> > > > 35109.486 ms for last loop, 149 items to freed.
> > >
> > > 35ms to free 149MB (38144 4KB pages), so roughly 1us per page. That
> > > does seem like a lot, but freeing a page (vm_page_free(m)) is much
> > > more expensive than freeing an item to UMA (i.e., uma_zfree()).
> > > Most of that time will be spent in _kmem_unback().
> > > ======
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20190908201819.GA49837>