Date: Tue, 15 Mar 2016 12:05:28 -0700 (PDT) From: "Jeffrey Bouquet" <jbtakk@iherebuywisely.com> To: "Ian Lepore" <ian@freebsd.org> Cc: "FreeBSD Current" <freebsd-current@freebsd.org>, "current@freebsd.org" <current@freebsd.org>, "Adrian Chadd" <adrian.chadd@gmail.com>, "Mark Johnston" <markj@freebsd.org>, "Gary Jennejohn" <gljennjohn@gmail.com> Subject: Re: how to recycle Inact memory more aggressively? Message-ID: <E1afuHY-0004MX-Gn@rmm6prod02.runbox.com> In-Reply-To: <1458055811.68920.24.camel@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 15 Mar 2016 09:30:11 -0600, Ian Lepore <ian@freebsd.org> wrote: > On Tue, 2016-03-15 at 07:20 -0700, Jeffrey Bouquet wrote: > > rsync... see bottom posting > >=20 > > On Tue, 15 Mar 2016 07:43:46 +0100, olli hauer <ohauer@gmx.de> wrote: > >=20 > > > On 2016-03-14 15:19, Ian Lepore wrote: > > > > On Sun, 2016-03-13 at 19:08 -0700, Adrian Chadd wrote: > > > > > On 13 March 2016 at 18:51, Mark Johnston <markj@freebsd.org> > > > > > wrote: > > > > > > On Sun, Mar 13, 2016 at 06:33:46PM -0700, Adrian Chadd wrote: > > > > > > > Hi, > > > > > > >=20 > > > > > > > I can reproduce this by doing a mkimage on a large > > > > > > > destination > > > > > > > file > > > > > > > image. it looks like it causes all the desktop processes to > > > > > > > get > > > > > > > paged > > > > > > > out whilst it's doing so, and then the whole UI freezes > > > > > > > until it > > > > > > > catches up. > > > > > >=20 > > > > > > mkimg(1) maps the destination file with MAP_NOSYNC, so if > > > > > > it's > > > > > > larger > > > > > > than RAM, I think it'll basically force the pagedaemon to > > > > > > write out > > > > > > the > > > > > > image as it tries to reclaim pages from the inactive queue. > > > > > > This > > > > > > can > > > > > > cause stalls if the pagedaemon blocks waiting for some I/O to > > > > > > complete. > > > > > > The user/alc/PQ_LAUNDRY branch helps alleviate this problem > > > > > > by > > > > > > using a > > > > > > different thread to launder dirty pages. I use mkimg on > > > > > > various > > > > > > desktop > > > > > > machines to build bhyve images and have noticed the problem > > > > > > you're > > > > > > describing; PQ_LAUNDRY helps quite a bit in that case. But I > > > > > > don't > > > > > > know > > > > > > why this would be a new problem. > > > > > >=20 > > > > >=20 > > > > > That's why I'm confused. I just know that it didn't used to > > > > > cause the > > > > > whole UI to hang due to paging. > > > > >=20 > > > >=20 > > > > I've been noticing this too. This machine runs 10-stable and > > > > this use > > > > of the swap began happening recently when I updated from 10 > > > > -stable > > > > around the 10.2 release time to 10-stable right about when the > > > > 10.3 > > > > code freeze began. > > > >=20 > > > > In my case I have no zfs anything here. I noticed the problem > > > > bigtime > > > > yesterday when rsync was syncing a ufs filesystem of about 500GB > > > > from > > > > one disk to another (probably 70-80 GB actually needed copying).=20 > > > > My > > > > desktop apps were noticibly unresponsive when I activated a > > > > window that > > > > had been idle for a while (like it would take a couple seconds > > > > for the > > > > app to begin responding). I could see lots of swap In happening > > > > in top > > > > during this unresponsiveness, and noticible amounts of Out > > > > activity > > > > when nothing was happening except the rsync. > > > >=20 > > > > This is amd64, 12GB ram, 16GB swap, a tmpfs had about 400MB in it > > > > at > > > > the time. Prior to the update around the 10.3 freeze, this > > > > machine > > > > would never touch the swap no matter what workload I threw at it > > > > (this > > > > rsync stuff happens every day, it's the usual workload). > > > >=20 > > >=20 > > > I'm not sure if it is the same problem, or port related. > > >=20 > > > On two systems without zfs but with many files e.g. svn servers I > > > see now > > > from time to time they are running out of swap. > > >=20 > > > kernel: swap_pager_getswapspace(9): failed > > > kernel: swap_pager_getswapspace(16): failed > > > ... > > >=20 > > > It also happened on one system during the nightly periodic tasks > > > holding > > > only millions of backup files. > > >=20 > > > $ freebsd-version -ku > > > 10.2-RELEASE-p9 > > > 10.2-RELEASE-p13 > > >=20 > > >=20 > > > _______________________________________________ > > > freebsd-current@freebsd.org mailing list > > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > > To unsubscribe, send any mail to " > > > freebsd-current-unsubscribe@freebsd.org" > >=20 > >=20 > > Just a point I've bought up elsewhere... > > I've, if I recall, wrecked several filesystems (although EIDE) using > > rsync at the normal bus rate, and sometimes > > thumbdrives with whatever filesystem type on them. > >=20 > > I settled on --bwlimit=3D1500, max for unattended rsync usage and > > almost every day > > use --bwlimit=3D700. > >=20 > > The latter enables several resource-intensive processes ( music, > > classical music videos, svn, pkg, browsing, etc) to > > proceed apace concurrently on the desktop (SATA not EIDE) with nary a > > hang nor slowdown. > >=20 > > If I recall, the usual speed is 10000 so that is less than ten > > percent, if I recall, of the usual speed. > >=20 > > YMMV. > >=20 > > J. > >=20 > > PS as an afterthough, it would be useful if that were more prominent > > on the man page somewhere or even > > in the port's pkg-message or pkg-description.=20=20 > > The SATA more robust than EIDE on FreeBSD that I've come across, > > though I prefer not to hint at because I > > believe it to be the fault of EIDE firmware rather than FreeBSD code. > > FWIW. >=20 > I have no real idea what any of that is about, but before it turns into Just trying to insert a helpful parameter into anyone's shell scripts > some kind of "rsync is bad" mythology, let me just say that I've been Quite not the point, I apologize if that was the tone. > using rsync to copy gigabytes of backup data every day for years now.=20 > I've never had any kind of problem, especially system responsiveness > problems, until this increased swapfile activity thing started > happening on 10-stable in the past few months. >=20 > To reiterate: rsync is not in any way at fault here, and any suggestion > that the unresponsiveness should be "fixed" by screwing around with > rsync parms that have worked fine for a decade is just something I > completely reject. I should maybe have put forth more prominently that the parameter was meant= to allow other resource-intensive processes to coexist, on a desktop machine.= =20=20 Apologies for the unintended oversight. >=20 > I'm sure I'd see the same kind of increased swapping with ANY process > that read and wrote gigabytes of data in a short period of time. And > that's what this thread is about: What has changed to cause this > regression that multiple people are seeing where lots of IO now drives > an unusual amount of swapfile activity on systems that used to NEVER > write anything to swap? >=20 > -- Ian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?E1afuHY-0004MX-Gn>