Date: Tue, 15 Mar 2016 09:30:11 -0600 From: Ian Lepore <ian@freebsd.org> To: jbtakk@iherebuywisely.com, FreeBSD Current <freebsd-current@freebsd.org> Cc: "current@freebsd.org" <current@freebsd.org>, Adrian Chadd <adrian.chadd@gmail.com>, Mark Johnston <markj@freebsd.org>, Gary Jennejohn <gljennjohn@gmail.com> Subject: Re: how to recycle Inact memory more aggressively? Message-ID: <1458055811.68920.24.camel@freebsd.org> In-Reply-To: <E1afpq7-0005Cs-NX@rmm6prod02.runbox.com> References: <E1afpq7-0005Cs-NX@rmm6prod02.runbox.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Tue, 2016-03-15 at 07:20 -0700, Jeffrey Bouquet wrote: > rsync... see bottom posting > > On Tue, 15 Mar 2016 07:43:46 +0100, olli hauer <ohauer@gmx.de> wrote: > > > On 2016-03-14 15:19, Ian Lepore wrote: > > > On Sun, 2016-03-13 at 19:08 -0700, Adrian Chadd wrote: > > > > On 13 March 2016 at 18:51, Mark Johnston <markj@freebsd.org> > > > > wrote: > > > > > On Sun, Mar 13, 2016 at 06:33:46PM -0700, Adrian Chadd wrote: > > > > > > Hi, > > > > > > > > > > > > I can reproduce this by doing a mkimage on a large > > > > > > destination > > > > > > file > > > > > > image. it looks like it causes all the desktop processes to > > > > > > get > > > > > > paged > > > > > > out whilst it's doing so, and then the whole UI freezes > > > > > > until it > > > > > > catches up. > > > > > > > > > > mkimg(1) maps the destination file with MAP_NOSYNC, so if > > > > > it's > > > > > larger > > > > > than RAM, I think it'll basically force the pagedaemon to > > > > > write out > > > > > the > > > > > image as it tries to reclaim pages from the inactive queue. > > > > > This > > > > > can > > > > > cause stalls if the pagedaemon blocks waiting for some I/O to > > > > > complete. > > > > > The user/alc/PQ_LAUNDRY branch helps alleviate this problem > > > > > by > > > > > using a > > > > > different thread to launder dirty pages. I use mkimg on > > > > > various > > > > > desktop > > > > > machines to build bhyve images and have noticed the problem > > > > > you're > > > > > describing; PQ_LAUNDRY helps quite a bit in that case. But I > > > > > don't > > > > > know > > > > > why this would be a new problem. > > > > > > > > > > > > > That's why I'm confused. I just know that it didn't used to > > > > cause the > > > > whole UI to hang due to paging. > > > > > > > > > > I've been noticing this too. This machine runs 10-stable and > > > this use > > > of the swap began happening recently when I updated from 10 > > > -stable > > > around the 10.2 release time to 10-stable right about when the > > > 10.3 > > > code freeze began. > > > > > > In my case I have no zfs anything here. I noticed the problem > > > bigtime > > > yesterday when rsync was syncing a ufs filesystem of about 500GB > > > from > > > one disk to another (probably 70-80 GB actually needed copying). > > > My > > > desktop apps were noticibly unresponsive when I activated a > > > window that > > > had been idle for a while (like it would take a couple seconds > > > for the > > > app to begin responding). I could see lots of swap In happening > > > in top > > > during this unresponsiveness, and noticible amounts of Out > > > activity > > > when nothing was happening except the rsync. > > > > > > This is amd64, 12GB ram, 16GB swap, a tmpfs had about 400MB in it > > > at > > > the time. Prior to the update around the 10.3 freeze, this > > > machine > > > would never touch the swap no matter what workload I threw at it > > > (this > > > rsync stuff happens every day, it's the usual workload). > > > > > > > I'm not sure if it is the same problem, or port related. > > > > On two systems without zfs but with many files e.g. svn servers I > > see now > > from time to time they are running out of swap. > > > > kernel: swap_pager_getswapspace(9): failed > > kernel: swap_pager_getswapspace(16): failed > > ... > > > > It also happened on one system during the nightly periodic tasks > > holding > > only millions of backup files. > > > > $ freebsd-version -ku > > 10.2-RELEASE-p9 > > 10.2-RELEASE-p13 > > > > > > _______________________________________________ > > freebsd-current@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to " > > freebsd-current-unsubscribe@freebsd.org" > > > Just a point I've bought up elsewhere... > I've, if I recall, wrecked several filesystems (although EIDE) using > rsync at the normal bus rate, and sometimes > thumbdrives with whatever filesystem type on them. > > I settled on --bwlimit=1500, max for unattended rsync usage and > almost every day > use --bwlimit=700. > > The latter enables several resource-intensive processes ( music, > classical music videos, svn, pkg, browsing, etc) to > proceed apace concurrently on the desktop (SATA not EIDE) with nary a > hang nor slowdown. > > If I recall, the usual speed is 10000 so that is less than ten > percent, if I recall, of the usual speed. > > YMMV. > > J. > > PS as an afterthough, it would be useful if that were more prominent > on the man page somewhere or even > in the port's pkg-message or pkg-description. > The SATA more robust than EIDE on FreeBSD that I've come across, > though I prefer not to hint at because I > believe it to be the fault of EIDE firmware rather than FreeBSD code. > FWIW. I have no real idea what any of that is about, but before it turns into some kind of "rsync is bad" mythology, let me just say that I've been using rsync to copy gigabytes of backup data every day for years now. I've never had any kind of problem, especially system responsiveness problems, until this increased swapfile activity thing started happening on 10-stable in the past few months. To reiterate: rsync is not in any way at fault here, and any suggestion that the unresponsiveness should be "fixed" by screwing around with rsync parms that have worked fine for a decade is just something I completely reject. I'm sure I'd see the same kind of increased swapping with ANY process that read and wrote gigabytes of data in a short period of time. And that's what this thread is about: What has changed to cause this regression that multiple people are seeing where lots of IO now drives an unusual amount of swapfile activity on systems that used to NEVER write anything to swap? -- Ian
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?1458055811.68920.24.camel>