From nobody Wed Oct 6 21:29:13 2021 X-Original-To: ports-bugs@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 56B8012DE37E for ; Wed, 6 Oct 2021 21:29:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "R3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4HPndk1vZmz3lcH for ; Wed, 6 Oct 2021 21:29:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 205CF274BE for ; Wed, 6 Oct 2021 21:29:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 196LTE2C034803 for ; Wed, 6 Oct 2021 21:29:14 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 196LTEME034802 for ports-bugs@FreeBSD.org; Wed, 6 Oct 2021 21:29:14 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: ports-bugs@FreeBSD.org Subject: [Bug 258974] devel/kf5-kcoreaddons: KDE KDirWatch class + libinotify + lots of messages == sluggish KMail on FreeBSD Date: Wed, 06 Oct 2021 21:29:13 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Ports & Packages X-Bugzilla-Component: Individual Port(s) X-Bugzilla-Version: Latest X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Some People X-Bugzilla-Who: noisetube@gmail.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: kde@FreeBSD.org X-Bugzilla-Flags: maintainer-feedback? X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter flagtypes.name Message-ID: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated List-Id: Ports bug reports List-Archive: https://lists.freebsd.org/archives/freebsd-ports-bugs List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-ports-bugs@freebsd.org X-BeenThere: freebsd-ports-bugs@freebsd.org MIME-Version: 1.0 X-ThisMailContainsUnwantedMimeParts: N https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D258974 Bug ID: 258974 Summary: devel/kf5-kcoreaddons: KDE KDirWatch class + libinotify + lots of messages =3D=3D sluggish KMail on FreeBSD Product: Ports & Packages Version: Latest Hardware: Any OS: Any Status: New Severity: Affects Some People Priority: --- Component: Individual Port(s) Assignee: kde@FreeBSD.org Reporter: noisetube@gmail.com Flags: maintainer-feedback?(kde@FreeBSD.org) Assignee: kde@FreeBSD.org I recently migrated my work desktop e-mail client from KMail1 on an old ins= tall of Ubuntu Linux to KMail2 on FreeBSD 12.2-RELEASE. Since I've been working = at the same place for over 20 years, my old KMail instance had accumulated over 400,000 e-mails in various maildir folders. This may sound like a lot, but = the old setup had no real problems handling this. Unfortunately KMail2 on FreeBSD did have some problems, which I traced down= to an interaction between the KDE KDirWatch class and the libinotify library t= hat FreeBSD uses to emulate the inotify(7) facility from Linux. I use fetchmail and procmail to take incoming messages from a Microsoft IMAP server and deposit them directly in the inbox folder in KMail's maildir directory (under ~/.local/share/local-mail). KMail needs to be able to dete= ct when new files appear in the folder. Behind the scenes, this is handled by = the akonadi_maildir_resource program (plugin?) which in turn uses the KDirWatch class from libKF5CoreAddons.so. The KDirWatch class supports a couple of different options for directory monitoring, including using FAM, using inotify(7) and (as a last resort) periodically calling stat(2) on the direc= tory to detect updates. The default is to use inotify(7), because that's the preferred strategy for Linux. Since inotify(7) is a Linux custom API, the FreeBSD kernel doesn't have a direct analog for it. Instead, there is a devel/libinotify port which provi= des a compatible API built on top of kqueue(2). Note however that in Linux, you= can call inotify_add_watch() on a directory and monitor it for a number of different kinds of events, including changes to files within the directory. There is a limitation with libinotify though: it has to maintain open file descriptors for the filesystem objects that it monitors. If you choose to monitor a directory and monitor changes to files within it, then libinotify will need to maintain an open file descriptor not just for the directory it= self but also for _every_ regular file it contains. This happens for exaple if y= ou call inotify_add_watch() with the IN_ATTRIB and/or the IN_MODIFY flags, as described in the README here: https://github.com/libinotify-kqueue/libinotify-kqueue Note the admonition about possibly running out of file descriptors, and the possibility of needing to increase the file descriptor limit. (I wouldn't consider that a reasonable way to deal with the matter though: it's like sa= ying "if you shoot yourself in the foot, just add more feet.") It turns out that the KDirWatch class does exactly this: https://invent.kde.org/frameworks/kcoreaddons/-/blob/master/src/lib/io/kdir= watch.cpp#L710 The end result: - If you use KMail with maildir folders - And you have hundreds of thousands of messages in those folders - Then akonadi_maildir_resource will end up with hundreds of thousands of o= pen file descriptors (one for _every_ message) This makes KMail/Akonadi very sluggish and has a fairly negative impact on = the system in general. akonadi_maildir_resource may get stuck consuming gobs of= CPU cycles without getting any real work done, and if you run fstat(1) it will = just hang. (lsof won't, but it'll spit out 400000 lines of output.) Also, the only reason KMail works at all in this scenario is that the defau= lt process limit for open file descriptors in FreeBSD these days can be over 900000. And if you reduce the descriptor limit to something less than the number of messages in your maildir folder, Akonadi and KMail will crash. I'm not exactly sure of the right way to address this, but something should= be done, otherwise KMail is just not usable on FreeBSD. It may *seem* like it = is if you have only a moderately small number of messages, but it will get progressively worse the more you use it. I think the best thing would be if someone found a way to rework libinotify= to eliminate this behavior, but I'm not sure of the best way to do that, since= it may be beyond the limits of what's possible with kqueue(2) or the FreeBSD kernel in general. (Also I wasn't sure if I should file this bug against the kcoreaddons port or the libinotify port either, so flipped a coin.) I did want to file a PR though just so there would be some publicly visible info on the matter, and so that people could potentially take advantage of = the workarounds I'm about to describe. There are basically two workarounds: 1) Rebuild the libinotify.so.0.0.0 shared library from the devel/libinotify port using the patch shown here: https://people.freebsd.org/~wpaul/libinotify/ This forces off the IN_ATTRIB and IN_MODIFY flags for directories. This at least prevents KDirWatch from making it spiral out of control. 2) The KDirWatch class can be configured to change the preferred monitoring mechanism, as described here: https://api.kde.org/frameworks/kcoreaddons/html/classKDirWatch.html I don't think that KDirWatch is compiled with FAM support on FreeBSD, so the only reasonable alternative is to select the "Stat" method, which while not= as ideal as modifying libinotify, at least doesn't require recompiling any cod= e. I *think* what you can do is modify the .kde/share/config/kdeglobals file to include the following: [DirWatch] PreferredMethod=3DStat Alternatively, you can use the KDIRWATCH_METHOD environment variable, e.g.: % setenv KDIRWATCH_METHOD Stat You have to make sure you do this before you launch KMail though. I elected to use option 1, because I also discovered that the Qt library al= so has a directory/file watcher class, and it also uses the inotify interface = and sets the IN_ATTRIB flag when monitoring directories. I decided fixing both problems with one change would be better. Now that I've patched libinotify, KMail/Akonadi is much more responsive, ev= en with my 20 years worth of mail. So I'm not in urgent need of a fix myself. = But responsible parties might want to at least consider changing the default wa= tch method for FreeBSD from "inotify" to "Stat", otherwise KMail isn't really v= ery usable on FreeBSD. -Bill --=20 You are receiving this mail because: You are the assignee for the bug.=