From owner-freebsd-stable@freebsd.org Mon Oct 29 12:34:51 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B6D3A10D6C88 for ; Mon, 29 Oct 2018 12:34:51 +0000 (UTC) (envelope-from SRS0=eyDw=NJ=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4A1426F2B4 for ; Mon, 29 Oct 2018 12:34:51 +0000 (UTC) (envelope-from SRS0=eyDw=NJ=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 23EF128423 for ; Mon, 29 Oct 2018 13:34:42 +0100 (CET) Received: from illbsd.quip.test (ip-86-49-16-209.net.upcbroadband.cz [86.49.16.209]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id C8D0628417 for ; Mon, 29 Oct 2018 13:34:40 +0100 (CET) To: freebsd-stable@freebsd.org From: Miroslav Lachman <000.fbsd@quip.cz> Subject: sync command runs for hours after upgrade to 11.2 Message-ID: <21a79514-0ada-83bd-bd4a-77fb838042fe@quip.cz> Date: Mon, 29 Oct 2018 13:34:40 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.3 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Oct 2018 12:34:51 -0000 I upgraded about 10 machines from 10.4 to 11.2 this month. The last machines were upgrade few days ago and one of them exhibits strange behaviour with sync(8) command. We have sync in one script for MRTG which is called each 5 minutes. Every day between 8 - 15h the sync calls are not finished in few seconds (or fraction of second) as usual but they run for hours and cannot be killed. # ps auxww | grep sync mrtg 22897 23.9 0.0 4232 2048 - R 09:20 69:30.86 sync mrtg 22186 23.8 0.0 4232 2048 - R 09:15 72:14.36 sync mrtg 29535 23.2 0.0 4232 2048 - R 10:05 48:05.80 sync mrtg 46040 22.5 0.0 4232 2048 - R 11:25 22:24.87 sync mrtg 59472 22.4 0.0 4232 2048 - R 12:40 4:30.85 sync mrtg 25683 22.2 0.0 4232 2048 - R 09:40 59:20.61 sync mrtg 51972 22.1 0.0 4232 2048 - R 12:05 12:19.05 sync mrtg 27983 22.0 0.0 4232 2048 - R 09:55 52:25.62 sync mrtg 6534 21.8 0.0 4232 2048 - R 08:00 126:28.14 sync mrtg 49674 21.7 0.0 4232 2048 - R 11:50 15:57.96 sync mrtg 58720 21.6 0.0 4232 2048 - R 12:35 5:33.81 sync mrtg 7419 21.5 0.0 4232 2048 - R 08:05 122:03.06 sync mrtg 16371 21.5 0.0 4232 2048 - R 08:50 88:07.14 sync mrtg 26454 21.5 0.0 4232 2048 - R 09:45 56:50.32 sync mrtg 36149 21.5 0.0 4232 2048 - R 10:35 37:25.89 sync mrtg 37607 21.5 0.0 4232 2048 - R 10:45 34:19.37 sync mrtg 6588 21.3 0.0 4232 2048 - R 08:00 126:31.50 sync mrtg 30283 21.1 0.0 4232 2048 - R 10:10 46:04.96 sync mrtg 11030 21.0 0.0 4232 2048 - R 08:15 114:01.56 sync mrtg 52724 21.0 0.0 4232 2048 - R 12:10 11:07.51 sync mrtg 61965 21.0 0.0 4232 2048 - R 12:55 1:23.08 sync mrtg 35360 20.9 0.0 4232 2048 - R 10:30 38:59.89 sync mrtg 40764 20.9 0.0 4232 2048 - R 11:05 27:59.73 sync mrtg 28855 20.8 0.0 4232 2048 - R 10:00 50:07.67 sync mrtg 33199 20.8 0.0 4232 2048 - R 10:15 44:08.61 sync mrtg 57984 20.8 0.0 4232 2048 - R 12:30 6:39.88 sync mrtg 17866 20.7 0.0 4232 2048 - R 09:00 81:18.44 sync mrtg 57226 20.7 0.0 4232 2048 - R 12:25 7:49.24 sync mrtg 34610 20.6 0.0 4232 2048 - R 10:25 40:34.89 sync mrtg 27242 20.5 0.0 4232 2048 - R 09:50 54:37.43 sync mrtg 8180 20.4 0.0 4232 2048 - R 08:10 117:47.97 sync mrtg 12422 20.4 0.0 4232 2048 - R 08:25 106:02.85 sync mrtg 17071 20.4 0.0 4232 2048 - R 08:55 84:41.97 sync mrtg 48920 20.4 0.0 4232 2048 - R 11:45 17:14.73 sync mrtg 23521 20.3 0.0 4232 2048 - R 09:25 66:46.53 sync mrtg 39911 20.3 0.0 4232 2048 - R 11:00 29:28.89 sync mrtg 56437 20.2 0.0 4232 2048 - R 12:20 8:56.43 sync mrtg 61234 20.2 0.0 4232 2048 - R 12:50 2:23.54 sync mrtg 14981 20.1 0.0 4232 2048 - R 08:40 95:00.02 sync mrtg 46832 20.1 0.0 4232 2048 - R 11:30 21:03.80 sync mrtg 50447 20.1 0.0 4232 2048 - R 11:55 14:42.42 sync mrtg 51273 20.1 0.0 4232 2048 - R 12:00 13:33.55 sync mrtg 14088 20.0 0.0 4232 2048 - R 08:35 98:16.55 sync mrtg 39126 20.0 0.0 4232 2048 - R 10:55 31:12.26 sync mrtg 62764 20.0 0.0 4232 2048 - R 13:00 0:21.80 sync mrtg 15716 19.9 0.0 4232 2048 - R 08:45 91:27.30 sync mrtg 55646 19.9 0.0 4232 2048 - R 12:15 10:03.76 sync mrtg 33863 19.7 0.0 4232 2048 - R 10:20 42:24.80 sync mrtg 48174 19.6 0.0 4232 2048 - R 11:40 18:29.01 sync mrtg 24297 19.5 0.0 4232 2048 - R 09:30 64:03.59 sync mrtg 44522 19.5 0.0 4232 2048 - R 11:15 25:02.50 sync root 61355 19.5 0.0 4232 2048 2 R+ 12:51 2:06.40 sync mrtg 38358 19.4 0.0 4232 2048 - R 10:50 32:39.67 sync mrtg 36847 19.1 0.0 4232 2048 - R 10:40 35:44.49 sync mrtg 45339 19.1 0.0 4232 2048 - R 11:20 23:45.21 sync mrtg 60318 19.1 0.0 4232 2048 - R 12:45 3:27.70 sync mrtg 13200 18.8 0.0 4232 2048 - R 08:30 102:11.46 sync mrtg 18627 18.8 0.0 4232 2048 - R 09:05 78:06.64 sync mrtg 41498 18.8 0.0 4232 2048 - R 11:10 26:30.43 sync mrtg 47563 18.8 0.0 4232 2048 - R 11:35 19:42.28 sync mrtg 11700 18.5 0.0 4232 2048 - R 08:20 110:05.39 sync mrtg 19368 18.3 0.0 4232 2048 - R 09:10 75:03.78 sync mrtg 25043 18.0 0.0 4232 2048 - R 09:35 61:45.86 sync The same script is used on each of our servers without any problem. This machine is running Apache and PHP without any problems. Web is still responding even if the system load is about 60: last pid: 59734; load averages: 61.76, 59.62, 57.24 up 2+12:33:30 12:42:31 387 processes: 64 running, 318 sleeping, 5 zombie CPU: 14.1% user, 0.0% nice, 85.7% system, 0.1% interrupt, 0.1% idle Mem: 1006M Active, 9907M Inact, 9056M Laundry, 2703M Wired, 1405M Buf, 1224M Free Swap: 16G Total, 63M Used, 16G Free The one thing that can be of some coincidence is rsync backup which is running in the time between 6 - 17h. Can anybody help where should I look to solve this problem? # uname -srmi FreeBSD 11.2-RELEASE-p4 amd64 GENERIC /dev/mirror/gm0root on / (ufs, local, noatime) /dev/mirror/gm0usr on /usr (ufs, local, noatime, soft-updates) /dev/mirror/gm0var on /var (ufs, local, noatime, nosuid, soft-updates) /dev/mirror/gm0tmp on /tmp (ufs, local, noatime, noexec, nosuid, soft-updates) /dev/mirror/gm0vol0 on /vol0 (ufs, local, noatime, noexec, nosuid, soft-updates) /dev/mirror/gm0vol1 on /vol1 (ufs, local, noatime, noexec, nosuid, soft-updates) No ZFS involved. Kind regards Miroslav Lachman