From owner-freebsd-virtualization@freebsd.org Sun Mar 29 21:11:09 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 62D16264366 for ; Sun, 29 Mar 2020 21:11:09 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48r7YS59mtz3LfQ for ; Sun, 29 Mar 2020 21:11:08 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id E6DF3264331; Sun, 29 Mar 2020 21:11:00 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E54CF264330 for ; Sun, 29 Mar 2020 21:11:00 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48r7YH0sVNz3LV7 for ; Sun, 29 Mar 2020 21:10:58 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E311B7087 for ; Sun, 29 Mar 2020 21:01:00 +0000 (UTC) (envelope-from bugzilla-noreply@FreeBSD.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02TL10l8054802 for ; Sun, 29 Mar 2020 21:01:00 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Received: (from bugzilla@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02TL106E054801 for virtualization@FreeBSD.org; Sun, 29 Mar 2020 21:01:00 GMT (envelope-from bugzilla-noreply@FreeBSD.org) Message-Id: <202003292101.02TL106E054801@kenobi.freebsd.org> X-Authentication-Warning: kenobi.freebsd.org: bugzilla set sender to bugzilla-noreply@FreeBSD.org using -f From: bugzilla-noreply@FreeBSD.org To: virtualization@FreeBSD.org Subject: Problem reports for virtualization@FreeBSD.org that need special attention Date: Sun, 29 Mar 2020 21:01:00 +0000 MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 29 Mar 2020 21:11:09 -0000 To view an individual PR, use: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=(Bug Id). The following is a listing of current problems submitted by FreeBSD users, which need special attention. These represent problem reports covering all versions including experimental development code and obsolete releases. Status | Bug Id | Description ------------+-----------+--------------------------------------------------- New | 240945 | [hyper-v] [netvsc] hn network driver incorrectly 1 problems total for which you should take action. From owner-freebsd-virtualization@freebsd.org Mon Mar 30 14:18:00 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9EC7D2A4C36 for ; Mon, 30 Mar 2020 14:18:00 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48rZLH3dwGz4RXK for ; Mon, 30 Mar 2020 14:17:59 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id D85522A4A66; Mon, 30 Mar 2020 14:12:03 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id D736F2A4A65 for ; Mon, 30 Mar 2020 14:12:03 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48rZCP0wVWz4Pk8 for ; Mon, 30 Mar 2020 14:12:01 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 70CBC1D551 for ; Mon, 30 Mar 2020 14:11:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02UEBqah074070 for ; Mon, 30 Mar 2020 14:11:52 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02UEBq1K074069 for virtualization@FreeBSD.org; Mon, 30 Mar 2020 14:11:52 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Mon, 30 Mar 2020 14:11:52 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mail@rubenvos.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Mar 2020 14:18:00 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #44 from mail@rubenvos.com --- (In reply to mail from comment #43) Well, the instance is still online. It appears to be in some sort of troubl= e, probably since: Mar 25 03:05:38 zfs01 kernel: nvme1: Missing interrupt Mar 25 03:07:04 zfs01 syslogd: last message repeated 1 times Mar 25 03:07:13 zfs01 syslogd: last message repeated 1 times logging on with ssh works, but there is a huge lag noticable. There is a 5-= 15 second delay between characters I type on my keyboard and them appearing on= the terminal. There are no other obvious log entries and top shows: root@zfs01:/usr/home/ruben # top -n -b 10 last pid: 6417; load averages: 0.29, 0.56, 0.53 up 18+00:53:17=20=20= =20 14:09:45 49 processes: 1 running, 48 sleeping CPU: 0.8% user, 0.0% nice, 2.9% system, 0.1% interrupt, 96.2% idle Mem: 23M Active, 3300K Inact, 25M Laundry, 14G Wired, 714M Buf, 1539M Free ARC: 10G Total, 6090M MFU, 4031M MRU, 420K Anon, 80M Header, 508M Other 9037M Compressed, 12G Uncompressed, 1.35:1 Ratio Swap: 8192M Total, 8192M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMM= AND 92501 root 1 25 0 21M 11M select 1 0:02 6.98% ssh 97476 root 1 22 0 11M 2528K select 0 0:01 2.98% pv 1145 root 1 21 0 11M 2528K select 0 0:01 0.98% nc 46323 root 20 20 0 11M 2528K rpcsvc 1 84:50 0.00% nfsd 73646 ntpd 1 20 0 16M 16M select 0 1:20 0.00% ntpd 30467 _pflogd 1 20 0 12M 3072K bpf 0 1:15 0.00% pflo= gd 81756 root 1 21 0 12M 3892K wait 0 0:13 0.00% sh 70675 root 1 34 0 11M 968K nanslp 1 0:07 0.00% cron 58098 root 1 20 0 11M 2672K select 0 0:07 0.00% sysl= ogd 29034 root 1 20 0 15M 5648K select 0 0:05 0.00% nrpe3 root@zfs01:/usr/home/ruben #=20 The NFS clients for this nfs server also seem to be performing their normal tasks on the dataset. Strange. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Mon Mar 30 17:25:43 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E402B260E94 for ; Mon, 30 Mar 2020 17:25:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48rfVv0CFYz3BxM for ; Mon, 30 Mar 2020 17:25:43 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 48D04260E8E; Mon, 30 Mar 2020 17:25:38 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 484FF260E8D for ; Mon, 30 Mar 2020 17:25:38 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48rfVn685Hz3Btw for ; Mon, 30 Mar 2020 17:25:37 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8EB111FFDB for ; Mon, 30 Mar 2020 17:17:11 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 02UHHB1e033961 for ; Mon, 30 Mar 2020 17:17:11 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 02UHHBhr033960 for virtualization@FreeBSD.org; Mon, 30 Mar 2020 17:17:11 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Mon, 30 Mar 2020 17:17:11 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cperciva@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 30 Mar 2020 17:25:44 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #45 from Colin Percival --- Hi Ruben, Getting more "missing interrupt" messages is an expected result of reducing= the timeout period, so that's kind of a good sign. I don't know what would be making the system very slow though -- not unless this is a T3 and you've run out of burstable CPU quota. Very strange indee= d. Any errors logged recently relating to ENA? --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Thu Apr 2 22:01:54 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CABA52678A4 for ; Thu, 2 Apr 2020 22:01:54 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48tcV91dnXz48Tp for ; Thu, 2 Apr 2020 22:01:52 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 44B3C26789A; Thu, 2 Apr 2020 22:01:47 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3E721267899 for ; Thu, 2 Apr 2020 22:01:47 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48tcV20z50z48Rh for ; Thu, 2 Apr 2020 22:01:45 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 743A820BE5 for ; Thu, 2 Apr 2020 22:01:39 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 032M1dMv075113 for ; Thu, 2 Apr 2020 22:01:39 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 032M1dv7075112 for virtualization@FreeBSD.org; Thu, 2 Apr 2020 22:01:39 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Thu, 02 Apr 2020 22:01:39 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mail@rubenvos.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2020 22:01:55 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #46 from mail@rubenvos.com --- (In reply to Colin Percival from comment #45) Hi Colin, Well, another crash this morning. A coworker also had to reboot the instance again. There are a lot of messages regarding ENA in the logs. This is /var/log/messages from the event: Apr 1 13:33:45 zfs01 ntpd[73646]: leapsecond file ('/var/db/ntpd.leap-seconds.list'): expired less than 96 days ago Apr 2 03:04:51 zfs01 kernel: nvme1: Missing interrupt Apr 2 03:04:58 zfs01 syslogd: last message repeated 1 times Apr 2 03:05:12 zfs01 kernel: ena0: The number of lost tx completion is abo= ve the threshold (129 > 128). Reset the device Apr 2 03:05:12 zfs01 kernel: ena0: Trigger reset is on Apr 2 03:05:12 zfs01 kernel: ena0: device is going DOWN Apr 2 03:05:13 zfs01 kernel: ena0: free uncompleted tx mbuf qid 0 idx 0x25d Apr 2 03:05:14 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Apr 2 03:05:14 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Apr 2 03:05:14 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Apr 2 03:05:14 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Apr 2 03:05:14 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Apr 2 03:05:14 zfs01 kernel: ena0: device is going UP Apr 2 03:05:14 zfs01 kernel: ena0: link is UP Apr 2 03:05:26 zfs01 kernel: nvme1: Missing interrupt Apr 2 03:05:40 zfs01 syslogd: last message repeated 1 times Apr 2 03:09:20 zfs01 kernel: ena0: The number of lost tx completion is abo= ve the threshold (129 > 128). Reset the device Apr 2 03:09:20 zfs01 kernel: ena0: Trigger reset is on Apr 2 03:09:20 zfs01 kernel: ena0: device is going DOWN Apr 2 03:09:20 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:09:20 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 03:09:20 zfs01 kernel: ena0: free uncompleted tx mbuf qid 0 idx 0x25bena0: free uncompleted tx mbuf qid 1 idx 0x2ad Apr 2 03:09:21 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Apr 2 03:09:21 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Apr 2 03:09:21 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Apr 2 03:09:21 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Apr 2 03:09:21 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Apr 2 03:09:21 zfs01 kernel: ena0: device is going UP Apr 2 03:09:21 zfs01 kernel: ena0: link is UP Apr 2 03:11:29 zfs01 kernel: nvme1: Missing interrupt Apr 2 03:11:40 zfs01 kernel: nvme1: cpl does not map to outstanding cmd Apr 2 03:11:40 zfs01 kernel: cdw0:00000000 sqhd:0013 sqid:0001 cid:001c p:0 sc:00 sct:0 m:0 dnr:0 Apr 2 03:11:40 zfs01 kernel: nvme1: Missing interrupt Apr 2 03:11:40 zfs01 kernel: nvme1: Resetting controller due to a timeout. Apr 2 03:11:40 zfs01 kernel: nvme1: resetting controller Apr 2 03:11:40 zfs01 kernel: nvme1: temperature threshold not supported Apr 2 03:11:40 zfs01 kernel: nvme1: aborting outstanding i/o Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14013800= 00 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14013808= 00 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14013815= 76 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14013824= 00 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14014058= 64 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14014082= 64 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14014127= 20 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14014132= 40 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:14014145= 04 len:256 Apr 2 03:11:40 zfs01 kernel: nvme1: resubmitting queued i/o Apr 2 03:11:40 zfs01 kernel: nvme1: WRITE sqid:2 cid:0 nsid:1 lba:19386367= 92 len:8 Apr 2 03:11:46 zfs01 kernel: nvme1: Missing interrupt Apr 2 03:11:56 zfs01 kernel: nvme1: Resetting controller due to a timeout. Apr 2 03:11:56 zfs01 kernel: nvme1: resetting controller Apr 2 03:11:56 zfs01 kernel: nvme1: temperature threshold not supported Apr 2 03:11:56 zfs01 kernel: nvme1: aborting outstanding i/o Apr 2 03:11:56 zfs01 syslogd: last message repeated 10 times Apr 2 03:12:02 zfs01 kernel: nvme1: WRITE sqid:1 cid:5 nsid:1 lba:3847102 len:64 Apr 2 03:12:02 zfs01 kernel: nvme1: INVALID OPCODE (00/01) sqid:1 cid:28 cdw0:0 Apr 2 03:12:02 zfs01 kernel: nvme1: Missing interrupt Apr 2 03:14:18 zfs01 kernel: ena0: Keep alive watchdog timeout. Apr 2 03:14:18 zfs01 kernel: ena0: Trigger reset is on Apr 2 03:14:18 zfs01 kernel: ena0: device is going DOWN Apr 2 03:15:36 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:15:36 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 03:15:39 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Apr 2 03:15:39 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Apr 2 03:15:39 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Apr 2 03:15:39 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Apr 2 03:15:39 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Apr 2 03:15:39 zfs01 kernel: ena0: ena0: device is going UP Apr 2 03:15:39 zfs01 kernel: link is UP Apr 2 03:17:22 zfs01 dhclient[69443]: send_packet: Network is down Apr 2 03:17:40 zfs01 syslogd: last message repeated 4 times Apr 2 03:17:44 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:17:44 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 03:17:50 zfs01 dhclient[69443]: send_packet: Network is down Apr 2 03:18:21 zfs01 syslogd: last message repeated 1 times Apr 2 03:19:37 zfs01 syslogd: last message repeated 2 times Apr 2 03:19:39 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:19:39 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 03:23:20 zfs01 kernel: ena0: Keep alive watchdog timeout. Apr 2 03:23:20 zfs01 kernel: ena0: Trigger reset is on Apr 2 03:23:20 zfs01 kernel: ena0: device is going DOWN Apr 2 03:23:20 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:23:20 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 03:23:23 zfs01 kernel: ena0: free uncompleted tx mbuf qid 1 idx 0x2ac Apr 2 03:23:24 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Apr 2 03:23:24 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Apr 2 03:23:24 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Apr 2 03:23:24 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Apr 2 03:23:24 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Apr 2 03:23:24 zfs01 kernel: ena0: device is going UP Apr 2 03:23:24 zfs01 kernel: ena0: link is UP Apr 2 03:25:09 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:25:09 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 03:27:00 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:27:00 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 03:28:49 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 03:28:49 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 04:37:52 zfs01 nrpe[41394]: Could not read request from client 172.28.8.16, bailing out... Apr 2 04:43:45 zfs01 kernel: ena0: The number of lost tx completion is abo= ve the threshold (129 > 128). Reset the device Apr 2 04:43:45 zfs01 kernel: ena0: Trigger reset is on Apr 2 04:43:45 zfs01 kernel: ena0: device is going DOWN Apr 2 04:44:22 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 04:44:22 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 04:44:25 zfs01 kernel: ena0: free uncompleted tx mbuf qid 1 idx 0x28f Apr 2 04:44:26 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Apr 2 04:44:26 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Apr 2 04:44:26 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Apr 2 04:44:26 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Apr 2 04:44:26 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Apr 2 04:44:26 zfs01 kernel: stray irq260 Apr 2 04:44:26 zfs01 kernel: ena0: ena0: device is going UP Apr 2 04:44:26 zfs01 kernel: link is UP Apr 2 04:46:33 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 04:46:33 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 04:48:36 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 04:48:36 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 04:50:25 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 04:50:25 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 04:52:37 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 04:52:37 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 04:54:40 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 04:54:40 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:09:52 zfs01 kernel: ena0: The number of lost tx completion is abo= ve the threshold (129 > 128). Reset the device Apr 2 05:09:52 zfs01 kernel: ena0: Trigger reset is on Apr 2 05:09:52 zfs01 kernel: ena0: device is going DOWN Apr 2 05:11:03 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:11:03 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:11:04 zfs01 kernel: ena0: free uncompleted tx mbuf qid 1 idx 0x22b Apr 2 05:11:05 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Apr 2 05:11:05 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Apr 2 05:11:05 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Apr 2 05:11:05 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Apr 2 05:11:05 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Apr 2 05:11:05 zfs01 kernel: stray irq260 Apr 2 05:11:05 zfs01 kernel: ena0: device is going UP Apr 2 05:11:05 zfs01 kernel: ena0: link is UP Apr 2 05:13:00 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:13:00 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:14:57 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:14:57 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:16:46 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:16:46 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:18:39 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:18:39 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:20:41 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:20:41 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:22:33 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:22:33 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:28:52 zfs01 kernel: ena0: Keep alive watchdog timeout. Apr 2 05:28:52 zfs01 kernel: ena0: Trigger reset is on Apr 2 05:28:52 zfs01 kernel: ena0: device is going DOWN Apr 2 05:28:52 zfs01 dhclient[7435]: send_packet6: Network is down Apr 2 05:28:52 zfs01 dhclient[7435]: dhc6: send_packet6() sent -1 of 52 by= tes Apr 2 05:29:24 zfs01 kernel: ena0: attempting to allocate 3 MSI-X vectors = (9 supported) Apr 2 05:29:24 zfs01 kernel: msi: routing MSI-X IRQ 259 to local APIC 0 ve= ctor 52 Apr 2 05:29:24 zfs01 kernel: msi: routing MSI-X IRQ 260 to local APIC 0 ve= ctor 53 Apr 2 05:29:24 zfs01 kernel: msi: routing MSI-X IRQ 261 to local APIC 0 ve= ctor 54 Apr 2 05:29:24 zfs01 kernel: ena0: using IRQs 259-261 for MSI-X Apr 2 05:29:24 zfs01 kernel: stray irq260 Apr 2 05:29:24 zfs01 kernel: ena0: ena0: device is going UP Apr 2 05:29:24 zfs01 kernel: link is UP Apr 2 05:57:05 zfs01 syslogd: kernel boot file is /boot/kernel/kernel Apr 2 05:57:05 zfs01 kernel: pflog0: promiscuous mode disabled Apr 2 05:57:05 zfs01 kernel:=20 Apr 2 05:57:05 zfs01 syslogd: last message repeated 1 times Apr 2 05:57:05 zfs01 kernel: Fatal trap 12: page fault while in kernel mode Apr 2 05:57:05 zfs01 kernel: cpuid =3D 0; apic id =3D 00 Apr 2 05:57:05 zfs01 kernel: fault virtual address =3D 0x78 Apr 2 05:57:05 zfs01 kernel: fault code =3D supervisor read= data, page not present Apr 2 05:57:05 zfs01 kernel: instruction pointer =3D 0x20:0xffffffff80b17c97 Apr 2 05:57:05 zfs01 kernel: stack pointer =3D 0x28:0xfffffe000045d1e0 Apr 2 05:57:05 zfs01 kernel: frame pointer =3D 0x28:0xfffffe000045d210 Apr 2 05:57:05 zfs01 kernel: code segment =3D base 0x0, limit 0xfffff, type 0x1b Apr 2 05:57:05 zfs01 kernel: =3D DPL 0, pres 1, long 1, = def32 0, gran 1 Apr 2 05:57:05 zfs01 kernel: processor eflags =3D interrupt enabled, resu= me, IOPL =3D 0 Apr 2 05:57:05 zfs01 kernel: current process =3D 12 (irq257: nvme0:io0) Apr 2 05:57:05 zfs01 kernel: trap number =3D 12 Apr 2 05:57:05 zfs01 kernel: panic: page fault Apr 2 05:57:05 zfs01 kernel: cpuid =3D 0 Apr 2 05:57:05 zfs01 kernel: time =3D 1585806979 Apr 2 05:57:05 zfs01 kernel: KDB: stack backtrace: Apr 2 05:57:05 zfs01 kernel: #0 0xffffffff80c1d2b7 at kdb_backtrace+0x67 Apr 2 05:57:05 zfs01 kernel: #1 0xffffffff80bd05ed at vpanic+0x19d Apr 2 05:57:05 zfs01 kernel: #2 0xffffffff80bd0443 at panic+0x43 Apr 2 05:57:05 zfs01 kernel: #3 0xffffffff810a7dcc at trap_fatal+0x39c Apr 2 05:57:05 zfs01 kernel: #4 0xffffffff810a7e19 at trap_pfault+0x49 Apr 2 05:57:05 zfs01 kernel: #5 0xffffffff810a740f at trap+0x29f Apr 2 05:57:05 zfs01 kernel: #6 0xffffffff81081a2c at calltrap+0x8 Apr 2 05:57:05 zfs01 kernel: #7 0xffffffff8079c5ac at nvme_qpair_complete_tracker+0x1bc Apr 2 05:57:05 zfs01 kernel: #8 0xffffffff8079c2c4 at nvme_qpair_process_completions+0xd4 Apr 2 05:57:05 zfs01 kernel: #9 0xffffffff80b93dd4 at ithread_loop+0x1d4 Apr 2 05:57:05 zfs01 kernel: #10 0xffffffff80b90c43 at fork_exit+0x83 Apr 2 05:57:05 zfs01 kernel: #11 0xffffffff81082a6e at fork_trampoline+0xe Apr 2 05:57:05 zfs01 kernel: Uptime: 20d16h39m50s Apr 2 05:57:05 zfs01 kernel: Rebooting... Apr 2 05:57:05 zfs01 kernel: ---<>--- This occured only minutes again after running the periodic framework: root@zfs01:~ # ls -lahtuT /etc/periodic/daily/ total 128 -rwxr-xr-x 1 root wheel 451B Apr 2 03:02:09 2020 480.leapfile-ntpd -rwxr-xr-x 1 root wheel 2.0K Apr 2 03:01:58 2020 460.status-mail-rejec= ts -rwxr-xr-x 1 root wheel 1.0K Apr 2 03:01:00 2020 450.status-security -rwxr-xr-x 1 root wheel 1.4K Apr 2 03:01:00 2020 440.status-mailq -rwxr-xr-x 1 root wheel 705B Apr 2 03:01:00 2020 430.status-uptime -rwxr-xr-x 1 root wheel 611B Apr 2 03:01:00 2020 420.status-network -rwxr-xr-x 1 root wheel 684B Apr 2 03:01:00 2020 410.status-mfi -rwxr-xr-x 1 root wheel 590B Apr 2 03:01:00 2020 409.status-gconcat -rwxr-xr-x 1 root wheel 590B Apr 2 03:01:00 2020 408.status-gstripe -rwxr-xr-x 1 root wheel 591B Apr 2 03:01:00 2020 407.status-graid3 -rwxr-xr-x 1 root wheel 596B Apr 2 03:01:00 2020 406.status-gmirror -rwxr-xr-x 1 root wheel 807B Apr 2 03:01:00 2020 404.status-zfs -rwxr-xr-x 1 root wheel 583B Apr 2 03:01:00 2020 401.status-graid -rwxr-xr-x 1 root wheel 773B Apr 2 03:01:00 2020 400.status-disks -rwxr-xr-x 1 root wheel 724B Apr 2 03:01:00 2020 330.news -r-xr-xr-x 1 root wheel 1.4K Apr 2 03:01:00 2020 310.accounting -rwxr-xr-x 1 root wheel 693B Apr 2 03:01:00 2020 300.calendar -rwxr-xr-x 1 root wheel 1.0K Apr 2 03:01:00 2020 210.backup-aliases -rwxr-xr-x 1 root wheel 1.7K Apr 2 03:01:00 2020 200.backup-passwd -rwxr-xr-x 1 root wheel 603B Apr 2 03:01:00 2020 150.clean-hoststat -rwxr-xr-x 1 root wheel 1.0K Apr 2 03:01:00 2020 140.clean-rwho -rwxr-xr-x 1 root wheel 709B Apr 2 03:01:00 2020 130.clean-msgs -rwxr-xr-x 1 root wheel 1.1K Apr 2 03:01:00 2020 120.clean-preserve -rwxr-xr-x 1 root wheel 1.5K Apr 2 03:01:00 2020 110.clean-tmps -rwxr-xr-x 1 root wheel 1.3K Apr 2 03:01:00 2020 100.clean-disks -rwxr-xr-x 1 root wheel 811B Apr 1 03:54:15 2020 999.local -rwxr-xr-x 1 root wheel 2.8K Apr 1 03:54:15 2020 800.scrub-zfs -rwxr-xr-x 1 root wheel 845B Apr 1 03:54:15 2020 510.status-world-kern= el -rwxr-xr-x 1 root wheel 737B Apr 1 03:54:15 2020 500.queuerun -rwxr-xr-x 1 root wheel 498B Apr 1 03:54:15 2020 480.status-ntpd drwxr-xr-x 2 root wheel 1.0K Dec 7 06:23:36 2018 . drwxr-xr-x 6 root wheel 512B Dec 7 06:23:36 2018 .. root@zfs01:~ # ls -lahtuT /etc/periodic/security/ total 68 -r--r--r-- 1 root wheel 2.8K Apr 2 03:01:49 2020 security.functions -rwxr-xr-x 1 root wheel 2.3K Apr 2 03:01:49 2020 900.tcpwrap -rwxr-xr-x 1 root wheel 2.3K Apr 2 03:01:49 2020 800.loginfail -rwxr-xr-x 1 root wheel 1.9K Apr 2 03:01:49 2020 700.kernelmsg -rwxr-xr-x 1 root wheel 2.0K Apr 2 03:01:49 2020 610.ipf6denied -rwxr-xr-x 1 root wheel 2.2K Apr 2 03:01:49 2020 550.ipfwlimit -rwxr-xr-x 1 root wheel 2.1K Apr 2 03:01:49 2020 520.pfdenied -rwxr-xr-x 1 root wheel 1.9K Apr 2 03:01:49 2020 510.ipfdenied -rwxr-xr-x 1 root wheel 2.0K Apr 2 03:01:49 2020 500.ipfwdenied -rwxr-xr-x 1 root wheel 1.9K Apr 2 03:01:49 2020 410.logincheck -rwxr-xr-x 1 root wheel 1.9K Apr 2 03:01:49 2020 400.passwdless -rwxr-xr-x 1 root wheel 1.9K Apr 2 03:01:49 2020 300.chkuid0 -rwxr-xr-x 1 root wheel 2.3K Apr 2 03:01:49 2020 200.chkmounts -rwxr-xr-x 1 root wheel 2.2K Apr 2 03:01:25 2020 110.neggrpperm -rwxr-xr-x 1 root wheel 2.2K Apr 2 03:01:00 2020 100.chksetuid drwxr-xr-x 2 root wheel 512B Dec 7 06:23:36 2018 . drwxr-xr-x 6 root wheel 512B Dec 7 06:23:36 2018 .. root@zfs01:~ #=20 I don't know how long we will be able to keep this installation running und= er these circumstances; the platform it provides NFS for will be "LIVE" in a couple of weeks and this issue needs to be resolved prior to that. Kind regards, Ruben --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Thu Apr 2 22:15:58 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0EDF9267F06 for ; Thu, 2 Apr 2020 22:15:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48tcpN2bsWz4F58 for ; Thu, 2 Apr 2020 22:15:56 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id 12374267F00; Thu, 2 Apr 2020 22:15:51 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0EB81267EFF for ; Thu, 2 Apr 2020 22:15:51 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48tcpD5K1tz4F3H for ; Thu, 2 Apr 2020 22:15:48 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id E799220D97 for ; Thu, 2 Apr 2020 22:06:08 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 032M68jc084204 for ; Thu, 2 Apr 2020 22:06:08 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 032M68pb084203 for virtualization@FreeBSD.org; Thu, 2 Apr 2020 22:06:08 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Thu, 02 Apr 2020 22:06:08 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cperciva@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 02 Apr 2020 22:15:58 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #47 from Colin Percival --- Hi Ruben, Good timing. Can you test the patch in https://reviews.freebsd.org/D24256 = ?=20 It will probably add "nvme_timeout" messages to your console log -- I'd be interested to know how many -- but it also fixes a nasty race condition whi= ch I suspect is causing some of the breakage you're seeing. --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Fri Apr 3 00:20:53 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 59C8B26C956; Fri, 3 Apr 2020 00:20:53 +0000 (UTC) (envelope-from doctor@doctor.nl2k.ab.ca) Received: from doctor.nl2k.ab.ca (doctor.nl2k.ab.ca [204.209.81.1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48tgZM4vp0z42JN; Fri, 3 Apr 2020 00:20:43 +0000 (UTC) (envelope-from doctor@doctor.nl2k.ab.ca) Received: from doctor by doctor.nl2k.ab.ca with local (Exim 4.93.0.4 (FreeBSD)) (envelope-from ) id 1jKA51-0005bH-0b; Thu, 02 Apr 2020 18:21:03 -0600 Date: Thu, 2 Apr 2020 18:21:02 -0600 From: The Doctor To: freebsd-virtualization@freebsd.org, freebsd-ports@freebsd.org Subject: grub-bhyve Message-ID: <20200403002102.GA13362@doctor.nl2k.ab.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Rspamd-Queue-Id: 48tgZM4vp0z42JN X-Spamd-Bar: ++ X-Spamd-Result: default: False [2.25 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(0.00)[+a]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_MEDIUM(0.23)[0.234,0]; BAD_REP_POLICIES(0.10)[]; MIME_TRACE(0.00)[0:+]; URIBL_PBL(0.02)[empire.kred]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(0.00)[nl2k.ab.ca,quarantine]; NEURAL_SPAM_LONG(0.16)[0.158,0]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; INTRODUCTION(2.00)[]; ASN(0.00)[asn:6171, ipnet:204.209.81.0/24, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-0.16)[ip: (-0.38), ipnet: 204.209.81.0/24(-0.19), asn: 6171(-0.15), country: CA(-0.09)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2020 00:20:53 -0000 Grub-bhyve is using Gcc 4.8.5 to copile with. I recall compiling with gcc 9.0 with a few updates needed to the code. Has anyone else tried this? -- Member - Liberal International This is doctor@@nl2k.ab.ca Ici doctor@@nl2k.ab.ca Yahweh, Queen & country!Never Satan President Republic!Beware AntiChrist rising! https://www.empire.kred/ROOTNK?t=94a1f39b Look at Psalms 14 and 53 on Atheism Marvelous Truth, confront us at every turn, in every guise. -Denise Levertov From owner-freebsd-virtualization@freebsd.org Fri Apr 3 02:31:50 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 882B1270BD0 for ; Fri, 3 Apr 2020 02:31:50 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from vito.onthenet.com.au (vito.onthenet.com.au [203.22.124.72]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48tkTc2q0nz3M6F for ; Fri, 3 Apr 2020 02:31:48 +0000 (UTC) (envelope-from grehan@freebsd.org) Received: from alto.onthenet.com.au (alto2.onthenet.com.au [203.13.68.14]) by vito.onthenet.com.au (Postfix) with ESMTPS id 467D720D1586 for ; Fri, 3 Apr 2020 12:31:36 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [203.13.68.150]) by alto.onthenet.com.au (Postfix) with ESMTPS id 2D59120B49B1 for ; Fri, 3 Apr 2020 12:31:36 +1000 (AEST) Received: from iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) by iredmail.onthenet.com.au (Postfix) with ESMTP id 23D962069EBA for ; Fri, 3 Apr 2020 12:31:36 +1000 (AEST) Received: from iredmail.onthenet.com.au ([127.0.0.1]) by iredmail.onthenet.com.au (iredmail.onthenet.com.au [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id ssyRo4IL5JDP for ; Fri, 3 Apr 2020 12:31:36 +1000 (AEST) Received: from MacBook-Air-4.local (otn-120-29-62-248.broadband.onthenet.net [120.29.62.248]) by iredmail.onthenet.com.au (Postfix) with ESMTPSA id 7E9C7206B14B; Fri, 3 Apr 2020 12:31:35 +1000 (AEST) Subject: Re: grub-bhyve To: The Doctor References: <20200403002102.GA13362@doctor.nl2k.ab.ca> Cc: freebsd-virtualization@freebsd.org, freebsd-ports@freebsd.org From: Peter Grehan Message-ID: Date: Fri, 3 Apr 2020 12:31:35 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: <20200403002102.GA13362@doctor.nl2k.ab.ca> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=UIBp4xXy c=1 sm=1 tr=0 a=A6CF0fG5TOl4vs6YHvqXgw==:117 a=DPSYfp6L38iglyTf480Cpw==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=IkcTkHD0fZMA:10 a=cl8xLZFz6L8A:10 a=pBpp5M7Eg4yIJLAhGUMA:9 a=QEXdDO2ut3YA:10 a=pHzHmUro8NiASowvMSCR:22 a=n87TN5wuljxrRezIQYnT:22 X-CMAE-Score: 0 X-CMAE-Analysis: v=2.3 cv=d81BNirE c=1 sm=1 tr=0 a=eF5wA05EhuPcU0j7PNT2Vg==:117 a=DPSYfp6L38iglyTf480Cpw==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=IkcTkHD0fZMA:10 a=cl8xLZFz6L8A:10 a=pBpp5M7Eg4yIJLAhGUMA:9 a=QEXdDO2ut3YA:10 a=pHzHmUro8NiASowvMSCR:22 a=n87TN5wuljxrRezIQYnT:22 X-Rspamd-Queue-Id: 48tkTc2q0nz3M6F X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-1.96 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-0.98)[-0.984,0]; NEURAL_HAM_LONG(-0.97)[-0.971,0]; ASN(0.00)[asn:9313, ipnet:203.22.124.0/23, country:AU] X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 Apr 2020 02:31:50 -0000 > Grub-bhyve is using Gcc 4.8.5 to copile with. > > I recall compiling with gcc 9.0 with a few updates needed to the> code. Has anyone else tried this? The ports version is using gcc 9 so no issues there. The grub2-bhyve github README has just been updated to remove the versions from gcc/gdb - the most recent versions of those are fine to use. later, Peter. From owner-freebsd-virtualization@freebsd.org Sat Apr 4 14:15:28 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C2F312ADAB5 for ; Sat, 4 Apr 2020 14:15:28 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 48vf330grzz4NW0 for ; Sat, 4 Apr 2020 14:15:27 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id C80F72ADAAB; Sat, 4 Apr 2020 14:15:16 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C78D32ADAAA for ; Sat, 4 Apr 2020 14:15:16 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48vf2q12Lsz4NRf for ; Sat, 4 Apr 2020 14:15:14 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 47ACC2269B for ; Sat, 4 Apr 2020 14:15:06 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 034EF5LI099938 for ; Sat, 4 Apr 2020 14:15:05 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 034EF5tr099937 for virtualization@FreeBSD.org; Sat, 4 Apr 2020 14:15:05 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Sat, 04 Apr 2020 14:15:05 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: mail@rubenvos.com X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Apr 2020 14:15:28 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #48 from mail@rubenvos.com --- (In reply to Colin Percival from comment #47) Hi Colin, I've checked out the releng/12.1 branch,, applied the patch, compiled, installed and rebooted. uname -a FreeBSD zfs01.kpnmsg.net 12.1-RELEASE-p3 FreeBSD 12.1-RELEASE-p3 #0 r359631= M: Sat Apr 4 13:41:41 UTC 2020=20=20=20=20 root@zfs01.kpnmsg.net:/data/obj/data/src/amd64.amd64/sys/GENERIC amd64 I still have the=20 hw.nvme.timeout_period=3D1 active in /boot/loader.conf , is that OK or should I switch that to somethi= ng else ? Will keep you updated on /var/log/messages contents. Hope the issue will trigger next week, because after that we really have to switch instance typ= es (for the instance that got triggered the most ( until now :P) ). Regards, Ruben --=20 You are receiving this mail because: You are the assignee for the bug.= From owner-freebsd-virtualization@freebsd.org Sat Apr 4 17:51:05 2020 Return-Path: Delivered-To: freebsd-virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id EF29927B900 for ; Sat, 4 Apr 2020 17:51:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mailman.nyi.freebsd.org (unknown [127.0.1.3]) by mx1.freebsd.org (Postfix) with ESMTP id 48vkqs25gyz3J7q for ; Sat, 4 Apr 2020 17:51:05 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: by mailman.nyi.freebsd.org (Postfix) id D05B527B8CC; Sat, 4 Apr 2020 17:50:58 +0000 (UTC) Delivered-To: virtualization@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id CC70F27B8CA for ; Sat, 4 Apr 2020 17:50:58 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48vkqj6vH0z3J5n for ; Sat, 4 Apr 2020 17:50:57 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org (kenobi.freebsd.org [IPv6:2610:1c1:1:606c::50:1d]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 8D14425026 for ; Sat, 4 Apr 2020 17:34:15 +0000 (UTC) (envelope-from bugzilla-noreply@freebsd.org) Received: from kenobi.freebsd.org ([127.0.1.5]) by kenobi.freebsd.org (8.15.2/8.15.2) with ESMTP id 034HYFtw051097 for ; Sat, 4 Apr 2020 17:34:15 GMT (envelope-from bugzilla-noreply@freebsd.org) Received: (from www@localhost) by kenobi.freebsd.org (8.15.2/8.15.2/Submit) id 034HYFWn051096 for virtualization@FreeBSD.org; Sat, 4 Apr 2020 17:34:15 GMT (envelope-from bugzilla-noreply@freebsd.org) X-Authentication-Warning: kenobi.freebsd.org: www set sender to bugzilla-noreply@freebsd.org using -f From: bugzilla-noreply@freebsd.org To: virtualization@FreeBSD.org Subject: [Bug 235856] FreeBSD freezes on AWS EC2 t3 machines Date: Sat, 04 Apr 2020 17:34:14 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: changed X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: Base System X-Bugzilla-Component: kern X-Bugzilla-Version: 12.0-RELEASE X-Bugzilla-Keywords: X-Bugzilla-Severity: Affects Only Me X-Bugzilla-Who: cperciva@FreeBSD.org X-Bugzilla-Status: New X-Bugzilla-Resolution: X-Bugzilla-Priority: --- X-Bugzilla-Assigned-To: virtualization@FreeBSD.org X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: Message-ID: In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: https://bugs.freebsd.org/bugzilla/ Auto-Submitted: auto-generated MIME-Version: 1.0 X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Apr 2020 17:51:07 -0000 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D235856 --- Comment #49 from Colin Percival --- Thanks! Yes, it would be great if you can keep that tunable in place so we= can see how it affects this. I'm particularly interested in seeing anything lo= gged concerning nvme -- there will probably be additional "nvme_timeout" messages being logged even when nothing is visibly going wrong, and I'd like to see = how often (and under what conditions) those appear. --=20 You are receiving this mail because: You are the assignee for the bug.=