From owner-freebsd-stable@freebsd.org Fri Nov 20 00:51:02 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AD391478F85 for ; Fri, 20 Nov 2020 00:51:02 +0000 (UTC) (envelope-from dmagda+stable@ee.ryerson.ca) Received: from eccles.ee.ryerson.ca (eccles.ee.ryerson.ca [141.117.1.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CcdJj4xmKz3rqj for ; Fri, 20 Nov 2020 00:51:01 +0000 (UTC) (envelope-from dmagda+stable@ee.ryerson.ca) Received: from [192.168.10.101] (216-154-8-25.cpe.teksavvy.com [216.154.8.25] (may be forged)) (authenticated bits=0) by eccles.ee.ryerson.ca (8.16.1/8.15.2) with ESMTPSA id 0AK0ouUk021408 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 19 Nov 2020 19:50:58 -0500 (EST) (envelope-from dmagda+stable@ee.ryerson.ca) X-Authentication-Warning: eccles.ee.ryerson.ca: Host 216-154-8-25.cpe.teksavvy.com [216.154.8.25] (may be forged) claimed to be [192.168.10.101] From: David Magda Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.15\)) Subject: NTP, UTC, and (negative) leap seconds Message-Id: <64E52A56-4C8F-4B9E-A4D3-33117C270D43@ee.ryerson.ca> Date: Thu, 19 Nov 2020 19:50:52 -0500 To: freebsd-stable X-Mailer: Apple Mail (2.3445.104.15) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.6.2 (eccles.ee.ryerson.ca [141.117.1.2]); Thu, 19 Nov 2020 19:50:58 -0500 (EST) X-Rspamd-Queue-Id: 4CcdJj4xmKz3rqj X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of dmagda@ee.ryerson.ca designates 141.117.1.2 as permitted sender) smtp.mailfrom=dmagda@ee.ryerson.ca X-Spamd-Result: default: False [-2.90 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+mx]; HAS_XAW(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCVD_IN_DNSWL_LOW(-0.10)[141.117.1.2:from]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[141.117.1.2:from]; ASN(0.00)[asn:26996, ipnet:141.117.0.0/17, country:CA]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[stable]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[ryerson.ca]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[141.117.1.2:from:127.0.2.255]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2020 00:51:02 -0000 Hello, So there was a recent weblog post on a timekeeping observation: https://fanf.dreamwidth.org/133823.html https://news.ycombinator.com/item?id=3D25145870 (via) https://en.wikipedia.org/wiki/Leap_second In all circumstances in the past, leap seconds were added: 23:59:57Z 23:59:58 23:59:59 23:59:60 00:00:00 00:00:01 The post brings up the possibility of a =E2=80=9Cnegative=E2=80=9D leap = second, i.e., a skip: 23:59:57Z 23:59:58 00:00:00 00:00:01 Has anyone tested this scenario on FreeBSD? Thanks for any info. -- David Magda= From owner-freebsd-stable@freebsd.org Fri Nov 20 01:40:35 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id AE4B447A489 for ; Fri, 20 Nov 2020 01:40:35 +0000 (UTC) (envelope-from ian@freebsd.org) Received: from outbound2k.ore.mailhop.org (outbound2k.ore.mailhop.org [54.148.219.64]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CcfPv0w7wz3vNS for ; Fri, 20 Nov 2020 01:40:34 +0000 (UTC) (envelope-from ian@freebsd.org) ARC-Seal: i=1; a=rsa-sha256; t=1605836434; cv=none; d=outbound.mailhop.org; s=arc-outbound20181012; b=Je+8pghU2VndES9r6dGK+SXRja3WBWx4rDdvKlU0Vre8eRfM4feMxArltklFTpONN8EuUatyyf01p B/UGU6xJMo6OwAAF4tr2aQY06KHltM78aFVkStY9Vf22u+mLs5AGStVmmEhMp0HAEy6G0W1HUPGzOl sudG1kSGQm29qp8A8Vmest28uX4uqenfyLMBG0qKnOKh3YiQn+ofdpI82OpMZCD6ituRQnfpmIjnMy QvNkPlx3dKd3fIbK+eHHFo3NKAL5Nzaw6ltSaOJwKXbuHv3Fcj+TuNLaKe13ZdknxFIes1hn7yfC5f 8Pz20gr+ey8MwuL65DgMmIsyO3+IUdA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=arc-outbound20181012; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:to:from:subject:message-id:dkim-signature:from; bh=aGnRJpk6BTJhTPLKKVRf4R/2mGLNDhZbLN8NboBTyHo=; b=aHIpofPdmS4NgZan9i0hDsVRsmgaclQ+OxtyVmIRn0KS2Es1ULJUDL9grrtDxXGJRcW4AS4DtGDlu Fv06Th2aLxBIZHCk5h3uAlz+KDsZQaLdQIX612xa/0n8q9G6x1bS6o3jFITyC3mIozE5k3/bF8bQ+u RpETbepFmR8wSGG43fpsok45+NjpQ5d6Ruua4ud712l5C7KeYFMBbnNUPOT4138A80QL/fcRLjxgNt OCnNSiJ094fTmRMr52doCvutq+ODtEIWO09fQaVGUKZja66J8K7mFzvhY1CtRlFb6p/EfrQ9eyhefa uSooAVjq50tcdMG7IXk1TJXJ1y/LpzQ== ARC-Authentication-Results: i=1; outbound4.ore.mailhop.org; spf=softfail smtp.mailfrom=freebsd.org smtp.remote-ip=67.177.211.60; dmarc=none header.from=freebsd.org; arc=none header.oldest-pass=0; DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=outbound.mailhop.org; s=dkim-high; h=content-transfer-encoding:mime-version:content-type:references:in-reply-to: date:to:from:subject:message-id:from; bh=aGnRJpk6BTJhTPLKKVRf4R/2mGLNDhZbLN8NboBTyHo=; b=jEz81RsvZ1aTMmMlmdKUbK0q+a+DMimdZSuOfahLciog3LEDTzf4o4sr7n8JYfMJDuYTZjJM9MpgO 856hAUlajYe8OdhGqrj7H7KhSVVhPRRSzv/SotJFoHtoULga5w82pnFi9VLcMenH0lN43Ds5qpc+oI 1YgzNu2vWyqgvqBJckX3BxeyAKPOq74gB9VQRlOHiPubr7yzyFCvvZcRbroXhkjwXLGZCNpxr8iePA OSwEJpuN6ff/c7dfwgiPMMy0PUoCag7gFpXlnHXmMiG07sRGZEvxIRDRIKQJ4JL7m0WdvAWdx0F/0F aXoaC7itWTVr9Vov0xUkWgKkXM7kmag== X-MHO-RoutePath: aGlwcGll X-MHO-User: 5fd3d11c-2ad1-11eb-9e11-df46ed8f892f X-Report-Abuse-To: https://support.duocircle.com/support/solutions/articles/5000540958-duocircle-standard-smtp-abuse-information X-Originating-IP: 67.177.211.60 X-Mail-Handler: DuoCircle Outbound SMTP Received: from ilsoft.org (c-67-177-211-60.hsd1.co.comcast.net [67.177.211.60]) by outbound4.ore.mailhop.org (Halon) with ESMTPSA id 5fd3d11c-2ad1-11eb-9e11-df46ed8f892f; Fri, 20 Nov 2020 01:40:32 +0000 (UTC) Received: from rev (rev [172.22.42.240]) by ilsoft.org (8.15.2/8.15.2) with ESMTP id 0AK1eU0f051775; Thu, 19 Nov 2020 18:40:30 -0700 (MST) (envelope-from ian@freebsd.org) Message-ID: <32809ec01dab023d5526b0b47fe0ad70ce558914.camel@freebsd.org> Subject: Re: NTP, UTC, and (negative) leap seconds From: Ian Lepore To: David Magda , freebsd-stable Date: Thu, 19 Nov 2020 18:40:30 -0700 In-Reply-To: <64E52A56-4C8F-4B9E-A4D3-33117C270D43@ee.ryerson.ca> References: <64E52A56-4C8F-4B9E-A4D3-33117C270D43@ee.ryerson.ca> Content-Type: text/plain; charset="iso-8859-13" X-Mailer: Evolution 3.28.5 FreeBSD GNOME Team Mime-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4CcfPv0w7wz3vNS X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [0.00 / 15.00]; TAGGED_RCPT(0.00)[stable]; ASN(0.00)[asn:16509, ipnet:54.148.0.0/15, country:US]; local_wl_from(0.00)[freebsd.org] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2020 01:40:35 -0000 On Thu, 2020-11-19 at 19:50 -0500, David Magda wrote: > Hello, > > So there was a recent weblog post on a timekeeping observation: > > https://fanf.dreamwidth.org/133823.html > https://news.ycombinator.com/item?id=251458700 (via) > https://en.wikipedia.org/wiki/Leap_second > > In all circumstances in the past, leap seconds were added: > > 23:59:57Z > 23:59:58 > 23:59:59 > 23:59:60 > 00:00:00 > 00:00:01 > > The post brings up the possibility of a ´negative¡ leap second, i.e., > a skip: > > 23:59:57Z > 23:59:58 > 00:00:00 > 00:00:01 > > Has anyone tested this scenario on FreeBSD? > > Thanks for any info. > > I run the freebsd kernel and ntpd through "negative" leap second events usually a couple times a year as part of testing our timekeeping products at $work. I'm skeptical that we'll ever see a negative leap, but we have some customers who insist on having it demonstrated to them as working correctly. The kernel and ntpd handle it just fine. No telling what any given application might do in reaction to a skipped second. I just did it now on one of our products. I scheduled a negative leap (from offset 37 to 36) to happen at the end of the day 2020-12-31, then I used a debugging command to manually force the date/time on the unit to 2020-12-31-23:58:00 and let it run. In a terminal window I had a bourne shell on that unit running while true; do sleep 1; ntptime; echo; done The output (trimmed to the range 23:59:55 - 00:00:10) looks like: ntp_gettime() returns code 2 (DEL) time e398e47b.82ebd4d0 Thu, Dec 31 2020 23:59:55.511, (.511411404), maximum error 4000 us, estimated error 5 us, TAI offset 37 ntp_adjtime() returns code 2 (DEL) modes 0x0 (), offset 4.976 us, frequency 29.182 ppm, interval 128 s, maximum error 4000 us, estimated error 5 us, status 0x2121 (PLL,DEL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 0.937 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. ntp_gettime() returns code 2 (DEL) time e398e47c.8d714b9c Thu, Dec 31 2020 23:59:56.552, (.552510729), maximum error 4500 us, estimated error 5 us, TAI offset 37 ntp_adjtime() returns code 2 (DEL) modes 0x0 (), offset 4.957 us, frequency 29.182 ppm, interval 128 s, maximum error 4500 us, estimated error 5 us, status 0x2121 (PLL,DEL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 0.988 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. ntp_gettime() returns code 2 (DEL) time e398e47d.92dbb5a0 Thu, Dec 31 2020 23:59:57.573, (.573665426), maximum error 5000 us, estimated error 5 us, TAI offset 37 ntp_adjtime() returns code 2 (DEL) modes 0x0 (), offset 4.937 us, frequency 29.182 ppm, interval 128 s, maximum error 5000 us, estimated error 5 us, status 0x2121 (PLL,DEL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 0.855 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. ntp_gettime() returns code 2 (DEL) time e398e47e.9878316c Thu, Dec 31 2020 23:59:58.595, (.595584418), maximum error 5500 us, estimated error 5 us, TAI offset 37 ntp_adjtime() returns code 2 (DEL) modes 0x0 (), offset 4.918 us, frequency 29.182 ppm, interval 128 s, maximum error 5500 us, estimated error 5 us, status 0x2121 (PLL,DEL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 1.006 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. ntp_gettime() returns code 4 (WAIT) time e398e480.9dd6cf84 Fri, Jan 1 2021 0:00:00.616, (.616559624), maximum error 6000 us, estimated error 5 us, TAI offset 36 ntp_adjtime() returns code 4 (WAIT) modes 0x0 (), offset 4.899 us, frequency 29.182 ppm, interval 128 s, maximum error 6000 us, estimated error 5 us, status 0x2121 (PLL,DEL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 1.010 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. ntp_gettime() returns code 4 (WAIT) time e398e481.a323571c Fri, Jan 1 2021 0:00:01.637, (.637258156), maximum error 6500 us, estimated error 5 us, TAI offset 36 ntp_adjtime() returns code 4 (WAIT) modes 0x0 (), offset 4.880 us, frequency 29.182 ppm, interval 128 s, maximum error 6500 us, estimated error 5 us, status 0x2121 (PLL,DEL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 1.025 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. ntp_gettime() returns code 4 (WAIT) time e398e482.a8977424 Fri, Jan 1 2021 0:00:02.658, (.658561327), maximum error 7000 us, estimated error 5 us, TAI offset 36 ntp_adjtime() returns code 4 (WAIT) modes 0x0 (), offset 4.861 us, frequency 29.182 ppm, interval 128 s, maximum error 7000 us, estimated error 5 us, status 0x2121 (PLL,DEL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 1.037 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. ntp_gettime() returns code 4 (WAIT) time e398e483.ade1f3a4 Fri, Jan 1 2021 0:00:03.679, (.679229584), maximum error 7500 us, estimated error 5 us, TAI offset 36 ntp_adjtime() returns code 4 (WAIT) modes 0x0 (), offset 4.842 us, frequency 29.182 ppm, interval 128 s, maximum error 7500 us, estimated error 5 us, status 0x2121 (PLL,DEL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 1.014 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. ntp_gettime() returns code 4 (WAIT) time e398e484.b3448064 Fri, Jan 1 2021 0:00:04.700, (.700264111), maximum error 8000 us, estimated error 5 us, TAI offset 36 ntp_adjtime() returns code 4 (WAIT) modes 0x0 (), offset 4.823 us, frequency 29.182 ppm, interval 128 s, maximum error 8000 us, estimated error 5 us, status 0x2121 (PLL,DEL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 1.051 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. ntp_gettime() returns code 4 (WAIT) time e398e485.b8a6dacc Fri, Jan 1 2021 0:00:05.721, (.721296182), maximum error 8500 us, estimated error 5 us, TAI offset 36 ntp_adjtime() returns code 4 (WAIT) modes 0x0 (), offset 4.804 us, frequency 29.182 ppm, interval 128 s, maximum error 8500 us, estimated error 5 us, status 0x2121 (PLL,DEL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 1.079 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. ntp_gettime() returns code 4 (WAIT) time e398e486.bddfdac8 Fri, Jan 1 2021 0:00:06.741, (.741697917), maximum error 1000 us, estimated error 4 us, TAI offset 36 ntp_adjtime() returns code 4 (WAIT) modes 0x0 (), offset 4.060 us, frequency 29.182 ppm, interval 128 s, maximum error 1000 us, estimated error 4 us, status 0x2101 (PLL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 0.911 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. ntp_gettime() returns code 0 (OK) time e398e487.cf8a7e74 Fri, Jan 1 2021 0:00:07.810, (.810707099), maximum error 1500 us, estimated error 4 us, TAI offset 36 ntp_adjtime() returns code 0 (OK) modes 0x0 (), offset 4.044 us, frequency 29.182 ppm, interval 128 s, maximum error 1500 us, estimated error 4 us, status 0x2101 (PLL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 0.826 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. ntp_gettime() returns code 0 (OK) time e398e488.d4da7f3c Fri, Jan 1 2021 0:00:08.831, (.831459235), maximum error 2000 us, estimated error 4 us, TAI offset 36 ntp_adjtime() returns code 0 (OK) modes 0x0 (), offset 4.028 us, frequency 29.182 ppm, interval 128 s, maximum error 2000 us, estimated error 4 us, status 0x2101 (PLL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 0.809 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. ntp_gettime() returns code 0 (OK) time e398e489.da869404 Fri, Jan 1 2021 0:00:09.853, (.853616381), maximum error 2500 us, estimated error 4 us, TAI offset 36 ntp_adjtime() returns code 0 (OK) modes 0x0 (), offset 4.012 us, frequency 29.182 ppm, interval 128 s, maximum error 2500 us, estimated error 4 us, status 0x2101 (PLL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 0.796 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. ntp_gettime() returns code 0 (OK) time e398e48a.e0216c60 Fri, Jan 1 2021 0:00:10.875, (.875510596), maximum error 3000 us, estimated error 4 us, TAI offset 36 ntp_adjtime() returns code 0 (OK) modes 0x0 (), offset 3.996 us, frequency 29.182 ppm, interval 128 s, maximum error 3000 us, estimated error 4 us, status 0x2101 (PLL,PPSSIGNAL,NANO), time constant 4, precision 0.001 us, tolerance 496 ppm, pps frequency 29.089 ppm, stability 0.128 ppm, jitter 0.653 us, intervals 23, jitter exceeded 6, stability exceeded 0, errors 2. -- Ian From owner-freebsd-stable@freebsd.org Fri Nov 20 10:19:21 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 308102C47B6 for ; Fri, 20 Nov 2020 10:19:21 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from smtpq5.tb.mail.iss.as9143.net (smtpq5.tb.mail.iss.as9143.net [212.54.42.168]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CcswS071Dz4rFN for ; Fri, 20 Nov 2020 10:19:19 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from [212.54.42.134] (helo=smtp10.tb.mail.iss.as9143.net) by smtpq5.tb.mail.iss.as9143.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kg3Ve-0003v2-As for freebsd-stable@freebsd.org; Fri, 20 Nov 2020 11:19:18 +0100 Received: from 82-101-198-11.cable.dynamic.v4.ziggo.nl ([82.101.198.11] helo=wan0.bsd4all.org) by smtp10.tb.mail.iss.as9143.net with esmtp (Exim 4.90_1) (envelope-from ) id 1kg3Ve-0002kg-0C for freebsd-stable@freebsd.org; Fri, 20 Nov 2020 11:19:18 +0100 Received: from newnas.bsd4all.local (localhost [127.0.0.1]) by wan0.bsd4all.org (Postfix) with ESMTP id 6B36649 for ; Fri, 20 Nov 2020 11:19:56 +0100 (CET) X-Virus-Scanned: amavisd-new at bsd4all.org Received: from wan0.bsd4all.org ([127.0.0.1]) by newnas.bsd4all.local (newnas.bsd4all.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OcPLffWEQrFw for ; Fri, 20 Nov 2020 11:19:56 +0100 (CET) Received: from mpro.bsd4all.local (mpro.bsd4all.local [192.168.1.65]) by wan0.bsd4all.org (Postfix) with ESMTPSA id 1D95E29F for ; Fri, 20 Nov 2020 11:19:56 +0100 (CET) From: Peter Blok Content-Type: multipart/signed; boundary="Apple-Mail=_6650814F-BC76-4F7A-AA21-D2A2261A4D96"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Commit 367705+367706 causes a pabic Message-Id: <0AEB0453-F35F-4DBA-9FD6-D389C4975AB1@bsd4all.org> Date: Fri, 20 Nov 2020 11:19:20 +0100 To: FreeBSD Stable X-Mailer: Apple Mail (2.3608.120.23.2.4) X-SourceIP: 82.101.198.11 X-Ziggo-spambar: / X-Ziggo-spamscore: 0.0 X-Ziggo-spamreport: CMAE Analysis: v=2.4 cv=HZuq8gI8 c=1 sm=1 tr=0 ts=5fb79826 a=3epKYzPC7TQzyyYZMm6bHQ==:17 a=nNwsprhYR40A:10 a=4ulB1z43gstThfcF4qIA:9 a=QEXdDO2ut3YA:10 a=g2jUX3k5JNHoXd8hjKIA:9 a=ZVk8-NSrHBgA:10 X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No X-Rspamd-Queue-Id: 4CcswS071Dz4rFN X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pblok@bsd4all.org designates 212.54.42.168 as permitted sender) smtp.mailfrom=pblok@bsd4all.org X-Spamd-Result: default: False [-5.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+a:smtp.ziggo.nl/16]; HAS_ATTACHMENT(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RECEIVED_SPAMHAUS_PBL(0.00)[82.101.198.11:received]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[212.54.42.168:from]; ASN(0.00)[asn:33915, ipnet:212.54.32.0/20, country:NL]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RCVD_COUNT_FIVE(0.00)[6]; RCVD_IN_DNSWL_LOW(-0.10)[212.54.42.168:from]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; SIGNED_SMIME(-2.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[bsd4all.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[212.54.42.168:from:127.0.2.255]; NEURAL_HAM_LONG(-1.00)[-1.000]; MAILMAN_DEST(0.00)[freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2020 10:19:21 -0000 --Apple-Mail=_6650814F-BC76-4F7A-AA21-D2A2261A4D96 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi, I=E2=80=99m afraid the last Epoch fix for bridge is not solving the = problem ( or perhaps creates a new ). This seems to happen when the jail epair is added to the bridge. Removing both fixes solves the problem. Peter kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid =3D 6; apic id =3D 06 fault virtual address =3D 0xc10 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff80695e76 stack pointer =3D 0x28:0xfffffe00bf14e6e0 frame pointer =3D 0x28:0xfffffe00bf14e720 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D resume, IOPL =3D 0 current process =3D 1686 (jail) trap number =3D 12 panic: page fault cpuid =3D 6 time =3D 1605811310 KDB: stack backtrace: #0 0xffffffff8069bb85 at kdb_backtrace+0x65 #1 0xffffffff80650a4b at vpanic+0x17b #2 0xffffffff806508c3 at panic+0x43 #3 0xffffffff809d0351 at trap_fatal+0x391 #4 0xffffffff809d03af at trap_pfault+0x4f #5 0xffffffff809cf9f6 at trap+0x286 #6 0xffffffff809a98c8 at calltrap+0x8 #7 0xffffffff80368a8d at ck_epoch_synchronize_wait+0x8d #8 0xffffffff80695c8a at epoch_wait_preempt+0xaa #9 0xffffffff80757d40 at vnet_if_init+0x120 #10 0xffffffff8078c994 at vnet_alloc+0x114 #11 0xffffffff8061e3f7 at kern_jail_set+0x1bb7 #12 0xffffffff80620190 at sys_jail_set+0x40 #13 0xffffffff809d0f07 at amd64_syscall+0x387 #14 0xffffffff809aa1ee at fast_syscall_common+0xf8= --Apple-Mail=_6650814F-BC76-4F7A-AA21-D2A2261A4D96 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCBSAw ggUcMIIEBKADAgECAhEAq2wFIs+rCK6H6/2jbblXhDANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgwNDE0MDAwMDAwWhcNMjEwNDEzMjM1 OTU5WjBEMQswCQYDVQQGEwJOTDETMBEGA1UEAxMKUGV0ZXIgQmxvazEgMB4GCSqGSIb3DQEJARYR cGJsb2tAYnNkNGFsbC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPT/3evs2a zLSIVepGa9qFVcSISd5HzoJt9xAyQ4od7NM6Qzwm446OyhzWsIN/a6+nDNB4AxzSg00QXKx4afEa FrdLzmREEfv24f88j2UZYqHAls0j26jyED5FZ068xs4gWZBG2U7EVTUNNJuUrrmqBNZkGxTIrFrD Cgr1EpRULpN+HrEelHHh7uR0twAjvwcyXkG9DbDJXnw8HzKGR80ik4+13HDxx4mDxOY4NOvWSSiM kEFS2Z2AKtxXSMBQZHazAUvbka27c1m93/QsjnDF+P6Aef9NEvUDL9mU9Jbf/+5V+anT2KdPGP4p rQ9gA/Nup61qxDkwc+RupiXD5NSbAgMBAAGjggGzMIIBrzAfBgNVHSMEGDAWgBSCr2yM+MX+lmF8 6B89K3FIXsSLwDAdBgNVHQ4EFgQUjwe7n1zvxFkTeCUYWrsaJpOGP14wDgYDVR0PAQH/BAQDAgWg MAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMEYGA1UdIAQ/MD0w OwYMKwYBBAGyMQECAQMFMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQv Q1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNs aWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgYsGCCsGAQUFBwEBBH8wfTBV BggrBgEFBQcwAoZJaHR0cDovL2NydC5jb21vZG9jYS5jb20vQ09NT0RPUlNBQ2xpZW50QXV0aGVu dGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29t b2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQC85hVlqTVwt218IJR/WjMiMnDtZ7hY860XKjzO uB3sUUQwHxHj+ZYuMbAfVLZGGqh1EekbwDMVgkK9cezIHM+ZzxrNGX2SJyl1YW+3FLn52P0uIlmA VPFjUowf5qBhOHl2NJo+WXYZhQY7rT/xSygE81o3oLE/A4zO6WtO3PeZpFpZNrBvizAsjTDfPeXW iQzXz6NLrgwert0Wml95ov2rG5oCzHYPijabubSNm2NdUjPRtcVylcqAThXOvp6X4UvW8/L0uhkp 9WsKP2JEJ3Zukv7Ib+vMBsdE4tf4rmv89pQC+lLpD08ze/QDCIeFBCRIihcC2PycDQrnNIp1RAIh MYIDyjCCA8YCAQEwga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0 ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQD EzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEA q2wFIs+rCK6H6/2jbblXhDANBglghkgBZQMEAgEFAKCCAe0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMjAxMTIwMTAxOTIwWjAvBgkqhkiG9w0BCQQxIgQgARJXMRmF AJuhAxnmZsBooso4oJXlvZqay5w3giiVAnYwgb4GCSsGAQQBgjcQBDGBsDCBrTCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCrbAUiz6sIrofr/aNtuVeEMIHABgsqhkiG 9w0BCRACCzGBsKCBrTCBlzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3Rl cjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMT NENPTU9ETyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCr bAUiz6sIrofr/aNtuVeEMA0GCSqGSIb3DQEBAQUABIIBAH9Lr0L/RH9XVDl+S76Re0z9P0EggwnB b2lNN1HImaWChUb8mQGT7XEmyqNEyAw1W2t2LeDfaBUJgc0hzwqlzjsV2ToiEaz0y3ORtXeoO+Yb D0nKrHQVPFXrB1NDUzQQBQERMipFxqrX8BQMdfgZh/XUarlbJN1AvRnSNNjfbSvCz4T26PgaqSXG vI1Wf7JZeHYqekrJtoUqvVXhLn36/T2x4ouq7Q/fNsosgDk2OrHcbR9vGFOgAXFmVTSJBVTBx2QJ 4K5ptyzq7rPY/qG4MmRly7E4NCNNj5nDdsDNIBuhVWhGGF2WDw5pyhZfXbvhavZr7pZPk/5Jn1o9 bDfTLzQAAAAAAAA= --Apple-Mail=_6650814F-BC76-4F7A-AA21-D2A2261A4D96-- From owner-freebsd-stable@freebsd.org Fri Nov 20 10:30:42 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6D87C2C4CA2 for ; Fri, 20 Nov 2020 10:30:42 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cct9Z2nSyz4sFM; Fri, 20 Nov 2020 10:30:42 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id 24A0DD359; Fri, 20 Nov 2020 10:30:42 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id CAF6F4AE6C; Fri, 20 Nov 2020 11:30:38 +0100 (CET) From: "Kristof Provost" To: peter.blok@bsd4all.org Cc: "FreeBSD Stable" Subject: Re: Commit 367705+367706 causes a pabic Date: Fri, 20 Nov 2020 11:30:37 +0100 X-Mailer: MailMate (1.13.2r5673) Message-ID: <1753B4A3-2FFC-47A5-9D0C-DC0B71BA22E8@FreeBSD.org> In-Reply-To: References: MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2020 10:30:42 -0000 On 20 Nov 2020, at 11:18, peter.blok@bsd4all.org wrote: > I’m afraid the last Epoch fix for bridge is not solving the problem > ( or perhaps creates a new ). > We’re talking about the stable/12 branch, right? > This seems to happen when the jail epair is added to the bridge. > There must be something more to it than that. I’ve run the bridge tests on stable/12 without issue, and this is a problem we didn’t see when the bridge epochification initially went into stable/12. Do you have a custom kernel config? Other patches? What exact commands do you run to trigger the panic? > kernel trap 12 with interrupts disabled > > > Fatal trap 12: page fault while in kernel mode > cpuid = 6; apic id = 06 > fault virtual address = 0xc10 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff80695e76 > stack pointer = 0x28:0xfffffe00bf14e6e0 > frame pointer = 0x28:0xfffffe00bf14e720 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = resume, IOPL = 0 > current process = 1686 (jail) > trap number = 12 > panic: page fault > cpuid = 6 > time = 1605811310 > KDB: stack backtrace: > #0 0xffffffff8069bb85 at kdb_backtrace+0x65 > #1 0xffffffff80650a4b at vpanic+0x17b > #2 0xffffffff806508c3 at panic+0x43 > #3 0xffffffff809d0351 at trap_fatal+0x391 > #4 0xffffffff809d03af at trap_pfault+0x4f > #5 0xffffffff809cf9f6 at trap+0x286 > #6 0xffffffff809a98c8 at calltrap+0x8 > #7 0xffffffff80368a8d at ck_epoch_synchronize_wait+0x8d > #8 0xffffffff80695c8a at epoch_wait_preempt+0xaa > #9 0xffffffff80757d40 at vnet_if_init+0x120 > #10 0xffffffff8078c994 at vnet_alloc+0x114 > #11 0xffffffff8061e3f7 at kern_jail_set+0x1bb7 > #12 0xffffffff80620190 at sys_jail_set+0x40 > #13 0xffffffff809d0f07 at amd64_syscall+0x387 > #14 0xffffffff809aa1ee at fast_syscall_common+0xf8 This panic is rather odd. This isn’t even the bridge code. This is during initial creation of the vnet. I don’t really see how this could even trigger panics. That panic looks as if something corrupted the net_epoch_preempt, by overwriting the epoch->e_epoch. The bridge patches only access this variable through the well-established functions and macros. I see no obvious way that they could corrupt it. Best regards, Kristof From owner-freebsd-stable@freebsd.org Fri Nov 20 11:02:16 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BF9F02C5B95 for ; Fri, 20 Nov 2020 11:02:16 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from smtpq2.tb.mail.iss.as9143.net (smtpq2.tb.mail.iss.as9143.net [212.54.42.165]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cctt045DPz4v0Z; Fri, 20 Nov 2020 11:02:15 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from [212.54.42.134] (helo=smtp10.tb.mail.iss.as9143.net) by smtpq2.tb.mail.iss.as9143.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kg4BC-0000qK-K6; Fri, 20 Nov 2020 12:02:14 +0100 Received: from 82-101-198-11.cable.dynamic.v4.ziggo.nl ([82.101.198.11] helo=wan0.bsd4all.org) by smtp10.tb.mail.iss.as9143.net with esmtp (Exim 4.90_1) (envelope-from ) id 1kg4BC-0005qp-Ee; Fri, 20 Nov 2020 12:02:14 +0100 Received: from newnas.bsd4all.local (localhost [127.0.0.1]) by wan0.bsd4all.org (Postfix) with ESMTP id E6FE511C; Fri, 20 Nov 2020 12:02:52 +0100 (CET) X-Virus-Scanned: amavisd-new at bsd4all.org Received: from wan0.bsd4all.org ([127.0.0.1]) by newnas.bsd4all.local (newnas.bsd4all.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 2OG6dGEULseD; Fri, 20 Nov 2020 12:02:52 +0100 (CET) Received: from mpro.bsd4all.local (mpro.bsd4all.local [192.168.1.65]) by wan0.bsd4all.org (Postfix) with ESMTPSA id 18E48314; Fri, 20 Nov 2020 12:02:52 +0100 (CET) From: Peter Blok Message-Id: <665757BF-DA06-4503-9ACD-8A4630E23FF4@bsd4all.org> Content-Type: multipart/signed; boundary="Apple-Mail=_D4E6FEDC-7B80-4B5B-BF86-174BD15DAD20"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: Commit 367705+367706 causes a pabic Date: Fri, 20 Nov 2020 12:02:16 +0100 In-Reply-To: <1753B4A3-2FFC-47A5-9D0C-DC0B71BA22E8@FreeBSD.org> Cc: FreeBSD Stable To: Kristof Provost References: <1753B4A3-2FFC-47A5-9D0C-DC0B71BA22E8@FreeBSD.org> X-Mailer: Apple Mail (2.3608.120.23.2.4) X-SourceIP: 82.101.198.11 X-Ziggo-spambar: / X-Ziggo-spamscore: 0.0 X-Ziggo-spamreport: CMAE Analysis: v=2.4 cv=HZuq8gI8 c=1 sm=1 tr=0 ts=5fb7a236 a=3epKYzPC7TQzyyYZMm6bHQ==:17 a=nNwsprhYR40A:10 a=6I5d2MoRAAAA:8 a=6Q3WNqvRAAAA:8 a=XkTKGFxqhWCpKb6hJtcA:9 a=QEXdDO2ut3YA:10 a=fn2EAvp5IHDLcdF3:21 a=_W_S_7VecoQA:10 a=g2jUX3k5JNHoXd8hjKIA:9 a=ZVk8-NSrHBgA:10 a=IjZwj45LgO3ly-622nXo:22 a=I8PBwKCn76L9oNdl0isp:22 X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No X-Rspamd-Queue-Id: 4Cctt045DPz4v0Z X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2020 11:02:16 -0000 --Apple-Mail=_D4E6FEDC-7B80-4B5B-BF86-174BD15DAD20 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hi Kristof, This is 12-stable. With the previous bridge epochification that was = backed out my config had a panic too. I don=E2=80=99t have any local modifications. I did a clean rebuild = after removing /usr/obj/usr My kernel is custom - I only have zfs.ko, opensolaris.ko, vmm.ko and = nmdm.ko as modules. Everything else is statically linked. I have removed = all drivers not needed for the hardware at hand. My bridge is between two vlans from the same trunk and the jail epair = devices as well as the bhyve tap devices. The panic happens when the jails are starting. I can try to narrow it down over the weekend and make the crash dump = available for analysis. Previously I had the following crash with 363492 kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid =3D 2; apic id =3D 02 fault virtual address =3D 0xffffffff00000410 fault code =3D supervisor read data, page not present instruction pointer =3D 0x20:0xffffffff80692326 stack pointer =3D 0x28:0xfffffe00c06097b0 frame pointer =3D 0x28:0xfffffe00c06097f0 code segment =3D base 0x0, limit 0xfffff, type 0x1b =3D DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags =3D resume, IOPL =3D 0 current process =3D 2030 (ifconfig) trap number =3D 12 panic: page fault cpuid =3D 2 time =3D 1595683412 KDB: stack backtrace: #0 0xffffffff80698165 at kdb_backtrace+0x65 #1 0xffffffff8064d67b at vpanic+0x17b #2 0xffffffff8064d4f3 at panic+0x43 #3 0xffffffff809cc311 at trap_fatal+0x391 #4 0xffffffff809cc36f at trap_pfault+0x4f #5 0xffffffff809cb9b6 at trap+0x286 #6 0xffffffff809a5b28 at calltrap+0x8 #7 0xffffffff803677fd at ck_epoch_synchronize_wait+0x8d #8 0xffffffff8069213a at epoch_wait_preempt+0xaa #9 0xffffffff807615b7 at ipsec_ioctl+0x3a7 #10 0xffffffff8075274f at ifioctl+0x47f #11 0xffffffff806b5ea7 at kern_ioctl+0x2b7 #12 0xffffffff806b5b4a at sys_ioctl+0xfa #13 0xffffffff809ccec7 at amd64_syscall+0x387 #14 0xffffffff809a6450 at fast_syscall_common+0x101 > On 20 Nov 2020, at 11:30, Kristof Provost wrote: >=20 > On 20 Nov 2020, at 11:18, peter.blok@bsd4all.org = wrote: >> I=E2=80=99m afraid the last Epoch fix for bridge is not solving the = problem ( or perhaps creates a new ). >>=20 > We=E2=80=99re talking about the stable/12 branch, right? >=20 >> This seems to happen when the jail epair is added to the bridge. >>=20 > There must be something more to it than that. I=E2=80=99ve run the = bridge tests on stable/12 without issue, and this is a problem we = didn=E2=80=99t see when the bridge epochification initially went into = stable/12. >=20 > Do you have a custom kernel config? Other patches? What exact commands = do you run to trigger the panic? >=20 >> kernel trap 12 with interrupts disabled >>=20 >>=20 >> Fatal trap 12: page fault while in kernel mode >> cpuid =3D 6; apic id =3D 06 >> fault virtual address =3D 0xc10 >> fault code =3D supervisor read data, page not present >> instruction pointer =3D 0x20:0xffffffff80695e76 >> stack pointer =3D 0x28:0xfffffe00bf14e6e0 >> frame pointer =3D 0x28:0xfffffe00bf14e720 >> code segment =3D base 0x0, limit 0xfffff, type 0x1b >> =3D DPL 0, pres 1, long 1, def32 0, gran 1 >> processor eflags =3D resume, IOPL =3D 0 >> current process =3D 1686 (jail) >> trap number =3D 12 >> panic: page fault >> cpuid =3D 6 >> time =3D 1605811310 >> KDB: stack backtrace: >> #0 0xffffffff8069bb85 at kdb_backtrace+0x65 >> #1 0xffffffff80650a4b at vpanic+0x17b >> #2 0xffffffff806508c3 at panic+0x43 >> #3 0xffffffff809d0351 at trap_fatal+0x391 >> #4 0xffffffff809d03af at trap_pfault+0x4f >> #5 0xffffffff809cf9f6 at trap+0x286 >> #6 0xffffffff809a98c8 at calltrap+0x8 >> #7 0xffffffff80368a8d at ck_epoch_synchronize_wait+0x8d >> #8 0xffffffff80695c8a at epoch_wait_preempt+0xaa >> #9 0xffffffff80757d40 at vnet_if_init+0x120 >> #10 0xffffffff8078c994 at vnet_alloc+0x114 >> #11 0xffffffff8061e3f7 at kern_jail_set+0x1bb7 >> #12 0xffffffff80620190 at sys_jail_set+0x40 >> #13 0xffffffff809d0f07 at amd64_syscall+0x387 >> #14 0xffffffff809aa1ee at fast_syscall_common+0xf8 >=20 > This panic is rather odd. This isn=E2=80=99t even the bridge code. = This is during initial creation of the vnet. I don=E2=80=99t really see = how this could even trigger panics. > That panic looks as if something corrupted the net_epoch_preempt, by = overwriting the epoch->e_epoch. The bridge patches only access this = variable through the well-established functions and macros. I see no = obvious way that they could corrupt it. >=20 > Best regards, > Kristof --Apple-Mail=_D4E6FEDC-7B80-4B5B-BF86-174BD15DAD20 Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCBSAw ggUcMIIEBKADAgECAhEAq2wFIs+rCK6H6/2jbblXhDANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgwNDE0MDAwMDAwWhcNMjEwNDEzMjM1 OTU5WjBEMQswCQYDVQQGEwJOTDETMBEGA1UEAxMKUGV0ZXIgQmxvazEgMB4GCSqGSIb3DQEJARYR cGJsb2tAYnNkNGFsbC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPT/3evs2a zLSIVepGa9qFVcSISd5HzoJt9xAyQ4od7NM6Qzwm446OyhzWsIN/a6+nDNB4AxzSg00QXKx4afEa FrdLzmREEfv24f88j2UZYqHAls0j26jyED5FZ068xs4gWZBG2U7EVTUNNJuUrrmqBNZkGxTIrFrD Cgr1EpRULpN+HrEelHHh7uR0twAjvwcyXkG9DbDJXnw8HzKGR80ik4+13HDxx4mDxOY4NOvWSSiM kEFS2Z2AKtxXSMBQZHazAUvbka27c1m93/QsjnDF+P6Aef9NEvUDL9mU9Jbf/+5V+anT2KdPGP4p rQ9gA/Nup61qxDkwc+RupiXD5NSbAgMBAAGjggGzMIIBrzAfBgNVHSMEGDAWgBSCr2yM+MX+lmF8 6B89K3FIXsSLwDAdBgNVHQ4EFgQUjwe7n1zvxFkTeCUYWrsaJpOGP14wDgYDVR0PAQH/BAQDAgWg MAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMEYGA1UdIAQ/MD0w OwYMKwYBBAGyMQECAQMFMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQv Q1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNs aWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgYsGCCsGAQUFBwEBBH8wfTBV BggrBgEFBQcwAoZJaHR0cDovL2NydC5jb21vZG9jYS5jb20vQ09NT0RPUlNBQ2xpZW50QXV0aGVu dGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29t b2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQC85hVlqTVwt218IJR/WjMiMnDtZ7hY860XKjzO uB3sUUQwHxHj+ZYuMbAfVLZGGqh1EekbwDMVgkK9cezIHM+ZzxrNGX2SJyl1YW+3FLn52P0uIlmA VPFjUowf5qBhOHl2NJo+WXYZhQY7rT/xSygE81o3oLE/A4zO6WtO3PeZpFpZNrBvizAsjTDfPeXW iQzXz6NLrgwert0Wml95ov2rG5oCzHYPijabubSNm2NdUjPRtcVylcqAThXOvp6X4UvW8/L0uhkp 9WsKP2JEJ3Zukv7Ib+vMBsdE4tf4rmv89pQC+lLpD08ze/QDCIeFBCRIihcC2PycDQrnNIp1RAIh MYIDyjCCA8YCAQEwga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0 ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQD EzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEA q2wFIs+rCK6H6/2jbblXhDANBglghkgBZQMEAgEFAKCCAe0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMjAxMTIwMTEwMjE2WjAvBgkqhkiG9w0BCQQxIgQgkoj7KDrS tF64PFkyAH79LZTJPVnVFDuWu8BfBmv8JHowgb4GCSsGAQQBgjcQBDGBsDCBrTCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCrbAUiz6sIrofr/aNtuVeEMIHABgsqhkiG 9w0BCRACCzGBsKCBrTCBlzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3Rl cjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMT NENPTU9ETyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCr bAUiz6sIrofr/aNtuVeEMA0GCSqGSIb3DQEBAQUABIIBAGud4BgzeYAF7/TbwvKXS2emx+F4/Von +ghazxpbvcPBhTvAdrNXSSUVkiD8jrWS5EmBDQXHHad6NsfYOB7r+crXCneGFaJ60J4qTYf6Ev5D YoZ2fGbsEieC8mPHwuQ52RrnGKMECbRD8iRPp2dgdmuw80ykkDsh/wxZFwtS37Kg+HUspxlmwb0y g24cpU16LJ3kKjxqcynvSeEs6CqZ30dEehq6V8GbdMP45lt4awP8PfugSZj75WTHmKAzkmzMP0lP didcqVuYJzAcrKIZpow7Lx8DlqvgCfrsy373sEEGav6o5HJGPbouUDw2CUHbX530bC9J+McaA5C6 Gq5HLyYAAAAAAAA= --Apple-Mail=_D4E6FEDC-7B80-4B5B-BF86-174BD15DAD20-- From owner-freebsd-stable@freebsd.org Fri Nov 20 11:53:36 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7BFB12C72FA for ; Fri, 20 Nov 2020 11:53:36 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Ccw1D20HPz3Gk8; Fri, 20 Nov 2020 11:53:36 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id A4A9EE821; Fri, 20 Nov 2020 11:53:35 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id C50B74B44B; Fri, 20 Nov 2020 12:53:32 +0100 (CET) From: "Kristof Provost" To: "Peter Blok" Cc: "FreeBSD Stable" Subject: Re: Commit 367705+367706 causes a pabic Date: Fri, 20 Nov 2020 12:53:29 +0100 X-Mailer: MailMate (1.13.2r5673) Message-ID: In-Reply-To: <665757BF-DA06-4503-9ACD-8A4630E23FF4@bsd4all.org> References: <1753B4A3-2FFC-47A5-9D0C-DC0B71BA22E8@FreeBSD.org> <665757BF-DA06-4503-9ACD-8A4630E23FF4@bsd4all.org> MIME-Version: 1.0 Embedded-HTML: [{"HTML":[775, 17404], "plain":[399, 4597], "uuid":"74904DD7-503B-4D4B-A800-4DC3AB57CB61"}] Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2020 11:53:36 -0000 Can you share your kernel config file (and src.conf / make.conf if they exist)? This second panic is in the IPSec code. My current thinking is that your kernel config is triggering a bug that’s manifesting in multiple places, but not actually caused by those places. I’d like to be able to reproduce it so we can debug it. Best regards, Kristof On 20 Nov 2020, at 12:02, Peter Blok wrote: > Hi Kristof, > > This is 12-stable. With the previous bridge epochification that was > backed out my config had a panic too. > > I don’t have any local modifications. I did a clean rebuild after > removing /usr/obj/usr > > My kernel is custom - I only have zfs.ko, opensolaris.ko, vmm.ko and > nmdm.ko as modules. Everything else is statically linked. I have > removed all drivers not needed for the hardware at hand. > > My bridge is between two vlans from the same trunk and the jail epair > devices as well as the bhyve tap devices. > > The panic happens when the jails are starting. > > I can try to narrow it down over the weekend and make the crash dump > available for analysis. > > Previously I had the following crash with 363492 > > kernel trap 12 with interrupts disabled > > > Fatal trap 12: page fault while in kernel mode > cpuid = 2; apic id = 02 > fault virtual address = 0xffffffff00000410 > fault code = supervisor read data, page not present > instruction pointer = 0x20:0xffffffff80692326 > stack pointer = 0x28:0xfffffe00c06097b0 > frame pointer = 0x28:0xfffffe00c06097f0 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = resume, IOPL = 0 > current process = 2030 (ifconfig) > trap number = 12 > panic: page fault > cpuid = 2 > time = 1595683412 > KDB: stack backtrace: > #0 0xffffffff80698165 at kdb_backtrace+0x65 > #1 0xffffffff8064d67b at vpanic+0x17b > #2 0xffffffff8064d4f3 at panic+0x43 > #3 0xffffffff809cc311 at trap_fatal+0x391 > #4 0xffffffff809cc36f at trap_pfault+0x4f > #5 0xffffffff809cb9b6 at trap+0x286 > #6 0xffffffff809a5b28 at calltrap+0x8 > #7 0xffffffff803677fd at ck_epoch_synchronize_wait+0x8d > #8 0xffffffff8069213a at epoch_wait_preempt+0xaa > #9 0xffffffff807615b7 at ipsec_ioctl+0x3a7 > #10 0xffffffff8075274f at ifioctl+0x47f > #11 0xffffffff806b5ea7 at kern_ioctl+0x2b7 > #12 0xffffffff806b5b4a at sys_ioctl+0xfa > #13 0xffffffff809ccec7 at amd64_syscall+0x387 > #14 0xffffffff809a6450 at fast_syscall_common+0x101 > > > > >> On 20 Nov 2020, at 11:30, Kristof Provost wrote: >> >> On 20 Nov 2020, at 11:18, peter.blok@bsd4all.org >> wrote: >>> I’m afraid the last Epoch fix for bridge is not solving the >>> problem ( or perhaps creates a new ). >>> >> We’re talking about the stable/12 branch, right? >> >>> This seems to happen when the jail epair is added to the bridge. >>> >> There must be something more to it than that. I’ve run the bridge >> tests on stable/12 without issue, and this is a problem we didn’t >> see when the bridge epochification initially went into stable/12. >> >> Do you have a custom kernel config? Other patches? What exact >> commands do you run to trigger the panic? >> >>> kernel trap 12 with interrupts disabled >>> >>> >>> Fatal trap 12: page fault while in kernel mode >>> cpuid = 6; apic id = 06 >>> fault virtual address = 0xc10 >>> fault code = supervisor read data, page not present >>> instruction pointer = 0x20:0xffffffff80695e76 >>> stack pointer = 0x28:0xfffffe00bf14e6e0 >>> frame pointer = 0x28:0xfffffe00bf14e720 >>> code segment = base 0x0, limit 0xfffff, type 0x1b >>> = DPL 0, pres 1, long 1, def32 0, gran 1 >>> processor eflags = resume, IOPL = 0 >>> current process = 1686 (jail) >>> trap number = 12 >>> panic: page fault >>> cpuid = 6 >>> time = 1605811310 >>> KDB: stack backtrace: >>> #0 0xffffffff8069bb85 at kdb_backtrace+0x65 >>> #1 0xffffffff80650a4b at vpanic+0x17b >>> #2 0xffffffff806508c3 at panic+0x43 >>> #3 0xffffffff809d0351 at trap_fatal+0x391 >>> #4 0xffffffff809d03af at trap_pfault+0x4f >>> #5 0xffffffff809cf9f6 at trap+0x286 >>> #6 0xffffffff809a98c8 at calltrap+0x8 >>> #7 0xffffffff80368a8d at ck_epoch_synchronize_wait+0x8d >>> #8 0xffffffff80695c8a at epoch_wait_preempt+0xaa >>> #9 0xffffffff80757d40 at vnet_if_init+0x120 >>> #10 0xffffffff8078c994 at vnet_alloc+0x114 >>> #11 0xffffffff8061e3f7 at kern_jail_set+0x1bb7 >>> #12 0xffffffff80620190 at sys_jail_set+0x40 >>> #13 0xffffffff809d0f07 at amd64_syscall+0x387 >>> #14 0xffffffff809aa1ee at fast_syscall_common+0xf8 >> >> This panic is rather odd. This isn’t even the bridge code. This is >> during initial creation of the vnet. I don’t really see how this >> could even trigger panics. >> That panic looks as if something corrupted the net_epoch_preempt, by >> overwriting the epoch->e_epoch. The bridge patches only access this >> variable through the well-established functions and macros. I see no >> obvious way that they could corrupt it. >> >> Best regards, >> Kristof From owner-freebsd-stable@freebsd.org Fri Nov 20 13:53:25 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 6E4302EB75A for ; Fri, 20 Nov 2020 13:53:25 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from smtpq1.tb.mail.iss.as9143.net (smtpq1.tb.mail.iss.as9143.net [212.54.42.164]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CcygS6K4Xz3Ph1; Fri, 20 Nov 2020 13:53:24 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from [212.54.42.135] (helo=smtp11.tb.mail.iss.as9143.net) by smtpq1.tb.mail.iss.as9143.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kg6qo-00024P-DP; Fri, 20 Nov 2020 14:53:22 +0100 Received: from 82-101-198-11.cable.dynamic.v4.ziggo.nl ([82.101.198.11] helo=wan0.bsd4all.org) by smtp11.tb.mail.iss.as9143.net with esmtp (Exim 4.90_1) (envelope-from ) id 1kg6qo-0008L4-1H; Fri, 20 Nov 2020 14:53:22 +0100 Received: from newnas.bsd4all.local (localhost [127.0.0.1]) by wan0.bsd4all.org (Postfix) with ESMTP id 13B2B31B; Fri, 20 Nov 2020 14:54:00 +0100 (CET) X-Virus-Scanned: amavisd-new at bsd4all.org Received: from wan0.bsd4all.org ([127.0.0.1]) by newnas.bsd4all.local (newnas.bsd4all.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3ZENyZVZevhZ; Fri, 20 Nov 2020 14:53:58 +0100 (CET) Received: from mpro.bsd4all.local (mpro.bsd4all.local [192.168.1.65]) by wan0.bsd4all.org (Postfix) with ESMTPSA id B63F738F; Fri, 20 Nov 2020 14:53:58 +0100 (CET) From: Peter Blok Message-Id: <5DA2A6D4-1FDA-48B3-A319-F26CB802E01B@bsd4all.org> Content-Type: multipart/signed; boundary="Apple-Mail=_A2B34ADC-6DE9-4C15-A605-0C85F060E27E"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: Commit 367705+367706 causes a pabic Date: Fri, 20 Nov 2020 14:53:23 +0100 In-Reply-To: Cc: FreeBSD Stable To: Kristof Provost References: <1753B4A3-2FFC-47A5-9D0C-DC0B71BA22E8@FreeBSD.org> <665757BF-DA06-4503-9ACD-8A4630E23FF4@bsd4all.org> X-Mailer: Apple Mail (2.3608.120.23.2.4) X-SourceIP: 82.101.198.11 X-Ziggo-spambar: / X-Ziggo-spamscore: 0.0 X-Ziggo-spamreport: CMAE Analysis: v=2.4 cv=RMMNo6u+ c=1 sm=1 tr=0 ts=5fb7ca52 a=3epKYzPC7TQzyyYZMm6bHQ==:17 a=nNwsprhYR40A:10 a=6I5d2MoRAAAA:8 a=6Q3WNqvRAAAA:8 a=5JVGq2EulSsSO7zaG_kA:9 a=QEXdDO2ut3YA:10 a=g2jUX3k5JNHoXd8hjKIA:9 a=ZVk8-NSrHBgA:10 a=IjZwj45LgO3ly-622nXo:22 a=I8PBwKCn76L9oNdl0isp:22 X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No X-Rspamd-Queue-Id: 4CcygS6K4Xz3Ph1 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-4.00 / 15.00]; REPLY(-4.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2020 13:53:25 -0000 --Apple-Mail=_A2B34ADC-6DE9-4C15-A605-0C85F060E27E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 The panic with ipsec code in the backtrace was already very strange. I = was using IPsec, but only on one interface totally separate from the = members of the bridge as well as the bridge itself. The jails were not = doing any ipsec as well. Note that panic was a while ago and it was = after the 1st bridge epochification was done on stable-12 which was = later backed out Today the system is no longer using ipsec, but it is still compiled in. = I can remove it if need be for a test src.conf WITHOUT_KERBEROS=3Dyes WITHOUT_GSSAPI=3Dyes WITHOUT_SENDMAIL=3Dtrue WITHOUT_MAILWRAPPER=3Dtrue WITHOUT_DMAGENT=3Dtrue WITHOUT_GAMES=3Dtrue WITHOUT_IPFILTER=3Dtrue WITHOUT_UNBOUND=3Dtrue WITHOUT_PROFILE=3Dtrue WITHOUT_ATM=3Dtrue WITHOUT_BSNMP=3Dtrue #WITHOUT_CROSS_COMPILER=3Dtrue WITHOUT_DEBUG_FILES=3Dtrue WITHOUT_DICT=3Dtrue WITHOUT_FLOPPY=3Dtrue WITHOUT_HTML=3Dtrue WITHOUT_HYPERV=3Dtrue WITHOUT_NDIS=3Dtrue WITHOUT_NIS=3Dtrue WITHOUT_PPP=3Dtrue WITHOUT_TALK=3Dtrue WITHOUT_TESTS=3Dtrue WITHOUT_WIRELESS=3Dtrue #WITHOUT_LIB32=3Dtrue WITHOUT_LPR=3Dtrue make.conf KERNCONF=3DBHYVE MODULES_OVERRIDE=3Dopensolaris dtrace zfs vmm nmdm if_bridge bridgestp = if_vxlan pflog libmchain libiconv smbfs linux linux64 linux_common = linuxkpi linprocfs linsysfs ext2fs DEFAULT_VERSIONS+=3Dperl5=3D5.30 mysql=3D5.7 python=3D3.8 python3=3D3.8 OPTIONS_UNSET=3DDOCS NLS MANPAGES BHYVE cpu HAMMER ident BHYVE makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug = symbols makeoptions WITH_CTF=3D1 # Run ctfconvert(1) for DTrace = support options CAMDEBUG options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread = preemption options INET # InterNETworking options INET6 # IPv6 communications protocols options IPSEC options TCP_OFFLOAD # TCP offload options TCP_RFC7413 # TCP FASTOPEN options SCTP # Stream Control Transmission = Protocol options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates = support options UFS_ACL # Support for access control = lists options UFS_DIRHASH # Improve performance on big = directories options UFS_GJOURNAL # Enable gjournal-based UFS = journaling options QUOTA # Enable disk quotas for UFS options SUIDDIR options NFSCL # Network Filesystem Client options NFSD # Network Filesystem Server options NFSLOCKD # Network Lock Manager options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options FUSEFS options NULLFS # NULL filesystem options UNIONFS options FDESCFS # File descriptor filesystem options PROCFS # Process filesystem (requires = PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_RAID # Soft RAID functionality. options GEOM_LABEL # Provides labelization options GEOM_ELI # Disk encryption. options COMPAT_FREEBSD32 # Compatible with i386 binaries options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options COMPAT_FREEBSD6 # Compatible with FreeBSD6 options COMPAT_FREEBSD7 # Compatible with FreeBSD7 options COMPAT_FREEBSD9 # Compatible with FreeBSD9 options COMPAT_FREEBSD10 # Compatible with FreeBSD10 options COMPAT_FREEBSD11 # Compatible with FreeBSD11 options SCSI_DELAY=3D5000 # Delay (in ms) before = probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time = extensions options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being = interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for = hwpmc(4) options AUDIT # Security event auditing options CAPABILITY_MODE # Capsicum capability mode options CAPABILITIES # Capsicum capabilities options MAC # TrustedBSD MAC Framework options MAC_PORTACL options MAC_NTPD options KDTRACE_FRAME # Ensure frames are compiled in options KDTRACE_HOOKS # Kernel DTrace hooks options DDB_CTF # Kernel ELF linker loads CTF = data options INCLUDE_CONFIG_FILE # Include this file in kernel # Debugging support. Always need this: options KDB # Enable kernel debugger = support. options KDB_TRACE # Print a stack trace for a = panic. options KDB_UNATTENDED # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcessor = Kernel options EARLY_AP_STARTUP # CPU frequency control device cpufreq device cpuctl device coretemp # Bus support. device acpi options ACPI_DMAR device pci options PCI_IOV # PCI SR-IOV support device iicbus device iicbb device iic device ic device iicsmb device ichsmb device smbus device smb #device jedec_dimm # ATA controllers device ahci # AHCI-compatible SATA = controllers device mvs # Marvell = 88SX50XX/88SX60XX/88SX70XX/SoC SATA # SCSI Controllers device mps # LSI-Logic MPT-Fusion 2 # ATA/SCSI peripherals device scbus # SCSI bus (required for = ATA/SCSI) device da # Direct Access (disks) device cd # CD device pass # Passthrough device (direct = ATA/SCSI access) device ses # Enclosure Services (SES and = SAF-TE) device sg device cfiscsi device ctl # CAM Target Layer device iscsi # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer # vt is the new video console driver device vt device vt_vga device vt_efifb # Serial (COM) ports device uart # Generic UART driver # PCI/PCI-X/PCIe Ethernet NICs that use iflib infrastructure device iflib device em # Intel PRO/1000 Gigabit = Ethernet Family device ix # Intel PRO/10GbE PCIE PF = Ethernet # Network stack virtualization. options VIMAGE # Pseudo devices. device crypto device cryptodev device loop # Network loopback device random # Entropy device device padlock_rng # VIA Padlock RNG device rdrand_rng # Intel Bull Mountain RNG device ipmi device smbios device vpd device aesni # AES-NI OpenCrypto module device ether # Ethernet support device lagg device vlan # 802.1Q VLAN support device tuntap # Packet tunnel. device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device firmware # firmware assist module device pf #device pflog #device pfsync # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter # The `epair' device implements a virtual back-to-back connected = Ethernet # like interface pair. device epair # USB support options USB_DEBUG # enable debug msgs device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB = 2.0) device xhci # XHCI PCI->USB interface (USB = 3.0) device usb # USB Bus (required) device uhid device ukbd # Keyboard device umass # Disks/Mass storage - Requires = scbus and da device ums device filemon device if_bridge > On 20 Nov 2020, at 12:53, Kristof Provost wrote: >=20 > Can you share your kernel config file (and src.conf / make.conf if = they exist)? >=20 > This second panic is in the IPSec code. My current thinking is that = your kernel config is triggering a bug that=E2=80=99s manifesting in = multiple places, but not actually caused by those places. >=20 > I=E2=80=99d like to be able to reproduce it so we can debug it. >=20 > Best regards, > Kristof >=20 > On 20 Nov 2020, at 12:02, Peter Blok wrote: >> Hi Kristof, >>=20 >> This is 12-stable. With the previous bridge epochification that was = backed out my config had a panic too. >>=20 >> I don=E2=80=99t have any local modifications. I did a clean rebuild = after removing /usr/obj/usr >>=20 >> My kernel is custom - I only have zfs.ko, opensolaris.ko, vmm.ko and = nmdm.ko as modules. Everything else is statically linked. I have removed = all drivers not needed for the hardware at hand. >>=20 >> My bridge is between two vlans from the same trunk and the jail epair = devices as well as the bhyve tap devices. >>=20 >> The panic happens when the jails are starting. >>=20 >> I can try to narrow it down over the weekend and make the crash dump = available for analysis. >>=20 >> Previously I had the following crash with 363492 >>=20 >> kernel trap 12 with interrupts disabled >>=20 >>=20 >> Fatal trap 12: page fault while in kernel mode >> cpuid =3D 2; apic id =3D 02 >> fault virtual address =3D 0xffffffff00000410 >> fault code =3D supervisor read data, page not present >> instruction pointer =3D 0x20:0xffffffff80692326 >> stack pointer =3D 0x28:0xfffffe00c06097b0 >> frame pointer =3D 0x28:0xfffffe00c06097f0 >> code segment =3D base 0x0, limit 0xfffff, type 0x1b >> =3D DPL 0, pres 1, long 1, def32 0, gran 1 >> processor eflags =3D resume, IOPL =3D 0 >> current process =3D 2030 (ifconfig) >> trap number =3D 12 >> panic: page fault >> cpuid =3D 2 >> time =3D 1595683412 >> KDB: stack backtrace: >> #0 0xffffffff80698165 at kdb_backtrace+0x65 >> #1 0xffffffff8064d67b at vpanic+0x17b >> #2 0xffffffff8064d4f3 at panic+0x43 >> #3 0xffffffff809cc311 at trap_fatal+0x391 >> #4 0xffffffff809cc36f at trap_pfault+0x4f >> #5 0xffffffff809cb9b6 at trap+0x286 >> #6 0xffffffff809a5b28 at calltrap+0x8 >> #7 0xffffffff803677fd at ck_epoch_synchronize_wait+0x8d >> #8 0xffffffff8069213a at epoch_wait_preempt+0xaa >> #9 0xffffffff807615b7 at ipsec_ioctl+0x3a7 >> #10 0xffffffff8075274f at ifioctl+0x47f >> #11 0xffffffff806b5ea7 at kern_ioctl+0x2b7 >> #12 0xffffffff806b5b4a at sys_ioctl+0xfa >> #13 0xffffffff809ccec7 at amd64_syscall+0x387 >> #14 0xffffffff809a6450 at fast_syscall_common+0x101 >>=20 >>=20 >>=20 >>=20 >>> On 20 Nov 2020, at 11:30, Kristof Provost wrote: >>>=20 >>> On 20 Nov 2020, at 11:18, peter.blok@bsd4all.org = wrote: >>>> I=E2=80=99m afraid the last Epoch fix for bridge is not solving the = problem ( or perhaps creates a new ). >>>>=20 >>> We=E2=80=99re talking about the stable/12 branch, right? >>>=20 >>>> This seems to happen when the jail epair is added to the bridge. >>>>=20 >>> There must be something more to it than that. I=E2=80=99ve run the = bridge tests on stable/12 without issue, and this is a problem we = didn=E2=80=99t see when the bridge epochification initially went into = stable/12. >>>=20 >>> Do you have a custom kernel config? Other patches? What exact = commands do you run to trigger the panic? >>>=20 >>>> kernel trap 12 with interrupts disabled >>>>=20 >>>>=20 >>>> Fatal trap 12: page fault while in kernel mode >>>> cpuid =3D 6; apic id =3D 06 >>>> fault virtual address =3D 0xc10 >>>> fault code =3D supervisor read data, page not present >>>> instruction pointer =3D 0x20:0xffffffff80695e76 >>>> stack pointer =3D 0x28:0xfffffe00bf14e6e0 >>>> frame pointer =3D 0x28:0xfffffe00bf14e720 >>>> code segment =3D base 0x0, limit 0xfffff, type 0x1b >>>> =3D DPL 0, pres 1, long 1, def32 0, gran 1 >>>> processor eflags =3D resume, IOPL =3D 0 >>>> current process =3D 1686 (jail) >>>> trap number =3D 12 >>>> panic: page fault >>>> cpuid =3D 6 >>>> time =3D 1605811310 >>>> KDB: stack backtrace: >>>> #0 0xffffffff8069bb85 at kdb_backtrace+0x65 >>>> #1 0xffffffff80650a4b at vpanic+0x17b >>>> #2 0xffffffff806508c3 at panic+0x43 >>>> #3 0xffffffff809d0351 at trap_fatal+0x391 >>>> #4 0xffffffff809d03af at trap_pfault+0x4f >>>> #5 0xffffffff809cf9f6 at trap+0x286 >>>> #6 0xffffffff809a98c8 at calltrap+0x8 >>>> #7 0xffffffff80368a8d at ck_epoch_synchronize_wait+0x8d >>>> #8 0xffffffff80695c8a at epoch_wait_preempt+0xaa >>>> #9 0xffffffff80757d40 at vnet_if_init+0x120 >>>> #10 0xffffffff8078c994 at vnet_alloc+0x114 >>>> #11 0xffffffff8061e3f7 at kern_jail_set+0x1bb7 >>>> #12 0xffffffff80620190 at sys_jail_set+0x40 >>>> #13 0xffffffff809d0f07 at amd64_syscall+0x387 >>>> #14 0xffffffff809aa1ee at fast_syscall_common+0xf8 >>>=20 >>> This panic is rather odd. This isn=E2=80=99t even the bridge code. = This is during initial creation of the vnet. I don=E2=80=99t really see = how this could even trigger panics. >>> That panic looks as if something corrupted the net_epoch_preempt, by = overwriting the epoch->e_epoch. The bridge patches only access this = variable through the well-established functions and macros. I see no = obvious way that they could corrupt it. >>>=20 >>> Best regards, >>> Kristof >=20 >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" --Apple-Mail=_A2B34ADC-6DE9-4C15-A605-0C85F060E27E Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCBSAw ggUcMIIEBKADAgECAhEAq2wFIs+rCK6H6/2jbblXhDANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgwNDE0MDAwMDAwWhcNMjEwNDEzMjM1 OTU5WjBEMQswCQYDVQQGEwJOTDETMBEGA1UEAxMKUGV0ZXIgQmxvazEgMB4GCSqGSIb3DQEJARYR cGJsb2tAYnNkNGFsbC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPT/3evs2a zLSIVepGa9qFVcSISd5HzoJt9xAyQ4od7NM6Qzwm446OyhzWsIN/a6+nDNB4AxzSg00QXKx4afEa FrdLzmREEfv24f88j2UZYqHAls0j26jyED5FZ068xs4gWZBG2U7EVTUNNJuUrrmqBNZkGxTIrFrD Cgr1EpRULpN+HrEelHHh7uR0twAjvwcyXkG9DbDJXnw8HzKGR80ik4+13HDxx4mDxOY4NOvWSSiM kEFS2Z2AKtxXSMBQZHazAUvbka27c1m93/QsjnDF+P6Aef9NEvUDL9mU9Jbf/+5V+anT2KdPGP4p rQ9gA/Nup61qxDkwc+RupiXD5NSbAgMBAAGjggGzMIIBrzAfBgNVHSMEGDAWgBSCr2yM+MX+lmF8 6B89K3FIXsSLwDAdBgNVHQ4EFgQUjwe7n1zvxFkTeCUYWrsaJpOGP14wDgYDVR0PAQH/BAQDAgWg MAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMEYGA1UdIAQ/MD0w OwYMKwYBBAGyMQECAQMFMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQv Q1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNs aWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgYsGCCsGAQUFBwEBBH8wfTBV BggrBgEFBQcwAoZJaHR0cDovL2NydC5jb21vZG9jYS5jb20vQ09NT0RPUlNBQ2xpZW50QXV0aGVu dGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29t b2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQC85hVlqTVwt218IJR/WjMiMnDtZ7hY860XKjzO uB3sUUQwHxHj+ZYuMbAfVLZGGqh1EekbwDMVgkK9cezIHM+ZzxrNGX2SJyl1YW+3FLn52P0uIlmA VPFjUowf5qBhOHl2NJo+WXYZhQY7rT/xSygE81o3oLE/A4zO6WtO3PeZpFpZNrBvizAsjTDfPeXW iQzXz6NLrgwert0Wml95ov2rG5oCzHYPijabubSNm2NdUjPRtcVylcqAThXOvp6X4UvW8/L0uhkp 9WsKP2JEJ3Zukv7Ib+vMBsdE4tf4rmv89pQC+lLpD08ze/QDCIeFBCRIihcC2PycDQrnNIp1RAIh MYIDyjCCA8YCAQEwga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0 ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQD EzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEA q2wFIs+rCK6H6/2jbblXhDANBglghkgBZQMEAgEFAKCCAe0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMjAxMTIwMTM1MzIzWjAvBgkqhkiG9w0BCQQxIgQgDz2ZdeTh aYMkkQg+6GjrYvfLjWgNbIMXZXTfv7auV6cwgb4GCSsGAQQBgjcQBDGBsDCBrTCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCrbAUiz6sIrofr/aNtuVeEMIHABgsqhkiG 9w0BCRACCzGBsKCBrTCBlzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3Rl cjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMT NENPTU9ETyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCr bAUiz6sIrofr/aNtuVeEMA0GCSqGSIb3DQEBAQUABIIBAHX2f6lsci746bTdjY6DvGOZVPq0ia2X iY1yO94e5Z6EUAU1gV7xSiysUef9NhNU49vPlQBwMutDE1RWQ+OSMtYaSPlYMpLMQXxZDkIjLbh7 P3CXItXaT/2/MAvTx0Lbq8RsSfQTaKgnxqwLg9OnsWQjeIpeSgVgPOnYGFvzPgV4CfYrhLdiFYI/ +izslPjfgLpmvq1JtGcVjwFEXGMKKqI+7pGN/1iuMeDQain8ovB7SS+ZZsWG4P2kvIXmcaVdDLm4 l3xVfJcIhMjtTZ9QeEjBtDCpeY8oRZPjKLY5alYEbcLIySO/Hv4AWGPMCrnsneGp4Br8T+Idohp3 MpvM2WoAAAAAAAA= --Apple-Mail=_A2B34ADC-6DE9-4C15-A605-0C85F060E27E-- From owner-freebsd-stable@freebsd.org Fri Nov 20 14:53:51 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 439A02ED001 for ; Fri, 20 Nov 2020 14:53:51 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Cd01C1Vfxz3kjN; Fri, 20 Nov 2020 14:53:51 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: from venus.codepro.be (venus.codepro.be [5.9.86.228]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.codepro.be", Issuer "Let's Encrypt Authority X3" (verified OK)) (Authenticated sender: kp) by smtp.freebsd.org (Postfix) with ESMTPSA id BD1D1FB8A; Fri, 20 Nov 2020 14:53:50 +0000 (UTC) (envelope-from kp@FreeBSD.org) Received: by venus.codepro.be (Postfix, authenticated sender kp) id 65ECB4B93B; Fri, 20 Nov 2020 15:53:48 +0100 (CET) From: "Kristof Provost" To: "Peter Blok" Cc: "FreeBSD Stable" Subject: Re: Commit 367705+367706 causes a pabic Date: Fri, 20 Nov 2020 15:53:47 +0100 X-Mailer: MailMate (1.13.2r5673) Message-ID: <5489BF90-E1F6-44C2-ABD5-1C52BABA206D@FreeBSD.org> In-Reply-To: <5DA2A6D4-1FDA-48B3-A319-F26CB802E01B@bsd4all.org> References: <1753B4A3-2FFC-47A5-9D0C-DC0B71BA22E8@FreeBSD.org> <665757BF-DA06-4503-9ACD-8A4630E23FF4@bsd4all.org> <5DA2A6D4-1FDA-48B3-A319-F26CB802E01B@bsd4all.org> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Nov 2020 14:53:51 -0000 I still can’t reproduce that panic. Does it happen immediately after you start a vnet jail? Does it also happen with a GENERIC kernel? Regards, Kristof On 20 Nov 2020, at 14:53, Peter Blok wrote: > The panic with ipsec code in the backtrace was already very strange. I > was using IPsec, but only on one interface totally separate from the > members of the bridge as well as the bridge itself. The jails were not > doing any ipsec as well. Note that panic was a while ago and it was > after the 1st bridge epochification was done on stable-12 which was > later backed out > > Today the system is no longer using ipsec, but it is still compiled > in. I can remove it if need be for a test > > > src.conf > WITHOUT_KERBEROS=yes > WITHOUT_GSSAPI=yes > WITHOUT_SENDMAIL=true > WITHOUT_MAILWRAPPER=true > WITHOUT_DMAGENT=true > WITHOUT_GAMES=true > WITHOUT_IPFILTER=true > WITHOUT_UNBOUND=true > WITHOUT_PROFILE=true > WITHOUT_ATM=true > WITHOUT_BSNMP=true > #WITHOUT_CROSS_COMPILER=true > WITHOUT_DEBUG_FILES=true > WITHOUT_DICT=true > WITHOUT_FLOPPY=true > WITHOUT_HTML=true > WITHOUT_HYPERV=true > WITHOUT_NDIS=true > WITHOUT_NIS=true > WITHOUT_PPP=true > WITHOUT_TALK=true > WITHOUT_TESTS=true > WITHOUT_WIRELESS=true > #WITHOUT_LIB32=true > WITHOUT_LPR=true > > make.conf > KERNCONF=BHYVE > MODULES_OVERRIDE=opensolaris dtrace zfs vmm nmdm if_bridge bridgestp > if_vxlan pflog libmchain libiconv smbfs linux linux64 linux_common > linuxkpi linprocfs linsysfs ext2fs > DEFAULT_VERSIONS+=perl5=5.30 mysql=5.7 python=3.8 python3=3.8 > OPTIONS_UNSET=DOCS NLS MANPAGES > > BHYVE > cpu HAMMER > ident BHYVE > > makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols > makeoptions WITH_CTF=1 # Run ctfconvert(1) for DTrace support > > options CAMDEBUG > > options SCHED_ULE # ULE scheduler > options PREEMPTION # Enable kernel thread preemption > options INET # InterNETworking > options INET6 # IPv6 communications protocols > options IPSEC > options TCP_OFFLOAD # TCP offload > options TCP_RFC7413 # TCP FASTOPEN > options SCTP # Stream Control Transmission Protocol > options FFS # Berkeley Fast Filesystem > options SOFTUPDATES # Enable FFS soft updates support > options UFS_ACL # Support for access control lists > options UFS_DIRHASH # Improve performance on big directories > options UFS_GJOURNAL # Enable gjournal-based UFS journaling > options QUOTA # Enable disk quotas for UFS > options SUIDDIR > options NFSCL # Network Filesystem Client > options NFSD # Network Filesystem Server > options NFSLOCKD # Network Lock Manager > options MSDOSFS # MSDOS Filesystem > options CD9660 # ISO 9660 Filesystem > options FUSEFS > options NULLFS # NULL filesystem > options UNIONFS > options FDESCFS # File descriptor filesystem > options PROCFS # Process filesystem (requires PSEUDOFS) > options PSEUDOFS # Pseudo-filesystem framework > options GEOM_PART_GPT # GUID Partition Tables. > options GEOM_RAID # Soft RAID functionality. > options GEOM_LABEL # Provides labelization > options GEOM_ELI # Disk encryption. > options COMPAT_FREEBSD32 # Compatible with i386 binaries > options COMPAT_FREEBSD4 # Compatible with FreeBSD4 > options COMPAT_FREEBSD5 # Compatible with FreeBSD5 > options COMPAT_FREEBSD6 # Compatible with FreeBSD6 > options COMPAT_FREEBSD7 # Compatible with FreeBSD7 > options COMPAT_FREEBSD9 # Compatible with FreeBSD9 > options COMPAT_FREEBSD10 # Compatible with FreeBSD10 > options COMPAT_FREEBSD11 # Compatible with FreeBSD11 > options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI > options KTRACE # ktrace(1) support > options STACK # stack(9) support > options SYSVSHM # SYSV-style shared memory > options SYSVMSG # SYSV-style message queues > options SYSVSEM # SYSV-style semaphores > options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time > extensions > options PRINTF_BUFR_SIZE=128 # Prevent printf output being > interspersed. > options KBD_INSTALL_CDEV # install a CDEV entry in /dev > options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) > options AUDIT # Security event auditing > options CAPABILITY_MODE # Capsicum capability mode > options CAPABILITIES # Capsicum capabilities > options MAC # TrustedBSD MAC Framework > options MAC_PORTACL > options MAC_NTPD > options KDTRACE_FRAME # Ensure frames are compiled in > options KDTRACE_HOOKS # Kernel DTrace hooks > options DDB_CTF # Kernel ELF linker loads CTF data > options INCLUDE_CONFIG_FILE # Include this file in kernel > > # Debugging support. Always need this: > options KDB # Enable kernel debugger support. > options KDB_TRACE # Print a stack trace for a panic. > options KDB_UNATTENDED > > # Make an SMP-capable kernel by default > options SMP # Symmetric MultiProcessor Kernel > options EARLY_AP_STARTUP > > # CPU frequency control > device cpufreq > device cpuctl > device coretemp > > # Bus support. > device acpi > options ACPI_DMAR > device pci > options PCI_IOV # PCI SR-IOV support > > device iicbus > device iicbb > > device iic > device ic > device iicsmb > > device ichsmb > device smbus > device smb > > #device jedec_dimm > > # ATA controllers > device ahci # AHCI-compatible SATA controllers > device mvs # Marvell 88SX50XX/88SX60XX/88SX70XX/SoC SATA > > # SCSI Controllers > device mps # LSI-Logic MPT-Fusion 2 > > # ATA/SCSI peripherals > device scbus # SCSI bus (required for ATA/SCSI) > device da # Direct Access (disks) > device cd # CD > device pass # Passthrough device (direct ATA/SCSI access) > device ses # Enclosure Services (SES and SAF-TE) > device sg > > device cfiscsi > device ctl # CAM Target Layer > device iscsi > > # atkbdc0 controls both the keyboard and the PS/2 mouse > device atkbdc # AT keyboard controller > device atkbd # AT keyboard > device psm # PS/2 mouse > > device kbdmux # keyboard multiplexer > > # vt is the new video console driver > device vt > device vt_vga > device vt_efifb > > # Serial (COM) ports > device uart # Generic UART driver > > # PCI/PCI-X/PCIe Ethernet NICs that use iflib infrastructure > device iflib > device em # Intel PRO/1000 Gigabit Ethernet Family > device ix # Intel PRO/10GbE PCIE PF Ethernet > > # Network stack virtualization. > options VIMAGE > > # Pseudo devices. > device crypto > device cryptodev > device loop # Network loopback > device random # Entropy device > device padlock_rng # VIA Padlock RNG > device rdrand_rng # Intel Bull Mountain RNG > device ipmi > device smbios > device vpd > device aesni # AES-NI OpenCrypto module > device ether # Ethernet support > device lagg > device vlan # 802.1Q VLAN support > device tuntap # Packet tunnel. > device md # Memory "disks" > device gif # IPv6 and IPv4 tunneling > device firmware # firmware assist module > > device pf > #device pflog > #device pfsync > > # The `bpf' device enables the Berkeley Packet Filter. > # Be aware of the administrative consequences of enabling this! > # Note that 'bpf' is required for DHCP. > device bpf # Berkeley packet filter > > # The `epair' device implements a virtual back-to-back connected > Ethernet > # like interface pair. > device epair > > # USB support > options USB_DEBUG # enable debug msgs > device uhci # UHCI PCI->USB interface > device ohci # OHCI PCI->USB interface > device ehci # EHCI PCI->USB interface (USB 2.0) > device xhci # XHCI PCI->USB interface (USB 3.0) > device usb # USB Bus (required) > device uhid > device ukbd # Keyboard > device umass # Disks/Mass storage - Requires scbus and da > device ums > > device filemon > > device if_bridge > >> On 20 Nov 2020, at 12:53, Kristof Provost wrote: >> >> Can you share your kernel config file (and src.conf / make.conf if >> they exist)? >> >> This second panic is in the IPSec code. My current thinking is that >> your kernel config is triggering a bug that’s manifesting in >> multiple places, but not actually caused by those places. >> >> I’d like to be able to reproduce it so we can debug it. >> >> Best regards, >> Kristof >> >> On 20 Nov 2020, at 12:02, Peter Blok wrote: >>> Hi Kristof, >>> >>> This is 12-stable. With the previous bridge epochification that was >>> backed out my config had a panic too. >>> >>> I don’t have any local modifications. I did a clean rebuild after >>> removing /usr/obj/usr >>> >>> My kernel is custom - I only have zfs.ko, opensolaris.ko, vmm.ko and >>> nmdm.ko as modules. Everything else is statically linked. I have >>> removed all drivers not needed for the hardware at hand. >>> >>> My bridge is between two vlans from the same trunk and the jail >>> epair devices as well as the bhyve tap devices. >>> >>> The panic happens when the jails are starting. >>> >>> I can try to narrow it down over the weekend and make the crash dump >>> available for analysis. >>> >>> Previously I had the following crash with 363492 >>> >>> kernel trap 12 with interrupts disabled >>> >>> >>> Fatal trap 12: page fault while in kernel mode >>> cpuid = 2; apic id = 02 >>> fault virtual address = 0xffffffff00000410 >>> fault code = supervisor read data, page not present >>> instruction pointer = 0x20:0xffffffff80692326 >>> stack pointer = 0x28:0xfffffe00c06097b0 >>> frame pointer = 0x28:0xfffffe00c06097f0 >>> code segment = base 0x0, limit 0xfffff, type 0x1b >>> = DPL 0, pres 1, long 1, def32 0, gran 1 >>> processor eflags = resume, IOPL = 0 >>> current process = 2030 (ifconfig) >>> trap number = 12 >>> panic: page fault >>> cpuid = 2 >>> time = 1595683412 >>> KDB: stack backtrace: >>> #0 0xffffffff80698165 at kdb_backtrace+0x65 >>> #1 0xffffffff8064d67b at vpanic+0x17b >>> #2 0xffffffff8064d4f3 at panic+0x43 >>> #3 0xffffffff809cc311 at trap_fatal+0x391 >>> #4 0xffffffff809cc36f at trap_pfault+0x4f >>> #5 0xffffffff809cb9b6 at trap+0x286 >>> #6 0xffffffff809a5b28 at calltrap+0x8 >>> #7 0xffffffff803677fd at ck_epoch_synchronize_wait+0x8d >>> #8 0xffffffff8069213a at epoch_wait_preempt+0xaa >>> #9 0xffffffff807615b7 at ipsec_ioctl+0x3a7 >>> #10 0xffffffff8075274f at ifioctl+0x47f >>> #11 0xffffffff806b5ea7 at kern_ioctl+0x2b7 >>> #12 0xffffffff806b5b4a at sys_ioctl+0xfa >>> #13 0xffffffff809ccec7 at amd64_syscall+0x387 >>> #14 0xffffffff809a6450 at fast_syscall_common+0x101 >>> >>> >>> >>> >>>> On 20 Nov 2020, at 11:30, Kristof Provost wrote: >>>> >>>> On 20 Nov 2020, at 11:18, peter.blok@bsd4all.org >>>> wrote: >>>>> I’m afraid the last Epoch fix for bridge is not solving the >>>>> problem ( or perhaps creates a new ). >>>>> >>>> We’re talking about the stable/12 branch, right? >>>> >>>>> This seems to happen when the jail epair is added to the bridge. >>>>> >>>> There must be something more to it than that. I’ve run the bridge >>>> tests on stable/12 without issue, and this is a problem we didn’t >>>> see when the bridge epochification initially went into stable/12. >>>> >>>> Do you have a custom kernel config? Other patches? What exact >>>> commands do you run to trigger the panic? >>>> >>>>> kernel trap 12 with interrupts disabled >>>>> >>>>> >>>>> Fatal trap 12: page fault while in kernel mode >>>>> cpuid = 6; apic id = 06 >>>>> fault virtual address = 0xc10 >>>>> fault code = supervisor read data, page not present >>>>> instruction pointer = 0x20:0xffffffff80695e76 >>>>> stack pointer = 0x28:0xfffffe00bf14e6e0 >>>>> frame pointer = 0x28:0xfffffe00bf14e720 >>>>> code segment = base 0x0, limit 0xfffff, type 0x1b >>>>> = DPL 0, pres 1, long 1, def32 0, gran 1 >>>>> processor eflags = resume, IOPL = 0 >>>>> current process = 1686 (jail) >>>>> trap number = 12 >>>>> panic: page fault >>>>> cpuid = 6 >>>>> time = 1605811310 >>>>> KDB: stack backtrace: >>>>> #0 0xffffffff8069bb85 at kdb_backtrace+0x65 >>>>> #1 0xffffffff80650a4b at vpanic+0x17b >>>>> #2 0xffffffff806508c3 at panic+0x43 >>>>> #3 0xffffffff809d0351 at trap_fatal+0x391 >>>>> #4 0xffffffff809d03af at trap_pfault+0x4f >>>>> #5 0xffffffff809cf9f6 at trap+0x286 >>>>> #6 0xffffffff809a98c8 at calltrap+0x8 >>>>> #7 0xffffffff80368a8d at ck_epoch_synchronize_wait+0x8d >>>>> #8 0xffffffff80695c8a at epoch_wait_preempt+0xaa >>>>> #9 0xffffffff80757d40 at vnet_if_init+0x120 >>>>> #10 0xffffffff8078c994 at vnet_alloc+0x114 >>>>> #11 0xffffffff8061e3f7 at kern_jail_set+0x1bb7 >>>>> #12 0xffffffff80620190 at sys_jail_set+0x40 >>>>> #13 0xffffffff809d0f07 at amd64_syscall+0x387 >>>>> #14 0xffffffff809aa1ee at fast_syscall_common+0xf8 >>>> >>>> This panic is rather odd. This isn’t even the bridge code. This >>>> is during initial creation of the vnet. I don’t really see how >>>> this could even trigger panics. >>>> That panic looks as if something corrupted the net_epoch_preempt, >>>> by overwriting the epoch->e_epoch. The bridge patches only access >>>> this variable through the well-established functions and macros. I >>>> see no obvious way that they could corrupt it. >>>> >>>> Best regards, >>>> Kristof >> >> >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to >> "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@freebsd.org Sat Nov 21 16:22:59 2020 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 61C8346A8CE for ; Sat, 21 Nov 2020 16:22:59 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from smtpq6.tb.mail.iss.as9143.net (smtpq6.tb.mail.iss.as9143.net [212.54.42.169]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4CddxY56rJz4sKg; Sat, 21 Nov 2020 16:22:57 +0000 (UTC) (envelope-from pblok@bsd4all.org) Received: from [212.54.42.136] (helo=smtp12.tb.mail.iss.as9143.net) by smtpq6.tb.mail.iss.as9143.net with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1kgVf4-0006eL-UT; Sat, 21 Nov 2020 17:22:54 +0100 Received: from 82-101-198-11.cable.dynamic.v4.ziggo.nl ([82.101.198.11] helo=wan0.bsd4all.org) by smtp12.tb.mail.iss.as9143.net with esmtp (Exim 4.90_1) (envelope-from ) id 1kgVf4-0006CV-IL; Sat, 21 Nov 2020 17:22:54 +0100 Received: from newnas.bsd4all.local (localhost [127.0.0.1]) by wan0.bsd4all.org (Postfix) with ESMTP id A09E2311; Sat, 21 Nov 2020 17:23:35 +0100 (CET) X-Virus-Scanned: amavisd-new at bsd4all.org Received: from wan0.bsd4all.org ([127.0.0.1]) by newnas.bsd4all.local (newnas.bsd4all.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Uky2o2zIv_l4; Sat, 21 Nov 2020 17:23:34 +0100 (CET) Received: from mpro.bsd4all.local (mpro.bsd4all.local [192.168.1.65]) by wan0.bsd4all.org (Postfix) with ESMTPSA id 43E7A289; Sat, 21 Nov 2020 17:23:34 +0100 (CET) From: Peter Blok Message-Id: <9E2E1843-1537-41AA-8FFE-BCE5EC9FF509@bsd4all.org> Content-Type: multipart/signed; boundary="Apple-Mail=_1BF7CE6F-A1E8-4585-9AE0-54ABC001B5AA"; protocol="application/pkcs7-signature"; micalg=sha-256 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.4\)) Subject: Re: Commit 367705+367706 causes a pabic Date: Sat, 21 Nov 2020 17:22:55 +0100 In-Reply-To: <5489BF90-E1F6-44C2-ABD5-1C52BABA206D@FreeBSD.org> Cc: FreeBSD Stable To: Kristof Provost References: <1753B4A3-2FFC-47A5-9D0C-DC0B71BA22E8@FreeBSD.org> <665757BF-DA06-4503-9ACD-8A4630E23FF4@bsd4all.org> <5DA2A6D4-1FDA-48B3-A319-F26CB802E01B@bsd4all.org> <5489BF90-E1F6-44C2-ABD5-1C52BABA206D@FreeBSD.org> X-Mailer: Apple Mail (2.3608.120.23.2.4) X-SourceIP: 82.101.198.11 X-Ziggo-spambar: / X-Ziggo-spamscore: 0.0 X-Ziggo-spamreport: CMAE Analysis: v=2.4 cv=fshi2H0f c=1 sm=1 tr=0 ts=5fb93ede a=3epKYzPC7TQzyyYZMm6bHQ==:17 a=nNwsprhYR40A:10 a=6I5d2MoRAAAA:8 a=6Q3WNqvRAAAA:8 a=-4xoYQfftjp4DLVD92UA:9 a=QEXdDO2ut3YA:10 a=g2jUX3k5JNHoXd8hjKIA:9 a=ZVk8-NSrHBgA:10 a=IjZwj45LgO3ly-622nXo:22 a=I8PBwKCn76L9oNdl0isp:22 X-Ziggo-Spam-Status: No X-Spam-Status: No X-Spam-Flag: No X-Rspamd-Queue-Id: 4CddxY56rJz4sKg X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of pblok@bsd4all.org designates 212.54.42.169 as permitted sender) smtp.mailfrom=pblok@bsd4all.org X-Spamd-Result: default: False [-5.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; MV_CASE(0.50)[]; R_SPF_ALLOW(-0.20)[+a:smtp.ziggo.nl/16]; HAS_ATTACHMENT(0.00)[]; TO_DN_ALL(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-1.000]; RCPT_COUNT_TWO(0.00)[2]; RECEIVED_SPAMHAUS_PBL(0.00)[82.101.198.11:received]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[212.54.42.169:from]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; ASN(0.00)[asn:33915, ipnet:212.54.32.0/20, country:NL]; RCVD_TLS_LAST(0.00)[]; ARC_NA(0.00)[]; RCVD_COUNT_FIVE(0.00)[6]; RCVD_IN_DNSWL_LOW(-0.10)[212.54.42.169:from]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[bsd4all.org]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; SPAMHAUS_ZRD(0.00)[212.54.42.169:from:127.0.2.255]; RWL_MAILSPIKE_POSSIBLE(0.00)[212.54.42.169:from]; MAILMAN_DEST(0.00)[freebsd-stable] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 21 Nov 2020 16:22:59 -0000 --Apple-Mail=_1BF7CE6F-A1E8-4585-9AE0-54ABC001B5AA Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Kristof, With a GENERIC kernel it does NOT happen. I do have a different iflib = related panic at reboot, but I=E2=80=99ll report that separately. I brought the two config files closer together and found out that if I = remove if_bridge from the config file and have it loaded dynamically = when the bridge is created, the problem no longer happens and everything = works ok. Peter > On 20 Nov 2020, at 15:53, Kristof Provost wrote: >=20 > I still can=E2=80=99t reproduce that panic. >=20 > Does it happen immediately after you start a vnet jail? >=20 > Does it also happen with a GENERIC kernel? >=20 > Regards, > Kristof >=20 > On 20 Nov 2020, at 14:53, Peter Blok wrote: >=20 >> The panic with ipsec code in the backtrace was already very strange. = I was using IPsec, but only on one interface totally separate from the = members of the bridge as well as the bridge itself. The jails were not = doing any ipsec as well. Note that panic was a while ago and it was = after the 1st bridge epochification was done on stable-12 which was = later backed out >>=20 >> Today the system is no longer using ipsec, but it is still compiled = in. I can remove it if need be for a test >>=20 >>=20 >> src.conf >> WITHOUT_KERBEROS=3Dyes >> WITHOUT_GSSAPI=3Dyes >> WITHOUT_SENDMAIL=3Dtrue >> WITHOUT_MAILWRAPPER=3Dtrue >> WITHOUT_DMAGENT=3Dtrue >> WITHOUT_GAMES=3Dtrue >> WITHOUT_IPFILTER=3Dtrue >> WITHOUT_UNBOUND=3Dtrue >> WITHOUT_PROFILE=3Dtrue >> WITHOUT_ATM=3Dtrue >> WITHOUT_BSNMP=3Dtrue >> #WITHOUT_CROSS_COMPILER=3Dtrue >> WITHOUT_DEBUG_FILES=3Dtrue >> WITHOUT_DICT=3Dtrue >> WITHOUT_FLOPPY=3Dtrue >> WITHOUT_HTML=3Dtrue >> WITHOUT_HYPERV=3Dtrue >> WITHOUT_NDIS=3Dtrue >> WITHOUT_NIS=3Dtrue >> WITHOUT_PPP=3Dtrue >> WITHOUT_TALK=3Dtrue >> WITHOUT_TESTS=3Dtrue >> WITHOUT_WIRELESS=3Dtrue >> #WITHOUT_LIB32=3Dtrue >> WITHOUT_LPR=3Dtrue >>=20 >> make.conf >> KERNCONF=3DBHYVE >> MODULES_OVERRIDE=3Dopensolaris dtrace zfs vmm nmdm if_bridge = bridgestp if_vxlan pflog libmchain libiconv smbfs linux linux64 = linux_common linuxkpi linprocfs linsysfs ext2fs >> DEFAULT_VERSIONS+=3Dperl5=3D5.30 mysql=3D5.7 python=3D3.8 python3=3D3.8= >> OPTIONS_UNSET=3DDOCS NLS MANPAGES >>=20 >> BHYVE >> cpu HAMMER >> ident BHYVE >>=20 >> makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug = symbols >> makeoptions WITH_CTF=3D1 # Run ctfconvert(1) for DTrace = support >>=20 >> options CAMDEBUG >>=20 >> options SCHED_ULE # ULE scheduler >> options PREEMPTION # Enable kernel thread = preemption >> options INET # InterNETworking >> options INET6 # IPv6 communications protocols >> options IPSEC >> options TCP_OFFLOAD # TCP offload >> options TCP_RFC7413 # TCP FASTOPEN >> options SCTP # Stream Control Transmission = Protocol >> options FFS # Berkeley Fast Filesystem >> options SOFTUPDATES # Enable FFS soft updates = support >> options UFS_ACL # Support for access control = lists >> options UFS_DIRHASH # Improve performance on big = directories >> options UFS_GJOURNAL # Enable gjournal-based UFS = journaling >> options QUOTA # Enable disk quotas for UFS >> options SUIDDIR >> options NFSCL # Network Filesystem Client >> options NFSD # Network Filesystem Server >> options NFSLOCKD # Network Lock Manager >> options MSDOSFS # MSDOS Filesystem >> options CD9660 # ISO 9660 Filesystem >> options FUSEFS >> options NULLFS # NULL filesystem >> options UNIONFS >> options FDESCFS # File descriptor = filesystem >> options PROCFS # Process filesystem (requires = PSEUDOFS) >> options PSEUDOFS # Pseudo-filesystem framework >> options GEOM_PART_GPT # GUID Partition Tables. >> options GEOM_RAID # Soft RAID functionality. >> options GEOM_LABEL # Provides labelization >> options GEOM_ELI # Disk encryption. >> options COMPAT_FREEBSD32 # Compatible with i386 binaries >> options COMPAT_FREEBSD4 # Compatible with FreeBSD4 >> options COMPAT_FREEBSD5 # Compatible with FreeBSD5 >> options COMPAT_FREEBSD6 # Compatible with FreeBSD6 >> options COMPAT_FREEBSD7 # Compatible with FreeBSD7 >> options COMPAT_FREEBSD9 # Compatible with FreeBSD9 >> options COMPAT_FREEBSD10 # Compatible with FreeBSD10 >> options COMPAT_FREEBSD11 # Compatible with FreeBSD11 >> options SCSI_DELAY=3D5000 # Delay (in ms) before = probing SCSI >> options KTRACE # ktrace(1) support >> options STACK # stack(9) support >> options SYSVSHM # SYSV-style shared memory >> options SYSVMSG # SYSV-style message queues >> options SYSVSEM # SYSV-style semaphores >> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time = extensions >> options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being = interspersed. >> options KBD_INSTALL_CDEV # install a CDEV entry in /dev >> options HWPMC_HOOKS # Necessary kernel hooks for = hwpmc(4) >> options AUDIT # Security event auditing >> options CAPABILITY_MODE # Capsicum capability mode >> options CAPABILITIES # Capsicum capabilities >> options MAC # TrustedBSD MAC Framework >> options MAC_PORTACL >> options MAC_NTPD >> options KDTRACE_FRAME # Ensure frames are compiled in >> options KDTRACE_HOOKS # Kernel DTrace hooks >> options DDB_CTF # Kernel ELF linker loads CTF = data >> options INCLUDE_CONFIG_FILE # Include this file in kernel >>=20 >> # Debugging support. Always need this: >> options KDB # Enable kernel debugger = support. >> options KDB_TRACE # Print a stack trace for a = panic. >> options KDB_UNATTENDED >>=20 >> # Make an SMP-capable kernel by default >> options SMP # Symmetric MultiProcessor = Kernel >> options EARLY_AP_STARTUP >>=20 >> # CPU frequency control >> device cpufreq >> device cpuctl >> device coretemp >>=20 >> # Bus support. >> device acpi >> options ACPI_DMAR >> device pci >> options PCI_IOV # PCI SR-IOV support >>=20 >> device iicbus >> device iicbb >>=20 >> device iic >> device ic >> device iicsmb >>=20 >> device ichsmb >> device smbus >> device smb >>=20 >> #device jedec_dimm >>=20 >> # ATA controllers >> device ahci # AHCI-compatible SATA = controllers >> device mvs # Marvell = 88SX50XX/88SX60XX/88SX70XX/SoC SATA >>=20 >> # SCSI Controllers >> device mps # LSI-Logic MPT-Fusion 2 >>=20 >> # ATA/SCSI peripherals >> device scbus # SCSI bus (required for = ATA/SCSI) >> device da # Direct Access (disks) >> device cd # CD >> device pass # Passthrough device = (direct ATA/SCSI access) >> device ses # Enclosure Services = (SES and SAF-TE) >> device sg >>=20 >> device cfiscsi >> device ctl # CAM Target Layer >> device iscsi >>=20 >> # atkbdc0 controls both the keyboard and the PS/2 mouse >> device atkbdc # AT keyboard controller >> device atkbd # AT keyboard >> device psm # PS/2 mouse >>=20 >> device kbdmux # keyboard multiplexer >>=20 >> # vt is the new video console driver >> device vt >> device vt_vga >> device vt_efifb >>=20 >> # Serial (COM) ports >> device uart # Generic UART driver >>=20 >> # PCI/PCI-X/PCIe Ethernet NICs that use iflib infrastructure >> device iflib >> device em # Intel PRO/1000 Gigabit = Ethernet Family >> device ix # Intel PRO/10GbE PCIE = PF Ethernet >>=20 >> # Network stack virtualization. >> options VIMAGE >>=20 >> # Pseudo devices. >> device crypto >> device cryptodev >> device loop # Network loopback >> device random # Entropy device >> device padlock_rng # VIA Padlock RNG >> device rdrand_rng # Intel Bull Mountain = RNG >> device ipmi >> device smbios >> device vpd >> device aesni # AES-NI OpenCrypto = module >> device ether # Ethernet support >> device lagg >> device vlan # 802.1Q VLAN support >> device tuntap # Packet tunnel. >> device md # Memory "disks" >> device gif # IPv6 and IPv4 = tunneling >> device firmware # firmware assist module >>=20 >> device pf >> #device pflog >> #device pfsync >>=20 >> # The `bpf' device enables the Berkeley Packet Filter. >> # Be aware of the administrative consequences of enabling this! >> # Note that 'bpf' is required for DHCP. >> device bpf # Berkeley packet filter >>=20 >> # The `epair' device implements a virtual back-to-back connected = Ethernet >> # like interface pair. >> device epair >>=20 >> # USB support >> options USB_DEBUG # enable debug msgs >> device uhci # UHCI PCI->USB = interface >> device ohci # OHCI PCI->USB = interface >> device ehci # EHCI PCI->USB = interface (USB 2.0) >> device xhci # XHCI PCI->USB = interface (USB 3.0) >> device usb # USB Bus (required) >> device uhid >> device ukbd # Keyboard >> device umass # Disks/Mass storage - = Requires scbus and da >> device ums >>=20 >> device filemon >>=20 >> device if_bridge >>=20 >>> On 20 Nov 2020, at 12:53, Kristof Provost wrote: >>>=20 >>> Can you share your kernel config file (and src.conf / make.conf if = they exist)? >>>=20 >>> This second panic is in the IPSec code. My current thinking is that = your kernel config is triggering a bug that=E2=80=99s manifesting in = multiple places, but not actually caused by those places. >>>=20 >>> I=E2=80=99d like to be able to reproduce it so we can debug it. >>>=20 >>> Best regards, >>> Kristof >>>=20 >>> On 20 Nov 2020, at 12:02, Peter Blok wrote: >>>> Hi Kristof, >>>>=20 >>>> This is 12-stable. With the previous bridge epochification that was = backed out my config had a panic too. >>>>=20 >>>> I don=E2=80=99t have any local modifications. I did a clean rebuild = after removing /usr/obj/usr >>>>=20 >>>> My kernel is custom - I only have zfs.ko, opensolaris.ko, vmm.ko = and nmdm.ko as modules. Everything else is statically linked. I have = removed all drivers not needed for the hardware at hand. >>>>=20 >>>> My bridge is between two vlans from the same trunk and the jail = epair devices as well as the bhyve tap devices. >>>>=20 >>>> The panic happens when the jails are starting. >>>>=20 >>>> I can try to narrow it down over the weekend and make the crash = dump available for analysis. >>>>=20 >>>> Previously I had the following crash with 363492 >>>>=20 >>>> kernel trap 12 with interrupts disabled >>>>=20 >>>>=20 >>>> Fatal trap 12: page fault while in kernel mode >>>> cpuid =3D 2; apic id =3D 02 >>>> fault virtual address =3D 0xffffffff00000410 >>>> fault code =3D supervisor read data, page not present >>>> instruction pointer =3D 0x20:0xffffffff80692326 >>>> stack pointer =3D 0x28:0xfffffe00c06097b0 >>>> frame pointer =3D 0x28:0xfffffe00c06097f0 >>>> code segment =3D base 0x0, limit 0xfffff, type 0x1b >>>> =3D DPL 0, pres 1, long 1, def32 0, gran 1 >>>> processor eflags =3D resume, IOPL =3D 0 >>>> current process =3D 2030 (ifconfig) >>>> trap number =3D 12 >>>> panic: page fault >>>> cpuid =3D 2 >>>> time =3D 1595683412 >>>> KDB: stack backtrace: >>>> #0 0xffffffff80698165 at kdb_backtrace+0x65 >>>> #1 0xffffffff8064d67b at vpanic+0x17b >>>> #2 0xffffffff8064d4f3 at panic+0x43 >>>> #3 0xffffffff809cc311 at trap_fatal+0x391 >>>> #4 0xffffffff809cc36f at trap_pfault+0x4f >>>> #5 0xffffffff809cb9b6 at trap+0x286 >>>> #6 0xffffffff809a5b28 at calltrap+0x8 >>>> #7 0xffffffff803677fd at ck_epoch_synchronize_wait+0x8d >>>> #8 0xffffffff8069213a at epoch_wait_preempt+0xaa >>>> #9 0xffffffff807615b7 at ipsec_ioctl+0x3a7 >>>> #10 0xffffffff8075274f at ifioctl+0x47f >>>> #11 0xffffffff806b5ea7 at kern_ioctl+0x2b7 >>>> #12 0xffffffff806b5b4a at sys_ioctl+0xfa >>>> #13 0xffffffff809ccec7 at amd64_syscall+0x387 >>>> #14 0xffffffff809a6450 at fast_syscall_common+0x101 >>>>=20 >>>>=20 >>>>=20 >>>>=20 >>>>> On 20 Nov 2020, at 11:30, Kristof Provost wrote: >>>>>=20 >>>>> On 20 Nov 2020, at 11:18, peter.blok@bsd4all.org = wrote: >>>>>> I=E2=80=99m afraid the last Epoch fix for bridge is not solving = the problem ( or perhaps creates a new ). >>>>>>=20 >>>>> We=E2=80=99re talking about the stable/12 branch, right? >>>>>=20 >>>>>> This seems to happen when the jail epair is added to the bridge. >>>>>>=20 >>>>> There must be something more to it than that. I=E2=80=99ve run the = bridge tests on stable/12 without issue, and this is a problem we = didn=E2=80=99t see when the bridge epochification initially went into = stable/12. >>>>>=20 >>>>> Do you have a custom kernel config? Other patches? What exact = commands do you run to trigger the panic? >>>>>=20 >>>>>> kernel trap 12 with interrupts disabled >>>>>>=20 >>>>>>=20 >>>>>> Fatal trap 12: page fault while in kernel mode >>>>>> cpuid =3D 6; apic id =3D 06 >>>>>> fault virtual address =3D 0xc10 >>>>>> fault code =3D supervisor read data, page not = present >>>>>> instruction pointer =3D 0x20:0xffffffff80695e76 >>>>>> stack pointer =3D 0x28:0xfffffe00bf14e6e0 >>>>>> frame pointer =3D 0x28:0xfffffe00bf14e720 >>>>>> code segment =3D base 0x0, limit 0xfffff, type 0x1b >>>>>> =3D DPL 0, pres 1, long 1, def32 0, gran 1 >>>>>> processor eflags =3D resume, IOPL =3D 0 >>>>>> current process =3D 1686 (jail) >>>>>> trap number =3D 12 >>>>>> panic: page fault >>>>>> cpuid =3D 6 >>>>>> time =3D 1605811310 >>>>>> KDB: stack backtrace: >>>>>> #0 0xffffffff8069bb85 at kdb_backtrace+0x65 >>>>>> #1 0xffffffff80650a4b at vpanic+0x17b >>>>>> #2 0xffffffff806508c3 at panic+0x43 >>>>>> #3 0xffffffff809d0351 at trap_fatal+0x391 >>>>>> #4 0xffffffff809d03af at trap_pfault+0x4f >>>>>> #5 0xffffffff809cf9f6 at trap+0x286 >>>>>> #6 0xffffffff809a98c8 at calltrap+0x8 >>>>>> #7 0xffffffff80368a8d at ck_epoch_synchronize_wait+0x8d >>>>>> #8 0xffffffff80695c8a at epoch_wait_preempt+0xaa >>>>>> #9 0xffffffff80757d40 at vnet_if_init+0x120 >>>>>> #10 0xffffffff8078c994 at vnet_alloc+0x114 >>>>>> #11 0xffffffff8061e3f7 at kern_jail_set+0x1bb7 >>>>>> #12 0xffffffff80620190 at sys_jail_set+0x40 >>>>>> #13 0xffffffff809d0f07 at amd64_syscall+0x387 >>>>>> #14 0xffffffff809aa1ee at fast_syscall_common+0xf8 >>>>>=20 >>>>> This panic is rather odd. This isn=E2=80=99t even the bridge code. = This is during initial creation of the vnet. I don=E2=80=99t really see = how this could even trigger panics. >>>>> That panic looks as if something corrupted the net_epoch_preempt, = by overwriting the epoch->e_epoch. The bridge patches only access this = variable through the well-established functions and macros. I see no = obvious way that they could corrupt it. >>>>>=20 >>>>> Best regards, >>>>> Kristof >>>=20 >>>=20 >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" --Apple-Mail=_1BF7CE6F-A1E8-4585-9AE0-54ABC001B5AA Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCBSAw ggUcMIIEBKADAgECAhEAq2wFIs+rCK6H6/2jbblXhDANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgwNDE0MDAwMDAwWhcNMjEwNDEzMjM1 OTU5WjBEMQswCQYDVQQGEwJOTDETMBEGA1UEAxMKUGV0ZXIgQmxvazEgMB4GCSqGSIb3DQEJARYR cGJsb2tAYnNkNGFsbC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPT/3evs2a zLSIVepGa9qFVcSISd5HzoJt9xAyQ4od7NM6Qzwm446OyhzWsIN/a6+nDNB4AxzSg00QXKx4afEa FrdLzmREEfv24f88j2UZYqHAls0j26jyED5FZ068xs4gWZBG2U7EVTUNNJuUrrmqBNZkGxTIrFrD Cgr1EpRULpN+HrEelHHh7uR0twAjvwcyXkG9DbDJXnw8HzKGR80ik4+13HDxx4mDxOY4NOvWSSiM kEFS2Z2AKtxXSMBQZHazAUvbka27c1m93/QsjnDF+P6Aef9NEvUDL9mU9Jbf/+5V+anT2KdPGP4p rQ9gA/Nup61qxDkwc+RupiXD5NSbAgMBAAGjggGzMIIBrzAfBgNVHSMEGDAWgBSCr2yM+MX+lmF8 6B89K3FIXsSLwDAdBgNVHQ4EFgQUjwe7n1zvxFkTeCUYWrsaJpOGP14wDgYDVR0PAQH/BAQDAgWg MAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMEYGA1UdIAQ/MD0w OwYMKwYBBAGyMQECAQMFMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQv Q1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNs aWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgYsGCCsGAQUFBwEBBH8wfTBV BggrBgEFBQcwAoZJaHR0cDovL2NydC5jb21vZG9jYS5jb20vQ09NT0RPUlNBQ2xpZW50QXV0aGVu dGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29t b2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQC85hVlqTVwt218IJR/WjMiMnDtZ7hY860XKjzO uB3sUUQwHxHj+ZYuMbAfVLZGGqh1EekbwDMVgkK9cezIHM+ZzxrNGX2SJyl1YW+3FLn52P0uIlmA VPFjUowf5qBhOHl2NJo+WXYZhQY7rT/xSygE81o3oLE/A4zO6WtO3PeZpFpZNrBvizAsjTDfPeXW iQzXz6NLrgwert0Wml95ov2rG5oCzHYPijabubSNm2NdUjPRtcVylcqAThXOvp6X4UvW8/L0uhkp 9WsKP2JEJ3Zukv7Ib+vMBsdE4tf4rmv89pQC+lLpD08ze/QDCIeFBCRIihcC2PycDQrnNIp1RAIh MYIDyjCCA8YCAQEwga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0 ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQD EzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEA q2wFIs+rCK6H6/2jbblXhDANBglghkgBZQMEAgEFAKCCAe0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMjAxMTIxMTYyMjU1WjAvBgkqhkiG9w0BCQQxIgQg2lnqsKLW ohIuOQjw1vzmZKK+V+gbeoeyk/LVUFsig4owgb4GCSsGAQQBgjcQBDGBsDCBrTCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCrbAUiz6sIrofr/aNtuVeEMIHABgsqhkiG 9w0BCRACCzGBsKCBrTCBlzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3Rl cjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMT NENPTU9ETyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCr bAUiz6sIrofr/aNtuVeEMA0GCSqGSIb3DQEBAQUABIIBAJ2bRojW3XsYwMHChArVS/Nv/jvCy7e5 XtMUTyFl0CLVziXnZoAh9wD/vSpCkXiWuZ8adnDNGEPA7o+9HMEbdTSprDBNjaIsnCsHSh1dZhh8 cwfCNjF/vCHJMa/K5wAMgcFNWpr+ZBUUAgj9CV2AIpOI2L/BCditXoZktPin5wYrthc8XuV1F73+ VmYkTenfWha4DOt+XnclxE2LabgO40g8cQjUCXPd5bfk5TBdRsN6jpiAg9qMGPwFNS3Wuf5Q3B0/ KkiYEYsANkvHFxyR4NYPVypgBhxvCBLWK9/c98fnzcBX3Upcl06era8rBV4uVjDhWsv/d20Bkddn KdL93z0AAAAAAAA= --Apple-Mail=_1BF7CE6F-A1E8-4585-9AE0-54ABC001B5AA--