Date: Fri, 20 Nov 2020 14:53:23 +0100 From: Peter Blok <pblok@bsd4all.org> To: Kristof Provost <kp@FreeBSD.org> Cc: FreeBSD Stable <freebsd-stable@freebsd.org> Subject: Re: Commit 367705+367706 causes a pabic Message-ID: <5DA2A6D4-1FDA-48B3-A319-F26CB802E01B@bsd4all.org> In-Reply-To: <BD8D114E-9F12-4580-A0D5-5A5BAC75DF27@FreeBSD.org> References: <CD3B0F62-3790-4C63-A92C-9694256823CD@bsd4all.org> <1753B4A3-2FFC-47A5-9D0C-DC0B71BA22E8@FreeBSD.org> <665757BF-DA06-4503-9ACD-8A4630E23FF4@bsd4all.org> <BD8D114E-9F12-4580-A0D5-5A5BAC75DF27@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--Apple-Mail=_A2B34ADC-6DE9-4C15-A605-0C85F060E27E Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 The panic with ipsec code in the backtrace was already very strange. I = was using IPsec, but only on one interface totally separate from the = members of the bridge as well as the bridge itself. The jails were not = doing any ipsec as well. Note that panic was a while ago and it was = after the 1st bridge epochification was done on stable-12 which was = later backed out Today the system is no longer using ipsec, but it is still compiled in. = I can remove it if need be for a test src.conf WITHOUT_KERBEROS=3Dyes WITHOUT_GSSAPI=3Dyes WITHOUT_SENDMAIL=3Dtrue WITHOUT_MAILWRAPPER=3Dtrue WITHOUT_DMAGENT=3Dtrue WITHOUT_GAMES=3Dtrue WITHOUT_IPFILTER=3Dtrue WITHOUT_UNBOUND=3Dtrue WITHOUT_PROFILE=3Dtrue WITHOUT_ATM=3Dtrue WITHOUT_BSNMP=3Dtrue #WITHOUT_CROSS_COMPILER=3Dtrue WITHOUT_DEBUG_FILES=3Dtrue WITHOUT_DICT=3Dtrue WITHOUT_FLOPPY=3Dtrue WITHOUT_HTML=3Dtrue WITHOUT_HYPERV=3Dtrue WITHOUT_NDIS=3Dtrue WITHOUT_NIS=3Dtrue WITHOUT_PPP=3Dtrue WITHOUT_TALK=3Dtrue WITHOUT_TESTS=3Dtrue WITHOUT_WIRELESS=3Dtrue #WITHOUT_LIB32=3Dtrue WITHOUT_LPR=3Dtrue make.conf KERNCONF=3DBHYVE MODULES_OVERRIDE=3Dopensolaris dtrace zfs vmm nmdm if_bridge bridgestp = if_vxlan pflog libmchain libiconv smbfs linux linux64 linux_common = linuxkpi linprocfs linsysfs ext2fs DEFAULT_VERSIONS+=3Dperl5=3D5.30 mysql=3D5.7 python=3D3.8 python3=3D3.8 OPTIONS_UNSET=3DDOCS NLS MANPAGES BHYVE cpu HAMMER ident BHYVE makeoptions DEBUG=3D-g # Build kernel with gdb(1) debug = symbols makeoptions WITH_CTF=3D1 # Run ctfconvert(1) for DTrace = support options CAMDEBUG options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread = preemption options INET # InterNETworking options INET6 # IPv6 communications protocols options IPSEC options TCP_OFFLOAD # TCP offload options TCP_RFC7413 # TCP FASTOPEN options SCTP # Stream Control Transmission = Protocol options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates = support options UFS_ACL # Support for access control = lists options UFS_DIRHASH # Improve performance on big = directories options UFS_GJOURNAL # Enable gjournal-based UFS = journaling options QUOTA # Enable disk quotas for UFS options SUIDDIR options NFSCL # Network Filesystem Client options NFSD # Network Filesystem Server options NFSLOCKD # Network Lock Manager options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options FUSEFS options NULLFS # NULL filesystem options UNIONFS options FDESCFS # File descriptor filesystem options PROCFS # Process filesystem (requires = PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_RAID # Soft RAID functionality. options GEOM_LABEL # Provides labelization options GEOM_ELI # Disk encryption. options COMPAT_FREEBSD32 # Compatible with i386 binaries options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options COMPAT_FREEBSD6 # Compatible with FreeBSD6 options COMPAT_FREEBSD7 # Compatible with FreeBSD7 options COMPAT_FREEBSD9 # Compatible with FreeBSD9 options COMPAT_FREEBSD10 # Compatible with FreeBSD10 options COMPAT_FREEBSD11 # Compatible with FreeBSD11 options SCSI_DELAY=3D5000 # Delay (in ms) before = probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time = extensions options PRINTF_BUFR_SIZE=3D128 # Prevent printf output being = interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for = hwpmc(4) options AUDIT # Security event auditing options CAPABILITY_MODE # Capsicum capability mode options CAPABILITIES # Capsicum capabilities options MAC # TrustedBSD MAC Framework options MAC_PORTACL options MAC_NTPD options KDTRACE_FRAME # Ensure frames are compiled in options KDTRACE_HOOKS # Kernel DTrace hooks options DDB_CTF # Kernel ELF linker loads CTF = data options INCLUDE_CONFIG_FILE # Include this file in kernel # Debugging support. Always need this: options KDB # Enable kernel debugger = support. options KDB_TRACE # Print a stack trace for a = panic. options KDB_UNATTENDED # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcessor = Kernel options EARLY_AP_STARTUP # CPU frequency control device cpufreq device cpuctl device coretemp # Bus support. device acpi options ACPI_DMAR device pci options PCI_IOV # PCI SR-IOV support device iicbus device iicbb device iic device ic device iicsmb device ichsmb device smbus device smb #device jedec_dimm # ATA controllers device ahci # AHCI-compatible SATA = controllers device mvs # Marvell = 88SX50XX/88SX60XX/88SX70XX/SoC SATA # SCSI Controllers device mps # LSI-Logic MPT-Fusion 2 # ATA/SCSI peripherals device scbus # SCSI bus (required for = ATA/SCSI) device da # Direct Access (disks) device cd # CD device pass # Passthrough device (direct = ATA/SCSI access) device ses # Enclosure Services (SES and = SAF-TE) device sg device cfiscsi device ctl # CAM Target Layer device iscsi # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer # vt is the new video console driver device vt device vt_vga device vt_efifb # Serial (COM) ports device uart # Generic UART driver # PCI/PCI-X/PCIe Ethernet NICs that use iflib infrastructure device iflib device em # Intel PRO/1000 Gigabit = Ethernet Family device ix # Intel PRO/10GbE PCIE PF = Ethernet # Network stack virtualization. options VIMAGE # Pseudo devices. device crypto device cryptodev device loop # Network loopback device random # Entropy device device padlock_rng # VIA Padlock RNG device rdrand_rng # Intel Bull Mountain RNG device ipmi device smbios device vpd device aesni # AES-NI OpenCrypto module device ether # Ethernet support device lagg device vlan # 802.1Q VLAN support device tuntap # Packet tunnel. device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device firmware # firmware assist module device pf #device pflog #device pfsync # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter # The `epair' device implements a virtual back-to-back connected = Ethernet # like interface pair. device epair # USB support options USB_DEBUG # enable debug msgs device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB = 2.0) device xhci # XHCI PCI->USB interface (USB = 3.0) device usb # USB Bus (required) device uhid device ukbd # Keyboard device umass # Disks/Mass storage - Requires = scbus and da device ums device filemon device if_bridge > On 20 Nov 2020, at 12:53, Kristof Provost <kp@FreeBSD.org> wrote: >=20 > Can you share your kernel config file (and src.conf / make.conf if = they exist)? >=20 > This second panic is in the IPSec code. My current thinking is that = your kernel config is triggering a bug that=E2=80=99s manifesting in = multiple places, but not actually caused by those places. >=20 > I=E2=80=99d like to be able to reproduce it so we can debug it. >=20 > Best regards, > Kristof >=20 > On 20 Nov 2020, at 12:02, Peter Blok wrote: >> Hi Kristof, >>=20 >> This is 12-stable. With the previous bridge epochification that was = backed out my config had a panic too. >>=20 >> I don=E2=80=99t have any local modifications. I did a clean rebuild = after removing /usr/obj/usr >>=20 >> My kernel is custom - I only have zfs.ko, opensolaris.ko, vmm.ko and = nmdm.ko as modules. Everything else is statically linked. I have removed = all drivers not needed for the hardware at hand. >>=20 >> My bridge is between two vlans from the same trunk and the jail epair = devices as well as the bhyve tap devices. >>=20 >> The panic happens when the jails are starting. >>=20 >> I can try to narrow it down over the weekend and make the crash dump = available for analysis. >>=20 >> Previously I had the following crash with 363492 >>=20 >> kernel trap 12 with interrupts disabled >>=20 >>=20 >> Fatal trap 12: page fault while in kernel mode >> cpuid =3D 2; apic id =3D 02 >> fault virtual address =3D 0xffffffff00000410 >> fault code =3D supervisor read data, page not present >> instruction pointer =3D 0x20:0xffffffff80692326 >> stack pointer =3D 0x28:0xfffffe00c06097b0 >> frame pointer =3D 0x28:0xfffffe00c06097f0 >> code segment =3D base 0x0, limit 0xfffff, type 0x1b >> =3D DPL 0, pres 1, long 1, def32 0, gran 1 >> processor eflags =3D resume, IOPL =3D 0 >> current process =3D 2030 (ifconfig) >> trap number =3D 12 >> panic: page fault >> cpuid =3D 2 >> time =3D 1595683412 >> KDB: stack backtrace: >> #0 0xffffffff80698165 at kdb_backtrace+0x65 >> #1 0xffffffff8064d67b at vpanic+0x17b >> #2 0xffffffff8064d4f3 at panic+0x43 >> #3 0xffffffff809cc311 at trap_fatal+0x391 >> #4 0xffffffff809cc36f at trap_pfault+0x4f >> #5 0xffffffff809cb9b6 at trap+0x286 >> #6 0xffffffff809a5b28 at calltrap+0x8 >> #7 0xffffffff803677fd at ck_epoch_synchronize_wait+0x8d >> #8 0xffffffff8069213a at epoch_wait_preempt+0xaa >> #9 0xffffffff807615b7 at ipsec_ioctl+0x3a7 >> #10 0xffffffff8075274f at ifioctl+0x47f >> #11 0xffffffff806b5ea7 at kern_ioctl+0x2b7 >> #12 0xffffffff806b5b4a at sys_ioctl+0xfa >> #13 0xffffffff809ccec7 at amd64_syscall+0x387 >> #14 0xffffffff809a6450 at fast_syscall_common+0x101 >>=20 >>=20 >>=20 >>=20 >>> On 20 Nov 2020, at 11:30, Kristof Provost <kp@FreeBSD.org> wrote: >>>=20 >>> On 20 Nov 2020, at 11:18, peter.blok@bsd4all.org = <mailto:peter.blok@bsd4all.org> wrote: >>>> I=E2=80=99m afraid the last Epoch fix for bridge is not solving the = problem ( or perhaps creates a new ). >>>>=20 >>> We=E2=80=99re talking about the stable/12 branch, right? >>>=20 >>>> This seems to happen when the jail epair is added to the bridge. >>>>=20 >>> There must be something more to it than that. I=E2=80=99ve run the = bridge tests on stable/12 without issue, and this is a problem we = didn=E2=80=99t see when the bridge epochification initially went into = stable/12. >>>=20 >>> Do you have a custom kernel config? Other patches? What exact = commands do you run to trigger the panic? >>>=20 >>>> kernel trap 12 with interrupts disabled >>>>=20 >>>>=20 >>>> Fatal trap 12: page fault while in kernel mode >>>> cpuid =3D 6; apic id =3D 06 >>>> fault virtual address =3D 0xc10 >>>> fault code =3D supervisor read data, page not present >>>> instruction pointer =3D 0x20:0xffffffff80695e76 >>>> stack pointer =3D 0x28:0xfffffe00bf14e6e0 >>>> frame pointer =3D 0x28:0xfffffe00bf14e720 >>>> code segment =3D base 0x0, limit 0xfffff, type 0x1b >>>> =3D DPL 0, pres 1, long 1, def32 0, gran 1 >>>> processor eflags =3D resume, IOPL =3D 0 >>>> current process =3D 1686 (jail) >>>> trap number =3D 12 >>>> panic: page fault >>>> cpuid =3D 6 >>>> time =3D 1605811310 >>>> KDB: stack backtrace: >>>> #0 0xffffffff8069bb85 at kdb_backtrace+0x65 >>>> #1 0xffffffff80650a4b at vpanic+0x17b >>>> #2 0xffffffff806508c3 at panic+0x43 >>>> #3 0xffffffff809d0351 at trap_fatal+0x391 >>>> #4 0xffffffff809d03af at trap_pfault+0x4f >>>> #5 0xffffffff809cf9f6 at trap+0x286 >>>> #6 0xffffffff809a98c8 at calltrap+0x8 >>>> #7 0xffffffff80368a8d at ck_epoch_synchronize_wait+0x8d >>>> #8 0xffffffff80695c8a at epoch_wait_preempt+0xaa >>>> #9 0xffffffff80757d40 at vnet_if_init+0x120 >>>> #10 0xffffffff8078c994 at vnet_alloc+0x114 >>>> #11 0xffffffff8061e3f7 at kern_jail_set+0x1bb7 >>>> #12 0xffffffff80620190 at sys_jail_set+0x40 >>>> #13 0xffffffff809d0f07 at amd64_syscall+0x387 >>>> #14 0xffffffff809aa1ee at fast_syscall_common+0xf8 >>>=20 >>> This panic is rather odd. This isn=E2=80=99t even the bridge code. = This is during initial creation of the vnet. I don=E2=80=99t really see = how this could even trigger panics. >>> That panic looks as if something corrupted the net_epoch_preempt, by = overwriting the epoch->e_epoch. The bridge patches only access this = variable through the well-established functions and macros. I see no = obvious way that they could corrupt it. >>>=20 >>> Best regards, >>> Kristof >=20 >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" --Apple-Mail=_A2B34ADC-6DE9-4C15-A605-0C85F060E27E Content-Disposition: attachment; filename=smime.p7s Content-Type: application/pkcs7-signature; name=smime.p7s Content-Transfer-Encoding: base64 MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgEFADCABgkqhkiG9w0BBwEAAKCCBSAw ggUcMIIEBKADAgECAhEAq2wFIs+rCK6H6/2jbblXhDANBgkqhkiG9w0BAQsFADCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0EwHhcNMTgwNDE0MDAwMDAwWhcNMjEwNDEzMjM1 OTU5WjBEMQswCQYDVQQGEwJOTDETMBEGA1UEAxMKUGV0ZXIgQmxvazEgMB4GCSqGSIb3DQEJARYR cGJsb2tAYnNkNGFsbC5vcmcwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDPT/3evs2a zLSIVepGa9qFVcSISd5HzoJt9xAyQ4od7NM6Qzwm446OyhzWsIN/a6+nDNB4AxzSg00QXKx4afEa FrdLzmREEfv24f88j2UZYqHAls0j26jyED5FZ068xs4gWZBG2U7EVTUNNJuUrrmqBNZkGxTIrFrD Cgr1EpRULpN+HrEelHHh7uR0twAjvwcyXkG9DbDJXnw8HzKGR80ik4+13HDxx4mDxOY4NOvWSSiM kEFS2Z2AKtxXSMBQZHazAUvbka27c1m93/QsjnDF+P6Aef9NEvUDL9mU9Jbf/+5V+anT2KdPGP4p rQ9gA/Nup61qxDkwc+RupiXD5NSbAgMBAAGjggGzMIIBrzAfBgNVHSMEGDAWgBSCr2yM+MX+lmF8 6B89K3FIXsSLwDAdBgNVHQ4EFgQUjwe7n1zvxFkTeCUYWrsaJpOGP14wDgYDVR0PAQH/BAQDAgWg MAwGA1UdEwEB/wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwQGCCsGAQUFBwMCMEYGA1UdIAQ/MD0w OwYMKwYBBAGyMQECAQMFMCswKQYIKwYBBQUHAgEWHWh0dHBzOi8vc2VjdXJlLmNvbW9kby5uZXQv Q1BTMFoGA1UdHwRTMFEwT6BNoEuGSWh0dHA6Ly9jcmwuY29tb2RvY2EuY29tL0NPTU9ET1JTQUNs aWVudEF1dGhlbnRpY2F0aW9uYW5kU2VjdXJlRW1haWxDQS5jcmwwgYsGCCsGAQUFBwEBBH8wfTBV BggrBgEFBQcwAoZJaHR0cDovL2NydC5jb21vZG9jYS5jb20vQ09NT0RPUlNBQ2xpZW50QXV0aGVu dGljYXRpb25hbmRTZWN1cmVFbWFpbENBLmNydDAkBggrBgEFBQcwAYYYaHR0cDovL29jc3AuY29t b2RvY2EuY29tMA0GCSqGSIb3DQEBCwUAA4IBAQC85hVlqTVwt218IJR/WjMiMnDtZ7hY860XKjzO uB3sUUQwHxHj+ZYuMbAfVLZGGqh1EekbwDMVgkK9cezIHM+ZzxrNGX2SJyl1YW+3FLn52P0uIlmA VPFjUowf5qBhOHl2NJo+WXYZhQY7rT/xSygE81o3oLE/A4zO6WtO3PeZpFpZNrBvizAsjTDfPeXW iQzXz6NLrgwert0Wml95ov2rG5oCzHYPijabubSNm2NdUjPRtcVylcqAThXOvp6X4UvW8/L0uhkp 9WsKP2JEJ3Zukv7Ib+vMBsdE4tf4rmv89pQC+lLpD08ze/QDCIeFBCRIihcC2PycDQrnNIp1RAIh MYIDyjCCA8YCAQEwga0wgZcxCzAJBgNVBAYTAkdCMRswGQYDVQQIExJHcmVhdGVyIE1hbmNoZXN0 ZXIxEDAOBgNVBAcTB1NhbGZvcmQxGjAYBgNVBAoTEUNPTU9ETyBDQSBMaW1pdGVkMT0wOwYDVQQD EzRDT01PRE8gUlNBIENsaWVudCBBdXRoZW50aWNhdGlvbiBhbmQgU2VjdXJlIEVtYWlsIENBAhEA q2wFIs+rCK6H6/2jbblXhDANBglghkgBZQMEAgEFAKCCAe0wGAYJKoZIhvcNAQkDMQsGCSqGSIb3 DQEHATAcBgkqhkiG9w0BCQUxDxcNMjAxMTIwMTM1MzIzWjAvBgkqhkiG9w0BCQQxIgQgDz2ZdeTh aYMkkQg+6GjrYvfLjWgNbIMXZXTfv7auV6cwgb4GCSsGAQQBgjcQBDGBsDCBrTCBlzELMAkGA1UE BhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3RlcjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgG A1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMTNENPTU9ETyBSU0EgQ2xpZW50IEF1dGhl bnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCrbAUiz6sIrofr/aNtuVeEMIHABgsqhkiG 9w0BCRACCzGBsKCBrTCBlzELMAkGA1UEBhMCR0IxGzAZBgNVBAgTEkdyZWF0ZXIgTWFuY2hlc3Rl cjEQMA4GA1UEBxMHU2FsZm9yZDEaMBgGA1UEChMRQ09NT0RPIENBIExpbWl0ZWQxPTA7BgNVBAMT NENPTU9ETyBSU0EgQ2xpZW50IEF1dGhlbnRpY2F0aW9uIGFuZCBTZWN1cmUgRW1haWwgQ0ECEQCr bAUiz6sIrofr/aNtuVeEMA0GCSqGSIb3DQEBAQUABIIBAHX2f6lsci746bTdjY6DvGOZVPq0ia2X iY1yO94e5Z6EUAU1gV7xSiysUef9NhNU49vPlQBwMutDE1RWQ+OSMtYaSPlYMpLMQXxZDkIjLbh7 P3CXItXaT/2/MAvTx0Lbq8RsSfQTaKgnxqwLg9OnsWQjeIpeSgVgPOnYGFvzPgV4CfYrhLdiFYI/ +izslPjfgLpmvq1JtGcVjwFEXGMKKqI+7pGN/1iuMeDQain8ovB7SS+ZZsWG4P2kvIXmcaVdDLm4 l3xVfJcIhMjtTZ9QeEjBtDCpeY8oRZPjKLY5alYEbcLIySO/Hv4AWGPMCrnsneGp4Br8T+Idohp3 MpvM2WoAAAAAAAA= --Apple-Mail=_A2B34ADC-6DE9-4C15-A605-0C85F060E27E--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5DA2A6D4-1FDA-48B3-A319-F26CB802E01B>