Date: Mon, 19 Apr 2021 17:45:23 -0700 From: jon <jon@brainville.io> To: Mark Delany <n7w@delta.emu.st> Cc: freebsd-hackers@freebsd.org Subject: Re: Various problems with 13.0 amd64 on vultr.com Message-ID: <YH4kIz31MwE0nMIf@brainville.io> In-Reply-To: <0.2.0-final-1618742820.474-0x878fa2@qmda.emu.st> References: <0.2.0-final-1618742820.474-0x878fa2@qmda.emu.st>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sun, Apr 18, 2021 at 10:47:00AM +0000, Mark Delany wrote: > Hi all. > > I rarely if ever post here so if there's a better place, LMK. > > I've been running 12.2 on vultr.com instances for a long time without any issues. However > I recently attempted an upgrade to 13.0 and the system now exhibits a number of issues. > > The most critical issue is that the system randomly wedged after running for a while > (anywhere from 10 minutes to a couple of hours) requiring a reboot to recover. No console > response or messages and limited network response (see below). No messages logged anywhere > as best I can tell. > > The second issue is more annoying than critical: the system doesn't reboot with the > reboot/shutdown commands. The shutdown sequence seems to complete but the reboot never > occurs. I compiled and ran a "reboot(RB_AUTOBOOT | RB_VERBOSE)" but nothing interesting > showed up. > > I have no idea whether the two issues are related excepting that neither occur with 12.2 > > > Some details: > > - I first upgraded with freebsd-update and then tried with a fresh ISO image and > completely overwrote the original file system. > > - I've tried both UFS and ZFS root file systems. > > - I tried with a fresh VM instance in case there was some sort of per-instance glitch > > - The system is 99% idle with no memory pressure. It normally runs nsd, openntpd and a few > other processes installed via pkg, but nothing wierd as best I can tell. > > - it has no kernel modules manually loaded > > - It's configured with ipv4 and ipv6 and when it gets wedged I get a ping response from > the ipv6 address, but not from ipv4. Furthermore, if I try a tcp connection to ipv6 I > get a connection setup, but no data. > > - The VM is configured as a single-CPU system > > - I haven't raised the issue with vultr yet. Thought I'd see what the hive-mind thinks > first. > > Not that it will surprise anyone, but I recently spun up 13.0 in Virtualbox on a lab > machine as well as on a different VM provider without any problems, so it's probably > something relatively unique to vultr. > > That this is a virtually idle system on a single CPU with no oddball or unusual kernel > modules or network configs makes the situation surprising to me. There is no pattern that > I'm yet able to discern. The main thing I have left to try is to boot the system without > any networking activated, but apart from that I'm out of ideas in terms of identifying the > root cause. > > > So my questions are: > > 1. Anyone else having the same issue? Or not having the same issue? > 2. Clues on how to diagnose? This is a non-critical system so I can try anything that > anyone suggests but I'm not particularly familiar with kernel-level debugging so a bit > of hand-holding might be needed if you have suggestions. > > For those unfamiliar with vultr's VMs, here's the first part of dmesg: > > FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr 9 04:24:09 UTC 2021 > root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 > FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe) > VT(vga): text 80x25 > CPU: Intel Xeon Processor (Cascadelake) (2993.02-MHz K8-class CPU) > Origin="GenuineIntel" Id=0x50656 Family=0x6 Model=0x55 Stepping=6 > Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2> > Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV> > AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> > AMD Features2=0x21<LAHF,ABM> > Structured Extended Features=0xd18307a9<FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,AVX512F,AVX512DQ,CLFLUSHOPT,CLWB,AVX512CD,AVX512BW,AVX512VL> > Structured Extended Features2=0x808<PKU,AVX512VNNI> > Structured Extended Features3=0xa4000000<IBPB,ARCH_CAP,SSBD> > XSAVE Features=0x1<XSAVEOPT> > IA32_ARCH_CAPS=0x2b<RDCL_NO,IBRS_ALL,SKIP_L1DFL_VME,MDS_NO> > Hypervisor: Origin = "KVMKVMKVM" > real memory = 1073741824 (1024 MB) > avail memory = 997744640 (951 MB) > Event timer "LAPIC" quality 600 > ACPI APIC Table: <BOCHS BXPCAPIC> > random: registering fast source Intel Secure Key RNG > random: fast provider: "Intel Secure Key RNG" > random: unblocking device. > ioapic0 <Version 1.1> irqs 0-23 > Timecounter "TSC-low" frequency 1496510010 Hz quality 800 > > in case it shows up anything odd to those who can decode this sort of stuff. > Hello, I happen to be running FreeBSD 13.0-RELEASE on a Vultr instance as well, but haven't had any problems in the ~4 days since I updated from 12 RELEASE. My VM is a single CPU with 2G memory and UFS for a filesystem. I do see that our VMs have different CPUs listed. Here is the first part of my dmesg : FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr 9 04:24:09 UTC 2021 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe) VT(vga): text 80x25 CPU: Intel Core Processor (Skylake, IBRS) (3792.08-MHz K8-class CPU) Origin="GenuineIntel" Id=0x506e3 Family=0x6 Model=0x5e Stepping=3 Features=0x783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2> Features2=0xfffa3203<SSE3,PCLMULQDQ,SSSE3,FMA,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND,HV> AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM> AMD Features2=0x21<LAHF,ABM> Structured Extended Features=0xfb9<FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM> Structured Extended Features3=0x84000000<IBPB,SSBD> XSAVE Features=0x1<XSAVEOPT> Hypervisor: Origin = "KVMKVMKVM" real memory = 2147483648 (2048 MB) avail memory = 2047262720 (1952 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <BOCHS BXPCAPIC> random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 <Version 1.1> irqs 0-23 Timecounter "TSC-low" frequency 1896040542 Hz quality 800 KTLS: Initialized 1 threads - Jon
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?YH4kIz31MwE0nMIf>