From owner-freebsd-hackers@freebsd.org Sun Apr 18 10:47:12 2021 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C68A65E48CF for ; Sun, 18 Apr 2021 10:47:12 +0000 (UTC) (envelope-from n7w@delta.emu.st) Received: from f3.bushwire.net (f3.bushwire.net [203.0.120.11]) by mx1.freebsd.org (Postfix) with ESMTP id 4FNRTq0pZvz3HJm for ; Sun, 18 Apr 2021 10:47:10 +0000 (UTC) (envelope-from n7w@delta.emu.st) Received: by f3.bushwire.net (Postfix, from userid 1001) id 8C2F33AE9B; Sun, 18 Apr 2021 20:47:00 +1000 (AEST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/simple; d=emu.st; s=2019; t=1618742820; bh=qVlQ/V+rqEKcswlPgbYXfZYQLI8=; h=Comments:Received:From:Comments:Message-ID:Content-Type:To: Subject:Mime-Version:Content-Disposition:Date; b=KBsVxYGmyCNDHbFmEgtTHMGWdukDDOJqrgQYUbmQ5A3boXLMQmWu+dZbO59A44GuH WtOWefltdh2jpRsobwEkN9UZZo5h3Is+Wq54eOlEBmHJdIAnHfYTSUU9Y8s93ZOgLG zQB1Xsg93xYNV5YInCQ8X/rPJF8GN/1+iuyEVisI=EVisI= Comments: QMDA 0.3a Received: (qmail 42282 invoked by uid 1001); 18 Apr 2021 10:47:00 -0000 From: "Mark Delany" Comments: QMDASubmit submit() 0.2.0-final Message-ID: <0.2.0-final-1618742820.474-0x878fa2@qmda.emu.st> Content-Type: text/plain; charset=utf-8 To: freebsd-hackers@freebsd.org Subject: Various problems with 13.0 amd64 on vultr.com Mime-Version: 1.0 Content-Disposition: inline Date: Sun, 18 Apr 2021 10:47:00 +0000 X-Rspamd-Queue-Id: 4FNRTq0pZvz3HJm X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; dkim=fail (headers rsa verify failed) header.d=emu.st header.s=2019 header.b=KBsVxYGm; dmarc=none; spf=pass (mx1.freebsd.org: domain of n7w@delta.emu.st designates 203.0.120.11 as permitted sender) smtp.mailfrom=n7w@delta.emu.st X-Spamd-Result: default: False [1.02 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; MV_CASE(0.50)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[emu.st]; NEURAL_SPAM_MEDIUM(0.89)[0.892]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[203.0.120.11:from:127.0.2.255]; R_SPF_ALLOW(-0.20)[+ip4:203.0.120.0/24]; DKIM_TRACE(0.00)[emu.st:-]; NEURAL_HAM_SHORT(-0.07)[-0.071]; R_DKIM_REJECT(1.00)[emu.st:s=2019]; NEURAL_HAM_LONG(-1.00)[-1.000]; RCVD_COUNT_ZERO(0.00)[0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RBL_DBL_DONT_QUERY_IPS(0.00)[203.0.120.11:from]; ASN(0.00)[asn:4764, ipnet:203.0.120.0/24, country:AU]; MAILMAN_DEST(0.00)[freebsd-hackers] X-Mailman-Approved-At: Sun, 18 Apr 2021 15:52:21 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Technical discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 18 Apr 2021 10:47:12 -0000 Hi all. I rarely if ever post here so if there's a better place, LMK. I've been running 12.2 on vultr.com instances for a long time without any issues. However I recently attempted an upgrade to 13.0 and the system now exhibits a number of issues. The most critical issue is that the system randomly wedged after running for a while (anywhere from 10 minutes to a couple of hours) requiring a reboot to recover. No console response or messages and limited network response (see below). No messages logged anywhere as best I can tell. The second issue is more annoying than critical: the system doesn't reboot with the reboot/shutdown commands. The shutdown sequence seems to complete but the reboot never occurs. I compiled and ran a "reboot(RB_AUTOBOOT | RB_VERBOSE)" but nothing interesting showed up. I have no idea whether the two issues are related excepting that neither occur with 12.2 Some details: - I first upgraded with freebsd-update and then tried with a fresh ISO image and completely overwrote the original file system. - I've tried both UFS and ZFS root file systems. - I tried with a fresh VM instance in case there was some sort of per-instance glitch - The system is 99% idle with no memory pressure. It normally runs nsd, openntpd and a few other processes installed via pkg, but nothing wierd as best I can tell. - it has no kernel modules manually loaded - It's configured with ipv4 and ipv6 and when it gets wedged I get a ping response from the ipv6 address, but not from ipv4. Furthermore, if I try a tcp connection to ipv6 I get a connection setup, but no data. - The VM is configured as a single-CPU system - I haven't raised the issue with vultr yet. Thought I'd see what the hive-mind thinks first. Not that it will surprise anyone, but I recently spun up 13.0 in Virtualbox on a lab machine as well as on a different VM provider without any problems, so it's probably something relatively unique to vultr. That this is a virtually idle system on a single CPU with no oddball or unusual kernel modules or network configs makes the situation surprising to me. There is no pattern that I'm yet able to discern. The main thing I have left to try is to boot the system without any networking activated, but apart from that I'm out of ideas in terms of identifying the root cause. So my questions are: 1. Anyone else having the same issue? Or not having the same issue? 2. Clues on how to diagnose? This is a non-critical system so I can try anything that anyone suggests but I'm not particularly familiar with kernel-level debugging so a bit of hand-holding might be needed if you have suggestions. For those unfamiliar with vultr's VMs, here's the first part of dmesg: FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr 9 04:24:09 UTC 2021 root@releng1.nyi.freebsd.org:/usr/obj/usr/src/amd64.amd64/sys/GENERIC amd64 FreeBSD clang version 11.0.1 (git@github.com:llvm/llvm-project.git llvmorg-11.0.1-0-g43ff75f2c3fe) VT(vga): text 80x25 CPU: Intel Xeon Processor (Cascadelake) (2993.02-MHz K8-class CPU) Origin="GenuineIntel" Id=0x50656 Family=0x6 Model=0x55 Stepping=6 Features=0x783fbff Features2=0xfffa3203 AMD Features=0x2c100800 AMD Features2=0x21 Structured Extended Features=0xd18307a9 Structured Extended Features2=0x808 Structured Extended Features3=0xa4000000 XSAVE Features=0x1 IA32_ARCH_CAPS=0x2b Hypervisor: Origin = "KVMKVMKVM" real memory = 1073741824 (1024 MB) avail memory = 997744640 (951 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" random: unblocking device. ioapic0 irqs 0-23 Timecounter "TSC-low" frequency 1496510010 Hz quality 800 in case it shows up anything odd to those who can decode this sort of stuff. Mark.