Date: Fri, 29 Dec 2017 15:45:49 -0800 From: Peter Grehan <grehan@freebsd.org> To: Andriy Gapon <avg@FreeBSD.org> Cc: freebsd-virtualization@freebsd.org Subject: Re: bhyve/amd: interrupt delivered when it shouldn't be? Message-ID: <450137ba-52dd-8b4c-63d2-3c3ce1909d69@freebsd.org> In-Reply-To: <42c22179-ae42-e4bb-e77d-a1d49fe634ed@FreeBSD.org> References: <42c22179-ae42-e4bb-e77d-a1d49fe634ed@FreeBSD.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Andriy, > The hardware is AMD. Ryzen ? > But what I see suggests that at this point a Local APIC timer interrupt gets > delivered to the thread. And that causes all the mess as the thread holding the > spinlock gets preempted. > > Does this ring a bell to anyone? I have seen something similar to this after about ~20 mins when doing a current -j 16 buildworld in a guest, with the symptom being a spinlock timeout, with one vCPU spinning in smp_targeted_tlb_shootdown() at smp_targeted_tlb_shootdown+0x352/frame 0xfffffe02c80098d0 smp_masked_invlpg() at smp_masked_invlpg+0x4c/frame 0xfffffe02c8009900 pmap_invalidate_page() at pmap_invalidate_page+0x191/frame 0xfffffe02c8009950 pmap_ts_referenced() at pmap_ts_referenced+0x7b3/frame 0xfffffe02c8009a00 vm_pageout() at vm_pageout+0xe04/frame 0xfffffe02c8009a70 ... and all the others eventually spinning on that held lock. However, NMIs are still able to get through (the post-panic ddb NMI IPI) so the VM isn't completely locked up - either an interrupt is missed, or a write isn't seen by the vCPU issuing the tlb shootdown. > Is there any suspect code? Not sure yet, but the interrupt-injection path could do with a close inspection. > It seems that we set v_intr_masking bit, so the rFLAGS / eFLAGS should be > completely virtualized. So, maybe a hardware issue? Hard to say. Running with all vCPUs pinned makes the problem go away, but that could just mean the issue is isolated to when vCPUs migrate. later, Peter.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?450137ba-52dd-8b4c-63d2-3c3ce1909d69>