From owner-freebsd-current@FreeBSD.ORG Wed Jun 23 05:40:22 2004 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B075516A4CE; Wed, 23 Jun 2004 05:40:22 +0000 (GMT) Received: from apollo.backplane.com (apollo.backplane.com [216.240.41.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 764E043D53; Wed, 23 Jun 2004 05:40:22 +0000 (GMT) (envelope-from dillon@apollo.backplane.com) Received: from apollo.backplane.com (localhost [127.0.0.1]) i5N5e2x7043867; Tue, 22 Jun 2004 22:40:02 -0700 (PDT) (envelope-from dillon@apollo.backplane.com) Received: (from dillon@localhost) by apollo.backplane.com (8.12.9p2/8.12.9/Submit) id i5N5e1N7043866; Tue, 22 Jun 2004 22:40:01 -0700 (PDT) (envelope-from dillon) Date: Tue, 22 Jun 2004 22:40:01 -0700 (PDT) From: Matthew Dillon Message-Id: <200406230540.i5N5e1N7043866@apollo.backplane.com> To: Gerrit Nagelhout References: cc: Julian Elischer cc: John Baldwin cc: current@FreeBSD.org Subject: RE: STI, HLT in acpi_cpu_idle_c1 X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 23 Jun 2004 05:40:22 -0000 :Thanks for the detailed info on this. It looks like CPU1 is trying :to service the interrupt because PPR = 0xf0, and TPR = 0x00. It is Yup. :also the only CPU that has a bit set in ISR. In this case, CPU 3 :was initiating the IPI (although I don't know why its icr_lo is :0xc00f6 because I was expecting it to be 0xc00f3 (and it was in :previous lockups). I still have no idea why CPU1 is not handling :this interrupt though. I am still getting used to this emulator, but :I think the values I am reading are believable: Yup. It would be nice to see the contents of the TMR and IRR as well, but the ISR is very important. If the ISR bit is set it means that the APIC delivered the interrupt to the cpu but the cpu has not yet EOI'd it. This would account for the contents of the PPR (it is based on the highest pending ISR bit. When you EOI the APIC it knows what to EOI based on the PPR). It is possible that the cpu got the interrupt and started processing it, then deadlocked in a mutex and went to idle before it had a chance to EOI it. I haven't looked at the IPI handling path in 5.x recently so this might not be a possible case, but so far it's the only thing that fits the bill. Perhaps it tried to use a sleep mutex when it really should have been using a spin mutex. Once the APIC has delivered the interrupt to the cpu nothing more happens to that interrupt until it is EOId. If the cpu has not EOI'd that interrupt and HLT's, it will *NOT* be reinterrupted by that particular interrupt. What I would do now is (attempt to) get some sort of core that you can gdb on or attach gdb too. I'm afraid I can't help much there. But what you want to do is get a stack backtrace for every single thread in the system, looking for a stuck mutex in an IPI path interrupting the normal thread's operation. Again, I don't know if this scenario is possible, but it's the only thing I can think of. Perhaps John can narrow down the possibilities some more. On the ICR_LO values: I don't know what one should expect for the vector portion. The 'c' in the c00f6 is definitely correct (the last command was sent to all excluding self). The status fields are 0 (Edge trigger, APIC has accepted the command, physical mode, fixed command). Since all APICs have S=0 we know that the problem is NOT an APIC-APIC deadlock. The APICs look to be in good shape. :CPU 0 :TPR: 0x0 << not priority masked :PPR: 0x0 << nothing pending :icr_lo:0xf3 << ready to accept command :CPU 1 :ID: 0x7000000 :TPR: 0x0 << not priority masked :PPR: 0xf0 << priority of delivered but not-yet EOI'd interrupt :icr_lo:0xf3 << ready to accept command :ISR7: 0x80000 << interrupt delivered and is in-service, not yet EOI'd :CPU 2 :TPR: 0x0 << not priority masked :PPR: 0x0 << nothing pending :icr_lo:0xfb << ready to accept command :CPU 3 :ID: 0x1000000 :TPR: 0x0 << not priority masked :PPR: 0x0 << nothing pending :icr_lo:0xc00f6 << ready to accept new command (previous command was accepted), last sent command was IPI to all-but-self. : :Gerrit -Matt Matthew Dillon