From owner-freebsd-arch@FreeBSD.ORG Thu Nov 13 09:38:33 2014 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 89FB8E58; Thu, 13 Nov 2014 09:38:33 +0000 (UTC) Received: from mail-wg0-x22f.google.com (mail-wg0-x22f.google.com [IPv6:2a00:1450:400c:c00::22f]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2CEBDCA2; Thu, 13 Nov 2014 09:38:33 +0000 (UTC) Received: by mail-wg0-f47.google.com with SMTP id a1so16412461wgh.34 for ; Thu, 13 Nov 2014 01:38:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=wntqBGuWdvav0aCjACtVxdh1XpfQJBCGJg6iJI/AwtA=; b=PaMPpZEZDvG3KJIR90BvDPF0BIKq24WJ7xOSUKWkpWD2oki9jpbNrLK7vruFYJoNJO CjaxLEOw75bNO1yC3v4q+Dszu+glK6Gv6ygKgVKuPsReyyI8qZuunCDd168EmCYevywA C+Vz2KTRZYFE6jBvB511CrQeFA7ajyw4llOr1m6R/XfnBt5ljaf0uJySCK7WPw5sESK5 wIK0oML2qoj0Zy/n/PIzKPUXWN+Y7yGcCK+1e4/9cOD9LJul8CBUmgzhaHKjzlAMAFH0 1MuFytXa2AYCX6m2/ojJ8LcQqaSh+vNIdL7Q4k1zD0xnp11Hc4J8Dvn6ObkenqHe9uiZ A0jQ== MIME-Version: 1.0 X-Received: by 10.194.47.226 with SMTP id g2mr2176667wjn.68.1415871511518; Thu, 13 Nov 2014 01:38:31 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.216.106.136 with HTTP; Thu, 13 Nov 2014 01:38:31 -0800 (PST) In-Reply-To: <5464764E.9080308@freebsd.org> References: <20141112212613.21037929@kan> <546472DA.3080006@freebsd.org> <5464764E.9080308@freebsd.org> Date: Thu, 13 Nov 2014 01:38:31 -0800 X-Google-Sender-Auth: 3mNnIBUBkZ65LoLVAEE1Bvq8xrU Message-ID: Subject: Re: Questions about locking; turnstiles and sleeping threads From: Adrian Chadd To: Alfred Perlstein Content-Type: text/plain; charset=UTF-8 Cc: "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 13 Nov 2014 09:38:33 -0000 It looks like the initial firings are because the check I put in didn't check to see if it's MPSAFE. eg: ip6_input -> tcp6_input -> tcp_input -> tcp_do_segment -> tcp_timer_active -> callout_stop_safe; called with tcpinp held. closefp() -> closef() -> fdrop -> soclose() -> sofree() -> tcp_usr_detach() -> tcp_discardcb() -> callout_stop_safe() with the tcpinp and tcp locks held. ioctl -> sys_ioctl-> devfs_ioctl_f -> acpi_ackSleepState -> callout_stop_safe; with ACPI global lock held; suspend path -> hdaa_suspend -> callout_stop_safe() with HDA driver mutex held So we can't just put the simple witness check from _sleep() in _callout_stop_safe(), it looks like it's mis-firing on MPSAFE callouts (which the tcp timers all are) and that won't go via the sleepq. It looks like the acpi callout is also mpsafe, as well as the HDA jack callout. However, I did pick up this: detach path -> usbd_transfer_drain() -> usbd_transfer_stop() -> ehci_device_intr_close() -> usbd_transfer_done() -> callout_stop_safe() with USB HUB mutex held The usbd_transfer_done() callout is initialised with a mutex, but the witness code should've detected it wasn't callout->c_lock and thus warned. -adrian On 13 November 2014 01:13, Alfred Perlstein wrote: > > On 11/13/14, 1:09 AM, Adrian Chadd wrote: >> >> On 13 November 2014 00:59, Alfred Perlstein wrote: >>> >>> On 11/12/14, 11:25 PM, Adrian Chadd wrote: >>>> >>>> On 12 November 2014 19:48, Adrian Chadd wrote: >>>>> >>>>> kan pointed out that we should likely do a WITNESS_WARN() in the >>>>> relevant spots in the callout code so we catch these before it >>>>> happens. >>>>> >>>>> I'll see what we can add to -HEAD to be pro-active about it. >>>> >>>> Amusingly, I tried adding it and it made my laptop turn to soup very >>>> quickly - among other things, the TCP timer callouts are all setup >>>> with non sleep locks held. >>>> >>> Howso? You only have to worry about callout_drain(), most other callout >>> functions should be safe-ish.... >> >> yeah, except for all the places where they drain the timer once >> something happens so it doesn't fire. >> >> :) > > > What is the backtrace...? > >