From owner-freebsd-hackers@freebsd.org Thu Aug 4 11:13:38 2016 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2C438BAECC9 for ; Thu, 4 Aug 2016 11:13:38 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-yb0-x231.google.com (mail-yb0-x231.google.com [IPv6:2607:f8b0:4002:c09::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id DCD9712CD; Thu, 4 Aug 2016 11:13:37 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-yb0-x231.google.com with SMTP id g133so4085037ybf.2; Thu, 04 Aug 2016 04:13:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=m9Fzn99ZR2NVpraEr2YHWKnFlQREaul/ZB0I9jMepyI=; b=LLsWE9QRf41gpIGe/I4Zm13RErRSl0LLuJ8STa8L4IkPnzRkx4nz11zr0VqhHPMxyW dQLRYoGFVyjzrxhPVWdEZ4lkxrDyDZ1mW4S7Gt0kHoY7uhk130Dv2NKp6/lRRe46wFmX 0fjCMwngoOQuXIuUdVmVgwlUOf5t0zwVm0mSwP+fbgJjp59At9ZYwocpYZVYhd++OHLd vPczxqQSx87ocfFrXZf6nOKI03e5b/PjtSoa/r8nAPQCpJVpFAu8S9NcqM8z6NbTbQel TFFCCy2HMRsGWI/E7lhNDeQHMcYTaf/PWRKhIPrhl6xLxtbJiSaBKWkc8nlIFvHux0S8 rSaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=m9Fzn99ZR2NVpraEr2YHWKnFlQREaul/ZB0I9jMepyI=; b=b+2WNEO7cUV2XfNNQTReHb8ZSlAVE2kkCEd+6lugNEEtj3itnyjvvc4P8azmDa/T0x WzPIMHp/KSokwGVAlgLmFcrejjOk293FzfiWaRr8Ml/HsvAe1C0gcW6RhvNlEhGZKcf3 zpbwlMr2lMHxDSFcFzbUsL1C7P6ldQTGXI6tgFWMZHSESL64oW9PHXBIA5mSpU99Anz1 rFyPNYrr0n4BoiVg+PH1HGLmr+KOZfG0856Wkpp+DfsKNNrMH9zvTUf46aROmpSERXeO 6CyEL+RTcKyVxyja9bN8n7L5Xsp96VFuGhsJOdtBiitfnnMsZ0koWgHB1Kl3f1ZovM0A vTzw== X-Gm-Message-State: AEkoouuxYx37oFHPyOUbDjrJqgjEKEar83aOnMlf2ltYnwSRCSETAvvIk3MU458NSjsgDpfmYPn/vb2oS9OHyA== X-Received: by 10.37.78.133 with SMTP id c127mr52401198ybb.41.1470309216899; Thu, 04 Aug 2016 04:13:36 -0700 (PDT) MIME-Version: 1.0 Received: by 10.37.161.37 with HTTP; Thu, 4 Aug 2016 04:13:36 -0700 (PDT) In-Reply-To: References: <5cc825d5-9ed7-efac-b711-60a8d4b18cc4@FreeBSD.org> From: Zaphod Beeblebrox Date: Thu, 4 Aug 2016 07:13:36 -0400 Message-ID: Subject: Re: How to get better debugging for the kernel. To: "K. Macy" Cc: Andriy Gapon , Samy Bahra , FreeBSD Hackers , Ryan Stone Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 04 Aug 2016 11:13:38 -0000 Thanks for the tips. So far, (and this is in diagnosing bug 211535), I've managed to compile the kernel with -O0. Someone on #FreeBSD told me to do it with "CFLAGS=-g -O0 make" ... which seems to have worked. At the very least, I have managed to find a frame on the stack that told me where in the filesystem was triggering the problem. Turns out that icinga2 (a monitoring package) manages to create a directory of 530k small files. It renames() two files each minute into the directory. At 530k files, this panic's the kernel every 12h or so. At lower levels of files, it panic's the kernel less. I have not setup a test system at home to try to induce this ... and if shutting off icinga2 fixes this (or shutting off the statistics module), then I'm loathe to keep the production server as a guinea pig, but I do have dumps for people to look at if they're interested. On Thu, Aug 4, 2016 at 4:07 AM, K. Macy wrote: > On Wed, Aug 3, 2016 at 12:53 PM, Andriy Gapon wrote: > > On 03/08/2016 20:14, Ryan Stone wrote: > >> Are you using the kgdb from the base system or from ports(it's a part of > >> devel/gdb)? The kgdb in ports is significantly better. If you haven't > >> tried the version from ports already, definitely do that first. > > > > kgdb 7.x from ports is certainly more powerful than the old base kgdb, > > but clang with O2 optimizations seems to be too much even for it. > > Samy did a good presentation about this issue. I'm hoping I can get > him to put his slides on line. Evidently clang is much more simplistic > about how it treats callee saved registers. In essence clang will > always err on the side of saying "optimized out" even when it has > sufficient state to know otherwise. Gcc, on the other hand will > sometimes incorrectly infer that a value is valid when it is in fact > not. > > I have been building some kernels with clang with dwarf4 enabled (and > thus needed to use kgdb 7.x from ports). Contrary to what I have heard > from some others I have found it to have virtually no added benefit. > > There is evidently some mechanism for appending an epilogue to > functions. This would allow us to conditionally force clang to treat > all function argument as being live to the end of the function. > > -M > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >