From owner-freebsd-stable@FreeBSD.ORG Tue Sep 30 23:56:01 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 70D8FFB7; Tue, 30 Sep 2014 23:56:01 +0000 (UTC) Received: from mail-ig0-x231.google.com (mail-ig0-x231.google.com [IPv6:2607:f8b0:4001:c05::231]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 21B7986C; Tue, 30 Sep 2014 23:56:01 +0000 (UTC) Received: by mail-ig0-f177.google.com with SMTP id h3so11732igd.4 for ; Tue, 30 Sep 2014 16:56:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=PGpQ8S1vbIFu/vXcTSe0hjUI+ntLl+9zoPKHtJZ7K5E=; b=zfr/hVLksoIjIwLac3kjy9YmLeg2ZhnTRPAyGjv+vVdNV3dRXa2MIUOzmDBh3hXlmR 4ss4oaQeh4ZP6MAKsNZmuw4Z29TF9CmPmAXEi7erDQFScYjC1O1T1+P4nXh78chg325Q cWA5QU94QntF5miOBddv3dvl67cXJLlW29UtVhdUPI0Tb+yvE3UwQnENvGhTGoZd72oy b6raldLNh1xZUkuqiLQjpOZPTTrF0jwdD5CfBjSIBW+UUtqVGTwOxD6xaCcr/OCM1n+2 3TFre5JP6PHnRP2RsQRu8H//gljHnKhlvypUBQ/GH9O0LciINCL7kGETtSqo2+dbTZXr CHcg== MIME-Version: 1.0 X-Received: by 10.43.65.12 with SMTP id xk12mr62697539icb.9.1412121360355; Tue, 30 Sep 2014 16:56:00 -0700 (PDT) Received: by 10.50.87.130 with HTTP; Tue, 30 Sep 2014 16:56:00 -0700 (PDT) In-Reply-To: <542AFAE3.9030705@FreeBSD.org> References: <542AAA3C.1080803@ipfw.ru> <542AE376.6000003@FreeBSD.org> <542AFAE3.9030705@FreeBSD.org> Date: Tue, 30 Sep 2014 16:56:00 -0700 Message-ID: Subject: Re: 10.1-BETA2 possible kernel memory leak in routing table From: Rumen Telbizov To: glebius@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "Alexander V. Chernikov" , "freebsd-stable@freebsd.org" , brian@freebsd.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Sep 2014 23:56:01 -0000 Hello everyone, I think I got one step closer. At this point I am almost certain that the *leak is related to PF reloading tables.* Adding Gleb Smirnoff here since it might be something related to the rewrite of PF in FreeBSD 10. Here are the additional details. The following dtrace script shows kernel memory leaks per probe: #!/usr/sbin/dtrace -s #pragma D option quiet BEGIN { printf("%-20s %20s %20s %20s\n", "FUNCTION", "ALLOCATED", "FREE", "TOTAL"); } dtmalloc::$1:malloc { @malloc[probefunc] = sum(args[3]); @total[probefunc] = sum(args[3]); } dtmalloc::$1:free { @free[probefunc] = sum(args[3]); @total[probefunc] = sum(-args[3]); } tick-1s { printa("%-20.20s %20@d %20@d %20@d\n", @malloc, @free, @total); } Running it like this: # ./script2.d routetbl FUNCTION ALLOCATED FREE TOTAL routetbl 512 512 0 ... routetbl 46592 46592 0 routetbl 808960 444416 364544 ... routetbl 861184 496640 364544 routetbl 1623552 894464 729088 ... routetbl 1641984 912896 729088 The interval between the increase of 356KB is almost exactly 30 seconds. This made me think about what I run on this machine every 30 seconds. I do in fact have a service which every 30 seconds processes data, compiles PF tables and reloads the PF tables like this: /sbin/pfctl -T load -f /etc/firewall/pf.conf I stopped this service and the amount of memory remained static! No more leaks. I checked vmstat -m | grep routetbl and it had stopped growing. A manual pfctl -f /etc/firewall/pf.conf causes a leak of 356KB of memory every single time I run it. Brian Somers and I are currently looking into the source of PF in latest 10-STABLE and trying to figure out what is going on. We were able to replicate this problem on a 11-CURRENT (Sep 12th) machine as well. A simple PF ruleset with 1 rule and 1 table. Every few reloads of the firewall and vmstat -m | grep routetbl shows increased memory usage. For completeness - Alexander's dtrace script, with an aggregate twist, shows this (and only this) after about a minute: #!/usr/sbin/dtrace -s fbt:kernel:rt_msg2:entry { @rt_msg2[stack()] = count(); } fbt:kernel:rn_addroute:entry { @rn_addroute[stack()] = count(); } Output: kernel`sysctl_rtsock+0x274 kernel`sysctl_root+0x214 kernel`userland_sysctl+0x1d8 kernel`sys___sysctl+0x74 kernel`amd64_syscall+0x334 kernel`0xffffffff80900e0b 2340 kernel`sysctl_rtsock+0x64c kernel`sysctl_root+0x214 kernel`userland_sysctl+0x1d8 kernel`sys___sysctl+0x74 kernel`amd64_syscall+0x334 kernel`0xffffffff80900e0b 4680 Getting pretty close ... Regards, -- Rumen Telbizov Unix Systems Administrator