From owner-freebsd-stable@FreeBSD.ORG Tue Sep 30 18:04:43 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6269ECB6; Tue, 30 Sep 2014 18:04:43 +0000 (UTC) Received: from mail-ie0-x233.google.com (mail-ie0-x233.google.com [IPv6:2607:f8b0:4001:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 21B9B217; Tue, 30 Sep 2014 18:04:43 +0000 (UTC) Received: by mail-ie0-f179.google.com with SMTP id ar1so118447iec.24 for ; Tue, 30 Sep 2014 11:04:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=8HVfsh5IdlMZannZ3E1ByEpBCL1PWpkG+Tww4gOkwb0=; b=IKU3KJn5WpagYvM8rWGJvLIpPxhYKkxo3Kza+8pQLzG3h2KdY7qbpeLurRgrRSIViZ xisY2diZ8IqvyztECf3y8lli+nWpeGyj0xIkYxIGanHQRsz2KXAAaLOw7o90SNXP1wcZ p9CVlykTLrgJcCGlZAWOtylpXAV71QdAb0bZ4AW+5bxVzWUn2wXOaZjXgSrgKz19HX5p Ge40BNYX98NQJzl/eR+busiLrcRZY4hWaJGAYVdIWjF4AMMOHX/UfDqiITNGyxNaLOAO C3EvoaB5iDBvCJk2AwFotU9XAiEFfKB0BJY9XxPkx3YtRsW8hmCIA9IZ9gRn2hGr1Ntw LUJA== MIME-Version: 1.0 X-Received: by 10.50.61.144 with SMTP id p16mr10687313igr.38.1412100282379; Tue, 30 Sep 2014 11:04:42 -0700 (PDT) Received: by 10.50.87.130 with HTTP; Tue, 30 Sep 2014 11:04:42 -0700 (PDT) In-Reply-To: References: <542AAA3C.1080803@ipfw.ru> <542AE376.6000003@FreeBSD.org> Date: Tue, 30 Sep 2014 11:04:42 -0700 Message-ID: Subject: Re: 10.1-BETA2 possible kernel memory leak in routing table From: Rumen Telbizov To: "Alexander V. Chernikov" Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 Cc: "freebsd-stable@freebsd.org" X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 30 Sep 2014 18:04:43 -0000 Also worth showing the "progress" overnight: This is yesterday from my first report: Type InUse MemUse HighUse Requests Size(s) Mon Sep 29 18:27:55 UTC 2014 routetbl 5988792 2888491K - 14285826 32,64,128,256,512,2048 This is now: Tue Sep 30 18:00:07 UTC 2014 routetbl 7948594 3835284K - 18822335 32,64,128,256,512,2048 Leaked 946793K (3835284-2888491) for close to 24 hours. So I am leaking about 1GB per day Here's the graph: [image: Inline image 1] On Tue, Sep 30, 2014 at 10:56 AM, Rumen Telbizov wrote: > Hi Alexander, > > >> Sorry. That meant to be "route -Arn". >> It would be even better if you show something like >> "route -Arn ; sleep 10; route -Arn" >> >> > OK I would assume you really meant netstat ;) I apologize for not pasting > the entire routing table (security considerations). If you're only after > the size of it - like I said it's pretty static and doesn't change very > often. Is there anything specific that you're after?: > > netstat -rna | wc -l ; sleep 10; netstat -rna | wc -l > 145 > 145 > > If you do need something specific from it I'll spend some time to > anonymize it later. > > > >> Ok. Are there any (relevant) messages floating inside route socket? >> e.g does "route -n monitor" shows a lot of output? >> > > time route -n monitor > > real 13m15.960s > user 0m0.000s > sys 0m0.002s > > No output. No changes whatsoever! dmesg doesn't report anything either. > > > >> Very strange. Do you have "normal" ipv4/ipv6 traffic? (e.g. not tunneled, >> not ipsec, not all-fragmented, etc?) >> ixgbe marks all packets by received queue id number, and queue 0 is >> always selected for "corner case" traffic. >> I see more or less equal traffic distribution in you original post, so >> this looks very strange for me. >> > > > Most of it should be pretty standard. Very few fragments and some ipsec > (not terminated on the firewall). Again without this option - nothing goes > over ix1 for some reason. > > > >> Can you show "sysctl dev.ix" output? (and any ixgbe tunables set if any)? >> > > I do have 3 settings that I changed via loader.conf as reported in the > original email. Please find the full output of loader.conf, sysctl.conf > as well as sysctl hw.ix and sysctl dev.ix at http://pastebin.com/M5Stnfvt > > > Thank you for your help. > > Regards, > -- > Rumen Telbizov > Unix Systems Administrator > -- Rumen Telbizov Unix Systems Administrator