From owner-freebsd-current@FreeBSD.ORG Sat Dec 3 22:59:16 2005 Return-Path: X-Original-To: current@FreeBSD.org Delivered-To: freebsd-current@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2642916A420 for ; Sat, 3 Dec 2005 22:59:16 +0000 (GMT) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 28A1F43D55 for ; Sat, 3 Dec 2005 22:59:14 +0000 (GMT) (envelope-from andre@freebsd.org) Received: (qmail 95738 invoked from network); 3 Dec 2005 22:56:34 -0000 Received: from c00l3r.networx.ch (HELO freebsd.org) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 3 Dec 2005 22:56:34 -0000 Message-ID: <43922340.4CAF7157@freebsd.org> Date: Sat, 03 Dec 2005 23:59:12 +0100 From: Andre Oppermann X-Mailer: Mozilla 4.8 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Robert Watson References: <20051203150119.B3216@fledge.watson.org> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: current@FreeBSD.org Subject: Re: mbuf cluster leaks in -CURRENT X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 03 Dec 2005 22:59:16 -0000 Robert Watson wrote: > > Yesterday I sat down to run some benchmarks on phk's changes to the process > time measurement system for scheduling, and discovered SMP boxes were wedging > in [zonelimit] when running netperf tests. I quickly tracked this down to an > mbuf cluster leak: > > /zoo/rwatson/netperf/bin/netserver > while (1) > echo "" > netstat -m | grep mbuf > /zoo/rwatson/netperf/bin/netperf -l 30 >& /dev/null > end > > Result of: > > CVS Date Description Leak? > 2005/12/3 sample yes > 2005/11/28-2005/11/29 rwatson sosend changes - > 2005/11/25 sample yes > 2005/11/15 sample yes > 2005/11/02-2005/11/05 andre cluster changes - > 2005/10/25 sample no > 2005/10/15 sample no > 2005/10/1 sample no > 2005/09/27 rwatson removes mbuf counters - > 2005/09/16 sample no > > The reason for the wedge is that NFS based systems don't like running out of > mbuf clusters. It turns out that the reason I likely didn't notice this > previously was that I was running the test boxes in question without ACPI, and > for whatever reason, the race becomes many times more serious with ACPI turned > on. It was leaking without ACPI, but since it was slower, I wasn't noticing > since I had the machines up for much shorter tests. Here's a sampling of > kernel dates and whether or not the leak was present in a kernel from the > date, as well as the dates of a few changes I was worried were likely causes: > > 769/641/1410 mbufs in use (current/cache/total) > 768/204/972/25600 mbuf clusters in use (current/cache/total/max) > > 769/4991/5760 mbufs in use (current/cache/total) > 4341/905/5246/25600 mbuf clusters in use (current/cache/total/max) > > 769/8456/9225 mbufs in use (current/cache/total) > 7901/801/8702/25600 mbuf clusters in use (current/cache/total/max) > > 769/11786/12555 mbufs in use (current/cache/total) > 11242/788/12030/25600 mbuf clusters in use (current/cache/total/max) > > 769/15236/16005 mbufs in use (current/cache/total) > 14570/916/15486/25600 mbuf clusters in use (current/cache/total/max) > > 769/18566/19335 mbufs in use (current/cache/total) > 17948/866/18814/25600 mbuf clusters in use (current/cache/total/max) > > I've not really had a chance to investigate the details of the leak -- the > number of used (allocated) mbufs remains low, but the cache number grows > steadily. However, the dates suggest that it was the mbuf cluster cleanup > work you did that introduced the problem (although don't guarantee it). This seems to be the same problem I described in rev. 1.14 of kern_mbuf.c where mbuf+clusters from the packet zone (pre-combined m+c) never get free'd back to their original pools. The numbers from netstat -m support that assumption. It doesn't (and can't) show the number of cached m+c in the packet zone. Mbuf's in packet zone account as cached in the mbuf zone because the packet zone is a secondary zone to it. The clusters in use are not leaked but attached to all those mbufs in the packet zone. The cluster zone doesn't know about the packet zone and accounts them as used. This pseudo-leak is not from my changes (as it is a UMA bug) but gets amplified by use of kernel subsystems which make heavy use of m+c from the packet zone. While my changes triggered this problem too by changing the way packets get free'd back to the UMA mbuf, cluster and packet zones it was reverted in 1.14. The other refcount changes do not cause any such effect. It may very well be that some part of the network stack switched from allocating mbuf and cluster separately to pre-combined packets. That would explain the 'sudden' appearance of the problem. The right fix is to have UMA free back mbuf+clusters from the packet zone to their native zones. This should not be done with high/low watermarks but a median and positive/negative deviation method. Refills and drains to/from the packet zone should happen in batches and not for single requests or free's to be efficient. I'll look into it tomorrow. I may have to summon Bosko for some help on the secondary zone stuff as he introduced this feature. -- Andre