From owner-freebsd-embedded@FreeBSD.ORG Sun Jul 15 10:31:48 2012 Return-Path: Delivered-To: freebsd-embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 334441065676; Sun, 15 Jul 2012 10:31:48 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id E667F8FC08; Sun, 15 Jul 2012 10:31:47 +0000 (UTC) Received: by pbbro2 with SMTP id ro2so8984058pbb.13 for ; Sun, 15 Jul 2012 03:31:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=R5/FWtjGHqXTYcdzcX51khL37Dv3k6XjHvwnBbQ7Jn8=; b=fAHWciHoymqOq4d6rauPFcfEJJb8qa/L5vq94cBmlRmFYoaAhFTlu0kUru8lJlkNWQ wC4gmPx7ubgXMjIG1gQCIX1KXKFiOIXe2hINYZxYtLqiqL0W1WzXO2TuuAJkwsV+Nmo3 Bp0mS9ctp17drXe1Yabb4Kv9zU4DFzsEOZ4U/JpF5tqnx+NKQFa7phbeYutyhY1gOBEe vYLcDuwLx9PN0hqAEcEhHJtWC4GobdzlDskJOYPu/ycJLPVFh6v2K3HXhw1126edz2sb SGaDAco7gh3pAtSNEeYs3a01hlowdFL3UtGz9Hhgq/xol5ZZEg33uK878DKvIzMs60Rp NJaw== MIME-Version: 1.0 Received: by 10.68.220.193 with SMTP id py1mr18612640pbc.4.1342348307385; Sun, 15 Jul 2012 03:31:47 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.195.102 with HTTP; Sun, 15 Jul 2012 03:31:47 -0700 (PDT) In-Reply-To: <4B538596-937B-46F3-AF8F-17F34BE0C92D@bsdimp.com> References: <1341745590.2740.17.camel@manbearpig.dynamic.weites.net> <201207081805.33574.bschmidt@freebsd.org> <1341841445.2540.10.camel@manbearpig.dynamic.weites.net> <1341849727.2540.11.camel@manbearpig.dynamic.weites.net> <1342195983.2336.35.camel@manbearpig.dynamic.weites.net> <4B538596-937B-46F3-AF8F-17F34BE0C92D@bsdimp.com> Date: Sun, 15 Jul 2012 03:31:47 -0700 X-Google-Sender-Auth: 3LqaV0qiGlHjs_JBD7-ntyLejfc Message-ID: From: Adrian Chadd To: Warner Losh Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Bernhard Schmidt , freebsd-embedded@freebsd.org, Harm Weites Subject: Re: TP-Link wr1043nd out of swap space X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Jul 2012 10:31:48 -0000 Hi, I would really appreciate it if people (read; not me) would be able to do the digging needed to get to the bottom of user/kernel memory usage. I really need to focus on just the net80211/wifi stack side of things. I'm going to focus on getting the ath(4) memory usage down over the next few months so it remains feasible to run on 32MB platforms, as those still ship. But I can't keep the rest of the kernel and userland in check. Thanks, Adrian On 13 July 2012 09:45, Warner Losh wrote: > > On Jul 13, 2012, at 10:13 AM, Harm Weites wrote: > >> Hi, >> >> the firmware posted on Adrian's google projects page works ok (which is >> from last December), it even says 10+M free memory. I've flashed my unit >> with a (old) kernel/world from r230847, it leaves ~5M mem available. The >> biggest difference between both is probably my inclusion of gif/pf >> devices, and leaving out the WITNESS/INVARIANTS stuff. Though that >> should probably not make such a huge difference. >> >> I've tried updating my tree to some revisions later, but at r233000 I'm >> left with ~4M available. There are some nice fixes in various places >> after that revision, so I'm eager to get something higher to work. And >> it would be nice to pin-point the cause of the error :) > > It has been my experience that if you want to run general purpose things = on a device, you need at least 8M[*] of ram free after boot with the daemo= ns running. Any less than that, and it becomes very hard to cope with fluc= tuations in load and usage. You'll likely need to go looking for things to= trim, I'm sorry to say. > > Warner > > [*] I've run certain special purpose machines closer to the edge, but the= load on them didn't vary much at all, and they had a very constrained set = of tasks they had to accomplish. And this was back in the 4.x time frame, = and dynamic memory in the kernel is somewhat more vigorous these days. > >> Regards >> >> Marcelo Araujo schreef op vr 13-07-2012 om 10:32 [+0800]: >>> Hello Harm, >>> >>> Did you have any progress to fix your problem, I'm quite interested on >>> it. >>> >>> Best Regards, >>> - Araujo >>> >>> 2012/7/10 Harm Weites >>> Hi, >>> >>> just a typo in my message, not in make.conf :) >>> >>> regards >>> >>> Marcelo Araujo schreef op ma 09-07-2012 om 23:27 [+0800]: >>>> Hello Harm, >>>> >>>> >>>> There is a typo is must be MALLOC_PRODUCTION=3DYES instead of >>> MALLOC >>>> PRODUCION=3DYES. Maybe you could double check! >>>> >>>> >>>> Best Regards, >>>> - Araujo >>>> >>>> 2012/7/9 Harm Weites >>>> Hi Bernard, >>>> >>>> thanks for your suggestion. I've added >>> MALLOC_PRODUCION=3DYES >>>> to /etc/make.conf, and also removed make option >>> DEBUG=3D-g from >>>> the kernel >>>> config. The error still exists though. >>>> >>>> Regards >>>> >>>> Bernhard Schmidt schreef op zo 08-07-2012 om 18:05 >>> [+0200]: >>>>> On Sunday 08 July 2012 13:06:30 Harm Weites wrote: >>>>>> Hi list, >>>>>> >>>>>> After flashing my firmware image on the TP-Link >>> it apears >>>> to run out of >>>>>> swap space when executing /etc/rc, thus halting >>> further >>>> system startup. >>>>>> My mfsroot is actually ~500KB smaller compared >>> to the >>>> result of the >>>>>> standard scripts, and the mount_mfs commands >>> in /etc/rc >>>> are building >>>>>> 512K devices instead of the standard 1M. What >>> could be the >>>> issue here, >>>>>> since there should be even more RAM available >>> compared to >>>> using the >>>>>> image produced by the standard build scripts? >>>>>> >>>>>> Furthermore, I've compiled the kernel without >>> WITNESS, >>>> WITNESS_SKIPSPIN, >>>>>> INVARIANTS and INVARIANT_SUPPORT. >>>>> >>>>> Do you have MALLOC_PRODUCTION defined in your >>> make.conf? If >>>> not, you >>>>> should try to do so. >>>>> >>>> >>>> >>>> _______________________________________________ >>>> freebsd-embedded@freebsd.org mailing list >>>> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-embedded >>>> To unsubscribe, send any mail to >>>> "freebsd-embedded-unsubscribe@freebsd.org" >>>> >>>> >>>> >>>> >>>> -- >>>> Marcelo Araujo >>>> araujo@FreeBSD.org >>>> >>> >>> >>> >>> >>> >>> >>> -- >>> Marcelo Araujo >>> araujo@FreeBSD.org >> >> >> _______________________________________________ >> freebsd-embedded@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-embedded >> To unsubscribe, send any mail to "freebsd-embedded-unsubscribe@freebsd.o= rg" > > _______________________________________________ > freebsd-embedded@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-embedded > To unsubscribe, send any mail to "freebsd-embedded-unsubscribe@freebsd.or= g" From owner-freebsd-embedded@FreeBSD.ORG Sun Jul 15 12:39:39 2012 Return-Path: Delivered-To: freebsd-embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7EFC41065672 for ; Sun, 15 Jul 2012 12:39:39 +0000 (UTC) (envelope-from freebsd@damnhippie.dyndns.org) Received: from qmta01.emeryville.ca.mail.comcast.net (qmta01.emeryville.ca.mail.comcast.net [76.96.30.16]) by mx1.freebsd.org (Postfix) with ESMTP id 575A38FC1E for ; Sun, 15 Jul 2012 12:39:39 +0000 (UTC) Received: from omta10.emeryville.ca.mail.comcast.net ([76.96.30.28]) by qmta01.emeryville.ca.mail.comcast.net with comcast id acdC1j0040cQ2SLA1cfZGe; Sun, 15 Jul 2012 12:39:33 +0000 Received: from damnhippie.dyndns.org ([24.8.232.202]) by omta10.emeryville.ca.mail.comcast.net with comcast id acfX1j00f4NgCEG8WcfYoV; Sun, 15 Jul 2012 12:39:33 +0000 Received: from [172.22.42.240] (revolution.hippie.lan [172.22.42.240]) by damnhippie.dyndns.org (8.14.3/8.14.3) with ESMTP id q6FCdU5X018736; Sun, 15 Jul 2012 06:39:30 -0600 (MDT) (envelope-from freebsd@damnhippie.dyndns.org) From: Ian Lepore To: Adrian Chadd In-Reply-To: References: <1341745590.2740.17.camel@manbearpig.dynamic.weites.net> <201207081805.33574.bschmidt@freebsd.org> <1341841445.2540.10.camel@manbearpig.dynamic.weites.net> <1341849727.2540.11.camel@manbearpig.dynamic.weites.net> <1342195983.2336.35.camel@manbearpig.dynamic.weites.net> <4B538596-937B-46F3-AF8F-17F34BE0C92D@bsdimp.com> Content-Type: text/plain; charset="us-ascii" Date: Sun, 15 Jul 2012 06:39:29 -0600 Message-ID: <1342355969.5473.6.camel@revolution.hippie.lan> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port Content-Transfer-Encoding: 7bit Cc: Harm Weites , freebsd-embedded@freebsd.org, Bernhard Schmidt Subject: Re: TP-Link wr1043nd out of swap space X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 15 Jul 2012 12:39:39 -0000 On Sun, 2012-07-15 at 03:31 -0700, Adrian Chadd wrote: > Hi, > > I would really appreciate it if people (read; not me) would be able to > do the digging needed to get to the bottom of user/kernel memory > usage. > > I really need to focus on just the net80211/wifi stack side of things. > I'm going to focus on getting the ath(4) memory usage down over the > next few months so it remains feasible to run on 32MB platforms, as > those still ship. But I can't keep the rest of the kernel and userland > in check. > > Thanks, > > > Adrian I had to chase down "out of swap space" aborts on an ARM platform with 64MB not long ago, and I discovered that the kernel by default allocates 1/4 of available ram for vfs buffers (up to some limit, then it's 1/10 after that). I added "option NBUF=128" to our kernel config and that limited wired vfs buffer space to about 2MB, which seems much more reasonable for an embedded platform that does relatively little disk IO. I suspect the NBUF value could go even lower, but I'm also afraid that making it too low will lead to other problems; I don't really know enough to make an informed decision. So far the 128 value is working well in testing, but we haven't actually put any units in the field with that setting (I think we will pretty soon). -- Ian From owner-freebsd-embedded@FreeBSD.ORG Mon Jul 16 11:08:48 2012 Return-Path: Delivered-To: freebsd-embedded@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 2CB34106566C for ; Mon, 16 Jul 2012 11:08:48 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 15D098FC0A for ; Mon, 16 Jul 2012 11:08:48 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q6GB8lsU093938 for ; Mon, 16 Jul 2012 11:08:47 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q6GB8kpb093934 for freebsd-embedded@FreeBSD.org; Mon, 16 Jul 2012 11:08:46 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 16 Jul 2012 11:08:46 GMT Message-Id: <201207161108.q6GB8kpb093934@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-embedded@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-embedded@FreeBSD.org X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jul 2012 11:08:48 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o misc/52256 embedded [picobsd] picobsd build script does not read in user/s o kern/42728 embedded [picobsd] many problems in src/usr.sbin/ppp/* after c 2 problems total. From owner-freebsd-embedded@FreeBSD.ORG Mon Jul 16 21:02:10 2012 Return-Path: Delivered-To: freebsd-embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A48D2106564A; Mon, 16 Jul 2012 21:02:10 +0000 (UTC) (envelope-from harm@weites.com) Received: from server1.weites.net (server1.weites.net [89.188.29.39]) by mx1.freebsd.org (Postfix) with ESMTP id 3F94A8FC0A; Mon, 16 Jul 2012 21:02:10 +0000 (UTC) Received: from [192.168.99.209] (52486A57.cm-4-1b.dynamic.ziggo.nl [82.72.106.87]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: harm@weites.com) by server1.weites.net (Postfix) with ESMTPSA id CE17A71C9D; Mon, 16 Jul 2012 23:02:02 +0200 (CEST) Message-ID: <1342472522.2336.97.camel@manbearpig.dynamic.weites.net> From: Harm Weites To: Ian Lepore Date: Mon, 16 Jul 2012 23:02:02 +0200 In-Reply-To: <1342355969.5473.6.camel@revolution.hippie.lan> References: <1341745590.2740.17.camel@manbearpig.dynamic.weites.net> <201207081805.33574.bschmidt@freebsd.org> <1341841445.2540.10.camel@manbearpig.dynamic.weites.net> <1341849727.2540.11.camel@manbearpig.dynamic.weites.net> <1342195983.2336.35.camel@manbearpig.dynamic.weites.net> <4B538596-937B-46F3-AF8F-17F34BE0C92D@bsdimp.com> <1342355969.5473.6.camel@revolution.hippie.lan> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.4.3 (3.4.3-1.fc17) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Cc: freebsd-embedded@freebsd.org, Bernhard Schmidt Subject: Re: TP-Link wr1043nd out of swap space X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Jul 2012 21:02:10 -0000 Hi, setting NBUF to 128 didn't bring any noticable change. I've changed /etc/rc to just start /bin/sh to make it easier to run some diagnostics right after kernel boot, here are some of my findings. r238194 this is quite interesting since there are no (user) processes running, apart from /bin/sh. ---------------- rtl8366rb0port0: link state changed to UP *** Start /bin/sh pid 18 (sh), uid 0, was killed: out of swap space Jul 16 11:58:11 init: /bin/sh on /etc/rc terminated abnormally, going to single user mode^M Enter full pathname of shell or RETURN for /bin/sh: # vmstat pid 20 (sh), uid 0, was killed: out of swap space Jul 16 11:59:41 init: single user shell terminated r235767 ---------------- # vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr fl0 md0 in sy cs us sy id 0 0 0 25360k 1796k 564 5 3 0 436 1420 29 0 0 63 83 2 18 80 r228268 Right after kernel boot (so without active networking/services): ---------------- # vmstat procs memory page disk faults cpu r b w avm fre flt re pi po fr sr fl0 in sy cs us sy id 0 0 0 35088k 17M 36 0 2 0 57 0 30 0 18 72 0 9 91 And after initializing networking (and starting hostapd): # vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr fl0 md0 in sy cs us sy id 0 0 0 49548k 8620k 140 0 1 0 89 0 0 0 0 74 93 1 5 94 Furthermore, after manually starting all scripts and observing vmstat after each step, I noticed a decrease from 13M to 10M after starting the wifi script (this starting hostapd). r231714 with the following processes: -hostapd -dropbear -dhcprelay -syslogd -rtadvd -dhclient ---------------- # vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr fl0 md0 in sy cs us sy id 1 1 0 176M 3080k 58 0 0 0 47 30 0 0 0 83 223 1 2 97 Starting bsnmpd/ntpd takes away another 2500k, which mostly resulted in the 'out of swap space' error. Hopefully I can at least tweak those services a little, or perhaps there is something with a smaller footprint already in ports :) I can only hope ~ 3000k is enough to route traffic... r228256:228258 This is from the image Adrian put online, where hostapd isn't running; just inetd. ---------------- # vmstat procs memory page disks faults cpu r b w avm fre flt re pi po fr sr fl0 md0 in sy cs us sy id 0 0 0 49928k 11M 255 1 3 0 186 0 0 0 0 144 122 2 18 80 I am by no means a kernel adept, so I can't do much but show my observations upon different kernel/userland configurations. Any tips/pointers to aid in the dig are greatly appreciated. Perhaps someone else with a 1043ND can offer his/her findings with any particular kernel revision. regards Ian Lepore schreef op zo 15-07-2012 om 06:39 [-0600]: > On Sun, 2012-07-15 at 03:31 -0700, Adrian Chadd wrote: > > Hi, > > > > I would really appreciate it if people (read; not me) would be able to > > do the digging needed to get to the bottom of user/kernel memory > > usage. > > > > I really need to focus on just the net80211/wifi stack side of things. > > I'm going to focus on getting the ath(4) memory usage down over the > > next few months so it remains feasible to run on 32MB platforms, as > > those still ship. But I can't keep the rest of the kernel and userland > > in check. > > > > Thanks, > > > > > > Adrian > > I had to chase down "out of swap space" aborts on an ARM platform with > 64MB not long ago, and I discovered that the kernel by default allocates > 1/4 of available ram for vfs buffers (up to some limit, then it's 1/10 > after that). I added "option NBUF=128" to our kernel config and that > limited wired vfs buffer space to about 2MB, which seems much more > reasonable for an embedded platform that does relatively little disk IO. > > I suspect the NBUF value could go even lower, but I'm also afraid that > making it too low will lead to other problems; I don't really know > enough to make an informed decision. So far the 128 value is working > well in testing, but we haven't actually put any units in the field with > that setting (I think we will pretty soon). > > -- Ian > > From owner-freebsd-embedded@FreeBSD.ORG Wed Jul 18 17:08:54 2012 Return-Path: Delivered-To: freebsd-embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id EB0A31065675; Wed, 18 Jul 2012 17:08:54 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 3301F8FC1B; Wed, 18 Jul 2012 17:08:53 +0000 (UTC) Received: by lbon10 with SMTP id n10so3005056lbo.13 for ; Wed, 18 Jul 2012 10:08:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=q+QXcfYUSvJNHVhYTlJR6Xm7ci+Hf1f5aoOWe4YNiMM=; b=UoFkv3EclI7l5flZzu3rQmaT84jtU47iWkfVtnG3JFZKBmPtsaWqq5FKVrdW8pkLpe W67u/YqTNVoGpsvxGMhRlh2WJIO70sU3DnBDbqwKM8CcvKXg6b3eTlluR2G0/5LA/KND IiN4gIGLu6PryzM/ctCd2hNxU5MuQtx/nXX1vEfTFzo6GAcmsz5VxpgMAL9FPn3o00tn AZQ3qxy07+t4VwlqvcLit3Q2qR22z/4uOi56xA6F2qCPz19bSygzjHvJPiwXV6Xp1D+D eus4RXILK2/RqaYfdKcdj/j3vBp8C6fh4ZCi7lZGPOREb+WBN+Vz1ulAwpE316Nvjfnm Wkfw== MIME-Version: 1.0 Received: by 10.112.47.104 with SMTP id c8mr2109853lbn.101.1342631332622; Wed, 18 Jul 2012 10:08:52 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.112.20.197 with HTTP; Wed, 18 Jul 2012 10:08:52 -0700 (PDT) In-Reply-To: <1342472522.2336.97.camel@manbearpig.dynamic.weites.net> References: <1341745590.2740.17.camel@manbearpig.dynamic.weites.net> <201207081805.33574.bschmidt@freebsd.org> <1341841445.2540.10.camel@manbearpig.dynamic.weites.net> <1341849727.2540.11.camel@manbearpig.dynamic.weites.net> <1342195983.2336.35.camel@manbearpig.dynamic.weites.net> <4B538596-937B-46F3-AF8F-17F34BE0C92D@bsdimp.com> <1342355969.5473.6.camel@revolution.hippie.lan> <1342472522.2336.97.camel@manbearpig.dynamic.weites.net> Date: Wed, 18 Jul 2012 10:08:52 -0700 X-Google-Sender-Auth: RIIN43yvbnlkalGOkAtx-1w8yAk Message-ID: From: Adrian Chadd To: Harm Weites Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-embedded@freebsd.org, Bernhard Schmidt Subject: Re: TP-Link wr1043nd out of swap space X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jul 2012 17:08:55 -0000 .. christ, has this really broken so significantly? I haven't updated my 1043nd in a couple months (as I have other test devices); are you sure you're correctly defining MALLOC_PRODUCTION? I'll see if I can/should just add NBUF=128 to the kernel configuration files, to save a little extra RAM. Thanks for that pointer. FWIW, I'm running this: FreeBSD home-11bg-ap 10.0-CURRENT FreeBSD 10.0-CURRENT #2 r234855:234941M: Wed Dec 31 16:00:00 PST 1969 adrian@dummy:/home/adrian/work/freebsd/svn/obj/mipseb/mips.mips/usr/home/adrian/work/freebsd/svn/src/sys/TP-WN1043ND mips .. so maybe try updating to that revision and see if you still see good/bad memory usage? I'd really appreciate it if both of you could build some kernel/world revisions and help me track down where this memory usage went up. I need the help. :) Thanks! Adrian On 16 July 2012 14:02, Harm Weites wrote: > Hi, > > setting NBUF to 128 didn't bring any noticable change. > > I've changed /etc/rc to just start /bin/sh to make it easier to run some > diagnostics right after kernel boot, here are some of my findings. > > r238194 > this is quite interesting since there are no (user) processes running, > apart from /bin/sh. > ---------------- > rtl8366rb0port0: link state changed to UP > *** Start /bin/sh > pid 18 (sh), uid 0, was killed: out of swap space > Jul 16 11:58:11 init: /bin/sh on /etc/rc terminated abnormally, going to > single user mode^M > Enter full pathname of shell or RETURN for /bin/sh: > # vmstat > pid 20 (sh), uid 0, was killed: out of swap space > Jul 16 11:59:41 init: single user shell terminated > > > r235767 > ---------------- > # vmstat > procs memory page disks faults > cpu > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > us sy id > 0 0 0 25360k 1796k 564 5 3 0 436 1420 29 0 0 63 > 83 2 18 80 > > r228268 > Right after kernel boot (so without active networking/services): > ---------------- > # vmstat > procs memory page disk faults cpu > r b w avm fre flt re pi po fr sr fl0 in sy cs us > sy id > 0 0 0 35088k 17M 36 0 2 0 57 0 30 0 18 72 0 > 9 91 > > And after initializing networking (and starting hostapd): > # vmstat > procs memory page disks faults > cpu > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > us sy id > 0 0 0 49548k 8620k 140 0 1 0 89 0 0 0 0 74 93 > 1 5 94 > > Furthermore, after manually starting all scripts and observing vmstat > after each step, I noticed a decrease from 13M to 10M after starting the > wifi script (this starting hostapd). > > r231714 with the following processes: > -hostapd > -dropbear > -dhcprelay > -syslogd > -rtadvd > -dhclient > ---------------- > # vmstat > procs memory page disks faults > cpu > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > us sy id > 1 1 0 176M 3080k 58 0 0 0 47 30 0 0 0 83 223 > 1 2 97 > > Starting bsnmpd/ntpd takes away another 2500k, which mostly resulted in > the 'out of swap space' error. Hopefully I can at least tweak those > services a little, or perhaps there is something with a smaller > footprint already in ports :) > > I can only hope ~ 3000k is enough to route traffic... > > r228256:228258 > This is from the image Adrian put online, where hostapd isn't running; > just inetd. > ---------------- > # vmstat > procs memory page disks faults > cpu > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > us sy id > 0 0 0 49928k 11M 255 1 3 0 186 0 0 0 0 144 122 > 2 18 80 > > I am by no means a kernel adept, so I can't do much but show my > observations upon different kernel/userland configurations. > > Any tips/pointers to aid in the dig are greatly appreciated. > > Perhaps someone else with a 1043ND can offer his/her findings with any > particular kernel revision. > > regards > > Ian Lepore schreef op zo 15-07-2012 om 06:39 [-0600]: >> On Sun, 2012-07-15 at 03:31 -0700, Adrian Chadd wrote: >> > Hi, >> > >> > I would really appreciate it if people (read; not me) would be able to >> > do the digging needed to get to the bottom of user/kernel memory >> > usage. >> > >> > I really need to focus on just the net80211/wifi stack side of things. >> > I'm going to focus on getting the ath(4) memory usage down over the >> > next few months so it remains feasible to run on 32MB platforms, as >> > those still ship. But I can't keep the rest of the kernel and userland >> > in check. >> > >> > Thanks, >> > >> > >> > Adrian >> >> I had to chase down "out of swap space" aborts on an ARM platform with >> 64MB not long ago, and I discovered that the kernel by default allocates >> 1/4 of available ram for vfs buffers (up to some limit, then it's 1/10 >> after that). I added "option NBUF=128" to our kernel config and that >> limited wired vfs buffer space to about 2MB, which seems much more >> reasonable for an embedded platform that does relatively little disk IO. >> >> I suspect the NBUF value could go even lower, but I'm also afraid that >> making it too low will lead to other problems; I don't really know >> enough to make an informed decision. So far the 128 value is working >> well in testing, but we haven't actually put any units in the field with >> that setting (I think we will pretty soon). >> >> -- Ian >> >> > > From owner-freebsd-embedded@FreeBSD.ORG Wed Jul 18 19:22:04 2012 Return-Path: Delivered-To: freebsd-embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9140F106564A; Wed, 18 Jul 2012 19:22:04 +0000 (UTC) (envelope-from harm@weites.com) Received: from server1.weites.net (server1.weites.net [89.188.29.39]) by mx1.freebsd.org (Postfix) with ESMTP id 2DB468FC08; Wed, 18 Jul 2012 19:22:03 +0000 (UTC) Received: from [192.168.99.209] (52486A57.cm-4-1b.dynamic.ziggo.nl [82.72.106.87]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: harm@weites.com) by server1.weites.net (Postfix) with ESMTPSA id B723C71C9D; Wed, 18 Jul 2012 21:21:55 +0200 (CEST) Message-ID: <1342639315.2698.21.camel@manbearpig.dynamic.weites.net> From: Harm Weites To: Adrian Chadd Date: Wed, 18 Jul 2012 21:21:55 +0200 In-Reply-To: References: <1341745590.2740.17.camel@manbearpig.dynamic.weites.net> <201207081805.33574.bschmidt@freebsd.org> <1341841445.2540.10.camel@manbearpig.dynamic.weites.net> <1341849727.2540.11.camel@manbearpig.dynamic.weites.net> <1342195983.2336.35.camel@manbearpig.dynamic.weites.net> <4B538596-937B-46F3-AF8F-17F34BE0C92D@bsdimp.com> <1342355969.5473.6.camel@revolution.hippie.lan> <1342472522.2336.97.camel@manbearpig.dynamic.weites.net> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.4.3 (3.4.3-2.fc17) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Cc: Bernhard, freebsd-embedded@freebsd.org, Schmidt Subject: Re: TP-Link wr1043nd out of swap space X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jul 2012 19:22:04 -0000 No luck, both immediately crash with the out-of-swap message. I've checked out r234855, deleted ./root and ./obj and then did the make-steps. Flashed the device, observed the error and noticed this (nothing is started at this point, not even networking): # vmstat procs memory page disk faults cpu r b w avm fre flt re pi po fr sr fl0 in sy cs us sy id 0 0 0 25340k 1776k 872 4 5 0 624 2604 29 0 103 119 3 27 70 # ps fauxwww USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAN [..] 0 20 0.7 29.6 10780 9712 u0 Ss 6:54PM 0:00.19 -sh (sh) 0 22 0.0 28.4 10516 9296 u0 R+ 6:55PM 0:00.15 ps fauxwww [..] The processes listed didn't use that much %MEM before... in r231714 those where both at 4-5. Sadly, an svn update of the tree to r234941 did not bring any improvements. I did put "MALLOC_PRODUCTION=YES" in /etc/make.conf, though I'm not sure if that is correct; the example (in /usr/share/examples/) does not list it. Hence I configured it with makeoptions aswell. My kernel config: --- sys/mips/conf/TP-WN1043ND (revision 234941) +++ sys/mips/conf/TP-WN1043ND (working copy) @@ -15,6 +15,9 @@ # Force the board memory - 32mb options AR71XX_REALMEM=32*1024*1024 +makeoptions MODULES_OVERRIDE="random gpio ar71xx if_gif if_gre if_bridge bridgestp wlan wlan_xauth wlan_acl wlan_tkip wlan_ccmp wlan_rssadapt wlan_amrr ath ath_ahb hwpmc pf if_vlan" +makeoptions MALLOC_PRODUCTION + # read MSDOS formatted disks - USB options MSDOSFS options GEOM_PART_BSD @@ -33,3 +36,15 @@ # Boot off of the rootfs, as defined in the geom_map setup. options ROOTDEVNAME=\"ufs:map/rootfs.uzip\" + +nooptions INVARIANTS +nooptions INVARIANT_SUPPORT +nooptions WITNESS +nooptions WITNESS_SKIPSPIN + +options NBUF=128 + +device pf +device gif +device vlan + regards Adrian Chadd schreef op wo 18-07-2012 om 10:08 [-0700]: > .. christ, has this really broken so significantly? > > I haven't updated my 1043nd in a couple months (as I have other test > devices); are you sure you're correctly defining MALLOC_PRODUCTION? > > I'll see if I can/should just add NBUF=128 to the kernel configuration > files, to save a little extra RAM. Thanks for that pointer. > > FWIW, I'm running this: > > FreeBSD home-11bg-ap 10.0-CURRENT FreeBSD 10.0-CURRENT #2 > r234855:234941M: Wed Dec 31 16:00:00 PST 1969 > adrian@dummy:/home/adrian/work/freebsd/svn/obj/mipseb/mips.mips/usr/home/adrian/work/freebsd/svn/src/sys/TP-WN1043ND > mips > > .. so maybe try updating to that revision and see if you still see > good/bad memory usage? > > I'd really appreciate it if both of you could build some kernel/world > revisions and help me track down where this memory usage went up. I > need the help. :) > > Thanks! > > > > Adrian > > > On 16 July 2012 14:02, Harm Weites wrote: > > Hi, > > > > setting NBUF to 128 didn't bring any noticable change. > > > > I've changed /etc/rc to just start /bin/sh to make it easier to run some > > diagnostics right after kernel boot, here are some of my findings. > > > > r238194 > > this is quite interesting since there are no (user) processes running, > > apart from /bin/sh. > > ---------------- > > rtl8366rb0port0: link state changed to UP > > *** Start /bin/sh > > pid 18 (sh), uid 0, was killed: out of swap space > > Jul 16 11:58:11 init: /bin/sh on /etc/rc terminated abnormally, going to > > single user mode^M > > Enter full pathname of shell or RETURN for /bin/sh: > > # vmstat > > pid 20 (sh), uid 0, was killed: out of swap space > > Jul 16 11:59:41 init: single user shell terminated > > > > > > r235767 > > ---------------- > > # vmstat > > procs memory page disks faults > > cpu > > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > > us sy id > > 0 0 0 25360k 1796k 564 5 3 0 436 1420 29 0 0 63 > > 83 2 18 80 > > > > r228268 > > Right after kernel boot (so without active networking/services): > > ---------------- > > # vmstat > > procs memory page disk faults cpu > > r b w avm fre flt re pi po fr sr fl0 in sy cs us > > sy id > > 0 0 0 35088k 17M 36 0 2 0 57 0 30 0 18 72 0 > > 9 91 > > > > And after initializing networking (and starting hostapd): > > # vmstat > > procs memory page disks faults > > cpu > > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > > us sy id > > 0 0 0 49548k 8620k 140 0 1 0 89 0 0 0 0 74 93 > > 1 5 94 > > > > Furthermore, after manually starting all scripts and observing vmstat > > after each step, I noticed a decrease from 13M to 10M after starting the > > wifi script (this starting hostapd). > > > > r231714 with the following processes: > > -hostapd > > -dropbear > > -dhcprelay > > -syslogd > > -rtadvd > > -dhclient > > ---------------- > > # vmstat > > procs memory page disks faults > > cpu > > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > > us sy id > > 1 1 0 176M 3080k 58 0 0 0 47 30 0 0 0 83 223 > > 1 2 97 > > > > Starting bsnmpd/ntpd takes away another 2500k, which mostly resulted in > > the 'out of swap space' error. Hopefully I can at least tweak those > > services a little, or perhaps there is something with a smaller > > footprint already in ports :) > > > > I can only hope ~ 3000k is enough to route traffic... > > > > r228256:228258 > > This is from the image Adrian put online, where hostapd isn't running; > > just inetd. > > ---------------- > > # vmstat > > procs memory page disks faults > > cpu > > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > > us sy id > > 0 0 0 49928k 11M 255 1 3 0 186 0 0 0 0 144 122 > > 2 18 80 > > > > I am by no means a kernel adept, so I can't do much but show my > > observations upon different kernel/userland configurations. > > > > Any tips/pointers to aid in the dig are greatly appreciated. > > > > Perhaps someone else with a 1043ND can offer his/her findings with any > > particular kernel revision. > > > > regards > > > > Ian Lepore schreef op zo 15-07-2012 om 06:39 [-0600]: > >> On Sun, 2012-07-15 at 03:31 -0700, Adrian Chadd wrote: > >> > Hi, > >> > > >> > I would really appreciate it if people (read; not me) would be able to > >> > do the digging needed to get to the bottom of user/kernel memory > >> > usage. > >> > > >> > I really need to focus on just the net80211/wifi stack side of things. > >> > I'm going to focus on getting the ath(4) memory usage down over the > >> > next few months so it remains feasible to run on 32MB platforms, as > >> > those still ship. But I can't keep the rest of the kernel and userland > >> > in check. > >> > > >> > Thanks, > >> > > >> > > >> > Adrian > >> > >> I had to chase down "out of swap space" aborts on an ARM platform with > >> 64MB not long ago, and I discovered that the kernel by default allocates > >> 1/4 of available ram for vfs buffers (up to some limit, then it's 1/10 > >> after that). I added "option NBUF=128" to our kernel config and that > >> limited wired vfs buffer space to about 2MB, which seems much more > >> reasonable for an embedded platform that does relatively little disk IO. > >> > >> I suspect the NBUF value could go even lower, but I'm also afraid that > >> making it too low will lead to other problems; I don't really know > >> enough to make an informed decision. So far the 128 value is working > >> well in testing, but we haven't actually put any units in the field with > >> that setting (I think we will pretty soon). > >> > >> -- Ian > >> > >> > > > > From owner-freebsd-embedded@FreeBSD.ORG Thu Jul 19 08:28:47 2012 Return-Path: Delivered-To: freebsd-embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id ECEF1106567E; Thu, 19 Jul 2012 08:28:47 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-lb0-f182.google.com (mail-lb0-f182.google.com [209.85.217.182]) by mx1.freebsd.org (Postfix) with ESMTP id 2EA368FC20; Thu, 19 Jul 2012 08:28:46 +0000 (UTC) Received: by lbon10 with SMTP id n10so4128039lbo.13 for ; Thu, 19 Jul 2012 01:28:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=CTuz74yRXnE3xucxUKyVntmArNlvOkcKKT1qmWzHeMk=; b=I6xoAPa0vJDBf3yyFrcTYcb+iY9HieXZ7d6v8srlM/sCftPL6WBtY1gBd9bZraN+vS VmbdE8IU2iF6PgK2anJJs0ofuRPSKXiApxE0MIC/9bC66oXaoe+Bf9vT57fUinyoexjk YYHtD8ZaUIUd9pnCG2eOCZhO90hbvLJrbsm7JmYtBa1XZB8+sCEKRIp3pDj+xCd0b6ps qdLiQjG9LJu0yXWtdJ5XgBIDQKcQN3Ip01WaQHKyI4OcOfmxyUWjd6d4KNfzEuolMKed ckPx1vQROi6AX43RMGFhslpMEQ7OyUtg74WMr+dKgpPJtzVUsJet1sgQghONNurOKTrU pp5Q== MIME-Version: 1.0 Received: by 10.112.44.163 with SMTP id f3mr681247lbm.59.1342686526140; Thu, 19 Jul 2012 01:28:46 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.112.20.197 with HTTP; Thu, 19 Jul 2012 01:28:46 -0700 (PDT) In-Reply-To: <1342639315.2698.21.camel@manbearpig.dynamic.weites.net> References: <1341745590.2740.17.camel@manbearpig.dynamic.weites.net> <201207081805.33574.bschmidt@freebsd.org> <1341841445.2540.10.camel@manbearpig.dynamic.weites.net> <1341849727.2540.11.camel@manbearpig.dynamic.weites.net> <1342195983.2336.35.camel@manbearpig.dynamic.weites.net> <4B538596-937B-46F3-AF8F-17F34BE0C92D@bsdimp.com> <1342355969.5473.6.camel@revolution.hippie.lan> <1342472522.2336.97.camel@manbearpig.dynamic.weites.net> <1342639315.2698.21.camel@manbearpig.dynamic.weites.net> Date: Thu, 19 Jul 2012 01:28:46 -0700 X-Google-Sender-Auth: OdHHLXTosenq2f-QvW6fV115oyk Message-ID: From: Adrian Chadd To: Harm Weites Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-embedded@freebsd.org, Bernhard Schmidt Subject: Re: TP-Link wr1043nd out of swap space X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jul 2012 08:28:48 -0000 Hi, re: MALLOC_PRODUCTION: It's not a kernel thing. It's part of the userland build configuration. Check out what my build scripts do (specifically build_freebsd) - I set it correctly in the relevant place. I'm more interested in any changes in available memory reported during kernel startup. Those need to be tracked down immediately. Same deal with any change in the post-boot malloc/slab allocator pools. Adrian On 18 July 2012 12:21, Harm Weites wrote: > No luck, both immediately crash with the out-of-swap message. > > I've checked out r234855, deleted ./root and ./obj and then did the > make-steps. Flashed the device, observed the error and noticed this > (nothing is started at this point, not even networking): > > # vmstat > procs memory page disk faults > cpu > r b w avm fre flt re pi po fr sr fl0 in sy cs us > sy id > 0 0 0 25340k 1776k 872 4 5 0 624 2604 29 0 103 119 3 > 27 70 > > # ps fauxwww > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAN > [..] > 0 20 0.7 29.6 10780 9712 u0 Ss 6:54PM 0:00.19 -sh (sh) > 0 22 0.0 28.4 10516 9296 u0 R+ 6:55PM 0:00.15 ps fauxwww > [..] > > The processes listed didn't use that much %MEM before... in r231714 > those where both at 4-5. > > Sadly, an svn update of the tree to r234941 did not bring any > improvements. > > I did put "MALLOC_PRODUCTION=YES" in /etc/make.conf, though I'm not sure > if that is correct; the example (in /usr/share/examples/) does not list > it. Hence I configured it with makeoptions aswell. > > My kernel config: > --- sys/mips/conf/TP-WN1043ND (revision 234941) > +++ sys/mips/conf/TP-WN1043ND (working copy) > @@ -15,6 +15,9 @@ > # Force the board memory - 32mb > options AR71XX_REALMEM=32*1024*1024 > > +makeoptions MODULES_OVERRIDE="random gpio ar71xx if_gif if_gre > if_bridge bridgestp wlan wlan_xauth wlan_acl wlan_tkip wlan_ccmp > wlan_rssadapt wlan_amrr ath ath_ahb hwpmc pf if_vlan" > +makeoptions MALLOC_PRODUCTION > + > # read MSDOS formatted disks - USB > options MSDOSFS > options GEOM_PART_BSD > @@ -33,3 +36,15 @@ > > # Boot off of the rootfs, as defined in the geom_map setup. > options ROOTDEVNAME=\"ufs:map/rootfs.uzip\" > + > +nooptions INVARIANTS > +nooptions INVARIANT_SUPPORT > +nooptions WITNESS > +nooptions WITNESS_SKIPSPIN > + > +options NBUF=128 > + > +device pf > +device gif > +device vlan > + > > regards > > Adrian Chadd schreef op wo 18-07-2012 om 10:08 [-0700]: >> .. christ, has this really broken so significantly? >> >> I haven't updated my 1043nd in a couple months (as I have other test >> devices); are you sure you're correctly defining MALLOC_PRODUCTION? >> >> I'll see if I can/should just add NBUF=128 to the kernel configuration >> files, to save a little extra RAM. Thanks for that pointer. >> >> FWIW, I'm running this: >> >> FreeBSD home-11bg-ap 10.0-CURRENT FreeBSD 10.0-CURRENT #2 >> r234855:234941M: Wed Dec 31 16:00:00 PST 1969 >> adrian@dummy:/home/adrian/work/freebsd/svn/obj/mipseb/mips.mips/usr/home/adrian/work/freebsd/svn/src/sys/TP-WN1043ND >> mips >> >> .. so maybe try updating to that revision and see if you still see >> good/bad memory usage? >> >> I'd really appreciate it if both of you could build some kernel/world >> revisions and help me track down where this memory usage went up. I >> need the help. :) >> >> Thanks! >> >> >> >> Adrian >> >> >> On 16 July 2012 14:02, Harm Weites wrote: >> > Hi, >> > >> > setting NBUF to 128 didn't bring any noticable change. >> > >> > I've changed /etc/rc to just start /bin/sh to make it easier to run some >> > diagnostics right after kernel boot, here are some of my findings. >> > >> > r238194 >> > this is quite interesting since there are no (user) processes running, >> > apart from /bin/sh. >> > ---------------- >> > rtl8366rb0port0: link state changed to UP >> > *** Start /bin/sh >> > pid 18 (sh), uid 0, was killed: out of swap space >> > Jul 16 11:58:11 init: /bin/sh on /etc/rc terminated abnormally, going to >> > single user mode^M >> > Enter full pathname of shell or RETURN for /bin/sh: >> > # vmstat >> > pid 20 (sh), uid 0, was killed: out of swap space >> > Jul 16 11:59:41 init: single user shell terminated >> > >> > >> > r235767 >> > ---------------- >> > # vmstat >> > procs memory page disks faults >> > cpu >> > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs >> > us sy id >> > 0 0 0 25360k 1796k 564 5 3 0 436 1420 29 0 0 63 >> > 83 2 18 80 >> > >> > r228268 >> > Right after kernel boot (so without active networking/services): >> > ---------------- >> > # vmstat >> > procs memory page disk faults cpu >> > r b w avm fre flt re pi po fr sr fl0 in sy cs us >> > sy id >> > 0 0 0 35088k 17M 36 0 2 0 57 0 30 0 18 72 0 >> > 9 91 >> > >> > And after initializing networking (and starting hostapd): >> > # vmstat >> > procs memory page disks faults >> > cpu >> > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs >> > us sy id >> > 0 0 0 49548k 8620k 140 0 1 0 89 0 0 0 0 74 93 >> > 1 5 94 >> > >> > Furthermore, after manually starting all scripts and observing vmstat >> > after each step, I noticed a decrease from 13M to 10M after starting the >> > wifi script (this starting hostapd). >> > >> > r231714 with the following processes: >> > -hostapd >> > -dropbear >> > -dhcprelay >> > -syslogd >> > -rtadvd >> > -dhclient >> > ---------------- >> > # vmstat >> > procs memory page disks faults >> > cpu >> > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs >> > us sy id >> > 1 1 0 176M 3080k 58 0 0 0 47 30 0 0 0 83 223 >> > 1 2 97 >> > >> > Starting bsnmpd/ntpd takes away another 2500k, which mostly resulted in >> > the 'out of swap space' error. Hopefully I can at least tweak those >> > services a little, or perhaps there is something with a smaller >> > footprint already in ports :) >> > >> > I can only hope ~ 3000k is enough to route traffic... >> > >> > r228256:228258 >> > This is from the image Adrian put online, where hostapd isn't running; >> > just inetd. >> > ---------------- >> > # vmstat >> > procs memory page disks faults >> > cpu >> > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs >> > us sy id >> > 0 0 0 49928k 11M 255 1 3 0 186 0 0 0 0 144 122 >> > 2 18 80 >> > >> > I am by no means a kernel adept, so I can't do much but show my >> > observations upon different kernel/userland configurations. >> > >> > Any tips/pointers to aid in the dig are greatly appreciated. >> > >> > Perhaps someone else with a 1043ND can offer his/her findings with any >> > particular kernel revision. >> > >> > regards >> > >> > Ian Lepore schreef op zo 15-07-2012 om 06:39 [-0600]: >> >> On Sun, 2012-07-15 at 03:31 -0700, Adrian Chadd wrote: >> >> > Hi, >> >> > >> >> > I would really appreciate it if people (read; not me) would be able to >> >> > do the digging needed to get to the bottom of user/kernel memory >> >> > usage. >> >> > >> >> > I really need to focus on just the net80211/wifi stack side of things. >> >> > I'm going to focus on getting the ath(4) memory usage down over the >> >> > next few months so it remains feasible to run on 32MB platforms, as >> >> > those still ship. But I can't keep the rest of the kernel and userland >> >> > in check. >> >> > >> >> > Thanks, >> >> > >> >> > >> >> > Adrian >> >> >> >> I had to chase down "out of swap space" aborts on an ARM platform with >> >> 64MB not long ago, and I discovered that the kernel by default allocates >> >> 1/4 of available ram for vfs buffers (up to some limit, then it's 1/10 >> >> after that). I added "option NBUF=128" to our kernel config and that >> >> limited wired vfs buffer space to about 2MB, which seems much more >> >> reasonable for an embedded platform that does relatively little disk IO. >> >> >> >> I suspect the NBUF value could go even lower, but I'm also afraid that >> >> making it too low will lead to other problems; I don't really know >> >> enough to make an informed decision. So far the 128 value is working >> >> well in testing, but we haven't actually put any units in the field with >> >> that setting (I think we will pretty soon). >> >> >> >> -- Ian >> >> >> >> >> > >> > > > From owner-freebsd-embedded@FreeBSD.ORG Thu Jul 19 17:11:52 2012 Return-Path: Delivered-To: freebsd-embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 04420106566B; Thu, 19 Jul 2012 17:11:52 +0000 (UTC) (envelope-from harm@weites.com) Received: from server1.weites.net (server1.weites.net [89.188.29.39]) by mx1.freebsd.org (Postfix) with ESMTP id 9A9FE8FC17; Thu, 19 Jul 2012 17:11:51 +0000 (UTC) Received: from [192.168.99.209] (52486A57.cm-4-1b.dynamic.ziggo.nl [82.72.106.87]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: harm@weites.com) by server1.weites.net (Postfix) with ESMTPSA id 9919B71C9D; Thu, 19 Jul 2012 19:11:49 +0200 (CEST) Message-ID: <1342717909.2637.20.camel@manbearpig.dynamic.weites.net> From: Harm Weites To: Adrian Chadd Date: Thu, 19 Jul 2012 19:11:49 +0200 In-Reply-To: References: <1341745590.2740.17.camel@manbearpig.dynamic.weites.net> <201207081805.33574.bschmidt@freebsd.org> <1341841445.2540.10.camel@manbearpig.dynamic.weites.net> <1341849727.2540.11.camel@manbearpig.dynamic.weites.net> <1342195983.2336.35.camel@manbearpig.dynamic.weites.net> <4B538596-937B-46F3-AF8F-17F34BE0C92D@bsdimp.com> <1342355969.5473.6.camel@revolution.hippie.lan> <1342472522.2336.97.camel@manbearpig.dynamic.weites.net> <1342639315.2698.21.camel@manbearpig.dynamic.weites.net> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.4.3 (3.4.3-2.fc17) Mime-Version: 1.0 Content-Transfer-Encoding: 7bit Cc: Bernhard, freebsd-embedded@freebsd.org, Schmidt Subject: Re: TP-Link wr1043nd out of swap space X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jul 2012 17:11:52 -0000 Damn, time for me to feel stupid... (which is a good thing, in some way) I defined it ok (in /etc/make.conf), but hadn't looked close enough at the build_freebsd script. The make command is appended with '__MAKE_CONF=/dev/null', rendering my change in /etc/make.conf useless since it wasn't beeing read anyway. So, changing that to /etc/make.conf, rebuilding the lot and flashing the unit gave 16M after initial kernel boot. After starting the scripts I'm left with ~3500k - ~5500k, with the following active processes: dropbear, dhcprelya, rtadvd, hostapd, syslogd, openntpd. I'm now on r238194, thanks! :) Time to play with arge0/arge1/etherswitchcfg! (since things don't work anymore like they did when there was only arge0) Adrian Chadd schreef op do 19-07-2012 om 01:28 [-0700]: > Hi, > > re: MALLOC_PRODUCTION: It's not a kernel thing. It's part of the > userland build configuration. > > Check out what my build scripts do (specifically build_freebsd) - I > set it correctly in the relevant place. > > I'm more interested in any changes in available memory reported during > kernel startup. Those need to be tracked down immediately. Same deal > with any change in the post-boot malloc/slab allocator pools. > > > > Adrian > > On 18 July 2012 12:21, Harm Weites wrote: > > No luck, both immediately crash with the out-of-swap message. > > > > I've checked out r234855, deleted ./root and ./obj and then did the > > make-steps. Flashed the device, observed the error and noticed this > > (nothing is started at this point, not even networking): > > > > # vmstat > > procs memory page disk faults > > cpu > > r b w avm fre flt re pi po fr sr fl0 in sy cs us > > sy id > > 0 0 0 25340k 1776k 872 4 5 0 624 2604 29 0 103 119 3 > > 27 70 > > > > # ps fauxwww > > USER PID %CPU %MEM VSZ RSS TT STAT STARTED TIME COMMAN > > [..] > > 0 20 0.7 29.6 10780 9712 u0 Ss 6:54PM 0:00.19 -sh (sh) > > 0 22 0.0 28.4 10516 9296 u0 R+ 6:55PM 0:00.15 ps fauxwww > > [..] > > > > The processes listed didn't use that much %MEM before... in r231714 > > those where both at 4-5. > > > > Sadly, an svn update of the tree to r234941 did not bring any > > improvements. > > > > I did put "MALLOC_PRODUCTION=YES" in /etc/make.conf, though I'm not sure > > if that is correct; the example (in /usr/share/examples/) does not list > > it. Hence I configured it with makeoptions aswell. > > > > My kernel config: > > --- sys/mips/conf/TP-WN1043ND (revision 234941) > > +++ sys/mips/conf/TP-WN1043ND (working copy) > > @@ -15,6 +15,9 @@ > > # Force the board memory - 32mb > > options AR71XX_REALMEM=32*1024*1024 > > > > +makeoptions MODULES_OVERRIDE="random gpio ar71xx if_gif if_gre > > if_bridge bridgestp wlan wlan_xauth wlan_acl wlan_tkip wlan_ccmp > > wlan_rssadapt wlan_amrr ath ath_ahb hwpmc pf if_vlan" > > +makeoptions MALLOC_PRODUCTION > > + > > # read MSDOS formatted disks - USB > > options MSDOSFS > > options GEOM_PART_BSD > > @@ -33,3 +36,15 @@ > > > > # Boot off of the rootfs, as defined in the geom_map setup. > > options ROOTDEVNAME=\"ufs:map/rootfs.uzip\" > > + > > +nooptions INVARIANTS > > +nooptions INVARIANT_SUPPORT > > +nooptions WITNESS > > +nooptions WITNESS_SKIPSPIN > > + > > +options NBUF=128 > > + > > +device pf > > +device gif > > +device vlan > > + > > > > regards > > > > Adrian Chadd schreef op wo 18-07-2012 om 10:08 [-0700]: > >> .. christ, has this really broken so significantly? > >> > >> I haven't updated my 1043nd in a couple months (as I have other test > >> devices); are you sure you're correctly defining MALLOC_PRODUCTION? > >> > >> I'll see if I can/should just add NBUF=128 to the kernel configuration > >> files, to save a little extra RAM. Thanks for that pointer. > >> > >> FWIW, I'm running this: > >> > >> FreeBSD home-11bg-ap 10.0-CURRENT FreeBSD 10.0-CURRENT #2 > >> r234855:234941M: Wed Dec 31 16:00:00 PST 1969 > >> adrian@dummy:/home/adrian/work/freebsd/svn/obj/mipseb/mips.mips/usr/home/adrian/work/freebsd/svn/src/sys/TP-WN1043ND > >> mips > >> > >> .. so maybe try updating to that revision and see if you still see > >> good/bad memory usage? > >> > >> I'd really appreciate it if both of you could build some kernel/world > >> revisions and help me track down where this memory usage went up. I > >> need the help. :) > >> > >> Thanks! > >> > >> > >> > >> Adrian > >> > >> > >> On 16 July 2012 14:02, Harm Weites wrote: > >> > Hi, > >> > > >> > setting NBUF to 128 didn't bring any noticable change. > >> > > >> > I've changed /etc/rc to just start /bin/sh to make it easier to run some > >> > diagnostics right after kernel boot, here are some of my findings. > >> > > >> > r238194 > >> > this is quite interesting since there are no (user) processes running, > >> > apart from /bin/sh. > >> > ---------------- > >> > rtl8366rb0port0: link state changed to UP > >> > *** Start /bin/sh > >> > pid 18 (sh), uid 0, was killed: out of swap space > >> > Jul 16 11:58:11 init: /bin/sh on /etc/rc terminated abnormally, going to > >> > single user mode^M > >> > Enter full pathname of shell or RETURN for /bin/sh: > >> > # vmstat > >> > pid 20 (sh), uid 0, was killed: out of swap space > >> > Jul 16 11:59:41 init: single user shell terminated > >> > > >> > > >> > r235767 > >> > ---------------- > >> > # vmstat > >> > procs memory page disks faults > >> > cpu > >> > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > >> > us sy id > >> > 0 0 0 25360k 1796k 564 5 3 0 436 1420 29 0 0 63 > >> > 83 2 18 80 > >> > > >> > r228268 > >> > Right after kernel boot (so without active networking/services): > >> > ---------------- > >> > # vmstat > >> > procs memory page disk faults cpu > >> > r b w avm fre flt re pi po fr sr fl0 in sy cs us > >> > sy id > >> > 0 0 0 35088k 17M 36 0 2 0 57 0 30 0 18 72 0 > >> > 9 91 > >> > > >> > And after initializing networking (and starting hostapd): > >> > # vmstat > >> > procs memory page disks faults > >> > cpu > >> > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > >> > us sy id > >> > 0 0 0 49548k 8620k 140 0 1 0 89 0 0 0 0 74 93 > >> > 1 5 94 > >> > > >> > Furthermore, after manually starting all scripts and observing vmstat > >> > after each step, I noticed a decrease from 13M to 10M after starting the > >> > wifi script (this starting hostapd). > >> > > >> > r231714 with the following processes: > >> > -hostapd > >> > -dropbear > >> > -dhcprelay > >> > -syslogd > >> > -rtadvd > >> > -dhclient > >> > ---------------- > >> > # vmstat > >> > procs memory page disks faults > >> > cpu > >> > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > >> > us sy id > >> > 1 1 0 176M 3080k 58 0 0 0 47 30 0 0 0 83 223 > >> > 1 2 97 > >> > > >> > Starting bsnmpd/ntpd takes away another 2500k, which mostly resulted in > >> > the 'out of swap space' error. Hopefully I can at least tweak those > >> > services a little, or perhaps there is something with a smaller > >> > footprint already in ports :) > >> > > >> > I can only hope ~ 3000k is enough to route traffic... > >> > > >> > r228256:228258 > >> > This is from the image Adrian put online, where hostapd isn't running; > >> > just inetd. > >> > ---------------- > >> > # vmstat > >> > procs memory page disks faults > >> > cpu > >> > r b w avm fre flt re pi po fr sr fl0 md0 in sy cs > >> > us sy id > >> > 0 0 0 49928k 11M 255 1 3 0 186 0 0 0 0 144 122 > >> > 2 18 80 > >> > > >> > I am by no means a kernel adept, so I can't do much but show my > >> > observations upon different kernel/userland configurations. > >> > > >> > Any tips/pointers to aid in the dig are greatly appreciated. > >> > > >> > Perhaps someone else with a 1043ND can offer his/her findings with any > >> > particular kernel revision. > >> > > >> > regards > >> > > >> > Ian Lepore schreef op zo 15-07-2012 om 06:39 [-0600]: > >> >> On Sun, 2012-07-15 at 03:31 -0700, Adrian Chadd wrote: > >> >> > Hi, > >> >> > > >> >> > I would really appreciate it if people (read; not me) would be able to > >> >> > do the digging needed to get to the bottom of user/kernel memory > >> >> > usage. > >> >> > > >> >> > I really need to focus on just the net80211/wifi stack side of things. > >> >> > I'm going to focus on getting the ath(4) memory usage down over the > >> >> > next few months so it remains feasible to run on 32MB platforms, as > >> >> > those still ship. But I can't keep the rest of the kernel and userland > >> >> > in check. > >> >> > > >> >> > Thanks, > >> >> > > >> >> > > >> >> > Adrian > >> >> > >> >> I had to chase down "out of swap space" aborts on an ARM platform with > >> >> 64MB not long ago, and I discovered that the kernel by default allocates > >> >> 1/4 of available ram for vfs buffers (up to some limit, then it's 1/10 > >> >> after that). I added "option NBUF=128" to our kernel config and that > >> >> limited wired vfs buffer space to about 2MB, which seems much more > >> >> reasonable for an embedded platform that does relatively little disk IO. > >> >> > >> >> I suspect the NBUF value could go even lower, but I'm also afraid that > >> >> making it too low will lead to other problems; I don't really know > >> >> enough to make an informed decision. So far the 128 value is working > >> >> well in testing, but we haven't actually put any units in the field with > >> >> that setting (I think we will pretty soon). > >> >> > >> >> -- Ian > >> >> > >> >> > >> > > >> > > > > > From owner-freebsd-embedded@FreeBSD.ORG Fri Jul 20 22:22:41 2012 Return-Path: Delivered-To: freebsd-embedded@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 464DE106566B; Fri, 20 Jul 2012 22:22:41 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-pb0-f54.google.com (mail-pb0-f54.google.com [209.85.160.54]) by mx1.freebsd.org (Postfix) with ESMTP id 0576E8FC16; Fri, 20 Jul 2012 22:22:40 +0000 (UTC) Received: by pbbro2 with SMTP id ro2so7411632pbb.13 for ; Fri, 20 Jul 2012 15:22:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=QDjkfa7PlM3Zx4tCIHn4RVdFdUW1danl5kRMHrloQVk=; b=kbiUY+l98BXrltAN5JBJZB2JV1prVr7WjCM8RiuxqwHxt5XJGABK5CAClp5FsQevX7 RYvYndYEtn/9yEoddot6R+Q+MNQIw38B9/9jPAEMIaQET+JWe3AgRiqD6toV9bYo18sY yT+tSrzCtirrmW/b2jqNuhaVzcbf9rchBRHB7Ew+NIOBvtrvsuCUrhSQaY0rmc3Fd14q 2mo8NHKzK1QMXdL1j5I13KSdXNOuNKec/trreHxpGpnEjdCQjY4jFKz7EbNnovQi9YQR hbnfK/aa1id341IxoFHGE0bAFuTLtIEJVCrR3HNm0xJmNa1RRMWkGnS4yiGqsJcCC6VM 5JSQ== MIME-Version: 1.0 Received: by 10.66.83.65 with SMTP id o1mr14510054pay.17.1342822960793; Fri, 20 Jul 2012 15:22:40 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.68.191.138 with HTTP; Fri, 20 Jul 2012 15:22:40 -0700 (PDT) In-Reply-To: <1342717909.2637.20.camel@manbearpig.dynamic.weites.net> References: <1341745590.2740.17.camel@manbearpig.dynamic.weites.net> <201207081805.33574.bschmidt@freebsd.org> <1341841445.2540.10.camel@manbearpig.dynamic.weites.net> <1341849727.2540.11.camel@manbearpig.dynamic.weites.net> <1342195983.2336.35.camel@manbearpig.dynamic.weites.net> <4B538596-937B-46F3-AF8F-17F34BE0C92D@bsdimp.com> <1342355969.5473.6.camel@revolution.hippie.lan> <1342472522.2336.97.camel@manbearpig.dynamic.weites.net> <1342639315.2698.21.camel@manbearpig.dynamic.weites.net> <1342717909.2637.20.camel@manbearpig.dynamic.weites.net> Date: Fri, 20 Jul 2012 15:22:40 -0700 X-Google-Sender-Auth: _4P77DU4e4arhrzecYBul4f_MqM Message-ID: From: Adrian Chadd To: Harm Weites Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-embedded@freebsd.org, Bernhard Schmidt Subject: Re: TP-Link wr1043nd out of swap space X-BeenThere: freebsd-embedded@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Dedicated and Embedded Systems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Jul 2012 22:22:41 -0000 That's great to hear! I think the 1043nd has one ethernet port hooked up to the switch. The other ethernet port isn't connected. :) Adrian