From owner-freebsd-hackers@FreeBSD.ORG Thu Apr 10 01:09:19 2014 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BE25D952; Thu, 10 Apr 2014 01:09:19 +0000 (UTC) Received: from mail-qc0-x233.google.com (mail-qc0-x233.google.com [IPv6:2607:f8b0:400d:c01::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5B88B1A71; Thu, 10 Apr 2014 01:09:19 +0000 (UTC) Received: by mail-qc0-f179.google.com with SMTP id m20so3641293qcx.38 for ; Wed, 09 Apr 2014 18:09:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type:content-transfer-encoding; bh=264EbCMXLHTmbq+aW9cndHLnV5e9ZHW389ZjxR05FuU=; b=dErqPCaQBQCSJFnT5mv2UZptgdxBTXFbtG+SuiyB3DlouolyEokw33WtU92DIghv9C rtw+hfe6MFxcO+53eb/zBiM8Ui+hdNpkPqXD+lW6Ps1T63XUWMUpUJuYNXKeR6vdcZWc V0hF3nU3Gj9CDZsQMjTgwwy1ZMvTg4kK6fRMXf3Ox/PLVgzuFxnNYB3f7mQtWUEeg4vT NsLxNjNktvPu+xI3SoPMaYX9aGSEERTOTOHg6lfQSWSMM84N/yIM27GjQ90seJ64DBe4 lq7ZWSukPtird9e+pywSV09EijZ+MYbJoZzE4LIwxDA2rO4MAlJH4xzJi0iHy45lw5Ib DvMQ== MIME-Version: 1.0 X-Received: by 10.140.42.38 with SMTP id b35mr15773695qga.87.1397092158477; Wed, 09 Apr 2014 18:09:18 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.50.206 with HTTP; Wed, 9 Apr 2014 18:09:18 -0700 (PDT) In-Reply-To: <1397077963.756961709.gspkmzvd@frv35.fwdcdn.com> References: <1397077963.756961709.gspkmzvd@frv35.fwdcdn.com> Date: Wed, 9 Apr 2014 18:09:18 -0700 X-Google-Sender-Auth: hJIhuCiqDd6ywEO6kT0dNkZDL8s Message-ID: Subject: Re: Some gruesome moments with performance of FreeBSD at over 20K interfaces From: Adrian Chadd To: Vladislav Prodan Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: "stable@freebsd.org" , "freebsd-hackers@freebsd.org" , "net@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Apr 2014 01:09:19 -0000 Hi, There's likely many more places where these aren't O(1) operations. The patch in question should be in -HEAD now. a-a On 9 April 2014 14:14, Vladislav Prodan wrote: > Dear Colleagues! > > I had a task, using FreeBSD 10.0-STABLE: > 1) Receive 20-30 Q-in-Q VLAN (IEEE 802.1ad ), inside of which 2k-4k vlan = (IEEE 802.1Q). Total ~60K vlan > 2) To every vlan interface assign ipv4 and ipv6 addresses, define routes = to ipv4 and ipv6 addresses on another side of vlan (ip unnumbered), and als= o prescribe ipv6 network /64 by size through ipv6 address on another side o= f vlan. > 3) Perform routing from the world to all of these ipv4/ipv6 addresses =D0= =B8 ipv6 networks inside ~60K vlan > > > > To accomplish the 1st task I have no alternatives to using Netgraph. > I noticed incorrect behavior of ngctl(8) after addition of 560th vlan (bi= n/187835) > Than speed of addition 4k, 8k, 12k vlans was damnably slow: > 10 minutes for first 4k vlans > 18 minutes for first 5k vlans > 28 minutes for first 6k vlans > 52 minutes for first 8k vlans > Than I added more 4=D0=BA vlans > 20 minutes - 9500 vlans > 33 minutes - 10500 vlans > 58 minutes - 12=D0=BA vlans > > In total speed of addition of 4k, 8k, 12k vlans was subsequently 10m/52m/= 110m > It=E2=80=99s hard to imagine, how many time is needed to add ~60K vlan :( > Process was accelerated a little by shooting off devd, bsnmpd, ntpd servi= ces, but it found another problems and limitations. > > For example, > a) Service ntpd refuse to start at 12K interfaces: > ntpd[2195]: Too many sockets in use, FD_SETSIZE 16384 exceeded > I remind, that in files /usr/src/sys/sys/select.h and /usr/include/sys/se= lect.h FD_SETSIZE value is only 1024U > > b) Service bsnmpd started at 12K interfaces, but immediately loaded CPU a= t 80-100% > > last pid: 64011; load averages: 1.00, 0.97, 0.90 = up 0+05:25:39 21:26:36 > 58 processes: 3 running, 54 sleeping, 1 waiting > CPU: 68.2% user, 0.0% nice, 30.6% system, 1.2% interrupt, 0.0% idle > Mem: 125M Active, 66M Inact, 435M Wired, 200K Cache, 525M Free > ARC: 66M Total, 28M MFU, 36M MRU, 16K Anon, 614K Header, 2035K Other > Swap: 1024M Total, 1024M Free > > PID USERNAME THR PRI NICE SIZE RES STATE TIME WCPU COMMAN= D > 63863 root 1 96 0 136M 119M RUN 35:31 79.98% bsnmpd > ... > > c) Size of fields during output of command netstat(1) - netstat -inW is u= nsufficient (bin/188153) > > d) If indicate in command netstat of interface it=E2=80=99s impossible to= understand, which ipv4/ipv6 neworks are indicated here. > > # netstat -I ngeth123.223 -nW > Name Mtu Network Address Ipkts Ierrs Idrop Opk= ts Oerrs Coll > ngeth12 1500 08:00:27:cd:9b:8e 0 0 0 = 1 5 0 > ngeth12 - 172.18.206.13 172.18.206.139 0 - - = 0 - - > ngeth12 - fe80::a00:27f fe80::a00:27ff:fe 0 - - = 1 - - > ngeth12 - 2001:570:28:1 2001:570:28:140:: 0 - - = 0 - - > > e) Very low output of command arp: > # ngctl list | grep ngeth | wc -l > 12003 > # ifconfig -a | egrep -e 'inet ' | wc -l > 12007 > # time /usr/sbin/arp -na > /dev/null > 150.661u 551.002s 11:53.71 98.3% 20+172k 1+0io 0pf+0w > > > More info at http://freebsd.1045724.n5.nabble.com/arp-8-performance-use-i= f-nameindex-instead-of-if-indextoname-td5898205.html > > After using of patch, speed became acceptable: > > # time /usr/sbin/arp -na > /dev/null > 0.114u 0.090s 0:00.14 142.8% 20+170k 0+0io 0pf+0w > > I suspect, that output of standard network stack will be too low to accom= plish a 3rd task, routing of ~60K vlan > I have no idea, how to use netmap(4) in this situation :( > Please, help me in fulfillment of assigned task. > > P.S. > Colleague-Linuxoid is adjusting the same task and bragging: > At Debian, in test (kernel 3.13), 80K vlans arose in 20 minutes. It takes= 3 GB RAM. And deleting of these vlans also took 20 minutes. > > -- > Vladislav V. Prodan > System & Network Administrator > http://support.od.ua > +380 67 4584408, +380 99 4060508 > VVP88-RIPE > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"