From owner-freebsd-net@freebsd.org Fri Dec 20 18:08:57 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5D2971DFFC6 for ; Fri, 20 Dec 2019 18:08:57 +0000 (UTC) (envelope-from darkfiberiru@gmail.com) Received: from mail-qt1-x82c.google.com (mail-qt1-x82c.google.com [IPv6:2607:f8b0:4864:20::82c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47fcFN2DpLz4FLk for ; Fri, 20 Dec 2019 18:08:56 +0000 (UTC) (envelope-from darkfiberiru@gmail.com) Received: by mail-qt1-x82c.google.com with SMTP id e6so8937651qtq.7 for ; Fri, 20 Dec 2019 10:08:56 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=LfgakmNsz6lbGFa/nUIPM6j/8xjgp3sSNzBwUXJrBXc=; b=U7aN3fK52uh814OVq7+VnFraveehYQ9ezIEQRqUePflspvAs7v34A4kFCbOvNliwpE ZdS9M3oRsHx58TiIH2DMGo59RRCj2Mjcb5RHFSVrKZ2rz3n8UkVVplAfmW17CHehdVUu YkIyDhmmSWdjS9ZloKnWeos/AYuBiWxMXqtorQsrJ1brqfp8RZ45UhJzVYvYMs7ICnWv KPG6RbOCDt1OaduKgsk7/TeBHBvzlg0fAAJWzg9NVyronZx6hZkFVn5OpDYAaMy5hO04 1NPBxRD6FJQNOwKWsrwFJ2Vk7rv4+qCY+tkiVb8Skv5xDTHWGCCJ9IzOPMryzmkEjsbt A10w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=LfgakmNsz6lbGFa/nUIPM6j/8xjgp3sSNzBwUXJrBXc=; b=BK6NyERtKOI7C34tIouM7B7Flxbhzjd8NGOOXGZPYPxBU5KRYBucgzcaftSdYk844X byn36bVjLiXYzyF2EbsW0OKMHEbdZjsKv9ft2B16l+Suqm+FlVgtp3zYviXSaEyEqt2F sGRUjFp5nM6xkMjQIkGJssXEGTdKJ+nrjHqnp8cLkYiTrm4ARLOgaigQj+1sDbGmPInV l06sgUAxfwhgyQioa1ShONXL61rzqmU539Aa5NaBGoDTv69YQ0XPFaGWZuItPC/ZV6th 4QCGh8lXhNArqJhrVb6FFNt1gXxzhxg2FzdqePrbHWvbDmBJutoNfDvn3vqIKi44JJCN ReKQ== X-Gm-Message-State: APjAAAUB+UXbPbxqdlrC5gxjsMUUeU5mfdkHzBUs+ptxZdBKH/tc26Zj /R1/ABgO1i3l3VfnwsJHYlMW1r/hymmtBp0Ltp9ERQ== X-Google-Smtp-Source: APXvYqz0pLE2cpBl7OrtPaVELUDnBbWTNuBe6IUn5OEhnK8n0zNGyO0qIpIiNdY2YKqhs7BUCZ0hLziRiFkk19EzKdU= X-Received: by 2002:ac8:145:: with SMTP id f5mr12509001qtg.194.1576865334784; Fri, 20 Dec 2019 10:08:54 -0800 (PST) MIME-Version: 1.0 References: <20191220122256.76942c07@x23> In-Reply-To: <20191220122256.76942c07@x23> From: Nick Wolff Date: Fri, 20 Dec 2019 13:09:52 -0500 Message-ID: Subject: Re: Continuing problems in a bridged VNET setup To: Marko Zec Cc: "Patrick M. Hausen" , Kristof Provost , "freebsd-net@freebsd.org" X-Rspamd-Queue-Id: 47fcFN2DpLz4FLk X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=U7aN3fK5; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of darkfiberiru@gmail.com designates 2607:f8b0:4864:20::82c as permitted sender) smtp.mailfrom=darkfiberiru@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (-9.24), ipnet: 2607:f8b0::/32(-2.18), asn: 15169(-1.89), country: US(-0.05)]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[c.2.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Dec 2019 18:08:57 -0000 Marko, Are you aware of any write ups for using ng_eiface and ng_bridge instead of if_bridge? Thanks, Nick Wolff On Fri, Dec 20, 2019 at 6:22 AM Marko Zec wrote: > Perhaps you could ditch if_bridge(4) and epair(4), and try ng_eiface(4) > with ng_bridge(4) instead? Works rock-solid 24/7 here on 11.2 / 11.3. > > Marko > > On Fri, 20 Dec 2019 11:19:24 +0100 > "Patrick M. Hausen" wrote: > > > Hi all, > > > > we still experience occasional network outages in production, > > yet have not been able to find the root cause. > > > > We run around 50 servers with VNET jails. some of them with > > a handful, the busiest ones with 50 or more jails each. > > > > Every now and then the jails are not reachable over the net, > > anymore. The server itself is up and running, all jails are > > up and running, one can ssh to the server but none of the > > jails can communicate over the network. > > > > There seems to be no pattern to the time of occurrance except > > that more jails on one system make it "more likely". > > Also having more than one bridge, e.g. for private networks > > between jails seems to increase the probability. > > When a server shows the problem it tends to get into the state > > rather frequently, a couple of hours inbetween. Then again > > most servers run for weeks without exhibiting the problem. > > That's what makes it so hard to reproduce. The last couple of > > days one system was failing regularly until we reduced the number > > of jails from around 80 to around 50. Now it seems stable again. > > > > I have a test system with lots of jails that I work with gatling > > that did not show a single failure so far :-( > > > > > > Setup: > > > > All jails are iocage jails with VNET interfaces. They are > > connected to at least one bridge that starts with the > > physical external interface as a member and gets jails' > > epair interfaces added as they start up. All jails are managed > > by iocage. > > > > ifconfig_igb0="-rxcsum -rxcsum6 -txcsum -txcsum6 -vlanhwtag > > -vlanhwtso up" cloned_interfaces="bridge0" > > ifconfig_bridge0_name="inet0" > > ifconfig_inet0="addm igb0 up" > > ifconfig_inet0_ipv6="inet6 /64 auto_linklocal" > > > > $ iocage get interfaces vpro0087 > > vnet0:inet0 > > > > $ ifconfig inet0 > > inet0: flags=8843 metric 0 > > mtu 1500 ether 90:1b:0e:63:ef:51 > > inet6 fe80::921b:eff:fe63:ef51%inet0 prefixlen 64 scopeid 0x4 > > inet6 prefixlen 64 > > nd6 options=21 > > groups: bridge > > id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 > > maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200 > > root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 > > member: vnet0.4 flags=143 > > ifmaxaddr 0 port 7 priority 128 path cost 2000 > > member: vnet0.1 flags=143 > > ifmaxaddr 0 port 6 priority 128 path cost 2000 > > member: igb0 flags=143 > > ifmaxaddr 0 port 1 priority 128 path cost 2000000 > > > > > > What we tried: > > > > At first we suspected the bridge to become "wedged" somehow. This was > > corroborated by talking to various people at devsummits and EuroBSDCon > > with Kristof Provost specifically suggesting that if_bridge was > > still under giant lock and there might be a problem here that the > > lock is not released under some race condition and then the entire > > bridge subsystem would be stalled. That sounds plausible given the > > random occurrance. > > > > But I think we can rule out that one, because: > > > > - ifconfig up/down does not help > > - the host is still communicating fine over the same bridge interface > > - tearing down the bridge, kldunload (!) of if_bridge.ko followed by > > a new kldload and reconstructing the members with `ifconfig addm` > > does not help, either > > - only a host reboot restores function > > > > Finally I created a not iocage managed jail on the problem host. > > Please ignore the `iocage` in the path, I used it to populate the > > root directory. But it is not started by iocage at boot time and > > the manual config is this: > > > > testjail { > > host.hostname = "testjail"; # hostname > > path = "/iocage/jails/testjail/root"; # root directory > > exec.clean; > > exec.system_user = "root"; > > exec.jail_user = "root"; > > vnet; > > vnet.interface = "epair999b"; > > exec.prestart += "ifconfig epair999 create; ifconfig > > epair999a inet6 2A00:B580:8000:8000::1/64 auto_linklocal"; > > exec.poststop += "sleep 2; ifconfig epair999a destroy; sleep 2"; > > # Standard stuff > > exec.start += "/bin/sh /etc/rc"; > > exec.stop = "/bin/sh /etc/rc.shutdown"; > > exec.consolelog = "/var/log/jail_testjail_console.log"; > > mount.devfs; #mount devfs > > allow.raw_sockets; #allow ping-pong > > devfs_ruleset="4"; #devfs ruleset for this jail > > } > > > > $ cat /iocage/jails/testjail/root/etc/rc.conf > > hostname="testjail" > > > > ifconfig_epair999b_ipv6="inet6 2A00:B580:8000:8000::2/64 > > auto_linklocal" > > > > When I do `service jail onestart testjail` I can then ping6 the jail > > from the host and the host from the jail. As you can see the > > if_bridge is not involved in this traffic. > > > > When the host is in the wedged state and I start this testjail the > > same way, no communication across the epair interface is possible. > > > > To me this seems to indicate that not the bridge but all epair > > interfaces stop working at the very same time. > > > > > > OS is RELENG_11_3, hardware and specifically network adapters vary, > > we have igb, ix, ixl, bnxt ... > > > > > > Does anyone have a suggestion what diagnostic measures could help to > > pinpoint the culprit? The random occurrance and the fact that the > > problem seems to prefer the production environment only makes this a > > real pain ... > > > > > > Thanks and kind regards, > > Patrick > > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >