From owner-freebsd-current@FreeBSD.ORG Mon Feb 17 17:21:34 2014 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id C8DEC98F for ; Mon, 17 Feb 2014 17:21:34 +0000 (UTC) Received: from mail-vc0-x233.google.com (mail-vc0-x233.google.com [IPv6:2607:f8b0:400c:c03::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 806281985 for ; Mon, 17 Feb 2014 17:21:34 +0000 (UTC) Received: by mail-vc0-f179.google.com with SMTP id lh14so11798132vcb.24 for ; Mon, 17 Feb 2014 09:21:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=MR+j5qlBW+2pqvV2EXz7ovc0RJjrrMF2efWjDF4wtYc=; b=KwN1qxmCI6AOvACySEN0mPFDBK07hCOfZ5mO5uN7DL5N8SmFz42R2qUXlP424pEh5R VJk/uGGcIfb5EkQ4mA2yF0wGEIncMAo2KAdf2NamTPhxDk/o6a3zXlRJjwAieOuU54qk hy9stbArxJXRavXBYOw6a1aeyeyEc7VIT8/NmBRU8b9ltIU+0hHMw13489Sl0nDcEpm8 A9XQtqWoUMYWZVBWxc2S+7VyIdCJ+4QnGSYp3ROmh3AGfGmbbqH0jJ0G2bSpvDlrWtnb Yc2FS2V1fUlfUOfDfsjNzZbi9pHEWqZtROldreaqJL4PlfCNCNybNxKcOW8Qaw0dc2g/ olUw== MIME-Version: 1.0 X-Received: by 10.52.27.132 with SMTP id t4mr5286444vdg.11.1392657693509; Mon, 17 Feb 2014 09:21:33 -0800 (PST) Received: by 10.220.11.130 with HTTP; Mon, 17 Feb 2014 09:21:33 -0800 (PST) In-Reply-To: <52FD297E.6040502@allanjude.com> References: <52FD297E.6040502@allanjude.com> Date: Mon, 17 Feb 2014 11:21:33 -0600 Message-ID: Subject: Re: ezjails, systat -ifstat, and multiple network cards From: Preston Hagar To: Allan Jude Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.17 Cc: freebsd-current@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Feb 2014 17:21:34 -0000 On Thu, Feb 13, 2014 at 2:22 PM, Allan Jude wrote: > On 2014-02-13 13:59, Preston Hagar wrote: > > I have a server setup with FreeBSD-10.0-RELEASE. It has 3 Intel gigabit > > network cards in it, em0, em1, and em2. I have multiple ezjails setup > that > > run various things. > > > > One jail, called db, runs a postgresql database. It was my intention to > > give it em0 all to itself. The other jails and host machine should be > > going through em2. em1 currently isn't being used. > > > > If I do an ifconfig, I see that em0 has the alias IP for my db jail and > em2 > > has the alias IP for all other jails. All the jails respond to network > > traffic as expected and seemingly work fine. > > > > The weird thing is when I do a systat -ifstat from the host, it should > > essentially all traffic going through em0. Some of the jails that run > off > > of em2 (as defined in their jail config files and seen in ifconfig) have > > large data transfers and/or are web servers with lots of photos. I have > > even tried to manually scp a large file out of a jail setup through em2 > and > > the numbers don't seem to budge. > > > > If I do netstat -i -b -n -I and check em0 and em2, it seems to support > the > > numbers shown by systat -ifstat. However, if I use trafshow or iftop > (both > > of which require choosing one interface at a time), they both seem to > > indicate the traffic flowing through the interfaces as I would expect. > > > > So I was curious if anyone had seen something like this before or had any > > ideas of what is going on. I have net.fibs=2 set in /boot/loader.conf, > but > > in all the jails I current have jail_name_fib="" as I haven't got around > to > > fullying setting up fibs. Is that perhaps the issue? Is there any way > to > > determine with certainty which jail is using which interface short of > > physically pulling a network cable and seeing what stops working? > > > > Here are the relevant lines from my db (the one that should be on em0) > > config: > > > > export jail_db_hostname="db" > > export jail_db_ip="em0|10.1.10.2" > > > > From another jail on em2 called www: > > > > export jail_www_hostname="www" > > export jail_www_ip="em2|10.1.10.7" > > > > from ifconfig > > > > em0: flags=8843 metric 0 mtu 1500 > > > options=4219b > > ether 08:60:6e:13:94:06 > > inet 10.1.1.4 netmask 0xffff0000 broadcast 10.1.255.255 > > inet6 fe80::a60:6eff:fe13:9406%em0 prefixlen 64 scopeid 0x1 > > inet 10.1.10.2 netmask 0xffffffff broadcast 10.1.10.2 > > nd6 options=29 > > media: Ethernet autoselect (1000baseT ) > > status: active > > > > em2: flags=8843 metric 0 mtu 1500 > > > options=4219b > > ether 68:05:ca:13:74:2a > > inet 10.1.1.2 netmask 0xffff0000 broadcast 10.1.255.255 > > inet6 fe80::6a05:caff:fe13:742a%em2 prefixlen 64 scopeid 0x3 > > inet 10.1.10.3 netmask 0xffffffff broadcast 10.1.10.3 > > inet 10.1.10.1 netmask 0xffffffff broadcast 10.1.10.1 > > inet 10.1.10.8 netmask 0xffffffff broadcast 10.1.10.8 > > inet 10.1.10.10 netmask 0xffffffff broadcast 10.1.10.10 > > inet 10.1.10.4 netmask 0xffffffff broadcast 10.1.10.4 > > inet 10.1.10.9 netmask 0xffffffff broadcast 10.1.10.9 > > inet 10.1.10.7 netmask 0xffffffff broadcast 10.1.10.7 > > nd6 options=29 > > media: Ethernet autoselect (1000baseT ) > > status: active > > > > > > Let me know if any more detail would be helpful or if you have any ideas > of > > things to check. > > > > Thanks, > > > > Preston > > _______________________________________________ > > freebsd-current@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-current > > To unsubscribe, send any mail to " > freebsd-current-unsubscribe@freebsd.org" > > > > All traffic going out from the jails will using the routing table from > the host system. The routing table will use the network card that is in > the same subnet as your default gateway to route the traffic to the > internet. > > In your case, I would imagine this is 10.1.1.4/16 (and 10.1.1.2/16). > > 'netstat -rn' will tell the tale, but I imagine it is whichever was > added first. > > If you want to have separate routing tables per jail, you'd have to > either use FIBs, and set the jails to use the different FIBs, or use > VNET jails and have a routing table in each jail. > > -- > Allan Jude > > Makes sense, thank you. I'll setup the FIBs. Preston