Skip site navigation (1)Skip section navigation (2)
Date:      Tue, 9 Mar 2004 21:57:27 +1100
From:      Tony Frank <>
To:        Jimmy Scott <>
Subject:   Re: ipfw counters (field 3)
Message-ID:  <20040309105727.GA8528@marvin.home.local>
In-Reply-To: <>
References:  <>

Next in thread | Previous in thread | Raw E-Mail | Index | Archive | Help
Hi there,

On Mon, Mar 08, 2004 at 02:55:57PM +0100, Jimmy Scott wrote:
> Me and a friend of my are located in a datacenter on the same switch,
> We both have the same 3 rules, to count the traffic.
> Our ISP uses MRTG with the 95% rule to charge us,
> now i was wondering, i have 2MB traffic echt day, theire MRTG says 70MB
> i thought it was because of all the ARP traffic.
> But my friend counted 1,6944 GB traffic, while our ISP's MRTG points 2GB
> i've let crond mail me his results from 00:01 till 00:01, so these are
> both mails:
> 00010 11345045 1068348938 count ip from any to any via xl0
> 00011  6826150  466872667 count ip from any to any in recv xl0
> 00012  4518893  601476157 count ip from any to any out xmit xl0
> 00010 27743578 2887729820 count ip from any to any via xl0
> 00011 17168887 1405455507 count ip from any to any in recv xl0
> 00012 10574689 1482274199 count ip from any to any out xmit xl0
> 1) is there something except ARP that isn't counted here? i thought this
> is with IP headers inclusive. (we don't have any ESP traffic)
> 2) is it because of theire 95% rule?
> 3) is theire system absolutely not correct?
> in case of number 3, how can i prove it to my ISP, just normal ipfw logs
> and kernel ipfw counters won't do imho

I believe that counting at ipfw 'ip' level should count IP traffic only.
Ie no ethernet headers, ARP, STP, CDP, IPX or whatever other protocols come
Note that ethernet headers will add minimum 14 bytes per packet.  
Possibly also more for any padding/checksums needed, also if you use
extras like vlan tagging these would be counted by a switch but not at IP

Depending on your ISP equipment, their switch port may count all those 
bytes transmitted at layer2 level.

You perhaps need to identify 100% exactly what is measured by ISP.
MRTG is just a tool that will gather statistics - you can run it on your
freebsd computer too and measure your interface traffic.
If they are counting layer3 traffic (IP) to your address, then the numbers 
seem to mismatch.  If they are counting layer2 traffic to your switch port, 
then there are a lot of extras you will not see at IP level where ipfw counts.

You can see the raw traffic levels by using "netstat -ib" command. 
I believe this will show you everything that the interface sees.
See the netstat(1) man page for more info on the -i options.

If you are using ipfw2 you might be able to use the sysctl to make ipfw see the layer2 packets and count them.
I'd strongly suggest careful reading of the ipfw man page before trying
that however or you might find unexpected results.

I'd also look at any one of the many accounting packages in the ports

Personally I would probably setup mrtg or cricket and use them to 
count & graph the traffic at your end.  You then have your own records.
If you feel you are being unfairly charged you can compare the numbers.

Just be sure you are both measuring the same numbers.

In my case:

> netstat -ibd
Name    Mtu Network       Address              Ipkts Ierrs     Ibytes    Opkts Oerrs     Obytes  Coll Drop
fxp1   1500 <Link#2>    00:04:ac:e5:d3:59  1252413     0  157415021   142427     0   18572480     0   0
fxp1   1500 netname     hostname             37303     -   26725212   142424     -   16577800     -   -

'ipfw show' for same time period gives me:

00500   188899    90900442 count ip from any to any in recv fxp1
00501   141977    16560196 count ip from any to any out xmit fxp1

As you can see there is a bit of a difference between layer2 and layer3 counts.
In my case that is 99.9% due to excess ARP messages seen on my cable modem.
Fortunately my ISP only counts the layer3 unicast traffic to my address.
So while the extra ARP clogs up the interface it is not counted towards usage.

Though I am not currently sure why the inet 'ipkts' figure is so low on the netstat print.
Possibly something to do with the natd/divert I'm using.

Hope that helps,


Want to link to this message? Use this URL: <>