Date: Mon, 12 May 2008 12:19:19 +0300 From: Stefan Lambrev <stefan.lambrev@moneybookers.com> To: Paul <paul@gtcomm.net> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Discrepancy on netstat -w x -I and what Cisco reports Message-ID: <48280B97.1020708@moneybookers.com> In-Reply-To: <4827E79C.8050608@gtcomm.net> References: <4827E79C.8050608@gtcomm.net>
next in thread | previous in thread | raw e-mail | index | archive | help
Greetings, I just have a question: is "netstat -w 100 lagg0" a typo ? If you want to see the traffic only on single interface you should use -I I do not know if this is bug, but netstat -w 100 something_non_existing works on my bsd and just shows "Total" So may be from here comes the confusion. You think, that netstat count only traffic on lagg0, but it shows you the Total traffic? Paul wrote: > This is very strange.. I can do: > > netstat -w 10 -I lagg0 > input (lagg0) output > packets errs bytes packets errs bytes colls > 57806 0 41751685 232442 0 51062425 0 > 56459 0 38341591 225146 0 48865209 0 > 60687 0 43552946 227987 0 52008241 0 > > which is roughly 23,000 pps and the Cisco switch reports > 30 second input rate 44544000 bits/sec, 16198 packets/sec > > > Another example: > > netstat -w 10 -I lagg0 > input (lagg0) output > packets errs bytes packets errs bytes colls > 71111 0 52180947 89734 0 25304669 0 > 66847 0 49028588 81737 0 21614941 0 > 63530 0 43502418 83419 0 24599547 0 > > 8,300 or so pps > > Cisco: > 30 second input rate 19230000 bits/sec, 4594 packets/sec > > > In some cases it's pretty close, cisco says 6500 and bsd says 7500.. > but sometimes it is way off > I even checked the em interfaces directly to see if it was a problem > with the lagg interface code and they report weird issues, check below. > Kind of weird.. I'm not sure if this is a Cisco issue or Fbsd issue > with the counters. > It's two different Cisco switches and two fbsd machines that have the > same kernel, etc. > > Maybe this is a side effect of setting my kernel HZ at 4000? > Gets even worse like this: > netstat -w 100 lagg0 > input (Total) output > packets errs bytes packets errs bytes colls > 9229322 0 3337789024 9424932 0 3510341158 0 > > 100 seconds, 9.4 million packets? That's 94,000 pps and cisco reports > 2 minute input rate 44130000 bits/sec, 14892 packets/sec Again this is Total not lagg0 > and even check this out: > > netstat -w 1 lagg0 > input *(Total) * output > packets errs bytes packets errs bytes colls > 92481 0 31630795 94952 0 34193131 0 > 89078 0 32498082 91460 0 35094821 0 > 87540 0 34526292 89992 0 37159101 0 > 88987 0 32391984 91765 0 35394351 0 > > > netstat -w 1 em0 > input *(Total) * output > packets errs bytes packets errs bytes colls > 96802 0 39474036 99897 0 42814245 0 > 93277 0 37018533 95943 0 39860879 0 > 95916 0 37559076 99032 0 40739640 0 > > netstat -w 1 em1 > input * (Total) * output > packets errs bytes packets errs bytes colls > 97102 0 38369949 99508 0 40841183 0 > 92321 0 35375169 94305 0 37384073 0 > 92225 0 33171455 94253 0 35209658 0 > > What in the world?? em0 + em1 is almost 200k pps but lagg0 says 100k > and i KNOW it's not doing anywhere near.. > cisco report > 2 minute output rate 32928000 bits/sec, 5823 packets/sec Again missing -I :) > > Now all lagg interfaces are reporting > netstat -w 1 lagg1 > input * (Total) * output > packets errs bytes packets errs bytes colls > 89324 0 30824353 91518 0 32770482 0 > 85875 0 31924738 87813 0 33552137 0 > 84105 0 31176932 85666 0 32393051 0 > 83617 0 32175677 84871 0 33120271 0 > 90611 0 37313093 92403 0 38818721 0 > > lagg1 goes directly to another freebsd box and on the other freebsd > box I do: > netstat -w 1 -I lagg1 > input (lagg1) output > packets errs bytes packets errs bytes colls > 45 0 3078 2213 0 1890198 0 > 48 0 3245 1958 0 1545642 0 > 43 0 3186 1975 0 1628916 0 > 43 0 2905 2169 0 1918250 0 > 46 0 3464 1859 0 1729764 0 > 46 0 3134 1873 0 1739662 0 > > and the other one > netstat -w 1 lagg1 > input * (Total)* output > packets errs bytes packets errs bytes colls > 92149 0 31706183 93523 0 32673138 0 > 89737 0 28119643 91323 0 28958816 0 > > > Doing all these reports now seems to have the interfaces stuck at > packets errs bytes packets errs bytes colls > 96937 0 31749525 98551 0 32678863 0 > 85892 0 29411078 87233 0 30182355 0 > 90435 0 31628680 91620 0 32215244 0 > 87383 0 30616741 88278 0 31026608 0 > > > every interface on the machine is reporting the same PPS and bytes.. > lol :) > > So something is extremely fishy about the counters.. I'm going to try > and update to -STABLE to see if it makes any difference. It's not > just the lagg interface either because all the em's are showing it as > well. > > This is using 4 port Intel Server PCI Express NIC > > ifstat seems to report correct usage in Kbps and seems to report > correct packet count. Maybe it's just a netstat problem? > > I will see if stable fixes it. Also, feel free to make any comments > on my config file for routing. > > FreeBSD 7.0-RELEASE-p1 FreeBSD 7.0-RELEASE-p1 #6: Thu Apr 17 18:11:49 > EDT 2008 amd64 > > > UPDATE.. Changed 1 router to stable: > FreeBSD 7.0-STABLE FreeBSD 7.0-STABLE #5: Sun May 11 19:00:57 EDT > 2008 amd64 > > Still see: > 2 minute input rate 10463000 bits/sec, 2481 packets/sec > 2 minute output rate 40075000 bits/sec, 6847 packets/sec > > and > input (lagg0) output > packets errs bytes packets errs bytes colls > 6940 0 5172153 4841 0 1345660 0 > 5922 0 4252074 3963 0 1087205 0 > 6673 0 4982394 4116 0 1056933 0 > 6659 0 4467398 4140 0 1068919 0 > 7085 0 4692973 4777 0 1665109 0 > 7140 0 4654486 4713 0 1658303 0 > 7070 0 4558384 5078 0 1994666 0 > 6375 0 4575464 4037 0 1121385 0 > 6257 0 3932910 4321 0 1607862 0 > 6504 0 4345014 4370 0 1278819 0 > > > Hmmmm.. > > > > > em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port > 0xece0-0xecff mem 0xd5ee0000-0xd5efffff,0xd5ec0000-0xd5edffff irq 17 > at device 0.0 on pci12 > em0: Using MSI interrupt > em0: Ethernet address: 00:15:17:5d:18:00 > em0: [FILTER] > em1: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port > 0xecc0-0xecdf mem 0xd5ea0000-0xd5ebffff,0xd5e80000-0xd5e9ffff irq 18 > at device 0.1 on pci12 > em1: Using MSI interrupt > em1: Ethernet address: 00:15:17:5d:18:01 > em1: [FILTER] > .......etc.. to em7 > > > Copyright (c) 1992-2008 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 7.0-RELEASE-p1 #6: Thu Apr 17 18:11:49 EDT 2008 > root@CR1.MTL3.Gtcomm.net:/usr/obj/usr/src/sys/ROUTER > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2329.28-MHz > K8-class CPU) > Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6 > > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > > > Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA> > > AMD Features=0x20100800<SYSCALL,NX,LM> > AMD Features2=0x1<LAHF> > Cores per package: 2 > usable memory = 4285833216 (4087 MB) > avail memory = 4124545024 (3933 MB) > ACPI APIC Table: <DELL PE_SC3 > > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 6 > cpu3 (AP): APIC ID: 7 > ioapic0: Changing APIC ID to 8 > ioapic1: Changing APIC ID to 9 > > > > Kernel config file: (if you have any suggestions about the config let > me know for faster routing speed) > > cpu HAMMER > ident GENERIC > > #makeoptions DEBUG=-g # Build kernel with gdb(1) > debug symbols > > options SCHED_4BSD # 4BSD scheduler > #options PREEMPTION # Enable kernel thread preemption > options INET # InterNETworking > options INET6 # IPv6 communications protocols > options FFS # Berkeley Fast Filesystem > options SOFTUPDATES # Enable FFS soft updates support > options UFS_ACL # Support for access control > lists > options UFS_DIRHASH # Improve performance on big > directories > options UFS_GJOURNAL # Enable gjournal-based UFS > journaling > options MD_ROOT # MD is a potential root device > options NTFS # NT File System > options MSDOSFS # MSDOS Filesystem > options CD9660 # ISO 9660 Filesystem > options PROCFS # Process filesystem (requires > PSEUDOFS) > options PSEUDOFS # Pseudo-filesystem framework > options GEOM_PART_GPT # GUID Partition Tables. > options GEOM_LABEL # Provides labelization > options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP THIS!] > options COMPAT_IA32 # Compatible with i386 binaries > options COMPAT_FREEBSD4 # Compatible with FreeBSD4 > options COMPAT_FREEBSD5 # Compatible with FreeBSD5 > options COMPAT_FREEBSD6 # Compatible with FreeBSD6 > options SCSI_DELAY=5000 # Delay (in ms) before probing > SCSI > options KTRACE # ktrace(1) support > options SYSVSHM # SYSV-style shared memory > options SYSVMSG # SYSV-style message queues > options SYSVSEM # SYSV-style semaphores > options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time > extensions > options KBD_INSTALL_CDEV # install a CDEV entry in /dev > #options ADAPTIVE_GIANT # Giant mutex is adaptive. > options NO_ADAPTIVE_MUTEXES ## > options STOP_NMI # Stop CPUS using NMI instead > of IPI > options AUDIT # Security event auditing > options IPSEC ## for tcp md5 > options TCP_SIGNATURE ##include support for RFC 2385 > device crypto ## for md5 > device cryptodev ## for md5 > > # Make an SMP-capable kernel by default > options SMP # Symmetric MultiProcessor Kernel > > # CPU frequency control > device cpufreq > > # Bus support. > device acpi > device pci > > # Floppy drives > device fdc > > # ATA and ATAPI devices > device ata > device atadisk # ATA disk drives > device ataraid # ATA RAID drives > device atapicd # ATAPI CDROM drives > device atapifd # ATAPI floppy drives > device atapist # ATAPI tape drives > options ATA_STATIC_ID # Static device numbering > > # SCSI peripherals > device scbus # SCSI bus (required for SCSI) > device ch # SCSI media changers > device da # Direct Access (disks) > device sa # Sequential Access (tape etc) > device cd # CD > device pass # Passthrough device (direct SCSI access) > device ses # SCSI Environmental Services (and > SAF-TE) > > # RAID controllers > device mfi # LSI MegaRAID SAS > > # atkbdc0 controls both the keyboard and the PS/2 mouse > device atkbdc # AT keyboard controller > device atkbd # AT keyboard > device psm # PS/2 mouse > > device kbdmux # keyboard multiplexer > > device vga # VGA video card driver > > device splash # Splash screen and screen saver support > > # syscons is the default console driver, resembling an SCO console > device sc > > device agp # support several AGP chipsets > > # Serial (COM) ports > device sio # 8250, 16[45]50 based serial ports > device uart # Generic UART driver > > # PCI Ethernet NICs. > device em # Intel PRO/1000 adapter Gigabit > Ethernet Card > # PCI Ethernet NICs that use the common MII bus controller code. > # NOTE: Be sure to keep the 'device miibus' line in order to use these > NICs! > device miibus > device bce # Broadcom BCM5706/BCM5708 Gigabit > Ethernet > > # Pseudo devices. > device loop # Network loopback > device random # Entropy device > device ether # Ethernet support > device sl # Kernel SLIP > device ppp # Kernel PPP > device tun # Packet tunnel. > device pty # Pseudo-ttys (telnet etc) > device md # Memory "disks" > device gif # IPv6 and IPv4 tunneling > device faith # IPv6-to-IPv4 relaying (translation) > device firmware # firmware assist module > > # The `bpf' device enables the Berkeley Packet Filter. > # Be aware of the administrative consequences of enabling this! > # Note that 'bpf' is required for DHCP. > device bpf # Berkeley packet filter > > # USB support > device uhci # UHCI PCI->USB interface > device ohci # OHCI PCI->USB interface > device ehci # EHCI PCI->USB interface (USB 2.0) > device usb # USB Bus (required) > #device udbp # USB Double Bulk Pipe devices > device ugen # Generic > device uhid # "Human Interface Devices" > device ukbd # Keyboard > device umass # Disks/Mass storage - Requires scbus > and da > device ums # Mouse > > ### OPTIONS > > > options MP_WATCHDOG > options DEVICE_POLLING > device pf > device pflog > device pfsync > device carp > device vlan > device gre > device if_bridge > device tun > device lagg > device stf #6to4 IPv6 over IPv4 encapsulation > > options ALTQ > options ALTQ_CBQ # Class Bases Queuing (CBQ) > options ALTQ_RED # Random Early Detection (RED) > options ALTQ_RIO # RED In/Out > options ALTQ_HFSC # Hierarchical Packet Scheduler (HFSC) > options ALTQ_CDNR > options ALTQ_PRIQ # Priority Queuing (PRIQ) > options ALTQ_NOPCC # Required for SMP build > > > > options NETGRAPH > options NETGRAPH_CISCO > options NETGRAPH_FEC > options NETGRAPH_ETHER > > > > > > Copyright (c) 1992-2008 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 7.0-STABLE #5: Sun May 11 19:00:57 EDT 2008 > root@CR2.MTL3.Gtcomm.net:/usr/obj/usr/src/sys/ROUTER > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Xeon(R) CPU 5140 @ 2.33GHz (2329.26-MHz > K8-class CPU) > Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6 > > Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > > > Features2=0x4e3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA> > > AMD Features=0x20100800<SYSCALL,NX,LM> > AMD Features2=0x1<LAHF> > Cores per package: 2 > usable memory = 4286042112 (4087 MB) > avail memory = 4124753920 (3933 MB) > ACPI APIC Table: <DELL PE_SC3 > > FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > cpu2 (AP): APIC ID: 6 > cpu3 (AP): APIC ID: 7 > ioapic0: Changing APIC ID to 8 > ioapic1: Changing APIC ID to 9 > ioapic0 <Version 2.0> irqs 0-23 on motherboard > ioapic1 <Version 2.0> irqs 64-87 on motherboard > kbd1 at kbdmux0 > cryptosoft0: <software crypto> on motherboard > acpi0: <DELL PE_SC3> on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 > acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff > on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 900 > cpu0: <ACPI CPU> on acpi0 > est0: <Enhanced SpeedStep Frequency Control> on cpu0 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 728072806000728 > device_attach: est0 attach returned 6 > p4tcc0: <CPU Frequency Thermal Control> on cpu0 > cpu1: <ACPI CPU> on acpi0 > est1: <Enhanced SpeedStep Frequency Control> on cpu1 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 728072806000728 > device_attach: est1 attach returned 6 > p4tcc1: <CPU Frequency Thermal Control> on cpu1 > cpu2: <ACPI CPU> on acpi0 > est2: <Enhanced SpeedStep Frequency Control> on cpu2 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 728072806000728 > device_attach: est2 attach returned 6 > p4tcc2: <CPU Frequency Thermal Control> on cpu2 > cpu3: <ACPI CPU> on acpi0 > est3: <Enhanced SpeedStep Frequency Control> on cpu3 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 728072806000728 > device_attach: est3 attach returned 6 > p4tcc3: <CPU Frequency Thermal Control> on cpu3 > pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0 > pci0: <ACPI PCI bus> on pcib0 > pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0 > pci6: <ACPI PCI bus> on pcib1 > pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci6 > pci7: <ACPI PCI bus> on pcib2 > pcib3: <ACPI PCI-PCI bridge> at device 0.0 on pci7 > pci8: <ACPI PCI bus> on pcib3 > pcib4: <PCI-PCI bridge> at device 0.0 on pci8 > pci9: <PCI bus> on pcib4 > bce0: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem > 0xd6000000-0xd7ffffff irq 16 at device 0.0 on pci9 > miibus0: <MII bus> on bce0 > brgphy0: <BCM5708C 10/100/1000baseTX PHY> PHY 1 on miibus0 > brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, > 1000baseT-FDX, auto > bce0: Ethernet address: 00:19:b9:cd:60:44 > bce0: [ITHREAD] > bce0: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W > (0x02090105); Flags( MFW MSI ) > pcib5: <ACPI PCI-PCI bridge> at device 1.0 on pci7 > pci10: <ACPI PCI bus> on pcib5 > pcib6: <PCI-PCI bridge> at device 0.0 on pci10 > pci11: <PCI bus> on pcib6 > pcib7: <PCI-PCI bridge> at device 0.0 on pci11 > pci12: <PCI bus> on pcib7 > em0: <Intel(R) PRO/1000 Network Connection 6.9.0> port 0xece0-0xecff > mem 0xd5ee0000-0xd5efffff,0xd5ec0000-0xd5edffff irq 17 at device 0.0 > on pci12 > em0: Using MSI interrupt > em0: [FILTER] > em0: Ethernet address: 00:15:17:5d:2a:40 > em1: <Intel(R) PRO/1000 Network Connection 6.9.0> port 0xecc0-0xecdf > mem 0xd5ea0000-0xd5ebffff,0xd5e80000-0xd5e9ffff irq 18 at device 0.1 > on pci12 > em1: Using MSI interrupt > em1: [FILTER] > em1: Ethernet address: 00:15:17:5d:2a:41 > pcib8: <PCI-PCI bridge> at device 1.0 on pci11 > pci13: <PCI bus> on pcib8 > em2: <Intel(R) PRO/1000 Network Connection 6.9.0> port 0xdce0-0xdcff > mem 0xd5ce0000-0xd5cfffff,0xd5cc0000-0xd5cdffff irq 18 at device 0. > > > lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu > 1500 > options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4> > ether 00:15:17:5d:2a:40 > media: Ethernet autoselect > status: active > laggproto lacp > laggport: em1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > laggport: em0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > > > lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu > 1500 > options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4> > ether 00:15:17:5d:28:62 > inet netmask 0xfffffffc broadcast > media: Ethernet autoselect > status: active > laggproto lacp > laggport: em7 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > laggport: em6 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > > lagg2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu > 1500 > options=19b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4> > ether 00:15:17:5d:28:60 > media: Ethernet autoselect > status: active > laggproto lacp > laggport: em5 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > laggport: em4 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING> > > > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" -- Best Wishes, Stefan Lambrev ICQ# 24134177
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?48280B97.1020708>