Date: Wed, 27 Feb 2008 11:49:42 +0100 From: Holger Kipp <hk@alogis.com> To: freebsd-stable@freebsd.org Subject: em very slow, shared irq... on 6.3p8 Message-ID: <20080227104942.GA14530@intserv.int1.b.intern>
next in thread | raw e-mail | index | archive | help
Hello, I updated a system with 12 dc-interfaces to a new hardware with 14 em-interfaces. Yes, it is a firewall. New System is 6.2-RELEASE-p8. What I now experience between two internal networks (100MBit/s each) is the following: 1318 packets transmitted, 1317 packets received, 0% packet loss round-trip min/avg/max/stddev = 0.387/246.153/2441.392/324.142 ms tcpdump on the firewall shows similar delays (on the outgoing interface). tcpdump on the system I ping however shows very quick responses for incoming packages (ie usually less than a millisecond). I therefore assume that the problem is between receiving the irq from em<x> and getting the data from the interface on the firewall itself. My first option would be to activate polling on em-interfaces - but as I did not experience this sort of notieceable slowdown with the old dc-based firewall (without polling), maybe someone can shed some light on this strange behaviour or has other suggestions as well? More details and dmesg below. Help and suggestions welcome :-) Best regards, Holger Kipp PS: Regarding the MPSAFE-issue - I need to use ipsec, so don't have a choice here. On the firewall I see many irqs (sometimes more than 5000) very often on one of the em-interfaces using sysctl/vmstat: -------------------------------------------------------------------------------- 2 users Load 0.00 0.00 0.04 Feb 27 12:20 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 14804 3724 32800 3800 1327728 count All 722324 5248 76723364 5696 pages Interrupts Proc:r p d s w Csw Trp Sys Int Sof Flt cow 7606 total 27 7862 1 462 4105 158840 wire 1: atkb 14988 act 6: fdc0 0.2%Sys 3.0%Intr 0.0%User 0.0%Nice 96.8%Idl 547868 inact 12: psm | | | | | | | | | | 1244 cache 14: ata ++ 1326484 free 359 16: em4 daefr 2761 17: em5 Namei Name-cache Dir-cache prcfr 28: twe Calls hits % hits % react 306 52: em0 pdwak 1 53: em1 zfod pdpgs 177 55: em3 Disks twed0 ozfod intrn 2001 cpu0: time KB/t 21.33 %slo-z 114464 buf 2001 cpu1: time tps 1 1 tfree 10 dirtybuf MB/s 0.01 100000 desiredvnodes % busy 0 66594 numvnodes 19758 freevnodes -------------------------------------------------------------------------------- dmesg Copyright (c) 1992-2007 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 6.2-RELEASE-p8 #1: Thu Oct 4 16:07:31 CEST 2007 WARNING: debug.mpsafenet forced to 0 as ipsec requires Giant WARNING: MPSAFE network stack disabled, expect reduced performance. MPTable: <INTEL MUKILTEO > Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 CPU 6420 @ 2.13GHz (2128.01-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x6f6 Stepping = 6 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0xe3bd<SSE3,RSVD2,MON,DS_CPL,VMX,EST,TM2,<b9>,CX16,<b14>,<b15>> AMD Features=0x20100000<NX,LM> AMD Features2=0x1<LAHF> Cores per package: 2 real memory = 2146349056 (2046 MB) avail memory = 2095296512 (1998 MB) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0: Assuming intbase of 0 ioapic1: Assuming intbase of 24 ioapic2: Assuming intbase of 48 ioapic0 <Version 2.0> irqs 0-23 on motherboard ioapic1 <Version 2.0> irqs 24-47 on motherboard ioapic2 <Version 2.0> irqs 48-71 on motherboard kbd1 at kbdmux0 cpu0 on motherboard cpu1 on motherboard pcib0: <MPTable Host-PCI bridge> pcibus 0 on motherboard pci0: <PCI bus> on pcib0 pcib0: unable to route slot 28 INTB pcib1: <PCI-PCI bridge> irq 16 at device 1.0 on pci0 pci1: <PCI bus> on pcib1 pcib2: <MPTable PCI-PCI bridge> at device 0.0 on pci1 pci2: <PCI bus> on pcib2 twe0: <3ware Storage Controller. Driver version 1.50.01.002> port 0x4000-0x400f mem 0xe0100000-0xe010000f,0xe0800000-0xe0ffffff irq 28 at device 2.0 on pci2 twe0: [GIANT-LOCKED] twe0: 2 ports, Firmware FE8S 1.05.00.068, BIOS BE7X 1.08.00.048 pci1: <base peripheral, interrupt controller> at device 0.1 (no driver attached) pcib3: <PCI-PCI bridge> at device 0.2 on pci1 pci3: <PCI bus> on pcib3 pcib4: <MPTable PCI-PCI bridge> at device 3.0 on pci3 pci4: <PCI bus> on pcib4 em0: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x5000-0x503f mem 0xe1000000-0xe101ffff irq 52 at device 4.0 on pci4 em0: Ethernet address: 00:0e:0c:c6:cc:54 em0: [GIANT-LOCKED] em1: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x5040-0x507f mem 0xe1020000-0xe103ffff irq 53 at device 4.1 on pci4 em1: Ethernet address: 00:0e:0c:c6:cc:55 em1: [GIANT-LOCKED] em2: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x5080-0x50bf mem 0xe1040000-0xe105ffff irq 54 at device 6.0 on pci4 em2: Ethernet address: 00:0e:0c:c6:cc:56 em2: [GIANT-LOCKED] em3: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x50c0-0x50ff mem 0xe1060000-0xe107ffff irq 55 at device 6.1 on pci4 em3: Ethernet address: 00:0e:0c:c6:cc:57 em3: [GIANT-LOCKED] pci1: <base peripheral, interrupt controller> at device 0.3 (no driver attached) pcib5: <PCI-PCI bridge> irq 16 at device 3.0 on pci0 pci10: <PCI bus> on pcib5 pcib6: <PCI-PCI bridge> at device 0.0 on pci10 pci11: <PCI bus> on pcib6 pcib7: <MPTable PCI-PCI bridge> at device 0.0 on pci11 pci12: <PCI bus> on pcib7 em4: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x6000-0x601f mem 0xe1120000-0xe113ffff,0xe1100000-0xe111ffff irq 16 at device 0.0 on pci12 em4: Memory Access and/or Bus Master bits were not set! em4: Ethernet address: 00:15:17:25:48:88 em4: [GIANT-LOCKED] em5: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x6020-0x603f mem 0xe1160000-0xe117ffff,0xe1140000-0xe115ffff irq 17 at device 0.1 on pci12 em5: Memory Access and/or Bus Master bits were not set! em5: Ethernet address: 00:15:17:25:48:89 em5: [GIANT-LOCKED] pcib8: <MPTable PCI-PCI bridge> at device 1.0 on pci11 pci13: <PCI bus> on pcib8 em6: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x7000-0x701f mem 0xe1220000-0xe123ffff,0xe1200000-0xe121ffff irq 16 at device 0.0 on pci13 em6: Memory Access and/or Bus Master bits were not set! em6: Ethernet address: 00:15:17:25:48:8a em6: [GIANT-LOCKED] em7: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x7020-0x703f mem 0xe1260000-0xe127ffff,0xe1240000-0xe125ffff irq 17 at device 0.1 on pci13 em7: Memory Access and/or Bus Master bits were not set! em7: Ethernet address: 00:15:17:25:48:8b em7: [GIANT-LOCKED] pcib9: <PCI-PCI bridge> irq 17 at device 28.0 on pci0 pci14: <PCI bus> on pcib9 pcib10: <PCI-PCI bridge> at device 0.0 on pci14 pci15: <PCI bus> on pcib10 pcib11: <MPTable PCI-PCI bridge> at device 0.0 on pci15 pci16: <PCI bus> on pcib11 em8: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x8000-0x801f mem 0xe1320000-0xe133ffff,0xe1300000-0xe131ffff irq 16 at device 0.0 on pci16 em8: Memory Access and/or Bus Master bits were not set! em8: Ethernet address: 00:15:17:25:48:64 em8: [GIANT-LOCKED] em9: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x8020-0x803f mem 0xe1360000-0xe137ffff,0xe1340000-0xe135ffff irq 17 at device 0.1 on pci16 em9: Memory Access and/or Bus Master bits were not set! em9: Ethernet address: 00:15:17:25:48:65 em9: [GIANT-LOCKED] pcib12: <MPTable PCI-PCI bridge> at device 1.0 on pci15 pci17: <PCI bus> on pcib12 em10: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x9000-0x901f mem 0xe1420000-0xe143ffff,0xe1400000-0xe141ffff irq 16 at device 0.0 on pci17 em10: Memory Access and/or Bus Master bits were not set! em10: Ethernet address: 00:15:17:25:48:66 em10: [GIANT-LOCKED] em11: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0x9020-0x903f mem 0xe1460000-0xe147ffff,0xe1440000-0xe145ffff irq 17 at device 0.1 on pci17 em11: Memory Access and/or Bus Master bits were not set! em11: Ethernet address: 00:15:17:25:48:67 em11: [GIANT-LOCKED] pcib13: <MPTable PCI-PCI bridge> irq 17 at device 28.4 on pci0 pci18: <PCI bus> on pcib13 em12: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0xa000-0xa01f mem 0xe1500000-0xe151ffff irq 16 at device 0.0 on pci18 em12: Ethernet address: 00:30:48:90:86:7a em12: [GIANT-LOCKED] pcib14: <MPTable PCI-PCI bridge> irq 7 at device 28.5 on pci0 pci19: <PCI bus> on pcib14 em13: <Intel(R) PRO/1000 Network Connection Version - 6.2.9> port 0xb000-0xb01f mem 0xe1600000-0xe161ffff irq 17 at device 0.0 on pci19 em13: Ethernet address: 00:30:48:90:86:7b em13: [GIANT-LOCKED] pcib15: <MPTable PCI-PCI bridge> at device 30.0 on pci0 pci20: <PCI bus> on pcib15 pci20: <display, VGA> at device 4.0 (no driver attached) isab0: <PCI-ISA bridge> at device 31.0 on pci0 isa0: <ISA bus> on isab0 atapci0: <Intel ICH7 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x3020-0x302f at device 31.1 on pci0 ata0: <ATA channel 0> on atapci0 ata1: <ATA channel 1> on atapci0 pci0: <serial bus, SMBus> at device 31.3 (no driver attached) pmtimer0 on isa0 orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0xcdfff on isa0 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] psm0: <PS/2 Mouse> irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: model MouseMan+, device ID 0 fdc0: <Enhanced floppy controller> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0 fdc0: [FAST] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 ppc0: parallel port not found. sc0: <System console> at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0 sio0: type 16550A sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 unknown: <PNP0303> can't assign resources (port) unknown: <INT0800> can't assign resources (memory) unknown: <PNP0c02> can't assign resources (memory) unknown: <PNP0f13> can't assign resources (irq) unknown: <PNP0501> can't assign resources (port) unknown: <PNP0700> can't assign resources (port) Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing. IP Filter: v4.1.13 initialized. Default = pass all, Logging = enabled acd0: DVDROM <HL-DT-STDVD-ROM GDR8164B/0L06> at ata0-slave UDMA33 twed0: <Unit 0, TwinStor, Normal> on twe0 twed0: 76318MB (156299440 sectors) SMP: AP CPU #1 Launched! Trying to mount root from ufs:/dev/twed0s1a em9: link state changed to UP em8: link state changed to UP em10: link state changed to UP em0: link state changed to UP em3: link state changed to UP [...]
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20080227104942.GA14530>