Date: Mon, 29 Sep 2008 10:41:41 +0300 (EEST) From: "Zin'kov Oleg" <noc@bg.net.ua> To: "Dmitriy" <_pppp@mail.ru> Cc: freebsd-net@freebsd.org, Zin'kov Oleg <noc@bg.net.ua> Subject: Re: Problem with process parallelization Message-ID: <38eb68e1e4bd90145fe7e90708c8979f.squirrel@webmail.bg.net.ua> In-Reply-To: <E1KcfmE-0000KR-00._pppp-mail-ru@f106.mail.ru> References: <79dc33e3f3737f5beeadce88e96004bc.squirrel@webmail.bg.net.ua> <E1KcfmE-0000KR-00._pppp-mail-ru@f106.mail.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
>> Hello, freebsd-net mailing list. >> >> We have server such configurtion: >> - 2 quadcore AMD Opteron processors; >> - 4 GB RAM; >> - NIC Intel Pro/1000 PT, Dual Port Server Adapter. >> >> ########################################################### >> >> Problem: >> >> in some moments of time, at the growth of the network activity, one of >> the processors is fully loaded at 100%. >> >> ########################################################### >> >> Kernel configuration: >> >> FreeBSD atlantis.bg.net.ua 7.0-STABLE FreeBSD 7.0-STABLE #1: Tue Apr 1 >> 15:06:30 EEST 2008 >> root@atlantis.bg.net.ua:/usr/obj/usr/src/sys/ATLANTIS amd64 >> >> /etc/sysctl.conf: >> >> net.inet.tcp.blackhole=2 >> net.inet.udp.blackhole=1 >> kern.ipc.somaxconn=16384 >> net.inet.ip.fastforwarding=1 >> net.inet.ip.maxfragpackets=2000 >> net.inet.ip.intr_queue_maxlen=1000 >> net.inet.ip.dummynet.hash_size=2048 >> net.inet.tcp.recvspace=65536 >> net.inet.udp.recvspace=65536 >> net.inet.raw.recvspace=32768 >> net.local.stream.recvspace=32768 >> net.local.dgram.recvspace=32768 >> net.local.stream.sendspace=32768 >> net.inet.tcp.sendspace=65536 >> net.inet.icmp.icmplim=500 >> dev.em.0.rx_int_delay=500 >> dev.em.0.tx_int_delay=500 >> dev.em.0.rx_abs_int_delay=800 >> dev.em.0.tx_abs_int_delay=800 >> dev.em.1.rx_int_delay=500 >> dev.em.1.tx_int_delay=500 >> dev.em.1.rx_abs_int_delay=800 >> dev.em.1.tx_abs_int_delay=800 >> net.link.ether.inet.max_age=600 >> >> /boot/loader.conf: >> >> hw.em.rxd=4096 >> hw.em.txd=4096 >> >> /etc/rc.firewall: >> >> 82 pipes like theese: >> >> pipe 387 ip from any to 193.227.x.x in recv vlan10 >> pipe 388 ip from 193.227.x.x to any out xmit vlan10 >> >> >> ######################################### >> Kernel: >> >> >> cpu HAMMER >> ident ATLANTIS >> >> # To statically compile in device wiring instead of /boot/device.hints >> #hints "GENERIC.hints" # Default places to look for >> devices. >> >> makeoptions DEBUG=-g # Build kernel with gdb(1) debug >> symbols >> >> options SCHED_ULE # 4BSD scheduler >> options PREEMPTION # Enable kernel thread >> preemption >> options INET # InterNETworking >> #options SCTP # Stream Control Transmission >> Protocol >> options FFS # Berkeley Fast Filesystem >> options >> SOFTUPDATES # Enable FFS soft updates support >> options >> UFS_ACL # Support for access control lists >> options >> UFS_DIRHASH # Improve performance on big directories >> options PROCFS # Process filesystem (requires >> PSEUDOFS) >> options PSEUDOFS # Pseudo-filesystem framework >> options GEOM_PART_GPT # GUID Partition Tables. >> options GEOM_LABEL # Provides labelization >> options COMPAT_43TTY # BSD 4.3 TTY compat [KEEP >> THIS!] >> options COMPAT_IA32 # Compatible with i386 binaries >> options COMPAT_FREEBSD4 # Compatible with FreeBSD4 >> options >> COMPAT_FREEBSD5 # Compatible with FreeBSD5 options >> COMPAT_FREEBSD6 # Compatible with FreeBSD6 options >> KTRACE >> # ktrace(1) support >> options SYSVSHM # SYSV-style shared memory >> options >> SYSVMSG # SYSV-style message queues options >> SYSVSEM # SYSV-style semaphores >> options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time >> extensions >> options KBD_INSTALL_CDEV # install a CDEV entry in /dev >> options ADAPTIVE_GIANT # Giant mutex is adaptive. >> options >> STOP_NMI # Stop CPUS using NMI instead of IPI >> options AUDIT # Security event auditing >> >> # Make an SMP-capable kernel by default >> options SMP # Symmetric MultiProcessor >> Kernel >> >> # Bus support. >> device acpi >> device pci >> >> # ATA and ATAPI devices >> device ata >> >> device atadisk # ATA disk drives >> options ATA_STATIC_ID # Static device numbering >> >> # RAID controllers >> device twe # 3ware ATA RAID >> >> # atkbdc0 controls both the keyboard and the PS/2 mouse >> device atkbdc # AT keyboard controller >> device atkbd # AT keyboard >> >> device vga # VGA video card driver >> >> device splash # Splash screen and screen saver support >> >> # syscons is the default console driver, resembling an SCO console >> device >> sc >> >> ### COM >> device sio >> >> # PCI Ethernet NICs. >> device em # Intel PRO/1000 adapter Gigabit >> Ethernet >> Card >> >> # PCI Ethernet NICs that use the common MII bus controller code. >> # NOTE: Be sure to keep the 'device miibus' line in order to use these >> NICs! device miibus # MII bus support >> device bge # Broadcom BCM570xx Gigabit Ethernet >> device fxp # Intel EtherExpress PRO/100B (82557, >> 82558) >> >> # Pseudo devices. >> device loop # Network loopback >> device random # Entropy device >> device ether # Ethernet support >> device pty # Pseudo-ttys (telnet etc) >> device vlan >> >> # The `bpf' device enables the Berkeley Packet Filter. >> # Be aware of the administrative consequences of enabling this! >> # Note that 'bpf' is required for DHCP. >> device bpf # Berkeley packet filter >> >> ## Custom options >> # NetGraph >> options NETGRAPH >> options NETGRAPH_ONE2MANY >> options NETGRAPH_NETFLOW >> options NETGRAPH_CISCO >> options NETGRAPH_ETHER >> options NETGRAPH_KSOCKET >> options NETGRAPH_SOCKET >> options NETGRAPH_TEE >> >> options IPFIREWALL >> options IPFIREWALL_VERBOSE >> options IPFIREWALL_FORWARD >> options IPFIREWALL_VERBOSE_LIMIT=1000 >> options IPFIREWALL_DEFAULT_TO_ACCEPT >> options DUMMYNET >> options HZ=1000 >> options DEVICE_POLLING >> ##################################################### >> >> Interfaces: >> - em0 >> - em1 >> - bge0 >> - bge1 >> - vlan (61 virtual interfaces) >> >> ##################################################### >> top -S >> >> last pid: 9673; load averages: 1.94, 1.75, 1.57 >> up 0+19:17:21 >> 19:45:01 >> 77 processes: 11 running, 49 sleeping, 17 waiting >> CPU states: 0.0% user, 0.0% nice, 22.6% system, 0.3% interrupt, 77.0% >> idle Mem: 198M Active, 410M Inact, 455M Wired, 228K Cache, 214M Buf, >> 2874M >> Free Swap: 4096M Total, 4096M Free >> >> PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU >> COMMAND >> 11 root 1 171 ki31 0K 16K CPU7 7 19.0H 100.00% idle: >> cpu7 >> 16 root 1 171 ki31 0K 16K CPU2 2 18.9H 100.00% idle: >> cpu2 >> 17 root 1 171 ki31 0K 16K RUN 1 18.8H 100.00% idle: >> cpu1 >> 13 root 1 171 ki31 0K 16K CPU5 5 18.8H 100.00% idle: >> cpu5 >> 18 root 1 171 ki31 0K 16K CPU0 0 916:13 100.00% idle: >> cpu0 >> 12 root 1 171 ki31 0K 16K CPU6 6 18.8H 99.85% idle: >> cpu6 >> 35 root 1 -68 - 0K 16K CPU4 4 466:17 96.00% em1 >> taskq >> 34 root 1 -68 - 0K 16K CPU3 3 482:01 90.38% em0 >> taskq >> 15 root 1 171 ki31 0K 16K RUN 3 655:20 13.38% idle: >> cpu3 >> 14 root 1 171 ki31 0K 16K RUN 4 671:52 3.08% idle: >> cpu4 >> >> >> ############################################## >> 19:45[p0]root@atlantis#~>netstat -w 1 -I em0 >> input (em0) output >> packets errs bytes packets errs bytes colls >> 57381 0 36442155 68726 0 69126050 0 >> 56817 0 37480502 67656 0 66053093 0 >> 57847 0 39532712 68603 0 67037042 0 >> 56908 0 37197022 68924 0 68660108 0 >> 57107 0 37643382 68398 0 68113937 0 >> 56847 0 35944754 68394 0 67896267 0 >> 58754 0 39763361 68966 0 70029090 0 >> 58343 0 38301796 69635 0 69948678 0 >> ^C >> 19:46[p0]root@atlantis#~>netstat -w 1 -I em1 >> input (em1) output >> packets errs bytes packets errs bytes colls >> 67944 0 68877031 55376 0 36252905 0 >> 65943 0 66722222 54575 0 37710643 0 >> 64639 0 67149621 53298 0 35423539 0 >> 63988 0 65035759 51787 0 35402337 0 >> 63849 0 65968513 50727 0 31683425 0 >> 64301 0 66684912 50193 0 30917339 0 >> >> >> >> ################################################################### >> >> >> How can we solve this problem and parallelize em1:taskq kernel processes >> between all 8 processors? > > # sysctl net.isr.direct=0 > would add one more kernel thread to handle your network traffic. > > Regards, Dmitriy. > A problem remained :( >> >> >> -- >> ISP BGNet >> 288-03-53 >> 246-68-98 >> >> Zin'kov Oleg >> System administrator >> >> >> >> >> >> >> >> >> >> >> >> >> -- >> ISP BGNet >> 288-03-53 >> 246-68-98 >> >> Zin'kov Oleg >> System administrator >> >> _______________________________________________ >> freebsd-net@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-net >> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >> > > -- ISP BGNet 288-03-53 246-68-98 Zin'kov Oleg System administrator
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?38eb68e1e4bd90145fe7e90708c8979f.squirrel>