Date: Wed, 02 Jul 2008 14:24:53 -0400 From: Paul <paul@gtcomm.net> To: Ingo Flaschberger <if@xip.at> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] Message-ID: <486BC7F5.5070604@gtcomm.net> In-Reply-To: <alpine.LFD.1.10.0807021155280.557@filebunker.xip.at> References: <4867420D.7090406@gtcomm.net> <200806301944.m5UJifJD081781@lava.sentex.ca> <20080701004346.GA3898@stlux503.dsto.defence.gov.au> <alpine.LFD.1.10.0807010257570.19444@filebunker.xip.at> <20080701010716.GF3898@stlux503.dsto.defence.gov.au> <alpine.LFD.1.10.0807010308320.19444@filebunker.xip.at> <486986D9.3000607@monkeybrains.net> <48699960.9070100@gtcomm.net> <ea7b9c170806302005n2a66f592h2127f87a0ba2c6d2@mail.gmail.com> <20080701033117.GH83626@cdnetworks.co.kr> <ea7b9c170806302050p2a3a5480t29923a4ac2d7c852@mail.gmail.com> <4869ACFC.5020205@gtcomm.net> <4869B025.9080006@gtcomm.net> <486A7E45.3030902@gtcomm.net> <486A8F24.5010000@gtcomm.net> <486A9A0E.6060308@elischer.org> <486B41D5.3060609@gtcomm.net> <alpine.LFD.1.10.0807021052041.557@filebunker.xip.at> <486B4F11.6040906@gtcomm.net> <alpine.LFD.1.10.0807021155280.557@filebunker.xip.at>
next in thread | previous in thread | raw e-mail | index | archive | help
Fastforward works with lagg , lagg just has some issues that need to be
fixed, even on UP system. It has the same issue as IPFW.
kern.polling.idlepoll_sleeping: 1
kern.polling.stalled: 806
kern.polling.suspect: 97861
kern.polling.phase: 0
kern.polling.enable: 0
kern.polling.handlers: 2
kern.polling.residual_burst: 0
kern.polling.pending_polls: 0
kern.polling.lost_polls: 128535
kern.polling.short_ticks: 1455
kern.polling.reg_frac: 1200
kern.polling.user_frac: 0
kern.polling.idle_poll: 0
kern.polling.each_burst: 50
kern.polling.burst_max: 1440
kern.polling.burst: 377
It's doing 720kpps right now, and it's having a lot of errors, but the
cpu is 40% idle! I don't understand? Is it reporting the wrong cpu usage
in TOP?
Is this a bug?
input (em0) output
packets errs bytes packets errs bytes colls
722012 42861 44764748 1 0 178 0
704432 52679 43674800 2 0 580 0
693297 53536 42984418 1 0 178 0
704046 42525 43650854 2 0 220 0
714959 37876 44327462 1 0 178 0
744923 24202 46185230 1 0 178 0
726069 34699 45016282 1 0 178 0
681837 78581 42273898 1 0 178 0
663106 85699 41112576 1 0 178 0
708274 55414 43912992 1 0 178 0
659659 94430 40898862 1 0 178 0
669235 100248 41492574 1 0 178 0
676510 100102 41943624 1 0 178 0
679847 98972 42150518 1 0 178 0
677700 92586 42017416 2 0 356 0
672639 86454 41703622 1 0 178 0
675841 72821 41902146 1 0 178 0
679522 86423 42130368 1 0 178 0
660737 72883 40965698 1 0 178 0
637085 81303 39499274 1 0 178 0
655463 98183 40638710 1 0 178 0
input (em0) output
packets errs bytes packets errs bytes colls
683650 66140 42386304 1 0 178 0
654910 110089 40604424 1 0 290 0
647969 120709 40174082 1 0 178 0
666260 67037 41308124 1 0 178 0
671570 68276 41637344 1 0 178 0
691683 60819 42884350 1 0 178 0
663656 79528 41146728 2 0 244 0
703917 47860 43642870 2 0 356 0
710988 55792 44081258 2 0 220 0
697062 77661 43217848 1 0 178 0
65 processes: 2 running, 46 sleeping, 17 waiting
CPU: 0.0% user, 0.0% nice, 10.8% system, 45.7% interrupt, 43.5% idle
Mem: 7968K Active, 6028K Inact, 42M Wired, 84K Cache, 8768K Buf, 1925M Free
Swap: 8192M Total, 8192M Free
PID USERNAME PRI NICE SIZE RES STATE TIME WCPU COMMAND
10 root 171 ki31 0K 16K RUN 4:06 86.43% idle
36 root -68 - 0K 16K - 0:57 3.81% em3 taskq
13 root -44 - 0K 16K WAIT 9:22 1.42% swi1: net
1429 root 44 0 8084K 2052K RUN 0:00 0.29% top
11 root -32 - 0K 16K WAIT 0:01 0.05% swi4: clock sio
net.isr.swi_count: 39442306
net.isr.drop: 0
net.isr.queued: 8
net.isr.deferred: 0
net.isr.directed: 2189
net.isr.count: 2189
net.isr.direct: 1
net.route.netisr_maxqlen: 16384
net.inet.ip.intr_queue_maxlen: 16384
net.inet.ip.intr_queue_drops: 0
em0: Excessive collisions = 0
em0: Sequence errors = 0
em0: Defer count = 0
em0: Missed Packets = 22574958
em0: Receive No Buffers = 65713041
em0: Receive Length Errors = 0
em0: Receive errors = 0
em0: Crc errors = 0
em0: Alignment errors = 0
em0: Collision/Carrier extension errors = 0
em0: RX overruns = 52679
em0: watchdog timeouts = 0
em0: RX MSIX IRQ = 0 TX MSIX IRQ = 0 LINK MSIX IRQ = 0
em0: XON Rcvd = 0
em0: XON Xmtd = 0
em0: XOFF Rcvd = 0
em0: XOFF Xmtd = 0
em0: Good Packets Rcvd = 547984791
em0: Good Packets Xmtd = 5000
em0: TSO Contexts Xmtd = 18
em0: TSO Contexts Failed = 0
kern.hz=2000
hw.em.rxd=512
hw.em.txd=512
-----------Reboot with 4096/4096........(my guess is that it will be a
lot worse, more errors..)
........
Without polling, 4096 is horrible, about 200kpps less ... :/
Turning on polling..
polling on, 4096 is bad,
input (em0) output
packets errs bytes packets errs bytes colls
622379 307753 38587506 1 0 178 0
635689 277303 39412718 1 0 178 0
625552 291235 38784244 2 0 580 0
630143 287872 39068870 1 0 178 0
620225 292071 38453954 1 0 178 0
627499 295329 38904942 1 0 178 0
623854 288086 38678952 1 0 178 0
632433 267698 39210850 1 0 178 0
619177 279541 38388978 1 0 178 0
618049 265926 38319038 2 0 356 0
627026 263882 38875616 1 0 178 0
em0: Missed Packets = 16570461
em0: Receive No Buffers = 9220592
em0: Receive Length Errors = 0
em0: Receive errors = 0
em0: Crc errors = 0
em0: Alignment errors = 0
em0: Collision/Carrier extension errors = 0
em0: RX overruns = 40539
------Rebooting with 256/256 descriptors..........
..........
No polling:
843762 25337 52313248 1 0 178 0
763555 0 47340414 1 0 178 0
830189 0 51471722 1 0 178 0
838724 0 52000892 1 0 178 0
813594 939 50442832 1 0 178 0
807303 763 50052790 1 0 178 0
791024 0 49043492 1 0 178 0
768316 1106 47635596 1 0 178 0
Machine is maxed and is unresponsive..
Polling ON:
input (em0) output
packets errs bytes packets errs bytes colls
784138 179079 48616564 1 0 226 0
788815 129608 48906530 2 0 356 0
755555 142997 46844426 2 0 468 0
803670 144459 49827544 1 0 178 0
777649 147120 48214242 1 0 178 0
779539 146820 48331422 1 0 178 0
786201 148215 48744478 2 0 356 0
776013 101660 48112810 1 0 178 0
774239 145041 48002834 2 0 356 0
771774 102969 47850004 1 0 178 0
Machine is responsive and has 40% idle cpu.. Why ALWAYS 40% ? I'm
really mistified by this..
Every time it maxes out and gets errors, top reports:
CPU: 0.0% user, 0.0% nice, 10.1% system, 45.3% interrupt, 44.6% idle
pretty much the same line every time
256/256 blows away 4096 , probably fits the descriptors into the cache
lines on the cpu and 4096 has too many cache misses and causes worse
performance.
This is probably just some nasty programming and they could optimize it
for 4096 by taking larger chunks, we don't need the latency to be 0.08ms
for each packet, i don't care if it's 0.3ms as long as it doesn't drop any.
Setting HZ=100
and 256/256 gets a maximum higher than 2000 polling with 256/256 but the
box is
unresponsive around 850kpps
If this only worked with SMP and was optimized it could do millions of
pps :/
Ingo Flaschberger wrote:
> Dear Paul,
>
>> I still don't like the huge hit ipfw and lagg take :/
>
> I think, you can't use fastforward with with lagg.
>
>> ** I tried polling in UP mode and I got some VERY interesting results..
>> CPU is 44% idle (idle polling isn't on) but I'm getting errors!
>> It's doing 530kpps with ipfw loaded, which without polling uses 100%
>> cpu but now it says my cpu is 44% idle? that makes no sense.. If it
>> was idle why am I getting errors? I only get errors when em taskq
>> was eating 100% cpu..
>> Idle polling on/off makes no difference.
>> user_frac is set to 5 ..
>
> what are your values:
> kern.polling.reg_frac=
> kern.polling.user_frac=
> kern.polling.burst_max=
>
> I use:
> kern.polling.reg_frac=20
> kern.polling.user_frac=20
> kern.polling.burst_max=512
>
> if you need more than 1000, you need to change the code:
> src/sys/kern/kern_poll.c
> #define MAX_POLL_BURST_MAX 1000
>
>> So my maximum without polling is close to 800kpps but if I push that
>> it starts locking me from doing things, or
>
> how many kpps do you want to achieve?
>
>> HZ=2000 for this test (512/512 descriptors)
>
> you mean:
> hw.em.rxd=512
> hw.em.txd=512
> ?
>
> can you try with polling:
> hw.em.rxd=4096
> hw.em.txd=4096
>
> Kind regards,
> Ingo Flaschberger
>
> _______________________________________________
> freebsd-net@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"
>
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?486BC7F5.5070604>