Date: Sun, 29 Jun 2008 12:48:35 -0400 From: Paul <paul@gtcomm.net> To: Mike Tancsa <mike@sentex.net> Cc: FreeBSD Net <freebsd-net@freebsd.org> Subject: Re: Freebsd IP Forwarding performance (question, and some info) [7-stable, current, em, smp] Message-ID: <4867BCE3.2000909@gtcomm.net> In-Reply-To: <200806291539.m5TFdvCO075285@lava.sentex.ca> References: <4867420D.7090406@gtcomm.net> <200806291539.m5TFdvCO075285@lava.sentex.ca>
next in thread | previous in thread | raw e-mail | index | archive | help
[RTM_MISS information added] I have noticed something weird.. It doesn't generate the RTM_MISS with all traffic... Check this out.. flooding packets through the router 11:29:56.147177 IP (tos 0x0, ttl 255, id 51487, offset 0, flags [none], proto TCP (6), length 40) 87.42.160.8.8195 > 10.3.9.50.623: ., cksum 0x999e (incorrect (-> 0xb7e6), 2714784062:2714784062(0) ack 1638282847 win 16384 11:29:56.147182 IP (tos 0x0, ttl 255, id 22197, offset 0, flags [none], proto TCP (6), length 40) 43.253.49.61.13857 > 10.3.9.50.6232: ., cksum 0x51b4 (incorrect (-> 0xeb52), 2685378913:2685378913(0) ack 3576016642 win 16384 11:29:56.147186 IP (tos 0x0, ttl 255, id 19160, offset 0, flags [none], proto TCP (6), length 40) 148.198.237.1.31784 > 10.3.9.50.63: ., cksum 0x7d2b (incorrect (-> 0xcedf), 2790811715:2790811715(0) ack 3733955172 win 16384 11:29:56.147190 IP (tos 0x0, ttl 255, id 45483, offset 0, flags [none], proto TCP (6), length 40) 50.140.153.13.23035 > 10.3.9.50.324: ., cksum 0x5e56 (incorrect (-> 0xdb81), 2403102209:2403102209(0) ack 923149825 win 16384 11:29:56.147194 IP (tos 0x0, ttl 255, id 53653, offset 0, flags [none], proto TCP (6), length 40) 215.20.25.100.18066 > 10.3.9.50.6232: ., cksum 0x592b (incorrect (-> 0x9f26), 3678810149:3678810149(0) ack 916597768 win 16384 11:29:56.147198 IP (tos 0x0, ttl 255, id 25330, offset 0, flags [none], proto TCP (6), length 40) 30.216.125.43.23715 > 10.3.9.50.5325: ., cksum 0x7d62 (incorrect (-> 0xdbb8), 4058239037:4058239037(0) ack 1225812033 win 16384 11:29:56.147210 IP (tos 0x0, ttl 255, id 56031, offset 0, flags [none], proto TCP (6), length 40) 92.202.56.54.1180 > 10.3.9.50.623: ., cksum 0x45fb (incorrect (-> 0xc35d), 1051146817:1051146817(0) ack 1914442290 win 16384 11:29:56.147214 IP (tos 0x0, ttl 255, id 17414, offset 0, flags [none], proto TCP (6), length 40) 243.139.107.120.22763 > 10.3.9.50.63: ., cksum 0x3f12 (incorrect (-> 0x983d), 343363109:343363109(0) ack 2790458180 win 16384 11:29:56.147219 IP (tos 0x0, ttl 255, id 15785, offset 0, flags [none], proto TCP (6), length 40) 193.30.160.14.19071 > 10.3.9.50.324: ., cksum 0x0021 (incorrect (-> 0x3f33), 3909194617:3909194617(0) ack 1335272554 win 16384 11:29:56.147223 IP (tos 0x0, ttl 255, id 36379, offset 0, flags [none], proto TCP (6), length 40) 93.46.193.34.22779 > 10.3.9.50.5325: ., cksum 0x2f95 (incorrect (-> 0x2fb6), 1506935083:1506935083(0) ack 2882181640 win 16384 11:29:56.147228 IP (tos 0x0, ttl 255, id 43639, offset 0, flags [none], proto TCP (6), length 40) 50.78.186.1.1437 > 10.3.9.50.623: ., cksum 0xba44 (incorrect (-> 0xe9d9), 3585077271:3585077271(0) ack 1754157076 win 16384 11:29:56.147232 IP (tos 0x0, ttl 255, id 31282, offset 0, flags [none], proto TCP (6), length 40) 230.61.232.98.9799 > 10.3.9.50.6232: ., cksum 0xe26a (incorrect (-> 0x9caf), 2184338509:2184338509(0) ack 1881236495 win 16384 11:29:56.147242 IP (tos 0x0, ttl 255, id 18266, offset 0, flags [none], proto TCP (6), length 40) 185.133.224.35.18694 > 10.3.9.50.63: ., cksum 0x5bb9 (incorrect (-> 0x3e24), 1265308989:1265308989(0) ack 898540886 win 16384 11:29:56.147247 IP (tos 0x0, ttl 255, id 63295, offset 0, flags [none], proto TCP (6), length 40) 60.149.107.97.14907 > 10.3.9.50.324: ., cksum 0x75ac (incorrect (-> 0xd165), 680418413:680418413(0) ack 1399770970 win 16384 11:29:56.147264 IP (tos 0x0, ttl 255, id 44265, offset 0, flags [none], proto TCP (6), length 40) 226.37.11.125.16380 > 10.3.9.50.5325: ., cksum 0xb763 (incorrect (-> 0x2d10), 1640245563:1640245563(0) ack 3534655349 win 16384 11:29:56.147276 IP (tos 0x0, ttl 255, id 62142, offset 0, flags [none], proto TCP (6), length 40) 76.66.176.108.8173 > 10.3.9.50.623: ., cksum 0xf66b (incorrect (-> 0xadcf), 1200243821:1200243821(0) ack 398846984 win 16384 11:29:56.147281 IP (tos 0x0, ttl 255, id 60663, offset 0, flags [none], proto TCP (6), length 40) 241.35.125.100.4600 > 10.3.9.50.324: ., cksum 0x7299 (incorrect (-> 0x63e8), 4145462333:4145462333(0) ack 1113096518 win 16384 11:29:56.147286 IP (tos 0x0, ttl 255, id 55417, offset 0, flags [none], proto TCP (6), length 40) 152.7.87.5.14990 > 10.3.9.50.6232: ., cksum 0xd955 (incorrect (-> 0xcfc1), 892720934:892720934(0) ack 1334374150 win 16384 ^C11:29:56.147290 IP (tos 0x0, ttl 255, id 29468, offset 0, flags [none], proto TCP (6), length 40) 246.253.188.115.23425 > 10.3.9.50.63: ., cksum 0xf14e (incorrect (-> 0xcaa4), 1030196069:1030196069(0) ack 2661620567 win 16384 these generate RTM_MISS for what looks like every packet : got message of size 160 on Sun Jun 29 11:31:04 2008 RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno 0, flags:<DONE> locks: inits: sockaddrs: <DST> default got message of size 160 on Sun Jun 29 11:31:04 2008 RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno 0, flags:<DONE> locks: inits: sockaddrs: <DST> default got message of size 160 on Sun Jun 29 11:31:04 2008 RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno 0, flags:<DONE> locks: inits: sockaddrs: <DST> default got message of size 160 on Sun Jun 29 11:31:04 2008 RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno 0, flags:<DONE> locks: inits: sockaddrs: <DST> default got message of size 160 on Sun Jun 29 11:31:04 2008 RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno 0, flags:<DONE> locks: inits: sockaddrs: <DST> default got message of size 160 on Sun Jun 29 11:31:04 2008 RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno 0, flags:<DONE> locks: inits: sockaddrs: <DST> default got message of size 160 on Sun Jun 29 11:31:04 2008 RTM_MISS: Lookup failed on this address: len 160, pid: 0, seq 0, errno 0, flags:<DONE> locks: inits: sockaddrs: <DST> default And then these packets: 11:32:50.540714 IP (tos 0x0, ttl 128, id 60717, offset 0, flags [DF], proto TCP (6), length 48) 199.253.132.217.1145 > 10.3.9.50.1230: S, cksum 0xe22a (correct), 2254729531:2254729531(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540719 IP (tos 0x0, ttl 128, id 29798, offset 0, flags [DF], proto TCP (6), length 48) 128.201.66.139.1074 > 10.3.9.50.1076: S, cksum 0xbc56 (correct), 699104812:699104812(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540737 IP (tos 0x0, ttl 128, id 24548, offset 0, flags [DF], proto TCP (6), length 48) 64.6.56.150.1137 > 10.3.9.50.1065: S, cksum 0x3d9f (correct), 3231494262:3231494262(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540742 IP (tos 0x0, ttl 128, id 24872, offset 0, flags [DF], proto TCP (6), length 48) 209.236.188.202.1119 > 10.3.9.50.1137: S, cksum 0xb4bb (correct), 3403159757:3403159757(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540746 IP (tos 0x0, ttl 128, id 7860, offset 0, flags [DF], proto TCP (6), length 48) 193.42.23.202.1147 > 10.3.9.50.1047: S, cksum 0x7900 (correct), 2067225130:2067225130(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540751 IP (tos 0x0, ttl 128, id 6369, offset 0, flags [DF], proto TCP (6), length 48) 198.222.98.165.1070 > 10.3.9.50.1197: S, cksum 0x8245 (correct), 1566580196:1566580196(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540759 IP (tos 0x0, ttl 128, id 29494, offset 0, flags [DF], proto TCP (6), length 48) 194.32.233.216.1217 > 10.3.9.50.1035: S, cksum 0x0036 (correct), 608655014:608655014(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540772 IP (tos 0x0, ttl 128, id 57975, offset 0, flags [DF], proto TCP (6), length 48) 205.206.20.208.1220 > 10.3.9.50.1232: S, cksum 0x52fc (correct), 1339728095:1339728095(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540777 IP (tos 0x0, ttl 128, id 5551, offset 0, flags [DF], proto TCP (6), length 48) 4.42.66.89.1142 > 10.3.9.50.1178: S, cksum 0xaa57 (correct), 1309337587:1309337587(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540781 IP (tos 0x0, ttl 128, id 64726, offset 0, flags [DF], proto TCP (6), length 48) 208.166.197.140.1052 > 10.3.9.50.1084: S, cksum 0x4dfd (correct), 3514659298:3514659298(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540786 IP (tos 0x0, ttl 128, id 51126, offset 0, flags [DF], proto TCP (6), length 48) 206.67.34.143.1071 > 10.3.9.50.1100: S, cksum 0xbf84 (correct), 3369643581:3369643581(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540790 IP (tos 0x0, ttl 128, id 59088, offset 0, flags [DF], proto TCP (6), length 48) 210.246.25.177.1277 > 10.3.9.50.1205: S, cksum 0x7080 (correct), 1667720615:1667720615(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540795 IP (tos 0x0, ttl 128, id 2326, offset 0, flags [DF], proto TCP (6), length 48) 129.251.33.231.1154 > 10.3.9.50.1195: S, cksum 0x28e5 (correct), 312297303:312297303(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540799 IP (tos 0x0, ttl 128, id 8268, offset 0, flags [DF], proto TCP (6), length 48) 216.84.213.152.1115 > 10.3.9.50.1027: S, cksum 0x7a81 (correct), 2232908292:2232908292(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540803 IP (tos 0x0, ttl 128, id 15857, offset 0, flags [DF], proto TCP (6), length 48) 199.228.246.182.1053 > 10.3.9.50.1053: S, cksum 0xe187 (correct), 376795414:376795414(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540808 IP (tos 0x0, ttl 128, id 33924, offset 0, flags [DF], proto TCP (6), length 48) 128.92.244.80.1060 > 10.3.9.50.1148: S, cksum 0xc4e1 (correct), 2038527032:2038527032(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540812 IP (tos 0x0, ttl 128, id 12114, offset 0, flags [DF], proto TCP (6), length 48) 64.118.145.169.1252 > 10.3.9.50.1148: S, cksum 0xb270 (correct), 3489780214:3489780214(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540816 IP (tos 0x0, ttl 128, id 23385, offset 0, flags [DF], proto TCP (6), length 48) 211.212.238.186.1217 > 10.3.9.50.1083: S, cksum 0xc082 (correct), 2887120836:2887120836(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540821 IP (tos 0x0, ttl 128, id 10979, offset 0, flags [DF], proto TCP (6), length 48) 24.119.38.75.1088 > 10.3.9.50.1027: S, cksum 0xee10 (correct), 3079816000:3079816000(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540830 IP (tos 0x0, ttl 128, id 44562, offset 0, flags [DF], proto TCP (6), length 48) 193.99.52.25.1249 > 10.3.9.50.1047: S, cksum 0x9ef4 (correct), 1263552303:1263552303(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540835 IP (tos 0x0, ttl 128, id 45332, offset 0, flags [DF], proto TCP (6), length 48) 209.168.140.136.1275 > 10.3.9.50.1053: S, cksum 0x03b5 (correct), 1521904180:1521904180(0) win 16384 <mss 1460,nop,[bad opt]> 11:32:50.540840 IP (tos 0x0, ttl 128, id 10244, offset 0, flags [DF], proto TCP (6), length 48) 198.114.145.88.1214 > 10.3.9.50.1083: S, cksum 0x51a9 (correct), 2907689003:2907689003(0) win 16384 <mss 1460,nop,[bad opt]> I get no RTM MISS. So this absoutely makes no sense?? It should not depend on the TYPE of packet at ALL. It's just forwarding IP regardless of what's in the TCP header so why on earth would the tcp header many any difference to how it routes.. It should not even be looking at the tcp header unless I have a firewall turned on or any type of rules that would filter based on tcp options. Forwarding a packet doesn't require looking at TCP option bits or even TCP port numbers. I suppose it could be checking the mss or trying to calculate checksums but I would prefer it didn't even do that :/ Raw performance is all I care about at this point, I don't even need to use any firewall at all. It's also funny to notice, that tcpdump does not see those packets as TCP.. tcpdump -n -i em0 tcp does not show either of my 'floods' tcpdump -n -i em1 tcp shows my floods tcpdump -n -i em0 shows them all.. and they say TCP ..lol :) Weird I tell ya! So the bottom line is, why do one type of packet generate RTM_MISS and another doesn't? I haven't confirmed this, but I don't think it does it on 7.0-RELEASE, only 7.0-STABLE and CURRENT. I will try and confirm though. Paul Mike Tancsa wrote: > At 04:04 AM 6/29/2008, Paul wrote: >> This is just a question but who can get more than 400k pps forwarding >> performance ? >> I have tested fbsd 6/7/8 so far with many different configs. (all >> using intel pci-ex nic and SMP) >> fbsd 7-stable/8(current) seem to be the fastest and always hit this >> ceiling of 400k pps. Soon as it hits that I get errors galore. > > In your testing of current, or even RELENG_7, did you ever solve the > problem of > > http://www.freebsd.org/cgi/query-pr.cgi?pr=123621&cat=kern > that you also ran into in a previous thread ? I wonder if the > generation of all those seemingly bogus RTM_MISS messages is hampering > performance. > > ---Mike > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4867BCE3.2000909>