Date: Mon, 08 Feb 2010 11:01:44 +0000 From: Vincent Hoffman <vince@unsane.co.uk> To: "Daniel O'Connor" <doconnor@gsoft.com.au> Cc: freebsd-chat@freebsd.org Subject: Re: Flow analysis tools Message-ID: <4B6FEF18.8050900@unsane.co.uk> In-Reply-To: <201002081237.56872.doconnor@gsoft.com.au> References: <201002081237.56872.doconnor@gsoft.com.au>
next in thread | previous in thread | raw e-mail | index | archive | help
Daniel O'Connor wrote: > Can anyone recommend some flow analysis tools? > I am mostly interested in who (from inside my network) is downloading > how much (ie who is costing us money :) > > I have enabled netflow in mpd and I can capture it but I haven't really > found a suitable analysis tool yet. > > I tried nfsen and stager but I couldn't get them to break down based on > IP, just AS - not nearly fine grained enough for my needs. > > I'd have said nfdump/nfsen if you hadnt ruled them out. I havent had any problems getting IP level reports from it. I'f you are willing to use the command line rather than the web interface its easy to cron a daily report: for example, we only have netflow setup on the router to do netflow for outgoing traffic from this location so for a report on the usage of the top 20 source IPs (destination is interesting but irrelevant for this exercise) Please note the -K option is just to anonymize the IPs for this excercise. [root@seaurchin ~/bin]# nfdump -M /usr/local/var/nfsen/profiles-data/live/rsh1 -T -K sgjkouik67juhyt689076stegncitfds -R 2010/02/07/nfcapd.201002072355:2010/02/08/nfcapd.201002080945 -n 20 -s srcip/bytes "src net 88.111.160.0/22" Top 20 Src IP Addr ordered by bytes: Date first seen Duration Proto Src IP Addr Flows(%) Packets(%) Bytes(%) pps bps bpp 2010-02-07 23:54:33.133 35692.169 any 88.111.163.199 8566( 0.1) 53.2 M(31.9) 77.1 G(42.0) 1491 17.3 M 1447 2010-02-07 23:54:46.281 35718.377 any 88.111.163.156 329868( 2.1) 12.1 M( 7.3) 14.8 G( 8.1) 339 3.3 M 1219 2010-02-07 23:54:36.561 35722.425 any 88.111.162.41 52807( 0.3) 6.7 M( 4.0) 8.3 G( 4.5) 188 1.9 M 1233 2010-02-07 23:55:00.465 35699.941 any 88.111.163.36 28073( 0.2) 5.8 M( 3.5) 8.0 G( 4.3) 162 1.8 M 1372 2010-02-07 23:55:25.553 35639.753 any 88.111.163.158 90460( 0.6) 4.3 M( 2.6) 5.6 G( 3.1) 120 1.3 M 1315 2010-02-07 23:54:44.289 35725.773 any 88.111.162.65 166080( 1.0) 3.0 M( 1.8) 3.8 G( 2.1) 85 851674 1247 2010-02-07 23:55:09.025 35688.921 any 88.111.163.53 18458( 0.1) 2.6 M( 1.6) 3.7 G( 2.0) 73 820182 1396 2010-02-07 23:54:42.717 35718.217 any 88.111.160.160 10498( 0.1) 2.4 M( 1.4) 3.5 G( 1.9) 66 784449 1468 2010-02-07 23:55:02.245 35696.169 any 88.111.162.42 355185( 2.2) 3.5 M( 2.1) 2.9 G( 1.6) 96 640147 825 2010-02-07 23:54:51.069 35715.737 any 88.111.163.150 185657( 1.2) 2.8 M( 1.7) 2.8 G( 1.5) 79 631644 990 2010-02-07 23:55:09.053 35681.053 any 88.111.162.72 9766( 0.1) 1.3 M( 0.8) 1.8 G( 1.0) 36 410339 1399 2010-02-07 23:55:01.993 35698.869 any 88.111.162.113 53386( 0.3) 1.4 M( 0.9) 1.8 G( 1.0) 40 408840 1262 2010-02-07 23:53:15.833 35807.625 any 88.111.162.91 197756( 1.2) 2.5 M( 1.5) 1.7 G( 0.9) 70 385855 688 2010-02-07 23:55:14.321 35682.961 any 88.111.160.134 8925( 0.1) 1.2 M( 0.7) 1.7 G( 0.9) 34 370990 1352 2010-02-07 23:54:18.257 35744.017 any 88.111.163.69 82420( 0.5) 1.3 M( 0.8) 1.6 G( 0.9) 35 366048 1296 2010-02-07 23:54:02.605 35755.409 any 88.111.163.112 120445( 0.8) 1.6 M( 0.9) 1.6 G( 0.9) 43 355841 1024 2010-02-07 23:55:06.065 35686.533 any 88.111.162.132 20215( 0.1) 1.1 M( 0.7) 1.4 G( 0.8) 30 322267 1312 2010-02-07 23:53:40.965 35782.949 any 88.111.163.157 136187( 0.9) 1.5 M( 0.9) 1.4 G( 0.8) 40 315334 969 2010-02-07 23:55:09.173 35688.901 any 88.111.161.247 18823( 0.1) 1.0 M( 0.6) 1.4 G( 0.8) 28 313011 1357 2010-02-07 23:54:58.133 35701.225 any 88.111.162.25 25457( 0.2) 1.1 M( 0.6) 1.4 G( 0.8) 30 310943 1284 IP addresses anonymized Summary: total flows: 15946440, total bytes: 183.7 G, total packets: 166.7 M, avg bps: 41.0 M, avg pps: 4654, avg bpp: 1101 Time window: 2010-02-07 23:53:15 - 2010-02-08 09:50:12 Total flows processed: 15946440, Blocks skipped: 0, Bytes read: 829226960 Sys: 4.534s flows/second: 3516363.2 Wall: 4.521s flows/second: 3526931.0 The same query is entirely doable in the nfsen web interface as well, just a report with "stat topN", top: 20, stat: SRC IP Addresses order by bytes, with a filter of "src net 88.111.160.0/22" We also use a customer written alert plugin to detect pps based DOS/DDOS attacks (if a single host exceeds 100K pps then tell us who it is and who is sending the packets.) pmacct isnt bad but I didnt like any of the interfaces I could find and didnt want to write my own, plus storing it all in a mysql database meant it had a much greater overhead than nfsen/dump. ntop was fine for smaller traffic but went belly up pretty quickly at higher traffic levels. didnt try stager. Flow tools looked ok if you want to roll your own, but no gain on nfdump really just without nfsen. Vince
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?4B6FEF18.8050900>