Date: Mon, 3 Mar 2014 19:29:47 -0500 From: Chad Gross <avatar4d@gmail.com> To: Muhammad Moinur Rahman <5u623l20@gmail.com> Cc: Alex Samorukov <samm@os2.kiev.ua>, lx@freebsd.org, FreeBSD Ports <freebsd-ports@freebsd.org> Subject: Re: [patch] net-mgmt/flowviewer and security/silktools patches Message-ID: <CAHP1p-VD_RZpWd31424%2BRrRXWbY6QVOqVVJbBTK4=vXbDB2=ag@mail.gmail.com> In-Reply-To: <CA%2BnPUkyDyph9HSV3M1XBgFvT6M3XY4nhxDwZPeE-uM64MWwAqw@mail.gmail.com> References: <CAHP1p-Xq_Kct7=U3nXsPO_ariQZ7x=vc3ybXj7ekMjmG_iR4uA@mail.gmail.com> <CAHP1p-UDykoxtVpuTq6gMPw3AGNe0kgsgod9wqWee4zEE29pKA@mail.gmail.com> <CA%2BnPUkyDyph9HSV3M1XBgFvT6M3XY4nhxDwZPeE-uM64MWwAqw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--001a11c22c70970b9204f3bd0175 Content-Type: text/plain; charset=ISO-8859-1 On Sun, Mar 2, 2014 at 11:42 PM, Muhammad Moinur Rahman <5u623l20@gmail.com>wrote: > Hi, > > Can you please send me the patches as attachment rather than inline. I > will try to rebuild it from scratch and check it out again with Silktools. > > Regards, > Muhammad > > > On Wed, Feb 19, 2014 at 12:57 AM, Chad Gross <avatar4d@gmail.com> wrote: > >> On Tue, Feb 18, 2014 at 10:33 AM, Chad Gross <avatar4d@gmail.com> wrote: >> >> > I managed to configure net-mgmt/flowviewer with security/silktools, but >> > had to make some modifications to get it working. FlowViewer is >> configured >> > by defaut to pass the $silk_data_dir + $device_name as the root data >> > directory to the rwfilter tool, when the root directory should be the >> same >> > as $silk_data_dir. I've confirmed it is still the configured this way in >> > the latest version (4.3, released 2/11/14) so I could be misconfiguring >> > something, but I don't see how since I following the documentation ( >> > >> http://sourceforge.net/projects/flowviewer/files/FlowViewer.pdf/download >> ). >> > I also manually ran the commands out of working/DEBUG_VIEWER and it >> > produced nothing until I updated --data-rootdir=/data/flows/S0 to >> > --data-rootdir=/data/flows. >> > >> > Here are patches for the 4 affected files: >> > >> > >> > --- FlowGrapher_Main.cgi.orig 2014-02-18 08:49:42.000000000 -0500 >> > >> > +++ FlowGrapher_Main.cgi 2014-02-18 09:09:58.000000000 -0500 >> > >> > @@ -535,7 +535,7 @@ >> > >> > $silk_flow_type =~ s/\s+//g; >> > >> > } >> > >> > >> > >> > - $data_root_dir = $silk_data_directory ."/". $device_name; >> > >> > + $data_root_dir = $silk_data_directory; >> > >> > >> > >> > # Prepare rwfilter start and end time parameters, filter >> criteria >> > and window type >> > >> > >> > --- FlowTracker_Recreate.orig 2014-02-16 15:50:35.000000000 -0500 >> > >> > +++ FlowTracker_Recreate 2014-02-18 09:09:58.000000000 -0500 >> > >> > @@ -245,7 +245,7 @@ >> > >> > $cat_start = >> > epoch_to_date($cat_start_epoch,"LOCAL"); >> > >> > $cat_end = >> epoch_to_date($cat_end_epoch,"LOCAL"); >> > >> > >> > >> > - $data_root_dir = $silk_data_directory ."/". >> > $device_name; >> > >> > + $data_root_dir = $silk_data_directory; >> > >> > >> > >> > $silk_flow_type = ""; >> > >> > >> > >> > --- FlowTracker_Collector.orig 2014-02-18 08:48:54.000000000 -0500 >> > >> > +++ FlowTracker_Collector 2014-02-18 09:09:58.000000000 -0500 >> > >> > @@ -303,7 +303,7 @@ >> > >> > >> > >> > # Set up silk data sources >> > >> > >> > >> > - $data_root_dir = $silk_data_directory ."/". >> > $device_name; >> > >> > + $data_root_dir = $silk_data_directory; >> > >> > >> > >> > $silk_flow_type = ""; >> > >> > >> > >> > --- FlowViewer_Main.cgi.orig 2014-02-18 08:52:30.000000000 -0500 >> > >> > +++ FlowViewer_Main.cgi 2014-02-18 09:09:58.000000000 -0500 >> > >> > @@ -431,7 +431,7 @@ >> > >> > $silk_flow_type =~ s/\s+//g; >> > >> > } >> > >> > >> > >> > - $data_root_dir = $silk_data_directory ."/". $device_name; >> > >> > + $data_root_dir = $silk_data_directory; >> > >> > >> > >> > # Prepare rwfilter start and end time parameters >> > >> > >> > >> > >> > I also found that security/silktools uses UTC by default, but has a >> > configuration option to enable localtime ( >> > https://tools.netsa.cert.org/silk/faq.html#timestamp-mismatch). >> > >> > Here is a patch to the Makefile containing a config option for >> localtime: >> > >> > >> > --- /usr/ports/silktools/Makefile.orig 2014-02-18 09:29:28.000000000 >> -0500 >> > >> > +++ /usr/ports/silktools/Makefile 2014-02-18 09:41:48.000000000 >> -0500 >> > >> > @@ -23,6 +23,11 @@ >> > >> > USES= perl5 >> > >> > USE_PERL5= build >> > >> > >> > +HAS_CONFIGURE= yes >> > >> > +OPTIONS_DEFINE= LOCALTIME >> > >> > +LOCALTIME_DESC= Use localtime instead of UTC >> > >> > + >> > >> > + >> > >> > MAN1= mapsid.1 num2dot.1 rwaddrcount.1 rwappend.1 \ >> > >> > rwbag.1 rwbagbuild.1 rwbagcat.1 rwbagtool.1 \ >> > >> > rwcat.1 rwcount.1 rwcut.1 rwdedupe.1 rwfglob.1 \ >> > >> > @@ -51,6 +56,13 @@ >> > >> > rwsender.8 >> > >> > >> > NO_STAGE= yes >> > >> > + >> > >> > +.include <bsd.port.options.mk> >> > >> > + >> > >> > +.if ${PORT_OPTIONS:MLOCALTIME} >> > >> > +CONFIGURE_ARGS+=--enable-localtime >> > >> > +.endif >> > >> > + >> > >> > post-patch: >> > >> > @${REINPLACE_CMD} -e 's|echo aout|echo elf|' ${WRKSRC}/configure >> > >> > >> > >> > Thanks, >> > >> > >> > Chad >> > >> >> >> >> Here is another patch for net-mgmt/flowview so sensor filtering works. I >> am >> not sure why, but this file is originally trying to use the exporter as >> the >> sensor for SiLK devices. This is interesting since the PDF above indicated >> that the @exporter array was only used for flow-tools, not SiLK but alas >> here it is using it. If anything I think it would make more sense to use >> the "device" as the sensor, especially since @ipfix_devices is already >> defined as a sensor per the documentation. To make matters worse it is >> grepping for the probes and not the sensors in order to populate the >> --sensors= flag. >> >> >> >> --- FlowViewer_Utilities.pm.orig 2014-02-18 12:52:42.000000000 -0500 >> >> +++ FlowViewer_Utilities.pm 2014-02-18 13:50:09.000000000 -0500 >> >> @@ -2339,50 +2339,50 @@ >> >> >> >> # Set up exporter address filtering, if any >> >> >> >> - if ($exporter ne "") { >> >> + if ($device_name ne "") { >> >> >> >> - $exporter =~ s/\s+//g; >> >> - $num_include_probe = 0; >> >> - @valid_probes = (); >> >> + $device_name =~ s/\s+//g; >> >> + $num_include_sensor = 0; >> >> + @valid_sensors = (); >> >> >> >> - # Get valid probes (exporters) from the sensor.conf file >> >> + # Get valid sensors (device_names) from the sensor.conf file >> >> >> >> - $probe_command = "cat $sensor_config_directory/sensor.conf | grep probe >> > >> $work_directory/valid_probes_$suffix"; >> >> - system ($probe_command); >> >> + $sensor_command = "cat $sensor_config_directory/sensor.conf | grep >> sensor >> > $work_directory/valid_sensors_$suffix"; >> >> + system ($sensor_command); >> >> >> >> - open (PROBES,"<$work_directory/valid_probes_$suffix"); >> >> + open (PROBES,"<$work_directory/valid_sensors_$suffix"); >> >> while (<PROBES>) { >> >> - ($probe_label,$probe) = split(/\s+/,$_); >> >> - if ($probe_label eq "probe") { push (@valid_probes,$probe); } >> >> + ($sensor_label,$sensor) = split(/\s+/,$_); >> >> + if ($sensor_label eq "sensor") { push (@valid_sensors,$sensor); } >> >> } >> >> >> >> while ($still_more) { >> >> >> >> - ($exporter_name) = split(/,/,$exporter); >> >> - $start_char = length($exporter_name) + 1; >> >> - $exporter = substr($exporter,$start_char); >> >> + ($device_name_name) = split(/,/,$device_name); >> >> + $start_char = length($device_name_name) + 1; >> >> + $device_name = substr($device_name,$start_char); >> >> >> >> - if (substr($exporter_name,0,1) eq "-") { >> >> - &print_error("SiLK software does not support exclusion of Exporters >> (Sensors) at this time: -$exporter_name"); last; >> >> + if (substr($device_name_name,0,1) eq "-") { >> >> + &print_error("SiLK software does not support exclusion of Exporters >> (Sensors) at this time: -$device_name_name"); last; >> >> } else { >> >> - foreach $probe (@valid_probes) { >> >> - if ($exporter_name eq $probe) { >> >> - $num_include_probe++; >> >> - if ($num_include_probe < 2) { >> >> - $sensor_field .= $exporter_name; >> >> + foreach $sensor (@valid_sensors) { >> >> + if ($device_name_name eq $sensor) { >> >> + $num_include_sensor++; >> >> + if ($num_include_sensor < 2) { >> >> + $sensor_field .= $device_name_name; >> >> } else { >> >> - $sensor_field .= "," . $exporter_name; >> >> + $sensor_field .= "," . $device_name_name; >> >> } >> >> } >> >> } >> >> } >> >> >> >> - if ($exporter eq "") { last; } >> >> + if ($device_name eq "") { last; } >> >> } >> >> >> >> $sensor_field = " --sensors=" . $sensor_field; >> >> >> >> - $save_file .= "_" . $exporter_name; >> >> + $save_file .= "_" . $device_name; >> >> } >> >> >> >> # Set up Next Hop IP filtering, if any >> _______________________________________________ >> freebsd-ports@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-ports >> To unsubscribe, send any mail to "freebsd-ports-unsubscribe@freebsd.org" >> > > Here are the patches from above as attachments. I also attached a patch for tools/flowtracker_restart (inline at the end as well), which should go in /usr/local/etc/rc.d/, but it is just hacked together at this point to get it to start the collector/grapher tools. It doesn't look like the tools are working yet though since I see some errors in the FlowTracker_Collector.log (see below) and the graphs have a bunch of nans. The first line is most likely because it is missing the --data-rootdir= flag the second time rwfilter is called in the FlowTracker_Collector script, but I am not sure about why the other two errors are occurring. rwfilter: Site configuration file not found rwcount: Error processing headers on file '-': Unexpected end of file while reading header ERROR: /usr/local/www/flowviewer/rrdtools/outweb.rrd: illegal attempt to update using time 1393890300 when last update time is 1393890300 (minimum one second step) PATCH: --- tools/flowtracker_restart.orig 2014-03-02 20:50:32.000000000 -0500 +++ tools/flowtracker_restart 2014-03-02 23:59:39.000000000 -0500 @@ -11,9 +11,9 @@ # can be restarted using the following command: # sudo /etc/init.d/flowtracker_restart restart -FlowViewer=/var/www/cgi-bin/FlowViewer_4.2.1 -user=flowviewer -RRDTOOL_PATH=/usr/bin +FlowViewer=/usr/local/www/flowviewer +user=www +RRDTOOL_PATH=/usr/local/bin RETVAL=0 start() { @@ -21,9 +21,8 @@ for i in FlowTracker_Collector FlowTracker_Grapher; do echo Starting $i: - if ! /bin/su --shell=/bin/sh $user \ - -c "cd $FlowViewer && - nice -20 env PATH=$PATH:$RRDTOOL_PATH ./$i&"; then + if ! /usr/bin/su -m $user \ + -c "cd $FlowViewer && ./$i&"; then echo Problem starting $i >&2 RETVAL=1 fi --001a11c22c70970b9204f3bd0175 Content-Type: application/octet-stream; name="patch-FlowGrapher_Main.cgi" Content-Disposition: attachment; filename="patch-FlowGrapher_Main.cgi" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hscfd1610 LS0tIEZsb3dHcmFwaGVyX01haW4uY2dpLm9yaWcJMjAxNC0wMi0xOCAwODo0OTo0Mi4wMDAwMDAw MDAgLTA1MDAKKysrIEZsb3dHcmFwaGVyX01haW4uY2dpCTIwMTQtMDItMTggMDk6MDk6NTguMDAw MDAwMDAwIC0wNTAwCkBAIC01MzUsNyArNTM1LDcgQEAKIAkJJHNpbGtfZmxvd190eXBlID1+IHMv XHMrLy9nOwogCX0KIAotCSRkYXRhX3Jvb3RfZGlyID0gJHNpbGtfZGF0YV9kaXJlY3RvcnkgLiIv Ii4gJGRldmljZV9uYW1lOworCSRkYXRhX3Jvb3RfZGlyID0gJHNpbGtfZGF0YV9kaXJlY3Rvcnk7 CiAKIAkjIFByZXBhcmUgcndmaWx0ZXIgc3RhcnQgYW5kIGVuZCB0aW1lIHBhcmFtZXRlcnMsIGZp bHRlciBjcml0ZXJpYSBhbmQgd2luZG93IHR5cGUKIAo= --001a11c22c70970b9204f3bd0175 Content-Type: application/octet-stream; name=patch-FlowTracker_Collector Content-Disposition: attachment; filename=patch-FlowTracker_Collector Content-Transfer-Encoding: base64 X-Attachment-Id: f_hscfd16z1 LS0tIEZsb3dUcmFja2VyX0NvbGxlY3Rvci5vcmlnCTIwMTQtMDItMTggMDg6NDg6NTQuMDAwMDAw MDAwIC0wNTAwCisrKyBGbG93VHJhY2tlcl9Db2xsZWN0b3IJMjAxNC0wMi0xOCAwOTowOTo1OC4w MDAwMDAwMDAgLTA1MDAKQEAgLTMwMyw3ICszMDMsNyBAQAogCiAJCQkjIFNldCB1cCBzaWxrIGRh dGEgc291cmNlcwogCi0JCSAgICAgICAgJGRhdGFfcm9vdF9kaXIgPSAkc2lsa19kYXRhX2RpcmVj dG9yeSAuIi8iLiAkZGV2aWNlX25hbWU7CisJCSAgICAgICAgJGRhdGFfcm9vdF9kaXIgPSAkc2ls a19kYXRhX2RpcmVjdG9yeTsKIAkJCiAJCQkkc2lsa19mbG93X3R5cGUgPSAiIjsKIAo= --001a11c22c70970b9204f3bd0175 Content-Type: application/octet-stream; name=patch-FlowTracker_Recreate Content-Disposition: attachment; filename=patch-FlowTracker_Recreate Content-Transfer-Encoding: base64 X-Attachment-Id: f_hscfd1782 LS0tIEZsb3dUcmFja2VyX1JlY3JlYXRlLm9yaWcJMjAxNC0wMi0xNiAxNTo1MDozNS4wMDAwMDAw MDAgLTA1MDAKKysrIEZsb3dUcmFja2VyX1JlY3JlYXRlCTIwMTQtMDItMTggMDk6MDk6NTguMDAw MDAwMDAwIC0wNTAwCkBAIC0yNDUsNyArMjQ1LDcgQEAKIAkJCSRjYXRfc3RhcnQgPSBlcG9jaF90 b19kYXRlKCRjYXRfc3RhcnRfZXBvY2gsIkxPQ0FMIik7CiAJCQkkY2F0X2VuZCAgID0gZXBvY2hf dG9fZGF0ZSgkY2F0X2VuZF9lcG9jaCwiTE9DQUwiKTsKIAotCQkgICAgICAgICRkYXRhX3Jvb3Rf ZGlyID0gJHNpbGtfZGF0YV9kaXJlY3RvcnkgLiIvIi4gJGRldmljZV9uYW1lOworCQkgICAgICAg ICRkYXRhX3Jvb3RfZGlyID0gJHNpbGtfZGF0YV9kaXJlY3Rvcnk7CiAJCQogCQkgICAgICAgICRz aWxrX2Zsb3dfdHlwZSA9ICIiOwogCg== --001a11c22c70970b9204f3bd0175 Content-Type: application/octet-stream; name="patch-FlowViewer_Main.cgi" Content-Disposition: attachment; filename="patch-FlowViewer_Main.cgi" Content-Transfer-Encoding: base64 X-Attachment-Id: f_hscfd17e3 LS0tIEZsb3dWaWV3ZXJfTWFpbi5jZ2kub3JpZwkyMDE0LTAyLTE4IDA4OjUyOjMwLjAwMDAwMDAw MCAtMDUwMAorKysgRmxvd1ZpZXdlcl9NYWluLmNnaQkyMDE0LTAyLTE4IDA5OjA5OjU4LjAwMDAw MDAwMCAtMDUwMApAQCAtNDMxLDcgKzQzMSw3IEBACiAgICAgICAgICAgICAgICAgJHNpbGtfZmxv d190eXBlID1+IHMvXHMrLy9nOwogICAgICAgICB9CiAKLSAgICAgICAgJGRhdGFfcm9vdF9kaXIg PSAkc2lsa19kYXRhX2RpcmVjdG9yeSAuIi8iLiAkZGV2aWNlX25hbWU7CisgICAgICAgICRkYXRh X3Jvb3RfZGlyID0gJHNpbGtfZGF0YV9kaXJlY3Rvcnk7CiAKICAgICAgICAgIyBQcmVwYXJlIHJ3 ZmlsdGVyIHN0YXJ0IGFuZCBlbmQgdGltZSBwYXJhbWV0ZXJzCiAK --001a11c22c70970b9204f3bd0175 Content-Type: application/octet-stream; name=patch-silktools-Makefile Content-Disposition: attachment; filename=patch-silktools-Makefile Content-Transfer-Encoding: base64 X-Attachment-Id: f_hscfdokq5 LS0tIE1ha2VmaWxlLm9yaWcJMjAxNC0wMi0xOCAwOToyOToyOC4wMDAwMDAwMDAgLTA1MDAKKysr IE1ha2VmaWxlCTIwMTQtMDItMTggMDk6NDE6NDguMDAwMDAwMDAwIC0wNTAwCkBAIC0yMyw2ICsy MywxMSBAQAogVVNFUz0JCXBlcmw1CiBVU0VfUEVSTDU9CWJ1aWxkCiAKK0hBU19DT05GSUdVUkU9 CXllcworT1BUSU9OU19ERUZJTkU9IExPQ0FMVElNRQorTE9DQUxUSU1FX0RFU0M9IFVzZSBsb2Nh bHRpbWUgaW5zdGVhZCBvZiBVVEMKKworCiBNQU4xPQltYXBzaWQuMSBudW0yZG90LjEgcndhZGRy Y291bnQuMSByd2FwcGVuZC4xIFwKIAlyd2JhZy4xIHJ3YmFnYnVpbGQuMSByd2JhZ2NhdC4xIHJ3 YmFndG9vbC4xIFwKIAlyd2NhdC4xIHJ3Y291bnQuMSByd2N1dC4xIHJ3ZGVkdXBlLjEgcndmZ2xv Yi4xIFwKQEAgLTUxLDYgKzU2LDEzIEBACiAJCXJ3c2VuZGVyLjgKIAogTk9fU1RBR0U9CXllcwor CisuaW5jbHVkZSA8YnNkLnBvcnQub3B0aW9ucy5taz4KKworLmlmICR7UE9SVF9PUFRJT05TOk1M T0NBTFRJTUV9CitDT05GSUdVUkVfQVJHUys9LS1lbmFibGUtbG9jYWx0aW1lCisuZW5kaWYKKwog cG9zdC1wYXRjaDoKIAlAJHtSRUlOUExBQ0VfQ01EfSAtZSAnc3xlY2hvIGFvdXR8ZWNobyBlbGZ8 JyAke1dSS1NSQ30vY29uZmlndXJlCiAK --001a11c22c70970b9204f3bd0175 Content-Type: application/octet-stream; name=patch-flowtracker_restart Content-Disposition: attachment; filename=patch-flowtracker_restart Content-Transfer-Encoding: base64 X-Attachment-Id: f_hscfsze16 LS0tIHRvb2xzL2Zsb3d0cmFja2VyX3Jlc3RhcnQub3JpZwkyMDE0LTAzLTAyIDIwOjUwOjMyLjAw MDAwMDAwMCAtMDUwMAorKysgdG9vbHMvZmxvd3RyYWNrZXJfcmVzdGFydAkyMDE0LTAzLTAyIDIz OjU5OjM5LjAwMDAwMDAwMCAtMDUwMApAQCAtMTEsOSArMTEsOSBAQAogIyBjYW4gYmUgcmVzdGFy dGVkIHVzaW5nIHRoZSBmb2xsb3dpbmcgY29tbWFuZDoKICMgc3VkbyAvZXRjL2luaXQuZC9mbG93 dHJhY2tlcl9yZXN0YXJ0IHJlc3RhcnQKIAotRmxvd1ZpZXdlcj0vdmFyL3d3dy9jZ2ktYmluL0Zs b3dWaWV3ZXJfNC4yLjEKLXVzZXI9Zmxvd3ZpZXdlcgotUlJEVE9PTF9QQVRIPS91c3IvYmluCitG bG93Vmlld2VyPS91c3IvbG9jYWwvd3d3L2Zsb3d2aWV3ZXIKK3VzZXI9d3d3CitSUkRUT09MX1BB VEg9L3Vzci9sb2NhbC9iaW4KIFJFVFZBTD0wCiAKIHN0YXJ0KCkgewpAQCAtMjEsOSArMjEsOCBA QAogCiAgICAgZm9yIGkgaW4gRmxvd1RyYWNrZXJfQ29sbGVjdG9yIEZsb3dUcmFja2VyX0dyYXBo ZXI7IGRvCiAgICAgICAgIGVjaG8gU3RhcnRpbmcgJGk6Ci0gICAgICAgIGlmICEgL2Jpbi9zdSAt LXNoZWxsPS9iaW4vc2ggJHVzZXIgXAotICAgICAgICAgICAgICAgIC1jICJjZCAkRmxvd1ZpZXdl ciAmJgotICAgICAgICAgICAgICAgIG5pY2UgLTIwIGVudiBQQVRIPSRQQVRIOiRSUkRUT09MX1BB VEggLi8kaSYiOyB0aGVuCisgICAgICAgIGlmICEgL3Vzci9iaW4vc3UgLW0gJHVzZXIgXAorICAg ICAgICAgICAgICAgIC1jICJjZCAkRmxvd1ZpZXdlciAmJiAuLyRpJiI7IHRoZW4KICAgICAgICAg ICAgICAgICAgICAgICAgIGVjaG8gUHJvYmxlbSBzdGFydGluZyAkaSA+JjIKICAgICAgICAgICAg ICAgICAgICAgICAgIFJFVFZBTD0xCiAgICAgICAgIGZpCg== --001a11c22c70970b9204f3bd0175--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAHP1p-VD_RZpWd31424%2BRrRXWbY6QVOqVVJbBTK4=vXbDB2=ag>