Date: Tue, 07 May 2002 22:47:30 -0600 From: RichardH <richardh@wsonline.net> To: freebsd-questions@FreeBSD.ORG Subject: Re: Parsing Log Files Message-ID: <5.1.0.14.0.20020507224720.00ad6cc8@pop.wsonline.net>
next in thread | raw e-mail | index | archive | help
Thx for the parsing suggestions, we are working on a custom script for=20 parsing access logs out as needed to a users home dir under a "log" dir for= =20 multiple users from 1 to 10000+. I don't know if this would help with=20 getting this developed (we are working on it but any help would be=20 appreciated and recognized as such). We run hashed user dir, like username= =20 is under /home/u/s/username, keeps dir structure more ?define-able?, do=20 same with zone files, try it, you'll like it. Anyway, if we get a script=20 together that can parse out the access logs without running massive=20 processes (ie. transferlog directive) to do it we will post it in here so=20 any help will be appreciated by all in the long run. Also, adding user logs= =20 to the script should be fairly painless, we are working on this part also=20 in that the whole script would not have to be opened and edited for each=20 user add, possible to write into the adduser??For more info on what we are= =20 wanting to do go to webhostingtalk.com and do a search for user storm2k and= =20 read the thread. Possibly at this link (may or may not work, if not do the= =20 search for user storm2k) http://www.webhostingtalk.com/showthread.php?s=3D0785248167d55ea6c36f39866be= 96f78&threadid=3D46871 now for a stoopid question, I have the large banner for FBSD on my site but= =20 I want a smaller button, where are those located (banners, buttons, linking= =20 stuff, etc.)? I cannot locate it for the life of me and I went through damn= =20 near the whole .org site. please send a link to that page asap, info on the= =20 other is appreciated but not expected asap :-) Thanks for input, Richard Hutson There are two major products that came out of Berkeley: LSD and BSD. We=20 don't believe this to be a coincidence. =97Jeremy S. Anderson At 07:48 PM 5/2/2002, RichardH wrote: >>Delivered-To: freebsd-questions@freebsd.org >>Date: Thu, 2 May 2002 09:24:35 -0700 >>To: questions@FreeBSD.ORG >>Subject: Re: Parsing Log Files >>X-Mailer: Sylpheed version 0.7.4 (GTK+ 1.2.10; i386-portbld-freebsd4.5) >>Sender: owner-freebsd-questions@FreeBSD.ORG >>List-ID: <freebsd-questions.FreeBSD.ORG> >>List-Archive: <http://docs.freebsd.org/mail/>; (Web Archive) >>List-Help: <mailto:majordomo@FreeBSD.ORG?subject=3Dhelp> (List= Instructions) >>List-Subscribe:=20 >><mailto:majordomo@FreeBSD.ORG?subject=3Dsubscribe%20freebsd-questions> >>List-Unsubscribe:=20 >><mailto:majordomo@FreeBSD.ORG?subject=3Dunsubscribe%20freebsd-questions> >>X-Loop: FreeBSD.ORG >> >>On Thu, 2 May 2002 11:02:03 -0400 >>Rob Ellis wrote: >> >> > On Wed, May 01, 2002 at 07:29:29PM -0600, RichardH wrote: >> > > By parsing out the files with a script, it reduces overall server >> > > load AND permits the use of rewrite rules, that allow you to use a >> > > virtmap.txt type of setup for hosting entries (in which case the >> > > transferlog entry does not work at all). >> > >> > Assuming the domain name is the first thing on each log line, >> > you could do something like >> > >> > #! /usr/bin/perl -w >> > use FileCache; # opens/closes file descriptors as required >> > no strict "refs"; # FileCache generates "strict refs" warnings >> > $log =3D "/usr/local/apache/logs/access_log"; >> > $outdir =3D "/usr/local/var/weblogs"; >> > open(LOG, $log) || die $!; >> > while (<LOG>) { >> > if (/^([\w\.-]+)\s+/) { >> > $domain =3D $1; >> > $outfile =3D "$outdir/$domain/access_log"; >> > die $! unless (cacheout $outfile); >> > print $outfile $_; >> > } >> > # do something here with junk lines >> > } >> > close(LOG); >> > 1; >> >>Here are some snips from a small script that I put together to parse the >>apache log (/var/log/httpd-access.log) to find suspect log entries >>containing lame attempts to exploit IIS vulnerabilities. If found, it >>will try to send an email to "abuse" at whatever domain the user was at. >> It doesn't write anything to an output file, but it does selectively >>choose entries from the current date only. You could possibly modify >>this to append each days activities to each users log file. Again, the >>below doesn't necessarily speak to your particular problem, but maybe >>some tidbits of this could be a start, along with the post from Rob >>Ellis. >> >>#!/usr/bin/perl -w >> >>use strict; >>use Mail::Sendmail; >> >>my ($line, $host, $rcpt, $dstamp, $body); # some scalars >>my @date; # an array >>my (%mail, %offenders); # some hashes >> >>@date =3D split(" ", `date`); # get current date into >>an array$dstamp =3D "$date[2]/$date[1]/$date[5]"; # rearrange to >>match date in apache log file >> >> >>open (FILE, "/var/log/httpd-access.log"); # open log file for >>reading >> >>while ($line =3D <FILE>) { >> # find log entries from today that also contain mischevious keywords >> if ( (grep(/.*\[$dstamp:/, $line)) && >>(grep(/scripts|winnt|cmd\.exe|root\.exe|system32/, $line)) ) { >>$line =3D~ /^(\S+).*\[(.+)\].*GET\s(\S+)/; # parse interesting line >>$1=3Dhost $2=3Ddate/time $3=3DGET command push @{$offenders{$1}},"$2 >>$3\n"; # put values into a hash for later processing } >>} >> >>foreach $host (keys(%offenders)) { >> if ($host !~ /\.\d+$/) { # only act if $host is an actual host >>name to which we can construct an email $host =3D~ /^\S+\.(.*)$/; # >>get domain portion of $host $rcpt =3D $1; # assign >>$rcpt to value of previous regex $body =3D ( # create >>the email body "Email Body" >> ); >> %mail =3D ( # create some email headers >> 'Date' =3D> Mail::Sendmail::time_to_date(), >> 'To' =3D> "abuse\@$rcpt", >> 'From' =3D> 'somebody@somewhere.org', >> 'Subject' =3D> 'Notification of malicious user or system', >> 'Body' =3D> "$body" >> ); >> sendmail(%mail); # send the mail >> } >>} >> >>close (FILE); # close the file log file >> >>To Unsubscribe: send mail to majordomo@FreeBSD.org >>with "unsubscribe freebsd-questions" in the body of the message To Unsubscribe: send mail to majordomo@FreeBSD.org with "unsubscribe freebsd-questions" in the body of the message
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5.1.0.14.0.20020507224720.00ad6cc8>