From owner-freebsd-questions@freebsd.org Tue Apr 18 17:40:15 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D162BD4476B for ; Tue, 18 Apr 2017 17:40:15 +0000 (UTC) (envelope-from andipersti@gmail.com) Received: from mail-wr0-x22e.google.com (mail-wr0-x22e.google.com [IPv6:2a00:1450:400c:c0c::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 63898165E for ; Tue, 18 Apr 2017 17:40:15 +0000 (UTC) (envelope-from andipersti@gmail.com) Received: by mail-wr0-x22e.google.com with SMTP id c55so107584904wrc.3 for ; Tue, 18 Apr 2017 10:40:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language:content-transfer-encoding; bh=cWr46x58MObZ5cRTTL2DKfU5lRzZ6Q/XxUqOTBoHlSI=; b=Cls0ytOhv09wMJ9syDmgS7R/bY9Z5hCyEuv+1gic8isDzQOFq6QQuA94nO3+y6PS9c sC30lKOVzKkZNJwEkMWhLx2Rvnqy1bM5XlgYeh2Q5M0Q0PgCmuuKA+D4yCY7KIPi1VIZ Yw1uDRaBdbLhAtX/MGjgiUu2GsQL7utSfUwoxgga6S3CJu5AQO1a6gXJ+pxP9wqb/Xpg RR2WyrZPHwFUuHMUvCbKIaiQtzbu0aNpHnGoyOdSZajw70JowvANgiQW0plT2fUmg9bH 3lsZ3jb4HAmoebKYUdf1OqCNnjWG+z/CxQMjixHjS5hV2WKsqegZPHlEwEHPVBE4psr1 pZ7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=cWr46x58MObZ5cRTTL2DKfU5lRzZ6Q/XxUqOTBoHlSI=; b=Hj0VsuFPm3ckerQmRWM3FEOigyvmJxfRGr9yEjCDtest0ugv0PUwM2L4AzUZXq03dw oJ3XlpRXrGlqZG9EX4NO3epdqozCG5gdOIlA51LWzVWK+/ekYpw/TIw1bHpE2ex7LfSW jkLtuHtkX8B3ogB4zz2+3ZNVltHgLg9u0kTNzGucFxicFx9MR8nZoQuxwC9zvmMKepXQ 8MSlmKJ6hl4KYCclLESmMbvm+TyTHEElEqhbGe2D66p4tjdoINW3C4j83bVJal5+UnoH 54j48UGcOabLJxopRYMpMQs8i3jlzk3AkNKVNCLrTZdRB57g2gYBFxkv/M1dkwQ57xpF i2ig== X-Gm-Message-State: AN3rC/4jb0rhxZHbDYWmm9RV0HUbqHhaCHmyywF/1MkFmmX5P81fAONR 9m54RFlz2e29qDDj X-Received: by 10.223.164.141 with SMTP id g13mr9653609wrb.61.1492537213380; Tue, 18 Apr 2017 10:40:13 -0700 (PDT) Received: from [192.168.178.25] (91-119-212-211.dsl.dynamic.surfer.at. [91.119.212.211]) by smtp.googlemail.com with ESMTPSA id z84sm15805272wmh.27.2017.04.18.10.40.12 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Apr 2017 10:40:12 -0700 (PDT) Subject: Re: awk help To: freebsd-questions@freebsd.org References: <58F25A01.1060208@gmail.com> <7951DF71-5CD3-4B53-9CB4-13CAA8945983@huiekin.org> <58F4CD14.7090008@gmail.com> <58F53EEA.2030206@gmail.com> <7b381f8f-e2a5-26ea-075e-96ae35efb25d@rogers.com> <58F61027.3090100@gmail.com> From: Andreas Perstinger Message-ID: Date: Tue, 18 Apr 2017 19:40:11 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.0 MIME-Version: 1.0 In-Reply-To: <58F61027.3090100@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: de_AT Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 18 Apr 2017 17:40:15 -0000 I think awk is the better tool for your task but you could still simplify your shell script a little bit: > hits_rpt="/etc/ipf_pool_hits_rpt" > rm $hits_rpt> touch $hits_rpt> > hits_new="/etc/ipf_pool.hits.yes"> rm $hits_new> touch $hits_new> > hits_no="/etc/ipf_pool.hits.no"> rm $hits_no> touch $hits_no Using the "truncate" command you could reduce these lines to hits_rpt="/etc/ipf_pool_hits_rpt" hits_new="/etc/ipf_pool.hits.yes" hits_no="/etc/ipf_pool.hits.no" truncate -s 0 $hits_rpt $hits_yes $hits_no > ippool -l -d -m probing_ips > $hits_rpt 2> /dev/null > > for line in `cat $hits_rpt`; do> > # drop the first 3 rpt lines> poollist_line=""> poollist_line=`echo -n $line | grep poollist`> [ -n "${poollist_line}" ] && continue> > role_line=""> role_line=`echo -n $line | grep Role`> [ -n "${role_line}" ] && continue> > nodes_line=""> nodes_line=`echo -n $line | grep Nodes`> [ -n "${nodes_line}" ] && continue Assuming that your file always contains the three unneeded lines at the beginning you could use the "tail" command to start at line 4. And I would also use a while-loop in combination with the "read" built-in function. Then you are able to process pairs of lines: tail -n +4 $hits_rpt | while read line do > in_line1=`echo -n $line | grep Address:` > [ -n "${in_line1}" ] && save_in_line1="${in_line1}" > > in_line2=`echo -n $line | grep Hits` > [ -n "${in_line2}" ] && save_in_line2="${in_line2}" > > if [ "${save_in_line1}" -a "${save_in_line2}" ]; then > build_line1=${save_in_line1##*:} > build_line1=${build_line1%%/*} > build_line1="${build_line1};" > build_line2=${save_in_line2##*Hits } > # So remove everything to the right of the word Bytes. > build_line2=${build_line2%%Bytes*} Using sed instead of grep would simplify the extraction of the IP address and the number of hits: ip=$(echo $line | sed -e 's/ *Address: *\([^/]*\).*/\1/') read line hits=$(echo $line | sed -e 's/ *Hits *\([[:digit:]*]\).*/\1/') > if [ ${build_line2} -gt 0 ]; then > db_rec="$added_date ${build_line1}" > echo "${db_rec}" >> $hits_new > fi > build_line="${build_line2} ${build_line1}" > echo "${build_line}" >> $hits_no > in_line1="" > in_line2="" > save_in_line1="" > save_in_line2="" > else > continue > fi > done > exit 0 So the complete shell script would be: #!/bin/sh added_date="`date +%Y%m%d`" hits_rpt="/etc/ipf_pool_hits_rpt" hits_new="/etc/ipf_pool.hits.yes" hits_no="/etc/ipf_pool.hits.no" truncate -s 0 $hits_rpt $hits_yes $hits_no ippool -l -d -m probing_ips > $hits_rpt 2> /dev/null tail -n +4 $hits_rpt | while read line do ip=$(echo $line | sed -e 's/ *Address: *\([^/]*\).*/\1/') read line hits=$(echo $line | sed -e 's/ *Hits *\([[:digit:]*]\).*/\1/') if [ "$hits" -gt 0 ]; then echo "$added_date ${ip};" >> $hits_new fi echo "$hits ${ip};" >> $hits_no done exit 0 Does this still take minutes to process your data? Bye, Andreas