From owner-freebsd-questions@freebsd.org Sat Sep 16 14:24:18 2017 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F0D11E19B25 for ; Sat, 16 Sep 2017 14:24:18 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: from mail-it0-x243.google.com (mail-it0-x243.google.com [IPv6:2607:f8b0:4001:c0b::243]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B4EB37C026 for ; Sat, 16 Sep 2017 14:24:18 +0000 (UTC) (envelope-from luzar722@gmail.com) Received: by mail-it0-x243.google.com with SMTP id u2so2941971itb.2 for ; Sat, 16 Sep 2017 07:24:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-transfer-encoding; bh=YFWhzJIDLqbeM12+5ci7/PfAuOKhHJgIHGStyeq48iQ=; b=Xoi1wgksuJRgjQupAigIRkSxqftHCEXNKgsMJSs+bM/ibeVeP5QnHqwDm/aG610r/i /7PQn4s29AAHl+lDKqxQvbbHTh/xgbNHvM0EZDAtuLUs9ls4atEEoDNMapIkyuy19ie8 wCj1N2cQObeJF6tfVdBJIFbFXSAw0e9SjcdT6q8hrsb7IStQS0N9H3gMN3uflVW6AKMM wXgoDVNf+/Buay4YG+mqOsAP/JVd5a2ZuPTCxCaLSnma+ZdVPvXdv62BHD3VwnNCphQf rOuau4SUboDQpk5wZfj9UTtW7sVG4zZSgxIL9RhzaF3uSa02gyDSEzcJ+MKiHII1GjLv vp0Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:date:from:user-agent:mime-version:to :cc:subject:references:in-reply-to:content-transfer-encoding; bh=YFWhzJIDLqbeM12+5ci7/PfAuOKhHJgIHGStyeq48iQ=; b=fV8816m9NeyjYdopnmdPi6lBXO2dVx/0dbyjLj8jeCAshyyvcC7h87xwiN7CeYQa4l fkWez/17uAqnP+15wAwRK6uL5k5q5nNdjkjMrQGeRbBiCWnBaLWDcAXfvGt1CV9G14xj 3Bw/roRZiI0g8Pe5SbEqWi3Ep+2PkD5cunz7D0D3g5W2TtxZOeyNu4AoIpnAttbNwSjV T0X6ZInACG6hFfoXOE+YhV24ZfU0CPbaANTVxoEpDrdR1Umj2JEHxqLnjXR0bokRDa6i c617iyj/4vUawfrUrrygG7e4u+UzwJ87k5f6pW2YvVHss9hOU068W150J4FNNH8BfFLg F6iw== X-Gm-Message-State: AHPjjUjXt6klTHFVKnDgeYL9pau4MQ81uw6IIhdejZFKd/DRSWNDtTQ2 ee1dDaKHZL9fu6PM X-Google-Smtp-Source: AOwi7QCX/r8//fEdNYvhUr75b4/W16XgjZVjmx6wO68c9L78AE5iIwPB+YKKGMJ1d3QqQCymI7Vcpw== X-Received: by 10.36.213.193 with SMTP id a184mr9223779itg.121.1505571857970; Sat, 16 Sep 2017 07:24:17 -0700 (PDT) Received: from [10.0.10.3] (cpe-74-141-88-147.neo.res.rr.com. [74.141.88.147]) by smtp.googlemail.com with ESMTPSA id g9sm67051itg.41.2017.09.16.07.24.17 (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Sat, 16 Sep 2017 07:24:17 -0700 (PDT) Message-ID: <59BD3410.80708@gmail.com> Date: Sat, 16 Sep 2017 10:24:16 -0400 From: Ernie Luzar User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: Jonathan McKeown CC: freebsd-questions@freebsd.org Subject: Re: Help scripting dns lookup using awk References: <59BB24E4.6060908@gmail.com> <20170915143019.2e02d386@gumby.homeunix.com> <59BC6036.8040709@gmail.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Sep 2017 14:24:19 -0000 Jonathan McKeown wrote: > Ernie, I've been following your questions over the last month or so. I > think I can guess your problem domain, and I suspect if you told the > list what you're trying to achieve you'd get much better suggested > solutions. > > As it is I think you have one approach in mind, and all your questions > relate to implementing parts of your idea. > > My humble apologies if I'm wrong; but please consider explaining what > the overall problem is. (Someone mentioned x-y problems - I think you > have one here.) Yes all my different posts over the last month are related to a solution I am trying to development. It all started with what looked like a very simple request from top management. "Stop employees from using social media from company PCs while at work" The one and only Freebsd system is the front door to the Company LAN and wifi. All LAN devices are WINDOW machines either cabled or wifi including hand held smart phones. So needed a single point solution that would effect the whole digital shop. You ask what about smart phones accessing their wireless service. In the USA a wireless signal jammer is not legal if the people being effected are un-aware of its existence. On being hired all employees sign a legal contract containing security requirements and are made aware that a cell phone wireless signal jammer is employed covering the Company estate and that Company land lines phone service is the only allowed way for phone contact with the public for personal and Company business. As the result of questions posted here, I learned about online providers of "host" lists. These lists contain '127.0.0.1 domain-name" records of known malware sites. These "host" lists can be used on WINDOWS and Unix flavored operating systems by populating those machines host file. This was not a single point solution. Along comes using dns as a single point solution. The 3 main players being bind, unbound, and Dnsmasq which all have the function to be populated with domain-names to be blocked at the local host level as not found. I chose unbound, but am having problems with /etc/resolv.conf and resolvconf not working as documented. Also could not get the built-in local-unbound to work with any local changes. Posted questions here which went un-answered. local-unbound and resolvconf are new and don't have a user base yet to draw answers from. So pretty much a dead end. I finally installed the port version of unbound and got it working. Using the public host files and unbound became a single point solution to provide protection LAN wide that is un-seen by the user base. No more installing browser plug-ins that tries to do the same blocking function. The dns solution provides protection to the LAN users from LAN machines that may become infected. There is no absolute solution just more layers of protection. These public available "host" files contain a lot of un-necessary junk that needed to be cleaned away. I wrote a .sh script to do this, but it was very slow. Got help from this list to convert it to awk. Using the same sample input file .sh took 7+ minutes, the awk version took 4 seconds. No brainier about which version I plan to use. As the last step in massaging the raw "host" file content what'd to do a dns lookup to verify those host domain-names were really good. Feeding unbound bogus domain-names is not going to hurt anything, but just wanted to be thorough. Again I started with a .sh script using the host system command which is very slow, I got help here from the list to convert it to awk and it was only a few seconds faster over all. I changed the .sh script to use the drill system command and it ran in half the time the host command version took. In reply to the subject of this post, I got the following; " Almost certainly the reason it's slow is that you are doing sequential synchronous lookups. Switching to another language isn't going help much. To speed it up you either need to switch to a language with a DNS library that supports asynchronous lookups or fire-off parallel child processes. The latter is easier." So I posted my last reply asking; How would I go about coding a sh script to fire-off parallel child processes? The only "other language" installed on my front door host is perl because its part of the apache pkg. I don't want to install another language just because it has a fast pre-canned dns lookup. So if anyone knows of a perl dns lookup solution I sure would be interested in hearing about it. While waiting for a reply to that last question I have done more testing. Using the drill command version of the .sh script against a "host" file containing 409 records which is the smallest file I have, found that 174 host names return mddomain or serverror. So it's oblivious that all 12 host files need dns verification. Thats 900,000+ records. If I run that .sh script against the same host file I start receiving this console message; Error: error sending query: Could not send or receive, because of network error The results indicate all the hosts were looked up. My isp provides 1gb upload and 3gb download speeds so limited speed is not the cause of the network error. Does anyone have any ideas about what is going here?