Skip site navigation (1)Skip section navigation (2)
Date:      Sat, 16 Sep 2017 10:24:16 -0400
From:      Ernie Luzar <>
To:        Jonathan McKeown <>
Subject:   Re: Help scripting dns lookup using awk
Message-ID:  <>
In-Reply-To: <>
References:  <> <> <> <>

Next in thread | Previous in thread | Raw E-Mail | Index | Archive | Help
Jonathan McKeown wrote:
> Ernie, I've been following your questions over the last month or so. I 
> think I can guess your problem domain, and I suspect if you told the 
> list what you're trying to achieve you'd get much better suggested 
> solutions.
> As it is I think you have one approach in mind, and all your questions 
> relate to implementing parts of your idea.
> My humble apologies if I'm wrong; but please consider explaining what 
> the overall problem is. (Someone mentioned x-y problems - I think you 
> have one here.) 

Yes all my different posts over the last month are related to a solution 
I am trying to development. It all started with what looked like a very 
simple request from top management. "Stop employees from using social 
media from company PCs while at work"  The one and only Freebsd system 
is the front door to the Company LAN and wifi. All LAN devices are 
WINDOW machines either cabled or wifi including hand held smart phones. 
So needed a single point solution that would effect the whole digital shop.

You ask what about smart phones accessing their wireless service. In the 
USA a wireless signal jammer is not legal if the people being effected 
are un-aware of its existence. On being hired all employees sign a legal 
contract containing security requirements and are made aware that a cell 
phone wireless signal jammer is employed covering the Company estate and 
that Company land lines phone service is the only allowed way for phone 
contact with the public for personal and Company business.

As the result of questions posted here, I learned about online providers 
of "host" lists. These lists contain ' domain-name" records of 
known malware sites. These "host" lists can be used on WINDOWS and Unix 
flavored operating systems by populating those machines host file. This 
was not a single point solution.

Along comes using dns as a single point solution. The 3 main players 
being bind, unbound, and Dnsmasq which all have the function to be 
populated with domain-names to be blocked at the local host level as not 
found. I chose unbound, but am having problems with /etc/resolv.conf and 
resolvconf not working as documented. Also could not get the built-in 
local-unbound to work with any local changes. Posted questions here 
which went un-answered. local-unbound and resolvconf are new and don't 
have a user base yet to draw answers from. So pretty much a dead end. I 
finally installed the port version of unbound and got it working.

Using the public host files and unbound became a single point solution 
to provide protection LAN wide that is un-seen by the user base. No more 
installing browser plug-ins that tries to do the same blocking function. 
The dns solution provides protection to the LAN users from LAN machines 
that may become infected. There is no absolute solution just more layers 
of protection.

These public available "host" files contain a lot of un-necessary junk 
that needed to be cleaned away. I wrote a .sh script to do this, but it 
was very slow. Got help from this list to convert it to awk. Using the 
same sample input file .sh took 7+ minutes, the awk version took 4 
seconds. No brainier about which version I plan to use.

As the last step in massaging the raw "host" file content what'd to do a 
dns lookup to verify those host domain-names were really good. Feeding 
unbound bogus domain-names is not going to hurt anything, but just 
wanted to be thorough. Again I started with a .sh script using the host 
system command which is very slow, I got help here from the list to 
convert it to awk and it was only a few seconds faster over all. I 
changed the .sh script to use the drill system command and it ran in 
half the time the host command version took.

In reply to the subject of this post, I got the following;

" Almost certainly the reason it's slow is that you are doing sequential
   synchronous lookups. Switching  to another language isn't going help
   much. To speed it up you either need to switch to a language with a
   DNS library that supports asynchronous lookups or fire-off parallel
   child processes. The latter is easier."

So I posted my last reply asking;

How would I go about coding a sh script to fire-off parallel child 

The only "other language" installed on my front door host is perl 
because its part of the apache pkg. I don't want to install another 
language just because it has a fast pre-canned dns lookup.

So if anyone knows of a perl dns lookup solution I sure would be 
interested in hearing about it.

While waiting for a reply to that last question I have done more 
testing. Using the drill command version of the .sh script against a 
"host" file containing 409 records which is the smallest file I have, 
found that 174 host names return mddomain or serverror. So it's 
oblivious that all 12 host files need dns verification. Thats 900,000+ 

If I run that .sh script against the same host file I start receiving 
this console message;

Error: error sending query: Could not send or receive, because of 
network error

The results indicate all the hosts were looked up. My isp provides 1gb 
upload and 3gb download speeds so limited speed is not the cause of the 
network error.

Does anyone have any ideas about what is going here?

Want to link to this message? Use this URL: <>