Date: Sat, 16 Sep 2017 20:27:50 -0500 From: Edgar Pettijohn <edgar@pettijohn-web.com> To: Ernie Luzar <luzar722@gmail.com> Cc: freebsd-questions@freebsd.org Subject: Re: Help scripting dns lookup using awk Message-ID: <20170917012749.GA50553@FreeBSD> In-Reply-To: <59BD3410.80708@gmail.com> References: <59BB24E4.6060908@gmail.com> <20170915143019.2e02d386@gumby.homeunix.com> <59BC6036.8040709@gmail.com> <CAOLAi33Uj-XmxSpvAgtMwQAKKAxLH60eqmVV177BYLrv7vQs9g@mail.gmail.com> <59BD3410.80708@gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Sat, Sep 16, 2017 at 10:24:16AM -0400, Ernie Luzar wrote: > Jonathan McKeown wrote: > > Ernie, I've been following your questions over the last month or so. I > > think I can guess your problem domain, and I suspect if you told the > > list what you're trying to achieve you'd get much better suggested > > solutions. > > > > As it is I think you have one approach in mind, and all your questions > > relate to implementing parts of your idea. > > > > My humble apologies if I'm wrong; but please consider explaining what > > the overall problem is. (Someone mentioned x-y problems - I think you > > have one here.) > > Yes all my different posts over the last month are related to a solution > I am trying to development. It all started with what looked like a very > simple request from top management. "Stop employees from using social > media from company PCs while at work" The one and only Freebsd system > is the front door to the Company LAN and wifi. All LAN devices are > WINDOW machines either cabled or wifi including hand held smart phones. > So needed a single point solution that would effect the whole digital shop. > > You ask what about smart phones accessing their wireless service. In the > USA a wireless signal jammer is not legal if the people being effected > are un-aware of its existence. On being hired all employees sign a legal > contract containing security requirements and are made aware that a cell > phone wireless signal jammer is employed covering the Company estate and > that Company land lines phone service is the only allowed way for phone > contact with the public for personal and Company business. > > As the result of questions posted here, I learned about online providers > of "host" lists. These lists contain '127.0.0.1 domain-name" records of > known malware sites. These "host" lists can be used on WINDOWS and Unix > flavored operating systems by populating those machines host file. This > was not a single point solution. > > Along comes using dns as a single point solution. The 3 main players > being bind, unbound, and Dnsmasq which all have the function to be > populated with domain-names to be blocked at the local host level as not > found. I chose unbound, but am having problems with /etc/resolv.conf and > resolvconf not working as documented. Also could not get the built-in > local-unbound to work with any local changes. Posted questions here > which went un-answered. local-unbound and resolvconf are new and don't > have a user base yet to draw answers from. So pretty much a dead end. I > finally installed the port version of unbound and got it working. > > Using the public host files and unbound became a single point solution > to provide protection LAN wide that is un-seen by the user base. No more > installing browser plug-ins that tries to do the same blocking function. > The dns solution provides protection to the LAN users from LAN machines > that may become infected. There is no absolute solution just more layers > of protection. > > These public available "host" files contain a lot of un-necessary junk > that needed to be cleaned away. I wrote a .sh script to do this, but it > was very slow. Got help from this list to convert it to awk. Using the > same sample input file .sh took 7+ minutes, the awk version took 4 > seconds. No brainier about which version I plan to use. > > As the last step in massaging the raw "host" file content what'd to do a > dns lookup to verify those host domain-names were really good. Feeding > unbound bogus domain-names is not going to hurt anything, but just > wanted to be thorough. Again I started with a .sh script using the host > system command which is very slow, I got help here from the list to > convert it to awk and it was only a few seconds faster over all. I > changed the .sh script to use the drill system command and it ran in > half the time the host command version took. > > In reply to the subject of this post, I got the following; > > " Almost certainly the reason it's slow is that you are doing sequential > synchronous lookups. Switching to another language isn't going help > much. To speed it up you either need to switch to a language with a > DNS library that supports asynchronous lookups or fire-off parallel > child processes. The latter is easier." > > So I posted my last reply asking; > > How would I go about coding a sh script to fire-off parallel child > processes? > > The only "other language" installed on my front door host is perl > because its part of the apache pkg. I don't want to install another > language just because it has a fast pre-canned dns lookup. > > So if anyone knows of a perl dns lookup solution I sure would be > interested in hearing about it. > > While waiting for a reply to that last question I have done more > testing. Using the drill command version of the .sh script against a > "host" file containing 409 records which is the smallest file I have, > found that 174 host names return mddomain or serverror. So it's > oblivious that all 12 host files need dns verification. Thats 900,000+ > records. > > If I run that .sh script against the same host file I start receiving > this console message; > > Error: error sending query: Could not send or receive, because of > network error Looking at the source for drill. I believe this may be a generic error for the sending, or receiving of the dns packet. Perhaps a snippet of the script you are running showing how you are using drill may shed some light on it. > > The results indicate all the hosts were looked up. My isp provides 1gb > upload and 3gb download speeds so limited speed is not the cause of the > network error. > > Does anyone have any ideas about what is going here? > > > > > > > > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to "freebsd-questions-unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20170917012749.GA50553>