Date: Mon, 7 Aug 2017 10:22:03 +0100 From: Matthew Seaman <matthew@FreeBSD.org> To: freebsd-questions@freebsd.org Subject: Re: log centralizer? Message-ID: <9df870be-21bb-94e3-924a-bedc54b7152c@FreeBSD.org> In-Reply-To: <1502086823.5923.150.camel@pki2.com> References: <CACcSE1xP0c8yA44LLwTfX2nWWBqJf=VEgR07TbqmZwKTaLCd%2Bw@mail.gmail.com> <1502086823.5923.150.camel@pki2.com>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --pvWQ4bVN8TA0XHtcwL79n69vE0ek9QuRs Content-Type: multipart/mixed; boundary="pbXcf2OkJHDQdTplMbDQQeHcnFTloD4CG"; protected-headers="v1" From: Matthew Seaman <matthew@FreeBSD.org> To: freebsd-questions@freebsd.org Message-ID: <9df870be-21bb-94e3-924a-bedc54b7152c@FreeBSD.org> Subject: Re: log centralizer? References: <CACcSE1xP0c8yA44LLwTfX2nWWBqJf=VEgR07TbqmZwKTaLCd+w@mail.gmail.com> <1502086823.5923.150.camel@pki2.com> In-Reply-To: <1502086823.5923.150.camel@pki2.com> --pbXcf2OkJHDQdTplMbDQQeHcnFTloD4CG Content-Type: text/plain; charset=utf-8 Content-Language: en-GB Content-Transfer-Encoding: quoted-printable On 07/08/2017 07:20, Dennis Glatting wrote: > On Sun, 2017-08-06 at 22:39 -0700, Aleksandr Miroslav wrote: >> I'm looking for a mechanism to collect and store all logs into a >> centralized location. I'm not looking for a fancy graphical interface >> (a la Splunk) to search those logs just yet, just collecting them on >> a >> centralized server is fine for the moment. >> >> Is there something available in ports/base that I can use for this >> purpose? I took a quick look at ELK, it seems overly complicated, but >> iIve never used it. >=20 > The simple approach is to have a central MySQL database fed from > rsyslog across the servers of interest. Costume devices, such as HVAC, > could point to a rsyslog server which then feeds the database.=20 >=20 > Periodically run scripts against the database to generate summary > information, build firewall rule sets, and for maintenance. >=20 > For weird things, such as netflow off the switches and routers,=20 > forward the flows to a server, parse it, and then stuff it into the > database. >=20 > You can also create multi-master databases in case one goes offline or > local optimization. I was looking at Cassandra for multi-master. You can just use the default system syslog to collect the logs onto a central logging server, but this would write everything to a log file, so probably only really satisfactory for quite a low-traffic setup. rsyslog will allow you much greater flexibility in where and how you write the logging data, including creating separate log files for each day or hour, or writing into a database or interfacing with various ELK type things like logstash. Note that anything based on (r)syslog doesn't guarantee successful delivery of log data to your server. Anything that fails to be received will be silently dropped. There's no concept like queuing up log messages for later delivery should the log server be temporarily off-line[*]. This is a fairly typical requirement: you don't want your webserver to stop responding simply because it cannot send syslog messages for a while. If you want more resilience, then consider an ElasticSearch cluster -- this will work best if you use a parser on the incoming log data to structure the messages appropriately for searching. Use Kibana as a query tool or to generate dashboards showing live performance data. Something like logstash will work for processing the raw log messages into something more readily searchable in ElasticSearch. However, think twice about running logstash clients on your frontend machines -- that's a big fat dollop of java or python to add to the load on your alreay hardworking servers. You can use (r)syslog to feed data into a remote Logstash setup pretty well. Cheers, Matthew --pbXcf2OkJHDQdTplMbDQQeHcnFTloD4CG-- --pvWQ4bVN8TA0XHtcwL79n69vE0ek9QuRs Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQJ8BAEBCgBmBQJZiDFBXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ2NTNBNjhCOTEzQTRFNkNGM0UxRTEzMjZC QjIzQUY1MThFMUE0MDEzAAoJELsjr1GOGkATYksP/AunFjeLlUtjJs/g6S14kJEi 52TJiTFL+46i40pMQE6ynA038fhTFnYnYz/9JjMVDxXW2Gz1KUeoS9E6/BLRz4V6 iLL6FVHsrTkxLFHHhktHiA/BITEo+HkTomcV1lsyN9lw2YqKPa4d6hHrZbQvLQDG 1647c/U9CucqusSTnfPjZDmx0ppPbrrZn7k2uTE1p5Qee3+NQUdkL3qNYYzthJD3 wI3Z0JUn2PCvDjV6SuR+4GKXOyEDZFrJyfIAhBM9PM2dy5cDtfRy62pWR8VFBjGx LiOqI4n8blLyEnwUQaDK9+REkWi8ATOfZqQ/lpWEbnWSnxF8yErmXjgk+c+2FXER eTia/Q4BPgsYeJov5WHOmwGPYRqEeXgh0qOJxfA7zqB0VBBDAAF8WjIveqIiAI4o 5jM/jmh5MbFdc3UnsiVsqze6TH3FQNlRl4Wq3IZspzTAIT1gR98VlIBb0ZrTN7nx VMnf9S1kP0usqD4d9jXjYTOlYDFCgvz+ZEnc5SdUJBIOZBDNLoodWMDIdFPTPr0h x3qpohqGwJPABr9OqV53h4uv4F7HTCMQ+OLAjg63//fCJRGwcGAoWtGrdxLnVBcJ 4PssVOJxX1XIPv8tZ68qwCZN6nxMp+u7ini1sfZlZPyd0fMMDBuPttBan6v8MBYc TI9x9JxKUVTJkdae7yXM =Ef5w -----END PGP SIGNATURE----- --pvWQ4bVN8TA0XHtcwL79n69vE0ek9QuRs--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9df870be-21bb-94e3-924a-bedc54b7152c>