Date: Wed, 07 Dec 2005 11:56:25 -0800 From: Jin Guojun <jinmtb@sbcglobal.net> To: Sean Chittenden <sean@gigave.com> Cc: Imri Zvik <imriz@co.zahav.net.il>, freebsd-performance@freebsd.org Subject: Re: very busy syslog server Message-ID: <43973E69.5000504@sbcglobal.net> In-Reply-To: <20051207184103.GD56998@sean.gigave.com> References: <A6E1736261841F40B12286B0D04B194503CE7839@IGMAIL.InetGold> <20051207184103.GD56998@sean.gigave.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Sean Chittenden wrote:
>>I'm trying to setup a syslog server to serve a large group of
>>servers. For the syslog daemon, I have chosen rsyslogd, and the
>>backend is mysql (on a different machine).
>>
>>The machine has 2 Intel Xeon 2.80GHz CPUs, and 1GB of RAM, and it is
>>running FreeBSD 6 (6.0-STABLE).
>>
>>The problem is, that I see a lot of UDP packets being dropped:
>>
>>udp:
>> 390202 datagrams received
>> 0 with incomplete header
>> 0 with bad data length field
>> 0 with bad checksum
>> 6 with no checksum
>> 0 dropped due to no socket
>> 0 broadcast/multicast datagrams dropped due to no socket
>>->>> 123677 dropped due to full socket buffers
>> 0 not for hashed pcb
>> 266525 delivered
>> 133260 datagrams output
>>
>>I have tried to increase net.inet.udp.recvspace, but it didn't solve
>>the problem.
>>
>>I would appreciate any hint or tips.
>>
>>
>
>When you're doing a large number of packets per second, you may want
>to look into enabling device polling(4). Right now, every packet
>results in an interrupt. With device polling, you can handle more
>than one packet per interrupt. See the man page for details.
>
Not quite, the interrupt interval depends on the device driver, or which
NIC is used.
A number NICs are able to to interrupt coalescence, which requires to
increase buffer
descriptor ring size (just for receiving buffer descriptors). Of course,
polling is a simple thing
to try.
Before we can come up a better way to alter a better solution for this
case, you also need to
monitor a few things:
What is NIC on this machine?
What is the CUP utilization in average and in case the packet drops? You
can simply write a
script to do this instead of instructing kernel to do so (since this
needs no super accurate):
run vmstat to record CPU utilization in every 1 to 3 seconds for use
when following event happens:
use netstat watch UDP and pipe it to awk "netstat -udp | awk
'$2=="drooped" {print $1; exit}'"
every 3-5 seconds, and compare the result with previous one to see if
any changes. If so,
grep the last couple of line from vmstat output records.
From your information, it seems that this machine has enough memory
bandwidth for syslog needs,
since it is not clear what this machine is for rlog daemon or sql
server, or both are on the same machine.
If the third case is true, then you may run out of memory bandwidth.
Under this circumstance,
you need to obtain the packet rate and the average packet size in order
to determine the I/O
and memory bandwidth requirements.
-Jin Guojun
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?43973E69.5000504>