Date: Thu, 8 Dec 2005 13:54:51 +0200 From: "Imri Zvik" <imriz@co.zahav.net.il> To: "jaws" <jaws@skyinet.net> Cc: freebsd-performance@freebsd.org Subject: RE: very busy syslog server Message-ID: <A6E1736261841F40B12286B0D04B194503CE7B1B@IGMAIL.InetGold>
next in thread | raw e-mail | index | archive | help
Already tried different values, even some really high arbitrary ones... I think I will try splitting the work load on a couple of instances of syslog...=20 I hope that would do the trick -- Imri Zvik PGP (2.6.3ia) Public Key: http://mariska.inter.net.il/~imriz/imriz.pgp =20 -----Original Message----- From: jaws [mailto:jaws@skyinet.net]=20 Sent: Thursday, December 08, 2005 1:48 PM To: Imri Zvik Cc: j_guojun@lbl.gov; freebsd-performance@freebsd.org Subject: Re: very busy syslog server What you need is more sockets available and a much larger sockbufs. Try=20 adjusting the following parameters: net.inet.udp.maxdgram net.inet.udp.recvspace Dont rely on the default values rather set it to a much higher values. Regards, jaws Imri Zvik wrote: >Hi, > >1. The NIC being used is "Intel(R) PRO/1000" (the em(4) driver). >2. The CPU utilization in average is between 15% and 20%. >3. This machine is being used _only_ for the sysloging - the database resides on another server. > >Meanwhile, I have added some more memory to the machine, and now it has 3GB of RAM, but I am still seeing packets being dropped due to full socket buffers. > >Thanks, > >-- >Imri Zvik >PGP (2.6.3ia) Public Key: http://mariska.inter.net.il/~imriz/imriz.pgp >=20 >________________________________________ >From: Jin Guojun [mailto:jinmtb@sbcglobal.net]=20 >Sent: Wednesday, December 07, 2005 9:56 PM >To: Sean Chittenden >Cc: Imri Zvik; freebsd-performance@freebsd.org >Subject: Re: very busy syslog server > >Sean Chittenden wrote:=20 >I'm trying to setup a syslog server to serve a large group of >servers. For the syslog daemon, I have chosen rsyslogd, and the >backend is mysql (on a different machine). > >The machine has 2 Intel Xeon 2.80GHz CPUs, and 1GB of RAM, and it is >running FreeBSD 6 (6.0-STABLE). > >The problem is, that I see a lot of UDP packets being dropped: > >udp: > 390202 datagrams received > 0 with incomplete header > 0 with bad data length field > 0 with bad checksum > 6 with no checksum > 0 dropped due to no socket > 0 broadcast/multicast datagrams dropped due to no socket >->>> 123677 dropped due to full socket buffers > 0 not for hashed pcb > 266525 delivered > 133260 datagrams output > >I have tried to increase net.inet.udp.recvspace, but it didn't solve >the problem. > >I would appreciate any hint or tips. > =20 > >When you're doing a large number of packets per second, you may want >to look into enabling device polling(4). Right now, every packet >results in an interrupt. With device polling, you can handle more >than one packet per interrupt. See the man page for details.=20 >Not quite, the interrupt interval depends on the device driver, or which NIC is used. >A number NICs are able to to interrupt coalescence, which requires to increase buffer >descriptor ring size (just for receiving buffer descriptors). Of course, polling is a simple thing >to try. > >Before we can come up a better way to alter a better solution for this case, you also need to >monitor a few things: > >What is NIC on this machine? > >What is the CUP utilization in average and in case the packet drops? You can simply write a >script to do this instead of instructing kernel to do so (since this needs no super accurate): > >run vmstat to record CPU utilization in every 1 to 3 seconds for use when following event happens: >use netstat watch UDP and pipe it to awk "netstat -udp | awk '$2=3D=3D"drooped" {print $1; exit}'" >every 3-5 seconds, and compare the result with previous one to see if any changes. If so, >grep the last couple of line from vmstat output records. > >>From your information, it seems that this machine has enough memory bandwidth for syslog needs, >since it is not clear what this machine is for rlog daemon or sql server, or both are on the same machine. >If the third case is true, then you may run out of memory bandwidth. Under this circumstance, >you need to obtain the packet rate and the average packet size in order to determine the I/O >and memory bandwidth requirements. > > -Jin Guojun >_______________________________________________ >freebsd-performance@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-performance >To unsubscribe, send any mail to "freebsd-performance-unsubscribe@freebsd.org" > > =20 >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A6E1736261841F40B12286B0D04B194503CE7B1B>