Date: Wed, 4 May 2011 14:42:33 -0700 From: Jeremy Chadwick <freebsd@jdc.parodius.com> To: "Mikhail T." <mi+thun@aldan.algebra.com> Cc: freebsd-stable@freebsd.org Subject: Re: syslogd spinning the CPU, not logging... Message-ID: <20110504214233.GA97428@icarus.home.lan> In-Reply-To: <4DC1A1CC.4080901@aldan.algebra.com> References: <4DC1A1CC.4080901@aldan.algebra.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Wed, May 04, 2011 at 02:58:20PM -0400, Mikhail T. wrote: > On FreeBSD/amd64 8.2-STABLE as of Sun Feb 27. I dismissed this > problem the first time (a week or two ago), but just saw it again. > Something triggers syslogd into spinning all available CPU -- while > not logging anything. > > Attempts to log anything -- such as by invoking logger(1) -- simply > hang. sendmail hangs too -- now new e-mail is coming. It is not > pretty. > > ktrace-ing the process yields empty file -- it is not doing > system-calls, while spinning. > > Attaching the debugger several times shows the same stack: > > #0 0x00000008007cb1d6 in _pthread_mutex_init_calloc_cb () from /lib/libc.so.7 > #1 0x00000008007cd9fb in _pthread_mutex_init_calloc_cb () from /lib/libc.so.7 > #2 0x00000008007d4825 in free () from /lib/libc.so.7 > #3 0x00000008007c5aab in catopen () from /lib/libc.so.7 > #4 0x00000008007c5410 in strerror_r () from /lib/libc.so.7 > #5 0x00000008007c556d in strerror () from /lib/libc.so.7 > #6 0x0000000000403c10 in ?? () > ... > > Among the logging-destinations in my syslog.conf there is a program, > that exits sometimes -- but never "too fast". Normally, syslogd > simply restarts it without an issue... > > Any ideas? Thanks! If ktrace doesn't show any syscalls (did you use "ktrace -i -t+"?), then I'm not sure how the process is actually spinning (100% CPU). "procstat -kk" on the PID of syslogd might be more useful in this case. The output is mainly useful for kernel folks. You may end up having to drop to ddb to examine what's going on within the pid. I'm not sure if attaching gdb to a running process on FreeBSD works with things like threads, etc.. Meaning I'm not sure if the above stack trace is indicative of a problem or not. No offence to anyone intended, but I try to avoid gdb on FreeBSD wherever possible (it's so old that I don't know how useful it is for extreme debugging). I've never seen syslogd spiral out of control on any of our systems, but all logging destinations in syslog.conf on our systems are files on local disks. If you can find a way to reliably reproduce the issue that would be useful, otherwise this sounds like one of those things that's going to be a problem to track down. :-( -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP 4BD6C0CB |
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20110504214233.GA97428>