Date: Sat, 22 Oct 2011 13:15:28 GMT From: Steven Hartland <killing@multiplay.co.uk> To: freebsd-gnats-submit@FreeBSD.org Subject: kern/161899: Repeating RTM_MISS packets causing high CPU load for ntpd Message-ID: <201110221315.p9MDFSTN066935@red.freebsd.org> Resent-Message-ID: <201110221320.p9MDK987040515@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 161899 >Category: kern >Synopsis: Repeating RTM_MISS packets causing high CPU load for ntpd >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sat Oct 22 13:20:08 UTC 2011 >Closed-Date: >Last-Modified: >Originator: Steven Hartland >Release: 8.2-STABLE >Organization: Multiplay >Environment: FreeBSD db4.multiplay.co.uk 8.2-STABLE FreeBSD 8.2-STABLE #16: Tue Oct 4 09:53:17 UTC 2011 root@db3-old.multiplay.co.uk:/usr/obj/usr/src/sys/MULTIPLAY amd64 >Description: We just updated a machine to 8-STABLE and I've noticed that ntpd is using notible amounts of CPU 5-7% which is very high for such a trivial daemon. 8.2-STABLE FreeBSD 8.2-STABLE #16: Tue Oct 4 09:53:17 UTC 2011 truss indicates its constantly checking and reading from a socket 0.047297485 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1) 0.047513160 clock_gettime(0,{1317770389.969538247 }) = 0 (0x0) 0.047604515 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 }) = 1 (0x1) 0.047668212 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) = 184 (0xb8) 0.049395293 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1) 0.049503689 clock_gettime(0,{1317770389.971526820 }) = 0 (0x0) 0.049606219 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 }) = 1 (0x1) 0.049669916 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) = 184 (0xb8) 0.049809882 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) = 1 (0x1) .. running with debug enabled it sits looping outputting:- routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored routing message op = 7: ignored .. It seems socket 28 is a duplicate of an internal routing socket as seen here in the trace:- 0.044544269 socket(PF_ROUTE,SOCK_RAW,0) = 4 (0x4) 0.044595394 fcntl(4,F_DUPFD,0x14) = 28 (0x1c) 0.044645960 close(4) = 0 (0x0) 0.044695968 fcntl(28,F_SETFL,O_NONBLOCK) = 0 (0x0) Now this looks like its RTM_MISS as defined:- sys/net/route.h:#define RTM_MISS 0x7 /* Lookup failed on this address */ So the question was why is PF_ROUTE socket constantly spamming RTM_MISS? route -n monitor on this machines shows:- got message of size 184 on Tue Oct 4 23:46:36 2011 RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno 0, flags:<DONE> locks: inits: sockaddrs: <DST> ::A.B.C.D got message of size 184 on Tue Oct 4 23:46:36 2011 RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno 0, flags:<DONE> locks: inits: sockaddrs: <DST> ::A.B.C.D This seems very much like the following pr which was fixed:- "Remove a bogusly introduced rtalloc_ign() in rev. 1.335/SVN 178029, generating an RTM_MISS for every IP packet forwarded making user space routing daemons unhappy":- http://www.freebsd.org/cgi/query-pr.cgi?pr=124540 The box is doing no routing, its fairly basic install with 1 main IP on em0 + 1 alias + gw addres and 1 private ip on em1. Its running mysql and thats about it. Additional discussion can be found in the the thread topic freebsd-stable list "serious packet routing issue causing ntpd high load?" >How-To-Repeat: According to Vlad Galu, which Li Qing confirms on the original discussion thread: "serious packet routing issue causing ntpd high load?" 1. Start listening for routing messages on a non-default FIB, e.g. setfib 2 route monitor 2. Add any static route within that FIB. 3. The machine I run the test on is a heavy DNS client, it generates a few dozen requests per second. The monitor process starts getting RTM_MISS messages for each outgoing DNS request (the dst sockaddr is the same as my first resolv.conf entry, seq is always 0). >Fix: >Release-Note: >Audit-Trail: >Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201110221315.p9MDFSTN066935>