Date: Wed, 5 Oct 2011 23:32:50 +0000 From: "Li, Qing" <qing.li@bluecoat.com> To: Steven Hartland <killing@multiplay.co.uk>, "freebsd-stable@freebsd.org" <freebsd-stable@freebsd.org> Cc: "liv3d@multiplay.co.uk" <liv3d@multiplay.co.uk> Subject: RE: serious packet routing issue causing ntpd high load? Message-ID: <B143A8975061C446AD5E29742C53172313BC17@PWSVL-EXCMBX-01.internal.cacheflow.com> In-Reply-To: <2740DA0E7C0F495BA00E6D4BC0FE8F37@multiplay.co.uk> References: <2740DA0E7C0F495BA00E6D4BC0FE8F37@multiplay.co.uk>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,
>
> RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno
> 0, flags:<DONE>
> locks: inits:
> sockaddrs: <DST>
> ::A.B.C.D
>
Would it be possible for you to email me what exactly does "::A.B.C.D"=20
map into WRT your system or infrastructure ?
And are you able to share your "ifconfig -a" and "netstat -rn" output
with me privately ?
--Qing
> -----Original Message-----
> From: owner-freebsd-stable@freebsd.org [mailto:owner-freebsd-
> stable@freebsd.org] On Behalf Of Steven Hartland
> Sent: Tuesday, October 04, 2011 5:00 PM
> To: freebsd-stable@freebsd.org
> Cc: liv3d@multiplay.co.uk
> Subject: serious packet routing issue causing ntpd high load?
>=20
> We just updated a machine to 8-STABLE and I've noticed
> that ntpd is using notible amounts of CPU 5-7% which
> is very high for such a trivial daemon.
>=20
> 8.2-STABLE FreeBSD 8.2-STABLE #16: Tue Oct 4 09:53:17 UTC 2011
>=20
> truss indicates its constantly checking and reading
> from a socket
>=20
> 0.047297485 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) =3D 1
> (0x1)
> 0.047513160 clock_gettime(0,{1317770389.969538247 }) =3D 0 (0x0)
> 0.047604515 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 })
> =3D 1 (0x1)
> 0.047668212 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) =3D 184
> (0xb8)
> 0.049395293 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) =3D 1
> (0x1)
> 0.049503689 clock_gettime(0,{1317770389.971526820 }) =3D 0 (0x0)
> 0.049606219 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,{0.000000 })
> =3D 1 (0x1)
> 0.049669916 read(28,"\M-8\0\^E\a\0\0\0\0@\0\0\0\^A\0"...,5120) =3D 184
> (0xb8)
> 0.049809882 select(29,{20 21 22 23 24 25 26 27 28},0x0,0x0,0x0) =3D 1
> (0x1)
> ...
>=20
> running with debug enabled it sits looping outputting:-
> routing message op =3D 7: ignored
> routing message op =3D 7: ignored
> routing message op =3D 7: ignored
> routing message op =3D 7: ignored
> routing message op =3D 7: ignored
> routing message op =3D 7: ignored
> routing message op =3D 7: ignored
> ...
>=20
> It seems socket 28 is a duplicate of an internal routing socket
> as seen here in the trace:-
> 0.044544269 socket(PF_ROUTE,SOCK_RAW,0) =3D 4 (0x4)
> 0.044595394 fcntl(4,F_DUPFD,0x14) =3D 28 (0x1c)
> 0.044645960 close(4) =3D 0 (0x0)
> 0.044695968 fcntl(28,F_SETFL,O_NONBLOCK) =3D 0 (0x0)
>=20
> Now this looks like its RTM_MISS as defined:-
> sys/net/route.h:#define RTM_MISS 0x7 /* Lookup failed on
> this address */
>=20
> So the question was why is PF_ROUTE socket constantly
> spamming RTM_MISS?
>=20
> route -n monitor on this machines shows:-
> got message of size 184 on Tue Oct 4 23:46:36 2011
> RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno
> 0, flags:<DONE>
> locks: inits:
> sockaddrs: <DST>
> ::A.B.C.D
>=20
> got message of size 184 on Tue Oct 4 23:46:36 2011
> RTM_MISS: Lookup failed on this address: len 184, pid: 0, seq 0, errno
> 0, flags:<DONE>
> locks: inits:
> sockaddrs: <DST>
> ::A.B.C.D
>=20
> This seems very much like the following pr which was fixed:-
> "Remove a bogusly introduced rtalloc_ign() in rev. 1.335/SVN 178029,
> generating an RTM_MISS for every IP packet forwarded making user space
> routing daemons unhappy":-
> http://www.freebsd.org/cgi/query-pr.cgi?pr=3D124540
>=20
> The box is doing no routing, its fairly basic install with
> 1 main IP on em0 + 1 alias + gw addres and 1 private ip on em1.
>=20
> Its running mysql and thats about it.
>=20
> Any ideas?
>=20
> Regards
> Steve
>=20
>=20
>=20
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> This e.mail is private and confidential between Multiplay (UK) Ltd. and
> the person or entity to whom it is addressed. In the event of
> misdirection, the recipient is prohibited from using, copying, printing
> or otherwise disseminating it or any information contained in it.
>=20
> In the event of misdirection, illegible or incomplete transmission
> please telephone +44 845 868 1337
> or return the E.mail to postmaster@multiplay.co.uk.
>=20
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to "freebsd-stable-
> unsubscribe@freebsd.org"
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B143A8975061C446AD5E29742C53172313BC17>