Date: Tue, 7 Oct 2014 13:21:47 +0200 From: Julien Cigar <jcigar@ulb.ac.be> To: hiren panchasara <hiren@freebsd.org> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org>, FreeBSD Stable Mailing List <freebsd-stable@freebsd.org> Subject: Re: [jcigar@ulb.ac.be: Listen queue overflow: 8 already in queue awaiting acceptance] Message-ID: <20141007112147.GI63350@mordor.lan> In-Reply-To: <CALCpEUFATvCbVd6MJ-_Ry%2BfzBAEAo535xfbGq4kPX2Y6Yz=yVA@mail.gmail.com> References: <20141002164202.GI29361@mordor.lan> <CALCpEUGjeLo3jid5vb0SxCRM7NBkWe2FdKLAoUM=%2B34ALZs3Vw@mail.gmail.com> <20141002181958.GJ29361@mordor.lan> <CALCpEUFb=z3p-BhydcB01uiAFndOGE%2BD1Dvj7mm8PF%2BsnNQ37Q@mail.gmail.com> <20141003080150.GL29361@mordor.lan> <CALCpEUFATvCbVd6MJ-_Ry%2BfzBAEAo535xfbGq4kPX2Y6Yz=yVA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--VSaCG/zfRnOiPJtU Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Oct 03, 2014 at 10:31:56AM -0700, hiren panchasara wrote: > On Fri, Oct 3, 2014 at 1:01 AM, Julien Cigar <jcigar@ulb.ac.be> wrote: > > On Thu, Oct 02, 2014 at 04:36:49PM -0700, hiren panchasara wrote: > >> On Thu, Oct 2, 2014 at 11:19 AM, Julien Cigar <jcigar@ulb.ac.be> wrote: > >> > On Thu, Oct 02, 2014 at 10:24:13AM -0700, hiren panchasara wrote: > >> >> On Thu, Oct 2, 2014 at 9:42 AM, Julien Cigar <jcigar@ulb.ac.be> wro= te: > >> >> > sorry for cross-posting, I'm forwarding this as it seems that par= t of > >> >> > the problem is also related to: > >> >> > https://lists.freebsd.org/pipermail/freebsd-net/2014-September/03= 9664.html > >> >> > >> >> Umm, this looks like a different problem than the subject of this e= mail. > >> > > >> > yes and no, seems the same hardware (HP and igb) and I have also some > >> > "requests for mbufs denied" (https://dpaste.de/t8kJ/raw) without any > >> > reasons. I should add that the box hanged a week ago and I had to do= a > >> > hard reboot, I have the feeling that it's somewhat related to this > >> > problem .. > >> > > >> I suggest you try to debug these 2 problems separately. Did you get a > >> chance to look at kgdb to find the culprit process as I suggested > >> below? > > > > I tried what you suggested, but I get a "No struct type named inpcb" > > Any idea ? :) >=20 > Is your kernel not build with debug symbols? Can you share your kernconf? indeed.. I always disable debug symbols on the production machines this is my kernconf: https://dpaste.de/8658/raw maybe I should give 10.1-RELEASE a try now that it's out soon .. >=20 > I have following in my kernconf: >=20 > makeoptions DEBUG=3D-g > options KDB > options KDB_TRACE > options DDB >=20 > cheers, > Hiren >=20 > >> > >> cheers, > >> Hiren > >> >> > > >> >> > I also wonder if something has been fixed in -STABLE in this area= .. > >> >> > > >> >> > (please keep me in CC as I'm not subscribed on freebsd-net@ an > >> >> > freebsd-stable@) > >> >> > > >> >> > -- > >> >> > Julien Cigar > >> >> > Belgian Biodiversity Platform (http://www.biodiversity.be) > >> >> > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23= C0 > >> >> > No trees were killed in the creation of this message. > >> >> > However, many electrons were terribly inconvenienced. > >> >> > > >> >> > > >> >> > ---------- Forwarded message ---------- > >> >> > From: Julien Cigar <jcigar@ulb.ac.be> > >> >> > To: freebsd-questions@freebsd.org > >> >> > Cc: > >> >> > Date: Thu, 2 Oct 2014 11:52:06 +0200 > >> >> > Subject: Listen queue overflow: 8 already in queue awaiting accep= tance > >> >> > Hello, > >> >> > > >> >> > I'm running 10-RELEASE on a HP Proliant DL160 Gen8 and I'm seeing= the > >> >> > following in my kernel logs: > >> >> > sonewconn: pcb 0xfffff8010e561310: Listen queue overflow: 8 alrea= dy in > >> >> > queue awaiting acceptance > >> >> > >> >> This usually means the application is not keeping up with the incom= ing > >> >> connections. > >> >> > > >> >> > I already raised kern.ipc.soacceptqueue to 1024 and netstat -naA= | grep > >> >> > "fffff8010e561310" returns nothing > >> >> > >> >> This is the usual way of finding the culprit process. If this doesn= 't > >> >> return anything, it probably means that it is a short-lived process. > >> >> > >> >> Here is an example of what you could do: > >> >> > >> >> sonewconn: pcb 0xfffff8008f40cb10: Listen queue overflow: 1 already= in queue > >> >> awaiting acceptance > >> >> > >> >> From kgdb, > >> >> (kgdb) p ((struct inpcb *)0xfffff8008f40cb10)->inp_inc > >> >> $3 =3D {inc_flags =3D 0 '\0', inc_len =3D 0 '\0', inc_fibnum =3D 0,= inc_ie =3D {ie_fport > >> >> =3D 0, ie_lport =3D 10295, ie_dependfaddr =3D { > >> >> ie46_foreign =3D {ia46_pad32 =3D {0, 0, 0}, ia46_addr4 =3D {s= _addr =3D 0}}, > >> >> ie6_foreign =3D {__u6_addr =3D { > >> >> __u6_addr8 =3D '\0' <repeats 15 times>, __u6_addr16 =3D {= 0, 0, 0, 0, 0, > >> >> 0, 0, 0}, __u6_addr32 =3D {0, 0, 0, 0}}}}, > >> >> ie_dependladdr =3D {ie46_local =3D {ia46_pad32 =3D {0, 0, 0}, i= a46_addr4 =3D > >> >> {s_addr =3D 0}}, ie6_local =3D {__u6_addr =3D { > >> >> __u6_addr8 =3D '\0' <repeats 15 times>, __u6_addr16 =3D {= 0, 0, 0, 0, 0, > >> >> 0, 0, 0}, __u6_addr32 =3D {0, 0, 0, 0}}}}}} > >> >> > >> >> Here, ie_lport =3D 10295 which is in n/w byte order and converting = it to host > >> >> byte order, 10295 -> 0x2837 and swapping them gives us 0x3728 which= is 14120. > >> >> > >> >> Now, use sockstat to find out what process is on that port: > >> >> > >> >> $ sockstat -l | grep 14120 > >> >> > >> >> cheers, > >> >> Hiren > >> > > >> > -- > >> > Julien Cigar > >> > Belgian Biodiversity Platform (http://www.biodiversity.be) > >> > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > >> > No trees were killed in the creation of this message. > >> > However, many electrons were terribly inconvenienced. > > > > -- > > Julien Cigar > > Belgian Biodiversity Platform (http://www.biodiversity.be) > > PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 > > No trees were killed in the creation of this message. > > However, many electrons were terribly inconvenienced. --=20 Julien Cigar Belgian Biodiversity Platform (http://www.biodiversity.be) PGP fingerprint: EEF9 F697 4B68 D275 7B11 6A25 B2BB 3710 A204 23C0 No trees were killed in the creation of this message. However, many electrons were terribly inconvenienced. --VSaCG/zfRnOiPJtU Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQIcBAABCgAGBQJUM8zLAAoJEAi2KiTKQR5pRKUQALrPkS3bp0m0eQsZW5x0kH8L B3Mvkenq2GlaK8F5WtrQCwMsjoTyDrI2AOZ959nal2zOKY+ZxlF9b0F+Umjz25Pc XkgdMJdCKcQ/LRicfz2EzYUzfygcYA4DB8OOc3O+12Oaz/J75s/AeFUoo54gjRb8 ylAlfGWkos+dr6xp/Sws6k+G2WYTYNehirdrx+RsKg+6wDvlOH47gMvqiIb8c3FR RvSZ3BxDRpLZRkkNMlPgs/fAXxuK80mwgTEW3DAomr50GyuN15Dnr9B6Hoybyvfr ngCs0gXVyaK06RLnGTS9jn9FAeOWPOd4ok6ChRZMPx/3FwRhpfK5zHcNGpowq3u1 zbYgRCuSAH+QTY+nfOW4O/jjaQ5goGOiOcSHtr0azHaMNW8KfHmXu/AeG+xqPP7y tYXj4Xx4nl5kmhyscROcRjng+cRA1EXWnuU+moUhgaDU2sBIv8o/JvjY642poNHu njjr37Ya7o7nFZ4iQ73PWgG20bQOlFp0Knk8hCkIWjohQVRiQIZ3tZw7VCOK3dI3 DuwdGI13Oe54tJIUG1iMlxgOC0ZyDK9EKJeIZ9T4odrRO+kYcQbOjMj3Mv3Qj5/8 O20+43aa27XgwHEOyycNRycA4Czf0so8UKMpFpmw9u3m5jQD5r7MC1xu67PLBmEc mBbrpxpTcPVabVnFh0WE =+Pub -----END PGP SIGNATURE----- --VSaCG/zfRnOiPJtU--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20141007112147.GI63350>