Date: Tue, 03 Jun 2014 12:42:46 +0200 From: Julien Charbon <jcharbon@verisign.com> To: freebsd-net@freebsd.org Subject: Re: [rfc] TCP timewait and credential handling - why would we get a TW with no credential? Message-ID: <538DA6A6.6040201@verisign.com> In-Reply-To: <CAJ-Vmo=JwL7gFE2xfXa9cx4rYdkDf87qoy32H%2BSguRPogT2p=w@mail.gmail.com> References: <CAJ-Vmo=JwL7gFE2xfXa9cx4rYdkDf87qoy32H%2BSguRPogT2p=w@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Adrian, On 01/06/14 17:21, Adrian Chadd wrote: > I've been seeing this panic under high very short-term connection TCP > throughput tests (30,000 connections a second) with SO_REUSEPORT: > > Current language: auto; currently minimal > > (kgdb) frame 11 > > #11 0xffffffff80a6bdf1 in tcp_twclose (tw=0xfffff801348b5780, reuse=0) > at /usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/tcp_timewait.c:644 > > 644 crfree(tw->tw_cred); > > (kgdb) frame 11 > > #11 0xffffffff80a6bdf1 in tcp_twclose (tw=0xfffff801348b5780, reuse=0) > at /usr/home/adrian/git/github/erikarn/freebsd/sys/netinet/tcp_timewait.c:644 > > 644 crfree(tw->tw_cred); > > (kgdb) print tw > > $1 = (struct tcptw *) 0xfffff801348b5780 > > (kgdb) print tw->tw_cred > > $2 = (struct ucred *) 0x0 > [...] > > ... there's only one path to deleting the credentials and that's via > tcp_twclose() -> tcp_tw_2msl_stop(). > > Has anyone seen this kind of problem before? Interesting question: I already got this issue also under short-lived high connection rate (~120k connections per second) and using SO_REUSEPORT _however_ it was with our TCP scaling patches applied (thus it might be a red herring for you) and I fixed it with a: tcp_tw_2msl_stop(): ... if (tw->tw_cred != NULL) crfree(tw->tw_cred); https://github.com/verisign/freebsd/commit/918fe94ad3f3ccd1f6bf577e7e79a138378262a0#diff-4f0c0d9222bfb69f2f358113d424cf89R610 At that time I thought you could create a tw with its tw->tw_cred field set to NULL and then tcp_tw_2msl_stop() simply missed a sanity check before calling crfree(). Now I re-checked how tw->tw_cred is set, its seems impossible to create a tw with tw->tw_cred = NULL (aha!). As this tw_cred field seems well protected by both INP_INFO_WLOCK and INP_WLOCK from race conditions, I am currently checking if something could inadvertently overwrite tw_cred in struct tcptw. My 2 cents. -- Julien
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?538DA6A6.6040201>