Date: Fri, 14 Oct 2016 11:48:38 +0200 From: Julien Charbon <jch@freebsd.org> To: Slawa Olhovchenkov <slw@zxy.spb.ru> Cc: Konstantin Belousov <kostikbel@gmail.com>, freebsd-stable@FreeBSD.org, hiren panchasara <hiren@strugglingcoder.info> Subject: Re: 11.0 stuck on high network load Message-ID: <9f2eb420-897f-e231-834e-c8b256c81130@freebsd.org> In-Reply-To: <20161014093546.GN57714@zxy.spb.ru> References: <20161012121322.GB57876@zxy.spb.ru> <62d8861c-673e-6d86-e96e-751399e505e5@freebsd.org> <20161012130103.GD57714@zxy.spb.ru> <e8a46471-576d-e074-8a50-5c316fb98bce@freebsd.org> <20161012154229.GC57876@zxy.spb.ru> <bc6810d8-af4c-1716-cbc7-e64819b78ccd@freebsd.org> <20161013143825.GK57714@zxy.spb.ru> <33ab0bfc-7009-95a7-7752-c2c439092e85@freebsd.org> <20161013151715.GL57714@zxy.spb.ru> <be331d47-dafc-10be-12dd-2b1bf9679a90@freebsd.org> <20161014093546.GN57714@zxy.spb.ru>
next in thread | previous in thread | raw e-mail | index | archive | help
This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--oQtGu5WxAv55Vq1jCjCv1IBIOsQGk4tTP
Content-Type: multipart/mixed; boundary="OqVWeOCDacKtisbe40sROktEM7EB10mp7";
protected-headers="v1"
From: Julien Charbon <jch@freebsd.org>
To: Slawa Olhovchenkov <slw@zxy.spb.ru>
Cc: Konstantin Belousov <kostikbel@gmail.com>, freebsd-stable@FreeBSD.org,
hiren panchasara <hiren@strugglingcoder.info>
Message-ID: <9f2eb420-897f-e231-834e-c8b256c81130@freebsd.org>
Subject: Re: 11.0 stuck on high network load
References: <20161012121322.GB57876@zxy.spb.ru>
<62d8861c-673e-6d86-e96e-751399e505e5@freebsd.org>
<20161012130103.GD57714@zxy.spb.ru>
<e8a46471-576d-e074-8a50-5c316fb98bce@freebsd.org>
<20161012154229.GC57876@zxy.spb.ru>
<bc6810d8-af4c-1716-cbc7-e64819b78ccd@freebsd.org>
<20161013143825.GK57714@zxy.spb.ru>
<33ab0bfc-7009-95a7-7752-c2c439092e85@freebsd.org>
<20161013151715.GL57714@zxy.spb.ru>
<be331d47-dafc-10be-12dd-2b1bf9679a90@freebsd.org>
<20161014093546.GN57714@zxy.spb.ru>
In-Reply-To: <20161014093546.GN57714@zxy.spb.ru>
--OqVWeOCDacKtisbe40sROktEM7EB10mp7
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable
Hi,
On 10/14/16 11:35 AM, Slawa Olhovchenkov wrote:
> On Thu, Oct 13, 2016 at 06:14:29PM +0200, Julien Charbon wrote:
>> On 10/13/16 5:17 PM, Slawa Olhovchenkov wrote:
>>> On Thu, Oct 13, 2016 at 05:06:00PM +0200, Julien Charbon wrote:
>>>
>>>>>> will give you that trace in the core, and without INVARIANT then i=
t is
>>>>>> better to use dtrace:
>>>>>>
>>>>>> $ cat tcp-twstart-dropped.d
>>>>>> fbt::tcp_twstart:entry
>>>>>> /args[0]->t_inpcb->inp_flags & 0x04000000/
>>>>>> {
>>>>>> stack();
>>>>>> printf("INP_DROPPED in tcp_twstart: %x", args[0]->t_inpcb->inp_f=
lags);
>>>>>> }
>>>>>
>>>>> Same code may be insert there too, IMHO.
>>>>
>>>> Hmm, I don't think so:
>>>>
>>>> - If you have INVARIANT, the kernel will panic in tcp_twstart() or
>>>> tcp_detach() and you will have everything you need to debug.
>>>> - If you don't, dtrace is the right tool to use in all cases anyway=
=2E
>>>
>>> dtrace don't executed in may case w/ diagnostic "dtrace: processing
>>> aborted: Abort due to systemic unresponsiveness". This is for
>>> tcp_close. May be tcp_twstart will be more successuful, may be not.
>>
>> It does and will.
>>
>>> Also, using dtrace too complex in production (need complex startup
>>> under screen and capture output) and for many peoples.
>>> kdb_backtrace() have too less administrative overhead.
>>
>> I still think it is overkill. The main goal of this change is to fix=
a
>> quite tricky and old TCP stack locking issue. Let's try to do that
>> first, it is complex enough by itself.
>>
>> Once the fix is validated and pushed, feel free to propose your own
>> patch/review to add kdb_backtrace(), log(), etc.. to get other devs
>> point of view.
>>
>> I don't remember who said: "Never ever optimize error cases"...
>=20
> This is not optimeze error cases, this is error recovery and
> diagnostic of error cases in other subsystems.
Sure, I guess this quote is more geared toward: "Always spend 50x more
time on improving the main path than the error path".
> Currently FreeBSD internals too complex for just always trust on
> correct of other subsystem or do panic on any incosystency.
>=20
> INVARIANTS too expensive now (20Gbit drops to 8Gbits).
I do agree. I am not expert enough to see all the side effects of
calling kdb_backtrace() from the TCP stack, might be way too slow,
tricky in interruption context, etc. You can see that kdb_backtrace()
is rarely called in the kernel source. That's why it is better if you
propose a review on adding this line to get comments from other devs on
just this question.
> PS: I am applay patch. Wait till monday.
>=20
> Thanks very match for this hard work!
No problem, thanks for your time. But it is not over yet: We have to
wait for final test.
--
Julien
--OqVWeOCDacKtisbe40sROktEM7EB10mp7--
--oQtGu5WxAv55Vq1jCjCv1IBIOsQGk4tTP
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - https://gpgtools.org
iQEcBAEBCgAGBQJYAKn6AAoJEKVlQ5Je6dhxz3YH/3ucJr741f6lrt7Pk367/lX+
PwduNL80rE+WfLsp47UCbk3yUinpWjaTk5VzjA/IUKcQ8SsNS7JLbaEfhCL3WYfI
RhJDdiwU3rljVMhAFA58NJgOwXX+RVlA/kw5ba38FGxUO9xGMwx3eZd8H/d3tz0r
rFVf1ng+RnTVuTj8Lhm/REiEvx9KhIT/sVW2iQquHriK582Js7XQ9ZBi5UfbkalQ
ueYAsM+5PIjSVKShSL4ZaE5HSZ0sqTLVxIXcpPUmweH3wzjOvW5uwT+XWxtwFCur
hLTlTqi4TGjmT8NhnWd/hAglsjQ0VuPLoA9i6uC+uto5sqqqdAB/VvtF9P2mzpU=
=BTqZ
-----END PGP SIGNATURE-----
--oQtGu5WxAv55Vq1jCjCv1IBIOsQGk4tTP--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?9f2eb420-897f-e231-834e-c8b256c81130>