Date: Tue, 8 Aug 2017 13:33:43 +0200 From: Julien Charbon <jch@freebsd.org> To: Hans Petter Selasky <hps@selasky.org>, Ben RUBSON <ben.rubson@gmail.com>, FreeBSD Net <freebsd-net@freebsd.org>, hiren <hiren@strugglingcoder.info>, Slawa Olhovchenkov <slw@zxy.spb.ru>, FreeBSD Stable <freebsd-stable@freebsd.org> Subject: Re: mlx4en, timer irq @100%... (11.0 stuck on high network load ???) Message-ID: <0a5787c5-8a53-ab09-971a-dc1cd5f3aca0@freebsd.org> In-Reply-To: <c05c2b1c-b5a8-c39c-6dff-e6cc0d8642bf@selasky.org> References: <BF3A3E47-A726-49FB-B83F-666EFC5C0FF1@gmail.com> <4DF74CB8-23D2-4CCF-B699-5B86DAEA65E5@gmail.com> <40602CEA-D417-4E5B-8C68-916958D49A0B@gmail.com> <9c306f10-7c05-d28d-e551-a930603aaafa@selasky.org> <896dd782-cb2c-0259-65d1-b00daae452de@FreeBSD.org> <0DB9F6FF-8BC9-48F5-B359-AC1905B9EB06@gmail.com> <7f14c95d-1ef8-bf82-c469-e6566c3aba66@selasky.org> <76A5EE7E-1D2E-46B4-86F1-F219C3DCE6EA@gmail.com> <e6f9df1c-8b55-8a3b-9f44-e67c26561543@selasky.org> <4C91C6E5-0725-42E7-9813-1F3ACF3DDD6E@gmail.com> <5840c25e-7472-3276-6df9-1ed4183078ad@selasky.org> <2ADA8C57-2C2D-4F97-9F0B-82D53EDDC649@gmail.com> <061cdf72-6285-8239-5380-58d9d19a1ef7@selasky.org> <92BEE83D-498F-47D5-A53C-39DCDC00A0FD@gmail.com> <5d8960d8-e1ff-8719-320f-d3ae84054714@selasky.org> <6B4A35F7-5694-4945-9575-19ADB678F9FA@gmail.com> <297a784a-3d80-b1a6-652e-a78621fe5a8b@selasky.org> <3ECCFBF1-18D9-4E33-8F39-0C366C3BB8B4@gmail.com> <c05c2b1c-b5a8-c39c-6dff-e6cc0d8642bf@selasky.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,
On 8/8/17 10:31 AM, Hans Petter Selasky wrote:
> On 08/08/17 10:06, Ben RUBSON wrote:
>>> On 08 Aug 2017, at 10:02, Hans Petter Selasky <hps@selasky.org> wrote:
>>>
>>> On 08/08/17 10:00, Ben RUBSON wrote:
>>>> kgdb) print *twq_2msl.tqh_first
>>>> $2 = {
>>>> tw_inpcb = 0xfffff8031c570740,
>>>
>>> print *twq_2msl.tqh_first->tw_inpcb
>>
>
> Here is the conclusion:
>
> The following code is going in an infinite loop:
>
>
>> for (;;) {
>> TW_RLOCK(V_tw_lock);
>> tw = TAILQ_FIRST(&V_twq_2msl);
>> if (tw == NULL || (!reuse && (tw->tw_time - ticks) >
>> 0)) {
>> TW_RUNLOCK(V_tw_lock);
>> break;
>> }
>> KASSERT(tw->tw_inpcb != NULL, ("%s: tw->tw_inpcb ==
>> NULL",
>> __func__));
>>
>> inp = tw->tw_inpcb;
>> in_pcbref(inp);
>> TW_RUNLOCK(V_tw_lock);
>>
>> if (INP_INFO_TRY_RLOCK(&V_tcbinfo)) {
>>
>> INP_WLOCK(inp);
>> tw = intotw(inp);
>> if (in_pcbrele_wlocked(inp)) {
>
> in_pcbrele_wlocked() returns (1) because INP_FREED (16) is set in
> inp->inp_flags2. I guess you have invariants disabled, because the
> KASSERT() below should have caused a panic.
>
>> KASSERT(tw == NULL, ("%s: held last inp "
>> "reference but tw not NULL",
>> __func__));
>> INP_INFO_RUNLOCK(&V_tcbinfo);
>> continue;
>> }
>
> This is a regression issue after:
>
>> commit 5630210a7f1dbbd903b77b2aef939cd47c63da58
>> Author: jch <jch@FreeBSD.org>
>> Date: Thu Oct 30 08:53:56 2014 +0000
>>
>> Fix a race condition in TCP timewait between tcp_tw_2msl_reuse() and
>> tcp_tw_2msl_scan(). This race condition drives unplanned timewait
>> timeout cancellation. Also simplify implementation by holding inpcb
>> reference and removing tcptw reference counting.
>
> Suggested fix attached.
I agree we your conclusion. Just for the record, more precisely this
regression seems to have been introduced with:
commit b02d40ddcda08b51a49e5667e6808f5dc5ec0472
Author: fabient <fabient@FreeBSD.org>
Date: Wed Nov 25 14:45:43 2015 +0000
The r241129 description was wrong that the scenario is possible
only for read locks on pcbs. The same race can happen with write
lock semantics as well.
The race scenario:
- Two threads (1 and 2) locate pcb with writer semantics
(INPLOOKUP_WLOCKPCB)
and do in_pcbref() on it.
- 1 and 2 both drop the inp hash lock.
- Another thread (3) grabs the inp hash lock. Then it runs in_pcbfree(),
which wlocks the pcb. They must happen faster than 1 or 2 come
INP_WLOCK()!
- 1 and 2 congest in INP_WLOCK().
- 3 does in_pcbremlists(), drops hash lock, and runs
in_pcbrele_wlocked(),
which doesn't free the pcb due to two references on it.
Then it unlocks the pcb.
- 1 (or 2) gets wlock on the pcb, runs in_pcbrele_wlocked(), which
doesn't
report inp as freed, due to 2 (or 1) still helding extra reference
on it.
The thread tries to do smth with a disconnected pcb and crashes.
Submitted by: emeric.poupon@stormshield.eu
Reviewed by: gleb@
MFC after: 1 week
Sponsored by: Stormshield
Tested by: Cassiano Peixoto, Stormshield
Notes:
svn path=/head/; revision=291301
Before this change in_pcbrele_wlocked() returned 1 only on the last
reference which is what tcp_tw_2msl_scan() currently expects.
Thus good catch, and your patch looks good. I am going to just verify
the other in_pcbrele_wlocked() calls in TCP stack.
Thanks.
--
Julien
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?0a5787c5-8a53-ab09-971a-dc1cd5f3aca0>