Date: Fri, 3 May 2013 19:01:18 -0700 From: Richard Sharpe <realrichardsharpe@gmail.com> To: Eric van Gyzen <eric@vangyzen.net> Cc: freebsd-net@freebsd.org, Alfred Perlstein <bright@mu.org> Subject: Re: Seeing EINVAL from writev on 8.0 to a non-blocking socket even though the data seems to hit the wire Message-ID: <CACyXjPwz1R9q22Th6NOLYAdBNk-HDW67vyTTtc2_E3UggEN1gA@mail.gmail.com> In-Reply-To: <CACyXjPxhUXC3ONRxv-gTwYSN8ebD8chVhv0LmRBDDVbw7rKSLw@mail.gmail.com> References: <CACyXjPwojx1vBo-7bDmN=Pjc2Qp3mRd3Ek2FUjLR_4DC=aUnWA@mail.gmail.com> <5181ECDF.1040905@mu.org> <CACyXjPy8ctxs1vG0KPLHtaJxrM_YTs6XfLEbhQUBKTkZAjewzA@mail.gmail.com> <51827DAA.2020009@vangyzen.net> <CACyXjPzML0b5VyCNnc3vCJDDM5JZaRdeTLc9v29iBAK_-1eYyw@mail.gmail.com> <5183CC06.9020806@vangyzen.net> <CACyXjPxhUXC3ONRxv-gTwYSN8ebD8chVhv0LmRBDDVbw7rKSLw@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, May 3, 2013 at 10:18 AM, Richard Sharpe
<realrichardsharpe@gmail.com> wrote:
> On Fri, May 3, 2013 at 7:39 AM, Eric van Gyzen <eric@vangyzen.net> wrote:
>> On 05/02/2013 19:00, Richard Sharpe wrote:
>>> On Thu, May 2, 2013 at 7:52 AM, Eric van Gyzen <eric@vangyzen.net> wrot=
e:
>>>> On 05/02/2013 08:48, Richard Sharpe wrote:
>>>>> On Wed, May 1, 2013 at 9:34 PM, Alfred Perlstein <bright@mu.org> wrot=
e:
>>>>>> On 5/1/13 8:03 PM, Richard Sharpe wrote:
>>>>>>> Hi folks,
>>>>>>>
>>>>>>> I am checking to see if there are any known bugs with respect to th=
is
>>>>>>> in FreeBSD 8.0.
>>>>>>>
>>>>>>> Situation is that Samba 3.6.6 uses writev to a non-blocking socket =
to
>>>>>>> get the SMB2 requests on the wire.
>>>>>>>
>>>>>>> Intermittently, we see the writev return EINVAL even though the dat=
a
>>>>>>> has gotten on the wire. This I have verified by grabbing a capture =
and
>>>>>>> comparing the SMB Sequence number in the last outgoing packet on th=
e
>>>>>>> wire vs the in-memory contents when we get EINVAL.
>>>>>>>
>>>>>>> Sometimes it occurs on a four-element IOVEC, sometimes we get EAGAI=
N
>>>>>>> on the four-element IOVEC and then we get EINVAL when retrying on a
>>>>>>> smaller IOVEC.
>>>>>>>
>>>>>>> Where should I look to check if there is some path where this might=
be
>>>>>>> happening? Is this even the correct mailing list?
>>>>>>>
>>>>>> What does the iovec look like when you get EINVAL? Can you sanity ch=
eck
>>>>>> it? Is there anything special about it? (zero length vecs?)
>>>>>>
>>>>>> I think there are a few "maxvals" that if overrun cause EINVAL to be
>>>>>> returned. example is if your iovec is somehow huge or has many, many
>>>>>> elements.
>>>>> Can anyone tell me the call graph down to the TCP code?
>>>>>
>>>> writev kern/sys_generic.c
>>>> kern_writev
>>>> dofilewrite
>>>> fo_write in sys/file.h
>>>> soo_write in kern/sys_socket.c
>>>> sosend in kern/uipc_socket.c
>>>> sosend_generic
>>>> tcp_usr_send in netinet/tcp_usrreq.c
>>> Is there a tool that generates call graphs?
>>
>> I'm not aware of one that works in the kernel--other than the kernel
>> itself, of course. With DDB compiled in, you could set a breakpoint on,
>> say, tcp_output, and show the call stack with bt.
>>
>> Also, take a look at stack(9).
>>
>>> I have been able to demonstrate that I am getting EINVAL returned from
>>> writev kern/sys_generic.c, kern_writev, dofilewrite and soo_write,
>>> but when I add printfs to sosend/sosend_generic it becomes very hard
>>> to provoke this problem.
>>
>> So, either relocating code or changing the timing has changed the
>> behavior--a Heisenbug.
>>
>> If your code looks like this:
>>
>> if (error =3D=3D EINVAL)
>> printf("you are here\n");
>>
>> You might add __predict_false, like this:
>>
>> if (__predict_false(error =3D=3D EINVAL))
>> printf("you are here\n");
>>
>> That /might/ reduce the impact on runtime behavior.
>
> Thanks for that. The problem does not appear to be in the TCP or IP
> layers. Rather, it appears to be in the ixgbe driver.
>
> The problem takes a little more effort to provoke, but simple printfs
> are doing the job so far.
The version of the ixgbe driver we are using seems to set the max size
of a dma element to 65535 (IXGBE_TSO_SIZE) and, even though large
numbers of iovecs are sent where the last element is 65536 bytes in
size, sometimes this causes EINVAL to be returned ...
--=20
Regards,
Richard Sharpe
(=A6=F3=A5H=B8=D1=BC~=A1H=B0=DF=A6=B3=A7=F9=B1d=A1C--=B1=E4=BE=DE)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACyXjPwz1R9q22Th6NOLYAdBNk-HDW67vyTTtc2_E3UggEN1gA>