Date: Fri, 3 May 2013 10:18:50 -0700 From: Richard Sharpe <realrichardsharpe@gmail.com> To: Eric van Gyzen <eric@vangyzen.net> Cc: freebsd-net@freebsd.org, Alfred Perlstein <bright@mu.org> Subject: Re: Seeing EINVAL from writev on 8.0 to a non-blocking socket even though the data seems to hit the wire Message-ID: <CACyXjPxhUXC3ONRxv-gTwYSN8ebD8chVhv0LmRBDDVbw7rKSLw@mail.gmail.com> In-Reply-To: <5183CC06.9020806@vangyzen.net> References: <CACyXjPwojx1vBo-7bDmN=Pjc2Qp3mRd3Ek2FUjLR_4DC=aUnWA@mail.gmail.com> <5181ECDF.1040905@mu.org> <CACyXjPy8ctxs1vG0KPLHtaJxrM_YTs6XfLEbhQUBKTkZAjewzA@mail.gmail.com> <51827DAA.2020009@vangyzen.net> <CACyXjPzML0b5VyCNnc3vCJDDM5JZaRdeTLc9v29iBAK_-1eYyw@mail.gmail.com> <5183CC06.9020806@vangyzen.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Fri, May 3, 2013 at 7:39 AM, Eric van Gyzen <eric@vangyzen.net> wrote:
> On 05/02/2013 19:00, Richard Sharpe wrote:
>> On Thu, May 2, 2013 at 7:52 AM, Eric van Gyzen <eric@vangyzen.net> wrote=
:
>>> On 05/02/2013 08:48, Richard Sharpe wrote:
>>>> On Wed, May 1, 2013 at 9:34 PM, Alfred Perlstein <bright@mu.org> wrote=
:
>>>>> On 5/1/13 8:03 PM, Richard Sharpe wrote:
>>>>>> Hi folks,
>>>>>>
>>>>>> I am checking to see if there are any known bugs with respect to thi=
s
>>>>>> in FreeBSD 8.0.
>>>>>>
>>>>>> Situation is that Samba 3.6.6 uses writev to a non-blocking socket t=
o
>>>>>> get the SMB2 requests on the wire.
>>>>>>
>>>>>> Intermittently, we see the writev return EINVAL even though the data
>>>>>> has gotten on the wire. This I have verified by grabbing a capture a=
nd
>>>>>> comparing the SMB Sequence number in the last outgoing packet on the
>>>>>> wire vs the in-memory contents when we get EINVAL.
>>>>>>
>>>>>> Sometimes it occurs on a four-element IOVEC, sometimes we get EAGAIN
>>>>>> on the four-element IOVEC and then we get EINVAL when retrying on a
>>>>>> smaller IOVEC.
>>>>>>
>>>>>> Where should I look to check if there is some path where this might =
be
>>>>>> happening? Is this even the correct mailing list?
>>>>>>
>>>>> What does the iovec look like when you get EINVAL? Can you sanity che=
ck
>>>>> it? Is there anything special about it? (zero length vecs?)
>>>>>
>>>>> I think there are a few "maxvals" that if overrun cause EINVAL to be
>>>>> returned. example is if your iovec is somehow huge or has many, many
>>>>> elements.
>>>> Can anyone tell me the call graph down to the TCP code?
>>>>
>>> writev kern/sys_generic.c
>>> kern_writev
>>> dofilewrite
>>> fo_write in sys/file.h
>>> soo_write in kern/sys_socket.c
>>> sosend in kern/uipc_socket.c
>>> sosend_generic
>>> tcp_usr_send in netinet/tcp_usrreq.c
>> Is there a tool that generates call graphs?
>
> I'm not aware of one that works in the kernel--other than the kernel
> itself, of course. With DDB compiled in, you could set a breakpoint on,
> say, tcp_output, and show the call stack with bt.
>
> Also, take a look at stack(9).
>
>> I have been able to demonstrate that I am getting EINVAL returned from
>> writev kern/sys_generic.c, kern_writev, dofilewrite and soo_write,
>> but when I add printfs to sosend/sosend_generic it becomes very hard
>> to provoke this problem.
>
> So, either relocating code or changing the timing has changed the
> behavior--a Heisenbug.
>
> If your code looks like this:
>
> if (error =3D=3D EINVAL)
> printf("you are here\n");
>
> You might add __predict_false, like this:
>
> if (__predict_false(error =3D=3D EINVAL))
> printf("you are here\n");
>
> That /might/ reduce the impact on runtime behavior.
Thanks for that. The problem does not appear to be in the TCP or IP
layers. Rather, it appears to be in the ixgbe driver.
The problem takes a little more effort to provoke, but simple printfs
are doing the job so far.
--=20
Regards,
Richard Sharpe
(=A6=F3=A5H=B8=D1=BC~=A1H=B0=DF=A6=B3=A7=F9=B1d=A1C--=B1=E4=BE=DE)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CACyXjPxhUXC3ONRxv-gTwYSN8ebD8chVhv0LmRBDDVbw7rKSLw>