From owner-freebsd-net@FreeBSD.ORG Fri May 3 14:40:12 2013 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 724A2C34 for ; Fri, 3 May 2013 14:40:12 +0000 (UTC) (envelope-from eric@vangyzen.net) Received: from aussmtpmrkps320.us.dell.com (aussmtpmrkps320.us.dell.com [143.166.224.254]) by mx1.freebsd.org (Postfix) with ESMTP id 4633E14E4 for ; Fri, 3 May 2013 14:40:11 +0000 (UTC) X-Loopcount0: from 64.238.244.148 X-IronPort-AV: E=Sophos;i="4.87,605,1363150800"; d="scan'208";a="27221275" Message-ID: <5183CC06.9020806@vangyzen.net> Date: Fri, 3 May 2013 09:39:02 -0500 From: Eric van Gyzen User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130413 Thunderbird/17.0.5 MIME-Version: 1.0 To: Richard Sharpe Subject: Re: Seeing EINVAL from writev on 8.0 to a non-blocking socket even though the data seems to hit the wire References: <5181ECDF.1040905@mu.org> <51827DAA.2020009@vangyzen.net> In-Reply-To: Content-Type: text/plain; charset="Big5" Content-Transfer-Encoding: 7bit Cc: freebsd-net@freebsd.org, Alfred Perlstein X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 03 May 2013 14:40:12 -0000 On 05/02/2013 19:00, Richard Sharpe wrote: > On Thu, May 2, 2013 at 7:52 AM, Eric van Gyzen wrote: >> On 05/02/2013 08:48, Richard Sharpe wrote: >>> On Wed, May 1, 2013 at 9:34 PM, Alfred Perlstein wrote: >>>> On 5/1/13 8:03 PM, Richard Sharpe wrote: >>>>> Hi folks, >>>>> >>>>> I am checking to see if there are any known bugs with respect to this >>>>> in FreeBSD 8.0. >>>>> >>>>> Situation is that Samba 3.6.6 uses writev to a non-blocking socket to >>>>> get the SMB2 requests on the wire. >>>>> >>>>> Intermittently, we see the writev return EINVAL even though the data >>>>> has gotten on the wire. This I have verified by grabbing a capture and >>>>> comparing the SMB Sequence number in the last outgoing packet on the >>>>> wire vs the in-memory contents when we get EINVAL. >>>>> >>>>> Sometimes it occurs on a four-element IOVEC, sometimes we get EAGAIN >>>>> on the four-element IOVEC and then we get EINVAL when retrying on a >>>>> smaller IOVEC. >>>>> >>>>> Where should I look to check if there is some path where this might be >>>>> happening? Is this even the correct mailing list? >>>>> >>>> What does the iovec look like when you get EINVAL? Can you sanity check >>>> it? Is there anything special about it? (zero length vecs?) >>>> >>>> I think there are a few "maxvals" that if overrun cause EINVAL to be >>>> returned. example is if your iovec is somehow huge or has many, many >>>> elements. >>> Can anyone tell me the call graph down to the TCP code? >>> >> writev kern/sys_generic.c >> kern_writev >> dofilewrite >> fo_write in sys/file.h >> soo_write in kern/sys_socket.c >> sosend in kern/uipc_socket.c >> sosend_generic >> tcp_usr_send in netinet/tcp_usrreq.c > Is there a tool that generates call graphs? I'm not aware of one that works in the kernel--other than the kernel itself, of course. With DDB compiled in, you could set a breakpoint on, say, tcp_output, and show the call stack with bt. Also, take a look at stack(9). > I have been able to demonstrate that I am getting EINVAL returned from > writev kern/sys_generic.c, kern_writev, dofilewrite and soo_write, > but when I add printfs to sosend/sosend_generic it becomes very hard > to provoke this problem. So, either relocating code or changing the timing has changed the behavior--a Heisenbug. If your code looks like this: if (error == EINVAL) printf("you are here\n"); You might add __predict_false, like this: if (__predict_false(error == EINVAL)) printf("you are here\n"); That /might/ reduce the impact on runtime behavior. > I wonder if the compiler is generating code that allows some functions > to leave variables in registers but they are not treated as volatile.