From owner-freebsd-stable@FreeBSD.ORG Fri Mar 21 02:45:17 2014 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0E6E92DF; Fri, 21 Mar 2014 02:45:17 +0000 (UTC) Received: from esa-annu.net.uoguelph.ca (esa-annu.mail.uoguelph.ca [131.104.91.36]) by mx1.freebsd.org (Postfix) with ESMTP id A08FF9C0; Fri, 21 Mar 2014 02:45:16 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AqUEAD2nK1ODaFve/2dsb2JhbABZg0FXgwe4FIZjUYErdIIlAQEBAwEBAQEgBCcgCxsYAgINGQIpAQkmBgEHBwQBCBQEh1AIDa0XomAXgSmMZQYBARsBMweCb4FJBJVyhAmQf4NJITF8CBci X-IronPort-AV: E=Sophos;i="4.97,700,1389762000"; d="scan'208";a="107529372" Received: from muskoka.cs.uoguelph.ca (HELO zcs3.mail.uoguelph.ca) ([131.104.91.222]) by esa-annu.net.uoguelph.ca with ESMTP; 20 Mar 2014 22:45:15 -0400 Received: from zcs3.mail.uoguelph.ca (localhost.localdomain [127.0.0.1]) by zcs3.mail.uoguelph.ca (Postfix) with ESMTP id 8430BB3F19; Thu, 20 Mar 2014 22:45:15 -0400 (EDT) Date: Thu, 20 Mar 2014 22:45:15 -0400 (EDT) From: Rick Macklem To: Markus Gebert , Christopher Forgeron Message-ID: <429006400.647323.1395369915529.JavaMail.root@uoguelph.ca> In-Reply-To: Subject: Re: Network stack returning EFBIG? MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Originating-IP: [172.17.91.203] X-Mailer: Zimbra 7.2.1_GA_2790 (ZimbraWebClient - FF3.0 (Win)/7.2.1_GA_2790) Cc: jfv@freebsd.org, freebsd-net@freebsd.org, freebsd-stable@freebsd.org, wollman@bimajority.org X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 21 Mar 2014 02:45:17 -0000 Markus Gebert wrote: >=20 > On 20.03.2014, at 14:51, wollman@bimajority.org wrote: >=20 > > In article <21290.60558.750106.630804@hergotha.csail.mit.edu>, I > > wrote: > >=20 > >> Since we put this server into production, random network system > >> calls > >> have started failing with [EFBIG] or maybe sometimes [EIO]. I've > >> observed this with a simple ping, but various daemons also log the > >> errors: > >> Mar 20 09:22:04 nfs-prod-4 sshd[42487]: fatal: Write failed: File > >> too > >> large [preauth] > >> Mar 20 09:23:44 nfs-prod-4 nrpe[42492]: Error: Could not complete > >> SSL > >> handshake. 5 > >=20 > > I found at least one call stack where this happens and it does get > > returned all the way to userspace: > >=20 > > 17 15547 _bus_dmamap_load_buffer:return > > kernel`_bus_dmamap_load_mbuf_sg+0x5f > > kernel`bus_dmamap_load_mbuf_sg+0x38 > > kernel`ixgbe_xmit+0xcf > > kernel`ixgbe_mq_start_locked+0x94 > > kernel`ixgbe_mq_start+0x12a > > if_lagg.ko`lagg_transmit+0xc4 > > kernel`ether_output_frame+0x33 > > kernel`ether_output+0x4fe > > kernel`ip_output+0xd74 > > kernel`tcp_output+0xfea > > kernel`tcp_usr_send+0x325 > > kernel`sosend_generic+0x3f6 > > kernel`soo_write+0x5e > > kernel`dofilewrite+0x85 > > kernel`kern_writev+0x6c > > kernel`sys_write+0x64 > > kernel`amd64_syscall+0x5ea > > kernel`0xffffffff808443c7 >=20 > This looks pretty similar to what we=E2=80=99ve seen when we got EFBIG: >=20 > 3 28502 _bus_dmamap_load_buffer:return > kernel`_bus_dmamap_load_mbuf_sg+0x5f > kernel`bus_dmamap_load_mbuf_sg+0x38 > kernel`ixgbe_xmit+0xcf > kernel`ixgbe_mq_start_locked+0x94 > kernel`ixgbe_mq_start+0x12a > kernel`ether_output_frame+0x33 > kernel`ether_output+0x4fe > kernel`ip_output+0xd74 > kernel`rip_output+0x229 > kernel`sosend_generic+0x3f6 > kernel`kern_sendit+0x1a3 > kernel`sendit+0xdc > kernel`sys_sendto+0x4d > kernel`amd64_syscall+0x5ea > kernel`0xffffffff80d35667 >=20 > In our case it looks like some of the ixgbe tx queues get stuck, and > some don=E2=80=99t. You can test, wether your server shows the same sympt= oms > with this command: >=20 > # for CPU in {0..7}; do echo "CPU${CPU}"; cpuset -l ${CPU} ping -i > 0.5 -c 2 -W 1 10.0.0.1 | grep sendto; done >=20 > We also use 82599EB based ixgbe controllers on affected systems. >=20 > Also see these two threads on freebsd-net: >=20 > http://lists.freebsd.org/pipermail/freebsd-net/2014-February/037967.html > http://lists.freebsd.org/pipermail/freebsd-net/2014-March/038061.html >=20 > I have started the second one, and there are some more details of > what we were seeing in case you=E2=80=99re interested. >=20 > Then there is: >=20 > http://www.freebsd.org/cgi/query-pr.cgi?pr=3D183390 > and: > https://bugs.freenas.org/issues/4560 >=20 Well, the "before" printf() from my patch is indicating a packet > 65535 and that will definitely result in a EFBIG. (There is no way that m_defrag(= ) can squeeze > 64K into 32 MCLBYTES mbufs.) Note that the EFBIG will be returned by the call that dequeues this packet and tries to transmit it (not necessarily the one that generated/queued the packet). This was pointed out by Ryan in a previous discussion of this. The code snippet from sys/netinet/tcp_output.c looks pretty straightforward= : /* 772 =09* Limit a burst to t_tsomax minus IP, 773 =09* TCP and options length to keep ip->ip_len 774 =09* from overflowing or exceeding the maximum 775 =09* length allowed by the network interface. 776 =09*/ 777 =09if (len > tp->t_tsomax - hdrlen) { 778 =09 len =3D tp->t_tsomax - hdrlen; 779 =09 sendalot =3D 1; 780 =09} If it is a TSO segment of > 65535, at a glance it would seem that this "if" is busted. Just to see, you could try replacing line# 777-778 with if (len > IP_MAXPACKET - hdrlen) { len =3D IP_MAXPACKET - hdrlen; which was what it was in 9.1. (Maybe t_tsomax isn't set correctly or someho= w screws up the calculation? rick >=20 > Markus > _______________________________________________ > freebsd-net@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to > "freebsd-net-unsubscribe@freebsd.org" >=20