From owner-freebsd-hackers@FreeBSD.ORG Wed Aug 27 19:29:47 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 2068944C; Wed, 27 Aug 2014 19:29:47 +0000 (UTC) Received: from webmail2.jnielsen.net (webmail2.jnielsen.net [50.114.224.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "webmail2.jnielsen.net", Issuer "freebsdsolutions.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id F13143F42; Wed, 27 Aug 2014 19:29:46 +0000 (UTC) Received: from [10.10.1.198] (office.betterlinux.com [199.58.199.60]) (authenticated bits=0) by webmail2.jnielsen.net (8.14.9/8.14.9) with ESMTP id s7RJQ6pl080635 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Wed, 27 Aug 2014 13:26:09 -0600 (MDT) (envelope-from lists@jnielsen.net) X-Authentication-Warning: webmail2.jnielsen.net: Host office.betterlinux.com [199.58.199.60] claimed to be [10.10.1.198] Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Subject: Re: Bug in FreeBSD VirtIO network driver (only with pf enabled) From: John Nielsen In-Reply-To: <53FE2B37.7050309@gameservers.com> Date: Wed, 27 Aug 2014 13:26:05 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <13EFC786-FFFB-479A-A65F-F33B07A78B9F@jnielsen.net> References: <53FE2B37.7050309@gameservers.com> To: Brian Rak , Bryan Venteicher X-Mailer: Apple Mail (2.1878.6) X-Virus-Scanned: clamav-milter 0.98.4 at webmail2.jnielsen.net X-Virus-Status: Clean Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Aug 2014 19:29:47 -0000 The virtio maintainer is bryanv (CC'ed). I have KVM+CentOS hosts available at $WORK so let me know if anyone = needs help reproducing this or testing fixes. On Aug 27, 2014, at 1:02 PM, Brian Rak wrote: > I have a FreeBSD 10 x64 guest installed inside a KVM instance on = Linux. When pf is active, and the server is sending data it causes the = Linux host to report warnings related to GSO. >=20 > I've talked to some Linux developers, and they believe it to be a bug = inside the FreeBSD VirtIO drivers. Based on what I'm seeing, I'm = inclined to agree with them. >=20 > Reproduction is mildly annoying, you need: > 1) A KVM guest setup with bridged networking, with FreeBSD running = side using the VirtIO network interface. (We tested with CentOS, but = the exact distribution should not matter) > 2) pf enabled (I'm using a single rule: "scrub in all") > 3) Some method of sending a bit of traffic from the guest (I use = netcat) >=20 > So, when I do this: >=20 > # service pf start > # cat /root/test | nc vultr.com 80 >=20 > The Linux kernel on the host will report: >=20 > kernel: WARNING: CPU: 7 PID: 7772 at net/core/dev.c:2246 = skb_warn_bad_offload+0xc3/0xd0() > kernel: igb: caps=3D(0x0000000640114bb3, 0x0000000000000000) len=3D1498 = data_len=3D0 gso_size=3D1380 gso_type=3D5 ip_summed=3D0 >=20 > If I do: >=20 > # service pf stop > # cat /root/test | nc vultr.com 80 >=20 > No such warning is reported. I can only reproduce this with pf = enabled. The contents of the /root/test don't matter, I'm using 4k of = data from /dev/urandom. The test file just needs to be bigger then the = MTU of the host's network interface. >=20 > I was able to track this down to the virtio_net_hdr being sent by the = FreeBSD guest. With pf enabled, this outbound traffic has the following = header: >=20 > flags =3D 0 > gso_type =3D VIRTIO_NET_HDR_GSO_TCPV4 > hdr_len =3D 66 > gso_size =3D 1440 > csum_start =3D 0 > csum_offset =3D 0 >=20 > But, this is not a valid configuration. With VIRTIO_NET_HDR_GSO_TCPV4 = enabled, you should also be setting VIRTIO_NET_HDR_F_NEEDS_CSUM and = populating the csum_start and csum_offset fields.