From owner-freebsd-hackers@FreeBSD.ORG Wed Aug 27 19:07:44 2014 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 6B7CD771 for ; Wed, 27 Aug 2014 19:07:44 +0000 (UTC) Received: from mail.choopa.net (mail.choopa.net [216.155.136.52]) by mx1.freebsd.org (Postfix) with ESMTP id 416903C7C for ; Wed, 27 Aug 2014 19:07:43 +0000 (UTC) Received: from localhost (unknown [127.0.0.1]) by mail.choopa.net (iRedMail) with ESMTP id 761674AF416 for ; Wed, 27 Aug 2014 15:02:21 -0400 (EDT) X-Virus-Scanned: amavisd-new at mail.choopa.net Received: from mail.choopa.net ([127.0.0.1]) by localhost (mail.choopa.net [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 08WYFgH5s4uR for ; Wed, 27 Aug 2014 15:02:16 -0400 (EDT) Received: from [10.5.5.117] (office-nat.choopa.net [208.167.225.40]) by mail.choopa.net (iRedMail) with ESMTPSA id 59A6F4AF3E7 for ; Wed, 27 Aug 2014 15:02:16 -0400 (EDT) Message-ID: <53FE2B37.7050309@gameservers.com> Date: Wed, 27 Aug 2014 15:02:15 -0400 From: Brian Rak User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20100101 Thunderbird/24.6.0 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Bug in FreeBSD VirtIO network driver (only with pf enabled) Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Aug 2014 19:07:44 -0000 I have a FreeBSD 10 x64 guest installed inside a KVM instance on Linux. When pf is active, and the server is sending data it causes the Linux host to report warnings related to GSO. I've talked to some Linux developers, and they believe it to be a bug inside the FreeBSD VirtIO drivers. Based on what I'm seeing, I'm inclined to agree with them. Reproduction is mildly annoying, you need: 1) A KVM guest setup with bridged networking, with FreeBSD running side using the VirtIO network interface. (We tested with CentOS, but the exact distribution should not matter) 2) pf enabled (I'm using a single rule: "scrub in all") 3) Some method of sending a bit of traffic from the guest (I use netcat) So, when I do this: # service pf start # cat /root/test | nc vultr.com 80 The Linux kernel on the host will report: kernel: WARNING: CPU: 7 PID: 7772 at net/core/dev.c:2246 skb_warn_bad_offload+0xc3/0xd0() kernel: igb: caps=(0x0000000640114bb3, 0x0000000000000000) len=1498 data_len=0 gso_size=1380 gso_type=5 ip_summed=0 If I do: # service pf stop # cat /root/test | nc vultr.com 80 No such warning is reported. I can only reproduce this with pf enabled. The contents of the /root/test don't matter, I'm using 4k of data from /dev/urandom. The test file just needs to be bigger then the MTU of the host's network interface. I was able to track this down to the virtio_net_hdr being sent by the FreeBSD guest. With pf enabled, this outbound traffic has the following header: flags = 0 gso_type = VIRTIO_NET_HDR_GSO_TCPV4 hdr_len = 66 gso_size = 1440 csum_start = 0 csum_offset = 0 But, this is not a valid configuration. With VIRTIO_NET_HDR_GSO_TCPV4 enabled, you should also be setting VIRTIO_NET_HDR_F_NEEDS_CSUM and populating the csum_start and csum_offset fields. http://www.spinics.net/lists/netdev/msg293976.html gives more detail on this. I don't fully understand this, so I'd probably mangle the explanation if I tried to give more detail. I can reproduce this at will now, but fixing it is beyond my abilities. Is there a better place to report this? I'm not entirely sure who is responsible for maintaining the virtio driver.