From owner-freebsd-xen@FreeBSD.ORG Tue Sep 2 17:25:19 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 87605C2A for ; Tue, 2 Sep 2014 17:25:19 +0000 (UTC) Received: from SMTP02.CITRIX.COM (smtp02.citrix.com [66.165.176.63]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (Client CN "mail.citrix.com", Issuer "Cybertrust Public SureServer SV CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BB4E219A3 for ; Tue, 2 Sep 2014 17:25:18 +0000 (UTC) X-IronPort-AV: E=Sophos;i="5.04,450,1406592000"; d="scan'208";a="168408834" Received: from [IPv6:::1] (10.80.16.47) by smtprelay.citrix.com (10.13.107.78) with Microsoft SMTP Server id 14.3.181.6; Tue, 2 Sep 2014 13:25:12 -0400 Message-ID: <5405FD77.2000102@citrix.com> Date: Tue, 2 Sep 2014 19:25:11 +0200 From: =?UTF-8?B?Um9nZXIgUGF1IE1vbm7DqQ==?= User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:31.0) Gecko/20100101 Thunderbird/31.0 MIME-Version: 1.0 To: Marko Lerota Subject: Re: Poor disk IO on Xenserver 6.2 References: <86lhq7odub.fsf@arch.perpetuum.hr> <5400B39E.3090904@citrix.com> <86tx4un8bd.fsf@cosmos.claresco.hr> In-Reply-To: <86tx4un8bd.fsf@cosmos.claresco.hr> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 8bit X-DLP: MIA1 Cc: FreeBSD XEN X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Sep 2014 17:25:19 -0000 El 30/08/14 a les 2.13, Marko Lerota ha escrit: > Roger Pau Monné writes: > >> El 29/08/14 a les 11.16, Marko Lerota ha escrit: >>> I have installed FreeBSD 10 release, stable and current on >>> Xenserver 6.2 with latest patches. All of them have poor disk >>> write speed. Performance from 12MBps - 25MBps. I tried it on >>> other servers but the performance is the same. Latest Debian >>> stable on the same machine have 300MBps disk speed. Am I missing >>> something after installation or this is the default speed? >>> >> >> I've certainly seen higher speeds, around 25-35MB/s on my box, which is >> the same speed I can get with a Linux guest. Could you describe how you >> are running your tests? > > You copy some *.tar.gz or *.iso file that have 600 MB. If it finish > in 2-3 seconds than it's fine. If it runs 40 seconds than it's very, > very bad. You can see current disk speed with 'iostat 1'. Today I have > installed FreeBSD 10-Release on KVM and it runs smooth. Files are > copied at 150-300MB per second. That's how it should be. Suppose that > you need to copy or sync files with another disk/partition very often. > It would be very painful and long process. There is definitely something > wrong here. I've just did some _very_ simple disk benchmarks using fio on both Linux and FreeBSD using a 1GB ramdisk as the backend: 'phy:/dev/ram0,hdd,w' This was added to the guests, both have 1GB of RAM and 8vcpus. Then I've used the following fio workload: [global] rw=write # or read for the read tests size=900m bs=32k direct=1 # tested both 0 and 1 [job1] FreeBSD was able to deliver >30000iops in all cases, and the throughput was around 1000MB/s for all test cases also. On the other hand, Linux was only able to deliver around 10000iops, with a throughput of ~400MB/s. This was tested using xen-unstable and a Linux v3.15 Dom0. Could you try to run this benchmark on both your FreeBSD and Linux guests? Please make sure guests are set to use the same amount of RAM/CPUs. Roger. From owner-freebsd-xen@FreeBSD.ORG Fri Sep 5 19:05:35 2014 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 515D796D for ; Fri, 5 Sep 2014 19:05:35 +0000 (UTC) Received: from mail-oi0-x230.google.com (mail-oi0-x230.google.com [IPv6:2607:f8b0:4003:c06::230]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1776D1F55 for ; Fri, 5 Sep 2014 19:05:34 +0000 (UTC) Received: by mail-oi0-f48.google.com with SMTP id a141so8027715oig.35 for ; Fri, 05 Sep 2014 12:05:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=averesystems.com; s=google; h=mime-version:date:message-id:subject:from:to:content-type; bh=hw+g9+TnF6G4CaAjucoSSzCCcsJNrrG08SYuE+jzeBE=; b=FxjgHeYYs2Z5PfzXvUrKZM3rWkYPhkVFN85ZN2Ha9vJuetcykxaoovrN94BGRoL9sH 8DRasIYEhA//Tgt68610bclAiHDkhQj+B0iA6OiaFHVsTluCJf+z5dJITyiKZ44pIZmv KnoCl9+SSMVeR/b62aCcmp0fdRQxhE0oeb4GA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to :content-type; bh=hw+g9+TnF6G4CaAjucoSSzCCcsJNrrG08SYuE+jzeBE=; b=iVqF4v1qjwfDQ8mEgykhsYaiw5onhWGDPdU4FZEV3up4xoxlC6Jxf673e56uv1ib7g TFBNL5BofTfJy4gMwgvfrTtyAOs01pNxZ5oGyDdsrdPY5ZearaK4Nju5vITa7yUrkU91 M8ZLEshvseZvoMCs/sHkAs3ZyvyyLsUSzYmNrqLW/a59I7uF6UXHxFoxiW73XdL/1RyF XMGWPPOWNJUek0gY7+w6+ckpBc5alQoqcnx8JSDqvaQMDYDj7lbDGtLxHy5pU4WiMcnM bRbEOeXL+jK8bC02T9xPrysKmlmYiDQyetKI/hIsaqRJdCqo6pIGAIieThQjAX71OjHh MEYQ== X-Gm-Message-State: ALoCoQkFOJicJxieorUlTX22l45q5tI5k5VvafVt95VYrajeYKMAejYtHgbCPof/ohJf/0bgPlPv MIME-Version: 1.0 X-Received: by 10.182.60.98 with SMTP id g2mr16195525obr.6.1409943934291; Fri, 05 Sep 2014 12:05:34 -0700 (PDT) Received: by 10.76.12.70 with HTTP; Fri, 5 Sep 2014 12:05:34 -0700 (PDT) Date: Fri, 5 Sep 2014 15:05:34 -0400 Message-ID: Subject: netfront driver with mbufs > 4K From: Jeremiah Lott To: freebsd-xen@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.18-1 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 05 Sep 2014 19:05:35 -0000 It seems the netback driver (at least in linux) cannot handle a fragment passed to it that crosses a 4K boundary. Here is the code from the linux netback driver that enforces this: if (unlikely((txp->offset + txp->size) > PAGE_SIZE)) { netdev_err(vif->dev, "Cross page boundary, txp->offset: %x, size: %u\n", txp->offset, txp->size); xenvif_fatal_tx_err(vif); return -EINVAL; } The netfront driver in FreeBSD creates a fragment for each mbuf in the chain, but does not check if the mbuf crosses a 4K boundary. For mbufs allocated from the zone_jumbo9 or zone_jumbo16 uma zones, it can create fragments that are too large and/or cross a page boundary. In this case the netback stops responding, and the network interface appears to "hang". It is also a problem for mbufs that have external data attached to them (as the data could be any size or alignment). It seems that the netfront driver needs split mbufs that span page boundaries into multiple fragments. I'm starting to think about code to do that, but I figured I'd put this out there if anyone has suggestions on handling this. Thanks, Jeremiah Lott Avere Systems