From owner-freebsd-net@freebsd.org Fri Apr 20 23:45:25 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 92B4EFAC45B for ; Fri, 20 Apr 2018 23:45:25 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: from mail-io0-x22b.google.com (mail-io0-x22b.google.com [IPv6:2607:f8b0:4001:c06::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2B1057EE50 for ; Fri, 20 Apr 2018 23:45:24 +0000 (UTC) (envelope-from kevin.bowling@kev009.com) Received: by mail-io0-x22b.google.com with SMTP id o7-v6so10526642iob.8 for ; Fri, 20 Apr 2018 16:45:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kev009.com; s=google; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=AQaFWWUEfJFG3oy+RBK2r8ywM0XT/Dj5QhhW8bw52SA=; b=HSj5BqQc/FKo1GsUZa0d8b9AxNIdVaNp1rNN7ofgnjWNqQkiy9fymqa6BAP92Bj0ML 9YyC/tX+0HIpfivFpVh2jSE5e0/XOj2BD0XVs7nnoXvju17slHIwObvL+YmXJ1v2U4GW +YEsgK0DDTqcLl37XI0UmT5FlTGCvz8sCE0YQ= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=AQaFWWUEfJFG3oy+RBK2r8ywM0XT/Dj5QhhW8bw52SA=; b=mUWvlFxDr25+g35lLZ/DJkCSAN9U5x5lErRZGaLMf91OW2LHyACYxKRRM3WG1EyRrZ EWUBaCqXbky2MEYVl3lt63UjXzErpvAgCfHCGBrWChCMeh1Ge3uOxSkknS+rjD9CvSUY w/UdEWZ0t5QQy2S2E9/DpkNrGX0oNjG2Ws3CVH8SqNK7FQjVxz7QDmpoHTEHhajleBZE j0+YagQis8vN1H6dgstF9SNS9iznXCtqwipIdEemJOWPeQs9oJpEKwcJlO919Efe79pU QYN8/9Iiqh1JrygpS01hIP7B+MCMzT5VZ+Rma6vQ+reP/VlJuBH3cQx55GH+MuP0+LOK 5FYA== X-Gm-Message-State: ALQs6tDL70g0lwXW/PWkCfhz239Zbj1ByJqb7y/8ktwZ0uAiB5+6rRib LmwTPjQHF/5faeXD9X0iKtXAIz7Z8z0VrAAc1UiAaLTl X-Google-Smtp-Source: AB8JxZp0yIsNuh33p77CblT612Vit1JYMVDVjbU+P56onk0tlCTOW4SSPgK2vx1aB3TQqO1NC/USv+tP4eiq/KBaMZ0= X-Received: by 2002:a6b:afdb:: with SMTP id p88-v6mr12396183ioo.257.1524267924245; Fri, 20 Apr 2018 16:45:24 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a02:9046:0:0:0:0:0 with HTTP; Fri, 20 Apr 2018 16:45:23 -0700 (PDT) In-Reply-To: <23257.26265.720293.659892@hergotha.csail.mit.edu> References: <23257.26265.720293.659892@hergotha.csail.mit.edu> From: Kevin Bowling Date: Fri, 20 Apr 2018 16:45:23 -0700 Message-ID: Subject: Re: Diagnosing terrible ixl performance To: Garrett Wollman Cc: FreeBSD Net Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 20 Apr 2018 23:45:25 -0000 These are all pretty much known (to intel and NF/LLNW) issues with the ixl driver. If you must run 11.1 you are best off just buying a chelsio T580. If you can run HEAD, or perhaps an eventual 11.3 MFC, https://github.com/intel-wired-ethernet/freebsd/tree/ixl-iflib may fix this but there are a few remaining issues before that hits HEAD. Regards, On Thu, Apr 19, 2018 at 9:03 PM, Garrett Wollman wrote: > I'm commissioning a new NFS server with an Intel dual-40G XL710 > interface, running 11.1. I have a few other servers with this > adapter, although not running 40G, and they work fine so long as you > disable TSO. This one ... not so much. On the receive side, it gets > about 600 Mbit/s with lots of retransmits. On the *sending* side, > though, it's not even able to sustain 10 Mbit/s -- but there's no > evidence of retransmissions, it's just sending really really slowly. > (Other machines with XL710 adapters are able to sustain full 10G.) > There is no evidence of any errors on either the adapter or the switch > it's connected to. > > So far, I've tried: > > - Using the latest Intel driver (no change) > - Using the latest Intel firmware (breaks the adapter) > - Disabling performance tweaks in loader.conf and sysctl.conf > - Changing congestion-control algorithms > > Anyone have suggestions while I still have time to test this? (My > plan B is to fall back to an X520 card that I have in my spares kit, > because I *know* those work great with no faffing about.) Any > relevant MIBs to inspect? > > The test I'm doing here is simple iperf over TCP, with MTU 9120. It > takes about 10 seconds for the sending side to complete, but buffers > are severely constipated for 20 seconds after that (delaying all > traffic, including ssh connections). > > I'm at the point of trying different switch ports just to eliminate > that as a possibility. > > -GAWollman > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"