From owner-freebsd-net@freebsd.org Sun Jan 24 07:23:05 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 211BAA8F40D for ; Sun, 24 Jan 2016 07:23:05 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: from mail-lf0-x22a.google.com (mail-lf0-x22a.google.com [IPv6:2a00:1450:4010:c07::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9BBCB1D71 for ; Sun, 24 Jan 2016 07:23:04 +0000 (UTC) (envelope-from rizzo.unipi@gmail.com) Received: by mail-lf0-x22a.google.com with SMTP id m198so68524531lfm.0 for ; Sat, 23 Jan 2016 23:23:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:content-type; bh=qmQgtJ9h6EOwgB7X8+Lz0O4xFcymzN5bQKqYjt7WR30=; b=rZh5Qc/j1oK5RcZmO2g8u06LcMi35KvgEh4Qn0M29odZrmBK9rrK3n0Bkld+QAqAUs As7qBw7CMH7ojcy9HtU9463FhdgtL0qrXeAUhHwgaHhzeuOXcUae1Jy3qNrmMhS3cm8I nR3NRPs4i+FOdmV+bPrGj5XCPmBhax62dBLj5lavnvam8ytCk45zXzNgntU86v7t2R/E 4DnmP1Jk0kUxlxxATN8wzgvGl+XnKV5XJsQepKw3YhMF7cxs6k5L9HoCS0hv6Igif42I yh0lRk9UU9e217XjsHXwDwVIfkgRawtI+5IYPe4XlKAuGWCqUC05ftG4Lw3tq+r1lJ7p JJsA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=qmQgtJ9h6EOwgB7X8+Lz0O4xFcymzN5bQKqYjt7WR30=; b=dUbzNGhmdIBD/ySUMUmoX/UEpp0LZGji28DbfxIlBG5eT6PJ/Z3LdMnMnFij4U76Ct G83tJNBpddwlNDCgLztLhruDLDVJ2ZyHS5UXD+R6myGKL2UT9ybtSBb60Ho/nq381rGQ ixqPdomrtgBFzYoRhJJzl5ZMj64V8x0wuaP45c41RFz9yhyyAVJYDmtAmeGc7ZtWL+v3 tHxe29JvcR6SWeVp05rYul6eYBxQCj/wDpct8nFm12UuXgyFyF5+dwqS8TwteqKRVmNb qdW/QWJvMf50svW9m3hlwYtecfwWEeuQf8R6MBbrEhVXOwQp1Tl8U+GFZ5hhqS655fsX 0FiQ== X-Gm-Message-State: AG10YOTpOCUx0tH077PalS0pn+mf7pXOCWvvWhcgqVu4j9PDbmi3W11uARJJfhi5/bHGMr6wt6t0WA8zitLjtg== MIME-Version: 1.0 X-Received: by 10.25.209.138 with SMTP id i132mr4181254lfg.4.1453620181539; Sat, 23 Jan 2016 23:23:01 -0800 (PST) Sender: rizzo.unipi@gmail.com Received: by 10.114.4.232 with HTTP; Sat, 23 Jan 2016 23:23:01 -0800 (PST) In-Reply-To: <20160124064217.GB7567@ox> References: <20160124042830.3D674A0128@smtp.hushmail.com> <20160124064217.GB7567@ox> Date: Sat, 23 Jan 2016 23:23:01 -0800 X-Google-Sender-Auth: 7WIfQixi2sMrNMogw4wHgYJF1HU Message-ID: Subject: Re: solved: Re: Chelsio T520-SO-CR low performance (netmap tested) for RX From: Luigi Rizzo To: Luigi Rizzo , Marcus Cenzatti , "freebsd-net@freebsd.org" , Navdeep Parhar Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 24 Jan 2016 07:23:05 -0000 On Sat, Jan 23, 2016 at 10:42 PM, Navdeep Parhar wrote: > On Sat, Jan 23, 2016 at 09:33:32PM -0800, Luigi Rizzo wrote: >> On Sat, Jan 23, 2016 at 8:28 PM, Marcus Cenzatti wrote: >> > >> > >> > On 1/24/2016 at 1:10 AM, "Luigi Rizzo" wrote: ... >> One last attempt: try use -l 64 on the sender, this will generate 64+4 byte >> packets, which may become just 64 on the receiver if the chelsio is configured >> to strip the CRC. This should result in well aligned PCIe transactions and >> reduced PCIe traffic, which may help (the ix driver has a similar problem, >> but since it does not strip the CRC can rx at line rate with 60 bytes but not >> with 64). > > Keep hw.cxgbe.fl_pktshift in mind for these kind of tests. The default > value is 2 so the chip DMAs payload at an offset of 2B from the start of > the rx buffer. So you'll need to adjust your frame size by 2 (66B on > the wire, 62B after CRC is removed, making it exactly 64B across PCIe if > pktshift is 2) or just set hw.cxgbe.fl_pktshift=0 in /boot/loader.conf. This sounds like something to fix. In netmap packets are supposed to be aligned with the beginning of the buffer, so when operating in netmap mode at least the pktshift should be set to 0. If it is not possible to have it per- queue, I would suggest to set it to 0 unconditionally when you compile a netmap-enabled kernel. How much of a performance boost do you see with a shift of 2 with regular traffic ? Modern CPUs seem to be pretty good with unaligned memory accesses. cheers luigi > Regards, > Navdeep -- -----------------------------------------+------------------------------- Prof. Luigi RIZZO, rizzo@iet.unipi.it . Dip. di Ing. dell'Informazione http://www.iet.unipi.it/~luigi/ . Universita` di Pisa TEL +39-050-2217533 . via Diotisalvi 2 Mobile +39-338-6809875 . 56122 PISA (Italy) -----------------------------------------+-------------------------------