From owner-freebsd-net@freebsd.org  Mon Feb  3 20:17:39 2020
Return-Path: <owner-freebsd-net@freebsd.org>
Delivered-To: freebsd-net@mailman.nyi.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1])
 by mailman.nyi.freebsd.org (Postfix) with ESMTP id E5B7322D924
 for <freebsd-net@mailman.nyi.freebsd.org>;
 Mon,  3 Feb 2020 20:17:39 +0000 (UTC) (envelope-from slw@zxy.spb.ru)
Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 48BJz64Zs5z44rg
 for <freebsd-net@freebsd.org>; Mon,  3 Feb 2020 20:17:38 +0000 (UTC)
 (envelope-from slw@zxy.spb.ru)
Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD))
 (envelope-from <slw@zxy.spb.ru>) id 1iyi9w-000OgO-SC
 for freebsd-net@freebsd.org; Mon, 03 Feb 2020 23:17:28 +0300
Date: Mon, 3 Feb 2020 23:17:28 +0300
From: Slawa Olhovchenkov <slw@zxy.spb.ru>
To: freebsd-net@freebsd.org
Subject: Chelsio NETMAP performance
Message-ID: <20200203201728.GC8028@zxy.spb.ru>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: slw@zxy.spb.ru
X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false
X-Rspamd-Queue-Id: 48BJz64Zs5z44rg
X-Spamd-Bar: /
Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none;
 spf=none (mx1.freebsd.org: domain of slw@zxy.spb.ru has no SPF policy when
 checking 195.70.199.98) smtp.mailfrom=slw@zxy.spb.ru
X-Spamd-Result: default: False [-0.48 / 15.00]; ARC_NA(0.00)[];
 NEURAL_HAM_MEDIUM(-0.69)[-0.687,0]; FROM_HAS_DN(0.00)[];
 TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.78)[-0.778,0];
 MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[];
 DMARC_NA(0.00)[zxy.spb.ru]; AUTH_NA(1.00)[];
 RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[];
 R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[];
 MIME_TRACE(0.00)[0:+];
 ASN(0.00)[asn:5495, ipnet:195.70.192.0/19, country:RU];
 MID_RHS_MATCH_FROM(0.00)[];
 IP_SCORE(0.08)[asn: 5495(0.40), country: RU(0.01)];
 RCVD_COUNT_TWO(0.00)[2]
X-BeenThere: freebsd-net@freebsd.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: Networking and TCP/IP with FreeBSD <freebsd-net.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-net/>
List-Post: <mailto:freebsd-net@freebsd.org>
List-Help: <mailto:freebsd-net-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-net>,
 <mailto:freebsd-net-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Mon, 03 Feb 2020 20:17:40 -0000

I am try to use Chelsio T540-CR in netmap mode and see poor (compared
to Intel 82599ES) performance.

Same application ac receive only about 8.9Mpss, compared to 12.5Mpps
at Intel.

pmc profile show mostly time spend in:

49.76%  [17802]    service_nm_rxq @ /boot/kernel/if_cxgbe.ko
 100.0%  [17802]     t4_vi_intr
  100.0%  [17802]      ithread_loop @ /boot/kernel/kernel
   100.0%  [17802]       fork_exit


to be exact at line

        while ((d->rsp.u.type_gen & F_RSPD_GEN) == nm_rxq->iq_gen) {

Is this maximum limit for this vendor?