From owner-freebsd-drivers@freebsd.org Wed Sep 2 09:43:52 2020 Return-Path: Delivered-To: freebsd-drivers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4765B3D2810; Wed, 2 Sep 2020 09:43:52 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: from mail-oi1-x242.google.com (mail-oi1-x242.google.com [IPv6:2607:f8b0:4864:20::242]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BhJsz2JXQz4WG9; Wed, 2 Sep 2020 09:43:51 +0000 (UTC) (envelope-from rajfbsd@gmail.com) Received: by mail-oi1-x242.google.com with SMTP id y6so3850458oie.5; Wed, 02 Sep 2020 02:43:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=CXP6M2e9nMepjz/nQtdC62+uoPYITJvLzcX3ke47eEU=; b=q4bOXUFY5BuHszYdz2+L2d1fRT8eAWqcqfm/PApqQFvmAuVaIlUalVKPfPeLOKSF3X X2zc5Apy/FvaKOmqbzxD+HezruEZAVP3/twNVsZlTuWfDDcC62nURN0SIHlCtn8q6wt2 CF45RbQVkGk6qdJ7r/5oEsRwMlIX8b04tvYlCGqmBcHHUremsJU7UrutbEZiLCzFgWQb Y7PPQIzdN7pTZKQVkLUFTk+fETI3TTujeGFB+p8NvA2iy7zECF81C4Fu1zC9IS/X/+k1 ZhelUDRY34sjUHywhFeti4XQ9FeE6INXwWzjoyQSyb9SyPlg3TwIm/lq+h9vkHyB+nlg yeOg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=CXP6M2e9nMepjz/nQtdC62+uoPYITJvLzcX3ke47eEU=; b=qgnvtlG94h85qlwME8uBYQngwCn43U/lEcqrgdkQYXUdKNXxj1NQTqBOy0/Vi6hJGN ip2n9so6YdNe1RxvqfSnR7XyqxNptK4oX+6Hj654ThjLc9h1fcf0hpN0Xyyct2G1M4/1 1APqBgYV00WR9OHtw58nkEmdZ1ooyLqHCT+SdjvawtvbPe7LS20Gm85mnZvX/DbsDNjo YshsI8Z2kje9P8NoSHCJlkS+o+hPsgBdB+fqx6Lu+JYdJzhf8RX/bmXA3B72WczGoTv4 41KI+DbkdGd6l/O5XcNKwcuEOjL2846fATYpebn+Y9c5cCtOID8KU6Z5o3H8tviVTnB9 GnLg== X-Gm-Message-State: AOAM532ldixMLh/ylWN0M7/X7G9DaUVA8Wcnzncg4gQv/iYBr5UBX2jf 3cxXP6714j5eOBc5Z3Hbcn5DE/Dx5BE4AEFd5RoZwoij X-Google-Smtp-Source: ABdhPJw3myAWYhREsXYC6rq0gw45yDpAYTa6Z7XrMgjKiWhzrIAJLy4Z7u4LeX4uq7ZK2oA9ZjjnJeT3Wt74FuQcpnM= X-Received: by 2002:aca:49ca:: with SMTP id w193mr1552808oia.46.1599039829785; Wed, 02 Sep 2020 02:43:49 -0700 (PDT) MIME-Version: 1.0 References: <7cfc7c52-b548-19bd-343b-899aca45c654@selasky.org> In-Reply-To: From: Rajesh Kumar Date: Wed, 2 Sep 2020 15:13:39 +0530 Message-ID: Subject: Re: Network throughput not reaching line rate. Need clarification on iflib. To: Daniel Ebdrup Jensen Cc: freebsd-drivers@freebsd.org, FreeBSD Hackers X-Rspamd-Queue-Id: 4BhJsz2JXQz4WG9 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=q4bOXUFY; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of rajfbsd@gmail.com designates 2607:f8b0:4864:20::242 as permitted sender) smtp.mailfrom=rajfbsd@gmail.com X-Spamd-Result: default: False [-3.09 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.03)[-1.034]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.04)[-1.035]; TO_DN_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::242:from]; NEURAL_HAM_SHORT(-0.02)[-0.018]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-drivers,freebsd-hackers]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.33 X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 02 Sep 2020 09:43:52 -0000 Hi Guys, Apologies for posting a question on an old thread. I started this thread early this year and it happened to pause working on this topic due to COVID lockdown. I recently resumed working on this and hence posting my questions on the same thread. To restate the issue, I am observing low network throughput numbers in the Receive path (on a 10G link) and trying to improve the performance numbers. As suggested in this thread, I tried using "iperf" instead of "iperf3" and also tried multiple threads. The behavior remains the same between iperf and iperf3. With a single thread I don't see any improvement in the performance numbers. But with multiple threads, I see slight improvement on the numbers, but still not touching the line rate for a 10G link. The base issue here is that the Receive buffers are not getting refilled fast enough for the Received packets. Hence I see a lot of retries (iperf3 output showing retries) and the performance drops. I could see the CPU utilization more than 90% when running iperf/iperf3 traffic. This could be the reason why Receive buffers are not filled faster. Receive path in my driver is pretty straight forward using the iflib framework. "rxd_available" interface returns the number of newly received packets. "rxd_refill" and "rxd_flush" interfaces fills the receive descriptors with buffers from iflib and update the buffer pointer registers. "rxd_pkt_get" interface traverses through all the descriptors for a packet return the received pkt info to iflib. I don't see any unwanted lock, loops and other evident bottlenecks in this path. So, what could be the reason for higher CPU utilization? and How can I debug this? Please let me know if any details are needed. Thanks, Rajesh. On Mon, Mar 2, 2020 at 4:37 AM Rajesh Kumar wrote: > Hi Guys, > > Thanks for your time in responding. > > As you people stated, the problem here doesn't seem to be single threaded > vs multi threaded. I am experience similar behavior with "iperf". It's > mostly to do with iflib and my driver(mostly my driver). It looks like the > receiver is slow in reading the packets than the sender sending the > packets. In this case, iflib drives the packet read through the exposed > interfaces (rxd_available, rxd_pkt_get etc.,). So, how to make the > receiver side read the packet faster with iflib? Is there anything that I > should take care in my driver in this regard? > > Thanks, > Rajesh. > > > > > On Sat, Feb 29, 2020 at 3:47 AM Daniel Ebdrup Jensen > wrote: > >> On Fri, Feb 28, 2020 at 7:39 PM Bruce A. Mah wrote: >> >> > [Resending with a From: address that hopefully works better.] >> > >> > If memory serves me right, Daniel Ebdrup Jensen wrote: >> > > Yes, iperf3 will default to single-threaded packet generation, et al. >> > which >> > > favours fast cores with frequency boosting facilities. >> > > You might want to use iperf2 as that's properly multi-threaded, or you >> > can >> > > use pkt-gen out of src/tools/tools/netmap/ or ports/net/pkt-gen. >> > >> > While it's true that iperf3 is single-threaded, it should be capable of >> > saturating a 10GE link with a single TCP connection, given proper >> > command-line arguments (in particular, specifying a sufficiently large >> > socket-buffer size with the -w option). >> > >> > But based on the symptom of packet loss, I'd say the single-threaded vs. >> > multi-threaded argument might not be relevant to the problem that the OP >> > has. >> > >> > Bruce. >> > >> > > On Fri, Feb 28, 2020 at 10:35 AM Hans Petter Selasky > > >> > > wrote: >> > > >> > >> On 2020-02-28 10:03, Rajesh Kumar wrote: >> > >>> Hi FreeBSD team, >> > >>> >> > >>> I am writing a network driver using iflib framework and using >> "iperf3" >> > >> tool >> > >>> for performance testing. >> > >>> >> > >> >> > >> Is there any difference with "iperf" tool and using multiple >> threads? I >> > >> think iperf3 is single threaded ??? >> > >> >> > >> --HPS >> > >> >> > >> _______________________________________________ >> > >> freebsd-hackers@freebsd.org mailing list >> > >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> > >> To unsubscribe, send any mail to " >> > freebsd-hackers-unsubscribe@freebsd.org" >> > >> >> > > _______________________________________________ >> > > freebsd-hackers@freebsd.org mailing list >> > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> > > To unsubscribe, send any mail to " >> > freebsd-hackers-unsubscribe@freebsd.org" >> > > >> > >> >> Oh, I didn't mean to imply that that wasn't part of the issue - I'm sorry >> if I made it sound like that. >> I was just confirming what Hans was asking, and possibly using the excuse >> to mention some things in base/ports that I think are also pretty neat. :) >> >> Also no longer top-posting, which was rather ghastly of me. I apologise. >> _______________________________________________ >> freebsd-drivers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-drivers >> To unsubscribe, send any mail to "freebsd-drivers-unsubscribe@freebsd.org >> " >> >