From owner-freebsd-hackers@freebsd.org Wed Jun 27 16:38:26 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A0CBE102C028 for ; Wed, 27 Jun 2018 16:38:26 +0000 (UTC) (envelope-from steevanxperia@gmail.com) Received: from mail-ot0-x235.google.com (mail-ot0-x235.google.com [IPv6:2607:f8b0:4003:c0f::235]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 31605851A1 for ; Wed, 27 Jun 2018 16:38:26 +0000 (UTC) (envelope-from steevanxperia@gmail.com) Received: by mail-ot0-x235.google.com with SMTP id n24-v6so2883177otl.9 for ; Wed, 27 Jun 2018 09:38:26 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=RSkWBMoHqYGlnZvJMGzTO6ABFkoUnG+5WDdTWvLYZSU=; b=KE4ab8yeeprN/PRwwqWxhLdLRtDXDpuOm8GFN8Yd3rUZFpdGPoyFuTOfF4LrGiHNiS gpPmplfLPyD9B6rPx3JadjQorTQSm6BB6hnuogkSsEp8/ubB7eCgrN4Lmmzn8dq3Wuvy C+YtkkbTzEcOmLrnCEvFj0JRJMp1x+GoFxFAJqDgXzkn7lKX1VCTcOTDJ2f++VYtf9F5 +cqFP7M71lGz70ghycOP0+8QCA83Gaxfjp7jhQBFmBpJQ3V37HaqbyW0JIdKthdJpt8Z MZ0IJQ39U4zhmHEVMYygXjdwxahrODH0+XtW09HKl1xtezmQXTBlnG2Er744FJHgEELI tV1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=RSkWBMoHqYGlnZvJMGzTO6ABFkoUnG+5WDdTWvLYZSU=; b=YnqrYQxZeBtX31iweeAwQetoF8fdgNMFepYRoVtuoFSgEzbiJimvAvO3bfCuZ2cbo7 2df8MZ6FHu70HQfN/lUx1NEdRtUrIGmJS/EZ67xhgC7WIdh8CXhWmUjbaoB+d2kSC5VF NWDJF6eRcvON65JOV2VoN1uc82r3R4uBEuwXZY/1Obg5VTy3AAQQEAzyGfqu0jtVvYis 5AmZ61iPVkDYA0nfYHCG9VJKzlHlvjRmftBuKtMUgOZ/kEl4ZmSNSRVRkvcS3hUKY6Bu ewyd7Mu+NUM/EQpZtHiEpjuJsinQC5o7xtr84AK2vquVyvFhF+7+oFlRlm5ZkuQV3QQg Rmug== X-Gm-Message-State: APt69E3qq/JlZN+p4BQrd08T6bfGivk1XWQZjW1UgXY52dXM2NhsfnDD ZxVol/fRqzs/5BUBBxWBtq7qmKlUQJ08gyMq3cy4Ug== X-Google-Smtp-Source: AAOMgpfyOCV9Sq+w7Q2HpqjQGGk+AOxCQ3YRXvuBIeE7IBsdN1j365i4SsOTjHDnlhqaUidWKBQ1zkq84Yap9I8ZAFs= X-Received: by 2002:a9d:4a49:: with SMTP id d9-v6mr2452821otj.374.1530117505234; Wed, 27 Jun 2018 09:38:25 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:ac9:73cf:0:0:0:0:0 with HTTP; Wed, 27 Jun 2018 09:38:24 -0700 (PDT) In-Reply-To: References: From: Steevan Rodrigues Date: Wed, 27 Jun 2018 22:08:24 +0530 Message-ID: Subject: Re: high CPU usage in FreeBSD for a PCIe card driver To: Warner Losh Cc: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jun 2018 16:38:26 -0000 Thank you for the quickly reply. The same driver gives pretty good performance with low CPU usage in kernel mode. I see high CPU usage only for user mode application. Here is the Dtrace hotkernel output details provided in the earlier attachment. CPU usage details on a 12 core 24 thread server PC. part of the perl hotkernel output kernel`free 1314 0.2% kernel`_sx_slock 1415 0.3% kernel`dev_relthread 1435 0.3% kernel`devvn_refthread 1842 0.3% kernel`_mtx_lock_spin_cookie 2059 0.4% kernel`copyin 2164 0.4% kernel`amd64_syscall 2235 0.4% kernel`__mtx_lock_spin_flags 3566 0.6% kernel`bzero 4889 0.9% kernel`__mtx_lock_sleep 9323 1.7% kernel`acpi_cpu_idle 11166 2.0% kernel`pmap_extract 17491 3.1% kernel`spinlock_exit 64850 11.5% kernel`cpu_idle 134617 23.9% kernel`lock_delay 263330 46.8% ------------------------------------------------- CPU usage details on a 2 core 4 thread Intel i3 based Desktop PC. part of the perl hotkernel output kernel`0xffffffff80 558 0.6% kernel`copyout 592 0.6% 0xffffffff822368a3 639 0.7% kernel`amd64_syscall 683 0.7% kernel`__mtx_unlock_flags 728 0.8% kernel`copyin 1311 1.4% kernel`bzero 1434 1.5% kernel`pmap_extract 1615 1.7% kernel`lock_delay 2329 2.4% 0xffffffff822b0b5e 2528 2.6% kernel`spinlock_exit 20950 21.8% kernel`cpu_idle 48124 50.2% ------------------------------------------------------ Thanks Steevan On Wed, Jun 27, 2018 at 7:41 PM, Warner Losh wrote: > No. It's not from mtx_lock. It's likely from lock contention. Since the > list ate the attachment, I can't check for sure, but high CPU usage in > locking primitives is a classic sign of poor lock design leading to > contention. > > Warner > > On Wed, Jun 27, 2018 at 6:28 AM, Steevan Rodrigues < > steevanxperia@gmail.com> wrote: > >> Hi >> >> I come from Linux background and new to FreeBSD. >> >> I am investigating a high CPU usage issue on FreeBSD for PCIe driver of a >> card . The card can process data at a rate of 30 to 50 Gbps . However, I >> see very high CPU usage even when I run a single threaded user side >> application which uses this card . >> >> I ran hotkernel script from Dtrace toolkit. A portion of the output is >> given in the attached file dtraceDiff.txt. Here I see that lock_delay >> seems to be the reason for high CPU usage. Looks like it comes from mutex >> locks >> >> This driver uses mutex locks ( mtx_lock). >> >> The CPU usage is very high on a multicore (Xeon CPU) server node than a >> dual core (Intel i3) Desktop . >> >> It shows 30 to 40 % CPU usage on a Intel i3 ( 2 core 4 threads) and 60 to >> 70% CPU usage on a Xeon CPU with 12 core (24 threads) for same output >> data rate . >> >> I wonder whether usage of mtx_lock is the cause of high CPU usage ? Any >> pointers or suggestions welcome. >> >> Using FreeBSD 11.1 Release >> >> Thanks >> >> Steev >> >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org >> " >> >> >