From owner-freebsd-questions@freebsd.org Sat May 8 13:45:31 2021 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 7565C635A5F for ; Sat, 8 May 2021 13:45:31 +0000 (UTC) (envelope-from orca@tira.com.ua) Received: from mail-ej1-x62c.google.com (mail-ej1-x62c.google.com [IPv6:2a00:1450:4864:20::62c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FcpVL4YJRz4tYy for ; Sat, 8 May 2021 13:45:30 +0000 (UTC) (envelope-from orca@tira.com.ua) Received: by mail-ej1-x62c.google.com with SMTP id l4so17805344ejc.10 for ; Sat, 08 May 2021 06:45:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tira-com-ua.20150623.gappssmtp.com; s=20150623; h=mime-version:from:date:message-id:subject:to; bh=X88O/D4Ku1TAPlD94zPfj1lvKWLMNVK5NSwnWTmM9XI=; b=vLTqYBGDygJ2+Yu9Zufbm59pObzP/fmyjY3AZ3cJMRGb1Cl4kWy/WTv1KDfFGXGZdm l/kYcctkvlj4ti5pnwDRpaeBrUe25xGg0uU96agY6WreKc13fP0CY7sb+BY6YeLWy8ie Z/rcGLHuOTPUgDy8Pj+eReeJvg8SQGfI+QSB+h9cJ/Vey8flypZJHGhVxyAimzGYkqLZ q40AoJTtxwHeOo5/4b9r4jK84QOpmqOeC4ty8PBfe1a+jUKO9CgskgfvE6UNva7G+U3w 360sTR6UPMBP13lxPhNrp1g+LjpWdAma2B5XqDZmSz5v9Q41ieskXnHOfmP8SwX9OGdl z3kQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=X88O/D4Ku1TAPlD94zPfj1lvKWLMNVK5NSwnWTmM9XI=; b=W7EV9LOJOxrWbpRDdI/HVK4FN8w/hocTlH545oN28UId1i8/9zAvo41WaWbt/xa4tm N8feP9CqQtcn4jQ3iD7VeJok5msxWwEKd26e4M99K8rWKVtfsLE2s0olU5wfXiNr4gjd csvnPmO/VjbWAmnW0iRqI+brlpUIG51u06E+pM6//HV1p97yBWuJIZo2+TPjFXNx9qUE XgN764uLXt5dBL3sZp8cvN4+XnT2Sw1PzwzgZoVY3M+CaDTUuM2w/K5iD9kTGLezl9gc WX+nHz7ZN8rDIwATWH6iUFpbzrrVxiT+H06YBy58YK5eiM/ND4HCuxhuoGdYF1oyVYti yRWg== X-Gm-Message-State: AOAM532WKNwUjh9DEfZZ14D8GzBse3BFVBa6Z/GpZDu/8v1zlSN78MHs CSMf+k48uCZqIfW1GlhqMpWWSwwRSrye9msU5lkKp+wriSFvxA== X-Google-Smtp-Source: ABdhPJyk8AJfhcu//o4H25qyyhqzELGp2VreeJao4qxb/PsMU3OGB4+8zgDE+7+7YJq0zo3T7MGmntalySDjSfhRXPc= X-Received: by 2002:a17:906:edcf:: with SMTP id sb15mr15641004ejb.202.1620481528176; Sat, 08 May 2021 06:45:28 -0700 (PDT) MIME-Version: 1.0 From: =?UTF-8?B?0JDQvdC00YDQtdC5INCT0LXRgNCw0YHQuNC80L7Qsg==?= Date: Sat, 8 May 2021 16:45:16 +0300 Message-ID: Subject: FreeBSD 13, Iflib, 2 CPU problem To: freebsd-questions@freebsd.org X-Rspamd-Queue-Id: 4FcpVL4YJRz4tYy X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tira-com-ua.20150623.gappssmtp.com header.s=20150623 header.b=vLTqYBGD; dmarc=none; spf=none (mx1.freebsd.org: domain of orca@tira.com.ua has no SPF policy when checking 2a00:1450:4864:20::62c) smtp.mailfrom=orca@tira.com.ua X-Spamd-Result: default: False [-3.20 / 15.00]; RCVD_TLS_ALL(0.00)[]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[tira-com-ua.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[2a00:1450:4864:20::62c:from:127.0.2.255]; DMARC_NA(0.00)[tira.com.ua]; DKIM_TRACE(0.00)[tira-com-ua.20150623.gappssmtp.com:+]; MIME_BASE64_TEXT(0.10)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::62c:from]; NEURAL_HAM_SHORT(-1.00)[-1.000]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[2a00:1450:4864:20::62c:from]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-questions] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 May 2021 13:45:31 -0000 Recently I have updated to FreeBSD 13, but it had begun since iflib was added to FreeBSD12. Behavior is the same on different servers. CPU 2 x CPU E5-2670 NIC driver is ix. It is standard. As I understand, when the system is starting it creates threads for every core of the processor. procstat -at | grep if_io 0 100019 kernel if_io_tqg_0 -1 24 sleep - 0 100020 kernel if_io_tqg_1 -1 24 sleep - 0 100021 kernel if_io_tqg_2 -1 24 sleep - 0 100022 kernel if_io_tqg_3 -1 24 sleep - 0 100023 kernel if_io_tqg_4 -1 24 sleep - 0 100024 kernel if_io_tqg_5 -1 24 sleep - 0 100025 kernel if_io_tqg_6 -1 24 sleep - 0 100026 kernel if_io_tqg_7 -1 24 sleep - 0 100027 kernel if_io_tqg_8 -1 24 sleep - 0 100028 kernel if_io_tqg_9 -1 24 sleep - 0 100029 kernel if_io_tqg_10 -1 24 sleep - 0 100030 kernel if_io_tqg_11 -1 24 sleep - 0 100031 kernel if_io_tqg_12 -1 24 sleep - 0 100032 kernel if_io_tqg_13 -1 24 sleep - 0 100033 kernel if_io_tqg_14 -1 24 sleep - 0 100034 kernel if_io_tqg_15 -1 24 sleep - When the network is loaded as I see in "top" only the first 8 threads are working which are binded to the first processor. So the second processor is not used at all. I've been trying a different combination of sysctl tuning and couldn't load the second processor. The only thing that helped was installing ix driver from ports collection which does not use iflib. And only after this i see 8 threads per NIC port that can be binded to different CPU cores. But when a server has more than 16 cores then it does not use excess ones, and if server has less than 16 cores, then few threads go to each core. I thought that it would be changed with FreeBSD13 but it wasn't