From owner-freebsd-net@freebsd.org Mon Nov 12 12:51:33 2018 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD9651105400 for ; Mon, 12 Nov 2018 12:51:33 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mx1.sbone.de (cross.sbone.de [195.201.62.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mx1.sbone.de", Issuer "SBone.DE" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 93D6B6E05D for ; Mon, 12 Nov 2018 12:51:32 +0000 (UTC) (envelope-from bzeeb-lists@lists.zabbadoz.net) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id B2F258D4A24D; Mon, 12 Nov 2018 12:51:23 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id E000ED1F801; Mon, 12 Nov 2018 12:51:22 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id kSTRNpB2OVeW; Mon, 12 Nov 2018 12:51:20 +0000 (UTC) Received: from [192.168.1.88] (fresh-ayiya.sbone.de [IPv6:fde9:577b:c1a9:f001::2]) (using TLSv1 with cipher AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id B5021D1F7FD; Mon, 12 Nov 2018 12:51:19 +0000 (UTC) From: "Bjoern A. Zeeb" To: "Eugene Grosbein" Cc: "Wolfgang Zenker" , freebsd-net@freebsd.org Subject: Re: epair(4) and net.route.netisr_maxqlen Date: Mon, 12 Nov 2018 12:51:29 +0000 X-Mailer: MailMate (2.0BETAr6127) Message-ID: In-Reply-To: <5afe5838-d2df-009d-af85-99e793f1927c@grosbein.net> References: <20181111232320.GA87730@lyxys.ka.sub.org> <5afe5838-d2df-009d-af85-99e793f1927c@grosbein.net> MIME-Version: 1.0 Content-Type: text/plain; format=flowed X-Rspamd-Queue-Id: 93D6B6E05D X-Spamd-Result: default: False [-3.73 / 200.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_FIVE(0.00)[5]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:195.201.62.131]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[zabbadoz.net]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cross.sbone.de]; NEURAL_HAM_SHORT(-0.83)[-0.832,0]; IP_SCORE(-0.59)[asn: 24940(-2.93), country: DE(-0.02)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:24940, ipnet:195.201.0.0/16, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-Rspamd-Server: mx1.freebsd.org X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 12 Nov 2018 12:51:33 -0000 On 12 Nov 2018, at 3:44, Eugene Grosbein wrote: My additions are mostly for Wolfgang, > 12.11.2018 6:23, Wolfgang Zenker wrote: > >> on a jail with quite a lot of somewhat bursty network traffic I get >> warnings from netdata recently about packets being dropped because of >> net.route.netisr_maxqlen being to small. Before I start setting this >> value to some random value I'ld like to find out what it actually >> means. >> A search for documentation turned up nothing; a look at the sources >> found that it is used for the size of a "software interrupt queue" in >> epair(4). But what does it mean? And does this give me enough >> information to find a good value to set for this sysctl? > > netisr packet queues keep packets received by the interface and > not yet processed by destined subsystem or userland application > that may be short of CPU cycles or blocked for some reason. > > First, the system won't allow you to raise net.route.netisr_maxqlen > over the limit net.isr.maxqlimit. > The limit itself can be changed with /boot/loader.conf and reboot. > Default value of limit is 10240. I generally raise the limit upto > 102400 > for hosts with heavy/bursty traffic. Note that actual increase of > net.route.netisr_maxqlen > somewhat increases usage of kernel memory and that could be important > for 32 bit kernel > and/or system with very low amount of RAM. > > There may be several netisr packet queues in the system and raising > net.route.netisr_maxqlen > allows all of them to grow. epair(4) has its own setting > net.link.epair.netisr_maxqlen > that defaults to net.route.netisr_maxqlen if unset, so you may be > start experimenting with > net.link.epair.netisr_maxqlen first, instead of system global > net.route.netisr_maxqlen. > > Don't set net.route.netisr_maxqlen to random value but double its > current value > and see if that would be enough. If not, double it again. If 4096 > apears not enough, > you should check your applications why they can't keep with incoming > traffic rate. Also if you have multiple epair interface-combinations and a multi-core CPU you might also want to try (which never became default I think) to experiment with these settings in loader.conf: net.isr.bindthreads=1 net.isr.maxthreads=256 # net.isr will always reduce it to mp_cpus which should help balancing the load across the cores. Note: these changes also affect all other possible traffic going through the netisr framework. The netisr(9) man page has some documentation of these fields but not everything. The source code does have a lot of comments and if someone would improve the man page that might be a good start: https://svnweb.freebsd.org/base/head/sys/net/netisr.c?annotate=326272#l159 netstat -Q is also a good source for monitoring and diagnostics. /bz