From owner-freebsd-stable@freebsd.org Wed Jan 16 14:31:54 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7F535148A3CF for ; Wed, 16 Jan 2019 14:31:54 +0000 (UTC) (envelope-from thomas@gibfest.dk) Received: from smtp2.servers.tyknet.dk (smtp2.servers.tyknet.dk [IPv6:2a01:3a0:1:1900:89:233:43:78]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 045B570161 for ; Wed, 16 Jan 2019 14:31:52 +0000 (UTC) (envelope-from thomas@gibfest.dk) Received: from [10.137.3.13] (nat2.hq.bornfiber.dk [185.96.91.2]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp2.servers.tyknet.dk (Postfix) with ESMTPSA id DD0BDF08F; Wed, 16 Jan 2019 14:31:49 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 smtp2.servers.tyknet.dk DD0BDF08F DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=gibfest.dk; s=default; t=1547649110; bh=WN+MuKcCdi81IbpbluCD6VPaoCxXl29EV2FQCQr1EUg=; h=Subject:To:References:From:Date:In-Reply-To; b=ZPZTivCXST0p4tt+mU7Pn9ixExpeCCT9vNeWGva7R2VXsM4j3GIw2mvWSLCgd5/zH xqTu2h23tS2gkRR+aedyfgQ3lQtVHhBSqGqhuVC0hEWdU2Q4f48ew8tGZTxltG7qwQ 2YLVu25o1Ue+KTgiuudXpZ67gXcgdt8Pb/AtCrmtUuK2nga6qmpLfn+GJF6S8Arjge X5yUOAIcJfu5LFt7NXNlWYzCiymfBVXcV5A6h3dV3BlySuZAXe404qeJkq3wcQH4lK bLL2YXF1grqV4y1kzH3lCJdkBel/1VHbIqv25W9dxwjz0cuD0vTQfYb2JJdA+Fpjii VVd1zA9J2dtgA== Subject: Re: CARP stopped working after upgrade from 11 to 12 To: Pete French , freebsd-stable@freebsd.org References: From: Thomas Steen Rasmussen Message-ID: Date: Wed, 16 Jan 2019 15:31:47 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: 045B570161 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gibfest.dk header.s=default header.b=ZPZTivCX; spf=pass (mx1.freebsd.org: domain of thomas@gibfest.dk designates 2a01:3a0:1:1900:89:233:43:78 as permitted sender) smtp.mailfrom=thomas@gibfest.dk X-Spamd-Result: default: False [-2.17 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gibfest.dk:s=default]; NEURAL_HAM_MEDIUM(-0.92)[-0.921,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-0.99)[-0.987,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[gibfest.dk]; NEURAL_SPAM_SHORT(0.25)[0.254,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gibfest.dk:+]; RCPT_COUNT_TWO(0.00)[2]; MX_GOOD(-0.01)[mail.tyknet.dk]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-0.00)[country: DK(-0.02)]; ASN(0.00)[asn:9167, ipnet:2a01:3a0::/32, country:DK]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jan 2019 14:31:54 -0000 On 1/16/19 3:14 PM, Pete French wrote: > I just upgraded my pair of firewalls from 11 to 12, and am now in the > situation where CARP no longer works between them to faiilover the > virtual addresse. Both machines come up thinking that they > are the master. If I manually set the advskew on the interfaces to > a high number on what should be passive then it briefly goes to backup > mode, but then goes back to master with the message: > > BACKUP -> MASTER (preempting a slower master) > > This is kind of a big problem! Indeed. I am seeing the same thing. Which revision of 12 are you running? I am currently (yesterday and today) bisecting revisions to find the commit which broke this, because it worked in 12-BETA2 but doesn't work on latest 12-STABLE. I have narrowed it down to somewhere between 12-STABLE-342037 which works, and 12-STABLE-342055 which does not. Only 4 commits touch 12-STABLE branch in that range: ------------------------------------------------------------------------ r342038 | eugen | 2018-12-13 10:52:40 +0000 (Thu, 13 Dec 2018) | 5 lines MFC r340394: ipfw.8: Fix part of the SYNOPSIS documenting LIST OF RULES AND PREPROCESSING that is still referred as last section of the SYNOPSIS later but was erroneously situated in the section IN-KERNEL NAT. ------------------------------------------------------------------------ r342047 | markj | 2018-12-13 15:51:07 +0000 (Thu, 13 Dec 2018) | 3 lines MFC r341638: Let kern.trap_enotcap be set as a tunable. ------------------------------------------------------------------------ r342048 | markj | 2018-12-13 16:07:35 +0000 (Thu, 13 Dec 2018) | 3 lines MFC r340405: Add accounting to per-domain UMA full bucket caches. ------------------------------------------------------------------------ r342051 | kp | 2018-12-13 20:00:11 +0000 (Thu, 13 Dec 2018) | 20 lines pfsync: Performance improvement pfsync code is called for every new state, state update and state deletion in pf. While pf itself can operate on multiple states at the same time (on different cores, assuming the states hash to a different hashrow), pfsync only had a single lock. This greatly reduced throughput on multicore systems. Address this by splitting the pfsync queues into buckets, based on the state id. This ensures that updates for a given connection always end up in the same bucket, which allows pfsync to still collapse multiple updates into one, while allowing multiple cores to proceed at the same time. The number of buckets is tunable, but defaults to 2 x number of cpus. Benchmarking has shown improvement, depending on hardware and setup, from ~30% to ~100%. Sponsored by:   Orange Business Services ------------------------------------------------------------------------ Of these I thought r342051 sounded most likely, so I am currently building r342050. I will write again in a few hours when I have isolated the commit. Best regards, Thomas Steen Rasmussen