From owner-freebsd-stable@freebsd.org Wed Jan 16 14:53:41 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3CC36148AE09 for ; Wed, 16 Jan 2019 14:53:41 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-ed1-x52f.google.com (mail-ed1-x52f.google.com [IPv6:2a00:1450:4864:20::52f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4134970D64 for ; Wed, 16 Jan 2019 14:53:40 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-ed1-x52f.google.com with SMTP id b14so5629506edt.6 for ; Wed, 16 Jan 2019 06:53:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:references:from:message-id:date:user-agent:mime-version :in-reply-to:content-language; bh=ZybVJdPxOByna08W9ngpU95EzOQ6ywzePRUI7noGQo4=; b=NWkjQRMYMWWkFUuQTaunCSa7m+LgjfWLSBpnPU9swvkj8xwcAuu/rl9TQF3I8qXwVI kHFOq28t/XSnoIG7HlFNL0JXCidwRQkTiO3jG/kWAuLcDToEc5ftE+2mRkSRfLdcxc/y b+ICiibdQW17bAQVVTzDh0VYo7DH/r02PhSxQk8I9Eezljhp7QIdkubs5miYIkZR8/FD 79gf3BkXDxJVAMhqHVICQlX0QHPsU80PrP38B9MByT9DHCXEnIfNk/pM6A8AoQR9Ad8k VXBw+Cgm/z7mqd23UYbhw7DdRdOwPoLpW9dNtQdUpiVqk6z6osZlNyqIFmoktynIf3u/ IzTQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=ZybVJdPxOByna08W9ngpU95EzOQ6ywzePRUI7noGQo4=; b=TRUYpYo0WOSa7MYpaInH71zkJDJJy9nMv4YWuOUYqLe9bb6qd816rJ+F0h1vg0ohzX kaVQA1pUdqIEElkpYqEj4JnFD9//sEuw23CUgILfYF38jDzTwQ6C5KpDAv/34yCTHjrU G+SwEOsI/dqVzn4GjdcXTTgP4wXI+3MyMtPwahIZ8oKb3QFMuZPpjlVQQ4eAuEU5/tdF /T1h6BSLp4uUXcbSayYBT1SG/C119pSxSYvY7M7++lVwklWgSVCa1IGSKmmbO/LCuP4f M+8YhIhtU3cckEuAlY06nlt5QXKS9wGelxyZ0oE8D72bsNCov1Z15EDW5VYrYQd7Fap2 VG+g== X-Gm-Message-State: AJcUukcI/flirdRCnaMsu5lPD1YE2k9pNcyovhpaPQNMk6NcsS0Wx0ge 9rRPhAEfSEuVfO7ZGBo5anc6YW8pa/UXew== X-Google-Smtp-Source: ALg8bN4qqIPRmkC3tGib4S7obrucBbtyHieTOOeZf0bGDYNVsXseMEwzVRcA5EobE/OtnCiwO0sDvw== X-Received: by 2002:a17:906:cb2:: with SMTP id k18-v6mr7161893ejh.129.1547650418436; Wed, 16 Jan 2019 06:53:38 -0800 (PST) Received: from [10.44.128.75] ([161.12.40.153]) by smtp.gmail.com with ESMTPSA id n10sm5493043edq.33.2019.01.16.06.53.37 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 16 Jan 2019 06:53:37 -0800 (PST) Subject: Re: CARP stopped working after upgrade from 11 to 12 To: Thomas Steen Rasmussen , Pete French , freebsd-stable@freebsd.org References: From: Steven Hartland Message-ID: Date: Wed, 16 Jan 2019 14:53:38 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: Content-Language: en-US X-Rspamd-Queue-Id: 4134970D64 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=multiplay-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=NWkjQRMY; spf=pass (mx1.freebsd.org: domain of killing@multiplay.co.uk designates 2a00:1450:4864:20::52f as permitted sender) smtp.mailfrom=killing@multiplay.co.uk X-Spamd-Result: default: False [-6.08 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[multiplay.co.uk]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[multiplay-co-uk.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[cached: ASPMX.L.GOOGLE.COM]; RCVD_IN_DNSWL_NONE(0.00)[f.2.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.99)[-0.995,0]; IP_SCORE(-2.58)[ip: (-8.98), ipnet: 2a00:1450::/32(-2.04), asn: 15169(-1.80), country: US(-0.08)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Jan 2019 14:53:41 -0000 I can't see how any of those would impact carp unless pf is now incorrectly blocking carp packets, which seems unlikely from that commit. Questions: * Are you running a firewall? * What does sysctl net.inet.carp report? * What exactly does ifconfig report about your carp on both hosts? * Have you tried enabling more detailed carp logging using sysctl net.inet.carp.log?     Regards     Steve On 16/01/2019 14:31, Thomas Steen Rasmussen wrote: > On 1/16/19 3:14 PM, Pete French wrote: >> I just upgraded my pair of firewalls from 11 to 12, and am now in the >> situation where CARP no longer works between them to faiilover the >> virtual addresse. Both machines come up thinking that they >> are the master. If I manually set the advskew on the interfaces to >> a high number on what should be passive then it briefly goes to backup >> mode, but then goes back to master with the message: >> >>     BACKUP -> MASTER (preempting a slower master) >> >> This is kind of a big problem! > > Indeed. I am seeing the same thing. Which revision of 12 are you running? > > I am currently (yesterday and today) bisecting revisions to find the > commit which broke this, because it worked in 12-BETA2 but doesn't > work on latest 12-STABLE. > > I have narrowed it down to somewhere between 12-STABLE-342037 which > works, and 12-STABLE-342055 which does not. > > Only 4 commits touch 12-STABLE branch in that range: > > ------------------------------------------------------------------------ > r342038 | eugen | 2018-12-13 10:52:40 +0000 (Thu, 13 Dec 2018) | 5 lines > > MFC r340394: ipfw.8: Fix part of the SYNOPSIS documenting > LIST OF RULES AND PREPROCESSING that is still referred > as last section of the SYNOPSIS later but was erroneously situated > in the section IN-KERNEL NAT. > > ------------------------------------------------------------------------ > r342047 | markj | 2018-12-13 15:51:07 +0000 (Thu, 13 Dec 2018) | 3 lines > > MFC r341638: > Let kern.trap_enotcap be set as a tunable. > > ------------------------------------------------------------------------ > r342048 | markj | 2018-12-13 16:07:35 +0000 (Thu, 13 Dec 2018) | 3 lines > > MFC r340405: > Add accounting to per-domain UMA full bucket caches. > > ------------------------------------------------------------------------ > r342051 | kp | 2018-12-13 20:00:11 +0000 (Thu, 13 Dec 2018) | 20 lines > > pfsync: Performance improvement > > pfsync code is called for every new state, state update and state > deletion in pf. While pf itself can operate on multiple states at the > same time (on different cores, assuming the states hash to a different > hashrow), pfsync only had a single lock. > This greatly reduced throughput on multicore systems. > > Address this by splitting the pfsync queues into buckets, based on the > state id. This ensures that updates for a given connection always end up > in the same bucket, which allows pfsync to still collapse multiple > updates into one, while allowing multiple cores to proceed at the same > time. > > The number of buckets is tunable, but defaults to 2 x number of cpus. > Benchmarking has shown improvement, depending on hardware and setup, > from ~30% > to ~100%. > > Sponsored by:   Orange Business Services > > ------------------------------------------------------------------------ > > Of these I thought r342051 sounded most likely, so I am currently > building r342050. > > I will write again in a few hours when I have isolated the commit. > > Best regards, > > Thomas Steen Rasmussen > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"