From owner-freebsd-stable@freebsd.org Sun Dec 22 06:18:16 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 351821C9AEA for ; Sun, 22 Dec 2019 06:18:16 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) Received: from kabab.cs.huji.ac.il (kabab.cs.huji.ac.il [132.65.116.210]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47gXNQ69Jrz45ND for ; Sun, 22 Dec 2019 06:18:14 +0000 (UTC) (envelope-from danny@cs.huji.ac.il) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=cs.huji.ac.il; s=57791128; h=To:References:Message-Id:Content-Transfer-Encoding:Cc:Date:In-Reply-To:From:Subject:Mime-Version:Content-Type; bh=oCHr4N3YunchyBOu6VOLNaRlOweK7dC3TyloGwjYh4k=; b=YD69MFH0rPYjgiY6E+t+rAe+YPg66cPacehgQ4LqXfCvY9eh+S/pExmRTatirC+btASJNiPRxj6bOPFw6H97Jv/pccGe8r+uZOjtoH4VzDYIXsfdFCmZ6Pr5Q4T4D5PuHEEL3fA1/amceoK6ER5TzXlFA1cxABc0WuX3QQKupUeQOBJuSZpVhaGNK+MOYmxWgvFumfgY3pcvv2+7yeR9btRbINFzM2hxbfotjek7+pIS5yMqNDfCgq//prlu23NFag/miTV3aAWfux2bEfv1PG5HCBlxYuzf62A0N56l/8v+X6LkAYjEMOK/y5asNFoRxD9VY1TtJzlNLQHqHU9fBw==; Received: from bach.cs.huji.ac.il ([132.65.80.20]) by kabab.cs.huji.ac.il with esmtp id 1iiuZ6-000B7H-Hu; Sun, 22 Dec 2019 08:18:08 +0200 Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: nfs lockd errors after NetApp software upgrade. From: Daniel Braniss In-Reply-To: Date: Sun, 22 Dec 2019 08:18:08 +0200 Cc: Adam McDougall , "freebsd-stable@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> <8A78F67B-C244-45CF-B9BF-D7062669B33B@cs.huji.ac.il> To: Rick Macklem X-Mailer: Apple Mail (2.3445.9.1) X-Rspamd-Queue-Id: 47gXNQ69Jrz45ND X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cs.huji.ac.il header.s=57791128 header.b=YD69MFH0; dmarc=pass (policy=none) header.from=huji.ac.il; spf=none (mx1.freebsd.org: domain of danny@cs.huji.ac.il has no SPF policy when checking 132.65.116.210) smtp.mailfrom=danny@cs.huji.ac.il X-Spamd-Result: default: False [-3.96 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[cs.huji.ac.il:s=57791128]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-1.66)[ip: (-4.11), ipnet: 132.64.0.0/13(-2.34), asn: 378(-1.87), country: IL(0.05)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[cs.huji.ac.il:+]; DMARC_POLICY_ALLOW(-0.50)[huji.ac.il,none]; RCVD_IN_DNSWL_NONE(0.00)[210.116.65.132.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:378, ipnet:132.64.0.0/13, country:IL]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Dec 2019 06:18:16 -0000 > On 21 Dec 2019, at 19:32, Rick Macklem wrote: >=20 > Daniel Braniss wrote: >>> On 20 Dec 2019, at 19:19, Rick Macklem = >>> wrote: >>>=20 >>> Adam McDougall wrote: >>>> Try changing bool_t do_tcp =3D FALSE; to TRUE in >>>> /usr/src/sys/nlm/nlm_prot_impl.c, recompile the kernel and try = again. I >>>> think this makes it match Linux client behavior. I suspect I ran = into >>>> the same issue as you. I do think I used nolockd is a workaround >>>> temporarily. I can provide some more details if it works. >>> If this fixes the problem, please let me know. >>>=20 >>> I'm not sure I'd want to change the default, since it might break = things for >>> others, but I can definitely make it a tunable, so that people don't = need to >>> recompile a kernel to deal with it. >>>=20 >>>=20 >> great! I was just about to see how it can be done(tunable) but need = to check if it can >be done >> at any time, or just at boot time. > I haven't looked at the code, but I suspect changing it on the fly = could cause problems, > so I am inclined to make it a tunable (boot time only). my feelings too. >=20 >> thanks. >> btw, currently, from several hours of analysing the traffic, it seems = that nlm is UDP. > I assume that means you haven't tried flipping it to TCP yet. I will soon, but I have my doubts, the problem is caused my multiple = events, i.e, it happened once while I was doing svn checkout, but i have done it several times since, and no = issues. So it must be an aggregation of factors. Other hosts are reporting locks times too. danny >=20 > Please let us know how it goes, rick >=20 > danny >=20 >=20 > rick >=20 > On 12/19/19 9:21 AM, Daniel Braniss wrote: >=20 >=20 > On 19 Dec 2019, at 16:09, Rick Macklem = > wrote: >=20 > Daniel Braniss wrote: > [stuff snipped] > all mounts are nfsv3/tcp > This doesn't affect what the NLM code (rpc.lockd) uses. I honestly = don't know when > the NLM uses tcp vs udp. I think rpc.statd still uses IP broadcast at = times. > can the replay cache have any influence here? I tend to remember way = back issues > with it, >=20 > To me, it looks like a network configuration issue. > that was/is my gut feelings too, but, as far as we can tell, nothing = has changed in the network infrastructure, > the problems appeared after the NetAPP=E2=80=99s software was updated, = it was working fine till then. >=20 > the problems are also happening on freebsd 12.1 >=20 > You could capture packets (maybe when a client first starts rpc.statd = and rpc.lockd) > and then look at them in wireshark. I'd disable statup of rpc.lockd = and rpc.statd > at boot for a test client and then run something like: > # tcpdump -s 0 -s out.pcap host > - and then start rpc.statd and rpc.lockd > Then I'd look at out.pcap in wireshark (much better at decoding this = stuff than > tcpdump). I'd look for things like different reply IP addresses from = the Netapp, > which might confuse this tired old NLM protocol Sun devised in the = mid-1980s. >=20 > it=E2=80=99s going to be an interesting week end :-( >=20 > the error is also appearing on freebsd-11.2-stable, I=E2=80=99m now = checking if it=E2=80=99s also > happening on 12.1 > btw, the NetApp version is 9.3P17 > Yes. I wasn't the author of the NSM and NLM code (long ago I refused = to even > try to implement it, because I knew the protocol was badly broken) and = I avoid > fiddling with. As such, it won't have change much since around = FreeBSD7. > and we haven=E2=80=99t had any issues with it for years, so you must = have done something good >=20 > cheers, > danny >=20 >=20 > rick >=20 > cheers, > danny >=20 > rick >=20 > Cheers >=20 > Richard > (NetApp admin) >=20 > On Wed, 18 Dec 2019 at 15:46, Daniel Braniss = > wrote: >=20 >=20 > On 18 Dec 2019, at 16:55, Rick Macklem = > wrote: >=20 > Daniel Braniss wrote: >=20 > Hi, > The server with the problems is running FreeBSD 11.1 stable, it was = working fine for >several months, > but after a software upgrade of our NetAPP server it=E2=80=99s = reporting many lockd errors >and becomes catatonic, > ... > Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not = responding > Dec 18 13:11:45 moo-09 last message repeated 7 times > Dec 18 13:12:55 moo-09 last message repeated 8 times > Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is = alive again > Dec 18 13:13:10 moo-09 last message repeated 8 times > Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 194 already in queue awaiting acceptance (1 = occurrences) > Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 193 already in queue awaiting acceptance (3957 = occurrences) > Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: = Listen queue >overflow: 193 already in queue awaiting acceptance =E2=80=A6= > Seems like their software upgrade didn't improve handling of NLM RPCs? > Appears to be handling RPCs slowly and/or intermittently. Note that no = one > tests it with IPv6, so at least make sure you are still using IPv4 for = the mounts and > try and make sure IP broadcast works between client and Netapp. I = think the NLM > and NSM (rpc.statd) still use IP broadcast sometimes. >=20 > we are ipv4 - we have our own class c :-) > Maybe the network guys can suggest more w.r.t. why, but as I've stated = before, > the NLM is a fundamentally broken protocol which was never published = by Sun, > so I suggest you avoid using it if at all possible. > well, at the moment the ball is on NetAPP court, and switching to = NFSv4 at the moment is out of the question, it=E2=80=99s > a production server used by several thousand students. >=20 >=20 > - If the locks don't need to be seen by other clients, you can just = use the "nolockd" > mount option. > or > - If locks need to be seen by other clients, try NFSv4 mounts. Netapp = filers > should support NFSv4.1, which is a much better protocol that NFSv4.0. >=20 > Good luck with it, rick > thanks > danny >=20 > =E2=80=A6 > any ideas? >=20 > thanks, > danny >=20 > _______________________________________________ > = freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >=20 > _______________________________________________ > = freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >=20 >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing = list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >=20 >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing = list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-stable@freebsd.org mailing = list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >=20 From owner-freebsd-stable@freebsd.org Sun Dec 22 17:01:29 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 357B91D7766 for ; Sun, 22 Dec 2019 17:01:29 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660056.outbound.protection.outlook.com [40.107.66.56]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47gpfc35yHz4ZV8 for ; Sun, 22 Dec 2019 17:01:28 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=IzTx503JEXNDmcababWz38EKJdD+/dn78iK6NTkEKz2Moj3tERnMw0uDSdS7l2W4sDLMNhhQtXx9wgmRlkrqiUBSHIok9rNNT0wLJvO2Jxo9b5N2CRsZSNQRs3ZQDj4E+IETUtsoR5g/lQ9CMoV5lh2bmWm/BdZA5KMtJhyAGdt/ZwSwMRlrShiKfvbqyci/6olzWBcGF5fG6XSYta52hV0bQJ3JhCdy2RK4031mAwNu1hu9u4kHBYEIck9DDauMDc+RtltSF8RRLaAI9QhtGTmnfcFe97/m/Ts/TVjwVZJXav8kXf6OAR1NITDfg6wmkUvAOYzdP9CccD5NnZuglw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=FJz7uBOnfOW+W3b2XR+QP4mjtEfQFiU/bFewya/fK9Q=; b=ZA8MvEsWgzdXItQRFDHOrYGoQjoOZVvUOUnON1owCTFSxYg8NMlbeWbpd9ThEi2s1WgzAHTjioQvt1FuwRLssnMk+WJzNC09p6cRFgRnF82pYdJ/eIepN5PHmYkqmSB1+Ci7Pahs4HEegSyx7MqBJZFDIQz18kmGMEb31MGwHbnzONZjKCH0HKOtBlVX2VPDozpAwrWRNKwz13tmsJWSPf1Bv0wllcDqHeN2ckOAhs8RkonrORrQ343mWrOL6GbH5zadDAdP6YZkbyCuTPu/NBTu3jTSHCv/FpvJKKXF+S/1wJPIxCnttEOTyu8ZpBQVecDp38PUiH9lUL7j5l+u+A== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM (52.132.69.153) by YQBPR0101MB1252.CANPRD01.PROD.OUTLOOK.COM (52.132.72.141) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.16; Sun, 22 Dec 2019 17:01:24 +0000 Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75]) by YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75%5]) with mapi id 15.20.2538.023; Sun, 22 Dec 2019 17:01:24 +0000 From: Rick Macklem To: Daniel Braniss CC: Adam McDougall , "freebsd-stable@freebsd.org" Subject: Re: nfs lockd errors after NetApp software upgrade. Thread-Topic: nfs lockd errors after NetApp software upgrade. Thread-Index: AQHVtawq+ga5QLcdVkqBDG/GW9zFg6e/+Am+gAARTACAAANHAIAAi7Y3gACf34CAAEVO6IAABk4AgADWGACAAO1eZYAA7uGAgACmPw2AANdsAIAAsCi6 Date: Sun, 22 Dec 2019 17:01:24 +0000 Message-ID: References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> <8A78F67B-C244-45CF-B9BF-D7062669B33B@cs.huji.ac.il> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 46953a5e-7daf-48c3-c256-08d7870093e9 x-ms-traffictypediagnostic: YQBPR0101MB1252: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:4125; x-forefront-prvs: 02596AB7DA x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(136003)(346002)(366004)(396003)(376002)(189003)(199004)(55016002)(5660300002)(66446008)(54906003)(53546011)(478600001)(66946007)(966005)(66556008)(64756008)(66476007)(6506007)(2906002)(52536014)(9686003)(4326008)(6916009)(7696005)(33656002)(316002)(786003)(71200400001)(76116006)(86362001)(81156014)(8676002)(81166006)(186003)(8936002); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB1252; H:YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: lmLn9FT5e3FOERjC+hUL4EBxxWzuPYSSvIrbqhHRFfkvcdIYTSutgIuAavuVvqIrREGadair0WbjTUaCZdhPaD9ZYqF6s+fx2fcZjr/CwaAUJp8lAOwKr7dXzSqiwTSXwra7W0rI9yP781nP4SGV18VPR2b5ncEpIeaYm2cOKUzfcw0InrsInnI4jWJ0AysPZ3Jh1LdcJ2vg3Ue0NTqekCYxbTOdrQBPNQdD48p8qeJksC8KWssTBpZuw9pD1cBmWuwGZX7YI1rFFGdZ7qyUhDih+f23rXitbKK1Ye6vSEGLyrRldzryoUUBg3kZ8AxrOIWGJXoSd+EueppSbZZlWHHKgOx/zsM9J1y0jKUoLDFtW7uHn5+nJZxqX3iXNG0Dg33UyzgoZAyMyiLp6VNvf32lfGL7s8RFlDw23lIKgP8uNGajxSrer7PXQULOs1XNKETbnpPUawmHnywYDJpKSrelouy5o5t3ESRp2aApbzs= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 46953a5e-7daf-48c3-c256-08d7870093e9 X-MS-Exchange-CrossTenant-originalarrivaltime: 22 Dec 2019 17:01:24.5835 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: LVjDWtCObw32wSqgLMU91jc6LGOlfoB0aP0U+yNCw0756WwVK0wJ8uKi0kAeM736OJeItq1iCoSLXtJZ59i1BQ== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB1252 X-Rspamd-Queue-Id: 47gpfc35yHz4ZV8 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.56 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.66 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[uoguelph.ca]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[56.66.107.40.list.dnswl.org : 127.0.3.0]; IP_SCORE(-1.36)[ipnet: 40.64.0.0/10(-3.84), asn: 8075(-2.93), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Dec 2019 17:01:29 -0000 Daniel Braniss wrote:=0A= >> On 21 Dec 2019, at 19:32, Rick Macklem wrote:=0A= >>=0A= >> Daniel Braniss wrote:=0A= >>>> On 20 Dec 2019, at 19:19, Rick Macklem >>>> wrote:=0A= >>>>=0A= >>>> Adam McDougall wrote:=0A= >>>>> Try changing bool_t do_tcp =3D FALSE; to TRUE in=0A= >>>>> /usr/src/sys/nlm/nlm_prot_impl.c, recompile the kernel and try again.= I=0A= >>>>> think this makes it match Linux client behavior. I suspect I ran into= =0A= >>>>> the same issue as you. I do think I used nolockd is a workaround=0A= >>>>> temporarily. I can provide some more details if it works.=0A= >>>> If this fixes the problem, please let me know.=0A= >>>>=0A= >>>> I'm not sure I'd want to change the default, since it might break thin= gs for=0A= >>>> others, but I can definitely make it a tunable, so that people don't n= eed to=0A= >>>> recompile a kernel to deal with it.=0A= >>>>=0A= >>>>=0A= >>> great! I was just about to see how it can be done(tunable) but need to = check if it can >be done=0A= >>> at any time, or just at boot time.=0A= >> I haven't looked at the code, but I suspect changing it on the fly could= cause problems,=0A= >> so I am inclined to make it a tunable (boot time only).=0A= my feelings too.=0A= >>=0A= >>> thanks.=0A= >>> btw, currently, from several hours of analysing the traffic, it seems t= hat nlm is UDP.=0A= >> I assume that means you haven't tried flipping it to TCP yet.=0A= >I will soon, but I have my doubts, the problem is caused my multiple event= s, i.e, it >happened once while=0A= >I was doing svn checkout, but i have done it several times since, and no i= ssues. So it >must be=0A= >an aggregation of factors. Other hosts are reporting locks times too.=0A= Well, I've noted the flawed protocol. Here's an example (from my limited un= derstanding of these protocols, where there has never been a published spec= ) :=0A= - The NLM supports a "blocking lock request" that goes something like this.= ..=0A= - client requests lock and is willing to wait for it=0A= - if server has a conflicting lock on the file, it replies "I'll acquire= the lock for=0A= you when I can and let you know".=0A= --> When the conflicting lock is released, the server acquires the loc= k and does=0A= a callback (server->client RPC) to tell the client it now has t= he lock.=0A= You don't have to think about this for long to realize that any network unr= eliability=0A= or partitioning could result in trouble.=0A= The kernel RPC layer may do some retries of the RPCs (this is controlled by= the=0A= parameters set for the RPC), but at some point the protocol asks the NSM=0A= (rpc.statd) if the machine is "up" and then uses the NSM's answer to deal w= ith it.=0A= (The NSM basically pokes other systems and notes they are "up" if they get= =0A= replies to these pokes. It uses IP broadcast at some point.)=0A= =0A= Now, maybe switching to TCP will make the RPCs reliable enough that it will= =0A= work, or maybe it won't? (It certainly sounds like the Netapp upgrade is ca= using=0A= some kind of network issue, and the NLM doesn't tolerate that well.)=0A= =0A= rick=0A= =0A= danny=0A= =0A= >=0A= > Please let us know how it goes, rick=0A= >=0A= > danny=0A= >=0A= >=0A= > rick=0A= >=0A= > On 12/19/19 9:21 AM, Daniel Braniss wrote:=0A= >=0A= >=0A= > On 19 Dec 2019, at 16:09, Rick Macklem > wrote:=0A= >=0A= > Daniel Braniss wrote:=0A= > [stuff snipped]=0A= > all mounts are nfsv3/tcp=0A= > This doesn't affect what the NLM code (rpc.lockd) uses. I honestly don't = know when=0A= > the NLM uses tcp vs udp. I think rpc.statd still uses IP broadcast at tim= es.=0A= > can the replay cache have any influence here? I tend to remember way back= issues=0A= > with it,=0A= >=0A= > To me, it looks like a network configuration issue.=0A= > that was/is my gut feelings too, but, as far as we can tell, nothing has = changed in the network infrastructure,=0A= > the problems appeared after the NetAPP=92s software was updated, it was w= orking fine till then.=0A= >=0A= > the problems are also happening on freebsd 12.1=0A= >=0A= > You could capture packets (maybe when a client first starts rpc.statd and= rpc.lockd)=0A= > and then look at them in wireshark. I'd disable statup of rpc.lockd and r= pc.statd=0A= > at boot for a test client and then run something like:=0A= > # tcpdump -s 0 -s out.pcap host =0A= > - and then start rpc.statd and rpc.lockd=0A= > Then I'd look at out.pcap in wireshark (much better at decoding this stuf= f than=0A= > tcpdump). I'd look for things like different reply IP addresses from the = Netapp,=0A= > which might confuse this tired old NLM protocol Sun devised in the mid-19= 80s.=0A= >=0A= > it=92s going to be an interesting week end :-(=0A= >=0A= > the error is also appearing on freebsd-11.2-stable, I=92m now checking if= it=92s also=0A= > happening on 12.1=0A= > btw, the NetApp version is 9.3P17=0A= > Yes. I wasn't the author of the NSM and NLM code (long ago I refused to e= ven=0A= > try to implement it, because I knew the protocol was badly broken) and I = avoid=0A= > fiddling with. As such, it won't have change much since around FreeBSD7.= =0A= > and we haven=92t had any issues with it for years, so you must have done = something good=0A= >=0A= > cheers,=0A= > danny=0A= >=0A= >=0A= > rick=0A= >=0A= > cheers,=0A= > danny=0A= >=0A= > rick=0A= >=0A= > Cheers=0A= >=0A= > Richard=0A= > (NetApp admin)=0A= >=0A= > On Wed, 18 Dec 2019 at 15:46, Daniel Braniss > wrote:=0A= >=0A= >=0A= > On 18 Dec 2019, at 16:55, Rick Macklem > wrote:=0A= >=0A= > Daniel Braniss wrote:=0A= >=0A= > Hi,=0A= > The server with the problems is running FreeBSD 11.1 stable, it was worki= ng fine for >several months,=0A= > but after a software upgrade of our NetAPP server it=92s reporting many l= ockd errors >and becomes catatonic,=0A= > ...=0A= > Dec 18 13:11:02 moo-09 kernel: nfs server fr-06:/web/www: lockd not respo= nding=0A= > Dec 18 13:11:45 moo-09 last message repeated 7 times=0A= > Dec 18 13:12:55 moo-09 last message repeated 8 times=0A= > Dec 18 13:13:10 moo-09 kernel: nfs server fr-06:/web/www: lockd is alive = again=0A= > Dec 18 13:13:10 moo-09 last message repeated 8 times=0A= > Dec 18 13:13:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue >overflow: 194 already in queue awaiting acceptance (1 occurrences)= =0A= > Dec 18 13:14:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue >overflow: 193 already in queue awaiting acceptance (3957 occurrences= )=0A= > Dec 18 13:15:29 moo-09 kernel: sonewconn: pcb 0xfffff8004cc051d0: Listen = queue >overflow: 193 already in queue awaiting acceptance =85=0A= > Seems like their software upgrade didn't improve handling of NLM RPCs?=0A= > Appears to be handling RPCs slowly and/or intermittently. Note that no on= e=0A= > tests it with IPv6, so at least make sure you are still using IPv4 for th= e mounts and=0A= > try and make sure IP broadcast works between client and Netapp. I think t= he NLM=0A= > and NSM (rpc.statd) still use IP broadcast sometimes.=0A= >=0A= > we are ipv4 - we have our own class c :-)=0A= > Maybe the network guys can suggest more w.r.t. why, but as I've stated be= fore,=0A= > the NLM is a fundamentally broken protocol which was never published by S= un,=0A= > so I suggest you avoid using it if at all possible.=0A= > well, at the moment the ball is on NetAPP court, and switching to NFSv4 a= t the moment is out of the question, it=92s=0A= > a production server used by several thousand students.=0A= >=0A= >=0A= > - If the locks don't need to be seen by other clients, you can just use t= he "nolockd"=0A= > mount option.=0A= > or=0A= > - If locks need to be seen by other clients, try NFSv4 mounts. Netapp fil= ers=0A= > should support NFSv4.1, which is a much better protocol that NFSv4.0.=0A= >=0A= > Good luck with it, rick=0A= > thanks=0A= > danny=0A= >=0A= > =85=0A= > any ideas?=0A= >=0A= > thanks,=0A= > danny=0A= >=0A= > _______________________________________________=0A= > freebsd-stable@freebsd.org mailing list=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org<= mailto:freebsd-stable-unsubscribe@freebsd.org>"=0A= >=0A= > _______________________________________________=0A= > freebsd-stable@freebsd.org mailing list=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org<= mailto:freebsd-stable-unsubscribe@freebsd.org>"=0A= >=0A= >=0A= > _______________________________________________=0A= > freebsd-stable@freebsd.org mailing lis= t=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"= =0A= >=0A= >=0A= > _______________________________________________=0A= > freebsd-stable@freebsd.org mailing lis= t=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org<= mailto:freebsd-stable-unsubscribe@freebsd.org>"=0A= > _______________________________________________=0A= > freebsd-stable@freebsd.org mailing lis= t=0A= > https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org<= mailto:freebsd-stable-unsubscribe@freebsd.org>"=0A= >=0A= =0A= _______________________________________________=0A= freebsd-stable@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"= =0A= From owner-freebsd-stable@freebsd.org Mon Dec 23 13:55:21 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 279661C8643 for ; Mon, 23 Dec 2019 13:55:21 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47hLTN2MYlz3CrS for ; Mon, 23 Dec 2019 13:55:19 +0000 (UTC) (envelope-from slw@zxy.spb.ru) Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD)) (envelope-from ) id 1ijOAw-000PXD-3B; Mon, 23 Dec 2019 16:55:10 +0300 Date: Mon, 23 Dec 2019 16:55:10 +0300 From: Slawa Olhovchenkov To: Ryan Stone Cc: freebsd-stable@freebsd.org Subject: Re: Access to NETMAP from c++ program Message-ID: <20191223135509.GE38096@zxy.spb.ru> References: <20191119213319.GD38096@zxy.spb.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.5.24 (2015-08-30) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false X-Rspamd-Queue-Id: 47hLTN2MYlz3CrS X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of slw@zxy.spb.ru has no SPF policy when checking 195.70.199.98) smtp.mailfrom=slw@zxy.spb.ru X-Spamd-Result: default: False [-0.30 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.84)[-0.845,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-0.45)[-0.445,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[zxy.spb.ru]; AUTH_NA(1.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:5495, ipnet:195.70.192.0/19, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.09)[asn: 5495(0.44), country: RU(0.01)]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Dec 2019 13:55:21 -0000 On Mon, Nov 25, 2019 at 03:36:21PM -0500, Ryan Stone wrote: > Remove "using namespace std;" from your program. I am don't have "using namespace std;". Example: === #include #include #include #include #include #include #include #include #include #include #include === Yes, only includes. c++ -c tatomic.cc In file included from tatomic.cc:11: In file included from /usr/include/net/netmap_user.h:104: In file included from /usr/include/net/netmap.h:816: /usr/include/stdatomic.h:186:17: error: unknown type name '_Bool' typedef _Atomic(_Bool) atomic_bool; ^ /usr/include/stdatomic.h:186:26: error: C++ requires a type specifier for all declarations typedef _Atomic(_Bool) atomic_bool; ~~~~~~ ^ /usr/include/stdatomic.h:379:17: error: unknown type name '_Bool' static __inline _Bool ^ /usr/include/stdatomic.h:383:10: error: address argument to atomic operation must be a pointer to _Atomic type ('volatile atomic_bool *' (aka 'volatile int *') invalid) return (atomic_exchange_explicit(&__object->__flag, 1, __order)); ^ ~~~~~~~~~~~~~~~~~ /usr/include/stdatomic.h:242:2: note: expanded from macro 'atomic_exchange_explicit' __c11_atomic_exchange(object, desired, order) ^ ~~~~~~ /usr/include/stdatomic.h:390:2: error: address argument to atomic operation must be a pointer to _Atomic type ('volatile atomic_bool *' (aka 'volatile int *') invalid) atomic_store_explicit(&__object->__flag, 0, __order); ^ ~~~~~~~~~~~~~~~~~ /usr/include/stdatomic.h:256:2: note: expanded from macro 'atomic_store_explicit' __c11_atomic_store(object, desired, order) ^ ~~~~~~ /usr/include/stdatomic.h:394:17: error: unknown type name '_Bool' static __inline _Bool ^ 6 errors generated. Ok, try ugly hack for _Bool: === #include #include #include #include #include #include #include #include #include #include typedef int _Bool; #include === No, errors. Now try === #include #include #include #include #include #include #include #include #include #include typedef int _Bool; #include #include #include === Many errors: In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1166:49: error: expected ')' atomic_is_lock_free(const volatile atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/c++/v1/atomic:1166:1: note: to match this '(' atomic_is_lock_free(const volatile atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/stdatomic.h:176:32: note: expanded from macro 'atomic_is_lock_free' __atomic_is_lock_free(sizeof(*(obj)), obj) ^ In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1166:1: error: expected expression atomic_is_lock_free(const volatile atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/stdatomic.h:176:37: note: expanded from macro 'atomic_is_lock_free' __atomic_is_lock_free(sizeof(*(obj)), obj) ^ In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1166:21: error: expected expression atomic_is_lock_free(const volatile atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/c++/v1/atomic:1166:53: error: expected ';' at end of declaration atomic_is_lock_free(const volatile atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/c++/v1/atomic:1166:54: error: expected unqualified-id atomic_is_lock_free(const volatile atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/c++/v1/__config:839:21: note: expanded from macro '_NOEXCEPT' # define _NOEXCEPT noexcept ^ In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1174:1: error: redefinition of '__atomic_is_lock_free' atomic_is_lock_free(const atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/stdatomic.h:176:2: note: expanded from macro 'atomic_is_lock_free' __atomic_is_lock_free(sizeof(*(obj)), obj) ^ /usr/include/c++/v1/atomic:1166:1: note: previous definition is here atomic_is_lock_free(const volatile atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/stdatomic.h:176:2: note: expanded from macro 'atomic_is_lock_free' __atomic_is_lock_free(sizeof(*(obj)), obj) ^ In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1174:40: error: expected ')' atomic_is_lock_free(const atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/c++/v1/atomic:1174:1: note: to match this '(' atomic_is_lock_free(const atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/stdatomic.h:176:32: note: expanded from macro 'atomic_is_lock_free' __atomic_is_lock_free(sizeof(*(obj)), obj) ^ In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1174:1: error: expected expression atomic_is_lock_free(const atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/stdatomic.h:176:37: note: expanded from macro 'atomic_is_lock_free' __atomic_is_lock_free(sizeof(*(obj)), obj) ^ In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1174:21: error: expected expression atomic_is_lock_free(const atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/c++/v1/atomic:1174:44: error: expected ';' at end of declaration atomic_is_lock_free(const atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/c++/v1/atomic:1174:45: error: expected unqualified-id atomic_is_lock_free(const atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/c++/v1/__config:839:21: note: expanded from macro '_NOEXCEPT' # define _NOEXCEPT noexcept ^ In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1202:1: error: unknown type name 'memory_order_seq_cst' atomic_store(volatile atomic<_Tp>* __o, _Tp __d) _NOEXCEPT ^ /usr/include/stdatomic.h:363:41: note: expanded from macro 'atomic_store' atomic_store_explicit(object, desired, memory_order_seq_cst) ^ In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1210:1: error: unknown type name 'memory_order_seq_cst' atomic_store(atomic<_Tp>* __o, _Tp __d) _NOEXCEPT ^ /usr/include/stdatomic.h:363:41: note: expanded from macro 'atomic_store' atomic_store_explicit(object, desired, memory_order_seq_cst) ^ In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1240:1: error: unknown type name 'memory_order_seq_cst' atomic_load(const volatile atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/stdatomic.h:361:31: note: expanded from macro 'atomic_load' atomic_load_explicit(object, memory_order_seq_cst) ^ In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1248:1: error: unknown type name 'memory_order_seq_cst' atomic_load(const atomic<_Tp>* __o) _NOEXCEPT ^ /usr/include/stdatomic.h:361:31: note: expanded from macro 'atomic_load' atomic_load_explicit(object, memory_order_seq_cst) ^ In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1278:1: error: unknown type name 'memory_order_seq_cst' atomic_exchange(volatile atomic<_Tp>* __o, _Tp __d) _NOEXCEPT ^ /usr/include/stdatomic.h:349:44: note: expanded from macro 'atomic_exchange' atomic_exchange_explicit(object, desired, memory_order_seq_cst) ^ In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1286:1: error: unknown type name 'memory_order_seq_cst' atomic_exchange(atomic<_Tp>* __o, _Tp __d) _NOEXCEPT ^ /usr/include/stdatomic.h:349:44: note: expanded from macro 'atomic_exchange' atomic_exchange_explicit(object, desired, memory_order_seq_cst) ^ In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1314:1: error: C++ requires a type specifier for all declarations atomic_compare_exchange_weak(volatile atomic<_Tp>* __o, _Tp* __e, _Tp __d) _NOEXCEPT ^ /usr/include/stdatomic.h:347:15: note: expanded from macro 'atomic_compare_exchange_weak' desired, memory_order_seq_cst, memory_order_seq_cst) ^ In file included from tatomic.cc:13: In file included from /usr/include/c++/v1/memory:668: /usr/include/c++/v1/atomic:1314:1: error: unknown type name 'memory_order_seq_cst' /usr/include/stdatomic.h:347:37: note: expanded from macro 'atomic_compare_exchange_weak' desired, memory_order_seq_cst, memory_order_seq_cst) ^ fatal error: too many errors emitted, stopping now [-ferror-limit=] 20 errors generated. Try different order: === #include #include #include #include #include #include #include #include #include #include #include #include typedef int _Bool; #include === No errors, no ugly hack for _Bool required. Is this normal? From owner-freebsd-stable@freebsd.org Mon Dec 23 15:13:26 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 001481CA1D0 for ; Mon, 23 Dec 2019 15:13:25 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mail.egr.msu.edu (boomhauer.egr.msu.edu [35.9.37.164]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47hNCT17r8z3HTZ for ; Mon, 23 Dec 2019 15:13:24 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from boomhauer (localhost [127.0.0.1]) by mail.egr.msu.edu (Postfix) with ESMTP id 9FFA9EB053 for ; Mon, 23 Dec 2019 10:13:23 -0500 (EST) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mail.egr.msu.edu ([127.0.0.1]) by boomhauer (boomhauer.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id W9TQjm-bDwTJ for ; Mon, 23 Dec 2019 10:13:23 -0500 (EST) Received: from EGR authenticated sender mcdouga9 Subject: Re: nfs lockd errors after NetApp software upgrade. To: freebsd-stable@freebsd.org References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> <8A78F67B-C244-45CF-B9BF-D7062669B33B@cs.huji.ac.il> From: Adam McDougall Message-ID: Date: Mon, 23 Dec 2019 10:13:22 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:68.0) Gecko/20100101 Thunderbird/68.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 47hNCT17r8z3HTZ X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=msu.edu; spf=pass (mx1.freebsd.org: domain of mcdouga9@egr.msu.edu designates 35.9.37.164 as permitted sender) smtp.mailfrom=mcdouga9@egr.msu.edu X-Spamd-Result: default: False [-3.09 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+a:boomhauer.egr.msu.edu]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_MED(-0.20)[164.37.9.35.list.dnswl.org : 127.0.11.2]; DMARC_POLICY_ALLOW(-0.50)[msu.edu,none]; IP_SCORE(-0.09)[asn: 237(-0.40), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:237, ipnet:35.0.0.0/10, country:US]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Dec 2019 15:13:26 -0000 On 12/22/19 12:01 PM, Rick Macklem wrote: > Well, I've noted the flawed protocol. Here's an example (from my limited understanding of these protocols, where there has never been a published spec) : > - The NLM supports a "blocking lock request" that goes something like this... > - client requests lock and is willing to wait for it > - if server has a conflicting lock on the file, it replies "I'll acquire the lock for > you when I can and let you know". > --> When the conflicting lock is released, the server acquires the lock and does > a callback (server->client RPC) to tell the client it now has the lock. > You don't have to think about this for long to realize that any network unreliability > or partitioning could result in trouble. > The kernel RPC layer may do some retries of the RPCs (this is controlled by the > parameters set for the RPC), but at some point the protocol asks the NSM > (rpc.statd) if the machine is "up" and then uses the NSM's answer to deal with it. > (The NSM basically pokes other systems and notes they are "up" if they get > replies to these pokes. It uses IP broadcast at some point.) > > Now, maybe switching to TCP will make the RPCs reliable enough that it will > work, or maybe it won't? (It certainly sounds like the Netapp upgrade is causing > some kind of network issue, and the NLM doesn't tolerate that well.) > > rick tl;dr I think netapp effectively nerfed UDP lockd performance in newer versions, maybe cluster mode. >From my very un-fun experience after migrating our volumes off an older netapp onto a new netapp with flash drives (plenty fast) running Ontap 9.x ("cluster mode"), our typical IO load from idle time IMAP connections was enough to overwhelm the new netapp and drive performance into the ground. The same IO that was perfectly fine on the old netapp. Going into a workday in this state was absolutely not possible. I opened a high priority ticket with netapp, didn't really get anywhere that very long day and settled on nolockd so I could go home and sleep. Both my hunch later and netapp support suggested switching lockd traffic to TCP even though I had no network problems (the old netapp was fine). I think I still run into occasional load issues but the newer netapp OS seemed way more capable of this load when using TCP lockd. Of course they also suggested switching to nfsv4 but I could not seriously entertain validating that type of change for production in less than a day. From owner-freebsd-stable@freebsd.org Mon Dec 23 17:57:15 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 226491CD0C5 for ; Mon, 23 Dec 2019 17:57:15 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: from mail-wr1-x42c.google.com (mail-wr1-x42c.google.com [IPv6:2a00:1450:4864:20::42c]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47hRrV276Pz3Pc1 for ; Mon, 23 Dec 2019 17:57:14 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: by mail-wr1-x42c.google.com with SMTP id b6so17489157wrq.0 for ; Mon, 23 Dec 2019 09:57:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=hoUPKzwH1OR6rfvmS75upDwVihWH24L2mUVwN42vlMY=; b=eAw34ojVE3hOv9K5qkPDKpH0b2aXNVk5H8OXeCE7mlqa8FVMe6y/NAr4hCGyUmFq5w ziG3O0vKnInfmv4AoqWZ1Dbwlq+H5C/G48t1tqy3T8qaGRe8I9J9dSRK2fx+jhMAw9il 2LS6cS7e+oG5zRuVlRqL9CEIbNYKV1tzuEJ2WDgZP/iGJ1Zz0gypjJErS5fRoO2l/VWU jgj46SOTNEAcA2+jU/h5fBzGhO/gvx7MlQqcOJahAnb1WgCx4zQ7WRx3zCiv87ukUUSQ YYEsJTVdjkgyUztHpqF4SxrmQrvdscE8Qmrf8uZ/+y1y5jsDlIWMlETvx8aiZQddiD7m 4Ljw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=hoUPKzwH1OR6rfvmS75upDwVihWH24L2mUVwN42vlMY=; b=Nj5zguT5QSp78jBcd1vw6MAgVcX8r9+m18BkMFAjhl83dNksTAOs2XOmKnTCLc5F5h dBLMehRzDi0JQIvloP9y63ArxXqlrxw2h5zckWbkU6VI5CZQ65srYE4T/M/6t0k77ENp khZXorXe16aacSwE73W+m7sitcdj0wD5ZxV3/z2fT7eJZYe/svKoitO7v9CgRl55wBwV 7xaF6yFsWFrKIAzn0XDlPHQobJhk+WeGKNPJsno+5JfDuuQFSjEaRQq3W8GpphwOpKzj /npdNzfk6UdDT9CuIhGn4rd8msC2UvVQuxyZnEleCLJh5TIwDApJko0Fkn0HAPf7Bkoi AqzA== X-Gm-Message-State: APjAAAUC/paaupYofu5qx++L0oZ/T7s90552AoACNWobrOpwYn9M3qtE U5QIKnXIfsPcw01UONaz1U7Hji5ybP5FqxxehoeQXRJHcB8= X-Google-Smtp-Source: APXvYqzwMDUEzSIf8amzvyFvzS7TDohhXR2FVqF/c3Vf+Q8bq0of/yXky+zM/RiWkRrNRmIYxbiNn0CXDXEyPI1a+uk= X-Received: by 2002:adf:f411:: with SMTP id g17mr30645711wro.89.1577123832274; Mon, 23 Dec 2019 09:57:12 -0800 (PST) MIME-Version: 1.0 References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> <8A78F67B-C244-45CF-B9BF-D7062669B33B@cs.huji.ac.il> In-Reply-To: From: Richard P Mackerras Date: Mon, 23 Dec 2019 17:57:01 +0000 Message-ID: Subject: Re: nfs lockd errors after NetApp software upgrade. To: Adam McDougall Cc: freebsd-stable@freebsd.org X-Rspamd-Queue-Id: 47hRrV276Pz3Pc1 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=eAw34ojV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mack63richard@gmail.com designates 2a00:1450:4864:20::42c as permitted sender) smtp.mailfrom=mack63richard@gmail.com X-Spamd-Result: default: False [-3.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[c.2.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; IP_SCORE(0.00)[ip: (-9.10), ipnet: 2a00:1450::/32(-2.65), asn: 15169(-1.89), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 23 Dec 2019 17:57:15 -0000 Hi, We had some bully type workloads emerge when we moved a lot of block storage from old XIV to new all flash 3PAR. I wonder if your IMAP issue might have emerged just because suddenly there was the opportunity with all flash. QOS is good on 9.x ONTAP. If anyone says it=E2=80=99s not then they = last looked on 8.x. So I suggest you QOS the IMAP workload. Nobody should be using UDP with NFS unless they have a very specific set of circumstances. TCP was a real step forward. Cheers Richard From owner-freebsd-stable@freebsd.org Tue Dec 24 03:02:20 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E547B1D9483 for ; Tue, 24 Dec 2019 03:02:20 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-QB1-obe.outbound.protection.outlook.com (mail-eopbgr660087.outbound.protection.outlook.com [40.107.66.87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47hgxR5Wgfz4MPN for ; Tue, 24 Dec 2019 03:02:19 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=npMgOHJxx8k3oazY6690I7WYLKTvMVcwpDRvSPz1yceOiFHeaQkvl3ORwVNZlq47lfcgmpzBldkaiImwZ5TSjZD9+4ojX14sanG60tFP5FG0VKuRnwzwDpl9wf7Fw1baJMg9Hy+AQGPbf8LSfbGAT5s9VOOa3pSc7U84x2y6aFSdX9aoptjVh+NrGHPpYUFodAr4dsqCMXdEv+KpKSImr7Rw5dEQQSxv2hW6NBQMdsrBEH8QApdyH5ymwAljUPkUyrrLXhUhBL0LwOJoS+InrCo9LBb9Sqy2N1g8A4bvRBVtaoxBJyt/zhdyeXkI75RaPr2rjfe/gEpMVyL8czB4uQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=xdtP6UZq6AZh4rsLh0vNhPWxqrl7m3JcrFfBYJlGa1U=; b=AWLHBvwMt75ouAyzUfKN6enrtuUPhoJWVPnlbMddjzf4Dh7JjK/GVlDbuELPEbQLQR3se3f7xKwKDo/9Y0QHnjYbzr5e2w9neJ8KURtmmGMCyUZFKNfbZHgZnxBsKd/4vMkclG7nO2WcrXMR6CbjE6qRgn8GsmzFvANJB1PyFHdLH0IwkqFWrFbomW1OPF3xwICTkMb5MVvVvp0aJvKZ6PhldfEr65lOIUsTdFq4EVe7t7sr7ZcYVhq4a5xpDqQJUhuLv/gfFlw/mfFW/OYN7rUi97OX8lEGFing5tx6NlOPUqeQ3WjTfN7EuCUOX4ON4XRWd+L4SrvnPJj2WSt4Rw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM (52.132.69.153) by YQBPR0101MB0707.CANPRD01.PROD.OUTLOOK.COM (52.132.73.32) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2559.13; Tue, 24 Dec 2019 03:02:17 +0000 Received: from YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75]) by YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM ([fe80::9504:a50d:ee12:b75%5]) with mapi id 15.20.2538.023; Tue, 24 Dec 2019 03:02:17 +0000 From: Rick Macklem To: Richard P Mackerras , Adam McDougall CC: "freebsd-stable@freebsd.org" Subject: Re: nfs lockd errors after NetApp software upgrade. Thread-Topic: nfs lockd errors after NetApp software upgrade. Thread-Index: AQHVtawq+ga5QLcdVkqBDG/GW9zFg6e/+Am+gAARTACAAANHAIAAi7Y3gACf34CAAEVO6IAABk4AgADWGACAAO1eZYAA7uGAgACmPw2AANdsAIAAsCi6gAF3uACAAC25gIAAlUcY Date: Tue, 24 Dec 2019 03:02:17 +0000 Message-ID: References: <0121E289-D2AE-44BA-ADAC-4814CAEE676F@cs.huji.ac.il> <854B6E5A-C6BC-44B3-A656-FC9B8EF19881@cs.huji.ac.il> <8770BD0D-4B72-431A-B4F5-A29D4DBA03B1@cs.huji.ac.il> <8A78F67B-C244-45CF-B9BF-D7062669B33B@cs.huji.ac.il> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 792a707e-385b-4897-8f6d-08d7881daf9f x-ms-traffictypediagnostic: YQBPR0101MB0707: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:10000; x-forefront-prvs: 0261CCEEDF x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(396003)(376002)(39860400002)(136003)(346002)(366004)(199004)(189003)(54094003)(66556008)(64756008)(81156014)(81166006)(7696005)(2906002)(8936002)(33656002)(86362001)(4326008)(52536014)(66446008)(66476007)(66946007)(8676002)(5660300002)(76116006)(71200400001)(110136005)(55016002)(6506007)(9686003)(316002)(478600001)(966005)(26005)(786003)(186003); DIR:OUT; SFP:1101; SCL:1; SRVR:YQBPR0101MB0707; H:YQBPR0101MB1427.CANPRD01.PROD.OUTLOOK.COM; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 5WrSspnVaQgxX5qiB9+eKMvjJO2Pd7cOOlg6DA13ORS12bSZs31cl4yfCDGD8k+nuIc2uCjlSxkiYLNpRAALc5ZxFG4OIeGcpyXLSekQkvj3w1dNIlyCF9jlVMoms0hywR8KqqUEBe1L7gl0wmiZEtebjkvtRzcukIOLxnRYdiEC7pGpBwx3/avdsENPuOVGdhQ97ZYZ3cg2aVGurJDKXnTkQFWyhA7hUWVQNY1XzgVVibbg9Rv8aUtP5rzsjFgsKQNqDIYdg/QLsVgZUElJ591qzQaUt7Lb3r/u8OlIRBgf1pJO0cPT9/34uon7cmI7P4+1qcsWBcWXQ2/2FadJcmr1aPfOrnLejWbpujVzbCID5PMp9xyzVjrd97o5nkXopfw1okrsde3gvjhC/8YfeWDIelkfmKZNkOMETFHOUaeQ3XxWhEtSGo6OnLRnumzNzyvYfdg9qOTAZ4im/NIJ23M3u+nzye8CWkv5XEIjlxSuXKAPlN1WLnYBMcwJeY9K3C0TiWkWvIA8rdHnbODmBB0Qtf+Q/WerBgNlc7e324k= x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: 792a707e-385b-4897-8f6d-08d7881daf9f X-MS-Exchange-CrossTenant-originalarrivaltime: 24 Dec 2019 03:02:17.7373 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 7szHPYE0aBXIHyfh0J5w6IUZh9GbJFcR1l/nL2GhtQZeLDo7CJb01WmUdg4PbZV0VKKWca236r8NnYVeiD535w== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YQBPR0101MB0707 X-Rspamd-Queue-Id: 47hgxR5Wgfz4MPN X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.66.87 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.66 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[uoguelph.ca]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[87.66.107.40.list.dnswl.org : 127.0.3.0]; IP_SCORE(-1.36)[ipnet: 40.64.0.0/10(-3.83), asn: 8075(-2.93), country: US(-0.05)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 24 Dec 2019 03:02:21 -0000 Richard P Mackerras wrote:=0A= >Hi,=0A= >=0A= >We had some bully type workloads emerge when we moved a lot of block=0A= >storage from old XIV to new all flash 3PAR. I wonder if your IMAP issue=0A= >might have emerged just because suddenly there was the opportunity with al= l=0A= >flash. QOS is good on 9.x ONTAP. If anyone says it=92s not then they last= =0A= >looked on 8.x. So I suggest you QOS the IMAP workload.=0A= >=0A= > Nobody should be using UDP with NFS unless they have a very specific set= =0A= >of circumstances. TCP was a real step forward.=0A= Well, I can't argue with this, considering I did the first working implemen= tation=0A= of NFS over TCP. It was actually Mike Karels that suggested I try doing so,= =0A= There's a paper in a very old Usenix Conference Proceedings, but it is so o= ld=0A= that it isn't on the Usenix web page (around 1988 in Denver, if I recall). = I don't=0A= even have a copy myself, although I was the author.=0A= =0A= Now, having said that, I must note that the Network Lock Manager (NLM) and= =0A= Network Status Monitor (NSM) were not NFS. They were separate stateful=0A= protocols (poorly designed imho) that Sun never published.=0A= =0A= NFS as Sun designed it (NFSv2 and NFSv3) were "stateless server" protocols,= =0A= so that they could work reliably without server crash recovery.=0A= However, the NLM was inherently stateful, since it was dealing with file lo= cks.=0A= =0A= So, you can't really lump the NLM with NFS (and you should avoid use of the= =0A= NLM over any transport imho).=0A= =0A= NFSv4 tackled the difficult problem of having a "stateful server" and crash= recovery,=0A= which resulted in a much more complex protocol (compare the size of RFC-181= 3=0A= vs RFC-5661 to get some idea of this).=0A= =0A= rick=0A= =0A= Cheers=0A= =0A= Richard=0A= _______________________________________________=0A= freebsd-stable@freebsd.org mailing list=0A= https://lists.freebsd.org/mailman/listinfo/freebsd-stable=0A= To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org"= =0A= From owner-freebsd-stable@freebsd.org Thu Dec 26 13:15:01 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 37D9B1D0300; Thu, 26 Dec 2019 13:15:01 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: from freefall.freebsd.org (freefall.freebsd.org [96.47.72.132]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47k9RT0b8Jz3QBq; Thu, 26 Dec 2019 13:15:01 +0000 (UTC) (envelope-from lwhsu@freebsd.org) Received: by freefall.freebsd.org (Postfix, from userid 1129) id 0C64F137F7; Thu, 26 Dec 2019 13:15:01 +0000 (UTC) Date: Thu, 26 Dec 2019 13:15:01 +0000 From: Li-Wen Hsu To: freebsd-testing@freebsd.org Cc: freebsd-current@freebsd.org, freebsd-stable@freebsd.org Subject: FreeBSD CI Weekly Report 2019-12-22 Message-ID: <20191226131501.GA909@freefall.freebsd.org> Reply-To: freebsd-testing@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.11.4 (2019-03-13) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 26 Dec 2019 13:15:01 -0000 (Please send the followup to freebsd-testing@ and note Reply-To is set.) FreeBSD CI Weekly Report 2019-12-22 =================================== Here is a summary of the FreeBSD Continuous Integration results for the period from 2019-12-15 to 2019-12-22. During this period, we have: * 2352 builds (98.5% (+2.2) passed, 1.5% (-2.2) failed) of buildworld and buildkernel (GENERIC and LINT) were executed on aarch64, amd64, armv6, armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, sparc64 architectures for head, stable/12, stable/11 branches. * 313 test runs (86.6% (-9.8) passed, 7.0% (+5.2) unstable, 6.4% (+4.6) exception) were executed on amd64, i386, riscv64 architectures for head, stable/12, stable/11 branches. * 16 doc builds (100% (+7) passed) Test case status (on 2019-12-22 23:59): | Branch/Architecture | Total | Pass | Fail | Skipped | | ------------------- | --------- | --------- | ----- | ------- | | head/amd64 | 7636 (+1) | 7570 (+4) | 0 (0) | 66 (-3) | | head/i386 | 7634 (+1) | 7560 (0) | 0 (0) | 74 (+1) | | 12-STABLE/amd64 | 7491 (+3) | 7443 (+3) | 0 (0) | 48 (0) | | 12-STABLE/i386 | 7489 (+3) | 7434 (+6) | 0 (0) | 55 (-3) | | 11-STABLE/amd64 | 6860 (+2) | 6810 (-1) | 0 (0) | 50 (+3) | | 11-STABLE/i386 | 6858 (+2) | 6809 (+2) | 0 (0) | 49 (0) | (The statistics from experimental jobs are omitted) If any of the issues found by CI are in your area of interest or expertise please investigate the PRs listed below. The latest web version of this report is available at https://hackmd.io/@FreeBSD-CI/report-20191222 and archive is available at https://hackmd.io/@FreeBSD-CI/ , any help is welcome. ## News * FreeBSD Foundation 2019 in Review: CI and Testing Advancements https://www.freebsdfoundation.org/blog/2019-in-review-ci-and-testing-advancements/ ## Failing and Flaky Tests (from experimental jobs) * https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/ * cddl.usr.sbin.dtrace.common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237641 * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/ * There are ~13 failing and ~109 skipped cases, including flakey ones, see https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/lastCompletedBuild/testReport/ for more details * Work for cleaning these failing cass are in progress ## Disabled Tests * sys.fs.tmpfs.mount_test.large https://bugs.freebsd.org/212862 * sys.fs.tmpfs.link_test.kqueue https://bugs.freebsd.org/213662 * sys.kqueue.libkqueue.kqueue_test.main https://bugs.freebsd.org/233586 * sys.kern.ptrace_test.ptrace__PT_KILL_competing_stop https://bugs.freebsd.org/220841 * lib.libc.regex.exhaust_test.regcomp_too_big (i386 only) https://bugs.freebsd.org/237450 * sys.netinet.socket_afinet.socket_afinet_bind_zero https://bugs.freebsd.org/238781 * sys.netpfil.pf.names.names * sys.netpfil.pf.synproxy.synproxy https://bugs.freebsd.org/238870 * sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger https://bugs.freebsd.org/239292 * sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger https://bugs.freebsd.org/239397 * sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger https://bugs.freebsd.org/239399 * sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger https://bugs.freebsd.org/239425 * lib.libc.gen.getmntinfo_test.getmntinfo_test https://bugs.freebsd.org/240049 * sys.sys.qmath_test.qdivq_s64q https://bugs.freebsd.org/240219 * sys.kern.ptrace_test.ptrace__getppid https://bugs.freebsd.org/240510 * lib.libc.sys.stat_test.stat_socket https://bugs.freebsd.org/240621 * lib.libarchive.functional_test.test_write_filter_zstd https://bugs.freebsd.org/240683 * lib.libcasper.services.cap_dns.dns_test.main https://bugs.freebsd.org/241435 * local.kyua.* (31 cases) & local.lutok.* (3 cases) on 11-i386 https://ci.freebsd.org/job/FreeBSD-stable-11-i386-test/2278/testReport/ * sys.geom.class.multipath.failloop.failloop https://bugs.freebsd.org/242689 ## Issues ### Cause build fails * https://bugs.freebsd.org/233735 Possible build race: genoffset.o /usr/src/sys/sys/types.h: error: machine/endian.h: No such file or directory * https://bugs.freebsd.org/233769 Possible build race: ld: error: unable to find library -lgcc_s ### Cause kernel panics * https://bugs.freebsd.org/238870 sys.netpfil.pf.names.names and sys.netpfil.pf.synproxy.synproxy cause panic Patch exists: * https://reviews.freebsd.org/D20868 * https://reviews.freebsd.org/D20869 ### Open * https://bugs.freebsd.org/242689 sys.geom.class.multipath.failloop.failloop fails due to too many CTF entries * https://bugs.freebsd.org/237403 Tests in sys/opencrypto should be converted to Python3 * https://bugs.freebsd.org/237641 Flakey test case: common.misc.t_dtrace_contrib.tst_dynopt_d * https://bugs.freebsd.org/237656 "Freed UMA keg (rtentry) was not empty (18 items). Lost 1 pages of memory." seen when running sys/netipsec tests * https://bugs.freebsd.org/238781 sys.netinet.socket_afinet.socket_afinet_bind_zero does not work when mac_portacl(4) loaded * https://bugs.freebsd.org/239292 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_child_detached_unrelated_debugger * https://bugs.freebsd.org/239397 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_both_attached_unrelated_debugger * https://bugs.freebsd.org/239399 Flakey test case: sys.kern.ptrace_test.ptrace__parent_sees_exit_after_child_debugger * https://bugs.freebsd.org/239425 Flakey test case: sys.kern.ptrace_test.ptrace__follow_fork_parent_detached_unrelated_debugger * https://bugs.freebsd.org/241662 Flakey test case: lib.libarchive.functional_test.test_fuzz_iso9660 ### Others * [Tickets related to testing@](https://preview.tinyurl.com/y9maauwg) From owner-freebsd-stable@freebsd.org Fri Dec 27 17:10:24 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 730B21C8FBA for ; Fri, 27 Dec 2019 17:10:24 +0000 (UTC) (envelope-from thenomad@gmail.com) Received: from mail-pg1-x530.google.com (mail-pg1-x530.google.com [IPv6:2607:f8b0:4864:20::530]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47ktcb4fKnz4K31 for ; Fri, 27 Dec 2019 17:10:23 +0000 (UTC) (envelope-from thenomad@gmail.com) Received: by mail-pg1-x530.google.com with SMTP id b9so14692781pgk.12 for ; Fri, 27 Dec 2019 09:10:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:to:from:subject:message-id:date:user-agent:mime-version :content-language:content-transfer-encoding; bh=ikvwgyng90sFsRGAg+Ag0zph20EhEcxYqop9LDEcHfo=; b=GRvUi5tfZq5dlsCggNh8kaHXX5aKTzbA475UyqkoDCHTw/foPLBqp1s3KQHms5XvTS 2/9S/7iEAY6EE3twBTq/RmniIqHZB9y+1/Sd75Bn9Pv3Rm51LofTy89X7GPk8Bh3HEfl u3jwhOdGfJDcL/qV6iHnaczy7unuemlrzSwL3E0WBQnOGYE5gT0twdJYhumN3OaZorlZ 6uMreWKHXZvPpaUj48aIFPcpLYnmWdDljn7IyvUeIKY2EBYUg8txhWxQ5QZCCQQZju/d WWQG7G4sNWHE0Ky2DLTalnh2QC+d/tS6eaj88Bk8ZrjqCVmBEa+n/wfmROM3nRaiosvE b82Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:to:from:subject:message-id:date :user-agent:mime-version:content-language:content-transfer-encoding; bh=ikvwgyng90sFsRGAg+Ag0zph20EhEcxYqop9LDEcHfo=; b=ovJtT19/fQFkxyNWzR/2dta7VQyrYfK3grc8mTu6jbuoC1tFCyZ84Fp4/vSQq/LL5V ECU+rT3wRfb/NPg7VfnfGuz0ghJz4ctoVxJzjz1gpEuHqsLb/ovRiwYzSowVXC99AN0g gQxr+nO/aZ2yhOFeWZh3g6e2Q607YrkXZULv0tGPXLh5kdkVAk6qVQe01s3BnjFlqYvr 98EFOuABS0yDnfgqfN4+9i90cZe9O4pPsfej3awMeELjqx1TQhaAap1FAI8uEUTeq4/z lGV/r5jcuh/G7nFOC8uTk59SZ/ATTELfU9zrUzfDj9Za8W93DsFx+RdKIPQjNA1ai/2q 3vmw== X-Gm-Message-State: APjAAAUZJ/KTHtH2kXoAYpp93VRGRzEviFtBZjuScwWbqfrB1iCy2Dzj NglRnqixsdhMrCrr8dWM+HgsBaoe X-Google-Smtp-Source: APXvYqxFZW09B2IfdlBIs/n0Wym4PsXYcIpiTccT8JzMA0v8QP3IU2Lw7QqveGLb65OoVCdowoIpxg== X-Received: by 2002:a63:7705:: with SMTP id s5mr54472783pgc.379.1577466621336; Fri, 27 Dec 2019 09:10:21 -0800 (PST) Received: from vanyel.ee.washington.edu (vanyel.ee.washington.edu. [128.208.232.99]) by smtp.gmail.com with ESMTPSA id q6sm41573815pfh.127.2019.12.27.09.10.20 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 27 Dec 2019 09:10:20 -0800 (PST) Sender: Lee Damon To: freebsd-stable@freebsd.org From: Lee Damon Subject: ldapsearch stops working after ~4-12 hours (one host of 4) Message-ID: <23f18d16-7f86-8e94-8cd5-9bed61ea3405@castle.org> Date: Fri, 27 Dec 2019 09:10:20 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.3.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 47ktcb4fKnz4K31 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=GRvUi5tf; dmarc=none; spf=pass (mx1.freebsd.org: domain of thenomad@gmail.com designates 2607:f8b0:4864:20::530 as permitted sender) smtp.mailfrom=thenomad@gmail.com X-Spamd-Result: default: False [-4.84 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[castle.org]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_IN_DNSWL_NONE(0.00)[0.3.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-2.64)[ip: (-9.10), ipnet: 2607:f8b0::/32(-2.16), asn: 15169(-1.88), country: US(-0.05)]; FORGED_SENDER(0.30)[nomad@castle.org,thenomad@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[nomad@castle.org,thenomad@gmail.com]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Dec 2019 17:10:24 -0000 I have four hosts all running 11.3-RELEASE-p5 on the same subnet. Yesterday I did the usual "freebsd-update fetch && freebsd-update install" on all four. They came back up fine but about 4 hours later one of them started reporting problems with ldap search. I poked at it a bit but it was struggling to do anything so I had to reboot it. It came back up clean but the problem resurfaced early this morning. from /var/log/messages: Dec 27 03:30:00 [redacted] root: 3:30AM up 11:16, 1 user, load averages: 0.04, 0.07, 0.08 Dec 27 03:35:00 [redacted] root: 3:35AM up 11:21, 1 user, load averages: 0.21, 0.23, 0.15 Dec 27 03:35:10 [redacted] chgrp: nss_ldap: could not search LDAP server - Server is unavailable Dec 27 03:35:12 [redacted] top: nss_ldap: could not search LDAP server - Server is unavailable Dec 27 03:35:35 [redacted] nrpe[76163]: nss_ldap: could not search LDAP server - Server is unavailable Both times I observed this: : ldapsearch -v -LLL -x -h [redacted].ee.washington.edu -b dc=ee,dc=washington,dc=edu uid=[redacted] ldap_initialize( ldap://[redacted].ee.washington.edu ) ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1) The other hosts on the same subnet have no such problems. I re-ran fetch & update but it said there was nothing to do. Any hints where I should start poking this? thanks, nomad From owner-freebsd-stable@freebsd.org Fri Dec 27 17:35:29 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BCDA11C9EFD for ; Fri, 27 Dec 2019 17:35:29 +0000 (UTC) (envelope-from matt.garber@gmail.com) Received: from mail-qk1-x72e.google.com (mail-qk1-x72e.google.com [IPv6:2607:f8b0:4864:20::72e]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47kv9X49lPz4LVt for ; Fri, 27 Dec 2019 17:35:28 +0000 (UTC) (envelope-from matt.garber@gmail.com) Received: by mail-qk1-x72e.google.com with SMTP id z76so22345864qka.2 for ; Fri, 27 Dec 2019 09:35:28 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=TfEtP29dZ/gxuF7L2JoNgqwZzkejlqCx05X6Avfl0F4=; b=Wh8cwWjhl0DdPbqeVh9nWu8Cbl35q3rYriPAnVOXhN399gJtFYu8yhZUbeOc6jxf1V R0nuPGzn11uzpTp0bPq13BfiTyPoyl6YV43IuHqbUXUbtQBWa/bGmNf7axvk9HVeGCD8 Dx2uD9KUG+nbS9y4fGcJqyR0lHAoAiHRsdfhLXQZ2jn0Rg+SJha8uc6cG+JRNC5dZ4NW 09esZP1NzjMWy3Q8b3THttKY4VTOrngFTQk4ccj3Me+F7f3Ewtk7hprpTgv7T4IhHHb3 PlgamWcx/TYEjXC+CDY1ayMTi9knKTdQK6sefLzXtZGKARAj/ASVTeNwFlpXxtPrQCoA 1Wgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=TfEtP29dZ/gxuF7L2JoNgqwZzkejlqCx05X6Avfl0F4=; b=B1drRavUPmc93v8x5MpPAIcKe+Ko2joFlw9OBncx9bUpVRN6ZXhw+EHQLkf/qZAqzr EpwnluM03ddPGNbDqQBDhASTFO7KN6/dhkaZPXZjDSVlIl/hN3wICr/QwatPdrvqPwCC kXqUfDsP6936zZ2ZdUR6HldTI6d+aYirJYrxYVuNIshLFgu9e0803W5nQcWra02YzW+r 33aph/3A2A4nAJaH0ibIOSIeJAYl9lE1myOE/QBCM+5UJTJUQbfDfn3Hbt/1u+EXo6Ic kKa71KBbl/YW74J3izQMQz6l3xbkTnISqfBncMrhzu4kspqTvJIFX/bWQluD8xO8z2oe XaIQ== X-Gm-Message-State: APjAAAW3IVR1rHd8gRsVtoAnQccWQk2qD4DvIY5N0/+oVqgaLM0HxyPa RKkM3x2Sf8ydUUpWBZmhR+Lua6l3obhhiZNxW8CSUQ== X-Google-Smtp-Source: APXvYqzLTcOgh+LVQNJy8qjnWP1PJgkv/bDe+MRq7dGGI/JDfYLBUXigcJeRPBS60ac9+9uots5MT/mrI316oHyQ2Qc= X-Received: by 2002:a05:620a:1102:: with SMTP id o2mr42997782qkk.278.1577468127241; Fri, 27 Dec 2019 09:35:27 -0800 (PST) MIME-Version: 1.0 References: <23f18d16-7f86-8e94-8cd5-9bed61ea3405@castle.org> In-Reply-To: <23f18d16-7f86-8e94-8cd5-9bed61ea3405@castle.org> From: Matt Garber Date: Fri, 27 Dec 2019 12:35:16 -0500 Message-ID: Subject: Re: ldapsearch stops working after ~4-12 hours (one host of 4) To: Lee Damon Cc: freebsd-stable@freebsd.org X-Rspamd-Queue-Id: 47kv9X49lPz4LVt X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Wh8cwWjh; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mattgarber@gmail.com designates 2607:f8b0:4864:20::72e as permitted sender) smtp.mailfrom=mattgarber@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (-9.23), ipnet: 2607:f8b0::/32(-2.16), asn: 15169(-1.88), country: US(-0.05)]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[e.2.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Dec 2019 17:35:29 -0000 On Fri, Dec 27, 2019 at 12:10 PM Lee Damon wrote: > > Both times I observed this: > > : ldapsearch -v -LLL -x -h [redacted].ee.washington.edu -b > dc=3Dee,dc=3Dwashington,dc=3Dedu uid=3D[redacted] > ldap_initialize( ldap://[redacted].ee.washington.edu ) > ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1) Do you have connection/access logs on the LDAP server to verify whether a connection is even being established? Also, are you able to try running those same ldapsearch queries with the IP address(es) rather than DNS names for your server? The =E2=80=9Ccan=E2=80=99t contact=E2=80=9D initially seem= s more like potentially DNS resolution or firewall/connectivity than something LDAP related like failure to bind successfully=E2=80=A6 Thanks, -Matt From owner-freebsd-stable@freebsd.org Fri Dec 27 18:53:17 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 856D71CC061 for ; Fri, 27 Dec 2019 18:53:17 +0000 (UTC) (envelope-from thenomad@gmail.com) Received: from mail-pf1-x436.google.com (mail-pf1-x436.google.com [IPv6:2607:f8b0:4864:20::436]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47kwvG6jwgz4RML for ; Fri, 27 Dec 2019 18:53:14 +0000 (UTC) (envelope-from thenomad@gmail.com) Received: by mail-pf1-x436.google.com with SMTP id x6so14027845pfo.10 for ; Fri, 27 Dec 2019 10:53:14 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:references:cc:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=P0PKiFrPe4UX20sXvXbfqpN9FfeoAL6Jr1lBdNcys3s=; b=cMIq0CyEyaR/zfMAkZAe85a+j8F7R9rFKzSalsnm7vo2VbR2STTu5NTFF74ffoDqt8 4+vufzf5C/Ma3pJk/3Ze2hBcu4il5+0NAo9n6M+AW0g6243BAh/kXdAVihJW+aT1IZ3F gER48TkK5C174yJYLGMo+sCVaZJHoOvNFSFFrsbVjy0Gbal8X9I7onY1ZeaYWidhx3f9 /tafOyWHkmbxcpRcMSqusSCZl1BLZmWPD/KWiDq9HGCbhYCT01qK/kHIHaB9FaY895nM TTH14wwU8AbNYHZZaNoLaQFRQ9EpjanetADzAJ4esvF1SuLmT7WwRpHFd2goo/nWMSbV H96w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:references:cc:from:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=P0PKiFrPe4UX20sXvXbfqpN9FfeoAL6Jr1lBdNcys3s=; b=BlaY3lKkgZRmnfgyS+sZjSsvdblwwVUZxp/IAU0SP2QcxfYp3+Qe7vLPBSSYrpvCFD vLitsUYUk0W3GQg5dgt2IuezZxUXTA8n5czrX85k7g0UKljy+ZJK87+gN4wUDRHEY2q/ py4iO5BDHiAPPmWPlTENED7CX0+CiSL/QsRqPie274K9wyA9t6cf0RL/mXekUakGt+38 q+1xi6l9biOh0k1wj3R34l0qDuGS7VAWMH9SzlWkxc4kdX4WDc6MbrMcYMrr8fB9xWRo H9Cf6mGsxLzL+lSPDWbY6rt27EoBmur8S+AAKoeoQBrF7iscdGIrir7Ng0LwEZ6uM7hD aZhw== X-Gm-Message-State: APjAAAVNeaQbtP93ZWQOQPnOiIE3h8OBe47QIZPpzWktL5XCAZYoAo6+ +4OfwTp8kvY5VljT+lP8WHFJaX9a X-Google-Smtp-Source: APXvYqzhfbX1CFRM/iHGvoI+1Aa6MSoeZ58xDKeT7KTlokdAf7MR7PzBFfcjIszV11HCubpdwKuuJg== X-Received: by 2002:a62:7683:: with SMTP id r125mr56858939pfc.132.1577472793055; Fri, 27 Dec 2019 10:53:13 -0800 (PST) Received: from vanyel.ee.washington.edu (vanyel.ee.washington.edu. [128.208.232.99]) by smtp.gmail.com with ESMTPSA id h12sm26469730pfo.12.2019.12.27.10.53.12 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 27 Dec 2019 10:53:12 -0800 (PST) Sender: Lee Damon Subject: Re: ldapsearch stops working after ~4-12 hours (one host of 4) To: Matt Garber References: <23f18d16-7f86-8e94-8cd5-9bed61ea3405@castle.org> Cc: freebsd-stable@freebsd.org From: Lee Damon Message-ID: <492d412f-042d-645d-4f29-1e12aacc2d3d@castle.org> Date: Fri, 27 Dec 2019 10:53:12 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.3.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 47kwvG6jwgz4RML X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=cMIq0CyE; dmarc=none; spf=pass (mx1.freebsd.org: domain of thenomad@gmail.com designates 2607:f8b0:4864:20::436 as permitted sender) smtp.mailfrom=thenomad@gmail.com X-Spamd-Result: default: False [-4.90 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; FORGED_SENDER(0.30)[nomad@castle.org,thenomad@gmail.com]; FREEMAIL_TO(0.00)[gmail.com]; IP_SCORE(-2.70)[ip: (-9.43), ipnet: 2607:f8b0::/32(-2.16), asn: 15169(-1.88), country: US(-0.05)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[nomad@castle.org,thenomad@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[castle.org]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[6.3.4.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Dec 2019 18:53:17 -0000 On 12/27/19 09:35 , Matt Garber wrote: > On Fri, Dec 27, 2019 at 12:10 PM Lee Damon > wrote: > > > Both times I observed this: > > : ldapsearch -v -LLL -x -h [redacted].ee.washington.edu > -b > dc=ee,dc=washington,dc=edu uid=[redacted] > ldap_initialize( ldap://[redacted].ee.washington.edu > ) > ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1) > > > Do you have connection/access logs on the LDAP server to verify whether > a connection is even being established? I've asked the people who run those servers for that information. > Also, are you able to try > running those same ldapsearch queries with the IP address(es) rather > than DNS names for your server? The “can’t contact” initially seems more > like potentially DNS resolution or firewall/connectivity than something > LDAP related like failure to bind successfully… The host command returned the correct IP address when I queried it. I don't remember substituting IP addresses when this happened yesterday and I know I didn't do it this morning. I'll try that the next time this happens. nomad From owner-freebsd-stable@freebsd.org Fri Dec 27 22:32:21 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 199091D178F for ; Fri, 27 Dec 2019 22:32:21 +0000 (UTC) (envelope-from thenomad@gmail.com) Received: from mail-pg1-x535.google.com (mail-pg1-x535.google.com [IPv6:2607:f8b0:4864:20::535]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47l1m40J7Lz4dwX for ; Fri, 27 Dec 2019 22:32:19 +0000 (UTC) (envelope-from thenomad@gmail.com) Received: by mail-pg1-x535.google.com with SMTP id b137so15065196pga.6 for ; Fri, 27 Dec 2019 14:32:19 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:from:subject:to:message-id:date:user-agent:mime-version :content-language; bh=bs0M3jlBLpo0Gfc1R+SVKr+lfZvZinU1EZa3IQ+OiyY=; b=Tk9y+N0axuQzLp2BvJkh3G+mNcaJLyXo0t2J+BVn5+IzK49seYW9qfnVMyqhGz2Z0i yXUT8yOWIyiwHjCiel5NshjGjbDrH5+akraLGAGVBzTYrz1l0aMFgCGR2DBxF3UuCJKI YRxVNip7yiM7ONBffcE9pNumChpxnpgvXQShGv01obLZ2MvFXuE0OihbO5orBpgzT6PR 0VJ4eP7tfkhJfR69N5b5c6yLwmakTrTQdNVolfGoFvoG7J/Xg8xRX/wxmqrfJiL9n/xh w8DFPQOGIiVLyOnrj999TrpH+nbzZonMpsP0BEWVn1yYT9qlgBk64I+lHP436lkqVArR vROQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:from:subject:to:message-id:date :user-agent:mime-version:content-language; bh=bs0M3jlBLpo0Gfc1R+SVKr+lfZvZinU1EZa3IQ+OiyY=; b=o2nmrb3kWF9mBN5gFMhTZNBkUHXpI0SyunBLaNU7WYonJIIe6S+zfJaEP+b0uHJLNX oV+a2YYfKh4g/AB7k3EDkha3f6bSoRAhWH4Aqj/R52tZZx8kOB9VAcODZpLFRb8ym9Eb NkXTiruMuv9Btq2LDUfVLkBshPNisBUAC9f50hcCEfQh2p5cXbkvukwZGoxAPKA6JCZx ze1sW5tCMHXoOIs035Kmu1pDc+p+cTmSJrB7dhXRGIXOyZ3xB5IQFutvdzW1m2OnbWg7 EqolRahdXrqBbqmUY0HuvSOd0K4h7OUQT/FaUiopXDuEa1DrLHlOXzRW5rP25S31Xmiv B4mw== X-Gm-Message-State: APjAAAWssC7ZOj89WQk64oWqK8Rty+pBBFj1L8jwwQrMKtJtcmqsXuDl nvEGe3rjp/+3hKfAVFkl0Dr6rcNH X-Google-Smtp-Source: APXvYqyRwJQ4uE3QbawvyUTWEmhATIHRYwX4j1LsrHlxUxJiPcg+uWgtUzJaY+fuo2iuVYOxrF+3oQ== X-Received: by 2002:aa7:9556:: with SMTP id w22mr57280871pfq.198.1577485938127; Fri, 27 Dec 2019 14:32:18 -0800 (PST) Received: from vanyel.ee.washington.edu (vanyel.ee.washington.edu. [128.208.232.99]) by smtp.gmail.com with ESMTPSA id o10sm38362571pgq.68.2019.12.27.14.32.17 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 27 Dec 2019 14:32:17 -0800 (PST) Sender: Lee Damon From: Lee Damon Subject: amd and LDAP - a sad story To: freebsd-stable@freebsd.org Message-ID: <44772535-277d-c50d-f93f-57b3e4162ed7@castle.org> Date: Fri, 27 Dec 2019 14:32:16 -0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:68.0) Gecko/20100101 Thunderbird/68.3.1 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------E6CE60CD2513FF880F77D12F" Content-Language: en-US X-Rspamd-Queue-Id: 47l1m40J7Lz4dwX X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Tk9y+N0a; dmarc=none; spf=pass (mx1.freebsd.org: domain of thenomad@gmail.com designates 2607:f8b0:4864:20::535 as permitted sender) smtp.mailfrom=thenomad@gmail.com X-Spamd-Result: default: False [-4.87 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_MATCH_ENVRCPT_ALL(0.00)[]; HAS_ATTACHMENT(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[castle.org]; DKIM_TRACE(0.00)[gmail.com:+]; RCVD_IN_DNSWL_NONE(0.00)[5.3.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-2.67)[ip: (-9.25), ipnet: 2607:f8b0::/32(-2.16), asn: 15169(-1.88), country: US(-0.05)]; FORGED_SENDER(0.30)[nomad@castle.org,thenomad@gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~,3:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[nomad@castle.org,thenomad@gmail.com]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 27 Dec 2019 22:32:21 -0000 This is a multi-part message in MIME format. --------------E6CE60CD2513FF880F77D12F Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit (Different host than the LDAP problem I emailed about earlier today). I have a newly installed host running 11.3-RELEASE-p5 (patched to latest). I have installed and configured it the same way an existing 11.3-RELEASE-p5 host on the same subnet is set up. amd launches and runs just fine on the host ... until I install and activate nss_ldap. Mind you, this is with _exactly_ the same ldap config files and amd maps as the working host is using. amd_log.good shows a regular startup, mount of a filesystem, and shutdown (no nss_ldap) while amd_log.bad shows what happens when pkg nss_ldap is installed and /etc/nssswitch.conf is changed from: group: compat passwd: compat to passwd: files ldap group: files ldap No other changes were made between these runs. Any hints? thanks, nomad --------------E6CE60CD2513FF880F77D12F Content-Type: application/octet-stream; name="amd_log.bad" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="amd_log.bad" RGVjIDI3IDE0OjE5OjM2IGR1Y2sgYW1kWzgzMl0vaW5mbzogIHN3aXRjaGVkIHRvIGxvZ2Zp bGUgIi92YXIvbG9nL2FtZCIKRGVjIDI3IDE0OjE5OjM2IGR1Y2sgYW1kWzgzMl0vaW5mbzog IEFNLVVUSUxTIFZFUlNJT04gSU5GT1JNQVRJT046CkRlYyAyNyAxNDoxOTozNiBkdWNrIGFt ZFs4MzJdL2luZm86ICBDb3B5cmlnaHQgKGMpIDE5OTctMjAxNCBFcmV6IFphZG9rCkRlYyAy NyAxNDoxOTozNiBkdWNrIGFtZFs4MzJdL2luZm86ICBDb3B5cmlnaHQgKGMpIDE5OTAgSmFu LVNpbW9uIFBlbmRyeQpEZWMgMjcgMTQ6MTk6MzYgZHVjayBhbWRbODMyXS9pbmZvOiAgQ29w eXJpZ2h0IChjKSAxOTkwIEltcGVyaWFsIENvbGxlZ2Ugb2YgU2NpZW5jZSwgVGVjaG5vbG9n eSAmIE1lZGljaW5lCkRlYyAyNyAxNDoxOTozNiBkdWNrIGFtZFs4MzJdL2luZm86ICBDb3B5 cmlnaHQgKGMpIDE5OTAgVGhlIFJlZ2VudHMgb2YgdGhlIFVuaXZlcnNpdHkgb2YgQ2FsaWZv cm5pYS4KRGVjIDI3IDE0OjE5OjM2IGR1Y2sgYW1kWzgzMl0vaW5mbzogIGFtLXV0aWxzIHZl cnNpb24gNi4yIChidWlsZCAxMTAzMDAwKS4KRGVjIDI3IDE0OjE5OjM2IGR1Y2sgYW1kWzgz Ml0vaW5mbzogIFJlcG9ydCBidWdzIHRvIGh0dHBzOi8vYnVnemlsbGEuYW0tdXRpbHMub3Jn LyBvciBhbS11dGlsc0BhbS11dGlscy5vcmcuCkRlYyAyNyAxNDoxOTozNiBkdWNrIGFtZFs4 MzJdL2luZm86ICBjcHU9YW1kNjQgKGxpdHRsZS1lbmRpYW4pLCBhcmNoPWFtZDY0LCBrYXJj aD1hbWQ2NC4KRGVjIDI3IDE0OjE5OjM2IGR1Y2sgYW1kWzgzMl0vaW5mbzogIGZ1bGxfb3M9 ZnJlZWJzZDExLjMsIG9zPWZyZWVic2QxMSwgb3N2ZXI9MTEuMywgdmVuZG9yPXVuZGVybXlk ZXNrLCBkaXN0cm89VGhlIEZyZWVCU0QgUHJvamVjdC4KRGVjIDI3IDE0OjE5OjM2IGR1Y2sg YW1kWzgzMl0vaW5mbzogIGRvbWFpbj1lZS53YXNoaW5ndG9uLmVkdSwgaG9zdD1kdWNrLCBo b3N0ZD1kdWNrLmVlLndhc2hpbmd0b24uZWR1LgpEZWMgMjcgMTQ6MTk6MzYgZHVjayBhbWRb ODMyXS9pbmZvOiAgTWFwIHN1cHBvcnQgZm9yOiByb290LCBwYXNzd2QsIHVuaW9uLCBuaXMs IG5kYm0sIGZpbGUsIGV4ZWMsIGVycm9yLgpEZWMgMjcgMTQ6MTk6MzYgZHVjayBhbWRbODMy XS9pbmZvOiAgQU1GUzogbmZzLCBsaW5rLCBuZnN4LCBuZnNsLCBob3N0LCBsaW5reCwgcHJv Z3JhbSwgdW5pb24sIHVmcywgY2RmcywgCkRlYyAyNyAxNDoxOTozNiBkdWNrIGFtZFs4MzJd L2luZm86ICAgICAgICBwY2ZzLCB0bXBmcywgdWRmLCBhdXRvLCBkaXJlY3QsIHRvcGx2bCwg ZXJyb3IsIGluaGVyaXQuCkRlYyAyNyAxNDoxOTozNiBkdWNrIGFtZFs4MzJdL2luZm86ICBG UzogY2Q5NjYwLCBtZnMsIG5mcywgbmZzMywgbnVsbGZzLCBtc2Rvc2ZzLCB0bXBmcywgdWZz LCB1bmlvbmZzLgpEZWMgMjcgMTQ6MTk6MzYgZHVjayBhbWRbODMyXS9pbmZvOiAgTmV0d29y ayAxOiB3aXJlPSIxMjguMjA4LjIzMi4wIiAobmV0bnVtYmVyPTEyOC4yMDguMjMyKS4KRGVj IDI3IDE0OjE5OjM2IGR1Y2sgYW1kWzgzMl0vaW5mbzogIE5ldHdvcmsgMjogd2lyZT0iMTky LjE2OC4yMzIuMCIgKG5ldG51bWJlcj0xOTIuMTY4LjIzMikuCkRlYyAyNyAxNDoxOTozNiBk dWNrIGFtZFs4MzJdL2luZm86ICBNeSBpcCBhZGRyIGlzIDEyNy4wLjAuMQpEZWMgMjcgMTQ6 MTk6MzYgZHVjayBhbWRbODMzXS9pbmZvOiAgcmVsZWFzZWQgY29udHJvbGxpbmcgdHR5IHVz aW5nIHNldHNpZCgpCkRlYyAyNyAxNDoxOTozNiBkdWNrIGFtZFs4MzNdL2luZm86ICBMb2Nr ZWQgcHJvY2VzcyBwYWdlcyBpbiBtZW1vcnkKRGVjIDI3IDE0OjE5OjM2IGR1Y2sgYW1kWzgz M10vaW5mbzogIGZpbGUgc2VydmVyIGxvY2FsaG9zdCwgdHlwZSBsb2NhbCwgc3RhdGUgc3Rh cnRzIHVwCkRlYyAyNyAxNDoxOTozNiBkdWNrIGFtZFs4MzNdL2luZm86ICB6cm9vdC9ST09U L2RlZmF1bHQgcmVzdGFydGVkIGZzdHlwZSBsaW5rIG9uIC8sIGZsYWdzIDB4ODkKRGVjIDI3 IDE0OjE5OjM2IGR1Y2sgYW1kWzgzM10vaW5mbzogIGRldmZzIHJlc3RhcnRlZCBmc3R5cGUg bGluayBvbiAvZGV2LCBmbGFncyAweDg5CkRlYyAyNyAxNDoxOTozNiBkdWNrIGFtZFs4MzNd L2luZm86ICB6cm9vdCByZXN0YXJ0ZWQgZnN0eXBlIGxpbmsgb24gL3pyb290LCBmbGFncyAw eDg5CkRlYyAyNyAxNDoxOTozNiBkdWNrIGFtZFs4MzNdL2luZm86ICB6cm9vdC9uaWtvbGEg cmVzdGFydGVkIGZzdHlwZSBsaW5rIG9uIC9uaWtvbGEsIGZsYWdzIDB4ODkKRGVjIDI3IDE0 OjE5OjM2IGR1Y2sgYW1kWzgzM10vaW5mbzogIHpyb290L3Zhci9jcmFzaCByZXN0YXJ0ZWQg ZnN0eXBlIGxpbmsgb24gL3Zhci9jcmFzaCwgZmxhZ3MgMHg4OQpEZWMgMjcgMTQ6MTk6MzYg ZHVjayBhbWRbODMzXS9pbmZvOiAgenJvb3QvdmFyL2xvZyByZXN0YXJ0ZWQgZnN0eXBlIGxp bmsgb24gL3Zhci9sb2csIGZsYWdzIDB4ODkKRGVjIDI3IDE0OjE5OjM2IGR1Y2sgYW1kWzgz M10vaW5mbzogIHpyb290L3Vzci9ob21lIHJlc3RhcnRlZCBmc3R5cGUgbGluayBvbiAvdXNy L2hvbWUsIGZsYWdzIDB4ODkKRGVjIDI3IDE0OjE5OjM2IGR1Y2sgYW1kWzgzM10vaW5mbzog IHpyb290L3Zhci9hdWRpdCByZXN0YXJ0ZWQgZnN0eXBlIGxpbmsgb24gL3Zhci9hdWRpdCwg ZmxhZ3MgMHg4OQpEZWMgMjcgMTQ6MTk6MzYgZHVjayBhbWRbODMzXS9pbmZvOiAgenJvb3Qv dXNyL3NyYyByZXN0YXJ0ZWQgZnN0eXBlIGxpbmsgb24gL3Vzci9zcmMsIGZsYWdzIDB4ODkK RGVjIDI3IDE0OjE5OjM2IGR1Y2sgYW1kWzgzM10vaW5mbzogIHpyb290L3RtcCByZXN0YXJ0 ZWQgZnN0eXBlIGxpbmsgb24gL3RtcCwgZmxhZ3MgMHg4OQpEZWMgMjcgMTQ6MTk6MzYgZHVj ayBhbWRbODMzXS9pbmZvOiAgenJvb3QvdmFyL21haWwgcmVzdGFydGVkIGZzdHlwZSBsaW5r IG9uIC92YXIvbWFpbCwgZmxhZ3MgMHg4OQpEZWMgMjcgMTQ6MTk6MzYgZHVjayBhbWRbODMz XS9pbmZvOiAgenJvb3QvdmFyL3RtcCByZXN0YXJ0ZWQgZnN0eXBlIGxpbmsgb24gL3Zhci90 bXAsIGZsYWdzIDB4ODkKRGVjIDI3IDE0OjE5OjM2IGR1Y2sgYW1kWzgzM10vaW5mbzogIHpy b290L3Vzci9wb3J0cyByZXN0YXJ0ZWQgZnN0eXBlIGxpbmsgb24gL3Vzci9wb3J0cywgZmxh Z3MgMHg4OQpEZWMgMjcgMTQ6MTk6MzYgZHVjayBhbWRbODMzXS9pbmZvOiAgenJvb3Qvbmlr b2xhL25pa29sYSByZXN0YXJ0ZWQgZnN0eXBlIGxpbmsgb24gL25pa29sYS9uaWtvbGEsIGZs YWdzIDB4ODkKRGVjIDI3IDE0OjE5OjM2IGR1Y2sgYW1kWzgzM10vbWFwOiAgIFRyeWluZyBt b3VudCBvZiAvZXRjL2FtZC9hbWQuaG9tZXMgb24gL3VzZXJzIGZzdHlwZSB0b3BsdmwgbW91 bnRfdHlwZSBub24tYXV0b2ZzCkRlYyAyNyAxNDoxOTozNiBkdWNrIGFtZFs4MzNdL2luZm86 ICBjcmVhdGluZyBtb3VudHBvaW50IGRpcmVjdG9yeSAnL3VzZXJzJwpEZWMgMjcgMTQ6MTk6 MzYgZHVjayBhbWRbODMzXS9tYXA6ICAgVHJ5aW5nIG1vdW50IG9mIC9ldGMvYW1kL2FtZC5u ZXR3b3JrIG9uIC9uIGZzdHlwZSB0b3BsdmwgbW91bnRfdHlwZSBub24tYXV0b2ZzCkRlYyAy NyAxNDoxOTozNiBkdWNrIGFtZFs4MzNdL2luZm86ICBjcmVhdGluZyBtb3VudHBvaW50IGRp cmVjdG9yeSAnL24nCkRlYyAyNyAxNDoxOTozNiBkdWNrIGFtZFs4MzRdL2luZm86ICAvdXNl cnM6IGRpc2FibGluZyBuZnMgY29uZ2VzdGlvbiB3aW5kb3cKRGVjIDI3IDE0OjE5OjM2IGR1 Y2sgYW1kWzgzM10vbWFwOiAgIFRyeWluZyBtb3VudCBvZiAvZXRjL2FtZC9hbWQuZ3JvdXAg b24gL2cgZnN0eXBlIHRvcGx2bCBtb3VudF90eXBlIG5vbi1hdXRvZnMKRGVjIDI3IDE0OjE5 OjM2IGR1Y2sgYW1kWzgzM10vaW5mbzogIGNyZWF0aW5nIG1vdW50cG9pbnQgZGlyZWN0b3J5 ICcvZycKRGVjIDI3IDE0OjE5OjM2IGR1Y2sgYW1kWzgzNV0vaW5mbzogIC9uOiBkaXNhYmxp bmcgbmZzIGNvbmdlc3Rpb24gd2luZG93CkRlYyAyNyAxNDoxOTozNiBkdWNrIGFtZFs4MzZd L2luZm86ICAvZzogZGlzYWJsaW5nIG5mcyBjb25nZXN0aW9uIHdpbmRvdwpEZWMgMjcgMTQ6 MTk6MzYgZHVjayBhbWRbODMzXS9mYXRhbDogdW5hYmxlIHRvIHJlZ2lzdGVyIChBTVFfUFJP R1JBTT0zMDAwMTksIEFNUV9WRVJTSU9OLCB0Y3ApCkRlYyAyNyAxNDoxOTozNiBkdWNrIGFt ZFs4MzNdL2ZhdGFsOiB1bmFibGUgdG8gcmVnaXN0ZXIgKEFNUV9QUk9HUkFNPTMwMDAxOSwg QU1RX1ZFUlNJT04sIHVkcCkKRGVjIDI3IDE0OjE5OjM2IGR1Y2sgYW1kWzgzM10vaW5mbzog IEZpbmlzaGluZyB3aXRoIHN0YXR1cyAzCg== --------------E6CE60CD2513FF880F77D12F Content-Type: application/octet-stream; name="amd_log.good" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="amd_log.good" RGVjIDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4MF0vaW5mbzogIHN3aXRjaGVkIHRvIGxvZ2Zp bGUgIi92YXIvbG9nL2FtZCIKRGVjIDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4MF0vaW5mbzog IEFNLVVUSUxTIFZFUlNJT04gSU5GT1JNQVRJT046CkRlYyAyNyAxNDoxNzo0OCBkdWNrIGFt ZFs1ODBdL2luZm86ICBDb3B5cmlnaHQgKGMpIDE5OTctMjAxNCBFcmV6IFphZG9rCkRlYyAy NyAxNDoxNzo0OCBkdWNrIGFtZFs1ODBdL2luZm86ICBDb3B5cmlnaHQgKGMpIDE5OTAgSmFu LVNpbW9uIFBlbmRyeQpEZWMgMjcgMTQ6MTc6NDggZHVjayBhbWRbNTgwXS9pbmZvOiAgQ29w eXJpZ2h0IChjKSAxOTkwIEltcGVyaWFsIENvbGxlZ2Ugb2YgU2NpZW5jZSwgVGVjaG5vbG9n eSAmIE1lZGljaW5lCkRlYyAyNyAxNDoxNzo0OCBkdWNrIGFtZFs1ODBdL2luZm86ICBDb3B5 cmlnaHQgKGMpIDE5OTAgVGhlIFJlZ2VudHMgb2YgdGhlIFVuaXZlcnNpdHkgb2YgQ2FsaWZv cm5pYS4KRGVjIDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4MF0vaW5mbzogIGFtLXV0aWxzIHZl cnNpb24gNi4yIChidWlsZCAxMTAzMDAwKS4KRGVjIDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4 MF0vaW5mbzogIFJlcG9ydCBidWdzIHRvIGh0dHBzOi8vYnVnemlsbGEuYW0tdXRpbHMub3Jn LyBvciBhbS11dGlsc0BhbS11dGlscy5vcmcuCkRlYyAyNyAxNDoxNzo0OCBkdWNrIGFtZFs1 ODBdL2luZm86ICBjcHU9YW1kNjQgKGxpdHRsZS1lbmRpYW4pLCBhcmNoPWFtZDY0LCBrYXJj aD1hbWQ2NC4KRGVjIDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4MF0vaW5mbzogIGZ1bGxfb3M9 ZnJlZWJzZDExLjMsIG9zPWZyZWVic2QxMSwgb3N2ZXI9MTEuMywgdmVuZG9yPXVuZGVybXlk ZXNrLCBkaXN0cm89VGhlIEZyZWVCU0QgUHJvamVjdC4KRGVjIDI3IDE0OjE3OjQ4IGR1Y2sg YW1kWzU4MF0vaW5mbzogIGRvbWFpbj1lZS53YXNoaW5ndG9uLmVkdSwgaG9zdD1kdWNrLCBo b3N0ZD1kdWNrLmVlLndhc2hpbmd0b24uZWR1LgpEZWMgMjcgMTQ6MTc6NDggZHVjayBhbWRb NTgwXS9pbmZvOiAgTWFwIHN1cHBvcnQgZm9yOiByb290LCBwYXNzd2QsIHVuaW9uLCBuaXMs IG5kYm0sIGZpbGUsIGV4ZWMsIGVycm9yLgpEZWMgMjcgMTQ6MTc6NDggZHVjayBhbWRbNTgw XS9pbmZvOiAgQU1GUzogbmZzLCBsaW5rLCBuZnN4LCBuZnNsLCBob3N0LCBsaW5reCwgcHJv Z3JhbSwgdW5pb24sIHVmcywgY2RmcywgCkRlYyAyNyAxNDoxNzo0OCBkdWNrIGFtZFs1ODBd L2luZm86ICAgICAgICBwY2ZzLCB0bXBmcywgdWRmLCBhdXRvLCBkaXJlY3QsIHRvcGx2bCwg ZXJyb3IsIGluaGVyaXQuCkRlYyAyNyAxNDoxNzo0OCBkdWNrIGFtZFs1ODBdL2luZm86ICBG UzogY2Q5NjYwLCBtZnMsIG5mcywgbmZzMywgbnVsbGZzLCBtc2Rvc2ZzLCB0bXBmcywgdWZz LCB1bmlvbmZzLgpEZWMgMjcgMTQ6MTc6NDggZHVjayBhbWRbNTgwXS9pbmZvOiAgTmV0d29y ayAxOiB3aXJlPSIxMjguMjA4LjIzMi4wIiAobmV0bnVtYmVyPTEyOC4yMDguMjMyKS4KRGVj IDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4MF0vaW5mbzogIE5ldHdvcmsgMjogd2lyZT0iMTky LjE2OC4yMzIuMCIgKG5ldG51bWJlcj0xOTIuMTY4LjIzMikuCkRlYyAyNyAxNDoxNzo0OCBk dWNrIGFtZFs1ODBdL2luZm86ICBNeSBpcCBhZGRyIGlzIDEyNy4wLjAuMQpEZWMgMjcgMTQ6 MTc6NDggZHVjayBhbWRbNTgxXS9pbmZvOiAgcmVsZWFzZWQgY29udHJvbGxpbmcgdHR5IHVz aW5nIHNldHNpZCgpCkRlYyAyNyAxNDoxNzo0OCBkdWNrIGFtZFs1ODFdL2luZm86ICBMb2Nr ZWQgcHJvY2VzcyBwYWdlcyBpbiBtZW1vcnkKRGVjIDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4 MV0vaW5mbzogIGZpbGUgc2VydmVyIGxvY2FsaG9zdCwgdHlwZSBsb2NhbCwgc3RhdGUgc3Rh cnRzIHVwCkRlYyAyNyAxNDoxNzo0OCBkdWNrIGFtZFs1ODFdL2luZm86ICB6cm9vdC9ST09U L2RlZmF1bHQgcmVzdGFydGVkIGZzdHlwZSBsaW5rIG9uIC8sIGZsYWdzIDB4ODkKRGVjIDI3 IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4MV0vaW5mbzogIGRldmZzIHJlc3RhcnRlZCBmc3R5cGUg bGluayBvbiAvZGV2LCBmbGFncyAweDg5CkRlYyAyNyAxNDoxNzo0OCBkdWNrIGFtZFs1ODFd L2luZm86ICB6cm9vdCByZXN0YXJ0ZWQgZnN0eXBlIGxpbmsgb24gL3pyb290LCBmbGFncyAw eDg5CkRlYyAyNyAxNDoxNzo0OCBkdWNrIGFtZFs1ODFdL2luZm86ICB6cm9vdC9uaWtvbGEg cmVzdGFydGVkIGZzdHlwZSBsaW5rIG9uIC9uaWtvbGEsIGZsYWdzIDB4ODkKRGVjIDI3IDE0 OjE3OjQ4IGR1Y2sgYW1kWzU4MV0vaW5mbzogIHpyb290L3Zhci9jcmFzaCByZXN0YXJ0ZWQg ZnN0eXBlIGxpbmsgb24gL3Zhci9jcmFzaCwgZmxhZ3MgMHg4OQpEZWMgMjcgMTQ6MTc6NDgg ZHVjayBhbWRbNTgxXS9pbmZvOiAgenJvb3QvdmFyL2xvZyByZXN0YXJ0ZWQgZnN0eXBlIGxp bmsgb24gL3Zhci9sb2csIGZsYWdzIDB4ODkKRGVjIDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4 MV0vaW5mbzogIHpyb290L3Vzci9ob21lIHJlc3RhcnRlZCBmc3R5cGUgbGluayBvbiAvdXNy L2hvbWUsIGZsYWdzIDB4ODkKRGVjIDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4MV0vaW5mbzog IHpyb290L3Zhci9hdWRpdCByZXN0YXJ0ZWQgZnN0eXBlIGxpbmsgb24gL3Zhci9hdWRpdCwg ZmxhZ3MgMHg4OQpEZWMgMjcgMTQ6MTc6NDggZHVjayBhbWRbNTgxXS9pbmZvOiAgenJvb3Qv dXNyL3NyYyByZXN0YXJ0ZWQgZnN0eXBlIGxpbmsgb24gL3Vzci9zcmMsIGZsYWdzIDB4ODkK RGVjIDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4MV0vaW5mbzogIHpyb290L3RtcCByZXN0YXJ0 ZWQgZnN0eXBlIGxpbmsgb24gL3RtcCwgZmxhZ3MgMHg4OQpEZWMgMjcgMTQ6MTc6NDggZHVj ayBhbWRbNTgxXS9pbmZvOiAgenJvb3QvdmFyL21haWwgcmVzdGFydGVkIGZzdHlwZSBsaW5r IG9uIC92YXIvbWFpbCwgZmxhZ3MgMHg4OQpEZWMgMjcgMTQ6MTc6NDggZHVjayBhbWRbNTgx XS9pbmZvOiAgenJvb3QvdmFyL3RtcCByZXN0YXJ0ZWQgZnN0eXBlIGxpbmsgb24gL3Zhci90 bXAsIGZsYWdzIDB4ODkKRGVjIDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4MV0vaW5mbzogIHpy b290L3Vzci9wb3J0cyByZXN0YXJ0ZWQgZnN0eXBlIGxpbmsgb24gL3Vzci9wb3J0cywgZmxh Z3MgMHg4OQpEZWMgMjcgMTQ6MTc6NDggZHVjayBhbWRbNTgxXS9pbmZvOiAgenJvb3Qvbmlr b2xhL25pa29sYSByZXN0YXJ0ZWQgZnN0eXBlIGxpbmsgb24gL25pa29sYS9uaWtvbGEsIGZs YWdzIDB4ODkKRGVjIDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4MV0vbWFwOiAgIFRyeWluZyBt b3VudCBvZiAvZXRjL2FtZC9hbWQuaG9tZXMgb24gL3VzZXJzIGZzdHlwZSB0b3BsdmwgbW91 bnRfdHlwZSBub24tYXV0b2ZzCkRlYyAyNyAxNDoxNzo0OCBkdWNrIGFtZFs1ODFdL2luZm86 ICBjcmVhdGluZyBtb3VudHBvaW50IGRpcmVjdG9yeSAnL3VzZXJzJwpEZWMgMjcgMTQ6MTc6 NDggZHVjayBhbWRbNTgxXS9tYXA6ICAgVHJ5aW5nIG1vdW50IG9mIC9ldGMvYW1kL2FtZC5u ZXR3b3JrIG9uIC9uIGZzdHlwZSB0b3BsdmwgbW91bnRfdHlwZSBub24tYXV0b2ZzCkRlYyAy NyAxNDoxNzo0OCBkdWNrIGFtZFs1ODFdL2luZm86ICBjcmVhdGluZyBtb3VudHBvaW50IGRp cmVjdG9yeSAnL24nCkRlYyAyNyAxNDoxNzo0OCBkdWNrIGFtZFs1ODJdL2luZm86ICAvdXNl cnM6IGRpc2FibGluZyBuZnMgY29uZ2VzdGlvbiB3aW5kb3cKRGVjIDI3IDE0OjE3OjQ4IGR1 Y2sgYW1kWzU4MV0vbWFwOiAgIFRyeWluZyBtb3VudCBvZiAvZXRjL2FtZC9hbWQuZ3JvdXAg b24gL2cgZnN0eXBlIHRvcGx2bCBtb3VudF90eXBlIG5vbi1hdXRvZnMKRGVjIDI3IDE0OjE3 OjQ4IGR1Y2sgYW1kWzU4MV0vaW5mbzogIGNyZWF0aW5nIG1vdW50cG9pbnQgZGlyZWN0b3J5 ICcvZycKRGVjIDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4M10vaW5mbzogIC9uOiBkaXNhYmxp bmcgbmZzIGNvbmdlc3Rpb24gd2luZG93CkRlYyAyNyAxNDoxNzo0OCBkdWNrIGFtZFs1ODRd L2luZm86ICAvZzogZGlzYWJsaW5nIG5mcyBjb25nZXN0aW9uIHdpbmRvdwpEZWMgMjcgMTQ6 MTc6NDggZHVjayBhbWRbNTgxXS9pbmZvOiAgaW5pdGlhbGl6aW5nIGFtZC5jb25mIG1hcCAv ZXRjL2FtZC9hbWQuZ3JvdXAgb2YgdHlwZSBmaWxlCkRlYyAyNyAxNDoxNzo0OCBkdWNrIGFt ZFs1ODFdL2luZm86ICBmaXJzdCB0aW1lIGxvYWQgb2YgbWFwIC9ldGMvYW1kL2FtZC5ncm91 cCBzdWNjZWVkZWQKRGVjIDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4MV0vaW5mbzogIC9ldGMv YW1kL2FtZC5ncm91cCBtb3VudGVkIGZzdHlwZSB0b3Bsdmwgb24gL2cKRGVjIDI3IDE0OjE3 OjQ4IGR1Y2sgYW1kWzU4MV0vaW5mbzogIC9nIHNldCB0byBuZXZlciB0aW1lb3V0CkRlYyAy NyAxNDoxNzo0OCBkdWNrIGFtZFs1ODFdL2luZm86ICBpbml0aWFsaXppbmcgYW1kLmNvbmYg bWFwIC9ldGMvYW1kL2FtZC5ob21lcyBvZiB0eXBlIGZpbGUKRGVjIDI3IDE0OjE3OjQ4IGR1 Y2sgYW1kWzU4MV0vaW5mbzogIGZpcnN0IHRpbWUgbG9hZCBvZiBtYXAgL2V0Yy9hbWQvYW1k LmhvbWVzIHN1Y2NlZWRlZApEZWMgMjcgMTQ6MTc6NDggZHVjayBhbWRbNTgxXS9pbmZvOiAg L2V0Yy9hbWQvYW1kLmhvbWVzIG1vdW50ZWQgZnN0eXBlIHRvcGx2bCBvbiAvdXNlcnMKRGVj IDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4MV0vaW5mbzogIC91c2VycyBzZXQgdG8gbmV2ZXIg dGltZW91dApEZWMgMjcgMTQ6MTc6NDggZHVjayBhbWRbNTgxXS9pbmZvOiAgaW5pdGlhbGl6 aW5nIGFtZC5jb25mIG1hcCAvZXRjL2FtZC9hbWQubmV0d29yayBvZiB0eXBlIGZpbGUKRGVj IDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4MV0vaW5mbzogIGZpcnN0IHRpbWUgbG9hZCBvZiBt YXAgL2V0Yy9hbWQvYW1kLm5ldHdvcmsgc3VjY2VlZGVkCkRlYyAyNyAxNDoxNzo0OCBkdWNr IGFtZFs1ODFdL2luZm86ICAvZXRjL2FtZC9hbWQubmV0d29yayBtb3VudGVkIGZzdHlwZSB0 b3Bsdmwgb24gL24KRGVjIDI3IDE0OjE3OjQ4IGR1Y2sgYW1kWzU4MV0vaW5mbzogIC9uIHNl dCB0byBuZXZlciB0aW1lb3V0CkRlYyAyNyAxNDoxODoyOSBkdWNrIGFtZFs1ODFdL21hcDog ICBUcnlpbmcgbW91bnQgb2YgLi9uL2NoaWNrZW4vaG9tZS1sdmQgb24gLi9uL2NoaWNrZW4v aG9tZS1sdmQgZnN0eXBlIGxpbmsgbW91bnRfdHlwZSBub24tYXV0b2ZzCkRlYyAyNyAxNDox ODoyOSBkdWNrIGFtZFs1ODFdL2luZm86ICAuL24vY2hpY2tlbi9ob21lLWx2ZCBtb3VudGVk IGZzdHlwZSBsaW5rIG9uIC4vbi9jaGlja2VuL2hvbWUtbHZkCkRlYyAyNyAxNDoxODoyOSBk dWNrIGFtZFs1ODFdL2luZm86ICAvdXNlcnMvbHZkIHNldCB0byB0aW1lb3V0IGluIDMwMCBz ZWNvbmRzCkRlYyAyNyAxNDoxODoyOSBkdWNrIGFtZFs1ODFdL21hcDogICBtYXRjaGVkIGRl ZmF1bHQgc2VsZWN0b3JzICJ0eXBlOj1uZnM7b3B0czo9aGFyZCxwcm90bz10Y3AiCkRlYyAy NyAxNDoxODoyOSBkdWNrIGFtZFs1ODFdL21hcDogICBUcnlpbmcgbW91bnQgb2YgL2V0Yy9h bWQvYW1kLm5ldHdvcmsgb24gL24vY2hpY2tlbiBmc3R5cGUgYXV0byBtb3VudF90eXBlIG5v bi1hdXRvZnMKRGVjIDI3IDE0OjE4OjI5IGR1Y2sgYW1kWzU4MV0vaW5mbzogIC9ldGMvYW1k L2FtZC5uZXR3b3JrIG1vdW50ZWQgZnN0eXBlIGF1dG8gb24gL24vY2hpY2tlbgpEZWMgMjcg MTQ6MTg6MjkgZHVjayBhbWRbNTgxXS9pbmZvOiAgL24vY2hpY2tlbiBzZXQgdG8gdGltZW91 dCBpbiAzMDAgc2Vjb25kcwpEZWMgMjcgMTQ6MTg6MjkgZHVjayBhbWRbNTgxXS9tYXA6ICAg bWF0Y2hlZCBkZWZhdWx0IHNlbGVjdG9ycyAidHlwZTo9bmZzO29wdHM6PWhhcmQscHJvdG89 dGNwIgpEZWMgMjcgMTQ6MTg6MjkgZHVjayBhbWRbNTgxXS9tYXA6ICAgVHJ5aW5nIG1vdW50 IG9mIC4vbi9jaGlja2VuL2hvbWUvbHZkIG9uIC4vbi9jaGlja2VuL2hvbWUvbHZkIGZzdHlw ZSBsaW5rIG1vdW50X3R5cGUgbm9uLWF1dG9mcwpEZWMgMjcgMTQ6MTg6MjkgZHVjayBhbWRb NTgxXS9pbmZvOiAgLi9uL2NoaWNrZW4vaG9tZS9sdmQgbW91bnRlZCBmc3R5cGUgbGluayBv biAuL24vY2hpY2tlbi9ob21lL2x2ZApEZWMgMjcgMTQ6MTg6MjkgZHVjayBhbWRbNTgxXS9p bmZvOiAgL24vY2hpY2tlbi9ob21lLWx2ZCBzZXQgdG8gdGltZW91dCBpbiAzMDAgc2Vjb25k cwpEZWMgMjcgMTQ6MTg6MjkgZHVjayBhbWRbNTgxXS9tYXA6ICAgbWF0Y2hlZCBkZWZhdWx0 IHNlbGVjdG9ycyAidHlwZTo9bmZzO29wdHM6PWhhcmQscHJvdG89dGNwIgpEZWMgMjcgMTQ6 MTg6MjkgZHVjayBhbWRbNTgxXS9tYXA6ICAgVHJ5aW5nIG1vdW50IG9mIC9ldGMvYW1kL2Ft ZC5uZXR3b3JrIG9uIC9uL2NoaWNrZW4vaG9tZSBmc3R5cGUgYXV0byBtb3VudF90eXBlIG5v bi1hdXRvZnMKRGVjIDI3IDE0OjE4OjI5IGR1Y2sgYW1kWzU4MV0vaW5mbzogIC9ldGMvYW1k L2FtZC5uZXR3b3JrIG1vdW50ZWQgZnN0eXBlIGF1dG8gb24gL24vY2hpY2tlbi9ob21lCkRl YyAyNyAxNDoxODoyOSBkdWNrIGFtZFs1ODFdL2luZm86ICAvbi9jaGlja2VuL2hvbWUgc2V0 IHRvIHRpbWVvdXQgaW4gMzAwIHNlY29uZHMKRGVjIDI3IDE0OjE4OjI5IGR1Y2sgYW1kWzU4 MV0vbWFwOiAgIG1hdGNoZWQgZGVmYXVsdCBzZWxlY3RvcnMgInR5cGU6PW5mcztvcHRzOj1o YXJkLHByb3RvPXRjcCIKRGVjIDI3IDE0OjE4OjI5IGR1Y2sgYW1kWzU4MV0vaW5mbzogIGdl dF9uZnNfdmVyc2lvbjogcmV0dXJuaW5nIE5GUygzLHRjcCkgb24gaG9zdCBjaGlja2VuLmVl Lndhc2hpbmd0b24uZWR1CkRlYyAyNyAxNDoxODoyOSBkdWNrIGFtZFs1ODFdL2luZm86ICBV c2luZyBORlMgdmVyc2lvbiAzLCBwcm90b2NvbCB0Y3Agb24gaG9zdCBjaGlja2VuLmVlLndh c2hpbmd0b24uZWR1CkRlYyAyNyAxNDoxODoyOSBkdWNrIGFtZFs1ODFdL2luZm86ICBpbml0 aWFsaXppbmcgY2hpY2tlbi5lZS53YXNoaW5ndG9uLmVkdSdzIHBpbmdlciB0byAzMCBzZWMK RGVjIDI3IDE0OjE4OjI5IGR1Y2sgYW1kWzU4MV0vaW5mbzogIGNyZWF0ZV9waW5nX3BheWxv YWQ6IG5mc192ZXJzaW9uOiAzCkRlYyAyNyAxNDoxODoyOSBkdWNrIGFtZFs1ODFdL21hcDog ICBUcnlpbmcgbW91bnQgb2YgY2hpY2tlbjovdm9sL2hvbWUvbHZkIG9uIC9hdG0vY2hpY2tl bi92b2wvaG9tZS9sdmQgZnN0eXBlIG5mcyBtb3VudF90eXBlIG5vbi1hdXRvZnMKRGVjIDI3 IDE0OjE4OjI5IGR1Y2sgYW1kWzU4MV0vaW5mbzogIGZpbGUgc2VydmVyIGNoaWNrZW4uZWUu d2FzaGluZ3Rvbi5lZHUsIHR5cGUgbmZzLCBzdGF0ZSBzdGFydHMgdXAKRGVjIDI3IDE0OjE4 OjI5IGR1Y2sgYW1kWzU4MV0vaW5mbzogIEZsdXNoZWQgL24vY2hpY2tlbi9ob21lL2x2ZDsg ZGVwZW5kZW50IG9uIGNoaWNrZW4uZWUud2FzaGluZ3Rvbi5lZHUKRGVjIDI3IDE0OjE4OjI5 IGR1Y2sgYW1kWzU4MV0vaW5mbzogIHJlY29tcHV0ZV9wb3J0bWFwOiBORlMgdmVyc2lvbiAz IG9uIGNoaWNrZW4uZWUud2FzaGluZ3Rvbi5lZHUKRGVjIDI3IDE0OjE4OjI5IGR1Y2sgYW1k WzU4MV0vaW5mbzogIFVzaW5nIE1PVU5UIHZlcnNpb246IDMKRGVjIDI3IDE0OjE4OjI5IGR1 Y2sgYW1kWzU4MV0vbWFwOiAgIFRyeWluZyBtb3VudCBvZiBjaGlja2VuOi92b2wvaG9tZS9s dmQgb24gL2F0bS9jaGlja2VuL3ZvbC9ob21lL2x2ZCBmc3R5cGUgbmZzIG1vdW50X3R5cGUg bm9uLWF1dG9mcwpEZWMgMjcgMTQ6MTg6MjkgZHVjayBhbWRbNTgxXS9pbmZvOiAgY2FsbF9t b3VudGQ6IE5GUyB2ZXJzaW9uIDMsIG1vdW50IHZlcnNpb24gMwpEZWMgMjcgMTQ6MTg6Mjkg ZHVjayBhbWRbNTgxXS9tYXA6ICAgVHJ5aW5nIG1vdW50IG9mIGNoaWNrZW46L3ZvbC9ob21l L2x2ZCBvbiAvYXRtL2NoaWNrZW4vdm9sL2hvbWUvbHZkIGZzdHlwZSBuZnMgbW91bnRfdHlw ZSBub24tYXV0b2ZzCkRlYyAyNyAxNDoxODoyOSBkdWNrIGFtZFs1ODFdL2luZm86ICBwcmlt ZV9uZnNfZmhhbmRsZV9jYWNoZTogTkZTIHZlcnNpb24gMwpEZWMgMjcgMTQ6MTg6MjkgZHVj ayBhbWRbNTgxXS9pbmZvOiAgY3JlYXRpbmcgbW91bnRwb2ludCBkaXJlY3RvcnkgJy9hdG0v Y2hpY2tlbi92b2wvaG9tZS9sdmQnCkRlYyAyNyAxNDoxODoyOSBkdWNrIGFtZFs3NjddL2lu Zm86ICBtb3VudF9uZnNfZmg6IE5GUyB2ZXJzaW9uIDMKRGVjIDI3IDE0OjE4OjI5IGR1Y2sg YW1kWzc2N10vaW5mbzogIG1vdW50X25mc19maDogdXNpbmcgTkZTIHRyYW5zcG9ydCB0Y3AK RGVjIDI3IDE0OjE4OjI5IGR1Y2sgYW1kWzU4MV0vaW5mbzogIGNoaWNrZW46L3ZvbC9ob21l L2x2ZCBtb3VudGVkIGZzdHlwZSBuZnMgb24gL2F0bS9jaGlja2VuL3ZvbC9ob21lL2x2ZApE ZWMgMjcgMTQ6MTg6MjkgZHVjayBhbWRbNTgxXS9pbmZvOiAgL24vY2hpY2tlbi9ob21lL2x2 ZCBzZXQgdG8gdGltZW91dCBpbiAzMDAgc2Vjb25kcwpEZWMgMjcgMTQ6MTg6MzUgZHVjayBh bWRbNTgxXS93YXJuOiAgV0FSTklORzogYXV0b21vdW50ZXIgZ29pbmcgZG93biBvbiBzaWdu YWwgMTUKRGVjIDI3IDE0OjE4OjM1IGR1Y2sgYW1kWzU4MV0vaW5mbzogIHByaW1lX25mc19m aGFuZGxlX2NhY2hlOiBORlMgdmVyc2lvbiAzCkRlYyAyNyAxNDoxODozNSBkdWNrIGFtZFs1 ODFdL2luZm86ICByZWNvbXB1dGVfcG9ydG1hcDogTkZTIHZlcnNpb24gMyBvbiBjaGlja2Vu LmVlLndhc2hpbmd0b24uZWR1CkRlYyAyNyAxNDoxODozNSBkdWNrIGFtZFs1ODFdL2luZm86 ICBVc2luZyBNT1VOVCB2ZXJzaW9uOiAzCkRlYyAyNyAxNDoxODozNSBkdWNrIGFtZFs1ODFd L2luZm86ICBjYWxsX21vdW50ZDogTkZTIHZlcnNpb24gMywgbW91bnQgdmVyc2lvbiAzCkRl YyAyNyAxNDoxODozNSBkdWNrIGFtZFs1ODFdL2luZm86ICByZW1vdmluZyBtb3VudHBvaW50 IGRpcmVjdG9yeSAnL2F0bS9jaGlja2VuL3ZvbC9ob21lL2x2ZCcKRGVjIDI3IDE0OjE4OjM1 IGR1Y2sgYW1kWzU4MV0vaW5mbzogIGNoaWNrZW46L3ZvbC9ob21lL2x2ZCB1bm1vdW50ZWQg ZnN0eXBlIG5mcyBmcm9tIC9hdG0vY2hpY2tlbi92b2wvaG9tZS9sdmQKRGVjIDI3IDE0OjE4 OjM1IGR1Y2sgYW1kWzU4MV0vaW5mbzogIC9ldGMvYW1kL2FtZC5uZXR3b3JrIHVubW91bnRl ZCBmc3R5cGUgYXV0byBmcm9tIC9uL2NoaWNrZW4vaG9tZQpEZWMgMjcgMTQ6MTg6MzUgZHVj ayBhbWRbNTgxXS9pbmZvOiAgLi9uL2NoaWNrZW4vaG9tZS9sdmQgdW5tb3VudGVkIGZzdHlw ZSBsaW5rIGZyb20gLi9uL2NoaWNrZW4vaG9tZS9sdmQKRGVjIDI3IDE0OjE4OjM1IGR1Y2sg YW1kWzU4MV0vaW5mbzogIC9ldGMvYW1kL2FtZC5uZXR3b3JrIHVubW91bnRlZCBmc3R5cGUg YXV0byBmcm9tIC9uL2NoaWNrZW4KRGVjIDI3IDE0OjE4OjM1IGR1Y2sgYW1kWzU4MV0vaW5m bzogIC4vbi9jaGlja2VuL2hvbWUtbHZkIHVubW91bnRlZCBmc3R5cGUgbGluayBmcm9tIC4v bi9jaGlja2VuL2hvbWUtbHZkCkRlYyAyNyAxNDoxODozNSBkdWNrIGFtZFs1ODFdL2luZm86 ICByZW1vdmluZyBtb3VudHBvaW50IGRpcmVjdG9yeSAnL2cnCkRlYyAyNyAxNDoxODozNSBk dWNrIGFtZFs1ODFdL2luZm86ICAvZXRjL2FtZC9hbWQuZ3JvdXAgdW5tb3VudGVkIGZzdHlw ZSB0b3BsdmwgZnJvbSAvZwpEZWMgMjcgMTQ6MTg6MzYgZHVjayBhbWRbNTgxXS9pbmZvOiAg cmVtb3ZpbmcgbW91bnRwb2ludCBkaXJlY3RvcnkgJy9uJwpEZWMgMjcgMTQ6MTg6MzYgZHVj ayBhbWRbNTgxXS9pbmZvOiAgL2V0Yy9hbWQvYW1kLm5ldHdvcmsgdW5tb3VudGVkIGZzdHlw ZSB0b3BsdmwgZnJvbSAvbgpEZWMgMjcgMTQ6MTg6MzcgZHVjayBhbWRbNTgxXS9pbmZvOiAg cmVtb3ZpbmcgbW91bnRwb2ludCBkaXJlY3RvcnkgJy91c2VycycKRGVjIDI3IDE0OjE4OjM3 IGR1Y2sgYW1kWzU4MV0vaW5mbzogIC9ldGMvYW1kL2FtZC5ob21lcyB1bm1vdW50ZWQgZnN0 eXBlIHRvcGx2bCBmcm9tIC91c2VycwpEZWMgMjcgMTQ6MTg6MzggZHVjayBhbWRbNTgxXS9p bmZvOiAgRmluaXNoaW5nIHdpdGggc3RhdHVzIDAK --------------E6CE60CD2513FF880F77D12F--