From owner-freebsd-toolchain@freebsd.org Mon Jun 18 22:34:01 2018 Return-Path: Delivered-To: freebsd-toolchain@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B743F1002443; Mon, 18 Jun 2018 22:34:01 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2610:1c1:1:6074::16:84]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "freefall.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 498BF7766F; Mon, 18 Jun 2018 22:34:01 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (unknown [127.0.1.132]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by freefall.freebsd.org (Postfix) with ESMTPS id EFF1431B3; Mon, 18 Jun 2018 22:34:00 +0000 (UTC) (envelope-from bdrewery@FreeBSD.org) Received: from mail.xzibition.com (localhost [172.31.3.2]) by mail.xzibition.com (Postfix) with ESMTP id 1D267A4E8; Mon, 18 Jun 2018 22:34:00 +0000 (UTC) X-Virus-Scanned: amavisd-new at mail.xzibition.com Received: from mail.xzibition.com ([172.31.3.2]) by mail.xzibition.com (mail.xzibition.com [172.31.3.2]) (amavisd-new, port 10026) with LMTP id Z0ObkPuIcP_D; Mon, 18 Jun 2018 22:33:57 +0000 (UTC) Subject: Re: A head buildworld race visible in the ci.freebsd.org build history DKIM-Filter: OpenDKIM Filter v2.10.3 mail.xzibition.com D9DD5A4E3 To: Li-Wen Hsu Cc: FreeBSD Current , FreeBSD Toolchain References: <74EAD684-0E0B-453A-B746-156777CF604A@yahoo.com> <1884103f-d1fb-aca6-2edd-062e11d05617@FreeBSD.org> <20180618204517.GD2430@kib.kiev.ua> <068108ab-76f2-0f2d-fd92-11c838a4d391@FreeBSD.org> From: Bryan Drewery Openpgp: id=F9173CB2C3AAEA7A5C8A1F0935D771BB6E4697CF; url=http://www.shatow.net/bryan/bryan2.asc Autocrypt: addr=bdrewery@FreeBSD.org; prefer-encrypt=mutual; keydata= xsBNBFJphmsBCADiFgmS4bIzwZijrS31SjEMzg+n5zNellgM+HkShwehpqCiyhXdWrvH6dTZ a6u50pbUIX7doTR7W7PQHCjCTqtpwvcj0eulZva+iHFp+XrbgSFHn+VVXgkYP2MFySyZRFab D2qqzJBEJofhpv4HvY6uQI5K99pMqKr1Z/lHqsijYYu4RH2OfwB5PinId7xeldzWEonVoCr+ rfxzO/UrgA6v/3layGZcKNHFjmc3NqoN1DXtdaEHqtjIozzbndVkH6lkFvIpIrI6i5ox8pwp VxsxLCr/4Musd5CWgHiet5kSw2SzNeA8FbxdLYCpXNVu+uBACEbCUP+CSNy3NVfEUxsBABEB AAHNJEJyeWFuIERyZXdlcnkgPGJkcmV3ZXJ5QEZyZWVCU0Qub3JnPsLAgAQTAQoAKgIbAwUL CQgHAwUVCgkICwUWAwIBAAIeAQIXgAIZAQUCWujOIgUJCmB7NwAKCRA113G7bkaXz/xpB/9b /UWIPbieY1IeIuHF2pyYPE7Hytkh3HVsxMA0F5Ma2AYQsXZZeKNKWrF7RPyDyDwUklLHJkhm k3EfClBbHxf08kMIm1vWCJRtgxic9knY/bzYGiWMpHjg3cSd1XfrYH1autYqTZAjDwIkgOjU dR//Tbn4V36sY7y2jz+kdMVWvK53U32aZqiwBbCn4DPe1wSZcUs17mV/0uZdIoGdj74B1orN A/0py5vHYo6HcbBNoaR8pKRLf5VZNRsxqGIMhTucx4SJWcHpuRBWYyvJSFzwvxdK4ZD4Yqoc kFGPVtOXktVMai9exrLvP3G77fKMu8DI6j4QRU4wCesnHuIfRPFuzsBNBFJphmsBCACiVFPf kNfaFtUSuY0395ueo/rMyHPGPQ2iwvERFCpeFGSQSgagpenNHLpFQKTg/dl6FOoST5tqyxMq fyHGHDzzU51bvA/IfaGoNi/BIhTe/toZNMRvpcI3PLjiGcnJnuwCCbAVOAGdb+t5cZtpNdOI cKYmrYG3u9RiBpe6dTF+qLrD/8Bs1wjhduQ8fcNNgnkXu8xDH4ZxY0lIc3QgvYWp9vimlQe6 iKjUd2/DX28ETZcD5h6pYV331KMPTrEI0p0yvFijUZce8c1XHFyL1j9sBAha5qpszJl6Uq5i LolhKRcGfcdmtD72vHQjUYglUyudSJUVyo2gMYjdbiFKzJulABEBAAHCwGUEGAEKAA8CGwwF AlrozigFCQpgez0ACgkQNddxu25Gl8+m5Af/R3VEdxNMAcDIes9ADhQyofj20SPV3eCJ3HYR OebTSuNdOudGt4AAyA8Ks94u9hiIp5IGsc6RDsT9W7O2vgXhd6eV3eiY5Oif5xLIYrIDVu1Y 1GyRxRrPEn/QOqDN6uFZCPwK1aOapGcYCrO9lB0gMuTVfgHanU61rgC9tMX0OoAOyRd+V3/M 8lDNhjJdF/IpO3SdYzKfkwduy4qamw4Gphcx/RfYQvYLq/eDkP8d50PphWdboqWBwNRHayro W/07OGzfxM5fJ5mBsXPQcO2QcRjkyHf6xCM6Hi1qQL4OnXMNE/ZTX0lnOj1/pH93TlzSHZMP TaiiA/MBD3vGsXBmBg== Organization: FreeBSD Message-ID: Date: Mon, 18 Jun 2018 15:33:56 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="qOqODywPw7sr1YLQU9do3M2p18zqeWgUz" X-BeenThere: freebsd-toolchain@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Maintenance of FreeBSD's integrated toolchain List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jun 2018 22:34:02 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --qOqODywPw7sr1YLQU9do3M2p18zqeWgUz Content-Type: multipart/mixed; boundary="qlTVijLjr7Vess76ZBjTCf4WFmsuLr5Od"; protected-headers="v1" From: Bryan Drewery To: Li-Wen Hsu Cc: FreeBSD Current , FreeBSD Toolchain Message-ID: Subject: Re: A head buildworld race visible in the ci.freebsd.org build history References: <74EAD684-0E0B-453A-B746-156777CF604A@yahoo.com> <1884103f-d1fb-aca6-2edd-062e11d05617@FreeBSD.org> <20180618204517.GD2430@kib.kiev.ua> <068108ab-76f2-0f2d-fd92-11c838a4d391@FreeBSD.org> In-Reply-To: --qlTVijLjr7Vess76ZBjTCf4WFmsuLr5Od Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 6/18/2018 3:31 PM, Li-Wen Hsu wrote: > On Mon, Jun 18, 2018 at 6:27 PM Bryan Drewery wr= ote: >> >> On 6/18/2018 1:45 PM, Konstantin Belousov wrote: >>> On Mon, Jun 18, 2018 at 12:42:46PM -0700, Bryan Drewery wrote: >>>> On 6/15/2018 10:55 PM, Mark Millard wrote: >>>>> In watching ci.freebsd.org builds I've seen a notable >>>>> number of one time failures, such as (example from >>>>> powerpc64): >>>>> >>>>> --- all_subdir_lib/libufs --- >>>>> ranlib -D libufs.a >>>>> ranlib: fatal: Failed to open 'libufs.a' >>>>> *** [libufs.a] Error code 70 >>>>> >>>>> where the next build works despite the change being >>>>> irrelevant to whatever ranlib complained about. >>>>> >>>>> Other builds failed similarly: >>>>> >>>>> --- all_subdir_lib/libbsm --- >>>>> ranlib -D libbsm_p.a >>>>> ranlib: fatal: Failed to open 'libbsm_p.a' >>>>> *** [libbsm_p.a] Error code 70 >>>>> >>>>> and: >>>>> >>>>> --- kerberos5/lib__L --- >>>>> ranlib -D libgssapi_spnego_p.a >>>>> --- libgssapi_spnego.a --- >>>>> ranlib -D libgssapi_spnego.a >>>>> --- libgssapi_spnego_p.a --- >>>>> ranlib: fatal: Failed to open 'libgssapi_spnego_p.a' >>>>> *** [libgssapi_spnego_p.a] Error code 70 >>>>> >>>>> and so on. >>>>> >>>>> >>>>> It is not limited to powerpc64. For example, for aarch64 >>>>> there are: >>>>> >>>>> --- libpam_exec.a --- >>>>> building static pam_exec library >>>>> ar -crD libpam_exec.a `NM=3D'nm' NMFLAGS=3D'' lorder pam_exec.o |= tsort -q` >>>>> ranlib -D libpam_exec.a >>>>> ranlib: fatal: Failed to open 'libpam_exec.a' >>>>> *** [libpam_exec.a] Error code 70 >>>>> >>>>> and: >>>>> >>>>> --- all_subdir_lib/libusb --- >>>>> ranlib -D libusb.a >>>>> ranlib: fatal: Failed to open 'libusb.a' >>>>> *** [libusb.a] Error code 70 >>>>> >>>>> and: >>>>> >>>>> --- all_subdir_lib/libbsnmp --- >>>>> ranlib: fatal: Failed to open 'libbsnmp.a' >>>>> --- all_subdir_lib/ncurses --- >>>>> --- all_subdir_lib/ncurses/panelw --- >>>>> --- panel.pico --- >>>>> --- all_subdir_lib/libbsnmp --- >>>>> *** [libbsnmp.a] Error code 70 >>>>> >>>>> >>>>> Even amd64 gets such: >>>>> >>>>> --- libpcap.a --- >>>>> ranlib -D libpcap.a >>>>> ranlib: fatal: Failed to open 'libpcap.a' >>>>> *** [libpcap.a] Error code 70 >>>>> >>>>> and: >>>>> >>>>> >>>>> --- libkafs5.a --- >>>>> ranlib: fatal: Failed to open 'libkafs5.a' >>>>> --- libkafs5_p.a --- >>>>> ranlib: fatal: Failed to open 'libkafs5_p.a' >>>>> --- cddl/lib__L --- >>>>> /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/lua/lbaseli= b.c:60:26: note: include the header or explicitly provide a dec= laration for 'toupper' >>>>> --- kerberos5/lib__L --- >>>>> *** [libkafs5_p.a] Error code 70 >>>>> >>>>> make[5]: stopped in /usr/src/kerberos5/lib/libkafs5 >>>>> --- libkafs5.a --- >>>>> *** [libkafs5.a] Error code 70 >>>>> >>>>> and: >>>>> >>>>> >>>>> --- lib__L --- >>>>> ranlib -D libclang_rt.asan_cxx-i386.a >>>>> ranlib: fatal: Failed to open 'libclang_rt.asan_cxx-i386.a' >>>>> *** [libclang_rt.asan_cxx-i386.a] Error code 70 >>>>> >>>>> >>>>> (Notice the variability in what .a the ranlib's fail for.) >>>>> >>>>> >>>>> >>>>> >>>>> >>>> >>>> >>>> I looked at this a few days ago and don't believe it's actually a bu= ild >>>> race. I think there is something wrong with the ar/ranlib on that sy= stem >>>> or something else. I've found no evidence of concurrent building of = the >>>> .a files in question. >>> >>> FWIW, I got the similar failure when I did last checks for the OFED >>> commit. For me, it was libgcc.a. >>> >> >> If it was -lgcc_s then it's a known rare build race due to >> tools/install.sh not handling -S. >=20 > It seems a more general problem, this one: >=20 > https://ci.freebsd.org/job/FreeBSD-head-aarch64-build/8190/console >=20 > calls for libcuse_p.a, while this one: >=20 > https://ci.freebsd.org/job/FreeBSD-head-mips-build/2919/console >=20 > calls for libfifolog.a >=20 Well why is ar -> ranlib so special? Nothing else is failing. What filesystem are these using for objdirs? What revision is the host kernel? --=20 Regards, Bryan Drewery --qlTVijLjr7Vess76ZBjTCf4WFmsuLr5Od-- --qOqODywPw7sr1YLQU9do3M2p18zqeWgUz Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJbKDNUAAoJEDXXcbtuRpfPizUIALxtnZDaono3T4sjzUTN9qIM GqoVNd6mb+d61ntQ+UKp6hdgKni7tikPsqFiSf4aAC0NSj2FurbB4sgKKjdsCl8R WtzerM8WpPEy/ro2Ow5hxntGfW+F308W5dePXGS+ugkTqz3FMuJ0tGfNre/cEfHX f91iVrouAW45Bfg8IRIqdk2Py7aaDTRXUbcSYyiFb9jwQf0EWS5seR9SEu30Yjck MStso/CDuNBUpEzPfObww2lr7TaJFdg51KfshQ6cFkby6mzlylRMigsUlXcyR3Z7 Xzb7qvL3XPsBxgPYmFhemEqCIlX94lEvxBGtVzGrPT0k55BEB/DYoxVj5ym98N8= =O4Hw -----END PGP SIGNATURE----- --qOqODywPw7sr1YLQU9do3M2p18zqeWgUz--