From owner-freebsd-threads@FreeBSD.ORG Thu Feb 12 00:30:24 2015 Return-Path: Delivered-To: threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 5A1D131F for ; Thu, 12 Feb 2015 00:30:24 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 206223F8 for ; Thu, 12 Feb 2015 00:30:24 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.9/8.14.9) with ESMTP id t1C0UN7p084890 for ; Thu, 12 Feb 2015 00:30:23 GMT (envelope-from bdrewery@freefall.freebsd.org) Received: (from bdrewery@localhost) by freefall.freebsd.org (8.14.9/8.14.9/Submit) id t1C0UNLX084887 for threads@freebsd.org; Thu, 12 Feb 2015 00:30:23 GMT (envelope-from bdrewery) Received: (qmail 94482 invoked from network); 11 Feb 2015 18:30:21 -0600 Received: from unknown (HELO ?10.10.1.139?) (freebsd@shatow.net@10.10.1.139) by sweb.xzibition.com with ESMTPA; 11 Feb 2015 18:30:21 -0600 Message-ID: <54DBF417.4050206@FreeBSD.org> Date: Wed, 11 Feb 2015 18:30:15 -0600 From: Bryan Drewery Organization: FreeBSD User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: Fixing dlopen("libpthread.so") References: <20141226165337.GJ1754@kib.kiev.ua> In-Reply-To: <20141226165337.GJ1754@kib.kiev.ua> OpenPGP: id=6E4697CF; url=http://www.shatow.net/bryan/bryan2.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="uqQesVRhT3Cr78iS9TWbomwC5tot1vB4H" Cc: threads@freebsd.org, arch@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Feb 2015 00:30:24 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --uqQesVRhT3Cr78iS9TWbomwC5tot1vB4H Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 12/26/2014 10:53 AM, Konstantin Belousov wrote: > [Long] > Proposed patch does the following: [...] It seems libthr.3 needs to be updated for the dlopen(3) support, to remove some of r272070. Also note the ordering comment (which I know you may not be ready to change yet). > INTERACTION WITH RUN-TIME LINKER > The libthr library must appear before libc in the global order of > depended objects. >=20 > Loading libthr with the dlopen(3) call in the process after the pr= ogram > binary is activated is not supported, and causes miscellaneous and= hard- > to-diagnose misbehaviour. This is due to libthr interposing sever= al > important libc symbols to provide thread-safe services. In partic= ular, > errno and the locking stubs from libc are affected. This requirem= ent is > currently not enforced. >=20 > If the program loads any modules at run-time, and those modules ma= y > require threading services, the main program binary must be linked= with > libpthread, even if it does not require any services from the libr= ary. >=20 > libthr cannot be unloaded; the dlclose(3) function does not perfor= m any > action when called with a handle for libthr. One of the reasons i= s that > the interposing of libc functions cannot be undone. As for the dlclose(3) refusing to work on libthr, I cannot find the supporting code. Where is it? Thanks, Bryan --uqQesVRhT3Cr78iS9TWbomwC5tot1vB4H Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJU2/QfAAoJEDXXcbtuRpfPNEQIAJHLs6BDBmKoHfKdtNJF/Fqq WFfnHY8niMTgsF9EXQfOmBNxROXYKNK8oQjvJ0jLD1k0Ep1bhKDARBntxf55R6yc oueO2rdMH7014WEcNLGbLQF5jkpcpN0SsgGaNfNpMVqlUeVDnAhkglVG/xKPUgC6 bIjeja4/YWD5lXA4P/lwIPbJHbuhAoYfdoz2oyJ1umhQskbPjj1GYU3fpUr9hX1C zQBdSDlC/hkDFuO6gsan2/S5471tEMV0YcaeLRVaqjJYjk6MO0pmcF8Ta7OPdRkr 3sBQfWt/wTygDwyoL+xcuRDbibEfi0Kk8/FaPonyiN/gdf6ll8uOLx0W+as1v8g= =Vz97 -----END PGP SIGNATURE----- --uqQesVRhT3Cr78iS9TWbomwC5tot1vB4H-- From owner-freebsd-threads@FreeBSD.ORG Thu Feb 12 09:27:38 2015 Return-Path: Delivered-To: threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id DE82626D; Thu, 12 Feb 2015 09:27:37 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6836E33E; Thu, 12 Feb 2015 09:27:34 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t1C9RTnr061044 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 12 Feb 2015 11:27:29 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t1C9RTnr061044 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t1C9RTsi061043; Thu, 12 Feb 2015 11:27:29 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 12 Feb 2015 11:27:28 +0200 From: Konstantin Belousov To: Bryan Drewery Subject: Re: Fixing dlopen("libpthread.so") Message-ID: <20150212092728.GE34251@kib.kiev.ua> References: <20141226165337.GJ1754@kib.kiev.ua> <54DBF417.4050206@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <54DBF417.4050206@FreeBSD.org> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: threads@freebsd.org, arch@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Feb 2015 09:27:38 -0000 On Wed, Feb 11, 2015 at 06:30:15PM -0600, Bryan Drewery wrote: > On 12/26/2014 10:53 AM, Konstantin Belousov wrote: > > [Long] > > Proposed patch does the following: > [...] > > It seems libthr.3 needs to be updated for the dlopen(3) support, to > remove some of r272070. Also note the ordering comment (which I know you > may not be ready to change yet). Thanks for noting, I forgot about the man page and about the text. See the patch at the end of the message. > As for the dlclose(3) refusing to work on libthr, I cannot find the > supporting code. Where is it? libthr is linked with -z nodelete linker flag. diff --git a/lib/libthr/libthr.3 b/lib/libthr/libthr.3 index 4b636ce..8108e2f 100644 --- a/lib/libthr/libthr.3 +++ b/lib/libthr/libthr.3 @@ -1,5 +1,5 @@ .\" Copyright (c) 2005 Robert N. M. Watson -.\" Copyright (c) 2014 The FreeBSD Foundation, Inc. +.\" Copyright (c) 2014,2015 The FreeBSD Foundation, Inc. .\" All rights reserved. .\" .\" Part of this documentation was written by @@ -29,7 +29,7 @@ .\" .\" $FreeBSD$ .\" -.Dd September 26, 2014 +.Dd February 12, 2015 .Dt LIBTHR 3 .Os .Sh NAME @@ -200,45 +200,25 @@ Bigger values reduce the frequency of the FIFO discipline. The value must be between 0 and 255. .El .Sh INTERACTION WITH RUN-TIME LINKER -The +On load, .Nm -library must appear before -.Li libc -in the global order of depended objects. -.Pp -Loading -.Nm -with the -.Xr dlopen 3 -call in the process after the program binary is activated -is not supported, and causes miscellaneous and hard-to-diagnose misbehaviour. -This is due to -.Nm -interposing several important -.Li libc -symbols to provide thread-safe services. -In particular, -.Dv errno -and the locking stubs from -.Li libc -are affected. -This requirement is currently not enforced. -.Pp -If the program loads any modules at run-time, and those modules may require -threading services, the main program binary must be linked with -.Li libpthread , -even if it does not require any services from the library. +installs interposing handlers into the hooks exported by +.Li libc . +The interposers provide real locking implementation instead of the +stubs for single-threaded processes in +.Li , +cancellation support and some modifications to the signal operations. .Pp .Nm cannot be unloaded; the .Xr dlclose 3 function does not perform any action when called with a handle for .Nm . -One of the reasons is that the interposing of +One of the reasons is that the internal interposing of .Li libc functions cannot be undone. .Sh SIGNALS -The implementation also interposes the user-installed +The implementation interposes the user-installed .Xr signal 3 handlers. This interposing is done to postpone signal delivery to threads which From owner-freebsd-threads@FreeBSD.ORG Thu Feb 12 16:57:13 2015 Return-Path: Delivered-To: threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0B05AC85 for ; Thu, 12 Feb 2015 16:57:13 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C73CDCB5 for ; Thu, 12 Feb 2015 16:57:12 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.9/8.14.9) with ESMTP id t1CGvCDJ023301 for ; Thu, 12 Feb 2015 16:57:12 GMT (envelope-from bdrewery@freefall.freebsd.org) Received: (from bdrewery@localhost) by freefall.freebsd.org (8.14.9/8.14.9/Submit) id t1CGvCjP023300 for threads@freebsd.org; Thu, 12 Feb 2015 16:57:12 GMT (envelope-from bdrewery) Received: (qmail 25728 invoked from network); 12 Feb 2015 10:57:07 -0600 Received: from unknown (HELO ?10.10.1.139?) (freebsd@shatow.net@10.10.1.139) by sweb.xzibition.com with ESMTPA; 12 Feb 2015 10:57:07 -0600 Message-ID: <54DCDB69.1000609@FreeBSD.org> Date: Thu, 12 Feb 2015 10:57:13 -0600 From: Bryan Drewery Organization: FreeBSD User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: Fixing dlopen("libpthread.so") References: <20141226165337.GJ1754@kib.kiev.ua> <54DBF417.4050206@FreeBSD.org> <20150212092728.GE34251@kib.kiev.ua> In-Reply-To: <20150212092728.GE34251@kib.kiev.ua> OpenPGP: id=6E4697CF; url=http://www.shatow.net/bryan/bryan2.asc Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="wIg4sUGK1H0xGfQ40t3FjNs3iaWXiGWCx" Cc: threads@freebsd.org, arch@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Feb 2015 16:57:13 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --wIg4sUGK1H0xGfQ40t3FjNs3iaWXiGWCx Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2/12/2015 3:27 AM, Konstantin Belousov wrote: > On Wed, Feb 11, 2015 at 06:30:15PM -0600, Bryan Drewery wrote: >> On 12/26/2014 10:53 AM, Konstantin Belousov wrote: >>> [Long] >>> Proposed patch does the following: >> [...] >> >> It seems libthr.3 needs to be updated for the dlopen(3) support, to >> remove some of r272070. Also note the ordering comment (which I know y= ou >> may not be ready to change yet). > Thanks for noting, I forgot about the man page and about the text. > See the patch at the end of the message. Looks good. >=20 >> As for the dlclose(3) refusing to work on libthr, I cannot find the >> supporting code. Where is it? > libthr is linked with -z nodelete linker flag. >=20 Ah! Thanks. Regards, Bryan Drewery --wIg4sUGK1H0xGfQ40t3FjNs3iaWXiGWCx Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQEcBAEBAgAGBQJU3NtpAAoJEDXXcbtuRpfPg8QH/2P4iJ2iJKASkZmT4XWkAdGw Hqivvl5wn+tpE8YNE0PL+EzkqoFvka/KohyVX1jw1NRsDtnL5sVAQ7nxI0bDZrR1 x+gfETl8y/QOa3+ORd+7CcaDhuuc3NP0TA/DNMTMTlvEUyqnO/5IY3NqSQJzq/Mt 4OHqlqyjy39XwiDv1feXKurZIjpkBSpT7clF3KYKDITZ3TMHRdZUF+P31vJfC7t8 ohylRpRH49cV0CDEQiZNTEmKOM+IRUYnuPcf9jMKwkI10BUY1YZMUU1GVYqcT/y/ kJMGUeyetk11jz0xMFwrUrUXtsESRbaOcn1cBq0/4Ucfm8rYu+yuXfC/pTDtGDo= =i3vX -----END PGP SIGNATURE----- --wIg4sUGK1H0xGfQ40t3FjNs3iaWXiGWCx-- From owner-freebsd-threads@FreeBSD.ORG Thu Feb 12 22:59:12 2015 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id BD7E747D; Thu, 12 Feb 2015 22:59:12 +0000 (UTC) Received: from thyme.infocus-llc.com (thyme.infocus-llc.com [199.15.120.10]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9745BA30; Thu, 12 Feb 2015 22:59:12 +0000 (UTC) Received: from draco.over-yonder.net (c-75-65-60-66.hsd1.ms.comcast.net [75.65.60.66]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by thyme.infocus-llc.com (Postfix) with ESMTPSA id 96DA937B593; Thu, 12 Feb 2015 16:59:10 -0600 (CST) Received: by draco.over-yonder.net (Postfix, from userid 100) id 3kjtZ609VSz2Mm; Thu, 12 Feb 2015 16:59:10 -0600 (CST) Date: Thu, 12 Feb 2015 16:59:09 -0600 From: "Matthew D. Fuller" To: freebsd-threads@freebsd.org Subject: Troubles with BIND + r276630 (/r277317 MFC) Message-ID: <20150212225909.GB15127@over-yonder.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Editor: vi X-OS: FreeBSD User-Agent: Mutt/1.5.23-fullermd.4 (2014-03-12) X-Virus-Scanned: clamav-milter 0.98.6 at thyme.infocus-llc.com X-Virus-Status: Clean Cc: kib@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Feb 2015 22:59:12 -0000 I've been setting up a stable/10 system (currently r278451, ~early this week), and had some disturbing troubles out of BIND (net/bind910, but also tested bind99). Sporadically, on restart, I get errors like Feb 12 10:42:37 dns named[4071]: dns_master_load: /usr/local/etc/namedb/master/empty.db:1: isc_lex_gettoken() failed: I/O error Feb 12 10:42:37 dns named[4071]: dns_master_load: /usr/local/etc/namedb/master/empty.db:1: I/O error Feb 12 10:42:37 dns named[4071]: zone 112.100.in-addr.arpa/IN: loading from master file /usr/local/etc/namedb/master/empty.db failed: I/O error Feb 12 10:42:37 dns named[4071]: zone 112.100.in-addr.arpa/IN: not loaded due to errors. and similar. Nothing wrong with the files, and it's wildly inconsistent. Easy to reproduce; just hammering 'service named restart' while tail -f'ing the messages log will show some startups going problem-free, while others will get random errors in random files. I can also reproduce it on my -CURRENT workstation (r278359, ~late last week), but not on several stable/10 systems of late-Dec vintage. In some testing on the -CURRENT box, if I build libc/libthr as of r276629 and do LD_LIBRARY_PATH hackery (and obviously not the 'service' wrapper) to use them, I can kick the reload without ever getting troubles (tried at least a hundred times). Going to 276630, or using the system libs, it'll happen sporadically (maybe once a dozen or so on average?). So far, I haven't seen anything odd in anything but BIND. I was running a mid-Jan -CURRENT before this for a couple weeks too, so there should have been time for Firefox to throw a hissy fit or something, but no hint of trouble. So it's not clear whether the problem is in the libc/libthr changes or in BIND 9.9 and 9.10, but certainly something between the two is unhappy... -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. From owner-freebsd-threads@FreeBSD.ORG Fri Feb 13 08:44:50 2015 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id B4FE4BE8; Fri, 13 Feb 2015 08:44:50 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2740FE06; Fri, 13 Feb 2015 08:44:49 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t1D8iifO001171 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 13 Feb 2015 10:44:44 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t1D8iifO001171 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t1D8ih1O001170; Fri, 13 Feb 2015 10:44:43 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 13 Feb 2015 10:44:43 +0200 From: Konstantin Belousov To: "Matthew D. Fuller" Subject: Re: Troubles with BIND + r276630 (/r277317 MFC) Message-ID: <20150213084443.GJ34251@kib.kiev.ua> References: <20150212225909.GB15127@over-yonder.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150212225909.GB15127@over-yonder.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: kib@freebsd.org, freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Feb 2015 08:44:50 -0000 On Thu, Feb 12, 2015 at 04:59:09PM -0600, Matthew D. Fuller wrote: > I've been setting up a stable/10 system (currently r278451, ~early > this week), and had some disturbing troubles out of BIND (net/bind910, > but also tested bind99). Sporadically, on restart, I get errors like > > > Feb 12 10:42:37 dns named[4071]: dns_master_load: /usr/local/etc/namedb/master/empty.db:1: isc_lex_gettoken() failed: I/O error > Feb 12 10:42:37 dns named[4071]: dns_master_load: /usr/local/etc/namedb/master/empty.db:1: I/O error > Feb 12 10:42:37 dns named[4071]: zone 112.100.in-addr.arpa/IN: loading from master file /usr/local/etc/namedb/master/empty.db failed: I/O error > Feb 12 10:42:37 dns named[4071]: zone 112.100.in-addr.arpa/IN: not loaded due to errors. > > > and similar. Nothing wrong with the files, and it's wildly > inconsistent. Easy to reproduce; just hammering 'service named > restart' while tail -f'ing the messages log will show some startups > going problem-free, while others will get random errors in random > files. I can also reproduce it on my -CURRENT workstation (r278359, > ~late last week), but not on several stable/10 systems of late-Dec > vintage. > > In some testing on the -CURRENT box, if I build libc/libthr as of > r276629 and do LD_LIBRARY_PATH hackery (and obviously not the > 'service' wrapper) to use them, I can kick the reload without ever > getting troubles (tried at least a hundred times). Going to 276630, > or using the system libs, it'll happen sporadically (maybe once a > dozen or so on average?). > > > So far, I haven't seen anything odd in anything but BIND. I was > running a mid-Jan -CURRENT before this for a couple weeks too, so > there should have been time for Firefox to throw a hissy fit or > something, but no hint of trouble. So it's not clear whether the > problem is in the libc/libthr changes or in BIND 9.9 and 9.10, but > certainly something between the two is unhappy... Start with ktrace/kdump around the place where the error is thrown out. Does 'reload' involves full process restart, or only reinitialization of the existing program ? From owner-freebsd-threads@FreeBSD.ORG Fri Feb 13 20:48:53 2015 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 53990559 for ; Fri, 13 Feb 2015 20:48:53 +0000 (UTC) Received: from thyme.infocus-llc.com (thyme.infocus-llc.com [199.15.120.10]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2B23A8CA for ; Fri, 13 Feb 2015 20:48:52 +0000 (UTC) Received: from draco.over-yonder.net (c-75-65-60-66.hsd1.ms.comcast.net [75.65.60.66]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by thyme.infocus-llc.com (Postfix) with ESMTPSA id 61C2237B55B; Fri, 13 Feb 2015 14:48:46 -0600 (CST) Received: by draco.over-yonder.net (Postfix, from userid 100) id 3kkRd95fkzz2vc; Fri, 13 Feb 2015 14:48:45 -0600 (CST) Date: Fri, 13 Feb 2015 14:48:45 -0600 From: "Matthew D. Fuller" To: Konstantin Belousov Subject: Re: Troubles with BIND + r276630 (/r277317 MFC) Message-ID: <20150213204845.GF15127@over-yonder.net> References: <20150212225909.GB15127@over-yonder.net> <20150213084443.GJ34251@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150213084443.GJ34251@kib.kiev.ua> X-Editor: vi X-OS: FreeBSD User-Agent: Mutt/1.5.23-fullermd.4 (2014-03-12) X-Virus-Scanned: clamav-milter 0.98.6 at thyme.infocus-llc.com X-Virus-Status: Clean Cc: freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Feb 2015 20:48:53 -0000 On Fri, Feb 13, 2015 at 10:44:43AM +0200 I heard the voice of Konstantin Belousov, and lo! it spake thus: > > Start with ktrace/kdump around the place where the error is thrown > out. > > Does 'reload' involves full process restart, or only > reinitialization of the existing program ? Doing the full start of named. The I/O err in isc_lex_gettoken() appears only triggered when getc() hits EOF. ktrace is a little tough to get through with all the threads. But the thread that logs that error tracing back from that error being sent out a fd does stuff that looks like prepping for syslog like getpid, and then (priorly; working backward) read()'s the file apparently just peachy. RET read 228/0xe4 % wc empty.db 11 38 228 empty.db and the text reads just fine. So something is getting squirreled up above the syscall level. -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. From owner-freebsd-threads@FreeBSD.ORG Fri Feb 13 20:51:04 2015 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 3721D928 for ; Fri, 13 Feb 2015 20:51:04 +0000 (UTC) Received: from thyme.infocus-llc.com (thyme.infocus-llc.com [199.15.120.10]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0E647911 for ; Fri, 13 Feb 2015 20:51:03 +0000 (UTC) Received: from draco.over-yonder.net (c-75-65-60-66.hsd1.ms.comcast.net [75.65.60.66]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by thyme.infocus-llc.com (Postfix) with ESMTPSA id 3F73C37B58D; Fri, 13 Feb 2015 14:51:03 -0600 (CST) Received: by draco.over-yonder.net (Postfix, from userid 100) id 3kkRgp5FHdz2vn; Fri, 13 Feb 2015 14:51:02 -0600 (CST) Date: Fri, 13 Feb 2015 14:51:02 -0600 From: "Matthew D. Fuller" To: Konstantin Belousov Subject: Re: Troubles with BIND + r276630 (/r277317 MFC) Message-ID: <20150213205102.GG15127@over-yonder.net> References: <20150212225909.GB15127@over-yonder.net> <20150213084443.GJ34251@kib.kiev.ua> <20150213204845.GF15127@over-yonder.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150213204845.GF15127@over-yonder.net> X-Editor: vi X-OS: FreeBSD User-Agent: Mutt/1.5.23-fullermd.4 (2014-03-12) X-Virus-Scanned: clamav-milter 0.98.6 at thyme.infocus-llc.com X-Virus-Status: Clean Cc: freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Feb 2015 20:51:04 -0000 > The I/O err in isc_lex_gettoken() appears only triggered when getc() > hits EOF. Err, wait, I'm backward on this. It's in ferror(), so not EOF. -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. From owner-freebsd-threads@FreeBSD.ORG Fri Feb 13 21:08:20 2015 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id EA52BE87 for ; Fri, 13 Feb 2015 21:08:20 +0000 (UTC) Received: from thyme.infocus-llc.com (thyme.infocus-llc.com [199.15.120.10]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id BB3B7AE7 for ; Fri, 13 Feb 2015 21:08:20 +0000 (UTC) Received: from draco.over-yonder.net (c-75-65-60-66.hsd1.ms.comcast.net [75.65.60.66]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by thyme.infocus-llc.com (Postfix) with ESMTPSA id DD55937B58D; Fri, 13 Feb 2015 15:08:18 -0600 (CST) Received: by draco.over-yonder.net (Postfix, from userid 100) id 3kkS3j6wnHz2w9; Fri, 13 Feb 2015 15:08:17 -0600 (CST) Date: Fri, 13 Feb 2015 15:08:17 -0600 From: "Matthew D. Fuller" To: Konstantin Belousov Subject: Re: Troubles with BIND + r276630 (/r277317 MFC) Message-ID: <20150213210817.GH15127@over-yonder.net> References: <20150212225909.GB15127@over-yonder.net> <20150213084443.GJ34251@kib.kiev.ua> <20150213204845.GF15127@over-yonder.net> <20150213205102.GG15127@over-yonder.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150213205102.GG15127@over-yonder.net> X-Editor: vi X-OS: FreeBSD User-Agent: Mutt/1.5.23-fullermd.4 (2014-03-12) X-Virus-Scanned: clamav-milter 0.98.6 at thyme.infocus-llc.com X-Virus-Status: Clean Cc: freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 Feb 2015 21:08:21 -0000 On Fri, Feb 13, 2015 at 02:51:02PM -0600 I heard the voice of Matthew D. Fuller, and lo! it spake thus: > > The I/O err in isc_lex_gettoken() appears only triggered when getc() > > hits EOF. > > Err, wait, I'm backward on this. It's in ferror(), so not EOF. And dropping in an errno check says getc is setting EBADF. But remember, this is pretty random. I'm also sometimes getting "unexpected end of input" sometimes, with all sorts of line numbers. Then here's one that gives an "ignoring out-of-zone data" talking about something that isn't on the referenced line at all. Here's a "loading [...] failed: bad ttl". It's all over the place. Every place I've checked, the read() data in ktrace looks fine; it's getting mangled after that. -- Matthew Fuller (MF4839) | fullermd@over-yonder.net Systems/Network Administrator | http://www.over-yonder.net/~fullermd/ On the Internet, nobody can hear you scream. From owner-freebsd-threads@FreeBSD.ORG Sat Feb 14 09:39:37 2015 Return-Path: Delivered-To: freebsd-threads@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id D53B3440 for ; Sat, 14 Feb 2015 09:39:37 +0000 (UTC) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 46EC09E9 for ; Sat, 14 Feb 2015 09:39:37 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.9/8.14.9) with ESMTP id t1E9dVCs087771 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 14 Feb 2015 11:39:31 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.9.2 kib.kiev.ua t1E9dVCs087771 Received: (from kostik@localhost) by tom.home (8.14.9/8.14.9/Submit) id t1E9dVUd087770; Sat, 14 Feb 2015 11:39:31 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 14 Feb 2015 11:39:31 +0200 From: Konstantin Belousov To: "Matthew D. Fuller" Subject: Re: Troubles with BIND + r276630 (/r277317 MFC) Message-ID: <20150214093930.GV34251@kib.kiev.ua> References: <20150212225909.GB15127@over-yonder.net> <20150213084443.GJ34251@kib.kiev.ua> <20150213204845.GF15127@over-yonder.net> <20150213205102.GG15127@over-yonder.net> <20150213210817.GH15127@over-yonder.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150213210817.GH15127@over-yonder.net> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.0 X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on tom.home Cc: freebsd-threads@freebsd.org X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 14 Feb 2015 09:39:37 -0000 On Fri, Feb 13, 2015 at 03:08:17PM -0600, Matthew D. Fuller wrote: > On Fri, Feb 13, 2015 at 02:51:02PM -0600 I heard the voice of > Matthew D. Fuller, and lo! it spake thus: > > > The I/O err in isc_lex_gettoken() appears only triggered when getc() > > > hits EOF. > > > > Err, wait, I'm backward on this. It's in ferror(), so not EOF. > > And dropping in an errno check says getc is setting EBADF. > > > But remember, this is pretty random. I'm also sometimes getting > "unexpected end of input" sometimes, with all sorts of line numbers. > Then here's one that gives an "ignoring out-of-zone data" talking > about something that isn't on the referenced line at all. Here's a > "loading [...] failed: bad ttl". > > It's all over the place. Every place I've checked, the read() data in > ktrace looks fine; it's getting mangled after that. Look, I asked for data. The texts you posted probably could be qualified as your walkthrough notes for some investigation, but may be not. Anyway, I am even unable to completely parse the notes, and I do not see how to start drilling down without data.