From owner-svn-src-head@FreeBSD.ORG Sun May 6 18:04:03 2012 Return-Path: Delivered-To: svn-src-head@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1AC2A1065674; Sun, 6 May 2012 18:04:03 +0000 (UTC) (envelope-from bz@FreeBSD.org) Received: from mx1.sbone.de (mx1.sbone.de [IPv6:2a01:4f8:130:3ffc::401:25]) by mx1.freebsd.org (Postfix) with ESMTP id 4DC4F8FC20; Sun, 6 May 2012 18:04:02 +0000 (UTC) Received: from mail.sbone.de (mail.sbone.de [IPv6:fde9:577b:c1a9:31::2013:587]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.sbone.de (Postfix) with ESMTPS id CEB6125D3A00; Sun, 6 May 2012 18:04:00 +0000 (UTC) Received: from content-filter.sbone.de (content-filter.sbone.de [IPv6:fde9:577b:c1a9:31::2013:2742]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPS id BABC9BE6619; Sun, 6 May 2012 18:03:59 +0000 (UTC) X-Virus-Scanned: amavisd-new at sbone.de Received: from mail.sbone.de ([IPv6:fde9:577b:c1a9:31::2013:587]) by content-filter.sbone.de (content-filter.sbone.de [fde9:577b:c1a9:31::2013:2742]) (amavisd-new, port 10024) with ESMTP id W736mFn0vNlT; Sun, 6 May 2012 18:03:57 +0000 (UTC) Received: from orange-en1.sbone.de (orange-en1.sbone.de [IPv6:fde9:577b:c1a9:31:cabc:c8ff:fecf:e8e3]) (using TLSv1 with cipher AES128-SHA (128/128 bits)) (No client certificate requested) by mail.sbone.de (Postfix) with ESMTPSA id DDAEDBE6618; Sun, 6 May 2012 18:03:55 +0000 (UTC) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: "Bjoern A. Zeeb" In-Reply-To: <20120421171128.GA6732@lor.one-eyed-alien.net> Date: Sun, 6 May 2012 18:03:54 +0000 Content-Transfer-Encoding: quoted-printable Message-Id: <3D17FEF6-8CC5-4BBB-B098-BEA7D28F377D@FreeBSD.org> References: <201204202137.q3KLbhNj056524@svn.freebsd.org> <20120421171128.GA6732@lor.one-eyed-alien.net> To: Brooks Davis , Ryan Stone X-Mailer: Apple Mail (2.1084) Cc: svn-src-head@freebsd.org, svn-src-all@freebsd.org, src-committers@freebsd.org Subject: Re: svn commit: r234504 - in head/sys: amd64/conf i386/conf X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 May 2012 18:04:03 -0000 On 21. Apr 2012, at 17:11 , Brooks Davis wrote: > On Sat, Apr 21, 2012 at 09:45:57AM -0400, Ryan Stone wrote: >> On Fri, Apr 20, 2012 at 5:37 PM, Brooks Davis = wrote: >>> Author: brooks >>> Date: Fri Apr 20 21:37:42 2012 >>> New Revision: 234504 >>> URL: http://svn.freebsd.org/changeset/base/234504 >>>=20 >>> Log: >>> ?Enable DTrace hooks in GENERIC. >>>=20 >>> ?Reviewed by: ?gnn >>> ?Approved by: ?core (jhb, imp) >>> ?Requested by: a cast of thousands >>> ?MFC after: ? ?3 days >>=20 >> Excellent! Thanks to everybody who helped make this happen, starting >> with the participants at dtrace.conf who gave us the requisite whacks >> with the clue-by-four. >>=20 >> However, what is our policy for enabling features in -STABLE that are >> known to be unstable? If we MFC this I don't have the slightest = worry >> that somebody might see instability in their system just because the >> hooks are all of a sudden there, but I would worry that somebody make >> take DTrace hooks being enabled in GENERIC on -STABLE to imply that >> DTrace is stable, start using it and being upset when they trip over = a >> DTrace bug. >=20 > I think we should note that it remains experimental and somewhat buggy > in the release notes and probably in the MFC message. Otherwise, it's > like any new feature. If you start using a feature in production > without extensive testing you may be surprised. I am sorry but I am close to requesting a backout now but I hope someone else might be able to debug it further, so I'll wait for another couple of days. I have heard this from multiple people and I am now at the point where I wasted almost a day or work on this over the course of two weeks. I am not able to finish HEAD snapshot release builds even after updating the underlying kernel and world to a today's HEAD (I had to disable the WITH_CTF=3D to get the kernel now booted to build). I have seen this on multiple setups for about two weeks. The common factor is always that ctfmerge is hanging forever. I have = seen this on both i386 and amd64. I have seen it with modules and with kernels built with NO_MODULES. I have seen it on NFS and on local = disks. As son as I add the # Temprary as ctfmerge hangs. nomakeoptions WITH_CTF things are fine again. At the moment it looks like this: root 34611 0.0 0.1 35908 8736 - I 4:19PM 0:00.07 ctfmerge -L = VERSION -g -o if_ath.ko.debug if_ath.o if_ath_debug.o if_ath_keycache.o = if_ath_sysctl.o if_ath_tx.o if_ath_tx_ht.o if_ath_led.o ah_osdep.o ah.o = ah_regdom root 39282 0.0 0.2 44104 14516 - I 4:20PM 0:00.06 ctfmerge -L = VERSION -g -o kernel.debug locore.o aic7xxx_reg_print.o = aic79xx_reg_print.o cam.o cam_periph.o cam_queue.o cam_sim.o cam_xpt.o = ata_all.o ata_xpt.o ata_pm root@bz1:/home/bz # procstat -k 34611 PID TID COMM TDNAME KSTACK = =20 34611 100228 ctfmerge - mi_switch = sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex = do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20 34611 100452 ctfmerge - mi_switch = sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex = do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20 34611 100453 ctfmerge - mi_switch = sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex = do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20 34611 100454 ctfmerge - mi_switch = sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex = do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20 34611 100455 ctfmerge - mi_switch = sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex = do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20 34611 100456 ctfmerge - mi_switch = sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex = do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20 root@bz1:/home/bz # procstat -k 39282 PID TID COMM TDNAME KSTACK = =20 39282 100348 ctfmerge - mi_switch = sleepq_catch_signals sleepq_wait_sig _sleep do_wait = __umtx_op_wait_uint_private amd64_syscall Xfast_syscall=20 39282 100457 ctfmerge - mi_switch = sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex = do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20 39282 100458 ctfmerge - mi_switch = sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex = do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20 39282 100459 ctfmerge - mi_switch = sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex = do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20 39282 100460 ctfmerge - mi_switch = sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex = do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20 39282 100461 ctfmerge - mi_switch = sleepq_catch_signals sleepq_wait_sig _sleep _do_lock_umutex = do_lock_umutex __umtx_op_wait_umutex amd64_syscall Xfast_syscall=20 root@bz1:/home/bz # procstat -t 34611 PID TID COMM TDNAME CPU PRI STATE WCHAN =20= 34611 100228 ctfmerge - 7 152 sleep umtxn = =20 34611 100452 ctfmerge - 1 152 sleep umtxn = =20 34611 100453 ctfmerge - 1 152 sleep umtxn = =20 34611 100454 ctfmerge - 4 152 sleep umtxn = =20 34611 100455 ctfmerge - 0 152 sleep umtxn = =20 34611 100456 ctfmerge - 5 152 sleep umtxn = =20 root@bz1:/home/bz # procstat -t 39282 PID TID COMM TDNAME CPU PRI STATE WCHAN =20= 39282 100348 ctfmerge - 7 152 sleep uwait = =20 39282 100457 ctfmerge - 2 152 sleep umtxn = =20 39282 100458 ctfmerge - 4 152 sleep umtxn = =20 39282 100459 ctfmerge - 1 152 sleep umtxn = =20 39282 100460 ctfmerge - 5 152 sleep umtxn = =20 39282 100461 ctfmerge - 0 152 sleep umtxn = =20 root@bz1:/home/bz # procstat -i 34611 | grep -v -- '---$' PID COMM SIG FLAGS 34611 ctfmerge INT --C 34611 ctfmerge QUIT --C 34611 ctfmerge TERM --C 34611 ctfmerge URG -I- 34611 ctfmerge TSTP -I- 34611 ctfmerge CHLD -I- 34611 ctfmerge TTIN -I- 34611 ctfmerge TTOU -I- 34611 ctfmerge IO -I- 34611 ctfmerge WINCH -I- 34611 ctfmerge INFO -I- 34611 ctfmerge 32 --C root@bz1:/home/bz # procstat -i 39282 | grep -v -- '---$' PID COMM SIG FLAGS 39282 ctfmerge INT --C 39282 ctfmerge QUIT --C 39282 ctfmerge TERM --C 39282 ctfmerge URG -I- 39282 ctfmerge TSTP -I- 39282 ctfmerge CHLD -I- 39282 ctfmerge TTIN -I- 39282 ctfmerge TTOU -I- 39282 ctfmerge IO -I- 39282 ctfmerge WINCH -I- 39282 ctfmerge INFO -I- 39282 ctfmerge 32 --C root@bz1:/home/bz # procstat -j 34611 | grep -v -- '--$' PID TID COMM SIG FLAGS 34611 100452 ctfmerge INT -B 34611 100452 ctfmerge QUIT -B 34611 100452 ctfmerge TERM -B 34611 100453 ctfmerge INT -B 34611 100453 ctfmerge QUIT -B 34611 100453 ctfmerge TERM -B 34611 100454 ctfmerge INT -B 34611 100454 ctfmerge QUIT -B 34611 100454 ctfmerge TERM -B 34611 100455 ctfmerge INT -B 34611 100455 ctfmerge QUIT -B 34611 100455 ctfmerge TERM -B 34611 100456 ctfmerge INT -B 34611 100456 ctfmerge QUIT -B 34611 100456 ctfmerge TERM -B root@bz1:/home/bz # procstat -j 39282 | grep -v -- '--$' PID TID COMM SIG FLAGS 39282 100457 ctfmerge INT -B 39282 100457 ctfmerge QUIT -B 39282 100457 ctfmerge TERM -B 39282 100458 ctfmerge INT -B 39282 100458 ctfmerge QUIT -B 39282 100458 ctfmerge TERM -B 39282 100459 ctfmerge INT -B 39282 100459 ctfmerge QUIT -B 39282 100459 ctfmerge TERM -B 39282 100460 ctfmerge INT -B 39282 100460 ctfmerge QUIT -B 39282 100460 ctfmerge TERM -B 39282 100461 ctfmerge INT -B 39282 100461 ctfmerge QUIT -B 39282 100461 ctfmerge TERM -B So given I have been upset I started to investigate some more. make -C /usr/src buildworld did not work either (no -j). So I started and limited the build to 1 core: cpuset -l 1 make -C /usr/src buildkernel did not work either. So added the nomakeoptions WITH_CTF and make -C /usr/src -j8 buildkernel completes. So removed the makeoptions WITH_CTF from GENRIC and the nomakeoptions = from my kernel including GENERIC and added it to the command line: make -C /usr/src -j8 buildkernel WITH_CTF=3D=20 and that doesn't seem to work at all anymore? "/usr/src/share/mk/bsd.own.mk", line 482: WITH_CTF and WITHOUT_CTF can't = both be set. So I did: echo WITH_CTF=3D >> /etc/src.conf make -C /usr/src -j8 buildkernel and of course it did not work again. Given all this and given it has reliably worked in the past, my = conclusions is that it must be something related to ctfmerge or = libraries? /usr/bin/ctfmerge: libctf.so.2 =3D> /lib/libctf.so.2 (0x800829000) libdwarf.so.3 =3D> /usr/lib/libdwarf.so.3 (0x800a36000) libelf.so.1 =3D> /usr/lib/libelf.so.1 (0x800c3f000) libz.so.6 =3D> /lib/libz.so.6 (0x800e58000) libthr.so.3 =3D> /lib/libthr.so.3 (0x80106e000) libc.so.7 =3D> /lib/libc.so.7 (0x801293000) What has changed the last 3ish months? /bz --=20 Bjoern A. Zeeb You have to have visions! It does not matter how good you are. It matters what good you do!