From owner-freebsd-fs@FreeBSD.ORG Thu Apr 3 14:35:30 2014 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1AA719BB for ; Thu, 3 Apr 2014 14:35:30 +0000 (UTC) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id EF4CE367 for ; Thu, 3 Apr 2014 14:35:29 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.8/8.14.8) with ESMTP id s33EZTiS044424 for ; Thu, 3 Apr 2014 14:35:29 GMT (envelope-from bdrewery@freefall.freebsd.org) Received: (from bdrewery@localhost) by freefall.freebsd.org (8.14.8/8.14.8/Submit) id s33EZTTG044423 for freebsd-fs@FreeBSD.org; Thu, 3 Apr 2014 14:35:29 GMT (envelope-from bdrewery) Received: (qmail 20902 invoked from network); 3 Apr 2014 09:35:27 -0500 Received: from unknown (HELO roundcube.xk42.net) (10.10.5.5) by sweb.xzibition.com with SMTP; 3 Apr 2014 09:35:27 -0500 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Date: Thu, 03 Apr 2014 09:35:27 -0500 From: Bryan Drewery To: freebsd-fs@FreeBSD.org Subject: Poudriere: rm -rf: Directory not empty Organization: FreeBSD Message-ID: X-Sender: bdrewery@FreeBSD.org User-Agent: Roundcube Webmail/0.9.5 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.17 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 03 Apr 2014 14:35:30 -0000 Hi, While using Poudriere to build packages on segregated tmpfs jails we sometimes get the following errors: ====>> [08] Starting build of devel/qt4-qt3support ====>> [08] Starting build of graphics/qt4-opengl rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5/include/Qt: Directory not empty rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5/include: Directory not empty rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5: Directory not empty rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work: Directory not empty rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support: Directory not empty rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel: Directory not empty rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports: Directory not empty rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr: Directory not empty ====>> [08] Starting build of math/py-numpy rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5/include/Qt: Directory not empty rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5/include: Directory not empty rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5: Directory not empty rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work: Directory not empty rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support: Directory not empty rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel: Directory not empty rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports: Directory not empty rm: /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr: Directory not empty What is happening here is that the devel/qt4-qt3support finishes, fails to cleanup itself, then the next build tries to cleanup the previous tempdir and fails. The next build then fails, and so on. Eventually crashing the whole build. This is the result of just "rm -rf /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/ports/devel/qt4-qt3support/work". devel/qt4-qt3support runs rm -rf, fails. kill -9 -1 is ran in jail. graphics/qt4-opengl starts, runs jail -r [kills processes], tries rm -rf, fails math/py-numpy starts, runs jail -r [kills processes], tries rm -rf, fails Another example is at the bottom of http://beefy1.isc.freebsd.org/bulk/83i386-default/2014-02-12_03h42m23s/logs/eclipse-3.7.1_4.log The eclipse one involved a process crashing and a coredump as well. I thought perhaps there was a race between writing core and removing the directory, but I found no evidence of that either by code inspection or testing. As shown above, no processes should be running in the jail at this point. Poudriere itself is not touching these directories outside of the jail either. There's no nullfs mounts of these files to elsewhere either that may be getting touched. What might cause this? It's very difficult to reproduce and is reported about once every 2 months or less. Note well this is not due to flags. A rerun of these same ports won't hit the issue. So far the workaround is to umount the tmpfs and remount it, but this is not a solution as tmpfs is optional for Poudriere. From past research it was found to not be tmpfs-specific, but my confidence level is not 100% on that. This has been seen on at least 9.2-R, and 10.0-R. I can't recreate this with simple tests though on ZFS or TMPFS. cd /tmp ( rm -rf test; mkdir test; cat /dev/random > test/foo & sleep 1; rm -rf test; kill $! ) ( rm -rf test; mkdir test; mkfifo test/foo; cat test/foo & sleep 1; rm -rf test; kill $! ) ( rm -rf test; mkdir test; cd test; rm -rf ../test ) In the other cases it's not clear if looping on rm -rf would work or if it would spin forever. We have not tried it since it's so difficult to reproduce. -- Regards, Bryan Drewery