Date: Thu, 3 Apr 2014 20:30:44 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Bryan Drewery <bdrewery@FreeBSD.org> Cc: freebsd-fs@FreeBSD.org Subject: Re: Poudriere: rm -rf: Directory not empty Message-ID: <20140403173044.GY21331@kib.kiev.ua> In-Reply-To: <dc09beae8394e6f23567846bab277ac3@shatow.net> References: <dc09beae8394e6f23567846bab277ac3@shatow.net>
next in thread | previous in thread | raw e-mail | index | archive | help
--qnuS/wU1MXEWeKjo Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Apr 03, 2014 at 09:35:27AM -0500, Bryan Drewery wrote: > Hi, >=20 > While using Poudriere to build packages on segregated tmpfs jails > we sometimes get the following errors: >=20 > =3D=3D=3D=3D>> [08] Starting build of devel/qt4-qt3support > =3D=3D=3D=3D>> [08] Starting build of graphics/qt4-opengl > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5/include/Qt:= =20 > Directory not empty > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5/include:=20 > Directory not empty > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5:=20 > Directory not empty > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts/devel/qt4-qt3support/work:=20 > Directory not empty > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts/devel/qt4-qt3support:=20 > Directory not empty > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts/devel:=20 > Directory not empty > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts:=20 > Directory not empty > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr:=20 > Directory not empty > =3D=3D=3D=3D>> [08] Starting build of math/py-numpy > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5/include/Qt:= =20 > Directory not empty > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5/include:=20 > Directory not empty > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts/devel/qt4-qt3support/work/qt-everywhere-opensource-src-4.8.5:=20 > Directory not empty > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts/devel/qt4-qt3support/work:=20 > Directory not empty > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts/devel/qt4-qt3support:=20 > Directory not empty > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts/devel:=20 > Directory not empty > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts:=20 > Directory not empty > rm:=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr:=20 > Directory not empty >=20 > What is happening here is that the devel/qt4-qt3support finishes, > fails to cleanup itself, then the next build tries to cleanup the > previous tempdir and fails. The next build then fails, and so on. > Eventually crashing the whole build. >=20 > This is the result of just "rm -rf=20 > /usr/local/poudriere/data/build/92amd64-default/ref/../08/wrkdirs/usr/por= ts/devel/qt4-qt3support/work". >=20 > devel/qt4-qt3support runs rm -rf, fails. kill -9 -1 is ran in jail. > graphics/qt4-opengl starts, runs jail -r [kills processes], tries rm=20 > -rf, fails > math/py-numpy starts, runs jail -r [kills processes], tries rm -rf,=20 > fails >=20 > Another example is at the bottom of=20 > http://beefy1.isc.freebsd.org/bulk/83i386-default/2014-02-12_03h42m23s/lo= gs/eclipse-3.7.1_4.log > The eclipse one involved a process crashing and a coredump as well. > I thought perhaps there was a race between writing core and > removing the directory, but I found no evidence of that either > by code inspection or testing. >=20 > As shown above, no processes should be running in the jail at this > point. Poudriere itself is not touching these directories outside > of the jail either. There's no nullfs mounts of these > files to elsewhere either that may be getting touched. >=20 > What might cause this? It's very difficult to reproduce and is > reported about once every 2 months or less. Note well this is > not due to flags. A rerun of these same ports won't hit the > issue. >=20 > So far the workaround is to umount the tmpfs and remount it, but this > is not a solution as tmpfs is optional for Poudriere. From past research > it was found to not be tmpfs-specific, but my confidence level is not=20 > 100% > on that. >=20 > This has been seen on at least 9.2-R, and 10.0-R. >=20 > I can't recreate this with simple tests though on ZFS or TMPFS. >=20 > cd /tmp > ( rm -rf test; mkdir test; cat /dev/random > test/foo & sleep 1; rm=20 > -rf test; kill $! ) > ( rm -rf test; mkdir test; mkfifo test/foo; cat test/foo & sleep 1; rm= =20 > -rf test; kill $! ) > ( rm -rf test; mkdir test; cd test; rm -rf ../test ) >=20 > In the other cases it's not clear if looping on rm -rf would work or > if it would spin forever. We have not tried it since it's so difficult > to reproduce. >=20 When the situation occured and you notes it, do you still have an access to the tmpfs directory which failed rm -rf ? If yes, try to do ls -la there, and ktrace the "rm -rf". Another approach is to patch tmpfs_rmdir() in tmpfs_vnops.c and dump some information when ENOTEMPTY error is returned; e.g. you could print the directory content and tn_size. --qnuS/wU1MXEWeKjo Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJTPZrDAAoJEJDCuSvBvK1BPVQP/jehDsdsYAgdnbzjKFe+Iiaw RNW+ZPN/1jHNxxB9o2OWs9cVZH9N0TT/yLSkTVmmPSbtIaYD9DS1ele0urgy2tNz +J85tBhjzacP0utYNZlUCJ0wN3cq9hfOkLU9MVY9XNy3NhJ8xjm34I0emBXk9mDU gDJhBkPUwMFsl+KVAD4gwaIGFDX3LQLivoiUCQG0OX2hQB0ixwKGa800QJVkW4BI uU4nLkNzvfuDCgM+sUxBiJT8fkIZAQymomwoa4aaK/6t+RN8Wd2zNr8jZvfeVY+F KmZ+XnckLWLF1jLaXPvglGhpFSpQJmAXXK3V8lA+LfPvhYH3tQnh5VS2H2Tt82c1 UZVQhbBKjsbS1PjRmLxCmxFVCIZQ+PWuPVTJjYOg+f6bzJokaN/dKsJ0TIl5O0BP TorZjTLGRVtfAQ50Vgbh4uWGcgCTQlvxf8tVdiTQkrRx9Nr7MqIpjfLYQwS5ETmL RvXlXLNYqeLzzCtQPc08OYqOq8xp3WQEFcrKxocUs2iEmXNVQyuAvXF24j8aQXA6 BnvDNR/jUqVD3/Km3DCyVBj4Ls53tKBUyABMOFujiFZASXc1AbHADv1lp7cWn4my QzsBJEJb2A5bBJwC/dr+K7MtlLInV6dM35FkMnDG3TmqteKzpnJtZ4VkMz+RvBNX OTIqTSqE+ShvJDYkHaGG =C245 -----END PGP SIGNATURE----- --qnuS/wU1MXEWeKjo--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20140403173044.GY21331>