Skip site navigation (1)Skip section navigation (2)
Date:      Wed, 27 Sep 2017 16:30:12 +0000
From:      Glen Barber <gjb@FreeBSD.org>
To:        freebsd-pkgbase@freebsd.org
Subject:   Problems with packaging /etc and etcupdate(8)
Message-ID:  <20170927163012.GA99112@FreeBSD.org>

next in thread | raw e-mail | index | archive | help

--DKU6Jbt7q3WqK7+M
Content-Type: multipart/mixed; boundary="Nq2Wo0NMKNjxTN9z"
Content-Disposition: inline


--Nq2Wo0NMKNjxTN9z
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline

Hi,

Over the past two weeks, I spent quite a bit of time looking into using
etcupdate(8) to bootstrap /var/db/etcupdate for use with pkg(8), and
here are some of the issues I have run into.  The most current diff
tested is attached.

1) A new 'create-packages-files' target was added, which invokes the
   etcupdate(8) 'extract' command using WSTAGEDIR for the DESTDIR.
   Despite passing 'TAGS=package=etc' and 'PACKAGE=etc' to etcupdate(8)
   '-M' flag, in addition to '-DNO_ROOT', some files still fail to be
   tagged in WSTAGEDIR/var/db/etcupdate/current/METALOG.  To work around
   this, a hack was added to release/scripts/mtree-to-plist.awk to trap
   and tag files when '-v etcupdate=yes' is passed to awk(1) in this
   target.

2) Some files do not automatically get created by etcupdate(8), as noted
   in the manual page.  For example, pwd_mkdb(8), cap_mkdb(1),
   newaliases(1), tzsetup(8), and services_mkdb(8) are invoked after
   merging files with the newly-created tree.  This causes, for example,
   files like /etc/passwd to be nonexistent when creating a package.  To
   work around this, another hack was added to the mtree-to-plist.awk
   script to skip nonexistent files in WSTAGEDIR/var/db/etcupdate.  This
   is still non-ideal, I think.

3) Setting 'prefix = /var/db/etcupdate' in the release/packages/etc.ucl
   file appears to be ignored.  Files in the 'etc' package conflict with
   files installed by other packages, such as runtime, autofs, and so
   on, with conflicting files being things like /etc/autofs/include_ldap
   or other similarly-installed files within /etc.  To work around this
   when packaging files, a hack was added to the mtree-to-plist.awk
   script (again) to prepend '/var/db/etcupdate' to the file path in
   order to get the correct files included in the package.  (This is
   somewhat in parallel with (2) above.)

4) When running etcupdate(8) after hacking around some of the issues
   above, I discovered that /var/db/etcupdate/current in the root of the
   system in which it is being run disappears.  Additionally, I had to
   restore files within /etc (such as master.passwd(5), passwd(5), etc.)
   in order to make my test system usable again.  In addition to
   replacing files that should have been merged, 'pkg check' against the
   'FreeBSD-etc' package reports all of the files from the 'current'
   directory have vanished.

In addition to the above items, I also did some (albeit minimal) testing
with mergemaster(8), as it creates /var/db/mergemaster.mtree, however in
my brief testing it seems the mtree(8) file created is not compatible
with pkg(8) or the mtree-to-plist.awk script.  (Note: the mergemaster(8)
bits from my testing are not included in the diff, as I had to revert
a number of changes to ensure the correct diff was included in this
email.

I am at the point now where I am out of ideas on how to proceed, and
could use some help on figuring this out.  Any help on the above issues,
or anything in the Wiki would be greatly appreciated.

https://wiki.freebsd.org/PkgBase

Thanks,

Glen


--Nq2Wo0NMKNjxTN9z
Content-Type: text/plain; charset=us-ascii
Content-Disposition: attachment; filename="etc.txt"
Content-Transfer-Encoding: quoted-printable

Index: Makefile
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- Makefile	(revision 324062)
+++ Makefile	(working copy)
@@ -132,6 +132,7 @@
 	builddtb xdev xdev-build xdev-install \
 	xdev-links native-xtools stageworld stagekernel stage-packages \
 	create-packages-world create-packages-kernel create-packages \
+	create-files-packages \
 	packages installconfig real-packages sign-packages package-pkg \
 	print-dir test-system-compiler
=20
Index: Makefile.inc1
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- Makefile.inc1	(revision 324063)
+++ Makefile.inc1	(working copy)
@@ -1555,6 +1555,7 @@
=20
 .ORDER:		stage-packages create-packages
 .ORDER:		create-packages create-world-packages
+.ORDER:		create-packages create-files-packages
 .ORDER:		create-packages create-kernel-packages
 .ORDER:		create-packages sign-packages
=20
@@ -1596,6 +1597,12 @@
 			DESTDIR=3D${WSTAGEDIR} \
 			PKG_VERSION=3D${PKG_VERSION} create-world-packages
=20
+create-packages-files:	_pkgbootstrap _repodir .PHONY
+	${_+_}@cd ${.CURDIR}; \
+		${MAKE} -f Makefile.inc1 \
+			DESTDIR=3D${WSTAGEDIR} \
+			PKG_VERSION=3D${PKG_VERSION} create-files-packages
+
 create-packages-kernel:	_pkgbootstrap _repodir .PHONY
 	${_+_}@cd ${.CURDIR}; \
 		${MAKE} -f Makefile.inc1 \
@@ -1603,8 +1610,25 @@
 			PKG_VERSION=3D${PKG_VERSION} DISTDIR=3Dkernel \
 			create-kernel-packages
=20
-create-packages: .PHONY create-packages-world create-packages-kernel
+create-packages: .PHONY create-packages-world create-packages-files create=
-packages-kernel
=20
+create-files-packages:	_pkgbootstrap .PHONY
+	mkdir -p ${WSTAGEDIR}/var/db/etcupdate/current
+	cd ${WSTAGEDIR} ; \
+	${WSTAGEDIR}/usr/sbin/etcupdate extract -B -M \
+		"TARGET=3D${TARGET} TARGET_ARCH=3D${TARGET_ARCH} -DNO_ROOT TAGS=3Dpackag=
e=3Detc PACKAGE=3Detc" \
+		-s ${SRCDIR} -d ${WSTAGEDIR}/var/db/etcupdate
+	awk -f ${SRCDIR}/release/scripts/mtree-to-plist.awk \
+		-v etcupdate=3Dyes -v ws=3D${WSTAGEDIR} \
+		${WSTAGEDIR}/var/db/etcupdate/current/METALOG
+	sh ${SRCDIR}/release/packages/generate-ucl.sh -o etc \
+		-s ${SRCDIR} -u ${WSTAGEDIR}/etc.ucl
+	${PKG_CMD} -o ABI_FILE=3D${WSTAGEDIR}/bin/sh -o ALLOW_BASE_SHLIBS=3Dyes \
+		create -M ${WSTAGEDIR}/etc.ucl \
+		-p ${WSTAGEDIR}/etc.plist \
+		-r ${WSTAGEDIR} \
+		-o ${REPODIR}/$$(${PKG_CMD} -o ABI_FILE=3D${WSTAGEDIR}/bin/sh config ABI=
)/${PKG_VERSION}
+
 create-world-packages:	_pkgbootstrap .PHONY
 	@rm -f ${WSTAGEDIR}/*.plist 2>/dev/null || :
 	@cd ${WSTAGEDIR} ; \
Index: release/packages/etc.ucl
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- release/packages/etc.ucl	(nonexistent)
+++ release/packages/etc.ucl	(working copy)
@@ -0,0 +1,22 @@
+#
+# $FreeBSD: head/release/packages/runtime.ucl 322327 2017-08-09 19:16:54Z =
gjb $
+#
+
+name =3D "FreeBSD-etc"
+origin =3D "base"
+version =3D "%VERSION%"
+comment =3D "etc package"
+categories =3D [ base ]
+maintainer =3D "re@FreeBSD.org"
+www =3D "https://www.FreeBSD.org"
+prefix =3D "/var/db/etcupdate"
+licenselogic =3D "single"
+licenses =3D [ BSD2CLAUSE ]
+desc =3D <<EOD
+etc package
+EOD
+scripts: {
+	post-install =3D <<EOD
+	echo "Please use etcupdate(8) or mergemaster(8) to update"
+	echo "files under /etc."
+EOD
Index: release/scripts/mtree-to-plist.awk
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
--- release/scripts/mtree-to-plist.awk	(revision 324062)
+++ release/scripts/mtree-to-plist.awk	(working copy)
@@ -28,6 +28,13 @@
 			tags=3Dtags""_kernconf
 		}
 	}
+	if (etcupdate !=3D "") {
+		if (length(gsub(/tags=3D/, "", tags) =3D=3D 0))
+			tags=3D"package=3Detc"
+		if ((length(type) !=3D 0) && (type =3D=3D "dir")) {
+			next
+		}
+	}
 	if (length(tags) =3D=3D 0)
 		next
 	if (tags ~ /package=3D/) {
@@ -76,5 +83,17 @@
 		output=3Dpkgname".plist"
 	}
=20
+	if (etcupdate !=3D "") {
+		pfx=3D"/var/db/etcupdate/current"
+		filename=3Dws""pfx""$1
+		if (system("test -e " filename) =3D=3D 0) {
+			print "@"type"("uname","gname","mode","flags") " pfx""$1 > output
+			next
+		} else {
+				next
+		}
+		next
+	}
+
 	print "@"type"("uname","gname","mode","flags") " $1 > output
 }

--Nq2Wo0NMKNjxTN9z--

--DKU6Jbt7q3WqK7+M
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEEVVuz/A7vpH93hEhKuWzd6q+LXtAFAlnL0hQACgkQuWzd6q+L
XtBAYA/8Dai/lJ7VM+7uks8oULtpSz3IZFfkApxBXGYFvS+Ptpryjjk681x/s+23
vGf7ua1x+E8U0t68YZCWNyc1fBtj9aoQmnTA66Dm8SJu/i1c4jzbZmRP3dl31RFU
NUkCrewIfTkhMoJbvFyv5tdFVcIpSDrwtAKVTDCJ4UQJXfSFcOtGfAToPB9x1EOy
oYHOOh8TxoSGPE3sZv4mV87nJRKoIzlw2ueRKFB6IghKHvWpBb0ROK37g5HhPS4g
F8kvVMmhYBPnFF+wZtI99ZH3Zv4ekGj0V9gNi2WyV3XYNkRAsSmCJfdpTuBYpcux
s0HpCOEDH8/wa3rKuLq93/2v/dQ6yX6i0ZwmkYR+zbZQJaievs1q0PHq9zVctxp/
wSiljzSixGNmFZe+S/Gi4juBLTzK+dpl4qHVJZJKArla5A2j1xfAimo3WrcE0W/C
OBVMEiRWk43BNWgsBy9Cz2N7od+6ffRKAmg8mFZ5N5hkK7omH8E4vOWlY5PjFwwO
iXNtwji16b0bbDSKqS2I0nXF7ouWa/MLXoMsuJzvM88TovVRQ95Etgx+wD7s5Y3t
k46ib7l/8ZK/nXJtxgI4PfPOfBcHpzJxskkS6j9af+KIQ7OUbQi5RjowuEWtSuuT
YftNdz3eFr5lxA3cFKlU4RnqnMoJ3tvKfgSp2BbgMcjDy0JmYbo=
=E4QT
-----END PGP SIGNATURE-----

--DKU6Jbt7q3WqK7+M--



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20170927163012.GA99112>