Date: Sun, 31 Jan 2016 08:38:33 +0000 (UTC) From: Kubilay Kocak <koobs@FreeBSD.org> To: ports-committers@freebsd.org, svn-ports-all@freebsd.org, svn-ports-head@freebsd.org Subject: svn commit: r407589 - in head/sysutils/py-diffoscope: . files Message-ID: <201601310838.u0V8cXXZ074315@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: koobs Date: Sun Jan 31 08:38:33 2016 New Revision: 407589 URL: https://svnweb.freebsd.org/changeset/ports/407589 Log: [NEW] sysutils/py-diffoscope: In-depth comparison of files, archives, and directories diffoscope will try to get to the bottom of what makes files or directories different. It will recursively unpack archives of many kinds and transform various binary formats into more human readable form to compare them. It can compare two tarballs, ISO images, or PDF just as easily. It can be scripted through error codes, and a report can be produced with the detected differences. The report can be text or HTML. When no type of report has been selected, diffoscope defaults to write a text report on the standard output. diffoscope is developed as part of the 'reproducible builds' Deian project. It is meant to be able to quickly understand why two builds of the same package produce different outputs. diffoscope was previously named debbindiff. WWW: https://diffoscope.org/ Thank you to bapt for providing the background context, information and the getfacl patch to get this port going. Requested by: emaste, bapt (for reproducible builds & FOSDEM talk) Added: head/sysutils/py-diffoscope/ head/sysutils/py-diffoscope/Makefile (contents, props changed) head/sysutils/py-diffoscope/distinfo (contents, props changed) head/sysutils/py-diffoscope/files/ head/sysutils/py-diffoscope/files/patch-diffoscope_____init____.py (contents, props changed) head/sysutils/py-diffoscope/files/patch-diffoscope_comparators_deb.py (contents, props changed) head/sysutils/py-diffoscope/files/patch-diffoscope_comparators_directory.py (contents, props changed) head/sysutils/py-diffoscope/files/patch-diffoscope_comparators_elf.py (contents, props changed) head/sysutils/py-diffoscope/files/patch-diffoscope_comparators_utils.py (contents, props changed) head/sysutils/py-diffoscope/files/patch-tests_comparators_test__deb.py (contents, props changed) head/sysutils/py-diffoscope/files/patch-tests_comparators_test__elf.py (contents, props changed) head/sysutils/py-diffoscope/files/patch-tests_data_deb__metadata__expected__diff (contents, props changed) head/sysutils/py-diffoscope/files/patch-tests_data_elf__lib__metadata__expected__diff (contents, props changed) head/sysutils/py-diffoscope/pkg-descr (contents, props changed) head/sysutils/py-diffoscope/pkg-message (contents, props changed) Added: head/sysutils/py-diffoscope/Makefile ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sysutils/py-diffoscope/Makefile Sun Jan 31 08:38:33 2016 (r407589) @@ -0,0 +1,36 @@ +# Created by: Kubilay Kocak <koobs@FreeBSD.org> +# $FreeBSD$ + +PORTNAME= diffoscope +PORTVERSION= 48 +CATEGORIES= sysutils python +MASTER_SITES= CHEESESHOP +PKGNAMEPREFIX= ${PYTHON_PKGNAMEPREFIX} + +MAINTAINER= koobs@FreeBSD.org +COMMENT= In-depth comparison of files, archives, and directories + +LICENSE= GPLv3+ +LICENSE_FILE= ${WRKSRC}/COPYING + +RUN_DEPENDS= ${PYTHON_PKGNAMEPREFIX}libarchive-c>=0:${PORTSDIR}/archivers/py3-libarchive-c \ + ${PYTHON_PKGNAMEPREFIX}python-magic>=0:${PORTSDIR}/devel/py3-python-magic +TEST_DEPENDS= ${PYTHON_PKGNAMEPREFIX}pytest>=0:${PORTSDIR}/devel/py-pytest \ + zipinfo:${PORTSDIR}/archivers/unzip \ + isoinfo:${PORTSDIR}/sysutils/cdrkit \ + rpm2cpio:${PORTSDIR}/archivers/rpm2cpio + +USES= python:3.4+ +USE_PYTHON= autoplist distutils + +NO_ARCH= yes + +post-extract: + @${RM} -rf \ + ${WRKSRC}/tests/__pycache__ \ + ${WRKSRC}/tests/comparators/__pycache__ + +do-test: + @cd ${WRKSRC} && ${PYTHON_CMD} ${PYDISTUTILS_SETUP} test --pytest-args="-rsx -l -vv" + +.include <bsd.port.mk> Added: head/sysutils/py-diffoscope/distinfo ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sysutils/py-diffoscope/distinfo Sun Jan 31 08:38:33 2016 (r407589) @@ -0,0 +1,2 @@ +SHA256 (diffoscope-48.tar.gz) = 3f60db78501f10739f96663542d53b4586c4bee86d3ee0569a3fec97ac9dc287 +SIZE (diffoscope-48.tar.gz) = 357441 Added: head/sysutils/py-diffoscope/files/patch-diffoscope_____init____.py ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sysutils/py-diffoscope/files/patch-diffoscope_____init____.py Sun Jan 31 08:38:33 2016 (r407589) @@ -0,0 +1,24 @@ +commit 8df464ebf0db8b04d0baae6a14504f3796e355a0 +Author: Jeremy Bobbio <lunar@debian.org> +Date: Sat Jan 30 13:47:47 2016 +0000 + + Also use libarchive to read metadata from ar archives + + The output is more precise and less dependent on binutils version. + + The command line `ar` tool is not used any more so remove it from the + required tools. + +--- diffoscope/__init__.py.orig 2016-01-31 06:30:01 UTC ++++ diffoscope/__init__.py +@@ -39,9 +39,7 @@ OS_NAMES = { 'arch': 'Arch Linux' + } + + class RequiredToolNotFound(Exception): +- PROVIDERS = { 'ar': { 'debian': 'binutils-multiarch', +- 'arch': 'binutils'} +- , 'bzip2': { 'debian': 'bzip2', ++ PROVIDERS = { 'bzip2': { 'debian': 'bzip2', + 'arch': 'bzip2'} + , 'cbfstool': {} + , 'cmp': { 'debian': 'diffutils', Added: head/sysutils/py-diffoscope/files/patch-diffoscope_comparators_deb.py ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sysutils/py-diffoscope/files/patch-diffoscope_comparators_deb.py Sun Jan 31 08:38:33 2016 (r407589) @@ -0,0 +1,36 @@ +commit 8df464ebf0db8b04d0baae6a14504f3796e355a0 +Author: Jeremy Bobbio <lunar@debian.org> +Date: Sat Jan 30 13:47:47 2016 +0000 + + Also use libarchive to read metadata from ar archives + + The output is more precise and less dependent on binutils version. + + The command line `ar` tool is not used any more so remove it from the + required tools. + +--- diffoscope/comparators/deb.py.orig 2016-01-31 06:31:13 UTC ++++ diffoscope/comparators/deb.py +@@ -29,7 +29,7 @@ import diffoscope.comparators + from diffoscope.comparators.binary import File + from diffoscope.comparators.libarchive import LibarchiveContainer, list_libarchive + from diffoscope.comparators.utils import \ +- Archive, ArchiveMember, get_ar_content ++ Archive, ArchiveMember + from diffoscope.comparators.tar import TarContainer + + +@@ -86,10 +86,9 @@ class DebFile(File): + return self._control + + def compare_details(self, other, source=None): +- my_content = get_ar_content(self.path) +- other_content = get_ar_content(other.path) +- return [Difference.from_text(my_content, other_content, self.path, other.path, source="metadata")] +- ++ return [Difference.from_text_readers(list_libarchive(self.path), ++ list_libarchive(other.path), ++ self.path, other.path, source="file list")] + + class Md5sumsFile(File): + @staticmethod Added: head/sysutils/py-diffoscope/files/patch-diffoscope_comparators_directory.py ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sysutils/py-diffoscope/files/patch-diffoscope_comparators_directory.py Sun Jan 31 08:38:33 2016 (r407589) @@ -0,0 +1,47 @@ +--- diffoscope/comparators/directory.py.orig 2016-01-30 11:54:52 UTC ++++ diffoscope/comparators/directory.py +@@ -72,8 +72,11 @@ def lsattr(path): + class Getfacl(Command): + @tool_required('getfacl') + def cmdline(self): +- return ['getfacl', '-p', '-c', self.path] +- ++ osname = os.uname()[0] ++ if osname == 'Linux': ++ return ['getfacl', '-p', '-c', self.path] ++ else: # FreeBSD ++ return ['getfacl', '-q', '-h', self.path] + + def compare_meta(path1, path2): + logger.debug('compare_meta(%s, %s)', path1, path2) +@@ -84,17 +87,19 @@ def compare_meta(path1, path2): + logger.warn("'stat' not found! Is PATH wrong?") + if os.path.islink(path1) or os.path.islink(path2): + return [d for d in differences if d is not None] +- try: +- lsattr1 = lsattr(path1) +- lsattr2 = lsattr(path2) +- differences.append(Difference.from_text( +- lsattr1, lsattr2, path1, path2, source="lattr")) +- except RequiredToolNotFound: +- logger.info("Unable to find 'lsattr'.") +- try: +- differences.append(Difference.from_command(Getfacl, path1, path2)) +- except RequiredToolNotFound: +- logger.info("Unable to find 'getfacl'.") ++ osname = os.uname()[0] ++ if osname == "Linux" or osname == "FreeBSD": ++ try: ++ lsattr1 = lsattr(path1) ++ lsattr2 = lsattr(path2) ++ differences.append(Difference.from_text( ++ lsattr1, lsattr2, path1, path2, source="lattr")) ++ except RequiredToolNotFound: ++ logger.info("Unable to find 'lsattr'.") ++ try: ++ differences.append(Difference.from_command(Getfacl, path1, path2)) ++ except RequiredToolNotFound: ++ logger.info("Unable to find 'getfacl'.") + return [d for d in differences if d is not None] + + Added: head/sysutils/py-diffoscope/files/patch-diffoscope_comparators_elf.py ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sysutils/py-diffoscope/files/patch-diffoscope_comparators_elf.py Sun Jan 31 08:38:33 2016 (r407589) @@ -0,0 +1,38 @@ +commit 8df464ebf0db8b04d0baae6a14504f3796e355a0 +Author: Jeremy Bobbio <lunar@debian.org> +Date: Sat Jan 30 13:47:47 2016 +0000 + + Also use libarchive to read metadata from ar archives + + The output is more precise and less dependent on binutils version. + + The command line `ar` tool is not used any more so remove it from the + required tools. + +--- diffoscope/comparators/elf.py.orig 2016-01-31 06:32:02 UTC ++++ diffoscope/comparators/elf.py +@@ -24,8 +24,9 @@ import subprocess + from diffoscope import tool_required, OutputParsingError + from diffoscope import logger + from diffoscope.comparators.binary import File ++from diffoscope.comparators.libarchive import list_libarchive + from diffoscope.comparators.deb import DebFile, get_build_id_map +-from diffoscope.comparators.utils import get_ar_content, Command, Container ++from diffoscope.comparators.utils import Command, Container + from diffoscope.difference import Difference + + +@@ -415,10 +416,8 @@ class StaticLibFile(File): + + def compare_details(self, other, source=None): + differences = [] +- # look up differences in metadata +- content1 = get_ar_content(self.path) +- content2 = get_ar_content(other.path) +- differences.append(Difference.from_text( +- content1, content2, self.path, other.path, source="metadata")) ++ differences.append(Difference.from_text_readers(list_libarchive(self.path), ++ list_libarchive(other.path), ++ self.path, other.path, source="file list")) + differences.extend(_compare_elf_data(self.path, other.path)) + return differences Added: head/sysutils/py-diffoscope/files/patch-diffoscope_comparators_utils.py ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sysutils/py-diffoscope/files/patch-diffoscope_comparators_utils.py Sun Jan 31 08:38:33 2016 (r407589) @@ -0,0 +1,28 @@ +commit 8df464ebf0db8b04d0baae6a14504f3796e355a0 +Author: Jeremy Bobbio <lunar@debian.org> +Date: Sat Jan 30 13:47:47 2016 +0000 + + Also use libarchive to read metadata from ar archives + + The output is more precise and less dependent on binutils version. + + The command line `ar` tool is not used any more so remove it from the + required tools. + +--- diffoscope/comparators/utils.py.orig 2016-01-31 06:33:12 UTC ++++ diffoscope/comparators/utils.py +@@ -36,14 +36,6 @@ from diffoscope.difference import Differ + from diffoscope import logger, tool_required, get_temporary_directory + + +-@tool_required('ar') +-def get_ar_content(path): +- if path == '/dev/null': +- return '' +- return subprocess.check_output( +- ['ar', 'tv', path], stderr=subprocess.STDOUT, shell=False).decode('utf-8') +- +- + class Command(object, metaclass=ABCMeta): + def __init__(self, path): + self._path = path Added: head/sysutils/py-diffoscope/files/patch-tests_comparators_test__deb.py ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sysutils/py-diffoscope/files/patch-tests_comparators_test__deb.py Sun Jan 31 08:38:33 2016 (r407589) @@ -0,0 +1,67 @@ +commit 8df464ebf0db8b04d0baae6a14504f3796e355a0 +Author: Jeremy Bobbio <lunar@debian.org> +Date: Sat Jan 30 13:47:47 2016 +0000 + + Also use libarchive to read metadata from ar archives + + The output is more precise and less dependent on binutils version. + + The command line `ar` tool is not used any more so remove it from the + required tools. + +--- tests/comparators/test_deb.py.orig 2016-01-31 06:34:26 UTC ++++ tests/comparators/test_deb.py +@@ -48,12 +48,10 @@ def test_no_differences(deb1): + def differences(deb1, deb2): + return deb1.compare(deb2).details + +-@pytest.mark.skipif(tool_missing('ar'), reason='missing ar') + def test_metadata(differences): + expected_diff = open(os.path.join(os.path.dirname(__file__), '../data/deb_metadata_expected_diff')).read() + assert differences[0].unified_diff == expected_diff + +-@pytest.mark.skipif(tool_missing('ar'), reason='missing ar') + def test_compressed_files(differences): + assert differences[1].source1 == 'control.tar.gz' + assert differences[2].source1 == 'data.tar.gz' +@@ -64,7 +62,6 @@ def test_identification_of_md5sums_outsi + f = specialize(FilesystemFile(path)) + assert type(f) is FilesystemFile + +-@pytest.mark.skipif(tool_missing('ar'), reason='missing ar') + def test_identification_of_md5sums_in_deb(deb1, deb2, monkeypatch): + orig_func = Md5sumsFile.recognizes + @staticmethod +@@ -78,16 +75,13 @@ def test_identification_of_md5sums_in_de + deb1.compare(deb2) + assert test_identification_of_md5sums_in_deb.found + +-@pytest.mark.skipif(tool_missing('ar'), reason='missing ar') + def test_md5sums(differences): + assert differences[1].details[0].details[1].comment == 'Files in package differs' + +-@pytest.mark.skipif(tool_missing('ar'), reason='missing ar') + def test_identical_files_in_md5sums(deb1, deb2): + for name in ['./usr/share/doc/test/README.Debian', './usr/share/doc/test/copyright']: + assert deb1.md5sums[name] == deb2.md5sums[name] + +-@pytest.mark.skipif(tool_missing('ar'), reason='missing ar') + def test_identification_of_data_tar(deb1, deb2, monkeypatch): + orig_func = DebDataTarFile.recognizes + @staticmethod +@@ -101,7 +95,6 @@ def test_identification_of_data_tar(deb1 + deb1.compare(deb2) + assert test_identification_of_data_tar.found + +-@pytest.mark.skipif(tool_missing('ar'), reason='missing ar') + def test_skip_comparison_of_known_identical_files(deb1, deb2, monkeypatch): + compared = set() + orig_func = diffoscope.comparators.compare_files +@@ -112,7 +105,6 @@ def test_skip_comparison_of_known_identi + deb1.compare(deb2) + assert './usr/share/doc/test/README.Debian' not in compared + +-@pytest.mark.skipif(tool_missing('ar'), reason='missing ar') + def test_compare_non_existing(monkeypatch, deb1): + monkeypatch.setattr(Config.general, 'new_file', True) + difference = deb1.compare(NonExistingFile('/nonexisting', deb1)) Added: head/sysutils/py-diffoscope/files/patch-tests_comparators_test__elf.py ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sysutils/py-diffoscope/files/patch-tests_comparators_test__elf.py Sun Jan 31 08:38:33 2016 (r407589) @@ -0,0 +1,22 @@ +commit 8df464ebf0db8b04d0baae6a14504f3796e355a0 +Author: Jeremy Bobbio <lunar@debian.org> +Date: Sat Jan 30 13:47:47 2016 +0000 + + Also use libarchive to read metadata from ar archives + + The output is more precise and less dependent on binutils version. + + The command line `ar` tool is not used any more so remove it from the + required tools. + +--- tests/comparators/test_elf.py.orig 2016-01-31 06:35:23 UTC ++++ tests/comparators/test_elf.py +@@ -92,7 +92,7 @@ def lib_differences(lib1, lib2): + @pytest.mark.skipif(tool_missing('readelf') or tool_missing('objdump'), reason='missing readelf or objdump') + def test_lib_differences(lib_differences): + assert len(lib_differences) == 2 +- assert lib_differences[0].source1 == 'metadata' ++ assert lib_differences[0].source1 == 'file list' + expected_metadata_diff = open(os.path.join(os.path.dirname(__file__), '../data/elf_lib_metadata_expected_diff')).read() + assert lib_differences[0].unified_diff == expected_metadata_diff + assert 'objdump' in lib_differences[1].source1 Added: head/sysutils/py-diffoscope/files/patch-tests_data_deb__metadata__expected__diff ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sysutils/py-diffoscope/files/patch-tests_data_deb__metadata__expected__diff Sun Jan 31 08:38:33 2016 (r407589) @@ -0,0 +1,26 @@ +commit 8df464ebf0db8b04d0baae6a14504f3796e355a0 +Author: Jeremy Bobbio <lunar@debian.org> +Date: Sat Jan 30 13:47:47 2016 +0000 + + Also use libarchive to read metadata from ar archives + + The output is more precise and less dependent on binutils version. + + The command line `ar` tool is not used any more so remove it from the + required tools. + +--- tests/data/deb_metadata_expected_diff.orig 2016-01-31 06:42:09 UTC ++++ tests/data/deb_metadata_expected_diff +@@ -1,6 +1,7 @@ + @@ -1,3 +1,3 @@ +- rw-r--r-- 0/0 4 Jun 24 17:40 2015 debian-binary +--rw-r--r-- 0/0 444 Jun 24 17:40 2015 control.tar.gz +--rw-r--r-- 0/0 1626 Jun 24 17:40 2015 data.tar.gz +-+rw-r--r-- 0/0 442 Jun 24 17:40 2015 control.tar.gz +-+rw-r--r-- 0/0 1754 Jun 24 17:40 2015 data.tar.gz ++--rw-r--r-- 0 0 0 4 2015-06-24 17:40:03.000000 debian-binary ++--rw-r--r-- 0 0 0 444 2015-06-24 17:40:03.000000 control.tar.gz ++--rw-r--r-- 0 0 0 1626 2015-06-24 17:40:03.000000 data.tar.gz +++-rw-r--r-- 0 0 0 4 2015-06-24 17:40:26.000000 debian-binary +++-rw-r--r-- 0 0 0 442 2015-06-24 17:40:26.000000 control.tar.gz +++-rw-r--r-- 0 0 0 1754 2015-06-24 17:40:26.000000 data.tar.gz Added: head/sysutils/py-diffoscope/files/patch-tests_data_elf__lib__metadata__expected__diff ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sysutils/py-diffoscope/files/patch-tests_data_elf__lib__metadata__expected__diff Sun Jan 31 08:38:33 2016 (r407589) @@ -0,0 +1,22 @@ +commit 8df464ebf0db8b04d0baae6a14504f3796e355a0 +Author: Jeremy Bobbio <lunar@debian.org> +Date: Sat Jan 30 13:47:47 2016 +0000 + + Also use libarchive to read metadata from ar archives + + The output is more precise and less dependent on binutils version. + + The command line `ar` tool is not used any more so remove it from the + required tools. + +--- tests/data/elf_lib_metadata_expected_diff.orig 2016-01-31 06:43:41 UTC ++++ tests/data/elf_lib_metadata_expected_diff +@@ -1,3 +1,5 @@ +-@@ -1 +1 @@ +--rw-r--r-- 1000/1000 1216 Jun 24 12:13 2015 test.o +-+rw-r--r-- 1000/1000 1216 Jun 24 12:14 2015 test.o ++@@ -1,2 +1,2 @@ ++----------- 0 0 0 10 2015-06-24 12:14:19.000000 / ++--rw-r--r-- 0 1000 1000 1216 2015-06-24 12:13:47.000000 test.o +++---------- 0 0 0 10 2015-06-24 12:14:29.000000 / +++-rw-r--r-- 0 1000 1000 1216 2015-06-24 12:14:27.000000 test.o Added: head/sysutils/py-diffoscope/pkg-descr ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sysutils/py-diffoscope/pkg-descr Sun Jan 31 08:38:33 2016 (r407589) @@ -0,0 +1,17 @@ +diffoscope will try to get to the bottom of what makes files or +directories different. It will recursively unpack archives of many kinds +and transform various binary formats into more human readable form to +compare them. It can compare two tarballs, ISO images, or PDF just as +easily. + +It can be scripted through error codes, and a report can be produced +with the detected differences. The report can be text or HTML. When no +type of report has been selected, diffoscope defaults to write a text +report on the standard output. + +diffoscope is developed as part of the 'reproducible builds' Deian +project. It is meant to be able to quickly understand why two builds +of the same package produce different outputs. diffoscope was previously +named debbindiff. + +WWW: https://diffoscope.org/ Added: head/sysutils/py-diffoscope/pkg-message ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/sysutils/py-diffoscope/pkg-message Sun Jan 31 08:38:33 2016 (r407589) @@ -0,0 +1,12 @@ +=========================================================================== + + diffoscope currently relies on /dev/fd. To enable this feature on your + system, run: + + # mount -t fdescfs fdesc /dev/fd + + To make it permanent, add the to /etc/fstab: + + fdesc /dev/fd fdescfs rw 0 0 + +===========================================================================
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201601310838.u0V8cXXZ074315>