Date: Fri, 17 Feb 2017 23:23:47 +0000 (UTC) From: Bryan Drewery <bdrewery@FreeBSD.org> To: ports-committers@freebsd.org, svn-ports-all@freebsd.org, svn-ports-head@freebsd.org Subject: svn commit: r434319 - in head/devel: . ccache ccache-memcached ccache/files Message-ID: <201702172323.v1HNNlRO070024@repo.freebsd.org>
next in thread | raw e-mail | index | archive | help
Author: bdrewery Date: Fri Feb 17 23:23:47 2017 New Revision: 434319 URL: https://svnweb.freebsd.org/changeset/ports/434319 Log: Add a patch for memcached to ccache along with a slave devel/ccache-memcached port. This patch is not safe for WITH_CCACHE_BUILD support yet as that causes all ports to depend on devel/ccache. Enabling that patch would then cause the new devel/libmemcached dependency to require devel/ccache which is a cyclic dependency. The autoconf dependency also causes issues. Add a devel/ccache-memcached slave port that would allow a user to use the ccache+memcached package manually with ports without WITH_CCACHE_BUILD. This patch comes from https://github.com/ccache/ccache/pull/58 and has been an ongoing effort over a few years to be merged into the mainline of ccache. Documenation for it can be found in the MANUAL file at: /usr/local/share/doc/ccache/MANUAL.txt Sponsored by: Dell EMC Isilon Added: head/devel/ccache-memcached/ head/devel/ccache-memcached/Makefile (contents, props changed) head/devel/ccache/files/extra-patch-memcached (contents, props changed) head/devel/ccache/files/patch-configure.ac (contents, props changed) Modified: head/devel/Makefile head/devel/ccache/Makefile Modified: head/devel/Makefile ============================================================================== --- head/devel/Makefile Fri Feb 17 23:12:18 2017 (r434318) +++ head/devel/Makefile Fri Feb 17 23:23:47 2017 (r434319) @@ -234,6 +234,7 @@ SUBDIR += cbrowser SUBDIR += cc65 SUBDIR += ccache + SUBDIR += ccache-memcached SUBDIR += cccc SUBDIR += ccdoc SUBDIR += ccons Added: head/devel/ccache-memcached/Makefile ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/devel/ccache-memcached/Makefile Fri Feb 17 23:23:47 2017 (r434319) @@ -0,0 +1,11 @@ +# $FreeBSD$ + +PKGNAMESUFFIX= -memcached + +MASTERDIR= ${.CURDIR}/../ccache + +OPTIONS_SLAVE= MEMCACHED + +CONFLICTS_INSTALL= ccache-[0-9]* + +.include "${MASTERDIR}/Makefile" Modified: head/devel/ccache/Makefile ============================================================================== --- head/devel/ccache/Makefile Fri Feb 17 23:12:18 2017 (r434318) +++ head/devel/ccache/Makefile Fri Feb 17 23:23:47 2017 (r434319) @@ -13,26 +13,43 @@ COMMENT= Tool to minimize the compile ti LICENSE= GPLv3 +CONFLICTS_INSTALL= ccache-memcached-[0-9]* + GNU_CONFIGURE= yes HOWTO= ccache-howto-freebsd.txt CCLINKDIR= libexec/ccache SUB_FILES= ${HOWTO} world-ccache pkg-message ccache-update-links.sh -PORTDOCS= ccache-howto-freebsd.txt MANUAL.html +PORTDOCS= ccache-howto-freebsd.txt MANUAL.html MANUAL.txt -OPTIONS_DEFINE= CLANGLINK LLVMLINK STATIC DOCS TINDERBOX +OPTIONS_DEFINE= CLANGLINK LLVMLINK STATIC DOCS TINDERBOX MEMCACHED OPTIONS_DEFAULT=CLANGLINK LLVMLINK CLANGLINK_DESC= Create clang compiler links if clang is installed LLVMLINK_DESC= Create llvm compiler links if llvm is installed TINDERBOX_DESC= Create tarball for tinderbox usage +MEMCACHED_DESC= Build in experimental Memcached support USES= compiler + +MEMCACHED_EXTRA_PATCHES= ${FILESDIR}/extra-patch-memcached:-p1 +MEMCACHED_CONFIGURE_ENABLE= memcached +MEMCACHED_USES= autoreconf pkgconfig +MEMCACHED_LIB_DEPENDS= libmemcached.so:databases/libmemcached +MEMCACHED_LDFLAGS= -L${LOCALBASE}/lib +MEMCACHED_CFLAGS= -I${LOCALBASE}/include + +.if defined(WITH_CCACHE_BUILD) && empty(OPTIONS_SLAVE:MMEMCACHED) # Don't allow autoreconf. We want no dependencies on this to keep # WITH_CCACHE_BUILD working. USES:= ${USES:Nautoreconf} +MEMCACHED_IGNORE= MEMCACHED cannot be combined with WITH_CCACHE_BUILD. Use devel/ccache-memcached +# XXX: This needs more testing with Poudriere before enabling. Also bsd.options.mk support. +#MEMCACHED_DEPENDS_ARGS+= NO_CCACHE=1 +.endif + OPTIONS_SUB= yes STATIC_LDFLAGS= -static @@ -93,6 +110,7 @@ do-install-TINDERBOX-on: do-install-DOCS-on: ${MKDIR} ${STAGEDIR}${DOCSDIR} ${INSTALL_DATA} ${WRKSRC}/MANUAL.html ${STAGEDIR}${DOCSDIR} + ${INSTALL_DATA} ${WRKSRC}/MANUAL.txt ${STAGEDIR}${DOCSDIR} ${INSTALL_DATA} ${WRKDIR}/${HOWTO} ${STAGEDIR}${DOCSDIR} .include <bsd.port.post.mk> Added: head/devel/ccache/files/extra-patch-memcached ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/devel/ccache/files/extra-patch-memcached Fri Feb 17 23:23:47 2017 (r434319) @@ -0,0 +1,2396 @@ +https://github.com/ccache/ccache/pull/58 +Retrieved on February 13th 2017. +Changes to .travis.yml removed since it is not in the release image. + +diff --git a/MANUAL.txt b/MANUAL.txt +index ab01886..c78bb6e 100644 +--- a/MANUAL.txt ++++ b/MANUAL.txt +@@ -418,6 +418,20 @@ WRAPPERS>>. + The default value is 5G. Available suffixes: k, M, G, T (decimal) and Ki, + Mi, Gi, Ti (binary). The default suffix is "G". + ++*memcached_conf* (*CCACHE_MEMCACHED_CONF*):: ++ ++ The memcached_conf option sets the memcached(3) configuration to use for ++ storing and getting cache values, if any. Example configuration: +++ ++------------------------------------------------------------------------------- ++CCACHE_MEMCACHED_CONF=--SERVER=localhost:11211 ++------------------------------------------------------------------------------- ++ ++*memcached_only* (*CCACHE_MEMCACHED_ONLY*):: ++ ++ Only store files in memcached, don't store them in the local filesystems. ++ The manifests (for direct mode) and stats are still being stored locally. ++ + *path* (*CCACHE_PATH*):: + + If set, ccache will search directories in this list when looking for the +@@ -451,6 +465,11 @@ WRAPPERS>>. + from the cache using the direct mode, not the preprocessor mode. See + documentation for *read_only* regarding using a read-only ccache directory. + ++*read_only_memcached* (*CCACHE_READONLY_MEMCACHED* or *CCACHE_NOREADONLY_MEMCACHED*), see <<_boolean_values,Boolean values>> above):: ++ ++ If true, ccache will attempt to get previously cached values from memcached, ++ but will not try to store any new values in memcached. ++ + *recache* (*CCACHE_RECACHE* or *CCACHE_NORECACHE*, see <<_boolean_values,Boolean values>> above):: + + If true, ccache will not use any previously stored result. New results will +@@ -769,6 +788,29 @@ A tip is to set *temporary_dir* to a directory on the local host to avoid NFS + traffic for temporary files. + + ++Sharing a cache with memcached ++------------------------------ ++ ++When using the *memcached* (<http://memcached.org>) feature, the most recently ++used cache entries are also available from the configured memcached servers. ++ ++The local cache directory will be searched first, but then it will still be ++possible to get cache hits (over the network) before having to run the ++compiler. ++ ++Using a local *moxi* (memcached proxy) will enable multiple ccache invocations ++to share memcached connections and thus avoid some of the network overhead. ++ ++It will also allow you to fine-tune connection timeouts and other settings. You ++can optionally replace your memcached servers with Couchbase servers. ++ ++Example: ++ ++------------------------------------------------------------------------------- ++moxi -z 11211=mc_server1:11211,mc_server2:11211 ++------------------------------------------------------------------------------- ++ ++ + Using ccache with other compiler wrappers + ----------------------------------------- + +diff --git a/Makefile.in b/Makefile.in +index 5aee02d..08b3633 100644 +--- a/Makefile.in ++++ b/Makefile.in +@@ -37,6 +37,7 @@ non_3pp_sources = \ + lockfile.c \ + manifest.c \ + mdfour.c \ ++ memccached.c \ + stats.c \ + unify.c \ + util.c \ +@@ -101,7 +102,7 @@ perf: ccache$(EXEEXT) + .PHONY: test + test: ccache$(EXEEXT) test/main$(EXEEXT) + test/main$(EXEEXT) +- CC='$(CC)' $(srcdir)/test.sh ++ CC='$(CC)' @ccache_memcached@$(srcdir)/test.sh + + .PHONY: quicktest + quicktest: test/main$(EXEEXT) +diff --git a/ccache.c b/ccache.c +index 88e0ec5..12026c7 100644 +--- a/ccache.c ++++ b/ccache.c +@@ -102,6 +102,9 @@ static char *output_dia = NULL; + // Split dwarf information (GCC 4.8 andup). Contains pathname if not NULL. + static char *output_dwo = NULL; + ++// The cached key. ++static char *cached_key; ++ + // Array for storing -arch options. + #define MAX_ARCH_ARGS 10 + static size_t arch_args_size = 0; +@@ -123,6 +126,9 @@ static char *cached_stderr; + // (cachedir/a/b/cdef[...]-size.d). + static char *cached_dep; + ++// The manifest key. ++static char *manifest_name; ++ + // Full path to the file containing the coverage information + // (cachedir/a/b/cdef[...]-size.gcno). + static char *cached_cov; +@@ -239,6 +245,18 @@ static pid_t compiler_pid = 0; + // stored in the cache changes in a backwards-incompatible way. + static const char HASH_PREFIX[] = "3"; + ++static void from_fscache(enum fromcache_call_mode mode, ++ bool put_object_in_manifest); ++static void to_fscache(struct args *args); ++#ifdef HAVE_LIBMEMCACHED ++static void from_memcached(enum fromcache_call_mode mode, ++ bool put_object_in_manifest); ++static void to_memcached(struct args *args); ++#endif ++static void (*from_cache)(enum fromcache_call_mode mode, ++ bool put_object_in_manifest); ++static void (*to_cache)(struct args *args); ++ + static void + add_prefix(struct args *args, char *prefix_command) + { +@@ -952,6 +970,28 @@ put_file_in_cache(const char *source, const char *dest) + stats_update_size(file_size(&st), 1); + } + ++#ifdef HAVE_LIBMEMCACHED ++// Copy data to the cache. ++static void ++put_data_in_cache(void *data, size_t size, const char *dest) ++{ ++ int ret; ++ ++ assert(!conf->read_only); ++ assert(!conf->read_only_direct); ++ ++ /* already compressed (in cache) */ ++ ret = write_file(data, dest, size); ++ if (ret != 0) { ++ cc_log("Failed to write to %s: %s", dest, strerror(errno)); ++ stats_update(STATS_ERROR); ++ failed(); ++ } ++ cc_log("Stored in cache: %zu bytes -> %s", size, dest); ++ stats_update_size(size, 1); ++} ++#endif ++ + // Copy or link a file from the cache. + static void + get_file_from_cache(const char *source, const char *dest) +@@ -1006,6 +1046,11 @@ send_cached_stderr(void) + // Create or update the manifest file. + void update_manifest_file(void) + { ++#ifdef HAVE_LIBMEMCACHED ++ char *data; ++ size_t size; ++#endif ++ + if (!conf->direct_mode + || !included_files + || conf->read_only +@@ -1023,6 +1068,14 @@ void update_manifest_file(void) + update_mtime(manifest_path); + if (x_stat(manifest_path, &st) == 0) { + stats_update_size(file_size(&st) - old_size, old_size == 0 ? 1 : 0); ++#if HAVE_LIBMEMCACHED ++ if (strlen(conf->memcached_conf) > 0 && !conf->read_only_memcached && ++ read_file(manifest_path, st.st_size, &data, &size)) { ++ cc_log("Storing %s in memcached", manifest_name); ++ memccached_raw_set(manifest_name, data, size); ++ free(data); ++ } ++#endif + } + } else { + cc_log("Failed to add object file hash to %s", manifest_path); +@@ -1031,8 +1084,12 @@ void update_manifest_file(void) + + // Run the real compiler and put the result in cache. + static void +-to_cache(struct args *args) ++to_fscache(struct args *args) + { ++#ifdef HAVE_LIBMEMCACHED ++ char *data_obj, *data_stderr, *data_dia, *data_dep; ++ size_t size_obj, size_stderr, size_dia, size_dep; ++#endif + char *tmp_stdout = format("%s.tmp.stdout", cached_obj); + int tmp_stdout_fd = create_tmp_fd(&tmp_stdout); + char *tmp_stderr = format("%s.tmp.stderr", cached_obj); +@@ -1288,6 +1345,40 @@ to_cache(struct args *args) + } + } + ++#ifdef HAVE_LIBMEMCACHED ++ if (strlen(conf->memcached_conf) > 0 && !conf->read_only_memcached && ++ !using_split_dwarf && /* no support for the dwo files just yet */ ++ !generating_coverage) { /* coverage refers to local paths anyway */ ++ cc_log("Storing %s in memcached", cached_key); ++ if (!read_file(cached_obj, 0, &data_obj, &size_obj)) { ++ data_obj = NULL; ++ size_obj = 0; ++ } ++ if (!read_file(cached_stderr, 0, &data_stderr, &size_stderr)) { ++ data_stderr = NULL; ++ size_stderr = 0; ++ } ++ if (!read_file(cached_dia, 0, &data_dia, &size_dia)) { ++ data_dia = NULL; ++ size_dia = 0; ++ } ++ if (!read_file(cached_dep, 0, &data_dep, &size_dep)) { ++ data_dep = NULL; ++ size_dep = 0; ++ } ++ ++ if (data_obj) { ++ memccached_set(cached_key, ++ data_obj, data_stderr, data_dia, data_dep, ++ size_obj, size_stderr, size_dia, size_dep); ++ } ++ ++ free(data_obj); ++ free(data_stderr); ++ free(data_dia); ++ free(data_dep); ++ } ++#endif + // Everything OK. + send_cached_stderr(); + update_manifest_file(); +@@ -1298,6 +1389,226 @@ to_cache(struct args *args) + free(tmp_dwo); + } + ++#ifdef HAVE_LIBMEMCACHED ++// Run the real compiler and put the result in cache. ++static void ++to_memcached(struct args *args) ++{ ++ const char *tmp_dir = temp_dir(); ++ char *tmp_stdout, *tmp_stderr; ++ char *stderr_d, *obj_d, *dia_d = NULL, *dep_d = NULL; ++ size_t stderr_l = 0, obj_l = 0, dia_l = 0, dep_l = 0; ++ struct stat st; ++ int status, tmp_stdout_fd, tmp_stderr_fd; ++ ++ tmp_stdout = format("%s/%s.tmp.stdout.%s", tmp_dir, cached_obj, tmp_string()); ++ tmp_stdout_fd = create_tmp_fd(&tmp_stdout); ++ tmp_stderr = format("%s/%s.tmp.stderr.%s", tmp_dir, cached_obj, tmp_string()); ++ tmp_stderr_fd = create_tmp_fd(&tmp_stderr); ++ ++ if (generating_coverage) { ++ cc_log("No memcached support for coverage yet"); ++ failed(); ++ } ++ if (using_split_dwarf) { ++ cc_log("No memcached support for split dwarf yet"); ++ failed(); ++ } ++ ++ if (create_parent_dirs(tmp_stdout) != 0) { ++ fatal("Failed to create parent directory for %s: %s", ++ tmp_stdout, strerror(errno)); ++ } ++ ++ args_add(args, "-o"); ++ args_add(args, output_obj); ++ ++ if (output_dia) { ++ args_add(args, "--serialize-diagnostics"); ++ args_add(args, output_dia); ++ } ++ ++ /* Turn off DEPENDENCIES_OUTPUT when running cc1, because ++ * otherwise it will emit a line like ++ * ++ * tmp.stdout.vexed.732.o: /home/mbp/.ccache/tmp.stdout.vexed.732.i ++ */ ++ x_unsetenv("DEPENDENCIES_OUTPUT"); ++ ++ if (conf->run_second_cpp) { ++ args_add(args, input_file); ++ } else { ++ args_add(args, i_tmpfile); ++ } ++ ++ cc_log("Running real compiler"); ++ status = execute(args->argv, tmp_stdout_fd, tmp_stderr_fd, &compiler_pid); ++ args_pop(args, 3); ++ ++ if (x_stat(tmp_stdout, &st) != 0) { ++ /* The stdout file was removed - cleanup in progress? Better bail out. */ ++ stats_update(STATS_MISSING); ++ tmp_unlink(tmp_stdout); ++ tmp_unlink(tmp_stderr); ++ failed(); ++ } ++ if (st.st_size != 0) { ++ cc_log("Compiler produced stdout"); ++ stats_update(STATS_STDOUT); ++ tmp_unlink(tmp_stdout); ++ tmp_unlink(tmp_stderr); ++ failed(); ++ } ++ tmp_unlink(tmp_stdout); ++ ++ /* ++ * Merge stderr from the preprocessor (if any) and stderr from the real ++ * compiler into tmp_stderr. ++ */ ++ if (cpp_stderr) { ++ int fd_cpp_stderr; ++ int fd_real_stderr; ++ int fd_result; ++ char *tmp_stderr2; ++ ++ tmp_stderr2 = format("%s.2", tmp_stderr); ++ if (x_rename(tmp_stderr, tmp_stderr2)) { ++ cc_log("Failed to rename %s to %s: %s", tmp_stderr, tmp_stderr2, ++ strerror(errno)); ++ failed(); ++ } ++ fd_cpp_stderr = open(cpp_stderr, O_RDONLY | O_BINARY); ++ if (fd_cpp_stderr == -1) { ++ cc_log("Failed opening %s: %s", cpp_stderr, strerror(errno)); ++ failed(); ++ } ++ fd_real_stderr = open(tmp_stderr2, O_RDONLY | O_BINARY); ++ if (fd_real_stderr == -1) { ++ cc_log("Failed opening %s: %s", tmp_stderr2, strerror(errno)); ++ failed(); ++ } ++ fd_result = open(tmp_stderr, O_WRONLY | O_CREAT | O_TRUNC | O_BINARY, 0666); ++ if (fd_result == -1) { ++ cc_log("Failed opening %s: %s", tmp_stderr, strerror(errno)); ++ failed(); ++ } ++ copy_fd(fd_cpp_stderr, fd_result); ++ copy_fd(fd_real_stderr, fd_result); ++ close(fd_cpp_stderr); ++ close(fd_real_stderr); ++ close(fd_result); ++ tmp_unlink(tmp_stderr2); ++ free(tmp_stderr2); ++ } ++ ++ if (status != 0) { ++ int fd; ++ cc_log("Compiler gave exit status %d", status); ++ stats_update(STATS_STATUS); ++ ++ fd = open(tmp_stderr, O_RDONLY | O_BINARY); ++ if (fd != -1) { ++ /* We can output stderr immediately instead of rerunning the compiler. */ ++ copy_fd(fd, 2); ++ close(fd); ++ tmp_unlink(tmp_stderr); ++ ++ x_exit(status); ++ } ++ ++ tmp_unlink(tmp_stderr); ++ failed(); ++ } ++ ++ if (stat(output_obj, &st) != 0) { ++ cc_log("Compiler didn't produce an object file"); ++ stats_update(STATS_NOOUTPUT); ++ failed(); ++ } ++ if (st.st_size == 0) { ++ cc_log("Compiler produced an empty object file"); ++ stats_update(STATS_EMPTYOUTPUT); ++ failed(); ++ } ++ ++ if (x_stat(tmp_stderr, &st) != 0) { ++ stats_update(STATS_ERROR); ++ failed(); ++ } ++ /* cache stderr */ ++ if (!read_file(tmp_stderr, 0, &stderr_d, &stderr_l)) { ++ stats_update(STATS_ERROR); ++ failed(); ++ } ++ tmp_unlink(tmp_stderr); ++ ++ if (output_dia) { ++ if (x_stat(output_dia, &st) != 0) { ++ stats_update(STATS_ERROR); ++ failed(); ++ } ++ /* cache dia */ ++ if (!read_file(output_dia, 0, &dia_d, &dia_l)) { ++ stats_update(STATS_ERROR); ++ failed(); ++ } ++ } ++ ++ /* cache output */ ++ if (!read_file(output_obj, 0, &obj_d, &obj_l)) { ++ stats_update(STATS_ERROR); ++ failed(); ++ } ++ ++ if (generating_dependencies) { ++ if (!read_file(output_dep, 0, &dep_d, &dep_l)) { ++ stats_update(STATS_ERROR); ++ failed(); ++ } ++ } ++ ++ if (memccached_set(cached_key, obj_d, stderr_d, dia_d, dep_d, ++ obj_l, stderr_l, dia_l, dep_l) < 0) { ++ stats_update(STATS_ERROR); ++ failed(); ++ } ++ ++ cc_log("Storing %s in memcached", cached_key); ++ ++ stats_update(STATS_TOCACHE); ++ ++ /* Make sure we have a CACHEDIR.TAG in the cache part of cache_dir. This can ++ * be done almost anywhere, but we might as well do it near the end as we ++ * save the stat call if we exit early. ++ */ ++ { ++ char *first_level_dir = dirname(stats_file); ++ if (create_cachedirtag(first_level_dir) != 0) { ++ cc_log("Failed to create %s/CACHEDIR.TAG (%s)\n", ++ first_level_dir, strerror(errno)); ++ stats_update(STATS_ERROR); ++ failed(); ++ } ++ free(first_level_dir); ++ ++ /* Remove any CACHEDIR.TAG on the cache_dir level where it was located in ++ * previous ccache versions. */ ++ if (getpid() % 1000 == 0) { ++ char *path = format("%s/CACHEDIR.TAG", conf->cache_dir); ++ x_unlink(path); ++ free(path); ++ } ++ } ++ ++ /* Everything OK. */ ++ send_cached_stderr(); ++ update_manifest_file(); ++ ++ free(tmp_stderr); ++ free(tmp_stdout); ++} ++#endif ++ + // Find the object file name by running the compiler in preprocessor mode. + // Returns the hash as a heap-allocated hex string. + static struct file_hash * +@@ -1408,6 +1719,7 @@ static void + update_cached_result_globals(struct file_hash *hash) + { + char *object_name = format_hash_as_string(hash->hash, hash->size); ++ cached_key = strdup(object_name); + cached_obj_hash = hash; + cached_obj = get_path_in_cache(object_name, ".o"); + cached_stderr = get_path_in_cache(object_name, ".stderr"); +@@ -1599,6 +1911,11 @@ calculate_common_hash(struct args *args, struct mdfour *hash) + static struct file_hash * + calculate_object_hash(struct args *args, struct mdfour *hash, int direct_mode) + { ++#if HAVE_LIBMEMCACHED ++ char *data; ++ size_t size; ++#endif ++ + if (direct_mode) { + hash_delimiter(hash, "manifest version"); + hash_int(hash, MANIFEST_VERSION); +@@ -1791,7 +2108,27 @@ calculate_object_hash(struct args *args, struct mdfour *hash, int direct_mode) + } + char *manifest_name = hash_result(hash); + manifest_path = get_path_in_cache(manifest_name, ".manifest"); +- free(manifest_name); ++ /* Check if the manifest file is there. */ ++ struct stat st; ++ if (stat(manifest_path, &st) != 0) { ++#if HAVE_LIBMEMCACHED ++ void *cache = NULL; ++#endif ++ cc_log("Manifest file %s not in cache", manifest_path); ++#if HAVE_LIBMEMCACHED ++ if (strlen(conf->memcached_conf) > 0) { ++ cc_log("Getting %s from memcached", manifest_name); ++ cache = memccached_raw_get(manifest_name, &data, &size); ++ } ++ if (cache) { ++ cc_log("Added object file hash to %s", manifest_path); ++ write_file(data, manifest_path, size); ++ stats_update_size(size, 1); ++ free(cache); ++ } else ++#endif ++ return NULL; ++ } + cc_log("Looking for object file hash in %s", manifest_path); + object_hash = manifest_get(conf, manifest_path); + if (object_hash) { +@@ -1828,8 +2165,13 @@ calculate_object_hash(struct args *args, struct mdfour *hash, int direct_mode) + // Try to return the compile result from cache. If we can return from cache + // then this function exits with the correct status code, otherwise it returns. + static void +-from_cache(enum fromcache_call_mode mode, bool put_object_in_manifest) ++from_fscache(enum fromcache_call_mode mode, bool put_object_in_manifest) + { ++#if HAVE_LIBMEMCACHED ++ char *data_obj, *data_stderr, *data_dia, *data_dep; ++ size_t size_obj, size_stderr, size_dia, size_dep; ++#endif ++ + // The user might be disabling cache hits. + if (conf->recache) { + return; +@@ -1837,7 +2179,33 @@ from_cache(enum fromcache_call_mode mode, bool put_object_in_manifest) + + struct stat st; + if (stat(cached_obj, &st) != 0) { ++#if HAVE_LIBMEMCACHED ++ void *cache = NULL; ++#endif + cc_log("Object file %s not in cache", cached_obj); ++#if HAVE_LIBMEMCACHED ++ if (strlen(conf->memcached_conf) > 0 && ++ !using_split_dwarf && ++ !generating_coverage) { ++ cc_log("Getting %s from memcached", cached_key); ++ cache = memccached_get(cached_key, ++ &data_obj, &data_stderr, &data_dia, &data_dep, ++ &size_obj, &size_stderr, &size_dia, &size_dep); ++ } ++ if (cache) { ++ put_data_in_cache(data_obj, size_obj, cached_obj); ++ if (size_stderr > 0) { ++ put_data_in_cache(data_stderr, size_stderr, cached_stderr); ++ } ++ if (size_dia > 0) { ++ put_data_in_cache(data_dia, size_dia, cached_dia); ++ } ++ if (size_dep > 0) { ++ put_data_in_cache(data_dep, size_dep, cached_dep); ++ } ++ memccached_free(cache); ++ } else ++#endif + return; + } + +@@ -1947,6 +2315,97 @@ from_cache(enum fromcache_call_mode mode, bool put_object_in_manifest) + x_exit(0); + } + ++#ifdef HAVE_LIBMEMCACHED ++/* ++ * Try to return the compile result from cache. If we can return from cache ++ * then this function exits with the correct status code, otherwise it returns. ++ */ ++static void ++from_memcached(enum fromcache_call_mode mode, bool put_object_in_manifest) ++{ ++ bool produce_dep_file = false; ++ int ret; ++ void *cache; ++ char *data_obj, *data_stderr, *data_dia, *data_dep; ++ size_t size_obj, size_stderr, size_dia, size_dep; ++ ++ /* the user might be disabling cache hits */ ++ if (conf->recache || using_split_dwarf || generating_coverage) { ++ return; ++ } ++ ++ cc_log("Getting %s from memcached", cached_key); ++ cache = memccached_get(cached_key, ++ &data_obj, &data_stderr, &data_dia, &data_dep, ++ &size_obj, &size_stderr, &size_dia, &size_dep); ++ if (!cache) { ++ return; ++ } ++ ++ /* ++ * (If mode != FROMCACHE_DIRECT_MODE, the dependency file is created by ++ * gcc.) ++ */ ++ produce_dep_file = generating_dependencies && mode == FROMCACHE_DIRECT_MODE; ++ ++ if (!str_eq(output_obj, "/dev/null")) { ++ x_unlink(output_obj); ++ ret = write_file(data_obj, output_obj, size_obj); ++ } else { ++ ret = 0; ++ } ++ if (ret < 0) { ++ cc_log("Problem creating %s from %s", output_obj, cached_key); ++ failed(); ++ } ++ ++ if (produce_dep_file) { ++ x_unlink(output_dep); ++ ret = write_file(data_dep, output_dep, size_dep); ++ if (ret < 0) { ++ cc_log("Problem creating %s from %s", output_dep, cached_key); ++ failed(); ++ } ++ } ++ if (output_dia) { ++ x_unlink(output_dia); ++ ret = write_file(data_dia, output_dia, size_dia); ++ if (ret < 0) { ++ cc_log("Problem creating %s from %s", output_dia, cached_key); ++ failed(); ++ } ++ } ++ ++ if (generating_dependencies && mode == FROMCACHE_CPP_MODE) { ++ /* Store the dependency file in the cache. */ ++ cc_log("Does not support non direct mode"); ++ } ++ ++ /* Send the stderr, if any. */ ++ safe_write(2, data_stderr, size_stderr); ++ ++ if (put_object_in_manifest) { ++ update_manifest_file(); ++ } ++ ++ /* log the cache hit */ ++ switch (mode) { ++ case FROMCACHE_DIRECT_MODE: ++ cc_log("Succeeded getting cached result"); ++ stats_update(STATS_CACHEHIT_DIR); ++ break; ++ ++ case FROMCACHE_CPP_MODE: ++ cc_log("Succeeded getting cached result"); ++ stats_update(STATS_CACHEHIT_CPP); ++ break; ++ } ++ ++ /* and exit with the right status code */ ++ x_exit(0); ++} ++#endif ++ + // Find the real compiler. We just search the PATH to find an executable of the + // same name that isn't a link to ourselves. + static void +@@ -3059,6 +3518,19 @@ initialize(void) + create_initial_config_file(conf, primary_config_path); + } + ++ from_cache = from_fscache; ++ to_cache = to_fscache; ++ ++#ifdef HAVE_LIBMEMCACHED ++ if (strlen(conf->memcached_conf) > 0) { ++ memccached_init(conf->memcached_conf); ++ } ++ ++ if (conf->memcached_only) { ++ from_cache = from_memcached; ++ to_cache = to_memcached; ++ } ++#endif + exitfn_init(); + exitfn_add_nullary(stats_flush); + exitfn_add_nullary(clean_up_pending_tmp_files); +@@ -3089,6 +3561,7 @@ cc_reset(void) + free(output_dep); output_dep = NULL; + free(output_cov); output_cov = NULL; + free(output_dia); output_dia = NULL; ++ free(cached_key); cached_key = NULL; + free(cached_obj_hash); cached_obj_hash = NULL; + free(cached_obj); cached_obj = NULL; + free(cached_dwo); cached_dwo = NULL; +@@ -3096,6 +3569,7 @@ cc_reset(void) + free(cached_dep); cached_dep = NULL; + free(cached_cov); cached_cov = NULL; + free(cached_dia); cached_dia = NULL; ++ free(manifest_name); manifest_name = NULL; + free(manifest_path); manifest_path = NULL; + time_of_compilation = 0; + for (size_t i = 0; i < ignore_headers_len; i++) { +@@ -3119,6 +3593,10 @@ cc_reset(void) + free(stats_file); stats_file = NULL; + output_is_precompiled_header = false; + ++#ifdef HAVE_LIBMEMCACHED ++ memccached_release(); ++#endif ++ + conf = conf_create(); + using_split_dwarf = false; + } +@@ -3285,8 +3763,14 @@ ccache(int argc, char *argv[]) + put_object_in_manifest = true; + } + +- // If we can return from cache at this point then do. +- from_cache(FROMCACHE_CPP_MODE, put_object_in_manifest); ++ /* don't hit memcached twice */ ++ if (conf->memcached_only && object_hash_from_manifest ++ && file_hashes_equal(object_hash_from_manifest, object_hash)) { ++ cc_log("Already searched for %s", cached_key); ++ } else { ++ // If we can return from cache at this point then do. ++ from_cache(FROMCACHE_CPP_MODE, put_object_in_manifest); ++ } + + if (conf->read_only) { + cc_log("Read-only mode; running real compiler"); +diff --git a/ccache.h b/ccache.h +index 7b29bb8..1c1e38d 100644 +--- a/ccache.h ++++ b/ccache.h +@@ -126,6 +126,8 @@ void cc_log_argv(const char *prefix, char **argv); + void fatal(const char *format, ...) ATTR_FORMAT(printf, 1, 2) ATTR_NORETURN; + + void copy_fd(int fd_in, int fd_out); ++int safe_write(int fd_out, const char *data, size_t length); ++int write_file(const char *data, const char *dest, size_t length); + int copy_file(const char *src, const char *dest, int compress_level); + int move_file(const char *src, const char *dest, int compress_level); + int move_uncompressed_file(const char *src, const char *dest, +@@ -185,6 +187,23 @@ char *read_text_file(const char *path, size_t size_hint); + char *subst_env_in_string(const char *str, char **errmsg); + + // ---------------------------------------------------------------------------- ++// memccached.c ++ ++int memccached_init(char *conf); ++int memccached_raw_set(const char *key, const char* data, size_t len); ++int memccached_set( ++ const char *key, ++ const char *out, const char *err, const char *dia, const char *dep, ++ size_t out_len, size_t err_len, size_t dia_len, size_t dep_len); ++void *memccached_raw_get(const char *key, char **data, size_t *len); ++void* memccached_get( ++ const char *key, ++ char **out, char **err, char **dia, char **dep, ++ size_t *out_len, size_t *err_len, size_t *dia_len, size_t *dep_len); ++void memccached_free(void *blob); ++int memccached_release(void); ++ ++// ---------------------------------------------------------------------------- + // stats.c + + void stats_update(enum stats stat); +diff --git a/conf.c b/conf.c +index cfa2874..bf4e365 100644 +--- a/conf.c ++++ b/conf.c +@@ -329,11 +329,14 @@ conf_create(void) + conf->log_file = x_strdup(""); + conf->max_files = 0; + conf->max_size = (uint64_t)5 * 1000 * 1000 * 1000; ++ conf->memcached_conf = x_strdup(""); ++ conf->memcached_only = false; + conf->path = x_strdup(""); + conf->prefix_command = x_strdup(""); + conf->prefix_command_cpp = x_strdup(""); + conf->read_only = false; + conf->read_only_direct = false; ++ conf->read_only_memcached = false; + conf->recache = false; + conf->run_second_cpp = true; + conf->sloppiness = 0; +@@ -362,6 +365,7 @@ conf_free(struct conf *conf) + free(conf->extra_files_to_hash); + free(conf->ignore_headers_in_manifest); + free(conf->log_file); ++ free(conf->memcached_conf); + free(conf->path); + free(conf->prefix_command); + free(conf->prefix_command_cpp); +@@ -594,6 +598,12 @@ conf_print_items(struct conf *conf, + printer(s, conf->item_origins[find_conf("max_size")->number], context); + free(s2); + ++ reformat(&s, "memcached_conf = %s", conf->memcached_conf); ++ printer(s, conf->item_origins[find_conf("memcached_conf")->number], context); ++ ++ reformat(&s, "memcached_only = %s", bool_to_string(conf->memcached_only)); ++ printer(s, conf->item_origins[find_conf("memcached_only")->number], context); ++ + reformat(&s, "path = %s", conf->path); + printer(s, conf->item_origins[find_conf("path")->number], context); + +@@ -611,6 +621,11 @@ conf_print_items(struct conf *conf, + printer(s, conf->item_origins[find_conf("read_only_direct")->number], + context); + ++ reformat(&s, "read_only_memcached = %s", ++ bool_to_string(conf->read_only_memcached)); ++ printer(s, conf->item_origins[find_conf("read_only_memcached")->number], ++ context); ++ + reformat(&s, "recache = %s", bool_to_string(conf->recache)); + printer(s, conf->item_origins[find_conf("recache")->number], context); + +diff --git a/conf.h b/conf.h +index 232dcfd..1e22016 100644 +--- a/conf.h ++++ b/conf.h +@@ -23,11 +23,14 @@ struct conf { + char *log_file; + unsigned max_files; + uint64_t max_size; ++ char *memcached_conf; ++ bool memcached_only; + char *path; + char *prefix_command; + char *prefix_command_cpp; + bool read_only; + bool read_only_direct; ++ bool read_only_memcached; + bool recache; + bool run_second_cpp; + unsigned sloppiness; +diff --git a/configure.ac b/configure.ac +index a35fac0..7ef33e1 100644 +--- a/configure.ac ++++ b/configure.ac +@@ -16,6 +16,7 @@ case $host in + ;; + esac + ++AC_SUBST(ccache_memcached) + AC_SUBST(extra_libs) + AC_SUBST(include_dev_mk) + AC_SUBST(test_suites) +@@ -84,6 +85,31 @@ HW_FUNC_ASPRINTF + dnl Check if -lm is needed. + AC_SEARCH_LIBS(cos, m) + ++AC_ARG_ENABLE(static, ++ [AS_HELP_STRING([--enable-static], ++ [enable static link])]) ++ ++if test x${enable_static} != x; then ++ extra_ldflags="-static" ++fi ++ ++AC_ARG_ENABLE(memcached, ++ [AS_HELP_STRING([--enable-memcached], ++ [enable memcached as a cache backend])]) ++ ++dnl enable-memcached: Check if -lmemcached is needed. ++if test x${enable_memcached} != x; then ++ if test x${enable_static} != x; then ++ AC_CHECK_LIB(stdc++, __gxx_personality_v0,[]) ++ fi ++ AC_CHECK_LIB(pthread, pthread_once) ++ AC_CHECK_LIB(memcached, memcached,[],[ ++ echo ' WARNING: recent version libmemcached not found' ++ echo ' please install libmemcached > 1.0 with development files' ++ exit 1 ++ ]) ++ ccache_memcached='CCACHE_MEMCACHED=1 ' ++fi + + dnl Check for zlib + AC_ARG_WITH(bundled-zlib, +diff --git a/confitems.gperf b/confitems.gperf +index 531bc92..fd43765 100644 +--- a/confitems.gperf ++++ b/confitems.gperf +@@ -26,15 +26,18 @@ limit_multiple, 15, ITEM(limit_multiple, float) + log_file, 16, ITEM(log_file, env_string) + max_files, 17, ITEM(max_files, unsigned) + max_size, 18, ITEM(max_size, size) +-path, 19, ITEM(path, env_string) +-prefix_command, 20, ITEM(prefix_command, env_string) +-prefix_command_cpp, 21, ITEM(prefix_command_cpp, env_string) +-read_only, 22, ITEM(read_only, bool) +-read_only_direct, 23, ITEM(read_only_direct, bool) +-recache, 24, ITEM(recache, bool) +-run_second_cpp, 25, ITEM(run_second_cpp, bool) +-sloppiness, 26, ITEM(sloppiness, sloppiness) +-stats, 27, ITEM(stats, bool) +-temporary_dir, 28, ITEM(temporary_dir, env_string) +-umask, 29, ITEM(umask, umask) +-unify, 30, ITEM(unify, bool) ++memcached_conf, 19, ITEM(memcached_conf, string) ++memcached_only, 20, ITEM(memcached_only, bool) ++path, 21, ITEM(path, env_string) ++prefix_command, 22, ITEM(prefix_command, env_string) ++prefix_command_cpp, 23, ITEM(prefix_command_cpp, env_string) ++read_only, 24, ITEM(read_only, bool) ++read_only_direct, 25, ITEM(read_only_direct, bool) ++read_only_memcached, 26, ITEM(read_only_memcached, bool) ++recache, 27, ITEM(recache, bool) ++run_second_cpp, 28, ITEM(run_second_cpp, bool) ++sloppiness, 29, ITEM(sloppiness, sloppiness) ++stats, 30, ITEM(stats, bool) *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201702172323.v1HNNlRO070024>