From owner-svn-src-all@freebsd.org Thu May 17 19:50:56 2018 Return-Path: Delivered-To: svn-src-all@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A7BADEDEC8D; Thu, 17 May 2018 19:50:56 +0000 (UTC) (envelope-from mmacy@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5605973A88; Thu, 17 May 2018 19:50:56 +0000 (UTC) (envelope-from mmacy@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 384EF5E86; Thu, 17 May 2018 19:50:56 +0000 (UTC) (envelope-from mmacy@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id w4HJou3x029611; Thu, 17 May 2018 19:50:56 GMT (envelope-from mmacy@FreeBSD.org) Received: (from mmacy@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id w4HJou5W029610; Thu, 17 May 2018 19:50:56 GMT (envelope-from mmacy@FreeBSD.org) Message-Id: <201805171950.w4HJou5W029610@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: mmacy set sender to mmacy@FreeBSD.org using -f From: Matt Macy Date: Thu, 17 May 2018 19:50:56 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r333759 - head/sys/kern X-SVN-Group: head X-SVN-Commit-Author: mmacy X-SVN-Commit-Paths: head/sys/kern X-SVN-Commit-Revision: 333759 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 17 May 2018 19:50:56 -0000 Author: mmacy Date: Thu May 17 19:50:55 2018 New Revision: 333759 URL: https://svnweb.freebsd.org/changeset/base/333759 Log: epoch(9): eliminate the need to wait when polling for callbacks to run by using ck's own callback handling mechanism we can simply check which callbacks have had a grace period elapse Approved by: sbruno Modified: head/sys/kern/subr_epoch.c Modified: head/sys/kern/subr_epoch.c ============================================================================== --- head/sys/kern/subr_epoch.c Thu May 17 19:41:58 2018 (r333758) +++ head/sys/kern/subr_epoch.c Thu May 17 19:50:55 2018 (r333759) @@ -64,6 +64,7 @@ static MALLOC_DEFINE(M_EPOCH, "epoch", "epoch based re #endif CTASSERT(sizeof(epoch_section_t) == sizeof(ck_epoch_section_t)); +CTASSERT(sizeof(ck_epoch_entry_t) == sizeof(struct epoch_context)); SYSCTL_NODE(_kern, OID_AUTO, epoch, CTLFLAG_RW, 0, "epoch information"); SYSCTL_NODE(_kern_epoch, OID_AUTO, stats, CTLFLAG_RW, 0, "epoch stats"); @@ -84,13 +85,11 @@ static counter_u64_t switch_count; SYSCTL_COUNTER_U64(_kern_epoch_stats, OID_AUTO, switches, CTLFLAG_RW, &switch_count, "# of times a thread voluntarily context switched in epoch_wait"); -typedef struct epoch_cb { - void (*ec_callback)(epoch_context_t); - STAILQ_ENTRY(epoch_cb) ec_link; -} *epoch_cb_t; - TAILQ_HEAD(threadlist, thread); +CK_STACK_CONTAINER(struct ck_epoch_entry, stack_entry, + ck_epoch_entry_container) + typedef struct epoch_record { ck_epoch_record_t er_record; volatile struct threadlist er_tdlist; @@ -100,7 +99,6 @@ typedef struct epoch_record { struct epoch_pcpu_state { struct epoch_record eps_record; - STAILQ_HEAD(, epoch_cb) eps_cblist; } __aligned(EPOCH_ALIGN); struct epoch { @@ -179,7 +177,6 @@ epoch_init_numa(epoch_t epoch) for (int i = 0; i < domcount[domain]; i++, eps++) { epoch->e_pcpu[cpu_offset + i] = eps; er = &eps->eps_record; - STAILQ_INIT(&eps->eps_cblist); ck_epoch_register(&epoch->e_epoch, &er->er_record, NULL); TAILQ_INIT((struct threadlist *)(uintptr_t)&er->er_tdlist); er->er_cpuid = cpu_offset + i; @@ -200,7 +197,6 @@ epoch_init_legacy(epoch_t epoch) er = &eps->eps_record; ck_epoch_register(&epoch->e_epoch, &er->er_record, NULL); TAILQ_INIT((struct threadlist *)(uintptr_t)&er->er_tdlist); - STAILQ_INIT(&eps->eps_cblist); er->er_cpuid = i; } } @@ -525,25 +521,24 @@ void epoch_call(epoch_t epoch, epoch_context_t ctx, void (*callback) (epoch_context_t)) { struct epoch_pcpu_state *eps; - epoch_cb_t cb; + ck_epoch_entry_t *cb; cb = (void *)ctx; MPASS(callback); /* too early in boot to have epoch set up */ - if (__predict_false(epoch == NULL)) { - callback(ctx); - return; - } - MPASS(cb->ec_callback == NULL); - MPASS(cb->ec_link.stqe_next == NULL); - cb->ec_callback = callback; + if (__predict_false(epoch == NULL)) + goto boottime; + counter_u64_add(epoch->e_frees, 1); critical_enter(); eps = epoch->e_pcpu[curcpu]; - STAILQ_INSERT_HEAD(&eps->eps_cblist, cb, ec_link); + ck_epoch_call(&eps->eps_record.er_record, cb, (ck_epoch_cb_t*)callback); critical_exit(); + return; + boottime: + callback(ctx); } static void @@ -551,28 +546,26 @@ epoch_call_task(void *context) { struct epoch_pcpu_state *eps; epoch_t epoch; - epoch_cb_t cb; struct thread *td; + ck_stack_entry_t *cursor; + ck_stack_t deferred; int cpu; - STAILQ_HEAD(, epoch_cb) tmp_head; epoch = context; - STAILQ_INIT(&tmp_head); td = curthread; + ck_stack_init(&deferred); thread_lock(td); CPU_FOREACH(cpu) { sched_bind(td, cpu); eps = epoch->e_pcpu[cpu]; - if (!STAILQ_EMPTY(&eps->eps_cblist)) - STAILQ_CONCAT(&tmp_head, &eps->eps_cblist); + ck_epoch_poll_deferred(&eps->eps_record.er_record, &deferred); } sched_unbind(td); thread_unlock(td); - epoch_wait(epoch); - - while ((cb = STAILQ_FIRST(&tmp_head)) != NULL) { - STAILQ_REMOVE_HEAD(&tmp_head, ec_link); - cb->ec_callback((void*)cb); + while((cursor = ck_stack_pop_npsc(&deferred)) != NULL) { + struct ck_epoch_entry *entry = + ck_epoch_entry_container(cursor); + entry->function(entry); } }