From owner-svn-src-head@freebsd.org Wed Jul 3 19:22:46 2019 Return-Path: Delivered-To: svn-src-head@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D40315DE3CD; Wed, 3 Jul 2019 19:22:46 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from mxrelay.nyi.freebsd.org (mxrelay.nyi.freebsd.org [IPv6:2610:1c1:1:606c::19:3]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "mxrelay.nyi.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 869528BDFF; Wed, 3 Jul 2019 19:22:45 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org (repo.freebsd.org [IPv6:2610:1c1:1:6068::e6a:0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mxrelay.nyi.freebsd.org (Postfix) with ESMTPS id 581DB1839B; Wed, 3 Jul 2019 19:22:45 +0000 (UTC) (envelope-from vangyzen@FreeBSD.org) Received: from repo.freebsd.org ([127.0.1.37]) by repo.freebsd.org (8.15.2/8.15.2) with ESMTP id x63JMjbZ062944; Wed, 3 Jul 2019 19:22:45 GMT (envelope-from vangyzen@FreeBSD.org) Received: (from vangyzen@localhost) by repo.freebsd.org (8.15.2/8.15.2/Submit) id x63JMiAa062941; Wed, 3 Jul 2019 19:22:44 GMT (envelope-from vangyzen@FreeBSD.org) Message-Id: <201907031922.x63JMiAa062941@repo.freebsd.org> X-Authentication-Warning: repo.freebsd.org: vangyzen set sender to vangyzen@FreeBSD.org using -f From: Eric van Gyzen Date: Wed, 3 Jul 2019 19:22:44 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org Subject: svn commit: r349677 - in head: sys/kern tools/test/callout_free X-SVN-Group: head X-SVN-Commit-Author: vangyzen X-SVN-Commit-Paths: in head: sys/kern tools/test/callout_free X-SVN-Commit-Revision: 349677 X-SVN-Commit-Repository: base MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 869528BDFF X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-2.96 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; NEURAL_HAM_SHORT(-0.96)[-0.961,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; ASN(0.00)[asn:11403, ipnet:2610:1c1:1::/48, country:US] X-BeenThere: svn-src-head@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: SVN commit messages for the src tree for head/-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Jul 2019 19:22:46 -0000 Author: vangyzen Date: Wed Jul 3 19:22:44 2019 New Revision: 349677 URL: https://svnweb.freebsd.org/changeset/base/349677 Log: Save the last callout function executed on each CPU Save the last callout function pointer (and its argument) executed on each CPU for inspection by a debugger. Add a ddb `show callout_last` command to show these pointers. Add a kernel module that I used for testing that command. Relocate `ce_migration_cpu` to reduce padding and therefore preserve the size of `struct callout_cpu` (320 bytes on amd64) despite the added members. This should help diagnose reference-after-free bugs where the callout's mutex has already been freed when `softclock_call_cc` tries to unlock it. You might hope that the pointer would still be available, but it isn't. The argument to that function is on the stack (because `softclock_call_cc` uses it later), and that might be enough in some cases, but even then, it's very laborious. A pointer to the callout is saved right before these newly added fields, but that callout might have been freed. We still have the pointer to its associated mutex, and the name within might be enough, but it might also have been freed. Reviewed by: markj jhb MFC after: 2 weeks Sponsored by: Dell EMC Isilon Differential Revision: https://reviews.freebsd.org/D20794 Added: head/tools/test/callout_free/ head/tools/test/callout_free/Makefile (contents, props changed) head/tools/test/callout_free/callout_free.c (contents, props changed) Modified: head/sys/kern/kern_timeout.c Modified: head/sys/kern/kern_timeout.c ============================================================================== --- head/sys/kern/kern_timeout.c Wed Jul 3 19:22:25 2019 (r349676) +++ head/sys/kern/kern_timeout.c Wed Jul 3 19:22:44 2019 (r349677) @@ -65,6 +65,7 @@ __FBSDID("$FreeBSD$"); #ifdef DDB #include +#include #include #endif @@ -143,12 +144,14 @@ u_int callwheelsize, callwheelmask; struct cc_exec { struct callout *cc_curr; void (*cc_drain)(void *); + void *cc_last_func; + void *cc_last_arg; #ifdef SMP void (*ce_migration_func)(void *); void *ce_migration_arg; - int ce_migration_cpu; sbintime_t ce_migration_time; sbintime_t ce_migration_prec; + int ce_migration_cpu; #endif bool cc_cancel; bool cc_waiting; @@ -177,6 +180,8 @@ struct callout_cpu { #define callout_migrating(c) ((c)->c_iflags & CALLOUT_DFRMIGRATION) #define cc_exec_curr(cc, dir) cc->cc_exec_entity[dir].cc_curr +#define cc_exec_last_func(cc, dir) cc->cc_exec_entity[dir].cc_last_func +#define cc_exec_last_arg(cc, dir) cc->cc_exec_entity[dir].cc_last_arg #define cc_exec_drain(cc, dir) cc->cc_exec_entity[dir].cc_drain #define cc_exec_next(cc) cc->cc_next #define cc_exec_cancel(cc, dir) cc->cc_exec_entity[dir].cc_cancel @@ -686,6 +691,8 @@ softclock_call_cc(struct callout *c, struct callout_cp c->c_iflags &= ~CALLOUT_PENDING; cc_exec_curr(cc, direct) = c; + cc_exec_last_func(cc, direct) = c_func; + cc_exec_last_arg(cc, direct) = c_arg; cc_exec_cancel(cc, direct) = false; cc_exec_drain(cc, direct) = NULL; CC_UNLOCK(cc); @@ -1669,5 +1676,43 @@ DB_SHOW_COMMAND(callout, db_show_callout) } _show_callout((struct callout *)addr); +} + +static void +_show_last_callout(int cpu, int direct, const char *dirstr) +{ + struct callout_cpu *cc; + void *func, *arg; + + cc = CC_CPU(cpu); + func = cc_exec_last_func(cc, direct); + arg = cc_exec_last_arg(cc, direct); + db_printf("cpu %d last%s callout function: %p ", cpu, dirstr, func); + db_printsym((db_expr_t)func, DB_STGY_ANY); + db_printf("\ncpu %d last%s callout argument: %p\n", cpu, dirstr, arg); +} + +DB_SHOW_COMMAND(callout_last, db_show_callout_last) +{ + int cpu, last; + + if (have_addr) { + if (addr < 0 || addr > mp_maxid || CPU_ABSENT(addr)) { + db_printf("no such cpu: %d\n", (int)addr); + return; + } + cpu = last = addr; + } else { + cpu = 0; + last = mp_maxid; + } + + while (cpu <= last) { + if (!CPU_ABSENT(cpu)) { + _show_last_callout(cpu, 0, ""); + _show_last_callout(cpu, 1, " direct"); + } + cpu++; + } } #endif /* DDB */ Added: head/tools/test/callout_free/Makefile ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/tools/test/callout_free/Makefile Wed Jul 3 19:22:44 2019 (r349677) @@ -0,0 +1,6 @@ +# $FreeBSD$ + +KMOD= callout_free +SRCS= callout_free.c + +.include Added: head/tools/test/callout_free/callout_free.c ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ head/tools/test/callout_free/callout_free.c Wed Jul 3 19:22:44 2019 (r349677) @@ -0,0 +1,87 @@ +/*- + * SPDX-License-Identifier: BSD-2-Clause + * + * Copyright (c) 2019 Eric van Gyzen + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + * + * $FreeBSD$ + */ + +/* + * Free a pending callout. This was useful for testing the + * "show callout_last" ddb command. + */ + +#include +#include +#include +#include +#include +#include +#include + +static struct callout callout_free; +static struct mtx callout_free_mutex; +static int callout_free_arg; + +static void +callout_free_func(void *arg) +{ + printf("squirrel!\n"); + mtx_destroy(&callout_free_mutex); + memset(&callout_free, 'C', sizeof(callout_free)); +} + +static int +callout_free_load(module_t mod, int cmd, void *arg) +{ + int error; + + switch (cmd) { + case MOD_LOAD: + mtx_init(&callout_free_mutex, "callout_free", NULL, MTX_DEF); + /* + * Do not pass CALLOUT_RETURNUNLOCKED so the callout + * subsystem will unlock the "destroyed" mutex. + */ + callout_init_mtx(&callout_free, &callout_free_mutex, 0); + printf("callout_free_func = %p\n", callout_free_func); + printf("callout_free_arg = %p\n", &callout_free_arg); + callout_reset(&callout_free, hz/10, callout_free_func, + &callout_free_arg); + error = 0; + break; + + case MOD_UNLOAD: + error = 0; + break; + + default: + error = EOPNOTSUPP; + break; + } + + return (error); +} + +DEV_MODULE(callout_free, callout_free_load, NULL);