From owner-freebsd-current@FreeBSD.ORG Fri May 13 13:43:29 2011 Return-Path: Delivered-To: freebsd-current@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 594C2106566B; Fri, 13 May 2011 13:43:29 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 554618FC14; Fri, 13 May 2011 13:43:27 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id QAA18228; Fri, 13 May 2011 16:43:26 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <4DCD357D.6000109@FreeBSD.org> Date: Fri, 13 May 2011 16:43:25 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.2.17) Gecko/20110504 Lightning/1.0b2 Thunderbird/3.1.10 MIME-Version: 1.0 To: FreeBSD current X-Enigmail-Version: 1.1.2 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: Subject: proposed smp_rendezvous change X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 13 May 2011 13:43:29 -0000 This is a change in vein of what I've been doing in the xcpu branch and it's supposed to fix the issue by the recent commit that (probably unintentionally) stress-tests smp_rendezvous in TSC code. Non-essential changes: - ditch initial, and in my opinion useless, pre-setup rendezvous in smp_rendezvous_action() - shift smp_rv_waiters indexes by one because of the above Essential changes (the fix): - re-use freed smp_rv_waiters[2] to indicate that a slave/target is really fully done with rendezvous (i.e. it's not going to access any members of smp_rv_* pseudo-structure) - spin on smp_rv_waiters[2] upon _entry_ to smp_rendezvous_cpus() to not re-use the smp_rv_* pseudo-structure too early Index: sys/kern/subr_smp.c =================================================================== --- sys/kern/subr_smp.c (revision 221835) +++ sys/kern/subr_smp.c (working copy) @@ -316,19 +316,14 @@ void (*local_action_func)(void*) = smp_rv_action_func; void (*local_teardown_func)(void*) = smp_rv_teardown_func; - /* Ensure we have up-to-date values. */ - atomic_add_acq_int(&smp_rv_waiters[0], 1); - while (smp_rv_waiters[0] < smp_rv_ncpus) - cpu_spinwait(); - /* setup function */ if (local_setup_func != smp_no_rendevous_barrier) { if (smp_rv_setup_func != NULL) smp_rv_setup_func(smp_rv_func_arg); /* spin on entry rendezvous */ - atomic_add_int(&smp_rv_waiters[1], 1); - while (smp_rv_waiters[1] < smp_rv_ncpus) + atomic_add_int(&smp_rv_waiters[0], 1); + while (smp_rv_waiters[0] < smp_rv_ncpus) cpu_spinwait(); } @@ -337,12 +332,16 @@ local_action_func(local_func_arg); /* spin on exit rendezvous */ - atomic_add_int(&smp_rv_waiters[2], 1); - if (local_teardown_func == smp_no_rendevous_barrier) + atomic_add_int(&smp_rv_waiters[1], 1); + if (local_teardown_func == smp_no_rendevous_barrier) { + atomic_add_int(&smp_rv_waiters[2], 1); return; - while (smp_rv_waiters[2] < smp_rv_ncpus) + } + while (smp_rv_waiters[1] < smp_rv_ncpus) cpu_spinwait(); + atomic_add_int(&smp_rv_waiters[2], 1); + /* teardown function */ if (local_teardown_func != NULL) local_teardown_func(local_func_arg); @@ -377,6 +376,10 @@ /* obtain rendezvous lock */ mtx_lock_spin(&smp_ipi_mtx); + /* Wait for any previous unwaited rendezvous to finish. */ + while (atomic_load_acq_int(&smp_rv_waiters[2]) < ncpus) + cpu_spinwait(); + /* set static function pointers */ smp_rv_ncpus = ncpus; smp_rv_setup_func = setup_func; @@ -395,7 +398,7 @@ smp_rendezvous_action(); if (teardown_func == smp_no_rendevous_barrier) - while (atomic_load_acq_int(&smp_rv_waiters[2]) < ncpus) + while (atomic_load_acq_int(&smp_rv_waiters[1]) < ncpus) cpu_spinwait(); /* release lock */ -- Andriy Gapon