From owner-svn-src-all@FreeBSD.ORG Sat Sep 11 07:08:22 2010 Return-Path: Delivered-To: svn-src-all@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C8861106564A; Sat, 11 Sep 2010 07:08:22 +0000 (UTC) (envelope-from mav@FreeBSD.org) Received: from svn.freebsd.org (svn.freebsd.org [IPv6:2001:4f8:fff6::2c]) by mx1.freebsd.org (Postfix) with ESMTP id 9DCCF8FC14; Sat, 11 Sep 2010 07:08:22 +0000 (UTC) Received: from svn.freebsd.org (localhost [127.0.0.1]) by svn.freebsd.org (8.14.3/8.14.3) with ESMTP id o8B78MRc060494; Sat, 11 Sep 2010 07:08:22 GMT (envelope-from mav@svn.freebsd.org) Received: (from mav@localhost) by svn.freebsd.org (8.14.3/8.14.3/Submit) id o8B78MSK060492; Sat, 11 Sep 2010 07:08:22 GMT (envelope-from mav@svn.freebsd.org) Message-Id: <201009110708.o8B78MSK060492@svn.freebsd.org> From: Alexander Motin Date: Sat, 11 Sep 2010 07:08:22 +0000 (UTC) To: src-committers@freebsd.org, svn-src-all@freebsd.org, svn-src-head@freebsd.org X-SVN-Group: head MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: svn commit: r212455 - head/sys/kern X-BeenThere: svn-src-all@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "SVN commit messages for the entire src tree \(except for " user" and " projects" \)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Sep 2010 07:08:22 -0000 Author: mav Date: Sat Sep 11 07:08:22 2010 New Revision: 212455 URL: http://svn.freebsd.org/changeset/base/212455 Log: Merge some SCHED_ULE features to SCHED_4BSD: - Teach SCHED_4BSD to inform cpu_idle() about high sleep/wakeup rate to choose optimized handler. In case of x86 it is MONITOR/MWAIT. Also it will be needed to bypass forthcoming idle tick skipping logic to not consume resources on events rescheduling when it won't give any benefits. - Teach SCHED_4BSD to wake up idle CPUs without using IPI. In case of x86, when MONITOR/MWAIT is active, it require just single memory write. This doubles performance on some heavily switching test loads. Modified: head/sys/kern/sched_4bsd.c Modified: head/sys/kern/sched_4bsd.c ============================================================================== --- head/sys/kern/sched_4bsd.c Sat Sep 11 06:38:34 2010 (r212454) +++ head/sys/kern/sched_4bsd.c Sat Sep 11 07:08:22 2010 (r212455) @@ -157,6 +157,12 @@ static struct runq runq_pcpu[MAXCPU]; long runq_length[MAXCPU]; #endif +struct pcpuidlestat { + u_int idlecalls; + u_int oldidlecalls; +}; +static DPCPU_DEFINE(struct pcpuidlestat, idlestat); + static void setup_runqs(void) { @@ -684,6 +690,7 @@ sched_rr_interval(void) void sched_clock(struct thread *td) { + struct pcpuidlestat *stat; struct td_sched *ts; THREAD_LOCK_ASSERT(td, MA_OWNED); @@ -703,6 +710,10 @@ sched_clock(struct thread *td) if (!TD_IS_IDLETHREAD(td) && ticks - PCPU_GET(switchticks) >= sched_quantum) td->td_flags |= TDF_NEEDRESCHED; + + stat = DPCPU_PTR(idlestat); + stat->oldidlecalls = stat->idlecalls; + stat->idlecalls = 0; } /* @@ -1137,7 +1148,15 @@ forward_wakeup(int cpunum) } if (map) { forward_wakeups_delivered++; - ipi_selected(map, IPI_AST); + SLIST_FOREACH(pc, &cpuhead, pc_allcpu) { + id = pc->pc_cpumask; + if ((map & id) == 0) + continue; + if (cpu_idle_wakeup(pc->pc_cpuid)) + map &= ~id; + } + if (map) + ipi_selected(map, IPI_AST); return (1); } if (cpunum == NOCPU) @@ -1154,7 +1173,8 @@ kick_other_cpu(int pri, int cpuid) pcpu = pcpu_find(cpuid); if (idle_cpus_mask & pcpu->pc_cpumask) { forward_wakeups_delivered++; - ipi_cpu(cpuid, IPI_AST); + if (!cpu_idle_wakeup(cpuid)) + ipi_cpu(cpuid, IPI_AST); return; } @@ -1537,12 +1557,16 @@ sched_tick(void) void sched_idletd(void *dummy) { + struct pcpuidlestat *stat; + stat = DPCPU_PTR(idlestat); for (;;) { mtx_assert(&Giant, MA_NOTOWNED); - while (sched_runnable() == 0) - cpu_idle(0); + while (sched_runnable() == 0) { + cpu_idle(stat->idlecalls + stat->oldidlecalls > 64); + stat->idlecalls++; + } mtx_lock_spin(&sched_lock); mi_switch(SW_VOL | SWT_IDLE, NULL);