From owner-freebsd-arch@freebsd.org Tue Jan 5 15:20:06 2016 Return-Path: Delivered-To: freebsd-arch@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8282DA63A73 for ; Tue, 5 Jan 2016 15:20:06 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 64A4318B8 for ; Tue, 5 Jan 2016 15:20:06 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from ralph.baldwin.cx (c-73-231-226-104.hsd1.ca.comcast.net [73.231.226.104]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id D341BB946 for ; Tue, 5 Jan 2016 10:20:02 -0500 (EST) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Expected taskqueue behavior Date: Tue, 05 Jan 2016 07:19:52 -0800 Message-ID: <9174153.lS3QljxB7A@ralph.baldwin.cx> User-Agent: KMail/4.14.3 (FreeBSD/10.2-STABLE; KDE/4.14.3; amd64; ; ) MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 05 Jan 2016 10:20:02 -0500 (EST) X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Jan 2016 15:20:06 -0000 Given this bit of code: void cpu_hog(void *context, int pending) { for (;;) ; } static struct taskqueue *tq; static struct task t; void mod_init(void) { int i; TASK_INIT(&t, 0, cpu_hog, NULL); tq = taskqueue_create("test", M_WAITOK, taskqueue_thread_enqueue, &tq); taskqueue_start_threads(&tq, mp_ncpu, PRI_MAX_IDLE, "test taskq"); for (i = 0; i < mp_ncpu; i++) { taskqueue_enqueue(tq, &t); pause("foo", 1); } } How many threads would you expect to be busy? The taskqueue(9) manpage does not describe what happens when you queue a task that is already executing. Currently our implementation will allow the task to execute on a second thread (or a third, fourth, fifth, etc.). I think this behavior is probably "surprising" as I expect most folks assume that a task won't be executed concurrently and rely on this assumption to avoid the need for certain types of serialization in task handlers. Linux workqueues explicitly document work-items as being non-reentrant. >From https://www.kernel.org/doc/Documentation/workqueue.txt Note that the flag WQ_NON_REENTRANT no longer exists as all workqueues are now non-reentrant - any work item is guaranteed to be executed by at most one worker system-wide at any given time. Windows work queues also forbid this though indirectly becuase they forbid extra queueing (which bumps 'pending' on FreeBSD) outright. >From https://msdn.microsoft.com/en-us/library/windows/hardware/ff564587%28v=vs.85%29.aspx Do not call IoQueueWorkItem or IoQueueWorkItemEx to queue a work item that is already in the queue. In checked builds, this error causes a bug check. In retail builds, the error is not detected but can cause corruption of system data structures. I think our current behavior is "surprising" and is a bug from when taskqueue thread pools were added (the case simply wasn't handled). At a minimum we should document the expected behavior (and fix it if its wrong). If we don't change our current behavior then the Linux KPI workqueue implementation will need additional changes as it can no longer simply call taskqueue_enqueue(). -- John Baldwin