From owner-freebsd-rc@freebsd.org Fri Oct 16 16:06:03 2015 Return-Path: Delivered-To: freebsd-rc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 71E4CA165A7 for ; Fri, 16 Oct 2015 16:06:03 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 487E31AB3 for ; Fri, 16 Oct 2015 16:06:03 +0000 (UTC) (envelope-from feld@FreeBSD.org) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 85DF42088A for ; Fri, 16 Oct 2015 12:06:01 -0400 (EDT) Received: from web3 ([10.202.2.213]) by compute1.internal (MEProxy); Fri, 16 Oct 2015 12:06:01 -0400 DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; d= messagingengine.com; h=content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-sasl-enc:x-sasl-enc; s=smtpout; bh=dN7Kioy9eGQNt1l Bcysa4IkU8gY=; b=AJ6IUm/jOoxPYlVzVRMRB0Xw7nihtXOe9WCR79BnYoqXis4 AqX+T19bnmMWBuYNiBu1pMIpWnRxxhRtUDVeoAmrtCDu096LUf7atVaLdHiYhp9b qrr38d6EQRTxn5q5UML92AN9WV2VdGAOjI9tmeXoofATIjyx34n6ptg66hEs= Received: by web3.nyi.internal (Postfix, from userid 99) id 4BDBF11101C; Fri, 16 Oct 2015 12:06:01 -0400 (EDT) Message-Id: <1445011561.1233840.412174489.688C5822@webmail.messagingengine.com> X-Sasl-Enc: dSD1SdLLWDCvnSxW39Tluk1pP1Ns4PS3dhY9QYBkJwr9 1445011561 From: Mark Felder To: Cyril Vechera , dougb@FreeBSD.org, freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Type: text/plain X-Mailer: MessagingEngine.com Webmail Interface - ajax-25d3ec43 Subject: Re: rc(8) parallel tasks Date: Fri, 16 Oct 2015 11:06:01 -0500 In-Reply-To: <560EAC05.6050308@jet9.net> References: <560EAC05.6050308@jet9.net> X-BeenThere: freebsd-rc@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion related to /etc/rc.d design and implementation." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2015 16:06:03 -0000 On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote: > Hi there. > > We've got a small launcher script (~250 loc) for parallel services > start/stop etc. It is used on our embedded systems and our users > containers. And I've done a proof of concept for implanting it to the > FreeBSD's standard /etc/rc for execution starting scripts in parallel. > It gave me a boot time reduction of rc part from 27 to 7 seconds, mostly > on eliminating jams for network or other long-latency resources waiting. > > The launcher is written in pure POSIX shell and uses FIFOs (named pipes) > as a mutexes for synchronization. So it is embedded into /etc/rc and > /etc/rc.d preserving rc.subr preloading. As a primary requirement, it > guarantees topological order (strict partial order) defined by > dependencies. It requires only POSIX shell, FreeBSD or Linux kernel, > mkfifo and a writeable file system. Due to last requirement, it can be > run on the late stage or should be supplied by some kinf of writtable > fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder > annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to change > rc.d scripts > > It's not a full init replacement or a kind of services supervision tool. > It only starts or invokes a group of scripts in parallel with resolving > and assuring execution in dependencies order. > > Please take a look at the script and patch set for FreeBSD: > > https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-init > https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd > > Your first link is a 404, but this looks really nice! -- Mark Felder ports-secteam member feld@FreeBSD.org From owner-freebsd-rc@freebsd.org Fri Oct 16 18:47:27 2015 Return-Path: Delivered-To: freebsd-rc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0BAABA17E4A; Fri, 16 Oct 2015 18:47:27 +0000 (UTC) (envelope-from cv@jet9.net) Received: from mail1.ssnab.net (mail1.ssnab.net [85.143.179.24]) by mx1.freebsd.org (Postfix) with ESMTP id 81A55795; Fri, 16 Oct 2015 18:47:25 +0000 (UTC) (envelope-from cv@jet9.net) Received: from inc.ru (localhost [127.0.0.1]) by mail1.ssnab.net (Postfix) with SMTP id E004EC0A39; Fri, 16 Oct 2015 21:37:27 +0300 (MSK) X-Antispam-passed: yes X-Antispam: yes Received: from [95.27.88.77] (account cv@serversnab.ru HELO new-cv.home) by inc.ru (CommuniGate Pro SMTP 4.1.8) with ESMTP-TLS id 24040011; Fri, 16 Oct 2015 21:37:27 +0300 Message-ID: <562143E7.3030104@jet9.net> Date: Fri, 16 Oct 2015 21:37:27 +0300 From: Cyril Vechera User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130330 Thunderbird/17.0.5 MIME-Version: 1.0 To: Mark Felder CC: marck@rinet.ru, freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org Subject: Re: rc(8) parallel tasks References: <560EAC05.6050308@jet9.net> <1445011561.1233840.412174489.688C5822@webmail.messagingengine.com> In-Reply-To: <1445011561.1233840.412174489.688C5822@webmail.messagingengine.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-rc@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion related to /etc/rc.d design and implementation." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2015 18:47:27 -0000 On 10/16/2015 07:06 PM, Mark Felder wrote: > On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote: >> Hi there. >> >> We've got a small launcher script (~250 loc) for parallel services >> start/stop etc. It is used on our embedded systems and our users >> containers. And I've done a proof of concept for implanting it to the >> FreeBSD's standard /etc/rc for execution starting scripts in parallel. >> It gave me a boot time reduction of rc part from 27 to 7 seconds, mostly >> on eliminating jams for network or other long-latency resources waiting. >> >> The launcher is written in pure POSIX shell and uses FIFOs (named pipes) >> as a mutexes for synchronization. So it is embedded into /etc/rc and >> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it >> guarantees topological order (strict partial order) defined by >> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel, >> mkfifo and a writeable file system. Due to last requirement, it can be >> run on the late stage or should be supplied by some kinf of writtable >> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder >> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to change >> rc.d scripts >> >> It's not a full init replacement or a kind of services supervision tool. >> It only starts or invokes a group of scripts in parallel with resolving >> and assuring execution in dependencies order. >> >> Please take a look at the script and patch set for FreeBSD: >> >> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-init >> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd >> >> > Your first link is a 404, but this looks really nice! In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to 'jet9-multitask-flow' to avoid confusion with naming, because it's not as it seems, from name, an init replacement, but just a parallel task launcher. And now the actual repository URL is https://github.com/cvss/jet9-multitask-flow In this commit I've complete the FreeBSD compatibility. Now a script or dependency name can include minuses `-` and dots `.` (the first stone I stumbled over was ftp-proxy). And I've cleaned up the main script code https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow and have split it to more functions that can be redefined. So it's now easier to rewrite dependency extraction for FreeBSD rc-scripts - current implementation is too rough and takes three 'awk' runs for each rc-script. The last is not only time loss, but as DMarck mentioned before, using awk restricts parallel rc to be run only after FILESYSTEMS stage is done. Maybe it would be better to add to the rcorder(8) some new option to dump the gathered dependencies in tsort-compatible listing and insert them directly to flow execution plan. I've also added rc.conf variable `rc_parallel` to turn on and off parallel execution. There's a risk to discover an incomplete dependency annotations in some rc-scripts that earlier were masked by serial execution. I've done some checks by enabling as much rc.conf variables as possible to start more rc-scripts, and didn't found any error. But it looks too good to be true and I'm afraid that it's just to poor testing. I think if some ordering conflict will be found, it could be worked-around with introducing a white-list for script names that must be run only in sequentially. So it remained first to check if it really works in different conditions. -- Cyril Vechera From owner-freebsd-rc@freebsd.org Fri Oct 16 22:14:10 2015 Return-Path: Delivered-To: freebsd-rc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84DE1A16037; Fri, 16 Oct 2015 22:14:10 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from mail.ijs.si (mail.ijs.si [IPv6:2001:1470:ff80::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2EFA5ED5; Fri, 16 Oct 2015 22:14:09 +0000 (UTC) (envelope-from Mark.Martinec+freebsd@ijs.si) Received: from amavis-ori.ijs.si (localhost [IPv6:::1]) by mail.ijs.si (Postfix) with ESMTP id 3nd1wY64LJzKm; Sat, 17 Oct 2015 00:14:05 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=ijs.si; h= user-agent:message-id:references:in-reply-to:organization :subject:subject:from:from:date:date:content-transfer-encoding :content-type:content-type:mime-version:received:received :received:received; s=jakla4; t=1445033642; x=1447625643; bh=zmt pQaUSTtZbXPjXmo0pGW/UaQGoOZdU2ncQOebgH+I=; b=hgnuVkSnLhPM/pd52qZ OeCwWdHT5SMLYkvrKw3+20NDI5n0wljRlnerVsHSGfIuQUXOw9HwgEWbGThle6Ae uUA1/ZS/dNEKObEHQF3phBKnMfAyOtre4BS/j1Xq1wIdKhcWvMO7lrCV50qJFaCt n7ZNGSaKWG6DCyUIN7yLukjc= X-Virus-Scanned: amavisd-new at ijs.si Received: from mail.ijs.si ([IPv6:::1]) by amavis-ori.ijs.si (mail.ijs.si [IPv6:::1]) (amavisd-new, port 10026) with LMTP id uyPcUiMYkl6h; Sat, 17 Oct 2015 00:14:02 +0200 (CEST) Received: from mildred.ijs.si (mailbox.ijs.si [IPv6:2001:1470:ff80::143:1]) by mail.ijs.si (Postfix) with ESMTP id 3nd1wT2pw2zKl; Sat, 17 Oct 2015 00:14:01 +0200 (CEST) Received: from nabiralnik.ijs.si (nabiralnik.ijs.si [IPv6:2001:1470:ff80::80:16]) by mildred.ijs.si (Postfix) with ESMTP id 3nd1wT1Jr7zwf; Sat, 17 Oct 2015 00:14:01 +0200 (CEST) Received: from sleepy.ijs.si (2001:1470:ff80:e001::1:1) by nabiralnik.ijs.si with HTTP (HTTP/1.1 POST); Sat, 17 Oct 2015 00:14:01 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII; format=flowed Content-Transfer-Encoding: 7bit Date: Sat, 17 Oct 2015 00:14:01 +0200 From: Mark Martinec To: freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org Cc: Cyril Vechera Subject: Re: rc(8) parallel tasks Organization: Jozef Stefan Institute In-Reply-To: <562143E7.3030104@jet9.net> References: <560EAC05.6050308@jet9.net> <1445011561.1233840.412174489.688C5822@webmail.messagingengine.com> <562143E7.3030104@jet9.net> Message-ID: <2c307d32ddd8ed582ac496d56a75c3e4@mailbox.ijs.si> X-Sender: Mark.Martinec+freebsd@ijs.si User-Agent: Roundcube Webmail/1.1.3 X-BeenThere: freebsd-rc@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion related to /etc/rc.d design and implementation." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Oct 2015 22:14:10 -0000 Tried a today's git checkout on one of our small virtual 10.2 boxes with not many services. The installation is almost trivial and a reboot went without any problems. Time for a reboot was shrunk by about 10 seconds (which is understandable with a small number of services). Very good, almost too good to be true for such fairly small and non-intrusive piece of shell/awk scripting - thanks! Mark 2015-10-16 20:37, Cyril Vechera wrote: > On 10/16/2015 07:06 PM, Mark Felder wrote: >> On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote: >>> Hi there. >>> >>> We've got a small launcher script (~250 loc) for parallel services >>> start/stop etc. It is used on our embedded systems and our users >>> containers. And I've done a proof of concept for implanting it to the >>> FreeBSD's standard /etc/rc for execution starting scripts in >>> parallel. >>> It gave me a boot time reduction of rc part from 27 to 7 seconds, >>> mostly >>> on eliminating jams for network or other long-latency resources >>> waiting. >>> >>> The launcher is written in pure POSIX shell and uses FIFOs (named >>> pipes) >>> as a mutexes for synchronization. So it is embedded into /etc/rc and >>> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it >>> guarantees topological order (strict partial order) defined by >>> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel, >>> mkfifo and a writeable file system. Due to last requirement, it can >>> be >>> run on the late stage or should be supplied by some kinf of writtable >>> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder >>> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to >>> change >>> rc.d scripts >>> >>> It's not a full init replacement or a kind of services supervision >>> tool. >>> It only starts or invokes a group of scripts in parallel with >>> resolving >>> and assuring execution in dependencies order. >>> >>> Please take a look at the script and patch set for FreeBSD: >>> >>> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-init >>> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd >>> >>> >> Your first link is a 404, but this looks really nice! > > In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to > 'jet9-multitask-flow' to avoid confusion with naming, because it's not > as it seems, from name, an init replacement, but just a parallel task > launcher. And now the actual repository URL is > https://github.com/cvss/jet9-multitask-flow > > In this commit I've complete the FreeBSD compatibility. Now a script > or dependency name can include minuses `-` and dots `.` (the first > stone I stumbled over was ftp-proxy). And I've cleaned up the main > script code > https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow and > have split it to more functions that can be redefined. So it's now > easier to rewrite dependency extraction for FreeBSD rc-scripts - > current implementation is too rough and takes three 'awk' runs for > each rc-script. The last is not only time loss, but as DMarck > mentioned before, using awk restricts parallel rc to be run only after > FILESYSTEMS stage is done. Maybe it would be better to add to the > rcorder(8) some new option to dump the gathered dependencies in > tsort-compatible listing and insert them directly to flow execution > plan. > > I've also added rc.conf variable `rc_parallel` to turn on and off > parallel execution. There's a risk to discover an incomplete > dependency annotations in some rc-scripts that earlier were masked by > serial execution. I've done some checks by enabling as much rc.conf > variables as possible to start more rc-scripts, and didn't found any > error. But it looks too good to be true and I'm afraid that it's just > to poor testing. I think if some ordering conflict will be found, it > could be worked-around with introducing a white-list for script names > that must be run only in sequentially. > > So it remained first to check if it really works in different > conditions. From owner-freebsd-rc@freebsd.org Sat Oct 17 12:25:19 2015 Return-Path: Delivered-To: freebsd-rc@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 41BC9A1759E for ; Sat, 17 Oct 2015 12:25:19 +0000 (UTC) (envelope-from zcwalinski@metroland.com) Received: from na01-bn1-obe.outbound.protection.outlook.com (mail-bn1bon0070.outbound.protection.outlook.com [157.56.111.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "MSIT Machine Auth CA 2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BF2E1107F for ; Sat, 17 Oct 2015 12:25:17 +0000 (UTC) (envelope-from zcwalinski@metroland.com) Received: from CY1PR08MB1415.namprd08.prod.outlook.com (10.161.172.142) by CY1PR08MB1989.namprd08.prod.outlook.com (10.164.222.21) with Microsoft SMTP Server (TLS) id 15.1.300.14; Sat, 17 Oct 2015 12:25:15 +0000 Received: from CY1PR08MB1413.namprd08.prod.outlook.com (10.161.172.140) by CY1PR08MB1415.namprd08.prod.outlook.com (10.161.172.142) with Microsoft SMTP Server (TLS) id 15.1.300.14; Sat, 17 Oct 2015 12:25:12 +0000 Received: from CY1PR08MB1413.namprd08.prod.outlook.com ([10.161.172.140]) by CY1PR08MB1413.namprd08.prod.outlook.com ([10.161.172.140]) with mapi id 15.01.0300.010; Sat, 17 Oct 2015 12:25:12 +0000 From: "Cwalinski, Zygmunt" To: "freebsd-rc-request@freebsd.org" , "freebsd-rc@freebsd.org" Subject: Re: freebsd-rc Digest, Vol 470, Issue 1 Thread-Topic: freebsd-rc Digest, Vol 470, Issue 1 Thread-Index: AQHRCNbew6EpJhUqrUuXNABeON5t0A== Date: Sat, 17 Oct 2015 12:25:12 +0000 Message-ID: <20151017122510.30285913.76124.33808@metroland.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-client-id: 33808 authentication-results: spf=none (sender IP is ) smtp.mailfrom=zcwalinski@metroland.com; x-originating-ip: [69.165.175.208] x-microsoft-exchange-diagnostics: 1; CY1PR08MB1415; 5:C5P0uDFuHIvixuH9UAnVnBghLnM/VvdWERsMd6ePvNZaJ4j3WRLwGAqkUc+mukZuR5Qca2BgHeuLvr+Lx2NNgH6w8LIXsLM9VQsj6E905EsSl5FrxReYfLXdHpn5hztr+sJGFCqV3anroyeFJv0LbQ==; 24:yoX083EwkBY2HpcICdZyyPcH5kBIC53dplclw0nxTgVPDWo4BOCj06IQIcoh+NBfLicHFIJZSUHiz5mjnwWOoI5w1JTr0qGzJbyWiaSX5Sk=; 20:3JdLUyDdsJ9uz+NUuZGHf+JSbCSWo6sHU5HYyZr+eVWq2ozpssiZxJDTOJ3vxOBbV1qhJ8Iddg0Hc1cy4oemsA== x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(42134001)(42139001); SRVR:CY1PR08MB1415; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(601004)(2401047)(520078)(5005006)(8121501046)(3002001); SRVR:CY1PR08MB1415; BCL:0; PCL:0; RULEID:; SRVR:CY1PR08MB1415; x-forefront-prvs: 07326CFBC4 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(6009001)(377424004)(24454002)(479174004)(27574002)(18543002)(5403001)(199003)(10533003)(189002)(377454003)(66066001)(106356001)(50986999)(77096005)(122556002)(101416001)(64706001)(102836002)(5008740100001)(36756003)(103116003)(92566002)(81156007)(97736004)(40100003)(566704002)(11100500001)(4001150100001)(54356999)(5001770100001)(5002640100001)(99286002)(86362001)(450100001)(15975445007)(10400500002)(2900100001)(5007970100001)(100306002)(189998001)(105586002)(19580405001)(106116001)(5004730100002)(87936001)(5001960100002)(19580395003)(46102003)(107886002)(2501003)(491001); DIR:OUT; SFP:1101; SCL:1; SRVR:CY1PR08MB1415; H:CY1PR08MB1413.namprd08.prod.outlook.com; FPR:; SPF:None; PTR:InfoNoRecords; A:1; MX:1; LANG:en; received-spf: None (protection.outlook.com: metroland.com does not designate permitted sender hosts) spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-ID: <3A3AE62B261D8D488BA4BA0B468C4A47@torstarcorp.onmicrosoft.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Oct 2015 12:25:12.4124 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: e4142d33-522c-4393-ac12-a97843392e58 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY1PR08MB1415 X-Microsoft-Exchange-Diagnostics: 1; CY1PR08MB1989; 2:vBA9MjymWOGxjKIdQ2Za7zED32h3al3f75Hu2zIZPgrMxvF5D3riQdQtPb+cCtTUTc7twvEI19Zme4/ogv54UV5kBzgkdjgL55RCY1xkBohJCWDBHhUEqRmZmxVMIuCdic0H1PTmp9MzrE1d9RvqcIkJvGdeCyhHzk0yOMKIbo8=; 23:ZH60WuQ9BisyIlmZL7AZ50N/6TpYyujwlntajrjNgKuxd6UrcOGlYWXneiDX4okQZzXaA7qs/xUPVBLOdxtasO19Su1ciaeHchRCOniL20lrkKCLymmScVO6S9x6Y7SVoADtT/mE2AXNckuJz+8urKN999NDmf3KvTYZ3jgy3OtTqhCH3Oiik4SV7BcATzJy X-OriginatorOrg: metroland.com X-BeenThere: freebsd-rc@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: "Discussion related to /etc/rc.d design and implementation." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 17 Oct 2015 12:25:19 -0000 freebsd-rc mailing list submissions to freebsd-rc@freebsd.org To subscribe or unsubscribe via the World Wide Web, visit https://lists.freebsd.org/mailman/listinfo/freebsd-rc or, via email, send a message with subject or body 'help' to freebsd-rc-request@freebsd.org You can reach the person managing the list at freebsd-rc-owner@freebsd.org When replying, please edit your Subject line so it is more specific than "Re: Contents of freebsd-rc digest..." Today's Topics: 1. Re: rc(8) parallel tasks (Mark Felder) 2. Re: rc(8) parallel tasks (Cyril Vechera) 3. Re: rc(8) parallel tasks (Mark Martinec) ---------------------------------------------------------------------- Message: 1 Date: Fri, 16 Oct 2015 11:06:01 -0500 From: Mark Felder To: Cyril Vechera , dougb@FreeBSD.org, freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org Subject: Re: rc(8) parallel tasks Message-ID: <1445011561.1233840.412174489.688C5822@webmail.messagingengine.com> Content-Type: text/plain On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote: > Hi there. >=20 > We've got a small launcher script (~250 loc) for parallel services=20 > start/stop etc. It is used on our embedded systems and our users=20 > containers. And I've done a proof of concept for implanting it to the=20 > FreeBSD's standard /etc/rc for execution starting scripts in parallel.=20 > It gave me a boot time reduction of rc part from 27 to 7 seconds, mostly= =20 > on eliminating jams for network or other long-latency resources waiting. >=20 > The launcher is written in pure POSIX shell and uses FIFOs (named pipes)= =20 > as a mutexes for synchronization. So it is embedded into /etc/rc and=20 > /etc/rc.d preserving rc.subr preloading. As a primary requirement, it=20 > guarantees topological order (strict partial order) defined by=20 > dependencies. It requires only POSIX shell, FreeBSD or Linux kernel,=20 > mkfifo and a writeable file system. Due to last requirement, it can be=20 > run on the late stage or should be supplied by some kinf of writtable=20 > fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder=20 > annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to change=20 > rc.d scripts >=20 > It's not a full init replacement or a kind of services supervision tool.= =20 > It only starts or invokes a group of scripts in parallel with resolving=20 > and assuring execution in dependencies order. >=20 > Please take a look at the script and patch set for FreeBSD: >=20 > https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-in= it > https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd >=20 >=20 Your first link is a 404, but this looks really nice! --=20 Mark Felder ports-secteam member feld@FreeBSD.org ------------------------------ Message: 2 Date: Fri, 16 Oct 2015 21:37:27 +0300 From: Cyril Vechera To: Mark Felder Cc: marck@rinet.ru, freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org Subject: Re: rc(8) parallel tasks Message-ID: <562143E7.3030104@jet9.net> Content-Type: text/plain; charset=3DISO-8859-1; format=3Dflowed On 10/16/2015 07:06 PM, Mark Felder wrote: > On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote: >> Hi there. >> >> We've got a small launcher script (~250 loc) for parallel services >> start/stop etc. It is used on our embedded systems and our users >> containers. And I've done a proof of concept for implanting it to the >> FreeBSD's standard /etc/rc for execution starting scripts in parallel. >> It gave me a boot time reduction of rc part from 27 to 7 seconds, mostly >> on eliminating jams for network or other long-latency resources waiting. >> >> The launcher is written in pure POSIX shell and uses FIFOs (named pipes) >> as a mutexes for synchronization. So it is embedded into /etc/rc and >> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it >> guarantees topological order (strict partial order) defined by >> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel, >> mkfifo and a writeable file system. Due to last requirement, it can be >> run on the late stage or should be supplied by some kinf of writtable >> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder >> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to change >> rc.d scripts >> >> It's not a full init replacement or a kind of services supervision tool. >> It only starts or invokes a group of scripts in parallel with resolving >> and assuring execution in dependencies order. >> >> Please take a look at the script and patch set for FreeBSD: >> >> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-i= nit >> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebsd >> >> > Your first link is a 404, but this looks really nice! In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to=20 'jet9-multitask-flow' to avoid confusion with naming, because it's not=20 as it seems, from name, an init replacement, but just a parallel task=20 launcher. And now the actual repository URL is=20 https://github.com/cvss/jet9-multitask-flow In this commit I've complete the FreeBSD compatibility. Now a script or=20 dependency name can include minuses `-` and dots `.` (the first stone I=20 stumbled over was ftp-proxy). And I've cleaned up the main script code=20 https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow and have=20 split it to more functions that can be redefined. So it's now easier to=20 rewrite dependency extraction for FreeBSD rc-scripts - current=20 implementation is too rough and takes three 'awk' runs for each=20 rc-script. The last is not only time loss, but as DMarck mentioned=20 before, using awk restricts parallel rc to be run only after FILESYSTEMS=20 stage is done. Maybe it would be better to add to the rcorder(8) some=20 new option to dump the gathered dependencies in tsort-compatible listing=20 and insert them directly to flow execution plan. I've also added rc.conf variable `rc_parallel` to turn on and off=20 parallel execution. There's a risk to discover an incomplete dependency=20 annotations in some rc-scripts that earlier were masked by serial=20 execution. I've done some checks by enabling as much rc.conf variables=20 as possible to start more rc-scripts, and didn't found any error. But it=20 looks too good to be true and I'm afraid that it's just to poor testing.=20 I think if some ordering conflict will be found, it could be=20 worked-around with introducing a white-list for script names that must=20 be run only in sequentially. So it remained first to check if it really works in different conditions. --=20 Cyril Vechera ------------------------------ Message: 3 Date: Sat, 17 Oct 2015 00:14:01 +0200 From: Mark Martinec To: freebsd-hackers@freebsd.org, freebsd-rc@freebsd.org Cc: Cyril Vechera Subject: Re: rc(8) parallel tasks Message-ID: <2c307d32ddd8ed582ac496d56a75c3e4@mailbox.ijs.si> Content-Type: text/plain; charset=3DUS-ASCII; format=3Dflowed Tried a today's git checkout on one of our small virtual 10.2 boxes with not many services. The installation is almost trivial and a reboot went without any problems. Time for a reboot was shrunk by about 10 seconds (which is understandable with a small number of services). Very good, almost too good to be true for such fairly small and non-intrusive piece of shell/awk scripting - thanks! Mark 2015-10-16 20:37, Cyril Vechera wrote: > On 10/16/2015 07:06 PM, Mark Felder wrote: >> On Fri, Oct 2, 2015, at 11:08, Cyril Vechera wrote: >>> Hi there. >>>=20 >>> We've got a small launcher script (~250 loc) for parallel services >>> start/stop etc. It is used on our embedded systems and our users >>> containers. And I've done a proof of concept for implanting it to the >>> FreeBSD's standard /etc/rc for execution starting scripts in=20 >>> parallel. >>> It gave me a boot time reduction of rc part from 27 to 7 seconds,=20 >>> mostly >>> on eliminating jams for network or other long-latency resources=20 >>> waiting. >>>=20 >>> The launcher is written in pure POSIX shell and uses FIFOs (named=20 >>> pipes) >>> as a mutexes for synchronization. So it is embedded into /etc/rc and >>> /etc/rc.d preserving rc.subr preloading. As a primary requirement, it >>> guarantees topological order (strict partial order) defined by >>> dependencies. It requires only POSIX shell, FreeBSD or Linux kernel, >>> mkfifo and a writeable file system. Due to last requirement, it can=20 >>> be >>> run on the late stage or should be supplied by some kinf of writtable >>> fs, ie tmpfs. The FreeBSD-integrated version uses standard rcorder >>> annotations (REQUIRE, BEFORE and PROVIDE) and there's no need to=20 >>> change >>> rc.d scripts >>>=20 >>> It's not a full init replacement or a kind of services supervision=20 >>> tool. >>> It only starts or invokes a group of scripts in parallel with=20 >>> resolving >>> and assuring execution in dependencies order. >>>=20 >>> Please take a look at the script and patch set for FreeBSD: >>>=20 >>> https://github.com/cvss/jet9-multitask-init/blob/master/jet9-multitask-= init >>> https://github.com/cvss/jet9-multitask-init/tree/master/examples/freebs= d >>>=20 >>>=20 >> Your first link is a 404, but this looks really nice! >=20 > In the last commit (v1.3.0) I've renamed 'jet9-multitask-init' to > 'jet9-multitask-flow' to avoid confusion with naming, because it's not > as it seems, from name, an init replacement, but just a parallel task > launcher. And now the actual repository URL is > https://github.com/cvss/jet9-multitask-flow >=20 > In this commit I've complete the FreeBSD compatibility. Now a script > or dependency name can include minuses `-` and dots `.` (the first > stone I stumbled over was ftp-proxy). And I've cleaned up the main > script code > https://github.com/cvss/jet9-multitask-flow/jet9-multitask-flow and > have split it to more functions that can be redefined. So it's now > easier to rewrite dependency extraction for FreeBSD rc-scripts - > current implementation is too rough and takes three 'awk' runs for > each rc-script. The last is not only time loss, but as DMarck > mentioned before, using awk restricts parallel rc to be run only after > FILESYSTEMS stage is done. Maybe it would be better to add to the > rcorder(8) some new option to dump the gathered dependencies in > tsort-compatible listing and insert them directly to flow execution > plan. >=20 > I've also added rc.conf variable `rc_parallel` to turn on and off > parallel execution. There's a risk to discover an incomplete > dependency annotations in some rc-scripts that earlier were masked by > serial execution. I've done some checks by enabling as much rc.conf > variables as possible to start more rc-scripts, and didn't found any > error. But it looks too good to be true and I'm afraid that it's just > to poor testing. I think if some ordering conflict will be found, it > could be worked-around with introducing a white-list for script names > that must be run only in sequentially. >=20 > So it remained first to check if it really works in different=20 > conditions. ------------------------------ Subject: Digest Footer _______________________________________________ freebsd-rc@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-rc To unsubscribe, send any mail to "freebsd-rc-unsubscribe@freebsd.org" ------------------------------ End of freebsd-rc Digest, Vol 470, Issue 1 ******************************************