From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 09:51:03 2011 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 233DE1065676 for ; Sat, 11 Jun 2011 09:51:03 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id D89A38FC15 for ; Sat, 11 Jun 2011 09:51:02 +0000 (UTC) Received: by gxk28 with SMTP id 28so3156715gxk.13 for ; Sat, 11 Jun 2011 02:51:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:date:x-google-sender-auth :message-id:subject:from:to:content-type; bh=1DQOoLSH+7rtBM2VlISeNQHjZn4v3rzkaprpKDAOR1s=; b=V4HcMslT6V8bd0mkMnq90En+2NrISoEIknLid6XuFBOzQBhN/WZUBj6Wzaiobgrf8W iJc2wOBVWabQj4aLCpVCAbjZFghbHQG7Vl4jJcWxzqvsaKv9F5A0M37Y6GqZ/R7u7pN0 M0byMj1d2AXf3oalFUIdrIGzR+1XLUXiSKUK8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:content-type; b=dYRqCKZCf9r73nuVxB1DAkGAQxRieUCSnZ3/K1qt8ra4bBxz6Y3KivZI+FwHNKj2EK K4NIUyEHfhQ4q+Nh15o1t0fEED8N92ZEPWK6ezraqX8OebalU3s5ZHNy3Yj85Gr0dcD7 TP9Wsq2sv1uNEu0WM7fw7fbCiJptbQ1UKax5k= MIME-Version: 1.0 Received: by 10.150.74.17 with SMTP id w17mr4067589yba.274.1307784080815; Sat, 11 Jun 2011 02:21:20 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.150.216.3 with HTTP; Sat, 11 Jun 2011 02:21:20 -0700 (PDT) Date: Sat, 11 Jun 2011 17:21:20 +0800 X-Google-Sender-Auth: s-lWIVYwV8RHGjR_pTxYCyi053w Message-ID: From: Adrian Chadd To: freebsd-arch@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 09:51:03 -0000 Hi guys, Has there been any further thought as of late about shipping kernels with modules only by default, rather than monolithic kernels? I tried this experiment a couple years ago and besides a little trickery with ACPI module loading, it worked out fine. Is there any reason we aren't doing this at the moment? Eg by having a default loader modules list populated from the kernel config file? Adrian From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 11:14:13 2011 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F0CA7106566B; Sat, 11 Jun 2011 11:14:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 708E58FC15; Sat, 11 Jun 2011 11:14:13 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id p5BAhtq4061535 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 11 Jun 2011 13:43:56 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id p5BAhtTE017001; Sat, 11 Jun 2011 13:43:55 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id p5BAht6c017000; Sat, 11 Jun 2011 13:43:55 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 11 Jun 2011 13:43:55 +0300 From: Kostik Belousov To: Adrian Chadd Message-ID: <20110611104355.GH48734@deviant.kiev.zoral.com.ua> References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="qVdROpInvt/uIoBf" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: freebsd-arch@freebsd.org Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 11:14:14 -0000 --qVdROpInvt/uIoBf Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jun 11, 2011 at 05:21:20PM +0800, Adrian Chadd wrote: > Hi guys, >=20 > Has there been any further thought as of late about shipping kernels > with modules only by default, rather than monolithic kernels? >=20 > I tried this experiment a couple years ago and besides a little > trickery with ACPI module loading, it worked out fine. >=20 > Is there any reason we aren't doing this at the moment? Eg by having a > default loader modules list populated from the kernel config file? I use highly modularized kernel on all machines, because it allows me to use (almost) the same kernel config. Will you provide a prototype change for your suggestion ? --qVdROpInvt/uIoBf Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk3zRuoACgkQC3+MBN1Mb4h1PwCgz22Yo5a5yqOR+aIt/V+dJZYS IpIAn34gXyQckN0dZ+RUoFfRhoYp8FJf =C+6t -----END PGP SIGNATURE----- --qVdROpInvt/uIoBf-- From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 13:21:58 2011 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 991BB106564A for ; Sat, 11 Jun 2011 13:21:58 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-yw0-f54.google.com (mail-yw0-f54.google.com [209.85.213.54]) by mx1.freebsd.org (Postfix) with ESMTP id 545128FC0C for ; Sat, 11 Jun 2011 13:21:58 +0000 (UTC) Received: by ywf7 with SMTP id 7so1961301ywf.13 for ; Sat, 11 Jun 2011 06:21:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=J8RfFZxeRBiXukMi+xcXmnkf5Lop05qbncehztGARTc=; b=v9MjVY4aWWm6jw4DnMLqYkDMcPfnpf4fIuZOVx41diR/0ue5/nHbCmUYL04xmUcu1M WofPUcW95asBxODIOiGtZ5oXCO1LhNRSDYO1Z+oS5t3/4UazTehnUQN9GB94gp4N24te CANbIkFF0crfiuFjN2qc62gMwWGYpGYrtNTak= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; b=PiSSqUfU/9O7/o06NpIwWzW6NijSLlaNtQCYT+HFXWn+ZKPJ8e63/KQNE1crS/NtML 3QtaEt3ZHM+zw94e5LlvfCV6RaSyw2A94N4wcgQM0UiTNta0qCOP3nxj1/+2+KXBqJTG b+Y8zfI33zbYFOVzPLCNbbv54g3LQCNvxgHXo= MIME-Version: 1.0 Received: by 10.150.72.23 with SMTP id u23mr4622259yba.160.1307798517496; Sat, 11 Jun 2011 06:21:57 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.150.216.3 with HTTP; Sat, 11 Jun 2011 06:21:57 -0700 (PDT) In-Reply-To: <20110611104355.GH48734@deviant.kiev.zoral.com.ua> References: <20110611104355.GH48734@deviant.kiev.zoral.com.ua> Date: Sat, 11 Jun 2011 21:21:57 +0800 X-Google-Sender-Auth: -XPzLlbyTD1Ez5VPUj5qVRUPkhw Message-ID: From: Adrian Chadd To: Kostik Belousov Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-arch@freebsd.org Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 13:21:58 -0000 On 11 June 2011 18:43, Kostik Belousov wrote: >> Is there any reason we aren't doing this at the moment? Eg by having a >> default loader modules list populated from the kernel config file? > I use highly modularized kernel on all machines, because it allows me > to use (almost) the same kernel config. > > Will you provide a prototype change for your suggestion ? Sure, but in a couple weeks, or when I decide next to avoid study and need a break from if_ath hackery, whichever comes first. Thanks, Adrian From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 16:59:58 2011 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0B7261065672; Sat, 11 Jun 2011 16:59:58 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 9EDE38FC0C; Sat, 11 Jun 2011 16:59:57 +0000 (UTC) Received: from [10.0.0.63] (63.imp.bsdimp.com [10.0.0.63]) (authenticated bits=0) by harmony.bsdimp.com (8.14.4/8.14.3) with ESMTP id p5BGsxRn055543 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES128-SHA bits=128 verify=NO); Sat, 11 Jun 2011 10:55:02 -0600 (MDT) (envelope-from imp@bsdimp.com) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: Date: Sat, 11 Jun 2011 10:54:52 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <64005736-63ED-47DB-82BD-964F237AD933@bsdimp.com> References: <20110611104355.GH48734@deviant.kiev.zoral.com.ua> To: Adrian Chadd X-Mailer: Apple Mail (2.1084) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (harmony.bsdimp.com [10.0.0.6]); Sat, 11 Jun 2011 10:55:02 -0600 (MDT) Cc: Kostik Belousov , freebsd-arch@freebsd.org Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 16:59:58 -0000 On Jun 11, 2011, at 7:21 AM, Adrian Chadd wrote: > On 11 June 2011 18:43, Kostik Belousov wrote: >>> Is there any reason we aren't doing this at the moment? Eg by having = a >>> default loader modules list populated from the kernel config file? >> I use highly modularized kernel on all machines, because it allows me >> to use (almost) the same kernel config. >>=20 >> Will you provide a prototype change for your suggestion ? >=20 > Sure, but in a couple weeks, or when I decide next to avoid study and > need a break from if_ath hackery, whichever comes first. For 9, I'd suggest compiling MINIMAL config file that takes everything = out of GENERIC that can be a module. This would be the install kernel. = We'd create a loader.conf that lists everything currently in GENERIC. = This would give us most of the benefits and allow our users to edit = loader.conf to include the minimal set. For 10, I'd suggest that we'd standardize the driver data such that we = can determine from the plug and play strings what to load. This should = be done with devd, not the boot loader. The boot loader would need to = load all device's necessary to get / mounted. Robert suggested having an /etc/driverdb file. Unless that file is = automatically generated, it will be stale all the time. One last thing to note is that we'd only need to know which modules to = load from the plug and play strings. There's a few instances in the = tree where from the plug and play strings we'd need to load two = different modules (look at the re/rl attach routines for one instance of = why). Oh, the other last thing to consider is that the module system doesn't = have any notion of drivers that are attached (or even bound to device = nodes unattached). If we had some notion, it would be easier to = automatically unload the unnecessary drivers. Warner= From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 17:29:03 2011 Return-Path: Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CA0E5106564A; Sat, 11 Jun 2011 17:29:03 +0000 (UTC) (envelope-from das@FreeBSD.ORG) Received: from zim.MIT.EDU (ZIM.MIT.EDU [18.95.3.101]) by mx1.freebsd.org (Postfix) with ESMTP id 93E808FC0A; Sat, 11 Jun 2011 17:29:03 +0000 (UTC) Received: from zim.MIT.EDU (localhost [127.0.0.1]) by zim.MIT.EDU (8.14.4/8.14.2) with ESMTP id p5BHIZam038335; Sat, 11 Jun 2011 13:18:35 -0400 (EDT) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by zim.MIT.EDU (8.14.4/8.14.2/Submit) id p5BHIZIh038334; Sat, 11 Jun 2011 13:18:35 -0400 (EDT) (envelope-from das@FreeBSD.ORG) Date: Sat, 11 Jun 2011 13:18:35 -0400 From: David Schultz To: Adrian Chadd Message-ID: <20110611171834.GA38142@zim.MIT.EDU> Mail-Followup-To: Adrian Chadd , freebsd-arch@freebsd.org References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Cc: freebsd-arch@FreeBSD.ORG Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 17:29:03 -0000 On Sat, Jun 11, 2011, Adrian Chadd wrote: > Hi guys, > > Has there been any further thought as of late about shipping kernels > with modules only by default, rather than monolithic kernels? > > I tried this experiment a couple years ago and besides a little > trickery with ACPI module loading, it worked out fine. > > Is there any reason we aren't doing this at the moment? Eg by having a > default loader modules list populated from the kernel config file? I've been doing this for years, and it has come in quite handy. For instance, when my if_msk gets wedged, the only way to fix it short of rebooting seems to be reloading the driver. One issue, however, is that the boot loader is horrendously slow at loading modules. (Either that or my BIOS has a braindead int 13h handler.) Most of these modules aren't actually needed until much later in the boot process, so a mechanism to load non-essential modules after the file systems are mounted might provide a good solution. OS X has an interesting solution, intended to preserve the flexibility of dynamic modules, while minimizing boot time. It provides a kextcache utility, which packages the kernel and all of the needed modules into a single binary for better locality on disk. Unlike recompiling the kernel, running kextcache is fast, and the system runs it automatically when hardware or driver changes necessitate it. From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 18:34:39 2011 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id C81081065673; Sat, 11 Jun 2011 18:34:39 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from [127.0.0.1] (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 541F51A8D90; Sat, 11 Jun 2011 18:34:27 +0000 (UTC) Message-ID: <4DF3B532.6020908@FreeBSD.org> Date: Sat, 11 Jun 2011 11:34:26 -0700 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10 MIME-Version: 1.0 To: Adrian Chadd References: In-Reply-To: X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-arch@freebsd.org Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 18:34:39 -0000 On 6/11/2011 2:21 AM, Adrian Chadd wrote: > Hi guys, > > Has there been any further thought as of late about shipping kernels > with modules only by default, rather than monolithic kernels? > > I tried this experiment a couple years ago and besides a little > trickery with ACPI module loading, it worked out fine. > > Is there any reason we aren't doing this at the moment? Eg by having a > default loader modules list populated from the kernel config file? Has anyone benchmarked monolithic vs. modular? I think that should be done before we move in this direction. Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 18:56:59 2011 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 35E8E106566C for ; Sat, 11 Jun 2011 18:56:59 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-wy0-f182.google.com (mail-wy0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id B999C8FC12 for ; Sat, 11 Jun 2011 18:56:58 +0000 (UTC) Received: by wyf23 with SMTP id 23so3613650wyf.13 for ; Sat, 11 Jun 2011 11:56:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; bh=xrrfAG0oq7bRRZ6uxMFClqEybLnAhiuMiCuwg7GlSB4=; b=KXSOsFCGqDjA1Sll3szP0Y5JGxOB59ESq6CLjJYDcTLRn401wK4le/1w2u6fjDsy/y 9M5fIb79v8YNNrm6YVVpVBhPgdVr9dMqiES30DDNG2vi6o9rG4Gx+Q55gbr4n7b/MyZH 2ku7oLF8+OscruJAGM17xiamq0OCqSigofAiA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; b=U3PsMF1sBIShCvMT9ZgvmmXQzjM0F+q6uj25bb5D2pQiFAqTpRf0MLfv51IiKS9cZx aH4AgFGN5H7nh1FMy6CvgJGC52l0FSjEmjgnl3pSqT/YqYfDudcuvwCVIuvYfvnPNFOD 34b9EpGDgJneA1QqM8Qt87yu0fuPBWWQfWWC8= MIME-Version: 1.0 Received: by 10.216.237.155 with SMTP id y27mr912935weq.82.1307817151977; Sat, 11 Jun 2011 11:32:31 -0700 (PDT) Sender: mdf356@gmail.com Received: by 10.216.52.141 with HTTP; Sat, 11 Jun 2011 11:32:31 -0700 (PDT) In-Reply-To: <20110611171834.GA38142@zim.MIT.EDU> References: <20110611171834.GA38142@zim.MIT.EDU> Date: Sat, 11 Jun 2011 11:32:31 -0700 X-Google-Sender-Auth: BYYvdrmyoQrZtlvvhYDpJsw_4es Message-ID: From: mdf@FreeBSD.org To: Adrian Chadd , freebsd-arch@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 18:56:59 -0000 On Sat, Jun 11, 2011 at 10:18 AM, David Schultz wrote: > On Sat, Jun 11, 2011, Adrian Chadd wrote: >> Hi guys, >> >> Has there been any further thought as of late about shipping kernels >> with modules only by default, rather than monolithic kernels? >> >> I tried this experiment a couple years ago and besides a little >> trickery with ACPI module loading, it worked out fine. >> >> Is there any reason we aren't doing this at the moment? Eg by having a >> default loader modules list populated from the kernel config file? > > I've been doing this for years, and it has come in quite handy. > For instance, when my if_msk gets wedged, the only way to fix it > short of rebooting seems to be reloading the driver. > > One issue, however, is that the boot loader is horrendously slow > at loading modules. =A0(Either that or my BIOS has a braindead int 13h > handler.) =A0Most of these modules aren't actually needed until much > later in the boot process, so a mechanism to load non-essential > modules after the file systems are mounted might provide a good > solution. Indeed, at $WORK we're trying to get shutdown -> restart under 2 minutes. Several seconds of this is moving things *into* the kernel that need to be there (disk drivers), and everything else to a point in init where modules can be loaded in parallel, using the faster disk driver, rather than in serial with slow BIOS handlers. I'm sure some things about our solution will be generally applicable, but I'm not sure there's a obvious answer that works for GENERIC. Any disk driver that could have a root filesystem under it needs to be in the kernel (or, more slowly loaded as a module by the loader). But then there's a need maybe for network drivers in GENERIC to support PXE boot? And as soon as modules aren't loaded by loader or part of kernel, parts of init will break and need fixing, since they previously relied on modules already being loaded... > OS X has an interesting solution, intended to preserve the > flexibility of dynamic modules, while minimizing boot time. > It provides a kextcache utility, which packages the kernel > and all of the needed modules into a single binary for better > locality on disk. =A0Unlike recompiling the kernel, running > kextcache is fast, and the system runs it automatically when > hardware or driver changes necessitate it. This would be interesting -- isn't it essentially a re-link of the kernel + modules? Cheers, matthew From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 20:06:52 2011 Return-Path: Delivered-To: freebsd-arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 33378106566B; Sat, 11 Jun 2011 20:06:52 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id E5E818FC08; Sat, 11 Jun 2011 20:06:51 +0000 (UTC) Received: from [10.0.0.63] (63.imp.bsdimp.com [10.0.0.63]) (authenticated bits=0) by harmony.bsdimp.com (8.14.4/8.14.3) with ESMTP id p5BK0PZL056706 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES128-SHA bits=128 verify=NO); Sat, 11 Jun 2011 14:00:28 -0600 (MDT) (envelope-from imp@bsdimp.com) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <4DF3B532.6020908@FreeBSD.org> Date: Sat, 11 Jun 2011 14:00:20 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <4DF3B532.6020908@FreeBSD.org> To: Doug Barton X-Mailer: Apple Mail (2.1084) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (harmony.bsdimp.com [10.0.0.6]); Sat, 11 Jun 2011 14:00:28 -0600 (MDT) Cc: Adrian Chadd , freebsd-arch@FreeBSD.org Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 20:06:52 -0000 On Jun 11, 2011, at 12:34 PM, Doug Barton wrote: > On 6/11/2011 2:21 AM, Adrian Chadd wrote: >> Hi guys, >>=20 >> Has there been any further thought as of late about shipping kernels >> with modules only by default, rather than monolithic kernels? >>=20 >> I tried this experiment a couple years ago and besides a little >> trickery with ACPI module loading, it worked out fine. >>=20 >> Is there any reason we aren't doing this at the moment? Eg by having = a >> default loader modules list populated from the kernel config file? >=20 > Has anyone benchmarked monolithic vs. modular? I think that should be = done before we move in this direction. I haven't noticed a difference, but I haven't done any specific = benchmarking. Warner From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 20:17:12 2011 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BBB25106564A; Sat, 11 Jun 2011 20:17:12 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mail.zoral.com.ua (mx0.zoral.com.ua [91.193.166.200]) by mx1.freebsd.org (Postfix) with ESMTP id 364158FC0A; Sat, 11 Jun 2011 20:17:11 +0000 (UTC) Received: from deviant.kiev.zoral.com.ua (root@deviant.kiev.zoral.com.ua [10.1.1.148]) by mail.zoral.com.ua (8.14.2/8.14.2) with ESMTP id p5BKH35S008777 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 11 Jun 2011 23:17:04 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: from deviant.kiev.zoral.com.ua (kostik@localhost [127.0.0.1]) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4) with ESMTP id p5BKH3wD019296; Sat, 11 Jun 2011 23:17:03 +0300 (EEST) (envelope-from kostikbel@gmail.com) Received: (from kostik@localhost) by deviant.kiev.zoral.com.ua (8.14.4/8.14.4/Submit) id p5BKH33E019295; Sat, 11 Jun 2011 23:17:03 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: deviant.kiev.zoral.com.ua: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 11 Jun 2011 23:17:03 +0300 From: Kostik Belousov To: Warner Losh Message-ID: <20110611201703.GO48734@deviant.kiev.zoral.com.ua> References: <4DF3B532.6020908@FreeBSD.org> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="YbCBl//VW3xXyIiK" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-Virus-Scanned: clamav-milter 0.95.2 at skuns.kiev.zoral.com.ua X-Virus-Status: Clean X-Spam-Status: No, score=-3.3 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DNS_FROM_OPENWHOIS autolearn=no version=3.2.5 X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on skuns.kiev.zoral.com.ua Cc: Adrian Chadd , Doug Barton , freebsd-arch@freebsd.org Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 20:17:12 -0000 --YbCBl//VW3xXyIiK Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Sat, Jun 11, 2011 at 02:00:20PM -0600, Warner Losh wrote: >=20 > On Jun 11, 2011, at 12:34 PM, Doug Barton wrote: > > On 6/11/2011 2:21 AM, Adrian Chadd wrote: > >> Hi guys, > >>=20 > >> Has there been any further thought as of late about shipping kernels > >> with modules only by default, rather than monolithic kernels? > >>=20 > >> I tried this experiment a couple years ago and besides a little > >> trickery with ACPI module loading, it worked out fine. > >>=20 > >> Is there any reason we aren't doing this at the moment? Eg by having a > >> default loader modules list populated from the kernel config file? > >=20 > > Has anyone benchmarked monolithic vs. modular? I think that should be d= one before we move in this direction. >=20 > I haven't noticed a difference, but I haven't done any specific benchmark= ing. There might be some measurable difference on i386, where we use dso for modules. As a consequence, the overhead of GOT/PLT indirection, and, more important, stolen %ebx on the register-starved architecture, may make a difference. I doubt that any difference can be measured on amd64. --YbCBl//VW3xXyIiK Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (FreeBSD) iEYEARECAAYFAk3zzT8ACgkQC3+MBN1Mb4jOPQCfV62nloWb6iVteeZ5dGfMLLPJ BkoAn2hGp2Z6Qg6XxRbHa/I5Bzj6X8a3 =jA/5 -----END PGP SIGNATURE----- --YbCBl//VW3xXyIiK-- From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 20:43:27 2011 Return-Path: Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 70F02106564A; Sat, 11 Jun 2011 20:43:27 +0000 (UTC) (envelope-from das@FreeBSD.ORG) Received: from zim.MIT.EDU (ZIM.MIT.EDU [18.95.3.101]) by mx1.freebsd.org (Postfix) with ESMTP id 316A28FC0A; Sat, 11 Jun 2011 20:43:26 +0000 (UTC) Received: from zim.MIT.EDU (localhost [127.0.0.1]) by zim.MIT.EDU (8.14.4/8.14.2) with ESMTP id p5BKhQnS051715; Sat, 11 Jun 2011 16:43:26 -0400 (EDT) (envelope-from das@FreeBSD.ORG) Received: (from das@localhost) by zim.MIT.EDU (8.14.4/8.14.2/Submit) id p5BKhQCc051714; Sat, 11 Jun 2011 16:43:26 -0400 (EDT) (envelope-from das@FreeBSD.ORG) Date: Sat, 11 Jun 2011 16:43:26 -0400 From: David Schultz To: mdf@FreeBSD.ORG Message-ID: <20110611204326.GA51320@zim.MIT.EDU> Mail-Followup-To: mdf@freebsd.org, Adrian Chadd , freebsd-arch@freebsd.org References: <20110611171834.GA38142@zim.MIT.EDU> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Cc: Adrian Chadd , freebsd-arch@FreeBSD.ORG Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 20:43:27 -0000 On Sat, Jun 11, 2011, mdf@freebsd.org wrote: > On Sat, Jun 11, 2011 at 10:18 AM, David Schultz wrote: > > On Sat, Jun 11, 2011, Adrian Chadd wrote: > >> Hi guys, > >> > >> Has there been any further thought as of late about shipping kernels > >> with modules only by default, rather than monolithic kernels? > >> > >> I tried this experiment a couple years ago and besides a little > >> trickery with ACPI module loading, it worked out fine. > >> > >> Is there any reason we aren't doing this at the moment? Eg by having a > >> default loader modules list populated from the kernel config file? > > > > I've been doing this for years, and it has come in quite handy. > > For instance, when my if_msk gets wedged, the only way to fix it > > short of rebooting seems to be reloading the driver. > > > > One issue, however, is that the boot loader is horrendously slow > > at loading modules.  (Either that or my BIOS has a braindead int 13h > > handler.)  Most of these modules aren't actually needed until much > > later in the boot process, so a mechanism to load non-essential > > modules after the file systems are mounted might provide a good > > solution. > > Indeed, at $WORK we're trying to get shutdown -> restart under 2 > minutes. Several seconds of this is moving things *into* the kernel > that need to be there (disk drivers), and everything else to a point > in init where modules can be loaded in parallel, using the faster disk > driver, rather than in serial with slow BIOS handlers. Have you found that drivers can be reliably loaded in parallel these days? I'm always waiting for timeouts on four card readers and two optical drives, so that would be a big win for me. IIRC, nothing can happen in parallel during boot because the scheduler is initialized very late in the process. I'm not a device driver person, but I imagine there might be other assumptions that might get in the way as well. > > OS X has an interesting solution, intended to preserve the > > flexibility of dynamic modules, while minimizing boot time. > > It provides a kextcache utility, which packages the kernel > > and all of the needed modules into a single binary for better > > locality on disk.  Unlike recompiling the kernel, running > > kextcache is fast, and the system runs it automatically when > > hardware or driver changes necessitate it. > > This would be interesting -- isn't it essentially a re-link of the > kernel + modules? They actually go further than that and "pre-bind" the modules, essentially doing the work of the dynamic linker ahead of time. I have doubts that this trick saves much time. It also doesn't address Doug's concern about the overhead of the additional level of indirection needed for dynamic symbols; however, that overhead probably isn't significant either. From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 20:49:26 2011 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C3AC1106564A for ; Sat, 11 Jun 2011 20:49:26 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-vx0-f182.google.com (mail-vx0-f182.google.com [209.85.220.182]) by mx1.freebsd.org (Postfix) with ESMTP id 614218FC18 for ; Sat, 11 Jun 2011 20:49:26 +0000 (UTC) Received: by vxc34 with SMTP id 34so4212821vxc.13 for ; Sat, 11 Jun 2011 13:49:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; bh=fW+G/2qgiaqq8OdAazWg2UjmpNIqDIVnqV3tkrv9FQk=; b=B77NxgERzdi5TUURT1A0r8NCrPvZQrxyRZDgrCCPVgnXFTb7mOk9wZ7lFWz919ZROD xQGgbO9PaRSjj/FK8kWaN31aoAolelaMRzHzB2PctRrV5YdeEUcHSYiOQlrIgOGhmU8S 7KJmNWm/DZohu8zkdFyKd078n0P5TrzD8Igcw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; b=cQv+9TWynLmh3Jeg7AzaUkTdPIZIM8WQkjMb35/PPrxQiYA8P/dG6RPE/tmNLytfNC rHkTvhomKsM9QOGca0Rr4G9NmoPAvrjRfwRasaQkSb7EXwJdnixnJZVuk6KbEgEsk/HT bZrl5xMLxLCiZsV7tn6iMFX+HpGRY3PJiImlw= MIME-Version: 1.0 Received: by 10.52.177.234 with SMTP id ct10mr4700503vdc.2.1307825365549; Sat, 11 Jun 2011 13:49:25 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.52.187.74 with HTTP; Sat, 11 Jun 2011 13:49:25 -0700 (PDT) In-Reply-To: <20110611204326.GA51320@zim.MIT.EDU> References: <20110611171834.GA38142@zim.MIT.EDU> <20110611204326.GA51320@zim.MIT.EDU> Date: Sat, 11 Jun 2011 22:49:25 +0200 X-Google-Sender-Auth: qM6z3YWTvIUsGOv99CPtvfyXXDM Message-ID: From: "K. Macy" To: mdf@freebsd.org, freebsd-arch@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 20:49:26 -0000 >> Indeed, at $WORK we're trying to get shutdown -> restart under 2 >> minutes. =A0Several seconds of this is moving things *into* the kernel >> that need to be there (disk drivers), and everything else to a point >> in init where modules can be loaded in parallel, using the faster disk >> driver, rather than in serial with slow BIOS handlers. > > Have you found that drivers can be reliably loaded in parallel > these days? =A0I'm always waiting for timeouts on four card readers > and two optical drives, so that would be a big win for me. =A0IIRC, > nothing can happen in parallel during boot because the scheduler > is initialized very late in the process. =A0I'm not a device driver > person, but I imagine there might be other assumptions that might > get in the way as well. Although I imagine that many drivers silently benefit from being loaded serially, to the best of my knowledge there is nothing architecturally requiring this apart from the fact that the scheduler isn't started until everything else tied to initialization happens. The absence of any sort of preemption was a bit of a thorn in my side back when I was working on "xenbus", as the linux implementation relies on the use of multiple thread contexts. I don't know how much effort to date has been put in to making boot fast. Cheers, Kip From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 20:49:57 2011 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id E8F58106566C; Sat, 11 Jun 2011 20:49:57 +0000 (UTC) (envelope-from dougb@FreeBSD.org) Received: from [127.0.0.1] (hub.freebsd.org [IPv6:2001:4f8:fff6::36]) by mx2.freebsd.org (Postfix) with ESMTP id 24005157F87; Sat, 11 Jun 2011 20:49:47 +0000 (UTC) Message-ID: <4DF3D4EB.5030805@FreeBSD.org> Date: Sat, 11 Jun 2011 13:49:47 -0700 From: Doug Barton Organization: http://www.FreeBSD.org/ User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.17) Gecko/20110414 Thunderbird/3.1.10 MIME-Version: 1.0 To: Kostik Belousov References: <4DF3B532.6020908@FreeBSD.org> <20110611201703.GO48734@deviant.kiev.zoral.com.ua> In-Reply-To: <20110611201703.GO48734@deviant.kiev.zoral.com.ua> X-Enigmail-Version: 1.1.1 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Adrian Chadd , freebsd-arch@freebsd.org Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 20:49:58 -0000 On 6/11/2011 1:17 PM, Kostik Belousov wrote: > On Sat, Jun 11, 2011 at 02:00:20PM -0600, Warner Losh wrote: >> >> On Jun 11, 2011, at 12:34 PM, Doug Barton wrote: >>> On 6/11/2011 2:21 AM, Adrian Chadd wrote: >>>> Hi guys, >>>> >>>> Has there been any further thought as of late about shipping kernels >>>> with modules only by default, rather than monolithic kernels? >>>> >>>> I tried this experiment a couple years ago and besides a little >>>> trickery with ACPI module loading, it worked out fine. >>>> >>>> Is there any reason we aren't doing this at the moment? Eg by having a >>>> default loader modules list populated from the kernel config file? >>> >>> Has anyone benchmarked monolithic vs. modular? I think that should be done before we move in this direction. >> >> I haven't noticed a difference, but I haven't done any specific benchmarking. > > There might be some measurable difference on i386, where we use dso for > modules. As a consequence, the overhead of GOT/PLT indirection, and, more > important, stolen %ebx on the register-starved architecture, may make > a difference. I doubt that any difference can be measured on amd64. And no one has ever been surprised that assumptions proved invalid in the light of actual testing? :) Theorizing on this point is of less-than-zero utility. Who is going to volunteer to do the actual benchmarking? Doug -- Nothin' ever doesn't change, but nothin' changes much. -- OK Go Breadth of IT experience, and depth of knowledge in the DNS. Yours for the right price. :) http://SupersetSolutions.com/ From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 22:54:09 2011 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C557F106564A; Sat, 11 Jun 2011 22:54:09 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-ww0-f50.google.com (mail-ww0-f50.google.com [74.125.82.50]) by mx1.freebsd.org (Postfix) with ESMTP id 0A5FC8FC08; Sat, 11 Jun 2011 22:54:08 +0000 (UTC) Received: by wwc33 with SMTP id 33so3750710wwc.31 for ; Sat, 11 Jun 2011 15:54:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; bh=iPFC7eba6Pv7br9fE70/PUUo6ppNu3xXYXhWwUyCEMQ=; b=tv1Bk6EKyP5qwzPN8mDJfq1gmwHgBmA3PXZw009qm2+jgsfhpht0wVq5lWEGOduum/ kykZ+TjlE1EQ+DL2z0lz1n34orNBK/3sIBU7DSVZC+boxNwi6e5Lw0L4YN13PLS3fxl/ 16Wer9lsAtw0GelRMl9Ak5sMYkhXw9pcrCMiw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:content-type :content-transfer-encoding; b=lbw/GQfb4kAvASpBR6lpwbM+LCkW4Kj7G/mcMuphrI7rU0vEUQl5mWL29pfhS4nP9o UHwGBdxI2Vdhr6iMlA6TnDhkV76i20Ygb0rpCgEF13RLJZ5T0wXHIdGZFEfRDDHw8leI HWBRF2/F2zpzWn2X6HhrJwwJ+jg+xilmcgdbE= MIME-Version: 1.0 Received: by 10.216.209.212 with SMTP id s62mr3462667weo.67.1307832847902; Sat, 11 Jun 2011 15:54:07 -0700 (PDT) Sender: mdf356@gmail.com Received: by 10.216.52.141 with HTTP; Sat, 11 Jun 2011 15:54:07 -0700 (PDT) In-Reply-To: <20110611204326.GA51320@zim.MIT.EDU> References: <20110611171834.GA38142@zim.MIT.EDU> <20110611204326.GA51320@zim.MIT.EDU> Date: Sat, 11 Jun 2011 15:54:07 -0700 X-Google-Sender-Auth: l5SjN7t0-Dgbx4mmSwaQrxehpro Message-ID: From: mdf@FreeBSD.org To: mdf@freebsd.org, Adrian Chadd , freebsd-arch@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 22:54:10 -0000 On Sat, Jun 11, 2011 at 1:43 PM, David Schultz wrote: > On Sat, Jun 11, 2011, mdf@freebsd.org wrote: >> On Sat, Jun 11, 2011 at 10:18 AM, David Schultz wrote: >> > On Sat, Jun 11, 2011, Adrian Chadd wrote: >> >> Hi guys, >> >> >> >> Has there been any further thought as of late about shipping kernels >> >> with modules only by default, rather than monolithic kernels? >> >> >> >> I tried this experiment a couple years ago and besides a little >> >> trickery with ACPI module loading, it worked out fine. >> >> >> >> Is there any reason we aren't doing this at the moment? Eg by having = a >> >> default loader modules list populated from the kernel config file? >> > >> > I've been doing this for years, and it has come in quite handy. >> > For instance, when my if_msk gets wedged, the only way to fix it >> > short of rebooting seems to be reloading the driver. >> > >> > One issue, however, is that the boot loader is horrendously slow >> > at loading modules. =A0(Either that or my BIOS has a braindead int 13h >> > handler.) =A0Most of these modules aren't actually needed until much >> > later in the boot process, so a mechanism to load non-essential >> > modules after the file systems are mounted might provide a good >> > solution. >> >> Indeed, at $WORK we're trying to get shutdown -> restart under 2 >> minutes. =A0Several seconds of this is moving things *into* the kernel >> that need to be there (disk drivers), and everything else to a point >> in init where modules can be loaded in parallel, using the faster disk >> driver, rather than in serial with slow BIOS handlers. > > Have you found that drivers can be reliably loaded in parallel > these days? =A0I'm always waiting for timeouts on four card readers > and two optical drives, so that would be a big win for me. =A0IIRC, > nothing can happen in parallel during boot because the scheduler > is initialized very late in the process. =A0I'm not a device driver > person, but I imagine there might be other assumptions that might > get in the way as well. With the specific set we have at $WORK it seems fine. IIRC, Giant is held when the module's init function is called, but it's dropped during sleep so there's still some parallelism that can be achieved -- I would guess most driver init is essentially instant on fast modern CPUs except when waiting for device probes, etc. Cheers, matthew >> > OS X has an interesting solution, intended to preserve the >> > flexibility of dynamic modules, while minimizing boot time. >> > It provides a kextcache utility, which packages the kernel >> > and all of the needed modules into a single binary for better >> > locality on disk. =A0Unlike recompiling the kernel, running >> > kextcache is fast, and the system runs it automatically when >> > hardware or driver changes necessitate it. >> >> This would be interesting -- isn't it essentially a re-link of the >> kernel + modules? > > They actually go further than that and "pre-bind" the modules, > essentially doing the work of the dynamic linker ahead of time. > I have doubts that this trick saves much time. =A0It also doesn't > address Doug's concern about the overhead of the additional level > of indirection needed for dynamic symbols; however, that overhead > probably isn't significant either. > From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 23:09:30 2011 Return-Path: Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EB43F106564A; Sat, 11 Jun 2011 23:09:30 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id A665C8FC16; Sat, 11 Jun 2011 23:09:30 +0000 (UTC) Received: from [10.0.0.63] (63.imp.bsdimp.com [10.0.0.63]) (authenticated bits=0) by harmony.bsdimp.com (8.14.4/8.14.3) with ESMTP id p5BN8pUf057709 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES128-SHA bits=128 verify=NO); Sat, 11 Jun 2011 17:08:51 -0600 (MDT) (envelope-from imp@bsdimp.com) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: Date: Sat, 11 Jun 2011 17:08:45 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <9349A935-F13D-4265-A59C-C1E9B35F2B73@bsdimp.com> References: <20110611171834.GA38142@zim.MIT.EDU> <20110611204326.GA51320@zim.MIT.EDU> To: "K. Macy" X-Mailer: Apple Mail (2.1084) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (harmony.bsdimp.com [10.0.0.6]); Sat, 11 Jun 2011 17:08:51 -0600 (MDT) Cc: mdf@FreeBSD.ORG, freebsd-arch@FreeBSD.ORG Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 23:09:31 -0000 On Jun 11, 2011, at 2:49 PM, K. Macy wrote: >>> Indeed, at $WORK we're trying to get shutdown -> restart under 2 >>> minutes. Several seconds of this is moving things *into* the kernel >>> that need to be there (disk drivers), and everything else to a point >>> in init where modules can be loaded in parallel, using the faster = disk >>> driver, rather than in serial with slow BIOS handlers. >>=20 >> Have you found that drivers can be reliably loaded in parallel >> these days? I'm always waiting for timeouts on four card readers >> and two optical drives, so that would be a big win for me. IIRC, >> nothing can happen in parallel during boot because the scheduler >> is initialized very late in the process. I'm not a device driver >> person, but I imagine there might be other assumptions that might >> get in the way as well. >=20 > Although I imagine that many drivers silently benefit from being > loaded serially, to the best of my knowledge there is nothing > architecturally requiring this apart from the fact that the scheduler > isn't started until everything else tied to initialization happens. > The absence of any sort of preemption was a bit of a thorn in my side > back when I was working on "xenbus", as the linux implementation > relies on the use of multiple thread contexts. I don't know how much > effort to date has been put in to making boot fast. Right now newbus uses Giant for all its locking. That's the biggest = problem preventing parallel probe/attach. Also, each and every bus = calls probe, then calls attach for each device in sequence. Fixing that = would require changing all the bus drivers. Warner= From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 23:09:31 2011 Return-Path: Delivered-To: freebsd-arch@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 95A7F1065675; Sat, 11 Jun 2011 23:09:31 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from harmony.bsdimp.com (bsdimp.com [199.45.160.85]) by mx1.freebsd.org (Postfix) with ESMTP id 347C08FC18; Sat, 11 Jun 2011 23:09:31 +0000 (UTC) Received: from [10.0.0.63] (63.imp.bsdimp.com [10.0.0.63]) (authenticated bits=0) by harmony.bsdimp.com (8.14.4/8.14.3) with ESMTP id p5BN7SAx057707 (version=TLSv1/SSLv3 cipher=DHE-DSS-AES128-SHA bits=128 verify=NO); Sat, 11 Jun 2011 17:07:31 -0600 (MDT) (envelope-from imp@bsdimp.com) Mime-Version: 1.0 (Apple Message framework v1084) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <20110611204326.GA51320@zim.MIT.EDU> Date: Sat, 11 Jun 2011 17:07:24 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: <20110611171834.GA38142@zim.MIT.EDU> <20110611204326.GA51320@zim.MIT.EDU> To: David Schultz X-Mailer: Apple Mail (2.1084) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.0.1 (harmony.bsdimp.com [10.0.0.6]); Sat, 11 Jun 2011 17:07:31 -0600 (MDT) Cc: mdf@FreeBSD.ORG, Adrian Chadd , freebsd-arch@FreeBSD.ORG Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 23:09:31 -0000 On Jun 11, 2011, at 2:43 PM, David Schultz wrote: > On Sat, Jun 11, 2011, mdf@freebsd.org wrote: >> On Sat, Jun 11, 2011 at 10:18 AM, David Schultz = wrote: >>> On Sat, Jun 11, 2011, Adrian Chadd wrote: >>>> Hi guys, >>>>=20 >>>> Has there been any further thought as of late about shipping = kernels >>>> with modules only by default, rather than monolithic kernels? >>>>=20 >>>> I tried this experiment a couple years ago and besides a little >>>> trickery with ACPI module loading, it worked out fine. >>>>=20 >>>> Is there any reason we aren't doing this at the moment? Eg by = having a >>>> default loader modules list populated from the kernel config file? >>>=20 >>> I've been doing this for years, and it has come in quite handy. >>> For instance, when my if_msk gets wedged, the only way to fix it >>> short of rebooting seems to be reloading the driver. >>>=20 >>> One issue, however, is that the boot loader is horrendously slow >>> at loading modules. (Either that or my BIOS has a braindead int 13h >>> handler.) Most of these modules aren't actually needed until much >>> later in the boot process, so a mechanism to load non-essential >>> modules after the file systems are mounted might provide a good >>> solution. >>=20 >> Indeed, at $WORK we're trying to get shutdown -> restart under 2 >> minutes. Several seconds of this is moving things *into* the kernel >> that need to be there (disk drivers), and everything else to a point >> in init where modules can be loaded in parallel, using the faster = disk >> driver, rather than in serial with slow BIOS handlers. >=20 > Have you found that drivers can be reliably loaded in parallel > these days? I'm always waiting for timeouts on four card readers > and two optical drives, so that would be a big win for me. IIRC, > nothing can happen in parallel during boot because the scheduler > is initialized very late in the process. I'm not a device driver > person, but I imagine there might be other assumptions that might > get in the way as well. Loading isn't the problem. The timeouts that you are waiting for are = part of the probe/attach sequence. And that's strictly serialized... Warner= From owner-freebsd-arch@FreeBSD.ORG Sat Jun 11 23:51:02 2011 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 36DE4106566B; Sat, 11 Jun 2011 23:51:02 +0000 (UTC) (envelope-from kmacybsd@gmail.com) Received: from mail-vw0-f54.google.com (mail-vw0-f54.google.com [209.85.212.54]) by mx1.freebsd.org (Postfix) with ESMTP id CB5348FC0A; Sat, 11 Jun 2011 23:51:01 +0000 (UTC) Received: by vws18 with SMTP id 18so4263083vws.13 for ; Sat, 11 Jun 2011 16:51:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=7rfPbv1gEnspB/zo60Jf1ujTkOH0YftQuYQgCK/CRyE=; b=xSgJVhUiaIMcesYq9goEu6q+O70GotC1avm5B7GKD8/bdoahTnF4a3gFgspFPhTeOs 6vmBzcQai9e2ZSWOWcAoL7sBUFM8aGO3KBLnZMKVD4FCj6TnobVG/2KlUrB9Pipsy4yr gxCi0kuUtULK4phuowFWOtSdBtfo+8XQPCklY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; b=mtTx8YrJdoVi5Wjn76rEIx1cPXOR7FV7xVPZWFw+W8VNOCYe+JzF7rpOpLZo8BXB7f YGNJOT4xeJfGLlSUwBhOLOQQRLZn5uJ8Ctidv4CRRklLLXpjuqzgWm5RQK9bMQHIed2X lmVTa7mZzssNrB4K2rFNsfAR4w8wJg3Hslk7w= MIME-Version: 1.0 Received: by 10.52.176.74 with SMTP id cg10mr3789854vdc.242.1307836260910; Sat, 11 Jun 2011 16:51:00 -0700 (PDT) Sender: kmacybsd@gmail.com Received: by 10.52.187.74 with HTTP; Sat, 11 Jun 2011 16:51:00 -0700 (PDT) In-Reply-To: <9349A935-F13D-4265-A59C-C1E9B35F2B73@bsdimp.com> References: <20110611171834.GA38142@zim.MIT.EDU> <20110611204326.GA51320@zim.MIT.EDU> <9349A935-F13D-4265-A59C-C1E9B35F2B73@bsdimp.com> Date: Sun, 12 Jun 2011 01:51:00 +0200 X-Google-Sender-Auth: 3yj2vn0kBXotehl8MCcl7F8CC5M Message-ID: From: "K. Macy" To: Warner Losh Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: mdf@freebsd.org, freebsd-arch@freebsd.org Subject: Re: [RFC] shipping kernels with default modules? X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Jun 2011 23:51:02 -0000 >> Although I imagine that many drivers silently benefit from being >> loaded serially, to the best of my knowledge there is nothing >> architecturally requiring this apart from the fact that the scheduler >> isn't started until everything else tied to initialization happens. >> The absence of any sort of preemption was a bit of a thorn in my side >> back when I was working on "xenbus", as the linux implementation >> relies on the use of multiple thread contexts. I don't know how much >> effort to date has been put in to making boot fast. > > Right now newbus uses Giant for all its locking. =A0That's the biggest pr= oblem preventing parallel probe/attach. =A0Also, each and every bus calls p= robe, then calls attach for each device in sequence. =A0Fixing that would r= equire changing all the bus drivers. Fair enough. That would only be worthwhile in the presence of a coordinated push to shorten boot / reset times. Thanks