From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 00:07:57 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 596A1586 for ; Sun, 26 May 2013 00:07:57 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from smtpauth2.wiscmail.wisc.edu (wmauth2.doit.wisc.edu [144.92.197.222]) by mx1.freebsd.org (Postfix) with ESMTP id 34A28F88 for ; Sun, 26 May 2013 00:07:56 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from avs-daemon.smtpauth2.wiscmail.wisc.edu by smtpauth2.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) id <0MND00E00MEHB700@smtpauth2.wiscmail.wisc.edu> for freebsd-hackers@freebsd.org; Sat, 25 May 2013 18:07:50 -0500 (CDT) X-Spam-PmxInfo: Server=avs-2, Version=6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2013.5.25.230019, SenderIP=0.0.0.0 X-Spam-Report: AuthenticatedSender=yes, SenderIP=0.0.0.0 Received: from comporellon.tachypleus.net (adsl-76-208-69-84.dsl.mdsnwi.sbcglobal.net [76.208.69.84]) by smtpauth2.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) with ESMTPSA id <0MND0047UMX1FL20@smtpauth2.wiscmail.wisc.edu> for freebsd-hackers@freebsd.org; Sat, 25 May 2013 18:07:50 -0500 (CDT) Message-id: <51A14445.4060305@freebsd.org> Date: Sat, 25 May 2013 18:07:49 -0500 From: Nathan Whitehorn User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130410 Thunderbird/17.0.5 To: freebsd-hackers@freebsd.org Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> In-reply-to: <51A1025A.2020607@cran.org.uk> X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 00:07:57 -0000 On 05/25/13 13:26, Bruce Cran wrote: > On 25/05/2013 17:15, Matt Olander wrote: >> From my vague recollection, we discussed improving bsdinstall by tying >> it in with pc-sysinstall, which we've been threatening to do for at >> least a year. Also, there was much discussion about Devin's bsdconfig >> perhaps tying in with a Google SoC Project. >> >> I think Devin was nominated for most of the work, since he was unable >> to defend himself :P > > Thanks. From previous discussions with Devin I think he has other > plans for the installer that don't involve pc-sysinstall. But since it > seems the future is all sh(1) code, I won't be able to contribute. > > https://wiki.freebsd.org/PCBSDInstallMerge lists a few limitations > with pc-sysinstall - are these being fixed? > I'm not aware of any movement there (on either side of the table). I'd personally be very suspicious of an all-sh(1) future -- by far the cleanest parts of bsdinstall are in C -- and this is especially true for interacting with geom. That said, since I've lost nearly all of my free time and ability to work on bsdinstall, I won't get in the way of anyone else working on things -Nathan From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 01:01:28 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8E9E0399 for ; Sun, 26 May 2013 01:01:28 +0000 (UTC) (envelope-from erdgeist@erdgeist.org) Received: from elektropost.org (elektropost.org [217.13.206.130]) by mx1.freebsd.org (Postfix) with ESMTP id E0C071AE for ; Sun, 26 May 2013 01:01:27 +0000 (UTC) Received: (qmail 45642 invoked from network); 26 May 2013 01:01:19 -0000 Received: from elektropost.org (HELO elektropost.org) (erdgeist@erdgeist.org) by elektropost.org with CAMELLIA256-SHA encrypted SMTP; 26 May 2013 01:01:19 -0000 Message-ID: <51A15EDF.6050600@erdgeist.org> Date: Sun, 26 May 2013 03:01:19 +0200 From: Dirk Engling User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Nathan Whitehorn Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> In-Reply-To: <51A14445.4060305@freebsd.org> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 01:01:28 -0000 On 26.05.13 01:07, Nathan Whitehorn wrote: > I'm not aware of any movement there (on either side of the table). I'd > personally be very suspicious of an all-sh(1) future -- by far the > cleanest parts of bsdinstall are in C -- and this is especially true for > interacting with geom. That said, since I've lost nearly all of my free > time and ability to work on bsdinstall, I won't get in the way of anyone > else working on things As discussed at BSDCan, I'd be willing to participate in the development and at least implement setting up zpools/zfs and geli/gbde providers. I have done similar things in sh in my ezjail tools and think I can glue the rest together. Scanning through the pc-sysinstall code, I find nothing too fancy there regarding either interaction with zfs nor geom tools. I do not think it is necessary as a back end just for these features. Nathan, is there any design rationale available for the scripts, e.g. on why you chose sh versus C and were you provided with some kind of wish list/requirements in the first place? Any particular mail thread to scan through beforehand? Regards, erdgeist From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 02:51:18 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 51B6B369; Sun, 26 May 2013 02:51:18 +0000 (UTC) (envelope-from superbisquit@gmail.com) Received: from mail-oa0-f45.google.com (mail-oa0-f45.google.com [209.85.219.45]) by mx1.freebsd.org (Postfix) with ESMTP id 12B9A3DC; Sun, 26 May 2013 02:51:17 +0000 (UTC) Received: by mail-oa0-f45.google.com with SMTP id j6so7547044oag.4 for ; Sat, 25 May 2013 19:51:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=XIUnXez8Nvi+sGe/mrRXD9SyKCgMVjuuwcV4+kJSCR4=; b=e6ITqgfCIrKBdBlCStnZos2yREBzPRQUR+GoCksS/G7zajMV+sxWNhuNUtjKU8yo7p I6GK50zQ2acqeYeTfgsb5+hsUHkYIXbYLq1e0N7ixHkTHZuxajYbJTSeaBc+YAd8mfdI hnt6lfDM/Fw5E+V5WatyYrjsG1OTLD8obeM61YKcE0i2PY9xkB+V8fb3/1Gb+mDzmpBO w59pa4HAJ9waOyuxfizR1TteWNJ+xmFhinbPlCaz7xlYdXujckOQit8Hveia/YtwD4ID V78LTi26MPMRaXIensg7Rhrvk0UCKZ4L5PL+cCgQ0cVs5P7CJUBrt7LLwqqQuSJv1BSd EJSQ== MIME-Version: 1.0 X-Received: by 10.60.134.204 with SMTP id pm12mr15828373oeb.67.1369536676096; Sat, 25 May 2013 19:51:16 -0700 (PDT) Received: by 10.182.53.231 with HTTP; Sat, 25 May 2013 19:51:16 -0700 (PDT) In-Reply-To: <51A15EDF.6050600@erdgeist.org> References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> Date: Sat, 25 May 2013 22:51:16 -0400 Message-ID: Subject: Re: FreeBSD installers and future direction From: Super Bisquit To: Dirk Engling Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Hackers , Nathan Whitehorn X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 02:51:18 -0000 Please don't turn this into an architecture dependent mess. PCBSD is i386 & AMD64 only. On Sat, May 25, 2013 at 9:01 PM, Dirk Engling wrote: > On 26.05.13 01:07, Nathan Whitehorn wrote: > > > I'm not aware of any movement there (on either side of the table). I'd > > personally be very suspicious of an all-sh(1) future -- by far the > > cleanest parts of bsdinstall are in C -- and this is especially true for > > interacting with geom. That said, since I've lost nearly all of my free > > time and ability to work on bsdinstall, I won't get in the way of anyone > > else working on things > > As discussed at BSDCan, I'd be willing to participate in the development > and at least implement setting up zpools/zfs and geli/gbde providers. I > have done similar things in sh in my ezjail tools and think I can glue > the rest together. > > Scanning through the pc-sysinstall code, I find nothing too fancy there > regarding either interaction with zfs nor geom tools. I do not think it > is necessary as a back end just for these features. > > Nathan, is there any design rationale available for the scripts, e.g. on > why you chose sh versus C and were you provided with some kind of wish > list/requirements in the first place? Any particular mail thread to scan > through beforehand? > > Regards, > > erdgeist > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 02:58:54 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2129F486 for ; Sun, 26 May 2013 02:58:54 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id E4731401 for ; Sun, 26 May 2013 02:58:53 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.17]) by ltcfislmsgpa02.fnfis.com (8.14.5/8.14.5) with ESMTP id r4Q2weot018082 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sat, 25 May 2013 21:58:40 -0500 Received: from LTCFISWMSGMB26.FNFIS.com ([10.132.99.18]) by LTCFISWMSGHT06.FNFIS.com ([10.132.206.17]) with mapi id 14.02.0309.002; Sat, 25 May 2013 21:58:40 -0500 From: "Teske, Devin" To: "" Subject: Re: FreeBSD installers and future direction Thread-Topic: FreeBSD installers and future direction Thread-Index: AQHOWV60z6k72swSkkmGANUzy/EMLJkWZ1wAgACzkIA= Date: Sun, 26 May 2013 02:58:40 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F5B088@ltcfiswmsgmb26> References: <51A0DC3F.9030301@cran.org.uk> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] Content-Type: text/plain; charset="Windows-1252" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-25_07:2013-05-24,2013-05-25,1970-01-01 signatures=0 Cc: Bruce Cran , "" , "Teske, Devin" , Kris Moore X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 02:58:54 -0000 On May 25, 2013, at 9:15 AM, Matt Olander wrote: > On Sat, May 25, 2013 at 8:43 AM, Bruce Cran wrote: >>=20 >> I heard there was some discussion at BSDCan about the direction of a fut= ure FreeBSD installer. Considering we currently have bsdinstall, pc-sysins= tall, and an effort to revive sysinstall, I'd be interested to know what wa= s decided (if anything) and whether I could help make progress towards gett= ing a single really good installer/frontend - instead of the current situat= ion with several, none of which have a much-needed UI for setting up an ins= tallation on ZFS. >=20 > Hey Bruce, >=20 > From my vague recollection, we discussed improving bsdinstall by tying > it in with pc-sysinstall, which we've been threatening to do for at > least a year. Also, there was much discussion about Devin's bsdconfig > perhaps tying in with a Google SoC Project. >=20 There is indeed a GSoC project to address the situation w/regard to install= ers. The GSoC project is one submitted by Harsh Bhatt and I've submitted that I = wish to Mentor. The project is titled: "Making pc-sysinstall FreeBSD ready by porting it to multiple architectures" The project is designed as such to increase possibilities w/respect to inst= allers. To elaborate=85 Integrating bsdinstall with pc-sysinstall would introduce a regression beca= use bsdinstall currently supports all platforms, whereas pc-sysinstall has = hard-coded assumptions specific to the x86 platform (i386 and amd64 include= d). If bsdinstall was ever going to be programmed to use pc-sysinstall as a= back-end, pc-sysinstall would have to be cleaned up. I'm in no way pushing to integrate bsdinstall with pc-sysinstall=85 but I *= am* saying this is a GSoC project to look out for. The goal of the project = is to see if it is even possible to remedy any possibility that tying bsdin= stall to pc-sysinstall would introduce a regression in platform support. If= the project is successful, PC-BSD should be able to immediately benefit fr= om the fruits thereof (as -- correct me if I'm wrong -- the graphical PC-BS= D installer uses pc-sysinstall as a back-end). ASIDE: For what it's worth, it's somewhat convenient to think that in a sim= ple world -- because pc-sysinstall is already the backend of the PC-BSD GUI= installer -- FreeBSD would have bsdinstall as the TUI front-end to the sam= e back-end. But however, I'm not na=EFve and can't imagine that as being cl= ean or elegant unless we can clean up pc-sysinstall. Cleaning up the code i= s a another major focal-point of the aforementioned GSoC project. With respect to my bsdconfig=85 (how it relates to installers) Just as there were blockers preventing pc-sysinstall from being integrated = into bsdinstall. There are blockers that prevent bsdinstall from being integrated into the l= arger bsdconfig framework. In the case of pc-sysinstall integrating to bsdinstall, pc-sysinstall doesn= 't support all the target architectures that bsdinstall does. Meanwhile, in= the case of bsdinstall integrating into bsdconfig, bsdinstall doesn't supp= ort all the features of bsdconfig. Luckily, introduction of most features to bring bsdinstall on-par with bsdc= onfig will be easy as we can just make bsdinstall use the subroutine includ= es from bsdconfig. However, there are other things that just can't be done = without plain sweat and labor=85 For example, making bsdinstall i18n-ready (as bsdconfig is). I expect this = to take (with a medium effort) a week or two, aided by the fact that the di= alog(1) abstraction API offered by bsdconfig makes most of that transparent= (for example, you don't have to worry about whether the "Yes" button says = "Ja", "Oui", or "Yes" -- you just call f_dialog_yesno() and it worried abou= t how to call dialog(1) to form the proper i18n-compatible prompt). Another big change to bsdinstall would be to make it *more* modular and rew= rite the C components to be in sh. > I think Devin was nominated for most of the work, since he was unable > to defend himself :P >=20 I don't mind being volunteered. In earnest, I figure the code I'm working o= n is volunteering enough. --=20 Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 03:08:04 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 71A3E6DE; Sun, 26 May 2013 03:08:04 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 4291078F; Sun, 26 May 2013 03:08:03 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.15]) by ltcfislmsgpa02.fnfis.com (8.14.5/8.14.5) with ESMTP id r4Q37v2N027666 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sat, 25 May 2013 22:07:57 -0500 Received: from LTCFISWMSGMB26.FNFIS.com ([10.132.99.18]) by LTCFISWMSGHT04.FNFIS.com ([10.132.206.15]) with mapi id 14.02.0309.002; Sat, 25 May 2013 22:08:25 -0500 From: "Teske, Devin" To: Bruce Cran Subject: Re: FreeBSD installers and future direction Thread-Topic: FreeBSD installers and future direction Thread-Index: AQHOWV60z6k72swSkkmGANUzy/EMLJkWZ1wAgAAkfQCAAJGrAA== Date: Sun, 26 May 2013 03:07:56 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F5B0FB@ltcfiswmsgmb26> References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> In-Reply-To: <51A1025A.2020607@cran.org.uk> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] Content-Type: text/plain; charset="Windows-1252" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-25_07:2013-05-24,2013-05-25,1970-01-01 signatures=0 Cc: FreeBSD Hackers , Devin Teske X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 03:08:04 -0000 On May 25, 2013, at 11:26 AM, Bruce Cran wrote: > On 25/05/2013 17:15, Matt Olander wrote: >> From my vague recollection, we discussed improving bsdinstall by tying >> it in with pc-sysinstall, which we've been threatening to do for at >> least a year. Also, there was much discussion about Devin's bsdconfig >> perhaps tying in with a Google SoC Project. >>=20 >> I think Devin was nominated for most of the work, since he was unable >> to defend himself :P >=20 > Thanks. From previous discussions with Devin I think he has other plans f= or the installer that don't involve pc-sysinstall. But since it seems the f= uture is all sh(1) code, I won't be able to contribute. >=20 The future of how these softwares become entangled to produce something bet= ter: + bsdinstall + pc-sysinstall + bsdconfig Is in my mind entirely still open as I work to finish up bsdconfig. I was thinking=85 Perhaps just rewrite bsdinstall from scratch (using the existing code as a = template). Have to put pc-sysinstall on the back-burner for any/all considerations unt= il the code is cleaned up and actually usable on all architectures (there's= a GSoC project to do exactly that -- I'm the potential mentor; project is = pending status). So unless this GSoC goes through and we are able to (as we plan) clean up t= hat mess=85 Defacto plan is to just rewrite bsdinstall from scratch (again=85 using the= existing code as a template). In said rewrite=85 I'd *heavily* leverate usr.sbin/bsdconfig/share/*.subr (= specifically "dialog.subr" -- the abstraction layer that allows me to have = nice i18n-ready dialogs and also gracefully handle dialog in a way that mak= es my code look very clean). > https://wiki.freebsd.org/PCBSDInstallMerge lists a few limitations with p= c-sysinstall - are these being fixed? >=20 I can see if the GSoC student for the "Making pc-sysinstall FreeBSD ready b= y porting it to multiple architectures" project is willing to incorporate a= ny of those limitations into his work. But I think the focus of the project= should be to clean up the code and make it work on at least one other impo= rtant non-x86 architecture. Barring that=85 You're absolutely right in stating that any/all discussions on merging pc-s= ysinstall with bsdinstall would result in a regression with pc-sysinstall i= n its current state. --=20 Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 03:12:04 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8B2607DF; Sun, 26 May 2013 03:12:04 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 58C737AE; Sun, 26 May 2013 03:12:04 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.15]) by ltcfislmsgpa06.fnfis.com (8.14.5/8.14.5) with ESMTP id r4Q3C0d1014811 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sat, 25 May 2013 22:12:00 -0500 Received: from LTCFISWMSGMB26.FNFIS.com ([10.132.99.18]) by LTCFISWMSGHT04.FNFIS.com ([10.132.206.15]) with mapi id 14.02.0309.002; Sat, 25 May 2013 22:12:12 -0500 From: "Teske, Devin" To: Nathan Whitehorn Subject: Re: FreeBSD installers and future direction Thread-Topic: FreeBSD installers and future direction Thread-Index: AQHOWV60z6k72swSkkmGANUzy/EMLJkWZ1wAgAAkfQCAAE6UgIAARDoA Date: Sun, 26 May 2013 03:12:00 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F5B146@ltcfiswmsgmb26> References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> In-Reply-To: <51A14445.4060305@freebsd.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] Content-Type: text/plain; charset="Windows-1252" Content-ID: <4DE6D26514B1814582C6830C90E9740E@fisglobal.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-25_07:2013-05-24,2013-05-25,1970-01-01 signatures=0 Cc: FreeBSD Hackers , Devin Teske X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 03:12:04 -0000 On May 25, 2013, at 4:07 PM, Nathan Whitehorn wrote: > On 05/25/13 13:26, Bruce Cran wrote: >> On 25/05/2013 17:15, Matt Olander wrote: >>> From my vague recollection, we discussed improving bsdinstall by tying >>> it in with pc-sysinstall, which we've been threatening to do for at >>> least a year. Also, there was much discussion about Devin's bsdconfig >>> perhaps tying in with a Google SoC Project. >>>=20 >>> I think Devin was nominated for most of the work, since he was unable >>> to defend himself :P >>=20 >> Thanks. From previous discussions with Devin I think he has other plans = for the installer that don't involve pc-sysinstall. But since it seems the = future is all sh(1) code, I won't be able to contribute. >>=20 >> https://wiki.freebsd.org/PCBSDInstallMerge lists a few limitations with = pc-sysinstall - are these being fixed? >>=20 >=20 > I'm not aware of any movement there (on either side of the table). I'd pe= rsonally be very suspicious of an all-sh(1) future -- by far the cleanest p= arts of bsdinstall are in C -- and this is especially true for interacting = with geom. So that I can get a feel for the expectations on quality of product=85 You say the cleanest parts of bsdinstall are in C. What makes the sh parts unclean? Not looking for any particular answer=85 j= ust want your opinion. I've been working very hard to produce a heavy-lifting "dialog.subr" (see b= ase/head/usr.sbin/bsdconfig/share/dialog.subr) which makes dialog(1) work v= ery nice and clean in sh(1). However, this may not be your concern, and you= may instead have some other "cleanliness" insight that I can address in my= work. --=20 Devin > That said, since I've lost nearly all of my free time and ability to work= on bsdinstall, I won't get in the way of anyone else working on things > -Nathan > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 03:43:08 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 85417AA6; Sun, 26 May 2013 03:43:08 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 526BF87F; Sun, 26 May 2013 03:43:07 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.15]) by ltcfislmsgpa01.fnfis.com (8.14.5/8.14.5) with ESMTP id r4Q3gxVo016440 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sat, 25 May 2013 22:42:59 -0500 Received: from LTCFISWMSGMB26.FNFIS.com ([10.132.99.18]) by LTCFISWMSGHT04.FNFIS.com ([10.132.206.15]) with mapi id 14.02.0309.002; Sat, 25 May 2013 22:42:59 -0500 From: "Teske, Devin" To: Dirk Engling Subject: Re: FreeBSD installers and future direction Thread-Topic: FreeBSD installers and future direction Thread-Index: AQHOWV60z6k72swSkkmGANUzy/EMLJkWZ1wAgAAkfQCAAE6UgIAAH7eAgAAtKgA= Date: Sun, 26 May 2013 03:42:58 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F5B2E7@ltcfiswmsgmb26> References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> In-Reply-To: <51A15EDF.6050600@erdgeist.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] Content-Type: text/plain; charset="Windows-1252" Content-ID: <5DBEBE4CE4BA274A955546731DBE4B7C@fisglobal.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-25_07:2013-05-24,2013-05-25,1970-01-01 signatures=0 Cc: FreeBSD Hackers , Devin Teske , Nathan Whitehorn X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 03:43:08 -0000 On May 25, 2013, at 6:01 PM, Dirk Engling wrote: > On 26.05.13 01:07, Nathan Whitehorn wrote: >=20 >> I'm not aware of any movement there (on either side of the table). I'd >> personally be very suspicious of an all-sh(1) future -- by far the >> cleanest parts of bsdinstall are in C -- and this is especially true for >> interacting with geom. That said, since I've lost nearly all of my free >> time and ability to work on bsdinstall, I won't get in the way of anyone >> else working on things >=20 > As discussed at BSDCan, I'd be willing to participate in the development > and at least implement setting up zpools/zfs and geli/gbde providers. I > have done similar things in sh in my ezjail tools and think I can glue > the rest together. >=20 Very Cool! I do encourage you to take a peak at bsdconfig either from HEAD or from por= ts (in sysutils)=85 However, I'm getting ready to drastically change the API very soon (as prev= iously threatened). I'm in the home-stretch and I'm trying to get in all th= e API changes before broaching the idea of taking off the "WITH_BSDCONFIG" = build-option requirement. > Scanning through the pc-sysinstall code, I find nothing too fancy there > regarding either interaction with zfs nor geom tools. I do not think it > is necessary as a back end just for these features. >=20 I tend to agree. In fact=85 the directory structure of pc-sysinstall and even the way it sto= res the subroutines in *.sh files means I would be compelled to restructure= the thing into proper "includes". Since I intend to rework bsdinstall to use bsdconfig's libraries (e.g. "dia= log.subr"), it sounds like a really nice path would be to develop: base/head/usr.sbin/bsdconfig/share/zfs.subr base/head/usr.sbin/bsdconfig/share/geom/ base/head/usr.sbin/bsdconfig/share/geom/geli.subr And then bsdinstall could include those. The reason I tell myself that it w= ould be better if they lived in bsdconfig and got imported at runtime by bs= dinstall (rather than living in bsdinstall) is that I want to move to brand= the utilities in this way: bsdinstall is the program designed to install things (bare metal, jails, et= c.) bsdconfig is the program designed to configure things either during install= ation time or post-installation time bsdinstall would be used on the boot media to install bare metal (but durin= g that installation, made use bsdconfig for things like configuring users, = boot services, networking, etc.). It will also be used after installation a= nd running in multi-user land to do things like install jails. So in other words=85 anytime something could conceivably be wanted by bsdco= nfig=85 we should just birth it there and have bsdinstall reach out for it.= Seems to make sense=85 a person installs once but configures many times. Guess I'm saying of course, that there'd be a great use for a zfs and geli = library to bsdconfig for managing zfs after booting, etc. > Nathan, is there any design rationale available for the scripts, e.g. on > why you chose sh versus C and were you provided with some kind of wish > list/requirements in the first place? Any particular mail thread to scan > through beforehand? >=20 Can't answer for Nathan (and not sure if you want my opinion), but=85 I chose 100% sh for bsdconfig because of a few good reasons=85 I ultimately wanted to (and did) make it scriptable. The scripts are actual= ly `sourced' shell scripts. So=85 Because bsdconfig is sh, and the script is sh, the script has access to all= of the bsdconfig internals, every API call, every variable, and can do nea= rly anything. There are no frustrating moments where a C aspect doesn't sup= port running in a desired mode because that particular aspect was not made = configurable or tunable. Not that this was (or is) a problem at all current= ly=85 just that it *was* a problem with sysinstall. For example=85 sysinstall had a deviceRescan() function but the dispatch.c system did not = make *that* particular function available through the system of hard-coded = resWords (reserved words) that were allowed in a script. So=85 if you wante= d to have a sysinstall script that (a) formatted some disks with some slice= s and (b) then formatted those slices with some BSD labels =85 well=85 you = couldn't (note: this is not to say couldn't get sysinstall to write both sl= ices and labels, just that if you wanted to do this *OUT* of the context of= a full install, you couldn't). The problem was that you really needed to c= all "deviceRescan" after doing the slice action on the bare metal so that s= ysinstall would pick up the "s1" slice on which you wanted to later write l= abels on. In the case of sysinstall=85 a one-line change of adding the deviceRescan()= function to the resWord mapping in dispatch.c would have made that functio= nality available to scripts. bsdconfig doesn't have that problem because the scripts are actually shell = scripts and the old sysinstall commands are replicated by shell functions. = There is never any need to "map" a function before it becomes available to = the scripting back-end. Any/all functions are simply available *AND* (heh, = unlike sysinstall) you can pass arguments and do anything else you can imag= ine in shell. --=20 Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 03:45:27 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D39DAB98; Sun, 26 May 2013 03:45:27 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id A0243891; Sun, 26 May 2013 03:45:27 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.17]) by ltcfislmsgpa01.fnfis.com (8.14.5/8.14.5) with ESMTP id r4Q3jMVs019033 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sat, 25 May 2013 22:45:22 -0500 Received: from LTCFISWMSGMB26.FNFIS.com ([10.132.99.18]) by LTCFISWMSGHT06.FNFIS.com ([10.132.206.17]) with mapi id 14.02.0309.002; Sat, 25 May 2013 22:45:22 -0500 From: "Teske, Devin" To: Super Bisquit Subject: Re: FreeBSD installers and future direction Thread-Topic: FreeBSD installers and future direction Thread-Index: AQHOWV60z6k72swSkkmGANUzy/EMLJkWZ1wAgAAkfQCAAE6UgIAAH7eAgAAeuACAAA8cgA== Date: Sun, 26 May 2013 03:45:21 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] Content-Type: text/plain; charset="Windows-1252" Content-ID: <3F46DE392190CE458006DEF689110FD5@fisglobal.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-25_07:2013-05-24,2013-05-25,1970-01-01 signatures=0 Cc: Devin Teske , FreeBSD Hackers , Dirk Engling , Nathan Whitehorn X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 03:45:27 -0000 On May 25, 2013, at 7:51 PM, Super Bisquit wrote: > Please don't turn this into an architecture dependent mess. PCBSD is i386= & > AMD64 only. >=20 There's a GSoC project (of which I'm potential mentor) to fix that. However, you are entirely right=85 we can't in all seriousness even think a= bout using pc-sysinstall until it is solid on all architectures as bsdinsta= ll already is. GSoC project is: "Making pc-sysinstall FreeBSD ready by porting it to multi= ple architectures" --=20 Devin >=20 > On Sat, May 25, 2013 at 9:01 PM, Dirk Engling wro= te: >=20 >> On 26.05.13 01:07, Nathan Whitehorn wrote: >>=20 >>> I'm not aware of any movement there (on either side of the table). I'd >>> personally be very suspicious of an all-sh(1) future -- by far the >>> cleanest parts of bsdinstall are in C -- and this is especially true for >>> interacting with geom. That said, since I've lost nearly all of my free >>> time and ability to work on bsdinstall, I won't get in the way of anyone >>> else working on things >>=20 >> As discussed at BSDCan, I'd be willing to participate in the development >> and at least implement setting up zpools/zfs and geli/gbde providers. I >> have done similar things in sh in my ezjail tools and think I can glue >> the rest together. >>=20 >> Scanning through the pc-sysinstall code, I find nothing too fancy there >> regarding either interaction with zfs nor geom tools. I do not think it >> is necessary as a back end just for these features. >>=20 >> Nathan, is there any design rationale available for the scripts, e.g. on >> why you chose sh versus C and were you provided with some kind of wish >> list/requirements in the first place? Any particular mail thread to scan >> through beforehand? >>=20 >> Regards, >>=20 >> erdgeist >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.or= g" >>=20 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 06:30:40 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4AEB962C for ; Sun, 26 May 2013 06:30:40 +0000 (UTC) (envelope-from vhaisman@gmail.com) Received: from mail-ee0-f49.google.com (mail-ee0-f49.google.com [74.125.83.49]) by mx1.freebsd.org (Postfix) with ESMTP id D40F2B3F for ; Sun, 26 May 2013 06:30:39 +0000 (UTC) Received: by mail-ee0-f49.google.com with SMTP id d17so3381047eek.8 for ; Sat, 25 May 2013 23:30:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:subject:references :in-reply-to:x-enigmail-version:content-type; bh=nJu/U1v3f9yL8quHLYe1jVTFL61wOHS0JMCtB+Ytab4=; b=c3niv5Qgi/lSg5ABec0yhJGK7O7MvxQO9LEHWpQUAvOItxuY6k22MDM9jFbTbppgqF CYVNtY++DVC/ZmIg5A1FetSCj8Um8OOGqHSp5sZn6hJhfa5Y0SpFksb74z53Y3f/SXvN jHJILheKjLtipxE7OkurzcaIqMf1fQiVSlrZbwc+1qgGm6Qw6D8YPKlK7P4rdddE9ig4 r5hjQHVvk3qkrM7/BHZpeUbRZyRRWCHdhAt8117SP9nxvcvvgueEckZ5HPtTzCgt4mCx /pwW8fJ7wViniNgpWUCPKBQEnVIgexl+5g8qXGgePJ4LmTNl5Gp6C5WivV1ZOJSHAJYQ H66Q== X-Received: by 10.14.177.130 with SMTP id d2mr5823410eem.133.1369549411267; Sat, 25 May 2013 23:23:31 -0700 (PDT) Received: from [10.0.0.1] (242.91.broadband5.iol.cz. [88.100.91.242]) by mx.google.com with ESMTPSA id l6sm33759496eef.12.2013.05.25.23.23.29 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Sat, 25 May 2013 23:23:30 -0700 (PDT) Message-ID: <51A1AA60.9050209@gmail.com> Date: Sun, 26 May 2013 08:23:28 +0200 From: =?ISO-8859-1?Q?V=E1clav_Zeman?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130510 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: Performance improvement to strnlen(). References: In-Reply-To: X-Enigmail-Version: 1.4.6 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="------------enig1FEBB2302363A6AC4C0260A6" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 06:30:40 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enig1FEBB2302363A6AC4C0260A6 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 05/25/2013 10:27 PM, Lee Thomas wrote: > Hello FreeBSD devs, >=20 > I have found a performance improvement to libc's strnlen(). > lib/libc/string/strnlen.c is a trivial byte-by-byte implementation, > where strlen.c has a smarter word-by-word implementation. I have > implemented strnlen similarly to strlen. It runs about 4x as fast, at > the cost of a binary codesize increase from 30 bytes to 221. >=20 > In writing this I needed a test, and there isn't one in > tools/regression/lib/libc/string, so I wrote a unit test for strnlen. > This really only makes sense for a word-by-word implementation, as it > tests each combination of string and limit length, overread characters,= > and starting alignment. >=20 > Could someone please review these patches? I am not very experienced in= > C, and am even less experienced with FreeBSD's style guidelines, so the= y > likely have a bunch of style and idiom issues. Even if one or both of > them prove not worth committing, it would still be useful for my learni= ng. >=20 > If/When these patches prove worthy of submitting, what is the next step= > after that? Should I submit a PR, or is there some other process? This > code is quite similar to the existing strlen.c, and doesn't do anything= > fancy with e.g. endianness, but I haven't tested it for correctness or > performance on anything other than x86... >=20 > And finally, there is some other low-hanging fruit in the other strn* > functions. Would it be worth it for me to give those the same treatment= ? >=20 > Thanks, > Lee Thomas >=20 > Test platform: > $uname -a > FreeBSD LeeDesktop 9.1-STABLE FreeBSD 9.1-STABLE #15 r250831: > Mon May 20 17:31:28 EDT 2013 > lthomas@LeeDesktop:/usr/obj/usr/src/sys/NOSOUND amd64 > $dmesg | grep CPU: > CPU: Intel(R) Xeon(R) CPU X5650 @ 2.67GHz > (2666.81-MHz K8-class CPU) > $clang --version > FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 201212= 21 > Target: x86_64-unknown-freebsd9.1 > Thread model: posix >=20 > My timing test was a file of 10000 strings, of uniformly random length,= > 50% between 0 and 20 bytes long, and 50% between 21 and 1000 bytes long= =2E > Each was filled with random generated bytes in the range [1, 255]. The > test executables run their strlen on each string in the file 1000 times= , > and a shell script ran the test programs alternately 100 times. Here ar= e > the clang results; gcc results are roughly the same. I will share the > actual timing code if someone wants it, but it is pretty ugly C++ and > shell and I don't guarantee it working :-). >=20 > Results: >=20 > x ./times_bsd_strnlen.txt > + ./times_my_strnlen.txt > +----------------------------------------------------------------------= ----+ >=20 > |+ = x| > |+ = x| > |+ = x| > |+ = x| > |+ = x| > |+ = x| > |+ = x| > |+ = x| > |+ = x| > |+ = x| > |+ = x| > |+ = x| > |+ = x| > |+ = x| > |+ = x| > |+ = x| > |+ = x| > |A = A| > +----------------------------------------------------------------------= ----+ >=20 > N Min Max Median Avg St= ddev > x 101 1.8685486 1.8693889 1.8687506 1.8687894 0.000148= 4903 > + 101 0.42704683 0.42779486 0.42712813 0.42715597 0.0001068= 1494 > Difference at 95.0% confidence > -1.44163 +/- 3.56739e-05 > -77.1426% +/- 0.00190893% > (Student's t, pooled s =3D 0.000129342) >=20 > Note: I manually shortened the histogram, as it was 100 lines long of > exactly the same. >=20 >=20 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.o= rg" >=20 > Index: strnlen.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > diff --git a/head/lib/libc/string/strnlen.c b/head/lib/libc/string/strn= len.c > --- a/head/lib/libc/string/strnlen.c (revision 250951) > +++ b/head/lib/libc/string/strnlen.c (working copy) > @@ -1,5 +1,6 @@ > /*- > - * Copyright (c) 2009 David Schultz > + * Copyright (c) 2009, 2010 Xin LI > + * Copyright (c) 2013 Lee Thomas > * All rights reserved. > * > * Redistribution and use in source and binary forms, with or without > @@ -27,16 +28,91 @@ > #include > __FBSDID("$FreeBSD$"); > =20 > +#include > +#include > #include > =20 > +/* > + * Portable strnlen() for 32-bit and 64-bit systems. > + * > + * Rationale: it is generally much more efficient to do word length > + * operations and avoid branches on modern computer systems, as > + * compared to byte-length operations with a lot of branches. > + * > + * The expression: > + * > + * ((x - 0x01....01) & ~x & 0x80....80) > + * > + * would evaluate to a non-zero value iff any of the bytes in the > + * original word is zero. > + * > + * On multi-issue processors, we can divide the above expression into:= > + * a) (x - 0x01....01) > + * b) (~x & 0x80....80) > + * c) a & b > + * > + * Where, a) and b) can be partially computed in parallel. > + * > + * The algorithm above is found on "Hacker's Delight" by > + * Henry S. Warren, Jr. > + */ > + > +/* Magic numbers for the algorithm */ > +#if LONG_BIT =3D=3D 32 > +static const unsigned long mask01 =3D 0x01010101; > +static const unsigned long mask80 =3D 0x80808080; > +#elif LONG_BIT =3D=3D 64 > +static const unsigned long mask01 =3D 0x0101010101010101; > +static const unsigned long mask80 =3D 0x8080808080808080; > +#else > +#error Unsupported word size > +#endif > + > +#define LONGPTR_MASK (sizeof(long) - 1) > + > size_t > -strnlen(const char *s, size_t maxlen) > +strnlen(const char *str, size_t maxlen) > { > - size_t len; > + const char *stop, *short_stop; > + const char *p; > + const unsigned long *lp; > + long va, vb; > =20 > - for (len =3D 0; len < maxlen; len++, s++) { > - if (!*s) > - break; > + if (maxlen=3D=3D0) return 0; > + > + stop=3Dstr+maxlen; > + /* > + * Before trying the hard (unaligned byte-by-byte access) way > + * to figure out whether there is a nul character, try to see > + * if there is a nul character is within this accessible word > + * first. > + * > + * p and (p & ~LONGPTR_MASK) must be equally accessible since > + * they always fall in the same memory page, as long as page > + * boundaries is integral multiple of word size. > + */ > + lp =3D (const unsigned long *)((uintptr_t)str & ~LONGPTR_MASK); > + va =3D (*lp - mask01); > + vb =3D ((~*lp) & mask80); I do not think that this correct C. This is type punning violating the rules of the language. One way you can rewrite this is to use memcpy() into a temporary. Do not be afraid of the memcpy(), modern GCC, and I hope even Clang, can optimize it into a single move instruction: { long tmp; memcpy(&tmp, lp, sizeof(tmp)); va =3D (tmp - mask01); vb =3D ((~tmp) & mask80); } Another way could be using the union trick. > + lp++; > + if (va & vb) { > + /* Check if we have \0 in the first part */ > + short_stop=3D(const char *)lp; > + if (stop + for (p=3Dstr; p !=3D short_stop; p++) > + if (*p =3D=3D '\0') > + return (p-str); > } > - return (len); > + /* Scan the rest of the string using word sized operation */ > + for (; (const char *)lp < stop; lp++) { > + va =3D (*lp - mask01); > + vb =3D ((~*lp) & mask80); Ditto. > + if (va & vb) { > + for (p=3D(const char *)lp; p !=3D stop; p++) > + if (*p =3D=3D '\0') > + break; > + return (p-str); > + } > + } > + return (maxlen); > } --=20 VZ --------------enig1FEBB2302363A6AC4C0260A6 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.19 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iF4EAREIAAYFAlGhqmEACgkQ6OWUyaYNY6OgEAD/XZbTU+/A4DiqlCuF31LB2WFz 9Qmcy8o2eALuL9HCkkgBAIxcLCprHbfby4hx84Laz36uFBiF3EnibKW/DuVxzqar =v93F -----END PGP SIGNATURE----- --------------enig1FEBB2302363A6AC4C0260A6-- From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 11:09:41 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C609BB76 for ; Sun, 26 May 2013 11:09:41 +0000 (UTC) (envelope-from oritm@mellanox.com) Received: from eu1sys200aog116.obsmtp.com (eu1sys200aog116.obsmtp.com [207.126.144.141]) by mx1.freebsd.org (Postfix) with ESMTP id F307E253 for ; Sun, 26 May 2013 11:09:40 +0000 (UTC) Received: from MTLCAS01.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob116.postini.com ([207.126.147.11]) with SMTP ID DSNKUaHtUjhGflFrqwtTip0teXsDD6S1yyQk@postini.com; Sun, 26 May 2013 11:09:41 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS01.mtl.com ([10.0.8.71]) with mapi id 14.03.0123.003; Sun, 26 May 2013 14:09:04 +0300 From: Orit Moskovich To: "freebsd-hackers@freebsd.org" Subject: Re: preemptive kernel Thread-Topic: Re: preemptive kernel Thread-Index: Ac5aAPD3Vi5EqI24R2Ks+QyWmjdUCw== Date: Sun, 26 May 2013 11:09:03 +0000 Message-ID: <981733489AB3BD4DB24B48340F53E0A55B0D5590@MTLDAG01.mtl.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 11:09:41 -0000 Can a filter routine preempt another filter routine? And can an interrupt t= hread (or a filter routine) preempt another ithread? From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 15:47:57 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2792390F for ; Sun, 26 May 2013 15:47:57 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 93C6FA2 for ; Sun, 26 May 2013 15:47:56 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r4QFlqST082136; Sun, 26 May 2013 18:47:52 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r4QFlqST082136 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r4QFlqaY082135; Sun, 26 May 2013 18:47:52 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 26 May 2013 18:47:52 +0300 From: Konstantin Belousov To: Orit Moskovich Subject: Re: preemptive kernel Message-ID: <20130526154752.GT3047@kib.kiev.ua> References: <981733489AB3BD4DB24B48340F53E0A55B0D5590@MTLDAG01.mtl.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="w2cbNFf0kg38sO0z" Content-Disposition: inline In-Reply-To: <981733489AB3BD4DB24B48340F53E0A55B0D5590@MTLDAG01.mtl.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 15:47:57 -0000 --w2cbNFf0kg38sO0z Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Sun, May 26, 2013 at 11:09:03AM +0000, Orit Moskovich wrote: > Can a filter routine preempt another filter routine? And can an interrupt thread (or a filter routine) preempt another ithread? Filter handler borrows the context from the thread executed at the time of the interrupt. At least on x86 (i.e. i386 and amd64) interrupt handlers are executed with the interrupts disabled. So the filter cannot interrupt another filter. Interrupt threads are executed with the interrupts enabled. So if an interrupt is delivered to the CPU while the CPU executed the ithread, the filter 'preempts' the ithread by executing in its context. For the same reason, if the interrupt delivered causes a higher-priority thread become ready to run, then the new runnable thread could preempt the interrupt thread according to the priority, affinity and other factors considered by a scheduler. --w2cbNFf0kg38sO0z Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRoi6nAAoJEJDCuSvBvK1Bky0P/AsWMdAScPYAkRfmVNL/1nae AZvawnUlLzekMXs2LwDQzkMiHvHMXSrDuIer3LY5tCY46C56JJ7rVhbZBYuYn0qC ciaA5QNxLQinufRAyU2kfUPMjBnh7hl2cPaz2kDULHCiyAQmFkgp6UM5+AHKOFFR TELiMIk5fG6/MWlRB5esz8tnVnlSxt/jPstH2Gp9jB752GPgFicPBulM2xOEP6S1 oKkw153jdhrNJFS/xAzYB5u+aFCzJYQfvRU0Go0cVgSplZPyzqAZqG+QTwdN1sFR 0v5UD112cZ7QeI2uP7SHbzgkYeq+sPacJS1hPH9Wl5bQqAjTn1s4uiYxJTfwMAd9 xSvr7SAiHy43lBUGP8kTZtVmQl5OSAhccviaSjSHQWKGmlWbSB/ExCc89TdAgxlo 8D1QQEzfrXRQ/vdmONOsmXO92yRslDghgBypuQullKF23BT6keU4LwmsYZCuJ/2M ZllRTKeNmMwspiZQEv9JQTkA8kKZfC8d/u/0NTtinscM9ZmLjk1Sr/eq3dP0poM0 HL+GDhsBdamdhGPipaL4jk2Bnf+A9+J782h9CEedIKSKq8w95uKjGkJHw0CSBbDe ZvTTDuAxtvyGLpOrgijVumO2ogBjkRejVA3R4A/pHKZm/91SmU/ybU0VTsYCzfUb aDANes1Af1pj06ZbRkDP =p8O7 -----END PGP SIGNATURE----- --w2cbNFf0kg38sO0z-- From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 16:58:46 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 87BD37C5; Sun, 26 May 2013 16:58:46 +0000 (UTC) (envelope-from superbisquit@gmail.com) Received: from mail-ob0-x22d.google.com (mail-ob0-x22d.google.com [IPv6:2607:f8b0:4003:c01::22d]) by mx1.freebsd.org (Postfix) with ESMTP id 3AB6D285; Sun, 26 May 2013 16:58:46 +0000 (UTC) Received: by mail-ob0-f173.google.com with SMTP id eh20so7117873obb.4 for ; Sun, 26 May 2013 09:58:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=j6owUBVaFAaaYR7sSEcFF/3ANBk2sHWApispB/sI+xw=; b=xMP/rkCJgqE6hja83ASu1d5mnyQks/4Q4C2CH1JrqISAWCadrOBAG1uExxS2k4kEms RnQISAgLYqxLWDgx5PkYmb3e9Hoy/H4B9EY+x07gZw324pMjC1FjidJum6j0qSP2/0gg lFrzWEfTYGOKkwufnz00jlP6TGkHzNrBzGGw6M7Gzhe4xmIcN9lLdRMNzev8XRAK7hGJ QwhWn9zdncPHbz/llb1ktfWqEv2m+LE3TOLWRRWBU8lZuphzE2rEyLCdvU4vlX+l+JhF bx4yL6ZYZLSk0I39P8v/AaqVeljlOipynHallKAkB7tR40Bd8nUEBi08nt5ZDNLMEw2B AKjg== MIME-Version: 1.0 X-Received: by 10.60.83.103 with SMTP id p7mr16195365oey.130.1369587524727; Sun, 26 May 2013 09:58:44 -0700 (PDT) Received: by 10.182.53.231 with HTTP; Sun, 26 May 2013 09:58:44 -0700 (PDT) In-Reply-To: <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> Date: Sun, 26 May 2013 12:58:44 -0400 Message-ID: Subject: Re: FreeBSD installers and future direction From: Super Bisquit To: Devin Teske Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: FreeBSD Hackers , Dirk Engling , Nathan Whitehorn X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 16:58:46 -0000 May I- and others- see the hyperlink to the project, please? On Sat, May 25, 2013 at 11:45 PM, Teske, Devin w= rote: > > On May 25, 2013, at 7:51 PM, Super Bisquit wrote: > > > Please don't turn this into an architecture dependent mess. PCBSD is > i386 & > > AMD64 only. > > > > There's a GSoC project (of which I'm potential mentor) to fix that. > > However, you are entirely right=85 we can't in all seriousness even think > about using pc-sysinstall until it is solid on all architectures as > bsdinstall already is. > > GSoC project is: "Making pc-sysinstall FreeBSD ready by porting it to > multiple architectures" > -- > Devin > > > > > > > On Sat, May 25, 2013 at 9:01 PM, Dirk Engling > wrote: > > > >> On 26.05.13 01:07, Nathan Whitehorn wrote: > >> > >>> I'm not aware of any movement there (on either side of the table). I'= d > >>> personally be very suspicious of an all-sh(1) future -- by far the > >>> cleanest parts of bsdinstall are in C -- and this is especially true > for > >>> interacting with geom. That said, since I've lost nearly all of my fr= ee > >>> time and ability to work on bsdinstall, I won't get in the way of > anyone > >>> else working on things > >> > >> As discussed at BSDCan, I'd be willing to participate in the developme= nt > >> and at least implement setting up zpools/zfs and geli/gbde providers. = I > >> have done similar things in sh in my ezjail tools and think I can glue > >> the rest together. > >> > >> Scanning through the pc-sysinstall code, I find nothing too fancy ther= e > >> regarding either interaction with zfs nor geom tools. I do not think i= t > >> is necessary as a back end just for these features. > >> > >> Nathan, is there any design rationale available for the scripts, e.g. = on > >> why you chose sh versus C and were you provided with some kind of wish > >> list/requirements in the first place? Any particular mail thread to sc= an > >> through beforehand? > >> > >> Regards, > >> > >> erdgeist > >> _______________________________________________ > >> freebsd-hackers@freebsd.org mailing list > >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > >> To unsubscribe, send any mail to " > freebsd-hackers-unsubscribe@freebsd.org" > >> > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to " > freebsd-hackers-unsubscribe@freebsd.org" > > _____________ > The information contained in this message is proprietary and/or > confidential. If you are not the intended recipient, please: (i) delete t= he > message and all copies; (ii) do not disclose, distribute or use the messa= ge > in any manner; and (iii) notify the sender immediately. In addition, plea= se > be aware that any message addressed to our domain is subject to archiving > and review by persons other than the intended recipient. Thank you. > From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 17:03:17 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 723ACAFA for ; Sun, 26 May 2013 17:03:17 +0000 (UTC) (envelope-from erdgeist@erdgeist.org) Received: from elektropost.org (elektropost.org [217.13.206.130]) by mx1.freebsd.org (Postfix) with ESMTP id C38CB2B6 for ; Sun, 26 May 2013 17:03:16 +0000 (UTC) Received: (qmail 81061 invoked from network); 26 May 2013 17:03:11 -0000 Received: from elektropost.org (HELO elektropost.org) (erdgeist@erdgeist.org) by elektropost.org with CAMELLIA256-SHA encrypted SMTP; 26 May 2013 17:03:11 -0000 Message-ID: <51A2404F.5040908@erdgeist.org> Date: Sun, 26 May 2013 19:03:11 +0200 From: Dirk Engling User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Super Bisquit Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> In-Reply-To: X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: FreeBSD Hackers , Nathan Whitehorn X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 17:03:17 -0000 On 26.05.13 04:51, Super Bisquit wrote: > Please don't turn this into an architecture dependent mess. PCBSD is > i386 & AMD64 only. Read my email thoroughly and notice that I never seriously considered using pc-sysinstall after looking into it. Don't worry. erdgeist From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 17:54:34 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9A35B226; Sun, 26 May 2013 17:54:34 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 071ED5FF; Sun, 26 May 2013 17:54:33 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.31]) by ltcfislmsgpa05.fnfis.com (8.14.5/8.14.5) with ESMTP id r4QHsQfa006108 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sun, 26 May 2013 12:54:27 -0500 Received: from LTCFISWMSGMB21.FNFIS.com ([10.132.99.23]) by LTCFISWMSGHT03.FNFIS.com ([10.132.206.31]) with mapi id 14.02.0309.002; Sun, 26 May 2013 12:54:26 -0500 From: "Teske, Devin" To: Super Bisquit Subject: Re: FreeBSD installers and future direction Thread-Topic: FreeBSD installers and future direction Thread-Index: AQHOWV60z6k72swSkkmGANUzy/EMLJkWZ1wAgAAkfQCAAE6UgIAAH7eAgAAeuACAAA8cgIAA3asAgAAPj4A= Date: Sun, 26 May 2013 17:54:26 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F603E6@ltcfiswmsgmb21> References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-26_05:2013-05-24,2013-05-26,1970-01-01 signatures=0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Dirk Engling , Devin Teske , Nathan Whitehorn , FreeBSD Hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 17:54:34 -0000 On May 26, 2013, at 9:58 AM, Super Bisquit wrote: May I- and others- see the hyperlink to the project, please? Absolutely=85 http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/harshbha= tt/1 -- Devin On Sat, May 25, 2013 at 11:45 PM, Teske, Devin > wrote: On May 25, 2013, at 7:51 PM, Super Bisquit wrote: > Please don't turn this into an architecture dependent mess. PCBSD is i386= & > AMD64 only. > There's a GSoC project (of which I'm potential mentor) to fix that. However, you are entirely right=85 we can't in all seriousness even think a= bout using pc-sysinstall until it is solid on all architectures as bsdinsta= ll already is. GSoC project is: "Making pc-sysinstall FreeBSD ready by porting it to multi= ple architectures" -- Devin > > On Sat, May 25, 2013 at 9:01 PM, Dirk Engling > wrote: > >> On 26.05.13 01:07, Nathan Whitehorn wrote: >> >>> I'm not aware of any movement there (on either side of the table). I'd >>> personally be very suspicious of an all-sh(1) future -- by far the >>> cleanest parts of bsdinstall are in C -- and this is especially true for >>> interacting with geom. That said, since I've lost nearly all of my free >>> time and ability to work on bsdinstall, I won't get in the way of anyone >>> else working on things >> >> As discussed at BSDCan, I'd be willing to participate in the development >> and at least implement setting up zpools/zfs and geli/gbde providers. I >> have done similar things in sh in my ezjail tools and think I can glue >> the rest together. >> >> Scanning through the pc-sysinstall code, I find nothing too fancy there >> regarding either interaction with zfs nor geom tools. I do not think it >> is necessary as a back end just for these features. >> >> Nathan, is there any design rationale available for the scripts, e.g. on >> why you chose sh versus C and were you provided with some kind of wish >> list/requirements in the first place? Any particular mail thread to scan >> through beforehand? >> >> Regards, >> >> erdgeist >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing = list >> http://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.or= g" >> > _______________________________________________ > freebsd-hackers@freebsd.org mailing l= ist > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 18:14:30 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id DE2B48A6; Sun, 26 May 2013 18:14:30 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [IPv6:2a01:348:0:15:5d59:5c40:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id 9DB726C8; Sun, 26 May 2013 18:14:30 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id B299CE6A36; Sun, 26 May 2013 19:14:25 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=mail; bh=edRYe6ZbVQgS yYfTG/6vjJA0nmA=; b=tJc+RBAcKCxYwVbaa9w1gXPAYBlG65ZYj1Kip5Tl7U1D +wH69ZIusgn5jqq2myfWIqKDwQsfn3RzIoPJyl+GIflvEUn8E2N1/lXM9uqRUnav TPlVBA+vHCgGgVXC/iybJGspXYAS8WnEG1+KK1YTyeLP7sDQzXJn6INvrL4EN/Q= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=mail; b=UOnT4u 7Y15NFpnPLoPcSVITyH3g2vLtJTXqGVaBUpgEXdT1GR8VxZk4/RHmBX3n2qacAcK 8kBg4vRbaQ1LHfpTfufy2qeBVSNEvVPEs3kL9Jm5v7veqZTLxiBoNtFUGiSD0GIt cbZbRhlQEFI8E3incL/T6yKJ/BzfB4FJOkM7Q= Received: from [192.168.2.61] (unknown [93.89.81.205]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id 6CB62E6A27; Sun, 26 May 2013 19:14:25 +0100 (BST) Message-ID: <51A250FE.3070206@cran.org.uk> Date: Sun, 26 May 2013 19:14:22 +0100 From: Bruce Cran User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Devin Teske Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <13CA24D6AB415D428143D44749F57D7201F603E6@ltcfiswmsgmb21> In-Reply-To: <13CA24D6AB415D428143D44749F57D7201F603E6@ltcfiswmsgmb21> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: Super Bisquit , Dirk Engling , "Teske, Devin" , FreeBSD Hackers , Nathan Whitehorn X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 18:14:30 -0000 On 26/05/2013 18:54, Teske, Devin wrote: > http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/harshbhatt/1 "This proposal is not made public, and you are not the student who submitted the proposal, nor are you a mentor for the organization it was submitted to." -- Bruce Cran From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 19:37:16 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E25B7CDE; Sun, 26 May 2013 19:37:16 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id AD178A29; Sun, 26 May 2013 19:37:16 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.31]) by ltcfislmsgpa07.fnfis.com (8.14.5/8.14.5) with ESMTP id r4QJb8ln029964 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sun, 26 May 2013 14:37:08 -0500 Received: from LTCFISWMSGMB21.FNFIS.com ([10.132.99.23]) by LTCFISWMSGHT03.FNFIS.com ([10.132.206.31]) with mapi id 14.02.0309.002; Sun, 26 May 2013 14:37:08 -0500 From: "Teske, Devin" To: Bruce Cran Subject: Re: FreeBSD installers and future direction Thread-Topic: FreeBSD installers and future direction Thread-Index: AQHOWV60z6k72swSkkmGANUzy/EMLJkWZ1wAgAAkfQCAAE6UgIAAH7eAgAAeuACAAA8cgIAA3asAgAAPj4CAAAWTAIAAFx4A Date: Sun, 26 May 2013 19:37:06 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F615E8@ltcfiswmsgmb21> References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <13CA24D6AB415D428143D44749F57D7201F603E6@ltcfiswmsgmb21> <51A250FE.3070206@cran.org.uk> In-Reply-To: <51A250FE.3070206@cran.org.uk> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] Content-Type: text/plain; charset="Windows-1252" Content-ID: <80C3FC48BC139047AE1AA0FB88E61E50@fisglobal.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-26_05:2013-05-24,2013-05-26,1970-01-01 signatures=0 Cc: Super Bisquit , Devin Teske , Nathan Whitehorn , Dirk Engling , FreeBSD Hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 19:37:17 -0000 On May 26, 2013, at 11:14 AM, Bruce Cran wrote: > On 26/05/2013 18:54, Teske, Devin wrote: >> http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/harsh= bhatt/1 >=20 > "This proposal is not made public, and you are not the student who submit= ted the proposal, nor are you a mentor for the organization it was submitte= d to." >=20 So, uh=85 register? I can see all private projects and all I did was register with google-melan= ge. I'm not aware of a way to make it public versus private. --=20 Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 23:27:33 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id BE600821 for ; Sun, 26 May 2013 23:27:33 +0000 (UTC) (envelope-from erdgeist@erdgeist.org) Received: from elektropost.org (elektropost.org [217.13.206.130]) by mx1.freebsd.org (Postfix) with ESMTP id 35B3D211 for ; Sun, 26 May 2013 23:27:32 +0000 (UTC) Received: (qmail 33238 invoked from network); 26 May 2013 23:27:27 -0000 Received: from elektropost.org (HELO elektropost.org) (erdgeist@erdgeist.org) by elektropost.org with CAMELLIA256-SHA encrypted SMTP; 26 May 2013 23:27:27 -0000 Message-ID: <51A29A5F.7010106@erdgeist.org> Date: Mon, 27 May 2013 01:27:27 +0200 From: Dirk Engling User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Devin Teske Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B2E7@ltcfiswmsgmb26> In-Reply-To: <13CA24D6AB415D428143D44749F57D7201F5B2E7@ltcfiswmsgmb26> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit Cc: FreeBSD Hackers , "Teske, Devin" , Nathan Whitehorn X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 23:27:33 -0000 On 26.05.13 05:42, Teske, Devin wrote: > I chose 100% sh for bsdconfig because of a few good reasons… First, the partedit tool makes heavy use of libgeom and the structs returned from that lib, so I've rather wondered why for some parts C was preferred, and not the other way around. Still, thanks for pointing all that out, but I rather wanted to look at the installer from another angle, as it is supposed to provide everyone from FreeBSD novices to experts with a comfortable way to do things the right way and yet be flexible enough to avoid abandoning the tool once the requirements differ. So I wonder if there has ever been a best practices document on how to "properly" set up zpools, when to advice the user against using zfs at all, whether it makes sense to use geli on the boot device, when it is better to have multiple zpools and only encrypt the data pool(s). Maybe the installer should be advocating concepts like manageBE, pre-setting noexec-flags on /var, setting some default quotas. The second part, of course, is to find visual concepts on how the user is guided through the default and expert bsdinstall/bsdconfig screens to cover the most common scenarios and still offer enough options. All this doesn't need a developer but a bunch of veteran FreeBSD admins, a wiki and a lot of bike sheds to paint. If there's no such document yet, I propose editing one in the wiki. erdgeist From owner-freebsd-hackers@FreeBSD.ORG Sun May 26 23:39:01 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4E791AE6; Sun, 26 May 2013 23:39:01 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [IPv6:2a01:348:0:15:5d59:5c40:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id 0F71D28A; Sun, 26 May 2013 23:39:01 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id A2FC3E6A39; Mon, 27 May 2013 00:38:46 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=mail; bh=IEWF2yaMH86+ FQmA/ptmTv43PGA=; b=EbcDhaBSBx0k9XNTgBLrAVs3UO3UcEzpyU6CI1iwRhxD W/64mDt0Vx9CuzqmVynQMycGnyh+klYNol0F7bj8huXIZ+6kJBZpzP8uyY+cKfZK 58+7NQamN/p8pE2Ot+aSV2upupyn1Mq5UloGxotOj7GIzf4Vp5A5rJic+PzLDpc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=mail; b=mboNAJ i0xbIDuJAtHLZjy8QbdBHbGhK+061Wl36ifCB+LVgqmlZlPV7TIm1XmZs3X1xzXt GF9Ifk/bRsfNNNcm1uFmtjTrJr3+lu1j8HcxHGT5mzkpAcWaXO+0FbmQwdtsmfSq vKz8ZhZErTrUw1aLeTVc1Wow5TFAEShWcDoPc= Received: from [IPv6:2a01:348:301:2:ccf1:925a:43f8:4bbc] (unknown [IPv6:2a01:348:301:2:ccf1:925a:43f8:4bbc]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id 6EA6CE6A27; Mon, 27 May 2013 00:38:46 +0100 (BST) Message-ID: <51A29D03.60609@cran.org.uk> Date: Mon, 27 May 2013 00:38:43 +0100 From: Bruce Cran User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Dirk Engling Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B2E7@ltcfiswmsgmb26> <51A29A5F.7010106@erdgeist.org> In-Reply-To: <51A29A5F.7010106@erdgeist.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: FreeBSD Hackers , Devin Teske , "Teske, Devin" , Nathan Whitehorn X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 26 May 2013 23:39:01 -0000 On 27/05/2013 00:27, Dirk Engling wrote: > Still, thanks for pointing all that out, but I rather wanted to look at > the installer from another angle, as it is supposed to provide everyone > from FreeBSD novices to experts with a comfortable way to do things the > right way and yet be flexible enough to avoid abandoning the tool once > the requirements differ. I'd like to see an option of different front-ends for the installer/configurator to cater for different users - at least an X11 application, but there was also an idea of having a http-based installation UI. -- Bruce Cran From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 01:54:32 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2798AF25; Mon, 27 May 2013 01:54:32 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id E75DD9AE; Mon, 27 May 2013 01:54:31 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.15]) by ltcfislmsgpa02.fnfis.com (8.14.5/8.14.5) with ESMTP id r4R1sPIS027346 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Sun, 26 May 2013 20:54:25 -0500 Received: from LTCFISWMSGMB21.FNFIS.com ([10.132.99.23]) by LTCFISWMSGHT04.FNFIS.com ([10.132.206.15]) with mapi id 14.02.0309.002; Sun, 26 May 2013 20:54:25 -0500 From: "Teske, Devin" To: Dirk Engling Subject: Re: FreeBSD installers and future direction Thread-Topic: FreeBSD installers and future direction Thread-Index: AQHOWV60z6k72swSkkmGANUzy/EMLJkWZ1wAgAAkfQCAAE6UgIAAH7eAgAAtKgCAAUrxgIAAKQ4A Date: Mon, 27 May 2013 01:54:24 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F61FE0@ltcfiswmsgmb21> References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B2E7@ltcfiswmsgmb26> <51A29A5F.7010106@erdgeist.org> In-Reply-To: <51A29A5F.7010106@erdgeist.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] Content-Type: text/plain; charset="Windows-1252" Content-ID: <181571E007F07E4E98C01B62A7626EFE@fisglobal.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-27_01:2013-05-24,2013-05-27,1970-01-01 signatures=0 Cc: FreeBSD Hackers , Devin Teske , Nathan Whitehorn X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 01:54:32 -0000 On May 26, 2013, at 4:27 PM, Dirk Engling wrote: > On 26.05.13 05:42, Teske, Devin wrote: >=20 >> I chose 100% sh for bsdconfig because of a few good reasons=85 >=20 > First, the partedit tool makes heavy use of libgeom and the structs > returned from that lib, so I've rather wondered why for some parts C was > preferred, and not the other way around. >=20 I tend to be of the mind that shell (gpart in this case) would be a better choice for the installer than tapping into geom=85 (for the following) Using gpart makes you exercise both the end-user utilities and the developer framework (by-way of those utilities) whereas if you stick to just the developer framework (libgeom in this case) there's a chance that the installer can do things that can't be done from the command-line. Just a thought. If structs are what you want, I have those in bsdconfig: dteske@scribe9.vicor.com share $ pwd /home/dteske/src/freebsd_svn/base/head/usr.sbin/bsdconfig/share dteske@scribe9.vicor.com share $ grep '^# f_' struct.subr=20 # f_struct_define $type $member_name1 ... # f_struct_new $type $name # f_struct $name # f_struct $name get $property [$var_to_set] # f_struct $name set $property $new_value # f_struct $name unset $property # f_struct_free $name # f_struct_copy $from_name $to_name Naturally, because it's shell, the struct members are loosely typed (so no need to define their type). So I think there's great potential to write a very clean "gpart.subr" that combines the power of the above "struct.subr" with perhaps another of my modules=85 dteske@scribe9.vicor.com share $ grep '^# f_' device.subr=20 # f_device_try $name [$i [$var_path]] # f_device_register $name $desc $devname $type $enabled $init_function \ # f_device_reset # f_device_get_all # f_device_name_get $type $name type|desc|max [$var_to_set] # f_device_name_set $type $name $desc [$max] # f_device_desc $device_name $device_type [$var_to_set] # f_device_rescan # f_device_find $name [$type [$var_to_set]]=20 # f_device_init $name # f_device_get $name $file [$probe] # f_device_shutdown $name # f_device_menu $title $prompt $hline $device_type [$helpfile] NOTE: device.subr already uses struct.subr. For example, $device_type is actually a struct type, and when you call f_device_get_all() it creates a series of structs named "device_{name}" (where {name} is something like "da0") and the struct holds many properties for the device. I don't think there's any reason why we have to write it in C if we can wri= te it in sh. > Still, thanks for pointing all that out, but I rather wanted to look at > the installer from another angle, as it is supposed to provide everyone > from FreeBSD novices to experts with a comfortable way to do things the > right way and yet be flexible enough to avoid abandoning the tool once > the requirements differ. >=20 I don't see your angle as perpendicular to my own=85 but rather that I would like to seek to define in concrete terms what that means. Working backwards from your list=85 Flexibility: I think using a conflated shell namespace (where the entire installer is sh and is in-turn scriptable by sh) gives the most flexibility= . Much more than flexibility, I think this approach also lowers the barrier-to-ent= ry for integration and automation engineers. Open questions: was the fact that sysinstall was all C with a few forks to external programs a barrier to sys= tem- integration and system-automation engineers trying to automate the install- and-configure process? Maybe. The BSD-P certification exam requires you to script sysinstall in FreeBSD-8. If we go all-shell, this would become mu= ch easier IMHO. Novices to Experts: I'd like to strive to hit a full spectrum between those= two points. I recognize that these are not two discrete ends of a dichotomous tree, but instead a journey that is traversed (assuming the novice stays wi= th FreeBSD after trying it out). There are no defined sets of travel along this line either, some people skip being a novice when they come to FreeBSD for the first time because they have experience with similar operating syst= ems. That's why I've taken to developing every aspect of the utility. I want to = see the installer and configurator helpful to all people from developers to gra= nd- mothers and everybody in-between (that's the modular part=85 if you don't like the way something works=85 the sh(1) nature allows you to simply override it with your own module). > So I wonder if there has ever been a best practices document on how to > "properly" set up zpools, when to advice the user against using zfs at > all, whether it makes sense to use geli on the boot device, when it is > better to have multiple zpools and only encrypt the data pool(s). Maybe > the installer should be advocating concepts like manageBE, pre-setting > noexec-flags on /var, setting some default quotas. >=20 You'll get no complaints from me here. The installer is (in my view) not only where we need to provide the most he= lpful functionality, but the best place to stash real developer insight when it c= omes to analyzing the installation environment and providing "canned" recipes. > The second part, of course, is to find visual concepts on how the user > is guided through the default and expert bsdinstall/bsdconfig screens to > cover the most common scenarios and still offer enough options. >=20 I think there's two approaches to solving the issue of custom screens that = are not producible with normal dialog(1) or Xdialog(1) (the two interfaces that= are currently supported by bsdconfig; bsdinstall currently only supports dialog= (1)): 1. Use libdialog to produce variations that dialog(1) cannot produce. NOTE: This is what bsdinstall does. 2. Write a helper utility like ssh-askpass to be called from the shell code The latter can take more patience, but has (imho) greater potential for re-= use. > All this doesn't need a developer but a bunch of veteran FreeBSD admins, > a wiki and a lot of bike sheds to paint. >=20 Well=85 a developer doesn't hurt. Someone to make sure the code is clean and re-usable. > If there's no such document yet, I propose editing one in the wiki. >=20 Not aware of one. What would it talk about? --=20 Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 02:10:44 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 98646217; Mon, 27 May 2013 02:10:44 +0000 (UTC) (envelope-from deischen@freebsd.org) Received: from mail.netplex.net (mail.netplex.net [204.213.176.9]) by mx1.freebsd.org (Postfix) with ESMTP id 58D809FF; Mon, 27 May 2013 02:10:44 +0000 (UTC) Received: from sea.ntplx.net (sea.ntplx.net [204.213.176.11]) by mail.netplex.net (8.14.6/8.14.6/NETPLEX) with ESMTP id r4R2Abr9022709; Sun, 26 May 2013 22:10:37 -0400 X-Virus-Scanned: by AMaViS and Clam AntiVirus (mail.netplex.net) X-Greylist: Message whitelisted by DRAC access database, not delayed by milter-greylist-4.4.1 (mail.netplex.net [204.213.176.9]); Sun, 26 May 2013 22:10:38 -0400 (EDT) Date: Sun, 26 May 2013 22:10:37 -0400 (EDT) From: Daniel Eischen X-X-Sender: eischen@sea.ntplx.net To: Devin Teske Subject: Re: FreeBSD installers and future direction In-Reply-To: <13CA24D6AB415D428143D44749F57D7201F61FE0@ltcfiswmsgmb21> Message-ID: References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B2E7@ltcfiswmsgmb26> <51A29A5F.7010106@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F61FE0@ltcfiswmsgmb21> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: FreeBSD Hackers , Dirk Engling , Nathan Whitehorn X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Daniel Eischen List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 02:10:44 -0000 On Mon, 27 May 2013, Teske, Devin wrote: > > I don't think there's any reason why we have to write it in C if we can write > it in sh. I don't really care one way or the other (C or sh), but I can say that I can understand(*) well structured C a lot better than well structured sh. Having something more strongly typed certainly helps understanding. (*) Assuming some level of complexity (I know that's subjective). -- DE From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 03:54:56 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5FAFD351 for ; Mon, 27 May 2013 03:54:56 +0000 (UTC) (envelope-from lee_thomas@aslantools.com) Received: from roproxy1-pub.unifiedlayer.com (roproxy1-pub.unifiedlayer.com [69.89.25.95]) by mx1.freebsd.org (Postfix) with SMTP id 1FC24F0E for ; Mon, 27 May 2013 03:54:55 +0000 (UTC) Received: (qmail 20179 invoked by uid 0); 27 May 2013 03:54:52 -0000 Received: from unknown (HELO mailchannelsproxy2.unifiedlayer.com) (66.147.243.71) by roproxy1.unifiedlayer.com with SMTP; 27 May 2013 03:54:52 -0000 X-Sender-Id: {:fast21.fastdomain.com:lthomasn:fast21.fastdomain.com} {sentby:program running on server} Received: from fast21.fastdomain.com (fast21.fastdomain.com [74.220.199.21]) by 0.0.0.0:2500 (trex/4.8.69); Mon, 27 May 2013 03:54:52 GMT Received: from localhost ([127.0.0.1]:49473 helo=fast21.fastdomain.com) by fast21.fastdomain.com with esmtp (Exim 4.80) (envelope-from ) id 1Uga8I-0007nG-Hp for freebsd-hackers@freebsd.org; Sun, 26 May 2013 06:33:06 -0600 To: Subject: Re: Performance improvement to strnlen(). X-PHP-Script: aslantools.com/mail/index.php for 173.71.114.222 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Sun, 26 May 2013 08:33:06 -0400 From: Lee Thomas Organization: Aslan Tools In-Reply-To: References: Message-ID: <3004a58e86f2eca25a3541db518ea133@lthomas.net> X-Sender: lee_thomas@aslantools.com User-Agent: Roundcube Webmail/0.8.4 X-Identified-User: {:fast21.fastdomain.com:lthomasn:fast21.fastdomain.com} {sentby:program running on server} X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 03:54:56 -0000 On 2013-05-26 08:00, Václav Zeman wrote: > On 05/25/2013 10:27 PM, Lee Thomas wrote: >> + lp = (const unsigned long *)((uintptr_t)str & ~LONGPTR_MASK); >> + va = (*lp - mask01); >> + vb = ((~*lp) & mask80); > I do not think that this correct C. This is type punning violating > the > rules of the language. Hello Václav, The aliasing here is safe, because there are no writes through either of the pointers. Regards, Lee Thomas From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 03:55:05 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E6205360 for ; Mon, 27 May 2013 03:55:05 +0000 (UTC) (envelope-from lee_thomas@aslantools.com) Received: from roproxy1-pub.unifiedlayer.com (roproxy1-pub.unifiedlayer.com [69.89.25.95]) by mx1.freebsd.org (Postfix) with SMTP id A6652F13 for ; Mon, 27 May 2013 03:55:05 +0000 (UTC) Received: (qmail 21866 invoked by uid 0); 27 May 2013 03:55:02 -0000 Received: from unknown (HELO mailchannelsproxy2.unifiedlayer.com) (66.147.243.71) by roproxy1.unifiedlayer.com with SMTP; 27 May 2013 03:55:02 -0000 X-Sender-Id: {:fast21.fastdomain.com:lthomasn:fast21.fastdomain.com} {sentby:program running on server} Received: from fast21.fastdomain.com (fast21.fastdomain.com [74.220.199.21]) by 0.0.0.0:2500 (trex/4.8.69); Mon, 27 May 2013 03:55:02 GMT Received: from localhost ([127.0.0.1]:55340 helo=fast21.fastdomain.com) by fast21.fastdomain.com with esmtp (Exim 4.80) (envelope-from ) id 1Uge0M-0003kT-CN for freebsd-hackers@freebsd.org; Sun, 26 May 2013 10:41:10 -0600 To: Subject: Re: Performance improvement to strnlen(). X-PHP-Script: aslantools.com/mail/index.php for 173.71.114.222 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Sun, 26 May 2013 12:41:10 -0400 From: Lee Thomas Organization: Aslan Tools In-Reply-To: <3004a58e86f2eca25a3541db518ea133@lthomas.net> References: <3004a58e86f2eca25a3541db518ea133@lthomas.net> Message-ID: <1da20503f41ab6834ee6bf3db542ae31@lthomas.net> X-Sender: lee_thomas@aslantools.com User-Agent: Roundcube Webmail/0.8.4 X-Identified-User: {:fast21.fastdomain.com:lthomasn:fast21.fastdomain.com} {sentby:program running on server} X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 03:55:06 -0000 On 2013-05-26 08:00, Václav Zeman wrote: > On 05/25/2013 10:27 PM, Lee Thomas wrote: >> + lp = (const unsigned long *)((uintptr_t)str & ~LONGPTR_MASK); >> + va = (*lp - mask01); >> + vb = ((~*lp) & mask80); > I do not think that this correct C. This is type punning violating > the > rules of the language. Hello Václav, The aliasing here is safe, because there are no writes through either of the pointers. Regards, Lee Thomas From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 03:55:16 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6D7A843D for ; Mon, 27 May 2013 03:55:16 +0000 (UTC) (envelope-from lee_thomas@aslantools.com) Received: from roproxy1-pub.unifiedlayer.com (roproxy1-pub.unifiedlayer.com [69.89.25.95]) by mx1.freebsd.org (Postfix) with SMTP id 303D5F15 for ; Mon, 27 May 2013 03:55:16 +0000 (UTC) Received: (qmail 24297 invoked by uid 0); 27 May 2013 03:55:15 -0000 Received: from unknown (HELO mailchannelsproxy2.unifiedlayer.com) (66.147.243.71) by roproxy1.unifiedlayer.com with SMTP; 27 May 2013 03:55:15 -0000 X-Sender-Id: {:fast21.fastdomain.com:lthomasn:fast21.fastdomain.com} {sentby:program running on server} Received: from fast21.fastdomain.com (fast21.fastdomain.com [74.220.199.21]) by 0.0.0.0:2500 (trex/4.8.69); Mon, 27 May 2013 03:55:15 GMT Received: from localhost ([127.0.0.1]:47619 helo=fast21.fastdomain.com) by fast21.fastdomain.com with esmtp (Exim 4.80) (envelope-from ) id 1UggCD-0004pt-SA for freebsd-hackers@freebsd.org; Sun, 26 May 2013 13:01:33 -0600 To: Subject: Re: Performance improvement to strnlen(). X-PHP-Script: aslantools.com/mail/index.php for 173.71.114.222 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Sun, 26 May 2013 15:01:33 -0400 From: Lee Thomas Organization: Aslan Tools Message-ID: <2adc4d8e7c55e92d935f61efb4c9d723@lthomas.net> X-Sender: lee_thomas@aslantools.com User-Agent: Roundcube Webmail/0.8.4 X-Identified-User: {:fast21.fastdomain.com:lthomasn:fast21.fastdomain.com} {sentby:program running on server} X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 03:55:16 -0000 On 2013-05-26 08:00, Václav Zeman wrote: > On 05/25/2013 10:27 PM, Lee Thomas wrote: >> + lp = (const unsigned long *)((uintptr_t)str & ~LONGPTR_MASK); >> + va = (*lp - mask01); >> + vb = ((~*lp) & mask80); > I do not think that this correct C. This is type punning violating > the > rules of the language. Hello Václav, The aliasing here is safe, because there are no writes through either of the pointers, and the reads are correctly aligned. Regards, Lee Thomas From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 05:03:14 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2078EADC for ; Mon, 27 May 2013 05:03:14 +0000 (UTC) (envelope-from oritm@mellanox.com) Received: from eu1sys200aog112.obsmtp.com (eu1sys200aog112.obsmtp.com [207.126.144.133]) by mx1.freebsd.org (Postfix) with ESMTP id 5E248149 for ; Mon, 27 May 2013 05:03:12 +0000 (UTC) Received: from MTLCAS02.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob112.postini.com ([207.126.147.11]) with SMTP ID DSNKUaLpB3JYaVuE8br66hyowDqPImsY7O9k@postini.com; Mon, 27 May 2013 05:03:13 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS02.mtl.com ([10.0.8.72]) with mapi id 14.03.0123.003; Mon, 27 May 2013 08:00:14 +0300 From: Orit Moskovich To: Konstantin Belousov Subject: RE: preemptive kernel Thread-Topic: preemptive kernel Thread-Index: AQHOWihm0YFG89T8IkWOFRvrz7kZiJkYcsLw Date: Mon, 27 May 2013 05:00:13 +0000 Message-ID: <981733489AB3BD4DB24B48340F53E0A55B0D56E0@MTLDAG01.mtl.com> References: <981733489AB3BD4DB24B48340F53E0A55B0D5590@MTLDAG01.mtl.com> <20130526154752.GT3047@kib.kiev.ua> In-Reply-To: <20130526154752.GT3047@kib.kiev.ua> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 05:03:14 -0000 Just to be more specific - On x86, during a filter routine all interrupts a= re disabled on the cpu currently handling a filter routine or only interrup= ts on the IRQ that generated the interrupt?=20 Is it possible that on a different cpu an interrupt (filter or ithread) of = the same IRQ will run? Should worry about locking data because an ithread c= an preempt the same ithread or they'll run concurrently on different cpus? -----Original Message----- From: Konstantin Belousov [mailto:kostikbel@gmail.com]=20 Sent: Sunday, May 26, 2013 06:48 PM To: Orit Moskovich Cc: freebsd-hackers@freebsd.org Subject: Re: preemptive kernel On Sun, May 26, 2013 at 11:09:03AM +0000, Orit Moskovich wrote: > Can a filter routine preempt another filter routine? And can an interrupt= thread (or a filter routine) preempt another ithread? Filter handler borrows the context from the thread executed at the time of = the interrupt. At least on x86 (i.e. i386 and amd64) interrupt handlers ar= e executed with the interrupts disabled. So the filter cannot interrupt an= other filter. Interrupt threads are executed with the interrupts enabled. So if an interr= upt is delivered to the CPU while the CPU executed the ithread, the filter = 'preempts' the ithread by executing in its context. For the same reason, i= f the interrupt delivered causes a higher-priority thread become ready to r= un, then the new runnable thread could preempt the interrupt thread accordi= ng to the priority, affinity and other factors considered by a scheduler. From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 06:34:38 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9B106CE6 for ; Mon, 27 May 2013 06:34:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id EA42C65E for ; Mon, 27 May 2013 06:34:37 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r4R6YYL4063806; Mon, 27 May 2013 09:34:34 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r4R6YYL4063806 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r4R6YW6L063805; Mon, 27 May 2013 09:34:32 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 27 May 2013 09:34:32 +0300 From: Konstantin Belousov To: Orit Moskovich Subject: Re: preemptive kernel Message-ID: <20130527063432.GY3047@kib.kiev.ua> References: <981733489AB3BD4DB24B48340F53E0A55B0D5590@MTLDAG01.mtl.com> <20130526154752.GT3047@kib.kiev.ua> <981733489AB3BD4DB24B48340F53E0A55B0D56E0@MTLDAG01.mtl.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="JoKCChanR9DHu5K+" Content-Disposition: inline In-Reply-To: <981733489AB3BD4DB24B48340F53E0A55B0D56E0@MTLDAG01.mtl.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 06:34:38 -0000 --JoKCChanR9DHu5K+ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, May 27, 2013 at 05:00:13AM +0000, Orit Moskovich wrote: > Just to be more specific - On x86, during a filter routine all interrupts= are disabled on the cpu currently handling a filter routine or only interr= upts on the IRQ that generated the interrupt?=20 The CPU enters the handler through the interrupt gate, which clears the PSL_I. Interrupts are not re-enabled until the handler return. > Is it possible that on a different cpu an interrupt (filter or ithread) o= f the same IRQ will run? Should worry about locking data because an ithread= can preempt the same ithread or they'll run concurrently on different cpus? The APIC or PCI MSI registers are programmed in the physical destination mode. This, together with the fact that interrupts are disabled on the target CPU, prevents parallel or nested execution of the filter. Having both filter and ithread for the same interrupt is apparently possible but weird. I do not see anything which would prevent interrupt filter =66rom being executed while the ithread is running. But again, this is very unusual setup. ithread has single stack, it cannot be run simultaneously on several CPUs. >=20 > -----Original Message----- > From: Konstantin Belousov [mailto:kostikbel@gmail.com]=20 > Sent: Sunday, May 26, 2013 06:48 PM > To: Orit Moskovich > Cc: freebsd-hackers@freebsd.org > Subject: Re: preemptive kernel >=20 > On Sun, May 26, 2013 at 11:09:03AM +0000, Orit Moskovich wrote: > > Can a filter routine preempt another filter routine? And can an interru= pt thread (or a filter routine) preempt another ithread? >=20 > Filter handler borrows the context from the thread executed at the time o= f the interrupt. At least on x86 (i.e. i386 and amd64) interrupt handlers = are executed with the interrupts disabled. So the filter cannot interrupt = another filter. >=20 > Interrupt threads are executed with the interrupts enabled. So if an inte= rrupt is delivered to the CPU while the CPU executed the ithread, the filte= r 'preempts' the ithread by executing in its context. For the same reason,= if the interrupt delivered causes a higher-priority thread become ready to= run, then the new runnable thread could preempt the interrupt thread accor= ding to the priority, affinity and other factors considered by a scheduler. --JoKCChanR9DHu5K+ Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRov53AAoJEJDCuSvBvK1BElQP/3cEU/CRzJ9dKzNP7N5j87vX aMM8+sSNBrQPwtYKt46ruQspHvkkZwhSJSIeih7PJtOYAmjhHPPiqwTDzLulD8tQ baEOzZEQJrSZGwk5g14LC5J2hjr4B8KxV7s1ZN/Cso2N1b7N28aTyGmOW8dcbMTz k57MV1zeXe7aZVVZWOEVh4A1N7DsbWT6pzUgbzGZQ1bBkxZzF8LxVgcI03wG/Zg9 DmnCWgp93beC4qWf13PizA7Asw6X0xOA9i30V5tsfp6WDfTGrdhu++YSDpPuVk1B j7PA7XdJP1l8oce5EpOIX9bvmt8cZbH0ro2H6AJx5A5JFFOa5gdkYrhJCyUzemZy 96lfHmiicYNeLFd8Ky/H9xtc0YJA0bbj/3+kakTeHzeNpGaq1hi5H1e9Xl30vGoL 4UAJYfHy2YTng/10M3vbH0NtY2lXfxqvf4gcvvepKH1OBjGJOifqZ9SEoAw+yZhm eSukpLM5Tx+eIhC0UB1tP5U3Kkk16gIo6yUUnDKUBP05cktBad4BGgldfO/MJn2z 5l8v8tfsfkX00Z7GsdK87RNnBpk0ThZhPBdgsWEksC7x9IihP/5Z3Aj5Pr2hWDvD wZi2qZJjhIt7Iw9wf3NUL3n1GMONYy89akluqEYukffE+X2wL5W4kY7JIBOKgHGx hYnTOdnkg1xu5fOMxoWo =ZfNo -----END PGP SIGNATURE----- --JoKCChanR9DHu5K+-- From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 07:11:05 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 084B5279 for ; Mon, 27 May 2013 07:11:05 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 24695874 for ; Mon, 27 May 2013 07:11:03 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id KAA11934; Mon, 27 May 2013 10:10:24 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1UgrZY-0006mU-AF; Mon, 27 May 2013 10:10:24 +0300 Message-ID: <51A306A8.1010201@FreeBSD.org> Date: Mon, 27 May 2013 10:09:28 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130405 Thunderbird/17.0.5 MIME-Version: 1.0 To: Konstantin Belousov Subject: Re: preemptive kernel References: <981733489AB3BD4DB24B48340F53E0A55B0D5590@MTLDAG01.mtl.com> <20130526154752.GT3047@kib.kiev.ua> <981733489AB3BD4DB24B48340F53E0A55B0D56E0@MTLDAG01.mtl.com> <20130527063432.GY3047@kib.kiev.ua> In-Reply-To: <20130527063432.GY3047@kib.kiev.ua> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" , Orit Moskovich X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 07:11:05 -0000 on 27/05/2013 09:34 Konstantin Belousov said the following: > Having both filter and ithread for the same interrupt is apparently > possible but weird. I do not see anything which would prevent interrupt > filter from being executed while the ithread is running. But again, this > is very unusual setup. I wouldn't call it weird, but, yes, it is rare. It's a pretty normal configuration when the filter acts as a filter and the handler acts as a handler (in ithread). In other words, it would be a replacement for a configuration where a filter is used and the filter offloads actual work to non-interrupt context via a e.g. taskqueue. But, hmm, this functionality is probably locked under INTR_FILTER option. -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 07:24:07 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id AF8D1733; Mon, 27 May 2013 07:24:07 +0000 (UTC) (envelope-from oritm@mellanox.com) Received: from eu1sys200aog108.obsmtp.com (eu1sys200aog108.obsmtp.com [207.126.144.125]) by mx1.freebsd.org (Postfix) with ESMTP id 7B26E901; Mon, 27 May 2013 07:24:05 +0000 (UTC) Received: from MTLCAS02.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob108.postini.com ([207.126.147.11]) with SMTP ID DSNKUaMKET/kK4Lr0bQ5DnYmy8SVhHGQFg7j@postini.com; Mon, 27 May 2013 07:24:06 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS02.mtl.com ([10.0.8.72]) with mapi id 14.03.0123.003; Mon, 27 May 2013 10:21:15 +0300 From: Orit Moskovich To: Andriy Gapon , Konstantin Belousov Subject: RE: preemptive kernel Thread-Topic: preemptive kernel Thread-Index: AQHOWihm0YFG89T8IkWOFRvrz7kZiJkYcsLw///vIACAAAnDAIAANDrg Date: Mon, 27 May 2013 07:21:15 +0000 Message-ID: <981733489AB3BD4DB24B48340F53E0A55B0D57D1@MTLDAG01.mtl.com> References: <981733489AB3BD4DB24B48340F53E0A55B0D5590@MTLDAG01.mtl.com> <20130526154752.GT3047@kib.kiev.ua> <981733489AB3BD4DB24B48340F53E0A55B0D56E0@MTLDAG01.mtl.com> <20130527063432.GY3047@kib.kiev.ua> <51A306A8.1010201@FreeBSD.org> In-Reply-To: <51A306A8.1010201@FreeBSD.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 07:24:07 -0000 What is actually the difference between deferring a filter routine's work u= sing an ithread given to bus_setup_intr, or using the global taskqueue_swi = (implemented using interrupt thread)? What do you mean that the functionality is locked under INTR_FILTER? -----Original Message----- From: Andriy Gapon [mailto:avg@FreeBSD.org]=20 Sent: Monday, May 27, 2013 10:11 AM To: Konstantin Belousov Cc: Orit Moskovich; freebsd-hackers@freebsd.org Subject: Re: preemptive kernel on 27/05/2013 09:34 Konstantin Belousov said the following: > Having both filter and ithread for the same interrupt is apparently=20 > possible but weird. I do not see anything which would prevent=20 > interrupt filter from being executed while the ithread is running. =20 > But again, this is very unusual setup. I wouldn't call it weird, but, yes, it is rare. It's a pretty normal confi= guration when the filter acts as a filter and the handler acts as a handler= (in ithread). In other words, it would be a replacement for a configurati= on where a filter is used and the filter offloads actual work to non-interr= upt context via a e.g. taskqueue. But, hmm, this functionality is probably locked under INTR_FILTER option. -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 08:37:42 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9D723687 for ; Mon, 27 May 2013 08:37:42 +0000 (UTC) (envelope-from vhaisman@gmail.com) Received: from mail-la0-x235.google.com (mail-la0-x235.google.com [IPv6:2a00:1450:4010:c03::235]) by mx1.freebsd.org (Postfix) with ESMTP id 2ACE5D7C for ; Mon, 27 May 2013 08:37:41 +0000 (UTC) Received: by mail-la0-f53.google.com with SMTP id ea20so6287801lab.12 for ; Mon, 27 May 2013 01:37:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=BYIeOUlrnTW2QwJDD/P4nS/3a95JQ1tQKmT6EgirZQI=; b=SdTmKtyn+SpWFcrq2TGYl+pp0vEqGwKTrAabQXFN83RmuCFjF5JvZgM3cCfMWLw0Gk 9wvLF0Z4UoUmvI5D8J+1t4XePDGDVhTP6Wtjg83MelxAL+k6LvQxHKSBG9PvtG1xw9Cg MoOsGusGKH+iOC60ZCd/5r8iQkKqyfazwl3Z6JKPerOlq6GNy2NoqRVVa/ZfX1qAWTlD 02G7BVVMZLpGpLCNBP2wxgNOgRQ1Q6FLDFnm/uT6a91kCaXYmkSTU/rDxBo4pyCP6tAA hh/HJunkO5aPCzfV8OWjtra3Vxd+0mH6AqcIahosxqtrEIJPcDZVjMa56tT2iIrLevZC ZGWg== MIME-Version: 1.0 X-Received: by 10.112.20.3 with SMTP id j3mr13483988lbe.90.1369643860141; Mon, 27 May 2013 01:37:40 -0700 (PDT) Received: by 10.114.98.129 with HTTP; Mon, 27 May 2013 01:37:40 -0700 (PDT) In-Reply-To: <2adc4d8e7c55e92d935f61efb4c9d723@lthomas.net> References: <2adc4d8e7c55e92d935f61efb4c9d723@lthomas.net> Date: Mon, 27 May 2013 10:37:40 +0200 Message-ID: Subject: Re: Performance improvement to strnlen(). From: =?UTF-8?Q?V=C3=A1clav_Zeman?= To: Lee Thomas Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 08:37:42 -0000 On 26 May 2013 21:01, Lee Thomas wrote: > On 2013-05-26 08:00, V=C3=A1clav Zeman wrote: >> >> On 05/25/2013 10:27 PM, Lee Thomas wrote: >>> >>> + lp =3D (const unsigned long *)((uintptr_t)str & ~LONGPTR_MASK); >>> + va =3D (*lp - mask01); >>> + vb =3D ((~*lp) & mask80); >> >> I do not think that this correct C. This is type punning violating the >> rules of the language. > > > Hello V=C3=A1clav, > > The aliasing here is safe, because there are no writes through either of = the > pointers, and the reads are correctly aligned. I disagree. IANALL but AFAIK, this is simply not allowed by the language =3D> UB =3D> even though it seems to work in this instance, you are just lucky the UB is actually doing what you expect. -- VZ From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 09:43:01 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DFCFC9C5 for ; Mon, 27 May 2013 09:43:01 +0000 (UTC) (envelope-from florent@peterschmitt.fr) Received: from peterschmitt.fr (peterschmitt.fr [5.135.177.31]) by mx1.freebsd.org (Postfix) with ESMTP id A8C1CC1 for ; Mon, 27 May 2013 09:43:01 +0000 (UTC) Received: from [172.29.180.39] (unknown [194.214.114.46]) by peterschmitt.fr (Postfix) with ESMTPSA id 32A4595D3 for ; Mon, 27 May 2013 11:43:01 +0200 (CEST) Message-ID: <51A32AE6.9010205@peterschmitt.fr> Date: Mon, 27 May 2013 11:44:06 +0200 From: Florent Peterschmitt User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/20130518 Icedove/17.0.5 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: Performance improvement to strnlen(). References: <2adc4d8e7c55e92d935f61efb4c9d723@lthomas.net> In-Reply-To: X-Enigmail-Version: 1.5.1 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="----enig2IFLGDRTAWGBPDRXLKSGF" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: florent+FreeBSD-hackers@peterschmitt.fr List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 09:43:01 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) ------enig2IFLGDRTAWGBPDRXLKSGF Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Le 27/05/2013 10:37, V=C3=A1clav Zeman a =C3=A9crit : > On 26 May 2013 21:01, Lee Thomas wrote: >> On 2013-05-26 08:00, V=C3=A1clav Zeman wrote: >>> >>> On 05/25/2013 10:27 PM, Lee Thomas wrote: >>>> >>>> + lp =3D (const unsigned long *)((uintptr_t)str & ~LONGPTR_MAS= K); >>>> + va =3D (*lp - mask01); >>>> + vb =3D ((~*lp) & mask80); >>> >>> I do not think that this correct C. This is type punning violating th= e >>> rules of the language. >> >> >> Hello V=C3=A1clav, >> >> The aliasing here is safe, because there are no writes through either = of the >> pointers, and the reads are correctly aligned. > I disagree. IANALL but AFAIK, this is simply not allowed by the > language =3D> UB =3D> even though it seems to work in this instance, yo= u > are just lucky the UB is actually doing what you expect. In that case we should rewrite strlen, it's the same code. That doesn't mean it's a good code but I really don't think it's bad. Using signedness is totally valid and what is done here appears valid too. > -- > VZ --=20 Florent Peterschmitt | /"\ ASCII Ribbon Campaign florent@peterschmitt.fr | \ / - No HTML/RTF in E-mail +33 (0)6 64 33 97 92 | X - No proprietary attachments http://florent.peterschmitt.fr | / \ - Respect for open standards ------enig2IFLGDRTAWGBPDRXLKSGF Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (GNU/Linux) Comment: Using GnuPG with Icedove - http://www.enigmail.net/ iQEcBAEBAgAGBQJRoyrmAAoJEMtO2Sol0IImqq8H/j1LDig+rnjwM870atgOonEA KWGPYQSgoi1yyYCcHR/74hX6F1nkSK+2xCDI+ZLPge5BUFwKlxw6qNHzG+smOWlB iS2omHNgApV9XsNgVxWf8iGO9s74VkmVBuaY1TFIZEk2Qd6J29dLF70dhQN83+24 H4AByalis6L/3Pxe1/wN3R0x26Dcjw7R6tiw4G6IrNzqlsmKdREIzQPL9q3vFti2 t0T/ZsYkYzyA7A2CJt4Mpq+KBLEDJ9U8BZ2FwssfiwQi9QSSsfBc03ZSvFBUN0xs mT3oSEzha0Vm1qPg3Y7T5EQi6KkyeI7rol69doqRQZn29Or/xsweHgjRcSzrH6E= =j9T0 -----END PGP SIGNATURE----- ------enig2IFLGDRTAWGBPDRXLKSGF-- From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 10:27:15 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 94F2E7BD for ; Mon, 27 May 2013 10:27:15 +0000 (UTC) (envelope-from lee_thomas@aslantools.com) Received: from roproxy2-pub.unifiedlayer.com (roproxy2-pub.unifiedlayer.com [69.89.18.3]) by mx1.freebsd.org (Postfix) with SMTP id 6CFB4285 for ; Mon, 27 May 2013 10:27:15 +0000 (UTC) Received: (qmail 12220 invoked by uid 0); 27 May 2013 10:20:32 -0000 Received: from unknown (HELO mailchannelsproxy2.unifiedlayer.com) (66.147.243.71) by roproxy2.unifiedlayer.com with SMTP; 27 May 2013 10:20:32 -0000 X-Sender-Id: {:fast21.fastdomain.com:lthomasn:fast21.fastdomain.com} {sentby:program running on server} Received: from fast21.fastdomain.com (fast21.fastdomain.com [74.220.199.21]) by 0.0.0.0:2500 (trex/4.8.69); Mon, 27 May 2013 10:20:32 GMT Received: from localhost ([127.0.0.1]:54553 helo=fast21.fastdomain.com) by fast21.fastdomain.com with esmtp (Exim 4.80) (envelope-from ) id 1UguXY-0000IC-8x for freebsd-hackers@freebsd.org; Mon, 27 May 2013 04:20:32 -0600 To: Subject: Re: Performance improvement to strnlen(). X-PHP-Script: aslantools.com/mail/index.php for 173.71.114.222 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Mon, 27 May 2013 06:20:32 -0400 From: Lee Thomas Organization: Aslan Tools In-Reply-To: References: <2adc4d8e7c55e92d935f61efb4c9d723@lthomas.net> Message-ID: X-Sender: lee_thomas@aslantools.com User-Agent: Roundcube Webmail/0.8.4 X-Identified-User: {:fast21.fastdomain.com:lthomasn:fast21.fastdomain.com} {sentby:program running on server} X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 10:27:15 -0000 On 2013-05-27 04:37, Václav Zeman wrote: > On 26 May 2013 21:01, Lee Thomas wrote: >> On 2013-05-26 08:00, Václav Zeman wrote: >>> >>> On 05/25/2013 10:27 PM, Lee Thomas wrote: >>>> >>>> + lp = (const unsigned long *)((uintptr_t)str & >>>> ~LONGPTR_MASK); >>>> + va = (*lp - mask01); >>>> + vb = ((~*lp) & mask80); >>> >>> I do not think that this correct C. This is type punning violating >>> the >>> rules of the language. >> >> >> Hello Václav, >> >> The aliasing here is safe, because there are no writes through >> either of the >> pointers, and the reads are correctly aligned. > I disagree. IANALL but AFAIK, this is simply not allowed by the > language => UB => even though it seems to work in this instance, you > are just lucky the UB is actually doing what you expect. > > -- > VZ Hello Václav, I am not an expert in C either, so you may be right that this is technically illegal. However, I copied this code from strlen.c, which has had it, and still has it, for 4.5 years, and I can't see any way any alias analysis done by the compiler could invalidate this code. In addition, there are many places in the kernel, and in other codebases I've worked on, where this kind of type conversion is done. See for instance /sys/amd64/amd64/vm_macdep.c:200, where we compute the base of a thread's stackframe from a pointer to an unrelated type of 'struct pcb', and then write to it. I am willing to uglify the code in the way you suggest if that is the general concensus, but I think the code as it stands is both safe and more legible. Thanks, Lee From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 10:31:02 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B24E88FD for ; Mon, 27 May 2013 10:31:02 +0000 (UTC) (envelope-from nowakpl@platinum.linux.pl) Received: from platinum.linux.pl (platinum.edu.pl [81.161.192.4]) by mx1.freebsd.org (Postfix) with ESMTP id 771732B7 for ; Mon, 27 May 2013 10:31:01 +0000 (UTC) Received: by platinum.linux.pl (Postfix, from userid 87) id 5087D5FD07; Mon, 27 May 2013 12:25:39 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on platinum.linux.pl X-Spam-Level: X-Spam-Status: No, score=-1.3 required=3.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.3.2 Received: from [10.255.0.2] (unknown [83.151.38.73]) by platinum.linux.pl (Postfix) with ESMTPA id 17BD15FD05 for ; Mon, 27 May 2013 12:25:39 +0200 (CEST) Message-ID: <51A33493.9010702@platinum.linux.pl> Date: Mon, 27 May 2013 12:25:23 +0200 From: Adam Nowacki User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: Performance improvement to strnlen(). References: <2adc4d8e7c55e92d935f61efb4c9d723@lthomas.net> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 10:31:02 -0000 On 2013-05-27 10:37, Václav Zeman wrote: > On 26 May 2013 21:01, Lee Thomas wrote: >> On 2013-05-26 08:00, Václav Zeman wrote: >>> >>> On 05/25/2013 10:27 PM, Lee Thomas wrote: >>>> >>>> + lp = (const unsigned long *)((uintptr_t)str & ~LONGPTR_MASK); >>>> + va = (*lp - mask01); >>>> + vb = ((~*lp) & mask80); >>> >>> I do not think that this correct C. This is type punning violating the >>> rules of the language. >> >> >> Hello Václav, >> >> The aliasing here is safe, because there are no writes through either of the >> pointers, and the reads are correctly aligned. > I disagree. IANALL but AFAIK, this is simply not allowed by the > language => UB => even though it seems to work in this instance, you > are just lucky the UB is actually doing what you expect. It is not possible to tell if the result would be undefined by looking at the strnlen function alone. Internally it doesn't break any aliasing rules as char and long are allowed to alias. UB can only happen when the input string was created with incompatible type (not char and not long) and the strnlen function got inlined. Preventing inlining would be sufficient to guarantee correctness in any case. From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 11:46:06 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 44BA27D1 for ; Mon, 27 May 2013 11:46:06 +0000 (UTC) (envelope-from vhaisman@gmail.com) Received: from mail-lb0-f179.google.com (mail-lb0-f179.google.com [209.85.217.179]) by mx1.freebsd.org (Postfix) with ESMTP id C5E99A80 for ; Mon, 27 May 2013 11:46:05 +0000 (UTC) Received: by mail-lb0-f179.google.com with SMTP id r11so6683712lbv.38 for ; Mon, 27 May 2013 04:45:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=si9OEuknzCUOgEwNbKP6f68BuX8IgDFAIcxqNuE2ZKA=; b=K8egLS0yp3kmTHhiPEoaXwH09DGGVafE6AFLiT7HRo5+zcOj/9caUKHijkouTqoxsM VspFAkYmRCUfqIKe6CulYZjGKlIg8oVVmkgRhpDvwey52iYeKtV9nJX4vB+U3/whc+92 Yu4PyBlXpZ8Xs3MflwGTfJyBBALRA/qXimZ/EsfAcZ5G9mMmdcLe8TOZzLmwLr9S8x1f yOS8GzGlloz2mBugD8kT0rJLZG46mpKfHTPmsE+vfUxclcxcLbiaEM7BkSDlMP3B9ksq IqgUjh2Saykkdp5nb8JDjzlpvNm4Mv7xKj07cjRYF3ji/XFuXeL1PProi6jjiz7lvPYz l9Vg== MIME-Version: 1.0 X-Received: by 10.112.126.9 with SMTP id mu9mr13939501lbb.99.1369655158526; Mon, 27 May 2013 04:45:58 -0700 (PDT) Received: by 10.114.98.129 with HTTP; Mon, 27 May 2013 04:45:58 -0700 (PDT) In-Reply-To: <51A33493.9010702@platinum.linux.pl> References: <2adc4d8e7c55e92d935f61efb4c9d723@lthomas.net> <51A33493.9010702@platinum.linux.pl> Date: Mon, 27 May 2013 13:45:58 +0200 Message-ID: Subject: Re: Performance improvement to strnlen(). From: =?UTF-8?Q?V=C3=A1clav_Zeman?= To: Adam Nowacki Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 11:46:06 -0000 On 27 May 2013 12:25, Adam Nowacki wrote: > On 2013-05-27 10:37, V=C3=A1clav Zeman wrote: >> >> On 26 May 2013 21:01, Lee Thomas wrote: >>> >>> On 2013-05-26 08:00, V=C3=A1clav Zeman wrote: >>>> >>>> >>>> On 05/25/2013 10:27 PM, Lee Thomas wrote: >>>>> >>>>> >>>>> + lp =3D (const unsigned long *)((uintptr_t)str & ~LONGPTR_MASK= ); >>>>> + va =3D (*lp - mask01); >>>>> + vb =3D ((~*lp) & mask80); >>>> >>>> >>>> I do not think that this correct C. This is type punning violating the >>>> rules of the language. >>> >>> >>> >>> Hello V=C3=A1clav, >>> >>> The aliasing here is safe, because there are no writes through either o= f >>> the >>> pointers, and the reads are correctly aligned. >> >> I disagree. IANALL but AFAIK, this is simply not allowed by the >> language =3D> UB =3D> even though it seems to work in this instance, you >> are just lucky the UB is actually doing what you expect. > > > It is not possible to tell if the result would be undefined by looking at > the strnlen function alone. IMHO, it is possible to look at the strnlen() function code and see the UB there. UB is is there even if it does not manifest itself to you by producing executable that gives you unexpected results or that crashes. > Internally it doesn't break any aliasing rules > as char and long are allowed to alias. UB can only happen when the input > string was created with incompatible type (not char and not long) and the > strnlen function got inlined. No, you got it wrong here. You can access any type of object by char pointer. However the reverse is not true. > Preventing inlining would be sufficient to > guarantee correctness in any case. Preventing inlining is masking the problem but it does not make it go away. Also, in these days when compilers can optimize across translation units, you cannot be sure the UB will be masked even if no inlining happens. -- VZ From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 11:55:49 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 208E5BA8 for ; Mon, 27 May 2013 11:55:49 +0000 (UTC) (envelope-from vhaisman@gmail.com) Received: from mail-la0-x22b.google.com (mail-la0-x22b.google.com [IPv6:2a00:1450:4010:c03::22b]) by mx1.freebsd.org (Postfix) with ESMTP id A1138B15 for ; Mon, 27 May 2013 11:55:48 +0000 (UTC) Received: by mail-la0-f43.google.com with SMTP id ez20so6427291lab.30 for ; Mon, 27 May 2013 04:55:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; bh=lF2t13jXSyNvsiPoqlPMNLRokW8ApAArmdGlThWz9hQ=; b=gEQbm8AXDINCvshlKInMdwF21FISzJXrAjLfBhY9sm5z/AEQ/Yv0rVn5QkhXV9wlCh 0Qw5sqju14PGEhH8UU+XdjDM+DkiO9x5r47Ntw8208A12+3gFSyStbjzphRpvSAAZz8o kQvFPSc2DaH+iU3OxijbozI+30YI5Vg6JRt8qhps/5Oesr7+RCyrKw9GPWz3f7O1vVv3 2t0V4Um+0gKEc+d3UL52cfObB5vV7LeLboH1FgHt8ZmUTCsAcVVHORyV3tQ6IuO/W81Z KANayczBTSUv06HyEuxXUE/JRGnWMB34Dmwk12iQQb8piY6dIs3xHN0QlIYBOGzISbeb Vd5Q== MIME-Version: 1.0 X-Received: by 10.152.27.194 with SMTP id v2mr3091007lag.22.1369655267298; Mon, 27 May 2013 04:47:47 -0700 (PDT) Received: by 10.114.98.129 with HTTP; Mon, 27 May 2013 04:47:47 -0700 (PDT) In-Reply-To: References: <2adc4d8e7c55e92d935f61efb4c9d723@lthomas.net> Date: Mon, 27 May 2013 13:47:47 +0200 Message-ID: Subject: Re: Performance improvement to strnlen(). From: =?UTF-8?Q?V=C3=A1clav_Zeman?= To: Lee Thomas Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 11:55:49 -0000 On 27 May 2013 12:20, Lee Thomas wrote: > On 2013-05-27 04:37, V=C3=A1clav Zeman wrote: >> >> On 26 May 2013 21:01, Lee Thomas wrote: >>> >>> On 2013-05-26 08:00, V=C3=A1clav Zeman wrote: >>>> >>>> >>>> On 05/25/2013 10:27 PM, Lee Thomas wrote: >>>>> >>>>> >>>>> + lp =3D (const unsigned long *)((uintptr_t)str & ~LONGPTR_MASK= ); >>>>> + va =3D (*lp - mask01); >>>>> + vb =3D ((~*lp) & mask80); >>>> >>>> >>>> I do not think that this correct C. This is type punning violating the >>>> rules of the language. >>> >>> >>> >>> Hello V=C3=A1clav, >>> >>> The aliasing here is safe, because there are no writes through either o= f >>> the >>> pointers, and the reads are correctly aligned. >> >> I disagree. IANALL but AFAIK, this is simply not allowed by the >> language =3D> UB =3D> even though it seems to work in this instance, you >> are just lucky the UB is actually doing what you expect. >> >> -- >> VZ > > > Hello V=C3=A1clav, > > I am not an expert in C either, so you may be right that this is technica= lly > illegal. However, I copied this code from strlen.c, which has had it, and > still has it, for 4.5 years, and I can't see any way any alias analysis d= one > by the compiler could invalidate this code. In addition, there are many > places in the kernel, and in other codebases I've worked on, where this k= ind > of type conversion is done. See for instance > /sys/amd64/amd64/vm_macdep.c:200, where we compute the base of a thread's > stackframe from a pointer to an unrelated type of 'struct pcb', and then > write to it. > > I am willing to uglify the code in the way you suggest if that is the > general concensus, but I think the code as it stands is both safe and mor= e > legible. You could always put the three lines into a macro to keep the strnlen() more readable. -- VZ From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 12:17:54 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E1B7994B for ; Mon, 27 May 2013 12:17:54 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 3A369E59 for ; Mon, 27 May 2013 12:17:53 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id PAA19511; Mon, 27 May 2013 15:17:47 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <51A34EEA.9050609@FreeBSD.org> Date: Mon, 27 May 2013 15:17:46 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130517 Thunderbird/17.0.6 MIME-Version: 1.0 To: Orit Moskovich Subject: Re: preemptive kernel References: <981733489AB3BD4DB24B48340F53E0A55B0D5590@MTLDAG01.mtl.com> <20130526154752.GT3047@kib.kiev.ua> <981733489AB3BD4DB24B48340F53E0A55B0D56E0@MTLDAG01.mtl.com> <20130527063432.GY3047@kib.kiev.ua> <51A306A8.1010201@FreeBSD.org> <981733489AB3BD4DB24B48340F53E0A55B0D57D1@MTLDAG01.mtl.com> In-Reply-To: <981733489AB3BD4DB24B48340F53E0A55B0D57D1@MTLDAG01.mtl.com> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: Konstantin Belousov , "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 12:17:54 -0000 on 27/05/2013 10:21 Orit Moskovich said the following: > What is actually the difference between deferring a filter routine's work using an ithread given to bus_setup_intr, or using the global taskqueue_swi (implemented using interrupt thread)? I think you mean taskqueue_fast. The difference is only in how much code you need to write. I do not think there is any significant difference in the resulting functionality. > What do you mean that the functionality is locked under INTR_FILTER? Please see the code. You have to use option INTR_FILTER to get the behavior I described earlier. > -----Original Message----- > From: Andriy Gapon [mailto:avg@FreeBSD.org] > Sent: Monday, May 27, 2013 10:11 AM > To: Konstantin Belousov > Cc: Orit Moskovich; freebsd-hackers@freebsd.org > Subject: Re: preemptive kernel > > on 27/05/2013 09:34 Konstantin Belousov said the following: >> Having both filter and ithread for the same interrupt is apparently >> possible but weird. I do not see anything which would prevent >> interrupt filter from being executed while the ithread is running. >> But again, this is very unusual setup. > > I wouldn't call it weird, but, yes, it is rare. It's a pretty normal configuration when the filter acts as a filter and the handler acts as a handler (in ithread). In other words, it would be a replacement for a configuration where a filter is used and the filter offloads actual work to non-interrupt context via a e.g. taskqueue. > But, hmm, this functionality is probably locked under INTR_FILTER option. > > -- > Andriy Gapon > -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 12:29:47 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 6CFD8EBA; Mon, 27 May 2013 12:29:47 +0000 (UTC) (envelope-from oritm@mellanox.com) Received: from eu1sys200aog120.obsmtp.com (eu1sys200aog120.obsmtp.com [207.126.144.149]) by mx1.freebsd.org (Postfix) with ESMTP id DC101EF1; Mon, 27 May 2013 12:29:44 +0000 (UTC) Received: from MTLCAS01.mtl.com ([193.47.165.155]) (using TLSv1) by eu1sys200aob120.postini.com ([207.126.147.11]) with SMTP ID DSNKUaNRoDa4ba30RnxD60XJmtW7fUIvaSZv@postini.com; Mon, 27 May 2013 12:29:46 UTC Received: from MTLDAG01.mtl.com ([10.0.8.75]) by MTLCAS01.mtl.com ([10.0.8.71]) with mapi id 14.03.0123.003; Mon, 27 May 2013 15:29:19 +0300 From: Orit Moskovich To: Andriy Gapon Subject: RE: preemptive kernel Thread-Topic: preemptive kernel Thread-Index: AQHOWihm0YFG89T8IkWOFRvrz7kZiJkYcsLw///vIACAAAnDAIAANDrggAAh6QCAADRysA== Date: Mon, 27 May 2013 12:29:19 +0000 Message-ID: <981733489AB3BD4DB24B48340F53E0A55B0D59A0@MTLDAG01.mtl.com> References: <981733489AB3BD4DB24B48340F53E0A55B0D5590@MTLDAG01.mtl.com> <20130526154752.GT3047@kib.kiev.ua> <981733489AB3BD4DB24B48340F53E0A55B0D56E0@MTLDAG01.mtl.com> <20130527063432.GY3047@kib.kiev.ua> <51A306A8.1010201@FreeBSD.org> <981733489AB3BD4DB24B48340F53E0A55B0D57D1@MTLDAG01.mtl.com> <51A34EEA.9050609@FreeBSD.org> In-Reply-To: <51A34EEA.9050609@FreeBSD.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.0.13.1] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: Konstantin Belousov , "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 12:29:47 -0000 >From what I've read in subr_taskqueue.c taskqueue_swi, taskqueue_swi_giant = and taskqueue_fast are all implemented using swi_add which calls ithread_cr= eate(). Is there any performance difference between them. Is one of the above or it= hread given to bus_setup_intr preferable on the other? -----Original Message----- From: Andriy Gapon [mailto:avg@FreeBSD.org]=20 Sent: Monday, May 27, 2013 03:18 PM To: Orit Moskovich Cc: Konstantin Belousov; freebsd-hackers@freebsd.org Subject: Re: preemptive kernel on 27/05/2013 10:21 Orit Moskovich said the following: > What is actually the difference between deferring a filter routine's work= using an ithread given to bus_setup_intr, or using the global taskqueue_sw= i (implemented using interrupt thread)? I think you mean taskqueue_fast. The difference is only in how much code you need to write. I do not think = there is any significant difference in the resulting functionality. > What do you mean that the functionality is locked under INTR_FILTER? Please see the code. You have to use option INTR_FILTER to get the behavio= r I described earlier. > -----Original Message----- > From: Andriy Gapon [mailto:avg@FreeBSD.org] > Sent: Monday, May 27, 2013 10:11 AM > To: Konstantin Belousov > Cc: Orit Moskovich; freebsd-hackers@freebsd.org > Subject: Re: preemptive kernel >=20 > on 27/05/2013 09:34 Konstantin Belousov said the following: >> Having both filter and ithread for the same interrupt is apparently=20 >> possible but weird. I do not see anything which would prevent=20 >> interrupt filter from being executed while the ithread is running. >> But again, this is very unusual setup. >=20 > I wouldn't call it weird, but, yes, it is rare. It's a pretty normal con= figuration when the filter acts as a filter and the handler acts as a handl= er (in ithread). In other words, it would be a replacement for a configura= tion where a filter is used and the filter offloads actual work to non-inte= rrupt context via a e.g. taskqueue. > But, hmm, this functionality is probably locked under INTR_FILTER option. >=20 > -- > Andriy Gapon >=20 -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 14:35:58 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7BC08675 for ; Mon, 27 May 2013 14:35:58 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id C69317E3 for ; Mon, 27 May 2013 14:35:57 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id RAA22152; Mon, 27 May 2013 17:35:49 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <51A36F44.8050206@FreeBSD.org> Date: Mon, 27 May 2013 17:35:48 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130517 Thunderbird/17.0.6 MIME-Version: 1.0 To: Orit Moskovich Subject: Re: preemptive kernel References: <981733489AB3BD4DB24B48340F53E0A55B0D5590@MTLDAG01.mtl.com> <20130526154752.GT3047@kib.kiev.ua> <981733489AB3BD4DB24B48340F53E0A55B0D56E0@MTLDAG01.mtl.com> <20130527063432.GY3047@kib.kiev.ua> <51A306A8.1010201@FreeBSD.org> <981733489AB3BD4DB24B48340F53E0A55B0D57D1@MTLDAG01.mtl.com> <51A34EEA.9050609@FreeBSD.org> <981733489AB3BD4DB24B48340F53E0A55B0D59A0@MTLDAG01.mtl.com> In-Reply-To: <981733489AB3BD4DB24B48340F53E0A55B0D59A0@MTLDAG01.mtl.com> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 14:35:58 -0000 [trimmed cc] on 27/05/2013 15:29 Orit Moskovich said the following: >>From what I've read in subr_taskqueue.c taskqueue_swi, taskqueue_swi_giant and taskqueue_fast are all implemented using swi_add which calls ithread_create(). > Is there any performance difference between them. Is one of the above or ithread given to bus_setup_intr preferable on the other? The differences are described in taskqueue(9) "Predefined Task Queues" section. -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 15:49:48 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 99873DA6 for ; Mon, 27 May 2013 15:49:48 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 88728ACD for ; Mon, 27 May 2013 15:49:48 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id 7749A1A3C47 for ; Mon, 27 May 2013 08:49:43 -0700 (PDT) Message-ID: <51A38095.5070704@mu.org> Date: Mon, 27 May 2013 08:49:41 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <51A2404F.5040908@erdgeist.org> In-Reply-To: <51A2404F.5040908@erdgeist.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 15:49:48 -0000 On 5/26/13 10:03 AM, Dirk Engling wrote: > On 26.05.13 04:51, Super Bisquit wrote: >> Please don't turn this into an architecture dependent mess. PCBSD is >> i386 & AMD64 only. > Read my email thoroughly and notice that I never seriously considered > using pc-sysinstall after looking into it. Don't worry. > Why is that exactly? A number of people are using it successfully to install ZFS based systems. -Alfred From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 15:57:13 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 162B64D0 for ; Mon, 27 May 2013 15:57:13 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 0642AB2E for ; Mon, 27 May 2013 15:57:12 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id E27DE1A3C42 for ; Mon, 27 May 2013 08:48:35 -0700 (PDT) Message-ID: <51A38051.8040909@mu.org> Date: Mon, 27 May 2013 08:48:33 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> In-Reply-To: <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 15:57:13 -0000 On 5/25/13 8:45 PM, Teske, Devin wrote: > On May 25, 2013, at 7:51 PM, Super Bisquit wrote: > >> Please don't turn this into an architecture dependent mess. PCBSD is i386 & >> AMD64 only. >> > There's a GSoC project (of which I'm potential mentor) to fix that. > > However, you are entirely right… we can't in all seriousness even think about using pc-sysinstall until it is solid on all architectures as bsdinstall already is. > > GSoC project is: "Making pc-sysinstall FreeBSD ready by porting it to multiple architectures" Why can we not use in the interim use pc-sysinstall on the platforms that it performs best on and use bsdinstall on the others? It doesn't make sense for us to hold up some platform like this at all. Maybe no one has thought of this? Basically use pc-sysinstall on amd64 and i386 and use bsdinstall on the other platforms until they catch up? -Alfred From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 16:48:03 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9A8FBBC8; Mon, 27 May 2013 16:48:03 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-ie0-x236.google.com (mail-ie0-x236.google.com [IPv6:2607:f8b0:4001:c03::236]) by mx1.freebsd.org (Postfix) with ESMTP id 5140FE15; Mon, 27 May 2013 16:48:03 +0000 (UTC) Received: by mail-ie0-f182.google.com with SMTP id a14so19126234iee.27 for ; Mon, 27 May 2013 09:48:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=XQ2GptU8sPSL6kTTzvVDg27aO9kpFjwAp8vy3mrDuIw=; b=fKgrJFIUhYgkUODY89PAmDstof71AQa6EO05J538q8vHOGwEV3v5+H91xP55ijx6Bg ABHPVsizphT8+1Wab18DBVqFVgWfE9jdSfzLKOl1ykCBb3mMiTxAzbY5j9NxYTUT/8ml Anmldumr+b88Al4RiFwO5k2AUB9ZsDgFw7mRAa1s2eo2/3V3WLQFMGYHHlRoxC5hxwMd OvdcNB1ykGbFx3T4ygTpw90Yu5UYCsoldRm8gdUgDuwt4MQHrNks3JWYZxXgGf6vStHD sG7GuK2f3XeDy+evd5XlAAFgTmlWlAJcA+JiMB95rx+ui0ATl+l2W+OEv+ACPu806cM9 vYBQ== MIME-Version: 1.0 X-Received: by 10.50.153.113 with SMTP id vf17mr5071844igb.101.1369673281570; Mon, 27 May 2013 09:48:01 -0700 (PDT) Received: by 10.64.23.243 with HTTP; Mon, 27 May 2013 09:48:01 -0700 (PDT) Received: by 10.64.23.243 with HTTP; Mon, 27 May 2013 09:48:01 -0700 (PDT) In-Reply-To: References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B2E7@ltcfiswmsgmb26> <51A29A5F.7010106@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F61FE0@ltcfiswmsgmb21> Date: Mon, 27 May 2013 17:48:01 +0100 Message-ID: Subject: Re: FreeBSD installers and future direction From: Chris Rees To: Daniel Eischen Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Devin Teske , freebsd-hackers@freebsd.org, Dirk Engling , Nathan Whitehorn X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 16:48:03 -0000 On 27 May 2013 03:10, "Daniel Eischen" wrote: > > On Mon, 27 May 2013, Teske, Devin wrote: >> >> >> I don't think there's any reason why we have to write it in C if we can write >> it in sh. > > > I don't really care one way or the other (C or sh), but > I can say that I can understand(*) well structured C a lot > better than well structured sh. Having something more > strongly typed certainly helps understanding. > > (*) Assuming some level of complexity (I know that's > subjective). I think it's all down to familiarity. I suppose sh is more resistant to many stupid bugs and handles strings well.... But it has its own troubles too of course. Chris From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 16:56:32 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E2238EF1 for ; Mon, 27 May 2013 16:56:32 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [IPv6:2a01:348:0:15:5d59:5c40:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id A4B9CE5D for ; Mon, 27 May 2013 16:56:32 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id A03F9E6A40; Mon, 27 May 2013 17:56:28 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=mail; bh=HiJN35mrkSsB Pt5TVMJ3IVDlZlw=; b=s6e8Hut8Xk/de9v/9u6PRG/fBJ5Wg75tJPRTrQSaWet/ 5owjGqhJDZcEbKgOCRfbENgLibiiA/9AahEm4sdDFAqijsju16t8pY8gc529gKC/ JHtAAiziTPnox/HjeLeibnghoA26AJ0X9edORSE9r644TWNN2HsFHJnXYpIYAJw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=mail; b=NuHVcV 2VutpKTKurBcouEJ5+j1MZt0a7HHi+e3ySwK59Yx0eAB8Zsh/K5/Vr74AnL70dYQ p9K4IADfHkmtSAVbmLXEofO/PH1oMA7wlMDsaizcI4/y2cUK9AKUTH4UaA3LcGuL +7V0B1GifK17tAND+esDbnUMiuCbbyPbiNWAQ= Received: from [IPv6:2a01:348:301:2:ccf1:925a:43f8:4bbc] (unknown [IPv6:2a01:348:301:2:ccf1:925a:43f8:4bbc]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id 7E9B2E6A3D; Mon, 27 May 2013 17:56:28 +0100 (BST) Message-ID: <51A39039.1070202@cran.org.uk> Date: Mon, 27 May 2013 17:56:25 +0100 From: Bruce Cran User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Alfred Perlstein Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> In-Reply-To: <51A38051.8040909@mu.org> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 16:56:32 -0000 On 27/05/2013 16:48, Alfred Perlstein wrote: > Why can we not use in the interim use pc-sysinstall on the platforms > that it performs best on and use bsdinstall on the others? Because pc-sysinstall doesn't have a UI - it's only a backend. If we update bsdinstall to use it, then it won't work on other platforms. -- Bruce From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 17:59:34 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id DD90F4B4 for ; Mon, 27 May 2013 17:59:34 +0000 (UTC) (envelope-from dt71@gmx.com) Received: from mout.gmx.net (mout.gmx.net [212.227.15.15]) by mx1.freebsd.org (Postfix) with ESMTP id 68665189 for ; Mon, 27 May 2013 17:59:34 +0000 (UTC) Received: from [192.168.1.80] ([78.92.216.13]) by mail.gmx.com (mrgmx002) with ESMTPSA (Nemesis) id 0MAgvb-1UZcnB21c3-00BrMC for ; Mon, 27 May 2013 19:59:30 +0200 Message-ID: <51A39EF8.6020700@gmx.com> Date: Mon, 27 May 2013 19:59:20 +0200 From: dt71@gmx.com User-Agent: Mozilla/5.0 (X11; FreeBSD i386; rv:20.0) Gecko/20100101 Firefox/20.0 SeaMonkey/2.17 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: >3955MiB of swap space Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Provags-ID: V03:K0:r8KdtqtY2ODEiNfbuth/ccypistekwzomxol2ttq4hUvjVFr0Ni XBbK4GdlTR0rPY6vqVpSiMPGVR47dyOcWx3j5bq1aYmfJ9nMSh5y2h/wBHzhVXDFCLrEJad 1pI/e+tAQkmQ531SBhobvyCC3LdFF55Wb9rqxriez7BFS9xSoFcKAVwS03dbGEHsXUJbaNx jucHNpOfAjTbbnLqabr4w== X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 17:59:34 -0000 I have 4 hard drives, each containing a swap partition of size 1023MiB. I get: warning: total configured swap (1178880 pages) exceeds maximum recommended amount (1012480 pages). warning: increase kern.maxswzone or reduce amount of swap. Is the warning safe to ignore? I assume that only 3955MiB of swap space will be used instead of 4092MiB, because using more would cause overhead. I do not prefer to repartition the drives for them to have swap partitions each of size (3955/4)MiB. By the way, is swapping distributed evenly among the drives? How? Is there a downfall when one of the drives is outstandingly slow? From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 18:03:31 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 7C6E16AB for ; Mon, 27 May 2013 18:03:31 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 6C3E51E3 for ; Mon, 27 May 2013 18:03:31 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id 50E6D1A3C39; Mon, 27 May 2013 11:03:26 -0700 (PDT) Message-ID: <51A39FEC.5070402@mu.org> Date: Mon, 27 May 2013 11:03:24 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Bruce Cran Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> <51A39039.1070202@cran.org.uk> In-Reply-To: <51A39039.1070202@cran.org.uk> Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 18:03:31 -0000 On 5/27/13 9:56 AM, Bruce Cran wrote: > On 27/05/2013 16:48, Alfred Perlstein wrote: >> Why can we not use in the interim use pc-sysinstall on the platforms >> that it performs best on and use bsdinstall on the others? > > Because pc-sysinstall doesn't have a UI - it's only a backend. If we > update bsdinstall to use it, then it won't work on other platforms. > This still doesn't make sense to me. Why can bsdinstall not conditionally use it? Do we always have to seek the lowest common denominator for our user experience? -Alfred From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 18:26:22 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 069144B1 for ; Mon, 27 May 2013 18:26:22 +0000 (UTC) (envelope-from kientzle@freebsd.org) Received: from monday.kientzle.com (99-115-135-74.uvs.sntcca.sbcglobal.net [99.115.135.74]) by mx1.freebsd.org (Postfix) with ESMTP id E1A9034A for ; Mon, 27 May 2013 18:26:21 +0000 (UTC) Received: (from root@localhost) by monday.kientzle.com (8.14.4/8.14.4) id r4RIQGkB007907; Mon, 27 May 2013 18:26:16 GMT (envelope-from kientzle@freebsd.org) Received: from [192.168.2.123] (CiscoE3000 [192.168.1.65]) by kientzle.com with SMTP id 365ruxq877ru9tmbnw6ukkjqb6; Mon, 27 May 2013 18:26:16 +0000 (UTC) (envelope-from kientzle@freebsd.org) Subject: Re: Performance improvement to strnlen(). Mime-Version: 1.0 (Apple Message framework v1283) Content-Type: text/plain; charset=windows-1252 From: Tim Kientzle In-Reply-To: Date: Mon, 27 May 2013 11:26:14 -0700 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Lee Thomas X-Mailer: Apple Mail (2.1283) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 18:26:22 -0000 >=20 > Index: strnlen.c > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > diff --git a/head/lib/libc/string/strnlen.c = b/head/lib/libc/string/strnlen.c > --- a/head/lib/libc/string/strnlen.c (revision 250951) > +++ b/head/lib/libc/string/strnlen.c (working copy) > @@ -1,5 +1,6 @@ > /*- > - * Copyright (c) 2009 David Schultz > + * Copyright (c) 2009, 2010 Xin LI > + * Copyright (c) 2013 Lee Thomas > * All rights reserved. > * > * Redistribution and use in source and binary forms, with or without > @@ -27,16 +28,91 @@ > #include > __FBSDID("$FreeBSD$"); > =20 > +#include > +#include > #include > =20 > +/* > + * Portable strnlen() for 32-bit and 64-bit systems. > + * > + * Rationale: it is generally much more efficient to do word length > + * operations and avoid branches on modern computer systems, as > + * compared to byte-length operations with a lot of branches. > + * > + * The expression: > + * > + * ((x - 0x01....01) & ~x & 0x80....80) > + * > + * would evaluate to a non-zero value iff any of the bytes in the > + * original word is zero. > + * > + * On multi-issue processors, we can divide the above expression = into: > + * a) (x - 0x01....01) > + * b) (~x & 0x80....80) > + * c) a & b > + * > + * Where, a) and b) can be partially computed in parallel. > + * > + * The algorithm above is found on "Hacker's Delight" by > + * Henry S. Warren, Jr. > + */ > + > +/* Magic numbers for the algorithm */ > +#if LONG_BIT =3D=3D 32 > +static const unsigned long mask01 =3D 0x01010101; > +static const unsigned long mask80 =3D 0x80808080; > +#elif LONG_BIT =3D=3D 64 > +static const unsigned long mask01 =3D 0x0101010101010101; > +static const unsigned long mask80 =3D 0x8080808080808080; > +#else > +#error Unsupported word size > +#endif > + > +#define LONGPTR_MASK (sizeof(long) - 1) I would not use this at all; (sizeof(long) - 1) is a common expression that your audience probably understands more quickly than "LONGPTR_MASK". > + > size_t > -strnlen(const char *s, size_t maxlen) > +strnlen(const char *str, size_t maxlen) > { > - size_t len; > + const char *stop, *short_stop; > + const char *p; > + const unsigned long *lp; > + long va, vb; > =20 > - for (len =3D 0; len < maxlen; len++, s++) { > - if (!*s) > - break; > + if (maxlen=3D=3D0) return 0; > + > + stop=3Dstr+maxlen; > + /* > + * Before trying the hard (unaligned byte-by-byte access) way > + * to figure out whether there is a nul character, try to see > + * if there is a nul character is within this accessible word > + * first. > + * > + * p and (p & ~LONGPTR_MASK) must be equally accessible since > + * they always fall in the same memory page, as long as page > + * boundaries is integral multiple of word size. > + */ > + lp =3D (const unsigned long *)((uintptr_t)str & ~LONGPTR_MASK); > + lp++; Have you tested to see whether the extra work you're doing for the initial segment really gains you much? I would have used a simpler byte-by-byte check as follows: p =3D s; while (maxlen-- > 0 && p < (const char *)lp) { if (*p =3D=3D '\0') return (p - s); } while (maxlen > 0) { =85 complicated test of *lp =85 maxlen -=3D sizeof(*lp); } Few points here: * This form of the initial segment test doesn't get surprised by a NUL byte just before the string. Yours does a bunch of extra work in that case. * Duff's device might help unroll the first loop; would require testing to see if it was faster. For something this simple, it might not be. * Your version tests the first word twice (once in the initial check and then again at the start of the word-sized pass). This version doesn't. * Comparison with zero is often free, so a countdown to zero is often faster than a count up to a computed limit. This assumes that 'lp' is pointing to the first aligned word at or after the beginning of the string. Your version does not leave lp pointing to the beginning of the string when the string is initially already aligned. If the string is already fully aligned, my version avoids the initial check entirely. To compute 'lp' as a pointer to the first full word at or after the beginning of the string: lp =3D (const unsigned long *) ( ( (uintptr_t)str + sizeof(*lp) - 1 ) & ( sizeof(*lp) - 1 ) ); I've broken this onto multiple lines to illustrate the structure; the final code would of course be a little more compact. Also note the use of sizeof(*lp) rather than sizeof(long) or sizeof(unsigned long); this way, you guarantee that the sizeof() matches lp even if someone comes along later and changes the declaration of lp. > + va =3D (*lp - mask01); > + vb =3D ((~*lp) & mask80); > + if (va & vb) { > + /* Check if we have \0 in the first part */ > + short_stop=3D(const char *)lp; > + if (stop + for (p=3Dstr; p !=3D short_stop; p++) > + if (*p =3D=3D '\0') > + return (p-str); > } > - return (len); > + /* Scan the rest of the string using word sized operation */ > + for (; (const char *)lp < stop; lp++) { > + va =3D (*lp - mask01); > + vb =3D ((~*lp) & mask80); > + if (va & vb) { I don't think the extra variables are helpful. Just write it directly: if ((*lp - mask01) & ~*lp & mask80) { That matches the comment you put at the beginning. > + for (p=3D(const char *)lp; p !=3D stop; p++) > + if (*p =3D=3D '\0') > + break; > + return (p-str); > + } > + } > + return (maxlen); > } You should, of course, compare my suggestions above against your version to see which is faster. (Preferably run such tests with both clang and gcc, on at least two architectures, and with a couple of different optimization flags.) Tim From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 18:40:23 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3E7DC1DE for ; Mon, 27 May 2013 18:40:23 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [IPv6:2a01:348:0:15:5d59:5c40:0:1]) by mx1.freebsd.org (Postfix) with ESMTP id F342B5F5 for ; Mon, 27 May 2013 18:40:22 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id B6EB0E6A40; Mon, 27 May 2013 19:40:20 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=mail; bh=4sASSCwVVkub ut7F0OitZVknmko=; b=FzplPz62/+49pJmao6FJQUyVFhinYII0xywDxz5kHEKW FeesWnXjucu28vpynidM81CesrTU4aAm6n6tHJygtfGNpfwPWMTAbNJNZH3uiYfq w5aZxTJlPPId/Iqqhb0dmCiJPU4ngmXyjqAntOhszoebKAzx9mbgaIYhmCDAv3M= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=mail; b=qFbVMv fqya0WQ/AN49tEexJWcJivhJAzY2MrNzwR+NSIl2csXevLYcb5250Q1yBp4HQLmJ +6DPcCVpwJRcgVWIQ/3efHCblWkurcyBnKPB6pXkkqa7WoNX3AoghyoA5TuU315l 1ZhO6SQjT7T4oZvOvPwXIFMkA4T38OUAONjAg= Received: from [192.168.2.61] (unknown [93.89.81.205]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id 99AB4E6A01; Mon, 27 May 2013 19:40:20 +0100 (BST) Message-ID: <51A3A891.5060103@cran.org.uk> Date: Mon, 27 May 2013 19:40:17 +0100 From: Bruce Cran User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Alfred Perlstein Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> <51A39039.1070202@cran.org.uk> <51A39FEC.5070402@mu.org> In-Reply-To: <51A39FEC.5070402@mu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 18:40:23 -0000 On 27/05/2013 19:03, Alfred Perlstein wrote: > Do we always have to seek the lowest common denominator for our user > experience? Yes. -- Bruce From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 19:31:26 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id BC168CA1; Mon, 27 May 2013 19:31:26 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 88087813; Mon, 27 May 2013 19:31:26 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.17]) by ltcfislmsgpa06.fnfis.com (8.14.5/8.14.5) with ESMTP id r4RJVJfZ010135 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Mon, 27 May 2013 14:31:19 -0500 Received: from LTCFISWMSGMB21.FNFIS.com ([10.132.99.23]) by LTCFISWMSGHT06.FNFIS.com ([10.132.206.17]) with mapi id 14.02.0309.002; Mon, 27 May 2013 14:31:19 -0500 From: "Teske, Devin" To: Alfred Perlstein Subject: Re: FreeBSD installers and future direction Thread-Topic: FreeBSD installers and future direction Thread-Index: AQHOWV60z6k72swSkkmGANUzy/EMLJkWZ1wAgAAkfQCAAE6UgIAAH7eAgAAeuACAAA8cgIACXGSAgAAS94CAABK3AIAAGI8A Date: Mon, 27 May 2013 19:31:18 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F631FD@ltcfiswmsgmb21> References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> <51A39039.1070202@cran.org.uk> <51A39FEC.5070402@mu.org> In-Reply-To: <51A39FEC.5070402@mu.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] Content-Type: text/plain; charset="Windows-1252" Content-ID: <0153016D4C54814D8138A845F0064412@fisglobal.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-27_04:2013-05-27,2013-05-27,1970-01-01 signatures=0 Cc: Bruce Cran , FreeBSD Hackers , Devin Teske X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 19:31:26 -0000 On May 27, 2013, at 11:03 AM, Alfred Perlstein wrote: > On 5/27/13 9:56 AM, Bruce Cran wrote: >> On 27/05/2013 16:48, Alfred Perlstein wrote: >>> Why can we not use in the interim use pc-sysinstall on the platforms th= at it performs best on and use bsdinstall on the others? >>=20 >> Because pc-sysinstall doesn't have a UI - it's only a backend. If we upd= ate bsdinstall to use it, then it won't work on other platforms. >>=20 > This still doesn't make sense to me. Why can bsdinstall not conditionall= y use it? >=20 Because, believe it or not, not all programs are split in twain, having a f= ront-end and a back-end. I'm not defending it, and I'm not promoting it, but bsdinstall does things = in a different way where pc-sysinstall wants to be just a back-end. bsdinstall would have to go thro= ugh a major revision to make it use any external back-end. Currently it's one self-enclosed enti= ty. > Do we always have to seek the lowest common denominator for our user expe= rience? >=20 When the situation*is* that the release engineers can only embrace one inst= aller for the release media of more than just x86 architecture, the answer is=85 yes (common deno= minator required). My thoughts are=85 let me toil on a new installer and then we can think abo= ut opening the can of worms that is "how to enable conditional installers." HINT: I already solved that problem=85 by modifying the boot loader Forth m= enu to allow the creation of custom boot menus. Now all we need is another installer to elic= it the use of that code to present the choice the user. However=85 (and this is a big however), unl= ess the 2nd choice installer is as half-as-bad as bsdinstall=85 I don't see why it wouldn't ju= st replace it (therefore negating the need to invoke that code I put in to allow multiple choice ins= tallers at beastie menu). --=20 Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 19:42:39 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 13C86F43 for ; Mon, 27 May 2013 19:42:39 +0000 (UTC) (envelope-from rank1seeker@gmail.com) Received: from mail-ea0-x236.google.com (mail-ea0-x236.google.com [IPv6:2a00:1450:4013:c01::236]) by mx1.freebsd.org (Postfix) with ESMTP id A4C45877 for ; Mon, 27 May 2013 19:42:38 +0000 (UTC) Received: by mail-ea0-f182.google.com with SMTP id r16so4124638ead.41 for ; Mon, 27 May 2013 12:42:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:from:to:subject:date:content-type :content-transfer-encoding:x-mailer; bh=N3abaX84KS/CaIurlyjXvlAzLVxkjL+5o6hjhUqtn1Q=; b=PkJG1dIasT6wXCvQ/arPNwvIu5mtb1s2DWzzOl1lEI5RZTMx61AkyoDURwgn26lt1W U5GPKK61fKPQeadVZCylbqABDcIn/xn2kXdekujhPoM8bs/cyyLToLO5tXxBaPYX7UqF n3rlZlynN7jEdfCG1L+lsrJ3XbPr236JEYmTI4qZqXwzcucSu1bumRFqg/gQEK8fdDfA BPt+LSLiGgR4MsNOUXbb5MTnUC5F2BYiIbl6aGkKReTA/0VVcqu1ZfHRchOfubrK3i/m 5vaC1lAov3Q2BPIXnxiMsxgT7fayLpY7mO/0B6buXMn6Ck8iXuKDg8hAE3/79Gin9K7t Y6Bw== X-Received: by 10.14.181.131 with SMTP id l3mr54754651eem.16.1369683756812; Mon, 27 May 2013 12:42:36 -0700 (PDT) Received: from DOMYPC ([82.193.208.225]) by mx.google.com with ESMTPSA id a5sm43526853ees.6.2013.05.27.12.42.35 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Mon, 27 May 2013 12:42:35 -0700 (PDT) Message-ID: <20130527.194235.693.1@DOMY-PC> From: rank1seeker@gmail.com To: hackers@freebsd.org Subject: /bin/sh => STDIN & functions, var scope messing Date: Mon, 27 May 2013 21:42:35 +0200 Content-Type: text/plain; charset="Windows-1250" Content-Transfer-Encoding: quoted-printable X-Mailer: POP Peeper (3.8.1.0) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 19:42:39 -0000 9.1-RELEASE-p3=0D=0A---------------=0D=0A#!/bin/sh=0D=0A=0D=0Ash_f = ()=0D=0A{=0D=0A global_scope_var=3D7463457=0D=0A}=0D=0A=0D=0Ayes | = sh_f=0D=0Aecho "$global_scope_var"=0D=0A=0D=0Aecho '=0D=0ANow without = /usr/bin/yes (maybe it is STDIN issue, instead) = ?!?=0D=0A'=0D=0A=0D=0Ash_f=0D=0Aecho = "$global_scope_var"=0D=0A---------------=0D=0A=0D=0A=0D=0ADomagoj = Smol=E8i=E6 From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 19:58:29 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9ECE293D for ; Mon, 27 May 2013 19:58:28 +0000 (UTC) (envelope-from linnemannr@gmail.com) Received: from mail-oa0-f43.google.com (mail-oa0-f43.google.com [209.85.219.43]) by mx1.freebsd.org (Postfix) with ESMTP id BE4CC929 for ; Mon, 27 May 2013 19:58:28 +0000 (UTC) Received: by mail-oa0-f43.google.com with SMTP id o6so9118142oag.16 for ; Mon, 27 May 2013 12:58:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=m/Lu68rN7xRYBnOV9g5MDoHpL3lch01/dFmXm8R1KqI=; b=BvQrYRDIp/Gm03zzneK4NiEQJ5A6oSsf8hZlPbLNI0xAnRaxrBaYPQxe6mQ8ajOjLm yZcer1px75/ucAyAK/+0ZoOPc++2KSXS5WPXTVqlHnyKZcpsFGxhTXC/GW+j++Txqaui FgQI+re1TUoe5XJOSWFvnRKgbeLPzyGR7bhs6QSWLmrYeWki1gbsXvWtoPfTaHeXdpZ2 jskh0cvCu3LdXhj58NKobnuuyqgPLBGhnvjlnPHQehNmWehjxuX7CKkxtLi8rSdHbTeH HvdExmxJUjMrhG98WUDCXRtD+8xMaVjzcuDTZeiXN4My7rLkXYxM4Wgw+DADUM3OlXjV caEA== MIME-Version: 1.0 X-Received: by 10.182.129.129 with SMTP id nw1mr19247765obb.100.1369684700957; Mon, 27 May 2013 12:58:20 -0700 (PDT) Received: by 10.182.251.229 with HTTP; Mon, 27 May 2013 12:58:20 -0700 (PDT) In-Reply-To: <20130527.194235.693.1@DOMY-PC> References: <20130527.194235.693.1@DOMY-PC> Date: Mon, 27 May 2013 14:58:20 -0500 Message-ID: Subject: Re: /bin/sh => STDIN & functions, var scope messing From: Reid Linnemann To: rank1seeker@gmail.com Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 19:58:29 -0000 from SH(1) "Note that unlike some other shells, sh executes each process in a pipe- line with more than one command in a subshell environment and as a child of the sh process." I'm taking this to mean that redirecting to sh_f has sh_f execute in a subshell in which global_scope_var changes, but the original shell's copy is uncahnged. On Mon, May 27, 2013 at 2:42 PM, wrote: > 9.1-RELEASE-p3 > --------------- > #!/bin/sh > > sh_f () > { > global_scope_var=3D7463457 > } > > yes | sh_f > echo "$global_scope_var" > > echo ' > Now without /usr/bin/yes (maybe it is STDIN issue, instead) ?!? > ' > > sh_f > echo "$global_scope_var" > --------------- > > > Domagoj Smol=C4=8Di=C4=87 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 20:01:56 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1BE6CDBF for ; Mon, 27 May 2013 20:01:56 +0000 (UTC) (envelope-from lee_thomas@aslantools.com) Received: from roproxy1-pub.unifiedlayer.com (roproxy1-pub.unifiedlayer.com [69.89.25.95]) by mx1.freebsd.org (Postfix) with SMTP id EEBB1966 for ; Mon, 27 May 2013 20:01:55 +0000 (UTC) Received: (qmail 4632 invoked by uid 0); 27 May 2013 20:01:52 -0000 Received: from unknown (HELO mailchannelsproxy2.unifiedlayer.com) (66.147.243.71) by roproxy1.unifiedlayer.com with SMTP; 27 May 2013 20:01:52 -0000 X-Sender-Id: {:fast21.fastdomain.com:lthomasn:fast21.fastdomain.com} {sentby:program running on server} Received: from fast21.fastdomain.com (fast21.fastdomain.com [74.220.199.21]) by 0.0.0.0:2500 (trex/4.8.69); Mon, 27 May 2013 20:01:52 GMT Received: from localhost ([127.0.0.1]:45886 helo=fast21.fastdomain.com) by fast21.fastdomain.com with esmtp (Exim 4.80) (envelope-from ) id 1Uh3c8-0000E5-34 for freebsd-hackers@freebsd.org; Mon, 27 May 2013 14:01:52 -0600 To: Subject: Re: Performance improvement to strnlen(). X-PHP-Script: aslantools.com/mail/index.php for 173.71.114.222 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit Date: Mon, 27 May 2013 16:01:51 -0400 From: Lee Thomas Organization: Aslan Tools In-Reply-To: References: Message-ID: <361355916ec746edbf91c3f258f2416e@lthomas.net> X-Sender: lee_thomas@aslantools.com User-Agent: Roundcube Webmail/0.8.4 X-Identified-User: {:fast21.fastdomain.com:lthomasn:fast21.fastdomain.com} {sentby:program running on server} X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 20:01:56 -0000 Hello Tim, Thank you for the review; replies inline. Note that all my performance discussion is for amd64, with a few tests of x86, and it's all on the machine described in my initial email (a Lynnfield Xeon), as I don't have any other FreeBSD platform to test on. Testing and performance measurement on other platforms would be appreciated. Thanks, Lee On 2013-05-27 14:26, Tim Kientzle wrote: >> >> Index: strnlen.c >> =================================================================== >> diff --git a/head/lib/libc/string/strnlen.c >> b/head/lib/libc/string/strnlen.c >> --- a/head/lib/libc/string/strnlen.c (revision 250951) >> +++ b/head/lib/libc/string/strnlen.c (working copy) >> @@ -1,5 +1,6 @@ >> /*- >> - * Copyright (c) 2009 David Schultz >> + * Copyright (c) 2009, 2010 Xin LI >> + * Copyright (c) 2013 Lee Thomas >> * All rights reserved. >> * >> * Redistribution and use in source and binary forms, with or >> without >> @@ -27,16 +28,91 @@ >> #include >> __FBSDID("$FreeBSD$"); >> >> +#include >> +#include >> #include >> >> +/* >> + * Portable strnlen() for 32-bit and 64-bit systems. >> + * >> + * Rationale: it is generally much more efficient to do word length >> + * operations and avoid branches on modern computer systems, as >> + * compared to byte-length operations with a lot of branches. >> + * >> + * The expression: >> + * >> + * ((x - 0x01....01) & ~x & 0x80....80) >> + * >> + * would evaluate to a non-zero value iff any of the bytes in the >> + * original word is zero. >> + * >> + * On multi-issue processors, we can divide the above expression >> into: >> + * a) (x - 0x01....01) >> + * b) (~x & 0x80....80) >> + * c) a & b >> + * >> + * Where, a) and b) can be partially computed in parallel. >> + * >> + * The algorithm above is found on "Hacker's Delight" by >> + * Henry S. Warren, Jr. >> + */ >> + >> +/* Magic numbers for the algorithm */ >> +#if LONG_BIT == 32 >> +static const unsigned long mask01 = 0x01010101; >> +static const unsigned long mask80 = 0x80808080; >> +#elif LONG_BIT == 64 >> +static const unsigned long mask01 = 0x0101010101010101; >> +static const unsigned long mask80 = 0x8080808080808080; >> +#else >> +#error Unsupported word size >> +#endif >> + >> +#define LONGPTR_MASK (sizeof(long) - 1) > > I would not use this at all; (sizeof(long) - 1) is a common > expression that your audience probably understands > more quickly than "LONGPTR_MASK". > This is simply copy-pasted from our strlen implementation; I wanted to leave it as close to that implementation as possible, rather than rewrite the whole thing. I could refactor the common stuff out of both, though it would require a common implementation header for both (The style guidelines for which I don't know), and I'm not sure it would be a benefit to code legibility, anyways. Would it be worth cleaning strnlen.c up, at the cost of an increased diff with strlen.c? Maybe both should be cleaned up in the same ways? I'm willing to do any of these if you think it would be worth the code churn. >> + >> size_t >> -strnlen(const char *s, size_t maxlen) >> +strnlen(const char *str, size_t maxlen) >> { >> - size_t len; >> + const char *stop, *short_stop; >> + const char *p; >> + const unsigned long *lp; >> + long va, vb; >> >> - for (len = 0; len < maxlen; len++, s++) { >> - if (!*s) >> - break; >> + if (maxlen==0) return 0; >> + >> + stop=str+maxlen; >> + /* >> + * Before trying the hard (unaligned byte-by-byte access) way >> + * to figure out whether there is a nul character, try to see >> + * if there is a nul character is within this accessible word >> + * first. >> + * >> + * p and (p & ~LONGPTR_MASK) must be equally accessible since >> + * they always fall in the same memory page, as long as page >> + * boundaries is integral multiple of word size. >> + */ >> + lp = (const unsigned long *)((uintptr_t)str & ~LONGPTR_MASK); >> + lp++; > > Have you tested to see whether the extra work you're > doing for the initial segment really gains you much? I > would have used a simpler byte-by-byte check as > follows: Yes, it is worth it, with a pretty good margin for small strings, even under pretty high assumptions about the frequency of zero in the data before the start of the string. I tried various things, and this is the fastest I found. > > p = s; > while (maxlen-- > 0 && p < (const char *)lp) { > if (*p == '\0') > return (p - s); > } > > while (maxlen > 0) { > … complicated test of *lp … > maxlen -= sizeof(*lp); > } > Few points here: > * This form of the initial segment test doesn't get surprised > by a NUL byte just before the string. Yours does a bunch of > extra work in that case. > * Duff's device might help unroll the first loop; would require > testing to see if it was faster. For something this simple, it > might not be. > * Your version tests the first word twice (once in the > initial check and then again at the start of the word-sized > pass). This version doesn't. Actually, it doesn't. Note the 'lp++' on line 97, after computing va and vb, but before the if. It's before the if because it's used in computing short_stop. I tried it after the if, because it would be cleaner there (as your missing it clearly shows), but one of either gcc or clang, I forget which, didn't manage to do the common sub expression elimination with the computation inside the if(va & vb) {}, leading to a slowdown in the case where one of the junk bytes before the string is zero. > * Comparison with zero is often free, so a countdown to zero > is often faster than a count up to a computed limit. > > This assumes that 'lp' is pointing to the first aligned word > at or after the beginning of the string. > Your version does not leave lp pointing to > the beginning of the string when the string is initially > already aligned. If the string is already fully aligned, my > version avoids the initial check entirely. > > To compute 'lp' as a pointer to the first full word > at or after the beginning of the string: > > lp = (const unsigned long *) > ( > ( > (uintptr_t)str + sizeof(*lp) - 1 > ) > & > ( > sizeof(*lp) - 1 > ) > ); > > I've broken this onto multiple lines to illustrate the structure; > the final code would of course be a little more compact. > > Also note the use of sizeof(*lp) rather than > sizeof(long) or sizeof(unsigned long); this way, > you guarantee that the sizeof() matches lp even if someone > comes along later and changes the declaration of lp. >> + va = (*lp - mask01); >> + vb = ((~*lp) & mask80); >> + if (va & vb) { >> + /* Check if we have \0 in the first part */ >> + short_stop=(const char *)lp; >> + if (stop> + for (p=str; p != short_stop; p++) >> + if (*p == '\0') >> + return (p-str); >> } >> - return (len); >> + /* Scan the rest of the string using word sized operation */ >> + for (; (const char *)lp < stop; lp++) { >> + va = (*lp - mask01); >> + vb = ((~*lp) & mask80); >> + if (va & vb) { > > I don't think the extra variables are helpful. Just write it > directly: > > if ((*lp - mask01) & ~*lp & mask80) { > > That matches the comment you put at the beginning. > Again, this is directly from the strlen code. As before, I'm willing to rewrite one or both if you think it'd be worth it. >> + for (p=(const char *)lp; p != stop; p++) >> + if (*p == '\0') >> + break; >> + return (p-str); >> + } >> + } >> + return (maxlen); >> } > > You should, of course, compare my suggestions above > against your version to see which is faster. (Preferably run > such tests with both clang and gcc, on at least two > architectures, and with a couple of different optimization > flags.) > > Tim I think my email did say some about my test methods, but I'll go into a little more detail. Let me know if you want the actual code, but be aware it's ugly as heck C++. I tested with both the gcc and clang in stable/9, though I think I forgot to mention in my email the opt flags I used, which were "-O2 -fno-strict-aliasing", "-O2", and "-O3". None of them seemed to make very much difference with this code, and gcc and clang both did pretty equivalently. I tried many variants, testing them against an artificial dataset of randomly generated strings, of 50% length between 0 and 20, and 50% between 21 and 1000, where the max_len value was effectively infinity (assuming that most of the time the string fits in the buffer). The dataset used randomly-generated overread bytes on both sides of the string, which were 50% NULL and 50% anything else. I used ministat for comparison, which by the way is the best tool I have ever encountered; We use it for 1000 things I had always used a spreadsheet for in the past. The code as posted is the fastest I could find. For instance, the strlen code unrolls the main loop; This is faster for strlen, but the extra length check makes the not-unrolled variant faster for strnlen. I think farther significant performance improvements are likely only to come through machine-dependent assembly versions, but I'm not going to write or maintain that :-). Linux is a little faster for ASCII (01-7F) strings, but quite a bit slower for general (01-FF) strings. I could do that, but FreeBSD had such a variant several years ago, and moved away from it, which I personally think was wise. Thanks again for the code review, Lee From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 20:29:00 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1A21379C for ; Mon, 27 May 2013 20:29:00 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 07A0BA6E for ; Mon, 27 May 2013 20:28:59 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id E54251A3C5B; Mon, 27 May 2013 13:28:53 -0700 (PDT) Message-ID: <51A3C202.9030802@mu.org> Date: Mon, 27 May 2013 13:28:50 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Bruce Cran Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> <51A39039.1070202@cran.org.uk> <51A39FEC.5070402@mu.org> <51A3A891.5060103@cran.org.uk> In-Reply-To: <51A3A891.5060103@cran.org.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 20:29:00 -0000 On 5/27/13 11:40 AM, Bruce Cran wrote: > On 27/05/2013 19:03, Alfred Perlstein wrote: >> Do we always have to seek the lowest common denominator for our user >> experience? > > Yes. > Is this a joke? -Alfred From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 20:37:08 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id EA443DFF for ; Mon, 27 May 2013 20:37:08 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [188.252.31.196]) by mx1.freebsd.org (Postfix) with ESMTP id 7984DACD for ; Mon, 27 May 2013 20:37:08 +0000 (UTC) Received: from wojtek.tensor.gdynia.pl (localhost [127.0.0.1]) by wojtek.tensor.gdynia.pl (8.14.6/8.14.6) with ESMTP id r4RKalD4002958; Mon, 27 May 2013 22:36:47 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.6/8.14.6/Submit) with ESMTP id r4RKaljB002955; Mon, 27 May 2013 22:36:47 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Mon, 27 May 2013 22:36:47 +0200 (CEST) From: Wojciech Puchar To: Bruce Cran Subject: Re: FreeBSD installers and future direction In-Reply-To: <51A0DC3F.9030301@cran.org.uk> Message-ID: References: <51A0DC3F.9030301@cran.org.uk> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (wojtek.tensor.gdynia.pl [127.0.0.1]); Mon, 27 May 2013 22:36:47 +0200 (CEST) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 20:37:09 -0000 > I heard there was some discussion at BSDCan about the direction of a future > FreeBSD installer. Considering we currently have bsdinstall, pc-sysinstall, the best would be removing it at all and adding instruction how to install by hand. At least someone that install FreeBSD will know what he/she actually did, and what to do then in case of emergency or how to say.. move the system From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 21:23:10 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 41C69D49 for ; Mon, 27 May 2013 21:23:10 +0000 (UTC) (envelope-from bruce@cran.org.uk) Received: from muon.cran.org.uk (muon.cran.org.uk [93.89.92.64]) by mx1.freebsd.org (Postfix) with ESMTP id 05996E1D for ; Mon, 27 May 2013 21:23:09 +0000 (UTC) Received: from muon.cran.org.uk (localhost [127.0.0.1]) by muon.cran.org.uk (Postfix) with ESMTP id 389BCE6A40; Mon, 27 May 2013 22:23:05 +0100 (BST) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; s=mail; bh=PGjyvj8Rkfkk V4MXJ82wXMZxtQs=; b=O1eesFmD0YkZRCKCQF4icN+CD+/Ffiuxsakpq9S/qa6x /JBvAmNhotgNX76Gj84GAbdQkVpXuBZMvodTeSipTSEA3lgXr9bXuQkAcayO0OsQ 3I+E1i7KhJUsKZv4wKaXM+uFbVHUQ0Cm3zeW8pbFsg+2Ip1JWlWDnINqI2ZxrBg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=cran.org.uk; h=message-id :date:from:mime-version:to:cc:subject:references:in-reply-to :content-type:content-transfer-encoding; q=dns; s=mail; b=cArYHa zMX93q8+JL13bpEBJBReXOztXbJ+ZL13X3LmFtoFpOV5Y5PQMeCG0y0BdV203erW JmTk01pFzGv4cuy8QeuNpf3HIkbNAg0fuUD4m/bdsQJUWlX7NVbsvRTBVaWqZ1tG PiQ8vT/btDVMRMoAbDX+LLiuGIqmAQ+TGUbEU= Received: from [192.168.2.61] (unknown [93.89.81.205]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by muon.cran.org.uk (Postfix) with ESMTPSA id 0FD30E6A01; Mon, 27 May 2013 22:23:05 +0100 (BST) Message-ID: <51A3CEB6.3070200@cran.org.uk> Date: Mon, 27 May 2013 22:23:02 +0100 From: Bruce Cran User-Agent: Mozilla/5.0 (Windows NT 6.2; WOW64; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Alfred Perlstein Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> <51A39039.1070202@cran.org.uk> <51A39FEC.5070402@mu.org> <51A3A891.5060103@cran.org.uk> <51A3C202.9030802@mu.org> In-Reply-To: <51A3C202.9030802@mu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 21:23:10 -0000 On 27/05/2013 21:28, Alfred Perlstein wrote: > On 5/27/13 11:40 AM, Bruce Cran wrote: >> Yes. > Is this a joke? It probably /was/ too short a reply. Personally I think there should be a single UI and scripting interface across all platforms. We should try and get pc-sysinstall running on all of them first in case there's some problem that means it can't be done, in which case we'd need to use a different backend. -- Bruce From owner-freebsd-hackers@FreeBSD.ORG Mon May 27 23:10:35 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A93DCB8E for ; Mon, 27 May 2013 23:10:35 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from smtpauth3.wiscmail.wisc.edu (wmauth3.doit.wisc.edu [144.92.197.226]) by mx1.freebsd.org (Postfix) with ESMTP id 80AA8250 for ; Mon, 27 May 2013 23:10:35 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from avs-daemon.smtpauth3.wiscmail.wisc.edu by smtpauth3.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) id <0MNH00F00951AL00@smtpauth3.wiscmail.wisc.edu> for freebsd-hackers@freebsd.org; Mon, 27 May 2013 17:10:26 -0500 (CDT) X-Spam-PmxInfo: Server=avs-3, Version=6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2013.5.27.220028, SenderIP=0.0.0.0 X-Spam-Report: AuthenticatedSender=yes, SenderIP=0.0.0.0 Received: from comporellon.tachypleus.net (adsl-76-208-69-84.dsl.mdsnwi.sbcglobal.net [76.208.69.84]) by smtpauth3.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) with ESMTPSA id <0MNH000AY9LB1X20@smtpauth3.wiscmail.wisc.edu>; Mon, 27 May 2013 17:10:25 -0500 (CDT) Message-id: <51A3D9CF.4070509@freebsd.org> Date: Mon, 27 May 2013 17:10:23 -0500 From: Nathan Whitehorn User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130410 Thunderbird/17.0.5 To: Bruce Cran Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> <51A39039.1070202@cran.org.uk> <51A39FEC.5070402@mu.org> <51A3A891.5060103@cran.org.uk> <51A3C202.9030802@mu.org> <51A3CEB6.3070200@cran.org.uk> In-reply-to: <51A3CEB6.3070200@cran.org.uk> Cc: freebsd-hackers@freebsd.org, Alfred Perlstein X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 May 2013 23:10:35 -0000 On 05/27/13 16:23, Bruce Cran wrote: > On 27/05/2013 21:28, Alfred Perlstein wrote: >> On 5/27/13 11:40 AM, Bruce Cran wrote: >>> Yes. >> Is this a joke? > > It probably /was/ too short a reply. Personally I think there should > be a single UI and scripting interface across all platforms. We should > try and get pc-sysinstall running on all of them first in case there's > some problem that means it can't be done, in which case we'd need to > use a different backend. > I'd point out that bsdinstall does have a scripting interface now as well. -Nathan From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 01:40:06 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A4A87234 for ; Tue, 28 May 2013 01:40:06 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 93483DA1 for ; Tue, 28 May 2013 01:40:06 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id 24E4D1A3D13; Mon, 27 May 2013 18:40:03 -0700 (PDT) Message-ID: <51A40AF2.2010108@mu.org> Date: Mon, 27 May 2013 18:40:02 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Bruce Cran Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> <51A39039.1070202@cran.org.uk> <51A39FEC.5070402@mu.org> <51A3A891.5060103@cran.org.uk> <51A3C202.9030802@mu.org> <51A3CEB6.3070200@cran.org.uk> In-Reply-To: <51A3CEB6.3070200@cran.org.uk> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 01:40:06 -0000 On 5/27/13 2:23 PM, Bruce Cran wrote: > On 27/05/2013 21:28, Alfred Perlstein wrote: >> On 5/27/13 11:40 AM, Bruce Cran wrote: >>> Yes. >> Is this a joke? > > It probably /was/ too short a reply. Personally I think there should > be a single UI and scripting interface across all platforms. We should > try and get pc-sysinstall running on all of them first in case there's > some problem that means it can't be done, in which case we'd need to > use a different backend. > There are just going to be certain platforms that make it EASY to do cool things. We should embrace that! That's why there are different platforms! Some are great for low power, others are great for graphics, cpu power, gpu, networking etc. If we always go for the lowest common denominator then we are crippling all the platforms for no one's benefit. Even if something CAN be done, if it is very difficult, or just never happening, then we can't limit everyone's experience based on the more difficult and/or resource strapped platforms. It's just not good business. -Alfred From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 01:54:09 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id CA5DE3BB for ; Tue, 28 May 2013 01:54:09 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from smtpauth3.wiscmail.wisc.edu (wmauth3.doit.wisc.edu [144.92.197.226]) by mx1.freebsd.org (Postfix) with ESMTP id 9D453E01 for ; Tue, 28 May 2013 01:54:09 +0000 (UTC) MIME-version: 1.0 Content-transfer-encoding: 7BIT Content-type: text/plain; CHARSET=US-ASCII; format=flowed Received: from avs-daemon.smtpauth3.wiscmail.wisc.edu by smtpauth3.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) id <0MNH00500JVBXN00@smtpauth3.wiscmail.wisc.edu> for freebsd-hackers@freebsd.org; Mon, 27 May 2013 20:54:07 -0500 (CDT) X-Spam-PmxInfo: Server=avs-3, Version=6.0.2.2308539, Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2013.5.28.13929, SenderIP=0.0.0.0 X-Spam-Report: AuthenticatedSender=yes, SenderIP=0.0.0.0 Received: from comporellon.tachypleus.net (adsl-76-208-69-84.dsl.mdsnwi.sbcglobal.net [76.208.69.84]) by smtpauth3.wiscmail.wisc.edu (Oracle Communications Messaging Server 7u4-27.01(7.0.4.27.0) 64bit (built Aug 30 2012)) with ESMTPSA id <0MNH00GQ5JXZPP10@smtpauth3.wiscmail.wisc.edu>; Mon, 27 May 2013 20:54:01 -0500 (CDT) Message-id: <51A40E37.9060702@freebsd.org> Date: Mon, 27 May 2013 20:53:59 -0500 From: Nathan Whitehorn User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130410 Thunderbird/17.0.5 To: Alfred Perlstein Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> <51A39039.1070202@cran.org.uk> <51A39FEC.5070402@mu.org> <51A3A891.5060103@cran.org.uk> <51A3C202.9030802@mu.org> <51A3CEB6.3070200@cran.org.uk> <51A40AF2.2010108@mu.org> In-reply-to: <51A40AF2.2010108@mu.org> Cc: Bruce Cran , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 01:54:09 -0000 On 05/27/13 20:40, Alfred Perlstein wrote: > On 5/27/13 2:23 PM, Bruce Cran wrote: >> On 27/05/2013 21:28, Alfred Perlstein wrote: >>> On 5/27/13 11:40 AM, Bruce Cran wrote: >>>> Yes. >>> Is this a joke? >> >> It probably /was/ too short a reply. Personally I think there should >> be a single UI and scripting interface across all platforms. We >> should try and get pc-sysinstall running on all of them first in case >> there's some problem that means it can't be done, in which case we'd >> need to use a different backend. >> > > There are just going to be certain platforms that make it EASY to do > cool things. We should embrace that! That's why there are different > platforms! > > Some are great for low power, others are great for graphics, cpu > power, gpu, networking etc. > > If we always go for the lowest common denominator then we are > crippling all the platforms for no one's benefit. Even if something > CAN be done, if it is very difficult, or just never happening, then we > can't limit everyone's experience based on the more difficult and/or > resource strapped platforms. > > It's just not good business. Yes, and all of this cuts both ways: pc-sysinstall has no wireless setup support, for instance. Right now we support what we support because it is the most feature-complete thing we have, not just on tier-2 platforms but also on x86. To bring this discussion back to the ground, the fact is that we lack an installer that has both internal support for ZFS and a UI. One of the reasons for this is that making a good expressive UI for ZFS is a non-trivial undertaking given its enormous flexibility. The bsdinstall partition editor has been written to be extensible for this, and several people have started writing code to do it, but no one ended up having time to finish. Probably a reasonable thing to do is to start with supporting only a minimal set of features. If anyone felt like actually writing this code, I'm sure it would be appreciated by all and be more productive than email exchanges. -Nathan From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 04:17:49 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4991C822; Tue, 28 May 2013 04:17:49 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 111A166C; Tue, 28 May 2013 04:17:48 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.31]) by ltcfislmsgpa04.fnfis.com (8.14.5/8.14.5) with ESMTP id r4S4Hd7a002293 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Mon, 27 May 2013 23:17:39 -0500 Received: from LTCFISWMSGMB21.FNFIS.com ([10.132.99.23]) by LTCFISWMSGHT03.FNFIS.com ([10.132.206.31]) with mapi id 14.02.0309.002; Mon, 27 May 2013 23:17:39 -0500 From: "Teske, Devin" To: Devin Teske Subject: Re: FreeBSD installers and future direction Thread-Topic: FreeBSD installers and future direction Thread-Index: AQHOWV60z6k72swSkkmGANUzy/EMLJkWZ1wAgAAkfQCAAE6UgIAAH7eAgAAeuACAAA8cgIAA3asAgAAPj4CAAAWTAIAAFx4AgAIjxAA= Date: Tue, 28 May 2013 04:17:38 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F638D8@ltcfiswmsgmb21> References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <13CA24D6AB415D428143D44749F57D7201F603E6@ltcfiswmsgmb21> <51A250FE.3070206@cran.org.uk> <13CA24D6AB415D428143D44749F57D7201F615E8@ltcfiswmsgmb21> In-Reply-To: <13CA24D6AB415D428143D44749F57D7201F615E8@ltcfiswmsgmb21> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] Content-Type: text/plain; charset="Windows-1252" Content-ID: <7FF863CDC8ADF347BF9168FE01B93DFB@fisglobal.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-28_02:2013-05-27,2013-05-28,1970-01-01 signatures=0 Cc: Bruce Cran , Super Bisquit , FreeBSD Hackers , Dirk Engling , Nathan Whitehorn , Josh Paetzel , Devin Teske X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 04:17:49 -0000 On May 26, 2013, at 12:37 PM, Teske, Devin wrote: >=20 > On May 26, 2013, at 11:14 AM, Bruce Cran wrote: >=20 >> On 26/05/2013 18:54, Teske, Devin wrote: >>> http://www.google-melange.com/gsoc/proposal/review/google/gsoc2013/hars= hbhatt/1 >>=20 >> "This proposal is not made public, and you are not the student who submi= tted the proposal, nor are you a mentor for the organization it was submitt= ed to." >>=20 >=20 > So, uh=85 register? >=20 > I can see all private projects and all I did was register with google-mel= ange. >=20 > I'm not aware of a way to make it public versus private. The project was not slotted. :( --=20 Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 04:36:20 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 1DFC9D18; Tue, 28 May 2013 04:36:20 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id 0A05877C; Tue, 28 May 2013 04:36:19 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id 5C6A21A3C1A; Mon, 27 May 2013 21:36:16 -0700 (PDT) Message-ID: <51A4343F.3070605@mu.org> Date: Mon, 27 May 2013 21:36:15 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Nathan Whitehorn Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> <51A39039.1070202@cran.org.uk> <51A39FEC.5070402@mu.org> <51A3A891.5060103@cran.org.uk> <51A3C202.9030802@mu.org> <51A3CEB6.3070200@cran.org.uk> <51A40AF2.2010108@mu.org> <51A40E37.9060702@freebsd.org> In-Reply-To: <51A40E37.9060702@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Bruce Cran , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 04:36:20 -0000 On 5/27/13 6:53 PM, Nathan Whitehorn wrote: > On 05/27/13 20:40, Alfred Perlstein wrote: >> On 5/27/13 2:23 PM, Bruce Cran wrote: >>> On 27/05/2013 21:28, Alfred Perlstein wrote: >>>> On 5/27/13 11:40 AM, Bruce Cran wrote: >>>>> Yes. >>>> Is this a joke? >>> >>> It probably /was/ too short a reply. Personally I think there should >>> be a single UI and scripting interface across all platforms. We >>> should try and get pc-sysinstall running on all of them first in >>> case there's some problem that means it can't be done, in which case >>> we'd need to use a different backend. >>> >> >> There are just going to be certain platforms that make it EASY to do >> cool things. We should embrace that! That's why there are different >> platforms! >> >> Some are great for low power, others are great for graphics, cpu >> power, gpu, networking etc. >> >> If we always go for the lowest common denominator then we are >> crippling all the platforms for no one's benefit. Even if something >> CAN be done, if it is very difficult, or just never happening, then >> we can't limit everyone's experience based on the more difficult >> and/or resource strapped platforms. >> >> It's just not good business. > > Yes, and all of this cuts both ways: pc-sysinstall has no wireless > setup support, for instance. Right now we support what we support > because it is the most feature-complete thing we have, not just on > tier-2 platforms but also on x86. > > To bring this discussion back to the ground, the fact is that we lack > an installer that has both internal support for ZFS and a UI. One of > the reasons for this is that making a good expressive UI for ZFS is a > non-trivial undertaking given its enormous flexibility. The bsdinstall > partition editor has been written to be extensible for this, and > several people have started writing code to do it, but no one ended up > having time to finish. Probably a reasonable thing to do is to start > with supporting only a minimal set of features. If anyone felt like > actually writing this code, I'm sure it would be appreciated by all > and be more productive than email exchanges. > -Nathan I'm sure if there was a list of reasonable things, such as wireless then pc-sysinstall could be augmented. This is the first I've heard of that. All the other complaints have been based on portability. Is that all that is required now, wireless? -Alfred -Alfred From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 08:33:35 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id BDF4BF81 for ; Tue, 28 May 2013 08:33:35 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [188.252.31.196]) by mx1.freebsd.org (Postfix) with ESMTP id 338122BC for ; Tue, 28 May 2013 08:33:34 +0000 (UTC) Received: from wojtek.tensor.gdynia.pl (localhost [127.0.0.1]) by wojtek.tensor.gdynia.pl (8.14.6/8.14.6) with ESMTP id r4S8XKDG008419; Tue, 28 May 2013 10:33:20 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.6/8.14.6/Submit) with ESMTP id r4S8XK46008416; Tue, 28 May 2013 10:33:20 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Tue, 28 May 2013 10:33:20 +0200 (CEST) From: Wojciech Puchar To: dt71@gmx.com Subject: Re: >3955MiB of swap space In-Reply-To: <51A39EF8.6020700@gmx.com> Message-ID: References: <51A39EF8.6020700@gmx.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (wojtek.tensor.gdynia.pl [127.0.0.1]); Tue, 28 May 2013 10:33:20 +0200 (CEST) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 08:33:35 -0000 > prefer to repartition the drives for them to have swap partitions each of > size (3955/4)MiB. > > By the way, is swapping distributed evenly among the drives? How? Is there a yes. > downfall when one of the drives is outstandingly slow? no really. sometimes swapin would be slower, 3/4 times of cases faster, by average would be a bit slower. From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 09:48:50 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 0DE3C8A2 for ; Tue, 28 May 2013 09:48:50 +0000 (UTC) (envelope-from vhaisman@gmail.com) Received: from mail-lb0-f178.google.com (mail-lb0-f178.google.com [209.85.217.178]) by mx1.freebsd.org (Postfix) with ESMTP id 8FD7C891 for ; Tue, 28 May 2013 09:48:49 +0000 (UTC) Received: by mail-lb0-f178.google.com with SMTP id w10so7401199lbi.23 for ; Tue, 28 May 2013 02:48:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=krDz0G1Z3c9BlWi51hQxnXsdgYpPOMWz7765pv+L+Eo=; b=U9xGgpA+l4zwir1rzq64WUZPw0KDKOApvjwvRgCIL3kc1uKSXtGZBvWQF02Pf0L6G6 qptUlFSnFPGH9rnmMbaswU8SuyztwYTVHrXyo2555dhjnerw5DhHB3n85zur10icrSHa qP//DU32tg7UHcI8H9CvlpVGrynPxIFTEDEfMJokrXc6AzTP17y7lPpMDj9d1fwpo9A1 BBhoWTSbsq3rydLNlBLPwLipEaKWF0midmbUkii+Ha0AhvwIl+7wKDmg2mv1/4OHpGJK QH00ETNnq5QKnR+8n6cNJAUROy5lUuG35AaUC8+6k30iicNgF2P3lEnnjtNz8LfbRcfa /ICw== MIME-Version: 1.0 X-Received: by 10.112.89.195 with SMTP id bq3mr16130901lbb.19.1369734527914; Tue, 28 May 2013 02:48:47 -0700 (PDT) Received: by 10.114.98.129 with HTTP; Tue, 28 May 2013 02:48:47 -0700 (PDT) In-Reply-To: References: <20130527.194235.693.1@DOMY-PC> Date: Tue, 28 May 2013 11:48:47 +0200 Message-ID: Subject: Re: /bin/sh => STDIN & functions, var scope messing From: =?UTF-8?Q?V=C3=A1clav_Zeman?= To: Reid Linnemann Content-Type: text/plain; charset=UTF-8 Cc: rank1seeker@gmail.com, hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 09:48:50 -0000 On 27 May 2013 21:58, Reid Linnemann wrote: > from SH(1) > > "Note that unlike some other shells, sh executes each process in a pipe- > line with more than one command in a subshell environment and as a > child > of the sh process." > > I'm taking this to mean that redirecting to sh_f has sh_f execute in a > subshell in which global_scope_var changes, but the original shell's copy > is uncahnged. Curious. Which of the two behaviours is POSIXly correct? -- VZ From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 12:00:57 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4C069C4B for ; Tue, 28 May 2013 12:00:57 +0000 (UTC) (envelope-from rysto32@gmail.com) Received: from mail-oa0-f44.google.com (mail-oa0-f44.google.com [209.85.219.44]) by mx1.freebsd.org (Postfix) with ESMTP id 1B3A62B4 for ; Tue, 28 May 2013 12:00:56 +0000 (UTC) Received: by mail-oa0-f44.google.com with SMTP id n12so9867369oag.17 for ; Tue, 28 May 2013 05:00:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=hG2bVdhsfuXZt9Rn4NpCSHi/IXAGtI+dIz/Qu3qrOJQ=; b=jC7WNdVKoOd+kM2zQR9sHzC1fMO/vhYFn9hTjRf6DDFFSd+GRohiWFpcQe44GAIgws x3kPNwcaE+1tbPQcAztPclnRLTbyJc7Xw5auOahcxbG6UKqfBSmO7VL9/v8sbsSY1z5q N1Xczycr0mBJNBxf6fXx9Mkw6y5oPED2lyao5tOgerE/adW4DuxmqAXIsegORPJCJqe7 WfiLEYHjo2x5ucAANK1PxwnR5RaPvAmAIyFj62dAqmIje+/FATG7y8324ExXe9zBdsTm SjRtS7Hh8PCeLJvmNm2Rbw+5m815wCHxLqUGEAalv0FjdQwOLnuaY9yiNmePbvf+ygai 2d9w== MIME-Version: 1.0 X-Received: by 10.182.97.168 with SMTP id eb8mr20544699obb.89.1369742456400; Tue, 28 May 2013 05:00:56 -0700 (PDT) Received: by 10.76.91.163 with HTTP; Tue, 28 May 2013 05:00:56 -0700 (PDT) In-Reply-To: References: <20130527.194235.693.1@DOMY-PC> Date: Tue, 28 May 2013 08:00:56 -0400 Message-ID: Subject: Re: /bin/sh => STDIN & functions, var scope messing From: Ryan Stone To: =?ISO-8859-1?Q?V=E1clav_Zeman?= Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: rank1seeker@gmail.com, Reid Linnemann , FreeBSD Hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 12:00:57 -0000 On Tue, May 28, 2013 at 5:48 AM, V=E1clav Zeman wrote: > Curious. Which of the two behaviours is POSIXly correct? > I believe that /bin/sh's behaviour is correct. I don't know what shell the manpage is referring to, but it's not bash (bash does the same thing in a pipeline). Perhaps it's referring to csh? If that is the case that line is probably causing more confusion rather than alleviating it. From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 12:58:04 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 820A59C9 for ; Tue, 28 May 2013 12:58:04 +0000 (UTC) (envelope-from afischer@marvell.com) Received: from na3sys009aog127.obsmtp.com (na3sys009aog127.obsmtp.com [74.125.149.107]) by mx1.freebsd.org (Postfix) with ESMTP id 13D5285F for ; Tue, 28 May 2013 12:58:03 +0000 (UTC) Received: from SC-OWA.marvell.com ([199.233.58.135]) (using TLSv1) by na3sys009aob127.postini.com ([74.125.148.12]) with SMTP ID DSNKUaSpyhcXFTkDcOGJpdBd0YqxyACvutTL@postini.com; Tue, 28 May 2013 05:58:04 PDT Received: from maili.marvell.com (10.93.76.43) by SC-OWA.marvell.com (10.93.76.28) with Microsoft SMTP Server id 8.3.213.0; Tue, 28 May 2013 05:57:16 -0700 Received: from [10.9.2.76] (unknown [10.9.2.76]) by maili.marvell.com (Postfix) with ESMTP id 405B41CCD9C; Tue, 28 May 2013 05:57:15 -0700 (PDT) Subject: Re: Low Tx-Rx performance with 10Gb NICs From: Axel Fischer To: Igor Mozolevsky In-Reply-To: References: <175CCF5F49938B4D99B2E3EF7F558EBE381FA6E5AA@SC-VEXCH4.marvell.com> <1369406798.20748.30.camel@EL-DT095.site> Content-Type: text/plain; charset="UTF-8" Date: Tue, 28 May 2013 14:57:11 +0200 Message-ID: <1369745831.14405.28.camel@EL-DT095.site> MIME-Version: 1.0 X-Mailer: Evolution 2.32.1 Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Tue, 28 May 2013 13:08:52 +0000 Cc: Hackers freeBSD , Ralf Assmann , Lino Sanfilippo , Markus Althoff X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 12:58:04 -0000 Hi Igor, all, thank you for your quick response regarding the 10GBit NIC performance. We noticed the following when using an Intel NIC as a reference NIC for our performance measurements: - We got the expected performance on FreeBSD 9.0 (32bit) and 9.1 (64bit) with: 1) LRO enabled (SW in Kernel,not on NIC) and 2) Processor affinity set for the receive interrupts of the NIC to one CPU (e.g.7). and 3) Processor affinity of the netperf process set to CPU 0 (rx) and CPU 1 (tx). But if we do not enable LRO or do not set processor affinity of the rx interrupts of the NIC we got a very bad performance of about 11 Gbit/s. Especially tx performance descreased dramatically durning a tx/rx netperf test using 4 tx streams and 4 rx streams ... => So my questions are: 1) Are "processor affinity" and "LRO" an essential requirement for appropriate duplex 10GBit performance ? 2) Why does tx performance decrease dramatically if we do not use LRO or proc.-affinity on the receive side ... ? 3) Is the behaviour probably related to the Intel platform (CPU related) (Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz) ...? Thanks you very much in advance, Axel Fischer Machine info: ============ hw.machine: amd64 hw.model: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz hw.ncpu: 4 hw.byteorder: 1234 hw.physmem: 17052545024 hw.usermem: 16792178688 hw.pagesize: 4096 hw.floatingpoint: 1 hw.machine_arch: amd64 hw.realmem: 17706254336 ========================================================= -------- Weitergeleitete Nachricht -------- Von: Igor Mozolevsky An: Axel Fischer Kopie: Igor Mozolevsky , Lino Sanfilippo , Hackers freeBSD , Ralf Assmann , Markus Althoff Betreff: Low Tx-Rx performance with 10Gb NICs Datum: Fri, 24 May 2013 08:41:25 -0700 On Friday, 24 May 2013, Axel Fischer wrote: Additionally I noticed the following TCP errors with netstat -s ...: 1186 data packets (1717328 bytes) retransmitted 6847875 window update packets 2319 duplicate acks 25831 out-of-order packets (37403288 bytes) 3733 discarded due to memory problems (drops) 1186 segment rexmits in SACK recovery episodes 1717328 byte rexmits in SACK recovery episodes Looks like your data is flooding your memory buffers, have a look through https://calomel.org/freebsd_network_tuning.html -- Igor M. -- Axel Fischer | R&D Software / SW Engineer | Marvell Semiconductor Germany GmbH Office +49 (7243) 502 370 | Fax +49 (7243) 502 982 afischer@marvell.com M A R V E L L |www.marvell.com This communication, together with any attachments hereto or links contained herein, is for the sole use of the intended recipient(s) and may contain information that is confidential or legally protected. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, dissemination, distribution or use of this communication is STRICTLY PROHIBITED. If you have received this communication in error, please notify the sender immediately by return e-mail message and delete the original and all copies of the communication, along with any attachments hereto or links herein, from your system. Marvell Semiconductor Germany GmbH, Siemensstr. 23, 76275 Ettlingen, Amtsgericht Mannheim HRB 361620 Geschäftsführer: Dipl.-Volksw. Mathias Horak From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 14:05:13 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D79BAF61 for ; Tue, 28 May 2013 14:05:13 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 419A7D8A for ; Tue, 28 May 2013 14:05:12 +0000 (UTC) Received: (qmail 92785 invoked from network); 28 May 2013 15:03:41 -0000 Received: from unknown (HELO [62.48.0.94]) ([62.48.0.94]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 28 May 2013 15:03:41 -0000 Message-ID: <51A4B991.3070805@freebsd.org> Date: Tue, 28 May 2013 16:05:05 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Orit Moskovich Subject: Re: preemptive kernel References: <981733489AB3BD4DB24B48340F53E0A55B0D5590@MTLDAG01.mtl.com> <20130526154752.GT3047@kib.kiev.ua> <981733489AB3BD4DB24B48340F53E0A55B0D56E0@MTLDAG01.mtl.com> <20130527063432.GY3047@kib.kiev.ua> <51A306A8.1010201@FreeBSD.org> <981733489AB3BD4DB24B48340F53E0A55B0D57D1@MTLDAG01.mtl.com> <51A34EEA.9050609@FreeBSD.org> <981733489AB3BD4DB24B48340F53E0A55B0D59A0@MTLDAG01.mtl.com> In-Reply-To: <981733489AB3BD4DB24B48340F53E0A55B0D59A0@MTLDAG01.mtl.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Konstantin Belousov , "freebsd-hackers@freebsd.org" , Andriy Gapon X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 14:05:13 -0000 On 27.05.2013 14:29, Orit Moskovich wrote: >>From what I've read in subr_taskqueue.c taskqueue_swi, taskqueue_swi_giant and taskqueue_fast are all implemented using swi_add which calls ithread_create(). > Is there any performance difference between them. Is one of the above or ithread given to bus_setup_intr preferable on the other? It depends on what you intend to do. If it is a network packet based work load then you want to avoid locking in the driver RX path. With MSI-X and a dedicated RX vector (per queue if you have more than one, RSS) you can do a lock-free hybrid interrupt/polling and life-lock avoiding RX model. The RX handler thereby is split into two parts: 1) the interrupt acknowledger quieting the interrupt until re-enabled (many modern cards also support fire-once interrupts and you can omit this step) and schedule the RX ithread/task; 2) run the dedicated (one per RX queue) ithread/task and pull the packets out of the RX ring in a while(1) loop with a maybe_yield() every so many packets to check your quantum and send them up the stack. Re-enable the interrupt once no more packets are available. When packets contiguously come in you never get an interrupt and always stay in the RX loop. The maybe_yield() lets other processes run if you've used a full quantum and returns when your next quantum is available. To completely omit the locking on the RX path you must make sure to prevent the driver from going away while the RX ithread/task is running. Also you have to make sure that nothing else is touching the RX ring and other directly related data structures. To stop the driver the ithread/task and the interrupt handler have to be torn down and drained. When that function returns both are gone and you're safe. Pseudo-code: error = bus_setup_intr(dev, sc->drv_irq, INTR_TYPE_NET | INTR_MPSAFE, dev_intr_handler, dev_intr_task, sc, &sc->drv_intrcookie); int drv_intr_handler(void *arg) { struct drv_softc *sc = arg; uint32_t status; status = CSR_READ_4(sc, DRV_INTR_STAUS); /* not necessary on MSI-X */ if (!status) return (FILTER_STRAY); /* Was not for us, try next handler on shared IRQ */ CSR_WRITE_4(sc, DRV_INTR_SUPPRESS); /* only if not fire-once */ return (FILTER_SCHEDULE_THREAD); /* Run ithread/task */ } void drv_intr_task(void *arg) { /* look mom, no locks! */ struct drv_softc *sc = arg; bus_dmamap_sync(...); while (rx_pkts) { bus_dmamap_sync(...); bus_dmamap_unload(...); ... (*ifp->if_input)(ifp, m); maybe_yield(); if (i++ > 32) { drv_rx_refill(sc, ...); /* Amortize over a number of RX packets */ i = 0; } } CSR_WRITE_4(sc, DRV_INTR_ENABLE); } bus_teardown_intr(dev, sc->drv_irq, &sc->drv_intrcookie); I'm soon starting work to revamp the lower stack / driver interface, to document the expected behavior of both sides and the best common practices. Also the proliferation of code-duplication for certain advanced features will be tackled. Both the RX and the TX side are part of this project. -- Andre > -----Original Message----- > From: Andriy Gapon [mailto:avg@FreeBSD.org] > Sent: Monday, May 27, 2013 03:18 PM > To: Orit Moskovich > Cc: Konstantin Belousov; freebsd-hackers@freebsd.org > Subject: Re: preemptive kernel > > on 27/05/2013 10:21 Orit Moskovich said the following: >> What is actually the difference between deferring a filter routine's work using an ithread given to bus_setup_intr, or using the global taskqueue_swi (implemented using interrupt thread)? > > I think you mean taskqueue_fast. > The difference is only in how much code you need to write. I do not think there is any significant difference in the resulting functionality. > >> What do you mean that the functionality is locked under INTR_FILTER? > > Please see the code. You have to use option INTR_FILTER to get the behavior I described earlier. > >> -----Original Message----- >> From: Andriy Gapon [mailto:avg@FreeBSD.org] >> Sent: Monday, May 27, 2013 10:11 AM >> To: Konstantin Belousov >> Cc: Orit Moskovich; freebsd-hackers@freebsd.org >> Subject: Re: preemptive kernel >> >> on 27/05/2013 09:34 Konstantin Belousov said the following: >>> Having both filter and ithread for the same interrupt is apparently >>> possible but weird. I do not see anything which would prevent >>> interrupt filter from being executed while the ithread is running. >>> But again, this is very unusual setup. >> >> I wouldn't call it weird, but, yes, it is rare. It's a pretty normal configuration when the filter acts as a filter and the handler acts as a handler (in ithread). In other words, it would be a replacement for a configuration where a filter is used and the filter offloads actual work to non-interrupt context via a e.g. taskqueue. >> But, hmm, this functionality is probably locked under INTR_FILTER option. >> >> -- >> Andriy Gapon >> > > -- > Andriy Gapon > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" > > From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 14:14:23 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B32BC813 for ; Tue, 28 May 2013 14:14:23 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 34BF9E0E for ; Tue, 28 May 2013 14:14:21 +0000 (UTC) Received: (qmail 92833 invoked from network); 28 May 2013 15:12:55 -0000 Received: from unknown (HELO [62.48.0.94]) ([62.48.0.94]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 28 May 2013 15:12:55 -0000 Message-ID: <51A4BBBC.8020405@freebsd.org> Date: Tue, 28 May 2013 16:14:20 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: afischer@marvell.com Subject: Re: Low Tx-Rx performance with 10Gb NICs References: <175CCF5F49938B4D99B2E3EF7F558EBE381FA6E5AA@SC-VEXCH4.marvell.com> <1369406798.20748.30.camel@EL-DT095.site> In-Reply-To: <1369406798.20748.30.camel@EL-DT095.site> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 14:14:23 -0000 On 24.05.2013 16:46, Axel Fischer wrote: > Hi Igor, > > my name is Axel Fischer. working at Marvell SC. Hi Axel, > In addition to your reply to my colleague Lino > Sanfilippo I did some performance measurements > on FreeBSD 9 with a commercial 10 GBit network > card. Which driver? > Unlike on other OS the FreeBSD performance > for duplex rx/tx operation never exceeded the > limit about 9.5 GBit/s. Normally a performance > of at least 16 GBit (up to line speed 20 GBit > in duplex mode) is expected. > As Lino already mentioned the CPU/bus system > (in general the HW) does not set a limit. > Furthermore I noticed that the CPU(s) load > is not very high, about 30 %. Your RX/TX is probably serialized in the driver and you can only make use of one core. > Here is an overview of the measurements: > > netperf rx-tx 4 streams / 60s > > 1768.16 Mb/s Port=2001 RX > 999.33 Mb/s Port=2002 RX > 72.16 Mb/s Port=1001 TX > 61.49 Mb/s Port=1002 TX > 2302.76 Mb/s Port=2003 RX > 73.48 Mb/s Port=1003 TX > 2416.23 Mb/s Port=2004 RX > 76.02 Mb/s Port=1004 TX > ==== > RX+TX Total Result: Mb/s 7769.63 > > > CPU load: > > last pid: 1739; load averages: 0.97, 0.49, 0.21 up 0+00:02:26 > 11:02:52 > 46 processes: 2 running, 44 sleeping > CPU 0: 2.0% user, 0.0% nice, 23.2% system, 0.4% interrupt, 74.4% idle > CPU 1: 1.2% user, 0.0% nice, 19.7% system, 0.4% interrupt, 78.7% idle > CPU 2: 0.0% user, 0.0% nice, 0.0% system, 80.7% interrupt, 19.3% idle > CPU 3: 0.0% user, 0.0% nice, 0.8% system, 1.6% interrupt, 97.6% idle > CPU 4: 2.4% user, 0.0% nice, 25.6% system, 0.0% interrupt, 72.0% idle > CPU 5: 3.1% user, 0.0% nice, 25.6% system, 0.0% interrupt, 71.3% idle > CPU 6: 0.0% user, 0.0% nice, 0.4% system, 32.7% interrupt, 66.9% idle > CPU 7: 0.4% user, 0.0% nice, 1.6% system, 0.0% interrupt, 98.0% idle > Mem: 14M Active, 7548K Inact, 66M Wired, 24K Cache, 16M Buf, 3326M Free > Swap: 4096M Total, 4096M Free > > Additionally I noticed the following TCP errors > with netstat -s ...: > > 1186 data packets (1717328 bytes) retransmitted This may happen and is typically not cause for concern on a loaded system. > 6847875 window update packets Normal. > 2319 duplicate acks Related to the retransmits. > 25831 out-of-order packets (37403288 bytes) This is unusual. What kind of test setup do you have, back-to-back cards or a switch in between? Out of order normally shouldn't happen unless over the internet. > 3733 discarded due to memory problems (drops) > 1186 segment rexmits in SACK recovery episodes > 1717328 byte rexmits in SACK recovery episodes Again related to retransmits. > My questions: > > - What is the max. performance (duplex) on > FreeBSD 9 that you have measured with a 10 GBit > NIC ? > (Expected > 16 GBit/s on appropriate HW) Certain large CDN are known to push more than 20Gbit/s production traffic per machine. Please also my other message from today to hackers@ "Re: preemptive kernel" with Message-ID: <51A4B991.3070805@freebsd.org>. -- Andre > Thank you in advance, > Axel > > > -------- Weitergeleitete Nachricht -------- > Von: Igor Mozolevsky > An: Lino Sanfilippo > Kopie: Hackers freeBSD , Axel Fischer > , Ralf Assmann , Markus > Althoff > Betreff: Re: Low Tx-Rx performance with 10Gb NICs > Datum: Thu, 23 May 2013 11:21:08 -0700 > > On 23 May 2013 19:00, Lino Sanfilippo wrote: > >> Is there a known issue concerning high traffic on Tx and Rx paths? Are there any system >> settings I could adjust to get the expected performance? Any hints are very appreciated. > > check your ierrs and oerrs: netstat -s 1, I've noticed I'm getting > ierrs on em chips, but none on fxp chips (connected to the same > wire/switch); might be unrelated to yours, but worth a check... > > > From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 15:07:32 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8FD3D9F3 for ; Tue, 28 May 2013 15:07:32 +0000 (UTC) (envelope-from linnemannr@gmail.com) Received: from mail-pa0-f47.google.com (mail-pa0-f47.google.com [209.85.220.47]) by mx1.freebsd.org (Postfix) with ESMTP id 6BC3E124 for ; Tue, 28 May 2013 15:07:32 +0000 (UTC) Received: by mail-pa0-f47.google.com with SMTP id kl13so1872323pab.34 for ; Tue, 28 May 2013 08:07:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; bh=6M98iu3k9P57kT1m3PL3lTBOyLeICYqu0bEnqmY7rig=; b=dJGl68N3BtMtHWlVYprCPvA6FXRycpPHmHdFa19r2nBRFoqc/SJnEfVTT2XGj5fEOH zAtlXHXjs73s6dQtA5qcH2+jmRDtgXBHs984u1voqXkmcIsMaC+b7hil/mIGuoHgChB8 4yTjRvCR3gg+LBzSUrH22GP9MnDF93UcmAZ65x7wUdmw5EWEeQdVnck4ozWPzwfHKEza uVZd1S4dyOQaadtoRlzcIEzLUSzKVsVT5nwRhx4fQE68/KvsKsrV0CAJqpzQ0YLMfndv x/Ob4M8v+xFEULTyB5GkxEUXfIWq2XOEmJIbVpida9Gm5fIkjgR+kkSBUwak54zrKvHG YSUw== X-Received: by 10.68.196.196 with SMTP id io4mr33670794pbc.166.1369753651762; Tue, 28 May 2013 08:07:31 -0700 (PDT) Received: from [192.168.1.144] (ip70-185-206-95.ok.ok.cox.net. [70.185.206.95]) by mx.google.com with ESMTPSA id ig4sm5123332pbc.18.2013.05.28.08.07.29 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 28 May 2013 08:07:30 -0700 (PDT) Content-Type: text/plain; charset=iso-8859-1 Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: /bin/sh => STDIN & functions, var scope messing From: Reid Linnemann In-Reply-To: Date: Tue, 28 May 2013 10:07:58 -0500 Content-Transfer-Encoding: quoted-printable Message-Id: <47252E1F-0965-4772-AE40-865BE5D05CD8@gmail.com> References: <20130527.194235.693.1@DOMY-PC> To: Ryan Stone X-Mailer: Apple Mail (2.1503) Cc: rank1seeker@gmail.com, FreeBSD Hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 15:07:32 -0000 On May 28, 2013, at 7:00 AM, Ryan Stone wrote: > On Tue, May 28, 2013 at 5:48 AM, V=E1clav Zeman = wrote: > Curious. Which of the two behaviours is POSIXly correct? >=20 > I believe that /bin/sh's behaviour is correct. I don't know what = shell the manpage is referring to, but it's not bash (bash does the same = thing in a pipeline). Perhaps it's referring to csh? If that is the = case that line is probably causing more confusion rather than = alleviating it. I believe it's referring to csh, possible ksh as well. I tried a similar = experiment with tcsh: #!/bin/tcsh alias fn set var=3D12 set var=3D echo $var yes | fn echo $var= From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 15:07:49 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 59C81AE3 for ; Tue, 28 May 2013 15:07:49 +0000 (UTC) (envelope-from nwhitehorn@freebsd.org) Received: from mail.icecube.wisc.edu (trout.icecube.wisc.edu [128.104.255.119]) by mx1.freebsd.org (Postfix) with ESMTP id 1ED96131 for ; Tue, 28 May 2013 15:07:48 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by mail.icecube.wisc.edu (Postfix) with ESMTP id 6E5755835D; Tue, 28 May 2013 09:49:21 -0500 (CDT) X-Virus-Scanned: amavisd-new at icecube.wisc.edu Received: from mail.icecube.wisc.edu ([127.0.0.1]) by localhost (trout.icecube.wisc.edu [127.0.0.1]) (amavisd-new, port 10030) with ESMTP id MRxVYd2VWTCj; Tue, 28 May 2013 09:49:21 -0500 (CDT) Received: from terminus.icecube.wisc.edu (terminus.icecube.wisc.edu [172.16.223.97]) by mail.icecube.wisc.edu (Postfix) with ESMTP id 3507E58344; Tue, 28 May 2013 09:49:21 -0500 (CDT) Message-ID: <51A4C3F1.2010604@freebsd.org> Date: Tue, 28 May 2013 09:49:21 -0500 From: Nathan Whitehorn User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130404 Thunderbird/17.0.5 MIME-Version: 1.0 To: Alfred Perlstein Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> <51A39039.1070202@cran.org.uk> <51A39FEC.5070402@mu.org> <51A3A891.5060103@cran.org.uk> <51A3C202.9030802@mu.org> <51A3CEB6.3070200@cran.org.uk> <51A40AF2.2010108@mu.org> <51A40E37.9060702@freebsd.org> <51A4343F.3070605@mu.org> In-Reply-To: <51A4343F.3070605@mu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Bruce Cran , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 15:07:49 -0000 On 05/27/13 23:36, Alfred Perlstein wrote: > On 5/27/13 6:53 PM, Nathan Whitehorn wrote: >> On 05/27/13 20:40, Alfred Perlstein wrote: >>> On 5/27/13 2:23 PM, Bruce Cran wrote: >>>> On 27/05/2013 21:28, Alfred Perlstein wrote: >>>>> On 5/27/13 11:40 AM, Bruce Cran wrote: >>>>>> Yes. >>>>> Is this a joke? >>>> >>>> It probably /was/ too short a reply. Personally I think there >>>> should be a single UI and scripting interface across all platforms. >>>> We should try and get pc-sysinstall running on all of them first in >>>> case there's some problem that means it can't be done, in which >>>> case we'd need to use a different backend. >>>> >>> >>> There are just going to be certain platforms that make it EASY to do >>> cool things. We should embrace that! That's why there are >>> different platforms! >>> >>> Some are great for low power, others are great for graphics, cpu >>> power, gpu, networking etc. >>> >>> If we always go for the lowest common denominator then we are >>> crippling all the platforms for no one's benefit. Even if something >>> CAN be done, if it is very difficult, or just never happening, then >>> we can't limit everyone's experience based on the more difficult >>> and/or resource strapped platforms. >>> >>> It's just not good business. >> >> Yes, and all of this cuts both ways: pc-sysinstall has no wireless >> setup support, for instance. Right now we support what we support >> because it is the most feature-complete thing we have, not just on >> tier-2 platforms but also on x86. >> >> To bring this discussion back to the ground, the fact is that we lack >> an installer that has both internal support for ZFS and a UI. One of >> the reasons for this is that making a good expressive UI for ZFS is a >> non-trivial undertaking given its enormous flexibility. The >> bsdinstall partition editor has been written to be extensible for >> this, and several people have started writing code to do it, but no >> one ended up having time to finish. Probably a reasonable thing to do >> is to start with supporting only a minimal set of features. If anyone >> felt like actually writing this code, I'm sure it would be >> appreciated by all and be more productive than email exchanges. >> -Nathan > > I'm sure if there was a list of reasonable things, such as wireless > then pc-sysinstall could be augmented. This is the first I've heard > of that. All the other complaints have been based on portability. > > Is that all that is required now, wireless? There are more, as well. A partial list of missing features on both sides is here: https://wiki.freebsd.org/PCBSDInstallMerge. Other major ones are IPv6 (maybe this has changed?) and no jail setup feature. Most of the existing disk partitioning code in pc-sysinstall, which is the only thing in a FreeBSD installer that is at all complicated, is also *extremely* fragile and needs in all likelihood to be entirely replaced. The merge effort stalled because of this kind of issue -- doing a "merge" rapidly became rewriting both from scratch. -Nathan From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 14:51:21 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E2D7A60F; Tue, 28 May 2013 14:51:21 +0000 (UTC) (envelope-from afischer@marvell.com) Received: from na3sys009aog101.obsmtp.com (na3sys009aog101.obsmtp.com [74.125.149.67]) by mx1.freebsd.org (Postfix) with ESMTP id 1D62AFDB; Tue, 28 May 2013 14:51:20 +0000 (UTC) Received: from sc-owa02.marvell.com ([199.233.58.137]) (using TLSv1) by na3sys009aob101.postini.com ([74.125.148.12]) with SMTP ID DSNKUaTEYSee9enMk03OF4IysqFNWj38JMcA@postini.com; Tue, 28 May 2013 07:51:21 PDT Received: from maili.marvell.com (10.93.76.43) by sc-owa02.marvell.com (10.93.76.22) with Microsoft SMTP Server id 8.3.213.0; Tue, 28 May 2013 07:48:30 -0700 Received: from [10.9.2.76] (unknown [10.9.2.76]) by maili.marvell.com (Postfix) with ESMTP id EE9941CCD9C; Tue, 28 May 2013 07:48:29 -0700 (PDT) Subject: Re: Low Tx-Rx performance with 10Gb NICs From: Axel Fischer To: Andre Oppermann , , In-Reply-To: <51A4BBBC.8020405@freebsd.org> References: <175CCF5F49938B4D99B2E3EF7F558EBE381FA6E5AA@SC-VEXCH4.marvell.com> <1369406798.20748.30.camel@EL-DT095.site> <51A4BBBC.8020405@freebsd.org> Content-Type: text/plain; charset="UTF-8" Date: Tue, 28 May 2013 16:48:16 +0200 Message-ID: <1369752496.14405.42.camel@EL-DT095.site> MIME-Version: 1.0 X-Mailer: Evolution 2.32.1 Content-Transfer-Encoding: 8bit X-Mailman-Approved-At: Tue, 28 May 2013 15:34:54 +0000 Cc: freebsd-hackers X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 14:51:22 -0000 Hi Andre, all The driver we used is an Intel ixgbe driver. We use this driver as a reference for our own Marvell driver. As on Linux our approach is to guarantee a lock free data transmission between rx and tx, since our HW supports this. The ixgbe driver should not perform any serialization between rx and tx too. Furthermore we know from the FreeBSD performance guide that the Intel NIC really shows a performance of more than 18GBit/s .... The problem (low performance) that we have is likely related to the fact that we first did not enable LRO on the Intel driver, that means that we feed the protocol stack with single or multiple rx frames on a per interrupt base. Enabling LRO and setting processor affinity changed the behaviour and we finally saw 18 Gbit/s on Intel .... See my results / questions below: ================================= - We got the expected performance on FreeBSD 9.0 (32bit) and 9.1 (64bit) with (ixgbe): 1) LRO enabled (SW in Kernel,not on NIC) and 2) Processor affinity set for the receive interrupts of the NIC to one CPU (e.g.7). and 3) Processor affinity of the netperf process set to CPU 0 (rx) and CPU 1 (tx). But if we do not enable LRO or do not set processor affinity of the rx interrupts of the NIC we got a very bad performance of about 11 Gbit/s. Especially tx performance descreased dramatically durning a tx/rx netperf test using 4 tx streams and 4 rx streams ... => So my questions are: 1) Are "processor affinity" and "LRO" an essential requirement for appropriate duplex 10GBit performance ? 2) Why does tx performance decrease dramatically if we do not use LRO or proc.-affinity on the receive side ... ? 3) Is the behaviour probably related to the Intel platform (CPU related) (Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz) ...? Best Regards, Axel -------- Weitergeleitete Nachricht -------- Von: Andre Oppermann An: Axel Fischer Kopie: freebsd-hackers Betreff: Re: Low Tx-Rx performance with 10Gb NICs Datum: Tue, 28 May 2013 07:14:20 -0700 On 24.05.2013 16:46, Axel Fischer wrote: > Hi Igor, > > my name is Axel Fischer. working at Marvell SC. Hi Axel, > In addition to your reply to my colleague Lino > Sanfilippo I did some performance measurements > on FreeBSD 9 with a commercial 10 GBit network > card. Which driver? > Unlike on other OS the FreeBSD performance > for duplex rx/tx operation never exceeded the > limit about 9.5 GBit/s. Normally a performance > of at least 16 GBit (up to line speed 20 GBit > in duplex mode) is expected. > As Lino already mentioned the CPU/bus system > (in general the HW) does not set a limit. > Furthermore I noticed that the CPU(s) load > is not very high, about 30 %. Your RX/TX is probably serialized in the driver and you can only make use of one core. > Here is an overview of the measurements: > > netperf rx-tx 4 streams / 60s > > 1768.16 Mb/s Port=2001 RX > 999.33 Mb/s Port=2002 RX > 72.16 Mb/s Port=1001 TX > 61.49 Mb/s Port=1002 TX > 2302.76 Mb/s Port=2003 RX > 73.48 Mb/s Port=1003 TX > 2416.23 Mb/s Port=2004 RX > 76.02 Mb/s Port=1004 TX > ==== > RX+TX Total Result: Mb/s 7769.63 > > > CPU load: > > last pid: 1739; load averages: 0.97, 0.49, 0.21 up 0+00:02:26 > 11:02:52 > 46 processes: 2 running, 44 sleeping > CPU 0: 2.0% user, 0.0% nice, 23.2% system, 0.4% interrupt, 74.4% idle > CPU 1: 1.2% user, 0.0% nice, 19.7% system, 0.4% interrupt, 78.7% idle > CPU 2: 0.0% user, 0.0% nice, 0.0% system, 80.7% interrupt, 19.3% idle > CPU 3: 0.0% user, 0.0% nice, 0.8% system, 1.6% interrupt, 97.6% idle > CPU 4: 2.4% user, 0.0% nice, 25.6% system, 0.0% interrupt, 72.0% idle > CPU 5: 3.1% user, 0.0% nice, 25.6% system, 0.0% interrupt, 71.3% idle > CPU 6: 0.0% user, 0.0% nice, 0.4% system, 32.7% interrupt, 66.9% idle > CPU 7: 0.4% user, 0.0% nice, 1.6% system, 0.0% interrupt, 98.0% idle > Mem: 14M Active, 7548K Inact, 66M Wired, 24K Cache, 16M Buf, 3326M Free > Swap: 4096M Total, 4096M Free > > Additionally I noticed the following TCP errors > with netstat -s ...: > > 1186 data packets (1717328 bytes) retransmitted This may happen and is typically not cause for concern on a loaded system. > 6847875 window update packets Normal. > 2319 duplicate acks Related to the retransmits. > 25831 out-of-order packets (37403288 bytes) This is unusual. What kind of test setup do you have, back-to-back cards or a switch in between? Out of order normally shouldn't happen unless over the internet. > 3733 discarded due to memory problems (drops) > 1186 segment rexmits in SACK recovery episodes > 1717328 byte rexmits in SACK recovery episodes Again related to retransmits. > My questions: > > - What is the max. performance (duplex) on > FreeBSD 9 that you have measured with a 10 GBit > NIC ? > (Expected > 16 GBit/s on appropriate HW) Certain large CDN are known to push more than 20Gbit/s production traffic per machine. Please also my other message from today to hackers@ "Re: preemptive kernel" with Message-ID: <51A4B991.3070805@freebsd.org>. -- Axel Fischer | R&D Software / SW Engineer | Marvell Semiconductor Germany GmbH Office +49 (7243) 502 370 | Fax +49 (7243) 502 982 afischer@marvell.com M A R V E L L |www.marvell.com This communication, together with any attachments hereto or links contained herein, is for the sole use of the intended recipient(s) and may contain information that is confidential or legally protected. If you are not the intended recipient, you are hereby notified that any review, disclosure, copying, dissemination, distribution or use of this communication is STRICTLY PROHIBITED. If you have received this communication in error, please notify the sender immediately by return e-mail message and delete the original and all copies of the communication, along with any attachments hereto or links herein, from your system. Marvell Semiconductor Germany GmbH, Siemensstr. 23, 76275 Ettlingen, Amtsgericht Mannheim HRB 361620 Geschäftsführer: Dipl.-Volksw. Mathias Horak From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 15:54:27 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D09B778D; Tue, 28 May 2013 15:54:27 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id B1F113C5; Tue, 28 May 2013 15:54:27 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id 855741A3C37; Tue, 28 May 2013 08:54:18 -0700 (PDT) Message-ID: <51A4D329.5060103@mu.org> Date: Tue, 28 May 2013 08:54:17 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Nathan Whitehorn Subject: Re: FreeBSD installers and future direction References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> <51A39039.1070202@cran.org.uk> <51A39FEC.5070402@mu.org> <51A3A891.5060103@cran.org.uk> <51A3C202.9030802@mu.org> <51A3CEB6.3070200@cran.org.uk> <51A40AF2.2010108@mu.org> <51A40E37.9060702@freebsd.org> <51A4343F.3070605@mu.org> <51A4C3F1.2010604@freebsd.org> In-Reply-To: <51A4C3F1.2010604@freebsd.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Bruce Cran , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 15:54:27 -0000 On 5/28/13 7:49 AM, Nathan Whitehorn wrote: > On 05/27/13 23:36, Alfred Perlstein wrote: >> On 5/27/13 6:53 PM, Nathan Whitehorn wrote: >>> On 05/27/13 20:40, Alfred Perlstein wrote: >>>> On 5/27/13 2:23 PM, Bruce Cran wrote: >>>>> On 27/05/2013 21:28, Alfred Perlstein wrote: >>>>>> On 5/27/13 11:40 AM, Bruce Cran wrote: >>>>>>> Yes. >>>>>> Is this a joke? >>>>> >>>>> It probably /was/ too short a reply. Personally I think there >>>>> should be a single UI and scripting interface across all >>>>> platforms. We should try and get pc-sysinstall running on all of >>>>> them first in case there's some problem that means it can't be >>>>> done, in which case we'd need to use a different backend. >>>>> >>>> >>>> There are just going to be certain platforms that make it EASY to >>>> do cool things. We should embrace that! That's why there are >>>> different platforms! >>>> >>>> Some are great for low power, others are great for graphics, cpu >>>> power, gpu, networking etc. >>>> >>>> If we always go for the lowest common denominator then we are >>>> crippling all the platforms for no one's benefit. Even if >>>> something CAN be done, if it is very difficult, or just never >>>> happening, then we can't limit everyone's experience based on the >>>> more difficult and/or resource strapped platforms. >>>> >>>> It's just not good business. >>> >>> Yes, and all of this cuts both ways: pc-sysinstall has no wireless >>> setup support, for instance. Right now we support what we support >>> because it is the most feature-complete thing we have, not just on >>> tier-2 platforms but also on x86. >>> >>> To bring this discussion back to the ground, the fact is that we >>> lack an installer that has both internal support for ZFS and a UI. >>> One of the reasons for this is that making a good expressive UI for >>> ZFS is a non-trivial undertaking given its enormous flexibility. The >>> bsdinstall partition editor has been written to be extensible for >>> this, and several people have started writing code to do it, but no >>> one ended up having time to finish. Probably a reasonable thing to >>> do is to start with supporting only a minimal set of features. If >>> anyone felt like actually writing this code, I'm sure it would be >>> appreciated by all and be more productive than email exchanges. >>> -Nathan >> >> I'm sure if there was a list of reasonable things, such as wireless >> then pc-sysinstall could be augmented. This is the first I've heard >> of that. All the other complaints have been based on portability. >> >> Is that all that is required now, wireless? > > There are more, as well. A partial list of missing features on both > sides is here: https://wiki.freebsd.org/PCBSDInstallMerge. Other major > ones are IPv6 (maybe this has changed?) and no jail setup feature. > Most of the existing disk partitioning code in pc-sysinstall, which is > the only thing in a FreeBSD installer that is at all complicated, is > also *extremely* fragile and needs in all likelihood to be entirely > replaced. The merge effort stalled because of this kind of issue -- > doing a "merge" rapidly became rewriting both from scratch. > -Nathan > Ah this is so cool. I'll bring it up with the PCBSD folks today. Thank you Nathan. -Alfred From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 17:15:55 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DEB3F972; Tue, 28 May 2013 17:15:55 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 963EF9C1; Tue, 28 May 2013 17:15:55 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.15]) by ltcfislmsgpa06.fnfis.com (8.14.5/8.14.5) with ESMTP id r4SHFjA1001585 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Tue, 28 May 2013 12:15:45 -0500 Received: from LTCFISWMSGMB21.FNFIS.com ([10.132.99.23]) by LTCFISWMSGHT04.FNFIS.com ([10.132.206.15]) with mapi id 14.02.0309.002; Tue, 28 May 2013 12:15:45 -0500 From: "Teske, Devin" To: Alfred Perlstein Subject: Re: FreeBSD installers and future direction Thread-Topic: FreeBSD installers and future direction Thread-Index: AQHOWV60z6k72swSkkmGANUzy/EMLJkWZ1wAgAAkfQCAAE6UgIAAH7eAgAAeuACAAA8cgIACXGSAgAAS94CAABK3AIAACk6AgAAeVACAAA8kAIAAR84AgAAD5oCAAC1WgIAAq02AgAASJICAABbCAA== Date: Tue, 28 May 2013 17:15:45 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F6484E@ltcfiswmsgmb21> References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> <51A39039.1070202@cran.org.uk> <51A39FEC.5070402@mu.org> <51A3A891.5060103@cran.org.uk> <51A3C202.9030802@mu.org> <51A3CEB6.3070200@cran.org.uk> <51A40AF2.2010108@mu.org> <51A40E37.9060702@freebsd.org> <51A4343F.3070605@mu.org> <51A4C3F1.2010604@freebsd.org> <51A4D329.5060103@mu.org> In-Reply-To: <51A4D329.5060103@mu.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-28_07:2013-05-28,2013-05-28,1970-01-01 signatures=0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Bruce Cran , FreeBSD Hackers , Devin Teske , Nathan Whitehorn X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 17:15:55 -0000 On May 28, 2013, at 8:54 AM, Alfred Perlstein wrote: On 5/28/13 7:49 AM, Nathan Whitehorn wrote: On 05/27/13 23:36, Alfred Perlstein wrote: On 5/27/13 6:53 PM, Nathan Whitehorn wrote: On 05/27/13 20:40, Alfred Perlstein wrote: On 5/27/13 2:23 PM, Bruce Cran wrote: On 27/05/2013 21:28, Alfred Perlstein wrote: On 5/27/13 11:40 AM, Bruce Cran wrote: Yes. Is this a joke? It probably /was/ too short a reply. Personally I think there should be a s= ingle UI and scripting interface across all platforms. We should try and ge= t pc-sysinstall running on all of them first in case there's some problem t= hat means it can't be done, in which case we'd need to use a different back= end. There are just going to be certain platforms that make it EASY to do cool t= hings. We should embrace that! That's why there are different platforms! Some are great for low power, others are great for graphics, cpu power, gpu= , networking etc. If we always go for the lowest common denominator then we are crippling all= the platforms for no one's benefit. Even if something CAN be done, if it = is very difficult, or just never happening, then we can't limit everyone's = experience based on the more difficult and/or resource strapped platforms. It's just not good business. Yes, and all of this cuts both ways: pc-sysinstall has no wireless setup su= pport, for instance. Right now we support what we support because it is the= most feature-complete thing we have, not just on tier-2 platforms but also= on x86. To bring this discussion back to the ground, the fact is that we lack an in= staller that has both internal support for ZFS and a UI. One of the reasons= for this is that making a good expressive UI for ZFS is a non-trivial unde= rtaking given its enormous flexibility. The bsdinstall partition editor has= been written to be extensible for this, and several people have started wr= iting code to do it, but no one ended up having time to finish. Probably a = reasonable thing to do is to start with supporting only a minimal set of fe= atures. If anyone felt like actually writing this code, I'm sure it would b= e appreciated by all and be more productive than email exchanges. -Nathan I'm sure if there was a list of reasonable things, such as wireless then pc= -sysinstall could be augmented. This is the first I've heard of that. All= the other complaints have been based on portability. Is that all that is required now, wireless? There are more, as well. A partial list of missing features on both sides i= s here: https://wiki.freebsd.org/PCBSDInstallMerge. Other major ones are IP= v6 (maybe this has changed?) and no jail setup feature. Most of the existin= g disk partitioning code in pc-sysinstall, which is the only thing in a Fre= eBSD installer that is at all complicated, is also *extremely* fragile and = needs in all likelihood to be entirely replaced. The merge effort stalled b= ecause of this kind of issue -- doing a "merge" rapidly became rewriting bo= th from scratch. -Nathan Ah this is so cool. I'll bring it up with the PCBSD folks today. Thank you Nathan. I had my own look at the pc-sysinstall and bsdinstall code and came to the = same conclusions, plus some. One of the biggest obstacles I see is actually a high-level issue that I've= self-identified through extensive work on bsdconfig (which is both a back-= end and a front-end). This is the issue of debugging and namespaces. I've sat down and made lists of other issues=85 but when I review, I find t= hese issues to be secondary to the above-stated larger issues. Concretely, I'm saying thus: + bsdinstall lacks debugging (debugging is different than logging; from wha= t I could see BSDINSTALL_LOG -- although utilized by both the sh(1) side an= d the C side -- is only populated during an installation). The ability to h= ave the type of debugging that is in bsdconfig would greatly diminish the a= mount of time developing important new features. + pc-sysinstall lacks debugging (similar situation=85 producing a log for s= ome action is not the same as being able to have debug statements for the p= urpose of enhancement the program or troubleshooting an enhancement) + bsdinstall separates the backend functionality and the front-end function= ality into two separate namespaces (and in the case of C binaries, a third = namespace) + pc-sysinstall separates one backend into more than one namespace =3D=3D=3D To get an idea of the type of debugging I'm talking about, install sysutils= /bsdconfig from the ports tree or install it from a HEAD checkout of base (= it's in usr.sbin) and execute: bsdconfig -d # produce debugging statements on stdout collated in realtime with the dial= og screens or bsdconfig -D fooLogfile # produce debugging statements in "fooLogfile" (debugging statements are hi= dden from stdout) or bsdconfig -D +fooLogfile # produce debugging statements in both "fooLogfile" *and* on stdout (this i= s the same as "-dD fooLogfile") As this stuff gets more modular and there are back-ends, front-ends, global= APIs, local APIs, etc. etc. Having the ability to (after noticing a problem) flip a switch and then get= an almost exact location of where you currently-are within the code just b= y looking at debug statements is extremely helpful in being able to locate = the problem. =3D=3D=3D The namespace argument is a bit harder to explain (and quantify for compari= son). In bsdinstall, we see namespace separation most readily by looking at the w= ay the C binaries work. The namespace separation in this case means that (despite the fact that the= C components do a getenv(3) to acquire $BSDINSTALL_LOG, for example) for t= he large part, the C aspects cannot enjoy code written to extend the sh(1) = aspect, and vice-versa. So if you want a nice debug library that acts in a single way for bsdinstal= l=85 that's going to be difficult (but not impossible=85 you could perhaps = cheat by having both the sh(1)-side and the C-side use a logger(1)/syslog(3= ) facility =85 but then getting that debug information integrated in the ab= ove manner is still non-trivial). On the other-hand=85 pc-sysinstall doesn't suffer from the same namespace p= roblems (it's 100% shell), but it's still not conflated as-is done in bsdco= nfig. In pc-sysinstall what you'll find is that instead of putting functionality = into actual functions=85 it creates shell scripts that operate in a separat= e namespace as they are executed as binaries rather than taking advantage o= f their shell-based nature (in other words=85 because it's 100% shell, it s= hould perhaps embrace the ability to avoid forking all the time and run eve= rything in a conflated [single] namespace). =3D=3D=3D And as Nathan points out=85 it quickly starts to look like a rewrite of bot= h to do a merge. However=85 the type of "merge" that was being talked about in the above URL= (reproduced below from the above reply text for convenience): https://wiki.freebsd.org/PCBSDInstallMerge Is not a merge that would see a single namespace emerge. In the above URL, the type of "merge" that's being posited is the kind wher= e bsdinstall becomes a front-end only and will call-out to everything that = pc-sysinstall provides. This could only be bad if pc-sysinstall is left in its current state, becau= se pc-sysinstall expects you to treat the largest portions of its functiona= lity as black-box executables (e.g. you call "delete-part.sh" with argument= s=85 rather than loading a shell library with a delete-part() function). Debugging is harder when you have to cross a namespace threshold. Perhaps a better style of merge would be the traditional use of the word=85 That is to say, that pc-sysinstall should be slurped *into* bsdinstall (bsd= install being the newer entity on the block=85 so it has more up-to-date co= ding style, albeit not the latest; contrasted to pc-sysinstall's dated inco= nsistencies within its own code-base -- so a merge in the opposite directio= n, of giving pc-sysinstall a user interface, would be harder). However (again, with these however statements)=85 If the idea is to merge pc-sysinstall and bsdinstall to solve the issue of = too many namespaces and the equally-distression problem of not having a deb= ug layer (which is only marginally helpful if you have a fractured namespac= e design)=85 =85 I think we could do a lot better. Perhaps not better with the out-come (which would be hard to judge before t= he product is truly envisaged), but with respect to disruptiveness. I recognize that any forward motion in the bsdinstall or pc-sysinstall camp= would be disruptive to the people dependent on those technologies. Meanwhile, nobody is depending on bsdconfig at the moment. A less (or perhaps, totally non-) disruptive path would be to merge the two= entities (bsdinstall and pc-sysinstall) into a third entity (bsdconfig) wh= ere the third entity has much more freedom to play. The end result would be that, when bsdconfig does indeed incorporate all th= e migrated functionality (and rewritten to achieve the desired enhancements= to make development and maintenance easier), we then -- and only then -- d= isrupt things by wholly replacing them. -- Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 17:28:49 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A9E08C0C; Tue, 28 May 2013 17:28:49 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 761CAA36; Tue, 28 May 2013 17:28:48 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.15]) by ltcfislmsgpa01.fnfis.com (8.14.5/8.14.5) with ESMTP id r4SHSlwa027766 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Tue, 28 May 2013 12:28:47 -0500 Received: from LTCFISWMSGMB21.FNFIS.com ([10.132.99.23]) by LTCFISWMSGHT04.FNFIS.com ([10.132.206.15]) with mapi id 14.02.0309.002; Tue, 28 May 2013 12:28:47 -0500 From: "Teske, Devin" To: Reid Linnemann Subject: Re: /bin/sh => STDIN & functions, var scope messing Thread-Topic: /bin/sh => STDIN & functions, var scope messing Thread-Index: AQHOWxJjvvyJSeBFUUSS2KYxPIq87pkZxr8AgADoBoCAACTsAIAANEIAgAAnVwA= Date: Tue, 28 May 2013 17:28:47 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F64904@ltcfiswmsgmb21> References: <20130527.194235.693.1@DOMY-PC> <47252E1F-0965-4772-AE40-865BE5D05CD8@gmail.com> In-Reply-To: <47252E1F-0965-4772-AE40-865BE5D05CD8@gmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] Content-Type: text/plain; charset="Windows-1252" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-28_08:2013-05-28,2013-05-28,1970-01-01 signatures=0 Cc: "rank1seeker@gmail.com" , Devin Teske , FreeBSD Hackers , Ryan Stone X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 17:28:49 -0000 On May 28, 2013, at 8:07 AM, Reid Linnemann wrote: >=20 > On May 28, 2013, at 7:00 AM, Ryan Stone wrote: >=20 >> On Tue, May 28, 2013 at 5:48 AM, V=E1clav Zeman wro= te: >> Curious. Which of the two behaviours is POSIXly correct? >>=20 >> I believe that /bin/sh's behaviour is correct. I don't know what shell = the manpage is referring to, but it's not bash (bash does the same thing in= a pipeline). Perhaps it's referring to csh? If that is the case that lin= e is probably causing more confusion rather than alleviating it. >=20 > I believe it's referring to csh, possible ksh as well. I tried a similar = experiment with tcsh: >=20 > #!/bin/tcsh > alias fn set var=3D12 > set var=3D > echo $var > yes | fn > echo $var I wonder why the sh(1) manual would be referring to csh(1). CSH does more than this, so you know=85 #!/bin/csh true | true | false | true # returns false #!/bin/sh true | true | false | true # returns true So not only must you be aware that sh(1) throws away variables assigned wit= hin a shell running as an rvalue to any pipe (because said statements are r= un in a sub-shell with a transient namespace initialized from the parent)=85 You must also be aware that return status of an lvalue operand of a pipe is= not retained along the chain. The entire chain is traversed (all rvalues o= f all pipes are invoked), but in sh(1) only the return status of last rvalu= e is returned. I see this problem running rampant in the pc-sysinstall code. For example=85 in pc-sysinstall=85 ./backend/functions-extractimage.sh- tar cvf - . 2>/dev/null | tar -xp= v -C ${FSMNT} ${TAROPTS} -f - 2>&1 | tee -a ${FSMNT}/.tar-extract.log ./backend/functions-extractimage.sh: if [ $? -ne 0 ] That's a big no-no. If your first tar (the initial lvalue to the first pipe) fails=85 but your = second tar succeeds (rvalue to the first pipe)=85 you won't catch that fail= ure in your checking of $? Also, if the first tar succeeds, but the second tar fails, AND the final rv= alue (the tee) succeeds=85 you also miss the error code. I call this the "piped return-status conflation issue". Basically=85 anytim= e you want to check the return-status in shell=85 and you care about lvalue= -failures in a pipe-chain=85 you must rewrite it to either: (a) be aware of the problem (I've in the past written wrappers that will te= st the previous return status and abort the chain in such an event) (b) undo the pipe-chain and store your results for incremental processing= =85 checking the return status after each filter. --=20 Devin NOTE: I'm responding on another thread that is related to pc-sysinstall cha= nges. In that thread, I didn't mention the above faux-pas of pc-sysinstall = because I really do consider things like this to be secondary to the over-a= rching namespace and debugging issues. _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 20:35:06 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 401817D9; Tue, 28 May 2013 20:35:06 +0000 (UTC) (envelope-from alfred@ixsystems.com) Received: from mail.iXsystems.com (newknight.ixsystems.com [206.40.55.70]) by mx1.freebsd.org (Postfix) with ESMTP id 1357F609; Tue, 28 May 2013 20:35:06 +0000 (UTC) Received: from localhost (mail.ixsystems.com [10.2.55.1]) by mail.iXsystems.com (Postfix) with ESMTP id 76192FDEC; Tue, 28 May 2013 13:35:05 -0700 (PDT) Received: from mail.iXsystems.com ([10.2.55.1]) by localhost (mail.ixsystems.com [10.2.55.1]) (maiad, port 10024) with ESMTP id 84638-08; Tue, 28 May 2013 13:35:05 -0700 (PDT) Received: from Alfreds-MacBook-Pro-9.local (unknown [10.8.0.22]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mail.iXsystems.com (Postfix) with ESMTPSA id 295D4FDE6; Tue, 28 May 2013 13:35:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=ixsystems.com; s=newknight0; t=1369773305; bh=AGCcC6e5/D+65vmVLr1NOeopCEfDOIcPYeKp6NMHPS4=; h=Date:From:To:CC:Subject:References:In-Reply-To; b=bFp/0Y+6OIVpEgUO3Ek7y9Z3GMqJVBJnq+KaS4sTnZcBW9bkqhRhNVYcajoDqZxaZ 72foWOvkvbCimj+CPHTNnfUWMhnDcvOsMKesoo6+eOsxLtnr4vGzvWbmfbzqgYWdbY l7GpPtEjF+CgvJr1d0zg4MDKsxY9lJRPVJPhYgEc= Message-ID: <51A514F5.9050405@ixsystems.com> Date: Tue, 28 May 2013 13:35:01 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: John Baldwin Subject: Re: please review, patch for lost camisr References: <51A44B0C.8010908@ixsystems.com> <201305281204.14146.jhb@freebsd.org> <51A505A5.7030105@ixsystems.com> <201305281613.32414.jhb@freebsd.org> In-Reply-To: <201305281613.32414.jhb@freebsd.org> Content-Type: text/plain; charset=ISO-8859-15; format=flowed Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Tue, 28 May 2013 21:35:20 +0000 Cc: Alexander Motin , hackers@freebsd.org, Xin Li X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 20:35:06 -0000 [[ moved to hackers@ from private mail. ]] On 5/28/13 1:13 PM, John Baldwin wrote: > On Tuesday, May 28, 2013 3:29:41 pm Alfred Perlstein wrote: >> On 5/28/13 9:04 AM, John Baldwin wrote: >>> On Tuesday, May 28, 2013 2:13:32 am Alfred Perlstein wrote: >>>> Hey folks, >>>> >>>> I had a talk with Nathan Whitehorn about the camisr issue. The issue we >>>> are seeing we mostly know, but to summarize, we are losing the camisr >>>> signal and the camisr is not being run. >>>> >>>> I gave him a summary of what we have been seeing and pointed him to the >>>> code I am concerned about here: >>>> http://pastebin.com/tLKr7mCV (this is inside of kern_intr.c). >>>> >>>> What I think that is happening is that the setting of it_need to 0 >>>> inside of sys/kern/kern_intr.c:ithread_loop() is not being scheduled >>>> correctly and it is being delayed until AFTER the call to >>>> ithread_execute_handlers() right below the atomic_store_rel_int(). >>> This seems highly unlikely, to the extent that if this were true all our >>> locking primitives would be broken. The store_rel is actually a release >>> barrier which acts like more a read/write fence. No memory accesses (read or >>> write) from before the release can be scheduled after the associated store, >>> either by the compiler or CPU. That is what Konstantin is referring to in his >>> commit when he says "release" semantics". >> Yes, that makes sense, however does it specify that the writes *must* >> occur at that *point*? If it only enforces ordering then we may have >> some issue, specifically because the setting of it to '1' inside of >> intr_event_schedule_thread has no barrier other than the acq semantics >> of the thread lock. I am wondering what is forcing out the '1' there. > Nothing ever forces writes. You would have to invalidate the cache to do that > and that is horribly expensive. It is always only about ordering and knowing > that if you can complete another operation on the same "cookie" variable with > acquire semantics that earlier writes will be visible. By cookie, you mean a specific memory address, basically a lock? This is starting to reinforce my suspicions as the setting of it_need is done with release semantics, however the acq on the other CPU is done on the thread lock. Maybe that is irrelevant. We will find out shortly. > >> See below as I think we have proof that this is somehow happening. > Having ih_need of 1 and it_need of 0 is certainly busted. The simplest fix > is probably to stop using atomics on it_need and just grab the thread lock > in the main ithread loop and hold it while checking and clearing it_need. > OK, we have some code that will either prove this, or perturb the memory ordering enough to make the bug go away, or prove this assertion wrong. We will update on our findings hopefully in the next few days. Thank you for your advice. -Alfred -Alfred From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 23:16:28 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 05083F11; Tue, 28 May 2013 23:16:28 +0000 (UTC) (envelope-from superbisquit@gmail.com) Received: from mail-ob0-x232.google.com (mail-ob0-x232.google.com [IPv6:2607:f8b0:4003:c01::232]) by mx1.freebsd.org (Postfix) with ESMTP id A74251B3; Tue, 28 May 2013 23:16:27 +0000 (UTC) Received: by mail-ob0-f178.google.com with SMTP id fb19so2545403obc.9 for ; Tue, 28 May 2013 16:16:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=QGxcfMAUMJfXIRLMN8jVtz5UY8mdidPKN+Isrx1vsJQ=; b=QxB6LS08QxoOIsljJb8KX4LZpV1xapN3taOqEuQE7CGyVztOWOZkE2nSGbRkVo5fpF qLDiqA+KKr6x3QVEyFnQ2ehWQWlvhFpzj+3R3lGICRKlnM4v1grXCjAQ8CGUyBiZWwD3 pBgq1sEn80h/257Op9E88MWEXj/dNNKWV2o9qzq+QkDnWCQXbcvhn3VArB5Nl1Fi7HNC OAtvJOXzKVzLMUtbbyBLJ+JLhJszPtyzOwjZp6kfgmQeGJcbrhoKYSo2v6B6dqEvB/f1 qtECBU6eysDZnOHAajvdKEqJ99BoigcYeNIVDs/n+X3TX5Ac6VV0275OU8Z7G/cUeA3v 19Fg== MIME-Version: 1.0 X-Received: by 10.60.134.204 with SMTP id pm12mr36826oeb.67.1369782987274; Tue, 28 May 2013 16:16:27 -0700 (PDT) Received: by 10.182.53.231 with HTTP; Tue, 28 May 2013 16:16:27 -0700 (PDT) In-Reply-To: <13CA24D6AB415D428143D44749F57D7201F6484E@ltcfiswmsgmb21> References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> <51A39039.1070202@cran.org.uk> <51A39FEC.5070402@mu.org> <51A3A891.5060103@cran.org.uk> <51A3C202.9030802@mu.org> <51A3CEB6.3070200@cran.org.uk> <51A40AF2.2010108@mu.org> <51A40E37.9060702@freebsd.org> <51A4343F.3070605@mu.org> <51A4C3F1.2010604@freebsd.org> <51A4D329.5060103@mu.org> <13CA24D6AB415D428143D44749F57D7201F6484E@ltcfiswmsgmb21> Date: Tue, 28 May 2013 19:16:27 -0400 Message-ID: Subject: Re: FreeBSD installers and future direction From: Super Bisquit To: Devin Teske Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Bruce Cran , FreeBSD Hackers , Alfred Perlstein , Nathan Whitehorn X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 23:16:28 -0000 In the case of firmware loaded systems, all of them aren't going to work with a single boot loader. On Tue, May 28, 2013 at 1:15 PM, Teske, Devin wr= ote: > > On May 28, 2013, at 8:54 AM, Alfred Perlstein wrote: > > On 5/28/13 7:49 AM, Nathan Whitehorn wrote: > On 05/27/13 23:36, Alfred Perlstein wrote: > On 5/27/13 6:53 PM, Nathan Whitehorn wrote: > On 05/27/13 20:40, Alfred Perlstein wrote: > On 5/27/13 2:23 PM, Bruce Cran wrote: > On 27/05/2013 21:28, Alfred Perlstein wrote: > On 5/27/13 11:40 AM, Bruce Cran wrote: > Yes. > Is this a joke? > > It probably /was/ too short a reply. Personally I think there should be a > single UI and scripting interface across all platforms. We should try and > get pc-sysinstall running on all of them first in case there's some probl= em > that means it can't be done, in which case we'd need to use a different > backend. > > > There are just going to be certain platforms that make it EASY to do cool > things. We should embrace that! That's why there are different platform= s! > > Some are great for low power, others are great for graphics, cpu power, > gpu, networking etc. > > If we always go for the lowest common denominator then we are crippling > all the platforms for no one's benefit. Even if something CAN be done, i= f > it is very difficult, or just never happening, then we can't limit > everyone's experience based on the more difficult and/or resource strappe= d > platforms. > > It's just not good business. > > Yes, and all of this cuts both ways: pc-sysinstall has no wireless setup > support, for instance. Right now we support what we support because it is > the most feature-complete thing we have, not just on tier-2 platforms but > also on x86. > > To bring this discussion back to the ground, the fact is that we lack an > installer that has both internal support for ZFS and a UI. One of the > reasons for this is that making a good expressive UI for ZFS is a > non-trivial undertaking given its enormous flexibility. The bsdinstall > partition editor has been written to be extensible for this, and several > people have started writing code to do it, but no one ended up having tim= e > to finish. Probably a reasonable thing to do is to start with supporting > only a minimal set of features. If anyone felt like actually writing this > code, I'm sure it would be appreciated by all and be more productive than > email exchanges. > -Nathan > > I'm sure if there was a list of reasonable things, such as wireless then > pc-sysinstall could be augmented. This is the first I've heard of that. > All the other complaints have been based on portability. > > Is that all that is required now, wireless? > > There are more, as well. A partial list of missing features on both sides > is here: https://wiki.freebsd.org/PCBSDInstallMerge. Other major ones are > IPv6 (maybe this has changed?) and no jail setup feature. Most of the > existing disk partitioning code in pc-sysinstall, which is the only thing > in a FreeBSD installer that is at all complicated, is also *extremely* > fragile and needs in all likelihood to be entirely replaced. The merge > effort stalled because of this kind of issue -- doing a "merge" rapidly > became rewriting both from scratch. > -Nathan > > Ah this is so cool. I'll bring it up with the PCBSD folks today. > > Thank you Nathan. > > > I had my own look at the pc-sysinstall and bsdinstall code and came to th= e > same conclusions, plus some. > > One of the biggest obstacles I see is actually a high-level issue that > I've self-identified through extensive work on bsdconfig (which is both a > back-end and a front-end). > > This is the issue of debugging and namespaces. > > I've sat down and made lists of other issues=85 but when I review, I find > these issues to be secondary to the above-stated larger issues. > > Concretely, I'm saying thus: > > + bsdinstall lacks debugging (debugging is different than logging; from > what I could see BSDINSTALL_LOG -- although utilized by both the sh(1) si= de > and the C side -- is only populated during an installation). The ability = to > have the type of debugging that is in bsdconfig would greatly diminish th= e > amount of time developing important new features. > > + pc-sysinstall lacks debugging (similar situation=85 producing a log for > some action is not the same as being able to have debug statements for th= e > purpose of enhancement the program or troubleshooting an enhancement) > > + bsdinstall separates the backend functionality and the front-end > functionality into two separate namespaces (and in the case of C binaries= , > a third namespace) > > + pc-sysinstall separates one backend into more than one namespace > > =3D=3D=3D > > To get an idea of the type of debugging I'm talking about, install > sysutils/bsdconfig from the ports tree or install it from a HEAD checkout > of base (it's in usr.sbin) and execute: > > bsdconfig -d > # produce debugging statements on stdout collated in realtime with the > dialog screens > > or > > bsdconfig -D fooLogfile > # produce debugging statements in "fooLogfile" (debugging statements are > hidden from stdout) > > or > > bsdconfig -D +fooLogfile > # produce debugging statements in both "fooLogfile" *and* on stdout (this > is the same as "-dD fooLogfile") > > > As this stuff gets more modular and there are back-ends, front-ends, > global APIs, local APIs, etc. etc. > > Having the ability to (after noticing a problem) flip a switch and then > get an almost exact location of where you currently-are within the code > just by looking at debug statements is extremely helpful in being able to > locate the problem. > > =3D=3D=3D > > The namespace argument is a bit harder to explain (and quantify for > comparison). > > In bsdinstall, we see namespace separation most readily by looking at the > way the C binaries work. > > The namespace separation in this case means that (despite the fact that > the C components do a getenv(3) to acquire $BSDINSTALL_LOG, for example) > for the large part, the C aspects cannot enjoy code written to extend the > sh(1) aspect, and vice-versa. > > So if you want a nice debug library that acts in a single way for > bsdinstall=85 that's going to be difficult (but not impossible=85 you cou= ld > perhaps cheat by having both the sh(1)-side and the C-side use a > logger(1)/syslog(3) facility =85 but then getting that debug information > integrated in the above manner is still non-trivial). > > On the other-hand=85 pc-sysinstall doesn't suffer from the same namespace > problems (it's 100% shell), but it's still not conflated as-is done in > bsdconfig. > > In pc-sysinstall what you'll find is that instead of putting functionalit= y > into actual functions=85 it creates shell scripts that operate in a separ= ate > namespace as they are executed as binaries rather than taking advantage o= f > their shell-based nature (in other words=85 because it's 100% shell, it > should perhaps embrace the ability to avoid forking all the time and run > everything in a conflated [single] namespace). > > =3D=3D=3D > > And as Nathan points out=85 it quickly starts to look like a rewrite of b= oth > to do a merge. > > However=85 the type of "merge" that was being talked about in the above U= RL > (reproduced below from the above reply text for convenience): > > https://wiki.freebsd.org/PCBSDInstallMerge > > Is not a merge that would see a single namespace emerge. > > In the above URL, the type of "merge" that's being posited is the kind > where bsdinstall becomes a front-end only and will call-out to everything > that pc-sysinstall provides. > > This could only be bad if pc-sysinstall is left in its current state, > because pc-sysinstall expects you to treat the largest portions of its > functionality as black-box executables (e.g. you call "delete-part.sh" wi= th > arguments=85 rather than loading a shell library with a delete-part() > function). > > Debugging is harder when you have to cross a namespace threshold. > > Perhaps a better style of merge would be the traditional use of the word= =85 > > That is to say, that pc-sysinstall should be slurped *into* bsdinstall > (bsdinstall being the newer entity on the block=85 so it has more up-to-d= ate > coding style, albeit not the latest; contrasted to pc-sysinstall's dated > inconsistencies within its own code-base -- so a merge in the opposite > direction, of giving pc-sysinstall a user interface, would be harder). > > However (again, with these however statements)=85 > > If the idea is to merge pc-sysinstall and bsdinstall to solve the issue o= f > too many namespaces and the equally-distression problem of not having a > debug layer (which is only marginally helpful if you have a fractured > namespace design)=85 > > =85 I think we could do a lot better. > > Perhaps not better with the out-come (which would be hard to judge before > the product is truly envisaged), but with respect to disruptiveness. > > I recognize that any forward motion in the bsdinstall or pc-sysinstall > camp would be disruptive to the people dependent on those technologies. > > Meanwhile, nobody is depending on bsdconfig at the moment. > > A less (or perhaps, totally non-) disruptive path would be to merge the > two entities (bsdinstall and pc-sysinstall) into a third entity (bsdconfi= g) > where the third entity has much more freedom to play. > > The end result would be that, when bsdconfig does indeed incorporate all > the migrated functionality (and rewritten to achieve the desired > enhancements to make development and maintenance easier), we then -- and > only then -- disrupt things by wholly replacing them. > -- > Devin > > _____________ > The information contained in this message is proprietary and/or > confidential. If you are not the intended recipient, please: (i) delete t= he > message and all copies; (ii) do not disclose, distribute or use the messa= ge > in any manner; and (iii) notify the sender immediately. In addition, plea= se > be aware that any message addressed to our domain is subject to archiving > and review by persons other than the intended recipient. Thank you. > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org= " > From owner-freebsd-hackers@FreeBSD.ORG Tue May 28 23:56:06 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 80F197DD; Tue, 28 May 2013 23:56:06 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 2B4E361E; Tue, 28 May 2013 23:56:05 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.15]) by ltcfislmsgpa07.fnfis.com (8.14.5/8.14.5) with ESMTP id r4SNtwmS026812 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Tue, 28 May 2013 18:55:58 -0500 Received: from LTCFISWMSGMB21.FNFIS.com ([10.132.99.23]) by LTCFISWMSGHT04.FNFIS.com ([10.132.206.15]) with mapi id 14.02.0309.002; Tue, 28 May 2013 18:55:59 -0500 From: "Teske, Devin" To: Super Bisquit Subject: Re: FreeBSD installers and future direction Thread-Topic: FreeBSD installers and future direction Thread-Index: AQHOWV60z6k72swSkkmGANUzy/EMLJkWZ1wAgAAkfQCAAE6UgIAAH7eAgAAeuACAAA8cgIACXGSAgAAS94CAABK3AIAACk6AgAAeVACAAA8kAIAAR84AgAAD5oCAAC1WgIAAq02AgAASJICAABbCAIAAZMiAgAALCoA= Date: Tue, 28 May 2013 23:55:58 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F65269@ltcfiswmsgmb21> References: <51A0DC3F.9030301@cran.org.uk> <51A1025A.2020607@cran.org.uk> <51A14445.4060305@freebsd.org> <51A15EDF.6050600@erdgeist.org> <13CA24D6AB415D428143D44749F57D7201F5B337@ltcfiswmsgmb26> <51A38051.8040909@mu.org> <51A39039.1070202@cran.org.uk> <51A39FEC.5070402@mu.org> <51A3A891.5060103@cran.org.uk> <51A3C202.9030802@mu.org> <51A3CEB6.3070200@cran.org.uk> <51A40AF2.2010108@mu.org> <51A40E37.9060702@freebsd.org> <51A4343F.3070605@mu.org> <51A4C3F1.2010604@freebsd.org> <51A4D329.5060103@mu.org> <13CA24D6AB415D428143D44749F57D7201F6484E@ltcfiswmsgmb21> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-28_10:2013-05-28,2013-05-28,1970-01-01 signatures=0 Content-Type: text/plain; charset="Windows-1252" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Bruce Cran , FreeBSD Hackers , Devin Teske , Alfred Perlstein , Nathan Whitehorn X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 May 2013 23:56:06 -0000 On May 28, 2013, at 4:16 PM, Super Bisquit wrote: In the case of firmware loaded systems, all of them aren't going to work wi= th a single boot loader. Uh=85 On the surface, what you're talking about seems to be unrelated to the disc= ussion at-hand. Nobody said anything about unifying the boot loader. We're talking about the installer=85 not the boot loader. *usually* the installer doesn't care about how it got to its installation e= nvironment. NOTE: I say *usually* because there *are* specialized situations like cobbl= er + mfsbsd + pc-sysinstall in which case the boot loader needs to provide = certain information (which is then accessible through kenv(1)). That type o= f installation setup will fail if the specialized boot loader does not boot= on the desired target installation platform *or* the boot loader that _doe= s_ work on said platform doesn't provide the necessary information. That being said=85 bsdinstall doesn't require special knowledge from the bo= ot loader (to the best of my knowledge). sysinstall didn't need special info from the boot loader either. I don't see why talking about a unified installer has to include a discussi= on involving unifying the boot loader (a topic in my mind that is unattaina= ble). -- Devin On Tue, May 28, 2013 at 1:15 PM, Teske, Devin > wrote: On May 28, 2013, at 8:54 AM, Alfred Perlstein wrote: On 5/28/13 7:49 AM, Nathan Whitehorn wrote: On 05/27/13 23:36, Alfred Perlstein wrote: On 5/27/13 6:53 PM, Nathan Whitehorn wrote: On 05/27/13 20:40, Alfred Perlstein wrote: On 5/27/13 2:23 PM, Bruce Cran wrote: On 27/05/2013 21:28, Alfred Perlstein wrote: On 5/27/13 11:40 AM, Bruce Cran wrote: Yes. Is this a joke? It probably /was/ too short a reply. Personally I think there should be a s= ingle UI and scripting interface across all platforms. We should try and ge= t pc-sysinstall running on all of them first in case there's some problem t= hat means it can't be done, in which case we'd need to use a different back= end. There are just going to be certain platforms that make it EASY to do cool t= hings. We should embrace that! That's why there are different platforms! Some are great for low power, others are great for graphics, cpu power, gpu= , networking etc. If we always go for the lowest common denominator then we are crippling all= the platforms for no one's benefit. Even if something CAN be done, if it = is very difficult, or just never happening, then we can't limit everyone's = experience based on the more difficult and/or resource strapped platforms. It's just not good business. Yes, and all of this cuts both ways: pc-sysinstall has no wireless setup su= pport, for instance. Right now we support what we support because it is the= most feature-complete thing we have, not just on tier-2 platforms but also= on x86. To bring this discussion back to the ground, the fact is that we lack an in= staller that has both internal support for ZFS and a UI. One of the reasons= for this is that making a good expressive UI for ZFS is a non-trivial unde= rtaking given its enormous flexibility. The bsdinstall partition editor has= been written to be extensible for this, and several people have started wr= iting code to do it, but no one ended up having time to finish. Probably a = reasonable thing to do is to start with supporting only a minimal set of fe= atures. If anyone felt like actually writing this code, I'm sure it would b= e appreciated by all and be more productive than email exchanges. -Nathan I'm sure if there was a list of reasonable things, such as wireless then pc= -sysinstall could be augmented. This is the first I've heard of that. All= the other complaints have been based on portability. Is that all that is required now, wireless? There are more, as well. A partial list of missing features on both sides i= s here: https://wiki.freebsd.org/PCBSDInstallMerge. Other major ones are = IPv6 (maybe this has changed?) and no jail setup feature. Most of the exist= ing disk partitioning code in pc-sysinstall, which is the only thing in a F= reeBSD installer that is at all complicated, is also *extremely* fragile an= d needs in all likelihood to be entirely replaced. The merge effort stalled= because of this kind of issue -- doing a "merge" rapidly became rewriting = both from scratch. -Nathan Ah this is so cool. I'll bring it up with the PCBSD folks today. Thank you Nathan. I had my own look at the pc-sysinstall and bsdinstall code and came to the = same conclusions, plus some. One of the biggest obstacles I see is actually a high-level issue that I've= self-identified through extensive work on bsdconfig (which is both a back-= end and a front-end). This is the issue of debugging and namespaces. I've sat down and made lists of other issues=85 but when I review, I find t= hese issues to be secondary to the above-stated larger issues. Concretely, I'm saying thus: + bsdinstall lacks debugging (debugging is different than logging; from wha= t I could see BSDINSTALL_LOG -- although utilized by both the sh(1) side an= d the C side -- is only populated during an installation). The ability to h= ave the type of debugging that is in bsdconfig would greatly diminish the a= mount of time developing important new features. + pc-sysinstall lacks debugging (similar situation=85 producing a log for s= ome action is not the same as being able to have debug statements for the p= urpose of enhancement the program or troubleshooting an enhancement) + bsdinstall separates the backend functionality and the front-end function= ality into two separate namespaces (and in the case of C binaries, a third = namespace) + pc-sysinstall separates one backend into more than one namespace =3D=3D=3D To get an idea of the type of debugging I'm talking about, install sysutils= /bsdconfig from the ports tree or install it from a HEAD checkout of base (= it's in usr.sbin) and execute: bsdconfig -d # produce debugging statements on stdout collated in realtime with the dial= og screens or bsdconfig -D fooLogfile # produce debugging statements in "fooLogfile" (debugging statements are hi= dden from stdout) or bsdconfig -D +fooLogfile # produce debugging statements in both "fooLogfile" *and* on stdout (this i= s the same as "-dD fooLogfile") As this stuff gets more modular and there are back-ends, front-ends, global= APIs, local APIs, etc. etc. Having the ability to (after noticing a problem) flip a switch and then get= an almost exact location of where you currently-are within the code just b= y looking at debug statements is extremely helpful in being able to locate = the problem. =3D=3D=3D The namespace argument is a bit harder to explain (and quantify for compari= son). In bsdinstall, we see namespace separation most readily by looking at the w= ay the C binaries work. The namespace separation in this case means that (despite the fact that the= C components do a getenv(3) to acquire $BSDINSTALL_LOG, for example) for t= he large part, the C aspects cannot enjoy code written to extend the sh(1) = aspect, and vice-versa. So if you want a nice debug library that acts in a single way for bsdinstal= l=85 that's going to be difficult (but not impossible=85 you could perhaps = cheat by having both the sh(1)-side and the C-side use a logger(1)/syslog(3= ) facility =85 but then getting that debug information integrated in the ab= ove manner is still non-trivial). On the other-hand=85 pc-sysinstall doesn't suffer from the same namespace p= roblems (it's 100% shell), but it's still not conflated as-is done in bsdco= nfig. In pc-sysinstall what you'll find is that instead of putting functionality = into actual functions=85 it creates shell scripts that operate in a separat= e namespace as they are executed as binaries rather than taking advantage o= f their shell-based nature (in other words=85 because it's 100% shell, it s= hould perhaps embrace the ability to avoid forking all the time and run eve= rything in a conflated [single] namespace). =3D=3D=3D And as Nathan points out=85 it quickly starts to look like a rewrite of bot= h to do a merge. However=85 the type of "merge" that was being talked about in the above URL= (reproduced below from the above reply text for convenience): https://wiki.freebsd.org/PCBSDInstallMerge Is not a merge that would see a single namespace emerge. In the above URL, the type of "merge" that's being posited is the kind wher= e bsdinstall becomes a front-end only and will call-out to everything that = pc-sysinstall provides. This could only be bad if pc-sysinstall is left in its current state, becau= se pc-sysinstall expects you to treat the largest portions of its functiona= lity as black-box executables (e.g. you call "delete-part.sh" with argument= s=85 rather than loading a shell library with a delete-part() function). Debugging is harder when you have to cross a namespace threshold. Perhaps a better style of merge would be the traditional use of the word=85 That is to say, that pc-sysinstall should be slurped *into* bsdinstall (bsd= install being the newer entity on the block=85 so it has more up-to-date co= ding style, albeit not the latest; contrasted to pc-sysinstall's dated inco= nsistencies within its own code-base -- so a merge in the opposite directio= n, of giving pc-sysinstall a user interface, would be harder). However (again, with these however statements)=85 If the idea is to merge pc-sysinstall and bsdinstall to solve the issue of = too many namespaces and the equally-distression problem of not having a deb= ug layer (which is only marginally helpful if you have a fractured namespac= e design)=85 =85 I think we could do a lot better. Perhaps not better with the out-come (which would be hard to judge before t= he product is truly envisaged), but with respect to disruptiveness. I recognize that any forward motion in the bsdinstall or pc-sysinstall camp= would be disruptive to the people dependent on those technologies. Meanwhile, nobody is depending on bsdconfig at the moment. A less (or perhaps, totally non-) disruptive path would be to merge the two= entities (bsdinstall and pc-sysinstall) into a third entity (bsdconfig) wh= ere the third entity has much more freedom to play. The end result would be that, when bsdconfig does indeed incorporate all th= e migrated functionality (and rewritten to achieve the desired enhancements= to make development and maintenance easier), we then -- and only then -- d= isrupt things by wholly replacing them. -- Devin _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. _______________________________________________ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-hackers@FreeBSD.ORG Wed May 29 05:08:59 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id F15B8575; Wed, 29 May 2013 05:08:58 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id 6770299E; Wed, 29 May 2013 05:08:58 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r4T58Z8A062005; Wed, 29 May 2013 08:08:35 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r4T58Z8A062005 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r4T58ZtY062004; Wed, 29 May 2013 08:08:35 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 29 May 2013 08:08:35 +0300 From: Konstantin Belousov To: Alfred Perlstein Subject: Re: please review, patch for lost camisr Message-ID: <20130529050835.GP3047@kib.kiev.ua> References: <51A44B0C.8010908@ixsystems.com> <201305281204.14146.jhb@freebsd.org> <51A505A5.7030105@ixsystems.com> <201305281613.32414.jhb@freebsd.org> <51A514F5.9050405@ixsystems.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="0b/DLdpkOOCEj3jc" Content-Disposition: inline In-Reply-To: <51A514F5.9050405@ixsystems.com> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: Alexander Motin , hackers@freebsd.org, Xin Li X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 May 2013 05:08:59 -0000 --0b/DLdpkOOCEj3jc Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 28, 2013 at 01:35:01PM -0700, Alfred Perlstein wrote: > [[ moved to hackers@ from private mail. ]] >=20 > On 5/28/13 1:13 PM, John Baldwin wrote: > > On Tuesday, May 28, 2013 3:29:41 pm Alfred Perlstein wrote: > >> On 5/28/13 9:04 AM, John Baldwin wrote: > >>> On Tuesday, May 28, 2013 2:13:32 am Alfred Perlstein wrote: > >>>> Hey folks, > >>>> > >>>> I had a talk with Nathan Whitehorn about the camisr issue. The issu= e we > >>>> are seeing we mostly know, but to summarize, we are losing the camisr > >>>> signal and the camisr is not being run. > >>>> > >>>> I gave him a summary of what we have been seeing and pointed him to = the > >>>> code I am concerned about here: > >>>> http://pastebin.com/tLKr7mCV (this is inside of kern_intr.c). > >>>> > >>>> What I think that is happening is that the setting of it_need to 0 > >>>> inside of sys/kern/kern_intr.c:ithread_loop() is not being scheduled > >>>> correctly and it is being delayed until AFTER the call to > >>>> ithread_execute_handlers() right below the atomic_store_rel_int(). > >>> This seems highly unlikely, to the extent that if this were true all = our > >>> locking primitives would be broken. The store_rel is actually a rele= ase > >>> barrier which acts like more a read/write fence. No memory accesses = (read or > >>> write) from before the release can be scheduled after the associated = store, > >>> either by the compiler or CPU. That is what Konstantin is referring = to in his > >>> commit when he says "release" semantics". > >> Yes, that makes sense, however does it specify that the writes *must* > >> occur at that *point*? If it only enforces ordering then we may have > >> some issue, specifically because the setting of it to '1' inside of > >> intr_event_schedule_thread has no barrier other than the acq semantics > >> of the thread lock. I am wondering what is forcing out the '1' there. > > Nothing ever forces writes. You would have to invalidate the cache to = do that > > and that is horribly expensive. It is always only about ordering and k= nowing > > that if you can complete another operation on the same "cookie" variabl= e with > > acquire semantics that earlier writes will be visible. >=20 > By cookie, you mean a specific memory address, basically a lock? This is= =20 > starting to reinforce my suspicions as the setting of it_need is done=20 > with release semantics, however the acq on the other CPU is done on the= =20 > thread lock. Maybe that is irrelevant. We will find out shortly. >=20 > > > >> See below as I think we have proof that this is somehow happening. > > Having ih_need of 1 and it_need of 0 is certainly busted. The simplest= fix > > is probably to stop using atomics on it_need and just grab the thread l= ock > > in the main ithread loop and hold it while checking and clearing it_nee= d. > > >=20 > OK, we have some code that will either prove this, or perturb the memory= =20 > ordering enough to make the bug go away, or prove this assertion wrong. >=20 > We will update on our findings hopefully in the next few days. IMO the read of it_need in the 'while (ithd->it_need)' should have acquire semantic, otherwise the future reads in the ithread_execute_handlers(), in particular, of the ih_need, could pass the read of it_need and cause the situation you reported. I do not see any acquire barrier between a condition in the while() statement and the read of ih_need in the execute_handlers(). It is probably true that the issue you see was caused by r236456, in the sense that implicitely locked xchgl instruction on x86 has a full barrier semantic. As result, the store_rel() was actually an acquire too, making this reordering impossible. I argue that this is not a bug in r236456, but the issue in the kern_intr.c. On the other hand, the John' suggestion to move the manipulations of it_need under the lock is probably the best anyway. --0b/DLdpkOOCEj3jc Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRpY1SAAoJEJDCuSvBvK1BnGYP/jEd2YKjWUCNIAhpA9U2B0Jq N1ckhmCnmp+NI4ierhsosT0+COAclCrpYFgrq6sEIz0HdU+CmYzjoh04fbN1OOph XY57D6hehssszwGW7NTUn9dMtt0Wiy47D10i9SSF2Gtyj38K+hEZm2y0Pqr1MEOy jMrE6Rfm5F4h2W0Lt3oJZCu8MtgWK0mTy3l6jre1f9uNCJt5ovleO0e6/PvwfIir LRTdBre3ETK3mCz04mAuHqFNb7YYpYiUdE5QYPFricXwXEdcmYQFVlGctUkI36Ue 2sjii/KmCwoXGDIf6+xY0fDWsd7cBsrK0PGWJGhagYIW4eazoBX5mvNY7pq9Md33 cTzVe1KDdtGTOAFBKTBpBlIUkdF4N23sxYO17ZgLXntLCGUIpA40h1BZF7S7BYQ+ hnXu0qCotxVsGDwQVaRaz544L1mRPUwSL8UAS3HuXkSqJ/nbCN6G7NjJ10fXyT+4 hUmXuPsnN5R5dM2yLDywf1Hy3UUBFudUPKk2Bq29fGbXxqTEMr0zMvTY/fGjFQo2 dJGGGD0q2XQnIcrDWpxQKh5z68+hyekocPTwMuoNCoMA+rz4VsDDRIZP26DElRMV 5JZDTalPNgPCDOxR6yymrus1P2BG4TEFl3dhuAaMvWMekBqdURXrSusaqTy3nMfq YcTIVY3XLgUUcaFO+iKf =XR5q -----END PGP SIGNATURE----- --0b/DLdpkOOCEj3jc-- From owner-freebsd-hackers@FreeBSD.ORG Wed May 29 05:31:44 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D6BCB3F7 for ; Wed, 29 May 2013 05:31:44 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id C719BA83 for ; Wed, 29 May 2013 05:31:44 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id 2D51D1A3C1E for ; Tue, 28 May 2013 22:31:41 -0700 (PDT) Message-ID: <51A592BC.1050100@mu.org> Date: Tue, 28 May 2013 22:31:40 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: please review, patch for lost camisr References: <51A44B0C.8010908@ixsystems.com> <201305281204.14146.jhb@freebsd.org> <51A505A5.7030105@ixsystems.com> <201305281613.32414.jhb@freebsd.org> <51A514F5.9050405@ixsystems.com> <20130529050835.GP3047@kib.kiev.ua> In-Reply-To: <20130529050835.GP3047@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 May 2013 05:31:44 -0000 On 5/28/13 10:08 PM, Konstantin Belousov wrote: > On Tue, May 28, 2013 at 01:35:01PM -0700, Alfred Perlstein wrote: >> [[ moved to hackers@ from private mail. ]] >> >> On 5/28/13 1:13 PM, John Baldwin wrote: >>> On Tuesday, May 28, 2013 3:29:41 pm Alfred Perlstein wrote: >>>> On 5/28/13 9:04 AM, John Baldwin wrote: >>>>> On Tuesday, May 28, 2013 2:13:32 am Alfred Perlstein wrote: >>>>>> Hey folks, >>>>>> >>>>>> I had a talk with Nathan Whitehorn about the camisr issue. The issue we >>>>>> are seeing we mostly know, but to summarize, we are losing the camisr >>>>>> signal and the camisr is not being run. >>>>>> >>>>>> I gave him a summary of what we have been seeing and pointed him to the >>>>>> code I am concerned about here: >>>>>> http://pastebin.com/tLKr7mCV (this is inside of kern_intr.c). >>>>>> >>>>>> What I think that is happening is that the setting of it_need to 0 >>>>>> inside of sys/kern/kern_intr.c:ithread_loop() is not being scheduled >>>>>> correctly and it is being delayed until AFTER the call to >>>>>> ithread_execute_handlers() right below the atomic_store_rel_int(). >>>>> This seems highly unlikely, to the extent that if this were true all our >>>>> locking primitives would be broken. The store_rel is actually a release >>>>> barrier which acts like more a read/write fence. No memory accesses (read or >>>>> write) from before the release can be scheduled after the associated store, >>>>> either by the compiler or CPU. That is what Konstantin is referring to in his >>>>> commit when he says "release" semantics". >>>> Yes, that makes sense, however does it specify that the writes *must* >>>> occur at that *point*? If it only enforces ordering then we may have >>>> some issue, specifically because the setting of it to '1' inside of >>>> intr_event_schedule_thread has no barrier other than the acq semantics >>>> of the thread lock. I am wondering what is forcing out the '1' there. >>> Nothing ever forces writes. You would have to invalidate the cache to do that >>> and that is horribly expensive. It is always only about ordering and knowing >>> that if you can complete another operation on the same "cookie" variable with >>> acquire semantics that earlier writes will be visible. >> By cookie, you mean a specific memory address, basically a lock? This is >> starting to reinforce my suspicions as the setting of it_need is done >> with release semantics, however the acq on the other CPU is done on the >> thread lock. Maybe that is irrelevant. We will find out shortly. >> >>>> See below as I think we have proof that this is somehow happening. >>> Having ih_need of 1 and it_need of 0 is certainly busted. The simplest fix >>> is probably to stop using atomics on it_need and just grab the thread lock >>> in the main ithread loop and hold it while checking and clearing it_need. >>> >> OK, we have some code that will either prove this, or perturb the memory >> ordering enough to make the bug go away, or prove this assertion wrong. >> >> We will update on our findings hopefully in the next few days. > IMO the read of it_need in the 'while (ithd->it_need)' should > have acquire semantic, otherwise the future reads in the > ithread_execute_handlers(), in particular, of the ih_need, could pass > the read of it_need and cause the situation you reported. I do not > see any acquire barrier between a condition in the while() statement > and the read of ih_need in the execute_handlers(). > > It is probably true that the issue you see was caused by r236456, in the > sense that implicitely locked xchgl instruction on x86 has a full barrier > semantic. As result, the store_rel() was actually an acquire too, making > this reordering impossible. I argue that this is not a bug in r236456, > but the issue in the kern_intr.c. If I remember the code correctly that would probably explain why we see it only on 9.1 system. > > On the other hand, the John' suggestion to move the manipulations of > it_need under the lock is probably the best anyway. > I was wondering if it would be lower latency to maintain it_need, however to keep another variable it_needlocked under the thread lock. This would result in potential superfluous interrupts, however under load you would allow the ithread to loop without taking the thread lock some number of times. I am not really sure if this is really worth the optimization (especially since it can result in superfluous interrupts) however it may reduce latency and that might be important. Is there some people that I can pass the patch onto for help with performance once we confirm that this is the actual bug? We can do internal testing, but I am worried about regressing performance of any form of IO for the kernel. I'll show the patch soon. Thank you for the information. This is promising. -Alfred From owner-freebsd-hackers@FreeBSD.ORG Wed May 29 07:16:55 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6ED92CBB for ; Wed, 29 May 2013 07:16:55 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id BFA4681 for ; Wed, 29 May 2013 07:16:54 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r4T7GjPO088490; Wed, 29 May 2013 10:16:45 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r4T7GjPO088490 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r4T7Gj2s088489; Wed, 29 May 2013 10:16:45 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 29 May 2013 10:16:45 +0300 From: Konstantin Belousov To: Alfred Perlstein Subject: Re: please review, patch for lost camisr Message-ID: <20130529071645.GR3047@kib.kiev.ua> References: <51A44B0C.8010908@ixsystems.com> <201305281204.14146.jhb@freebsd.org> <51A505A5.7030105@ixsystems.com> <201305281613.32414.jhb@freebsd.org> <51A514F5.9050405@ixsystems.com> <20130529050835.GP3047@kib.kiev.ua> <51A592BC.1050100@mu.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Kp2K9UuanVOCcgkO" Content-Disposition: inline In-Reply-To: <51A592BC.1050100@mu.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 May 2013 07:16:55 -0000 --Kp2K9UuanVOCcgkO Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, May 28, 2013 at 10:31:40PM -0700, Alfred Perlstein wrote: > On 5/28/13 10:08 PM, Konstantin Belousov wrote: > > On Tue, May 28, 2013 at 01:35:01PM -0700, Alfred Perlstein wrote: > >> [[ moved to hackers@ from private mail. ]] > >> > >> On 5/28/13 1:13 PM, John Baldwin wrote: > >>> On Tuesday, May 28, 2013 3:29:41 pm Alfred Perlstein wrote: > >>>> On 5/28/13 9:04 AM, John Baldwin wrote: > >>>>> On Tuesday, May 28, 2013 2:13:32 am Alfred Perlstein wrote: > >>>>>> Hey folks, > >>>>>> > >>>>>> I had a talk with Nathan Whitehorn about the camisr issue. The is= sue we > >>>>>> are seeing we mostly know, but to summarize, we are losing the cam= isr > >>>>>> signal and the camisr is not being run. > >>>>>> > >>>>>> I gave him a summary of what we have been seeing and pointed him t= o the > >>>>>> code I am concerned about here: > >>>>>> http://pastebin.com/tLKr7mCV (this is inside of kern_intr.c). > >>>>>> > >>>>>> What I think that is happening is that the setting of it_need to 0 > >>>>>> inside of sys/kern/kern_intr.c:ithread_loop() is not being schedul= ed > >>>>>> correctly and it is being delayed until AFTER the call to > >>>>>> ithread_execute_handlers() right below the atomic_store_rel_int(). > >>>>> This seems highly unlikely, to the extent that if this were true al= l our > >>>>> locking primitives would be broken. The store_rel is actually a re= lease > >>>>> barrier which acts like more a read/write fence. No memory accesse= s (read or > >>>>> write) from before the release can be scheduled after the associate= d store, > >>>>> either by the compiler or CPU. That is what Konstantin is referrin= g to in his > >>>>> commit when he says "release" semantics". > >>>> Yes, that makes sense, however does it specify that the writes *must* > >>>> occur at that *point*? If it only enforces ordering then we may have > >>>> some issue, specifically because the setting of it to '1' inside of > >>>> intr_event_schedule_thread has no barrier other than the acq semanti= cs > >>>> of the thread lock. I am wondering what is forcing out the '1' ther= e. > >>> Nothing ever forces writes. You would have to invalidate the cache t= o do that > >>> and that is horribly expensive. It is always only about ordering and= knowing > >>> that if you can complete another operation on the same "cookie" varia= ble with > >>> acquire semantics that earlier writes will be visible. > >> By cookie, you mean a specific memory address, basically a lock? This = is > >> starting to reinforce my suspicions as the setting of it_need is done > >> with release semantics, however the acq on the other CPU is done on the > >> thread lock. Maybe that is irrelevant. We will find out shortly. > >> > >>>> See below as I think we have proof that this is somehow happening. > >>> Having ih_need of 1 and it_need of 0 is certainly busted. The simple= st fix > >>> is probably to stop using atomics on it_need and just grab the thread= lock > >>> in the main ithread loop and hold it while checking and clearing it_n= eed. > >>> > >> OK, we have some code that will either prove this, or perturb the memo= ry > >> ordering enough to make the bug go away, or prove this assertion wrong. > >> > >> We will update on our findings hopefully in the next few days. > > IMO the read of it_need in the 'while (ithd->it_need)' should > > have acquire semantic, otherwise the future reads in the > > ithread_execute_handlers(), in particular, of the ih_need, could pass > > the read of it_need and cause the situation you reported. I do not > > see any acquire barrier between a condition in the while() statement > > and the read of ih_need in the execute_handlers(). > > > > It is probably true that the issue you see was caused by r236456, in the > > sense that implicitely locked xchgl instruction on x86 has a full barri= er > > semantic. As result, the store_rel() was actually an acquire too, maki= ng > > this reordering impossible. I argue that this is not a bug in r236456, > > but the issue in the kern_intr.c. > If I remember the code correctly that would probably explain why we see= =20 > it only on 9.1 system. > > > > On the other hand, the John' suggestion to move the manipulations of > > it_need under the lock is probably the best anyway. > > > I was wondering if it would be lower latency to maintain it_need,=20 > however to keep another variable it_needlocked under the thread lock. =20 > This would result in potential superfluous interrupts, however under=20 > load you would allow the ithread to loop without taking the thread lock= =20 > some number of times. >=20 > I am not really sure if this is really worth the optimization=20 > (especially since it can result in superfluous interrupts) however it=20 > may reduce latency and that might be important. >=20 > Is there some people that I can pass the patch onto for help with=20 > performance once we confirm that this is the actual bug? We can do=20 > internal testing, but I am worried about regressing performance of any=20 > form of IO for the kernel. >=20 > I'll show the patch soon. >=20 > Thank you for the information. This is promising. Well, if you and I are right, the minimal patch should be diff --git a/sys/kern/kern_intr.c b/sys/kern/kern_intr.c index 8d63c9b..7c21015 100644 --- a/sys/kern/kern_intr.c +++ b/sys/kern/kern_intr.c @@ -1349,7 +1349,7 @@ ithread_loop(void *arg) * we are running, it will set it_need to note that we * should make another pass. */ - while (ithd->it_need) { + while (atomic_load_acq_int(&ithd->it_need)) { /* * This might need a full read and write barrier * to make sure that this write posts before any --Kp2K9UuanVOCcgkO Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRpatcAAoJEJDCuSvBvK1BB84P/0yJ0u/G3Ddiw2Psk4UmIW1W oDFF3Z3qDW25qQxyH7ER/35EfayVdugBvR4BwNt//V//nNlEXcS56d+qAG1Fxgd/ cCV/QSxBjb7sLka93FX8Ku5wvwKbeGbunBOG8DSVDL34+MJy+tyriw9LudYmegVf 4oT/TjtsJDAiXfl+vS1EhEoQy49QCQwThc2CA5DalrydF3xJc5RJRitR8f9yfjf5 fhPPm24g0wHYbG2ctUGl1oK7xnJUVVQT0MlEfMoB4F/tamXIeS/hP09dLJuWQK3j wqzxD6KndKtLi5m/JSbpuxdE0nWpdewpLe66VEDW7bcxf98qZ/qzp3bjImYBE6dC SYgMtgTwZfBpV7LuyPlR7lQo2Rc0XKKdJwCtdF0MSSBy8kzLK+WQsiCLJjJth1y/ k2mm+SndZgNKXjsYjUITxm1YZx6obVNiyiwUwoRilwizkMpdFyyO9FvGRBywDGQz bQOjptIEvOzcUVHSFgSWon7+6xbaV7JEDe1ja8ApeLki3Cm699HZia4ZmlC5dNxP OOfHp0CY9lCIdK/ib3I6yPY2g3Q8RWJRtBbhkU2ChxXzB4WiuAyumgWjWmYhpX3t /PdWUXk3IGdmvp9w+vt8HzIqkl7RIGh3T3f3YKWypRhjB/00wmY1UKIGISzTUQc1 5jvmi4ZCGmTwuFhEbp5X =/X81 -----END PGP SIGNATURE----- --Kp2K9UuanVOCcgkO-- From owner-freebsd-hackers@FreeBSD.ORG Wed May 29 07:27:48 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 1780470 for ; Wed, 29 May 2013 07:27:48 +0000 (UTC) (envelope-from bright@mu.org) Received: from elvis.mu.org (elvis.mu.org [192.203.228.196]) by mx1.freebsd.org (Postfix) with ESMTP id EC161F3 for ; Wed, 29 May 2013 07:27:47 +0000 (UTC) Received: from Alfreds-MacBook-Pro-9.local (c-67-180-208-218.hsd1.ca.comcast.net [67.180.208.218]) by elvis.mu.org (Postfix) with ESMTPSA id 71CB41A3C1D for ; Wed, 29 May 2013 00:27:46 -0700 (PDT) Message-ID: <51A5ADF2.3070004@mu.org> Date: Wed, 29 May 2013 00:27:46 -0700 From: Alfred Perlstein User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-hackers@freebsd.org Subject: Re: please review, patch for lost camisr References: <51A44B0C.8010908@ixsystems.com> <201305281204.14146.jhb@freebsd.org> <51A505A5.7030105@ixsystems.com> <201305281613.32414.jhb@freebsd.org> <51A514F5.9050405@ixsystems.com> <20130529050835.GP3047@kib.kiev.ua> <51A592BC.1050100@mu.org> <20130529071645.GR3047@kib.kiev.ua> In-Reply-To: <20130529071645.GR3047@kib.kiev.ua> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 May 2013 07:27:48 -0000 On 5/29/13 12:16 AM, Konstantin Belousov wrote: > On Tue, May 28, 2013 at 10:31:40PM -0700, Alfred Perlstein wrote: >> On 5/28/13 10:08 PM, Konstantin Belousov wrote: >>> On Tue, May 28, 2013 at 01:35:01PM -0700, Alfred Perlstein wrote: >>>> [[ moved to hackers@ from private mail. ]] >>>> >>>> On 5/28/13 1:13 PM, John Baldwin wrote: >>>>> On Tuesday, May 28, 2013 3:29:41 pm Alfred Perlstein wrote: >>>>>> On 5/28/13 9:04 AM, John Baldwin wrote: >>>>>>> On Tuesday, May 28, 2013 2:13:32 am Alfred Perlstein wrote: >>>>>>>> Hey folks, >>>>>>>> >>>>>>>> I had a talk with Nathan Whitehorn about the camisr issue. The issue we >>>>>>>> are seeing we mostly know, but to summarize, we are losing the camisr >>>>>>>> signal and the camisr is not being run. >>>>>>>> >>>>>>>> I gave him a summary of what we have been seeing and pointed him to the >>>>>>>> code I am concerned about here: >>>>>>>> http://pastebin.com/tLKr7mCV (this is inside of kern_intr.c). >>>>>>>> >>>>>>>> What I think that is happening is that the setting of it_need to 0 >>>>>>>> inside of sys/kern/kern_intr.c:ithread_loop() is not being scheduled >>>>>>>> correctly and it is being delayed until AFTER the call to >>>>>>>> ithread_execute_handlers() right below the atomic_store_rel_int(). >>>>>>> This seems highly unlikely, to the extent that if this were true all our >>>>>>> locking primitives would be broken. The store_rel is actually a release >>>>>>> barrier which acts like more a read/write fence. No memory accesses (read or >>>>>>> write) from before the release can be scheduled after the associated store, >>>>>>> either by the compiler or CPU. That is what Konstantin is referring to in his >>>>>>> commit when he says "release" semantics". >>>>>> Yes, that makes sense, however does it specify that the writes *must* >>>>>> occur at that *point*? If it only enforces ordering then we may have >>>>>> some issue, specifically because the setting of it to '1' inside of >>>>>> intr_event_schedule_thread has no barrier other than the acq semantics >>>>>> of the thread lock. I am wondering what is forcing out the '1' there. >>>>> Nothing ever forces writes. You would have to invalidate the cache to do that >>>>> and that is horribly expensive. It is always only about ordering and knowing >>>>> that if you can complete another operation on the same "cookie" variable with >>>>> acquire semantics that earlier writes will be visible. >>>> By cookie, you mean a specific memory address, basically a lock? This is >>>> starting to reinforce my suspicions as the setting of it_need is done >>>> with release semantics, however the acq on the other CPU is done on the >>>> thread lock. Maybe that is irrelevant. We will find out shortly. >>>> >>>>>> See below as I think we have proof that this is somehow happening. >>>>> Having ih_need of 1 and it_need of 0 is certainly busted. The simplest fix >>>>> is probably to stop using atomics on it_need and just grab the thread lock >>>>> in the main ithread loop and hold it while checking and clearing it_need. >>>>> >>>> OK, we have some code that will either prove this, or perturb the memory >>>> ordering enough to make the bug go away, or prove this assertion wrong. >>>> >>>> We will update on our findings hopefully in the next few days. >>> IMO the read of it_need in the 'while (ithd->it_need)' should >>> have acquire semantic, otherwise the future reads in the >>> ithread_execute_handlers(), in particular, of the ih_need, could pass >>> the read of it_need and cause the situation you reported. I do not >>> see any acquire barrier between a condition in the while() statement >>> and the read of ih_need in the execute_handlers(). >>> >>> It is probably true that the issue you see was caused by r236456, in the >>> sense that implicitely locked xchgl instruction on x86 has a full barrier >>> semantic. As result, the store_rel() was actually an acquire too, making >>> this reordering impossible. I argue that this is not a bug in r236456, >>> but the issue in the kern_intr.c. >> If I remember the code correctly that would probably explain why we see >> it only on 9.1 system. >>> On the other hand, the John' suggestion to move the manipulations of >>> it_need under the lock is probably the best anyway. >>> >> I was wondering if it would be lower latency to maintain it_need, >> however to keep another variable it_needlocked under the thread lock. >> This would result in potential superfluous interrupts, however under >> load you would allow the ithread to loop without taking the thread lock >> some number of times. >> >> I am not really sure if this is really worth the optimization >> (especially since it can result in superfluous interrupts) however it >> may reduce latency and that might be important. >> >> Is there some people that I can pass the patch onto for help with >> performance once we confirm that this is the actual bug? We can do >> internal testing, but I am worried about regressing performance of any >> form of IO for the kernel. >> >> I'll show the patch soon. >> >> Thank you for the information. This is promising. > Well, if you and I are right, the minimal patch should be > > diff --git a/sys/kern/kern_intr.c b/sys/kern/kern_intr.c > index 8d63c9b..7c21015 100644 > --- a/sys/kern/kern_intr.c > +++ b/sys/kern/kern_intr.c > @@ -1349,7 +1349,7 @@ ithread_loop(void *arg) > * we are running, it will set it_need to note that we > * should make another pass. > */ > - while (ithd->it_need) { > + while (atomic_load_acq_int(&ithd->it_need)) { > /* > * This might need a full read and write barrier > * to make sure that this write posts before any > OK we can try this. I've been pretty good at locking when using mutexes, but when we get into the atomic ops like this it gets a little tough for me to follow without extensive research. I know that the signalling thread (swi_sched caller) does not use any atomic ops... is this OK? -Alfred From owner-freebsd-hackers@FreeBSD.ORG Wed May 29 07:40:43 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 8DD1537E for ; Wed, 29 May 2013 07:40:43 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id BCF1217E for ; Wed, 29 May 2013 07:40:42 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r4T7eXfN093450; Wed, 29 May 2013 10:40:33 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r4T7eXfN093450 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r4T7eXUi093449; Wed, 29 May 2013 10:40:33 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 29 May 2013 10:40:33 +0300 From: Konstantin Belousov To: Alfred Perlstein Subject: Re: please review, patch for lost camisr Message-ID: <20130529074033.GS3047@kib.kiev.ua> References: <51A44B0C.8010908@ixsystems.com> <201305281204.14146.jhb@freebsd.org> <51A505A5.7030105@ixsystems.com> <201305281613.32414.jhb@freebsd.org> <51A514F5.9050405@ixsystems.com> <20130529050835.GP3047@kib.kiev.ua> <51A592BC.1050100@mu.org> <20130529071645.GR3047@kib.kiev.ua> <51A5ADF2.3070004@mu.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="aN4sSdpAXbmVieFt" Content-Disposition: inline In-Reply-To: <51A5ADF2.3070004@mu.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 May 2013 07:40:43 -0000 --aN4sSdpAXbmVieFt Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, May 29, 2013 at 12:27:46AM -0700, Alfred Perlstein wrote: > On 5/29/13 12:16 AM, Konstantin Belousov wrote: > > On Tue, May 28, 2013 at 10:31:40PM -0700, Alfred Perlstein wrote: > >> On 5/28/13 10:08 PM, Konstantin Belousov wrote: > >>> On Tue, May 28, 2013 at 01:35:01PM -0700, Alfred Perlstein wrote: > >>>> [[ moved to hackers@ from private mail. ]] > >>>> > >>>> On 5/28/13 1:13 PM, John Baldwin wrote: > >>>>> On Tuesday, May 28, 2013 3:29:41 pm Alfred Perlstein wrote: > >>>>>> On 5/28/13 9:04 AM, John Baldwin wrote: > >>>>>>> On Tuesday, May 28, 2013 2:13:32 am Alfred Perlstein wrote: > >>>>>>>> Hey folks, > >>>>>>>> > >>>>>>>> I had a talk with Nathan Whitehorn about the camisr issue. The = issue we > >>>>>>>> are seeing we mostly know, but to summarize, we are losing the c= amisr > >>>>>>>> signal and the camisr is not being run. > >>>>>>>> > >>>>>>>> I gave him a summary of what we have been seeing and pointed him= to the > >>>>>>>> code I am concerned about here: > >>>>>>>> http://pastebin.com/tLKr7mCV (this is inside of kern_intr.c). > >>>>>>>> > >>>>>>>> What I think that is happening is that the setting of it_need to= 0 > >>>>>>>> inside of sys/kern/kern_intr.c:ithread_loop() is not being sched= uled > >>>>>>>> correctly and it is being delayed until AFTER the call to > >>>>>>>> ithread_execute_handlers() right below the atomic_store_rel_int(= ). > >>>>>>> This seems highly unlikely, to the extent that if this were true = all our > >>>>>>> locking primitives would be broken. The store_rel is actually a = release > >>>>>>> barrier which acts like more a read/write fence. No memory acces= ses (read or > >>>>>>> write) from before the release can be scheduled after the associa= ted store, > >>>>>>> either by the compiler or CPU. That is what Konstantin is referr= ing to in his > >>>>>>> commit when he says "release" semantics". > >>>>>> Yes, that makes sense, however does it specify that the writes *mu= st* > >>>>>> occur at that *point*? If it only enforces ordering then we may h= ave > >>>>>> some issue, specifically because the setting of it to '1' inside of > >>>>>> intr_event_schedule_thread has no barrier other than the acq seman= tics > >>>>>> of the thread lock. I am wondering what is forcing out the '1' th= ere. > >>>>> Nothing ever forces writes. You would have to invalidate the cache= to do that > >>>>> and that is horribly expensive. It is always only about ordering a= nd knowing > >>>>> that if you can complete another operation on the same "cookie" var= iable with > >>>>> acquire semantics that earlier writes will be visible. > >>>> By cookie, you mean a specific memory address, basically a lock? Thi= s is > >>>> starting to reinforce my suspicions as the setting of it_need is done > >>>> with release semantics, however the acq on the other CPU is done on = the > >>>> thread lock. Maybe that is irrelevant. We will find out shortly. > >>>> > >>>>>> See below as I think we have proof that this is somehow happening. > >>>>> Having ih_need of 1 and it_need of 0 is certainly busted. The simp= lest fix > >>>>> is probably to stop using atomics on it_need and just grab the thre= ad lock > >>>>> in the main ithread loop and hold it while checking and clearing it= _need. > >>>>> > >>>> OK, we have some code that will either prove this, or perturb the me= mory > >>>> ordering enough to make the bug go away, or prove this assertion wro= ng. > >>>> > >>>> We will update on our findings hopefully in the next few days. > >>> IMO the read of it_need in the 'while (ithd->it_need)' should > >>> have acquire semantic, otherwise the future reads in the > >>> ithread_execute_handlers(), in particular, of the ih_need, could pass > >>> the read of it_need and cause the situation you reported. I do not > >>> see any acquire barrier between a condition in the while() statement > >>> and the read of ih_need in the execute_handlers(). > >>> > >>> It is probably true that the issue you see was caused by r236456, in = the > >>> sense that implicitely locked xchgl instruction on x86 has a full bar= rier > >>> semantic. As result, the store_rel() was actually an acquire too, ma= king > >>> this reordering impossible. I argue that this is not a bug in r23645= 6, > >>> but the issue in the kern_intr.c. > >> If I remember the code correctly that would probably explain why we see > >> it only on 9.1 system. > >>> On the other hand, the John' suggestion to move the manipulations of > >>> it_need under the lock is probably the best anyway. > >>> > >> I was wondering if it would be lower latency to maintain it_need, > >> however to keep another variable it_needlocked under the thread lock. > >> This would result in potential superfluous interrupts, however under > >> load you would allow the ithread to loop without taking the thread lock > >> some number of times. > >> > >> I am not really sure if this is really worth the optimization > >> (especially since it can result in superfluous interrupts) however it > >> may reduce latency and that might be important. > >> > >> Is there some people that I can pass the patch onto for help with > >> performance once we confirm that this is the actual bug? We can do > >> internal testing, but I am worried about regressing performance of any > >> form of IO for the kernel. > >> > >> I'll show the patch soon. > >> > >> Thank you for the information. This is promising. > > Well, if you and I are right, the minimal patch should be > > > > diff --git a/sys/kern/kern_intr.c b/sys/kern/kern_intr.c > > index 8d63c9b..7c21015 100644 > > --- a/sys/kern/kern_intr.c > > +++ b/sys/kern/kern_intr.c > > @@ -1349,7 +1349,7 @@ ithread_loop(void *arg) > > * we are running, it will set it_need to note that we > > * should make another pass. > > */ > > - while (ithd->it_need) { > > + while (atomic_load_acq_int(&ithd->it_need)) { > > /* > > * This might need a full read and write barrier > > * to make sure that this write posts before any > > >=20 > OK we can try this. >=20 > I've been pretty good at locking when using mutexes, but when we get=20 > into the atomic ops like this it gets a little tough for me to follow=20 > without extensive research. I know that the signalling thread=20 > (swi_sched caller) does not use any atomic ops... is this OK? The sequence of the actions there is atomic_store_rel_int(&ih->ih_need, 1); later in intr_event_schedule_thread(): it->it_need =3D 1; thread_lock(td); sched_add(td); thread_unlock(td); There are two things to worry about, as I see: 1. The possibiblity of seeing it_need =3D=3D 1 but ih_need =3D=3D 0. This = could happen on some architectures which allow the write reordering, so it might make sense to move the store_rel from ih_need to it_need. Effectively, we would need to have rel in both places, since=20 the call to intr_event_schedule_thread() is conditional. But the reordering is impossible on x86. 2. A possibility of scheduling the interrupt thread without CPU running it noticing the it_need =3D 1 write. Since the thread lock is locked and released, the release barrier is applied, so the write should become visible before thread is scheduled on processor. I would start with the only addition of load_acq() for now and see if it helps. --aN4sSdpAXbmVieFt Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRpbDwAAoJEJDCuSvBvK1BCYEP/3rXtRdLZe2SGCBYhIMuSOJU uYhNnd9rFRhgm6GMzSenbIMr/mEGfvsAFWkQMBZ0SQ1XHQjJ1oS90rVcutdMFdDl mv8CxAG+dsebpd75e9RZXEcsVze6NmA2Iqv79JE9eCo8Cgw0Mpks+Jo2ys9w0D+E CkOOTi2Ga6VuR0+jEMWuJRsgLXKzWxoKuIJzDdZfXQiTrh45s87EXExbF/QgYDXz 51Fcqd01BSLeDDnfrAozy2zxHkxnOq69pxhRr/SSoG/1VHJnxL5QhOT8fFVZ8hJq KfcXj6dCROPP4nsn17EAmQuxYMJCewsi6Ub5573wpSw6JWtg4tgyi+GQfrSUXX4e Czhj+fCsp+HPTstYLOAGm2Bwyzv31PBUAbPLNi5vaVQIlTXs3jl0IiCpkVgLrTbO F6JjctxMO6giSc73UmEi1miQv2aSGRFSwBq4PpAqHo6cdCeSC7E11Jv4nnPO9TWo Wlw7KlsKSTaZksBG4dhAe8y9is02SqjLOMRaFDwd3NwzTZTqZ60eQjxQDkf57qy2 MR2h3thrDwl/Q9oRXPSJd9MBhVY2mDidwHy3VTfTmpCxmCVBp7w0tffNpGRQ8p/w PtW9VQhDtBFeaJ+Jw0AFM+j9+bkA//FlpnhjJKXmL1fxJiPjqbx0Qc74uR6H3Tw9 PoWTUACr3VWqaxmwaw9Y =R2ZG -----END PGP SIGNATURE----- --aN4sSdpAXbmVieFt-- From owner-freebsd-hackers@FreeBSD.ORG Wed May 29 16:03:22 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C6C2A41A; Wed, 29 May 2013 16:03:22 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-ie0-x22f.google.com (mail-ie0-x22f.google.com [IPv6:2607:f8b0:4001:c03::22f]) by mx1.freebsd.org (Postfix) with ESMTP id 8D0A664F; Wed, 29 May 2013 16:03:22 +0000 (UTC) Received: by mail-ie0-f175.google.com with SMTP id tp5so8596183ieb.20 for ; Wed, 29 May 2013 09:03:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=Y/zf121WgqklytmqnP9Z2OuO196go5jzuafbf/WD8Pk=; b=RePDd3n3do6QViq4SL20+ATSpUX6OXwnQDVFfjNlE4g4DAizmK8WrPuzyPSRPLJYEj RzUEEOunNfuqFqrBiutIMopQQFr3TVsj3idOUJXW/I2jjyKalj2MYmjoU/WOMCQObfPg oO8MdnHA8idUc40WH3kdnFgAeI/DIP0IluFisswMXjQVndgxytSLur8RbJZyl2Pr45tQ 8idSzMxZcfE79wS0a+fw0KNt0qN8eXzpY+qhZ7WeCfTSLCpuUzV6wJTcoVIlB5qke6NE J11AIQe99mPtMpS9gk/xUoxi6hPbk/9q6N3Mdl7p3IMxAK3RUybUbiT9ccXrDqoFJNh3 OOAA== X-Received: by 10.43.106.202 with SMTP id dv10mr1350012icc.37.1369843402259; Wed, 29 May 2013 09:03:22 -0700 (PDT) MIME-Version: 1.0 Received: by 10.64.71.101 with HTTP; Wed, 29 May 2013 09:02:52 -0700 (PDT) From: Chris Rees Date: Wed, 29 May 2013 17:02:52 +0100 Message-ID: Subject: Order of canonical upgrade sequence To: "freebsd-hackers@freebsd.org" , Warner Losh , Alexander Leidinger , Glen Barber Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 May 2013 16:03:22 -0000 Hi all! Back in 2005, when Alexander Leidinger wrote the make delete-old target, he documented the order of upgrade such that it should be run before mergemaster [1]; # 7. `make installworld' # 8. `make delete-old' # 9. `mergemaster' I have merged the delete-old section of the Handbook into the upgrading chapter, and independently decided to put mergemaster first, because I thought it would be safer, but checked here before I committed. I think that steps 8 and 9 should be reversed, because of the possibility of an unbootable system being made, when an rc script references an executable that has just been removed for example. I cannot think of an example where the system is left unbootable/damaged if make delete-old is run after mergemaster. What do people think of the patch at [2]? Chris [1] http://svnweb.freebsd.org/base/head/Makefile?r1=148329&r2=148330& [2] http://www.bayofrum.net/~crees/patches/delete-old-order.diff From owner-freebsd-hackers@FreeBSD.ORG Wed May 29 16:18:30 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 906C87BF for ; Wed, 29 May 2013 16:18:30 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ob0-x235.google.com (mail-ob0-x235.google.com [IPv6:2607:f8b0:4003:c01::235]) by mx1.freebsd.org (Postfix) with ESMTP id 5C24A79D for ; Wed, 29 May 2013 16:18:30 +0000 (UTC) Received: by mail-ob0-f181.google.com with SMTP id 16so1530007obc.12 for ; Wed, 29 May 2013 09:18:30 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=dOu5gNj0ilH11mfk3tU7MIay/9VDj4IEz6DUqd95NAM=; b=co0UtEitIQiw+nr2+O2IcVz+PJTcYgoyhsdGJaH/pqAz0uenMVHqw3t7RZDqg2I1iL 0auL6gYcIVJVyTTtCAsPH2uVpdJn95W6Fj/wXbRsCmPbq15El3N82G/fmwNEYJhSyUjn Ohsp/ZJDigPEUruiHdF6jctRQhZYIYC5A0F3v0rX4YZbJKu8vecnz4NCaJVLCtG2LNc5 4ENV52uaXo4CoVVySSeN2P8cY3FCJyU4lRv1VEXRH8paV1Hoov/8lM6AsEdJuBdkh0yv Gnd/fOMmzyht8PWD2YN9pUPrrCnge03RC/OEsbeW/NtUVbj8PfRfOQkm/Mim7hNzmCs6 ER5Q== X-Received: by 10.60.36.136 with SMTP id q8mr2057221oej.90.1369844309874; Wed, 29 May 2013 09:18:29 -0700 (PDT) Received: from monkey-bot.int.fusionio.com ([209.117.142.2]) by mx.google.com with ESMTPSA id x10sm41549472oes.6.2013.05.29.09.18.27 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 29 May 2013 09:18:28 -0700 (PDT) Sender: Warner Losh Subject: Re: Order of canonical upgrade sequence Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: Date: Wed, 29 May 2013 10:18:26 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <05285081-3881-4AD7-8F17-EF919F2A0076@bsdimp.com> References: To: Chris Rees X-Mailer: Apple Mail (2.1085) X-Gm-Message-State: ALoCoQnrZIoYckuwGXj1Q4NVpOKmVkOIjX0DKQOZeltDwooT2t8n0X6O6tmTg/fLuitHCt2UZzVu Cc: "freebsd-hackers@freebsd.org" , Alexander Leidinger , Glen Barber X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 May 2013 16:18:30 -0000 On May 29, 2013, at 10:02 AM, Chris Rees wrote: > Hi all! >=20 > Back in 2005, when Alexander Leidinger wrote the make delete-old > target, he documented the order of upgrade such that it should be run > before mergemaster [1]; >=20 > # 7. `make installworld' > # 8. `make delete-old' > # 9. `mergemaster' >=20 > I have merged the delete-old section of the Handbook into the > upgrading chapter, and independently decided to put mergemaster first, > because I thought it would be safer, but checked here before I > committed. >=20 > I think that steps 8 and 9 should be reversed, because of the > possibility of an unbootable system being made, when an rc script > references an executable that has just been removed for example. While I don't care, any old scripts that are removed won't make the = system unbootable. delete-old deletes both the executable and the rc = script in those cases where we've moved functionality out of the base = system. The rc scripts themselves have protection against executables = not found, and the damage will be limited to at most the one script that = references that binary. Since the binary is gone, the ill effects are = minimal. So I'm finding it hard to believe this is a credible example of danger. > I cannot think of an example where the system is left > unbootable/damaged if make delete-old is run after mergemaster. I'm not sure that it will be that much safer, but to be honest, I run = delete-old about once every presidential election cycle on my machines. > What do people think of the patch at [2]? It's likely a tiny bit safer, but I don't think it is necessary. Warner > Chris >=20 > [1] http://svnweb.freebsd.org/base/head/Makefile?r1=3D148329&r2=3D148330= & >=20 > [2] http://www.bayofrum.net/~crees/patches/delete-old-order.diff From owner-freebsd-hackers@FreeBSD.ORG Wed May 29 17:53:51 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id AED5625A for ; Wed, 29 May 2013 17:53:51 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-oa0-f48.google.com (mail-oa0-f48.google.com [209.85.219.48]) by mx1.freebsd.org (Postfix) with ESMTP id 7B2ABCE5 for ; Wed, 29 May 2013 17:53:51 +0000 (UTC) Received: by mail-oa0-f48.google.com with SMTP id i4so11959363oah.21 for ; Wed, 29 May 2013 10:53:44 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=3Ok7P0HNzBekNR/A9bJhVzu24Vii3EzbC+YC6LF5Alg=; b=VpJ5PB6HX34dwws9eN9RUtS/ucm3E0a/50eopjIBQGJ1zTCoTanl4EF4VdqqxAiOGv wi+QE+/rpb2PglfmtI+T/WwWsI54ZW9k2A+yigIwZ+DGlOTROz/4VS0f60/UykB8nXH7 2J6AQB5RjDqsHZrSuU9qk+DZMyFBOBn0qbGL6+WbcYYPiRaaUuWrqWDb04dV7Q+a8AwX OmPPgRHqHfi76qtcMMaSv+KR/ei1uRv2gD2FhRaJg6TfdPtFFcocXvc2PAcQwzwFGb0j c6dcF4Wk19TE1+6SH/N/VtGEDj/kztFsGEbMb926t+EeoNGDdX5z6Ycr2DuJmZa2fZsh oGNw== X-Received: by 10.60.94.48 with SMTP id cz16mr2326250oeb.5.1369850024781; Wed, 29 May 2013 10:53:44 -0700 (PDT) Received: from monkey-bot.int.fusionio.com ([209.117.142.2]) by mx.google.com with ESMTPSA id jt1sm42006383oeb.5.2013.05.29.10.53.43 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Wed, 29 May 2013 10:53:43 -0700 (PDT) Sender: Warner Losh Subject: Re: Order of canonical upgrade sequence Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: Date: Wed, 29 May 2013 11:53:41 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <9AD6CC08-8C59-4941-8C1D-54ABE74A162C@bsdimp.com> References: To: Chris Rees X-Mailer: Apple Mail (2.1085) X-Gm-Message-State: ALoCoQmkXPQnSwLVLriYJIuuStCA6iVXiCjRPNstcyA4n6C1PEZ9vX4C5mSGNi1ZRUr2mv8riCin Cc: "freebsd-hackers@freebsd.org" , Alexander Leidinger , Glen Barber X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 May 2013 17:53:51 -0000 [[ summarizing a conversation in irc ]] The below fragment doesn't match UPDATING. Since I don't think the order = matters; and since we've had no reports that UPDATING is wrong; and = since I think way more people follow updating than the Makefile; we = should fix the makefile and make the docs match both. Warner On May 29, 2013, at 10:02 AM, Chris Rees wrote: > Hi all! >=20 > Back in 2005, when Alexander Leidinger wrote the make delete-old > target, he documented the order of upgrade such that it should be run > before mergemaster [1]; >=20 > # 7. `make installworld' > # 8. `make delete-old' > # 9. `mergemaster' >=20 > I have merged the delete-old section of the Handbook into the > upgrading chapter, and independently decided to put mergemaster first, > because I thought it would be safer, but checked here before I > committed. >=20 > I think that steps 8 and 9 should be reversed, because of the > possibility of an unbootable system being made, when an rc script > references an executable that has just been removed for example. >=20 > I cannot think of an example where the system is left > unbootable/damaged if make delete-old is run after mergemaster. >=20 > What do people think of the patch at [2]? >=20 > Chris >=20 > [1] http://svnweb.freebsd.org/base/head/Makefile?r1=3D148329&r2=3D148330= & >=20 > [2] http://www.bayofrum.net/~crees/patches/delete-old-order.diff From owner-freebsd-hackers@FreeBSD.ORG Wed May 29 19:25:57 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 84D531AD; Wed, 29 May 2013 19:25:57 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-ie0-x22c.google.com (mail-ie0-x22c.google.com [IPv6:2607:f8b0:4001:c03::22c]) by mx1.freebsd.org (Postfix) with ESMTP id 512D426B; Wed, 29 May 2013 19:25:57 +0000 (UTC) Received: by mail-ie0-f172.google.com with SMTP id 17so5332927iea.17 for ; Wed, 29 May 2013 12:25:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=sU3iydEr35wYfZoxeMN6KLznwoSqnFFPY2QDGFYRITQ=; b=uqJi9C+dH3B9KB7hjNxKbRbfYe0QO7mlJlw0rxgzx6+FTORqW+6ODzwQE52NTHGBS2 WorCT3WzwSvxUNBdNRzWxOgSgo++YedMD5Rq+zwOIr29UY12pA7L7aDzcrR4ppydPEPF sZ/KgKKpohxXvTw5TZJjbykWexE5OuC6bFgCNNyKEXtdIylAttGshkQYoapFiZ4+yOcQ XVubWRr6KYZsBme5d7J7OTPPld7B3cgvmfTnQXYAdzBp7SfZa97GVnAeU3pj07qZ9k4i ZJsuLY49lbAtLyI4eWx9qrA5uKPxnkT+wS/iGdK7KEAJcjMj/mSjQuxgglFAH4/BhQEg LZxw== X-Received: by 10.50.72.49 with SMTP id a17mr9931264igv.36.1369855556981; Wed, 29 May 2013 12:25:56 -0700 (PDT) MIME-Version: 1.0 Received: by 10.64.71.101 with HTTP; Wed, 29 May 2013 12:25:26 -0700 (PDT) In-Reply-To: <20130529205711.00000111@unknown> References: <9AD6CC08-8C59-4941-8C1D-54ABE74A162C@bsdimp.com> <20130529205711.00000111@unknown> From: Chris Rees Date: Wed, 29 May 2013 20:25:26 +0100 Message-ID: Subject: Re: Order of canonical upgrade sequence To: Alexander Leidinger Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-hackers@freebsd.org" , Glen Barber X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 May 2013 19:25:57 -0000 On 29 May 2013 19:57, Alexander Leidinger wrote: > On Wed, 29 May 2013 11:53:41 -0600 > Warner Losh wrote: > >> [[ summarizing a conversation in irc ]] >> >> The below fragment doesn't match UPDATING. Since I don't think the >> order matters; and since we've had no reports that UPDATING is wrong; >> and since I think way more people follow updating than the Makefile; >> we should fix the makefile and make the docs match both. > > The order matters, mergemaster first, then delete-old. UPDATING is > correct. At least regarding the order all places should be corrected > which tell to use the reverse order. > >> Warner >> >> On May 29, 2013, at 10:02 AM, Chris Rees wrote: >> >> > Hi all! >> > >> > Back in 2005, when Alexander Leidinger wrote the make delete-old >> > target, he documented the order of upgrade such that it should be >> > run before mergemaster [1]; >> > >> > # 7. `make installworld' >> > # 8. `make delete-old' >> > # 9. `mergemaster' > > This is the wrong order. > >> > I have merged the delete-old section of the Handbook into the >> > upgrading chapter, and independently decided to put mergemaster >> > first, because I thought it would be safer, but checked here before >> > I committed. >> > >> > I think that steps 8 and 9 should be reversed, because of the >> > possibility of an unbootable system being made, when an rc script >> > references an executable that has just been removed for example. > > I agree. > Committed. Thanks! Chris From owner-freebsd-hackers@FreeBSD.ORG Wed May 29 18:57:34 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 9AFB778E; Wed, 29 May 2013 18:57:34 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from mail.ebusiness-leidinger.de (mail.ebusiness-leidinger.de [217.11.53.44]) by mx1.freebsd.org (Postfix) with ESMTP id 56C88126; Wed, 29 May 2013 18:57:34 +0000 (UTC) Received: from outgoing.leidinger.net (p57A38BC2.dip0.t-ipconnect.de [87.163.139.194]) by mail.ebusiness-leidinger.de (Postfix) with ESMTPSA id 6B5FF84476B; Wed, 29 May 2013 20:57:13 +0200 (CEST) Received: from unknown (Titan.Leidinger.net [192.168.1.17]) by outgoing.leidinger.net (Postfix) with ESMTP id C76091155; Wed, 29 May 2013 20:57:10 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=leidinger.net; s=outgoing-alex; t=1369853830; bh=Innpq4ui3l6lran5CQJUyNpXKv3m4o2Nr2apHRAY93o=; h=Date:From:To:Cc:Subject:In-Reply-To:References; b=LSoq7m70OJBSW9bLMj+w2YsJ38EhPNZ4X8gEdoaIy96tdEiiXqLrB8YoBfmBWT9z6 baM1VCW3GETbSApeonMIG2GSJC1GagJjK2lsPEhVotaHG4APIT26xOGNK/CuSKfg64 3nUbkkvGxngLOQds1DQyl5pYZvlmibfi7seUqUT3vr/hU3jT2dUu9aZ/3rzuWiiAB5 r34H/JYq1T5pp2lofgaz797K9MZvVcKBOjJv/yARFlB3A+fs6wmM4Jxrp84gxpwdH5 0WSHDjo3t10H1S2AvZpaE8uvwkyg7C7Fvjz2vqENp4wz71eruT0ljH5O7dHwQ6J8on IIladbUKFcL8g== Date: Wed, 29 May 2013 20:57:11 +0200 From: Alexander Leidinger To: Warner Losh Subject: Re: Order of canonical upgrade sequence Message-ID: <20130529205711.00000111@unknown> In-Reply-To: <9AD6CC08-8C59-4941-8C1D-54ABE74A162C@bsdimp.com> References: <9AD6CC08-8C59-4941-8C1D-54ABE74A162C@bsdimp.com> X-Mailer: Claws Mail 3.9.1-2-g66aa06 (GTK+ 2.16.6; i586-pc-mingw32msvc) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-EBL-MailScanner-Information: Please contact the ISP for more information X-EBL-MailScanner-ID: 6B5FF84476B.A3D64 X-EBL-MailScanner: Found to be clean X-EBL-MailScanner-SpamCheck: not spam, spamhaus-ZEN, SpamAssassin (not cached, score=-1.141, required 6, autolearn=disabled, ALL_TRUSTED -1.00, AWL -0.03, DKIM_SIGNED 0.10, DKIM_VALID -0.10, DKIM_VALID_AU -0.10, T_RP_MATCHES_RCVD -0.01, URIBL_BLOCKED 0.00) X-EBL-MailScanner-From: alexander@leidinger.net X-EBL-MailScanner-Watermark: 1370458634.5863@t+f5AEyEa1r0rESTUBBUZg X-EBL-Spam-Status: No X-Mailman-Approved-At: Wed, 29 May 2013 19:37:23 +0000 Cc: "freebsd-hackers@freebsd.org" , Chris Rees , Glen Barber X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 May 2013 18:57:34 -0000 On Wed, 29 May 2013 11:53:41 -0600 Warner Losh wrote: > [[ summarizing a conversation in irc ]] > > The below fragment doesn't match UPDATING. Since I don't think the > order matters; and since we've had no reports that UPDATING is wrong; > and since I think way more people follow updating than the Makefile; > we should fix the makefile and make the docs match both. The order matters, mergemaster first, then delete-old. UPDATING is correct. At least regarding the order all places should be corrected which tell to use the reverse order. > Warner > > On May 29, 2013, at 10:02 AM, Chris Rees wrote: > > > Hi all! > > > > Back in 2005, when Alexander Leidinger wrote the make delete-old > > target, he documented the order of upgrade such that it should be > > run before mergemaster [1]; > > > > # 7. `make installworld' > > # 8. `make delete-old' > > # 9. `mergemaster' This is the wrong order. > > I have merged the delete-old section of the Handbook into the > > upgrading chapter, and independently decided to put mergemaster > > first, because I thought it would be safer, but checked here before > > I committed. > > > > I think that steps 8 and 9 should be reversed, because of the > > possibility of an unbootable system being made, when an rc script > > references an executable that has just been removed for example. I agree. Bye, Alexander. -- http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-hackers@FreeBSD.ORG Wed May 29 20:05:58 2013 Return-Path: Delivered-To: hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 899F0F21 for ; Wed, 29 May 2013 20:05:58 +0000 (UTC) (envelope-from bcr@FreeBSD.org) Received: from mxout1.bln1.prohost.de (mxout1.bln1.prohost.de [213.160.84.47]) by mx1.freebsd.org (Postfix) with ESMTP id 2B811639 for ; Wed, 29 May 2013 20:05:57 +0000 (UTC) Received: from Benedicts-Macbook-Pro.local (p4FC73854.dip0.t-ipconnect.de [79.199.56.84]) (authenticated bits=0) by mx1.bln1.prohost.de (8.14.4/8.14.4) with ESMTP id r4TK5sqw002130 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 29 May 2013 22:05:54 +0200 Message-ID: <51A65FA3.4040602@FreeBSD.org> Date: Wed, 29 May 2013 22:05:55 +0200 From: Benedict Reuschling Organization: The FreeBSD Project User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: hackers@FreeBSD.org Subject: Summary of the Documentation Working Group at the BSDCan 2013 DevSummit X-Enigmail-Version: 1.5.1 OpenPGP: id=4A819348 Content-Type: text/plain; charset=ISO-8859-15 Content-Transfer-Encoding: 7bit X-Null-Tag: 94a2244d38d221c97cadb67134b1325f X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: bcr@FreeBSD.org List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 29 May 2013 20:05:58 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello all, this a summary of the doc working group at the BSDCan 2013 DevSummit for those who could not attend but wanted to be kept in the loop about what is going on. Excuse my non-brevity, we had a lot to talk about. Some personal notes are in there, but the rest is written in a neutral style. No presentations were given in the session itself, we worked with the wiki page and recorded everything in an openetherpad document, upon which this summary is based. Overview: The documentation working group had formulated a number of topics on the wiki [1] before the working group began and we managed to cover all of those. I, as one of the two chairpersons for this group (the other being Dru Lavigne like in previous years), was pleasantly surprised that we had more attendees and even some that had never been to one of our doc working groups before. On a personal note, it was also the first time that all of my present and past doc mentees were in the same room together, which made me very happy that this was possible. The previous day at the vendor summit, a number of issues came up that fall into the documentation area like the website. George Neville-Neil, who lead that session, took these items for himself. Having recently been awarded a doc commit bit himself, he was also in our documentation session and provided his unique insights not only as a long time src committer, but also as a book author, which in my opinion helped a lot in some of the discussions we had. Topic #1: Handbook print edition We talked about this almost half the time of the first session before the break. Dru Lavigne gave an update on what work had been done up to this point by her in the ISBN repo branch [2]. The first pass of the handbook is mostly completed, which resulted in a number of writing style and whitespace fixes in preparation for bigger content changes to come. In addition to the already huge todo list [3], she prepared a second one with notes [4] (will be merged into [3] at some point) about each individual section. For the second round of content changes that are required to bring the handbook into FreeBSD 9.X-only world, we need to tap specific developers on the shoulders to get their information for missing content. The observation was that asking on a mailing list does not bring enough people in to help. A list of people was created (table 2 in [3]) that have done work in specific areas where we need their insights about whether content is still accurate or needs to be rewritten to reflect the current state of the system. Doc developers that are interested in these sections will contact these developers and ask for their help. The working group especially reiterated an important point and I would like to repeat it here to make it clear for everyone who is reluctant (to say the least) to write documentation: It is content we need, not markup. If you as a developer provide us with a few paragraphs of a textual description, we will gladly take it and take care of the markup/formatting/editing work that is required ourselves. We might get back to you to clarify content things, but everything else is something that doc developers would be happy to take off your shoulders if it is too much of a hassle for you developers! Another topic that came up with making drastic content changes and chapter/section rearrangements on the handbook in the ISBN branch is that some doc developers are worried that they might receive pushback from others worrying about the state of the handbook and asking for reverts of changes. This has happened in the past with the deactivation of the PGP keys section in the branch (not the live handbook on www.freebsd.org), which would reduce the current PDF from 1491 to 954 pages, which is still a huge number. The argument was that having the keys (or at least a few important ones like the secteam keys) are important for verification, but the room consensus was that these are unnecessary in a printed book. While this specific issue has already been resolved, it is an example of what might occur in the future once the doc team starts to make big changes which others won't like. The consensus was (and gnn@ confirmed this to me after the session was over by independently talking to core members about it) as follows: Editorial choices are the choice of the doc team, However, edits which inadvertently break technical meaning need to be reverted/fixed. I think this is in everyones interest. We don't want people to be completely silent with feedback and discussions about these changes, while at the same time stopping progress that needs to happen in a handbook that has a huge need for fresh and the removal of obsolete content. Print on demand was briefly touched on by David Chisnall and Michael Lucas who could provide contacts with publishers specialising in low-volume print runs for people willing to write FreeBSD-related books. Topic #2: www.freebsd.org website redesign This was previously discussed at the Cambridge DevSummit last year and some of the feedback from the vendor summit the day before (see above) was also brought in. Many suggestions were made in improving the search feature, which does not work very well at the moment. Things like the need for a content management system (separating content from layout), missing links to the wiki and the FreeBSD forums, twitter/social media integration ideas while still making the site easy to manage were debated. Generally, the website is another big project for the documentation team and we agreed that a small team of interested individuals should form and send a mail to ask for participation and to brainstorm ideas. Topic #3: documentation infrastructural changes This was only briefly touched on as Glen Barber reported that he was in contact with Gabor Kovesdan for importing a newer DocBook version into the main tree, which Gabor has been working in a branch for a while now. EPUB support that this change will bring (along with other important improvements like conditional text to mark release specific sections) is something that I personally have been looking forward to. The reason is that once the handbook is ready for publishing via the FreeBSD Mall, we will also provide ebook versions for people to pay for with receipts going back to the FreeBSD foundation as donations. This will hopefully provide another incentive for people to work on documentation when they see that this supports the FreeBSD project in a different way than it has now. Topic #4: Recruiting new doc committers/contributors Recruiting new doc committers is another topic we talked about, since our ranks are still low compared to the ports or source people. Clearly, part of our current problems is that we lack the people to do many of the things we want to accomplish in our documentation set. Ideas revolved around talking to people who write howtos on the FreeBSD forums, create a separate user wiki for content submissions, an option for each handbook page to add notes (like the PostgreSQL project has) and finding a suitable DocBook Editor so people don't have to worry too much about the markup and can just focus on providing content. Topic #5: Helping our translation projects keeping up with changes Helping the translation teams was a topic that I put in being a member of the FreeBSD German Documentation Project and observing similar things happening in the other language teams: we can't keep up. In most of our translation projects, only two people (the former mentee and his/her mentor) try to catch up with the work that few very active (which is a good thing, mind you!) committers are doing in the english reference tree. I'm investigating textproc/pootle at the moment as an alternative way of getting additional translations by reusing already translated strings. PC-BSD apparently has some success with it and I was thinking about sharing their database since both systems use the same basic terminology. I had the chance during the conference to talk to Kris Moore about this possibility and he said that Pootle needed a lot of extra scripts to be written in order to automate things and to be as helpful as it it for them now. Sharing just the database might be an option, but there are many PC-BSD graphical menu strings in there that simply don't exist in FreeBSD so there won't be much reuse of already translated strings. However, I think this is still better than starting from scratch with a fresh Pootle installation. I'm willing to do more experimenting in that area to come up with a solution. Topic #6: Separate doc hackathon We talked about organizing a separate doc hackathon to get things up to speed, especially in the print edition of the handbook. This would be more focused on doing actual work rather than talking about what needs to be done, with the possibility to talk to people in the room when the need arises. David Chisnall offered room space and logistics for such an event at the Cambridge DevSummit, which has recently been announced. Topic #7 and 8: Blanket doc commit rights A brief topic was giving members of some non-doc teams blanket commit rights (after a mentoring period) to make commits on documents that they maintain mostly, but need to go through a doc person at the moment to do the actual commit. A prominent example is the porters handbook, which is actively maintained by the ports team sending patches in PRs. The goal with such a blanket commit rule is to reduce the workload on the doc committers while allowing these teams to maintain their documents faster. We will work with teams that are open to this idea to have at least one member of their team get a commit bit so that they can channel changes through that person. if a team feels that this is of no added value to them or prefer to go through the doc team first, that is also fine with us. Topic #9: Quality assurance Quality assurance in our documentation set was only briefly touched on as we were already running late in the session. We talked about the possibilities of regular textproc/igor runs that display their findings on a webpage or mailing the results to the doc team. This would either result in igor being improved (due to false positives) or people starting to fix bugs that were reported. Regular link checker runs also should be done as finding broken links is something that a machine is better at than humans are. All in all, it was a very productive session and we continued talking after the session ended, either over dinner or at the Hacking Lounge. I also want to highlight that we had a few people in the doc lounge (running parallel to the Hacking Lounge) that were interested in helping out with documentation work. Doc committers were there to help them get started and a few PRs were filed into GNATS and later resolved as a direct result of these sessions. Clearly, showing someone in person what needs to be done is more effective and yields results faster than doing it via any other means. My thanks go to all who participated and provided insights, feedback, ideas for improvement, support and their experience during the working group session. A big THANK YOU goes to John Baldwin for organizing the DevSummit so well again and all the sponsors that made it possible. If you have questions on any of these topics, feel free to drop me a line. Regards Benedict Reuschling FreeBSD Documentation Committer The FreeBSD Documentation Project FreeBSD German Documentation Project - https://doc.bsdgroup.de [1] https://wiki.freebsd.org/201305DevSummit/Documentation [2] http://svnweb.freebsd.org/doc/projects/ISBN_1-57176-407-0/en_US.ISO8859-1/books/handbook/ [3] https://wiki.freebsd.org/PublishedFormats [4] http://openetherpad.org/R6vMaCQ1GI -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.8 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEARECAAYFAlGmX6MACgkQTSZQLkqBk0iihwCgtP/FO44TPRkP94RN0w+gJO57 6owAn2dc1ZEBOIPUnLGz12rvzgzjzZWk =gu9w -----END PGP SIGNATURE----- From owner-freebsd-hackers@FreeBSD.ORG Thu May 30 13:42:16 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id BE9CCC86; Thu, 30 May 2013 13:42:16 +0000 (UTC) (envelope-from steve@mouf.net) Received: from mouf.net (mouf.net [IPv6:2607:fc50:0:4400:216:3eff:fe69:33b3]) by mx1.freebsd.org (Postfix) with ESMTP id 7FD57238; Thu, 30 May 2013 13:42:16 +0000 (UTC) Received: from meatwad.mouf.net (cpe-098-122-135-254.nc.res.rr.com [98.122.135.254]) (authenticated bits=0) by mouf.net (8.14.5/8.14.5) with ESMTP id r4UDg5cn087046 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NOT); Thu, 30 May 2013 13:42:10 GMT (envelope-from steve@mouf.net) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=mouf.net; s=mail; t=1369921334; bh=bj4PxZIVqf0TlTqKuzl8ud+L8KEfVQM38iOiQRLzAWw=; h=Date:From:To:CC:Subject:References:In-Reply-To; b=crT4Z1Jj8dT7h7syfPuY0axtTFi+z2XLTKa2RL9CYd9iWmJf28He7jEG4C04h4IA7 dCTZhtXxKpZ70l4Q+enUdVbuxwLPFOImOfBe2uRmr6bEE5PpM8KEeWoQRK6dfWXl9X UVMlMHE8RCnBKy/RvVNvmpGtfTJFqqWvYhM0FQ7Q= Message-ID: <51A7572D.7050808@mouf.net> Date: Thu, 30 May 2013 13:42:05 +0000 From: Steve Wills User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130413 Thunderbird/17.0.5 MIME-Version: 1.0 To: Chris Rees Subject: Re: Order of canonical upgrade sequence References: In-Reply-To: X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.4.3 (mouf.net [199.48.129.64]); Thu, 30 May 2013 13:42:11 +0000 (UTC) X-Spam-Status: No, score=0.0 required=4.5 tests=none autolearn=unavailable version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on mouf.net X-Virus-Scanned: clamav-milter 0.97.8 at mouf.net X-Virus-Status: Clean Cc: "freebsd-hackers@freebsd.org" , Alexander Leidinger , Glen Barber X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 May 2013 13:42:16 -0000 On 05/29/13 16:02, Chris Rees wrote: > Hi all! > > Back in 2005, when Alexander Leidinger wrote the make delete-old > target, he documented the order of upgrade such that it should be run > before mergemaster [1]; > > # 7. `make installworld' > # 8. `make delete-old' > # 9. `mergemaster' > It would be good to mention that it's wise to "make check-old", and rebuilding any ports that depend on the old libs, before doing "make delete-old". Steve From owner-freebsd-hackers@FreeBSD.ORG Thu May 30 13:52:57 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 33869339; Thu, 30 May 2013 13:52:57 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-ea0-x232.google.com (mail-ea0-x232.google.com [IPv6:2a00:1450:4013:c01::232]) by mx1.freebsd.org (Postfix) with ESMTP id 6D5D633F; Thu, 30 May 2013 13:52:56 +0000 (UTC) Received: by mail-ea0-f178.google.com with SMTP id q15so353969ead.23 for ; Thu, 30 May 2013 06:52:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=4cSYseJ3gBWWFrmAklJGW0io5nQNzz485m+kocF2jlg=; b=Md9lWTqsAQ2/FjprAb9p7q9k3B5i9uQ7QUFVFrsvSzskwqDg3MbKHgGw3k3Wz2CIPG mYd3xiTNN4eHwZ/3m+aXYGePNiiiwAy5M4viVYPzIaptp5Cjno+6nLLg8xr6aqOjXfNV T6Mmh+P7PwInX5SVMAxC+uXE7773vMkMe0X8kIG5O/Aeuvm/Fzx2RRftBB2P5f+dlnQx nCyXOoWmbBW0qtkJZ0wye6DyNX7t4mC6qwx+mfm9FqzjNN7WrcqgdSHfksFZkv6ttAH8 pJO7XeXR829rQePSqPE5PoDWJ499CHvvTKM1RCxwNsrKaCJcoIcCiGBUqDisDnS/acCo EZXw== X-Received: by 10.14.174.8 with SMTP id w8mr9920979eel.115.1369921975580; Thu, 30 May 2013 06:52:55 -0700 (PDT) MIME-Version: 1.0 Received: by 10.15.23.76 with HTTP; Thu, 30 May 2013 06:52:25 -0700 (PDT) In-Reply-To: <51A7572D.7050808@mouf.net> References: <51A7572D.7050808@mouf.net> From: Chris Rees Date: Thu, 30 May 2013 14:52:25 +0100 Message-ID: Subject: Re: Order of canonical upgrade sequence To: Steve Wills Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-hackers@freebsd.org" , Alexander Leidinger , Glen Barber X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 May 2013 13:52:57 -0000 On 30 May 2013 14:42, Steve Wills wrote: > On 05/29/13 16:02, Chris Rees wrote: >> Hi all! >> >> Back in 2005, when Alexander Leidinger wrote the make delete-old >> target, he documented the order of upgrade such that it should be run >> before mergemaster [1]; >> >> # 7. `make installworld' >> # 8. `make delete-old' >> # 9. `mergemaster' >> > > It would be good to mention that it's wise to "make check-old", and > rebuilding any ports that depend on the old libs, before doing "make > delete-old". make delete-old doesn't touch the libraries; it's the next steps that do that (make delete-old-libs). The Handbook section is far more verbose on this, and I think that this reference in the Makefile is more as a quick reminder than a step-by-step walkthrough. Chris From owner-freebsd-hackers@FreeBSD.ORG Thu May 30 15:21:12 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 1683B447; Thu, 30 May 2013 15:21:12 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 233A9D04; Thu, 30 May 2013 15:21:10 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id SAA16485; Thu, 30 May 2013 18:21:03 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <51A76E5E.5050206@FreeBSD.org> Date: Thu, 30 May 2013 18:21:02 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130517 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-current@FreeBSD.org, freebsd-hackers Subject: hwpmc with opteron 6128 X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 May 2013 15:21:12 -0000 I am trying to do a very basic thing with hwpmc on this CPU: CPU: AMD Opteron(tm) Processor 6128 (1999.05-MHz K8-class CPU) hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 K8/4/48/0x1ff What I am trying is: $ pmcstat -T -S instructions What I am getting is just: PMC: [FR_RETIRED_X86_INSTRUCTIONS] Samples: 0 (0.0%) , 0 unresolved and nothing else on the screen. Has anyone had a success with this class of processors? Should it be supported? Any ideas/suggestions/hints? P.S. pmccontrol -L reports a whole bunch of "K8" counters, just a small random sub-sample: BU_FILL_INTO_L2 IC_FETCH IC_MISS IC_REFILL_FROM_L2 IC_REFILL_FROM_SYSTEM IC_L1_ITLB_MISS_AND_L2_ITLB_HIT IC_L1_ITLB_MISS_AND_L2_ITLB_MISS IC_MICROARCHITECTURAL_RESYNC_BY_SNOOP IC_INSTRUCTION_FETCH_STALL IC_RETURN_STACK_HIT IC_RETURN_STACK_OVERFLOW FR_RETIRED_X86_INSTRUCTIONS FR_RETIRED_UOPS FR_RETIRED_BRANCHES FR_RETIRED_BRANCHES_MISPREDICTED FR_RETIRED_TAKEN_BRANCHES FR_RETIRED_TAKEN_BRANCHES_MISPREDICTED -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Thu May 30 17:19:22 2013 Return-Path: Delivered-To: freebsd-hackers@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6A62CE19; Thu, 30 May 2013 17:19:22 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 766AF7D9; Thu, 30 May 2013 17:19:20 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id UAA18529; Thu, 30 May 2013 20:19:19 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <51A78A16.4050705@FreeBSD.org> Date: Thu, 30 May 2013 20:19:18 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:17.0) Gecko/20130517 Thunderbird/17.0.6 MIME-Version: 1.0 To: freebsd-current@FreeBSD.org, freebsd-hackers Subject: Re: hwpmc with opteron 6128 References: <51A76E5E.5050206@FreeBSD.org> In-Reply-To: <51A76E5E.5050206@FreeBSD.org> X-Enigmail-Version: 1.5.1 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 May 2013 17:19:22 -0000 on 30/05/2013 18:21 Andriy Gapon said the following: > > I am trying to do a very basic thing with hwpmc on this CPU: > > CPU: AMD Opteron(tm) Processor 6128 (1999.05-MHz K8-class CPU) > hwpmc: SOFT/16/64/0x67 TSC/1/64/0x20 > K8/4/48/0x1ff I didn't realize that the system was running in a VM. Sorry for the noise. > What I am trying is: > $ pmcstat -T -S instructions > > What I am getting is just: > PMC: [FR_RETIRED_X86_INSTRUCTIONS] Samples: 0 (0.0%) , 0 unresolved > > and nothing else on the screen. > > Has anyone had a success with this class of processors? > Should it be supported? > Any ideas/suggestions/hints? > > P.S. pmccontrol -L reports a whole bunch of "K8" counters, just a small random > sub-sample: > BU_FILL_INTO_L2 > IC_FETCH > IC_MISS > IC_REFILL_FROM_L2 > IC_REFILL_FROM_SYSTEM > IC_L1_ITLB_MISS_AND_L2_ITLB_HIT > IC_L1_ITLB_MISS_AND_L2_ITLB_MISS > IC_MICROARCHITECTURAL_RESYNC_BY_SNOOP > IC_INSTRUCTION_FETCH_STALL > IC_RETURN_STACK_HIT > IC_RETURN_STACK_OVERFLOW > FR_RETIRED_X86_INSTRUCTIONS > FR_RETIRED_UOPS > FR_RETIRED_BRANCHES > FR_RETIRED_BRANCHES_MISPREDICTED > FR_RETIRED_TAKEN_BRANCHES > FR_RETIRED_TAKEN_BRANCHES_MISPREDICTED > -- Andriy Gapon From owner-freebsd-hackers@FreeBSD.ORG Thu May 30 22:30:48 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 402E5413 for ; Thu, 30 May 2013 22:30:48 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (unknown [IPv6:2001:610:1108:5012::107]) by mx1.freebsd.org (Postfix) with ESMTP id 07E57EEF for ; Thu, 30 May 2013 22:30:48 +0000 (UTC) Received: from snail.stack.nl (snail.stack.nl [IPv6:2001:610:1108:5010::131]) by mx1.stack.nl (Postfix) with ESMTP id A285612013D; Fri, 31 May 2013 00:30:31 +0200 (CEST) Received: by snail.stack.nl (Postfix, from userid 1677) id 8900928493; Fri, 31 May 2013 00:30:31 +0200 (CEST) Date: Fri, 31 May 2013 00:30:31 +0200 From: Jilles Tjoelker To: =?iso-8859-1?Q?V=E1clav?= Zeman Subject: Re: /bin/sh => STDIN & functions, var scope messing Message-ID: <20130530223031.GA1672@stack.nl> References: <20130527.194235.693.1@DOMY-PC> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: User-Agent: Mutt/1.5.21 (2010-09-15) Cc: rank1seeker@gmail.com, Reid Linnemann , hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 30 May 2013 22:30:48 -0000 On Tue, May 28, 2013 at 11:48:47AM +0200, Václav Zeman wrote: > On 27 May 2013 21:58, Reid Linnemann wrote: > > from SH(1) > > "Note that unlike some other shells, sh executes each process in a pipe- > > line with more than one command in a subshell environment and as a > > child > > of the sh process." > > I'm taking this to mean that redirecting to sh_f has sh_f execute in > > a subshell in which global_scope_var changes, but the original > > shell's copy is uncahnged. > Curious. Which of the two behaviours is POSIXly correct? Both. As per XCU 2.12 Shell Execution Environment, each command in a multi-command pipeline may or may not be executed in a subshell environment. Behaviour different from our sh is most often encountered in the various versions of the real Korn shell (ksh88 and ksh93), which execute the last command in a pipeline in the current shell environment. If things like jobs | cat work, that can also be explained using this rule. -- Jilles Tjoelker From owner-freebsd-hackers@FreeBSD.ORG Fri May 31 01:43:30 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2891ADEA for ; Fri, 31 May 2013 01:43:30 +0000 (UTC) (envelope-from nparhar@gmail.com) Received: from mail-pd0-f173.google.com (mail-pd0-f173.google.com [209.85.192.173]) by mx1.freebsd.org (Postfix) with ESMTP id 086CDD6D for ; Fri, 31 May 2013 01:43:29 +0000 (UTC) Received: by mail-pd0-f173.google.com with SMTP id v14so1356837pde.4 for ; Thu, 30 May 2013 18:43:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=date:from:to:subject:message-id:mail-followup-to:mime-version :content-type:content-disposition:user-agent; bh=1h8n2tSnWAWETDTZVnATTJ4prfBGnhdxc0B1+apyo6c=; b=UgOACXagxHyrZg9AZSW3eo7Kvv0E8QcQA+3632Y1tLGyBda7ypaFl2RyYzmvmATZVt /0OR4awY017KVMxZH0Akp5jisSe2cLTbglWic5euPlFcEnyTN6Pv8XbBChewftRwClFL zwYFN2EX47lEGR6r+GMgVujgcmNLZsEbfpNgw57g079Ez5MJMpCiG9B24mxcRxa0kcat ROodXdXO9rf7hYGn36adR33zyBhCj1u4rKwxddsiwgc39s55E2bDRwyG8z3wSG1V+EQT QDM/BQNtFMJ/FaLno5D8oG8SguGQt9MoSUMUNO/pKOpzdqkEz0zkCBWRuCIEkTRs03Do ryig== X-Received: by 10.68.92.165 with SMTP id cn5mr10822979pbb.50.1369964609372; Thu, 30 May 2013 18:43:29 -0700 (PDT) Received: from itx (c-24-6-45-85.hsd1.ca.comcast.net. [24.6.45.85]) by mx.google.com with ESMTPSA id gi2sm44269973pbb.2.2013.05.30.18.43.27 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Thu, 30 May 2013 18:43:28 -0700 (PDT) Date: Thu, 30 May 2013 18:43:20 -0700 From: Navdeep Parhar To: freebsd-hackers@freebsd.org Subject: UNIVERSE_TARGET doesn't seem to work Message-ID: <20130531014320.GA22257@itx> Mail-Followup-To: freebsd-hackers@freebsd.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.20 (2009-06-14) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 01:43:30 -0000 I build kernel-toolchain and MAKE_JUST_KERNELS (often with NO_CLEAN, but not this time) as part of my pre-commit checklist. It doesn't seem to work after the switch to bmake. What am I missing? This on a system at r251171 with nothing in make.conf or src.conf: # make -j12 universe UNIVERSE_TARGET=kernel-toolchain --- universe_prologue --- -------------------------------------------------------------- >>> make universe started on Thu May 30 18:19:44 PDT 2013 -------------------------------------------------------------- `universe_amd64_prologue' was not built (made 0, flags 2009, type b000001)! `universe_arm_prologue' was not built (made 0, flags 2009, type b000001)! `universe_i386_prologue' was not built (made 0, flags 2009, type b000001)! `universe_ia64_prologue' was not built (made 0, flags 2009, type b000001)! `universe_mips_prologue' was not built (made 0, flags 2009, type b000001)! `universe_pc98_prologue' was not built (made 0, flags 2009, type b000001)! `universe_powerpc_prologue' was not built (made 0, flags 2009, type b000001)! `universe_sparc64_prologue' was not built (made 0, flags 2009, type b000001)! `universe_epilogue' was not built (made 1, flags 2009, type b000001)! `universe_epilogue' has .ORDER dependency against universe_amd64 (made 1, flags 3009, type 3000001) `universe_epilogue' has .ORDER dependency against universe_arm (made 1, flags 3009, type 3000001) `universe_epilogue' has .ORDER dependency against universe_i386 (made 1, flags 3009, type 3000001) `universe_epilogue' has .ORDER dependency against universe_ia64 (made 1, flags 3009, type 3000001) `universe_epilogue' has .ORDER dependency against universe_mips (made 1, flags 3009, type 3000001) `universe_epilogue' has .ORDER dependency against universe_pc98 (made 1, flags 3009, type 3000001) `universe_epilogue' has .ORDER dependency against universe_powerpc (made 1, flags 3009, type 3000001) `universe_epilogue' has .ORDER dependency against universe_sparc64 (made 1, flags 3009, type 3000001) # make -j12 -DMAKE_JUST_KERNELS JFLAG=-j12 universe (same result) Regards, Navdeep From owner-freebsd-hackers@FreeBSD.ORG Fri May 31 10:26:56 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5520779A for ; Fri, 31 May 2013 10:26:56 +0000 (UTC) (envelope-from dirkx@webweaving.org) Received: from pikmeer.webweaving.org (pikmeer.webweaving.org [178.18.23.51]) by mx1.freebsd.org (Postfix) with ESMTP id CE801323 for ; Fri, 31 May 2013 10:26:55 +0000 (UTC) Received: from beeb.leiden.webweaving.org (5ED28243.cm-7-3c.dynamic.ziggo.nl [94.210.130.67]) (authenticated bits=0) by pikmeer.webweaving.org (8.14.5/8.14.5) with ESMTP id r4VA12Lq079007 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO) for ; Fri, 31 May 2013 10:01:02 GMT (envelope-from dirkx@webweaving.org) From: Dirk-Willem van Gulik Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Subject: seeding randomness in zee cloud Message-Id: <0BF6FBDD-47E8-44F1-BA71-A355EDCDEDB6@webweaving.org> Date: Fri, 31 May 2013 12:01:02 +0200 To: freebsd-hackers@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) X-Mailer: Apple Mail (2.1503) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.5 (pikmeer.webweaving.org [178.18.23.51]); Fri, 31 May 2013 10:01:02 +0000 (UTC) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 10:26:56 -0000 Thanks to a badly-written mngt script - we've rencently noticed a = freshly generated ssh-key on a new AWS instances to be indentical to one = seen a few months prior.=20 Careful analysis of some other logs showed that we've had similar = clashes on another script just after startup generating a very short = x509 CSR. It happens quite rarely though. But still. I am surmising that perhaps the (micro-T) images do not have that much = entropy on startup. So I am wondering how to best make our images 'more random' -- and want = to avoid the linux/openstack suggestion[1] of doing this through the = boot-params [2] (as in our case it is the operator of the machine we're protecting/guarding against = accusations/temptations). Now we happen to have very easy access to blocks of 1024bits of = randomness from a remote server in already nicely PKI signed packages = (as it is needed later for something else). Is it safe to simply *add* those with: set -1 # fetch randomness & check signature .. snipped... # Seed Software random generator # cat rnd > /dev/random # Activate software random generator as an additional source sysctl kern.random.sys.harvest.swi=3D1 Or does this cause a loss/reset of all entropy gathered by the hardware = sofar ? Or is there a cleaner way to add a additional seed as a one-off = with disturbing as little as possible (in the few seconds just after the = network is brought up). =09 Thanks, Dw. FWIIW: this is the output of sysctl kern.random. kern.random.yarrow.gengateinterval: 10 kern.random.yarrow.bins: 10 kern.random.yarrow.fastthresh: 192 kern.random.yarrow.slowthresh: 256 kern.random.yarrow.slowoverthresh: 2 kern.random.sys.seeded: 1 kern.random.sys.harvest.ethernet: 1 kern.random.sys.harvest.point_to_point: 1 kern.random.sys.harvest.interrupt: 1 kern.random.sys.harvest.swi: 0 1: = http://blog.dustinkirkland.com/2012/10/entropy-or-lack-thereof-in-openstac= k.html 2: https://review.openstack.org/#/c/14550/= From owner-freebsd-hackers@FreeBSD.ORG Fri May 31 12:02:46 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id F28EC2F7 for ; Fri, 31 May 2013 12:02:46 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: from mail-ea0-x22b.google.com (mail-ea0-x22b.google.com [IPv6:2a00:1450:4013:c01::22b]) by mx1.freebsd.org (Postfix) with ESMTP id 8BC44B1B for ; Fri, 31 May 2013 12:02:46 +0000 (UTC) Received: by mail-ea0-f171.google.com with SMTP id b15so1513347eae.2 for ; Fri, 31 May 2013 05:02:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=date:from:to:subject:message-id:in-reply-to:references:x-mailer :mime-version:content-type:content-transfer-encoding; bh=qlTC3ZuSc2K05suRZ3soUyDzgYHNgETj9LvMkK9fbvY=; b=gaTk/tTrvLHSp4pwNJ9pEIXjvkhmkxTJJMxjQXbaUP5LJmMocCdbH9ZWGN7LaeVmbM RjYp6m89+gb8Mgbq+YMTH2kUTkBq0YK6PLqfo/0LJz2gxmoQqo4ojwX+/VnfGVYLmjpe fu4VUw4bhcGwPXjTSfIozFpoNT35dZmt8GZ33YRHttQhCI4TMUbPh1TVKfxJ6PjbFuv9 kjenQFFXHrfnU8hZMqXnjG3aRdxyJI3LzZoWMacK3yyss16LXTNSnGPAc4/FqSGIjlWz GlRQzP8dxC4TXjFzQPs2U3iLCYWOja97BRZCLD0SkpfTMyeQq500yTjbc2ppz/Kt7Rlj sI1Q== X-Received: by 10.15.108.141 with SMTP id cd13mr13398196eeb.46.1370001765661; Fri, 31 May 2013 05:02:45 -0700 (PDT) Received: from gumby.homeunix.com (87-194-105-247.bethere.co.uk. [87.194.105.247]) by mx.google.com with ESMTPSA id s43sm66550925eem.13.2013.05.31.05.02.44 for (version=SSLv3 cipher=RC4-SHA bits=128/128); Fri, 31 May 2013 05:02:45 -0700 (PDT) Date: Fri, 31 May 2013 13:02:43 +0100 From: RW To: freebsd-hackers@freebsd.org Subject: Re: seeding randomness in zee cloud Message-ID: <20130531130243.18fb9a30@gumby.homeunix.com> In-Reply-To: <0BF6FBDD-47E8-44F1-BA71-A355EDCDEDB6@webweaving.org> References: <0BF6FBDD-47E8-44F1-BA71-A355EDCDEDB6@webweaving.org> X-Mailer: Claws Mail 3.9.0 (GTK+ 2.24.17; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 12:02:47 -0000 On Fri, 31 May 2013 12:01:02 +0200 Dirk-Willem van Gulik wrote: > Now we happen to have very easy access to blocks of 1024bits of > randomness from a remote server in already nicely PKI signed packages > (as it is needed later for something else). > > Is it safe to simply *add* those with: > > set -1 > # fetch randomness & check signature > .. snipped... > > # Seed Software random generator > # > cat rnd > /dev/random To be on the safe side you should sleep for about 0.5 seconds after this > > # Activate software random generator as an additional source > sysctl kern.random.sys.harvest.swi=1 IIRC this doesn't do anything From owner-freebsd-hackers@FreeBSD.ORG Fri May 31 12:26:41 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A1FBA7B2 for ; Fri, 31 May 2013 12:26:41 +0000 (UTC) (envelope-from dirkx@webweaving.org) Received: from pikmeer.webweaving.org (pikmeer.webweaving.org [178.18.23.51]) by mx1.freebsd.org (Postfix) with ESMTP id 3EBC0CEE for ; Fri, 31 May 2013 12:26:40 +0000 (UTC) Received: from beeb.leiden.webweaving.org (5ED28243.cm-7-3c.dynamic.ziggo.nl [94.210.130.67]) (authenticated bits=0) by pikmeer.webweaving.org (8.14.5/8.14.5) with ESMTP id r4VCQdv9081297 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NO); Fri, 31 May 2013 12:26:39 GMT (envelope-from dirkx@webweaving.org) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.3 \(1503\)) Subject: Re: seeding randomness in zee cloud From: Dirk-Willem van Gulik In-Reply-To: <20130531130243.18fb9a30@gumby.homeunix.com> Date: Fri, 31 May 2013 14:26:39 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <0BF6FBDD-47E8-44F1-BA71-A355EDCDEDB6@webweaving.org> <20130531130243.18fb9a30@gumby.homeunix.com> To: RW X-Mailer: Apple Mail (2.1503) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.5 (pikmeer.webweaving.org [178.18.23.51]); Fri, 31 May 2013 12:26:40 +0000 (UTC) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 12:26:41 -0000 Op 31 mei 2013, om 14:02 heeft RW het = volgende geschreven: > On Fri, 31 May 2013 12:01:02 +0200 Dirk-Willem van Gulik wrote: >> # Seed Software random generator >> # >> cat rnd > /dev/random >=20 > To be on the safe side you should sleep for about 0.5 seconds after > this=20 >=20 >>=20 >> # Activate software random generator as an additional source >> sysctl kern.random.sys.harvest.swi=3D1 >=20 > IIRC this doesn't do anything Thanks. So the man page says: The kern.random.sys.harvest.swi variable is used to select software interrupts as an entropy source. A 0 (zero) value means software = inter- rupts are not considered as an entropy source. Set the variable to = 1 (one) if you wish to use them for entropy harvesting. but it is fair to assume that even when it is set to '0' (the default = observerd on 9.1-RELEASE) - that the randomness sent to /dev/random is = still mixed in ? Thanks, Dw. From owner-freebsd-hackers@FreeBSD.ORG Fri May 31 13:45:53 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2AD2FE5 for ; Fri, 31 May 2013 13:45:53 +0000 (UTC) (envelope-from rwmaillists@googlemail.com) Received: from mail-wg0-x229.google.com (mail-wg0-x229.google.com [IPv6:2a00:1450:400c:c00::229]) by mx1.freebsd.org (Postfix) with ESMTP id B7FC22F5 for ; Fri, 31 May 2013 13:45:52 +0000 (UTC) Received: by mail-wg0-f41.google.com with SMTP id k13so732594wgh.4 for ; Fri, 31 May 2013 06:45:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=googlemail.com; s=20120113; h=date:from:to:subject:message-id:in-reply-to:references:x-mailer :mime-version:content-type:content-transfer-encoding; bh=Kf7hPZfGk1FatimGSgoVO5EITHiTuPCyZYl7lVoJkaY=; b=pKSjm5O2bL+cxXl7i7lWJsGICSvl5L4dxE7hSHZy7z9nWwM5yQwYbXcxnewpMEM5jU n6VtJJ/UAcMK3/X3aVByNOXk2oZbfH6cz+I16XI0+E5etwPgL4GA8kOoyoXFLCIlRS13 1umaFh3nepcQgHv/ROoKdYzrxdE74RTTvEZH18Fmq41m+cdhewPPa6zbKPFefpqQYxSV oW+o1bQ2H9aAos4DNRmbhhYxKpNtTg8vgR2AfK/G89SeuSPDXqAYwgDiYdFdFbcqPutI IWBbOh1dEhPhEKRhm+adyr/eiW4YpGJ65NfeM2Rcg8tRAHBNVYhrnOkUnjCBuYqkB90Y OLPA== X-Received: by 10.180.9.80 with SMTP id x16mr3274997wia.63.1370007951913; Fri, 31 May 2013 06:45:51 -0700 (PDT) Received: from gumby.homeunix.com (87-194-105-247.bethere.co.uk. [87.194.105.247]) by mx.google.com with ESMTPSA id fv11sm4043799wic.11.2013.05.31.06.45.50 for (version=SSLv3 cipher=RC4-SHA bits=128/128); Fri, 31 May 2013 06:45:51 -0700 (PDT) Date: Fri, 31 May 2013 14:45:49 +0100 From: RW To: freebsd-hackers@freebsd.org Subject: Re: seeding randomness in zee cloud Message-ID: <20130531144549.1193d3c4@gumby.homeunix.com> In-Reply-To: References: <0BF6FBDD-47E8-44F1-BA71-A355EDCDEDB6@webweaving.org> <20130531130243.18fb9a30@gumby.homeunix.com> X-Mailer: Claws Mail 3.9.0 (GTK+ 2.24.17; amd64-portbld-freebsd10.0) Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 13:45:53 -0000 On Fri, 31 May 2013 14:26:39 +0200 Dirk-Willem van Gulik wrote: > > Op 31 mei 2013, om 14:02 heeft RW het > >> # Activate software random generator as an additional > >> source sysctl kern.random.sys.harvest.swi=1 > > > > IIRC this doesn't do anything > > Thanks. So the man page says: > > The kern.random.sys.harvest.swi variable is used to select > software interrupts as an entropy source. A 0 (zero) value means > software inter- rupts are not considered as an entropy source. Set > the variable to 1 (one) if you wish to use them for entropy > harvesting. I don't think it ever got implemented, but for some reason the sysctl got left in. All it would have done is turn-on an additional entropy source. > but it is fair to assume that even when it is set to '0' (the default > observerd on 9.1-RELEASE) - that the randomness sent to /dev/random > is still mixed in ? Yes, if you are using the software generator then it's used. If you have direct hardware support you wont see the harvest sysctls and the input is harmlessly discarded. Most Ivy Bridge and newer AMD processors have RdRand these days. From owner-freebsd-hackers@FreeBSD.ORG Fri May 31 13:59:55 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 3E06B43D for ; Fri, 31 May 2013 13:59:55 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [188.252.31.196]) by mx1.freebsd.org (Postfix) with ESMTP id BBD583A7 for ; Fri, 31 May 2013 13:59:54 +0000 (UTC) Received: from wojtek.tensor.gdynia.pl (localhost [127.0.0.1]) by wojtek.tensor.gdynia.pl (8.14.7/8.14.6) with ESMTP id r4VDxc9T008858; Fri, 31 May 2013 15:59:38 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.6/8.14.6/Submit) with ESMTP id r4VDxcsH008855; Fri, 31 May 2013 15:59:38 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Fri, 31 May 2013 15:59:38 +0200 (CEST) From: Wojciech Puchar To: Dirk-Willem van Gulik Subject: Re: seeding randomness in zee cloud In-Reply-To: <0BF6FBDD-47E8-44F1-BA71-A355EDCDEDB6@webweaving.org> Message-ID: References: <0BF6FBDD-47E8-44F1-BA71-A355EDCDEDB6@webweaving.org> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (wojtek.tensor.gdynia.pl [127.0.0.1]); Fri, 31 May 2013 15:59:38 +0200 (CEST) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 13:59:55 -0000 > .. snipped... > > # Seed Software random generator > # > cat rnd > /dev/random > > # Activate software random generator as an additional source > sysctl kern.random.sys.harvest.swi=1 > > Or does this cause a loss/reset of all entropy gathered by the hardware sofar ? no - writing to /dev/random "adds randomness" From owner-freebsd-hackers@FreeBSD.ORG Fri May 31 14:02:30 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 283DC589 for ; Fri, 31 May 2013 14:02:30 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-ie0-x232.google.com (mail-ie0-x232.google.com [IPv6:2607:f8b0:4001:c03::232]) by mx1.freebsd.org (Postfix) with ESMTP id 037CF5F4 for ; Fri, 31 May 2013 14:02:29 +0000 (UTC) Received: by mail-ie0-f178.google.com with SMTP id f4so3930805iea.23 for ; Fri, 31 May 2013 07:02:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to:content-type; bh=ztOGiTh1PVlCdo77I1KqQrJajSFYPdUtFqBzIcgQLMI=; b=pob5lmN+Mr4I6DbRbtbVS6T41lmPK0dtPy2fa3cr4y1hoKIKysUigtoFjXXGTLJHlH zmWVX4uMuYOvOpKKu80+xsp1BRpB3/hEWrhSAT4kzr2YmTmEJ/rST5P3kSk4Fl4rFOvc EIBxmD9A7EdJUjOZMZPHWucejoDoQ6F98Ci0ODGXDqIt1R5f5xj3DpLUau7wCOUP51Ps sOgTUlSOLbQGhKWZ0ItuTKTcBnqcfZeYcvUnVBbD//mllsn0xUC8Q0BmH+WiTULipo/I /BQHeDElbxd5oaNeWRqwQvNvU1HQwOWJbCCYhKGKH550VLoQI14tXa5KxYdpw/+LkuLG 1s8A== X-Received: by 10.43.106.202 with SMTP id dv10mr5047561icc.37.1370008949717; Fri, 31 May 2013 07:02:29 -0700 (PDT) MIME-Version: 1.0 Received: by 10.64.71.101 with HTTP; Fri, 31 May 2013 07:01:59 -0700 (PDT) From: Chris Rees Date: Fri, 31 May 2013 15:01:59 +0100 Message-ID: Subject: sed query To: "freebsd-hackers@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 14:02:30 -0000 Hi all, I think I've discovered a strange behaviour of sed perhaps triggered by the length of a regex passed to it. I noticed that a certain expression I passed took a very long time, and suspected the usual backtracking loop, so I started trimming it... and discovered this: [crees@pegasus]~% time sed -ne "s,^BitchX-[0-9][^|]*[\|]/usr/por,," /var/db/pkg/INDEX-9 4.699u 0.007s 0:04.70 99.7% 40+2733k 0+0io 0pf+0w [crees@pegasus]~% time sed -ne "s,^BitchX-[0-9][^|]*[\|]/usr/po,," /var/db/pkg/INDEX-9 0.042u 0.000s 0:00.04 100.0% 48+3216k 0+0io 0pf+0w I've looked at the code, and can't from a brief glance figure out why a slightly longer regex makes such a difference-- does it start to split it? Chris From owner-freebsd-hackers@FreeBSD.ORG Fri May 31 14:10:43 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1173B9F6 for ; Fri, 31 May 2013 14:10:43 +0000 (UTC) (envelope-from paul-freebsd@fletchermoorland.co.uk) Received: from hydra.fletchermoorland.co.uk (78-33-207-170.static.enta.net [78.33.207.170]) by mx1.freebsd.org (Postfix) with ESMTP id 5F796689 for ; Fri, 31 May 2013 14:10:41 +0000 (UTC) Received: from demophon.fletchermoorland.co.uk ([192.168.0.112]) by hydra.fletchermoorland.co.uk (8.14.3/8.14.3) with ESMTP id r4VE6YNS038257; Fri, 31 May 2013 15:06:34 +0100 (BST) (envelope-from paul-freebsd@fletchermoorland.co.uk) Message-ID: <51A8AE6A.5000300@fletchermoorland.co.uk> Date: Fri, 31 May 2013 15:06:34 +0100 From: Paul Wootton User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:12.0) Gecko/20120530 Thunderbird/12.0.1 MIME-Version: 1.0 To: "Welcome, Traiano" Subject: Re: Writing a (BSD like) Operating Systems From Scratch References: <8F56C8EF8265DF489B64A19B10910AC7025C53B2@ex10-mbx-14001.ant.amazon.com> <8F56C8EF8265DF489B64A19B10910AC7025E4E12@ex10-mbx-14001.ant.amazon.com> In-Reply-To: <8F56C8EF8265DF489B64A19B10910AC7025E4E12@ex10-mbx-14001.ant.amazon.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.9 required=7.0 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on hydra.fletchermoorland.co.uk Cc: "freebsd-hackers@freebsd.org" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 14:10:43 -0000 On 05/25/13 06:44, Welcome, Traiano wrote: > a) What kind of hardware (processor) would I use as a development >> platform, given the requirements of cheap, well documented, easily >> obtainable, easy to debug etc ... I believe the hardware platform chosen >> should satisfy the following requirements: >> >> any except PCs unless you will like to deal with CPU and other >> (over)complexity. > Exactly my thinking. Most of the online links to operating system development involve x86 hardware, although more and more Microcontrollers are appearing for embedded market with features that previously only existed in mainstream microprocessors. Ideally, the platform I'd choose would have a small enough instruction set to learn (small relative to Intel's mainstream processors), maybe something like the ARM processor used on Raspberry Pi, or Zilog's ez80 Acclaim series. In that case, why not look at some of the Microchip's PIC32 development suites. There are quite a few options from Microchip, their dev hardware is not that expensive and could prove as a good starting point Looking at their site, dev kit DM320015 has a microcontroller, 4.3" WQVGA screen, capacitive touch screen and USB OTG all built on one unit Or maybe DM320004 + DM320002 + other modules would give a nice platform (microcontroller, serial, ethernet, SD socket, wireless interface etc) (as a side note) I have the DM320004 (uC board), DM320002 (backplane), AC164122 (SD card adapter), AC164129 (Audio adapter) and AC164126 (breadboards) They have loads of source code available, so you could use that as a starting point, then go on to writing everything your self, including a kernel with a full GUI and touchscreen control if you so wish. Or, why not try a MaxiMite unit (http://geoffg.net/maximite.html). The schematics are available on the website and the PCBs should be available (with all combinations from blank PCB to fully populated) But remember, there are a thousand ways to skin a cat and this is just one... Paul From owner-freebsd-hackers@FreeBSD.ORG Fri May 31 18:00:00 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D9C53797 for ; Fri, 31 May 2013 18:00:00 +0000 (UTC) (envelope-from rank1seeker@gmail.com) Received: from mail-bk0-x235.google.com (mail-bk0-x235.google.com [IPv6:2a00:1450:4008:c01::235]) by mx1.freebsd.org (Postfix) with ESMTP id 6FF1E2F3 for ; Fri, 31 May 2013 18:00:00 +0000 (UTC) Received: by mail-bk0-f53.google.com with SMTP id mx10so892101bkb.40 for ; Fri, 31 May 2013 10:59:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:from:to:subject:date:content-type :content-transfer-encoding:in-reply-to:references:x-mailer; bh=6nwhI8PWqLRuwfiBkFKCaCBcXTD1W7639GGoEafjBzg=; b=xindJZy0OQlk1AisEWXfowPItjHEoYw0YBtTE333IVT5cKzGwtQ4q2ScUnFHKbN4VH /rTV/X0PSIoWIpBO7PijUcb4VTB5ticUDORMciVEwIwqnTts+vsSZADbAE2evZvs2BA2 kqLPDbZc7HpVov+F+oDn5mzEIAXAdqQp7MsVnTFj0di96NkwvhLDaCHCSb1bWcrWqLWE OgQhNWQRakz2LZbBnj+uQtO8xiRkdKQAQ7oU8vpIeUOVWVio68wbgZ9Sd4bmAzxMFc7Z xrkNuPGh1Vrgx72B31iPrV13pL00ppIhc2jU2u9EwP0Rs0n8INpna3vfwsZULp5gjdxD 3p9w== X-Received: by 10.204.187.209 with SMTP id cx17mr3745277bkb.143.1370023199497; Fri, 31 May 2013 10:59:59 -0700 (PDT) Received: from DOMYPC ([82.193.208.225]) by mx.google.com with ESMTPSA id tl1sm15598246bkb.7.2013.05.31.10.59.57 for (version=TLSv1 cipher=RC4-SHA bits=128/128); Fri, 31 May 2013 10:59:58 -0700 (PDT) Message-ID: <20130531.175959.745.2@DOMY-PC> From: rank1seeker@gmail.com To: "Jilles Tjoelker" , hackers@freebsd.org Subject: Re: /bin/sh => STDIN & functions, var scope messing Date: Fri, 31 May 2013 19:59:59 +0200 Content-Type: text/plain; charset="Windows-1250" Content-Transfer-Encoding: quoted-printable In-Reply-To: <20130530223031.GA1672@stack.nl> References: <20130527.194235.693.1@DOMY-PC> <20130530223031.GA1672@stack.nl> X-Mailer: POP Peeper (3.8.1.0) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 18:00:00 -0000 Redirections > and >> don't put it in a subshell.=0D=0AOnly pipe |, which = means only STDIN affects/triggers this behaviour.=0D=0ADoes < operator = also does it, as it is also STDIN?=0D=0A=0D=0AAnyway, I don't care for = executing binaries, but I do care if that is part of sh's code, as = function is.=0D=0AIt messes var scopes.=0D=0A=0D=0AAnd finally if we take = into account this:=0D=0A=0D=0A> For example=85 in pc-sysinstall=85=0D=0A> = =0D=0A> ./backend/functions-extractimage.sh- tar cvf - . 2>/dev/null = | tar -xpv -C ${FSMNT} ${TAROPTS} -f - 2>&1 | tee -a = ${FSMNT}/.tar-extract.log=0D=0A> ./backend/functions-extractimage.sh: = if [ $? -ne 0 ]=0D=0A> =0D=0A> That's a big no-no.=0D=0A> =0D=0A> If = your first tar (the initial lvalue to the first pipe) fails=85 but your = second tar succeeds (rvalue to the first pipe)=85 you won't catch that = failure in your checking of $?=0D=0A> =0D=0A> Also, if the first tar = succeeds, but the second tar fails, AND the final rvalue (the tee) = succeeds=85 you also miss the error code.=0D=0A> =0D=0A> I call this the = "piped return-status conflation issue". Basically=85 anytime you want to = check the return-status in shell=85 and you care about lvalue-failures in = a pipe-chain=85 you must rewrite it to either:=0D=0A> =0D=0A> (a) be = aware of the problem (I've in the past written wrappers that will test = the previous return status and abort the chain in such an event)=0D=0A> = =0D=0A> (b) undo the pipe-chain and store your results for incremental = processing=85 checking the return status after each filter.=0D=0A> = =0D=0A> -- =0D=0A> Devin=0D=0A=0D=0A=0D=0Ash's behaviour must be changed = regarding pipeing=0D=0A=0D=0A> > Curious. Which of the two behaviours is = POSIXly correct?=0D=0A> =0D=0A> Both. As per XCU 2.12 Shell Execution = Environment, each command in a=0D=0A> multi-command pipeline may or may = not be executed in a subshell=0D=0A> environment.=0D=0A> =0D=0A> = Behaviour different from our sh is most often encountered in the = various=0D=0A> versions of the real Korn shell (ksh88 and ksh93), which = execute the=0D=0A> last command in a pipeline in the current shell = environment.=0D=0A> =0D=0A> If things like jobs | cat work, that can = also be explained using this=0D=0A> rule.=0D=0A> =0D=0A> -- =0D=0A> = Jilles Tjoelker=0D=0A>=0D=0A=0D=0A=0D=0ADomagoj Smol=E8i=E6=0D=0A From owner-freebsd-hackers@FreeBSD.ORG Fri May 31 18:12:06 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id AD8DBB58; Fri, 31 May 2013 18:12:06 +0000 (UTC) (envelope-from Devin.Teske@fisglobal.com) Received: from mx1.fisglobal.com (mx1.fisglobal.com [199.200.24.190]) by mx1.freebsd.org (Postfix) with ESMTP id 7A4D03E7; Fri, 31 May 2013 18:12:06 +0000 (UTC) Received: from smtp.fisglobal.com ([10.132.206.17]) by ltcfislmsgpa01.fnfis.com (8.14.5/8.14.5) with ESMTP id r4VIC3Yw028168 (version=TLSv1/SSLv3 cipher=AES128-SHA bits=128 verify=NOT); Fri, 31 May 2013 13:12:03 -0500 Received: from LTCFISWMSGMB21.FNFIS.com ([10.132.99.23]) by LTCFISWMSGHT06.FNFIS.com ([10.132.206.17]) with mapi id 14.02.0309.002; Fri, 31 May 2013 13:12:03 -0500 From: "Teske, Devin" To: " " Subject: Re: /bin/sh => STDIN & functions, var scope messing Thread-Topic: /bin/sh => STDIN & functions, var scope messing Thread-Index: AQHOWxJjvvyJSeBFUUSS2KYxPIq87pkZxr8AgADoBoCAA/l9gIABRr+AgAADXgA= Date: Fri, 31 May 2013 18:12:03 +0000 Message-ID: <13CA24D6AB415D428143D44749F57D7201F6AD11@ltcfiswmsgmb21> References: <20130527.194235.693.1@DOMY-PC> <20130530223031.GA1672@stack.nl> <20130531.175959.745.2@DOMY-PC> In-Reply-To: <20130531.175959.745.2@DOMY-PC> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.132.253.126] Content-Type: text/plain; charset="windows-1250" Content-ID: <6DC94DA7C65DE545A69C9962A2FCEC08@fisglobal.com> Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:5.10.8626, 1.0.431, 0.0.0000 definitions=2013-05-31_08:2013-05-31,2013-05-31,1970-01-01 signatures=0 Cc: Devin Teske , FreeBSD Hackers , Jilles Tjoelker X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: Devin Teske List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 18:12:06 -0000 On May 31, 2013, at 10:59 AM, wrote: > Redirections > and >> don't put it in a subshell. Correct. (note: I made no such insinuation; But thanks for clarifying for o= thers that perhaps were not aware). > Only pipe |, which means only STDIN affects/triggers this behaviour. > Does < operator also does it, as it is also STDIN? >=20 > Anyway, I don't care for executing binaries, but I do care if that is par= t of sh's code, as function is. > It messes var scopes. >=20 > And finally if we take into account this: >=20 >> For example=85 in pc-sysinstall=85 >>=20 >> ./backend/functions-extractimage.sh- tar cvf - . 2>/dev/null | tar = -xpv -C ${FSMNT} ${TAROPTS} -f - 2>&1 | tee -a ${FSMNT}/.tar-extract.log >> ./backend/functions-extractimage.sh: if [ $? -ne 0 ] >>=20 >> That's a big no-no. >>=20 >> If your first tar (the initial lvalue to the first pipe) fails=85 but yo= ur second tar succeeds (rvalue to the first pipe)=85 you won't catch that f= ailure in your checking of $? >>=20 >> Also, if the first tar succeeds, but the second tar fails, AND the final= rvalue (the tee) succeeds=85 you also miss the error code. >>=20 >> I call this the "piped return-status conflation issue". Basically=85 any= time you want to check the return-status in shell=85 and you care about lva= lue-failures in a pipe-chain=85 you must rewrite it to either: >>=20 >> (a) be aware of the problem (I've in the past written wrappers that will= test the previous return status and abort the chain in such an event) >>=20 >> (b) undo the pipe-chain and store your results for incremental processin= g=85 checking the return status after each filter. >>=20 >> --=20 >> Devin >=20 >=20 > sh's behaviour must be changed regarding piping >=20 If you're arguing we have to change sh's behavior to be more compliant, jil= les already quoted XCU 2.12 (our shell is well within its right to run any/= all lvalue/rvalue operands of a pipe in a sub-shell without contradicting t= he guidelines). But if you're arguing that it has to change to make things better or easier= =85 I don't know about that. Might just make people lulled into using a sty= le that's non-portable. I'd like to keep things the way they are so that if= you program for FreeBSD, you're inherently going to program in a fashion t= hat is more portable. --=20 Devin >>> Curious. Which of the two behaviours is POSIXly correct? >>=20 >> Both. As per XCU 2.12 Shell Execution Environment, each command in a >> multi-command pipeline may or may not be executed in a subshell >> environment. >>=20 >> Behaviour different from our sh is most often encountered in the various >> versions of the real Korn shell (ksh88 and ksh93), which execute the >> last command in a pipeline in the current shell environment. >>=20 >> If things like jobs | cat work, that can also be explained using this >> rule. >>=20 >> --=20 >> Jilles Tjoelker >>=20 >=20 >=20 > Domagoj Smol=E8i=E6 >=20 > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" _____________ The information contained in this message is proprietary and/or confidentia= l. If you are not the intended recipient, please: (i) delete the message an= d all copies; (ii) do not disclose, distribute or use the message in any ma= nner; and (iii) notify the sender immediately. In addition, please be aware= that any message addressed to our domain is subject to archiving and revie= w by persons other than the intended recipient. Thank you. From owner-freebsd-hackers@FreeBSD.ORG Fri May 31 18:49:22 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 4234D40E for ; Fri, 31 May 2013 18:49:22 +0000 (UTC) (envelope-from linnemannr@gmail.com) Received: from mail-ob0-x235.google.com (mail-ob0-x235.google.com [IPv6:2607:f8b0:4003:c01::235]) by mx1.freebsd.org (Postfix) with ESMTP id 0F1A2780 for ; Fri, 31 May 2013 18:49:22 +0000 (UTC) Received: by mail-ob0-f181.google.com with SMTP id 16so3612964obc.40 for ; Fri, 31 May 2013 11:49:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=0L1jDwDLlCplsXEwbeAb8UqUAAwJYbLjYZU/FOoSaWw=; b=TLXYZdSpW7xfWxeywgeWioPZmr7uwJ2t6PwrQKBuVlVWLrajoUvbhEieTYSwgCC9Yh 28BOfKKAD8z0SueCdV88AZp91YB8MBeX3XiuLkaPvIZFtjbrjNnVtO4EAfHE8Mx/+xNA kxjNkz+UpaJPurguCIFu1QI2ikudsSjSNjqmIJYkjnD72GiksdKbQhnG8CkURohdfjyk /F0ZIrskd61FqdDO+HjlKRBh4sw9NDljSPdtUSbUyP3n2GgS+/YaaBmXtHutEUJyqrqP 13UoFC8yQwLXDFfjTZWqq+QJawirAqTCZHz8ZqbrlImJpjaJhrrHMVqwGcHxD0zb7cXg 3mZg== MIME-Version: 1.0 X-Received: by 10.60.33.102 with SMTP id q6mr6549242oei.111.1370026161717; Fri, 31 May 2013 11:49:21 -0700 (PDT) Received: by 10.182.217.98 with HTTP; Fri, 31 May 2013 11:49:21 -0700 (PDT) In-Reply-To: <13CA24D6AB415D428143D44749F57D7201F6AD11@ltcfiswmsgmb21> References: <20130527.194235.693.1@DOMY-PC> <20130530223031.GA1672@stack.nl> <20130531.175959.745.2@DOMY-PC> <13CA24D6AB415D428143D44749F57D7201F6AD11@ltcfiswmsgmb21> Date: Fri, 31 May 2013 13:49:21 -0500 Message-ID: Subject: Re: /bin/sh => STDIN & functions, var scope messing From: Reid Linnemann To: FreeBSD Hackers Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: "" , Jilles Tjoelker X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 18:49:22 -0000 On Fri, May 31, 2013 at 1:12 PM, Teske, Devin wr= ote: > > If you're arguing we have to change sh's behavior to be more compliant, > jilles already quoted XCU 2.12 (our shell is well within its right to run > any/all lvalue/rvalue operands of a pipe in a sub-shell without > contradicting the guidelines). > > But if you're arguing that it has to change to make things better or > easier=85 I don't know about that. Might just make people lulled into usi= ng a > style that's non-portable. I'd like to keep things the way they are so th= at > if you program for FreeBSD, you're inherently going to program in a fashi= on > that is more portable. > -- > Devin > FWIW bash (invoked as sh) on RHEL-based linux systems exhibits the same behavior- sh-3.2$ var=3D1 sh-3.2$ yes|var=3D2 sh-3.2$ echo $var 1 sh-3.2$ If my opinion matters at all, I would agree that for the sake of portability that behavior be consistent with the majority of sh implementations rather than "right" according to arbitrary ruling. From owner-freebsd-hackers@FreeBSD.ORG Fri May 31 19:50:01 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5E1A2FA4 for ; Fri, 31 May 2013 19:50:01 +0000 (UTC) (envelope-from dan@dan.emsphone.com) Received: from email2.allantgroup.com (email2.emsphone.com [199.67.51.116]) by mx1.freebsd.org (Postfix) with ESMTP id 0C932975 for ; Fri, 31 May 2013 19:50:00 +0000 (UTC) Received: from dan.emsphone.com (dan.emsphone.com [172.17.17.101]) by email2.allantgroup.com (8.14.5/8.14.5) with ESMTP id r4VJl4Np064343 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 31 May 2013 14:47:04 -0500 (CDT) (envelope-from dan@dan.emsphone.com) Received: from dan.emsphone.com (smmsp@localhost [127.0.0.1]) by dan.emsphone.com (8.14.7/8.14.6) with ESMTP id r4VJl3ea015331 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 31 May 2013 14:47:03 -0500 (CDT) (envelope-from dan@dan.emsphone.com) Received: (from dan@localhost) by dan.emsphone.com (8.14.7/8.14.6/Submit) id r4VJl3tt015330; Fri, 31 May 2013 14:47:03 -0500 (CDT) (envelope-from dan) Date: Fri, 31 May 2013 14:47:03 -0500 From: Dan Nelson To: Reid Linnemann Subject: Re: /bin/sh => STDIN & functions, var scope messing Message-ID: <20130531194703.GC5410@dan.emsphone.com> References: <20130527.194235.693.1@DOMY-PC> <20130530223031.GA1672@stack.nl> <20130531.175959.745.2@DOMY-PC> <13CA24D6AB415D428143D44749F57D7201F6AD11@ltcfiswmsgmb21> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-OS: FreeBSD 9.1-STABLE User-Agent: Mutt/1.5.21 (2010-09-15) X-Virus-Scanned: clamav-milter 0.97.8 at email2.allantgroup.com X-Virus-Status: Clean X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (email2.allantgroup.com [172.17.19.78]); Fri, 31 May 2013 14:47:04 -0500 (CDT) X-Spam-Status: No, score=-3.9 required=5.0 tests=ALL_TRUSTED,BAYES_00, RP_MATCHES_RCVD autolearn=ham version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on email2.allantgroup.com X-Scanned-By: MIMEDefang 2.73 Cc: "" , FreeBSD Hackers , Jilles Tjoelker X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 19:50:01 -0000 In the last episode (May 31), Reid Linnemann said: > On Fri, May 31, 2013 at 1:12 PM, Teske, Devin wrote: > > If you're arguing we have to change sh's behavior to be more compliant, > > jilles already quoted XCU 2.12 (our shell is well within its right to > > run any/all lvalue/rvalue operands of a pipe in a sub-shell without > > contradicting the guidelines). > > > > But if you're arguing that it has to change to make things better or > > easier... I don't know about that. Might just make people lulled into > > using a style that's non-portable. I'd like to keep things the way they > > are so that if you program for FreeBSD, you're inherently going to > > program in a fashion that is more portable. > > FWIW bash (invoked as sh) on RHEL-based linux systems exhibits the same > behavior- > > sh-3.2$ var=1 > sh-3.2$ yes|var=2 > sh-3.2$ echo $var > 1 > sh-3.2$ > > If my opinion matters at all, I would agree that for the sake of > portability that behavior be consistent with the majority of sh > implementations rather than "right" according to arbitrary ruling. On the other hand, zsh runs the last component of a pipeline in the parent shell. The usual model is "do work in pipeline, process results in final component", and being able to simply set variables there that can be used in the rest of the script is very elegant. -- Dan Nelson dnelson@allantgroup.com From owner-freebsd-hackers@FreeBSD.ORG Fri May 31 20:05:53 2013 Return-Path: Delivered-To: hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id DD7FD580 for ; Fri, 31 May 2013 20:05:53 +0000 (UTC) (envelope-from utisoft@gmail.com) Received: from mail-ie0-x232.google.com (mail-ie0-x232.google.com [IPv6:2607:f8b0:4001:c03::232]) by mx1.freebsd.org (Postfix) with ESMTP id AFF42A27 for ; Fri, 31 May 2013 20:05:53 +0000 (UTC) Received: by mail-ie0-f178.google.com with SMTP id f4so5123245iea.37 for ; Fri, 31 May 2013 13:05:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=0A/Uc9ikeXJu3fv2TB0IaVKs1hc3i0Cjtnu6Hww//G8=; b=oPjpOKpk2TVja1Unk0kobYR1hlkpCy9+Y3AHjyYkdzt7xqbEBK6TI8XwoJqwYXGdMg irVkH0RxQg6XkxFTSYXP/qKZNSUuSz9msM61RPHgOLOiE0AQHuzlIq9VFIW5p7mQ6v4+ I0HDTotcb6Vc8ZyhMeF7RQsIFXOwv3CCD6XuzbDJdXa/Xn3MXj0ZFCr/bV/nQeix9wlP XKd1gqYBh6Hja2jgJ4PEz+gbAevt2gX5bjUpWz2GQPuDVolD/yMGH9TYP4XYVyCUzLDJ ORk2GI0HT/2Ivd9zul7/NDYX3QRfd0oVRnL14DkQ53iOpAO4S8NdgN+kkuuVyqQIh942 VZoA== MIME-Version: 1.0 X-Received: by 10.50.118.69 with SMTP id kk5mr2410883igb.36.1370030753456; Fri, 31 May 2013 13:05:53 -0700 (PDT) Received: by 10.64.71.101 with HTTP; Fri, 31 May 2013 13:05:53 -0700 (PDT) Received: by 10.64.71.101 with HTTP; Fri, 31 May 2013 13:05:53 -0700 (PDT) In-Reply-To: <20130531194703.GC5410@dan.emsphone.com> References: <20130527.194235.693.1@DOMY-PC> <20130530223031.GA1672@stack.nl> <20130531.175959.745.2@DOMY-PC> <13CA24D6AB415D428143D44749F57D7201F6AD11@ltcfiswmsgmb21> <20130531194703.GC5410@dan.emsphone.com> Date: Fri, 31 May 2013 21:05:53 +0100 Message-ID: Subject: Re: /bin/sh => STDIN & functions, var scope messing From: Chris Rees To: Dan Nelson Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: rank1seeker@gmail.com, Reid Linnemann , FreeBSD Hackers , Jilles Tjoelker X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 31 May 2013 20:05:53 -0000 On 31 May 2013 20:50, "Dan Nelson" wrote: > > In the last episode (May 31), Reid Linnemann said: > > On Fri, May 31, 2013 at 1:12 PM, Teske, Devin wrote: > > > If you're arguing we have to change sh's behavior to be more compliant, > > > jilles already quoted XCU 2.12 (our shell is well within its right to > > > run any/all lvalue/rvalue operands of a pipe in a sub-shell without > > > contradicting the guidelines). > > > > > > But if you're arguing that it has to change to make things better or > > > easier... I don't know about that. Might just make people lulled into > > > using a style that's non-portable. I'd like to keep things the way they > > > are so that if you program for FreeBSD, you're inherently going to > > > program in a fashion that is more portable. > > > > FWIW bash (invoked as sh) on RHEL-based linux systems exhibits the same > > behavior- > > > > sh-3.2$ var=1 > > sh-3.2$ yes|var=2 > > sh-3.2$ echo $var > > 1 > > sh-3.2$ > > > > If my opinion matters at all, I would agree that for the sake of > > portability that behavior be consistent with the majority of sh > > implementations rather than "right" according to arbitrary ruling. > > On the other hand, zsh runs the last component of a pipeline in the parent > shell. The usual model is "do work in pipeline, process results in final > component", and being able to simply set variables there that can be used in > the rest of the script is very elegant. Right... But it's not portable. Chris From owner-freebsd-hackers@FreeBSD.ORG Sat Jun 1 01:46:00 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 6DDF449B for ; Sat, 1 Jun 2013 01:46:00 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vps.rulingia.com (host-122-100-2-194.octopus.com.au [122.100.2.194]) by mx1.freebsd.org (Postfix) with ESMTP id E945377D for ; Sat, 1 Jun 2013 01:45:58 +0000 (UTC) Received: from server.rulingia.com (c220-239-237-213.belrs5.nsw.optusnet.com.au [220.239.237.213]) by vps.rulingia.com (8.14.5/8.14.5) with ESMTP id r511jsxg067065 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Sat, 1 Jun 2013 11:45:55 +1000 (EST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.14.5/8.14.5) with ESMTP id r511jgUc043540 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 1 Jun 2013 11:45:42 +1000 (EST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.14.5/8.14.5/Submit) id r511jfJK043539; Sat, 1 Jun 2013 11:45:41 +1000 (EST) (envelope-from peter) Date: Sat, 1 Jun 2013 11:45:40 +1000 From: Peter Jeremy To: Dirk-Willem van Gulik Subject: Re: seeding randomness in zee cloud Message-ID: <20130601014540.GF79250@server.rulingia.com> References: <0BF6FBDD-47E8-44F1-BA71-A355EDCDEDB6@webweaving.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="QTprm0S8XgL7H0Dt" Content-Disposition: inline In-Reply-To: <0BF6FBDD-47E8-44F1-BA71-A355EDCDEDB6@webweaving.org> X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Jun 2013 01:46:00 -0000 --QTprm0S8XgL7H0Dt Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2013-May-31 12:01:02 +0200, Dirk-Willem van Gulik = wrote: > Thanks to a badly-written mngt script - >we've rencently noticed a freshly generated ssh-key on a new AWS >instances to be indentical to one seen a few months prior. =2E.. >I am surmising that perhaps the (micro-T) images do not have that >much entropy on startup. This is a fairly common issue - typically, the first thing a newly installed system does immediately after a boot (when it has the least entropy availab= le) is to generate its SSH host keys. >Now we happen to have very easy access to blocks of 1024bits of >randomness from a remote server in already nicely PKI signed packages >(as it is needed later for something else). Obtaining entropy from another machine is an option but you need to ensure that the source is trustworthy, you only use the entropy once and that the entropy can't be intercepted by anyone else. >Or does this cause a loss/reset of all entropy gathered by the hardware so= far ? As others have indicated, no. Writing to /dev/random can't reduce the available entropy. > Or is there a cleaner way to add a additional seed as a one-off with >disturbing as little as possible (in the few seconds just after the >network is brought up). If this needs to be done automatically, not really. If there's a person available, you could use the "please type a screen full of random junk" approach and feed both the inter-character timings (which should be done automatically via IRQ harvesting) and junk into /dev/random. --=20 Peter Jeremy --QTprm0S8XgL7H0Dt Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iEYEARECAAYFAlGpUkQACgkQ/opHv/APuIfiQACfW6DsCUhclpUYxT4crFZ8a1Qu kJcAoI7mB2H5lYHh2Re9eELeW8nQBLFj =0341 -----END PGP SIGNATURE----- --QTprm0S8XgL7H0Dt-- From owner-freebsd-hackers@FreeBSD.ORG Sat Jun 1 08:40:02 2013 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 8EFB9123 for ; Sat, 1 Jun 2013 08:40:02 +0000 (UTC) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from wojtek.tensor.gdynia.pl (wojtek.tensor.gdynia.pl [188.252.31.196]) by mx1.freebsd.org (Postfix) with ESMTP id 174EE6C8 for ; Sat, 1 Jun 2013 08:40:01 +0000 (UTC) Received: from wojtek.tensor.gdynia.pl (localhost [127.0.0.1]) by wojtek.tensor.gdynia.pl (8.14.7/8.14.6) with ESMTP id r518doCV007159; Sat, 1 Jun 2013 10:39:50 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Received: from localhost (wojtek@localhost) by wojtek.tensor.gdynia.pl (8.14.6/8.14.6/Submit) with ESMTP id r518dmP4007156; Sat, 1 Jun 2013 10:39:49 +0200 (CEST) (envelope-from wojtek@wojtek.tensor.gdynia.pl) Date: Sat, 1 Jun 2013 10:39:48 +0200 (CEST) From: Wojciech Puchar To: Peter Jeremy Subject: Re: seeding randomness in zee cloud In-Reply-To: <20130601014540.GF79250@server.rulingia.com> Message-ID: References: <0BF6FBDD-47E8-44F1-BA71-A355EDCDEDB6@webweaving.org> <20130601014540.GF79250@server.rulingia.com> User-Agent: Alpine 2.00 (BSF 1167 2008-08-23) MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (wojtek.tensor.gdynia.pl [127.0.0.1]); Sat, 01 Jun 2013 10:39:50 +0200 (CEST) Cc: Dirk-Willem van Gulik , freebsd-hackers@freebsd.org X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 Jun 2013 08:40:02 -0000 >> Or is there a cleaner way to add a additional seed as a one-off with >> disturbing as little as possible (in the few seconds just after the >> network is brought up). > > If this needs to be done automatically, not really. If there's a > person available, you could use the "please type a screen full of > random junk" approach and feed both the inter-character timings (which > should be done automatically via IRQ harvesting) and junk into > /dev/random. > why just not put entropy files before installing from image?