From owner-freebsd-arch@FreeBSD.ORG Sun Nov 3 17:56:09 2013 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id BBD6B65D; Sun, 3 Nov 2013 17:56:09 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qc0-x233.google.com (mail-qc0-x233.google.com [IPv6:2607:f8b0:400d:c01::233]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 65C8A2A35; Sun, 3 Nov 2013 17:56:09 +0000 (UTC) Received: by mail-qc0-f179.google.com with SMTP id k18so3509871qcv.38 for ; Sun, 03 Nov 2013 09:56:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=NlrG4wNrL09iDomEUipaW/rTe8So5H02pro7JNWh9Ok=; b=TYA2Uv6WBUrCYvIbB5X6Uwca3I+ee9qCvpMYWXb4A3MYpYppWq8pP0MPmGyEaaAFiC Z9QWKQ/uCVCwMcC02Xfrdzx4CfTJmBLgWUb6jgV+2/cYhaLk1vZdMh3LRt7hnPJaeklW fFZ+ZeqxP+KmG9I7hpomilMqMO4eFFdLCFHClg5iXJVujl4EcHv8yp7F2oVFutA00J2m EMDK6opUS8eL5vcL+okgSDghGZ38k6Rm0+CcY7POASDcazURunFj8DP/q/nAcUx8FYwx lmGnM72eru8BYcHJXrbAaLY6wkXimN1t0nDLbDUQj723/OP6iLq2hBxt91n6HvhDaUyn s1vQ== MIME-Version: 1.0 X-Received: by 10.224.51.131 with SMTP id d3mr17707474qag.0.1383501368569; Sun, 03 Nov 2013 09:56:08 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.224.207.66 with HTTP; Sun, 3 Nov 2013 09:56:08 -0800 (PST) In-Reply-To: <1383419596.3253.42.camel@eva02.mbsd> References: <1382572583.1862.39.camel@eva02.mbsd> <1382589020.1846.36.camel@eva02.mbsd> <1383331203.12614.1.camel@eva02.mbsd> <1383336004.13657.18.camel@eva02.mbsd> <1383338117.13657.53.camel@eva02.mbsd> <1383382815.31973.1.camel@eva02.mbsd> <1383419596.3253.42.camel@eva02.mbsd> Date: Sun, 3 Nov 2013 09:56:08 -0800 X-Google-Sender-Auth: 6LW7WgQ8fQeIU01Y_JcFzla5h3M Message-ID: Subject: Re: service netif restart [iface] runs a wpa_supplicant twice From: Adrian Chadd To: clutton , "freebsd-arch@freebsd.org" Content-Type: text/plain; charset=ISO-8859-1 Cc: "freebsd-wireless@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Nov 2013 17:56:09 -0000 On 2 November 2013 12:13, clutton wrote: [snip] > What was happened? netif tries to setup wlan0 (clone, wpa, dhcp, etc), > when wlan0 interface occurs, devd runs another copy of netif. Well, it sounds like we need to pick an architecture _and_ fix the behaviour here. Which is: * I think wpa-supplicant should always run if it's required in /etc/rc.conf; * netif should check if devd is configured and if so, just leave the configuration up to devd * if it isn't running, then devd should be responsible for dhclient/add-to-wpa-config What we first have to establish is whether add_interface and remove_interface (or whatever they're called) are correctly working, for ethernet and wifi driver types. Then, we need to ensure they can coexist (ie, one wpa_supplicant, but with both ethernet/wifi drivers loaded and active on their relevant interfaces.) _then_ we can break out the "stuff devd does" out of netif and have _either_ netif (x)or devd call this new script to setup/teardown the interface runtime state. How's that sound? -adrian From owner-freebsd-arch@FreeBSD.ORG Tue Nov 5 18:23:17 2013 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 8130580; Tue, 5 Nov 2013 18:23:17 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 5773F2C2E; Tue, 5 Nov 2013 18:23:17 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 57349B968; Tue, 5 Nov 2013 13:23:15 -0500 (EST) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: service netif restart [iface] runs a wpa_supplicant twice Date: Tue, 5 Nov 2013 11:54:18 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; ) References: <1382572583.1862.39.camel@eva02.mbsd> <1383419596.3253.42.camel@eva02.mbsd> In-Reply-To: MIME-Version: 1.0 Message-Id: <201311051154.18872.jhb@freebsd.org> Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 05 Nov 2013 13:23:15 -0500 (EST) Cc: clutton , Adrian Chadd , "freebsd-wireless@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Nov 2013 18:23:17 -0000 On Sunday, November 03, 2013 12:56:08 pm Adrian Chadd wrote: > On 2 November 2013 12:13, clutton wrote: > > [snip] > > > What was happened? netif tries to setup wlan0 (clone, wpa, dhcp, etc), > > when wlan0 interface occurs, devd runs another copy of netif. > > Well, it sounds like we need to pick an architecture _and_ fix the > behaviour here. > > Which is: > > > * I think wpa-supplicant should always run if it's required in /etc/rc.conf; > * netif should check if devd is configured and if so, just leave the > configuration up to devd > * if it isn't running, then devd should be responsible for > dhclient/add-to-wpa-config > > What we first have to establish is whether add_interface and > remove_interface (or whatever they're called) are correctly working, > for ethernet and wifi driver types. Then, we need to ensure they can > coexist (ie, one wpa_supplicant, but with both ethernet/wifi drivers > loaded and active on their relevant interfaces.) _then_ we can break > out the "stuff devd does" out of netif and have _either_ netif (x)or > devd call this new script to setup/teardown the interface runtime > state. > > How's that sound? Note that devd just runs netif (via /etc/pccard_ether), so it's already just one script, and having netif bail if devd is running would make netif not do anything in the common case. What normally happens during boot is that '/etc/rc.d/netif start' creates wlan0 and runs wpa_supplicant via 'childif_create' making a nested call to ifn_start for wlan0. That is, childif_create autoruns /etc/rc.d/netif explicitly after it creates the device. Probably that is what should be removed. That would let devd always start wpa_supplicant via /etc/pccard_ether. I've just tested this by doing a stop/start on iwn0 (parent of wlan0, so wlan0 gets destroyed and re-created) and it started wpa_supplicant correctly. Index: head/etc/network.subr =================================================================== --- network.subr (revision 257705) +++ network.subr (working copy) @@ -1429,9 +1429,6 @@ childif_create() fi ${IFCONFIG_CMD} $i name $child && cfg=0 fi - if autoif $child; then - ifn_start $child - fi done # Create vlan interfaces I also tested vlans created via vlans_ and they should use the same fix as well. Note that this model is more consistent with how cloned_interfaces works where ifn_start is not explicitly run when each interface is created. Instead, we rely on devd kicking off pccard_ether for those as well. -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Tue Nov 5 19:03:49 2013 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id CD660652; Tue, 5 Nov 2013 19:03:49 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qe0-x22e.google.com (mail-qe0-x22e.google.com [IPv6:2607:f8b0:400d:c02::22e]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 6C43B2EFB; Tue, 5 Nov 2013 19:03:49 +0000 (UTC) Received: by mail-qe0-f46.google.com with SMTP id s14so5318656qeb.33 for ; Tue, 05 Nov 2013 11:03:48 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date:message-id:subject :from:to:cc:content-type; bh=NRYgCyKCjrafZHaWFLMoH708tu1m5BkpkvpvMT8gQAY=; b=YqzfkOJmN+zOJPkhsFvwEzrekVZZzX+p/B+GkT2wZicTVSrZkSn3zIm5+6Cr+8hOdj d8XQXqmTeYO56okM1SQ1CJaIgeqRpkv/tIALGvZNeonDafC1C4iXp9xFxAXxuBReehGe /p4ujE14zVzNKC16/JSXRx5KQGIHez1zltivyE5g4HOL9FDyAa0p3QLU6uwkE+aTGhbx 6ed98xMvHwvjA+7DUOfWQknjg+LVa2v+5x8O2+0FG+0A63xWf5xJYQAyeGJu6dNmpM65 9n5IhQd52DrxNv0iZFaam/oX60iqmqGzpji0OV/MbQBprBhISkr/jZe5y0joQ1E+PdLX m1PA== MIME-Version: 1.0 X-Received: by 10.49.59.115 with SMTP id y19mr31529010qeq.8.1383678228585; Tue, 05 Nov 2013 11:03:48 -0800 (PST) Sender: adrian.chadd@gmail.com Received: by 10.224.207.66 with HTTP; Tue, 5 Nov 2013 11:03:48 -0800 (PST) In-Reply-To: <201311051154.18872.jhb@freebsd.org> References: <1382572583.1862.39.camel@eva02.mbsd> <1383419596.3253.42.camel@eva02.mbsd> <201311051154.18872.jhb@freebsd.org> Date: Tue, 5 Nov 2013 11:03:48 -0800 X-Google-Sender-Auth: VQVObu6YbYwAdTlLt0xOukFlQ24 Message-ID: Subject: Re: service netif restart [iface] runs a wpa_supplicant twice From: Adrian Chadd To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Cc: clutton , "freebsd-wireless@freebsd.org" , "freebsd-arch@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Nov 2013 19:03:49 -0000 On 5 November 2013 08:54, John Baldwin wrote: > Note that devd just runs netif (via /etc/pccard_ether), so it's already > just one script, and having netif bail if devd is running would make > netif not do anything in the common case. > > What normally happens during boot is that '/etc/rc.d/netif start' creates > wlan0 and runs wpa_supplicant via 'childif_create' making a nested call to > ifn_start for wlan0. That is, childif_create autoruns /etc/rc.d/netif > explicitly after it creates the device. Probably that is what should be > removed. That would let devd always start wpa_supplicant via > /etc/pccard_ether. I've just tested this by doing a stop/start on iwn0 > (parent of wlan0, so wlan0 gets destroyed and re-created) and it started > wpa_supplicant correctly. > > Index: head/etc/network.subr > =================================================================== > --- network.subr (revision 257705) > +++ network.subr (working copy) > @@ -1429,9 +1429,6 @@ childif_create() > fi > ${IFCONFIG_CMD} $i name $child && cfg=0 > fi > - if autoif $child; then > - ifn_start $child > - fi > done > > # Create vlan interfaces > > I also tested vlans created via vlans_ and they should use the same fix as > well. Note that this model is more consistent with how cloned_interfaces > works where ifn_start is not explicitly run when each interface is created. > Instead, we rely on devd kicking off pccard_ether for those as well. Well, if you'd like to do the honours.. ? I'd like this fixed before 10.0-REL. It should make the normal use case for wifi actually work reliably again. -adrian From owner-freebsd-arch@FreeBSD.ORG Tue Nov 5 19:33:53 2013 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 02059AA7 for ; Tue, 5 Nov 2013 19:33:52 +0000 (UTC) (envelope-from bschmidt@techwires.net) Received: from mail-wi0-f173.google.com (mail-wi0-f173.google.com [209.85.212.173]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 70FAF2117 for ; Tue, 5 Nov 2013 19:33:52 +0000 (UTC) Received: by mail-wi0-f173.google.com with SMTP id ey11so2665171wid.12 for ; Tue, 05 Nov 2013 11:33:50 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc:content-type; bh=RG92irbN9hCPPVhzWP35JsMJyAuanIyipef5rTDe6Lc=; b=JsUoijR6ygLCVJoOttXTbQgbQ0+gog6AREFX48z2Gxhz6CS+V441IqaGTgCg84VNG1 Aqf5gwcONuX9v0C3JyUVm+r8N2K4xZVJlhHTScP6pGw6l1W/k4EEnbQ6GyU2FrH/tjE+ qRHec0EYn1DMVdK0U01pXsoyAY3uDCn7UoU7C3vMMjuzw/A1I6t+SjKu5cJFxeIOw3IH FMbEUj4DsdXA3DbMjZYlisD8U9XmkbY9S9wWKiLRDXEMSer4G8Wo7vyoWqHeunesDJw5 Bm+5QyZyLjbQxPP8vZB6EJS2dNxYKGrJOZ/i0Ng5lCgiD2FdWJLrnAUMKuM8UR+jvzM9 qDTw== X-Gm-Message-State: ALoCoQl+a98lMzomllT/mlmgoH8ZBmf7oWJBEtVyyd3SZqsiit2macEI/f/CVA25g8pxKX0unzrh MIME-Version: 1.0 X-Received: by 10.194.240.197 with SMTP id wc5mr19642081wjc.23.1383680030616; Tue, 05 Nov 2013 11:33:50 -0800 (PST) Received: by 10.227.226.196 with HTTP; Tue, 5 Nov 2013 11:33:50 -0800 (PST) X-Originating-IP: [88.67.223.128] In-Reply-To: <201311051154.18872.jhb@freebsd.org> References: <1382572583.1862.39.camel@eva02.mbsd> <1383419596.3253.42.camel@eva02.mbsd> <201311051154.18872.jhb@freebsd.org> Date: Tue, 5 Nov 2013 20:33:50 +0100 Message-ID: Subject: Re: service netif restart [iface] runs a wpa_supplicant twice From: Bernhard Schmidt To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 Cc: clutton , "freebsd-wireless@freebsd.org" , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Nov 2013 19:33:53 -0000 On Tue, Nov 5, 2013 at 5:54 PM, John Baldwin wrote: > On Sunday, November 03, 2013 12:56:08 pm Adrian Chadd wrote: >> On 2 November 2013 12:13, clutton wrote: >> >> [snip] >> >> > What was happened? netif tries to setup wlan0 (clone, wpa, dhcp, etc), >> > when wlan0 interface occurs, devd runs another copy of netif. >> >> Well, it sounds like we need to pick an architecture _and_ fix the >> behaviour here. >> >> Which is: >> >> >> * I think wpa-supplicant should always run if it's required in /etc/rc.conf; >> * netif should check if devd is configured and if so, just leave the >> configuration up to devd >> * if it isn't running, then devd should be responsible for >> dhclient/add-to-wpa-config >> >> What we first have to establish is whether add_interface and >> remove_interface (or whatever they're called) are correctly working, >> for ethernet and wifi driver types. Then, we need to ensure they can >> coexist (ie, one wpa_supplicant, but with both ethernet/wifi drivers >> loaded and active on their relevant interfaces.) _then_ we can break >> out the "stuff devd does" out of netif and have _either_ netif (x)or >> devd call this new script to setup/teardown the interface runtime >> state. >> >> How's that sound? > > Note that devd just runs netif (via /etc/pccard_ether), so it's already > just one script, and having netif bail if devd is running would make > netif not do anything in the common case. > > What normally happens during boot is that '/etc/rc.d/netif start' creates > wlan0 and runs wpa_supplicant via 'childif_create' making a nested call to > ifn_start for wlan0. That is, childif_create autoruns /etc/rc.d/netif > explicitly after it creates the device. Probably that is what should be > removed. That would let devd always start wpa_supplicant via > /etc/pccard_ether. I've just tested this by doing a stop/start on iwn0 > (parent of wlan0, so wlan0 gets destroyed and re-created) and it started > wpa_supplicant correctly. > > Index: head/etc/network.subr > =================================================================== > --- network.subr (revision 257705) > +++ network.subr (working copy) > @@ -1429,9 +1429,6 @@ childif_create() > fi > ${IFCONFIG_CMD} $i name $child && cfg=0 > fi > - if autoif $child; then > - ifn_start $child > - fi > done > > # Create vlan interfaces > > I also tested vlans created via vlans_ and they should use the same fix as > well. Note that this model is more consistent with how cloned_interfaces > works where ifn_start is not explicitly run when each interface is created. > Instead, we rely on devd kicking off pccard_ether for those as well. That looks sane too me. Just one question, I remember that devd is disabled during boot and activated later through a sysctl (to ignore events entirely), is this the case before or after netif is running? I guess it is activated after netif, otherwise we would have seen this issue on booting and not just during netif restart. -- Bernhard From owner-freebsd-arch@FreeBSD.ORG Tue Nov 5 22:19:31 2013 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id EDA65A58; Tue, 5 Nov 2013 22:19:31 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id A7F4A2BA3; Tue, 5 Nov 2013 22:19:31 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 4D992B98F; Tue, 5 Nov 2013 17:19:30 -0500 (EST) From: John Baldwin To: Bernhard Schmidt Subject: Re: service netif restart [iface] runs a wpa_supplicant twice Date: Tue, 5 Nov 2013 17:17:30 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; ) References: <1382572583.1862.39.camel@eva02.mbsd> <201311051154.18872.jhb@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201311051717.30519.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 05 Nov 2013 17:19:30 -0500 (EST) Cc: clutton , "freebsd-wireless@freebsd.org" , freebsd-arch@freebsd.org X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Nov 2013 22:19:32 -0000 On Tuesday, November 05, 2013 2:33:50 pm Bernhard Schmidt wrote: > On Tue, Nov 5, 2013 at 5:54 PM, John Baldwin wrote: > > On Sunday, November 03, 2013 12:56:08 pm Adrian Chadd wrote: > >> On 2 November 2013 12:13, clutton wrote: > >> > >> [snip] > >> > >> > What was happened? netif tries to setup wlan0 (clone, wpa, dhcp, etc), > >> > when wlan0 interface occurs, devd runs another copy of netif. > >> > >> Well, it sounds like we need to pick an architecture _and_ fix the > >> behaviour here. > >> > >> Which is: > >> > >> > >> * I think wpa-supplicant should always run if it's required in /etc/rc.conf; > >> * netif should check if devd is configured and if so, just leave the > >> configuration up to devd > >> * if it isn't running, then devd should be responsible for > >> dhclient/add-to-wpa-config > >> > >> What we first have to establish is whether add_interface and > >> remove_interface (or whatever they're called) are correctly working, > >> for ethernet and wifi driver types. Then, we need to ensure they can > >> coexist (ie, one wpa_supplicant, but with both ethernet/wifi drivers > >> loaded and active on their relevant interfaces.) _then_ we can break > >> out the "stuff devd does" out of netif and have _either_ netif (x)or > >> devd call this new script to setup/teardown the interface runtime > >> state. > >> > >> How's that sound? > > > > Note that devd just runs netif (via /etc/pccard_ether), so it's already > > just one script, and having netif bail if devd is running would make > > netif not do anything in the common case. > > > > What normally happens during boot is that '/etc/rc.d/netif start' creates > > wlan0 and runs wpa_supplicant via 'childif_create' making a nested call to > > ifn_start for wlan0. That is, childif_create autoruns /etc/rc.d/netif > > explicitly after it creates the device. Probably that is what should be > > removed. That would let devd always start wpa_supplicant via > > /etc/pccard_ether. I've just tested this by doing a stop/start on iwn0 > > (parent of wlan0, so wlan0 gets destroyed and re-created) and it started > > wpa_supplicant correctly. > > > > Index: head/etc/network.subr > > =================================================================== > > --- network.subr (revision 257705) > > +++ network.subr (working copy) > > @@ -1429,9 +1429,6 @@ childif_create() > > fi > > ${IFCONFIG_CMD} $i name $child && cfg=0 > > fi > > - if autoif $child; then > > - ifn_start $child > > - fi > > done > > > > # Create vlan interfaces > > > > I also tested vlans created via vlans_ and they should use the same fix as > > well. Note that this model is more consistent with how cloned_interfaces > > works where ifn_start is not explicitly run when each interface is created. > > Instead, we rely on devd kicking off pccard_ether for those as well. > > That looks sane too me. > > Just one question, I remember that devd is disabled during boot and > activated later through a sysctl (to ignore events entirely), is this > the case before or after netif is running? I guess it is activated > after netif, otherwise we would have seen this issue on booting and > not just during netif restart. Hmm, devd starts after netif, but it just worked fine for me when I booted up. I also misspoke about cloned_interfaces. We manually add the cloned_interface list to the list of interfaces /etc/rc.d/netif iterates over. What I am puzzled by is that this just worked for me during a test boot. Hmm, it looks like devctl is no longer disabled during boot and then explicitly enabled by devd. devctl is now always enabled during boot, but capped at 1000 entries to avoid leaking memory. In fact, it looks like devd tries to recreate a few interfaces after netif finishes and is generally confused. I tried again with devd_flags set to "-n" to flush the initial set of events on boot. This removed the multiple calls to netif on boot on my laptop, but somehow wpa_supplicant is still being started by devd (and I'm not sure how now). Maybe we should bring back the ifn_start calls, but only do them if devd is not running (i.e. during boot) as Adrian suggested? -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Wed Nov 6 19:18:24 2013 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 81C4B759; Wed, 6 Nov 2013 19:18:24 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) (using TLSv1 with cipher ADH-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 375482647; Wed, 6 Nov 2013 19:18:24 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id 3AF05B96C; Wed, 6 Nov 2013 14:18:23 -0500 (EST) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: service netif restart [iface] runs a wpa_supplicant twice Date: Wed, 6 Nov 2013 11:59:14 -0500 User-Agent: KMail/1.13.5 (FreeBSD/8.4-CBSD-20130906; KDE/4.5.5; amd64; ; ) References: <1382572583.1862.39.camel@eva02.mbsd> <201311051717.30519.jhb@freebsd.org> In-Reply-To: <201311051717.30519.jhb@freebsd.org> MIME-Version: 1.0 Message-Id: <201311061159.14824.jhb@freebsd.org> Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Wed, 06 Nov 2013 14:18:23 -0500 (EST) Cc: clutton , Bernhard Schmidt , "freebsd-wireless@freebsd.org" X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Nov 2013 19:18:24 -0000 On Tuesday, November 05, 2013 5:17:30 pm John Baldwin wrote: > On Tuesday, November 05, 2013 2:33:50 pm Bernhard Schmidt wrote: > > On Tue, Nov 5, 2013 at 5:54 PM, John Baldwin wrote: > > > On Sunday, November 03, 2013 12:56:08 pm Adrian Chadd wrote: > > >> On 2 November 2013 12:13, clutton wrote: > > >> > > >> [snip] > > >> > > >> > What was happened? netif tries to setup wlan0 (clone, wpa, dhcp, etc), > > >> > when wlan0 interface occurs, devd runs another copy of netif. > > >> > > >> Well, it sounds like we need to pick an architecture _and_ fix the > > >> behaviour here. > > >> > > >> Which is: > > >> > > >> > > >> * I think wpa-supplicant should always run if it's required in /etc/rc.conf; > > >> * netif should check if devd is configured and if so, just leave the > > >> configuration up to devd > > >> * if it isn't running, then devd should be responsible for > > >> dhclient/add-to-wpa-config > > >> > > >> What we first have to establish is whether add_interface and > > >> remove_interface (or whatever they're called) are correctly working, > > >> for ethernet and wifi driver types. Then, we need to ensure they can > > >> coexist (ie, one wpa_supplicant, but with both ethernet/wifi drivers > > >> loaded and active on their relevant interfaces.) _then_ we can break > > >> out the "stuff devd does" out of netif and have _either_ netif (x)or > > >> devd call this new script to setup/teardown the interface runtime > > >> state. > > >> > > >> How's that sound? > > > > > > Note that devd just runs netif (via /etc/pccard_ether), so it's already > > > just one script, and having netif bail if devd is running would make > > > netif not do anything in the common case. > > > > > > What normally happens during boot is that '/etc/rc.d/netif start' creates > > > wlan0 and runs wpa_supplicant via 'childif_create' making a nested call to > > > ifn_start for wlan0. That is, childif_create autoruns /etc/rc.d/netif > > > explicitly after it creates the device. Probably that is what should be > > > removed. That would let devd always start wpa_supplicant via > > > /etc/pccard_ether. I've just tested this by doing a stop/start on iwn0 > > > (parent of wlan0, so wlan0 gets destroyed and re-created) and it started > > > wpa_supplicant correctly. > > > > > > Index: head/etc/network.subr > > > =================================================================== > > > --- network.subr (revision 257705) > > > +++ network.subr (working copy) > > > @@ -1429,9 +1429,6 @@ childif_create() > > > fi > > > ${IFCONFIG_CMD} $i name $child && cfg=0 > > > fi > > > - if autoif $child; then > > > - ifn_start $child > > > - fi > > > done > > > > > > # Create vlan interfaces > > > > > > I also tested vlans created via vlans_ and they should use the same fix as > > > well. Note that this model is more consistent with how cloned_interfaces > > > works where ifn_start is not explicitly run when each interface is created. > > > Instead, we rely on devd kicking off pccard_ether for those as well. > > > > That looks sane too me. > > > > Just one question, I remember that devd is disabled during boot and > > activated later through a sysctl (to ignore events entirely), is this > > the case before or after netif is running? I guess it is activated > > after netif, otherwise we would have seen this issue on booting and > > not just during netif restart. > > Hmm, devd starts after netif, but it just worked fine for me when I booted up. > I also misspoke about cloned_interfaces. We manually add the cloned_interface > list to the list of interfaces /etc/rc.d/netif iterates over. What I am > puzzled by is that this just worked for me during a test boot. Hmm, it looks like > devctl is no longer disabled during boot and then explicitly enabled by devd. > devctl is now always enabled during boot, but capped at 1000 entries to avoid > leaking memory. In fact, it looks like devd tries to recreate a few interfaces > after netif finishes and is generally confused. I tried again with devd_flags > set to "-n" to flush the initial set of events on boot. This removed the > multiple calls to netif on boot on my laptop, but somehow wpa_supplicant is > still being started by devd (and I'm not sure how now). I've hacked devd some more and can now see what is going on. -n doesn't do what I thought it does. It does not throw away pending events on startup, it just makes devd not fork until it has walked the initial set of events. The kernel changed (a while ago) to queue the first 1000 events until devd starts up. This means that in practice devd gets arrival events for all devices in the system as soon as it starts up and triggers duplicate invocations of netif after netif finishes. However, /etc/pccard_ether ignores attempts to start a device that is already up, so this should be a no-op on bootup (if my change is reverted) as the interfaces should already be configured by the time devd starts. I suspect what happens in multiuser is that devd fires off pccard_ether and sees that the interface isn't up before the original netif has a chance to invoke the nested ifn_start. We could perhaps change it so we only invoke ifn_start if devd isn't running. One other thought: I restart my wireless interfaces by doing 'sh /etc/rc.d/netif restart wlan0', not 'iwn0'. This doesn't teardown/recreate the wlan0 device, so it doesn't suffer from the issue reported by the OP. Here is a change I've tested that seems to do the right thing both at boot time and doing a restart of either iwn0 or wlan0 at runtime. If devd is running it leaves the task of starting an interface up to devd, otherwise (such as during boot), it configures the new child interface synchronously. Note that pgrep is in /bin. Index: network.subr =================================================================== --- network.subr (revision 257747) +++ network.subr (working copy) @@ -1406,10 +1406,14 @@ clone_down() # childif_create() { - local cfg child child_vlans child_wlans create_args debug_flags ifn i + local cfg child child_vlans child_wlans create_args debug_flags devd \ + ifn i cfg=1 ifn=$1 + # Check if devd is running + devd=$(pgrep devd) + # Create wireless interfaces child_wlans=`get_if_var $ifn wlans_IF` @@ -1429,6 +1433,9 @@ childif_create() fi ${IFCONFIG_CMD} $i name $child && cfg=0 fi + if [ -z "$devd" ] && autoif $child; then + ifn_start $child + fi done # Create vlan interfaces @@ -1452,6 +1459,9 @@ childif_create() ${IFCONFIG_CMD} $i name $child && cfg=0 fi fi + if [ -z "$devd" ] && autoif $child; then + ifn_start $child + fi done return ${cfg} -- John Baldwin From owner-freebsd-arch@FreeBSD.ORG Thu Nov 7 22:22:17 2013 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id E05B5E99; Thu, 7 Nov 2013 22:22:17 +0000 (UTC) (envelope-from clutton@zoho.com) Received: from sender1.zohomail.com (sender1.zohomail.com [72.5.230.95]) by mx1.freebsd.org (Postfix) with ESMTP id BDC862286; Thu, 7 Nov 2013 22:22:17 +0000 (UTC) DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws; s=zapps768; d=zoho.com; h=subject:from:to:cc:in-reply-to:references:content-type:date:message-id:mime-version; b=KXl98ZqPxYsesq9eENVQxu9CLi9nTeQtuA5LadivjIxGwEmmVjek6E9ZDrHOpEbF9W8w2bZPTMLS RvFQ+NxEfJMxmUexLke4To6tUk8VQGwo1Wnu0Sef6Xrh+xR0TALj Received: from [192.168.1.100] (svann.corbina.com.ua [31.43.139.125]) by mx.zohomail.com with SMTPS id 1383862929208753.7064245195025; Thu, 7 Nov 2013 14:22:09 -0800 (PST) Subject: Re: service netif restart [iface] runs a wpa_supplicant twice From: clutton To: freebsd-wireless@freebsd.org, freebsd-arch@freebsd.org In-Reply-To: <201311061159.14824.jhb@freebsd.org> References: <1382572583.1862.39.camel@eva02.mbsd> <201311051717.30519.jhb@freebsd.org> <201311061159.14824.jhb@freebsd.org> Content-Type: multipart/signed; micalg="pgp-sha1"; protocol="application/pgp-signature"; boundary="=-BVm0CXnSENNJNebfmTOo" Date: Fri, 08 Nov 2013 00:22:03 +0200 Message-ID: <1383862923.70321.87.camel@eva02.mbsd> Mime-Version: 1.0 X-Mailer: Evolution 2.32.1 FreeBSD GNOME Team Port X-Zoho-Virus-Status: 1 X-ZohoMail: Ss SS_10 UW UB UW UB SGR3_1_06113_64 X-ZohoMail-Owner: <1383862923.70321.87.camel@eva02.mbsd>+zmo_0_ X-ZohoMail-Sender: 31.43.139.125 X-ZohoMailClient: External Cc: Bernhard Schmidt , Adrian Chadd X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Nov 2013 22:22:18 -0000 --=-BVm0CXnSENNJNebfmTOo Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Wed, 2013-11-06 at 11:59 -0500, John Baldwin wrote: > On Tuesday, November 05, 2013 5:17:30 pm John Baldwin wrote: > > On Tuesday, November 05, 2013 2:33:50 pm Bernhard Schmidt wrote: > > > On Tue, Nov 5, 2013 at 5:54 PM, John Baldwin wrote: > > > > On Sunday, November 03, 2013 12:56:08 pm Adrian Chadd wrote: > > > >> On 2 November 2013 12:13, clutton wrote: > > > >> > > > >> [snip] > > > >> > > > >> > What was happened? netif tries to setup wlan0 (clone, wpa, dhcp,= etc), > > > >> > when wlan0 interface occurs, devd runs another copy of netif. > > > >> > > > >> Well, it sounds like we need to pick an architecture _and_ fix the > > > >> behaviour here. > > > >> > > > >> Which is: > > > >> > > > >> > > > >> * I think wpa-supplicant should always run if it's required in /et= c/rc.conf; > > > >> * netif should check if devd is configured and if so, just leave t= he > > > >> configuration up to devd > > > >> * if it isn't running, then devd should be responsible for > > > >> dhclient/add-to-wpa-config > > > >> > > > >> What we first have to establish is whether add_interface and > > > >> remove_interface (or whatever they're called) are correctly workin= g, > > > >> for ethernet and wifi driver types. Then, we need to ensure they c= an > > > >> coexist (ie, one wpa_supplicant, but with both ethernet/wifi drive= rs > > > >> loaded and active on their relevant interfaces.) _then_ we can bre= ak > > > >> out the "stuff devd does" out of netif and have _either_ netif (x)= or > > > >> devd call this new script to setup/teardown the interface runtime > > > >> state. > > > >> > > > >> How's that sound? > > > > > > > > Note that devd just runs netif (via /etc/pccard_ether), so it's alr= eady > > > > just one script, and having netif bail if devd is running would mak= e > > > > netif not do anything in the common case. > > > > > > > > What normally happens during boot is that '/etc/rc.d/netif start' c= reates > > > > wlan0 and runs wpa_supplicant via 'childif_create' making a nested = call to > > > > ifn_start for wlan0. That is, childif_create autoruns /etc/rc.d/ne= tif > > > > explicitly after it creates the device. Probably that is what shou= ld be > > > > removed. That would let devd always start wpa_supplicant via > > > > /etc/pccard_ether. I've just tested this by doing a stop/start on = iwn0 > > > > (parent of wlan0, so wlan0 gets destroyed and re-created) and it st= arted > > > > wpa_supplicant correctly. > > > > > > > > Index: head/etc/network.subr > > > > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > > > > --- network.subr (revision 257705) > > > > +++ network.subr (working copy) > > > > @@ -1429,9 +1429,6 @@ childif_create() > > > > fi > > > > ${IFCONFIG_CMD} $i name $child && cfg=3D0 > > > > fi > > > > - if autoif $child; then > > > > - ifn_start $child > > > > - fi > > > > done > > > > > > > > # Create vlan interfaces > > > > > > > > I also tested vlans created via vlans_ and they should use the = same fix as > > > > well. Note that this model is more consistent with how cloned_inte= rfaces > > > > works where ifn_start is not explicitly run when each interface is = created. > > > > Instead, we rely on devd kicking off pccard_ether for those as well= . > > >=20 > > > That looks sane too me. > > >=20 > > > Just one question, I remember that devd is disabled during boot and > > > activated later through a sysctl (to ignore events entirely), is this > > > the case before or after netif is running? I guess it is activated > > > after netif, otherwise we would have seen this issue on booting and > > > not just during netif restart. > >=20 > > Hmm, devd starts after netif, but it just worked fine for me when I boo= ted up. > > I also misspoke about cloned_interfaces. We manually add the cloned_in= terface > > list to the list of interfaces /etc/rc.d/netif iterates over. What I a= m > > puzzled by is that this just worked for me during a test boot. Hmm, it= looks like > > devctl is no longer disabled during boot and then explicitly enabled by= devd. > > devctl is now always enabled during boot, but capped at 1000 entries to= avoid > > leaking memory. In fact, it looks like devd tries to recreate a few in= terfaces > > after netif finishes and is generally confused. I tried again with dev= d_flags > > set to "-n" to flush the initial set of events on boot. This removed t= he > > multiple calls to netif on boot on my laptop, but somehow wpa_supplican= t is > > still being started by devd (and I'm not sure how now). >=20 > I've hacked devd some more and can now see what is going on. -n doesn't = do what > I thought it does. It does not throw away pending events on startup, it = just > makes devd not fork until it has walked the initial set of events. The k= ernel > changed (a while ago) to queue the first 1000 events until devd starts up= . This > means that in practice devd gets arrival events for all devices in the sy= stem as > soon as it starts up and triggers duplicate invocations of netif after ne= tif > finishes. However, /etc/pccard_ether ignores attempts to start a device = that > is already up, so this should be a no-op on bootup (if my change is rever= ted) > as the interfaces should already be configured by the time devd starts. = I suspect > what happens in multiuser is that devd fires off pccard_ether and sees th= at the > interface isn't up before the original netif has a chance to invoke the n= ested > ifn_start. We could perhaps change it so we only invoke ifn_start if dev= d > isn't running. >=20 > One other thought: I restart my wireless interfaces by doing > 'sh /etc/rc.d/netif restart wlan0', not 'iwn0'. This doesn't teardown/re= create > the wlan0 device, so it doesn't suffer from the issue reported by the OP. >=20 > Here is a change I've tested that seems to do the right thing both at boo= t time > and doing a restart of either iwn0 or wlan0 at runtime. If devd is runni= ng > it leaves the task of starting an interface up to devd, otherwise (such a= s during > boot), it configures the new child interface synchronously. >=20 > Note that pgrep is in /bin. >=20 > Index: network.subr > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > --- network.subr (revision 257747) > +++ network.subr (working copy) > @@ -1406,10 +1406,14 @@ clone_down() > # > childif_create() > { > - local cfg child child_vlans child_wlans create_args debug_flags ifn i > + local cfg child child_vlans child_wlans create_args debug_flags devd \ > + ifn i > cfg=3D1 > ifn=3D$1 > =20 > + # Check if devd is running > + devd=3D$(pgrep devd) > + > # Create wireless interfaces > child_wlans=3D`get_if_var $ifn wlans_IF` > =20 > @@ -1429,6 +1433,9 @@ childif_create() > fi > ${IFCONFIG_CMD} $i name $child && cfg=3D0 > fi > + if [ -z "$devd" ] && autoif $child; then > + ifn_start $child > + fi > done > =20 > # Create vlan interfaces > @@ -1452,6 +1459,9 @@ childif_create() > ${IFCONFIG_CMD} $i name $child && cfg=3D0 > fi > fi > + if [ -z "$devd" ] && autoif $child; then > + ifn_start $child > + fi > done > =20 > return ${cfg} >=20 Yes, the "service netif restart wlan0" doesn't teardown/recreate the wlan0 device. Anyway, a "service netif restart" does. What about removing this functionality, instead? See the patch below. The pros: 1) creating the wlan interface by hand (by ifconfig) means that the further configuration is going to be in that way, by hand (by ifconfig, route, dhcpclient, etc). 2) already written down configuration (in rc.conf) means working with rc subsystem (netif) 3) I have no idea why somebody would expect from a command "ifconfig wlan0 create wlandev ath0" the same behaviour as from a "service netif start wlan0". 4) Let's remove the unexpected behaviour at all, it's prone error, it's not obviously at first glance, some kind of clever computer which knows better what do you need. I think that we have this functionality occasionally, no one had designed this on purpose, am I wrong? The cons: I have none. You told me. =CE=9E ~ =E2=86=92 diff -u /usr/src/etc/devd.conf /etc/devd.conf =20 --- /usr/src/etc/devd.conf 2013-09-29 17:24:16.759250174 +0300 +++ /etc/devd.conf 2013-11-07 23:43:17.833616197 +0200 @@ -38,7 +38,7 @@ # notify 0 { match "system" "IFNET"; - match "subsystem" "!usbus[0-9]+"; + match "subsystem" "!(usbus|wlan|vlan)[0-9]+"; match "type" "ATTACH"; action "/etc/pccard_ether $subsystem start"; }; --=-BVm0CXnSENNJNebfmTOo Content-Type: application/pgp-signature; name="signature.asc" Content-Description: This is a digitally signed message part -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAABAgAGBQJSfBKLAAoJECNkWbjnbjuixB0QAKRQ5n8KY0kVxCdw+bxNMSeK cbNeAyunZ6dx90sPTmw1eIsdlt0smvXJBWFVhclEhaT/yYKhTziO1GH3OPvZlB3H wkYzy3HzpO8MoP0ue7TpzPuJaTIdG30fSD/XiUq5y3rhc8AioIKFFmS386gaJA8h 4eOxhQQdtHpbfFxSg7kzp2n9KaHlguZqCzZn6VOpPbTCRBjvuV+cIFIgUpPFrIXc +7q5qtzLsRGZ2OXwFPfd3r8gqNXxIvl66BqEPXGXEobtEpdH5wOVV+TUz32AEYr6 XzYvnIYwdF9Dl5/Y/qnCmACWBduKeRvGFH6PRaV2f0d6YoyhP1QXgOsbrwW3mQbp WxPZm8/TSTco3gjm1u7CBac/iB3hr/N+t7OUy5PuxVW6dWI3u6aGO7PY9ovzgDOG iTAe4fCPkunTccPn1MDOMyDRzRpqqgv8JiSGj3jlnJs9BzC+gqcAUbgnkQKNQE1k PtMGmRnYWBuXU0WN8Y5TnvPmif8VQgmKNhBO2y13tXAyjW195kZnna/KUIWNQw84 YB0koK6xQblNcxjipFQt5ldYlPTFv7CXCiOzLH+X+BXUrKxp+fmSNlTBSOKJ5XDf nTCIxVLFBxB7DADtsHH8VOi0s8yCr/3xNSGJ50bAjLSPG1KNVN3LC+/FCvjxG1Dx FR0M5eiP9h6uZncZdAra =Fusc -----END PGP SIGNATURE----- --=-BVm0CXnSENNJNebfmTOo--