From owner-freebsd-arch@FreeBSD.ORG Sun Jan 25 19:17:40 2015 Return-Path: Delivered-To: arch@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1C9F5D74; Sun, 25 Jan 2015 19:17:40 +0000 (UTC) Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CBF29BC6; Sun, 25 Jan 2015 19:17:39 +0000 (UTC) Received: from slw by zxy.spb.ru with local (Exim 4.84 (FreeBSD)) (envelope-from ) id 1YFSgf-000Ov8-6V; Sun, 25 Jan 2015 22:17:33 +0300 Date: Sun, 25 Jan 2015 22:17:33 +0300 From: Slawa Olhovchenkov To: Peter Jeremy Subject: Re: [RFC] Set the default locale to en_US.UTF-8 Message-ID: <20150125191733.GS3698@zxy.spb.ru> References: <20150124143357.GI81001@ivaldir.etoilebsd.net> <20150125143243.GB76051@zxy.spb.ru> <7B1D8345-248B-4C44-9568-079BA29614C2@ixsystems.com> <20150125155000.GD76051@zxy.spb.ru> <20150125185951.GC23253@server.rulingia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150125185951.GC23253@server.rulingia.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-SA-Exim-Connect-IP: X-SA-Exim-Mail-From: slw@zxy.spb.ru X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false Cc: arch@FreeBSD.org, Baptiste Daroussin , Jordan Hubbard X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 25 Jan 2015 19:17:40 -0000 On Mon, Jan 26, 2015 at 05:59:51AM +1100, Peter Jeremy wrote: > On 2015-Jan-25 18:50:00 +0300, Slawa Olhovchenkov wrote: > >On Sun, Jan 25, 2015 at 06:58:13AM -0800, Jordan Hubbard wrote: > >> > On Jan 25, 2015, at 6:32 AM, Slawa Olhovchenkov wrote: > >> > > >> > NO! Please, NOT! > >> > Not all bytestring allowed in UTF-8, as result -- unpedicable failed > >> > execution of sed, grep, vi, ed and etc. > > I switched to en_AU.UTF-8 about 5 years ago with relatively little pain > (though I had very little non-ASCII text). > > The downside of UTF-8 in that random non-ASCII bytestrings are unlikely to > be valid UTF-8 and will therefore get rejected. About the only time I get > bitten by this is that my random password generator: > dd if=/dev/random bs=32 count=1 | tr -cd '!-~' > will die with an "tr: Illegal byte sequence" and needs a "LC_ALL=C" to > placate it. Yes, I now remeber -- other case will be tr. > >I am years use ru_RU.KOI8-R. Now I try use ru_RU.UTF8 and got some > >issuse (on 10-STABLE). 9.x and OS may have dufferent version of > >software and don't touch this. > > Once you've started using any 8-bit locale, switching to UTF-8 (or any > other 8-bit locale) will be a PITA because you need to re-encode everything. > And, since it's very difficult to run with multiple locales, you need to > do a complete sweep when you change locales. If you are running into > specific issues with incorrect handling of ru_RU.UTF8, that is a bug and > you need to report it. No, I don't have incorrect handling of ru_RU.UTF8 (for correct UTF8 files), I have trouble with processing non-utf files (like example with tr) > Note that we're talking about changing the default - you already override > the default so it won't affect you. This is dangerous change -- you can lost data, incorrectly proccess previosly correctly processed files (script with tr and etc.). And this may be surprised for you. > >This is (change from one-byte tu multi-bytes locale) may be do > >individualy, after inspecting systems. This is may be OK for new > >install, but not [automatic] for update/upgrade. > > Either an existing system has already overridden the default locale, so > changing the default will have no impact, or the treatment of non-ASCII > data is currently undefined so changing the default is changing undefined > behaviour to explicitly warning the the user that they have problems with > their data. Currently defaul locale is C. This locale accept any byte string. UTF8 locale may be more strict. This is may be break existing systems.