From owner-freebsd-bugs@FreeBSD.ORG Sun Sep 16 16:30:08 2007 Return-Path: Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5560B16A41A for ; Sun, 16 Sep 2007 16:30:08 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 3073713C4A8 for ; Sun, 16 Sep 2007 16:30:08 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.14.1/8.14.1) with ESMTP id l8GGU8oK074927 for ; Sun, 16 Sep 2007 16:30:08 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.1/8.14.1/Submit) id l8GGU7ef074926; Sun, 16 Sep 2007 16:30:08 GMT (envelope-from gnats) Date: Sun, 16 Sep 2007 16:30:08 GMT Message-Id: <200709161630.l8GGU7ef074926@freefall.freebsd.org> To: freebsd-bugs@FreeBSD.org From: Hye-Shik Chang Cc: Subject: Re: gnu/116363: isspace broken for UTF-8 locales X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Hye-Shik Chang List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Sep 2007 16:30:08 -0000 The following reply was made to PR gnu/116363; it has been noted by GNATS. From: Hye-Shik Chang To: Andrey Chernov , Petr Hroudny , freebsd-gnats-submit@FreeBSD.ORG, jkoshy@FreeBSD.ORG, i18n@FreeBSD.ORG Cc: Subject: Re: gnu/116363: isspace broken for UTF-8 locales Date: Mon, 17 Sep 2007 01:22:14 +0900 On Sun, Sep 16, 2007 at 12:54:33PM +0400, Andrey Chernov wrote: > On Sat, Sep 15, 2007 at 09:08:01AM +0000, Petr Hroudny wrote: > > > > >Number: 116363 > > >Category: gnu > > >Synopsis: isspace broken for UTF-8 locales > > >Confidential: no > > >Severity: non-critical > > >Priority: medium > > >Responsible: freebsd-bugs > > >State: open > > >Quarter: > > >Keywords: > > >Date-Required: > > >Class: sw-bug > > >Submitter-Id: current-users > > >Arrival-Date: Sat Sep 15 09:10:02 GMT 2007 > > >Closed-Date: > > >Last-Modified: > > >Originator: Petr Hroudny > > >Release: 6-stable, 7-current > > >Organization: > > >Environment: > > >Description: > > In UTF-8 locales, isspace(0xA0) returns 1 which is wrong. > > > > In UTF-8, 0xA0 could only be the second or third byte of multibyte character, but never a space. > > > > As a consequence, operations like str.upper() and/or str.split() are broken, when > > UTF-8 character with 0xA0 byte is encountered. If you are saying about Python's str.split(), the problem is due to our libc bug (or feature) which is described many times before, and Python already includes a workaround for the problem. http://mail.python.org/pipermail/python-checkins/2004-August/042343.html > It seems that our UTF-8.src is completely wrong, it is just plain Unicode > and not UTF-8 which multibyte values should start from > C2-DF > E0-EF > F0-F4 > only (as stated in http://en.wikipedia.org/wiki/UTF-8 f.e.) > Can anybody write replacement for it? In fact, UTF-8.src defines values for not UTF-8 but Unicode codepoints. Using the Unicode codepoint as wchar_t's internal representation gives much benefit. I think we would be better to make isspace() and other ctypes functions aware of "encoding". IIRC, tjr@ provided the workaround as in the URL mentioned above and said that it would get a chance to be fixed in 6 or 7 on 2004. Hye-Shik