Date: Sun, 16 Sep 2007 16:30:08 GMT From: Hye-Shik Chang <perky@FreeBSD.ORG> To: freebsd-bugs@FreeBSD.org Subject: Re: gnu/116363: isspace broken for UTF-8 locales Message-ID: <200709161630.l8GGU7ef074926@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
The following reply was made to PR gnu/116363; it has been noted by GNATS.
From: Hye-Shik Chang <perky@FreeBSD.ORG>
To: Andrey Chernov <ache@nagual.pp.ru>, Petr Hroudny <petr.hroudny@gmail.com>,
freebsd-gnats-submit@FreeBSD.ORG, jkoshy@FreeBSD.ORG, i18n@FreeBSD.ORG
Cc:
Subject: Re: gnu/116363: isspace broken for UTF-8 locales
Date: Mon, 17 Sep 2007 01:22:14 +0900
On Sun, Sep 16, 2007 at 12:54:33PM +0400, Andrey Chernov wrote:
> On Sat, Sep 15, 2007 at 09:08:01AM +0000, Petr Hroudny wrote:
> >
> > >Number: 116363
> > >Category: gnu
> > >Synopsis: isspace broken for UTF-8 locales
> > >Confidential: no
> > >Severity: non-critical
> > >Priority: medium
> > >Responsible: freebsd-bugs
> > >State: open
> > >Quarter:
> > >Keywords:
> > >Date-Required:
> > >Class: sw-bug
> > >Submitter-Id: current-users
> > >Arrival-Date: Sat Sep 15 09:10:02 GMT 2007
> > >Closed-Date:
> > >Last-Modified:
> > >Originator: Petr Hroudny
> > >Release: 6-stable, 7-current
> > >Organization:
> > >Environment:
> > >Description:
> > In UTF-8 locales, isspace(0xA0) returns 1 which is wrong.
> >
> > In UTF-8, 0xA0 could only be the second or third byte of multibyte character, but never a space.
> >
> > As a consequence, operations like str.upper() and/or str.split() are broken, when
> > UTF-8 character with 0xA0 byte is encountered.
If you are saying about Python's str.split(), the problem is due
to our libc bug (or feature) which is described many times before,
and Python already includes a workaround for the problem.
http://mail.python.org/pipermail/python-checkins/2004-August/042343.html
> It seems that our UTF-8.src is completely wrong, it is just plain Unicode
> and not UTF-8 which multibyte values should start from
> C2-DF
> E0-EF
> F0-F4
> only (as stated in http://en.wikipedia.org/wiki/UTF-8 f.e.)
> Can anybody write replacement for it?
In fact, UTF-8.src defines values for not UTF-8 but Unicode codepoints.
Using the Unicode codepoint as wchar_t's internal representation gives
much benefit. I think we would be better to make isspace() and
other ctypes functions aware of "encoding". IIRC, tjr@ provided the
workaround as in the URL mentioned above and said that it would get
a chance to be fixed in 6 or 7 on 2004.
Hye-Shik
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200709161630.l8GGU7ef074926>