From owner-freebsd-questions@FreeBSD.ORG Sun Apr 22 10:45:59 2012 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 505D01065675 for ; Sun, 22 Apr 2012 10:45:59 +0000 (UTC) (envelope-from m.seaman@infracaninophile.co.uk) Received: from smtp.infracaninophile.co.uk (smtp6.infracaninophile.co.uk [IPv6:2001:8b0:151:1:3cd3:cd67:fafa:3d78]) by mx1.freebsd.org (Postfix) with ESMTP id A480B8FC12 for ; Sun, 22 Apr 2012 10:45:58 +0000 (UTC) Received: from seedling.black-earth.co.uk (seedling.black-earth.co.uk [IPv6:2001:8b0:151:1:fa1e:dfff:feda:c0bb]) (authenticated bits=0) by smtp.infracaninophile.co.uk (8.14.5/8.14.5) with ESMTP id q3MAjr5x073833 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO) for ; Sun, 22 Apr 2012 11:45:54 +0100 (BST) (envelope-from m.seaman@infracaninophile.co.uk) X-DKIM: OpenDKIM Filter v2.5.2 smtp.infracaninophile.co.uk q3MAjr5x073833 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=infracaninophile.co.uk; s=201001-infracaninophile; t=1335091554; bh=X3u5tuakToPBxhUtkBAiENEaM9Q5R5ZcZE/MUVOQ7UI=; h=Date:From:To:Subject:References:In-Reply-To:Cc:Content-Type: Message-ID:Mime-Version; b=H1apqH3BGq/FANTxgwsglc39kZBo29e4TpNl37MTiA1obThTGS/UVqJJ2cvihPwXK NT84OFezsj+i2O38aeLYb9DHIY4SbxjHce2SV9JfXXU/Q67ZHX8f62TUXXrDzydUxG K2/8tGK5RLR8nrx9hGB9XmbFj8lvZ+zyJwjjVgIs= Message-ID: <4F93E159.7020807@infracaninophile.co.uk> Date: Sun, 22 Apr 2012 11:45:45 +0100 From: Matthew Seaman User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:11.0) Gecko/20120327 Thunderbird/11.0.1 MIME-Version: 1.0 To: freebsd-questions@freebsd.org References: <20120421055823.GA6788@tinyCurrent> <4F9253D7.7010609@locolomo.org> <4F9278A2.1020301@locolomo.org> <4F93CC95.5050209@locolomo.org> In-Reply-To: <4F93CC95.5050209@locolomo.org> X-Enigmail-Version: 1.4 OpenPGP: id=60AE908C Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="------------enigBE8224C100E6251ECE879E77" X-Virus-Scanned: clamav-milter 0.97.4 at lucid-nonsense.infracaninophile.co.uk X-Virus-Status: Clean X-Spam-Status: No, score=-2.1 required=5.0 tests=ALL_TRUSTED,AWL,BAYES_00, DKIM_ADSP_ALL,DKIM_SIGNED,T_DKIM_INVALID autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on lucid-nonsense.infracaninophile.co.uk Subject: Re: converting UTF-8 to HTML X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 22 Apr 2012 10:45:59 -0000 This is an OpenPGP/MIME signed message (RFC 2440 and 3156) --------------enigBE8224C100E6251ECE879E77 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 22/04/2012 10:17, Erik N=F8rgaard wrote: > UTF-8 is variable with, ascii characters are stored as single bytes (no= t > sure about iso-8859-1) while other characters are stored as two byte ch= ars. ascii uses the low 128 values that you can assign to an unsigned char, ie. those where the high-order bit is zero. iso-8859-1 and the various other iso-8859-X character sets fill in the remaining 128 characters with various other glyphs useful in latin alphabets, so it's still one char per glyph. Other alphabets (greek, cyrillic, etc) have similar one byte-per glyph encodings. But you have to know what the encoding is to display the content correctly, and it is difficult to mix chunks of text in different encodings in the same docume= nt. UTF has various different forms, based on different word sizes, but the commonly used UTF-8 works in units of 1-byte chars. However, glyphs may be represented by sequences of from 1 to 4 bytes. The 1-byte glyphs are identical to ascii. Any byte with the high-order bit set indicates the beginning of a multibyte glyph -- the number of bytes is indicated by the bit pattern of the first byte and all the other bytes of that glyph will have the high order bit set. All million-plus glyphs available through Unicode can be expressed this way, so the encoding is universal and suitable for all languages and alphabets or non-alphabetic languages.= Not all possible byte sequences are valid UTF-8 text, but the design of the encoding means that an interpreter can skip over an invalid sequence of bytes and find the beginning of the next valid sequence easily. Whoever it was upthread had the misfortune to run into a text editor that just gave up and truncated their document at an invalid sequence needs to vent their ire on the lazy and stupid programmers of whatever app it was, rather than on the concept of UTF-8 itself. Yes, with UTF-8 encoded text, you can no-longer equate the number of glyphs[*] in a piece of text (and hence the space required to display the text) with the memory required to store that text. There's a lot of legacy code out there which makes this assumption, but this is overshadowed by the amount of legacy code out there which can only handle ascii text. Fixing all that code is pretty long-winded, but not conceptually too difficult. Programming a text-only display to assume everything is UTF-8 would be quite viable, and backwardly compatible with ascii-only displays. The hard part is creating a font with a more-or-less complete set of Unicode glyphs. Cheers, Matthew [*] Let's not even mention the concept of 'combining characters' here. --=20 Dr Matthew J Seaman MA, D.Phil. 7 Priory Courtyard Flat 3 PGP: http://www.infracaninophile.co.uk/pgpkey Ramsgate JID: matthew@infracaninophile.co.uk Kent, CT11 9PW --------------enigBE8224C100E6251ECE879E77 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.16 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk+T4WAACgkQ8Mjk52CukIzOmgCfUBg1eIOuCYblRHCct3xBX7MZ eLYAn1oBBmoAi6DGL37siUceAboi9aGA =VQj4 -----END PGP SIGNATURE----- --------------enigBE8224C100E6251ECE879E77--