From owner-svn-soc-all@FreeBSD.ORG Tue Aug 26 20:38:44 2014 Return-Path: Delivered-To: svn-soc-all@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 1A5C256D for ; Tue, 26 Aug 2014 20:38:44 +0000 (UTC) Received: from socsvn.freebsd.org (socsvn.freebsd.org [IPv6:2001:1900:2254:206a::50:2]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 038AE3A9C for ; Tue, 26 Aug 2014 20:38:44 +0000 (UTC) Received: from socsvn.freebsd.org ([127.0.1.124]) by socsvn.freebsd.org (8.14.9/8.14.9) with ESMTP id s7QKchB9084370 for ; Tue, 26 Aug 2014 20:38:43 GMT (envelope-from ghostmansd@FreeBSD.org) Received: (from www@localhost) by socsvn.freebsd.org (8.14.9/8.14.9/Submit) id s7QKcfUB084347 for svn-soc-all@FreeBSD.org; Tue, 26 Aug 2014 20:38:41 GMT (envelope-from ghostmansd@FreeBSD.org) Date: Tue, 26 Aug 2014 20:38:41 GMT Message-Id: <201408262038.s7QKcfUB084347@socsvn.freebsd.org> X-Authentication-Warning: socsvn.freebsd.org: www set sender to ghostmansd@FreeBSD.org using -f From: ghostmansd@FreeBSD.org To: svn-soc-all@FreeBSD.org Subject: socsvn commit: r273177 - in soc2014/ghostmansd/head: bin/colldb lib/libc/locale lib/libc/string lib/libc/unicode lib/libcolldb share/examples/colldb MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-BeenThere: svn-soc-all@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: SVN commit messages for the entire Summer of Code repository List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 26 Aug 2014 20:38:44 -0000 Author: ghostmansd Date: Tue Aug 26 20:38:40 2014 New Revision: 273177 URL: http://svnweb.FreeBSD.org/socsvn/?view=rev&rev=273177 Log: significant fixes; source code reorganization; full support for root collation Added: soc2014/ghostmansd/head/bin/colldb/ soc2014/ghostmansd/head/bin/colldb/colldb.1 soc2014/ghostmansd/head/lib/libc/locale/unicode.h soc2014/ghostmansd/head/share/examples/colldb/ soc2014/ghostmansd/head/share/examples/colldb/colldb.py - copied unchanged from r273130, soc2014/ghostmansd/head/lib/libcolldb/colldb.py Deleted: soc2014/ghostmansd/head/lib/libcolldb/colldb.py Modified: soc2014/ghostmansd/head/lib/libc/locale/collate.c soc2014/ghostmansd/head/lib/libc/locale/collate.h soc2014/ghostmansd/head/lib/libc/locale/xlocale.c soc2014/ghostmansd/head/lib/libc/locale/xlocale_private.h soc2014/ghostmansd/head/lib/libc/string/strcoll.c soc2014/ghostmansd/head/lib/libc/string/strxfrm.c soc2014/ghostmansd/head/lib/libc/string/wcscoll.c soc2014/ghostmansd/head/lib/libc/string/wcsxfrm.c soc2014/ghostmansd/head/lib/libc/unicode/coll.h soc2014/ghostmansd/head/lib/libc/unicode/ucscoll.c soc2014/ghostmansd/head/lib/libc/unicode/ucsxfrm.c soc2014/ghostmansd/head/lib/libcolldb/colldb.3 Added: soc2014/ghostmansd/head/bin/colldb/colldb.1 ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ soc2014/ghostmansd/head/bin/colldb/colldb.1 Tue Aug 26 20:38:40 2014 (r273177) @@ -0,0 +1,49 @@ +.\"- +.\" Copyright (c) 2014 Dmitry Selyutin +.\" All rights reserved. +.\" +.\" Redistribution and use in source and binary forms, with or without +.\" modification, are permitted provided that the following conditions +.\" are met: +.\" 1. Redistributions of source code must retain the above copyright +.\" notice, this list of conditions and the following disclaimer. +.\" 2. Redistributions in binary form must reproduce the above copyright +.\" notice, this list of conditions and the following disclaimer in the +.\" documentation and/or other materials provided with the distribution. +.\" +.\" THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND +.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +.\" ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE +.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS +.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) +.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT +.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY +.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF +.\" SUCH DAMAGE. +.\" + +.Dd August 26, 2014 +.Dt COLLDB 1 +.Os +.Sh NAME +.Nm colldb +.Nd convert Unicode collation file to *BSD collation database +.Sh SYNOPSIS +.Nm +.Ar source_file target_file +.Sh DESCRIPTION +The +.Nm +utility converts Unicode collation file to *BSD collation database. +Such collation database are stored in network byte order, so they can be +reused on every system which supports Berkley Database format. +For example, root collation which is used as a fallback one, is expected +to be converted this way: +.Pp +.Dl # src="/usr/src/share/colldb/root.src" +.Dl # dst="/usr/share/locale/UTF-8/LC_COLLATE" +.Dl # colldb $src $dst +.Sh SEE ALSO +.Xr colldb 3 Modified: soc2014/ghostmansd/head/lib/libc/locale/collate.c ============================================================================== --- soc2014/ghostmansd/head/lib/libc/locale/collate.c Tue Aug 26 19:58:48 2014 (r273176) +++ soc2014/ghostmansd/head/lib/libc/locale/collate.c Tue Aug 26 20:38:40 2014 (r273177) @@ -51,290 +51,6 @@ #include "libc_private.h" -#define ROOT_COLLDB "/usr/share/locale/UTF-8/LC_COLLATE" - - -/* Collation Database. */ -struct __colldb { - uint32_t version; - void *handle; -}; - -void * -__colldb_create(const char *path, int mode) -{ - DBT key; - DBT value; - DB *db = NULL; - int error = 0; - void *colldb = NULL; - uint32_t version = 0; - int flags = (O_RDWR | O_CREAT | O_TRUNC); - - colldb = calloc(1, sizeof(struct __colldb)); - if (colldb == NULL) { - errno = ENOMEM; - return (NULL); - } - db = dbopen(path, flags, mode, DB_HASH, NULL); - if (db == NULL) { - error = errno; - free(colldb); - errno = error; - return (NULL); - } - ((struct __colldb*)colldb)->version = __COLLDB_VERSION; - - key.data = "TYPE"; - value.data = "COLLATION"; - key.size = (strlen("TYPE") + 1); - value.size = (strlen("COLLATION") + 1); - if (db->put(db, &key, &value, 0) == -1) { - error = errno; - goto failure; - } - - key.data = "VERSION"; - version = htonl(((struct __colldb*)colldb)->version); - value.data = &version; - key.size = (strlen("VERSION") + 1); - value.size = sizeof(((struct __colldb*)colldb)->version); - if (db->put(db, &key, &value, 0) == -1) { - error = errno; - goto failure; - } - - ((struct __colldb*)colldb)->handle = db; - return (colldb); - -failure: - (void) db->close(db); - free(colldb); - errno = error; - return (NULL); -} - -void * -__colldb_open(const char *path) -{ - DBT key; - DBT value; - DB *db = NULL; - int error = 0; - int state = 0; - void *colldb = NULL; - int flags = O_RDONLY; - - colldb = calloc(1, sizeof(struct __colldb)); - if (colldb == NULL) { - errno = ENOMEM; - return (NULL); - } - db = dbopen(path, flags, 0, DB_HASH, NULL); - if (db == NULL) { - error = errno; - free(colldb); - errno = error; - return (NULL); - } - - key.data = "TYPE"; - key.size = (strlen("TYPE") + 1); - state = db->get(db, &key, &value, 0); - if (state != 0) { - if (state < 0) - error = errno; - else - error = EFTYPE; - goto failure; - } - if (strcmp(value.data, "COLLATION") != 0) { - error = EFTYPE; - goto failure; - } - - key.data = "VERSION"; - key.size = (strlen("VERSION") + 1); - state = db->get(db, &key, &value, 0); - if (state != 0) { - if (state < 0) - error = errno; - else - error = EFTYPE; - goto failure; - } - ((struct __colldb*)colldb)->version = - ntohl(*(const uint32_t*)value.data); - - ((struct __colldb*)colldb)->handle = db; - return (colldb); - -failure: - (void) db->close(db); - free(colldb); - errno = error; - return (NULL); -} - -int -__colldb_close(void *colldb) -{ - DB *db = NULL; - int error = 0; - - if (colldb == NULL) { - errno = EINVAL; - return (-1); - } - db = ((struct __colldb*)colldb)->handle; - if (db == NULL) { - errno = EINVAL; - return (-1); - } - if (db->close(db) == -1) { - error = errno; - free(colldb); - errno = error; - return (-1); - } - free(colldb); - return (0); -} - -int -__colldb_sync(void *colldb) -{ - DB *db = NULL; - - if (colldb == NULL) { - errno = EINVAL; - return (-1); - } - db = ((struct __colldb*)colldb)->handle; - if (db == NULL) { - errno = EINVAL; - return (-1); - } - return db->sync(db, 0); -} - -int -__colldb_get(void *colldb, - struct __colldb_key *key, - struct __colldb_value *value) -{ - DBT dbkey; - DBT dbvalue; - DB *db = NULL; - size_t i = 0; - int error = 0; - int state = 0; - uint32_t *keybuf = NULL; - struct __colldb_weight *weights = NULL; - - if ((colldb == NULL) || (key == NULL) || (value == NULL)) { - errno = EINVAL; - return (-1); - } - db = ((struct __colldb*)colldb)->handle; - if ((db == NULL) || (key->chars == NULL) || (key->count == 0)) { - errno = EINVAL; - return (-1); - } - - keybuf = malloc(key->count * sizeof(*key->chars)); - if (keybuf == NULL) { - errno = ENOMEM; - return (-1); - } - for (i = 0; i < key->count; ++i) - keybuf[i] = htonl(key->chars[i]); - - dbkey.data = keybuf; - dbkey.size = (key->count * sizeof(*key->chars)); - state = db->get(db, &dbkey, &dbvalue, 0); - if (state != 0) { - error = errno; - free(keybuf); - errno = error; - return (state); - } - - weights = dbvalue.data; - if ((dbvalue.size / sizeof(*weights)) > value->count) { - free(keybuf); - errno = ERANGE; - return (-1); - } - value->count = (dbvalue.size / sizeof(*weights)); - for (i = 0; i < value->count; ++i) { - value->weights[i].alternate = weights[i].alternate; - value->weights[i].level1 = ntohl(weights[i].level1); - value->weights[i].level2 = ntohl(weights[i].level2); - value->weights[i].level3 = ntohl(weights[i].level3); - value->weights[i].level4 = ntohl(weights[i].level4); - } - free(dbvalue.data); - free(keybuf); - return (0); -} - -int -__colldb_put(void *colldb, - struct __colldb_key *key, - struct __colldb_value *value) -{ - DBT dbkey; - DBT dbvalue; - DB *db = NULL; - size_t i = 0; - int state = 0; - int error = 0; - uint32_t *keybuf = NULL; - struct __colldb_weight *valuebuf = NULL; - - if ((colldb == NULL) || (key == NULL) || (value == NULL)) { - errno = EINVAL; - return (-1); - } - db = ((struct __colldb*)colldb)->handle; - if ((db == NULL) || (key->chars == NULL) || (key->count == 0)) { - errno = EINVAL; - return (-1); - } - - valuebuf = value->weights; - keybuf = malloc(key->count * sizeof(*key->chars)); - valuebuf = malloc(value->count * sizeof(*value->weights)); - if ((keybuf == NULL) || (valuebuf == NULL)) { - errno = ENOMEM; - return (-1); - } - for (i = 0; i < key->count; ++i) - keybuf[i] = htonl(key->chars[i]); - for (i = 0; i < value->count; ++i) { - valuebuf[i].alternate = value->weights[i].alternate; - valuebuf[i].level1 = htonl(value->weights[i].level1); - valuebuf[i].level2 = htonl(value->weights[i].level2); - valuebuf[i].level3 = htonl(value->weights[i].level3); - valuebuf[i].level4 = htonl(value->weights[i].level4); - } - - dbkey.data = keybuf; - dbvalue.data = valuebuf; - dbkey.size = (key->count * sizeof(*key->chars)); - dbvalue.size = (value->count * sizeof(*value->weights)); - state = db->put(db, &dbkey, &dbvalue, 0); - if (state != 0) { - error = errno; - free(keybuf); - free(valuebuf); - errno = error; - } - return state; -} - - /* * To avoid modifying the original (single-threaded) code too much, we'll just * define the old globals as fields inside the table. @@ -349,6 +65,26 @@ int __collate_load_error; +void *__colldb_root = NULL; + +__attribute__((constructor)) +static void root_construct(void) +{ + const int error_init = errno; + + __colldb_root = __colldb_open(__COLLDB_ROOT); + if (__colldb_root == NULL) + errno = error_init; +} + +__attribute__((destructor)) +static void root_destructor(void) +{ + __colldb_close(__colldb_root); +} + + + struct xlocale_collate __xlocale_global_collate = { {{0}, "C"}, NULL, 1, 0 }; @@ -370,7 +106,7 @@ free(__collate_chain_pri_table); __collate_chain_pri_table = NULL; } - if (table->__colldb != NULL) { + if ((table->__colldb != NULL) && (table->__colldb != __colldb_root)) { __colldb_close(table->__colldb); table->__colldb = NULL; } @@ -411,7 +147,6 @@ FILE *fp; int i, saverr, chains; uint32_t u32; - void *colldb = NULL; char strbuf[STR_LEN], buf[PATH_MAX]; void *TMP_substitute_table, *TMP_char_pri_table, *TMP_chain_pri_table; @@ -429,15 +164,19 @@ (void)strcat(buf, "/LC_COLLATE"); /* - * Try to use collation database for the given locale. - * If collation database does not exist for the given locale, - * try to use root collation database. - * If it fails too, use original collation algorithm. + * Collation functions are going to use both root and custom collation + * databases, so both must be opened for each locale. The first one is + * used as a fallback one in case if match is not found inside custom + * database. If root collation database is not found, use previous + * collation algorithms as fallback. */ saverr = errno; - if ((table->__colldb = __colldb_open(buf)) == NULL) - table->__colldb = __colldb_open(ROOT_COLLDB); - if (colldb != NULL) + table->__colldb = __colldb_open(buf); + if (table->__colldb == NULL) { + table->__colldb = __colldb_root; + errno = saverr; + } + if (table->__colldb != NULL) return (_LDP_LOADED); errno = saverr; Modified: soc2014/ghostmansd/head/lib/libc/locale/collate.h ============================================================================== --- soc2014/ghostmansd/head/lib/libc/locale/collate.h Tue Aug 26 19:58:48 2014 (r273176) +++ soc2014/ghostmansd/head/lib/libc/locale/collate.h Tue Aug 26 20:38:40 2014 (r273177) @@ -46,38 +46,6 @@ #define COLLATE_VERSION1_2 "1.2\n" -#define __COLLDB_VERSION 0x00000001 -#define __COLLDB_WEIGHTS_MAX 16 -struct __colldb_weight { - uint8_t alternate; - uint32_t level1; - uint32_t level2; - uint32_t level3; - uint32_t level4; -}; -struct __colldb_key { - size_t count; - const uint32_t *chars; -}; -struct __colldb_value { - size_t count; - struct __colldb_weight *weights; -}; -void* __colldb_create(const char*, int mode); -void* __colldb_open(const char*); -int __colldb_close(void*); -int __colldb_sync(void*); -int __colldb_get(void*, - struct __colldb_key*, - struct __colldb_value*); -int __colldb_put(void*, - struct __colldb_key*, - struct __colldb_value*); - -size_t __ucsxfrm(uint32_t*, const uint32_t*, size_t, void*); -int __ucscoll(const uint32_t*, const uint32_t*, void*); - - struct __collate_st_char_pri { int prim, sec; }; Added: soc2014/ghostmansd/head/lib/libc/locale/unicode.h ============================================================================== --- /dev/null 00:00:00 1970 (empty, because file is newly added) +++ soc2014/ghostmansd/head/lib/libc/locale/unicode.h Tue Aug 26 20:38:40 2014 (r273177) @@ -0,0 +1,79 @@ +/*- + * Copyright (c) 2014 Dmitry Selyutin. + * All rights reserved. + * + * Redistribution and use in source and binary forms, with or without + * modification, are permitted provided that the following conditions + * are met: + * 1. Redistributions of source code must retain the above copyright + * notice, this list of conditions and the following disclaimer. + * 2. Redistributions in binary form must reproduce the above copyright + * notice, this list of conditions and the following disclaimer in the + * documentation and/or other materials provided with the distribution. + * + * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND + * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE + * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE + * ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE + * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL + * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS + * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) + * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT + * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY + * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF + * SUCH DAMAGE. + */ + +#ifndef _UNICODE_H_ +#define _UNICODE_H_ + + +/* Unicode collation. */ +#define __COLLDB_ROOT "/usr/share/locale/UTF-8/LC_COLLATE" +#define __COLLDB_VERSION 0x00000001 +#define __COLLDB_WEIGHTS_MAX 16 +extern void *__colldb_root; +struct __colldb_weight { + uint8_t alternate; + uint32_t level1; + uint32_t level2; + uint32_t level3; + uint32_t level4; +}; +struct __colldb_key { + size_t count; + const uint32_t *chars; +}; +struct __colldb_value { + size_t count; + struct __colldb_weight *weights; +}; +void* __colldb_create(const char*, int mode); +void* __colldb_open(const char*); +int __colldb_close(void*); +int __colldb_sync(void*); +int __colldb_get(void*, + struct __colldb_key*, + struct __colldb_value*); +int __colldb_put(void*, + struct __colldb_key*, + struct __colldb_value*); + +size_t __ucsxfrm(uint32_t*, const uint32_t*, size_t, void*); +int __ucscoll(const uint32_t*, const uint32_t*, void*); + + +/* Unicode canonicalization. */ +int __uccclass(uint32_t); +int __ucscanon(uint32_t*); + + +/* Unicode normalization. */ +#define __UC_NFD 1 +#define __UC_NFKD 2 +#define __UC_NFC 3 +#define __UC_NFKC 4 +size_t __ucsnorm(uint32_t*, const uint32_t*, size_t, int); + + +#endif /* _UNICODE_H_ */ Modified: soc2014/ghostmansd/head/lib/libc/locale/xlocale.c ============================================================================== --- soc2014/ghostmansd/head/lib/libc/locale/xlocale.c Tue Aug 26 19:58:48 2014 (r273176) +++ soc2014/ghostmansd/head/lib/libc/locale/xlocale.c Tue Aug 26 20:38:40 2014 (r273177) @@ -39,10 +39,293 @@ #include #include #include +#include +#include +#include +#include #include "libc_private.h" #include "xlocale_private.h" +/* Collation Database. */ +struct __colldb { + uint32_t version; + void *handle; +}; + +void * +__colldb_create(const char *path, int mode) +{ + DBT key; + DBT value; + DB *db = NULL; + int error = 0; + void *colldb = NULL; + uint32_t version = 0; + int flags = (O_RDWR | O_CREAT | O_TRUNC); + + colldb = calloc(1, sizeof(struct __colldb)); + if (colldb == NULL) { + errno = ENOMEM; + return (NULL); + } + db = dbopen(path, flags, mode, DB_HASH, NULL); + if (db == NULL) { + error = errno; + free(colldb); + errno = error; + return (NULL); + } + ((struct __colldb*)colldb)->version = __COLLDB_VERSION; + + key.data = "TYPE"; + value.data = "COLLATION"; + key.size = (strlen("TYPE") + 1); + value.size = (strlen("COLLATION") + 1); + if (db->put(db, &key, &value, 0) == -1) { + error = errno; + goto failure; + } + + key.data = "VERSION"; + version = htonl(((struct __colldb*)colldb)->version); + value.data = &version; + key.size = (strlen("VERSION") + 1); + value.size = sizeof(((struct __colldb*)colldb)->version); + if (db->put(db, &key, &value, 0) == -1) { + error = errno; + goto failure; + } + + ((struct __colldb*)colldb)->handle = db; + return (colldb); + +failure: + (void) db->close(db); + free(colldb); + errno = error; + return (NULL); +} + +void * +__colldb_open(const char *path) +{ + DBT key; + DBT value; + DB *db = NULL; + int error = 0; + int state = 0; + void *colldb = NULL; + int flags = O_RDONLY; + + colldb = calloc(1, sizeof(struct __colldb)); + if (colldb == NULL) { + errno = ENOMEM; + return (NULL); + } + db = dbopen(path, flags, 0, DB_HASH, NULL); + if (db == NULL) { + error = errno; + free(colldb); + errno = error; + return (NULL); + } + + key.data = "TYPE"; + key.size = (strlen("TYPE") + 1); + state = db->get(db, &key, &value, 0); + if (state != 0) { + if (state < 0) + error = errno; + else + error = EFTYPE; + goto failure; + } + if (strcmp(value.data, "COLLATION") != 0) { + error = EFTYPE; + goto failure; + } + + key.data = "VERSION"; + key.size = (strlen("VERSION") + 1); + state = db->get(db, &key, &value, 0); + if (state != 0) { + if (state < 0) + error = errno; + else + error = EFTYPE; + goto failure; + } + ((struct __colldb*)colldb)->version = + ntohl(*(const uint32_t*)value.data); + + ((struct __colldb*)colldb)->handle = db; + return (colldb); + +failure: + (void) db->close(db); + free(colldb); + errno = error; + return (NULL); +} + +int +__colldb_close(void *colldb) +{ + DB *db = NULL; + int error = 0; + + if (colldb == NULL) + return (0); + db = ((struct __colldb*)colldb)->handle; + if (db == NULL) { + errno = EINVAL; + return (-1); + } + if (db->close(db) == -1) { + error = errno; + free(colldb); + errno = error; + return (-1); + } + free(colldb); + return (0); +} + +int +__colldb_sync(void *colldb) +{ + DB *db = NULL; + + if (colldb == NULL) { + errno = EINVAL; + return (-1); + } + db = ((struct __colldb*)colldb)->handle; + if (db == NULL) { + errno = EINVAL; + return (-1); + } + return db->sync(db, 0); +} + +int +__colldb_get(void *colldb, + struct __colldb_key *key, + struct __colldb_value *value) +{ + DBT dbkey; + DBT dbvalue; + DB *db = NULL; + size_t i = 0; + int error = 0; + int state = 0; + uint32_t *keybuf = NULL; + struct __colldb_weight *weights = NULL; + + if ((colldb == NULL) || (key == NULL) || (value == NULL)) { + errno = EINVAL; + return (-1); + } + db = ((struct __colldb*)colldb)->handle; + if ((db == NULL) || (key->chars == NULL) || (key->count == 0)) { + errno = EINVAL; + return (-1); + } + + keybuf = malloc(key->count * sizeof(*key->chars)); + if (keybuf == NULL) { + errno = ENOMEM; + return (-1); + } + for (i = 0; i < key->count; ++i) + keybuf[i] = htonl(key->chars[i]); + + dbkey.data = keybuf; + dbkey.size = (key->count * sizeof(*key->chars)); + state = db->get(db, &dbkey, &dbvalue, 0); + if (state != 0) { + error = errno; + free(keybuf); + errno = error; + return (state); + } + + weights = dbvalue.data; + if ((dbvalue.size / sizeof(*weights)) > value->count) { + free(keybuf); + errno = ERANGE; + return (-1); + } + value->count = (dbvalue.size / sizeof(*weights)); + for (i = 0; i < value->count; ++i) { + value->weights[i].alternate = weights[i].alternate; + value->weights[i].level1 = ntohl(weights[i].level1); + value->weights[i].level2 = ntohl(weights[i].level2); + value->weights[i].level3 = ntohl(weights[i].level3); + value->weights[i].level4 = ntohl(weights[i].level4); + } + free(dbvalue.data); + free(keybuf); + return (0); +} + +int +__colldb_put(void *colldb, + struct __colldb_key *key, + struct __colldb_value *value) +{ + DBT dbkey; + DBT dbvalue; + DB *db = NULL; + size_t i = 0; + int state = 0; + int error = 0; + uint32_t *keybuf = NULL; + struct __colldb_weight *valuebuf = NULL; + + if ((colldb == NULL) || (key == NULL) || (value == NULL)) { + errno = EINVAL; + return (-1); + } + db = ((struct __colldb*)colldb)->handle; + if ((db == NULL) || (key->chars == NULL) || (key->count == 0)) { + errno = EINVAL; + return (-1); + } + + valuebuf = value->weights; + keybuf = malloc(key->count * sizeof(*key->chars)); + valuebuf = malloc(value->count * sizeof(*value->weights)); + if ((keybuf == NULL) || (valuebuf == NULL)) { + errno = ENOMEM; + return (-1); + } + for (i = 0; i < key->count; ++i) + keybuf[i] = htonl(key->chars[i]); + for (i = 0; i < value->count; ++i) { + valuebuf[i].alternate = value->weights[i].alternate; + valuebuf[i].level1 = htonl(value->weights[i].level1); + valuebuf[i].level2 = htonl(value->weights[i].level2); + valuebuf[i].level3 = htonl(value->weights[i].level3); + valuebuf[i].level4 = htonl(value->weights[i].level4); + } + + dbkey.data = keybuf; + dbvalue.data = valuebuf; + dbkey.size = (key->count * sizeof(*key->chars)); + dbvalue.size = (value->count * sizeof(*value->weights)); + state = db->put(db, &dbkey, &dbvalue, 0); + if (state != 0) { + error = errno; + free(keybuf); + free(valuebuf); + errno = error; + } + return (state); +} + + /** * Each locale loader declares a global component. This is used by setlocale() * and also by xlocale with LC_GLOBAL_LOCALE.. @@ -88,7 +371,6 @@ &__xlocale_global_time, &__xlocale_global_messages }, - NULL, 1, 0, 1, @@ -102,7 +384,6 @@ &__xlocale_C_ctype, 0, 0, 0, 0 }, - NULL, 1, 0, 1, Modified: soc2014/ghostmansd/head/lib/libc/locale/xlocale_private.h ============================================================================== --- soc2014/ghostmansd/head/lib/libc/locale/xlocale_private.h Tue Aug 26 19:58:48 2014 (r273176) +++ soc2014/ghostmansd/head/lib/libc/locale/xlocale_private.h Tue Aug 26 20:38:40 2014 (r273177) @@ -40,17 +40,7 @@ #include #include #include "setlocale.h" - -/* Unicode canonicalization. */ -int __uccclass(uint32_t); -int __ucscanon(uint32_t*); - -/* Unicode normalization. */ -#define __UC_NFD 1 -#define __UC_NFKD 2 -#define __UC_NFC 3 -#define __UC_NFKC 4 -size_t __ucsnorm(uint32_t*, const uint32_t*, size_t, int); +#include "unicode.h" enum { Modified: soc2014/ghostmansd/head/lib/libc/string/strcoll.c ============================================================================== --- soc2014/ghostmansd/head/lib/libc/string/strcoll.c Tue Aug 26 19:58:48 2014 (r273176) +++ soc2014/ghostmansd/head/lib/libc/string/strcoll.c Tue Aug 26 20:38:40 2014 (r273177) @@ -35,12 +35,13 @@ #include #include +#include #include #include "collate.h" static wchar_t * -__wcsdup(const char *s) +__wcsdup(const char *s, locale_t locale) { static const mbstate_t initial; mbstate_t st = initial; @@ -50,12 +51,12 @@ cp = s; st = initial; - if ((len = mbsrtowcs(NULL, &cp, 0, &st)) == (size_t)-1) + if ((len = mbsrtowcs_l(NULL, &cp, 0, &st, locale)) == (size_t)-1) return (NULL); if ((wcs = malloc((len + 1) * sizeof(wchar_t))) == NULL) return (NULL); st = initial; - mbsrtowcs(wcs, &s, len + 1, &st); + mbsrtowcs_l(wcs, &s, len + 1, &st, locale); return (wcs); } @@ -64,18 +65,53 @@ int strcoll_l(const char *s, const char *s2, locale_t locale) { + const int error_init = errno; + int len = 0; + int len2 = 0; + int prim = 0; + int prim2 = 0; + int sec = 0; + int sec2 = 0; + int ret = 0; + int ret2 = 0; + int error = 0; + char *tt = NULL; + char *tt2 = NULL; wchar_t *ws = NULL; wchar_t *ws2 = NULL; - int len, len2, prim, prim2, sec, sec2, ret, ret2; - const char *t, *t2; - char *tt, *tt2; - FIX_LOCALE(locale); - struct xlocale_collate *table = - (struct xlocale_collate*)locale->components[XLC_COLLATE]; + const char *t = NULL; + const char *t2 = NULL; + struct xlocale_collate *table = NULL; + FIX_LOCALE(locale); + table = (struct xlocale_collate*)locale->components[XLC_COLLATE]; if (table->__collate_load_error) return strcmp(s, s2); + do { + if (table->__colldb != NULL) { + ws = __wcsdup(s, locale); + ws2 = __wcsdup(s2, locale); + if ((ws == NULL) || (ws2 == NULL)) { + errno = error_init; + free(ws); + break; + } + error = errno; + errno = 0; + ret = wcscoll_l(ws, ws2, locale); + if (errno != 0) { + errno = error_init; + free(ws); + free(ws2); + break; + } + free(ws); + free(ws2); + return (ret); + } + } while (0); + len = len2 = 1; ret = ret2 = 0; if (table->__collate_substitute_nontrivial) { @@ -118,6 +154,7 @@ return ret; } + int strcoll(const char *s, const char *s2) { Modified: soc2014/ghostmansd/head/lib/libc/string/strxfrm.c ============================================================================== --- soc2014/ghostmansd/head/lib/libc/string/strxfrm.c Tue Aug 26 19:58:48 2014 (r273176) +++ soc2014/ghostmansd/head/lib/libc/string/strxfrm.c Tue Aug 26 20:38:40 2014 (r273177) @@ -34,28 +34,49 @@ __FBSDID("$FreeBSD$"); #include +#include #include +#include #include "collate.h" -size_t -strxfrm_l(char * __restrict dest, const char * __restrict src, size_t len, locale_t loc); -size_t -strxfrm(char * __restrict dest, const char * __restrict src, size_t len) +static wchar_t * +__wcsdup(const char *s, locale_t locale) { - return strxfrm_l(dest, src, len, __get_locale()); + static const mbstate_t initial; + mbstate_t st = initial; + size_t len = 0; + wchar_t *wcs = NULL; + const char *cp = s; + + cp = s; + st = initial; + if ((len = mbsrtowcs_l(NULL, &cp, 0, &st, locale)) == (size_t)-1) *** DIFF OUTPUT TRUNCATED AT 1000 LINES ***