From owner-freebsd-hackers@freebsd.org Sat Jan 20 23:59:51 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 56D1BECEC1E for ; Sat, 20 Jan 2018 23:59:51 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) Received: from sonic309-21.consmr.mail.ne1.yahoo.com (sonic309-21.consmr.mail.ne1.yahoo.com [66.163.184.147]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2A1A570FBA for ; Sat, 20 Jan 2018 23:59:50 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) X-YMail-OSG: G3DJ7PQVM1n5ncjEYlrmnpHQKiEMKCzkt6ts4rQAGKGjXe4zYGsjW7193BcjCCJ xYTZoVjliu1tL8MGFG_BChOw5ReWrRFQBCYe_67jF.wnX09LTlpsxqm5.mDTvV5p26bQ4vtlLWgH BG0lMreIYly5qs2gCH11ksnT3jvpBgeyJTBoX92X2yAe62WLP7GhK3j6rrTN4dmdFGT_eumMSd8_ _vnBo.ohP59CZL4oyfrx1hDcmpmdYLSLpkmawKvVwG2rsB9Ct_M8IRD7CrnfB.R7KsjsUqRdq7aL XqvyB3_cYiqdTsiWv.ouMxqbKXMmWTkGpmow.zxQT7JO2ilNWnrcxl.aq2q.32Bty44otGAWPaZU vW9ew4j3YbaPXaM.QRNFdb4mOVhd5ZM3oF.Q3LtK9XNJJHW1b_9II1xzXsJAkCtE2EMM68d41pgr 3.a48UyXfe3WS0pIA3TzQPuuJqpQ1v0puOqp6ZXfz0tzN9_aWLMO4RLiCmcwFPQEOSMge Received: from sonic.gate.mail.ne1.yahoo.com by sonic309.consmr.mail.ne1.yahoo.com with HTTP; Sat, 20 Jan 2018 23:59:44 +0000 Received: from smtpgate103.mail.ne1.yahoo.com (EHLO [192.168.1.25]) ([216.155.193.162]) by smtp416.mail.ne1.yahoo.com (JAMES SMTP Server ) with ESMTPA ID 3966b0e8031e630d539b8a5755cf2897; Sat, 20 Jan 2018 23:59:40 +0000 (UTC) From: Mark Millard Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: Attribute alloc__size use and clang 5.0.1 vs. gcc7 (e.g.): __builtin_object_size(p,1) and __builtin_object_size(p,3) disagreements result Date: Sat, 20 Jan 2018 15:59:38 -0800 References: <1AA0993D-81E4-4DC0-BBD9-CC42B26ADB1C@yahoo.com> To: Pedro Giffuni , FreeBSD Toolchain , FreeBSD Hackers In-Reply-To: <1AA0993D-81E4-4DC0-BBD9-CC42B26ADB1C@yahoo.com> Message-Id: X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 20 Jan 2018 23:59:51 -0000 [Noting a typo in the program source, and so in the output text: the 2nd occurance of: "my_calloc_alt0 should have been: "my_calloc_alt1 . Hand edited corrections below for clarity.] On 2018-Jan-20, at 3:27 PM, Mark Millard = wrote: > [Bugzilla 225197 indirectly lead to this. > Avoiding continuing there.] >=20 > I decided to compare some alternate uses of > __attribute__((alloc_size(. . .))) compiled > and run under clang 5.0.1 and gcc7. I did not > get what I expected based on prior discussion > material. >=20 > This is an FYI since I do not know how important > the distinctions that I found are. >=20 > Here is the quick program: >=20 > # more alloc_size_attr_test.c=20 > #include > #include >=20 > __attribute__((alloc_size(1,2))) > void* my_calloc_alt0(size_t n, size_t s) > { > void* p =3D calloc(n,s); > printf("calloc __builtin_object_size 0,1,2,3: %ld, %ld, %ld, %ld\n" > ,(long) __builtin_object_size(p, 0) > ,(long) __builtin_object_size(p, 1) > ,(long) __builtin_object_size(p, 2) > ,(long) __builtin_object_size(p, 3) > ); > return p; > } >=20 > __attribute__((alloc_size(1))) __attribute__((alloc_size(2))) > void* my_calloc_alt1(size_t n, size_t s) > { > void* p =3D calloc(n,s); > printf("calloc __builtin_object_size 0,1,2,3: %ld, %ld, %ld, %ld\n" > ,(long) __builtin_object_size(p, 0) > ,(long) __builtin_object_size(p, 1) > ,(long) __builtin_object_size(p, 2) > ,(long) __builtin_object_size(p, 3) > ); > return p; > } >=20 > int main() > { > void* p =3D my_calloc_alt0(2,7); > printf("my_calloc_alt0 __builtin_object_size 0,1,2,3: %ld, %ld, %ld, = %ld\n" > ,(long) __builtin_object_size(p, 0) > ,(long) __builtin_object_size(p, 1) > ,(long) __builtin_object_size(p, 2) > ,(long) __builtin_object_size(p, 3) > ); > void* q =3D my_calloc_alt1(2,7); > printf("my_calloc_alt0 __builtin_object_size 0,1,2,3: %ld, %ld, %ld, = %ld\n" The above line should have been: printf("my_calloc_alt1 __builtin_object_size 0,1,2,3: %ld, %ld, %ld, = %ld\n" > ,(long) __builtin_object_size(q, 0) > ,(long) __builtin_object_size(q, 1) > ,(long) __builtin_object_size(q, 2) > ,(long) __builtin_object_size(q, 3) > ); > } >=20 > # uname -apKU > FreeBSD FBSDFSSD 12.0-CURRENT FreeBSD 12.0-CURRENT r327485M amd64 = amd64 1200054 1200054 >=20 > The system-clang 5.0.1 result was: >=20 > # clang -O2 alloc_size_attr_test.c The later outputs are edited for clarity: > # ./a.out > calloc __builtin_object_size 0,1,2,3: 14, 14, 14, 0 > my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 > calloc __builtin_object_size 0,1,2,3: 14, 14, 14, 0 my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >=20 > The lang/gcc7 result was: >=20 > # gcc7 -O2 alloc_size_attr_test.c >=20 > # ./a.out > calloc __builtin_object_size 0,1,2,3: -1, -1, 0, 0 > my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 14 > calloc __builtin_object_size 0,1,2,3: -1, -1, 0, 0 my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 7, 14, 14 >=20 > I'll ignore that gcc does not provide actual sizes > via __builtin_object_size for calloc use. >=20 > Pairing the other lines for easy comparison, with > some notes mixed in: >=20 > __attribute__((alloc_size(1,2))) style: > my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 (system = clang) > my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 14 (gcc7) >=20 > __attribute__((alloc_size(1))) __attribute__((alloc_size(2))) style: my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 (system = clang) my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 7, 14, 14 (gcc7) >=20 > Thus. . . >=20 > For __attribute__((alloc_size(1))) __attribute__((alloc_size(2))): > __builtin_object_size(p,1) is not equivalent (clang vs. gcc7) >=20 > For both of the alloc_size usage styles: > __builtin_object_size(p,3) is not equivalent (clang vs. gcc7) >=20 > This means that the two style of alloc_size use are not > equivalent across some major compilers/toolchains. >=20 > But I do not know if either of the differences is a problem or > not. >=20 >=20 > Note: without a sufficient -O all the figures can be > the mix of -1's and 0's. =3D=3D=3D Mark Millard marklmi at yahoo.com ( markmi at dsl-only.net is going away in 2018-Feb, late) From owner-freebsd-hackers@freebsd.org Sun Jan 21 01:05:57 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 57E3CED22BC for ; Sun, 21 Jan 2018 01:05:57 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from sonic312-37.consmr.mail.ne1.yahoo.com (sonic312-37.consmr.mail.ne1.yahoo.com [66.163.191.218]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2965E7338D for ; Sun, 21 Jan 2018 01:05:56 +0000 (UTC) (envelope-from pfg@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1516496750; bh=G5Q6Fpc/nM9NOPjWGBD1moffyv9tb+m82MDIwe44XOs=; h=Subject:To:References:From:Date:In-Reply-To:From:Subject; b=nO1r2DKFoDPudVZyERTC0bmQ2z5tuWRbgHhY3OmhntQ4/JonkOFZ7911qHh+UYEV5EoHH5GXRQSP1lMvU0A7ZTVZx/piAaqirXq6Lmi/oXMFDIAVau+JW7a3UWDAxIyHjSEzdCWFpVd827HbokglUhAZE5C0vmi93DOav4nPtJPAxE0kO/wcAe8gEVcZu+//isWyPct0GixkyRO0NJ11p5detW30pJAB1IINYkwgr9VcNS9RMB3GyUjFbUS8+So+cF1RlVX7KzxjQgVcfpEBakMwGV9sewlnRhg+6nqJZ6ZtDQOfjesinm6JEKvIEooFuW5GeHeqOyQmxx64tYJEfQ== X-YMail-OSG: 1WAF3FoVM1mWIts4T6Esu_LtUSHVBK8xXY47nQ6mQxBnzRss3wnYplZkFeOZrg7 79RDFxvnivGReqgG031rNMXVQWc7o5BEaUqdI48CqBlhE0NFeQ8kxYAc5Dira.lHkb6YLqDm6SKS c.ebFp3XsGXZ90uik5t3WzredpwisHUPH5iCpyV7iIRahKWr4gETS_FPpU9A9OjPKBV9KxWSiUs8 Yk5NKtNhiJqudlZD97tVlNbccMFGqg9k7vZjWLp3DzFGGXmXl5LoLXfhHuhrBgbuAmxP5HQ1msD4 E_kKehUjTbXstKmKGF8AAmwjRLPXwbTyqCG9ZmSJIYQU_QQuBeyucaahFjXdZ8S6c8hLWFF_ynFk xfFZDbWQZDmHUBK6r3WaUTVieNaklc0oHDUWoAz4PMMMJwAwfloJ.mO6jZ11jXBu10t_zqKpzwL6 Y8BGXxavay1HANqFIeJ.EgxjK_wcZ9sbWe_gwnMcBehZGb5btYKVTB_9PSZWU18Opxc4Ry2ryqFT AlaNXNu.8uQ-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic312.consmr.mail.ne1.yahoo.com with HTTP; Sun, 21 Jan 2018 01:05:50 +0000 Received: from smtp235.mail.ne1.yahoo.com (EHLO [192.168.0.8]) ([10.218.253.206]) by smtp406.mail.ne1.yahoo.com (JAMES SMTP Server ) with ESMTPA ID 57f354ca5d4b331749f470663f89717a; Sun, 21 Jan 2018 01:05:48 +0000 (UTC) Subject: Re: Attribute alloc__size use and clang 5.0.1 vs. gcc7 (e.g.): __builtin_object_size(p,1) and __builtin_object_size(p,3) disagreements result To: Mark Millard , FreeBSD Toolchain , FreeBSD Hackers References: <1AA0993D-81E4-4DC0-BBD9-CC42B26ADB1C@yahoo.com> From: Pedro Giffuni Message-ID: <8c4c5985-885a-abf7-93df-0698c645d36e@FreeBSD.org> Date: Sat, 20 Jan 2018 20:05:48 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jan 2018 01:05:57 -0000 Very interesting , thanks for running such tests ... On 01/20/18 18:59, Mark Millard wrote: > [Noting a typo in the program source, and > so in the output text: the 2nd occurance of: "my_calloc_alt0 > should have been: "my_calloc_alt1 > . Hand edited corrections below for clarity.] > > On 2018-Jan-20, at 3:27 PM, Mark Millard wrote: > >> [Bugzilla 225197 indirectly lead to this. >> Avoiding continuing there.] >> >> I decided to compare some alternate uses of >> __attribute__((alloc_size(. . .))) compiled >> and run under clang 5.0.1 and gcc7. I did not >> get what I expected based on prior discussion >> material. >> >> This is an FYI since I do not know how important >> the distinctions that I found are. >> >> Here is the quick program: >> >> # more alloc_size_attr_test.c >> #include >> #include >> >> __attribute__((alloc_size(1,2))) >> void* my_calloc_alt0(size_t n, size_t s) >> { >> void* p = calloc(n,s); >> printf("calloc __builtin_object_size 0,1,2,3: %ld, %ld, %ld, %ld\n" >> ,(long) __builtin_object_size(p, 0) >> ,(long) __builtin_object_size(p, 1) >> ,(long) __builtin_object_size(p, 2) >> ,(long) __builtin_object_size(p, 3) >> ); >> return p; >> } >> >> __attribute__((alloc_size(1))) __attribute__((alloc_size(2))) >> void* my_calloc_alt1(size_t n, size_t s) >> { >> void* p = calloc(n,s); >> printf("calloc __builtin_object_size 0,1,2,3: %ld, %ld, %ld, %ld\n" >> ,(long) __builtin_object_size(p, 0) >> ,(long) __builtin_object_size(p, 1) >> ,(long) __builtin_object_size(p, 2) >> ,(long) __builtin_object_size(p, 3) >> ); >> return p; >> } >> >> int main() >> { >> void* p = my_calloc_alt0(2,7); >> printf("my_calloc_alt0 __builtin_object_size 0,1,2,3: %ld, %ld, %ld, %ld\n" >> ,(long) __builtin_object_size(p, 0) >> ,(long) __builtin_object_size(p, 1) >> ,(long) __builtin_object_size(p, 2) >> ,(long) __builtin_object_size(p, 3) >> ); >> void* q = my_calloc_alt1(2,7); >> printf("my_calloc_alt0 __builtin_object_size 0,1,2,3: %ld, %ld, %ld, %ld\n" > The above line should have been: > > printf("my_calloc_alt1 __builtin_object_size 0,1,2,3: %ld, %ld, %ld, %ld\n" > >> ,(long) __builtin_object_size(q, 0) >> ,(long) __builtin_object_size(q, 1) >> ,(long) __builtin_object_size(q, 2) >> ,(long) __builtin_object_size(q, 3) >> ); >> } >> >> # uname -apKU >> FreeBSD FBSDFSSD 12.0-CURRENT FreeBSD 12.0-CURRENT r327485M amd64 amd64 1200054 1200054 >> >> The system-clang 5.0.1 result was: >> >> # clang -O2 alloc_size_attr_test.c > The later outputs are edited for clarity: > >> # ./a.out >> calloc __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >> calloc __builtin_object_size 0,1,2,3: 14, 14, 14, 0 > my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >> The lang/gcc7 result was: >> >> # gcc7 -O2 alloc_size_attr_test.c >> >> # ./a.out >> calloc __builtin_object_size 0,1,2,3: -1, -1, 0, 0 >> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 14 >> calloc __builtin_object_size 0,1,2,3: -1, -1, 0, 0 > my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 7, 14, 14 >> I'll ignore that gcc does not provide actual sizes >> via __builtin_object_size for calloc use. >> >> Pairing the other lines for easy comparison, with >> some notes mixed in: >> >> __attribute__((alloc_size(1,2))) style: >> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 (system clang) >> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 14 (gcc7) >> >> __attribute__((alloc_size(1))) __attribute__((alloc_size(2))) style: > my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 (system clang) > my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 7, 14, 14 (gcc7) So on GCC7 it appears  __attribute__((alloc_size(1,2))) != __attribute__((alloc_size(1))) __attribute__((alloc_size(2))) This is not good as it was the base for r280801 .. related to the old discussion about deprecating old compilers that don't accept VA_ARGS. I am unsure if its a regression but it appears that for clang it is the same thing though. >> Thus. . . >> >> For __attribute__((alloc_size(1))) __attribute__((alloc_size(2))): >> __builtin_object_size(p,1) is not equivalent (clang vs. gcc7) >> >> For both of the alloc_size usage styles: >> __builtin_object_size(p,3) is not equivalent (clang vs. gcc7) >> >> This means that the two style of alloc_size use are not >> equivalent across some major compilers/toolchains. This is actually not a surprise: GCC and clang implementation of __alloc_size__ has differences due to limitations on the LLVM IR (or the fact there is one). The alloc_size attribute is basically only used for the so-called FORTIFY_SOURCE feature that depends on GCC with some support from the C-library: last time I looked clang didn't support the compile-time checks very well. The attributes are mostly unused in FreeBSD at this time but, GCC7 -Walloc-size-larger-than=size depends on them (I have never tested that though). FWIW, we had an unfinished GSoC that attempted to implement FORTIFY_SOURCE but we got stuck on the lack of clang support and other issues. Lately Google has been spending some effort on it but it is more limited and doesn't match the GCC behavior. >> But I do not know if either of the differences is a problem or >> not. >> >> >> Note: without a sufficient -O all the figures can be >> the mix of -1's and 0's. > > === > Mark Millard > marklmi at yahoo.com > ( markmi at dsl-only.net is > going away in 2018-Feb, late) > From owner-freebsd-hackers@freebsd.org Sun Jan 21 16:56:59 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 87A2DED1A3C for ; Sun, 21 Jan 2018 16:56:59 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) Received: from sonic313-10.consmr.mail.ne1.yahoo.com (sonic313-10.consmr.mail.ne1.yahoo.com [66.163.185.33]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5A5C372B05 for ; Sun, 21 Jan 2018 16:56:59 +0000 (UTC) (envelope-from marklmi26-fbsd@yahoo.com) X-YMail-OSG: 3DHC.90VM1lCiYWc0ThMOnR6USw8HlQrQs0rXVcK6C33EmknCr2pFqD.kSK7coH aYA2pvncyIm61JfzzEREpHptJ_OxevNhrhNUoTcodW32kXOdMPyw9xsZkpdFQcAXq4_5Y1x_xkdA y4n_ciPzbgNkKe0YA3yDv_1gMkozQwnkFrv7vMwTQzQNNRJP.cRDgSkaebUeLbgMXzh9JE6Q0hNz GWQ2o9lnN_Ltek2Pei92qrO9bd0bMXjNGyGuvT6U1D5Khp1v70gcBHG5oa1g1F7sGysXJ.FYto9Y Dl3Kv6swYTAjAd7WuPHjELXm8MNCmm1oaNwJcr3Y3kK8RNgGjNkcGDE7aGDGX99Ezky96f6ChIUZ VB._K0wS891rcF7Ns7XKulaypMuEvBEwgbFG6BROf07732RN6rM5LBn95Xqcd9mhrOPRWOfoC3_Q oagI5hXj7nyXqsTnGxEw2HMhH2CNgEDXY2mgpVf2IaiZY5Udy9L23_CTO72VvwAp26vME Received: from sonic.gate.mail.ne1.yahoo.com by sonic313.consmr.mail.ne1.yahoo.com with HTTP; Sun, 21 Jan 2018 16:56:52 +0000 Received: from smtp229.mail.ne1.yahoo.com (EHLO [192.168.1.25]) ([10.218.253.212]) by smtp415.mail.ne1.yahoo.com (JAMES SMTP Server ) with ESMTPA ID fe075fbd47d9537ac5d0ccfac4678709; Sun, 21 Jan 2018 16:56:49 +0000 (UTC) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: Attribute alloc__size use and clang 5.0.1 vs. gcc7 (e.g.): __builtin_object_size(p,1) and __builtin_object_size(p,3) disagreements result From: Mark Millard In-Reply-To: <8c4c5985-885a-abf7-93df-0698c645d36e@FreeBSD.org> Date: Sun, 21 Jan 2018 08:56:48 -0800 Cc: FreeBSD Toolchain , FreeBSD Hackers Content-Transfer-Encoding: quoted-printable Message-Id: <9DE674C6-EAA3-4E8A-906F-446E74D82FC4@yahoo.com> References: <1AA0993D-81E4-4DC0-BBD9-CC42B26ADB1C@yahoo.com> <8c4c5985-885a-abf7-93df-0698c645d36e@FreeBSD.org> To: Pedro Giffuni X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jan 2018 16:56:59 -0000 [May be an __alloc_size2(n,s) should be added and used?] On 2018-Jan-20, at 5:05 PM, Pedro Giffuni wrote: > Very interesting , thanks for running such tests ... >=20 >=20 > On 01/20/18 18:59, Mark Millard wrote: >> [Noting a typo in the program source, and >> so in the output text: the 2nd occurance of: "my_calloc_alt0 >> should have been: "my_calloc_alt1 >> . Hand edited corrections below for clarity.] >>=20 >> On 2018-Jan-20, at 3:27 PM, Mark Millard = wrote: >>=20 >>> [Bugzilla 225197 indirectly lead to this. >>> Avoiding continuing there.] >>>=20 >>> I decided to compare some alternate uses of >>> __attribute__((alloc_size(. . .))) compiled >>> and run under clang 5.0.1 and gcc7. I did not >>> get what I expected based on prior discussion >>> material. >>>=20 >>> This is an FYI since I do not know how important >>> the distinctions that I found are. >>>=20 >>> Here is the quick program: >>>=20 >>> # more alloc_size_attr_test.c >>> #include >>> #include >>>=20 >>> __attribute__((alloc_size(1,2))) >>> void* my_calloc_alt0(size_t n, size_t s) >>> { >>> void* p =3D calloc(n,s); >>> printf("calloc __builtin_object_size 0,1,2,3: %ld, %ld, %ld, = %ld\n" >>> ,(long) __builtin_object_size(p, 0) >>> ,(long) __builtin_object_size(p, 1) >>> ,(long) __builtin_object_size(p, 2) >>> ,(long) __builtin_object_size(p, 3) >>> ); >>> return p; >>> } >>>=20 >>> __attribute__((alloc_size(1))) __attribute__((alloc_size(2))) >>> void* my_calloc_alt1(size_t n, size_t s) >>> { >>> void* p =3D calloc(n,s); >>> printf("calloc __builtin_object_size 0,1,2,3: %ld, %ld, %ld, = %ld\n" >>> ,(long) __builtin_object_size(p, 0) >>> ,(long) __builtin_object_size(p, 1) >>> ,(long) __builtin_object_size(p, 2) >>> ,(long) __builtin_object_size(p, 3) >>> ); >>> return p; >>> } >>>=20 >>> int main() >>> { >>> void* p =3D my_calloc_alt0(2,7); >>> printf("my_calloc_alt0 __builtin_object_size 0,1,2,3: %ld, %ld, = %ld, %ld\n" >>> ,(long) __builtin_object_size(p, 0) >>> ,(long) __builtin_object_size(p, 1) >>> ,(long) __builtin_object_size(p, 2) >>> ,(long) __builtin_object_size(p, 3) >>> ); >>> void* q =3D my_calloc_alt1(2,7); >>> printf("my_calloc_alt0 __builtin_object_size 0,1,2,3: %ld, %ld, = %ld, %ld\n" >> The above line should have been: >>=20 >> printf("my_calloc_alt1 __builtin_object_size 0,1,2,3: %ld, %ld, %ld, = %ld\n" >>=20 >>> ,(long) __builtin_object_size(q, 0) >>> ,(long) __builtin_object_size(q, 1) >>> ,(long) __builtin_object_size(q, 2) >>> ,(long) __builtin_object_size(q, 3) >>> ); >>> } >>>=20 >>> # uname -apKU >>> FreeBSD FBSDFSSD 12.0-CURRENT FreeBSD 12.0-CURRENT r327485M amd64 = amd64 1200054 1200054 >>>=20 >>> The system-clang 5.0.1 result was: >>>=20 >>> # clang -O2 alloc_size_attr_test.c >> The later outputs are edited for clarity: >>=20 >>> # ./a.out >>> calloc __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >>> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >>> calloc __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >> my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >>> The lang/gcc7 result was: >>>=20 >>> # gcc7 -O2 alloc_size_attr_test.c >>>=20 >>> # ./a.out >>> calloc __builtin_object_size 0,1,2,3: -1, -1, 0, 0 >>> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 14 >>> calloc __builtin_object_size 0,1,2,3: -1, -1, 0, 0 >> my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 7, 14, 14 >>> I'll ignore that gcc does not provide actual sizes >>> via __builtin_object_size for calloc use. >>>=20 >>> Pairing the other lines for easy comparison, with >>> some notes mixed in: >>>=20 >>> __attribute__((alloc_size(1,2))) style: >>> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 (system = clang) >>> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 14 (gcc7) >>>=20 >>> __attribute__((alloc_size(1))) __attribute__((alloc_size(2))) style: >> my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 (system = clang) >> my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 7, 14, 14 (gcc7) > So on GCC7 it appears > __attribute__((alloc_size(1,2))) !=3D __attribute__((alloc_size(1))) = __attribute__((alloc_size(2))) >=20 > This is not good as it was the base for r280801 .. related to the old = discussion about deprecating old compilers that don't accept VA_ARGS. >=20 > I am unsure if its a regression but it appears that for clang it is = the same thing though. May be there should be a __alloc_size2(n,s) that translates to __attribute__((alloc_size(n,s))) when it is not a no-op. This avoids the VA_ARGS issue and gcc's documentation makes no mention of a more than 2 argument variant of __attribute__((alloc_size(. . .))) . Looking up glib it has: G_GNUC_ALLOC_SIZE(x) G_GNUC_ALLOC_SIZE2(x,y) but none for any other count of arguments. >>> Thus. . . >>>=20 >>> For __attribute__((alloc_size(1))) __attribute__((alloc_size(2))): >>> __builtin_object_size(p,1) is not equivalent (clang vs. gcc7) >>>=20 >>> For both of the alloc_size usage styles: >>> __builtin_object_size(p,3) is not equivalent (clang vs. gcc7) >>>=20 >>> This means that the two style of alloc_size use are not >>> equivalent across some major compilers/toolchains. >=20 > This is actually not a surprise: GCC and clang implementation of = __alloc_size__ has differences due to limitations on the LLVM IR (or the = fact there is one). >=20 > The alloc_size attribute is basically only used for the so-called = FORTIFY_SOURCE feature that depends on GCC with some support from the = C-library: last time I looked clang didn't support the compile-time = checks very well. The attributes are mostly unused in FreeBSD at this = time but, GCC7 -Walloc-size-larger-than=3Dsize depends on them (I have = never tested that though). >=20 > FWIW, we had an unfinished GSoC that attempted to implement = FORTIFY_SOURCE but we got stuck on the lack of clang support and other = issues. Lately Google has been spending some effort on it but it is more = limited and doesn't match the GCC behavior. >=20 >=20 >=20 >=20 >>> But I do not know if either of the differences is a problem or >>> not. >>>=20 >>>=20 >>> Note: without a sufficient -O all the figures can be >>> the mix of -1's and 0's. =3D=3D=3D Mark Millard marklmi at yahoo.com ( markmi at dsl-only.net is going away in 2018-Feb, late) From owner-freebsd-hackers@freebsd.org Sun Jan 21 18:09:23 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 570B3ED4992 for ; Sun, 21 Jan 2018 18:09:23 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from sonic316-22.consmr.mail.gq1.yahoo.com (sonic316-22.consmr.mail.gq1.yahoo.com [98.137.69.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2CE66754FF for ; Sun, 21 Jan 2018 18:09:22 +0000 (UTC) (envelope-from pfg@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1516558161; bh=zFkLeS3OEN5xNiCBn7yXSRzXotqC1SG7ntXVA31wjeQ=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From:Subject; b=IPBNCYyotAqiuzI7T5FzxsuiKgE+Ox85Wumo3DpLUXIIRlDp7AmssplYNnXCfqIdil/ZOcUuUhXfR/J7CYevyJMHsJjm8qyqb4PxPc+ayeJQ8t0mGm3VPEy2VDrFT0grgKO6A7MyqsvSKI0LYpCAs+NplKyQ3r9yhEiinvXbhHQr59jqeLdl5ET07/hV/RVG4+Pr4s2KIRb6JpVdxhPuSPjkx2DkrNuww4AYPZZ5Fa33fJ+x+GNDcWj58ui4mqcdD5md5+a+vMIDE/LmCiC1LglzIuKMAZk+UHR6c5VHXA+MrQLeDNVM1zHRS2nQ4/LDasxkl9qlkM2nz3aOCoKHiA== X-YMail-OSG: x6Fn9iEVM1k2bBj3CR_045KbWh9snN.pLpzfnh3S483vZGxITQIQvHDxjVvepfu WS4Fv97DjDENku0xvpKgpnaHwAgiNTjySN5vdBDZVrHjcK3dPHPayGYGFQJUYFoCPD1ajhnQlaXC jBtg1nILFH3pfjXFCzaWOD2dISeckcHyZWTVEGZZN9RWhPkhnq8QL32.nvXyTIze.X.QGAv_hAXA 3d3Cnhu2r2eKF7Wzv2d1o5l2j4K2OiNmhYiMswNrQ_UtgTxWsfRhMu4W5fzZRdJPrB8qw4Ikwdmw lp3CNvsTrAP3FpTXDgHf5WjkCiP_aqXAdN29ePgEGuYen2qsA51sM5DqFx.s5Mudr20xwJumnyxl DbVuGGqYe8vOxGicPGE21p38dJL7fK0zK5krwznPPKxaMKJ.9RfXNTIp3tMIUs7YUZilYHaLX9Nl G2Eyqxnp8eMVLjI80CRqjwNPA3h.LYP.3aGBTOyBSm2jPUUzfD6_WjGNpsg.9GIMgXyX.xoJwyDa .fUojjZWyyw-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Sun, 21 Jan 2018 18:09:21 +0000 Received: from smtp103.rhel.mail.gq1.yahoo.com (EHLO [192.168.0.8]) ([68.180.227.11]) by smtp403.mail.gq1.yahoo.com (JAMES SMTP Server ) with ESMTPA ID b273a8aa0a7b2e548ec7d53afcbd0f7f; Sun, 21 Jan 2018 17:59:11 +0000 (UTC) Subject: Re: Attribute alloc__size use and clang 5.0.1 vs. gcc7 (e.g.): __builtin_object_size(p,1) and __builtin_object_size(p,3) disagreements result To: Mark Millard Cc: FreeBSD Toolchain , FreeBSD Hackers References: <1AA0993D-81E4-4DC0-BBD9-CC42B26ADB1C@yahoo.com> <8c4c5985-885a-abf7-93df-0698c645d36e@FreeBSD.org> <9DE674C6-EAA3-4E8A-906F-446E74D82FC4@yahoo.com> From: Pedro Giffuni Message-ID: <67a7b385-b554-1a5e-ef30-fb29a0c6cf08@FreeBSD.org> Date: Sun, 21 Jan 2018 12:59:12 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: <9DE674C6-EAA3-4E8A-906F-446E74D82FC4@yahoo.com> Content-Type: multipart/mixed; boundary="------------EFD43819BA4DE658890B95F7" Content-Language: en-US X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jan 2018 18:09:23 -0000 This is a multi-part message in MIME format. --------------EFD43819BA4DE658890B95F7 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hi; On 01/21/18 11:56, Mark Millard wrote: > [May be an __alloc_size2(n,s) should be added and used?] > > On 2018-Jan-20, at 5:05 PM, Pedro Giffuni wrote: > >> Very interesting , thanks for running such tests ... >> >> >> On 01/20/18 18:59, Mark Millard wrote: >>> [Noting a typo in the program source, and >>> so in the output text: the 2nd occurance of: "my_calloc_alt0 >>> should have been: "my_calloc_alt1 >>> . Hand edited corrections below for clarity.] >>> >>> On 2018-Jan-20, at 3:27 PM, Mark Millard wrote: >>> >>>> [Bugzilla 225197 indirectly lead to this. >>>> Avoiding continuing there.] >>>> >>>> I decided to compare some alternate uses of >>>> __attribute__((alloc_size(. . .))) compiled >>>> and run under clang 5.0.1 and gcc7. I did not >>>> get what I expected based on prior discussion >>>> material. >>>> >>>> This is an FYI since I do not know how important >>>> the distinctions that I found are. >>>> >>>> Here is the quick program: >>>> >>>> # more alloc_size_attr_test.c >>>> #include >>>> #include >>>> >>>> __attribute__((alloc_size(1,2))) >>>> void* my_calloc_alt0(size_t n, size_t s) >>>> { >>>> void* p = calloc(n,s); >>>> printf("calloc __builtin_object_size 0,1,2,3: %ld, %ld, %ld, %ld\n" >>>> ,(long) __builtin_object_size(p, 0) >>>> ,(long) __builtin_object_size(p, 1) >>>> ,(long) __builtin_object_size(p, 2) >>>> ,(long) __builtin_object_size(p, 3) >>>> ); >>>> return p; >>>> } >>>> >>>> __attribute__((alloc_size(1))) __attribute__((alloc_size(2))) >>>> void* my_calloc_alt1(size_t n, size_t s) >>>> { >>>> void* p = calloc(n,s); >>>> printf("calloc __builtin_object_size 0,1,2,3: %ld, %ld, %ld, %ld\n" >>>> ,(long) __builtin_object_size(p, 0) >>>> ,(long) __builtin_object_size(p, 1) >>>> ,(long) __builtin_object_size(p, 2) >>>> ,(long) __builtin_object_size(p, 3) >>>> ); >>>> return p; >>>> } >>>> >>>> int main() >>>> { >>>> void* p = my_calloc_alt0(2,7); >>>> printf("my_calloc_alt0 __builtin_object_size 0,1,2,3: %ld, %ld, %ld, %ld\n" >>>> ,(long) __builtin_object_size(p, 0) >>>> ,(long) __builtin_object_size(p, 1) >>>> ,(long) __builtin_object_size(p, 2) >>>> ,(long) __builtin_object_size(p, 3) >>>> ); >>>> void* q = my_calloc_alt1(2,7); >>>> printf("my_calloc_alt0 __builtin_object_size 0,1,2,3: %ld, %ld, %ld, %ld\n" >>> The above line should have been: >>> >>> printf("my_calloc_alt1 __builtin_object_size 0,1,2,3: %ld, %ld, %ld, %ld\n" >>> >>>> ,(long) __builtin_object_size(q, 0) >>>> ,(long) __builtin_object_size(q, 1) >>>> ,(long) __builtin_object_size(q, 2) >>>> ,(long) __builtin_object_size(q, 3) >>>> ); >>>> } >>>> >>>> # uname -apKU >>>> FreeBSD FBSDFSSD 12.0-CURRENT FreeBSD 12.0-CURRENT r327485M amd64 amd64 1200054 1200054 >>>> >>>> The system-clang 5.0.1 result was: >>>> >>>> # clang -O2 alloc_size_attr_test.c >>> The later outputs are edited for clarity: >>> >>>> # ./a.out >>>> calloc __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >>>> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >>>> calloc __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >>> my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >>>> The lang/gcc7 result was: >>>> >>>> # gcc7 -O2 alloc_size_attr_test.c >>>> >>>> # ./a.out >>>> calloc __builtin_object_size 0,1,2,3: -1, -1, 0, 0 >>>> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 14 >>>> calloc __builtin_object_size 0,1,2,3: -1, -1, 0, 0 >>> my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 7, 14, 14 >>>> I'll ignore that gcc does not provide actual sizes >>>> via __builtin_object_size for calloc use. >>>> >>>> Pairing the other lines for easy comparison, with >>>> some notes mixed in: >>>> >>>> __attribute__((alloc_size(1,2))) style: >>>> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 (system clang) >>>> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 14 (gcc7) >>>> >>>> __attribute__((alloc_size(1))) __attribute__((alloc_size(2))) style: >>> my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 (system clang) >>> my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 7, 14, 14 (gcc7) >> So on GCC7 it appears >> __attribute__((alloc_size(1,2))) != __attribute__((alloc_size(1))) __attribute__((alloc_size(2))) >> >> This is not good as it was the base for r280801 .. related to the old discussion about deprecating old compilers that don't accept VA_ARGS. >> >> I am unsure if its a regression but it appears that for clang it is the same thing though. > May be there should be a __alloc_size2(n,s) that translates to > __attribute__((alloc_size(n,s))) when it is not a no-op. This > avoids the VA_ARGS issue and gcc's documentation makes no mention > of a more than 2 argument variant of > __attribute__((alloc_size(. . .))) . > > Looking up glib it has: > > G_GNUC_ALLOC_SIZE(x) > G_GNUC_ALLOC_SIZE2(x,y) > > but none for any other count of arguments. Yes, that would work fine for alloc_array. It would work for the nonnull attribute but that is deprecated on FreeBSD :). Concept patch attached. Pedro. >>>> Thus. . . >>>> >>>> For __attribute__((alloc_size(1))) __attribute__((alloc_size(2))): >>>> __builtin_object_size(p,1) is not equivalent (clang vs. gcc7) >>>> >>>> For both of the alloc_size usage styles: >>>> __builtin_object_size(p,3) is not equivalent (clang vs. gcc7) >>>> >>>> This means that the two style of alloc_size use are not >>>> equivalent across some major compilers/toolchains. >> This is actually not a surprise: GCC and clang implementation of __alloc_size__ has differences due to limitations on the LLVM IR (or the fact there is one). >> >> The alloc_size attribute is basically only used for the so-called FORTIFY_SOURCE feature that depends on GCC with some support from the C-library: last time I looked clang didn't support the compile-time checks very well. The attributes are mostly unused in FreeBSD at this time but, GCC7 -Walloc-size-larger-than=size depends on them (I have never tested that though). >> >> FWIW, we had an unfinished GSoC that attempted to implement FORTIFY_SOURCE but we got stuck on the lack of clang support and other issues. Lately Google has been spending some effort on it but it is more limited and doesn't match the GCC behavior. >> >> >> >> >>>> But I do not know if either of the differences is a problem or >>>> not. >>>> >>>> >>>> Note: without a sufficient -O all the figures can be >>>> the mix of -1's and 0's. > === > Mark Millard > marklmi at yahoo.com > ( markmi at dsl-only.net is > going away in 2018-Feb, late) > --------------EFD43819BA4DE658890B95F7 Content-Type: text/x-patch; name="alloc_size2.diff" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="alloc_size2.diff" Index: include/stdlib.h =================================================================== --- include/stdlib.h (revision 328218) +++ include/stdlib.h (working copy) @@ -92,7 +92,7 @@ void *bsearch(const void *, const void *, size_t, size_t, int (*)(const void * _Nonnull, const void *)); void *calloc(size_t, size_t) __malloc_like __result_use_check - __alloc_size(1) __alloc_size(2); + __alloc_size2(1, 2); div_t div(int, int) __pure2; _Noreturn void exit(int); void free(void *); @@ -302,8 +302,8 @@ int (*)(void *, const void *, const void *)); int radixsort(const unsigned char **, int, const unsigned char *, unsigned); -void *reallocarray(void *, size_t, size_t) __result_use_check __alloc_size(2) - __alloc_size(3); +void *reallocarray(void *, size_t, size_t) __result_use_check + __alloc_size2(2, 3); void *reallocf(void *, size_t) __result_use_check __alloc_size(2); int rpmatch(const char *); void setprogname(const char *); Index: sys/sys/cdefs.h =================================================================== --- sys/sys/cdefs.h (revision 328218) +++ sys/sys/cdefs.h (working copy) @@ -230,8 +230,10 @@ #endif #if __GNUC_PREREQ__(4, 3) || __has_attribute(__alloc_size__) #define __alloc_size(x) __attribute__((__alloc_size__(x))) +#define __alloc_size(n, x) __attribute__((__alloc_size__(n, x))) #else #define __alloc_size(x) +#define __alloc_size2(n, x) #endif #if __GNUC_PREREQ__(4, 9) || __has_attribute(__alloc_align__) #define __alloc_align(x) __attribute__((__alloc_align__(x))) Index: sys/sys/malloc.h =================================================================== --- sys/sys/malloc.h (revision 328218) +++ sys/sys/malloc.h (working copy) @@ -188,7 +188,7 @@ __malloc_like __result_use_check __alloc_size(1); void *mallocarray(size_t nmemb, size_t size, struct malloc_type *type, int flags) __malloc_like __result_use_check - __alloc_size(1) __alloc_size(2); + __alloc_size2(1, 2); void malloc_init(void *); int malloc_last_fail(void); void malloc_type_allocated(struct malloc_type *type, unsigned long size); --------------EFD43819BA4DE658890B95F7-- From owner-freebsd-hackers@freebsd.org Sun Jan 21 18:55:59 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 48DF3EB44CE for ; Sun, 21 Jan 2018 18:55:59 +0000 (UTC) (envelope-from pfg@FreeBSD.org) Received: from sonic316-22.consmr.mail.gq1.yahoo.com (sonic316-22.consmr.mail.gq1.yahoo.com [98.137.69.46]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1F5EB773AA for ; Sun, 21 Jan 2018 18:55:58 +0000 (UTC) (envelope-from pfg@FreeBSD.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1516560957; bh=Oeb2XetJ+ct/IKmS4UKj4/rbGZKTJcjTmBbduFAdhd4=; h=Subject:From:To:Cc:References:Date:In-Reply-To:From:Subject; b=X4rV2GZuvdbafkp1woDqmvFOcs7Rhs3IgkSnEv9BXZr2GpseT/vUbg0fskw28StXxgirU2ldr9YBKuwBDRWgVrfyfiqbuqj1HKXulNz5vYjCIKFdSsLnXcyrdkg6BpKnedEwtaJUHojMnMlaZWj7PQzeNpICW1AJ8dXlSyOUsn/JIAJpT5ayX6bGKmvt7Zraabzi4EG+IeG60MjCMvhOyS7vgGHz7C54TFRBvBqrXpVwhVby9tZhvc+NNI4dmWvjjPS1C7wd0/uO3cROM9pv4QjiG0od0iQbGUYeBn4lngcHIoTo7uxm+iEPfUbV4gph31LvmUvpgU73MM9kvayXLw== X-YMail-OSG: ZzOTs3UVM1neXQb.Mfn_IpOr6MxQ2RDWt8rFG8EUhn5Q_hHUyZU66aGWihNNrfa 7eTPK36cVpEm0n3YAzaiWHdfyQB6boBPY.LZpsTNlMu75cprXmuWEudmFyYFkBMruHjDunXwU0aY BgXucOBmDzOvJqKfKJHahu8aMcBcwdtIIW3Ax2Lhggl14nw46jsxPgZyVeD6RJ.SOWxDenHwIHWG eoXVFNHr91hqs6xjyrpWXfdavFnXLNarsDCT_rFnwG_M7.5Sb1KFdvJ61WcBomyVHJypYwuK9.Jm tMkf4yq8ZQzF5ZUemFXI5SRiw8Y5MQX0EFZPuz8mzaAa1BLXfIFKbza5WTmoJLzuc7A1CR41.c7G jXe.dZv1x1z9_O.92fiWBdQGHT2B4ploBK5cSoAQIpl4ErMij9T56cKfbOgjkVYciTvItOy2vQaq r8zCuGvNVhQcYsCdxKlmuRkmwIO2T.CmUQA8RRkmFUT3tXzhmE2TSDja4.qQOqnt_5CyqqNmg.u6 OzKo6X1pg2Q-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic316.consmr.mail.gq1.yahoo.com with HTTP; Sun, 21 Jan 2018 18:55:57 +0000 Received: from smtp105.rhel.mail.gq1.yahoo.com (EHLO [192.168.0.8]) ([68.180.227.8]) by smtp412.mail.gq1.yahoo.com (JAMES SMTP Server ) with ESMTPA ID 521ad2623a0d6f915e996ea87fc2db5c; Sun, 21 Jan 2018 18:55:54 +0000 (UTC) Subject: Re: Attribute alloc__size use and clang 5.0.1 vs. gcc7 (e.g.): __builtin_object_size(p,1) and __builtin_object_size(p,3) disagreements result From: Pedro Giffuni To: Mark Millard Cc: FreeBSD Toolchain , FreeBSD Hackers References: <1AA0993D-81E4-4DC0-BBD9-CC42B26ADB1C@yahoo.com> <8c4c5985-885a-abf7-93df-0698c645d36e@FreeBSD.org> <9DE674C6-EAA3-4E8A-906F-446E74D82FC4@yahoo.com> <67a7b385-b554-1a5e-ef30-fb29a0c6cf08@FreeBSD.org> Message-ID: <9f07664c-6b39-8201-94ed-62c6b91cffbf@FreeBSD.org> Date: Sun, 21 Jan 2018 13:55:55 -0500 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: <67a7b385-b554-1a5e-ef30-fb29a0c6cf08@FreeBSD.org> Content-Type: multipart/mixed; boundary="------------520FA10BA9E7194FE65018A2" Content-Language: en-US X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jan 2018 18:55:59 -0000 This is a multi-part message in MIME format. --------------520FA10BA9E7194FE65018A2 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit On 01/21/18 12:59, Pedro Giffuni wrote: > Hi; > > > On 01/21/18 11:56, Mark Millard wrote: >> [May be an __alloc_size2(n,s) should be added and used?] >> >> On 2018-Jan-20, at 5:05 PM, Pedro Giffuni wrote: >> >>> Very interesting , thanks for running such tests ... >>> >>> >>> On 01/20/18 18:59, Mark Millard wrote: >>>> [Noting a typo in the program source, and >>>> so in the output text: the 2nd occurance of: "my_calloc_alt0 >>>> should have been: "my_calloc_alt1 >>>> . Hand edited corrections below for clarity.] >>>> >>>> On 2018-Jan-20, at 3:27 PM, Mark Millard >>>> wrote: >>>> >>>>> [Bugzilla 225197 indirectly lead to this. >>>>> Avoiding continuing there.] >>>>> >>>>> I decided to compare some alternate uses of >>>>> __attribute__((alloc_size(. . .))) compiled >>>>> and run under clang 5.0.1 and gcc7. I did not >>>>> get what I expected based on prior discussion >>>>> material. >>>>> >>>>> This is an FYI since I do not know how important >>>>> the distinctions that I found are. >>>>> >>>>> Here is the quick program: >>>>> >>>>> # more alloc_size_attr_test.c >>>>> #include >>>>> #include >>>>> >>>>> __attribute__((alloc_size(1,2))) >>>>> void* my_calloc_alt0(size_t n, size_t s) >>>>> { >>>>>    void* p = calloc(n,s); >>>>>    printf("calloc __builtin_object_size 0,1,2,3: %ld, %ld, %ld, >>>>> %ld\n" >>>>>          ,(long) __builtin_object_size(p, 0) >>>>>          ,(long) __builtin_object_size(p, 1) >>>>>          ,(long) __builtin_object_size(p, 2) >>>>>          ,(long) __builtin_object_size(p, 3) >>>>>          ); >>>>>    return p; >>>>> } >>>>> >>>>> __attribute__((alloc_size(1))) __attribute__((alloc_size(2))) >>>>> void* my_calloc_alt1(size_t n, size_t s) >>>>> { >>>>>    void* p = calloc(n,s); >>>>>    printf("calloc __builtin_object_size 0,1,2,3: %ld, %ld, %ld, >>>>> %ld\n" >>>>>          ,(long) __builtin_object_size(p, 0) >>>>>          ,(long) __builtin_object_size(p, 1) >>>>>          ,(long) __builtin_object_size(p, 2) >>>>>          ,(long) __builtin_object_size(p, 3) >>>>>          ); >>>>>    return p; >>>>> } >>>>> >>>>> int main() >>>>> { >>>>>    void* p = my_calloc_alt0(2,7); >>>>>    printf("my_calloc_alt0 __builtin_object_size 0,1,2,3: %ld, %ld, >>>>> %ld, %ld\n" >>>>>          ,(long) __builtin_object_size(p, 0) >>>>>          ,(long) __builtin_object_size(p, 1) >>>>>          ,(long) __builtin_object_size(p, 2) >>>>>          ,(long) __builtin_object_size(p, 3) >>>>>          ); >>>>>    void* q = my_calloc_alt1(2,7); >>>>>    printf("my_calloc_alt0 __builtin_object_size 0,1,2,3: %ld, %ld, >>>>> %ld, %ld\n" >>>> The above line should have been: >>>> >>>> printf("my_calloc_alt1 __builtin_object_size 0,1,2,3: %ld, %ld, >>>> %ld, %ld\n" >>>> >>>>>          ,(long) __builtin_object_size(q, 0) >>>>>          ,(long) __builtin_object_size(q, 1) >>>>>          ,(long) __builtin_object_size(q, 2) >>>>>          ,(long) __builtin_object_size(q, 3) >>>>>          ); >>>>> } >>>>> >>>>> # uname -apKU >>>>> FreeBSD FBSDFSSD 12.0-CURRENT FreeBSD 12.0-CURRENT r327485M  amd64 >>>>> amd64 1200054 1200054 >>>>> >>>>> The system-clang 5.0.1 result was: >>>>> >>>>> # clang -O2 alloc_size_attr_test.c >>>> The later outputs are edited for clarity: >>>> >>>>> # ./a.out >>>>> calloc __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >>>>> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >>>>> calloc __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >>>> my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 >>>>> The lang/gcc7 result was: >>>>> >>>>> # gcc7 -O2 alloc_size_attr_test.c >>>>> >>>>> # ./a.out >>>>> calloc __builtin_object_size 0,1,2,3: -1, -1, 0, 0 >>>>> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 14 >>>>> calloc __builtin_object_size 0,1,2,3: -1, -1, 0, 0 >>>> my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 7, 14, 14 >>>>> I'll ignore that gcc does not provide actual sizes >>>>> via __builtin_object_size for calloc use. >>>>> >>>>> Pairing the other lines for easy comparison, with >>>>> some notes mixed in: >>>>> >>>>> __attribute__((alloc_size(1,2))) style: >>>>> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 0  >>>>> (system clang) >>>>> my_calloc_alt0 __builtin_object_size 0,1,2,3: 14, 14, 14, 14 (gcc7) >>>>> >>>>> __attribute__((alloc_size(1))) __attribute__((alloc_size(2))) style: >>>> my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 14, 14, 0 (system >>>> clang) >>>> my_calloc_alt1 __builtin_object_size 0,1,2,3: 14, 7, 14, 14 (gcc7) >>> So on GCC7 it appears >>>   __attribute__((alloc_size(1,2))) != __attribute__((alloc_size(1))) >>> __attribute__((alloc_size(2))) >>> >>> This is not good as it was the base for r280801 .. related to the >>> old discussion about deprecating old compilers that don't accept >>> VA_ARGS. >>> >>> I am unsure if its a regression but it appears that for clang it is >>> the same thing though. >> May be there should be a __alloc_size2(n,s) that translates to >> __attribute__((alloc_size(n,s))) when it is not a no-op. This >> avoids the VA_ARGS issue and gcc's documentation makes no mention >> of a more than 2 argument variant of >> __attribute__((alloc_size(. . .))) . >> >> Looking up glib it has: >> >> G_GNUC_ALLOC_SIZE(x) >> G_GNUC_ALLOC_SIZE2(x,y) >> >> but none for any other count of arguments. > > > Yes, that would work fine for alloc_array. It would work for the > nonnull attribute but that is deprecated on FreeBSD :). > > > Concept patch attached. And of course there was a bug. Pedro. --------------520FA10BA9E7194FE65018A2 Content-Type: text/x-patch; name="alloc_size2.diff" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="alloc_size2.diff" Index: include/stdlib.h =================================================================== --- include/stdlib.h (revision 328218) +++ include/stdlib.h (working copy) @@ -92,7 +92,7 @@ void *bsearch(const void *, const void *, size_t, size_t, int (*)(const void * _Nonnull, const void *)); void *calloc(size_t, size_t) __malloc_like __result_use_check - __alloc_size(1) __alloc_size(2); + __alloc_size2(1, 2); div_t div(int, int) __pure2; _Noreturn void exit(int); void free(void *); @@ -302,8 +302,8 @@ int (*)(void *, const void *, const void *)); int radixsort(const unsigned char **, int, const unsigned char *, unsigned); -void *reallocarray(void *, size_t, size_t) __result_use_check __alloc_size(2) - __alloc_size(3); +void *reallocarray(void *, size_t, size_t) __result_use_check + __alloc_size2(2, 3); void *reallocf(void *, size_t) __result_use_check __alloc_size(2); int rpmatch(const char *); void setprogname(const char *); Index: sys/sys/cdefs.h =================================================================== --- sys/sys/cdefs.h (revision 328218) +++ sys/sys/cdefs.h (working copy) @@ -230,8 +230,10 @@ #endif #if __GNUC_PREREQ__(4, 3) || __has_attribute(__alloc_size__) #define __alloc_size(x) __attribute__((__alloc_size__(x))) +#define __alloc_size2(n, x) __attribute__((__alloc_size__(n, x))) #else #define __alloc_size(x) +#define __alloc_size2(n, x) #endif #if __GNUC_PREREQ__(4, 9) || __has_attribute(__alloc_align__) #define __alloc_align(x) __attribute__((__alloc_align__(x))) Index: sys/sys/malloc.h =================================================================== --- sys/sys/malloc.h (revision 328218) +++ sys/sys/malloc.h (working copy) @@ -188,7 +188,7 @@ __malloc_like __result_use_check __alloc_size(1); void *mallocarray(size_t nmemb, size_t size, struct malloc_type *type, int flags) __malloc_like __result_use_check - __alloc_size(1) __alloc_size(2); + __alloc_size2(1, 2); void malloc_init(void *); int malloc_last_fail(void); void malloc_type_allocated(struct malloc_type *type, unsigned long size); --------------520FA10BA9E7194FE65018A2-- From owner-freebsd-hackers@freebsd.org Sun Jan 21 21:15:02 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 66E37EBBAD2 for ; Sun, 21 Jan 2018 21:15:02 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: from mail-wm0-x22e.google.com (mail-wm0-x22e.google.com [IPv6:2a00:1450:400c:c09::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EFDDC7CAA1 for ; Sun, 21 Jan 2018 21:15:01 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: by mail-wm0-x22e.google.com with SMTP id f3so13214762wmc.1 for ; Sun, 21 Jan 2018 13:15:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:from:date:message-id:subject:to; bh=1vCg4tN+wLXUuzQPpJWSK6XX2QnlG6QCvRB4nPVgybw=; b=DxDkFCgUyJM8E1C74SEER5UBP+sb8xTk7J0AnYhGG1PeAm5HL+2zKyLv+Ax3CUlj3u yY6sJ/5Nilco32tXBt4uZDrKVeIRUA7JuRTsC+KxkRrsS9/gBaIlkdnHvty954xkux4T gWoRCJW3EWUIil32Cxrbbfpg37j1Y07whB5dARfk+Ey58sJ2kcLOrZSEsun/j+dmFjBG JbMM5aUMQN1nWNgfUtDcWYxTsJ6wAV/rci8yCm6ZFosBiYfZ9ufrpmkZLg6f0dSZSWwM CT7pVDvDECPQym0EGMcercEK5xCLkEKTH2Hda71hT9kdUgYB1ceMNjNtF/JXte1mROg+ fXBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:from:date:message-id:subject :to; bh=1vCg4tN+wLXUuzQPpJWSK6XX2QnlG6QCvRB4nPVgybw=; b=naExQE3kCP7xDNKZ3eitQwOMr+BOtuHtJRvDMhW0x6AeoU6b63aumQG+z4tZeKHyQM IHYY1IZVXSIVLpkfDNwLsI9opzWSePw398uu4t66tIYZ2SVuOek6GfqZYKDA0++WnIvf LlqKhp88Ys0a41RNd33mpFFRXYpvqw00q447ymxH57EplAakyX1gZtoQAJ0plC7Y5V0Z USj+lz64x9eIPSUbAC9flbqZRj2J+jhwNWkyeOmIgZcz5ecumEooOUIjs8uTbirWqhe3 Om0lPr6+LuWMAYnGaGc4y1oqCy1QQhkS+LaDaUI//qlLyYwySvNtLl77bLFMsBgPOdbK YTnA== X-Gm-Message-State: AKwxytfvS7HWuJf/hAdJ/GVe8bHBGwo1Fti1Qbu5zuOU8PChj7FUnNBD d3UxFd5Whx7lta+pEpe8SCZZoZsc5T7Vcz8ex2LavK+/ X-Google-Smtp-Source: AH8x227FF7Q/8ePLzf5OejFLHiVLqxu1qRHgl5AxTmSrad7w54oOWzS0iyo+DFAe2Tk2Y1jSfYoBXqzFr0EqpkqVbQw= X-Received: by 10.28.59.69 with SMTP id i66mr3381514wma.159.1516569299924; Sun, 21 Jan 2018 13:14:59 -0800 (PST) MIME-Version: 1.0 Sender: lwhsu.freebsd@gmail.com Received: by 10.28.124.11 with HTTP; Sun, 21 Jan 2018 13:14:59 -0800 (PST) From: Li-Wen Hsu Date: Mon, 22 Jan 2018 05:14:59 +0800 X-Google-Sender-Auth: z4nVB8u9Up5iK7rgK1DrUn5IVJQ Message-ID: Subject: Calling getaddrinfo(3) in 32-bit binary on 64-bit host To: freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jan 2018 21:15:02 -0000 Hi all, Recently I found a strange case: calling getaddrinfo(3) cannot resolve IPv6 address in 32-bit binary on 64-bit host. It happens on vanilla installed 11.1-R and also on r327788 snapshot build. For a program like this: https://gist.github.com/lwhsu/1288aa5be90b9e7da934a3e2bfc55aa3 It works fine when compiled as a 32-bit binary and run on a 32-bit host. As expected, It is also works fine when compiled as a 64-bit binary and run on a 64-bit host However, when taking the 32-bit binary and run on a 64 bit system (with /usr/lib32 installed), getaddrinfo(3) just returns: "Non-recoverable failure in name resolution" The ktrace/kdump outputs are as following: 64 on 64: https://gist.github.com/lwhsu/a89eb72b34a5ce59af075566c65db8ab 32 on 32: https://gist.github.com/lwhsu/c0b763695a44e5f3798276096d9379c0 32 on 64: https://gist.github.com/lwhsu/b2048789726f674681e7646603d2bd75 It seems that it even did not call socket(2). Any suggested way to debug this? I've tried to use gdb, set breaking point at err = getaddrinfo(host, service, &hints, &res); and did "step", but it went to next line in the file: if (err != 0) Instead of stepping into getaddrinfo.c as debugging on 32-bit or 64-bit systems. Best, Li-Wen -- Li-Wen Hsu https://lwhsu.org From owner-freebsd-hackers@freebsd.org Sun Jan 21 23:33:26 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6192FEC1899 for ; Sun, 21 Jan 2018 23:33:26 +0000 (UTC) (envelope-from yuripv@icloud.com) Received: from pv33p00im-asmtp003.me.com (pv33p00im-asmtp003.me.com [17.142.194.252]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 44ECC81310; Sun, 21 Jan 2018 23:33:26 +0000 (UTC) (envelope-from yuripv@icloud.com) Received: from process-dkim-sign-daemon.pv33p00im-asmtp003.me.com by pv33p00im-asmtp003.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) id <0P2X00F00I0ZSV00@pv33p00im-asmtp003.me.com>; Sun, 21 Jan 2018 23:33:18 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=04042017; t=1516577598; bh=rOppae8Tlmnddz2Zvmhas+EpnEcqp5vslH24Lc/Tbio=; h=Subject:To:From:Message-id:Date:MIME-version:Content-type; b=WmhxUPAkViWmLKVa0KF5WZmB5SbWhm63RSMwiAt/ur3kouBlWbcdD8ua4hCYY/8j/ bfsDPRE/fc0Nk7fi0fOFgRyTuXTF5WWBNfCJ8sr/Fy/QwvBgkmc4cBJPFWoFsVOzT2 kY0Rtmik+xmk9F7Jo4SdJB9YO696y4RfU4GcLANDD6CIWgNgKMfZEEenUImAz2jtt/ R3CC/EEdcL2gFvvp3OgTHWxSBoAD17kem2+lDCQT+vhBklFWL1CRT6ibLWmFjmJfqj 10WT0Ugog0+ZNXkkQ1B/RCGFzbC3OVf2YkM8Yzoj9bUy9Y9kCVkyQBIX8DsjrAGvKx zWPO+rp2k72xw== Received: from icloud.com ([127.0.0.1]) by pv33p00im-asmtp003.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) with ESMTPSA id <0P2X001FMIRCMR10@pv33p00im-asmtp003.me.com>; Sun, 21 Jan 2018 23:33:16 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-01-21_12:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 clxscore=1011 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1801210338 Subject: Re: Calling getaddrinfo(3) in 32-bit binary on 64-bit host To: Li-Wen Hsu , freebsd-hackers@freebsd.org References: From: Yuri Pankov Message-id: <8c6dc5b5-7640-61fc-b687-08efd1e621ee@icloud.com> Date: Mon, 22 Jan 2018 02:33:12 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-version: 1.0 In-reply-to: Content-type: text/plain; charset=utf-8; format=flowed Content-language: en-US Content-transfer-encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jan 2018 23:33:26 -0000 On Mon, Jan 22, 2018 at 05:14:59AM +0800, Li-Wen Hsu wrote: > Hi all, > > Recently I found a strange case: calling getaddrinfo(3) cannot resolve IPv6 > address in 32-bit binary on 64-bit host. > It happens on vanilla installed 11.1-R and also on r327788 snapshot build. > > For a program like this: > https://gist.github.com/lwhsu/1288aa5be90b9e7da934a3e2bfc55aa3 > > It works fine when compiled as a 32-bit binary and run on a 32-bit host. > As expected, It is also works fine when compiled as a 64-bit binary and run > on a 64-bit host > > However, when taking the 32-bit binary and run on a 64 bit system (with > /usr/lib32 installed), > getaddrinfo(3) just returns: "Non-recoverable failure in name resolution" Apparently, it goes through addrconfig() down to getifaddrs() returning bogus data for IPv6 addresses. This most likely has to with SALIGN being incorrect for 32-bit binary trying to parse route messages from 64-bit kernel. I'm not sure about proper fix here, but changing SALIGN to be 7 (that is, "sizeof(long) - 1" on amd64 platform) makes your test case return correct data. From owner-freebsd-hackers@freebsd.org Mon Jan 22 15:47:22 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 83BEFEC7512 for ; Mon, 22 Jan 2018 15:47:22 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6B150815B3; Mon, 22 Jan 2018 15:47:19 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w0MFl9Ec050678 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 22 Jan 2018 17:47:13 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w0MFl9Ec050678 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w0MFl9I4050677; Mon, 22 Jan 2018 17:47:09 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 22 Jan 2018 17:47:09 +0200 From: Konstantin Belousov To: Yuri Pankov Cc: Li-Wen Hsu , freebsd-hackers@freebsd.org Subject: Re: Calling getaddrinfo(3) in 32-bit binary on 64-bit host Message-ID: <20180122154709.GF55707@kib.kiev.ua> References: <8c6dc5b5-7640-61fc-b687-08efd1e621ee@icloud.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <8c6dc5b5-7640-61fc-b687-08efd1e621ee@icloud.com> User-Agent: Mutt/1.9.2 (2017-12-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jan 2018 15:47:22 -0000 On Mon, Jan 22, 2018 at 02:33:12AM +0300, Yuri Pankov wrote: > On Mon, Jan 22, 2018 at 05:14:59AM +0800, Li-Wen Hsu wrote: > > Hi all, > > > > Recently I found a strange case: calling getaddrinfo(3) cannot resolve IPv6 > > address in 32-bit binary on 64-bit host. > > It happens on vanilla installed 11.1-R and also on r327788 snapshot build. > > > > For a program like this: > > https://gist.github.com/lwhsu/1288aa5be90b9e7da934a3e2bfc55aa3 > > > > It works fine when compiled as a 32-bit binary and run on a 32-bit host. > > As expected, It is also works fine when compiled as a 64-bit binary and run > > on a 64-bit host > > > > However, when taking the 32-bit binary and run on a 64 bit system (with > > /usr/lib32 installed), > > getaddrinfo(3) just returns: "Non-recoverable failure in name resolution" > > Apparently, it goes through addrconfig() down to getifaddrs() returning > bogus data for IPv6 addresses. This most likely has to with SALIGN > being incorrect for 32-bit binary trying to parse route messages from > 64-bit kernel. I'm not sure about proper fix here, but changing SALIGN > to be 7 (that is, "sizeof(long) - 1" on amd64 platform) makes your test > case return correct data. Thank you for the diagnostic. The following worked for me. Most likely there are may be more issues, since there are more SA_SIZE() uses from sysctl context. Also, it is probably impossible to provide COMPAT32 for rtsock itself. diff --git a/sys/net/rtsock.c b/sys/net/rtsock.c index eea4cb9459d..26b888f2153 100644 --- a/sys/net/rtsock.c +++ b/sys/net/rtsock.c @@ -112,6 +112,12 @@ struct ifa_msghdrl32 { int32_t ifam_metric; struct if_data ifam_data; }; + +#define SA_SIZE32(sa) \ + ( (((struct sockaddr *)(sa))->sa_len == 0) ? \ + sizeof(int) : \ + 1 + ( (((struct sockaddr *)(sa))->sa_len - 1) | (sizeof(int) - 1) ) ) + #endif /* COMPAT_FREEBSD32 */ MALLOC_DEFINE(M_RTABLE, "routetbl", "routing tables"); @@ -1116,6 +1122,9 @@ rtsock_msg_buffer(int type, struct rt_addrinfo *rtinfo, struct walkarg *w, int * struct sockaddr_storage ss; struct sockaddr_in6 *sin6; #endif +#ifdef COMPAT_FREEBSD32 + bool compat32 = false; +#endif switch (type) { @@ -1123,9 +1132,10 @@ rtsock_msg_buffer(int type, struct rt_addrinfo *rtinfo, struct walkarg *w, int * case RTM_NEWADDR: if (w != NULL && w->w_op == NET_RT_IFLISTL) { #ifdef COMPAT_FREEBSD32 - if (w->w_req->flags & SCTL_MASK32) + if (w->w_req->flags & SCTL_MASK32) { len = sizeof(struct ifa_msghdrl32); - else + compat32 = true; + } else #endif len = sizeof(struct ifa_msghdrl); } else @@ -1139,6 +1149,7 @@ rtsock_msg_buffer(int type, struct rt_addrinfo *rtinfo, struct walkarg *w, int * len = sizeof(struct if_msghdrl32); else len = sizeof(struct if_msghdr32); + compat32 = true; break; } #endif @@ -1169,7 +1180,12 @@ rtsock_msg_buffer(int type, struct rt_addrinfo *rtinfo, struct walkarg *w, int * if ((sa = rtinfo->rti_info[i]) == NULL) continue; rtinfo->rti_addrs |= (1 << i); - dlen = SA_SIZE(sa); +#ifdef COMPAT_FREEBSD32 + if (compat32) + dlen = SA_SIZE32(sa); + else +#endif + dlen = SA_SIZE(sa); if (cp != NULL && buflen >= dlen) { #ifdef INET6 if (V_deembed_scopeid && sa->sa_family == AF_INET6) { From owner-freebsd-hackers@freebsd.org Mon Jan 22 17:00:59 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 659C2ECA53D for ; Mon, 22 Jan 2018 17:00:59 +0000 (UTC) (envelope-from mail@kkoenig.net) Received: from mx1.outerhaven.de (mx1.outerhaven.de [81.14.236.89]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id F07308490D for ; Mon, 22 Jan 2018 17:00:58 +0000 (UTC) (envelope-from mail@kkoenig.net) Received: from [192.168.2.245] (big-shell.lan [192.168.2.245]) by mx1.outerhaven.de (OpenSMTPD) with ESMTPSA id a4810e8f TLS version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NO for ; Mon, 22 Jan 2018 18:00:54 +0100 (CET) To: freebsd-hackers@freebsd.org From: =?UTF-8?Q?Karsten_K=c3=b6nig?= Subject: Question about 'anonymous' UMA zones Message-ID: Date: Mon, 22 Jan 2018 18:00:54 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Mon, 22 Jan 2018 17:07:31 +0000 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jan 2018 17:00:59 -0000 Hello y'all, I'm sorry for cross-posting but after second examination this list feels more appropriate for my question. I just played a little bit with the zone allocator of the FreeBSD kernel and observed the following: All 'anonymous' zones (is this the appropriate term?) with a size smaller than 512 Bytes have no slabs in my 11.1-VM. This is not the case on my bare metal server (11.1, too). This is the output: # vmstat -z ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP ... 16: 16, 0, 0, 0, 0, 0, 0 32: 32, 0, 0, 0, 0, 0, 0 64: 64, 0, 0, 0, 0, 0, 0 128: 128, 0, 0, 0, 0, 0, 0 256: 256, 0, 0, 0, 0, 0, 0 512: 512, 0, 7906, 70, 97681, 0, 0 ... I would like to know why this is the case; can someone explain this to me? What happens if I allocate e.g. a 256 Byte buffer? Best, Karsten From owner-freebsd-hackers@freebsd.org Mon Jan 22 18:55:27 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BA358ECFF07 for ; Mon, 22 Jan 2018 18:55:27 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: from mail-wr0-x22d.google.com (mail-wr0-x22d.google.com [IPv6:2a00:1450:400c:c0c::22d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4BA0468C73 for ; Mon, 22 Jan 2018 18:55:27 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: by mail-wr0-x22d.google.com with SMTP id v15so6992451wrb.8 for ; Mon, 22 Jan 2018 10:55:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=+sxocHCKJsWnBI8P3FENjOP2lMCxh+amWgE3QCnnmuE=; b=V7AA7g5w2PixT8vNDZLBNiZAGF/jtLWe+i/pEuKq/4RaYUREpWSh+PBJxxn0bohH9A LfxxwHSDO8tJJlT2W+Sy9wrMyN8lta86xyGB/6o/wNRepYGBEDFs6qHMyvHyTGQZrkbE xTVsGZv+/o8pj7dpN8TU2O6MhgLS5g6cNj/3KR0hSC0LJRjbWKlZ3gXuI8xgjTD718Lx yECIQBFjxNo/e5weIQNwoUDGcw6szUMxOUSpwV6XZbNX2VL3vN9QIDRaI/KoeEGhVw2W +fBfyGQOtYlgldMKfVAVDkigB4eKQ7qLw5rj0O1Uyxe0sohYSE/vUCKmN8dH8Lq2mtB/ vLlA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=+sxocHCKJsWnBI8P3FENjOP2lMCxh+amWgE3QCnnmuE=; b=cQScpjAD9AbrHv6kxBbA6lT2B4IHfBj234+lxjEzof2NemwY1Kofz5RtPKJrA1QsMK QLFe0Tt09EPHnxgf8tgIhmp1MgiKEDtx2vmJkmzMCpPRYnOANheLQAnpPFKeuWtW88OD vesQnFgm+nhwO6z3kygN5QZzZNFLgrOdILZM6TRBJ6+NmAclbOwQN/PX/C0KKQQHLg8v 53jEbFKX+Nfc/9vb9cfsxbv3ms0czzuWHMbpKNnNm1L0kOmgnjrtdg6PXOuT04KaBbER UsyRCYae9fWJJfDATIjJ6UOJ2W3TXIXf/IaWA0Tcv4JNmVCx/Nht1ArScKbyfDXFI6ZR /ojg== X-Gm-Message-State: AKwxytdriWHNRNBC0da0eKrmG5qnmHZaOumaW0Ftce/y6Zlt+6+8+MZi ueZ3OhQffipEiXd0FbJA5sbwdOVN1wss5qzTGpA= X-Google-Smtp-Source: AH8x224AwNwSCo2Ul6jN15hd8nHvW9FK6/I/tWpdnM4XrTy+d85zhyu7ltHB/x0/PgyJn4V+heNuZMVoLXxY5JNqnNI= X-Received: by 10.223.129.67 with SMTP id 61mr7219365wrm.271.1516647325534; Mon, 22 Jan 2018 10:55:25 -0800 (PST) MIME-Version: 1.0 Sender: lwhsu.freebsd@gmail.com Received: by 10.28.124.11 with HTTP; Mon, 22 Jan 2018 10:55:24 -0800 (PST) In-Reply-To: <20180122154709.GF55707@kib.kiev.ua> References: <8c6dc5b5-7640-61fc-b687-08efd1e621ee@icloud.com> <20180122154709.GF55707@kib.kiev.ua> From: Li-Wen Hsu Date: Tue, 23 Jan 2018 02:55:24 +0800 X-Google-Sender-Auth: FECnlbNzvbPRYIg4IZhkz0OuPZ4 Message-ID: Subject: Re: Calling getaddrinfo(3) in 32-bit binary on 64-bit host To: Konstantin Belousov Cc: Yuri Pankov , freebsd-hackers@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jan 2018 18:55:27 -0000 On Mon, Jan 22, 2018 at 11:47 PM, Konstantin Belousov wrote: > On Mon, Jan 22, 2018 at 02:33:12AM +0300, Yuri Pankov wrote: > > On Mon, Jan 22, 2018 at 05:14:59AM +0800, Li-Wen Hsu wrote: > > > Hi all, > > > > > > Recently I found a strange case: calling getaddrinfo(3) cannot resolve > IPv6 > > > address in 32-bit binary on 64-bit host. > > > It happens on vanilla installed 11.1-R and also on r327788 snapshot > build. > > > > > > For a program like this: > > > https://gist.github.com/lwhsu/1288aa5be90b9e7da934a3e2bfc55aa3 > > > > > > It works fine when compiled as a 32-bit binary and run on a 32-bit > host. > > > As expected, It is also works fine when compiled as a 64-bit binary > and run > > > on a 64-bit host > > > > > > However, when taking the 32-bit binary and run on a 64 bit system (with > > > /usr/lib32 installed), > > > getaddrinfo(3) just returns: "Non-recoverable failure in name > resolution" > > > > Apparently, it goes through addrconfig() down to getifaddrs() returning > > bogus data for IPv6 addresses. This most likely has to with SALIGN > > being incorrect for 32-bit binary trying to parse route messages from > > 64-bit kernel. I'm not sure about proper fix here, but changing SALIGN > > to be 7 (that is, "sizeof(long) - 1" on amd64 platform) makes your test > > case return correct data. > > Thank you for the diagnostic. The following worked for me. > Most likely there are may be more issues, since there are more SA_SIZE() > uses from sysctl context. Also, it is probably impossible to provide > COMPAT32 for rtsock itself. > Also thanks Yuri for the diagnostic. Just be curious, did you find this out by looking at the code or with some other tools? Another thing I don't understand is why gdb desn't step into getaddrinfo.c in compat32 mode. Thanks Konstantin for fixing this. I've also tested and it works here. I'll try to find other uses of SA_SIZE() and see if they would be run in compat32 mode and works or not. Best, Li-Wen -- Li-Wen Hsu https://lwhsu.org From owner-freebsd-hackers@freebsd.org Mon Jan 22 19:29:34 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5CD17ED16A1 for ; Mon, 22 Jan 2018 19:29:34 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E6C446A649; Mon, 22 Jan 2018 19:29:33 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w0MJTPgv000206 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 22 Jan 2018 21:29:29 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w0MJTPgv000206 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w0MJTPVv000205; Mon, 22 Jan 2018 21:29:25 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Mon, 22 Jan 2018 21:29:25 +0200 From: Konstantin Belousov To: Li-Wen Hsu Cc: Yuri Pankov , freebsd-hackers@freebsd.org Subject: Re: Calling getaddrinfo(3) in 32-bit binary on 64-bit host Message-ID: <20180122192925.GI55707@kib.kiev.ua> References: <8c6dc5b5-7640-61fc-b687-08efd1e621ee@icloud.com> <20180122154709.GF55707@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.2 (2017-12-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jan 2018 19:29:34 -0000 On Tue, Jan 23, 2018 at 02:55:24AM +0800, Li-Wen Hsu wrote: > I'll try to find other uses of SA_SIZE() and see if they would be run in > compat32 mode and works or not. There are enough uses of SA_SIZE() by functions which are perhaps called both from a sysctl handler and for queuing the routing messages into routing socket queue. In the later case, we cannot know ABI of the reader. This is what I referenced when I noted that routing socket's ABI compat is impossible to implement correctly. From owner-freebsd-hackers@freebsd.org Mon Jan 22 21:03:08 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 634F6ED5A1A for ; Mon, 22 Jan 2018 21:03:08 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [78.47.246.247]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E8D156EB5C; Mon, 22 Jan 2018 21:03:07 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (root@eg.sd.rdtc.ru [62.231.161.221] (may be forged)) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id w0ML2xME015312 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Mon, 22 Jan 2018 22:03:00 +0100 (CET) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: kostikbel@gmail.com Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id w0ML2tc5008769 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 23 Jan 2018 04:02:55 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: Calling getaddrinfo(3) in 32-bit binary on 64-bit host To: Konstantin Belousov , Li-Wen Hsu References: <8c6dc5b5-7640-61fc-b687-08efd1e621ee@icloud.com> <20180122154709.GF55707@kib.kiev.ua> <20180122192925.GI55707@kib.kiev.ua> Cc: Yuri Pankov , freebsd-hackers@freebsd.org From: Eugene Grosbein Message-ID: <5A66517B.4040802@grosbein.net> Date: Tue, 23 Jan 2018 04:02:51 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: <20180122192925.GI55707@kib.kiev.ua> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=2.2 required=5.0 tests=BAYES_00, LOCAL_FROM, RDNS_NONE autolearn=no autolearn_force=no version=3.4.1 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * 1.9 RDNS_NONE Delivered to internal network by a host with no rDNS * 2.6 LOCAL_FROM From my domains X-Spam-Level: ** X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on hz.grosbein.net X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 22 Jan 2018 21:03:08 -0000 23.01.2018 2:29, Konstantin Belousov пишет: > On Tue, Jan 23, 2018 at 02:55:24AM +0800, Li-Wen Hsu wrote: >> I'll try to find other uses of SA_SIZE() and see if they would be run in >> compat32 mode and works or not. > There are enough uses of SA_SIZE() by functions which are perhaps called > both from a sysctl handler and for queuing the routing messages into routing > socket queue. In the later case, we cannot know ABI of the reader. This > is what I referenced when I noted that routing socket's ABI compat is > impossible to implement correctly. Can't we supply sysctl handler with some kind of back reference to original userland caller of the syscall to find its ABI? From owner-freebsd-hackers@freebsd.org Tue Jan 23 00:53:33 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E90BEEBB5BB for ; Tue, 23 Jan 2018 00:53:33 +0000 (UTC) (envelope-from yuripv@icloud.com) Received: from pv33p00im-asmtp001.me.com (pv33p00im-asmtp001.me.com [17.142.194.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C26A477671; Tue, 23 Jan 2018 00:53:33 +0000 (UTC) (envelope-from yuripv@icloud.com) Received: from process-dkim-sign-daemon.pv33p00im-asmtp001.me.com by pv33p00im-asmtp001.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) id <0P2Z00L00G8JON00@pv33p00im-asmtp001.me.com>; Tue, 23 Jan 2018 00:53:27 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=04042017; t=1516668807; bh=syXfrKCTfWHU5vOkSThlQ38TRx8kp4tCOAHxQVjj08o=; h=To:From:Subject:Message-id:Date:MIME-version:Content-type; b=L7TxuNvwXb7OyPZ7G80jbnm2a3fZudFvhoJaHcqVNK1K19lkAhtWCK9O/dmhF3cgO SOtSr+p8peuwJXxm0a7pGTWZnQmcc7ggvsL1ilpeYI2Dx6+vvtSzMWbv4PHuPZtgxM CO9Wa2ZtGgayk9168S9dw8tiGkTmGhO6O6FBqEzdUn9xmfRF7E3kQWKT9mDvAyUEoq D+/pFSrlQh4PFncazH20XuoWkFk8wcDSzFcNh8HdUbnBNnE8iVPsxSX6oWPHB9XMu4 23Q6V6MYWGJnFb2y3ipULRCe4dq8E3M3c5+d735suiaP1u/5tUSpOWIu/wZC+ymw/n LzR5CnxUWhBkg== Received: from icloud.com ([127.0.0.1]) by pv33p00im-asmtp001.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) with ESMTPSA id <0P2Z002HMH4WYM00@pv33p00im-asmtp001.me.com>; Tue, 23 Jan 2018 00:53:25 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-01-22_10:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 clxscore=1011 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1801230008 To: freebsd-hackers , Kyle Evans From: Yuri Pankov Subject: libc/regex: r302824 added invalid check breaking collating ranges Message-id: Date: Tue, 23 Jan 2018 03:53:19 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-version: 1.0 Content-type: text/plain; charset=utf-8; format=flowed Content-language: en-US Content-transfer-encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jan 2018 00:53:34 -0000 (CCing Kyle as he's working on regex at the moment and not because he broke something) Hi, r302284 added an invalid check which breaks collating ranges: -if (table->__collate_load_error) { - (void)REQUIRE((uch)start <= (uch)finish, REG_ERANGE); +if (table->__collate_load_error || MB_CUR_MAX > 1) { + (void)REQUIRE(start <= finish, REG_ERANGE); The "MB_CUR_MAX > 1" is wrong, we should be doing proper comparison according to current locale's collation and not simply comparing the wchar_t values. Example -- see Table 1 in http://www.unicode.org/reports/tr10/: Let's try Swedish collation: $ echo 'test' | LC_COLLATE=se_SE.UTF-8 grep '[ö-z]' grep: invalid character range $ echo 'test' | LC_COLLATE=se_SE.UTF-8 grep '[z-ö]' OK, the above seems to be correct, 'ö' > 'z' in Swedish collation, but we just got lucky here, as wchar_t comparison gives us the same result. Now German one: $ echo 'test' | LC_COLLATE=de_DE.UTF-8 grep '[ö-z]' grep: invalid character range $ echo 'test' | LC_COLLATE=de_DE.UTF-8 grep '[z-ö]' Same, but according to the table, 'ö' < 'z' in German collation! I think the fix here would be to drop the "if (table->__collate_load_error || MB_CUR_MAX > 1)" block entirely as we no longer use the "table" so there's no point in getting it and checking error, wcscoll() which would be called eventually in p_range_cmp() does the table handling itself, and we can't use the direct comparison for anything other than 'C' locale (not sure if it's applicable even there). From owner-freebsd-hackers@freebsd.org Tue Jan 23 06:53:41 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 213C9EB4E8C for ; Tue, 23 Jan 2018 06:53:41 +0000 (UTC) (envelope-from byond.lenox@gmail.com) Received: from mail-it0-f41.google.com (mail-it0-f41.google.com [209.85.214.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E20AF182C for ; Tue, 23 Jan 2018 06:53:40 +0000 (UTC) (envelope-from byond.lenox@gmail.com) Received: by mail-it0-f41.google.com with SMTP id x42so12864895ita.4 for ; Mon, 22 Jan 2018 22:53:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=gsnQvoQ3IsorqMoKazL7eTOKWVOScE7At1dKlboyD9k=; b=Tk6Z2AEfJB0aMEuTPG59uDUK6ldUNPT5zd0jqd0f5M0Z6dz9nohwwV7WQtc1WI+Urm fCQW0wcevFdygIBb+K0aCQ5i4JgijR3INfjeudxJuEe4IHpsZIS6REm2F93J8Vfn7IDA E9MULHdE7T9quCduCJm0P0TAooYHiEGO0Ppkpji8buym+wd16CvptA6wsQmqNv5LWIJT tmqmU5XqpHqd72wnMqtD4qgpr6toBNs3CeKeKetdv4SwCamN1PG2gGW0/LJb2TXz80Rl D0j6FuKkkk//jaSYLeolEIlBhxfuCGJWhCGxx2/qKSsgnneLXsDUjkW6kVQes4AoU1z/ PxYw== X-Gm-Message-State: AKwxytenwBMdQR9VLgl6/eLd/QgnOuN5ZQknl2pwGmU+/4r+OYrgZJxr +u6qYVMzX5xJ+jddYkVJ62k4bZ0b X-Google-Smtp-Source: AH8x226GULjLmy8rM+Tj4DEyxlGQWwnc+viZlUuRaFYZx6CTIZGsaURtyUbUEsL7vDyLLHypYUUA9w== X-Received: by 10.36.190.15 with SMTP id i15mr2368042itf.109.1516686868550; Mon, 22 Jan 2018 21:54:28 -0800 (PST) Received: from mail-io0-f178.google.com (mail-io0-f178.google.com. [209.85.223.178]) by smtp.gmail.com with ESMTPSA id l69sm9084248ioi.11.2018.01.22.21.54.28 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Jan 2018 21:54:28 -0800 (PST) Received: by mail-io0-f178.google.com with SMTP id d13so7480218iog.5 for ; Mon, 22 Jan 2018 21:54:28 -0800 (PST) X-Received: by 10.107.50.210 with SMTP id y201mr352171ioy.224.1516686867978; Mon, 22 Jan 2018 21:54:27 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.157.12 with HTTP; Mon, 22 Jan 2018 21:54:07 -0800 (PST) In-Reply-To: References: From: Kyle Evans Date: Mon, 22 Jan 2018 23:54:07 -0600 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: libc/regex: r302824 added invalid check breaking collating ranges To: Yuri Pankov Cc: freebsd-hackers Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jan 2018 06:53:41 -0000 On Mon, Jan 22, 2018 at 6:53 PM, Yuri Pankov wrote: > (CCing Kyle as he's working on regex at the moment and not because he bro= ke > something) Phew, brief moment of panic until I read this line and went back to actually look at the full revision number you wrote. =3Dp I note here that I know incredibly little about collation issues, so feel free to enlighten me. > Hi, > > r302284 added an invalid check which breaks collating ranges: > > -if (table->__collate_load_error) { > - (void)REQUIRE((uch)start <=3D (uch)finish, REG_ERANGE); > +if (table->__collate_load_error || MB_CUR_MAX > 1) { > + (void)REQUIRE(start <=3D finish, REG_ERANGE); > > The "MB_CUR_MAX > 1" is wrong, we should be doing proper comparison > according to current locale's collation and not simply comparing the wcha= r_t > values. Right; that seems like almost the complete opposite of when you might be able to do a direct comparison like this. As you mention, though, even MB_CUR_MAX =3D=3D 1 wouldn't necessarily be safe. > Example -- see Table 1 in http://www.unicode.org/reports/tr10/: > > Let's try Swedish collation: > $ echo 'test' | LC_COLLATE=3Dse_SE.UTF-8 grep '[=C3=B6-z]' > grep: invalid character range > $ echo 'test' | LC_COLLATE=3Dse_SE.UTF-8 grep '[z-=C3=B6]' > > OK, the above seems to be correct, '=C3=B6' > 'z' in Swedish collation, b= ut we > just got lucky here, as wchar_t comparison gives us the same result. > > Now German one: > $ echo 'test' | LC_COLLATE=3Dde_DE.UTF-8 grep '[=C3=B6-z]' > grep: invalid character range > $ echo 'test' | LC_COLLATE=3Dde_DE.UTF-8 grep '[z-=C3=B6]' > > Same, but according to the table, '=C3=B6' < 'z' in German collation! > > I think the fix here would be to drop the "if (table->__collate_load_erro= r > || MB_CUR_MAX > 1)" block entirely as we no longer use the "table" so > there's no point in getting it and checking error, wcscoll() which would = be > called eventually in p_range_cmp() does the table handling itself, and we > can't use the direct comparison for anything other than 'C' locale (not s= ure > if it's applicable even there). I've arrived at the same conclusion, and it makes me both sad and happy at the same time. I had a lot more written here, but erased it because I clearly fail to read some of this stuff. I have a proposed patch at [1] that addresses this in theory, but the patch is not right and breaks things in en_US.UTF-8 at the very least. `echo "TEST" | sed -e 's/[s-t]/x/g'` outputs "TExT"; debug output shows that it's adding to the character set: 'S', 's', 't', and U+00DF. I've not been able to dig any deeper and figure out why p_range_comp/wcscoll place these things in the range of [s-t] (115 - 116) with en_US.UTF-8 -- the answer isn't immediately obvious to me. Do note that this patch might not apply against a clean -head, but the spirit of it is still there. It implements your suggestion, but rewrites the loop in the (former) second branch to keep track of contiguous ranges of characters. This is a (pehaps misguided) attempt to take more of the performance hit at expr compilation time rather than execution, reducing cs->wides as much as possible so that we don't have to do as many comparisons in not worst-case scenarios. I've also rearranged the loops in regex2.h:CHIN1 so that we check the smaller/faster comparisons before checking the larger character set at wide. I don't know if it really makes for any of a performance difference, but it seems worth it to at least benchmark for some more complex character sets one can build in different locales with mixtures of actually contiguous ranges (in the current locale) and randomly distributed non-contiguous characters. [1] https://people.freebsd.org/~kevans/regex-collation.diff From owner-freebsd-hackers@freebsd.org Tue Jan 23 07:15:17 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B921EB658A for ; Tue, 23 Jan 2018 07:15:17 +0000 (UTC) (envelope-from byond.lenox@gmail.com) Received: from mail-io0-f182.google.com (mail-io0-f182.google.com [209.85.223.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id E70393C38 for ; Tue, 23 Jan 2018 07:15:16 +0000 (UTC) (envelope-from byond.lenox@gmail.com) Received: by mail-io0-f182.google.com with SMTP id f89so12252452ioj.4 for ; Mon, 22 Jan 2018 23:15:16 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=CQXGmdXom7YSMziybkk9hSTd357uV4qcVpZLNrvSj98=; b=J3jdxenf0rRQV6BH2dmcyJzl06T89638h1B/j5OQisgDmCKgodZFU2Cjhc09slvajY h9RprHQI8vzNCXfNxmxPrh6DuD5u7pJXk+vTINUE9Lmsh8EFXbIifba7D+mipzY5Hd47 HqGcNPqsBrX9zsZ/omcX/VP777YvlsucCAg6ESDcMQ/cMUioG59nRFiUEqrAzO7BQc8A bQ/wg5Ai4h7PxLdZcAn12HKCMxkTk1OXB0J1j5Xjc2FSEoEMxSyZ5OCkks6peuHJLg2z EtrYFsTt0WEo3qNQoSLYPUIpSkGpGhtGC5SsAnLVDwTxtPWf9laSxhhKJ0gtJdV0xBL2 6awA== X-Gm-Message-State: AKwxyteFGMDfsqfEovqlLtxpXMbSk9V+NSdpod5yu98eUFS6Fz8LbFsd tEQag6UgaT95yZ5/UCXLWYWGNBE6 X-Google-Smtp-Source: AH8x224WQjwEdAJFHvH6p5GH9ZxciMw2d293g40dOLvQLAOMJn5Gu+NhTBQN8MeFxPgzdL4nYCuvAA== X-Received: by 10.107.139.78 with SMTP id n75mr2434323iod.252.1516688076691; Mon, 22 Jan 2018 22:14:36 -0800 (PST) Received: from mail-io0-f180.google.com (mail-io0-f180.google.com. [209.85.223.180]) by smtp.gmail.com with ESMTPSA id d189sm8772276iog.77.2018.01.22.22.14.36 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 22 Jan 2018 22:14:36 -0800 (PST) Received: by mail-io0-f180.google.com with SMTP id m11so12097573iob.2 for ; Mon, 22 Jan 2018 22:14:36 -0800 (PST) X-Received: by 10.107.53.221 with SMTP id k90mr2377237ioo.6.1516688076195; Mon, 22 Jan 2018 22:14:36 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.157.12 with HTTP; Mon, 22 Jan 2018 22:14:35 -0800 (PST) Received: by 10.107.157.12 with HTTP; Mon, 22 Jan 2018 22:14:35 -0800 (PST) In-Reply-To: References: From: Kyle Evans Date: Tue, 23 Jan 2018 00:14:35 -0600 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: libc/regex: r302824 added invalid check breaking collating ranges To: Yuri Pankov Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jan 2018 07:15:17 -0000 (apologies for broken quoting, on mobile) On Jan 22, 2018 11:54 PM, "Kyle Evans" wrote: On Mon, Jan 22, 2018 at 6:53 PM, Yuri Pankov wrote: > (CCing Kyle as he's working on regex at the moment and not because he broke > something) Phew, brief moment of panic until I read this line and went back to actually look at the full revision number you wrote. =3Dp I note here that I know incredibly little about collation issues, so feel free to enlighten me. > Hi, > > r302284 added an invalid check which breaks collating ranges: > > -if (table->__collate_load_error) { > - (void)REQUIRE((uch)start <=3D (uch)finish, REG_ERANGE); > +if (table->__collate_load_error || MB_CUR_MAX > 1) { > + (void)REQUIRE(start <=3D finish, REG_ERANGE); > > The "MB_CUR_MAX > 1" is wrong, we should be doing proper comparison > according to current locale's collation and not simply comparing the wchar_t > values. Right; that seems like almost the complete opposite of when you might be able to do a direct comparison like this. As you mention, though, even MB_CUR_MAX =3D=3D 1 wouldn't necessarily be safe. > Example -- see Table 1 in http://www.unicode.org/reports/tr10/: > > Let's try Swedish collation: > $ echo 'test' | LC_COLLATE=3Dse_SE.UTF-8 grep '[=C3=B6-z]' > grep: invalid character range > $ echo 'test' | LC_COLLATE=3Dse_SE.UTF-8 grep '[z-=C3=B6]' > > OK, the above seems to be correct, '=C3=B6' > 'z' in Swedish collation, b= ut we > just got lucky here, as wchar_t comparison gives us the same result. > > Now German one: > $ echo 'test' | LC_COLLATE=3Dde_DE.UTF-8 grep '[=C3=B6-z]' > grep: invalid character range > $ echo 'test' | LC_COLLATE=3Dde_DE.UTF-8 grep '[z-=C3=B6]' > > Same, but according to the table, '=C3=B6' < 'z' in German collation! > > I think the fix here would be to drop the "if (table->__collate_load_erro= r > || MB_CUR_MAX > 1)" block entirely as we no longer use the "table" so > there's no point in getting it and checking error, wcscoll() which would be > called eventually in p_range_cmp() does the table handling itself, and we > can't use the direct comparison for anything other than 'C' locale (not sure > if it's applicable even there). I've arrived at the same conclusion, and it makes me both sad and happy at the same time. I had a lot more written here, but erased it because I clearly fail to read some of this stuff. I have a proposed patch at [1] that addresses this in theory, but the patch is not right and breaks things in en_US.UTF-8 at the very least. `echo "TEST" | sed -e 's/[s-t]/x/g'` outputs "TExT"; debug output shows that it's adding to the character set: 'S', 's', 't', and U+00DF. I've not been able to dig any deeper and figure out why p_range_comp/wcscoll place these things in the range of [s-t] (115 - 116) with en_US.UTF-8 -- the answer isn't immediately obvious to me. I see now where I goofed this up- please disregard. Do note that this patch might not apply against a clean -head, but the spirit of it is still there. It implements your suggestion, but rewrites the loop in the (former) second branch to keep track of contiguous ranges of characters. This is a (pehaps misguided) attempt to take more of the performance hit at expr compilation time rather than execution, reducing cs->wides as much as possible so that we don't have to do as many comparisons in not worst-case scenarios. I've also rearranged the loops in regex2.h:CHIN1 so that we check the smaller/faster comparisons before checking the larger character set at wide. I don't know if it really makes for any of a performance difference, but it seems worth it to at least benchmark for some more complex character sets one can build in different locales with mixtures of actually contiguous ranges (in the current locale) and randomly distributed non-contiguous characters. [1] https://people.freebsd.org/~kevans/regex-collation.diff From owner-freebsd-hackers@freebsd.org Tue Jan 23 08:41:38 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 093F0EBE8F9 for ; Tue, 23 Jan 2018 08:41:38 +0000 (UTC) (envelope-from yuripv@icloud.com) Received: from pv33p00im-asmtp003.me.com (pv33p00im-asmtp003.me.com [17.142.194.252]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CBE346CF61 for ; Tue, 23 Jan 2018 08:41:37 +0000 (UTC) (envelope-from yuripv@icloud.com) Received: from process-dkim-sign-daemon.pv33p00im-asmtp003.me.com by pv33p00im-asmtp003.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) id <0P2Z00L00SKL7K00@pv33p00im-asmtp003.me.com> for freebsd-hackers@freebsd.org; Tue, 23 Jan 2018 05:37:08 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=04042017; t=1516685828; bh=IDp+F9eUOuPodPWkEJKNExLxzE4w3VnNYHrWUDBF8L0=; h=Subject:From:To:Message-id:Date:MIME-version:Content-type; b=E8039XoqL6oXR5klmKcg9YotJVrNaL0hDagjQl6hYF86IRFRbq9zWIHlF2Lfuk5OY 4eZjXwpxbZWw3CzWNqXnSXg5CAbSazfmX7XENXJgzUyKWrElS9SCkG9ypAR7hhFcoo VRJg0t7FWGazKQo1fPWGBQG7pY0j5iqcMrNzv0hNmego2wMKbYgB/8WB+9GpVAT5F8 wILamBg+k5y93T4335nBk5GWf5XEHRtdyFod1nCMwd3liIC8BB9pgbuIBP1Qvj68ON M48tv/UL67vDKqmBaAuHOexFwg69QN0SH2hsHWHBWg+1xgTedCl79zAJ8EdkFc+VoR cn3Ud3xBIa/AQ== Received: from icloud.com ([127.0.0.1]) by pv33p00im-asmtp003.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) with ESMTPSA id <0P2Z00DBIU9OLM20@pv33p00im-asmtp003.me.com> for freebsd-hackers@freebsd.org; Tue, 23 Jan 2018 05:37:07 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-01-23_02:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 clxscore=1015 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1801230078 Subject: Re: libc/regex: r302824 added invalid check breaking collating ranges From: Yuri Pankov To: freebsd-hackers References: Message-id: Date: Tue, 23 Jan 2018 08:36:59 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-version: 1.0 In-reply-to: Content-type: text/plain; charset=utf-8; format=flowed Content-language: en-US Content-transfer-encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jan 2018 08:41:38 -0000 On Tue, Jan 23, 2018 at 03:53:19AM +0300, Yuri Pankov wrote: > (CCing Kyle as he's working on regex at the moment and not because he > broke something) > > Hi, > > r302284 added an invalid check which breaks collating ranges: > > -if (table->__collate_load_error) { > - (void)REQUIRE((uch)start <= (uch)finish, REG_ERANGE); > +if (table->__collate_load_error || MB_CUR_MAX > 1) { > + (void)REQUIRE(start <= finish, REG_ERANGE); > > The "MB_CUR_MAX > 1" is wrong, we should be doing proper comparison > according to current locale's collation and not simply comparing the > wchar_t values. After re-reading the specification I now see that what looked like a bug is actually an implementation choice, though the one that needs to be documentated. I'll update the man page if anyone is willing to review (and commit) the changes. From owner-freebsd-hackers@freebsd.org Tue Jan 23 14:17:09 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A0C6FED6101 for ; Tue, 23 Jan 2018 14:17:09 +0000 (UTC) (envelope-from byond.lenox@gmail.com) Received: from mail-io0-f182.google.com (mail-io0-f182.google.com [209.85.223.182]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 707A07D69D for ; Tue, 23 Jan 2018 14:17:09 +0000 (UTC) (envelope-from byond.lenox@gmail.com) Received: by mail-io0-f182.google.com with SMTP id l17so1070337ioc.3 for ; Tue, 23 Jan 2018 06:17:09 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=RnD9rOnAyVDlLQF4AxQznLbQtQswgXj4rDW9jEerqjU=; b=khMQIFIfUwB8YecPnPWZsmhtC91rU5io0pLKaaejIy1NBfXKAQ9CgN8MVWgXMQniGf gWiqUHpHtH/UKVQch6E3+d93EP1a6YeayoqTmfhIooPEfdHGOp7fWy+3rhEm4yZBp3GU FWq09+r8juQKMfliYZkYhv4aoYNrdoSFQkrf9df/Dh8QnKf54/TIZjOuIvcqZ03s+cjc Nk+jc8yl+0WXfruP/pMaslc6HnU+iw6lloBx2I/l1U1gDPUKVM8PFK77jAnLcOIj/PAh tAu8Et9sZ4dsy7FpPvaG+w6lQyZ5WbhlXGWRX9u/u0j67ydkVJJlmdR5wkWusp5WdRiP fo3A== X-Gm-Message-State: AKwxytdxt7IGUSLxURFurAqrKO03HwhLtfkBklAKrheF+6kpLvNl3scs dOs9g6z1Rnt02trcrUOfg//+UuXL X-Google-Smtp-Source: AH8x224BWyEWRG0TZYGb9cHpeAEykmhshF0ICYuJqP8to8DuYXr5nlHVoDGoZsgX00j6xCfwzipojg== X-Received: by 10.107.164.134 with SMTP id d6mr3752063ioj.176.1516716654405; Tue, 23 Jan 2018 06:10:54 -0800 (PST) Received: from mail-io0-f169.google.com (mail-io0-f169.google.com. [209.85.223.169]) by smtp.gmail.com with ESMTPSA id f139sm3890622iof.35.2018.01.23.06.10.53 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Jan 2018 06:10:54 -0800 (PST) Received: by mail-io0-f169.google.com with SMTP id 25so1038962ioj.9 for ; Tue, 23 Jan 2018 06:10:53 -0800 (PST) X-Received: by 10.107.53.221 with SMTP id k90mr3713669ioo.6.1516716653378; Tue, 23 Jan 2018 06:10:53 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.157.12 with HTTP; Tue, 23 Jan 2018 06:10:32 -0800 (PST) In-Reply-To: References: From: Kyle Evans Date: Tue, 23 Jan 2018 08:10:32 -0600 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: libc/regex: r302824 added invalid check breaking collating ranges To: Yuri Pankov Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jan 2018 14:17:09 -0000 On Mon, Jan 22, 2018 at 11:36 PM, Yuri Pankov wrote: > On Tue, Jan 23, 2018 at 03:53:19AM +0300, Yuri Pankov wrote: >> >> (CCing Kyle as he's working on regex at the moment and not because he >> broke something) >> >> Hi, >> >> r302284 added an invalid check which breaks collating ranges: >> >> -if (table->__collate_load_error) { >> - (void)REQUIRE((uch)start <= (uch)finish, REG_ERANGE); >> +if (table->__collate_load_error || MB_CUR_MAX > 1) { >> + (void)REQUIRE(start <= finish, REG_ERANGE); >> >> The "MB_CUR_MAX > 1" is wrong, we should be doing proper comparison >> according to current locale's collation and not simply comparing the >> wchar_t values. > > > After re-reading the specification I now see that what looked like a bug is > actually an implementation choice, though the one that needs to be > documentated. I'll update the man page if anyone is willing to review (and > commit) the changes. Can you point to the section of specification that indicates this is OK behavior? It doesn't seem desirable, but I see that GNU systems will operate in the same manner that we do now. From owner-freebsd-hackers@freebsd.org Tue Jan 23 19:11:09 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 56A4AEC1008 for ; Tue, 23 Jan 2018 19:11:09 +0000 (UTC) (envelope-from yuripv@icloud.com) Received: from pv33p00im-asmtp002.me.com (pv33p00im-asmtp002.me.com [17.142.194.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 3880D69FFE; Tue, 23 Jan 2018 19:11:09 +0000 (UTC) (envelope-from yuripv@icloud.com) Received: from process-dkim-sign-daemon.pv33p00im-asmtp002.me.com by pv33p00im-asmtp002.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) id <0P3000000VEKPM00@pv33p00im-asmtp002.me.com>; Tue, 23 Jan 2018 19:10:57 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=04042017; t=1516734657; bh=hRrqkbv7FzRKg+04ixA7QSHxnn2WIJg5ps+UXR3v8LI=; h=Subject:To:From:Message-id:Date:MIME-version:Content-type; b=oYH2+FETj+NOFKdw5ZHCaGJypJPk2lH4MZIApl7yPFhzwifpUrDf2+OFS38K1KUWd hTAlO75yuPLraHW1p4rtCJS5t2HMw2P8ux7NJ5CQ3B7Qmss/W1nUyQDFsvoDC66g7K 6rnfIF2HcYAeFtZgOtWiAHzh4MkQjTMQQh6QQqEf4kat8jdATekjGHJeR28i0wjt8w GsUwoKdZO4HKsurABWiWqUZtPUbIVZRX+HrbDWqrnZhZNUL/e3bAXCLy5AKPyY/lHi /aBytcVx45REMJrv0d7FPEJEhYSKtPmvaxHMrIeb9DXRk/cYGnuY7HK1qOrZf/ghsI ppwb7KQ3/Mknw== Received: from icloud.com ([127.0.0.1]) by pv33p00im-asmtp002.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) with ESMTPSA id <0P3000HR4VY28D10@pv33p00im-asmtp002.me.com>; Tue, 23 Jan 2018 19:10:55 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-01-23_07:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 clxscore=1015 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1801230262 Subject: Re: libc/regex: r302824 added invalid check breaking collating ranges To: Kyle Evans Cc: FreeBSD Hackers References: From: Yuri Pankov Message-id: Date: Tue, 23 Jan 2018 22:10:49 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-version: 1.0 In-reply-to: Content-type: text/plain; charset=utf-8; format=flowed Content-language: en-US Content-transfer-encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jan 2018 19:11:09 -0000 On Tue, Jan 23, 2018 at 08:10:32AM -0600, Kyle Evans wrote: > On Mon, Jan 22, 2018 at 11:36 PM, Yuri Pankov wrote: >> On Tue, Jan 23, 2018 at 03:53:19AM +0300, Yuri Pankov wrote: >>> >>> (CCing Kyle as he's working on regex at the moment and not because he >>> broke something) >>> >>> Hi, >>> >>> r302284 added an invalid check which breaks collating ranges: >>> >>> -if (table->__collate_load_error) { >>> - (void)REQUIRE((uch)start <= (uch)finish, REG_ERANGE); >>> +if (table->__collate_load_error || MB_CUR_MAX > 1) { >>> + (void)REQUIRE(start <= finish, REG_ERANGE); >>> >>> The "MB_CUR_MAX > 1" is wrong, we should be doing proper comparison >>> according to current locale's collation and not simply comparing the >>> wchar_t values. >> >> >> After re-reading the specification I now see that what looked like a bug is >> actually an implementation choice, though the one that needs to be >> documentated. I'll update the man page if anyone is willing to review (and >> commit) the changes. > > Can you point to the section of specification that indicates this is > OK behavior? It doesn't seem desirable, but I see that GNU systems > will operate in the same manner that we do now. Here -- http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html: ------------------------------------------------------------------------ In the POSIX locale, a range expression represents the set of collating elements that fall between two elements in the collation sequence, inclusive. In other locales, a range expression has unspecified behavior: strictly conforming applications shall not rely on whether the range expression is valid, or on the set of collating elements matched. ------------------------------------------------------------------------ I've tried to "fix" what I was seeing as well, and yes, everything outside of ASCII is ugly, e.g. Cyrillic 'а-я' would match much more than you could expect if you are doing lookups based on collation order (capital chars and a lot of other symbols). So what we have currently looks the least evil to me: - non-collating ASCII lookups for any locale -- looking at the log for regcomp.c there was an attempt to "fix" this, but it was reverted as a lot of existing code relies on this; - non-collating multi-byte locale lookups -- they will work for almost all cases, and where they don't, well POSIX says it's undefined :D - collating single-byte locale lookups for outside of ASCII range -- they make sense as collation order there doesn't seem to mix small/caps/other characters together. What I think we need to do is document this as implementation choice in the code and regex(3) "IMPLEMENTATION NOTES" so that another poor soul doesn't come trying to fix it as I did :-) From owner-freebsd-hackers@freebsd.org Tue Jan 23 19:28:24 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E7065EC1FF8 for ; Tue, 23 Jan 2018 19:28:24 +0000 (UTC) (envelope-from byond.lenox@gmail.com) Received: from mail-io0-f177.google.com (mail-io0-f177.google.com [209.85.223.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B53306ACC1 for ; Tue, 23 Jan 2018 19:28:24 +0000 (UTC) (envelope-from byond.lenox@gmail.com) Received: by mail-io0-f177.google.com with SMTP id f34so2126274ioi.13 for ; Tue, 23 Jan 2018 11:28:24 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=iCbtsUDVGJ37fr6Qut1EpyFURUB6EyAGTOvUf6BXHjk=; b=FsZqYI1JpjR6IBoyaewN7GZHnQADFAxAtDsF6yYdl/EZ3ie0ZiRvi3+lYnvUbsIuCi gcXsfHFdNjd9y3xsW6QsMRQoD8E1j/6rfwaQ+nacGv0hQCTmkA0UlIDGrmPLEbLwX/oi qstf3Gv2J4wB4ME+se5HidpJZZ9qN7magErsTB5Q2aJYoIKQCl500jkQ7qXNLUjm36l5 G3ohsMaMLKHXT+69I5Un6Y6IMgBDzeI3LcJUbbCFltHDDxJkTRxlZq4LlhmMhmzVdnJK SZ2bFZ1AOJ4+wpeDuduG7qp9pS+QQZkV3i+J9FixOrSQBXXDkKspMPG/eGzaL+T48/LZ GXiQ== X-Gm-Message-State: AKwxytf8ArjR2meM2XBc9UpW75zJaME8x7tXxejgDh6RVJ695nKxvaV4 6msItX+mvGPsYarOzEGtLLwg/Wtk X-Google-Smtp-Source: AH8x225dc2NJQ9Eh+LtZw9Bv6a1on+lttvFBIc9mu0hRIW79FyoMdTGdRL/hcUI//au3IHhCtmwREQ== X-Received: by 10.107.53.22 with SMTP id c22mr5162982ioa.189.1516735345685; Tue, 23 Jan 2018 11:22:25 -0800 (PST) Received: from mail-it0-f51.google.com (mail-it0-f51.google.com. [209.85.214.51]) by smtp.gmail.com with ESMTPSA id e18sm5621921itc.4.2018.01.23.11.22.25 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 23 Jan 2018 11:22:25 -0800 (PST) Received: by mail-it0-f51.google.com with SMTP id 196so2277128iti.5 for ; Tue, 23 Jan 2018 11:22:25 -0800 (PST) X-Received: by 10.36.179.67 with SMTP id z3mr5484443iti.67.1516735344921; Tue, 23 Jan 2018 11:22:24 -0800 (PST) MIME-Version: 1.0 Received: by 10.107.157.12 with HTTP; Tue, 23 Jan 2018 11:22:04 -0800 (PST) In-Reply-To: References: From: Kyle Evans Date: Tue, 23 Jan 2018 13:22:04 -0600 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: libc/regex: r302824 added invalid check breaking collating ranges To: Yuri Pankov Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 23 Jan 2018 19:28:25 -0000 On Tue, Jan 23, 2018 at 1:10 PM, Yuri Pankov wrote: > On Tue, Jan 23, 2018 at 08:10:32AM -0600, Kyle Evans wrote: >> >> On Mon, Jan 22, 2018 at 11:36 PM, Yuri Pankov wrote: >>> >>> On Tue, Jan 23, 2018 at 03:53:19AM +0300, Yuri Pankov wrote: >>>> >>>> >>>> (CCing Kyle as he's working on regex at the moment and not because he >>>> broke something) >>>> >>>> Hi, >>>> >>>> r302284 added an invalid check which breaks collating ranges: >>>> >>>> -if (table->__collate_load_error) { >>>> - (void)REQUIRE((uch)start <=3D (uch)finish, REG_ERANGE); >>>> +if (table->__collate_load_error || MB_CUR_MAX > 1) { >>>> + (void)REQUIRE(start <=3D finish, REG_ERANGE); >>>> >>>> The "MB_CUR_MAX > 1" is wrong, we should be doing proper comparison >>>> according to current locale's collation and not simply comparing the >>>> wchar_t values. >>> >>> >>> >>> After re-reading the specification I now see that what looked like a bu= g >>> is >>> actually an implementation choice, though the one that needs to be >>> documentated. I'll update the man page if anyone is willing to review >>> (and >>> commit) the changes. >> >> >> Can you point to the section of specification that indicates this is >> OK behavior? It doesn't seem desirable, but I see that GNU systems >> will operate in the same manner that we do now. > > > Here -- > http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html: > ------------------------------------------------------------------------ > In the POSIX locale, a range expression represents the set of collating > elements that fall between two elements in the collation sequence, > inclusive. In other locales, a range expression has unspecified behavior: > strictly conforming applications shall not rely on whether the range > expression is valid, or on the set of collating elements matched. > ------------------------------------------------------------------------ > Thanks- our current behavior seems reasonable in that context. > I've tried to "fix" what I was seeing as well, and yes, everything outsid= e > of ASCII is ugly, e.g. Cyrillic '=D0=B0-=D1=8F' would match much more tha= n you could > expect if you are doing lookups based on collation order (capital chars a= nd > a lot of other symbols). > > So what we have currently looks the least evil to me: > > - non-collating ASCII lookups for any locale -- looking at the log for > regcomp.c there was an attempt to "fix" this, but it was reverted as > a lot of existing code relies on this; > - non-collating multi-byte locale lookups -- they will work for almost > all cases, and where they don't, well POSIX says it's undefined :D > - collating single-byte locale lookups for outside of ASCII range -- > they make sense as collation order there doesn't seem to mix > small/caps/other characters together. > > What I think we need to do is document this as implementation choice in t= he > code and regex(3) "IMPLEMENTATION NOTES" so that another poor soul doesn'= t > come trying to fix it as I did :-) I agree with your assessment- such a patch would be welcomed, especially before I go and revise a bunch of this for clarification in a future libregex world. From owner-freebsd-hackers@freebsd.org Wed Jan 24 00:45:31 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C6E11ED6A1A for ; Wed, 24 Jan 2018 00:45:31 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: from mail-io0-x22e.google.com (mail-io0-x22e.google.com [IPv6:2607:f8b0:4001:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 66CFC6F71F for ; Wed, 24 Jan 2018 00:45:31 +0000 (UTC) (envelope-from chuck@tuffli.net) Received: by mail-io0-x22e.google.com with SMTP id 72so2973917iom.10 for ; Tue, 23 Jan 2018 16:45:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=tuffli-net.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=8DCWxXceq1XRyNa61N1I/H+oz0f0PiKWND6NeC2sAjk=; b=LfH76WaPU6RWjqpexVkbrYv3+3hl54i7BfI5EeFkvbnq/iRrVdTOiCfcQFA+CiPL5L 0a7vRmfN1VbzYzpDF4tetELmcihB2+T+ABzg9ibRBXR7f4pXHov2D52oeApBucQTdZp5 F0wuyjh0jQ3XJqT6nswznLDfrny6RihDDYbSOZgIbj7/b10v+lCG5frTsqkTBMtKzT81 dgcbHdt71jdiwRTfCQa0bw3QE9A5GkVJV0jcehtarOi3Aan8glT7KHWcoMQmL2gCvFha A4yiEwZwpVqrHgBhQfBGMD1HJFxADB9I1SrjlMSm2HJbXHNJiQbhbi3DV/AcpB60WATL paBQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=8DCWxXceq1XRyNa61N1I/H+oz0f0PiKWND6NeC2sAjk=; b=PuEpTCgPH0UKAVSgkwoScWV5YvYAo43ESqsBWJdCxhYs28RytTRs817zot5RoDKvp2 WnmhgZxXZnxAFnJIPZjvU3q0xJ7SgeVMgwntIAzf/IdBnRi5pPFu+Tk9qnjIwULQNOUD 2VddpFNERYCUvXUYG7VxmiulAJZZs42HQMPZnqxU8ZOx9btFUSROtcDSWc0nSzctQGH/ UVrMYhMBh97DaDXmCuAJQDFZLECIOKakCtYnFMkVvzA9ECwmt9EH4BqjlqOXPh85eOIR AzDwFwZ3Yw/TnpJwJZfRgmWAmLW0DsXYReAIf1b9mgiWvtN3F7cCpmsbgmx+s2fbAq5e y/GQ== X-Gm-Message-State: AKwxytdTmtOKg1s0oNWORJ9c0ZUWIB4MBtNLzl0tmdi9mGEy4zqFDGxB D1RrT9Bx6cOuDrL6EEaPX0oP0rVjoJNWCLTgXn+bew== X-Google-Smtp-Source: AH8x224n2h2z5Dbg+JzBP8BHXdBcZ6P79vB66I0dM+QQerY5Tjg+DoUZcWHEBw+6qWDEatv9v2ttoz+KyYmB4YDlCek= X-Received: by 10.107.51.149 with SMTP id z143mr6446840ioz.287.1516754730483; Tue, 23 Jan 2018 16:45:30 -0800 (PST) MIME-Version: 1.0 Received: by 10.2.21.86 with HTTP; Tue, 23 Jan 2018 16:45:30 -0800 (PST) In-Reply-To: <32C3FADA-073C-4942-B12F-AEB7F345766E@dons.net.au> References: <20180116095908.GQ1684@kib.kiev.ua> <32C3FADA-073C-4942-B12F-AEB7F345766E@dons.net.au> From: Chuck Tuffli Date: Tue, 23 Jan 2018 16:45:30 -0800 Message-ID: Subject: Re: examining Linux core file? To: "O'Connor, Daniel" Cc: Konstantin Belousov , FreeBSD Hackers , Pedro Giffuni Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jan 2018 00:45:32 -0000 On Tue, Jan 16, 2018 at 5:30 PM, O'Connor, Daniel wrote: > > >> On 16 Jan 2018, at 20:29, Konstantin Belousov wrote: >>> >>> My memory is sketchy but you may need to build a complete >>> cross-toolchain for that. Long-long ago we actually did that but the >>> resulting linux binaries were weird and needed branding always. >> >> No cross-toolchain will help there. Problem is that the binary is Linux, >> while core is FreeBSD. There are enough details significant to the debugger >> that make such combination a new platform. > > Using ktrace / linux_kdump might help - or at least give some clues as to where it's failing too. The ktrace output is interesting. Is there any way to show the address causing the segfault? dmesg shows: pid 1123 (apt-get), uid 0: exited on signal 11 (core dumped) ktrace shows: ... 1123 apt-get 1516724569.278210 CALL L64 write(0x1,0x800644000,0x21) 1123 apt-get 1516724569.284901 GIO L64 fd 1 wrote 33 bytes "\rBuilding dependency tree... 50%\r" 1123 apt-get 1516724569.291711 RET L64 write 33/0x21 1123 apt-get 1516724569.298353 CALL L64 gettimeofday(0x7fffffffd410,0) 1123 apt-get 1516724569.305308 RET L64 gettimeofday 0 1123 apt-get 1516724569.312174 PSIG L64 SIGSEGV SIG_DFL code=SEGV_MAPERR 1123 apt-get 1516724569.312183 NAMI L64 "apt-get.core" --chuck From owner-freebsd-hackers@freebsd.org Wed Jan 24 02:54:04 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3A205EB5FCB for ; Wed, 24 Jan 2018 02:54:04 +0000 (UTC) (envelope-from darius@dons.net.au) Received: from ipmail07.adl2.internode.on.net (ipmail07.adl2.internode.on.net [150.101.137.131]) by mx1.freebsd.org (Postfix) with ESMTP id 5342D73B5E for ; Wed, 24 Jan 2018 02:54:02 +0000 (UTC) (envelope-from darius@dons.net.au) Received: from 124-169-232-53.dyn.iinet.net.au (HELO midget.dons.net.au) ([124.169.232.53]) by ipmail07.adl2.internode.on.net with ESMTP; 24 Jan 2018 13:23:54 +1030 Received: from midget.dons.net.au (localhost [127.0.0.1]) by midget.dons.net.au (8.15.1/8.14.9) with ESMTPS id w0O2rc2J033537 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Wed, 24 Jan 2018 13:23:49 +1030 (CST) (envelope-from darius@dons.net.au) Received: (from mailnull@localhost) by midget.dons.net.au (8.15.1/8.14.9/Submit) id w0O2nPD7030247 for ; Wed, 24 Jan 2018 13:19:25 +1030 (CST) (envelope-from darius@dons.net.au) X-Authentication-Warning: midget.dons.net.au: mailnull set sender to using -f Received: from [10.176.138.114] (ns.dons.net.au [10.0.2.1]) by ns.dons.net.au (envelope-sender ) (MIMEDefang) with ESMTP id w0O2nJPJ030244; Wed, 24 Jan 2018 13:19:25 +1030 Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: examining Linux core file? From: "O'Connor, Daniel" In-Reply-To: Date: Wed, 24 Jan 2018 13:19:51 +1030 Cc: Konstantin Belousov , FreeBSD Hackers , Pedro Giffuni Content-Transfer-Encoding: quoted-printable Message-Id: <8ECC1C72-58D3-46D2-B577-22123C7A81DB@dons.net.au> References: <20180116095908.GQ1684@kib.kiev.ua> <32C3FADA-073C-4942-B12F-AEB7F345766E@dons.net.au> To: Chuck Tuffli X-Mailer: Apple Mail (2.3445.5.20) X-Spam-Score: -1 () No, score=-1.0 required=5.0 tests=ALL_TRUSTED, T_RP_MATCHES_RCVD autolearn=unavailable autolearn_force=no version=3.4.0 X-Scanned-By: MIMEDefang 2.75 on 10.0.2.1 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jan 2018 02:54:04 -0000 > On 24 Jan 2018, at 11:15, Chuck Tuffli wrote: >>=20 >> Using ktrace / linux_kdump might help - or at least give some clues = as to where it's failing too. >=20 > The ktrace output is interesting. Is there any way to show the address > causing the segfault? Not AFAIK, because ktrace only knows about syscalls :( > dmesg shows: > pid 1123 (apt-get), uid 0: exited on signal 11 (core dumped) >=20 > ktrace shows: > ... > 1123 apt-get 1516724569.278210 CALL L64 = write(0x1,0x800644000,0x21) > 1123 apt-get 1516724569.284901 GIO L64 fd 1 wrote 33 bytes > "\rBuilding dependency tree... 50%\r" > 1123 apt-get 1516724569.291711 RET L64 write 33/0x21 > 1123 apt-get 1516724569.298353 CALL L64 = gettimeofday(0x7fffffffd410,0) > 1123 apt-get 1516724569.305308 RET L64 gettimeofday 0 > 1123 apt-get 1516724569.312174 PSIG L64 SIGSEGV SIG_DFL = code=3DSEGV_MAPERR > 1123 apt-get 1516724569.312183 NAMI L64 "apt-get.core" Looks pretty innocent.. :-/ -- Daniel O'Connor "The nice thing about standards is that there are so many of them to choose from." -- Andrew Tanenbaum GPG Fingerprint - 5596 B766 97C0 0E94 4347 295E E593 DC20 7B3F CE8C From owner-freebsd-hackers@freebsd.org Wed Jan 24 04:18:15 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6E347EB9D13 for ; Wed, 24 Jan 2018 04:18:15 +0000 (UTC) (envelope-from yuripv@icloud.com) Received: from pv33p00im-asmtp003.me.com (pv33p00im-asmtp003.me.com [17.142.194.252]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0C98C765D3; Wed, 24 Jan 2018 04:18:15 +0000 (UTC) (envelope-from yuripv@icloud.com) Received: from process-dkim-sign-daemon.pv33p00im-asmtp003.me.com by pv33p00im-asmtp003.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) id <0P3100400KGQ0L00@pv33p00im-asmtp003.me.com>; Wed, 24 Jan 2018 04:17:34 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=04042017; t=1516767453; bh=i+xpt3gW6Ia+48y+ATXC0xbteQMTwuBk9v8ocxFF5/4=; h=Subject:To:From:Message-id:Date:MIME-version:Content-type; b=GwGNkfj6ebyDAi8S7lVTPcFBxG8LrK50ymVUet65Enaa7SDfNZ9hXZD3oAJGiAFpZ vIDtkMNRh2IiOcea9YujaUqfC3AHrcIhK1RYWVLG+y9v3Qzf20c38tAiRDLi5tpHuY MyloyRbed0mnZZLbipathQh5rMjixGgdWuTG7MlvaA6Ncr+cAB3NtNQQm23AEQN3Kr +KeA3Ih89xzl8w0JJc/AZ/sfy2KNarajFavMF3rFxRDCGg1FKXOnW/6ZZyo6lolvkk L67IXVI73L5RYSj1GNJwfy/tHRZg7C5LgiXI3zAdMRQDsRi77Nou15mmXzHZvoecEG eJ1SSmpCZDPmw== Received: from icloud.com ([127.0.0.1]) by pv33p00im-asmtp003.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) with ESMTPSA id <0P31009ULL950F30@pv33p00im-asmtp003.me.com>; Wed, 24 Jan 2018 04:17:33 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-01-24_01:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 clxscore=1015 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1801240053 Subject: Re: libc/regex: r302824 added invalid check breaking collating ranges To: Kyle Evans Cc: FreeBSD Hackers References: From: Yuri Pankov Message-id: <2c9ebf81-c06a-13ed-9cf9-9b42a00c76ee@icloud.com> Date: Wed, 24 Jan 2018 07:17:27 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-version: 1.0 In-reply-to: Content-type: text/plain; charset=utf-8; format=flowed Content-language: en-US Content-transfer-encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jan 2018 04:18:15 -0000 On Tue, Jan 23, 2018 at 01:22:04PM -0600, Kyle Evans wrote: > On Tue, Jan 23, 2018 at 1:10 PM, Yuri Pankov wrote: >> On Tue, Jan 23, 2018 at 08:10:32AM -0600, Kyle Evans wrote: >>> >>> On Mon, Jan 22, 2018 at 11:36 PM, Yuri Pankov wrote: >>>> >>>> On Tue, Jan 23, 2018 at 03:53:19AM +0300, Yuri Pankov wrote: >>>>> >>>>> >>>>> (CCing Kyle as he's working on regex at the moment and not because he >>>>> broke something) >>>>> >>>>> Hi, >>>>> >>>>> r302284 added an invalid check which breaks collating ranges: >>>>> >>>>> -if (table->__collate_load_error) { >>>>> - (void)REQUIRE((uch)start <= (uch)finish, REG_ERANGE); >>>>> +if (table->__collate_load_error || MB_CUR_MAX > 1) { >>>>> + (void)REQUIRE(start <= finish, REG_ERANGE); >>>>> >>>>> The "MB_CUR_MAX > 1" is wrong, we should be doing proper comparison >>>>> according to current locale's collation and not simply comparing the >>>>> wchar_t values. >>>> >>>> >>>> >>>> After re-reading the specification I now see that what looked like a bug >>>> is >>>> actually an implementation choice, though the one that needs to be >>>> documentated. I'll update the man page if anyone is willing to review >>>> (and >>>> commit) the changes. >>> >>> >>> Can you point to the section of specification that indicates this is >>> OK behavior? It doesn't seem desirable, but I see that GNU systems >>> will operate in the same manner that we do now. >> >> >> Here -- >> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html: >> ------------------------------------------------------------------------ >> In the POSIX locale, a range expression represents the set of collating >> elements that fall between two elements in the collation sequence, >> inclusive. In other locales, a range expression has unspecified behavior: >> strictly conforming applications shall not rely on whether the range >> expression is valid, or on the set of collating elements matched. >> ------------------------------------------------------------------------ >> > > Thanks- our current behavior seems reasonable in that context. > >> I've tried to "fix" what I was seeing as well, and yes, everything outside >> of ASCII is ugly, e.g. Cyrillic 'а-я' would match much more than you could >> expect if you are doing lookups based on collation order (capital chars and >> a lot of other symbols). >> >> So what we have currently looks the least evil to me: >> >> - non-collating ASCII lookups for any locale -- looking at the log for >> regcomp.c there was an attempt to "fix" this, but it was reverted as >> a lot of existing code relies on this; >> - non-collating multi-byte locale lookups -- they will work for almost >> all cases, and where they don't, well POSIX says it's undefined :D >> - collating single-byte locale lookups for outside of ASCII range -- >> they make sense as collation order there doesn't seem to mix >> small/caps/other characters together. >> >> What I think we need to do is document this as implementation choice in the >> code and regex(3) "IMPLEMENTATION NOTES" so that another poor soul doesn't >> come trying to fix it as I did :-) > > I agree with your assessment- such a patch would be welcomed, > especially before I go and revise a bunch of this for clarification in > a future libregex world. Actually, it's broken even more than I thought: $ echo 'TEST' | LC_ALL=en_US.ISO8859-1 grep '[a-z]' TEST That's a result of using collation lookup for singlebyte locales. Now I just think that using collations for range expressions in *any* locale is just plain wrong. Another side effect of all this "sometimes non-collating" nonsense is inability to deal with multibyte characters whose corresponding wide character is in 128-255 range -- try adding 'µ' (\302\265, U+00B5) to the pattern and observe a nice ~1GB core from grep after endless loop in regcomp(). This is due to NC (I guess meaning "non-collating") being defined as (CHAR_MAX - CHAR_MIN + 1) which is 256. To sum the above, how about we drop the "non-collating" notion, and just use binary wide character comparison everywhere? From owner-freebsd-hackers@freebsd.org Wed Jan 24 04:22:36 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 81D3FEBB1E0 for ; Wed, 24 Jan 2018 04:22:36 +0000 (UTC) (envelope-from yuripv@icloud.com) Received: from pv33p00im-asmtp002.me.com (pv33p00im-asmtp002.me.com [17.142.194.251]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1F4B076A20; Wed, 24 Jan 2018 04:22:36 +0000 (UTC) (envelope-from yuripv@icloud.com) Received: from process-dkim-sign-daemon.pv33p00im-asmtp002.me.com by pv33p00im-asmtp002.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) id <0P3100400KGPP600@pv33p00im-asmtp002.me.com>; Wed, 24 Jan 2018 04:22:28 +0000 (GMT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=icloud.com; s=04042017; t=1516767748; bh=XcbFr2h4OucEU9ViJwQnInlMTE4TnO5pnZtUzBL9ziU=; h=Subject:From:To:Message-id:Date:MIME-version:Content-type; b=ARxAqlWOw1JBQLCTsgys9LdPdqQtxKJns0r2bmnq4oPawkJFSlB0e19z+hfBYUJvr vsJip8PWKESEv3H/tRS9GcClrwJk12zoBiMw33BfUofh/DnMhVql/GgK9x45ld6aiK DkE42jYkr8gyXQQKBQ9b4X1KcoGGhFhGOZONZXcbg2gglFzE42CuQ4kmXwG8xWkh46 72tYreNSFxptkgSl9VK5gd2/8fNTT/UmjEsJZv0Ykrwy3QdIkKxYpUiyTLC35Ve8f/ 5L2vp4/3Ohm2gKWUf4aU+yhksThaNtM5QFeGptJBqOVfxKqjAvRhuq/cP5lt+8eQf8 hQ3elXDvuG3qg== Received: from icloud.com ([127.0.0.1]) by pv33p00im-asmtp002.me.com (Oracle Communications Messaging Server 8.0.1.2.20170607 64bit (built Jun 7 2017)) with ESMTPSA id <0P31007J2LH95I10@pv33p00im-asmtp002.me.com>; Wed, 24 Jan 2018 04:22:24 +0000 (GMT) X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-01-24_01:,, signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 spamscore=0 clxscore=1015 suspectscore=0 malwarescore=0 phishscore=0 adultscore=0 bulkscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1707230000 definitions=main-1801240054 Subject: Re: libc/regex: r302824 added invalid check breaking collating ranges From: Yuri Pankov To: Kyle Evans Cc: FreeBSD Hackers References: <2c9ebf81-c06a-13ed-9cf9-9b42a00c76ee@icloud.com> Message-id: <6c84d7ad-26bd-ec67-143b-b6e41d6018e6@icloud.com> Date: Wed, 24 Jan 2018 07:22:20 +0300 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-version: 1.0 In-reply-to: <2c9ebf81-c06a-13ed-9cf9-9b42a00c76ee@icloud.com> Content-type: text/plain; charset=utf-8; format=flowed Content-language: en-US Content-transfer-encoding: 8bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 24 Jan 2018 04:22:36 -0000 On Wed, Jan 24, 2018 at 07:17:27AM +0300, Yuri Pankov wrote: > On Tue, Jan 23, 2018 at 01:22:04PM -0600, Kyle Evans wrote: >> On Tue, Jan 23, 2018 at 1:10 PM, Yuri Pankov wrote: >>> On Tue, Jan 23, 2018 at 08:10:32AM -0600, Kyle Evans wrote: >>>> >>>> On Mon, Jan 22, 2018 at 11:36 PM, Yuri Pankov wrote: >>>>> >>>>> On Tue, Jan 23, 2018 at 03:53:19AM +0300, Yuri Pankov wrote: >>>>>> >>>>>> >>>>>> (CCing Kyle as he's working on regex at the moment and not because he >>>>>> broke something) >>>>>> >>>>>> Hi, >>>>>> >>>>>> r302284 added an invalid check which breaks collating ranges: >>>>>> >>>>>> -if (table->__collate_load_error) { >>>>>> - (void)REQUIRE((uch)start <= (uch)finish, REG_ERANGE); >>>>>> +if (table->__collate_load_error || MB_CUR_MAX > 1) { >>>>>> + (void)REQUIRE(start <= finish, REG_ERANGE); >>>>>> >>>>>> The "MB_CUR_MAX > 1" is wrong, we should be doing proper comparison >>>>>> according to current locale's collation and not simply comparing the >>>>>> wchar_t values. >>>>> >>>>> >>>>> >>>>> After re-reading the specification I now see that what looked like a bug >>>>> is >>>>> actually an implementation choice, though the one that needs to be >>>>> documentated. I'll update the man page if anyone is willing to review >>>>> (and >>>>> commit) the changes. >>>> >>>> >>>> Can you point to the section of specification that indicates this is >>>> OK behavior? It doesn't seem desirable, but I see that GNU systems >>>> will operate in the same manner that we do now. >>> >>> >>> Here -- >>> http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html: >>> ------------------------------------------------------------------------ >>> In the POSIX locale, a range expression represents the set of collating >>> elements that fall between two elements in the collation sequence, >>> inclusive. In other locales, a range expression has unspecified behavior: >>> strictly conforming applications shall not rely on whether the range >>> expression is valid, or on the set of collating elements matched. >>> ------------------------------------------------------------------------ >>> >> >> Thanks- our current behavior seems reasonable in that context. >> >>> I've tried to "fix" what I was seeing as well, and yes, everything outside >>> of ASCII is ugly, e.g. Cyrillic 'а-я' would match much more than you could >>> expect if you are doing lookups based on collation order (capital chars and >>> a lot of other symbols). >>> >>> So what we have currently looks the least evil to me: >>> >>> - non-collating ASCII lookups for any locale -- looking at the log for >>> regcomp.c there was an attempt to "fix" this, but it was reverted as >>> a lot of existing code relies on this; >>> - non-collating multi-byte locale lookups -- they will work for almost >>> all cases, and where they don't, well POSIX says it's undefined :D >>> - collating single-byte locale lookups for outside of ASCII range -- >>> they make sense as collation order there doesn't seem to mix >>> small/caps/other characters together. >>> >>> What I think we need to do is document this as implementation choice in the >>> code and regex(3) "IMPLEMENTATION NOTES" so that another poor soul doesn't >>> come trying to fix it as I did :-) >> >> I agree with your assessment- such a patch would be welcomed, >> especially before I go and revise a bunch of this for clarification in >> a future libregex world. > > Actually, it's broken even more than I thought: > > $ echo 'TEST' | LC_ALL=en_US.ISO8859-1 grep '[a-z]' > TEST > > That's a result of using collation lookup for singlebyte locales. Now I > just think that using collations for range expressions in *any* locale > is just plain wrong. > > Another side effect of all this "sometimes non-collating" nonsense is > inability to deal with multibyte characters whose corresponding wide > character is in 128-255 range -- try adding 'µ' (\302\265, U+00B5) to > the pattern and observe a nice ~1GB core from grep after endless loop in > regcomp(). This is due to NC (I guess meaning "non-collating") being > defined as (CHAR_MAX - CHAR_MIN + 1) which is 256. Oh, and the lookup should be case-insensitive above to reproduce the issue. > To sum the above, how about we drop the "non-collating" notion, and just > use binary wide character comparison everywhere? From owner-freebsd-hackers@freebsd.org Thu Jan 25 16:52:38 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 17A4DEBEC82 for ; Thu, 25 Jan 2018 16:52:38 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A5735863A0 for ; Thu, 25 Jan 2018 16:52:37 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: by mailman.ysv.freebsd.org (Postfix) id 5FE42EBEC7D; Thu, 25 Jan 2018 16:52:37 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4C1ADEBEC7C for ; Thu, 25 Jan 2018 16:52:37 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from mailhost.dlr.de (mailhost.dlr.de [129.247.252.33]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mailhost.dlr.de", Issuer "DLR CA - G02" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C0F878639F for ; Thu, 25 Jan 2018 16:52:36 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from DLREXHUB01.intra.dlr.de (172.21.152.130) by mailhost.dlr.de (172.21.163.101) with Microsoft SMTP Server (TLS) id 14.3.361.1; Thu, 25 Jan 2018 17:52:23 +0100 Received: from KNOP-BEAGLE.kn.op.dlr.de (129.247.178.136) by smtp.dlr.de (172.21.152.151) with Microsoft SMTP Server (TLS) id 14.3.361.1; Thu, 25 Jan 2018 17:52:28 +0100 Date: Thu, 25 Jan 2018 17:52:57 +0100 From: Hartmut Brandt X-X-Sender: brandt_h@KNOP-BEAGLE.kn.op.dlr.de To: Subject: allocating large piece of contiguous kernel memory Message-ID: User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; format=flowed; charset="US-ASCII" X-TM-AS-Product-Ver: SMEX-11.0.0.4283-8.100.1062-23620.000 X-TM-AS-Result: No--0.126000-5.000000-31 X-TM-AS-MatchedID: 708712-708060-705262-708218-861157-303277-705388-709584-1 48004-148050-10018-41000-42000-42003-63 X-TM-AS-User-Approved-Sender: No X-TM-AS-User-Blocked-Sender: No X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jan 2018 16:52:38 -0000 Hi all, for a device driver communicating with an FPGA I would like to allocate an 8GByte piece or two 4GByte pieces of contiguous memory in the device driver. The machine is amd64 and has 256GByte. The maximum I can allocate is 512MByte. If I try more (in one piece or in two pieces) contigfree() crashes in pmap_resident_count_dec() with an "pmap resident count underflow". Is this something which is not supposed to work? Or is there something tunable or is this a bug? Regards, harti From owner-freebsd-hackers@freebsd.org Thu Jan 25 17:21:41 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4C00EEC0C09 for ; Thu, 25 Jan 2018 17:21:41 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id D99A087A1A for ; Thu, 25 Jan 2018 17:21:40 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 92322EC0C08; Thu, 25 Jan 2018 17:21:40 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E37BEC0C06 for ; Thu, 25 Jan 2018 17:21:40 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E7DE187A16 for ; Thu, 25 Jan 2018 17:21:39 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w0PHLGY8042938 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 25 Jan 2018 19:21:19 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w0PHLGY8042938 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w0PHLGfe042937; Thu, 25 Jan 2018 19:21:16 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Thu, 25 Jan 2018 19:21:16 +0200 From: Konstantin Belousov To: Hartmut Brandt Cc: hackers@freebsd.org Subject: Re: allocating large piece of contiguous kernel memory Message-ID: <20180125172116.GS55707@kib.kiev.ua> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.2 (2017-12-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 25 Jan 2018 17:21:41 -0000 On Thu, Jan 25, 2018 at 05:52:57PM +0100, Hartmut Brandt wrote: > Hi all, > > for a device driver communicating with an FPGA I would like to allocate > an 8GByte piece or two 4GByte pieces of contiguous memory in the device > driver. The machine is amd64 and has 256GByte. > > The maximum I can allocate is 512MByte. If I try more (in one piece or in > two pieces) contigfree() crashes in pmap_resident_count_dec() with an > "pmap resident count underflow". > > Is this something which is not supposed to work? Or is there something > tunable or is this a bug? I suspect a bug. Print out the value of kernel_pmap->pm_stats.resident_count before and after contigmalloc. Also interesting are the values of the resident count and decrement at the panic time (but they are already printed by the panic, you did not shown them). From owner-freebsd-hackers@freebsd.org Fri Jan 26 09:03:17 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2EC91ECB201 for ; Fri, 26 Jan 2018 09:03:17 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id BB3CD6A5E5 for ; Fri, 26 Jan 2018 09:03:16 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: by mailman.ysv.freebsd.org (Postfix) id 75484ECB200; Fri, 26 Jan 2018 09:03:16 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5085EECB1FF for ; Fri, 26 Jan 2018 09:03:16 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from mailhost.dlr.de (mailhost.dlr.de [129.247.252.32]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (Client CN "mailhost.dlr.de", Issuer "DLR CA - G02" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id C6BAA6A5E4 for ; Fri, 26 Jan 2018 09:03:15 +0000 (UTC) (envelope-from Hartmut.Brandt@dlr.de) Received: from DLREXHUB01.intra.dlr.de (172.21.152.130) by mailhost.dlr.de (172.21.163.100) with Microsoft SMTP Server (TLS) id 14.3.361.1; Fri, 26 Jan 2018 10:03:01 +0100 Received: from KNOP-BEAGLE.kn.op.dlr.de (129.247.178.136) by smtp.dlr.de (172.21.152.151) with Microsoft SMTP Server (TLS) id 14.3.361.1; Fri, 26 Jan 2018 10:03:06 +0100 Date: Fri, 26 Jan 2018 10:05:14 +0100 From: Hartmut Brandt X-X-Sender: brandt_h@KNOP-BEAGLE.kn.op.dlr.de To: Konstantin Belousov CC: Subject: Re: allocating large piece of contiguous kernel memory In-Reply-To: <20180125172116.GS55707@kib.kiev.ua> Message-ID: References: <20180125172116.GS55707@kib.kiev.ua> User-Agent: Alpine 2.20 (BSF 67 2015-01-07) MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" X-TM-AS-Product-Ver: SMEX-11.0.0.4283-8.100.1062-23620.005 X-TM-AS-Result: No--5.400600-5.000000-31 X-TM-AS-MatchedID: 150567-700075-708712-708060-705262-708218-861157-705388-7 09584-703523-707760-702807-186027-121202-188199-702688-106420-105630-704421 -700918-704318-708497-701428-701465-702098-701461-701236-703788-148004-1480 50-29997-42000-42003-63 X-TM-AS-User-Approved-Sender: No X-TM-AS-User-Blocked-Sender: No X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Jan 2018 09:03:17 -0000 Hi, On Thu, 25 Jan 2018, Konstantin Belousov wrote: KB>On Thu, Jan 25, 2018 at 05:52:57PM +0100, Hartmut Brandt wrote: KB>> Hi all, KB>> KB>> for a device driver communicating with an FPGA I would like to allocate KB>> an 8GByte piece or two 4GByte pieces of contiguous memory in the device KB>> driver. The machine is amd64 and has 256GByte. KB>> KB>> The maximum I can allocate is 512MByte. If I try more (in one piece or in KB>> two pieces) contigfree() crashes in pmap_resident_count_dec() with an KB>> "pmap resident count underflow". KB>> KB>> Is this something which is not supposed to work? Or is there something KB>> tunable or is this a bug? KB> KB>I suspect a bug. Print out the value of KB>kernel_pmap->pm_stats.resident_count before and after contigmalloc. Also KB>interesting are the values of the resident count and decrement at the KB>panic time (but they are already printed by the panic, you did not shown KB>them). This is the panic: knot_attach: attaching kernel_pmap->pm_stats.resident_count=57322 cmem=0xfffffe0101000000 kernel_pmap->pm_stats.resident_count=57712 kernel_pmap->pm_stats.resident_count=57712 panic: pmap 0xffffffff80e40188 resident count underflow 368 512 cpuid = 3 time = 1516956458 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0100d0d1c0 vpanic() at vpanic+0x19c/frame 0xfffffe0100d0d240 kassert_panic() at kassert_panic+0x126/frame 0xfffffe0100d0d2b0 pmap_remove_pde() at pmap_remove_pde+0x645/frame 0xfffffe0100d0d340 pmap_remove() at pmap_remove+0x484/frame 0xfffffe0100d0d3c0 kmem_unback() at kmem_unback+0x3a/frame 0xfffffe0100d0d400 kmem_free() at kmem_free+0x3d/frame 0xfffffe0100d0d430 contigfree() at contigfree+0x27/frame 0xfffffe0100d0d460 knot_attach() at knot_attach+0xec/frame 0xfffffe0100d0d4c0 device_attach() at device_attach+0x3f7/frame 0xfffffe0100d0d510 pci_driver_added() at pci_driver_added+0xe9/frame 0xfffffe0100d0d550 devclass_driver_added() at devclass_driver_added+0x7d/frame 0xfffffe0100d0d590 devclass_add_driver() at devclass_add_driver+0x144/frame 0xfffffe0100d0d5d0 module_register_init() at module_register_init+0xc0/frame 0xfffffe0100d0d600 linker_load_module() at linker_load_module+0xb78/frame 0xfffffe0100d0d910 kern_kldload() at kern_kldload+0xa9/frame 0xfffffe0100d0d950 sys_kldload() at sys_kldload+0x5b/frame 0xfffffe0100d0d980 amd64_syscall() at amd64_syscall+0x271/frame 0xfffffe0100d0dab0 Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0100d0dab0 --- syscall (304, FreeBSD ELF64, sys_kldload), rip = 0x80087228a, rsp = 0x7fffffffd508, rbp = 0x7fffffffda80 --- KDB: enter: panic [ thread pid 1150 tid 100146 ] Stopped at kdb_enter+0x3b: movq $0,kdb_why db> The first printf() is just before contigmalloc(), then it prints the returned pointer. The next to prints() are just before the contigfree(). Doesn't the change of the count by 390 look strange when allocating 8GByte? Regards, harti From owner-freebsd-hackers@freebsd.org Sat Jan 27 13:50:39 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D87A0EC27A8 for ; Sat, 27 Jan 2018 13:50:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5D22A69A9C for ; Sat, 27 Jan 2018 13:50:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 1DC67EC27A6; Sat, 27 Jan 2018 13:50:38 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE08CEC27A5 for ; Sat, 27 Jan 2018 13:50:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 727D469A98 for ; Sat, 27 Jan 2018 13:50:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w0RDoDcs038530 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 27 Jan 2018 15:50:16 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w0RDoDcs038530 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w0RDoDKA038528; Sat, 27 Jan 2018 15:50:13 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 27 Jan 2018 15:50:13 +0200 From: Konstantin Belousov To: Hartmut Brandt Cc: hackers@freebsd.org Subject: Re: allocating large piece of contiguous kernel memory Message-ID: <20180127135013.GA55707@kib.kiev.ua> References: <20180125172116.GS55707@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.2 (2017-12-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jan 2018 13:50:39 -0000 On Fri, Jan 26, 2018 at 10:05:14AM +0100, Hartmut Brandt wrote: > Hi, > > On Thu, 25 Jan 2018, Konstantin Belousov wrote: > > KB>On Thu, Jan 25, 2018 at 05:52:57PM +0100, Hartmut Brandt wrote: > KB>> Hi all, > KB>> > KB>> for a device driver communicating with an FPGA I would like to allocate > KB>> an 8GByte piece or two 4GByte pieces of contiguous memory in the device > KB>> driver. The machine is amd64 and has 256GByte. > KB>> > KB>> The maximum I can allocate is 512MByte. If I try more (in one piece or in > KB>> two pieces) contigfree() crashes in pmap_resident_count_dec() with an > KB>> "pmap resident count underflow". > KB>> > KB>> Is this something which is not supposed to work? Or is there something > KB>> tunable or is this a bug? > KB> > KB>I suspect a bug. Print out the value of > KB>kernel_pmap->pm_stats.resident_count before and after contigmalloc. Also > KB>interesting are the values of the resident count and decrement at the > KB>panic time (but they are already printed by the panic, you did not shown > KB>them). > > This is the panic: > > knot_attach: attaching > kernel_pmap->pm_stats.resident_count=57322 > cmem=0xfffffe0101000000 > kernel_pmap->pm_stats.resident_count=57712 > kernel_pmap->pm_stats.resident_count=57712 > panic: pmap 0xffffffff80e40188 resident count underflow 368 512 > cpuid = 3 > time = 1516956458 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0100d0d1c0 > vpanic() at vpanic+0x19c/frame 0xfffffe0100d0d240 > kassert_panic() at kassert_panic+0x126/frame 0xfffffe0100d0d2b0 > pmap_remove_pde() at pmap_remove_pde+0x645/frame 0xfffffe0100d0d340 > pmap_remove() at pmap_remove+0x484/frame 0xfffffe0100d0d3c0 > kmem_unback() at kmem_unback+0x3a/frame 0xfffffe0100d0d400 > kmem_free() at kmem_free+0x3d/frame 0xfffffe0100d0d430 > contigfree() at contigfree+0x27/frame 0xfffffe0100d0d460 > knot_attach() at knot_attach+0xec/frame 0xfffffe0100d0d4c0 > device_attach() at device_attach+0x3f7/frame 0xfffffe0100d0d510 > pci_driver_added() at pci_driver_added+0xe9/frame 0xfffffe0100d0d550 > devclass_driver_added() at devclass_driver_added+0x7d/frame 0xfffffe0100d0d590 > devclass_add_driver() at devclass_add_driver+0x144/frame 0xfffffe0100d0d5d0 > module_register_init() at module_register_init+0xc0/frame 0xfffffe0100d0d600 > linker_load_module() at linker_load_module+0xb78/frame 0xfffffe0100d0d910 > kern_kldload() at kern_kldload+0xa9/frame 0xfffffe0100d0d950 > sys_kldload() at sys_kldload+0x5b/frame 0xfffffe0100d0d980 > amd64_syscall() at amd64_syscall+0x271/frame 0xfffffe0100d0dab0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0100d0dab0 > --- syscall (304, FreeBSD ELF64, sys_kldload), rip = 0x80087228a, rsp = > 0x7fffffffd508, rbp = 0x7fffffffda80 --- > KDB: enter: panic > [ thread pid 1150 tid 100146 ] > Stopped at kdb_enter+0x3b: movq $0,kdb_why > db> > > The first printf() is just before contigmalloc(), then it prints the > returned pointer. The next to prints() are just before the contigfree(). > > Doesn't the change of the count by 390 look strange when allocating > 8GByte? I cannot reproduce your issue. I suspect this is something local to your system. With the patch below, if I do # sysctl debug.a=1 I see # sysctl debug.a=1 debug.a: before 31310 0after alloc 2128462 after free 31310 -> 0 The numbers are exactly as what I expect. diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c index 12c5b695e23..3c707444169 100644 --- a/sys/amd64/amd64/pmap.c +++ b/sys/amd64/amd64/pmap.c @@ -8040,3 +8040,26 @@ DB_SHOW_COMMAND(phys2dmap, pmap_phys2dmap) } } #endif + +static int +pmap_contigalloc(SYSCTL_HANDLER_ARGS) +{ + void *ptr; + size_t sz; + int error, i; + + i = 0; + error = sysctl_handle_int(oidp, &i, 0, req); + if (error != 0 || req->newptr == NULL) + return (error); + sz = 8UL * 1024 * 1024 * 1024; + printf("before %ld\n", kernel_pmap->pm_stats.resident_count); + ptr = contigmalloc(sz, M_TEMP, M_WAITOK, 0, ~0UL, 0, 0); + printf("after alloc %ld\n", kernel_pmap->pm_stats.resident_count); + contigfree(ptr, sz, M_TEMP); + printf("after free %ld\n", kernel_pmap->pm_stats.resident_count); + return (0); +} +SYSCTL_PROC(_debug, OID_AUTO, a, CTLTYPE_INT | CTLFLAG_RW | + CTLFLAG_MPSAFE, NULL, 0, pmap_contigalloc, "I", + ""); From owner-freebsd-hackers@freebsd.org Sat Jan 27 16:55:32 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1959EECB5E3 for ; Sat, 27 Jan 2018 16:55:32 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id A847B703F9 for ; Sat, 27 Jan 2018 16:55:31 +0000 (UTC) (envelope-from hps@selasky.org) Received: by mailman.ysv.freebsd.org (Postfix) id 64EF0ECB5E2; Sat, 27 Jan 2018 16:55:31 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 27DDCECB5E1 for ; Sat, 27 Jan 2018 16:55:31 +0000 (UTC) (envelope-from hps@selasky.org) Received: from mail.turbocat.net (turbocat.net [88.99.82.50]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id ADE27703F8 for ; Sat, 27 Jan 2018 16:55:30 +0000 (UTC) (envelope-from hps@selasky.org) Received: from hps2016.home.selasky.org (unknown [62.141.128.70]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mail.turbocat.net (Postfix) with ESMTPSA id 59704260115; Sat, 27 Jan 2018 17:55:20 +0100 (CET) Subject: Re: allocating large piece of contiguous kernel memory To: Konstantin Belousov , Hartmut Brandt Cc: hackers@freebsd.org References: <20180125172116.GS55707@kib.kiev.ua> <20180127135013.GA55707@kib.kiev.ua> From: Hans Petter Selasky Message-ID: Date: Sat, 27 Jan 2018 17:52:28 +0100 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: <20180127135013.GA55707@kib.kiev.ua> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jan 2018 16:55:32 -0000 On 01/27/18 14:50, Konstantin Belousov wrote: > On Fri, Jan 26, 2018 at 10:05:14AM +0100, Hartmut Brandt wrote: >> Hi, >> >> On Thu, 25 Jan 2018, Konstantin Belousov wrote: >> >> KB>On Thu, Jan 25, 2018 at 05:52:57PM +0100, Hartmut Brandt wrote: >> KB>> Hi all, >> KB>> >> KB>> for a device driver communicating with an FPGA I would like to allocate >> KB>> an 8GByte piece or two 4GByte pieces of contiguous memory in the device >> KB>> driver. The machine is amd64 and has 256GByte. >> KB>> >> KB>> The maximum I can allocate is 512MByte. If I try more (in one piece or in >> KB>> two pieces) contigfree() crashes in pmap_resident_count_dec() with an >> KB>> "pmap resident count underflow". >> KB>> >> KB>> Is this something which is not supposed to work? Or is there something >> KB>> tunable or is this a bug? >> KB> >> KB>I suspect a bug. Print out the value of >> KB>kernel_pmap->pm_stats.resident_count before and after contigmalloc. Also >> KB>interesting are the values of the resident count and decrement at the >> KB>panic time (but they are already printed by the panic, you did not shown >> KB>them). >> >> This is the panic: >> >> knot_attach: attaching >> kernel_pmap->pm_stats.resident_count=57322 >> cmem=0xfffffe0101000000 >> kernel_pmap->pm_stats.resident_count=57712 >> kernel_pmap->pm_stats.resident_count=57712 >> panic: pmap 0xffffffff80e40188 resident count underflow 368 512 >> cpuid = 3 >> time = 1516956458 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0100d0d1c0 >> vpanic() at vpanic+0x19c/frame 0xfffffe0100d0d240 >> kassert_panic() at kassert_panic+0x126/frame 0xfffffe0100d0d2b0 >> pmap_remove_pde() at pmap_remove_pde+0x645/frame 0xfffffe0100d0d340 >> pmap_remove() at pmap_remove+0x484/frame 0xfffffe0100d0d3c0 >> kmem_unback() at kmem_unback+0x3a/frame 0xfffffe0100d0d400 >> kmem_free() at kmem_free+0x3d/frame 0xfffffe0100d0d430 >> contigfree() at contigfree+0x27/frame 0xfffffe0100d0d460 >> knot_attach() at knot_attach+0xec/frame 0xfffffe0100d0d4c0 >> device_attach() at device_attach+0x3f7/frame 0xfffffe0100d0d510 >> pci_driver_added() at pci_driver_added+0xe9/frame 0xfffffe0100d0d550 >> devclass_driver_added() at devclass_driver_added+0x7d/frame 0xfffffe0100d0d590 >> devclass_add_driver() at devclass_add_driver+0x144/frame 0xfffffe0100d0d5d0 >> module_register_init() at module_register_init+0xc0/frame 0xfffffe0100d0d600 >> linker_load_module() at linker_load_module+0xb78/frame 0xfffffe0100d0d910 >> kern_kldload() at kern_kldload+0xa9/frame 0xfffffe0100d0d950 >> sys_kldload() at sys_kldload+0x5b/frame 0xfffffe0100d0d980 >> amd64_syscall() at amd64_syscall+0x271/frame 0xfffffe0100d0dab0 >> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0100d0dab0 >> --- syscall (304, FreeBSD ELF64, sys_kldload), rip = 0x80087228a, rsp = >> 0x7fffffffd508, rbp = 0x7fffffffda80 --- >> KDB: enter: panic >> [ thread pid 1150 tid 100146 ] >> Stopped at kdb_enter+0x3b: movq $0,kdb_why >> db> >> >> The first printf() is just before contigmalloc(), then it prints the >> returned pointer. The next to prints() are just before the contigfree(). >> >> Doesn't the change of the count by 390 look strange when allocating >> 8GByte? > I cannot reproduce your issue. I suspect this is something local to your > system. With the patch below, if I do > # sysctl debug.a=1 > I see > # sysctl debug.a=1 > debug.a: before 31310 > 0after alloc 2128462 > after free 31310 > -> 0 > > The numbers are exactly as what I expect. > > diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c > index 12c5b695e23..3c707444169 100644 > --- a/sys/amd64/amd64/pmap.c > +++ b/sys/amd64/amd64/pmap.c > @@ -8040,3 +8040,26 @@ DB_SHOW_COMMAND(phys2dmap, pmap_phys2dmap) > } > } > #endif > + > +static int > +pmap_contigalloc(SYSCTL_HANDLER_ARGS) > +{ > + void *ptr; > + size_t sz; > + int error, i; > + > + i = 0; > + error = sysctl_handle_int(oidp, &i, 0, req); > + if (error != 0 || req->newptr == NULL) > + return (error); > + sz = 8UL * 1024 * 1024 * 1024; > + printf("before %ld\n", kernel_pmap->pm_stats.resident_count); > + ptr = contigmalloc(sz, M_TEMP, M_WAITOK, 0, ~0UL, 0, 0); > + printf("after alloc %ld\n", kernel_pmap->pm_stats.resident_count); > + contigfree(ptr, sz, M_TEMP); > + printf("after free %ld\n", kernel_pmap->pm_stats.resident_count); > + return (0); > +} > +SYSCTL_PROC(_debug, OID_AUTO, a, CTLTYPE_INT | CTLFLAG_RW | > + CTLFLAG_MPSAFE, NULL, 0, pmap_contigalloc, "I", > + ""); Hi, What is the earliest point in the SYSINIT sequence which contigmalloc() can be called? Is contigmalloc() being called too early for this big allocations? --HPS From owner-freebsd-hackers@freebsd.org Sat Jan 27 17:34:49 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 10822ECD4FC for ; Sat, 27 Jan 2018 17:34:49 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8806071CAA for ; Sat, 27 Jan 2018 17:34:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 4CB4BECD4FA; Sat, 27 Jan 2018 17:34:48 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2582AECD4F8 for ; Sat, 27 Jan 2018 17:34:48 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9B4BA71CA9 for ; Sat, 27 Jan 2018 17:34:47 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w0RHYJcS089678 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 27 Jan 2018 19:34:23 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w0RHYJcS089678 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w0RHYJiY089677; Sat, 27 Jan 2018 19:34:19 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 27 Jan 2018 19:34:19 +0200 From: Konstantin Belousov To: Hans Petter Selasky Cc: Hartmut Brandt , hackers@freebsd.org Subject: Re: allocating large piece of contiguous kernel memory Message-ID: <20180127173419.GC55707@kib.kiev.ua> References: <20180125172116.GS55707@kib.kiev.ua> <20180127135013.GA55707@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.2 (2017-12-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jan 2018 17:34:49 -0000 On Sat, Jan 27, 2018 at 05:52:28PM +0100, Hans Petter Selasky wrote: > On 01/27/18 14:50, Konstantin Belousov wrote: > > On Fri, Jan 26, 2018 at 10:05:14AM +0100, Hartmut Brandt wrote: > >> Hi, > >> > >> On Thu, 25 Jan 2018, Konstantin Belousov wrote: > >> > >> KB>On Thu, Jan 25, 2018 at 05:52:57PM +0100, Hartmut Brandt wrote: > >> KB>> Hi all, > >> KB>> > >> KB>> for a device driver communicating with an FPGA I would like to allocate > >> KB>> an 8GByte piece or two 4GByte pieces of contiguous memory in the device > >> KB>> driver. The machine is amd64 and has 256GByte. > >> KB>> > >> KB>> The maximum I can allocate is 512MByte. If I try more (in one piece or in > >> KB>> two pieces) contigfree() crashes in pmap_resident_count_dec() with an > >> KB>> "pmap resident count underflow". > >> KB>> > >> KB>> Is this something which is not supposed to work? Or is there something > >> KB>> tunable or is this a bug? > >> KB> > >> KB>I suspect a bug. Print out the value of > >> KB>kernel_pmap->pm_stats.resident_count before and after contigmalloc. Also > >> KB>interesting are the values of the resident count and decrement at the > >> KB>panic time (but they are already printed by the panic, you did not shown > >> KB>them). > >> > >> This is the panic: > >> > >> knot_attach: attaching > >> kernel_pmap->pm_stats.resident_count=57322 > >> cmem=0xfffffe0101000000 > >> kernel_pmap->pm_stats.resident_count=57712 > >> kernel_pmap->pm_stats.resident_count=57712 > >> panic: pmap 0xffffffff80e40188 resident count underflow 368 512 > >> cpuid = 3 > >> time = 1516956458 > >> KDB: stack backtrace: > >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0100d0d1c0 > >> vpanic() at vpanic+0x19c/frame 0xfffffe0100d0d240 > >> kassert_panic() at kassert_panic+0x126/frame 0xfffffe0100d0d2b0 > >> pmap_remove_pde() at pmap_remove_pde+0x645/frame 0xfffffe0100d0d340 > >> pmap_remove() at pmap_remove+0x484/frame 0xfffffe0100d0d3c0 > >> kmem_unback() at kmem_unback+0x3a/frame 0xfffffe0100d0d400 > >> kmem_free() at kmem_free+0x3d/frame 0xfffffe0100d0d430 > >> contigfree() at contigfree+0x27/frame 0xfffffe0100d0d460 > >> knot_attach() at knot_attach+0xec/frame 0xfffffe0100d0d4c0 > >> device_attach() at device_attach+0x3f7/frame 0xfffffe0100d0d510 > >> pci_driver_added() at pci_driver_added+0xe9/frame 0xfffffe0100d0d550 > >> devclass_driver_added() at devclass_driver_added+0x7d/frame 0xfffffe0100d0d590 > >> devclass_add_driver() at devclass_add_driver+0x144/frame 0xfffffe0100d0d5d0 > >> module_register_init() at module_register_init+0xc0/frame 0xfffffe0100d0d600 > >> linker_load_module() at linker_load_module+0xb78/frame 0xfffffe0100d0d910 > >> kern_kldload() at kern_kldload+0xa9/frame 0xfffffe0100d0d950 > >> sys_kldload() at sys_kldload+0x5b/frame 0xfffffe0100d0d980 > >> amd64_syscall() at amd64_syscall+0x271/frame 0xfffffe0100d0dab0 > >> Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0100d0dab0 > >> --- syscall (304, FreeBSD ELF64, sys_kldload), rip = 0x80087228a, rsp = > >> 0x7fffffffd508, rbp = 0x7fffffffda80 --- > >> KDB: enter: panic > >> [ thread pid 1150 tid 100146 ] > >> Stopped at kdb_enter+0x3b: movq $0,kdb_why > >> db> > >> > >> The first printf() is just before contigmalloc(), then it prints the > >> returned pointer. The next to prints() are just before the contigfree(). > >> > >> Doesn't the change of the count by 390 look strange when allocating > >> 8GByte? > > I cannot reproduce your issue. I suspect this is something local to your > > system. With the patch below, if I do > > # sysctl debug.a=1 > > I see > > # sysctl debug.a=1 > > debug.a: before 31310 > > 0after alloc 2128462 > > after free 31310 > > -> 0 > > > > The numbers are exactly as what I expect. > > > > diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c > > index 12c5b695e23..3c707444169 100644 > > --- a/sys/amd64/amd64/pmap.c > > +++ b/sys/amd64/amd64/pmap.c > > @@ -8040,3 +8040,26 @@ DB_SHOW_COMMAND(phys2dmap, pmap_phys2dmap) > > } > > } > > #endif > > + > > +static int > > +pmap_contigalloc(SYSCTL_HANDLER_ARGS) > > +{ > > + void *ptr; > > + size_t sz; > > + int error, i; > > + > > + i = 0; > > + error = sysctl_handle_int(oidp, &i, 0, req); > > + if (error != 0 || req->newptr == NULL) > > + return (error); > > + sz = 8UL * 1024 * 1024 * 1024; > > + printf("before %ld\n", kernel_pmap->pm_stats.resident_count); > > + ptr = contigmalloc(sz, M_TEMP, M_WAITOK, 0, ~0UL, 0, 0); > > + printf("after alloc %ld\n", kernel_pmap->pm_stats.resident_count); > > + contigfree(ptr, sz, M_TEMP); > > + printf("after free %ld\n", kernel_pmap->pm_stats.resident_count); > > + return (0); > > +} > > +SYSCTL_PROC(_debug, OID_AUTO, a, CTLTYPE_INT | CTLFLAG_RW | > > + CTLFLAG_MPSAFE, NULL, 0, pmap_contigalloc, "I", > > + ""); > > Hi, > > What is the earliest point in the SYSINIT sequence which contigmalloc() > can be called? I think it is callable at the same moment as the normal malloc(9). > Is contigmalloc() being called too early for this big > allocations? I have no idea, but I doubt that it is the problem reported. Long time ago there existed type errors in the *malloc function arguments, which caused overflow for large allocations. I am not sure what version of the system was used, hopefully not too old. From owner-freebsd-hackers@freebsd.org Sat Jan 27 17:42:46 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 065D4ECDBAC for ; Sat, 27 Jan 2018 17:42:46 +0000 (UTC) (envelope-from fernando.apesteguia@gmail.com) Received: from mail-lf0-x236.google.com (mail-lf0-x236.google.com [IPv6:2a00:1450:4010:c07::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 65FAC72430 for ; Sat, 27 Jan 2018 17:42:45 +0000 (UTC) (envelope-from fernando.apesteguia@gmail.com) Received: by mail-lf0-x236.google.com with SMTP id h92so4457029lfi.7 for ; Sat, 27 Jan 2018 09:42:45 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=onm4+dqe2pp8jVcRdfwRSN66cBVBvO3WsRNSRtxNXeA=; b=VLo7Ptr/MdoHFdSshbP1RHM1xeYARbcqxtD9oQN979SJlxEvgzpjaMUP0IjUeKu57g L734pKEPYEbv8EsFMWCCnocaAfddRltuqxYzk4EHkHloPDlWjGgfMz28p/fCBHVGsW+w zKC9mhGOSUAr43dB7YDFM9zEErj6GXrFSZIPYZIRCn3iiAJ+FZzHE/phdKOTN6thlJAY u5+INUQcL7r0hVAycPEB6EJDukvQWwOzXmAtYPSR+tVYtrAywRhM2EwyzIEB/KAM/Eij uZWYbMHC02QGOwiewqzv1+G5Hd16cGFPi9Tneeh+d8vaRBnckU559IRhxtMyvr7gz7ps HWYQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=onm4+dqe2pp8jVcRdfwRSN66cBVBvO3WsRNSRtxNXeA=; b=VkZwJ2v8yZi9a7IQ9itQmfpLMt30MFglD0lqny7u97Z/ztXNqEDitauVRq6uvlATxG PRbTzdmgMGjQaeU/XRbhh0SbbnX/HZ2ZfNE+E+XGh0UAtLVapxFUftcJGnnf2AXiAx8G J5Mr4/oK8dWT4fYqDR7uYTRHklcIrgx+xBNyo0YxZY7G2tCFvtxVC01Tafa6xkVbxPEt TTEc46w0ADspgdBwtD2rNe/c21v/CGBMSKe77A+WjdaRP2DI1KVvv7bh9gCp/ZKGwS/6 zlsjvnkk0tS3ebevGmutlxkPc7Q/d1V9XNjkWDHYcpTLYIBD2x5r3m68Mb56Szc5jPQE JKzQ== X-Gm-Message-State: AKwxytf5McUybs6StxllITpQpYyb/bhSZq7xvCmLdEM86mQtQPq8i/hl p0CnOaIWzmod+pHsVMqE4DQDK+vcpaFJ+V2xQ4VOTQ== X-Google-Smtp-Source: AH8x224s7RUWniBbjW3npx0mpcpXwKlLgtlpDOkkmSSnPPa5+P0CcLce9j+ygnudQQt4T9ASAHLKIvWQI+VmiOi/DbM= X-Received: by 10.46.66.197 with SMTP id h66mr10056178ljf.42.1517074963492; Sat, 27 Jan 2018 09:42:43 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.225.218 with HTTP; Sat, 27 Jan 2018 09:42:42 -0800 (PST) From: =?UTF-8?Q?Fernando_Apestegu=C3=ADa?= Date: Sat, 27 Jan 2018 18:42:42 +0100 Message-ID: Subject: cad/stepccode fails on -CURRENT since clang-6.0.0 was imported To: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jan 2018 17:42:46 -0000 Hi there, Since clang-6.0.0 was imported in -CURRENT (https://svnweb.freebsd.org/base/head/usr.bin/clang/llvm-cov/?view=log), cad/stepcode fails to build. It built fine in -CURRENT with clang-5.0.0 on both i386 and amd64. The full log can be found here (although I can't acces theat URL anymore, how long do they last?): Log URL: http://beefy11.nyi.freebsd.org/data/head-i386-default/p459899_s328300/logs/stepcode-0.8_3.log Build URL: http://beefy11.nyi.freebsd.org/build.html?mastername=head-i386-default&build=p459899_s328300 Last lines of the error log are as follows: In file included from /wrkdirs/usr/ports/cad/stepcode/work/stepcode-0.8/src/cllazyfile/lazy_test.cc:1: In file included from /wrkdirs/usr/ports/cad/stepcode/work/stepcode-0.8/src/cllazyfile/lazyInstMgr.h:8: In file included from /wrkdirs/usr/ports/cad/stepcode/work/stepcode-0.8/src/cllazyfile/lazyDataSectionReader.h:6: In file included from /wrkdirs/usr/ports/cad/stepcode/work/stepcode-0.8/src/cllazyfile/sectionReader.h:6: In file included from /wrkdirs/usr/ports/cad/stepcode/work/stepcode-0.8/src/cllazyfile/lazyTypes.h:11: /wrkdirs/usr/ports/cad/stepcode/work/stepcode-0.8/src/base/judy/src/judyL2Array.h:169:28: error: assigning to 'const std::__1::vector > *' from incompatible type 'unsigned long long' kv.value = ( JudyValue ) 0; ^~~~~~~~~~~~~~~ /wrkdirs/usr/ports/cad/stepcode/work/stepcode-0.8/src/base/judy/src/judyL2Array.h:180:20: note: in instantiation of member function 'judyL2Array::mostRecentPair' requested here return mostRecentPair(); ^ /wrkdirs/usr/ports/cad/stepcode/work/stepcode-0.8/src/cllazyfile/lazy_test.cc:48:37: note: in instantiation of member function 'judyL2Array::begin' requested here instanceRefs_t::cpair p = refs->begin(); ^ 1 error generated. ninja: build stopped: subcommand failed. *** Error code 1 I have no idea what the reason for the fail is. I had a look at the change log from 5 to 6 but a lot of things have gone on and I can not find any specific commit to blame. Sorry for the delay reporting this, but I didn't have time to have a look until a couple of days ago. Can anyone help here? Thanks in advance. From owner-freebsd-hackers@freebsd.org Sat Jan 27 21:42:03 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CF20EEC412E for ; Sat, 27 Jan 2018 21:42:03 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from tensor.andric.com (tensor.andric.com [87.251.56.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "tensor.andric.com", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A14EE7B0D6 for ; Sat, 27 Jan 2018 21:42:02 +0000 (UTC) (envelope-from dim@FreeBSD.org) Received: from coleburn.home.andric.com (coleburn.home.andric.com [192.168.0.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by tensor.andric.com (Postfix) with ESMTPSA id 58D02EF51; Sat, 27 Jan 2018 22:41:55 +0100 (CET) From: Dimitry Andric Message-Id: Content-Type: multipart/signed; boundary="Apple-Mail=_65B3958D-8733-4363-8E53-AB839C8BDCC7"; protocol="application/pgp-signature"; micalg=pgp-sha1 Mime-Version: 1.0 (Mac OS X Mail 10.3 \(3273\)) Subject: Re: cad/stepccode fails on -CURRENT since clang-6.0.0 was imported Date: Sat, 27 Jan 2018 22:41:54 +0100 In-Reply-To: Cc: FreeBSD Hackers To: =?utf-8?Q?Fernando_Apestegu=C3=ADa?= References: X-Mailer: Apple Mail (2.3273) X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jan 2018 21:42:04 -0000 --Apple-Mail=_65B3958D-8733-4363-8E53-AB839C8BDCC7 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 On 27 Jan 2018, at 18:42, Fernando Apestegu=C3=ADa = wrote: >=20 > Since clang-6.0.0 was imported in -CURRENT > = (https://svnweb.freebsd.org/base/head/usr.bin/clang/llvm-cov/?view=3Dlog),= > cad/stepcode fails to build. It built fine in -CURRENT with > clang-5.0.0 on both i386 and amd64. ... > = /wrkdirs/usr/ports/cad/stepcode/work/stepcode-0.8/src/base/judy/src/judyL2= Array.h:169:28: > error: assigning to 'const std::__1::vector std::__1::allocator > *' from incompatible type > 'unsigned long long' > kv.value =3D ( JudyValue ) 0; > ^~~~~~~~~~~~~~~ Like gcc 6 and higher, clang 6 now defaults to -std=3Dgnu++14, and from C++11 onwards, you must use 'nullptr' if you mean a null pointer, not the integer 0. You can either force the port to be compiled with -std=3Dgnu++98 (by adding USE_CXXSTD=3Dgnu++98 to the port Makefile), or change the two instances where these assignments are being done, e.g.: --- src/base/judy/src/judyL2Array.h.orig 2014-12-26 20:12:05 UTC +++ src/base/judy/src/judyL2Array.h @@ -166,7 +166,7 @@ class judyL2Array { kv.value =3D *_lastSlot; _success =3D true; } else { - kv.value =3D ( JudyValue ) 0; + kv.value =3D nullptr; _success =3D false; } kv.key =3D _buff[0]; --- src/base/judy/src/judyS2Array.h.orig 2014-12-26 20:12:05 UTC +++ src/base/judy/src/judyS2Array.h @@ -191,7 +191,7 @@ class judyS2Array { kv.value =3D *_lastSlot; _success =3D true; } else { - kv.value =3D ( JudyValue ) 0; + kv.value =3D nullptr; _success =3D false; } kv.key =3D _buff; -Dimitry --Apple-Mail=_65B3958D-8733-4363-8E53-AB839C8BDCC7 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.2 iF0EARECAB0WIQR6tGLSzjX8bUI5T82wXqMKLiCWowUCWmzyIgAKCRCwXqMKLiCW o9KIAKCL8nnZ9SjFMBPfL81i+jNcniqbQgCeMov2kYTgA64AFu/R3ik6q9w+Ga8= =6I9S -----END PGP SIGNATURE----- --Apple-Mail=_65B3958D-8733-4363-8E53-AB839C8BDCC7-- From owner-freebsd-hackers@freebsd.org Sat Jan 27 22:45:14 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 24A65EC7A9C for ; Sat, 27 Jan 2018 22:45:14 +0000 (UTC) (envelope-from fernando.apesteguia@gmail.com) Received: from mail-lf0-x22c.google.com (mail-lf0-x22c.google.com [IPv6:2a00:1450:4010:c07::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8452C7D8F0; Sat, 27 Jan 2018 22:45:13 +0000 (UTC) (envelope-from fernando.apesteguia@gmail.com) Received: by mail-lf0-x22c.google.com with SMTP id x196so4903357lfd.12; Sat, 27 Jan 2018 14:45:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=tDi+wxhS2WWZBsQPC2qtR4F32/3owCLIgh+dm7WQmtM=; b=gmV+EzVY1y59j+YfbwovussL8rD4WaBEmHusvHOztBvGHGYo1pp3IAI35AAGum5T7X m/F5/f1AXK1XngPiY4DcEZQIBz+79RArUoZ2SkO1FoqnkWUTmqIBpr7RiaokbC2blspE g/OElhfVFsDOW556/wywjhnqLaEk3hL+LAzZGbBkaYrC/+AHtGRv9A1tpqui+t+w4TfR CP3TdTi+WYImATHxDLg3aK02BgNnG/eMCpnrXGg5UNjJMvvGmGxJWaPl1ML7BuSelJVw +KY1I6SastaxYWkpXFFvONzImRO9N78uDSNbIayltMhIpmEpAAaimjLiwg1pbWmdh9jy 1lPw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=tDi+wxhS2WWZBsQPC2qtR4F32/3owCLIgh+dm7WQmtM=; b=Ru0qL6Ear0FxNy52LElhnSjwTCfo+pur51EV43+fUc3yJPYqx32cJmuNdJDU/kPS9G tiWvFkdgK3cVzh+gy/fs8hMimc1rFgghOOZ5XUBBABdC6Ho4aW0sdvUgoL9W5/6EzQ8F dVcYcMMFysxm3cvsBxhU2AHdF+d/LTf6fGNk6GYVB4aTwhLM29ogphS/3tDIfnYYR5QH 6tFsaYU8InVz2nXEg5+aUoE0J5Lh14ltAfaLBJoFHXmaU0YxAmwtpCjOea2PGGX2JjCi oURgvs1yT1bpl9V8/9l5zNIRBYDGVmUGsLGUnpVbe3ahmJ73xwCpjQu3pPOG3YM+D+fd FMAg== X-Gm-Message-State: AKwxytfnhE5frKTUhVNOkcn/8T8LPSN8fNy08P5Hqj8LCTTiWAoBIj5c 1zgHDC6yfQ7qnxSYKRt+OjUdyGxwbbRav4RiCMs= X-Google-Smtp-Source: AH8x225epqjfag97blHK0DbRXcvDrILVeVN3WReG8bUMrKK9AQVWhUomtXY4eDWhyLyP4OpvmRe3Xm7SuZOpFcn9V04= X-Received: by 10.25.90.81 with SMTP id o78mr9820932lfb.123.1517093111959; Sat, 27 Jan 2018 14:45:11 -0800 (PST) MIME-Version: 1.0 Received: by 10.25.225.218 with HTTP; Sat, 27 Jan 2018 14:45:10 -0800 (PST) Received: by 10.25.225.218 with HTTP; Sat, 27 Jan 2018 14:45:10 -0800 (PST) In-Reply-To: References: From: =?UTF-8?Q?Fernando_Apestegu=C3=ADa?= Date: Sat, 27 Jan 2018 23:45:10 +0100 Message-ID: Subject: Re: cad/stepccode fails on -CURRENT since clang-6.0.0 was imported To: Dimitry Andric Cc: FreeBSD Hackers Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jan 2018 22:45:14 -0000 El 27 ene. 2018 22:41, "Dimitry Andric" escribi=C3=B3: On 27 Jan 2018, at 18:42, Fernando Apestegu=C3=ADa wrote: > > Since clang-6.0.0 was imported in -CURRENT > (https://svnweb.freebsd.org/base/head/usr.bin/clang/llvm-cov/?view=3Dlog)= , > cad/stepcode fails to build. It built fine in -CURRENT with > clang-5.0.0 on both i386 and amd64. ... > /wrkdirs/usr/ports/cad/stepcode/work/stepcode-0.8/ src/base/judy/src/judyL2Array.h:169:28: > error: assigning to 'const std::__1::vector std::__1::allocator > *' from incompatible type > 'unsigned long long' > kv.value =3D ( JudyValue ) 0; > ^~~~~~~~~~~~~~~ Like gcc 6 and higher, clang 6 now defaults to -std=3Dgnu++14, and from C++11 onwards, you must use 'nullptr' if you mean a null pointer, not the integer 0. You can either force the port to be compiled with -std=3Dgnu++98 (by adding USE_CXXSTD=3Dgnu++98 to the port Makefile), or change the two instances where these assignments are being done, e.g.: --- src/base/judy/src/judyL2Array.h.orig 2014-12-26 20:12:05 UTC +++ src/base/judy/src/judyL2Array.h @@ -166,7 +166,7 @@ class judyL2Array { kv.value =3D *_lastSlot; _success =3D true; } else { - kv.value =3D ( JudyValue ) 0; + kv.value =3D nullptr; _success =3D false; } kv.key =3D _buff[0]; --- src/base/judy/src/judyS2Array.h.orig 2014-12-26 20:12:05 UTC +++ src/base/judy/src/judyS2Array.h @@ -191,7 +191,7 @@ class judyS2Array { kv.value =3D *_lastSlot; _success =3D true; } else { - kv.value =3D ( JudyValue ) 0; + kv.value =3D nullptr; _success =3D false; } kv.key =3D _buff; -Dimitry I will try the later and send the patch upstream. Thanks!