From owner-freebsd-ipfw@freebsd.org Thu Mar 10 07:20:13 2016 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4E3F6ACAA62; Thu, 10 Mar 2016 07:20:13 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from gw.catspoiler.org (unknown [IPv6:2602:304:b010:ef20::f2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "gw.catspoiler.org", Issuer "gw.catspoiler.org" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 2A5B0F3D; Thu, 10 Mar 2016 07:20:13 +0000 (UTC) (envelope-from truckman@FreeBSD.org) Received: from FreeBSD.org (mousie.catspoiler.org [192.168.101.2]) by gw.catspoiler.org (8.15.2/8.15.2) with ESMTP id u2A7K1v4013479; Wed, 9 Mar 2016 23:20:06 -0800 (PST) (envelope-from truckman@FreeBSD.org) Message-Id: <201603100720.u2A7K1v4013479@gw.catspoiler.org> Date: Wed, 9 Mar 2016 23:20:01 -0800 (PST) From: Don Lewis Subject: Re: Dummynet AQM v0.1- CoDel and FQ-CoDel for FreeBSD's ipfw/dummynet To: ralsaadi@swin.edu.au cc: freebsd-net@FreeBSD.org, freebsd-ipfw@FreeBSD.org, garmitage@swin.edu.au In-Reply-To: <6545444AE21C2749939E637E56594CEA3C1B0A7C@gsp-ex02.ds.swin.edu.au> MIME-Version: 1.0 Content-Type: TEXT/plain; charset=us-ascii X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Mar 2016 07:20:13 -0000 On 10 Mar, Rasool Al-Saadi wrote: > > > On Wednesday, 9 March 2016, Don Lewis wrote: >> >> On 26 Feb, Rasool Al-Saadi wrote: >> > Dear all, >> > >> > I would like to announce that we (myself and Grenville Armitage) >> > released >> Dummynet AQM v0.1, which is an independent implementation of CoDel >> and FQ-CoDel for FreeBSD's ipfw/dummynet framework, based on the IETF >> CoDel [1] and FQ-CoDel [2] Internet-Drafts. >> > We prepared patches for FreeBSD11-CURRENT-r295345 and FreeBSD 10.x- >> RELEASE (10.0, 10.1, 10.2), and a technical report of our >> implementation. >> > >> > Patches and documentation can be found in: >> > http://caia.swin.edu.au/freebsd/aqm >> >> Without the patch below, the dummynet module fails to load >> >> # kldload dummynet.ko >> kldload: can't load dummynet.ko: No such file or directory >> >> and the following is printed to /var/log/messages: >> >> link_elf: symbol sysctl__net_inet_ip_dummynet_children undefined > > Thanks again for testing the patch and for providing feedback. > > It seems that this error (and the compilation error in your previous > email) appears in i386 versions of FreeBSD. I was testing on FreeBSD 10.3-PRERELEASE and would have been willing to bet that the difference in behavior was due to a recent change in FreeBSD. After testing, this second bug is also a difference between i386 and amd64. Digging into the problem, this line in ip_dn_io.c: static SYSCTL_NODE(_net_inet_ip, OID_AUTO, dummynet, CTLFLAG_RW, 0, "Dummynet"); gets translated into: static struct sysctl_oid_list sysctl__net_inet_ip_dummynet_children; static struct sysctl_oid sysctl___net_inet_ip_dummynet = { &sysctl__net_inet_ip_children, { ((void *)0) }, (-1), 1|((0x80000000|0x40000000)), (void*)&sysctl__net_inet_ip_dummynet_children, 0, "dummynet", 0, "N", 0, 0, "Dummynet" }; __asm__(".globl " "__start_set_sysctl_set"); __asm__(".globl " "__stop_set_sysctl_set"); static void const * const __set_sysctl_set_sym_sysctl___net_inet_ip_dummynet __attribute__((__section__("set_" "sysctl_set"))) __attribute__((__used__)) = &sysctl___net_inet_ip_dummynet; _Static_assert((((0x80000000|0x40000000)) & 0xf) == 0 || (((0x80000000|0x40000000)) & 0) == 1, "compile-time assertion failed"); by the C preprocessor, and it shows up in the symbol table of the .o file (on amd64): 0000000000000140 b sysctl__net_inet_ip_dummynet_children where "b" indicates that it is a local symbol in the uninitialized data section. The dn_aqm_codel.c and dn_sched_fq_codel.c files also what access to this SYSCTL_NODE, which they indicated by using: SYSCTL_DECL(_net_inet_ip_dummynet); which gets pre-processed to: extern struct sysctl_oid_list sysctl__net_inet_ip_dummynet_children; which results in an undefined symbol in the compiled .o file: U sysctl__net_inet_ip_dummynet_children A symbol declared as an extern in one compilation until should not get linked to a matching symbol defined as a static in another compilation unit. Looking at the symbol table for the module dummynet.ko, I see both the static and undefined versions of the symbol: 00000000000001b0 b sysctl__net_inet_ip_dummynet_children U sysctl__net_inet_ip_dummynet_children For some reason, the kernel linker on amd64 accepts this module, whereas on i386 it is (correctly in my opinion) rejected. I'll try to put together a simple example that I can use to file a FreeBSD bug report. With the patch (on i386), the symbol is defined as global in ip_dn_io.o: 000000b8 B sysctl__net_inet_ip_dummynet_children and there are no undefined versions of this symbol in the .ko file: 00010e4c b sysctl__net_inet_ip_dummynet_children however I don't know why it is no longer a global in the .ko file.