Date: Mon, 9 May 2011 16:21:15 +0200 From: Marko Zec <zec@fer.hr> To: freebsd-virtualization@freebsd.org Cc: Mikolaj Golub <trociny@freebsd.org>, Kostik Belousov <kib@freebsd.org> Subject: Re: vnet: acessing module's virtualized global variables from another module Message-ID: <201105091621.16414.zec@fer.hr> In-Reply-To: <86aaewdopy.fsf@kopusha.home.net> References: <86aaewdopy.fsf@kopusha.home.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Monday 09 May 2011 14:48:25 Mikolaj Golub wrote: > Hi, > > Trying ipfw_nat under VIMAGE kernel I got this panic on the module load: Hi, I think the problem here is that curvnet context is not set properly on entry to ipfw_nat_modevent(). The canonical way to initialize VNET-enabled subsystems is to trigger them using VNET_SYSINIT() macros (instead of using modevent mechanisms), which in turn ensure that: a) that the initializer function gets invoked for each existing vnet b) curvnet context is set properly on entry to initializer functions and Cheers, Marko > Fatal trap 12: page fault while in kernel mode > cpuid = 1; apic id = 01 > fault virtual address = 0x4 > fault code = supervisor read, page not present > instruction pointer = 0x20:0xc09f098e > stack pointer = 0x28:0xf563b944 > frame pointer = 0x28:0xf563b998 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = interrupt enabled, resume, IOPL = 0 > current process = 4264 (kldload) > > witness_checkorder(c6d5e91c,9,ca0ac2e3,223,0,...) at > witness_checkorder+0x6e _rw_wlock(c6d5e91c,ca0ac2e3,223,0,c0e8f795,...) at > _rw_wlock+0x82 > ipfw_nat_modevent(c98a48c0,0,0,75,0,...) at ipfw_nat_modevent+0x41 > module_register_init(ca0ad508,0,c0e8d834,e6,0,...) at > module_register_init+0xa7 > linker_load_module(0,f563bc18,c0e8d834,3fc,f563bc28,...) at > linker_load_module+0xa05 > kern_kldload(c86835c0,c72d3400,f563bc40,0,c8d0d000,...) at > kern_kldload+0x133 kldload(c86835c0,f563bcec,c09e8940,c86835c0,0,...) at > kldload+0x74 syscallenter(c86835c0,f563bce4,c0ce05dd,c1022150,0,...) at > syscallenter+0x263 syscall(f563bd28) at syscall+0x34 > Xint0x80_syscall() at Xint0x80_syscall+0x21 > --- syscall (304, FreeBSD ELF32, kldload), eip = 0x280da00b, esp = > 0xbfbfe79c, ebp = 0xbfbfec88 - > > It crashed on acessing data from virtualized global variable V_layer3_chain > in ipfw_nat_modevent(). V_layer3_chain is defined in ipfw module and it > turns out that &V_layer3_chain returns wrong location from anywhere but > ipfw.ko. > > May be this is a known issue, but I have not found info about this, so > below are details of investigation why this happens. > > Virtualized global variables are defined using the VNET_DEFINE() macro, > which places them in the 'set_vnet' linker set (in the base kernel or in > module). This is used to > > 1) copy these "default" values to each virtual network stack instance when > created; > > 2) act as unique global names by which the variable can be referred to. The > location of a per-virtual instance variable is calculated at run-time like > in the example below for layer3_chain variable in the default vnet (vnet0): > > vnet0->vnet_data_base + (uintptr_t) & vnet_entry_layer3_chain (1) > > For modules the thing is more complicated. When a module is loaded its > global variables from 'set_vnet' linker set are copied to the kernel > 'set_vnet', and for module to be able to access them the linker reallocates > all references accordingly (kern/link_elf.c:elf_relocaddr()): > > if (x >= ef->vnet_start && x < ef->vnet_stop) > return ((x - ef->vnet_start) + ef->vnet_base); > > So from inside the module the access to its virtualized variables works, > but from the outside we get wrong location using calculation like above > (1), because &vnet_entry_layer3_chain returns address of the variable in > the module's 'set_vnet'. > > The workaround is to compile such modules into the kernel or use a hack I > have done for ipfw_nat -- add the function to ipfw module which returns the > location of virtualized layer3_chain variable and use this location instead > of V_layer3_chain macro (see the attached patch). > > But I suppose the problem is not a new and there might be better approach > already invented to deal with this?
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?201105091621.16414.zec>