Date: Thu, 6 Aug 2009 22:23:30 GMT From: Julian Elischer <julian@FreeBSD.org> To: Perforce Change Reviews <perforce@freebsd.org> Subject: PERFORCE change 167074 for review Message-ID: <200908062223.n76MNUTu012625@repoman.freebsd.org>
next in thread | raw e-mail | index | archive | help
http://perforce.freebsd.org/chv.cgi?CH=167074 Change 167074 by julian@julian-mac on 2009/08/06 22:22:52 wordsmithing. Add a comment to clear up somethings that I didn't express well originally. Still needs a lot of work. Affected files ... .. //depot/projects/vimage/porting_to_vimage.txt#11 edit Differences ... ==== //depot/projects/vimage/porting_to_vimage.txt#11 (text+ko) ==== @@ -9,9 +9,9 @@ to operate on multiple independent instances of its state so that it can participate in a virtual machine / virtual environment scenario. It refers to a part of the Jail infrastructure in FreeBSD. For historical reasons -Virtual network stack enabled jails are know a vimage enabled jails or vnet +Virtual network stack enabled jails are known a vimage enabled jails or vnet enabled jails. The currently correct term is the latter, which is a -contraction ofthe first. In the future other parts of the system may be +contraction of the first. In the future other parts of the system may be virtualized using the same technology and the term to cover all such components would be VIMAGE enhanced modules. @@ -23,26 +23,26 @@ things in order to allow for all the changes to be conditionally compilable, i.e. permitting the virtualized code to fall back to operation on global state. -The rest of this documant will discuss NETWORK virtualization -thoughthe concepts will be true in the future for other parts of the +The rest of this document will discuss NETWORK virtualization +though the concepts may be true in the future for other parts of the system. The most visible change throughout the existing code is typically replacement of direct references to global variables with macros; foo_bar thus becomes -V_foo_bar. V_foo_bar macros will resolve back to foo_bar global in default -kernel builds, and alternatively to teh logical equivalent of +V_foo_bar. V_foo_bar macros will resolve back to the foo_bar global in +default kernel builds, and alternatively to the logical equivalent of some_base_pointer->_foo_bar for "options VIMAGE" kernel configs. -Prepending of "V_" prefixes to variable references -helps in visual discrimination between global and virtualized state. +Prepending of "V_" prefixes to variable references helps in +visual discrimination between global and virtualized state. It is also possible to use an alternative syntax, of VNET(foo_bar) to achieve the same thing. The deveopers felt that V_foo_bar was less visually distracting while still providing enough clues to the reader -that the variable is virtualized. In fact the V_foo_bar symtac is +that the variable is virtualized. In fact the V_foo_bar macro is locally defined near the definition of foo_bar to be an alias for VNET(foo_bar) so the two are not only equivalent, they are the same. -The framework extends the sysctl infrastructure to support access to +The framework also extends the sysctl infrastructure to support access to virtualized state through introduction of the SYSCTL_VNET family of macros; those also automatically fall back to their standard SYSCTL counterparts in default kernel builds. @@ -51,26 +51,28 @@ which permits userland binaries such as netstat to operate unmodified on "options VIMAGE" kernels, though this may have some security implications. -Vnets are associated with jails. Each process is associated with a jail, -usually the default (null) jail, and jails currently hang off of -ucred-s. This relationship defines a process's administrative affinity -to a vnet and thus indirectly to all of its state. All network interfaces -and sockets hold pointers back to their parent vnets; this relationship -is obviously entirely independent from proc->ucred->jail bindings. -Hence, when a process opens a socket, the socket will get bound to a vnet -instance hanging off of proc->ucred->jail->vnet, but once such a socket->vnet -binding gets established, it cannot be changed for the entire socket lifetime. +Vnets are associated with jails. In 8.0, every process is associated with +a jail, usually the default (null) jail, and jails currently hang off of +a processes ucred. This relationship defines a process's administrative +affinity to a vnet and thus indirectly to all of its state. All network +interfaces and sockets hold pointers back to their parent vnets. +This relationship is obviously entirely independent from proc->ucred->jail +bindings. Hence, when a process opens a socket, the socket will get bound +to a vnet instance hanging off of proc->ucred->jail->vnet, but once such a +socket->vnet binding gets established, it cannot be changed for the entire +socket lifetime. + The mapping of a from a thread to a vnet is should always be doen via the TD_TO_VNET macro as the path may change in the future as we get more -experience with usinghte system. +experience with using the system. Certain classes of network interfaces (Ethernet in particular) can be -assigned from one vnet to another at any time. By definition all vnets -are are independent and can communicate only if they are explicitly -provided with communication paths; currently mainly netgraph is used to +reassigned from one vnet to another at any time. By definition all vnets +are independent and can communicate only if they are explicitly +provided with communication paths. Currently mainly netgraph is used to establish inter-vnet datapaths, though other paths are being explored such as the 'epair' back-to-back virtual interface pair, in which -teh different sides may exist in different jails. +the different sides may exist in different jails. In network traffic processing the vnet affinity is defined either by the inbound interface or by the socket / pcb -> vnet binding. However, there @@ -89,7 +91,7 @@ events, given that those are inherently vnet-contextless on entry. The current rule is that when not in networking code, the result of -the curvnet macro will return NULL and evaluating a V_xxx (or VNET(xxx)) +the 'curvnet' macro will return NULL and evaluating a V_xxx (or VNET(xxx)) macro will result in an kernel page-fault error. While this is not strictly necessary, it aids in debugging and assurance of program correctness. Note this does NOT mean that TD_TO_VNET(curthread) is invalid. @@ -105,12 +107,12 @@ 1/ decide whether the module needs to be virtualised. - if the module is a driver for specific hardware, it makes sense that + If the module is a driver for specific hardware, it makes sense that there be only one instance of the driver as there is only one piece of physical hardware. There are changes in the networking code to allow physical (or virtual) interfaces to be moved between vnets. This generally requires NO changes to the network drivers of the classes - covered (e.g. ethernet). Currently if your module is doe snot have any + covered (e.g. ethernet). Currently if your module is does not have any networking facet, the answer is "no" by default. 2/ If the module is to be virtualised, decide which attributes of the @@ -121,12 +123,12 @@ lock guarding it. It might also make sense if the "foo_debug" sysctl controls all the instances at once, while on the other hand, the "foo_mode" sysctl might make better sense if it were controllable - on a virtual system by virtual system basis. + on a virtual system by virtual system basis. 3/ Work out what global variables and structures are to be virtualised to - achieve the behaviour required for part #3. + achieve the behaviour required for part #2. -4/ Work out for all the code paths through the module, how the path entering +4/ Work out for all the code paths through the module, how the thread entering the module can divine which virtual environment it is on. Some examples: @@ -296,6 +298,9 @@ /* maybe sleep 1 mSec or something to let threads get out */ break; + case MOD_SHUTDOWN: + break; + default: err = EOPNOTSUPP; break; @@ -351,15 +356,26 @@ VNET_SYSUNINIT called for every jail/vnet SYSUNINIT -On system shutdown: - effectively the same as unload - {with exception of modevent?} +On system shutdown: [check/fix this ] + MODEVENT(shutdown) + ? VNET_SYSUNINIT called for every jail/vnet + ? SYSUNINIT NOTICE that while the order of the SYSINIT and VNET_SYSINIT is reversed from that of SYSUNINIT and VNET_SYSUNINIT, MODEVENTS do not follow this rule and thus it is dangerous to initialise and uninitialise things which are order dependent using MODEVENTs. +Or, put another way, +Since MODEVENT is called first during l module load, it would, by the +assumption that everything is reversed, be easy to assume that MODEVENT +is called AFTER the SYSINITS during unload. This is in fact not +the case. (and I have the scars to prove it). + +It might be make some sense if the "QUIESCE" was called before the +SYSINIT/SYSUNINIT and the UNLOAD called after.. with a millisecond +sleep between them, but this is not the case either. + Since initial values are copied into the virtualized variables on each new instantiatin, it is quite possible to have modules for which some of the above methods are not needed, and they may be left out.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200908062223.n76MNUTu012625>