Date: Mon, 19 Apr 2010 00:00:41 +0530 From: "C. Jayachandran" <c.jayachandran@gmail.com> To: Randall Stewart <rrs@lakerest.net> Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. Message-ID: <v2q98a59be81004181130t44d115c2ld94dae1e6ad5880f@mail.gmail.com> In-Reply-To: <B878BB7E-B7E2-4D0A-B99E-7B8DA1BF84E7@lakerest.net> References: <w2z98a59be81004171540t2f0d5193nca2ec9e2540502e2@mail.gmail.com> <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> <DEF66A16-B4B8-4459-87C7-B695813FB68C@freebsd.org> <B878BB7E-B7E2-4D0A-B99E-7B8DA1BF84E7@lakerest.net>
next in thread | previous in thread | raw e-mail | index | archive | help
I did not have invariants enabled when I moved to HEAD, and I got to multiuser login prompt. The panic is easily seen with INVARIANTS enabled with all 32 CPUs. But it does not happen if I cut down the cpus to 16 [ userapp_mask_cpus 0xffff on boot prompt]. Looks like there is some race I'm hitting with more CPUs. Will look at this... JC. On Sun, Apr 18, 2010 at 11:07 PM, Randall Stewart <rrs@lakerest.net> wrote: > OK, > > it appears in my early morning fog as I started I missed a patch ;-) > Anyway... I know better than to work before my first cup of coffee ... > > I am rebuilding and will try again with all the patches ;-) > > I suspect I will see your panic.. my guess is the lockup JC saw > was a LOR and he ran with NO WITNESS/INVARIANT in his build. > > We should be able to fix this pretty easily ;-) > > Give me a bit of time... > > R > > On Apr 18, 2010, at 10:17 AM, Rui Paulo wrote: > >> On 18 Apr 2010, at 13:09, Rui Paulo wrote: >> >>> On 17 Apr 2010, at 23:40, C. Jayachandran wrote: >>> >>>> I've a set of initial patches to enable SMP for RMI processors. It >>>> comes up in multi-user with 32 CPUs. =A0I could do buildworld before I >>>> updated to HEAD - with head there is a hang during buildworld which >>>> I'm looking at, but I think the initial work can be checked in. >>>> >>>> Neel, can you have a look at the first two patches - one is to enable >>>> ULE scheduler and the second one is to move platform_init_ap to >>>> slightly later in the initialization sequence. >>>> >>>> The patches are : >>>> 1. mips-ule-support.patch >>>> - Enable ULE scheduler for MIPS >>>> >>>> 2. mips-smp-move-platform.patch >>>> - We need a hook to setup message ring and its interrupts, we use >>>> platform_init_ap now, and move it be called later for XLR >>>> >>>> 3. rmi-prid.patch >>>> Add RMI processor ID prints - right now it prints unknown processor. >>>> >>>> 4. rmi-pcib-fix.patch >>>> XLR pci bridge should be off the IODI bus, currently it is off the >>>> nexus bus which will cause issue when bus_setup_intr on nexus is >>>> called >>>> >>>> 5. cleanup-reorg-cpuid-rge-kx.patch >>>> This set has a many changes rolled up: >>>> - clean up rge driver, remove unused code paths and commented code >>>> (long way to go still) >>>> - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage updat= ed >>>> - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses >>>> here, I have made changes for using XKPHYS to do the same process. >>>> - the KX bit is enabled and interrupts are disabled before the >>>> physical memory is read, to avoid the KX bit setting from affecting >>>> other code. >>>> - move message ring code to on_chip.c from xlr_machdep.c, now all >>>> message ring code is in on_chip.c >>>> >>>> 6. rmi-xlr-smp.patch >>>> SMP Support for XLR >>>> - mpwait.S added to get the other CPUs out of bootloader code at >>>> startup. >>>> - SMP platform related functions. >>>> - disable shared TLB code for SMP - since threads have different >>>> mappings. >>>> >>>> The patches are also available at >>>> http://sites.google.com/site/cjayachandran/files >>>> >>>> Enabling all 32 threads will need a minor fix in the SMP code, see the >>>> patch subr_smp-fix.patch at the same place. >>> >>> On the netperf XLR, this panics with SCHED_ULE activated: >>> >>> Copyright (c) 1992-2010 The FreeBSD Project. >>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 199= 4 >>> =A0 =A0 =A0 =A0The Regents of the University of California. All rights = reserved. >>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>> FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010 >>> >>> =A0rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpa= ulo/freebsd/head/sys/XLR >>> mips >>> real memory =A0=3D 3488608256 (3406844K bytes) >>> avail memory =3D 3395207168 (3237MB) >>> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs >>> nexus0: <MIPS32 root nexus> >>> iodi0 on nexus0 >>> uart0: <16550 or compatible> on iodi0 >>> uart0: [FILTER] >>> uart0: console (38551,n,8,1) >>> pcib0: <XLR PCI bus> on iodi0 >>> pci0: <PCI bus> on pcib0 >>> pci0: <mass storage> at device 1.0 (no driver attached) >>> pci0: <network, ethernet> at device 3.0 (no driver attached) >>> rge0 on iodi0 >>> rge0: [ITHREAD] >>> rge0: Ethernet address: >>> rge0: [10Mbps] >>> Total Active Core 8 >>> rge1 on iodi0 >>> rge1: [ITHREAD] >>> rge1: Ethernet address: >>> rge1: [1000Mbps] >>> rge2 on iodi0 >>> rge2: [ITHREAD] >>> rge2: Ethernet address: >>> rge2: [10Mbps] >>> rge3 on iodi0 >>> rge3: [ITHREAD] >>> rge3: Ethernet address: >>> rge3: [10Mbps] >>> Enabling MDIO interrupts >>> Timecounter "MIPS32" frequency 66000000 Hz quality 800 >>> Timecounters tick every 1.000 msec >>> bootpc_init: wired to interface 'rge1' >>> Sending DHCP Discover packet from interface rge1 >>> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root >>> path) >>> Sending DHCP Request packet from interface rge1 >>> Sending DHCP Discover packet from interface rge1 >>> DHCP/BOOTP timeout for server 255.255.255.255 >>> DHCP/BOOTP timeout for server 255.255.255.255 >>> DHCP/BOOTP timeout for server 255.255.255.255 >>> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root >>> path) >>> Sending DHCP Request packet from interface rge1 >>> DHCP/BOOTP timeout for server 255.255.255.255 >>> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root >>> path) >>> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boo= t >>> file /zoo/rpaulo/lama1/boot/kernel/kernel >>> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs >>> /zoo/rpaulo/lama1 rootopts nolockd >>> Adjusted interface rge1 >>> panic: mutex sched lock 4 not owned at >>> /usr/home/rpaulo/freebsd/head/sys/kern/sched_ule.c:2054 >>> cpuid =3D 4 >>> KDB: enter: panic >> >> If I use SCHED_4BSD this happens: (am I missing something?) >> >> Copyright (c) 1992-2010 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 >> =A0 =A0 =A0 =A0The Regents of the University of California. All rights r= eserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 9.0-CURRENT #4 r206791M: Sun Apr 18 17:10:55 UTC 2010 >> >> rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpaulo/= freebsd/head/sys/XLR >> mips >> real memory =A0=3D 3488608256 (3406844K bytes) >> avail memory =3D 3395252224 (3237MB) >> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs >> nexus0: <MIPS32 root nexus> >> iodi0 on nexus0 >> uart0: <16550 or compatible> on iodi0 >> uart0: [FILTER] >> uart0: console (38551,n,8,1) >> pcib0: <XLR PCI bus> on iodi0 >> pci0: <PCI bus> on pcib0 >> pci0: <mass storage> at device 1.0 (no driver attached) >> pci0: <network, ethernet> at device 3.0 (no driver attached) >> rge0 on iodi0 >> rge0: [ITHREAD] >> rge0: Ethernet address: 00:0f:30:00:20:4e >> rge0: [10Mbps] >> Total Active Core 8 >> rge1 on iodi0 >> rge1: [ITHREAD] >> rge1: Ethernet address: 00:0f:30:00:20:4f >> rge1: [1000Mbps] >> rge2 on iodi0 >> rge2: [ITHREAD] >> rge2: Ethernet address: 00:0f:30:00:20:50 >> rge2: [10Mbps] >> rge3 on iodi0 >> rge3: [ITHREAD] >> rge3: Ethernet address: 00:0f:30:00:20:51 >> rge3: [10Mbps] >> Enabling MDIO interrupts >> Timecounter "MIPS32" frequency 66000000 Hz quality 800 >> Timecounters tick every 1.000 msec >> bootpc_init: wired to interface 'rge1' >> Sending DHCP Discover packet from interface rge1 (00:0f:30:00:20:4f) >> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root >> path) >> Sending DHCP Request packet from interface rge1 (00:0f:30:00:20:4f) >> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root >> path) >> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boot >> file /zoo/rpaulo/lama1/boot/kernel/kernel >> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/rpaulo/lam= a1 >> rootopts nolockd >> Adjusted interface rge1 >> spin lock 0x804b8ea4 (sched lock) held by 0xc98c0260 (tid 100027) too lo= ng >> panic: spin lock held too long >> cpuid =3D 12 >> KDB: enter: panic >> >> >> Regards, >> -- >> Rui Paulo >> >> > > ------------------------------ > Randall Stewart > 803-317-4952 (cell) > 803-345-0391(direct) > > --=20 C. Jayachandran c.jayachandran@gmail.com
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?v2q98a59be81004181130t44d115c2ld94dae1e6ad5880f>