Date: Sun, 18 Apr 2010 18:17:43 +0100 From: Rui Paulo <rpaulo@freebsd.org> To: "C. Jayachandran" <c.jayachandran@gmail.com> Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. Message-ID: <DEF66A16-B4B8-4459-87C7-B695813FB68C@freebsd.org> In-Reply-To: <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> References: <w2z98a59be81004171540t2f0d5193nca2ec9e2540502e2@mail.gmail.com> <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
On 18 Apr 2010, at 13:09, Rui Paulo wrote: > On 17 Apr 2010, at 23:40, C. Jayachandran wrote: >=20 >> I've a set of initial patches to enable SMP for RMI processors. It >> comes up in multi-user with 32 CPUs. I could do buildworld before I >> updated to HEAD - with head there is a hang during buildworld which >> I'm looking at, but I think the initial work can be checked in. >>=20 >> Neel, can you have a look at the first two patches - one is to enable >> ULE scheduler and the second one is to move platform_init_ap to >> slightly later in the initialization sequence. >>=20 >> The patches are : >> 1. mips-ule-support.patch >> - Enable ULE scheduler for MIPS >>=20 >> 2. mips-smp-move-platform.patch >> - We need a hook to setup message ring and its interrupts, we use >> platform_init_ap now, and move it be called later for XLR >>=20 >> 3. rmi-prid.patch >> Add RMI processor ID prints - right now it prints unknown processor. >>=20 >> 4. rmi-pcib-fix.patch >> XLR pci bridge should be off the IODI bus, currently it is off the >> nexus bus which will cause issue when bus_setup_intr on nexus is >> called >>=20 >> 5. cleanup-reorg-cpuid-rge-kx.patch >> This set has a many changes rolled up: >> - clean up rge driver, remove unused code paths and commented code >> (long way to go still) >> - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage = updated >> - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses >> here, I have made changes for using XKPHYS to do the same process. >> - the KX bit is enabled and interrupts are disabled before the >> physical memory is read, to avoid the KX bit setting from affecting >> other code. >> - move message ring code to on_chip.c from xlr_machdep.c, now all >> message ring code is in on_chip.c >>=20 >> 6. rmi-xlr-smp.patch >> SMP Support for XLR >> - mpwait.S added to get the other CPUs out of bootloader code at = startup. >> - SMP platform related functions. >> - disable shared TLB code for SMP - since threads have different = mappings. >>=20 >> The patches are also available at >> http://sites.google.com/site/cjayachandran/files >>=20 >> Enabling all 32 threads will need a minor fix in the SMP code, see = the >> patch subr_smp-fix.patch at the same place. >=20 > On the netperf XLR, this panics with SCHED_ULE activated: >=20 > Copyright (c) 1992-2010 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, = 1994 > The Regents of the University of California. All rights = reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010 > = rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpaulo/fr= eebsd/head/sys/XLR mips > real memory =3D 3488608256 (3406844K bytes) > avail memory =3D 3395207168 (3237MB) > FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs > nexus0: <MIPS32 root nexus> > iodi0 on nexus0 > uart0: <16550 or compatible> on iodi0 > uart0: [FILTER] > uart0: console (38551,n,8,1) > pcib0: <XLR PCI bus> on iodi0 > pci0: <PCI bus> on pcib0 > pci0: <mass storage> at device 1.0 (no driver attached) > pci0: <network, ethernet> at device 3.0 (no driver attached) > rge0 on iodi0 > rge0: [ITHREAD] > rge0: Ethernet address:=20 > rge0: [10Mbps] > Total Active Core 8 > rge1 on iodi0 > rge1: [ITHREAD] > rge1: Ethernet address: > rge1: [1000Mbps] > rge2 on iodi0 > rge2: [ITHREAD] > rge2: Ethernet address:=20 > rge2: [10Mbps] > rge3 on iodi0 > rge3: [ITHREAD] > rge3: Ethernet address:=20 > rge3: [10Mbps] > Enabling MDIO interrupts > Timecounter "MIPS32" frequency 66000000 Hz quality 800 > Timecounters tick every 1.000 msec > bootpc_init: wired to interface 'rge1' > Sending DHCP Discover packet from interface rge1 > Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no = root path) > Sending DHCP Request packet from interface rge1 > Sending DHCP Discover packet from interface rge1 > DHCP/BOOTP timeout for server 255.255.255.255 > DHCP/BOOTP timeout for server 255.255.255.255 > DHCP/BOOTP timeout for server 255.255.255.255 > Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no = root path) > Sending DHCP Request packet from interface rge1 > DHCP/BOOTP timeout for server 255.255.255.255 > Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root = path) > rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org = boot file /zoo/rpaulo/lama1/boot/kernel/kernel > subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs = /zoo/rpaulo/lama1 rootopts nolockd=20 > Adjusted interface rge1 > panic: mutex sched lock 4 not owned at = /usr/home/rpaulo/freebsd/head/sys/kern/sched_ule.c:2054 > cpuid =3D 4 > KDB: enter: panic If I use SCHED_4BSD this happens: (am I missing something?) Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights = reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-CURRENT #4 r206791M: Sun Apr 18 17:10:55 UTC 2010 = rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/rpaulo/fr= eebsd/head/sys/XLR mips real memory =3D 3488608256 (3406844K bytes) avail memory =3D 3395252224 (3237MB) FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs nexus0: <MIPS32 root nexus> iodi0 on nexus0 uart0: <16550 or compatible> on iodi0 uart0: [FILTER] uart0: console (38551,n,8,1) pcib0: <XLR PCI bus> on iodi0 pci0: <PCI bus> on pcib0 pci0: <mass storage> at device 1.0 (no driver attached) pci0: <network, ethernet> at device 3.0 (no driver attached) rge0 on iodi0 rge0: [ITHREAD] rge0: Ethernet address: 00:0f:30:00:20:4e rge0: [10Mbps] Total Active Core 8 rge1 on iodi0 rge1: [ITHREAD] rge1: Ethernet address: 00:0f:30:00:20:4f rge1: [1000Mbps] rge2 on iodi0 rge2: [ITHREAD] rge2: Ethernet address: 00:0f:30:00:20:50 rge2: [10Mbps] rge3 on iodi0 rge3: [ITHREAD] rge3: Ethernet address: 00:0f:30:00:20:51 rge3: [10Mbps] Enabling MDIO interrupts Timecounter "MIPS32" frequency 66000000 Hz quality 800 Timecounters tick every 1.000 msec bootpc_init: wired to interface 'rge1' Sending DHCP Discover packet from interface rge1 (00:0f:30:00:20:4f) Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no root = path) Sending DHCP Request packet from interface rge1 (00:0f:30:00:20:4f) Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got root = path) rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org boot = file /zoo/rpaulo/lama1/boot/kernel/kernel subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs = /zoo/rpaulo/lama1 rootopts nolockd=20 Adjusted interface rge1 spin lock 0x804b8ea4 (sched lock) held by 0xc98c0260 (tid 100027) too = long panic: spin lock held too long cpuid =3D 12 KDB: enter: panic Regards, -- Rui Paulo
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?DEF66A16-B4B8-4459-87C7-B695813FB68C>