Date: Sun, 18 Apr 2010 10:47:34 -0700 From: Randall Stewart <rrs@lakerest.net> To: Rui Paulo <rpaulo@freebsd.org> Cc: freebsd-mips@freebsd.org Subject: Re: SMP support for XLR processors. Message-ID: <A1FC32B9-1105-43C5-91C1-C4A81F78066B@lakerest.net> In-Reply-To: <DEF66A16-B4B8-4459-87C7-B695813FB68C@freebsd.org> References: <w2z98a59be81004171540t2f0d5193nca2ec9e2540502e2@mail.gmail.com> <544BD79C-24C0-4015-AE04-F7F8D70CF537@freebsd.org> <DEF66A16-B4B8-4459-87C7-B695813FB68C@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
Rui: What type of load are you doing.. NFS or MFS fs? I am using a root NFS mounted and with all the patchs applied the machine no longer recognizes the DHCP responses from my daemon. I can clearly see these in wireshark... I will revert all patches out and see what happens without the rge-kx cleanup.. R On Apr 18, 2010, at 10:17 AM, Rui Paulo wrote: > On 18 Apr 2010, at 13:09, Rui Paulo wrote: > >> On 17 Apr 2010, at 23:40, C. Jayachandran wrote: >> >>> I've a set of initial patches to enable SMP for RMI processors. It >>> comes up in multi-user with 32 CPUs. I could do buildworld before I >>> updated to HEAD - with head there is a hang during buildworld which >>> I'm looking at, but I think the initial work can be checked in. >>> >>> Neel, can you have a look at the first two patches - one is to >>> enable >>> ULE scheduler and the second one is to move platform_init_ap to >>> slightly later in the initialization sequence. >>> >>> The patches are : >>> 1. mips-ule-support.patch >>> - Enable ULE scheduler for MIPS >>> >>> 2. mips-smp-move-platform.patch >>> - We need a hook to setup message ring and its interrupts, we use >>> platform_init_ap now, and move it be called later for XLR >>> >>> 3. rmi-prid.patch >>> Add RMI processor ID prints - right now it prints unknown processor. >>> >>> 4. rmi-pcib-fix.patch >>> XLR pci bridge should be off the IODI bus, currently it is off the >>> nexus bus which will cause issue when bus_setup_intr on nexus is >>> called >>> >>> 5. cleanup-reorg-cpuid-rge-kx.patch >>> This set has a many changes rolled up: >>> - clean up rge driver, remove unused code paths and commented code >>> (long way to go still) >>> - xlr_cpu_id(), xlr_core_id() and xlr_thr_id() updated and usage >>> updated >>> - fix a bug - we cannot use MIPS_PHYS_TO_KSEG0 on physical addresses >>> here, I have made changes for using XKPHYS to do the same process. >>> - the KX bit is enabled and interrupts are disabled before the >>> physical memory is read, to avoid the KX bit setting from affecting >>> other code. >>> - move message ring code to on_chip.c from xlr_machdep.c, now all >>> message ring code is in on_chip.c >>> >>> 6. rmi-xlr-smp.patch >>> SMP Support for XLR >>> - mpwait.S added to get the other CPUs out of bootloader code at >>> startup. >>> - SMP platform related functions. >>> - disable shared TLB code for SMP - since threads have different >>> mappings. >>> >>> The patches are also available at >>> http://sites.google.com/site/cjayachandran/files >>> >>> Enabling all 32 threads will need a minor fix in the SMP code, see >>> the >>> patch subr_smp-fix.patch at the same place. >> >> On the netperf XLR, this panics with SCHED_ULE activated: >> >> Copyright (c) 1992-2010 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, >> 1994 >> The Regents of the University of California. All rights reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 9.0-CURRENT #2 r206791M: Sun Apr 18 12:02:52 UTC 2010 >> rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/ >> rpaulo/freebsd/head/sys/XLR mips >> real memory = 3488608256 (3406844K bytes) >> avail memory = 3395207168 (3237MB) >> FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs >> nexus0: <MIPS32 root nexus> >> iodi0 on nexus0 >> uart0: <16550 or compatible> on iodi0 >> uart0: [FILTER] >> uart0: console (38551,n,8,1) >> pcib0: <XLR PCI bus> on iodi0 >> pci0: <PCI bus> on pcib0 >> pci0: <mass storage> at device 1.0 (no driver attached) >> pci0: <network, ethernet> at device 3.0 (no driver attached) >> rge0 on iodi0 >> rge0: [ITHREAD] >> rge0: Ethernet address: >> rge0: [10Mbps] >> Total Active Core 8 >> rge1 on iodi0 >> rge1: [ITHREAD] >> rge1: Ethernet address: >> rge1: [1000Mbps] >> rge2 on iodi0 >> rge2: [ITHREAD] >> rge2: Ethernet address: >> rge2: [10Mbps] >> rge3 on iodi0 >> rge3: [ITHREAD] >> rge3: Ethernet address: >> rge3: [10Mbps] >> Enabling MDIO interrupts >> Timecounter "MIPS32" frequency 66000000 Hz quality 800 >> Timecounters tick every 1.000 msec >> bootpc_init: wired to interface 'rge1' >> Sending DHCP Discover packet from interface rge1 >> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no >> root path) >> Sending DHCP Request packet from interface rge1 >> Sending DHCP Discover packet from interface rge1 >> DHCP/BOOTP timeout for server 255.255.255.255 >> DHCP/BOOTP timeout for server 255.255.255.255 >> DHCP/BOOTP timeout for server 255.255.255.255 >> Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no >> root path) >> Sending DHCP Request packet from interface rge1 >> DHCP/BOOTP timeout for server 255.255.255.255 >> Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got >> root path) >> rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org >> boot file /zoo/rpaulo/lama1/boot/kernel/kernel >> subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/ >> rpaulo/lama1 rootopts nolockd >> Adjusted interface rge1 >> panic: mutex sched lock 4 not owned at /usr/home/rpaulo/freebsd/ >> head/sys/kern/sched_ule.c:2054 >> cpuid = 4 >> KDB: enter: panic > > If I use SCHED_4BSD this happens: (am I missing something?) > > Copyright (c) 1992-2010 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, > 1994 > The Regents of the University of California. All rights reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 9.0-CURRENT #4 r206791M: Sun Apr 18 17:10:55 UTC 2010 > rpaulo@freebsd-vmware:/usr/home/rpaulo/freebsd/obj/mips/usr/home/ > rpaulo/freebsd/head/sys/XLR mips > real memory = 3488608256 (3406844K bytes) > avail memory = 3395252224 (3237MB) > FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs > nexus0: <MIPS32 root nexus> > iodi0 on nexus0 > uart0: <16550 or compatible> on iodi0 > uart0: [FILTER] > uart0: console (38551,n,8,1) > pcib0: <XLR PCI bus> on iodi0 > pci0: <PCI bus> on pcib0 > pci0: <mass storage> at device 1.0 (no driver attached) > pci0: <network, ethernet> at device 3.0 (no driver attached) > rge0 on iodi0 > rge0: [ITHREAD] > rge0: Ethernet address: 00:0f:30:00:20:4e > rge0: [10Mbps] > Total Active Core 8 > rge1 on iodi0 > rge1: [ITHREAD] > rge1: Ethernet address: 00:0f:30:00:20:4f > rge1: [1000Mbps] > rge2 on iodi0 > rge2: [ITHREAD] > rge2: Ethernet address: 00:0f:30:00:20:50 > rge2: [10Mbps] > rge3 on iodi0 > rge3: [ITHREAD] > rge3: Ethernet address: 00:0f:30:00:20:51 > rge3: [10Mbps] > Enabling MDIO interrupts > Timecounter "MIPS32" frequency 66000000 Hz quality 800 > Timecounters tick every 1.000 msec > bootpc_init: wired to interface 'rge1' > Sending DHCP Discover packet from interface rge1 (00:0f:30:00:20:4f) > Received DHCP Offer packet on rge1 from 192.168.5.1 (accepted) (no > root path) > Sending DHCP Request packet from interface rge1 (00:0f:30:00:20:4f) > Received DHCP Ack packet on rge1 from 192.168.5.1 (accepted) (got > root path) > rge1 at 192.168.5.32 server 192.168.5.1 server name zoo.freebsd.org > boot file /zoo/rpaulo/lama1/boot/kernel/kernel > subnet mask 255.255.255.0 root_server 192.168.5.1 rootfs /zoo/rpaulo/ > lama1 rootopts nolockd > Adjusted interface rge1 > spin lock 0x804b8ea4 (sched lock) held by 0xc98c0260 (tid 100027) > too long > panic: spin lock held too long > cpuid = 12 > KDB: enter: panic > > > Regards, > -- > Rui Paulo > > ------------------------------ Randall Stewart 803-317-4952 (cell) 803-345-0391(direct)
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?A1FC32B9-1105-43C5-91C1-C4A81F78066B>