Date: Wed, 20 Apr 2011 23:33:00 +0400 From: Sergey Kandaurov <pluknet@gmail.com> To: Ian Smith <smithi@nimnet.asn.au> Cc: Daniel Gerzo <danger@freebsd.org>, freebsd-stable@freebsd.org, Alexander Motin <mav@freebsd.org>, John Baldwin <jhb@freebsd.org> Subject: Re: kern.smp.maxid error on i386 UP [was: powerd / cpufreq question] Message-ID: <BANLkTimUCDg7-t9DRubh8LHDdcQv-8Gr0Q@mail.gmail.com> In-Reply-To: <BANLkTi=7eS5CC0BMO2WdzasD61g5oCzZpA@mail.gmail.com> References: <4D9EEDAF.3020803@rulez.sk> <20110411125416.S35056@sola.nimnet.asn.au> <4DA37E31.4020700@FreeBSD.org> <20110413024230.Y35056@sola.nimnet.asn.au> <20110420164100.Y43371@sola.nimnet.asn.au> <BANLkTi=7eS5CC0BMO2WdzasD61g5oCzZpA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--002354471dd84ebfd304a15eb14c Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable On 20 April 2011 22:29, Sergey Kandaurov <pluknet@gmail.com> wrote: > On 20 April 2011 21:02, Ian Smith <smithi@nimnet.asn.au> wrote: >> On Wed, 13 Apr 2011, Ian Smith wrote: >> =A0> On Tue, 12 Apr 2011, Daniel Gerzo wrote: >> =A0> =A0> On 11.4.2011 6:08, Ian Smith wrote: >> [..] >> =A0> =A0> > Are those kern.cp_times values as they came, or did you remo= ve trailing >> =A0> =A0> > zeroes? =A0Reason I ask is that on my Thinkpad T23, single-c= ore 1133/733 >> =A0> =A0> > MHz, sysctl kern.cp_time shows the usual 5 values, but kern.= cp_times has >> =A0> =A0> > the same 5 values for cpu0, but then 5 zeroes for each of cp= u1 through >> =A0> =A0> > cpu31, on 8.2-PRE about early January. =A0I need to update t= he script to >> =A0> =A0> > remove surplus data for non-existing cpus, but wonder if the= extra data >> =A0> =A0> > also appeared on your 12 core box? >> =A0> =A0> >> =A0> =A0> I haven't removed anything, it's a pure copy&paste. >> =A0> >> =A0> Thanks. =A0I'll check the single-cpu case again after updating to 8= .2-R >> >> Ok, still a problem on at least my i386 single core Thinkpad T23 at >> 8.2-R, since 8.0 I think, certainly evident in a sysctl -a at 8.1-R >> >> FreeBSD t23.smithi.id.au 8.2-RELEASE FreeBSD 8.2-RELEASE #1: Thu Apr 14 >> 21:45:47 EST 2011 root@t23.smithi.id.au:/usr/obj/usr/src/sys/GENERIC i38= 6 >> >> Verbose dmesg: http://smithi.id.au/t23_dmesg_boot-v.8.2-R.txt >> sysctl -a: =A0 =A0 http://smithi.id.au/t23_sysctl-a_8.2-R.txt >> >> kern.ccpu: 0 >> =A0<cpu count=3D"1" mask=3D"0x1">0</cpu> >> kern.smp.forward_signal_enabled: 1 >> kern.smp.topology: 0 >> kern.smp.cpus: 1 >> kern.smp.disabled: 0 >> kern.smp.active: 0 >> kern.smp.maxcpus: 32 >> kern.smp.maxid: 31 =A0 =A0 =A0<<<<<<< >> hw.ncpu: 1 >> >> kern.cp_times: 38548 1 120437 195677 9660939 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 >> >> /usr/src/sys/kern/kern_clock.c: >> return SYSCTL_OUT(req, 0, sizeof(long) * CPUSTATES * (mp_maxid + 1)); >> >> Consumers of kern.cp_times like powerd, top, dtrace? and others have to >> loop over 32 cpus, all but one non-existent, and there seem to be many >> places in the kernel doing eg: for (cpu =3D 0; cpu <=3D mp_maxid; cpu++)= { >> and while CPU_FOREACH / CPU_ABSENT will skip over them, seems wasteful >> at best on machines least likely to have cycles to spare. >> >> eg: powerd parses kern.cp_times to count cpus, wasting cycles adding >> up the 31 'empty' cpus. =A0I haven't explored other userland consumers. >> >> Clearly kern.smp.maxid (ie mp_maxid) should be 0, not 31. =A0On i386, >> non-APIC i386 at least, mp_maxid is not set to (mp_ncpus - 1) as on some >> other archs .. after having being initialised to (MAXCPU - 1) in >> /sys/i386/i386/mp_machdep.c it's never updated for non-smp machines. >> >> I haven't chased all of these rabbits down all of their holes by any >> means, but it seems that making /sys/i386/i386/mp_machdep.c do what it >> says it's gonna do ('with an id of 0') should help. =A0Paste, tabs lost: >> >> int >> cpu_mp_probe(void) >> { >> =A0 =A0 =A0 =A0/* >> =A0 =A0 =A0 =A0 * Always record BSP in CPU map so that the mbuf init cod= e works >> =A0 =A0 =A0 =A0 * correctly. >> =A0 =A0 =A0 =A0 */ >> =A0 =A0 =A0 =A0all_cpus =3D 1; >> =A0 =A0 =A0 =A0if (mp_ncpus =3D=3D 0) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0/* >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * No CPUs were found, so this must be a = UP system. =A0Setup >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * the variables to represent a system wi= th a single CPU >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * with an id of 0. >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 */ >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0mp_ncpus =3D 1; >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 mp_maxid =3D 0; >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return (0); >> =A0 =A0 =A0 =A0} >> >> =A0 =A0 =A0 =A0/* At least one CPU was found. */ >> =A0 =A0 =A0 =A0if (mp_ncpus =3D=3D 1) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0/* >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * One CPU was found, so this must be a U= P system with >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 * an I/O APIC. >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 */ >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 mp_maxid =3D 0; >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return (0); >> =A0 =A0 =A0 =A0} >> >> =A0 =A0 =A0 =A0/* At least two CPUs were found. */ >> =A0 =A0 =A0 =A0return (1); >> } >> >> Note that the second added line above already exists in >> /sys/amd64/amd64/mp_machdep.c, maybe to fix a similar problem, though >> that should only apply to 'a UP system with an I/O APIC'. =A0Maybe bette= r >> could be to fix this in cpu_mp_probe's caller, /sys/kern/subr_smp.c: >> >> static void >> mp_start(void *dummy) >> { >> =A0 =A0 =A0 =A0mtx_init(&smp_ipi_mtx, "smp rendezvous", NULL, MTX_SPIN); >> >> =A0 =A0 =A0 =A0/* Probe for MP hardware. */ >> =A0 =A0 =A0 =A0if (smp_disabled !=3D 0 || cpu_mp_probe() =3D=3D 0) { >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0mp_ncpus =3D 1; >> + =A0 =A0 =A0 =A0 =A0 =A0 =A0 mp_maxid =3D 0; >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0all_cpus =3D PCPU_GET(cpumask); >> =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0return; >> =A0 =A0 =A0 =A0} >> >> =A0 =A0 =A0 =A0cpu_mp_start(); >> =A0 =A0 =A0 =A0printf("FreeBSD/SMP: Multiprocessor System Detected: %d C= PUs\n", >> =A0 =A0 =A0 =A0 =A0 =A0mp_ncpus); >> =A0 =A0 =A0 =A0cpu_mp_announce(); >> } >> >> I'm probably a long way off base for a solution, but think I've located >> the problem. =A0Thoughts? =A0Is this a known issue? =A0Might any develop= ers >> actually still have a single-cpu i386 system to check this on? :) >> >> Very happy to test any patches etc. >> > > Ouch. > Looks like that affects a system with 2 cores as well. > Intel Core2 E7200, 8.2-R i386 SMP: > > kern.smp.forward_signal_enabled: 1 > kern.smp.topology: 0 > kern.smp.cpus: 2 > kern.smp.disabled: 0 > kern.smp.active: 1 > kern.smp.maxcpus: 32 > kern.smp.maxid: 31 > > kern.cp_times: 867360 171 429180 70114 170549535 1385294 306 176659 82618 > 170270900 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 > 0 0 0 0 0 0 0 Perhaps that was so from the begging: the problem is seen also on 6.4 i386 SMP 8 core. Here it was just set to (MAXCPU-1). The buggy mp_maxid was fixed in HEAD with r215009, though not merged. The patch works on my 8.2 Core2 SMP i386 system. --=20 wbr, pluknet --002354471dd84ebfd304a15eb14c Content-Type: text/x-diff; charset=US-ASCII; name="mp_maxid.82.patch" Content-Disposition: attachment; filename="mp_maxid.82.patch" Content-Transfer-Encoding: base64 X-Attachment-Id: f_gmqnk58a0 SW5kZXg6IHg4Ni94ODYvbXB0YWJsZS5jCj09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIHg4Ni94ODYvbXB0YWJsZS5j CShyZXZpc2lvbiAyMTUwMDgpCisrKyB4ODYveDg2L21wdGFibGUuYwkocmV2aXNpb24gMjE1MDA5 KQpAQCAtMzg5LDcgKzM4OSw4IEBACiAKIAlhcGljX3JlZ2lzdGVyX2VudW1lcmF0b3IoJm1wdGFi bGVfZW51bWVyYXRvcik7CiB9Ci1TWVNJTklUKG1wdGFibGVfcmVnaXN0ZXIsIFNJX1NVQl9NUFRC TCwgU0lfT1JERVJfRklSU1QsIG1wdGFibGVfcmVnaXN0ZXIsIE5VTEwpOworU1lTSU5JVChtcHRh YmxlX3JlZ2lzdGVyLCBTSV9TVUJfVFVOQUJMRVMgLSAxLCBTSV9PUkRFUl9GSVJTVCwgbXB0YWJs ZV9yZWdpc3RlciwKKyAgICBOVUxMKTsKIAogLyoKICAqIENhbGwgdGhlIGhhbmRsZXIgcm91dGlu ZSBmb3IgZWFjaCBlbnRyeSBpbiB0aGUgTVAgY29uZmlnIHRhYmxlLgpJbmRleDogeDg2L3g4Ni9s b2NhbF9hcGljLmMKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PQotLS0geDg2L3g4Ni9sb2NhbF9hcGljLmMJKHJldmlzaW9u IDIxNTAwOCkKKysrIHg4Ni94ODYvbG9jYWxfYXBpYy5jCShyZXZpc2lvbiAyMTUwMDkpCkBAIC0x Mjg1LDcgKzEyODUsNyBAQAogCWlmIChyZXNvdXJjZV9kaXNhYmxlZCgiYXBpYyIsIDApKQogCQly ZXR1cm47CiAKLQkvKiBGaXJzdCwgcHJvYmUgYWxsIHRoZSBlbnVtZXJhdG9ycyB0byBmaW5kIHRo ZSBiZXN0IG1hdGNoLiAqLworCS8qIFByb2JlIGFsbCB0aGUgZW51bWVyYXRvcnMgdG8gZmluZCB0 aGUgYmVzdCBtYXRjaC4gKi8KIAliZXN0X2VudW0gPSBOVUxMOwogCWJlc3QgPSAwOwogCVNMSVNU X0ZPUkVBQ0goZW51bWVyYXRvciwgJmVudW1lcmF0b3JzLCBhcGljX25leHQpIHsKQEAgLTEzMjEs MTMgKzEzMjEsMTIgQEAKIAl9CiAjZW5kaWYKIAotCS8qIFNlY29uZCwgcHJvYmUgdGhlIENQVSdz IGluIHRoZSBzeXN0ZW0uICovCisJLyogUHJvYmUgdGhlIENQVSdzIGluIHRoZSBzeXN0ZW0uICov CiAJcmV0dmFsID0gYmVzdF9lbnVtLT5hcGljX3Byb2JlX2NwdXMoKTsKIAlpZiAocmV0dmFsICE9 IDApCiAJCXByaW50ZigiJXM6IEZhaWxlZCB0byBwcm9iZSBDUFVzOiByZXR1cm5lZCAlZFxuIiwK IAkJICAgIGJlc3RfZW51bS0+YXBpY19uYW1lLCByZXR2YWwpOwogCi0jaWZkZWYgX19hbWQ2NF9f CiB9CiBTWVNJTklUKGFwaWNfaW5pdCwgU0lfU1VCX1RVTkFCTEVTIC0gMSwgU0lfT1JERVJfU0VD T05ELCBhcGljX2luaXQsIE5VTEwpOwogCkBAIC0xMzQyLDE5ICsxMzQxLDE0IEBACiAgCiAJaWYg KGJlc3RfZW51bSA9PSBOVUxMKQogCQlyZXR1cm47Ci0jZW5kaWYKLQkvKiBUaGlyZCwgaW5pdGlh bGl6ZSB0aGUgbG9jYWwgQVBJQy4gKi8KKworCS8qIEluaXRpYWxpemUgdGhlIGxvY2FsIEFQSUMu ICovCiAJcmV0dmFsID0gYmVzdF9lbnVtLT5hcGljX3NldHVwX2xvY2FsKCk7CiAJaWYgKHJldHZh bCAhPSAwKQogCQlwcmludGYoIiVzOiBGYWlsZWQgdG8gc2V0dXAgdGhlIGxvY2FsIEFQSUM6IHJl dHVybmVkICVkXG4iLAogCQkgICAgYmVzdF9lbnVtLT5hcGljX25hbWUsIHJldHZhbCk7CiB9Ci0j aWZkZWYgX19hbWQ2NF9fCi1TWVNJTklUKGFwaWNfc2V0dXBfbG9jYWwsIFNJX1NVQl9DUFUsIFNJ X09SREVSX1NFQ09ORCwgYXBpY19zZXR1cF9sb2NhbCwKLSAgICBOVUxMKTsKLSNlbHNlCi1TWVNJ TklUKGFwaWNfaW5pdCwgU0lfU1VCX0NQVSwgU0lfT1JERVJfU0VDT05ELCBhcGljX2luaXQsIE5V TEwpOwotI2VuZGlmCitTWVNJTklUKGFwaWNfc2V0dXBfbG9jYWwsIFNJX1NVQl9DUFUsIFNJX09S REVSX1NFQ09ORCwgYXBpY19zZXR1cF9sb2NhbCwgTlVMTCk7CiAKIC8qCiAgKiBTZXR1cCB0aGUg SS9PIEFQSUNzLgpJbmRleDogaTM4Ni9hY3BpY2EvbWFkdC5jCj09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT0KLS0tIGkzODYv YWNwaWNhL21hZHQuYwkocmV2aXNpb24gMjE1MDA4KQorKysgaTM4Ni9hY3BpY2EvbWFkdC5jCShy ZXZpc2lvbiAyMTUwMDkpCkBAIC0yMDMsNyArMjAzLDcgQEAKIAogCWFwaWNfcmVnaXN0ZXJfZW51 bWVyYXRvcigmbWFkdF9lbnVtZXJhdG9yKTsKIH0KLVNZU0lOSVQobWFkdF9yZWdpc3RlciwgU0lf U1VCX0NQVSAtIDEsIFNJX09SREVSX1NFQ09ORCwgbWFkdF9yZWdpc3RlciwgTlVMTCk7CitTWVNJ TklUKG1hZHRfcmVnaXN0ZXIsIFNJX1NVQl9UVU5BQkxFUyAtIDEsIFNJX09SREVSX0ZJUlNULCBt YWR0X3JlZ2lzdGVyLCBOVUxMKTsKIAogLyoKICAqIENhbGwgdGhlIGhhbmRsZXIgcm91dGluZSBm b3IgZWFjaCBlbnRyeSBpbiB0aGUgTUFEVCB0YWJsZS4KSW5kZXg6IGkzODYvaTM4Ni9tcF9tYWNo ZGVwLmMKPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PQotLS0gaTM4Ni9pMzg2L21wX21hY2hkZXAuYwkocmV2aXNpb24gMjE1 MDA4KQorKysgaTM4Ni9pMzg2L21wX21hY2hkZXAuYwkocmV2aXNpb24gMjE1MDA5KQpAQCAtNDY1 LDggKzQ2NSwxMCBAQAogCQlib290X2NwdV9pZCA9IGFwaWNfaWQ7CiAJCWNwdV9pbmZvW2FwaWNf aWRdLmNwdV9ic3AgPSAxOwogCX0KLQlpZiAobXBfbmNwdXMgPCBNQVhDUFUpCisJaWYgKG1wX25j cHVzIDwgTUFYQ1BVKSB7CiAJCW1wX25jcHVzKys7CisJCW1wX21heGlkID0gbXBfbmNwdXMgLSAx OworCX0KIAlpZiAoYm9vdHZlcmJvc2UpCiAJCXByaW50ZigiU01QOiBBZGRlZCBDUFUgJWQgKCVz KVxuIiwgYXBpY19pZCwgYm9vdF9jcHUgPyAiQlNQIiA6CiAJCSAgICAiQVAiKTsKQEAgLTQ3Niw3 ICs0NzgsMTkgQEAKIGNwdV9tcF9zZXRtYXhpZCh2b2lkKQogewogCi0JbXBfbWF4aWQgPSBNQVhD UFUgLSAxOworCS8qCisJICogbXBfbWF4aWQgc2hvdWxkIGJlIGFscmVhZHkgc2V0IGJ5IGNhbGxz IHRvIGNwdV9hZGQoKS4KKwkgKiBKdXN0IHNhbml0eSBjaGVjayBpdHMgdmFsdWUgaGVyZS4KKwkg Ki8KKwlpZiAobXBfbmNwdXMgPT0gMCkKKwkJS0FTU0VSVChtcF9tYXhpZCA9PSAwLAorCQkgICAg KCIlczogbXBfbmNwdXMgaXMgemVybywgYnV0IG1wX21heGlkIGlzIG5vdCIsIF9fZnVuY19fKSk7 CisJZWxzZSBpZiAobXBfbmNwdXMgPT0gMSkKKwkJbXBfbWF4aWQgPSAwOworCWVsc2UKKwkJS0FT U0VSVChtcF9tYXhpZCA+PSBtcF9uY3B1cyAtIDEsCisJCSAgICAoIiVzOiBjb3VudGVycyBvdXQg b2Ygc3luYzogbWF4ICVkLCBjb3VudCAlZCIsIF9fZnVuY19fLAorCQkJbXBfbWF4aWQsIG1wX25j cHVzKSk7CiB9CiAKIGludApAQCAtNTA0LDYgKzUxOCw3IEBACiAJCSAqIE9uZSBDUFUgd2FzIGZv dW5kLCBzbyB0aGlzIG11c3QgYmUgYSBVUCBzeXN0ZW0gd2l0aAogCQkgKiBhbiBJL08gQVBJQy4K IAkJICovCisJCW1wX21heGlkID0gMDsKIAkJcmV0dXJuICgwKTsKIAl9CiAKSW5kZXg6IGkzODYv eGVuL21wdGFibGUuYwo9PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09 PT09PT09PT09PT09PT09PT09PT09PT09Ci0tLSBpMzg2L3hlbi9tcHRhYmxlLmMJKHJldmlzaW9u IDIxNTAwOCkKKysrIGkzODYveGVuL21wdGFibGUuYwkocmV2aXNpb24gMjE1MDA5KQpAQCAtMTA5 LDcgKzEwOSw3IEBACiAKIAlhcGljX3JlZ2lzdGVyX2VudW1lcmF0b3IoJm1wdGFibGVfZW51bWVy YXRvcik7CiB9Ci1TWVNJTklUKG1wdGFibGVfcmVnaXN0ZXIsIFNJX1NVQl9DUFUgLSAxLCBTSV9P UkRFUl9GSVJTVCwgbXB0YWJsZV9yZWdpc3RlciwKK1NZU0lOSVQobXB0YWJsZV9yZWdpc3Rlciwg U0lfU1VCX1RVTkFCTEVTIC0gMSwgU0lfT1JERVJfRklSU1QsIG1wdGFibGVfcmVnaXN0ZXIsCiAg ICAgTlVMTCk7CiAKIAo= --002354471dd84ebfd304a15eb14c--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?BANLkTimUCDg7-t9DRubh8LHDdcQv-8Gr0Q>