Date: Wed, 16 Oct 2013 09:00:10 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: Alan Cox <alc@rice.edu> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>, alc@freebsd.org Subject: Re: 9.2 + ZFS + i386 = panic: pmap_enter: attempted pmap_enter on 4MB page Message-ID: <20131016060010.GO3865@kib.kiev.ua> In-Reply-To: <525D7784.5000808@rice.edu> References: <CABXB=RQd_yT%2BsEA0qBnyCK-3ZsvxfSiSCcMVjnHoSyCB%2BZ5%2BRg@mail.gmail.com> <20131015164537.GH3865@kib.kiev.ua> <525D7784.5000808@rice.edu>
next in thread | previous in thread | raw e-mail | index | archive | help
--MmQIYbZiCoQ2kDro Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Oct 15, 2013 at 12:12:36PM -0500, Alan Cox wrote: > On 10/15/2013 11:45, Konstantin Belousov wrote: > > On Tue, Oct 15, 2013 at 11:17:53AM -0400, J David wrote: > >> What are the necessary loader.conf / kernel config invocations to make > >> ZFS stable on 9.2-RELEASE i386 node with 3GB RAM? > >> > >> This machine was rock solid under 8.4, but since upgrading to 9.2 it > >> has been a disaster. It crashes every few hours with "panic: > >> pmap_enter: attempted pmap_enter on 4MB page." > >> > >> Here are a couple of stack traces: > >> > >> panic: pmap_enter: attempted pmap_enter on 4MB page > >> cpuid =3D 0 > >> KDB: stack backtrace: > >> #0 0xc0afc092 at kdb_backtrace+0x52 > >> #1 0xc0ac249c at panic+0x1bc > >> #2 0xc0f1825d at pmap_enter+0x63d > >> #3 0xc0d34ee5 at vm_fault_hold+0x1c45 > >> #4 0xc0d33262 at vm_fault+0x82 > >> #5 0xc0f1f5d6 at trap_pfault+0x186 > >> #6 0xc0f1ed4b at trap+0x51b > >> #7 0xc0f0887c at calltrap+0x6 > >> #8 0xc16770b6 at zio_execute+0x116 > >> #9 0xc15d9460 at taskq_run_safe+0x10 > >> #10 0xc0b08f26 at taskqueue_run_locked+0xe6 > >> #11 0xc0b097f7 at taskqueue_thread_loop+0xb7 > >> #12 0xc0a90a73 at fork_exit+0xa3 > >> #13 0xc0f08924 at fork_trampoline+0x8 > >> > >> panic: pmap_enter: attempted pmap_enter on 4MB page > >> cpuid =3D 1 > >> KDB: stack backtrace: > >> #0 0xc0afc092 at kdb_backtrace+0x52 > >> #1 0xc0ac249c at panic+0x1bc > >> #2 0xc0f1825d at pmap_enter+0x63d > >> #3 0xc0d34ee5 at vm_fault_hold+0x1c45 > >> #4 0xc0d33262 at vm_fault+0x82 > >> #5 0xc0f1f5d6 at trap_pfault+0x186 > >> #6 0xc0f1ed4b at trap+0x51b > >> #7 0xc0f0887c at calltrap+0x6 > >> #8 0xc1679bdf at zio_dva_allocate+0x9f > >> #9 0xc16770b6 at zio_execute+0x116 > >> #10 0xc15d9460 at taskq_run_safe+0x10 > >> #11 0xc0b08f26 at taskqueue_run_locked+0xe6 > >> #12 0xc0b097f7 at taskqueue_thread_loop+0xb7 > >> #13 0xc0a90a73 at fork_exit+0xa3 > >> #14 0xc0f08924 at fork_trampoline+0x8 > >> > >> For the last crash, top was running and this is what was on the screen > >> when it died: > >> > >> CPU: 0.6% user, 0.0% nice, 2.2% system, 0.2% interrupt, 97.1% idle > >> Mem: 442M Active, 91M Inact, 79M Wired, 3764K Cache, 4960K Buf, 2380M = Free > >> ARC: 40M Total, 8520K MFU, 30M MRU, 304K Anon, 494K Header, 1062K Other > >> Swap: 4096M Total, 4096M Free > >> Write failed: Broken pipe > >> > >> It doesn't seem to matter what KVA_PAGES, vm.kmem_size, > >> vfs.zfs.arc_max or vfs.zfs.vdev.cache.size is set to, and the ZFS > >> tuning guides in the wiki (albeit appearing to be dating to the 7.x > >> era) provides guidelines for tuning down to 768M. So 3GB should be > >> enough for a machine that is 99% idle. (Particularly given that it > >> dies with >2GB free.) > >> > >> This seems to be specific to i386, this problem hasn't cropped up on > >> any amd64 nodes. No compression, deduplication, snapshots or anything > >> like that is in use. > > It seems that i386 pmap_enter() cannot deal with the superpages, while > > amd64 can. On the other hand, I am not quite sure that this is your > > problem, because you get traps on accessing KVA, and I suspect that > > ZFS uses wired memory. Obtain the kernel dump and do full backtrace > > for the panic. >=20 > Try MFCing r253949. I'm not confident that r253949 is the fix for this > particular problem, but it should be MFCed. Probably not. In addition, please note that the reporter has i386. >=20 > > Anyway, I believe that i386 pmap_enter() should do a demotion when need= ed. >=20 > A kernel space pmap_enter should never be overwriting an existing > superpage mapping. Yes, this is why I requested the backtrace. Still, I also think that demotion is missed in pmap_enter() on i386. >=20 > > Below is the prototyped change for this. > > > > diff --git a/sys/i386/i386/pmap.c b/sys/i386/i386/pmap.c > > index 64bf1a3..91453da 100644 > > --- a/sys/i386/i386/pmap.c > > +++ b/sys/i386/i386/pmap.c > > @@ -3473,17 +3473,21 @@ pmap_enter(pmap_t pmap, vm_offset_t va, vm_prot= _t access, vm_page_t m, > > PMAP_LOCK(pmap); > > sched_pin(); > > =20 > > - /* > > - * In the case that a page table page is not > > - * resident, we are creating it here. > > - */ > > - if (va < VM_MAXUSER_ADDRESS) { > > + pde =3D pmap_pde(pmap, va); > > + if ((*pde & PG_PS) !=3D 0) { > > + /* PG_V is asserted by pmap_demote_pde */ > > + pmap_demote_pde(pmap, pde, va); > > + if (va < VM_MAXUSER_ADDRESS) { > > + mpte =3D PHYS_TO_VM_PAGE(*pde & PG_FRAME); > > + mpte->wire_count++; > > + } > > + } else if (va < VM_MAXUSER_ADDRESS) { > > + /* > > + * In the case that a page table page is not resident, > > + * we are creating it here. > > + */ > > mpte =3D pmap_allocpte(pmap, va, M_WAITOK); > > } > > - > > - pde =3D pmap_pde(pmap, va); > > - if ((*pde & PG_PS) !=3D 0) > > - panic("pmap_enter: attempted pmap_enter on 4MB page"); > > pte =3D pmap_pte_quick(pmap, va); > > =20 > > /* --MmQIYbZiCoQ2kDro Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (FreeBSD) iQIcBAEBAgAGBQJSXitpAAoJEJDCuSvBvK1B0HEQAIxJilHSStiud8mlKHWYjFdt idgzYT3pOSdmaYTUNSyi5j9SG8PZ2vU+y5ROrxwD5kZfDyA2IXbF/8jqOU/hTzJd gjE3gZnHTlc0eEaFmyy5MmXgrKnAgEcPwk+IN+7kBjzldnHpC1m0AKHDUcIpZiUd Pih3gIFXHF/234tw6iXw3DWhPvIwjiKoZrTVBg6/CqFXytnN3MZNj3WNn0mcL4V6 OigUH3+vDjbEUYD/Q3PQI9Fixu5WyEsk8BWvP6GvvTsh8FBE7Ztlzm4R505L41TW LoVHryke3web8rlFtSl7Jd6NDn+5XCBIEWFw79loZ73uiqAb6JXK/31LH5wGvP9v 57CMBIbDgn89G4s6vt3lP2gOl3UdneIa2xT0lx8cuEWaO+YmQFLw+C1M76IXP1U2 D9Pbch5u1lvETV+ODne6DMX/jg5poRWoebODhyW1R8F9xx7Bll9G2a7Y6niZbog9 il6B8+XVWouqEMW3R5lhOu5cg/4kdsrp3Ddw/F2fv9ezRgNTcHIH3GhADfio58u+ x3zEtaUYujnF8+T416U2xtWlLuXbViTw+KRoMqfX7cz3CHY8wL6NWJYgNeE9YNB4 OxYDQJsKGnGA8JBpRdhsShAcBmaeu41ny3KHv8VggiYDJ0m2IF6HhmTh7a5yKPTy bOUJU1poHgv7taDv3bcJ =7P+/ -----END PGP SIGNATURE----- --MmQIYbZiCoQ2kDro--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20131016060010.GO3865>