From owner-freebsd-hackers@freebsd.org Sat Jan 27 13:50:39 2018 Return-Path: Delivered-To: freebsd-hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D87A0EC27A8 for ; Sat, 27 Jan 2018 13:50:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5D22A69A9C for ; Sat, 27 Jan 2018 13:50:38 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: by mailman.ysv.freebsd.org (Postfix) id 1DC67EC27A6; Sat, 27 Jan 2018 13:50:38 +0000 (UTC) Delivered-To: hackers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EE08CEC27A5 for ; Sat, 27 Jan 2018 13:50:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 727D469A98 for ; Sat, 27 Jan 2018 13:50:37 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id w0RDoDcs038530 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Sat, 27 Jan 2018 15:50:16 +0200 (EET) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua w0RDoDcs038530 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id w0RDoDKA038528; Sat, 27 Jan 2018 15:50:13 +0200 (EET) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sat, 27 Jan 2018 15:50:13 +0200 From: Konstantin Belousov To: Hartmut Brandt Cc: hackers@freebsd.org Subject: Re: allocating large piece of contiguous kernel memory Message-ID: <20180127135013.GA55707@kib.kiev.ua> References: <20180125172116.GS55707@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.9.2 (2017-12-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 27 Jan 2018 13:50:39 -0000 On Fri, Jan 26, 2018 at 10:05:14AM +0100, Hartmut Brandt wrote: > Hi, > > On Thu, 25 Jan 2018, Konstantin Belousov wrote: > > KB>On Thu, Jan 25, 2018 at 05:52:57PM +0100, Hartmut Brandt wrote: > KB>> Hi all, > KB>> > KB>> for a device driver communicating with an FPGA I would like to allocate > KB>> an 8GByte piece or two 4GByte pieces of contiguous memory in the device > KB>> driver. The machine is amd64 and has 256GByte. > KB>> > KB>> The maximum I can allocate is 512MByte. If I try more (in one piece or in > KB>> two pieces) contigfree() crashes in pmap_resident_count_dec() with an > KB>> "pmap resident count underflow". > KB>> > KB>> Is this something which is not supposed to work? Or is there something > KB>> tunable or is this a bug? > KB> > KB>I suspect a bug. Print out the value of > KB>kernel_pmap->pm_stats.resident_count before and after contigmalloc. Also > KB>interesting are the values of the resident count and decrement at the > KB>panic time (but they are already printed by the panic, you did not shown > KB>them). > > This is the panic: > > knot_attach: attaching > kernel_pmap->pm_stats.resident_count=57322 > cmem=0xfffffe0101000000 > kernel_pmap->pm_stats.resident_count=57712 > kernel_pmap->pm_stats.resident_count=57712 > panic: pmap 0xffffffff80e40188 resident count underflow 368 512 > cpuid = 3 > time = 1516956458 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0100d0d1c0 > vpanic() at vpanic+0x19c/frame 0xfffffe0100d0d240 > kassert_panic() at kassert_panic+0x126/frame 0xfffffe0100d0d2b0 > pmap_remove_pde() at pmap_remove_pde+0x645/frame 0xfffffe0100d0d340 > pmap_remove() at pmap_remove+0x484/frame 0xfffffe0100d0d3c0 > kmem_unback() at kmem_unback+0x3a/frame 0xfffffe0100d0d400 > kmem_free() at kmem_free+0x3d/frame 0xfffffe0100d0d430 > contigfree() at contigfree+0x27/frame 0xfffffe0100d0d460 > knot_attach() at knot_attach+0xec/frame 0xfffffe0100d0d4c0 > device_attach() at device_attach+0x3f7/frame 0xfffffe0100d0d510 > pci_driver_added() at pci_driver_added+0xe9/frame 0xfffffe0100d0d550 > devclass_driver_added() at devclass_driver_added+0x7d/frame 0xfffffe0100d0d590 > devclass_add_driver() at devclass_add_driver+0x144/frame 0xfffffe0100d0d5d0 > module_register_init() at module_register_init+0xc0/frame 0xfffffe0100d0d600 > linker_load_module() at linker_load_module+0xb78/frame 0xfffffe0100d0d910 > kern_kldload() at kern_kldload+0xa9/frame 0xfffffe0100d0d950 > sys_kldload() at sys_kldload+0x5b/frame 0xfffffe0100d0d980 > amd64_syscall() at amd64_syscall+0x271/frame 0xfffffe0100d0dab0 > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe0100d0dab0 > --- syscall (304, FreeBSD ELF64, sys_kldload), rip = 0x80087228a, rsp = > 0x7fffffffd508, rbp = 0x7fffffffda80 --- > KDB: enter: panic > [ thread pid 1150 tid 100146 ] > Stopped at kdb_enter+0x3b: movq $0,kdb_why > db> > > The first printf() is just before contigmalloc(), then it prints the > returned pointer. The next to prints() are just before the contigfree(). > > Doesn't the change of the count by 390 look strange when allocating > 8GByte? I cannot reproduce your issue. I suspect this is something local to your system. With the patch below, if I do # sysctl debug.a=1 I see # sysctl debug.a=1 debug.a: before 31310 0after alloc 2128462 after free 31310 -> 0 The numbers are exactly as what I expect. diff --git a/sys/amd64/amd64/pmap.c b/sys/amd64/amd64/pmap.c index 12c5b695e23..3c707444169 100644 --- a/sys/amd64/amd64/pmap.c +++ b/sys/amd64/amd64/pmap.c @@ -8040,3 +8040,26 @@ DB_SHOW_COMMAND(phys2dmap, pmap_phys2dmap) } } #endif + +static int +pmap_contigalloc(SYSCTL_HANDLER_ARGS) +{ + void *ptr; + size_t sz; + int error, i; + + i = 0; + error = sysctl_handle_int(oidp, &i, 0, req); + if (error != 0 || req->newptr == NULL) + return (error); + sz = 8UL * 1024 * 1024 * 1024; + printf("before %ld\n", kernel_pmap->pm_stats.resident_count); + ptr = contigmalloc(sz, M_TEMP, M_WAITOK, 0, ~0UL, 0, 0); + printf("after alloc %ld\n", kernel_pmap->pm_stats.resident_count); + contigfree(ptr, sz, M_TEMP); + printf("after free %ld\n", kernel_pmap->pm_stats.resident_count); + return (0); +} +SYSCTL_PROC(_debug, OID_AUTO, a, CTLTYPE_INT | CTLFLAG_RW | + CTLFLAG_MPSAFE, NULL, 0, pmap_contigalloc, "I", + "");