From owner-freebsd-hackers@FreeBSD.ORG Fri Mar 26 20:14:29 2010 Return-Path: Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id ACEBD106566B for ; Fri, 26 Mar 2010 20:14:29 +0000 (UTC) (envelope-from tom@tomjudge.com) Received: from tomjudge.vm.bytemark.co.uk (tomjudge.vm.bytemark.co.uk [80.68.91.100]) by mx1.freebsd.org (Postfix) with ESMTP id 2D7C78FC12 for ; Fri, 26 Mar 2010 20:14:28 +0000 (UTC) Received: from localhost (localhost.localdomain [127.0.0.1]) by tomjudge.vm.bytemark.co.uk (Postfix) with ESMTP id 067A2489D5; Fri, 26 Mar 2010 20:14:28 +0000 (GMT) X-Virus-Scanned: Debian amavisd-new at tomjudge.vm.bytemark.co.uk Received: from tomjudge.vm.bytemark.co.uk ([127.0.0.1]) by localhost (tomjudge.vm.bytemark.co.uk [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Lp+b3b89wDML; Fri, 26 Mar 2010 20:14:25 +0000 (GMT) Received: from rita.nodomain (unknown [192.168.205.6]) by tomjudge.vm.bytemark.co.uk (Postfix) with ESMTP id 0646E489D3; Fri, 26 Mar 2010 20:14:23 +0000 (GMT) Message-ID: <4BAD1498.5040402@tomjudge.com> Date: Fri, 26 Mar 2010 20:10:00 +0000 From: Tom Judge User-Agent: Thunderbird 2.0.0.23 (X11/20090822) MIME-Version: 1.0 To: Kostik Belousov References: <4BACF92E.60600@tomjudge.com> <20100326195659.GU2415@deviant.kiev.zoral.com.ua> In-Reply-To: <20100326195659.GU2415@deviant.kiev.zoral.com.ua> X-Enigmail-Version: 0.96.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-hackers@freebsd.org Subject: Re: Panic in vm_map_stack X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 26 Mar 2010 20:14:29 -0000 -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Interesting this is in some custom code to calculate and track memory usage in jails. #0 sched_switch (td=0xffffff0004325370, newtd=0xffffffff804aeed8, flags=Variable "flags" is not available. ) at /usr/src/sys/kern/sched_ule.c:1944 #1 0x0000000000000000 in ?? () #2 0xffffffff804960a1 in mi_switch (flags=70407024, newtd=0x1) at /usr/src/sys/kern/kern_synch.c:440 #3 0xffffffff804c24a0 in sleepq_timedwait (wchan=0xffffffff80a09748) at /usr/src/sys/kern/subr_sleepqueue.c:615 #4 0xffffffff8049663d in _sleep (ident=0xffffffff80a09748, lock=0x0, priority=83314928, wmesg=0xffffffff807c4219 "vmo_de", timo=1) at /usr/src/sys/kern/kern_synch.c:222 #5 0xffffffff8064b5d4 in vm_object_deallocate (object=0xffffff07335f8c30) at /usr/src/sys/vm/vm_object.c:512 #6 0xffffffff8064440e in vm_map_delete (map=0xffffff007f8191a0, start=18446742980708383584, end=140737488355328) at /usr/src/sys/vm/vm_map.c:2315 #7 0xffffffff806445df in vm_map_remove (map=0xffffff007f8191a0, start=0, end=140737488355328) at /usr/src/sys/vm/vm_map.c:2423 #8 0xffffffff80644bb9 in vmspace_free (vm=0xffffff007f8191a0) at /usr/src/sys/vm/vm_map.c:324 #9 0xffffffff80470403 in prison_memory (pr=0xffffff0004fa2000) at /usr/src/sys/kern/kern_jail.c:756 #10 0xffffffff804705f8 in jpager_td (arg=Variable "arg" is not available. ) at /usr/src/sys/kern/kern_jail.c:192 #11 0xffffffff8046ad93 in fork_exit (callout=0xffffffff804705a0 , arg=0xffffff0004fa2000, frame=0xffffffffb8986c80) - ---Type to continue, or q to quit--- at /usr/src/sys/kern/kern_fork.c:804 #12 0xffffffff806be49e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:455 This is the function, I am guessing that I need to unlock the proc before calling vmspace_free ? 673 /* Given credential, return memory usage in bytes. */ 674 void 675 prison_memory(struct prison *pr) 676 { 677 struct proc *p; 678 struct thread *td; 679 struct vmspace *vm; 680 long mem_used = 0; 681 long full_mem_used = 0; 682 long proc_res = 0; 683 684 /* 685 * TODO: this is a really bad way of doing the 686 * search, as we end up going across all processes 687 * for each jail. It'd be more efficient to just do 688 * this once in a period and update the relevant jail. 689 * 690 */ 691 sx_slock(&allproc_lock); 692 693 FOREACH_PROC_IN_SYSTEM(p) { 694 int breakout; 695 proc_res=0; 696 vm = NULL; 697 if (PROC_TRYLOCK(p) == 0) 698 continue; 699 /* 700 * If this is a system or protected process, skip it. 701 */ 702 if ((p->p_flag & P_SYSTEM) || (p->p_pid == 1) || 703 (p->p_flag & P_PROTECTED) || 704 (p->p_pid < 48)) { 705 PROC_UNLOCK(p); 706 continue; 707 } 708 /* 709 * If the process is in a non-running type state, 710 * don't touch it. Check all the threads individually. 711 */ 712 breakout = 0; 713 FOREACH_THREAD_IN_PROC(p, td) { 714 thread_lock(td); 715 if (!TD_ON_RUNQ(td) && 716 !TD_IS_RUNNING(td) && 717 !TD_IS_SLEEPING(td)) { 718 thread_unlock(td); 719 breakout = 1; 720 break; 721 } 722 thread_unlock(td); 723 } 724 if (breakout) { 725 PROC_UNLOCK(p); 726 continue; 727 } 728 729 if (p->p_state == PRS_NEW || 730 p->p_state == PRS_ZOMBIE || 731 !jailed(p->p_ucred) || 732 (pr != p->p_ucred->cr_prison) || 733 !p->p_vmspace) { 734 PROC_UNLOCK(p); 735 continue; 736 } 737 /* 738 * get the process size 739 */ 740 vm = vmspace_acquire_ref(p); 741 if (vm == NULL) { 742 PROC_UNLOCK(p); 743 continue; 744 } 745 746 if (!vm_map_trylock_read(&vm->vm_map)) { 747 vmspace_free(vm); 748 PROC_UNLOCK(p); 749 continue; 750 } 751 full_mem_used += vmspace_swap_count(vm); 752 vm_map_unlock_read(&vm->vm_map); 753 proc_res = vmspace_resident_count(vm); 754 full_mem_used += proc_res; 755 mem_used += proc_res; 756 vmspace_free(vm); 757 PROC_UNLOCK(p); 758 } 759 sx_sunlock(&allproc_lock); 760 761 mem_used *= PAGE_SIZE; 762 full_mem_used *= PAGE_SIZE; 763 /* Copy the current memory usage to the prison struct */ 764 mtx_lock(&pr->pr_mtx); 765 pr->pr_mem_usage = mem_used; 766 pr->pr_full_mem_usage = full_mem_used; 767 mtx_unlock(&pr->pr_mtx); 768 } 769 Tom Kostik Belousov wrote: > On Fri, Mar 26, 2010 at 06:13:02PM +0000, Tom Judge wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> Hi, >> >> I am seeing the following rare but re occurring panic in the >> vm_map_stack code. >> >> I have the core file and symbols available should more information be >> required. >> >> Tom >> >> - ---- >> >> >>> $ uname -a >> FreeBSD XXX 7.1-RELEASE-p4 FreeBSD 7.1-RELEASE-p4 #0 @718:841: Mon Feb >> 8 20:14:39 UTC 2010 tj@XXX:/usr/obj/usr/src/sys/XXXv8 amd64 >> >> >>> $ kgdb /boot/kernel/kernel /var/crash/vmcore.0 >> GNU gdb 6.1.1 [FreeBSD] >> Copyright 2004 Free Software Foundation, Inc. >> GDB is free software, covered by the GNU General Public License, and you are >> welcome to change it and/or distribute copies of it under certain >> conditions. >> Type "show copying" to see the conditions. >> There is absolutely no warranty for GDB. Type "show warranty" for details. >> This GDB was configured as "amd64-marcel-freebsd"... >> >> Unread portion of the kernel message buffer: >> Sleeping thread (tid 100119, pid 1554) owns a non-sleepable lock >> panic: sleeping thread >> cpuid = 0 >> Uptime: 44d17h36m22s >> Physical memory: 32746 MB >> Dumping 1467 MB: 1452 1436 1420 1404 1388 1372 1356 1340 1324 1308 1292 >> 1276 1260 1244 1228 1212 1196 1180 1164 1148 1132 1116 1100 1084 1068 >> 1052 1036 1020 1004 988 972 956 940 924 908 892 876 860 844 828 812 796 >> 780 764 748 732 716 700 684 668 652 636 620 604 588 572 556 540 524 508 >> 492 476 460 444 428 412 396 380 364 348 332 316 300 284 268 252 236 220 >> 204 188 172 156 140 124 108 92 76 60 44 28 12 >> >> #0 doadump () at pcpu.h:195 >> 195 pcpu.h: No such file or directory. >> in pcpu.h >> (kgdb) bt >> #0 doadump () at pcpu.h:195 >> #1 0x0000000000000004 in ?? () >> #2 0xffffffff8048e2a9 in boot (howto=260) at >> /usr/src/sys/kern/kern_shutdown.c:418 >> #3 0xffffffff8048e6b2 in panic (fmt=0x104
> bounds>) at /usr/src/sys/kern/kern_shutdown.c:574 >> #4 0xffffffff804c4a95 in propagate_priority (td=Variable "td" is not >> available. >> ) at /usr/src/sys/kern/subr_turnstile.c:222 >> #5 0xffffffff804c58a5 in turnstile_wait (ts=Variable "ts" is not available. >> ) at /usr/src/sys/kern/subr_turnstile.c:740 >> #6 0xffffffff804816af in _mtx_lock_sleep (m=0xffffff0654567118, >> tid=18446742976254672896, opts=Variable "opts" is not available. >> ) at /usr/src/sys/kern/kern_mutex.c:420 >> #7 0xffffffff80646057 in vm_map_stack (map=0xffffff000447cd00, >> addrbos=140736951484416, max_ssize=536870912, prot=7 '\a', max=7 '\a', >> cow=4096) at /usr/src/sys/vm/vm_map.c:2722 >> #8 0xffffffff80467115 in exec_new_vmspace (imgp=0xffffffffb92b4a10, >> sv=0xffffffff809b0a40) at /usr/src/sys/kern/kern_exec.c:971 >> #9 0xffffffff8044e534 in exec_elf64_imgact (imgp=0xffffffffb92b4a10) at >> /usr/src/sys/kern/imgact_elf.c:680 >> #10 0xffffffff80467532 in kern_execve (td=0xffffff007a978000, >> args=0xffffffffb92b4b00, mac_p=Variable "mac_p" is not available. >> ) at /usr/src/sys/kern/kern_exec.c:432 >> #11 0xffffffff80468477 in execve (td=0xffffff007a978000, uap=Variable >> "uap" is not available. >> ) at /usr/src/sys/kern/kern_exec.c:201 >> #12 0xffffffff806d8217 in syscall (frame=0xffffffffb92b4c80) at >> /usr/src/sys/amd64/amd64/trap.c:907 >> #13 0xffffffff806be2db in Xfast_syscall () at >> /usr/src/sys/amd64/amd64/exception.S:330 >> #14 0x000000080091bcec in ?? () >> Previous frame inner to this frame (corrupt stack?) >> (kgdb) > > Switch to the thread 100119, and get a backtrace for it. > You can look up the kgdb thread number by kernel tid by > issuing "threads" command. - -- TJU13-ARIN -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.13 (FreeBSD) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQEcBAEBAgAGBQJLrRSYAAoJEMSwVS7lr0OdAjUH/2pOJl0r7sl79+rDgABt9QLw IF+geLoM0Htb71xf8FJ6xXZ4rarclrzCx4I1ZJCgmkTE44EKx9929/e0pqDwGbZ+ O+lj+zYs3wrppo8u/GGicJYYyhMblIpVfr/OVvIEtC6otTFwo+scyeXg7VHL2qqG HaMakCZwR65P1ydbTqz7aSWKay4l1SNq8uoeeaaE6ujVXmUEWYC231IsoirR8L4L E5teJBlLoilK2rJN0vGvECMXiuvuT4hjraTJXOO99Tzge3BP663We3LI+o4+4Tam eD6D2cOT2PdrTbvfzLZWCq+EAwolk8MKCZqGLzz5dkFtEIu4Q/JToOu4UEj9GQU= =//on -----END PGP SIGNATURE-----