From owner-freebsd-threads@FreeBSD.ORG Mon Sep 11 08:00:51 2006 Return-Path: X-Original-To: freebsd-threads@hub.freebsd.org Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C574A16A620 for ; Mon, 11 Sep 2006 08:00:51 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id A1F7243D6A for ; Mon, 11 Sep 2006 08:00:45 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id k8B80isG041872 for ; Mon, 11 Sep 2006 08:00:44 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id k8B80ic9041871; Mon, 11 Sep 2006 08:00:44 GMT (envelope-from gnats) Resent-Date: Mon, 11 Sep 2006 08:00:44 GMT Resent-Message-Id: <200609110800.k8B80ic9041871@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-threads@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Anders Nordby Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EB98016A412 for ; Mon, 11 Sep 2006 07:54:34 +0000 (UTC) (envelope-from anders@fupp.net) Received: from fupp.net (totem.fix.no [80.91.36.20]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4AC5043D45 for ; Mon, 11 Sep 2006 07:54:34 +0000 (GMT) (envelope-from anders@fupp.net) Received: from localhost (totem.fix.no [80.91.36.20]) by fupp.net (Postfix) with ESMTP id A357B8D9888; Mon, 11 Sep 2006 09:54:32 +0200 (CEST) Received: from fupp.net ([80.91.36.20]) by localhost (totem.fix.no [80.91.36.20]) (amavisd-new, port 10024) with LMTP id 50875-01-8; Mon, 11 Sep 2006 09:54:31 +0200 (CEST) Received: by fupp.net (Postfix, from userid 1000) id D12758D9874; Mon, 11 Sep 2006 09:54:31 +0200 (CEST) Message-Id: <20060911075431.D12758D9874@fupp.net> Date: Mon, 11 Sep 2006 09:54:31 +0200 (CEST) From: Anders Nordby To: FreeBSD-gnats-submit@FreeBSD.org X-Send-Pr-Version: 3.113 Cc: tmseck@netcologne.de, adrian@creative.net.au Subject: threads/103127: Kernel panic while using thread features in Squid 2.6 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Anders Nordby List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2006 08:00:52 -0000 >Number: 103127 >Category: threads >Synopsis: Kernel panic while using thread features in Squid 2.6 >Confidential: no >Severity: serious >Priority: medium >Responsible: freebsd-threads >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Mon Sep 11 08:00:44 GMT 2006 >Closed-Date: >Last-Modified: >Originator: Anders Nordby >Release: FreeBSD 6.1-RELEASE-p5 i386 >Organization: - >Environment: System: FreeBSD cache3.foobar.no 6.1-RELEASE-p5 FreeBSD 6.1-RELEASE-p5 #0: Thu Sep 7 15:53:24 CEST 2006 root@cache3.foobar.no:/usr/obj/usr/src/sys/CACHE i386 Squid Cache: Version 2.6.STABLE3 configure options: '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid' '--localstatedir=/usr/local/squid' '--sysconfdir=/usr/local/etc/squid' '--enable-removal-policies=lru heap' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-epoll' '--enable-auth=basic ntlm digest' '--enable-basic-auth-helpers=NCSA PAM MSNT SMB YP' '--enable-digest-auth-helpers=password' '--enable-external-acl-helpers=ip_user session unix_group wbinfo_group' '--enable-ntlm-auth-helpers=SMB' '--enable-async-io' '--with-pthreads' '--enable-storeio=ufs diskd null aufs' '--enable-kqueue' '--enable-err-languages=Azerbaijani Bulgarian Catalan Czech Danish Dutch English Estonian Finnish French German Greek Hebrew Hungarian Italian Japanese Korean Lithuanian Polish Portuguese Romanian Russian-1251 Russian-koi8-r Serbian Simplify_Chinese Slovak Spanish Swedish Traditional_Chinese Turkish' '--enable-default-err-language=E nglish' '--enable-dlmalloc' '--enable-snmp' '--enable-async-io=50' '--prefix=/usr/local' 'i386-portbld-freebsd6.1' 'LDFLAGS=' 'CFLAGS=-O2 -fno-strict-aliasing -pipe ' 'CPPFLAGS=' 'host_alias=i386-portbld-freebsd6.1' 'build_alias=i386-portbld-freebsd6.1' 'target_alias=i386-portbld-freebsd6.1' 'CC=cc' >Description: Recently I've been trying to switch to threads-based scheduling of disk I/O requests, that is to use the aufs cache store type. It seems fast, by using it Squid doesn't block on disk I/O so easily when under load. Now and then I get a kernel panic, however: root@cache3:/usr/src# kgdb /usr/obj/usr/src/sys/CACHE/kernel.debug /var/crash/vmcore.0 [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"] GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "i386-marcel-freebsd". Unread portion of the kernel message buffer: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 06 fault virtual address = 0x0 fault code = supervisor read, page not present instruction pointer = 0x20:0x0 stack pointer = 0x28:0xeb9a7b98 frame pointer = 0x28:0xeb9a7bcc code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 542 (squid) trap number = 12 panic: page fault cpuid = 0 Uptime: 2d4h20m9s Dumping 3839 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 3839MB (982778 pages) 3824 3808 3792 3776 3760 3744 3728 3712 3696 3680 3664 3648 3632 3616 3600 3584 3568 3552 3536 3520 3504 3488 3472 3456 3440 3424 3408 3392 3376 3360 3344 3328 3312 3296 3280 3264 3248 3232 3216 3200 3184 3168 3152 3136 3120 3104 3088 3072 3056 3040 3024 3008 2992 2976 2960 2944 2928 2912 2896 2880 2864 2848 2832 2816 2800 2784 2768 2752 2736 2720 2704 2688 2672 2656 2640 2624 2608 2592 2576 2560 2544 2528 2512 2496 2480 2464 2448 2432 2416 2400 2384 2368 2352 2336 2320 2304 2288 2272 2256 2240 2224 2208 2192 2176 2160 2144 2128 2112 2096 2080 2064 2048 2032 2016 2000 1984 1968 1952 1936 1920 1904 1888 1872 1856 1840 1824 1808 1792 1776 1760 1744 1728 1712 1696 1680 1664 1648 1632 1616 1600 1584 1568 1552 1536 1520 1504 1488 1472 1456 1440 1424 1408 1392 1376 1360 1344 1328 1312 1296 1280 1264 1248 1232 1216 1200 1184 1168 1152 1136 1120 1104 1088 1072 1056 1040 1024 1008 992 976 960 944 928 912 896 880 864 848 832 816 800 784 768 752 736 720 704 688 672 656 640 624 608 592 576 560 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16 #0 doadump () at pcpu.h:165 165 pcpu.h: No such file or directory. in pcpu.h (kgdb) bt #0 doadump () at pcpu.h:165 #1 0xc0672901 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:402 #2 0xc0672c59 in panic (fmt=0xc087f113 "%s") at /usr/src/sys/kern/kern_shutdown.c:558 #3 0xc08493e6 in trap_fatal (frame=0xeb9a7b58, eva=0) at /usr/src/sys/i386/i386/trap.c:836 #4 0xc08490ef in trap_pfault (frame=0xeb9a7b58, usermode=0, eva=0) at /usr/src/sys/i386/i386/trap.c:744 #5 0xc0848d05 in trap (frame= {tf_fs = -916914168, tf_es = -916914136, tf_ds = -896335832, tf_edi = -917610112, tf_esi = -342197140, tf_ebp = -342197300, tf_isp = -342197372, tf_ebx = 4, tf_edx = -919717272, tf_ecx = -917610156, tf_eax = -1064162880, tf_trapno = 12, tf_err = 0, tf_eip = 0, tf_cs = 32, tf_eflags = 66118, tf_esp = -1067102273, tf_ss = -917610156}) at /usr/src/sys/i386/i386/trap.c:434 #6 0xc08361fa in calltrap () at /usr/src/sys/i386/i386/exception.s:139 #7 0x00000000 in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb) quit root@cache3:/usr/src# exit The panics seems to happen when load is low as well as high. >How-To-Repeat: Install FreeBSD 6.1, and then ports/www/squid26: cd /usr/ports/www/squid26 make SQUID_CONFIGURE_ARGS="--enable-dlmalloc --enable-snmp --enable-async-io=50" WITH_SQUID_AUFS=yes OPTIONS="" install Set up some aufs cache dirs on different disks (I am using SCSI disks in HP Proliant DL 380): cache_dir aufs /data/cache 15000 64 512 cache_dir aufs /data01/cache 15000 64 512 cache_dir aufs /data02/cache 15000 64 512 cache_dir aufs /data03/cache 15000 64 512 cache_dir aufs /data04/cache 15000 64 512 >Fix: N/A I am going to be trying libthr(3), meanwhile, to see if I can avoid the problem then. Let me know if there is anything more I can do to help debug the problem. >Release-Note: >Audit-Trail: >Unformatted: From owner-freebsd-threads@FreeBSD.ORG Mon Sep 11 11:08:39 2006 Return-Path: X-Original-To: freebsd-threads@FreeBSD.org Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1984E16A416 for ; Mon, 11 Sep 2006 11:08:39 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4D0FC43D49 for ; Mon, 11 Sep 2006 11:08:38 +0000 (GMT) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (linimon@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id k8BB8cBx063346 for ; Mon, 11 Sep 2006 11:08:38 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id k8BB8aWQ063342 for freebsd-threads@FreeBSD.org; Mon, 11 Sep 2006 11:08:36 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 11 Sep 2006 11:08:36 GMT Message-Id: <200609111108.k8BB8aWQ063342@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: linimon set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-threads@FreeBSD.org Cc: Subject: Current problem reports assigned to you X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Sep 2006 11:08:39 -0000 Current FreeBSD problem reports Critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- s threa/76690 threads fork hang in child for (-lc_r & -lthr) 1 problem total. Serious problems S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/20016 threads pthreads: Cannot set scheduling timer/Cannot set virtu s threa/24472 threads libc_r does not honor SO_SNDTIMEO/SO_RCVTIMEO socket o s threa/24632 threads libc_r delicate deviation from libc in handling SIGCHL s bin/32295 threads pthread dont dequeue signals s threa/34536 threads accept() blocks other threads o kern/38549 threads the procces compiled whith pthread stopped in pthread_ s threa/39922 threads [threads] [patch] Threaded applications executed with s threa/48856 threads Setting SIGCHLD to SIG_IGN still leaves zombies under s threa/49087 threads Signals lost in programs linked with libc_r s kern/64313 threads FreeBSD (OpenBSD) pthread implicit set/unset O_NONBLOC o threa/70975 threads unexpected and unreliable behaviour when using SYSV se o threa/72353 threads Assertion fails in /usr/src/lib/libpthread/sys/lock.c, o threa/72429 threads threads blocked in stdio (fgets, etc) are not cancella o threa/72953 threads fork() unblocks blocked signals w/o PTHREAD_SCOPE_SYST o threa/75273 threads FBSD 5.3 libpthread (KSE) bug o threa/75374 threads pthread_kill() ignores SA_SIGINFO flag s threa/76694 threads fork cause hang in dup()/close() function in child (-l o threa/79683 threads svctcp_create() fails if multiple threads call at the o threa/80435 threads panic on high loads o threa/83914 threads [libc] popen() doesn't work in static threaded program s threa/84483 threads problems with devel/nspr and -lc_r on 4.x o threa/85160 threads [libthr] [patch] libobjc + libpthread/libthr crash pro p threa/89262 threads [kernel] [patch] multi-threaded process hangs in kerne o threa/90278 threads libthr, ULE and -current produces >100% WCPU with apac o kern/91266 threads [threads] Trying sleep, but thread marked as sleeping s threa/94467 threads send(), sendto() and sendmsg() are not correct in libc f threa/98256 threads gnome-system-monitor core dumps from pthread_testcance s threa/100815 threads FBSD 5.5 broke nanosleep in libc_r o threa/101323 threads fork(2) in threaded programs broken. o threa/101355 threads threaded application spents too much time in _umtx_op o threa/103127 threads Kernel panic while using thread features in Squid 2.6 31 problems total. Non-critical problems S Tracker Resp. Description -------------------------------------------------------------------------------- s kern/19247 threads uthread_sigaction.c does not do anything wrt SA_NOCLDW s kern/22190 threads A threaded read(2) from a socketpair(2) fd can sometim s threa/30464 threads pthread mutex attributes -- pshared s threa/37676 threads libc_r: msgsnd(), msgrcv(), pread(), pwrite() need wra s threa/40671 threads pthread_cancel doesn't remove thread from condition qu s threa/69020 threads pthreads library leaks _gc_mutex o threa/74180 threads KSE problem. Applications those riched maximum possibl o threa/79887 threads [patch] freopen() isn't thread-safe o threa/80992 threads abort() sometimes not caught by gdb depending on threa o threa/81534 threads [libc_r] [patch] libc_r close() will fail on any fd ty 10 problems total. From owner-freebsd-threads@FreeBSD.ORG Tue Sep 12 03:31:50 2006 Return-Path: X-Original-To: freebsd-threads@hub.freebsd.org Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0E3AA16A40F; Tue, 12 Sep 2006 03:31:50 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id BA92743D45; Tue, 12 Sep 2006 03:31:49 +0000 (GMT) (envelope-from delphij@FreeBSD.org) Received: from freefall.freebsd.org (delphij@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id k8C3VnJM064854; Tue, 12 Sep 2006 03:31:49 GMT (envelope-from delphij@freefall.freebsd.org) Received: (from delphij@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id k8C3Vn1T064850; Tue, 12 Sep 2006 03:31:49 GMT (envelope-from delphij) Date: Tue, 12 Sep 2006 03:31:49 GMT From: Xin LI Message-Id: <200609120331.k8C3Vn1T064850@freefall.freebsd.org> To: garry@NetworkPhysics.COM, delphij@FreeBSD.org, freebsd-threads@FreeBSD.org, davidxu@FreeBSD.org Cc: Subject: Re: threads/89262: [kernel] [patch] multi-threaded process hangs in kernel in fork() X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Sep 2006 03:31:50 -0000 Synopsis: [kernel] [patch] multi-threaded process hangs in kernel in fork() State-Changed-From-To: patched->closed State-Changed-By: delphij State-Changed-When: Tue Sep 12 03:29:52 UTC 2006 State-Changed-Why: Patch was merged to REENG_6. Responsible-Changed-From-To: freebsd-threads->davidxu Responsible-Changed-By: delphij Responsible-Changed-When: Tue Sep 12 03:29:52 UTC 2006 Responsible-Changed-Why: The issue was fixed by davidxu@ so over to him. http://www.freebsd.org/cgi/query-pr.cgi?pr=89262 From owner-freebsd-threads@FreeBSD.ORG Tue Sep 12 19:56:02 2006 Return-Path: X-Original-To: freebsd-threads@FreeBSD.org Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4998216A407; Tue, 12 Sep 2006 19:56:02 +0000 (UTC) (envelope-from anders@FreeBSD.org) Received: from fupp.net (totem.fix.no [80.91.36.20]) by mx1.FreeBSD.org (Postfix) with ESMTP id F005C43D62; Tue, 12 Sep 2006 19:55:49 +0000 (GMT) (envelope-from anders@FreeBSD.org) Received: from localhost (totem.fix.no [80.91.36.20]) by fupp.net (Postfix) with ESMTP id 4BAB48D9896; Tue, 12 Sep 2006 21:55:48 +0200 (CEST) Received: from fupp.net ([80.91.36.20]) by localhost (totem.fix.no [80.91.36.20]) (amavisd-new, port 10024) with LMTP id 73858-01-4; Tue, 12 Sep 2006 21:55:47 +0200 (CEST) Received: by fupp.net (Postfix, from userid 1000) id AE26E8D9888; Tue, 12 Sep 2006 21:55:47 +0200 (CEST) Date: Tue, 12 Sep 2006 21:55:47 +0200 From: Anders Nordby To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-threads@FreeBSD.org Message-ID: <20060912195547.GA71462@totem.fix.no> References: <20060911075431.D12758D9874@fupp.net> <200609110800.k8B80ie3041853@freefall.freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200609110800.k8B80ie3041853@freefall.freebsd.org> X-PGP-Key: http://anders.fix.no/pgp/ X-PGP-Key-FingerPrint: 1E0F C53C D8DF 6A8F EAAD 19C5 D12A BC9F 0083 5956 User-Agent: Mutt/1.5.11 Cc: Suleiman Souhlal Subject: Re: threads/103127: Kernel panic while using thread features in Squid 2.6 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Sep 2006 19:56:02 -0000 Using libthr instead of pthread didn't help at all. It might be a kqueue issue after all. On recommendation by Pawel Worach, I set these sysctls: debug.trace_on_panic=1 debug.debugger_on_panic=0 Then I could get more debugging info: # dmesg -M /var/crash/vmcore.0 -N /usr/obj/usr/src/sys/CACHE/kernel.debug (..) Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 06 fault virtual address = 0x0 fault code = supervisor read, page not present instruction pointer = 0x20:0x0 stack pointer = 0x28:0xeb886b98 frame pointer = 0x28:0xeb886bcc code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 545 (squid) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: kdb_backtrace(100,c9599480,28,eb886b58,c) at kdb_backtrace+0x29 panic(c087f113,c08ce651,0,fffff,c959d49b) at panic+0x114 trap_fatal(eb886b58,0,c9599480,c8f8a708,c) at trap_fatal+0x2ce trap_pfault(eb886b58,0,0) at trap_pfault+0x1d7 trap(8,c9590028,ce020028,c9feb0b4,eb886c30) at trap+0x2fd calltrap() at calltrap+0x5 --- trap 0xc, eip = 0, esp = 0xeb886b98, ebp = 0xeb886bcc --- (null)(c9560800,eb886c1c,c9599480,1,0) at 0 kern_kevent(c9599480,3,37,80,eb886cc8) at kern_kevent+0xc9 kevent(c9599480,eb886d04,6,4bacc,202) at kevent+0x55 syscall(3b,3b,bfbf003b,82180b1,0) at syscall+0x2bf Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (363, FreeBSD ELF32, kevent), eip = 0xb71e24af, esp = 0xbfbfe9cc, ebp = 0xbfbfea18 --- Uptime: 21h18m39s Dumping 3839 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 3839MB (982778 pages) 3824 3808 3792 3776 3760 3744 3728 3712 3696 3680 3664 3648 3632 3616 3600 3584 3568 3552 3536 3520 3504 3488 3472 3456 3440 3424 3408 3392 3376 3360 3344 3328 3312 3296 3280 3264 3248 3232 3216 3200 3184 3168 3152 3136 3120 3104 3088 3072 3056 3040 3024 3008 2992 2976 2960 2944 2928 2912 2896 2880 2864 2848 2832 2816 2800 2784 2768 2752 2736 2720 2704 2688 2672 2656 2640 2624 2608 2592 2576 2560 2544 2528 2512 2496 2480 2464 2448 2432 2416 2400 2384 2368 2352 2336 2320 2304 2288 2272 2256 2240 2224 2208 2192 2176 2160 2144 2128 2112 2096 2080 2064 2048 2032 2016 2000 1984 1968 1952 1936 1920 1904 1888 1872 1856 1840 1824 1808 1792 1776 1760 1744 1728 1712 1696 1680 1664 1648 1632 1616 1600 1584 1568 1552 1536 1520 1504 1488 1472 1456 1440 1424 1408 1392 1376 1360 1344 1328 1312 1296 1280 1264 1248 1232 1216 1200 1184 1168 1152 1136 1120 1104 1088 1072 1056 1040 1024 1008 992 976 960 944 928 912 896 880 864 848 832 816 800 784 768 752 736 720 704 688 672 656 640 624 608 592 576 560 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16 Checking those kevent's, I get: (kgdb) l *kevent+0x55 0xc06542fd is in kevent (/usr/src/sys/kern/kern_event.c:571). 566 return (error); 567 tsp = &ts; 568 } else 569 tsp = NULL; 570 571 return (kern_kevent(td, uap->fd, uap->nchanges, uap->nevents, 572 &k_ops, tsp)); 573 } 574 575 /* (kgdb) l *kern_kevent+0xc9 0xc0654439 is in kern_kevent (/usr/src/sys/kern/kern_event.c:637). 632 goto done; 633 changes = keva; 634 for (i = 0; i < n; i++) { 635 kevp = &changes[i]; 636 kevp->flags &= ~EV_SYSFLAGS; 637 error = kqueue_register(kq, kevp, td, 1); 638 if (error) { 639 if (nevents != 0) { 640 kevp->flags = EV_ERROR; 641 kevp->data = error; (kgdb) This is in FreeBSD 6.1-RELEASE-p5. One physical processor, but two logical (HyperThreading enabled). Regards, Anders. From owner-freebsd-threads@FreeBSD.ORG Tue Sep 12 20:01:11 2006 Return-Path: X-Original-To: freebsd-threads@hub.freebsd.org Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D8A7016A47E for ; Tue, 12 Sep 2006 20:01:11 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 18BBA43D72 for ; Tue, 12 Sep 2006 20:00:47 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id k8CK0lCm065521 for ; Tue, 12 Sep 2006 20:00:47 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id k8CK0lhG065516; Tue, 12 Sep 2006 20:00:47 GMT (envelope-from gnats) Date: Tue, 12 Sep 2006 20:00:47 GMT Message-Id: <200609122000.k8CK0lhG065516@freefall.freebsd.org> To: freebsd-threads@FreeBSD.org From: Anders Nordby Cc: Subject: Re: threads/103127: Kernel panic while using thread features in Squid 2.6 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Anders Nordby List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Sep 2006 20:01:11 -0000 The following reply was made to PR kern/103127; it has been noted by GNATS. From: Anders Nordby To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-threads@FreeBSD.org Cc: Suleiman Souhlal , Pawel Worach Subject: Re: threads/103127: Kernel panic while using thread features in Squid 2.6 Date: Tue, 12 Sep 2006 21:55:47 +0200 Using libthr instead of pthread didn't help at all. It might be a kqueue issue after all. On recommendation by Pawel Worach, I set these sysctls: debug.trace_on_panic=1 debug.debugger_on_panic=0 Then I could get more debugging info: # dmesg -M /var/crash/vmcore.0 -N /usr/obj/usr/src/sys/CACHE/kernel.debug (..) Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 06 fault virtual address = 0x0 fault code = supervisor read, page not present instruction pointer = 0x20:0x0 stack pointer = 0x28:0xeb886b98 frame pointer = 0x28:0xeb886bcc code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 545 (squid) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: kdb_backtrace(100,c9599480,28,eb886b58,c) at kdb_backtrace+0x29 panic(c087f113,c08ce651,0,fffff,c959d49b) at panic+0x114 trap_fatal(eb886b58,0,c9599480,c8f8a708,c) at trap_fatal+0x2ce trap_pfault(eb886b58,0,0) at trap_pfault+0x1d7 trap(8,c9590028,ce020028,c9feb0b4,eb886c30) at trap+0x2fd calltrap() at calltrap+0x5 --- trap 0xc, eip = 0, esp = 0xeb886b98, ebp = 0xeb886bcc --- (null)(c9560800,eb886c1c,c9599480,1,0) at 0 kern_kevent(c9599480,3,37,80,eb886cc8) at kern_kevent+0xc9 kevent(c9599480,eb886d04,6,4bacc,202) at kevent+0x55 syscall(3b,3b,bfbf003b,82180b1,0) at syscall+0x2bf Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (363, FreeBSD ELF32, kevent), eip = 0xb71e24af, esp = 0xbfbfe9cc, ebp = 0xbfbfea18 --- Uptime: 21h18m39s Dumping 3839 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 3839MB (982778 pages) 3824 3808 3792 3776 3760 3744 3728 3712 3696 3680 3664 3648 3632 3616 3600 3584 3568 3552 3536 3520 3504 3488 3472 3456 3440 3424 3408 3392 3376 3360 3344 3328 3312 3296 3280 3264 3248 3232 3216 3200 3184 3168 3152 3136 3120 3104 3088 3072 3056 3040 3024 3008 2992 2976 2960 2944 2928 2912 2896 2880 2864 2848 2832 2816 2800 2784 2768 2752 2736 2720 2704 2688 2672 2656 2640 2624 2608 2592 2576 2560 2544 2528 2512 2496 2480 2464 2448 2432 2416 2400 2384 2368 2352 2336 2320 2304 2288 2272 2256 2240 2224 2208 2192 2176 2160 2144 2128 2112 2096 2080 2064 2048 2032 2016 2000 1984 1968 1952 1936 1920 1904 1888 1872 1856 1840 1824 1808 1792 1776 1760 1744 1728 1712 1696 1680 1664 1648 1632 1616 1600 1584 1568 1552 1536 1520 1504 1488 1472 1456 1440 1424 1408 1392 1376 1360 1344 1328 1312 1296 1280 1264 1248 1232 1216 1200 1184 1168 1152 1136 1120 1104 1088 1072 1056 1040 1024 1008 992 976 960 944 928 912 896 880 864 848 832 816 800 784 768 752 736 720 704 688 672 656 640 624 608 592 576 560 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16 Checking those kevent's, I get: (kgdb) l *kevent+0x55 0xc06542fd is in kevent (/usr/src/sys/kern/kern_event.c:571). 566 return (error); 567 tsp = &ts; 568 } else 569 tsp = NULL; 570 571 return (kern_kevent(td, uap->fd, uap->nchanges, uap->nevents, 572 &k_ops, tsp)); 573 } 574 575 /* (kgdb) l *kern_kevent+0xc9 0xc0654439 is in kern_kevent (/usr/src/sys/kern/kern_event.c:637). 632 goto done; 633 changes = keva; 634 for (i = 0; i < n; i++) { 635 kevp = &changes[i]; 636 kevp->flags &= ~EV_SYSFLAGS; 637 error = kqueue_register(kq, kevp, td, 1); 638 if (error) { 639 if (nevents != 0) { 640 kevp->flags = EV_ERROR; 641 kevp->data = error; (kgdb) This is in FreeBSD 6.1-RELEASE-p5. One physical processor, but two logical (HyperThreading enabled). Regards, Anders. From owner-freebsd-threads@FreeBSD.ORG Wed Sep 13 18:58:36 2006 Return-Path: X-Original-To: freebsd-threads@FreeBSD.org Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9612316A416; Wed, 13 Sep 2006 18:58:36 +0000 (UTC) (envelope-from anders@fupp.net) Received: from fupp.net (totem.fix.no [80.91.36.20]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0E66643D98; Wed, 13 Sep 2006 18:58:05 +0000 (GMT) (envelope-from anders@fupp.net) Received: from localhost (totem.fix.no [80.91.36.20]) by fupp.net (Postfix) with ESMTP id E264D8D9896; Wed, 13 Sep 2006 20:58:03 +0200 (CEST) Received: from fupp.net ([80.91.36.20]) by localhost (totem.fix.no [80.91.36.20]) (amavisd-new, port 10024) with LMTP id 81684-01-6; Wed, 13 Sep 2006 20:58:03 +0200 (CEST) Received: by fupp.net (Postfix, from userid 1000) id 68A958D9874; Wed, 13 Sep 2006 20:58:03 +0200 (CEST) Date: Wed, 13 Sep 2006 20:58:03 +0200 From: Anders Nordby To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-threads@FreeBSD.org Message-ID: <20060913185803.GA27955@totem.fix.no> References: <20060911075431.D12758D9874@fupp.net> <200609110800.k8B80ie3041853@freefall.freebsd.org> <20060912195547.GA71462@totem.fix.no> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="LZvS9be/3tNcYl/X" Content-Disposition: inline In-Reply-To: <20060912195547.GA71462@totem.fix.no> X-PGP-Key: http://anders.fix.no/pgp/ X-PGP-Key-FingerPrint: 1E0F C53C D8DF 6A8F EAAD 19C5 D12A BC9F 0083 5956 User-Agent: Mutt/1.5.11 Cc: Thomas-Martin Seck , Suleiman Souhlal Subject: Re: threads/103127: Kernel panic while using thread features in Squid 2.6 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2006 18:58:36 -0000 --LZvS9be/3tNcYl/X Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, After talking with Pawel, I tried the attached patch to get some kqueue debug info. I just got it: Sep 13 20:18:56 cache3 kernel: NULL f_event in new kn Sep 13 20:18:56 cache3 kernel: f_event == NULL Then Squid stops responding to new requests, and I can not even kill it with kill -9: root@cache3:~# pgrep -l squid 547 squid root@cache3:~# kill -9 547 root@cache3:~# pgrep -l squid 547 squid Have to reboot to get the system working again, but it seems now it doesn't panic. Regards, -- Anders. --LZvS9be/3tNcYl/X Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="kqueue-debug.patch" --- sys/kern/kern_event.c.orig Wed Sep 13 08:44:57 2006 +++ sys/kern/kern_event.c Wed Sep 13 08:51:03 2006 @@ -900,6 +900,8 @@ goto done; } KN_LIST_LOCK(kn); + if (kn->kn_fop->f_event == NULL) + printf("NULL f_event in new kn\n"); } else { /* * The user may change some filter values after the @@ -912,6 +914,8 @@ kn->kn_sfflags = kev->fflags; kn->kn_sdata = kev->data; kn->kn_kevent.udata = kev->udata; + if (kn->kn_fop->f_event == NULL) + printf("NULL f_event in existing kn\n"); } /* @@ -921,6 +925,12 @@ * is called on a zombie process. It will call filt_proc * which will remove it from the list, and NULL kn_knlist. */ + if (kn->kn_fop->f_event == NULL) { + printf("f_event == NULL\n"); + KN_LIST_UNLOCK(kn); + error = EAGAIN; + goto done; + } event = kn->kn_fop->f_event(kn, 0); KQ_LOCK(kq); if (event) --LZvS9be/3tNcYl/X-- From owner-freebsd-threads@FreeBSD.ORG Wed Sep 13 19:00:49 2006 Return-Path: X-Original-To: freebsd-threads@hub.freebsd.org Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E989016A412 for ; Wed, 13 Sep 2006 19:00:49 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7AE1F43D73 for ; Wed, 13 Sep 2006 19:00:43 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id k8DJ0hSH019044 for ; Wed, 13 Sep 2006 19:00:43 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id k8DJ0hkR019043; Wed, 13 Sep 2006 19:00:43 GMT (envelope-from gnats) Date: Wed, 13 Sep 2006 19:00:43 GMT Message-Id: <200609131900.k8DJ0hkR019043@freefall.freebsd.org> To: freebsd-threads@FreeBSD.org From: Anders Nordby Cc: Subject: Re: threads/103127: Kernel panic while using thread features in Squid 2.6 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Anders Nordby List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Sep 2006 19:00:50 -0000 The following reply was made to PR kern/103127; it has been noted by GNATS. From: Anders Nordby To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-threads@FreeBSD.org Cc: Suleiman Souhlal , Pawel Worach , Thomas-Martin Seck Subject: Re: threads/103127: Kernel panic while using thread features in Squid 2.6 Date: Wed, 13 Sep 2006 20:58:03 +0200 --LZvS9be/3tNcYl/X Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hi, After talking with Pawel, I tried the attached patch to get some kqueue debug info. I just got it: Sep 13 20:18:56 cache3 kernel: NULL f_event in new kn Sep 13 20:18:56 cache3 kernel: f_event == NULL Then Squid stops responding to new requests, and I can not even kill it with kill -9: root@cache3:~# pgrep -l squid 547 squid root@cache3:~# kill -9 547 root@cache3:~# pgrep -l squid 547 squid Have to reboot to get the system working again, but it seems now it doesn't panic. Regards, -- Anders. --LZvS9be/3tNcYl/X Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="kqueue-debug.patch" --- sys/kern/kern_event.c.orig Wed Sep 13 08:44:57 2006 +++ sys/kern/kern_event.c Wed Sep 13 08:51:03 2006 @@ -900,6 +900,8 @@ goto done; } KN_LIST_LOCK(kn); + if (kn->kn_fop->f_event == NULL) + printf("NULL f_event in new kn\n"); } else { /* * The user may change some filter values after the @@ -912,6 +914,8 @@ kn->kn_sfflags = kev->fflags; kn->kn_sdata = kev->data; kn->kn_kevent.udata = kev->udata; + if (kn->kn_fop->f_event == NULL) + printf("NULL f_event in existing kn\n"); } /* @@ -921,6 +925,12 @@ * is called on a zombie process. It will call filt_proc * which will remove it from the list, and NULL kn_knlist. */ + if (kn->kn_fop->f_event == NULL) { + printf("f_event == NULL\n"); + KN_LIST_UNLOCK(kn); + error = EAGAIN; + goto done; + } event = kn->kn_fop->f_event(kn, 0); KQ_LOCK(kq); if (event) --LZvS9be/3tNcYl/X-- From owner-freebsd-threads@FreeBSD.ORG Sat Sep 16 20:51:09 2006 Return-Path: X-Original-To: freebsd-threads@FreeBSD.org Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F390E16A403; Sat, 16 Sep 2006 20:51:08 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.FreeBSD.org (Postfix) with ESMTP id 94CD943D58; Sat, 16 Sep 2006 20:50:45 +0000 (GMT) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (macjaiscmamqcn0k@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.6/8.13.3) with ESMTP id k8GKojdJ065567; Sat, 16 Sep 2006 13:50:45 -0700 (PDT) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.6/8.13.3/Submit) id k8GKoi8A065566; Sat, 16 Sep 2006 13:50:44 -0700 (PDT) (envelope-from jmg) Date: Sat, 16 Sep 2006 13:50:44 -0700 From: John-Mark Gurney To: Anders Nordby Message-ID: <20060916205044.GF9421@funkthat.com> Mail-Followup-To: Anders Nordby , FreeBSD-gnats-submit@FreeBSD.org, freebsd-threads@FreeBSD.org, Thomas-Martin Seck , Suleiman Souhlal References: <20060911075431.D12758D9874@fupp.net> <200609110800.k8B80ie3041853@freefall.freebsd.org> <20060912195547.GA71462@totem.fix.no> <20060913185803.GA27955@totem.fix.no> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20060913185803.GA27955@totem.fix.no> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: Suleiman Souhlal , FreeBSD-gnats-submit@FreeBSD.org, Thomas-Martin Seck , freebsd-threads@FreeBSD.org Subject: Re: threads/103127: Kernel panic while using thread features in Squid 2.6 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Sep 2006 20:51:09 -0000 Anders Nordby wrote this message on Wed, Sep 13, 2006 at 20:58 +0200: > After talking with Pawel, I tried the attached patch to get some kqueue > debug info. I just got it: > > Sep 13 20:18:56 cache3 kernel: NULL f_event in new kn > Sep 13 20:18:56 cache3 kernel: f_event == NULL This means that the f_attach method for that event isn't setting f_event properly... We need to figure out which event this is happening w/... Try the following modifications: > --- sys/kern/kern_event.c.orig Wed Sep 13 08:44:57 2006 > +++ sys/kern/kern_event.c Wed Sep 13 08:51:03 2006 > @@ -900,6 +900,8 @@ > goto done; > } > KN_LIST_LOCK(kn); > + if (kn->kn_fop->f_event == NULL) { > + printf("NULL f_event in new kn\n"); printf("kn: ident: %d, filter: %d\n", kn->kn_ident, kn->kn_filter); } > } else { > /* > * The user may change some filter values after the I have a feeling that this is a similar panic that jhb is seeing... Which means that filter will be EVFILT_VNODE (-4)... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-threads@FreeBSD.ORG Sat Sep 16 20:59:10 2006 Return-Path: X-Original-To: freebsd-threads@FreeBSD.org Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9094E16A500; Sat, 16 Sep 2006 20:59:10 +0000 (UTC) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (gate.funkthat.com [69.17.45.168]) by mx1.FreeBSD.org (Postfix) with ESMTP id B05B843D55; Sat, 16 Sep 2006 20:59:09 +0000 (GMT) (envelope-from jmg@hydrogen.funkthat.com) Received: from hydrogen.funkthat.com (ivo8j3xnjq7wkrqz@localhost.funkthat.com [127.0.0.1]) by hydrogen.funkthat.com (8.13.6/8.13.3) with ESMTP id k8GKx9Bu065710; Sat, 16 Sep 2006 13:59:09 -0700 (PDT) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.13.6/8.13.3/Submit) id k8GKx9lg065709; Sat, 16 Sep 2006 13:59:09 -0700 (PDT) (envelope-from jmg) Date: Sat, 16 Sep 2006 13:59:08 -0700 From: John-Mark Gurney To: Anders Nordby Message-ID: <20060916205908.GG9421@funkthat.com> Mail-Followup-To: Anders Nordby , FreeBSD-gnats-submit@FreeBSD.org, freebsd-threads@FreeBSD.org, Thomas-Martin Seck , Suleiman Souhlal References: <20060911075431.D12758D9874@fupp.net> <200609110800.k8B80ie3041853@freefall.freebsd.org> <20060912195547.GA71462@totem.fix.no> <20060913185803.GA27955@totem.fix.no> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="oC1+HKm2/end4ao3" Content-Disposition: inline In-Reply-To: <20060913185803.GA27955@totem.fix.no> User-Agent: Mutt/1.4.2.1i X-Operating-System: FreeBSD 5.4-RELEASE-p6 i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Cc: Suleiman Souhlal , FreeBSD-gnats-submit@FreeBSD.org, Thomas-Martin Seck , freebsd-threads@FreeBSD.org Subject: Re: threads/103127: Kernel panic while using thread features in Squid 2.6 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Sep 2006 20:59:10 -0000 --oC1+HKm2/end4ao3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Anders Nordby wrote this message on Wed, Sep 13, 2006 at 20:58 +0200: > After talking with Pawel, I tried the attached patch to get some kqueue > debug info. I just got it: > > Sep 13 20:18:56 cache3 kernel: NULL f_event in new kn > Sep 13 20:18:56 cache3 kernel: f_event == NULL > > Then Squid stops responding to new requests, and I can not even kill it > with kill -9: Please try the attached patch.. It appears that the badfo_kqfilter was returning sucess instead of error since it was introcuded many years ago... This many still cause other problems w/ squid, but will fix the panics... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." --oC1+HKm2/end4ao3 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="badfo_kqfilter.patch" Index: kern_descrip.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_descrip.c,v retrieving revision 1.297 diff -u -r1.297 kern_descrip.c --- kern_descrip.c 21 Jul 2006 20:24:00 -0000 1.297 +++ kern_descrip.c 16 Sep 2006 20:58:40 -0000 @@ -2632,7 +2632,7 @@ badfo_kqfilter(struct file *fp, struct knote *kn) { - return (0); + return (EINVAL); } static int --oC1+HKm2/end4ao3-- From owner-freebsd-threads@FreeBSD.ORG Sat Sep 16 21:00:46 2006 Return-Path: X-Original-To: freebsd-threads@hub.freebsd.org Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id B777216A492 for ; Sat, 16 Sep 2006 21:00:46 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id 80A9743D55 for ; Sat, 16 Sep 2006 21:00:46 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id k8GL0kU4032100 for ; Sat, 16 Sep 2006 21:00:46 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id k8GL0kGi032099; Sat, 16 Sep 2006 21:00:46 GMT (envelope-from gnats) Date: Sat, 16 Sep 2006 21:00:46 GMT Message-Id: <200609162100.k8GL0kGi032099@freefall.freebsd.org> To: freebsd-threads@FreeBSD.org From: John-Mark Gurney Cc: Subject: Re: threads/103127: Kernel panic while using thread features in Squid 2.6 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Sep 2006 21:00:46 -0000 The following reply was made to PR kern/103127; it has been noted by GNATS. From: John-Mark Gurney To: Anders Nordby Cc: FreeBSD-gnats-submit@FreeBSD.org, freebsd-threads@FreeBSD.org, Thomas-Martin Seck , Suleiman Souhlal Subject: Re: threads/103127: Kernel panic while using thread features in Squid 2.6 Date: Sat, 16 Sep 2006 13:50:44 -0700 Anders Nordby wrote this message on Wed, Sep 13, 2006 at 20:58 +0200: > After talking with Pawel, I tried the attached patch to get some kqueue > debug info. I just got it: > > Sep 13 20:18:56 cache3 kernel: NULL f_event in new kn > Sep 13 20:18:56 cache3 kernel: f_event == NULL This means that the f_attach method for that event isn't setting f_event properly... We need to figure out which event this is happening w/... Try the following modifications: > --- sys/kern/kern_event.c.orig Wed Sep 13 08:44:57 2006 > +++ sys/kern/kern_event.c Wed Sep 13 08:51:03 2006 > @@ -900,6 +900,8 @@ > goto done; > } > KN_LIST_LOCK(kn); > + if (kn->kn_fop->f_event == NULL) { > + printf("NULL f_event in new kn\n"); printf("kn: ident: %d, filter: %d\n", kn->kn_ident, kn->kn_filter); } > } else { > /* > * The user may change some filter values after the I have a feeling that this is a similar panic that jhb is seeing... Which means that filter will be EVFILT_VNODE (-4)... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-threads@FreeBSD.ORG Sat Sep 16 21:00:53 2006 Return-Path: X-Original-To: freebsd-threads@hub.freebsd.org Delivered-To: freebsd-threads@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4168816A4A7 for ; Sat, 16 Sep 2006 21:00:53 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id C494843D55 for ; Sat, 16 Sep 2006 21:00:52 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id k8GL0qRr032127 for ; Sat, 16 Sep 2006 21:00:52 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id k8GL0q29032125; Sat, 16 Sep 2006 21:00:52 GMT (envelope-from gnats) Date: Sat, 16 Sep 2006 21:00:52 GMT Message-Id: <200609162100.k8GL0q29032125@freefall.freebsd.org> To: freebsd-threads@FreeBSD.org From: John-Mark Gurney Cc: Subject: Re: threads/103127: Kernel panic while using thread features in Squid 2.6 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: John-Mark Gurney List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 16 Sep 2006 21:00:53 -0000 The following reply was made to PR kern/103127; it has been noted by GNATS. From: John-Mark Gurney To: Anders Nordby Cc: FreeBSD-gnats-submit@FreeBSD.org, freebsd-threads@FreeBSD.org, Thomas-Martin Seck , Suleiman Souhlal Subject: Re: threads/103127: Kernel panic while using thread features in Squid 2.6 Date: Sat, 16 Sep 2006 13:59:08 -0700 --oC1+HKm2/end4ao3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Anders Nordby wrote this message on Wed, Sep 13, 2006 at 20:58 +0200: > After talking with Pawel, I tried the attached patch to get some kqueue > debug info. I just got it: > > Sep 13 20:18:56 cache3 kernel: NULL f_event in new kn > Sep 13 20:18:56 cache3 kernel: f_event == NULL > > Then Squid stops responding to new requests, and I can not even kill it > with kill -9: Please try the attached patch.. It appears that the badfo_kqfilter was returning sucess instead of error since it was introcuded many years ago... This many still cause other problems w/ squid, but will fix the panics... -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." --oC1+HKm2/end4ao3 Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="badfo_kqfilter.patch" Index: kern_descrip.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_descrip.c,v retrieving revision 1.297 diff -u -r1.297 kern_descrip.c --- kern_descrip.c 21 Jul 2006 20:24:00 -0000 1.297 +++ kern_descrip.c 16 Sep 2006 20:58:40 -0000 @@ -2632,7 +2632,7 @@ badfo_kqfilter(struct file *fp, struct knote *kn) { - return (0); + return (EINVAL); } static int --oC1+HKm2/end4ao3--