From owner-freebsd-current@FreeBSD.ORG Tue Oct 15 08:43:56 2013 Return-Path: Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 654E0D85 for ; Tue, 15 Oct 2013 08:43:56 +0000 (UTC) (envelope-from mexas@bristol.ac.uk) Received: from eu1sys200aog117.obsmtp.com (eu1sys200aog117.obsmtp.com [207.126.144.143]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id 671D42D2C for ; Tue, 15 Oct 2013 08:43:54 +0000 (UTC) Received: from mail-wg0-f51.google.com ([74.125.82.51]) (using TLSv1) by eu1sys200aob117.postini.com ([207.126.147.11]) with SMTP ID DSNKUl0ASamvSHv8ND3C6dkFNxh7x0przGNz@postini.com; Tue, 15 Oct 2013 08:43:55 UTC Received: by mail-wg0-f51.google.com with SMTP id l18so4260021wgh.30 for ; Tue, 15 Oct 2013 01:43:53 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:sender:date:from:message-id:to:subject:cc :reply-to:in-reply-to; bh=qIZOnKs6cocoqi24maySE41oy63uT+WI8F4kkYMQedY=; b=m6urSfwkEZk/kazrc/6YCSRNbhhGCuSAvdfcYM0wXhHeT+TOAGtN+XdxeJoSMFvZoU VrHdCklqzZrj7Q+aNm0UUmzB7JpTVOT2hROUcWnpS2RoGZZQWELqX14sguB8DwNtFW7Y xCffBF4wkvaeC7PqXTAOS6Yq3OJGR6V96vfZqEyu9p9zmaOOWfLtAZ+zfIsjEJdxyDXj TrXuwu6WU31YFezTGKq7xzpHysgz5uczYO+vrCg5PnjZvwRqijA8CGmhHrZ1qhWIzT45 NVKAGXjPoFfOzZHgk7YpEq9HtDgLPMSDlaxqaCV+oWHUyOzks17eN2vT++KRGsX7XrgY YhYA== X-Gm-Message-State: ALoCoQlXbRezJcg+8OvT9FMiCi90r0kDoaNq74ya8lP/ugLTb4MoQ6W3L6UGDQIwbbw/IB0ayQQMOtIetykWXJugnH9REa/oWqiQg0pebx+byIX6rx3BWfDYfl/7bCi+zTQAfjRe5QHDkF5mqxBp8rtkdHmisBNiJ4Z95iOFkq5kc4CAtKhx8As= X-Received: by 10.180.187.169 with SMTP id ft9mr18481612wic.14.1381826633644; Tue, 15 Oct 2013 01:43:53 -0700 (PDT) X-Received: by 10.180.187.169 with SMTP id ft9mr18481606wic.14.1381826633515; Tue, 15 Oct 2013 01:43:53 -0700 (PDT) Received: from mech-cluster241.men.bris.ac.uk (mech-cluster241.men.bris.ac.uk. [137.222.187.241]) by mx.google.com with ESMTPSA id om10sm3947130wic.5.2013.10.15.01.43.51 for (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Oct 2013 01:43:52 -0700 (PDT) Sender: Anton Shterenlikht Received: from mech-cluster241.men.bris.ac.uk (localhost [127.0.0.1]) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6) with ESMTP id r9F8hok0043236 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 15 Oct 2013 09:43:50 +0100 (BST) (envelope-from mexas@mech-cluster241.men.bris.ac.uk) Received: (from mexas@localhost) by mech-cluster241.men.bris.ac.uk (8.14.7/8.14.6/Submit) id r9F8ho1c043235; Tue, 15 Oct 2013 09:43:50 +0100 (BST) (envelope-from mexas) Date: Tue, 15 Oct 2013 09:43:50 +0100 (BST) From: Anton Shterenlikht Message-Id: <201310150843.r9F8ho1c043235@mech-cluster241.men.bris.ac.uk> To: davide@freebsd.org, mexas@bris.ac.uk Subject: panic: wrong page state m 0xe00000027a9adb40 + savecore deadlock In-Reply-To: Cc: freebsd-current@freebsd.org, freebsd-ia64@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list Reply-To: mexas@bris.ac.uk List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 15 Oct 2013 08:43:56 -0000 >From davide.italiano@gmail.com Mon Oct 14 12:50:44 2013 > >This is fair enough -- If you're still at the ddb prompt, please print >the whole panic message (or at least the address of the lock reported >as deadlocked by DEADLKRES), so that we can at least have a candidate. > Here's another one, followed by savecore deadlock. ia64 r255488 panic: wrong page state m 0xe00000027a9adb40 cpuid = 0 KDB: stack backtrace: db_trace_self(0x9ffc000000158380) at db_trace_self+0x40 db_trace_self_wrapper(0x9ffc000000607370) at db_trace_self_wrapper+0x70 kdb_backtrace(0x9ffc000000ed0e10, 0x9ffc00000058e660, 0x40c, 0x9ffc0000010a44a0) at kdb_backtrace+0xc0 vpanic(0x9ffc000000dd3fe0, 0xa00000009de61118, 0x9ffc000000ef9670, 0x9ffc000000ed0bc0) at vpanic+0x260 kassert_panic(0x9ffc000000dd3fe0, 0xe00000027a9adb40, 0x81f, 0xe0000002013cf400, 0x9ffc0000006a0220, 0x2c60, 0xe0000002013cf400, 0xe0000002013cf418) at kassert_panic+0x120 vn_sendfile(0x8df, 0xd, 0x0, 0x0, 0x0, 0x8df, 0x7fffffffffffdfe0, 0x0) at vn_sendfile+0x15d0 sys_sendfile(0xe000000012aef200, 0xa00000009de614e8, 0x10, 0xa00000009de61360) at sys_sendfile+0x2b0 syscall(0xe0000000154f2940, 0xd, 0x0, 0xe000000012aef200, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return KDB: enter: panic [ thread pid 5989 tid 100111 ] Stopped at kdb_enter+0x92: [I2] addl r14=0xffffffffffe2c990,gp ;; db> db> scripts lockinfo=show locks; show alllocks; show lockedvnods zzz=textdump set; capture on; run lockinfo; show pcpu; bt; ps; alltrace; capture off; call doadump; reset db> db> run zzz I get to db:0:alltrace> capture off db:0:off> call doadump Dumping 10220 MB (25 chunks) chunk 0: 1 pages ... ok chunk 1: 159 pages ... ok chunk 2: 256 pages ... ok chunk 3: 7680 pages ... ok chunk 4: 8192 pages ... ok chunk 5: 239734 pages ... ok chunk 6: 748 pages ... ok chunk 7: 533 pages ... ok chunk 8: 21 pages ... ok chunk 9: 1572862 pages ... ok chunk 10: 781683 pages ... ok chunk 11: 512 pages ... ok chunk 12: 139 pages ... ok chunk 13: 484 pages ... ok chunk 14: 1565 pages ... ok chunk 15: 1 pages ... ok chunk 16: 506 pages ... ok chunk 17: 1 pages ... ok chunk 18: 3 pages ... ok chunk 19: 566 pages ... ok chunk 20: 66 pages ... ok chunk 21: 1 pages ... ok chunk 22: 285 pages ... ok chunk 23: 6 pages ... ok chunk 24: 354 pages ... ok Dump complete = 0 db:0:doadump> reset So far, so good. On reboot I get: Starting ddb. ddb: sysctl: debug.ddb.scripting.scripts: Invalid argument /etc/rc: WARNING: failed to start ddb This probably already indicates some problem? Eventually I get to: savecore: reboot after panic: wrong page state m 0xe00000027a9adb40 Oct 15 09:05:50 mech-as28 savecore: reboot after panic: wrong page state m 0xe00000027a9adb40 savecore: writing core to /var/crash/vmcore.9 So here I'm confused. I think I set up textdump as in the man page. So I think the core should not be written. Instead I was expecting ddb.txt, config.txt, etc., as in textdump(4). Anyway, savecore eventually deadlocks: panic: deadlkres: possible deadlock detected for 0xe0000000127b7b00, blocked for 901401 ticks cpuid = 0 KDB: stack backtrace: db_trace_self(0x9ffc000000158380) at db_trace_self+0x40 db_trace_self_wrapper(0x9ffc000000607370) at db_trace_self_wrapper+0x70 kdb_backtrace(0x9ffc000000ed0e10, 0x9ffc00000058e660, 0x40c, 0x9ffc0000010a44a0) at kdb_backtrace+0xc0 vpanic(0x9ffc000000db8a18, 0xa00000009dca7518) at vpanic+0x260 panic(0x9ffc000000db8a18, 0x9ffc000000db8c70, 0xe0000000127b7b00, 0xdc119) at panic+0x80 deadlkres(0xdc119, 0xe0000000127b7b00, 0x9ffc000000dbb648, 0x9ffc000000db89a8) at deadlkres+0x420 fork_exit(0x9ffc000000e0fca0, 0x0, 0xa00000009dca7550) at fork_exit+0x120 enter_userland() at enter_userland KDB: enter: panic [ thread pid 0 tid 100053 ] Stopped at kdb_enter+0x92: [I2] addl r14=0xffffffffffe2c990,gp ;; db> db> scripts lockinfo=show locks; show alllocks; show lockedvnods db> run lockinfo db:0:lockinfo> show locks db:0:locks> show alllocks db:0:alllocks> show lockedvnods Locked vnodes 0xe0000000127cbba8: tag devfs, type VCHR usecount 1, writecount 0, refcount 19 mountedhere 0xe0000000126ab200 flags (VI_ACTIVE) v_object 0xe0000000127c2b00 ref 0 pages 422 lock type devfs: EXCL by thread 0xe000000012690000 (pid 21, syncer, tid 100062) dev da3p1 0xe0000000127f4ec0: tag ufs, type VREG usecount 1, writecount 1, refcount 32934 mountedhere 0 flags (VI_ACTIVE) v_object 0xe0000000127f7200 ref 0 pages 1242850 lock type ufs: EXCL by thread 0xe0000000127b7b00 (pid 805, savecore, tid 100079) ino 6500740, on dev da3p1 db> db> ps pid ppid pgrp uid state wmesg wchan cmd 805 803 24 0 L+ *vm page 0xe000000012402fc0 savecore 803 24 24 0 DL+ vm map ( 0xe00000001285fa88 sh 801 1 801 0 Ss select 0xe000000010c296c0 syslogd 792 1 792 0 Ss select 0xe000000010c29840 ipmon 670 1 670 0 Ss select 0xe0000000126d5740 devd 155 1 155 0 SWs pause 0xe0000000127fe9e8 adjkerntz 24 1 24 0 LLs+ *vm activ 0xe000000012403bc0 sh 23 0 0 0 DL sdflush 0x9ffc000000ed1ff0 [softdepflush] 22 0 0 0 DL vlruwt 0xe000000012409720 [vnlru] 21 0 0 0 LL *vm objec 0xe000000010c30240 [syncer] 20 0 0 0 DL psleep 0x9ffc000000ed1390 [bufdaemon] 19 0 0 0 DL pgzero 0x9ffc000000ed239c [pagezero] 18 0 0 0 RL [vmdaemon] 17 0 0 0 LL *pmap pv 0xe000000010c2e180 [pagedaemon] 9 0 0 0 DL ccb_scan 0x9ffc000000ed26a0 [xpt_thrd] 6 0 0 0 DL waiting_ 0x9ffc0000010f33b0 [sctp_iterator] 5 0 0 0 DL (threaded) [zfskern] 100052 D l2arc_fe 0x9ffc000001547f40 [l2arc_feed_thread] 100051 D arc_recl 0x9ffc000001538120 [arc_reclaim_thread] --More-- 4 0 0 0 DL ispf 0xe000000011a02000 [isp0: fc_thrd0] 3 0 0 0 DL idle 0xa000000000a16000 [mpt_recovery1] 2 0 0 0 DL idle 0xa0000000009fe000 [mpt_recovery0] 16 0 0 0 DL (threaded) [usb] 100039 D - 0xa0000000009fae18 [usbus2] 100038 D - 0xa0000000009fadc0 [usbus2] 100037 D - 0xa0000000009fad68 [usbus2] 100036 D - 0xa0000000009fad10 [usbus2] 100034 D - 0xa0000000009f1460 [usbus1] 100033 D - 0xa0000000009f1408 [usbus1] 100032 D - 0xa0000000009f13b0 [usbus1] 100031 D - 0xa0000000009f1358 [usbus1] 100029 D - 0xa0000000009ed460 [usbus0] 100028 D - 0xa0000000009ed408 [usbus0] 100027 D - 0xa0000000009ed3b0 [usbus0] 100026 D - 0xa0000000009ed358 [usbus0] 15 0 0 0 DL tzpoll 0x9ffc000000ecfd20 [acpi_thermal] 14 0 0 0 DL - 0x9ffc000000ed0008 [rand_harvestq] 13 0 0 0 LL (threaded) [geom] 100012 D - 0x9ffc000000ed0540 [g_down] --More-- 100011 L *vm objec 0xe000000010c30240 [g_up] 100010 D - 0x9ffc000000ed0528 [g_event] 12 0 0 0 RL (threaded) [intr] 100048 I [swi0: uart uart+++] 100046 I [irq71: isp0] 100042 I [irq28: mpt1] 100040 I [irq27: mpt0] 100035 I [irq18: ehci0] 100030 I [irq17: ohci1] 100025 I [irq16: ohci0] 100022 I [swi6: task queue] 100021 I [swi6: Giant taskq] 100019 I [swi5: fast taskq] 100015 I [swi2: cambio] 100008 I [swi4: clock] 100007 Run CPU 1 [swi4: clock] 100006 I [swi3: vm] 100005 I [swi1: netisr 0] 11 0 0 0 RL (threaded) [idle] 100004 CanRun [idle: cpu1] 100003 CanRun [idle: cpu0] 1 0 1 0 SLs wait 0xe000000011176de0 [init] 10 0 0 0 DL audit_wo 0x9ffc0000010f7a48 [audit] 0 0 0 0 RLs (threaded) [kernel] 100053 Run CPU 0 [deadlkres] 100050 D - 0xe000000011a2d300 [system_taskq_1] 100049 D - 0xe000000011a2d300 [system_taskq_0] 100045 D - 0xe000000011198500 [em1 taskq] 100044 D - 0xe000000011198600 [em0 taskq] 100023 D - 0xe00000001119b300 [ffs_trim taskq] 100020 D - 0xe00000001119b600 [thread taskq] 100018 D - 0xe00000001119b800 [acpi_task_2] 100017 D - 0xe00000001119b800 [acpi_task_1] 100016 D - 0xe00000001119b800 [acpi_task_0] 100014 D - 0xe00000001119ba00 [kqueue taskq] 100009 D - 0xe00000001119bd00 [firmware taskq] 100000 D swapin 0x9ffc000000eedd28 [swapper] db> db> show pcpu cpuid = 0 dynamic pcpu = 0x40040040ff13ca80 curthread = 0xe000000011a26d80: pid 0 "deadlkres" curpcb = 0 fpcurthread = none idlethread = 0xe000000011178900: tid 100003 "idle: cpu0" MD: vhpt = 0xe0000040ff000000 MD: lid = 0 MD: clock = 0x2351ad02dce MD: clock_mode = 2 MD: clock_load = 0 MD: stats = 0x9ffc0000010ff130 MD: pmap = 0x9ffc000000eee760 spin locks held: db> db> bt Tracing pid 0 tid 100053 td 0xe000000011a26d80 kdb_enter(0x9ffc000000dc2fd0, 0x9ffc000000dc2fd0, 0x9ffc00000058e6b0, 0x40c) at kdb_enter+0x92 vpanic(0x9ffc000000db8a18, 0xa00000009dca7518) at vpanic+0x2b0 panic(0x9ffc000000db8a18, 0x9ffc000000db8c70, 0xe0000000127b7b00, 0xdc119) at panic+0x80 deadlkres(0xdc119, 0xe0000000127b7b00, 0x9ffc000000dbb648, 0x9ffc000000db89a8) at deadlkres+0x420 fork_exit(0x9ffc000000e0fca0, 0x0, 0xa00000009dca7550) at fork_exit+0x120 enter_userland() at enter_userland db> db> alltrace Tracing command savecore pid 805 tid 100079 td 0xe0000000127b7b00 cpu_switch(0xe0000000127b7b00, 0xe000000011178900, 0xe000000012402fc0, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe0000000127b7b00, 0xe000000011178900, 0x9ffc000000f15698, 0x9ffc000000f15680) at sched_switch+0x890 mi_switch(0x103, 0x0, 0xe0000000127b7b00, 0x9ffc00000062d1f0) at mi_switch+0x3f0 turnstile_wait(0xe000000012402fc0, 0xe000000012400480, 0x0, 0x9ffc000000dcb698) at turnstile_wait+0x960 __mtx_lock_sleep(0x9ffc0000010f9998, 0xe0000000127b7b00, 0xe000000012402fc0, 0x9ffc000000dc0558, 0x742) at __mtx_lock_sleep+0x2f0 __mtx_lock_flags(0x9ffc0000010f9980, 0x0, 0x9ffc000000dd4a90, 0x742) at __mtx_lock_flags+0x1e0 vfs_vmio_release(0xa00000009ebe72f0, 0xe00000027ed2ab70, 0x3, 0xa00000009ebe736c, 0xa00000009ebe7498, 0xa00000009ebe72f8, 0x9ffc000000dd4a90, 0x9ffc0000010f9680) at vfs_vmio_release+0x290 getnewbuf(0xe0000000127f4ec0, 0x0, 0x0, 0x8000, 0xa00000009ebe99a8, 0x0, 0x9ffc0000010f0798, 0xa00000009ebe72f0) at getnewbuf+0x7e0 getblk(0xe0000000127f4ec0, 0x4cbaa, 0x8000, 0x0, 0x0, 0x0, 0x0, 0x0) at getblk+0xee0 ffs_balloc_ufs2(0xe0000000127f4ec0, 0x4cbaa, 0xa0000000c60ba000, 0xe000000011165a00, 0x7f050000, 0xa00000009dd79160) at ffs_balloc_ufs2+0x2950 ffs_write(0xa00000009dd79248, 0x3000, 0x265d50000) at ffs_write+0x5c0 VOP_WRITE_APV(0x9ffc000000e94ac0, 0xa00000009dd79248, 0x0, 0x0) at VOP_WRITE_APV+0x330 vn_write(0xe0000000129ae820, 0xa00000009dd79360, 0xe000000011165a00, 0x0, 0xe0000000129ae830, 0xe0000000127f4ec0) at vn_write+0x450 vn_io_fault(0xe0000000129ae820, 0xa00000009dd79360, 0xe000000011165a00, 0x0, 0xe0000000127b7b00) at vn_io_fault+0x330 dofilewrite(0xe0000000127b7b00, 0x7, 0xe0000000129ae820, 0xa00000009dd79360, 0xffffffffffffffff, 0x0) at dofilewrite+0x180 kern_writev(0xe0000000127b7b00, 0x7, 0xa00000009dd79360) at kern_writev+0xa0 sys_write(0xe0000000127b7b00, 0xa00000009dd794e8, 0x9ffc000000abac80, 0x48d) at sys_write+0x100 syscall(0xe0000000129d04a0, 0x140857000, 0x8000, 0xe0000000127b7b00, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 --More-- epc_syscall_return() at epc_syscall_return Tracing command sh pid 803 tid 100075 td 0xe0000000127b6000 cpu_switch(0xe0000000127b6000, 0xe000000011178900, 0x9ffc000000f4c570, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe0000000127b6000, 0xe000000011178900, 0x9ffc000000f15698, 0x9ffc000000f15680) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe0000000127b6000, 0x9ffc000000620b70) at mi_switch+0x3f0 sleepq_switch(0xe00000001285fa88, 0x0, 0xe0000000127b6378, 0xe0000000127b6000) at sleepq_switch+0x2e0 sleepq_wait(0xe00000001285fa88, 0x0, 0x9ffc000000dca928, 0xe0000000127b6000, 0x9ffc00000059d350) at sleepq_wait+0xb0 _sx_slock_hard(0xe00000001285fa88, 0x0, 0xe00000001285faa0, 0xe000000012400000, 0xe00000001285fa88) at _sx_slock_hard+0x5f0 _sx_slock(0xe00000001285fa88, 0x0, 0x9ffc000000dfd218, 0xefa) at _sx_slock+0x190 _vm_map_lock_read(0xe00000001285fa08, 0x9ffc000000dfd218, 0xefa) at _vm_map_lock_read+0x80 vm_map_lookup(0xa00000009dd58928, 0x100046000, 0x1, 0xa00000009dd58930, 0xa00000009dd58918, 0xa00000009dd58920, 0xa00000009dd588f0, 0xa00000009dd588f4) at vm_map_lookup+0x70 vm_fault_hold(0xe00000001285fa08, 0x100046000, 0x1, 0x0, 0x0) at vm_fault_hold+0x280 vm_fault(0xe00000001285fa08, 0x100046000, 0x1, 0x0, 0xe0000000127b6000, 0x9ffc000000abbce0, 0x716, 0xe0000000127fe5a0) at vm_fault+0xf0 trap(0x14, 0xa00000009dd59400) at trap+0x9e0 ivt_Data_TLB() at ivt_Data_TLB+0x1d0 --- trapframe at 0xa00000009dd59400 (null)(...) at 0x8000000000000120 Tracing command syslogd pid 801 tid 100078 td 0xe00000001282c000 cpu_switch(0xe00000001282c000, 0xe000000011178480, 0x9ffc000000f4c110, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe00000001282c000, 0xe000000011178480, 0x9ffc000000f16398, 0x9ffc000000f16380) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001282c000, 0x9ffc000000620b70) at mi_switch+0x3f0 sleepq_switch(0xe000000010c296c0, 0x0, 0xe00000001282c378, 0xe00000001282c000) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000010c296c0, 0x0, 0x9ffc000000dca928, 0x9ffc00000056ab00) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000010c296c0, 0x0, 0x9ffc0000004f7230, 0x611, 0x9ffc0000004f7270) at sleepq_wait_sig+0x30 _cv_wait_sig(0xe000000010c296c0, 0xe000000010c296a0, 0xe00000001282c000, 0x0, 0x9ffc000000ef0710) at _cv_wait_sig+0x4d0 seltdwait(0xe000000010c29680, 0xffffffffffffffff, 0x0, 0xe000000010c296b8, 0x9ffc000000dcdfe0, 0x9ffc000000645bf0) at seltdwait+0x110 kern_select(0xe00000001282c000, 0x0, 0x140866180, 0x0, 0x0, 0xffffffffffffffff, 0x9) at kern_select+0xb40 sys_select(0xe00000001282c000, 0xa00000009dd714e8, 0x9ffc000000abac80, 0x48d) at sys_select+0xb0 syscall(0xe0000000129d0940, 0x140866180, 0x0, 0xe00000001282c000, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command ipmon pid 792 tid 100077 td 0xe00000001282c480 cpu_switch(0xe00000001282c480, 0xe000000011178900, 0x9ffc000000f4d560, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe00000001282c480, 0xe000000011178900, 0x9ffc000000f15698, 0x9ffc000000f15680) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe00000001282c480, 0x9ffc000000620b70) at mi_switch+0x3f0 sleepq_switch(0xe000000010c29840, 0x0, 0xe00000001282c7f8, 0xe00000001282c480) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe000000010c29840, 0x0, 0x9ffc000000dca928, 0x9ffc00000056ab00) at sleepq_catch_signals+0x190 sleepq_wait_sig(0xe000000010c29840, 0x0, 0x9ffc0000004f7230, 0x611, 0x9ffc0000004f7270) at sleepq_wait_sig+0x30 --More-- _cv_wait_sig(0xe000000010c29840, 0xe000000010c29820, 0xe00000001282c480, 0x0, 0x9ffc000000ef0710) at _cv_wait_sig+0x4d0 seltdwait(0xe000000010c29800, 0xffffffffffffffff, 0x0, 0xe000000010c29838, 0x9ffc000000dcdfe0, 0x9ffc000000645bf0) at seltdwait+0x110 kern_select(0xe00000001282c480, 0x0, 0x7fffffffffff6b50, 0x0, 0x0, 0xffffffffffffffff, 0x4) at kern_select+0xb40 sys_select(0xe00000001282c480, 0xa00000009dd694e8, 0x9ffc000000abac80, 0x48d) at sys_select+0xb0 syscall(0xe0000000129d0de0, 0x7fffffffffff6b50, 0x0, 0xe00000001282c480, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return Tracing command devd pid 670 tid 100056 td 0xe000000012400d80 cpu_switch(0xe000000012400d80, 0xe000000011178480, 0x9ffc000000f4b738, 0x9ffc0000005e7e80) at cpu_switch+0xd0 sched_switch(0xe000000012400d80, 0xe000000011178480, 0x9ffc000000f16398, 0x9ffc000000f16380) at sched_switch+0x890 mi_switch(0x104, 0x0, 0xe000000012400d80, 0x9ffc000000620b70) at mi_switch+0x3f0 sleepq_switch(0xe0000000126d5740, 0x0, 0xe0000000124010f8, 0xe000000012400d80) at sleepq_switch+0x2e0 sleepq_catch_signals(0xe0000000126d5740, 0x0, 0x9ffc000000dca928, 0xe0000000126d5720) at sleepq_catch_signals+0x190 sleepq_timedwait_sig(0xe0000000126d5740, 0x0, 0x9ffc0000004f6430, 0x9ffc0000004f6470, 0x713) at sleepq_timedwait_sig+0x30 _cv_timedwait_sig_sbt(0xe0000000126d5740, 0xe0000000126d5720, 0x5bbba38f388, 0x3c0000000, 0x200) at _cv_timedwait_sig_sbt+0x4f0 seltdwait(0xe0000000126d5700, 0x5bbba38f388, 0x3c0000000, 0xe0000000126d5738, 0x9ffc000000dcdfe0, 0x9ffc000000645bf0) at seltdwait+0xf0 kern_select(0xe000000012400d80, 0x0, 0x7fffffffffffcd00, 0x0, 0x0, 0x5bbba38f388, 0x5) at kern_select+0xb40 sys_select(0xe000000012400d80, 0xa00000009dcc14e8, 0x9ffc000000abac80, 0x48d) at sys_select+0xb0 syscall(0xe0000000111cb280, 0x7fffffffffffcd00, 0x0, 0xe000000012400d80, 0x0, 0x0, 0x9ffc000000ab7280, 0x8) at syscall+0x5e0 epc_syscall_return() at epc_syscall_return --More-- Tracing command adjkerntz pid 155 tid 100074 td 0xe0000000127b6480 Tracing command sh pid 24 tid 100065 td 0xe000000012401200 Tracing command softdepflush pid 23 tid 100064 td 0xe000000012401680 Tracing command vnlru pid 22 tid 100063 td 0xe000000012401b00 Tracing command syncer pid 21 tid 100062 td 0xe000000012690000 Tracing command bufdaemon pid 20 tid 100061 td 0xe000000012690480 Tracing command pagezero pid 19 tid 100060 td 0xe000000011a27b00 Tracing command vmdaemon pid 18 tid 100059 td 0xe000000012400000 Tracing command pagedaemon pid 17 tid 100058 td 0xe000000012400480 Tracing command xpt_thrd pid 9 tid 100057 td 0xe000000012400900 --More-- Tracing command sctp_iterator pid 6 tid 100054 td 0xe000000011a26900 Tracing command zfskern pid 5 tid 100052 td 0xe000000011a27200 Tracing command zfskern pid 5 tid 100051 td 0xe000000011a27680 Tracing command isp0: fc_thrd0 pid 4 tid 100047 td 0xe000000011931b00 Tracing command mpt_recovery1 pid 3 tid 100043 td 0xe000000011930000 Tracing command mpt_recovery0 pid 2 tid 100041 td 0xe000000011930900 Tracing command usb pid 16 tid 100039 td 0xe000000011924480 Tracing command usb pid 16 tid 100038 td 0xe000000011924900 Tracing command usb pid 16 tid 100037 td 0xe000000011924d80 Tracing command usb pid 16 tid 100036 td 0xe000000011925200 --More-- Tracing command usb pid 16 tid 100034 td 0xe000000011300d80 Tracing command usb pid 16 tid 100033 td 0xe000000011301200 Tracing command usb pid 16 tid 100032 td 0xe000000011301680 Tracing command usb pid 16 tid 100031 td 0xe000000011301b00 Tracing command usb pid 16 tid 100029 td 0xe000000011219680 Tracing command usb pid 16 tid 100028 td 0xe000000011219b00 Tracing command usb pid 16 tid 100027 td 0xe000000011300000 Tracing command usb pid 16 tid 100026 td 0xe000000011300480 Tracing command acpi_thermal pid 15 tid 100024 td 0xe000000011218000 Tracing command rand_harvestq pid 14 tid 100013 td 0xe000000011197680 --More-- Tracing command geom pid 13 tid 100012 td 0xe000000011197b00 Tracing command geom pid 13 tid 100011 td 0xe0000000111a6000 Tracing command geom pid 13 tid 100010 td 0xe000000011179680 Tracing command intr pid 12 tid 100048 td 0xe000000011931680 Tracing command intr pid 12 tid 100046 td 0xe000000011a26000 Tracing command intr pid 12 tid 100042 td 0xe000000011930480 Tracing command intr pid 12 tid 100040 td 0xe000000011924000 Tracing command intr pid 12 tid 100035 td 0xe000000011300900 Tracing command intr pid 12 tid 100030 td 0xe000000011219200 Tracing command intr pid 12 tid 100025 td 0xe0000000111a7b00 --More-- Tracing command intr pid 12 tid 100022 td 0xe000000011218900 Tracing command intr pid 12 tid 100021 td 0xe000000011218d80 Tracing command intr pid 12 tid 100019 td 0xe0000000111a6900 Tracing command intr pid 12 tid 100015 td 0xe000000011196d80 Tracing command intr pid 12 tid 100008 td 0xe000000011196000 Tracing command intr pid 12 tid 100007 td 0xe000000011196480 Tracing command intr pid 12 tid 100006 td 0xe000000011196900 Tracing command intr pid 12 tid 100005 td 0xe000000011178000 Tracing command idle pid 11 tid 100004 td 0xe000000011178480 Tracing command idle pid 11 tid 100003 td 0xe000000011178900 --More-- Tracing command init pid 1 tid 100002 td 0xe000000011178d80 Tracing command audit pid 10 tid 100001 td 0xe000000011179200 Tracing command kernel pid 0 tid 100053 td 0xe000000011a26d80 Tracing command kernel pid 0 tid 100050 td 0xe000000011930d80 Tracing command kernel pid 0 tid 100049 td 0xe000000011931200 Tracing command kernel pid 0 tid 100045 td 0xe000000011925680 Tracing command kernel pid 0 tid 100044 td 0xe000000011925b00 Tracing command kernel pid 0 tid 100023 td 0xe000000011218480 Tracing command kernel pid 0 tid 100020 td 0xe0000000111a6480 Tracing command kernel pid 0 tid 100018 td 0xe0000000111a6d80 --More-- Tracing command kernel pid 0 tid 100017 td 0xe0000000111a7200 Tracing command kernel pid 0 tid 100016 td 0xe0000000111a7680 Tracing command kernel pid 0 tid 100014 td 0xe000000011197200 Tracing command kernel pid 0 tid 100009 td 0xe000000011179b00 Tracing command kernel pid 0 tid 100000 td 0x9ffc000000eee1d0 db> I'll go check textdump setup again... Many thanks Anton