From owner-freebsd-fs@FreeBSD.ORG Sun Mar 7 01:13:48 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D3ECC106566B; Sun, 7 Mar 2010 01:13:48 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from esa-jnhn.mail.uoguelph.ca (esa-jnhn.mail.uoguelph.ca [131.104.91.44]) by mx1.freebsd.org (Postfix) with ESMTP id 438518FC12; Sun, 7 Mar 2010 01:13:47 +0000 (UTC) X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: AvsEAOOMkkuDaFvH/2dsb2JhbACbSXO2WIR4BIMXiyQ X-IronPort-AV: E=Sophos;i="4.49,596,1262581200"; d="scan'208";a="67794263" Received: from danube.cs.uoguelph.ca ([131.104.91.199]) by esa-jnhn-pri.mail.uoguelph.ca with ESMTP; 06 Mar 2010 20:13:47 -0500 Received: from localhost (localhost.localdomain [127.0.0.1]) by danube.cs.uoguelph.ca (Postfix) with ESMTP id 05C5310843E2; Sat, 6 Mar 2010 20:13:47 -0500 (EST) X-Virus-Scanned: amavisd-new at danube.cs.uoguelph.ca Received: from danube.cs.uoguelph.ca ([127.0.0.1]) by localhost (danube.cs.uoguelph.ca [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id lyEyZ0nar1B2; Sat, 6 Mar 2010 20:13:46 -0500 (EST) Received: from muncher.cs.uoguelph.ca (muncher.cs.uoguelph.ca [131.104.91.102]) by danube.cs.uoguelph.ca (Postfix) with ESMTP id 0A3381084106; Sat, 6 Mar 2010 20:13:46 -0500 (EST) Received: from localhost (rmacklem@localhost) by muncher.cs.uoguelph.ca (8.11.7p3+Sun/8.11.6) with ESMTP id o271Q4s18320; Sat, 6 Mar 2010 20:26:05 -0500 (EST) X-Authentication-Warning: muncher.cs.uoguelph.ca: rmacklem owned process doing -bs Date: Sat, 6 Mar 2010 20:26:04 -0500 (EST) From: Rick Macklem X-X-Sender: rmacklem@muncher.cs.uoguelph.ca To: Daniel Braniss In-Reply-To: Message-ID: References: <20100226174021.8feadad9.gerrit@pmp.uni-hannover.de> <20100226224320.8c4259bf.gerrit@pmp.uni-hannover.de> <4B884757.9040001@digiware.nl> <20100227080220.ac6a2e4d.gerrit@pmp.uni-hannover.de> <4B892918.4080701@digiware.nl> <20100227202105.f31cbef7.gerrit@pmp.uni-hannover.de> <20100227193819.GA60576@icarus.home.lan> <4B89943C.70704@digiware.nl> <20100227220310.GA65110@icarus.home.lan> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: stable@freebsd.org, freebsd-fs@freebsd.org, Willem Jan Withagen , =?utf-8?B?RWlyaWsgw5h2ZXJieQ==?= , rwatson@freebsd.org, Jeremy Chadwick Subject: Re: mbuf leakage with nfs/zfs? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Mar 2010 01:13:48 -0000 On Sat, 6 Mar 2010, Daniel Braniss wrote: > it works ok in 7.2, so it would be interesting to compare changes ... > The sys/rpc in FreeBSD8 is completely different code than what is used in FreeBSD7, so I'm afraid thay're apples vs oranges. rick From owner-freebsd-fs@FreeBSD.ORG Mon Mar 8 11:06:56 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C1C02106570A for ; Mon, 8 Mar 2010 11:06:56 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 839218FC20 for ; Mon, 8 Mar 2010 11:06:56 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o28B6uIi073672 for ; Mon, 8 Mar 2010 11:06:56 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o28B6tGN073670 for freebsd-fs@FreeBSD.org; Mon, 8 Mar 2010 11:06:55 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 8 Mar 2010 11:06:55 GMT Message-Id: <201003081106.o28B6tGN073670@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-fs@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-fs@FreeBSD.org X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Mar 2010 11:06:56 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/144458 fs [nfs] [patch] nfsd fails as a kld o kern/144447 fs [zfs] sharenfs fsunshare() & fsshare_main() non functi o kern/144416 fs [panic] Kernel panic on online filesystem optimization o kern/144415 fs [zfs] [panic] kernel panics on boot after zfs crash o kern/144402 fs [zfs] [panic] panic at zfs_znode_dmu_init: existing zn o kern/144330 fs [nfs] mbuf leakage in nfsd with zfs o kern/144234 fs [zfs] Cannot boot machine with recent gptzfsboot code o bin/144214 fs zfsboot fails on gang block after upgrade to zfs v14 o kern/143825 fs [nfs] [panic] Kernel panic on NFS client o kern/143345 fs [ext2fs] [patch] extfs minor header cleanups to better o kern/143343 fs [zfs] bug in sunlink flag on directories o kern/143212 fs [nfs] NFSv4 client strange work ... o kern/143184 fs [zfs] [lor] zfs/bufwait LOR o kern/142924 fs [ext2fs] [patch] Small cleanup for the inode struct in o kern/142914 fs [zfs] ZFS performance degradation over time o kern/142878 fs [zfs] [vfs] lock order reversal o kern/142872 fs [zfs] ZFS ZVOL Lockmgr Deadlock o kern/142597 fs [ext2fs] ext2fs does not work on filesystems with real o kern/142594 fs [zfs] Modification time reset to 1 Jan 1970 after fsyn o kern/142489 fs [zfs] [lor] allproc/zfs LOR o kern/142466 fs Update 7.2 -> 8.0 on Raid 1 ends with screwed raid [re o kern/142401 fs [ntfs] [patch] Minor updates to NTFS from NetBSD o kern/142306 fs [zfs] [panic] ZFS drive (from OSX Leopard) causes two o kern/142068 fs [ufs] BSD labels are got deleted spontaneously o kern/141950 fs [unionfs] [lor] ufs/unionfs/ufs Lock order reversal o kern/141897 fs [msdosfs] [panic] Kernel panic. msdofs: file name leng o kern/141718 fs [zfs] [panic] kernel panic when 'zfs rename' is used o o kern/141685 fs [zfs] zfs corruption on adaptec 5805 raid controller o kern/141463 fs [nfs] [panic] Frequent kernel panics after upgrade fro o kern/141305 fs [zfs] FreeBSD ZFS+sendfile severe performance issues ( o kern/141177 fs [zfs] fsync() on FIFO causes panic() on zfs o kern/141091 fs [patch] [nullfs] fix panics with DIAGNOSTIC enabled o kern/141086 fs [nfs] [panic] panic("nfs: bioread, not dir") on FreeBS o kern/141010 fs [zfs] "zfs scrub" fails when backed by files in UFS2 o kern/140888 fs [zfs] boot fail from zfs root while the pool resilveri o kern/140682 fs [netgraph] [panic] random panic in netgraph o kern/140661 fs [zfs] [patch] /boot/loader fails to work on a GPT/ZFS- o kern/140640 fs [zfs] snapshot crash o kern/140433 fs [zfs] [panic] panic while replaying ZIL after crash o kern/140134 fs [msdosfs] write and fsck destroy filesystem integrity o kern/140068 fs [smbfs] [patch] smbfs does not allow semicolon in file o kern/139725 fs [zfs] zdb(1) dumps core on i386 when examining zpool c o kern/139715 fs [zfs] vfs.numvnodes leak on busy zfs o bin/139651 fs [nfs] mount(8): read-only remount of NFS volume does n o kern/139597 fs [patch] [tmpfs] tmpfs initializes va_gen but doesn't u o kern/139564 fs [zfs] [panic] 8.0-RC1 - Fatal trap 12 at end of shutdo o kern/139407 fs [smbfs] [panic] smb mount causes system crash if remot o kern/139363 fs [nfs] diskless root nfs mount from non FreeBSD server o kern/138790 fs [zfs] ZFS ceases caching when mem demand is high o kern/138524 fs [msdosfs] disks and usb flashes/cards with Russian lab o kern/138421 fs [ufs] [patch] remove UFS label limitations o kern/138202 fs mount_msdosfs(1) see only 2Gb f kern/137037 fs [zfs] [hang] zfs rollback on root causes FreeBSD to fr o kern/136968 fs [ufs] [lor] ufs/bufwait/ufs (open) o kern/136945 fs [ufs] [lor] filedesc structure/ufs (poll) o kern/136944 fs [ffs] [lor] bufwait/snaplk (fsync) o kern/136873 fs [ntfs] Missing directories/files on NTFS volume o kern/136865 fs [nfs] [patch] NFS exports atomic and on-the-fly atomic o kern/136470 fs [nfs] Cannot mount / in read-only, over NFS o kern/135594 fs [zfs] Single dataset unresponsive with Samba o kern/135546 fs [zfs] zfs.ko module doesn't ignore zpool.cache filenam o kern/135469 fs [ufs] [panic] kernel crash on md operation in ufs_dirb o kern/135050 fs [zfs] ZFS clears/hides disk errors on reboot o kern/134491 fs [zfs] Hot spares are rather cold... o kern/133676 fs [smbfs] [panic] umount -f'ing a vnode-based memory dis o kern/133614 fs [panic] panic: ffs_truncate: read-only filesystem o kern/133174 fs [msdosfs] [patch] msdosfs must support utf-encoded int f kern/133150 fs [zfs] Page fault with ZFS on 7.1-RELEASE/amd64 while w o kern/132960 fs [ufs] [panic] panic:ffs_blkfree: freeing free frag o kern/132397 fs reboot causes filesystem corruption (failure to sync b o kern/132331 fs [ufs] [lor] LOR ufs and syncer o kern/132237 fs [msdosfs] msdosfs has problems to read MSDOS Floppy o kern/132145 fs [panic] File System Hard Crashes o kern/131995 fs [nfs] Failure to mount NFSv4 server o kern/131441 fs [unionfs] [nullfs] unionfs and/or nullfs not combineab o kern/131360 fs [nfs] poor scaling behavior of the NFS server under lo o kern/131342 fs [nfs] mounting/unmounting of disks causes NFS to fail o bin/131341 fs makefs: error "Bad file descriptor" on the mount poin o kern/130979 fs [smbfs] [panic] boot/kernel/smbfs.ko o kern/130920 fs [msdosfs] cp(1) takes 100% CPU time while copying file o kern/130229 fs [iconv] usermount fails on fs that need iconv o kern/130210 fs [nullfs] Error by check nullfs o kern/129760 fs [nfs] after 'umount -f' of a stale NFS share FreeBSD l o kern/129488 fs [smbfs] Kernel "bug" when using smbfs in smbfs_smb.c: o kern/129231 fs [ufs] [patch] New UFS mount (norandom) option - mostly o kern/129152 fs [panic] non-userfriendly panic when trying to mount(8) o kern/129059 fs [zfs] [patch] ZFS bootloader whitelistable via WITHOUT f kern/128829 fs smbd(8) causes periodic panic on 7-RELEASE o kern/127420 fs [gjournal] [panic] Journal overflow on gmirrored gjour o kern/127029 fs [panic] mount(8): trying to mount a write protected zi o kern/126287 fs [ufs] [panic] Kernel panics while mounting an UFS file s kern/125738 fs [zfs] [request] SHA256 acceleration in ZFS p kern/124621 fs [ext3] [patch] Cannot mount ext2fs partition f bin/124424 fs [zfs] zfs(8): zfs list -r shows strange snapshots' siz o kern/123939 fs [msdosfs] corrupts new files o kern/122380 fs [ffs] ffs_valloc:dup alloc (Soekris 4801/7.0/USB Flash o bin/122172 fs [fs]: amd(8) automount daemon dies on 6.3-STABLE i386, o bin/121898 fs [nullfs] pwd(1)/getcwd(2) fails with Permission denied o bin/121779 fs [ufs] snapinfo(8) (and related tools?) only work for t o bin/121366 fs [zfs] [patch] Automatic disk scrubbing from periodic(8 o bin/121072 fs [smbfs] mount_smbfs(8) cannot normally convert the cha f kern/120991 fs [panic] [fs] [snapshot] System crashes when manipulati o kern/120483 fs [ntfs] [patch] NTFS filesystem locking changes o kern/120482 fs [ntfs] [patch] Sync style changes between NetBSD and F f kern/119735 fs [zfs] geli + ZFS + samba starting on boot panics 7.0-B o kern/118912 fs [2tb] disk sizing/geometry problem with large array o kern/118713 fs [minidump] [patch] Display media size required for a k o bin/118249 fs mv(1): moving a directory changes its mtime o kern/118107 fs [ntfs] [panic] Kernel panic when accessing a file at N o bin/117315 fs [smbfs] mount_smbfs(8) and related options can't mount o kern/117314 fs [ntfs] Long-filename only NTFS fs'es cause kernel pani o kern/117158 fs [zfs] zpool scrub causes panic if geli vdevs detach on o bin/116980 fs [msdosfs] [patch] mount_msdosfs(8) resets some flags f o kern/116913 fs [ffs] [panic] ffs_blkfree: freeing free block p kern/116608 fs [msdosfs] [patch] msdosfs fails to check mount options o kern/116583 fs [ffs] [hang] System freezes for short time when using o kern/116170 fs [panic] Kernel panic when mounting /tmp o kern/115645 fs [snapshots] [panic] lockmgr: thread 0xc4c00d80, not ex o bin/115361 fs [zfs] mount(8) gets into a state where it won't set/un o kern/114955 fs [cd9660] [patch] [request] support for mask,dirmask,ui o kern/114847 fs [ntfs] [patch] [request] dirmask support for NTFS ala o kern/114676 fs [ufs] snapshot creation panics: snapacct_ufs2: bad blo o bin/114468 fs [patch] [request] add -d option to umount(8) to detach o kern/113852 fs [smbfs] smbfs does not properly implement DFS referral o bin/113838 fs [patch] [request] mount(8): add support for relative p o bin/113049 fs [patch] [request] make quot(8) use getopt(3) and show o kern/112658 fs [smbfs] [patch] smbfs and caching problems (resolves b o kern/111843 fs [msdosfs] Long Names of files are incorrectly created o kern/111782 fs [ufs] dump(8) fails horribly for large filesystems s bin/111146 fs [2tb] fsck(8) fails on 6T filesystem o kern/109024 fs [msdosfs] mount_msdosfs: msdosfs_iconv: Operation not o kern/109010 fs [msdosfs] can't mv directory within fat32 file system o bin/107829 fs [2TB] fdisk(8): invalid boundary checking in fdisk / w o kern/106030 fs [ufs] [panic] panic in ufs from geom when a dead disk o kern/104406 fs [ufs] Processes get stuck in "ufs" state under persist o kern/104133 fs [ext2fs] EXT2FS module corrupts EXT2/3 filesystems o kern/103035 fs [ntfs] Directories in NTFS mounted disc images appear o kern/101324 fs [smbfs] smbfs sometimes not case sensitive when it's s o kern/99290 fs [ntfs] mount_ntfs ignorant of cluster sizes o kern/97377 fs [ntfs] [patch] syntax cleanup for ntfs_ihash.c o kern/95222 fs [iso9660] File sections on ISO9660 level 3 CDs ignored o kern/94849 fs [ufs] rename on UFS filesystem is not atomic o kern/94769 fs [ufs] Multiple file deletions on multi-snapshotted fil o kern/94733 fs [smbfs] smbfs may cause double unlock o kern/93942 fs [vfs] [patch] panic: ufs_dirbad: bad dir (patch from D o kern/92272 fs [ffs] [hang] Filling a filesystem while creating a sna f kern/91568 fs [ufs] [panic] writing to UFS/softupdates DVD media in o kern/91134 fs [smbfs] [patch] Preserve access and modification time a kern/90815 fs [smbfs] [patch] SMBFS with character conversions somet o kern/88657 fs [smbfs] windows client hang when browsing a samba shar o kern/88266 fs [smbfs] smbfs does not implement UIO_NOCOPY and sendfi o kern/87859 fs [smbfs] System reboot while umount smbfs. o kern/86587 fs [msdosfs] rm -r /PATH fails with lots of small files o kern/85326 fs [smbfs] [panic] saving a file via samba to an overquot o kern/84589 fs [2TB] 5.4-STABLE unresponsive during background fsck 2 o kern/80088 fs [smbfs] Incorrect file time setting on NTFS mounted vi o kern/73484 fs [ntfs] Kernel panic when doing `ls` from the client si o bin/73019 fs [ufs] fsck_ufs(8) cannot alloc 607016868 bytes for ino o kern/71774 fs [ntfs] NTFS cannot "see" files on a WinXP filesystem o kern/68978 fs [panic] [ufs] crashes with failing hard disk, loose po o kern/65920 fs [nwfs] Mounted Netware filesystem behaves strange o kern/65901 fs [smbfs] [patch] smbfs fails fsx write/truncate-down/tr o kern/61503 fs [smbfs] mount_smbfs does not work as non-root o kern/55617 fs [smbfs] Accessing an nsmb-mounted drive via a smb expo o kern/51685 fs [hang] Unbounded inode allocation causes kernel to loc o kern/51583 fs [nullfs] [patch] allow to work with devices and socket o kern/36566 fs [smbfs] System reboot with dead smb mount and umount o kern/33464 fs [ufs] soft update inconsistencies after system crash o kern/18874 fs [2TB] 32bit NFS servers export wrong negative values t 169 problems total. From owner-freebsd-fs@FreeBSD.ORG Mon Mar 8 17:11:10 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E40B1065679 for ; Mon, 8 Mar 2010 17:11:10 +0000 (UTC) (envelope-from mwlucas@bewilderbeast.blackhelicopters.org) Received: from bewilderbeast.blackhelicopters.org (bewilderbeast.blackhelicopters.org [198.22.63.8]) by mx1.freebsd.org (Postfix) with ESMTP id 4D5A28FC17 for ; Mon, 8 Mar 2010 17:11:09 +0000 (UTC) Received: from bewilderbeast.blackhelicopters.org (localhost [127.0.0.1]) by bewilderbeast.blackhelicopters.org (8.14.3/8.14.3) with ESMTP id o28Gid1v082005 for ; Mon, 8 Mar 2010 11:44:39 -0500 (EST) (envelope-from mwlucas@bewilderbeast.blackhelicopters.org) Received: (from mwlucas@localhost) by bewilderbeast.blackhelicopters.org (8.14.3/8.14.3/Submit) id o28GidEH082004 for freebsd-fs@freebsd.org; Mon, 8 Mar 2010 11:44:39 -0500 (EST) (envelope-from mwlucas) Date: Mon, 8 Mar 2010 11:44:39 -0500 From: "Michael W. Lucas" To: freebsd-fs@freebsd.org Message-ID: <20100308164439.GA81791@bewilderbeast.blackhelicopters.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.2 (bewilderbeast.blackhelicopters.org [127.0.0.1]); Mon, 08 Mar 2010 11:44:39 -0500 (EST) Subject: diskless clients, OpenSolaris servers, NFS errors X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Mar 2010 17:11:10 -0000 Hi, I'm attempting to boot FreeBSD 8/amd64 and 9 systems diskless, using an OpenSolaris 2009.06 server. It seems that I've hit NFS interoperability problem, and I'd like to know if anyone can suggest solutions or workarounds, or if anyone else is working on this problem. It appears that others have had this problem before, e.g.: http://lists.freebsd.org/pipermail/freebsd-hackers/2008-January/022792.html I asked about this earlier on freebsd-questions, and was referred here (http://lists.freebsd.org/pipermail/freebsd-questions/2009-November/208255.html). But my earlier workaround of using nfsv2 on 9-current systems doesn't seem to work for 8-stable. Symptoms: The client loads pxeboot, loads the kernel, and proceeds to boot. Once it tries to mount root, however, I get the error: NFS ROOT: XXX.XXX.XXX.XXX:/data1/knocker Lookup of /dev for devfs, error: 13 exec /sbin/init: error 13 exec /sbin/oinit: error 13 ... panic no init Packet capture between the FreeBSD client and the OpenSolaris server show that the client uses NFSv2 to load the kernel after pxeboot. When we try to remount root, though, the client switches to NFSv3. When this happens, the server rejects the requests with NFS3ERR_STALE. I can boot a FreeBSD 9 client by forcing it to use nfsv2. When I force nfsv2 on FreeBSD 8, however, I get the error: Lookup of /dev for devfs, error: 2 Packet sniffer shows the error is now NFS_NOENT. Yet pxeboot mounted the root filesystem and loaded the kernel! I'm happy to provide packet traces, debugging, whatever, to sort this out. Thanks, ==ml -- Michael W. Lucas mwlucas@BlackHelicopters.org http://www.MichaelWLucas.com/ Latest book: Cisco Routers for the Desperate, 2nd Edition http://www.CiscoRoutersForTheDesperate.com/ From owner-freebsd-fs@FreeBSD.ORG Mon Mar 8 18:06:12 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F40CA1065672 for ; Mon, 8 Mar 2010 18:06:11 +0000 (UTC) (envelope-from korvus@comcast.net) Received: from mx04.pub.collaborativefusion.com (mx04.pub.collaborativefusion.com [206.210.72.84]) by mx1.freebsd.org (Postfix) with ESMTP id C28518FC19 for ; Mon, 8 Mar 2010 18:06:11 +0000 (UTC) Received: from [192.168.2.164] ([206.210.89.202]) by mx04.pub.collaborativefusion.com (StrongMail Enterprise 4.1.1.4(4.1.1.4-47689)); Mon, 08 Mar 2010 13:23:49 -0500 X-VirtualServerGroup: Default X-MailingID: 00000::00000::00000::00000::::3 X-SMHeaderMap: mid="X-MailingID" X-Destination-ID: freebsd-fs@freebsd.org X-SMFBL: ZnJlZWJzZC1mc0BmcmVlYnNkLm9yZw== Message-ID: <4B953C92.5080606@comcast.net> Date: Mon, 08 Mar 2010 13:06:10 -0500 From: Steve Polyack User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.7) Gecko/20100211 Thunderbird/3.0.1 MIME-Version: 1.0 To: freebsd-stable Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: ZFS hot spares X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Mar 2010 18:06:12 -0000 ZFS in FreeBSD lacks at least one major feature from the Solaris version: hot spares. There is a PR open at http://www.freebsd.org/cgi/query-pr.cgi?pr=134491, but there hasn't been any motion/thoughts posted on it since its creation almost one year ago. I'm aware that on Solaris, hot spare replacement is handled by a few Solaris-specific daemons, zfs-retire and zfs-diagnose, which both plug into the Solaris FMA (Fault Management Architecture). Have there been any thoughts on porting these over or getting something similar running within FreeBSD? With all of the recent SATA/SAS CAM hotplug work now committed, it would be nice to have automatic replacement of hot spares with a future hot-replacement of the failed drive. On the other side, I'd be interested in hearing if anyone has had success in rolling their own scripted solution: i.e. something which polls 'zpool status' looking for failed drives and performing hot-spare replacements automatically. Thanks, Steve Polyack From owner-freebsd-fs@FreeBSD.ORG Mon Mar 8 18:51:54 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9B158106564A for ; Mon, 8 Mar 2010 18:51:54 +0000 (UTC) (envelope-from mattjreimer@gmail.com) Received: from mail-yw0-f204.google.com (mail-yw0-f204.google.com [209.85.211.204]) by mx1.freebsd.org (Postfix) with ESMTP id 55C908FC14 for ; Mon, 8 Mar 2010 18:51:54 +0000 (UTC) Received: by ywh42 with SMTP id 42so2953714ywh.7 for ; Mon, 08 Mar 2010 10:51:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=FQpTCa1pNdWfkVS4xXNasSbKPDsB8W618VAGvW3jRUo=; b=uk68aFjBEQk8j9/hVC6rJU8R2Su9LJ/ZHY+WKp0Gd1dwvz37h5nkTWeoZh7NY1yXMZ aO93h+i/w5fr7ZZu1dqzA4Q3ilijKaHpS+DEi3LZNhfmaa4NM3ROZ02BG+ntgut3Pw2u XB+gKby1ix6uLECYIymIBguGO5IU4SdvLEKeM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=jx/dqYEH9Fy4ZYKHW0r0lvnUaylK4EEffcT+P3OFNDOyxDxVUTSlgm4+NvH1/ClIp9 wmmauwL2IWab9vogBARVwBImu3z5DkWPLbw0d68wHxKRu8wSpOoRd99dzPcKmUgGsZh7 QO33Nxghfm4/PIPHtH/2dCC6yHI5tD/RxM728= MIME-Version: 1.0 Received: by 10.150.65.3 with SMTP id n3mr4470027yba.322.1268074313455; Mon, 08 Mar 2010 10:51:53 -0800 (PST) In-Reply-To: <9EB2D14A-ABED-4DEC-A88F-09B0AB32905E@onpuls.com> References: <9EB2D14A-ABED-4DEC-A88F-09B0AB32905E@onpuls.com> Date: Mon, 8 Mar 2010 10:51:53 -0800 Message-ID: From: Matt Reimer To: Morrison Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS Boot problem after hard shutdown while resilvering RAIDZ1 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Mar 2010 18:51:54 -0000 On Sat, Mar 6, 2010 at 12:04 AM, Morrison wrote: > Hello folks. > > Here are my specs: > FreeBSD Current 8.0 (amd64) > 3x 2TB HDD combined to a RAIDZ1 > GPT > Root is running on ZFS, > I used the following Guideline ( > http://wiki.freebsd.org/RootOnZFS/GPTZFSBoot/RAIDZ1) > > > Here is the problem: > > I had to replace a drive, but chose by accident a spare which hat some > strange malfunction as well. > I used the command: > zpool replace zroot gpt/disk1 > > zpool status > gave me the information that resilvering would take 2500h (increasing). > After a couple of MB were resilvered (very slowly), I decided to shut the > system down safely, which didn't work, so I powered it off by holding the > power switch. > I detached the malfunctioned spare drive and tried to start the system with > the two HDDs, but now I am getting > > ZFS: can only boot from disk, mirror or raidz vdevs > ZFS: i/o error - all block copies unavailable > ZFS: can't read object set for dataset lld > Can't find root filesystem - giving up > ZFS: unexpected object set type lld > ZFS: unexpected object set type lld > > FreeBSD/i386 boot > Default: zroot:/boot/kernel/kernel > boot: > ZFS: unexpected object set type lld > > FreeBSD/i386 boot > Default: zroot:/boot/kernel/kernel > boot:_ > Those 'type lld' suggests you're running an older gptzfsboot. Try the latest from -CURRENT or -STABLE. Newer gptzfsboot can boot from a degraded array. > I used my LiveFS USB and tried with Fixit, after loading opensolaris.ko and > zfs.ko: > zpool import -f zroot > > After that, all HDDs started making noises as if they try to heal/restore > them self. But all commands I entered then, gave me errors like: > /libexec/ld-elf.so.1: Shared object "libavl.so.2" not found, > required by "zpool" > Even sImple commands like "ls" produce simular messages. > I waited one night until the HDDs went silent and rebooted, but without any > luck. > Maybe you should import the pool using an alternate root, like "zpool import -f -R /mnt" so you're not replacing /usr etc. out from under Fixit? Matt From owner-freebsd-fs@FreeBSD.ORG Tue Mar 9 06:45:58 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B2131065670 for ; Tue, 9 Mar 2010 06:45:58 +0000 (UTC) (envelope-from howard@leadmon.net) Received: from mail.leadmon.net (mail.leadmon.net [173.13.218.154]) by mx1.freebsd.org (Postfix) with ESMTP id B62D38FC14 for ; Tue, 9 Mar 2010 06:45:57 +0000 (UTC) Received: from HDLDESKTOP64 (hdl-desktop-64.leadmon.net [10.0.0.3]) (authenticated bits=0) by mail.leadmon.net (8.14.4/8.14.4/LNSG+SCOP+PSBL+LUBL+NJABL+SBL+DSBL+SORBS+CBL+RHSBL) with ESMTP id o296Ced9073508 for ; Tue, 9 Mar 2010 01:12:40 -0500 (EST) (envelope-from howard@leadmon.net) X-DKIM: Sendmail DKIM Filter v2.8.3 mail.leadmon.net o296Ced9073508 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=leadmon.net; s=default; t=1268115160; bh=7S40MuIThjVIGfJNO2iRCeuAntOjIudVGanQdF7hH8g=; h=From:To:Subject:Date:Message-ID:MIME-Version:Content-Type; b=ByD1fePJGuQvqwgsKIBrtNXQHQ/hojtRS/PcFgcSiSxQoVRCG6sSYstpimcZTV3d+ Nndlia8pzHjQS51k3+PBA3FAvisXnh+FCO46dp+iTnM3b7iObbyMVvOCpZ7YrByKlC PX+48frHYpV32IEuvvrKDo6EBGcn3vO5Wy0EHweE= X-DomainKeys: Sendmail DomainKeys Filter v1.0.2 mail.leadmon.net o296Ced9073508 DomainKey-Signature: a=rsa-sha1; s=default; d=leadmon.net; c=simple; q=dns; h=x-senderid:authentication-results:from:to:subject:date: message-id:mime-version:content-type:x-mailer:thread-index:content-language; b=gV+ofSls3EXpp2Pq26o8dDh1fyFMtcjChjpaciG+aQKlqIxLz1DYNcaSwYfWxc+4b 37BMLXX7/4a6bk8EEbQ2YKBZS0t8p8f9raeAXwzAPXB99jmLpAkfYVt4UGMRhEA0NSo vXzLyWU6q3hZquHwK5bgk6859arVKMs6VqcYBfU= X-SenderID: Sendmail Sender-ID Filter v1.0.0 mail.leadmon.net o296Ced9073508 Authentication-Results: mail.leadmon.net; sender-id=pass header.from=howard@leadmon.net; spf=pass smtp.mfrom=howard@leadmon.net From: "Howard Leadmon" To: Date: Tue, 9 Mar 2010 01:12:37 -0500 Message-ID: <00d201cabf4f$8634f6a0$929ee3e0$@net> MIME-Version: 1.0 X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Acq/T4RLyGnQBzLQQv6grOy7KsHf8g== Content-Language: en-us X-Virus-Scanned: clamav-milter 0.95.3 at vorlon.leadmon.net X-Virus-Status: Clean Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Error Message with FBSD 8 & ZFS after doing a cvsup and rebuild, any ideas? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 06:45:58 -0000 I just did a cvsup and updated my amd64 8.0-STABLE machine, and after rebooting with the new kernel and all I am seeing the following on directory access: default_perms_for_dir: sys_acl_get_file(., ACL_TYPE_DEFAULT): Invalid argument, falling back on umask default_perms_for_dir: sys_acl_get_file(., ACL_TYPE_DEFAULT): Invalid argument, falling back on umask Now I know it wasn't doing this before, so something has apparently changed since the update and rebuild. Has anyone run into this issue, or machine something that is known in the current STABLE source tree? Just not sure if this is something I got to worry about, but not really sure how to go about fixing it either.. --- Howard Leadmon From owner-freebsd-fs@FreeBSD.ORG Tue Mar 9 06:48:26 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A6DC4106566B for ; Tue, 9 Mar 2010 06:48:26 +0000 (UTC) (envelope-from michael@fuckner.net) Received: from dedihh.fuckner.net (dedihh.fuckner.net [81.209.183.161]) by mx1.freebsd.org (Postfix) with ESMTP id 652318FC17 for ; Tue, 9 Mar 2010 06:48:26 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by dedihh.fuckner.net (Postfix) with ESMTP id A374B1A7D9; Tue, 9 Mar 2010 07:49:37 +0100 (CET) X-Virus-Scanned: amavisd-new at example.com Received: from dedihh.fuckner.net ([127.0.0.1]) by localhost (dedihh.fuckner.net [127.0.0.1]) (amavisd-new, port 10024) with SMTP id p6Jw5E3TTaji; Tue, 9 Mar 2010 07:49:33 +0100 (CET) Received: from c64.rebootking.de (e176130024.adsl.alicedsl.de [85.176.130.24]) by dedihh.fuckner.net (Postfix) with ESMTPA id E224C1A7C9; Tue, 9 Mar 2010 07:49:32 +0100 (CET) Message-ID: <4B95EF13.5080405@fuckner.net> Date: Tue, 09 Mar 2010 07:47:47 +0100 From: Michael Fuckner User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.8) Gecko/20100303 Thunderbird/3.0.3 MIME-Version: 1.0 To: Howard Leadmon References: <00d201cabf4f$8634f6a0$929ee3e0$@net> In-Reply-To: <00d201cabf4f$8634f6a0$929ee3e0$@net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org Subject: Re: Error Message with FBSD 8 & ZFS after doing a cvsup and rebuild, any ideas? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 06:48:26 -0000 On 03/09/10 07:12, Howard Leadmon wrote: > > > I just did a cvsup and updated my amd64 8.0-STABLE machine, and after > > rebooting with the new kernel and all I am seeing the following on directory > > > > default_perms_for_dir: sys_acl_get_file(., ACL_TYPE_DEFAULT): Invalid > > argument, falling back on umask > > > > default_perms_for_dir: sys_acl_get_file(., ACL_TYPE_DEFAULT): Invalid > > argument, falling back on umask > > Did you upgrade rsync as well? Where exactly are the messages coming from? Regards, Michael! From owner-freebsd-fs@FreeBSD.ORG Tue Mar 9 08:07:17 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 22E3F106566B for ; Tue, 9 Mar 2010 08:07:17 +0000 (UTC) (envelope-from howard@leadmon.net) Received: from mail.leadmon.net (mail.leadmon.net [173.13.218.154]) by mx1.freebsd.org (Postfix) with ESMTP id 8CFE08FC0C for ; Tue, 9 Mar 2010 08:07:16 +0000 (UTC) Received: from HDLDESKTOP64 (hdl-desktop-64.leadmon.net [10.0.0.3]) (authenticated bits=0) by mail.leadmon.net (8.14.4/8.14.4/LNSG+SCOP+PSBL+LUBL+NJABL+SBL+DSBL+SORBS+CBL+RHSBL) with ESMTP id o2987FP0058394; Tue, 9 Mar 2010 03:07:15 -0500 (EST) (envelope-from howard@leadmon.net) X-DKIM: Sendmail DKIM Filter v2.8.3 mail.leadmon.net o2987FP0058394 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=leadmon.net; s=default; t=1268122035; bh=DZ/TDwtEwvt7FZCKnNfzB5JR395Hx5Jl/2zCjmdOh64=; h=From:To:Cc:References:In-Reply-To:Subject:Date:Message-ID: MIME-Version:Content-Type:Content-Transfer-Encoding; b=VVrbhLfnDkh6KWyBWAuK3aICetvZU99WhK/Ip7cAMjSHlvICG+bsOoUDZAcyGXqOz x2pwOUe+v/gbrrDaWVy1xji1/7rMa//fKUtzuvyVuoK/cMNBe/DD3dB9Hg8wtNEa1x Y/60tlLq5nPze7zpO9uGvcMV9/lXOK1yGQ2JDfuc= X-DomainKeys: Sendmail DomainKeys Filter v1.0.2 mail.leadmon.net o2987FP0058394 DomainKey-Signature: a=rsa-sha1; s=default; d=leadmon.net; c=simple; q=dns; h=x-senderid:authentication-results:from:to:cc:references: in-reply-to:subject:date:message-id:mime-version:content-type: content-transfer-encoding:x-mailer:thread-index:content-language; b=JIh6ujQ3K1zF4SUfFOLAgHk2cAtxJakax8h+QtdFhYZEQIL3geJCfIVmIyi8EWD6Z 7XQ17XjGtHouAEs6mhTyBh/JL6mxBI8K9jV/2QLH6/ltRDec2zrF730n76qXi2ys+tr 4BSs4DMlytov+0eWHWiQ1MKk5IiQv1eOeIeXrbM= X-SenderID: Sendmail Sender-ID Filter v1.0.0 mail.leadmon.net o2987FP0058394 Authentication-Results: mail.leadmon.net; sender-id=pass header.from=howard@leadmon.net; spf=pass smtp.mfrom=howard@leadmon.net From: "Howard Leadmon" To: "'Michael Fuckner'" References: <00d201cabf4f$8634f6a0$929ee3e0$@net> <4B95EF13.5080405@fuckner.net> In-Reply-To: <4B95EF13.5080405@fuckner.net> Date: Tue, 9 Mar 2010 03:07:11 -0500 Message-ID: <00e001cabf5f$87bcdbe0$973693a0$@net> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Mailer: Microsoft Office Outlook 12.0 Thread-Index: Acq/VImWMmny35/vTCuCI3qWaGun4QAClkKw Content-Language: en-us X-Virus-Scanned: clamav-milter 0.95.3 at vorlon.leadmon.net X-Virus-Status: Clean Cc: freebsd-fs@freebsd.org Subject: RE: Error Message with FBSD 8 & ZFS after doing a cvsup and rebuild, any ideas? X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 08:07:17 -0000 Yep, your right, it's rsync causing the generation of the messages. I did also run a portupgrade as well, and I do believe there was an update to rsync if I recall, but am not 100% positive. I just ran the update script for the clamav databases, and saw the following during execution: + rsync --no-motd --files-from=/var/db/clamav-addon/configs/ss-include-dbs.txt -crtz --timeout=30 --stats rsync://69.63.144.18/sanesecurity /var/db/clamav-addon/ss-dbs default_perms_for_dir: sys_acl_get_file(., ACL_TYPE_DEFAULT): Invalid argument, falling back on umask default_perms_for_dir: sys_acl_get_file(., ACL_TYPE_DEFAULT): Invalid argument, falling back on umask + rsync -cqrt /var/db/clamav-addon/ss-dbs/jurlbl.ndb /var/db/clamav default_perms_for_dir: sys_acl_get_file(., ACL_TYPE_DEFAULT): Invalid argument, falling back on umask default_perms_for_dir: sys_acl_get_file(., ACL_TYPE_DEFAULT): Invalid argument, falling back on umask + rsync -cqrt /var/db/clamav-addon/ss-dbs/jurlbla.ndb /var/db/clamav default_perms_for_dir: sys_acl_get_file(., ACL_TYPE_DEFAULT): Invalid argument, falling back on umask default_perms_for_dir: sys_acl_get_file(., ACL_TYPE_DEFAULT): Invalid argument, falling back on umask So I take it there is some loss of love between rsync and zfs, but hopefully nothing major to worry about.. Thanks for the quick reply.. --- Howard Leadmon > -----Original Message----- > From: Michael Fuckner [mailto:michael@fuckner.net] > Sent: Tuesday, March 09, 2010 1:48 AM > To: Howard Leadmon > Cc: freebsd-fs@freebsd.org > Subject: Re: Error Message with FBSD 8 & ZFS after doing a cvsup and > rebuild,any ideas? > > On 03/09/10 07:12, Howard Leadmon wrote: > > > > > > I just did a cvsup and updated my amd64 8.0-STABLE machine, and after > > > > rebooting with the new kernel and all I am seeing the following on > directory > > > > > > > > default_perms_for_dir: sys_acl_get_file(., ACL_TYPE_DEFAULT): Invalid > > > > argument, falling back on umask > > > > > > > > default_perms_for_dir: sys_acl_get_file(., ACL_TYPE_DEFAULT): Invalid > > > > argument, falling back on umask > > > > > Did you upgrade rsync as well? Where exactly are the messages coming from? > > Regards, > Michael! From owner-freebsd-fs@FreeBSD.ORG Tue Mar 9 09:18:24 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1A87F106575E; Tue, 9 Mar 2010 09:18:23 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 28EDD8FC0C; Tue, 9 Mar 2010 09:18:22 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 903B045C9F; Tue, 9 Mar 2010 10:18:20 +0100 (CET) Received: from localhost (pdawidek.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 86E5B45C9B; Tue, 9 Mar 2010 10:18:15 +0100 (CET) Date: Tue, 9 Mar 2010 10:18:16 +0100 From: Pawel Jakub Dawidek To: Steve Polyack Message-ID: <20100309091815.GB3155@garage.freebsd.pl> References: <4B953C92.5080606@comcast.net> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="s9fJI615cBHmzTOP" Content-Disposition: inline In-Reply-To: <4B953C92.5080606@comcast.net> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org, Michel Bouissou , freebsd-stable Subject: Re: ZFS hot spares X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 09:18:24 -0000 --s9fJI615cBHmzTOP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Mar 08, 2010 at 01:06:10PM -0500, Steve Polyack wrote: > ZFS in FreeBSD lacks at least one major feature from the Solaris=20 > version: hot spares. There is a PR open at=20 > http://www.freebsd.org/cgi/query-pr.cgi?pr=3D134491, but there hasn't bee= n=20 > any motion/thoughts posted on it since its creation almost one year ago. >=20 > I'm aware that on Solaris, hot spare replacement is handled by a few=20 > Solaris-specific daemons, zfs-retire and zfs-diagnose, which both plug=20 > into the Solaris FMA (Fault Management Architecture). Have there been=20 > any thoughts on porting these over or getting something similar running= =20 > within FreeBSD? With all of the recent SATA/SAS CAM hotplug work now=20 > committed, it would be nice to have automatic replacement of hot spares= =20 > with a future hot-replacement of the failed drive. >=20 > On the other side, I'd be interested in hearing if anyone has had=20 > success in rolling their own scripted solution: i.e. something which=20 > polls 'zpool status' looking for failed drives and performing hot-spare= =20 > replacements automatically. Currently FreeBSD's ZFS sends various events to devd. It should be possible to implement some scripts (or maybe reuse zfs-retire/zfs-diagnose?) to perform 'zpool replace' when disk disappears, etc. This shouldn't be very hard modulo bugs in FreeBSD/ZFS as this functionality, because unused, wasn't tested. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --s9fJI615cBHmzTOP Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkuWElcACgkQForvXbEpPzQuMACfTjbHEMV5MsAbZmmP7rK4yw0y aIMAoO1gVToaXGpeGNsUwvS3SPC5KuzC =bzbO -----END PGP SIGNATURE----- --s9fJI615cBHmzTOP-- From owner-freebsd-fs@FreeBSD.ORG Tue Mar 9 14:44:28 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 862A3106564A for ; Tue, 9 Mar 2010 14:44:28 +0000 (UTC) (envelope-from johan@stromnet.se) Received: from core.stromnet.se (core.stromnet.se [83.218.84.131]) by mx1.freebsd.org (Postfix) with ESMTP id C4E408FC12 for ; Tue, 9 Mar 2010 14:44:27 +0000 (UTC) Received: from localhost (core.stromnet.se [83.218.84.131]) by core.stromnet.se (Postfix) with ESMTP id E59E9F57419 for ; Tue, 9 Mar 2010 15:26:28 +0100 (CET) X-Virus-Scanned: amavisd-new at stromnet.se X-Spam-Flag: NO X-Spam-Score: 0.727 X-Spam-Level: X-Spam-Status: No, score=0.727 tagged_above=-1000 required=6.2 tests=[AWL=0.751, BAYES_00=-2.599, RCVD_IN_PBL=0.905, RCVD_IN_SORBS_DUL=0.877, RDNS_DYNAMIC=0.1, SPF_FAIL=0.693] autolearn=no Received: from core.stromnet.se ([83.218.84.131]) by localhost (core.stromnet.se [83.218.84.131]) (amavisd-new, port 10024) with ESMTP id Y0uvUZM-mHly for ; Tue, 9 Mar 2010 15:26:25 +0100 (CET) Received: from johan-mp.stromnet.se (c-b4fe70d5.016-325-67626721.cust.bredbandsbolaget.se [213.112.254.180]) by core.stromnet.se (Postfix) with ESMTP id D14D2F5740D for ; Tue, 9 Mar 2010 15:26:25 +0100 (CET) Message-Id: <8BA8475A-424E-489B-B643-8757F375320B@stromnet.se> From: =?ISO-8859-1?Q?Johan_Str=F6m?= To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v936) Date: Tue, 9 Mar 2010 15:26:25 +0100 X-Mailer: Apple Mail (2.936) Subject: ZFS: zpool import hang on "zio->io_cv)" (with DDB output). Help needed! X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 14:44:28 -0000 Hi List! I'm in the tedious process of upgrading my pool on my FreeBSD 8.0 box (with a pool from 7.x). Yesterday I pulled a few disks from my mirrored pool (one pool with multiple mirrors), in order to be able to (free ports) plug in new ones and build my brand new pool. After removing disks, the pool got into state DEGRADED ofcourse, since some of the disks where gone, but data was still there. I did zpool detach on the removed disks, and the pool was ONLINE again, all fine! Then I rebooted to make another disk available through my old LSI MegaRaid card (didnt take time to figure out cryptic syntax of the megarc CLI util). On boot, I was met by something similar to this: pool: tank state: UNAVAIL status: One or more devices could not be used because the label is missing or invalid. There are insufficient replicas for the pool to continue functioning. action: Destroy and re-create the pool from a backup source. see: http://www.sun.com/msg/ZFS-8000-5E scrub: none requested config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas ad10s1d ONLINE 0 0 0 mirror DEGRADED 0 0 0 ad12 FAULTED 0 0 0 corrupted data ad16 UNAVAIL 0 0 0 corrupted data mirror DEGRADED 0 0 0 ad20 FAULTED 0 0 0 corrupted data ad18 UNAVAIL 0 0 0 corrupted data mirror UNAVAIL 0 0 0 insufficient replicas ad6 UNAVAIL 0 0 0 corrupted data ad4 FAULTED 0 0 0 corrupted data mirror ONLINE 0 0 0 amrd0 ONLINE 0 0 0 amrd1 ONLINE 0 0 0 The reason the disks was still there was due to me haveing an old zfs.cache file in the boot (it boots from anothre drive, but thats another story..) In this case, a export/import did the trick, after reimporting, the pool was back online. All fine. I created a new pool and did some bonnie++ testing on it, and suddenly the box paniced or something (I didn't have dumpon enabled :/ and didn't see the screen until it rebooted).. Now my problems come. The box came up again, with the above output. I tried zpool export again, fine. Zpool import however, hang. Waited an hour, nothing. After rebuilding the kernel with DDB/witness and did the import again, i've managed to get this output (im not really sure what is usable here): back-1 # zpool import load: 0.29 cmd: zpool 3193 [zio->io_cv)] 2.54r 0.00u 0.01s 0% 2236k In DDB: > tr 3193 Tracing pid 3193 tid 100122 td 0xffffff00035ca390 sched_switch() at sched_switch+0xde mi_switch() at mi_switch+0x170 sleepq_wait() at sleepq_wait+0x44 _cv_wait() at _cv_wait+0x13c zio_wait() at zio_wait+0x61 arc_read_nolock() at arc_read_nolock+0x345 dmu_objset_open_impl() at dmu_objset_open_impl+0xd0 dsl_pool_open() at dsl_pool_open+0x5a spa_load() at spa_load+0x31b spa_tryimport() at spa_tryimport+0xa9 zfs_ioc_pool_tryimport() at zfs_ioc_pool_tryimport+0x3f zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xf6 ioctl() at ioctl+0xfd syscall() at syscall+0x19e Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x8010eb86c, rsp = 0x7fffffff8e28, rbp = 0x801323300 --- db> show thread 100122 Thread 100122 at 0xffffff00035ca390: proc (pid 3193): 0xffffff000383c460 name: zpool stack: 0xffffff805740a000-0xffffff805740dfff flags: 0x44 pflags: 0x10000 state: INHIBITED: {SLEEPING} wmesg: zio->io_cv) wchan: 0xffffff00506e5858 priority: 131 container lock: sleepq chain (0xffffffff80c61e68) db> db> show sleepchain 10012 db> show sleepchain 3193 thread 100122 (pid 3193, zpool) sleeping on 0xffffff00506e5858 "zio- >io_cv)" db> show lock 0xffffff00506e5858 class: spin mutex name: zio->io_cv) flags: {SPIN} state: {OWNED} The box is currenlty in this state, so if you reply to me now I can continue to do debugging according to instructions. Since I cannot get anywhere on my own with this, all and any help is appreciated since I really need this pool to get online.. dmesg is posted below (including some LOR?) Thanks! Johan DMESG: Copyright (c) 1992-2009 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-RELEASE-p2 #10: Tue Mar 9 12:44:15 CET 2010 johan@back-1.stromnet.se:/usr/obj/usr/src/sys/BACK1 WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz (2666.68-MHz K8- class CPU) Origin = "GenuineIntel" Id = 0x6fb Stepping = 11 Features = 0xbfebfbff < FPU ,VME ,DE ,PSE ,TSC ,MSR ,PAE ,MCE ,CX8 ,APIC ,SEP ,MTRR ,PGE ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2 =0xe3fd AMD Features=0x20100800 AMD Features2=0x1 TSC: P-state invariant real memory = 2147483648 (2048 MB) avail memory = 2040631296 (1946 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs FreeBSD/SMP: 1 package(s) x 2 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 ioapic0: Changing APIC ID to 2 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 cryptosoft0: on motherboard acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, 7f4e0000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 vgapci0: port 0xe000-0xe007 mem 0xe6300000-0xe637ffff,0xd0000000-0xdfffffff,0xe6000000-0xe60fffff irq 16 at device 2.0 on pci0 agp0: on vgapci0 agp0: detected 7164k stolen memory agp0: aperture size is 256M uhci0: port 0xe100-0xe11f irq 16 at device 26.0 on pci0 uhci0: [ITHREAD] uhci0: LegSup = 0x2f00 usbus0: on uhci0 uhci1: port 0xe200-0xe21f irq 21 at device 26.1 on pci0 uhci1: [ITHREAD] uhci1: LegSup = 0x2f00 usbus1: on uhci1 uhci2: port 0xe600-0xe61f irq 18 at device 26.2 on pci0 uhci2: [ITHREAD] uhci2: LegSup = 0x2f00 usbus2: on uhci2 ehci0: mem 0xe6384000-0xe63843ff irq 18 at device 26.7 on pci0 ehci0: [ITHREAD] usbus3: EHCI version 1.0 usbus3: on ehci0 hdac0: mem 0xe6380000-0xe6383fff irq 22 at device 27.0 on pci0 hdac0: HDA Driver Revision: 20090624_0136 hdac0: [ITHREAD] pcib1: irq 16 at device 28.0 on pci0 pci1: on pcib1 pcib2: irq 18 at device 28.2 on pci0 pci2: on pcib2 em0: port 0xa000-0xa01f mem 0xe1020000-0xe103ffff,0xe1000000-0xe101ffff irq 18 at device 0.0 on pci2 em0: Using MSI interrupt em0: [FILTER] em0: Ethernet address: 00:1b:21:05:00:b4 pcib3: irq 19 at device 28.3 on pci0 pci3: on pcib3 atapci0: port 0xb000-0xb007,0xb100-0xb103,0xb200-0xb207,0xb300-0xb303,0xb400-0xb40f mem 0xe6100000-0xe6101fff irq 19 at device 0.0 on pci3 atapci0: [ITHREAD] atapci0: AHCI called from vendor specific driver atapci0: AHCI v1.00 controller with 2 3Gbps ports, PM supported ata2: on atapci0 ata2: [ITHREAD] ata3: on atapci0 ata3: [ITHREAD] ata4: on atapci0 ata4: [ITHREAD] pcib4: irq 16 at device 28.4 on pci0 pci4: on pcib4 re0: port 0xc000-0xc0ff mem 0xe3000000-0xe3000fff irq 16 at device 0.0 on pci4 re0: Using 1 MSI messages re0: Chip rev. 0x38000000 re0: MAC rev. 0x00000000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto re0: Ethernet address: 00:1a:4d:5a:97:87 re0: [FILTER] uhci3: port 0xe300-0xe31f irq 23 at device 29.0 on pci0 uhci3: [ITHREAD] uhci3: LegSup = 0x2f00 usbus4: on uhci3 uhci4: port 0xe400-0xe41f irq 19 at device 29.1 on pci0 uhci4: [ITHREAD] uhci4: LegSup = 0x2f00 usbus5: on uhci4 uhci5: port 0xe500-0xe51f irq 18 at device 29.2 on pci0 uhci5: [ITHREAD] uhci5: LegSup = 0x2f00 usbus6: on uhci5 ehci1: mem 0xe6385000-0xe63853ff irq 23 at device 29.7 on pci0 ehci1: [ITHREAD] usbus7: EHCI version 1.0 usbus7: on ehci1 pcib5: at device 30.0 on pci0 pci5: on pcib5 amr0: mem 0xe6200000-0xe620ffff irq 20 at device 0.0 on pci5 amr0: Using 64-bit DMA amr0: [ITHREAD] amr0: delete logical drives supported by controller amr0: Firmware 713S, BIOS G121, 64MB RAM xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xd000-0xd07f mem 0xe5004000-0xe500407f irq 19 at device 1.0 on pci5 miibus1: on xl0 xlphy0: <3c905C 10/100 internal PHY> PHY 24 on miibus1 xlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto xl0: Ethernet address: 00:04:76:ef:c6:36 xl0: [ITHREAD] skc0: port 0xd100-0xd1ff mem 0xe5000000-0xe5003fff irq 18 at device 2.0 on pci5 pci0:5:2:0: invalid VPD data, remain 0xfc skc0: SysKonnect SK-NET Gigabit Ethernet Adapter SK-9843 SX rev. (0x0) sk0: on skc0 sk0: Ethernet address: 00:00:5a:98:43:68 miibus2: on sk0 xmphy0: PHY 0 on miibus2 xmphy0: 1000baseSX, 1000baseSX-FDX, auto skc0: [ITHREAD] isab0: at device 31.0 on pci0 isa0: on isab0 atapci1: port 0xe700-0xe707,0xe800-0xe803,0xe900-0xe907,0xea00-0xea03,0xeb00-0xeb1f mem 0xe6386000-0xe63867ff irq 19 at device 31.2 on pci0 atapci1: [ITHREAD] atapci1: AHCI called from vendor specific driver atapci1: AHCI v1.20 controller with 6 3Gbps ports, PM supported ata5: on atapci1 ata5: [ITHREAD] ata6: on atapci1 ata6: [ITHREAD] ata7: on atapci1 ata7: [ITHREAD] ata8: on atapci1 ata8: [ITHREAD] ata9: on atapci1 ata9: [ITHREAD] ata10: on atapci1 ata10: [ITHREAD] pci0: at device 31.3 (no driver attached) atrtc0: port 0x70-0x73 on acpi0 fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0 fdc0: [FILTER] uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] cpu0: on acpi0 est0: on cpu0 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 82a082a0600082a device_attach: est0 attach returned 6 p4tcc0: on cpu0 cpu1: on acpi0 est1: on cpu1 est: CPU supports Enhanced Speedstep, but is not recognized. est: cpu_vendor GenuineIntel, msr 82a082a0600082a device_attach: est1 attach returned 6 p4tcc1: on cpu1 orm0: at iomem 0xcc000-0xcc7ff,0xcd000-0xcefff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present; to enable, add "vfs.zfs.prefetch_disable=0" to /boot/ loader.conf. ZFS filesystem version 13 ZFS storage pool version 13 Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing. usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 usbus4: 12Mbps Full Speed USB v1.0 usbus5: 12Mbps Full Speed USB v1.0 usbus6: 12Mbps Full Speed USB v1.0 usbus7: 480Mbps High Speed USB v2.0 ad4: 476938MB at ata2-master SATA300 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ugen4.1: at usbus4 uhub4: on usbus4 ugen5.1: at usbus5 uhub5: on usbus5 ugen6.1: at usbus6 uhub6: on usbus6 ugen7.1: at usbus7 uhub7: on usbus7 ad6: 1907729MB at ata3-master SATA300 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered uhub4: 2 ports with 2 removable, self powered uhub5: 2 ports with 2 removable, self powered uhub6: 2 ports with 2 removable, self powered ad10: 715404MB at ata5-master SATA300 ad12: 305245MB at ata6-master SATA150 ad14: 715404MB at ata7-master SATA300 ad16: 1907729MB at ata8-master SATA300 GEOM_MIRROR: Device mirror/gm1a launched (1/1). GEOM_MIRROR: Device mirror/gm1b launched (1/1). ad18: 1907729MB at ata9-master SATA300 GEOM_MIRROR: Device mirror/swap launched (1/1). ad20: 286187MB at ata10-master SATA150 hdac0: HDA Codec #2: Realtek ALC885 pcm0: at cad 2 nid 1 on hdac0 pcm1: at cad 2 nid 1 on hdac0 pcm2: at cad 2 nid 1 on hdac0 pcm3: at cad 2 nid 1 on hdac0 pcm4: at cad 2 nid 1 on hdac0 pcm5: at cad 2 nid 1 on hdac0 amr0: delete logical drives supported by controller amrd0: on amr0 amrd0: 476935MB (976762880 sectors) RAID 0 (optimal) amrd1: on amr0 amrd1: 476935MB (976762880 sectors) RAID 0 (optimal) amrd2: on amr0 amrd2: 1907724MB (3907018752 sectors) RAID 0 (optimal) SMP: AP CPU #1 Launched! WARNING: WITNESS option enabled, expect reduced performance. Root mount waiting for: usbus7 usbus3 uhub3: 6 ports with 6 removable, self powered uhub7: 6 ports with 6 removable, self powered Trying to mount root from zfs:zroot ugen0.2: at usbus0 ugen1.2: at usbus1 uma_zalloc_arg: zone "256" with the following non-sleepable locks held: exclusive rw ifnet_rw (ifnet_rw) r = 0 (0xffffffff80e01f60) locked @ / usr/src/sys/net/if.c:402 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2c witness_warn() at witness_warn+0x2c2 uma_zalloc_arg() at uma_zalloc_arg+0x29d malloc() at malloc+0x5d if_grow() at if_grow+0x2f if_alloc() at if_alloc+0x2b3 gif_clone_create() at gif_clone_create+0x53 ifc_simple_create() at ifc_simple_create+0x89 if_clone_createif() at if_clone_createif+0x64 ifioctl() at ifioctl+0x6b5 kern_ioctl() at kern_ioctl+0xf6 ioctl() at ioctl+0xfd syscall() at syscall+0x19e Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800b8286c, rsp = 0x7fffffffe4a8, rbp = 0x7fffffffef6e --- lock order reversal: 1st 0xffffffff80c093e0 pf task mtx (pf task mtx) @ /usr/src/sys/ contrib/pf/net/pf_ioctl.c:1393 2nd 0xffffffff80e01f60 ifnet_rw (ifnet_rw) @ /usr/src/sys/net/if.c: 2034 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2c witness_checkorder() at witness_checkorder+0x66f _rw_rlock() at _rw_rlock+0x29 ifunit() at ifunit+0x22 pfioctl() at pfioctl+0x262a devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xf6 ioctl() at ioctl+0xfd syscall() at syscall+0x19e Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x80099886c, rsp = 0x7fffffffdb68, rbp = 0x7fffffffdc20 --- lock order reversal: 1st 0xffffff00500c3098 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1054 2nd 0xffffff005010f448 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c: 2083 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2c witness_checkorder() at witness_checkorder+0x66f __lockmgr_args() at __lockmgr_args+0x475 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 _vn_lock() at _vn_lock+0x47 vget() at vget+0x56 devfs_allocv() at devfs_allocv+0x103 devfs_root() at devfs_root+0x48 vfs_donmount() at vfs_donmount+0xf43 nmount() at nmount+0x63 syscall() at syscall+0x19e Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (378, FreeBSD ELF64, nmount), rip = 0x8007b04dc, rsp = 0x7fffffffdd28, rbp = 0x800a04048 --- n r tun0: link state changed to UP lock order reversal: 1st 0xffffff00500c3098 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1200 2nd 0xffffff005010f270 syncer (syncer) @ /usr/src/sys/kern/ vfs_subr.c:2188 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2c witness_checkorder() at witness_checkorder+0x66f __lockmgr_args() at __lockmgr_args+0x475 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 _vn_lock() at _vn_lock+0x47 vrele() at vrele+0xc3 dounmount() at dounmount+0x269 unmount() at unmount+0x27e syscall() at syscall+0x19e Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (22, FreeBSD ELF64, unmount), rip = 0x8006a09bc, rsp = 0x7fffffffde18, rbp = 0 --- KDB: enter: manual escape to debugger From owner-freebsd-fs@FreeBSD.ORG Tue Mar 9 15:28:39 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id DF04F1065672 for ; Tue, 9 Mar 2010 15:28:39 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 31A528FC14 for ; Tue, 9 Mar 2010 15:28:38 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id RAA27204 for ; Tue, 09 Mar 2010 17:28:37 +0200 (EET) (envelope-from avg@icyb.net.ua) Message-ID: <4B966925.1040609@icyb.net.ua> Date: Tue, 09 Mar 2010 17:28:37 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.23 (X11/20100211) MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org X-Enigmail-Version: 0.95.7 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: ZFS ACL usage question X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 15:28:40 -0000 I have a usage question on ZFS ACL. Perhaps it's something trivial that should have been asked on questions@, apologies in that case. Let's say for simplicity that I want some directory to be equally shared by two users. Both should have full access and new files created by one user should still be fully accessible by the other. I can't seem to be able to configure AСLs to get that. Let's say the users are user1 and user2. The original directory is owned by user1: $ ls -ld ~/testdir drwxrwxr-x+ 22 user1 group0 26 9 Mar 13:01 /home/user1/testdir I then issue the following commands: $ setfacl -b -m user:user1:rwxAWCo:fd:allow ~/testdir $ setfacl -m user:user1::fd:deny ~/testdir $ setfacl -m user:user2:rwxAWCo:fd:allow ~/testdir $ setfacl -m user:user2::fd:deny ~/testdir $ getfacl ~/testdir # file: /home/user1/testdir # owner: user1 # group: group0 user:user2:--------------:fd----:deny user:user2:rwx----A-W-Co-:fd----:allow user:user1:--------------:fd----:deny user:user1:rwx----A-W-Co-:fd----:allow owner@:--------------:------:deny owner@:rwxp---A-W-Co-:------:allow group@:--------------:------:deny group@:rwxp----------:------:allow everyone@:-w-p---A-W-Co-:------:deny everyone@:r-x---a-R-c--s:------:allow Then I create a new file as user1 like this (umask is set to 022): $ touch ~/testdir/test $ ls -ld ~/testdir/test -rw-r--r--+ 1 user1 group0 0 9 Mar 13:01 /home/user1/testdir/test $ getfacl ~/testdir/test # file: /home/user1/testdir/test # owner: user1 # group: group0 user:user2:--------------:------:deny user:user2:-wx-----------:------:deny user:user2:rwx----A-W----:------:allow user:user1:--------------:------:deny user:user1:--x-----------:------:deny user:user1:rwx----A-W----:------:allow owner@:--x-----------:------:deny owner@:rw-p---A-W-Co-:------:allow group@:-wxp----------:------:deny group@:r-------------:------:allow everyone@:-wxp---A-W-Co-:------:deny everyone@:r-----a-R-c--s:------:allow So now there two deny entries for both users and one of them makes sure that user2 can not modify the file. What am I doing wrong? :-) -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Tue Mar 9 16:22:27 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 235AD1065672 for ; Tue, 9 Mar 2010 16:22:27 +0000 (UTC) (envelope-from johan@stromnet.se) Received: from core.stromnet.se (core.stromnet.se [83.218.84.131]) by mx1.freebsd.org (Postfix) with ESMTP id 38E538FC2E for ; Tue, 9 Mar 2010 16:22:25 +0000 (UTC) Received: from localhost (core.stromnet.se [83.218.84.131]) by core.stromnet.se (Postfix) with ESMTP id EAA82F57419; Tue, 9 Mar 2010 17:22:23 +0100 (CET) X-Virus-Scanned: amavisd-new at stromnet.se X-Spam-Flag: NO X-Spam-Score: 0.724 X-Spam-Level: X-Spam-Status: No, score=0.724 tagged_above=-1000 required=6.2 tests=[AWL=0.748, BAYES_00=-2.599, RCVD_IN_PBL=0.905, RCVD_IN_SORBS_DUL=0.877, RDNS_DYNAMIC=0.1, SPF_FAIL=0.693] autolearn=no Received: from core.stromnet.se ([83.218.84.131]) by localhost (core.stromnet.se [83.218.84.131]) (amavisd-new, port 10024) with ESMTP id jMYs0GRePEA0; Tue, 9 Mar 2010 17:22:20 +0100 (CET) Received: from johan-mp.stromnet.se (c-b4fe70d5.016-325-67626721.cust.bredbandsbolaget.se [213.112.254.180]) by core.stromnet.se (Postfix) with ESMTP id A4839F5740D; Tue, 9 Mar 2010 17:22:20 +0100 (CET) Message-Id: From: =?ISO-8859-1?Q?Johan_Str=F6m?= To: =?ISO-8859-1?Q?Johan_Str=F6m?= In-Reply-To: <8BA8475A-424E-489B-B643-8757F375320B@stromnet.se> Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Apple Message framework v936) Date: Tue, 9 Mar 2010 17:22:20 +0100 References: <8BA8475A-424E-489B-B643-8757F375320B@stromnet.se> X-Mailer: Apple Mail (2.936) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS: zpool import hang on "zio->io_cv)" (with DDB output). Help needed! X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 16:22:27 -0000 Some followup, pjd tried to help me on IRC, without getting much =20 further on the actual problem. Some more information: Output from DDB PS: http://www.stromnet.se/~johan/back-1-ddb-ps.txt Output from alltrace: = http://www.stromnet.se/~johan/back-1-ddb-alltrace.txt db> show alllocks Process 2738 (sshd) thread 0xffffff0050418000 (100178) Process 12 (intr) thread 0xffffff0002407ab0 (100018) db> show lockedvnods Locked vnodes db > Also tried accessing the disks (to make sure interrupts etc where =20 alive) ,using dd if=3D/dev/adXX of=3D/dev/null count=3D1 on every disk = on =20 the system, works fine. Rebooted the system and tried with zfs debugging enabled too, and =20 doing zpool import tank instead of just zpool import. no difference. =20 Hanging there now. Johan On Mar 9, 2010, at 15:26 , Johan Str=F6m wrote: > Hi List! > > I'm in the tedious process of upgrading my pool on my FreeBSD 8.0 =20 > box (with a pool from 7.x). Yesterday I pulled a few disks from my =20 > mirrored pool (one pool with multiple mirrors), in order to be able =20= > to (free ports) plug in new ones and build my brand new pool. > After removing disks, the pool got into state DEGRADED ofcourse, =20 > since some of the disks where gone, but data was still there. I did =20= > zpool detach on the removed disks, and the pool was ONLINE again, =20 > all fine! > Then I rebooted to make another disk available through my old LSI =20 > MegaRaid card (didnt take time to figure out cryptic syntax of the =20 > megarc CLI util). On boot, I was met by something similar to this: > > pool: tank > state: UNAVAIL > status: One or more devices could not be used because the label is =20 > missing > or invalid. There are insufficient replicas for the pool to =20= > continue > functioning. > action: Destroy and re-create the pool from a backup source. > see: http://www.sun.com/msg/ZFS-8000-5E > scrub: none requested > config: > > NAME STATE READ WRITE CKSUM > tank UNAVAIL 0 0 0 insufficient replicas > ad10s1d ONLINE 0 0 0 > mirror DEGRADED 0 0 0 > ad12 FAULTED 0 0 0 corrupted data > ad16 UNAVAIL 0 0 0 corrupted data > mirror DEGRADED 0 0 0 > ad20 FAULTED 0 0 0 corrupted data > ad18 UNAVAIL 0 0 0 corrupted data > mirror UNAVAIL 0 0 0 insufficient replicas > ad6 UNAVAIL 0 0 0 corrupted data > ad4 FAULTED 0 0 0 corrupted data > mirror ONLINE 0 0 0 > amrd0 ONLINE 0 0 0 > amrd1 ONLINE 0 0 0 > > The reason the disks was still there was due to me haveing an old =20 > zfs.cache file in the boot (it boots from anothre drive, but thats =20 > another story..) > > In this case, a export/import did the trick, after reimporting, the =20= > pool was back online. All fine. I created a new pool and did some =20 > bonnie++ testing on it, and suddenly the box paniced or something (I =20= > didn't have dumpon enabled :/ and didn't see the screen until it =20 > rebooted).. > > Now my problems come. The box came up again, with the above output. =20= > I tried zpool export again, fine. Zpool import however, hang. Waited =20= > an hour, nothing. After rebuilding the kernel with DDB/witness and =20 > did the import again, i've managed to get this output (im not really =20= > sure what is usable here): > > back-1 # zpool import > load: 0.29 cmd: zpool 3193 [zio->io_cv)] 2.54r 0.00u 0.01s 0% 2236k > > > > In DDB: > > tr 3193 > > Tracing pid 3193 tid 100122 td 0xffffff00035ca390 > sched_switch() at sched_switch+0xde > mi_switch() at mi_switch+0x170 > sleepq_wait() at sleepq_wait+0x44 > _cv_wait() at _cv_wait+0x13c > zio_wait() at zio_wait+0x61 > arc_read_nolock() at arc_read_nolock+0x345 > dmu_objset_open_impl() at dmu_objset_open_impl+0xd0 > dsl_pool_open() at dsl_pool_open+0x5a > spa_load() at spa_load+0x31b > spa_tryimport() at spa_tryimport+0xa9 > zfs_ioc_pool_tryimport() at zfs_ioc_pool_tryimport+0x3f > zfsdev_ioctl() at zfsdev_ioctl+0x8d > devfs_ioctl_f() at devfs_ioctl_f+0x76 > kern_ioctl() at kern_ioctl+0xf6 > ioctl() at ioctl+0xfd > syscall() at syscall+0x19e > Xfast_syscall() at Xfast_syscall+0xe1 > --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x8010eb86c, rsp =3D =20= > 0x7fffffff8e28, rbp =3D 0x801323300 --- > > > db> show thread 100122 > > Thread 100122 at 0xffffff00035ca390: > proc (pid 3193): 0xffffff000383c460 > name: zpool > stack: 0xffffff805740a000-0xffffff805740dfff > flags: 0x44 pflags: 0x10000 > state: INHIBITED: {SLEEPING} > wmesg: zio->io_cv) wchan: 0xffffff00506e5858 > priority: 131 > container lock: sleepq chain (0xffffffff80c61e68) > db> > > db> show sleepchain 10012 > > db> show sleepchain 3193 > > thread 100122 (pid 3193, zpool) sleeping on 0xffffff00506e5858 "zio-=20= > >io_cv)" > db> show lock 0xffffff00506e5858 > > class: spin mutex > name: zio->io_cv) > flags: {SPIN} > state: {OWNED} > > > > > The box is currenlty in this state, so if you reply to me now I can =20= > continue to do debugging according to instructions. > Since I cannot get anywhere on my own with this, all and any help is =20= > appreciated since I really need this pool to get online.. > > dmesg is posted below (including some LOR?) > > Thanks! > Johan > > > > DMESG: > > Copyright (c) 1992-2009 The FreeBSD Project. > Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, =20= > 1994 > The Regents of the University of California. All rights =20 > reserved. > FreeBSD is a registered trademark of The FreeBSD Foundation. > FreeBSD 8.0-RELEASE-p2 #10: Tue Mar 9 12:44:15 CET 2010 > johan@back-1.stromnet.se:/usr/obj/usr/src/sys/BACK1 > WARNING: WITNESS option enabled, expect reduced performance. > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz (2666.68-MHz K8-=20= > class CPU) > Origin =3D "GenuineIntel" Id =3D 0x6fb Stepping =3D 11 > =20 > Features=20 > =3D=20 > 0xbfebfbff=20 > <=20 > FPU=20 > ,VME=20 > ,DE=20 > ,PSE=20 > ,TSC=20 > ,MSR=20 > ,PAE=20 > ,MCE=20 > ,CX8=20 > ,APIC=20 > ,SEP=20 > ,MTRR=20 > ,PGE=20 > ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> > =20 > Features2=20 > =3D0xe3fd > AMD Features=3D0x20100800 > AMD Features2=3D0x1 > TSC: P-state invariant > real memory =3D 2147483648 (2048 MB) > avail memory =3D 2040631296 (1946 MB) > ACPI APIC Table: > FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs > FreeBSD/SMP: 1 package(s) x 2 core(s) > cpu0 (BSP): APIC ID: 0 > cpu1 (AP): APIC ID: 1 > ioapic0: Changing APIC ID to 2 > ioapic0 irqs 0-23 on motherboard > kbd1 at kbdmux0 > cryptosoft0: on motherboard > acpi0: on motherboard > acpi0: [ITHREAD] > acpi0: Power Button (fixed) > acpi0: reservation of 0, a0000 (3) failed > acpi0: reservation of 100000, 7f4e0000 (3) failed > Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 > acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 > acpi_hpet0: iomem 0xfed00000-0xfed003ff =20= > on acpi0 > Timecounter "HPET" frequency 14318180 Hz quality 900 > acpi_button0: on acpi0 > pcib0: port 0xcf8-0xcff on acpi0 > pci0: on pcib0 > vgapci0: port 0xe000-0xe007 mem =20 > 0xe6300000-0xe637ffff,0xd0000000-0xdfffffff,0xe6000000-0xe60fffff =20 > irq 16 at device 2.0 on pci0 > agp0: on vgapci0 > agp0: detected 7164k stolen memory > agp0: aperture size is 256M > uhci0: port 0xe100-0xe11f irq =20 > 16 at device 26.0 on pci0 > uhci0: [ITHREAD] > uhci0: LegSup =3D 0x2f00 > usbus0: on uhci0 > uhci1: port 0xe200-0xe21f irq =20 > 21 at device 26.1 on pci0 > uhci1: [ITHREAD] > uhci1: LegSup =3D 0x2f00 > usbus1: on uhci1 > uhci2: port 0xe600-0xe61f irq =20 > 18 at device 26.2 on pci0 > uhci2: [ITHREAD] > uhci2: LegSup =3D 0x2f00 > usbus2: on uhci2 > ehci0: mem =20 > 0xe6384000-0xe63843ff irq 18 at device 26.7 on pci0 > ehci0: [ITHREAD] > usbus3: EHCI version 1.0 > usbus3: on ehci0 > hdac0: mem =20 > 0xe6380000-0xe6383fff irq 22 at device 27.0 on pci0 > hdac0: HDA Driver Revision: 20090624_0136 > hdac0: [ITHREAD] > pcib1: irq 16 at device 28.0 on pci0 > pci1: on pcib1 > pcib2: irq 18 at device 28.2 on pci0 > pci2: on pcib2 > em0: port =20 > 0xa000-0xa01f mem 0xe1020000-0xe103ffff,0xe1000000-0xe101ffff irq 18 =20= > at device 0.0 on pci2 > em0: Using MSI interrupt > em0: [FILTER] > em0: Ethernet address: 00:1b:21:05:00:b4 > pcib3: irq 19 at device 28.3 on pci0 > pci3: on pcib3 > atapci0: port =20 > 0xb000=20 > -0xb007,0xb100-0xb103,0xb200-0xb207,0xb300-0xb303,0xb400-0xb40f mem =20= > 0xe6100000-0xe6101fff irq 19 at device 0.0 on pci3 > atapci0: [ITHREAD] > atapci0: AHCI called from vendor specific driver > atapci0: AHCI v1.00 controller with 2 3Gbps ports, PM supported > ata2: on atapci0 > ata2: [ITHREAD] > ata3: on atapci0 > ata3: [ITHREAD] > ata4: on atapci0 > ata4: [ITHREAD] > pcib4: irq 16 at device 28.4 on pci0 > pci4: on pcib4 > re0: 8111CP/8111DP PCIe Gigabit Ethernet> port 0xc000-0xc0ff mem =20 > 0xe3000000-0xe3000fff irq 16 at device 0.0 on pci4 > re0: Using 1 MSI messages > re0: Chip rev. 0x38000000 > re0: MAC rev. 0x00000000 > miibus0: on re0 > rgephy0: PHY 1 on miibus0 > rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, =20= > 1000baseT-FDX, auto > re0: Ethernet address: 00:1a:4d:5a:97:87 > re0: [FILTER] > uhci3: port 0xe300-0xe31f irq =20 > 23 at device 29.0 on pci0 > uhci3: [ITHREAD] > uhci3: LegSup =3D 0x2f00 > usbus4: on uhci3 > uhci4: port 0xe400-0xe41f irq =20 > 19 at device 29.1 on pci0 > uhci4: [ITHREAD] > uhci4: LegSup =3D 0x2f00 > usbus5: on uhci4 > uhci5: port 0xe500-0xe51f irq =20 > 18 at device 29.2 on pci0 > uhci5: [ITHREAD] > uhci5: LegSup =3D 0x2f00 > usbus6: on uhci5 > ehci1: mem =20 > 0xe6385000-0xe63853ff irq 23 at device 29.7 on pci0 > ehci1: [ITHREAD] > usbus7: EHCI version 1.0 > usbus7: on ehci1 > pcib5: at device 30.0 on pci0 > pci5: on pcib5 > amr0: mem 0xe6200000-0xe620ffff irq 20 at =20 > device 0.0 on pci5 > amr0: Using 64-bit DMA > amr0: [ITHREAD] > amr0: delete logical drives supported by controller > amr0: Firmware 713S, BIOS G121, 64MB =20= > RAM > xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xd000-0xd07f mem =20 > 0xe5004000-0xe500407f irq 19 at device 1.0 on pci5 > miibus1: on xl0 > xlphy0: <3c905C 10/100 internal PHY> PHY 24 on miibus1 > xlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto > xl0: Ethernet address: 00:04:76:ef:c6:36 > xl0: [ITHREAD] > skc0: port 0xd100-0xd1ff mem =20 > 0xe5000000-0xe5003fff irq 18 at device 2.0 on pci5 > pci0:5:2:0: invalid VPD data, remain 0xfc > skc0: SysKonnect SK-NET Gigabit Ethernet Adapter SK-9843 SX rev. (0x0) > sk0: on skc0 > sk0: Ethernet address: 00:00:5a:98:43:68 > miibus2: on sk0 > xmphy0: PHY 0 on miibus2 > xmphy0: 1000baseSX, 1000baseSX-FDX, auto > skc0: [ITHREAD] > isab0: at device 31.0 on pci0 > isa0: on isab0 > atapci1: port =20 > 0xe700=20 > -0xe707,0xe800-0xe803,0xe900-0xe907,0xea00-0xea03,0xeb00-0xeb1f mem =20= > 0xe6386000-0xe63867ff irq 19 at device 31.2 on pci0 > atapci1: [ITHREAD] > atapci1: AHCI called from vendor specific driver > atapci1: AHCI v1.20 controller with 6 3Gbps ports, PM supported > ata5: on atapci1 > ata5: [ITHREAD] > ata6: on atapci1 > ata6: [ITHREAD] > ata7: on atapci1 > ata7: [ITHREAD] > ata8: on atapci1 > ata8: [ITHREAD] > ata9: on atapci1 > ata9: [ITHREAD] > ata10: on atapci1 > ata10: [ITHREAD] > pci0: at device 31.3 (no driver attached) > atrtc0: port 0x70-0x73 on acpi0 > fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 =20 > on acpi0 > fdc0: [FILTER] > uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on =20 > acpi0 > uart0: [FILTER] > ppc0: port 0x378-0x37f irq 7 on acpi0 > ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode > ppc0: [ITHREAD] > ppbus0: on ppc0 > plip0: on ppbus0 > plip0: [ITHREAD] > lpt0: on ppbus0 > lpt0: [ITHREAD] > lpt0: Interrupt-driven port > ppi0: on ppbus0 > atkbdc0: port 0x60,0x64 irq 1 on acpi0 > atkbd0: irq 1 on atkbdc0 > kbd0 at atkbd0 > atkbd0: [GIANT-LOCKED] > atkbd0: [ITHREAD] > cpu0: on acpi0 > est0: on cpu0 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 82a082a0600082a > device_attach: est0 attach returned 6 > p4tcc0: on cpu0 > cpu1: on acpi0 > est1: on cpu1 > est: CPU supports Enhanced Speedstep, but is not recognized. > est: cpu_vendor GenuineIntel, msr 82a082a0600082a > device_attach: est1 attach returned 6 > p4tcc1: on cpu1 > orm0: at iomem 0xcc000-0xcc7ff,0xcd000-0xcefff on =20= > isa0 > sc0: at flags 0x100 on isa0 > sc0: VGA <16 virtual consoles, flags=3D0x300> > vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on =20= > isa0 > ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM =20= > is present; > to enable, add "vfs.zfs.prefetch_disable=3D0" to /boot/=20 > loader.conf. > ZFS filesystem version 13 > ZFS storage pool version 13 > Timecounters tick every 1.000 msec > IPsec: Initialized Security Association Processing. > usbus0: 12Mbps Full Speed USB v1.0 > usbus1: 12Mbps Full Speed USB v1.0 > usbus2: 12Mbps Full Speed USB v1.0 > usbus3: 480Mbps High Speed USB v2.0 > usbus4: 12Mbps Full Speed USB v1.0 > usbus5: 12Mbps Full Speed USB v1.0 > usbus6: 12Mbps Full Speed USB v1.0 > usbus7: 480Mbps High Speed USB v2.0 > ad4: 476938MB at ata2-master SATA300 > ugen0.1: at usbus0 > uhub0: on =20 > usbus0 > ugen1.1: at usbus1 > uhub1: on =20 > usbus1 > ugen2.1: at usbus2 > uhub2: on =20 > usbus2 > ugen3.1: at usbus3 > uhub3: on =20 > usbus3 > ugen4.1: at usbus4 > uhub4: on =20 > usbus4 > ugen5.1: at usbus5 > uhub5: on =20 > usbus5 > ugen6.1: at usbus6 > uhub6: on =20 > usbus6 > ugen7.1: at usbus7 > uhub7: on =20 > usbus7 > ad6: 1907729MB at ata3-master SATA300 > uhub0: 2 ports with 2 removable, self powered > uhub1: 2 ports with 2 removable, self powered > uhub2: 2 ports with 2 removable, self powered > uhub4: 2 ports with 2 removable, self powered > uhub5: 2 ports with 2 removable, self powered > uhub6: 2 ports with 2 removable, self powered > ad10: 715404MB at ata5-master SATA300 > ad12: 305245MB at ata6-master SATA150 > ad14: 715404MB at ata7-master SATA300 > ad16: 1907729MB at ata8-master SATA300 > GEOM_MIRROR: Device mirror/gm1a launched (1/1). > GEOM_MIRROR: Device mirror/gm1b launched (1/1). > ad18: 1907729MB at ata9-master SATA300 > GEOM_MIRROR: Device mirror/swap launched (1/1). > ad20: 286187MB at ata10-master SATA150 > hdac0: HDA Codec #2: Realtek ALC885 > pcm0: at cad 2 nid 1 on hdac0 > pcm1: at cad 2 nid 1 on hdac0 > pcm2: at cad 2 nid 1 on hdac0 > pcm3: at cad 2 nid 1 on hdac0 > pcm4: at cad 2 nid 1 on hdac0 > pcm5: at cad 2 nid 1 on hdac0 > amr0: delete logical drives supported by controller > amrd0: on amr0 > amrd0: 476935MB (976762880 sectors) RAID 0 (optimal) > amrd1: on amr0 > amrd1: 476935MB (976762880 sectors) RAID 0 (optimal) > amrd2: on amr0 > amrd2: 1907724MB (3907018752 sectors) RAID 0 (optimal) > SMP: AP CPU #1 Launched! > WARNING: WITNESS option enabled, expect reduced performance. > Root mount waiting for: usbus7 usbus3 > uhub3: 6 ports with 6 removable, self powered > uhub7: 6 ports with 6 removable, self powered > Trying to mount root from zfs:zroot > ugen0.2: at usbus0 > ugen1.2: at usbus1 > uma_zalloc_arg: zone "256" with the following non-sleepable locks =20 > held: > exclusive rw ifnet_rw (ifnet_rw) r =3D 0 (0xffffffff80e01f60) locked =20= > @ /usr/src/sys/net/if.c:402 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > _witness_debugger() at _witness_debugger+0x2c > witness_warn() at witness_warn+0x2c2 > uma_zalloc_arg() at uma_zalloc_arg+0x29d > malloc() at malloc+0x5d > if_grow() at if_grow+0x2f > if_alloc() at if_alloc+0x2b3 > gif_clone_create() at gif_clone_create+0x53 > ifc_simple_create() at ifc_simple_create+0x89 > if_clone_createif() at if_clone_createif+0x64 > ifioctl() at ifioctl+0x6b5 > kern_ioctl() at kern_ioctl+0xf6 > ioctl() at ioctl+0xfd > syscall() at syscall+0x19e > Xfast_syscall() at Xfast_syscall+0xe1 > --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800b8286c, rsp =3D =20= > 0x7fffffffe4a8, rbp =3D 0x7fffffffef6e --- > lock order reversal: > 1st 0xffffffff80c093e0 pf task mtx (pf task mtx) @ /usr/src/sys/=20 > contrib/pf/net/pf_ioctl.c:1393 > 2nd 0xffffffff80e01f60 ifnet_rw (ifnet_rw) @ /usr/src/sys/net/if.c:=20 > 2034 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > _witness_debugger() at _witness_debugger+0x2c > witness_checkorder() at witness_checkorder+0x66f > _rw_rlock() at _rw_rlock+0x29 > ifunit() at ifunit+0x22 > pfioctl() at pfioctl+0x262a > devfs_ioctl_f() at devfs_ioctl_f+0x76 > kern_ioctl() at kern_ioctl+0xf6 > ioctl() at ioctl+0xfd > syscall() at syscall+0x19e > Xfast_syscall() at Xfast_syscall+0xe1 > --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x80099886c, rsp =3D =20= > 0x7fffffffdb68, rbp =3D 0x7fffffffdc20 --- > lock order reversal: > 1st 0xffffff00500c3098 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1054 > 2nd 0xffffff005010f448 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:=20= > 2083 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > _witness_debugger() at _witness_debugger+0x2c > witness_checkorder() at witness_checkorder+0x66f > __lockmgr_args() at __lockmgr_args+0x475 > vop_stdlock() at vop_stdlock+0x39 > VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 > _vn_lock() at _vn_lock+0x47 > vget() at vget+0x56 > devfs_allocv() at devfs_allocv+0x103 > devfs_root() at devfs_root+0x48 > vfs_donmount() at vfs_donmount+0xf43 > nmount() at nmount+0x63 > syscall() at syscall+0x19e > Xfast_syscall() at Xfast_syscall+0xe1 > --- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x8007b04dc, rsp =3D = =20 > 0x7fffffffdd28, rbp =3D 0x800a04048 --- > n > > r > tun0: link state changed to UP > lock order reversal: > 1st 0xffffff00500c3098 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1200 > 2nd 0xffffff005010f270 syncer (syncer) @ /usr/src/sys/kern/=20 > vfs_subr.c:2188 > KDB: stack backtrace: > db_trace_self_wrapper() at db_trace_self_wrapper+0x2a > _witness_debugger() at _witness_debugger+0x2c > witness_checkorder() at witness_checkorder+0x66f > __lockmgr_args() at __lockmgr_args+0x475 > vop_stdlock() at vop_stdlock+0x39 > VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 > _vn_lock() at _vn_lock+0x47 > vrele() at vrele+0xc3 > dounmount() at dounmount+0x269 > unmount() at unmount+0x27e > syscall() at syscall+0x19e > Xfast_syscall() at Xfast_syscall+0xe1 > --- syscall (22, FreeBSD ELF64, unmount), rip =3D 0x8006a09bc, rsp =3D = =20 > 0x7fffffffde18, rbp =3D 0 --- > KDB: enter: manual escape to debugger > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@FreeBSD.ORG Tue Mar 9 17:15:51 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6E0E5106566C for ; Tue, 9 Mar 2010 17:15:51 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id F05558FC17 for ; Tue, 9 Mar 2010 17:15:50 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1Np30s-00067e-Q5 for freebsd-fs@freebsd.org; Tue, 09 Mar 2010 18:14:34 +0100 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 09 Mar 2010 18:14:34 +0100 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Tue, 09 Mar 2010 18:14:34 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Tue, 09 Mar 2010 11:11:39 +0100 Lines: 59 Message-ID: References: <4B953C92.5080606@comcast.net> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.5) Gecko/20100118 Thunderbird/3.0 In-Reply-To: <4B953C92.5080606@comcast.net> Cc: freebsd-stable@freebsd.org Subject: Re: ZFS hot spares X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 17:15:51 -0000 On 03/08/10 19:06, Steve Polyack wrote: > ZFS in FreeBSD lacks at least one major feature from the Solaris > version: hot spares. There is a PR open at > http://www.freebsd.org/cgi/query-pr.cgi?pr=134491, but there hasn't been > any motion/thoughts posted on it since its creation almost one year ago. > > I'm aware that on Solaris, hot spare replacement is handled by a few > Solaris-specific daemons, zfs-retire and zfs-diagnose, which both plug > into the Solaris FMA (Fault Management Architecture). Have there been > any thoughts on porting these over or getting something similar running > within FreeBSD? With all of the recent SATA/SAS CAM hotplug work now > committed, it would be nice to have automatic replacement of hot spares > with a future hot-replacement of the failed drive. > > On the other side, I'd be interested in hearing if anyone has had > success in rolling their own scripted solution: i.e. something which > polls 'zpool status' looking for failed drives and performing hot-spare > replacements automatically. You don't have to exactly poll it. See /etc/devd.conf: # Sample ZFS problem reports handling. notify 10 { match "system" "ZFS"; match "type" "zpool"; action "logger -p kern.err 'ZFS: failed to load zpool $pool'"; }; notify 10 { match "system" "ZFS"; match "type" "vdev"; action "logger -p kern.err 'ZFS: vdev failure, zpool=$pool type=$type'"; }; notify 10 { match "system" "ZFS"; match "type" "data"; action "logger -p kern.warn 'ZFS: zpool I/O failure, zpool=$pool error=$zio_err'"; }; notify 10 { match "system" "ZFS"; match "type" "io"; action "logger -p kern.warn 'ZFS: vdev I/O failure, zpool=$pool path=$vdev_path offset=$zio_offset size=$zio_size error=$zio_err'"; }; notify 10 { match "system" "ZFS"; match "type" "checksum"; action "logger -p kern.warn 'ZFS: checksum mismatch, zpool=$pool path=$vdev_path offset=$zio_offset size=$zio_size'"; }; I don't really know if these notifications actually work since I don't have hot-plug test machines, but if they do, this looks like a decent starting point. From owner-freebsd-fs@FreeBSD.ORG Tue Mar 9 18:33:29 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 509F0106564A for ; Tue, 9 Mar 2010 18:33:29 +0000 (UTC) (envelope-from korvus@comcast.net) Received: from mx04.pub.collaborativefusion.com (mx04.pub.collaborativefusion.com [206.210.72.84]) by mx1.freebsd.org (Postfix) with ESMTP id 0CBFC8FC0C for ; Tue, 9 Mar 2010 18:33:28 +0000 (UTC) Received: from [192.168.2.164] ([206.210.89.202]) by mx04.pub.collaborativefusion.com (StrongMail Enterprise 4.1.1.4(4.1.1.4-47689)); Tue, 09 Mar 2010 13:50:54 -0500 X-VirtualServerGroup: Default X-MailingID: 00000::00000::00000::00000::::380 X-SMHeaderMap: mid="X-MailingID" X-Destination-ID: freebsd-fs@freebsd.org X-SMFBL: ZnJlZWJzZC1mc0BmcmVlYnNkLm9yZw== Message-ID: <4B969477.70706@comcast.net> Date: Tue, 09 Mar 2010 13:33:27 -0500 From: Steve Polyack User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.7) Gecko/20100211 Thunderbird/3.0.1 MIME-Version: 1.0 To: Ivan Voras References: <4B953C92.5080606@comcast.net> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: ZFS hot spares X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 18:33:29 -0000 On 03/09/10 05:11, Ivan Voras wrote: > On 03/08/10 19:06, Steve Polyack wrote: >> ZFS in FreeBSD lacks at least one major feature from the Solaris >> version: hot spares. There is a PR open at >> http://www.freebsd.org/cgi/query-pr.cgi?pr=134491, but there hasn't been >> any motion/thoughts posted on it since its creation almost one year ago. >> >> I'm aware that on Solaris, hot spare replacement is handled by a few >> Solaris-specific daemons, zfs-retire and zfs-diagnose, which both plug >> into the Solaris FMA (Fault Management Architecture). Have there been >> any thoughts on porting these over or getting something similar running >> within FreeBSD? With all of the recent SATA/SAS CAM hotplug work now >> committed, it would be nice to have automatic replacement of hot spares >> with a future hot-replacement of the failed drive. >> >> On the other side, I'd be interested in hearing if anyone has had >> success in rolling their own scripted solution: i.e. something which >> polls 'zpool status' looking for failed drives and performing hot-spare >> replacements automatically. > > You don't have to exactly poll it. See /etc/devd.conf: > > # Sample ZFS problem reports handling. > notify 10 { > match "system" "ZFS"; > match "type" "zpool"; > action "logger -p kern.err 'ZFS: failed to load zpool $pool'"; > }; > > notify 10 { > match "system" "ZFS"; > match "type" "vdev"; > action "logger -p kern.err 'ZFS: vdev failure, zpool=$pool > type=$type'"; > }; > > notify 10 { > match "system" "ZFS"; > match "type" "data"; > action "logger -p kern.warn 'ZFS: zpool I/O failure, > zpool=$pool error=$zio_err'"; > }; > > notify 10 { > match "system" "ZFS"; > match "type" "io"; > action "logger -p kern.warn 'ZFS: vdev I/O failure, > zpool=$pool path=$vdev_path offset=$zio_offset size=$zio_size > error=$zio_err'"; > }; > > notify 10 { > match "system" "ZFS"; > match "type" "checksum"; > action "logger -p kern.warn 'ZFS: checksum mismatch, > zpool=$pool path=$vdev_path offset=$zio_offset size=$zio_size'"; > }; > > I don't really know if these notifications actually work since I don't > have hot-plug test machines, but if they do, this looks like a decent > starting point. > Thanks for the suggestions. I received a similar one from someone else. If I get time to build a ZFS lab machine then I will certainly try these out and provide feedback on how they work. From owner-freebsd-fs@FreeBSD.ORG Tue Mar 9 19:07:50 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6D8ED106566B for ; Tue, 9 Mar 2010 19:07:50 +0000 (UTC) (envelope-from johan@stromnet.se) Received: from core.stromnet.se (core.stromnet.se [83.218.84.131]) by mx1.freebsd.org (Postfix) with ESMTP id B5D2E8FC14 for ; Tue, 9 Mar 2010 19:07:49 +0000 (UTC) Received: from localhost (core.stromnet.se [83.218.84.131]) by core.stromnet.se (Postfix) with ESMTP id 70A87F57419; Tue, 9 Mar 2010 20:07:47 +0100 (CET) X-Virus-Scanned: amavisd-new at stromnet.se X-Spam-Flag: NO X-Spam-Score: 0.723 X-Spam-Level: X-Spam-Status: No, score=0.723 tagged_above=-1000 required=6.2 tests=[AWL=0.747, BAYES_00=-2.599, RCVD_IN_PBL=0.905, RCVD_IN_SORBS_DUL=0.877, RDNS_DYNAMIC=0.1, SPF_FAIL=0.693] autolearn=no Received: from core.stromnet.se ([83.218.84.131]) by localhost (core.stromnet.se [83.218.84.131]) (amavisd-new, port 10024) with ESMTP id CiHGz-vvB3ei; Tue, 9 Mar 2010 20:07:44 +0100 (CET) Received: from johan-mb.vpn.stromnet.se (c-b4fe70d5.016-325-67626721.cust.bredbandsbolaget.se [213.112.254.180]) by core.stromnet.se (Postfix) with ESMTP id CFC2CF5740D; Tue, 9 Mar 2010 20:07:43 +0100 (CET) Message-Id: <4231C45D-499B-4FC9-90C3-BCC34DF20965@stromnet.se> From: =?ISO-8859-1?Q?Johan_Str=F6m?= To: =?ISO-8859-1?Q?Johan_Str=F6m?= In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Apple Message framework v936) Date: Tue, 9 Mar 2010 20:07:43 +0100 References: <8BA8475A-424E-489B-B643-8757F375320B@stromnet.se> X-Mailer: Apple Mail (2.936) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS: zpool import hang on "zio->io_cv)" (with DDB output). Help needed! X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 19:07:50 -0000 Got it working again, by ripping out my WD20EARS bench drives. Some =20 more background: The pool I was testing on was created like this (for 3 disks, ad6, =20 ad16, ad18): gpart create -s gpt adN gpart add -t freebsd-zfs -b 40 adN ...for all 3 drives.. zpool create bench raidz1 ad6 ad16 ad18 The reason for this was to test performance difference between running =20= straight on disk, and running on a 4k aligned partition (I was =20 benching and doing experiments on my new WD20EARS 4k sector size SATA =20= disks). Now, it was during bonnie++'ing these drives that the system paniced. =20= If that has anything at all to do with the panic, I do not know since =20= I was unable to obtain any dump/output. However, after booting again, it failed to import as described earlier. I now powered the machine down, and disconnected all the drives in the =20= above pool (ad6, ad16, ad18 and another amrd2 which was not in use at =20= the time). After power on, the import went fine! So, where was the problem? I don't really know. Could have been a =20 bunch of things I guess? Some thoughts: - ZFS failed to import from the GPT partitions for some reason - The ZFS labels was borked due to me having booted with invalid /=20 boot/zfs/zpool.cache (booting from old drive, new/correct loaders & =20 kernels though) - Borked ZFS labels due to panic. will try to plug the other disks =20 back in tomorrow and try to zdb out the labels. As a side note, from what I could get out of the testing, zfs on 4k =20 aligned partitions was not faster. However, for anyone using these =20 drives and UFS, aligning does make for a pretty nice improvement! Some =20= details can be found here: http://www.stromnet.se/~johan/bonnie-wd20ears-align-test.html (for =20 UFS, compare first two tables, unaligned vs aligned, on for example =20 Sequential output; block K/Sec and the latency). Disclaimer: this was not very scientific and I'm not sure if I'm =20 interpreting the results correctly, but it would seem that the average =20= values are 77-80MiB/s vs avg 55MiB/s, and latency sub 1000ms vs avg =20 2500ms. Exactly what that means I'm not sure but it IS better. :) =46rom what I've understood regarding zfs and 4k drivs, since ZFS =20 internally works with 512b blocks, it doesnt matter if I try to align =20= anyway, until the underlying system can report 4k sectors properly =20 (including getting these frikkin drives to do that, which they =20 currently don't). Anyhow, besides the big side note, if anyone cares to try to =20 reproduce.. Feel free :) I'll get back with zdb dumps on the other =20 drives later on. Thanks for ZFS and freebsd :) Johan On Mar 9, 2010, at 17:22 , Johan Str=F6m wrote: > Some followup, pjd tried to help me on IRC, without getting much =20 > further on the actual problem. > Some more information: > > Output from DDB PS: http://www.stromnet.se/~johan/back-1-ddb-ps.txt > Output from alltrace: = http://www.stromnet.se/~johan/back-1-ddb-alltrace.txt > > db> show alllocks > Process 2738 (sshd) thread 0xffffff0050418000 (100178) > Process 12 (intr) thread 0xffffff0002407ab0 (100018) > db> show lockedvnods > Locked vnodes > db > > > > Also tried accessing the disks (to make sure interrupts etc where =20 > alive) ,using dd if=3D/dev/adXX of=3D/dev/null count=3D1 on every disk = on =20 > the system, works fine. > Rebooted the system and tried with zfs debugging enabled too, and =20 > doing zpool import tank instead of just zpool import. no difference. =20= > Hanging there now. > > Johan > > > On Mar 9, 2010, at 15:26 , Johan Str=F6m wrote: > >> Hi List! >> >> I'm in the tedious process of upgrading my pool on my FreeBSD 8.0 =20 >> box (with a pool from 7.x). Yesterday I pulled a few disks from my =20= >> mirrored pool (one pool with multiple mirrors), in order to be able =20= >> to (free ports) plug in new ones and build my brand new pool. >> After removing disks, the pool got into state DEGRADED ofcourse, =20 >> since some of the disks where gone, but data was still there. I did =20= >> zpool detach on the removed disks, and the pool was ONLINE again, =20 >> all fine! >> Then I rebooted to make another disk available through my old LSI =20 >> MegaRaid card (didnt take time to figure out cryptic syntax of the =20= >> megarc CLI util). On boot, I was met by something similar to this: >> >> pool: tank >> state: UNAVAIL >> status: One or more devices could not be used because the label is =20= >> missing >> or invalid. There are insufficient replicas for the pool to =20= >> continue >> functioning. >> action: Destroy and re-create the pool from a backup source. >> see: http://www.sun.com/msg/ZFS-8000-5E >> scrub: none requested >> config: >> >> NAME STATE READ WRITE CKSUM >> tank UNAVAIL 0 0 0 insufficient replicas >> ad10s1d ONLINE 0 0 0 >> mirror DEGRADED 0 0 0 >> ad12 FAULTED 0 0 0 corrupted data >> ad16 UNAVAIL 0 0 0 corrupted data >> mirror DEGRADED 0 0 0 >> ad20 FAULTED 0 0 0 corrupted data >> ad18 UNAVAIL 0 0 0 corrupted data >> mirror UNAVAIL 0 0 0 insufficient replicas >> ad6 UNAVAIL 0 0 0 corrupted data >> ad4 FAULTED 0 0 0 corrupted data >> mirror ONLINE 0 0 0 >> amrd0 ONLINE 0 0 0 >> amrd1 ONLINE 0 0 0 >> >> The reason the disks was still there was due to me haveing an old =20 >> zfs.cache file in the boot (it boots from anothre drive, but thats =20= >> another story..) >> >> In this case, a export/import did the trick, after reimporting, the =20= >> pool was back online. All fine. I created a new pool and did some =20 >> bonnie++ testing on it, and suddenly the box paniced or something =20 >> (I didn't have dumpon enabled :/ and didn't see the screen until it =20= >> rebooted).. >> >> Now my problems come. The box came up again, with the above output. =20= >> I tried zpool export again, fine. Zpool import however, hang. =20 >> Waited an hour, nothing. After rebuilding the kernel with DDB/=20 >> witness and did the import again, i've managed to get this output =20 >> (im not really sure what is usable here): >> >> back-1 # zpool import >> load: 0.29 cmd: zpool 3193 [zio->io_cv)] 2.54r 0.00u 0.01s 0% 2236k >> >> >> >> In DDB: >> > tr 3193 >> >> Tracing pid 3193 tid 100122 td 0xffffff00035ca390 >> sched_switch() at sched_switch+0xde >> mi_switch() at mi_switch+0x170 >> sleepq_wait() at sleepq_wait+0x44 >> _cv_wait() at _cv_wait+0x13c >> zio_wait() at zio_wait+0x61 >> arc_read_nolock() at arc_read_nolock+0x345 >> dmu_objset_open_impl() at dmu_objset_open_impl+0xd0 >> dsl_pool_open() at dsl_pool_open+0x5a >> spa_load() at spa_load+0x31b >> spa_tryimport() at spa_tryimport+0xa9 >> zfs_ioc_pool_tryimport() at zfs_ioc_pool_tryimport+0x3f >> zfsdev_ioctl() at zfsdev_ioctl+0x8d >> devfs_ioctl_f() at devfs_ioctl_f+0x76 >> kern_ioctl() at kern_ioctl+0xf6 >> ioctl() at ioctl+0xfd >> syscall() at syscall+0x19e >> Xfast_syscall() at Xfast_syscall+0xe1 >> --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x8010eb86c, rsp =3D =20= >> 0x7fffffff8e28, rbp =3D 0x801323300 --- >> >> >> db> show thread 100122 >> >> Thread 100122 at 0xffffff00035ca390: >> proc (pid 3193): 0xffffff000383c460 >> name: zpool >> stack: 0xffffff805740a000-0xffffff805740dfff >> flags: 0x44 pflags: 0x10000 >> state: INHIBITED: {SLEEPING} >> wmesg: zio->io_cv) wchan: 0xffffff00506e5858 >> priority: 131 >> container lock: sleepq chain (0xffffffff80c61e68) >> db> >> >> db> show sleepchain 10012 >> >> db> show sleepchain 3193 >> >> thread 100122 (pid 3193, zpool) sleeping on 0xffffff00506e5858 "zio-=20= >> >io_cv)" >> db> show lock 0xffffff00506e5858 >> >> class: spin mutex >> name: zio->io_cv) >> flags: {SPIN} >> state: {OWNED} >> >> >> >> >> The box is currenlty in this state, so if you reply to me now I can =20= >> continue to do debugging according to instructions. >> Since I cannot get anywhere on my own with this, all and any help =20 >> is appreciated since I really need this pool to get online.. >> >> dmesg is posted below (including some LOR?) >> >> Thanks! >> Johan >> >> >> >> DMESG: >> >> Copyright (c) 1992-2009 The FreeBSD Project. >> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, =20= >> 1994 >> The Regents of the University of California. All rights =20 >> reserved. >> FreeBSD is a registered trademark of The FreeBSD Foundation. >> FreeBSD 8.0-RELEASE-p2 #10: Tue Mar 9 12:44:15 CET 2010 >> johan@back-1.stromnet.se:/usr/obj/usr/src/sys/BACK1 >> WARNING: WITNESS option enabled, expect reduced performance. >> Timecounter "i8254" frequency 1193182 Hz quality 0 >> CPU: Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz (2666.68-MHz =20 >> K8-class CPU) >> Origin =3D "GenuineIntel" Id =3D 0x6fb Stepping =3D 11 >> Features=20 >> =3D=20 >> 0xbfebfbff=20 >> <=20 >> FPU=20 >> ,VME=20 >> ,DE=20 >> ,PSE=20 >> ,TSC=20 >> ,MSR=20 >> ,PAE=20 >> ,MCE=20 >> ,CX8=20 >> ,APIC=20 >> ,SEP=20 >> ,MTRR=20 >> ,PGE=20 >> ,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> >> Features2=20 >> =3D0xe3fd >> AMD Features=3D0x20100800 >> AMD Features2=3D0x1 >> TSC: P-state invariant >> real memory =3D 2147483648 (2048 MB) >> avail memory =3D 2040631296 (1946 MB) >> ACPI APIC Table: >> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >> FreeBSD/SMP: 1 package(s) x 2 core(s) >> cpu0 (BSP): APIC ID: 0 >> cpu1 (AP): APIC ID: 1 >> ioapic0: Changing APIC ID to 2 >> ioapic0 irqs 0-23 on motherboard >> kbd1 at kbdmux0 >> cryptosoft0: on motherboard >> acpi0: on motherboard >> acpi0: [ITHREAD] >> acpi0: Power Button (fixed) >> acpi0: reservation of 0, a0000 (3) failed >> acpi0: reservation of 100000, 7f4e0000 (3) failed >> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 >> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >> acpi_hpet0: iomem =20 >> 0xfed00000-0xfed003ff on acpi0 >> Timecounter "HPET" frequency 14318180 Hz quality 900 >> acpi_button0: on acpi0 >> pcib0: port 0xcf8-0xcff on acpi0 >> pci0: on pcib0 >> vgapci0: port 0xe000-0xe007 mem =20 >> 0xe6300000-0xe637ffff,0xd0000000-0xdfffffff,0xe6000000-0xe60fffff =20 >> irq 16 at device 2.0 on pci0 >> agp0: on vgapci0 >> agp0: detected 7164k stolen memory >> agp0: aperture size is 256M >> uhci0: port 0xe100-0xe11f irq =20= >> 16 at device 26.0 on pci0 >> uhci0: [ITHREAD] >> uhci0: LegSup =3D 0x2f00 >> usbus0: on uhci0 >> uhci1: port 0xe200-0xe21f irq =20= >> 21 at device 26.1 on pci0 >> uhci1: [ITHREAD] >> uhci1: LegSup =3D 0x2f00 >> usbus1: on uhci1 >> uhci2: port 0xe600-0xe61f irq =20= >> 18 at device 26.2 on pci0 >> uhci2: [ITHREAD] >> uhci2: LegSup =3D 0x2f00 >> usbus2: on uhci2 >> ehci0: mem =20 >> 0xe6384000-0xe63843ff irq 18 at device 26.7 on pci0 >> ehci0: [ITHREAD] >> usbus3: EHCI version 1.0 >> usbus3: on ehci0 >> hdac0: mem =20 >> 0xe6380000-0xe6383fff irq 22 at device 27.0 on pci0 >> hdac0: HDA Driver Revision: 20090624_0136 >> hdac0: [ITHREAD] >> pcib1: irq 16 at device 28.0 on pci0 >> pci1: on pcib1 >> pcib2: irq 18 at device 28.2 on pci0 >> pci2: on pcib2 >> em0: port =20 >> 0xa000-0xa01f mem 0xe1020000-0xe103ffff,0xe1000000-0xe101ffff irq =20 >> 18 at device 0.0 on pci2 >> em0: Using MSI interrupt >> em0: [FILTER] >> em0: Ethernet address: 00:1b:21:05:00:b4 >> pcib3: irq 19 at device 28.3 on pci0 >> pci3: on pcib3 >> atapci0: port =20 >> 0xb000=20 >> -0xb007,0xb100-0xb103,0xb200-0xb207,0xb300-0xb303,0xb400-0xb40f mem =20= >> 0xe6100000-0xe6101fff irq 19 at device 0.0 on pci3 >> atapci0: [ITHREAD] >> atapci0: AHCI called from vendor specific driver >> atapci0: AHCI v1.00 controller with 2 3Gbps ports, PM supported >> ata2: on atapci0 >> ata2: [ITHREAD] >> ata3: on atapci0 >> ata3: [ITHREAD] >> ata4: on atapci0 >> ata4: [ITHREAD] >> pcib4: irq 16 at device 28.4 on pci0 >> pci4: on pcib4 >> re0: > 8111CP/8111DP PCIe Gigabit Ethernet> port 0xc000-0xc0ff mem =20 >> 0xe3000000-0xe3000fff irq 16 at device 0.0 on pci4 >> re0: Using 1 MSI messages >> re0: Chip rev. 0x38000000 >> re0: MAC rev. 0x00000000 >> miibus0: on re0 >> rgephy0: PHY 1 on miibus0 >> rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, =20 >> 1000baseT, 1000baseT-FDX, auto >> re0: Ethernet address: 00:1a:4d:5a:97:87 >> re0: [FILTER] >> uhci3: port 0xe300-0xe31f irq =20= >> 23 at device 29.0 on pci0 >> uhci3: [ITHREAD] >> uhci3: LegSup =3D 0x2f00 >> usbus4: on uhci3 >> uhci4: port 0xe400-0xe41f irq =20= >> 19 at device 29.1 on pci0 >> uhci4: [ITHREAD] >> uhci4: LegSup =3D 0x2f00 >> usbus5: on uhci4 >> uhci5: port 0xe500-0xe51f irq =20= >> 18 at device 29.2 on pci0 >> uhci5: [ITHREAD] >> uhci5: LegSup =3D 0x2f00 >> usbus6: on uhci5 >> ehci1: mem =20 >> 0xe6385000-0xe63853ff irq 23 at device 29.7 on pci0 >> ehci1: [ITHREAD] >> usbus7: EHCI version 1.0 >> usbus7: on ehci1 >> pcib5: at device 30.0 on pci0 >> pci5: on pcib5 >> amr0: mem 0xe6200000-0xe620ffff irq 20 at =20= >> device 0.0 on pci5 >> amr0: Using 64-bit DMA >> amr0: [ITHREAD] >> amr0: delete logical drives supported by controller >> amr0: Firmware 713S, BIOS G121, =20 >> 64MB RAM >> xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xd000-0xd07f mem =20 >> 0xe5004000-0xe500407f irq 19 at device 1.0 on pci5 >> miibus1: on xl0 >> xlphy0: <3c905C 10/100 internal PHY> PHY 24 on miibus1 >> xlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto >> xl0: Ethernet address: 00:04:76:ef:c6:36 >> xl0: [ITHREAD] >> skc0: port 0xd100-0xd1ff mem =20 >> 0xe5000000-0xe5003fff irq 18 at device 2.0 on pci5 >> pci0:5:2:0: invalid VPD data, remain 0xfc >> skc0: SysKonnect SK-NET Gigabit Ethernet Adapter SK-9843 SX rev. =20 >> (0x0) >> sk0: on skc0 >> sk0: Ethernet address: 00:00:5a:98:43:68 >> miibus2: on sk0 >> xmphy0: PHY 0 on miibus2 >> xmphy0: 1000baseSX, 1000baseSX-FDX, auto >> skc0: [ITHREAD] >> isab0: at device 31.0 on pci0 >> isa0: on isab0 >> atapci1: port =20 >> 0xe700=20 >> -0xe707,0xe800-0xe803,0xe900-0xe907,0xea00-0xea03,0xeb00-0xeb1f mem =20= >> 0xe6386000-0xe63867ff irq 19 at device 31.2 on pci0 >> atapci1: [ITHREAD] >> atapci1: AHCI called from vendor specific driver >> atapci1: AHCI v1.20 controller with 6 3Gbps ports, PM supported >> ata5: on atapci1 >> ata5: [ITHREAD] >> ata6: on atapci1 >> ata6: [ITHREAD] >> ata7: on atapci1 >> ata7: [ITHREAD] >> ata8: on atapci1 >> ata8: [ITHREAD] >> ata9: on atapci1 >> ata9: [ITHREAD] >> ata10: on atapci1 >> ata10: [ITHREAD] >> pci0: at device 31.3 (no driver attached) >> atrtc0: port 0x70-0x73 on acpi0 >> fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 =20= >> on acpi0 >> fdc0: [FILTER] >> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on =20 >> acpi0 >> uart0: [FILTER] >> ppc0: port 0x378-0x37f irq 7 on acpi0 >> ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode >> ppc0: [ITHREAD] >> ppbus0: on ppc0 >> plip0: on ppbus0 >> plip0: [ITHREAD] >> lpt0: on ppbus0 >> lpt0: [ITHREAD] >> lpt0: Interrupt-driven port >> ppi0: on ppbus0 >> atkbdc0: port 0x60,0x64 irq 1 on acpi0 >> atkbd0: irq 1 on atkbdc0 >> kbd0 at atkbd0 >> atkbd0: [GIANT-LOCKED] >> atkbd0: [ITHREAD] >> cpu0: on acpi0 >> est0: on cpu0 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 82a082a0600082a >> device_attach: est0 attach returned 6 >> p4tcc0: on cpu0 >> cpu1: on acpi0 >> est1: on cpu1 >> est: CPU supports Enhanced Speedstep, but is not recognized. >> est: cpu_vendor GenuineIntel, msr 82a082a0600082a >> device_attach: est1 attach returned 6 >> p4tcc1: on cpu1 >> orm0: at iomem 0xcc000-0xcc7ff,0xcd000-0xcefff on =20= >> isa0 >> sc0: at flags 0x100 on isa0 >> sc0: VGA <16 virtual consoles, flags=3D0x300> >> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff =20 >> on isa0 >> ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM =20= >> is present; >> to enable, add "vfs.zfs.prefetch_disable=3D0" to /boot/=20 >> loader.conf. >> ZFS filesystem version 13 >> ZFS storage pool version 13 >> Timecounters tick every 1.000 msec >> IPsec: Initialized Security Association Processing. >> usbus0: 12Mbps Full Speed USB v1.0 >> usbus1: 12Mbps Full Speed USB v1.0 >> usbus2: 12Mbps Full Speed USB v1.0 >> usbus3: 480Mbps High Speed USB v2.0 >> usbus4: 12Mbps Full Speed USB v1.0 >> usbus5: 12Mbps Full Speed USB v1.0 >> usbus6: 12Mbps Full Speed USB v1.0 >> usbus7: 480Mbps High Speed USB v2.0 >> ad4: 476938MB at ata2-master SATA300 >> ugen0.1: at usbus0 >> uhub0: on =20 >> usbus0 >> ugen1.1: at usbus1 >> uhub1: on =20 >> usbus1 >> ugen2.1: at usbus2 >> uhub2: on =20 >> usbus2 >> ugen3.1: at usbus3 >> uhub3: on =20 >> usbus3 >> ugen4.1: at usbus4 >> uhub4: on =20 >> usbus4 >> ugen5.1: at usbus5 >> uhub5: on =20 >> usbus5 >> ugen6.1: at usbus6 >> uhub6: on =20 >> usbus6 >> ugen7.1: at usbus7 >> uhub7: on =20 >> usbus7 >> ad6: 1907729MB at ata3-master SATA300 >> uhub0: 2 ports with 2 removable, self powered >> uhub1: 2 ports with 2 removable, self powered >> uhub2: 2 ports with 2 removable, self powered >> uhub4: 2 ports with 2 removable, self powered >> uhub5: 2 ports with 2 removable, self powered >> uhub6: 2 ports with 2 removable, self powered >> ad10: 715404MB at ata5-master =20 >> SATA300 >> ad12: 305245MB at ata6-master SATA150 >> ad14: 715404MB at ata7-master =20 >> SATA300 >> ad16: 1907729MB at ata8-master SATA300 >> GEOM_MIRROR: Device mirror/gm1a launched (1/1). >> GEOM_MIRROR: Device mirror/gm1b launched (1/1). >> ad18: 1907729MB at ata9-master SATA300 >> GEOM_MIRROR: Device mirror/swap launched (1/1). >> ad20: 286187MB at ata10-master SATA150 >> hdac0: HDA Codec #2: Realtek ALC885 >> pcm0: at cad 2 nid 1 on hdac0 >> pcm1: at cad 2 nid 1 on hdac0 >> pcm2: at cad 2 nid 1 on hdac0 >> pcm3: at cad 2 nid 1 on hdac0 >> pcm4: at cad 2 nid 1 on hdac0 >> pcm5: at cad 2 nid 1 on hdac0 >> amr0: delete logical drives supported by controller >> amrd0: on amr0 >> amrd0: 476935MB (976762880 sectors) RAID 0 (optimal) >> amrd1: on amr0 >> amrd1: 476935MB (976762880 sectors) RAID 0 (optimal) >> amrd2: on amr0 >> amrd2: 1907724MB (3907018752 sectors) RAID 0 (optimal) >> SMP: AP CPU #1 Launched! >> WARNING: WITNESS option enabled, expect reduced performance. >> Root mount waiting for: usbus7 usbus3 >> uhub3: 6 ports with 6 removable, self powered >> uhub7: 6 ports with 6 removable, self powered >> Trying to mount root from zfs:zroot >> ugen0.2: at usbus0 >> ugen1.2: at usbus1 >> uma_zalloc_arg: zone "256" with the following non-sleepable locks =20 >> held: >> exclusive rw ifnet_rw (ifnet_rw) r =3D 0 (0xffffffff80e01f60) locked =20= >> @ /usr/src/sys/net/if.c:402 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> _witness_debugger() at _witness_debugger+0x2c >> witness_warn() at witness_warn+0x2c2 >> uma_zalloc_arg() at uma_zalloc_arg+0x29d >> malloc() at malloc+0x5d >> if_grow() at if_grow+0x2f >> if_alloc() at if_alloc+0x2b3 >> gif_clone_create() at gif_clone_create+0x53 >> ifc_simple_create() at ifc_simple_create+0x89 >> if_clone_createif() at if_clone_createif+0x64 >> ifioctl() at ifioctl+0x6b5 >> kern_ioctl() at kern_ioctl+0xf6 >> ioctl() at ioctl+0xfd >> syscall() at syscall+0x19e >> Xfast_syscall() at Xfast_syscall+0xe1 >> --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800b8286c, rsp =3D =20= >> 0x7fffffffe4a8, rbp =3D 0x7fffffffef6e --- >> lock order reversal: >> 1st 0xffffffff80c093e0 pf task mtx (pf task mtx) @ /usr/src/sys/=20 >> contrib/pf/net/pf_ioctl.c:1393 >> 2nd 0xffffffff80e01f60 ifnet_rw (ifnet_rw) @ /usr/src/sys/net/if.c:=20= >> 2034 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> _witness_debugger() at _witness_debugger+0x2c >> witness_checkorder() at witness_checkorder+0x66f >> _rw_rlock() at _rw_rlock+0x29 >> ifunit() at ifunit+0x22 >> pfioctl() at pfioctl+0x262a >> devfs_ioctl_f() at devfs_ioctl_f+0x76 >> kern_ioctl() at kern_ioctl+0xf6 >> ioctl() at ioctl+0xfd >> syscall() at syscall+0x19e >> Xfast_syscall() at Xfast_syscall+0xe1 >> --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x80099886c, rsp =3D =20= >> 0x7fffffffdb68, rbp =3D 0x7fffffffdc20 --- >> lock order reversal: >> 1st 0xffffff00500c3098 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1054 >> 2nd 0xffffff005010f448 devfs (devfs) @ /usr/src/sys/kern/vfs_subr.c:=20= >> 2083 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> _witness_debugger() at _witness_debugger+0x2c >> witness_checkorder() at witness_checkorder+0x66f >> __lockmgr_args() at __lockmgr_args+0x475 >> vop_stdlock() at vop_stdlock+0x39 >> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 >> _vn_lock() at _vn_lock+0x47 >> vget() at vget+0x56 >> devfs_allocv() at devfs_allocv+0x103 >> devfs_root() at devfs_root+0x48 >> vfs_donmount() at vfs_donmount+0xf43 >> nmount() at nmount+0x63 >> syscall() at syscall+0x19e >> Xfast_syscall() at Xfast_syscall+0xe1 >> --- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x8007b04dc, rsp =3D = =20 >> 0x7fffffffdd28, rbp =3D 0x800a04048 --- >> n >> >> r >> tun0: link state changed to UP >> lock order reversal: >> 1st 0xffffff00500c3098 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:1200 >> 2nd 0xffffff005010f270 syncer (syncer) @ /usr/src/sys/kern/=20 >> vfs_subr.c:2188 >> KDB: stack backtrace: >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >> _witness_debugger() at _witness_debugger+0x2c >> witness_checkorder() at witness_checkorder+0x66f >> __lockmgr_args() at __lockmgr_args+0x475 >> vop_stdlock() at vop_stdlock+0x39 >> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 >> _vn_lock() at _vn_lock+0x47 >> vrele() at vrele+0xc3 >> dounmount() at dounmount+0x269 >> unmount() at unmount+0x27e >> syscall() at syscall+0x19e >> Xfast_syscall() at Xfast_syscall+0xe1 >> --- syscall (22, FreeBSD ELF64, unmount), rip =3D 0x8006a09bc, rsp =3D = =20 >> 0x7fffffffde18, rbp =3D 0 --- >> KDB: enter: manual escape to debugger >> >> _______________________________________________ >> freebsd-fs@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >> To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > From owner-freebsd-fs@FreeBSD.ORG Tue Mar 9 23:28:00 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBC401065672 for ; Tue, 9 Mar 2010 23:28:00 +0000 (UTC) (envelope-from toasty@dragondata.com) Received: from mail-gw0-f54.google.com (mail-gw0-f54.google.com [74.125.83.54]) by mx1.freebsd.org (Postfix) with ESMTP id A57678FC16 for ; Tue, 9 Mar 2010 23:28:00 +0000 (UTC) Received: by gwaa20 with SMTP id a20so4229316gwa.13 for ; Tue, 09 Mar 2010 15:27:59 -0800 (PST) Received: by 10.150.128.6 with SMTP id a6mr101441ybd.281.1268175823565; Tue, 09 Mar 2010 15:03:43 -0800 (PST) Received: from vpn177.ord02.your.org (vpn177.ord02.your.org [204.9.55.177]) by mx.google.com with ESMTPS id 21sm5951597iwn.11.2010.03.09.15.03.42 (version=TLSv1/SSLv3 cipher=RC4-MD5); Tue, 09 Mar 2010 15:03:43 -0800 (PST) From: Kevin Day Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Date: Tue, 9 Mar 2010 17:03:41 -0600 Message-Id: <7418ECC2-55C1-4A28-82EA-0972AFE745EF@dragondata.com> To: freebsd-fs@freebsd.org Mime-Version: 1.0 (Apple Message framework v1077) X-Mailer: Apple Mail (2.1077) Subject: iscsi over HAST backed storage partial success X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Mar 2010 23:28:01 -0000 I'm running istgt (iscsi target) using HAST backed storage. For the most = part, it seems to work really well. I have ucarp running to change the = IP that istgt is bound to, and modified the ucarp scripts to start/stop = istgt depending on which side is the master. If I shut down the primary, = the secondary takes over and all seems well. However, if I reboot the secondary, the primary starts freezing up for = long periods: Mar 9 22:46:27 cs04 hastd: [iscsi1] (primary) Unable to r: Socket is = not connected. Mar 9 22:46:27 cs04 hastd: [iscsi1] (primary) Unable to co: Connection = refused. Mar 9 22:46:42 cs04 last message repeated 3 times Mar 9 22:46:53 cs04 istgt[14298]: ABORT_TASK Mar 9 22:47:35 cs04 last message repeated 3 times Mar 9 22:48:02 cs04 hastd: [iscsi1] (primary) Unable to co: Operation = timed out. Mar 9 22:48:02 cs04 istgt[14298]: CmdSN(45748), OP=3D0x2a, = ElapsedTime=3D74 cleared=20 Mar 9 22:48:02 cs04 istgt[14298]: istgt_iscsi.c: = 640:istgt_iscsi_write_pdu: ***ERROR*** iscsi_write() failed (errno=3D32) Mar 9 22:48:02 cs04 istgt[14298]: = istgt_iscsi.c:3327:istgt_iscsi_op_task: ***ERROR*** iscsi_write_pdu() = failed Mar 9 22:48:02 cs04 istgt[14298]: = istgt_iscsi.c:3867:istgt_iscsi_execute: ***ERROR*** iscsi_op_task() = failed =20 Mar 9 22:48:02 cs04 istgt[14298]: istgt_iscsi.c:4337:worker: = ***ERROR*** iscsi_execute() failed Mar 9 22:48:02 cs04 istgt[14298]: CmdSN(490802), OP=3D0x2a, = ElapsedTime=3D73 cleared Mar 9 22:48:02 cs04 istgt[14298]: CmdSN(28387), OP=3D0x2a, = ElapsedTime=3D73 cleared=20 Mar 9 22:48:14 cs04 istgt[14298]: ABORT_TASK Mar 9 22:48:52 cs04 last message repeated 2 times Mar 9 22:49:22 cs04 hastd: [iscsi1] (primary) Unable to co: Operation = timed out. As soon as the secondary comes back online, everything starts behaving = again and all is well. Is this expected behavior at this point, or should hastd not block like = this? -- Kevin From owner-freebsd-fs@FreeBSD.ORG Wed Mar 10 12:32:05 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A2413106564A; Wed, 10 Mar 2010 12:32:05 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop1.sarenet.es (proxypop1.sarenet.es [194.30.0.99]) by mx1.freebsd.org (Postfix) with ESMTP id AF2828FC21; Wed, 10 Mar 2010 12:32:04 +0000 (UTC) Received: from [172.16.1.204] (unknown [192.148.167.2]) by proxypop1.sarenet.es (Postfix) with ESMTP id 2D7B66078; Wed, 10 Mar 2010 13:32:03 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20100310110202.GA1715@garage.freebsd.pl> Date: Wed, 10 Mar 2010 13:32:02 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> To: Pawel Jakub Dawidek X-Mailer: Apple Mail (2.1077) Cc: freebsd-fs@freebsd.org, FreeBSD Stable Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 12:32:05 -0000 =09 On Mar 10, 2010, at 12:02 PM, Pawel Jakub Dawidek wrote: > On Wed, Mar 10, 2010 at 10:24:49AM +0100, Borja Marcos wrote: >> Tested. Same deadlock remains. >=20 > Ok, to track this down I need the following: >=20 > Uncomment 'CFLAGS+=3D-DDEBUG=3D1' line in sys/modules/zfs/Makefile. >=20 > Add the following lines to your kernel config: >=20 > options WITNESS > options WITNESS_SKIPSPIN > options INVARIANTS > options INVARIANT_SUPPORT > options DEBUG_VFS_LOCKS > options DEBUG_LOCKS > options KDB > options DDB >=20 > Recompile your kernel and modules and reboot. >=20 > Once the deadlock occur, enter DDB and send me the output of: >=20 > ps > show alllocks > show lockedvnods > show allchains > alltrace Trying. I started my typical test: Machine 1 doing a make buildworld on a dataset with src and obj on it. Machine 1 replicating incremental snapshots of the dataset to machine 2. Machine 2 running some "tar cf - . | ( cd /pool/anotherdataset && tar xf = - )" from the dataset being replicated, ie, doing read operations on the = target dataset. This time, with all those debug options, there was no deadlock, but = almost an instant trap entering DDB. Unfortunately, I tried to capture = the output of "alltrace", etc using the capture option. But couldn't = come back to the system to read it. Any ideas? I'm using VMWare Fusion to run FreeBSD for these tests and = seems I'm out of luck, I don't see any console output mechanism. When rebooting I was greeted by some LORs lock order reversal: 1st 0xffffff000286c2e8 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:549 2nd 0xffffff000286b0d8 dn->dn_mtx (dn->dn_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:1173 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_block_freed() at dnode_block_freed+0x8e dbuf_read() at dbuf_read+0x155 dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x12a dmu_read() at dmu_read+0x80 load_nvlist() at load_nvlist+0x85 spa_load() at spa_load+0x49a spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffffd808, rbp =3D 0x801224140 --- lock order reversal: 1st 0xffffff0002864e70 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode_sync.c:381 2nd 0xffffff00026e5140 osi->os_lock (osi->os_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:323 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_destroy() at dnode_destroy+0xa6 dnode_buf_pageout() at dnode_buf_pageout+0xb2 dbuf_evict_user() at dbuf_evict_user+0x55 dbuf_clear() at dbuf_clear+0x5e dnode_evict_dbufs() at dnode_evict_dbufs+0x98 dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0x11c dmu_objset_evict() at dmu_objset_evict+0xbf dsl_pool_close() at dsl_pool_close+0x52 spa_unload() at spa_unload+0xb2 spa_load() at spa_load+0x4da spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffffd808, rbp =3D 0x801224140 --- lock order reversal: 1st 0xffffff000286c058 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1116 2nd 0xffffff0002591a38 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1120 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_dirty() at dbuf_dirty+0x892 dnode_setdirty() at dnode_setdirty+0x1a9 dbuf_dirty() at dbuf_dirty+0xa53 bplist_vacate() at bplist_vacate+0x4d spa_sync() at spa_sync+0x297 txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002905638 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 2nd 0xffffff000250c2f0 spa->spa_sync_bplist.bpl_lock = (spa->spa_sync_bplist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:235 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 bplist_enqueue_deferred() at bplist_enqueue_deferred+0x47 zio_free() at zio_free+0x105 arc_free() at arc_free+0x11c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x483 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x3eb dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync() at dmu_objset_sync+0x134 dsl_pool_sync() at dsl_pool_sync+0x200 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff000293bcc8 zfs (zfs) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/gfs.c:437 2nd 0xffffff0002916310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23c zfs_root() at zfs_root+0x50 zfsctl_create() at zfsctl_create+0x82 zfs_mount() at zfs_mount+0x7ef vfs_donmount() at vfs_donmount+0xcde nmount() at nmount+0x63 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x800f48f1c, rsp =3D = 0x7fffffffced8, rbp =3D 0x7fffffffcef8 --- lock order reversal: 1st 0xffffff00025e4078 zp->z_name_lock (zp->z_name_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_dir.c:212 2nd 0xffffff0002916330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23c zfs_dirent_lock() at zfs_dirent_lock+0x4a0 zfs_dirlook() at zfs_dirlook+0x90 zfs_lookup() at zfs_lookup+0x256 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800fd8acc, rsp =3D = 0x7fffffffcf38, rbp =3D 0x7fffffffd3d0 --- lock order reversal: 1st 0xffffff0002916210 zfsvfs->z_teardown_inactive_lock = (zfsvfs->z_teardown_inactive_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_vnops.c:3724 2nd 0xffffff0002916330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:1027 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zinactive() at zfs_zinactive+0x95 zfs_inactive() at zfs_inactive+0x7e zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5 vinactive() at vinactive+0x90 vputx() at vputx+0x2fc kern_statat_vnhook() at kern_statat_vnhook+0xfa kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800fd8acc, rsp =3D = 0x7fffffffcf38, rbp =3D 0x7fffffffd3d0 --- lock order reversal: 1st 0xffffff0002861958 buf->b_lock (buf->b_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/arc.c:2509 2nd 0xffffff0002956430 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:421 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_read_done() at dbuf_read_done+0x3b arc_read_done() at arc_read_done+0x1d2 zio_done() at zio_done+0x308 zio_execute() at zio_execute+0xb1 arc_read_nolock() at arc_read_nolock+0x3d0 arc_read() at arc_read+0xaf dbuf_read() at dbuf_read+0x62b dmu_buf_hold() at dmu_buf_hold+0xcc zap_lockdir() at zap_lockdir+0x6e zap_cursor_retrieve() at zap_cursor_retrieve+0x1bc zfs_unlinked_drain() at zfs_unlinked_drain+0xd8 zfsvfs_setup() at zfsvfs_setup+0xfa zfs_mount() at zfs_mount+0x7df vfs_donmount() at vfs_donmount+0xcde nmount() at nmount+0x63 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x800f48f1c, rsp =3D = 0x7fffffffced8, rbp =3D 0x7fffffffcef8 --- Expensive timeout(9) function: 0xffffffff80329580(0xffffff8000284000) = 0.012686807 s ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based = forwarding disabled, default to deny, logging disabled lock order reversal: 1st 0xffffff800a2f5d08 bufwait (bufwait) @ = /pool/newsrc/src/sys/kern/vfs_bio.c:2559 2nd 0xffffff0002c69800 dirhash (dirhash) @ = /pool/newsrc/src/sys/ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 ufsdirhash_acquire() at ufsdirhash_acquire+0x44 ufsdirhash_add() at ufsdirhash_add+0x19 ufs_direnter() at ufs_direnter+0x88b ufs_mkdir() at ufs_mkdir+0x623 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xb9 kern_mkdirat() at kern_mkdirat+0x264 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (136, FreeBSD ELF64, mkdir), rip =3D 0x80072bb0c, rsp =3D = 0x7fffffffec88, rbp =3D 0x7fffffffef66 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 2nd 0xffffff000286ab88 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:130 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dnode_verify() at dnode_verify+0x70 dnode_hold_impl() at dnode_hold_impl+0x73 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 2nd 0xffffff0002f6b0d8 dn->dn_mtx (dn->dn_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:606 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_hold_impl() at dnode_hold_impl+0x184 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 2nd 0xffffff00028842e8 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1724 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_rele() at dbuf_rele+0x2d dnode_hold_impl() at dnode_hold_impl+0x20f dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 2nd 0xffffff00027d5d40 osi->os_lock (osi->os_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:687 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_setdirty() at dnode_setdirty+0xbc dbuf_dirty() at dbuf_dirty+0x516 bplist_enqueue() at bplist_enqueue+0xbd dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002c84938 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 2nd 0xffffff0002f6b7b0 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:543 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dbuf_read() at dbuf_read+0x2ad dbuf_will_dirty() at dbuf_will_dirty+0x53 dsl_dataset_block_kill() at dsl_dataset_block_kill+0xe9 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x159 dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync() at dmu_objset_sync+0x134 dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff000291a210 zfsvfs->z_teardown_inactive_lock = (zfsvfs->z_teardown_inactive_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_vfsops.c:917 2nd 0xffffff00027450f8 ds->ds_rwlock (ds->ds_rwlock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dsl_dataset.c:2864 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dsl_dataset_clone_swap() at dsl_dataset_clone_swap+0x5a dmu_recv_end() at dmu_recv_end+0x94 zfs_ioc_recv() at zfs_ioc_recv+0x29d zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffff8e98, rbp =3D 0x7fffffff9bd0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:94 2nd 0xffffff0002f6b330 dn->dn_dbufs_mtx (dn->dn_dbufs_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1518 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_destroy() at dbuf_destroy+0x58 bplist_close() at bplist_close+0x37 dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0x506 dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 dsl_pool_sync() at dsl_pool_sync+0x122 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002745038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:94 2nd 0xffffffff81152650 h->hash_mutexes[i] (h->hash_mutexes[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:191 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_destroy() at dbuf_destroy+0x111 bplist_close() at bplist_close+0x37 dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0x506 dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 dsl_pool_sync() at dsl_pool_sync+0x122 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff000291a250 zfsvfs->z_znodes_lock (zfsvfs->z_znodes_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_vfsops.c:1317 2nd 0xffffff000291a310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:966 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_rezget() at zfs_rezget+0x4a zfs_resume_fs() at zfs_resume_fs+0x158 zfs_ioc_recv() at zfs_ioc_recv+0x2b4 zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffff8e98, rbp =3D 0x7fffffff9bd0 --- #=20 Borja. From owner-freebsd-fs@FreeBSD.ORG Wed Mar 10 12:56:48 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 455521065672; Wed, 10 Mar 2010 12:56:48 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 824E08FC15; Wed, 10 Mar 2010 12:56:46 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 2CD0545C99; Wed, 10 Mar 2010 13:56:45 +0100 (CET) Received: from localhost (pdawidek.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id E663745683; Wed, 10 Mar 2010 13:56:39 +0100 (CET) Date: Wed, 10 Mar 2010 13:56:39 +0100 From: Pawel Jakub Dawidek To: Borja Marcos Message-ID: <20100310125639.GB1715@garage.freebsd.pl> References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="qlTNgmc+xy1dBmNv" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org, FreeBSD Stable Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 12:56:48 -0000 --qlTNgmc+xy1dBmNv Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Mar 10, 2010 at 01:32:02PM +0100, Borja Marcos wrote: > Trying. I started my typical test: >=20 > Machine 1 doing a make buildworld on a dataset with src and obj on it. >=20 > Machine 1 replicating incremental snapshots of the dataset to machine 2. >=20 > Machine 2 running some "tar cf - . | ( cd /pool/anotherdataset && tar xf = - )" from the dataset being replicated, ie, doing read operations on the ta= rget dataset. >=20 > This time, with all those debug options, there was no deadlock, but almos= t an instant trap entering DDB. Unfortunately, I tried to capture the outpu= t of "alltrace", etc using the capture option. But couldn't come back to th= e system to read it. >=20 > Any ideas? I'm using VMWare Fusion to run FreeBSD for these tests and see= ms I'm out of luck, I don't see any console output mechanism. You should be able to use text dumps to capture output of those commands on panic. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --qlTNgmc+xy1dBmNv Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkuXlwcACgkQForvXbEpPzRzNACfTNHrGE+pPTvYnZlXaRaPpxGc 2u8An3VA/URKFHEEeca5ItcVTdP5G+Uy =z7Mk -----END PGP SIGNATURE----- --qlTNgmc+xy1dBmNv-- From owner-freebsd-fs@FreeBSD.ORG Wed Mar 10 14:37:50 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBFA0106564A; Wed, 10 Mar 2010 14:37:50 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop1.sarenet.es (proxypop1.sarenet.es [194.30.0.99]) by mx1.freebsd.org (Postfix) with ESMTP id D9EDC8FC13; Wed, 10 Mar 2010 14:37:48 +0000 (UTC) Received: from [172.16.1.204] (unknown [192.148.167.2]) by proxypop1.sarenet.es (Postfix) with ESMTP id 9C7375FC5; Wed, 10 Mar 2010 15:37:47 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1077) From: Borja Marcos In-Reply-To: <20100310110202.GA1715@garage.freebsd.pl> Date: Wed, 10 Mar 2010 15:37:46 +0100 Message-Id: <548451D6-E62C-4B7F-A846-E9CF08A180C8@sarenet.es> References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> To: Pawel Jakub Dawidek X-Mailer: Apple Mail (2.1077) X-Mailman-Approved-At: Wed, 10 Mar 2010 15:03:13 +0000 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org, FreeBSD Stable Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 14:37:51 -0000 =09 On Mar 10, 2010, at 12:02 PM, Pawel Jakub Dawidek wrote: > Recompile your kernel and modules and reboot. >=20 > Once the deadlock occur, enter DDB and send me the output of: >=20 > ps > show alllocks > show lockedvnods > show allchains > alltrace (copying to freebsd-fs, as this is filesystem-related) There we go. Thanks for the textdump hint :) Instead of a deadlock I had a crash. Maybe some of the additional = debugging options triggered it. It is a 8-stable cvsupped on 2nd March, = and with this patch applied: http://people.freebsd.org/~pjd/patches/zfs_3way_deadlock.patch This is ddb.txt: #=20 #=20 # cat ddb.txt=20 db> set $terminal=3D0 Unknown variable db> set $lines=3D0 db> ps pid ppid pgrp uid state wmesg wchan cmd 1425 1423 1423 0 S+ zfsvfs-> 0xffffff000256d310 bsdtar 1424 1423 1423 0 S+ pipdwt 0xffffff00025d6000 bsdtar 1423 1384 1423 0 S+ wait 0xffffff0005106000 sh 1422 1420 1420 0 R+ bsdtar 1421 1420 1420 0 S+ pipdwt 0xffffff00025d65b0 bsdtar 1420 1384 1420 0 S+ wait 0xffffff0005107000 sh 1419 1417 1417 0 S+ zfs 0xffffff00057ac578 bsdtar 1418 1417 1417 0 S+ pipdwt 0xffffff00025d4000 bsdtar 1417 1384 1417 0 S+ wait 0xffffff0002e79000 sh 1416 1414 1414 0 S+ zio->io_ 0xffffff00026c5b28 bsdtar 1415 1414 1414 0 S+ pipdwt 0xffffff00025d6b60 bsdtar 1414 1384 1414 0 S+ wait 0xffffff0002e788c0 sh 1413 1409 1413 0 S+ select 0xffffff00029d56c0 top 1409 1408 1409 0 S+ pause 0xffffff0002d83500 csh 1408 1405 1408 1001 S+ wait 0xffffff0002e7a000 su 1405 1404 1405 1001 Ss+ pause 0xffffff0002e78500 csh 1404 1401 1401 1001 S select 0xffffff0002cce1c0 sshd 1401 1280 1401 0 Ss sbwait 0xffffff0002f63694 sshd 1384 1383 1384 0 S+ ttyin 0xffffff00024b54a8 csh 1383 1377 1383 1001 S+ wait 0xffffff0002e77460 su 1377 1376 1377 1001 Ss+ pause 0xffffff0002e79500 csh 1376 1373 1373 1001 S select 0xffffff0002916440 sshd 1373 1280 1373 0 Ss sbwait 0xffffff0002cff3ec sshd 1372 1 1372 0 Ss+ ttyin 0xffffff00024daca8 getty 1371 1 1371 0 Ss+ ttyin 0xffffff00024dc0a8 getty 1370 1 1370 0 Ss+ ttyin 0xffffff00024cdca8 getty 1369 1 1369 0 Ss+ ttyin 0xffffff00024c94a8 getty 1368 1 1368 0 Ss+ ttyin 0xffffff00024dd4a8 getty 1367 1 1367 0 Ss+ ttyin 0xffffff00024c28a8 getty 1366 1 1366 0 Ss+ ttyin 0xffffff00024db4a8 getty 1365 1 1365 0 Ss+ ttyin 0xffffff00024dc8a8 getty 1301 1 1301 0 Ss nanslp 0xffffffff80c4d168 cron 1294 1 1294 25 Ss pause 0xffffff0002623960 sendmail 1288 1 1288 0 Ss select 0xffffff0002917cc0 sendmail 1280 1 1280 0 Ss select 0xffffff00029d5340 sshd 1005 1 1005 0 Rs syslogd 827 1 827 0 Ss select 0xffffff0002ccf2c0 devd 20 0 0 0 SL flowclea 0xffffffff80e0e950 = [flowcleaner] 19 0 0 0 SL sdflush 0xffffffff80e1d8b8 = [softdepflush] 18 0 0 0 RL [syncer] 17 0 0 0 SL vlruwt 0xffffff000256a8c0 [vnlru] 16 0 0 0 SL psleep 0xffffffff80e0e1c8 [bufdaemon] 15 0 0 0 SL pgzero 0xffffffff80e1f34c [pagezero] 14 0 0 0 SL psleep 0xffffffff80e1e6e8 [vmdaemon] 9 0 0 0 RL = [pagedaemon] 8 0 0 0 SL ccb_scan 0xffffffff80c16060 [xpt_thrd] 7 0 0 0 RL (threaded) zfskern 100085 Run CPU 0 = [txg_thread_enter] 100084 RunQ = [txg_thread_enter] 100082 D vgeom:io 0xffffff00026acd90 [vdev da2] 100036 D l2arc_fe 0xffffffff81150620 = [l2arc_feed_thread] 100035 D arc_recl 0xffffffff81147fe0 = [arc_reclaim_thread] 6 0 0 0 SL - 0xffffff000244e648 [fdc0] 5 0 0 0 SL idle 0xffffff800023c000 = [mpt_recovery0] 13 0 0 0 SL - 0xffffffff80c4ce44 [yarrow] 4 0 0 0 RL [g_down] 3 0 0 0 RL [g_up] 2 0 0 0 RL [g_event] 12 0 0 0 WL (threaded) intr 100033 I [swi0: uart = uart] 100032 I [irq7: = ppc0] 100031 I [irq12: = psm0] 100030 I [irq1: = atkbd0] 100024 I [irq17: em2 = mpt0] 100023 I [irq15: = ata1] 100022 I [irq14: = ata0] 100021 I [irq9: = acpi0] 100019 I [swi5: +] 100018 I [swi2: = cambio] 100013 I [swi6: task = queue] 100012 I [swi6: = Giant taskq] 100006 I [swi1: = netisr 0] 100005 I [swi4: = clock] 100004 I [swi3: vm] 11 0 0 0 RL [idle: = cpu0] 1 0 1 0 SLs wait 0xffffff00021e18c0 [init] 10 0 0 0 SL audit_wo 0xffffffff80e1cc10 [audit] 0 0 0 0 SLs (threaded) kernel 100088 D - 0xffffff0002914880 [zil_clean] 100087 D - 0xffffff0002916780 [zil_clean] 100086 D - 0xffffff0002913900 [zil_clean] 100083 D - 0xffffff0002628480 = [zfs_vn_rele_taskq] 100081 D - 0xffffff00026acc00 [spa_zio] 100080 D - 0xffffff00026acb80 [spa_zio] 100079 D - 0xffffff00026acb00 [spa_zio] 100078 D - 0xffffff00026aca80 [spa_zio] 100077 D - 0xffffff00026aca00 [spa_zio] 100076 D - 0xffffff00026ac980 [spa_zio] 100075 D - 0xffffff00026ac900 [spa_zio] 100074 D - 0xffffff00026ac880 [spa_zio_7] 100073 D - 0xffffff00026ac880 [spa_zio_6] 100072 D - 0xffffff00026ac880 [spa_zio_5] 100071 D - 0xffffff00026ac880 [spa_zio_4] 100070 D - 0xffffff00026ac880 [spa_zio_3] 100069 D - 0xffffff00026ac880 [spa_zio_2] 100068 D - 0xffffff00026ac880 [spa_zio_1] 100067 D - 0xffffff00026ac880 [spa_zio_0] 100066 D - 0xffffff00026ac800 [spa_zio_7] 100065 D - 0xffffff00026ac800 [spa_zio_6] 100064 D - 0xffffff00026ac800 [spa_zio_5] 100063 D - 0xffffff00026ac800 [spa_zio_4] 100062 D - 0xffffff00026ac800 [spa_zio_3] 100061 D - 0xffffff00026ac800 [spa_zio_2] 100060 D - 0xffffff00026ac800 [spa_zio_1] 100059 D - 0xffffff00026ac800 [spa_zio_0] 100058 D - 0xffffff00026ac780 [spa_zio] 100057 D - 0xffffff00026ac700 [spa_zio] 100056 D - 0xffffff00026ac680 [spa_zio] 100029 D - 0xffffff0002441980 [em3 taskq] 100028 D - 0xffffff0002440d80 [em2 taskq] 100027 D - 0xffffff0002436200 [em1 taskq] 100026 D - 0xffffff000242fb80 [em0 taskq] 100020 D - 0xffffff0002358600 [thread = taskq] 100017 D - 0xffffff0002316b80 = [acpi_task_2] 100016 D - 0xffffff0002316b80 = [acpi_task_1] 100015 D - 0xffffff0002316b80 = [acpi_task_0] 100014 D - 0xffffff0002316c00 [kqueue = taskq] 100010 D - 0xffffff00021e0180 [firmware = taskq] 100000 D sched 0xffffffff80c49740 [swapper] db> show alllocks Process 1425 (bsdtar) thread 0xffffff000528b3a0 (100115) Process 1422 (bsdtar) thread 0xffffff000528c000 (100112) Process 1419 (bsdtar) thread 0xffffff0002d81000 (100109) Process 1416 (bsdtar) thread 0xffffff000262a3a0 (100091) Process 1401 (sshd) thread 0xffffff0002510740 (100055) Process 1373 (sshd) thread 0xffffff000262a740 (100090) Process 7 (zfskern) thread 0xffffff000262bae0 (100085) db> show lockedvnods Locked vnodes db> show allchains db> alltrace Tracing command bsdtar pid 1425 tid 100115 td 0xffffff000528b3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_xlock_hard() at _sx_xlock_hard+0x267 _sx_xlock() at _sx_xlock+0xc1 zfs_zinactive() at zfs_zinactive+0x95 zfs_inactive() at zfs_inactive+0x7e zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5 vinactive() at vinactive+0x90 vputx() at vputx+0x2fc kern_statat_vnhook() at kern_statat_vnhook+0xfa kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800e35acc, rsp =3D = 0x7fffffffe768, rbp =3D 0x801025380 --- Tracing command bsdtar pid 1424 tid 100114 td 0xffffff000528b740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_write() at pipe_write+0x13fc dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e44d5c, rsp =3D = 0x7fffffffe0d8, rbp =3D 0x801004140 --- Tracing command sh pid 1423 tid 100113 td 0xffffff000528bae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x80093554c, rsp =3D = 0x7fffffffe8b8, rbp =3D 0x58f --- Tracing command bsdtar pid 1422 tid 100112 td 0xffffff000528c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a txg_wait_open() at txg_wait_open+0x95 zfs_freebsd_setattr() at zfs_freebsd_setattr+0x862 VOP_SETATTR_APV() at VOP_SETATTR_APV+0xaf setutimes() at setutimes+0x214 kern_lutimes() at kern_lutimes+0xf2 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (276, FreeBSD ELF64, lutimes), rip =3D 0x800da66dc, rsp =3D = 0x7fffffffe7e8, rbp =3D 0x801025380 --- Tracing command bsdtar pid 1421 tid 100111 td 0xffffff000528c3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_write() at pipe_write+0x13fc dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e44d5c, rsp =3D = 0x7fffffffe0d8, rbp =3D 0x801004140 --- Tracing command sh pid 1420 tid 100110 td 0xffffff000262dae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x80093554c, rsp =3D = 0x7fffffffe8b8, rbp =3D 0x58c --- Tracing command bsdtar pid 1419 tid 100109 td 0xffffff0002d81000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x7a3 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b cache_lookup() at cache_lookup+0x4e0 vfs_cache_lookup() at vfs_cache_lookup+0xc0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800e35acc, rsp =3D = 0x7fffffffe768, rbp =3D 0x801025380 --- Tracing command bsdtar pid 1418 tid 100108 td 0xffffff0002d813a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_write() at pipe_write+0x13fc dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e44d5c, rsp =3D = 0x7fffffffe0d8, rbp =3D 0x801004140 --- Tracing command sh pid 1417 tid 100101 td 0xffffff0002cfb000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x80093554c, rsp =3D = 0x7fffffffe8b8, rbp =3D 0x589 --- Tracing command bsdtar pid 1416 tid 100091 td 0xffffff000262a3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a zio_wait() at zio_wait+0x7d dbuf_read() at dbuf_read+0x2bb dnode_hold_impl() at dnode_hold_impl+0x114 dmu_bonus_hold() at dmu_bonus_hold+0x31 zfs_zget() at zfs_zget+0x259 zfs_dirent_lock() at zfs_dirent_lock+0x4a0 zfs_dirlook() at zfs_dirlook+0x90 zfs_lookup() at zfs_lookup+0x256 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x4a9 kern_mkdirat() at kern_mkdirat+0xc3 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (136, FreeBSD ELF64, mkdir), rip =3D 0x800e35b0c, rsp =3D = 0x7fffffffe748, rbp =3D 0x1ed --- Tracing command bsdtar pid 1415 tid 100105 td 0xffffff0002cfa000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_write() at pipe_write+0x13fc dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e44d5c, rsp =3D = 0x7fffffffe0d8, rbp =3D 0x801004140 --- Tracing command sh pid 1414 tid 100102 td 0xffffff0002cfaae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x80093554c, rsp =3D = 0x7fffffffe8b8, rbp =3D 0x586 --- Tracing command top pid 1413 tid 100054 td 0xffffff0002510ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x800ab7cfc, rsp =3D = 0x7fffffffea08, rbp =3D 0x11 --- Tracing command csh pid 1409 tid 100093 td 0xffffff0002645ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x80094d19c, rsp =3D = 0x7fffffffe638, rbp =3D 0x800c5a400 --- Tracing command su pid 1408 tid 100098 td 0xffffff0002cfbae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009e854c, rsp =3D = 0x7fffffffe598, rbp =3D 0x581 --- Tracing command csh pid 1405 tid 100103 td 0xffffff0002cfa740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x80094d19c, rsp =3D = 0x7fffffffe558, rbp =3D 0x800c0f900 --- Tracing command sshd pid 1404 tid 100104 td 0xffffff0002cfa3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d7cfc, rsp =3D = 0x7fffffffdcb8, rbp =3D 0x7fffffffdd40 --- Tracing command sshd pid 1401 tid 100055 td 0xffffff0002510740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x8013d7d7c, rsp =3D = 0x7fffffffdcd8, rbp =3D 0 --- Tracing command csh pid 1384 tid 100107 td 0xffffff0002d81740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x8009ebd7c, rsp =3D = 0x7fffffffe738, rbp =3D 0x1 --- Tracing command su pid 1383 tid 100106 td 0xffffff0002d81ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009e854c, rsp =3D = 0x7fffffffe598, rbp =3D 0x568 --- Tracing command csh pid 1377 tid 100100 td 0xffffff0002cfb3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x80094d19c, rsp =3D = 0x7fffffffe558, rbp =3D 0x800c0f600 --- Tracing command sshd pid 1376 tid 100099 td 0xffffff0002cfb740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d7cfc, rsp =3D = 0x7fffffffdcb8, rbp =3D 0x7fffffffdd40 --- Tracing command sshd pid 1373 tid 100090 td 0xffffff000262a740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x8013d7d7c, rsp =3D = 0x7fffffffdcd8, rbp =3D 0 --- Tracing command getty pid 1372 tid 100046 td 0xffffff00025b1ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1371 tid 100095 td 0xffffff00026453a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1370 tid 100096 td 0xffffff0002da63a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1369 tid 100097 td 0xffffff0002da6000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1368 tid 100092 td 0xffffff000262a000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1367 tid 100094 td 0xffffff0002645740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1366 tid 100089 td 0xffffff000262aae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1365 tid 100052 td 0xffffff00025b03a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command cron pid 1301 tid 100053 td 0xffffff00025b0000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip =3D 0x80093efac, rsp =3D = 0x7fffffffeb28, rbp =3D 0x3b --- Tracing command sendmail pid 1294 tid 100048 td 0xffffff00025b13a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800d3a19c, rsp =3D = 0x7fffffffcd68, rbp =3D 0x1 --- Tracing command sendmail pid 1288 tid 100051 td 0xffffff00025b0740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x800dd8cfc, rsp =3D = 0x7fffffffc1e8, rbp =3D 0x7fffffffc280 --- Tracing command sshd pid 1280 tid 100047 td 0xffffff00025b1740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d7cfc, rsp =3D = 0x7fffffffddd8, rbp =3D 0x2 --- Tracing command syslogd pid 1005 tid 100050 td 0xffffff00025b0ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x800851cfc, rsp =3D = 0x7fffffffe118, rbp =3D 0x800a310c8 --- Tracing command devd pid 827 tid 100049 td 0xffffff00025b1000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x440ecc, rsp =3D = 0x7fffffffe898, rbp =3D 0x7fffffffe8b0 --- Tracing command flowcleaner pid 20 tid 100045 td 0xffffff0002485ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c flowtable_cleaner() at flowtable_cleaner+0xde fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800018cd30, rbp =3D 0 --- Tracing command softdepflush pid 19 tid 100044 td 0xffffff000250d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 softdep_flush() at softdep_flush+0x259 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000187d30, rbp =3D 0 --- Tracing command syncer pid 18 tid 100043 td 0xffffff000250d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c sched_sync() at sched_sync+0x4db fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000182d30, rbp =3D 0 --- Tracing command vnlru pid 17 tid 100042 td 0xffffff000250d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vnlru_proc() at vnlru_proc+0x5cf fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800017dd30, rbp =3D 0 --- Tracing command bufdaemon pid 16 tid 100041 td 0xffffff000250dae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 buf_daemon() at buf_daemon+0x14a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000178d30, rbp =3D 0 --- Tracing command pagezero pid 15 tid 100040 td 0xffffff000250e000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000173d30, rbp =3D 0 --- Tracing command vmdaemon pid 14 tid 100039 td 0xffffff000250e3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800016ed30, rbp =3D 0 --- Tracing command pagedaemon pid 9 tid 100038 td 0xffffff000250e740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pageout() at vm_pageout+0x855 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000169d30, rbp =3D 0 --- Tracing command xpt_thrd pid 8 tid 100037 td 0xffffff000250eae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 xpt_scanner_thread() at xpt_scanner_thread+0xd5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000164d30, rbp =3D 0 --- Tracing command zfskern pid 7 tid 100085 td 0xffffff000262bae0 kdb_enter() at kdb_enter+0x3d panic() at panic+0x17b dblfault_handler() at dblfault_handler+0x96 Xdblfault() at Xdblfault+0xac --- trap 0x17, rip =3D 0xffffffff805d322d, rsp =3D 0xffffffff80e5ecd0, = rbp =3D 0xffffff8012415090 --- witness_checkorder() at witness_checkorder+0x2d _mtx_lock_spin_flags() at _mtx_lock_spin_flags+0x94 callout_tick() at callout_tick+0x41 hardclock_cpu() at hardclock_cpu+0xc8 hardclock() at hardclock+0x23 lapic_handle_timer() at lapic_handle_timer+0xd2 Xtimerint() at Xtimerint+0x8c --- interrupt, rip =3D 0xffffffff80866093, rsp =3D 0xffffff8012415270, = rbp =3D 0xffffff8012415290 --- spinlock_exit() at spinlock_exit+0x33 putcons() at putcons+0x40 putchar() at putchar+0x6a kvprintf() at kvprintf+0x81 vprintf() at vprintf+0x85 printf() at printf+0x67 db_putc() at db_putc+0x81 kvprintf() at kvprintf+0x81 db_printf() at db_printf+0x86 db_backtrace() at db_backtrace+0x162 db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dnode_verify() at dnode_verify+0x70 dnode_hold_impl() at dnode_hold_impl+0x73 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x3eb dbuf_sync_list() at dbuf_sync_list+0x17f dbuf_sync_list() at dbuf_sync_list+0x17f dbuf_sync_list() at dbuf_sync_list+0x17f dbuf_sync_list() at dbuf_sync_list+0x17f dbuf_sync_list() at dbuf_sync_list+0x17f dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync() at dmu_objset_sync+0x134 dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- Tracing command zfskern pid 7 tid 100084 td 0xffffff000262d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a txg_thread_wait() at txg_thread_wait+0x5b txg_quiesce_thread() at txg_quiesce_thread+0xb4 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012413d30, rbp =3D 0 --- Tracing command zfskern pid 7 tid 100082 td 0xffffff0002640000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vdev_geom_worker() at vdev_geom_worker+0xf3 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123ffd30, rbp =3D 0 --- Tracing command zfskern pid 7 tid 100036 td 0xffffff0002510000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c l2arc_feed_thread() at l2arc_feed_thread+0x162 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800015fd30, rbp =3D 0 --- Tracing command zfskern pid 7 tid 100035 td 0xffffff0002356740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c arc_reclaim_thread() at arc_reclaim_thread+0x2b7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800015ad30, rbp =3D 0 --- Tracing command fdc0 pid 6 tid 100034 td 0xffffff0002356ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 fdc_thread() at fdc_thread+0x774 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000155d30, rbp =3D 0 --- Tracing command mpt_recovery0 pid 5 tid 100025 td 0xffffff0002353ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 mpt_recovery_thread() at mpt_recovery_thread+0x251 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800011ed30, rbp =3D 0 --- Tracing command yarrow pid 13 tid 100011 td 0xffffff00021e53a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 random_kthread() at random_kthread+0x1ad fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800004cd30, rbp =3D 0 --- Tracing command g_down pid 4 tid 100009 td 0xffffff00021e5ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 g_io_schedule_down() at g_io_schedule_down+0x236 g_down_procbody() at g_down_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000042d30, rbp =3D 0 --- Tracing command g_up pid 3 tid 100008 td 0xffffff00021f3000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 g_io_schedule_up() at g_io_schedule_up+0x133 g_up_procbody() at g_up_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800003dd30, rbp =3D 0 --- Tracing command g_event pid 2 tid 100007 td 0xffffff00021f33a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 g_event_procbody() at g_event_procbody+0xa1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000038d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100033 td 0xffffff0002483000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100032 td 0xffffff00024833a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100031 td 0xffffff0002483740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100030 td 0xffffff0002483ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000137d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100024 td 0xffffff0002354000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80000d7d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100023 td 0xffffff00023543a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80000d2d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100022 td 0xffffff0002354740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100021 td 0xffffff0002354ae0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100019 td 0xffffff00023563a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000078d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100018 td 0xffffff00021f3ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000073d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100013 td 0xffffff0002353000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800005ad30, rbp =3D 0 --- Tracing command intr pid 12 tid 100012 td 0xffffff00023533a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100006 td 0xffffff00021f3740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000033d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100005 td 0xffffff00021e4000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800002ed30, rbp =3D 0 --- Tracing command intr pid 12 tid 100004 td 0xffffff00021e43a0 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100003 td 0xffffff00021e4740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d critical_exit() at critical_exit+0xa3 intr_event_handle() at intr_event_handle+0xd7 intr_execute_handlers() at intr_execute_handlers+0x5f lapic_handle_intr() at lapic_handle_intr+0x37 Xapic_isr1() at Xapic_isr1+0xa4 --- interrupt, rip =3D 0xffffffff808545f6, rsp =3D 0xffffff8000024b50, = rbp =3D 0xffffff8000024b60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000024d30, rbp =3D 0 --- Tracing command init pid 1 tid 100002 td 0xffffff00021e4ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x40c86c, rsp =3D = 0x7fffffffe808, rbp =3D 0x401d40 --- Tracing command audit pid 10 tid 100001 td 0xffffff00021e5000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a audit_worker() at audit_worker+0x77 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800001ad30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100088 td 0xffffff000262b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012426d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100087 td 0xffffff000262b3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012421d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100086 td 0xffffff000262b740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff801241cd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100083 td 0xffffff000263eae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012404d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100081 td 0xffffff00026403a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123fad30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100080 td 0xffffff000263d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123f5d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100079 td 0xffffff000263dae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123f0d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100078 td 0xffffff000263e000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleep#=20 And this is msgbuf.txt: # cat msgbuf.txt=20 Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights = reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-STABLE #3: Wed Mar 10 14:41:54 CET 2010 = root@pruebazfs1.sarenet.es:/pool/newsrc/obj/pool/newsrc/src/sys/DEBUG = amd64 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz (2116.63-MHz = K8-class CPU) Origin =3D "GenuineIntel" Id =3D 0x10676 Stepping =3D 6 = Features=3D0xfebfbff Features2=3D0x80082201> AMD Features=3D0x20100800 AMD Features2=3D0x1 TSC: P-state invariant real memory =3D 536870912 (512 MB) avail memory =3D 495988736 (473 MB) ACPI APIC Table: MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port = 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 7.3 (no driver attached) pci0: at device 7.7 (no driver attached) vgapci0: port 0x10d0-0x10df mem = 0xd0000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on = pci0 mpt0: port 0x1400-0x14ff mem = 0xd8820000-0xd883ffff,0xd8800000-0xd881ffff irq 17 at device 16.0 on = pci0 mpt0: [ITHREAD] mpt0: MPI Version=3D1.2.0.0 pcib2: at device 17.0 on pci0 pci2: on pcib2 em0: port 0x2000-0x203f = mem 0xd8940000-0xd895ffff,0xd8900000-0xd890ffff irq 18 at device 0.0 on = pci2 em0: Memory Access and/or Bus Master bits were not set! em0: [FILTER] em0: Ethernet address: 00:0c:29:11:4f:d5 em1: port 0x2040-0x207f = mem 0xd8960000-0xd897ffff,0xd8910000-0xd891ffff irq 19 at device 1.0 on = pci2 em1: Memory Access and/or Bus Master bits were not set! em1: [FILTER] em1: Ethernet address: 00:0c:29:11:4f:cb pci2: at device 2.0 (no driver attached) em2: port 0x20c0-0x20ff = mem 0xd8980000-0xd899ffff,0xd8920000-0xd892ffff irq 17 at device 3.0 on = pci2 em2: Memory Access and/or Bus Master bits were not set! em2: [FILTER] em2: Ethernet address: 00:0c:29:11:4f:df em3: port 0x2400-0x243f = mem 0xd89a0000-0xd89bffff,0xd8930000-0xd893ffff irq 19 at device 5.0 on = pci2 em3: Memory Access and/or Bus Master bits were not set! em3: [FILTER] em3: Ethernet address: 00:0c:29:11:4f:e9 pcib3: at device 21.0 on pci0 pci3: on pcib3 pcib4: at device 21.1 on pci0 pci4: on pcib4 pcib5: at device 21.2 on pci0 pci5: on pcib5 pcib6: at device 21.3 on pci0 pci6: on pcib6 pcib7: at device 21.4 on pci0 pci7: on pcib7 pcib8: at device 21.5 on pci0 pci8: on pcib8 pcib9: at device 21.6 on pci0 pci9: on pcib9 pcib10: at device 21.7 on pci0 pci10: on pcib10 pcib11: at device 22.0 on pci0 pci11: on pcib11 pcib12: at device 22.1 on pci0 pci12: on pcib12 pcib13: at device 22.2 on pci0 pci13: on pcib13 pcib14: at device 22.3 on pci0 pci14: on pcib14 pcib15: at device 22.4 on pci0 pci15: on pcib15 pcib16: at device 22.5 on pci0 pci16: on pcib16 pcib17: at device 22.6 on pci0 pci17: on pcib17 pcib18: at device 22.7 on pci0 pci18: on pcib18 pcib19: at device 23.0 on pci0 pci19: on pcib19 pcib20: at device 23.1 on pci0 pci20: on pcib20 pcib21: at device 23.2 on pci0 pci21: on pcib21 pcib22: at device 23.3 on pci0 pci22: on pcib22 pcib23: at device 23.4 on pci0 pci23: on pcib23 pcib24: at device 23.5 on pci0 pci24: on pcib24 pcib25: at device 23.6 on pci0 pci25: on pcib25 pcib26: at device 23.7 on pci0 pci26: on pcib26 pcib27: at device 24.0 on pci0 pci27: on pcib27 pcib28: at device 24.1 on pci0 pci28: on pcib28 pcib29: at device 24.2 on pci0 pci29: on pcib29 pcib30: at device 24.3 on pci0 pci30: on pcib30 pcib31: at device 24.4 on pci0 pci31: on pcib31 pcib32: at device 24.5 on pci0 pci32: on pcib32 pcib33: at device 24.6 on pci0 pci33: on pcib33 pcib34: at device 24.7 on pci0 pci34: on pcib34 acpi_acad0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on = acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 cpu0: on acpi0 acpi_throttle0: on cpu0 orm0: at iomem = 0xc0000-0xc7fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0x= cdfff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=3D0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on = isa0 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is = present; to enable, add "vfs.zfs.prefetch_disable=3D0" to = /boot/loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable = behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version 3 ZFS storage pool version 14 Timecounter "TSC" frequency 2116630847 Hz quality 800 Timecounters tick every 10.000 msec Expensive timeout(9) function: 0xffffffff805ed840(0) 0.002412596 s acd0: DVDR at ata1-master = UDMA33=20 da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device=20 da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C) da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-2 device=20 da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da1: Command Queueing enabled da1: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) da2 at mpt0 bus 0 scbus0 target 2 lun 0 da2: Fixed Direct Access SCSI-2 device=20 da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da2: Command Queueing enabled da2: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) da3 at mpt0 bus 0 scbus0 target 3 lun 0 da3: Fixed Direct Access SCSI-2 device=20 da3: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da3: Command Queueing enabled da3: 12288MB (25165824 512 byte sectors: 255H 63S/T 1566C) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0s1a WARNING: / was not properly dismounted /: mount pending error: blocks 4 files 1 <118>Setting hostuuid: 564d8c47-d7e5-fea4-92ef-f17760cfe4d2. <118>Setting hostid: 0x4d345a2c. lock order reversal: 1st 0xffffff000286d2e8 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:549 2nd 0xffffff000286c0d8 dn->dn_mtx (dn->dn_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:1173 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_block_freed() at dnode_block_freed+0x8e dbuf_read() at dbuf_read+0x155 dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x12a dmu_read() at dmu_read+0x80 load_nvlist() at load_nvlist+0x85 spa_load() at spa_load+0x49a spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffffd808, rbp =3D 0x801224140 --- Expensive timeout(9) function: 0xffffffff80892b50(0xffffff000244fe00) = 0.005049803 s lock order reversal: 1st 0xffffff0002868e70 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode_sync.c:381 2nd 0xffffff00026e7940 osi->os_lock (osi->os_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:323 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_destroy() at dnode_destroy+0xa6 dnode_buf_pageout() at dnode_buf_pageout+0xb2 dbuf_evict_user() at dbuf_evict_user+0x55 dbuf_clear() at dbuf_clear+0x5e dnode_evict_dbufs() at dnode_evict_dbufs+0x98 dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0x11c dmu_objset_evict() at dmu_objset_evict+0xbf dsl_pool_close() at dsl_pool_close+0x52 spa_unload() at spa_unload+0xb2 spa_load() at spa_load+0x4da spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffffd808, rbp =3D 0x801224140 --- lock order reversal: 1st 0xffffff0002864910 buf->b_lock (buf->b_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/arc.c:2509 2nd 0xffffff00028681a0 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:421 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_read_done() at dbuf_read_done+0x3b arc_read_done() at arc_read_done+0x1d2 zio_done() at zio_done+0x308 zio_execute() at zio_execute+0xb1 arc_read_nolock() at arc_read_nolock+0x3d0 arc_read() at arc_read+0xaf dbuf_read() at dbuf_read+0x62b dmu_buf_hold() at dmu_buf_hold+0xcc zap_lockdir() at zap_lockdir+0x6e zap_lookup_norm() at zap_lookup_norm+0x45 zap_lookup() at zap_lookup+0x2e dsl_prop_get_dd() at dsl_prop_get_dd+0xfc dsl_dataset_get_ref() at dsl_dataset_get_ref+0x4a7 dsl_dataset_hold() at dsl_dataset_hold+0xf2 dmu_objset_open() at dmu_objset_open+0xcf zil_check_log_chain() at zil_check_log_chain+0x2a dmu_objset_find_spa() at dmu_objset_find_spa+0x23f dmu_objset_find_spa() at dmu_objset_find_spa+0x324 dmu_objset_find_spa() at dmu_objset_find_spa+0x324 dmu_objset_find() at dmu_objset_find+0x2a spa_check_logs() at spa_check_logs+0x2a spa_load() at spa_load+0x6ca spa_load() at spa_load+0x4fe spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffffd808, rbp =3D 0x801224140 --- lock order reversal: 1st 0xffffff000286d058 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1116 2nd 0xffffff000259ac38 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1120 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_dirty() at dbuf_dirty+0x892 dnode_setdirty() at dnode_setdirty+0x1a9 dbuf_dirty() at dbuf_dirty+0xa53 bplist_vacate() at bplist_vacate+0x4d spa_sync() at spa_sync+0x297 txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff000292d538 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 2nd 0xffffff000250c2f0 spa->spa_sync_bplist.bpl_lock = (spa->spa_sync_bplist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:235 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 bplist_enqueue_deferred() at bplist_enqueue_deferred+0x47 zio_free() at zio_free+0x105 arc_free() at arc_free+0x11c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x483 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x3eb dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync() at dmu_objset_sync+0x134 dsl_pool_sync() at dsl_pool_sync+0x200 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002948c38 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 2nd 0xffffff00028a7000 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1859 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dbuf_check_blkptr() at dbuf_check_blkptr+0x11e dbuf_sync_list() at dbuf_sync_list+0x34a dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync_dnodes() at dmu_objset_sync_dnodes+0x60 dmu_objset_sync() at dmu_objset_sync+0x162 dsl_pool_sync() at dsl_pool_sync+0x200 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- <118>Entropy harvesting: <118> interrupts <118> ethernet <118> point_to_point <118> kickstart <118>. <118>Starting file system checks: <118>/dev/da0s1a: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) <118>/dev/da0s1a: SUMMARY INFORMATION BAD (SALVAGED) <118>/dev/da0s1a: BLK(S) MISSING IN BIT MAPS (SALVAGED) <118>/dev/da0s1a: 28961 files, 517324 used, 1004493 free (7877 frags, = 124577 blocks, 0.5% fragmentation) <118>Mounting local file systems: <118>. lock order reversal: 1st 0xffffff000263acc8 zfs (zfs) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/gfs.c:437 2nd 0xffffff000256e310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23c zfs_root() at zfs_root+0x50 zfsctl_create() at zfsctl_create+0x82 zfs_mount() at zfs_mount+0x7ef vfs_donmount() at vfs_donmount+0xcde nmount() at nmount+0x63 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x800f48f1c, rsp =3D = 0x7fffffffced8, rbp =3D 0x7fffffffcef8 --- lock order reversal: 1st 0xffffff0002996078 zp->z_name_lock (zp->z_name_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_dir.c:212 2nd 0xffffff000256e330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23c zfs_dirent_lock() at zfs_dirent_lock+0x4a0 zfs_dirlook() at zfs_dirlook+0x90 zfs_lookup() at zfs_lookup+0x256 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800fd8acc, rsp =3D = 0x7fffffffcf38, rbp =3D 0x7fffffffd3d0 --- lock order reversal: 1st 0xffffff000256e210 zfsvfs->z_teardown_inactive_lock = (zfsvfs->z_teardown_inactive_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_vnops.c:3724 2nd 0xffffff000256e330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:1027 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zinactive() at zfs_zinactive+0x95 zfs_inactive() at zfs_inactive+0x7e zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5 vinactive() at vinactive+0x90 vputx() at vputx+0x2fc kern_statat_vnhook() at kern_statat_vnhook+0xfa kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800fd8acc, rsp =3D = 0x7fffffffcf38, rbp =3D 0x7fffffffd3d0 --- <118>/etc/rc: WARNING: $hostname is not set -- see rc.conf(5). <118>Starting Network: lo0 em0 em1 em2 em3. <118>lo0: flags=3D8049 metric 0 mtu 16384 <118> options=3D3 <118> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6=20 <118> inet6 ::1 prefixlen 128=20 <118> inet 127.0.0.1 netmask 0xff000000=20 <118> nd6 options=3D3 <118>em0: flags=3D8843 metric 0 = mtu 1500 <118> options=3D9b <118> ether 00:0c:29:11:4f:d5 <118> inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255 <118> media: Ethernet autoselect (1000baseT ) <118> status: active <118>em1: flags=3D8843 metric 0 = mtu 1500 <118> options=3D9b <118> ether 00:0c:29:11:4f:cb <118> inet 10.0.1.2 netmask 0xffffff00 broadcast 10.0.1.255 <118> media: Ethernet autoselect (1000baseT ) <118> status: active <118>em2: flags=3D8843 metric 0 = mtu 1500 <118> options=3D9b <118> ether 00:0c:29:11:4f:df <118> inet 10.0.2.2 netmask 0xffffff00 broadcast 10.0.2.255 <118> media: Ethernet autoselect (1000baseT ) <118> status: active <118>em3: flags=3D8843 metric 0 = mtu 1500 <118> options=3D9b <118> ether 00:0c:29:11:4f:e9 <118> inet 172.16.2.118 netmask 0xffff0000 broadcast 172.16.255.255 <118> media: Ethernet autoselect (1000baseT ) <118> status: active <118>add net default: gateway 172.16.0.2 Expensive timeout(9) function: 0xffffffff80329580(0xffffff8000272000) = 0.006700013 s <118>Starting devd. ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based = forwarding disabled, default to deny, logging disabled <118>Flushed all rules. <118>00100 allow ip from any to any via lo0 <118>00200 deny ip from any to 127.0.0.0/8 <118>00300 deny ip from 127.0.0.0/8 to any <118>65000 allow ip from any to any <118>Firewall rules loaded. <118>Creating and/or trimming log files <118>. <118>Starting syslogd. <118>No core dumps found. <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib <118>32-bit compatibility ldconfig path: /usr/lib32 lock order reversal: 1st 0xffffff800a2f62e8 bufwait (bufwait) @ = /pool/newsrc/src/sys/kern/vfs_bio.c:2559 2nd 0xffffff0002c71400 dirhash (dirhash) @ = /pool/newsrc/src/sys/ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 ufsdirhash_acquire() at ufsdirhash_acquire+0x44 ufsdirhash_add() at ufsdirhash_add+0x19 ufs_direnter() at ufs_direnter+0x88b ufs_mkdir() at ufs_mkdir+0x623 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xb9 kern_mkdirat() at kern_mkdirat+0x264 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (136, FreeBSD ELF64, mkdir), rip =3D 0x80072bb0c, rsp =3D = 0x7fffffffec88, rbp =3D 0x7fffffffef66 --- <118>Clearing /tmp (X related). <118>Updating motd: <118>. <118>Configuring syscons: <118> blanktime <118>. <118>Starting sshd. <118>Starting cron. <118>Starting background file system checks in 60 seconds. <118> <118>Wed Mar 10 17:29:09 CET 2010 <118>Mar 10 17:29:21 su: BAD SU borjam to root on /dev/pts/0 <118>Mar 10 17:29:24 su: BAD SU borjam to root on /dev/pts/0 <118>Mar 10 17:29:26 su: borjam to root on /dev/pts/0 <118>Mar 10 17:30:36 su: borjam to root on /dev/pts/1 lock order reversal: 1st 0xffffff00027e2438 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 2nd 0xffffff000286bb88 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:130 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dnode_verify() at dnode_verify+0x70 dnode_hold_impl() at dnode_hold_impl+0x73 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x3eb dbuf_sync_list() at dbuf_sync_list+0x17f Fatal double fault rip =3D 0xffffffff805d322d rsp =3D 0xffffff8012414fe0 rbp =3D 0xffffff8012415090 cpuid =3D 0; apic id =3D 00 panic: double fault cpuid =3D 0 KDB: enter: panic shared sx zfsvfs->z_teardown_inactive_lock = (zfsvfs->z_teardown_inactive_lock) r =3D 0 (0xffffff000256d210) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_vnops.c:3724 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff00055b3a58) locked @ = /pool/newsrc/src/sys/kern/vfs_subr.c:2091 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff0005618308) locked @ = /pool/newsrc/src/sys/kern/vfs_syscalls.c:3174 shared lockmgr zfs (zfs) r =3D 0 (0xffffff00057bda58) locked @ = /pool/newsrc/src/sys/kern/vfs_subr.c:2091 exclusive sx zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) r =3D 0 = (0xffffff000256d310) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:866 shared sx zp->z_name_lock (zp->z_name_lock) r =3D 0 (0xffffff00057c7c38) = locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_dir.c:212 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff00057ac578) locked @ = /pool/newsrc/src/sys/kern/vfs_subr.c:2091 exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xffffff0002f63648) locked @ = /pool/newsrc/src/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xffffff0002cff3a0) locked @ = /pool/newsrc/src/sys/kern/uipc_sockbuf.c:148 exclusive sx ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) r =3D 0 = (0xffffff00027e2438) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 = (0xffffff0002948238) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 = (0xffffff0002963538) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 = (0xffffff0002949438) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 = (0xffffff0002948b38) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 = (0xffffff0002966238) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 = (0xffffff0002963438) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 0xffffff00055b39c0: tag zfs, type VDIR usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOINGINACT) lock type zfs: EXCL by thread 0xffffff000528b3a0 (pid 1425) #0 0xffffffff80578cef at __lockmgr_args+0x4ef #1 0xffffffff80610a59 at vop_stdlock+0x39 #2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062d957 at _vn_lock+0x57 #4 0xffffffff806210bb at vget+0x7b #5 0xffffffff8060e570 at cache_lookup+0x4e0 #6 0xffffffff8060e890 at vfs_cache_lookup+0xc0 #7 0xffffffff808cbdb7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615513 at lookup+0x3d3 #9 0xffffffff806164e9 at namei+0x4a9 #10 0xffffffff806255cf at kern_statat_vnhook+0x8f #11 0xffffffff80625775 at kern_statat+0x15 #12 0xffffffff8062583a at lstat+0x2a #13 0xffffffff80876b48 at syscall+0x118 #14 0xffffffff8085cf91 at Xfast_syscall+0xe1 0xffffff0005618270: tag zfs, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type zfs: EXCL by thread 0xffffff000528c000 (pid 1422) #0 0xffffffff80578f77 at __lockmgr_args+0x777 #1 0xffffffff80610a59 at vop_stdlock+0x39 #2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062d957 at _vn_lock+0x57 #4 0xffffffff806299ba at setutimes+0x7a #5 0xffffffff80629cb2 at kern_lutimes+0xf2 #6 0xffffffff80876b48 at syscall+0x118 #7 0xffffffff8085cf91 at Xfast_syscall+0xe1 0xffffff00057bd9c0: tag zfs, type VDIR usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type zfs: SHARED (count 1) #0 0xffffffff80578cef at __lockmgr_args+0x4ef #1 0xffffffff80610a59 at vop_stdlock+0x39 #2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062d957 at _vn_lock+0x57 #4 0xffffffff806210bb at vget+0x7b #5 0xffffffff8060e570 at cache_lookup+0x4e0 #6 0xffffffff8060e890 at vfs_cache_lookup+0xc0 #7 0xffffffff808cbdb7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615513 at lookup+0x3d3 #9 0xffffffff806164e9 at namei+0x4a9 #10 0xffffffff806255cf at kern_statat_vnhook+0x8f #11 0xffffffff80625775 at kern_statat+0x15 #12 0xffffffff8062583a at lstat+0x2a #13 0xffffffff80876b48 at syscall+0x118 #14 0xffffffff8085cf91 at Xfast_syscall+0xe1 0xffffff00057ac4e0: tag zfs, type VDIR usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type zfs: EXCL by thread 0xffffff000262a3a0 (pid 1416) with shared waiters pending #0 0xffffffff80578cef at __lockmgr_args+0x4ef #1 0xffffffff80610a59 at vop_stdlock+0x39 #2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062d957 at _vn_lock+0x57 #4 0xffffffff806210bb at vget+0x7b #5 0xffffffff8060e570 at cache_lookup+0x4e0 #6 0xffffffff8060e890 at vfs_cache_lookup+0xc0 #7 0xffffffff808cbdb7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615513 at lookup+0x3d3 #9 0xffffffff806164e9 at namei+0x4a9 #10 0xffffffff80624df3 at kern_mkdirat+0xc3 #11 0xffffffff80876b48 at syscall+0x118 #12 0xffffffff8085cf91 at Xfast_syscall+0xe1 Uptime: 2m16s #=20 Kernel configuration follows: # cat config.txt=20 options CONFIG_AUTOGENERATED ident DEBUG machine amd64 cpu HAMMER makeoptions DEBUG=3D-g options AH_SUPPORT_AR5416 options IEEE80211_SUPPORT_MESH options IEEE80211_AMPDU_AGE options IEEE80211_DEBUG options AHD_REG_PRETTY_PRINT options AHC_REG_PRETTY_PRINT options ATA_STATIC_ID options SMP options DEBUG_LOCKS options DEBUG_VFS_LOCKS options DIAGNOSTIC options INVARIANT_SUPPORT options INVARIANTS options DDB options KDB options WITNESS_SKIPSPIN options WITNESS options INCLUDE_CONFIG_FILE options FLOWTABLE options MAC options AUDIT options HWPMC_HOOKS options KBD_INSTALL_CDEV options PRINTF_BUFR_SIZE=3D128 options _KPOSIX_PRIORITY_SCHEDULING options P1003_1B_SEMAPHORES options SYSVSEM options SYSVMSG options SYSVSHM options STACK options KTRACE options SCSI_DELAY=3D5000 options COMPAT_FREEBSD7 options COMPAT_FREEBSD6 options COMPAT_FREEBSD5 options COMPAT_FREEBSD4 options COMPAT_IA32 options COMPAT_43TTY options GEOM_LABEL options GEOM_PART_GPT options PSEUDOFS options PROCFS options CD9660 options MSDOSFS options NFS_ROOT options NFSLOCKD options NFSSERVER options NFSCLIENT options MD_ROOT options UFS_GJOURNAL options UFS_DIRHASH options UFS_ACL options SOFTUPDATES options FFS options SCTP options INET6 options INET options PREEMPTION options SCHED_ULE options GEOM_PART_MBR options GEOM_PART_EBR_COMPAT options GEOM_PART_EBR options GEOM_PART_BSD device isa device mem device io device uart_ns8250 device cpufreq device acpi device pci device fdc device ata device atadisk device ataraid device atapicd device atapifd device atapist device ahc device ahd device amd device hptiop device isp device mpt device sym device trm device adv device adw device aic device bt device scbus device ch device da device sa device cd device pass device ses device amr device arcmsr device ciss device dpt device hptmv device hptrr device iir device ips device mly device twa device aac device aacp device ida device mfi device mlx device twe device atkbdc device atkbd device psm device kbdmux device vga device splash device sc device agp device cbb device pccard device cardbus device uart device ppc device ppbus device lpt device plip device ppi device de device em device igb device ixgbe device le device ti device txp device vx device miibus device ae device age device alc device ale device bce device bfe device bge device dc device et device fxp device jme device lge device msk device nfe device nge device pcn device re device rl device sf device sis device sk device ste device stge device tl device tx device vge device vr device wb device xl device cs device ed device ex device ep device fe device sn device xe device wlan device wlan_wep device wlan_ccmp device wlan_tkip device wlan_amrr device an device ath device ath_hal device ath_rate_sample device ral device wi device loop device random device ether device vlan device tun device pty device md device gif device faith device firmware device bpf device uhci device ohci device ehci device usb device uhid device ukbd device ulpt device umass device ums device urio device uark device ubsa device uftdi device uipaq device uplcom device uslcom device uvisor device uvscom device aue device axe device cdce device cue device kue device rue device udav device rum device uath device ural device zyd device firewire device fwe device fwip device dcons device dcons_crom #=20 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 10 15:12:44 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3F6301065670; Wed, 10 Mar 2010 15:12:44 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop2.sarenet.es (proxypop2.sarenet.es [194.30.0.95]) by mx1.freebsd.org (Postfix) with ESMTP id E44F58FC12; Wed, 10 Mar 2010 15:12:38 +0000 (UTC) Received: from [172.16.1.204] (unknown [192.148.167.2]) by proxypop2.sarenet.es (Postfix) with ESMTP id 955B0732A4; Wed, 10 Mar 2010 16:12:37 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20100310110202.GA1715@garage.freebsd.pl> Date: Wed, 10 Mar 2010 16:12:36 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> To: Pawel Jakub Dawidek X-Mailer: Apple Mail (2.1077) Cc: freebsd-fs@freebsd.org, FreeBSD Stable Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 15:12:44 -0000 =09 On Mar 10, 2010, at 12:02 PM, Pawel Jakub Dawidek wrote: > Once the deadlock occur, enter DDB and send me the output of: >=20 > ps > show alllocks > show lockedvnods > show allchains > alltrace (Again, crossposted to -fs, ZFS related) Previous one was a panic when performing the test with several tar jobs = running in parallel. Now this is a capture of the deadlock itself, instead of a panic. (I = called panic from the debugger to generate a dump) ddb.txt: # cat ddb.txt db> set $lines=3D0 db> ps pid ppid pgrp uid state wmesg wchan cmd 1821 1385 1821 0 R+ CPU 0 sysctl 1820 1818 1818 0 S tx->tx_s 0xffffff00027da650 zfs 1818 1816 1818 0 Ss pause 0xffffff0002e33960 csh 1816 1284 1816 0 Ss select 0xffffff001c6d3cc0 sshd 1699 1697 1697 0 S+ zfsvfs-> 0xffffff0002893210 bsdtar 1698 1697 1697 0 S+ pipdwt 0xffffff00025d62d8 bsdtar 1697 1385 1697 0 S+ wait 0xffffff0002d47000 sh 1385 1384 1385 0 R+ csh 1384 1381 1384 1001 S+ wait 0xffffff0002e33460 su 1381 1380 1381 1001 Ss+ pause 0xffffff0002e36500 csh 1380 1377 1377 1001 R sshd 1377 1284 1377 0 Ss sbwait 0xffffff0002c763ec sshd 1376 1 1376 0 Ss+ ttyin 0xffffff00024daca8 getty 1375 1 1375 0 Ss+ ttyin 0xffffff00024dc0a8 getty 1374 1 1374 0 Ss+ ttyin 0xffffff00024cdca8 getty 1373 1 1373 0 Ss+ ttyin 0xffffff00024c94a8 getty 1372 1 1372 0 Ss+ ttyin 0xffffff00024dd4a8 getty 1371 1 1371 0 Ss+ ttyin 0xffffff00024c28a8 getty 1370 1 1370 0 Ss+ ttyin 0xffffff00024db4a8 getty 1369 1 1369 0 Ss+ ttyin 0xffffff00024dc8a8 getty 1305 1 1305 0 Ss nanslp 0xffffffff80c4d168 cron 1298 1 1298 25 Ss pause 0xffffff000257d500 sendmail 1292 1 1292 0 Ss select 0xffffff000287f5c0 sendmail 1284 1 1284 0 Ss select 0xffffff000287f940 sshd 1005 1 1005 0 Ss select 0xffffff0002c6e2c0 syslogd 827 1 827 0 Ss select 0xffffff000296f440 devd 20 0 0 0 SL flowclea 0xffffffff80e0e950 = [flowcleaner] 19 0 0 0 SL sdflush 0xffffffff80e1d8b8 = [softdepflush] 18 0 0 0 SL rrl->rr_ 0xffffff0002893110 [syncer] 17 0 0 0 SL zfsvfs-> 0xffffff0002893210 [vnlru] 16 0 0 0 SL psleep 0xffffffff80e0e1c8 [bufdaemon] 15 0 0 0 SL pgzero 0xffffffff80e1f34c [pagezero] 14 0 0 0 SL psleep 0xffffffff80e1e6e8 [vmdaemon] 9 0 0 0 SL psleep 0xffffffff80e1e6ac = [pagedaemon] 8 0 0 0 SL ccb_scan 0xffffffff80c16060 [xpt_thrd] 7 0 0 0 SL (threaded) zfskern 100085 D tx->tx_q 0xffffff00027da670 = [txg_thread_enter] 100084 D tx->tx_c 0xffffff0002599d20 = [txg_thread_enter] 100082 D vgeom:io 0xffffff00026acd90 [vdev da2] 100036 D l2arc_fe 0xffffffff81150620 = [l2arc_feed_thread] 100035 D arc_recl 0xffffffff81147fe0 = [arc_reclaim_thread] 6 0 0 0 SL - 0xffffff000244e648 [fdc0] 5 0 0 0 SL idle 0xffffff800023c000 = [mpt_recovery0] 13 0 0 0 SL - 0xffffffff80c4ce44 [yarrow] 4 0 0 0 SL - 0xffffffff80c49648 [g_down] 3 0 0 0 SL - 0xffffffff80c49640 [g_up] 2 0 0 0 SL - 0xffffffff80c49630 [g_event] 12 0 0 0 WL (threaded) intr 100033 I [swi0: uart = uart] 100032 I [irq7: = ppc0] 100031 I [irq12: = psm0] 100030 I [irq1: = atkbd0] 100024 I [irq17: em2 = mpt0] 100023 I [irq15: = ata1] 100022 I [irq14: = ata0] 100021 I [irq9: = acpi0] 100019 I [swi5: +] 100018 I [swi2: = cambio] 100013 I [swi6: task = queue] 100012 I [swi6: = Giant taskq] 100006 I [swi1: = netisr 0] 100005 I [swi4: = clock] 100004 I [swi3: vm] 11 0 0 0 RL [idle: = cpu0] 1 0 1 0 SLs wait 0xffffff00021e18c0 [init] 10 0 0 0 SL audit_wo 0xffffffff80e1cc10 [audit] 0 0 0 0 SLs (threaded) kernel 100087 D - 0xffffff000287c180 [zil_clean] 100086 D - 0xffffff000287b200 [zil_clean] 100083 D - 0xffffff0002640480 = [zfs_vn_rele_taskq] 100081 D - 0xffffff00026acc00 [spa_zio] 100080 D - 0xffffff00026acb80 [spa_zio] 100079 D - 0xffffff00026acb00 [spa_zio] 100078 D - 0xffffff00026aca80 [spa_zio] 100077 D - 0xffffff00026aca00 [spa_zio] 100076 D - 0xffffff00026ac980 [spa_zio] 100075 D - 0xffffff00026ac900 [spa_zio] 100074 D - 0xffffff00026ac880 [spa_zio_7] 100073 D - 0xffffff00026ac880 [spa_zio_6] 100072 D - 0xffffff00026ac880 [spa_zio_5] 100071 D - 0xffffff00026ac880 [spa_zio_4] 100070 D - 0xffffff00026ac880 [spa_zio_3] 100069 D - 0xffffff00026ac880 [spa_zio_2] 100068 D - 0xffffff00026ac880 [spa_zio_1] 100067 D - 0xffffff00026ac880 [spa_zio_0] 100066 D - 0xffffff00026ac800 [spa_zio_7] 100065 D - 0xffffff00026ac800 [spa_zio_6] 100064 D - 0xffffff00026ac800 [spa_zio_5] 100063 D - 0xffffff00026ac800 [spa_zio_4] 100062 D - 0xffffff00026ac800 [spa_zio_3] 100061 D - 0xffffff00026ac800 [spa_zio_2] 100060 D - 0xffffff00026ac800 [spa_zio_1] 100059 D - 0xffffff00026ac800 [spa_zio_0] 100058 D - 0xffffff00026ac780 [spa_zio] 100057 D - 0xffffff00026ac700 [spa_zio] 100056 D - 0xffffff00026ac680 [spa_zio] 100029 D - 0xffffff0002441980 [em3 taskq] 100028 D - 0xffffff0002440d80 [em2 taskq] 100027 D - 0xffffff0002436200 [em1 taskq] 100026 D - 0xffffff000242fb80 [em0 taskq] 100020 D - 0xffffff0002358600 [thread = taskq] 100017 D - 0xffffff0002316b80 = [acpi_task_2] 100016 D - 0xffffff0002316b80 = [acpi_task_1] 100015 D - 0xffffff0002316b80 = [acpi_task_0] 100014 D - 0xffffff0002316c00 [kqueue = taskq] 100010 D - 0xffffff00021e0180 [firmware = taskq] 100000 D sched 0xffffffff80c49740 [swapper] db> show alllocks Process 1821 (sysctl) thread 0xffffff00025b0ae0 (100050) Process 1820 (zfs) thread 0xffffff0002ca4000 (100105) Process 1699 (bsdtar) thread 0xffffff000262dae0 (100093) Process 1377 (sshd) thread 0xffffff0002642740 (100090) Process 18 (syncer) thread 0xffffff000250d3a0 (100043) Process 17 (vnlru) thread 0xffffff000250d740 (100042) db> show lockedvnods Locked vnodes db> show allchains db> alltrace Tracing command sysctl pid 1821 tid 100050 td 0xffffff00025b0ae0 kdb_enter() at kdb_enter+0x3d kdb_sysctl_enter() at kdb_sysctl_enter+0x89 sysctl_root() at sysctl_root+0x113 userland_sysctl() at userland_sysctl+0x158 __sysctl() at __sysctl+0xaa syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (202, FreeBSD ELF64, __sysctl), rip =3D 0x80073cfac, rsp =3D = 0x7fffffffe298, rbp =3D 0x7fffffffe36c --- Tracing command zfs pid 1820 tid 100105 td 0xffffff0002ca4000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a txg_wait_synced() at txg_wait_synced+0x98 zfsvfs_teardown() at zfsvfs_teardown+0x1f6 zfs_suspend_fs() at zfs_suspend_fs+0x2b zfs_ioc_recv() at zfs_ioc_recv+0x28b zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffff8e98, rbp =3D 0x7fffffff9bd0 --- Tracing command csh pid 1818 tid 100102 td 0xffffff0002ca4ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x80094d19c, rsp =3D = 0x7fffffffe578, rbp =3D 0x800c0f500 --- Tracing command sshd pid 1816 tid 100098 td 0xffffff0002d2bae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d7cfc, rsp =3D = 0x7fffffffdcb8, rbp =3D 0x7fffffffdd40 --- Tracing command bsdtar pid 1699 tid 100093 td 0xffffff000262dae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_slock_hard() at _sx_slock_hard+0x1b7 _sx_slock() at _sx_slock+0xc1 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x63 VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 getnewvnode() at getnewvnode+0x24f zfs_znode_cache_constructor() at zfs_znode_cache_constructor+0x43 zfs_znode_alloc() at zfs_znode_alloc+0x38 zfs_mknode() at zfs_mknode+0x259 zfs_freebsd_create() at zfs_freebsd_create+0x661 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x473 kern_openat() at kern_openat+0x179 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip =3D 0x800e35b4c, rsp =3D = 0x7fffffffe748, rbp =3D 0x1a4 --- Tracing command bsdtar pid 1698 tid 100107 td 0xffffff0002c90740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_write() at pipe_write+0x13fc dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e44d5c, rsp =3D = 0x7fffffffe818, rbp =3D 0x801004140 --- Tracing command sh pid 1697 tid 100091 td 0xffffff00026423a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x80093554c, rsp =3D = 0x7fffffffe8a8, rbp =3D 0x6a1 --- Tracing command csh pid 1385 tid 100104 td 0xffffff0002ca43a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a fork1() at fork1+0xfe2 vfork() at vfork+0x1c syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (66, FreeBSD ELF64, vfork), rip =3D 0x80094f31d, rsp =3D = 0x7fffffffe6c0, rbp =3D 0 --- Tracing command su pid 1384 tid 100103 td 0xffffff0002ca4740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009e854c, rsp =3D = 0x7fffffffe598, rbp =3D 0x569 --- Tracing command csh pid 1381 tid 100100 td 0xffffff0002d2b3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x80094d19c, rsp =3D = 0x7fffffffe558, rbp =3D 0x800c0f900 --- Tracing command sshd pid 1380 tid 100099 td 0xffffff0002d2b740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d7cfc, rsp =3D = 0x7fffffffdcb8, rbp =3D 0x7fffffffdd40 --- Tracing command sshd pid 1377 tid 100090 td 0xffffff0002642740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x8013d7d7c, rsp =3D = 0x7fffffffdcd8, rbp =3D 0 --- Tracing command getty pid 1376 tid 100046 td 0xffffff00025b1ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1375 tid 100095 td 0xffffff000262d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1374 tid 100096 td 0xffffff0002cd43a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1373 tid 100097 td 0xffffff0002cd4000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1372 tid 100092 td 0xffffff0002642000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1371 tid 100094 td 0xffffff000262d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1370 tid 100048 td 0xffffff00025b13a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1369 tid 100054 td 0xffffff0002510ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084ed7c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command cron pid 1305 tid 100052 td 0xffffff00025b03a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip =3D 0x80093efac, rsp =3D = 0x7fffffffeb28, rbp =3D 0x3c --- Tracing command sendmail pid 1298 tid 100055 td 0xffffff0002510740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800d3a19c, rsp =3D = 0x7fffffffcd68, rbp =3D 0x1 --- Tracing command sendmail pid 1292 tid 100053 td 0xffffff00025b0000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x800dd8cfc, rsp =3D = 0x7fffffffc1e8, rbp =3D 0x7fffffffc280 --- Tracing command sshd pid 1284 tid 100047 td 0xffffff00025b1740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d7cfc, rsp =3D = 0x7fffffffddd8, rbp =3D 0x2 --- Tracing command syslogd pid 1005 tid 100051 td 0xffffff00025b0740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x800851cfc, rsp =3D = 0x7fffffffe118, rbp =3D 0x800a310c8 --- Tracing command devd pid 827 tid 100049 td 0xffffff00025b1000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x440ecc, rsp =3D = 0x7fffffffe898, rbp =3D 0x7fffffffe8b0 --- Tracing command flowcleaner pid 20 tid 100045 td 0xffffff0002485ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c flowtable_cleaner() at flowtable_cleaner+0xde fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800018cd30, rbp =3D 0 --- Tracing command softdepflush pid 19 tid 100044 td 0xffffff000250d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 softdep_flush() at softdep_flush+0x259 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000187d30, rbp =3D 0 --- Tracing command syncer pid 18 tid 100043 td 0xffffff000250d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a rrw_enter() at rrw_enter+0x135 zfs_sync() at zfs_sync+0x63 sync_fsync() at sync_fsync+0x13a VOP_FSYNC_APV() at VOP_FSYNC_APV+0xb5 sync_vnode() at sync_vnode+0x157 sched_sync() at sched_sync+0x1cc fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000182d30, rbp =3D 0 --- Tracing command vnlru pid 17 tid 100042 td 0xffffff000250d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_slock_hard() at _sx_slock_hard+0x1b7 _sx_slock() at _sx_slock+0xc1 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x63 VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 vnlru_proc() at vnlru_proc+0x58e fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800017dd30, rbp =3D 0 --- Tracing command bufdaemon pid 16 tid 100041 td 0xffffff000250dae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 buf_daemon() at buf_daemon+0x14a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000178d30, rbp =3D 0 --- Tracing command pagezero pid 15 tid 100040 td 0xffffff000250e000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000173d30, rbp =3D 0 --- Tracing command vmdaemon pid 14 tid 100039 td 0xffffff000250e3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800016ed30, rbp =3D 0 --- Tracing command pagedaemon pid 9 tid 100038 td 0xffffff000250e740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pageout() at vm_pageout+0x855 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000169d30, rbp =3D 0 --- Tracing command xpt_thrd pid 8 tid 100037 td 0xffffff000250eae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 xpt_scanner_thread() at xpt_scanner_thread+0xd5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000164d30, rbp =3D 0 --- Tracing command zfskern pid 7 tid 100085 td 0xffffff0002643ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a txg_thread_wait() at txg_thread_wait+0x5b txg_sync_thread() at txg_sync_thread+0x1d0 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- Tracing command zfskern pid 7 tid 100084 td 0xffffff0002645000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a txg_quiesce_thread() at txg_quiesce_thread+0x22b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012413d30, rbp =3D 0 --- Tracing command zfskern pid 7 tid 100082 td 0xffffff000263dae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vdev_geom_worker() at vdev_geom_worker+0xf3 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123ffd30, rbp =3D 0 --- Tracing command zfskern pid 7 tid 100036 td 0xffffff0002510000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c l2arc_feed_thread() at l2arc_feed_thread+0x162 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800015fd30, rbp =3D 0 --- Tracing command zfskern pid 7 tid 100035 td 0xffffff0002356740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c arc_reclaim_thread() at arc_reclaim_thread+0x2b7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800015ad30, rbp =3D 0 --- Tracing command fdc0 pid 6 tid 100034 td 0xffffff0002356ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 fdc_thread() at fdc_thread+0x774 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000155d30, rbp =3D 0 --- Tracing command mpt_recovery0 pid 5 tid 100025 td 0xffffff0002353ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 mpt_recovery_thread() at mpt_recovery_thread+0x251 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800011ed30, rbp =3D 0 --- Tracing command yarrow pid 13 tid 100011 td 0xffffff00021e53a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 random_kthread() at random_kthread+0x1ad fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800004cd30, rbp =3D 0 --- Tracing command g_down pid 4 tid 100009 td 0xffffff00021e5ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 g_io_schedule_down() at g_io_schedule_down+0x236 g_down_procbody() at g_down_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000042d30, rbp =3D 0 --- Tracing command g_up pid 3 tid 100008 td 0xffffff00021f3000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 g_io_schedule_up() at g_io_schedule_up+0x133 g_up_procbody() at g_up_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800003dd30, rbp =3D 0 --- Tracing command g_event pid 2 tid 100007 td 0xffffff00021f33a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 g_event_procbody() at g_event_procbody+0xa1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000038d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100033 td 0xffffff0002483000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100032 td 0xffffff00024833a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100031 td 0xffffff0002483740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100030 td 0xffffff0002483ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000137d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100024 td 0xffffff0002354000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80000d7d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100023 td 0xffffff00023543a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80000d2d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100022 td 0xffffff0002354740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100021 td 0xffffff0002354ae0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100019 td 0xffffff00023563a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000078d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100018 td 0xffffff00021f3ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000073d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100013 td 0xffffff0002353000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800005ad30, rbp =3D 0 --- Tracing command intr pid 12 tid 100012 td 0xffffff00023533a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100006 td 0xffffff00021f3740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000033d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100005 td 0xffffff00021e4000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800002ed30, rbp =3D 0 --- Tracing command intr pid 12 tid 100004 td 0xffffff00021e43a0 fork_trampoline() at fork_trampoline Tracing command idle pid 11 tid 100003 td 0xffffff00021e4740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d critical_exit() at critical_exit+0xa3 intr_event_handle() at intr_event_handle+0xd7 intr_execute_handlers() at intr_execute_handlers+0x5f lapic_handle_intr() at lapic_handle_intr+0x37 Xapic_isr1() at Xapic_isr1+0xa4 --- interrupt, rip =3D 0xffffffff808545f6, rsp =3D 0xffffff8000024b50, = rbp =3D 0xffffff8000024b60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000024d30, rbp =3D 0 --- Tracing command init pid 1 tid 100002 td 0xffffff00021e4ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x40c86c, rsp =3D = 0x7fffffffe808, rbp =3D 0x401d40 --- Tracing command audit pid 10 tid 100001 td 0xffffff00021e5000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a audit_worker() at audit_worker+0x77 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800001ad30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100087 td 0xffffff00026433a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012421d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100086 td 0xffffff0002643740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff801241cd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100083 td 0xffffff000263d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012404d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100081 td 0xffffff000263e000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123fad30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100080 td 0xffffff000263e3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123f5d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100079 td 0xffffff000263e740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123f0d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100078 td 0xffffff000263eae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123ebd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100077 td 0xffffff0002628000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123e6d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100076 td 0xffffff0002628740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123e1d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100075 td 0xffffff0002628ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123dcd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100074 td 0xffffff00025103a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123d7d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100073 td 0xffffff0002645740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123d2d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100072 td 0xffffff00026453a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123cdd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100071 td 0xffffff000262a000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80123c8d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100070 td 0xffffff000262a3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001ffd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100069 td 0xffffff000262a740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001fad30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100068 td 0xffffff000262aae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001d2d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100067 td 0xffffff000262b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001f5d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100066 td 0xffffff000262b3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001f0d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100065 td 0xffffff000262b740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001ebd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100064 td 0xffffff000262bae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001e6d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100063 td 0xffffff000262c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001e1d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100062 td 0xffffff000262c3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001dcd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100061 td 0xffffff000262c740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001d7d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100060 td 0xffffff000262cae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001cdd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100059 td 0xffffff000262d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001c8d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100058 td 0xffffff000263d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001c3d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100057 td 0xffffff000263d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012409d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100056 td 0xffffff00026283a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff801240ed30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100029 td 0xffffff0002485000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000132d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100028 td 0xffffff00024853a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800012dd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100027 td 0xffffff0002485740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000128d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100026 td 0xffffff0002353740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000123d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100020 td 0xffffff0002356000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800007dd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100017 td 0xffffff0002352000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800006ed30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100016 td 0xffffff00023523a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000069d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100015 td 0xffffff0002352740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000064d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100014 td 0xffffff0002352ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800005fd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100010 td 0xffffff00021e5740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000047d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff80c49ba0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 scheduler() at scheduler+0x29c mi_startup() at mi_startup+0x59 btext() at btext+0x2c db> textdump set textdump set db> help ahd_dump ahd_in ahd_out ahd_pause ahd_sunit ahd_unpause alltrace b break bt c call capture continue d delete dhwatch dwatch examine gdb halt hwatch kill match next p panic print ps reboot reset run s script scripts search set show step t textdump thread trace unscript until w watch watchdog where write x db> panic #=20 msgbuf.txt: #=20 #=20 # cat msgbuf.txt 2400-0x243f mem 0xd89a0000-0xd89bffff,0xd8930000-0xd893ffff irq 19 at = device 5.0 on pci2 em3: Memory Access and/or Bus Master bits were not set! em3: [FILTER] em3: Ethernet address: 00:0c:29:11:4f:e9 pcib3: at device 21.0 on pci0 pci3: on pcib3 pcib4: at device 21.1 on pci0 pci4: on pcib4 pcib5: at device 21.2 on pci0 pci5: on pcib5 pcib6: at device 21.3 on pci0 pci6: on pcib6 pcib7: at device 21.4 on pci0 pci7: on pcib7 pcib8: at device 21.5 on pci0 pci8: on pcib8 pcib9: at device 21.6 on pci0 pci9: on pcib9 pcib10: at device 21.7 on pci0 pci10: on pcib10 pcib11: at device 22.0 on pci0 pci11: on pcib11 pcib12: at device 22.1 on pci0 pci12: on pcib12 pcib13: at device 22.2 on pci0 pci13: on pcib13 pcib14: at device 22.3 on pci0 pci14: on pcib14 pcib15: at device 22.4 on pci0 pci15: on pcib15 pcib16: at device 22.5 on pci0 pci16: on pcib16 pcib17: at device 22.6 on pci0 pci17: on pcib17 pcib18: at device 22.7 on pci0 pci18: on pcib18 pcib19: at device 23.0 on pci0 pci19: on pcib19 pcib20: at device 23.1 on pci0 pci20: on pcib20 pcib21: at device 23.2 on pci0 pci21: on pcib21 pcib22: at device 23.3 on pci0 pci22: on pcib22 pcib23: at device 23.4 on pci0 pci23: on pcib23 pcib24: at device 23.5 on pci0 pci24: on pcib24 pcib25: at device 23.6 on pci0 pci25: on pcib25 pcib26: at device 23.7 on pci0 pci26: on pcib26 pcib27: at device 24.0 on pci0 pci27: on pcib27 pcib28: at device 24.1 on pci0 pci28: on pcib28 pcib29: at device 24.2 on pci0 pci29: on pcib29 pcib30: at device 24.3 on pci0 pci30: on pcib30 pcib31: at device 24.4 on pci0 pci31: on pcib31 pcib32: at device 24.5 on pci0 pci32: on pcib32 pcib33: at device 24.6 on pci0 pci33: on pcib33 pcib34: at device 24.7 on pci0 pci34: on pcib34 acpi_acad0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on = acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 cpu0: on acpi0 acpi_throttle0: on cpu0 orm0: at iomem = 0xc0000-0xc7fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0x= cdfff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=3D0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on = isa0 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is = present; to enable, add "vfs.zfs.prefetch_disable=3D0" to = /boot/loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable = behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version 3 ZFS storage pool version 14 Timecounter "TSC" frequency 2116630847 Hz quality 800 Timecounters tick every 10.000 msec Expensive timeout(9) function: 0xffffffff805ed840(0) 0.002412596 s acd0: DVDR at ata1-master = UDMA33=20 da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device=20 da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C) da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-2 device=20 da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da1: Command Queueing enabled da1: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) da2 at mpt0 bus 0 scbus0 target 2 lun 0 da2: Fixed Direct Access SCSI-2 device=20 da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da2: Command Queueing enabled da2: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) da3 at mpt0 bus 0 scbus0 target 3 lun 0 da3: Fixed Direct Access SCSI-2 device=20 da3: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da3: Command Queueing enabled da3: 12288MB (25165824 512 byte sectors: 255H 63S/T 1566C) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0s1a WARNING: / was not properly dismounted /: mount pending error: blocks 4 files 1 <118>Setting hostuuid: 564d8c47-d7e5-fea4-92ef-f17760cfe4d2. <118>Setting hostid: 0x4d345a2c. lock order reversal: 1st 0xffffff000286d2e8 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:549 2nd 0xffffff000286c0d8 dn->dn_mtx (dn->dn_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:1173 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_block_freed() at dnode_block_freed+0x8e dbuf_read() at dbuf_read+0x155 dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x12a dmu_read() at dmu_read+0x80 load_nvlist() at load_nvlist+0x85 spa_load() at spa_load+0x49a spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffffd808, rbp =3D 0x801224140 --- Expensive timeout(9) function: 0xffffffff80892b50(0xffffff000244fe00) = 0.005049803 s lock order reversal: 1st 0xffffff0002868e70 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode_sync.c:381 2nd 0xffffff00026e7940 osi->os_lock (osi->os_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:323 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_destroy() at dnode_destroy+0xa6 dnode_buf_pageout() at dnode_buf_pageout+0xb2 dbuf_evict_user() at dbuf_evict_user+0x55 dbuf_clear() at dbuf_clear+0x5e dnode_evict_dbufs() at dnode_evict_dbufs+0x98 dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0x11c dmu_objset_evict() at dmu_objset_evict+0xbf dsl_pool_close() at dsl_pool_close+0x52 spa_unload() at spa_unload+0xb2 spa_load() at spa_load+0x4da spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffffd808, rbp =3D 0x801224140 --- lock order reversal: 1st 0xffffff0002864910 buf->b_lock (buf->b_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/arc.c:2509 2nd 0xffffff00028681a0 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:421 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_read_done() at dbuf_read_done+0x3b arc_read_done() at arc_read_done+0x1d2 zio_done() at zio_done+0x308 zio_execute() at zio_execute+0xb1 arc_read_nolock() at arc_read_nolock+0x3d0 arc_read() at arc_read+0xaf dbuf_read() at dbuf_read+0x62b dmu_buf_hold() at dmu_buf_hold+0xcc zap_lockdir() at zap_lockdir+0x6e zap_lookup_norm() at zap_lookup_norm+0x45 zap_lookup() at zap_lookup+0x2e dsl_prop_get_dd() at dsl_prop_get_dd+0xfc dsl_dataset_get_ref() at dsl_dataset_get_ref+0x4a7 dsl_dataset_hold() at dsl_dataset_hold+0xf2 dmu_objset_open() at dmu_objset_open+0xcf zil_check_log_chain() at zil_check_log_chain+0x2a dmu_objset_find_spa() at dmu_objset_find_spa+0x23f dmu_objset_find_spa() at dmu_objset_find_spa+0x324 dmu_objset_find_spa() at dmu_objset_find_spa+0x324 dmu_objset_find() at dmu_objset_find+0x2a spa_check_logs() at spa_check_logs+0x2a spa_load() at spa_load+0x6ca spa_load() at spa_load+0x4fe spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffffd808, rbp =3D 0x801224140 --- lock order reversal: 1st 0xffffff000286d058 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1116 2nd 0xffffff000259ac38 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1120 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_dirty() at dbuf_dirty+0x892 dnode_setdirty() at dnode_setdirty+0x1a9 dbuf_dirty() at dbuf_dirty+0xa53 bplist_vacate() at bplist_vacate+0x4d spa_sync() at spa_sync+0x297 txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff000292d538 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 2nd 0xffffff000250c2f0 spa->spa_sync_bplist.bpl_lock = (spa->spa_sync_bplist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:235 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 bplist_enqueue_deferred() at bplist_enqueue_deferred+0x47 zio_free() at zio_free+0x105 arc_free() at arc_free+0x11c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x483 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x3eb dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync() at dmu_objset_sync+0x134 dsl_pool_sync() at dsl_pool_sync+0x200 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002948c38 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 2nd 0xffffff00028a7000 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1859 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dbuf_check_blkptr() at dbuf_check_blkptr+0x11e dbuf_sync_list() at dbuf_sync_list+0x34a dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync_dnodes() at dmu_objset_sync_dnodes+0x60 dmu_objset_sync() at dmu_objset_sync+0x162 dsl_pool_sync() at dsl_pool_sync+0x200 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- <118>Entropy harvesting: <118> interrupts <118> ethernet <118> point_to_point <118> kickstart <118>. <118>Starting file system checks: <118>/dev/da0s1a: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) <118>/dev/da0s1a: SUMMARY INFORMATION BAD (SALVAGED) <118>/dev/da0s1a: BLK(S) MISSING IN BIT MAPS (SALVAGED) <118>/dev/da0s1a: 28961 files, 517324 used, 1004493 free (7877 frags, = 124577 blocks, 0.5% fragmentation) <118>Mounting local file systems: <118>. lock order reversal: 1st 0xffffff000263acc8 zfs (zfs) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/gfs.c:437 2nd 0xffffff000256e310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23c zfs_root() at zfs_root+0x50 zfsctl_create() at zfsctl_create+0x82 zfs_mount() at zfs_mount+0x7ef vfs_donmount() at vfs_donmount+0xcde nmount() at nmount+0x63 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x800f48f1c, rsp =3D = 0x7fffffffced8, rbp =3D 0x7fffffffcef8 --- lock order reversal: 1st 0xffffff0002996078 zp->z_name_lock (zp->z_name_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_dir.c:212 2nd 0xffffff000256e330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23c zfs_dirent_lock() at zfs_dirent_lock+0x4a0 zfs_dirlook() at zfs_dirlook+0x90 zfs_lookup() at zfs_lookup+0x256 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800fd8acc, rsp =3D = 0x7fffffffcf38, rbp =3D 0x7fffffffd3d0 --- lock order reversal: 1st 0xffffff000256e210 zfsvfs->z_teardown_inactive_lock = (zfsvfs->z_teardown_inactive_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_vnops.c:3724 2nd 0xffffff000256e330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:1027 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zinactive() at zfs_zinactive+0x95 zfs_inactive() at zfs_inactive+0x7e zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5 vinactive() at vinactive+0x90 vputx() at vputx+0x2fc kern_statat_vnhook() at kern_statat_vnhook+0xfa kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800fd8acc, rsp =3D = 0x7fffffffcf38, rbp =3D 0x7fffffffd3d0 --- <118>/etc/rc: WARNING: $hostname is not set -- see rc.conf(5). <118>Starting Network: lo0 em0 em1 em2 em3. <118>lo0: flags=3D8049 metric 0 mtu 16384 <118> options=3D3 <118> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6=20 <118> inet6 ::1 prefixlen 128=20 <118> inet 127.0.0.1 netmask 0xff000000=20 <118> nd6 options=3D3 <118>em0: flags=3D8843 metric 0 = mtu 1500 <118> options=3D9b <118> ether 00:0c:29:11:4f:d5 <118> inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255 <118> media: Ethernet autoselect (1000baseT ) <118> status: active <118>em1: flags=3D8843 metric 0 = mtu 1500 <118> options=3D9b <118> ether 00:0c:29:11:4f:cb <118> inet 10.0.1.2 netmask 0xffffff00 broadcast 10.0.1.255 <118> media: Ethernet autoselect (1000baseT ) <118> status: active <118>em2: flags=3D8843 metric 0 = mtu 1500 <118> options=3D9b <118> ether 00:0c:29:11:4f:df <118> inet 10.0.2.2 netmask 0xffffff00 broadcast 10.0.2.255 <118> media: Ethernet autoselect (1000baseT ) <118> status: active <118>em3: flags=3D8843 metric 0 = mtu 1500 <118> options=3D9b <118> ether 00:0c:29:11:4f:e9 <118> inet 172.16.2.118 netmask 0xffff0000 broadcast 172.16.255.255 <118> media: Ethernet autoselect (1000baseT ) <118> status: active <118>add net default: gateway 172.16.0.2 Expensive timeout(9) function: 0xffffffff80329580(0xffffff8000272000) = 0.006700013 s <118>Starting devd. ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based = forwarding disabled, default to deny, logging disabled <118>Flushed all rules. <118>00100 allow ip from any to any via lo0 <118>00200 deny ip from any to 127.0.0.0/8 <118>00300 deny ip from 127.0.0.0/8 to any <118>65000 allow ip from any to any <118>Firewall rules loaded. <118>Creating and/or trimming log files <118>. <118>Starting syslogd. <118>No core dumps found. <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib <118>32-bit compatibility ldconfig path: /usr/lib32 lock order reversal: 1st 0xffffff800a2f62e8 bufwait (bufwait) @ = /pool/newsrc/src/sys/kern/vfs_bio.c:2559 2nd 0xffffff0002c71400 dirhash (dirhash) @ = /pool/newsrc/src/sys/ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 ufsdirhash_acquire() at ufsdirhash_acquire+0x44 ufsdirhash_add() at ufsdirhash_add+0x19 ufs_direnter() at ufs_direnter+0x88b ufs_mkdir() at ufs_mkdir+0x623 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xb9 kern_mkdirat() at kern_mkdirat+0x264 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (136, FreeBSD ELF64, mkdir), rip =3D 0x80072bb0c, rsp =3D = 0x7fffffffec88, rbp =3D 0x7fffffffef66 --- <118>Clearing /tmp (X related). <118>Updating motd: <118>. <118>Configuring syscons: <118> blanktime <118>. <118>Starting sshd. <118>Starting cron. <118>Starting background file system checks in 60 seconds. <118> <118>Wed Mar 10 17:29:09 CET 2010 <118>Mar 10 17:29:21 su: BAD SU borjam to root on /dev/pts/0 <118>Mar 10 17:29:24 su: BAD SU borjam to root on /dev/pts/0 <118>Mar 10 17:29:26 su: borjam to root on /dev/pts/0 <118>Mar 10 17:30:36 su: borjam to root on /dev/pts/1 lock order reversal: 1st 0xffffff00027e2438 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 2nd 0xffffff000286bb88 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:130 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dnode_verify() at dnode_verify+0x70 dnode_hold_impl() at dnode_hold_impl+0x73 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x3eb dbuf_sync_list() at dbuf_sync_list+0x17f Fatal double fault rip =3D 0xffffffff805d322d rsp =3D 0xffffff8012414fe0 rbp =3D 0xffffff8012415090 cpuid =3D 0; apic id =3D 00 panic: double fault cpuid =3D 0 KDB: enter: panic shared sx zfsvfs->z_teardown_inactive_lock = (zfsvfs->z_teardown_inactive_lock) r =3D 0 (0xffffff000256d210) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_vnops.c:3724 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff00055b3a58) locked @ = /pool/newsrc/src/sys/kern/vfs_subr.c:2091 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff0005618308) locked @ = /pool/newsrc/src/sys/kern/vfs_syscalls.c:3174 shared lockmgr zfs (zfs) r =3D 0 (0xffffff00057bda58) locked @ = /pool/newsrc/src/sys/kern/vfs_subr.c:2091 exclusive sx zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) r =3D 0 = (0xffffff000256d310) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:866 shared sx zp->z_name_lock (zp->z_name_lock) r =3D 0 (0xffffff00057c7c38) = locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_dir.c:212 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff00057ac578) locked @ = /pool/newsrc/src/sys/kern/vfs_subr.c:2091 exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xffffff0002f63648) locked @ = /pool/newsrc/src/sys/kern/uipc_sockbuf.c:148 exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xffffff0002cff3a0) locked @ = /pool/newsrc/src/sys/kern/uipc_sockbuf.c:148 exclusive sx ds->ds_deadlist.bpl_lock (ds->ds_deadlist.bpl_lock) r =3D 0 = (0xffffff00027e2438) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 = (0xffffff0002948238) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 = (0xffffff0002963538) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 = (0xffffff0002949438) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 = (0xffffff0002948b38) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 = (0xffffff0002966238) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 exclusive sx dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) r =3D 0 = (0xffffff0002963438) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 0xffffff00055b39c0: tag zfs, type VDIR usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOINGINACT) lock type zfs: EXCL by thread 0xffffff000528b3a0 (pid 1425) #0 0xffffffff80578cef at __lockmgr_args+0x4ef #1 0xffffffff80610a59 at vop_stdlock+0x39 #2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062d957 at _vn_lock+0x57 #4 0xffffffff806210bb at vget+0x7b #5 0xffffffff8060e570 at cache_lookup+0x4e0 #6 0xffffffff8060e890 at vfs_cache_lookup+0xc0 #7 0xffffffff808cbdb7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615513 at lookup+0x3d3 #9 0xffffffff806164e9 at namei+0x4a9 #10 0xffffffff806255cf at kern_statat_vnhook+0x8f #11 0xffffffff80625775 at kern_statat+0x15 #12 0xffffffff8062583a at lstat+0x2a #13 0xffffffff80876b48 at syscall+0x118 #14 0xffffffff8085cf91 at Xfast_syscall+0xe1 0xffffff0005618270: tag zfs, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type zfs: EXCL by thread 0xffffff000528c000 (pid 1422) #0 0xffffffff80578f77 at __lockmgr_args+0x777 #1 0xffffffff80610a59 at vop_stdlock+0x39 #2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062d957 at _vn_lock+0x57 #4 0xffffffff806299ba at setutimes+0x7a #5 0xffffffff80629cb2 at kern_lutimes+0xf2 #6 0xffffffff80876b48 at syscall+0x118 #7 0xffffffff8085cf91 at Xfast_syscall+0xe1 0xffffff00057bd9c0: tag zfs, type VDIR usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type zfs: SHARED (count 1) #0 0xffffffff80578cef at __lockmgr_args+0x4ef #1 0xffffffff80610a59 at vop_stdlock+0x39 #2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062d957 at _vn_lock+0x57 #4 0xffffffff806210bb at vget+0x7b #5 0xffffffff8060e570 at cache_lookup+0x4e0 #6 0xffffffff8060e890 at vfs_cache_lookup+0xc0 #7 0xffffffff808cbdb7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615513 at lookup+0x3d3 #9 0xffffffff806164e9 at namei+0x4a9 #10 0xffffffff806255cf at kern_statat_vnhook+0x8f #11 0xffffffff80625775 at kern_statat+0x15 #12 0xffffffff8062583a at lstat+0x2a #13 0xffffffff80876b48 at syscall+0x118 #14 0xffffffff8085cf91 at Xfast_syscall+0xe1 0xffffff00057ac4e0: tag zfs, type VDIR usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type zfs: EXCL by thread 0xffffff000262a3a0 (pid 1416) with shared waiters pending #0 0xffffffff80578cef at __lockmgr_args+0x4ef #1 0xffffffff80610a59 at vop_stdlock+0x39 #2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062d957 at _vn_lock+0x57 #4 0xffffffff806210bb at vget+0x7b #5 0xffffffff8060e570 at cache_lookup+0x4e0 #6 0xffffffff8060e890 at vfs_cache_lookup+0xc0 #7 0xffffffff808cbdb7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615513 at lookup+0x3d3 #9 0xffffffff806164e9 at namei+0x4a9 #10 0xffffffff80624df3 at kern_mkdirat+0xc3 #11 0xffffffff80876b48 at syscall+0x118 #12 0xffffffff8085cf91 at Xfast_syscall+0xe1 Uptime: 2m16s Textdump complete. Automatic reboot in 15 seconds - press a key on the console to abort --> Press a key on the console to reboot, --> or switch off the system now. Rebooting... Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights = reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-STABLE #3: Wed Mar 10 14:41:54 CET 2010 = root@pruebazfs1.sarenet.es:/pool/newsrc/obj/pool/newsrc/src/sys/DEBUG = amd64 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz (2116.63-MHz = K8-class CPU) Origin =3D "GenuineIntel" Id =3D 0x10676 Stepping =3D 6 = Features=3D0xfebfbff Features2=3D0x80082201> AMD Features=3D0x20100800 AMD Features2=3D0x1 TSC: P-state invariant real memory =3D 536870912 (512 MB) avail memory =3D 495988736 (473 MB) ACPI APIC Table: MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port = 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 7.3 (no driver attached) pci0: at device 7.7 (no driver attached) vgapci0: port 0x10d0-0x10df mem = 0xd0000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on = pci0 mpt0: port 0x1400-0x14ff mem = 0xd8820000-0xd883ffff,0xd8800000-0xd881ffff irq 17 at device 16.0 on = pci0 mpt0: [ITHREAD] mpt0: MPI Version=3D1.2.0.0 pcib2: at device 17.0 on pci0 pci2: on pcib2 em0: port 0x2000-0x203f = mem 0xd8940000-0xd895ffff,0xd8900000-0xd890ffff irq 18 at device 0.0 on = pci2 em0: Memory Access and/or Bus Master bits were not set! em0: [FILTER] em0: Ethernet address: 00:0c:29:11:4f:d5 em1: port 0x2040-0x207f = mem 0xd8960000-0xd897ffff,0xd8910000-0xd891ffff irq 19 at device 1.0 on = pci2 em1: Memory Access and/or Bus Master bits were not set! em1: [FILTER] em1: Ethernet address: 00:0c:29:11:4f:cb pci2: at device 2.0 (no driver attached) em2: port 0x20c0-0x20ff = mem 0xd8980000-0xd899ffff,0xd8920000-0xd892ffff irq 17 at device 3.0 on = pci2 em2: Memory Access and/or Bus Master bits were not set! em2: [FILTER] em2: Ethernet address: 00:0c:29:11:4f:df em3: port 0x2400-0x243f = mem 0xd89a0000-0xd89bffff,0xd8930000-0xd893ffff irq 19 at device 5.0 on = pci2 em3: Memory Access and/or Bus Master bits were not set! em3: [FILTER] em3: Ethernet address: 00:0c:29:11:4f:e9 pcib3: at device 21.0 on pci0 pci3: on pcib3 pcib4: at device 21.1 on pci0 pci4: on pcib4 pcib5: at device 21.2 on pci0 pci5: on pcib5 pcib6: at device 21.3 on pci0 pci6: on pcib6 pcib7: at device 21.4 on pci0 pci7: on pcib7 pcib8: at device 21.5 on pci0 pci8: on pcib8 pcib9: at device 21.6 on pci0 pci9: on pcib9 pcib10: at device 21.7 on pci0 pci10: on pcib10 pcib11: at device 22.0 on pci0 pci11: on pcib11 pcib12: at device 22.1 on pci0 pci12: on pcib12 pcib13: at device 22.2 on pci0 pci13: on pcib13 pcib14: at device 22.3 on pci0 pci14: on pcib14 pcib15: at device 22.4 on pci0 pci15: on pcib15 pcib16: at device 22.5 on pci0 pci16: on pcib16 pcib17: at device 22.6 on pci0 pci17: on pcib17 pcib18: at device 22.7 on pci0 pci18: on pcib18 pcib19: at device 23.0 on pci0 pci19: on pcib19 pcib20: at device 23.1 on pci0 pci20: on pcib20 pcib21: at device 23.2 on pci0 pci21: on pcib21 pcib22: at device 23.3 on pci0 pci22: on pcib22 pcib23: at device 23.4 on pci0 pci23: on pcib23 pcib24: at device 23.5 on pci0 pci24: on pcib24 pcib25: at device 23.6 on pci0 pci25: on pcib25 pcib26: at device 23.7 on pci0 pci26: on pcib26 pcib27: at device 24.0 on pci0 pci27: on pcib27 pcib28: at device 24.1 on pci0 pci28: on pcib28 pcib29: at device 24.2 on pci0 pci29: on pcib29 pcib30: at device 24.3 on pci0 pci30: on pcib30 pcib31: at device 24.4 on pci0 pci31: on pcib31 pcib32: at device 24.5 on pci0 pci32: on pcib32 pcib33: at device 24.6 on pci0 pci33: on pcib33 pcib34: at device 24.7 on pci0 pci34: on pcib34 acpi_acad0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on = acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 cpu0: on acpi0 acpi_throttle0: on cpu0 orm0: at iomem = 0xc0000-0xc7fff,0xca000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff,0xcd000-0x= cdfff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=3D0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on = isa0 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is = present; to enable, add "vfs.zfs.prefetch_disable=3D0" to = /boot/loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable = behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version 3 ZFS storage pool version 14 Timecounter "TSC" frequency 2116629050 Hz quality 800 Timecounters tick every 10.000 msec acd0: DVDR at ata1-master = UDMA33=20 Expensive timeout(9) function: 0xffffffff805ed840(0) 0.002413714 s Expensive timeout(9) function: 0xffffffff80885bf0(0xffffffff80e68900) = 0.003080000 s da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device=20 da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C) da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-2 device=20 da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da1: Command Queueing enabled da1: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) da2 at mpt0 bus 0 scbus0 target 2 lun 0 da2: Fixed Direct Access SCSI-2 device=20 da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da2: Command Queueing enabled da2: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) da3 at mpt0 bus 0 scbus0 target 3 lun 0 da3: Fixed Direct Access SCSI-2 device=20 da3: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da3: Command Queueing enabled da3: 12288MB (25165824 512 byte sectors: 255H 63S/T 1566C) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0s1a WARNING: / was not properly dismounted <118>Setting hostuuid: 564d8c47-d7e5-fea4-92ef-f17760cfe4d2. <118>Setting hostid: 0x4d345a2c. lock order reversal: 1st 0xffffff00028732e8 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:549 2nd 0xffffff00028720d8 dn->dn_mtx (dn->dn_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:1173 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_block_freed() at dnode_block_freed+0x8e dbuf_read() at dbuf_read+0x155 dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x12a dmu_read() at dmu_read+0x80 load_nvlist() at load_nvlist+0x85 spa_load() at spa_load+0x49a spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 Expensive timeout(9) function: 0xffffffff80892b50(0xffffff000244fe00) = 0.003086146 s kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffffd808, rbp =3D 0x801224140 --- lock order reversal: 1st 0xffffff0002868e70 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode_sync.c:381 2nd 0xffffff0002767940 osi->os_lock (osi->os_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:323 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_destroy() at dnode_destroy+0xa6 dnode_buf_pageout() at dnode_buf_pageout+0xb2 dbuf_evict_user() at dbuf_evict_user+0x55 dbuf_clear() at dbuf_clear+0x5e dnode_evict_dbufs() at dnode_evict_dbufs+0x98 dmu_objset_evict_dbufs() at dmu_objset_evict_dbufs+0x11c dmu_objset_evict() at dmu_objset_evict+0xbf dsl_pool_close() at dsl_pool_close+0x52 spa_unload() at spa_unload+0xb2 spa_load() at spa_load+0x4da spa_open_common() at spa_open_common+0x12d spa_get_stats() at spa_get_stats+0x42 zfs_ioc_pool_stats() at zfs_ioc_pool_stats+0x2c zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffffd808, rbp =3D 0x801224140 --- lock order reversal: 1st 0xffffff0002873058 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1116 2nd 0xffffff000259ac38 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1120 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_sExpensive timeout(9) function: = 0xffffffff805ed8c0(0) 0.003133917 s elf_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_dirty() at dbuf_dirty+0x892 dnode_setdirty() at dnode_setdirty+0x1a9 dbuf_dirty() at dbuf_dirty+0xa53 bplist_vacate() at bplist_vacate+0x4d spa_sync() at spa_sync+0x297 txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002926838 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 2nd 0xffffff000250c2f0 spa->spa_sync_bplist.bpl_lock = (spa->spa_sync_bplist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:235 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 bplist_enqueue_deferred() at bplist_enqueue_deferred+0x47 zio_free() at zio_free+0x105 arc_free() at arc_free+0x11c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x483 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x3eb dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync() at dmu_objset_sync+0x134 dsl_pool_sync() at dsl_pool_sync+0x200 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at Expensive timeout(9) function: = 0xffffffff80885bf0(0xffffffff80e68900) 0.003364952 s fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- <118>Entropy harvesting: <118> interrupts <118> ethernet <118> point_to_point <118> kickstart <118>. <118>Starting file system checks: <118>/dev/da0s1a: INCORRECT BLOCK COUNT I=3D259208 (16 should be 12) = (CORRECTED) <118>/dev/da0s1a: LINK COUNT DIR I=3D235521 OWNER=3Droot MODE=3D41777 <118>/dev/da0s1a: SIZE=3D3584 MTIME=3DMar 10 17:30 2010 COUNT 120 = SHOULD BE 119 (ADJUSTED) <118>/dev/da0s1a: SUMMARY INFORMATION BAD (SALVAGED) <118>/dev/da0s1a: BLK(S) MISSING IN BIT MAPS (SALVAGED) <118>/dev/da0s1a: 28967 files, 517344 used, 1004473 free (7889 frags, = 124573 blocks, 0.5% fragmentation) Expensive timeout(9) function: 0xffffffff80475420(0xffffffff80e77f60) = 0.004632152 s <118>Mounting local file systems: <118>. lock order reversal: 1st 0xffffff0002620cc8 zfs (zfs) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/gfs.c:437 2nd 0xffffff000256f310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23c zfs_root() at zfs_root+0x50 zfsctl_create() at zfsctl_create+0x82 zfs_mount() at zfs_mount+0x7ef vfs_donmount() at vfs_donmount+0xcde nmount() at nmount+0x63 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x800f48f1c, rsp =3D = 0x7fffffffced8, rbp =3D 0x7fffffffcef8 --- lock order reversal: 1st 0xffffff0002655078 zp->z_name_lock (zp->z_name_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_dir.c:212 2nd 0xffffff000256f330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23c zfs_dirent_lock() at zfs_dirent_lock+0x4a0 zfs_dirlook() at zfs_dirlook+0x90 zfs_lookup() at zfs_lookup+0x256 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800fd8acc, rsp =3D = 0x7fffffffcf38, rbp =3D 0x7fffffffd3d0 --- lock order reversal: 1st 0xffffff000256f210 zfsvfs->z_teardown_inactive_lock = (zfsvfs->z_teardown_inactive_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_vnops.c:3724 2nd 0xffffff000256f330 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:1027 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zinactive() at zfs_zinactive+0x95 zfs_inactive() at zfs_inactive+0x7e zfs_freebsd_inactive() at zfs_freebsd_inactive+0x1a VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5 vinactive() at vinactive+0x90 vputx() at vputx+0x2fc kern_statat_vnhook() at kern_statat_vnhook+0xfa kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800fd8acc, rsp =3D = 0x7fffffffcf38, rbp =3D 0x7fffffffd3d0 --- <118>/etc/rc: WARNING: $hostname is not set -- see rc.conf(5). <118>Starting Network: lo0 em0 em1 em2 em3. <118>lo0: flags=3D8049 metric 0 mtu 16384 <118> options=3D3 <118> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6=20 <118> inet6 ::1 prefixlen 128=20 <118> inet 127.0.0.1 netmask 0xff000000=20 <118> nd6 options=3D3 <118>em0: flags=3D8843 metric 0 = mtu 1500 <118> options=3D9b <118> ether 00:0c:29:11:4f:d5 <118> inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255 <118> media: Ethernet autoselect (1000baseT ) <118> status: active <118>em1: flags=3D8843 metric 0 = mtu 1500 <118> options=3D9b <118> ether 00:0c:29:11:4f:cb <118> inet 10.0.1.2 netmask 0xffffff00 broadcast 10.0.1.255 <118> media: Ethernet autoselect (1000baseT ) <118> status: active <118>em2: flags=3D8843 metric 0 = mtu 1500 <118> options=3D9b <118> ether 00:0c:29:11:4f:df <118> inet 10.0.2.2 netmask 0xffffff00 broadcast 10.0.2.255 <118> media: Ethernet autoselect (1000baseT ) <118> status: active <118>em3: flags=3D8843 metric 0 = mtu 1500 <118> options=3D9b <118> ether 00:0c:29:11:4f:e9 <118> inet 172.16.2.118 netmask 0xffff0000 broadcast 172.16.255.255 <118> media: Ethernet autoselect (1000baseT ) <118> status: active <118>add net default: gateway 172.16.0.2 <118>Starting devd. Expensive timeout(9) function: 0xffffffff80329580(0xffffff8000284000) = 0.010275607 s ipfw2 (+ipv6) initialized, divert loadable, nat loadable, rule-based = forwarding disabled, default to deny, logging disabled <118>Flushed all rules. <118>00100 allow ip from any to any via lo0 <118>00200 deny ip from any to 127.0.0.0/8 <118>00300 deny ip from 127.0.0.0/8 to any <118>65000 allow ip from any to any <118>Firewall rules loaded. <118>Creating and/or trimming log files <118>. <118>Starting syslogd. <118>savecore: reboot after panic: double fault <118>Mar 10 17:33:17 savecore: reboot after panic: double fault <118>savecore: writing core to textdump.tar.1 <118>/var/crash/vmcore.1 not found <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib <118>32-bit compatibility ldconfig path: /usr/lib32 lock order reversal: 1st 0xffffff800a2f8338 bufwait (bufwait) @ = /pool/newsrc/src/sys/kern/vfs_bio.c:2559 2nd 0xffffff0002ca5c00 dirhash (dirhash) @ = /pool/newsrc/src/sys/ufs/ufs/ufs_dirhash.c:285 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 ufsdirhash_acquire() at ufsdirhash_acquire+0x44 ufsdirhash_add() at ufsdirhash_add+0x19 ufs_direnter() at ufs_direnter+0x88b ufs_mkdir() at ufs_mkdir+0x623 VOP_MKDIR_APV() at VOP_MKDIR_APV+0xb9 kern_mkdirat() at kern_mkdirat+0x264 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (136, FreeBSD ELF64, mkdir), rip =3D 0x80072bb0c, rsp =3D = 0x7fffffffec88, rbp =3D 0x7fffffffef66 --- <118>Clearing /tmp (X related). <118>Updating motd: <118>. <118>Configuring syscons: <118> blanktime <118>. <118>Starting sshd. lock order reversal: 1st 0xffffff0002864b08 buf->b_lock (buf->b_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/arc.c:2509 2nd 0xffffff00025e9430 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:421 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_read_done() at dbuf_read_done+0x3b arc_read_nolock() at arc_read_nolock+0x526 arc_read() at arc_read+0xaf dbuf_read() at dbuf_read+0x62b dmu_buf_hold() at dmu_buf_hold+0xcc bplist_cache() at bplist_cache+0x4c bplist_iterate() at bplist_iterate+0xb3 spa_sync() at spa_sync+0x262 txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- <118>Starting cron. <118>Starting background file system checks in 60 seconds. <118> <118>Wed Mar 10 17:33:21 CET 2010 <118>Mar 10 17:33:30 su: borjam to root on /dev/pts/0 lock order reversal: 1st 0xffffff0002c6d238 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1905 2nd 0xffffff0002866000 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:543 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dbuf_read() at dbuf_read+0x2ad dbuf_will_dirty() at dbuf_will_dirty+0x53 dsl_dataset_block_kill() at dsl_dataset_block_kill+0xe9 dbuf_write() at dbuf_write+0x24c dbuf_sync_list() at dbuf_sync_list+0x159 dbuf_sync_list() at dbuf_sync_list+0x17f dnode_sync() at dnode_sync+0xc12 dmu_objset_sync() at dmu_objset_sync+0x134 dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff000248d038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 2nd 0xffffff0002871b88 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:130 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_slock() at _sx_slock+0x55 dnode_verify() at dnode_verify+0x70 dnode_hold_impl() at dnode_hold_impl+0x73 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff000248d038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 2nd 0xffffff0002f340d8 dn->dn_mtx (dn->dn_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:606 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_hold_impl() at dnode_hold_impl+0x184 dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff000248d038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 2nd 0xffffff00028686c0 db->db_mtx (db->db_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1724 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_rele() at dbuf_rele+0x2d dnode_hold_impl() at dnode_hold_impl+0x20f dmu_bonus_hold() at dmu_bonus_hold+0x31 bplist_hold() at bplist_hold+0x48 bplist_enqueue() at bplist_enqueue+0x4c dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff000248d038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:189 2nd 0xffffff00027d9540 osi->os_lock (osi->os_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dnode.c:687 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dnode_setdirty() at dnode_setdirty+0xbc dbuf_dirty() at dbuf_dirty+0x516 bplist_enqueue() at bplist_enqueue+0xbd dsl_dataset_block_kill() at dsl_dataset_block_kill+0x119 dmu_objset_sync() at dmu_objset_sync+0x1fe dsl_pool_sync() at dsl_pool_sync+0x88 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002893210 zfsvfs->z_teardown_inactive_lock = (zfsvfs->z_teardown_inactive_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_vfsops.c:917 2nd 0xffffff000248d0f8 ds->ds_rwlock (ds->ds_rwlock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dsl_dataset.c:2864 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dsl_dataset_clone_swap() at dsl_dataset_clone_swap+0x5a dmu_recv_end() at dmu_recv_end+0x94 zfs_ioc_recv() at zfs_ioc_recv+0x29d zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffff8e98, rbp =3D 0x7fffffff9bd0 --- lock order reversal: 1st 0xffffff000248d038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:152 2nd 0xffffff0002f34330 dn->dn_dbufs_mtx (dn->dn_dbufs_mtx) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:1518 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_destroy() at dbuf_destroy+0x58 bplist_cache() at bplist_cache+0x2e bplist_iterate() at bplist_iterate+0xb3 bplist_space_birthrange() at bplist_space_birthrange+0x60 dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 dsl_pool_sync() at dsl_pool_sync+0x122 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff000248d038 ds->ds_deadlist.bpl_lock = (ds->ds_deadlist.bpl_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/bplist.c:152 2nd 0xffffffff81150db0 h->hash_mutexes[i] (h->hash_mutexes[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dbuf.c:191 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 dbuf_destroy() at dbuf_destroy+0x111 bplist_cache() at bplist_cache+0x2e bplist_iterate() at bplist_iterate+0xb3 bplist_space_birthrange() at bplist_space_birthrange+0x60 dsl_dataset_clone_swap_sync() at dsl_dataset_clone_swap_sync+0xee dsl_sync_task_group_sync() at dsl_sync_task_group_sync+0x173 dsl_pool_sync() at dsl_pool_sync+0x122 spa_sync() at spa_sync+0x35e txg_sync_thread() at txg_sync_thread+0x2d7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8012417d30, rbp =3D 0 --- lock order reversal: 1st 0xffffff0002893250 zfsvfs->z_znodes_lock (zfsvfs->z_znodes_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_vfsops.c:1317 2nd 0xffffff0002893310 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:966 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_rezget() at zfs_rezget+0x4a zfs_resume_fs() at zfs_resume_fs+0x158 zfs_ioc_recv() at zfs_ioc_recv+0x2b4 zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe7d1c, rsp =3D = 0x7fffffff8e98, rbp =3D 0x7fffffff9bd0 --- lock order reversal: 1st 0xffffff0004e911d0 zp->z_parent_lock (zp->z_parent_lock) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_dir.c:379 2nd 0xffffff0002893350 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:866 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e _sx_xlock() at _sx_xlock+0x55 zfs_zget() at zfs_zget+0x23c zfs_dirlook() at zfs_dirlook+0x1fc zfs_lookup() at zfs_lookup+0x256 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x8d VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x4a9 kern_chdir() at kern_chdir+0x78 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (12, FreeBSD ELF64, chdir), rip =3D 0x800da74dc, rsp =3D = 0x7fffffffe898, rbp =3D 0x801046f10 --- KDB: enter: sysctl debug.kdb.enter exclusive sleep mutex Giant (Giant) r =3D 0 (0xffffffff80c4c540) locked = @ /pool/newsrc/src/sys/kern/kern_sysctl.c:1417 shared sx sysctl lock (sysctl lock) r =3D 0 (0xffffffff80c4cec0) locked = @ /pool/newsrc/src/sys/kern/kern_sysctl.c:1521 exclusive sx zfsvfs->z_teardown_inactive_lock = (zfsvfs->z_teardown_inactive_lock) r =3D 0 (0xffffff0002893210) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_vfsops.c:917 shared sx ds->ds_rwlock (ds->ds_rwlock) r =3D 0 (0xffffff00190f50f8) = locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/dsl_dataset.c:519 exclusive sx zfsvfs->z_online_recv_lock (zfsvfs->z_online_recv_lock) r =3D= 0 (0xffffff0002893288) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_ioctl.c:2492 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff0007d43308) locked @ = /pool/newsrc/src/sys/kern/vfs_subr.c:888 exclusive sx zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) r =3D 0 = (0xffffff00028963f0) locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_znode.c:768 shared sx zp->z_name_lock (zp->z_name_lock) r =3D 0 (0xffffff0007cf31f0) = locked @ = /pool/newsrc/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common= /fs/zfs/zfs_dir.c:212 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff0007d11098) locked @ = /pool/newsrc/src/sys/kern/vfs_subr.c:2091 exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xffffff0002c763a0) locked @ = /pool/newsrc/src/sys/kern/uipc_sockbuf.c:148 exclusive lockmgr syncer (syncer) r =3D 0 (0xffffff0002947cc8) locked @ = /pool/newsrc/src/sys/kern/vfs_subr.c:1701 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff0007d61cc8) locked @ = /pool/newsrc/src/sys/kern/vfs_subr.c:888 0xffffff0007d11000: tag zfs, type VDIR usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type zfs: EXCL by thread 0xffffff000262dae0 (pid 1699) #0 0xffffffff80578cef at __lockmgr_args+0x4ef #1 0xffffffff80610a59 at vop_stdlock+0x39 #2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062d957 at _vn_lock+0x57 #4 0xffffffff806210bb at vget+0x7b #5 0xffffffff8060e570 at cache_lookup+0x4e0 #6 0xffffffff8060e890 at vfs_cache_lookup+0xc0 #7 0xffffffff808cbdb7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615513 at lookup+0x3d3 #9 0xffffffff806164e9 at namei+0x4a9 #10 0xffffffff8062d1a2 at vn_open_cred+0xc2 #11 0xffffffff8062b569 at kern_openat+0x179 #12 0xffffffff80876b48 at syscall+0x118 #13 0xffffffff8085cf91 at Xfast_syscall+0xe1 0xffffff0002947c30: tag syncer, type VNON usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type syncer: EXCL by thread 0xffffff000250d3a0 (pid 18) #0 0xffffffff80578f77 at __lockmgr_args+0x777 #1 0xffffffff80610a59 at vop_stdlock+0x39 #2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062d957 at _vn_lock+0x57 #4 0xffffffff806229a0 at sync_vnode+0x130 #5 0xffffffff80622c2c at sched_sync+0x1cc #6 0xffffffff80566aca at fork_exit+0x12a #7 0xffffffff8085d18e at fork_trampoline+0xe 0xffffff0007d43270: tag zfs, type VREG usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED) v_object 0xffffff0007d2e870 ref 0 pages 0 lock type zfs: EXCL by thread 0xffffff000262dae0 (pid 1699) #0 0xffffffff80578f77 at __lockmgr_args+0x777 #1 0xffffffff80610a59 at vop_stdlock+0x39 #2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b #3 0xffffffff806231db at vnlru_free+0x20b #4 0xffffffff806235df at getnewvnode+0x24f #5 0xffffffff810a7533 at zfs_znode_cache_constructor+0x43 #6 0xffffffff810a9248 at zfs_znode_alloc+0x38 #7 0xffffffff810a96d9 at zfs_mknode+0x259 #8 0xffffffff810cda21 at zfs_freebsd_create+0x661 #9 0xffffffff808cbc43 at VOP_CREATE_APV+0xb3 #10 0xffffffff8062d553 at vn_open_cred+0x473 #11 0xffffffff8062b569 at kern_openat+0x179 #12 0xffffffff80876b48 at syscall+0x118 #13 0xffffffff8085cf91 at Xfast_syscall+0xe1 0xffffff0007d61c30: tag zfs, type VREG usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED) v_object 0xffffff0007d2e6c0 ref 0 pages 0 lock type zfs: EXCL by thread 0xffffff000250d740 (pid 17) #0 0xffffffff80578f77 at __lockmgr_args+0x777 #1 0xffffffff80610a59 at vop_stdlock+0x39 #2 0xffffffff808caedb at VOP_LOCK1_APV+0x9b #3 0xffffffff806231db at vnlru_free+0x20b #4 0xffffffff80623bce at vnlru_proc+0x58e #5 0xffffffff80566aca at fork_exit+0x12a #6 0xffffffff8085d18e at fork_trampoline+0xe panic: from debugger cpuid =3D 0 Uptime: 33m28s #=20 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 10 15:15:03 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 0CD771065670 for ; Wed, 10 Mar 2010 15:15:03 +0000 (UTC) (envelope-from md@scoutsengidsenvlaanderen.be) Received: from scoutsengidsenvlaanderen.be (mail.scoutsengidsenvlaanderen.be [81.83.17.184]) by mx1.freebsd.org (Postfix) with ESMTP id 470AC8FC16 for ; Wed, 10 Mar 2010 15:15:01 +0000 (UTC) Received: from [192.168.1.163] (account md [192.168.1.163] verified) by scoutsengidsenvlaanderen.be (CommuniGate Pro SMTP 5.3.3) with ESMTPA id 34070739 for freebsd-fs@freebsd.org; Wed, 10 Mar 2010 15:14:59 +0100 Message-ID: <4B97A963.9040300@scoutsengidsenvlaanderen.be> Date: Wed, 10 Mar 2010 15:14:59 +0100 From: Michiel Detailleur User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) MIME-Version: 1.0 To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: ZFS snapshot name length limit? (File name too long) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 15:15:03 -0000 Hi, I'm using sysutils/zfs-snapshot-mgmt for automatic creation and purging of ZFS snapshots. It creates snapshot names like this (zfsfilesys is not the actual name of the zfs filesystem, it's actually a bit longer. Why is this important? See below.): zfsfilesys@auto-2010-03-10_06.00 Clean and simple naming scheme you would think, but when I ls -la zfsfilesys/.zfs/snapshot, I get output from ls: ls: auto-2010-03-10_06.00: File name too long This happens for all snapshots with names longer than 16 characters for this particular zfs file system. Snapshots shorter than 16 characters are listed. drwxr-xr-x 12 root wheel 23 Dec 24 14:06 2010-02-02-00/ The name of the actual file system is 56 characters, starting from 'tank/' For file systems with shorters names, longer snapshot names *are* accessible. This seems to imply a maximum length of 72 characters (snapshot '@' sign excluded) for *usable* snapshots? (usable in the sense that we can mount/look into them) What limit am I (we?) hitting here? Is this a problem with ZFS itself or with FreeBSD? Sure looks like a silly limit for the filesystem that touts 'biggest everything'. I'm using FreeBSD 8.0, the zpool was freshly created on the 8.0 OS (not upgraded from older FreeBSD ZFS version). Thanks for any insight on this problem. -- Michiel Detailleur Medewerker Informatica Scouts en Gidsen Vlaanderen vzw Lange Kievitstraat 74 2018 Antwerpen tel 03 231 16 20 fax 03 232 63 92 www.scoutsengidsenvlaanderen.be From owner-freebsd-fs@FreeBSD.ORG Wed Mar 10 16:04:43 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9B504106566C; Wed, 10 Mar 2010 16:04:43 +0000 (UTC) (envelope-from jh@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 724908FC13; Wed, 10 Mar 2010 16:04:43 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2AG4h5H081813; Wed, 10 Mar 2010 16:04:43 GMT (envelope-from jh@freefall.freebsd.org) Received: (from jh@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2AG4hdF081808; Wed, 10 Mar 2010 16:04:43 GMT (envelope-from jh) Date: Wed, 10 Mar 2010 16:04:43 GMT Message-Id: <201003101604.o2AG4hdF081808@freefall.freebsd.org> To: jgarzik@pobox.com, jh@FreeBSD.org, freebsd-fs@FreeBSD.org, jh@FreeBSD.org From: jh@FreeBSD.org Cc: Subject: Re: kern/131995: [nfs] Failure to mount NFSv4 server X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 16:04:43 -0000 Synopsis: [nfs] Failure to mount NFSv4 server State-Changed-From-To: open->feedback State-Changed-By: jh State-Changed-When: Wed Mar 10 16:03:11 UTC 2010 State-Changed-Why: University of Michigan NFSv4 client has been removed from head and stable/8. A new, experimental implementation with support for NFSv4 is available in head and stable/8. Can you still reproduce the problems with the new implementation? The file system type for mount(8) command is newnfs. Responsible-Changed-From-To: freebsd-fs->jh Responsible-Changed-By: jh Responsible-Changed-When: Wed Mar 10 16:03:11 UTC 2010 Responsible-Changed-Why: Track. http://www.freebsd.org/cgi/query-pr.cgi?pr=131995 From owner-freebsd-fs@FreeBSD.ORG Wed Mar 10 17:31:53 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7FB8E1065674; Wed, 10 Mar 2010 17:31:53 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id B3B198FC27; Wed, 10 Mar 2010 17:31:51 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 614F045CDD; Wed, 10 Mar 2010 18:31:49 +0100 (CET) Received: from localhost (pdawidek.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 7FAEE45C9B; Wed, 10 Mar 2010 18:31:43 +0100 (CET) Date: Wed, 10 Mar 2010 18:31:43 +0100 From: Pawel Jakub Dawidek To: Borja Marcos Message-ID: <20100310173143.GD1715@garage.freebsd.pl> References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BI5RvnYi6R4T2M87" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org, FreeBSD Stable Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 17:31:53 -0000 --BI5RvnYi6R4T2M87 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Mar 10, 2010 at 04:12:36PM +0100, Borja Marcos wrote: > =09 > On Mar 10, 2010, at 12:02 PM, Pawel Jakub Dawidek wrote: >=20 > > Once the deadlock occur, enter DDB and send me the output of: > >=20 > > ps > > show alllocks > > show lockedvnods > > show allchains > > alltrace >=20 > (Again, crossposted to -fs, ZFS related) >=20 >=20 > Previous one was a panic when performing the test with several tar jobs r= unning in parallel. >=20 > Now this is a capture of the deadlock itself, instead of a panic. (I call= ed panic from the debugger to generate a dump) [...] Hmm, interesting. Especially those two traces: Tracing command zfs pid 1820 tid 100105 td 0xffffff0002ca4000 [...] _cv_wait() at _cv_wait+0x17a txg_wait_synced() at txg_wait_synced+0x98 zfsvfs_teardown() at zfsvfs_teardown+0x1f6 zfs_suspend_fs() at zfs_suspend_fs+0x2b zfs_ioc_recv() at zfs_ioc_recv+0x28b zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd [...] Tracing command bsdtar pid 1699 tid 100093 td 0xffffff000262dae0 [...] _sx_slock_hard() at _sx_slock_hard+0x1b7 _sx_slock() at _sx_slock+0xc1=20 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x63 VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 getnewvnode() at getnewvnode+0x24f zfs_znode_cache_constructor() at zfs_znode_cache_constructor+0x43 zfs_znode_alloc() at zfs_znode_alloc+0x38 zfs_mknode() at zfs_mknode+0x259 zfs_freebsd_create() at zfs_freebsd_create+0x661 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x473 kern_openat() at kern_openat+0x179 [...] This should be impossible. If we are that deep in zfsvfs_teardown(), it mea= ns that we hold the z_teardown_lock exclusively. And we do as 'show alllocks' output confirms. But if we are holding this lock exclusively we shouldn't be that deep in create code path, because we need hold this lock as reader. It isn't visible in 'show alllocks' output, because this lock is special (rrwlock.c). I see three possibilities: 1. We are looking at different file systems here. But where is deadlock coming from then? 2. There is a bug in rrwlock.c. Highly unlikely I think. 3. My thinking is incorrect somewhere. Let me do some more thinking and I'll get back to you (possibly with a patch that will help us to find right possibility). --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --BI5RvnYi6R4T2M87 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkuX134ACgkQForvXbEpPzRsuACgzsjOtg3CjoVm65QoYNmS6GKg LasAoN0poZ4eavwo2Pl/LCiRUCGb67Vm =LFmy -----END PGP SIGNATURE----- --BI5RvnYi6R4T2M87-- From owner-freebsd-fs@FreeBSD.ORG Wed Mar 10 17:42:47 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id D6E781065670; Wed, 10 Mar 2010 17:42:47 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id ACC468FC20; Wed, 10 Mar 2010 17:42:46 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id TAA18018; Wed, 10 Mar 2010 19:42:44 +0200 (EET) (envelope-from avg@icyb.net.ua) Message-ID: <4B97DA13.1040900@icyb.net.ua> Date: Wed, 10 Mar 2010 19:42:43 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.23 (X11/20100211) MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> <20100310173143.GD1715@garage.freebsd.pl> In-Reply-To: <20100310173143.GD1715@garage.freebsd.pl> X-Enigmail-Version: 0.95.7 Content-Type: text/plain; charset=KOI8-U Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, FreeBSD Stable Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 17:42:47 -0000 on 10/03/2010 19:31 Pawel Jakub Dawidek said the following: > This should be impossible. If we are that deep in zfsvfs_teardown(), it means > that we hold the z_teardown_lock exclusively. And we do as 'show alllocks' > output confirms. But if we are holding this lock exclusively we shouldn't be > that deep in create code path, because we need hold this lock as reader. > It isn't visible in 'show alllocks' output, because this lock is special > (rrwlock.c). BTW, it seems that our 'stock' rwlock implements exactly the same thing as rrwlock.c - recursive readers, etc. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Wed Mar 10 17:44:18 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3A68910656E0 for ; Wed, 10 Mar 2010 17:44:18 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 9ED8A8FC1F for ; Wed, 10 Mar 2010 17:44:17 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id C0BBC45EE5; Wed, 10 Mar 2010 18:44:15 +0100 (CET) Received: from localhost (pdawidek.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 9454D45E8E; Wed, 10 Mar 2010 18:44:11 +0100 (CET) Date: Wed, 10 Mar 2010 18:44:11 +0100 From: Pawel Jakub Dawidek To: Michiel Detailleur Message-ID: <20100310174411.GE1715@garage.freebsd.pl> References: <4B97A963.9040300@scoutsengidsenvlaanderen.be> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Qz2CZ664xQdCRdPu" Content-Disposition: inline In-Reply-To: <4B97A963.9040300@scoutsengidsenvlaanderen.be> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: ZFS snapshot name length limit? (File name too long) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 17:44:18 -0000 --Qz2CZ664xQdCRdPu Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Mar 10, 2010 at 03:14:59PM +0100, Michiel Detailleur wrote: > Hi, >=20 > I'm using sysutils/zfs-snapshot-mgmt for automatic creation and purging= =20 > of ZFS snapshots. It creates snapshot names like this (zfsfilesys is not= =20 > the actual name of the zfs filesystem, it's actually a bit longer. Why=20 > is this important? See below.): >=20 > zfsfilesys@auto-2010-03-10_06.00 >=20 > Clean and simple naming scheme you would think, but when I ls -la=20 > zfsfilesys/.zfs/snapshot, I get output from ls: >=20 > ls: auto-2010-03-10_06.00: File name too long >=20 > This happens for all snapshots with names longer than 16 characters for= =20 > this particular zfs file system. >=20 > Snapshots shorter than 16 characters are listed. >=20 > drwxr-xr-x 12 root wheel 23 Dec 24 14:06 2010-02-02-00/ >=20 > The name of the actual file system is 56 characters, starting from 'tank/' > For file systems with shorters names, longer snapshot names *are*=20 > accessible. >=20 > This seems to imply a maximum length of 72 characters (snapshot '@' sign= =20 > excluded) for *usable* snapshots? (usable in the sense that we can=20 > mount/look into them) >=20 > What limit am I (we?) hitting here? Is this a problem with ZFS itself or= =20 > with FreeBSD? Sure looks like a silly limit for the filesystem that=20 > touts 'biggest everything'. >=20 > I'm using FreeBSD 8.0, the zpool was freshly created on the 8.0 OS (not= =20 > upgraded from older FreeBSD ZFS version). >=20 > Thanks for any insight on this problem. =46rom looking at the code, I think you hitting this limit: /* * Be ultra-paranoid about making sure the type and fspath * variables will fit in our mp buffers, including the * terminating NUL. */ if (strlen(fstype) >=3D MFSNAMELEN || strlen(fspath) >=3D MNAMELEN) return (ENAMETOOLONG); in vfs_domount() or vfs_donmount(). This is FreeBSD limit caused by statfs structure: /* * filesystem statistics */ [...] #define MNAMELEN 88 /* size of on/from name bufs */ [...] struct statfs { [...] char f_mntfromname[MNAMELEN];/* mounted filesystem */ char f_mntonname[MNAMELEN]; /* directory on which mounted */ }; When you list .zfs/snapshot/ directory (especially with -l option) ZFS mounts snapshots on lookup and this is this mount that fails. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --Qz2CZ664xQdCRdPu Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkuX2msACgkQForvXbEpPzRZLwCgx7sMATavSxG6XkQz81tlAi/b JekAoLYNXmhmqQEq0TUTwu+cbw7hiMO4 =YbzN -----END PGP SIGNATURE----- --Qz2CZ664xQdCRdPu-- From owner-freebsd-fs@FreeBSD.ORG Wed Mar 10 17:46:19 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C62F910656A7 for ; Wed, 10 Mar 2010 17:46:19 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 2F42F8FC21 for ; Wed, 10 Mar 2010 17:46:19 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 6B7D445EB2; Wed, 10 Mar 2010 18:46:16 +0100 (CET) Received: from localhost (pdawidek.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id CC59745E8E; Wed, 10 Mar 2010 18:46:11 +0100 (CET) Date: Wed, 10 Mar 2010 18:46:12 +0100 From: Pawel Jakub Dawidek To: Kevin Day Message-ID: <20100310174612.GF1715@garage.freebsd.pl> References: <7418ECC2-55C1-4A28-82EA-0972AFE745EF@dragondata.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="0rSojgWGcpz+ezC3" Content-Disposition: inline In-Reply-To: <7418ECC2-55C1-4A28-82EA-0972AFE745EF@dragondata.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: iscsi over HAST backed storage partial success X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 17:46:20 -0000 --0rSojgWGcpz+ezC3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Mar 09, 2010 at 05:03:41PM -0600, Kevin Day wrote: >=20 > I'm running istgt (iscsi target) using HAST backed storage. For the most = part, it seems to work really well. I have ucarp running to change the IP t= hat istgt is bound to, and modified the ucarp scripts to start/stop istgt d= epending on which side is the master. If I shut down the primary, the secon= dary takes over and all seems well. >=20 > However, if I reboot the secondary, the primary starts freezing up for lo= ng periods: >=20 > Mar 9 22:46:27 cs04 hastd: [iscsi1] (primary) Unable to r: Socket is not= connected. > Mar 9 22:46:27 cs04 hastd: [iscsi1] (primary) Unable to co: Connection r= efused. > Mar 9 22:46:42 cs04 last message repeated 3 times > Mar 9 22:46:53 cs04 istgt[14298]: ABORT_TASK > Mar 9 22:47:35 cs04 last message repeated 3 times > Mar 9 22:48:02 cs04 hastd: [iscsi1] (primary) Unable to co: Operation ti= med out. > Mar 9 22:48:02 cs04 istgt[14298]: CmdSN(45748), OP=3D0x2a, ElapsedTime= =3D74 cleared=20 > Mar 9 22:48:02 cs04 istgt[14298]: istgt_iscsi.c: 640:istgt_iscsi_write_p= du: ***ERROR*** iscsi_write() failed (errno=3D32) > Mar 9 22:48:02 cs04 istgt[14298]: istgt_iscsi.c:3327:istgt_iscsi_op_task= : ***ERROR*** iscsi_write_pdu() failed > Mar 9 22:48:02 cs04 istgt[14298]: istgt_iscsi.c:3867:istgt_iscsi_execute= : ***ERROR*** iscsi_op_task() failed =20 > Mar 9 22:48:02 cs04 istgt[14298]: istgt_iscsi.c:4337:worker: ***ERROR***= iscsi_execute() failed > Mar 9 22:48:02 cs04 istgt[14298]: CmdSN(490802), OP=3D0x2a, ElapsedTime= =3D73 cleared > Mar 9 22:48:02 cs04 istgt[14298]: CmdSN(28387), OP=3D0x2a, ElapsedTime= =3D73 cleared=20 > Mar 9 22:48:14 cs04 istgt[14298]: ABORT_TASK > Mar 9 22:48:52 cs04 last message repeated 2 times > Mar 9 22:49:22 cs04 hastd: [iscsi1] (primary) Unable to co: Operation ti= med out. >=20 > As soon as the secondary comes back online, everything starts behaving ag= ain and all is well. >=20 > Is this expected behavior at this point, or should hastd not block like t= his? It shouldn't of course block like this. There is a separate thread responsible for reconnecting which shouldn't interact with I/O threads. I'll try to reproduce and will let you know. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --0rSojgWGcpz+ezC3 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkuX2uMACgkQForvXbEpPzT7ewCgu70tq0+kx0OyRe4PQAaR1HV3 KAsAn2mW+sMKaLvKpvfJWILmNUiQfYX6 =ONha -----END PGP SIGNATURE----- --0rSojgWGcpz+ezC3-- From owner-freebsd-fs@FreeBSD.ORG Wed Mar 10 18:04:09 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3C48C106564A; Wed, 10 Mar 2010 18:04:09 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 6E4938FC15; Wed, 10 Mar 2010 18:04:07 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 8D74C45CD8; Wed, 10 Mar 2010 19:04:06 +0100 (CET) Received: from localhost (pdawidek.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id A994C45683; Wed, 10 Mar 2010 19:04:01 +0100 (CET) Date: Wed, 10 Mar 2010 19:04:01 +0100 From: Pawel Jakub Dawidek To: Andriy Gapon Message-ID: <20100310180401.GG1715@garage.freebsd.pl> References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> <20100310173143.GD1715@garage.freebsd.pl> <4B97DA13.1040900@icyb.net.ua> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="UTZ8bGhNySVQ9LYl" Content-Disposition: inline In-Reply-To: <4B97DA13.1040900@icyb.net.ua> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@FreeBSD.org, FreeBSD Stable Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 18:04:09 -0000 --UTZ8bGhNySVQ9LYl Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Mar 10, 2010 at 07:42:43PM +0200, Andriy Gapon wrote: > on 10/03/2010 19:31 Pawel Jakub Dawidek said the following: > > This should be impossible. If we are that deep in zfsvfs_teardown(), it= means > > that we hold the z_teardown_lock exclusively. And we do as 'show allloc= ks' > > output confirms. But if we are holding this lock exclusively we shouldn= 't be > > that deep in create code path, because we need hold this lock as reader. > > It isn't visible in 'show alllocks' output, because this lock is special > > (rrwlock.c). >=20 > BTW, it seems that our 'stock' rwlock implements exactly the same thing as > rrwlock.c - recursive readers, etc. But you cannot sleep while holding our rwlock(9). --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --UTZ8bGhNySVQ9LYl Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkuX3xAACgkQForvXbEpPzTK3wCghF57qjGHuo2cJWJ8ySWpSNr3 bXQAnj0etk/l9wDe7BFzglE0vs3igsOw =pLu5 -----END PGP SIGNATURE----- --UTZ8bGhNySVQ9LYl-- From owner-freebsd-fs@FreeBSD.ORG Wed Mar 10 18:17:10 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F0547106564A for ; Wed, 10 Mar 2010 18:17:10 +0000 (UTC) (envelope-from johan@stromnet.se) Received: from core.stromnet.se (core.stromnet.se [83.218.84.131]) by mx1.freebsd.org (Postfix) with ESMTP id E93748FC19 for ; Wed, 10 Mar 2010 18:17:09 +0000 (UTC) Received: from localhost (core.stromnet.se [83.218.84.131]) by core.stromnet.se (Postfix) with ESMTP id 8E48EF57464; Wed, 10 Mar 2010 19:17:07 +0100 (CET) X-Virus-Scanned: amavisd-new at stromnet.se X-Spam-Flag: NO X-Spam-Score: 0.721 X-Spam-Level: X-Spam-Status: No, score=0.721 tagged_above=-1000 required=6.2 tests=[AWL=0.745, BAYES_00=-2.599, RCVD_IN_PBL=0.905, RCVD_IN_SORBS_DUL=0.877, RDNS_DYNAMIC=0.1, SPF_FAIL=0.693] autolearn=no Received: from core.stromnet.se ([83.218.84.131]) by localhost (core.stromnet.se [83.218.84.131]) (amavisd-new, port 10024) with ESMTP id rYP026E4xpXb; Wed, 10 Mar 2010 19:17:03 +0100 (CET) Received: from johan-mb.vpn.stromnet.se (c-b4fe70d5.016-325-67626721.cust.bredbandsbolaget.se [213.112.254.180]) by core.stromnet.se (Postfix) with ESMTP id 97E24F5743B; Wed, 10 Mar 2010 19:17:03 +0100 (CET) Message-Id: <02981D7A-09B4-4964-9E2F-63646B2C7129@stromnet.se> From: =?ISO-8859-1?Q?Johan_Str=F6m?= To: =?ISO-8859-1?Q?Johan_Str=F6m?= In-Reply-To: <4231C45D-499B-4FC9-90C3-BCC34DF20965@stromnet.se> Content-Type: text/plain; charset=ISO-8859-1; format=flowed; delsp=yes Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Apple Message framework v936) Date: Wed, 10 Mar 2010 19:17:03 +0100 References: <8BA8475A-424E-489B-B643-8757F375320B@stromnet.se> <4231C45D-499B-4FC9-90C3-BCC34DF20965@stromnet.se> X-Mailer: Apple Mail (2.936) Cc: freebsd-fs@freebsd.org Subject: Re: ZFS: zpool import hang on "zio->io_cv)" (with DDB output). Help needed! X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 18:17:11 -0000 Labels look alright: # zdb -l /dev/ad6 -------------------------------------------- LABEL 0 -------------------------------------------- failed to unpack label 0 -------------------------------------------- LABEL 1 -------------------------------------------- failed to unpack label 1 -------------------------------------------- LABEL 2 -------------------------------------------- version=3D13 name=3D'bench' state=3D2 txg=3D2166 pool_guid=3D10982925876172809874 hostid=3D224084357 hostname=3D'back-1.stromnet.se' top_guid=3D8240321046841083771 guid=3D6775655865709593495 vdev_tree type=3D'raidz' id=3D0 guid=3D8240321046841083771 nparity=3D1 metaslab_array=3D23 metaslab_shift=3D35 ashift=3D9 asize=3D6001182375936 is_log=3D0 children[0] type=3D'disk' id=3D0 guid=3D1346858432892182394 path=3D'/dev/ad6' whole_disk=3D0 children[1] type=3D'disk' id=3D1 guid=3D5713829410293466138 path=3D'/dev/ad16' whole_disk=3D0 children[2] type=3D'disk' id=3D2 guid=3D6775655865709593495 path=3D'/dev/ad18' whole_disk=3D0 -------------------------------------------- LABEL 3 -------------------------------------------- failed to unpack label 3 # zdb -l /dev/ad16 -------------------------------------------- LABEL 0 -------------------------------------------- failed to unpack label 0 -------------------------------------------- LABEL 1 -------------------------------------------- failed to unpack label 1 -------------------------------------------- LABEL 2 -------------------------------------------- version=3D13 name=3D'bench' state=3D2 txg=3D2166 pool_guid=3D10982925876172809874 hostid=3D224084357 hostname=3D'back-1.stromnet.se' top_guid=3D8240321046841083771 guid=3D5713829410293466138 vdev_tree type=3D'raidz' id=3D0 guid=3D8240321046841083771 nparity=3D1 metaslab_array=3D23 metaslab_shift=3D35 ashift=3D9 asize=3D6001182375936 is_log=3D0 children[0] type=3D'disk' id=3D0 guid=3D1346858432892182394 path=3D'/dev/ad6' whole_disk=3D0 children[1] type=3D'disk' id=3D1 guid=3D5713829410293466138 path=3D'/dev/ad16' whole_disk=3D0 children[2] type=3D'disk' id=3D2 guid=3D6775655865709593495 path=3D'/dev/ad18' whole_disk=3D0 -------------------------------------------- LABEL 3 -------------------------------------------- failed to unpack label 3 # zdb -l /dev/ad18 -------------------------------------------- LABEL 0 -------------------------------------------- version=3D13 name=3D'bench' state=3D2 txg=3D8 pool_guid=3D13542658689232285344 hostid=3D224084357 hostname=3D'back-1.stromnet.se' top_guid=3D14685823632815031808 guid=3D3228666435075435579 vdev_tree type=3D'raidz' id=3D0 guid=3D14685823632815031808 nparity=3D1 metaslab_array=3D23 metaslab_shift=3D36 ashift=3D9 asize=3D8001555529728 is_log=3D0 children[0] type=3D'disk' id=3D0 guid=3D6901292936454137887 path=3D'/dev/ad6' whole_disk=3D0 children[1] type=3D'disk' id=3D1 guid=3D10179612237289717409 path=3D'/dev/ad16' whole_disk=3D0 children[2] type=3D'disk' id=3D2 guid=3D9860099641022027014 path=3D'/dev/ad18' whole_disk=3D0 children[3] type=3D'disk' id=3D3 guid=3D3228666435075435579 path=3D'/dev/amrd2' whole_disk=3D0 -------------------------------------------- LABEL 1 -------------------------------------------- version=3D13 name=3D'bench' state=3D2 txg=3D8 pool_guid=3D13542658689232285344 hostid=3D224084357 hostname=3D'back-1.stromnet.se' top_guid=3D14685823632815031808 guid=3D3228666435075435579 vdev_tree type=3D'raidz' id=3D0 guid=3D14685823632815031808 nparity=3D1 metaslab_array=3D23 metaslab_shift=3D36 ashift=3D9 asize=3D8001555529728 is_log=3D0 children[0] type=3D'disk' id=3D0 guid=3D6901292936454137887 path=3D'/dev/ad6' whole_disk=3D0 children[1] type=3D'disk' id=3D1 guid=3D10179612237289717409 path=3D'/dev/ad16' whole_disk=3D0 children[2] type=3D'disk' id=3D2 guid=3D9860099641022027014 path=3D'/dev/ad18' whole_disk=3D0 children[3] type=3D'disk' id=3D3 guid=3D3228666435075435579 path=3D'/dev/amrd2' whole_disk=3D0 -------------------------------------------- LABEL 2 -------------------------------------------- version=3D13 name=3D'test' state=3D0 txg=3D4 pool_guid=3D4936907373550577113 hostname=3D'' top_guid=3D3609426615657631821 guid=3D13380697164729542683 vdev_tree type=3D'raidz' id=3D0 guid=3D3609426615657631821 nparity=3D1 metaslab_array=3D23 metaslab_shift=3D36 ashift=3D9 asize=3D8001576501248 is_log=3D0 children[0] type=3D'disk' id=3D0 guid=3D13380697164729542683 path=3D'/dev/ad4' whole_disk=3D0 children[1] type=3D'disk' id=3D1 guid=3D10325582179578809077 path=3D'/dev/ad6' whole_disk=3D0 children[2] type=3D'disk' id=3D2 guid=3D4289142720340552060 path=3D'/dev/ad8' whole_disk=3D0 children[3] type=3D'disk' id=3D3 guid=3D8524377709988872861 path=3D'/dev/ad10' whole_disk=3D0 -------------------------------------------- LABEL 3 -------------------------------------------- version=3D13 name=3D'test' state=3D0 txg=3D4 pool_guid=3D4936907373550577113 hostname=3D'' top_guid=3D3609426615657631821 guid=3D13380697164729542683 vdev_tree type=3D'raidz' id=3D0 guid=3D3609426615657631821 nparity=3D1 metaslab_array=3D23 metaslab_shift=3D36 ashift=3D9 asize=3D8001576501248 is_log=3D0 children[0] type=3D'disk' id=3D0 guid=3D13380697164729542683 path=3D'/dev/ad4' whole_disk=3D0 children[1] type=3D'disk' id=3D1 guid=3D10325582179578809077 path=3D'/dev/ad6' whole_disk=3D0 children[2] type=3D'disk' id=3D2 guid=3D4289142720340552060 path=3D'/dev/ad8' whole_disk=3D0 children[3] type=3D'disk' id=3D3 guid=3D8524377709988872861 path=3D'/dev/ad10' whole_disk=3D0 Won't try to import these, will clear the drives out and start =20 migrating data to my new zraid1. Johan On Mar 9, 2010, at 20:07 , Johan Str=F6m wrote: > Got it working again, by ripping out my WD20EARS bench drives. Some =20= > more background: > > The pool I was testing on was created like this (for 3 disks, ad6, =20 > ad16, ad18): > > gpart create -s gpt adN > gpart add -t freebsd-zfs -b 40 adN > ...for all 3 drives.. > > zpool create bench raidz1 ad6 ad16 ad18 > > The reason for this was to test performance difference between =20 > running straight on disk, and running on a 4k aligned partition (I =20 > was benching and doing experiments on my new WD20EARS 4k sector size =20= > SATA disks). > > Now, it was during bonnie++'ing these drives that the system =20 > paniced. If that has anything at all to do with the panic, I do not =20= > know since I was unable to obtain any dump/output. > > However, after booting again, it failed to import as described =20 > earlier. > I now powered the machine down, and disconnected all the drives in =20 > the above pool (ad6, ad16, ad18 and another amrd2 which was not in =20 > use at the time). > After power on, the import went fine! > > So, where was the problem? I don't really know. Could have been a =20 > bunch of things I guess? Some thoughts: > - ZFS failed to import from the GPT partitions for some reason > - The ZFS labels was borked due to me having booted with invalid /=20 > boot/zfs/zpool.cache (booting from old drive, new/correct loaders & =20= > kernels though) > - Borked ZFS labels due to panic. will try to plug the other disks =20= > back in tomorrow and try to zdb out the labels. > > As a side note, from what I could get out of the testing, zfs on 4k =20= > aligned partitions was not faster. However, for anyone using these =20 > drives and UFS, aligning does make for a pretty nice improvement! =20 > Some details can be found here: > http://www.stromnet.se/~johan/bonnie-wd20ears-align-test.html (for =20 > UFS, compare first two tables, unaligned vs aligned, on for example =20= > Sequential output; block K/Sec and the latency). > Disclaimer: this was not very scientific and I'm not sure if I'm =20 > interpreting the results correctly, but it would seem that the =20 > average values are 77-80MiB/s vs avg 55MiB/s, and latency sub 1000ms =20= > vs avg 2500ms. Exactly what that means I'm not sure but it IS =20 > better. :) > =46rom what I've understood regarding zfs and 4k drivs, since ZFS =20 > internally works with 512b blocks, it doesnt matter if I try to =20 > align anyway, until the underlying system can report 4k sectors =20 > properly (including getting these frikkin drives to do that, which =20 > they currently don't). > > Anyhow, besides the big side note, if anyone cares to try to =20 > reproduce.. Feel free :) I'll get back with zdb dumps on the other =20 > drives later on. > > Thanks for ZFS and freebsd :) > Johan > > > On Mar 9, 2010, at 17:22 , Johan Str=F6m wrote: > >> Some followup, pjd tried to help me on IRC, without getting much =20 >> further on the actual problem. >> Some more information: >> >> Output from DDB PS: http://www.stromnet.se/~johan/back-1-ddb-ps.txt >> Output from alltrace: = http://www.stromnet.se/~johan/back-1-ddb-alltrace.txt >> >> db> show alllocks >> Process 2738 (sshd) thread 0xffffff0050418000 (100178) >> Process 12 (intr) thread 0xffffff0002407ab0 (100018) >> db> show lockedvnods >> Locked vnodes >> db > >> >> >> Also tried accessing the disks (to make sure interrupts etc where =20 >> alive) ,using dd if=3D/dev/adXX of=3D/dev/null count=3D1 on every = disk on =20 >> the system, works fine. >> Rebooted the system and tried with zfs debugging enabled too, and =20 >> doing zpool import tank instead of just zpool import. no =20 >> difference. Hanging there now. >> >> Johan >> >> >> On Mar 9, 2010, at 15:26 , Johan Str=F6m wrote: >> >>> Hi List! >>> >>> I'm in the tedious process of upgrading my pool on my FreeBSD 8.0 =20= >>> box (with a pool from 7.x). Yesterday I pulled a few disks from my =20= >>> mirrored pool (one pool with multiple mirrors), in order to be =20 >>> able to (free ports) plug in new ones and build my brand new pool. >>> After removing disks, the pool got into state DEGRADED ofcourse, =20 >>> since some of the disks where gone, but data was still there. I =20 >>> did zpool detach on the removed disks, and the pool was ONLINE =20 >>> again, all fine! >>> Then I rebooted to make another disk available through my old LSI =20= >>> MegaRaid card (didnt take time to figure out cryptic syntax of the =20= >>> megarc CLI util). On boot, I was met by something similar to this: >>> >>> pool: tank >>> state: UNAVAIL >>> status: One or more devices could not be used because the label is =20= >>> missing >>> or invalid. There are insufficient replicas for the pool to =20= >>> continue >>> functioning. >>> action: Destroy and re-create the pool from a backup source. >>> see: http://www.sun.com/msg/ZFS-8000-5E >>> scrub: none requested >>> config: >>> >>> NAME STATE READ WRITE CKSUM >>> tank UNAVAIL 0 0 0 insufficient replicas >>> ad10s1d ONLINE 0 0 0 >>> mirror DEGRADED 0 0 0 >>> ad12 FAULTED 0 0 0 corrupted data >>> ad16 UNAVAIL 0 0 0 corrupted data >>> mirror DEGRADED 0 0 0 >>> ad20 FAULTED 0 0 0 corrupted data >>> ad18 UNAVAIL 0 0 0 corrupted data >>> mirror UNAVAIL 0 0 0 insufficient replicas >>> ad6 UNAVAIL 0 0 0 corrupted data >>> ad4 FAULTED 0 0 0 corrupted data >>> mirror ONLINE 0 0 0 >>> amrd0 ONLINE 0 0 0 >>> amrd1 ONLINE 0 0 0 >>> >>> The reason the disks was still there was due to me haveing an old =20= >>> zfs.cache file in the boot (it boots from anothre drive, but thats =20= >>> another story..) >>> >>> In this case, a export/import did the trick, after reimporting, =20 >>> the pool was back online. All fine. I created a new pool and did =20 >>> some bonnie++ testing on it, and suddenly the box paniced or =20 >>> something (I didn't have dumpon enabled :/ and didn't see the =20 >>> screen until it rebooted).. >>> >>> Now my problems come. The box came up again, with the above =20 >>> output. I tried zpool export again, fine. Zpool import however, =20 >>> hang. Waited an hour, nothing. After rebuilding the kernel with =20 >>> DDB/witness and did the import again, i've managed to get this =20 >>> output (im not really sure what is usable here): >>> >>> back-1 # zpool import >>> load: 0.29 cmd: zpool 3193 [zio->io_cv)] 2.54r 0.00u 0.01s 0% 2236k >>> >>> >>> >>> In DDB: >>> > tr 3193 >>> >>> Tracing pid 3193 tid 100122 td 0xffffff00035ca390 >>> sched_switch() at sched_switch+0xde >>> mi_switch() at mi_switch+0x170 >>> sleepq_wait() at sleepq_wait+0x44 >>> _cv_wait() at _cv_wait+0x13c >>> zio_wait() at zio_wait+0x61 >>> arc_read_nolock() at arc_read_nolock+0x345 >>> dmu_objset_open_impl() at dmu_objset_open_impl+0xd0 >>> dsl_pool_open() at dsl_pool_open+0x5a >>> spa_load() at spa_load+0x31b >>> spa_tryimport() at spa_tryimport+0xa9 >>> zfs_ioc_pool_tryimport() at zfs_ioc_pool_tryimport+0x3f >>> zfsdev_ioctl() at zfsdev_ioctl+0x8d >>> devfs_ioctl_f() at devfs_ioctl_f+0x76 >>> kern_ioctl() at kern_ioctl+0xf6 >>> ioctl() at ioctl+0xfd >>> syscall() at syscall+0x19e >>> Xfast_syscall() at Xfast_syscall+0xe1 >>> --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x8010eb86c, rsp =3D = =20 >>> 0x7fffffff8e28, rbp =3D 0x801323300 --- >>> >>> >>> db> show thread 100122 >>> >>> Thread 100122 at 0xffffff00035ca390: >>> proc (pid 3193): 0xffffff000383c460 >>> name: zpool >>> stack: 0xffffff805740a000-0xffffff805740dfff >>> flags: 0x44 pflags: 0x10000 >>> state: INHIBITED: {SLEEPING} >>> wmesg: zio->io_cv) wchan: 0xffffff00506e5858 >>> priority: 131 >>> container lock: sleepq chain (0xffffffff80c61e68) >>> db> >>> >>> db> show sleepchain 10012 >>> >>> db> show sleepchain 3193 >>> >>> thread 100122 (pid 3193, zpool) sleeping on 0xffffff00506e5858 =20 >>> "zio->io_cv)" >>> db> show lock 0xffffff00506e5858 >>> >>> class: spin mutex >>> name: zio->io_cv) >>> flags: {SPIN} >>> state: {OWNED} >>> >>> >>> >>> >>> The box is currenlty in this state, so if you reply to me now I =20 >>> can continue to do debugging according to instructions. >>> Since I cannot get anywhere on my own with this, all and any help =20= >>> is appreciated since I really need this pool to get online.. >>> >>> dmesg is posted below (including some LOR?) >>> >>> Thanks! >>> Johan >>> >>> >>> >>> DMESG: >>> >>> Copyright (c) 1992-2009 The FreeBSD Project. >>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, =20 >>> 1993, 1994 >>> The Regents of the University of California. All rights =20 >>> reserved. >>> FreeBSD is a registered trademark of The FreeBSD Foundation. >>> FreeBSD 8.0-RELEASE-p2 #10: Tue Mar 9 12:44:15 CET 2010 >>> johan@back-1.stromnet.se:/usr/obj/usr/src/sys/BACK1 >>> WARNING: WITNESS option enabled, expect reduced performance. >>> Timecounter "i8254" frequency 1193182 Hz quality 0 >>> CPU: Intel(R) Core(TM)2 Duo CPU E6750 @ 2.66GHz (2666.68-MHz =20= >>> K8-class CPU) >>> Origin =3D "GenuineIntel" Id =3D 0x6fb Stepping =3D 11 >>> Features=20 >>> =3D=20 >>> 0xbfebfbff=20 >>> <=20 >>> FPU=20 >>> ,VME=20 >>> ,DE=20 >>> ,PSE=20 >>> ,TSC=20 >>> ,MSR=20 >>> ,PAE=20 >>> ,MCE=20 >>> ,CX8=20 >>> ,APIC=20 >>> ,SEP=20 >>> ,MTRR=20 >>> ,PGE=20 >>> ,MCA=20 >>> ,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> >>> Features2=20 >>> =3D0xe3fd= >>> AMD Features=3D0x20100800 >>> AMD Features2=3D0x1 >>> TSC: P-state invariant >>> real memory =3D 2147483648 (2048 MB) >>> avail memory =3D 2040631296 (1946 MB) >>> ACPI APIC Table: >>> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs >>> FreeBSD/SMP: 1 package(s) x 2 core(s) >>> cpu0 (BSP): APIC ID: 0 >>> cpu1 (AP): APIC ID: 1 >>> ioapic0: Changing APIC ID to 2 >>> ioapic0 irqs 0-23 on motherboard >>> kbd1 at kbdmux0 >>> cryptosoft0: on motherboard >>> acpi0: on motherboard >>> acpi0: [ITHREAD] >>> acpi0: Power Button (fixed) >>> acpi0: reservation of 0, a0000 (3) failed >>> acpi0: reservation of 100000, 7f4e0000 (3) failed >>> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 >>> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 >>> acpi_hpet0: iomem =20 >>> 0xfed00000-0xfed003ff on acpi0 >>> Timecounter "HPET" frequency 14318180 Hz quality 900 >>> acpi_button0: on acpi0 >>> pcib0: port 0xcf8-0xcff on acpi0 >>> pci0: on pcib0 >>> vgapci0: port 0xe000-0xe007 mem =20 >>> 0xe6300000-0xe637ffff,0xd0000000-0xdfffffff,0xe6000000-0xe60fffff =20= >>> irq 16 at device 2.0 on pci0 >>> agp0: on vgapci0 >>> agp0: detected 7164k stolen memory >>> agp0: aperture size is 256M >>> uhci0: port 0xe100-0xe11f irq =20= >>> 16 at device 26.0 on pci0 >>> uhci0: [ITHREAD] >>> uhci0: LegSup =3D 0x2f00 >>> usbus0: on uhci0 >>> uhci1: port 0xe200-0xe21f irq =20= >>> 21 at device 26.1 on pci0 >>> uhci1: [ITHREAD] >>> uhci1: LegSup =3D 0x2f00 >>> usbus1: on uhci1 >>> uhci2: port 0xe600-0xe61f irq =20= >>> 18 at device 26.2 on pci0 >>> uhci2: [ITHREAD] >>> uhci2: LegSup =3D 0x2f00 >>> usbus2: on uhci2 >>> ehci0: mem =20 >>> 0xe6384000-0xe63843ff irq 18 at device 26.7 on pci0 >>> ehci0: [ITHREAD] >>> usbus3: EHCI version 1.0 >>> usbus3: on ehci0 >>> hdac0: mem =20 >>> 0xe6380000-0xe6383fff irq 22 at device 27.0 on pci0 >>> hdac0: HDA Driver Revision: 20090624_0136 >>> hdac0: [ITHREAD] >>> pcib1: irq 16 at device 28.0 on pci0 >>> pci1: on pcib1 >>> pcib2: irq 18 at device 28.2 on pci0 >>> pci2: on pcib2 >>> em0: port =20 >>> 0xa000-0xa01f mem 0xe1020000-0xe103ffff,0xe1000000-0xe101ffff irq =20= >>> 18 at device 0.0 on pci2 >>> em0: Using MSI interrupt >>> em0: [FILTER] >>> em0: Ethernet address: 00:1b:21:05:00:b4 >>> pcib3: irq 19 at device 28.3 on pci0 >>> pci3: on pcib3 >>> atapci0: port =20 >>> 0xb000=20 >>> -0xb007,0xb100-0xb103,0xb200-0xb207,0xb300-0xb303,0xb400-0xb40f =20 >>> mem 0xe6100000-0xe6101fff irq 19 at device 0.0 on pci3 >>> atapci0: [ITHREAD] >>> atapci0: AHCI called from vendor specific driver >>> atapci0: AHCI v1.00 controller with 2 3Gbps ports, PM supported >>> ata2: on atapci0 >>> ata2: [ITHREAD] >>> ata3: on atapci0 >>> ata3: [ITHREAD] >>> ata4: on atapci0 >>> ata4: [ITHREAD] >>> pcib4: irq 16 at device 28.4 on pci0 >>> pci4: on pcib4 >>> re0: >> 8111CP/8111DP PCIe Gigabit Ethernet> port 0xc000-0xc0ff mem =20 >>> 0xe3000000-0xe3000fff irq 16 at device 0.0 on pci4 >>> re0: Using 1 MSI messages >>> re0: Chip rev. 0x38000000 >>> re0: MAC rev. 0x00000000 >>> miibus0: on re0 >>> rgephy0: PHY 1 on miibus0 >>> rgephy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, =20 >>> 1000baseT, 1000baseT-FDX, auto >>> re0: Ethernet address: 00:1a:4d:5a:97:87 >>> re0: [FILTER] >>> uhci3: port 0xe300-0xe31f irq =20= >>> 23 at device 29.0 on pci0 >>> uhci3: [ITHREAD] >>> uhci3: LegSup =3D 0x2f00 >>> usbus4: on uhci3 >>> uhci4: port 0xe400-0xe41f irq =20= >>> 19 at device 29.1 on pci0 >>> uhci4: [ITHREAD] >>> uhci4: LegSup =3D 0x2f00 >>> usbus5: on uhci4 >>> uhci5: port 0xe500-0xe51f irq =20= >>> 18 at device 29.2 on pci0 >>> uhci5: [ITHREAD] >>> uhci5: LegSup =3D 0x2f00 >>> usbus6: on uhci5 >>> ehci1: mem =20 >>> 0xe6385000-0xe63853ff irq 23 at device 29.7 on pci0 >>> ehci1: [ITHREAD] >>> usbus7: EHCI version 1.0 >>> usbus7: on ehci1 >>> pcib5: at device 30.0 on pci0 >>> pci5: on pcib5 >>> amr0: mem 0xe6200000-0xe620ffff irq 20 at =20= >>> device 0.0 on pci5 >>> amr0: Using 64-bit DMA >>> amr0: [ITHREAD] >>> amr0: delete logical drives supported by controller >>> amr0: Firmware 713S, BIOS G121, =20 >>> 64MB RAM >>> xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xd000-0xd07f mem =20 >>> 0xe5004000-0xe500407f irq 19 at device 1.0 on pci5 >>> miibus1: on xl0 >>> xlphy0: <3c905C 10/100 internal PHY> PHY 24 on miibus1 >>> xlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto >>> xl0: Ethernet address: 00:04:76:ef:c6:36 >>> xl0: [ITHREAD] >>> skc0: port 0xd100-0xd1ff mem =20= >>> 0xe5000000-0xe5003fff irq 18 at device 2.0 on pci5 >>> pci0:5:2:0: invalid VPD data, remain 0xfc >>> skc0: SysKonnect SK-NET Gigabit Ethernet Adapter SK-9843 SX rev. =20 >>> (0x0) >>> sk0: on skc0 >>> sk0: Ethernet address: 00:00:5a:98:43:68 >>> miibus2: on sk0 >>> xmphy0: PHY 0 on miibus2 >>> xmphy0: 1000baseSX, 1000baseSX-FDX, auto >>> skc0: [ITHREAD] >>> isab0: at device 31.0 on pci0 >>> isa0: on isab0 >>> atapci1: port =20 >>> 0xe700=20 >>> -0xe707,0xe800-0xe803,0xe900-0xe907,0xea00-0xea03,0xeb00-0xeb1f =20 >>> mem 0xe6386000-0xe63867ff irq 19 at device 31.2 on pci0 >>> atapci1: [ITHREAD] >>> atapci1: AHCI called from vendor specific driver >>> atapci1: AHCI v1.20 controller with 6 3Gbps ports, PM supported >>> ata5: on atapci1 >>> ata5: [ITHREAD] >>> ata6: on atapci1 >>> ata6: [ITHREAD] >>> ata7: on atapci1 >>> ata7: [ITHREAD] >>> ata8: on atapci1 >>> ata8: [ITHREAD] >>> ata9: on atapci1 >>> ata9: [ITHREAD] >>> ata10: on atapci1 >>> ata10: [ITHREAD] >>> pci0: at device 31.3 (no driver attached) >>> atrtc0: port 0x70-0x73 on acpi0 >>> fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 =20= >>> on acpi0 >>> fdc0: [FILTER] >>> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on =20= >>> acpi0 >>> uart0: [FILTER] >>> ppc0: port 0x378-0x37f irq 7 on acpi0 >>> ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode >>> ppc0: [ITHREAD] >>> ppbus0: on ppc0 >>> plip0: on ppbus0 >>> plip0: [ITHREAD] >>> lpt0: on ppbus0 >>> lpt0: [ITHREAD] >>> lpt0: Interrupt-driven port >>> ppi0: on ppbus0 >>> atkbdc0: port 0x60,0x64 irq 1 on acpi0 >>> atkbd0: irq 1 on atkbdc0 >>> kbd0 at atkbd0 >>> atkbd0: [GIANT-LOCKED] >>> atkbd0: [ITHREAD] >>> cpu0: on acpi0 >>> est0: on cpu0 >>> est: CPU supports Enhanced Speedstep, but is not recognized. >>> est: cpu_vendor GenuineIntel, msr 82a082a0600082a >>> device_attach: est0 attach returned 6 >>> p4tcc0: on cpu0 >>> cpu1: on acpi0 >>> est1: on cpu1 >>> est: CPU supports Enhanced Speedstep, but is not recognized. >>> est: cpu_vendor GenuineIntel, msr 82a082a0600082a >>> device_attach: est1 attach returned 6 >>> p4tcc1: on cpu1 >>> orm0: at iomem 0xcc000-0xcc7ff,0xcd000-0xcefff =20 >>> on isa0 >>> sc0: at flags 0x100 on isa0 >>> sc0: VGA <16 virtual consoles, flags=3D0x300> >>> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff =20= >>> on isa0 >>> ZFS NOTICE: Prefetch is disabled by default if less than 4GB of =20 >>> RAM is present; >>> to enable, add "vfs.zfs.prefetch_disable=3D0" to /boot/=20 >>> loader.conf. >>> ZFS filesystem version 13 >>> ZFS storage pool version 13 >>> Timecounters tick every 1.000 msec >>> IPsec: Initialized Security Association Processing. >>> usbus0: 12Mbps Full Speed USB v1.0 >>> usbus1: 12Mbps Full Speed USB v1.0 >>> usbus2: 12Mbps Full Speed USB v1.0 >>> usbus3: 480Mbps High Speed USB v2.0 >>> usbus4: 12Mbps Full Speed USB v1.0 >>> usbus5: 12Mbps Full Speed USB v1.0 >>> usbus6: 12Mbps Full Speed USB v1.0 >>> usbus7: 480Mbps High Speed USB v2.0 >>> ad4: 476938MB at ata2-master SATA300 >>> ugen0.1: at usbus0 >>> uhub0: on =20= >>> usbus0 >>> ugen1.1: at usbus1 >>> uhub1: on =20= >>> usbus1 >>> ugen2.1: at usbus2 >>> uhub2: on =20= >>> usbus2 >>> ugen3.1: at usbus3 >>> uhub3: on =20= >>> usbus3 >>> ugen4.1: at usbus4 >>> uhub4: on =20= >>> usbus4 >>> ugen5.1: at usbus5 >>> uhub5: on =20= >>> usbus5 >>> ugen6.1: at usbus6 >>> uhub6: on =20= >>> usbus6 >>> ugen7.1: at usbus7 >>> uhub7: on =20= >>> usbus7 >>> ad6: 1907729MB at ata3-master SATA300 >>> uhub0: 2 ports with 2 removable, self powered >>> uhub1: 2 ports with 2 removable, self powered >>> uhub2: 2 ports with 2 removable, self powered >>> uhub4: 2 ports with 2 removable, self powered >>> uhub5: 2 ports with 2 removable, self powered >>> uhub6: 2 ports with 2 removable, self powered >>> ad10: 715404MB at ata5-master =20 >>> SATA300 >>> ad12: 305245MB at ata6-master SATA150 >>> ad14: 715404MB at ata7-master =20 >>> SATA300 >>> ad16: 1907729MB at ata8-master =20 >>> SATA300 >>> GEOM_MIRROR: Device mirror/gm1a launched (1/1). >>> GEOM_MIRROR: Device mirror/gm1b launched (1/1). >>> ad18: 1907729MB at ata9-master =20 >>> SATA300 >>> GEOM_MIRROR: Device mirror/swap launched (1/1). >>> ad20: 286187MB at ata10-master SATA150 >>> hdac0: HDA Codec #2: Realtek ALC885 >>> pcm0: at cad 2 nid 1 on hdac0 >>> pcm1: at cad 2 nid 1 on hdac0 >>> pcm2: at cad 2 nid 1 on hdac0 >>> pcm3: at cad 2 nid 1 on hdac0 >>> pcm4: at cad 2 nid 1 on hdac0 >>> pcm5: at cad 2 nid 1 on hdac0 >>> amr0: delete logical drives supported by controller >>> amrd0: on amr0 >>> amrd0: 476935MB (976762880 sectors) RAID 0 (optimal) >>> amrd1: on amr0 >>> amrd1: 476935MB (976762880 sectors) RAID 0 (optimal) >>> amrd2: on amr0 >>> amrd2: 1907724MB (3907018752 sectors) RAID 0 (optimal) >>> SMP: AP CPU #1 Launched! >>> WARNING: WITNESS option enabled, expect reduced performance. >>> Root mount waiting for: usbus7 usbus3 >>> uhub3: 6 ports with 6 removable, self powered >>> uhub7: 6 ports with 6 removable, self powered >>> Trying to mount root from zfs:zroot >>> ugen0.2: at usbus0 >>> ugen1.2: at usbus1 >>> uma_zalloc_arg: zone "256" with the following non-sleepable locks =20= >>> held: >>> exclusive rw ifnet_rw (ifnet_rw) r =3D 0 (0xffffffff80e01f60) locked = =20 >>> @ /usr/src/sys/net/if.c:402 >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >>> _witness_debugger() at _witness_debugger+0x2c >>> witness_warn() at witness_warn+0x2c2 >>> uma_zalloc_arg() at uma_zalloc_arg+0x29d >>> malloc() at malloc+0x5d >>> if_grow() at if_grow+0x2f >>> if_alloc() at if_alloc+0x2b3 >>> gif_clone_create() at gif_clone_create+0x53 >>> ifc_simple_create() at ifc_simple_create+0x89 >>> if_clone_createif() at if_clone_createif+0x64 >>> ifioctl() at ifioctl+0x6b5 >>> kern_ioctl() at kern_ioctl+0xf6 >>> ioctl() at ioctl+0xfd >>> syscall() at syscall+0x19e >>> Xfast_syscall() at Xfast_syscall+0xe1 >>> --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800b8286c, rsp =3D = =20 >>> 0x7fffffffe4a8, rbp =3D 0x7fffffffef6e --- >>> lock order reversal: >>> 1st 0xffffffff80c093e0 pf task mtx (pf task mtx) @ /usr/src/sys/=20 >>> contrib/pf/net/pf_ioctl.c:1393 >>> 2nd 0xffffffff80e01f60 ifnet_rw (ifnet_rw) @ /usr/src/sys/net/if.c:=20= >>> 2034 >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >>> _witness_debugger() at _witness_debugger+0x2c >>> witness_checkorder() at witness_checkorder+0x66f >>> _rw_rlock() at _rw_rlock+0x29 >>> ifunit() at ifunit+0x22 >>> pfioctl() at pfioctl+0x262a >>> devfs_ioctl_f() at devfs_ioctl_f+0x76 >>> kern_ioctl() at kern_ioctl+0xf6 >>> ioctl() at ioctl+0xfd >>> syscall() at syscall+0x19e >>> Xfast_syscall() at Xfast_syscall+0xe1 >>> --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x80099886c, rsp =3D = =20 >>> 0x7fffffffdb68, rbp =3D 0x7fffffffdc20 --- >>> lock order reversal: >>> 1st 0xffffff00500c3098 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:=20 >>> 1054 >>> 2nd 0xffffff005010f448 devfs (devfs) @ /usr/src/sys/kern/=20 >>> vfs_subr.c:2083 >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >>> _witness_debugger() at _witness_debugger+0x2c >>> witness_checkorder() at witness_checkorder+0x66f >>> __lockmgr_args() at __lockmgr_args+0x475 >>> vop_stdlock() at vop_stdlock+0x39 >>> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 >>> _vn_lock() at _vn_lock+0x47 >>> vget() at vget+0x56 >>> devfs_allocv() at devfs_allocv+0x103 >>> devfs_root() at devfs_root+0x48 >>> vfs_donmount() at vfs_donmount+0xf43 >>> nmount() at nmount+0x63 >>> syscall() at syscall+0x19e >>> Xfast_syscall() at Xfast_syscall+0xe1 >>> --- syscall (378, FreeBSD ELF64, nmount), rip =3D 0x8007b04dc, rsp =3D= =20 >>> 0x7fffffffdd28, rbp =3D 0x800a04048 --- >>> n >>> >>> r >>> tun0: link state changed to UP >>> lock order reversal: >>> 1st 0xffffff00500c3098 zfs (zfs) @ /usr/src/sys/kern/vfs_mount.c:=20 >>> 1200 >>> 2nd 0xffffff005010f270 syncer (syncer) @ /usr/src/sys/kern/=20 >>> vfs_subr.c:2188 >>> KDB: stack backtrace: >>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2a >>> _witness_debugger() at _witness_debugger+0x2c >>> witness_checkorder() at witness_checkorder+0x66f >>> __lockmgr_args() at __lockmgr_args+0x475 >>> vop_stdlock() at vop_stdlock+0x39 >>> VOP_LOCK1_APV() at VOP_LOCK1_APV+0x46 >>> _vn_lock() at _vn_lock+0x47 >>> vrele() at vrele+0xc3 >>> dounmount() at dounmount+0x269 >>> unmount() at unmount+0x27e >>> syscall() at syscall+0x19e >>> Xfast_syscall() at Xfast_syscall+0xe1 >>> --- syscall (22, FreeBSD ELF64, unmount), rip =3D 0x8006a09bc, rsp =3D= =20 >>> 0x7fffffffde18, rbp =3D 0 --- >>> KDB: enter: manual escape to debugger >>> >>> _______________________________________________ >>> freebsd-fs@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-fs >>> To unsubscribe, send any mail to "freebsd-fs-=20 >>> unsubscribe@freebsd.org" >> > From owner-freebsd-fs@FreeBSD.ORG Wed Mar 10 20:57:22 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id E55591065670 for ; Wed, 10 Mar 2010 20:57:21 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 2709C8FC28 for ; Wed, 10 Mar 2010 20:57:20 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 703E345E35; Wed, 10 Mar 2010 21:57:18 +0100 (CET) Received: from localhost (chello089077043238.chello.pl [89.77.43.238]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id D459E45C9B; Wed, 10 Mar 2010 21:57:12 +0100 (CET) Date: Wed, 10 Mar 2010 21:57:11 +0100 From: Pawel Jakub Dawidek To: Kevin Day Message-ID: <20100310205711.GA1847@garage.freebsd.pl> References: <7418ECC2-55C1-4A28-82EA-0972AFE745EF@dragondata.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="5mCyUwZo2JvN/JJP" Content-Disposition: inline In-Reply-To: <7418ECC2-55C1-4A28-82EA-0972AFE745EF@dragondata.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: iscsi over HAST backed storage partial success X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Mar 2010 20:57:22 -0000 --5mCyUwZo2JvN/JJP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Mar 09, 2010 at 05:03:41PM -0600, Kevin Day wrote: >=20 > I'm running istgt (iscsi target) using HAST backed storage. For the most = part, it seems to work really well. I have ucarp running to change the IP t= hat istgt is bound to, and modified the ucarp scripts to start/stop istgt d= epending on which side is the master. If I shut down the primary, the secon= dary takes over and all seems well. >=20 > However, if I reboot the secondary, the primary starts freezing up for lo= ng periods: >=20 > Mar 9 22:46:27 cs04 hastd: [iscsi1] (primary) Unable to r: Socket is not= connected. > Mar 9 22:46:27 cs04 hastd: [iscsi1] (primary) Unable to co: Connection r= efused. > Mar 9 22:46:42 cs04 last message repeated 3 times > Mar 9 22:46:53 cs04 istgt[14298]: ABORT_TASK > Mar 9 22:47:35 cs04 last message repeated 3 times > Mar 9 22:48:02 cs04 hastd: [iscsi1] (primary) Unable to co: Operation ti= med out. > Mar 9 22:48:02 cs04 istgt[14298]: CmdSN(45748), OP=3D0x2a, ElapsedTime= =3D74 cleared=20 > Mar 9 22:48:02 cs04 istgt[14298]: istgt_iscsi.c: 640:istgt_iscsi_write_p= du: ***ERROR*** iscsi_write() failed (errno=3D32) > Mar 9 22:48:02 cs04 istgt[14298]: istgt_iscsi.c:3327:istgt_iscsi_op_task= : ***ERROR*** iscsi_write_pdu() failed > Mar 9 22:48:02 cs04 istgt[14298]: istgt_iscsi.c:3867:istgt_iscsi_execute= : ***ERROR*** iscsi_op_task() failed =20 > Mar 9 22:48:02 cs04 istgt[14298]: istgt_iscsi.c:4337:worker: ***ERROR***= iscsi_execute() failed > Mar 9 22:48:02 cs04 istgt[14298]: CmdSN(490802), OP=3D0x2a, ElapsedTime= =3D73 cleared > Mar 9 22:48:02 cs04 istgt[14298]: CmdSN(28387), OP=3D0x2a, ElapsedTime= =3D73 cleared=20 > Mar 9 22:48:14 cs04 istgt[14298]: ABORT_TASK > Mar 9 22:48:52 cs04 last message repeated 2 times > Mar 9 22:49:22 cs04 hastd: [iscsi1] (primary) Unable to co: Operation ti= med out. >=20 > As soon as the secondary comes back online, everything starts behaving ag= ain and all is well. Could you try the following patch? http://people.freebsd.org/~pjd/patches/hastd_primary.c.patch --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --5mCyUwZo2JvN/JJP Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkuYB6cACgkQForvXbEpPzRh4gCfWohM4E1Z3WndDs1I12TNB17R 94wAnjAJ47himb+HnGIVn33Pq/X7GmiZ =gbrf -----END PGP SIGNATURE----- --5mCyUwZo2JvN/JJP-- From owner-freebsd-fs@FreeBSD.ORG Thu Mar 11 07:45:38 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A985E1065672; Thu, 11 Mar 2010 07:45:38 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from mail.ebusiness-leidinger.de (mail.ebusiness-leidinger.de [217.11.53.44]) by mx1.freebsd.org (Postfix) with ESMTP id 367388FC27; Thu, 11 Mar 2010 07:45:37 +0000 (UTC) Received: from outgoing.leidinger.net (pD9E2DD69.dip.t-dialin.net [217.226.221.105]) by mail.ebusiness-leidinger.de (Postfix) with ESMTPSA id D27708444D8; Thu, 11 Mar 2010 08:45:30 +0100 (CET) Received: from webmail.leidinger.net (webmail.leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id 8333F5402; Thu, 11 Mar 2010 08:45:27 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=Leidinger.net; s=outgoing-alex; t=1268293527; bh=CULMdTKHhWLorRPNyx8Tmm4zgUkFlilXiK9S8maKbvc=; h=Message-ID:Date:From:To:Cc:Subject:References:In-Reply-To: MIME-Version:Content-Type:Content-Transfer-Encoding; b=0OUAAca2VtQKrzprenZ3TcjifLVYWsF4i1eiuT9Qb58aMZhnOuMECIbIHmbQa7hJq LSInvSodmUywg0PwEslXAl2Or2DrDyjqoiUfThzj8r84Fc20e2xLTM2jmYEwRIw4W/ uWh/mzAmZdh+Wm2+RyX8/WgWlgeBiroKCxzmi8VOUENYtjfkF4Fo6YYk7fBhf3bVf/ FZVYsyvSqYbPR0qctCaXkwlEhbGBUBb4bMStUFIYsptzo0Uve4w4bUVTkHKrW7ZDEM 2PR7l+I43IINwwq96knryZSVaTs7WwmAFJVmMowKDep4dJa41JQdhRNQLiZZ6W6oFN oPfSNibwSKwcA== Received: (from www@localhost) by webmail.leidinger.net (8.14.3/8.13.8/Submit) id o2B7jRka056645; Thu, 11 Mar 2010 08:45:27 +0100 (CET) (envelope-from Alexander@Leidinger.net) Received: from pslux.cec.eu.int (pslux.cec.eu.int [158.169.9.14]) by webmail.leidinger.net (Horde Framework) with HTTP; Thu, 11 Mar 2010 08:45:27 +0100 Message-ID: <20100311084527.2934034895hvgxaw@webmail.leidinger.net> Date: Thu, 11 Mar 2010 08:45:27 +0100 From: Alexander Leidinger To: Pawel Jakub Dawidek References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> <20100310173143.GD1715@garage.freebsd.pl> In-Reply-To: <20100310173143.GD1715@garage.freebsd.pl> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Dynamic Internet Messaging Program (DIMP) H3 (1.1.4) X-EBL-MailScanner-Information: Please contact the ISP for more information X-EBL-MailScanner-ID: D27708444D8.58B03 X-EBL-MailScanner: Found to be clean X-EBL-MailScanner-SpamCheck: not spam, spamhaus-ZEN, SpamAssassin (not cached, score=-1.286, required 6, autolearn=disabled, ALL_TRUSTED -1.44, DKIM_SIGNED 0.00, DKIM_VERIFIED -0.00, TW_DV 0.08, TW_ZF 0.08) X-EBL-MailScanner-From: alexander@leidinger.net X-EBL-MailScanner-Watermark: 1268898333.40525@YCCAFIHqqWQfp0/PGkOy+Q X-EBL-Spam-Status: No Cc: freebsd-fs@FreeBSD.org, Stable , FreeBSD Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 07:45:38 -0000 Quoting Pawel Jakub Dawidek (from Wed, 10 Mar 2010 18:31:43 +0100): > On Wed, Mar 10, 2010 at 04:12:36PM +0100, Borja Marcos wrote: >> >> On Mar 10, 2010, at 12:02 PM, Pawel Jakub Dawidek wrote: >> >> > Once the deadlock occur, enter DDB and send me the output of: >> > >> > ps >> > show alllocks >> > show lockedvnods >> > show allchains >> > alltrace >> >> (Again, crossposted to -fs, ZFS related) >> >> >> Previous one was a panic when performing the test with several tar >> jobs running in parallel. >> >> Now this is a capture of the deadlock itself, instead of a panic. >> (I called panic from the debugger to generate a dump) > [...] > > Hmm, interesting. Especially those two traces: > > Tracing command zfs pid 1820 tid 100105 td 0xffffff0002ca4000 > [...] > _cv_wait() at _cv_wait+0x17a > txg_wait_synced() at txg_wait_synced+0x98 > zfsvfs_teardown() at zfsvfs_teardown+0x1f6 > zfs_suspend_fs() at zfs_suspend_fs+0x2b > zfs_ioc_recv() at zfs_ioc_recv+0x28b > zfsdev_ioctl() at zfsdev_ioctl+0x8d > devfs_ioctl_f() at devfs_ioctl_f+0x76 > kern_ioctl() at kern_ioctl+0xc5 > ioctl() at ioctl+0xfd > [...] > > Tracing command bsdtar pid 1699 tid 100093 td 0xffffff000262dae0 > [...] > _sx_slock_hard() at _sx_slock_hard+0x1b7 > _sx_slock() at _sx_slock+0xc1 > zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x63 > VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 > vgonel() at vgonel+0x119 > vnlru_free() at vnlru_free+0x345 > getnewvnode() at getnewvnode+0x24f > zfs_znode_cache_constructor() at zfs_znode_cache_constructor+0x43 > zfs_znode_alloc() at zfs_znode_alloc+0x38 > zfs_mknode() at zfs_mknode+0x259 > zfs_freebsd_create() at zfs_freebsd_create+0x661 > VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 > vn_open_cred() at vn_open_cred+0x473 > kern_openat() at kern_openat+0x179 > [...] > > This should be impossible. If we are that deep in zfsvfs_teardown(), it means > that we hold the z_teardown_lock exclusively. And we do as 'show alllocks' > output confirms. But if we are holding this lock exclusively we shouldn't be > that deep in create code path, because we need hold this lock as reader. > It isn't visible in 'show alllocks' output, because this lock is special > (rrwlock.c). > > I see three possibilities: > 1. We are looking at different file systems here. But where is deadlock > coming from then? > 2. There is a bug in rrwlock.c. Highly unlikely I think. > 3. My thinking is incorrect somewhere. There is a 4th possibility, if you can rule out everything else: bugs in the CPU. I stumbled upon this with ZFS (but UFS was exposing the problem much faster). The problem in my case was that the BIOS was not recognizing the CPU and as such was not uploading microcode updates. Borja, can you confirm that the CPU is correctly announced in FreeBSD (just look at "dmesg | grep CPU:" output, if it tells you it is a AMD or Intel XXX CPU it is correctly detected by the BIOS)? Bye, Alexander. -- Kissing a fish is like smoking a bicycle. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-fs@FreeBSD.ORG Thu Mar 11 08:54:49 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 88CA7106564A; Thu, 11 Mar 2010 08:54:49 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop1.sarenet.es (proxypop1.sarenet.es [194.30.0.99]) by mx1.freebsd.org (Postfix) with ESMTP id D19048FC0A; Thu, 11 Mar 2010 08:54:48 +0000 (UTC) Received: from [172.16.1.204] (unknown [192.148.167.2]) by proxypop1.sarenet.es (Postfix) with ESMTP id 8F1635CDE; Thu, 11 Mar 2010 09:54:47 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20100311084527.2934034895hvgxaw@webmail.leidinger.net> Date: Thu, 11 Mar 2010 09:54:47 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <764BD545-B86C-47DC-9004-964EB2216AF0@sarenet.es> References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> <20100310173143.GD1715@garage.freebsd.pl> <20100311084527.2934034895hvgxaw@webmail.leidinger.net> To: Alexander Leidinger X-Mailer: Apple Mail (2.1077) Cc: freebsd-fs@FreeBSD.org, Stable , FreeBSD@FreeBSD.ORG, Pawel Jakub Dawidek Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 08:54:49 -0000 On Mar 11, 2010, at 8:45 AM, Alexander Leidinger wrote: > Quoting Pawel Jakub Dawidek (from Wed, 10 Mar 2010 = 18:31:43 +0100): >=20 > There is a 4th possibility, if you can rule out everything else: bugs = in the CPU. I stumbled upon this with ZFS (but UFS was exposing the = problem much faster). The problem in my case was that the BIOS was not = recognizing the CPU and as such was not uploading microcode updates. >=20 > Borja, can you confirm that the CPU is correctly announced in FreeBSD = (just look at "dmesg | grep CPU:" output, if it tells you it is a AMD or = Intel XXX CPU it is correctly detected by the BIOS)? A CPU bug? Weird. Very. Let me explain the whole history of this. We are using ZFS to maintain a couple of servers in an active/passive = arrangement. At 30 second intervals we create a snapshot on the master = server and send it to the slave. Actually I prefer this scheme to = drbd-style arrangements, but that's another story ;) We started our tests and soon ran into problems: deadlocked filesystem. = At one point I remember that the deadlock affected UFS as well, not only = ZFS. I mean, having both ZFS and UFS, the system also lost access to the = UFS filesystems when this happened. Looking at the hours when it happened, it turned out to be one or two of = these events: periodic scripts running (which, among other things, = traverse the whole filesystem) and/or a backup being made with Bacula. = Either way, there seemed to be a problem: read activity on a dataset on = which I was receiving a snapshot at the same time could lead to a = deadlock. I am sure I have never tried to receive two snapshots = simultaneously, etc. The replicating program guaratees it. As the servers had to be rolled into production, and such tests with = real servers can be quite time consuming, I set up a couple of FreeBSD = virtual machines, using VMWare Fusion (version 2 then, now version 3) on = a Macbook (Macbook 4,1 Intel Core2Duo, 2.1 GHz) and tried to reproduce = it. To reproduce it, I set up a "master" machine, with /usr/src and /usr/obj = on a dataset (pool/src), replicating it at 30 second intervals to = another virtual machine, the slave. On the slave, I launch "tar" in an = infinite loop, so that the contents of the replicated dataser (pool/src) = is copied to another dataset (pool/thecopy). With that running, and, remember, there are replications at 30 second = intervals (longer if a replication takes a long time, of course) I run a = make buildworld on the master machine. The destination soon gets = deadlocked. I have tried to fiddle with the virtual machine, for example, trying to = offer a single or dual core CPU, and there's no difference. With dual = cores it *seems* to deadlock earlier, but I'm not sure. For the latest = test results I've posted, I was using a single core CPU.=20 The original machines on which I detected the problem (problem I have = subsequently reproduced successfully on virtual machines running on = VMWare Fusion) are Dell PowerEdge 2950, and this is the CPU description: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU L5420 @ 2.50GHz (2496.25-MHz = K8-class CPU) Origin =3D "GenuineIntel" Id =3D 0x1067a Stepping =3D 10 = Features=3D0xbfebfbff = Features2=3D0x40ce3bd AMD Features=3D0x20100800 AMD Features2=3D0x1 TSC: P-state invariant real memory =3D 8589934592 (8192 MB) avail memory =3D 8250003456 (7867 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 1 package(s) x 8 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 ioapic0: Changing APIC ID to 8 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on = acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 The virtual machine (VMWare Fusion 3.0.0, Macbook, Mac OS X 10.6.2) = reports this: Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz (2116.62-MHz = K8-class CPU) Origin =3D "GenuineIntel" Id =3D 0x10676 Stepping =3D 6 = Features=3D0xfebfbff Features2=3D0x80082201> AMD Features=3D0x20100800 AMD Features2=3D0x1 TSC: P-state invariant real memory =3D 1153433600 (1100 MB) avail memory =3D 1090441216 (1039 MB) ACPI APIC Table: MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 In order to compare to Solaris, I installed a virtual machine running = Solaris 10 as well, and used it as a target for the replication. The = same test didn't deadlock and it seemed to work like a charm. Sometimes I've tried to run more than one "tar" job in parallel instead = of just one. It just makes it deadlock earlier, no other difference. Any more tests I can do? Borja. From owner-freebsd-fs@FreeBSD.ORG Thu Mar 11 10:59:06 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 09EAE106566B for ; Thu, 11 Mar 2010 10:59:06 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id B19538FC24 for ; Thu, 11 Mar 2010 10:59:05 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1Npg6a-0007YT-JD for freebsd-fs@freebsd.org; Thu, 11 Mar 2010 11:59:04 +0100 Received: from lara.cc.fer.hr ([161.53.72.113]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 11 Mar 2010 11:59:04 +0100 Received: from ivoras by lara.cc.fer.hr with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 11 Mar 2010 11:59:04 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Ivan Voras Date: Thu, 11 Mar 2010 11:59:01 +0100 Lines: 13 Message-ID: References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> <20100310173143.GD1715@garage.freebsd.pl> <20100311084527.2934034895hvgxaw@webmail.leidinger.net> <764BD545-B86C-47DC-9004-964EB2216AF0@sarenet.es> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: lara.cc.fer.hr User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.5) Gecko/20100118 Thunderbird/3.0 In-Reply-To: <764BD545-B86C-47DC-9004-964EB2216AF0@sarenet.es> Cc: freebsd-stable@freebsd.org Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 10:59:06 -0000 On 03/11/10 09:54, Borja Marcos wrote: I don't know about the rest but this: > CPU: Intel(R) Xeon(R) CPU L5420 @ 2.50GHz (2496.25-MHz K8-class CPU) does not agree with this: > FreeBSD/SMP: 1 package(s) x 8 core(s) The Xeon 54xx series does not come in 8 core packages. Either it is 2xquad-core or a Xeon 55xx. From owner-freebsd-fs@FreeBSD.ORG Thu Mar 11 13:00:15 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AD0CB10656AE; Thu, 11 Mar 2010 13:00:15 +0000 (UTC) (envelope-from md@scoutsengidsenvlaanderen.be) Received: from scoutsengidsenvlaanderen.be (mail.scoutsengidsenvlaanderen.be [81.83.17.184]) by mx1.freebsd.org (Postfix) with ESMTP id 9AB3F8FC2E; Thu, 11 Mar 2010 13:00:13 +0000 (UTC) Received: from [192.168.1.163] (account md [192.168.1.163] verified) by scoutsengidsenvlaanderen.be (CommuniGate Pro SMTP 5.3.3) with ESMTPA id 34078327; Thu, 11 Mar 2010 14:00:11 +0100 Message-ID: <4B98E95B.90806@scoutsengidsenvlaanderen.be> Date: Thu, 11 Mar 2010 14:00:11 +0100 From: Michiel Detailleur User-Agent: Thunderbird 2.0.0.23 (Windows/20090812) MIME-Version: 1.0 To: Pawel Jakub Dawidek References: <4B97A963.9040300@scoutsengidsenvlaanderen.be> <20100310174411.GE1715@garage.freebsd.pl> In-Reply-To: <20100310174411.GE1715@garage.freebsd.pl> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-fs@freebsd.org Subject: Re: [Spam? ] Re: ZFS snapshot name length limit? (File name too long) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 13:00:15 -0000 Pawel Jakub Dawidek wrote: > On Wed, Mar 10, 2010 at 03:14:59PM +0100, Michiel Detailleur wrote: > >> This seems to imply a maximum length of 72 characters (snapshot '@' sign >> excluded) for *usable* snapshots? (usable in the sense that we can >> mount/look into them) >> >> What limit am I (we?) hitting here? Is this a problem with ZFS itself or >> with FreeBSD? Sure looks like a silly limit for the filesystem that >> touts 'biggest everything'. >> > > From looking at the code, I think you hitting this limit: > > This is FreeBSD limit caused by statfs structure: > > /* > * filesystem statistics > */ > [...] > #define MNAMELEN 88 /* size of on/from name bufs */ > [...] > > ZFS mounts snapshots on lookup and this is this mount that fails. > So all that would need to be done to allow mounting snapshots with longer names would be to increase this value? (I'm not a C programmer, so bear with me :) ) Is that something you could commit a change for as the ZFS committer? Or is there more to it than that? In any case: thanks for your answer and for all your hard work in getting ZFS in FreeBSD! :) Michiel From owner-freebsd-fs@FreeBSD.ORG Thu Mar 11 12:39:37 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1DF08106564A for ; Thu, 11 Mar 2010 12:39:37 +0000 (UTC) (envelope-from gallasch@free.de) Received: from smtp.free.de (smtp.free.de [91.204.6.103]) by mx1.freebsd.org (Postfix) with ESMTP id A6CDD8FC21 for ; Thu, 11 Mar 2010 12:39:34 +0000 (UTC) Received: (qmail 14165 invoked from network); 11 Mar 2010 13:39:28 +0100 Received: from smtp.free.de (HELO orwell.free.de) (gallasch@free.de@[91.204.4.103]) (envelope-sender ) by smtp.free.de (qmail-ldap-1.03) with AES128-SHA encrypted SMTP for ; 11 Mar 2010 13:39:28 +0100 Date: Thu, 11 Mar 2010 13:39:16 +0100 From: Kai Gallasch To: freebsd-stable@FreeBSD.org Message-ID: <20100311133916.42ba69b0@orwell.free.de> X-Mailer: Claws Mail 3.7.3 (GTK+ 2.18.6; powerpc-apple-darwin9.8.0) X-Face: 7"x0zA5=*cXGZw-xjU<">'+!3(KXTUXZVLD42KVN{'go[UQr"Mc.e(XW92N8plZ(9x.{x; I<|95e+b&GH-36\15F~L$YD*Y +u}o&KV?6.%"mJIkaY3G>BKNt`1|Y+%K1P4t; 47D65&(Y7h5Ll-[ltkhamx.-; ,jggK'}oMpUgEHFG YQ"9oXKAl>!d,J}T{)@uxvfu?YFWC*\~h+,^f Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailman-Approved-At: Thu, 11 Mar 2010 13:03:31 +0000 Cc: freebsd-fs@FreeBSD.org Subject: proliant server lockups with freebsd-amd64-stable (2010-03-10) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 12:39:37 -0000 Hi. I have some trouble with an opteron server locking up spontaneously. It looses all networks connectivity and even through console I can get no shell. Lockups occur mostly under disk load (periodic daily, bacula backup running, make buildworld/buildkernel) and I can provoke them easily. The following debug options are currently set in kernel.conf: options KDB options DDB options GDB options INVARIANTS options INVARIANT_SUPPORT options WITNESS options WITNESS_SKIPSPIN options SW_WATCHDOG # not available with ULE scheduler #options MP_WATCHDOG options DEBUG_LOCKS options DEBUG_VFS_LOCKS There is no special tuning in sysctl.conf done. Yesterday the server locked up again and through "sysctl debug.kdb.panic=1" I was able to enter the debugger, close before the console locked up completely. Below (below dmesg output) are my debug findings: In the end I did a "call doadump" but writing the dump failed, so it really must be a low level problem. Could this be ciss-related (show allchains), or is the ciss device just another victim? Sorry for the lengthy debug output, but I am glad I had the chance to collect the data. Regards, Kai. BTW: CC'ing fs@ as it might be fs-related. -------- dmesg: -------- FreeBSD 8.0-STABLE #10: Wed Mar 10 10:52:19 CET 2010 root@sonnenkraft.free.de:/usr/obj/usr/src/sys/DL385-G2 amd64 WARNING: WITNESS option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Quad-Core AMD Opteron(tm) Processor 2352 (2100.08-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f23 Stepping = 3 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x7ff TSC: P-state invariant real memory = 21474836480 (20480 MB) avail memory = 20685017088 (19726 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 ioapic0 irqs 0-15 on motherboard ioapic1 irqs 16-31 on motherboard ioapic2 irqs 32-47 on motherboard kbd1 at kbdmux0 cryptosoft0: on motherboard acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x920-0x923 on acpi0 acpi_hpet0: iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 900 pcib0: on acpi0 pci0: on pcib0 vgapci0: port 0x1000-0x10ff mem 0xe8000000-0xefffffff,0xf7ff0000-0xf7ffffff irq 44 at device 3.0 on pci0 pci0: at device 4.0 (no driver attached) pci0: at device 4.2 (no driver attached) uhci0: port 0x1800-0x181f irq 45 at device 4.4 on pci0 uhci0: [ITHREAD] usbus0: on uhci0 pci0: at device 4.6 (no driver attached) pcib1: at device 5.0 on pci0 pci1: on pcib1 pcib2: at device 13.0 on pci1 pci2: on pcib2 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x500-0x50f at device 6.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] isab0: at device 6.2 on pci0 isa0: on isab0 pcib3: irq 42 at device 15.0 on pci0 pci5: on pcib3 pcib4: irq 38 at device 16.0 on pci0 pci8: on pcib4 pcib5: irq 39 at device 17.0 on pci0 pci14: on pcib5 pcib6: irq 40 at device 18.0 on pci0 pci11: on pcib6 pcib7: irq 41 at device 19.0 on pci0 pci3: on pcib7 pcib8: at device 0.0 on pci3 pci4: on pcib8 bce0: mem 0xf8000000-0xf9ffffff irq 41 at device 0.0 on pci4 miibus0: on bce0 brgphy0: PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bce0: Ethernet address: 00:1b:78:38:dd:02 bce0: [ITHREAD] bce0: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (1.9.6); Flags (SPLT |MSI|MFW); MFW () pcib9: on acpi0 pci64: on pcib9 pcib10: irq 36 at device 15.0 on pci64 pci67: on pcib10 pcib11: irq 32 at device 16.0 on pci64 pci70: on pcib11 ciss0: port 0x4000-0x40ff mem 0xfdf00000-0xfdffffff,0xfdef0000-0xfdef0fff irq 32 at device 0.0 on pci70 ciss0: PERFORMANT Transport ciss0: [ITHREAD] pcib12: irq 33 at device 17.0 on pci64 pci73: on pcib12 pcib13: irq 34 at device 18.0 on pci64 pci65: on pcib13 pcib14: at device 0.0 on pci65 pci66: on pcib14 bce1: mem 0xfa000000-0xfbffffff irq 34 at device 0.0 on pci66 miibus1: on bce1 brgphy1: PHY 1 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-FDX, auto bce1: Ethernet address: 00:1b:78:38:dd:00 bce1: [ITHREAD] bce1: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C (1.9.6); Flags (SPLT |MSI|MFW); MFW () pcib15: irq 35 at device 19.0 on pci64 pci74: on pcib15 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 atrtc0: port 0x70-0x71 on acpi0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart0: console (9600,n,8,1) cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 cpu4: on acpi0 cpu5: on acpi0 cpu6: on acpi0 cpu7: on acpi0 orm0: at iomem 0xc0000-0xcafff,0xcb000-0xcefff,0xe5000-0xe6fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 ppc0: cannot reserve I/O port range uart1: at port 0x2f8-0x2ff irq 3 on isa0 uart1: [FILTER] Timecounters tick every 1.000 msec IPsec: Initialized Security Association Processing. usbus0: 12Mbps Full Speed USB v1.0 acd0: CDRW at ata0-master UDMA33 ugen0.1: <(0x103c)> at usbus0 uhub0: <(0x103c) UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0 uhub0: 2 ports with 2 removable, self powered ugen0.2: at usbus0 ukbd0: on usbus0 kbd2 at ukbd0 ums0: on usbus0 ums0: 3 buttons and [XY] coordinates ID=0 da0 at ciss0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 135.168MB/s transfers da0: Command Queueing enabled da0: 36863MB (75496320 512 byte sectors: 255H 32S/T 9252C) da1 at ciss0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 135.168MB/s transfers da1: Command Queueing enabled da1: 243098MB (497866080 512 byte sectors: 255H 32S/T 61013C) da2 at ciss0 bus 0 scbus0 target 2 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 135.168MB/s transfers da2: Command Queueing enabled da2: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) da3 at ciss0 bus 0 scbus0 target 3 lun 0 da3: Fixed Direct Access SCSI-5 device da3: 135.168MB/s transfers da3: Command Queueing enabled da3: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) da4 at ciss0 bus 0 scbus0 target 4 lun 0 da4: Fixed Direct Access SCSI-5 device da4: 135.168MB/s transfers da4: Command Queueing enabled da4: 139979MB (286677120 512 byte sectors: 255H 32S/T 35132C) SMP: AP CPU #3 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #5 Launched! WARNING: WITNESS option enabled, expect reduced performance. Trying to mount root from ufs:/dev/da0s1a WARNING: / was not properly dismounted ZFS filesystem version 3 ZFS storage pool version 14 -------- dmesg: END -------- ----- (0) "praeludium" ----- sysctl debug.kdb.panic=1 panic: kdb_sysctl_panic cpuid = 4 KDB: enter: panic [thread pid 68597 tid 100536 ] Stopped at kdb_enter+0x3d: movq $0,0x5246b0(%rip) ----- (1) "db> ps" ----- pid ppid pgrp uid state wmesg wchan cmd 68597 49133 68597 0 R+ CPU 4 sysctl 68596 68595 68595 22 SJ nanslp 0xffffffff808ca5c8 sshd 68595 1774 68595 0 SsJ sbwait 0xffffff0080fcce8c sshd 68593 6292 68593 65534 Ss ufs 0xffffff0243dc3098 inetd 68588 6292 68588 65534 Ss ufs 0xffffff0243dc3098 inetd 68587 6292 68587 65534 Ss ufs 0xffffff0243dc3098 inetd 68579 4427 4403 8 SJ zio->io_ 0xffffff02626da2b8 sh 68566 49896 68565 0 SE+ getblk 0xffffff819da189d8 grep 68564 49284 68564 0 SV+ getblk 0xffffff819da38be8 csh 68560 6292 68560 65534 Ss ufs 0xffffff0243dc3098 inetd 68559 68558 68558 22 SJ nanslp 0xffffffff808ca5c8 sshd 68558 5931 68558 0 SsJ sbwait 0xffffff0015b62694 sshd 68553 6292 68553 65534 Ss ufs 0xffffff0243dc3098 inetd 68552 6292 68552 65534 Ss ufs 0xffffff0243dc3098 inetd 68534 6292 68534 65534 Ss ufs 0xffffff0243dc3098 inetd 68532 68531 2558 82 SJ zio->io_ 0xffffff01ee145588 qmail-queue 68531 2583 2558 82 SJ wait 0xffffff01188398c0 qmail-smtpd 68527 68526 68526 22 SJ nanslp 0xffffffff808ca5c8 sshd 68526 5931 68526 0 SsJ sbwait 0xffffff03222e893c sshd 68525 6292 68525 65534 Ss ufs 0xffffff0243dc3098 inetd 68522 6292 68522 65534 Ss ufs 0xffffff0243dc3098 inetd 68521 6292 68521 65534 Ss getblk 0xffffff819da23ff8 python 68511 3778 3778 125 SJ kqread 0xffffff0118b26b00 proxymap 68510 3778 3778 0 SJ nanslp 0xffffffff808ca5c8 smtpd 68497 68495 49299 0 T+ biowr 0xffffff819da35f40 mktemp 68496 68494 49299 0 TE+ getblk 0xffffff819da38be8 tsort 68495 68494 49299 0 T+ piperd 0xffffff03b2025888 sh 68494 68493 49299 0 T+ sh 68493 68140 49299 0 T+ piperd 0xffffff01d7200b60 sh 68140 68139 49299 0 T+ make 68139 68138 49299 0 T+ sh 68138 68066 49299 0 T+ make 68066 66665 49299 0 T+ sh 67917 67915 67915 0 S zio->io_ 0xffffff0243466df8 zfs 67915 67913 67915 0 Ss pause 0xffffff014c8e7500 csh 67913 1399 67913 0 Ss select 0xffffff0226d7d4c0 sshd 67519 2051 2051 80 SJ select 0xffffff0080b0cdc0 httpd 66665 49405 49299 0 T+ make 49896 49816 49896 0 S+ pause 0xffffff011883e0a0 csh 49816 49531 49816 666 S+ wait 0xffffff011832e8c0 su 49531 49530 49531 666 Ss+ pause 0xffffff00696df0a0 tcsh 49530 49511 49511 666 S select 0xffffff0226f1b0c0 sshd 49511 1399 49511 0 Ss sbwait 0xffffff00a1106be4 sshd 49405 49299 49299 0 T+ make 49299 49284 49299 0 T+ make 49284 49283 49284 0 S+ ppwait 0xffffff04ed859450 csh 49283 49280 49283 666 S+ wait 0xffffff014cbac000 su 49280 49279 49280 666 Ss+ pause 0xffffff0007c6a0a0 tcsh 49279 49277 49277 666 S select 0xffffff0080990b40 sshd 49277 1399 49277 0 Ss sbwait 0xffffff01184e53ec sshd 49133 6335 49133 0 S+ pause 0xffffff03222870a0 csh 48183 2051 2051 80 SJ select 0xffffff04495163c0 httpd 47840 2051 2051 80 SJ select 0xffffff011813a7c0 httpd 47422 47420 47417 1001 SJ sbwait 0xffffff0118d9a3ec initial thread 47420 47419 47417 1001 SJ wait 0xffffff008083f000 bash 47419 47418 47417 1001 SJ wait 0xffffff04edcfc8c0 sh 47418 1 47417 0 SJ wait 0xffffff0322283000 atrun 47399 2051 2051 80 SJ select 0xffffff0449ab3440 httpd 46967 2051 2051 80 SJ select 0xffffff00a13a4240 httpd 46370 3778 3778 125 SJ kqread 0xffffff0322660700 pickup 46325 46323 46325 19420 S+J pause 0xffffff011883d0a0 screen 46323 46322 46323 19420 Ss+J wait 0xffffff014caf5460 bash 46322 46319 46319 19420 SJ select 0xffffff00a1184cc0 sshd 46319 3904 46319 0 SsJ sbwait 0xffffff014c9eb144 sshd 46245 2051 2051 80 SJ zio->io_ 0xffffff0007e0adf8 httpd 45112 2051 2051 80 SJ select 0xffffff0080b1c940 httpd 45014 2051 2051 80 SJ select 0xffffff008000e2c0 httpd 43851 4776 4776 80 SJ accept 0xffffff03b218685e httpd 43056 1752 1752 80 SJ select 0xffffff0080970cc0 httpd 43051 1752 1752 80 SJ select 0xffffff0080ca21c0 httpd 42965 1752 1752 80 SJ select 0xffffff03b20a6340 httpd 42955 1752 1752 80 SJ select 0xffffff02165c1a40 httpd 42939 1752 1752 80 SJ select 0xffffff014c475a40 httpd 42933 1752 1752 80 SJ select 0xffffff04331705c0 httpd 42520 1752 1752 80 SJ select 0xffffff0080b7abc0 httpd 42519 1752 1752 80 SJ lockf 0xffffff0226cde000 httpd 42326 1752 1752 80 SJ select 0xffffff00a1214340 httpd 42072 1752 1752 80 SJ kqread 0xffffff02160ae900 httpd 38848 2051 2051 80 SJ zfs 0xffffff021649e098 httpd 29104 3778 3778 125 SJ kqread 0xffffff0118d9dd00 qmgr 23197 5314 5314 80 SJ accept 0xffffff0449b73dae httpd 17921 17919 17921 19420 S+J select 0xffffff0253ada840 irssi 17919 17872 17919 19420 Ss+J wait 0xffffff0118839000 bash 17874 17872 17874 19420 Ss+J select 0xffffff0080ae80c0 initial thread 17872 1 17872 19420 SsJ select 0xffffff00a157acc0 screen 11037 11035 11037 19421 S+J select 0xffffff0080b7d8c0 irssi 11035 10729 11035 19421 Ss+J wait 0xffffff00696de460 bash 10731 10729 10731 19421 Ss+J select 0xffffff0071c5d140 initial thread 10729 1 10729 19421 SsJ select 0xffffff0423e9c9c0 screen 6589 2832 2832 80 SJ kqread 0xffffff01c0c0ae00 httpd 6335 1 6335 0 Ss+ wait 0xffffff0322284000 login 6334 1 6334 0 Ss+ ttyin 0xffffff00078d6ca8 getty 6333 1 6333 0 Ss+ ttyin 0xffffff00078d54a8 getty 6332 1 6332 0 Ss+ ttyin 0xffffff00078c4ca8 getty 6331 1 6331 0 Ss+ ttyin 0xffffff00078c48a8 getty 6330 1 6330 0 Ss+ ttyin 0xffffff00078d70a8 getty 6329 1 6329 0 Ss+ ttyin 0xffffff00078d78a8 getty 6328 1 6328 0 Ss+ ttyin 0xffffff00078d58a8 getty 6327 1 6327 0 Ss+ ttyin 0xffffff00078d84a8 getty 6314 1 6314 0 Ss select 0xffffff04edcac0c0 bsnmpd 6292 1 6292 0 Ss select 0xffffff04490cddc0 inetd 6258 1 6258 0 Ss nanslp 0xffffffff808ca5c8 cron 6251 1 6251 25 Ss pause 0xffffff04ed858500 sendmail 6247 1 6247 0 Ss select 0xffffff04eda4e9c0 sendmail 6221 1 6221 0 Ss (threaded) bacula-fd 100681 S ucond 0xffffff0449b6c100 bacula-fd 100154 S select 0xffffff04eda4ecc0 bacula-fd 6191 1 6191 0 SsJ nanslp 0xffffffff808ca5c8 cron 6182 1 6182 0 SsJ select 0xffffff04eda4f040 sshd 6072 1 6072 0 SsJ select 0xffffff04ed9987c0 syslogd 5948 5918 5918 80 SJ (threaded) httpd 100665 S lockf 0xffffff014c630000 httpd 100664 S ucond 0xffffff04ed656480 httpd 100663 S ucond 0xffffff04ed656400 httpd 100662 S ucond 0xffffff04ed656300 httpd 100661 S select 0xffffff00808aac40 httpd 100660 S ucond 0xffffff04ed656380 httpd 100659 S ucond 0xffffff04ed656280 httpd 100658 S ucond 0xffffff04ed656100 httpd 100657 S ucond 0xffffff04ed656000 httpd 100613 S ucond 0xffffff04ed656180 httpd 100656 S select 0xffffff00808ab3c0 httpd 100654 S ucond 0xffffff04ed672d80 httpd 100652 S ucond 0xffffff04ed672e00 httpd 100650 S ucond 0xffffff04ed672d00 httpd 100648 S ucond 0xffffff04ed672c80 httpd 100646 S ucond 0xffffff04ed672b80 httpd 100644 S ucond 0xffffff04ed672c00 httpd 100641 S ucond 0xffffff027de26680 httpd 100640 S ucond 0xffffff027de26600 httpd 100638 S ucond 0xffffff027de26580 httpd 100636 S ucond 0xffffff02a3988680 httpd 100635 S ucond 0xffffff02a3988700 httpd 100632 S ucond 0xffffff02a3988780 httpd 100630 S ucond 0xffffff0449b6ba00 httpd 100628 S ucond 0xffffff0449b6b900 httpd 100626 S ucond 0xffffff0449b6c000 httpd 100069 S piperd 0xffffff03b23d1b60 httpd 5947 5918 5918 80 SJ (threaded) httpd 100655 S kqread 0xffffff0007f75c00 httpd 100653 S ucond 0xffffff0449514700 httpd 100651 S ucond 0xffffff0449514680 httpd 100649 S ucond 0xffffff0449514480 httpd 100647 S ucond 0xffffff0449514500 httpd 100645 S ucond 0xffffff0449514900 httpd 100643 S ucond 0xffffff0449514880 httpd 100642 S select 0xffffff00800115c0 httpd 100639 S ucond 0xffffff00154ae100 httpd 100637 S ucond 0xffffff01eeccdd00 httpd 100634 S ucond 0xffffff020a053880 httpd 100633 S ucond 0xffffff0015f5f680 httpd 100631 S ucond 0xffffff020a00e680 httpd 100629 S ucond 0xffffff01eed29100 httpd 100627 S ucond 0xffffff01eed2aa80 httpd 100625 S ucond 0xffffff02a3988800 httpd 100624 S ucond 0xffffff02a3988880 httpd 100623 S ucond 0xffffff02a3988900 httpd 100622 S ucond 0xffffff02a3988980 httpd 100620 S ucond 0xffffff02a3988a00 httpd 100619 S ucond 0xffffff02a3988a80 httpd 100618 S ucond 0xffffff02a3988b00 httpd 100617 S ucond 0xffffff02a3988b80 httpd 100616 S ucond 0xffffff0449b6b800 httpd 100615 S ucond 0xffffff0449b6bd80 httpd 100614 S ucond 0xffffff0449b6bb80 httpd 100458 S piperd 0xffffff03b23d1b60 httpd 5940 1 5940 0 SsJ nanslp 0xffffffff808ca5c8 cron 5931 1 5931 0 SsJ select 0xffffff044961b740 sshd 5918 1 5918 0 SsJ select 0xffffff043390c6c0 httpd 5868 5867 5867 5121 SJ (threaded) python2.4 100701 S ucond 0xffffff02a304c280 python2.4 100700 S ucond 0xffffff02a304c300 python2.4 100699 S ucond 0xffffff02a304c400 python2.4 100698 S ucond 0xffffff026c4afa00 python2.4 100168 S select 0xffffff01c048c0c0 initial thread 5867 1 5867 0 SsJ select 0xffffff04ed045dc0 python2.4 5801 1 5801 0 SsJ select 0xffffff04ed0a0cc0 syslogd 5674 1 5674 0 SsJ nanslp 0xffffffff808ca5c8 cron 5665 1 5665 0 SsJ select 0xffffff0449e871c0 sshd 5639 5573 5570 88 S+J (threaded) mysqld 100612 S sigwait 0xffffff8243587a68 mysqld 100609 S ucond 0xffffff03b21af800 mysqld 100611 S select 0xffffff04494e22c0 mysqld 100610 S select 0xffffff03b2a623c0 mysqld 100608 S ucond 0xffffff03b2036580 mysqld 100607 S ucond 0xffffff03b21af580 mysqld 100606 S ucond 0xffffff03b21af600 mysqld 100605 S ucond 0xffffff03b21af880 mysqld 100577 S select 0xffffff0449e882c0 initial thread 5573 1 5570 88 S+J wait 0xffffff03fe7d08c0 sh 5470 1 5470 0 SsJ select 0xffffff0449d1e240 syslogd 5469 5314 5314 80 SJ accept 0xffffff0449b73dae httpd 5468 5314 5314 80 SJ accept 0xffffff0449b73dae httpd 5390 5389 5314 0 SJ select 0xffffff0449c0a040 ruby18 5389 5314 5314 0 SJ (threaded) ApplicationPoolServ 100783 S sbwait 0xffffff00a129fe8c ApplicationPoolServ 100782 S sbwait 0xffffff00718b9144 ApplicationPoolServ 100604 S fifoow 0xffffff0433f6b358 ApplicationPoolServ 100603 S ucond 0xffffff03fe5ddd00 ApplicationPoolServ 100225 S sbwait 0xffffff03b20873ec initial thread 5336 1 5336 0 SsJ nanslp 0xffffffff808ca5c8 cron 5327 1 5327 0 SsJ select 0xffffff0449b6b240 sshd 5314 1 5314 0 SsJ select 0xffffff0449b88640 httpd 5245 1 5245 0 SsJ accept 0xffffff03b230185e vsftpd 5176 1 5176 0 SsJ select 0xffffff0449ab2ac0 syslogd 5065 1 5065 0 SsJ select 0xffffff0423062a40 inetd 5051 1 5051 0 SsJ nanslp 0xffffffff808ca5c8 cron 5042 1 5042 0 SsJ select 0xffffff0423dc9440 sshd 4932 1 4932 0 SsJ select 0xffffff04495ff940 syslogd 4915 4776 4776 80 SJ accept 0xffffff03b218685e httpd 4914 4776 4776 80 SJ accept 0xffffff03b218685e httpd 4913 4776 4776 80 SJ accept 0xffffff03b218685e httpd 4912 4776 4776 80 SJ accept 0xffffff03b218685e httpd 4911 4776 4776 80 SJ accept 0xffffff03b218685e httpd 4807 1 4807 0 SsJ nanslp 0xffffffff808ca5c8 cron 4800 1 4800 25 SsJ pause 0xffffff03fe7cc500 sendmail 4796 1 4796 0 SsJ select 0xffffff044951d240 sendmail 4789 1 4789 0 SsJ select 0xffffff0423580440 sshd 4776 1 4776 0 SsJ select 0xffffff032295a1c0 httpd 4726 4725 4725 0 SJ (threaded) varnishd 100600 S ucond 0xffffff02a3988c80 cache-worker 100599 S ucond 0xffffff02a3988d00 cache-worker 100598 S ucond 0xffffff02a3988d80 cache-worker 100597 S ucond 0xffffff02a3988e00 cache-worker 100596 S ucond 0xffffff02a3989000 cache-worker 100595 S ucond 0xffffff02a3989080 cache-worker 100594 S ucond 0xffffff02a3989100 cache-worker 100593 S ucond 0xffffff02a3989180 cache-worker 100592 S ucond 0xffffff02a3989200 cache-worker 100591 S select 0xffffff0433697bc0 cache-acceptor 100590 S kqread 0xffffff0216dd2600 cache-kqueue 100589 S ucond 0xffffff02a3989280 cache-worker 100588 S nanslp 0xffffffff808ca5c8 cache-timeout 100587 S ucond 0xffffff03fe6be180 wrk_herder 100586 S nanslp 0xffffffff808ca5c8 wrk_herdtimer 100407 S select 0xffffff044951d7c0 cache-main 4725 1 4725 0 SsJ select 0xffffff0449124940 varnishd 4709 4708 4708 7001 SJ (threaded) python 100849 S ucond 0xffffff01185bb580 python 100848 S ucond 0xffffff00808bea00 python 100541 S select 0xffffff04331708c0 initial thread 4708 1 4708 0 SsJ select 0xffffff04493a9240 python 4628 1 4628 0 SsJ select 0xffffff03feda70c0 syslogd 4524 4426 4426 8 SJ select 0xffffff03b295f3c0 innfeed 4523 4426 4426 8 SJ sbwait 0xffffff03b2bbf694 perl 4512 1 4512 0 SsJ select 0xffffff03b2f8fbc0 inetd 4498 1 4498 0 SsJ nanslp 0xffffffff808ca5c8 cron 4492 1 4492 25 SsJ pause 0xffffff0322be4500 sendmail 4488 1 4488 0 SsJ select 0xffffff03fe6be6c0 sendmail 4481 1 4481 0 SsJ select 0xffffff03fe6a12c0 sshd 4428 1 4403 8 S+J nanslp 0xffffffff808ca5c8 perl 4427 1 4403 8 S+J wait 0xffffff03b2aee000 sh 4426 1 4426 8 SsJ zio->io_ 0xffffff01ee1832b8 innd 4340 1 4340 0 SsJ select 0xffffff03b29f4640 syslogd 4207 1 4207 0 SsJ nanslp 0xffffffff808ca5c8 cron 4201 1 4201 25 SsJ pause 0xffffff03b2af6500 sendmail 4197 1 4197 0 SsJ select 0xffffff03b2d0fbc0 sendmail 4185 1 4185 0 SsJ select 0xffffff03b2a3a240 sshd 4175 1 4175 0 SsJ select 0xffffff03b2b7b0c0 perl 4103 1 4103 53 SsJ (threaded) named 100529 S kqread 0xffffff01ee4afd00 named 100528 S ucond 0xffffff02a3057500 named 100527 S ucond 0xffffff02a36d8280 named 100526 S ucond 0xffffff027dd68a80 named 100525 S ucond 0xffffff02a36d8700 named 100524 S ucond 0xffffff02a36d8500 named 100523 S ucond 0xffffff02a36d8300 named 100522 S ucond 0xffffff02a36d8400 named 100521 S ucond 0xffffff02a36d8b80 named 100520 S ucond 0xffffff02a36d8900 named 100406 S sigwait 0xffffff8243181a68 named 4098 1 4098 0 SsJ select 0xffffff03b27f3640 syslogd 3947 3937 24 85 S+J piperd 0xffffff0015427888 qmail-clean 3946 3937 24 86 S+J select 0xffffff03b25846c0 qmail-rspawn 3945 3937 24 0 S+J select 0xffffff03229248c0 qmail-lspawn 3944 3937 24 83 S+J piperd 0xffffff02e77b2b60 splogger 3937 1 24 87 S+J select 0xffffff032278fa40 qmail-send 3930 1 3930 0 SsJ nanslp 0xffffffff808ca5c8 cron 3904 1 3904 0 SsJ select 0xffffff03b22f9440 sshd 3855 1 3855 0 SsJ select 0xffffff02a3a81cc0 iscsi-target 3843 1 3843 0 SsJ select 0xffffff0322f483c0 perl 3835 3806 3805 88 S+J (threaded) mysqld 100487 S sigwait 0xffffff8243316a68 mysqld 100484 S ucond 0xffffff0322cea900 mysqld 100486 S select 0xffffff03b20ae7c0 mysqld 100485 S select 0xffffff03b2236a40 mysqld 100483 S ucond 0xffffff02a36d8680 mysqld 100482 S ucond 0xffffff02a36d9880 mysqld 100481 S ucond 0xffffff03b21af500 mysqld 100480 S ucond 0xffffff03b203f480 mysqld 100397 S select 0xffffff03b2036a40 initial thread 3806 1 3805 88 S+J wait 0xffffff0322be5460 sh 3792 3791 3791 65534 SJ kqread 0xffffff0216eec000 nginx 3791 1 3791 0 SsJ pause 0xffffff027dfe80a0 nginx 3778 1 3778 0 SsJ kqread 0xffffff02e7f14300 master 3702 1 3702 0 SsJ select 0xffffff0322cebdc0 perl 3552 1 3552 0 SsJ select 0xffffff032283bd40 syslogd 3406 1 3406 0 SsJ nanslp 0xffffffff808ca5c8 cron 3397 1 3397 0 SsJ select 0xffffff03220466c0 sshd 3396 3391 3371 85 SJ piperd 0xffffff02e77b22d8 qmail-clean 3395 3391 3371 86 SJ select 0xffffff03222b7dc0 qmail-rspawn 3394 3391 3371 0 SJ select 0xffffff032225fbc0 qmail-lspawn 3393 3386 3371 83 SJ piperd 0xffffff01d720eb60 multilog 3392 3385 3371 82 SJ accept 0xffffff02a3cb4dae tcpserver 3391 3383 3371 87 SJ select 0xffffff02e768cdc0 qmail-send 3390 3384 3371 83 SJ piperd 0xffffff026ce4e888 multilog 3386 3373 3371 0 SJ select 0xffffff032225dac0 supervise 3385 3373 3371 0 SJ select 0xffffff02e77cb640 supervise 3384 3373 3371 0 SJ select 0xffffff03222b4440 supervise 3383 3373 3371 0 SJ select 0xffffff02a3fb79c0 supervise 3374 1 3371 0 SJ piperd 0xffffff02e77b05b0 readproctitle 3373 1 3371 0 SJ nanslp 0xffffffff808ca5c8 svscan 3274 1 3274 0 SsJ select 0xffffff02a35a67c0 syslogd 3250 3116 3116 80 SJ (threaded) httpd 100448 S accept 0xffffff02a3afc5b6 httpd 100447 S ucond 0xffffff02e70df400 httpd 100446 S ucond 0xffffff02e7d3e500 httpd 100445 S ucond 0xffffff02e7ecfa00 httpd 100444 S ucond 0xffffff02e7ecf980 httpd 100443 S ucond 0xffffff02e7ecf900 httpd 100442 S ucond 0xffffff02e7ecf880 httpd 100441 S ucond 0xffffff02e7ecf800 httpd 100440 S ucond 0xffffff02e7ecf780 httpd 100439 S ucond 0xffffff02e7ecf700 httpd 100438 S ucond 0xffffff02e7ecf680 httpd 100437 S ucond 0xffffff02e7ecf600 httpd 100436 S ucond 0xffffff02e7ecf580 httpd 100435 S ucond 0xffffff02e7ecf500 httpd 100434 S ucond 0xffffff02e7ecf480 httpd 100433 S ucond 0xffffff02e7ecf400 httpd 100432 S ucond 0xffffff02a30e4780 httpd 100431 S ucond 0xffffff02a317dc00 httpd 100430 S ucond 0xffffff02a317db80 httpd 100429 S ucond 0xffffff02a317db00 httpd 100428 S ucond 0xffffff02a317da80 httpd 100427 S ucond 0xffffff02a317da00 httpd 100426 S ucond 0xffffff02a317d980 httpd 100425 S ucond 0xffffff02a317d900 httpd 100424 S ucond 0xffffff02a317d880 httpd 100423 S ucond 0xffffff02a317d800 httpd 100404 S piperd 0xffffff02e77b0888 httpd 3146 1 3146 0 SsJ nanslp 0xffffffff808ca5c8 cron 3140 1 3140 25 SsJ pause 0xffffff0015bb30a0 sendmail 3136 1 3136 0 SsJ select 0xffffff02e76ab7c0 sendmail 3129 1 3129 0 SsJ select 0xffffff02e7993840 sshd 3116 1 3116 0 SsJ select 0xffffff02e7b68640 httpd 3061 1 3061 0 SsJ select 0xffffff02e769c8c0 proftpd 2998 1 2998 0 SsJ select 0xffffff02e7517740 syslogd 2864 1 2864 0 SsJ nanslp 0xffffffff808ca5c8 cron 2858 1 2858 25 SsJ pause 0xffffff027dfea0a0 sendmail 2854 1 2854 0 SsJ select 0xffffff02a3c6d2c0 sendmail 2853 2832 2832 80 SJ lockf 0xffffff014c4a8180 httpd 2852 2832 2832 80 SJ lockf 0xffffff00a18c6200 httpd 2845 1 2845 0 SsJ select 0xffffff02a3cd5e40 sshd 2832 1 2832 0 SsJ select 0xffffff02a3a8eb40 httpd 2804 1 2804 0 SsJ (threaded) nscd 100396 S kqread 0xffffff02a3928700 nscd 100395 S kqread 0xffffff02a3928700 nscd 100394 S kqread 0xffffff02a3928700 nscd 100393 S kqread 0xffffff02a3928700 nscd 100392 S kqread 0xffffff02a3928700 nscd 100391 S kqread 0xffffff02a3928700 nscd 100390 S kqread 0xffffff02a3928700 nscd 100389 S kqread 0xffffff02a3928700 nscd 100195 S uwait 0xffffff0007d50080 nscd 2722 1 2722 0 SsJ select 0xffffff02a3921440 syslogd 2593 1 2593 0 SsJ nanslp 0xffffffff808ca5c8 cron 2592 2587 2558 85 SJ piperd 0xffffff01d720e000 qmail-clean 2591 2587 2558 86 SJ select 0xffffff02a3284ac0 qmail-rspawn 2590 2587 2558 0 SJ select 0xffffff02a317a840 qmail-lspawn 2589 2576 2558 83 SJ piperd 0xffffff0007cbbb60 multilog 2587 2575 2558 87 SJ select 0xffffff02a3436c40 qmail-send 2586 2574 2558 83 SJ piperd 0xffffff0007cbb2d8 multilog 2583 2573 2558 82 SJ accept 0xffffff020ac94b06 tcpserver 2577 1 2577 0 SsJ select 0xffffff02a31d64c0 sshd 2576 2560 2558 0 SJ select 0xffffff025375fd40 supervise 2575 2560 2558 0 SJ select 0xffffff01c0abacc0 supervise 2574 2560 2558 0 SJ select 0xffffff02a30a1d40 supervise 2573 2560 2558 0 SJ select 0xffffff02a31d6540 supervise 2561 1 2558 0 SJ piperd 0xffffff0015b81000 readproctitle 2560 1 2558 0 SJ nanslp 0xffffffff808ca5c8 svscan 2478 2316 2316 80 SJ (threaded) httpd 100380 S lockf 0xffffff00a1214280 httpd 100379 S ucond 0xffffff027de26980 httpd 100378 S ucond 0xffffff027de26900 httpd 100377 S ucond 0xffffff027de26880 httpd 100376 S ucond 0xffffff027de26800 httpd 100375 S ucond 0xffffff027de26780 httpd 100374 S ucond 0xffffff027de26700 httpd 100373 S ucond 0xffffff0007c12e00 httpd 100372 S ucond 0xffffff01c0dfc400 httpd 100371 S ucond 0xffffff01c0dfc380 httpd 100370 S ucond 0xffffff020af7ad00 httpd 100328 S ucond 0xffffff020af7ad80 httpd 100369 S ucond 0xffffff020af7ae00 httpd 100367 S ucond 0xffffff02a30e4b80 httpd 100365 S ucond 0xffffff02a30e4b00 httpd 100362 S ucond 0xffffff02a30e4a80 httpd 100361 S select 0xffffff008064b9c0 httpd 100359 S ucond 0xffffff02a30e4980 httpd 100357 S select 0xffffff0036219e40 httpd 100355 S select 0xffffff0253d004c0 httpd 100353 S select 0xffffff007169e1c0 httpd 100350 S select 0xffffff007176a740 httpd 100347 S ucond 0xffffff020af7b000 httpd 100344 S select 0xffffff04492e2dc0 httpd 100341 S ucond 0xffffff0007d4f000 httpd 100339 S select 0xffffff0449517bc0 httpd 100327 S piperd 0xffffff0007cc7888 httpd 2477 2316 2316 80 SJ (threaded) httpd 100368 S kqread 0xffffff021697e800 httpd 100366 S ucond 0xffffff02a317d600 httpd 100364 S ucond 0xffffff02a317d580 httpd 100363 S ucond 0xffffff02a317d500 httpd 100360 S ucond 0xffffff00153bb380 httpd 100358 S ucond 0xffffff027dee0b80 httpd 100356 S ucond 0xffffff027dee2980 httpd 100354 S ucond 0xffffff027dee1500 httpd 100352 S ucond 0xffffff027dee1180 httpd 100351 S ucond 0xffffff027dee1100 httpd 100349 S ucond 0xffffff027dee0c00 httpd 100348 S ucond 0xffffff027dee3000 httpd 100346 S ucond 0xffffff027d416280 httpd 100345 S ucond 0xffffff027dee3100 httpd 100343 S ucond 0xffffff027d5bd900 httpd 100342 S select 0xffffff008064fb40 httpd 100340 S ucond 0xffffff027dee2b80 httpd 100338 S ucond 0xffffff027dee2e00 httpd 100337 S ucond 0xffffff027ded3200 httpd 100336 S ucond 0xffffff027ded0680 httpd 100334 S ucond 0xffffff027dcfd080 httpd 100333 S select 0xffffff00413a5840 httpd 100332 S ucond 0xffffff027dee1680 httpd 100331 S ucond 0xffffff027dee1980 httpd 100330 S ucond 0xffffff027dee1580 httpd 100329 S ucond 0xffffff027dee1a00 httpd 100326 S piperd 0xffffff0007cc7888 httpd 2459 1 2459 0 SsJ select 0xffffff027ded02c0 syslogd 2338 1 2338 0 SsJ nanslp 0xffffffff808ca5c8 cron 2329 1 2329 0 SsJ select 0xffffff027daa36c0 sshd 2316 1 2316 0 SsJ select 0xffffff027dbb8740 httpd 2267 2266 2266 5201 SJ (threaded) python2.4 100583 S ucond 0xffffff027de26c80 python2.4 100582 S ucond 0xffffff027de26c00 python2.4 100581 S ucond 0xffffff027de26b80 python2.4 100580 S ucond 0xffffff027de26b00 python2.4 100183 S select 0xffffff0423bd6840 initial thread 2266 1 2266 0 SsJ select 0xffffff026ccaa940 python2.4 2205 2051 2051 80 SJ zfs 0xffffff021649e098 httpd 2200 1 2200 0 SsJ select 0xffffff026ce6b5c0 syslogd 2073 1 2073 0 SsJ nanslp 0xffffffff808ca5c8 cron 2064 1 2064 0 SsJ select 0xffffff026c82a9c0 sshd 2051 1 2051 0 SsJ select 0xffffff026caf0cc0 httpd 2001 2000 2000 5351 SJ (threaded) python 100585 S ucond 0xffffff03fe6be200 python 100584 S ucond 0xffffff03b2f8f700 python 100243 S select 0xffffff044910f640 initial thread 2000 1 2000 0 SsJ select 0xffffff00153423c0 python 1986 1985 1985 5301 SJ (threaded) python 100567 D zio->io_ 0xffffff0007dc8588 python 100566 S ucond 0xffffff02a3989380 python 100249 S select 0xffffff03b2f8f640 initial thread 1985 1 1985 0 SsJ select 0xffffff0243fbc540 python 1921 1 1921 0 SsJ select 0xffffff0015cd5d40 syslogd 1785 1 1785 0 SsJ nanslp 0xffffffff808ca5c8 cron 1774 1 1774 0 SsJ select 0xffffff020aadd7c0 sshd 1773 1752 1752 0 SJ piperd 0xffffff0007cb72d8 initial thread 1772 1752 1752 0 SJ piperd 0xffffff01c05fa5b0 initial thread 1771 1752 1752 0 SJ piperd 0xffffff01c05fa2d8 initial thread 1769 1752 1752 0 SJ piperd 0xffffff01c05fa000 initial thread 1768 1752 1752 0 SJ piperd 0xffffff01c05f9b60 initial thread 1767 1752 1752 0 SJ piperd 0xffffff01c05f9888 initial thread 1752 1 1752 0 SsJ select 0xffffff020aa0e3c0 httpd 1688 1687 1687 5241 SJ (threaded) python 100261 S ucond 0xffffff0007d4f200 python 100260 S ucond 0xffffff0007d4f180 python 100181 S select 0xffffff026c2aa7c0 initial thread 1687 1 1687 0 SsJ select 0xffffff01d7db7640 python 1679 1678 1678 5221 SJ (threaded) python2.4 100253 S ucond 0xffffff0015479180 python2.4 100252 S ucond 0xffffff00154ae400 python2.4 100177 S select 0xffffff01eef34b40 initial thread 1678 1 1678 0 SsJ select 0xffffff01c0625840 python2.4 1674 1673 1673 5141 SJ (threaded) python2.4 100259 S ucond 0xffffff020af7b280 python2.4 100258 S ucond 0xffffff020af7b300 python2.4 100169 S select 0xffffff0216c249c0 initial thread 1673 1 1673 0 SsJ select 0xffffff01c0ab9a40 python2.4 1669 1668 1668 5161 SJ (threaded) python2.4 100255 S ucond 0xffffff00154ae700 python2.4 100254 S ucond 0xffffff00154ae380 python2.4 100166 S select 0xffffff020ab9a340 initial thread 1668 1 1668 0 SsJ select 0xffffff01c06231c0 python2.4 1664 1663 1663 5181 SJ (threaded) python2.4 100257 S ucond 0xffffff020af7c100 python2.4 100256 S ucond 0xffffff020af7c000 python2.4 100171 S select 0xffffff01d7a133c0 initial thread 1663 1 1663 0 SsJ select 0xffffff01c048bdc0 python2.4 1592 1 1592 0 SsJ select 0xffffff0015f57140 syslogd 1399 1 1399 0 Ss select 0xffffff0015465dc0 sshd 1380 1 1380 1 Ss sbwait 0xffffff0015c0e694 rwhod 1344 1 1344 0 Ss select 0xffffff0015340dc0 ntpd 1255 1254 1254 0 S (threaded) nfsd 100165 S rpcsvc 0xffffff0007e99b20 nfsd: service 100164 S rpcsvc 0xffffff0007e99ba0 nfsd: service 100163 S rpcsvc 0xffffff0007e99c20 nfsd: service 100066 S rpcsvc 0xffffff001530ce20 nfsd: master 1254 1 1254 0 Ss accept 0xffffff0015b875b6 nfsd 1252 1 1252 0 Ss select 0xffffff0015340040 mountd 1132 1 1132 0 Ss select 0xffffff0007c10c40 rpcbind 1109 1 1109 0 Ss select 0xffffff0007f79040 syslogd 936 1 936 0 Ss select 0xffffff0015340a40 devd 903 1 903 0 Ss select 0xffffff0015465840 moused 632 1 632 0 Ss select 0xffffff0015465c40 ospfd 626 1 626 0 Ss select 0xffffff0007e9b3c0 zebra 43 0 0 0 SL (threaded) zfskern 100112 D zio->io_ 0xffffff024351e2b8 [txg_thread_enter] 100111 D tx->tx_q 0xffffff0007e2aa60 [txg_thread_enter] 100109 D vgeom:io 0xffffff0007f60a10 [vdev da3] 100108 D vgeom:io 0xffffff0007f60b90 [vdev da2] 100081 D l2arc_fe 0xffffffff80f11640 [l2arc_feed_thread] 100070 D arc_recl 0xffffffff80f09060 [arc_reclaim_thread] 23 0 0 0 RL [flowcleaner] 22 0 0 0 SL sdflush 0xffffffff80aa5518 [softdepflush] 21 0 0 0 SL vlruwt 0xffffff0007b91460 [vnlru] 20 0 0 0 SL zio->io_ 0xffffff026271fb28 [syncer] 19 0 0 0 SL psleep 0xffffffff80a8b648 [bufdaemon] 18 0 0 0 SL pgzero 0xffffffff80aa6fac [pagezero] 17 0 0 0 SL psleep 0xffffffff80aa6348 [vmdaemon] 16 0 0 0 SL psleep 0xffffffff80aa630c [pagedaemon] 15 0 0 0 SL jsw:wait 0xffffffff808c6a90 [g_journal switcher] 9 0 0 0 SL ccb_scan 0xffffffff808aa160 [xpt_thrd] 8 0 0 0 SL pftm 0xffffffff801d43a0 [pfpurge] 7 0 0 0 SL idle 0xffffff80005ec0f0 [ciss_notify0] 14 0 0 0 SL (threaded) usb 100042 D - 0xffffff80005d6ef0 [usbus0] 100041 D - 0xffffff80005d6e98 [usbus0] 100040 D - 0xffffff80005d6e40 [usbus0] 100039 D - 0xffffff80005d6de8 [usbus0] 6 0 0 0 SL crypto_r 0xffffffff80aa4340 [crypto returns] 5 0 0 0 SL crypto_w 0xffffffff80aa4300 [crypto] 13 0 0 0 SL - 0xffffffff808ca2a4 [yarrow] 4 0 0 0 LL *cissmtx 0xffffff04ed820c00 [g_down] 3 0 0 0 SL - 0xffffffff808c6a00 [g_up] 2 0 0 0 SL - 0xffffffff808c69f0 [g_event] 12 0 0 0 RL (threaded) intr 100051 Run CPU 5 [swi0: uart uart] 100050 I [irq12: psm0] 100049 I [irq1: atkbd0] 100048 I [irq258: bce1] 100046 L *cissmtx 0xffffff04ed820c00 [irq257: ciss0] 100045 I [irq256: bce0] 100044 I [irq15: ata1] 100043 I [irq14: ata0] 100038 I [irq45: uhci0] 100037 I [irq9: acpi0] 100035 I [swi6: task queue] 100034 I [swi6: Giant taskq] 100032 I [swi5: +] 100028 Run CPU 2 [swi2: cambio] 100020 I [swi4: clock] 100019 I [swi4: clock] 100018 I [swi4: clock] 100017 I [swi4: clock] 100016 I [swi4: clock] 100015 I [swi4: clock] 100014 I [swi4: clock] 100013 L *cissmtx 0xffffff04ed820c00 [swi4: clock] 100012 I [swi3: vm] 100011 I [swi1: netisr 0] 11 0 0 0 RL (threaded) idle 100010 Run CPU 0 [idle: cpu0] 100009 Run CPU 1 [idle: cpu1] 100008 CanRun [idle: cpu2] 100007 Run CPU 3 [idle: cpu3] 100006 CanRun [idle: cpu4] 100005 CanRun [idle: cpu5] 100004 Run CPU 6 [idle: cpu6] 100003 Run CPU 7 [idle: cpu7] 1 0 1 0 SLs wait 0xffffff00073468c0 [init] 10 0 0 0 SL audit_wo 0xffffffff80aa4870 [audit] 0 0 0 0 SLs (threaded) kernel 100131 D - 0xffffff00153f5500 [zil_clean] 100130 D - 0xffffff00153f5c80 [zil_clean] 100129 D - 0xffffff001538db80 [zil_clean] 100128 D - 0xffffff00153ba300 [zil_clean] 100127 D - 0xffffff00153bab80 [zil_clean] 100126 D - 0xffffff00153bb480 [zil_clean] 100125 D - 0xffffff00153bbb00 [zil_clean] 100124 D - 0xffffff00153bc400 [zil_clean] 100123 D - 0xffffff00153bd480 [zil_clean] 100122 D - 0xffffff001538c400 [zil_clean] 100121 D - 0xffffff0015343c00 [zil_clean] 100120 D - 0xffffff001538a500 [zil_clean] 100119 D - 0xffffff001538ad80 [zil_clean] 100118 D - 0xffffff001538b880 [zil_clean] 100117 D - 0xffffff001538c280 [zil_clean] 100116 D - 0xffffff001538ca80 [zil_clean] 100115 D - 0xffffff0015082600 [zil_clean] 100114 D - 0xffffff00150e9280 [zil_clean] 100110 D - 0xffffff0007f60080 [zfs_vn_rele_taskq] 100107 D - 0xffffff0007f60d00 [spa_zio] 100106 D - 0xffffff0007f60d80 [spa_zio] 100105 D - 0xffffff0007f60e00 [spa_zio] 100104 D - 0xffffff0007f61000 [spa_zio] 100103 D - 0xffffff0007f61080 [spa_zio] 100102 D - 0xffffff0007f61100 [spa_zio] 100101 D - 0xffffff0007f61180 [spa_zio] 100100 D - 0xffffff0007f61200 [spa_zio_7] 100099 D - 0xffffff0007f61200 [spa_zio_6] 100098 D - 0xffffff0007f61200 [spa_zio_5] 100097 D - 0xffffff0007f61200 [spa_zio_4] 100096 D - 0xffffff0007f61200 [spa_zio_3] 100095 D - 0xffffff0007f61200 [spa_zio_2] 100094 D - 0xffffff0007f61200 [spa_zio_1] 100093 D - 0xffffff0007f61200 [spa_zio_0] 100092 D - 0xffffff0007f61280 [spa_zio_7] 100091 D - 0xffffff0007f61280 [spa_zio_6] 100090 D - 0xffffff0007f61280 [spa_zio_5] 100089 D - 0xffffff0007f61280 [spa_zio_4] 100088 D - 0xffffff0007f61280 [spa_zio_3] 100087 D - 0xffffff0007f61280 [spa_zio_2] 100086 D - 0xffffff0007f61280 [spa_zio_1] 100085 D - 0xffffff0007f61280 [spa_zio_0] 100084 D - 0xffffff0007f61300 [spa_zio] 100083 D - 0xffffff0007f61380 [spa_zio] 100082 D - 0xffffff0007f61e00 [spa_zio] 100036 D - 0xffffff000742ac00 [kqueue taskq] 100033 D - 0xffffff000742b000 [thread taskq] 100031 D - 0xffffff000742b180 [acpi_task_2] 100030 D - 0xffffff000742b180 [acpi_task_1] 100029 D - 0xffffff000742b180 [acpi_task_0] 100024 D - 0xffffff0007343400 [firmware taskq] 100000 D sched 0xffffffff808c6ba0 [swapper] ----- (2) "db> bt" ----- Tracing pid 68597 tid 100536 td 0xffffff02a31a63a0 kdb_enter() at kdb_enter+0x3d panic() at panic+0x17b kdb_trap() at kdb_trap sysctl_root() at sysctl_root+0x113 userland_sysctl() at userland_sysctl+0x158 __sysctl() at __sysctl+0xaa syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (202, FreeBSD ELF64, __sysctl), rip = 0x80073cb6c, rsp = 0x7fffffffe248, rbp = 0x7fffffffe31c --- ----- (3) "db> show alllocks" ----- Process 68597 (sysctl) thread 0xffffff02a31a63a0 (100536) exclusive sleep mutex Giant (Giant) r = 0 (0xffffffff808c99a0) locked @ /usr/src/sys/kern/kern_sysctl.c:1417 shared sx sysctl lock (sysctl lock) r = 0 (0xffffffff808ca320) locked @ /usr/src/sys/kern/kern_sysctl.c:1521 Process 68595 (sshd) thread 0xffffff021610c740 (100207) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0080fcce40) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 68593 (inetd) thread 0xffffff0015b753a0 (100213) shared lockmgr ufs (ufs) r = 0 (0xffffff00153f77e8) locked @ /usr/src/sys/kern/vfs_subr.c:2091 Process 68588 (inetd) thread 0xffffff02269333a0 (100878) shared lockmgr ufs (ufs) r = 0 (0xffffff00153f77e8) locked @ /usr/src/sys/kern/vfs_subr.c:2091 Process 68587 (inetd) thread 0xffffff04ed9733a0 (100758) shared lockmgr ufs (ufs) r = 0 (0xffffff00153f77e8) locked @ /usr/src/sys/kern/vfs_subr.c:2091 Process 68579 (sh) thread 0xffffff04ed96aae0 (100764) exclusive lockmgr zfs (zfs) r = 0 (0xffffff01d79247e8) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1212 Process 68566 (grep) thread 0xffffff0015bb73a0 (100217) exclusive lockmgr ufs (ufs) r = 0 (0xffffff00153f7578) locked @ /usr/src/sys/kern/vfs_subr.c:2204 Process 68564 (csh) thread 0xffffff04ed974740 (100672) exclusive lockmgr ufs (ufs) r = 0 (0xffffff01ee7a6578) locked @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 shared lockmgr ufs (ufs) r = 0 (0xffffff00153f77e8) locked @ /usr/src/sys/kern/vfs_subr.c:2091 Process 68560 (inetd) thread 0xffffff0041398000 (100709) shared lockmgr ufs (ufs) r = 0 (0xffffff00153f77e8) locked @ /usr/src/sys/kern/vfs_subr.c:2091 Process 68558 (sshd) thread 0xffffff0007c6f3a0 (100141) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0015b62648) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 68553 (inetd) thread 0xffffff02fb700000 (100891) shared lockmgr ufs (ufs) r = 0 (0xffffff00153f77e8) locked @ /usr/src/sys/kern/vfs_subr.c:2091 Process 68552 (inetd) thread 0xffffff020ab44ae0 (100250) shared lockmgr ufs (ufs) r = 0 (0xffffff00153f77e8) locked @ /usr/src/sys/kern/vfs_subr.c:2091 Process 68534 (inetd) thread 0xffffff0015ebc000 (100186) shared lockmgr ufs (ufs) r = 0 (0xffffff00153f77e8) locked @ /usr/src/sys/kern/vfs_subr.c:2091 Process 68532 (qmail-queue) thread 0xffffff0080909ae0 (100816) shared lockmgr zfs (zfs) r = 0 (0xffffff02260eb308) locked @ /usr/src/sys/kern/vfs_syscalls.c:3508 Process 68526 (sshd) thread 0xffffff0449bcf000 (100687) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff03222e88f0) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 68525 (inetd) thread 0xffffff01180d1ae0 (100882) shared lockmgr ufs (ufs) r = 0 (0xffffff00153f77e8) locked @ /usr/src/sys/kern/vfs_subr.c:2091 Process 68522 (inetd) thread 0xffffff00a11db740 (100853) shared lockmgr ufs (ufs) r = 0 (0xffffff00153f77e8) locked @ /usr/src/sys/kern/vfs_subr.c:2091 Process 68521 (python) thread 0xffffff02fb58b740 (100897) exclusive lockmgr ufs (ufs) r = 0 (0xffffff0243dc3098) locked @ /usr/src/sys/kern/vfs_subr.c:2204 Process 68497 (mktemp) thread 0xffffff026ce6f3a0 (100289) exclusive lockmgr bufwait (bufwait) r = 0 (0xffffff819da35fd8) locked @ /usr/src/sys/kern/vfs_bio.c:2559 exclusive lockmgr ufs (ufs) r = 0 (0xffffff04334ed098) locked @ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1465 exclusive lockmgr ufs (ufs) r = 0 (0xffffff0015b467e8) locked @ /usr/src/sys/kern/vfs_subr.c:2091 Process 68496 (tsort) thread 0xffffff03fe6cc000 (100715) exclusive lockmgr ufs (ufs) r = 0 (0xffffff024d04b578) locked @ /usr/src/sys/kern/vfs_subr.c:2204 Process 49511 (sshd) thread 0xffffff008090f000 (100789) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff00a1106b98) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 49277 (sshd) thread 0xffffff0080907ae0 (100824) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff01184e53a0) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 47422 (mplayer) thread 0xffffff020ab103a0 (100235) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff0118d9a3a0) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 46319 (sshd) thread 0xffffff03fe6cdae0 (100559) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff014c9eb0f8) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 46245 (httpd) thread 0xffffff008090d740 (100798) exclusive lockmgr zfs (zfs) r = 0 (0xffffff021649e098) locked @ /usr/src/sys/kern/vfs_vnops.c:607 Process 5389 (ApplicationPoolServ) thread 0xffffff0041391740 (100783) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff00a129fe40) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 5389 (ApplicationPoolServ) thread 0xffffff0041391ae0 (100782) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff00718b90f8) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 5389 (ApplicationPoolServ) thread 0xffffff0007d6c3a0 (100225) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff03b20873a0) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 4523 (perl) thread 0xffffff03b2dcb000 (100570) exclusive sx so_rcv_sx (so_rcv_sx) r = 0 (0xffffff03b2bbf648) locked @ /usr/src/sys/kern/uipc_sockbuf.c:148 Process 4426 (innd) thread 0xffffff03229c9ae0 (100517) exclusive lockmgr zfs (zfs) r = 0 (0xffffff024dfaacc8) locked @ /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1212 Process 1986 (python) thread 0xffffff02a39fd740 (100567) shared lockmgr zfs (zfs) r = 0 (0xffffff0216f0da58) locked @ /usr/src/sys/kern/vfs_vnops.c:536 Process 20 (syncer) thread 0xffffff0007b993a0 (100059) exclusive lockmgr syncer (syncer) r = 0 (0xffffff00153b97e8) locked @ /usr/src/sys/kern/vfs_subr.c:1701 Process 12 (intr) thread 0xffffff000739c3a0 (100028) exclusive sleep mutex cissmtx (cissmtx) r = 0 (0xffffff80005ec148) locked @ /usr/src/sys/cam/cam_xpt.c:4732 ----- (4) "db> show allchains" ----- chain 1: thread 100023 (pid 4, g_down) blocked on lock 0xffffff80005ec148 (sleep mutex) "cissmtx" thread 100028 (pid 12, swi2: cambio) running on CPU 2 chain 2: thread 100046 (pid 12, irq257: ciss0) blocked on lock 0xffffff80005ec148 (sleep mutex) "cissmtx" thread 100028 (pid 12, swi2: cambio) running on CPU 2 chain 3: thread 100013 (pid 12, swi4: clock) blocked on lock 0xffffff80005ec148 (sleep mutex) "cissmtx" thread 100028 (pid 12, swi2: cambio) running on CPU 2 ----- (5) "db> show lockedvnods" ----- Locked vnodes 0xffffff0015b46750: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags () v_object 0xffffff0015483a20 ref 0 pages 1 lock type ufs: EXCL by thread 0xffffff026ce6f3a0 (pid 68497) #0 0xffffffff8037fb3f at __lockmgr_args+0x4ef #1 0xffffffff805ae38c at ffs_lock+0x9c #2 0xffffffff806405cb at VOP_LOCK1_APV+0x9b #3 0xffffffff80434a87 at _vn_lock+0x57 #4 0xffffffff8042816b at vget+0x7b #5 0xffffffff80415690 at cache_lookup+0x4e0 #6 0xffffffff804159b0 at vfs_cache_lookup+0xc0 #7 0xffffffff806414a7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff8041c633 at lookup+0x3d3 #9 0xffffffff8041d5d9 at namei+0x479 #10 0xffffffff804342c2 at vn_open_cred+0xc2 #11 0xffffffff80432679 at kern_openat+0x179 #12 0xffffffff80604ea2 at syscall+0x102 #13 0xffffffff805eade1 at Xfast_syscall+0xe1 ino 47109, on dev da0s1a 0xffffff04334ed000: tag ufs, type VREG usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type ufs: EXCL by thread 0xffffff026ce6f3a0 (pid 68497) #0 0xffffffff8037fdc7 at __lockmgr_args+0x777 #1 0xffffffff805a987f at ffs_vgetf+0x1bf #2 0xffffffff8058f474 at ffs_valloc+0x174 #3 0xffffffff805b950c at ufs_makeinode+0x8c #4 0xffffffff80641333 at VOP_CREATE_APV+0xb3 #5 0xffffffff80434682 at vn_open_cred+0x482 #6 0xffffffff80432679 at kern_openat+0x179 #7 0xffffffff80604ea2 at syscall+0x102 #8 0xffffffff805eade1 at Xfast_syscall+0xe1 ino 49547, on dev da0s1a 0xffffff00153f7750: tag ufs, type VDIR usecount 10, writecount 0, refcount 13 mountedhere 0 flags () v_object 0xffffff0007cff6c0 ref 0 pages 2 lock type ufs: SHARED (count 10) #0 0xffffffff8037fb3f at __lockmgr_args+0x4ef #1 0xffffffff805ae38c at ffs_lock+0x9c #2 0xffffffff806405cb at VOP_LOCK1_APV+0x9b #3 0xffffffff80434a87 at _vn_lock+0x57 #4 0xffffffff80435665 at vn_close+0xc5 #5 0xffffffff804357aa at vn_closefile+0x5a #6 0xffffffff8035f113 at _fdrop+0x23 #7 0xffffffff80360b7b at closef+0x5b #8 0xffffffff80361160 at kern_close+0x110 #9 0xffffffff80604ea2 at syscall+0x102 #10 0xffffffff805eade1 at Xfast_syscall+0xe1 ino 32897, on dev da0s1e 0xffffff00153f74e0: tag ufs, type VREG usecount 0, writecount 0, refcount 2 mountedhere 0 flags (VI_DOINGINACT) v_object 0xffffff001535bbd0 ref 0 pages 21 lock type ufs: EXCL by thread 0xffffff0015bb73a0 (pid 68566) #0 0xffffffff8037fdc7 at __lockmgr_args+0x777 #1 0xffffffff805ae38c at ffs_lock+0x9c #2 0xffffffff806405cb at VOP_LOCK1_APV+0x9b #3 0xffffffff80434a87 at _vn_lock+0x57 #4 0xffffffff8042942e at vputx+0x2fe #5 0xffffffff8036be85 at exit1+0x415 #6 0xffffffff8036caae at sys_exit+0xe #7 0xffffffff80604ea2 at syscall+0x102 #8 0xffffffff805eade1 at Xfast_syscall+0xe1 ino 33428, on dev da0s1e 0xffffff0243dc3000: tag ufs, type VREG usecount 0, writecount 0, refcount 11 mountedhere 0 flags (VI_DOINGINACT) v_object 0xffffff00413156c0 ref 0 pages 3 lock type ufs: EXCL by thread 0xffffff02fb58b740 (pid 68521) with exclusive waiters pending #0 0xffffffff8037fdc7 at __lockmgr_args+0x777 #1 0xffffffff805ae38c at ffs_lock+0x9c #2 0xffffffff806405cb at VOP_LOCK1_APV+0x9b #3 0xffffffff80434a87 at _vn_lock+0x57 #4 0xffffffff8042942e at vputx+0x2fe #5 0xffffffff8036a4bf at kern_execve+0xbbf #6 0xffffffff8036aded at execve+0x3d #7 0xffffffff80604ea2 at syscall+0x102 #8 0xffffffff805eade1 at Xfast_syscall+0xe1 ino 35212, on dev da0s1e 0xffffff024d04b4e0: tag ufs, type VREG usecount 0, writecount 0, refcount 2 mountedhere 0 flags (VI_DOINGINACT) v_object 0xffffff024d370a20 ref 0 pages 3 lock type ufs: EXCL by thread 0xffffff03fe6cc000 (pid 68496) #0 0xffffffff8037fdc7 at __lockmgr_args+0x777 #1 0xffffffff805ae38c at ffs_lock+0x9c #2 0xffffffff806405cb at VOP_LOCK1_APV+0x9b #3 0xffffffff80434a87 at _vn_lock+0x57 #4 0xffffffff8042942e at vputx+0x2fe #5 0xffffffff8036be85 at exit1+0x415 #6 0xffffffff8039758f at sigexit+0x8f #7 0xffffffff80399660 at cursig+0 #8 0xffffffff803d3019 at ast+0x1a9 #9 0xffffffff805eba19 at doreti_ast+0x1f ino 35382, on dev da0s1e 0xffffff01ee7a64e0: tag ufs, type VNON usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type ufs: EXCL by thread 0xffffff04ed974740 (pid 68564) #0 0xffffffff8037fdc7 at __lockmgr_args+0x777 #1 0xffffffff805a987f at ffs_vgetf+0x1bf #2 0xffffffff805b41a6 at ufs_lookup_+0xb46 #3 0xffffffff8063f24f at VOP_CACHEDLOOKUP_APV+0xaf #4 0xffffffff804159e0 at vfs_cache_lookup+0xf0 #5 0xffffffff806414a7 at VOP_LOOKUP_APV+0xb7 #6 0xffffffff8041c633 at lookup+0x3d3 #7 0xffffffff8041d5d9 at namei+0x479 #8 0xffffffff80369ba3 at kern_execve+0x2a3 #9 0xffffffff8036aded at execve+0x3d #10 0xffffffff80604ea2 at syscall+0x102 #11 0xffffffff805eade1 at Xfast_syscall+0xe1 ino 35352, on dev da0s1e 0xffffff00153b9750: tag syncer, type VNON usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type syncer: EXCL by thread 0xffffff0007b993a0 (pid 20) #0 0xffffffff8037fdc7 at __lockmgr_args+0x777 #1 0xffffffff80417b79 at vop_stdlock+0x39 #2 0xffffffff806405cb at VOP_LOCK1_APV+0x9b #3 0xffffffff80434a87 at _vn_lock+0x57 #4 0xffffffff80429a30 at sync_vnode+0x130 #5 0xffffffff80429cbc at sched_sync+0x1cc #6 0xffffffff8036d91a at fork_exit+0x12a #7 0xffffffff805eafde at fork_trampoline+0xe 0xffffff024dfaac30: tag zfs, type VDIR usecount 1, writecount 0, refcount 1 mountedhere 0 flags () lock type zfs: EXCL by thread 0xffffff03229c9ae0 (pid 4426) #0 0xffffffff8037fb3f at __lockmgr_args+0x4ef #1 0xffffffff80417b79 at vop_stdlock+0x39 #2 0xffffffff806405cb at VOP_LOCK1_APV+0x9b #3 0xffffffff80434a87 at _vn_lock+0x57 #4 0xffffffff80ea2e22 at zfs_lookup+0x2c2 #5 0xffffffff80ea3c81 at zfs_freebsd_lookup+0x81 #6 0xffffffff8063f24f at VOP_CACHEDLOOKUP_APV+0xaf #7 0xffffffff804159e0 at vfs_cache_lookup+0xf0 #8 0xffffffff806414a7 at VOP_LOOKUP_APV+0xb7 #9 0xffffffff8041c633 at lookup+0x3d3 #10 0xffffffff8041d5d9 at namei+0x479 #11 0xffffffff804342c2 at vn_open_cred+0xc2 #12 0xffffffff80432679 at kern_openat+0x179 #13 0xffffffff80604ea2 at syscall+0x102 #14 0xffffffff805eade1 at Xfast_syscall+0xe1 0xffffff01d7924750: tag zfs, type VREG usecount 2, writecount 0, refcount 3 mountedhere 0 flags (VV_TEXT) v_object 0xffffff026c5d0e58 ref 1 pages 1 lock type zfs: EXCL by thread 0xffffff04ed96aae0 (pid 68579) #0 0xffffffff8037fdc7 at __lockmgr_args+0x777 #1 0xffffffff80417b79 at vop_stdlock+0x39 #2 0xffffffff806405cb at VOP_LOCK1_APV+0x9b #3 0xffffffff80434a87 at _vn_lock+0x57 #4 0xffffffff80ea2e22 at zfs_lookup+0x2c2 #5 0xffffffff80ea3c81 at zfs_freebsd_lookup+0x81 #6 0xffffffff8063f24f at VOP_CACHEDLOOKUP_APV+0xaf #7 0xffffffff804159e0 at vfs_cache_lookup+0xf0 #8 0xffffffff806414a7 at VOP_LOOKUP_APV+0xb7 #9 0xffffffff8041c633 at lookup+0x3d3 #10 0xffffffff8041d5d9 at namei+0x479 #11 0xffffffff80369ba3 at kern_execve+0x2a3 #12 0xffffffff8036aded at execve+0x3d #13 0xffffffff80604ea2 at syscall+0x102 #14 0xffffffff805eade1 at Xfast_syscall+0xe1 0xffffff02260eb270: tag zfs, type VREG usecount 1, writecount 1, refcount 1 mountedhere 0 flags () v_object 0xffffff026ca3bbd0 ref 0 pages 0 lock type zfs: SHARED (count 1) #0 0xffffffff8037fb3f at __lockmgr_args+0x4ef #1 0xffffffff80417b79 at vop_stdlock+0x39 #2 0xffffffff806405cb at VOP_LOCK1_APV+0x9b #3 0xffffffff80434a87 at _vn_lock+0x57 #4 0xffffffff80433880 at fsync+0xc0 #5 0xffffffff80604ea2 at syscall+0x102 #6 0xffffffff805eade1 at Xfast_syscall+0xe1 0xffffff021649e000: tag zfs, type VREG usecount 1, writecount 1, refcount 1 mountedhere 0 flags () v_object 0xffffff026cabe870 ref 0 pages 0 lock type zfs: EXCL by thread 0xffffff008090d740 (pid 46245) with exclusive waiters pending #0 0xffffffff8037fdc7 at __lockmgr_args+0x777 #1 0xffffffff80417b79 at vop_stdlock+0x39 #2 0xffffffff806405cb at VOP_LOCK1_APV+0x9b #3 0xffffffff80434a87 at _vn_lock+0x57 #4 0xffffffff80436098 at vn_write+0x218 #5 0xffffffff803dbea5 at dofilewrite+0x85 #6 0xffffffff803dd6a0 at kern_writev+0x60 #7 0xffffffff803dd7a5 at write+0x55 #8 0xffffffff80604ea2 at syscall+0x102 #9 0xffffffff805eade1 at Xfast_syscall+0xe1 0xffffff0216f0d9c0: tag zfs, type VREG usecount 1, writecount 1, refcount 1 mountedhere 0 flags () v_object 0xffffff01d7910a20 ref 0 pages 0 lock type zfs: SHARED (count 1) #0 0xffffffff8037fb3f at __lockmgr_args+0x4ef #1 0xffffffff80417b79 at vop_stdlock+0x39 #2 0xffffffff806405cb at VOP_LOCK1_APV+0x9b #3 0xffffffff80434a87 at _vn_lock+0x57 #4 0xffffffff80436327 at vn_read+0x1c7 #5 0xffffffff803dc051 at dofileread+0xa1 #6 0xffffffff803dc380 at kern_readv+0x60 #7 0xffffffff803dc485 at read+0x55 #8 0xffffffff80604ea2 at syscall+0x102 #9 0xffffffff805eade1 at Xfast_syscall+0xe1 ----- (6) "db> alltrace" ----- (very long output, for the sake of completeness!) Tracing command sysctl pid 68597 tid 100536 td 0xffffff02a31a63a0 kdb_enter() at kdb_enter+0x3d panic() at panic+0x17b kdb_trap() at kdb_trap sysctl_root() at sysctl_root+0x113 userland_sysctl() at userland_sysctl+0x158 __sysctl() at __sysctl+0xaa syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (202, FreeBSD ELF64, __sysctl), rip = 0x80073cb6c, rsp = 0x7fffffffe248, rbp = 0x7fffffffe31c --- Tracing command sshd pid 68596 tid 100684 td 0xffffff0449bcfae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8013bbb6c, rsp = 0x7fffffffa4d8, rbp = 0x26 --- Tracing command sshd pid 68595 tid 100207 td 0xffffff021610c740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013d693c, rsp = 0x7fffffffdbd8, rbp = 0 --- Tracing command inetd pid 68593 tid 100213 td 0xffffff0015b753a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b cache_lookup() at cache_lookup+0x4e0 vfs_cache_lookup() at vfs_cache_lookup+0xc0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x479 kern_execve() at kern_execve+0x2a3 execve() at execve+0x3d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (59, FreeBSD ELF64, execve), rip = 0x8009dc58c, rsp = 0x7fffffffdd78, rbp = 0x800c46200 --- Tracing command inetd pid 68588 tid 100878 td 0xffffff02269333a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b cache_lookup() at cache_lookup+0x4e0 vfs_cache_lookup() at vfs_cache_lookup+0xc0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x479 kern_execve() at kern_execve+0x2a3 execve() at execve+0x3d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (59, FreeBSD ELF64, execve), rip = 0x8009dc58c, rsp = 0x7fffffffdd78, rbp = 0x800c46200 --- Tracing command inetd pid 68587 tid 100758 td 0xffffff04ed9733a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b cache_lookup() at cache_lookup+0x4e0 vfs_cache_lookup() at vfs_cache_lookup+0xc0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x479 kern_execve() at kern_execve+0x2a3 execve() at execve+0x3d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (59, FreeBSD ELF64, execve), rip = 0x8009dc58c, rsp = 0x7fffffffdd78, rbp = 0x800c46200 --- Tracing command sh pid 68579 tid 100764 td 0xffffff04ed96aae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a zio_wait() at zio_wait+0x61 dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x22a dmu_buf_hold_array() at dmu_buf_hold_array+0x62 dmu_read_uio() at dmu_read_uio+0x3f zfs_freebsd_read() at zfs_freebsd_read+0x5c1 VOP_READ_APV() at VOP_READ_APV+0xaf vnode_pager_generic_getpages() at vnode_pager_generic_getpages+0x3f3 VOP_GETPAGES_APV() at VOP_GETPAGES_APV+0xb5 vnode_pager_getpages() at vnode_pager_getpages+0x8c exec_map_first_page() at exec_map_first_page+0x13c kern_execve() at kern_execve+0x32e execve() at execve+0x3d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (59, FreeBSD ELF64, execve), rip = 0x80093558c, rsp = 0x7fffffffccc8, rbp = 0x800c20430 --- Tracing command grep pid 68566 tid 100217 td 0xffffff0015bb73a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e getblk() at getblk+0xf4 breadn() at breadn+0x3f bread() at bread+0x1e ffs_update() at ffs_update+0x122 ufs_inactive() at ufs_inactive+0x15e VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5 vinactive() at vinactive+0x90 vputx() at vputx+0x2de exit1() at exit1+0x415 sys_exit() at sys_exit+0xe syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (1, FreeBSD ELF64, sys_exit), rip = 0x800a32b2c, rsp = 0x7fffffffe9e8, rbp = 0 --- Tracing command csh pid 68564 tid 100672 td 0xffffff04ed974740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e getblk() at getblk+0xf4 breadn() at breadn+0x3f bread() at bread+0x1e ffs_vgetf() at ffs_vgetf+0x2d4 ufs_lookup_() at ufs_lookup_+0xb46 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x479 kern_execve() at kern_execve+0x2a3 execve() at execve+0x3d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (59, FreeBSD ELF64, execve), rip = 0x80096458c, rsp = 0x7fffffffe5c8, rbp = 0x800c0d578 --- Tracing command inetd pid 68560 tid 100709 td 0xffffff0041398000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b cache_lookup() at cache_lookup+0x4e0 vfs_cache_lookup() at vfs_cache_lookup+0xc0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x479 kern_execve() at kern_execve+0x2a3 execve() at execve+0x3d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (59, FreeBSD ELF64, execve), rip = 0x8009dc58c, rsp = 0x7fffffffdd78, rbp = 0x800c46200 --- Tracing command sshd pid 68559 tid 100741 td 0xffffff0041392000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8013bbb6c, rsp = 0x7fffffffa4d8, rbp = 0x26 --- Tracing command sshd pid 68558 tid 100141 td 0xffffff0007c6f3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013d693c, rsp = 0x7fffffffdbd8, rbp = 0 --- Tracing command inetd pid 68553 tid 100891 td 0xffffff02fb700000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b cache_lookup() at cache_lookup+0x4e0 vfs_cache_lookup() at vfs_cache_lookup+0xc0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x479 kern_execve() at kern_execve+0x2a3 execve() at execve+0x3d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (59, FreeBSD ELF64, execve), rip = 0x8009dc58c, rsp = 0x7fffffffdd78, rbp = 0x800c46200 --- Tracing command inetd pid 68552 tid 100250 td 0xffffff020ab44ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b cache_lookup() at cache_lookup+0x4e0 vfs_cache_lookup() at vfs_cache_lookup+0xc0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x479 kern_execve() at kern_execve+0x2a3 execve() at execve+0x3d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (59, FreeBSD ELF64, execve), rip = 0x8009dc58c, rsp = 0x7fffffffdd78, rbp = 0x800c46200 --- Tracing command inetd pid 68534 tid 100186 td 0xffffff0015ebc000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b cache_lookup() at cache_lookup+0x4e0 vfs_cache_lookup() at vfs_cache_lookup+0xc0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x479 kern_execve() at kern_execve+0x2a3 execve() at execve+0x3d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (59, FreeBSD ELF64, execve), rip = 0x8009dc58c, rsp = 0x7fffffffdd78, rbp = 0x800c46200 --- Tracing command qmail-queue pid 68532 tid 100816 td 0xffffff0080909ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a zio_wait() at zio_wait+0x61 zil_commit() at zil_commit+0x3e1 zfs_freebsd_fsync() at zfs_freebsd_fsync+0xd7 VOP_FSYNC_APV() at VOP_FSYNC_APV+0xb5 fsync() at fsync+0x148 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (95, FreeBSD ELF64, fsync), rip = 0x8007396bc, rsp = 0x7fffffffeb48, rbp = 0x7fffffffeba0 --- Tracing command qmail-smtpd pid 68531 tid 100777 td 0xffffff0118850000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800aa55cc, rsp = 0x7fffffffea98, rbp = 0x7fffffffead4 --- Tracing command sshd pid 68527 tid 100621 td 0xffffff03b21f4000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x8013bbb6c, rsp = 0x7fffffffa4d8, rbp = 0x26 --- Tracing command sshd pid 68526 tid 100687 td 0xffffff0449bcf000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013d693c, rsp = 0x7fffffffdbd8, rbp = 0 --- Tracing command inetd pid 68525 tid 100882 td 0xffffff01180d1ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b cache_lookup() at cache_lookup+0x4e0 vfs_cache_lookup() at vfs_cache_lookup+0xc0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x479 kern_execve() at kern_execve+0x2a3 execve() at execve+0x3d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (59, FreeBSD ELF64, execve), rip = 0x8009dc58c, rsp = 0x7fffffffdd78, rbp = 0x800c46200 --- Tracing command inetd pid 68522 tid 100853 td 0xffffff00a11db740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e ffs_lock() at ffs_lock+0x9c VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b cache_lookup() at cache_lookup+0x4e0 vfs_cache_lookup() at vfs_cache_lookup+0xc0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x479 kern_execve() at kern_execve+0x2a3 execve() at execve+0x3d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (59, FreeBSD ELF64, execve), rip = 0x8009dc58c, rsp = 0x7fffffffdd78, rbp = 0x800c46200 --- Tracing command python pid 68521 tid 100897 td 0xffffff02fb58b740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e getblk() at getblk+0xf4 breadn() at breadn+0x3f bread() at bread+0x1e ffs_update() at ffs_update+0x122 ufs_inactive() at ufs_inactive+0x15e VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5 vinactive() at vinactive+0x90 vputx() at vputx+0x2de kern_execve() at kern_execve+0xbbf execve() at execve+0x3d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (0, FreeBSD ELF64, nosys), rip = 0x800605770, rsp = 0x7fffffffebf8, rbp = 0 --- Tracing command proxymap pid 68511 tid 100865 td 0xffffff01180d1740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800c49b9c, rsp = 0x7fffffffde38, rbp = 0x7fffffffde40 --- Tracing command smtpd pid 68510 tid 100890 td 0xffffff02fb7003a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x800c82b6c, rsp = 0x7fffffffe688, rbp = 0 --- Tracing command mktemp pid 68497 tid 100289 td 0xffffff026ce6f3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 bwait() at bwait+0x64 bufwait() at bufwait+0x23 bufwrite() at bufwrite+0x158 ffs_update() at ffs_update+0x1cc ufs_makeinode() at ufs_makeinode+0x1f4 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x482 kern_openat() at kern_openat+0x179 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip = 0x80072b70c, rsp = 0x7fffffffe228, rbp = 0x800a04092 --- Tracing command tsort pid 68496 tid 100715 td 0xffffff03fe6cc000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e getblk() at getblk+0xf4 breadn() at breadn+0x3f bread() at bread+0x1e ffs_update() at ffs_update+0x122 ufs_inactive() at ufs_inactive+0x15e VOP_INACTIVE_APV() at VOP_INACTIVE_APV+0xb5 vinactive() at vinactive+0x90 vputx() at vputx+0x2de exit1() at exit1+0x415 sigexit() at sigexit+0x8f cursig() at cursig ast() at ast+0x1a9 doreti_ast() at doreti_ast+0x1f Tracing command sh pid 68495 tid 100822 td 0xffffff00809083a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8009bc93c, rsp = 0x7fffffffe0c8, rbp = 0x800c072b3 --- Tracing command sh pid 68494 tid 100730 td 0xffffff0080912740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2c0 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009355cc, rsp = 0x7fffffffe058, rbp = 0xffffffff --- Tracing command sh pid 68493 tid 100838 td 0xffffff00a11d9ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8009bc93c, rsp = 0x7fffffffe1b8, rbp = 0x800c08891 --- Tracing command make pid 68140 tid 100245 td 0xffffff0216844ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2c0 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x41f89c, rsp = 0x7fffffffd888, rbp = 0x7fffffffd8d0 --- Tracing command sh pid 68139 tid 100138 td 0xffffff0007c70000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2c0 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009355cc, rsp = 0x7fffffffe1d8, rbp = 0xffffffff --- Tracing command make pid 68138 tid 100210 td 0xffffff01c04df000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2c0 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x41f89c, rsp = 0x7fffffffd988, rbp = 0x7fffffffd9d0 --- Tracing command sh pid 68066 tid 100831 td 0xffffff0080906000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2c0 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009355cc, rsp = 0x7fffffffe248, rbp = 0xffffffff --- Tracing command zfs pid 67917 tid 100843 td 0xffffff01180d0740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a zio_wait() at zio_wait+0x61 traverse_read() at traverse_read+0x1b3 find_block() at find_block+0x1c9 traverse_more() at traverse_more+0x22f traverse_dsl_dataset() at traverse_dsl_dataset+0x50 dmu_sendbackup() at dmu_sendbackup+0x1ea zfs_ioc_send() at zfs_ioc_send+0x171 zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip = 0x800fe78dc, rsp = 0x7fffffffba18, rbp = 0x80121d600 --- Tracing command csh pid 67915 tid 100790 td 0xffffff008090eae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x80094d21c, rsp = 0x7fffffffe578, rbp = 0x800c5b400 --- Tracing command sshd pid 67913 tid 100881 td 0xffffff01186aa740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdc48, rbp = 0x7fffffffdcd0 --- Tracing command httpd pid 67519 tid 100736 td 0xffffff00413483a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa128, rbp = 0x7fffffffa238 --- Tracing command make pid 66665 tid 100519 td 0xffffff03229c93a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2c0 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x41f89c, rsp = 0x7fffffffd998, rbp = 0x7fffffffd9e0 --- Tracing command csh pid 49896 tid 100767 td 0xffffff04ed96a000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x80094d21c, rsp = 0x7fffffffe4c8, rbp = 0x800c5c600 --- Tracing command su pid 49816 tid 100759 td 0xffffff04ed973000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e85cc, rsp = 0x7fffffffe5c8, rbp = 0xc2e8 --- Tracing command tcsh pid 49531 tid 100707 td 0xffffff04ed95aae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x80094d21c, rsp = 0x7fffffffe598, rbp = 0x800c5c200 --- Tracing command sshd pid 49530 tid 100868 td 0xffffff0226962ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdc48, rbp = 0x7fffffffdcd0 --- Tracing command sshd pid 49511 tid 100789 td 0xffffff008090f000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013d693c, rsp = 0x7fffffffdc68, rbp = 0 --- Tracing command make pid 49405 tid 100247 td 0xffffff02168443a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2c0 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x41f89c, rsp = 0x7fffffffdc28, rbp = 0x7fffffffdc70 --- Tracing command make pid 49299 tid 100858 td 0xffffff00a11d9000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2c0 sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x41f89c, rsp = 0x7fffffffdca8, rbp = 0x7fffffffdcf0 --- Tracing command csh pid 49284 tid 100502 td 0xffffff02a30e6740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a fork1() at fork1+0xfe2 vfork() at vfork+0x1c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (66, FreeBSD ELF64, vfork), rip = 0x80094f39d, rsp = 0x7fffffffe6c0, rbp = 0 --- Tracing command su pid 49283 tid 100829 td 0xffffff0080906740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009e85cc, rsp = 0x7fffffffe5c8, rbp = 0xc084 --- Tracing command tcsh pid 49280 tid 100139 td 0xffffff0007c6fae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x80094d21c, rsp = 0x7fffffffe598, rbp = 0x800c5c600 --- Tracing command sshd pid 49279 tid 100806 td 0xffffff008090a000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdc48, rbp = 0x7fffffffdcd0 --- Tracing command sshd pid 49277 tid 100824 td 0xffffff0080907ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013d693c, rsp = 0x7fffffffdc68, rbp = 0 --- Tracing command csh pid 49133 tid 100422 td 0xffffff02a3a27740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x80094d21c, rsp = 0x7fffffffe628, rbp = 0x800c5c300 --- Tracing command httpd pid 48183 tid 100494 td 0xffffff0322949000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa128, rbp = 0x7fffffffa238 --- Tracing command httpd pid 47840 tid 100875 td 0xffffff0226960000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa128, rbp = 0x7fffffffa238 --- Tracing command mplayer pid 47422 tid 100235 td 0xffffff020ab103a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f soreceive_generic() at soreceive_generic+0xf99 kern_recvit() at kern_recvit+0x1c1 recvit() at recvit+0x21 recvfrom() at recvfrom+0x82 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (29, FreeBSD ELF64, recvfrom), rip = 0x8019777ac, rsp = 0x7fffffffd908, rbp = 0x400 --- Tracing command bash pid 47420 tid 100719 td 0xffffff03fe6c93a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800bb85cc, rsp = 0x7fffffffe468, rbp = 0xb93e --- Tracing command sh pid 47419 tid 100688 td 0xffffff02a31a2740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009355cc, rsp = 0x7fffffffea88, rbp = 0xb939 --- Tracing command atrun pid 47418 tid 100460 td 0xffffff02a3a20740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8008cd5cc, rsp = 0x7fffffffe9e8, rbp = 0x4 --- Tracing command httpd pid 47399 tid 100161 td 0xffffff0007ca4ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa128, rbp = 0x7fffffffa238 --- Tracing command httpd pid 46967 tid 100761 td 0xffffff04ed96e740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa128, rbp = 0x7fffffffa238 --- Tracing command pickup pid 46370 tid 100727 td 0xffffff00809133a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800c4eb9c, rsp = 0x7fffffffda18, rbp = 0x7fffffffda20 --- Tracing command screen pid 46325 tid 100773 td 0xffffff0118852000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x800a5721c, rsp = 0x7fffffffe3a8, rbp = 0x12 --- Tracing command bash pid 46323 tid 100836 td 0xffffff02a319d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800b9b5cc, rsp = 0x7fffffffe8e8, rbp = 0x800e453c0 --- Tracing command sshd pid 46322 tid 100762 td 0xffffff04ed96e3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdc38, rbp = 0x7fffffffdcc0 --- Tracing command sshd pid 46319 tid 100559 td 0xffffff03fe6cdae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x8013d693c, rsp = 0x7fffffffdc58, rbp = 0 --- Tracing command httpd pid 46245 tid 100798 td 0xffffff008090d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a zio_wait() at zio_wait+0x61 dbuf_read() at dbuf_read+0x3e4 dmu_tx_check_ioerr() at dmu_tx_check_ioerr+0x9a dmu_tx_count_write() at dmu_tx_count_write+0x4c dmu_tx_hold_write() at dmu_tx_hold_write+0x4a zfs_freebsd_write() at zfs_freebsd_write+0x3ea VOP_WRITE_APV() at VOP_WRITE_APV+0x129 vn_write() at vn_write+0x2a2 dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x80117191c, rsp = 0x7fffffff61a8, rbp = 0x7fffffff82a0 --- Tracing command httpd pid 45112 tid 100546 td 0xffffff03229c9000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa128, rbp = 0x7fffffffa238 --- Tracing command httpd pid 45014 tid 100477 td 0xffffff032292d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa128, rbp = 0x7fffffffa238 --- Tracing command httpd pid 43851 tid 100230 td 0xffffff01c0ef0740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_accept() at kern_accept+0x1fc accept() at accept+0x75 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (30, FreeBSD ELF64, accept), rip = 0x800fd6c0c, rsp = 0x7fffffffe998, rbp = 0x7fffffffe9d8 --- Tracing command httpd pid 43056 tid 100667 td 0xffffff03b21f3740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa208, rbp = 0x7fffffffa308 --- Tracing command httpd pid 43051 tid 100731 td 0xffffff00809123a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa208, rbp = 0x7fffffffa308 --- Tracing command httpd pid 42965 tid 100788 td 0xffffff008090f3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa208, rbp = 0x7fffffffa308 --- Tracing command httpd pid 42955 tid 100785 td 0xffffff0118841000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa208, rbp = 0x7fffffffa308 --- Tracing command httpd pid 42939 tid 100805 td 0xffffff008090a3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa208, rbp = 0x7fffffffa308 --- Tracing command httpd pid 42933 tid 100280 td 0xffffff0015c2b740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa208, rbp = 0x7fffffffa308 --- Tracing command httpd pid 42520 tid 100718 td 0xffffff04ed9693a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa208, rbp = 0x7fffffffa308 --- Tracing command httpd pid 42519 tid 100703 td 0xffffff04ed95bae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 lf_advlockasync() at lf_advlockasync+0xddd lf_advlock() at lf_advlock+0x47 vop_stdadvlock() at vop_stdadvlock+0xb3 VOP_ADVLOCK_APV() at VOP_ADVLOCK_APV+0xb5 flock() at flock+0x133 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (131, FreeBSD ELF64, flock), rip = 0x801101fac, rsp = 0x7fffffffe978, rbp = 0x7fffffffe9c8 --- Tracing command httpd pid 42326 tid 100835 td 0xffffff02a318dae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111cefc, rsp = 0x7fffffffa208, rbp = 0x7fffffffa308 --- Tracing command httpd pid 42072 tid 100724 td 0xffffff04ed968740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x801157b9c, rsp = 0x7fffffffe938, rbp = 0x7fffffffe9c8 --- Tracing command httpd pid 38848 tid 100839 td 0xffffff0041399ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vn_write() at vn_write+0x218 dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x80117191c, rsp = 0x7fffffff61a8, rbp = 0x7fffffff82a0 --- Tracing command qmgr pid 29104 tid 100191 td 0xffffff01c0f7d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800c5cb9c, rsp = 0x7fffffffd9e8, rbp = 0x7fffffffd9f0 --- Tracing command httpd pid 23197 tid 100456 td 0xffffff02a3a21740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_accept() at kern_accept+0x1fc accept() at accept+0x75 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (30, FreeBSD ELF64, accept), rip = 0x8012c4c0c, rsp = 0x7fffffffe858, rbp = 0x801613be0 --- Tracing command irssi pid 17921 tid 100244 td 0xffffff0216845000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80191fefc, rsp = 0x7fffffffe5d8, rbp = 0x2 --- Tracing command bash pid 17919 tid 100779 td 0xffffff011884f740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800b9b5cc, rsp = 0x7fffffffe428, rbp = 0x800e55800 --- Tracing command centerim pid 17874 tid 100694 td 0xffffff0041398ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x801ccd8bc, rsp = 0x7fffffffd098, rbp = 0x5 --- Tracing command screen pid 17872 tid 100514 td 0xffffff0322918740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800af58bc, rsp = 0x7fffffffe288, rbp = 0x7fffffffe310 --- Tracing command irssi pid 11037 tid 100714 td 0xffffff04ed969740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80191fefc, rsp = 0x7fffffffe5f8, rbp = 0x2 --- Tracing command bash pid 11035 tid 100710 td 0xffffff04ed95a3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x800b9b5cc, rsp = 0x7fffffffe448, rbp = 0x800e5d0c0 --- Tracing command centerim pid 10731 tid 100199 td 0xffffff01c0ee53a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x801ccd8bc, rsp = 0x7fffffffd208, rbp = 0x4 --- Tracing command screen pid 10729 tid 100473 td 0xffffff0322d26000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800af58bc, rsp = 0x7fffffffe2b8, rbp = 0x7fffffffe340 --- Tracing command httpd pid 6589 tid 100691 td 0xffffff0041399740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x801170b9c, rsp = 0x7fffffffe958, rbp = 0x7fffffffe9fc --- Tracing command login pid 6335 tid 100457 td 0xffffff02a3a213a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009ea5cc, rsp = 0x7fffffffec48, rbp = 0 --- Tracing command getty pid 6334 tid 100556 td 0xffffff03fe6ce740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084e93c, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 6333 tid 100530 td 0xffffff02a3a0c740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084e93c, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 6332 tid 100281 td 0xffffff0015c2b3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084e93c, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 6331 tid 100459 td 0xffffff02a3a20ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084e93c, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 6330 tid 100489 td 0xffffff02a30eb3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084e93c, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 6329 tid 100495 td 0xffffff0322901ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084e93c, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 6328 tid 100474 td 0xffffff032292dae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084e93c, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command getty pid 6327 tid 100516 td 0xffffff0322918000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80084e93c, rsp = 0x7fffffffecc8, rbp = 0 --- Tracing command bsnmpd pid 6314 tid 100167 td 0xffffff00153c6000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800a6b8bc, rsp = 0x7fffffffc3d8, rbp = 0x7fffffffc5cc --- Tracing command inetd pid 6292 tid 100506 td 0xffffff02a30e0ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800a638bc, rsp = 0x7fffffffdd88, rbp = 0 --- Tracing command cron pid 6258 tid 100174 td 0xffffff0015c053a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffeb28, rbp = 0x3c --- Tracing command sendmail pid 6251 tid 100674 td 0xffffff04ed974000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x800d3921c, rsp = 0x7fffffffcd78, rbp = 0x1 --- Tracing command sendmail pid 6247 tid 100668 td 0xffffff03b21f33a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800dd78bc, rsp = 0x7fffffffc1e8, rbp = 0x7fffffffc280 --- Tracing command bacula-fd pid 6221 tid 100681 td 0xffffff0449bb8740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f do_cv_wait() at do_cv_wait+0x57a __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800ef52ac, rsp = 0x7fffffbfee78, rbp = 0x801c741c0 --- Tracing command bacula-fd pid 6221 tid 100154 td 0xffffff0007cc4740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x801a158bc, rsp = 0x7fffffffdb98, rbp = 0x4 --- Tracing command cron pid 6191 tid 100679 td 0xffffff04ed97cae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea98, rbp = 0x3c --- Tracing command sshd pid 6182 tid 100680 td 0xffffff04ed97c740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd18, rbp = 0 --- Tracing command syslogd pid 6072 tid 100286 td 0xffffff026cdfe000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command httpd pid 5948 tid 100665 td 0xffffff04ed821ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 lf_advlockasync() at lf_advlockasync+0xddd lf_advlock() at lf_advlock+0x47 vop_stdadvlock() at vop_stdadvlock+0xb3 VOP_ADVLOCK_APV() at VOP_ADVLOCK_APV+0xb5 flock() at flock+0x133 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (131, FreeBSD ELF64, flock), rip = 0x801104fac, rsp = 0x7ffffc7e4f08, rbp = 0x7ffffc7e4f7c --- Tracing command httpd pid 5948 tid 100664 td 0xffffff04ed823000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffc9e5e68, rbp = 0x804c14280 --- Tracing command httpd pid 5948 tid 100663 td 0xffffff04ed8233a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffcbe6e68, rbp = 0x804c14440 --- Tracing command httpd pid 5948 tid 100662 td 0xffffff04ed823740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffcde7e68, rbp = 0x804c14600 --- Tracing command httpd pid 5948 tid 100661 td 0xffffff04ed823ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111fefc, rsp = 0x7ffffcfe47b8, rbp = 0x7ffffcfe48b8 --- Tracing command httpd pid 5948 tid 100660 td 0xffffff04ed824000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd1e9e68, rbp = 0x804c14980 --- Tracing command httpd pid 5948 tid 100659 td 0xffffff04ed8243a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd3eae68, rbp = 0x804c14b40 --- Tracing command httpd pid 5948 tid 100658 td 0xffffff04ed824740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd5ebe68, rbp = 0x804c14d00 --- Tracing command httpd pid 5948 tid 100657 td 0xffffff04ed824ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd7ece68, rbp = 0x804c14ec0 --- Tracing command httpd pid 5948 tid 100613 td 0xffffff04ed825000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd9ede68, rbp = 0x804c15080 --- Tracing command httpd pid 5948 tid 100656 td 0xffffff04ed8253a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111fefc, rsp = 0x7ffffdbea7b8, rbp = 0x7ffffdbea8b8 --- Tracing command httpd pid 5948 tid 100654 td 0xffffff04ed825740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffddefe68, rbp = 0x804c15400 --- Tracing command httpd pid 5948 tid 100652 td 0xffffff04ed825ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffdff0e68, rbp = 0x804c155c0 --- Tracing command httpd pid 5948 tid 100650 td 0xffffff04ed826000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe1f1e68, rbp = 0x804c15780 --- Tracing command httpd pid 5948 tid 100648 td 0xffffff04ed8263a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe3f2e68, rbp = 0x804c15940 --- Tracing command httpd pid 5948 tid 100646 td 0xffffff04ed826740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe5f3e68, rbp = 0x804c15b00 --- Tracing command httpd pid 5948 tid 100644 td 0xffffff04ed826ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe7f4e68, rbp = 0x804c15cc0 --- Tracing command httpd pid 5948 tid 100641 td 0xffffff02a31a83a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe9f5e68, rbp = 0x804c15e80 --- Tracing command httpd pid 5948 tid 100640 td 0xffffff02a31a8740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffebf6e68, rbp = 0x804c16040 --- Tracing command httpd pid 5948 tid 100638 td 0xffffff02a31a8ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffedf7e68, rbp = 0x804c16200 --- Tracing command httpd pid 5948 tid 100636 td 0xffffff02a3856000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffeff8e68, rbp = 0x804c163c0 --- Tracing command httpd pid 5948 tid 100635 td 0xffffff02a38563a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff1f9e68, rbp = 0x804c16580 --- Tracing command httpd pid 5948 tid 100632 td 0xffffff02a3856740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff3fae68, rbp = 0x804c16740 --- Tracing command httpd pid 5948 tid 100630 td 0xffffff0449bb8ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff5fbe68, rbp = 0x804c16900 --- Tracing command httpd pid 5948 tid 100628 td 0xffffff0449bb9000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff7fce68, rbp = 0x804c16ac0 --- Tracing command httpd pid 5948 tid 100626 td 0xffffff0449bb93a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff9fde68, rbp = 0x804c16c80 --- Tracing command httpd pid 5948 tid 100069 td 0xffffff0007ca33a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80117493c, rsp = 0x7fffffffe948, rbp = 0 --- Tracing command httpd pid 5947 tid 100655 td 0xffffff04ed81a000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x80115ab9c, rsp = 0x7ffffc7e4ec8, rbp = 0x7ffffc7e4f7c --- Tracing command httpd pid 5947 tid 100653 td 0xffffff04ed81a3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffc9e5e68, rbp = 0x804c14280 --- Tracing command httpd pid 5947 tid 100651 td 0xffffff04ed81a740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffcbe6e68, rbp = 0x804c14440 --- Tracing command httpd pid 5947 tid 100649 td 0xffffff04ed81aae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffcde7e68, rbp = 0x804c14600 --- Tracing command httpd pid 5947 tid 100647 td 0xffffff04ed81c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffcfe8e68, rbp = 0x804c147c0 --- Tracing command httpd pid 5947 tid 100645 td 0xffffff04ed81c3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd1e9e68, rbp = 0x804c14980 --- Tracing command httpd pid 5947 tid 100643 td 0xffffff04ed81c740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd3eae68, rbp = 0x804c14b40 --- Tracing command httpd pid 5947 tid 100642 td 0xffffff04ed81cae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111fefc, rsp = 0x7ffffd5e77b8, rbp = 0x7ffffd5e78b8 --- Tracing command httpd pid 5947 tid 100639 td 0xffffff0015c05ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd7ece68, rbp = 0x804c14ec0 --- Tracing command httpd pid 5947 tid 100637 td 0xffffff01d7013000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd9ede68, rbp = 0x804c15080 --- Tracing command httpd pid 5947 tid 100634 td 0xffffff01d70133a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffdbeee68, rbp = 0x804c15240 --- Tracing command httpd pid 5947 tid 100633 td 0xffffff01d7013740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffddefe68, rbp = 0x804c15400 --- Tracing command httpd pid 5947 tid 100631 td 0xffffff01d7013ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffdff0e68, rbp = 0x804c155c0 --- Tracing command httpd pid 5947 tid 100629 td 0xffffff01c0cba000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe1f1e68, rbp = 0x804c15780 --- Tracing command httpd pid 5947 tid 100627 td 0xffffff01c0cba3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe3f2e68, rbp = 0x804c15940 --- Tracing command httpd pid 5947 tid 100625 td 0xffffff02a3856ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe5f3e68, rbp = 0x804c15b00 --- Tracing command httpd pid 5947 tid 100624 td 0xffffff02a397b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe7f4e68, rbp = 0x804c15cc0 --- Tracing command httpd pid 5947 tid 100623 td 0xffffff02a397b3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe9f5e68, rbp = 0x804c15e80 --- Tracing command httpd pid 5947 tid 100622 td 0xffffff02a397b740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffebf6e68, rbp = 0x804c16040 --- Tracing command httpd pid 5947 tid 100620 td 0xffffff02a397bae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffedf7e68, rbp = 0x804c16200 --- Tracing command httpd pid 5947 tid 100619 td 0xffffff02a388e000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffeff8e68, rbp = 0x804c163c0 --- Tracing command httpd pid 5947 tid 100618 td 0xffffff02a388e3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff1f9e68, rbp = 0x804c16580 --- Tracing command httpd pid 5947 tid 100617 td 0xffffff02a388e740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff3fae68, rbp = 0x804c16740 --- Tracing command httpd pid 5947 tid 100616 td 0xffffff0449bb9740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff5fbe68, rbp = 0x804c16900 --- Tracing command httpd pid 5947 tid 100615 td 0xffffff0449bb9ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff7fce68, rbp = 0x804c16ac0 --- Tracing command httpd pid 5947 tid 100614 td 0xffffff0449bba000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff9fde68, rbp = 0x804c16c80 --- Tracing command httpd pid 5947 tid 100458 td 0xffffff02a3a21000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80117493c, rsp = 0x7fffffffe948, rbp = 0 --- Tracing command cron pid 5940 tid 100305 td 0xffffff02162ff3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea98, rbp = 0x3c --- Tracing command sshd pid 5931 tid 100409 td 0xffffff026cd12000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd18, rbp = 0 --- Tracing command httpd pid 5918 tid 100198 td 0xffffff0007d18ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8011748bc, rsp = 0x7fffffffe9a8, rbp = 0 --- Tracing command python2.4 pid 5868 tid 100701 td 0xffffff02a30d6ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5c2ac, rsp = 0x7fffff8fb658, rbp = 0x8106cce40 --- Tracing command python2.4 pid 5868 tid 100700 td 0xffffff02a30d7000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5c2ac, rsp = 0x7fffff9fc658, rbp = 0x8108261c0 --- Tracing command python2.4 pid 5868 tid 100699 td 0xffffff02a30d73a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5c2ac, rsp = 0x7fffffafd658, rbp = 0x810826380 --- Tracing command python2.4 pid 5868 tid 100698 td 0xffffff02a30d7740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5c2ac, rsp = 0x7fffffbfe658, rbp = 0x810826540 --- Tracing command python2.4 pid 5868 tid 100168 td 0xffffff00153c5ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e558bc, rsp = 0x7fffffff1838, rbp = 0x12 --- Tracing command python2.4 pid 5867 tid 100152 td 0xffffff0007cc5000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e558bc, rsp = 0x7fffffff18d8, rbp = 0x4 --- Tracing command syslogd pid 5801 tid 100218 td 0xffffff0015bb7000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command cron pid 5674 tid 100500 td 0xffffff0322901000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea98, rbp = 0x3c --- Tracing command sshd pid 5665 tid 100531 td 0xffffff02a3a0c3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd18, rbp = 0x4 --- Tracing command mysqld pid 5639 tid 100612 td 0xffffff03b21f43a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigtimedwait() at kern_sigtimedwait+0x55d sigwait() at sigwait+0x74 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (429, FreeBSD ELF64, sigwait), rip = 0x801475b3c, rsp = 0x7ffffebf6f08, rbp = 0x80180a200 --- Tracing command mysqld pid 5639 tid 100609 td 0xffffff03b21f4740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x80131b2ac, rsp = 0x7ffffedf7e88, rbp = 0x80180a3c0 --- Tracing command mysqld pid 5639 tid 100611 td 0xffffff03b21f4ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8015148bc, rsp = 0x7ffffeff8f18, rbp = 0 --- Tracing command mysqld pid 5639 tid 100610 td 0xffffff03b21f5000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8015148bc, rsp = 0x7fffff1f9f08, rbp = 0 --- Tracing command mysqld pid 5639 tid 100608 td 0xffffff03b21f5740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x80131b2ac, rsp = 0x7fffff5fbbc8, rbp = 0x80180aac0 --- Tracing command mysqld pid 5639 tid 100607 td 0xffffff03b21f5ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x80131b2ac, rsp = 0x7fffff7fcbc8, rbp = 0x80180ac80 --- Tracing command mysqld pid 5639 tid 100606 td 0xffffff03b21f6000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x80131b2ac, rsp = 0x7fffff9fdbc8, rbp = 0x80180ae40 --- Tracing command mysqld pid 5639 tid 100605 td 0xffffff03b21f63a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x80131b2ac, rsp = 0x7fffffbfebc8, rbp = 0x802d161c0 --- Tracing command mysqld pid 5639 tid 100577 td 0xffffff03fe84f3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8015148bc, rsp = 0x7fffffffe3b8, rbp = 0xd --- Tracing command sh pid 5573 tid 100563 td 0xffffff03b2dd33a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009355cc, rsp = 0x7fffffffdf08, rbp = 0x15c2 --- Tracing command syslogd pid 5470 tid 100384 td 0xffffff027d9a8740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command httpd pid 5469 tid 100159 td 0xffffff0007ccb3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_accept() at kern_accept+0x1fc accept() at accept+0x75 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (30, FreeBSD ELF64, accept), rip = 0x8012c4c0c, rsp = 0x7fffffffe838, rbp = 0x801613be0 --- Tracing command httpd pid 5468 tid 100562 td 0xffffff03b2dd3740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_accept() at kern_accept+0x1fc accept() at accept+0x75 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (30, FreeBSD ELF64, accept), rip = 0x8012c4c0c, rsp = 0x7fffffffe838, rbp = 0x801613be0 --- Tracing command ruby18 pid 5390 tid 100229 td 0xffffff01c0ef0ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800c6c8bc, rsp = 0x7fffffff7598, rbp = 0x7fffffff7620 --- Tracing command ApplicationPoolServ pid 5389 tid 100783 td 0xffffff0041391740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800f1893c, rsp = 0x7fffffb78628, rbp = 0x7fffffb786b0 --- Tracing command ApplicationPoolServ pid 5389 tid 100782 td 0xffffff0041391ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800f1893c, rsp = 0x7fffffb99628, rbp = 0x7fffffb996b0 --- Tracing command ApplicationPoolServ pid 5389 tid 100604 td 0xffffff03fe6ccae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 fifo_open() at fifo_open+0x2f6 VOP_OPEN_APV() at VOP_OPEN_APV+0xaf vn_open_cred() at vn_open_cred+0x316 kern_openat() at kern_openat+0x179 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip = 0x800f0970c, rsp = 0x7fffffbdd8b8, rbp = 0x8 --- Tracing command ApplicationPoolServ pid 5389 tid 100603 td 0xffffff03fe6cd000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f do_cv_wait() at do_cv_wait+0x57a __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x8008e82ac, rsp = 0x7fffffbfe9b8, rbp = 0x80160ae40 --- Tracing command ApplicationPoolServ pid 5389 tid 100225 td 0xffffff0007d6c3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800f1893c, rsp = 0x7fffffffdeb8, rbp = 0x7fffffffdf40 --- Tracing command cron pid 5336 tid 100450 td 0xffffff02a3a27000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea98, rbp = 0x3c --- Tracing command sshd pid 5327 tid 100557 td 0xffffff03fe6ce3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd18, rbp = 0x4 --- Tracing command httpd pid 5314 tid 100421 td 0xffffff02e7e5d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013288bc, rsp = 0x7fffffffe9d8, rbp = 0 --- Tracing command vsftpd pid 5245 tid 100503 td 0xffffff02a30e63a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_accept() at kern_accept+0x1fc accept() at accept+0x75 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (30, FreeBSD ELF64, accept), rip = 0x800df5c0c, rsp = 0x7fffffffe8e8, rbp = 0x7fffffffe980 --- Tracing command syslogd pid 5176 tid 100228 td 0xffffff0007c6c3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command inetd pid 5065 tid 100560 td 0xffffff03fe6cd740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800a638bc, rsp = 0x7fffffffdd38, rbp = 0 --- Tracing command cron pid 5051 tid 100576 td 0xffffff03fe84f740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea98, rbp = 0x3c --- Tracing command sshd pid 5042 tid 100385 td 0xffffff027d9a83a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd18, rbp = 0x4 --- Tracing command syslogd pid 4932 tid 100572 td 0xffffff03fe818740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command httpd pid 4915 tid 100303 td 0xffffff02162ffae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_accept() at kern_accept+0x1fc accept() at accept+0x75 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (30, FreeBSD ELF64, accept), rip = 0x800fd6c0c, rsp = 0x7fffffffe998, rbp = 0x7fffffffe9d8 --- Tracing command httpd pid 4914 tid 100420 td 0xffffff02e7e5dae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_accept() at kern_accept+0x1fc accept() at accept+0x75 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (30, FreeBSD ELF64, accept), rip = 0x800fd6c0c, rsp = 0x7fffffffe998, rbp = 0x7fffffffe9d8 --- Tracing command httpd pid 4913 tid 100548 td 0xffffff0322be0740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_accept() at kern_accept+0x1fc accept() at accept+0x75 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (30, FreeBSD ELF64, accept), rip = 0x800fd6c0c, rsp = 0x7fffffffe998, rbp = 0x7fffffffe9d8 --- Tracing command httpd pid 4912 tid 100335 td 0xffffff02a318c740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_accept() at kern_accept+0x1fc accept() at accept+0x75 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (30, FreeBSD ELF64, accept), rip = 0x800fd6c0c, rsp = 0x7fffffffe998, rbp = 0x7fffffffe9d8 --- Tracing command httpd pid 4911 tid 100547 td 0xffffff0322be0ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_accept() at kern_accept+0x1fc accept() at accept+0x75 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (30, FreeBSD ELF64, accept), rip = 0x800fd6c0c, rsp = 0x7fffffffe998, rbp = 0x7fffffffe9d8 --- Tracing command cron pid 4807 tid 100508 td 0xffffff0322bda000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea98, rbp = 0x3b --- Tracing command sendmail pid 4800 tid 100578 td 0xffffff03fe84f000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x800d3921c, rsp = 0x7fffffffccf8, rbp = 0x1 --- Tracing command sendmail pid 4796 tid 100551 td 0xffffff032200cae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800dd78bc, rsp = 0x7fffffffc168, rbp = 0x7fffffffc200 --- Tracing command sshd pid 4789 tid 100569 td 0xffffff03b2dcb3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd18, rbp = 0x4 --- Tracing command httpd pid 4776 tid 100192 td 0xffffff01c0f7d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x80103a8bc, rsp = 0x7fffffffea48, rbp = 0 --- Tracing command varnishd pid 4726 tid 100600 td 0xffffff02a39e6000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800acb2ac, rsp = 0x7ffffdfee968, rbp = 0x806413cc0 --- Tracing command varnishd pid 4726 tid 100599 td 0xffffff02a39e63a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800acb2ac, rsp = 0x7ffffe1ef968, rbp = 0x806413e80 --- Tracing command varnishd pid 4726 tid 100598 td 0xffffff02a39e6740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800acb2ac, rsp = 0x7ffffe3f0968, rbp = 0x806414040 --- Tracing command varnishd pid 4726 tid 100597 td 0xffffff02a39e6ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800acb2ac, rsp = 0x7ffffe5f1968, rbp = 0x806414200 --- Tracing command varnishd pid 4726 tid 100596 td 0xffffff02a39e7000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800acb2ac, rsp = 0x7ffffe7f2968, rbp = 0x8064143c0 --- Tracing command varnishd pid 4726 tid 100595 td 0xffffff02a39e73a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800acb2ac, rsp = 0x7ffffe9f3968, rbp = 0x806414580 --- Tracing command varnishd pid 4726 tid 100594 td 0xffffff02a39e7740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800acb2ac, rsp = 0x7ffffebf4968, rbp = 0x806414740 --- Tracing command varnishd pid 4726 tid 100593 td 0xffffff02a39e7ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800acb2ac, rsp = 0x7ffffedf5968, rbp = 0x806414900 --- Tracing command varnishd pid 4726 tid 100592 td 0xffffff02a39fd000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800acb2ac, rsp = 0x7ffffeff6968, rbp = 0x806414ac0 --- Tracing command varnishd pid 4726 tid 100591 td 0xffffff03fe786000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x800d8eefc, rsp = 0x7fffff1f9ea8, rbp = 0 --- Tracing command varnishd pid 4726 tid 100590 td 0xffffff03fe7863a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800dc9b9c, rsp = 0x7fffff3f9fa8, rbp = 0x80100ac88 --- Tracing command varnishd pid 4726 tid 100589 td 0xffffff02a39fd3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800acb2ac, rsp = 0x7fffff5f9968, rbp = 0x806414c80 --- Tracing command varnishd pid 4726 tid 100588 td 0xffffff03fe786740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x800dc8b6c, rsp = 0x7fffff7fc668, rbp = 0 --- Tracing command varnishd pid 4726 tid 100587 td 0xffffff03fe786ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800acb2ac, rsp = 0x7fffff9fdec8, rbp = 0x8011181c0 --- Tracing command varnishd pid 4726 tid 100586 td 0xffffff03fe7a4000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x800dc8b6c, rsp = 0x7fffffbfec78, rbp = 0x801118388 --- Tracing command varnishd pid 4726 tid 100407 td 0xffffff026cd12740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x800d8eefc, rsp = 0x7fffffffe5d8, rbp = 0x801017c40 --- Tracing command varnishd pid 4725 tid 100555 td 0xffffff03fe6ceae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x800d8eefc, rsp = 0x7fffffffe5e8, rbp = 0x801033080 --- Tracing command python pid 4709 tid 100849 td 0xffffff0118841ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c602ac, rsp = 0x7fffffafd668, rbp = 0x8146ebd40 --- Tracing command python pid 4709 tid 100848 td 0xffffff01188453a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c602ac, rsp = 0x7fffffbfe668, rbp = 0x8146ebf00 --- Tracing command python pid 4709 tid 100541 td 0xffffff03b2dceae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e598bc, rsp = 0x7ffffffeb158, rbp = 0x9 --- Tracing command python pid 4708 tid 100292 td 0xffffff026cdc5740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e598bc, rsp = 0x7ffffffee288, rbp = 0x4 --- Tracing command syslogd pid 4628 tid 100574 td 0xffffff03fe818000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command innfeed pid 4524 tid 100515 td 0xffffff03229183a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x80077f8bc, rsp = 0x7fffffffd5e8, rbp = 0x800a28800 --- Tracing command perl pid 4523 tid 100570 td 0xffffff03b2dcb000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800cc393c, rsp = 0x7fffffffb7b8, rbp = 0 --- Tracing command inetd pid 4512 tid 100571 td 0xffffff03fe818ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800a638bc, rsp = 0x7fffffffdd18, rbp = 0 --- Tracing command cron pid 4498 tid 100282 td 0xffffff0015c2b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea98, rbp = 0x3c --- Tracing command sendmail pid 4492 tid 100467 td 0xffffff02a3a28740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x800d3921c, rsp = 0x7fffffffccf8, rbp = 0x1 --- Tracing command sendmail pid 4488 tid 100552 td 0xffffff032200c740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800dd78bc, rsp = 0x7fffffffc168, rbp = 0x7fffffffc200 --- Tracing command sshd pid 4481 tid 100553 td 0xffffff032200c3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd18, rbp = 0 --- Tracing command perl pid 4428 tid 100451 td 0xffffff02a3a24ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x800ca8b6c, rsp = 0x7fffffffd8f8, rbp = 0xcfb978 --- Tracing command sh pid 4427 tid 100543 td 0xffffff03b2dce3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009355cc, rsp = 0x7fffffffccc8, rbp = 0x1133 --- Tracing command innd pid 4426 tid 100517 td 0xffffff03229c9ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a zio_wait() at zio_wait+0x61 dbuf_read() at dbuf_read+0x3e4 dmu_buf_hold() at dmu_buf_hold+0xcc zap_lockdir() at zap_lockdir+0x55 zap_lookup_norm() at zap_lookup_norm+0x45 zap_lookup() at zap_lookup+0x2e zfs_dirent_lock() at zfs_dirent_lock+0x503 zfs_dirlook() at zfs_dirlook+0x69 zfs_lookup() at zfs_lookup+0x1f0 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x81 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x479 vn_open_cred() at vn_open_cred+0xc2 kern_openat() at kern_openat+0x179 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip = 0x800d1370c, rsp = 0x7fffffffcce8, rbp = 0 --- Tracing command syslogd pid 4340 tid 100542 td 0xffffff03b2dce740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command cron pid 4207 tid 100476 td 0xffffff032292d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea98, rbp = 0x3c --- Tracing command sendmail pid 4201 tid 100533 td 0xffffff02a39ffae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x800d3921c, rsp = 0x7fffffffccf8, rbp = 0x1 --- Tracing command sendmail pid 4197 tid 100532 td 0xffffff02a3a0c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800dd78bc, rsp = 0x7fffffffc168, rbp = 0x7fffffffc200 --- Tracing command sshd pid 4185 tid 100507 td 0xffffff0322bda3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd48, rbp = 0x4 --- Tracing command perl pid 4175 tid 100475 td 0xffffff032292d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800cb48bc, rsp = 0x7fffffffea98, rbp = 0x3 --- Tracing command named pid 4103 tid 100529 td 0xffffff02a3a0cae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800c82b9c, rsp = 0x7ffffe9f5e08, rbp = 0x201 --- Tracing command named pid 4103 tid 100528 td 0xffffff02a3a0d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f do_cv_wait() at do_cv_wait+0x57a __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800aa32ac, rsp = 0x7ffffebf6db8, rbp = 0x800e0a200 --- Tracing command named pid 4103 tid 100527 td 0xffffff02a3a0d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800aa32ac, rsp = 0x7ffffedf7eb8, rbp = 0x800e0a3c0 --- Tracing command named pid 4103 tid 100526 td 0xffffff02a3a0d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800aa32ac, rsp = 0x7ffffeff8eb8, rbp = 0x800e0a580 --- Tracing command named pid 4103 tid 100525 td 0xffffff02a3a0dae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800aa32ac, rsp = 0x7fffff1f9eb8, rbp = 0x800e0a740 --- Tracing command named pid 4103 tid 100524 td 0xffffff02a3a0e000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800aa32ac, rsp = 0x7fffff3faeb8, rbp = 0x800e0a900 --- Tracing command named pid 4103 tid 100523 td 0xffffff02a3a0e3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800aa32ac, rsp = 0x7fffff5fbeb8, rbp = 0x800e0aac0 --- Tracing command named pid 4103 tid 100522 td 0xffffff02a3a0e740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800aa32ac, rsp = 0x7fffff7fceb8, rbp = 0x800e0ac80 --- Tracing command named pid 4103 tid 100521 td 0xffffff02a3a0eae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800aa32ac, rsp = 0x7fffff9fdeb8, rbp = 0x800e0ae40 --- Tracing command named pid 4103 tid 100520 td 0xffffff02a3a0f000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800aa32ac, rsp = 0x7fffffbfeeb8, rbp = 0x800e9d1c0 --- Tracing command named pid 4103 tid 100406 td 0xffffff026cd12ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigtimedwait() at kern_sigtimedwait+0x55d sigwait() at sigwait+0x74 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (429, FreeBSD ELF64, sigwait), rip = 0x800bfdb3c, rsp = 0x7fffffffeae8, rbp = 0x800e041c0 --- Tracing command syslogd pid 4098 tid 100386 td 0xffffff027d9a8000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe058, rbp = 0x800a330c8 --- Tracing command qmail-clean pid 3947 tid 100512 td 0xffffff02a30d7ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80073b93c, rsp = 0x7fffffffec18, rbp = 0x503aa0 --- Tracing command qmail-rspawn pid 3946 tid 100511 td 0xffffff02a30e0000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x80073c8bc, rsp = 0x7fffffffec48, rbp = 0x1 --- Tracing command qmail-lspawn pid 3945 tid 100510 td 0xffffff02a30e03a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x80073c8bc, rsp = 0x7fffffffec28, rbp = 0x1 --- Tracing command splogger pid 3944 tid 100509 td 0xffffff02a30e0740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80073a93c, rsp = 0x7fffffffec98, rbp = 0x501dc0 --- Tracing command qmail-send pid 3937 tid 100505 td 0xffffff02a30e6000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8007438bc, rsp = 0x7fffffffebd8, rbp = 0x7fffffffebe0 --- Tracing command cron pid 3930 tid 100306 td 0xffffff02162ff000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea88, rbp = 0x3c --- Tracing command sshd pid 3904 tid 100295 td 0xffffff02161083a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd58, rbp = 0 --- Tracing command iscsi-target pid 3855 tid 100324 td 0xffffff02a303d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x800818efc, rsp = 0x7fffffffde18, rbp = 0x7fffffffde50 --- Tracing command perl pid 3843 tid 100283 td 0xffffff0015b80ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800ca68bc, rsp = 0x7fffffffeb08, rbp = 0x3 --- Tracing command mysqld pid 3835 tid 100487 td 0xffffff03229493a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigtimedwait() at kern_sigtimedwait+0x55d sigwait() at sigwait+0x74 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (429, FreeBSD ELF64, sigwait), rip = 0x8013e9b3c, rsp = 0x7ffffebf6f08, rbp = 0x80160a200 --- Tracing command mysqld pid 3835 tid 100484 td 0xffffff0322949740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x80128f2ac, rsp = 0x7ffffedf7e88, rbp = 0x80160a3c0 --- Tracing command mysqld pid 3835 tid 100486 td 0xffffff0322949ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8014888bc, rsp = 0x7ffffeff8f18, rbp = 0 --- Tracing command mysqld pid 3835 tid 100485 td 0xffffff0322cbc000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8014888bc, rsp = 0x7fffff1f9f08, rbp = 0 --- Tracing command mysqld pid 3835 tid 100483 td 0xffffff02a3a0f3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x80128f2ac, rsp = 0x7fffff5fbbb8, rbp = 0x80160aac0 --- Tracing command mysqld pid 3835 tid 100482 td 0xffffff02a3a0f740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x80128f2ac, rsp = 0x7fffff7fcbb8, rbp = 0x80160ac80 --- Tracing command mysqld pid 3835 tid 100481 td 0xffffff03b21f6740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x80128f2ac, rsp = 0x7fffff9fdbb8, rbp = 0x80160ae40 --- Tracing command mysqld pid 3835 tid 100480 td 0xffffff03b21f6ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x80128f2ac, rsp = 0x7fffffbfebb8, rbp = 0x8021bb1c0 --- Tracing command mysqld pid 3835 tid 100397 td 0xffffff026cc9d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8014888bc, rsp = 0x7fffffffe4d8, rbp = 0xd --- Tracing command sh pid 3806 tid 100464 td 0xffffff02e7e5d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x8009355cc, rsp = 0x7fffffffe258, rbp = 0xedd --- Tracing command nginx pid 3792 tid 100470 td 0xffffff0322d26ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800beab9c, rsp = 0x7fffffffe888, rbp = 0 --- Tracing command nginx pid 3791 tid 100381 td 0xffffff02a318c3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x800b6621c, rsp = 0x7fffffffe9d8, rbp = 0x1 --- Tracing command master pid 3778 tid 100472 td 0xffffff0322d263a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800c46b9c, rsp = 0x7fffffffdce8, rbp = 0x7fffffffdcf0 --- Tracing command perl pid 3702 tid 100215 td 0xffffff0015bb7ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800ca68bc, rsp = 0x7fffffffeaa8, rbp = 0x3 --- Tracing command syslogd pid 3552 tid 100463 td 0xffffff02a3a0fae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command cron pid 3406 tid 100151 td 0xffffff0007cc53a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea98, rbp = 0x3c --- Tracing command sshd pid 3397 tid 100455 td 0xffffff02a3a21ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdce8, rbp = 0x4 --- Tracing command qmail-clean pid 3396 tid 100319 td 0xffffff027dfee3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80073b93c, rsp = 0x7fffffffebf8, rbp = 0x503aa0 --- Tracing command qmail-rspawn pid 3395 tid 100248 td 0xffffff0216844000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x80073c8bc, rsp = 0x7fffffffec28, rbp = 0x1 --- Tracing command qmail-lspawn pid 3394 tid 100316 td 0xffffff0216843000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x80073c8bc, rsp = 0x7fffffffec08, rbp = 0x1 --- Tracing command multilog pid 3393 tid 100410 td 0xffffff0216847ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80073f93c, rsp = 0x7fffffffeb38, rbp = 0x1 --- Tracing command tcpserver pid 3392 tid 100077 td 0xffffff0007b963a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_accept() at kern_accept+0x1fc accept() at accept+0x75 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (30, FreeBSD ELF64, accept), rip = 0x80082ec0c, rsp = 0x7fffffffeae8, rbp = 0xffffffff --- Tracing command qmail-send pid 3391 tid 100417 td 0xffffff02e7e3b740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8007438bc, rsp = 0x7fffffffebb8, rbp = 0x7fffffffebc0 --- Tracing command multilog pid 3390 tid 100454 td 0xffffff02a3a24000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80073f93c, rsp = 0x7fffffffeb38, rbp = 0x1 --- Tracing command supervise pid 3386 tid 100411 td 0xffffff02e7118000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006e8efc, rsp = 0x7fffffffea98, rbp = 0x2 --- Tracing command supervise pid 3385 tid 100388 td 0xffffff026cc9d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006e8efc, rsp = 0x7fffffffea88, rbp = 0x2 --- Tracing command supervise pid 3384 tid 100401 td 0xffffff027dd5e3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006e8efc, rsp = 0x7fffffffea98, rbp = 0x2 --- Tracing command supervise pid 3383 tid 100317 td 0xffffff0216301ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006e8efc, rsp = 0x7fffffffea98, rbp = 0x2 --- Tracing command readproctitle pid 3374 tid 100399 td 0xffffff027dd5eae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80073993c, rsp = 0x7fffffffea48, rbp = 0x7fffffffed5d --- Tracing command svscan pid 3373 tid 100419 td 0xffffff02e7e3b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x800721b6c, rsp = 0x7fffffffec28, rbp = 0x7fffffffeca0 --- Tracing command syslogd pid 3274 tid 100416 td 0xffffff02e7e3bae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command httpd pid 3250 tid 100448 td 0xffffff02e71183a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_accept() at kern_accept+0x1fc accept() at accept+0x75 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (30, FreeBSD ELF64, accept), rip = 0x801128c0c, rsp = 0x7ffffc7e4dc8, rbp = 0x7ffffc7e4f08 --- Tracing command httpd pid 3250 tid 100447 td 0xffffff02e7118740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffc9e5e58, rbp = 0x806414280 --- Tracing command httpd pid 3250 tid 100446 td 0xffffff02e7118ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffcbe6e58, rbp = 0x806414440 --- Tracing command httpd pid 3250 tid 100445 td 0xffffff02e7ff4000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffcde7e58, rbp = 0x806414600 --- Tracing command httpd pid 3250 tid 100444 td 0xffffff02e7ff43a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffcfe8e58, rbp = 0x8064147c0 --- Tracing command httpd pid 3250 tid 100443 td 0xffffff02e7ff4740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffd1e9e58, rbp = 0x806414980 --- Tracing command httpd pid 3250 tid 100442 td 0xffffff02e7ff4ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffd3eae58, rbp = 0x806414b40 --- Tracing command httpd pid 3250 tid 100441 td 0xffffff02e7ff5000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffd5ebe58, rbp = 0x806414d00 --- Tracing command httpd pid 3250 tid 100440 td 0xffffff02e7ff53a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffd7ece58, rbp = 0x806414ec0 --- Tracing command httpd pid 3250 tid 100439 td 0xffffff02e7ff5740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffd9ede58, rbp = 0x806415080 --- Tracing command httpd pid 3250 tid 100438 td 0xffffff02e7ff5ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffdbeee58, rbp = 0x806415240 --- Tracing command httpd pid 3250 tid 100437 td 0xffffff02e7ff6000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffddefe58, rbp = 0x806415400 --- Tracing command httpd pid 3250 tid 100436 td 0xffffff02e7ff63a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffdff0e58, rbp = 0x8064155c0 --- Tracing command httpd pid 3250 tid 100435 td 0xffffff02e7ff6740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffe1f1e58, rbp = 0x806415780 --- Tracing command httpd pid 3250 tid 100434 td 0xffffff02e7ff6ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffe3f2e58, rbp = 0x806415940 --- Tracing command httpd pid 3250 tid 100433 td 0xffffff032200c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffe5f3e58, rbp = 0x806415b00 --- Tracing command httpd pid 3250 tid 100432 td 0xffffff02a30efae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffe7f4e58, rbp = 0x806415cc0 --- Tracing command httpd pid 3250 tid 100431 td 0xffffff02a3188000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffe9f5e58, rbp = 0x806415e80 --- Tracing command httpd pid 3250 tid 100430 td 0xffffff02a31883a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffebf6e58, rbp = 0x806416040 --- Tracing command httpd pid 3250 tid 100429 td 0xffffff02a3188740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffedf7e58, rbp = 0x806416200 --- Tracing command httpd pid 3250 tid 100428 td 0xffffff02a3188ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7ffffeff8e58, rbp = 0x8064163c0 --- Tracing command httpd pid 3250 tid 100427 td 0xffffff02a318a000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7fffff1f9e58, rbp = 0x806416580 --- Tracing command httpd pid 3250 tid 100426 td 0xffffff02a318a3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7fffff3fae58, rbp = 0x806416740 --- Tracing command httpd pid 3250 tid 100425 td 0xffffff02a318a740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7fffff5fbe58, rbp = 0x806416900 --- Tracing command httpd pid 3250 tid 100424 td 0xffffff02a318aae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7fffff7fce58, rbp = 0x806416ac0 --- Tracing command httpd pid 3250 tid 100423 td 0xffffff02a318c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f932ac, rsp = 0x7fffff9fde58, rbp = 0x806416c80 --- Tracing command httpd pid 3250 tid 100404 td 0xffffff0007d6b740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80118c93c, rsp = 0x7fffffffe958, rbp = 0 --- Tracing command cron pid 3146 tid 100383 td 0xffffff027d9a8ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea98, rbp = 0x3c --- Tracing command sendmail pid 3140 tid 100162 td 0xffffff0007d633a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x800d3921c, rsp = 0x7fffffffccf8, rbp = 0x1 --- Tracing command sendmail pid 3136 tid 100400 td 0xffffff027dd5e740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800dd78bc, rsp = 0x7fffffffc168, rbp = 0x7fffffffc200 --- Tracing command sshd pid 3129 tid 100300 td 0xffffff0216300740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd18, rbp = 0x4 --- Tracing command httpd pid 3116 tid 100293 td 0xffffff026cdc53a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x80118c8bc, rsp = 0x7fffffffe9a8, rbp = 0 --- Tracing command proftpd pid 3061 tid 100325 td 0xffffff02a3034ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800dfa8bc, rsp = 0x7fffffffeb48, rbp = 0x2 --- Tracing command syslogd pid 2998 tid 100398 td 0xffffff026cc9d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command cron pid 2864 tid 100403 td 0xffffff026cdfeae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea98, rbp = 0x3c --- Tracing command sendmail pid 2858 tid 100323 td 0xffffff02a303d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x800d3921c, rsp = 0x7fffffffccf8, rbp = 0x1 --- Tracing command sendmail pid 2854 tid 100322 td 0xffffff02a303d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800dd78bc, rsp = 0x7fffffffc168, rbp = 0x7fffffffc200 --- Tracing command httpd pid 2853 tid 100221 td 0xffffff02163013a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 lf_advlockasync() at lf_advlockasync+0xddd lf_advlock() at lf_advlock+0x47 vop_stdadvlock() at vop_stdadvlock+0xb3 VOP_ADVLOCK_APV() at VOP_ADVLOCK_APV+0xb5 flock() at flock+0x133 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (131, FreeBSD ELF64, flock), rip = 0x80111afac, rsp = 0x7fffffffe978, rbp = 0x7fffffffe9fc --- Tracing command httpd pid 2852 tid 100311 td 0xffffff021610cae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 lf_advlockasync() at lf_advlockasync+0xddd lf_advlock() at lf_advlock+0x47 vop_stdadvlock() at vop_stdadvlock+0xb3 VOP_ADVLOCK_APV() at VOP_ADVLOCK_APV+0xb5 flock() at flock+0x133 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (131, FreeBSD ELF64, flock), rip = 0x80111afac, rsp = 0x7fffffffe978, rbp = 0x7fffffffe9fc --- Tracing command sshd pid 2845 tid 100149 td 0xffffff0007d6e740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd18, rbp = 0x4 --- Tracing command httpd pid 2832 tid 100287 td 0xffffff026ce6fae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x80118a8bc, rsp = 0x7fffffffe9b8, rbp = 0 --- Tracing command nscd pid 2804 tid 100396 td 0xffffff02a39fe000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800a76b9c, rsp = 0x7ffffedf7e98, rbp = 0x7ffffedf7f80 --- Tracing command nscd pid 2804 tid 100395 td 0xffffff02a39fe3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800a76b9c, rsp = 0x7ffffeff8e98, rbp = 0x7ffffeff8f80 --- Tracing command nscd pid 2804 tid 100394 td 0xffffff02a39fe740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800a76b9c, rsp = 0x7fffff1f9e98, rbp = 0x7fffff1f9f80 --- Tracing command nscd pid 2804 tid 100393 td 0xffffff02a39feae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800a76b9c, rsp = 0x7fffff3fae98, rbp = 0x7fffff3faf80 --- Tracing command nscd pid 2804 tid 100392 td 0xffffff02a39ff000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800a76b9c, rsp = 0x7fffff5fbe98, rbp = 0x7fffff5fbf80 --- Tracing command nscd pid 2804 tid 100391 td 0xffffff0007d69ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800a76b9c, rsp = 0x7fffff7fce98, rbp = 0x7fffff7fcf80 --- Tracing command nscd pid 2804 tid 100390 td 0xffffff0007d6b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800a76b9c, rsp = 0x7fffff9fde98, rbp = 0x7fffff9fdf80 --- Tracing command nscd pid 2804 tid 100389 td 0xffffff0007d6b3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x800a76b9c, rsp = 0x7fffffbfee98, rbp = 0x7fffffbfef80 --- Tracing command nscd pid 2804 tid 100195 td 0xffffff0007d61ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_wait() at do_wait+0x6bb __umtx_op_wait() at __umtx_op_wait+0x5e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x8007872ac, rsp = 0x7fffffffe9f8, rbp = 0x800c0ae40 --- Tracing command syslogd pid 2722 tid 100147 td 0xffffff0007d71000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command cron pid 2593 tid 100307 td 0xffffff02162feae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffeab8, rbp = 0x3c --- Tracing command qmail-clean pid 2592 tid 100136 td 0xffffff0007c70740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80073b93c, rsp = 0x7fffffffec38, rbp = 0x502480 --- Tracing command qmail-rspawn pid 2591 tid 100231 td 0xffffff01c0ef03a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x80073c8bc, rsp = 0x7fffffffec58, rbp = 0x12c0 --- Tracing command qmail-lspawn pid 2590 tid 100309 td 0xffffff02162fe3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x80073d8bc, rsp = 0x7fffffffec38, rbp = 0x12c0 --- Tracing command multilog pid 2589 tid 100219 td 0xffffff00153c9ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80073f93c, rsp = 0x7fffffffeb28, rbp = 0x1 --- Tracing command qmail-send pid 2587 tid 100318 td 0xffffff027dfee740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8007448bc, rsp = 0x7fffffffeba8, rbp = 0x7fffffffec30 --- Tracing command multilog pid 2586 tid 100313 td 0xffffff027ddee3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80073f93c, rsp = 0x7fffffffeb28, rbp = 0x1 --- Tracing command tcpserver pid 2583 tid 100156 td 0xffffff0007cc4000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_accept() at kern_accept+0x1fc accept() at accept+0x75 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (30, FreeBSD ELF64, accept), rip = 0x80082ec0c, rsp = 0x7fffffffea78, rbp = 0xffffffff --- Tracing command sshd pid 2577 tid 100314 td 0xffffff0216843740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd18, rbp = 0 --- Tracing command supervise pid 2576 tid 100113 td 0xffffff0007c70ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006e8efc, rsp = 0x7fffffffea98, rbp = 0x2 --- Tracing command supervise pid 2575 tid 100080 td 0xffffff0007d183a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006e8efc, rsp = 0x7fffffffea88, rbp = 0x2 --- Tracing command supervise pid 2574 tid 100304 td 0xffffff02162ff740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006e8efc, rsp = 0x7fffffffea98, rbp = 0x2 --- Tracing command supervise pid 2573 tid 100308 td 0xffffff02162fe740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x8006e8efc, rsp = 0x7fffffffea88, rbp = 0x2 --- Tracing command readproctitle pid 2561 tid 100241 td 0xffffff0216845ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80073993c, rsp = 0x7fffffffea48, rbp = 0x7fffffffed5d --- Tracing command svscan pid 2560 tid 100132 td 0xffffff0007d73000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x800721b6c, rsp = 0x7fffffffec28, rbp = 0x7fffffffeca0 --- Tracing command httpd pid 2478 tid 100380 td 0xffffff02a31a6740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 lf_advlockasync() at lf_advlockasync+0xddd lf_advlock() at lf_advlock+0x47 vop_stdadvlock() at vop_stdadvlock+0xb3 VOP_ADVLOCK_APV() at VOP_ADVLOCK_APV+0xb5 flock() at flock+0x133 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (131, FreeBSD ELF64, flock), rip = 0x801104fac, rsp = 0x7ffffc7e4f08, rbp = 0x7ffffc7e4f7c --- Tracing command httpd pid 2478 tid 100379 td 0xffffff02a31a6ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffc9e5e68, rbp = 0x80524e440 --- Tracing command httpd pid 2478 tid 100378 td 0xffffff02a31a7000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffcbe6e68, rbp = 0x80524e600 --- Tracing command httpd pid 2478 tid 100377 td 0xffffff02a31a73a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffcde7e68, rbp = 0x80524e7c0 --- Tracing command httpd pid 2478 tid 100376 td 0xffffff02a31a7740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffcfe8e68, rbp = 0x80524e980 --- Tracing command httpd pid 2478 tid 100375 td 0xffffff02a31a7ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd1e9e68, rbp = 0x80524eb40 --- Tracing command httpd pid 2478 tid 100374 td 0xffffff02a31a8000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd3eae68, rbp = 0x80524ed00 --- Tracing command httpd pid 2478 tid 100373 td 0xffffff0007d6e3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd5ebe68, rbp = 0x80524eec0 --- Tracing command httpd pid 2478 tid 100372 td 0xffffff0015ebc740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd7ece68, rbp = 0x80524f080 --- Tracing command httpd pid 2478 tid 100371 td 0xffffff0015ebcae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd9ede68, rbp = 0x80524f240 --- Tracing command httpd pid 2478 tid 100370 td 0xffffff020a9b6000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffdbeee68, rbp = 0x80524f400 --- Tracing command httpd pid 2478 tid 100328 td 0xffffff020a9b63a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffddefe68, rbp = 0x80524f5c0 --- Tracing command httpd pid 2478 tid 100369 td 0xffffff020a9b6740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffdff0e68, rbp = 0x80524f780 --- Tracing command httpd pid 2478 tid 100367 td 0xffffff02a30ebae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe1f1e68, rbp = 0x80524f940 --- Tracing command httpd pid 2478 tid 100365 td 0xffffff02a30ee000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe3f2e68, rbp = 0x80524fb00 --- Tracing command httpd pid 2478 tid 100362 td 0xffffff02a30ee3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe5f3e68, rbp = 0x80524fcc0 --- Tracing command httpd pid 2478 tid 100361 td 0xffffff02a30ee740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111fefc, rsp = 0x7ffffe7f07b8, rbp = 0x7ffffe7f08b8 --- Tracing command httpd pid 2478 tid 100359 td 0xffffff02a30eeae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe9f5e68, rbp = 0x805250040 --- Tracing command httpd pid 2478 tid 100357 td 0xffffff02a30ef000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111fefc, rsp = 0x7ffffebf27b8, rbp = 0x7ffffebf28b8 --- Tracing command httpd pid 2478 tid 100355 td 0xffffff02a30ef3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111fefc, rsp = 0x7ffffedf37b8, rbp = 0x7ffffedf38b8 --- Tracing command httpd pid 2478 tid 100353 td 0xffffff02a30ef740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111fefc, rsp = 0x7ffffeff47b8, rbp = 0x7ffffeff48b8 --- Tracing command httpd pid 2478 tid 100350 td 0xffffff0007d6c740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111fefc, rsp = 0x7fffff1f57b8, rbp = 0x7fffff1f58b8 --- Tracing command httpd pid 2478 tid 100347 td 0xffffff020a9b6ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff3fae68, rbp = 0x805250900 --- Tracing command httpd pid 2478 tid 100344 td 0xffffff020ab21000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111fefc, rsp = 0x7fffff5f77b8, rbp = 0x7fffff5f78b8 --- Tracing command httpd pid 2478 tid 100341 td 0xffffff0007d6cae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff7fce68, rbp = 0x805250c80 --- Tracing command httpd pid 2478 tid 100339 td 0xffffff0007d6e000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111fefc, rsp = 0x7fffff9f97b8, rbp = 0x7fffff9f98b8 --- Tracing command httpd pid 2478 tid 100327 td 0xffffff02a30343a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80117493c, rsp = 0x7fffffffe948, rbp = 0 --- Tracing command httpd pid 2477 tid 100368 td 0xffffff02a318cae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_kevent() at kern_kevent+0x32b kevent() at kevent+0x90 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (363, FreeBSD ELF64, kevent), rip = 0x80115ab9c, rsp = 0x7ffffc7e4ec8, rbp = 0x7ffffc7e4f7c --- Tracing command httpd pid 2477 tid 100366 td 0xffffff02a318d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffc9e5e68, rbp = 0x80524e440 --- Tracing command httpd pid 2477 tid 100364 td 0xffffff02a318d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffcbe6e68, rbp = 0x80524e600 --- Tracing command httpd pid 2477 tid 100363 td 0xffffff02a318d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffcde7e68, rbp = 0x80524e7c0 --- Tracing command httpd pid 2477 tid 100360 td 0xffffff027dfeeae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffcfe8e68, rbp = 0x80524e980 --- Tracing command httpd pid 2477 tid 100358 td 0xffffff02a30cc000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd1e9e68, rbp = 0x80524eb40 --- Tracing command httpd pid 2477 tid 100356 td 0xffffff02a30cc3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd3eae68, rbp = 0x80524ed00 --- Tracing command httpd pid 2477 tid 100354 td 0xffffff02a30cc740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd5ebe68, rbp = 0x80524eec0 --- Tracing command httpd pid 2477 tid 100352 td 0xffffff02a30ccae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd7ece68, rbp = 0x80524f080 --- Tracing command httpd pid 2477 tid 100351 td 0xffffff02a30cd000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffd9ede68, rbp = 0x80524f240 --- Tracing command httpd pid 2477 tid 100349 td 0xffffff02a30cd3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffdbeee68, rbp = 0x80524f400 --- Tracing command httpd pid 2477 tid 100348 td 0xffffff02a30cd740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffddefe68, rbp = 0x80524f5c0 --- Tracing command httpd pid 2477 tid 100346 td 0xffffff02a30cdae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffdff0e68, rbp = 0x80524f780 --- Tracing command httpd pid 2477 tid 100345 td 0xffffff02a30ce000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe1f1e68, rbp = 0x80524f940 --- Tracing command httpd pid 2477 tid 100343 td 0xffffff02a30ce3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe3f2e68, rbp = 0x80524fb00 --- Tracing command httpd pid 2477 tid 100342 td 0xffffff02a30ce740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111fefc, rsp = 0x7ffffe5ef7b8, rbp = 0x7ffffe5ef8b8 --- Tracing command httpd pid 2477 tid 100340 td 0xffffff02a30ceae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe7f4e68, rbp = 0x80524fe80 --- Tracing command httpd pid 2477 tid 100338 td 0xffffff02a3178000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffe9f5e68, rbp = 0x805250040 --- Tracing command httpd pid 2477 tid 100337 td 0xffffff02a31783a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffebf6e68, rbp = 0x805250200 --- Tracing command httpd pid 2477 tid 100336 td 0xffffff02a3178740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffedf7e68, rbp = 0x8052503c0 --- Tracing command httpd pid 2477 tid 100334 td 0xffffff027ddeeae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7ffffeff8e68, rbp = 0x805250580 --- Tracing command httpd pid 2477 tid 100333 td 0xffffff02a3010000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x80111fefc, rsp = 0x7fffff1f57b8, rbp = 0x7fffff1f58b8 --- Tracing command httpd pid 2477 tid 100332 td 0xffffff02a30103a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff3fae68, rbp = 0x805250900 --- Tracing command httpd pid 2477 tid 100331 td 0xffffff02a3010740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff5fbe68, rbp = 0x805250ac0 --- Tracing command httpd pid 2477 tid 100330 td 0xffffff02a3010ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff7fce68, rbp = 0x805250c80 --- Tracing command httpd pid 2477 tid 100329 td 0xffffff02a3034000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800f7b2ac, rsp = 0x7fffff9fde68, rbp = 0x805250e40 --- Tracing command httpd pid 2477 tid 100326 td 0xffffff02a3034740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80117493c, rsp = 0x7fffffffe948, rbp = 0 --- Tracing command syslogd pid 2459 tid 100312 td 0xffffff027ddee740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command cron pid 2338 tid 100301 td 0xffffff02163003a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea98, rbp = 0x3c --- Tracing command sshd pid 2329 tid 100150 td 0xffffff0007cc5740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd18, rbp = 0 --- Tracing command httpd pid 2316 tid 100226 td 0xffffff0007c6cae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8011748bc, rsp = 0x7fffffffe9a8, rbp = 0 --- Tracing command python2.4 pid 2267 tid 100583 td 0xffffff02a31a53a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5c2ac, rsp = 0x7fffff8fb658, rbp = 0x80d4bc2c0 --- Tracing command python2.4 pid 2267 tid 100582 td 0xffffff02a31a5740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5c2ac, rsp = 0x7fffff9fc658, rbp = 0x80d4bc480 --- Tracing command python2.4 pid 2267 tid 100581 td 0xffffff02a31a5ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5c2ac, rsp = 0x7fffffafd658, rbp = 0x80d4bc640 --- Tracing command python2.4 pid 2267 tid 100580 td 0xffffff02a31a6000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5c2ac, rsp = 0x7fffffbfe658, rbp = 0x80d4bc800 --- Tracing command python2.4 pid 2267 tid 100183 td 0xffffff0015ebc3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e558bc, rsp = 0x7fffffff1838, rbp = 0x18 --- Tracing command python2.4 pid 2266 tid 100296 td 0xffffff0216301000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e558bc, rsp = 0x7fffffff18b8, rbp = 0x4 --- Tracing command httpd pid 2205 tid 100279 td 0xffffff0015c2bae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vn_write() at vn_write+0x218 dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip = 0x80117191c, rsp = 0x7fffffff6188, rbp = 0x7fffffff8280 --- Tracing command syslogd pid 2200 tid 100285 td 0xffffff026cdfe3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command cron pid 2073 tid 100251 td 0xffffff020ab44740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea98, rbp = 0x3c --- Tracing command sshd pid 2064 tid 100227 td 0xffffff0007c6c740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd18, rbp = 0 --- Tracing command httpd pid 2051 tid 100175 td 0xffffff0015c05000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8011718bc, rsp = 0x7fffffffe9b8, rbp = 0 --- Tracing command python pid 2001 tid 100585 td 0xffffff03fe7a43a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5b2ac, rsp = 0x7fffffafd658, rbp = 0x811169080 --- Tracing command python pid 2001 tid 100584 td 0xffffff03fe7a4740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5b2ac, rsp = 0x7fffffbfe658, rbp = 0x811169240 --- Tracing command python pid 2001 tid 100243 td 0xffffff02168453a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e548bc, rsp = 0x7ffffffebc48, rbp = 0x9 --- Tracing command python pid 2000 tid 100065 td 0xffffff0007ca43a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e548bc, rsp = 0x7ffffffee6c8, rbp = 0x4 --- Tracing command python pid 1986 tid 100567 td 0xffffff02a39fd740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a zio_wait() at zio_wait+0x61 dmu_buf_hold_array_by_dnode() at dmu_buf_hold_array_by_dnode+0x22a dmu_buf_hold_array() at dmu_buf_hold_array+0x62 dmu_read_uio() at dmu_read_uio+0x3f zfs_freebsd_read() at zfs_freebsd_read+0x571 VOP_READ_APV() at VOP_READ_APV+0xaf vn_read() at vn_read+0x256 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x800e5493c, rsp = 0x7fffffaf0fd8, rbp = 0x800f99cb0 --- Tracing command python pid 1986 tid 100566 td 0xffffff02a39fdae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5b2ac, rsp = 0x7fffffbfcaa8, rbp = 0x810fe83c0 --- Tracing command python pid 1986 tid 100249 td 0xffffff0216843ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e548bc, rsp = 0x7ffffffec6c8, rbp = 0x17 --- Tracing command python pid 1985 tid 100237 td 0xffffff0216847740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e548bc, rsp = 0x7ffffffeecb8, rbp = 0x4 --- Tracing command syslogd pid 1921 tid 100239 td 0xffffff02168473a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command cron pid 1785 tid 100211 td 0xffffff0015b75ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip = 0x80093eb6c, rsp = 0x7fffffffea98, rbp = 0x3c --- Tracing command sshd pid 1774 tid 100208 td 0xffffff01c04df3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd18, rbp = 0x1 --- Tracing command rotatelogs pid 1773 tid 100194 td 0xffffff0007d613a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80112093c, rsp = 0x7ffffffee538, rbp = 0x801413110 --- Tracing command rotatelogs pid 1772 tid 100190 td 0xffffff0007d61740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80112093c, rsp = 0x7ffffffee538, rbp = 0x801413110 --- Tracing command rotatelogs pid 1771 tid 100189 td 0xffffff0007d18740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80112093c, rsp = 0x7ffffffee538, rbp = 0x801413110 --- Tracing command rotatelogs pid 1769 tid 100188 td 0xffffff0007d61000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80112093c, rsp = 0x7ffffffee538, rbp = 0x801413110 --- Tracing command rotatelogs pid 1768 tid 100185 td 0xffffff0007d62000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80112093c, rsp = 0x7ffffffee538, rbp = 0x801413110 --- Tracing command rotatelogs pid 1767 tid 100184 td 0xffffff0007d63740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip = 0x80112093c, rsp = 0x7ffffffee538, rbp = 0x801413110 --- Tracing command httpd pid 1752 tid 100182 td 0xffffff0007d62ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8011718bc, rsp = 0x7fffffffe9c8, rbp = 0 --- Tracing command python pid 1688 tid 100261 td 0xffffff0007d6bae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5b2ac, rsp = 0x7fffffafd658, rbp = 0x810d446c0 --- Tracing command python pid 1688 tid 100260 td 0xffffff0007d6c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5b2ac, rsp = 0x7fffffbfe658, rbp = 0x810d44880 --- Tracing command python pid 1688 tid 100181 td 0xffffff0007d62740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e548bc, rsp = 0x7ffffffeb8f8, rbp = 0xf --- Tracing command python pid 1687 tid 100180 td 0xffffff0015b8bae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e548bc, rsp = 0x7ffffffee578, rbp = 0x4 --- Tracing command python2.4 pid 1679 tid 100253 td 0xffffff0015b61740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5b2ac, rsp = 0x7fffffafd658, rbp = 0x8105d0b00 --- Tracing command python2.4 pid 1679 tid 100252 td 0xffffff0015b61ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5b2ac, rsp = 0x7fffffbfe658, rbp = 0x8105d0cc0 --- Tracing command python2.4 pid 1679 tid 100177 td 0xffffff0015c28740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e548bc, rsp = 0x7fffffff1818, rbp = 0x9 --- Tracing command python2.4 pid 1678 tid 100176 td 0xffffff0015c28ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e548bc, rsp = 0x7fffffff18a8, rbp = 0x4 --- Tracing command python2.4 pid 1674 tid 100259 td 0xffffff020ab44000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5b2ac, rsp = 0x7fffffafd658, rbp = 0x8106feb00 --- Tracing command python2.4 pid 1674 tid 100258 td 0xffffff020ab443a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5b2ac, rsp = 0x7fffffbfe658, rbp = 0x8106fecc0 --- Tracing command python2.4 pid 1674 tid 100169 td 0xffffff00153c5740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e548bc, rsp = 0x7fffffff1818, rbp = 0xf --- Tracing command python2.4 pid 1673 tid 100153 td 0xffffff0007cc4ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e548bc, rsp = 0x7fffffff18b8, rbp = 0x4 --- Tracing command python2.4 pid 1669 tid 100255 td 0xffffff0015b61000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5b2ac, rsp = 0x7fffffafd658, rbp = 0x8106d2b00 --- Tracing command python2.4 pid 1669 tid 100254 td 0xffffff0015b613a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5b2ac, rsp = 0x7fffffbfe658, rbp = 0x8106d2cc0 --- Tracing command python2.4 pid 1669 tid 100166 td 0xffffff00153c63a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e548bc, rsp = 0x7fffffff1828, rbp = 0xe --- Tracing command python2.4 pid 1668 tid 100170 td 0xffffff00153c53a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e548bc, rsp = 0x7fffffff18c8, rbp = 0x4 --- Tracing command python2.4 pid 1664 tid 100257 td 0xffffff021610c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5b2ac, rsp = 0x7fffffafd658, rbp = 0x8106beb00 --- Tracing command python2.4 pid 1664 tid 100256 td 0xffffff021610c3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 do_cv_wait() at do_cv_wait+0x7d4 __umtx_op_cv_wait() at __umtx_op_cv_wait+0x5c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (454, FreeBSD ELF64, _umtx_op), rip = 0x800c5b2ac, rsp = 0x7fffffbfe658, rbp = 0x8106becc0 --- Tracing command python2.4 pid 1664 tid 100171 td 0xffffff00153c5000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e548bc, rsp = 0x7fffffff1848, rbp = 0xd --- Tracing command python2.4 pid 1663 tid 100172 td 0xffffff0007f6dae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800e548bc, rsp = 0x7fffffff18f8, rbp = 0x4 --- Tracing command syslogd pid 1592 tid 100178 td 0xffffff0015c283a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe098, rbp = 0x800a10058 --- Tracing command sshd pid 1399 tid 100160 td 0xffffff0007ccb000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8013d68bc, rsp = 0x7fffffffdd68, rbp = 0 --- Tracing command rwhod pid 1380 tid 100155 td 0xffffff0007cc43a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_dgram() at soreceive_dgram+0x180 kern_recvit() at kern_recvit+0x1c1 recvit() at recvit+0x21 recvfrom() at recvfrom+0x82 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (29, FreeBSD ELF64, recvfrom), rip = 0x8006f67ac, rsp = 0x7fffffffe7d8, rbp = 0x7fffffffe880 --- Tracing command ntpd pid 1344 tid 100158 td 0xffffff0007ccb740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800d508bc, rsp = 0x7fffffffec08, rbp = 0x7fffffffed30 --- Tracing command nfsd pid 1255 tid 100165 td 0xffffff0015b80000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c svc_run_internal() at svc_run_internal+0x7fe svc_thread_start() at svc_thread_start+0xb fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0xc, rip = 0x80069fdfc, rsp = 0x7fffffffe6b8, rbp = 0x7 --- Tracing command nfsd pid 1255 tid 100164 td 0xffffff0015b803a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c svc_run_internal() at svc_run_internal+0x7fe svc_thread_start() at svc_thread_start+0xb fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0xc, rip = 0x80069fdfc, rsp = 0x7fffffffe6b8, rbp = 0x7 --- Tracing command nfsd pid 1255 tid 100163 td 0xffffff0015b80740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c svc_run_internal() at svc_run_internal+0x7fe svc_thread_start() at svc_thread_start+0xb fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0xc, rip = 0x80069fdfc, rsp = 0x7fffffffe6b8, rbp = 0x7 --- Tracing command nfsd pid 1255 tid 100066 td 0xffffff0007ca4000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c svc_run_internal() at svc_run_internal+0x7fe svc_run() at svc_run+0x94 nfssvc_nfsd() at nfssvc_nfsd+0xa2 nfssvc_nfsserver() at nfssvc_nfsserver+0x5b nfssvc() at nfssvc+0x7c syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (155, FreeBSD ELF64, nfssvc), rip = 0x80069fdfc, rsp = 0x7fffffffe6b8, rbp = 0x7 --- Tracing command nfsd pid 1254 tid 100142 td 0xffffff0007c6f000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_accept() at kern_accept+0x1fc accept() at accept+0x75 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (30, FreeBSD ELF64, accept), rip = 0x8006d9c0c, rsp = 0x7fffffffe958, rbp = 0x800a04060 --- Tracing command mountd pid 1252 tid 100074 td 0xffffff0007c9fae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008528bc, rsp = 0x7fffffffec48, rbp = 0x7fffffffed40 --- Tracing command rpcbind pid 1132 tid 100078 td 0xffffff0007c9f3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 poll() at poll+0x2b6 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (209, FreeBSD ELF64, poll), rip = 0x800906efc, rsp = 0x7fffffffcab8, rbp = 0x800c130c0 --- Tracing command syslogd pid 1109 tid 100076 td 0xffffff0007b96740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8008518bc, rsp = 0x7fffffffe108, rbp = 0x800a310e0 --- Tracing command devd pid 936 tid 100071 td 0xffffff0007b973a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x440ecc, rsp = 0x7fffffffe898, rbp = 0x7fffffffe8b0 --- Tracing command moused pid 903 tid 100073 td 0xffffff0007b96ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8009708bc, rsp = 0x7fffffffe848, rbp = 0x7fffffffed10 --- Tracing command ospfd pid 632 tid 100072 td 0xffffff0007b97000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _cv_timedwait_sig() at _cv_timedwait_sig+0x18c seltdwait() at seltdwait+0x56 kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x800c2c8bc, rsp = 0x7fffffffe988, rbp = 0 --- Tracing command zebra pid 626 tid 100068 td 0xffffff0007ca3740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip = 0x8016fc8bc, rsp = 0x7fffffffe978, rbp = 0 --- Tracing command zfskern pid 43 tid 100112 td 0xffffff0007cc5ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a zio_wait() at zio_wait+0x61 dsl_pool_sync() at dsl_pool_sync+0xee spa_sync() at spa_sync+0x355 txg_sync_thread() at txg_sync_thread+0x28f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242bc3d30, rbp = 0 --- Tracing command zfskern pid 43 tid 100111 td 0xffffff0007d18000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a txg_thread_wait() at txg_thread_wait+0x79 txg_quiesce_thread() at txg_quiesce_thread+0xb5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242bbfd30, rbp = 0 --- Tracing command zfskern pid 43 tid 100109 td 0xffffff0007d63000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vdev_geom_worker() at vdev_geom_worker+0xf3 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242ba6d30, rbp = 0 --- Tracing command zfskern pid 43 tid 100108 td 0xffffff0007d64000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vdev_geom_worker() at vdev_geom_worker+0xf3 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242ba1d30, rbp = 0 --- Tracing command zfskern pid 43 tid 100081 td 0xffffff0007c9f000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c l2arc_feed_thread() at l2arc_feed_thread+0x144 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b29d30, rbp = 0 --- Tracing command zfskern pid 43 tid 100070 td 0xffffff0007ca3000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c arc_reclaim_thread() at arc_reclaim_thread+0x299 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242af2d30, rbp = 0 --- Tracing command flowcleaner pid 23 tid 100062 td 0xffffff0007b97740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sched_bind() at sched_bind+0x7f flowtable_cleaner() at flowtable_cleaner+0x177 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82407c7d30, rbp = 0 --- Tracing command softdepflush pid 22 tid 100061 td 0xffffff0007b97ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 softdep_flush() at softdep_flush+0x259 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82407c2d30, rbp = 0 --- Tracing command vnlru pid 21 tid 100060 td 0xffffff0007b99000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vnlru_proc() at vnlru_proc+0x5cf fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82407bdd30, rbp = 0 --- Tracing command syncer pid 20 tid 100059 td 0xffffff0007b993a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a zio_wait() at zio_wait+0x61 zil_commit() at zil_commit+0x3e1 zfs_sync() at zfs_sync+0xa6 sync_fsync() at sync_fsync+0x13a VOP_FSYNC_APV() at VOP_FSYNC_APV+0xb5 sync_vnode() at sync_vnode+0x157 sched_sync() at sched_sync+0x1cc fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82407b8d30, rbp = 0 --- Tracing command bufdaemon pid 19 tid 100058 td 0xffffff0007b99740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 buf_daemon() at buf_daemon+0x14a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82407b3d30, rbp = 0 --- Tracing command pagezero pid 18 tid 100057 td 0xffffff0007b99ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82407aed30, rbp = 0 --- Tracing command vmdaemon pid 17 tid 100056 td 0xffffff000742f3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82407a9d30, rbp = 0 --- Tracing command pagedaemon pid 16 tid 100055 td 0xffffff000742f740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pageout() at vm_pageout+0x855 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82407a4d30, rbp = 0 --- Tracing command g_journal switcher pid 15 tid 100054 td 0xffffff000742fae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 g_journal_switcher() at g_journal_switcher+0x4b fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82404f8d30, rbp = 0 --- Tracing command xpt_thrd pid 9 tid 100053 td 0xffffff0007880000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 xpt_scanner_thread() at xpt_scanner_thread+0xd5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82404f3d30, rbp = 0 --- Tracing command pfpurge pid 8 tid 100052 td 0xffffff00078803a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 pf_purge_thread() at pf_purge_thread+0x31 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82404eed30, rbp = 0 --- Tracing command ciss_notify0 pid 7 tid 100047 td 0xffffff0007883740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 ciss_notify_thread() at ciss_notify_thread+0x276 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82404c9d30, rbp = 0 --- Tracing command usb pid 14 tid 100042 td 0xffffff000742c740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8240417d30, rbp = 0 --- Tracing command usb pid 14 tid 100041 td 0xffffff000742cae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8240412d30, rbp = 0 --- Tracing command usb pid 14 tid 100040 td 0xffffff000742d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff824040dd30, rbp = 0 --- Tracing command usb pid 14 tid 100039 td 0xffffff000742d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8240408d30, rbp = 0 --- Tracing command crypto returns pid 6 tid 100027 td 0xffffff000739c740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 crypto_ret_proc() at crypto_ret_proc+0x19a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000108d30, rbp = 0 --- Tracing command crypto pid 5 tid 100026 td 0xffffff000735d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 crypto_proc() at crypto_proc+0x1d1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000103d30, rbp = 0 --- Tracing command yarrow pid 13 tid 100025 td 0xffffff000735dae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 random_kthread() at random_kthread+0x1ad fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000fed30, rbp = 0 --- Tracing command g_down pid 4 tid 100023 td 0xffffff000735e3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d turnstile_wait() at turnstile_wait+0x243 _mtx_lock_sleep() at _mtx_lock_sleep+0xd6 _mtx_lock_flags() at _mtx_lock_flags+0xe1 dastrategy() at dastrategy+0x46 g_disk_start() at g_disk_start+0xf4 g_io_schedule_down() at g_io_schedule_down+0x1e2 g_down_procbody() at g_down_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000f4d30, rbp = 0 --- Tracing command g_up pid 3 tid 100022 td 0xffffff000735e740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 g_io_schedule_up() at g_io_schedule_up+0x133 g_up_procbody() at g_up_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000efd30, rbp = 0 --- Tracing command g_event pid 2 tid 100021 td 0xffffff000735eae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 g_event_procbody() at g_event_procbody+0xa1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000ead30, rbp = 0 --- Tracing command intr pid 12 tid 100051 td 0xffffff0007880740 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x19a nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff80387589, rsp = 0xffffff8000062fe0, rbp = 0xffffff82404e9b70 --- _mtx_lock_sleep() at _mtx_lock_sleep+0x119 _mtx_lock_flags() at _mtx_lock_flags+0xe1 intr_event_execute_handlers() at intr_event_execute_handlers+0x5f ithread_loop() at ithread_loop+0xb2 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82404e9d30, rbp = 0 --- Tracing command intr pid 12 tid 100050 td 0xffffff0007880ae0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100049 td 0xffffff0007883000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff82404dfd30, rbp = 0 --- Tracing command intr pid 12 tid 100048 td 0xffffff00078833a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100046 td 0xffffff0007883ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d turnstile_wait() at turnstile_wait+0x243 _mtx_lock_sleep() at _mtx_lock_sleep+0xd6 _mtx_lock_flags() at _mtx_lock_flags+0xe1 ciss_perf_msi_intr() at ciss_perf_msi_intr+0x4c intr_event_execute_handlers() at intr_event_execute_handlers+0x66 ithread_loop() at ithread_loop+0xb2 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff824047ed30, rbp = 0 --- Tracing command intr pid 12 tid 100045 td 0xffffff000739cae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8240478d30, rbp = 0 --- Tracing command intr pid 12 tid 100044 td 0xffffff000742c000 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100043 td 0xffffff000742c3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff824043fd30, rbp = 0 --- Tracing command intr pid 12 tid 100038 td 0xffffff000742d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8240403d30, rbp = 0 --- Tracing command intr pid 12 tid 100037 td 0xffffff000742dae0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100035 td 0xffffff0007360740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000130d30, rbp = 0 --- Tracing command intr pid 12 tid 100034 td 0xffffff0007360ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800012bd30, rbp = 0 --- Tracing command intr pid 12 tid 100032 td 0xffffff000739a3a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100028 td 0xffffff000739c3a0 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x19a nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff80253d86, rsp = 0xffffff800004dfe0, rbp = 0xffffff800010cef0 --- ciss_get_request() at ciss_get_request+0x16 ciss_cam_action() at ciss_cam_action+0x2e2 xpt_run_dev_sendq() at xpt_run_dev_sendq+0x180 cam_periph_error() at cam_periph_error+0x465 dadone() at dadone+0xb3 camisr_runqueue() at camisr_runqueue+0x113 camisr() at camisr+0xd3 intr_event_execute_handlers() at intr_event_execute_handlers+0x66 ithread_loop() at ithread_loop+0xb2 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800010dd30, rbp = 0 --- Tracing command intr pid 12 tid 100020 td 0xffffff0007360000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000e5d30, rbp = 0 --- Tracing command intr pid 12 tid 100019 td 0xffffff00073603a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000e0d30, rbp = 0 --- Tracing command intr pid 12 tid 100018 td 0xffffff0007354ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000dbd30, rbp = 0 --- Tracing command intr pid 12 tid 100017 td 0xffffff000735c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000d6d30, rbp = 0 --- Tracing command intr pid 12 tid 100016 td 0xffffff000735c3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000d1d30, rbp = 0 --- Tracing command intr pid 12 tid 100015 td 0xffffff000735c740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000ccd30, rbp = 0 --- Tracing command intr pid 12 tid 100014 td 0xffffff000735cae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000c7d30, rbp = 0 --- Tracing command intr pid 12 tid 100013 td 0xffffff000735d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d turnstile_wait() at turnstile_wait+0x243 _mtx_lock_sleep() at _mtx_lock_sleep+0xd6 _mtx_lock_flags() at _mtx_lock_flags+0xe1 softclock() at softclock+0x248 intr_event_execute_handlers() at intr_event_execute_handlers+0x66 ithread_loop() at ithread_loop+0xb2 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000c2d30, rbp = 0 --- Tracing command intr pid 12 tid 100012 td 0xffffff000735d3a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100011 td 0xffffff000734a3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000b8d30, rbp = 0 --- Tracing command idle pid 11 tid 100010 td 0xffffff000734a740 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x19a nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff805e2446, rsp = 0xffffffff80ae73c0, rbp = 0xffffff80000b3b60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000b3d30, rbp = 0 --- Tracing command idle pid 11 tid 100009 td 0xffffff000734aae0 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x19a nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff805e2446, rsp = 0xffffff8000046fe0, rbp = 0xffffff80000aeb60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000aed30, rbp = 0 --- Tracing command idle pid 11 tid 100008 td 0xffffff0007354000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sched_idletd() at sched_idletd+0x26c fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000a9d30, rbp = 0 --- Tracing command idle pid 11 tid 100007 td 0xffffff00073543a0 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x19a nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff805e2446, rsp = 0xffffff8000054fe0, rbp = 0xffffff80000a4b60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000a4d30, rbp = 0 --- Tracing command idle pid 11 tid 100006 td 0xffffff0007354740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x70 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9b --- interrupt, rip = 0xffffffff805e2446, rsp = 0xffffff800009fb50, rbp = 0xffffff800009fb60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800009fd30, rbp = 0 --- Tracing command idle pid 11 tid 100005 td 0xffffff0007349000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sched_preempt() at sched_preempt+0xbc ipi_bitmap_handler() at ipi_bitmap_handler+0x70 Xipi_intr_bitmap_handler() at Xipi_intr_bitmap_handler+0x9b --- interrupt, rip = 0xffffffff805e2446, rsp = 0xffffff800009ab50, rbp = 0xffffff800009ab60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800009ad30, rbp = 0 --- Tracing command idle pid 11 tid 100004 td 0xffffff00073493a0 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x19a nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff805e2446, rsp = 0xffffff8000069fe0, rbp = 0xffffff8000095b60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000095d30, rbp = 0 --- Tracing command idle pid 11 tid 100003 td 0xffffff0007349740 cpustop_handler() at cpustop_handler+0x40 ipi_nmi_handler() at ipi_nmi_handler+0x30 trap() at trap+0x19a nmi_calltrap() at nmi_calltrap+0x8 --- trap 0x13, rip = 0xffffffff805e2446, rsp = 0xffffff8000070fe0, rbp = 0xffffff8000090b60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000090d30, rbp = 0 --- Tracing command init pid 1 tid 100002 td 0xffffff0007349ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x102 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip = 0x40c49c, rsp = 0x7fffffffe808, rbp = 0x401d40 --- Tracing command audit pid 10 tid 100001 td 0xffffff000734a000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a audit_worker() at audit_worker+0x77 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000086d30, rbp = 0 --- Tracing command kernel pid 0 tid 100131 td 0xffffff00153c6740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242c22d30, rbp = 0 --- Tracing command kernel pid 0 tid 100130 td 0xffffff00153c6ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242c1dd30, rbp = 0 --- Tracing command kernel pid 0 tid 100129 td 0xffffff00153c7000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242c18d30, rbp = 0 --- Tracing command kernel pid 0 tid 100128 td 0xffffff00153c73a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242c13d30, rbp = 0 --- Tracing command kernel pid 0 tid 100127 td 0xffffff00153c7740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242c0ed30, rbp = 0 --- Tracing command kernel pid 0 tid 100126 td 0xffffff00153c7ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242c09d30, rbp = 0 --- Tracing command kernel pid 0 tid 100125 td 0xffffff00153c9000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242c04d30, rbp = 0 --- Tracing command kernel pid 0 tid 100124 td 0xffffff00153c93a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242bffd30, rbp = 0 --- Tracing command kernel pid 0 tid 100123 td 0xffffff00153c9740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242bfad30, rbp = 0 --- Tracing command kernel pid 0 tid 100122 td 0xffffff0007d64740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242bf5d30, rbp = 0 --- Tracing command kernel pid 0 tid 100121 td 0xffffff0007d64ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242bf0d30, rbp = 0 --- Tracing command kernel pid 0 tid 100120 td 0xffffff0007d68000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242bebd30, rbp = 0 --- Tracing command kernel pid 0 tid 100119 td 0xffffff0007d683a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242be6d30, rbp = 0 --- Tracing command kernel pid 0 tid 100118 td 0xffffff0007d68740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242be1d30, rbp = 0 --- Tracing command kernel pid 0 tid 100117 td 0xffffff0007d68ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242bdcd30, rbp = 0 --- Tracing command kernel pid 0 tid 100116 td 0xffffff0007d69000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242bd7d30, rbp = 0 --- Tracing command kernel pid 0 tid 100115 td 0xffffff0007d693a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242bd2d30, rbp = 0 --- Tracing command kernel pid 0 tid 100114 td 0xffffff0007d69740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242bcdd30, rbp = 0 --- Tracing command kernel pid 0 tid 100110 td 0xffffff0007d643a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242babd30, rbp = 0 --- Tracing command kernel pid 0 tid 100107 td 0xffffff0007d63ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b9cd30, rbp = 0 --- Tracing command kernel pid 0 tid 100106 td 0xffffff0007c713a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b97d30, rbp = 0 --- Tracing command kernel pid 0 tid 100105 td 0xffffff0007c71740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b92d30, rbp = 0 --- Tracing command kernel pid 0 tid 100104 td 0xffffff0007c71ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b8dd30, rbp = 0 --- Tracing command kernel pid 0 tid 100103 td 0xffffff0007d73740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b65d30, rbp = 0 --- Tracing command kernel pid 0 tid 100102 td 0xffffff0007d73ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b83d30, rbp = 0 --- Tracing command kernel pid 0 tid 100101 td 0xffffff0007d733a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b6fd30, rbp = 0 --- Tracing command kernel pid 0 tid 100100 td 0xffffff0007f64000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b79d30, rbp = 0 --- Tracing command kernel pid 0 tid 100099 td 0xffffff0007f643a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b74d30, rbp = 0 --- Tracing command kernel pid 0 tid 100098 td 0xffffff0007f64740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b88d30, rbp = 0 --- Tracing command kernel pid 0 tid 100097 td 0xffffff0007f64ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b7ed30, rbp = 0 --- Tracing command kernel pid 0 tid 100096 td 0xffffff0007f6a000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b6ad30, rbp = 0 --- Tracing command kernel pid 0 tid 100095 td 0xffffff0007f6a3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b4cd30, rbp = 0 --- Tracing command kernel pid 0 tid 100094 td 0xffffff0007f6a740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b5bd30, rbp = 0 --- Tracing command kernel pid 0 tid 100093 td 0xffffff0007f6aae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b60d30, rbp = 0 --- Tracing command kernel pid 0 tid 100092 td 0xffffff0007f6b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b51d30, rbp = 0 --- Tracing command kernel pid 0 tid 100091 td 0xffffff0007f6b3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b3dd30, rbp = 0 --- Tracing command kernel pid 0 tid 100090 td 0xffffff0007f6b740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b56d30, rbp = 0 --- Tracing command kernel pid 0 tid 100089 td 0xffffff0007f6bae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b42d30, rbp = 0 --- Tracing command kernel pid 0 tid 100088 td 0xffffff0007f6c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b47d30, rbp = 0 --- Tracing command kernel pid 0 tid 100087 td 0xffffff0007f6c3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b38d30, rbp = 0 --- Tracing command kernel pid 0 tid 100086 td 0xffffff0007f6c740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b33d30, rbp = 0 --- Tracing command kernel pid 0 tid 100085 td 0xffffff0007f6cae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242b2ed30, rbp = 0 --- Tracing command kernel pid 0 tid 100084 td 0xffffff0007f6d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242bb5d30, rbp = 0 --- Tracing command kernel pid 0 tid 100083 td 0xffffff0007f6d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242bb0d30, rbp = 0 --- Tracing command kernel pid 0 tid 100082 td 0xffffff0007f6d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8242bbad30, rbp = 0 --- Tracing command kernel pid 0 tid 100036 td 0xffffff000742f000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000135d30, rbp = 0 --- Tracing command kernel pid 0 tid 100033 td 0xffffff000739a000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000126d30, rbp = 0 --- Tracing command kernel pid 0 tid 100031 td 0xffffff000739a740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff800011cd30, rbp = 0 --- Tracing command kernel pid 0 tid 100030 td 0xffffff000739aae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000117d30, rbp = 0 --- Tracing command kernel pid 0 tid 100029 td 0xffffff000739c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff8000112d30, rbp = 0 --- Tracing command kernel pid 0 tid 100024 td 0xffffff000735e000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff80000f9d30, rbp = 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff808c7000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 scheduler() at scheduler+0x29c mi_startup() at mi_startup+0x59 btext() at btext+0x2c ----- (7) "db> call doadump" ----- (fails!) Physical memory: 20466 MB Dumping 5722 MB: From owner-freebsd-fs@FreeBSD.ORG Thu Mar 11 13:08:16 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 61DD4106566B for ; Thu, 11 Mar 2010 13:08:16 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id A19898FC14 for ; Thu, 11 Mar 2010 13:08:15 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 103FB45C9B; Thu, 11 Mar 2010 14:08:13 +0100 (CET) Received: from localhost (pdawidek.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 4EA0E45683; Thu, 11 Mar 2010 14:08:08 +0100 (CET) Date: Thu, 11 Mar 2010 14:08:07 +0100 From: Pawel Jakub Dawidek To: Michiel Detailleur Message-ID: <20100311130807.GJ2350@garage.freebsd.pl> References: <4B97A963.9040300@scoutsengidsenvlaanderen.be> <20100310174411.GE1715@garage.freebsd.pl> <4B98E95B.90806@scoutsengidsenvlaanderen.be> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="Ublo+h3cBgJ33ahC" Content-Disposition: inline In-Reply-To: <4B98E95B.90806@scoutsengidsenvlaanderen.be> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: [Spam? ] Re: ZFS snapshot name length limit? (File name too long) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 13:08:16 -0000 --Ublo+h3cBgJ33ahC Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Mar 11, 2010 at 02:00:11PM +0100, Michiel Detailleur wrote: > >From looking at the code, I think you hitting this limit: > > > >This is FreeBSD limit caused by statfs structure: > > > >/* > > * filesystem statistics > > */ > >[...] > >#define MNAMELEN 88 /* size of on/from name bufs */ > >[...] > > > >ZFS mounts snapshots on lookup and this is this mount that fails. > > =20 > So all that would need to be done to allow mounting snapshots with=20 > longer names would be to increase this value? (I'm not a C programmer,=20 > so bear with me :) ) Such change will break ABI compatibility with tools compiled on previous FreeBSD versions. As you can see in sys/sys/mount.h this is the reason we still keep ostatfs structure. I'll discuss the possibilities with other FreeBSD committers and we will see what we can do about it. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --Ublo+h3cBgJ33ahC Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkuY6zcACgkQForvXbEpPzTzCgCgpzZjbPw4RrG3VuSllVOzjzW9 J68AoKH4Lfqg2DZ06R4psYAb0ESuhXpE =ZnVM -----END PGP SIGNATURE----- --Ublo+h3cBgJ33ahC-- From owner-freebsd-fs@FreeBSD.ORG Thu Mar 11 14:08:33 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1BD081065673; Thu, 11 Mar 2010 14:08:33 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from mail.ebusiness-leidinger.de (mail.ebusiness-leidinger.de [217.11.53.44]) by mx1.freebsd.org (Postfix) with ESMTP id 6613E8FC17; Thu, 11 Mar 2010 14:08:32 +0000 (UTC) Received: from outgoing.leidinger.net (pD9E2DD69.dip.t-dialin.net [217.226.221.105]) by mail.ebusiness-leidinger.de (Postfix) with ESMTPSA id 11FA48444D8; Thu, 11 Mar 2010 15:08:26 +0100 (CET) Received: from webmail.leidinger.net (webmail.leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id E5BC5542A; Thu, 11 Mar 2010 15:08:22 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=Leidinger.net; s=outgoing-alex; t=1268316503; bh=DMkCh8DidYAAaGwTEv1a018dtLXIZDuI6Jlt3phmOZo=; h=Message-ID:Date:From:To:Cc:Subject:References:In-Reply-To: MIME-Version:Content-Type:Content-Transfer-Encoding; b=NOL1fV8lCJXp0iH/JcnisWgYTL3AEd4kq9uBMeeb9u7pVDMpDrszTunnAtAYlMeYJ DpYhBieMEocA0337evOknyRmxm2APEuPef049aCegHuH9BYrG+SxicW6RQUZMUCn6v KsI+c3ilPBrDJQEe8+VTgCCInWZlWI4oeiBt0GY+bFyIO5RxDx1RHjvL/W4zTMNtsS 6Jyabcyl73wNX9aYOsBgX+eTerwfZTGjlbiGbyqAYDgMPtAvm4vxvLduq9FWotBR63 E4kDkOk0xG8vC6KgTxyH1KAziqxJDC+YI+b3YRbGtSqjQDNes3EAqeSMaNNAcLkDbj CgNU4/4slVQMA== Received: (from www@localhost) by webmail.leidinger.net (8.14.3/8.13.8/Submit) id o2BE8MTD042722; Thu, 11 Mar 2010 15:08:22 +0100 (CET) (envelope-from Alexander@Leidinger.net) Received: from pslux.cec.eu.int (pslux.cec.eu.int [158.169.9.14]) by webmail.leidinger.net (Horde Framework) with HTTP; Thu, 11 Mar 2010 15:08:22 +0100 Message-ID: <20100311150822.107231cvjvgs9gsg@webmail.leidinger.net> Date: Thu, 11 Mar 2010 15:08:22 +0100 From: Alexander Leidinger To: Borja Marcos References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> <20100310173143.GD1715@garage.freebsd.pl> <20100311084527.2934034895hvgxaw@webmail.leidinger.net> <764BD545-B86C-47DC-9004-964EB2216AF0@sarenet.es> In-Reply-To: <764BD545-B86C-47DC-9004-964EB2216AF0@sarenet.es> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Dynamic Internet Messaging Program (DIMP) H3 (1.1.4) X-EBL-MailScanner-Information: Please contact the ISP for more information X-EBL-MailScanner-ID: 11FA48444D8.551BF X-EBL-MailScanner: Found to be clean X-EBL-MailScanner-SpamCheck: not spam, spamhaus-ZEN, SpamAssassin (not cached, score=-1.363, required 6, autolearn=disabled, ALL_TRUSTED -1.44, DKIM_SIGNED 0.00, DKIM_VERIFIED -0.00, TW_ZF 0.08) X-EBL-MailScanner-From: alexander@leidinger.net X-EBL-MailScanner-Watermark: 1268921306.69505@9ibt2Iy+sh0nBQqflFRYlw X-EBL-Spam-Status: No Cc: freebsd-fs@FreeBSD.org, Pawel Jakub Dawidek , FreeBSD@FreeBSD.org, Stable Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 14:08:33 -0000 Quoting Borja Marcos (from Thu, 11 Mar 2010 09:54:47 +0100): > > On Mar 11, 2010, at 8:45 AM, Alexander Leidinger wrote: > >> Quoting Pawel Jakub Dawidek (from Wed, 10 Mar >> 2010 18:31:43 +0100): >> >> There is a 4th possibility, if you can rule out everything else: >> bugs in the CPU. I stumbled upon this with ZFS (but UFS was >> exposing the problem much faster). The problem in my case was that >> the BIOS was not recognizing the CPU and as such was not uploading >> microcode updates. >> >> Borja, can you confirm that the CPU is correctly announced in >> FreeBSD (just look at "dmesg | grep CPU:" output, if it tells you >> it is a AMD or Intel XXX CPU it is correctly detected by the BIOS)? > > A CPU bug? Weird. Very. It depends. CPUs have bugs. You do not want to run any modern CPU without an microcode update. The BIOS is doing it for you at system start. I do not want to say that this is the problem you have, I just want to point out that it may be possible (but see below). I got hit by this last december and I was finding the solution (replacing the complete system, as only replacing the CPU was not an option) in January. > As the servers had to be rolled into production, and such tests with > real servers can be quite time consuming, I set up a couple of > FreeBSD virtual machines, using VMWare Fusion (version 2 then, now > version 3) on a Macbook (Macbook 4,1 Intel Core2Duo, 2.1 GHz) and > tried to reproduce it. > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Xeon(R) CPU L5420 @ 2.50GHz (2496.25-MHz > K8-class CPU) > Origin = "GenuineIntel" Id = 0x1067a Stepping = 10 > The virtual machine (VMWare Fusion 3.0.0, Macbook, Mac OS X 10.6.2) > reports this: > Timecounter "i8254" frequency 1193182 Hz quality 0 > CPU: Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz (2116.62-MHz > K8-class CPU) > Origin = "GenuineIntel" Id = 0x10676 Stepping = 6 Summary: you confirmed the problem on a different kind of CPU. Because of this it makes it even more unlikely that it is a CPU problem. Bye, Alexander. -- A good plan today is better than a perfect plan tomorrow. -- Patton http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-fs@FreeBSD.ORG Thu Mar 11 14:09:41 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 755301065670; Thu, 11 Mar 2010 14:09:41 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from mail.ebusiness-leidinger.de (mail.ebusiness-leidinger.de [217.11.53.44]) by mx1.freebsd.org (Postfix) with ESMTP id 086AA8FC17; Thu, 11 Mar 2010 14:09:41 +0000 (UTC) Received: from outgoing.leidinger.net (pD9E2DD69.dip.t-dialin.net [217.226.221.105]) by mail.ebusiness-leidinger.de (Postfix) with ESMTPSA id C614C8444D9; Thu, 11 Mar 2010 15:09:36 +0100 (CET) Received: from webmail.leidinger.net (webmail.leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id 0216A542B; Thu, 11 Mar 2010 15:09:33 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=Leidinger.net; s=outgoing-alex; t=1268316574; bh=T4U5PK74rC/89MAeNS2ju+5QO61PySgkmqG4/FBVMRU=; h=Message-ID:Date:From:To:Cc:Subject:References:In-Reply-To: MIME-Version:Content-Type:Content-Transfer-Encoding; b=VIw09OfpFPsHM0te/srzc4ONgn+hkdmXFVKc6kFrqXRk75aq7ooaPSVX55UDZ807G RZToslMqoEFZuj3SoPg7zpvtyJh4dtIjYHkor5sY/glUEsJt0jbF5U33M7lFY5v0oI XdNt7aJ6z6DDDY0YAQ412/kraRXScfHOBcBlQxkInxoadsI+PCJdPKyxKCIPJ+qPD5 vSbn6Ksf/H9NpaTQJKSUinQVUBdHxSs7s90Avx4EFbU6vQjTAsjjPBBfvXEpAHjTb4 fhmV98m9erz6yfOg1ilZ3jfmd062cOouP5oLSuJKifnO+Vnd5mNGsZnEEy6CsPaGPp M2aLz8nu7Yofw== Received: (from www@localhost) by webmail.leidinger.net (8.14.3/8.13.8/Submit) id o2BE9XUp042953; Thu, 11 Mar 2010 15:09:33 +0100 (CET) (envelope-from Alexander@Leidinger.net) Received: from pslux.cec.eu.int (pslux.cec.eu.int [158.169.9.14]) by webmail.leidinger.net (Horde Framework) with HTTP; Thu, 11 Mar 2010 15:09:33 +0100 Message-ID: <20100311150933.18276bypviae5mw4@webmail.leidinger.net> Date: Thu, 11 Mar 2010 15:09:33 +0100 From: Alexander Leidinger To: Ivan Voras References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> <20100310173143.GD1715@garage.freebsd.pl> <20100311084527.2934034895hvgxaw@webmail.leidinger.net> <764BD545-B86C-47DC-9004-964EB2216AF0@sarenet.es> In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Dynamic Internet Messaging Program (DIMP) H3 (1.1.4) X-EBL-MailScanner-Information: Please contact the ISP for more information X-EBL-MailScanner-ID: C614C8444D9.78C0E X-EBL-MailScanner: Found to be clean X-EBL-MailScanner-SpamCheck: not spam, spamhaus-ZEN, SpamAssassin (not cached, score=-1.363, required 6, autolearn=disabled, ALL_TRUSTED -1.44, DKIM_SIGNED 0.00, DKIM_VERIFIED -0.00, TW_ZF 0.08) X-EBL-MailScanner-From: alexander@leidinger.net X-EBL-MailScanner-Watermark: 1268921377.22527@02IFKIAdn8Etlq6wt1Ru3g X-EBL-Spam-Status: No Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 14:09:41 -0000 Quoting Ivan Voras (from Thu, 11 Mar 2010 11:59:01 +0100): > On 03/11/10 09:54, Borja Marcos wrote: > > I don't know about the rest but this: > >> CPU: Intel(R) Xeon(R) CPU L5420 @ 2.50GHz (2496.25-MHz >> K8-class CPU) > > does not agree with this: > >> FreeBSD/SMP: 1 package(s) x 8 core(s) > > The Xeon 54xx series does not come in 8 core packages. Either it is > 2xquad-core or a Xeon 55xx. Can also be a problem in the layout detection logic... Bye, Alexander. -- If we can ever make red tape nutritional, we can feed the world. -- R. Schaeberle, "Management Accounting" http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-fs@FreeBSD.ORG Thu Mar 11 17:26:12 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AACC5106564A; Thu, 11 Mar 2010 17:26:12 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop1.sarenet.es (proxypop1.sarenet.es [194.30.0.99]) by mx1.freebsd.org (Postfix) with ESMTP id 5EE118FC0C; Thu, 11 Mar 2010 17:26:11 +0000 (UTC) Received: from [172.16.1.204] (unknown [192.148.167.2]) by proxypop1.sarenet.es (Postfix) with ESMTP id 8EAA06158; Thu, 11 Mar 2010 18:26:10 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20100311150822.107231cvjvgs9gsg@webmail.leidinger.net> Date: Thu, 11 Mar 2010 18:26:09 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <35ADB9B1-F571-4EE4-9089-5363ACEBE159@sarenet.es> References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> <20100310173143.GD1715@garage.freebsd.pl> <20100311084527.2934034895hvgxaw@webmail.leidinger.net> <764BD545-B86C-47DC-9004-964EB2216AF0@sarenet.es> <20100311150822.107231cvjvgs9gsg@webmail.leidinger.net> To: Alexander Leidinger X-Mailer: Apple Mail (2.1077) Cc: freebsd-fs@FreeBSD.org, Pawel Jakub Dawidek , FreeBSD@FreeBSD.org, Stable Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 17:26:12 -0000 On Mar 11, 2010, at 3:08 PM, Alexander Leidinger wrote: >>> Borja, can you confirm that the CPU is correctly announced in = FreeBSD (just look at "dmesg | grep CPU:" output, if it tells you it is = a AMD or Intel XXX CPU it is correctly detected by the BIOS)? >>=20 >> A CPU bug? Weird. Very. >=20 > It depends. CPUs have bugs. You do not want to run any modern CPU = without an microcode update. The BIOS is doing it for you at system = start. >=20 > I do not want to say that this is the problem you have, I just want to = point out that it may be possible (but see below). I got hit by this = last december and I was finding the solution (replacing the complete = system, as only replacing the CPU was not an option) in January. Of course CPUs have bugs, I don't doubt it. I was just wondering how I = coud reproduce the problem with a different hardware :) That's why I = said it was unlikely. Besides, such a low level fault should produce many more problems than = such a well defined failure mode, as far as I know. >> As the servers had to be rolled into production, and such tests with = real servers can be quite time consuming, I set up a couple of FreeBSD = virtual machines, using VMWare Fusion (version 2 then, now version 3) on = a Macbook (Macbook 4,1 Intel Core2Duo, 2.1 GHz) and tried to reproduce = it. >=20 >> Timecounter "i8254" frequency 1193182 Hz quality 0 >> CPU: Intel(R) Xeon(R) CPU L5420 @ 2.50GHz (2496.25-MHz = K8-class CPU) >> Origin =3D "GenuineIntel" Id =3D 0x1067a Stepping =3D 10 >=20 >> The virtual machine (VMWare Fusion 3.0.0, Macbook, Mac OS X 10.6.2) = reports this: >> Timecounter "i8254" frequency 1193182 Hz quality 0 >> CPU: Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz (2116.62-MHz = K8-class CPU) >> Origin =3D "GenuineIntel" Id =3D 0x10676 Stepping =3D 6 >=20 > Summary: you confirmed the problem on a different kind of CPU. >=20 > Because of this it makes it even more unlikely that it is a CPU = problem. Indeed :) Borja. >=20 > Bye, > Alexander. >=20 > --=20 > A good plan today is better than a perfect plan tomorrow. > -- Patton >=20 > http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID =3D = B0063FE7 > http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID =3D = 72077137 From owner-freebsd-fs@FreeBSD.ORG Thu Mar 11 18:30:05 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EFA69106566B for ; Thu, 11 Mar 2010 18:30:05 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id A98048FC16 for ; Thu, 11 Mar 2010 18:30:05 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1Npn92-0000pb-8e for freebsd-fs@freebsd.org; Thu, 11 Mar 2010 19:30:04 +0100 Received: from 207.155.204.151.ptr.us.xo.net ([207.155.204.151]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 11 Mar 2010 19:30:04 +0100 Received: from atkin901 by 207.155.204.151.ptr.us.xo.net with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 11 Mar 2010 19:30:04 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Mark Atkinson Date: Thu, 11 Mar 2010 10:25:17 -0800 Lines: 20 Message-ID: References: <20100311133916.42ba69b0@orwell.free.de> Mime-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: 207.155.204.151.ptr.us.xo.net User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.1.8) Gecko/20100303 Thunderbird/3.0.3 In-Reply-To: <20100311133916.42ba69b0@orwell.free.de> Cc: freebsd-stable@freebsd.org Subject: Re: proliant server lockups with freebsd-amd64-stable (2010-03-10) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 18:30:06 -0000 On 03/11/10 04:39, Kai Gallasch wrote: > Hi. > > I have some trouble with an opteron server locking up spontaneously. It looses > all networks connectivity and even through console I can get no shell. > > Lockups occur mostly under disk load (periodic daily, bacula backup > running, make buildworld/buildkernel) and I can provoke them easily. try adding: vm.pmap.pg_ps_enabled="0" to /boot/loader.conf and reboot. Long thread reference: http://thread.gmane.org/gmane.os.freebsd.current/121035/focus=121457 possible AMD "errata 147" From owner-freebsd-fs@FreeBSD.ORG Thu Mar 11 18:57:35 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A7F711065680; Thu, 11 Mar 2010 18:57:35 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id B494B8FC19; Thu, 11 Mar 2010 18:57:34 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id UAA09304; Thu, 11 Mar 2010 20:57:30 +0200 (EET) (envelope-from avg@icyb.net.ua) Message-ID: <4B993D19.60504@icyb.net.ua> Date: Thu, 11 Mar 2010 20:57:29 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.23 (X11/20100211) MIME-Version: 1.0 To: Mark Atkinson References: <20100311133916.42ba69b0@orwell.free.de> In-Reply-To: X-Enigmail-Version: 0.95.7 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: proliant server lockups with freebsd-amd64-stable (2010-03-10) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 18:57:35 -0000 on 11/03/2010 20:25 Mark Atkinson said the following: > On 03/11/10 04:39, Kai Gallasch wrote: >> Hi. >> >> I have some trouble with an opteron server locking up spontaneously. It looses >> all networks connectivity and even through console I can get no shell. >> >> Lockups occur mostly under disk load (periodic daily, bacula backup >> running, make buildworld/buildkernel) and I can provoke them easily. > > try adding: > > vm.pmap.pg_ps_enabled="0" The lockups happen with this setting too. This is likely a red herring, because Kai already has Alan's workaround from head. > to /boot/loader.conf and reboot. > > Long thread reference: > > http://thread.gmane.org/gmane.os.freebsd.current/121035/focus=121457 > > possible AMD "errata 147" 147? -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Thu Mar 11 19:09:57 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C9CE5106566B for ; Thu, 11 Mar 2010 19:09:57 +0000 (UTC) (envelope-from freebsd-fs@m.gmane.org) Received: from lo.gmane.org (lo.gmane.org [80.91.229.12]) by mx1.freebsd.org (Postfix) with ESMTP id 55B2A8FC17 for ; Thu, 11 Mar 2010 19:09:57 +0000 (UTC) Received: from list by lo.gmane.org with local (Exim 4.69) (envelope-from ) id 1Npnlc-0001JL-1S for freebsd-fs@freebsd.org; Thu, 11 Mar 2010 20:09:56 +0100 Received: from 207.155.204.151.ptr.us.xo.net ([207.155.204.151]) by main.gmane.org with esmtp (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 11 Mar 2010 20:09:56 +0100 Received: from atkin901 by 207.155.204.151.ptr.us.xo.net with local (Gmexim 0.1 (Debian)) id 1AlnuQ-0007hv-00 for ; Thu, 11 Mar 2010 20:09:56 +0100 X-Injected-Via-Gmane: http://gmane.org/ To: freebsd-fs@freebsd.org From: Mark Atkinson Date: Thu, 11 Mar 2010 11:09:43 -0800 Lines: 52 Message-ID: References: <20100311133916.42ba69b0@orwell.free.de> <4B993D19.60504@icyb.net.ua> Mime-Version: 1.0 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Complaints-To: usenet@dough.gmane.org X-Gmane-NNTP-Posting-Host: 207.155.204.151.ptr.us.xo.net User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.9.1.8) Gecko/20100303 Thunderbird/3.0.3 In-Reply-To: <4B993D19.60504@icyb.net.ua> Cc: freebsd-stable@freebsd.org Subject: Re: proliant server lockups with freebsd-amd64-stable (2010-03-10) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 19:09:57 -0000 On 03/11/10 10:57, Andriy Gapon wrote: > on 11/03/2010 20:25 Mark Atkinson said the following: >> On 03/11/10 04:39, Kai Gallasch wrote: >>> Hi. >>> >>> I have some trouble with an opteron server locking up spontaneously. It looses >>> all networks connectivity and even through console I can get no shell. >>> >>> Lockups occur mostly under disk load (periodic daily, bacula backup >>> running, make buildworld/buildkernel) and I can provoke them easily. >> >> try adding: >> >> vm.pmap.pg_ps_enabled="0" > > The lockups happen with this setting too. > This is likely a red herring, because Kai already has Alan's workaround from head. I'm not sure that the workaround covered the processor listed here > CPU: Quad-Core AMD Opteron(tm) Processor 2352 (2100.08-MHz K8-class CPU) > Origin = "AuthenticAMD" Id = 0x100f23 Stepping = 3 (mine is the same number and wasn't covered by the coded workaround commited to Linux, or IIRC Freebsd -- I'll have to look again). Turning off superpages totally solves the problem for me. But for Kai's problem, as Andriy points out, turning it off may not solve it. If I turn off superpages and enable MCE's I'll get the MCE when hitting the condition, so I just leave the MCE code off. >> to /boot/loader.conf and reboot. >> >> Long thread reference: >> >> http://thread.gmane.org/gmane.os.freebsd.current/121035/focus=121457 >> >> possible AMD "errata 147" > > 147? > I thought this problem was speculated to be from the bug: http://bugzilla.kernel.org/show_bug.cgi?id=11305 Which references an AMD "errata 147" from "Revision Guide for AMD Athlon™ 64 and AMD Opteron™ Processors." http://support.amd.com/us/Processor_TechDocs/25759.pdf From owner-freebsd-fs@FreeBSD.ORG Thu Mar 11 20:11:29 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 70F5F106564A for ; Thu, 11 Mar 2010 20:11:29 +0000 (UTC) (envelope-from jakub.rajman@gmail.com) Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.158]) by mx1.freebsd.org (Postfix) with ESMTP id 0CD428FC19 for ; Thu, 11 Mar 2010 20:11:28 +0000 (UTC) Received: by fg-out-1718.google.com with SMTP id 16so43942fgg.13 for ; Thu, 11 Mar 2010 12:11:27 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:date:message-id:subject :from:to:content-type; bh=m5xKhZMxvLCS1S8csz9kVankQeGVm+7UTk78HVDSe8A=; b=jPJrZXjy+rnYCwsSTvb5A0AL33LKtRnENHUkE0mAMIAXqfBR5tts/MxhULa7zsARCc btBJN6yhPfQ0LNDZMbpohppj3Iw/Ifw0lC6wD/GRj2+4G1SLLsnMbNh/rMZ93u6ttQDF lBRbu75B7n8A6dA5xVsxYZgdPch9pxqXhvmg8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=ttAsTcugX6r7KYqwx0dttXZsc2PByDA+G20M77gL3lYrO9vA3JU8xLvBRLQmKY2dE6 ecVsbLvp60jUOMaV3NUd/I+C/bZzMGi+gpS+qzGXLccC6Gh4EAyqncreX+FePyb2FKK2 a8Tw/RyJzc4j6AavAAuhywrmCM3pWg7/s4nrM= MIME-Version: 1.0 Received: by 10.103.48.21 with SMTP id a21mr1169355muk.98.1268336672129; Thu, 11 Mar 2010 11:44:32 -0800 (PST) Date: Thu, 11 Mar 2010 20:44:32 +0100 Message-ID: <37b87d3d1003111144j38d8c9edsb1c80d8009149d73@mail.gmail.com> From: Jakub Rajman To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: Example of simple filesystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Mar 2010 20:11:29 -0000 Hello, I am going to write my own simple file system and run it under the FreeBSD OS. I'm looking for some inspiration in existing file systems (UFS2, msdosfs,...), but they are too complex and complikated. So I would like to ask, if exists some simple "experimental" filesystems? Some workable (and mountable) examples with their source codes. Thank you for any help From owner-freebsd-fs@FreeBSD.ORG Fri Mar 12 08:10:45 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BA2E2106567A; Fri, 12 Mar 2010 08:10:45 +0000 (UTC) (envelope-from alexander@leidinger.net) Received: from mail.ebusiness-leidinger.de (mail.ebusiness-leidinger.de [217.11.53.44]) by mx1.freebsd.org (Postfix) with ESMTP id 3E6EC8FC13; Fri, 12 Mar 2010 08:10:45 +0000 (UTC) Received: from outgoing.leidinger.net (pD9E2D538.dip.t-dialin.net [217.226.213.56]) by mail.ebusiness-leidinger.de (Postfix) with ESMTPSA id 434288453C9; Fri, 12 Mar 2010 09:10:39 +0100 (CET) Received: from webmail.leidinger.net (webmail.leidinger.net [192.168.1.102]) by outgoing.leidinger.net (Postfix) with ESMTP id AB6A654CE; Fri, 12 Mar 2010 09:10:35 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=Leidinger.net; s=outgoing-alex; t=1268381435; bh=ztVDoIz8rrYJYFhIrv+VOWSa+dq8hHAR593pDFwMG/Q=; h=Message-ID:Date:From:To:Cc:Subject:References:In-Reply-To: MIME-Version:Content-Type:Content-Transfer-Encoding; b=Rq8NA3leV7MKGmhZAC2eaOzyT8/RGzuq4X0JUrVxDYxVU7ia6ML/f+CDpwN35jGRX D2lkIMc8GwdgIH75ZsmChI+JOWU5Sjd1mFJTyMoKazdVOh7SCEuJR0ERhV7+kNGdDM qHRtXzM+vN+oz5UDxBytIHMTm3oMqRY2qcfAQnR2YcZW9ebSw+ia8w4qo4MYGlzAyd 7ZV1CgH/jNrYUoA5KRXG7w71PXfAM2LXKwOYKJtLCE5DiT2uP39YVuXCcuFsgvl8uL GJFHsS7Ha8ElcCS4dW5yzW+QlfIEK+5vCUz9OXUd3c6jiJQddfoKwndvcjTkprkjjk NzFPXLLcBaHQQ== Received: (from www@localhost) by webmail.leidinger.net (8.14.3/8.13.8/Submit) id o2C8AZZf093080; Fri, 12 Mar 2010 09:10:35 +0100 (CET) (envelope-from Alexander@Leidinger.net) Received: from pslux.cec.eu.int (pslux.cec.eu.int [158.169.9.14]) by webmail.leidinger.net (Horde Framework) with HTTP; Fri, 12 Mar 2010 09:10:35 +0100 Message-ID: <20100312091035.961823cwec0rxijo@webmail.leidinger.net> Date: Fri, 12 Mar 2010 09:10:35 +0100 From: Alexander Leidinger To: Borja Marcos References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> <20100310173143.GD1715@garage.freebsd.pl> <20100311084527.2934034895hvgxaw@webmail.leidinger.net> <764BD545-B86C-47DC-9004-964EB2216AF0@sarenet.es> <20100311150822.107231cvjvgs9gsg@webmail.leidinger.net> <35ADB9B1-F571-4EE4-9089-5363ACEBE159@sarenet.es> In-Reply-To: <35ADB9B1-F571-4EE4-9089-5363ACEBE159@sarenet.es> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8; DelSp="Yes"; format="flowed" Content-Disposition: inline Content-Transfer-Encoding: 7bit User-Agent: Dynamic Internet Messaging Program (DIMP) H3 (1.1.4) X-EBL-MailScanner-Information: Please contact the ISP for more information X-EBL-MailScanner-ID: 434288453C9.E4D6B X-EBL-MailScanner: Found to be clean X-EBL-MailScanner-SpamCheck: not spam, spamhaus-ZEN, SpamAssassin (not cached, score=-1.363, required 6, autolearn=disabled, ALL_TRUSTED -1.44, DKIM_SIGNED 0.00, DKIM_VERIFIED -0.00, TW_ZF 0.08) X-EBL-MailScanner-From: alexander@leidinger.net X-EBL-MailScanner-Watermark: 1268986240.38003@u8CKQTup46wL395ZMBCtEw X-EBL-Spam-Status: No Cc: freebsd-fs@FreeBSD.org, Pawel Jakub Dawidek , FreeBSD@FreeBSD.org, Stable Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Mar 2010 08:10:45 -0000 Quoting Borja Marcos (from Thu, 11 Mar 2010 18:26:09 +0100): > Of course CPUs have bugs, I don't doubt it. I was just wondering how > I coud reproduce the problem with a different hardware :) That's why > I said it was unlikely. > > Besides, such a low level fault should produce many more problems > than such a well defined failure mode, as far as I know. In my case I had a 7.1 system which was running fine. After updating to 7.2 I got deadlocks after some minutes with UFS. Switching to ZFS for the main data partition extended the lifetime to 3-4 hours. After updating ZFS in 7-stable with the code from 8-stable this was extended to 6 hours (periodic daily triggered the problem faster), and after switching to exclusive locks instead of shared locks in ZFS the system survived a night with several jails running periodic daily (but I had to reboot in the morning because apache was not able to serve data anymore). Everything else was working correctly. I would say this was a very narrow problem case. Bye, Alexander. -- Adult, n.: One old enough to know better. http://www.Leidinger.net Alexander @ Leidinger.net: PGP ID = B0063FE7 http://www.FreeBSD.org netchild @ FreeBSD.org : PGP ID = 72077137 From owner-freebsd-fs@FreeBSD.ORG Fri Mar 12 10:12:55 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 31B98106566B; Fri, 12 Mar 2010 10:12:55 +0000 (UTC) (envelope-from borjam@sarenet.es) Received: from proxypop1.sarenet.es (proxypop1.sarenet.es [194.30.0.99]) by mx1.freebsd.org (Postfix) with ESMTP id 0D66D8FC1B; Fri, 12 Mar 2010 10:12:52 +0000 (UTC) Received: from [172.16.1.204] (unknown [192.148.167.2]) by proxypop1.sarenet.es (Postfix) with ESMTP id 7FEA45E5A; Fri, 12 Mar 2010 11:12:51 +0100 (CET) Mime-Version: 1.0 (Apple Message framework v1077) Content-Type: text/plain; charset=us-ascii From: Borja Marcos In-Reply-To: <20100310173143.GD1715@garage.freebsd.pl> Date: Fri, 12 Mar 2010 11:12:49 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <27A6DF40-2EA8-4CFC-9E42-DD995E2F9342@sarenet.es> References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> <20100310173143.GD1715@garage.freebsd.pl> To: Pawel Jakub Dawidek X-Mailer: Apple Mail (2.1077) Cc: freebsd-fs@freebsd.org Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Mar 2010 10:12:55 -0000 On Mar 10, 2010, at 6:31 PM, Pawel Jakub Dawidek wrote: > Hmm, interesting. Especially those two traces: >=20 > Tracing command zfs pid 1820 tid 100105 td 0xffffff0002ca4000 Just in case something was wrong, I wiped the two virtual machines and = started everything again. The two virtual systems were acting a bit = weird, giving lots of LORs. I don't know why, maybe something was = corrupted, these virtual machines have been panicked and restarted many = times. So, experiment repeated again, fresh start. Still deadlocking pretty = soon if I start, say, 5 or 6 tar processes. Traces follow. These are the new traces. pruebazfs3# cat ddb.txt=20 db> set $lines=3D0 db> ps pid ppid pgrp uid state wmesg wchan cmd 4089 1014 4089 0 R+ CPU 0 sysctl 4088 4077 4077 2 S zfsvfs-> 0xffffff0002e5a140 dd 4077 4074 4077 2 Ss wait 0xffffff0002eec460 sh 4076 4073 4076 0 Ss zfsvfs-> 0xffffff0002e5a140 newsyslog 4074 929 929 0 S piperd 0xffffff000577bb60 cron 4073 929 929 0 S piperd 0xffffff00026175b0 cron 4072 4070 4070 0 S tx->tx_s 0xffffff00029f2e50 zfs 4070 4068 4070 0 Ss pause 0xffffff00057120a0 csh 4068 918 4068 0 Ss select 0xffffff001d228e40 sshd 4056 4054 4054 0 S+ zfs 0xffffff002bce2308 bsdtar 4055 4054 4054 0 S+ pipdwt 0xffffff00359945b0 bsdtar 4054 1014 4054 0 S+ wait 0xffffff001d6ee000 sh 4053 4051 4051 0 S+ zfsvfs-> 0xffffff0002e5a140 bsdtar 4052 4051 4051 0 S+ pipdwt 0xffffff00026472d8 bsdtar 4051 1014 4051 0 S+ wait 0xffffff001d6ef8c0 sh 4050 4048 4048 0 S+ zfs 0xffffff002bd407e8 bsdtar 4049 4048 4048 0 S+ pipdwt 0xffffff0002617888 bsdtar 4048 1014 4048 0 S+ wait 0xffffff001d6d4000 sh 4047 4045 4045 0 S+ zfs 0xffffff002bce2308 bsdtar 4046 4045 4045 0 S+ pipdwt 0xffffff0002647888 bsdtar 4045 1014 4045 0 S+ wait 0xffffff0002eec8c0 sh 2719 2718 2719 0 S+ ttyin 0xffffff003de62ca8 csh 2718 2715 2718 1001 S+ wait 0xffffff0002eed000 su 2715 2714 2715 1001 Ss+ pause 0xffffff0002eeb960 csh 2714 2711 2711 1001 S select 0xffffff0002f160c0 sshd 2711 918 2711 0 Ss sbwait 0xffffff002bbf9694 sshd 2706 2704 2704 0 S+ piperd 0xffffff00026475b0 bsdtar 2705 2704 2704 0 S+ rrl->rr_ 0xffffff0002e5a110 bsdtar 2704 1014 2704 0 S+ wait 0xffffff0002eec000 sh 1014 1013 1014 0 R+ csh 1013 1010 1013 1001 S+ wait 0xffffff0002691460 su 1010 1009 1010 1001 Ss+ pause 0xffffff0002692500 csh 1009 1005 1005 1001 R sshd 1005 918 1005 0 Ss sbwait 0xffffff0002c01694 sshd 1001 1 1001 0 Ss+ ttyin 0xffffff000253e4a8 getty 1000 1 1000 0 Ss+ ttyin 0xffffff000252aca8 getty 999 1 999 0 Ss+ ttyin 0xffffff00025504a8 getty 998 1 998 0 Ss+ ttyin 0xffffff00025500a8 getty 997 1 997 0 Ss+ ttyin 0xffffff00025510a8 getty 996 1 996 0 Ss+ ttyin 0xffffff0002550ca8 getty 995 1 995 0 Ss+ ttyin 0xffffff000254f0a8 getty 994 1 994 0 Ss+ ttyin 0xffffff000254eca8 getty 929 1 929 0 Ss nanslp 0xffffffff80c4c768 cron 918 1 918 0 Ss select 0xffffff0002bbec40 sshd 643 1 643 0 Ss select 0xffffff0002bf0940 syslogd 521 1 521 0 Ss select 0xffffff00026bb9c0 devd 40 0 0 0 SL (threaded) zfskern 100093 D tx->tx_q 0xffffff00029f2e70 = [txg_thread_enter] 100092 D tx->tx_c 0xffffff00025f5520 = [txg_thread_enter] 100090 D vgeom:io 0xffffff00028c1590 [vdev da4] 100089 D vgeom:io 0xffffff00028bf790 [vdev da3] 100088 D vgeom:io 0xffffff00028c1410 [vdev da2] 100087 D vgeom:io 0xffffff00028c1290 [vdev da1] 100060 D l2arc_fe 0xffffffff81311640 = [l2arc_feed_thread] 100054 D arc_recl 0xffffffff81309060 = [arc_reclaim_thread] 20 0 0 0 SL flowclea 0xffffffff80e0df50 = [flowcleaner] 19 0 0 0 SL sdflush 0xffffffff80e1ceb8 = [softdepflush] 18 0 0 0 SL zfsvfs-> 0xffffff0002e5a140 [vnlru] 17 0 0 0 SL zfsvfs-> 0xffffff0002b73300 [syncer] 16 0 0 0 SL psleep 0xffffffff80e0d7c8 [bufdaemon] 15 0 0 0 SL pgzero 0xffffffff80e1e94c [pagezero] 9 0 0 0 SL psleep 0xffffffff80e1dce8 [vmdaemon] 8 0 0 0 SL psleep 0xffffffff80e1dcac = [pagedaemon] 7 0 0 0 SL ccb_scan 0xffffffff80c15660 [xpt_thrd] 6 0 0 0 SL - 0xffffff00024b6848 [fdc0] 14 0 0 0 SL (threaded) usb 100035 D - 0xffffff8000282dd0 [usbus1] 100034 D - 0xffffff8000282d78 [usbus1] 100033 D - 0xffffff8000282d20 [usbus1] 100032 D - 0xffffff8000282cc8 [usbus1] 100030 D - 0xffffff8000267ef0 [usbus0] 100029 D - 0xffffff8000267e98 [usbus0] 100028 D - 0xffffff8000267e40 [usbus0] 100027 D - 0xffffff8000267de8 [usbus0] 5 0 0 0 SL idle 0xffffff800024c000 = [mpt_recovery0] 13 0 0 0 SL - 0xffffffff80c4c444 [yarrow] 4 0 0 0 SL - 0xffffffff80c48c48 [g_down] 3 0 0 0 SL - 0xffffffff80c48c40 [g_up] 2 0 0 0 SL - 0xffffffff80c48c30 [g_event] 12 0 0 0 WL (threaded) intr 100039 I [swi0: uart = uart] 100038 I [irq7: = ppc0] 100037 I [irq12: = psm0] 100036 I [irq1: = atkbd0] 100026 I [irq18: = uhci0] 100024 I [irq17: = mpt0 ehci0] 100023 I [irq15: = ata1] 100022 I [irq14: = ata0] 100021 I [irq9: = acpi0] 100020 I [swi2: = cambio] 100015 I [swi6: task = queue] 100014 I [swi6: = Giant taskq] 100012 I [swi5: +] 100006 I [swi3: vm] 100005 I [swi1: = netisr 0] 100004 I [swi4: = clock] 11 0 0 0 RL [idle: = cpu0] 1 0 1 0 SLs wait 0xffffff00022408c0 [init] 10 0 0 0 SL audit_wo 0xffffffff80e1c210 [audit] 0 0 0 0 SLs (threaded) kernel 100097 D - 0xffffff0002bbe100 [zil_clean] 100095 D - 0xffffff0002bb4800 [zil_clean] 100094 D - 0xffffff00028c0900 [zil_clean] 100091 D - 0xffffff00026e4280 = [zfs_vn_rele_taskq] 100086 D - 0xffffff00028c1100 [spa_zio] 100085 D - 0xffffff00026e4880 [spa_zio] 100084 D - 0xffffff00028bf480 [spa_zio] 100083 D - 0xffffff00028bf400 [spa_zio] 100082 D - 0xffffff00028bf300 [spa_zio] 100081 D - 0xffffff00028bf500 [spa_zio] 100080 D - 0xffffff00025ccb00 [spa_zio] 100079 D - 0xffffff000255e700 [spa_zio_7] 100078 D - 0xffffff000255e700 [spa_zio_6] 100077 D - 0xffffff000255e700 [spa_zio_5] 100076 D - 0xffffff000255e700 [spa_zio_4] 100075 D - 0xffffff000255e700 [spa_zio_3] 100074 D - 0xffffff000255e700 [spa_zio_2] 100073 D - 0xffffff000255e700 [spa_zio_1] 100072 D - 0xffffff000255e700 [spa_zio_0] 100071 D - 0xffffff00025cc980 [spa_zio_7] 100070 D - 0xffffff00025cc980 [spa_zio_6] 100069 D - 0xffffff00025cc980 [spa_zio_5] 100068 D - 0xffffff00025cc980 [spa_zio_4] 100067 D - 0xffffff00025cc980 [spa_zio_3] 100066 D - 0xffffff00025cc980 [spa_zio_2] 100065 D - 0xffffff00025cc980 [spa_zio_1] 100064 D - 0xffffff00025cc980 [spa_zio_0] 100063 D - 0xffffff00026a9e00 [spa_zio] 100062 D - 0xffffff00026a9580 [spa_zio] 100061 D - 0xffffff00026a9500 [spa_zio] 100031 D - 0xffffff00024deb00 [em0 taskq] 100019 D - 0xffffff00023b5c00 = [acpi_task_2] 100018 D - 0xffffff00023b5c00 = [acpi_task_1] 100017 D - 0xffffff00023b5c00 = [acpi_task_0] 100016 D - 0xffffff00023b5c80 [kqueue = taskq] 100013 D - 0xffffff00023b6080 [thread = taskq] 100010 D - 0xffffff0002245480 [firmware = taskq] 100000 D sched 0xffffffff80c48d40 [swapper] db> show alllocks Process 4089 (sysctl) thread 0xffffff001d849ae0 (100129) Process 4088 (dd) thread 0xffffff001d84c000 (100138) Process 4076 (newsyslog) thread 0xffffff001d84aae0 (100133) Process 4053 (bsdtar) thread 0xffffff0002ef0740 (100118) Process 4050 (bsdtar) thread 0xffffff0002be5740 (100107) Process 2711 (sshd) thread 0xffffff0002be53a0 (100108) Process 2705 (bsdtar) thread 0xffffff0002be5000 (100109) Process 1005 (sshd) thread 0xffffff0002be6000 (100105) Process 18 (vnlru) thread 0xffffff00025d4740 (100047) Process 17 (syncer) thread 0xffffff00025d4ae0 (100046) db> show lockedvnods Locked vnodes db> show allchains db> alltrace Tracing command sysctl pid 4089 tid 100129 td 0xffffff001d849ae0 kdb_enter() at kdb_enter+0x3d kdb_sysctl_enter() at kdb_sysctl_enter+0x89 sysctl_root() at sysctl_root+0x113 userland_sysctl() at userland_sysctl+0x158 __sysctl() at __sysctl+0xaa syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (202, FreeBSD ELF64, __sysctl), rip =3D 0x80073cb6c, rsp =3D = 0x7fffffffe298, rbp =3D 0x7fffffffe36c --- Tracing command dd pid 4088 tid 100138 td 0xffffff001d84c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_slock_hard() at _sx_slock_hard+0x1b7 _sx_slock() at _sx_slock+0xc1 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 getnewvnode() at getnewvnode+0x24f ffs_vgetf() at ffs_vgetf+0xdf ffs_valloc() at ffs_valloc+0x174 ufs_makeinode() at ufs_makeinode+0x8c VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x473 kern_openat() at kern_openat+0x179 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip =3D 0x80072e70c, rsp =3D = 0x7fffffffecf8, rbp =3D 0x7fffffffed50 --- Tracing command sh pid 4077 tid 100111 td 0xffffff0002ef23a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009355cc, rsp =3D = 0x7fffffffe9d8, rbp =3D 0xfed --- Tracing command newsyslog pid 4076 tid 100133 td 0xffffff001d84aae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_slock_hard() at _sx_slock_hard+0x1b7 _sx_slock() at _sx_slock+0xc1 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 getnewvnode() at getnewvnode+0x24f ffs_vgetf() at ffs_vgetf+0xdf ufs_lookup_() at ufs_lookup_+0xb46 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 stat() at stat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (188, FreeBSD ELF64, stat), rip =3D 0x8007316ac, rsp =3D = 0x7fffffffc348, rbp =3D 0x800a095e0 --- Tracing command cron pid 4074 tid 100136 td 0xffffff001d84b740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80095993c, rsp =3D = 0x7fffffffe4b8, rbp =3D 0x800a9e920 --- Tracing command cron pid 4073 tid 100134 td 0xffffff001d84b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80095993c, rsp =3D = 0x7fffffffe4b8, rbp =3D 0x800a9e920 --- Tracing command zfs pid 4072 tid 100132 td 0xffffff001d84a740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a txg_wait_synced() at txg_wait_synced+0x7c zfsvfs_teardown() at zfsvfs_teardown+0x1d3 zfs_suspend_fs() at zfs_suspend_fs+0x2b zfs_ioc_recv() at zfs_ioc_recv+0x28b zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe78dc, rsp =3D = 0x7fffffff8e98, rbp =3D 0x7fffffff9bd0 --- Tracing command csh pid 4070 tid 100139 td 0xffffff0002ef0ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x80094d21c, rsp =3D = 0x7fffffffe578, rbp =3D 0x800c1f500 --- Tracing command sshd pid 4068 tid 100137 td 0xffffff001d84bae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d68bc, rsp =3D = 0x7fffffffdcb8, rbp =3D 0x7fffffffdd40 --- Tracing command bsdtar pid 4056 tid 100127 td 0xffffff001d84c3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 setfown() at setfown+0x79 kern_fchownat() at kern_fchownat+0xf9 kern_lchown() at kern_lchown+0x24 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (254, FreeBSD ELF64, lchown), rip =3D 0x800da583c, rsp =3D = 0x7fffffffe808, rbp =3D 0x801025380 --- Tracing command bsdtar pid 4055 tid 100123 td 0xffffff0002f053a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_write() at pipe_write+0x13fc dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e4391c, rsp =3D = 0x7fffffffe818, rbp =3D 0x801004140 --- Tracing command sh pid 4054 tid 100125 td 0xffffff0002be8ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009355cc, rsp =3D = 0x7fffffffe8a8, rbp =3D 0xfd6 --- Tracing command bsdtar pid 4053 tid 100118 td 0xffffff0002ef0740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_slock_hard() at _sx_slock_hard+0x1b7 _sx_slock() at _sx_slock+0xc1 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 getnewvnode() at getnewvnode+0x24f zfs_znode_cache_constructor() at zfs_znode_cache_constructor+0x35 zfs_znode_alloc() at zfs_znode_alloc+0x39 zfs_mknode() at zfs_mknode+0x204 zfs_freebsd_create() at zfs_freebsd_create+0x594 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x473 kern_openat() at kern_openat+0x179 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip =3D 0x800e3470c, rsp =3D = 0x7fffffffe738, rbp =3D 0x1a4 --- Tracing command bsdtar pid 4052 tid 100124 td 0xffffff0002f05000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_write() at pipe_write+0x13fc dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e4391c, rsp =3D = 0x7fffffffe818, rbp =3D 0x801004140 --- Tracing command sh pid 4051 tid 100120 td 0xffffff0002ef0000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009355cc, rsp =3D = 0x7fffffffe8a8, rbp =3D 0xfd3 --- Tracing command bsdtar pid 4050 tid 100107 td 0xffffff0002be5740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x7a3 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b cache_lookup() at cache_lookup+0x4e0 vfs_cache_lookup() at vfs_cache_lookup+0xc0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800e3468c, rsp =3D = 0x7fffffffe758, rbp =3D 0x801025380 --- Tracing command bsdtar pid 4049 tid 100126 td 0xffffff0002be8740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_write() at pipe_write+0x13fc dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e4391c, rsp =3D = 0x7fffffffe818, rbp =3D 0x801004140 --- Tracing command sh pid 4048 tid 100119 td 0xffffff0002ef03a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009355cc, rsp =3D = 0x7fffffffe8a8, rbp =3D 0xfd0 --- Tracing command bsdtar pid 4047 tid 100117 td 0xffffff0002ef1000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 setfown() at setfown+0x79 kern_fchownat() at kern_fchownat+0xf9 kern_lchown() at kern_lchown+0x24 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (254, FreeBSD ELF64, lchown), rip =3D 0x800da583c, rsp =3D = 0x7fffffffe808, rbp =3D 0x801025380 --- Tracing command bsdtar pid 4046 tid 100121 td 0xffffff0002f05ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_write() at pipe_write+0x13fc dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e4391c, rsp =3D = 0x7fffffffe818, rbp =3D 0x801004140 --- Tracing command sh pid 4045 tid 100096 td 0xffffff0002be83a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009355cc, rsp =3D = 0x7fffffffe8a8, rbp =3D 0xfcd --- Tracing command csh pid 2719 tid 100115 td 0xffffff0002ef13a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x8009eb93c, rsp =3D = 0x7fffffffe738, rbp =3D 0x1 --- Tracing command su pid 2718 tid 100110 td 0xffffff00026e2ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009e85cc, rsp =3D = 0x7fffffffe598, rbp =3D 0xa9f --- Tracing command csh pid 2715 tid 100113 td 0xffffff0002ef1ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x80094d21c, rsp =3D = 0x7fffffffe558, rbp =3D 0x800c21700 --- Tracing command sshd pid 2714 tid 100114 td 0xffffff0002ef1740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d68bc, rsp =3D = 0x7fffffffdcb8, rbp =3D 0x7fffffffdd40 --- Tracing command sshd pid 2711 tid 100108 td 0xffffff0002be53a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x8013d693c, rsp =3D = 0x7fffffffdcd8, rbp =3D 0 --- Tracing command bsdtar pid 2706 tid 100106 td 0xffffff0002be5ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x800e4393c, rsp =3D = 0x7fffffffe788, rbp =3D 0x80101c240 --- Tracing command bsdtar pid 2705 tid 100109 td 0xffffff0002be5000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a rrw_enter() at rrw_enter+0xb1 zfs_freebsd_read() at zfs_freebsd_read+0x55 VOP_READ_APV() at VOP_READ_APV+0xaf vn_read() at vn_read+0x256 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x800e4393c, rsp =3D = 0x7fffffffe878, rbp =3D 0x740 --- Tracing command sh pid 2704 tid 100112 td 0xffffff0002ef2000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009355cc, rsp =3D = 0x7fffffffe8a8, rbp =3D 0xa90 --- Tracing command csh pid 1014 tid 100052 td 0xffffff00025d33a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a fork1() at fork1+0xfe2 vfork() at vfork+0x1c syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (66, FreeBSD ELF64, vfork), rip =3D 0x80094f39d, rsp =3D = 0x7fffffffe6c0, rbp =3D 0 --- Tracing command su pid 1013 tid 100056 td 0xffffff000253d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009e85cc, rsp =3D = 0x7fffffffe598, rbp =3D 0x3f6 --- Tracing command csh pid 1010 tid 100053 td 0xffffff00025d3000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x80094d21c, rsp =3D = 0x7fffffffe558, rbp =3D 0x800c21700 --- Tracing command sshd pid 1009 tid 100104 td 0xffffff0002be63a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d68bc, rsp =3D = 0x7fffffffdcb8, rbp =3D 0x7fffffffdd40 --- Tracing command sshd pid 1005 tid 100105 td 0xffffff0002be6000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x8013d693c, rsp =3D = 0x7fffffffdcd8, rbp =3D 0 --- Tracing command getty pid 1001 tid 100103 td 0xffffff0002be6740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1000 tid 100102 td 0xffffff0002be6ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 999 tid 100101 td 0xffffff0002be7000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 998 tid 100100 td 0xffffff0002be73a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 997 tid 100099 td 0xffffff0002be7740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 996 tid 100098 td 0xffffff0002be7ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 995 tid 100055 td 0xffffff000253d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 994 tid 100050 td 0xffffff00025d3ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command cron pid 929 tid 100051 td 0xffffff00025d3740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip =3D 0x80093eb6c, rsp =3D = 0x7fffffffeb28, rbp =3D 0x3a --- Tracing command sshd pid 918 tid 100057 td 0xffffff00026a2ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d68bc, rsp =3D = 0x7fffffffddd8, rbp =3D 0x2 --- Tracing command syslogd pid 643 tid 100059 td 0xffffff00026a23a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8008518bc, rsp =3D = 0x7fffffffe118, rbp =3D 0x800a310c8 --- Tracing command devd pid 521 tid 100058 td 0xffffff00026a2740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x440ecc, rsp =3D = 0x7fffffffe898, rbp =3D 0x7fffffffe8b0 --- Tracing command zfskern pid 40 tid 100093 td 0xffffff00026adae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a txg_thread_wait() at txg_thread_wait+0x79 txg_sync_thread() at txg_sync_thread+0x19f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cf6d30, rbp =3D 0 --- Tracing command zfskern pid 40 tid 100092 td 0xffffff00026af000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a txg_quiesce_thread() at txg_quiesce_thread+0x1fb fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cf2d30, rbp =3D 0 --- Tracing command zfskern pid 40 tid 100090 td 0xffffff00026acae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vdev_geom_worker() at vdev_geom_worker+0xf3 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021ccfd30, rbp =3D 0 --- Tracing command zfskern pid 40 tid 100089 td 0xffffff00026ad000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vdev_geom_worker() at vdev_geom_worker+0xf3 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021ccad30, rbp =3D 0 --- Tracing command zfskern pid 40 tid 100088 td 0xffffff00026be000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vdev_geom_worker() at vdev_geom_worker+0xf3 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cc5d30, rbp =3D 0 --- Tracing command zfskern pid 40 tid 100087 td 0xffffff00026be3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vdev_geom_worker() at vdev_geom_worker+0xf3 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cc0d30, rbp =3D 0 --- Tracing command zfskern pid 40 tid 100060 td 0xffffff00026a2000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c l2arc_feed_thread() at l2arc_feed_thread+0x144 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001dad30, rbp =3D 0 --- Tracing command zfskern pid 40 tid 100054 td 0xffffff000253dae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c arc_reclaim_thread() at arc_reclaim_thread+0x299 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001bcd30, rbp =3D 0 --- Tracing command flowcleaner pid 20 tid 100049 td 0xffffff00025d4000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c flowtable_cleaner() at flowtable_cleaner+0xde fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001a3d30, rbp =3D 0 --- Tracing command softdepflush pid 19 tid 100048 td 0xffffff00025d43a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 softdep_flush() at softdep_flush+0x259 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800019ed30, rbp =3D 0 --- Tracing command vnlru pid 18 tid 100047 td 0xffffff00025d4740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_slock_hard() at _sx_slock_hard+0x1b7 _sx_slock() at _sx_slock+0xc1 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 vnlru_proc() at vnlru_proc+0x58e fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000199d30, rbp =3D 0 --- Tracing command syncer pid 17 tid 100046 td 0xffffff00025d4ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_xlock_hard() at _sx_xlock_hard+0x267 _sx_xlock() at _sx_xlock+0xc1 zfs_zget() at zfs_zget+0x20a zfs_get_data() at zfs_get_data+0x4d zil_commit() at zil_commit+0x532 zfs_sync() at zfs_sync+0xa6 sync_fsync() at sync_fsync+0x13a VOP_FSYNC_APV() at VOP_FSYNC_APV+0xb5 sync_vnode() at sync_vnode+0x157 sched_sync() at sched_sync+0x1cc fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000194d30, rbp =3D 0 --- Tracing command bufdaemon pid 16 tid 100045 td 0xffffff00024d5ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 buf_daemon() at buf_daemon+0x14a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800018fd30, rbp =3D 0 --- Tracing command pagezero pid 15 tid 100044 td 0xffffff000253a000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800018ad30, rbp =3D 0 --- Tracing command vmdaemon pid 9 tid 100043 td 0xffffff000253a3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000185d30, rbp =3D 0 --- Tracing command pagedaemon pid 8 tid 100042 td 0xffffff000253a740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pageout() at vm_pageout+0x855 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000180d30, rbp =3D 0 --- Tracing command xpt_thrd pid 7 tid 100041 td 0xffffff000253aae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 xpt_scanner_thread() at xpt_scanner_thread+0xd5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800017bd30, rbp =3D 0 --- Tracing command fdc0 pid 6 tid 100040 td 0xffffff000253b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 fdc_thread() at fdc_thread+0x774 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000176d30, rbp =3D 0 --- Tracing command usb pid 14 tid 100035 td 0xffffff00023bb740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000153d30, rbp =3D 0 --- Tracing command usb pid 14 tid 100034 td 0xffffff00023bbae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800014ed30, rbp =3D 0 --- Tracing command usb pid 14 tid 100033 td 0xffffff00024d3000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000149d30, rbp =3D 0 --- Tracing command usb pid 14 tid 100032 td 0xffffff00024d33a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000144d30, rbp =3D 0 --- Tracing command usb pid 14 tid 100030 td 0xffffff00024d3ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000139d30, rbp =3D 0 --- Tracing command usb pid 14 tid 100029 td 0xffffff00024d5000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000134d30, rbp =3D 0 --- Tracing command usb pid 14 tid 100028 td 0xffffff00024d53a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800012fd30, rbp =3D 0 --- Tracing command usb pid 14 tid 100027 td 0xffffff00024d5740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800012ad30, rbp =3D 0 --- Tracing command mpt_recovery0 pid 5 tid 100025 td 0xffffff00023b8ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 mpt_recovery_thread() at mpt_recovery_thread+0x251 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800011fd30, rbp =3D 0 --- Tracing command yarrow pid 13 tid 100011 td 0xffffff00022443a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 random_kthread() at random_kthread+0x1ad fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000051d30, rbp =3D 0 --- Tracing command g_down pid 4 tid 100009 td 0xffffff0002244ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 g_io_schedule_down() at g_io_schedule_down+0x236 g_down_procbody() at g_down_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000043d30, rbp =3D 0 --- Tracing command g_up pid 3 tid 100008 td 0xffffff0002253000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 g_io_schedule_up() at g_io_schedule_up+0x133 g_up_procbody() at g_up_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800003ed30, rbp =3D 0 --- Tracing command g_event pid 2 tid 100007 td 0xffffff00022533a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 g_event_procbody() at g_event_procbody+0xa1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000039d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100039 td 0xffffff000253b3a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100038 td 0xffffff000253b740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100037 td 0xffffff000253bae0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100036 td 0xffffff000253d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000158d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100026 td 0xffffff00023b8740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100024 td 0xffffff00023b9000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80000d8d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100023 td 0xffffff00023b93a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80000d3d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100022 td 0xffffff00023b9740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100021 td 0xffffff00023b9ae0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100020 td 0xffffff00023bb000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800007ed30, rbp =3D 0 --- Tracing command intr pid 12 tid 100015 td 0xffffff00023b7740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000065d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100014 td 0xffffff00023b7ae0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xffffff00023b83a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000056d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100006 td 0xffffff0002253740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100005 td 0xffffff0002243000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800002fd30, rbp =3D 0 --- Tracing command intr pid 12 tid 100004 td 0xffffff00022433a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800002ad30, rbp =3D 0 --- Tracing command idle pid 11 tid 100003 td 0xffffff0002243740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d critical_exit() at critical_exit+0xa3 intr_event_handle() at intr_event_handle+0xd7 intr_execute_handlers() at intr_execute_handlers+0x5f lapic_handle_intr() at lapic_handle_intr+0x37 Xapic_isr1() at Xapic_isr1+0xa4 --- interrupt, rip =3D 0xffffffff80854cb6, rsp =3D 0xffffff8000025b50, = rbp =3D 0xffffff8000025b60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000025d30, rbp =3D 0 --- Tracing command init pid 1 tid 100002 td 0xffffff0002243ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x40c49c, rsp =3D = 0x7fffffffe808, rbp =3D 0x401d40 --- Tracing command audit pid 10 tid 100001 td 0xffffff0002244000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a audit_worker() at audit_worker+0x77 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800001bd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100097 td 0xffffff0002be8000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021d0ad30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100095 td 0xffffff00026ad3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021d00d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100094 td 0xffffff00026ad740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cfbd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100091 td 0xffffff00026ac740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cd4d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100086 td 0xffffff00026be740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cbbd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100085 td 0xffffff00026beae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cb6d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100084 td 0xffffff00026bc3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cb1d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100083 td 0xffffff00026bc740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cacd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100082 td 0xffffff00026bcae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021ca7d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100081 td 0xffffff00026af740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021ca2d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100080 td 0xffffff00026afae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c9dd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100079 td 0xffffff00026e0000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c98d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100078 td 0xffffff00026e03a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c93d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100077 td 0xffffff00026af3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c8ed30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100076 td 0xffffff00026e0740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001eed30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100075 td 0xffffff00026e0ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c89d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100074 td 0xffffff00026e2000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c84d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100073 td 0xffffff00026e23a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c7fd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100072 td 0xffffff00026e2740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c7ad30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100071 td 0xffffff00026aa000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001fdd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100070 td 0xffffff00026aa3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001f8d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100069 td 0xffffff00026aa740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001f3d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100068 td 0xffffff00026aaae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001e9d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100067 td 0xffffff00026ab000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001e4d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100066 td 0xffffff00026ab3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001dfd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100065 td 0xffffff00026ab740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021ce8d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100064 td 0xffffff00026abae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021ce3d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100063 td 0xffffff00026ac000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cded30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100062 td 0xffffff00026ac3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cd9d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100061 td 0xffffff00026bc000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cedd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100031 td 0xffffff00024d3740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800013ed30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100019 td 0xffffff00023bb3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000079d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100018 td 0xffffff0002253ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000074d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100017 td 0xffffff00023b7000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800006fd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100016 td 0xffffff00023b73a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800006ad30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100013 td 0xffffff00023b8000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800005bd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100010 td 0xffffff0002244740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000048d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff80c491a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 scheduler() at scheduler+0x29c mi_startup() at mi_startup+0x59 btext() at btext+0x2c db> panic pruebazfs3#=20 pruebazfs3# cat ddb.txt=20 db> set $lines=3D0 db> ps pid ppid pgrp uid state wmesg wchan cmd 4089 1014 4089 0 R+ CPU 0 sysctl 4088 4077 4077 2 S zfsvfs-> 0xffffff0002e5a140 dd 4077 4074 4077 2 Ss wait 0xffffff0002eec460 sh 4076 4073 4076 0 Ss zfsvfs-> 0xffffff0002e5a140 newsyslog 4074 929 929 0 S piperd 0xffffff000577bb60 cron 4073 929 929 0 S piperd 0xffffff00026175b0 cron 4072 4070 4070 0 S tx->tx_s 0xffffff00029f2e50 zfs 4070 4068 4070 0 Ss pause 0xffffff00057120a0 csh 4068 918 4068 0 Ss select 0xffffff001d228e40 sshd 4056 4054 4054 0 S+ zfs 0xffffff002bce2308 bsdtar 4055 4054 4054 0 S+ pipdwt 0xffffff00359945b0 bsdtar 4054 1014 4054 0 S+ wait 0xffffff001d6ee000 sh 4053 4051 4051 0 S+ zfsvfs-> 0xffffff0002e5a140 bsdtar 4052 4051 4051 0 S+ pipdwt 0xffffff00026472d8 bsdtar 4051 1014 4051 0 S+ wait 0xffffff001d6ef8c0 sh 4050 4048 4048 0 S+ zfs 0xffffff002bd407e8 bsdtar 4049 4048 4048 0 S+ pipdwt 0xffffff0002617888 bsdtar 4048 1014 4048 0 S+ wait 0xffffff001d6d4000 sh 4047 4045 4045 0 S+ zfs 0xffffff002bce2308 bsdtar 4046 4045 4045 0 S+ pipdwt 0xffffff0002647888 bsdtar 4045 1014 4045 0 S+ wait 0xffffff0002eec8c0 sh 2719 2718 2719 0 S+ ttyin 0xffffff003de62ca8 csh 2718 2715 2718 1001 S+ wait 0xffffff0002eed000 su 2715 2714 2715 1001 Ss+ pause 0xffffff0002eeb960 csh 2714 2711 2711 1001 S select 0xffffff0002f160c0 sshd 2711 918 2711 0 Ss sbwait 0xffffff002bbf9694 sshd 2706 2704 2704 0 S+ piperd 0xffffff00026475b0 bsdtar 2705 2704 2704 0 S+ rrl->rr_ 0xffffff0002e5a110 bsdtar 2704 1014 2704 0 S+ wait 0xffffff0002eec000 sh 1014 1013 1014 0 R+ csh 1013 1010 1013 1001 S+ wait 0xffffff0002691460 su 1010 1009 1010 1001 Ss+ pause 0xffffff0002692500 csh 1009 1005 1005 1001 R sshd 1005 918 1005 0 Ss sbwait 0xffffff0002c01694 sshd 1001 1 1001 0 Ss+ ttyin 0xffffff000253e4a8 getty 1000 1 1000 0 Ss+ ttyin 0xffffff000252aca8 getty 999 1 999 0 Ss+ ttyin 0xffffff00025504a8 getty 998 1 998 0 Ss+ ttyin 0xffffff00025500a8 getty 997 1 997 0 Ss+ ttyin 0xffffff00025510a8 getty 996 1 996 0 Ss+ ttyin 0xffffff0002550ca8 getty 995 1 995 0 Ss+ ttyin 0xffffff000254f0a8 getty 994 1 994 0 Ss+ ttyin 0xffffff000254eca8 getty 929 1 929 0 Ss nanslp 0xffffffff80c4c768 cron 918 1 918 0 Ss select 0xffffff0002bbec40 sshd 643 1 643 0 Ss select 0xffffff0002bf0940 syslogd 521 1 521 0 Ss select 0xffffff00026bb9c0 devd 40 0 0 0 SL (threaded) zfskern 100093 D tx->tx_q 0xffffff00029f2e70 = [txg_thread_enter] 100092 D tx->tx_c 0xffffff00025f5520 = [txg_thread_enter] 100090 D vgeom:io 0xffffff00028c1590 [vdev da4] 100089 D vgeom:io 0xffffff00028bf790 [vdev da3] 100088 D vgeom:io 0xffffff00028c1410 [vdev da2] 100087 D vgeom:io 0xffffff00028c1290 [vdev da1] 100060 D l2arc_fe 0xffffffff81311640 = [l2arc_feed_thread] 100054 D arc_recl 0xffffffff81309060 = [arc_reclaim_thread] 20 0 0 0 SL flowclea 0xffffffff80e0df50 = [flowcleaner] 19 0 0 0 SL sdflush 0xffffffff80e1ceb8 = [softdepflush] 18 0 0 0 SL zfsvfs-> 0xffffff0002e5a140 [vnlru] 17 0 0 0 SL zfsvfs-> 0xffffff0002b73300 [syncer] 16 0 0 0 SL psleep 0xffffffff80e0d7c8 [bufdaemon] 15 0 0 0 SL pgzero 0xffffffff80e1e94c [pagezero] 9 0 0 0 SL psleep 0xffffffff80e1dce8 [vmdaemon] 8 0 0 0 SL psleep 0xffffffff80e1dcac = [pagedaemon] 7 0 0 0 SL ccb_scan 0xffffffff80c15660 [xpt_thrd] 6 0 0 0 SL - 0xffffff00024b6848 [fdc0] 14 0 0 0 SL (threaded) usb 100035 D - 0xffffff8000282dd0 [usbus1] 100034 D - 0xffffff8000282d78 [usbus1] 100033 D - 0xffffff8000282d20 [usbus1] 100032 D - 0xffffff8000282cc8 [usbus1] 100030 D - 0xffffff8000267ef0 [usbus0] 100029 D - 0xffffff8000267e98 [usbus0] 100028 D - 0xffffff8000267e40 [usbus0] 100027 D - 0xffffff8000267de8 [usbus0] 5 0 0 0 SL idle 0xffffff800024c000 = [mpt_recovery0] 13 0 0 0 SL - 0xffffffff80c4c444 [yarrow] 4 0 0 0 SL - 0xffffffff80c48c48 [g_down] 3 0 0 0 SL - 0xffffffff80c48c40 [g_up] 2 0 0 0 SL - 0xffffffff80c48c30 [g_event] 12 0 0 0 WL (threaded) intr 100039 I [swi0: uart = uart] 100038 I [irq7: = ppc0] 100037 I [irq12: = psm0] 100036 I [irq1: = atkbd0] 100026 I [irq18: = uhci0] 100024 I [irq17: = mpt0 ehci0] 100023 I [irq15: = ata1] 100022 I [irq14: = ata0] 100021 I [irq9: = acpi0] 100020 I [swi2: = cambio] 100015 I [swi6: task = queue] 100014 I [swi6: = Giant taskq] 100012 I [swi5: +] 100006 I [swi3: vm] 100005 I [swi1: = netisr 0] 100004 I [swi4: = clock] 11 0 0 0 RL [idle: = cpu0] 1 0 1 0 SLs wait 0xffffff00022408c0 [init] 10 0 0 0 SL audit_wo 0xffffffff80e1c210 [audit] 0 0 0 0 SLs (threaded) kernel 100097 D - 0xffffff0002bbe100 [zil_clean] 100095 D - 0xffffff0002bb4800 [zil_clean] 100094 D - 0xffffff00028c0900 [zil_clean] 100091 D - 0xffffff00026e4280 = [zfs_vn_rele_taskq] 100086 D - 0xffffff00028c1100 [spa_zio] 100085 D - 0xffffff00026e4880 [spa_zio] 100084 D - 0xffffff00028bf480 [spa_zio] 100083 D - 0xffffff00028bf400 [spa_zio] 100082 D - 0xffffff00028bf300 [spa_zio] 100081 D - 0xffffff00028bf500 [spa_zio] 100080 D - 0xffffff00025ccb00 [spa_zio] 100079 D - 0xffffff000255e700 [spa_zio_7] 100078 D - 0xffffff000255e700 [spa_zio_6] 100077 D - 0xffffff000255e700 [spa_zio_5] 100076 D - 0xffffff000255e700 [spa_zio_4] 100075 D - 0xffffff000255e700 [spa_zio_3] 100074 D - 0xffffff000255e700 [spa_zio_2] 100073 D - 0xffffff000255e700 [spa_zio_1] 100072 D - 0xffffff000255e700 [spa_zio_0] 100071 D - 0xffffff00025cc980 [spa_zio_7] 100070 D - 0xffffff00025cc980 [spa_zio_6] 100069 D - 0xffffff00025cc980 [spa_zio_5] 100068 D - 0xffffff00025cc980 [spa_zio_4] 100067 D - 0xffffff00025cc980 [spa_zio_3] 100066 D - 0xffffff00025cc980 [spa_zio_2] 100065 D - 0xffffff00025cc980 [spa_zio_1] 100064 D - 0xffffff00025cc980 [spa_zio_0] 100063 D - 0xffffff00026a9e00 [spa_zio] 100062 D - 0xffffff00026a9580 [spa_zio] 100061 D - 0xffffff00026a9500 [spa_zio] 100031 D - 0xffffff00024deb00 [em0 taskq] 100019 D - 0xffffff00023b5c00 = [acpi_task_2] 100018 D - 0xffffff00023b5c00 = [acpi_task_1] 100017 D - 0xffffff00023b5c00 = [acpi_task_0] 100016 D - 0xffffff00023b5c80 [kqueue = taskq] 100013 D - 0xffffff00023b6080 [thread = taskq] 100010 D - 0xffffff0002245480 [firmware = taskq] 100000 D sched 0xffffffff80c48d40 [swapper] db> show alllocks Process 4089 (sysctl) thread 0xffffff001d849ae0 (100129) Process 4088 (dd) thread 0xffffff001d84c000 (100138) Process 4076 (newsyslog) thread 0xffffff001d84aae0 (100133) Process 4053 (bsdtar) thread 0xffffff0002ef0740 (100118) Process 4050 (bsdtar) thread 0xffffff0002be5740 (100107) Process 2711 (sshd) thread 0xffffff0002be53a0 (100108) Process 2705 (bsdtar) thread 0xffffff0002be5000 (100109) Process 1005 (sshd) thread 0xffffff0002be6000 (100105) Process 18 (vnlru) thread 0xffffff00025d4740 (100047) Process 17 (syncer) thread 0xffffff00025d4ae0 (100046) db> show lockedvnods Locked vnodes db> show allchains db> alltrace Tracing command sysctl pid 4089 tid 100129 td 0xffffff001d849ae0 kdb_enter() at kdb_enter+0x3d kdb_sysctl_enter() at kdb_sysctl_enter+0x89 sysctl_root() at sysctl_root+0x113 userland_sysctl() at userland_sysctl+0x158 __sysctl() at __sysctl+0xaa syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (202, FreeBSD ELF64, __sysctl), rip =3D 0x80073cb6c, rsp =3D = 0x7fffffffe298, rbp =3D 0x7fffffffe36c --- Tracing command dd pid 4088 tid 100138 td 0xffffff001d84c000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_slock_hard() at _sx_slock_hard+0x1b7 _sx_slock() at _sx_slock+0xc1 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 getnewvnode() at getnewvnode+0x24f ffs_vgetf() at ffs_vgetf+0xdf ffs_valloc() at ffs_valloc+0x174 ufs_makeinode() at ufs_makeinode+0x8c VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x473 kern_openat() at kern_openat+0x179 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip =3D 0x80072e70c, rsp =3D = 0x7fffffffecf8, rbp =3D 0x7fffffffed50 --- Tracing command sh pid 4077 tid 100111 td 0xffffff0002ef23a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009355cc, rsp =3D = 0x7fffffffe9d8, rbp =3D 0xfed --- Tracing command newsyslog pid 4076 tid 100133 td 0xffffff001d84aae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_slock_hard() at _sx_slock_hard+0x1b7 _sx_slock() at _sx_slock+0xc1 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 getnewvnode() at getnewvnode+0x24f ffs_vgetf() at ffs_vgetf+0xdf ufs_lookup_() at ufs_lookup_+0xb46 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 stat() at stat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (188, FreeBSD ELF64, stat), rip =3D 0x8007316ac, rsp =3D = 0x7fffffffc348, rbp =3D 0x800a095e0 --- Tracing command cron pid 4074 tid 100136 td 0xffffff001d84b740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80095993c, rsp =3D = 0x7fffffffe4b8, rbp =3D 0x800a9e920 --- Tracing command cron pid 4073 tid 100134 td 0xffffff001d84b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80095993c, rsp =3D = 0x7fffffffe4b8, rbp =3D 0x800a9e920 --- Tracing command zfs pid 4072 tid 100132 td 0xffffff001d84a740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a txg_wait_synced() at txg_wait_synced+0x7c zfsvfs_teardown() at zfsvfs_teardown+0x1d3 zfs_suspend_fs() at zfs_suspend_fs+0x2b zfs_ioc_recv() at zfs_ioc_recv+0x28b zfsdev_ioctl() at zfsdev_ioctl+0x8d devfs_ioctl_f() at devfs_ioctl_f+0x76 kern_ioctl() at kern_ioctl+0xc5 ioctl() at ioctl+0xfd syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800fe78dc, rsp =3D = 0x7fffffff8e98, rbp =3D 0x7fffffff9bd0 --- Tracing command csh pid 4070 tid 100139 td 0xffffff0002ef0ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x80094d21c, rsp =3D = 0x7fffffffe578, rbp =3D 0x800c1f500 --- Tracing command sshd pid 4068 tid 100137 td 0xffffff001d84bae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d68bc, rsp =3D = 0x7fffffffdcb8, rbp =3D 0x7fffffffdd40 --- Tracing command bsdtar pid 4056 tid 100127 td 0xffffff001d84c3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 setfown() at setfown+0x79 kern_fchownat() at kern_fchownat+0xf9 kern_lchown() at kern_lchown+0x24 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (254, FreeBSD ELF64, lchown), rip =3D 0x800da583c, rsp =3D = 0x7fffffffe808, rbp =3D 0x801025380 --- Tracing command bsdtar pid 4055 tid 100123 td 0xffffff0002f053a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_write() at pipe_write+0x13fc dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e4391c, rsp =3D = 0x7fffffffe818, rbp =3D 0x801004140 --- Tracing command sh pid 4054 tid 100125 td 0xffffff0002be8ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009355cc, rsp =3D = 0x7fffffffe8a8, rbp =3D 0xfd6 --- Tracing command bsdtar pid 4053 tid 100118 td 0xffffff0002ef0740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_slock_hard() at _sx_slock_hard+0x1b7 _sx_slock() at _sx_slock+0xc1 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 getnewvnode() at getnewvnode+0x24f zfs_znode_cache_constructor() at zfs_znode_cache_constructor+0x35 zfs_znode_alloc() at zfs_znode_alloc+0x39 zfs_mknode() at zfs_mknode+0x204 zfs_freebsd_create() at zfs_freebsd_create+0x594 VOP_CREATE_APV() at VOP_CREATE_APV+0xb3 vn_open_cred() at vn_open_cred+0x473 kern_openat() at kern_openat+0x179 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (5, FreeBSD ELF64, open), rip =3D 0x800e3470c, rsp =3D = 0x7fffffffe738, rbp =3D 0x1a4 --- Tracing command bsdtar pid 4052 tid 100124 td 0xffffff0002f05000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_write() at pipe_write+0x13fc dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e4391c, rsp =3D = 0x7fffffffe818, rbp =3D 0x801004140 --- Tracing command sh pid 4051 tid 100120 td 0xffffff0002ef0000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009355cc, rsp =3D = 0x7fffffffe8a8, rbp =3D 0xfd3 --- Tracing command bsdtar pid 4050 tid 100107 td 0xffffff0002be5740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x7a3 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b cache_lookup() at cache_lookup+0x4e0 vfs_cache_lookup() at vfs_cache_lookup+0xc0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x3d3 namei() at namei+0x4a9 kern_statat_vnhook() at kern_statat_vnhook+0x8f kern_statat() at kern_statat+0x15 lstat() at lstat+0x2a syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (190, FreeBSD ELF64, lstat), rip =3D 0x800e3468c, rsp =3D = 0x7fffffffe758, rbp =3D 0x801025380 --- Tracing command bsdtar pid 4049 tid 100126 td 0xffffff0002be8740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_write() at pipe_write+0x13fc dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e4391c, rsp =3D = 0x7fffffffe818, rbp =3D 0x801004140 --- Tracing command sh pid 4048 tid 100119 td 0xffffff0002ef03a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009355cc, rsp =3D = 0x7fffffffe8a8, rbp =3D 0xfd0 --- Tracing command bsdtar pid 4047 tid 100117 td 0xffffff0002ef1000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d __lockmgr_args() at __lockmgr_args+0x78e vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 setfown() at setfown+0x79 kern_fchownat() at kern_fchownat+0xf9 kern_lchown() at kern_lchown+0x24 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (254, FreeBSD ELF64, lchown), rip =3D 0x800da583c, rsp =3D = 0x7fffffffe808, rbp =3D 0x801025380 --- Tracing command bsdtar pid 4046 tid 100121 td 0xffffff0002f05ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_write() at pipe_write+0x13fc dofilewrite() at dofilewrite+0x85 kern_writev() at kern_writev+0x60 write() at write+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x800e4391c, rsp =3D = 0x7fffffffe818, rbp =3D 0x801004140 --- Tracing command sh pid 4045 tid 100096 td 0xffffff0002be83a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009355cc, rsp =3D = 0x7fffffffe8a8, rbp =3D 0xfcd --- Tracing command csh pid 2719 tid 100115 td 0xffffff0002ef13a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x8009eb93c, rsp =3D = 0x7fffffffe738, rbp =3D 0x1 --- Tracing command su pid 2718 tid 100110 td 0xffffff00026e2ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009e85cc, rsp =3D = 0x7fffffffe598, rbp =3D 0xa9f --- Tracing command csh pid 2715 tid 100113 td 0xffffff0002ef1ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x80094d21c, rsp =3D = 0x7fffffffe558, rbp =3D 0x800c21700 --- Tracing command sshd pid 2714 tid 100114 td 0xffffff0002ef1740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d68bc, rsp =3D = 0x7fffffffdcb8, rbp =3D 0x7fffffffdd40 --- Tracing command sshd pid 2711 tid 100108 td 0xffffff0002be53a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x8013d693c, rsp =3D = 0x7fffffffdcd8, rbp =3D 0 --- Tracing command bsdtar pid 2706 tid 100106 td 0xffffff0002be5ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 pipe_read() at pipe_read+0x4a3 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x800e4393c, rsp =3D = 0x7fffffffe788, rbp =3D 0x80101c240 --- Tracing command bsdtar pid 2705 tid 100109 td 0xffffff0002be5000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a rrw_enter() at rrw_enter+0xb1 zfs_freebsd_read() at zfs_freebsd_read+0x55 VOP_READ_APV() at VOP_READ_APV+0xaf vn_read() at vn_read+0x256 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x800e4393c, rsp =3D = 0x7fffffffe878, rbp =3D 0x740 --- Tracing command sh pid 2704 tid 100112 td 0xffffff0002ef2000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009355cc, rsp =3D = 0x7fffffffe8a8, rbp =3D 0xa90 --- Tracing command csh pid 1014 tid 100052 td 0xffffff00025d33a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a fork1() at fork1+0xfe2 vfork() at vfork+0x1c syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (66, FreeBSD ELF64, vfork), rip =3D 0x80094f39d, rsp =3D = 0x7fffffffe6c0, rbp =3D 0 --- Tracing command su pid 1013 tid 100056 td 0xffffff000253d3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x8009e85cc, rsp =3D = 0x7fffffffe598, rbp =3D 0x3f6 --- Tracing command csh pid 1010 tid 100053 td 0xffffff00025d3000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_sigsuspend() at kern_sigsuspend+0x9a sigsuspend() at sigsuspend+0x34 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (4, FreeBSD ELF64, write), rip =3D 0x80094d21c, rsp =3D = 0x7fffffffe558, rbp =3D 0x800c21700 --- Tracing command sshd pid 1009 tid 100104 td 0xffffff0002be63a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d68bc, rsp =3D = 0x7fffffffdcb8, rbp =3D 0x7fffffffdd40 --- Tracing command sshd pid 1005 tid 100105 td 0xffffff0002be6000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 soreceive_generic() at soreceive_generic+0xf99 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x8013d693c, rsp =3D = 0x7fffffffdcd8, rbp =3D 0 --- Tracing command getty pid 1001 tid 100103 td 0xffffff0002be6740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 1000 tid 100102 td 0xffffff0002be6ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 999 tid 100101 td 0xffffff0002be7000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 998 tid 100100 td 0xffffff0002be73a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 997 tid 100099 td 0xffffff0002be7740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 996 tid 100098 td 0xffffff0002be7ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 995 tid 100055 td 0xffffff000253d740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command getty pid 994 tid 100050 td 0xffffff00025d3ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e tty_wait() at tty_wait+0x48 ttydisc_read() at ttydisc_read+0x2f1 ttydev_read() at ttydev_read+0xab devfs_read_f() at devfs_read_f+0x86 dofileread() at dofileread+0xa1 kern_readv() at kern_readv+0x60 read() at read+0x55 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (3, FreeBSD ELF64, read), rip =3D 0x80084e93c, rsp =3D = 0x7fffffffecc8, rbp =3D 0 --- Tracing command cron pid 929 tid 100051 td 0xffffff00025d3740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_timedwait_sig() at sleepq_timedwait_sig+0x19 _sleep() at _sleep+0x23f kern_nanosleep() at kern_nanosleep+0x118 nanosleep() at nanosleep+0x6e syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (240, FreeBSD ELF64, nanosleep), rip =3D 0x80093eb6c, rsp =3D = 0x7fffffffeb28, rbp =3D 0x3a --- Tracing command sshd pid 918 tid 100057 td 0xffffff00026a2ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8013d68bc, rsp =3D = 0x7fffffffddd8, rbp =3D 0x2 --- Tracing command syslogd pid 643 tid 100059 td 0xffffff00026a23a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x8008518bc, rsp =3D = 0x7fffffffe118, rbp =3D 0x800a310c8 --- Tracing command devd pid 521 tid 100058 td 0xffffff00026a2740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _cv_wait_sig() at _cv_wait_sig+0x17e seltdwait() at seltdwait+0xac kern_select() at kern_select+0x620 select() at select+0x5d syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (93, FreeBSD ELF64, select), rip =3D 0x440ecc, rsp =3D = 0x7fffffffe898, rbp =3D 0x7fffffffe8b0 --- Tracing command zfskern pid 40 tid 100093 td 0xffffff00026adae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a txg_thread_wait() at txg_thread_wait+0x79 txg_sync_thread() at txg_sync_thread+0x19f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cf6d30, rbp =3D 0 --- Tracing command zfskern pid 40 tid 100092 td 0xffffff00026af000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a txg_quiesce_thread() at txg_quiesce_thread+0x1fb fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cf2d30, rbp =3D 0 --- Tracing command zfskern pid 40 tid 100090 td 0xffffff00026acae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vdev_geom_worker() at vdev_geom_worker+0xf3 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021ccfd30, rbp =3D 0 --- Tracing command zfskern pid 40 tid 100089 td 0xffffff00026ad000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vdev_geom_worker() at vdev_geom_worker+0xf3 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021ccad30, rbp =3D 0 --- Tracing command zfskern pid 40 tid 100088 td 0xffffff00026be000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vdev_geom_worker() at vdev_geom_worker+0xf3 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cc5d30, rbp =3D 0 --- Tracing command zfskern pid 40 tid 100087 td 0xffffff00026be3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vdev_geom_worker() at vdev_geom_worker+0xf3 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cc0d30, rbp =3D 0 --- Tracing command zfskern pid 40 tid 100060 td 0xffffff00026a2000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c l2arc_feed_thread() at l2arc_feed_thread+0x144 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001dad30, rbp =3D 0 --- Tracing command zfskern pid 40 tid 100054 td 0xffffff000253dae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c arc_reclaim_thread() at arc_reclaim_thread+0x299 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001bcd30, rbp =3D 0 --- Tracing command flowcleaner pid 20 tid 100049 td 0xffffff00025d4000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _cv_timedwait() at _cv_timedwait+0x18c flowtable_cleaner() at flowtable_cleaner+0xde fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001a3d30, rbp =3D 0 --- Tracing command softdepflush pid 19 tid 100048 td 0xffffff00025d43a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 softdep_flush() at softdep_flush+0x259 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800019ed30, rbp =3D 0 --- Tracing command vnlru pid 18 tid 100047 td 0xffffff00025d4740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_slock_hard() at _sx_slock_hard+0x1b7 _sx_slock() at _sx_slock+0xc1 zfs_freebsd_reclaim() at zfs_freebsd_reclaim+0x4a VOP_RECLAIM_APV() at VOP_RECLAIM_APV+0xb5 vgonel() at vgonel+0x119 vnlru_free() at vnlru_free+0x345 vnlru_proc() at vnlru_proc+0x58e fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000199d30, rbp =3D 0 --- Tracing command syncer pid 17 tid 100046 td 0xffffff00025d4ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sx_xlock_hard() at _sx_xlock_hard+0x267 _sx_xlock() at _sx_xlock+0xc1 zfs_zget() at zfs_zget+0x20a zfs_get_data() at zfs_get_data+0x4d zil_commit() at zil_commit+0x532 zfs_sync() at zfs_sync+0xa6 sync_fsync() at sync_fsync+0x13a VOP_FSYNC_APV() at VOP_FSYNC_APV+0xb5 sync_vnode() at sync_vnode+0x157 sched_sync() at sched_sync+0x1cc fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000194d30, rbp =3D 0 --- Tracing command bufdaemon pid 16 tid 100045 td 0xffffff00024d5ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 buf_daemon() at buf_daemon+0x14a fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800018fd30, rbp =3D 0 --- Tracing command pagezero pid 15 tid 100044 td 0xffffff000253a000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pagezero() at vm_pagezero+0x73 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800018ad30, rbp =3D 0 --- Tracing command vmdaemon pid 9 tid 100043 td 0xffffff000253a3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 vm_daemon() at vm_daemon+0x4d fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000185d30, rbp =3D 0 --- Tracing command pagedaemon pid 8 tid 100042 td 0xffffff000253a740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 vm_pageout() at vm_pageout+0x855 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000180d30, rbp =3D 0 --- Tracing command xpt_thrd pid 7 tid 100041 td 0xffffff000253aae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 xpt_scanner_thread() at xpt_scanner_thread+0xd5 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800017bd30, rbp =3D 0 --- Tracing command fdc0 pid 6 tid 100040 td 0xffffff000253b000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 fdc_thread() at fdc_thread+0x774 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000176d30, rbp =3D 0 --- Tracing command usb pid 14 tid 100035 td 0xffffff00023bb740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000153d30, rbp =3D 0 --- Tracing command usb pid 14 tid 100034 td 0xffffff00023bbae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800014ed30, rbp =3D 0 --- Tracing command usb pid 14 tid 100033 td 0xffffff00024d3000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000149d30, rbp =3D 0 --- Tracing command usb pid 14 tid 100032 td 0xffffff00024d33a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000144d30, rbp =3D 0 --- Tracing command usb pid 14 tid 100030 td 0xffffff00024d3ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000139d30, rbp =3D 0 --- Tracing command usb pid 14 tid 100029 td 0xffffff00024d5000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000134d30, rbp =3D 0 --- Tracing command usb pid 14 tid 100028 td 0xffffff00024d53a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800012fd30, rbp =3D 0 --- Tracing command usb pid 14 tid 100027 td 0xffffff00024d5740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a usb_process() at usb_process+0x15f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800012ad30, rbp =3D 0 --- Tracing command mpt_recovery0 pid 5 tid 100025 td 0xffffff00023b8ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 mpt_recovery_thread() at mpt_recovery_thread+0x251 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800011fd30, rbp =3D 0 --- Tracing command yarrow pid 13 tid 100011 td 0xffffff00022443a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 random_kthread() at random_kthread+0x1ad fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000051d30, rbp =3D 0 --- Tracing command g_down pid 4 tid 100009 td 0xffffff0002244ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 g_io_schedule_down() at g_io_schedule_down+0x236 g_down_procbody() at g_down_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000043d30, rbp =3D 0 --- Tracing command g_up pid 3 tid 100008 td 0xffffff0002253000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 g_io_schedule_up() at g_io_schedule_up+0x133 g_up_procbody() at g_up_procbody+0x6f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800003ed30, rbp =3D 0 --- Tracing command g_event pid 2 tid 100007 td 0xffffff00022533a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 g_event_procbody() at g_event_procbody+0xa1 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000039d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100039 td 0xffffff000253b3a0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100038 td 0xffffff000253b740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100037 td 0xffffff000253bae0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100036 td 0xffffff000253d000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000158d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100026 td 0xffffff00023b8740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100024 td 0xffffff00023b9000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80000d8d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100023 td 0xffffff00023b93a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80000d3d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100022 td 0xffffff00023b9740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100021 td 0xffffff00023b9ae0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100020 td 0xffffff00023bb000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800007ed30, rbp =3D 0 --- Tracing command intr pid 12 tid 100015 td 0xffffff00023b7740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000065d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100014 td 0xffffff00023b7ae0 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100012 td 0xffffff00023b83a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000056d30, rbp =3D 0 --- Tracing command intr pid 12 tid 100006 td 0xffffff0002253740 fork_trampoline() at fork_trampoline Tracing command intr pid 12 tid 100005 td 0xffffff0002243000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800002fd30, rbp =3D 0 --- Tracing command intr pid 12 tid 100004 td 0xffffff00022433a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d ithread_loop() at ithread_loop+0x246 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800002ad30, rbp =3D 0 --- Tracing command idle pid 11 tid 100003 td 0xffffff0002243740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d critical_exit() at critical_exit+0xa3 intr_event_handle() at intr_event_handle+0xd7 intr_execute_handlers() at intr_execute_handlers+0x5f lapic_handle_intr() at lapic_handle_intr+0x37 Xapic_isr1() at Xapic_isr1+0xa4 --- interrupt, rip =3D 0xffffffff80854cb6, rsp =3D 0xffffff8000025b50, = rbp =3D 0xffffff8000025b60 --- acpi_cpu_c1() at acpi_cpu_c1+0x6 acpi_cpu_idle() at acpi_cpu_idle+0x1f6 sched_idletd() at sched_idletd+0x123 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000025d30, rbp =3D 0 --- Tracing command init pid 1 tid 100002 td 0xffffff0002243ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_catch_signals() at sleepq_catch_signals+0x2af sleepq_wait_sig() at sleepq_wait_sig+0x16 _sleep() at _sleep+0x309 kern_wait() at kern_wait+0x3f7 wait4() at wait4+0x35 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (7, FreeBSD ELF64, wait4), rip =3D 0x40c49c, rsp =3D = 0x7fffffffe808, rbp =3D 0x401d40 --- Tracing command audit pid 10 tid 100001 td 0xffffff0002244000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a audit_worker() at audit_worker+0x77 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800001bd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100097 td 0xffffff0002be8000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021d0ad30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100095 td 0xffffff00026ad3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021d00d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100094 td 0xffffff00026ad740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cfbd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100091 td 0xffffff00026ac740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cd4d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100086 td 0xffffff00026be740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cbbd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100085 td 0xffffff00026beae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cb6d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100084 td 0xffffff00026bc3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cb1d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100083 td 0xffffff00026bc740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cacd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100082 td 0xffffff00026bcae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021ca7d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100081 td 0xffffff00026af740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021ca2d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100080 td 0xffffff00026afae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c9dd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100079 td 0xffffff00026e0000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c98d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100078 td 0xffffff00026e03a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c93d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100077 td 0xffffff00026af3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c8ed30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100076 td 0xffffff00026e0740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001eed30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100075 td 0xffffff00026e0ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c89d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100074 td 0xffffff00026e2000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c84d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100073 td 0xffffff00026e23a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c7fd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100072 td 0xffffff00026e2740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021c7ad30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100071 td 0xffffff00026aa000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001fdd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100070 td 0xffffff00026aa3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001f8d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100069 td 0xffffff00026aa740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001f3d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100068 td 0xffffff00026aaae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001e9d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100067 td 0xffffff00026ab000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001e4d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100066 td 0xffffff00026ab3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff80001dfd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100065 td 0xffffff00026ab740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021ce8d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100064 td 0xffffff00026abae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021ce3d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100063 td 0xffffff00026ac000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cded30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100062 td 0xffffff00026ac3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cd9d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100061 td 0xffffff00026bc000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8021cedd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100031 td 0xffffff00024d3740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800013ed30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100019 td 0xffffff00023bb3a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000079d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100018 td 0xffffff0002253ae0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000074d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100017 td 0xffffff00023b7000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d msleep_spin() at msleep_spin+0x209 taskqueue_thread_loop() at taskqueue_thread_loop+0x62 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800006fd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100016 td 0xffffff00023b73a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800006ad30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100013 td 0xffffff00023b8000 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff800005bd30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100010 td 0xffffff0002244740 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _sleep() at _sleep+0x357 taskqueue_thread_loop() at taskqueue_thread_loop+0xb7 fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip =3D 0, rsp =3D 0xffffff8000048d30, rbp =3D 0 --- Tracing command kernel pid 0 tid 100000 td 0xffffffff80c491a0 sched_switch() at sched_switch+0x154 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_timedwait() at sleepq_timedwait+0x4d _sleep() at _sleep+0x341 scheduler() at scheduler+0x29c mi_startup() at mi_startup+0x59 btext() at btext+0x2c db> panic pruebazfs3#=20 pruebazfs3# ll total 116 -rw------- 1 root wheel 3339 Mar 12 11:04 config.txt -rw------- 1 root wheel 71969 Mar 12 11:04 ddb.txt -rw------- 1 root wheel 35268 Mar 12 11:04 msgbuf.txt -rw------- 1 root wheel 13 Mar 12 11:04 panic.txt -rw------- 1 root wheel 101 Mar 12 11:04 version.txt pruebazfs3# more msgbuf.txt=20 Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights = reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-STABLE #0: Thu Mar 11 15:50:59 UTC 2010 borjam@pruebazfs1:/usr/obj/usr/src/sys/DEBUG amd64 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz (2116.61-MHz = K8-class CPU) Origin =3D "GenuineIntel" Id =3D 0x10676 Stepping =3D 6 = Features=3D0xfebfbff Features2=3D0x80082201> AMD Features=3D0x20100800 AMD Features2=3D0x1 TSC: P-state invariant real memory =3D 1073741824 (1024 MB) avail memory =3D 1015681024 (968 MB) ACPI APIC Table: MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port = 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 7.3 (no driver attached) pci0: at device 7.7 (no driver attached) vgapci0: port 0x10d0-0x10df mem = 0xd0000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on = pci0 mpt0: port 0x1400-0x14ff mem = 0xd8820000-0xd883ffff,0xd8800000-0xd881ffff irq 17 at device 16.0 on = pci0 mpt0: [ITHREAD] mpt0: MPI Version=3D1.2.0.0 pcib2: at device 17.0 on pci0 pci2: on pcib2 uhci0: port 0x2080-0x209f irq = 18 at device 0.0 on pci2 uhci0: [ITHREAD] usbus0: on uhci0 em0: port 0x2000-0x203f = mem 0xd8920000-0xd893ffff,0xd8900000-0xd890ffff irq 19 at device 1.0 on = pci2 em0: Memory Access and/or Bus Master bits were not set! em0: [FILTER] em0: Ethernet address: 00:50:56:31:b0:87 pci2: at device 2.0 (no driver attached) ehci0: mem 0xd8910000-0xd8910fff irq = 17 at device 3.0 on pci2 ehci0: [ITHREAD] usbus1: EHCI version 1.0 usbus1: on ehci0 pcib3: at device 21.0 on pci0 pci3: on pcib3 pcib4: at device 21.1 on pci0 pci4: on pcib4 pcib5: at device 21.2 on pci0 pci5: on pcib5 pcib6: at device 21.3 on pci0 pci6: on pcib6 pcib7: at device 21.4 on pci0 pci7: on pcib7 pcib8: at device 21.5 on pci0 pci8: on pcib8 pcib9: at device 21.6 on pci0 pci9: on pcib9 pcib10: at device 21.7 on pci0 pci10: on pcib10 pcib11: at device 22.0 on pci0 pci11: on pcib11 pcib12: at device 22.1 on pci0 pci12: on pcib12 pcib13: at device 22.2 on pci0 pci13: on pcib13 pcib14: at device 22.3 on pci0 pci14: on pcib14 pcib15: at device 22.4 on pci0 pci15: on pcib15 pcib16: at device 22.5 on pci0 pci16: on pcib16 pcib17: at device 22.6 on pci0 pci17: on pcib17 pcib18: at device 22.7 on pci0 pci18: on pcib18 pcib19: at device 23.0 on pci0 pci19: on pcib19 pcib20: at device 23.1 on pci0 pci20: on pcib20 pcib21: at device 23.2 on pci0 pci21: on pcib21 pcib22: at device 23.3 on pci0 pci22: on pcib22 pcib23: at device 23.4 on pci0 pci23: on pcib23 pcib24: at device 23.5 on pci0 pci24: on pcib24 pcib25: at device 23.6 on pci0 pci25: on pcib25 pcib26: at device 23.7 on pci0 pci26: on pcib26 pcib27: at device 24.0 on pci0 pci27: on pcib27 pcib28: at device 24.1 on pci0 pci28: on pcib28 pcib29: at device 24.2 on pci0 pci29: on pcib29 pcib30: at device 24.3 on pci0 pci30: on pcib30 pcib31: at device 24.4 on pci0 pci31: on pcib31 pcib32: at device 24.5 on pci0 pci32: on pcib32 pcib33: at device 24.6 on pci0 pci33: on pcib33 pcib34: at device 24.7 on pci0 pci34: on pcib34 acpi_acad0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on = acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 orm0: at iomem = 0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=3D0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on = isa0 acpi_throttle0: on cpu0 Timecounter "TSC" frequency 2116614775 Hz quality 800 Timecounters tick every 10.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 480Mbps High Speed USB v2.0 acd0: CDROM at ata1-master = UDMA33=20 ugen0.1: at usbus0 uhub0: on usbus0 Expensive timeout(9) function: 0xffffffff805c9110(0xffffff00024d5000) = 0.005615797 s ugen1.1: <(0x15ad)> at usbus1 uhub1: <(0x15ad) EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on = usbus1 Expensive timeout(9) function: 0xffffffff805c9110(0xffffff00023bbae0) = 0.012154058 s uhub0: 2 ports with 2 removable, self powered uhub1: 6 ports with 6 removable, self powered da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device=20 da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C) da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-2 device=20 da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da1: Command Queueing enabled da1: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) da2 at mpt0 bus 0 scbus0 target 2 lun 0 da2: Fixed Direct Access SCSI-2 device=20 da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da2: Command Queueing enabled da2: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) da3 at mpt0 bus 0 scbus0 target 3 lun 0 da3: Fixed Direct Access SCSI-2 device=20 da3: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da3: Command Queueing enabled da3: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) da4 at mpt0 bus 0 scbus0 target 4 lun 0 da4: Fixed Direct Access SCSI-2 device=20 da4: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da4: Command Queueing enabled da4: 1024MB (2097152 512 byte sectors: 64H 32S/T 1024C) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. GEOM: da1: corrupt or invalid GPT detected. GEOM: da1: GPT rejected -- may not be recoverable. GEOM: da2: corrupt or invalid GPT detected. GEOM: da2: GPT rejected -- may not be recoverable. GEOM: da3: corrupt or invalid GPT detected. GEOM: da3: GPT rejected -- may not be recoverable. GEOM: da4: corrupt or invalid GPT detected. GEOM: da4: GPT rejected -- may not be recoverable. Trying to mount root from ufs:/dev/da0s1a WARNING: / was not properly dismounted <118>Setting hostuuid: 564d4950-1450-6307-6f3f-f4dbde2e248d. <118>Setting hostid: 0x9168d38c. <118>Entropy harvesting: <118> interrupts <118> ethernet <118> point_to_point <118> kickstart <118>. <118>Starting file system checks: <118>/dev/da0s1a: FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED) <118>/dev/da0s1a: SUMMARY INFORMATION BAD (SALVAGED) <118>/dev/da0s1a: BLK(S) MISSING IN BIT MAPS (SALVAGED) <118>/dev/da0s1a: 26042 files, 1618758 used, 1426289 free (20681 frags, = 175701 blocks, 0.7% fragmentation) <118>Mounting local file systems: <118>. <118>Setting hostname: pruebazfs3 <118>. <118>Starting Network: lo0 em0. <118>lo0: flags=3D8049 metric 0 mtu 16384 <118> options=3D3 <118> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3=20 <118> inet6 ::1 prefixlen 128=20 <118> inet 127.0.0.1 netmask 0xff000000=20 <118> nd6 options=3D3 <118>em0: flags=3D8843 metric 0 = mtu 1500 <118> options=3D9b <118> ether 00:50:56:31:b0:87 <118> inet 172.16.2.118 netmask 0xffff0000 broadcast 172.16.255.255 <118> media: Ethernet autoselect (1000baseT ) <118> status: active <118>add net default: gateway 172.16.0.2 <118>Starting devd. <118>Creating and/or trimming log files <118>. <118>Starting syslogd. <118>No core dumps found. <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib <118>32-bit compatibility ldconfig path: /usr/lib32 <118>Clearing /tmp (X related). <118>Updating motd: <118>. <118>Configuring syscons: <118> blanktime <118>. <118>Starting sshd. <118>Starting cron. <118>Starting background file system checks in 60 seconds. <118> <118>Fri Mar 12 08:52:42 UTC 2010 <118>Mar 12 09:02:55 pruebazfs3 su: borjam to root on /dev/pts/1 <118>Mar 12 09:03:03 pruebazfs3 su: borjam to root on /dev/pts/2 ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is = present; to enable, add "vfs.zfs.prefetch_disable=3D0" to = /boot/loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable = behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version 3 ZFS storage pool version 14 GEOM: da1: corrupt or invalid GPT detected. GEOM: da1: GPT rejected -- may not be recoverable. GEOM: da2: corrupt or invalid GPT detected. GEOM: da2: GPT rejected -- may not be recoverable. GEOM: da3: corrupt or invalid GPT detected. GEOM: da3: GPT rejected -- may not be recoverable. GEOM: da4: corrupt or invalid GPT detected. GEOM: da4: GPT rejected -- may not be recoverable. <118>Mar 12 09:34:35 pruebazfs3 shutdown: reboot by borjam:=20 <118>Stopping cron. <118>Stopping sshd. <118>Stopping devd. <118>Writing entropy file: <118>. <118>Terminated <118>. <118>Mar 12 09:34:39 pruebazfs3 syslogd: exiting on signal 15 <118>Mar 12 09:34:59 init: some processes would not die; ps axl advised Waiting (max 60 seconds) for system process `vnlru' to stop...done Waiting (max 60 seconds) for system process `bufdaemon' to stop...done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining...2 0 0 0 done All buffers synced. lock order reversal: 1st 0xffffff0002561308 ufs (ufs) @ /usr/src/sys/kern/vfs_mount.c:1204 2nd 0xffffff0002561cc8 devfs (devfs) @ = /usr/src/sys/kern/vfs_subr.c:2091 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a _witness_debugger() at _witness_debugger+0x2e witness_checkorder() at witness_checkorder+0x81e __lockmgr_args() at __lockmgr_args+0xd31 vop_stdlock() at vop_stdlock+0x39 VOP_LOCK1_APV() at VOP_LOCK1_APV+0x9b _vn_lock() at _vn_lock+0x57 vget() at vget+0x7b devfs_allocv() at devfs_allocv+0x100 devfs_root() at devfs_root+0x48 dounmount() at dounmount+0x474 vfs_unmountall() at vfs_unmountall+0x54 boot() at boot+0x7d3 reboot() at reboot+0x68 syscall() at syscall+0x118 Xfast_syscall() at Xfast_syscall+0xe1 --- syscall (55, FreeBSD ELF64, reboot), rip =3D 0x40892c, rsp =3D = 0x7fffffffe738, rbp =3D 0x4023d0 --- Uptime: 42m50s Rebooting... Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights = reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.0-STABLE #0: Thu Mar 11 15:50:59 UTC 2010 borjam@pruebazfs1:/usr/obj/usr/src/sys/DEBUG amd64 WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Duo CPU T8100 @ 2.10GHz (2158.56-MHz = K8-class CPU) Origin =3D "GenuineIntel" Id =3D 0x10676 Stepping =3D 6 = Features=3D0xfebfbff Features2=3D0x80082201> AMD Features=3D0x20100800 AMD Features2=3D0x1 TSC: P-state invariant real memory =3D 1073741824 (1024 MB) avail memory =3D 1015681024 (968 MB) ACPI APIC Table: MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 isab0: at device 7.0 on pci0 isa0: on isab0 atapci0: port = 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x10c0-0x10cf at device 7.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 7.3 (no driver attached) pci0: at device 7.7 (no driver attached) vgapci0: port 0x10d0-0x10df mem = 0xd0000000-0xd7ffffff,0xd8000000-0xd87fffff irq 16 at device 15.0 on = pci0 mpt0: port 0x1400-0x14ff mem = 0xd8820000-0xd883ffff,0xd8800000-0xd881ffff irq 17 at device 16.0 on = pci0 mpt0: [ITHREAD] mpt0: MPI Version=3D1.2.0.0 pcib2: at device 17.0 on pci0 pci2: on pcib2 uhci0: port 0x2080-0x209f irq = 18 at device 0.0 on pci2 uhci0: [ITHREAD] usbus0: on uhci0 em0: port 0x2000-0x203f = mem 0xd8920000-0xd893ffff,0xd8900000-0xd890ffff irq 19 at device 1.0 on = pci2 em0: Memory Access and/or Bus Master bits were not set! em0: [FILTER] em0: Ethernet address: 00:50:56:31:b0:87 pci2: at device 2.0 (no driver attached) ehci0: mem 0xd8910000-0xd8910fff irq = 17 at device 3.0 on pci2 ehci0: [ITHREAD] usbus1: EHCI version 1.0 usbus1: on ehci0 pcib3: at device 21.0 on pci0 pci3: on pcib3 pcib4: at device 21.1 on pci0 pci4: on pcib4 pcib5: at device 21.2 on pci0 pci5: on pcib5 pcib6: at device 21.3 on pci0 pci6: on pcib6 pcib7: at device 21.4 on pci0 pci7: on pcib7 pcib8: at device 21.5 on pci0 pci8: on pcib8 pcib9: at device 21.6 on pci0 pci9: on pcib9 pcib10: at device 21.7 on pci0 pci10: on pcib10 pcib11: at device 22.0 on pci0 pci11: on pcib11 pcib12: at device 22.1 on pci0 pci12: on pcib12 pcib13: at device 22.2 on pci0 pci13: on pcib13 pcib14: at device 22.3 on pci0 pci14: on pcib14 pcib15: at device 22.4 on pci0 pci15: on pcib15 pcib16: at device 22.5 on pci0 pci16: on pcib16 pcib17: at device 22.6 on pci0 pci17: on pcib17 pcib18: at device 22.7 on pci0 pci18: on pcib18 pcib19: at device 23.0 on pci0 pci19: on pcib19 pcib20: at device 23.1 on pci0 pci20: on pcib20 pcib21: at device 23.2 on pci0 pci21: on pcib21 pcib22: at device 23.3 on pci0 pci22: on pcib22 pcib23: at device 23.4 on pci0 pci23: on pcib23 pcib24: at device 23.5 on pci0 pci24: on pcib24 pcib25: at device 23.6 on pci0 pci25: on pcib25 pcib26: at device 23.7 on pci0 pci26: on pcib26 pcib27: at device 24.0 on pci0 pci27: on pcib27 pcib28: at device 24.1 on pci0 pci28: on pcib28 pcib29: at device 24.2 on pci0 pci29: on pcib29 pcib30: at device 24.3 on pci0 pci30: on pcib30 pcib31: at device 24.4 on pci0 pci31: on pcib31 pcib32: at device 24.5 on pci0 pci32: on pcib32 pcib33: at device 24.6 on pci0 pci33: on pcib33 pcib34: at device 24.7 on pci0 pci34: on pcib34 acpi_acad0: on acpi0 atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model IntelliMouse, device ID 3 ppc0: port 0x378-0x37f irq 7 on acpi0 ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode ppc0: [ITHREAD] ppbus0: on ppc0 plip0: on ppbus0 plip0: [ITHREAD] lpt0: on ppbus0 lpt0: [ITHREAD] lpt0: Interrupt-driven port ppi0: on ppbus0 uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart0: [FILTER] uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0 uart1: [FILTER] fdc0: port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on = acpi0 fdc0: [FILTER] fd0: <1440-KB 3.5" drive> on fdc0 drive 0 orm0: at iomem = 0xc0000-0xc7fff,0xca000-0xcafff,0xdc000-0xdffff,0xe0000-0xe3fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=3D0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on = isa0 acpi_throttle0: on cpu0 Timecounter "TSC" frequency 2158561733 Hz quality 800 Timecounters tick every 10.000 msec usbus0: 12Mbps Full Speed USB v1.0 usbus1: 480Mbps High Speed USB v2.0 Expensive timeout(9) function: 0xffffffff805edf70(0) 0.002324038 s ugen0.1: at usbus0 uhub0: on usbus0 Expensive timeout(9) function: 0xffffffff805c9110(0xffffff00024d5000) = 0.005129981 s ugen1.1: <(0x15ad)> at usbus1 uhub1: <(0x15ad) EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on = usbus1 acd0: CDROM at ata1-master = UDMA33=20 uhub0: 2 ports with 2 removable, self powered uhub1: 6 ports with 6 removable, self powered da0 at mpt0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-2 device=20 da0: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da0: Command Queueing enabled da0: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C) da1 at mpt0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-2 device=20 da1: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da1: Command Queueing enabled da1: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) da2 at mpt0 bus 0 scbus0 target 2 lun 0 da2: Fixed Direct Access SCSI-2 device=20 da2: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da2: Command Queueing enabled da2: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) da3 at mpt0 bus 0 scbus0 target 3 lun 0 da3: Fixed Direct Access SCSI-2 device=20 da3: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da3: Command Queueing enabled da3: 4096MB (8388608 512 byte sectors: 255H 63S/T 522C) da4 at mpt0 bus 0 scbus0 target 4 lun 0 da4: Fixed Direct Access SCSI-2 device=20 da4: 320.000MB/s transfers (160.000MHz, offset 127, 16bit) da4: Command Queueing enabled da4: 1024MB (2097152 512 byte sectors: 64H 32S/T 1024C) WARNING: WITNESS option enabled, expect reduced performance. WARNING: DIAGNOSTIC option enabled, expect reduced performance. Expensive timeout(9) function: 0xffffffff808862e0(0xffffffff80e67f00) = 0.005847391 s GEOM: da1: corrupt or invalid GPT detected. GEOM: da1: GPT rejected -- may not be recoverable. GEOM: da2: corrupt or invalid GPT detected. GEOM: da2: GPT rejected -- may not be recoverable. GEOM: da3: corrupt or invalid GPT detected. GEOM: da3: GPT rejected -- may not be recoverable. GEOM: da4: corrupt or invalid GPT detected. GEOM: da4: GPT rejected -- may not be recoverable. Trying to mount root from ufs:/dev/da0s1a <118>Setting hostuuid: 564d4950-1450-6307-6f3f-f4dbde2e248d. <118>Setting hostid: 0x9168d38c. ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is = present; to enable, add "vfs.zfs.prefetch_disable=3D0" to = /boot/loader.conf. ZFS WARNING: Recommended minimum kmem_size is 512MB; expect unstable = behavior. Consider tuning vm.kmem_size and vm.kmem_size_max in /boot/loader.conf. ZFS filesystem version 3 ZFS storage pool version 14 GEOM: da1: corrupt or invalid GPT detected. GEOM: da1: GPT rejected -- may not be recoverable. GEOM: da2: corrupt or invalid GPT detected. GEOM: da2: GPT rejected -- may not be recoverable. GEOM: da3: corrupt or invalid GPT detected. GEOM: da3: GPT rejected -- may not be recoverable. GEOM: da4: corrupt or invalid GPT detected. GEOM: da4: GPT rejected -- may not be recoverable. <118>Entropy harvesting: <118> interrupts <118> ethernet <118> point_to_point <118> kickstart <118>. <118>Starting file system checks: <118>/dev/da0s1a: FILE SYSTEM CLEAN; SKIPPING CHECKS <118>/dev/da0s1a: clean, 1375746 free (20682 frags, 169383 blocks, 0.7% = fragmentation) <118>Mounting local file systems: <118>. <118>Setting hostname: pruebazfs3 <118>. <118>Starting Network: lo0 em0. <118>lo0: flags=3D8049 metric 0 mtu 16384 <118> options=3D3 <118> inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3=20 <118> inet6 ::1 prefixlen 128=20 <118> inet 127.0.0.1 netmask 0xff000000=20 <118> nd6 options=3D3 <118>em0: flags=3D8843 metric 0 = mtu 1500 <118> options=3D9b <118> ether 00:50:56:31:b0:87 <118> inet 172.16.2.118 netmask 0xffff0000 broadcast 172.16.255.255 <118> media: Ethernet autoselect (1000baseT ) <118> status: active <118>add net default: gateway 172.16.0.2 <118>Starting devd. <118>Creating and/or trimming log files <118>. <118>Starting syslogd. <118>No core dumps found. <118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib <118>32-bit compatibility ldconfig path: /usr/lib32 <118>Clearing /tmp (X related). <118>Updating motd: <118>. <118>Configuring syscons: <118> blanktime <118>. <118>Starting sshd. <118>Starting cron. <118>Starting background file system checks in 60 seconds. <118> <118>Fri Mar 12 09:36:13 UTC 2010 Expensive timeout(9) function: 0xffffffff80329650(0xffffff800026a000) = 0.007047822 s <118>Mar 12 09:40:35 pruebazfs3 sshd[1005]: error: PAM: authentication = error for borjam from 172.16.1.204 <118>Mar 12 09:40:40 pruebazfs3 su: borjam to root on /dev/pts/0 Expensive timeout(9) function: 0xffffffff808862e0(0xffffffff80e67f00) = 0.008521194 s Expensive timeout(9) function: 0xffffffff80329650(0xffffff800026a000) = 0.013145804 s <118>Mar 12 10:30:20 pruebazfs3 su: borjam to root on /dev/pts/1 Expensive timeout(9) function: 0xffffffff804ae7e0(0xffffff8000261000) = 0.017813995 s KDB: enter: sysctl debug.kdb.enter exclusive sleep mutex Giant (Giant) r =3D 0 (0xffffffff80c4bb40) locked = @ /usr/src/sys/kern/kern_sysctl.c:1417 shared sx sysctl lock (sysctl lock) r =3D 0 (0xffffffff80c4c4c0) locked = @ /usr/src/sys/kern/kern_sysctl.c:1521 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff001dc8da58) locked @ = /usr/src/sys/kern/vfs_subr.c:888 exclusive lockmgr ufs (ufs) r =3D 0 (0xffffff00026dbcc8) locked @ = /usr/src/sys/kern/vfs_subr.c:2091 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff002b9677e8) locked @ = /usr/src/sys/kern/vfs_subr.c:888 shared lockmgr ufs (ufs) r =3D 0 (0xffffff0002b8f7e8) locked @ = /usr/src/sys/kern/vfs_subr.c:2091 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff001de647e8) locked @ = /usr/src/sys/kern/vfs_subr.c:888 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff002bd407e8) locked @ = /usr/src/sys/kern/vfs_subr.c:2091 shared lockmgr zfs (zfs) r =3D 0 (0xffffff002bce2308) locked @ = /usr/src/sys/kern/vfs_subr.c:2091 exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xffffff002bbf9648) locked @ = /usr/src/sys/kern/uipc_sockbuf.c:148 shared lockmgr zfs (zfs) r =3D 0 (0xffffff001da92a58) locked @ = /usr/src/sys/kern/vfs_vnops.c:536 exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xffffff0002c01648) locked @ = /usr/src/sys/kern/uipc_sockbuf.c:148 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff001d9d47e8) locked @ = /usr/src/sys/kern/vfs_subr.c:888 exclusive lockmgr syncer (syncer) r =3D 0 (0xffffff000269b7e8) locked @ = /usr/src/sys/kern/vfs_subr.c:1701 0xffffff0002b8f750: tag ufs, type VDIR usecount 1, writecount 0, refcount 4 mountedhere 0 flags () v_object 0xffffff000267d288 ref 0 pages 1 lock type ufs: SHARED (count 1) #0 0xffffffff80578d8f at __lockmgr_args+0x4ef #1 0xffffffff807beb8c at ffs_lock+0x9c #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062e047 at _vn_lock+0x57 #4 0xffffffff8062176b at vget+0x7b #5 0xffffffff8060ec20 at cache_lookup+0x4e0 #6 0xffffffff8060ef40 at vfs_cache_lookup+0xc0 #7 0xffffffff808cc4a7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615bc3 at lookup+0x3d3 #9 0xffffffff80616b99 at namei+0x4a9 #10 0xffffffff80625c7f at kern_statat_vnhook+0x8f #11 0xffffffff80625e25 at kern_statat+0x15 #12 0xffffffff80625fca at stat+0x2a #13 0xffffffff80877238 at syscall+0x118 #14 0xffffffff8085d651 at Xfast_syscall+0xe1 ino 188426, on dev da0s1a 0xffffff00026dbc30: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags () v_object 0xffffff0002ba9ca8 ref 0 pages 1 lock type ufs: EXCL by thread 0xffffff001d84c000 (pid 4088) #0 0xffffffff80578d8f at __lockmgr_args+0x4ef #1 0xffffffff807beb8c at ffs_lock+0x9c #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062e047 at _vn_lock+0x57 #4 0xffffffff8062176b at vget+0x7b #5 0xffffffff8060ec20 at cache_lookup+0x4e0 #6 0xffffffff8060ef40 at vfs_cache_lookup+0xc0 #7 0xffffffff808cc4a7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615bc3 at lookup+0x3d3 #9 0xffffffff80616b99 at namei+0x4a9 #10 0xffffffff8062d892 at vn_open_cred+0xc2 #11 0xffffffff8062bc59 at kern_openat+0x179 #12 0xffffffff80877238 at syscall+0x118 #13 0xffffffff8085d651 at Xfast_syscall+0xe1 ino 188436, on dev da0s1a 0xffffff000269b750: tag syncer, type VNON usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type syncer: EXCL by thread 0xffffff00025d4ae0 (pid 17) #0 0xffffffff80579017 at __lockmgr_args+0x777 #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062e047 at _vn_lock+0x57 #4 0xffffffff80623050 at sync_vnode+0x130 #5 0xffffffff806232dc at sched_sync+0x1cc #6 0xffffffff80566b6a at fork_exit+0x12a #7 0xffffffff8085d84e at fork_trampoline+0xe 0xffffff002bce2270: tag zfs, type VDIR usecount 3, writecount 0, refcount 4 mountedhere 0 flags () lock type zfs: SHARED (count 1) with exclusive waiters pending #0 0xffffffff80578d8f at __lockmgr_args+0x4ef #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062e047 at _vn_lock+0x57 #4 0xffffffff8062176b at vget+0x7b #5 0xffffffff8060ec20 at cache_lookup+0x4e0 #6 0xffffffff8060ef40 at vfs_cache_lookup+0xc0 #7 0xffffffff808cc4a7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615bc3 at lookup+0x3d3 #9 0xffffffff80616b99 at namei+0x4a9 #10 0xffffffff8062a727 at kern_fchownat+0xa7 #11 0xffffffff8062a7e4 at kern_lchown+0x24 #12 0xffffffff80877238 at syscall+0x118 #13 0xffffffff8085d651 at Xfast_syscall+0xe1 0xffffff002bd40750: tag zfs, type VDIR usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type zfs: EXCL by thread 0xffffff0002ef0740 (pid 4053) with shared waiters pending #0 0xffffffff80578d8f at __lockmgr_args+0x4ef #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062e047 at _vn_lock+0x57 #4 0xffffffff8062176b at vget+0x7b #5 0xffffffff8060ec20 at cache_lookup+0x4e0 #6 0xffffffff8060ef40 at vfs_cache_lookup+0xc0 #7 0xffffffff808cc4a7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615bc3 at lookup+0x3d3 #9 0xffffffff80616b99 at namei+0x4a9 #10 0xffffffff8062d892 at vn_open_cred+0xc2 #11 0xffffffff8062bc59 at kern_openat+0x179 #12 0xffffffff80877238 at syscall+0x118 #13 0xffffffff8085d651 at Xfast_syscall+0xe1 0xffffff001de64750: tag zfs, type VREG usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED) v_object 0xffffff001ab5f5e8 ref 0 pages 0 lock type zfs: EXCL by thread 0xffffff0002ef0740 (pid 4053) #0 0xffffffff80579017 at __lockmgr_args+0x777 #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062388b at vnlru_free+0x20b #4 0xffffffff80623c8f at getnewvnode+0x24f #5 0xffffffff81281445 at zfs_znode_cache_constructor+0x35 #6 0xffffffff812824a9 at zfs_znode_alloc+0x39 #7 0xffffffff81283104 at zfs_mknode+0x204 #8 0xffffffff812a48f4 at zfs_freebsd_create+0x594 #9 0xffffffff808cc333 at VOP_CREATE_APV+0xb3 #10 0xffffffff8062dc43 at vn_open_cred+0x473 #11 0xffffffff8062bc59 at kern_openat+0x179 #12 0xffffffff80877238 at syscall+0x118 #13 0xffffffff8085d651 at Xfast_syscall+0xe1 0xffffff001d9d4750: tag zfs, type VREG usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED) v_object 0xffffff001df7c6c0 ref 0 pages 0 lock type zfs: EXCL by thread 0xffffff00025d4740 (pid 18) #0 0xffffffff80579017 at __lockmgr_args+0x777 #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062388b at vnlru_free+0x20b #4 0xffffffff8062427e at vnlru_proc+0x58e #5 0xffffffff80566b6a at fork_exit+0x12a #6 0xffffffff8085d84e at fork_trampoline+0xe 0xffffff002b967750: tag zfs, type VREG usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED) v_object 0xffffff001dd251b0 ref 0 pages 0 lock type zfs: EXCL by thread 0xffffff001d84aae0 (pid 4076) #0 0xffffffff80579017 at __lockmgr_args+0x777 #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062388b at vnlru_free+0x20b #4 0xffffffff80623c8f at getnewvnode+0x24f #5 0xffffffff807b9f9f at ffs_vgetf+0xdf #6 0xffffffff807c49a6 at ufs_lookup_+0xb46 #7 0xffffffff808ca24f at VOP_CACHEDLOOKUP_APV+0xaf #8 0xffffffff8060ef70 at vfs_cache_lookup+0xf0 #9 0xffffffff808cc4a7 at VOP_LOOKUP_APV+0xb7 #10 0xffffffff80615bc3 at lookup+0x3d3 #11 0xffffffff80616b99 at namei+0x4a9 #12 0xffffffff80625c7f at kern_statat_vnhook+0x8f #13 0xffffffff80625e25 at kern_statat+0x15 #14 0xffffffff80625fca at stat+0x2a #15 0xffffffff80877238 at syscall+0x118 #16 0xffffffff8085d651 at Xfast_syscall+0xe1 0xffffff001dc8d9c0: tag zfs, type VREG usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED) v_object 0xffffff001d83bca8 ref 0 pages 0 lock type zfs: EXCL by thread 0xffffff001d84c000 (pid 4088) #0 0xffffffff80579017 at __lockmgr_args+0x777 #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062388b at vnlru_free+0x20b #4 0xffffffff80623c8f at getnewvnode+0x24f <118>Starting background file system checks in 60 seconds. <118> <118>Fri Mar 12 09:36:13 UTC 2010 Expensive timeout(9) function: 0xffffffff80329650(0xffffff800026a000) = 0.00704782 2 s <118>Mar 12 09:40:35 pruebazfs3 sshd[1005]: error: PAM: authentication = error for borjam from 172.16.1.204 <118>Mar 12 09:40:40 pruebazfs3 su: borjam to root on /dev/pts/0 Expensive timeout(9) function: 0xffffffff808862e0(0xffffffff80e67f00) = 0.00852119 4 s Expensive timeout(9) function: 0xffffffff80329650(0xffffff800026a000) = 0.01314580 4 s <118>Mar 12 10:30:20 pruebazfs3 su: borjam to root on /dev/pts/1 Expensive timeout(9) function: 0xffffffff804ae7e0(0xffffff8000261000) = 0.01781399 5 s KDB: enter: sysctl debug.kdb.enter exclusive sleep mutex Giant (Giant) r =3D 0 (0xffffffff80c4bb40) locked = @ /usr/src /sys/kern/kern_sysctl.c:1417 shared sx sysctl lock (sysctl lock) r =3D 0 (0xffffffff80c4c4c0) locked = @ /usr/src /sys/kern/kern_sysctl.c:1521 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff001dc8da58) locked @ = /usr/src/sys/ker n/vfs_subr.c:888 exclusive lockmgr ufs (ufs) r =3D 0 (0xffffff00026dbcc8) locked @ = /usr/src/sys/ker n/vfs_subr.c:2091 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff002b9677e8) locked @ = /usr/src/sys/kern/vfs_subr.c:888 shared lockmgr ufs (ufs) r =3D 0 (0xffffff0002b8f7e8) locked @ = /usr/src/sys/kern/vfs_subr.c:2091 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff001de647e8) locked @ = /usr/src/sys/kern/vfs_subr.c:888 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff002bd407e8) locked @ = /usr/src/sys/kern/vfs_subr.c:2091 shared lockmgr zfs (zfs) r =3D 0 (0xffffff002bce2308) locked @ = /usr/src/sys/kern/vfs_subr.c:2091 exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xffffff002bbf9648) locked @ = /usr/src/sys/kern/uipc_sockbuf.c:148 shared lockmgr zfs (zfs) r =3D 0 (0xffffff001da92a58) locked @ = /usr/src/sys/kern/vfs_vnops.c:536 exclusive sx so_rcv_sx (so_rcv_sx) r =3D 0 (0xffffff0002c01648) locked @ = /usr/src/sys/kern/uipc_sockbuf.c:148 exclusive lockmgr zfs (zfs) r =3D 0 (0xffffff001d9d47e8) locked @ = /usr/src/sys/kern/vfs_subr.c:888 exclusive lockmgr syncer (syncer) r =3D 0 (0xffffff000269b7e8) locked @ = /usr/src/sys/kern/vfs_subr.c:1701 0xffffff0002b8f750: tag ufs, type VDIR usecount 1, writecount 0, refcount 4 mountedhere 0 flags () v_object 0xffffff000267d288 ref 0 pages 1 lock type ufs: SHARED (count 1) #0 0xffffffff80578d8f at __lockmgr_args+0x4ef #1 0xffffffff807beb8c at ffs_lock+0x9c #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062e047 at _vn_lock+0x57 #4 0xffffffff8062176b at vget+0x7b #5 0xffffffff8060ec20 at cache_lookup+0x4e0 #6 0xffffffff8060ef40 at vfs_cache_lookup+0xc0 #7 0xffffffff808cc4a7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615bc3 at lookup+0x3d3 #9 0xffffffff80616b99 at namei+0x4a9 #10 0xffffffff80625c7f at kern_statat_vnhook+0x8f #11 0xffffffff80625e25 at kern_statat+0x15 #12 0xffffffff80625fca at stat+0x2a #13 0xffffffff80877238 at syscall+0x118 #14 0xffffffff8085d651 at Xfast_syscall+0xe1 ino 188426, on dev da0s1a 0xffffff00026dbc30: tag ufs, type VDIR usecount 1, writecount 0, refcount 3 mountedhere 0 flags () v_object 0xffffff0002ba9ca8 ref 0 pages 1 lock type ufs: EXCL by thread 0xffffff001d84c000 (pid 4088) #0 0xffffffff80578d8f at __lockmgr_args+0x4ef #1 0xffffffff807beb8c at ffs_lock+0x9c #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062e047 at _vn_lock+0x57 #4 0xffffffff8062176b at vget+0x7b #5 0xffffffff8060ec20 at cache_lookup+0x4e0 #6 0xffffffff8060ef40 at vfs_cache_lookup+0xc0 #7 0xffffffff808cc4a7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615bc3 at lookup+0x3d3 #9 0xffffffff80616b99 at namei+0x4a9 #10 0xffffffff8062d892 at vn_open_cred+0xc2 #11 0xffffffff8062bc59 at kern_openat+0x179 #12 0xffffffff80877238 at syscall+0x118 #13 0xffffffff8085d651 at Xfast_syscall+0xe1 ino 188436, on dev da0s1a 0xffffff000269b750: tag syncer, type VNON usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type syncer: EXCL by thread 0xffffff00025d4ae0 (pid 17) #0 0xffffffff80579017 at __lockmgr_args+0x777 #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062e047 at _vn_lock+0x57 #4 0xffffffff80623050 at sync_vnode+0x130 #5 0xffffffff806232dc at sched_sync+0x1cc #6 0xffffffff80566b6a at fork_exit+0x12a #7 0xffffffff8085d84e at fork_trampoline+0xe 0xffffff002bce2270: tag zfs, type VDIR usecount 3, writecount 0, refcount 4 mountedhere 0 flags () lock type zfs: SHARED (count 1) with exclusive waiters pending #0 0xffffffff80578d8f at __lockmgr_args+0x4ef #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062e047 at _vn_lock+0x57 #4 0xffffffff8062176b at vget+0x7b #5 0xffffffff8060ec20 at cache_lookup+0x4e0 #6 0xffffffff8060ef40 at vfs_cache_lookup+0xc0 #7 0xffffffff808cc4a7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615bc3 at lookup+0x3d3 #9 0xffffffff80616b99 at namei+0x4a9 #10 0xffffffff8062a727 at kern_fchownat+0xa7 #11 0xffffffff8062a7e4 at kern_lchown+0x24 #12 0xffffffff80877238 at syscall+0x118 #13 0xffffffff8085d651 at Xfast_syscall+0xe1 0xffffff002bd40750: tag zfs, type VDIR usecount 1, writecount 0, refcount 2 mountedhere 0 flags () lock type zfs: EXCL by thread 0xffffff0002ef0740 (pid 4053) with shared waiters pending #0 0xffffffff80578d8f at __lockmgr_args+0x4ef #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062e047 at _vn_lock+0x57 #4 0xffffffff8062176b at vget+0x7b #5 0xffffffff8060ec20 at cache_lookup+0x4e0 #6 0xffffffff8060ef40 at vfs_cache_lookup+0xc0 #7 0xffffffff808cc4a7 at VOP_LOOKUP_APV+0xb7 #8 0xffffffff80615bc3 at lookup+0x3d3 #9 0xffffffff80616b99 at namei+0x4a9 #10 0xffffffff8062d892 at vn_open_cred+0xc2 #11 0xffffffff8062bc59 at kern_openat+0x179 #12 0xffffffff80877238 at syscall+0x118 #13 0xffffffff8085d651 at Xfast_syscall+0xe1 0xffffff001de64750: tag zfs, type VREG usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED) v_object 0xffffff001ab5f5e8 ref 0 pages 0 lock type zfs: EXCL by thread 0xffffff0002ef0740 (pid 4053) #0 0xffffffff80579017 at __lockmgr_args+0x777 #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062388b at vnlru_free+0x20b #4 0xffffffff80623c8f at getnewvnode+0x24f #5 0xffffffff81281445 at zfs_znode_cache_constructor+0x35 #6 0xffffffff812824a9 at zfs_znode_alloc+0x39 #7 0xffffffff81283104 at zfs_mknode+0x204 #8 0xffffffff812a48f4 at zfs_freebsd_create+0x594 #9 0xffffffff808cc333 at VOP_CREATE_APV+0xb3 #10 0xffffffff8062dc43 at vn_open_cred+0x473 #11 0xffffffff8062bc59 at kern_openat+0x179 #12 0xffffffff80877238 at syscall+0x118 #13 0xffffffff8085d651 at Xfast_syscall+0xe1 0xffffff001d9d4750: tag zfs, type VREG usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED) v_object 0xffffff001df7c6c0 ref 0 pages 0 lock type zfs: EXCL by thread 0xffffff00025d4740 (pid 18) #0 0xffffffff80579017 at __lockmgr_args+0x777 #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062388b at vnlru_free+0x20b #4 0xffffffff8062427e at vnlru_proc+0x58e #5 0xffffffff80566b6a at fork_exit+0x12a #6 0xffffffff8085d84e at fork_trampoline+0xe 0xffffff002b967750: tag zfs, type VREG usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED) v_object 0xffffff001dd251b0 ref 0 pages 0 lock type zfs: EXCL by thread 0xffffff001d84aae0 (pid 4076) #0 0xffffffff80579017 at __lockmgr_args+0x777 #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062388b at vnlru_free+0x20b #4 0xffffffff80623c8f at getnewvnode+0x24f #5 0xffffffff807b9f9f at ffs_vgetf+0xdf #6 0xffffffff807c49a6 at ufs_lookup_+0xb46 #7 0xffffffff808ca24f at VOP_CACHEDLOOKUP_APV+0xaf #8 0xffffffff8060ef70 at vfs_cache_lookup+0xf0 #9 0xffffffff808cc4a7 at VOP_LOOKUP_APV+0xb7 #10 0xffffffff80615bc3 at lookup+0x3d3 #11 0xffffffff80616b99 at namei+0x4a9 #12 0xffffffff80625c7f at kern_statat_vnhook+0x8f #13 0xffffffff80625e25 at kern_statat+0x15 #14 0xffffffff80625fca at stat+0x2a #15 0xffffffff80877238 at syscall+0x118 #16 0xffffffff8085d651 at Xfast_syscall+0xe1 0xffffff001dc8d9c0: tag zfs, type VREG usecount 0, writecount 0, refcount 1 mountedhere 0 flags (VI_DOOMED) v_object 0xffffff001d83bca8 ref 0 pages 0 lock type zfs: EXCL by thread 0xffffff001d84c000 (pid 4088) #0 0xffffffff80579017 at __lockmgr_args+0x777 #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062388b at vnlru_free+0x20b #4 0xffffffff80623c8f at getnewvnode+0x24f #5 0xffffffff807b9f9f at ffs_vgetf+0xdf #6 0xffffffff8079fc74 at ffs_valloc+0x174 #7 0xffffffff807c9e5c at ufs_makeinode+0x8c #8 0xffffffff808cc333 at VOP_CREATE_APV+0xb3 #9 0xffffffff8062dc43 at vn_open_cred+0x473 #10 0xffffffff8062bc59 at kern_openat+0x179 #11 0xffffffff80877238 at syscall+0x118 #12 0xffffffff8085d651 at Xfast_syscall+0xe1 0xffffff001da929c0: tag zfs, type VREG usecount 1, writecount 0, refcount 1 mountedhere 0 flags () v_object 0xffffff002bfc5000 ref 0 pages 0 lock type zfs: SHARED (count 1) #0 0xffffffff80578d8f at __lockmgr_args+0x4ef #1 0xffffffff80611109 at vop_stdlock+0x39 #2 0xffffffff808cb5cb at VOP_LOCK1_APV+0x9b #3 0xffffffff8062e047 at _vn_lock+0x57 #4 0xffffffff8062f8d7 at vn_read+0x1c7 #5 0xffffffff805d53b1 at dofileread+0xa1 #6 0xffffffff805d56e0 at kern_readv+0x60 #7 0xffffffff805d57e5 at read+0x55 #8 0xffffffff80877238 at syscall+0x118 #9 0xffffffff8085d651 at Xfast_syscall+0xe1 panic: from debugger cpuid =3D 0 Uptime: 1h28m29s pruebazfs3# =20 pruebazfs3#=20 pruebazfs3#=20 From owner-freebsd-fs@FreeBSD.ORG Fri Mar 12 11:50:34 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 95066106564A; Fri, 12 Mar 2010 11:50:34 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 2A1E48FC17; Fri, 12 Mar 2010 11:50:33 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id B847945EE5; Fri, 12 Mar 2010 12:50:31 +0100 (CET) Received: from localhost (pdawidek.wheel.pl [10.0.1.1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id EC56345EB2; Fri, 12 Mar 2010 12:50:27 +0100 (CET) Date: Fri, 12 Mar 2010 12:50:28 +0100 From: Pawel Jakub Dawidek To: Kai Gallasch Message-ID: <20100312115028.GG1819@garage.freebsd.pl> References: <20100311133916.42ba69b0@orwell.free.de> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="7gGkHNMELEOhSGF6" Content-Disposition: inline In-Reply-To: <20100311133916.42ba69b0@orwell.free.de> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-5.9 required=4.5 tests=ALL_TRUSTED,BAYES_00 autolearn=ham version=3.0.4 Cc: freebsd-fs@FreeBSD.org, mav@FreeBSD.org, freebsd-stable@FreeBSD.org Subject: Re: proliant server lockups with freebsd-amd64-stable (2010-03-10) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Mar 2010 11:50:34 -0000 --7gGkHNMELEOhSGF6 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Mar 11, 2010 at 01:39:16PM +0100, Kai Gallasch wrote: > Hi. >=20 > I have some trouble with an opteron server locking up spontaneously. It l= ooses > all networks connectivity and even through console I can get no shell. >=20 > Lockups occur mostly under disk load (periodic daily, bacula backup > running, make buildworld/buildkernel) and I can provoke them easily. [...] > 4 0 0 0 LL *cissmtx 0xffffff04ed820c00 [g_down] [...] > 100046 L *cissmtx 0xffffff04ed820c00 [irq257: cis= s0] [...] I was analizing similar problem as potential ZFS bug. It turned out to be bug in ciss(4) and I believe mav@ (CCed) has fix for that. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --7gGkHNMELEOhSGF6 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkuaKoQACgkQForvXbEpPzQmpACfYv5IvtyEzVrdjxL7/dsS8QtI JNUAn065Ro15k663XOO3y8YFrMf15Owq =5M1X -----END PGP SIGNATURE----- --7gGkHNMELEOhSGF6-- From owner-freebsd-fs@FreeBSD.ORG Fri Mar 12 18:38:39 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B46FF106566B; Fri, 12 Mar 2010 18:38:39 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-fx0-f223.google.com (mail-fx0-f223.google.com [209.85.220.223]) by mx1.freebsd.org (Postfix) with ESMTP id 992518FC13; Fri, 12 Mar 2010 18:38:38 +0000 (UTC) Received: by fxm23 with SMTP id 23so1495594fxm.3 for ; Fri, 12 Mar 2010 10:38:37 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=2tSyOi4nNgSTPzhkO/2TebU726++GvNqCLZ0bMQmqJA=; b=FVMh/4AcGRUCrnko2IgXtXrTiOX+VfOxd0l3VkMABv3pE1Kul2VIdfEt1lccYUkGme qicz4Cvc7oesghoqr2dyE6YDRFwllEl2Upmw1xUeNxrpRQQ/Z0duKbDo9MOKDvo7j8ao H++q270RxZts9dSRPsETQmgzze/hilQ0MfcFo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=egaYpz25eFmIJYF3TCBe4rUjmeA85ZQTpqWEYdvXNC1r3Gx8EFFQ6ddnlKu+nuFOjD T4xYEFc62hrIaKE2wmeAMxw01CfpXDT3xfvToyOJO+w4Rtsu0bSoBh5RfwRWb3uDXOux 2nuSpS+DGL6h2LRt3vRTuiKhaHWUhw5S1HzmU= Received: by 10.223.164.75 with SMTP id d11mr1166233fay.68.1268419117543; Fri, 12 Mar 2010 10:38:37 -0800 (PST) Received: from mavbook.mavhome.dp.ua (s224.GtokyoFL6.vectant.ne.jp [222.228.90.224]) by mx.google.com with ESMTPS id 15sm1203260fxm.12.2010.03.12.10.38.34 (version=SSLv3 cipher=RC4-MD5); Fri, 12 Mar 2010 10:38:36 -0800 (PST) Sender: Alexander Motin Message-ID: <4B9A8A27.8050608@FreeBSD.org> Date: Fri, 12 Mar 2010 20:38:31 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.23 (X11/20091212) MIME-Version: 1.0 To: Kai Gallasch References: <20100311133916.42ba69b0@orwell.free.de> <20100312115028.GG1819@garage.freebsd.pl> In-Reply-To: <20100312115028.GG1819@garage.freebsd.pl> X-Enigmail-Version: 0.96.0 Content-Type: text/plain; charset=KOI8-R Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, freebsd-stable@FreeBSD.org, Pawel Jakub Dawidek Subject: Re: proliant server lockups with freebsd-amd64-stable (2010-03-10) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Mar 2010 18:38:39 -0000 Pawel Jakub Dawidek wrote: > On Thu, Mar 11, 2010 at 01:39:16PM +0100, Kai Gallasch wrote: >> I have some trouble with an opteron server locking up spontaneously. It looses >> all networks connectivity and even through console I can get no shell. >> >> Lockups occur mostly under disk load (periodic daily, bacula backup >> running, make buildworld/buildkernel) and I can provoke them easily. > [...] >> 4 0 0 0 LL *cissmtx 0xffffff04ed820c00 [g_down] > [...] >> 100046 L *cissmtx 0xffffff04ed820c00 [irq257: ciss0] > [...] > > I was analizing similar problem as potential ZFS bug. It turned out to > be bug in ciss(4) and I believe mav@ (CCed) has fix for that. That my patch is already at 8-STABLE since r204873 of 2010-03-08. Make sure you have it. In this case trap stopped process at ciss_get_request(), which indeed called holding cissmtx lock. But there is no place to sleep or loop there, so may be it was just spontaneous. With bugs I was fixing there was a chance to loop indefinitely between ciss and CAM on resource constraint. That increases chance for such situation to be caught. You may try also look what's going on with `top -HS` and `systat -vm 1`. -- Alexander Motin From owner-freebsd-fs@FreeBSD.ORG Fri Mar 12 21:27:04 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 7614B1065676 for ; Fri, 12 Mar 2010 21:27:04 +0000 (UTC) (envelope-from gallasch@free.de) Received: from smtp.free.de (smtp.free.de [91.204.6.103]) by mx1.freebsd.org (Postfix) with ESMTP id 7645D8FC21 for ; Fri, 12 Mar 2010 21:27:03 +0000 (UTC) Received: (qmail 31273 invoked from network); 12 Mar 2010 22:27:01 +0100 Received: from smtp.free.de (HELO orwell.free.de) (gallasch@free.de@[91.204.4.103]) (envelope-sender ) by smtp.free.de (qmail-ldap-1.03) with AES128-SHA encrypted SMTP for ; 12 Mar 2010 22:27:01 +0100 Date: Fri, 12 Mar 2010 22:26:59 +0100 From: Kai Gallasch To: Alexander Motin Message-ID: <20100312222659.0198dd03@orwell.free.de> In-Reply-To: <4B9A8A27.8050608@FreeBSD.org> References: <20100311133916.42ba69b0@orwell.free.de> <20100312115028.GG1819@garage.freebsd.pl> <4B9A8A27.8050608@FreeBSD.org> X-Mailer: Claws Mail 3.7.3 (GTK+ 2.18.7; powerpc-apple-darwin9.8.0) X-Face: 7"x0zA5=*cXGZw-xjU<">'+!3(KXTUXZVLD42KVN{'go[UQr"Mc.e(XW92N8plZ(9x.{x; I<|95e+b&GH-36\15F~L$YD*Y +u}o&KV?6.%"mJIkaY3G>BKNt`1|Y+%K1P4t; 47D65&(Y7h5Ll-[ltkhamx.-; ,jggK'}oMpUgEHFG YQ"9oXKAl>!d,J}T{)@uxvfu?YFWC*\~h+,^f Mime-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, Pawel Jakub Dawidek , freebsd-stable@FreeBSD.org Subject: Re: proliant server lockups with freebsd-amd64-stable (2010-03-10) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Mar 2010 21:27:04 -0000 Am Fri, 12 Mar 2010 20:38:31 +0200 schrieb Alexander Motin : > Pawel Jakub Dawidek wrote: > > On Thu, Mar 11, 2010 at 01:39:16PM +0100, Kai Gallasch wrote: > >> I have some trouble with an opteron server locking up > >> spontaneously. It looses all networks connectivity and even > >> through console I can get no shell. > >> > >> Lockups occur mostly under disk load (periodic daily, bacula backup > >> running, make buildworld/buildkernel) and I can provoke them > >> easily. > > [...] > >> 4 0 0 0 LL *cissmtx 0xffffff04ed820c00 > >> [g_down] > > [...] > >> 100046 L *cissmtx 0xffffff04ed820c00 > >> [irq257: ciss0] > > [...] > > > > I was analizing similar problem as potential ZFS bug. It turned out > > to be bug in ciss(4) and I believe mav@ (CCed) has fix for that. > > That my patch is already at 8-STABLE since r204873 of 2010-03-08. Make > sure you have it. Updating collection src-all/cvs .. .. Edit src/sys/dev/ciss/ciss.c Edit src/sys/dev/ciss/cissvar.h Didn't have it! Must have been just a few hours I missed it, when I built the last kernel. So I will rebuild my kernel and give it a spin later on. > In this case trap stopped process at ciss_get_request(), which indeed > called holding cissmtx lock. But there is no place to sleep or loop > there, so may be it was just spontaneous. With bugs I was fixing there > was a chance to loop indefinitely between ciss and CAM on resource > constraint. That increases chance for such situation to be caught. > > You may try also look what's going on with `top -HS` and `systat -vm > 1`. FYI. Without your patch of ciss.c and cissvar.h a lockup with top -HS and systat -vm running gives following information below. Is there a pattern visible relating to your patch of the ciss driver? Kai. ------ vmstat -vm ------ 3 users Load 0.21 0.36 0.45 Mar 12 21:47 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 2805456 40804 62699320 79936 12358k count All 6182560 95796 1136820k 212452 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 16018 total 491 533 28 576 19 179 174 zfod atkbd0 1 ozfod uart0 irq4 12.9%Sys 12.5%Intr 0.1%User 0.0%Nice 74.5%Idle %ozfod ata0 irq14 | | | | | | | | | | | daefr uhci0 45 ======+++++++ prcfr 2000 cpu0: time 87 dtbuf totfr 19 bce0 256 Namei Name-cache Dir-cache 100000 desvn react ciss0 257 Calls hits % hits % 40811 numvn pdwak 2000 cpu1: time 17300 frevn pdpgs 2000 cpu4: time intrn 2000 cpu5: time Disks da0 da1 da2 da3 da4 pass0 pass1 4995516 wire 2000 cpu6: time KB/t 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2593276 act 1999 cpu7: time tps 0 0 0 0 0 0 0 369560 inact 2000 cpu3: time MB/s 0.00 0.00 0.00 0.00 0.00 0.00 0.00 9568 cache 2000 cpu2: time %busy 0 0 0 0 0 0 0 12348752 free 1252272 buf ----------- top -HS ------------ last pid: 42561; load averages: 0.35, 0.38, 0.46 up 0+11:24:36 21:53:39 658 processes: 13 running, 623 sleeping, 21 waiting, 1 lock CPU: 0.6% user, 0.0% nice, 12.6% system, 25.0% interrupt, 61.8% idle Mem: 2559M Active, 363M Inact, 4892M Wired, 9548K Cache, 1223M Buf, 12G Free Swap: 21G Total, 21G Free PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 11 root 171 ki31 0K 128K CPU3 3 672:22 100.00% {idle: cpu3} 11 root 171 ki31 0K 128K CPU1 1 663:50 100.00% {idle: cpu1} 11 root 171 ki31 0K 128K RUN 4 649:18 100.00% {idle: cpu4} 12 root -32 - 0K 384K CPU7 0 6:38 100.00% {swi4: clock} 4 root -8 - 0K 16K CPU5 5 1:16 100.00% g_down 12 root -64 - 0K 384K CPU2 0 1:15 100.00% {swi2: cambio} 11 root 171 ki31 0K 128K CPU6 6 672:13 97.27% {idle: cpu6} 11 root 171 ki31 0K 128K CPU0 0 622:18 96.29% {idle: cpu0} 11 root 171 ki31 0K 128K RUN 7 676:57 10.99% {idle: cpu7} 2046 zope1 46 0 468M 379M ucond 6 7:30 1.76% {python} 11 root 171 ki31 0K 128K RUN 5 663:27 0.00% {idle: cpu5} 11 root 171 ki31 0K 128K RUN 2 656:03 0.00% {idle: cpu2} 12 root -68 - 0K 384K WAIT 0 14:54 0.00% {irq256: bce0} 2046 zope1 46 0 468M 379M ucond 4 6:02 0.00% {python} 1719 zope2 48 0 263M 178M ucond 3 2:00 0.00% {python2.4} 1724 zope2 46 0 321M 235M ucond 5 1:52 0.00% {python2.4} 6314 root 45 0 10432K 2564K ttyin 4 1:36 0.00% systat 3 root -8 - 0K 16K - 2 1:29 0.00% g_up 6301 zope4 46 0 311M 225M ucond 1 1:28 0.00% {python2.4} 2320 zope5 44 0 281M 197M ucond 1 1:22 0.00% {python2.4} 20 root 44 - 0K 16K zio->i 6 1:21 0.00% syncer 6301 zope4 46 0 311M 225M ucond 5 1:19 0.00% {python2.4} 1729 zope6 44 0 311M 228M ucond 4 1:15 0.00% {python2.4} 2320 zope5 45 0 281M 197M ucond 4 1:13 0.00% {python2.4} 2320 zope5 50 0 281M 197M ucond 7 1:08 0.00% {python2.4} 6301 zope4 44 0 311M 225M ucond 4 1:02 0.00% {python2.4} 5138 dave 44 0 341M 248M select 4 0:58 0.00% {initial thread} 1729 zope6 45 0 311M 228M ucond 6 0:56 0.00% {python2.4} 8158 19421 44 0 27644K 7560K select 4 0:52 0.00% centerim 1743 zope7 44 0 287M 201M zio->i 1 0:51 0.00% {python} 2046 zope1 44 0 468M 379M select 1 0:48 0.00% {initial thread} 19943 19420 44 0 31740K 13044K select 4 0:46 0.00% centerim 6301 zope4 44 0 311M 225M ucond 0 0:40 0.00% {python2.4} 1743 zope7 44 0 287M 201M select 0 0:38 0.00% {initial thread} 2052 zope8 44 0 300M 212M select 3 0:37 0.00% {initial thread} 1724 zope3 46 0 321M 235M ucond 4 0:33 0.00% {python2.4} 2320 zope5 44 0 281M 197M ucond 1 0:31 0.00% {python2.4} 2052 zope8 45 0 300M 212M ucond 7 0:30 0.00% {python} 2320 zope5 44 0 281M 197M select 6 0:29 0.00% {initial thread} 12 root -64 - 0K 384K *cissm 1 0:28 0.00% {irq257: ciss0} 6301 zope4 44 0 311M 225M select 6 0:28 0.00% {initial thread} 0 root -16 0 0K 816K - 0 0:25 0.00% {spa_zio} 2052 zope8 47 0 300M 212M ucond 2 0:24 0.00% {python} 1729 zope6 44 0 311M 228M select 3 0:24 0.00% {initial thread} 1719 zope2 44 0 263M 178M select 1 0:23 0.00% {initial thread} 1724 zope3 44 0 321M 235M select 1 0:23 0.00% {initial thread} 12 root -48 - 0K 384K WAIT 0 0:22 0.00% {swi0: uart uart} 1734 zope9 44 0 249M 163M select 4 0:21 0.00% {initial thread} 0 root -16 0 0K 816K - 4 0:16 0.00% {spa_zio_6} 0 root -16 0 0K 816K - 3 0:16 0.00% {spa_zio_4} 0 root -16 0 0K 816K - 0 0:16 0.00% {spa_zio_1} 0 root -16 0 0K 816K - 4 0:16 0.00% {spa_zio_2} 0 root -16 0 0K 816K - 1 0:16 0.00% {spa_zio_5} 0 root -16 0 0K 816K - 2 0:16 0.00% {spa_zio_0} 0 root -16 0 0K 816K - 1 0:16 0.00% {spa_zio_7} 0 root -16 0 0K 816K - 5 0:16 0.00% {spa_zio_3} 4288 mysql 44 0 62460K 21608K select 4 0:16 0.00% {mysqld} 59 root -8 - 0K 92K tx->tx 0 0:12 0.00% {txg_thread_enter} 6129 mysql 44 0 62788K 23420K select 0 0:12 0.00% {mysqld} 59 root -8 - 0K 92K vgeom: 5 0:09 0.00% {vdev da2} 59 root -8 - 0K 92K vgeom: 6 0:09 0.00% {vdev da3} 4288 mysql 44 0 62460K 21608K select 0 0:08 0.00% {mysqld} 6129 mysql 44 0 62788K 23420K select 0 0:08 0.00% {mysqld} 13 root 44 - 0K 16K - 0 0:08 0.00% yarrow 0 root -16 0 0K 816K - 0 0:08 0.00% {spa_zio_6} 0 root -16 0 0K 816K - 7 0:08 0.00% {spa_zio_5} 0 root -16 0 0K 816K - 5 0:08 0.00% {spa_zio_2} 0 root -16 0 0K 816K - 2 0:08 0.00% {spa_zio_4} 0 root -16 0 0K 816K - 3 0:08 0.00% {spa_zio_3} 0 root -16 0 0K 816K - 4 0:07 0.00% {spa_zio_1} 0 root -16 0 0K 816K - 0 0:07 0.00% {spa_zio_7} 0 root -16 0 0K 816K - 1 0:07 0.00% {spa_zio_0} 5099 root 44 0 11380K 4292K CPU4 4 0:07 0.00% top 12 root -44 - 0K 384K WAIT 0 0:07 0.00% {swi1: netisr 0} 4864 news 45 0 24964K 12108K nanslp 6 0:06 0.00% perl 5138 dave 76 0 341M 248M ucond 7 0:05 0.00% {python} 4863 news 76 0 8260K 1920K wait 5 0:05 0.00% sh 5155 root 44 0 442M 6592K kqread 1 0:04 0.00% {cache-kqueue} 1734 zope9 44 0 249M 163M ucond 4 0:04 0.00% {python2.4} 4862 news 44 0 139M 43180K select 6 0:04 0.00% innd 8173 19421 44 0 30928K 8496K select 4 0:04 0.00% irssi From owner-freebsd-fs@FreeBSD.ORG Fri Mar 12 21:52:33 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 3F663106566C; Fri, 12 Mar 2010 21:52:33 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-fx0-f223.google.com (mail-fx0-f223.google.com [209.85.220.223]) by mx1.freebsd.org (Postfix) with ESMTP id 7488B8FC16; Fri, 12 Mar 2010 21:52:32 +0000 (UTC) Received: by fxm23 with SMTP id 23so1688832fxm.3 for ; Fri, 12 Mar 2010 13:52:31 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:sender:message-id:date:from :user-agent:mime-version:to:cc:subject:references:in-reply-to :x-enigmail-version:content-type:content-transfer-encoding; bh=NV8tJFdp2BUjPxtwWpup2plB6nu8xJsEpuJRVbWmCy0=; b=VnfcaRrEPFriydgFxiWF7V+R80N2c1OPZYRHoAYFkKL7wZsaM8jHbezk5At23Zd3kE rPCrp2YTQnYGmucdgXtchDUryXvubqWbDu8zJS9S0FqJLCzXMnjQyrI9pra7w5YFzSCu +ovLWS3GnH0CsEO8aOHV24uZTFnBYNGD2PjqY= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:x-enigmail-version:content-type :content-transfer-encoding; b=OksR/9/dQBtaAWqlAEjI78JLrOsPB/jtcqhxs2GEDPbePEySim+rmkz9wCR1QHyWBw vtXARDVTKiVHCscuy6EH57nIIJDuheHwYe4pbo2MatW89bWVr5HAD/+5eNmoRGpdaMpH FZzQzj34kp4Nv7sXgscMm4C79oNSVIevKLLx4= Received: by 10.223.143.18 with SMTP id s18mr1651141fau.64.1268430751344; Fri, 12 Mar 2010 13:52:31 -0800 (PST) Received: from mavbook.mavhome.dp.ua (s224.GtokyoFL6.vectant.ne.jp [222.228.90.224]) by mx.google.com with ESMTPS id 16sm1342126fxm.7.2010.03.12.13.52.29 (version=SSLv3 cipher=RC4-MD5); Fri, 12 Mar 2010 13:52:30 -0800 (PST) Sender: Alexander Motin Message-ID: <4B9AB799.6070906@FreeBSD.org> Date: Fri, 12 Mar 2010 23:52:25 +0200 From: Alexander Motin User-Agent: Thunderbird 2.0.0.23 (X11/20091212) MIME-Version: 1.0 To: Kai Gallasch References: <20100311133916.42ba69b0@orwell.free.de> <20100312115028.GG1819@garage.freebsd.pl> <4B9A8A27.8050608@FreeBSD.org> <20100312222659.0198dd03@orwell.free.de> In-Reply-To: <20100312222659.0198dd03@orwell.free.de> X-Enigmail-Version: 0.96.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-fs@FreeBSD.org, Pawel Jakub Dawidek , freebsd-stable@FreeBSD.org Subject: Re: proliant server lockups with freebsd-amd64-stable (2010-03-10) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Mar 2010 21:52:33 -0000 Kai Gallasch wrote: > Am Fri, 12 Mar 2010 20:38:31 +0200 > schrieb Alexander Motin : >> Pawel Jakub Dawidek wrote: >>> I was analizing similar problem as potential ZFS bug. It turned out >>> to be bug in ciss(4) and I believe mav@ (CCed) has fix for that. >> That my patch is already at 8-STABLE since r204873 of 2010-03-08. Make >> sure you have it. > > Updating collection src-all/cvs > .. > .. > Edit src/sys/dev/ciss/ciss.c > Edit src/sys/dev/ciss/cissvar.h > > Didn't have it! Must have been just a few hours I missed it, > when I built the last kernel. So I will rebuild my kernel and > give it a spin later on. > >> In this case trap stopped process at ciss_get_request(), which indeed >> called holding cissmtx lock. But there is no place to sleep or loop >> there, so may be it was just spontaneous. With bugs I was fixing there >> was a chance to loop indefinitely between ciss and CAM on resource >> constraint. That increases chance for such situation to be caught. >> >> You may try also look what's going on with `top -HS` and `systat -vm >> 1`. > > FYI. Without your patch of ciss.c and cissvar.h a lockup with top -HS > and systat -vm running gives following information below. Is there a > pattern visible relating to your patch of the ciss driver? May be. You should try, -- Alexander Motin From owner-freebsd-fs@FreeBSD.ORG Sat Mar 13 12:10:30 2010 Return-Path: Delivered-To: freebsd-fs@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 6B4BE1065675 for ; Sat, 13 Mar 2010 12:10:30 +0000 (UTC) (envelope-from avg@icyb.net.ua) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id A27BE8FC08 for ; Sat, 13 Mar 2010 12:10:29 +0000 (UTC) Received: from porto.topspin.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id OAA04437 for ; Sat, 13 Mar 2010 14:10:28 +0200 (EET) (envelope-from avg@icyb.net.ua) Received: from localhost.topspin.kiev.ua ([127.0.0.1]) by porto.topspin.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1NqQAl-0005CT-OP for freebsd-fs@FreeBSD.org; Sat, 13 Mar 2010 14:10:27 +0200 Message-ID: <4B9B80B3.5050002@icyb.net.ua> Date: Sat, 13 Mar 2010 14:10:27 +0200 From: Andriy Gapon User-Agent: Thunderbird 2.0.0.23 (X11/20100211) MIME-Version: 1.0 To: freebsd-fs@FreeBSD.org References: <4B966925.1040609@icyb.net.ua> In-Reply-To: <4B966925.1040609@icyb.net.ua> X-Enigmail-Version: 0.96.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Cc: Subject: Re: ZFS ACL usage question [solved] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Mar 2010 12:10:30 -0000 I've got a resolution for the below issue thanks to the great help from Edward Tomasz Napierala (trasz@). on 09/03/2010 17:28 Andriy Gapon said the following: > I have a usage question on ZFS ACL. > Perhaps it's something trivial that should have been asked on questions@, > apologies in that case. > > Let's say for simplicity that I want some directory to be equally shared by two > users. Both should have full access and new files created by one user should > still be fully accessible by the other. > I can't seem to be able to configure AСLs to get that. > > Let's say the users are user1 and user2. > The original directory is owned by user1: > $ ls -ld ~/testdir > drwxrwxr-x+ 22 user1 group0 26 9 Mar 13:01 /home/user1/testdir > > I then issue the following commands: > $ setfacl -b -m user:user1:rwxAWCo:fd:allow ~/testdir > $ setfacl -m user:user1::fd:deny ~/testdir Just in case, this deny entry is redundant, I added it only in attempt to fight ACL inheritance issue. > $ setfacl -m user:user2:rwxAWCo:fd:allow ~/testdir > $ setfacl -m user:user2::fd:deny ~/testdir Ditto. > $ getfacl ~/testdir > # file: /home/user1/testdir > # owner: user1 > # group: group0 > user:user2:--------------:fd----:deny > user:user2:rwx----A-W-Co-:fd----:allow > user:user1:--------------:fd----:deny > user:user1:rwx----A-W-Co-:fd----:allow > owner@:--------------:------:deny > owner@:rwxp---A-W-Co-:------:allow > group@:--------------:------:deny > group@:rwxp----------:------:allow > everyone@:-w-p---A-W-Co-:------:deny > everyone@:r-x---a-R-c--s:------:allow > > Then I create a new file as user1 like this (umask is set to 022): > $ touch ~/testdir/test > $ ls -ld ~/testdir/test > -rw-r--r--+ 1 user1 group0 0 9 Mar 13:01 /home/user1/testdir/test > $ getfacl ~/testdir/test > # file: /home/user1/testdir/test > # owner: user1 > # group: group0 > user:user2:--------------:------:deny > user:user2:-wx-----------:------:deny > user:user2:rwx----A-W----:------:allow > user:user1:--------------:------:deny > user:user1:--x-----------:------:deny > user:user1:rwx----A-W----:------:allow > owner@:--x-----------:------:deny > owner@:rw-p---A-W-Co-:------:allow > group@:-wxp----------:------:deny > group@:r-------------:------:allow > everyone@:-wxp---A-W-Co-:------:deny > everyone@:r-----a-R-c--s:------:allow > > > So now there two deny entries for both users and one of them makes sure that user2 > can not modify the file. > What am I doing wrong? :-) So, I totally missed aclmode and aclinherit properties of the ZFS filesystem in question. Apparently with their default values ACL permissions are also implicitly subjected to limitations by regular Unix permissions. More information on aclmode and aclinherit can be obtained from these links: http://docs.sun.com/app/docs/doc/819-5461/gbaaz?a=view http://docs.sun.com/app/docs/doc/819-5461/gbaax?a=view http://docs.sun.com/app/docs/doc/819-5461/ftyxi?a=view Summary: to get pure ACL behavior unaffected by Unix permissions both aclmode and aclinherit should be set to passthrough. With those settings I got ACL inheritance working as I originally expected. Access works as expected too. -- Andriy Gapon From owner-freebsd-fs@FreeBSD.ORG Sat Mar 13 15:24:39 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F0A41106566C for ; Sat, 13 Mar 2010 15:24:39 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id 320E68FC22 for ; Sat, 13 Mar 2010 15:24:38 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 80AA945F22; Sat, 13 Mar 2010 16:24:36 +0100 (CET) Received: from localhost (chello089077043238.chello.pl [89.77.43.238]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 9028545E82; Sat, 13 Mar 2010 16:24:30 +0100 (CET) Date: Sat, 13 Mar 2010 16:24:29 +0100 From: Pawel Jakub Dawidek To: Borja Marcos Message-ID: <20100313152429.GH3209@garage.freebsd.pl> References: <864468D4-DCE9-493B-9280-00E5FAB2A05C@lassitu.de> <20100309122954.GE3155@garage.freebsd.pl> <20100309125815.GF3155@garage.freebsd.pl> <20100310110202.GA1715@garage.freebsd.pl> <20100310173143.GD1715@garage.freebsd.pl> <27A6DF40-2EA8-4CFC-9E42-DD995E2F9342@sarenet.es> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="BOKacYhQ+x31HxR3" Content-Disposition: inline In-Reply-To: <27A6DF40-2EA8-4CFC-9E42-DD995E2F9342@sarenet.es> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: Many processes stuck in zfs X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Mar 2010 15:24:40 -0000 --BOKacYhQ+x31HxR3 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Mar 12, 2010 at 11:12:49AM +0100, Borja Marcos wrote: >=20 > On Mar 10, 2010, at 6:31 PM, Pawel Jakub Dawidek wrote: >=20 > > Hmm, interesting. Especially those two traces: > >=20 > > Tracing command zfs pid 1820 tid 100105 td 0xffffff0002ca4000 >=20 > Just in case something was wrong, I wiped the two virtual machines and st= arted everything again. The two virtual systems were acting a bit weird, gi= ving lots of LORs. I don't know why, maybe something was corrupted, these v= irtual machines have been panicked and restarted many times. >=20 > So, experiment repeated again, fresh start. Still deadlocking pretty soon= if I start, say, 5 or 6 tar processes. Traces follow. Are you able to dump the kernel memory and make it available for me somewhere? I'd need to look at it wit GDB, as DDB is not enough here, I'm afraid. --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --BOKacYhQ+x31HxR3 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkubriwACgkQForvXbEpPzSs7ACeOrARD6DxJqpTwvIsHeGCWKk1 sKcAniaTOnPPvwqtev6B8U2gLr3krleO =fM69 -----END PGP SIGNATURE----- --BOKacYhQ+x31HxR3-- From owner-freebsd-fs@FreeBSD.ORG Sat Mar 13 15:30:09 2010 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B2D53106567D for ; Sat, 13 Mar 2010 15:30:09 +0000 (UTC) (envelope-from pjd@garage.freebsd.pl) Received: from mail.garage.freebsd.pl (chello089077043238.chello.pl [89.77.43.238]) by mx1.freebsd.org (Postfix) with ESMTP id DB2E98FC24 for ; Sat, 13 Mar 2010 15:30:08 +0000 (UTC) Received: by mail.garage.freebsd.pl (Postfix, from userid 65534) id 2831845F3D; Sat, 13 Mar 2010 16:30:07 +0100 (CET) Received: from localhost (chello089077043238.chello.pl [89.77.43.238]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.garage.freebsd.pl (Postfix) with ESMTP id 53045456B1; Sat, 13 Mar 2010 16:30:01 +0100 (CET) Date: Sat, 13 Mar 2010 16:30:00 +0100 From: Pawel Jakub Dawidek To: Jakub Rajman Message-ID: <20100313153000.GI3209@garage.freebsd.pl> References: <37b87d3d1003111144j38d8c9edsb1c80d8009149d73@mail.gmail.com> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="3uo+9/B/ebqu+fSQ" Content-Disposition: inline In-Reply-To: <37b87d3d1003111144j38d8c9edsb1c80d8009149d73@mail.gmail.com> User-Agent: Mutt/1.4.2.3i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 9.0-CURRENT i386 X-Spam-Checker-Version: SpamAssassin 3.0.4 (2005-06-05) on mail.garage.freebsd.pl X-Spam-Level: X-Spam-Status: No, score=-0.6 required=4.5 tests=BAYES_00,RCVD_IN_SORBS_DUL autolearn=no version=3.0.4 Cc: freebsd-fs@freebsd.org Subject: Re: Example of simple filesystem X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Mar 2010 15:30:09 -0000 --3uo+9/B/ebqu+fSQ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Mar 11, 2010 at 08:44:32PM +0100, Jakub Rajman wrote: > Hello, >=20 > I am going to write my own simple file system and run it under the > FreeBSD OS. I'm looking for some inspiration in existing file systems > (UFS2, msdosfs,...), but they are too complex and complikated. > So I would like to ask, if exists some simple "experimental" > filesystems? Some workable (and mountable) examples with their source > codes. MSDOSFS is probably the easiest example of read-write general-purpose file system. If you want to start with something easier, you can take a look at cd9660 file system, which is read-only. Be prepared that file systems aren't easy, especially because VFS layer is very complex. Good luck:) --=20 Pawel Jakub Dawidek http://www.wheelsystems.com pjd@FreeBSD.org http://www.FreeBSD.org FreeBSD committer Am I Evil? Yes, I Am! --3uo+9/B/ebqu+fSQ Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.14 (FreeBSD) iEYEARECAAYFAkubr3cACgkQForvXbEpPzTymwCfZkNz0dWnQ7l3UuUoxrGJ/TJH A48AoL0eBovQ3fOBHmIY4oN91OBc6OAJ =/lMm -----END PGP SIGNATURE----- --3uo+9/B/ebqu+fSQ-- From owner-freebsd-fs@FreeBSD.ORG Sat Mar 13 22:54:34 2010 Return-Path: Delivered-To: freebsd-fs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 2F8E9106566C; Sat, 13 Mar 2010 22:54:34 +0000 (UTC) (envelope-from linimon@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 065FE8FC0C; Sat, 13 Mar 2010 22:54:34 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id o2DMsXap091405; Sat, 13 Mar 2010 22:54:33 GMT (envelope-from linimon@freefall.freebsd.org) Received: (from linimon@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id o2DMsXXY091401; Sat, 13 Mar 2010 22:54:33 GMT (envelope-from linimon) Date: Sat, 13 Mar 2010 22:54:33 GMT Message-Id: <201003132254.o2DMsXXY091401@freefall.freebsd.org> To: linimon@FreeBSD.org, freebsd-bugs@FreeBSD.org, freebsd-fs@FreeBSD.org From: linimon@FreeBSD.org Cc: Subject: Re: kern/144720: [zfs][patch] zfs list improvements (vendor patches) X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Mar 2010 22:54:34 -0000 Synopsis: [zfs][patch] zfs list improvements (vendor patches) Responsible-Changed-From-To: freebsd-bugs->freebsd-fs Responsible-Changed-By: linimon Responsible-Changed-When: Sat Mar 13 22:54:23 UTC 2010 Responsible-Changed-Why: Over to maintainer(s). http://www.freebsd.org/cgi/query-pr.cgi?pr=144720