Skip site navigation (1)Skip section navigation (2)
Date:      Mon, 11 Oct 2010 12:48:17 -0500
From:      "Sam Fourman Jr." <sfourman@gmail.com>
To:        Jeremy Chadwick <freebsd@jdc.parodius.com>
Cc:        freebsd-fs@freebsd.org, Andriy Gapon <avg@icyb.net.ua>
Subject:   Re: Locked up processes after upgrade to ZFS v15
Message-ID:  <AANLkTi=18wHu07TFg0Vvk1E%2BjDMjNYVqmPNpCb487diB@mail.gmail.com>
In-Reply-To: <20101011154647.GA11532@icarus.home.lan>
References:  <39F05641-4E46-4BE0-81CA-4DEB175A5FBE@free.de> <20101009111241.GA58948@icarus.home.lan> <CF901B53-657E-49FC-A43B-27BC7D49F7A7@free.de> <4CB17983.3020907@icyb.net.ua> <20101011151508.GA10917@icarus.home.lan> <4CB32C75.2060000@icyb.net.ua> <AANLkTikCDxxqf1v9VOMqKZP7wNimq=6fFLDdyDGMDXQ6@mail.gmail.com> <20101011154647.GA11532@icarus.home.lan>

next in thread | previous in thread | raw e-mail | index | archive | help
On Mon, Oct 11, 2010 at 10:46 AM, Jeremy Chadwick
<freebsd@jdc.parodius.com> wrote:
> On Mon, Oct 11, 2010 at 10:40:11AM -0500, Sam Fourman Jr. wrote:
>> > Jeremy,
>> >
>> > were are in a process of debugging this issue with Kai and I think tha=
t we are
>> > onto something. =A0I would like to ask you for some additional testing=
.
>> >
>> > What kind of workload do your run?
>> > Do you have anything using sendfile(2)? =A0E.g. Apache with EnableSend=
file enabled
>> > (it might be by default, without explicit options).
>> > Can you try to disable sendfile(2) use, reboot and see how system beha=
ves?
>> >
>> > If you still experience the same problem after doing the above, then I=
'd like to
>> > ask you for shell access with root privileges; or establishing communi=
cation via
>> > IM and running some commands for me.
>> >
>> > Thank you!
>> > --
>> > Andriy Gapon
>> > _______________________________________________
>>
>>
>> Andriy,
>>
>> I am not sure if this is the same issue, but
>> I have a similar problem running FreeBSD 9-CURRENT amd64 (sources as
>> of 1 hour ago)
>> a zfs lockup appears very frequently, it is a NFS server it locks up
>> several times per day.
>>
>> I have been having these issues since before the ZFSv15 patch was
>> committed to HEAD a few months ago.
>> let me know if you need details, I have no problems giving you root
>> ssh on this machine... but it does =A0not have a serial port.
>> so I am not sure how useful it would be.
>
> Are the processes which livelock (get stuck and cannot be killed, even
> with kill -9) stuck in states "zfs" or "zfsmrb"? =A0You can check with "p=
s
> -axl" or top.
>
> If so, it's probably the same problem. =A0If not, it may be a different
> problem.
>
> --


ok so in ~3 hours I recived yet another zfs lockup

I was in the middle of a scrub and zpool status -v, will not return

FNFS#
FNFS# zpool status -v
  pool: Network
 state: ONLINE
status: One or more devices has experienced an error resulting in data
	corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
	entire pool from backup.
   see: http://www.sun.com/msg/ZFS-8000-8A
 scrub: scrub in progress for 2h1m, 14.52% done, 11h52m to go
config:

	NAME        STATE     READ WRITE CKSUM
	Network     ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    da5     ONLINE       0     0     0
	    da6     ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    da7     ONLINE       0     0     0
	    da8     ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    da9     ONLINE       0     0     0
	    da10    ONLINE       0     0     0
	  mirror    ONLINE       0     0     0
	    da11    ONLINE       0     0     0
	    da12    ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

        /Network/tv/Sam-tv/Lost/Lost.S05E08.HDTV.XviD-XOR.avi
        /Network/jail/mirror/usr/local/man/man3/XChangeDeviceProperty.3.gz
        /Network/jail/mirror/usr/local/man/man3/XForceScreenSaver.3.gz

  pool: zFNFS
 state: ONLINE
 scrub: none requested
config:

	NAME           STATE     READ WRITE CKSUM
	zFNFS          ONLINE       0     0     0
	  raidz2       ONLINE       0     0     0
	    gpt/disk0  ONLINE       0     0     0
	    gpt/disk1  ONLINE       0     0     0
	    gpt/disk2  ONLINE       0     0     0
	    gpt/disk3  ONLINE       0     0     0

errors: No known data errors
FNFS#
FNFS#
FNFS# zpool status -v







FNFS# ps axHwwl
  UID   PID  PPID CPU PRI NI    VSZ    RSS MWCHAN STAT  TT       TIME COMMA=
ND
    0     0     0   0 -16  0      0   2608 sched  DLs   ??    1:31.59 [kern=
el]
    0     0     0   0   8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0   8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0   8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0   8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0   8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0   8  0      0   2608 -      DLs   ??    0:00.01 [kern=
el]
    0     0     0   0 -68  0      0   2608 -      DLs   ??    0:01.39 [kern=
el]
    0     0     0   0 -84  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -84  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -84  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -84  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 deadlk DLs   ??    0:00.23 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.03 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.01 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.02 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.07 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.07 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.10 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.10 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.10 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.07 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.06 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.06 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.06 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.06 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.07 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.06 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.06 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.15 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.04 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    6:04.82 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    6:04.74 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    6:04.78 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.62 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.60 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.62 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.05 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.05 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.05 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.05 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.05 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.14 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.14 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.14 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.14 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.14 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.14 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.14 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.14 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.05 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.05 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.05 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.06 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.05 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0 -16  0      0   2608 -      DLs   ??    0:00.02 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.18 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     0     0   0  -8  0      0   2608 -      DLs   ??    0:00.00 [kern=
el]
    0     1     0   0  44  0   7296    436 wait   ILs   ??    0:00.02
/sbin/init --
    0     2     0   0  -8  0      0     16 -      DL    ??    0:00.44 [g_ev=
ent]
    0     3     0   0  -8  0      0     16 -      DL    ??    0:50.05 [g_up=
]
    0     4     0   0  -8  0      0     16 -      DL    ??    1:25.38 [g_do=
wn]
    0     5     0   0  -8  0      0    128 arc_re DL    ??    0:01.49 [zfsk=
ern]
    0     5     0   0  -8  0      0    128 l2arc_ DL    ??    0:00.04 [zfsk=
ern]
    0     5     0   0  -8  0      0    128 tx->tx DL    ??    0:00.00 [zfsk=
ern]
    0     5     0   0  -8  0      0    128 tx->tx DL    ??    0:00.07 [zfsk=
ern]
    0     5     0   0  -8  0      0    128 tx->tx DL    ??    0:00.00 [zfsk=
ern]
    0     5     0   0  -8  0      0    128 spa->s DL    ??    5:04.17 [zfsk=
ern]
    0     6     0   0 -16  0      0     16 waitin DL    ??    0:00.00
[sctp_iterator]
    0     7     0   0  76  0      0     16 ccb_sc DL    ??    0:05.10 [xpt_=
thrd]
    0     8     0   0 -16  0      0     16 psleep DL    ??    0:00.01
[pagedaemon]
    0     9     0   0 -16  0      0     16 psleep DL    ??    0:00.00 [vmda=
emon]
    0    10     0   0 -16  0      0     16 audit_ DL    ??    0:00.00 [audi=
t]
    0    11     0   0 171  0      0     64 -      RL    ??  197:54.91 [idle=
]
    0    11     0   0 171  0      0     64 -      RL    ??  195:59.68 [idle=
]
    0    11     0   0 171  0      0     64 -      RL    ??  196:58.20 [idle=
]
    0    11     0   0 171  0      0     64 -      RL    ??  170:17.14 [idle=
]
    0    12     0   0 -44  0      0    336 -      WL    ??    0:00.00 [intr=
]
    0    12     0   0 -32  0      0    336 -      WL    ??    0:07.07 [intr=
]
    0    12     0   0 -32  0      0    336 -      WL    ??    0:00.89 [intr=
]
    0    12     0   0 -32  0      0    336 -      WL    ??    0:00.42 [intr=
]
    0    12     0   0 -32  0      0    336 -      WL    ??    0:00.17 [intr=
]
    0    12     0   0 -36  0      0    336 -      WL    ??    0:00.00 [intr=
]
    0    12     0   0 -24  0      0    336 -      WL    ??    0:00.00 [intr=
]
    0    12     0   0 -24  0      0    336 -      WL    ??    0:00.00 [intr=
]
    0    12     0   0 -40  0      0    336 -      WL    ??    8:54.70 [intr=
]
    0    12     0   0 -28  0      0    336 -      WL    ??    0:00.04 [intr=
]
    0    12     0   0 -52  0      0    336 -      WL    ??    0:00.00 [intr=
]
    0    12     0   0 -64  0      0    336 -      WL    ??    0:00.00 [intr=
]
    0    12     0   0 -64  0      0    336 -      WL    ??   31:08.03 [intr=
]
    0    12     0   0 -64  0      0    336 -      WL    ??    0:00.00 [intr=
]
    0    12     0   0 -64  0      0    336 -      WL    ??    0:00.00 [intr=
]
    0    12     0   0 -64  0      0    336 -      WL    ??    0:00.00 [intr=
]
    0    12     0   0 -64  0      0    336 -      WL    ??    0:00.00 [intr=
]
    0    12     0   0 -64  0      0    336 -      WL    ??    0:00.00 [intr=
]
    0    12     0   0 -60  0      0    336 -      WL    ??    0:00.00 [intr=
]
    0    12     0   0 -48  0      0    336 -      WL    ??    0:00.00 [intr=
]
    0    12     0   0 -60  0      0    336 -      WL    ??    0:00.00 [intr=
]
    0    13     0   0  44  0      0     16 -      DL    ??    0:06.17 [yarr=
ow]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -68  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.07 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -68  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.08 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -68  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.09 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -68  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.08 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -68  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.09 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -68  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.09 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -68  0      0    448 -      DL    ??    0:00.00 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.06 [usb]
    0    14     0   0 -64  0      0    448 -      DL    ??    0:00.00 [usb]
    0    15     0   0  44  0      0     16 pgzero DL    ??    0:00.00 [page=
zero]
    0    16     0   0 -16  0      0     16 psleep DL    ??    0:00.04
[bufdaemon]
    0    17     0   0  44  0      0     16 vlruwt DL    ??    0:00.27 [vnlr=
u]
    0    18     0   0  44  0      0     16 zio->i DL    ??    0:04.80 [sync=
er]
    0    19     0   0 -16  0      0     16 sdflus DL    ??    0:00.10
[softdepflush]
    0    20     0   0 -16  0      0     16 flowcl DL    ??    0:00.04
[flowcleaner]
    0   158     1   0  76  0   2892    824 pause  Is    ??    0:00.00
adjkerntz -i
    0   743     1   0  44  0   7296    744 select Is    ??    0:00.00 /sbin=
/devd
    0   938     1   0  44  0  11284   1352 select Ss    ??    0:00.03
/usr/sbin/syslogd -s
    0   962     1   0  44  0  12212   1624 select Ss    ??    0:00.02
/usr/sbin/rpcbind
    0  1043     1   0  76  0  10056   1288 pause  Is    ??    0:00.00
nfsuserd: master (nfsuserd)
    0  1044  1043   0  44  0  10056   1300 select S     ??    0:00.01
nfsuserd: slave (nfsuserd)
    0  1045  1043   0  44  0  10056   1300 select S     ??    0:00.01
nfsuserd: slave (nfsuserd)
    0  1046  1043   0  44  0  10056   1300 select S     ??    0:00.01
nfsuserd: slave (nfsuserd)
    0  1047  1043   0  44  0  10056   1300 select S     ??    0:00.01
nfsuserd: slave (nfsuserd)
    0  1066     1   0  44  0  11280   1924 select Is    ??    0:00.06
/usr/sbin/mountd -e -r -l /etc/exports /etc/zfs/exports
    0  1074     1   0  50  0  10052   1564 select Is    ??    0:00.02
nfsd: master (nfsd)
    0  1081  1074   0  44  0  10052   1080 rpcsvc S     ??    0:00.03
nfsd: server (nfsd)
    0  1081  1074   0  44  0  10052   1080 rpcsvc S     ??    0:00.03
nfsd: server (nfsd)
    0  1081  1074   0  44  0  10052   1080 rpcsvc S     ??    0:00.03
nfsd: server (nfsd)
    0  1081  1074   0  44  0  10052   1080 rpcsvc S     ??    0:00.03
nfsd: server (nfsd)
    0  1082     1   0  44  0 273396   1452 select Ss    ??    0:00.01
/usr/sbin/rpc.statd
    0  1089     1   0  76  0  12340   1424 rpcsvc Ss    ??    0:00.01
/usr/sbin/rpc.lockd
    0  1249     1   0  44  0  30548   3060 select Is    ??    0:00.00
/usr/sbin/sshd
    0  1257     1   0  44  0  16408   3392 select Ss    ??    0:00.12
sendmail: accepting connections (sendmail)
   25  1263     1   0  44  0  16408   3240 pause  Is    ??    0:00.00
sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail)
    0  1270     1   0  44  0  12212   1412 nanslp Is    ??    0:00.02
/usr/sbin/cron -s
    0  1469     1   0  44  0   6888   1208 select SsJ   ??    0:00.02
/usr/sbin/syslogd -s
   70  1594     1   0  44  0  60780   5612 select SsJ   ??    0:00.14
/usr/local/bin/postgres -D /usr/local/pgsql/data
    0  1618     1   0  76  0  26004   3036 select IsJ   ??    0:00.00
/usr/sbin/sshd
    0  1625     1   0  44  0  11976   3360 select SsJ   ??    0:00.14
sendmail: accepting connections (sendmail)
   25  1631     1   0  44  0  11976   3068 pause  IsJ   ??    0:00.00
sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail)
    0  1638     1   0  44  0   7944   1276 nanslp IsJ   ??    0:00.02
/usr/sbin/cron -s
    0  1801     1   0  44  0   6888   1200 select SsJ   ??    0:00.02
/usr/sbin/syslogd -s
 1002  1867     1   0  76  0  28376   1852 accept IsJ   ??    0:00.00
/usr/local/bin/svnserve -d --listen-port=3D3690 --listen-host 0.0.0.0 -r
/usr/local/svn
   70  1916     1   0  44  0  53492   5412 select SsJ   ??    0:00.14
/usr/local/bin/postgres -D /usr/local/pgsql/data
    0  1948     1   0  76  0  26004   3056 select IsJ   ??    0:00.00
/usr/sbin/sshd
    0  1955     1   0  44  0  11976   3096 select SsJ   ??    0:00.14
sendmail: accepting connections (sendmail)
   25  1961     1   0  44  0  11976   2980 pause  IsJ   ??    0:00.00
sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail)
    0  1967     1   0  44  0   7944   1264 nanslp IsJ   ??    0:00.02
/usr/sbin/cron -s
    0  2118     1   0  44  0   6888   1200 select IsJ   ??    0:00.02
/usr/sbin/syslogd -s
 1002  2237     1   0  76  0   6704   1980 select IsJ   ??    0:00.00
/usr/local/sbin/cvsupd -e -C 8 -l @daemon -b /usr/local/etc/cvsup -s
sup.client
    0  2257     1   0  44  0  11976   2856 select SsJ   ??    0:00.14
sendmail: accepting connections (sendmail)
   25  2263     1   0  44  0  11976   2796 pause  IsJ   ??    0:00.00
sendmail: Queue runner@00:30:00 for /var/spool/clientmqueue (sendmail)
    0  2269     1   0  44  0   7944   1256 nanslp IsJ   ??    0:00.02
/usr/sbin/cron -s
    0  2375     1   0  76  0  13268   1544 select Is    ??    0:00.00
/usr/sbin/inetd -wW -C 60
   70  2419  1594   0  44  0  60780   5680 select SsJ   ??    0:01.35
postgres: writer process    (postgres)
   70  2420  1594   0  44  0  60780   5664 select SsJ   ??    0:00.73
postgres: wal writer process    (postgres)
   70  2421  1594   0  44  0  60780   5688 zio->i DsJ   ??    0:00.15
postgres: autovacuum launcher process    (postgres)
   70  2422  1594   0  44  0  23788   3704 select SsJ   ??    0:00.12
postgres: stats collector process    (postgres)
   70  2425  1916   0  44  0  53492   5520 select SsJ   ??    0:01.19
postgres: writer process    (postgres)
   70  2426  1916   0  44  0  53492   5512 select SsJ   ??    0:00.68
postgres: wal writer process    (postgres)
   70  2427  1916   0  44  0  53492   5492 zio->i DsJ   ??    0:00.16
postgres: autovacuum launcher process    (postgres)
   70  2428  1916   0  44  0  24900   3624 select SsJ   ??    0:00.12
postgres: stats collector process    (postgres)
    0  2621  1249   0  44  0  42480   4228 select Is    ??    0:00.03
sshd: root@pts/0 (sshd)
    0  3711  1967   0  44  0   7944   1332 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3712  3711   0  44  0   7940    952 zio->i DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3713  2269   0  44  0   7944   1328 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3714  3713   0  44  0   7940    936 zio->i DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3717  1638   0  44  0   7944   1336 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3718  3717   0  44  0   7940    952 zio->i DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3719  1967   0  44  0   7944   1332 ppwait DJ    ??    0:00.00
cron: running job (cron)
    2  3720  3719   0  44  0   7944   1332 zio->i DVsJ  ??    0:00.00
cron: running job (cron)
    0  3721  2269   0  44  0   7944   1328 ppwait DJ    ??    0:00.00
cron: running job (cron)
    2  3722  3721   0  44  0   7944   1328 zio->i DVsJ  ??    0:00.00
cron: running job (cron)
    0  3735  1638   0  44  0   7944   1336 ppwait DJ    ??    0:00.00
cron: running job (cron)
    2  3736  3735   0  44  0   7944   1336 zio->i DVsJ  ??    0:00.00
cron: running job (cron)
    0  3737  1967   0  44  0   7944   1332 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3738  3737   0  44  0   4884    840 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3739  2269   0  44  0   7944   1328 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3740  3739   0  44  0   4884    824 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3743  1638   0  44  0   7944   1336 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3744  3743   0  44  0   4884    840 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3745  1967   0  44  0   7944   1332 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3746  3745   0  44  0   4884    840 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3747  2269   0  44  0   7944   1328 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3748  3747   0  44  0   4884    824 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3751  1638   0  44  0   7944   1336 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3752  3751   0  44  0   4884    840 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3753  1967   0  44  0   7944   1332 ppwait DJ    ??    0:00.00
cron: running job (cron)
    2  3754  3753   0  44  0   7944   1332 db->db DVsJ  ??    0:00.00
cron: running job (cron)
    0  3755  2269   0  44  0   7944   1328 ppwait DJ    ??    0:00.00
cron: running job (cron)
    2  3756  3755   0  44  0   7944   1328 db->db DVsJ  ??    0:00.00
cron: running job (cron)
    0  3769  1638   0  44  0   7944   1336 ppwait DJ    ??    0:00.00
cron: running job (cron)
    2  3770  3769   0  44  0   7944   1336 db->db DVsJ  ??    0:00.00
cron: running job (cron)
    0  3771  1967   0  44  0   7944   1332 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3772  3771   0  44  0   4884    840 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3773  2269   0  44  0   7944   1328 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3774  3773   0  44  0   4884    824 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3777  1638   0  44  0   7944   1336 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3778  3777   0  44  0   4884    840 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3779  1967   0  44  0   7944   1332 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3780  3779   0  44  0   4884    840 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3781  2269   0  44  0   7944   1328 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3782  3781   0  44  0   4884    824 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3785  1638   0  44  0   7944   1336 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3786  3785   0  44  0   4884    840 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3788  1249   0  44  0  42480   4344 select Ss    ??    0:00.02
sshd: root@pts/1 (sshd)
    0  3796  1967   0  44  0   7944   1332 ppwait DJ    ??    0:00.00
cron: running job (cron)
    2  3797  3796   0  44  0   7944   1332 db->db DVsJ  ??    0:00.00
cron: running job (cron)
    0  3798  2269   0  44  0   7944   1328 ppwait DJ    ??    0:00.00
cron: running job (cron)
    2  3799  3798   0  44  0   7944   1328 db->db DVsJ  ??    0:00.00
cron: running job (cron)
    0  3812  1638   0  44  0   7944   1336 ppwait DJ    ??    0:00.00
cron: running job (cron)
    2  3813  3812   0  44  0   7944   1336 db->db DVsJ  ??    0:00.00
cron: running job (cron)
    0  3818  1967   0  44  0   7944   1332 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3819  3818   0  44  0   4884    840 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3820  2269   0  44  0   7944   1328 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3821  3820   0  44  0   4884    824 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  3824  1638   0  44  0   7944   1336 piperd IJ    ??    0:00.00
cron: running job (cron)
    1  3825  3824   0  44  0   4884    840 zfs    DsJ   ??    0:00.00
/usr/libexec/atrun
    0  2409     1   0  76  0  11152   1160 ttyin  Is+   v0    0:00.00
/usr/libexec/getty Pc ttyv0
    0  2410     1   0  76  0  11152   1160 ttyin  Is+   v1    0:00.00
/usr/libexec/getty Pc ttyv1
    0  2411     1   0  76  0  11152   1160 ttyin  Is+   v2    0:00.00
/usr/libexec/getty Pc ttyv2
    0  2412     1   0  76  0  11152   1160 ttyin  Is+   v3    0:00.00
/usr/libexec/getty Pc ttyv3
    0  2413     1   0  76  0  11152   1160 ttyin  Is+   v4    0:00.00
/usr/libexec/getty Pc ttyv4
    0  2414     1   0  76  0  11152   1160 ttyin  Is+   v5    0:00.00
/usr/libexec/getty Pc ttyv5
    0  2415     1   0  76  0  11152   1160 ttyin  Is+   v6    0:00.00
/usr/libexec/getty Pc ttyv6
    0  2416     1   0  76  0  11152   1160 ttyin  Is+   v7    0:00.00
/usr/libexec/getty Pc ttyv7
    0  2624  2621   0  44  0  14556   2700 pause  Is     0    0:00.03 -csh =
(csh)
    0  3787  2624   0  44  0  19948   1600 zio->i D+     0    0:00.00
zpool status -v
    0  3792  3788   0  44  0  14556   2724 pause  Ss     1    0:00.02 -csh =
(csh)
    0  3827  3792   0  44  0  12272   1604 -      R+     1    0:00.00 ps ax=
Hwwl
FNFS#



--=20

Sam Fourman Jr.
Fourman Networks
http://www.fourmannetworks.com



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?AANLkTi=18wHu07TFg0Vvk1E%2BjDMjNYVqmPNpCb487diB>