From owner-freebsd-stable@freebsd.org  Tue Sep 20 19:20:56 2016
Return-Path: <owner-freebsd-stable@freebsd.org>
Delivered-To: freebsd-stable@mailman.ysv.freebsd.org
Received: from mx1.freebsd.org (mx1.freebsd.org
 [IPv6:2001:1900:2254:206a::19:1])
 by mailman.ysv.freebsd.org (Postfix) with ESMTP id 49607BE29C0
 for <freebsd-stable@mailman.ysv.freebsd.org>;
 Tue, 20 Sep 2016 19:20:56 +0000 (UTC) (envelope-from slw@zxy.spb.ru)
Received: from zxy.spb.ru (zxy.spb.ru [195.70.199.98])
 (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits))
 (Client did not present a certificate)
 by mx1.freebsd.org (Postfix) with ESMTPS id 0FB2B839;
 Tue, 20 Sep 2016 19:20:56 +0000 (UTC) (envelope-from slw@zxy.spb.ru)
Received: from slw by zxy.spb.ru with local (Exim 4.86 (FreeBSD))
 (envelope-from <slw@zxy.spb.ru>)
 id 1bmQb7-000ARb-AO; Tue, 20 Sep 2016 22:20:53 +0300
Date: Tue, 20 Sep 2016 22:20:53 +0300
From: Slawa Olhovchenkov <slw@zxy.spb.ru>
To: John Baldwin <jhb@freebsd.org>
Cc: freebsd-stable@freebsd.org
Subject: Re: nginx and FreeBSD11
Message-ID: <20160920192053.GP2840@zxy.spb.ru>
References: <20160907191348.GD22212@zxy.spb.ru>
 <1823460.vTm8IvUQsF@ralph.baldwin.cx>
 <20160918162241.GE2960@zxy.spb.ru>
 <2122051.7RxZBKUSFc@ralph.baldwin.cx>
 <20160920065244.GO2840@zxy.spb.ru>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
In-Reply-To: <20160920065244.GO2840@zxy.spb.ru>
User-Agent: Mutt/1.5.24 (2015-08-30)
X-SA-Exim-Connect-IP: <locally generated>
X-SA-Exim-Mail-From: slw@zxy.spb.ru
X-SA-Exim-Scanned: No (on zxy.spb.ru); SAEximRunCond expanded to false
X-BeenThere: freebsd-stable@freebsd.org
X-Mailman-Version: 2.1.23
Precedence: list
List-Id: Production branch of FreeBSD source code <freebsd-stable.freebsd.org>
List-Unsubscribe: <https://lists.freebsd.org/mailman/options/freebsd-stable>, 
 <mailto:freebsd-stable-request@freebsd.org?subject=unsubscribe>
List-Archive: <http://lists.freebsd.org/pipermail/freebsd-stable/>
List-Post: <mailto:freebsd-stable@freebsd.org>
List-Help: <mailto:freebsd-stable-request@freebsd.org?subject=help>
List-Subscribe: <https://lists.freebsd.org/mailman/listinfo/freebsd-stable>,
 <mailto:freebsd-stable-request@freebsd.org?subject=subscribe>
X-List-Received-Date: Tue, 20 Sep 2016 19:20:56 -0000

On Tue, Sep 20, 2016 at 09:52:44AM +0300, Slawa Olhovchenkov wrote:

> On Mon, Sep 19, 2016 at 06:05:46PM -0700, John Baldwin wrote:
> 
> > > > If this panics, then vmspace_switch_aio() is not working for
> > > > some reason.
> > > 
> > > I am try using next DTrace script:
> > > ====
> > > #pragma D option dynvarsize=64m
> > > 
> > > int req[struct vmspace  *, void *];
> > > self int trace;
> > > 
> > > syscall:freebsd:aio_read:entry
> > > {
> > >         this->aio = *(struct aiocb *)copyin(arg0, sizeof(struct aiocb));
> > >         req[curthread->td_proc->p_vmspace, this->aio.aio_buf] = curthread->td_proc->p_pid; 
> > > }
> > > 
> > > fbt:kernel:aio_process_rw:entry
> > > {
> > >         self->job = args[0];
> > >         self->trace = 1;
> > > }
> > > 
> > > fbt:kernel:aio_process_rw:return
> > > /self->trace/
> > > {
> > >         req[self->job->userproc->p_vmspace, self->job->uaiocb.aio_buf] = 0;
> > >         self->job = 0;
> > >         self->trace = 0;
> > > }
> > > 
> > > fbt:kernel:vn_io_fault:entry
> > > /self->trace && !req[curthread->td_proc->p_vmspace, args[1]->uio_iov[0].iov_base]/
> > > {
> > >         this->buf = args[1]->uio_iov[0].iov_base;
> > >         printf("%Y vn_io_fault %p:%p pid %d\n", walltimestamp, curthread->td_proc->p_vmspace, this->buf, req[curthread->td_proc->p_vmspace, this->buf]);
> > > }
> > > ===
> > > 
> > > And don't got any messages near nginx core dump.
> > > What I can check next?
> > > May be check context/address space switch for kernel process?
> > 
> > Which CPU are you using?
> 
> CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.04-MHz K8-class CPU)
> 
> > Perhaps try disabling PCID support (I think vm.pmap.pcid_enabled=0 from
> > loader prompt or loader.conf)?  (Wondering if pmap_activate() is somehow not switching)

I am need some more time to test (day or two), but now this is like
workaround/solution: 12h runtime and peak hour w/o nginx crash.
(vm.pmap.pcid_enabled=0 in loader.conf).