From owner-freebsd-stable@freebsd.org Tue Sep 20 22:00:11 2016 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B0CF2BE3CF4 for ; Tue, 20 Sep 2016 22:00:11 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-io0-x229.google.com (mail-io0-x229.google.com [IPv6:2607:f8b0:4001:c06::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 80F025F6 for ; Tue, 20 Sep 2016 22:00:11 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-io0-x229.google.com with SMTP id m79so34505762ioo.3 for ; Tue, 20 Sep 2016 15:00:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to:cc; bh=N+sEnO3G/LxeidDHAO+0U2MJN0+jJwOzBwyShzuEbDA=; b=bfhoR7PekeYk369mP8hX19rXl3hk4sC26KSmMMtaCBcqjojFdCL9uT0AYIdutXyTKn dcibKDVwPCAKuUfVecZPRnLyp/uPm4qTH3TI4fjRA0Xu/6XF1grFcnZUHl9Boypl/gWc i6Ot74d3odwFRT8casoKGUQKqyWWYjowPTihXOkCIqaa1sgS7arIAfjfrA7YIK/5UR37 JJkG30ys+d07QuetlvgzoY/JBOKU5xsBdmpsl5llBNhWo1CMd+ToDw81I65EQulvzPV3 MihOf/zRG38YxxgrYk7psdC88PvGJ6Zk0p1vV7blxA03ocSb3ZfnxbelGwEr7sjO4qS6 Awbg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to:cc; bh=N+sEnO3G/LxeidDHAO+0U2MJN0+jJwOzBwyShzuEbDA=; b=OSUt+BiqysNDWLeMvs/dUZA8QYEZwX74LQQHh3BxwPu4KaMSexMN8K5B89dsgDGUFd tBeMtXdfD5EozPMoisKueK/8siy5H9jPKbljBz8LKnsDE0LCJjWNyEyZz1d9mNtwLrzW F0WwE3GeUauoqiCPehz5WIkDZSyKZ2n5MLh5xVM8VRi05Ki6oHCBT284VL475rNUZxuH sKNJIHR/OgnoKTdSER1S4ym/cpndKmMnAyVYfSJvxi3oByvuERKah47Lfed6SXBGIBwM gwHFm5JsqjKo4yyTq4WOxQbdiuBQ/RePakAX8jrKwzN7D4n7Fi0ufNayBRP3ZycG7xHb T0Yg== X-Gm-Message-State: AE9vXwP67u7YDHHQtqpFMR1ZsvgZYHsLdJ2bpuvzutOtBxzu/uazZajZlyeqP1edVaiG3Otks5MhAcF1Yf2OMw== X-Received: by 10.107.184.131 with SMTP id i125mr51553459iof.167.1474408810769; Tue, 20 Sep 2016 15:00:10 -0700 (PDT) MIME-Version: 1.0 Sender: wlosh@bsdimp.com Received: by 10.36.65.7 with HTTP; Tue, 20 Sep 2016 15:00:10 -0700 (PDT) X-Originating-IP: [69.53.245.200] In-Reply-To: <20160920214729.GS2840@zxy.spb.ru> References: <20160907191348.GD22212@zxy.spb.ru> <1823460.vTm8IvUQsF@ralph.baldwin.cx> <20160918162241.GE2960@zxy.spb.ru> <2122051.7RxZBKUSFc@ralph.baldwin.cx> <20160920065244.GO2840@zxy.spb.ru> <20160920192053.GP2840@zxy.spb.ru> <20160920201925.GI38409@kib.kiev.ua> <20160920203853.GR2840@zxy.spb.ru> <20160920211517.GJ38409@kib.kiev.ua> <20160920214729.GS2840@zxy.spb.ru> From: Warner Losh Date: Tue, 20 Sep 2016 16:00:10 -0600 X-Google-Sender-Auth: oeE2MriJ8mCLk8wyBN64xENAK4g Message-ID: Subject: Re: nginx and FreeBSD11 To: Slawa Olhovchenkov Cc: Konstantin Belousov , FreeBSD-STABLE Mailing List , John Baldwin Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 20 Sep 2016 22:00:11 -0000 On Tue, Sep 20, 2016 at 3:47 PM, Slawa Olhovchenkov wrote: > On Wed, Sep 21, 2016 at 12:15:17AM +0300, Konstantin Belousov wrote: > >> On Tue, Sep 20, 2016 at 11:38:54PM +0300, Slawa Olhovchenkov wrote: >> > On Tue, Sep 20, 2016 at 11:19:25PM +0300, Konstantin Belousov wrote: >> > >> > > On Tue, Sep 20, 2016 at 10:20:53PM +0300, Slawa Olhovchenkov wrote: >> > > > On Tue, Sep 20, 2016 at 09:52:44AM +0300, Slawa Olhovchenkov wrote: >> > > > >> > > > > On Mon, Sep 19, 2016 at 06:05:46PM -0700, John Baldwin wrote: >> > > > > >> > > > > > > > If this panics, then vmspace_switch_aio() is not working for >> > > > > > > > some reason. >> > > > > > > >> > > > > > > I am try using next DTrace script: >> > > > > > > ==== >> > > > > > > #pragma D option dynvarsize=64m >> > > > > > > >> > > > > > > int req[struct vmspace *, void *]; >> > > > > > > self int trace; >> > > > > > > >> > > > > > > syscall:freebsd:aio_read:entry >> > > > > > > { >> > > > > > > this->aio = *(struct aiocb *)copyin(arg0, sizeof(struct aiocb)); >> > > > > > > req[curthread->td_proc->p_vmspace, this->aio.aio_buf] = curthread->td_proc->p_pid; >> > > > > > > } >> > > > > > > >> > > > > > > fbt:kernel:aio_process_rw:entry >> > > > > > > { >> > > > > > > self->job = args[0]; >> > > > > > > self->trace = 1; >> > > > > > > } >> > > > > > > >> > > > > > > fbt:kernel:aio_process_rw:return >> > > > > > > /self->trace/ >> > > > > > > { >> > > > > > > req[self->job->userproc->p_vmspace, self->job->uaiocb.aio_buf] = 0; >> > > > > > > self->job = 0; >> > > > > > > self->trace = 0; >> > > > > > > } >> > > > > > > >> > > > > > > fbt:kernel:vn_io_fault:entry >> > > > > > > /self->trace && !req[curthread->td_proc->p_vmspace, args[1]->uio_iov[0].iov_base]/ >> > > > > > > { >> > > > > > > this->buf = args[1]->uio_iov[0].iov_base; >> > > > > > > printf("%Y vn_io_fault %p:%p pid %d\n", walltimestamp, curthread->td_proc->p_vmspace, this->buf, req[curthread->td_proc->p_vmspace, this->buf]); >> > > > > > > } >> > > > > > > === >> > > > > > > >> > > > > > > And don't got any messages near nginx core dump. >> > > > > > > What I can check next? >> > > > > > > May be check context/address space switch for kernel process? >> > > > > > >> > > > > > Which CPU are you using? >> > > > > >> > > > > CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.04-MHz K8-class CPU) >> > > Is this sandy bridge ? >> > >> > Sandy Bridge EP >> > >> > > Show me first 100 lines of the verbose dmesg, >> > >> > After day or two, after end of this test run -- I am need to enable verbose. >> > >> > > I want to see cpu features lines. In particular, does you CPU support >> > > the INVPCID feature. >> > >> > CPU: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz (2000.05-MHz K8-class CPU) >> > Origin="GenuineIntel" Id=0x206d7 Family=0x6 Model=0x2d Stepping=7 >> > Features=0xbfebfbff >> > Features2=0x1fbee3ff >> > AMD Features=0x2c100800 >> > AMD Features2=0x1 >> > XSAVE Features=0x1 >> > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID >> > TSC: P-state invariant, performance statistics >> > >> > I am don't see this feature before E5v3: >> > >> > CPU: Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz (2600.06-MHz K8-class CPU) >> > Origin="GenuineIntel" Id=0x306e4 Family=0x6 Model=0x3e Stepping=4 >> > Features=0xbfebfbff >> > Features2=0x7fbee3ff >> > AMD Features=0x2c100800 >> > AMD Features2=0x1 >> > Structured Extended Features=0x281 >> > XSAVE Features=0x1 >> > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr >> > TSC: P-state invariant, performance statistics >> > >> > (don't run 11.0 on this CPU) >> Ok. >> >> > >> > CPU: Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz (2600.05-MHz K8-class CPU) >> > Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 >> > Features=0xbfebfbff >> > Features2=0x7ffefbff >> > AMD Features=0x2c100800 >> > AMD Features2=0x21 >> > Structured Extended Features=0x37ab >> > XSAVE Features=0x1 >> > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr >> > TSC: P-state invariant, performance statistics >> > >> > (11.0 run w/o this issuse) >> Do you mean that similarly configured nginx+aio do not demonstrate the corruption on this machine ? > > Yes. > But different storage configuration and different pattern load. > > Also 11.0 run w/o this issuse on > > CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz (2200.04-MHz K8-class CPU) > Origin="GenuineIntel" Id=0x406f1 Family=0x6 Model=0x4f Stepping=1 > Features=0xbfebfbff > Features2=0x7ffefbff > AMD Features=0x2c100800 > AMD Features2=0x121 > Structured Extended Features=0x21cbfbb > XSAVE Features=0x1 > VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr > TSC: P-state invariant, performance statistics > > PS: all systems is dual-cpu. Does this mean 2 cores or two sockets? We've seen a similar hang with the following CPU: CPU: Intel(R) Xeon(R) CPU E5-2697 v2 @ 2.70GHz (2700.06-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306e4 Family=0x6 Model=0x3e Stepping=4 Features=0xbfebfbff Features2=0x7fbee3ff AMD Features=0x2c100800 AMD Features2=0x1 Structured Extended Features=0x281 XSAVE Features=0x1 VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 274877906944 (262144 MB) avail memory = 267146330112 (254770 MB) 12 cores x 2 SMT x 1 socket Warner