From owner-freebsd-testing@FreeBSD.ORG Mon Apr 27 18:16:35 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 52729556; Mon, 27 Apr 2015 18:16:35 +0000 (UTC) Received: from mail-ob0-x233.google.com (mail-ob0-x233.google.com [IPv6:2607:f8b0:4003:c01::233]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1337F1816; Mon, 27 Apr 2015 18:16:35 +0000 (UTC) Received: by oblw8 with SMTP id w8so90063218obl.0; Mon, 27 Apr 2015 11:16:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:subject:date:message-id:cc:to:mime-version; bh=Zl8xyfs2Jg9+/AMq0thMAGPFasWSGkVlsjc9Q7DgIEQ=; b=Nj/6KLNcNLH1gz4ti+YxFmG7aiLJwZx+Vh9HgBgt8YkbQvplmIFsI0N6dxzNsg2zUQ rkj/vYrwQcokLyK+CBsC2W1ZCj2qfqAj1VxVqcZviKr+QpgS1wTIasXiOkO7C0IuBWG6 sQd4bdyVHD8G+M7v5GCO8r9Pg/paNuYK8OSJeciXUJwVMSrI1YphuZHoGVkQBjtVIodf x9YOl3BQDZtaV/Sc1Dn33RyOrUggPQ1+GOd8Im2S96NpSDwTJp6WWKMtuYEJBpWXJHgb rIVvI0+kVRarWenbVv7yMc+8xdfewjhnRjqpOaewFN3W+312+3IZfRI2lpKXakM19Chx vYIQ== X-Received: by 10.202.91.194 with SMTP id p185mr10622755oib.9.1430158594318; Mon, 27 Apr 2015 11:16:34 -0700 (PDT) Received: from ?IPv6:2601:8:ab80:7d6:f039:b350:5840:757f? ([2601:8:ab80:7d6:f039:b350:5840:757f]) by mx.google.com with ESMTPSA id s3sm5361340obt.27.2015.04.27.11.16.33 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 27 Apr 2015 11:16:33 -0700 (PDT) From: Garrett Cooper X-Pgp-Agent: GPGMail 2.5b6 Content-Type: multipart/signed; boundary="Apple-Mail=_13AEAF56-63DF-48BE-8CBC-815BD69CC552"; protocol="application/pgp-signature"; micalg=pgp-sha512 Subject: cam(4) timeouts in bhyve/kyua runs up on Jenkins? Date: Mon, 27 Apr 2015 11:16:32 -0700 Message-Id: <28845361-E624-4CB9-B255-1B2C1FD0DF28@gmail.com> Cc: FreeBSD Current , "freebsd-testing@freebsd.org" To: Alexander Motin Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Apr 2015 18:16:35 -0000 --Apple-Mail=_13AEAF56-63DF-48BE-8CBC-815BD69CC552 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Hi mav! I was looking at the console log for the latest kyua run and = I=92ve noticed that it=92s timing out a bit more [1] than it was = previously [2]. I=92ve seen some of your commits recently to cam(4) = dealing with bhyve =97 has there been a performance regression there? Thanks! -NGie 1. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/940/console 2. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/983/console --Apple-Mail=_13AEAF56-63DF-48BE-8CBC-815BD69CC552 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJVPn0AAAoJEMZr5QU6S73ey0cH/AgoUne2Iev6ojMRW/2PPfyn 6Fwma5aaXZkyzEZ2mFGVoE0aQy+RjJP5Q5fUNPXhpwciq+CA3Gws6ye+z7RM4OHH lL8TUVk/+8Xp816jxqnyQJIHvfZUbU9XHVRIaNTTk+mW684n/0i/659K7p1yRW7w S0qohKXbbJy6sRzRy4lcrwulAxXSHFifBoYq2NX0kd/VGaKa+Ow7zQfpSlxkklsl a/4Ghl2992v5wvtm4d+gBhBYcF23jbb938gizgLfZesmBq/yHyPlkjNlnPAwXdsm A9lobeDiscbDR9RSVuOsOYLpqiE12LJeuIh98+Zl7t1AwsILsnWZodNOpvtEP8c= =CwX5 -----END PGP SIGNATURE----- --Apple-Mail=_13AEAF56-63DF-48BE-8CBC-815BD69CC552-- From owner-freebsd-testing@FreeBSD.ORG Mon Apr 27 18:17:34 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 9E5CC67D; Mon, 27 Apr 2015 18:17:34 +0000 (UTC) Received: from mail-oi0-x230.google.com (mail-oi0-x230.google.com [IPv6:2607:f8b0:4003:c06::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 67917182C; Mon, 27 Apr 2015 18:17:34 +0000 (UTC) Received: by oica37 with SMTP id a37so96401541oic.0; Mon, 27 Apr 2015 11:17:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to; bh=z4Fzs/bguAXDQlotsTdHQwpOgY31jPtv03prJdCbfGI=; b=H9zJM79oYaKxgsUczhqHxl8skiLCI3OOhzSBlXgR/BtWsuK/IyxxokSTbkOKlq7RZb 2RWAUh2JtQ6N9fBsausI+goKIcD9LDt/hP2zrqSBEAiHxakio82gLu3kMiip1dzbNuRP uUewK9bHidlDN73d243dbV57Xch5Q2+ed/Ew7zqQnE/EnflqGaQ14TLRODx/psW10omh YksIZ8WDejV3rM/oZp+Mzbxhkr6hZmRAB2GJfRwHb3YkF9i3EKvyDSUa61h3Rbf154oA lMZWJ+rSlYOJodWi0vXWcXjtRKzKaD5KukIgbwt09TGhJqezx0NtATLm7+80nHGo7utQ +yQQ== X-Received: by 10.60.177.73 with SMTP id co9mr11121916oec.5.1430158653593; Mon, 27 Apr 2015 11:17:33 -0700 (PDT) Received: from ?IPv6:2601:8:ab80:7d6:f039:b350:5840:757f? ([2601:8:ab80:7d6:f039:b350:5840:757f]) by mx.google.com with ESMTPSA id s3sm5361340obt.27.2015.04.27.11.17.32 (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 27 Apr 2015 11:17:32 -0700 (PDT) Subject: Re: cam(4) timeouts in bhyve/kyua runs up on Jenkins? Mime-Version: 1.0 (Mac OS X Mail 7.3 \(1878.6\)) Content-Type: multipart/signed; boundary="Apple-Mail=_1BCEB238-21DD-4009-BC57-E71DAB7B1FE1"; protocol="application/pgp-signature"; micalg=pgp-sha512 X-Pgp-Agent: GPGMail 2.5b6 From: Garrett Cooper In-Reply-To: <28845361-E624-4CB9-B255-1B2C1FD0DF28@gmail.com> Date: Mon, 27 Apr 2015 11:17:33 -0700 Cc: FreeBSD Current , "freebsd-testing@freebsd.org" Message-Id: <4BEE0A54-3AD2-42DB-AF4E-2B0A5E34DF4E@gmail.com> References: <28845361-E624-4CB9-B255-1B2C1FD0DF28@gmail.com> To: Alexander Motin X-Mailer: Apple Mail (2.1878.6) X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 27 Apr 2015 18:17:34 -0000 --Apple-Mail=_1BCEB238-21DD-4009-BC57-E71DAB7B1FE1 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 On Apr 27, 2015, at 11:16, Garrett Cooper wrote: > Hi mav! > I was looking at the console log for the latest kyua run and = I=92ve noticed that it=92s timing out a bit more [1] than it was = previously [2]. I=92ve seen some of your commits recently to cam(4) = dealing with bhyve =97 has there been a performance regression there? > Thanks! > -NGie >=20 > 1. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/940/console > 2. https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/983/console (Sorry for not being more explicit for the archives) These are the timeouts I=92m referring to: ahcich0: is 00000000 cs 00000000 ss 1f000000 rs 1f000000 tfd 50 serr = 00000000 cmd 1000dc17 (ada0:ahcich0:0:0:0): WRITE_FPDMA_QUEUED. ACB: 61 08 a8 54 1e 40 00 00 = 00 00 00 00 (ada0:ahcich0:0:0:0): CAM status: Command timeout (ada0:ahcich0:0:0:0): Retrying command --Apple-Mail=_1BCEB238-21DD-4009-BC57-E71DAB7B1FE1 Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename=signature.asc Content-Type: application/pgp-signature; name=signature.asc Content-Description: Message signed with OpenPGP using GPGMail -----BEGIN PGP SIGNATURE----- Comment: GPGTools - https://gpgtools.org iQEcBAEBCgAGBQJVPn09AAoJEMZr5QU6S73e8VsH/2m1z6UsYAFpE6qzKSIJkVy+ 2qXnfAiedfjpetvBILzqeWDFD9qnaTGv0HUo+vdt39NURJePA9lPMnUHQuq1x+rj s53xLub/HvFQkijyA7x8nBFtPW5U84cdesjfmzgpMk4DDulLmBS4yC29tcu7PUMK Whsgd815WceQ9HLUw78FdIIRmL8OJAbvW4X8KCODrfkAHmjBSItUGaJsgJQZrVm8 M4wnUFzjDAYK8Gh/iZ0IevYglfzwi19UC4uuSr5QBJgzXfo8VJ//MJXjJtldidJA Uq79Dwa/krTKgqZKqnv48DykCzm/brzX5qal0sKQLBT1zgrhmnUFeFu4z8Huots= =salu -----END PGP SIGNATURE----- --Apple-Mail=_1BCEB238-21DD-4009-BC57-E71DAB7B1FE1-- From owner-freebsd-testing@FreeBSD.ORG Tue Apr 28 07:54:21 2015 Return-Path: Delivered-To: freebsd-testing@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 0647EF6C; Tue, 28 Apr 2015 07:54:21 +0000 (UTC) Received: from mail-la0-x229.google.com (mail-la0-x229.google.com [IPv6:2a00:1450:4010:c03::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7D944157B; Tue, 28 Apr 2015 07:54:20 +0000 (UTC) Received: by labbd9 with SMTP id bd9so98031267lab.2; Tue, 28 Apr 2015 00:54:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=sender:message-id:date:from:user-agent:mime-version:to:cc:subject :references:in-reply-to:content-type:content-transfer-encoding; bh=uVciRwHOvrc8tSl9V3hORQZVT9HWfPA9qvTsy8XVxI4=; b=lrhPIGC0tVj4YCgqL402sOKglFlkGJFzdwQff3SwoJ4siMNI/bl4aVgtGu6LvCrUJf zlB9BOBuKShHEK/j7NiW9Bvy9qfDBpENSEScx1QseEmbRNqKQxti58MCybdwMtT/sVtL 6m+83S5SbxW2Er4fg4o98ludxkLakb1prSrBps+PRIJwUuOojwVrKnBNcj4sOe3x9OzD EySyldBYU15vtV6/Qucppqebq9RvCfQiXnbJ5JLmuk1bCLRlxureNB2/0CYlaE5FGJrQ RT7/V5g+4jv0arvQBeXFeSBFvdcPZThVgDVQ6zd7DEcE0fyPojJtz0ZdMWwTxjhK4zLO kNTQ== X-Received: by 10.112.57.197 with SMTP id k5mr13807683lbq.102.1430207658690; Tue, 28 Apr 2015 00:54:18 -0700 (PDT) Received: from mavbook.mavhome.dp.ua ([91.198.175.1]) by mx.google.com with ESMTPSA id o8sm5361947lal.2.2015.04.28.00.54.16 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 28 Apr 2015 00:54:17 -0700 (PDT) Sender: Alexander Motin Message-ID: <553F3CA7.9090601@FreeBSD.org> Date: Tue, 28 Apr 2015 10:54:15 +0300 From: Alexander Motin User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: Garrett Cooper CC: FreeBSD Current , "freebsd-testing@freebsd.org" Subject: Re: cam(4) timeouts in bhyve/kyua runs up on Jenkins? References: <28845361-E624-4CB9-B255-1B2C1FD0DF28@gmail.com> <4BEE0A54-3AD2-42DB-AF4E-2B0A5E34DF4E@gmail.com> In-Reply-To: <4BEE0A54-3AD2-42DB-AF4E-2B0A5E34DF4E@gmail.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-testing@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Testing on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Apr 2015 07:54:21 -0000 Hi. On 27.04.2015 21:17, Garrett Cooper wrote: > On Apr 27, 2015, at 11:16, Garrett Cooper > wrote: > >> I was looking at the console log for the latest kyua run and I’ve >> noticed that it’s timing out a bit more [1] than it was >> previously [2]. I’ve seen some of your commits recently to cam(4) >> dealing with bhyve — has there been a performance regression >> there? Thanks! -NGie >> >> 1. >> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/940/console >> 2. >> https://jenkins.freebsd.org/job/FreeBSD_HEAD-tests2/983/console > > (Sorry for not being more explicit for the archives) These are the > timeouts I’m referring to: > > ahcich0: is 00000000 cs 00000000 ss 1f000000 rs 1f000000 tfd 50 > serr 00000000 cmd 1000dc17 (ada0:ahcich0:0:0:0): > WRITE_FPDMA_QUEUED. ACB: 61 08 a8 54 1e 40 00 00 00 00 00 00 > (ada0:ahcich0:0:0:0): CAM status: Command timeout > (ada0:ahcich0:0:0:0): Retrying command Last time I was more working on bhyve host disk emulation, rather then on cam(4) running on guest. Considering that, what guest and what host versions are you running? Is there any other load on host except this VM that could cause I/O delays high enough to trigger timeouts? What are you using to back the virtual disk (file, zvol, ...)? -- Alexander Motin