From owner-freebsd-scsi@FreeBSD.ORG Mon Jun 8 14:13:11 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id F357F1000 for ; Mon, 8 Jun 2015 14:13:11 +0000 (UTC) (envelope-from sibananda.sahu@avagotech.com) Received: from mail-wi0-x234.google.com (mail-wi0-x234.google.com [IPv6:2a00:1450:400c:c05::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A54F1A94 for ; Mon, 8 Jun 2015 14:13:11 +0000 (UTC) (envelope-from sibananda.sahu@avagotech.com) Received: by wigg3 with SMTP id g3so53974752wig.1 for ; Mon, 08 Jun 2015 07:13:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=avagotech.com; s=google; h=from:mime-version:thread-index:date:message-id:subject:to:cc :content-type; bh=MbOw/p0BmX4BtW0n9v+RqG0AIttbSzd9ehLFAlJMQ+I=; b=xux6S1uDl3ToRr6HG8rPIFB/ET6l0+zKx8nwkz8qIZEIkiwlz19jRJNgqp5dB1y5OE NIZ6okxDqc8fLYlLsHll3NIo9iIPVzG/2wOQdZ1k0aZvhAw3+E1OQ8O/dFlBXqU0UtQX 6uehRPj4ofOP5mLFYi40HThYf2D3tjkAq2bko= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:mime-version:thread-index:date:message-id :subject:to:cc:content-type; bh=MbOw/p0BmX4BtW0n9v+RqG0AIttbSzd9ehLFAlJMQ+I=; b=D8rCsju0LYxLU51p5bWMaZQcrT7RHEkXw2t06YoW4JhM0LPMiBLvP/qm6KmncG8kbD P/yx47HvMCeFPXuQROYaF2gtqczV0Fokxszgrv7cJ0CbqKt3z+c/oDSIOhnBadjcIQL6 /8gbk73neZGiQqc2egQ5BFTtJOgFaCok+4tsJpvnODq04h0Ir80ruUyxd8ptzXbwjoTI PCA3fAAmcf1htbDDhrgwlTxZyNks1gRf2mYe+0V1yDA+xv99VbRbbuuJSbLhhigvCq0Y nN/ay/2FJKr0gT3CDu+/e4BYkZxjwsgrEOdOr9fZprfs6atk226q7/6eOTSA6G6s/jDM ZSNg== X-Gm-Message-State: ALoCoQlPESNT3MTUCNOJ+AVhUn8WObtE6/081dDFW9jpGrGLQcfKpZuDBQJkn2T03cMd6K/u2Xl7 X-Received: by 10.194.2.68 with SMTP id 4mr23677707wjs.82.1433772789844; Mon, 08 Jun 2015 07:13:09 -0700 (PDT) From: Sibananda Sahu MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AdCh9TDS5xEWt6+SSnqG+4nM6XvF1Q== Date: Mon, 8 Jun 2015 19:43:04 +0530 Message-ID: <74957f591aa372aee413a4f9fadda2e4@mail.gmail.com> Subject: IO stall in FreeBSD 9.1 To: freebsd-scsi@freebsd.org Cc: "Kenneth D. Merry" , scottl@freebsd.org, Scott Long , Konstantin Belousov Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Jun 2015 14:13:12 -0000 Hi all, I am working on mrsas(4) driver for AVAGO=E2=80=99s 3108 Invader controller= s in FreeBSD. Recently while testing on FreeBSD-9.1-RELEASE I have observed that IOs running on VDs behind AVAGO 3108 controller, stalls after a while. I have two VDs, both configured of RAID0. I use fio(1) for to fire IOs to the disks with posixaio as the ioengine. POST ISSUE ANALYSIS DATA: In the gstat(1) utility it shows some pending items in the Queue and there is no activity of IO but fio(1) is running in the background. dT: 1.002s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 11 0 0 0 0.0 0 0 0.0 0.0|da0 32 0 0 0 0.0 0 0 0.0 0.0| da1 0 0 0 0 0.0 0 0 0.0 0.0| ada0 0 0 0 0 0.0 0 0 0.0 0.0| ada0p1 0 0 0 0 0.0 0 0 0.0 0.0| ada0p2 0 0 0 0 0.0 0 0 0.0 0.0| ada0p3 I have also used the same configuration on FreeBSD-8.4-RELEASE and FreeBSD-10.0-RELEASE and the issue is not seen. IOs were running consistently and there was no IO stall at all. Furthermore, I have also checked that if mrsas(4) driver is causing this issue, but to my surprise mrsas(4) is not at all called after that IO stall= . I mean to say, the SIM driver=E2=80=99s action callback (mrsas_action() in = this case) is not at all called. Below is a dmesg log of camcontrol debug =E2=80=93TSX [bus:target:lun] afte= r IO stall: http://www.infysim.org/wp-content/dmesg_camcontrol_debug Can anyone provide any solution/suggestion to this issue? Thanks, Sibananda Sahu From owner-freebsd-scsi@FreeBSD.ORG Tue Jun 9 16:55:23 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [8.8.178.115]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 570438F0 for ; Tue, 9 Jun 2015 16:55:23 +0000 (UTC) (envelope-from sibananda.sahu@avagotech.com) Received: from mail-wg0-x229.google.com (mail-wg0-x229.google.com [IPv6:2a00:1450:400c:c00::229]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D65BA1434 for ; Tue, 9 Jun 2015 16:55:22 +0000 (UTC) (envelope-from sibananda.sahu@avagotech.com) Received: by wgme6 with SMTP id e6so18145398wgm.2 for ; Tue, 09 Jun 2015 09:55:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=avagotech.com; s=google; h=from:references:in-reply-to:mime-version:thread-index:date :message-id:subject:to:cc:content-type; bh=uh0BU1yMciD1HFtTbWz165+XQhafXhynWE8OocDm/ag=; b=KGg5M96+pcBq7rv2RLAs0/dNwSaDlJjE89PXJxmGyNUQBgsGbkwutFRXpsfrcFfpJV e5SWwVLy62N+d5z0LkUpcJw+JwqW1jtX85tHrXOFLhZes3T5VqEtI2xCN8lMq6dirqR7 hnZ1+UqOpIcFtL9GWJyPci31lDb1BG36s6zb0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:references:in-reply-to:mime-version :thread-index:date:message-id:subject:to:cc:content-type; bh=uh0BU1yMciD1HFtTbWz165+XQhafXhynWE8OocDm/ag=; b=DsH0DGy0NlWD80o0p+KfuWOv0hUF/ciqWCcD/c6n3BOLJsKL5dg9clO08Xs7Ur8m0+ 48nnTjbpYtWt7Z5dvsK5GKuCwSDXJ/gKL0pBvkDWDZ4NkYAbIDH4Zqm9gmG8yM5/YoKy QCrZ7P561G4uocUvScJLuPKcDYxZA3Hf+2KIbfEy0Dld5XeSD0JSFoTdDVpF93XHI85U x7IPSymXkk5rLbVhZP+TtcoU4RDRw0+a6tBs/3FWtVFQsW8pGn4wmQno5ZIspfpPQdXR h9HGNpqU2Cp3pavc+trpy4Omwa+Cyb+ZAd2mtj2E3ZSE8xbIIzujyJbrJ/h/J75RE/BE Nnug== X-Gm-Message-State: ALoCoQmJnX4kPbbqv/axc7hmaSI0GpOH1BCIRqN/DTt8njDYdpf6iNagu95PXh9RXgMOZ3oU46yE X-Received: by 10.180.100.164 with SMTP id ez4mr10260384wib.53.1433868921113; Tue, 09 Jun 2015 09:55:21 -0700 (PDT) From: Sibananda Sahu References: 74957f591aa372aee413a4f9fadda2e4@mail.gmail.com In-Reply-To: 74957f591aa372aee413a4f9fadda2e4@mail.gmail.com MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AdCh9TDS5xEWt6+SSnqG+4nM6XvF1QA3vWrg Date: Tue, 9 Jun 2015 22:25:14 +0530 Message-ID: Subject: RE: IO stall in FreeBSD 9.1 To: freebsd-scsi@freebsd.org Cc: "Kenneth D. Merry" , scottl@freebsd.org, Scott Long , Konstantin Belousov Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.20 X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Jun 2015 16:55:23 -0000 Is this a known issue on FreeBSD-9.1-RELEASE ? I went through the ERRATA and RELEASE NOTES of FreeBSD-9.1-RELEASE (where I observe the issue consistently) and FreeBSD-8.4-RELEASE (where this issue is not observed), but I really did not find any info on issues related to CAM or GEOM. Thanks, Sibananda Sahu *From:* Sibananda Sahu [mailto:sibananda.sahu@avagotech.com] *Sent:* Monday, June 08, 2015 7:43 PM *To:* 'freebsd-scsi@freebsd.org' *Cc:* 'Kenneth D. Merry'; 'scottl@freebsd.org'; 'Scott Long'; 'Konstantin Belousov' *Subject:* IO stall in FreeBSD 9.1 Hi all, I am working on mrsas(4) driver for AVAGO=E2=80=99s 3108 Invader controller= s in FreeBSD. Recently while testing on FreeBSD-9.1-RELEASE I have observed that IOs running on VDs behind AVAGO 3108 controller, stalls after a while. I have two VDs, both configured of RAID0. I use fio(1) for to fire IOs to the disks with posixaio as the ioengine. POST ISSUE ANALYSIS DATA: In the gstat(1) utility it shows some pending items in the Queue and there is no activity of IO but fio(1) is running in the background. dT: 1.002s w: 1.000s L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 11 0 0 0 0.0 0 0 0.0 0.0|da0 32 0 0 0 0.0 0 0 0.0 0.0| da1 0 0 0 0 0.0 0 0 0.0 0.0| ada0 0 0 0 0 0.0 0 0 0.0 0.0| ada0p1 0 0 0 0 0.0 0 0 0.0 0.0| ada0p2 0 0 0 0 0.0 0 0 0.0 0.0| ada0p3 I have also used the same configuration on FreeBSD-8.4-RELEASE and FreeBSD-10.0-RELEASE and the issue is not seen. IOs were running consistently and there was no IO stall at all. Furthermore, I have also checked that if mrsas(4) driver is causing this issue, but to my surprise mrsas(4) is not at all called after that IO stall= . I mean to say, the SIM driver=E2=80=99s action callback (mrsas_action() in = this case) is not at all called. Below is a dmesg log of camcontrol debug =E2=80=93TSX [bus:target:lun] afte= r IO stall: http://www.infysim.org/wp-content/dmesg_camcontrol_debug Can anyone provide any solution/suggestion to this issue? Thanks, Sibananda Sahu From owner-freebsd-scsi@FreeBSD.ORG Tue Jun 9 23:46:01 2015 Return-Path: Delivered-To: freebsd-scsi@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 794F0880 for ; Tue, 9 Jun 2015 23:46:01 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from mail.egr.msu.edu (hill.egr.msu.edu [35.9.37.162]) by mx1.freebsd.org (Postfix) with ESMTP id 5268E1AED for ; Tue, 9 Jun 2015 23:46:00 +0000 (UTC) (envelope-from mcdouga9@egr.msu.edu) Received: from hill (localhost [127.0.0.1]) by mail.egr.msu.edu (Postfix) with ESMTP id B9BDB51E70 for ; Tue, 9 Jun 2015 19:45:54 -0400 (EDT) X-Virus-Scanned: amavisd-new at egr.msu.edu Received: from mail.egr.msu.edu ([127.0.0.1]) by hill (hill.egr.msu.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tqynaRf-N8As for ; Tue, 9 Jun 2015 19:45:54 -0400 (EDT) Received: from EGR authenticated sender mcdouga9 Message-ID: <55777AB1.4040204@egr.msu.edu> Date: Tue, 09 Jun 2015 19:45:53 -0400 From: Adam McDougall User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 To: freebsd-scsi@freebsd.org Subject: Re: IO stall in FreeBSD 9.1 References: 74957f591aa372aee413a4f9fadda2e4@mail.gmail.com In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-scsi@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: SCSI subsystem List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Jun 2015 23:46:01 -0000 I don't know if this is a known issue, but you might try 9.3 (or 9.2 for comparison) because 9.1/9.2 are not supported anymore. On 06/09/2015 12:55, Sibananda Sahu wrote: > Is this a known issue on FreeBSD-9.1-RELEASE ? > > I went through the ERRATA and RELEASE NOTES of FreeBSD-9.1-RELEASE (where I > observe the issue consistently) and FreeBSD-8.4-RELEASE (where this issue > is not observed), but I really did not find any info on issues related to > CAM or GEOM. > > > > Thanks, > > Sibananda Sahu > > > > > > *From:* Sibananda Sahu [mailto:sibananda.sahu@avagotech.com] > *Sent:* Monday, June 08, 2015 7:43 PM > *To:* 'freebsd-scsi@freebsd.org' > *Cc:* 'Kenneth D. Merry'; 'scottl@freebsd.org'; 'Scott Long'; 'Konstantin > Belousov' > *Subject:* IO stall in FreeBSD 9.1 > > > > Hi all, > > > > I am working on mrsas(4) driver for AVAGO’s 3108 Invader controllers in > FreeBSD. > > Recently while testing on FreeBSD-9.1-RELEASE I have observed that IOs > running on VDs behind AVAGO 3108 controller, stalls after a while.