From owner-freebsd-stable@FreeBSD.ORG Tue Oct 2 10:03:10 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 0180E1065672 for ; Tue, 2 Oct 2012 10:03:10 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-bk0-f54.google.com (mail-bk0-f54.google.com [209.85.214.54]) by mx1.freebsd.org (Postfix) with ESMTP id 6A34B8FC1F for ; Tue, 2 Oct 2012 10:03:09 +0000 (UTC) Received: by bkcjf20 with SMTP id jf20so4395720bkc.13 for ; Tue, 02 Oct 2012 03:03:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date:message-id :references:to:x-mailer; bh=exaxxHSBEAnDEUM7wsv3962CW26SH+YbRj4FbKIh2RA=; b=Fuc1WXJ2if7X2Vn80oVrX//zqJFCjrQhsOGTV4WMpSo/Cp8pgJFYqyLOuNux+9I/Xf 3FFte13UDdwIqsZNwzrzqCNaHr6C89wNaXyyxpGvE7q52UtAPak1dkhDDjlxOBnESvx6 llcGUHYjaTXaJCjPT5L1eEQa/55iO6sDR49C3i8dfgBtzNWLsaHtKzG/tD9cC+HOv3ux JK0fjxrmpJuwxrGSdWUgwODCK5aL4HpIyPsXq4m+DhOMzD3ki0tPyTpAGup1ORXn+l9i EFdNzRKVPVunuBcvD3MECbC1JSiyxkZt774tJQxfORYk27Bke8UbACb/XYwK552OLun6 Qhqw== Received: by 10.204.148.12 with SMTP id n12mr6431450bkv.62.1349172188044; Tue, 02 Oct 2012 03:03:08 -0700 (PDT) Received: from ndenevsa.sf.moneybookers.net (g1.moneybookers.com. [217.18.249.148]) by mx.google.com with ESMTPS id o17sm623433bkw.9.2012.10.02.03.03.05 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 02 Oct 2012 03:03:06 -0700 (PDT) Mime-Version: 1.0 (Mac OS X Mail 6.1 \(1498\)) From: Nikolay Denev In-Reply-To: <72A4B763-D36B-4912-8C20-7373A0562EA1@gmail.com> Date: Tue, 2 Oct 2012 13:03:04 +0300 Message-Id: <11028C2E-9DB0-4B71-A7B1-98160D5AEA93@gmail.com> References: <72A4B763-D36B-4912-8C20-7373A0562EA1@gmail.com> To: "freebsd-stable@freebsd.org" X-Mailer: Apple Mail (2.1498) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: Re: CAM Target Layer and Linux (continued) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Oct 2012 10:03:10 -0000 On Sep 27, 2012, at 6:33 PM, Nikolay Denev wrote: > Hi All, >=20 > With the help of Chuck Tuffli, I'm now able to use CTL to export a = zvol over FC to a Linux host: >=20 > LUN Backend Size (Blocks) BS Serial Number Device ID =20= > 0 block 4185915392 512 FBSDZFS001 ORA_ASM_01 =20= > lun_type=3D0 > num_threads=3D14 > file=3D/dev/zvol/tank/oracle_asm_01 > 1 block 4185915392 512 FBSDZFS002 ORA_ASM_02 =20= > lun_type=3D0 > num_threads=3D14 > file=3D/dev/zvol/tank/oracle_asm_02 > 2 block 4185915392 512 FBSDZFS003 ORA_ASM_03 =20= > lun_type=3D0 > num_threads=3D14 > file=3D/dev/zvol/tank/oracle_asm_03 > 3 block 4185915392 512 FBSDZFS004 ORA_ASM_04 =20= > lun_type=3D0 > num_threads=3D14 > file=3D/dev/zvol/tank/oracle_asm_04 >=20 > Then we ran some tests using Oracle's ORION benchmark tool from the = Linux host. > We ran one test which passed successfully, > then I've just disabled zfs prefetch -> "vfs.zfs.prefetch_disable=3D1" > and rerun the test, which failed due to this error. >=20 > On the FreeBSD side: >=20 > (0:3:0:1): READ(10). CDB: 28 0 84 f9 58 0 0 4 0 0=20 > (0:3:0:1): Tag: 0x116220, Type: 1 > (0:3:0:1): CTL Status: SCSI Error > (0:3:0:1): SCSI Status: Check Condition > (0:3:0:1): SCSI sense: NOT READY asc:4b,0 (Data phase error) >=20 > Linux reported : >=20 > sd 4:0:0:1: Device not ready: <6>: Current: sense key: Not Ready > Add. Sense: Data phase error >=20 > end_request: I/O error, dev sdr, sector 2230933504 > device-mapper: multipath: Failing path 65:16. > sd 4:0:0:1: Device not ready: <6>: Current: sense key: Not Ready > Add. Sense: Data phase error >=20 > end_request: I/O error, dev sdr, sector 2230934528 >=20 > There are no other suspicious messages in dmesg. >=20 > Also, ctladm dumpooa does not show anything. > Here is dumpscructs output : >=20 > CTL IID to WWPN map start: > CTL IID to WWPN map end > CTL Persistent Reservation information start: > CTL Persistent Reservation information end > CTL Frontends: > Frontend CTL ioctl Type 4 pport 0 vport 0 WWNN 0 WWPN 0 > Frontend ctl2cam Type 8 pport 0 vport 0 WWNN 0x5000000995680700 WWPN = 0x5000000995680702 > Frontend CTL internal Type 8 pport 0 vport 0 WWNN 0 WWPN 0 > Frontend isp0 Type 1 pport 0 vport 0 WWNN 0x20000024ff376b98 WWPN = 0x21000024ff376b98 > isp0: max tagged openings: 4096, max dev openings: 4096 > isp0: max_ccbs: 20488, ccb_count: 79 > isp0: ccb_freeq is NOT empty > isp0: alloc_queue.entries 0, alloc_openings 4096 > isp0: qfrozen_cnt:0:0:0:0:0 > (ctl2:isp0:0:0:0): 0 requests total waiting for CCBs > (ctl2:isp0:0:0:0): 0 CCBs oustanding (17788811 allocated, 17788811 = freed) > (ctl2:isp0:0:0:0): 0 CTIOs outstanding (17788811 sent, 17788811 = returned > (ctl4:isp0:0:0:1): 0 requests total waiting for CCBs > (ctl4:isp0:0:0:1): 0 CCBs oustanding (16708305 allocated, 16708305 = freed) > (ctl4:isp0:0:0:1): 0 CTIOs outstanding (16708305 sent, 16708305 = returned > (ctl6:isp0:0:0:2): 0 requests total waiting for CCBs > (ctl6:isp0:0:0:2): 0 CCBs oustanding (16712865 allocated, 16712865 = freed) > (ctl6:isp0:0:0:2): 0 CTIOs outstanding (16712865 sent, 16712865 = returned > (ctl8:isp0:0:0:3): 0 requests total waiting for CCBs > (ctl8:isp0:0:0:3): 0 CCBs oustanding (16699727 allocated, 16699727 = freed) > (ctl8:isp0:0:0:3): 0 CTIOs outstanding (16699727 sent, 16699727 = returned > isp1: max tagged openings: 4096, max dev openings: 4096 > isp1: max_ccbs: 20488, ccb_count: 1 > isp1: ccb_freeq is NOT empty > isp1: alloc_queue.entries 0, alloc_openings 4096 > isp1: qfrozen_cnt:0:0:0:0:0 > (ctl3:isp1:0:0:0): 0 requests total waiting for CCBs > (ctl3:isp1:0:0:0): 0 CCBs oustanding (0 allocated, 0 freed) > (ctl3:isp1:0:0:0): 0 CTIOs outstanding (0 sent, 0 returned > (ctl5:isp1:0:0:1): 0 requests total waiting for CCBs > (ctl5:isp1:0:0:1): 0 CCBs oustanding (0 allocated, 0 freed) > (ctl5:isp1:0:0:1): 0 CTIOs outstanding (0 sent, 0 returned > (ctl7:isp1:0:0:2): 0 requests total waiting for CCBs > (ctl7:isp1:0:0:2): 0 CCBs oustanding (0 allocated, 0 freed) > (ctl7:isp1:0:0:2): 0 CTIOs outstanding (0 sent, 0 returned > (ctl9:isp1:0:0:3): 0 requests total waiting for CCBs > (ctl9:isp1:0:0:3): 0 CCBs oustanding (0 allocated, 0 freed) > (ctl9:isp1:0:0:3): 0 CTIOs outstanding (0 sent, 0 returned > Frontend isp1 Type 1 pport 1 vport 0 WWNN 0x20000024ff376b99 WWPN = 0x21000024ff376b99 > isp0: max tagged openings: 4096, max dev openings: 4096 > isp0: max_ccbs: 20488, ccb_count: 79 > isp0: ccb_freeq is NOT empty > isp0: alloc_queue.entries 0, alloc_openings 4096 > isp0: qfrozen_cnt:0:0:0:0:0 > (ctl2:isp0:0:0:0): 0 requests total waiting for CCBs > (ctl2:isp0:0:0:0): 0 CCBs oustanding (17788811 allocated, 17788811 = freed) > (ctl2:isp0:0:0:0): 0 CTIOs outstanding (17788811 sent, 17788811 = returned > (ctl4:isp0:0:0:1): 0 requests total waiting for CCBs > (ctl4:isp0:0:0:1): 0 CCBs oustanding (16708305 allocated, 16708305 = freed) > (ctl4:isp0:0:0:1): 0 CTIOs outstanding (16708305 sent, 16708305 = returned > (ctl6:isp0:0:0:2): 0 requests total waiting for CCBs > (ctl6:isp0:0:0:2): 0 CCBs oustanding (16712865 allocated, 16712865 = freed) > (ctl6:isp0:0:0:2): 0 CTIOs outstanding (16712865 sent, 16712865 = returned > (ctl8:isp0:0:0:3): 0 requests total waiting for CCBs > (ctl8:isp0:0:0:3): 0 CCBs oustanding (16699727 allocated, 16699727 = freed) > (ctl8:isp0:0:0:3): 0 CTIOs outstanding (16699727 sent, 16699727 = returned > isp1: max tagged openings: 4096, max dev openings: 4096 > isp1: max_ccbs: 20488, ccb_count: 1 > isp1: ccb_freeq is NOT empty > isp1: alloc_queue.entries 0, alloc_openings 4096 > isp1: qfrozen_cnt:0:0:0:0:0 > (ctl3:isp1:0:0:0): 0 requests total waiting for CCBs > (ctl3:isp1:0:0:0): 0 CCBs oustanding (0 allocated, 0 freed) > (ctl3:isp1:0:0:0): 0 CTIOs outstanding (0 sent, 0 returned > (ctl5:isp1:0:0:1): 0 requests total waiting for CCBs > (ctl5:isp1:0:0:1): 0 CCBs oustanding (0 allocated, 0 freed) > (ctl5:isp1:0:0:1): 0 CTIOs outstanding (0 sent, 0 returned > (ctl7:isp1:0:0:2): 0 requests total waiting for CCBs > (ctl7:isp1:0:0:2): 0 CCBs oustanding (0 allocated, 0 freed) > (ctl7:isp1:0:0:2): 0 CTIOs outstanding (0 sent, 0 returned > (ctl9:isp1:0:0:3): 0 requests total waiting for CCBs > (ctl9:isp1:0:0:3): 0 CCBs oustanding (0 allocated, 0 freed) > (ctl9:isp1:0:0:3): 0 CTIOs outstanding (0 sent, 0 returned > CTL Frontend information end >=20 > zpool status is showing no errors. >=20 > P.S.: > The machine is 8 core 2.0Ghz Xeon E5-2650 with 196G of RAM > Other things to note are that I'm running with "hw.mfi.max_cmds=3D254" > Also my zpool is on GELI hw encrypted disks: 24 JBOD/RAID0 drives on = mfi, each encrypted with GELI, and arranged in 3 raidz2 vdevs of 8 = drives. > And the machine also acts as a fairly loaded NFS server. >=20 After a whole day of orion tests without problems, we started an Oracle = ASM instance from the Linux host and again got an error, this time it was WRITE error : (0:3:0:3): WRITE(10). CDB: 2a 0 1 5b 10 0 0 4 0 0=20 (0:3:0:3): Tag: 0x110940, Type: 1 (0:3:0:3): CTL Status: SCSI Error (0:3:0:3): SCSI Status: Check Condition (0:3:0:3): SCSI sense: NOT READY asc:4b,0 (Data phase error) I've tried to track down this "Data phase error" in the CTL code and it = looks like it is something related to the isp(4) driver: default: /* * XXX KDM the isp(4) driver doesn't = really * seem to send errors back for data * transfers that I can tell. There is = one * case where it'll send = CAM_REQ_CMP_ERR, * but probably not that many more = cases. * So set a generic data phase error = here, * like the SXP driver sets. */ io->io_hdr.port_status =3D 0xbad1; ctl_set_data_phase_error(&io->scsiio); /* * XXX KDM figure out residual. */ break; But I can't find error statistics for the isp(4) driver anywhere. Any = ideas how this can be debugged?=