From owner-freebsd-stable@FreeBSD.ORG Thu Sep 27 15:33:30 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 90C87106566B for ; Thu, 27 Sep 2012 15:33:30 +0000 (UTC) (envelope-from ndenev@gmail.com) Received: from mail-we0-f182.google.com (mail-we0-f182.google.com [74.125.82.182]) by mx1.freebsd.org (Postfix) with ESMTP id 0B07F8FC12 for ; Thu, 27 Sep 2012 15:33:29 +0000 (UTC) Received: by weyx43 with SMTP id x43so836561wey.13 for ; Thu, 27 Sep 2012 08:33:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=from:content-type:date:subject:to:message-id:mime-version:x-mailer; bh=ymFFwpUtFgj14QMl++zhW8TwieH0zCPncfHqs2+xClk=; b=YJGu9pj3MtHW7Qbp8Ff9j8XT8LmzxZobGvrmuU6ES9zZC7UxCsy8M0W/AdbB4UcOr9 4K+DE5HBrO5LdTz7duFOe875GCf1n+tPeeprCVYsPR8TQJgFCqszKwBls07Y+0f5kBWK i5PcWJ8TC1TvsM3MWVMiV3bFs7PP6SEQ1JrX4m4gufmICTHZU9Jj+M/bHfLRrj5WhDXo qyx+aFtZ8pUEyMQfFBxjX/zYcvgwkowGYEf9ADm2+GXQ1XLOu+iCp9wJuA7TBeTpxoNt O0qGEdDGfUdbqBzmLuUTIB8tBZ57ufh8GwweadDggskQBPhTGdF0P2fWcM5WhCbjRb7Z mFBg== Received: by 10.180.86.133 with SMTP id p5mr9171277wiz.17.1348760008669; Thu, 27 Sep 2012 08:33:28 -0700 (PDT) Received: from ndenevsa.sf.moneybookers.net (g1.moneybookers.com. [217.18.249.148]) by mx.google.com with ESMTPS id cl8sm31751211wib.10.2012.09.27.08.33.26 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 27 Sep 2012 08:33:27 -0700 (PDT) From: Nikolay Denev Date: Thu, 27 Sep 2012 18:33:25 +0300 To: "freebsd-stable@freebsd.org" Message-Id: <72A4B763-D36B-4912-8C20-7373A0562EA1@gmail.com> Mime-Version: 1.0 (Mac OS X Mail 6.1 \(1498\)) X-Mailer: Apple Mail (2.1498) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Subject: CAM Target Layer and Linux (continued) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 27 Sep 2012 15:33:30 -0000 Hi All, With the help of Chuck Tuffli, I'm now able to use CTL to export a zvol = over FC to a Linux host: LUN Backend Size (Blocks) BS Serial Number Device ID =20 0 block 4185915392 512 FBSDZFS001 ORA_ASM_01 =20 lun_type=3D0 num_threads=3D14 file=3D/dev/zvol/tank/oracle_asm_01 1 block 4185915392 512 FBSDZFS002 ORA_ASM_02 =20 lun_type=3D0 num_threads=3D14 file=3D/dev/zvol/tank/oracle_asm_02 2 block 4185915392 512 FBSDZFS003 ORA_ASM_03 =20 lun_type=3D0 num_threads=3D14 file=3D/dev/zvol/tank/oracle_asm_03 3 block 4185915392 512 FBSDZFS004 ORA_ASM_04 =20 lun_type=3D0 num_threads=3D14 file=3D/dev/zvol/tank/oracle_asm_04 Then we ran some tests using Oracle's ORION benchmark tool from the = Linux host. We ran one test which passed successfully, then I've just disabled zfs prefetch -> "vfs.zfs.prefetch_disable=3D1" and rerun the test, which failed due to this error. On the FreeBSD side: (0:3:0:1): READ(10). CDB: 28 0 84 f9 58 0 0 4 0 0=20 (0:3:0:1): Tag: 0x116220, Type: 1 (0:3:0:1): CTL Status: SCSI Error (0:3:0:1): SCSI Status: Check Condition (0:3:0:1): SCSI sense: NOT READY asc:4b,0 (Data phase error) Linux reported : sd 4:0:0:1: Device not ready: <6>: Current: sense key: Not Ready Add. Sense: Data phase error end_request: I/O error, dev sdr, sector 2230933504 device-mapper: multipath: Failing path 65:16. sd 4:0:0:1: Device not ready: <6>: Current: sense key: Not Ready Add. Sense: Data phase error end_request: I/O error, dev sdr, sector 2230934528 There are no other suspicious messages in dmesg. Also, ctladm dumpooa does not show anything. Here is dumpscructs output : CTL IID to WWPN map start: CTL IID to WWPN map end CTL Persistent Reservation information start: CTL Persistent Reservation information end CTL Frontends: Frontend CTL ioctl Type 4 pport 0 vport 0 WWNN 0 WWPN 0 Frontend ctl2cam Type 8 pport 0 vport 0 WWNN 0x5000000995680700 WWPN = 0x5000000995680702 Frontend CTL internal Type 8 pport 0 vport 0 WWNN 0 WWPN 0 Frontend isp0 Type 1 pport 0 vport 0 WWNN 0x20000024ff376b98 WWPN = 0x21000024ff376b98 isp0: max tagged openings: 4096, max dev openings: 4096 isp0: max_ccbs: 20488, ccb_count: 79 isp0: ccb_freeq is NOT empty isp0: alloc_queue.entries 0, alloc_openings 4096 isp0: qfrozen_cnt:0:0:0:0:0 (ctl2:isp0:0:0:0): 0 requests total waiting for CCBs (ctl2:isp0:0:0:0): 0 CCBs oustanding (17788811 allocated, 17788811 = freed) (ctl2:isp0:0:0:0): 0 CTIOs outstanding (17788811 sent, 17788811 returned (ctl4:isp0:0:0:1): 0 requests total waiting for CCBs (ctl4:isp0:0:0:1): 0 CCBs oustanding (16708305 allocated, 16708305 = freed) (ctl4:isp0:0:0:1): 0 CTIOs outstanding (16708305 sent, 16708305 returned (ctl6:isp0:0:0:2): 0 requests total waiting for CCBs (ctl6:isp0:0:0:2): 0 CCBs oustanding (16712865 allocated, 16712865 = freed) (ctl6:isp0:0:0:2): 0 CTIOs outstanding (16712865 sent, 16712865 returned (ctl8:isp0:0:0:3): 0 requests total waiting for CCBs (ctl8:isp0:0:0:3): 0 CCBs oustanding (16699727 allocated, 16699727 = freed) (ctl8:isp0:0:0:3): 0 CTIOs outstanding (16699727 sent, 16699727 returned isp1: max tagged openings: 4096, max dev openings: 4096 isp1: max_ccbs: 20488, ccb_count: 1 isp1: ccb_freeq is NOT empty isp1: alloc_queue.entries 0, alloc_openings 4096 isp1: qfrozen_cnt:0:0:0:0:0 (ctl3:isp1:0:0:0): 0 requests total waiting for CCBs (ctl3:isp1:0:0:0): 0 CCBs oustanding (0 allocated, 0 freed) (ctl3:isp1:0:0:0): 0 CTIOs outstanding (0 sent, 0 returned (ctl5:isp1:0:0:1): 0 requests total waiting for CCBs (ctl5:isp1:0:0:1): 0 CCBs oustanding (0 allocated, 0 freed) (ctl5:isp1:0:0:1): 0 CTIOs outstanding (0 sent, 0 returned (ctl7:isp1:0:0:2): 0 requests total waiting for CCBs (ctl7:isp1:0:0:2): 0 CCBs oustanding (0 allocated, 0 freed) (ctl7:isp1:0:0:2): 0 CTIOs outstanding (0 sent, 0 returned (ctl9:isp1:0:0:3): 0 requests total waiting for CCBs (ctl9:isp1:0:0:3): 0 CCBs oustanding (0 allocated, 0 freed) (ctl9:isp1:0:0:3): 0 CTIOs outstanding (0 sent, 0 returned Frontend isp1 Type 1 pport 1 vport 0 WWNN 0x20000024ff376b99 WWPN = 0x21000024ff376b99 isp0: max tagged openings: 4096, max dev openings: 4096 isp0: max_ccbs: 20488, ccb_count: 79 isp0: ccb_freeq is NOT empty isp0: alloc_queue.entries 0, alloc_openings 4096 isp0: qfrozen_cnt:0:0:0:0:0 (ctl2:isp0:0:0:0): 0 requests total waiting for CCBs (ctl2:isp0:0:0:0): 0 CCBs oustanding (17788811 allocated, 17788811 = freed) (ctl2:isp0:0:0:0): 0 CTIOs outstanding (17788811 sent, 17788811 returned (ctl4:isp0:0:0:1): 0 requests total waiting for CCBs (ctl4:isp0:0:0:1): 0 CCBs oustanding (16708305 allocated, 16708305 = freed) (ctl4:isp0:0:0:1): 0 CTIOs outstanding (16708305 sent, 16708305 returned (ctl6:isp0:0:0:2): 0 requests total waiting for CCBs (ctl6:isp0:0:0:2): 0 CCBs oustanding (16712865 allocated, 16712865 = freed) (ctl6:isp0:0:0:2): 0 CTIOs outstanding (16712865 sent, 16712865 returned (ctl8:isp0:0:0:3): 0 requests total waiting for CCBs (ctl8:isp0:0:0:3): 0 CCBs oustanding (16699727 allocated, 16699727 = freed) (ctl8:isp0:0:0:3): 0 CTIOs outstanding (16699727 sent, 16699727 returned isp1: max tagged openings: 4096, max dev openings: 4096 isp1: max_ccbs: 20488, ccb_count: 1 isp1: ccb_freeq is NOT empty isp1: alloc_queue.entries 0, alloc_openings 4096 isp1: qfrozen_cnt:0:0:0:0:0 (ctl3:isp1:0:0:0): 0 requests total waiting for CCBs (ctl3:isp1:0:0:0): 0 CCBs oustanding (0 allocated, 0 freed) (ctl3:isp1:0:0:0): 0 CTIOs outstanding (0 sent, 0 returned (ctl5:isp1:0:0:1): 0 requests total waiting for CCBs (ctl5:isp1:0:0:1): 0 CCBs oustanding (0 allocated, 0 freed) (ctl5:isp1:0:0:1): 0 CTIOs outstanding (0 sent, 0 returned (ctl7:isp1:0:0:2): 0 requests total waiting for CCBs (ctl7:isp1:0:0:2): 0 CCBs oustanding (0 allocated, 0 freed) (ctl7:isp1:0:0:2): 0 CTIOs outstanding (0 sent, 0 returned (ctl9:isp1:0:0:3): 0 requests total waiting for CCBs (ctl9:isp1:0:0:3): 0 CCBs oustanding (0 allocated, 0 freed) (ctl9:isp1:0:0:3): 0 CTIOs outstanding (0 sent, 0 returned CTL Frontend information end zpool status is showing no errors. P.S.: The machine is 8 core 2.0Ghz Xeon E5-2650 with 196G of RAM Other things to note are that I'm running with "hw.mfi.max_cmds=3D254" Also my zpool is on GELI hw encrypted disks: 24 JBOD/RAID0 drives on = mfi, each encrypted with GELI, and arranged in 3 raidz2 vdevs of 8 = drives. And the machine also acts as a fairly loaded NFS server.