From owner-freebsd-fs@FreeBSD.ORG Thu Mar 5 17:10:15 2015 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTPS id 169EA67E; Thu, 5 Mar 2015 17:10:15 +0000 (UTC) Received: from gate.pik.ru (gate.pik.ru [IPv6:2a03:5a00:31:40::25]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C49D1838; Thu, 5 Mar 2015 17:10:14 +0000 (UTC) Received: from [internal] by relay.pik.ru (Postfix) with ESMTP id 52A7810800; Thu, 5 Mar 2015 20:10:02 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=hotplug.ru; s=mx; t=1425575403; bh=yK5uethunIEmolVnEEq6+BZngV3gkJKlbzRI6G3b5Vk=; h=Date:From:To:CC:Subject; b=TyHtmyCLp9mnoZFnfmm+tX87+TBfA1pDRG5HGxvV29yOIK/LDUos6KKglYDxWj/Lh EkHJdsI/XeYEXAnmmYdNo14KPhBRlmec/avjy/USIXwQNysFRwFnNTQMUWTiyNx1Gi imI+EUGqXxsHHaORx76VKGmvZ06S4rq2Q0v15gvM= Message-ID: <54F88DEA.2070301@hotplug.ru> Date: Thu, 05 Mar 2015 20:10:02 +0300 From: Emil Muratov User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.4.0 MIME-Version: 1.0 To: freebsd-fs@freebsd.org Subject: CAM Target over FC and UNMAP problem Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Cc: Alexander Motin X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.18-1 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 05 Mar 2015 17:10:15 -0000 I've got an issue with CTL UNMAP and zvol backends. Seems that UNMAP from the initiator passed to the underlying disks (without trim support) causes IO blocking to the whole pool. Not sure where to address this problem. My setup: - plain SATA 7.2 krpm drives attached to Adaptec aacraid SAS controller - zfs raidz pool over plain drives, no partitioning - zvol created with volmode=dev - Qlogic ISP 2532 FC HBA in target mode - FreeBSD 10.1-STABLE #1 r279593 Create a new LUN with a zvol backend ctladm realsync off ctladm port -o on -p 5 ctladm create -b block -o file=/dev/zvol/wd/tst1 -o unmap=on -l 0 -d wd.tst1 -S tst1 Both target an initiator hosts connected to the FC fabric. Initiator is Win2012 server, actually it is a VM with RDM LUN to the guest OS. Formating, reading and writing large amounts of data (file copy/IOmeter) - so far so good. But as soon as I've tried to delete large files all IO to the LUN blocks, initiator system just iowaits. gstat on target shows that underlying disk load bumped to 100%, queue up to 10, but no iowrites actually in progress, only decent amount of ioreads. After a minute or so IO unblocks for a second or two than blocks again and so on again until all UNMAPs are done, it could take up to 5 minutes to delete 10Gb file. I can see that 'logicalused' property of a zvol shows that the deleted space was actually released. System log is filled with CTL msgs: kernel: (ctl2:isp1:0:0:3): ctlfestart: aborted command 0x12aaf4 discarded kernel: (2:5:3/3): WRITE(10). CDB: 2a 00 2f d4 74 b8 00 00 08 00 kernel: (2:5:3/3): Tag: 0x12ab24, type 1 kernel: (2:5:3/3): ctl_process_done: 96 seconds kernel: (ctl2:isp1:0:0:3): ctlfestart: aborted command 0x12afa4 discarded kernel: (ctl2:isp1:0:0:3): ctlfestart: aborted command 0x12afd4 discarded kernel: ctlfedone: got XPT_IMMEDIATE_NOTIFY status 0x36 tag 0xffffffff seq 0x121104 kernel: (ctl2:isp1:0:0:3): ctlfe_done: returning task I/O tag 0xffffffff seq 0x1210d4 I've tried to tackle some sysctls, but no success so far. vfs.zfs.vdev.bio_flush_disable: 1 vfs.zfs.vdev.bio_delete_disable: 1 vfs.zfs.trim.enabled=0 Disabling UNMAP in CTL (-o unmap=off) resolves the issue completely but than there is no space reclamation for zvol. Any hints would be appreciated.