From owner-freebsd-fs@freebsd.org Wed Oct 4 16:15:30 2017 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 112A2E3CF76 for ; Wed, 4 Oct 2017 16:15:30 +0000 (UTC) (envelope-from javocado@gmail.com) Received: from mail-ua0-x236.google.com (mail-ua0-x236.google.com [IPv6:2607:f8b0:400c:c08::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BE29A68156 for ; Wed, 4 Oct 2017 16:15:29 +0000 (UTC) (envelope-from javocado@gmail.com) Received: by mail-ua0-x236.google.com with SMTP id 47so7169049uas.8 for ; Wed, 04 Oct 2017 09:15:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=VoR66OkqVQzR1dkDl5fIUJwKp7m+sxA4QL4wSYgBvM0=; b=dObOOk5fD14ov7FG8zlvOzsBxBkX2kwDxlnCwSTuNnuz3U1bkd3fj+LsEvaVaWU7i7 wzM2nUFiJLtCQlwqbCeUMtq3SNh2c68eHljzEYLG9fpavKpeZtFtxKN9caVSR66F5CuP AmswVFJtu/PoQq+R6mtk9JIRIqYRilWSvOwElfjnIt2eVZU/4AsCVmrGvRPcIVUj7hJ+ BA2sbUd79dfasw2uUszD00JFAhHkhXasu0iOLE2Yb4kkekVwITbluPCk4s9MePCngO80 5DtkcP7bINSaCA46+fa0Zsf0YI3UmOPDcWnW0icONUUNmd9fyjfXlMi2Ao4ejFqhZ3pY EZjQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=VoR66OkqVQzR1dkDl5fIUJwKp7m+sxA4QL4wSYgBvM0=; b=E8rxNMhHWBkyLi2/HEZ1s4RvM1pcdHsOUHLADx9ki3z/wdTKkTBTAVqYTHvcCmeAm1 jp0Q9+mh2YmUTo9eOzFf7lUwxL66pdzd/neaDjyTnnu+x2Ec+rq46VHfgCEOeZX3lwm4 zAJLKHl+zL8XtgmltpQxtXaUnf6MpXjNPH22HtQhLyynSRLLthC68YDzW/SaTpbMBkIk 5WrPFl+3fSVeqfotwT8dWU8fH4bOwiMuTVRd4jBLExPskTqy/5DT939exCmVvX5oOuD9 fiAnnDaWIL0UOBgKWrOz1kaqUaIq5/JLqCpQHwLt3KkNtDC0GB/6a4u8L6oZk9neftPW 1OnA== X-Gm-Message-State: AHPjjUi04sxpG2un+QTxRgETWqkMTGg+vlWs9z8ZRWjlR8peA/Wjb39t 031zKGufP7mMNRMZlL7YlvdwzXcqHCeEKuZfgYC7mA== X-Google-Smtp-Source: AOwi7QBkhSedf6YzySREy/pdCERdPYWe0nCWa4C92eaX/qollyGz6dKqa0v1TedoEmvvwrRQuWK4epNlCL7AD6eljgM= X-Received: by 10.159.36.74 with SMTP id 68mr10888560uaq.67.1507133728327; Wed, 04 Oct 2017 09:15:28 -0700 (PDT) MIME-Version: 1.0 Received: by 10.159.51.90 with HTTP; Wed, 4 Oct 2017 09:15:27 -0700 (PDT) From: javocado Date: Wed, 4 Oct 2017 09:15:27 -0700 Message-ID: Subject: lockup during zfs destroy To: FreeBSD Filesystems Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Oct 2017 16:15:30 -0000 I am trying to destroy a dense, large filesystem and it's not going well. Details: - zpool is a raidz3 with 3 x 12 drive vdevs. - target filesystem to be destroyed is ~2T with ~63M inodes. - OS: FreeBSD 10.3amd with 192 GB of RAM. - 120 GB of swap (90GB recently added as swap-on-disk) What happened initially is the system locked after a few hours up and I had to reboot. Upon rebooting and starting zfs, I see sustained disk activity in gstat *and* that the sustained activity is usually just 6 disks reading. Two raidz3 vdevs are involved in this filesystem I am deleting so there are 6 parity disks ... not sure if that is correlated or not. At about the 1h40m mark of uptime I see things start to happen in top: a sudden spike in load, and drop in the amount of "Free" memory as reported in top: ([CODE]Mem: 23M Active, 32M Inact, 28G Wired, 24M Buf, 159G Free[/CODE]) It drops down under a GB and then fluctuates up and down till eventually it reaches some small amount (41 MB). As this drop starts, I see gstat activity on zpool drives cease, and there's some light activity on the swap devices, but not much. Also, the amount of swap used is reported as very little, maybe less than a MB to 24 MB. swapinfo shows nothing used. After the memory usage settles the system eventually ends up in a locked state where: - nothing is going on in gstat; the only non-zero number is the queue length for the swap device which is stuck at 4 - load drops to nothing, and occasionally I see the zfskern and zpool procs stuck in vmwait state*. - shell is unresponsive, but carriage returns register - there are NO kernel/messages of any kind on console indicating a problem or resource exhaustion Finally, I cannot do this: # zdb -dddd pool/filesystem | grep DELETE_QUEUE zdb: can't open 'pool/filesystem': Device busy (presumably because it is pending destroy ...) I had set: vm.kmem_size="384G" (and nothing else in loader) but even removing that and setting more realistic figures like: vm.kmem_size=200862670848 vm.kmem_size_max=200862670848 vfs.zfs.arc_max=187904819200 have not resulted in a different outcome, *though I don't see the processes in vmwait any longer, the state is just "-" I've just lowered these to: vm.kmem_size=198642237440 vm.kmem_size_max=198642237440 vfs.zfs.arc_max=190052302848 to see if that will make a difference. No matter how many times I reboot, so far about 6, I never make it past the 1h40m mark and this memory dip. I don't know if I'm making any progress or just running into the same wall. My questions: - is this what it appears to be, a memory exhaustion? - if so, why isn't swap utilized? - how would I configure my way past this hurdle? - a filesystem has a DELETE_QUEUE ... does the zpool itself have a destroy queue of some kind? I am trying to see if I can see the zpool working and how far along it is, but I do not know what to query with zdb Thanks!