From owner-freebsd-cloud@freebsd.org Wed May 11 19:47:54 2016 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 052EDB373F4 for ; Wed, 11 May 2016 19:47:54 +0000 (UTC) (envelope-from bhoward@digitalocean.com) Received: from mail-vk0-x230.google.com (mail-vk0-x230.google.com [IPv6:2607:f8b0:400c:c05::230]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id BDB6910FD for ; Wed, 11 May 2016 19:47:53 +0000 (UTC) (envelope-from bhoward@digitalocean.com) Received: by mail-vk0-x230.google.com with SMTP id f66so70994422vkh.2 for ; Wed, 11 May 2016 12:47:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=digitalocean.com; s=google; h=mime-version:date:message-id:subject:from:to; bh=x8IuZFRcQcd0hpwvMsdyHhy1UdEzKXzulVmEuTtFQuo=; b=TxsxLTUWb4s/BjWYP53OYwb7H91q7xaF0pIT2iezqLam7Af79v93+TFTkrjdO6eFxJ htBl61ufpTrRQpDzt32wqIJ2blSiS/1vhOUpiZ4l24A2ojpXNPEaWgjrNb4hi0TeJPFR 7NZixiX+IWxNljwRcrWp1PHiD7LCn7EiQBGhs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:date:message-id:subject:from:to; bh=x8IuZFRcQcd0hpwvMsdyHhy1UdEzKXzulVmEuTtFQuo=; b=KcPA+qySkV3LtySkPdonPne/O8z0KzzGCP3uh34ZFfvLff3nfuYCeGxDQNr0d/QyQw h+KZbF0ljA7n0A3T3V1tzVm2OP9vMes65n+6xtpw2+UTVRmsrvRZu1+CtT/uhRlkxl1z cJNG41TwKgEhVlbHfE+peI+JzunfY4TAVH4MDom2DXDuEHd5kLz0tMXlB4iJzY4jg4ib 0nlnqLiAUS1xgVUYkebLF5Rfe6Td//OsXduBVVeBl0uszepBKRlMuvwcI5Do0BwnUrm7 X2h/xQ1qvwa9mNIn0wlfXItxTPWRo3meqIE2N1Buk8BXHHEqlGzGSmnJjbCBqIQCEFTD FJ4w== X-Gm-Message-State: AOPr4FXKFkhpc2naBUBLzT1wJFq+UiI1tdWNQudxLse1qTk+QzhU/VdL70TCctUkbPljB5hOxc6IjiktWTXjbtay MIME-Version: 1.0 X-Received: by 10.31.99.133 with SMTP id x127mr2303845vkb.146.1462996072762; Wed, 11 May 2016 12:47:52 -0700 (PDT) Received: by 10.31.154.13 with HTTP; Wed, 11 May 2016 12:47:52 -0700 (PDT) Date: Wed, 11 May 2016 13:47:52 -0600 Message-ID: Subject: Help with a performance issue From: Ben Howard To: freebsd-cloud@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 May 2016 19:47:54 -0000 Howdy, I work for DigitalOcean and am responsible for our base images. Currently we offer FreeBSD, and unfortunately, it has some rough edges. If anyone would like to help to make it better, please drop me an email. Thanks, Ben From owner-freebsd-cloud@freebsd.org Wed May 11 19:51:23 2016 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0F157B37595 for ; Wed, 11 May 2016 19:51:23 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) by mx1.freebsd.org (Postfix) with ESMTP id E363A13D6 for ; Wed, 11 May 2016 19:51:22 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (unknown [10.1.1.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id 32DA1D60F for ; Wed, 11 May 2016 19:51:16 +0000 (UTC) Subject: Re: Help with a performance issue To: freebsd-cloud@freebsd.org References: From: Allan Jude Message-ID: <57338D30.3000909@freebsd.org> Date: Wed, 11 May 2016 15:51:12 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="tNIjK06OI2r63ED5STOVcsmMw8iFhpLJj" X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 May 2016 19:51:23 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --tNIjK06OI2r63ED5STOVcsmMw8iFhpLJj Content-Type: multipart/mixed; boundary="LJWGIoENPO5brSXx12ROQRr0k9HTIbEhH" From: Allan Jude To: freebsd-cloud@freebsd.org Message-ID: <57338D30.3000909@freebsd.org> Subject: Re: Help with a performance issue References: In-Reply-To: --LJWGIoENPO5brSXx12ROQRr0k9HTIbEhH Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 2016-05-11 15:47, Ben Howard via freebsd-cloud wrote: > Howdy, >=20 > I work for DigitalOcean and am responsible for our base images. Current= ly > we offer FreeBSD, and unfortunately, it has some rough edges. >=20 > If anyone would like to help to make it better, please drop me an email= =2E >=20 > Thanks, > Ben > _______________________________________________ > freebsd-cloud@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-cloud > To unsubscribe, send any mail to "freebsd-cloud-unsubscribe@freebsd.org= " >=20 What are the problems you are having? --=20 Allan Jude --LJWGIoENPO5brSXx12ROQRr0k9HTIbEhH-- --tNIjK06OI2r63ED5STOVcsmMw8iFhpLJj Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.22 (MingW32) iQIcBAEBAgAGBQJXM40zAAoJEBmVNT4SmAt+Sl8P/jf1vPfp7wcN+BGjCmfv9jrz 6aKLb8kOQjGA72NN3gtXIZk9EUwvrLnhkyBJ4VEzLx6W9UsHb0iRuKke+Ygu69H2 0z02i/g65DkInU+QyTyMK7ccUVDx67WE1t+Omzm41vcJKQxfusQ84JKkla6T4fDl azHYbXBiVyusncnoZ73v2msGXyFRN754LNxL7IvnaJR09wr4rajnYxnamwY3LhjC +NdAD5rPqZauOCxeLTnIrAHzXU+bNId4/OnyAvS4I75ndOQtJWTRao2gaVxgML/u JthFIXa3FuAerblEwDFfGGdv7ymiIp7i0tvpM1A3jj4GFIhzZoZeHOMjYFEjD/5t 2Sq5ZhEA6UJE9yyHjQ5ngmje6Up6yAwYUDj9umJlpa3fz1uSCIG/ulRIwO2Xde/f QVUFkjTtxNdUHngqqcHWQ9/2h3M2gv4gc2oq+dNboQ8OzkSIScqYNqMlOacwoZF7 y6gtmBgSeaeiHf6C2a5O8GnhKLH8N2veb1FYVJUFsqTtnoDv9/828GbKtTDka7hT CpTa/kehEyWsaAZC1SDjfl57JpaGfLtWZ6UJ+a6w29/mEcJg7bb3wEXDt6eqDusb RnqV8o4clQh4arh2224k29pzSOF012Pqn0YdshOmW5ypd1kqX6+Ka8uKQ7ReueHT /7OVti6egaK2MuvMl9ar =xkSi -----END PGP SIGNATURE----- --tNIjK06OI2r63ED5STOVcsmMw8iFhpLJj-- From owner-freebsd-cloud@freebsd.org Wed May 11 19:53:30 2016 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 82E28B3767B for ; Wed, 11 May 2016 19:53:30 +0000 (UTC) (envelope-from jungleboogie0@gmail.com) Received: from mail-io0-x236.google.com (mail-io0-x236.google.com [IPv6:2607:f8b0:4001:c06::236]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5030116C3 for ; Wed, 11 May 2016 19:53:30 +0000 (UTC) (envelope-from jungleboogie0@gmail.com) Received: by mail-io0-x236.google.com with SMTP id d62so68463592iof.2 for ; Wed, 11 May 2016 12:53:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=xd9rqCxVZs5gMhmiAOA7vhHTHcz2wJDB8dOLnj8c980=; b=YIkwEBwV1kOHhbUgc90LlD26Z/K/1j6YNQeulvNJh1MjRDuOtcVwUOpoa/Kc6GOWDC 2xMYZCm9rz2brBbnT0ktx5Spus23kK+3I2oDm83lWjLvmOlQ2JvsURYI9QCmHL4Tt+mp aep0Teo2fBc3h6Of+7BC15C4skW9ga7gPvz4HiYwHXqwuEaRgA2/q4zDNSQ6CCfIk7Zb npontfa+JkWjeEOqKILj02AiQB84P0wtW7IjMQ4q323Bp3YH+Zt1qollexoBt5y7sfEV qrJiyuhcS+qdaOpGokY99fRQZBU56v7tDjlmzyykAu4LMS9bs6b4TO2in3elxZaNJJjI u9WA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=xd9rqCxVZs5gMhmiAOA7vhHTHcz2wJDB8dOLnj8c980=; b=lY4Pa84x6nsfO1qYgXxuwXNEmyBkOekUQ1r99IOk+4ASJ/bEdA3igSoQvo0zDn55i/ NOH9ePvcEXqSv2DOA4ksWduM94p+qWi0TmtZCmBYHCzNIl2RN6+ixEamQ1s6PRdf270Z TctEXgZNZNeHmuCrVVVSmozVg3O+EdXbkhgYos0kCzDBGpB2iyhBBW+GoWVF2sesrbC1 4jHCGPM3wCxb5l4DShtW26qpTCHRX9+51z65mAcW+tQJh2wFt/uhUfAD8xuHxj38LgxI Sigb4AMqZV3hoe2+wAnoUs/APqz3AO/RjOGVsZli0inhSkhayczlaJzgepSNYMTM6sjc aDIA== X-Gm-Message-State: AOPr4FXS5n6v3qCelY3f7nQDI9X1McrCWTDi/D/lYLl2tlHWJgmJA8NiABp6zMasH2UjYUJqhGXj/MP6bILgSg== MIME-Version: 1.0 X-Received: by 10.36.149.137 with SMTP id m131mr6499623itd.94.1462996409764; Wed, 11 May 2016 12:53:29 -0700 (PDT) Received: by 10.107.142.21 with HTTP; Wed, 11 May 2016 12:53:29 -0700 (PDT) In-Reply-To: References: Date: Wed, 11 May 2016 12:53:29 -0700 Message-ID: Subject: Re: Help with a performance issue From: jungle Boogie To: Ben Howard Cc: freebsd-cloud@freebsd.org Content-Type: text/plain; charset=UTF-8 X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 May 2016 19:53:30 -0000 On 11 May 2016 at 12:47, Ben Howard via freebsd-cloud wrote: > If anyone would like to help to make it better, please drop me an email. What is it you want to improve? -- ------- inum: 883510009027723 sip: jungleboogie@sip2sip.info xmpp: jungle-boogie@jit.si From owner-freebsd-cloud@freebsd.org Wed May 11 20:22:08 2016 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C6C01B370B6 for ; Wed, 11 May 2016 20:22:08 +0000 (UTC) (envelope-from bhoward@digitalocean.com) Received: from mail-vk0-x231.google.com (mail-vk0-x231.google.com [IPv6:2607:f8b0:400c:c05::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 91B831E50 for ; Wed, 11 May 2016 20:22:08 +0000 (UTC) (envelope-from bhoward@digitalocean.com) Received: by mail-vk0-x231.google.com with SMTP id o133so72187529vka.0 for ; Wed, 11 May 2016 13:22:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=digitalocean.com; s=google; h=mime-version:in-reply-to:references:date:message-id:subject:from:to; bh=/2DiDbV8odDT53sH2NuBlSZvcSUHr8Avszc5r4CxMfA=; b=SfLqeW1MqRUlEE4tO2x2k/tgG35l0hGJmG/5Mg7bbqjq4JOum6rUu2palJRQWS6JL5 yNvTaFgYiqSSrJaCw2Ps+OJfY1hSv+wD8q6GkZrm4BG6jzAO8Cy6O3CGWv1JW2Kt+vS9 Q6kO3jBn13M/R3FSah8F80/eKUPF0++8j2HRA= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to; bh=/2DiDbV8odDT53sH2NuBlSZvcSUHr8Avszc5r4CxMfA=; b=T/jrjbZXNl46ACeh+ciCg4XsEgFbeG97qEs+ddy13Qtb7ge+6yFTISBcMvQXgZEoij vvw5rJOWv+RehFQzrCdUENtXdGr1FKwPzpWwlH+Ra8kz4ZkqCJHQxgGkiPj8riitg/Lv RSEX4xInM/MimX3vgrp7W5qJB9lo+sljv0AuQFSZGn/cDzKaEovkAbvGrOtBKquPzcop mv4xHu3ASeHfcPfM0EIswVylEaDd3Ya9j270X70bmAbTTc4bsQCmoPQuZW1N7+LrEbkM CmSntH7QnJKRXvK9k9rELU3PzlrJ3xJnHWO4POycd3UXUDdHL8vwnO92omuiz7AU8dAW 0Q9w== X-Gm-Message-State: AOPr4FXlEkOFMZM8w1L3AWrnx1IlFV0GRUkjdqL0PTPJ45m0XjsD7vdO+/8bRxmdGQgCKAJYZ0UN41RE6wavNxRs MIME-Version: 1.0 X-Received: by 10.31.69.7 with SMTP id s7mr2806305vka.47.1462998127592; Wed, 11 May 2016 13:22:07 -0700 (PDT) Received: by 10.31.154.13 with HTTP; Wed, 11 May 2016 13:22:07 -0700 (PDT) In-Reply-To: References: Date: Wed, 11 May 2016 14:22:07 -0600 Message-ID: Subject: Re: Help with a performance issue From: Ben Howard To: freebsd-cloud@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.22 X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 May 2016 20:22:08 -0000 The short story is that we are seeing profound differences between UFS and ZFS in terms of network to disk writes. # FreeBSD with ZFS $ fetch http://speedtest.bahnhof.se/1000M.zip 1000M.zip 27% of 1000 MB 11 MBps 01m00s # Stock FreeBSD in NYC3 with UFS: $ fetch http://speedtest.bahnhof.se/1000M.zip 1000M.zip 1% of 1000 MB 529 kBps 32m12s The test case is flaky, unfortunately. Switching the backing filesystem, IMHO, should not illicit such a massive difference. Any ideas? Thanks, Ben On Wed, May 11, 2016 at 1:47 PM, Ben Howard wrote: > Howdy, > > I work for DigitalOcean and am responsible for our base images. Currently > we offer FreeBSD, and unfortunately, it has some rough edges. > > If anyone would like to help to make it better, please drop me an email. > > Thanks, > Ben > From owner-freebsd-cloud@freebsd.org Wed May 11 21:10:54 2016 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 704BEB37F4C for ; Wed, 11 May 2016 21:10:54 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from mx1.scaleengine.net (mx1.scaleengine.net [209.51.186.6]) by mx1.freebsd.org (Postfix) with ESMTP id 35C65104C for ; Wed, 11 May 2016 21:10:53 +0000 (UTC) (envelope-from allanjude@freebsd.org) Received: from [10.1.1.2] (unknown [10.1.1.2]) (Authenticated sender: allanjude.freebsd@scaleengine.com) by mx1.scaleengine.net (Postfix) with ESMTPSA id EAD26D831 for ; Wed, 11 May 2016 21:10:52 +0000 (UTC) Subject: Re: Help with a performance issue To: freebsd-cloud@freebsd.org References: From: Allan Jude Message-ID: <57339FDC.4090608@freebsd.org> Date: Wed, 11 May 2016 17:10:52 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Thunderbird/38.7.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.22 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 May 2016 21:10:54 -0000 On 2016-05-11 16:22, Ben Howard via freebsd-cloud wrote: > The short story is that we are seeing profound differences between UFS and > ZFS in terms of network to disk writes. > > # FreeBSD with ZFS > $ fetch http://speedtest.bahnhof.se/1000M.zip > 1000M.zip 27% of 1000 MB 11 MBps 01m00s > > # Stock FreeBSD in NYC3 with UFS: > $ fetch http://speedtest.bahnhof.se/1000M.zip > 1000M.zip 1% of 1000 MB 529 kBps 32m12s > > The test case is flaky, unfortunately. Switching the backing filesystem, > IMHO, should not illicit such a massive difference. > > Any ideas? > > Thanks, > Ben Part of this will be the way that ZFS works. In the case of an asynchronous write like downloading a file, it will buffer the writes to ram, and flush them to disk in one contiguous block every 5 seconds (unless the memory buffer gets full, or some other operation causes a sync) Using a slightly more predictable test: # dd if=/dev/zero of=zerofile bs=1m count=1k 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 97.438103 secs (11019732 bytes/sec) I get about the same performance on UFS as you were getting on ZFS. However, running 'gstat' (GEOM stat, GEOM is the FreeBSD storage abstraction layer), shows very high write latency, sometimes operations taking as much as 1000ms to complete. L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 11 7 0 0 0.0 7 884 1041 107.1| vtbd0 During the write I saw speeds as high as 25MB/s, but often much less than that. There also seem to be stalls where nothing happens for a second. Interestingly, I actually saw steadier performance when UFS was mounted with the 'sync' option, which I would have expected to decrease performance a great deal. The average latency went from 50-100ms to under 2ms: L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 0 587 0 0 0.0 587 18719 1.8 99.6| vtbd0 1 1551 1 32 0.2 1550 49564 0.5 74.5| vtbd0 When I did the test again... # dd if=/dev/zero of=zerofile3 bs=1m count=1k 1024+0 records in 1024+0 records out 1073741824 bytes transferred in 45.618443 secs (23537450 bytes/sec) It was faster. However, when mounted with 'sync', the amount of writes measured in gstat if twice as high as the amount generated by userland. Using mbuffer to rate-limit writes to 10mb/s, a 'sync' mounted / showed 20mb/s of writes to the drive. dd if=/dev/zero bs=1m count=1k | mbuffer -o zerofile -s 128k -b 64 -R 10m in @ 10.0 MiB/s, out @ 10.0 MiB/s, 243 MiB total, buffer 100% fulll L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 0 722 0 0 0.0 722 23061 0.6 41.8| vtbd0 Switching back to 'noasync' (the default): Suddenly the speed is very fast. in @ 123 MiB/s, out @ 123 MiB/s, 3446 MiB total, buffer 98% full L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name 3 1049 0 0 0.0 1049 133702 5.4 95.9| vtbd0 Very strange. I'll do some more tests with a fresh droplet, this was from an existing one. I wonder if it has to do with lazy allocation? I know ZFS developers have seen this issue on Amazon, where if they full the drive with 0's before its first use, performance is much better afterwards. -- Allan Jude