From owner-freebsd-cloud@freebsd.org Sat Nov 25 13:22:34 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6C14DDE3939 for ; Sat, 25 Nov 2017 13:22:34 +0000 (UTC) (envelope-from raf@rafal.net) Received: from fbo-1.mxes.net (mxout-23.mxes.net [216.86.168.223]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48A3380CCD for ; Sat, 25 Nov 2017 13:22:33 +0000 (UTC) (envelope-from raf@rafal.net) Received: from smtp-out-2.mxes.net (smtp-out-2.mxes.net [67.222.241.249]) by fbi-1.mxes.net (Postfix) with ESMTP id 17CAB274F8 for ; Sat, 25 Nov 2017 08:15:39 -0500 (EST) Received: from [172.20.10.2] (s147-130.psd.vodafone.ie [213.233.147.130]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 4DA0F274FB for ; Sat, 25 Nov 2017 08:15:14 -0500 (EST) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable From: Rafal Lukawiecki Mime-Version: 1.0 (1.0) Date: Sat, 25 Nov 2017 13:24:17 +0100 Subject: FreeBSD AWS AMI disk system Message-Id: <386518E3-1D01-4E1F-BB77-E9C530E05381@rafal.net> To: freebsd-cloud@freebsd.org X-Mailer: iPad Mail (15B202) X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Nov 2017 13:22:34 -0000 Pardon my ignorance, as a recent convert to running FreeBSD in AWS, but I wo= uld like to know a little more about the possibilities that regard the under= lying file system of the official RE and Marketplace FreeBSD 11 AMIs. Ideally, I would like to have the simplicity of using the official AMIs but h= ave the option of selecting a different initial volume size (both smaller, 4= -5GB, and larger), and to turn on the standard EBS encryption at the initial= instance launch time. I realise that I can create my own AMIs, as I have do= ne over the years using CentOS. Is there an easier way and one that would no= t require my own AMI rebuilding from my own snapshots each time FreeBSD is r= eleased? Then there is the question of the actual file system. Have you opinions abou= t any performance gains, especially startup/reboot time, for OpenZFS via EB= S? The usual ZFS advantages of versioning/ZFS snapshots and the ability to s= tream updates seem attractive to our way of running our (growing) server far= m. I was just reading the current issue of the FreeBSD Magazine and I have f= ound out that ScaleEngine use OpenZFS in their AWS set-up. As an aside, the EFS is pretty slow=E2=80=94I suppose I had higher expectati= ons, considering EBS performance. I do not think that has much to do with Fre= eBSD and more likely a limitation of AWS NFS, but I wonder if there is anyth= ing on the horizon that could improve it. On another note, will the >32 bit N= FS log spam disappear anytime soon, ie. when is FreeBSD likely to get 64-bit= handles? Many thanks for your thoughts and my regards from Ireland, Rafal Lukawiecki Data Scientist Project Botticelli Ltd From owner-freebsd-cloud@freebsd.org Sat Nov 25 23:59:43 2017 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B152DDF4034 for ; Sat, 25 Nov 2017 23:59:43 +0000 (UTC) (envelope-from 0100015ff59bd0cd-42713e64-05b2-42e6-a964-ccaafb1d2a28-000000@amazonses.com) Received: from a8-176.smtp-out.amazonses.com (a8-176.smtp-out.amazonses.com [54.240.8.176]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7876F74984 for ; Sat, 25 Nov 2017 23:59:42 +0000 (UTC) (envelope-from 0100015ff59bd0cd-42713e64-05b2-42e6-a964-ccaafb1d2a28-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ae7m2yrxjw65l2cqdpjxuucyrvy564tn; d=tarsnap.com; t=1511654151; h=Subject:To:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; bh=6mvuPVRTRm/i2m+mhtliwDTCrguXP2ctUiKis5AYULY=; b=lyat5AsR6wAilUzv9IA0XVUE+cRcHAoZa+XJabgrxDOUMWB1cb112Say8gWkG6ki HlQA8Q5/Biz9GU0A7l8HulF6ii6MeCP4G+aNe0n0SVEi9uEtd7+/5lewK5fEXIJTKKb osDg93IPgVZXnVabsd6dlEJ44F0ku9zowsbGMq50= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=224i4yxa5dv7c2xz3womw6peuasteono; d=amazonses.com; t=1511654150; h=Subject:To:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:Feedback-ID; bh=6mvuPVRTRm/i2m+mhtliwDTCrguXP2ctUiKis5AYULY=; b=kwNehNi1H6qY2ybDN6zkier79KuzKwlcCCUjZyaM3eZfPTD7mv+NL97tXFjJYWWS rtMxwYUdZ3SiAGdjHa0YTk5IFVosRSl7vWqJxxQtjt77prQtJ3nuQy27mFXpLRsUoQ5 bDIqm98D9a6wcCJvHuhUg0uSQ+7/5QAg8uGL2ung= Subject: Re: FreeBSD AWS AMI disk system To: Rafal Lukawiecki , freebsd-cloud@freebsd.org References: <386518E3-1D01-4E1F-BB77-E9C530E05381@rafal.net> From: Colin Percival Message-ID: <0100015ff59bd0cd-42713e64-05b2-42e6-a964-ccaafb1d2a28-000000@email.amazonses.com> Date: Sat, 25 Nov 2017 23:55:50 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: <386518E3-1D01-4E1F-BB77-E9C530E05381@rafal.net> Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 8bit X-SES-Outgoing: 2017.11.25-54.240.8.176 Feedback-ID: 1.us-east-1.Lv9FVjaNvvR5llaqfLoOVbo2VxOELl7cjN0AOyXnPlk=:AmazonSES X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Nov 2017 23:59:43 -0000 On 11/25/17 04:24, Rafal Lukawiecki wrote: > Ideally, I would like to have the simplicity of using the official AMIs but > have the option of selecting a different initial volume size (both smaller, > 4-5GB, and larger) This is something you can already do. The "default" volume size is 10 GB, but the actual disk image is 4 GB. (I think. It was 2 GB but has increased over the years.) When FreeBSD first boots in EC2, it automatically grows its root filesystem to fill the disk it's launched with. One minor issue: At one point there was a bug in the way that Marketplace images were handled which prevented launching with a disk smaller than 10 GB; I don't know if this has been fixed yet. But you can still launch with larger disks. > and to turn on the standard EBS encryption at the > initial instance launch time. Unless things have changed in how EBS works, this isn't possible. When you launch an EC2 instance, the newly created EBS volume isn't fully initialized yet -- that would slow things down far too much. Instead, EBS keeps track of which blocks haven't been initialized and loads data from the backing snapshot as needed. But if a disk is "encrypted", EBS will expect to load encrypted blocks from the snapshot stored in S3 -- AFAIK it can't handle having some of the backing blocks being encrypted and some of them being unencrypted. So (again, unless things have changed) if you want an encrypted volume you're going to have to launch it from an encrypted snapshot, meaning that you'll have to copy the AMI. > Then there is the question of the actual file system. Have you opinions > about any performance gains, especially startup/reboot time, for OpenZFS > via EBS? The usual ZFS advantages of versioning/ZFS snapshots and the > ability to stream updates seem attractive to our way of running our > (growing) server farm. I was just reading the current issue of the FreeBSD > Magazine and I have found out that ScaleEngine use OpenZFS in their AWS > set-up. I haven't measured performance for UFS vs. ZFS; I suspect that any differences will be insignificant given how fast the disks are. So far I've stuck with UFS images mainly because I expect that people who use ZFS will probably want to create a ZFS pool out of additional volumes they attach -- not out of the single 10 GB (by default) disk which has the base OS. > As an aside, the EFS is pretty slow—I suppose I had higher expectations, > considering EBS performance. I do not think that has much to do with > FreeBSD and more likely a limitation of AWS NFS, but I wonder if there is > anything on the horizon that could improve it. Assuming you're not running into the EFS I/O throttling, I suspect you're just seeing "NFS is slower than local filesystems". > On another note, will the > >32 bit NFS log spam disappear anytime soon, ie. when is FreeBSD likely to > get 64-bit handles? That's fixed in HEAD, but won't be MFCed because 64-bit inodes break all sorts of interfaces. So, 12.0 (aimed for early 2019, I believe). -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid