From owner-freebsd-cloud@freebsd.org Mon Jan 1 16:17:00 2018 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ECB8AEB0BFB for ; Mon, 1 Jan 2018 16:17:00 +0000 (UTC) (envelope-from raf@rafal.net) Received: from smtp-out-1.mxes.net (smtp-out-1.mxes.net [67.222.241.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C2E21720E0 for ; Mon, 1 Jan 2018 16:17:00 +0000 (UTC) (envelope-from raf@rafal.net) Received: from [192.168.40.41] (86-40-118-125-dynamic.agg2.bri.bbh-prp.eircom.net [86.40.118.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id D277427543; Mon, 1 Jan 2018 11:16:46 -0500 (EST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: AMI building AMI From: Rafal Lukawiecki In-Reply-To: <01000160a0283ad5-12b330d3-3f4b-43ea-9cdc-2524f5f42f4a-000000@email.amazonses.com> Date: Mon, 1 Jan 2018 16:16:44 +0000 Cc: freebsd-cloud@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <085D225B-2E99-4540-B693-223087B14A27@rafal.net> <0100016099578ea7-02da1d5c-997d-47be-9cd8-dc6b75bbeda6-000000@email.amazonses.com> <8BCB67B8-DE56-491A-8E6A-95AFB83F71D6@rafal.net> <0100016099671dff-32d986bf-3979-40a3-accf-4d17c2020403-000000@email.amazonses.com> <8538BFB9-B82C-4520-ADB3-E25040EA1B30@rafal.net> <01000160a0283ad5-12b330d3-3f4b-43ea-9cdc-2524f5f42f4a-000000@email.amazonses.com> To: Colin Percival X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Jan 2018 16:17:01 -0000 > On 29 Dec 2017, at 02:44, Colin Percival wrote: >=20 > us-east-1 ami-e6a6ea9c is a FreeBSD 11.1-RELEASE AMI-building AMI. Many thanks, Colin. And a Happy New Year! May I ask you if what I am hoping to do makes sense? Taking the gist = from = http://www.daemonology.net/blog/2015-11-21-FreeBSD-AMI-builder-AMI.html = I plan to make an AMI, install packages, patch and install the kernel, = all in one go. I am unsure how to =E2=80=9Cmake kernel=E2=80=9D so that = it ends up in the new AMI, rather than just in the builder AMI. Firs, I create an IAM role with a policy which grants access to the EC2 = CreateImage API. Then I launch your AMI building AMI ami-e6a6ea9c in = us-east-1 using config init script as follows, on a sufficiently large = c5: #!/bin/sh export ASSUME_ALWAYS_YES=3DYES PKGS_TO_INSTALL=3Dlots of packages go here pkg -r /mnt fetch -d $PKGS_TO_INSTALL pkg -c /mnt install $PKGS_TO_INSTALL svnlite co https://svn.freebsd.org/base/releng/11.1/ /usr/src/ cd /usr/src cat > /tmp/kernel.patch < Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9FF5AEB4893 for ; Mon, 1 Jan 2018 17:43:18 +0000 (UTC) (envelope-from raf@rafal.net) Received: from fbo-2.mxes.net (mxout-22.mxes.net [216.86.168.222]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 76A98756DE for ; Mon, 1 Jan 2018 17:43:17 +0000 (UTC) (envelope-from raf@rafal.net) Received: from smtp-out-1.mxes.net (smtp-out-1.mxes.net [67.222.241.250]) by fbi-2.mxes.net (Postfix) with ESMTP id BA3E02753E for ; Mon, 1 Jan 2018 12:43:09 -0500 (EST) Received: from [192.168.40.41] (86-40-118-125-dynamic.agg2.bri.bbh-prp.eircom.net [86.40.118.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 5295527553; Mon, 1 Jan 2018 12:42:46 -0500 (EST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: AMI building AMI From: Rafal Lukawiecki In-Reply-To: <01000160a0283ad5-12b330d3-3f4b-43ea-9cdc-2524f5f42f4a-000000@email.amazonses.com> Date: Mon, 1 Jan 2018 17:42:44 +0000 Cc: freebsd-cloud@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <0C1308D4-AFD6-4216-83A6-4687157EC8D3@rafal.net> References: <085D225B-2E99-4540-B693-223087B14A27@rafal.net> <0100016099578ea7-02da1d5c-997d-47be-9cd8-dc6b75bbeda6-000000@email.amazonses.com> <8BCB67B8-DE56-491A-8E6A-95AFB83F71D6@rafal.net> <0100016099671dff-32d986bf-3979-40a3-accf-4d17c2020403-000000@email.amazonses.com> <8538BFB9-B82C-4520-ADB3-E25040EA1B30@rafal.net> <01000160a0283ad5-12b330d3-3f4b-43ea-9cdc-2524f5f42f4a-000000@email.amazonses.com> To: Colin Percival X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Jan 2018 17:43:18 -0000 PS. Could it be as easy as make DESTDIR=3D/mnt installkernel in the = config init? > On 29 Dec 2017, at 02:44, Colin Percival wrote: >=20 > us-east-1 ami-e6a6ea9c is a FreeBSD 11.1-RELEASE AMI-building AMI. Many thanks, Colin. And a Happy New Year! May I ask you if what I am hoping to do makes sense? Taking the gist = from = http://www.daemonology.net/blog/2015-11-21-FreeBSD-AMI-builder-AMI.html = I plan to make an AMI, install packages, patch and install the kernel, = all in one go. I am unsure how to =E2=80=9Cmake kernel=E2=80=9D so that = it ends up in the new AMI, rather than just in the builder AMI. Firs, I create an IAM role with a policy which grants access to the EC2 = CreateImage API. Then I launch your AMI building AMI ami-e6a6ea9c in = us-east-1 using config init script as follows, on a sufficiently large = c5: #!/bin/sh export ASSUME_ALWAYS_YES=3DYES PKGS_TO_INSTALL=3Dlots of packages go here pkg -r /mnt fetch -d $PKGS_TO_INSTALL pkg -c /mnt install $PKGS_TO_INSTALL svnlite co https://svn.freebsd.org/base/releng/11.1/ /usr/src/ cd /usr/src cat > /tmp/kernel.patch < Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2F25EEB7953 for ; Mon, 1 Jan 2018 19:00:13 +0000 (UTC) (envelope-from raf@rafal.net) Received: from smtp-out-1.mxes.net (smtp-out-1.mxes.net [67.222.241.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 07DE77829E for ; Mon, 1 Jan 2018 19:00:12 +0000 (UTC) (envelope-from raf@rafal.net) Received: from [192.168.40.41] (86-40-118-125-dynamic.agg2.bri.bbh-prp.eircom.net [86.40.118.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 921D527552; Mon, 1 Jan 2018 14:00:10 -0500 (EST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: AMI building AMI From: Rafal Lukawiecki In-Reply-To: <0C1308D4-AFD6-4216-83A6-4687157EC8D3@rafal.net> Date: Mon, 1 Jan 2018 19:00:08 +0000 Cc: freebsd-cloud@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <085D225B-2E99-4540-B693-223087B14A27@rafal.net> <0100016099578ea7-02da1d5c-997d-47be-9cd8-dc6b75bbeda6-000000@email.amazonses.com> <8BCB67B8-DE56-491A-8E6A-95AFB83F71D6@rafal.net> <0100016099671dff-32d986bf-3979-40a3-accf-4d17c2020403-000000@email.amazonses.com> <8538BFB9-B82C-4520-ADB3-E25040EA1B30@rafal.net> <01000160a0283ad5-12b330d3-3f4b-43ea-9cdc-2524f5f42f4a-000000@email.amazonses.com> <0C1308D4-AFD6-4216-83A6-4687157EC8D3@rafal.net> To: Colin Percival X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Jan 2018 19:00:13 -0000 > On 1 Jan 2018, at 17:42, Rafal Lukawiecki wrote: >=20 > PS. Could it be as easy as make DESTDIR=3D/mnt installkernel in the = config init? I went ahead and tried the script. It is not working yet, and I need to = debug it further, which means connecting to the instance etc, which I = have not yet. So far a few observations are: - AMI builder does not like c5.4xlarge, it aborts when trying to install = the base FreeBSD image. It works with c4.4xlarge. - svnlite needed extra params to avoid hanging on keyboard input - pkg -c throws error about "no address record unable to update = repository=E2=80=9D, I am trying to force an update of the repository - kernel compiles, but I am not yet sure if it installed correctly - mkami failed to create an AMI, unsure yet why, need to rerun with more = debug output. My current script is: #!/bin/sh set -x export ASSUME_ALWAYS_YES=3DYES PKGS_TO_INSTALL=3D"awscli cli53 lots of other ones" pkg -r /mnt fetch -d $PKGS_TO_INSTALL pkg -r /mnt update -f pkg -c /mnt install $PKGS_TO_INSTALL svnlite --non-interactive --trust-server-cert-failures=3Dunknown-ca co = https://svn.freebsd.org/base/releng/11.1/ /usr/src/ cd /usr/src cat > /tmp/kernel.patch < Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8588EE87FF5 for ; Mon, 1 Jan 2018 22:46:10 +0000 (UTC) (envelope-from 01000160b3e11382-967ccc76-185e-434e-a28e-57741781aaaa-000000@amazonses.com) Received: from a8-237.smtp-out.amazonses.com (a8-237.smtp-out.amazonses.com [54.240.8.237]) (using TLSv1 with cipher ECDHE-RSA-AES128-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4814722CB for ; Mon, 1 Jan 2018 22:46:09 +0000 (UTC) (envelope-from 01000160b3e11382-967ccc76-185e-434e-a28e-57741781aaaa-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=ae7m2yrxjw65l2cqdpjxuucyrvy564tn; d=tarsnap.com; t=1514846360; h=Subject:To:References:Cc:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; bh=vkBOQUiCp9DqW/5YmsweSYMLPcDWyMKQu7UNKbo9LG4=; b=jhuMCf/Tln+0eWTM5cnuMDxlEK5STJ86j99MxLzMvemDKiamlxK2gzxLpXqNO/SG A31QEog4IdC6Jj3CANR+5wwffaja4mS857GH/0Jh3vaebOE6qWA5u9uMtzXDL/RDYOn sZc+f/UYeIIRGUl2Vcmsnovm/78g96OVSLRwg6jQ= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=224i4yxa5dv7c2xz3womw6peuasteono; d=amazonses.com; t=1514846360; h=Subject:To:References:Cc:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:Feedback-ID; bh=vkBOQUiCp9DqW/5YmsweSYMLPcDWyMKQu7UNKbo9LG4=; b=MRx/c8Q0D/c0LeWRxbdSOXYfsDIOMGGjL2yOnn+sw+6PmbyMeutqQDRteAnvKz0V qmV+oBlJCOEji3uSfJRP0XrVVOfq88Sne46EwD80mSN6JkHaqKGaXDQlvk2MoBh7K8v 5us5hP8qA7cZoa2g/9f8BGTu7XjPenNhhOKlFh7g= Subject: Re: AMI building AMI To: Rafal Lukawiecki References: <085D225B-2E99-4540-B693-223087B14A27@rafal.net> <0100016099578ea7-02da1d5c-997d-47be-9cd8-dc6b75bbeda6-000000@email.amazonses.com> <8BCB67B8-DE56-491A-8E6A-95AFB83F71D6@rafal.net> <0100016099671dff-32d986bf-3979-40a3-accf-4d17c2020403-000000@email.amazonses.com> <8538BFB9-B82C-4520-ADB3-E25040EA1B30@rafal.net> <01000160a0283ad5-12b330d3-3f4b-43ea-9cdc-2524f5f42f4a-000000@email.amazonses.com> <0C1308D4-AFD6-4216-83A6-4687157EC8D3@rafal.net> Cc: freebsd-cloud@freebsd.org From: Colin Percival Message-ID: <01000160b3e11382-967ccc76-185e-434e-a28e-57741781aaaa-000000@email.amazonses.com> Date: Mon, 1 Jan 2018 22:39:20 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:45.0) Gecko/20100101 Thunderbird/45.5.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-SES-Outgoing: 2018.01.01-54.240.8.237 Feedback-ID: 1.us-east-1.Lv9FVjaNvvR5llaqfLoOVbo2VxOELl7cjN0AOyXnPlk=:AmazonSES X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Jan 2018 22:46:10 -0000 On 01/01/18 11:00, Rafal Lukawiecki wrote: > On 1 Jan 2018, at 17:42, Rafal Lukawiecki wrote: >> PS. Could it be as easy as make DESTDIR=/mnt installkernel in the config init? That is indeed how to install the kernel you built into the right place. > I went ahead and tried the script. [...] Maybe a dumb question, but do you really need to use a configinit script for this? I know I showed that as an example in my blog post, but I expected that the main way the AMI builder would be used would be by SSHing in and setting things up "manually". -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid From owner-freebsd-cloud@freebsd.org Mon Jan 1 23:54:05 2018 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E935DEA321B for ; Mon, 1 Jan 2018 23:54:05 +0000 (UTC) (envelope-from raf@rafal.net) Received: from smtp-out-1.mxes.net (smtp-out-1.mxes.net [67.222.241.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C107F63C8E for ; Mon, 1 Jan 2018 23:54:05 +0000 (UTC) (envelope-from raf@rafal.net) Received: from [192.168.40.23] (86-40-118-125-dynamic.agg2.bri.bbh-prp.eircom.net [86.40.118.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 9F1F62754C; Mon, 1 Jan 2018 18:54:02 -0500 (EST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (1.0) Subject: Re: AMI building AMI From: Rafal Lukawiecki X-Mailer: iPad Mail (15C153) In-Reply-To: <01000160b3e11756-1d74d2fc-6486-4458-87cd-b88203a99a77-000000@email.amazonses.com> Date: Mon, 1 Jan 2018 23:54:01 +0000 Cc: freebsd-cloud@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <1F125915-3331-4F00-97E4-B6763DFCEBA4@rafal.net> References: <085D225B-2E99-4540-B693-223087B14A27@rafal.net> <0100016099578ea7-02da1d5c-997d-47be-9cd8-dc6b75bbeda6-000000@email.amazonses.com> <8BCB67B8-DE56-491A-8E6A-95AFB83F71D6@rafal.net> <0100016099671dff-32d986bf-3979-40a3-accf-4d17c2020403-000000@email.amazonses.com> <8538BFB9-B82C-4520-ADB3-E25040EA1B30@rafal.net> <01000160a0283ad5-12b330d3-3f4b-43ea-9cdc-2524f5f42f4a-000000@email.amazonses.com> <0C1308D4-AFD6-4216-83A6-4687157EC8D3@rafal.net> <01000160b3e11756-1d74d2fc-6486-4458-87cd-b88203a99a77-000000@email.amazonses.com> To: Colin Percival X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 01 Jan 2018 23:54:06 -0000 > On 1 Jan 2018, at 22:39, Colin Percival wrote: >=20 > Maybe a dumb question, but do you really need to use a configinit script > for this? I know I showed that as an example in my blog post, but I > expected that the main way the AMI builder would be used would be by > SSHing in and setting things up "manually". Automation is attractive but not not a primary goal this time. Being able to= kick it off from an EC2 launch template and come back to a finished AMI, ma= kes it more likely I=E2=80=99d have it done the next few times the AMI needs= remaking. I suppose I will need to do it manually the first time to see where the issu= es are. I will share the feedback with you in case it is useful for a future= post. Many thanks, Rafal= From owner-freebsd-cloud@freebsd.org Tue Jan 2 03:05:43 2018 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E3F0EAC98E for ; Tue, 2 Jan 2018 03:05:43 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 3CD226AC82 for ; Tue, 2 Jan 2018 03:05:42 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (203-206-51-137.dyn.iinet.net.au [203.206.51.137] (may be forged)) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id w0235abQ027498 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Mon, 1 Jan 2018 19:05:39 -0800 (PST) (envelope-from julian@freebsd.org) Subject: Re: AMI building AMI To: Rafal Lukawiecki , Colin Percival Cc: freebsd-cloud@freebsd.org References: <085D225B-2E99-4540-B693-223087B14A27@rafal.net> <0100016099578ea7-02da1d5c-997d-47be-9cd8-dc6b75bbeda6-000000@email.amazonses.com> <8BCB67B8-DE56-491A-8E6A-95AFB83F71D6@rafal.net> <0100016099671dff-32d986bf-3979-40a3-accf-4d17c2020403-000000@email.amazonses.com> <8538BFB9-B82C-4520-ADB3-E25040EA1B30@rafal.net> <01000160a0283ad5-12b330d3-3f4b-43ea-9cdc-2524f5f42f4a-000000@email.amazonses.com> From: Julian Elischer Message-ID: <004116c5-0938-5570-f119-fdc21e91b2ad@freebsd.org> Date: Tue, 2 Jan 2018 11:05:30 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jan 2018 03:05:43 -0000 On 2/1/18 12:16 am, Rafal Lukawiecki wrote: >> On 29 Dec 2017, at 02:44, Colin Percival wrote: >> >> us-east-1 ami-e6a6ea9c is a FreeBSD 11.1-RELEASE AMI-building AMI. > Many thanks, Colin. And a Happy New Year! > > May I ask you if what I am hoping to do makes sense? Taking the gist from http://www.daemonology.net/blog/2015-11-21-FreeBSD-AMI-builder-AMI.html I plan to make an AMI, install packages, patch and install the kernel, all in one go. I am unsure how to “make kernel” so that it ends up in the new AMI, rather than just in the builder AMI. > > Firs, I create an IAM role with a policy which grants access to the EC2 CreateImage API. Then I launch your AMI building AMI ami-e6a6ea9c in us-east-1 using config init script as follows, on a sufficiently large c5: > > #!/bin/sh > export ASSUME_ALWAYS_YES=YES > PKGS_TO_INSTALL=lots of packages go here > pkg -r /mnt fetch -d $PKGS_TO_INSTALL > pkg -c /mnt install $PKGS_TO_INSTALL please explain to me the  use of -r AND -c? if you use -r to populate a chroot for -c then should that be pkg add rater than pkg install? > svnlite co https://svn.freebsd.org/base/releng/11.1/ /usr/src/ > cd /usr/src > cat > /tmp/kernel.patch < Patch goes here > EOT > patch < /tmp/kernel.patch > make kernel -j16 > *** How to copy the new kernel into the AMI’s kernel on /mnt without messing anything up *** > mkami "FreeBSD 11.1 for PB” “PB Customised FreeBSD" > shutdown -p now > > Afterwards I copy my new AMI to eu-west-1. > > Does this make sense? Is there documentation to mkami? I could not find it. I am curious if I could parametrise it to create an encrypted volume snapshot of a given target size. > > Many, many thanks and my regards from Ireland, > Rafal > _______________________________________________ > freebsd-cloud@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-cloud > To unsubscribe, send any mail to "freebsd-cloud-unsubscribe@freebsd.org" > > From owner-freebsd-cloud@freebsd.org Tue Jan 2 19:19:58 2018 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AD0F2EB479F for ; Tue, 2 Jan 2018 19:19:58 +0000 (UTC) (envelope-from raf@rafal.net) Received: from smtp-out-1.mxes.net (smtp-out-1.mxes.net [67.222.241.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7C4A46E619; Tue, 2 Jan 2018 19:19:58 +0000 (UTC) (envelope-from raf@rafal.net) Received: from [192.168.40.37] (86-40-118-125-dynamic.agg2.bri.bbh-prp.eircom.net [86.40.118.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 5115727505; Tue, 2 Jan 2018 14:19:50 -0500 (EST) From: Rafal Lukawiecki Message-Id: <559A9E86-6E97-4A8D-96CC-95FC9180A483@rafal.net> Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: AMI building AMI Date: Tue, 2 Jan 2018 19:19:48 +0000 In-Reply-To: <004116c5-0938-5570-f119-fdc21e91b2ad@freebsd.org> Cc: Colin Percival , freebsd-cloud@freebsd.org To: Julian Elischer References: <085D225B-2E99-4540-B693-223087B14A27@rafal.net> <0100016099578ea7-02da1d5c-997d-47be-9cd8-dc6b75bbeda6-000000@email.amazonses.com> <8BCB67B8-DE56-491A-8E6A-95AFB83F71D6@rafal.net> <0100016099671dff-32d986bf-3979-40a3-accf-4d17c2020403-000000@email.amazonses.com> <8538BFB9-B82C-4520-ADB3-E25040EA1B30@rafal.net> <01000160a0283ad5-12b330d3-3f4b-43ea-9cdc-2524f5f42f4a-000000@email.amazonses.com> <004116c5-0938-5570-f119-fdc21e91b2ad@freebsd.org> X-Mailer: Apple Mail (2.3445.5.20) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jan 2018 19:19:58 -0000 > On 2 Jan 2018, at 03:05, Julian Elischer wrote: >> #!/bin/sh >> export ASSUME_ALWAYS_YES=3DYES >> PKGS_TO_INSTALL=3Dlots of packages go here >> pkg -r /mnt fetch -d $PKGS_TO_INSTALL >> pkg -c /mnt install $PKGS_TO_INSTALL > please explain to me the use of -r AND -c? > if you use -r to populate a chroot for -c then should that be pkg add rat= er than pkg install? Julian, I am afraid I have blindly followed the suggestion from http://www.= daemonology.net/blog/2015-11-21-FreeBSD-AMI-builder-AMI.html which suggested t= his script to preinstall the packages in the system mounted on /mnt #!/bin/sh export ASSUME_ALWAYS_YES=3DYES pkg -r /mnt fetch -d apache24 pkg -c /mnt install apache24 echo apache24_enable=3DYES >> /mnt/etc/rc.conf mkami "FreeBSD 10.2 w/ Apache 2.4" "FreeBSD with Apache pre-installed" shutdown -p now Perhaps Colin might be able to chime in on the original intent? My goal is to have packages installed, together with the patched kernel, th= en to power off the machine and use it as a template for others, saving the = 3 minutes of the initial package installation process, in our case. I have t= ried your suggestion of =E2=80=9Cadd=E2=80=9D but it did not work, either. I= wonder if this somehow related to inability of pkg -c to resolve names in t= his AMI. When I try this manually, not using the above scripts, these are the result= s and errors I get: $ pkg -r /mnt update -f Updating FreeBSD repository catalogue... pkg: Repository FreeBSD load error: access repo file(/mnt/var/db/pkg/repo-F= reeBSD.sqlite) failed: No such file or directory Fetching meta.txz: 100% 944 B 0.9kB/s 00:01 Fetching packagesite.txz: 100% 6 MiB 6.1MB/s 00:01 Processing entries: 100% FreeBSD repository update completed. 26993 packages processed. All repositories are up to date. $ pkg -c /mnt install awscli Updating FreeBSD repository catalogue... pkg: http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/meta.txz: No address= record repository FreeBSD has no meta file, using default settings pkg: http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/packagesite.txz: No = address record Unable to update repository FreeBSD Error updating repositories! $ pkg -c /mnt add awscli pkg: awscli: No such file or directory pkg: Was 'pkg install awscli' meant? Failed to install the following 1 package(s): awscli Many thanks, everyone. Rafal From owner-freebsd-cloud@freebsd.org Tue Jan 2 19:30:42 2018 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D54CEEB4D98 for ; Tue, 2 Jan 2018 19:30:42 +0000 (UTC) (envelope-from raf@rafal.net) Received: from smtp-out-1.mxes.net (smtp-out-1.mxes.net [67.222.241.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8BF8E6EB78; Tue, 2 Jan 2018 19:30:42 +0000 (UTC) (envelope-from raf@rafal.net) Received: from [192.168.40.37] (86-40-118-125-dynamic.agg2.bri.bbh-prp.eircom.net [86.40.118.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 546A727505; Tue, 2 Jan 2018 14:30:39 -0500 (EST) From: Rafal Lukawiecki Message-Id: <1832FE6C-5A3E-448C-8B6A-81A49C3C952E@rafal.net> Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: AMI building AMI Date: Tue, 2 Jan 2018 19:30:37 +0000 In-Reply-To: <559A9E86-6E97-4A8D-96CC-95FC9180A483@rafal.net> Cc: Colin Percival , freebsd-cloud@freebsd.org To: Julian Elischer References: <085D225B-2E99-4540-B693-223087B14A27@rafal.net> <0100016099578ea7-02da1d5c-997d-47be-9cd8-dc6b75bbeda6-000000@email.amazonses.com> <8BCB67B8-DE56-491A-8E6A-95AFB83F71D6@rafal.net> <0100016099671dff-32d986bf-3979-40a3-accf-4d17c2020403-000000@email.amazonses.com> <8538BFB9-B82C-4520-ADB3-E25040EA1B30@rafal.net> <01000160a0283ad5-12b330d3-3f4b-43ea-9cdc-2524f5f42f4a-000000@email.amazonses.com> <004116c5-0938-5570-f119-fdc21e91b2ad@freebsd.org> <559A9E86-6E97-4A8D-96CC-95FC9180A483@rafal.net> X-Mailer: Apple Mail (2.3445.5.20) Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jan 2018 19:30:42 -0000 Ah, perhaps pkg -c simply cannot access /var/db/pkg/repo-FreeBSD.sqlite beca= use it is chrooted? > On 2 Jan 2018, at 19:19, Rafal Lukawiecki wrote: > > >> On 2 Jan 2018, at 03:05, Julian Elischer > wrote: >>> #!/bin/sh >>> export ASSUME_ALWAYS_YES=3DYES >>> PKGS_TO_INSTALL=3Dlots of packages go here >>> pkg -r /mnt fetch -d $PKGS_TO_INSTALL >>> pkg -c /mnt install $PKGS_TO_INSTALL >> please explain to me the use of -r AND -c? >> if you use -r to populate a chroot for -c then should that be pkg add ra= ter than pkg install? > > Julian, I am afraid I have blindly followed the suggestion from http://ww= w.daemonology.net/blog/2015-11-21-FreeBSD-AMI-builder-AMI.html which suggested= this script to preinstall the packages in the system mounted on /mnt > > #!/bin/sh > export ASSUME_ALWAYS_YES=3DYES > pkg -r /mnt fetch -d apache24 > pkg -c /mnt install apache24 > echo apache24_enable=3DYES >> /mnt/etc/rc.conf > mkami "FreeBSD 10.2 w/ Apache 2.4" "FreeBSD with Apache pre-installed" > shutdown -p now > > Perhaps Colin might be able to chime in on the original intent? > > My goal is to have packages installed, together with the patched kernel, = then to power off the machine and use it as a template for others, saving th= e 3 minutes of the initial package installation process, in our case. I have= tried your suggestion of =E2=80=9Cadd=E2=80=9D but it did not work, either.= I wonder if this somehow related to inability of pkg -c to resolve names in= this AMI. > > When I try this manually, not using the above scripts, these are the resu= lts and errors I get: > > $ pkg -r /mnt update -f > Updating FreeBSD repository catalogue... > pkg: Repository FreeBSD load error: access repo file(/mnt/var/db/pkg/repo= -FreeBSD.sqlite) failed: No such file or directory > Fetching meta.txz: 100% 944 B 0.9kB/s 00:01 > Fetching packagesite.txz: 100% 6 MiB 6.1MB/s 00:01 > Processing entries: 100% > FreeBSD repository update completed. 26993 packages processed. > All repositories are up to date. > > $ pkg -c /mnt install awscli > Updating FreeBSD repository catalogue... > pkg: http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/meta.txz: No address record > repository FreeBSD has no meta file, using default settings > pkg: http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/packagesite.txz: <= http://pkg.freebsd.org/FreeBSD:11:amd64/quarterly/packagesite.txz:> No addre= ss record > Unable to update repository FreeBSD > Error updating repositories! > > $ pkg -c /mnt add awscli > pkg: awscli: No such file or directory > pkg: Was 'pkg install awscli' meant? > Failed to install the following 1 package(s): awscli > > Many thanks, everyone. > Rafal From owner-freebsd-cloud@freebsd.org Tue Jan 2 19:35:18 2018 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1590EEB50BB for ; Tue, 2 Jan 2018 19:35:18 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from vps1.elischer.org (vps1.elischer.org [204.109.63.16]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vps1.elischer.org", Issuer "CA Cert Signing Authority" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E43606F10E for ; Tue, 2 Jan 2018 19:35:17 +0000 (UTC) (envelope-from julian@freebsd.org) Received: from Julian-MBP3.local (203-206-51-137.dyn.iinet.net.au [203.206.51.137] (may be forged)) (authenticated bits=0) by vps1.elischer.org (8.15.2/8.15.2) with ESMTPSA id w02JZAlt031432 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NO); Tue, 2 Jan 2018 11:35:13 -0800 (PST) (envelope-from julian@freebsd.org) Subject: Re: AMI building AMI To: Rafal Lukawiecki Cc: Colin Percival , freebsd-cloud@freebsd.org References: <085D225B-2E99-4540-B693-223087B14A27@rafal.net> <0100016099578ea7-02da1d5c-997d-47be-9cd8-dc6b75bbeda6-000000@email.amazonses.com> <8BCB67B8-DE56-491A-8E6A-95AFB83F71D6@rafal.net> <0100016099671dff-32d986bf-3979-40a3-accf-4d17c2020403-000000@email.amazonses.com> <8538BFB9-B82C-4520-ADB3-E25040EA1B30@rafal.net> <01000160a0283ad5-12b330d3-3f4b-43ea-9cdc-2524f5f42f4a-000000@email.amazonses.com> <004116c5-0938-5570-f119-fdc21e91b2ad@freebsd.org> <559A9E86-6E97-4A8D-96CC-95FC9180A483@rafal.net> From: Julian Elischer Message-ID: <50f75b01-2ea3-f7fd-24a6-65a4d64ae16d@freebsd.org> Date: Wed, 3 Jan 2018 03:35:01 +0800 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:52.0) Gecko/20100101 Thunderbird/52.5.2 MIME-Version: 1.0 In-Reply-To: <559A9E86-6E97-4A8D-96CC-95FC9180A483@rafal.net> Content-Language: en-US Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jan 2018 19:35:18 -0000 On 3/1/18 3:19 am, Rafal Lukawiecki wrote: > >> On 2 Jan 2018, at 03:05, Julian Elischer > > wrote: >>> #!/bin/sh >>> export ASSUME_ALWAYS_YES=YES >>> PKGS_TO_INSTALL=lots of packages go here >>> pkg -r /mnt fetch -d $PKGS_TO_INSTALL >>> pkg -c /mnt install $PKGS_TO_INSTALL >> please explain to me the use of -r AND -c? >> if you use -r to populate a chroot for -c then should that be pkg >> add rater than pkg install? > > Julian, I am afraid I have blindly followed the suggestion from > http://www.daemonology.net/blog/2015-11-21-FreeBSD-AMI-builder-AMI.html which > suggested this script to preinstall the packages in the system > mounted on /mnt > > #!/bin/sh > export ASSUME_ALWAYS_YES=YES > pkg -r /mnt fetch -d apache24 > pkg -c /mnt install apache24 > echo apache24_enable=YES >> /mnt/etc/rc.conf > mkami "FreeBSD 10.2 w/ Apache 2.4" "FreeBSD with Apache pre-installed" > shutdown -p now > > Perhaps Colin might be able to chime in on the original intent? > > My goal is to have packages installed, together with the patched > kernel, then to power off the machine and use it as a template for > others, saving the 3 minutes of the initial package installation > process, in our case. I have tried your suggestion of “add” but it > did not work, either. I wonder if this somehow related to inability > of pkg -c to resolve names in this AMI. > > When I try this manually, not using the above scripts, these are the > results and errors I get: > > $ pkg -r /mnt update -f > Updating FreeBSD repository catalogue... > pkg: Repository FreeBSD load error: access repo > file(/mnt/var/db/pkg/repo-FreeBSD.sqlite) failed: No such file > or directory > Fetching meta.txz: 100%    944 B   0.9kB/s    00:01 > Fetching packagesite.txz: 100%    6 MiB   6.1MB/s    00:01 > Processing entries: 100% > FreeBSD repository update completed. 26993 packages processed. > All repositories are up to date. > > $ pkg -c /mnt install awscli > Updating FreeBSD repository catalogue... > pkg: http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/meta.txz: No > address record > repository FreeBSD has no meta file, using default settings > pkg: > http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/packagesite.txz: > No address record > Unable to update repository FreeBSD > Error updating repositories! this means that either in the jail/chroot or in your machine, dns lookups are failing.. did you remember to copy /etc/resolv.conf into the chroot/jail? > > $ pkg -c /mnt add awscli > pkg: awscli: No such file or directory > pkg: Was 'pkg install awscli' meant? > Failed to install the following 1 package(s): awscli > > Many thanks, everyone. > Rafal pkg install fetches over the network, pkg add installs a prefetched file, so I am guessing that if you use add, and point the preloaded file it may work.. Of course pkg may be smart enough to find the cached copy itself, but it still seems to want to look at the network. From owner-freebsd-cloud@freebsd.org Tue Jan 2 19:41:16 2018 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4A7FDEB53A1 for ; Tue, 2 Jan 2018 19:41:16 +0000 (UTC) (envelope-from raf@rafal.net) Received: from smtp-out-1.mxes.net (smtp-out-1.mxes.net [67.222.241.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 00AF26F40D; Tue, 2 Jan 2018 19:41:15 +0000 (UTC) (envelope-from raf@rafal.net) Received: from [192.168.40.37] (86-40-118-125-dynamic.agg2.bri.bbh-prp.eircom.net [86.40.118.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 536CE2755D; Tue, 2 Jan 2018 14:41:08 -0500 (EST) From: Rafal Lukawiecki Message-Id: <77A688A9-6272-4FC6-89DF-4F9231B7B3CE@rafal.net> Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: AMI building AMI Date: Tue, 2 Jan 2018 19:41:06 +0000 In-Reply-To: <50f75b01-2ea3-f7fd-24a6-65a4d64ae16d@freebsd.org> Cc: Colin Percival , freebsd-cloud@freebsd.org To: Julian Elischer References: <085D225B-2E99-4540-B693-223087B14A27@rafal.net> <0100016099578ea7-02da1d5c-997d-47be-9cd8-dc6b75bbeda6-000000@email.amazonses.com> <8BCB67B8-DE56-491A-8E6A-95AFB83F71D6@rafal.net> <0100016099671dff-32d986bf-3979-40a3-accf-4d17c2020403-000000@email.amazonses.com> <8538BFB9-B82C-4520-ADB3-E25040EA1B30@rafal.net> <01000160a0283ad5-12b330d3-3f4b-43ea-9cdc-2524f5f42f4a-000000@email.amazonses.com> <004116c5-0938-5570-f119-fdc21e91b2ad@freebsd.org> <559A9E86-6E97-4A8D-96CC-95FC9180A483@rafal.net> <50f75b01-2ea3-f7fd-24a6-65a4d64ae16d@freebsd.org> X-Mailer: Apple Mail (2.3445.5.20) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.25 X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jan 2018 19:41:16 -0000 > On 2 Jan 2018, at 19:35, Julian Elischer wrote: >> $ pkg -c /mnt install awscli >> Updating FreeBSD repository catalogue... >> pkg: http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/meta.txz: No address record >> repository FreeBSD has no meta file, using default settings >> pkg: http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly/packagesite.txz: = No addr= ess record >> Unable to update repository FreeBSD >> Error updating repositories! > > this means that either in the jail/chroot or in your machine, dns lookups= are failing.. did you remember to copy /etc/resolv.conf into the chroot/jai= l? Bingo. Copying it solved the problem. Many thanks, Julian! >> >> $ pkg -c /mnt add awscli >> pkg: awscli: No such file or directory >> pkg: Was 'pkg install awscli' meant? >> Failed to install the following 1 package(s): awscli >> >> Many thanks, everyone. >> Rafal > pkg install fetches over the network, pkg add installs a prefetched file,= > > so I am guessing that if you use add, and point the preloaded file it may= work.. Of course pkg may be smart enough to find the cached copy itself, bu= t it still seems to want to look at the network. > From owner-freebsd-cloud@freebsd.org Tue Jan 2 20:48:19 2018 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id AFA7FEB8BC0 for ; Tue, 2 Jan 2018 20:48:19 +0000 (UTC) (envelope-from raf@rafal.net) Received: from smtp-out-1.mxes.net (smtp-out-1.mxes.net [67.222.241.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 81D8672AB3; Tue, 2 Jan 2018 20:48:19 +0000 (UTC) (envelope-from raf@rafal.net) Received: from [192.168.40.37] (86-40-118-125-dynamic.agg2.bri.bbh-prp.eircom.net [86.40.118.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 5C6B227547; Tue, 2 Jan 2018 15:48:16 -0500 (EST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: AMI building AMI From: Rafal Lukawiecki In-Reply-To: <77A688A9-6272-4FC6-89DF-4F9231B7B3CE@rafal.net> Date: Tue, 2 Jan 2018 20:48:14 +0000 Cc: Colin Percival , freebsd-cloud@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: References: <085D225B-2E99-4540-B693-223087B14A27@rafal.net> <0100016099578ea7-02da1d5c-997d-47be-9cd8-dc6b75bbeda6-000000@email.amazonses.com> <8BCB67B8-DE56-491A-8E6A-95AFB83F71D6@rafal.net> <0100016099671dff-32d986bf-3979-40a3-accf-4d17c2020403-000000@email.amazonses.com> <8538BFB9-B82C-4520-ADB3-E25040EA1B30@rafal.net> <01000160a0283ad5-12b330d3-3f4b-43ea-9cdc-2524f5f42f4a-000000@email.amazonses.com> <004116c5-0938-5570-f119-fdc21e91b2ad@freebsd.org> <559A9E86-6E97-4A8D-96CC-95FC9180A483@rafal.net> <50f75b01-2ea3-f7fd-24a6-65a4d64ae16d@freebsd.org> <77A688A9-6272-4FC6-89DF-4F9231B7B3CE@rafal.net> To: Julian Elischer X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jan 2018 20:48:19 -0000 I have encountered another, hopefully minor issue running pkg in the = unattended script. When it installs a package (postfix-sasl) it expects = keyboard input to =E2=80=9CWould you like to active Postfix in = /usr/local/etc/mail/mailer.conf [n]?=E2=80=9D. I have tried both: export ASSUME_ALWAYS_YES=3DYES pkg -c /mnt -o ASSUME_ALWAYS_YES=3DYES install $PKGS_TO_INSTALL but the process still hangs waiting for input. Are there any other flags = or tricks I could pass to pkg, short of prefixing it with =E2=80=9Cyes n = |=E2=80=9D? Looking through man for pkg I do not see any options = designed for such unattended install cases. Many thanks for your help. Rafal From owner-freebsd-cloud@freebsd.org Tue Jan 2 21:17:53 2018 Return-Path: Delivered-To: freebsd-cloud@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65CBEEBA146 for ; Tue, 2 Jan 2018 21:17:53 +0000 (UTC) (envelope-from raf@rafal.net) Received: from smtp-out-1.mxes.net (smtp-out-1.mxes.net [67.222.241.250]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 380AA73E11; Tue, 2 Jan 2018 21:17:52 +0000 (UTC) (envelope-from raf@rafal.net) Received: from [192.168.40.37] (86-40-118-125-dynamic.agg2.bri.bbh-prp.eircom.net [86.40.118.125]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.mxes.net (Postfix) with ESMTPSA id 076A027542; Tue, 2 Jan 2018 16:17:49 -0500 (EST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: AMI building AMI From: Rafal Lukawiecki In-Reply-To: Date: Tue, 2 Jan 2018 21:17:48 +0000 Cc: freebsd-cloud@freebsd.org, Julian Elischer Content-Transfer-Encoding: quoted-printable Message-Id: References: <085D225B-2E99-4540-B693-223087B14A27@rafal.net> <0100016099578ea7-02da1d5c-997d-47be-9cd8-dc6b75bbeda6-000000@email.amazonses.com> <8BCB67B8-DE56-491A-8E6A-95AFB83F71D6@rafal.net> <0100016099671dff-32d986bf-3979-40a3-accf-4d17c2020403-000000@email.amazonses.com> <8538BFB9-B82C-4520-ADB3-E25040EA1B30@rafal.net> <01000160a0283ad5-12b330d3-3f4b-43ea-9cdc-2524f5f42f4a-000000@email.amazonses.com> <004116c5-0938-5570-f119-fdc21e91b2ad@freebsd.org> <559A9E86-6E97-4A8D-96CC-95FC9180A483@rafal.net> <50f75b01-2ea3-f7fd-24a6-65a4d64ae16d@freebsd.org> <77A688A9-6272-4FC6-89DF-4F9231B7B3CE@rafal.net> To: Colin Percival X-Mailer: Apple Mail (2.3445.5.20) X-BeenThere: freebsd-cloud@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "FreeBSD on cloud platforms \(EC2, GCE, Azure, etc.\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 02 Jan 2018 21:17:53 -0000 Colin, Everyone, Here is the final script that has managed to run, unattended! On a = c4.x4large it completes the task in 11 minutes, which is impressive, = considering the long list of packages (126). Following that, I am able = to encrypt the AMI while copying it across regions using: aws ec2 copy-image --source-region us-east-1 --region eu-west-1 = --source-image-id ami-x --encrypted --name =E2=80=9CMy FreeBSD" If you would like to update your blog post with this, now fully tested = script, perhaps others would benefit. Please note credit is due to = Julian, who got me unstuck regarding pkg -c and resolv.conf, many = thanks, Julian. #!/bin/sh set -x export ASSUME_ALWAYS_YES=3DYES PKGS_TO_INSTALL=3D=E2=80=9Clist of packages to install" pkg -r /mnt fetch -d $PKGS_TO_INSTALL pkg -r /mnt update -f mv /mnt/etc/resolv.conf /mnt/etc/resolv.conf.orig cp /etc/resolv.conf /mnt/etc/resolv.conf yes n | pkg -c /mnt -o ASSUME_ALWAYS_YES=3DYES install $PKGS_TO_INSTALL mv /mnt/etc/resolv.conf.orig /mnt/etc/resolv.conf svnlite --non-interactive --trust-server-cert-failures=3Dunknown-ca co = https://svn.freebsd.org/base/releng/11.1/ /usr/src/ cd /usr/src cat > /tmp/kernel.patch < On 2 Jan 2018, at 20:48, Rafal Lukawiecki wrote: >=20 > I have encountered another, hopefully minor issue running pkg in the = unattended script. When it installs a package (postfix-sasl) it expects = keyboard input to =E2=80=9CWould you like to active Postfix in = /usr/local/etc/mail/mailer.conf [n]?=E2=80=9D. I have tried both: >=20 > export ASSUME_ALWAYS_YES=3DYES > pkg -c /mnt -o ASSUME_ALWAYS_YES=3DYES install $PKGS_TO_INSTALL >=20 > but the process still hangs waiting for input. Are there any other = flags or tricks I could pass to pkg, short of prefixing it with =E2=80=9Cy= es n |=E2=80=9D? Looking through man for pkg I do not see any options = designed for such unattended install cases. >=20 > Many thanks for your help. >=20 > Rafal >=20 > _______________________________________________ > freebsd-cloud@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-cloud > To unsubscribe, send any mail to = "freebsd-cloud-unsubscribe@freebsd.org"