From owner-freebsd-arm@freebsd.org Thu Mar 21 04:05:19 2019 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 15236154CBED for ; Thu, 21 Mar 2019 04:05:19 +0000 (UTC) (envelope-from 010001699e6c9812-a0fc8d06-51d3-4252-ac06-2c1087d2c46f-000000@amazonses.com) Received: from a8-52.smtp-out.amazonses.com (a8-52.smtp-out.amazonses.com [54.240.8.52]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8D2716BFB6 for ; Thu, 21 Mar 2019 04:05:17 +0000 (UTC) (envelope-from 010001699e6c9812-a0fc8d06-51d3-4252-ac06-2c1087d2c46f-000000@amazonses.com) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=vnqrkfnvu6csdl6mwgk5t6ix3nnepx57; d=tarsnap.com; t=1553141111; h=Subject:To:Cc:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding; bh=7A5b7nk9wigh6p/oREGTdLpP+CQlzpci0N70YWCBoAI=; b=Ozhkhhvh1v8t2mMSduIa7+7NiFEJ+RCQqDq7N/UJJ/3ZelShb7DxDZA/mRY33//G pS3hE94xC+zVJcIZNmckKS1zCMcofQofR96mQZltke/VGE+aDLFswCN5qQ+hTFyBMBa j8KOAjs2FZKRVJklHBQwMFModBZOp8DA8EKFtT24= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/simple; s=6gbrjpgwjskckoa6a5zn6fwqkn67xbtw; d=amazonses.com; t=1553141110; h=Subject:To:Cc:References:From:Message-ID:Date:MIME-Version:In-Reply-To:Content-Type:Content-Transfer-Encoding:Feedback-ID; bh=7A5b7nk9wigh6p/oREGTdLpP+CQlzpci0N70YWCBoAI=; b=JM/ZcpM36M4pb3lL6ZXR/tXALgN4ZWFn9DsDcFdSkBFpLgxwF0ghS0dYG1PQdbHu QsVjbOQKQ7tqYTEEm2BAi5HBFxtYZg5tg/72gDWxsN+3TrHqI6OppXXQD4HuIRvWyKA cEG4cSjp/ZFxaxcw0sXbLl0Fu0mkBpoeIt9nEpQA= Subject: Re: ARM Graviton AWS Processor (AMI Image) To: =?UTF-8?Q?Micha=c5=82_Krawczyk?= Cc: Marcin Wojtas , Greg V , "Matushevsky, Alexander" , Martin Karrer , =?UTF-8?Q?Rafa=c5=82_Kozik?= , freebsd-arm@freebsd.org, "Tzalik, Guy" References: <79CC79B9-81AF-4563-BABE-429E6A57F476@bmalum.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazonses.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazo> <1548182399.2864.0@smtp.migadu.com> <1552255580.21373.0@unrelenting.technology> <01000169992ab14c-aa38b2f1-3f57-4805-9d8a-fa92207e4c2d-000000@email.amazonses.com> From: Colin Percival Openpgp: preference=signencrypt Autocrypt: addr=cperciva@tarsnap.com; prefer-encrypt=mutual; keydata= mQGhBElrAAcRBACDfDys4ZtK+ErCJ1HAzYeteKpm3OEsvT/49AjUTLihkF79HhIKrCQU+1KC zv7BwHCMLb6hq30As9L7iFKG7n5QFLFC4Te/VcITUnWHMG/c3ViLOfJGvi+9/nOEHaM1dVJY D6tEp5yM1nHmVQpo9932j4KGuGFR0LhOK5IHXOSfGwCgxSFDPdgxe2OEjWxjGgY+oV3EafcD +JROXCTjlcQiG/OguQH4Vks3mhHfFnEppLxTkDuYgHZQiUtpcT9ssH5khgqoTyMar05OUdAj ZIhNbWDh4LgTj+7ZmvLhXT5Zxw8LX9d7T36aTB8XDQSenDqEtinMWOb0TCBBLbsB8EFG1WTT ESbZci9jJS5yhtktuZoY/eM8uXMD/3k4FWFO80VRRkELSp+XSy/VlSQjyi/rhl2nQq/oOA9F oJbDaB0yq9VNhxP+uFBzBWSqeIX0t1ZWLtNfVFr4TRP5hihI5ICrg/0OpqgisKsU2NFe9xyO hyJLYmfD8ebpDJ/9k30C7Iju9pVrwLm1QgS4S2fqJRcR+U4WbjvP7CgStCVDb2xpbiBQZXJj aXZhbCA8Y3BlcmNpdmFAdGFyc25hcC5jb20+iGEEExECACEFAklrALYCGwMHCwkIBwMCAQQV AggDBBYCAwECHgECF4AACgkQOM7KaQxqam6/igCgn+z2k3V5ggNppmWrZstt1U2lugsAoL7L wS9V9yLtil3oWmHtwpUqYruEuQINBElrAAcQCAD3ZLMIsP4CIDoJORg+YY0lqLVBgcnF7pFb 4Uy2+KvdWofN+DKH61rZLjgXXkNE9M4EQC1B4lGttBP8IY2gs41y3AUogGdyFbidq99rCBz7 LTsgARHwFxZoaHmXyiZLEU1QZuMqwPZV1mCviRhN5E3rRqYNXVcrnXAAuhBpvNyj/ntHvcDN 2/m+ochiuBYueU4kX3lHya7sOj+mTsndcWmQ9soOUyr8O0r/BG088bMn4qqtUw4dl5/pglXk jbl7uOOPinKf0WVd2r6M0wLPJCD4NPHrCWRLLLAjwfjrtoSRvXxDbXhCdgGBa72+K8eYLzVs hgq7tJOoBWzjVK6XRxR7AAMGB/9Mo3iJ2DxqDecd02KCB5BsFDICbJGhPltU7FwrtbC7djSb XUrwsEVLHi4st4cbdGNCWCrp0BRezXZKohKnNAPFOTK++ZfgeKxrV2sJod+Q9RILF86tQ4XF 7A7Yme5hy92t/WgiU4vc/fWbgP8gV/19f8nunaT2E9NSa70mZFjZNu4iuwThoUUO5CV3Wo0Y UISsnRK8XD1+LR3A2qVyLiFRwh/miC1hgLFCTGCQ3GLxZeZzIpYSlGdQJ0L5lixW5ZQD9r1I 8i/8zhE6qRFAM0upUMI3Gt1Oq2w03DiXrZU0Fu/R8Rm8rlnkQKA+95mRTUq1xL5P5NZIi4gJ Z569OPMFiEkEGBECAAkFAklrAAcCGwwACgkQOM7KaQxqam41igCfbaldnFTu5uAdrnrghESv EI3CAo8AoLkNMks1pThl2BJNRm4CtTK9xZeH Message-ID: <010001699e6c9812-a0fc8d06-51d3-4252-ac06-2c1087d2c46f-000000@email.amazonses.com> Date: Thu, 21 Mar 2019 04:05:10 +0000 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.5.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-SES-Outgoing: 2019.03.21-54.240.8.52 Feedback-ID: 1.us-east-1.Lv9FVjaNvvR5llaqfLoOVbo2VxOELl7cjN0AOyXnPlk=:AmazonSES X-Rspamd-Queue-Id: 8D2716BFB6 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=tarsnap.com header.s=vnqrkfnvu6csdl6mwgk5t6ix3nnepx57 header.b=Ozhkhhvh; dkim=pass header.d=amazonses.com header.s=6gbrjpgwjskckoa6a5zn6fwqkn67xbtw header.b=JM/ZcpM3; spf=pass (mx1.freebsd.org: domain of 010001699e6c9812-a0fc8d06-51d3-4252-ac06-2c1087d2c46f-000000@amazonses.com designates 54.240.8.52 as permitted sender) smtp.mailfrom=010001699e6c9812-a0fc8d06-51d3-4252-ac06-2c1087d2c46f-000000@amazonses.com X-Spamd-Result: default: False [-2.44 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[tarsnap.com:s=vnqrkfnvu6csdl6mwgk5t6ix3nnepx57,amazonses.com:s=6gbrjpgwjskckoa6a5zn6fwqkn67xbtw]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:54.240.0.0/18]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[tarsnap.com]; NEURAL_HAM_SHORT(-0.97)[-0.972,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: feedback-smtp.us-east-1.amazonses.com]; DKIM_TRACE(0.00)[tarsnap.com:+,amazonses.com:+]; RCPT_COUNT_SEVEN(0.00)[8]; RCVD_IN_DNSWL_NONE(0.00)[52.8.240.54.list.dnswl.org : 127.0.15.0]; FORGED_SENDER(0.30)[cperciva@tarsnap.com,010001699e6c9812-a0fc8d06-51d3-4252-ac06-2c1087d2c46f-000000@amazonses.com]; RCVD_COUNT_ZERO(0.00)[0]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-1.76)[ip: (-0.82), ipnet: 54.240.8.0/21(-4.57), asn: 14618(-3.33), country: US(-0.07)]; ASN(0.00)[asn:14618, ipnet:54.240.8.0/21, country:US]; FORGED_MUA_THUNDERBIRD_MSGID_UNKNOWN(2.50)[]; FROM_NEQ_ENVFROM(0.00)[cperciva@tarsnap.com, 010001699e6c9812-a0fc8d06-51d3-4252-ac06-2c1087d2c46f-000000@amazonses.com] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Mar 2019 04:05:19 -0000 Hi Michał, Thanks, those two patches fix both problems. Can you get them into the FreeBSD tree or should I commit them? Colin Percival On 3/20/19 3:24 AM, Michał Krawczyk wrote: > Hi, > > we have solution for two of the issues with ENA, > > The double resource activation can be fixed by this patch: > https://reviews.freebsd.org/D19655 > > And the issue with reloading module/reconfiguring it by using ifconfig > can be fixed by using this patch: > https://reviews.freebsd.org/D19641 > > Please check if that fixes it for you. > > Thanks, > Michał > > > śr., 20 mar 2019 o 04:35 Colin Percival napisał(a): >> >> Hi ENA people, >> >> Any luck figuring out what was going on with the ENA driver here >> with interrupt activation? >> >> Colin Percival >> >> On 3/11/19 1:35 AM, Marcin Wojtas wrote: >>> +FreeBSD ENA maintainers >>> >>> W dniu pon., 11.03.2019 o 00:40 Greg V >>> napisał(a): >>> >>>> >>>> On Tue, Jan 22, 2019 at 9:39 PM, Greg V >>>> wrote: >>>>> On Mon, Jan 21, 2019 at 1:11 PM, Martin Karrer >>>>> wrote: >>>>>> My question is if there are any plans yet to support the Graviton >>>>>> ARM instances of AWS? >>>>>> >>>>>> We have a heavy load on FreeBSD and would also use the ARM >>>>>> instances. Are there any other interested parties? >>>>> >>>>> I have tried this. It should work very well in theory, e.g. the >>>>> network card driver (if_ena) compiles with no changes for aarch64, >>>>> and in fact NetBSD has ported this driver and is up and running on >>>>> these instances: https://dmesgd.nycbug.org/index.cgi?do=view&id=4623 >>>>> >>>>> But my result with FreeBSD was: nothing on the console after >>>>> loader.efi hands control to the kernel. >>>>> […] >>>> >>>> Hello everyone, big update: >>>> >>>> FreeBSD/aarch64 on Amazon EC2 a1 (AWS Graviton) instances WORKS! >>>> >>>> https://dmesgd.nycbug.org/index.cgi?do=view&id=4813 >>>> >>>> And you can try it (well, my -CURRENT build, NO WARRANTY etc) right now: >>>> >>>> ami-0c2829a0b82a62ca6 in eu-west-1 (Ireland) >>>> >>>> ----- >>>> >>>> So, what I had to do / what should be done / how others can help get >>>> this into a finished state: >>>> >>>> 1. Serial console: >>>> - I fixed it: https://reviews.freebsd.org/D19507 >>>> - (I learned some things about UARTs and their support in FreeBSD, >>>> should write a blog post about that) >>>> >>>> 2. aarch64 build configuration: >>>> - if_ena network driver module should be enabled: >>>> https://reviews.freebsd.org/D18372 >>>> - NVMe driver should be enabled in the GENERIC kernel config (device >>>> nvme, device nvd) >>>> - BTW, why not also go with hw.nvme.use_nvd="0" by default on >>>> aarch64, IIRC that was done on powerpc64 >>>> >>>> 3. VM image build system: >>>> - GPT+EFI should be used (amd64 was GPT with no EFI, and aarch64 was >>>> MBR with EFI (???)): https://reviews.freebsd.org/D18371 >>>> - bsdec2-image-upload --arm64 flag should be supported: included >>>> above ^^ >>>> - ec2.conf: amazon-ssm-agent shouldn't be installed when building >>>> for aarch64 TARGET, since that's written in Go, and Go isn't ported to >>>> FreeBSD/aarch64 yet: >>>> >>>> https://github.com/myfreeweb/freebsd/commit/5b530ebf7385d8320b9076cf84f50aad01689bc >>>> (untested patch, I actually used an interactive shell in between the >>>> image build commands) >>>> - qemu-aarch64-static should be used for preinstalling pkgs when >>>> chrooting into the image: rough version included above ^^ >>>> >>>> 4. ENA (Elastic Network Adapter) driver: >>>> - it works >>>> - except there's something funky with interrupt activation, and it >>>> hits panic("Attempt to double activation of resource id: %u\n", res_id) >>>> (for the management IRQ) on boot, so I applied the obvious silly >>>> workaround of "don't panic": >>>> >>>> https://github.com/myfreeweb/freebsd/commit/a7e7c6e48cdbdb0fdc6c4e0ba63392262938e62c >>>> - but still, it doesn't properly reactivate interrupts (and the box >>>> becomes unreachable over the net) after going down and up again — >>>> guess what does that on boot? dhclient applying the big jumbo MTU — >>>> so I set dhclient.conf to reject MTU changes: >>>> >>>> https://github.com/myfreeweb/freebsd/commit/03ec4d417b0b4252285baaf4e294cc6d8c870f7f >>>> >>>> >>>> Would be great if someone familiar with interrupts and stuff could help >>>> debug the ena driver and make it work without these hacks :) >> -- >> Colin Percival >> Security Officer Emeritus, FreeBSD | The power to serve >> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid > > -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid