Date: Thu, 21 Mar 2019 04:05:10 +0000 From: Colin Percival <cperciva@tarsnap.com> To: =?UTF-8?Q?Micha=c5=82_Krawczyk?= <mk@semihalf.com> Cc: Marcin Wojtas <mw@semihalf.com>, Greg V <greg@unrelenting.technology>, "Matushevsky, Alexander" <matua@amazon.com>, Martin Karrer <martin@bmalum.com>, =?UTF-8?Q?Rafa=c5=82_Kozik?= <rk@semihalf.com>, freebsd-arm@freebsd.org, "Tzalik, Guy" <gtzalik@amazon.com> Subject: Re: ARM Graviton AWS Processor (AMI Image) Message-ID: <010001699e6c9812-a0fc8d06-51d3-4252-ac06-2c1087d2c46f-000000@email.amazonses.com> In-Reply-To: <CAJMMOfOxOQeVwYcqQfCJ0RvfmNBCkBnd=sYN3dpkMhdGUKv%2BXA@mail.gmail.com> References: <79CC79B9-81AF-4563-BABE-429E6A57F476@bmalum.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazonses.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazo> <1548182399.2864.0@smtp.migadu.com> <1552255580.21373.0@unrelenting.technology> <CAPv3WKdV4BcpCTj%2BC=raa_Jh2Rce5wtSSu2XaEkP=qxa4zZi7Q@mail.gmail.com> <01000169992ab14c-aa38b2f1-3f57-4805-9d8a-fa92207e4c2d-000000@email.amazonses.com> <CAJMMOfOxOQeVwYcqQfCJ0RvfmNBCkBnd=sYN3dpkMhdGUKv%2BXA@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi Michał, Thanks, those two patches fix both problems. Can you get them into the FreeBSD tree or should I commit them? Colin Percival On 3/20/19 3:24 AM, Michał Krawczyk wrote: > Hi, > > we have solution for two of the issues with ENA, > > The double resource activation can be fixed by this patch: > https://reviews.freebsd.org/D19655 > > And the issue with reloading module/reconfiguring it by using ifconfig > can be fixed by using this patch: > https://reviews.freebsd.org/D19641 > > Please check if that fixes it for you. > > Thanks, > Michał > > > śr., 20 mar 2019 o 04:35 Colin Percival <cperciva@tarsnap.com> napisał(a): >> >> Hi ENA people, >> >> Any luck figuring out what was going on with the ENA driver here >> with interrupt activation? >> >> Colin Percival >> >> On 3/11/19 1:35 AM, Marcin Wojtas wrote: >>> +FreeBSD ENA maintainers >>> >>> W dniu pon., 11.03.2019 o 00:40 Greg V <greg@unrelenting.technology> >>> napisał(a): >>> >>>> >>>> On Tue, Jan 22, 2019 at 9:39 PM, Greg V <greg@unrelenting.technology> >>>> wrote: >>>>> On Mon, Jan 21, 2019 at 1:11 PM, Martin Karrer <martin@bmalum.com> >>>>> wrote: >>>>>> My question is if there are any plans yet to support the Graviton >>>>>> ARM instances of AWS? >>>>>> >>>>>> We have a heavy load on FreeBSD and would also use the ARM >>>>>> instances. Are there any other interested parties? >>>>> >>>>> I have tried this. It should work very well in theory, e.g. the >>>>> network card driver (if_ena) compiles with no changes for aarch64, >>>>> and in fact NetBSD has ported this driver and is up and running on >>>>> these instances: https://dmesgd.nycbug.org/index.cgi?do=view&id=4623 >>>>> >>>>> But my result with FreeBSD was: nothing on the console after >>>>> loader.efi hands control to the kernel. >>>>> […] >>>> >>>> Hello everyone, big update: >>>> >>>> FreeBSD/aarch64 on Amazon EC2 a1 (AWS Graviton) instances WORKS! >>>> >>>> https://dmesgd.nycbug.org/index.cgi?do=view&id=4813 >>>> >>>> And you can try it (well, my -CURRENT build, NO WARRANTY etc) right now: >>>> >>>> ami-0c2829a0b82a62ca6 in eu-west-1 (Ireland) >>>> >>>> ----- >>>> >>>> So, what I had to do / what should be done / how others can help get >>>> this into a finished state: >>>> >>>> 1. Serial console: >>>> - I fixed it: https://reviews.freebsd.org/D19507 >>>> - (I learned some things about UARTs and their support in FreeBSD, >>>> should write a blog post about that) >>>> >>>> 2. aarch64 build configuration: >>>> - if_ena network driver module should be enabled: >>>> https://reviews.freebsd.org/D18372 >>>> - NVMe driver should be enabled in the GENERIC kernel config (device >>>> nvme, device nvd) >>>> - BTW, why not also go with hw.nvme.use_nvd="0" by default on >>>> aarch64, IIRC that was done on powerpc64 >>>> >>>> 3. VM image build system: >>>> - GPT+EFI should be used (amd64 was GPT with no EFI, and aarch64 was >>>> MBR with EFI (???)): https://reviews.freebsd.org/D18371 >>>> - bsdec2-image-upload --arm64 flag should be supported: included >>>> above ^^ >>>> - ec2.conf: amazon-ssm-agent shouldn't be installed when building >>>> for aarch64 TARGET, since that's written in Go, and Go isn't ported to >>>> FreeBSD/aarch64 yet: >>>> >>>> https://github.com/myfreeweb/freebsd/commit/5b530ebf7385d8320b9076cf84f50aad01689bc >>>> (untested patch, I actually used an interactive shell in between the >>>> image build commands) >>>> - qemu-aarch64-static should be used for preinstalling pkgs when >>>> chrooting into the image: rough version included above ^^ >>>> >>>> 4. ENA (Elastic Network Adapter) driver: >>>> - it works >>>> - except there's something funky with interrupt activation, and it >>>> hits panic("Attempt to double activation of resource id: %u\n", res_id) >>>> (for the management IRQ) on boot, so I applied the obvious silly >>>> workaround of "don't panic": >>>> >>>> https://github.com/myfreeweb/freebsd/commit/a7e7c6e48cdbdb0fdc6c4e0ba63392262938e62c >>>> - but still, it doesn't properly reactivate interrupts (and the box >>>> becomes unreachable over the net) after going down and up again — >>>> guess what does that on boot? dhclient applying the big jumbo MTU — >>>> so I set dhclient.conf to reject MTU changes: >>>> >>>> https://github.com/myfreeweb/freebsd/commit/03ec4d417b0b4252285baaf4e294cc6d8c870f7f >>>> >>>> >>>> Would be great if someone familiar with interrupts and stuff could help >>>> debug the ena driver and make it work without these hacks :) >> -- >> Colin Percival >> Security Officer Emeritus, FreeBSD | The power to serve >> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid > > -- Colin Percival Security Officer Emeritus, FreeBSD | The power to serve Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoid
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?010001699e6c9812-a0fc8d06-51d3-4252-ac06-2c1087d2c46f-000000>