Skip site navigation (1)Skip section navigation (2)
Date:      Thu, 21 Mar 2019 11:53:41 +0100
From:      Marcin Wojtas <mw@semihalf.com>
To:        Colin Percival <cperciva@tarsnap.com>
Cc:        =?UTF-8?Q?Micha=C5=82_Krawczyk?= <mk@semihalf.com>,  "Matushevsky, Alexander" <matua@amazon.com>, freebsd-arm <freebsd-arm@freebsd.org>, "Tzalik, Guy" <gtzalik@amazon.com>
Subject:   Re: ARM Graviton AWS Processor (AMI Image)
Message-ID:  <CAPv3WKcy3udYxr3CY5DahR26_ef9OK5mAM%2BBr%2B3QHO5HtCN1Ow@mail.gmail.com>
In-Reply-To: <010001699e6c9812-a0fc8d06-51d3-4252-ac06-2c1087d2c46f-000000@email.amazonses.com>
References:  <79CC79B9-81AF-4563-BABE-429E6A57F476@bmalum.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazonses.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazo> <1548182399.2864.0@smtp.migadu.com> <1552255580.21373.0@unrelenting.technology> <CAPv3WKdV4BcpCTj%2BC=raa_Jh2Rce5wtSSu2XaEkP=qxa4zZi7Q@mail.gmail.com> <01000169992ab14c-aa38b2f1-3f57-4805-9d8a-fa92207e4c2d-000000@email.amazonses.com> <CAJMMOfOxOQeVwYcqQfCJ0RvfmNBCkBnd=sYN3dpkMhdGUKv%2BXA@mail.gmail.com> <010001699e6c9812-a0fc8d06-51d3-4252-ac06-2c1087d2c46f-000000@email.amazonses.com>

next in thread | previous in thread | raw e-mail | index | archive | help
Hi Colin,

Fixes pushed:
https://svnweb.freebsd.org/base?view=3Drevision&revision=3D345371
https://svnweb.freebsd.org/base?view=3Drevision&revision=3D345373

Best regards,
Marcin


czw., 21 mar 2019 o 06:05 Colin Percival <cperciva@tarsnap.com> napisa=C5=
=82(a):
>
> Hi Micha=C5=82,
>
> Thanks, those two patches fix both problems.  Can you get them into
> the FreeBSD tree or should I commit them?
>
> Colin Percival
>
> On 3/20/19 3:24 AM, Micha=C5=82 Krawczyk wrote:
> > Hi,
> >
> > we have solution for two of the issues with ENA,
> >
> > The double resource activation can be fixed by this patch:
> > https://reviews.freebsd.org/D19655
> >
> > And the issue with reloading module/reconfiguring it by using ifconfig
> > can be fixed by using this patch:
> > https://reviews.freebsd.org/D19641
> >
> > Please check if that fixes it for you.
> >
> > Thanks,
> > Micha=C5=82
> >
> >
> > =C5=9Br., 20 mar 2019 o 04:35 Colin Percival <cperciva@tarsnap.com> nap=
isa=C5=82(a):
> >>
> >> Hi ENA people,
> >>
> >> Any luck figuring out what was going on with the ENA driver here
> >> with interrupt activation?
> >>
> >> Colin Percival
> >>
> >> On 3/11/19 1:35 AM, Marcin Wojtas wrote:
> >>> +FreeBSD ENA maintainers
> >>>
> >>> W dniu pon., 11.03.2019 o 00:40 Greg V <greg@unrelenting.technology>
> >>> napisa=C5=82(a):
> >>>
> >>>>
> >>>> On Tue, Jan 22, 2019 at 9:39 PM, Greg V <greg@unrelenting.technology=
>
> >>>> wrote:
> >>>>> On Mon, Jan 21, 2019 at 1:11 PM, Martin Karrer <martin@bmalum.com>
> >>>>> wrote:
> >>>>>> My question is if there are any plans yet to support the Graviton
> >>>>>> ARM  instances of AWS?
> >>>>>>
> >>>>>> We have a heavy load on FreeBSD and would also use the ARM
> >>>>>> instances.  Are there any other interested parties?
> >>>>>
> >>>>> I have tried this. It should work very well in theory, e.g. the
> >>>>> network card driver (if_ena) compiles with no changes for aarch64,
> >>>>> and in fact NetBSD has ported this driver and is up and running on
> >>>>> these instances: https://dmesgd.nycbug.org/index.cgi?do=3Dview&id=
=3D4623
> >>>>>
> >>>>> But my result with FreeBSD was: nothing on the console after
> >>>>> loader.efi hands control to the kernel.
> >>>>> [=E2=80=A6]
> >>>>
> >>>> Hello everyone, big update:
> >>>>
> >>>> FreeBSD/aarch64 on Amazon EC2 a1 (AWS Graviton) instances WORKS!
> >>>>
> >>>> https://dmesgd.nycbug.org/index.cgi?do=3Dview&id=3D4813
> >>>>
> >>>> And you can try it (well, my -CURRENT build, NO WARRANTY etc) right =
now:
> >>>>
> >>>> ami-0c2829a0b82a62ca6 in eu-west-1 (Ireland)
> >>>>
> >>>> -----
> >>>>
> >>>> So, what I had to do / what should be done / how others can help get
> >>>> this into a finished state:
> >>>>
> >>>> 1. Serial console:
> >>>>    - I fixed it: https://reviews.freebsd.org/D19507
> >>>>    - (I learned some things about UARTs and their support in FreeBSD=
,
> >>>> should write a blog post about that)
> >>>>
> >>>> 2. aarch64 build configuration:
> >>>>    - if_ena network driver module should be enabled:
> >>>> https://reviews.freebsd.org/D18372
> >>>>    - NVMe driver should be enabled in the GENERIC kernel config (dev=
ice
> >>>> nvme, device nvd)
> >>>>    - BTW, why not also go with hw.nvme.use_nvd=3D"0" by default on
> >>>> aarch64, IIRC that was done on powerpc64
> >>>>
> >>>> 3. VM image build system:
> >>>>    - GPT+EFI should be used (amd64 was GPT with no EFI, and aarch64 =
was
> >>>> MBR with EFI (???)): https://reviews.freebsd.org/D18371
> >>>>    - bsdec2-image-upload --arm64 flag should be supported: included
> >>>> above ^^
> >>>>    - ec2.conf: amazon-ssm-agent shouldn't be installed when building
> >>>> for aarch64 TARGET, since that's written in Go, and Go isn't ported =
to
> >>>> FreeBSD/aarch64 yet:
> >>>>
> >>>> https://github.com/myfreeweb/freebsd/commit/5b530ebf7385d8320b9076cf=
84f50aad01689bc
> >>>> (untested patch, I actually used an interactive shell in between the
> >>>> image build commands)
> >>>>    - qemu-aarch64-static should be used for preinstalling pkgs when
> >>>> chrooting into the image: rough version included above ^^
> >>>>
> >>>> 4. ENA (Elastic Network Adapter) driver:
> >>>>    - it works
> >>>>    - except there's something funky with interrupt activation, and i=
t
> >>>> hits panic("Attempt to double activation of resource id: %u\n", res_=
id)
> >>>> (for the management IRQ) on boot, so I applied the obvious silly
> >>>> workaround of "don't panic":
> >>>>
> >>>> https://github.com/myfreeweb/freebsd/commit/a7e7c6e48cdbdb0fdc6c4e0b=
a63392262938e62c
> >>>>    - but still, it doesn't properly reactivate interrupts (and the b=
ox
> >>>> becomes unreachable over the net) after going down and up again =E2=
=80=94
> >>>> guess what does that on boot? dhclient applying the big jumbo MTU =
=E2=80=94
> >>>> so I set dhclient.conf to reject MTU changes:
> >>>>
> >>>> https://github.com/myfreeweb/freebsd/commit/03ec4d417b0b4252285baaf4=
e294cc6d8c870f7f
> >>>>
> >>>>
> >>>> Would be great if someone familiar with interrupts and stuff could h=
elp
> >>>> debug the ena driver and make it work without these hacks :)
> >> --
> >> Colin Percival
> >> Security Officer Emeritus, FreeBSD | The power to serve
> >> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly para=
noid
> >
> >
>
> --
> Colin Percival
> Security Officer Emeritus, FreeBSD | The power to serve
> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoi=
d
> _______________________________________________
> freebsd-arm@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-arm
> To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org"



Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CAPv3WKcy3udYxr3CY5DahR26_ef9OK5mAM%2BBr%2B3QHO5HtCN1Ow>