From owner-freebsd-arm@freebsd.org Thu Mar 21 10:53:57 2019 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A4DF3153219E for ; Thu, 21 Mar 2019 10:53:57 +0000 (UTC) (envelope-from mw@semihalf.com) Received: from mail-qt1-x833.google.com (mail-qt1-x833.google.com [IPv6:2607:f8b0:4864:20::833]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8DA39855AF for ; Thu, 21 Mar 2019 10:53:55 +0000 (UTC) (envelope-from mw@semihalf.com) Received: by mail-qt1-x833.google.com with SMTP id z16so4590564qtn.4 for ; Thu, 21 Mar 2019 03:53:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=QQNDow3+mdTic15Onw8mwEWbydV0qZCk45Ua28oHQdQ=; b=FdFmYoNxXoUuFBmUsAX4xNCi27E5hNZ6bVOyjeBuIXChBxlWN1OquTfvrDmqpCVHK6 dS60DiVXDF3ytjJrPC8N+8Ivoslu2C5Tu19m5ir/tMqSYgWAk0cyq1bDoykqpwKRsU+j R6P15/Tk+ValFRNKvI64ho1l0uYXB3ApcDRjlXGOmEcgeR/0D+aXR9DavC+Plvkn2i6E HS011cIAtEVglYVcBdQzlzfhq6q6ClnqrXsfg0XemT8sPYNlIdyMv9ee7UJgLXqCdF5z C4xAytismk3kykMejMXZPpETRkXmE5q2SJWuQU8yCNQe7iQQhO9ZKb9sKQioWqq4IF+S AnWA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=QQNDow3+mdTic15Onw8mwEWbydV0qZCk45Ua28oHQdQ=; b=Zz+IN3+TqI0EZpojMH05xp/OkrSBsU+bjuY8WrA9W6fEu82dAgQ29w8/JaMeDZyK80 U4Foc6eMiGovYM8P3ldAuT6RzeR0XChAmDkQkRtb9LcaqWHfeA4Stz3djDxh3JpH/McP pyI++Mb+PTXJ8iP/Ol/mu9x2kTR7LJy/OAiIfAzDxVahvaLwFvaBEXWCLMFw5Jp8/ngn ikM++6s4JAG+KD7sxiZilDYZF+9L6IeLYxYKa2nxZU6TJO4QaJlSSCipJlnoWfUMyELP j6IFXDvt/K+pCslOZgC+6ZmAYQ1JmrJR3iLGpwzRafRftXQW+ANVSPAEWjfRiskskOd0 8Osg== X-Gm-Message-State: APjAAAXiqwkNCtOKUTR+V8cfOlRKwwoQSmG9Zv+I6V6VuffJtOBXQvl1 wZl11lH28KconZsyFgu2eIg/3P+M5NhddEbouaLiVA== X-Google-Smtp-Source: APXvYqzd5jmfPA2g7Zt08K9m3tI/NNuzUw8ITmTZ/xU5CKruDC6XOYJh3fzVBe3ls9HekZJXXzTGhzZuLL/Je9JH/sc= X-Received: by 2002:a0c:9acc:: with SMTP id k12mr2454506qvf.211.1553165633876; Thu, 21 Mar 2019 03:53:53 -0700 (PDT) MIME-Version: 1.0 References: <79CC79B9-81AF-4563-BABE-429E6A57F476@bmalum.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazonses.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazo> <1548182399.2864.0@smtp.migadu.com> <1552255580.21373.0@unrelenting.technology> <01000169992ab14c-aa38b2f1-3f57-4805-9d8a-fa92207e4c2d-000000@email.amazonses.com> <010001699e6c9812-a0fc8d06-51d3-4252-ac06-2c1087d2c46f-000000@email.amazonses.com> In-Reply-To: <010001699e6c9812-a0fc8d06-51d3-4252-ac06-2c1087d2c46f-000000@email.amazonses.com> From: Marcin Wojtas Date: Thu, 21 Mar 2019 11:53:41 +0100 Message-ID: Subject: Re: ARM Graviton AWS Processor (AMI Image) To: Colin Percival Cc: =?UTF-8?Q?Micha=C5=82_Krawczyk?= , "Matushevsky, Alexander" , freebsd-arm , "Tzalik, Guy" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 8DA39855AF X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=semihalf-com.20150623.gappssmtp.com header.s=20150623 header.b=FdFmYoNx X-Spamd-Result: default: False [-5.73 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[semihalf-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arm@freebsd.org]; DMARC_NA(0.00)[semihalf.com]; RCPT_COUNT_FIVE(0.00)[5]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[semihalf-com.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[ASPMX2.GOOGLEMAIL.com,ALT2.ASPMX.L.GOOGLE.com,ASPMX.L.GOOGLE.com,ALT1.ASPMX.L.GOOGLE.com,ASPMX3.GOOGLEMAIL.com]; RCVD_IN_DNSWL_NONE(0.00)[3.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.69)[-0.687,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.73)[ip: (-8.65), ipnet: 2607:f8b0::/32(-2.82), asn: 15169(-2.12), country: US(-0.07)] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 21 Mar 2019 10:53:57 -0000 Hi Colin, Fixes pushed: https://svnweb.freebsd.org/base?view=3Drevision&revision=3D345371 https://svnweb.freebsd.org/base?view=3Drevision&revision=3D345373 Best regards, Marcin czw., 21 mar 2019 o 06:05 Colin Percival napisa=C5= =82(a): > > Hi Micha=C5=82, > > Thanks, those two patches fix both problems. Can you get them into > the FreeBSD tree or should I commit them? > > Colin Percival > > On 3/20/19 3:24 AM, Micha=C5=82 Krawczyk wrote: > > Hi, > > > > we have solution for two of the issues with ENA, > > > > The double resource activation can be fixed by this patch: > > https://reviews.freebsd.org/D19655 > > > > And the issue with reloading module/reconfiguring it by using ifconfig > > can be fixed by using this patch: > > https://reviews.freebsd.org/D19641 > > > > Please check if that fixes it for you. > > > > Thanks, > > Micha=C5=82 > > > > > > =C5=9Br., 20 mar 2019 o 04:35 Colin Percival nap= isa=C5=82(a): > >> > >> Hi ENA people, > >> > >> Any luck figuring out what was going on with the ENA driver here > >> with interrupt activation? > >> > >> Colin Percival > >> > >> On 3/11/19 1:35 AM, Marcin Wojtas wrote: > >>> +FreeBSD ENA maintainers > >>> > >>> W dniu pon., 11.03.2019 o 00:40 Greg V > >>> napisa=C5=82(a): > >>> > >>>> > >>>> On Tue, Jan 22, 2019 at 9:39 PM, Greg V > >>>> wrote: > >>>>> On Mon, Jan 21, 2019 at 1:11 PM, Martin Karrer > >>>>> wrote: > >>>>>> My question is if there are any plans yet to support the Graviton > >>>>>> ARM instances of AWS? > >>>>>> > >>>>>> We have a heavy load on FreeBSD and would also use the ARM > >>>>>> instances. Are there any other interested parties? > >>>>> > >>>>> I have tried this. It should work very well in theory, e.g. the > >>>>> network card driver (if_ena) compiles with no changes for aarch64, > >>>>> and in fact NetBSD has ported this driver and is up and running on > >>>>> these instances: https://dmesgd.nycbug.org/index.cgi?do=3Dview&id= =3D4623 > >>>>> > >>>>> But my result with FreeBSD was: nothing on the console after > >>>>> loader.efi hands control to the kernel. > >>>>> [=E2=80=A6] > >>>> > >>>> Hello everyone, big update: > >>>> > >>>> FreeBSD/aarch64 on Amazon EC2 a1 (AWS Graviton) instances WORKS! > >>>> > >>>> https://dmesgd.nycbug.org/index.cgi?do=3Dview&id=3D4813 > >>>> > >>>> And you can try it (well, my -CURRENT build, NO WARRANTY etc) right = now: > >>>> > >>>> ami-0c2829a0b82a62ca6 in eu-west-1 (Ireland) > >>>> > >>>> ----- > >>>> > >>>> So, what I had to do / what should be done / how others can help get > >>>> this into a finished state: > >>>> > >>>> 1. Serial console: > >>>> - I fixed it: https://reviews.freebsd.org/D19507 > >>>> - (I learned some things about UARTs and their support in FreeBSD= , > >>>> should write a blog post about that) > >>>> > >>>> 2. aarch64 build configuration: > >>>> - if_ena network driver module should be enabled: > >>>> https://reviews.freebsd.org/D18372 > >>>> - NVMe driver should be enabled in the GENERIC kernel config (dev= ice > >>>> nvme, device nvd) > >>>> - BTW, why not also go with hw.nvme.use_nvd=3D"0" by default on > >>>> aarch64, IIRC that was done on powerpc64 > >>>> > >>>> 3. VM image build system: > >>>> - GPT+EFI should be used (amd64 was GPT with no EFI, and aarch64 = was > >>>> MBR with EFI (???)): https://reviews.freebsd.org/D18371 > >>>> - bsdec2-image-upload --arm64 flag should be supported: included > >>>> above ^^ > >>>> - ec2.conf: amazon-ssm-agent shouldn't be installed when building > >>>> for aarch64 TARGET, since that's written in Go, and Go isn't ported = to > >>>> FreeBSD/aarch64 yet: > >>>> > >>>> https://github.com/myfreeweb/freebsd/commit/5b530ebf7385d8320b9076cf= 84f50aad01689bc > >>>> (untested patch, I actually used an interactive shell in between the > >>>> image build commands) > >>>> - qemu-aarch64-static should be used for preinstalling pkgs when > >>>> chrooting into the image: rough version included above ^^ > >>>> > >>>> 4. ENA (Elastic Network Adapter) driver: > >>>> - it works > >>>> - except there's something funky with interrupt activation, and i= t > >>>> hits panic("Attempt to double activation of resource id: %u\n", res_= id) > >>>> (for the management IRQ) on boot, so I applied the obvious silly > >>>> workaround of "don't panic": > >>>> > >>>> https://github.com/myfreeweb/freebsd/commit/a7e7c6e48cdbdb0fdc6c4e0b= a63392262938e62c > >>>> - but still, it doesn't properly reactivate interrupts (and the b= ox > >>>> becomes unreachable over the net) after going down and up again =E2= =80=94 > >>>> guess what does that on boot? dhclient applying the big jumbo MTU = =E2=80=94 > >>>> so I set dhclient.conf to reject MTU changes: > >>>> > >>>> https://github.com/myfreeweb/freebsd/commit/03ec4d417b0b4252285baaf4= e294cc6d8c870f7f > >>>> > >>>> > >>>> Would be great if someone familiar with interrupts and stuff could h= elp > >>>> debug the ena driver and make it work without these hacks :) > >> -- > >> Colin Percival > >> Security Officer Emeritus, FreeBSD | The power to serve > >> Founder, Tarsnap | www.tarsnap.com | Online backups for the truly para= noid > > > > > > -- > Colin Percival > Security Officer Emeritus, FreeBSD | The power to serve > Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoi= d > _______________________________________________ > freebsd-arm@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-arm > To unsubscribe, send any mail to "freebsd-arm-unsubscribe@freebsd.org"