From owner-freebsd-arm@freebsd.org Wed Mar 20 10:24:21 2019 Return-Path: Delivered-To: freebsd-arm@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C314B154F046 for ; Wed, 20 Mar 2019 10:24:20 +0000 (UTC) (envelope-from mk@semihalf.com) Received: from mail-ed1-x544.google.com (mail-ed1-x544.google.com [IPv6:2a00:1450:4864:20::544]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5264C8F92F for ; Wed, 20 Mar 2019 10:24:19 +0000 (UTC) (envelope-from mk@semihalf.com) Received: by mail-ed1-x544.google.com with SMTP id q3so1511088edg.0 for ; Wed, 20 Mar 2019 03:24:19 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=semihalf-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=zMlyCR0ry6wOsrQ0PEUoHthq3WCsonL14wkJo3H33yA=; b=jlhnMPav7mFLVRR89gIzk2N0jtGXNFpKRq8jgnFTuKNkwfNTj95dTTyb78xbgsHDgT z26oJG/tQ8pti7cHW+K+n6EA0nNfKLshhcW18nRft8flYqFyf8UolEb6a5WVyteptgZ1 8CZWCwzK5XD51Kn10TskU33s9f44rnoHknE/UdOcf4dk7uoOWdU2aLOM4TV6RT6Sz6De SzDvoY54QmWXMlXfIy5z0daU1I3ZMj8J0sDeoEyXLBLDQU7Ddcpf/H9gztHC+VY/1zRQ nV8Y0Hg4E4+/fuEzBTh152qNlWc+DyVrzFNSc3PCRR8AEA90TB/IgawdpQ3t3US8rI7V FbBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=zMlyCR0ry6wOsrQ0PEUoHthq3WCsonL14wkJo3H33yA=; b=Q8hQGZl2dA3PfxJX2lHk0+tOBtwDjMhvqP6/ooP3Fqw+Ytb5rin2cZtgps1RsJqaZW AsFDi8AqIRl4fD7LpOTbUqN4vEIXwJADwEsbBwE/wK+5l0G4hY9Valzm1X8SnJypqVtl 6u5RJOeRacZznrF6Ra2WnHHwz6uRq0WP0l1Nz5wV4BOHquk5ou3Kvj26zueU3XNys/rz AU18dQiMm8r+4y2AZlztKtRwGDVRcuDyLG1cf3PA6chzmLOcY2Va9qbJr7+Gmda9AaH8 eNfRDXaUuOIZzpYiH0DekUp2c9YuA49GTzY1Xd0R8EPU2Yp1Wi2u09f4DGCHSnQfxxMT J1WA== X-Gm-Message-State: APjAAAVZg6tMr33tMJeplosrPGcEKY1uzO8DLDFmlGep5qKRKTgQ/ShM qA1NGAt969fCk1UGTE4vB1HwwWeg0kZ8VgYhqx+9BxH04bk= X-Google-Smtp-Source: APXvYqypAH4471ohlQULJKGD2KehmONMJ3IT05QGlpLGuR9VD2dT4bCr9caUNm2Nm0yPrQ8yf0i8jFkjxAX9AaAZWWE= X-Received: by 2002:a17:906:b291:: with SMTP id q17mr12217700ejz.56.1553077458253; Wed, 20 Mar 2019 03:24:18 -0700 (PDT) MIME-Version: 1.0 References: <79CC79B9-81AF-4563-BABE-429E6A57F476@bmalum.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazonses.com> <010201686fe5047f-ed14af85-2b25-4480-a62a-a893f062eedd-000000@eu-west-1.amazo> <1548182399.2864.0@smtp.migadu.com> <1552255580.21373.0@unrelenting.technology> <01000169992ab14c-aa38b2f1-3f57-4805-9d8a-fa92207e4c2d-000000@email.amazonses.com> In-Reply-To: <01000169992ab14c-aa38b2f1-3f57-4805-9d8a-fa92207e4c2d-000000@email.amazonses.com> From: =?UTF-8?Q?Micha=C5=82_Krawczyk?= Date: Wed, 20 Mar 2019 11:24:07 +0100 Message-ID: Subject: Re: ARM Graviton AWS Processor (AMI Image) To: Colin Percival Cc: Marcin Wojtas , Greg V , "Matushevsky, Alexander" , Martin Karrer , =?UTF-8?B?UmFmYcWCIEtvemlr?= , freebsd-arm@freebsd.org, "Tzalik, Guy" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 5264C8F92F X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=semihalf-com.20150623.gappssmtp.com header.s=20150623 header.b=jlhnMPav X-Spamd-Result: default: False [-4.03 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.996,0]; R_DKIM_ALLOW(-0.20)[semihalf-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arm@freebsd.org]; DMARC_NA(0.00)[semihalf.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[semihalf-com.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[ASPMX2.GOOGLEMAIL.com,ALT2.ASPMX.L.GOOGLE.com,ASPMX.L.GOOGLE.com,ALT1.ASPMX.L.GOOGLE.com,ASPMX3.GOOGLEMAIL.com]; RCPT_COUNT_SEVEN(0.00)[8]; RCVD_IN_DNSWL_NONE(0.00)[4.4.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; NEURAL_HAM_SHORT(-0.78)[-0.778,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-0.95)[ip: (-0.18), ipnet: 2a00:1450::/32(-2.38), asn: 15169(-2.11), country: US(-0.07)] X-BeenThere: freebsd-arm@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: "Porting FreeBSD to ARM processors." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 20 Mar 2019 10:24:21 -0000 Hi, we have solution for two of the issues with ENA, The double resource activation can be fixed by this patch: https://reviews.freebsd.org/D19655 And the issue with reloading module/reconfiguring it by using ifconfig can be fixed by using this patch: https://reviews.freebsd.org/D19641 Please check if that fixes it for you. Thanks, Micha=C5=82 =C5=9Br., 20 mar 2019 o 04:35 Colin Percival napisa= =C5=82(a): > > Hi ENA people, > > Any luck figuring out what was going on with the ENA driver here > with interrupt activation? > > Colin Percival > > On 3/11/19 1:35 AM, Marcin Wojtas wrote: > > +FreeBSD ENA maintainers > > > > W dniu pon., 11.03.2019 o 00:40 Greg V > > napisa=C5=82(a): > > > >> > >> On Tue, Jan 22, 2019 at 9:39 PM, Greg V > >> wrote: > >>> On Mon, Jan 21, 2019 at 1:11 PM, Martin Karrer > >>> wrote: > >>>> My question is if there are any plans yet to support the Graviton > >>>> ARM instances of AWS? > >>>> > >>>> We have a heavy load on FreeBSD and would also use the ARM > >>>> instances. Are there any other interested parties? > >>> > >>> I have tried this. It should work very well in theory, e.g. the > >>> network card driver (if_ena) compiles with no changes for aarch64, > >>> and in fact NetBSD has ported this driver and is up and running on > >>> these instances: https://dmesgd.nycbug.org/index.cgi?do=3Dview&id=3D4= 623 > >>> > >>> But my result with FreeBSD was: nothing on the console after > >>> loader.efi hands control to the kernel. > >>> [=E2=80=A6] > >> > >> Hello everyone, big update: > >> > >> FreeBSD/aarch64 on Amazon EC2 a1 (AWS Graviton) instances WORKS! > >> > >> https://dmesgd.nycbug.org/index.cgi?do=3Dview&id=3D4813 > >> > >> And you can try it (well, my -CURRENT build, NO WARRANTY etc) right no= w: > >> > >> ami-0c2829a0b82a62ca6 in eu-west-1 (Ireland) > >> > >> ----- > >> > >> So, what I had to do / what should be done / how others can help get > >> this into a finished state: > >> > >> 1. Serial console: > >> - I fixed it: https://reviews.freebsd.org/D19507 > >> - (I learned some things about UARTs and their support in FreeBSD, > >> should write a blog post about that) > >> > >> 2. aarch64 build configuration: > >> - if_ena network driver module should be enabled: > >> https://reviews.freebsd.org/D18372 > >> - NVMe driver should be enabled in the GENERIC kernel config (devic= e > >> nvme, device nvd) > >> - BTW, why not also go with hw.nvme.use_nvd=3D"0" by default on > >> aarch64, IIRC that was done on powerpc64 > >> > >> 3. VM image build system: > >> - GPT+EFI should be used (amd64 was GPT with no EFI, and aarch64 wa= s > >> MBR with EFI (???)): https://reviews.freebsd.org/D18371 > >> - bsdec2-image-upload --arm64 flag should be supported: included > >> above ^^ > >> - ec2.conf: amazon-ssm-agent shouldn't be installed when building > >> for aarch64 TARGET, since that's written in Go, and Go isn't ported to > >> FreeBSD/aarch64 yet: > >> > >> https://github.com/myfreeweb/freebsd/commit/5b530ebf7385d8320b9076cf84= f50aad01689bc > >> (untested patch, I actually used an interactive shell in between the > >> image build commands) > >> - qemu-aarch64-static should be used for preinstalling pkgs when > >> chrooting into the image: rough version included above ^^ > >> > >> 4. ENA (Elastic Network Adapter) driver: > >> - it works > >> - except there's something funky with interrupt activation, and it > >> hits panic("Attempt to double activation of resource id: %u\n", res_id= ) > >> (for the management IRQ) on boot, so I applied the obvious silly > >> workaround of "don't panic": > >> > >> https://github.com/myfreeweb/freebsd/commit/a7e7c6e48cdbdb0fdc6c4e0ba6= 3392262938e62c > >> - but still, it doesn't properly reactivate interrupts (and the box > >> becomes unreachable over the net) after going down and up again =E2=80= =94 > >> guess what does that on boot? dhclient applying the big jumbo MTU =E2= =80=94 > >> so I set dhclient.conf to reject MTU changes: > >> > >> https://github.com/myfreeweb/freebsd/commit/03ec4d417b0b4252285baaf4e2= 94cc6d8c870f7f > >> > >> > >> Would be great if someone familiar with interrupts and stuff could hel= p > >> debug the ena driver and make it work without these hacks :) > -- > Colin Percival > Security Officer Emeritus, FreeBSD | The power to serve > Founder, Tarsnap | www.tarsnap.com | Online backups for the truly paranoi= d