From owner-freebsd-virtualization@freebsd.org Thu Nov 16 20:52:06 2017 Return-Path: Delivered-To: freebsd-virtualization@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 00390DE8E3C for ; Thu, 16 Nov 2017 20:52:06 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf0-f41.google.com (mail-lf0-f41.google.com [209.85.215.41]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A668969FBA for ; Thu, 16 Nov 2017 20:52:05 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf0-f41.google.com with SMTP id 73so350448lfu.10 for ; Thu, 16 Nov 2017 12:52:05 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=vJWOc99V9FgL1kEfB6FcbSPLvgtfhigIOfc5ygzwj9U=; b=p0yjJl7uBgVOJzSLBjIywLfOYqhWzp6khPV8v/FEvOihOv29PQA2TF1sgbAg4vOeXc xOFCkJf7xOM8n4GYAu5hV8OmHPzK4E4VyKI7lRB4ydeWGIy0j9tulzJTHP88m/OcwDTD ryQiR8rzMWhgpaWRCRFyuDOQE3zXKleVVsQCJ+T0axSUn6bOLMAt5LqbZ/rzY1n7Kty/ 9MjU97dV52YuiqnQFR8IDoWD/rOqtXAJgqGkkM0XORd77fNMavYkzsNsHxmVwfVtS3p4 sSEt/Jf3VvrnqGgPlTqbBUaikAv2HpIVLohqBM6scsps2iOtxyWxanefOi+ex/p6X1G0 j4mw== X-Gm-Message-State: AJaThX66h/04+tclvaZHDHT9AVZeGO5yq+mx0Upwz+oYtIzDZ2OIrm9A Zkm0aSm/JhSTZmxhZol8MsrVk4uN X-Google-Smtp-Source: AGs4zMaKDmmMvKlPiiRrLkqnKdQZ8cn+i5ajbZBUlxw7Qy+y+jh/GSN1YddCaKQQXi9Cuc13eOFmNg== X-Received: by 10.25.167.213 with SMTP id q204mr32794lfe.219.1510865517526; Thu, 16 Nov 2017 12:51:57 -0800 (PST) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id a12sm349547ljd.19.2017.11.16.12.51.56 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 16 Nov 2017 12:51:56 -0800 (PST) From: Andriy Gapon Subject: Re: problem with pass-through on amd To: Anish Cc: "freebsd-virtualization@freebsd.org" References: Message-ID: Date: Thu, 16 Nov 2017 22:51:55 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-BeenThere: freebsd-virtualization@freebsd.org X-Mailman-Version: 2.1.25 Precedence: list List-Id: "Discussion of various virtualization techniques FreeBSD supports." List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 16 Nov 2017 20:52:06 -0000 On 14/11/2017 06:22, Anish wrote: > [root@ryzen /home/anish/FreeBSD/head]# vmstat -ia |grep ivh > irq256: ivhd0:fault                    0          0 > irq257: ivhd1:fault                    0          0 After I fixed the MSI setup problem that I described in the previous email I was finally able to see the real problem. Now I have: irq263: ivhd0:fault 9 0 dev.ivhd.0.event_tail: 240 dev.ivhd.0.event_head: 240 dev.ivhd.0.event_intr_count: 9 I've got a bunch of messages like these in the log: [IO_PAGE_FAULT EVT: devId:0xa01 DomId:0x2 Addr:0x70f70400x30] [IO_PAGE_FAULT EVT: devId:0xa01 DomId:0x2 Addr:0x2fdf0400x30] (BTW, there seems to be a missing space before 0x30) Now it's obvious what the problem is. The controller has a RID of 0xa00 (its PCI locator is 10:0:0) but the requests are coming from 0xa01. The card actually has another PCI device at that location (10:0:1), it's a vestigial IDE controller (in a sense that it is not connected to any ports, so it cannot really provide any functionality). I've passed both of the PCI device to bhyve and everything started working. Thanks! I've googled a little bit and it seems that this is not an uncommon problem. Linux has a bunch of quirks for this and similar kinds of problems (they call it PCI or DMA aliasing): http://elixir.free-electrons.com/linux/v4.8.16/source/drivers/pci/quirks.c#L3701 The comments say that sometimes requests come from devices that do not exist at all (or hidden beyond non-transparent bridges). An interesting new world for me :-) By the way, my understanding is that the alias IVHD entries in IVRS are designed to report such aliasing issues when they are known to the platform. But it appears to me that FreeBSD currently ignores those aliases. Also, it looks like we are not making any use of IVHD entries beyond printing them. Based on the specification I think that we should use IVHD flags to set up appropriate bits in the corresponding device table entries. But we are not doing that. That's probably okay since we do not support the pass-though for devices with complex properties anyway. -- Andriy Gapon