Nowe posty

Autor Wątek: QEMU KVM gpu passthrough brak obrazu  (Przeczytany 4656 razy)

Offline Mate_

  • Users
  • Nowy na forum
  • *
  • Wiadomości: 3
    • Zobacz profil
QEMU KVM gpu passthrough brak obrazu
« dnia: 2020-02-10, 16:25:38 »
Witam, mam problem z brakiem obrazu na maszynie wirtualnej / błędem gdy podaje ścieżkę do biosu karty.
Błąd:
Błąd podczas uruchamiania domeny: internal error: qemu unexpectedly closed the monitor: 2020-02-10T15:10:07.010111Z qemu-system-x86_64: -device vfio-pci,host=0000:0b:00.0,id=hostdev1,bus=pci.7,addr=0x0,rombar=1,romfile=/home/mate/bios.bin: failed to find romfile "/home/mate/bios.bin"

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 111, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 66, in newfn
    ret = fn(self, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1279, in startup
    self._backend.create()
  File "/usr/lib/python3.8/site-packages/libvirt.py", line 1136, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-02-10T15:10:07.010111Z qemu-system-x86_64: -device vfio-pci,host=0000:0b:00.0,id=hostdev1,bus=pci.7,addr=0x0,rombar=1,romfile=/home/mate/bios.bin: failed to find romfile "/home/mate/bios.bin"

 ██████████████████  ████████     mate@mate-pc
 ██████████████████  ████████     OS: Manjaro 18.1.5 Juhraya
 ██████████████████  ████████     Kernel: x86_64 Linux 5.5.2-1-MANJARO
 ██████████████████  ████████     Uptime: 30m
 ████████            ████████     Packages: 1388
 ████████  ████████  ████████     Shell: zsh 5.7.1
 ████████  ████████  ████████     Resolution: 2560x1440
 ████████  ████████  ████████     DE: KDE 5.66.0 / Plasma 5.17.5
 ████████  ████████  ████████     WM: KWin
 ████████  ████████  ████████     GTK Theme: Nordic [GTK2/3]
 ████████  ████████  ████████     Icon Theme: Vimix-dark
 ████████  ████████  ████████     Disk: 1,1T / 1,2T (90%)
 ████████  ████████  ████████     CPU: AMD Ryzen 5 1600 Six-Core @ 12x 4GHz
 ████████  ████████  ████████     GPU: GeForce GT 710
                                  RAM: 4680MiB / 15943MiB

GPU gościa: GTX 1070, gpu hosta GT 710. GT 710 jest w pierwszym sloecie PCI-e, 1070 w drugim, wcześniej próbowałem odwrotnej konfiguracji.
Płyta główna: ASUS X370 PRO

#!/bin/bash
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;
Cytuj
IOMMU Group 0:
        00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 1:
        00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 10:
        00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 11:
        00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 12:
        00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
        00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13:
        00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
        00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
        00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
        00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
        00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
        00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
        00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
        00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU Group 14:
        01:00.0 Non-Volatile memory controller [0108]: Lite-On Technology Corporation M8Pe Series NVMe SSD [14a4:22f1] (rev 01)
IOMMU Group 15:
        02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset USB 3.1 xHCI Controller [1022:43b9] (rev 02)
        02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset SATA Controller [1022:43b5] (rev 02)
        02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset PCIe Upstream Port [1022:43b0] (rev 02)
        03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
        03:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
        03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
        03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
        03:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
        03:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
        07:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1143 USB 3.1 Host Controller [1b21:1343]
        08:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 16:
        0a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
        0a:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU Group 17:
        0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
        0b:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 18:
        0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
IOMMU Group 19:
        0c:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU Group 2:
        00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 20:
        0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]
IOMMU Group 21:
        0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
IOMMU Group 22:
        0d:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 23:
        0d:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]
IOMMU Group 3:
        00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 4:
        00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 5:
        00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 6:
        00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 7:
        00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 8:
        00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 9:
        00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]

~ lspci -nnk -d 10de:1b81     
0b:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] GP104 [GeForce GTX 1070] [1462:3302]
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau, nvidia_drm, nvidia

~ lspci -nnk -d 10de:10f0
0b:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] GP104 High Definition Audio Controller [1462:3302]
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

dmesg | grep -i vfio
[    1.496993] VFIO - User Level meta-driver version: 0.3
[    1.500736] vfio-pci 0000:0b:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[    1.515367] vfio_pci: add [10de:1b81[ffffffff:ffffffff]] class 0x000000/00000000
[    1.532136] vfio_pci: add [10de:10f0[ffffffff:ffffffff]] class 0x000000/00000000
[    1.553494] vfio-pci 0000:0b:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[   66.908357] vfio-pci 0000:0b:00.0: enabling device (0000 -> 0003)
[   66.908606] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[   66.925017] vfio-pci 0000:0b:00.1: enabling device (0000 -> 0002)
[  175.216340] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[  347.414442] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[  470.981870] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[  491.371976] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[  513.148747] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[  580.079597] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[  598.579816] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[  613.739947] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[  626.916761] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[  738.681050] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[ 1047.926562] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[ 1079.386961] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[ 1121.890853] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[ 1761.799815] vfio-pci 0000:0b:00.0: vfio_ecap_init: hiding ecap 0x19@0x900

dmesg | grep -i -e DMAR -e IOMMU
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.5-x86_64 root=UUID=74ab8c47-0649-4768-a42a-433c9b4fa920 rw quiet kvm.ignore_msrs=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1
[    0.138452] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-5.5-x86_64 root=UUID=74ab8c47-0649-4768-a42a-433c9b4fa920 rw quiet kvm.ignore_msrs=1 amd_iommu=on iommu=pt kvm_amd.npt=1 kvm_amd.avic=1
[    0.921658] iommu: Default domain type: Passthrough (set via kernel command line)
[    1.116426] pci 0000:00:00.2: AMD-Vi: IOMMU performance counters supported
[    1.116491] pci 0000:00:01.0: Adding to iommu group 0
[    1.116511] pci 0000:00:01.1: Adding to iommu group 1
[    1.116526] pci 0000:00:01.3: Adding to iommu group 2
[    1.116542] pci 0000:00:02.0: Adding to iommu group 3
[    1.116562] pci 0000:00:03.0: Adding to iommu group 4
[    1.116578] pci 0000:00:03.1: Adding to iommu group 5
[    1.116592] pci 0000:00:03.2: Adding to iommu group 6
[    1.116610] pci 0000:00:04.0: Adding to iommu group 7
[    1.116628] pci 0000:00:07.0: Adding to iommu group 8
[    1.116646] pci 0000:00:07.1: Adding to iommu group 9
[    1.116664] pci 0000:00:08.0: Adding to iommu group 10
[    1.116679] pci 0000:00:08.1: Adding to iommu group 11
[    1.116699] pci 0000:00:14.0: Adding to iommu group 12
[    1.116712] pci 0000:00:14.3: Adding to iommu group 12
[    1.116760] pci 0000:00:18.0: Adding to iommu group 13
[    1.116772] pci 0000:00:18.1: Adding to iommu group 13
[    1.116783] pci 0000:00:18.2: Adding to iommu group 13
[    1.116795] pci 0000:00:18.3: Adding to iommu group 13
[    1.116806] pci 0000:00:18.4: Adding to iommu group 13
[    1.116818] pci 0000:00:18.5: Adding to iommu group 13
[    1.116829] pci 0000:00:18.6: Adding to iommu group 13
[    1.116842] pci 0000:00:18.7: Adding to iommu group 13
[    1.116860] pci 0000:01:00.0: Adding to iommu group 14
[    1.116894] pci 0000:02:00.0: Adding to iommu group 15
[    1.116913] pci 0000:02:00.1: Adding to iommu group 15
[    1.116931] pci 0000:02:00.2: Adding to iommu group 15
[    1.116940] pci 0000:03:00.0: Adding to iommu group 15
[    1.116950] pci 0000:03:02.0: Adding to iommu group 15
[    1.116959] pci 0000:03:03.0: Adding to iommu group 15
[    1.116967] pci 0000:03:04.0: Adding to iommu group 15
[    1.116976] pci 0000:03:06.0: Adding to iommu group 15
[    1.116984] pci 0000:03:07.0: Adding to iommu group 15
[    1.116998] pci 0000:07:00.0: Adding to iommu group 15
[    1.117013] pci 0000:08:00.0: Adding to iommu group 15
[    1.117041] pci 0000:0a:00.0: Adding to iommu group 16
[    1.117060] pci 0000:0a:00.1: Adding to iommu group 16
[    1.117089] pci 0000:0b:00.0: Adding to iommu group 17
[    1.117109] pci 0000:0b:00.1: Adding to iommu group 17
[    1.117126] pci 0000:0c:00.0: Adding to iommu group 18
[    1.117143] pci 0000:0c:00.2: Adding to iommu group 19
[    1.117159] pci 0000:0c:00.3: Adding to iommu group 20
[    1.117175] pci 0000:0d:00.0: Adding to iommu group 21
[    1.117191] pci 0000:0d:00.2: Adding to iommu group 22
[    1.117210] pci 0000:0d:00.3: Adding to iommu group 23
[    1.117429] pci 0000:00:00.2: AMD-Vi: Found IOMMU cap 0x40
[    1.117795] perf/amd_iommu: Detected AMD IOMMU #0 (2 banks, 4 counters/bank).
[    1.161096] AMD-Vi: AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>

konfiguracja maszyny:
https://pastebin.com/GJCXFrMp

Powiedzmy trochę ponad rok temu cały proces się udał po kilku dniowej batalii, tylko możliwe, że zamiast 1070 używałem 1050Ti. Proszę o pomoc bo już nie mam pomysłów co jeszcze zmienić.


Offline Paweł Kraszewski

  • Administrator
  • Guru
  • *****
  • Wiadomości: 3047
  • Lenistwo jest matką potrzeby = babcią wynalazku
    • Zobacz profil
Odp: QEMU KVM gpu passthrough brak obrazu
« Odpowiedź #1 dnia: 2020-02-10, 20:30:28 »
Ummm....

failed to find romfile "/home/mate/bios.bin"

?
Paweł Kraszewski
~Arch/Void/Gentoo/FreeBSD/OpenBSD/Specjalizowane customy

Offline Mate_

  • Users
  • Nowy na forum
  • *
  • Wiadomości: 3
    • Zobacz profil
Odp: QEMU KVM gpu passthrough brak obrazu
« Odpowiedź #2 dnia: 2020-02-10, 20:47:03 »



Plik z biosem jest, uprawnienia są. Sam bios wyeksportowałem nvflash'em i zpatchowałem tym https://github.com/Matoking/NVIDIA-vBIOS-VFIO-Patcher narzędziem.

// edit udało mi się rozwiązać problem z odczytem biosu ( przeniosłem plik do /var/lib/libvirt/ ), ale dalej nie mam obrazu z gpu gościa :(
« Ostatnia zmiana: 2020-02-10, 21:26:24 wysłana przez Mate_ »

Offline Paweł Kraszewski

  • Administrator
  • Guru
  • *****
  • Wiadomości: 3047
  • Lenistwo jest matką potrzeby = babcią wynalazku
    • Zobacz profil
Odp: QEMU KVM gpu passthrough brak obrazu
« Odpowiedź #3 dnia: 2020-02-11, 07:21:25 »
To jeszcze zadbać, żeby absolutnie żadna usługa/moduł nie były podpięte pod kartę gościa podczas startu VM (czyli driver NV, driver dźwięku przez HDMI, itp)
Paweł Kraszewski
~Arch/Void/Gentoo/FreeBSD/OpenBSD/Specjalizowane customy

Offline Mate_

  • Users
  • Nowy na forum
  • *
  • Wiadomości: 3
    • Zobacz profil
Odp: QEMU KVM gpu passthrough brak obrazu
« Odpowiedź #4 dnia: 2020-02-11, 17:05:06 »
Udało znaleźć rozwiązanie, podczas tworzenia wirtualki zapędziłem się i nie zmieniłem z domyślnego BIOS na UEFI.