Kernel panic upon resume of Linux/KVM VM (OpenBSD 6.6)

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Kernel panic upon resume of Linux/KVM VM (OpenBSD 6.6)

Andreas Rottmann
Hi!

>Synopsis: panic: pvclock0: unstable result on stable clock
>Category: virtualization
>Environment:
        System      : OpenBSD 6.6
        Details     : OpenBSD 6.6 (GENERIC.MP) #372: Sat Oct 12 10:56:27 MDT 2019
                         [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

        Architecture: OpenBSD.amd64
        Machine     : amd64
>Description:

I've just experienced a kernel panic when resuming my laptop from
suspend-to-RAM while my OpenBSD 6.6 VM was running; the first few lines
of the crash read like this:

panic: pvclock0: unstable result on stable clock
Stopped at      db_enter+0x10:  popq    %rbp
    TID    PID    UID     PRFLAGS     PFLAGS  CPU  COMMAND
db_enter() at db_enter+0x10
panic() at panic+0x128
pvclock_get_timecount(ffffffff81f14360) at pvclock_get_timecount+0xc2

The full ddb session, including backtraces for both cores, and the `ps`
output is attached as `ddb.txt`.



panic: pvclock0: unstable result on stable clock
Stopped at      db_enter+0x10:  popq    %rbp          
    TID    PID    UID     PRFLAGS     PFLAGS  CPU  COMMAND
db_enter() at db_enter+0x10                              
panic() at panic+0x128    
pvclock_get_timecount(ffffffff81f14360) at pvclock_get_timecount+0xc2
tc_windup(0,0,0) at tc_windup+0xd0                                  
tc_ticktock() at tc_ticktock+0x52
hardclock(ffff800014a3c360) at hardclock+0x12c
lapic_clockintr(0,0) at lapic_clockintr+0x3f  
Xresume_lapic_ltimer(0,0,1388,0,ffff800000068020,ffffffff81f1b6f8) at Xresume_l
apic_ltimer+0x26                                                              
acpicpu_idle() at acpicpu_idle+0x261
sched_idle(ffffffff81f1aff0) at sched_idle+0x225
end trace frame: 0x0, count: 5                  
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports.  Insufficient info makes it difficult to find and fix bugs.
ddb{0}> machine ddbcpu 1
Stopped at      x86_ipi_db+0x12:        leave
x86_ipi_db(ffff800014918ff0) at x86_ipi_db+0x12
x86_ipi_handler() at x86_ipi_handler+0x80      
Xresume_lapic_ipi(0,0,1388,0,ffff800000068040,ffff8000149196f8) at Xresume_lapi
c_ipi+0x23                                                                    
acpicpu_idle() at acpicpu_idle+0x261
sched_idle(ffff800014918ff0) at sched_idle+0x225
end trace frame: 0x0, count: 10
ddb{1}> trace                  
x86_ipi_db(ffff800014918ff0) at x86_ipi_db+0x12
x86_ipi_handler() at x86_ipi_handler+0x80      
Xresume_lapic_ipi(0,0,1388,0,ffff800000068040,ffff8000149196f8) at Xresume_lapi
c_ipi+0x23                                                                    
acpicpu_idle() at acpicpu_idle+0x261
sched_idle(ffff800014918ff0) at sched_idle+0x225
end trace frame: 0x0, count: -5
ddb{0}> ps                      
   PID     TID   PPID    UID  S       FLAGS  WAIT          COMMAND
 39708  150541      1      0  3    0x100083  ttyin         getty  
 38611   41016      1      0  3    0x100083  ttyin         getty
 94774  269748      1      0  3    0x100083  ttyin         getty
 44011  319819      1      0  3    0x100083  ttyin         getty
 91206   96891      1      0  3    0x100083  ttyin         getty
 33996  335998      1      0  3    0x100083  ttyin         getty
 13087  186233      1      0  3    0x100098  poll          cron
 49267  356024      1     99  3    0x100090  poll          sndiod
 29103   20383      1    110  3    0x100090  poll          sndiod
 11112  130700    558     95  3    0x100092  kqread        smtpd
 60221  104818    558    103  3    0x100092  kqread        smtpd
 91315   98875    558     95  3    0x100092  kqread        smtpd
 44766  437017    558     95  3    0x100092  kqread        smtpd
 92451  425209    558     95  3    0x100092  kqread        smtpd
 57317  247401    558     95  3    0x100092  kqread        smtpd
   558   56707      1      0  3    0x100080  kqread        smtpd
 40884  156906      1      0  3        0x80  select        sshd
 93191  263044      1      0  3    0x100080  poll          ntpd
 70824   87212  32842     83  3    0x100092  poll          ntpd
 32842  205398      1     83  3    0x100092  poll          ntpd
 37959  288486  90259     74  3    0x100092  bpf           pflogd
 90259  164238      1      0  3        0x80  netio         pflogd
 30903  489022  63084     73  3    0x100090  kqread        syslogd
 63084  188163      1      0  3    0x100082  netio         syslogd
 30724  510384   9200    115  3    0x100092  kqread        slaacd
 60465  454290   9200    115  3    0x100092  kqread        slaacd
  9200  510960      1      0  3    0x100080  kqread        slaacd
 47096  304024      0      0  3     0x14200  pgzero        zerothread
 21877  137120      0      0  3     0x14200  aiodoned      aiodoned  
 32349  212694      0      0  3     0x14200  syncer        update  
 32681   43875      0      0  3     0x14200  cleaner       cleaner
 26707  437544      0      0  3     0x14200  reaper        reaper
 49598  303545      0      0  3     0x14200  pgdaemon      pagedaemon
 46418    5967      0      0  3     0x14200  bored         crynlk    
 54379  131524      0      0  3     0x14200  bored         crypto
 18111  464847      0      0  3     0x14200  bored         viomb
 56135   56594      0      0  3     0x14200  usbtsk        usbtask
 34167  241492      0      0  3     0x14200  usbatsk       usbatsk
 46718  141673      0      0  3  0x40014200  acpi0         acpi0  
 39849  211312      0      0  7  0x40014200                idle1
 29427  339100      0      0  3     0x14200  bored         softnet
 42678  320450      0      0  3     0x14200  bored         systqmp
 35573  456553      0      0  3     0x14200  bored         systq  
 68462  145843      0      0  3  0x40014200  bored         softclock
*39924  454465      0      0  7  0x40014200                idle0    
 85955  382369      0      0  3     0x14200  bored         smr  
     1  418871      0      0  3        0x82  wait          init
     0       0     -1      0  3     0x10200  scheduler     swapper


This is a fresh install of OpenBSD 6.6, just a few packages from ports
installed, no extra daemons. The host system is running:

- Debian 11 (aka "buster", and currently "testing")
- qemu-system-x86 1:4.1-1+b2
- Linux 5.2.0-3-amd64 #1 SMP Debian 5.2.17-1 (2019-09-26) x86_64
  GNU/Linux

This is a Thinkpad T460, running an i5-6200U CPU; I've attached the
output from `lscpu` on the host (lspcu.txt). The VM has been given 2
cores and 1GiB of RAM, the libvirt XML description of the VM is attached
as well (obsd-dev.xml).


Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   39 bits physical, 48 bits virtual
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              2
Core(s) per socket:              2
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           78
Model name:                      Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
Stepping:                        3
CPU MHz:                         588.091
CPU max MHz:                     2800.0000
CPU min MHz:                     400.0000
BogoMIPS:                        4800.00
Virtualization:                  VT-x
L1d cache:                       64 KiB
L1i cache:                       64 KiB
L2 cache:                        512 KiB
L3 cache:                        3 MiB
NUMA node0 CPU(s):               0-3
Vulnerability L1tf:              Mitigation; PTE Inversion; VMX conditional cache flushes, SMT vulnerable
Vulnerability Mds:               Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Meltdown:          Mitigation; PTI
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full generic retpoline, IBPB conditional, IBRS_FW, STIBP conditional, RSB filling
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb invpcid_single pti ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt intel_pt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp md_clear flush_l1d


>How-To-Repeat:
        This seems to happen only sometimes after the host was suspended
        and woken up again; I experienced it twice in a row, after an
        over-night suspend, but failed ro reproduce after that. I'll try
        to have the VM running whenever I suspend, and see if I can
        trigger it again.

dmesg:
OpenBSD 6.6 (GENERIC.MP) #372: Sat Oct 12 10:56:27 MDT 2019
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 1056804864 (1007MB)
avail mem = 1012129792 (965MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf5880 (10 entries)
bios0: vendor SeaBIOS version "1.12.0-1" date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S5
acpi0: tables DSDT FACP APIC
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel Core Processor (Skylake, IBRS), 2400.39 MHz, 06-5e-03
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1000MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel Core Processor (Skylake, IBRS), 2400.04 MHz, 06-5e-03
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,TSC_ADJUST,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,RDSEED,ADX,SMAP,CLFLUSHOPT,UMIP,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu1: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu1: smt 0, core 0, package 1
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: _OSC failed
acpicmos0 at acpi0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"QEMU0002" at acpi0 not configured
"ACPI0010" at acpi0 not configured
cpu0: using VERW MDS workaround
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0: unstable clock
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
cd0(pciide0:0:0): using PIO mode 4, DMA mode 2
pciide0: channel 1 disabled (no drives)
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Red Hat QXL Video" rev 0x04
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio0: address 52:54:00:82:57:22
virtio0: msix shared
azalia0 at pci0 dev 4 function 0 "Intel 82801FB HD Audio" rev 0x01: apic 0 int 11
azalia0: No codecs found
uhci0 at pci0 dev 5 function 0 "Intel 82801I USB" rev 0x03: apic 0 int 10
uhci1 at pci0 dev 5 function 1 "Intel 82801I USB" rev 0x03: apic 0 int 10
uhci2 at pci0 dev 5 function 2 "Intel 82801I USB" rev 0x03: apic 0 int 11
ehci0 at pci0 dev 5 function 7 "Intel 82801I USB" rev 0x03: apic 0 int 11
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
virtio1 at pci0 dev 6 function 0 "Qumranet Virtio Storage" rev 0x00
vioblk0 at virtio1
scsibus2 at vioblk0: 2 targets
sd0 at scsibus2 targ 0 lun 0: <VirtIO, Block Device, >
sd0: 20480MB, 512 bytes/sector, 41943040 sectors
virtio1: msix shared
virtio2 at pci0 dev 7 function 0 "Qumranet Virtio Memory Balloon" rev 0x00
viomb0 at virtio2
virtio2: apic 0 int 11
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
usb1 at uhci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb2 at uhci1: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci2: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "Intel UHCI root hub" rev 1.00/1.00 addr 1
vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation)
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (cb02cbb1a7da4429.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
fd0 at fdc0 drive 1: density unknown

usbdevs:
Controller /dev/usb0:
addr 01: 8086:0000 Intel, EHCI root hub
         high speed, self powered, config 1, rev 1.00
         driver: uhub0
Controller /dev/usb1:
addr 01: 8086:0000 Intel, UHCI root hub
         full speed, self powered, config 1, rev 1.00
         driver: uhub1
Controller /dev/usb2:
addr 01: 8086:0000 Intel, UHCI root hub
         full speed, self powered, config 1, rev 1.00
         driver: uhub2
Controller /dev/usb3:
addr 01: 8086:0000 Intel, UHCI root hub
         full speed, self powered, config 1, rev 1.00
         driver: uhub3

pcidump:
Domain /dev/pci0:
 0:0:0: Intel 82441FX
        0x0000: Vendor ID: 8086, Product ID: 1237
        0x0004: Command: 0103, Status: 0000
        0x0008: Class: 06 Bridge, Subclass: 00 Host,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0000: 12378086 00000103 06000002 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 00000000
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 80000000 11111000 33111111
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 000a0000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:1:0: Intel 82371SB ISA
        0x0000: Vendor ID: 8086, Product ID: 7000
        0x0004: Command: 0103, Status: 0200
        0x0008: Class: 06 Bridge, Subclass: 01 ISA,
                Interface: 00, Revision: 00
        0x000c: BIST: 00, Header Type: 80, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0000: 70008086 02000103 06010000 00800000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 00000000
        0x0040: 00000000 00000000 00000000 0003004d
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 0b0b0a0a 00000000 00000200 00000000
        0x0070: 00000080 0c0c0000 00000002 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000008 00000000 0000000f 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:1:1: Intel 82371SB IDE
        0x0000: Vendor ID: 8086, Product ID: 7010
        0x0004: Command: 0107, Status: 0280
        0x0008: Class: 01 Mass Storage, Subclass: 01 IDE,
                Interface: 80, Revision: 00
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR io addr: 0x0000c140/0x0010
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0000: 70108086 02800107 01018000 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 0000c141 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 00000000
        0x0040: 0000a307 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:1:3: Intel 82371AB Power
        0x0000: Vendor ID: 8086, Product ID: 7113
        0x0004: Command: 0103, Status: 0280
        0x0008: Class: 06 Bridge, Subclass: 80 Miscellaneous,
                Interface: 00, Revision: 03
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 09 Min Gnt: 00 Max Lat: 00
        0x0000: 71138086 02800103 06800003 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 00000109
        0x0040: 00000601 00000000 00000000 00000000
        0x0050: 00000000 00000000 02000000 10000000
        0x0060: 60000000 08000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000001 00000000 00000000 00000000
        0x0090: 00000701 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00090000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:2:0: Red Hat QXL Video
        0x0000: Vendor ID: 1b36, Product ID: 0100
        0x0004: Command: 0107, Status: 0000
        0x0008: Class: 03 Display, Subclass: 00 VGA,
                Interface: 00, Revision: 04
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR mem 32bit addr: 0xf4000000/0x04000000
        0x0014: BAR mem 32bit addr: 0xf8000000/0x04000000
        0x0018: BAR mem 32bit addr: 0xfc054000/0x00002000
        0x001c: BAR io addr: 0x0000c080/0x0020
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: fc040000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0a Min Gnt: 00 Max Lat: 00
        0x0000: 01001b36 00000107 03000004 00000000
        0x0010: f4000000 f8000000 fc054000 0000c081
        0x0020: 00000000 00000000 00000000 11001af4
        0x0030: fc040000 00000000 00000000 0000010a
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:3:0: Qumranet Virtio Network
        0x0000: Vendor ID: 1af4, Product ID: 1000
        0x0004: Command: 0107, Status: 0010
        0x0008: Class: 02 Network, Subclass: 00 Ethernet,
                Interface: 00, Revision: 00
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR io addr: 0x0000c0a0/0x0020
        0x0014: BAR mem 32bit addr: 0xfc056000/0x00001000
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR mem prefetchable 64bit addr: 0x00000000febf4000/0x00004000
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 0001
        0x0030: Expansion ROM Base Address: fc000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
        0x0098: Capability 0x11: Extended Message Signalled Interrupts (MSI-X)
                Enabled: yes; table size 3 (BAR 1:0)
        0x0084: Capability 0x09: Vendor Specific
        0x0070: Capability 0x09: Vendor Specific
        0x0060: Capability 0x09: Vendor Specific
        0x0050: Capability 0x09: Vendor Specific
        0x0040: Capability 0x09: Vendor Specific
        0x0000: 10001af4 00100107 02000000 00000000
        0x0010: 0000c0a1 fc056000 00000000 00000000
        0x0020: febf400c 00000000 00000000 00011af4
        0x0030: fc000000 00000098 00000000 0000010b
        0x0040: 01100009 00000004 00000000 00001000
        0x0050: 03104009 00000004 00001000 00001000
        0x0060: 04105009 00000004 00002000 00001000
        0x0070: 02146009 00000004 00003000 00001000
        0x0080: 00000004 05147009 00000000 00000000
        0x0090: 00000000 00000000 80028411 00000001
        0x00a0: 00000801 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:4:0: Intel 82801FB HD Audio
        0x0000: Vendor ID: 8086, Product ID: 2668
        0x0004: Command: 0107, Status: 0010
        0x0008: Class: 04 (unknown), Subclass: 03 (unknown),
                Interface: 00, Revision: 01
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR mem 32bit addr: 0xfc050000/0x00004000
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
        0x0060: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: no
        0x0000: 26688086 00100107 04030001 00000000
        0x0010: fc050000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 11001af4
        0x0030: 00000000 00000060 00000000 0000010b
        0x0040: 00000001 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00800005 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:5:0: Intel 82801I USB
        0x0000: Vendor ID: 8086, Product ID: 2934
        0x0004: Command: 0107, Status: 0000
        0x0008: Class: 0c Serial Bus, Subclass: 03 USB,
                Interface: 00, Revision: 03
        0x000c: BIST: 00, Header Type: 80, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR io addr: 0x0000c0c0/0x0020
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0a Min Gnt: 00 Max Lat: 00
        0x0000: 29348086 00000107 0c030003 00800000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 0000c0c1 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 0000010a
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000010 00000000 00010000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00002000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:5:1: Intel 82801I USB
        0x0000: Vendor ID: 8086, Product ID: 2935
        0x0004: Command: 0107, Status: 0000
        0x0008: Class: 0c Serial Bus, Subclass: 03 USB,
                Interface: 00, Revision: 03
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR io addr: 0x0000c0e0/0x0020
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 02 Line: 0a Min Gnt: 00 Max Lat: 00
        0x0000: 29358086 00000107 0c030003 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 0000c0e1 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 0000020a
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000010 00000000 00010000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00002000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:5:2: Intel 82801I USB
        0x0000: Vendor ID: 8086, Product ID: 2936
        0x0004: Command: 0107, Status: 0000
        0x0008: Class: 0c Serial Bus, Subclass: 03 USB,
                Interface: 00, Revision: 03
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR io addr: 0x0000c100/0x0020
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 03 Line: 0b Min Gnt: 00 Max Lat: 00
        0x0000: 29368086 00000107 0c030003 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 0000c101 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 0000030b
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000010 00000000 00010000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00002000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:5:7: Intel 82801I USB
        0x0000: Vendor ID: 8086, Product ID: 293a
        0x0004: Command: 0107, Status: 0000
        0x0008: Class: 0c Serial Bus, Subclass: 03 USB,
                Interface: 20, Revision: 03
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR mem 32bit addr: 0xfc057000/0x00001000
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 1100
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 04 Line: 0b Min Gnt: 00 Max Lat: 00
        0x0000: 293a8086 00000107 0c032003 00000000
        0x0010: fc057000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 11001af4
        0x0030: 00000000 00000000 00000000 0000040b
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00002020 00000000 00000001 c0000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:6:0: Qumranet Virtio Storage
        0x0000: Vendor ID: 1af4, Product ID: 1001
        0x0004: Command: 0107, Status: 0010
        0x0008: Class: 01 Mass Storage, Subclass: 00 SCSI,
                Interface: 00, Revision: 00
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR io addr: 0x0000c000/0x0080
        0x0014: BAR mem 32bit addr: 0xfc058000/0x00001000
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR mem prefetchable 64bit addr: 0x00000000febf8000/0x00004000
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 0002
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0a Min Gnt: 00 Max Lat: 00
        0x0098: Capability 0x11: Extended Message Signalled Interrupts (MSI-X)
                Enabled: yes; table size 2 (BAR 1:0)
        0x0084: Capability 0x09: Vendor Specific
        0x0070: Capability 0x09: Vendor Specific
        0x0060: Capability 0x09: Vendor Specific
        0x0050: Capability 0x09: Vendor Specific
        0x0040: Capability 0x09: Vendor Specific
        0x0000: 10011af4 00100107 01000000 00000000
        0x0010: 0000c001 fc058000 00000000 00000000
        0x0020: febf800c 00000000 00000000 00021af4
        0x0030: 00000000 00000098 00000000 0000010a
        0x0040: 01100009 00000004 00000000 00001000
        0x0050: 03104009 00000004 00001000 00001000
        0x0060: 04105009 00000004 00002000 00001000
        0x0070: 02146009 00000004 00003000 00001000
        0x0080: 00000004 05147009 00000000 00000000
        0x0090: 00000000 00000000 80018411 00000001
        0x00a0: 00000801 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:7:0: Qumranet Virtio Memory Balloon
        0x0000: Vendor ID: 1af4, Product ID: 1002
        0x0004: Command: 0107, Status: 0010
        0x0008: Class: 00 Prehistoric, Subclass: ff (unknown),
                Interface: 00, Revision: 00
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR io addr: 0x0000c120/0x0020
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR mem prefetchable 64bit addr: 0x00000000febfc000/0x00004000
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1af4 Product ID: 0005
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
        0x0084: Capability 0x09: Vendor Specific
        0x0070: Capability 0x09: Vendor Specific
        0x0060: Capability 0x09: Vendor Specific
        0x0050: Capability 0x09: Vendor Specific
        0x0040: Capability 0x09: Vendor Specific
        0x0000: 10021af4 00100107 00ff0000 00000000
        0x0010: 0000c121 00000000 00000000 00000000
        0x0020: febfc00c 00000000 00000000 00051af4
        0x0030: 00000000 00000084 00000000 0000010b
        0x0040: 01100009 00000004 00000000 00001000
        0x0050: 03104009 00000004 00001000 00001000
        0x0060: 04105009 00000004 00002000 00001000
        0x0070: 02146009 00000004 00003000 00001000
        0x0080: 00000004 05147009 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000

acpidump:
begin-base64 644 APIC.3
QVBJQ4AAAAAB2kJPQ0hTIEJYUENBUElDAQAAAEJYUEMBAAAAAADg/gEAAAAACAAAAQAAAAAIAQEB
AAAAAQwAAAAAwP4AAAAAAgoAAAIAAAAAAAIKAAUFAAAADQACCgAJCQAAAA0AAgoACgoAAAANAAIK
AAsLAAAADQAEBv8AAAE=
====
begin-base64 644 DSDT.2
RFNEVOoTAAABBUJPQ0hTIEJYUENEU0RUAQAAAEJYUEMBAAAAEEkEXABbgERCR18BCwIEAVuBC0RC
R18BREJHQggULERCVUcBmGhglmBgdIdgAWFwAGKiEJViYXCDiGBiAERCR0J1YnAKCkRCR0IQIl9T
Ql9bghtQQ0kwCF9ISUQMQdAKAwhfQURSAAhfVUlEARBNCF9TQl9bgkUISFBFVAhfSElEDEHQAQMI
X1VJRABbgEhQVE0ADAAA0P4LAARbgRBIUFRNE1ZFTkQgUFJEXyAUNl9TVEEAcFZFTkRgcFBSRF9h
emAKEGCgDJGTYACTYAv//6QAoA6Rk2EAlGEMAOH1BaQApAoPCF9DUlMREQoOhgkAAAAA0P4ABAAA
eQAQJS5fU0JfUENJMFuCGVBYMTMIX0FEUgwDAAEAW4BQMTNDAgAK/xBABS5fU0JfUENJMFuCQwRJ
U0FfCF9BRFIMAAABAFuAUDQwQwIKYAoEW4EmXi5QWDEzUDEzQwAASC8AB0xQRU4BADgAA0NBRU4B
AANDQkVOARBAHC8DX1NCX1BDSTBJU0FfW4ItUlRDXwhfSElEDEHQCwAIX0NSUxEYChVHAXAAcAAQ
AiIAAUcBcgByAAIGeQBbgjdLQkRfCF9ISUQMQdADAxQJX1NUQQCkCg8IX0NSUxEYChVHAWAAYAAB
AUcBZABkAAEBIgIAeQBbgidNT1VfCF9ISUQMQdAPExQJX1NUQQCkCg8IX0NSUxEICgUiABB5AFuC
TgRGREMwCF9ISUQMQdAHAAhfQ1JTERsKGEcB8gPyAwAERwH3A/cDAAEiQAAqBAB5AAhfRkRFERcK
FAAAAAAAAAAAAAAAAAAAAAACAAAAW4I+TFBUXwhfSElEDEHQBAAUGF9TVEEAcExQRU5goAaTYACk
AKEEpAoPCF9DUlMREAoNRwF4A3gDCAgigAB5AFuCRQRDT00xCF9ISUQMQdAFAQhfVUlEARQYX1NU
QQBwQ0FFTmCgBpNgAKQAoQSkCg8IX0NSUxEQCg1HAfgD+AMACCIQAHkAW4JGBENPTTIIX0hJRAxB
0AUBCF9VSUQKAhQYX1NUQQBwQ0JFTmCgBpNgAKQAoQSkCg8IX0NSUxEQCg1HAfgC+AIACCIIAHkA
EEgILl9TQl9QQ0kwW4BQQ1NUAQsArgoIW4EQUENTVENQQ0lVIFBDSUQgW4BTRUpfAQsIrgoEW4EL
U0VKX0NCMEVKIFuAQk5NUgELEK4KBFuBC0JOTVJDQk5VTSBbAUJMQ0sAFCVQQ0VKAlsjQkxDS///
cGhCTlVNcHkBaQBCMEVKWydCTENLpAAQTjZfU0JfEEsKUENJMBRECl9QUlQAcBICgGBwAGGiQgmV
YQqAcHphCgIAYnB7cmFiAAoDAGOgEJNjAHASCQQAAExOS0QAZKAkk2MBoBGTYQoEcBIJBAAATE5L
UwBkoQ1wEgkEAABMTktBAGSgEZNjCgJwEgkEAABMTktCAGSgEZNjCgNwEgkEAABMTktDAGRwfXli
ChAAC///AIhkAABwe2EKAwCIZAEAcGSIYGEAdWGkYFuBJC8DUENJMElTQV9QNDBDAVBSUTAIUFJR
MQhQUlEyCFBSUTMIFBNJUVNUAaAJewqAaACkCgmkCgsUNklRQ1IJCFBSUjARDgoLiQYACQEAAAAA
eQCKUFJSMAoFUFJSSaALlWgKgHBoUFJSSaRQUlIwW4JMB0xOS0EIX0hJRAxB0AwPCF9VSUQACF9Q
UlMRFgoTiQ4ACQMFAAAACgAAAAsAAAB5ABQPX1NUQQCkSVFTVFBSUTAUEV9ESVMAfVBSUTAKgFBS
UTAUD19DUlMApElRQ1JQUlEwFBdfU1JTAYpoCgVQUlJJcFBSUklQUlEwW4JMB0xOS0IIX0hJRAxB
0AwPCF9VSUQBCF9QUlMRFgoTiQ4ACQMFAAAACgAAAAsAAAB5ABQPX1NUQQCkSVFTVFBSUTEUEV9E
SVMAfVBSUTEKgFBSUTEUD19DUlMApElRQ1JQUlExFBdfU1JTAYpoCgVQUlJJcFBSUklQUlExW4JN
B0xOS0MIX0hJRAxB0AwPCF9VSUQKAghfUFJTERYKE4kOAAkDBQAAAAoAAAALAAAAeQAUD19TVEEA
pElRU1RQUlEyFBFfRElTAH1QUlEyCoBQUlEyFA9fQ1JTAKRJUUNSUFJRMhQXX1NSUwGKaAoFUFJS
SXBQUlJJUFJRMluCTQdMTktECF9ISUQMQdAMDwhfVUlECgMIX1BSUxEWChOJDgAJAwUAAAAKAAAA
CwAAAHkAFA9fU1RBAKRJUVNUUFJRMxQRX0RJUwB9UFJRMwqAUFJRMxQPX0NSUwCkSVFDUlBSUTMU
F19TUlMBimgKBVBSUklwUFJSSVBSUTNbgk8ETE5LUwhfSElEDEHQDA8IX1VJRAoECF9QUlMRDgoL
iQYACQEJAAAAeQAUCV9TVEEApAoLFAZfRElTABQLX0NSUwCkX1BSUxQGX1NSUwEQTjtfU0JfW4JD
ClwvA19TQl9QQ0kwUFJFUwhfSElEDEHQCgYIX1VJRA1DUFUgSG90cGx1ZyByZXNvdXJjZXMAWwFD
UExLAAhfQ1JTEQ0KCkcBAK8ArwEMeQBbgFBSU1QBCwCvCgxbgSNQUlNUQQAgQ1BFTgFDSU5TAUNS
TVYBQ0VKMAEABENDTUQIW4ESUFJTVANDU0VMIAAgQ0RBVCAUDF9JTkkIcABDU0VMW4JBMVwuX1NC
X0NQVVMIX0hJRA1BQ1BJMDAxMAAIX0NJRAxB0AoFFBxDVEZZAqAKk2gAhkMwMDBpoAqTaAGGQzAw
MWkUSAZDU1RBCVsjXC8EX1NCX1BDSTBQUkVTQ1BMS///cGhcLwRfU0JfUENJMFBSRVNDU0VMcABg
oBqTXC8EX1NCX1BDSTBQUkVTQ1BFTgFwCg9gWydcLwRfU0JfUENJMFBSRVNDUExLpGAUTQVDRUow
CVsjXC8EX1NCX1BDSTBQUkVTQ1BMS///cGhcLwRfU0JfUENJMFBSRVNDU0VMcAFcLwRfU0JfUENJ
MFBSRVNDRUowWydcLwRfU0JfUENJMFBSRVNDUExLFEgOQ1NDTghbI1wvBF9TQl9QQ0kwUFJFU0NQ
TEv//3ABYKJBC5NgAXAAYHAAXC8EX1NCX1BDSTBQUkVTQ0NNRKBHBJNcLwRfU0JfUENJMFBSRVND
SU5TAUNURllcLwRfU0JfUENJMFBSRVNDREFUAXABXC8EX1NCX1BDSTBQUkVTQ0lOU3ABYKFLBKBI
BJNcLwRfU0JfUENJMFBSRVNDUk1WAUNURllcLwRfU0JfUENJMFBSRVNDREFUCgNwAVwvBF9TQl9Q
Q0kwUFJFU0NSTVZwAWBbJ1wvBF9TQl9QQ0kwUFJFU0NQTEsUTQlDT1NUDFsjXC8EX1NCX1BDSTBQ
UkVTQ1BMS///cGhcLwRfU0JfUENJMFBSRVNDU0VMcAFcLwRfU0JfUENJMFBSRVNDQ01EcGlcLwRf
U0JfUENJMFBSRVNDREFUcAoCXC8EX1NCX1BDSTBQUkVTQ0NNRHBqXC8EX1NCX1BDSTBQUkVTQ0RB
VFsnXC8EX1NCX1BDSTBQUkVTQ1BMS1uDOEMwMDAAAAAAAAAUDF9TVEEIpENTVEEACF9NQVQRCwoI
AAgAAAEAAAAUDl9PU1QLQ09TVABoaWpbg0UEQzAwMQEAAAAAABQMX1NUQQikQ1NUQQEIX01BVBEL
CggACAEBAQAAABQLX0VKMAFDRUowARQOX09TVAtDT1NUAWhpahQbXC5fR1BFX0UwMgBcLwNfU0Jf
Q1BVU0NTQ04QTwRfR1BFCF9ISUQNQUNQSTAwMDYAFDlfRTAxAFsjXC8DX1NCX1BDSTBCTENL//9c
LwNfU0JfUENJMFBDTlRbJ1wvA19TQl9QQ0kwQkxDSxBDFFwuX1NCX1BDSTAIX0NSUxFACgqciA0A
AgwAAAAAAP8AAAAAAUcB+Az4DAEIiA0AAQwDAAAAAPcMAAD4DIgNAAEMAwAAAA3//wAAAPOHFwAA
DAMAAAAAAAAKAP//CwAAAAAAAAACAIcXAAAMAQAAAAAAAABA//+//gAAAAAAAMC+iisAAAwDAAAA
AAAAAAAAAAAAAQAAAP///38BAAAAAAAAAAAAAAAAAACAAAAAAHkAW4JDBEdQRTAIX0hJRA1QTlAw
QTA2AAhfVUlEDUdQRTAgcmVzb3VyY2VzAAhfU1RBCgsIX0NSUxENCgpHAeCv4K8BBHkAW4JKBFBI
UFIIX0hJRA1QTlAwQTA2AAhfVUlEDVBDSSBIb3RwbHVnIHJlc291cmNlcwAIX1NUQQoLCF9DUlMR
DQoKRwEArgCuARR5ABAPXAAIX1M1XxIGBAAAAAAQO1wuX1NCX1BDSTBbgi5GV0NGCF9ISUQNUUVN
VTAwMDIACF9TVEEKCwhfQ1JTEQ0KCkcBEAUQBQEMeQAQT21cX1NCXxBHbVBDSTAIQlNFTABbggtT
MDBfCF9BRFIAW4IrUzEwXwhfQURSDAAAAgAUCF9TMUQApAAUCF9TMkQApAAUCV9TM0QApAoDW4Ip
UzE4XwhfQURSDAAAAwAIX1NVTgoDFBJfRUowAVBDRUpCU0VMX1NVTluCKVMyMF8IX0FEUgwAAAQA
CF9TVU4KBBQSX0VKMAFQQ0VKQlNFTF9TVU5bgg9TMjhfCF9BRFIMAAAFAFuCKVMzMF8IX0FEUgwA
AAYACF9TVU4KBhQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTMzhfCF9BRFIMAAAHAAhfU1VOCgcUEl9F
SjABUENFSkJTRUxfU1VOW4IpUzQwXwhfU1VOCggIX0FEUgwAAAgAFBJfRUowAVBDRUpCU0VMX1NV
TluCKVM0OF8IX1NVTgoJCF9BRFIMAAAJABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTNTBfCF9TVU4K
CghfQURSDAAACgAUEl9FSjABUENFSkJTRUxfU1VOW4IpUzU4XwhfU1VOCgsIX0FEUgwAAAsAFBJf
RUowAVBDRUpCU0VMX1NVTluCKVM2MF8IX1NVTgoMCF9BRFIMAAAMABQSX0VKMAFQQ0VKQlNFTF9T
VU5bgilTNjhfCF9TVU4KDQhfQURSDAAADQAUEl9FSjABUENFSkJTRUxfU1VOW4IpUzcwXwhfU1VO
Cg4IX0FEUgwAAA4AFBJfRUowAVBDRUpCU0VMX1NVTluCKVM3OF8IX1NVTgoPCF9BRFIMAAAPABQS
X0VKMAFQQ0VKQlNFTF9TVU5bgilTODBfCF9TVU4KEAhfQURSDAAAEAAUEl9FSjABUENFSkJTRUxf
U1VOW4IpUzg4XwhfU1VOChEIX0FEUgwAABEAFBJfRUowAVBDRUpCU0VMX1NVTluCKVM5MF8IX1NV
TgoSCF9BRFIMAAASABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTOThfCF9TVU4KEwhfQURSDAAAEwAU
El9FSjABUENFSkJTRUxfU1VOW4IpU0EwXwhfU1VOChQIX0FEUgwAABQAFBJfRUowAVBDRUpCU0VM
X1NVTluCKVNBOF8IX1NVTgoVCF9BRFIMAAAVABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTQjBfCF9T
VU4KFghfQURSDAAAFgAUEl9FSjABUENFSkJTRUxfU1VOW4IpU0I4XwhfU1VOChcIX0FEUgwAABcA
FBJfRUowAVBDRUpCU0VMX1NVTluCKVNDMF8IX1NVTgoYCF9BRFIMAAAYABQSX0VKMAFQQ0VKQlNF
TF9TVU5bgilTQzhfCF9TVU4KGQhfQURSDAAAGQAUEl9FSjABUENFSkJTRUxfU1VOW4IpU0QwXwhf
U1VOChoIX0FEUgwAABoAFBJfRUowAVBDRUpCU0VMX1NVTluCKVNEOF8IX1NVTgobCF9BRFIMAAAb
ABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTRTBfCF9TVU4KHAhfQURSDAAAHAAUEl9FSjABUENFSkJT
RUxfU1VOW4IpU0U4XwhfU1VOCh0IX0FEUgwAAB0AFBJfRUowAVBDRUpCU0VMX1NVTluCKVNGMF8I
X1NVTgoeCF9BRFIMAAAeABQSX0VKMAFQQ0VKQlNFTF9TVU5bgilTRjhfCF9TVU4KHwhfQURSDAAA
HwAUEl9FSjABUENFSkJTRUxfU1VOFEsaRFZOVAKgDHtoCggAhlMxOF9poAx7aAoQAIZTMjBfaaAM
e2gKQACGUzMwX2mgDHtoCoAAhlMzOF9poA17aAsAAQCGUzQwX2mgDXtoCwACAIZTNDhfaaANe2gL
AAQAhlM1MF9poA17aAsACACGUzU4X2mgDXtoCwAQAIZTNjBfaaANe2gLACAAhlM2OF9poA17aAsA
QACGUzcwX2mgDXtoCwCAAIZTNzhfaaAPe2gMAAABAACGUzgwX2mgD3toDAAAAgAAhlM4OF9poA97
aAwAAAQAAIZTOTBfaaAPe2gMAAAIAACGUzk4X2mgD3toDAAAEAAAhlNBMF9poA97aAwAACAAAIZT
QThfaaAPe2gMAABAAACGU0IwX2mgD3toDAAAgAAAhlNCOF9poA97aAwAAAABAIZTQzBfaaAPe2gM
AAAAAgCGU0M4X2mgD3toDAAAAAQAhlNEMF9poA97aAwAAAAIAIZTRDhfaaAPe2gMAAAAEACGU0Uw
X2mgD3toDAAAACAAhlNFOF9poA97aAwAAABAAIZTRjBfaaAPe2gMAAAAgACGU0Y4X2kUH1BDTlQA
cABCTlVNRFZOVFBDSVUBRFZOVFBDSUQKAw==
====
begin-base64 644 FACP.1
RkFDUHQAAAAB50JPQ0hTIEJYUENGQUNQAQAAAEJYUEMBAAAAAAD+P0AA/j8BAAkAsgAAAPHwAAAA
BgAAAAAAAAQGAAAAAAAAAAAAAAgGAADgrwAAAAAAAAQCAAQEAAAA/w//DwAAAAAAAAAAMgAAAKWA
AAA=
====
begin-base64 644 RSDT.0
UlNEVCwAAAABBEJPQ0hTIEJYUENSU0RUAQAAAEJYUEMBAAAAKhT+P54U/j8=
====
begin-base64 644 headers
ClJTRCBQVFI6IENoZWNrc3VtPTIyNiwgT0VNSUQ9Qk9DSFMsIFJldmlzaW9uPTAsIFJzZHRBZGRy
ZXNzPTB4M2ZmZTE1MWUKCgpSU0RUOiBMZW5ndGg9NDQsIFJldmlzaW9uPTEsIENoZWNrc3VtPTQs
CglPRU1JRD1CT0NIUywgT0VNIFRhYmxlIElEPUJYUENSU0RULCBPRU0gUmV2aXNpb249MHgxLAoJ
Q3JlYXRvciBJRD1CWFBDLCBDcmVhdG9yIFJldmlzaW9uPTB4MQoKCglFbnRyaWVzPXsgMHgzZmZl
MTQyYSwgMHgzZmZlMTQ5ZSB9CgoKCURTRFQ9MHgzZmZlMDA0MAoJSU5UX01PREVMPUFQSUMKCVND
SV9JTlQ9OQoJU01JX0NNRD0weGIyLCBBQ1BJX0VOQUJMRT0weGYxLCBBQ1BJX0RJU0FCTEU9MHhm
MCwgUzRCSU9TX1JFUT0weDAKCVBNMWFfRVZUX0JMSz0weDYwMC0weDYwMwoJUE0xYV9DTlRfQkxL
PTB4NjA0LTB4NjA1CglQTTJfVE1SX0JMSz0weDYwOC0weDYwYgoJUE0yX0dQRTBfQkxLPTB4YWZl
MC0weGFmZTMKCVBfTFZMMl9MQVQ9NDA5NW1zLCBQX0xWTDNfTEFUPTQwOTVtcwoJRkxVU0hfU0la
RT0wLCBGTFVTSF9TVFJJREU9MAoJRFVUWV9PRkZTRVQ9MCwgRFVUWV9XSURUSD0wCglEQVlfQUxS
TT0wLCBNT05fQUxSTT0wLCBDRU5UVVJZPTUwCglGbGFncz17V0JJTlZELFBST0NfQzEsU0xQX0JV
VFRPTixSVENfUzR9CgoKRFNEVDogTGVuZ3RoPTUwOTgsIFJldmlzaW9uPTEsIENoZWNrc3VtPTUs
CglPRU1JRD1CT0NIUywgT0VNIFRhYmxlIElEPUJYUENEU0RULCBPRU0gUmV2aXNpb249MHgxLAoJ
Q3JlYXRvciBJRD1CWFBDLCBDcmVhdG9yIFJldmlzaW9uPTB4MQoKCkFQSUM6IExlbmd0aD0xMjgs
IFJldmlzaW9uPTEsIENoZWNrc3VtPTIxOCwKCU9FTUlEPUJPQ0hTLCBPRU0gVGFibGUgSUQ9QlhQ
Q0FQSUMsIE9FTSBSZXZpc2lvbj0weDEsCglDcmVhdG9yIElEPUJYUEMsIENyZWF0b3IgUmV2aXNp
b249MHgxCgo=
====

obsd-dev.xml (5K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Kernel panic upon resume of Linux/KVM VM (OpenBSD 6.6)

guenther
On Tue, 22 Oct 2019, Andreas Rottmann wrote:

> >Synopsis: panic: pvclock0: unstable result on stable clock
> >Category: virtualization
> >Environment:
> System      : OpenBSD 6.6
> Details     : OpenBSD 6.6 (GENERIC.MP) #372: Sat Oct 12 10:56:27 MDT 2019
> [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> Architecture: OpenBSD.amd64
> Machine     : amd64
> >Description:
>
> I've just experienced a kernel panic when resuming my laptop from
> suspend-to-RAM while my OpenBSD 6.6 VM was running; the first few lines
> of the crash read like this:
>
> panic: pvclock0: unstable result on stable clock
> Stopped at      db_enter+0x10:  popq    %rbp
>     TID    PID    UID     PRFLAGS     PFLAGS  CPU  COMMAND
> db_enter() at db_enter+0x10
> panic() at panic+0x128
> pvclock_get_timecount(ffffffff81f14360) at pvclock_get_timecount+0xc2
>
> The full ddb session, including backtraces for both cores, and the `ps`
> output is attached as `ddb.txt`.

So the immediate code of the panic is this:
        /* This bit must be set as we attached based on the stable flag */
        if ((flags & PVCLOCK_FLAG_TSC_STABLE) == 0)
                panic("%s: unstable result on stable clock", DEVNAME(sc));

That is, the pvclock driver currently assumes that if it advertises a
stable clock when the OpenBSD guest is booted, then it'll remain stable
forever.  That apparently is not a safe assumption across a suspend/resume
cycle in the Linux/KVM host.

To fix this, the driver would have to get the system to stop using it as
the active timecounter whenever its marked instable.  Perhaps it could
just adjust its quality (sc->sc_tc->tc_quality) downward while that's the
case?  I'm not sure if that would be enough, but you could try
implementing that.

Lacking that, I guess you'll want to have KVM stop the guest before you
suspend the host, and then on resume wait a bit until the clock
settles--not sure how long that takes or how you would know--before
restarting the guest.


Philip Guenther

Reply | Threaded
Open this post in threaded view
|

Re: Kernel panic upon resume of Linux/KVM VM (OpenBSD 6.6)

Mike Larkin-2
On Tue, Oct 22, 2019 at 04:25:19PM -0700, [hidden email] wrote:

> On Tue, 22 Oct 2019, Andreas Rottmann wrote:
> > >Synopsis: panic: pvclock0: unstable result on stable clock
> > >Category: virtualization
> > >Environment:
> > System      : OpenBSD 6.6
> > Details     : OpenBSD 6.6 (GENERIC.MP) #372: Sat Oct 12 10:56:27 MDT 2019
> > [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> >
> > Architecture: OpenBSD.amd64
> > Machine     : amd64
> > >Description:
> >
> > I've just experienced a kernel panic when resuming my laptop from
> > suspend-to-RAM while my OpenBSD 6.6 VM was running; the first few lines
> > of the crash read like this:
> >
> > panic: pvclock0: unstable result on stable clock
> > Stopped at      db_enter+0x10:  popq    %rbp
> >     TID    PID    UID     PRFLAGS     PFLAGS  CPU  COMMAND
> > db_enter() at db_enter+0x10
> > panic() at panic+0x128
> > pvclock_get_timecount(ffffffff81f14360) at pvclock_get_timecount+0xc2
> >
> > The full ddb session, including backtraces for both cores, and the `ps`
> > output is attached as `ddb.txt`.
>
> So the immediate code of the panic is this:
>         /* This bit must be set as we attached based on the stable flag */
>         if ((flags & PVCLOCK_FLAG_TSC_STABLE) == 0)
>                 panic("%s: unstable result on stable clock", DEVNAME(sc));
>
> That is, the pvclock driver currently assumes that if it advertises a
> stable clock when the OpenBSD guest is booted, then it'll remain stable
> forever.  That apparently is not a safe assumption across a suspend/resume
> cycle in the Linux/KVM host.
>

It probably also isn't a safe assumption in a live migration scenario,
either, if you're correct above.

-ml

> To fix this, the driver would have to get the system to stop using it as
> the active timecounter whenever its marked instable.  Perhaps it could
> just adjust its quality (sc->sc_tc->tc_quality) downward while that's the
> case?  I'm not sure if that would be enough, but you could try
> implementing that.
>
> Lacking that, I guess you'll want to have KVM stop the guest before you
> suspend the host, and then on resume wait a bit until the clock
> settles--not sure how long that takes or how you would know--before
> restarting the guest.
>
>
> Philip Guenther
>

Reply | Threaded
Open this post in threaded view
|

Re: Kernel panic upon resume of Linux/KVM VM (OpenBSD 6.6)

Klemens Nanni-2
On Tue, Oct 22, 2019 at 04:52:08PM -0700, Mike Larkin wrote:

> On Tue, Oct 22, 2019 at 04:25:19PM -0700, [hidden email] wrote:
> > On Tue, 22 Oct 2019, Andreas Rottmann wrote:
> > > >Synopsis: panic: pvclock0: unstable result on stable clock
> > > >Category: virtualization
> > > >Environment:
> > > System      : OpenBSD 6.6
> > > Details     : OpenBSD 6.6 (GENERIC.MP) #372: Sat Oct 12 10:56:27 MDT 2019
> > > [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> > >
> > > Architecture: OpenBSD.amd64
> > > Machine     : amd64
> > > >Description:
> > >
> > > I've just experienced a kernel panic when resuming my laptop from
> > > suspend-to-RAM while my OpenBSD 6.6 VM was running; the first few lines
> > > of the crash read like this:
> > >
> > > panic: pvclock0: unstable result on stable clock
> > > Stopped at      db_enter+0x10:  popq    %rbp
> > >     TID    PID    UID     PRFLAGS     PFLAGS  CPU  COMMAND
> > > db_enter() at db_enter+0x10
> > > panic() at panic+0x128
> > > pvclock_get_timecount(ffffffff81f14360) at pvclock_get_timecount+0xc2
> > >
> > > The full ddb session, including backtraces for both cores, and the `ps`
> > > output is attached as `ddb.txt`.
> >
> > So the immediate code of the panic is this:
> >         /* This bit must be set as we attached based on the stable flag */
> >         if ((flags & PVCLOCK_FLAG_TSC_STABLE) == 0)
> >                 panic("%s: unstable result on stable clock", DEVNAME(sc));
> >
> > That is, the pvclock driver currently assumes that if it advertises a
> > stable clock when the OpenBSD guest is booted, then it'll remain stable
> > forever.  That apparently is not a safe assumption across a suspend/resume
> > cycle in the Linux/KVM host.
> >
>
> It probably also isn't a safe assumption in a live migration scenario,
> either, if you're correct above.
>
> -ml
>
> > To fix this, the driver would have to get the system to stop using it as
> > the active timecounter whenever its marked instable.  Perhaps it could
> > just adjust its quality (sc->sc_tc->tc_quality) downward while that's the
> > case?  I'm not sure if that would be enough, but you could try
> > implementing that.
> >
> > Lacking that, I guess you'll want to have KVM stop the guest before you
> > suspend the host, and then on resume wait a bit until the clock
> > settles--not sure how long that takes or how you would know--before
> > restarting the guest.
> >
> >
> > Philip Guenther
> >
>
For the archives, the following commit fixed this panic.
I can successfully suspend and resume a linux host running OpenBSD 6.6
snapshots under QEMU/KVM:

RCS file: /cvs/src/sys/dev/pv/pvclock.c,v
Working file: dev/pv/pvclock.c
head: 1.5
branch:
locks: strict
access list:
keyword substitution: kv
total revisions: 5; selected revisions: 5
description:
----------------------------
revision 1.5
date: 2019/12/13 06:43:46;  author: pd;  state: Exp;  lines: +20 -12;  commitid: mpL92Q7XX7jEvgkn;
pvclock(4): attach even if when PVCLOCK_FLAG_TSC_STABLE is unset

Attaches pvclock with lower priority (500) in case of unstable tsc
(PVCLOCK_FLAG_TSC_STABLE) instead of not attaching at all.  In this state, we do
make sure to return a monotonically increasing number.

This mostly helps openbsd guests on openbsd vmm(4) where a pvclock with unstable
tsc is still better than i8254.

ok mlarkin@