Panic on boot with Hyper-V since Jun 17 snapshot

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Panic on boot with Hyper-V since Jun 17 snapshot

Andre Stoebe
>Synopsis: Panic on boot with Hyper-V since Jun 17 snapshot
>Category: kernel
>Environment:
        System      : OpenBSD 6.7
        Details     : OpenBSD 6.7-current (GENERIC.MP) #375: Sun Jul 26 11:26:37 MDT 2020
                         [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

        Architecture: OpenBSD.amd64
        Machine     : amd64
>Description:
        Booting -current on Hyper-V with hvn(4) results in a panic after
        "starting network".

        This seems to only affect the multi-user boot; hvn(4) still
        works flawlessly on the ramdisk.

        Last working snapshot:
        OpenBSD 6.7-current (GENERIC.MP) #273: Mon Jun 15 19:13:12 MDT 2020

        First non-working snapshot:
        OpenBSD 6.7-current (GENERIC.MP) #278: Wed Jun 17 12:18:35 MDT 2020

        Below is the serial output including ddb trace and ps.
>How-To-Repeat:
        Boot OpenBSD-current on Hyper-V with a hvn(4) network adapter.
>Fix:
        Unknown. As a workaround, disabling hvn(4) via "boot -c" does
        not result in a panic.

NOTE: random seed is being reused.
booting hd0a:/bsd: 14390600+3171336+344096+0+872448 [968918+128+1133640+857287]=0x14bde68
entry point at 0xffffffff81001000
[ using 2961000 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
        The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2020 OpenBSD. All rights reserved.  https://www.OpenBSD.org

OpenBSD 6.7-current (GENERIC.MP) #278: Wed Jun 17 12:18:35 MDT 2020
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4278124544 (4079MB)
avail mem = 4133519360 (3942MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.3 @ 0xf93d0 (338 entries)
bios0: vendor American Megatrends Inc. version "090008" date 12/07/2018
bios0: Microsoft Corporation Virtual Machine
acpi0 at bios0: ACPI 2.0
acpi0: sleep states S0 S5
acpi0: tables DSDT FACP WAET SLIC OEM0 SRAT APIC OEMB
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpihve0 at acpi0
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins, remapped
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz, 2770.10 MHz, 06-5e-03
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 166MHz
cpu1 at mainbus0: apid 1 (application processor)
cpu1: Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz, 2910.06 MHz, 06-5e-03
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz, 2910.09 MHz, 06-5e-03
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz, 2910.11 MHz, 06-5e-03
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
acpipci0 at acpi0 PCI0
extent `acpipci0 pcibus' (0x0 - 0xff), flags=0
extent `pciio' (0x0 - 0xffffffff), flags=0
     0x10000 - 0xffffffff
extent `pcimem' (0x0 - 0xffffffffffffffff), flags=0
     0x0 - 0xf7ffffff
     0x40000000000 - 0xffffffffffffffff
acpicmos0 at acpi0
"VMBus" at acpi0 not configured
"Hyper_V_Gen_Counter_V1" at acpi0 not configured
cpu0: using VERW MDS workaround (except on vmm entry)
pvbus0 at mainbus0: Hyper-V 10.0
hyperv0 at pvbus0: protocol 4.0, features 0x2e7f
hyperv0: heartbeat, kvp, shutdown, timesync
hvs0 at hyperv0 channel 2: ide, protocol 6.2
scsibus1 at hvs0: 2 targets
sd0 at scsibus1 targ 0 lun 0: <Msft, Virtual Disk, 1.0> naa.600224802050cb43ae008c0f97857048
sd0: 65536MB, 512 bytes/sector, 134217728 sectors, thin
hvs1 at hyperv0 channel 15: scsi, protocol 6.2
scsibus2 at hvs1: 2 targets
hvn0 at hyperv0 channel 16: NVS 5.0 NDIS 6.30, address 00:15:5d:9a:72:2f
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82443BX" rev 0x03
pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x01
pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
pciide0: channel 0 disabled (no drives)
atapiscsi0 at pciide0 channel 1 drive 0
scsibus3 at atapiscsi0: 2 targets
cd0 at scsibus3 targ 0 lun 0: <Msft, Virtual CD/ROM, 1.0> removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x02: SMBus disabled
vga1 at pci0 dev 8 function 0 "Microsoft VGA" rev 0x00
wsdisplay at vga1 not configured
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0 mux 1
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vscsi0 at root
scsibus4 at vscsi0: 256 targets
softraid0 at root
scsibus5 at softraid0: 256 targets
root on sd0a (1d6b6b3c33ab37de.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
Automatic boot in progress: starting file system checks.
/dev/sd0a (1d6b6b3c33ab37de.a): FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
/dev/sd0a (1d6b6b3c33ab37de.a): SUMMARY INFORMATION BAD (SALVAGED)
/dev/sd0a (1d6b6fd1 at fdc0 drive 1: density unknown
b3c33ab37de.a): BLK(S) MISSING IN BIT MAPS (SALVAGED)
/dev/sd0a (1d6b6b3c33ab37de.a): 16631 files, 246111 used, 268328 free (192 frags, 33517 blocks, 0.0% fragmentation)
/dev/sd0a (1d6b6b3c33ab37de.a): MARKING FILE SYSTEM CLEAN
/dev/sd0k (1d6b6b3c33ab37de.k): 2 files, 2 used, 13167524 free (20 frags, 1645938 blocks, 0.0% fragmentation)
/dev/sd0k (1d6b6b3c33ab37de.k): MARKING FILE SYSTEM CLEAN
/dev/sd0d (1d6b6b3c33ab37de.d): INCORRECT BLOCK COUNT I=3 (4 should be 0) (CORRECTED)
/dev/sd0d (1d6b6b3c33ab37de.d): UNREF FILE I=3  OWNER=root MODE=100600
/dev/sd0d: SIZE=0 MTIME=Jul 29 10:20 2020  (CLEARED)
/dev/sd0d (1d6b6b3c33ab37de.d): 6 files, 5 used, 2062690 free (42 frags, 257831 blocks, 0.0% fragmentation)
/dev/sd0d (1d6b6b3c33ab37de.d): MARKING FILE SYSTEM CLEAN
/dev/sd0f (1d6b6b3c33ab37de.f): 224680 files, 1157145 used, 389470 free (39678 frags, 43724 blocks, 2.6% fragmentation)
/dev/sd0f (1d6b6b3c33ab37de.f): MARKING FILE SYSTEM CLEAN
/dev/sd0g (1d6b6b3c33ab37de.g): 9160 files, 110467 used, 403972 free (7404 frags, 49571 blocks, 1.4% fragmentation)
/dev/sd0g (1d6b6b3c33ab37de.g): MARKING FILE SYSTEM CLEAN
/dev/sd0h (1d6b6b3c33ab37de.h): 13178 files, 260424 used, 4393111 free (6287 frags, 548353 blocks, 0.1% fragmentation)
/dev/sd0h (1d6b6b3c33ab37de.h): MARKING FILE SYSTEM CLEAN
/dev/sd0j (1d6b6b3c33ab37de.j): 3484 files, 186521 used, 2908358 free (582 frags, 363472 blocks, 0.0% fragmentation)
/dev/sd0j (1d6b6b3c33ab37de.j): MARKING FILE SYSTEM CLEAN
/dev/sd0i (1d6b6b3c33ab37de.i): 138604 files, 620123 used, 410404 free (1612 frags, 51099 blocks, 0.2% fragmentation)
/dev/sd0i (1d6b6b3c33ab37de.i): MARKING FILE SYSTEM CLEAN
/dev/sd0e (1d6b6b3c33ab37de.e): FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
/dev/sd0e (1d6b6b3c33ab37de.e): SUMMARY INFORMATION BAD (SALVAGED)
/dev/sd0e (1d6b6b3c33ab37de.e): BLK(S) MISSING IN BIT MAPS (SALVAGED)
/dev/sd0e (1d6b6b3c33ab37de.e): 561 files, 6789 used, 3619194 free (274 frags, 452365 blocks, 0.0% fragmentation)
/dev/sd0e (1d6b6b3c33ab37de.e): MARKING FILE SYSTEM CLEAN
kbd: keyboard mapping set to de
pf enabled
starting network
kernel: protection fault trap, code=0
Stopped at      setrunqueue+0xa2:       addl    $0x1,0x288(%r13)
ddb{0}> trace
setrunqueue(4632e58d2632e1b7,ffff8000235e79c0,32) at setrunqueue+0xa2
sched_barrier_task(ffff80002367a858) at sched_barrier_task+0x6c
taskq_thread(ffffffff820e01e0) at taskq_thread+0x8d
end trace frame: 0x0, count: -3
ddb{0}> machine ddbcpu 1
Stopped at      x86_ipi_db+0x12:        leave
ddb{1}> trace
x86_ipi_db(ffff800022415ff0) at x86_ipi_db+0x12
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
acpicpu_idle() at acpicpu_idle+0x271
sched_idle(ffff800022415ff0) at sched_idle+0x225
end trace frame: 0x0, count: -5
ddb{1}> machine ddbcpu 2
Stopped at      x86_ipi_db+0x12:        leave
ddb{2}> trace
x86_ipi_db(ffff800022426ff0) at x86_ipi_db+0x12
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
acpicpu_idle() at acpicpu_idle+0x271
sched_idle(ffff800022426ff0) at sched_idle+0x225
end trace frame: 0x0, count: -5
ddb{2}> machine ddbcpu 3
Stopped at      x86_ipi_db+0x12:        leave
ddb{3}> trace
x86_ipi_db(ffff80002242fff0) at x86_ipi_db+0x12
x86_ipi_handler() at x86_ipi_handler+0x80
Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
acpicpu_idle() at acpicpu_idle+0x271
sched_idle(ffff80002242fff0) at sched_idle+0x225
end trace frame: 0x0, count: -5
ddb{3}> ps
   PID     TID   PPID    UID  S       FLAGS  WAIT          COMMAND
 16940  381953   2194      0  3         0x3  sbar          ifconfig
  2194  467874  88209      0  3    0x10008b  pause         sh
 27795  175339  15600    115  3    0x100092  kqread        slaacd
  1268  460749  15600    115  3    0x100092  kqread        slaacd
 15600   87558      1      0  3    0x100080  kqread        slaacd
 75508  331379  15064      0  3    0x100083  nanosleep     sleep
 15064  228406      1      0  3    0x100089  pause         ksh
 88209  267771      1      0  3    0x10008b  pause         sh
 29800  381335      0      0  3     0x14200  bored         smr
 13503   94478      0      0  3     0x14200  pgzero        zerothread
 89041  215522      0      0  3     0x14200  aiodoned      aiodoned
 46332     915      0      0  3     0x14200  syncer        update
 20152  396910      0      0  3     0x14200  cleaner       cleaner
 54251  296627      0      0  3     0x14200  reaper        reaper
 66847  188480      0      0  3     0x14200  pgdaemon      pagedaemon
 78063  276360      0      0  3     0x14200  bored         crynlk
 25507  271821      0      0  3     0x14200  bored         crypto
 80421   84867      0      0  3     0x14200  bored         hvs1
 57199  382476      0      0  3     0x14200  bored         hvs0
 50781   12321      0      0  3  0x40014200  acpi0         acpi0
*99819  420836      0      0  7  0x40014200                idle3
 82224  107227      0      0  7  0x40014200                idle2
 36175  478289      0      0  7  0x40014200                idle1
  8587  296045      0      0  3     0x14200  bored         softnet
kernel: protection fault trap, code=0
Faulted in DDB; continuing...

Reply | Threaded
Open this post in threaded view
|

Re: Panic on boot with Hyper-V since Jun 17 snapshot

Jonathan Gray-11
On Wed, Jul 29, 2020 at 02:05:18PM +0200, Andre Stoebe wrote:

> >Synopsis: Panic on boot with Hyper-V since Jun 17 snapshot
> >Category: kernel
> >Environment:
> System      : OpenBSD 6.7
> Details     : OpenBSD 6.7-current (GENERIC.MP) #375: Sun Jul 26 11:26:37 MDT 2020
> [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> Architecture: OpenBSD.amd64
> Machine     : amd64
> >Description:
> Booting -current on Hyper-V with hvn(4) results in a panic after
> "starting network".
>
> This seems to only affect the multi-user boot; hvn(4) still
> works flawlessly on the ramdisk.
>
> Last working snapshot:
> OpenBSD 6.7-current (GENERIC.MP) #273: Mon Jun 15 19:13:12 MDT 2020
>
> First non-working snapshot:
> OpenBSD 6.7-current (GENERIC.MP) #278: Wed Jun 17 12:18:35 MDT 2020
>
> Below is the serial output including ddb trace and ps.
> >How-To-Repeat:
> Boot OpenBSD-current on Hyper-V with a hvn(4) network adapter.
> >Fix:
> Unknown. As a workaround, disabling hvn(4) via "boot -c" does
> not result in a panic.

more fallout of intr_barrier() now using the argument

Index: sys/dev/pv/if_hvn.c
===================================================================
RCS file: /cvs/src/sys/dev/pv/if_hvn.c,v
retrieving revision 1.41
diff -u -p -r1.41 if_hvn.c
--- sys/dev/pv/if_hvn.c 10 Jul 2020 13:26:40 -0000 1.41
+++ sys/dev/pv/if_hvn.c 29 Jul 2020 12:52:09 -0000
@@ -451,7 +451,7 @@ hvn_stop(struct hvn_softc *sc)
  }
 
  ifq_barrier(&ifp->if_snd);
- intr_barrier(sc->sc_chan);
+ sched_barrier(NULL);
 
  ifq_clr_oactive(&ifp->if_snd);
 }

>
> NOTE: random seed is being reused.
> booting hd0a:/bsd: 14390600+3171336+344096+0+872448 [968918+128+1133640+857287]=0x14bde68
> entry point at 0xffffffff81001000
> [ using 2961000 bytes of bsd ELF symbol table ]
> Copyright (c) 1982, 1986, 1989, 1991, 1993
>         The Regents of the University of California.  All rights reserved.
> Copyright (c) 1995-2020 OpenBSD. All rights reserved.  https://www.OpenBSD.org
>
> OpenBSD 6.7-current (GENERIC.MP) #278: Wed Jun 17 12:18:35 MDT 2020
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 4278124544 (4079MB)
> avail mem = 4133519360 (3942MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.3 @ 0xf93d0 (338 entries)
> bios0: vendor American Megatrends Inc. version "090008" date 12/07/2018
> bios0: Microsoft Corporation Virtual Machine
> acpi0 at bios0: ACPI 2.0
> acpi0: sleep states S0 S5
> acpi0: tables DSDT FACP WAET SLIC OEM0 SRAT APIC OEMB
> acpi0: wakeup devices
> acpitimer0 at acpi0: 3579545 Hz, 32 bits
> acpihve0 at acpi0
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins, remapped
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz, 2770.10 MHz, 06-5e-03
> cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 166MHz
> cpu1 at mainbus0: apid 1 (application processor)
> cpu1: Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz, 2910.06 MHz, 06-5e-03
> cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 0, core 1, package 0
> cpu2 at mainbus0: apid 2 (application processor)
> cpu2: Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz, 2910.09 MHz, 06-5e-03
> cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> cpu2: 256KB 64b/line 8-way L2 cache
> cpu2: smt 0, core 2, package 0
> cpu3 at mainbus0: apid 3 (application processor)
> cpu3: Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz, 2910.11 MHz, 06-5e-03
> cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> cpu3: 256KB 64b/line 8-way L2 cache
> cpu3: smt 0, core 3, package 0
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpicpu0 at acpi0: C1(@1 halt!)
> acpicpu1 at acpi0: C1(@1 halt!)
> acpicpu2 at acpi0: C1(@1 halt!)
> acpicpu3 at acpi0: C1(@1 halt!)
> acpipci0 at acpi0 PCI0
> extent `acpipci0 pcibus' (0x0 - 0xff), flags=0
> extent `pciio' (0x0 - 0xffffffff), flags=0
>      0x10000 - 0xffffffff
> extent `pcimem' (0x0 - 0xffffffffffffffff), flags=0
>      0x0 - 0xf7ffffff
>      0x40000000000 - 0xffffffffffffffff
> acpicmos0 at acpi0
> "VMBus" at acpi0 not configured
> "Hyper_V_Gen_Counter_V1" at acpi0 not configured
> cpu0: using VERW MDS workaround (except on vmm entry)
> pvbus0 at mainbus0: Hyper-V 10.0
> hyperv0 at pvbus0: protocol 4.0, features 0x2e7f
> hyperv0: heartbeat, kvp, shutdown, timesync
> hvs0 at hyperv0 channel 2: ide, protocol 6.2
> scsibus1 at hvs0: 2 targets
> sd0 at scsibus1 targ 0 lun 0: <Msft, Virtual Disk, 1.0> naa.600224802050cb43ae008c0f97857048
> sd0: 65536MB, 512 bytes/sector, 134217728 sectors, thin
> hvs1 at hyperv0 channel 15: scsi, protocol 6.2
> scsibus2 at hvs1: 2 targets
> hvn0 at hyperv0 channel 16: NVS 5.0 NDIS 6.30, address 00:15:5d:9a:72:2f
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 82443BX" rev 0x03
> pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x01
> pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
> pciide0: channel 0 disabled (no drives)
> atapiscsi0 at pciide0 channel 1 drive 0
> scsibus3 at atapiscsi0: 2 targets
> cd0 at scsibus3 targ 0 lun 0: <Msft, Virtual CD/ROM, 1.0> removable
> cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
> piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x02: SMBus disabled
> vga1 at pci0 dev 8 function 0 "Microsoft VGA" rev 0x00
> wsdisplay at vga1 not configured
> isa0 at pcib0
> isadma0 at isa0
> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com0: console
> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0 mux 1
> pms0 at pckbc0 (aux slot)
> wsmouse0 at pms0 mux 0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> vscsi0 at root
> scsibus4 at vscsi0: 256 targets
> softraid0 at root
> scsibus5 at softraid0: 256 targets
> root on sd0a (1d6b6b3c33ab37de.a) swap on sd0b dump on sd0b
> WARNING: / was not properly unmounted
> fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
> Automatic boot in progress: starting file system checks.
> /dev/sd0a (1d6b6b3c33ab37de.a): FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
> /dev/sd0a (1d6b6b3c33ab37de.a): SUMMARY INFORMATION BAD (SALVAGED)
> /dev/sd0a (1d6b6fd1 at fdc0 drive 1: density unknown
> b3c33ab37de.a): BLK(S) MISSING IN BIT MAPS (SALVAGED)
> /dev/sd0a (1d6b6b3c33ab37de.a): 16631 files, 246111 used, 268328 free (192 frags, 33517 blocks, 0.0% fragmentation)
> /dev/sd0a (1d6b6b3c33ab37de.a): MARKING FILE SYSTEM CLEAN
> /dev/sd0k (1d6b6b3c33ab37de.k): 2 files, 2 used, 13167524 free (20 frags, 1645938 blocks, 0.0% fragmentation)
> /dev/sd0k (1d6b6b3c33ab37de.k): MARKING FILE SYSTEM CLEAN
> /dev/sd0d (1d6b6b3c33ab37de.d): INCORRECT BLOCK COUNT I=3 (4 should be 0) (CORRECTED)
> /dev/sd0d (1d6b6b3c33ab37de.d): UNREF FILE I=3  OWNER=root MODE=100600
> /dev/sd0d: SIZE=0 MTIME=Jul 29 10:20 2020  (CLEARED)
> /dev/sd0d (1d6b6b3c33ab37de.d): 6 files, 5 used, 2062690 free (42 frags, 257831 blocks, 0.0% fragmentation)
> /dev/sd0d (1d6b6b3c33ab37de.d): MARKING FILE SYSTEM CLEAN
> /dev/sd0f (1d6b6b3c33ab37de.f): 224680 files, 1157145 used, 389470 free (39678 frags, 43724 blocks, 2.6% fragmentation)
> /dev/sd0f (1d6b6b3c33ab37de.f): MARKING FILE SYSTEM CLEAN
> /dev/sd0g (1d6b6b3c33ab37de.g): 9160 files, 110467 used, 403972 free (7404 frags, 49571 blocks, 1.4% fragmentation)
> /dev/sd0g (1d6b6b3c33ab37de.g): MARKING FILE SYSTEM CLEAN
> /dev/sd0h (1d6b6b3c33ab37de.h): 13178 files, 260424 used, 4393111 free (6287 frags, 548353 blocks, 0.1% fragmentation)
> /dev/sd0h (1d6b6b3c33ab37de.h): MARKING FILE SYSTEM CLEAN
> /dev/sd0j (1d6b6b3c33ab37de.j): 3484 files, 186521 used, 2908358 free (582 frags, 363472 blocks, 0.0% fragmentation)
> /dev/sd0j (1d6b6b3c33ab37de.j): MARKING FILE SYSTEM CLEAN
> /dev/sd0i (1d6b6b3c33ab37de.i): 138604 files, 620123 used, 410404 free (1612 frags, 51099 blocks, 0.2% fragmentation)
> /dev/sd0i (1d6b6b3c33ab37de.i): MARKING FILE SYSTEM CLEAN
> /dev/sd0e (1d6b6b3c33ab37de.e): FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
> /dev/sd0e (1d6b6b3c33ab37de.e): SUMMARY INFORMATION BAD (SALVAGED)
> /dev/sd0e (1d6b6b3c33ab37de.e): BLK(S) MISSING IN BIT MAPS (SALVAGED)
> /dev/sd0e (1d6b6b3c33ab37de.e): 561 files, 6789 used, 3619194 free (274 frags, 452365 blocks, 0.0% fragmentation)
> /dev/sd0e (1d6b6b3c33ab37de.e): MARKING FILE SYSTEM CLEAN
> kbd: keyboard mapping set to de
> pf enabled
> starting network
> kernel: protection fault trap, code=0
> Stopped at      setrunqueue+0xa2:       addl    $0x1,0x288(%r13)
> ddb{0}> trace
> setrunqueue(4632e58d2632e1b7,ffff8000235e79c0,32) at setrunqueue+0xa2
> sched_barrier_task(ffff80002367a858) at sched_barrier_task+0x6c
> taskq_thread(ffffffff820e01e0) at taskq_thread+0x8d
> end trace frame: 0x0, count: -3
> ddb{0}> machine ddbcpu 1
> Stopped at      x86_ipi_db+0x12:        leave
> ddb{1}> trace
> x86_ipi_db(ffff800022415ff0) at x86_ipi_db+0x12
> x86_ipi_handler() at x86_ipi_handler+0x80
> Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
> acpicpu_idle() at acpicpu_idle+0x271
> sched_idle(ffff800022415ff0) at sched_idle+0x225
> end trace frame: 0x0, count: -5
> ddb{1}> machine ddbcpu 2
> Stopped at      x86_ipi_db+0x12:        leave
> ddb{2}> trace
> x86_ipi_db(ffff800022426ff0) at x86_ipi_db+0x12
> x86_ipi_handler() at x86_ipi_handler+0x80
> Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
> acpicpu_idle() at acpicpu_idle+0x271
> sched_idle(ffff800022426ff0) at sched_idle+0x225
> end trace frame: 0x0, count: -5
> ddb{2}> machine ddbcpu 3
> Stopped at      x86_ipi_db+0x12:        leave
> ddb{3}> trace
> x86_ipi_db(ffff80002242fff0) at x86_ipi_db+0x12
> x86_ipi_handler() at x86_ipi_handler+0x80
> Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
> acpicpu_idle() at acpicpu_idle+0x271
> sched_idle(ffff80002242fff0) at sched_idle+0x225
> end trace frame: 0x0, count: -5
> ddb{3}> ps
>    PID     TID   PPID    UID  S       FLAGS  WAIT          COMMAND
>  16940  381953   2194      0  3         0x3  sbar          ifconfig
>   2194  467874  88209      0  3    0x10008b  pause         sh
>  27795  175339  15600    115  3    0x100092  kqread        slaacd
>   1268  460749  15600    115  3    0x100092  kqread        slaacd
>  15600   87558      1      0  3    0x100080  kqread        slaacd
>  75508  331379  15064      0  3    0x100083  nanosleep     sleep
>  15064  228406      1      0  3    0x100089  pause         ksh
>  88209  267771      1      0  3    0x10008b  pause         sh
>  29800  381335      0      0  3     0x14200  bored         smr
>  13503   94478      0      0  3     0x14200  pgzero        zerothread
>  89041  215522      0      0  3     0x14200  aiodoned      aiodoned
>  46332     915      0      0  3     0x14200  syncer        update
>  20152  396910      0      0  3     0x14200  cleaner       cleaner
>  54251  296627      0      0  3     0x14200  reaper        reaper
>  66847  188480      0      0  3     0x14200  pgdaemon      pagedaemon
>  78063  276360      0      0  3     0x14200  bored         crynlk
>  25507  271821      0      0  3     0x14200  bored         crypto
>  80421   84867      0      0  3     0x14200  bored         hvs1
>  57199  382476      0      0  3     0x14200  bored         hvs0
>  50781   12321      0      0  3  0x40014200  acpi0         acpi0
> *99819  420836      0      0  7  0x40014200                idle3
>  82224  107227      0      0  7  0x40014200                idle2
>  36175  478289      0      0  7  0x40014200                idle1
>   8587  296045      0      0  3     0x14200  bored         softnet
> kernel: protection fault trap, code=0
> Faulted in DDB; continuing...
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Panic on boot with Hyper-V since Jun 17 snapshot

Mark Kettenis
> Date: Wed, 29 Jul 2020 23:01:15 +1000
> From: Jonathan Gray <[hidden email]>
>
> On Wed, Jul 29, 2020 at 02:05:18PM +0200, Andre Stoebe wrote:
> > >Synopsis: Panic on boot with Hyper-V since Jun 17 snapshot
> > >Category: kernel
> > >Environment:
> > System      : OpenBSD 6.7
> > Details     : OpenBSD 6.7-current (GENERIC.MP) #375: Sun Jul 26 11:26:37 MDT 2020
> > [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> >
> > Architecture: OpenBSD.amd64
> > Machine     : amd64
> > >Description:
> > Booting -current on Hyper-V with hvn(4) results in a panic after
> > "starting network".
> >
> > This seems to only affect the multi-user boot; hvn(4) still
> > works flawlessly on the ramdisk.
> >
> > Last working snapshot:
> > OpenBSD 6.7-current (GENERIC.MP) #273: Mon Jun 15 19:13:12 MDT 2020
> >
> > First non-working snapshot:
> > OpenBSD 6.7-current (GENERIC.MP) #278: Wed Jun 17 12:18:35 MDT 2020
> >
> > Below is the serial output including ddb trace and ps.
> > >How-To-Repeat:
> > Boot OpenBSD-current on Hyper-V with a hvn(4) network adapter.
> > >Fix:
> > Unknown. As a workaround, disabling hvn(4) via "boot -c" does
> > not result in a panic.
>
> more fallout of intr_barrier() now using the argument

The change doesn't make an awful lot of sense though.  Just like the
current code doesn't make any sense since the driver doesn't do
interrupts.  It is unclear to me what the call is trying to achive.
My guess would be that if anything needs to be done here it is
flushing any pending work associated with the channel.  Which would
imply a taskq_barrier() on the task associated with the channel.


> Index: sys/dev/pv/if_hvn.c
> ===================================================================
> RCS file: /cvs/src/sys/dev/pv/if_hvn.c,v
> retrieving revision 1.41
> diff -u -p -r1.41 if_hvn.c
> --- sys/dev/pv/if_hvn.c 10 Jul 2020 13:26:40 -0000 1.41
> +++ sys/dev/pv/if_hvn.c 29 Jul 2020 12:52:09 -0000
> @@ -451,7 +451,7 @@ hvn_stop(struct hvn_softc *sc)
>   }
>  
>   ifq_barrier(&ifp->if_snd);
> - intr_barrier(sc->sc_chan);
> + sched_barrier(NULL);
>  
>   ifq_clr_oactive(&ifp->if_snd);
>  }
>
> >
> > NOTE: random seed is being reused.
> > booting hd0a:/bsd: 14390600+3171336+344096+0+872448 [968918+128+1133640+857287]=0x14bde68
> > entry point at 0xffffffff81001000
> > [ using 2961000 bytes of bsd ELF symbol table ]
> > Copyright (c) 1982, 1986, 1989, 1991, 1993
> >         The Regents of the University of California.  All rights reserved.
> > Copyright (c) 1995-2020 OpenBSD. All rights reserved.  https://www.OpenBSD.org
> >
> > OpenBSD 6.7-current (GENERIC.MP) #278: Wed Jun 17 12:18:35 MDT 2020
> >     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> > real mem = 4278124544 (4079MB)
> > avail mem = 4133519360 (3942MB)
> > random: good seed from bootblocks
> > mpath0 at root
> > scsibus0 at mpath0: 256 targets
> > mainbus0 at root
> > bios0 at mainbus0: SMBIOS rev. 2.3 @ 0xf93d0 (338 entries)
> > bios0: vendor American Megatrends Inc. version "090008" date 12/07/2018
> > bios0: Microsoft Corporation Virtual Machine
> > acpi0 at bios0: ACPI 2.0
> > acpi0: sleep states S0 S5
> > acpi0: tables DSDT FACP WAET SLIC OEM0 SRAT APIC OEMB
> > acpi0: wakeup devices
> > acpitimer0 at acpi0: 3579545 Hz, 32 bits
> > acpihve0 at acpi0
> > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> > ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins, remapped
> > cpu0 at mainbus0: apid 0 (boot processor)
> > cpu0: Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz, 2770.10 MHz, 06-5e-03
> > cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> > cpu0: 256KB 64b/line 8-way L2 cache
> > cpu0: smt 0, core 0, package 0
> > mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> > cpu0: apic clock running at 166MHz
> > cpu1 at mainbus0: apid 1 (application processor)
> > cpu1: Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz, 2910.06 MHz, 06-5e-03
> > cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> > cpu1: 256KB 64b/line 8-way L2 cache
> > cpu1: smt 0, core 1, package 0
> > cpu2 at mainbus0: apid 2 (application processor)
> > cpu2: Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz, 2910.09 MHz, 06-5e-03
> > cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> > cpu2: 256KB 64b/line 8-way L2 cache
> > cpu2: smt 0, core 2, package 0
> > cpu3 at mainbus0: apid 3 (application processor)
> > cpu3: Intel(R) Xeon(R) CPU E3-1245 v5 @ 3.50GHz, 2910.11 MHz, 06-5e-03
> > cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,FSGSBASE,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> > cpu3: 256KB 64b/line 8-way L2 cache
> > cpu3: smt 0, core 3, package 0
> > acpiprt0 at acpi0: bus 0 (PCI0)
> > acpicpu0 at acpi0: C1(@1 halt!)
> > acpicpu1 at acpi0: C1(@1 halt!)
> > acpicpu2 at acpi0: C1(@1 halt!)
> > acpicpu3 at acpi0: C1(@1 halt!)
> > acpipci0 at acpi0 PCI0
> > extent `acpipci0 pcibus' (0x0 - 0xff), flags=0
> > extent `pciio' (0x0 - 0xffffffff), flags=0
> >      0x10000 - 0xffffffff
> > extent `pcimem' (0x0 - 0xffffffffffffffff), flags=0
> >      0x0 - 0xf7ffffff
> >      0x40000000000 - 0xffffffffffffffff
> > acpicmos0 at acpi0
> > "VMBus" at acpi0 not configured
> > "Hyper_V_Gen_Counter_V1" at acpi0 not configured
> > cpu0: using VERW MDS workaround (except on vmm entry)
> > pvbus0 at mainbus0: Hyper-V 10.0
> > hyperv0 at pvbus0: protocol 4.0, features 0x2e7f
> > hyperv0: heartbeat, kvp, shutdown, timesync
> > hvs0 at hyperv0 channel 2: ide, protocol 6.2
> > scsibus1 at hvs0: 2 targets
> > sd0 at scsibus1 targ 0 lun 0: <Msft, Virtual Disk, 1.0> naa.600224802050cb43ae008c0f97857048
> > sd0: 65536MB, 512 bytes/sector, 134217728 sectors, thin
> > hvs1 at hyperv0 channel 15: scsi, protocol 6.2
> > scsibus2 at hvs1: 2 targets
> > hvn0 at hyperv0 channel 16: NVS 5.0 NDIS 6.30, address 00:15:5d:9a:72:2f
> > pci0 at mainbus0 bus 0
> > pchb0 at pci0 dev 0 function 0 "Intel 82443BX" rev 0x03
> > pcib0 at pci0 dev 7 function 0 "Intel 82371AB PIIX4 ISA" rev 0x01
> > pciide0 at pci0 dev 7 function 1 "Intel 82371AB IDE" rev 0x01: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
> > pciide0: channel 0 disabled (no drives)
> > atapiscsi0 at pciide0 channel 1 drive 0
> > scsibus3 at atapiscsi0: 2 targets
> > cd0 at scsibus3 targ 0 lun 0: <Msft, Virtual CD/ROM, 1.0> removable
> > cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
> > piixpm0 at pci0 dev 7 function 3 "Intel 82371AB Power" rev 0x02: SMBus disabled
> > vga1 at pci0 dev 8 function 0 "Microsoft VGA" rev 0x00
> > wsdisplay at vga1 not configured
> > isa0 at pcib0
> > isadma0 at isa0
> > fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> > com0: console
> > com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> > pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> > pckbd0 at pckbc0 (kbd slot)
> > wskbd0 at pckbd0 mux 1
> > pms0 at pckbc0 (aux slot)
> > wsmouse0 at pms0 mux 0
> > pcppi0 at isa0 port 0x61
> > spkr0 at pcppi0
> > vscsi0 at root
> > scsibus4 at vscsi0: 256 targets
> > softraid0 at root
> > scsibus5 at softraid0: 256 targets
> > root on sd0a (1d6b6b3c33ab37de.a) swap on sd0b dump on sd0b
> > WARNING: / was not properly unmounted
> > fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
> > Automatic boot in progress: starting file system checks.
> > /dev/sd0a (1d6b6b3c33ab37de.a): FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
> > /dev/sd0a (1d6b6b3c33ab37de.a): SUMMARY INFORMATION BAD (SALVAGED)
> > /dev/sd0a (1d6b6fd1 at fdc0 drive 1: density unknown
> > b3c33ab37de.a): BLK(S) MISSING IN BIT MAPS (SALVAGED)
> > /dev/sd0a (1d6b6b3c33ab37de.a): 16631 files, 246111 used, 268328 free (192 frags, 33517 blocks, 0.0% fragmentation)
> > /dev/sd0a (1d6b6b3c33ab37de.a): MARKING FILE SYSTEM CLEAN
> > /dev/sd0k (1d6b6b3c33ab37de.k): 2 files, 2 used, 13167524 free (20 frags, 1645938 blocks, 0.0% fragmentation)
> > /dev/sd0k (1d6b6b3c33ab37de.k): MARKING FILE SYSTEM CLEAN
> > /dev/sd0d (1d6b6b3c33ab37de.d): INCORRECT BLOCK COUNT I=3 (4 should be 0) (CORRECTED)
> > /dev/sd0d (1d6b6b3c33ab37de.d): UNREF FILE I=3  OWNER=root MODE=100600
> > /dev/sd0d: SIZE=0 MTIME=Jul 29 10:20 2020  (CLEARED)
> > /dev/sd0d (1d6b6b3c33ab37de.d): 6 files, 5 used, 2062690 free (42 frags, 257831 blocks, 0.0% fragmentation)
> > /dev/sd0d (1d6b6b3c33ab37de.d): MARKING FILE SYSTEM CLEAN
> > /dev/sd0f (1d6b6b3c33ab37de.f): 224680 files, 1157145 used, 389470 free (39678 frags, 43724 blocks, 2.6% fragmentation)
> > /dev/sd0f (1d6b6b3c33ab37de.f): MARKING FILE SYSTEM CLEAN
> > /dev/sd0g (1d6b6b3c33ab37de.g): 9160 files, 110467 used, 403972 free (7404 frags, 49571 blocks, 1.4% fragmentation)
> > /dev/sd0g (1d6b6b3c33ab37de.g): MARKING FILE SYSTEM CLEAN
> > /dev/sd0h (1d6b6b3c33ab37de.h): 13178 files, 260424 used, 4393111 free (6287 frags, 548353 blocks, 0.1% fragmentation)
> > /dev/sd0h (1d6b6b3c33ab37de.h): MARKING FILE SYSTEM CLEAN
> > /dev/sd0j (1d6b6b3c33ab37de.j): 3484 files, 186521 used, 2908358 free (582 frags, 363472 blocks, 0.0% fragmentation)
> > /dev/sd0j (1d6b6b3c33ab37de.j): MARKING FILE SYSTEM CLEAN
> > /dev/sd0i (1d6b6b3c33ab37de.i): 138604 files, 620123 used, 410404 free (1612 frags, 51099 blocks, 0.2% fragmentation)
> > /dev/sd0i (1d6b6b3c33ab37de.i): MARKING FILE SYSTEM CLEAN
> > /dev/sd0e (1d6b6b3c33ab37de.e): FREE BLK COUNT(S) WRONG IN SUPERBLK (SALVAGED)
> > /dev/sd0e (1d6b6b3c33ab37de.e): SUMMARY INFORMATION BAD (SALVAGED)
> > /dev/sd0e (1d6b6b3c33ab37de.e): BLK(S) MISSING IN BIT MAPS (SALVAGED)
> > /dev/sd0e (1d6b6b3c33ab37de.e): 561 files, 6789 used, 3619194 free (274 frags, 452365 blocks, 0.0% fragmentation)
> > /dev/sd0e (1d6b6b3c33ab37de.e): MARKING FILE SYSTEM CLEAN
> > kbd: keyboard mapping set to de
> > pf enabled
> > starting network
> > kernel: protection fault trap, code=0
> > Stopped at      setrunqueue+0xa2:       addl    $0x1,0x288(%r13)
> > ddb{0}> trace
> > setrunqueue(4632e58d2632e1b7,ffff8000235e79c0,32) at setrunqueue+0xa2
> > sched_barrier_task(ffff80002367a858) at sched_barrier_task+0x6c
> > taskq_thread(ffffffff820e01e0) at taskq_thread+0x8d
> > end trace frame: 0x0, count: -3
> > ddb{0}> machine ddbcpu 1
> > Stopped at      x86_ipi_db+0x12:        leave
> > ddb{1}> trace
> > x86_ipi_db(ffff800022415ff0) at x86_ipi_db+0x12
> > x86_ipi_handler() at x86_ipi_handler+0x80
> > Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
> > acpicpu_idle() at acpicpu_idle+0x271
> > sched_idle(ffff800022415ff0) at sched_idle+0x225
> > end trace frame: 0x0, count: -5
> > ddb{1}> machine ddbcpu 2
> > Stopped at      x86_ipi_db+0x12:        leave
> > ddb{2}> trace
> > x86_ipi_db(ffff800022426ff0) at x86_ipi_db+0x12
> > x86_ipi_handler() at x86_ipi_handler+0x80
> > Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
> > acpicpu_idle() at acpicpu_idle+0x271
> > sched_idle(ffff800022426ff0) at sched_idle+0x225
> > end trace frame: 0x0, count: -5
> > ddb{2}> machine ddbcpu 3
> > Stopped at      x86_ipi_db+0x12:        leave
> > ddb{3}> trace
> > x86_ipi_db(ffff80002242fff0) at x86_ipi_db+0x12
> > x86_ipi_handler() at x86_ipi_handler+0x80
> > Xresume_lapic_ipi() at Xresume_lapic_ipi+0x23
> > acpicpu_idle() at acpicpu_idle+0x271
> > sched_idle(ffff80002242fff0) at sched_idle+0x225
> > end trace frame: 0x0, count: -5
> > ddb{3}> ps
> >    PID     TID   PPID    UID  S       FLAGS  WAIT          COMMAND
> >  16940  381953   2194      0  3         0x3  sbar          ifconfig
> >   2194  467874  88209      0  3    0x10008b  pause         sh
> >  27795  175339  15600    115  3    0x100092  kqread        slaacd
> >   1268  460749  15600    115  3    0x100092  kqread        slaacd
> >  15600   87558      1      0  3    0x100080  kqread        slaacd
> >  75508  331379  15064      0  3    0x100083  nanosleep     sleep
> >  15064  228406      1      0  3    0x100089  pause         ksh
> >  88209  267771      1      0  3    0x10008b  pause         sh
> >  29800  381335      0      0  3     0x14200  bored         smr
> >  13503   94478      0      0  3     0x14200  pgzero        zerothread
> >  89041  215522      0      0  3     0x14200  aiodoned      aiodoned
> >  46332     915      0      0  3     0x14200  syncer        update
> >  20152  396910      0      0  3     0x14200  cleaner       cleaner
> >  54251  296627      0      0  3     0x14200  reaper        reaper
> >  66847  188480      0      0  3     0x14200  pgdaemon      pagedaemon
> >  78063  276360      0      0  3     0x14200  bored         crynlk
> >  25507  271821      0      0  3     0x14200  bored         crypto
> >  80421   84867      0      0  3     0x14200  bored         hvs1
> >  57199  382476      0      0  3     0x14200  bored         hvs0
> >  50781   12321      0      0  3  0x40014200  acpi0         acpi0
> > *99819  420836      0      0  7  0x40014200                idle3
> >  82224  107227      0      0  7  0x40014200                idle2
> >  36175  478289      0      0  7  0x40014200                idle1
> >   8587  296045      0      0  3     0x14200  bored         softnet
> > kernel: protection fault trap, code=0
> > Faulted in DDB; continuing...
> >
> >
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Panic on boot with Hyper-V since Jun 17 snapshot

Mike Belopuhov-5
On 29/07/2020 15:23, Mark Kettenis wrote:

>> Date: Wed, 29 Jul 2020 23:01:15 +1000
>> From: Jonathan Gray <[hidden email]>
>>
>> On Wed, Jul 29, 2020 at 02:05:18PM +0200, Andre Stoebe wrote:
>>>> Synopsis: Panic on boot with Hyper-V since Jun 17 snapshot
>>>> Category: kernel
>>>> Environment:
>>> System      : OpenBSD 6.7
>>> Details     : OpenBSD 6.7-current (GENERIC.MP) #375: Sun Jul 26 11:26:37 MDT 2020
>>> [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>>>
>>> Architecture: OpenBSD.amd64
>>> Machine     : amd64
>>>> Description:
>>> Booting -current on Hyper-V with hvn(4) results in a panic after
>>> "starting network".
>>>
>>> This seems to only affect the multi-user boot; hvn(4) still
>>> works flawlessly on the ramdisk.
>>>
>>> Last working snapshot:
>>> OpenBSD 6.7-current (GENERIC.MP) #273: Mon Jun 15 19:13:12 MDT 2020
>>>
>>> First non-working snapshot:
>>> OpenBSD 6.7-current (GENERIC.MP) #278: Wed Jun 17 12:18:35 MDT 2020
>>>
>>> Below is the serial output including ddb trace and ps.
>>>> How-To-Repeat:
>>> Boot OpenBSD-current on Hyper-V with a hvn(4) network adapter.
>>>> Fix:
>>> Unknown. As a workaround, disabling hvn(4) via "boot -c" does
>>> not result in a panic.
>>
>> more fallout of intr_barrier() now using the argument
>
> The change doesn't make an awful lot of sense though.  Just like the
> current code doesn't make any sense since the driver doesn't do
> interrupts.  It is unclear to me what the call is trying to achive.
> My guess would be that if anything needs to be done here it is
> flushing any pending work associated with the channel.  Which would
> imply a taskq_barrier() on the task associated with the channel.
>
>

Hi Mark,

hvn is interrupt driven; hvn_nvs_intr is the interrupt handler.
It lacks a conventional PCI interrupt establish dance because of
Hyper-V specifics: interrupts and device communication happens
over channels, not conventional buses.

I cannot attest to Jonathan's diff, but if that worked for xnf,
I don't see a reason why it wouldn't work here.

Regards,
Mike

Reply | Threaded
Open this post in threaded view
|

Re: Panic on boot with Hyper-V since Jun 17 snapshot

Mark Kettenis
> From: Mike Belopuhov <[hidden email]>
> Date: Thu, 30 Jul 2020 09:21:32 +0200
>
> On 29/07/2020 15:23, Mark Kettenis wrote:
> >> Date: Wed, 29 Jul 2020 23:01:15 +1000
> >> From: Jonathan Gray <[hidden email]>
> >>
> >> On Wed, Jul 29, 2020 at 02:05:18PM +0200, Andre Stoebe wrote:
> >>>> Synopsis: Panic on boot with Hyper-V since Jun 17 snapshot
> >>>> Category: kernel
> >>>> Environment:
> >>> System      : OpenBSD 6.7
> >>> Details     : OpenBSD 6.7-current (GENERIC.MP) #375: Sun Jul 26 11:26:37 MDT 2020
> >>> [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> >>>
> >>> Architecture: OpenBSD.amd64
> >>> Machine     : amd64
> >>>> Description:
> >>> Booting -current on Hyper-V with hvn(4) results in a panic after
> >>> "starting network".
> >>>
> >>> This seems to only affect the multi-user boot; hvn(4) still
> >>> works flawlessly on the ramdisk.
> >>>
> >>> Last working snapshot:
> >>> OpenBSD 6.7-current (GENERIC.MP) #273: Mon Jun 15 19:13:12 MDT 2020
> >>>
> >>> First non-working snapshot:
> >>> OpenBSD 6.7-current (GENERIC.MP) #278: Wed Jun 17 12:18:35 MDT 2020
> >>>
> >>> Below is the serial output including ddb trace and ps.
> >>>> How-To-Repeat:
> >>> Boot OpenBSD-current on Hyper-V with a hvn(4) network adapter.
> >>>> Fix:
> >>> Unknown. As a workaround, disabling hvn(4) via "boot -c" does
> >>> not result in a panic.
> >>
> >> more fallout of intr_barrier() now using the argument
> >
> > The change doesn't make an awful lot of sense though.  Just like the
> > current code doesn't make any sense since the driver doesn't do
> > interrupts.  It is unclear to me what the call is trying to achive.
> > My guess would be that if anything needs to be done here it is
> > flushing any pending work associated with the channel.  Which would
> > imply a taskq_barrier() on the task associated with the channel.
> >
> >
>
> Hi Mark,
>
> hvn is interrupt driven; hvn_nvs_intr is the interrupt handler.
> It lacks a conventional PCI interrupt establish dance because of
> Hyper-V specifics: interrupts and device communication happens
> over channels, not conventional buses.

Ah, now I remember.  The interrupts bypass the normal interrupt
establish logic.  These interrupts always run on cpu0?

> I cannot attest to Jonathan's diff, but if that worked for xnf,
> I don't see a reason why it wouldn't work here.

Slightly better there though since the operation is abstracted into
xen_intr_barrier().  Something similar should be done for hyperv.

Maybe for now it is good enough to add a hv_channel_intr_barier()
function that takes the channel as an argument and simply calls
sched_barrier(NULL)?

Reply | Threaded
Open this post in threaded view
|

Re: Panic on boot with Hyper-V since Jun 17 snapshot

Mike Belopuhov-5
On 30/07/2020 10:37, Mark Kettenis wrote:

>> From: Mike Belopuhov <[hidden email]>
>> Date: Thu, 30 Jul 2020 09:21:32 +0200
>>
>> On 29/07/2020 15:23, Mark Kettenis wrote:
>>>> Date: Wed, 29 Jul 2020 23:01:15 +1000
>>>> From: Jonathan Gray <[hidden email]>
>>>>
>>>> On Wed, Jul 29, 2020 at 02:05:18PM +0200, Andre Stoebe wrote:
>>>>>> Synopsis: Panic on boot with Hyper-V since Jun 17 snapshot
>>>>>> Category: kernel
>>>>>> Environment:
>>>>> System      : OpenBSD 6.7
>>>>> Details     : OpenBSD 6.7-current (GENERIC.MP) #375: Sun Jul 26 11:26:37 MDT 2020
>>>>> [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>>>>>
>>>>> Architecture: OpenBSD.amd64
>>>>> Machine     : amd64
>>>>>> Description:
>>>>> Booting -current on Hyper-V with hvn(4) results in a panic after
>>>>> "starting network".
>>>>>
>>>>> This seems to only affect the multi-user boot; hvn(4) still
>>>>> works flawlessly on the ramdisk.
>>>>>
>>>>> Last working snapshot:
>>>>> OpenBSD 6.7-current (GENERIC.MP) #273: Mon Jun 15 19:13:12 MDT 2020
>>>>>
>>>>> First non-working snapshot:
>>>>> OpenBSD 6.7-current (GENERIC.MP) #278: Wed Jun 17 12:18:35 MDT 2020
>>>>>
>>>>> Below is the serial output including ddb trace and ps.
>>>>>> How-To-Repeat:
>>>>> Boot OpenBSD-current on Hyper-V with a hvn(4) network adapter.
>>>>>> Fix:
>>>>> Unknown. As a workaround, disabling hvn(4) via "boot -c" does
>>>>> not result in a panic.
>>>>
>>>> more fallout of intr_barrier() now using the argument
>>>
>>> The change doesn't make an awful lot of sense though.  Just like the
>>> current code doesn't make any sense since the driver doesn't do
>>> interrupts.  It is unclear to me what the call is trying to achive.
>>> My guess would be that if anything needs to be done here it is
>>> flushing any pending work associated with the channel.  Which would
>>> imply a taskq_barrier() on the task associated with the channel.
>>>
>>>
>>
>> Hi Mark,
>>
>> hvn is interrupt driven; hvn_nvs_intr is the interrupt handler.
>> It lacks a conventional PCI interrupt establish dance because of
>> Hyper-V specifics: interrupts and device communication happens
>> over channels, not conventional buses.
>
> Ah, now I remember.  The interrupts bypass the normal interrupt
> establish logic.  These interrupts always run on cpu0?
>

All interrupts for PV devices are delivered via hv_intr which is
called by the IDTVEC(intr_hyperv_upcall) -- a handler for
LAPIC_HYPERV_VECTOR interrupt vector.  hv_init_interrupts and
hv_init_synic set it up for the boot CPU currently.

>> I cannot attest to Jonathan's diff, but if that worked for xnf,
>> I don't see a reason why it wouldn't work here.
>
> Slightly better there though since the operation is abstracted into
> xen_intr_barrier().  Something similar should be done for hyperv.
>
> Maybe for now it is good enough to add a hv_channel_intr_barier()
> function that takes the channel as an argument and simply calls
> sched_barrier(NULL)?
>

Sure, I must say I haven't followed recent improvements of interrupt
barriers, but AFAIR, neither Xen nor Hyper-V codebases aren't really
prepared to make a proper use of the intr barrier with an argument.
IIRC, these barriers are used to synchronize interrupt threads when
disabling interrupts during <if>_stop routines.

Reply | Threaded
Open this post in threaded view
|

Re: Panic on boot with Hyper-V since Jun 17 snapshot

Andre Stoebe
In reply to this post by Jonathan Gray-11
On 29.07.2020 15:01, Jonathan Gray wrote:

> more fallout of intr_barrier() now using the argument
>
> Index: sys/dev/pv/if_hvn.c
> ===================================================================
> RCS file: /cvs/src/sys/dev/pv/if_hvn.c,v
> retrieving revision 1.41
> diff -u -p -r1.41 if_hvn.c
> --- sys/dev/pv/if_hvn.c 10 Jul 2020 13:26:40 -0000 1.41
> +++ sys/dev/pv/if_hvn.c 29 Jul 2020 12:52:09 -0000
> @@ -451,7 +451,7 @@ hvn_stop(struct hvn_softc *sc)
>   }
>  
>   ifq_barrier(&ifp->if_snd);
> - intr_barrier(sc->sc_chan);
> + sched_barrier(NULL);
>  
>   ifq_clr_oactive(&ifp->if_snd);
>  }

Hi Jonathan,

I can confirm that this fixes the panic.

Thanks,
Andre

Reply | Threaded
Open this post in threaded view
|

Re: Panic on boot with Hyper-V since Jun 17 snapshot

Andre Stoebe
On 30.07.2020 12:05, Andre Stoebe wrote:

> On 29.07.2020 15:01, Jonathan Gray wrote:
>> more fallout of intr_barrier() now using the argument
>>
>> Index: sys/dev/pv/if_hvn.c
>> ===================================================================
>> RCS file: /cvs/src/sys/dev/pv/if_hvn.c,v
>> retrieving revision 1.41
>> diff -u -p -r1.41 if_hvn.c
>> --- sys/dev/pv/if_hvn.c 10 Jul 2020 13:26:40 -0000 1.41
>> +++ sys/dev/pv/if_hvn.c 29 Jul 2020 12:52:09 -0000
>> @@ -451,7 +451,7 @@ hvn_stop(struct hvn_softc *sc)
>>   }
>>  
>>   ifq_barrier(&ifp->if_snd);
>> - intr_barrier(sc->sc_chan);
>> + sched_barrier(NULL);
>>  
>>   ifq_clr_oactive(&ifp->if_snd);
>>  }
>
> Hi Jonathan,
>
> I can confirm that this fixes the panic.
>
> Thanks,
> Andre

Can this or a similar fix get committed? It's broken for some time now.

Reply | Threaded
Open this post in threaded view
|

Re: Panic on boot with Hyper-V since Jun 17 snapshot

Jonathan Gray-11
On Sun, Aug 30, 2020 at 12:20:50PM +0200, Andre Stoebe wrote:

> On 30.07.2020 12:05, Andre Stoebe wrote:
> > On 29.07.2020 15:01, Jonathan Gray wrote:
> >> more fallout of intr_barrier() now using the argument
> >>
> >> Index: sys/dev/pv/if_hvn.c
> >> ===================================================================
> >> RCS file: /cvs/src/sys/dev/pv/if_hvn.c,v
> >> retrieving revision 1.41
> >> diff -u -p -r1.41 if_hvn.c
> >> --- sys/dev/pv/if_hvn.c 10 Jul 2020 13:26:40 -0000 1.41
> >> +++ sys/dev/pv/if_hvn.c 29 Jul 2020 12:52:09 -0000
> >> @@ -451,7 +451,7 @@ hvn_stop(struct hvn_softc *sc)
> >>   }
> >>  
> >>   ifq_barrier(&ifp->if_snd);
> >> - intr_barrier(sc->sc_chan);
> >> + sched_barrier(NULL);
> >>  
> >>   ifq_clr_oactive(&ifp->if_snd);
> >>  }
> >
> > Hi Jonathan,
> >
> > I can confirm that this fixes the panic.
> >
> > Thanks,
> > Andre
>
> Can this or a similar fix get committed? It's broken for some time now.

Sorry, I lost track of this committed.
If people want to rework it further that can be a different diff.