6.8/arm64 on RaspberryPi 4B: kernel panics during booting under devicetree mode due to xhci

[utunbu]

Hi,

I've successfully installed OpenBSD/arm64-6.8 on a RPi4B following https://ftp.openbsd.org/pub/OpenBSD/6.8/arm64/INSTALL.arm64 and https://github.com/pftf/RPi4 on a USB device with UEFI (v1.24) boot from the SD card.

If I config
> UEFI ->  Device Manager -> Raspberry Pi Configuration -> Advanced Configuration -> System Table Selection
to Devicetree or ACPI + Devicetree (as in the below figure), booting resulted kernel panic; using ACPI alone allows me to boot, but does not allow me to access GPIO devices on RPi4 such as the wireless chip, as suggested in:
https://www.reddit.com/r/openbsd/comments/gp3tb8/bcmgpio_not_present_on_raspberry_pi_4/
and
https://github.com/AshyIsMe/openbsd-rpi4.







The booting message is:

> disks: sd0* sd1
> >> OpenBSD/arm64 BOOTAA64 1.2
> boot>
> NOTE: random seed is being reused.
> booting sd0a:/bsd: 8618988+1764280+552464+794752 [618710+109+1051200+616462]=0xf
> 775e8
> type 0x0 pa 0x0 va 0x0 pages 0x1d0 attr 0xe
> type 0x6 pa 0x1d0000 va 0x48f7cb5000 pages 0x20 attr 0x800000000000000e
> type 0x0 pa 0x1f0000 va 0x0 pages 0x10 attr 0xe
> type 0x2 pa 0x200000 va 0x0 pages 0x4000 attr 0xe
> type 0x7 pa 0x4200000 va 0x0 pages 0x2e96c attr 0xe
> type 0x2 pa 0x32b6c000 va 0x0 pages 0xde4 attr 0xe
> type 0x1 pa 0x33950000 va 0x0 pages 0x2a attr 0xe
> type 0x0 pa 0x3397a000 va 0x0 pages 0x56 attr 0xe
> type 0x6 pa 0x339d0000 va 0x492b4b5000 pages 0x50 attr 0x800000000000000e
> type 0x5 pa 0x33a20000 va 0x492b505000 pages 0x100 attr 0x800000000000000e
> type 0x9 pa 0x33b20000 va 0x0 pages 0x10 attr 0xe
> type 0x7 pa 0x33b30000 va 0x0 pages 0x16d3 attr 0xe
> type 0x4 pa 0x35203000 va 0x0 pages 0x4a0 attr 0xe
> type 0x7 pa 0x356a3000 va 0x0 pages 0x16 attr 0xe
> type 0x4 pa 0x356b9000 va 0x0 pages 0x136b attr 0xe
> type 0x7 pa 0x36a24000 va 0x0 pages 0x1ac attr 0xe
> type 0x3 pa 0x36bd0000 va 0x0 pages 0x430 attr 0xe
> type 0x5 pa 0x37000000 va 0x492eae5000 pages 0x190 attr 0x800000000000000e
> type 0x6 pa 0x37190000 va 0x492ec75000 pages 0x250 attr 0x800000000000000e
> type 0x7 pa 0x373e0000 va 0x0 pages 0x1f attr 0xe
> type 0x4 pa 0x373ff000 va 0x0 pages 0x1 attr 0xe
> type 0x7 pa 0x37400000 va 0x0 pages 0x30cd attr 0xe
> type 0x3 pa 0x3a4cd000 va 0x0 pages 0x38 attr 0xe
> type 0x4 pa 0x3a505000 va 0x0 pages 0xefb attr 0xe
> type 0x7 pa 0x40000000 va 0x0 pages 0x40000 attr 0xe
> [ using 2287448 bytes of bsd ELF symbol table ]
> Copyright (c) 1982, 1986, 1989, 1991, 1993
>         The Regents of the University of California.  All rights reserved.
> Copyright (c) 1995-2020 OpenBSD. All rights reserved.  https://www.OpenBSD.org <https://www.openbsd.org/>
>
> OpenBSD 6.8 (GENERIC.MP) #828: Sun Oct  4 20:35:47 MDT 2020
>     [hidden email]:/usr/src/sys/arch/arm64/compile/GENERIC.MP
> real mem  = 1999556608 (1906MB)
> avail mem = 1905778688 (1817MB)
> random: good seed from bootblocks
> mainbus0 at root: Raspberry Pi 4 Model B Rev 1.2
> psci0 at mainbus0: PSCI 1.1, SMCCC 1.2
> cpu0 at mainbus0 mpidr 0: ARM Cortex-A72 r0p3
> cpu0: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache
> cpu0: 1024KB 64b/line 16-way L2 cache
> cpu1 at mainbus0 mpidr 1: ARM Cortex-A72 r0p3
> cpu1: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache
> cpu1: 1024KB 64b/line 16-way L2 cache
> cpu2 at mainbus0 mpidr 2: ARM Cortex-A72 r0p3
> cpu2: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache
> cpu2: 1024KB 64b/line 16-way L2 cache
> cpu3 at mainbus0 mpidr 3: ARM Cortex-A72 r0p3
> cpu3: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache
> cpu3: 1024KB 64b/line 16-way L2 cache
> efi0 at mainbus0: UEFI 2.7
> efi0: https://github.com/pftf/RPi4 rev 0x10000
> smbios0 at efi0: SMBIOS 3.3.0
> smbios0: vendor https://github.com/pftf/RPi4 version "UEFI Firmware v1.24" date 02/26/2021
> smbios0: Raspberry Pi Foundation Raspberry Pi 4 Model B
> apm0 at mainbus0
> "system" at mainbus0 not configured
> "axi" at mainbus0 not configured
> simplebus0 at mainbus0: "soc"
> bcmclock0 at simplebus0
> bcmmbox0 at simplebus0
> bcmgpio0 at simplebus0
> bcmaux0 at simplebus0
> ampintc0 at simplebus0 nirq 256, ncpu 4 ipi: 0, 1: "interrupt-controller"
> bcmtmon0 at simplebus0
> bcmdmac0 at simplebus0: DMA0 DMA2 DMA4 DMA5 DMA6 DMA7 DMA8 DMA9
> "timer" at simplebus0 not configured
> pluart0 at simplebus0: console
> com0 at simplebus0: ns16550, no working fifo
> "local_intc" at simplebus0 not configured
> bcmdog0 at simplebus0
> bcmirng0 at simplebus0
> "firmware" at simplebus0 not configured
> "power" at simplebus0 not configured
> "mailbox" at simplebus0 not configured
> sdhc0 at simplebus0
> sdhc0: SDHC 3.0, 250 MHz base clock
> sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed
> "gpiomem" at simplebus0 not configured
> "fb" at simplebus0 not configured
> "vcsm" at simplebus0 not configured
> "clocks" at mainbus0 not configured
> "phy" at mainbus0 not configured
> "clk-27M" at mainbus0 not configured
> "clk-108M" at mainbus0 not configured
> simplebus1 at mainbus0: "emmc2bus"
> sdhc1 at simplebus1
> sdhc1: SDHC 3.0, 100 MHz base clock
> sdmmc1 at sdhc1: 8-bit, sd high-speed, mmc high-speed, ddr52, dma
> "arm-pmu" at mainbus0 not configured
> agtimer0 at mainbus0: tick rate 54000 KHz
> simplebus2 at mainbus0: "scb"
> bcmpcie0 at simplebus2
> pci0 at bcmpcie0
> ppb0 at pci0 dev 0 function 0 "Broadcom BCM2711" rev 0x10
> pci1 at ppb0 bus 1
> xhci0 at pci1 dev 0 function 0 "VIA VL805 xHCI" rev 0x01: intxpanic: uvm_fault failed: ffffff800055a884 esr 96000007 far ffffff8027e66ab4
> Stopped at      panic+0x158:    mov     w0, w20
>     TID    PID    UID     PRFLAGS     PFLAGS  CPU  COMMAND
> *     0      0      0     0x10000      0x200    0K swapper
> db_enter() at panic+0x154
> panic() at data_abort+0x284
> do_el0_sync() at handle_el1h_sync+0x6c
> handle_el1h_sync() at xhci_pci_takecontroller+0x78
> xhci_pci_takecontroller() at xhci_pci_takecontroller+0x78
> xhci_pci_takecontroller() at xhci_pci_attach+0x224
> xhci_pci_attach() at config_attach+0x21c
> https://www.openbsd.org/ddb.html describes the minimum info required in bug
> reports.  Insufficient info makes it difficult to find and fix bugs.
> ddb{0}>

ddb outputs:
(*the dmesg output is a subset of the above and is omitted*)

> ddb{0}> trace
> db_enter() at panic+0x154
> panic() at data_abort+0x284
> do_el0_sync() at handle_el1h_sync+0x6c
> handle_el1h_sync() at xhci_pci_takecontroller+0x78
> xhci_pci_takecontroller() at xhci_pci_takecontroller+0x78
> xhci_pci_takecontroller() at xhci_pci_attach+0x224
> xhci_pci_attach() at config_attach+0x21c
> config_attach() at pci_probe_device+0x3f0
> pci_probe_device() at pci_enumerate_bus+0x140
> pci_enumerate_bus() at config_attach+0x21c
> config_attach() at ppbattach+0x638
> ppbattach() at config_attach+0x21c
> config_attach() at pci_probe_device+0x3f0
> pci_probe_device() at pci_enumerate_bus+0x140
> pci_enumerate_bus() at config_attach+0x21c
> config_attach() at bcmpcie_attach+0x42c
> bcmpcie_attach() at config_attach+0x21c
> config_attach() at simplebus_attach_node+0x308
> simplebus_attach_node() at simplebus_attach+0x258
> simplebus_attach() at config_attach+0x21c
> config_attach() at mainbus_attach_node+0x2cc
> mainbus_attach_node() at mainbus_attach+0x2dc
> mainbus_attach() at config_attach+0x21c
> config_attach() at cpu_configure+0x2c
> cpu_configure() at main+0x318
> main() at $x.2+0x70

> ddb{0}> show panic
> uvm_fault failed: ffffff800055a884 esr 96000007 far ffffff8027e66ab4

> ddb{0}> ps
>    PID     TID   PPID    UID  S       FLAGS  WAIT          COMMAND
> *    0       0     -1      0  7     0x10200                swapper

> ddb{0}> cpuinfo
> No such command
> ddb{0}> machine cpuinfo
> *   0: ddb
>     1: stopping
>     2: stopping
>     3: stopping

> ddb{0}> show reg
> x0                               0x1
> x1                0xffffff8027e1a000
> x2                               0x4
> x3                                 0
> x4                0xffffff8000eb6120    initstack+0x3f20
> x5                0xffffff800778b000    _end+0x6842170
> x6                0xffffff800778b024    _end+0x6842194
> x7                0xffffff800778b024    _end+0x6842194
> x8                               0x1
> x9                0xa811e99eaa97da45
> x10                              0xb
> x11                              0x1
> x12                       0xffffffc8
> x13                             0x20    $d.1+0xd
> x14                       0xffffffff
> x15               0x5cd30d11ce5874a9
> x16                              0x5
> x17                      0x638001000
> x18               0xffffff8000eb6180    initstack+0x3f80
> x19               0xffffff8000c42669    $d.2+0xf
> x20                            0x100    $d.1+0xed
> x21               0xffffff8000e76948    kprintf_mutex
> x22               0xffffff8000ee6000    de_pq_table+0xbd8
> x23               0xffffff8000eb61d0    initstack+0x3fd0
> x24               0xffffff8027e66000
> x25                              0xe
> x26               0xffffff8000ede450    kernel_map_store
> x27               0xffffff8000eb2200    initstack
> x28                                0
> x29               0xffffff8000eb6180    initstack+0x3f80
> x30                                0
> sp                0xffffff8000eb6180    initstack+0x3f80
> spsr                      0x60000345
> elr               0xffffff800073b70c    db_enter+0x14
> lr                0xffffff800086bbac    panic+0x158
> panic+0x158:    mov     w0, w20

> ddb{0}> show uvm
> Current UVM status:
>   pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
>   465320 VM pages: 0 active, 0 inactive, 0 wired, 465159 free (0 zero)
>   min  10% (25) anon, 10% (25) vnode, 5% (12) vtext
>   freemin=0, free-target=0, inactive-target=0, wired-max=0
>   faults=1, traps=0, intrs=0, ctxswitch=0 fpuswitch=0
>   softint=0, syscalls=0, kmapent=16
>   fault counts:
>     noram=0, noanon=0, noamap=0, pgwait=0, pgrele=0
>     ok relocks(total)=0(0), anget(retries)=0(0), amapcopy=0
>     neighbor anon/obj pg=0/0, gets(lock/unlock)=0/0
>     cases: anon=0, anoncow=0, obj=0, prcopy=0, przero=0
>   daemon and swap counts:
>     woke=0, revs=0, scans=0, obscans=0, anscans=0
>     busy=0, freed=0, reactivate=0, deactivate=0
>     pageouts=0, pending=0, nswget=0
>     nswapdev=0
>     swpages=0, swpginuse=0, swpgonly=0 paging=0
>   kernel pointers:
>     objs(kern)=0xffffff8000ed9640

> ddb{0}> show bcstats
> Current Buffer Cache status:
> numbufs 0 busymapped 0, delwri 0
> kvaslots 2724 avail kva slots 2724
> bufpages 0, dmapages 0, dirtypages 0
> pendingreads 0, pendingwrites 0
> highflips 0, highflops 0, dmaflips 0

I noticed a similar problem report
http://openbsd-archive.7691.n7.nabble.com/aarch64-panic-at-do-el1h-sync-0x21c-td407163.html#a407183
that also involves ``do_el0_sync()``  and ``handle_el1h_sync()``.

Below is the ``sendbug(1)`` output after a successful boot *using "APCI" mode*:
> >Environment:
>         System      : OpenBSD 6.8
>         Details     : OpenBSD 6.8 (GENERIC.MP) #828: Sun Oct  4 20:35:47 MDT 2020
>                          [hidden email]:/usr/src/sys/arch/arm64/compile/GENERIC.MP

[Stuart Henderson]

On 2021/03/22 04:09, [hidden email] wrote:
> > smbios0 at efi0: SMBIOS 3.3.0
> > smbios0: vendor https://github.com/pftf/RPi4 version "UEFI Firmware v1.24" date 02/26/2021
> > smbios0: Raspberry Pi Foundation Raspberry Pi 4 Model B

Try different firmware versions. Mine is running 1.13 (and is stable so
I haven't seen a need to try others)..

[utunbu]

Thank you Stuart! I've downgraded to v1.21 and this issue has been resolved.

Only then did I find out that they explicitly mentioned in v1.22 release that newer releases break xHCI initialization; should have looked into UEFI first :)

On Mon, Mar 22, 2021, at 13:14, Stuart Henderson wrote:
> On 2021/03/22 04:09, [hidden email] <mailto:cs%40imap.cc> wrote:
> > > smbios0 at efi0: SMBIOS 3.3.0
> > > smbios0: vendor https://github.com/pftf/RPi4 version "UEFI Firmware v1.24" date 02/26/2021
> > > smbios0: Raspberry Pi Foundation Raspberry Pi 4 Model B
>
> Try different firmware versions. Mine is running 1.13 (and is stable so
> I haven't seen a need to try others)..
>
>