SB drive not detected after reboot on PC Engines APU2

[Marfaba Stewart]

Synopsis: USB drive not detected after reboot on PC Engines APU2
Category: amd64
Environment:
        System      : OpenBSD 6.9
        Details     : OpenBSD 6.9-beta (GENERIC.MP) #343: Thu Feb 18 23:09:55 MST 2021
                         [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

        Architecture: OpenBSD.amd64
        Machine     : amd64
Description:
        After leaving USB drive plugged in and then rebooting on
        PC Engines APU2, my USB drive isn't detected. If I unplug
        and replug into the same USB port, it doesn't change console
        output or dmesg, even though the USB drive does get power.
        Plugging into a different port triggers detection and the
        drive can then be mounted. The USB drive I've tested is a
        Western Digital Elements drive. The same thing happens on
        multiple PC Engines APU2 units running -current for a while,
        but unfortunately I procrastinated on reporting this. I can
        attach the same drive to a desktop and a laptop running
        current, and they detect it after reboot.  The usbdevs
        output below is after switching USB ports.

How-To-Repeat:
        Leave USB drive in port. Reboot. Even removing and putting
        in same port doesn't detect drive. Only switching to other
        USB port detects drives.
Fix:
        work around is put USB drive in other USB port after reboot.

dmesg:
OpenBSD 6.9-beta (GENERIC.MP) #343: Thu Feb 18 23:09:55 MST 2021
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4261076992 (4063MB)
avail mem = 4116594688 (3925MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xdffb7020 (7 entries)
bios0: vendor coreboot version "4.0.7" date 02/28/2017
bios0: PC Engines APU2
acpi0 at bios0: ACPI 4.0
acpi0: sleep states S0 S1 S2 S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC HEST SSDT SSDT HPET
acpi0: wakeup devices PWRB(S4) PBR4(S4) PBR5(S4) PBR6(S4) PBR7(S4) PBR8(S4) UOH1(S3) UOH3(S3) UOH5(S3) XHC0(S4)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD GX-412TC SOC, 998.36 MHz, 16-30-01
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu0: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD GX-412TC SOC, 998.01 MHz, 16-30-01
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu1: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu1: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu1: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: AMD GX-412TC SOC, 998.01 MHz, 16-30-01
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu2: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu2: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu2: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: AMD GX-412TC SOC, 998.01 MHz, 16-30-01
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu3: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu3: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu3: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 21, 24 pins
ioapic1 at mainbus0: apid 5 pa 0xfec20000, version 21, 32 pins, remapped
acpihpet0 at acpi0: 14318180 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (PBR4)
acpiprt2 at acpi0: bus 1 (PBR5)
acpiprt3 at acpi0: bus 2 (PBR6)
acpiprt4 at acpi0: bus 3 (PBR7)
acpiprt5 at acpi0: bus 4 (PBR8)
acpibtn0 at acpi0: PWRB
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
acpicpu0 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu1 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu2 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu3 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
cpu0: 998 MHz: speeds: 1000 800 600 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "AMD 16h Root Complex" rev 0x00
pchb1 at pci0 dev 2 function 0 "AMD 16h Host" rev 0x00
ppb0 at pci0 dev 2 function 2 "AMD 16h PCIE" rev 0x00: msi
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel I210" rev 0x03: msi, address xxx
ppb1 at pci0 dev 2 function 3 "AMD 16h PCIE" rev 0x00: msi
pci2 at ppb1 bus 2
em1 at pci2 dev 0 function 0 "Intel I210" rev 0x03: msi, address xxx
ppb2 at pci0 dev 2 function 4 "AMD 16h PCIE" rev 0x00: msi
pci3 at ppb2 bus 3
em2 at pci3 dev 0 function 0 "Intel I210" rev 0x03: msi, address xxx
ppb3 at pci0 dev 2 function 5 "AMD 16h PCIE" rev 0x00: msi
pci4 at ppb3 bus 4
athn0 at pci4 dev 0 function 0 "Atheros AR9281" rev 0x01: apic 5 int 16
athn0: AR9280 rev 2 (2T2R), ROM rev 22, address xxx
ccp0 at pci0 dev 8 function 0 "AMD 16h Crypto" rev 0x00
xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x11: msi, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "AMD xHCI root hub" rev 3.00/1.00 addr 1
ahci0 at pci0 dev 17 function 0 "AMD Hudson-2 SATA" rev 0x40: apic 4 int 19, AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, SATA SSD, SBFM> naa.0000000000000000
sd0: 28626MB, 512 bytes/sector, 58626288 sectors, thin
ehci0 at pci0 dev 19 function 0 "AMD Hudson-2 USB2" rev 0x39: apic 4 int 18
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
piixpm0 at pci0 dev 20 function 0 "AMD Hudson-2 SMBus" rev 0x42: SMI
iic0 at piixpm0
iic1 at piixpm0
pcib0 at pci0 dev 20 function 3 "AMD Hudson-2 LPC" rev 0x11
sdhc0 at pci0 dev 20 function 7 "AMD Bolton SD/MMC" rev 0x01: apic 4 int 16
sdhc0: SDHC 2.0, 50 MHz base clock
sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed, dma
pchb2 at pci0 dev 24 function 0 "AMD 16h Link Cfg" rev 0x00
pchb3 at pci0 dev 24 function 1 "AMD 16h Address Map" rev 0x00
pchb4 at pci0 dev 24 function 2 "AMD 16h DRAM Cfg" rev 0x00
km0 at pci0 dev 24 function 3 "AMD 16h Misc Cfg" rev 0x00
pchb5 at pci0 dev 24 function 4 "AMD 16h CPU Power" rev 0x00
pchb6 at pci0 dev 24 function 5 "AMD 16h Misc Cfg" rev 0x00
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
wbsio0 at isa0 port 0x2e/2: NCT5104D rev 0x53
vmm0 at mainbus0: SVM/RVI
uhub2 at uhub1 port 1 configuration 1 interface 0 "Advanced Micro Devices Hub" rev 2.00/0.18 addr 2
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (xxx.a) swap on sd0b dump on sd0b

usbdevs:
Controller /dev/usb0:
addr 01: 1022:0000 AMD, xHCI root hub
         super speed, self powered, config 1, rev 1.00
         driver: uhub0
addr 02: 1058:2620 Western Digital, Elements 2620
         super speed, power 224 mA, config 1, rev 10.18, iSerial xxx
         driver: umass0
Controller /dev/usb1:
addr 01: 1022:0000 AMD, EHCI root hub
         high speed, self powered, config 1, rev 1.00
         driver: uhub1
addr 02: 0438:7900 Advanced Micro Devices, Hub
         high speed, self powered, config 1, rev 0.18
         driver: uhub2

[Marcus Glocker]

On Fri, Feb 19, 2021 at 04:48:11PM +0000, marfabastewart wrote:

> Synopsis: USB drive not detected after reboot on PC Engines APU2
> Category: amd64
> Environment:
> System      : OpenBSD 6.9
> Details     : OpenBSD 6.9-beta (GENERIC.MP) #343: Thu Feb 18 23:09:55 MST 2021
> [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> Architecture: OpenBSD.amd64
> Machine     : amd64
> Description:
>         After leaving USB drive plugged in and then rebooting on
>         PC Engines APU2, my USB drive isn't detected. If I unplug
>         and replug into the same USB port, it doesn't change console
>         output or dmesg, even though the USB drive does get power.
>         Plugging into a different port triggers detection and the
>         drive can then be mounted. The USB drive I've tested is a
>         Western Digital Elements drive. The same thing happens on
>         multiple PC Engines APU2 units running -current for a while,
>         but unfortunately I procrastinated on reporting this. I can
>         attach the same drive to a desktop and a laptop running
>         current, and they detect it after reboot.  The usbdevs
>         output below is after switching USB ports.
>
> How-To-Repeat:
>         Leave USB drive in port. Reboot. Even removing and putting
>         in same port doesn't detect drive. Only switching to other
>         USB port detects drives.
> Fix:
>         work around is put USB drive in other USB port after reboot.

Is this dmesg showing the boot while the USB disk is attached?

On my APU2 I see something similar, but not exactly the same.  When a
USB device is attached, the controller sometimes comes up in a non-
halted state, and then the halt attempt fails.

Controller already halted at boot, all good:

xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x11: msi, xHCI 1.0
halted=1
i=0
usb0 at xhci0: USB revision 3.0

Controller not halted at boot, halt request fails/timeouts:

xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x11: msi, xHCI 1.0
halted=0
i=100
xhci0: halt timeout
xhci0: reset timeout
xhci0: init failed, error=5

But in my case the controller attachment entirely fails, so switching
port won't help.

It might be some quirk related to the AMD Bolton xHCI controller, but
though it looks like not exactly the same issue.

You could boot again with this diff compiled in, while your USB disk
is attached so we can get more clarity.


Index: sys/dev/usb/xhci.c
===================================================================
RCS file: /cvs/src/sys/dev/usb/xhci.c,v
retrieving revision 1.120
diff -u -p -u -p -r1.120 xhci.c
--- sys/dev/usb/xhci.c 24 Dec 2020 14:11:38 -0000 1.120
+++ sys/dev/usb/xhci.c 20 Feb 2021 10:48:57 -0000
@@ -560,6 +560,9 @@ xhci_reset(struct xhci_softc *sc)
  uint32_t hcr;
  int i;
 
+ hcr = XOREAD4(sc, XHCI_USBSTS) & XHCI_STS_HCH;
+ printf("halted=%d\n", hcr);
+
  XOWRITE4(sc, XHCI_USBCMD, 0); /* Halt controller */
  for (i = 0; i < 100; i++) {
  usb_delay_ms(&sc->sc_bus, 1);
@@ -567,6 +570,7 @@ xhci_reset(struct xhci_softc *sc)
  if (hcr)
  break;
  }
+ printf("i=%d\n", i);
 
  if (!hcr)
  printf("%s: halt timeout\n", DEVNAME(sc));

[Marfaba Stewart]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Saturday, February 20, 2021 4:51 AM, Marcus Glocker <[hidden email]> wrote:

> On Fri, Feb 19, 2021 at 04:48:11PM +0000, marfabastewart wrote:
>
> > Synopsis: USB drive not detected after reboot on PC Engines APU2
> > Category: amd64
> > Environment:
> > System : OpenBSD 6.9
> > Details : OpenBSD 6.9-beta (GENERIC.MP) #343: Thu Feb 18 23:09:55 MST 2021
> > [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> > Architecture: OpenBSD.amd64
> > Machine : amd64
> > Description:
> > After leaving USB drive plugged in and then rebooting on
> > PC Engines APU2, my USB drive isn't detected. If I unplug
> > and replug into the same USB port, it doesn't change console
> > output or dmesg, even though the USB drive does get power.
> > Plugging into a different port triggers detection and the
> > drive can then be mounted. The USB drive I've tested is a
> > Western Digital Elements drive. The same thing happens on
> > multiple PC Engines APU2 units running -current for a while,
> > but unfortunately I procrastinated on reporting this. I can
> > attach the same drive to a desktop and a laptop running
> > current, and they detect it after reboot. The usbdevs
> > output below is after switching USB ports.
> > How-To-Repeat:
> > Leave USB drive in port. Reboot. Even removing and putting
> > in same port doesn't detect drive. Only switching to other
> > USB port detects drives.
> > Fix:
> > work around is put USB drive in other USB port after reboot.
>
> Is this dmesg showing the boot while the USB disk is attached?
>
> On my APU2 I see something similar, but not exactly the same. When a
> USB device is attached, the controller sometimes comes up in a non-
> halted state, and then the halt attempt fails.
>
> Controller already halted at boot, all good:
>
> xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x11: msi, xHCI 1.0
> halted=1
> i=0
> usb0 at xhci0: USB revision 3.0
>
> Controller not halted at boot, halt request fails/timeouts:
>
> xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x11: msi, xHCI 1.0
> halted=0
> i=100
> xhci0: halt timeout
> xhci0: reset timeout
> xhci0: init failed, error=5
>
> But in my case the controller attachment entirely fails, so switching
> port won't help.
>
> It might be some quirk related to the AMD Bolton xHCI controller, but
> though it looks like not exactly the same issue.
>
> You could boot again with this diff compiled in, while your USB disk
> is attached so we can get more clarity.
>
> Index: sys/dev/usb/xhci.c
>
> =====================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================
>
> RCS file: /cvs/src/sys/dev/usb/xhci.c,v
> retrieving revision 1.120
> diff -u -p -u -p -r1.120 xhci.c
> --- sys/dev/usb/xhci.c 24 Dec 2020 14:11:38 -0000 1.120
> +++ sys/dev/usb/xhci.c 20 Feb 2021 10:48:57 -0000
> @@ -560,6 +560,9 @@ xhci_reset(struct xhci_softc *sc)
> uint32_t hcr;
> int i;
>
> -   hcr = XOREAD4(sc, XHCI_USBSTS) & XHCI_STS_HCH;
> -   printf("halted=%d\n", hcr);
> -   XOWRITE4(sc, XHCI_USBCMD, 0); /* Halt controller */
>     for (i = 0; i < 100; i++) {
>     usb_delay_ms(&sc->sc_bus, 1);
>
>
> @@ -567,6 +570,7 @@ xhci_reset(struct xhci_softc *sc)
> if (hcr)
> break;
> }
>
> -   printf("i=%d\n", i);
>
>     if (!hcr)
>     printf("%s: halt timeout\n", DEVNAME(sc));
>

Sorry for the delay. (I really have to build a faster build machine.) (Note: subject line with "SB" was copy/paste fail;
I missed the "U" in "USB" from the synopsis line.)

Below are the dmesg and usbdevs for an APU4 with the patch.
Rebooting with the USB drive plugged in detects the drive.

I will also install on an APU2 and send that as soon as I can.

dmesg:
OpenBSD 6.9-beta (GENERIC.MP) #2: Sun Feb 21 00:10:00 CST 2021
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4259950592 (4062MB)
avail mem = 4115492864 (3924MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xcfe9e020 (13 entries)
bios0: vendor coreboot version "v4.12.0.1" date 05/29/2020
bios0: PC Engines apu4
acpi0 at bios0: ACPI 6.0
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP SSDT MCFG APIC HEST SSDT SSDT HPET
acpi0: wakeup devices PWRB(S4) PBR4(S4) PBR5(S4) PBR6(S4) PBR7(S4) PBR8(S4) UOH1(S3) UOH2(S3) UOH3(S3) UOH4(S3) UOH5(S3) UOH6(S3) XHC0(S4)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimcfg0 at acpi0
acpimcfg0: addr 0xf8000000, bus 0-64
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD GX-412TC SOC, 998.27 MHz, 16-30-01
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu0: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD GX-412TC SOC, 998.13 MHz, 16-30-01
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu1: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu1: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu1: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: AMD GX-412TC SOC, 998.13 MHz, 16-30-01
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu2: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu2: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu2: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: AMD GX-412TC SOC, 998.13 MHz, 16-30-01
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu3: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu3: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu3: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 21, 24 pins
ioapic1 at mainbus0: apid 5 pa 0xfec20000, version 21, 32 pins
acpihpet0 at acpi0: 14318180 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PBR4)
acpiprt2 at acpi0: bus 2 (PBR5)
acpiprt3 at acpi0: bus 3 (PBR6)
acpiprt4 at acpi0: bus 4 (PBR7)
acpiprt5 at acpi0: bus -1 (PBR8)
acpibtn0 at acpi0: PWRB
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
amdgpio0 at acpi0 GPIO uid 0 addr 0xfed81500/0x300 irq 7, 184 pins
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"PRP0001" at acpi0 not configured
"BOOT0000" at acpi0 not configured
acpicpu0 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu1 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu2 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu3 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpitz0 at acpi0: critical temperature is 115 degC
cpu0: 998 MHz: speeds: 1000 800 600 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "AMD 16h Root Complex" rev 0x00
vendor "AMD", unknown product 0x1567 (class system subclass IOMMU, rev 0x00) at pci0 dev 0 function 2 not configured
pchb1 at pci0 dev 2 function 0 "AMD 16h Host" rev 0x00
ppb0 at pci0 dev 2 function 1 "AMD 16h PCIE" rev 0x00: msi
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel I211" rev 0x03: msi, address xxx
ppb1 at pci0 dev 2 function 2 "AMD 16h PCIE" rev 0x00: msi
pci2 at ppb1 bus 2
em1 at pci2 dev 0 function 0 "Intel I211" rev 0x03: msi, address xxx
ppb2 at pci0 dev 2 function 3 "AMD 16h PCIE" rev 0x00: msi
pci3 at ppb2 bus 3
em2 at pci3 dev 0 function 0 "Intel I211" rev 0x03: msi, address xxx
ppb3 at pci0 dev 2 function 4 "AMD 16h PCIE" rev 0x00: msi
pci4 at ppb3 bus 4
em3 at pci4 dev 0 function 0 "Intel I211" rev 0x03: msi, address xxx
ccp0 at pci0 dev 8 function 0 "AMD 16h Crypto" rev 0x00
xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x11: msi, xHCI 1.0
halted=0
i=3
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "AMD xHCI root hub" rev 3.00/1.00 addr 1
ahci0 at pci0 dev 17 function 0 "AMD Hudson-2 SATA" rev 0x40: apic 4 int 19, AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, SATA SSD, SBFM> t10.ATA_SATA_SSD_
sd0: 15272MB, 512 bytes/sector, 31277232 sectors, thin
ehci0 at pci0 dev 19 function 0 "AMD Hudson-2 USB2" rev 0x39: apic 4 int 18
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
piixpm0 at pci0 dev 20 function 0 "AMD Hudson-2 SMBus" rev 0x42: SMI
iic0 at piixpm0
iic1 at piixpm0
iic1: addr 0x4c 3e=00 48=00 4a=00 4e=00 fc=00 fe=00 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
pcib0 at pci0 dev 20 function 3 "AMD Hudson-2 LPC" rev 0x11
sdhc0 at pci0 dev 20 function 7 "AMD Bolton SD/MMC" rev 0x01: apic 4 int 16
sdhc0: SDHC 2.0, 50 MHz base clock
sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed, dma
pchb2 at pci0 dev 24 function 0 "AMD 16h Link Cfg" rev 0x00
pchb3 at pci0 dev 24 function 1 "AMD 16h Address Map" rev 0x00
pchb4 at pci0 dev 24 function 2 "AMD 16h DRAM Cfg" rev 0x00
km0 at pci0 dev 24 function 3 "AMD 16h Misc Cfg" rev 0x00
pchb5 at pci0 dev 24 function 4 "AMD 16h CPU Power" rev 0x00
pchb6 at pci0 dev 24 function 5 "AMD 16h Misc Cfg" rev 0x00
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
intr_establish: pic ioapic0 pin 7: can't share type 3 with 2
wbsio0 at isa0 port 0x2e/2: NCT5104D rev 0x53
vmm0 at mainbus0: SVM/RVI
umass0 at uhub0 port 1 configuration 1 interface 0 "Western Digital Elements 2620" rev 3.10/10.18 addr 2
umass0: using SCSI over Bulk-Only
scsibus2 at umass0: 2 targets, initiator 0
sd1 at scsibus2 targ 1 lun 0: <WD, Elements 2620, 1018> serial.
sd1: 953837MB, 512 bytes/sector, 1953458176 sectors
uhub2 at uhub1 port 1 configuration 1 interface 0 "Advanced Micro Devices Hub" rev 2.00/0.18 addr 2
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (xxx.a) swap on sd0b dump on sd0b

usbdevs:
Controller /dev/usb0:
addr 01: 1022:0000 AMD, xHCI root hub
         super speed, self powered, config 1, rev 1.00
         driver: uhub0
addr 02: 1058:2620 Western Digital, Elements 2620
         super speed, power 224 mA, config 1, rev 10.18, iSerial
         driver: umass0
Controller /dev/usb1:
addr 01: 1022:0000 AMD, EHCI root hub
         high speed, self powered, config 1, rev 1.00
         driver: uhub1
addr 02: 0438:7900 Advanced Micro Devices, Hub
         high speed, self powered, config 1, rev 0.18
         driver: uhub2

[Marfaba Stewart]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Sunday, February 21, 2021 9:47 AM, marfabastewart <[hidden email]> wrote:

> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Saturday, February 20, 2021 4:51 AM, Marcus Glocker [hidden email] wrote:
>
> > On Fri, Feb 19, 2021 at 04:48:11PM +0000, marfabastewart wrote:
> >
> > > Synopsis: USB drive not detected after reboot on PC Engines APU2
> > > Category: amd64
> > > Environment:
> > > System : OpenBSD 6.9
> > > Details : OpenBSD 6.9-beta (GENERIC.MP) #343: Thu Feb 18 23:09:55 MST 2021
> > > [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> > > Architecture: OpenBSD.amd64
> > > Machine : amd64
> > > Description:
> > > After leaving USB drive plugged in and then rebooting on
> > > PC Engines APU2, my USB drive isn't detected. If I unplug
> > > and replug into the same USB port, it doesn't change console
> > > output or dmesg, even though the USB drive does get power.
> > > Plugging into a different port triggers detection and the
> > > drive can then be mounted. The USB drive I've tested is a
> > > Western Digital Elements drive. The same thing happens on
> > > multiple PC Engines APU2 units running -current for a while,
> > > but unfortunately I procrastinated on reporting this. I can
> > > attach the same drive to a desktop and a laptop running
> > > current, and they detect it after reboot. The usbdevs
> > > output below is after switching USB ports.
> > > How-To-Repeat:
> > > Leave USB drive in port. Reboot. Even removing and putting
> > > in same port doesn't detect drive. Only switching to other
> > > USB port detects drives.
> > > Fix:
> > > work around is put USB drive in other USB port after reboot.
> >
> > Is this dmesg showing the boot while the USB disk is attached?
> > On my APU2 I see something similar, but not exactly the same. When a
> > USB device is attached, the controller sometimes comes up in a non-
> > halted state, and then the halt attempt fails.
> > Controller already halted at boot, all good:
> > xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x11: msi, xHCI 1.0
> > halted=1
> > i=0
> > usb0 at xhci0: USB revision 3.0
> > Controller not halted at boot, halt request fails/timeouts:
> > xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x11: msi, xHCI 1.0
> > halted=0
> > i=100
> > xhci0: halt timeout
> > xhci0: reset timeout
> > xhci0: init failed, error=5
> > But in my case the controller attachment entirely fails, so switching
> > port won't help.
> > It might be some quirk related to the AMD Bolton xHCI controller, but
> > though it looks like not exactly the same issue.
> > You could boot again with this diff compiled in, while your USB disk
> > is attached so we can get more clarity.
> > Index: sys/dev/usb/xhci.c
> > =====================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================
> > RCS file: /cvs/src/sys/dev/usb/xhci.c,v
> > retrieving revision 1.120
> > diff -u -p -u -p -r1.120 xhci.c
> > --- sys/dev/usb/xhci.c 24 Dec 2020 14:11:38 -0000 1.120
> > +++ sys/dev/usb/xhci.c 20 Feb 2021 10:48:57 -0000
> > @@ -560,6 +560,9 @@ xhci_reset(struct xhci_softc *sc)
> > uint32_t hcr;
> > int i;
> >
> > -   hcr = XOREAD4(sc, XHCI_USBSTS) & XHCI_STS_HCH;
> > -   printf("halted=%d\n", hcr);
> > -   XOWRITE4(sc, XHCI_USBCMD, 0); /* Halt controller */
> >     for (i = 0; i < 100; i++) {
> >     usb_delay_ms(&sc->sc_bus, 1);
> >
> >
> > @@ -567,6 +570,7 @@ xhci_reset(struct xhci_softc *sc)
> > if (hcr)
> > break;
> > }
> >
> > -   printf("i=%d\n", i);
> >     if (!hcr)
> >     printf("%s: halt timeout\n", DEVNAME(sc));
> >
>
> Sorry for the delay. (I really have to build a faster build machine.) (Note: subject line with "SB" was copy/paste fail;
> I missed the "U" in "USB" from the synopsis line.)
>
> Below are the dmesg and usbdevs for an APU4 with the patch.
> Rebooting with the USB drive plugged in detects the drive.
>
> I will also install on an APU2 and send that as soon as I can.
>
> dmesg:
> OpenBSD 6.9-beta (GENERIC.MP) #2: Sun Feb 21 00:10:00 CST 2021
> [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 4259950592 (4062MB)
> avail mem = 4115492864 (3924MB)
> random: good seed from bootblocks
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xcfe9e020 (13 entries)
> bios0: vendor coreboot version "v4.12.0.1" date 05/29/2020
> bios0: PC Engines apu4
> acpi0 at bios0: ACPI 6.0
> acpi0: sleep states S0 S1 S4 S5
> acpi0: tables DSDT FACP SSDT MCFG APIC HEST SSDT SSDT HPET
> acpi0: wakeup devices PWRB(S4) PBR4(S4) PBR5(S4) PBR6(S4) PBR7(S4) PBR8(S4) UOH1(S3) UOH2(S3) UOH3(S3) UOH4(S3) UOH5(S3) UOH6(S3) XHC0(S4)
> acpitimer0 at acpi0: 3579545 Hz, 32 bits
> acpimcfg0 at acpi0
> acpimcfg0: addr 0xf8000000, bus 0-64
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: AMD GX-412TC SOC, 998.27 MHz, 16-30-01
> cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
> cpu0: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
> cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
> cpu0: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 99MHz
> cpu0: mwait min=64, max=64, IBE
> cpu1 at mainbus0: apid 1 (application processor)
> cpu1: AMD GX-412TC SOC, 998.13 MHz, 16-30-01
> cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
> cpu1: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
> cpu1: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
> cpu1: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
> cpu1: smt 0, core 1, package 0
> cpu2 at mainbus0: apid 2 (application processor)
> cpu2: AMD GX-412TC SOC, 998.13 MHz, 16-30-01
> cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
> cpu2: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
> cpu2: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
> cpu2: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
> cpu2: smt 0, core 2, package 0
> cpu3 at mainbus0: apid 3 (application processor)
> cpu3: AMD GX-412TC SOC, 998.13 MHz, 16-30-01
> cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
> cpu3: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
> cpu3: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
> cpu3: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
> cpu3: smt 0, core 3, package 0
> ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 21, 24 pins
> ioapic1 at mainbus0: apid 5 pa 0xfec20000, version 21, 32 pins
> acpihpet0 at acpi0: 14318180 Hz
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 1 (PBR4)
> acpiprt2 at acpi0: bus 2 (PBR5)
> acpiprt3 at acpi0: bus 3 (PBR6)
> acpiprt4 at acpi0: bus 4 (PBR7)
> acpiprt5 at acpi0: bus -1 (PBR8)
> acpibtn0 at acpi0: PWRB
> acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
> acpicmos0 at acpi0
> amdgpio0 at acpi0 GPIO uid 0 addr 0xfed81500/0x300 irq 7, 184 pins
> "PRP0001" at acpi0 not configured
> "PRP0001" at acpi0 not configured
> "PRP0001" at acpi0 not configured
> "PRP0001" at acpi0 not configured
> "PRP0001" at acpi0 not configured
> "PRP0001" at acpi0 not configured
> "BOOT0000" at acpi0 not configured
> acpicpu0 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
> acpicpu1 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
> acpicpu2 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
> acpicpu3 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
> acpitz0 at acpi0: critical temperature is 115 degC
> cpu0: 998 MHz: speeds: 1000 800 600 MHz
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "AMD 16h Root Complex" rev 0x00
> vendor "AMD", unknown product 0x1567 (class system subclass IOMMU, rev 0x00) at pci0 dev 0 function 2 not configured
> pchb1 at pci0 dev 2 function 0 "AMD 16h Host" rev 0x00
> ppb0 at pci0 dev 2 function 1 "AMD 16h PCIE" rev 0x00: msi
> pci1 at ppb0 bus 1
> em0 at pci1 dev 0 function 0 "Intel I211" rev 0x03: msi, address xxx
> ppb1 at pci0 dev 2 function 2 "AMD 16h PCIE" rev 0x00: msi
> pci2 at ppb1 bus 2
> em1 at pci2 dev 0 function 0 "Intel I211" rev 0x03: msi, address xxx
> ppb2 at pci0 dev 2 function 3 "AMD 16h PCIE" rev 0x00: msi
> pci3 at ppb2 bus 3
> em2 at pci3 dev 0 function 0 "Intel I211" rev 0x03: msi, address xxx
> ppb3 at pci0 dev 2 function 4 "AMD 16h PCIE" rev 0x00: msi
> pci4 at ppb3 bus 4
> em3 at pci4 dev 0 function 0 "Intel I211" rev 0x03: msi, address xxx
> ccp0 at pci0 dev 8 function 0 "AMD 16h Crypto" rev 0x00
> xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x11: msi, xHCI 1.0
> halted=0
> i=3
> usb0 at xhci0: USB revision 3.0
> uhub0 at usb0 configuration 1 interface 0 "AMD xHCI root hub" rev 3.00/1.00 addr 1
> ahci0 at pci0 dev 17 function 0 "AMD Hudson-2 SATA" rev 0x40: apic 4 int 19, AHCI 1.3
> ahci0: port 0: 6.0Gb/s
> scsibus1 at ahci0: 32 targets
> sd0 at scsibus1 targ 0 lun 0: <ATA, SATA SSD, SBFM> t10.ATA_SATA_SSD_
> sd0: 15272MB, 512 bytes/sector, 31277232 sectors, thin
> ehci0 at pci0 dev 19 function 0 "AMD Hudson-2 USB2" rev 0x39: apic 4 int 18
> usb1 at ehci0: USB revision 2.0
> uhub1 at usb1 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
> piixpm0 at pci0 dev 20 function 0 "AMD Hudson-2 SMBus" rev 0x42: SMI
> iic0 at piixpm0
> iic1 at piixpm0
> iic1: addr 0x4c 3e=00 48=00 4a=00 4e=00 fc=00 fe=00 words 00=ffff 01=ffff 02=ffff 03=ffff 04=ffff 05=ffff 06=ffff 07=ffff
> pcib0 at pci0 dev 20 function 3 "AMD Hudson-2 LPC" rev 0x11
> sdhc0 at pci0 dev 20 function 7 "AMD Bolton SD/MMC" rev 0x01: apic 4 int 16
> sdhc0: SDHC 2.0, 50 MHz base clock
> sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed, dma
> pchb2 at pci0 dev 24 function 0 "AMD 16h Link Cfg" rev 0x00
> pchb3 at pci0 dev 24 function 1 "AMD 16h Address Map" rev 0x00
> pchb4 at pci0 dev 24 function 2 "AMD 16h DRAM Cfg" rev 0x00
> km0 at pci0 dev 24 function 3 "AMD 16h Misc Cfg" rev 0x00
> pchb5 at pci0 dev 24 function 4 "AMD 16h CPU Power" rev 0x00
> pchb6 at pci0 dev 24 function 5 "AMD 16h Misc Cfg" rev 0x00
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com0: console
> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> lpt0 at isa0 port 0x378/4 irq 7
> intr_establish: pic ioapic0 pin 7: can't share type 3 with 2
> wbsio0 at isa0 port 0x2e/2: NCT5104D rev 0x53
> vmm0 at mainbus0: SVM/RVI
> umass0 at uhub0 port 1 configuration 1 interface 0 "Western Digital Elements 2620" rev 3.10/10.18 addr 2
> umass0: using SCSI over Bulk-Only
> scsibus2 at umass0: 2 targets, initiator 0
> sd1 at scsibus2 targ 1 lun 0: <WD, Elements 2620, 1018> serial.
> sd1: 953837MB, 512 bytes/sector, 1953458176 sectors
> uhub2 at uhub1 port 1 configuration 1 interface 0 "Advanced Micro Devices Hub" rev 2.00/0.18 addr 2
> vscsi0 at root
> scsibus3 at vscsi0: 256 targets
> softraid0 at root
> scsibus4 at softraid0: 256 targets
> root on sd0a (xxx.a) swap on sd0b dump on sd0b
>
> usbdevs:
> Controller /dev/usb0:
> addr 01: 1022:0000 AMD, xHCI root hub
> super speed, self powered, config 1, rev 1.00
> driver: uhub0
> addr 02: 1058:2620 Western Digital, Elements 2620
> super speed, power 224 mA, config 1, rev 10.18, iSerial
> driver: umass0
> Controller /dev/usb1:
> addr 01: 1022:0000 AMD, EHCI root hub
> high speed, self powered, config 1, rev 1.00
> driver: uhub1
> addr 02: 0438:7900 Advanced Micro Devices, Hub
> high speed, self powered, config 1, rev 0.18
> driver: uhub2

------------------------

and this is the output of sendbug -P on an APU2
with the patch compiled in, after plugging in
the USB drive and rebooting. The USB drive is not
detected on the APU2 on boot, but the lines
from the patch are generated:
halted=1
i=0

output below:
dmesg:
OpenBSD 6.9-beta (GENERIC.MP) #2: Sun Feb 21 00:10:00 CST 2021
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4261076992 (4063MB)
avail mem = 4116594688 (3925MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xdffb7020 (7 entries)
bios0: vendor coreboot version "4.0.7" date 02/28/2017
bios0: PC Engines APU2
acpi0 at bios0: ACPI 4.0
acpi0: sleep states S0 S1 S2 S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC HEST SSDT SSDT HPET
acpi0: wakeup devices PWRB(S4) PBR4(S4) PBR5(S4) PBR6(S4) PBR7(S4) PBR8(S4) UOH1(S3) UOH3(S3) UOH5(S3) XHC0(S4)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD GX-412TC SOC, 998.26 MHz, 16-30-01
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu0: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD GX-412TC SOC, 998.17 MHz, 16-30-01
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu1: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu1: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu1: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: AMD GX-412TC SOC, 998.22 MHz, 16-30-01
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu2: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu2: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu2: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: AMD GX-412TC SOC, 998.13 MHz, 16-30-01
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PERFTSC,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu3: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu3: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu3: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 21, 24 pins
ioapic1 at mainbus0: apid 5 pa 0xfec20000, version 21, 32 pins, remapped
acpihpet0 at acpi0: 14318180 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus -1 (PBR4)
acpiprt2 at acpi0: bus 1 (PBR5)
acpiprt3 at acpi0: bus 2 (PBR6)
acpiprt4 at acpi0: bus 3 (PBR7)
acpiprt5 at acpi0: bus -1 (PBR8)
acpibtn0 at acpi0: PWRB
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
acpicpu0 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu1 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu2 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
acpicpu3 at acpi0: C2(0@400 io@0x1771), C1(@1 halt!), PSS
cpu0: 998 MHz: speeds: 1000 800 600 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "AMD 16h Root Complex" rev 0x00
pchb1 at pci0 dev 2 function 0 "AMD 16h Host" rev 0x00
ppb0 at pci0 dev 2 function 2 "AMD 16h PCIE" rev 0x00: msi
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel I210" rev 0x03: msi, address
ppb1 at pci0 dev 2 function 3 "AMD 16h PCIE" rev 0x00: msi
pci2 at ppb1 bus 2
em1 at pci2 dev 0 function 0 "Intel I210" rev 0x03: msi, address
ppb2 at pci0 dev 2 function 4 "AMD 16h PCIE" rev 0x00: msi
pci3 at ppb2 bus 3
em2 at pci3 dev 0 function 0 "Intel I210" rev 0x03: msi, address
ccp0 at pci0 dev 8 function 0 "AMD 16h Crypto" rev 0x00
xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x11: msi, xHCI 1.0
halted=1
i=0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "AMD xHCI root hub" rev 3.00/1.00 addr 1
ahci0 at pci0 dev 17 function 0 "AMD Hudson-2 SATA" rev 0x40: apic 4 int 19, AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, SATA SSD, SBFM> naa.0000000000000000
sd0: 57241MB, 512 bytes/sector, 117231408 sectors, thin
ehci0 at pci0 dev 19 function 0 "AMD Hudson-2 USB2" rev 0x39: apic 4 int 18
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
piixpm0 at pci0 dev 20 function 0 "AMD Hudson-2 SMBus" rev 0x42: SMI
iic0 at piixpm0
iic1 at piixpm0
pcib0 at pci0 dev 20 function 3 "AMD Hudson-2 LPC" rev 0x11
sdhc0 at pci0 dev 20 function 7 "AMD Bolton SD/MMC" rev 0x01: apic 4 int 16
sdhc0: SDHC 2.0, 50 MHz base clock
sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed, dma
pchb2 at pci0 dev 24 function 0 "AMD 16h Link Cfg" rev 0x00
pchb3 at pci0 dev 24 function 1 "AMD 16h Address Map" rev 0x00
pchb4 at pci0 dev 24 function 2 "AMD 16h DRAM Cfg" rev 0x00
km0 at pci0 dev 24 function 3 "AMD 16h Misc Cfg" rev 0x00
pchb5 at pci0 dev 24 function 4 "AMD 16h CPU Power" rev 0x00
pchb6 at pci0 dev 24 function 5 "AMD 16h Misc Cfg" rev 0x00
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
wbsio0 at isa0 port 0x2e/2: NCT5104D rev 0x53
vmm0 at mainbus0: SVM/RVI
uhub2 at uhub1 port 1 configuration 1 interface 0 "Advanced Micro Devices Hub" rev 2.00/0.18 addr 2
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (xxx.a) swap on sd0b dump on sd0b

usbdevs:
Controller /dev/usb0:
addr 01: 1022:0000 AMD, xHCI root hub
         super speed, self powered, config 1, rev 1.00
         driver: uhub0
Controller /dev/usb1:
addr 01: 1022:0000 AMD, EHCI root hub
         high speed, self powered, config 1, rev 1.00
         driver: uhub1
addr 02: 0438:7900 Advanced Micro Devices, Hub
         high speed, self powered, config 1, rev 0.18
         driver: uhub2

[Marcus Glocker]

On Sun, Feb 21, 2021 at 04:45:19PM +0000, marfabastewart wrote:

[...]
> and this is the output of sendbug -P on an APU2
> with the patch compiled in, after plugging in
> the USB drive and rebooting. The USB drive is not
> detected on the APU2 on boot, but the lines
> from the patch are generated:
> halted=1
> i=0
>
> output below:

Thanks for the reports.

I have tried various variants to change the halt/reset cycle slightly.
But unfortunately nothing helps.  It seems that when a certain kind
of USB device (like the umass) is plugged in to the AMD Bolton
controller, something goes not right with the device halt request.

I have no more ideas currently how to tackle this.

[Marfaba Stewart]

> On Sun, Feb 21, 2021 at 04:45:19PM +0000, marfabastewart wrote:
>
> [...]
>
> > and this is the output of sendbug -P on an APU2
> > with the patch compiled in, after plugging in
> > the USB drive and rebooting. The USB drive is not
> > detected on the APU2 on boot, but the lines
> > from the patch are generated:
> > halted=1
> > i=0
> > output below:
>
> Thanks for the reports.
>
> I have tried various variants to change the halt/reset cycle slightly.
> But unfortunately nothing helps. It seems that when a certain kind
> of USB device (like the umass) is plugged in to the AMD Bolton
> controller, something goes not right with the device halt request.

Thank you very much for looking into it. I emailed PCEngines
just now to see if they can provide any more information
about the possible differences between the USB controllers
on the APU2 and APU4. I will let you know when they reply.

[Stuart Henderson]

On 2021/02/23 13:48, Marfaba Stewart wrote:

> > On Sun, Feb 21, 2021 at 04:45:19PM +0000, marfabastewart wrote:
> >
> > [...]
> >
> > > and this is the output of sendbug -P on an APU2
> > > with the patch compiled in, after plugging in
> > > the USB drive and rebooting. The USB drive is not
> > > detected on the APU2 on boot, but the lines
> > > from the patch are generated:
> > > halted=1
> > > i=0
> > > output below:
> >
> > Thanks for the reports.
> >
> > I have tried various variants to change the halt/reset cycle slightly.
> > But unfortunately nothing helps. It seems that when a certain kind
> > of USB device (like the umass) is plugged in to the AMD Bolton
> > controller, something goes not right with the device halt request.
>
> Thank you very much for looking into it. I emailed PCEngines
> just now to see if they can provide any more information
> about the possible differences between the USB controllers
> on the APU2 and APU4. I will let you know when they reply.
>

They use the same system-on-chip, the differences are in network/pcie
configuration. The different firmware you are running on the two
machines is much more likely to be the difference, you have:

bios0: vendor coreboot version "4.0.7" date 02/28/2017
bios0: vendor coreboot version "v4.12.0.1" date 05/29/2020

Note that flashrom allows updating firmware on pcengines APU from OpenBSD.
flashrom's pkg-readme file has everything you need to know.

[Marfaba Stewart]

On Tuesday, February 23, 2021 8:40 AM, Stuart Henderson <[hidden email]> wrote:

[...]

> bios0: vendor coreboot version "4.0.7" date 02/28/2017
> bios0: vendor coreboot version "v4.12.0.1" date 05/29/2020
>
> Note that flashrom allows updating firmware on pcengines APU from OpenBSD.
> flashrom's pkg-readme file has everything you need to know.

Thank you very much! I will update and post my results
for the APU2.

I read PCEngines documentation after the fact (oops!) and
noticed that coreboot version v.4.10.0.x (and below, I
take it) have this same problem on Debian, Voyage,
OPNSense, and pfSense. Their note below is about
v.4.10.0.x:

"all apu boards have problems with USB 3.x stick detection in BIOS, system is working properly, but after reboot/warmboot/coldboot USB stick may not appear in BIOS boot menu (depends on the stick, well working sticks are mentioned in this document)"