ldom toolsuite (ldomd/ldomctl) fails to start on a T4-2

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

ldom toolsuite (ldomd/ldomctl) fails to start on a T4-2

Koakuma
Synopsis: ldom toolsuite (ldomd/ldomctl) fails to start on a T4-2
Category: user sparc64
Environment:
        System      : OpenBSD 6.7
        Details     : OpenBSD 6.7-stable (GENERIC.MP) #1: Tue Jun 16 18:28:00 WIB 2020
                         koakuma@ramen:/usr/src/sys/arch/sparc64/compile/GENERIC.MP

        Architecture: OpenBSD.sparc64
        Machine     : sparc64

Description:
        This is a T4-2 running OpenBSD 6.7-stable built according to
        the guide in https://www.openbsd.org/stable.html.

        Both ldomctl and ldomd always crashes on this machine. I'm not sure if it's
        related or not, so I'm including both the symptoms here for completeness.

        1. When running "ldomctl init-system ldom.conf", ldomctl crashes with
        a segmentation fault. Running a gdb backtrace on the core file gives me this:

        #0  0x000000b69bd0506c in md_add_prop () from /usr/sbin/ldomctl
        #1  0x000000b69bd050b4 in md_add_prop_arc () from /usr/sbin/ldomctl
        #2  0x000000b69bd0e2a8 in hvmd_finalize_pcie_device () from /usr/sbin/ldomctl
        #3  0x000000b69bd0e4d0 in hvmd_finalize_devices () from /usr/sbin/ldomctl
        #4  0x000000b69bd0e7c4 in hvmd_finalize () from /usr/sbin/ldomctl
        #5  0x000000b69bd0fe90 in build_config () from /usr/sbin/ldomctl
        #6  0x000000b69bd0238c in main () from /usr/sbin/ldomctl

        2. ldomd always crashes 10-30 minutes after starting. Running it manually
        with "ldomd -d" gives me this error:

        ldomd: Cannot allocate memory

        There's no core files or other logs that I can found, though.

How-To-Repeat:
        1. For the ldomctl crash, I followed the examples in
        ldom.conf(5) and ldomctl(8) manual pages:

        # ldomctl list
        factory-default [current]
        # mkdir factory-default && cd factory-default
        # ldomctl dump
        # cp -R factory-default openbsd
        # cd openbsd/
        # ldomctl create-vdisk -s 6G /storage/vmimages/obsd.img
        # cat <<EOF >ldom.conf
        domain "primary" {
                vcpu 4
                memory 4G
        }

        domain "obsd" {
                vcpu 2
                memory 2G
                vdisk "/storage/vmimages/obsd.img"
                vnet
        }
        EOF
        # ldomctl init-system ldom.conf
        Segmentation fault (core dumped)

        2. For the ldomd crash, to get the message I started it manually
        with "ldomd -d" and wait for 10-30 mins for it to crash.

        # ldomd -d
        ldomd: Cannot allocate memory

dmesg:
OpenBSD 6.7-stable (GENERIC.MP) #1: Tue Jun 16 18:28:00 WIB 2020
    koakuma@ramen:/usr/src/sys/arch/sparc64/compile/GENERIC.MP
real mem = 51002736640 (48640MB)
avail mem = 50083602432 (47763MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root: SPARC T4-2
cpu0 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu1 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu2 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu3 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu4 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu5 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu6 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu7 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu8 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu9 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu10 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu11 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu12 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu13 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu14 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu15 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu16 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu17 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu18 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu19 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu20 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu21 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu22 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu23 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu24 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu25 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu26 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu27 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu28 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu29 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu30 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu31 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu32 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu33 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu34 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu35 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu36 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu37 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu38 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu39 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu40 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu41 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu42 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu43 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu44 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu45 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu46 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu47 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu48 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu49 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu50 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu51 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu52 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu53 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu54 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu55 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu56 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu57 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu58 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu59 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu60 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu61 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu62 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu63 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu64 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu65 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu66 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu67 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu68 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu69 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu70 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu71 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu72 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu73 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu74 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu75 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu76 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu77 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu78 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu79 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu80 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu81 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu82 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu83 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu84 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu85 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu86 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu87 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu88 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu89 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu90 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu91 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu92 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu93 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu94 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu95 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu96 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu97 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu98 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu99 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu100 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu101 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu102 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu103 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu104 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu105 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu106 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu107 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu108 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu109 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu110 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu111 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu112 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu113 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu114 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu115 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu116 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu117 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu118 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu119 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu120 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu121 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu122 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu123 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu124 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu125 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu126 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu127 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
vbus0 at mainbus0
"flashprom" at vbus0 not configured
"tpm" at vbus0 not configured
cbus0 at vbus0
vldc0 at cbus0
vldcp0 at vldc0 chan 0x0: ivec 0x0, 0x1 channel "hvctl"
"ldom-primary" at vldc0 chan 0x1 not configured
"fmactl" at vldc0 chan 0x3 not configured
vldc1 at cbus0
"ldmfma" at vldc1 chan 0x4 not configured
vldc2 at cbus0
"spfma" at vldc2 chan 0x5 not configured
vldc3 at cbus0
vldcp1 at vldc3 chan 0x14: ivec 0x28, 0x29 channel "spds"
"sunvts" at vldc3 chan 0x6 not configured
"sunmc" at vldc3 chan 0x7 not configured
"explorer" at vldc3 chan 0x8 not configured
"flashupdate" at vldc3 chan 0x9 not configured
"ipmi" at vldc3 chan 0xb not configured
"virtual-domain-service" at cbus0 not configured
vcons0 at vbus0: ivec 0x111, console
vrtc0 at vbus0
vpci0 at mainbus0: bus 0 to 15, dvma map 80000000-ffffffff
pci0 at vpci0
ppb0 at pci0 dev 1 function 0 "Sun SPARC-T3/T4 PCIE" rev 0x01
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02
pci2 at ppb1 bus 2
ppb2 at pci2 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02
pci3 at ppb2 bus 3
ppb3 at pci3 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x02
pci4 at ppb3 bus 4
"ASPEED Technology AST2000" rev 0x10 at pci4 dev 0 function 0 not configured
ppb4 at pci2 dev 4 function 0 "IDT 89HPES64H16G2" rev 0x02
pci5 at ppb4 bus 5
em0 at pci5 dev 0 function 0 "Intel 82576" rev 0x01: msi, address 00:21:28:f1:e2:f6
em1 at pci5 dev 0 function 1 "Intel 82576" rev 0x01: msi, address 00:21:28:f1:e2:f7
ppb5 at pci2 dev 8 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci6 at ppb5 bus 6
ppb6 at pci2 dev 11 function 0 "IDT 89HPES64H16G2" rev 0x02
pci7 at ppb6 bus 7
ppb7 at pci7 dev 0 function 0 "PLX PEX 8112" rev 0xaa
pci8 at ppb7 bus 8
ohci0 at pci8 dev 0 function 0 "NEC USB" rev 0x43: ivec 0x401, version 1.0
ohci1 at pci8 dev 0 function 1 "NEC USB" rev 0x43: ivec 0x402, version 1.0
ehci0 at pci8 dev 0 function 2 "NEC USB" rev 0x04: ivec 0x403
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "NEC EHCI root hub" rev 2.00/1.00 addr 1
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "NEC OHCI root hub" rev 1.00/1.00 addr 1
usb2 at ohci1: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "NEC OHCI root hub" rev 1.00/1.00 addr 1
ppb8 at pci2 dev 12 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci9 at ppb8 bus 9
ppb9 at pci0 dev 2 function 0 "Sun SPARC-T3/T4 PCIE" rev 0x01
pci10 at ppb9 bus 10
ppb10 at pci10 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02
pci11 at ppb10 bus 11
ppb11 at pci11 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci12 at ppb11 bus 12
ppb12 at pci11 dev 4 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci13 at ppb12 bus 13
ppb13 at pci11 dev 8 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci14 at ppb13 bus 14
ppb14 at pci11 dev 14 function 0 "IDT 89HPES64H16G2" rev 0x02
pci15 at ppb14 bus 15
mpii0 at pci15 dev 0 function 0 "Symbios Logic SAS2008" rev 0x03: msi
mpii0: Solana On-Board, firmware 9.0.0.0 IR, MPI 2.0
scsibus1 at mpii0: 834 targets
vpci1 at mainbus0: bus 0 to 10, dvma map 80000000-ffffffff
pci16 at vpci1
ppb15 at pci16 dev 1 function 0 "Sun SPARC-T3/T4 PCIE" rev 0x01
pci17 at ppb15 bus 1
ppb16 at pci17 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02
pci18 at ppb16 bus 2
ppb17 at pci18 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci19 at ppb17 bus 3
ppb18 at pci18 dev 5 function 0 "IDT 89HPES64H16G2" rev 0x02
pci20 at ppb18 bus 4
em2 at pci20 dev 0 function 0 "Intel 82576" rev 0x01: msi, address 00:21:28:f1:e2:f8
em3 at pci20 dev 0 function 1 "Intel 82576" rev 0x01: msi, address 00:21:28:f1:e2:f9
ppb19 at pci18 dev 6 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci21 at ppb19 bus 5
ppb20 at pci16 dev 2 function 0 "Sun SPARC-T3/T4 PCIE" rev 0x01
pci22 at ppb20 bus 6
ppb21 at pci22 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02
pci23 at ppb21 bus 7
ppb22 at pci23 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci24 at ppb22 bus 8
ppb23 at pci23 dev 6 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci25 at ppb23 bus 9
ppb24 at pci23 dev 10 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci26 at ppb24 bus 10
"niu" at mainbus0 not configured
"niu" at mainbus0 not configured
"pci-performance-counters" at mainbus0 not configured
"pci-performance-counters" at mainbus0 not configured
sd0 at scsibus1 targ 2 lun 0: <HITACHI, H103014SCSUN146G, A2A8> naa.5000cca0152238bc
sd0: 140009MB, 512 bytes/sector, 286739329 sectors
cd0 at scsibus1 targ 9 lun 0: <TEAC, DV-W28SS-R, 1.0C> removable
uhub3 at uhub0 port 2 configuration 1 interface 0 "Cypress Semiconductor USB2 Hub" rev 2.00/0.0b addr 2
ugen0 at uhub3 port 3 "American Megatrends Inc. Generic Hub" rev 2.00/1.00 addr 3
uhidev0 at uhub3 port 4 configuration 1 interface 0 "American Megatrends Inc. Virtual Keyboard and Mouse" rev 1.10/1.00 addr 4
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0 mux 1
uhidev1 at uhub3 port 4 configuration 1 interface 1 "American Megatrends Inc. Virtual Keyboard and Mouse" rev 1.10/1.00 addr 4
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons
wsmouse0 at ums0 mux 0
uhub4 at uhub0 port 4 configuration 1 interface 0 "Cypress Semiconductor USB2 Hub" rev 2.00/0.0b addr 5
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
bootpath: /pci@400,0/pci@2,0/pci@0,0/pci@e,0/scsi@0,0/disk@5000cca0152238bd,0
root on sd0a (442f26575bfc7d59.a) swap on sd0b dump on sd0b

usbdevs:
Controller /dev/usb0:
addr 01: 1033:0000 NEC, EHCI root hub
         high speed, self powered, config 1, rev 1.00
         driver: uhub0
addr 02: 04b4:6560 Cypress Semiconductor, USB2 Hub
         high speed, self powered, config 1, rev 0.0b
         driver: uhub3
addr 03: 046b:ff01 American Megatrends Inc., Generic Hub
         high speed, self powered, config 1, rev 1.00
         driver: ugen0
addr 04: 046b:ff10 American Megatrends Inc., Virtual Keyboard and Mouse
         low speed, self powered, config 1, rev 1.00
         driver: uhidev0
         driver: uhidev1
addr 05: 04b4:6560 Cypress Semiconductor, USB2 Hub
         high speed, self powered, config 1, rev 0.0b
         driver: uhub4
Controller /dev/usb1:
addr 01: 1033:0000 NEC, OHCI root hub
         full speed, self powered, config 1, rev 1.00
         driver: uhub1
Controller /dev/usb2:
addr 01: 1033:0000 NEC, OHCI root hub
         full speed, self powered, config 1, rev 1.00
         driver: uhub2

pcidump:
Domain /dev/pci0:
 0:1:0: Sun SPARC-T3/T4 PCIE
        0x0000: Vendor ID: 108e, Product ID: 8186
        0x0004: Command: 0007, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 01
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 0, Secondary Bus: 1, Subordinate Bus: 9,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 01, I/O Limit: 11, Secondary Status: 0000
        0x0020: Memory Base: 0010, Memory Limit: 0110
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x01: Power Management
                State: D0
        0x0060: Capability 0x10: PCI Express
                Link Speed: 5.0 / 5.0 GT/s, Link Width: x8 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0138: Enhanced Capability 0x0b: Vendor-Specific
        0x014c: Enhanced Capability 0x0d: Access Control Services
        0x0000: 8186108e 00100007 06040001 00010000
        0x0010: 00000000 00000000 00090100 00001101
        0x0020: 01100010 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: c8036001 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00420010 00008001 00000000 00395c82
        0x0070: 70820000 00000000 01000000 00000000
        0x0080: 00000000 00000020 00000000 00000000
        0x0090: 00000002 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:2:0: Sun SPARC-T3/T4 PCIE
        0x0000: Vendor ID: 108e, Product ID: 8186
        0x0004: Command: 0007, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 01
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 0, Secondary Bus: 10, Subordinate Bus: 15,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 21, I/O Limit: 21, Secondary Status: 0000
        0x0020: Memory Base: 0120, Memory Limit: 0120
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x01: Power Management
                State: D0
        0x0060: Capability 0x10: PCI Express
                Link Speed: 5.0 / 5.0 GT/s, Link Width: x8 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0138: Enhanced Capability 0x0b: Vendor-Specific
        0x014c: Enhanced Capability 0x0d: Access Control Services
        0x0000: 8186108e 00100007 06040001 00010000
        0x0010: 00000000 00000000 000f0a00 00002121
        0x0020: 01200120 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: c8036001 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00420010 00008001 00000000 01395c82
        0x0070: 70820000 00000000 01000000 00000000
        0x0080: 00000000 00000020 00000000 00000000
        0x0090: 00000002 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 1:0:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 1, Secondary Bus: 2, Subordinate Bus: 9,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 01, I/O Limit: 11, Secondary Status: 0000
        0x0020: Memory Base: 0010, Memory Limit: 0110
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 5.0 / 5.0 GT/s, Link Width: x8 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x00: Reserved
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 00090201 00001101
        0x0020: 01100010 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0052c010 00008024 00000000 00016c82
        0x0050: 00820000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000002 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 2:0:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 2, Secondary Bus: 3, Subordinate Bus: 4,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 01, I/O Limit: 01, Secondary Status: 0000
        0x0020: Memory Base: 0010, Memory Limit: 00f0
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 5.0 GT/s, Link Width: x1 / x4
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: no
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 00040302 00000101
        0x0020: 00f00010 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0062c010 00008024 00000000 0a396c42
        0x0050: 20110000 00000000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00800005 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 2:4:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 2, Secondary Bus: 5, Subordinate Bus: 5,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 11, I/O Limit: 11, Secondary Status: 0000
        0x0020: Memory Base: 0100, Memory Limit: 0100
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 5.0 GT/s, Link Width: x4 / x4
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: no
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 00050502 00001111
        0x0020: 01000100 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0062c010 00008024 00000000 04396c42
        0x0050: 20410000 00000000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00800005 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 2:8:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 2, Secondary Bus: 6, Subordinate Bus: 6,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 21, I/O Limit: 11, Secondary Status: 0000
        0x0020: Memory Base: fff0, Memory Limit: 0000
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 5.0 GT/s, Link Width: x0 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 00060602 00001121
        0x0020: 0000fff0 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0162c010 00008024 00000000 08396c82
        0x0050: 00010000 00300000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00810005 7f800000 00000000 00000002
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 2:11:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 2, Secondary Bus: 7, Subordinate Bus: 8,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 21, I/O Limit: 11, Secondary Status: 0000
        0x0020: Memory Base: 0110, Memory Limit: 0110
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 5.0 GT/s, Link Width: x1 / x4
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: no
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 00080702 00001121
        0x0020: 01100110 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0062c010 00008024 00000000 0b396c42
        0x0050: 20110000 00000000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00800005 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 2:12:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 2, Secondary Bus: 9, Subordinate Bus: 9,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 21, I/O Limit: 11, Secondary Status: 0000
        0x0020: Memory Base: fff0, Memory Limit: 0000
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 5.0 GT/s, Link Width: x0 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 00090902 00001121
        0x0020: 0000fff0 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0162c010 00008024 00000000 0c396c82
        0x0050: 00010000 00400000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00810005 7f800000 00000000 00000003
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 3:0:0: ASPEED Technology AST1150 PCI
        0x0000: Vendor ID: 1a03, Product ID: 1150
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 3, Secondary Bus: 4, Subordinate Bus: 4,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 01, I/O Limit: 01, Secondary Status: 0220
        0x0020: Memory Base: 0010, Memory Limit: 00f0
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 01, Line: 00, Bridge Control: 0000
        0x0050: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: no
        0x0078: Capability 0x01: Power Management
                State: D0
        0x0080: Capability 0x10: PCI Express
                Link Speed: 2.5 / 2.5 GT/s, Link Width: x1 / x1
        0x0100: Enhanced Capability 0x02: Virtual Channel Capability
        0x00a4: Capability 0x0d: PCI-PCI
        0x0000: 11501a03 00100047 06040002 00010010
        0x0010: 00000000 00000000 00040403 02200101
        0x0020: 00f00010 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000050 00000000 00000100
        0x0040: 00000000 01826160 00000000 00000000
        0x0050: 00807805 00000000 00000000 00000000
        0x0060: 00000000 00000000 00007811 00000000
        0x0070: 00000001 00000000 ffc38001 00000008
        0x0080: 0071a410 00008000 00102810 0100dc11
        0x0090: 10110000 00000000 000003c0 00000000
        0x00a0: 00000000 0000000d 11501a03 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 4:0:0: ASPEED Technology AST2000
        0x0000: Vendor ID: 1a03, Product ID: 2000
        0x0004: Command: 0003, Status: 0210
        0x0008: Class: 03 Display, Subclass: 00 VGA,
                Interface: 00, Revision: 10
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR mem 32bit addr: 0x00800000/0x00800000
        0x0014: BAR mem 32bit addr: 0x00100000/0x00020000
        0x0018: BAR io addr: 0x00000000/0x0080
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1a03 Product ID: 2000
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0040: Capability 0x01: Power Management
                State: D0
        0x0000: 20001a03 02100003 03000010 00000000
        0x0010: 00800000 00100000 00000001 00000000
        0x0020: 00000000 00000000 00000000 20001a03
        0x0030: 00000000 00000040 00000000 00000100
        0x0040: ffc30001 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 5:0:0: Intel 82576
        0x0000: Vendor ID: 8086, Product ID: 10c9
        0x0004: Command: 0146, Status: 0010
        0x0008: Class: 02 Network, Subclass: 00 Ethernet,
                Interface: 00, Revision: 01
        0x000c: BIST: 00, Header Type: 80, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR mem 32bit addr: 0x01000000/0x00020000
        0x0014: BAR mem 32bit addr: 0x01020000/0x00020000
        0x0018: BAR io addr: 0x00001000/0x0020
        0x001c: BAR mem 32bit addr: 0x01040000/0x00004000
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 108e Product ID: 4848
        0x0030: Expansion ROM Base Address: 01060000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0040: Capability 0x01: Power Management
                State: D0
        0x0050: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x0070: Capability 0x11: Extended Message Signalled Interrupts (MSI-X)
                Enabled: no; table size 10 (BAR 3:0)
        0x00a0: Capability 0x10: PCI Express
                Link Speed: 2.5 / 2.5 GT/s, Link Width: x4 / x4
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0140: Enhanced Capability 0x03: Device Serial Number
                Serial Number: 002128fffff1e2f6
        0x0150: Enhanced Capability 0x0e: Alternate Routing ID
        0x0160: Enhanced Capability 0x10: Single Root I/O Virtualization
        0x0000: 10c98086 00100146 02000001 00800010
        0x0010: 01000000 01020000 00001001 01040000
        0x0020: 00000000 00000000 00000000 4848108e
        0x0030: 01060000 00000040 00000000 00000100
        0x0040: c8235001 1a002000 00000000 00000000
        0x0050: 01817005 7f800000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 0009a011 00000003 00002003 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00020010 10008cc2 00192810 04036c41
        0x00b0: 10410000 00000000 00000000 00000000
        0x00c0: 00000000 0000001f 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000003 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 5:0:1: Intel 82576
        0x0000: Vendor ID: 8086, Product ID: 10c9
        0x0004: Command: 0146, Status: 0010
        0x0008: Class: 02 Network, Subclass: 00 Ethernet,
                Interface: 00, Revision: 01
        0x000c: BIST: 00, Header Type: 80, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR mem 32bit addr: 0x01080000/0x00020000
        0x0014: BAR mem 32bit addr: 0x010a0000/0x00020000
        0x0018: BAR io addr: 0x00001020/0x0020
        0x001c: BAR mem 32bit addr: 0x01044000/0x00004000
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 108e Product ID: 4848
        0x0030: Expansion ROM Base Address: 010c0000
        0x0038: 00000000
        0x003c: Interrupt Pin: 02 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0040: Capability 0x01: Power Management
                State: D0
        0x0050: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x0070: Capability 0x11: Extended Message Signalled Interrupts (MSI-X)
                Enabled: no; table size 10 (BAR 3:0)
        0x00a0: Capability 0x10: PCI Express
                Link Speed: 2.5 / 2.5 GT/s, Link Width: x4 / x4
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0140: Enhanced Capability 0x03: Device Serial Number
                Serial Number: 002128fffff1e2f6
        0x0150: Enhanced Capability 0x0e: Alternate Routing ID
        0x0160: Enhanced Capability 0x10: Single Root I/O Virtualization
        0x0000: 10c98086 00100146 02000001 00800010
        0x0010: 01080000 010a0000 00001021 01044000
        0x0020: 00000000 00000000 00000000 4848108e
        0x0030: 010c0000 00000040 00000000 00000200
        0x0040: c8235001 1a002000 00000000 00000000
        0x0050: 01817005 7f800000 00000000 00000001
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 0009a011 00000003 00002003 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00020010 10008cc2 00192810 04036c41
        0x00b0: 10410000 00000000 00000000 00000000
        0x00c0: 00000000 0000001f 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000003 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 7:0:0: PLX PEX 8112
        0x0000: Vendor ID: 10b5, Product ID: 8112
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: aa
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 7, Secondary Bus: 8, Subordinate Bus: 8,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 20, I/O Limit: 10, Secondary Status: 0220
        0x0020: Memory Base: 0110, Memory Limit: 0110
        0x0024: Prefetch Memory Base: 0000, Prefetch Memory Limit: fff0
        0x0028: Prefetch Memory Base Upper 32 Bits: 00000000
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 01, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x01: Power Management
                State: D0 PME# asserted
        0x0050: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: no
        0x0060: Capability 0x10: PCI Express
                Link Speed: 2.5 / 2.5 GT/s, Link Width: x1 / x1
        0x0100: Enhanced Capability 0x04: Power Budgeting
        0x0000: 811210b5 00100047 060400aa 00010010
        0x0010: 00000000 00000000 00080807 02201020
        0x0020: 01100110 fff00000 00000000 00000000
        0x0030: 00000000 00000040 00000000 00000100
        0x0040: 5a025001 00008000 000e2011 00000000
        0x0050: 00806005 00000000 00000000 00000000
        0x0060: 00710010 00000020 00005007 00024c11
        0x0070: 00110000 00000c80 00400000 00000000
        0x0080: 00000000 00000000 00000033 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 8:0:0: NEC USB
        0x0000: Vendor ID: 1033, Product ID: 0035
        0x0004: Command: 0046, Status: 0210
        0x0008: Class: 0c Serial Bus, Subclass: 03 USB,
                Interface: 10, Revision: 43
        0x000c: BIST: 00, Header Type: 80, Latency Timer: 40,
                Cache Line Size: 10
        0x0010: BAR mem 32bit addr: 0x01100000/0x00001000
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1033 Product ID: 0035
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 00 Min Gnt: 01 Max Lat: 2a
        0x0040: Capability 0x01: Power Management
                State: D0
        0x0000: 00351033 02100046 0c031043 00804010
        0x0010: 01100000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 00351033
        0x0030: 00000000 00000040 00000000 2a010100
        0x0040: 7e020001 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 6cb03305 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 8:0:1: NEC USB
        0x0000: Vendor ID: 1033, Product ID: 0035
        0x0004: Command: 0046, Status: 0210
        0x0008: Class: 0c Serial Bus, Subclass: 03 USB,
                Interface: 10, Revision: 43
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 40,
                Cache Line Size: 10
        0x0010: BAR mem 32bit addr: 0x01102000/0x00001000
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1033 Product ID: 0035
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 02 Line: 00 Min Gnt: 01 Max Lat: 2a
        0x0040: Capability 0x01: Power Management
                State: D0
        0x0000: 00351033 02100046 0c031043 00004010
        0x0010: 01102000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 00351033
        0x0030: 00000000 00000040 00000000 2a010200
        0x0040: 7e020001 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 8:0:2: NEC USB
        0x0000: Vendor ID: 1033, Product ID: 00e0
        0x0004: Command: 0046, Status: 0210
        0x0008: Class: 0c Serial Bus, Subclass: 03 USB,
                Interface: 20, Revision: 04
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 40,
                Cache Line Size: 10
        0x0010: BAR mem 32bit addr: 0x01104000/0x00000100
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1033 Product ID: 00e0
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 03 Line: 00 Min Gnt: 10 Max Lat: 22
        0x0040: Capability 0x01: Power Management
                State: D0
        0x0000: 00e01033 02100046 0c032004 00004010
        0x0010: 01104000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 00e01033
        0x0030: 00000000 00000040 00000000 22100300
        0x0040: 7e020001 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 003f2020 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 6cb03305 00000000 00000001 c0080000
        0x00f0: 00000000 00000000 00000000 00000000
 10:0:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 10, Secondary Bus: 11, Subordinate Bus: 15,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 21, I/O Limit: 21, Secondary Status: 0000
        0x0020: Memory Base: 0120, Memory Limit: 0120
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 5.0 / 5.0 GT/s, Link Width: x8 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x00: Reserved
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 000f0b0a 00002121
        0x0020: 01200120 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0052c010 00008024 00000000 00016c82
        0x0050: 00820000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000002 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 11:0:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 11, Secondary Bus: 12, Subordinate Bus: 12,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 21, I/O Limit: 11, Secondary Status: 0000
        0x0020: Memory Base: fff0, Memory Limit: 0000
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 5.0 GT/s, Link Width: x0 / x4
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 000c0c0b 00001121
        0x0020: 0000fff0 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0162c010 00008024 00000000 0c396c42
        0x0050: 00010000 00200000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00810005 7f800000 00000000 00000004
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 11:4:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 11, Secondary Bus: 13, Subordinate Bus: 13,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 21, I/O Limit: 11, Secondary Status: 0000
        0x0020: Memory Base: fff0, Memory Limit: 0000
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 5.0 GT/s, Link Width: x0 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 000d0d0b 00001121
        0x0020: 0000fff0 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0162c010 00008024 00000000 04396c82
        0x0050: 00010000 00100000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00810005 7f800000 00000000 00000005
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 11:8:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 11, Secondary Bus: 14, Subordinate Bus: 14,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 21, I/O Limit: 11, Secondary Status: 0000
        0x0020: Memory Base: fff0, Memory Limit: 0000
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 5.0 GT/s, Link Width: x0 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 000e0e0b 00001121
        0x0020: 0000fff0 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0162c010 00008024 00000000 08396c82
        0x0050: 00010000 00000000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00810005 7f800000 00000000 00000006
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 11:14:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 11, Secondary Bus: 15, Subordinate Bus: 15,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 21, I/O Limit: 21, Secondary Status: 0000
        0x0020: Memory Base: 0120, Memory Limit: 0120
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 5.0 / 5.0 GT/s, Link Width: x8 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: no
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 000f0f0b 00002121
        0x0020: 01200120 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0062c010 00008024 00000000 0e396c82
        0x0050: 20820000 00000000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00800005 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 15:0:0: Symbios Logic SAS2008
        0x0000: Vendor ID: 1000, Product ID: 0072
        0x0004: Command: 0046, Status: 0010
        0x0008: Class: 01 Mass Storage, Subclass: 07 SAS,
                Interface: 00, Revision: 03
        0x000c: BIST: 00, Header Type: 00, Latency Timer: 00,
                Cache Line Size: 80
        0x0010: BAR io addr: 0x00002000/0x0100
        0x0014: BAR mem 64bit addr: 0x0000000001200000/0x00004000
        0x001c: BAR mem 64bit addr: 0x0000000001240000/0x00040000
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 1000 Product ID: 0072
        0x0030: Expansion ROM Base Address: 01280000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0050: Capability 0x01: Power Management
                State: D0
        0x0068: Capability 0x10: PCI Express
                Link Speed: 5.0 / 5.0 GT/s, Link Width: x8 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0138: Enhanced Capability 0x04: Power Budgeting
        0x0150: Enhanced Capability 0x10: Single Root I/O Virtualization
        0x0190: Enhanced Capability 0x0e: Alternate Routing ID
        0x00d0: Capability 0x03: Vital Product Data (VPD)
        0x00a8: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x00c0: Capability 0x11: Extended Message Signalled Interrupts (MSI-X)
                Enabled: no; table size 15 (BAR 1:8192)
        0x0000: 00721000 00100046 01070003 00000080
        0x0010: 00002001 01200004 00000000 01240004
        0x0020: 00000000 00000000 00000000 00721000
        0x0030: 01280000 00000050 00000000 00000100
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 06036801 00000008 00000000 00000000
        0x0060: 00000000 00000f00 0002d010 10008025
        0x0070: 00092810 00000482 10820000 00000000
        0x0080: 00000000 00000000 00000000 00000016
        0x0090: 00000000 00000000 00000002 00000000
        0x00a0: 00000000 00000000 0081c005 7f800000
        0x00b0: 00000000 00000007 00000000 00000000
        0x00c0: 000e0011 00002001 00003801 00000000
        0x00d0: 0000a803 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
Domain /dev/pci1:
 0:1:0: Sun SPARC-T3/T4 PCIE
        0x0000: Vendor ID: 108e, Product ID: 8186
        0x0004: Command: 0007, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 01
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 0, Secondary Bus: 1, Subordinate Bus: 5,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 01, I/O Limit: 01, Secondary Status: 0000
        0x0020: Memory Base: 0010, Memory Limit: 0010
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x01: Power Management
                State: D0
        0x0060: Capability 0x10: PCI Express
                Link Speed: 5.0 / 5.0 GT/s, Link Width: x8 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0138: Enhanced Capability 0x0b: Vendor-Specific
        0x014c: Enhanced Capability 0x0d: Access Control Services
        0x0000: 8186108e 00100007 06040001 00010000
        0x0010: 00000000 00000000 00050100 00000101
        0x0020: 00100010 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: c8036001 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00420010 00008001 00000000 00395c82
        0x0070: 70820000 00000000 01000000 00000000
        0x0080: 00000000 00000020 00000000 00000000
        0x0090: 00000002 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 0:2:0: Sun SPARC-T3/T4 PCIE
        0x0000: Vendor ID: 108e, Product ID: 8186
        0x0004: Command: 0007, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 01
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 0, Secondary Bus: 6, Subordinate Bus: 10,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 11, I/O Limit: 01, Secondary Status: 0000
        0x0020: Memory Base: fff0, Memory Limit: 0000
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x01: Power Management
                State: D0
        0x0060: Capability 0x10: PCI Express
                Link Speed: 5.0 / 5.0 GT/s, Link Width: x8 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0138: Enhanced Capability 0x0b: Vendor-Specific
        0x014c: Enhanced Capability 0x0d: Access Control Services
        0x0000: 8186108e 00100007 06040001 00010000
        0x0010: 00000000 00000000 000a0600 00000111
        0x0020: 0000fff0 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: c8036001 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00420010 00008001 00000000 01395c82
        0x0070: 70820000 00000000 01000000 00000000
        0x0080: 00000000 00000020 00000000 00000000
        0x0090: 00000002 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 1:0:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 1, Secondary Bus: 2, Subordinate Bus: 5,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 01, I/O Limit: 01, Secondary Status: 0000
        0x0020: Memory Base: 0010, Memory Limit: 0010
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 5.0 / 5.0 GT/s, Link Width: x8 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x00: Reserved
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 00050201 00000101
        0x0020: 00100010 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0052c010 00008024 00000000 02016c82
        0x0050: 00820000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000002 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 2:0:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 2, Secondary Bus: 3, Subordinate Bus: 3,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 11, I/O Limit: 01, Secondary Status: 0000
        0x0020: Memory Base: fff0, Memory Limit: 0000
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 5.0 GT/s, Link Width: x0 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 00030302 00000111
        0x0020: 0000fff0 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0162c010 00008024 00000000 0e396c82
        0x0050: 00010000 00480000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00810005 7f800000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 2:5:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 2, Secondary Bus: 4, Subordinate Bus: 4,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 01, I/O Limit: 01, Secondary Status: 0000
        0x0020: Memory Base: 0010, Memory Limit: 0010
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 5.0 GT/s, Link Width: x4 / x4
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: no
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 00040402 00000101
        0x0020: 00100010 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0062c010 00008024 00000000 05396c42
        0x0050: 20410000 00000000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00800005 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 2:6:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 2, Secondary Bus: 5, Subordinate Bus: 5,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 11, I/O Limit: 01, Secondary Status: 0000
        0x0020: Memory Base: fff0, Memory Limit: 0000
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 5.0 GT/s, Link Width: x0 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 00050502 00000111
        0x0020: 0000fff0 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0162c010 00008024 00000000 06396c82
        0x0050: 00010000 00380000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00810005 7f800000 00000000 00000003
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 4:0:0: Intel 82576
        0x0000: Vendor ID: 8086, Product ID: 10c9
        0x0004: Command: 0146, Status: 0010
        0x0008: Class: 02 Network, Subclass: 00 Ethernet,
                Interface: 00, Revision: 01
        0x000c: BIST: 00, Header Type: 80, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR mem 32bit addr: 0x00100000/0x00020000
        0x0014: BAR mem 32bit addr: 0x00120000/0x00020000
        0x0018: BAR io addr: 0x00000000/0x0020
        0x001c: BAR mem 32bit addr: 0x00140000/0x00004000
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 108e Product ID: 4848
        0x0030: Expansion ROM Base Address: 00160000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0040: Capability 0x01: Power Management
                State: D0
        0x0050: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x0070: Capability 0x11: Extended Message Signalled Interrupts (MSI-X)
                Enabled: no; table size 10 (BAR 3:0)
        0x00a0: Capability 0x10: PCI Express
                Link Speed: 2.5 / 2.5 GT/s, Link Width: x4 / x4
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0140: Enhanced Capability 0x03: Device Serial Number
                Serial Number: 002128fffff1e2f8
        0x0150: Enhanced Capability 0x0e: Alternate Routing ID
        0x0160: Enhanced Capability 0x10: Single Root I/O Virtualization
        0x0000: 10c98086 00100146 02000001 00800010
        0x0010: 00100000 00120000 00000001 00140000
        0x0020: 00000000 00000000 00000000 4848108e
        0x0030: 00160000 00000040 00000000 00000100
        0x0040: c8235001 1a002000 00000000 00000000
        0x0050: 01817005 7f800000 00000000 00000001
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 0009a011 00000003 00002003 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00020010 10008cc2 00192810 05036c41
        0x00b0: 10410000 00000000 00000000 00000000
        0x00c0: 00000000 0000001f 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000003 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 4:0:1: Intel 82576
        0x0000: Vendor ID: 8086, Product ID: 10c9
        0x0004: Command: 0146, Status: 0010
        0x0008: Class: 02 Network, Subclass: 00 Ethernet,
                Interface: 00, Revision: 01
        0x000c: BIST: 00, Header Type: 80, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR mem 32bit addr: 0x00180000/0x00020000
        0x0014: BAR mem 32bit addr: 0x001a0000/0x00020000
        0x0018: BAR io addr: 0x00000020/0x0020
        0x001c: BAR mem 32bit addr: 0x00144000/0x00004000
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 108e Product ID: 4848
        0x0030: Expansion ROM Base Address: 001c0000
        0x0038: 00000000
        0x003c: Interrupt Pin: 02 Line: 00 Min Gnt: 00 Max Lat: 00
        0x0040: Capability 0x01: Power Management
                State: D0
        0x0050: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x0070: Capability 0x11: Extended Message Signalled Interrupts (MSI-X)
                Enabled: no; table size 10 (BAR 3:0)
        0x00a0: Capability 0x10: PCI Express
                Link Speed: 2.5 / 2.5 GT/s, Link Width: x4 / x4
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0140: Enhanced Capability 0x03: Device Serial Number
                Serial Number: 002128fffff1e2f8
        0x0150: Enhanced Capability 0x0e: Alternate Routing ID
        0x0160: Enhanced Capability 0x10: Single Root I/O Virtualization
        0x0000: 10c98086 00100146 02000001 00800010
        0x0010: 00180000 001a0000 00000021 00144000
        0x0020: 00000000 00000000 00000000 4848108e
        0x0030: 001c0000 00000040 00000000 00000200
        0x0040: c8235001 1a002000 00000000 00000000
        0x0050: 01817005 7f800000 00000000 00000002
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 0009a011 00000003 00002003 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00020010 10008cc2 00192810 05036c41
        0x00b0: 10410000 00000000 00000000 00000000
        0x00c0: 00000000 0000001f 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000003 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000
 6:0:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 6, Secondary Bus: 7, Subordinate Bus: 10,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 11, I/O Limit: 01, Secondary Status: 0000
        0x0020: Memory Base: fff0, Memory Limit: 0000
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 5.0 / 5.0 GT/s, Link Width: x8 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x00: Reserved
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 000a0706 00000111
        0x0020: 0000fff0 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0052c010 00008024 00000000 02016c82
        0x0050: 00820000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000002 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 7:0:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 7, Secondary Bus: 8, Subordinate Bus: 8,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 11, I/O Limit: 01, Secondary Status: 0000
        0x0020: Memory Base: fff0, Memory Limit: 0000
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 5.0 GT/s, Link Width: x0 / x4
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 00080807 00000111
        0x0020: 0000fff0 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0162c010 00008024 00000000 0d396c42
        0x0050: 00010000 00280000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00810005 7f800000 00000000 00000004
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 7:6:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 7, Secondary Bus: 9, Subordinate Bus: 9,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 11, I/O Limit: 01, Secondary Status: 0000
        0x0020: Memory Base: fff0, Memory Limit: 0000
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 5.0 GT/s, Link Width: x0 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 00090907 00000111
        0x0020: 0000fff0 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0162c010 00008024 00000000 06396c82
        0x0050: 00010000 00180000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00810005 7f800000 00000000 00000005
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d
 7:10:0: IDT 89HPES64H16G2
        0x0000: Vendor ID: 111d, Product ID: 8077
        0x0004: Command: 0047, Status: 0010
        0x0008: Class: 06 Bridge, Subclass: 04 PCI,
                Interface: 00, Revision: 02
        0x000c: BIST: 00, Header Type: 01, Latency Timer: 00,
                Cache Line Size: 10
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: Primary Bus: 7, Secondary Bus: 10, Subordinate Bus: 10,
                Secondary Latency Timer: 00
        0x001c: I/O Base: 11, I/O Limit: 01, Secondary Status: 0000
        0x0020: Memory Base: fff0, Memory Limit: 0000
        0x0024: Prefetch Memory Base: fff1, Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: ffffffff
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000, I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00, Line: 00, Bridge Control: 0000
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 5.0 GT/s, Link Width: x0 / x8
        0x0100: Enhanced Capability 0x01: Advanced Error Reporting
        0x0200: Enhanced Capability 0x02: Virtual Channel Capability
        0x0320: Enhanced Capability 0x0d: Access Control Services
        0x0330: Enhanced Capability 0x12: Multicast
        0x00c0: Capability 0x01: Power Management
                State: D0
        0x00d0: Capability 0x05: Message Signalled Interrupts (MSI)
                Enabled: yes
        0x0000: 8077111d 00100047 06040002 00010010
        0x0010: 00000000 00000000 000a0a07 00000111
        0x0020: 0000fff0 0001fff1 ffffffff 00000000
        0x0030: 00000000 00000040 00000000 00000000
        0x0040: 0162c010 00008024 00000000 0a396c82
        0x0050: 00010000 00080000 00400000 00000000
        0x0060: 00000000 00000020 00000000 00000000
        0x0070: 00010042 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: c803d001 00000008 00000000 00000000
        0x00d0: 00810005 7f800000 00000000 00000006
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 0000000d 00000000 00000000 8077111d

acpidump:

Reply | Threaded
Open this post in threaded view
|

Re: ldom toolsuite (ldomd/ldomctl) fails to start on a T4-2

Klemens Nanni-2
On Fri, Jun 19, 2020 at 05:57:48AM +0000, Koakuma wrote:
> Both ldomctl and ldomd always crashes on this machine. I'm not sure if it's
> related or not, so I'm including both the symptoms here for completeness.
Thank you for the proper report.

> 1. When running "ldomctl init-system ldom.conf", ldomctl crashes with
> a segmentation fault. Running a gdb backtrace on the core file gives me this:
>
> #0  0x000000b69bd0506c in md_add_prop () from /usr/sbin/ldomctl
> #1  0x000000b69bd050b4 in md_add_prop_arc () from /usr/sbin/ldomctl
> #2  0x000000b69bd0e2a8 in hvmd_finalize_pcie_device () from /usr/sbin/ldomctl
> #3  0x000000b69bd0e4d0 in hvmd_finalize_devices () from /usr/sbin/ldomctl
> #4  0x000000b69bd0e7c4 in hvmd_finalize () from /usr/sbin/ldomctl
> #5  0x000000b69bd0fe90 in build_config () from /usr/sbin/ldomctl
> #6  0x000000b69bd0238c in main () from /usr/sbin/ldomctl
This seems to happen on T4-2 machines only, T200, T5220, T5240 and T4-1
are known to work.  I've briefly looked into this before but shelved it
since noone else reported it;  diff below works around the issue such
that `init-system' works as expected, but it does not fix the bug:

Index: config.c
===================================================================
RCS file: /cvs/src/usr.sbin/ldomctl/config.c,v
retrieving revision 1.40
diff -u -p -r1.40 config.c
--- config.c 24 May 2020 22:08:54 -0000 1.40
+++ config.c 19 Jun 2020 12:26:21 -0000
@@ -1144,6 +1144,7 @@ hvmd_finalize_pcie_device(struct md *md,
  TAILQ_FOREACH(subdevice, &device->guest->subdevice_list, link) {
  TAILQ_FOREACH(component, &components, link) {
  if (strcmp(subdevice->path, component->path) == 0)
+    if (component->hv_node == NULL) /* XXX fix components */
  md_link_node(md, parent, component->hv_node);
  }
  }

> 2. ldomd always crashes 10-30 minutes after starting. Running it manually
> with "ldomd -d" gives me this error:
>
> ldomd: Cannot allocate memory
I have never seen or reproduced this before.

Reply | Threaded
Open this post in threaded view
|

Re: ldom toolsuite (ldomd/ldomctl) fails to start on a T4-2

Koakuma
On Friday, June 19, 2020 7:42 PM, Klemens Nanni <[hidden email]> wrote:

> This seems to happen on T4-2 machines only, T200, T5220, T5240 and T4-1
> are known to work. I've briefly looked into this before but shelved it
> since noone else reported it; diff below works around the issue such
> that `init-system' works as expected, but it does not fix the bug:

I've tried your fix on tech@, it seems to solve the problem - at least up until
writing it to SP storage and changing the configuration. When I rebooted,
though, it panics with:

panic: rw_enter: vmmaplk locking against myself

Going back to factory-default config seems to make it not panic anymore, though.
I've attached the ddb log and relevant objdump parts too.

>> 2. ldomd always crashes 10-30 minutes after starting. Running it manually
>> with "ldomd -d" gives me this error:
>>
>> ldomd: Cannot allocate memory
> I have never seen or reproduced this before.

Uh, I am sorry, I forgot to include the ldomd backtrace.
Recompiling it with DEBUG="-g3 -O0" and running it under gdb gave me
this backtrace:

#0  *_libc_exit (status=1) at /usr/src/lib/libc/stdlib/exit.c:54
#1  0x000000a2c4acdb24 in *_libc_verr (eval=1, fmt=0x0, ap=0xfffffffffffbd0d0) at /usr/src/lib/libc/gen/verr.c:50
#2  0x000000a2c4b23050 in *_libc_err (eval=1, fmt=0x0) at /usr/src/lib/libc/gen/err.c:40
#3  0x000000a008607450 in xmalloc (size=24) at /usr/src/usr.sbin/ldomd/../ldomctl/util.c:36
#4  0x000000a0086018e8 in add_frag (base=44040192) at /usr/src/usr.sbin/ldomd/ldomd.c:348
#5  0x000000a0086017c0 in add_frag_mblock (node=0xa21182d7c0) at /usr/src/usr.sbin/ldomd/ldomd.c:333
#6  0x000000a0086016e8 in frag_init () at /usr/src/usr.sbin/ldomd/ldomd.c:320
#7  0x000000a008600f04 in main (argc=0, argv=0xfffffffffffbda78) at /usr/src/usr.sbin/ldomd/ldomd.c:192

Is there anything else that I could do to help?

ddb.log (7K) Download Attachment
objdump.txt (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: ldom toolsuite (ldomd/ldomctl) fails to start on a T4-2

Klemens Nanni-2
On Mon, Jun 22, 2020 at 05:07:36AM +0000, Koakuma wrote:
> I've tried your fix on tech@, it seems to solve the problem - at least up until
> writing it to SP storage and changing the configuration. When I rebooted,
> though, it panics with:
That is still on 6.7-stable, right?
Is the machine's firmware up to date?  Recent firmware on my T4-2 looks
like this:

        -> show /HOST hypervisor_version obp_version sysfw_version

          /HOST
            Properties:
                 hypervisor_version = Hypervisor 1.15.16 2018/11/28 07:41
                 obp_version = OpenBoot 4.38.16 2018/11/28 07:24
                 sysfw_version = Sun System Firmware 8.9.11 2018/11/28 07:59


I cannot downgrade my only T4-2 in production to test this, but using a
snapshot with my ldomctl diff from tech@ I did the following to recreate
its existing configuration and reboot with it:

        # sysctl -n kern.version
        OpenBSD 6.7-beta (GENERIC.MP) #299: Wed Apr 29 17:49:31 MDT 2020
            [hidden email]:/usr/src/sys/arch/sparc64/compile/GENERIC.MP
        ls factory-default/
        hv.md        pri          primary.md
        # cp -R factory-default production-new
        # cd production-new
        # cp ../production/ldom.conf .
        # ldomctl init-system ldom.conf
        # cd ..
        # ldomctl download production-new
        # ldomctl select production-new
        # halt

        -> reset /SYS

All domains come up, ldomd runs on the primary just fine and assigned
PCIe devices keep attaching in guest domains just like before.

Did you assign any PCIe devices to guest domains with `iodevice'?
Did you start with a fresh copy of the factory-default dump?
Please show the exact ldom.conf file you used.

> panic: rw_enter: vmmaplk locking against myself
>
> Going back to factory-default config seems to make it not panic anymore, though.
> I've attached the ddb log and relevant objdump parts too.
No clue yet, sorry.
 

> Uh, I am sorry, I forgot to include the ldomd backtrace.
> Recompiling it with DEBUG="-g3 -O0" and running it under gdb gave me
> this backtrace:
>
> #0  *_libc_exit (status=1) at /usr/src/lib/libc/stdlib/exit.c:54
> #1  0x000000a2c4acdb24 in *_libc_verr (eval=1, fmt=0x0, ap=0xfffffffffffbd0d0) at /usr/src/lib/libc/gen/verr.c:50
> #2  0x000000a2c4b23050 in *_libc_err (eval=1, fmt=0x0) at /usr/src/lib/libc/gen/err.c:40
> #3  0x000000a008607450 in xmalloc (size=24) at /usr/src/usr.sbin/ldomd/../ldomctl/util.c:36
> #4  0x000000a0086018e8 in add_frag (base=44040192) at /usr/src/usr.sbin/ldomd/ldomd.c:348
> #5  0x000000a0086017c0 in add_frag_mblock (node=0xa21182d7c0) at /usr/src/usr.sbin/ldomd/ldomd.c:333
> #6  0x000000a0086016e8 in frag_init () at /usr/src/usr.sbin/ldomd/ldomd.c:320
> #7  0x000000a008600f04 in main (argc=0, argv=0xfffffffffffbda78) at /usr/src/usr.sbin/ldomd/ldomd.c:192
Much better, thanks.

> Is there anything else that I could do to help?
Install the sysutils/mdprint package show the output of
`mdprint -q -g frag_space,frag_mblock hv.md', this will show us the
amount of memory ldomd is trying to allocate (in multiple fragments like
size=24 as seen in the backtrace).

Again: I never saw this on any machine, my T4-2 just works with it.
Perhaps you have a bogus machine description?  I vaguely remember
`ldomctl init-system' behaving differently/crashing when *.md files
existed already, so best to always run it with nothing but a clean copy
of the "factory-default" dump and a ldom.conf text file.

Can you try to reproduce this on recent snapshots?

Reply | Threaded
Open this post in threaded view
|

Re: ldom toolsuite (ldomd/ldomctl) fails to start on a T4-2

Koakuma
On Tuesday, June 23, 2020 3:11 AM, Klemens Nanni <[hidden email]> wrote:

> That is still on 6.7-stable, right?
> Is the machine's firmware up to date? Recent firmware on my T4-2 looks
> like this:
>
> -> show /HOST hypervisor_version obp_version sysfw_version
>
> /HOST
> Properties:
> hypervisor_version = Hypervisor 1.15.16 2018/11/28 07:41
> obp_version = OpenBoot 4.38.16 2018/11/28 07:24
> sysfw_version = Sun System Firmware 8.9.11 2018/11/28 07:59
Yes, it's still the same 6.7-stable, no change except for your patch and
recompiled ldomd.

The firmware version is this:
-> show /HOST hypervisor_version obp_version sysfw_version

  /HOST
    Properties:
        hypervisor_version = Hypervisor 1.10.1.b 2011/09/12 09:56
        obp_version = OpenBoot 4.33.1 2011/08/03 10:34
        sysfw_version = Sun System Firmware 8.1.1.c 2011/09/12 10:53

(I got this machine from a liquidation sale so I haven't been able to get the
firmware update files, sadly.)

> Did you assign any PCIe devices to guest domains with `iodevice'?
No.

> Did you start with a fresh copy of the factory-default dump?
Yes.

> Please show the exact ldom.conf file you used.
It's the same as in the previous mail, I'm just starting out on this machine.

domain "primary" {
        vcpu 4
        memory 4G
}

domain "obsd" {
        vcpu 2
        memory 2G
        vdisk "/storage/vmimages/obsd.img"
        vnet
}

> Install the sysutils/mdprint package show the output of
> `mdprint -q -g frag_space,frag_mblock hv.md', this will show us the amount of
> memory ldomd is trying to allocate (in multiple fragments like size=24 as seen
> in the backtrace). Again: I never saw this on any machine, my T4-2 just works
> with it. Perhaps you have a bogus machine description? I vaguely remember
> `ldomctl init-system' behaving differently/crashing when *.md files
> existed already, so best to always run it with nothing but a clean copy
> of the "factory-default" dump and a ldom.conf text file.
This is the output of the command, both for the factory-default hv.md and the
"openbsd" hv.md:

# mdprint -q -g frag_space,frag_mblock factory-default/hv.md
frag_space (node 0x5de) {
        bak <-- root (node 0x0)
        fwd --> frag_mblock (node 0x5e2)
}
frag_mblock (node 0x5e2) {
        base = 0x2a00000 /* 44040192 */
        size = 0x6000000 /* 100663296 */
        bak <-- frag_space (node 0x5de)
}

# mdprint -q -g frag_space,frag_mblock openbsd/hv.md
frag_space (node 0x12) {
        fragsize = 0x800000 /* 8388608 */
        bak <-- root (node 0x0)
        fwd --> frag_mblock (node 0x17)
}
frag_mblock (node 0x17) {
        base = 0x2a00000 /* 44040192 */
        size = 0x6000000 /* 100663296 */
        bak <-- frag_space (node 0x12)
}

> Can you try to reproduce this on recent snapshots?
I'll try to. I'll report back after upgrading to the latest snapshot.

Reply | Threaded
Open this post in threaded view
|

Re: ldom toolsuite (ldomd/ldomctl) fails to start on a T4-2

Klemens Nanni-2
On Tue, Jun 23, 2020 at 05:45:26AM +0000, Koakuma wrote:

> The firmware version is this:
> -> show /HOST hypervisor_version obp_version sysfw_version
>
>   /HOST
>     Properties:
>         hypervisor_version = Hypervisor 1.10.1.b 2011/09/12 09:56
>         obp_version = OpenBoot 4.33.1 2011/08/03 10:34
>         sysfw_version = Sun System Firmware 8.1.1.c 2011/09/12 10:53
>
> (I got this machine from a liquidation sale so I haven't been able to get the
> firmware update files, sadly.)
There are lots of fixes in the firmware changelog after your version,
I highly suggest you update it.

> This is the output of the command, both for the factory-default hv.md and the
> "openbsd" hv.md:
Both machine descriptions look good to me, nothing odd with those nodes.

> I'll try to. I'll report back after upgrading to the latest snapshot.
Given the old firmware, I now rather suspect the machine itself to be
the culprit.  Around relese time, I've been working on ldomctl on my
T4-2 and never experienced any of your issues, so it seems unlikely to
me that snapshots will fix it for you, but you should try nonetheless.

Reply | Threaded
Open this post in threaded view
|

Re: ldom toolsuite (ldomd/ldomctl) fails to start on a T4-2

Koakuma
In reply to this post by Klemens Nanni-2
On Tuesday, June 23, 2020 3:11 AM, Klemens Nanni <[hidden email]> wrote:

> Can you try to reproduce this on recent snapshots?

Tried the Jun 25 snapshot, it still has the same problem for ldomd.
As for the kernel itself, when I switched to my ldom config, it froze
somewhere after detecting the USB hubs (?), but there's no panics or errors.
factory-default works fine, though.

Below are the logs.

Kernel version:
OpenBSD 6.7-current (GENERIC.MP) #376: Thu Jun 25 14:04:09 MDT 2020
    [hidden email]:/usr/src/sys/arch/sparc64/compile/GENERIC.MP

ldomd backtrace:
#0  *_libc_exit (status=1) at /usr/src/lib/libc/stdlib/exit.c:54
#1  0x0000001bf29e4f24 in *_libc_verr (eval=1, fmt=0x0, ap=0xfffffffffffe41b0) at /usr/src/lib/libc/gen/verr.c:50
#2  0x0000001bf29e8af0 in *_libc_err (eval=1, fmt=0x0) at /usr/src/lib/libc/gen/err.c:40
#3  0x0000001994f04d48 in xmalloc () from /usr/sbin/ldomd
#4  0x0000001994f00838 in add_frag () from /usr/sbin/ldomd
#5  0x0000001994f00918 in add_frag_mblock () from /usr/sbin/ldomd
#6  0x0000001994f00a7c in frag_init () from /usr/sbin/ldomd
#7  0x0000001994f01894 in main () from /usr/sbin/ldomd

Boot log on "openbsd" ldom configuration:
Boot device: disk  File and args:
OpenBSD IEEE 1275 Bootblock 2.1
..>> OpenBSD BOOT 1.20

ERROR: /iscsi-hba: No iscsi-network-bootpath property
Trying bsd...
Booting /pci@400/pci@2/pci@0/pci@e/scsi@0/disk@w5000cca0152238bd,0:a/bsd
9987840@0x1000000+6400@0x1986700+194580@0x1c00000+3999724@0x1c2f814
symbols @ 0xfe526400 479922+165+641664+443225 start=0x1000000
[ using 1566016 bytes of bsd ELF symbol table ]
console is /virtual-devices@100/console@1
Copyright (c) 1982, 1986, 1989, 1991, 1993
        The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2020 OpenBSD. All rights reserved.  https://www.OpenBSD.org

OpenBSD 6.7-current (GENERIC.MP) #376: Thu Jun 25 14:04:09 MDT 2020
    [hidden email]:/usr/src/sys/arch/sparc64/compile/GENERIC.MP
real mem = 4294967296 (4096MB)
avail mem = 4198817792 (4004MB)
random: good seed from bootblocks
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root: SPARC T4-2
cpu0 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu1 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu2 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
cpu3 at mainbus0: SPARC-T4 (rev 0.0) @ 2847.862 MHz
vbus0 at mainbus0
"flashprom" at vbus0 not configured
"tpm" at vbus0 not configured
cbus0 at vbus0
vldc0 at cbus0
vldcp0 at vldc0 chan 0x0: ivec 0x0, 0x1 channel "hvctl"
"ldom-primary" at vldc0 chan 0x1 not configured
"fmactl" at vldc0 chan 0x3 not configured
vldc1 at cbus0
"ldmfma" at vldc1 chan 0x4 not configured
vldc2 at cbus0
"spfma" at vldc2 chan 0x5 not configured
vldc3 at cbus0
vldcp1 at vldc3 chan 0x14: ivec 0x28, 0x29 channel "spds"
"sunvts" at vldc3 chan 0x6 not configured
"sunmc" at vldc3 chan 0x7 not configured
"explorer" at vldc3 chan 0x8 not configured
"flashupdate" at vldc3 chan 0x9 not configured
"ipmi" at vldc3 chan 0xb not configured
vldcp2 at vldc3 chan 0x18: ivec 0x30, 0x31 channel "ldom-obsd"
"virtual-domain-service" at cbus0 not configured
vcc0 at cbus0
vcctty0 at vcc0 chan 0x17: ivec 0x2e, 0x2f domain "obsd"
vds0 at cbus0
vdsp0 at vds0 chan 0x19: ivec 0x32, 0x33
vsw0 at cbus0
vnet0 at vsw0 chan 0x1a: ivec 0x34, 0x35
vcons0 at vbus0: ivec 0x111, console
vrtc0 at vbus0
vpci0 at mainbus0: bus 0 to 15, dvma map 80000000-ffffffff
pci0 at vpci0
ppb0 at pci0 dev 1 function 0 "Sun SPARC-T3/T4 PCIE" rev 0x01
pci1 at ppb0 bus 1
ppb1 at pci1 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02
pci2 at ppb1 bus 2
ppb2 at pci2 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02
pci3 at ppb2 bus 3
ppb3 at pci3 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x02
pci4 at ppb3 bus 4
"ASPEED Technology AST2000" rev 0x10 at pci4 dev 0 function 0 not configured
ppb4 at pci2 dev 4 function 0 "IDT 89HPES64H16G2" rev 0x02
pci5 at ppb4 bus 5
em0 at pci5 dev 0 function 0 "Intel 82576" rev 0x01: msi, address 00:21:28:f1:e2:f6
em1 at pci5 dev 0 function 1 "Intel 82576" rev 0x01: msi, address 00:21:28:f1:e2:f7
ppb5 at pci2 dev 8 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci6 at ppb5 bus 6
ppb6 at pci2 dev 11 function 0 "IDT 89HPES64H16G2" rev 0x02
pci7 at ppb6 bus 7
ppb7 at pci7 dev 0 function 0 "PLX PEX 8112" rev 0xaa
pci8 at ppb7 bus 8
ohci0 at pci8 dev 0 function 0 "NEC USB" rev 0x43: ivec 0x401, version 1.0
ohci1 at pci8 dev 0 function 1 "NEC USB" rev 0x43: ivec 0x402, version 1.0
ehci0 at pci8 dev 0 function 2 "NEC USB" rev 0x04: ivec 0x403
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "NEC EHCI root hub" rev 2.00/1.00 addr 1
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "NEC OHCI root hub" rev 1.00/1.00 addr 1
usb2 at ohci1: USB revision 1.0
uhub2 at usb2 configuration 1 interface 0 "NEC OHCI root hub" rev 1.00/1.00 addr 1
ppb8 at pci2 dev 12 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci9 at ppb8 bus 9
ppb9 at pci0 dev 2 function 0 "Sun SPARC-T3/T4 PCIE" rev 0x01
pci10 at ppb9 bus 10
ppb10 at pci10 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02
pci11 at ppb10 bus 11
ppb11 at pci11 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci12 at ppb11 bus 12
ppb12 at pci11 dev 4 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci13 at ppb12 bus 13
ppb13 at pci11 dev 8 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci14 at ppb13 bus 14
ppb14 at pci11 dev 14 function 0 "IDT 89HPES64H16G2" rev 0x02
pci15 at ppb14 bus 15
mpii0 at pci15 dev 0 function 0 "Symbios Logic SAS2008" rev 0x03: msi
mpii0: Solana On-Board, firmware 9.0.0.0 IR, MPI 2.0
scsibus1 at mpii0: 834 targets
vpci1 at mainbus0: bus 0 to 10, dvma map 80000000-ffffffff
pci16 at vpci1
ppb15 at pci16 dev 1 function 0 "Sun SPARC-T3/T4 PCIE" rev 0x01
pci17 at ppb15 bus 1
ppb16 at pci17 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02
pci18 at ppb16 bus 2
ppb17 at pci18 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci19 at ppb17 bus 3
ppb18 at pci18 dev 5 function 0 "IDT 89HPES64H16G2" rev 0x02
pci20 at ppb18 bus 4
em2 at pci20 dev 0 function 0 "Intel 82576" rev 0x01: msi, address 00:21:28:f1:e2:f8
em3 at pci20 dev 0 function 1 "Intel 82576" rev 0x01: msi, address 00:21:28:f1:e2:f9
ppb19 at pci18 dev 6 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci21 at ppb19 bus 5
ppb20 at pci16 dev 2 function 0 "Sun SPARC-T3/T4 PCIE" rev 0x01
pci22 at ppb20 bus 6
ppb21 at pci22 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02
pci23 at ppb21 bus 7
ppb22 at pci23 dev 0 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci24 at ppb22 bus 8
ppb23 at pci23 dev 6 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci25 at ppb23 bus 9
ppb24 at pci23 dev 10 function 0 "IDT 89HPES64H16G2" rev 0x02: msi
pci26 at ppb24 bus 10
"niu" at mainbus0 not configured
"niu" at mainbus0 not configured
"pci-performance-counters" at mainbus0 not configured
"pci-performance-counters" at mainbus0 not configured
uhub3 at uhub0 port 2 configuration 1 interface 0 "Cypress Semiconductor USB2 Hub" rev 2.00/0.0b addr 2
ugen0 at uhub3 port 3 "American Megatrends Inc. Generic Hub" rev 2.00/1.00 addr 3
uhidev0 at uhub3 port 4 configuration 1 interface 0 "American Megatrends Inc. Virtual Keyboard and Mouse" rev 1.10/1.00 addr 4
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0 mux 1
uhidev1 at uhub3 port 4 configuration 1 interface 1 "American Megatrends Inc. Virtual Keyboard and Mouse" rev 1.10/1.00 addr 4
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons
wsmouse0 at ums0 mux 0
uhub4 at uhub0 port 4 configuration 1 interface 0 "Cypress Semiconductor USB2 Hub" rev 2.00/0.0b addr 5

(it then froze here, without any panics or errors)

Reply | Threaded
Open this post in threaded view
|

Re: ldom toolsuite (ldomd/ldomctl) fails to start on a T4-2

Koakuma
On Friday, June 26, 2020 10:52 AM, Koakuma <[hidden email]> wrote:

> On Tuesday, June 23, 2020 3:11 AM, Klemens Nanni [hidden email] wrote:
>
> > Can you try to reproduce this on recent snapshots?
>
> Kernel version:
> OpenBSD 6.7-current (GENERIC.MP) #376: Thu Jun 25 14:04:09 MDT 2020
> [hidden email]:/usr/src/sys/arch/sparc64/compile/GENERIC.MP

Okay, so I've updated the firmware.
Still using the same snapshot (+ the ldomctl patch from tech@), booting is
a bit slow but everything seems to be working fine now, thanks!

Reply | Threaded
Open this post in threaded view
|

Re: ldom toolsuite (ldomd/ldomctl) fails to start on a T4-2

Klemens Nanni-2
On Sun, Jun 28, 2020 at 02:32:19AM +0000, Koakuma wrote:
> Okay, so I've updated the firmware.
> Still using the same snapshot (+ the ldomctl patch from tech@), booting is
> a bit slow but everything seems to be working fine now, thanks!
Good to hear, thanks.