problem with round-robin redirection in pf (Bug)

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

problem with round-robin redirection in pf (Bug)

Sebastian Benoit
Hi,

Daniel an I found a rather bizarre problem with round-robin tables in pf.

We have a relayd setup with two webservers (IP's $A and $B). We observed that
when one of the webservers went down, so did the loadbalanced service (on IP
$C).

We first noticed this in 2011-11 when deploying a pair of carped
loadbalancers (running current) and debugged and narrowed it down to this case,
independently reproducible without relayd:

(pf.conf contains an anchor "relayd/*")

$ cat /etc/pf.testrule
pass in quick on rdomain 0 inet proto tcp from any \
        to $C port = 80 flags S/SA \
        keep state (tcp.established 600) tag RELAYDHTTP \
        rdr-to <http_foo> port 80 round-robin prio 0

$ pfctl -a relayd/http_foo -t http_foo -T add $A
$ pfctl -a relayd/http_foo -t http_foo -T add $B
$ pfctl -f /etc/pf.testrule  -a relayd/http_foo

This results in a working loadblanced setup:

$ pfctl -a relayd/http_foo -t http_foo -T show
$A
$B

Now consider the case where an IP is deleted from the table http_foo:

$ pfctl -a relayd/http_foo -t http_foo -T delete $A

or

$ pfctl -a relayd/http_foo -t http_foo -T delete $B

The bug triggers when this delete happens right after the _last_ redirection
address was $B, and $C becomes inaccesible.

It is independet of which of the addresses gets deleted, but the output _order_ of $A
and $B when printing the table with "-T show" is important.

The bug does not trigger with more than two addresses in the table.
I found it with IPv4 and IPv6 addresses.

For example real addresses used for A,B and C were

A=2a00:15a8:6:100:5054:ff:fe00:67
B=2a00:15a8:6:100:5054:ff:fe00:69
C=2a00:15a8:6:100::27c3

(and '69 has to be the last one used before deleting an address from the
table).

I also have a machine with IPv4 addresses where I can demonstrate the bug ;-)

We narrowed the cause of the problem down to snapshots between 2011-06-29
and 2011-07-16 and today found that a kernel build from CVS date '2011-07-08
12:00:00 UTC' does not have the bug, but '2011-07-09 12:00:00 UTC' has it.

We guess it might be rev. 1.92 of src/sys/net/pf_table.c.

If you need more details, please say so.
I can provide access to a system to demonstrate the bug if needed.

/Benno & Daniel

----------------------------------------

dmesg:
OpenBSD 5.0-current (GENERIC.MP) #123: Wed Nov 16 15:56:56 MST 2011
    [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Intel(R) Atom(TM) CPU D525 @ 1.80GHz ("GenuineIntel" 686-class) 1.81 GHz
cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,LAHF
real mem  = 2146471936 (2047MB)
avail mem = 2101256192 (2003MB)
mainbus0 at root
bios0 at mainbus0: AT/286+ BIOS, date 12/17/10, BIOS32 rev. 0 @ 0xf0010, SMBIOS rev. 2.6 @ 0x9ac00 (19 entries)
bios0: vendor American Megatrends Inc. version "1.1a" date 12/17/10
bios0: Supermicro X7SPA-HF
acpi0 at bios0: rev 2
acpi0: sleep states S0 S1 S4 S5
acpi0: tables DSDT FACP APIC MCFG OEMB HPET EINJ BERT ERST HEST
acpi0: wakeup devices P0P1(S4) PS2K(S4) PS2M(S4) USB0(S4) USB1(S4) USB2(S4) USB5(S4) EUSB(S4) USB3(S4) USB4(S4) USB6(S4) USBE(S4) P0P4(S4) P0P5(S4) P0P6(S4) P0P7(S4) P0P8(S4) P0P9(S4) GBE_(S4) SLPB(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: apic clock running at 199MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Atom(TM) CPU D525 @ 1.80GHz ("GenuineIntel" 686-class) 1.81 GHz
cpu1: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,MWAIT,DS-CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE,LAHF
ioapic0 at mainbus0: apid 3 pa 0xfec00000, version 20, 24 pins
ioapic0: misconfigured as apic 1, remapped to apid 3
acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 14318179 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 4 (P0P1)
acpiprt2 at acpi0: bus 1 (P0P4)
acpiprt3 at acpi0: bus -1 (P0P5)
acpiprt4 at acpi0: bus -1 (P0P6)
acpiprt5 at acpi0: bus -1 (P0P7)
acpiprt6 at acpi0: bus 2 (P0P8)
acpiprt7 at acpi0: bus 3 (P0P9)
acpicpu0 at acpi0
acpicpu1 at acpi0
acpibtn0 at acpi0: SLPB
acpibtn1 at acpi0: PWRB
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000
ipmi at mainbus0 not configured
pci0 at mainbus0 bus 0: configuration mode 1 (bios)
pchb0 at pci0 dev 0 function 0 "Intel Pineview DMI" rev 0x02
uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 3 int 16
uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 3 int 21
uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x02: apic 3 int 19
ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 3 int 18
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb0 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02: apic 3 int 17
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82572EI)" rev 0x06: msi, address 00:1b:21:b9:0e:b1
ppb1 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02: apic 3 int 17
pci2 at ppb1 bus 2
em1 at pci2 dev 0 function 0 "Intel PRO/1000 MT (82574L)" rev 0x00: msi, address 00:25:90:3b:37:e2
ppb2 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02: apic 3 int 16
pci3 at ppb2 bus 3
em2 at pci3 dev 0 function 0 "Intel PRO/1000 MT (82574L)" rev 0x00: msi, address 00:25:90:3b:37:e3
uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 3 int 23
uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 3 int 19
uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 3 int 18
ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 3 int 23
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
ppb3 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
pci4 at ppb3 bus 4
vga1 at pci4 dev 4 function 0 "Matrox MGA G200eW" rev 0x0a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ichpcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02: PM disabled
ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.2
scsibus0 at ahci0: 32 targets
sd0 at scsibus0 targ 0 lun 0: <ATA, KINGSTON SS100S2, D100> SCSI3 0/direct fixed t10.ATA_KINGSTON_SS100S28G_08GB40017544_
sd0: 7641MB, 512 bytes/sector, 15649200 sectors, thin
ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 3 int 18
iic0 at ichiic0
lm1 at iic0 addr 0x2d: unknown Winbond chip (ID 0x0)
spdmem0 at iic0 addr 0x51: 2GB DDR3 SDRAM PC3-8500 SO-DIMM
usb2 at uhci0: USB revision 1.0
uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb3 at uhci1: USB revision 1.0
uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb4 at uhci2: USB revision 1.0
uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb5 at uhci3: USB revision 1.0
uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb6 at uhci4: USB revision 1.0
uhub6 at usb6 "Intel UHCI root hub" rev 1.00/1.00 addr 1
usb7 at uhci5: USB revision 1.0
uhub7 at usb7 "Intel UHCI root hub" rev 1.00/1.00 addr 1
isa0 at ichpcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
wbsio0 at isa0 port 0x2e/2: W83627DHG rev 0x25
lm2 at wbsio0 port 0xca0/8: W83627DHG
npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
mtrr: Pentium Pro MTRR support
uhidev0 at uhub4 port 2 configuration 1 interface 0 "Winbond Electronics Corp Hermon USB hidmouse Device" rev 1.10/0.01 addr 2
uhidev0: iclass 3/1
ums0 at uhidev0: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
uhidev1 at uhub4 port 2 configuration 1 interface 1 "Winbond Electronics Corp Hermon USB hidmouse Device" rev 1.10/0.01 addr 2
uhidev1: iclass 3/1
ukbd0 at uhidev1: 8 modifier keys, 6 key codes
wskbd1 at ukbd0 mux 1
wskbd1: connecting to wsdisplay0
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
root on sd0a (1b0341781f8a2a6a.a) swap on sd0b dump on sd0b
WARNING: / was not properly unmounted
carp: carp0 demoted group carp by 1 to 129 (carpdev)
carp: carp1 demoted group carp by 1 to 130 (carpdev)
carp: pfsync0 demoted group carp by 1 to 131 (pfsync bulk start)
carp: pfsync0 demoted group pfsync by 1 to 1 (pfsync bulk start)

usbdevs:
Controller /dev/usb0:
addr 1: high speed, self powered, config 1, EHCI root hub(0x0000), Intel(0x8086), rev 1.00
 port 1 powered
 port 2 powered
 port 3 powered
 port 4 powered
 port 5 powered
 port 6 powered
Controller /dev/usb1:
addr 1: high speed, self powered, config 1, EHCI root hub(0x0000), Intel(0x8086), rev 1.00
 port 1 powered
 port 2 powered
 port 3 powered
 port 4 powered
 port 5 powered
 port 6 powered
Controller /dev/usb2:
addr 1: full speed, self powered, config 1, UHCI root hub(0x0000), Intel(0x8086), rev 1.00
 port 1 powered
 port 2 powered
Controller /dev/usb3:
addr 1: full speed, self powered, config 1, UHCI root hub(0x0000), Intel(0x8086), rev 1.00
 port 1 powered
 port 2 powered
Controller /dev/usb4:
addr 1: full speed, self powered, config 1, UHCI root hub(0x0000), Intel(0x8086), rev 1.00
 port 1 powered
 port 2 addr 2: full speed, power 100 mA, config 1, Hermon USB hidmouse Device(0x2221), Winbond Electronics Corp(0x0557), rev 0.01
Controller /dev/usb5:
addr 1: full speed, self powered, config 1, UHCI root hub(0x0000), Intel(0x8086), rev 1.00
 port 1 powered
 port 2 powered
Controller /dev/usb6:
addr 1: full speed, self powered, config 1, UHCI root hub(0x0000), Intel(0x8086), rev 1.00
 port 1 powered
 port 2 powered
Controller /dev/usb7:
addr 1: full speed, self powered, config 1, UHCI root hub(0x0000), Intel(0x8086), rev 1.00
 port 1 powered
 port 2 powered

pcidump:
Domain /dev/pci0:
 0:0:0: Intel Pineview DMI
        0x0000: Vendor ID: 8086 Product ID: a000
        0x0004: Command: 0006 Status ID: 2090
        0x0008: Class: 06 Subclass: 00 Interface: 00 Revision: 02
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
        0x00e0: Capability 0x09: Vendor Specific
        0x0000: a0008086 20900006 06000002 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 060a15d9
        0x0030: 00000000 000000e0 00000000 00000000
        0x0040: fed19001 00000000 fed14000 00000000
        0x0050: 00000000 00000001 00000000 00000000
        0x0060: e0000001 00000000 fed18001 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00011110 00333300 000003ff 00381a00
        0x00a0: 08000020 80000000 80000000 80000000
        0x00b0: 00008000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000049 e6000000
        0x00e0: c1080009 08111002 00000000 00000000
        0x00f0: 00000000 00000001 00020fad 00000000
 0:26:0: Intel 82801I USB
        0x0000: Vendor ID: 8086 Product ID: 2937
        0x0004: Command: 0005 Status ID: 0290
        0x0008: Class: 0c Subclass: 03 Interface: 00 Revision: 02
        0x000c: BIST: 00 Header Type: 80 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR io addr: 0x0000bc00/0x0020
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
        0x0050: Capability 0x13: PCI Advanced Features
        0x0000: 29378086 02900005 0c030002 00800000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 0000bc01 00000000 00000000 060a15d9
        0x0030: 00000000 00000050 00000000 0000010b
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 03060013 00000000 00000000 00000000
        0x0060: 00000010 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00002f00 00000000 00010000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00020f86 00000000
 0:26:1: Intel 82801I USB
        0x0000: Vendor ID: 8086 Product ID: 2938
        0x0004: Command: 0005 Status ID: 0290
        0x0008: Class: 0c Subclass: 03 Interface: 00 Revision: 02
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR io addr: 0x0000b880/0x0020
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 02 Line: 07 Min Gnt: 00 Max Lat: 00
        0x0050: Capability 0x13: PCI Advanced Features
        0x0000: 29388086 02900005 0c030002 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 0000b881 00000000 00000000 060a15d9
        0x0030: 00000000 00000050 00000000 00000207
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 03060013 00000000 00000000 00000000
        0x0060: 00000010 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00002f00 00000000 00010000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00020f86 00000000
 0:26:2: Intel 82801I USB
        0x0000: Vendor ID: 8086 Product ID: 2939
        0x0004: Command: 0005 Status ID: 0290
        0x0008: Class: 0c Subclass: 03 Interface: 00 Revision: 02
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR io addr: 0x0000b800/0x0020
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 04 Line: 0e Min Gnt: 00 Max Lat: 00
        0x0050: Capability 0x13: PCI Advanced Features
        0x0000: 29398086 02900005 0c030002 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 0000b801 00000000 00000000 060a15d9
        0x0030: 00000000 00000050 00000000 0000040e
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 03060013 00000000 00000000 00000000
        0x0060: 00000010 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00002f00 00000000 00010000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00020f86 00000000
 0:26:7: Intel 82801I USB
        0x0000: Vendor ID: 8086 Product ID: 293c
        0x0004: Command: 0006 Status ID: 0290
        0x0008: Class: 0c Subclass: 03 Interface: 20 Revision: 02
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR mem 32bit addr: 0xfebfbc00/0x00000400
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 03 Line: 0f Min Gnt: 00 Max Lat: 00
        0x0050: Capability 0x01: Power Management
        0x0058: Capability 0x0a: Debug Port
        0x0098: Capability 0x13: PCI Advanced Features
        0x0000: 293c8086 02900006 0c032002 00000000
        0x0010: febfbc00 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 060a15d9
        0x0030: 00000000 00000050 00000000 0000030f
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: c9c25801 00000000 20a0980a 00000000
        0x0060: 01ff2020 00000000 00000001 c0082000
        0x0070: 0fdf0000 00000000 00000000 00000000
        0x0080: 00000000 00000001 00000000 00000000
        0x0090: 00000000 00000000 03060013 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00ffaa00 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00408586 00020f86 2002130a
 0:28:0: Intel 82801I PCIE
        0x0000: Vendor ID: 8086 Product ID: 2940
        0x0004: Command: 0107 Status ID: 0010
        0x0008: Class: 06 Subclass: 04 Interface: 00 Revision: 02
        0x000c: BIST: 00 Header Type: 81 Latency Timer: 00 Cache Line Size: 08
        0x0010: 00000000
        0x0014: 00000000
        0x0018: Primary Bus: 0 Secondary Bus: 1 Subordinate Bus: 1
                Secondary Latency Timer: 00
        0x001c: I/O Base: c0 I/O Limit: c0 Secondary Status: 2000
        0x0020: Memory Base: fe80 Memory Limit: fe80
        0x0024: Prefetch Memory Base: fff1 Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: 00000000
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000 I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 01 Line: 0a Bridge Control: 0006
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 2.5 Gb/s Link Width: x1 / x4
        0x0080: Capability 0x05: Message Signaled Interrupts (MSI)
        0x0090: Capability 0x0d: PCI-PCI
        0x00a0: Capability 0x01: Power Management
        0x0000: 29408086 00100107 06040002 00810008
        0x0010: 00000000 00000000 00010100 2000c0c0
        0x0020: fe80fe80 0001fff1 00000000 00000000
        0x0030: 00000000 00000040 00000000 0006010a
        0x0040: 01418010 00008000 00100000 01112c41
        0x0050: 30110040 00000560 00400028 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00009005 00000000 00000000 00000000
        0x0090: 0000a00d 060a15d9 00000000 00000000
        0x00a0: c8020001 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 08110000 00000000
        0x00e0: 00c70000 00080706 00000030 00000000
        0x00f0: 00000000 00000000 00020f86 00000000
 0:28:4: Intel 82801I PCIE
        0x0000: Vendor ID: 8086 Product ID: 2948
        0x0004: Command: 0107 Status ID: 0010
        0x0008: Class: 06 Subclass: 04 Interface: 00 Revision: 02
        0x000c: BIST: 00 Header Type: 81 Latency Timer: 00 Cache Line Size: 08
        0x0010: 00000000
        0x0014: 00000000
        0x0018: Primary Bus: 0 Secondary Bus: 2 Subordinate Bus: 2
                Secondary Latency Timer: 00
        0x001c: I/O Base: d0 I/O Limit: d0 Secondary Status: 2000
        0x0020: Memory Base: fe90 Memory Limit: fe90
        0x0024: Prefetch Memory Base: fff1 Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: 00000000
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000 I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 01 Line: 0a Bridge Control: 0006
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 2.5 Gb/s Link Width: x1 / x1
        0x0080: Capability 0x05: Message Signaled Interrupts (MSI)
        0x0090: Capability 0x0d: PCI-PCI
        0x00a0: Capability 0x01: Power Management
        0x0000: 29488086 00100107 06040002 00810008
        0x0010: 00000000 00000000 00020200 2000d0d0
        0x0020: fe90fe90 0001fff1 00000000 00000000
        0x0030: 00000000 00000040 00000000 0006010a
        0x0040: 01418010 00008000 00100000 05112c11
        0x0050: 30110040 00000560 00400028 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00009005 00000000 00000000 00000000
        0x0090: 0000a00d 060a15d9 00000000 00000000
        0x00a0: c8020001 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 08110000 00000000
        0x00e0: 00c70000 00080706 00000030 00000000
        0x00f0: 00000000 00000000 00020f86 00000000
 0:28:5: Intel 82801I PCIE
        0x0000: Vendor ID: 8086 Product ID: 294a
        0x0004: Command: 0107 Status ID: 0010
        0x0008: Class: 06 Subclass: 04 Interface: 00 Revision: 02
        0x000c: BIST: 00 Header Type: 81 Latency Timer: 00 Cache Line Size: 08
        0x0010: 00000000
        0x0014: 00000000
        0x0018: Primary Bus: 0 Secondary Bus: 3 Subordinate Bus: 3
                Secondary Latency Timer: 00
        0x001c: I/O Base: e0 I/O Limit: e0 Secondary Status: 2000
        0x0020: Memory Base: fea0 Memory Limit: fea0
        0x0024: Prefetch Memory Base: fff1 Prefetch Memory Limit: 0001
        0x0028: Prefetch Memory Base Upper 32 Bits: 00000000
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000 I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 02 Line: 0b Bridge Control: 0006
        0x0040: Capability 0x10: PCI Express
                Link Speed: 2.5 / 2.5 Gb/s Link Width: x1 / x1
        0x0080: Capability 0x05: Message Signaled Interrupts (MSI)
        0x0090: Capability 0x0d: PCI-PCI
        0x00a0: Capability 0x01: Power Management
        0x0000: 294a8086 00100107 06040002 00810008
        0x0010: 00000000 00000000 00030300 2000e0e0
        0x0020: fea0fea0 0001fff1 00000000 00000000
        0x0030: 00000000 00000040 00000000 0006020b
        0x0040: 01418010 00008000 00100000 06112c11
        0x0050: 30110040 00000560 00400028 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00009005 00000000 00000000 00000000
        0x0090: 0000a00d 060a15d9 00000000 00000000
        0x00a0: c8020001 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 08110000 00000000
        0x00e0: 00c70000 00080706 00000030 00000000
        0x00f0: 00000000 00000000 00020f86 00000000
 0:29:0: Intel 82801I USB
        0x0000: Vendor ID: 8086 Product ID: 2934
        0x0004: Command: 0005 Status ID: 0290
        0x0008: Class: 0c Subclass: 03 Interface: 00 Revision: 02
        0x000c: BIST: 00 Header Type: 80 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR io addr: 0x0000b480/0x0020
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
        0x0050: Capability 0x13: PCI Advanced Features
        0x0000: 29348086 02900005 0c030002 00800000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 0000b481 00000000 00000000 060a15d9
        0x0030: 00000000 00000050 00000000 0000010b
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 03060013 00000000 00000000 00000000
        0x0060: 00000010 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00002f00 00000000 00010000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00020f86 00000000
 0:29:1: Intel 82801I USB
        0x0000: Vendor ID: 8086 Product ID: 2935
        0x0004: Command: 0005 Status ID: 0290
        0x0008: Class: 0c Subclass: 03 Interface: 00 Revision: 02
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR io addr: 0x0000b400/0x0020
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 02 Line: 0e Min Gnt: 00 Max Lat: 00
        0x0050: Capability 0x13: PCI Advanced Features
        0x0000: 29358086 02900005 0c030002 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 0000b401 00000000 00000000 060a15d9
        0x0030: 00000000 00000050 00000000 0000020e
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 03060013 00000000 00000000 00000000
        0x0060: 00000010 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00002f00 00000000 00010000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00020f86 00000000
 0:29:2: Intel 82801I USB
        0x0000: Vendor ID: 8086 Product ID: 2936
        0x0004: Command: 0005 Status ID: 0290
        0x0008: Class: 0c Subclass: 03 Interface: 00 Revision: 02
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR io addr: 0x0000b080/0x0020
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 03 Line: 0f Min Gnt: 00 Max Lat: 00
        0x0050: Capability 0x13: PCI Advanced Features
        0x0000: 29368086 02900005 0c030002 00000000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 0000b081 00000000 00000000 060a15d9
        0x0030: 00000000 00000050 00000000 0000030f
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 03060013 00000000 00000000 00000000
        0x0060: 00000010 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00002f00 00000000 00010000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00020f86 00000000
 0:29:7: Intel 82801I USB
        0x0000: Vendor ID: 8086 Product ID: 293a
        0x0004: Command: 0006 Status ID: 0290
        0x0008: Class: 0c Subclass: 03 Interface: 20 Revision: 02
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR mem 32bit addr: 0xfebfb800/0x00000400
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
        0x0050: Capability 0x01: Power Management
        0x0058: Capability 0x0a: Debug Port
        0x0098: Capability 0x13: PCI Advanced Features
        0x0000: 293a8086 02900006 0c032002 00000000
        0x0010: febfb800 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 060a15d9
        0x0030: 00000000 00000050 00000000 0000010b
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: c9c25801 00000000 20a0980a 00000000
        0x0060: 01ff2020 00000000 00000001 c0082000
        0x0070: 0fdf0000 00000000 00000000 00000000
        0x0080: 00000000 00000001 00000000 00000000
        0x0090: 00000000 00000000 03060013 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00ffaa00 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00408586 00020f86 2002130a
 0:30:0: Intel 82801BA Hub-to-PCI
        0x0000: Vendor ID: 8086 Product ID: 244e
        0x0004: Command: 0107 Status ID: 0010
        0x0008: Class: 06 Subclass: 04 Interface: 01 Revision: 92
        0x000c: BIST: 00 Header Type: 01 Latency Timer: 00 Cache Line Size: 00
        0x0010: 00000000
        0x0014: 00000000
        0x0018: Primary Bus: 0 Secondary Bus: 4 Subordinate Bus: 4
                Secondary Latency Timer: 20
        0x001c: I/O Base: f0 I/O Limit: 00 Secondary Status: 2280
        0x0020: Memory Base: fdf0 Memory Limit: fe70
        0x0024: Prefetch Memory Base: fc01 Prefetch Memory Limit: fcf1
        0x0028: Prefetch Memory Base Upper 32 Bits: 00000000
        0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
        0x0030: I/O Base Upper 16 Bits: 0000 I/O Limit Upper 16 Bits: 0000
        0x0038: Expansion ROM Base Address: 00000000
        0x003c: Interrupt Pin: 00 Line: ff Bridge Control: 001a
        0x0050: Capability 0x0d: PCI-PCI
        0x0000: 244e8086 00100107 06040192 00010000
        0x0010: 00000000 00000000 20040400 228000f0
        0x0020: fe70fdf0 fcf1fc01 00000000 00000000
        0x0030: 00000000 00000050 00000000 001a00ff
        0x0040: 00000000 00000000 00000000 00001200
        0x0050: 0000000d 060a15d9 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00020f86 00000000
 0:31:0: Intel 82801IR LPC
        0x0000: Vendor ID: 8086 Product ID: 2916
        0x0004: Command: 0007 Status ID: 0210
        0x0008: Class: 06 Subclass: 01 Interface: 00 Revision: 02
        0x000c: BIST: 00 Header Type: 80 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR empty (00000000)
        0x0014: BAR empty (00000000)
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
        0x00e0: Capability 0x09: Vendor Specific
        0x0000: 29168086 02100007 06010002 00800000
        0x0010: 00000000 00000000 00000000 00000000
        0x0020: 00000000 00000000 00000000 060a15d9
        0x0030: 00000000 000000e0 00000000 00000000
        0x0040: 00000801 00000080 00000481 00000010
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 0e0f0a0b 000000d0 0b0a0780 000000f8
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 34030010 000c02e1 000c03e1 000c1641
        0x0090: 000c0ca1 00000000 00000000 00000000
        0x00a0: 00000620 00000208 00000013 40000300
        0x00b0: 00f00000 00000000 20000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00112233 00004567 0000ffc0 00000000
        0x00e0: 100c0009 02c40000 00000064 00000000
        0x00f0: fed1c001 00000000 00020f86 00000000
 0:31:2: Intel 82801I AHCI
        0x0000: Vendor ID: 8086 Product ID: 2922
        0x0004: Command: 0007 Status ID: 02b0
        0x0008: Class: 01 Subclass: 06 Interface: 01 Revision: 02
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR io addr: 0x0000a480/0x0008
        0x0014: BAR io addr: 0x0000b000/0x0004
        0x0018: BAR io addr: 0x0000ac00/0x0008
        0x001c: BAR io addr: 0x0000a880/0x0004
        0x0020: BAR io addr: 0x0000a800/0x0020
        0x0024: BAR mem 32bit addr: 0xfebfb000/0x00000800
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 02 Line: 0e Min Gnt: 00 Max Lat: 00
        0x0080: Capability 0x05: Message Signaled Interrupts (MSI)
        0x0070: Capability 0x01: Power Management
        0x00a8: Capability 0x12: SATA
        0x00b0: Capability 0x13: PCI Advanced Features
        0x0000: 29228086 02b00007 01060102 00000000
        0x0010: 0000a481 0000b001 0000ac01 0000a881
        0x0020: 0000a801 febfb000 00000000 060a15d9
        0x0030: 00000000 00000080 00000000 0000020e
        0x0040: 80008000 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 4003a801 00000008 00000000 00000000
        0x0080: 00097005 fee00000 00000050 00000000
        0x0090: 813f0060 00000193 00000000 00000000
        0x00a0: 00000000 00000000 0010b012 00000048
        0x00b0: 03060013 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00020f86 00000000
 0:31:3: Intel 82801I SMBus
        0x0000: Vendor ID: 8086 Product ID: 2930
        0x0004: Command: 0003 Status ID: 0280
        0x0008: Class: 0c Subclass: 05 Interface: 00 Revision: 02
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
        0x0010: BAR mem 64bit addr: 0x00000000febfac00/0x00000100
        0x0018: BAR empty (00000000)
        0x001c: BAR empty (00000000)
        0x0020: BAR io addr: 0x00000400/0x0020
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 03 Line: 0f Min Gnt: 00 Max Lat: 00
        0x0000: 29308086 02800003 0c050002 00000000
        0x0010: febfac04 00000000 00000000 00000000
        0x0020: 00000401 00000000 00000000 060a15d9
        0x0030: 00000000 00000000 00000000 0000030f
        0x0040: 00000001 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00040402 080800f0 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000004 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00000000
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00020f86 00000000
 1:0:0: Intel PRO/1000 PT (82572EI)
        0x0000: Vendor ID: 8086 Product ID: 107d
        0x0004: Command: 0007 Status ID: 0010
        0x0008: Class: 02 Subclass: 00 Interface: 00 Revision: 06
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 08
        0x0010: BAR mem 32bit addr: 0xfe8e0000/0x00020000
        0x0014: BAR mem 32bit addr: 0xfe8c0000/0x00020000
        0x0018: BAR io addr: 0x0000cc00/0x0020
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 8086 Product ID: 1082
        0x0030: Expansion ROM Base Address: fe8a0000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
        0x00c8: Capability 0x01: Power Management
        0x00d0: Capability 0x05: Message Signaled Interrupts (MSI)
        0x00e0: Capability 0x10: PCI Express
                Link Speed: 2.5 / 2.5 Gb/s Link Width: x1 / x1
        0x0000: 107d8086 00100007 02000006 00000008
        0x0010: fe8e0000 fe8c0000 0000cc01 00000000
        0x0020: 00000000 00000000 00000000 10828086
        0x0030: fe8a0000 000000c8 00000000 0000010b
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 c822d001 21002000
        0x00d0: 0081e005 fee00000 00000000 00000060
        0x00e0: 00010010 00000cc1 001a2810 00036411
        0x00f0: 10110000 00000000 00000000 00000000
 2:0:0: Intel PRO/1000 MT (82574L)
        0x0000: Vendor ID: 8086 Product ID: 10d3
        0x0004: Command: 0007 Status ID: 0010
        0x0008: Class: 02 Subclass: 00 Interface: 00 Revision: 00
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 10
        0x0010: BAR mem 32bit addr: 0xfe9e0000/0x00020000
        0x0014: BAR empty (00000000)
        0x0018: BAR io addr: 0x0000dc00/0x0020
        0x001c: BAR mem 32bit addr: 0xfe9dc000/0x00004000
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 10d3
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
        0x00c8: Capability 0x01: Power Management
        0x00d0: Capability 0x05: Message Signaled Interrupts (MSI)
        0x00e0: Capability 0x10: PCI Express
                Link Speed: 2.5 / 2.5 Gb/s Link Width: x1 / x1
        0x00a0: Capability 0x11: Extended Message Signaled Interrupts (MSI-X)
        0x0000: 10d38086 00100007 02000000 00000010
        0x0010: fe9e0000 00000000 0000dc01 fe9dc000
        0x0020: 00000000 00000000 00000000 10d315d9
        0x0030: 00000000 000000c8 00000000 0000010b
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00040011 00000003 00002003 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 c822d001 0f002000
        0x00d0: 0081e005 fee00000 00000000 00000061
        0x00e0: 0001a010 00008cc1 00192810 00031c11
        0x00f0: 10110000 00000000 00000000 00000000
 3:0:0: Intel PRO/1000 MT (82574L)
        0x0000: Vendor ID: 8086 Product ID: 10d3
        0x0004: Command: 0007 Status ID: 0010
        0x0008: Class: 02 Subclass: 00 Interface: 00 Revision: 00
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 10
        0x0010: BAR mem 32bit addr: 0xfeae0000/0x00020000
        0x0014: BAR empty (00000000)
        0x0018: BAR io addr: 0x0000ec00/0x0020
        0x001c: BAR mem 32bit addr: 0xfeadc000/0x00004000
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 10d3
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0a Min Gnt: 00 Max Lat: 00
        0x00c8: Capability 0x01: Power Management
        0x00d0: Capability 0x05: Message Signaled Interrupts (MSI)
        0x00e0: Capability 0x10: PCI Express
                Link Speed: 2.5 / 2.5 Gb/s Link Width: x1 / x1
        0x00a0: Capability 0x11: Extended Message Signaled Interrupts (MSI-X)
        0x0000: 10d38086 00100007 02000000 00000010
        0x0010: feae0000 00000000 0000ec01 feadc000
        0x0020: 00000000 00000000 00000000 10d315d9
        0x0030: 00000000 000000c8 00000000 0000010a
        0x0040: 00000000 00000000 00000000 00000000
        0x0050: 00000000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00040011 00000003 00002003 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 c822d001 0f002000
        0x00d0: 0081e005 fee00000 00000000 00000062
        0x00e0: 0001a010 00008cc1 00192810 00031c11
        0x00f0: 10110000 00000000 00000000 00000000
 4:4:0: Matrox MGA G200eW
        0x0000: Vendor ID: 102b Product ID: 0532
        0x0004: Command: 0007 Status ID: 0290
        0x0008: Class: 03 Subclass: 00 Interface: 00 Revision: 0a
        0x000c: BIST: 00 Header Type: 00 Latency Timer: 40 Cache Line Size: 08
        0x0010: BAR mem prefetchable 32bit addr: 0xfc000000/0x01000000
        0x0014: BAR mem 32bit addr: 0xfdffc000/0x00004000
        0x0018: BAR mem 32bit addr: 0xfe000000/0x00800000
        0x001c: BAR empty (00000000)
        0x0020: BAR empty (00000000)
        0x0024: BAR empty (00000000)
        0x0028: Cardbus CIS: 00000000
        0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
        0x0030: Expansion ROM Base Address: 00000000
        0x0038: 00000000
        0x003c: Interrupt Pin: 01 Line: 0f Min Gnt: 10 Max Lat: 20
        0x00dc: Capability 0x01: Power Management
        0x0000: 0532102b 02900007 0300000a 00004008
        0x0010: fc000008 fdffc000 fe000000 00000000
        0x0020: 00000000 00000000 00000000 060a15d9
        0x0030: 00000000 000000dc 00000000 2010010f
        0x0040: 00000120 00003c00 01ff0001 060a15d9
        0x0050: 0000b000 00000000 00000000 00000000
        0x0060: 00000000 00000000 00000000 00000000
        0x0070: 00000000 00000000 00000000 00000000
        0x0080: 00000000 00000000 00000000 00000000
        0x0090: 00000000 00000000 00000000 00000000
        0x00a0: 00000000 00000000 00000000 00000000
        0x00b0: 00000000 00000000 00000000 00000000
        0x00c0: 00000000 00000000 00000000 00000000
        0x00d0: 00000000 00000000 00000000 00210001
        0x00e0: 00000000 00000000 00000000 00000000
        0x00f0: 00000000 00000000 00000000 00000000

Reply | Threaded
Open this post in threaded view
|

Re: problem with round-robin redirection in pf (Bug)

Joerg Jung
Hi,

Am 03.01.2012 um 20:37 schrieb Sebastian Benoit <[hidden email]>:

> Daniel an I found a rather bizarre problem with round-robin tables in pf.
>
> We have a relayd setup with two webservers (IP's $A and $B). We observed
that
> when one of the webservers went down, so did the loadbalanced service (on
IP
> $C).
>
> We first noticed this in 2011-11 when deploying a pair of carped
> loadbalancers (running current) and debugged and narrowed it down to this
case,

> independently reproducible without relayd:
>
> (pf.conf contains an anchor "relayd/*")
>
> $ cat /etc/pf.testrule
> pass in quick on rdomain 0 inet proto tcp from any \
>        to $C port = 80 flags S/SA \
>        keep state (tcp.established 600) tag RELAYDHTTP \
>        rdr-to <http_foo> port 80 round-robin prio 0
>
> $ pfctl -a relayd/http_foo -t http_foo -T add $A
> $ pfctl -a relayd/http_foo -t http_foo -T add $B
> $ pfctl -f /etc/pf.testrule  -a relayd/http_foo
>
> This results in a working loadblanced setup:
>
> $ pfctl -a relayd/http_foo -t http_foo -T show
> $A
> $B
>
> Now consider the case where an IP is deleted from the table http_foo:
>
> $ pfctl -a relayd/http_foo -t http_foo -T delete $A
>
> or
>
> $ pfctl -a relayd/http_foo -t http_foo -T delete $B
>
> The bug triggers when this delete happens right after the _last_
redirection
> address was $B, and $C becomes inaccesible.
>
> It is independet of which of the addresses gets deleted, but the output
_order_ of $A
> and $B when printing the table with "-T show" is important.

Can you confirm that this bug happens in a plain PF setup also (i.e. without
relayd beeing involved)?

Regards,
Joerg

> The bug does not trigger with more than two addresses in the table.
> I found it with IPv4 and IPv6 addresses.
>
> For example real addresses used for A,B and C were
>
> A=2a00:15a8:6:100:5054:ff:fe00:67
> B=2a00:15a8:6:100:5054:ff:fe00:69
> C=2a00:15a8:6:100::27c3
>
> (and '69 has to be the last one used before deleting an address from the
> table).
>
> I also have a machine with IPv4 addresses where I can demonstrate the bug
;-)
>
> We narrowed the cause of the problem down to snapshots between 2011-06-29
> and 2011-07-16 and today found that a kernel build from CVS date
'2011-07-08

> 12:00:00 UTC' does not have the bug, but '2011-07-09 12:00:00 UTC' has it.
>
> We guess it might be rev. 1.92 of src/sys/net/pf_table.c.
>
> If you need more details, please say so.
> I can provide access to a system to demonstrate the bug if needed.
>
> /Benno & Daniel
>
> ----------------------------------------
>
> dmesg:
> OpenBSD 5.0-current (GENERIC.MP) #123: Wed Nov 16 15:56:56 MST 2011
>    [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC.MP
> cpu0: Intel(R) Atom(TM) CPU D525 @ 1.80GHz ("GenuineIntel" 686-class) 1.81
GHz
> cpu0:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,MWAIT,DS-CPL,TM2,SSSE
3,CX16,xTPR,PDCM,MOVBE,LAHF
> real mem  = 2146471936 (2047MB)
> avail mem = 2101256192 (2003MB)
> mainbus0 at root
> bios0 at mainbus0: AT/286+ BIOS, date 12/17/10, BIOS32 rev. 0 @ 0xf0010,
SMBIOS rev. 2.6 @ 0x9ac00 (19 entries)
> bios0: vendor American Megatrends Inc. version "1.1a" date 12/17/10
> bios0: Supermicro X7SPA-HF
> acpi0 at bios0: rev 2
> acpi0: sleep states S0 S1 S4 S5
> acpi0: tables DSDT FACP APIC MCFG OEMB HPET EINJ BERT ERST HEST
> acpi0: wakeup devices P0P1(S4) PS2K(S4) PS2M(S4) USB0(S4) USB1(S4) USB2(S4)
USB5(S4) EUSB(S4) USB3(S4) USB4(S4) USB6(S4) USBE(S4) P0P4(S4) P0P5(S4)
P0P6(S4) P0P7(S4) P0P8(S4) P0P9(S4) GBE_(S4) SLPB(S4)
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: apic clock running at 199MHz
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel(R) Atom(TM) CPU D525 @ 1.80GHz ("GenuineIntel" 686-class) 1.81
GHz
> cpu1:
FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUS
H,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,SBF,NXE,LONG,SSE3,MWAIT,DS-CPL,TM2,SSSE
3,CX16,xTPR,PDCM,MOVBE,LAHF

> ioapic0 at mainbus0: apid 3 pa 0xfec00000, version 20, 24 pins
> ioapic0: misconfigured as apic 1, remapped to apid 3
> acpimcfg0 at acpi0 addr 0xe0000000, bus 0-255
> acpihpet0 at acpi0: 14318179 Hz
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 4 (P0P1)
> acpiprt2 at acpi0: bus 1 (P0P4)
> acpiprt3 at acpi0: bus -1 (P0P5)
> acpiprt4 at acpi0: bus -1 (P0P6)
> acpiprt5 at acpi0: bus -1 (P0P7)
> acpiprt6 at acpi0: bus 2 (P0P8)
> acpiprt7 at acpi0: bus 3 (P0P9)
> acpicpu0 at acpi0
> acpicpu1 at acpi0
> acpibtn0 at acpi0: SLPB
> acpibtn1 at acpi0: PWRB
> bios0: ROM list: 0xc0000/0x8000 0xc8000/0x1000
> ipmi at mainbus0 not configured
> pci0 at mainbus0 bus 0: configuration mode 1 (bios)
> pchb0 at pci0 dev 0 function 0 "Intel Pineview DMI" rev 0x02
> uhci0 at pci0 dev 26 function 0 "Intel 82801I USB" rev 0x02: apic 3 int 16
> uhci1 at pci0 dev 26 function 1 "Intel 82801I USB" rev 0x02: apic 3 int 21
> uhci2 at pci0 dev 26 function 2 "Intel 82801I USB" rev 0x02: apic 3 int 19
> ehci0 at pci0 dev 26 function 7 "Intel 82801I USB" rev 0x02: apic 3 int 18
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb0 at pci0 dev 28 function 0 "Intel 82801I PCIE" rev 0x02: apic 3 int 17
> pci1 at ppb0 bus 1
> em0 at pci1 dev 0 function 0 "Intel PRO/1000 PT (82572EI)" rev 0x06: msi,
address 00:1b:21:b9:0e:b1
> ppb1 at pci0 dev 28 function 4 "Intel 82801I PCIE" rev 0x02: apic 3 int 17
> pci2 at ppb1 bus 2
> em1 at pci2 dev 0 function 0 "Intel PRO/1000 MT (82574L)" rev 0x00: msi,
address 00:25:90:3b:37:e2
> ppb2 at pci0 dev 28 function 5 "Intel 82801I PCIE" rev 0x02: apic 3 int 16
> pci3 at ppb2 bus 3
> em2 at pci3 dev 0 function 0 "Intel PRO/1000 MT (82574L)" rev 0x00: msi,
address 00:25:90:3b:37:e3

> uhci3 at pci0 dev 29 function 0 "Intel 82801I USB" rev 0x02: apic 3 int 23
> uhci4 at pci0 dev 29 function 1 "Intel 82801I USB" rev 0x02: apic 3 int 19
> uhci5 at pci0 dev 29 function 2 "Intel 82801I USB" rev 0x02: apic 3 int 18
> ehci1 at pci0 dev 29 function 7 "Intel 82801I USB" rev 0x02: apic 3 int 23
> usb1 at ehci1: USB revision 2.0
> uhub1 at usb1 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb3 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0x92
> pci4 at ppb3 bus 4
> vga1 at pci4 dev 4 function 0 "Matrox MGA G200eW" rev 0x0a
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> ichpcib0 at pci0 dev 31 function 0 "Intel 82801IR LPC" rev 0x02: PM
disabled
> ahci0 at pci0 dev 31 function 2 "Intel 82801I AHCI" rev 0x02: msi, AHCI 1.2
> scsibus0 at ahci0: 32 targets
> sd0 at scsibus0 targ 0 lun 0: <ATA, KINGSTON SS100S2, D100> SCSI3 0/direct
fixed t10.ATA_KINGSTON_SS100S28G_08GB40017544_
> sd0: 7641MB, 512 bytes/sector, 15649200 sectors, thin
> ichiic0 at pci0 dev 31 function 3 "Intel 82801I SMBus" rev 0x02: apic 3 int
18

> iic0 at ichiic0
> lm1 at iic0 addr 0x2d: unknown Winbond chip (ID 0x0)
> spdmem0 at iic0 addr 0x51: 2GB DDR3 SDRAM PC3-8500 SO-DIMM
> usb2 at uhci0: USB revision 1.0
> uhub2 at usb2 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb3 at uhci1: USB revision 1.0
> uhub3 at usb3 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb4 at uhci2: USB revision 1.0
> uhub4 at usb4 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb5 at uhci3: USB revision 1.0
> uhub5 at usb5 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb6 at uhci4: USB revision 1.0
> uhub6 at usb6 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> usb7 at uhci5: USB revision 1.0
> uhub7 at usb7 "Intel UHCI root hub" rev 1.00/1.00 addr 1
> isa0 at ichpcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com0: console
> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> pckbc0 at isa0 port 0x60/5
> pckbd0 at pckbc0 (kbd slot)
> pckbc0: using irq 1 for kbd slot
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> wbsio0 at isa0 port 0x2e/2: W83627DHG rev 0x25
> lm2 at wbsio0 port 0xca0/8: W83627DHG
> npx0 at isa0 port 0xf0/16: reported by CPUID; using exception 16
> mtrr: Pentium Pro MTRR support
> uhidev0 at uhub4 port 2 configuration 1 interface 0 "Winbond Electronics
Corp Hermon USB hidmouse Device" rev 1.10/0.01 addr 2
> uhidev0: iclass 3/1
> ums0 at uhidev0: 3 buttons, Z dir
> wsmouse0 at ums0 mux 0
> uhidev1 at uhub4 port 2 configuration 1 interface 1 "Winbond Electronics
Corp Hermon USB hidmouse Device" rev 1.10/0.01 addr 2

> uhidev1: iclass 3/1
> ukbd0 at uhidev1: 8 modifier keys, 6 key codes
> wskbd1 at ukbd0 mux 1
> wskbd1: connecting to wsdisplay0
> vscsi0 at root
> scsibus1 at vscsi0: 256 targets
> softraid0 at root
> scsibus2 at softraid0: 256 targets
> root on sd0a (1b0341781f8a2a6a.a) swap on sd0b dump on sd0b
> WARNING: / was not properly unmounted
> carp: carp0 demoted group carp by 1 to 129 (carpdev)
> carp: carp1 demoted group carp by 1 to 130 (carpdev)
> carp: pfsync0 demoted group carp by 1 to 131 (pfsync bulk start)
> carp: pfsync0 demoted group pfsync by 1 to 1 (pfsync bulk start)
>
> usbdevs:
> Controller /dev/usb0:
> addr 1: high speed, self powered, config 1, EHCI root hub(0x0000),
Intel(0x8086), rev 1.00
> port 1 powered
> port 2 powered
> port 3 powered
> port 4 powered
> port 5 powered
> port 6 powered
> Controller /dev/usb1:
> addr 1: high speed, self powered, config 1, EHCI root hub(0x0000),
Intel(0x8086), rev 1.00
> port 1 powered
> port 2 powered
> port 3 powered
> port 4 powered
> port 5 powered
> port 6 powered
> Controller /dev/usb2:
> addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
Intel(0x8086), rev 1.00
> port 1 powered
> port 2 powered
> Controller /dev/usb3:
> addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
Intel(0x8086), rev 1.00
> port 1 powered
> port 2 powered
> Controller /dev/usb4:
> addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
Intel(0x8086), rev 1.00
> port 1 powered
> port 2 addr 2: full speed, power 100 mA, config 1, Hermon USB hidmouse
Device(0x2221), Winbond Electronics Corp(0x0557), rev 0.01
> Controller /dev/usb5:
> addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
Intel(0x8086), rev 1.00
> port 1 powered
> port 2 powered
> Controller /dev/usb6:
> addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
Intel(0x8086), rev 1.00
> port 1 powered
> port 2 powered
> Controller /dev/usb7:
> addr 1: full speed, self powered, config 1, UHCI root hub(0x0000),
Intel(0x8086), rev 1.00

> port 1 powered
> port 2 powered
>
> pcidump:
> Domain /dev/pci0:
> 0:0:0: Intel Pineview DMI
>    0x0000: Vendor ID: 8086 Product ID: a000
>    0x0004: Command: 0006 Status ID: 2090
>    0x0008: Class: 06 Subclass: 00 Interface: 00 Revision: 02
>    0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
>    0x0010: BAR empty (00000000)
>    0x0014: BAR empty (00000000)
>    0x0018: BAR empty (00000000)
>    0x001c: BAR empty (00000000)
>    0x0020: BAR empty (00000000)
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
>    0x00e0: Capability 0x09: Vendor Specific
>    0x0000: a0008086 20900006 06000002 00000000
>    0x0010: 00000000 00000000 00000000 00000000
>    0x0020: 00000000 00000000 00000000 060a15d9
>    0x0030: 00000000 000000e0 00000000 00000000
>    0x0040: fed19001 00000000 fed14000 00000000
>    0x0050: 00000000 00000001 00000000 00000000
>    0x0060: e0000001 00000000 fed18001 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00000000 00000000 00000000 00000000
>    0x0090: 00011110 00333300 000003ff 00381a00
>    0x00a0: 08000020 80000000 80000000 80000000
>    0x00b0: 00008000 00000000 00000000 00000000
>    0x00c0: 00000000 00000000 00000000 00000000
>    0x00d0: 00000000 00000000 00000049 e6000000
>    0x00e0: c1080009 08111002 00000000 00000000
>    0x00f0: 00000000 00000001 00020fad 00000000
> 0:26:0: Intel 82801I USB
>    0x0000: Vendor ID: 8086 Product ID: 2937
>    0x0004: Command: 0005 Status ID: 0290
>    0x0008: Class: 0c Subclass: 03 Interface: 00 Revision: 02
>    0x000c: BIST: 00 Header Type: 80 Latency Timer: 00 Cache Line Size: 00
>    0x0010: BAR empty (00000000)
>    0x0014: BAR empty (00000000)
>    0x0018: BAR empty (00000000)
>    0x001c: BAR empty (00000000)
>    0x0020: BAR io addr: 0x0000bc00/0x0020
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
>    0x0050: Capability 0x13: PCI Advanced Features
>    0x0000: 29378086 02900005 0c030002 00800000
>    0x0010: 00000000 00000000 00000000 00000000
>    0x0020: 0000bc01 00000000 00000000 060a15d9
>    0x0030: 00000000 00000050 00000000 0000010b
>    0x0040: 00000000 00000000 00000000 00000000
>    0x0050: 03060013 00000000 00000000 00000000
>    0x0060: 00000010 00000000 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00000000 00000000 00000000 00000000
>    0x0090: 00000000 00000000 00000000 00000000
>    0x00a0: 00000000 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00002f00 00000000 00010000 00000000
>    0x00d0: 00000000 00000000 00000000 00000000
>    0x00e0: 00000000 00000000 00000000 00000000
>    0x00f0: 00000000 00000000 00020f86 00000000
> 0:26:1: Intel 82801I USB
>    0x0000: Vendor ID: 8086 Product ID: 2938
>    0x0004: Command: 0005 Status ID: 0290
>    0x0008: Class: 0c Subclass: 03 Interface: 00 Revision: 02
>    0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
>    0x0010: BAR empty (00000000)
>    0x0014: BAR empty (00000000)
>    0x0018: BAR empty (00000000)
>    0x001c: BAR empty (00000000)
>    0x0020: BAR io addr: 0x0000b880/0x0020
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 02 Line: 07 Min Gnt: 00 Max Lat: 00
>    0x0050: Capability 0x13: PCI Advanced Features
>    0x0000: 29388086 02900005 0c030002 00000000
>    0x0010: 00000000 00000000 00000000 00000000
>    0x0020: 0000b881 00000000 00000000 060a15d9
>    0x0030: 00000000 00000050 00000000 00000207
>    0x0040: 00000000 00000000 00000000 00000000
>    0x0050: 03060013 00000000 00000000 00000000
>    0x0060: 00000010 00000000 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00000000 00000000 00000000 00000000
>    0x0090: 00000000 00000000 00000000 00000000
>    0x00a0: 00000000 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00002f00 00000000 00010000 00000000
>    0x00d0: 00000000 00000000 00000000 00000000
>    0x00e0: 00000000 00000000 00000000 00000000
>    0x00f0: 00000000 00000000 00020f86 00000000
> 0:26:2: Intel 82801I USB
>    0x0000: Vendor ID: 8086 Product ID: 2939
>    0x0004: Command: 0005 Status ID: 0290
>    0x0008: Class: 0c Subclass: 03 Interface: 00 Revision: 02
>    0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
>    0x0010: BAR empty (00000000)
>    0x0014: BAR empty (00000000)
>    0x0018: BAR empty (00000000)
>    0x001c: BAR empty (00000000)
>    0x0020: BAR io addr: 0x0000b800/0x0020
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 04 Line: 0e Min Gnt: 00 Max Lat: 00
>    0x0050: Capability 0x13: PCI Advanced Features
>    0x0000: 29398086 02900005 0c030002 00000000
>    0x0010: 00000000 00000000 00000000 00000000
>    0x0020: 0000b801 00000000 00000000 060a15d9
>    0x0030: 00000000 00000050 00000000 0000040e
>    0x0040: 00000000 00000000 00000000 00000000
>    0x0050: 03060013 00000000 00000000 00000000
>    0x0060: 00000010 00000000 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00000000 00000000 00000000 00000000
>    0x0090: 00000000 00000000 00000000 00000000
>    0x00a0: 00000000 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00002f00 00000000 00010000 00000000
>    0x00d0: 00000000 00000000 00000000 00000000
>    0x00e0: 00000000 00000000 00000000 00000000
>    0x00f0: 00000000 00000000 00020f86 00000000
> 0:26:7: Intel 82801I USB
>    0x0000: Vendor ID: 8086 Product ID: 293c
>    0x0004: Command: 0006 Status ID: 0290
>    0x0008: Class: 0c Subclass: 03 Interface: 20 Revision: 02
>    0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
>    0x0010: BAR mem 32bit addr: 0xfebfbc00/0x00000400
>    0x0014: BAR empty (00000000)
>    0x0018: BAR empty (00000000)
>    0x001c: BAR empty (00000000)
>    0x0020: BAR empty (00000000)
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 03 Line: 0f Min Gnt: 00 Max Lat: 00
>    0x0050: Capability 0x01: Power Management
>    0x0058: Capability 0x0a: Debug Port
>    0x0098: Capability 0x13: PCI Advanced Features
>    0x0000: 293c8086 02900006 0c032002 00000000
>    0x0010: febfbc00 00000000 00000000 00000000
>    0x0020: 00000000 00000000 00000000 060a15d9
>    0x0030: 00000000 00000050 00000000 0000030f
>    0x0040: 00000000 00000000 00000000 00000000
>    0x0050: c9c25801 00000000 20a0980a 00000000
>    0x0060: 01ff2020 00000000 00000001 c0082000
>    0x0070: 0fdf0000 00000000 00000000 00000000
>    0x0080: 00000000 00000001 00000000 00000000
>    0x0090: 00000000 00000000 03060013 00000000
>    0x00a0: 00000000 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00000000 00000000 00000000 00000000
>    0x00d0: 00000000 00ffaa00 00000000 00000000
>    0x00e0: 00000000 00000000 00000000 00000000
>    0x00f0: 00000000 00408586 00020f86 2002130a
> 0:28:0: Intel 82801I PCIE
>    0x0000: Vendor ID: 8086 Product ID: 2940
>    0x0004: Command: 0107 Status ID: 0010
>    0x0008: Class: 06 Subclass: 04 Interface: 00 Revision: 02
>    0x000c: BIST: 00 Header Type: 81 Latency Timer: 00 Cache Line Size: 08
>    0x0010: 00000000
>    0x0014: 00000000
>    0x0018: Primary Bus: 0 Secondary Bus: 1 Subordinate Bus: 1
>            Secondary Latency Timer: 00
>    0x001c: I/O Base: c0 I/O Limit: c0 Secondary Status: 2000
>    0x0020: Memory Base: fe80 Memory Limit: fe80
>    0x0024: Prefetch Memory Base: fff1 Prefetch Memory Limit: 0001
>    0x0028: Prefetch Memory Base Upper 32 Bits: 00000000
>    0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
>    0x0030: I/O Base Upper 16 Bits: 0000 I/O Limit Upper 16 Bits: 0000
>    0x0038: Expansion ROM Base Address: 00000000
>    0x003c: Interrupt Pin: 01 Line: 0a Bridge Control: 0006
>    0x0040: Capability 0x10: PCI Express
>            Link Speed: 2.5 / 2.5 Gb/s Link Width: x1 / x4
>    0x0080: Capability 0x05: Message Signaled Interrupts (MSI)
>    0x0090: Capability 0x0d: PCI-PCI
>    0x00a0: Capability 0x01: Power Management
>    0x0000: 29408086 00100107 06040002 00810008
>    0x0010: 00000000 00000000 00010100 2000c0c0
>    0x0020: fe80fe80 0001fff1 00000000 00000000
>    0x0030: 00000000 00000040 00000000 0006010a
>    0x0040: 01418010 00008000 00100000 01112c41
>    0x0050: 30110040 00000560 00400028 00000000
>    0x0060: 00000000 00000000 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00009005 00000000 00000000 00000000
>    0x0090: 0000a00d 060a15d9 00000000 00000000
>    0x00a0: c8020001 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00000000 00000000 00000000 00000000
>    0x00d0: 00000000 00000000 08110000 00000000
>    0x00e0: 00c70000 00080706 00000030 00000000
>    0x00f0: 00000000 00000000 00020f86 00000000
> 0:28:4: Intel 82801I PCIE
>    0x0000: Vendor ID: 8086 Product ID: 2948
>    0x0004: Command: 0107 Status ID: 0010
>    0x0008: Class: 06 Subclass: 04 Interface: 00 Revision: 02
>    0x000c: BIST: 00 Header Type: 81 Latency Timer: 00 Cache Line Size: 08
>    0x0010: 00000000
>    0x0014: 00000000
>    0x0018: Primary Bus: 0 Secondary Bus: 2 Subordinate Bus: 2
>            Secondary Latency Timer: 00
>    0x001c: I/O Base: d0 I/O Limit: d0 Secondary Status: 2000
>    0x0020: Memory Base: fe90 Memory Limit: fe90
>    0x0024: Prefetch Memory Base: fff1 Prefetch Memory Limit: 0001
>    0x0028: Prefetch Memory Base Upper 32 Bits: 00000000
>    0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
>    0x0030: I/O Base Upper 16 Bits: 0000 I/O Limit Upper 16 Bits: 0000
>    0x0038: Expansion ROM Base Address: 00000000
>    0x003c: Interrupt Pin: 01 Line: 0a Bridge Control: 0006
>    0x0040: Capability 0x10: PCI Express
>            Link Speed: 2.5 / 2.5 Gb/s Link Width: x1 / x1
>    0x0080: Capability 0x05: Message Signaled Interrupts (MSI)
>    0x0090: Capability 0x0d: PCI-PCI
>    0x00a0: Capability 0x01: Power Management
>    0x0000: 29488086 00100107 06040002 00810008
>    0x0010: 00000000 00000000 00020200 2000d0d0
>    0x0020: fe90fe90 0001fff1 00000000 00000000
>    0x0030: 00000000 00000040 00000000 0006010a
>    0x0040: 01418010 00008000 00100000 05112c11
>    0x0050: 30110040 00000560 00400028 00000000
>    0x0060: 00000000 00000000 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00009005 00000000 00000000 00000000
>    0x0090: 0000a00d 060a15d9 00000000 00000000
>    0x00a0: c8020001 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00000000 00000000 00000000 00000000
>    0x00d0: 00000000 00000000 08110000 00000000
>    0x00e0: 00c70000 00080706 00000030 00000000
>    0x00f0: 00000000 00000000 00020f86 00000000
> 0:28:5: Intel 82801I PCIE
>    0x0000: Vendor ID: 8086 Product ID: 294a
>    0x0004: Command: 0107 Status ID: 0010
>    0x0008: Class: 06 Subclass: 04 Interface: 00 Revision: 02
>    0x000c: BIST: 00 Header Type: 81 Latency Timer: 00 Cache Line Size: 08
>    0x0010: 00000000
>    0x0014: 00000000
>    0x0018: Primary Bus: 0 Secondary Bus: 3 Subordinate Bus: 3
>            Secondary Latency Timer: 00
>    0x001c: I/O Base: e0 I/O Limit: e0 Secondary Status: 2000
>    0x0020: Memory Base: fea0 Memory Limit: fea0
>    0x0024: Prefetch Memory Base: fff1 Prefetch Memory Limit: 0001
>    0x0028: Prefetch Memory Base Upper 32 Bits: 00000000
>    0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
>    0x0030: I/O Base Upper 16 Bits: 0000 I/O Limit Upper 16 Bits: 0000
>    0x0038: Expansion ROM Base Address: 00000000
>    0x003c: Interrupt Pin: 02 Line: 0b Bridge Control: 0006
>    0x0040: Capability 0x10: PCI Express
>            Link Speed: 2.5 / 2.5 Gb/s Link Width: x1 / x1
>    0x0080: Capability 0x05: Message Signaled Interrupts (MSI)
>    0x0090: Capability 0x0d: PCI-PCI
>    0x00a0: Capability 0x01: Power Management
>    0x0000: 294a8086 00100107 06040002 00810008
>    0x0010: 00000000 00000000 00030300 2000e0e0
>    0x0020: fea0fea0 0001fff1 00000000 00000000
>    0x0030: 00000000 00000040 00000000 0006020b
>    0x0040: 01418010 00008000 00100000 06112c11
>    0x0050: 30110040 00000560 00400028 00000000
>    0x0060: 00000000 00000000 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00009005 00000000 00000000 00000000
>    0x0090: 0000a00d 060a15d9 00000000 00000000
>    0x00a0: c8020001 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00000000 00000000 00000000 00000000
>    0x00d0: 00000000 00000000 08110000 00000000
>    0x00e0: 00c70000 00080706 00000030 00000000
>    0x00f0: 00000000 00000000 00020f86 00000000
> 0:29:0: Intel 82801I USB
>    0x0000: Vendor ID: 8086 Product ID: 2934
>    0x0004: Command: 0005 Status ID: 0290
>    0x0008: Class: 0c Subclass: 03 Interface: 00 Revision: 02
>    0x000c: BIST: 00 Header Type: 80 Latency Timer: 00 Cache Line Size: 00
>    0x0010: BAR empty (00000000)
>    0x0014: BAR empty (00000000)
>    0x0018: BAR empty (00000000)
>    0x001c: BAR empty (00000000)
>    0x0020: BAR io addr: 0x0000b480/0x0020
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
>    0x0050: Capability 0x13: PCI Advanced Features
>    0x0000: 29348086 02900005 0c030002 00800000
>    0x0010: 00000000 00000000 00000000 00000000
>    0x0020: 0000b481 00000000 00000000 060a15d9
>    0x0030: 00000000 00000050 00000000 0000010b
>    0x0040: 00000000 00000000 00000000 00000000
>    0x0050: 03060013 00000000 00000000 00000000
>    0x0060: 00000010 00000000 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00000000 00000000 00000000 00000000
>    0x0090: 00000000 00000000 00000000 00000000
>    0x00a0: 00000000 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00002f00 00000000 00010000 00000000
>    0x00d0: 00000000 00000000 00000000 00000000
>    0x00e0: 00000000 00000000 00000000 00000000
>    0x00f0: 00000000 00000000 00020f86 00000000
> 0:29:1: Intel 82801I USB
>    0x0000: Vendor ID: 8086 Product ID: 2935
>    0x0004: Command: 0005 Status ID: 0290
>    0x0008: Class: 0c Subclass: 03 Interface: 00 Revision: 02
>    0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
>    0x0010: BAR empty (00000000)
>    0x0014: BAR empty (00000000)
>    0x0018: BAR empty (00000000)
>    0x001c: BAR empty (00000000)
>    0x0020: BAR io addr: 0x0000b400/0x0020
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 02 Line: 0e Min Gnt: 00 Max Lat: 00
>    0x0050: Capability 0x13: PCI Advanced Features
>    0x0000: 29358086 02900005 0c030002 00000000
>    0x0010: 00000000 00000000 00000000 00000000
>    0x0020: 0000b401 00000000 00000000 060a15d9
>    0x0030: 00000000 00000050 00000000 0000020e
>    0x0040: 00000000 00000000 00000000 00000000
>    0x0050: 03060013 00000000 00000000 00000000
>    0x0060: 00000010 00000000 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00000000 00000000 00000000 00000000
>    0x0090: 00000000 00000000 00000000 00000000
>    0x00a0: 00000000 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00002f00 00000000 00010000 00000000
>    0x00d0: 00000000 00000000 00000000 00000000
>    0x00e0: 00000000 00000000 00000000 00000000
>    0x00f0: 00000000 00000000 00020f86 00000000
> 0:29:2: Intel 82801I USB
>    0x0000: Vendor ID: 8086 Product ID: 2936
>    0x0004: Command: 0005 Status ID: 0290
>    0x0008: Class: 0c Subclass: 03 Interface: 00 Revision: 02
>    0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
>    0x0010: BAR empty (00000000)
>    0x0014: BAR empty (00000000)
>    0x0018: BAR empty (00000000)
>    0x001c: BAR empty (00000000)
>    0x0020: BAR io addr: 0x0000b080/0x0020
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 03 Line: 0f Min Gnt: 00 Max Lat: 00
>    0x0050: Capability 0x13: PCI Advanced Features
>    0x0000: 29368086 02900005 0c030002 00000000
>    0x0010: 00000000 00000000 00000000 00000000
>    0x0020: 0000b081 00000000 00000000 060a15d9
>    0x0030: 00000000 00000050 00000000 0000030f
>    0x0040: 00000000 00000000 00000000 00000000
>    0x0050: 03060013 00000000 00000000 00000000
>    0x0060: 00000010 00000000 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00000000 00000000 00000000 00000000
>    0x0090: 00000000 00000000 00000000 00000000
>    0x00a0: 00000000 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00002f00 00000000 00010000 00000000
>    0x00d0: 00000000 00000000 00000000 00000000
>    0x00e0: 00000000 00000000 00000000 00000000
>    0x00f0: 00000000 00000000 00020f86 00000000
> 0:29:7: Intel 82801I USB
>    0x0000: Vendor ID: 8086 Product ID: 293a
>    0x0004: Command: 0006 Status ID: 0290
>    0x0008: Class: 0c Subclass: 03 Interface: 20 Revision: 02
>    0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
>    0x0010: BAR mem 32bit addr: 0xfebfb800/0x00000400
>    0x0014: BAR empty (00000000)
>    0x0018: BAR empty (00000000)
>    0x001c: BAR empty (00000000)
>    0x0020: BAR empty (00000000)
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
>    0x0050: Capability 0x01: Power Management
>    0x0058: Capability 0x0a: Debug Port
>    0x0098: Capability 0x13: PCI Advanced Features
>    0x0000: 293a8086 02900006 0c032002 00000000
>    0x0010: febfb800 00000000 00000000 00000000
>    0x0020: 00000000 00000000 00000000 060a15d9
>    0x0030: 00000000 00000050 00000000 0000010b
>    0x0040: 00000000 00000000 00000000 00000000
>    0x0050: c9c25801 00000000 20a0980a 00000000
>    0x0060: 01ff2020 00000000 00000001 c0082000
>    0x0070: 0fdf0000 00000000 00000000 00000000
>    0x0080: 00000000 00000001 00000000 00000000
>    0x0090: 00000000 00000000 03060013 00000000
>    0x00a0: 00000000 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00000000 00000000 00000000 00000000
>    0x00d0: 00000000 00ffaa00 00000000 00000000
>    0x00e0: 00000000 00000000 00000000 00000000
>    0x00f0: 00000000 00408586 00020f86 2002130a
> 0:30:0: Intel 82801BA Hub-to-PCI
>    0x0000: Vendor ID: 8086 Product ID: 244e
>    0x0004: Command: 0107 Status ID: 0010
>    0x0008: Class: 06 Subclass: 04 Interface: 01 Revision: 92
>    0x000c: BIST: 00 Header Type: 01 Latency Timer: 00 Cache Line Size: 00
>    0x0010: 00000000
>    0x0014: 00000000
>    0x0018: Primary Bus: 0 Secondary Bus: 4 Subordinate Bus: 4
>            Secondary Latency Timer: 20
>    0x001c: I/O Base: f0 I/O Limit: 00 Secondary Status: 2280
>    0x0020: Memory Base: fdf0 Memory Limit: fe70
>    0x0024: Prefetch Memory Base: fc01 Prefetch Memory Limit: fcf1
>    0x0028: Prefetch Memory Base Upper 32 Bits: 00000000
>    0x002c: Prefetch Memory Limit Upper 32 Bits: 00000000
>    0x0030: I/O Base Upper 16 Bits: 0000 I/O Limit Upper 16 Bits: 0000
>    0x0038: Expansion ROM Base Address: 00000000
>    0x003c: Interrupt Pin: 00 Line: ff Bridge Control: 001a
>    0x0050: Capability 0x0d: PCI-PCI
>    0x0000: 244e8086 00100107 06040192 00010000
>    0x0010: 00000000 00000000 20040400 228000f0
>    0x0020: fe70fdf0 fcf1fc01 00000000 00000000
>    0x0030: 00000000 00000050 00000000 001a00ff
>    0x0040: 00000000 00000000 00000000 00001200
>    0x0050: 0000000d 060a15d9 00000000 00000000
>    0x0060: 00000000 00000000 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00000000 00000000 00000000 00000000
>    0x0090: 00000000 00000000 00000000 00000000
>    0x00a0: 00000000 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00000000 00000000 00000000 00000000
>    0x00d0: 00000000 00000000 00000000 00000000
>    0x00e0: 00000000 00000000 00000000 00000000
>    0x00f0: 00000000 00000000 00020f86 00000000
> 0:31:0: Intel 82801IR LPC
>    0x0000: Vendor ID: 8086 Product ID: 2916
>    0x0004: Command: 0007 Status ID: 0210
>    0x0008: Class: 06 Subclass: 01 Interface: 00 Revision: 02
>    0x000c: BIST: 00 Header Type: 80 Latency Timer: 00 Cache Line Size: 00
>    0x0010: BAR empty (00000000)
>    0x0014: BAR empty (00000000)
>    0x0018: BAR empty (00000000)
>    0x001c: BAR empty (00000000)
>    0x0020: BAR empty (00000000)
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 00 Line: 00 Min Gnt: 00 Max Lat: 00
>    0x00e0: Capability 0x09: Vendor Specific
>    0x0000: 29168086 02100007 06010002 00800000
>    0x0010: 00000000 00000000 00000000 00000000
>    0x0020: 00000000 00000000 00000000 060a15d9
>    0x0030: 00000000 000000e0 00000000 00000000
>    0x0040: 00000801 00000080 00000481 00000010
>    0x0050: 00000000 00000000 00000000 00000000
>    0x0060: 0e0f0a0b 000000d0 0b0a0780 000000f8
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 34030010 000c02e1 000c03e1 000c1641
>    0x0090: 000c0ca1 00000000 00000000 00000000
>    0x00a0: 00000620 00000208 00000013 40000300
>    0x00b0: 00f00000 00000000 20000000 00000000
>    0x00c0: 00000000 00000000 00000000 00000000
>    0x00d0: 00112233 00004567 0000ffc0 00000000
>    0x00e0: 100c0009 02c40000 00000064 00000000
>    0x00f0: fed1c001 00000000 00020f86 00000000
> 0:31:2: Intel 82801I AHCI
>    0x0000: Vendor ID: 8086 Product ID: 2922
>    0x0004: Command: 0007 Status ID: 02b0
>    0x0008: Class: 01 Subclass: 06 Interface: 01 Revision: 02
>    0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
>    0x0010: BAR io addr: 0x0000a480/0x0008
>    0x0014: BAR io addr: 0x0000b000/0x0004
>    0x0018: BAR io addr: 0x0000ac00/0x0008
>    0x001c: BAR io addr: 0x0000a880/0x0004
>    0x0020: BAR io addr: 0x0000a800/0x0020
>    0x0024: BAR mem 32bit addr: 0xfebfb000/0x00000800
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 02 Line: 0e Min Gnt: 00 Max Lat: 00
>    0x0080: Capability 0x05: Message Signaled Interrupts (MSI)
>    0x0070: Capability 0x01: Power Management
>    0x00a8: Capability 0x12: SATA
>    0x00b0: Capability 0x13: PCI Advanced Features
>    0x0000: 29228086 02b00007 01060102 00000000
>    0x0010: 0000a481 0000b001 0000ac01 0000a881
>    0x0020: 0000a801 febfb000 00000000 060a15d9
>    0x0030: 00000000 00000080 00000000 0000020e
>    0x0040: 80008000 00000000 00000000 00000000
>    0x0050: 00000000 00000000 00000000 00000000
>    0x0060: 00000000 00000000 00000000 00000000
>    0x0070: 4003a801 00000008 00000000 00000000
>    0x0080: 00097005 fee00000 00000050 00000000
>    0x0090: 813f0060 00000193 00000000 00000000
>    0x00a0: 00000000 00000000 0010b012 00000048
>    0x00b0: 03060013 00000000 00000000 00000000
>    0x00c0: 00000000 00000000 00000000 00000000
>    0x00d0: 00000000 00000000 00000000 00000000
>    0x00e0: 00000000 00000000 00000000 00000000
>    0x00f0: 00000000 00000000 00020f86 00000000
> 0:31:3: Intel 82801I SMBus
>    0x0000: Vendor ID: 8086 Product ID: 2930
>    0x0004: Command: 0003 Status ID: 0280
>    0x0008: Class: 0c Subclass: 05 Interface: 00 Revision: 02
>    0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 00
>    0x0010: BAR mem 64bit addr: 0x00000000febfac00/0x00000100
>    0x0018: BAR empty (00000000)
>    0x001c: BAR empty (00000000)
>    0x0020: BAR io addr: 0x00000400/0x0020
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 03 Line: 0f Min Gnt: 00 Max Lat: 00
>    0x0000: 29308086 02800003 0c050002 00000000
>    0x0010: febfac04 00000000 00000000 00000000
>    0x0020: 00000401 00000000 00000000 060a15d9
>    0x0030: 00000000 00000000 00000000 0000030f
>    0x0040: 00000001 00000000 00000000 00000000
>    0x0050: 00000000 00000000 00000000 00000000
>    0x0060: 00040402 080800f0 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00000004 00000000 00000000 00000000
>    0x0090: 00000000 00000000 00000000 00000000
>    0x00a0: 00000000 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00000000 00000000 00000000 00000000
>    0x00d0: 00000000 00000000 00000000 00000000
>    0x00e0: 00000000 00000000 00000000 00000000
>    0x00f0: 00000000 00000000 00020f86 00000000
> 1:0:0: Intel PRO/1000 PT (82572EI)
>    0x0000: Vendor ID: 8086 Product ID: 107d
>    0x0004: Command: 0007 Status ID: 0010
>    0x0008: Class: 02 Subclass: 00 Interface: 00 Revision: 06
>    0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 08
>    0x0010: BAR mem 32bit addr: 0xfe8e0000/0x00020000
>    0x0014: BAR mem 32bit addr: 0xfe8c0000/0x00020000
>    0x0018: BAR io addr: 0x0000cc00/0x0020
>    0x001c: BAR empty (00000000)
>    0x0020: BAR empty (00000000)
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 8086 Product ID: 1082
>    0x0030: Expansion ROM Base Address: fe8a0000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
>    0x00c8: Capability 0x01: Power Management
>    0x00d0: Capability 0x05: Message Signaled Interrupts (MSI)
>    0x00e0: Capability 0x10: PCI Express
>            Link Speed: 2.5 / 2.5 Gb/s Link Width: x1 / x1
>    0x0000: 107d8086 00100007 02000006 00000008
>    0x0010: fe8e0000 fe8c0000 0000cc01 00000000
>    0x0020: 00000000 00000000 00000000 10828086
>    0x0030: fe8a0000 000000c8 00000000 0000010b
>    0x0040: 00000000 00000000 00000000 00000000
>    0x0050: 00000000 00000000 00000000 00000000
>    0x0060: 00000000 00000000 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00000000 00000000 00000000 00000000
>    0x0090: 00000000 00000000 00000000 00000000
>    0x00a0: 00000000 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00000000 00000000 c822d001 21002000
>    0x00d0: 0081e005 fee00000 00000000 00000060
>    0x00e0: 00010010 00000cc1 001a2810 00036411
>    0x00f0: 10110000 00000000 00000000 00000000
> 2:0:0: Intel PRO/1000 MT (82574L)
>    0x0000: Vendor ID: 8086 Product ID: 10d3
>    0x0004: Command: 0007 Status ID: 0010
>    0x0008: Class: 02 Subclass: 00 Interface: 00 Revision: 00
>    0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 10
>    0x0010: BAR mem 32bit addr: 0xfe9e0000/0x00020000
>    0x0014: BAR empty (00000000)
>    0x0018: BAR io addr: 0x0000dc00/0x0020
>    0x001c: BAR mem 32bit addr: 0xfe9dc000/0x00004000
>    0x0020: BAR empty (00000000)
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 10d3
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 01 Line: 0b Min Gnt: 00 Max Lat: 00
>    0x00c8: Capability 0x01: Power Management
>    0x00d0: Capability 0x05: Message Signaled Interrupts (MSI)
>    0x00e0: Capability 0x10: PCI Express
>            Link Speed: 2.5 / 2.5 Gb/s Link Width: x1 / x1
>    0x00a0: Capability 0x11: Extended Message Signaled Interrupts (MSI-X)
>    0x0000: 10d38086 00100007 02000000 00000010
>    0x0010: fe9e0000 00000000 0000dc01 fe9dc000
>    0x0020: 00000000 00000000 00000000 10d315d9
>    0x0030: 00000000 000000c8 00000000 0000010b
>    0x0040: 00000000 00000000 00000000 00000000
>    0x0050: 00000000 00000000 00000000 00000000
>    0x0060: 00000000 00000000 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00000000 00000000 00000000 00000000
>    0x0090: 00000000 00000000 00000000 00000000
>    0x00a0: 00040011 00000003 00002003 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00000000 00000000 c822d001 0f002000
>    0x00d0: 0081e005 fee00000 00000000 00000061
>    0x00e0: 0001a010 00008cc1 00192810 00031c11
>    0x00f0: 10110000 00000000 00000000 00000000
> 3:0:0: Intel PRO/1000 MT (82574L)
>    0x0000: Vendor ID: 8086 Product ID: 10d3
>    0x0004: Command: 0007 Status ID: 0010
>    0x0008: Class: 02 Subclass: 00 Interface: 00 Revision: 00
>    0x000c: BIST: 00 Header Type: 00 Latency Timer: 00 Cache Line Size: 10
>    0x0010: BAR mem 32bit addr: 0xfeae0000/0x00020000
>    0x0014: BAR empty (00000000)
>    0x0018: BAR io addr: 0x0000ec00/0x0020
>    0x001c: BAR mem 32bit addr: 0xfeadc000/0x00004000
>    0x0020: BAR empty (00000000)
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 10d3
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 01 Line: 0a Min Gnt: 00 Max Lat: 00
>    0x00c8: Capability 0x01: Power Management
>    0x00d0: Capability 0x05: Message Signaled Interrupts (MSI)
>    0x00e0: Capability 0x10: PCI Express
>            Link Speed: 2.5 / 2.5 Gb/s Link Width: x1 / x1
>    0x00a0: Capability 0x11: Extended Message Signaled Interrupts (MSI-X)
>    0x0000: 10d38086 00100007 02000000 00000010
>    0x0010: feae0000 00000000 0000ec01 feadc000
>    0x0020: 00000000 00000000 00000000 10d315d9
>    0x0030: 00000000 000000c8 00000000 0000010a
>    0x0040: 00000000 00000000 00000000 00000000
>    0x0050: 00000000 00000000 00000000 00000000
>    0x0060: 00000000 00000000 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00000000 00000000 00000000 00000000
>    0x0090: 00000000 00000000 00000000 00000000
>    0x00a0: 00040011 00000003 00002003 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00000000 00000000 c822d001 0f002000
>    0x00d0: 0081e005 fee00000 00000000 00000062
>    0x00e0: 0001a010 00008cc1 00192810 00031c11
>    0x00f0: 10110000 00000000 00000000 00000000
> 4:4:0: Matrox MGA G200eW
>    0x0000: Vendor ID: 102b Product ID: 0532
>    0x0004: Command: 0007 Status ID: 0290
>    0x0008: Class: 03 Subclass: 00 Interface: 00 Revision: 0a
>    0x000c: BIST: 00 Header Type: 00 Latency Timer: 40 Cache Line Size: 08
>    0x0010: BAR mem prefetchable 32bit addr: 0xfc000000/0x01000000
>    0x0014: BAR mem 32bit addr: 0xfdffc000/0x00004000
>    0x0018: BAR mem 32bit addr: 0xfe000000/0x00800000
>    0x001c: BAR empty (00000000)
>    0x0020: BAR empty (00000000)
>    0x0024: BAR empty (00000000)
>    0x0028: Cardbus CIS: 00000000
>    0x002c: Subsystem Vendor ID: 15d9 Product ID: 060a
>    0x0030: Expansion ROM Base Address: 00000000
>    0x0038: 00000000
>    0x003c: Interrupt Pin: 01 Line: 0f Min Gnt: 10 Max Lat: 20
>    0x00dc: Capability 0x01: Power Management
>    0x0000: 0532102b 02900007 0300000a 00004008
>    0x0010: fc000008 fdffc000 fe000000 00000000
>    0x0020: 00000000 00000000 00000000 060a15d9
>    0x0030: 00000000 000000dc 00000000 2010010f
>    0x0040: 00000120 00003c00 01ff0001 060a15d9
>    0x0050: 0000b000 00000000 00000000 00000000
>    0x0060: 00000000 00000000 00000000 00000000
>    0x0070: 00000000 00000000 00000000 00000000
>    0x0080: 00000000 00000000 00000000 00000000
>    0x0090: 00000000 00000000 00000000 00000000
>    0x00a0: 00000000 00000000 00000000 00000000
>    0x00b0: 00000000 00000000 00000000 00000000
>    0x00c0: 00000000 00000000 00000000 00000000
>    0x00d0: 00000000 00000000 00000000 00210001
>    0x00e0: 00000000 00000000 00000000 00000000
>    0x00f0: 00000000 00000000 00000000 00000000

Reply | Threaded
Open this post in threaded view
|

Re: problem with round-robin redirection in pf (Bug)

Sebastian Benoit-2
Joerg Zinke([hidden email]) on 2012.01.03 21:42:08 +0100:

> Hi,
>
> Am 03.01.2012 um 20:37 schrieb Sebastian Benoit <[hidden email]>:
>
> > Daniel an I found a rather bizarre problem with round-robin tables in pf.
> >
> > We have a relayd setup with two webservers (IP's $A and $B). We observed that
> > when one of the webservers went down, so did the loadbalanced service (on IP
> > $C).
> >
> > We first noticed this in 2011-11 when deploying a pair of carped
> > loadbalancers (running current) and debugged and narrowed it down to this case,
> > independently reproducible without relayd:
> >
> > (pf.conf contains an anchor "relayd/*")
> >
> > $ cat /etc/pf.testrule
> > pass in quick on rdomain 0 inet proto tcp from any \
> >        to $C port = 80 flags S/SA \
> >        keep state (tcp.established 600) tag RELAYDHTTP \
> >        rdr-to <http_foo> port 80 round-robin prio 0
> >
> > $ pfctl -a relayd/http_foo -t http_foo -T add $A
> > $ pfctl -a relayd/http_foo -t http_foo -T add $B
> > $ pfctl -f /etc/pf.testrule  -a relayd/http_foo
> >
> > This results in a working loadblanced setup:
> >
> > $ pfctl -a relayd/http_foo -t http_foo -T show
> > $A
> > $B
> >
> > Now consider the case where an IP is deleted from the table http_foo:
> >
> > $ pfctl -a relayd/http_foo -t http_foo -T delete $A
> >
> > or
> >
> > $ pfctl -a relayd/http_foo -t http_foo -T delete $B
> >
> > The bug triggers when this delete happens right after the _last_ redirection
> > address was $B, and $C becomes inaccesible.
> >
> > It is independet of which of the addresses gets deleted, but the output _order_ of $A
> > and $B when printing the table with "-T show" is important.
>
> Can you confirm that this bug happens in a plain PF setup also (i.e. without relayd beeing involved)?

Yes, i just chose to use the relayd anchor (as in the commands above), but
in most of my test, especially today when i checked different kernels relayd
was never started.

/Benno

Reply | Threaded
Open this post in threaded view
|

Re: problem with round-robin redirection in pf (Bug)

Sebastian Benoit
Sebastian Benoit([hidden email]) on 2012.01.03 23:01:17 +0100:
> > Can you confirm that this bug happens in a plain PF setup also (i.e. without relayd beeing involved)?
>
> Yes, i just chose to use the relayd anchor (as in the commands above), but
> in most of my test, especially today when i checked different kernels relayd
> was never started.

just to be clear, this also works:

in main pf.conf:

 table <http_27c3> persist
 pass in quick on rdomain 0 inet6 proto tcp from any \
        to 2a00:15a8:6:100::27c3 port = 80 flags S/SA \
        keep state (tcp.established 600) tag RELAYDHTTP \
        rdr-to <http_27c3> port 80 round-robin prio 0

and then

# pfctl -t http_27c3 -T add 2a00:15a8:6:100:5054:ff:fe00:69
# pfctl -t http_27c3 -T add 2a00:15a8:6:100:5054:ff:fe00:67
# pfctl -t http_27c3 -T show                        
   2a00:15a8:6:100:5054:ff:fe00:67
   2a00:15a8:6:100:5054:ff:fe00:69

[get url until 2a00:15a8:6:100:5054:ff:fe00:69 was hit]

# pfctl -t http_27c3 -T delete 2a00:15a8:6:100:5054:ff:fe00:69

-> next get request times out

and also with ipv4...

/Benno

Reply | Threaded
Open this post in threaded view
|

Re: problem with round-robin redirection in pf (Bug)

Todd T. Fries-2
In reply to this post by Joerg Jung
Penned by Joerg Zinke on 20120103 14:42.08, we have:
| Hi,
|
| Am 03.01.2012 um 20:37 schrieb Sebastian Benoit <[hidden email]>:
|
| > Daniel an I found a rather bizarre problem with round-robin tables in pf.
| >
| > We have a relayd setup with two webservers (IP's $A and $B). We observed
| that
| > when one of the webservers went down, so did the loadbalanced service (on
| IP
| > $C).
| >
| > We first noticed this in 2011-11 when deploying a pair of carped
| > loadbalancers (running current) and debugged and narrowed it down to this
| case,
| > independently reproducible without relayd:
| >
| > (pf.conf contains an anchor "relayd/*")
| >
| > $ cat /etc/pf.testrule
| > pass in quick on rdomain 0 inet proto tcp from any \
| >        to $C port = 80 flags S/SA \
| >        keep state (tcp.established 600) tag RELAYDHTTP \
| >        rdr-to <http_foo> port 80 round-robin prio 0
| >
| > $ pfctl -a relayd/http_foo -t http_foo -T add $A
| > $ pfctl -a relayd/http_foo -t http_foo -T add $B
| > $ pfctl -f /etc/pf.testrule  -a relayd/http_foo
| >
| > This results in a working loadblanced setup:
| >
| > $ pfctl -a relayd/http_foo -t http_foo -T show
| > $A
| > $B
| >
| > Now consider the case where an IP is deleted from the table http_foo:
| >
| > $ pfctl -a relayd/http_foo -t http_foo -T delete $A
| >
| > or
| >
| > $ pfctl -a relayd/http_foo -t http_foo -T delete $B
| >
| > The bug triggers when this delete happens right after the _last_
| redirection
| > address was $B, and $C becomes inaccesible.
| >
| > It is independet of which of the addresses gets deleted, but the output
| _order_ of $A
| > and $B when printing the table with "-T show" is important.
|
| Can you confirm that this bug happens in a plain PF setup also (i.e. without
| relayd beeing involved)?

I can echo the above sentiments and confirm that independent of relayd this is
recreatable.  My magic workaround ended up putting an additional ip in the pool for
each host thus the table is either 4 or 2 never 1.  Obviously round-robin is broken
when the table reaches 1 host, but someone other than me will be needed to fix the
code.
--
Todd Fries .. [hidden email]

 _____________________________________________
|                                             \  1.636.410.0632 (voice)
| Free Daemon Consulting, LLC                 \  1.405.227.9094 (voice)
| http://FreeDaemonConsulting.com             \  1.866.792.3418 (FAX)
| 2525 NW Expy #525, Oklahoma City, OK 73112  \  sip:[hidden email]
| "..in support of free software solutions."  \  sip:[hidden email]
 \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
                                                 
              37E7 D3EB 74D0 8D66 A68D  B866 0326 204E 3F42 004A
                        http://todd.fries.net/pgp.txt

Reply | Threaded
Open this post in threaded view
|

Re: problem with round-robin redirection in pf (Bug)

Piotr Dzionek
Hi,
I am using OpenBSD 5.0 and I noticed similar bug. It happens when I delete higher IP address from the table (2 web servers). It is always the last one in pfctl -t table -T show command. When i delete it just after reloading pf everything is fine. However if there was some state or even if it was expired (no states after sometime) the problem occurs. Rdr rule stops working even though there is still one IP address in the load balancing table. Flushing States and Sources doesn't help. Only full reload of rule set and deleting the host address fix this.
So finally, I would like to know if this problem still exists in latest versions of OpenBSD or maybe it was fixed?


Reply | Threaded
Open this post in threaded view
|

Re: problem with round-robin redirection in pf (Bug)

Mike Belopuhov-5
On 14 December 2012 15:30, Piotr Dzionek <[hidden email]> wrote:

> Hi,
> I am using OpenBSD 5.0 and I noticed similar bug. It happens when I delete
> higher IP address from the table (2 web servers). It is always the last one
> in pfctl -t table -T show command. When i delete it just after reloading pf
> everything is fine. However if there was some state or even if it was
> expired (no states after sometime) the problem occurs. Rdr rule stops
> working even though there is still one IP address in the load balancing
> table. Flushing States and Sources doesn't help. Only full reload of rule
> set and deleting the host address fix this.
> So finally, I would like to know if this problem still exists in latest
> versions of OpenBSD or maybe it was fixed?

some problems in this are was fixed, there's a diff to solve some other.
you should update to 5.2-current and if it doesn't work for you, please
file a proper report.

Reply | Threaded
Open this post in threaded view
|

Re: problem with round-robin redirection in pf (Bug)

Piotr Dzionek
This post was updated on .
Thx for info. I will try an upgrade to 5.2 and next to 5.3 when it will
be available.

W dniu 14.12.2012 15:36, Mike Belopuhov pisze:
> On 14 December 2012 15:30, Piotr Dzionek <piotr.dzionek@intercon.pl> wrote:
>  
>> Hi,
>> I am using OpenBSD 5.0 and I noticed similar bug. It happens when I delete
>> higher IP address from the table (2 web servers). It is always the last one
>> in pfctl -t table -T show command. When i delete it just after reloading pf
>> everything is fine. However if there was some state or even if it was
>> expired (no states after sometime) the problem occurs. Rdr rule stops
>> working even though there is still one IP address in the load balancing
>> table. Flushing States and Sources doesn't help. Only full reload of rule
>> set and deleting the host address fix this.
>> So finally, I would like to know if this problem still exists in latest
>> versions of OpenBSD or maybe it was fixed?
>>    
> some problems in this are was fixed, there's a diff to solve some other.
> you should update to 5.2-current and if it doesn't work for you, please
> file a proper report.
>
>
>