Troubleshooting pf congestion

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Troubleshooting pf congestion

Scott Reese-2
Greetings:

I am troubleshooting an issue: users complaining about network performance. The firewall
is an OpenBSD 6.7 system with patches applied. I've traced the issue and I'm seeing the
congestion counter incrementing on system. The problems that we're seeing fit with what
I have been able to find about congestion - when the firewall is congested it continues
passing packets that match existing state entries but it will not create any new state
entries until the congestion clears.

I'm having trouble troubleshooting it beyond that point because I have not been able to
find any additional information about what the congestion counter is counting. There is
the information in the pfctl man page: "congestion: network interface queue congested",
but beyond that I can't really find any information about exactly what network interface
queue is congested.

I'm not seeing packets being dropped, either on the switch side or firewall side that
correspond with the congestion counter going up. The average on the congestion counter
stays around 10/s, but what it's really doing is going up by 100-300/s for short periods
and then not moving for longer periods.

If anyone could spare a couple of sentences or a share a link to a page detailing what
state causes the system to consider itself contested, I would appreciate it.

Thanks for your time.

-Scott


System dmesg:

OpenBSD 6.7 (GENERIC.MP) #6: Thu Sep  3 14:08:18 MDT 2020
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 8386699264 (7998MB)
avail mem = 8119902208 (7743MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7fb76000 (62 entries)
bios0: vendor American Megatrends Inc. version "2.2" date 05/23/2018
bios0: Supermicro X11SSL-F
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG HPET LPIT SSDT SSDT SSDT DBGP DBG2 SSDT SSDT UEFI SSDT DMAR EINJ ERST BERT HEST
acpi0: wakeup devices PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) PEGP(S4) RP09(S4) PXSX(S4) RP10(S4) PXSX(S4) RP11(S4) PXSX(S4) RP12(S4) PXSX(S4) RP13(S4) PXSX(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3901.62 MHz, 06-9e-09
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 24MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 1 (application processor)
cpu4: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 1, core 0, package 0
cpu5 at mainbus0: apid 3 (application processor)
cpu5: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.00 MHz, 06-9e-09
cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 1, core 1, package 0
cpu6 at mainbus0: apid 5 (application processor)
cpu6: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu6: 256KB 64b/line 8-way L2 cache
cpu6: smt 1, core 2, package 0
cpu7 at mainbus0: apid 7 (application processor)
cpu7: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu7: 256KB 64b/line 8-way L2 cache
cpu7: smt 1, core 3, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 23999999 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEG0)
acpiprt2 at acpi0: bus -1 (PEG1)
acpiprt3 at acpi0: bus -1 (PEG2)
acpiprt4 at acpi0: bus 2 (RP09)
acpiprt5 at acpi0: bus 3 (RP10)
acpiprt6 at acpi0: bus 4 (RP11)
acpiprt7 at acpi0: bus 5 (BR51)
acpiprt8 at acpi0: bus -1 (RP12)
acpiprt9 at acpi0: bus -1 (RP13)
acpiprt10 at acpi0: bus -1 (RP01)
acpiprt11 at acpi0: bus -1 (RP05)
acpiprt12 at acpi0: bus -1 (RP17)
acpiprt13 at acpi0: bus -1 (RP18)
acpiprt14 at acpi0: bus -1 (RP19)
acpiprt15 at acpi0: bus -1 (RP20)
acpiprt16 at acpi0: bus -1 (RP14)
acpiprt17 at acpi0: bus -1 (RP15)
acpiprt18 at acpi0: bus -1 (RP16)
acpiec0 at acpi0: not present
acpicpu0 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu2 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu3 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu4 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu5 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu6 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu7 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpipwrres0 at acpi0: PG00, resource for PEG0
acpipwrres1 at acpi0: PG01, resource for PEG1
acpipwrres2 at acpi0: PG02, resource for PEG2
acpipwrres3 at acpi0: WRST
acpipwrres4 at acpi0: WRST
acpipwrres5 at acpi0: WRST
acpipwrres6 at acpi0: WRST
acpipwrres7 at acpi0: WRST
acpipwrres8 at acpi0: WRST
acpipwrres9 at acpi0: WRST
acpipwrres10 at acpi0: WRST
acpipwrres11 at acpi0: WRST
acpipwrres12 at acpi0: WRST
acpipwrres13 at acpi0: WRST
acpipwrres14 at acpi0: WRST
acpipwrres15 at acpi0: WRST
acpipwrres16 at acpi0: WRST
acpipwrres17 at acpi0: WRST
acpipwrres18 at acpi0: WRST
acpipwrres19 at acpi0: WRST
acpipwrres20 at acpi0: WRST
acpipwrres21 at acpi0: WRST
acpipwrres22 at acpi0: WRST
acpipwrres23 at acpi0: FN00, resource for FAN0
acpipwrres24 at acpi0: FN01, resource for FAN1
acpipwrres25 at acpi0: FN02, resource for FAN2
acpipwrres26 at acpi0: FN03, resource for FAN3
acpipwrres27 at acpi0: FN04, resource for FAN4
acpitz0 at acpi0: critical temperature is 119 degC
acpitz1 at acpi0: critical temperature is 119 degC
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
"IPI0001" at acpi0 not configured
"INT0E0C" at acpi0 not configured
acpibtn0 at acpi0: SLPB
"INT33A1" at acpi0 not configured
acpibtn1 at acpi0: PWRB
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
acpivideo0 at acpi0: GFX0
acpivout0 at acpivideo0: DD1F
ipmi at mainbus0 not configured
cpu0: using Skylake AVX MDS workaround
cpu0: Enhanced SpeedStep 3901 MHz: speeds: 3901, 3900, 3700, 3500, 3200, 3000, 2800, 2600, 2300, 2100, 1900, 1700, 1500, 1200, 1000, 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 v6/7 Host" rev 0x05
ppb0 at pci0 dev 1 function 0 "Intel Core 6G PCIE" rev 0x05: msi
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel I350" rev 0x01: msi, address ac:1f:6b:96:bd:4c
em1 at pci1 dev 0 function 1 "Intel I350" rev 0x01: msi, address ac:1f:6b:96:bd:4d
em2 at pci1 dev 0 function 2 "Intel I350" rev 0x01: msi, address ac:1f:6b:96:bd:4e
em3 at pci1 dev 0 function 3 "Intel I350" rev 0x01: msi, address ac:1f:6b:96:bd:4f
"Intel 100 Series ISH" rev 0x31 at pci0 dev 19 function 0 not configured
xhci0 at pci0 dev 20 function 0 "Intel 100 Series xHCI" rev 0x31: msi, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1
pchtemp0 at pci0 dev 20 function 2 "Intel 100 Series Thermal" rev 0x31
"Intel 100 Series MEI" rev 0x31 at pci0 dev 22 function 0 not configured
ahci0 at pci0 dev 23 function 0 "Intel 100 Series AHCI" rev 0x31: msi, AHCI 1.3.1
ahci0: port 2: 6.0Gb/s
ahci0: port 3: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 2 lun 0: <ATA, WDC WD5000LPLX-0, 01.0> naa.50014ee6b30cf538
sd0: 476940MB, 512 bytes/sector, 976773168 sectors
sd1 at scsibus1 targ 3 lun 0: <ATA, WDC WD5000LPLX-0, 01.0> naa.50014ee6b30182ec
sd1: 476940MB, 512 bytes/sector, 976773168 sectors
ppb1 at pci0 dev 29 function 0 "Intel 100 Series PCIE" rev 0xf1: msi
pci2 at ppb1 bus 2
em4 at pci2 dev 0 function 0 "Intel I210" rev 0x03: msi, address ac:1f:6b:97:4a:bc
ppb2 at pci0 dev 29 function 1 "Intel 100 Series PCIE" rev 0xf1: msi
pci3 at ppb2 bus 3
em5 at pci3 dev 0 function 0 "Intel I210" rev 0x03: msi, address ac:1f:6b:97:4a:bd
ppb3 at pci0 dev 29 function 2 "Intel 100 Series PCIE" rev 0xf1: msi
pci4 at ppb3 bus 4
ppb4 at pci4 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x03
pci5 at ppb4 bus 5
vga1 at pci5 dev 0 function 0 "ASPEED Technology AST2000" rev 0x30
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel C232 LPC" rev 0x31
"Intel 100 Series PMC" rev 0x31 at pci0 dev 31 function 2 not configured
ichiic0 at pci0 dev 31 function 4 "Intel 100 Series SMBus" rev 0x31: apic 2 int 16
iic0 at ichiic0
iic0: addr 0x19 00=00 01=00 02=04 03=00 04=06 05=01 06=1c 07=22 08=00 words 00=00ef 01=000d 02=04a0 03=0000 04=0600 05=0150 06=1c85 07=2221
iic0: addr 0x1b 00=00 01=00 02=04 03=00 04=06 05=01 06=1c 07=22 08=00 words 00=00ef 01=000d 02=04a0 03=0000 04=0600 05=0154 06=1c85 07=2221
iic0: addr 0x4c 04=01 44=01 84=01 c0=03 c1=01 c4=01 c6=01 ca=01 cb=01 cc=01 cd=02 cf=01 d1=02 d4=07 da=0f db=02 dd=01 de=03 words 00=ffff 01=ffff 02=ffff 03=ffff 04=01ff 05=ffff 06=ffff 07=ffff
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT
uhub1 at uhub0 port 9 configuration 1 interface 0 "ATEN International product 0x7000" rev 2.00/0.00 addr 2
uhidev0 at uhub1 port 1 configuration 1 interface 0 "ATEN International product 0x2419" rev 1.10/1.00 addr 3
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub1 port 1 configuration 1 interface 1 "ATEN International product 0x2419" rev 1.10/1.00 addr 3
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
uhidev2 at uhub0 port 11 configuration 1 interface 0 "Dell Dell USB Keyboard" rev 1.10/3.52 addr 4
uhidev2: iclass 3/1
ukbd1 at uhidev2: 8 variable keys, 6 key codes
wskbd1 at ukbd1 mux 1
wskbd1: connecting to wsdisplay0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006>
sd2: 476937MB, 512 bytes/sector, 976767473 sectors
root on sd2a (9e550c811878345c.a) swap on sd2b dump on sd2b
carp52: state transition: BACKUP -> MASTER
carp302: state transition: BACKUP -> MASTER
carp99: state transition: BACKUP -> MASTER
carp56: state transition: BACKUP -> MASTER
carp48: state transition: BACKUP -> MASTER
carp303: state transition: BACKUP -> MASTER
carp22: state transition: BACKUP -> MASTER
carp20: state transition: BACKUP -> MASTER
carp17: state transition: BACKUP -> MASTER
carp17: state transition: MASTER -> BACKUP
carp20: state transition: MASTER -> BACKUP
carp22: state transition: MASTER -> BACKUP
carp302: state transition: MASTER -> BACKUP
carp303: state transition: MASTER -> BACKUP
carp48: state transition: MASTER -> BACKUP
carp120: state transition: BACKUP -> MASTER
carp52: state transition: MASTER -> BACKUP
carp99: state transition: MASTER -> BACKUP
carp56: state transition: MASTER -> BACKUP

Reply | Threaded
Open this post in threaded view
|

Re: Troubleshooting pf congestion

Uwe Werler-4
Without seeing a rule set what should one say?

Am 14. September 2020 15:19:46 GMT+00:00 schrieb Scott Reese <[hidden email]>:

>Greetings:
>
>I am troubleshooting an issue: users complaining about network
>performance. The firewall
>is an OpenBSD 6.7 system with patches applied. I've traced the issue
>and I'm seeing the
>congestion counter incrementing on system. The problems that we're
>seeing fit with what
>I have been able to find about congestion - when the firewall is
>congested it continues
>passing packets that match existing state entries but it will not
>create any new state
>entries until the congestion clears.
>
>I'm having trouble troubleshooting it beyond that point because I have
>not been able to
>find any additional information about what the congestion counter is
>counting. There is
>the information in the pfctl man page: "congestion: network interface
>queue congested",
>but beyond that I can't really find any information about exactly what
>network interface
>queue is congested.
>
>I'm not seeing packets being dropped, either on the switch side or
>firewall side that
>correspond with the congestion counter going up. The average on the
>congestion counter
>stays around 10/s, but what it's really doing is going up by 100-300/s
>for short periods
>and then not moving for longer periods.
>
>If anyone could spare a couple of sentences or a share a link to a page
>detailing what
>state causes the system to consider itself contested, I would
>appreciate it.
>
>Thanks for your time.
>
>-Scott
>
>
>System dmesg:
>
>OpenBSD 6.7 (GENERIC.MP) #6: Thu Sep  3 14:08:18 MDT 2020
>[hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>real mem = 8386699264 (7998MB)
>avail mem = 8119902208 (7743MB)
>mpath0 at root
>scsibus0 at mpath0: 256 targets
>mainbus0 at root
>bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7fb76000 (62 entries)
>bios0: vendor American Megatrends Inc. version "2.2" date 05/23/2018
>bios0: Supermicro X11SSL-F
>acpi0 at bios0: ACPI 5.0
>acpi0: sleep states S0 S4 S5
>acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG HPET LPIT SSDT SSDT
>SSDT DBGP DBG2 SSDT SSDT UEFI SSDT DMAR EINJ ERST BERT HEST
>acpi0: wakeup devices PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4)
>PEGP(S4) RP09(S4) PXSX(S4) RP10(S4) PXSX(S4) RP11(S4) PXSX(S4) RP12(S4)
>PXSX(S4) RP13(S4) PXSX(S4) [...]
>acpitimer0 at acpi0: 3579545 Hz, 24 bits
>acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
>cpu0 at mainbus0: apid 0 (boot processor)
>cpu0: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3901.62 MHz, 06-9e-09
>cpu0:
>FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
>cpu0: 256KB 64b/line 8-way L2 cache
>cpu0: smt 0, core 0, package 0
>mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
>cpu0: apic clock running at 24MHz
>cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE
>cpu1 at mainbus0: apid 2 (application processor)
>cpu1: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
>cpu1:
>FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
>cpu1: 256KB 64b/line 8-way L2 cache
>cpu1: smt 0, core 1, package 0
>cpu2 at mainbus0: apid 4 (application processor)
>cpu2: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
>cpu2:
>FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
>cpu2: 256KB 64b/line 8-way L2 cache
>cpu2: smt 0, core 2, package 0
>cpu3 at mainbus0: apid 6 (application processor)
>cpu3: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
>cpu3:
>FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
>cpu3: 256KB 64b/line 8-way L2 cache
>cpu3: smt 0, core 3, package 0
>cpu4 at mainbus0: apid 1 (application processor)
>cpu4: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
>cpu4:
>FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
>cpu4: 256KB 64b/line 8-way L2 cache
>cpu4: smt 1, core 0, package 0
>cpu5 at mainbus0: apid 3 (application processor)
>cpu5: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.00 MHz, 06-9e-09
>cpu5:
>FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
>cpu5: 256KB 64b/line 8-way L2 cache
>cpu5: smt 1, core 1, package 0
>cpu6 at mainbus0: apid 5 (application processor)
>cpu6: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
>cpu6:
>FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
>cpu6: 256KB 64b/line 8-way L2 cache
>cpu6: smt 1, core 2, package 0
>cpu7 at mainbus0: apid 7 (application processor)
>cpu7: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
>cpu7:
>FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
>cpu7: 256KB 64b/line 8-way L2 cache
>cpu7: smt 1, core 3, package 0
>ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
>acpimcfg0 at acpi0
>acpimcfg0: addr 0xe0000000, bus 0-255
>acpihpet0 at acpi0: 23999999 Hz
>acpiprt0 at acpi0: bus 0 (PCI0)
>acpiprt1 at acpi0: bus 1 (PEG0)
>acpiprt2 at acpi0: bus -1 (PEG1)
>acpiprt3 at acpi0: bus -1 (PEG2)
>acpiprt4 at acpi0: bus 2 (RP09)
>acpiprt5 at acpi0: bus 3 (RP10)
>acpiprt6 at acpi0: bus 4 (RP11)
>acpiprt7 at acpi0: bus 5 (BR51)
>acpiprt8 at acpi0: bus -1 (RP12)
>acpiprt9 at acpi0: bus -1 (RP13)
>acpiprt10 at acpi0: bus -1 (RP01)
>acpiprt11 at acpi0: bus -1 (RP05)
>acpiprt12 at acpi0: bus -1 (RP17)
>acpiprt13 at acpi0: bus -1 (RP18)
>acpiprt14 at acpi0: bus -1 (RP19)
>acpiprt15 at acpi0: bus -1 (RP20)
>acpiprt16 at acpi0: bus -1 (RP14)
>acpiprt17 at acpi0: bus -1 (RP15)
>acpiprt18 at acpi0: bus -1 (RP16)
>acpiec0 at acpi0: not present
>acpicpu0 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
>C1(1000@1 mwait.1), PSS
>acpicpu1 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
>C1(1000@1 mwait.1), PSS
>acpicpu2 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
>C1(1000@1 mwait.1), PSS
>acpicpu3 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
>C1(1000@1 mwait.1), PSS
>acpicpu4 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
>C1(1000@1 mwait.1), PSS
>acpicpu5 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
>C1(1000@1 mwait.1), PSS
>acpicpu6 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
>C1(1000@1 mwait.1), PSS
>acpicpu7 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33),
>C1(1000@1 mwait.1), PSS
>acpipwrres0 at acpi0: PG00, resource for PEG0
>acpipwrres1 at acpi0: PG01, resource for PEG1
>acpipwrres2 at acpi0: PG02, resource for PEG2
>acpipwrres3 at acpi0: WRST
>acpipwrres4 at acpi0: WRST
>acpipwrres5 at acpi0: WRST
>acpipwrres6 at acpi0: WRST
>acpipwrres7 at acpi0: WRST
>acpipwrres8 at acpi0: WRST
>acpipwrres9 at acpi0: WRST
>acpipwrres10 at acpi0: WRST
>acpipwrres11 at acpi0: WRST
>acpipwrres12 at acpi0: WRST
>acpipwrres13 at acpi0: WRST
>acpipwrres14 at acpi0: WRST
>acpipwrres15 at acpi0: WRST
>acpipwrres16 at acpi0: WRST
>acpipwrres17 at acpi0: WRST
>acpipwrres18 at acpi0: WRST
>acpipwrres19 at acpi0: WRST
>acpipwrres20 at acpi0: WRST
>acpipwrres21 at acpi0: WRST
>acpipwrres22 at acpi0: WRST
>acpipwrres23 at acpi0: FN00, resource for FAN0
>acpipwrres24 at acpi0: FN01, resource for FAN1
>acpipwrres25 at acpi0: FN02, resource for FAN2
>acpipwrres26 at acpi0: FN03, resource for FAN3
>acpipwrres27 at acpi0: FN04, resource for FAN4
>acpitz0 at acpi0: critical temperature is 119 degC
>acpitz1 at acpi0: critical temperature is 119 degC
>acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
>acpicmos0 at acpi0
>"IPI0001" at acpi0 not configured
>"INT0E0C" at acpi0 not configured
>acpibtn0 at acpi0: SLPB
>"INT33A1" at acpi0 not configured
>acpibtn1 at acpi0: PWRB
>"PNP0C0B" at acpi0 not configured
>"PNP0C0B" at acpi0 not configured
>"PNP0C0B" at acpi0 not configured
>"PNP0C0B" at acpi0 not configured
>"PNP0C0B" at acpi0 not configured
>acpivideo0 at acpi0: GFX0
>acpivout0 at acpivideo0: DD1F
>ipmi at mainbus0 not configured
>cpu0: using Skylake AVX MDS workaround
>cpu0: Enhanced SpeedStep 3901 MHz: speeds: 3901, 3900, 3700, 3500,
>3200, 3000, 2800, 2600, 2300, 2100, 1900, 1700, 1500, 1200, 1000, 800
>MHz
>pci0 at mainbus0 bus 0
>pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 v6/7 Host" rev 0x05
>ppb0 at pci0 dev 1 function 0 "Intel Core 6G PCIE" rev 0x05: msi
>pci1 at ppb0 bus 1
>em0 at pci1 dev 0 function 0 "Intel I350" rev 0x01: msi, address
>ac:1f:6b:96:bd:4c
>em1 at pci1 dev 0 function 1 "Intel I350" rev 0x01: msi, address
>ac:1f:6b:96:bd:4d
>em2 at pci1 dev 0 function 2 "Intel I350" rev 0x01: msi, address
>ac:1f:6b:96:bd:4e
>em3 at pci1 dev 0 function 3 "Intel I350" rev 0x01: msi, address
>ac:1f:6b:96:bd:4f
>"Intel 100 Series ISH" rev 0x31 at pci0 dev 19 function 0 not
>configured
>xhci0 at pci0 dev 20 function 0 "Intel 100 Series xHCI" rev 0x31: msi,
>xHCI 1.0
>usb0 at xhci0: USB revision 3.0
>uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev
>3.00/1.00 addr 1
>pchtemp0 at pci0 dev 20 function 2 "Intel 100 Series Thermal" rev 0x31
>"Intel 100 Series MEI" rev 0x31 at pci0 dev 22 function 0 not
>configured
>ahci0 at pci0 dev 23 function 0 "Intel 100 Series AHCI" rev 0x31: msi,
>AHCI 1.3.1
>ahci0: port 2: 6.0Gb/s
>ahci0: port 3: 6.0Gb/s
>scsibus1 at ahci0: 32 targets
>sd0 at scsibus1 targ 2 lun 0: <ATA, WDC WD5000LPLX-0, 01.0>
>naa.50014ee6b30cf538
>sd0: 476940MB, 512 bytes/sector, 976773168 sectors
>sd1 at scsibus1 targ 3 lun 0: <ATA, WDC WD5000LPLX-0, 01.0>
>naa.50014ee6b30182ec
>sd1: 476940MB, 512 bytes/sector, 976773168 sectors
>ppb1 at pci0 dev 29 function 0 "Intel 100 Series PCIE" rev 0xf1: msi
>pci2 at ppb1 bus 2
>em4 at pci2 dev 0 function 0 "Intel I210" rev 0x03: msi, address
>ac:1f:6b:97:4a:bc
>ppb2 at pci0 dev 29 function 1 "Intel 100 Series PCIE" rev 0xf1: msi
>pci3 at ppb2 bus 3
>em5 at pci3 dev 0 function 0 "Intel I210" rev 0x03: msi, address
>ac:1f:6b:97:4a:bd
>ppb3 at pci0 dev 29 function 2 "Intel 100 Series PCIE" rev 0xf1: msi
>pci4 at ppb3 bus 4
>ppb4 at pci4 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x03
>pci5 at ppb4 bus 5
>vga1 at pci5 dev 0 function 0 "ASPEED Technology AST2000" rev 0x30
>wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
>wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
>pcib0 at pci0 dev 31 function 0 "Intel C232 LPC" rev 0x31
>"Intel 100 Series PMC" rev 0x31 at pci0 dev 31 function 2 not
>configured
>ichiic0 at pci0 dev 31 function 4 "Intel 100 Series SMBus" rev 0x31:
>apic 2 int 16
>iic0 at ichiic0
>iic0: addr 0x19 00=00 01=00 02=04 03=00 04=06 05=01 06=1c 07=22 08=00
>words 00=00ef 01=000d 02=04a0 03=0000 04=0600 05=0150 06=1c85 07=2221
>iic0: addr 0x1b 00=00 01=00 02=04 03=00 04=06 05=01 06=1c 07=22 08=00
>words 00=00ef 01=000d 02=04a0 03=0000 04=0600 05=0154 06=1c85 07=2221
>iic0: addr 0x4c 04=01 44=01 84=01 c0=03 c1=01 c4=01 c6=01 ca=01 cb=01
>cc=01 cd=02 cf=01 d1=02 d4=07 da=0f db=02 dd=01 de=03 words 00=ffff
>01=ffff 02=ffff 03=ffff 04=01ff 05=ffff 06=ffff 07=ffff
>isa0 at pcib0
>isadma0 at isa0
>com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
>com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
>pckbc0 at isa0 port 0x60/5 irq 1 irq 12
>pcppi0 at isa0 port 0x61
>spkr0 at pcppi0
>vmm0 at mainbus0: VMX/EPT
>uhub1 at uhub0 port 9 configuration 1 interface 0 "ATEN International
>product 0x7000" rev 2.00/0.00 addr 2
>uhidev0 at uhub1 port 1 configuration 1 interface 0 "ATEN International
>product 0x2419" rev 1.10/1.00 addr 3
>uhidev0: iclass 3/1
>ukbd0 at uhidev0: 8 variable keys, 6 key codes
>wskbd0 at ukbd0: console keyboard, using wsdisplay0
>uhidev1 at uhub1 port 1 configuration 1 interface 1 "ATEN International
>product 0x2419" rev 1.10/1.00 addr 3
>uhidev1: iclass 3/1
>ums0 at uhidev1: 3 buttons, Z dir
>wsmouse0 at ums0 mux 0
>uhidev2 at uhub0 port 11 configuration 1 interface 0 "Dell Dell USB
>Keyboard" rev 1.10/3.52 addr 4
>uhidev2: iclass 3/1
>ukbd1 at uhidev2: 8 variable keys, 6 key codes
>wskbd1 at ukbd1 mux 1
>wskbd1: connecting to wsdisplay0
>vscsi0 at root
>scsibus2 at vscsi0: 256 targets
>softraid0 at root
>scsibus3 at softraid0: 256 targets
>sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006>
>sd2: 476937MB, 512 bytes/sector, 976767473 sectors
>root on sd2a (9e550c811878345c.a) swap on sd2b dump on sd2b
>carp52: state transition: BACKUP -> MASTER
>carp302: state transition: BACKUP -> MASTER
>carp99: state transition: BACKUP -> MASTER
>carp56: state transition: BACKUP -> MASTER
>carp48: state transition: BACKUP -> MASTER
>carp303: state transition: BACKUP -> MASTER
>carp22: state transition: BACKUP -> MASTER
>carp20: state transition: BACKUP -> MASTER
>carp17: state transition: BACKUP -> MASTER
>carp17: state transition: MASTER -> BACKUP
>carp20: state transition: MASTER -> BACKUP
>carp22: state transition: MASTER -> BACKUP
>carp302: state transition: MASTER -> BACKUP
>carp303: state transition: MASTER -> BACKUP
>carp48: state transition: MASTER -> BACKUP
>carp120: state transition: BACKUP -> MASTER
>carp52: state transition: MASTER -> BACKUP
>carp99: state transition: MASTER -> BACKUP
>carp56: state transition: MASTER -> BACKUP

--
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.
Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: Troubleshooting pf congestion

Scott Reese-2
Greetings:

----- Original Message -----
> From: "Uwe Werler" <[hidden email]>
> To: "misc" <[hidden email]>, "Scott Reese" <[hidden email]>, "misc" <[hidden email]>
> Sent: Monday, September 14, 2020 12:47:31 PM
> Subject: [EXTERNAL]  Re: Troubleshooting pf congestion

> Without seeing a rule set what should one say?
>

>>
>>If anyone could spare a couple of sentences or a share a link to a page
>>detailing what
>>state causes the system to consider itself contested, I would
>>appreciate it.

Thanks for your reply. The question that I can't find a good answer for is,
"What is pf congestion?". I would like to solve the actual problem myself, I'm just looking
for some information about what it means for pf to be congested.

-Scott

Reply | Threaded
Open this post in threaded view
|

Re: Troubleshooting pf congestion

Otto Moerbeek
In reply to this post by Scott Reese-2
On Mon, Sep 14, 2020 at 11:19:46AM -0400, Scott Reese wrote:

> Greetings:
>
> I am troubleshooting an issue: users complaining about network performance. The firewall
> is an OpenBSD 6.7 system with patches applied. I've traced the issue and I'm seeing the
> congestion counter incrementing on system. The problems that we're seeing fit with what
> I have been able to find about congestion - when the firewall is congested it continues
> passing packets that match existing state entries but it will not create any new state
> entries until the congestion clears.
>
> I'm having trouble troubleshooting it beyond that point because I have not been able to
> find any additional information about what the congestion counter is counting. There is
> the information in the pfctl man page: "congestion: network interface queue congested",
> but beyond that I can't really find any information about exactly what network interface
> queue is congested.
>
> I'm not seeing packets being dropped, either on the switch side or firewall side that
> correspond with the congestion counter going up. The average on the congestion counter
> stays around 10/s, but what it's really doing is going up by 100-300/s for short periods
> and then not moving for longer periods.
>
> If anyone could spare a couple of sentences or a share a link to a page detailing what
> state causes the system to consider itself contested, I would appreciate it.
>
> Thanks for your time.
>
> -Scott

openbsd-archive.7691.n7.nabble.com/PF-congestion-question-td156490.html

Description and potential remedy are stil true, afaik.

        -Otto

>
>
> System dmesg:
>
> OpenBSD 6.7 (GENERIC.MP) #6: Thu Sep  3 14:08:18 MDT 2020
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 8386699264 (7998MB)
> avail mem = 8119902208 (7743MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7fb76000 (62 entries)
> bios0: vendor American Megatrends Inc. version "2.2" date 05/23/2018
> bios0: Supermicro X11SSL-F
> acpi0 at bios0: ACPI 5.0
> acpi0: sleep states S0 S4 S5
> acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG HPET LPIT SSDT SSDT SSDT DBGP DBG2 SSDT SSDT UEFI SSDT DMAR EINJ ERST BERT HEST
> acpi0: wakeup devices PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) PEGP(S4) RP09(S4) PXSX(S4) RP10(S4) PXSX(S4) RP11(S4) PXSX(S4) RP12(S4) PXSX(S4) RP13(S4) PXSX(S4) [...]
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3901.62 MHz, 06-9e-09
> cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 24MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
> cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 0, core 1, package 0
> cpu2 at mainbus0: apid 4 (application processor)
> cpu2: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
> cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> cpu2: 256KB 64b/line 8-way L2 cache
> cpu2: smt 0, core 2, package 0
> cpu3 at mainbus0: apid 6 (application processor)
> cpu3: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
> cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> cpu3: 256KB 64b/line 8-way L2 cache
> cpu3: smt 0, core 3, package 0
> cpu4 at mainbus0: apid 1 (application processor)
> cpu4: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
> cpu4: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> cpu4: 256KB 64b/line 8-way L2 cache
> cpu4: smt 1, core 0, package 0
> cpu5 at mainbus0: apid 3 (application processor)
> cpu5: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.00 MHz, 06-9e-09
> cpu5: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> cpu5: 256KB 64b/line 8-way L2 cache
> cpu5: smt 1, core 1, package 0
> cpu6 at mainbus0: apid 5 (application processor)
> cpu6: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
> cpu6: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> cpu6: 256KB 64b/line 8-way L2 cache
> cpu6: smt 1, core 2, package 0
> cpu7 at mainbus0: apid 7 (application processor)
> cpu7: Intel(R) Xeon(R) CPU E3-1280 v6 @ 3.90GHz, 3900.01 MHz, 06-9e-09
> cpu7: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
> cpu7: 256KB 64b/line 8-way L2 cache
> cpu7: smt 1, core 3, package 0
> ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
> acpimcfg0 at acpi0
> acpimcfg0: addr 0xe0000000, bus 0-255
> acpihpet0 at acpi0: 23999999 Hz
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 1 (PEG0)
> acpiprt2 at acpi0: bus -1 (PEG1)
> acpiprt3 at acpi0: bus -1 (PEG2)
> acpiprt4 at acpi0: bus 2 (RP09)
> acpiprt5 at acpi0: bus 3 (RP10)
> acpiprt6 at acpi0: bus 4 (RP11)
> acpiprt7 at acpi0: bus 5 (BR51)
> acpiprt8 at acpi0: bus -1 (RP12)
> acpiprt9 at acpi0: bus -1 (RP13)
> acpiprt10 at acpi0: bus -1 (RP01)
> acpiprt11 at acpi0: bus -1 (RP05)
> acpiprt12 at acpi0: bus -1 (RP17)
> acpiprt13 at acpi0: bus -1 (RP18)
> acpiprt14 at acpi0: bus -1 (RP19)
> acpiprt15 at acpi0: bus -1 (RP20)
> acpiprt16 at acpi0: bus -1 (RP14)
> acpiprt17 at acpi0: bus -1 (RP15)
> acpiprt18 at acpi0: bus -1 (RP16)
> acpiec0 at acpi0: not present
> acpicpu0 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
> acpicpu1 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
> acpicpu2 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
> acpicpu3 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
> acpicpu4 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
> acpicpu5 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
> acpicpu6 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
> acpicpu7 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
> acpipwrres0 at acpi0: PG00, resource for PEG0
> acpipwrres1 at acpi0: PG01, resource for PEG1
> acpipwrres2 at acpi0: PG02, resource for PEG2
> acpipwrres3 at acpi0: WRST
> acpipwrres4 at acpi0: WRST
> acpipwrres5 at acpi0: WRST
> acpipwrres6 at acpi0: WRST
> acpipwrres7 at acpi0: WRST
> acpipwrres8 at acpi0: WRST
> acpipwrres9 at acpi0: WRST
> acpipwrres10 at acpi0: WRST
> acpipwrres11 at acpi0: WRST
> acpipwrres12 at acpi0: WRST
> acpipwrres13 at acpi0: WRST
> acpipwrres14 at acpi0: WRST
> acpipwrres15 at acpi0: WRST
> acpipwrres16 at acpi0: WRST
> acpipwrres17 at acpi0: WRST
> acpipwrres18 at acpi0: WRST
> acpipwrres19 at acpi0: WRST
> acpipwrres20 at acpi0: WRST
> acpipwrres21 at acpi0: WRST
> acpipwrres22 at acpi0: WRST
> acpipwrres23 at acpi0: FN00, resource for FAN0
> acpipwrres24 at acpi0: FN01, resource for FAN1
> acpipwrres25 at acpi0: FN02, resource for FAN2
> acpipwrres26 at acpi0: FN03, resource for FAN3
> acpipwrres27 at acpi0: FN04, resource for FAN4
> acpitz0 at acpi0: critical temperature is 119 degC
> acpitz1 at acpi0: critical temperature is 119 degC
> acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
> acpicmos0 at acpi0
> "IPI0001" at acpi0 not configured
> "INT0E0C" at acpi0 not configured
> acpibtn0 at acpi0: SLPB
> "INT33A1" at acpi0 not configured
> acpibtn1 at acpi0: PWRB
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> acpivideo0 at acpi0: GFX0
> acpivout0 at acpivideo0: DD1F
> ipmi at mainbus0 not configured
> cpu0: using Skylake AVX MDS workaround
> cpu0: Enhanced SpeedStep 3901 MHz: speeds: 3901, 3900, 3700, 3500, 3200, 3000, 2800, 2600, 2300, 2100, 1900, 1700, 1500, 1200, 1000, 800 MHz
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 v6/7 Host" rev 0x05
> ppb0 at pci0 dev 1 function 0 "Intel Core 6G PCIE" rev 0x05: msi
> pci1 at ppb0 bus 1
> em0 at pci1 dev 0 function 0 "Intel I350" rev 0x01: msi, address ac:1f:6b:96:bd:4c
> em1 at pci1 dev 0 function 1 "Intel I350" rev 0x01: msi, address ac:1f:6b:96:bd:4d
> em2 at pci1 dev 0 function 2 "Intel I350" rev 0x01: msi, address ac:1f:6b:96:bd:4e
> em3 at pci1 dev 0 function 3 "Intel I350" rev 0x01: msi, address ac:1f:6b:96:bd:4f
> "Intel 100 Series ISH" rev 0x31 at pci0 dev 19 function 0 not configured
> xhci0 at pci0 dev 20 function 0 "Intel 100 Series xHCI" rev 0x31: msi, xHCI 1.0
> usb0 at xhci0: USB revision 3.0
> uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1
> pchtemp0 at pci0 dev 20 function 2 "Intel 100 Series Thermal" rev 0x31
> "Intel 100 Series MEI" rev 0x31 at pci0 dev 22 function 0 not configured
> ahci0 at pci0 dev 23 function 0 "Intel 100 Series AHCI" rev 0x31: msi, AHCI 1.3.1
> ahci0: port 2: 6.0Gb/s
> ahci0: port 3: 6.0Gb/s
> scsibus1 at ahci0: 32 targets
> sd0 at scsibus1 targ 2 lun 0: <ATA, WDC WD5000LPLX-0, 01.0> naa.50014ee6b30cf538
> sd0: 476940MB, 512 bytes/sector, 976773168 sectors
> sd1 at scsibus1 targ 3 lun 0: <ATA, WDC WD5000LPLX-0, 01.0> naa.50014ee6b30182ec
> sd1: 476940MB, 512 bytes/sector, 976773168 sectors
> ppb1 at pci0 dev 29 function 0 "Intel 100 Series PCIE" rev 0xf1: msi
> pci2 at ppb1 bus 2
> em4 at pci2 dev 0 function 0 "Intel I210" rev 0x03: msi, address ac:1f:6b:97:4a:bc
> ppb2 at pci0 dev 29 function 1 "Intel 100 Series PCIE" rev 0xf1: msi
> pci3 at ppb2 bus 3
> em5 at pci3 dev 0 function 0 "Intel I210" rev 0x03: msi, address ac:1f:6b:97:4a:bd
> ppb3 at pci0 dev 29 function 2 "Intel 100 Series PCIE" rev 0xf1: msi
> pci4 at ppb3 bus 4
> ppb4 at pci4 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x03
> pci5 at ppb4 bus 5
> vga1 at pci5 dev 0 function 0 "ASPEED Technology AST2000" rev 0x30
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> pcib0 at pci0 dev 31 function 0 "Intel C232 LPC" rev 0x31
> "Intel 100 Series PMC" rev 0x31 at pci0 dev 31 function 2 not configured
> ichiic0 at pci0 dev 31 function 4 "Intel 100 Series SMBus" rev 0x31: apic 2 int 16
> iic0 at ichiic0
> iic0: addr 0x19 00=00 01=00 02=04 03=00 04=06 05=01 06=1c 07=22 08=00 words 00=00ef 01=000d 02=04a0 03=0000 04=0600 05=0150 06=1c85 07=2221
> iic0: addr 0x1b 00=00 01=00 02=04 03=00 04=06 05=01 06=1c 07=22 08=00 words 00=00ef 01=000d 02=04a0 03=0000 04=0600 05=0154 06=1c85 07=2221
> iic0: addr 0x4c 04=01 44=01 84=01 c0=03 c1=01 c4=01 c6=01 ca=01 cb=01 cc=01 cd=02 cf=01 d1=02 d4=07 da=0f db=02 dd=01 de=03 words 00=ffff 01=ffff 02=ffff 03=ffff 04=01ff 05=ffff 06=ffff 07=ffff
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> vmm0 at mainbus0: VMX/EPT
> uhub1 at uhub0 port 9 configuration 1 interface 0 "ATEN International product 0x7000" rev 2.00/0.00 addr 2
> uhidev0 at uhub1 port 1 configuration 1 interface 0 "ATEN International product 0x2419" rev 1.10/1.00 addr 3
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 variable keys, 6 key codes
> wskbd0 at ukbd0: console keyboard, using wsdisplay0
> uhidev1 at uhub1 port 1 configuration 1 interface 1 "ATEN International product 0x2419" rev 1.10/1.00 addr 3
> uhidev1: iclass 3/1
> ums0 at uhidev1: 3 buttons, Z dir
> wsmouse0 at ums0 mux 0
> uhidev2 at uhub0 port 11 configuration 1 interface 0 "Dell Dell USB Keyboard" rev 1.10/3.52 addr 4
> uhidev2: iclass 3/1
> ukbd1 at uhidev2: 8 variable keys, 6 key codes
> wskbd1 at ukbd1 mux 1
> wskbd1: connecting to wsdisplay0
> vscsi0 at root
> scsibus2 at vscsi0: 256 targets
> softraid0 at root
> scsibus3 at softraid0: 256 targets
> sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006>
> sd2: 476937MB, 512 bytes/sector, 976767473 sectors
> root on sd2a (9e550c811878345c.a) swap on sd2b dump on sd2b
> carp52: state transition: BACKUP -> MASTER
> carp302: state transition: BACKUP -> MASTER
> carp99: state transition: BACKUP -> MASTER
> carp56: state transition: BACKUP -> MASTER
> carp48: state transition: BACKUP -> MASTER
> carp303: state transition: BACKUP -> MASTER
> carp22: state transition: BACKUP -> MASTER
> carp20: state transition: BACKUP -> MASTER
> carp17: state transition: BACKUP -> MASTER
> carp17: state transition: MASTER -> BACKUP
> carp20: state transition: MASTER -> BACKUP
> carp22: state transition: MASTER -> BACKUP
> carp302: state transition: MASTER -> BACKUP
> carp303: state transition: MASTER -> BACKUP
> carp48: state transition: MASTER -> BACKUP
> carp120: state transition: BACKUP -> MASTER
> carp52: state transition: MASTER -> BACKUP
> carp99: state transition: MASTER -> BACKUP
> carp56: state transition: MASTER -> BACKUP
>

Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: Troubleshooting pf congestion

Stuart Henderson
In reply to this post by Scott Reese-2
On 2020-09-14, Scott Reese <[hidden email]> wrote:

> Greetings:
>
> ----- Original Message -----
>> From: "Uwe Werler" <[hidden email]>
>> To: "misc" <[hidden email]>, "Scott Reese" <[hidden email]>, "misc" <[hidden email]>
>> Sent: Monday, September 14, 2020 12:47:31 PM
>> Subject: [EXTERNAL]  Re: Troubleshooting pf congestion
>
>> Without seeing a rule set what should one say?
>>
>
>>>
>>>If anyone could spare a couple of sentences or a share a link to a page
>>>detailing what
>>>state causes the system to consider itself contested, I would
>>>appreciate it.
>
> Thanks for your reply. The question that I can't find a good answer for is,
> "What is pf congestion?". I would like to solve the actual problem myself, I'm just looking
> for some information about what it means for pf to be congested.

When enqueueing packets to an interface fails (queue is full), a
global congestion marker variable in the kernel is set to the current
timestamp.

When PF tests an inbound packet against rules (i.e. when it has a packet
that doesn't match an existing state) it checks if that congestion timestamp
is recent. If it is, the packet is dropped and the PF stats congestion
counter is incremented.

Look around if_congested/if_congestion in /sys/net and the mq_ functions
in /sys/kern/uipc_mbuf.c - the functions described in mq_init(9) as "If the
queue is full then XX will be dropped" trigger congestion.

You might get some suggestions if you post a description of your
configuration (at least which interface types - physical or virtual -
are in use, what they're connected to, what if any VPNs it's running,
and it may help to see the ruleset).

Output from these might help too:

netstat -m
systat mbuf | cat
vmstat -i
vmstat -m


Reply | Threaded
Open this post in threaded view
|

Re: [EXTERNAL] Re: Troubleshooting pf congestion

Scott Reese-2

> On 2020-09-14, Scott Reese <[hidden email]> wrote:
>> Greetings:
>>
>> ----- Original Message -----
>>> From: "Uwe Werler" <[hidden email]>
>>> To: "misc" <[hidden email]>, "Scott Reese" <[hidden email]>, "misc"
>>> <[hidden email]>
>>> Sent: Monday, September 14, 2020 12:47:31 PM
>>> Subject: [EXTERNAL]  Re: Troubleshooting pf congestion
>>
>>> Without seeing a rule set what should one say?
>>>
>>
>>>>
>>>>If anyone could spare a couple of sentences or a share a link to a page
>>>>detailing what
>>>>state causes the system to consider itself contested, I would
>>>>appreciate it.
>>
>> Thanks for your reply. The question that I can't find a good answer for is,
>> "What is pf congestion?". I would like to solve the actual problem myself, I'm
>> just looking
>> for some information about what it means for pf to be congested.
>
> When enqueueing packets to an interface fails (queue is full), a
> global congestion marker variable in the kernel is set to the current
> timestamp.
>
> When PF tests an inbound packet against rules (i.e. when it has a packet
> that doesn't match an existing state) it checks if that congestion timestamp
> is recent. If it is, the packet is dropped and the PF stats congestion
> counter is incremented.
>
> Look around if_congested/if_congestion in /sys/net and the mq_ functions
> in /sys/kern/uipc_mbuf.c - the functions described in mq_init(9) as "If the
> queue is full then XX will be dropped" trigger congestion.
>
> You might get some suggestions if you post a description of your
> configuration (at least which interface types - physical or virtual -
> are in use, what they're connected to, what if any VPNs it's running,
> and it may help to see the ruleset).
>
> Output from these might help too:
>
> netstat -m
> systat mbuf | cat
> vmstat -i
> vmstat -m

Greetings Stuart:

Thank you for your reply. It was very helpful and pointed me in the right
direction. The 1000+ windows workstations behind that firewall had been
converted from 7 to 10. Most aren't allowed to access the internet, and the
new OS is much more aggressive about trying to phone home. All of those
dropped packets had to traverse all of the rules before being dropped, and
that was the root cause of the issue. It didn't look like too much traffic
because it was just SYN packets, but it was a lot of SYN packets.

Again, thanks for your help.

-Scott