PF Congestion

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

PF Congestion

Eric Camirand
Hello,

I'm testing a new dns server (nsd) and i'm getting 'congestion' in the pf information output on very low level of traffic ( ~100 pps).

Can someone help me understand why pf congestion is kicking in ?

Thanks

Eric Camirand


===

Dns traffic is coming in on em0 trough 2 vlan using carp.

===

Status: Enabled for 0 days 00:46:04             Debug: info

Hostid:   0xc64f5e35
Checksum: 0x5b36311462da2ab76b362f191125df49

Interface Stats for egress            IPv4             IPv6
  Bytes In                        14878243          6533902
  Bytes Out                       21341439          7144887
  Packets In
    Passed                          198909            76866
    Blocked                            839                8
  Packets Out
    Passed                          143777            44494
    Blocked                              0               38

State Table                          Total             Rate
  current entries                     2106              
  half-open tcp                          0              
  searches                          464928          168.2/s
  inserts                           180405           65.3/s
  removals                          178299           64.5/s
Source Tracking Table
  current entries                        0              
  searches                               0            0.0/s
  inserts                                0            0.0/s
  removals                               0            0.0/s
Counters
  match                             181265           65.6/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                              3            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                            23            0.0/s
  ip-option                             18            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                         0            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
  translate                              0            0.0/s
  no-route                               0            0.0/s
Limit Counters
  max states per rule                    0            0.0/s
  max-src-states                         0            0.0/s
  max-src-nodes                          0            0.0/s
  max-src-conn                           0            0.0/s
  max-src-conn-rate                      0            0.0/s
  overload table insertion               0            0.0/s
  overload flush states                  0            0.0/s
  synfloods detected                     0            0.0/s
  syncookies sent                        0            0.0/s
  syncookies validated                   0            0.0/s
Adaptive Syncookies Watermarks
  start                              25000 states
  end                                12500 states

===

@0 block drop log all
  [ Evaluations: 170657    Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 64835 State Creations: 0     ]
@1 pass quick proto carp all
  [ Evaluations: 170657    Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 64835 State Creations: 0     ]
@2 block drop in quick on egress from <BAD:0> to any
  [ Evaluations: 170657    Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 64835 State Creations: 0     ]
@3 pass in quick on egress proto tcp from any to <DNS:8> port = 53 flags S/SA
  [ Evaluations: 170395    Packets: 2521      Bytes: 201134      States: 7     ]
  [ Inserted: uid 0 pid 64835 State Creations: 252   ]
@4 pass in quick on egress proto tcp from <MGMT:1> to any port = 22 flags S/SA
  [ Evaluations: 639       Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 64835 State Creations: 0     ]
@5 pass in quick on egress proto udp from any to <DNS:8> port = 53
  [ Evaluations: 170143    Packets: 338308    Bytes: 38583005    States: 2877  ]
  [ Inserted: uid 0 pid 64835 State Creations: 168655]
@6 pass in quick on egress inet proto icmp all
  [ Evaluations: 1488      Packets: 42        Bytes: 2728        States: 0     ]
  [ Inserted: uid 0 pid 64835 State Creations: 12    ]
@7 pass in quick on egress inet6 proto ipv6-icmp all
  [ Evaluations: 1476      Packets: 2781      Bytes: 197948      States: 3     ]
  [ Inserted: uid 0 pid 64835 State Creations: 736   ]
@8 block drop in log quick on egress all
  [ Evaluations: 740       Packets: 740       Bytes: 60473       States: 0     ]
  [ Inserted: uid 0 pid 64835 State Creations: 0     ]
@9 pass out quick on egress all flags S/SA
  [ Evaluations: 262       Packets: 780       Bytes: 55120       States: 2     ]
  [ Inserted: uid 0 pid 64835 State Creations: 262   ]
@10 block drop out quick on egress all
  [ Evaluations: 0         Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 64835 State Creations: 0     ]

===

OpenBSD 6.5-current (GENERIC.MP) #184: Wed Aug  7 21:37:16 MDT 2019
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 34156728320 (32574MB)
avail mem = 33111396352 (31577MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7fb76000 (68 entries)
bios0: vendor American Megatrends Inc. version "2.2" date 05/23/2018
bios0: Supermicro X11SSH-F
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG HPET LPIT SSDT SSDT SSDT DBGP DBG2 SSDT PRAD SSDT UEFI SSDT DMAR EINJ ERST BERT HEST
acpi0: wakeup devices PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) PEGP(S4) RP09(S4) PXSX(S4) RP10(S4) PXSX(S4) RP11(S4) PXSX(S4) RP12(S4) PXSX(S4) RP13(S4) PXSX(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E3-1240 v6 @ 3.70GHz, 3701.77 MHz, 06-9e-09
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
tsc_timecounter_init: TSC skew=0 observed drift=0
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 24MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
TSC skew=4
cpu1: Intel(R) Xeon(R) CPU E3-1240 v6 @ 3.70GHz, 3700.00 MHz, 06-9e-09
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
tsc_timecounter_init: TSC skew=4 observed drift=0
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
TSC skew=-4
cpu2: Intel(R) Xeon(R) CPU E3-1240 v6 @ 3.70GHz, 3700.00 MHz, 06-9e-09
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
tsc_timecounter_init: TSC skew=-4 observed drift=0
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
TSC skew=-4
cpu3: Intel(R) Xeon(R) CPU E3-1240 v6 @ 3.70GHz, 3700.01 MHz, 06-9e-09
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,MD_CLEAR,TSXFA,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
tsc_timecounter_init: TSC skew=-4 observed drift=0
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 23999999 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEG0)
acpiprt2 at acpi0: bus -1 (PEG1)
acpiprt3 at acpi0: bus -1 (PEG2)
acpiprt4 at acpi0: bus -1 (RP09)
acpiprt5 at acpi0: bus -1 (RP13)
acpiprt6 at acpi0: bus 2 (RP01)
acpiprt7 at acpi0: bus 3 (RP02)
acpiprt8 at acpi0: bus -1 (RP03)
acpiprt9 at acpi0: bus -1 (RP04)
acpiprt10 at acpi0: bus -1 (RP05)
acpiprt11 at acpi0: bus -1 (RP06)
acpiprt12 at acpi0: bus 4 (RP07)
acpiprt13 at acpi0: bus 5 (BR53)
acpiprt14 at acpi0: bus -1 (RP08)
acpiprt15 at acpi0: bus -1 (RP17)
acpiprt16 at acpi0: bus -1 (RP18)
acpiprt17 at acpi0: bus -1 (RP19)
acpiprt18 at acpi0: bus -1 (RP20)
acpiprt19 at acpi0: bus -1 (RP14)
acpiprt20 at acpi0: bus -1 (RP15)
acpiprt21 at acpi0: bus -1 (RP16)
acpiec0 at acpi0: not present
acpicpu0 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu2 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu3 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpipwrres0 at acpi0: PG00, resource for PEG0
acpipwrres1 at acpi0: PG01, resource for PEG1
acpipwrres2 at acpi0: PG02, resource for PEG2
acpipwrres3 at acpi0: WRST
acpipwrres4 at acpi0: WRST
acpipwrres5 at acpi0: WRST
acpipwrres6 at acpi0: WRST
acpipwrres7 at acpi0: WRST
acpipwrres8 at acpi0: WRST
acpipwrres9 at acpi0: WRST
acpipwrres10 at acpi0: WRST
acpipwrres11 at acpi0: WRST
acpipwrres12 at acpi0: WRST
acpipwrres13 at acpi0: WRST
acpipwrres14 at acpi0: WRST
acpipwrres15 at acpi0: WRST
acpipwrres16 at acpi0: WRST
acpipwrres17 at acpi0: WRST
acpipwrres18 at acpi0: WRST
acpipwrres19 at acpi0: WRST
acpipwrres20 at acpi0: WRST
acpipwrres21 at acpi0: WRST
acpipwrres22 at acpi0: WRST
acpipwrres23 at acpi0: FN00, resource for FAN0
acpipwrres24 at acpi0: FN01, resource for FAN1
acpipwrres25 at acpi0: FN02, resource for FAN2
acpipwrres26 at acpi0: FN03, resource for FAN3
acpipwrres27 at acpi0: FN04, resource for FAN4
acpitz0 at acpi0: critical temperature is 119 degC
acpitz1 at acpi0: critical temperature is 119 degC
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
"IPI0001" at acpi0 not configured
"INT0E0C" at acpi0 not configured
acpibtn0 at acpi0: SLPB
"INT33A1" at acpi0 not configured
acpibtn1 at acpi0: PWRB
"ACPI000D" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
acpivideo0 at acpi0: GFX0
acpivout0 at acpivideo0: DD1F
ipmi at mainbus0 not configured
cpu0: using VERW MDS workaround (except on vmm entry)
cpu0: Enhanced SpeedStep 3701 MHz: speeds: 3701, 3700, 3500, 3300, 3100, 2900, 2700, 2500, 2200, 2000, 1800, 1600, 1400, 1200, 1000, 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 v6/7 Host" rev 0x05
ppb0 at pci0 dev 1 function 0 "Intel Core 6G PCIE" rev 0x05: msi
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel 82571EB" rev 0x06: apic 2 int 16, address 00:15:17:a6:52:fa
em1 at pci1 dev 0 function 1 "Intel 82571EB" rev 0x06: apic 2 int 17, address 00:15:17:a6:52:fb
"Intel 100 Series ISH" rev 0x31 at pci0 dev 19 function 0 not configured
xhci0 at pci0 dev 20 function 0 "Intel 100 Series xHCI" rev 0x31: msi, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1
pchtemp0 at pci0 dev 20 function 2 "Intel 100 Series Thermal" rev 0x31
"Intel 100 Series MEI" rev 0x31 at pci0 dev 22 function 0 not configured
"Intel 100 Series MEI" rev 0x31 at pci0 dev 22 function 1 not configured
ahci0 at pci0 dev 23 function 0 "Intel 100 Series AHCI" rev 0x31: msi, AHCI 1.3.1
ahci0: port 4: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 4 lun 0: <ATA, INTEL SSDSC2BW24, 400i> SCSI3 0/direct fixed naa.5001517bb2af180f
sd0: 228936MB, 512 bytes/sector, 468862128 sectors, thin
ppb1 at pci0 dev 28 function 0 "Intel 100 Series PCIE" rev 0xf1: msi
pci2 at ppb1 bus 2
em2 at pci2 dev 0 function 0 "Intel I210" rev 0x03: msi, address ac:1f:6b:77:ed:c6
ppb2 at pci0 dev 28 function 1 "Intel 100 Series PCIE" rev 0xf1: msi
pci3 at ppb2 bus 3
em3 at pci3 dev 0 function 0 "Intel I210" rev 0x03: msi, address ac:1f:6b:77:ed:c7
ppb3 at pci0 dev 28 function 6 "Intel 100 Series PCIE" rev 0xf1: msi
pci4 at ppb3 bus 4
ppb4 at pci4 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x03
pci5 at ppb4 bus 5
vga1 at pci5 dev 0 function 0 "ASPEED Technology AST2000" rev 0x30
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel C236 LPC" rev 0x31
"Intel 100 Series PMC" rev 0x31 at pci0 dev 31 function 2 not configured
ichiic0 at pci0 dev 31 function 4 "Intel 100 Series SMBus" rev 0x31: apic 2 int 16
iic0 at ichiic0
sdtemp0 at iic0 addr 0x18: stts2004
sdtemp1 at iic0 addr 0x19: stts2004
sdtemp2 at iic0 addr 0x1a: stts2004
sdtemp3 at iic0 addr 0x1b: stts2004
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
uhub1 at uhub0 port 14 configuration 1 interface 0 "ATEN International product 0x7000" rev 2.00/0.00 addr 2
uhidev0 at uhub1 port 1 configuration 1 interface 0 "ATEN International product 0x2419" rev 1.10/1.00 addr 3
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub1 port 1 configuration 1 interface 1 "ATEN International product 0x2419" rev 1.10/1.00 addr 3
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (4594e024a043b9e7.a) swap on sd0b dump on sd0b

Reply | Threaded
Open this post in threaded view
|

Re: PF Congestion

Stuart Henderson
On 2019/08/08 20:40, Eric Camirand wrote:
> Hello,
>
> I'm testing a new dns server (nsd) and i'm getting 'congestion' in the pf information output on very low level of traffic ( ~100 pps).
>
> Can someone help me understand why pf congestion is kicking in ?

This happens when the network stack detects that the system is too
busy, it's not from some congestion internal to PF.

I suggest reasking on [hidden email].
Reply | Threaded
Open this post in threaded view
|

Re: PF Congestion

Matt Rowley
In reply to this post by Eric Camirand
> I'm testing a new dns server (nsd) and i'm getting 'congestion' in
> the pf information output on very low level of traffic ( ~100 pps).
>
> Can someone help me understand why pf congestion is kicking in ?

Are you really really sure about the low traffic levels?  In my
experience, pf congestion has only ever shown up when there was very
high levels of packets per second, or due to less than awesome NICs
being involved.

--Matt
Reply | Threaded
Open this post in threaded view
|

Re: PF Congestion

Eric Camirand
The switch port monitoring and collectd tell me there isnt much traffic there. I also don't have this issue with my older servers using em interfaces. I'm going to do more test on another new server and ask tech@openbsd if the issue persist.

Eric

Reply | Threaded
Open this post in threaded view
|

Re: PF Congestion

Stuart Henderson
On 2019/08/13 12:47, Eric Camirand wrote:
> The switch port monitoring and collectd tell me there isnt much traffic there. I also don't have this issue with my older servers using em interfaces. I'm going to do more test on another new server and ask tech@openbsd if the issue persist.
>
> Eric
>

There were some changes to the input queues/backpressure mechanism in
-current, so if your older servers are running 6.5 or an earlier release
that is one big difference between the two.

Please don't wait too long before mailing tech@, if changes are needed
the window for making these prior to the next OpenBSD release is shrinking
and the people who need to see this are more likely to see it there.

As well as dmesg and a brief outline of config (also mention any pseudo-
devices used e.g. vlan/trunk/aggr), it will probably be useful to
paste in the output from "systat mbuf" and "sysctl kern.netlivelocks".