pf congestion on low traffic

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

pf congestion on low traffic

Eric Camirand
Hello,

I'm testing new servers (same mboard/cpu) and getting 'congestion' in
the pf information output on very low level of traffic (~100 pps).

Traffic is coming in on em0:901 and em0:2906 using carp.
Traffic level is monitored on the switch port.

First tried 6.5-stable and now using 6.6-beta. Only default daemons
and nsd are running.

Can somebody help me figure this out ?
Thanks !

Eric Camirand

===

sysctl kern.netlivelocks
kern.netlivelocks=2

===

Status: Enabled for 0 days 19:25:59             Debug: info

Hostid:   0xdc985578
Checksum: 0x5b36311462da2ab76b362f191125df49

Interface Stats for egress            IPv4             IPv6
  Bytes In                       410275731        159715549
  Bytes Out                      645392204        219562746
  Packets In
    Passed                         5230566          1821014
    Blocked                          15434              298
  Packets Out
    Passed                         4496816          1353481
    Blocked                              1               22

State Table                          Total             Rate
  current entries                     2507              
  half-open tcp                          0              
  searches                        12917617          184.6/s
  inserts                          5637553           80.6/s
  removals                         5635046           80.5/s
Source Tracking Table
  current entries                        0              
  searches                               0            0.0/s
  inserts                                0            0.0/s
  removals                               0            0.0/s
Counters
  match                            5653419           80.8/s
  bad-offset                             0            0.0/s
  fragment                               0            0.0/s
  short                                  0            0.0/s
  normalize                             15            0.0/s
  memory                                 0            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                             5            0.0/s
  ip-option                            116            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                        13            0.0/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                              0            0.0/s
  synproxy                               0            0.0/s
  translate                              0            0.0/s
  no-route                               0            0.0/s
Limit Counters
  max states per rule                    0            0.0/s
  max-src-states                         0            0.0/s
  max-src-nodes                          0            0.0/s
  max-src-conn                           0            0.0/s
  max-src-conn-rate                      0            0.0/s
  overload table insertion               0            0.0/s
  overload flush states                  0            0.0/s
  synfloods detected                     0            0.0/s
  syncookies sent                        0            0.0/s
  syncookies validated                   0            0.0/s
Adaptive Syncookies Watermarks
  start                              25000 states
  end                                12500 states

===

@0 block drop log all
  [ Evaluations: 5652928   Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 62066 State Creations: 0     ]
@1 pass quick proto carp all
  [ Evaluations: 5652928   Packets: 105212    Bytes: 6943972     States: 2     ]
  [ Inserted: uid 0 pid 62066 State Creations: 2     ]
@2 block drop in quick on egress from <BAD:0> to any
  [ Evaluations: 5652926   Packets: 0         Bytes: 0           States: 0     ]
  [ Inserted: uid 0 pid 62066 State Creations: 0     ]
@3 pass in quick on egress proto tcp from any to <DNS:8> port = 53 flags S/SA
  [ Evaluations: 5646195   Packets: 141995    Bytes: 7940619     States: 52    ]
  [ Inserted: uid 0 pid 62066 State Creations: 21188 ]
@4 pass in quick on egress proto tcp from <MGMT:1> to any port = 22 flags S/SA
  [ Evaluations: 13461     Packets: 7958      Bytes: 5424896     States: 1     ]
  [ Inserted: uid 0 pid 62066 State Creations: 14    ]
@5 pass in quick on egress proto udp from any to <DNS:8> port = 53
  [ Evaluations: 5624993   Packets: 11196223  Bytes: 1279868426  States: 3051  ]
  [ Inserted: uid 0 pid 62066 State Creations: 5582822]
@6 pass in quick on egress inet proto icmp all
  [ Evaluations: 42171     Packets: 11503     Bytes: 727676      States: 2     ]
  [ Inserted: uid 0 pid 62066 State Creations: 5041  ]
@7 pass in quick on egress inet6 proto ipv6-icmp all
  [ Evaluations: 36988     Packets: 59374     Bytes: 4213306     States: 4     ]
  [ Inserted: uid 0 pid 62066 State Creations: 21286 ]
@8 block drop in log quick on egress all
  [ Evaluations: 15604     Packets: 15604     Bytes: 1183076     States: 0     ]
  [ Inserted: uid 0 pid 62066 State Creations: 0     ]
@9 pass out quick on egress all flags S/SA
  [ Evaluations: 6731      Packets: 24872     Bytes: 9727811     States: 2     ]
  [ Inserted: uid 0 pid 62066 State Creations: 6729  ]
@10 block drop out quick on egress all
  [ Evaluations: 2         Packets: 2         Bytes: 144         States: 0     ]
  [ Inserted: uid 0 pid 62066 State Creations: 0     ]

===

   1 users Load 0.04 0.05 0.00                     so-1-s-010 22:22:21

IFACE             LIVELOCKS  SIZE ALIVE   LWM   HWM   CWM
System                    0   256    47          16
                             2112    38          12
lo0
em0                          2050    38    10   256    38
em1
em2
em3
enc0
vlan901
pflog0
vlan2906
carp0
carp1
carp2
carp3

===

OpenBSD 6.6-beta (GENERIC.MP) #205: Mon Aug 12 21:58:39 MDT 2019
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 34156728320 (32574MB)
avail mem = 33108889600 (31575MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7fb76000 (68 entries)
bios0: vendor American Megatrends Inc. version "2.2" date 05/23/2018
bios0: Supermicro X11SSH-F
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC FPDT FIDT SPMI MCFG HPET LPIT SSDT SSDT SSDT DBGP DBG2 SSDT PRAD SSDT UEFI SSDT DMAR EINJ ERST BERT HEST
acpi0: wakeup devices PEG0(S4) PEGP(S4) PEG1(S4) PEGP(S4) PEG2(S4) PEGP(S4) RP09(S4) PXSX(S4) RP10(S4) PXSX(S4) RP11(S4) PXSX(S4) RP12(S4) PXSX(S4) RP13(S4) PXSX(S4) [...]
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E3-1240 v6 @ 3.70GHz, 3701.52 MHz, 06-9e-09
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
tsc_timecounter_init: TSC skew=0 observed drift=0
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 24MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.2.4.1, IBE
cpu1 at mainbus0: apid 2 (application processor)
TSC skew=-2
cpu1: Intel(R) Xeon(R) CPU E3-1240 v6 @ 3.70GHz, 3700.01 MHz, 06-9e-09
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
tsc_timecounter_init: TSC skew=-2 observed drift=0
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
TSC skew=-3
cpu2: Intel(R) Xeon(R) CPU E3-1240 v6 @ 3.70GHz, 3700.00 MHz, 06-9e-09
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
tsc_timecounter_init: TSC skew=-3 observed drift=0
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
TSC skew=1
cpu3: Intel(R) Xeon(R) CPU E3-1240 v6 @ 3.70GHz, 3700.00 MHz, 06-9e-09
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SGX,BMI1,HLE,AVX2,SMEP,BMI2,ERMS,INVPCID,RTM,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PT,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
tsc_timecounter_init: TSC skew=1 observed drift=0
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpihpet0 at acpi0: 23999999 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEG0)
acpiprt2 at acpi0: bus -1 (PEG1)
acpiprt3 at acpi0: bus -1 (PEG2)
acpiprt4 at acpi0: bus -1 (RP09)
acpiprt5 at acpi0: bus -1 (RP13)
acpiprt6 at acpi0: bus 2 (RP01)
acpiprt7 at acpi0: bus 3 (RP02)
acpiprt8 at acpi0: bus -1 (RP03)
acpiprt9 at acpi0: bus -1 (RP04)
acpiprt10 at acpi0: bus -1 (RP05)
acpiprt11 at acpi0: bus -1 (RP06)
acpiprt12 at acpi0: bus 4 (RP07)
acpiprt13 at acpi0: bus 5 (BR53)
acpiprt14 at acpi0: bus -1 (RP08)
acpiprt15 at acpi0: bus -1 (RP17)
acpiprt16 at acpi0: bus -1 (RP18)
acpiprt17 at acpi0: bus -1 (RP19)
acpiprt18 at acpi0: bus -1 (RP20)
acpiprt19 at acpi0: bus -1 (RP14)
acpiprt20 at acpi0: bus -1 (RP15)
acpiprt21 at acpi0: bus -1 (RP16)
acpiec0 at acpi0: not present
acpicpu0 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu1 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu2 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpicpu3 at acpi0: C3(200@256 mwait.1@0x40), C2(200@151 mwait.1@0x33), C1(1000@1 mwait.1), PSS
acpipwrres0 at acpi0: PG00, resource for PEG0
acpipwrres1 at acpi0: PG01, resource for PEG1
acpipwrres2 at acpi0: PG02, resource for PEG2
acpipwrres3 at acpi0: WRST
acpipwrres4 at acpi0: WRST
acpipwrres5 at acpi0: WRST
acpipwrres6 at acpi0: WRST
acpipwrres7 at acpi0: WRST
acpipwrres8 at acpi0: WRST
acpipwrres9 at acpi0: WRST
acpipwrres10 at acpi0: WRST
acpipwrres11 at acpi0: WRST
acpipwrres12 at acpi0: WRST
acpipwrres13 at acpi0: WRST
acpipwrres14 at acpi0: WRST
acpipwrres15 at acpi0: WRST
acpipwrres16 at acpi0: WRST
acpipwrres17 at acpi0: WRST
acpipwrres18 at acpi0: WRST
acpipwrres19 at acpi0: WRST
acpipwrres20 at acpi0: WRST
acpipwrres21 at acpi0: WRST
acpipwrres22 at acpi0: WRST
acpipwrres23 at acpi0: FN00, resource for FAN0
acpipwrres24 at acpi0: FN01, resource for FAN1
acpipwrres25 at acpi0: FN02, resource for FAN2
acpipwrres26 at acpi0: FN03, resource for FAN3
acpipwrres27 at acpi0: FN04, resource for FAN4
acpitz0 at acpi0: critical temperature is 119 degC
acpitz1 at acpi0: critical temperature is 119 degC
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
"IPI0001" at acpi0 not configured
"INT0E0C" at acpi0 not configured
acpibtn0 at acpi0: SLPB
"INT33A1" at acpi0 not configured
acpibtn1 at acpi0: PWRB
"ACPI000D" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
"PNP0C0B" at acpi0 not configured
acpivideo0 at acpi0: GFX0
acpivout0 at acpivideo0: DD1F
ipmi at mainbus0 not configured
cpu0: using Skylake AVX MDS workaround
cpu0: Enhanced SpeedStep 3701 MHz: speeds: 3701, 3700, 3500, 3300, 3100, 2900, 2700, 2500, 2200, 2000, 1800, 1600, 1400, 1200, 1000, 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel Xeon E3-1200 v6/7 Host" rev 0x05
ppb0 at pci0 dev 1 function 0 "Intel Core 6G PCIE" rev 0x05: msi
pci1 at ppb0 bus 1
em0 at pci1 dev 0 function 0 "Intel 82571EB" rev 0x06: apic 2 int 16, address 00:15:17:a6:52:fa
em1 at pci1 dev 0 function 1 "Intel 82571EB" rev 0x06: apic 2 int 17, address 00:15:17:a6:52:fb
"Intel 100 Series ISH" rev 0x31 at pci0 dev 19 function 0 not configured
xhci0 at pci0 dev 20 function 0 "Intel 100 Series xHCI" rev 0x31: msi, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1
pchtemp0 at pci0 dev 20 function 2 "Intel 100 Series Thermal" rev 0x31
"Intel 100 Series MEI" rev 0x31 at pci0 dev 22 function 0 not configured
"Intel 100 Series MEI" rev 0x31 at pci0 dev 22 function 1 not configured
ahci0 at pci0 dev 23 function 0 "Intel 100 Series AHCI" rev 0x31: msi, AHCI 1.3.1
ahci0: port 4: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 4 lun 0: <ATA, INTEL SSDSC2BW24, 400i> SCSI3 0/direct fixed naa.5001517bb2af180f
sd0: 228936MB, 512 bytes/sector, 468862128 sectors, thin
ppb1 at pci0 dev 28 function 0 "Intel 100 Series PCIE" rev 0xf1: msi
pci2 at ppb1 bus 2
em2 at pci2 dev 0 function 0 "Intel I210" rev 0x03: msi, address ac:1f:6b:77:ed:c6
ppb2 at pci0 dev 28 function 1 "Intel 100 Series PCIE" rev 0xf1: msi
pci3 at ppb2 bus 3
em3 at pci3 dev 0 function 0 "Intel I210" rev 0x03: msi, address ac:1f:6b:77:ed:c7
ppb3 at pci0 dev 28 function 6 "Intel 100 Series PCIE" rev 0xf1: msi
pci4 at ppb3 bus 4
ppb4 at pci4 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x03
pci5 at ppb4 bus 5
vga1 at pci5 dev 0 function 0 "ASPEED Technology AST2000" rev 0x30
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 31 function 0 "Intel C236 LPC" rev 0x31
"Intel 100 Series PMC" rev 0x31 at pci0 dev 31 function 2 not configured
ichiic0 at pci0 dev 31 function 4 "Intel 100 Series SMBus" rev 0x31: apic 2 int 16
iic0 at ichiic0
sdtemp0 at iic0 addr 0x18: stts2004
sdtemp1 at iic0 addr 0x19: stts2004
sdtemp2 at iic0 addr 0x1a: stts2004
sdtemp3 at iic0 addr 0x1b: stts2004
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT
uhub1 at uhub0 port 14 configuration 1 interface 0 "ATEN International product 0x7000" rev 2.00/0.00 addr 2
uhidev0 at uhub1 port 1 configuration 1 interface 0 "ATEN International product 0x2419" rev 1.10/1.00 addr 3
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub1 port 1 configuration 1 interface 1 "ATEN International product 0x2419" rev 1.10/1.00 addr 3
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (2b8c5fc37393b73b.a) swap on sd0b dump on sd0b

Reply | Threaded
Open this post in threaded view
|

Re: pf congestion on low traffic

Eric Camirand
Hi Richard,

I'm still getting congestion using acpihpet0. Not sure about
the arp, i don't think there is that much. Older hardware
using bsd.mp on the same lan doesn't have this issue.

Thanks!


On Aug 22, 2019, at 5:57 AM, Richard Procter <[hidden email]> wrote:

Hi Eric,

I was unfamiliar with how the congestion counter worked so I
looked at the code; I’m not an expert here.

I’m puzzled. Is it possible you’re seeing a lot of ARP packets?

Another shot in the dark: it might be timecounter related.
You might try:

# sysctl kern.timecounter.hardware=acpihpet0

and see if you can reproduce.

There has been a lot of work on time and the timecounter recently;
it may help to upgrade to an even later snapshot.

Please let me know how you get on.

best,
Richard.