Leaking mbufs on 6.4 and 6.5

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Leaking mbufs on 6.4 and 6.5

Lévai, Dániel
Hello again,

I've upgraded to 6.5 and if possible it has gotten worse. Now it gets unresponsive after ~half a day. With 6.4 I could get away with one reboot per day but it seems I'm going to need two or more with 6.5.
Does anyone have any suggestions or suspicions? Anyone else noticed this or similar throughout the previous two releases? I know a lot of changes went into and are underway in the network stack re multi-threading etc... is this maybe something related to this leak? I'm trying not to push unrelated or unnecessary buttons until there's a good candidate for the source of this problem.

Thanks in advance,
Daniel

> Hi!
>
> I have mbufs leaking after introducing a network topology change on a
> router:
>
> After a reboot + 20 minutes:
>    1 users Load 0.46 0.38 0.33                     16:40:43
> IFACE             LIVELOCKS  SIZE ALIVE   LWM   HWM   CWM
> System                    0   256  4100         260
>                              2048  3717         468
>                              2112    53           7
>                              4096   128          20
>                              9216    17          12
>
> After one day:
>    1 users Load 0.53 0.40 0.32                     16:14:53
> IFACE             LIVELOCKS  SIZE ALIVE   LWM   HWM   CWM
> System                    0   256  131K        8456
>                              2048     *       16022
>                              2112    69          23
>                              4096   128          21
>                              9216    17          19
>
> After two days:
>    1 users Load 0.14 0.20 0.22                     15:00:00
> IFACE             LIVELOCKS  SIZE ALIVE   LWM   HWM   CWM
> System                    0   256  215K       13780
>                              2048     *       26954
>                              2112    66           9
>                              4096   128          25
>                              9216    11          11
>
> I think this is either caused by starting to use an on-board re(4)
> device or starting to use vether(4) in a bridge(4) (dmesg, details
> below).
>
> In any case, 1 gigs worth of memory is consumed in about two days (give
> or take a couple of hours) with mild internet traffic (pppoe connected
> to re(4)).
>
> I was given the advice to inquire/report here, maybe someone can help
> track down the root cause.
>
> Any help is appreciated!
>
>
> This is my original e-mail [1] sent to misc@ (minus the writing about
> the processes running), it has a bunch of information, if someone's
> interested:
>
> <TL;DR>
> After a recent network configuration change (added re(4), vether(4)) I'm
> experiencing this memory leak from somewhere.
>
> How can I check/query how much memory the kernel (or parts of it) is
> using over time, besides running top(1) with system processes shown --
> I'm also staring at systat(1)'s `malloc' and `pool' views but I'm not
> really sure what I'm (or rather what I should be) looking at.
> </TL;DR>
>
>
> I have an on-board Realtek NIC and an add-on Intel PCIe 4 port NIC.
>
> re0 at pci3 dev 0 function 0 "Realtek 8168" rev 0x0c: RTL8168G/8111G (0x4c00), msi
>
> em0 at pci1 dev 0 function 0 "Intel I350" rev 0x01: msi
> em1 at pci1 dev 0 function 1 "Intel I350" rev 0x01: msi
> em2 at pci1 dev 0 function 2 "Intel I350" rev 0x01: msi
> em3 at pci1 dev 0 function 3 "Intel I350" rev 0x01: msi
>
> Before I started to experience the leak I had the on-board NIC disabled
> in BIOS and only used two ports of the Intel NIC. My setup looked like
> this:
>
>
> == Network
>
> ISP --- pppoe0 <-> em0 -- OpenBSD 6.4 -- em1(dhcpd) --- [switch] LAN
>                                       `- athn0 --- LAN
>                                       `- athn1 --- LAN
>                                       `- bridge0(em1, athn[01])
>
> Internal IP configured on em1.
> athn? devices don't have IPs, just bridged to the internal network.
> dhcpd(8) is running on em1.
>
>
> Now I started to use re0 and one more NIC on the Intel card:
>
> ISP --- pppoe0 <-> re0 -- OpenBSD 6.4 -- em0 --- LAN
>                                       `- em1 --- LAN
>                                       `- em2 --- LAN
>                                       `- athn0 --- LAN
>                                       `- athn1 --- LAN
>                                       `- vether0 (dhcpd)
>                                       `- bridge0(em[123], athn[01], vether0)
>
> Now the 4 port Intel NIC basically acts as a switch, the em? interfaces
> don't have IP addresses configured, and the athn? interfaces don't have
> one either.
> I've added a vether(4) interface with an internal IP (same as the em1
> have had before) to the bridge, and dhcpd(8) (and everything else that
> needs it) now uses that interface.
>
>
> # ifconfig bridge0
> bridge0: flags=41<UP,RUNNING>
>         index 10 llprio 3
>         groups: bridge
>         priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp
>         designated: id 00:00:00:00:00:00 priority 0
>
>         athn0 flags=3<LEARNING,DISCOVER>
>                 port 6 ifpriority 0 ifcost 0
> [bunch of `pass/block in/out on athn0 src/dst <MAC>' rules]
>
>         athn1 flags=3<LEARNING,DISCOVER>
>                 port 7 ifpriority 0 ifcost 0
> [bunch of `pass/block in/out on athn1 src/dst <MAC>' rules]
>
>         em0 flags=3<LEARNING,DISCOVER>
>                 port 1 ifpriority 0 ifcost 0
>         em1 flags=3<LEARNING,DISCOVER>
>                 port 2 ifpriority 0 ifcost 0
>         em2 flags=3<LEARNING,DISCOVER>
>                 port 3 ifpriority 0 ifcost 0
>         em3 flags=3<LEARNING,DISCOVER>
>                 port 4 ifpriority 0 ifcost 0
>         vether0 flags=3<LEARNING,DISCOVER>
>                 port 13 ifpriority 0 ifcost 0
>         Addresses (max cache: 100, timeout: 240):
> [...]
>
> # ifconfig vether0
> vether0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
>         lladdr fe:e1:ba:d0:58:44
>         description: Internal LAN
>         index 13 priority 0 llprio 3
>         groups: vether
>         media: Ethernet autoselect
>         status: active
>         inet 192.168.0.1 netmask 0xffff0000 broadcast 192.168.255.255
>         inet6 <> prefixlen 64 scopeid 0xd
>         inet6 <> prefixlen 64 pltime 573642 vltime 573642
>
> # ifconfig em
> em0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
>         lladdr
>         description: Internal LAN - Room1
>         index 1 priority 0 llprio 3
>         media: Ethernet autoselect (100baseTX full-duplex,rxpause,txpause)
>         status: active
> em1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
>         lladdr
>         description: Internal LAN - Room2
>         index 2 priority 0 llprio 3
>         media: Ethernet autoselect (1000baseT full-duplex,master,rxpause,txpause)
>         status: active
> em2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
>         lladdr
>         description: Internal LAN - Room3
>         index 3 priority 0 llprio 3
>         media: Ethernet autoselect (100baseTX full-duplex,rxpause,txpause)
>         status: active
> em3: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> mtu 1500
>         lladdr
>         index 4 priority 0 llprio 3
>         media: Ethernet autoselect (none)
>         status: no carrier
>
>
> Nothing fancy, just basically a "software switch" for connecting
> separate rooms in the house and adding a vether(4) interface for
> conveniency.
>
> Also, everything else remain unchanged, i.e.: no additional services
> running, no new software has been installed, no introduction of new pf
> rules, other than replacing the interface names of course.
>
> I'm using /etc/malloc.conf@ -> S on all my machines but never had any
> problems with it (and still don't anywhere else).
>
>
> == pf(4)
>
> pf(4) is used on the box, and the rules that have been changed are the
> ones related to the interfaces mentioned above:
>
> set skip on lo0
> set skip on bridge0
> pass on vether0 all flags S/SA allow-opts
> pass on em0 all flags S/SA allow-opts
> pass on em1 all flags S/SA allow-opts
> pass on em2 all flags S/SA allow-opts
> pass on em3 all flags S/SA allow-opts
> pass on athn0 all flags S/SA allow-opts
> pass on athn1 all flags S/SA allow-opts
> match in on vether0 inet proto tcp from 192.168.0.0/16 port = <port> to any set \
> (prio(2, 6), queue(ssh_bulk, ssh_rt)) block return on vether0 proto tcp from any to \
> any port = 25 block return on em0 proto tcp from any to any port = 25
> block return on em1 proto tcp from any to any port = 25
> block return on em2 proto tcp from any to any port = 25
> block return on em3 proto tcp from any to any port = 25
> block return on athn0 proto tcp from any to any port = 25
> block return on athn1 proto tcp from any to any port = 25
>
> Previously I've had `skip' on athn? as well.
>
> (I can upload the complete rule set somewhere if it is of interest)
>
>
>
> == dmesg
>
> OpenBSD 6.4 (GENERIC.MP) #9: Tue Mar 26 19:21:43 CET 2019
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 2059997184 (1964MB)
> avail mem = 1988288512 (1896MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xec130 (75 entries)
> bios0: vendor American Megatrends Inc. version "F2" date 08/06/2015
> bios0: Gigabyte Technology Co., Ltd. H81M-DS2
> acpi0 at bios0: rev 2
> acpi0: sleep states S0 S3 S4 S5
> acpi0: tables DSDT FACP APIC FPDT SSDT SSDT MCFG HPET SSDT SSDT
> acpi0: wakeup devices PXSX(S4) RP01(S4) PXSX(S4) PXSX(S4) RP03(S4) PXSX(S4) PXSX(S4) \
> RP05(S4) PXSX(S4) RP06(S4) PXSX(S4) PXSX(S4) GLAN(S4) EHC1(S4) EHC2(S4) XHC_(S4) \
> [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: Intel(R) Celeron(R) CPU G1840 @ 2.80GHz, 2793.95 MHz, 06-3c-03
> cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,D \
> S,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSS \
> E3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,XSAVE,RDRAND,NXE,PAGE1 \
> GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,ERMS,INVPCID,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 99MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
> cpu1 at mainbus0: apid 2 (application processor)
> cpu1: Intel(R) Celeron(R) CPU G1840 @ 2.80GHz, 2793.55 MHz, 06-3c-03
> cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,D \
> S,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSS \
> E3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,XSAVE,RDRAND,NXE,PAGE1 \
> GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,ERMS,INVPCID,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 0, core 1, package 0
> ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
> acpimcfg0 at acpi0
> acpimcfg0: addr 0xf8000000, bus 0-63
> acpihpet0 at acpi0: 14318179 Hz
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 2 (RP01)
> acpiprt2 at acpi0: bus 3 (RP03)
> acpiprt3 at acpi0: bus 4 (RP05)
> acpiprt4 at acpi0: bus 5 (RP06)
> acpiprt5 at acpi0: bus 1 (PEG0)
> acpiprt6 at acpi0: bus -1 (PEG1)
> acpiprt7 at acpi0: bus -1 (PEG2)
> acpiec0 at acpi0: not present
> acpicpu0 at acpi0: C2(500@67 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> acpicpu1 at acpi0: C2(500@67 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> acpipwrres0 at acpi0: FN00, resource for FAN0
> acpipwrres1 at acpi0: FN01, resource for FAN1
> acpipwrres2 at acpi0: FN02, resource for FAN2
> acpipwrres3 at acpi0: FN03, resource for FAN3
> acpipwrres4 at acpi0: FN04, resource for FAN4
> acpitz0 at acpi0: critical temperature is 105 degC
> acpitz1 at acpi0: critical temperature is 105 degC
> acpicmos0 at acpi0
> acpibtn0 at acpi0: PWRB
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> "PNP0C0B" at acpi0 not configured
> acpivideo0 at acpi0: GFX0
> acpivout0 at acpivideo0: DD1F
> cpu0: Enhanced SpeedStep 2793 MHz: speeds: 2800, 2700, 2500, 2400, 2300, 2100, 2000, \
> 1900, 1700, 1600, 1500, 1300, 1200, 1100, 900, 800 MHz pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel Core 4G Host" rev 0x06
> ppb0 at pci0 dev 1 function 0 "Intel Core 4G PCIE" rev 0x06: msi
> pci1 at ppb0 bus 1
> em0 at pci1 dev 0 function 0 "Intel I350" rev 0x01: msi, address
> em1 at pci1 dev 0 function 1 "Intel I350" rev 0x01: msi, address
> em2 at pci1 dev 0 function 2 "Intel I350" rev 0x01: msi, address
> em3 at pci1 dev 0 function 3 "Intel I350" rev 0x01: msi, address
> inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics" rev 0x06
> drm0 at inteldrm0
> inteldrm0: msi
> error: [drm:pid0:drm_edid_block_valid] *ERROR* EDID checksum is invalid, remainder is \
> 39 Raw EDID:
>         00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
>         fe 11 01 03 08 25 1e 78 ca bb 05 a3 56 47 99 24
>         f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> error: [drm:pid0:drm_edid_block_valid] *ERROR* EDID checksum is invalid, remainder is \
> 7 Raw EDID:
>         00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
>         de 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
>         d4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> error: [drm:pid0:drm_edid_block_valid] *ERROR* EDID checksum is invalid, remainder is \
> 39 Raw EDID:
>         00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
>         de 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
>         f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> error: [drm:pid0:drm_edid_block_valid] *ERROR* EDID checksum is invalid, remainder is \
> 71 Raw EDID:
>         00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
>         fe 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
>         f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
>         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> drm:pid0:drm_do_get_edid *WARNING* VGA-: EDID block 0 invalid.
> inteldrm0: 1024x768, 32bpp
> wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
> wsdisplay0: screen 1-5 added (std, vt100 emulation)
> azalia0 at pci0 dev 3 function 0 "Intel Core 4G HD Audio" rev 0x06: msi
> azalia0: No codecs found
> "Intel 8 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured
> ehci0 at pci0 dev 26 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 16
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> ppb1 at pci0 dev 28 function 0 "Intel 8 Series PCIE" rev 0xd5: msi
> pci2 at ppb1 bus 2
> ppb2 at pci0 dev 28 function 2 "Intel 8 Series PCIE" rev 0xd5: msi
> pci3 at ppb2 bus 3
> re0 at pci3 dev 0 function 0 "Realtek 8168" rev 0x0c: RTL8168G/8111G (0x4c00), msi, \
> address  rgephy0 at re0 phy 7: RTL8251 PHY, rev. 0
> ppb3 at pci0 dev 28 function 4 "Intel 8 Series PCIE" rev 0xd5: msi
> pci4 at ppb3 bus 4
> athn0 at pci4 dev 0 function 0 "Atheros AR9281" rev 0x01: apic 8 int 16
> athn0: AR9280 rev 2 (2T2R), ROM rev 16, address
> ppb4 at pci0 dev 28 function 5 "Intel 8 Series PCIE" rev 0xd5: msi
> pci5 at ppb4 bus 5
> athn1 at pci5 dev 0 function 0 "Atheros AR9281" rev 0x01: apic 8 int 17
> athn1: AR9280 rev 2 (2T2R), ROM rev 22, address
> ehci1 at pci0 dev 29 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 23
> usb1 at ehci1: USB revision 2.0
> uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> pcib0 at pci0 dev 31 function 0 "Intel H81 LPC" rev 0x05
> ahci0 at pci0 dev 31 function 2 "Intel 8 Series AHCI" rev 0x05: msi, AHCI 1.3
> ahci0: port 4: 3.0Gb/s
> ahci0: port 5: 3.0Gb/s
> scsibus1 at ahci0: 32 targets
> sd0 at scsibus1 targ 4 lun 0: <ATA, KINGSTON SHFS37A, 608A> SCSI3 0/direct fixed \
>                 naa.50026b726304fb22
> sd0: 114473MB, 512 bytes/sector, 234441648 sectors, thin
> sd1 at scsibus1 targ 5 lun 0: <ATA, KINGSTON SHFS37A, 608A> SCSI3 0/direct fixed \
>                 naa.50026b726304fbe2
> sd1: 114473MB, 512 bytes/sector, 234441648 sectors, thin
> ichiic0 at pci0 dev 31 function 3 "Intel 8 Series SMBus" rev 0x05: apic 8 int 18
> iic0 at ichiic0
> spdmem0 at iic0 addr 0x50: 2GB DDR3 SDRAM PC3-10600
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev \
> 2.00/0.05 addr 2 uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate \
> Matching Hub" rev 2.00/0.05 addr 2 uhidev0 at uhub3 port 2 configuration 1 interface \
>                 0 "SINO WEALTH USB KEYBOARD" rev 1.10/1.00 addr 3
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 variable keys, 6 key codes
> wskbd1 at ukbd0 mux 1
> wskbd1: connecting to wsdisplay0
> uhidev1 at uhub3 port 2 configuration 1 interface 1 "SINO WEALTH USB KEYBOARD" rev \
>                 1.10/1.00 addr 3
> uhidev1: iclass 3/0, 5 report ids
> uhid0 at uhidev1 reportid 2: input=1, output=0, feature=0
> uhid1 at uhidev1 reportid 3: input=3, output=0, feature=0
> uhid2 at uhidev1 reportid 5: input=0, output=0, feature=5
> uhidev2 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
>                 Back-UPS XS 1400U  FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 4
> uhidev2: iclass 3/0, 123 report ids
> upd0 at uhidev2
> vscsi0 at root
> scsibus2 at vscsi0: 256 targets
> softraid0 at root
> scsibus3 at softraid0: 256 targets
> sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006> SCSI2 0/direct fixed
> sd2: 114470MB, 512 bytes/sector, 234435953 sectors
> root on sd2a (59f8b4a610bc10ff.a) swap on sd2b dump on sd2b
> pppoe: GENERIC ERROR: RP-PPPoE: Child pppd process terminated
> wskbd1: disconnecting from wsdisplay0
> wskbd1 detached
> ukbd0 detached
> uhidev0 detached
> uhid0 detached
> uhid1 detached
> uhid2 detached
> uhidev1 detached
> pppoe0: received unexpected PADO
> pppoe0: received unexpected PADO
> pppoe0: received unexpected PADO
> upd0 detached
> uhidev2 detached
> uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
>                 Back-UPS XS 1400U  FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> uhidev0: iclass 3/0, 123 report ids
> upd0 at uhidev0
> upd0 detached
> uhidev0 detached
> uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
>                 Back-UPS XS 1400U  FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> uhidev0: iclass 3/0, 123 report ids
> upd0 at uhidev0
> upd0 detached
> uhidev0 detached
> uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
>                 Back-UPS XS 1400U  FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> uhidev0: iclass 3/0, 123 report ids
> upd0 at uhidev0
> upd0 detached
> uhidev0 detached
> uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
>                 Back-UPS XS 1400U  FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> uhidev0: iclass 3/0, 123 report ids
> upd0 at uhidev0
> upd0 detached
> uhidev0 detached
> uhub3: device problem, disabling port 3
> uhub3: port error, restarting port 3
> uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
>                 Back-UPS XS 1400U  FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> uhidev0: iclass 3/0, 123 report ids
> upd0 at uhidev0
>
>
>
> [1] - https://marc.info/?l=openbsd-misc&m=155532485003397&w=2

-- 
Lévai, Dániel

Reply | Threaded
Open this post in threaded view
|

Re: Leaking mbufs on 6.4 and 6.5

Claudio Jeker-3
On Thu, May 02, 2019 at 07:48:30AM +0000, Lévai, Dániel wrote:

> Hello again,
>
> I've upgraded to 6.5 and if possible it has gotten worse. Now it gets
> unresponsive after ~half a day. With 6.4 I could get away with one
> reboot per day but it seems I'm going to need two or more with 6.5.
> Does anyone have any suggestions or suspicions? Anyone else noticed this
> or similar throughout the previous two releases? I know a lot of changes
> went into and are underway in the network stack re multi-threading
> etc... is this maybe something related to this leak? I'm trying not to
> push unrelated or unnecessary buttons until there's a good candidate for
> the source of this problem.
>

Can you try to identify which of the many bits in your setup cause the
mbuf leak? Easiest is to use a second system to split some of the
functionality out. Try to remove vether(4) or the pppoe(4) part from your
setup and check if this changes the behaviour. Also try to remove the MAC
filters on bridge0 and see if that changes behavior.
The leaked mbufs are from the system, re(4) or vether(4). Neither em(4)
nor athn(4) use 2048 byte buffers on the RX side.

There is something special in your setup and you need to help us identify
what it is. I know many people run similar setups without any troubles.

> Thanks in advance,
> Daniel
>
> > Hi!
> >
> > I have mbufs leaking after introducing a network topology change on a
> > router:
> >
> > After a reboot + 20 minutes:
> >    1 users Load 0.46 0.38 0.33                     16:40:43
> > IFACE             LIVELOCKS  SIZE ALIVE   LWM   HWM   CWM
> > System                    0   256  4100         260
> >                              2048  3717         468
> >                              2112    53           7
> >                              4096   128          20
> >                              9216    17          12
> >
> > After one day:
> >    1 users Load 0.53 0.40 0.32                     16:14:53
> > IFACE             LIVELOCKS  SIZE ALIVE   LWM   HWM   CWM
> > System                    0   256  131K        8456
> >                              2048     *       16022
> >                              2112    69          23
> >                              4096   128          21
> >                              9216    17          19
> >
> > After two days:
> >    1 users Load 0.14 0.20 0.22                     15:00:00
> > IFACE             LIVELOCKS  SIZE ALIVE   LWM   HWM   CWM
> > System                    0   256  215K       13780
> >                              2048     *       26954
> >                              2112    66           9
> >                              4096   128          25
> >                              9216    11          11
> >
> > I think this is either caused by starting to use an on-board re(4)
> > device or starting to use vether(4) in a bridge(4) (dmesg, details
> > below).
> >
> > In any case, 1 gigs worth of memory is consumed in about two days (give
> > or take a couple of hours) with mild internet traffic (pppoe connected
> > to re(4)).
> >
> > I was given the advice to inquire/report here, maybe someone can help
> > track down the root cause.
> >
> > Any help is appreciated!
> >
> >
> > This is my original e-mail [1] sent to misc@ (minus the writing about
> > the processes running), it has a bunch of information, if someone's
> > interested:
> >
> > <TL;DR>
> > After a recent network configuration change (added re(4), vether(4)) I'm
> > experiencing this memory leak from somewhere.
> >
> > How can I check/query how much memory the kernel (or parts of it) is
> > using over time, besides running top(1) with system processes shown --
> > I'm also staring at systat(1)'s `malloc' and `pool' views but I'm not
> > really sure what I'm (or rather what I should be) looking at.
> > </TL;DR>
> >
> >
> > I have an on-board Realtek NIC and an add-on Intel PCIe 4 port NIC.
> >
> > re0 at pci3 dev 0 function 0 "Realtek 8168" rev 0x0c: RTL8168G/8111G (0x4c00), msi
> >
> > em0 at pci1 dev 0 function 0 "Intel I350" rev 0x01: msi
> > em1 at pci1 dev 0 function 1 "Intel I350" rev 0x01: msi
> > em2 at pci1 dev 0 function 2 "Intel I350" rev 0x01: msi
> > em3 at pci1 dev 0 function 3 "Intel I350" rev 0x01: msi
> >
> > Before I started to experience the leak I had the on-board NIC disabled
> > in BIOS and only used two ports of the Intel NIC. My setup looked like
> > this:
> >
> >
> > == Network
> >
> > ISP --- pppoe0 <-> em0 -- OpenBSD 6.4 -- em1(dhcpd) --- [switch] LAN
> >                                       `- athn0 --- LAN
> >                                       `- athn1 --- LAN
> >                                       `- bridge0(em1, athn[01])
> >
> > Internal IP configured on em1.
> > athn? devices don't have IPs, just bridged to the internal network.
> > dhcpd(8) is running on em1.
> >
> >
> > Now I started to use re0 and one more NIC on the Intel card:
> >
> > ISP --- pppoe0 <-> re0 -- OpenBSD 6.4 -- em0 --- LAN
> >                                       `- em1 --- LAN
> >                                       `- em2 --- LAN
> >                                       `- athn0 --- LAN
> >                                       `- athn1 --- LAN
> >                                       `- vether0 (dhcpd)
> >                                       `- bridge0(em[123], athn[01], vether0)
> >
> > Now the 4 port Intel NIC basically acts as a switch, the em? interfaces
> > don't have IP addresses configured, and the athn? interfaces don't have
> > one either.
> > I've added a vether(4) interface with an internal IP (same as the em1
> > have had before) to the bridge, and dhcpd(8) (and everything else that
> > needs it) now uses that interface.
> >
> >
> > # ifconfig bridge0
> > bridge0: flags=41<UP,RUNNING>
> >         index 10 llprio 3
> >         groups: bridge
> >         priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp
> >         designated: id 00:00:00:00:00:00 priority 0
> >
> >         athn0 flags=3<LEARNING,DISCOVER>
> >                 port 6 ifpriority 0 ifcost 0
> > [bunch of `pass/block in/out on athn0 src/dst <MAC>' rules]
> >
> >         athn1 flags=3<LEARNING,DISCOVER>
> >                 port 7 ifpriority 0 ifcost 0
> > [bunch of `pass/block in/out on athn1 src/dst <MAC>' rules]
> >
> >         em0 flags=3<LEARNING,DISCOVER>
> >                 port 1 ifpriority 0 ifcost 0
> >         em1 flags=3<LEARNING,DISCOVER>
> >                 port 2 ifpriority 0 ifcost 0
> >         em2 flags=3<LEARNING,DISCOVER>
> >                 port 3 ifpriority 0 ifcost 0
> >         em3 flags=3<LEARNING,DISCOVER>
> >                 port 4 ifpriority 0 ifcost 0
> >         vether0 flags=3<LEARNING,DISCOVER>
> >                 port 13 ifpriority 0 ifcost 0
> >         Addresses (max cache: 100, timeout: 240):
> > [...]
> >
> > # ifconfig vether0
> > vether0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
> >         lladdr fe:e1:ba:d0:58:44
> >         description: Internal LAN
> >         index 13 priority 0 llprio 3
> >         groups: vether
> >         media: Ethernet autoselect
> >         status: active
> >         inet 192.168.0.1 netmask 0xffff0000 broadcast 192.168.255.255
> >         inet6 <> prefixlen 64 scopeid 0xd
> >         inet6 <> prefixlen 64 pltime 573642 vltime 573642
> >
> > # ifconfig em
> > em0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
> >         lladdr
> >         description: Internal LAN - Room1
> >         index 1 priority 0 llprio 3
> >         media: Ethernet autoselect (100baseTX full-duplex,rxpause,txpause)
> >         status: active
> > em1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
> >         lladdr
> >         description: Internal LAN - Room2
> >         index 2 priority 0 llprio 3
> >         media: Ethernet autoselect (1000baseT full-duplex,master,rxpause,txpause)
> >         status: active
> > em2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
> >         lladdr
> >         description: Internal LAN - Room3
> >         index 3 priority 0 llprio 3
> >         media: Ethernet autoselect (100baseTX full-duplex,rxpause,txpause)
> >         status: active
> > em3: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> mtu 1500
> >         lladdr
> >         index 4 priority 0 llprio 3
> >         media: Ethernet autoselect (none)
> >         status: no carrier
> >
> >
> > Nothing fancy, just basically a "software switch" for connecting
> > separate rooms in the house and adding a vether(4) interface for
> > conveniency.
> >
> > Also, everything else remain unchanged, i.e.: no additional services
> > running, no new software has been installed, no introduction of new pf
> > rules, other than replacing the interface names of course.
> >
> > I'm using /etc/malloc.conf@ -> S on all my machines but never had any
> > problems with it (and still don't anywhere else).
> >
> >
> > == pf(4)
> >
> > pf(4) is used on the box, and the rules that have been changed are the
> > ones related to the interfaces mentioned above:
> >
> > set skip on lo0
> > set skip on bridge0
> > pass on vether0 all flags S/SA allow-opts
> > pass on em0 all flags S/SA allow-opts
> > pass on em1 all flags S/SA allow-opts
> > pass on em2 all flags S/SA allow-opts
> > pass on em3 all flags S/SA allow-opts
> > pass on athn0 all flags S/SA allow-opts
> > pass on athn1 all flags S/SA allow-opts
> > match in on vether0 inet proto tcp from 192.168.0.0/16 port = <port> to any set \
> > (prio(2, 6), queue(ssh_bulk, ssh_rt)) block return on vether0 proto tcp from any to \
> > any port = 25 block return on em0 proto tcp from any to any port = 25
> > block return on em1 proto tcp from any to any port = 25
> > block return on em2 proto tcp from any to any port = 25
> > block return on em3 proto tcp from any to any port = 25
> > block return on athn0 proto tcp from any to any port = 25
> > block return on athn1 proto tcp from any to any port = 25
> >
> > Previously I've had `skip' on athn? as well.
> >
> > (I can upload the complete rule set somewhere if it is of interest)
> >
> >
> >
> > == dmesg
> >
> > OpenBSD 6.4 (GENERIC.MP) #9: Tue Mar 26 19:21:43 CET 2019
> >     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> > real mem = 2059997184 (1964MB)
> > avail mem = 1988288512 (1896MB)
> > mpath0 at root
> > scsibus0 at mpath0: 256 targets
> > mainbus0 at root
> > bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xec130 (75 entries)
> > bios0: vendor American Megatrends Inc. version "F2" date 08/06/2015
> > bios0: Gigabyte Technology Co., Ltd. H81M-DS2
> > acpi0 at bios0: rev 2
> > acpi0: sleep states S0 S3 S4 S5
> > acpi0: tables DSDT FACP APIC FPDT SSDT SSDT MCFG HPET SSDT SSDT
> > acpi0: wakeup devices PXSX(S4) RP01(S4) PXSX(S4) PXSX(S4) RP03(S4) PXSX(S4) PXSX(S4) \
> > RP05(S4) PXSX(S4) RP06(S4) PXSX(S4) PXSX(S4) GLAN(S4) EHC1(S4) EHC2(S4) XHC_(S4) \
> > [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits
> > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> > cpu0 at mainbus0: apid 0 (boot processor)
> > cpu0: Intel(R) Celeron(R) CPU G1840 @ 2.80GHz, 2793.95 MHz, 06-3c-03
> > cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,D \
> > S,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSS \
> > E3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,XSAVE,RDRAND,NXE,PAGE1 \
> > GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,ERMS,INVPCID,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> >
> > cpu0: 256KB 64b/line 8-way L2 cache
> > cpu0: smt 0, core 0, package 0
> > mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> > cpu0: apic clock running at 99MHz
> > cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
> > cpu1 at mainbus0: apid 2 (application processor)
> > cpu1: Intel(R) Celeron(R) CPU G1840 @ 2.80GHz, 2793.55 MHz, 06-3c-03
> > cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,D \
> > S,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSS \
> > E3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,XSAVE,RDRAND,NXE,PAGE1 \
> > GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,ERMS,INVPCID,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> >
> > cpu1: 256KB 64b/line 8-way L2 cache
> > cpu1: smt 0, core 1, package 0
> > ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
> > acpimcfg0 at acpi0
> > acpimcfg0: addr 0xf8000000, bus 0-63
> > acpihpet0 at acpi0: 14318179 Hz
> > acpiprt0 at acpi0: bus 0 (PCI0)
> > acpiprt1 at acpi0: bus 2 (RP01)
> > acpiprt2 at acpi0: bus 3 (RP03)
> > acpiprt3 at acpi0: bus 4 (RP05)
> > acpiprt4 at acpi0: bus 5 (RP06)
> > acpiprt5 at acpi0: bus 1 (PEG0)
> > acpiprt6 at acpi0: bus -1 (PEG1)
> > acpiprt7 at acpi0: bus -1 (PEG2)
> > acpiec0 at acpi0: not present
> > acpicpu0 at acpi0: C2(500@67 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> > acpicpu1 at acpi0: C2(500@67 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> > acpipwrres0 at acpi0: FN00, resource for FAN0
> > acpipwrres1 at acpi0: FN01, resource for FAN1
> > acpipwrres2 at acpi0: FN02, resource for FAN2
> > acpipwrres3 at acpi0: FN03, resource for FAN3
> > acpipwrres4 at acpi0: FN04, resource for FAN4
> > acpitz0 at acpi0: critical temperature is 105 degC
> > acpitz1 at acpi0: critical temperature is 105 degC
> > acpicmos0 at acpi0
> > acpibtn0 at acpi0: PWRB
> > "PNP0C0B" at acpi0 not configured
> > "PNP0C0B" at acpi0 not configured
> > "PNP0C0B" at acpi0 not configured
> > "PNP0C0B" at acpi0 not configured
> > "PNP0C0B" at acpi0 not configured
> > acpivideo0 at acpi0: GFX0
> > acpivout0 at acpivideo0: DD1F
> > cpu0: Enhanced SpeedStep 2793 MHz: speeds: 2800, 2700, 2500, 2400, 2300, 2100, 2000, \
> > 1900, 1700, 1600, 1500, 1300, 1200, 1100, 900, 800 MHz pci0 at mainbus0 bus 0
> > pchb0 at pci0 dev 0 function 0 "Intel Core 4G Host" rev 0x06
> > ppb0 at pci0 dev 1 function 0 "Intel Core 4G PCIE" rev 0x06: msi
> > pci1 at ppb0 bus 1
> > em0 at pci1 dev 0 function 0 "Intel I350" rev 0x01: msi, address
> > em1 at pci1 dev 0 function 1 "Intel I350" rev 0x01: msi, address
> > em2 at pci1 dev 0 function 2 "Intel I350" rev 0x01: msi, address
> > em3 at pci1 dev 0 function 3 "Intel I350" rev 0x01: msi, address
> > inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics" rev 0x06
> > drm0 at inteldrm0
> > inteldrm0: msi
> > error: [drm:pid0:drm_edid_block_valid] *ERROR* EDID checksum is invalid, remainder is \
> > 39 Raw EDID:
> >         00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> >         fe 11 01 03 08 25 1e 78 ca bb 05 a3 56 47 99 24
> >         f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > error: [drm:pid0:drm_edid_block_valid] *ERROR* EDID checksum is invalid, remainder is \
> > 7 Raw EDID:
> >         00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> >         de 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
> >         d4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > error: [drm:pid0:drm_edid_block_valid] *ERROR* EDID checksum is invalid, remainder is \
> > 39 Raw EDID:
> >         00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> >         de 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
> >         f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > error: [drm:pid0:drm_edid_block_valid] *ERROR* EDID checksum is invalid, remainder is \
> > 71 Raw EDID:
> >         00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> >         fe 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
> >         f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> >         ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > drm:pid0:drm_do_get_edid *WARNING* VGA-: EDID block 0 invalid.
> > inteldrm0: 1024x768, 32bpp
> > wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
> > wsdisplay0: screen 1-5 added (std, vt100 emulation)
> > azalia0 at pci0 dev 3 function 0 "Intel Core 4G HD Audio" rev 0x06: msi
> > azalia0: No codecs found
> > "Intel 8 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured
> > ehci0 at pci0 dev 26 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 16
> > usb0 at ehci0: USB revision 2.0
> > uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> > ppb1 at pci0 dev 28 function 0 "Intel 8 Series PCIE" rev 0xd5: msi
> > pci2 at ppb1 bus 2
> > ppb2 at pci0 dev 28 function 2 "Intel 8 Series PCIE" rev 0xd5: msi
> > pci3 at ppb2 bus 3
> > re0 at pci3 dev 0 function 0 "Realtek 8168" rev 0x0c: RTL8168G/8111G (0x4c00), msi, \
> > address  rgephy0 at re0 phy 7: RTL8251 PHY, rev. 0
> > ppb3 at pci0 dev 28 function 4 "Intel 8 Series PCIE" rev 0xd5: msi
> > pci4 at ppb3 bus 4
> > athn0 at pci4 dev 0 function 0 "Atheros AR9281" rev 0x01: apic 8 int 16
> > athn0: AR9280 rev 2 (2T2R), ROM rev 16, address
> > ppb4 at pci0 dev 28 function 5 "Intel 8 Series PCIE" rev 0xd5: msi
> > pci5 at ppb4 bus 5
> > athn1 at pci5 dev 0 function 0 "Atheros AR9281" rev 0x01: apic 8 int 17
> > athn1: AR9280 rev 2 (2T2R), ROM rev 22, address
> > ehci1 at pci0 dev 29 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 23
> > usb1 at ehci1: USB revision 2.0
> > uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> > pcib0 at pci0 dev 31 function 0 "Intel H81 LPC" rev 0x05
> > ahci0 at pci0 dev 31 function 2 "Intel 8 Series AHCI" rev 0x05: msi, AHCI 1.3
> > ahci0: port 4: 3.0Gb/s
> > ahci0: port 5: 3.0Gb/s
> > scsibus1 at ahci0: 32 targets
> > sd0 at scsibus1 targ 4 lun 0: <ATA, KINGSTON SHFS37A, 608A> SCSI3 0/direct fixed \
> >                 naa.50026b726304fb22
> > sd0: 114473MB, 512 bytes/sector, 234441648 sectors, thin
> > sd1 at scsibus1 targ 5 lun 0: <ATA, KINGSTON SHFS37A, 608A> SCSI3 0/direct fixed \
> >                 naa.50026b726304fbe2
> > sd1: 114473MB, 512 bytes/sector, 234441648 sectors, thin
> > ichiic0 at pci0 dev 31 function 3 "Intel 8 Series SMBus" rev 0x05: apic 8 int 18
> > iic0 at ichiic0
> > spdmem0 at iic0 addr 0x50: 2GB DDR3 SDRAM PC3-10600
> > isa0 at pcib0
> > isadma0 at isa0
> > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> > pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> > pckbd0 at pckbc0 (kbd slot)
> > wskbd0 at pckbd0: console keyboard, using wsdisplay0
> > pcppi0 at isa0 port 0x61
> > spkr0 at pcppi0
> > uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev \
> > 2.00/0.05 addr 2 uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate \
> > Matching Hub" rev 2.00/0.05 addr 2 uhidev0 at uhub3 port 2 configuration 1 interface \
> >                 0 "SINO WEALTH USB KEYBOARD" rev 1.10/1.00 addr 3
> > uhidev0: iclass 3/1
> > ukbd0 at uhidev0: 8 variable keys, 6 key codes
> > wskbd1 at ukbd0 mux 1
> > wskbd1: connecting to wsdisplay0
> > uhidev1 at uhub3 port 2 configuration 1 interface 1 "SINO WEALTH USB KEYBOARD" rev \
> >                 1.10/1.00 addr 3
> > uhidev1: iclass 3/0, 5 report ids
> > uhid0 at uhidev1 reportid 2: input=1, output=0, feature=0
> > uhid1 at uhidev1 reportid 3: input=3, output=0, feature=0
> > uhid2 at uhidev1 reportid 5: input=0, output=0, feature=5
> > uhidev2 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> >                 Back-UPS XS 1400U  FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 4
> > uhidev2: iclass 3/0, 123 report ids
> > upd0 at uhidev2
> > vscsi0 at root
> > scsibus2 at vscsi0: 256 targets
> > softraid0 at root
> > scsibus3 at softraid0: 256 targets
> > sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006> SCSI2 0/direct fixed
> > sd2: 114470MB, 512 bytes/sector, 234435953 sectors
> > root on sd2a (59f8b4a610bc10ff.a) swap on sd2b dump on sd2b
> > pppoe: GENERIC ERROR: RP-PPPoE: Child pppd process terminated
> > wskbd1: disconnecting from wsdisplay0
> > wskbd1 detached
> > ukbd0 detached
> > uhidev0 detached
> > uhid0 detached
> > uhid1 detached
> > uhid2 detached
> > uhidev1 detached
> > pppoe0: received unexpected PADO
> > pppoe0: received unexpected PADO
> > pppoe0: received unexpected PADO
> > upd0 detached
> > uhidev2 detached
> > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> >                 Back-UPS XS 1400U  FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > uhidev0: iclass 3/0, 123 report ids
> > upd0 at uhidev0
> > upd0 detached
> > uhidev0 detached
> > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> >                 Back-UPS XS 1400U  FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > uhidev0: iclass 3/0, 123 report ids
> > upd0 at uhidev0
> > upd0 detached
> > uhidev0 detached
> > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> >                 Back-UPS XS 1400U  FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > uhidev0: iclass 3/0, 123 report ids
> > upd0 at uhidev0
> > upd0 detached
> > uhidev0 detached
> > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> >                 Back-UPS XS 1400U  FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > uhidev0: iclass 3/0, 123 report ids
> > upd0 at uhidev0
> > upd0 detached
> > uhidev0 detached
> > uhub3: device problem, disabling port 3
> > uhub3: port error, restarting port 3
> > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> >                 Back-UPS XS 1400U  FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > uhidev0: iclass 3/0, 123 report ids
> > upd0 at uhidev0
> >
> >
> >
> > [1] - https://marc.info/?l=openbsd-misc&m=155532485003397&w=2
>
> -- 
> Lévai, Dániel
>

--
:wq Claudio

Reply | Threaded
Open this post in threaded view
|

Re: Leaking mbufs on 6.4 and 6.5

Lévai, Dániel
Thank you Claudio for taking the time to consider this!


Really appreciate the tip on the MAC filters in the bridge(4) -- I'll get to that in a second.

First of all, it seems that 6.5 exhibits a tad bit different behaviour then 6.4, and as I concluded I will not be able to reproduce or deduct the exact same thing(s), I just went ahead and retried everything with 6.5. More specifically, 6.5 seems to still leak mbufs without re0 and vether0 in the setup, but with a disabled re0 and a removed vether0, the leak became *much* slower.

Nevertheless, after removing re0 from the setup (disabled in the BIOS, nothing less) and removing the usage of vether0, mbufs for size 256 and 2048 were still growing slowly but steadily. And that's when I issued an `ifconfig bridge0 flushrule athn[01]'. And lo and behold, it's as if the leak had stopped. Still need to test this over more time but while without the block rules both (256, 2048) sized mbufs seemed to stagnate over a 10 minute period, after I reintroduced the block rules, over the same 10 minute period both (256, 2048) mbufs counter increased by ~900.

Now I'll try to bring back vether0 into the equation and see if that makes a difference without the bridge(4) block rules. I won't test re0 further, I'll just leave that disabled, because I still have a spare port in the 4 port Intel NIC -- so why bother..


Thanks again Claudio!!

Dani


-- 
Lévai, Dániel

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, 2 May 2019 11:32, Claudio Jeker <[hidden email]> wrote:

> On Thu, May 02, 2019 at 07:48:30AM +0000, Lévai, Dániel wrote:
>

> > Hello again,
> > I've upgraded to 6.5 and if possible it has gotten worse. Now it gets
> > unresponsive after ~half a day. With 6.4 I could get away with one
> > reboot per day but it seems I'm going to need two or more with 6.5.
> > Does anyone have any suggestions or suspicions? Anyone else noticed this
> > or similar throughout the previous two releases? I know a lot of changes
> > went into and are underway in the network stack re multi-threading
> > etc... is this maybe something related to this leak? I'm trying not to
> > push unrelated or unnecessary buttons until there's a good candidate for
> > the source of this problem.
>

> Can you try to identify which of the many bits in your setup cause the
> mbuf leak? Easiest is to use a second system to split some of the
> functionality out. Try to remove vether(4) or the pppoe(4) part from your
> setup and check if this changes the behaviour. Also try to remove the MAC
> filters on bridge0 and see if that changes behavior.
> The leaked mbufs are from the system, re(4) or vether(4). Neither em(4)
> nor athn(4) use 2048 byte buffers on the RX side.
>

> There is something special in your setup and you need to help us identify
> what it is. I know many people run similar setups without any troubles.
>

> > Thanks in advance,
> > Daniel
> >

> > > Hi!
> > > I have mbufs leaking after introducing a network topology change on a
> > > router:
> > > After a reboot + 20 minutes:
> > > 1 users Load 0.46 0.38 0.33 16:40:43
> > > IFACE LIVELOCKS SIZE ALIVE LWM HWM CWM
> > > System 0 256 4100 260
> > > 2048 3717 468
> > > 2112 53 7
> > > 4096 128 20
> > > 9216 17 12
> > > After one day:
> > > 1 users Load 0.53 0.40 0.32 16:14:53
> > > IFACE LIVELOCKS SIZE ALIVE LWM HWM CWM
> > > System 0 256 131K 8456
> > > 2048 * 16022
> > > 2112 69 23
> > > 4096 128 21
> > > 9216 17 19
> > > After two days:
> > > 1 users Load 0.14 0.20 0.22 15:00:00
> > > IFACE LIVELOCKS SIZE ALIVE LWM HWM CWM
> > > System 0 256 215K 13780
> > > 2048 * 26954
> > > 2112 66 9
> > > 4096 128 25
> > > 9216 11 11
> > > I think this is either caused by starting to use an on-board re(4)
> > > device or starting to use vether(4) in a bridge(4) (dmesg, details
> > > below).
> > > In any case, 1 gigs worth of memory is consumed in about two days (give
> > > or take a couple of hours) with mild internet traffic (pppoe connected
> > > to re(4)).
> > > I was given the advice to inquire/report here, maybe someone can help
> > > track down the root cause.
> > > Any help is appreciated!
> > > This is my original e-mail [1] sent to misc@ (minus the writing about
> > > the processes running), it has a bunch of information, if someone's
> > > interested:
> > > <TL;DR>
> > > After a recent network configuration change (added re(4), vether(4)) I'm
> > > experiencing this memory leak from somewhere.
> > > How can I check/query how much memory the kernel (or parts of it) is
> > > using over time, besides running top(1) with system processes shown --
> > > I'm also staring at systat(1)'s `malloc' and`pool' views but I'm not
> > > really sure what I'm (or rather what I should be) looking at.
> > > </TL;DR>
> > > I have an on-board Realtek NIC and an add-on Intel PCIe 4 port NIC.
> > > re0 at pci3 dev 0 function 0 "Realtek 8168" rev 0x0c: RTL8168G/8111G (0x4c00), msi
> > > em0 at pci1 dev 0 function 0 "Intel I350" rev 0x01: msi
> > > em1 at pci1 dev 0 function 1 "Intel I350" rev 0x01: msi
> > > em2 at pci1 dev 0 function 2 "Intel I350" rev 0x01: msi
> > > em3 at pci1 dev 0 function 3 "Intel I350" rev 0x01: msi
> > > Before I started to experience the leak I had the on-board NIC disabled
> > > in BIOS and only used two ports of the Intel NIC. My setup looked like
> > > this:
> > > == Network
> > > ISP --- pppoe0 <-> em0 -- OpenBSD 6.4 -- em1(dhcpd) --- [switch] LAN
> > > `- athn0 --- LAN`- athn1 --- LAN
> > > `- bridge0(em1, athn[01])
> > > Internal IP configured on em1.
> > > athn? devices don't have IPs, just bridged to the internal network.
> > > dhcpd(8) is running on em1.
> > > Now I started to use re0 and one more NIC on the Intel card:
> > > ISP --- pppoe0 <-> re0 -- OpenBSD 6.4 -- em0 --- LAN
> > > `- em1 --- LAN`- em2 --- LAN
> > > `- athn0 --- LAN`- athn1 --- LAN
> > > `- vether0 (dhcpd)`- bridge0(em[123], athn[01], vether0)
> > > Now the 4 port Intel NIC basically acts as a switch, the em? interfaces
> > > don't have IP addresses configured, and the athn? interfaces don't have
> > > one either.
> > > I've added a vether(4) interface with an internal IP (same as the em1
> > > have had before) to the bridge, and dhcpd(8) (and everything else that
> > > needs it) now uses that interface.
> > >

> > > ifconfig bridge0
> > >

> > > =================
> > >

> > > bridge0: flags=41<UP,RUNNING>
> > > index 10 llprio 3
> > > groups: bridge
> > > priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp
> > > designated: id 00:00:00:00:00:00 priority 0
> > >

> > >         athn0 flags=3<LEARNING,DISCOVER>
> > >                 port 6 ifpriority 0 ifcost 0
> > >    

> > >

> > > [bunch of `pass/block in/out on athn0 src/dst <MAC>' rules]
> > >

> > >         athn1 flags=3<LEARNING,DISCOVER>
> > >                 port 7 ifpriority 0 ifcost 0
> > >    

> > >

> > > [bunch of `pass/block in/out on athn1 src/dst <MAC>' rules]
> > >

> > >         em0 flags=3<LEARNING,DISCOVER>
> > >                 port 1 ifpriority 0 ifcost 0
> > >         em1 flags=3<LEARNING,DISCOVER>
> > >                 port 2 ifpriority 0 ifcost 0
> > >         em2 flags=3<LEARNING,DISCOVER>
> > >                 port 3 ifpriority 0 ifcost 0
> > >         em3 flags=3<LEARNING,DISCOVER>
> > >                 port 4 ifpriority 0 ifcost 0
> > >         vether0 flags=3<LEARNING,DISCOVER>
> > >                 port 13 ifpriority 0 ifcost 0
> > >         Addresses (max cache: 100, timeout: 240):
> > >    

> > >

> > > [...]
> > >

> > > ifconfig vether0
> > >

> > > =================
> > >

> > > vether0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
> > > lladdr fe:e1:ba:d0:58:44
> > > description: Internal LAN
> > > index 13 priority 0 llprio 3
> > > groups: vether
> > > media: Ethernet autoselect
> > > status: active
> > > inet 192.168.0.1 netmask 0xffff0000 broadcast 192.168.255.255
> > > inet6 <> prefixlen 64 scopeid 0xd
> > > inet6 <> prefixlen 64 pltime 573642 vltime 573642
> > >

> > > ifconfig em
> > >

> > > ============
> > >

> > > em0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
> > > lladdr
> > > description: Internal LAN - Room1
> > > index 1 priority 0 llprio 3
> > > media: Ethernet autoselect (100baseTX full-duplex,rxpause,txpause)
> > > status: active
> > > em1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
> > > lladdr
> > > description: Internal LAN - Room2
> > > index 2 priority 0 llprio 3
> > > media: Ethernet autoselect (1000baseT full-duplex,master,rxpause,txpause)
> > > status: active
> > > em2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
> > > lladdr
> > > description: Internal LAN - Room3
> > > index 3 priority 0 llprio 3
> > > media: Ethernet autoselect (100baseTX full-duplex,rxpause,txpause)
> > > status: active
> > > em3: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> mtu 1500
> > > lladdr
> > > index 4 priority 0 llprio 3
> > > media: Ethernet autoselect (none)
> > > status: no carrier
> > > Nothing fancy, just basically a "software switch" for connecting
> > > separate rooms in the house and adding a vether(4) interface for
> > > conveniency.
> > > Also, everything else remain unchanged, i.e.: no additional services
> > > running, no new software has been installed, no introduction of new pf
> > > rules, other than replacing the interface names of course.
> > > I'm using /etc/malloc.conf@ -> S on all my machines but never had any
> > > problems with it (and still don't anywhere else).
> > > == pf(4)
> > > pf(4) is used on the box, and the rules that have been changed are the
> > > ones related to the interfaces mentioned above:
> > > set skip on lo0
> > > set skip on bridge0
> > > pass on vether0 all flags S/SA allow-opts
> > > pass on em0 all flags S/SA allow-opts
> > > pass on em1 all flags S/SA allow-opts
> > > pass on em2 all flags S/SA allow-opts
> > > pass on em3 all flags S/SA allow-opts
> > > pass on athn0 all flags S/SA allow-opts
> > > pass on athn1 all flags S/SA allow-opts
> > > match in on vether0 inet proto tcp from 192.168.0.0/16 port = <port> to any set \
> > > (prio(2, 6), queue(ssh_bulk, ssh_rt)) block return on vether0 proto tcp from any to \
> > > any port = 25 block return on em0 proto tcp from any to any port = 25
> > > block return on em1 proto tcp from any to any port = 25
> > > block return on em2 proto tcp from any to any port = 25
> > > block return on em3 proto tcp from any to any port = 25
> > > block return on athn0 proto tcp from any to any port = 25
> > > block return on athn1 proto tcp from any to any port = 25
> > > Previously I've had `skip' on athn? as well.
> > > (I can upload the complete rule set somewhere if it is of interest)
> > > == dmesg
> > > OpenBSD 6.4 (GENERIC.MP) #9: Tue Mar 26 19:21:43 CET 2019
> > > [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> > > real mem = 2059997184 (1964MB)
> > > avail mem = 1988288512 (1896MB)
> > > mpath0 at root
> > > scsibus0 at mpath0: 256 targets
> > > mainbus0 at root
> > > bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xec130 (75 entries)
> > > bios0: vendor American Megatrends Inc. version "F2" date 08/06/2015
> > > bios0: Gigabyte Technology Co., Ltd. H81M-DS2
> > > acpi0 at bios0: rev 2
> > > acpi0: sleep states S0 S3 S4 S5
> > > acpi0: tables DSDT FACP APIC FPDT SSDT SSDT MCFG HPET SSDT SSDT
> > > acpi0: wakeup devices PXSX(S4) RP01(S4) PXSX(S4) PXSX(S4) RP03(S4) PXSX(S4) PXSX(S4) \
> > > RP05(S4) PXSX(S4) RP06(S4) PXSX(S4) PXSX(S4) GLAN(S4) EHC1(S4) EHC2(S4) XHC_(S4) \
> > > [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits
> > > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> > > cpu0 at mainbus0: apid 0 (boot processor)
> > > cpu0: Intel(R) Celeron(R) CPU G1840 @ 2.80GHz, 2793.95 MHz, 06-3c-03
> > > cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,D \
> > > S,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSS \
> > > E3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,XSAVE,RDRAND,NXE,PAGE1 \
> > > GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,ERMS,INVPCID,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> > > cpu0: 256KB 64b/line 8-way L2 cache
> > > cpu0: smt 0, core 0, package 0
> > > mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> > > cpu0: apic clock running at 99MHz
> > > cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
> > > cpu1 at mainbus0: apid 2 (application processor)
> > > cpu1: Intel(R) Celeron(R) CPU G1840 @ 2.80GHz, 2793.55 MHz, 06-3c-03
> > > cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,D \
> > > S,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSS \
> > > E3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,XSAVE,RDRAND,NXE,PAGE1 \
> > > GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,ERMS,INVPCID,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> > > cpu1: 256KB 64b/line 8-way L2 cache
> > > cpu1: smt 0, core 1, package 0
> > > ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
> > > acpimcfg0 at acpi0
> > > acpimcfg0: addr 0xf8000000, bus 0-63
> > > acpihpet0 at acpi0: 14318179 Hz
> > > acpiprt0 at acpi0: bus 0 (PCI0)
> > > acpiprt1 at acpi0: bus 2 (RP01)
> > > acpiprt2 at acpi0: bus 3 (RP03)
> > > acpiprt3 at acpi0: bus 4 (RP05)
> > > acpiprt4 at acpi0: bus 5 (RP06)
> > > acpiprt5 at acpi0: bus 1 (PEG0)
> > > acpiprt6 at acpi0: bus -1 (PEG1)
> > > acpiprt7 at acpi0: bus -1 (PEG2)
> > > acpiec0 at acpi0: not present
> > > acpicpu0 at acpi0: C2(500@67 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> > > acpicpu1 at acpi0: C2(500@67 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> > > acpipwrres0 at acpi0: FN00, resource for FAN0
> > > acpipwrres1 at acpi0: FN01, resource for FAN1
> > > acpipwrres2 at acpi0: FN02, resource for FAN2
> > > acpipwrres3 at acpi0: FN03, resource for FAN3
> > > acpipwrres4 at acpi0: FN04, resource for FAN4
> > > acpitz0 at acpi0: critical temperature is 105 degC
> > > acpitz1 at acpi0: critical temperature is 105 degC
> > > acpicmos0 at acpi0
> > > acpibtn0 at acpi0: PWRB
> > > "PNP0C0B" at acpi0 not configured
> > > "PNP0C0B" at acpi0 not configured
> > > "PNP0C0B" at acpi0 not configured
> > > "PNP0C0B" at acpi0 not configured
> > > "PNP0C0B" at acpi0 not configured
> > > acpivideo0 at acpi0: GFX0
> > > acpivout0 at acpivideo0: DD1F
> > > cpu0: Enhanced SpeedStep 2793 MHz: speeds: 2800, 2700, 2500, 2400, 2300, 2100, 2000, \
> > > 1900, 1700, 1600, 1500, 1300, 1200, 1100, 900, 800 MHz pci0 at mainbus0 bus 0
> > > pchb0 at pci0 dev 0 function 0 "Intel Core 4G Host" rev 0x06
> > > ppb0 at pci0 dev 1 function 0 "Intel Core 4G PCIE" rev 0x06: msi
> > > pci1 at ppb0 bus 1
> > > em0 at pci1 dev 0 function 0 "Intel I350" rev 0x01: msi, address
> > > em1 at pci1 dev 0 function 1 "Intel I350" rev 0x01: msi, address
> > > em2 at pci1 dev 0 function 2 "Intel I350" rev 0x01: msi, address
> > > em3 at pci1 dev 0 function 3 "Intel I350" rev 0x01: msi, address
> > > inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics" rev 0x06
> > > drm0 at inteldrm0
> > > inteldrm0: msi
> > > error: [drm:pid0:drm_edid_block_valid] ERROR EDID checksum is invalid, remainder is \
> > > 39 Raw EDID:
> > > 00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> > > fe 11 01 03 08 25 1e 78 ca bb 05 a3 56 47 99 24
> > > f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > error: [drm:pid0:drm_edid_block_valid] ERROR EDID checksum is invalid, remainder is \
> > > 7 Raw EDID:
> > > 00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> > > de 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
> > > d4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > error: [drm:pid0:drm_edid_block_valid] ERROR EDID checksum is invalid, remainder is \
> > > 39 Raw EDID:
> > > 00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> > > de 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
> > > f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > error: [drm:pid0:drm_edid_block_valid] ERROR EDID checksum is invalid, remainder is \
> > > 71 Raw EDID:
> > > 00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> > > fe 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
> > > f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > drm:pid0:drm_do_get_edid WARNING VGA-: EDID block 0 invalid.
> > > inteldrm0: 1024x768, 32bpp
> > > wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
> > > wsdisplay0: screen 1-5 added (std, vt100 emulation)
> > > azalia0 at pci0 dev 3 function 0 "Intel Core 4G HD Audio" rev 0x06: msi
> > > azalia0: No codecs found
> > > "Intel 8 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured
> > > ehci0 at pci0 dev 26 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 16
> > > usb0 at ehci0: USB revision 2.0
> > > uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> > > ppb1 at pci0 dev 28 function 0 "Intel 8 Series PCIE" rev 0xd5: msi
> > > pci2 at ppb1 bus 2
> > > ppb2 at pci0 dev 28 function 2 "Intel 8 Series PCIE" rev 0xd5: msi
> > > pci3 at ppb2 bus 3
> > > re0 at pci3 dev 0 function 0 "Realtek 8168" rev 0x0c: RTL8168G/8111G (0x4c00), msi, \
> > > address rgephy0 at re0 phy 7: RTL8251 PHY, rev. 0
> > > ppb3 at pci0 dev 28 function 4 "Intel 8 Series PCIE" rev 0xd5: msi
> > > pci4 at ppb3 bus 4
> > > athn0 at pci4 dev 0 function 0 "Atheros AR9281" rev 0x01: apic 8 int 16
> > > athn0: AR9280 rev 2 (2T2R), ROM rev 16, address
> > > ppb4 at pci0 dev 28 function 5 "Intel 8 Series PCIE" rev 0xd5: msi
> > > pci5 at ppb4 bus 5
> > > athn1 at pci5 dev 0 function 0 "Atheros AR9281" rev 0x01: apic 8 int 17
> > > athn1: AR9280 rev 2 (2T2R), ROM rev 22, address
> > > ehci1 at pci0 dev 29 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 23
> > > usb1 at ehci1: USB revision 2.0
> > > uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> > > pcib0 at pci0 dev 31 function 0 "Intel H81 LPC" rev 0x05
> > > ahci0 at pci0 dev 31 function 2 "Intel 8 Series AHCI" rev 0x05: msi, AHCI 1.3
> > > ahci0: port 4: 3.0Gb/s
> > > ahci0: port 5: 3.0Gb/s
> > > scsibus1 at ahci0: 32 targets
> > > sd0 at scsibus1 targ 4 lun 0: <ATA, KINGSTON SHFS37A, 608A> SCSI3 0/direct fixed \
> > > naa.50026b726304fb22
> > > sd0: 114473MB, 512 bytes/sector, 234441648 sectors, thin
> > > sd1 at scsibus1 targ 5 lun 0: <ATA, KINGSTON SHFS37A, 608A> SCSI3 0/direct fixed \
> > > naa.50026b726304fbe2
> > > sd1: 114473MB, 512 bytes/sector, 234441648 sectors, thin
> > > ichiic0 at pci0 dev 31 function 3 "Intel 8 Series SMBus" rev 0x05: apic 8 int 18
> > > iic0 at ichiic0
> > > spdmem0 at iic0 addr 0x50: 2GB DDR3 SDRAM PC3-10600
> > > isa0 at pcib0
> > > isadma0 at isa0
> > > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> > > pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> > > pckbd0 at pckbc0 (kbd slot)
> > > wskbd0 at pckbd0: console keyboard, using wsdisplay0
> > > pcppi0 at isa0 port 0x61
> > > spkr0 at pcppi0
> > > uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev \
> > > 2.00/0.05 addr 2 uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate \
> > > Matching Hub" rev 2.00/0.05 addr 2 uhidev0 at uhub3 port 2 configuration 1 interface \
> > > 0 "SINO WEALTH USB KEYBOARD" rev 1.10/1.00 addr 3
> > > uhidev0: iclass 3/1
> > > ukbd0 at uhidev0: 8 variable keys, 6 key codes
> > > wskbd1 at ukbd0 mux 1
> > > wskbd1: connecting to wsdisplay0
> > > uhidev1 at uhub3 port 2 configuration 1 interface 1 "SINO WEALTH USB KEYBOARD" rev \
> > > 1.10/1.00 addr 3
> > > uhidev1: iclass 3/0, 5 report ids
> > > uhid0 at uhidev1 reportid 2: input=1, output=0, feature=0
> > > uhid1 at uhidev1 reportid 3: input=3, output=0, feature=0
> > > uhid2 at uhidev1 reportid 5: input=0, output=0, feature=5
> > > uhidev2 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 4
> > > uhidev2: iclass 3/0, 123 report ids
> > > upd0 at uhidev2
> > > vscsi0 at root
> > > scsibus2 at vscsi0: 256 targets
> > > softraid0 at root
> > > scsibus3 at softraid0: 256 targets
> > > sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006> SCSI2 0/direct fixed
> > > sd2: 114470MB, 512 bytes/sector, 234435953 sectors
> > > root on sd2a (59f8b4a610bc10ff.a) swap on sd2b dump on sd2b
> > > pppoe: GENERIC ERROR: RP-PPPoE: Child pppd process terminated
> > > wskbd1: disconnecting from wsdisplay0
> > > wskbd1 detached
> > > ukbd0 detached
> > > uhidev0 detached
> > > uhid0 detached
> > > uhid1 detached
> > > uhid2 detached
> > > uhidev1 detached
> > > pppoe0: received unexpected PADO
> > > pppoe0: received unexpected PADO
> > > pppoe0: received unexpected PADO
> > > upd0 detached
> > > uhidev2 detached
> > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > uhidev0: iclass 3/0, 123 report ids
> > > upd0 at uhidev0
> > > upd0 detached
> > > uhidev0 detached
> > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > uhidev0: iclass 3/0, 123 report ids
> > > upd0 at uhidev0
> > > upd0 detached
> > > uhidev0 detached
> > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > uhidev0: iclass 3/0, 123 report ids
> > > upd0 at uhidev0
> > > upd0 detached
> > > uhidev0 detached
> > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > uhidev0: iclass 3/0, 123 report ids
> > > upd0 at uhidev0
> > > upd0 detached
> > > uhidev0 detached
> > > uhub3: device problem, disabling port 3
> > > uhub3: port error, restarting port 3
> > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > uhidev0: iclass 3/0, 123 report ids
> > > upd0 at uhidev0
> > > [1] - https://marc.info/?l=openbsd-misc&m=155532485003397&w=2
> >

> > -- 
> > Lévai, Dániel
>

> --
>

> :wq Claudio


publickey - leva@ecentrum.hu - 0x66E1F716.asc (904 bytes) Download Attachment
signature.asc (259 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Leaking mbufs on 6.4 and 6.5

Lévai, Dániel
So after about 8 hours running and reintroducing vether(4) into the setup and the bridge(4), the 2048 sized mbufs remained exactly 284, with an occasional bump here and there, but always reset back to that.
The 256 sized ones grew by about a hundred over this ~8 hour period, I'm not sure if that's normal, even though anywhere else I have at most a couple of hundred of them -- with different types of NICs, however.

   1 users Load 0.09 0.08 0.08

IFACE             LIVELOCKS  SIZE ALIVE   LWM   HWM   CWM
System                    0   256   763          94
                             2048   284          61
                             2112   178          62
                             4096   128          21
lo0
em0                          2050    31    10   256    31
em1                          2050    35    10   256    35
em2                          2050    27    10   256    27
em3                          2050    85    10   256    85
athn0
athn1
enc0
bridge0
pflog1
pppoe0
vether0



-- 
Lévai, Dániel

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, 2 May 2019 13:54, Lévai, Dániel <[hidden email]> wrote:

> Thank you Claudio for taking the time to consider this!
>
> Really appreciate the tip on the MAC filters in the bridge(4) -- I'll get to that in a second.
>
> First of all, it seems that 6.5 exhibits a tad bit different behaviour then 6.4, and as I concluded I will not be able to reproduce or deduct the exact same thing(s), I just went ahead and retried everything with 6.5. More specifically, 6.5 seems to still leak mbufs without re0 and vether0 in the setup, but with a disabled re0 and a removed vether0, the leak became much slower.
>
> Nevertheless, after removing re0 from the setup (disabled in the BIOS, nothing less) and removing the usage of vether0, mbufs for size 256 and 2048 were still growing slowly but steadily. And that's when I issued an `ifconfig bridge0 flushrule athn[01]'. And lo and behold, it's as if the leak had stopped. Still need to test this over more time but while without the block rules both (256, 2048) sized mbufs seemed to stagnate over a 10 minute period, after I reintroduced the block rules, over the same 10 minute period both (256, 2048) mbufs counter increased by ~900.
>
> Now I'll try to bring back vether0 into the equation and see if that makes a difference without the bridge(4) block rules. I won't test re0 further, I'll just leave that disabled, because I still have a spare port in the 4 port Intel NIC -- so why bother..
>
> Thanks again Claudio!!
>
> Dani
>
> -- 
> Lévai, Dániel
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, 2 May 2019 11:32, Claudio Jeker [hidden email] wrote:
>
> > On Thu, May 02, 2019 at 07:48:30AM +0000, Lévai, Dániel wrote:
>
> > > Hello again,
> > > I've upgraded to 6.5 and if possible it has gotten worse. Now it gets
> > > unresponsive after ~half a day. With 6.4 I could get away with one
> > > reboot per day but it seems I'm going to need two or more with 6.5.
> > > Does anyone have any suggestions or suspicions? Anyone else noticed this
> > > or similar throughout the previous two releases? I know a lot of changes
> > > went into and are underway in the network stack re multi-threading
> > > etc... is this maybe something related to this leak? I'm trying not to
> > > push unrelated or unnecessary buttons until there's a good candidate for
> > > the source of this problem.
>
> > Can you try to identify which of the many bits in your setup cause the
> > mbuf leak? Easiest is to use a second system to split some of the
> > functionality out. Try to remove vether(4) or the pppoe(4) part from your
> > setup and check if this changes the behaviour. Also try to remove the MAC
> > filters on bridge0 and see if that changes behavior.
> > The leaked mbufs are from the system, re(4) or vether(4). Neither em(4)
> > nor athn(4) use 2048 byte buffers on the RX side.
>
> > There is something special in your setup and you need to help us identify
> > what it is. I know many people run similar setups without any troubles.
>
> > > Thanks in advance,
> > > Daniel
>
> > > > Hi!
> > > > I have mbufs leaking after introducing a network topology change on a
> > > > router:
> > > > After a reboot + 20 minutes:
> > > > 1 users Load 0.46 0.38 0.33 16:40:43
> > > > IFACE LIVELOCKS SIZE ALIVE LWM HWM CWM
> > > > System 0 256 4100 260
> > > > 2048 3717 468
> > > > 2112 53 7
> > > > 4096 128 20
> > > > 9216 17 12
> > > > After one day:
> > > > 1 users Load 0.53 0.40 0.32 16:14:53
> > > > IFACE LIVELOCKS SIZE ALIVE LWM HWM CWM
> > > > System 0 256 131K 8456
> > > > 2048 * 16022
> > > > 2112 69 23
> > > > 4096 128 21
> > > > 9216 17 19
> > > > After two days:
> > > > 1 users Load 0.14 0.20 0.22 15:00:00
> > > > IFACE LIVELOCKS SIZE ALIVE LWM HWM CWM
> > > > System 0 256 215K 13780
> > > > 2048 * 26954
> > > > 2112 66 9
> > > > 4096 128 25
> > > > 9216 11 11
> > > > I think this is either caused by starting to use an on-board re(4)
> > > > device or starting to use vether(4) in a bridge(4) (dmesg, details
> > > > below).
> > > > In any case, 1 gigs worth of memory is consumed in about two days (give
> > > > or take a couple of hours) with mild internet traffic (pppoe connected
> > > > to re(4)).
> > > > I was given the advice to inquire/report here, maybe someone can help
> > > > track down the root cause.
> > > > Any help is appreciated!
> > > > This is my original e-mail [1] sent to misc@ (minus the writing about
> > > > the processes running), it has a bunch of information, if someone's
> > > > interested:
> > > > <TL;DR>
> > > > After a recent network configuration change (added re(4), vether(4)) I'm
> > > > experiencing this memory leak from somewhere.
> > > > How can I check/query how much memory the kernel (or parts of it) is
> > > > using over time, besides running top(1) with system processes shown --
> > > > I'm also staring at systat(1)'s `malloc' and`pool' views but I'm not
> > > > really sure what I'm (or rather what I should be) looking at.
> > > > </TL;DR>
> > > > I have an on-board Realtek NIC and an add-on Intel PCIe 4 port NIC.
> > > > re0 at pci3 dev 0 function 0 "Realtek 8168" rev 0x0c: RTL8168G/8111G (0x4c00), msi
> > > > em0 at pci1 dev 0 function 0 "Intel I350" rev 0x01: msi
> > > > em1 at pci1 dev 0 function 1 "Intel I350" rev 0x01: msi
> > > > em2 at pci1 dev 0 function 2 "Intel I350" rev 0x01: msi
> > > > em3 at pci1 dev 0 function 3 "Intel I350" rev 0x01: msi
> > > > Before I started to experience the leak I had the on-board NIC disabled
> > > > in BIOS and only used two ports of the Intel NIC. My setup looked like
> > > > this:
> > > > == Network
> > > > ISP --- pppoe0 <-> em0 -- OpenBSD 6.4 -- em1(dhcpd) --- [switch] LAN
> > > > `- athn0 --- LAN`- athn1 --- LAN
> > > > `- bridge0(em1, athn[01]) Internal IP configured on em1. athn? devices don't have IPs, just bridged to the internal network. dhcpd(8) is running on em1. Now I started to use re0 and one more NIC on the Intel card: ISP --- pppoe0 <-> re0 -- OpenBSD 6.4 -- em0 --- LAN`- em1 --- LAN`- em2 --- LAN`- athn0 --- LAN`- athn1 --- LAN`- vether0 (dhcpd)`- bridge0(em[123], athn[01], vether0)
> > > > Now the 4 port Intel NIC basically acts as a switch, the em? interfaces
> > > > don't have IP addresses configured, and the athn? interfaces don't have
> > > > one either.
> > > > I've added a vether(4) interface with an internal IP (same as the em1
> > > > have had before) to the bridge, and dhcpd(8) (and everything else that
> > > > needs it) now uses that interface.
>
> > > > ifconfig bridge0
>
> > > > =================
>
> > > > bridge0: flags=41<UP,RUNNING>
> > > > index 10 llprio 3
> > > > groups: bridge
> > > > priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp
> > > > designated: id 00:00:00:00:00:00 priority 0
>
> > > >         athn0 flags=3<LEARNING,DISCOVER>
> > > >                 port 6 ifpriority 0 ifcost 0
> > > >
>
> > > >
>
> > > > [bunch of `pass/block in/out on athn0 src/dst <MAC>' rules]
>
> > > >         athn1 flags=3<LEARNING,DISCOVER>
> > > >                 port 7 ifpriority 0 ifcost 0
> > > >
>
> > > >
>
> > > > [bunch of `pass/block in/out on athn1 src/dst <MAC>' rules]
>
> > > >         em0 flags=3<LEARNING,DISCOVER>
> > > >                 port 1 ifpriority 0 ifcost 0
> > > >         em1 flags=3<LEARNING,DISCOVER>
> > > >                 port 2 ifpriority 0 ifcost 0
> > > >         em2 flags=3<LEARNING,DISCOVER>
> > > >                 port 3 ifpriority 0 ifcost 0
> > > >         em3 flags=3<LEARNING,DISCOVER>
> > > >                 port 4 ifpriority 0 ifcost 0
> > > >         vether0 flags=3<LEARNING,DISCOVER>
> > > >                 port 13 ifpriority 0 ifcost 0
> > > >         Addresses (max cache: 100, timeout: 240):
> > > >
>
> > > >
>
> > > > [...]
>
> > > > ifconfig vether0
>
> > > > =================
>
> > > > vether0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
> > > > lladdr fe:e1:ba:d0:58:44
> > > > description: Internal LAN
> > > > index 13 priority 0 llprio 3
> > > > groups: vether
> > > > media: Ethernet autoselect
> > > > status: active
> > > > inet 192.168.0.1 netmask 0xffff0000 broadcast 192.168.255.255
> > > > inet6 <> prefixlen 64 scopeid 0xd
> > > > inet6 <> prefixlen 64 pltime 573642 vltime 573642
>
> > > > ifconfig em
>
> > > > ============
>
> > > > em0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
> > > > lladdr
> > > > description: Internal LAN - Room1
> > > > index 1 priority 0 llprio 3
> > > > media: Ethernet autoselect (100baseTX full-duplex,rxpause,txpause)
> > > > status: active
> > > > em1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
> > > > lladdr
> > > > description: Internal LAN - Room2
> > > > index 2 priority 0 llprio 3
> > > > media: Ethernet autoselect (1000baseT full-duplex,master,rxpause,txpause)
> > > > status: active
> > > > em2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
> > > > lladdr
> > > > description: Internal LAN - Room3
> > > > index 3 priority 0 llprio 3
> > > > media: Ethernet autoselect (100baseTX full-duplex,rxpause,txpause)
> > > > status: active
> > > > em3: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> mtu 1500
> > > > lladdr
> > > > index 4 priority 0 llprio 3
> > > > media: Ethernet autoselect (none)
> > > > status: no carrier
> > > > Nothing fancy, just basically a "software switch" for connecting
> > > > separate rooms in the house and adding a vether(4) interface for
> > > > conveniency.
> > > > Also, everything else remain unchanged, i.e.: no additional services
> > > > running, no new software has been installed, no introduction of new pf
> > > > rules, other than replacing the interface names of course.
> > > > I'm using /etc/malloc.conf@ -> S on all my machines but never had any
> > > > problems with it (and still don't anywhere else).
> > > > == pf(4)
> > > > pf(4) is used on the box, and the rules that have been changed are the
> > > > ones related to the interfaces mentioned above:
> > > > set skip on lo0
> > > > set skip on bridge0
> > > > pass on vether0 all flags S/SA allow-opts
> > > > pass on em0 all flags S/SA allow-opts
> > > > pass on em1 all flags S/SA allow-opts
> > > > pass on em2 all flags S/SA allow-opts
> > > > pass on em3 all flags S/SA allow-opts
> > > > pass on athn0 all flags S/SA allow-opts
> > > > pass on athn1 all flags S/SA allow-opts
> > > > match in on vether0 inet proto tcp from 192.168.0.0/16 port = <port> to any set \
> > > > (prio(2, 6), queue(ssh_bulk, ssh_rt)) block return on vether0 proto tcp from any to \
> > > > any port = 25 block return on em0 proto tcp from any to any port = 25
> > > > block return on em1 proto tcp from any to any port = 25
> > > > block return on em2 proto tcp from any to any port = 25
> > > > block return on em3 proto tcp from any to any port = 25
> > > > block return on athn0 proto tcp from any to any port = 25
> > > > block return on athn1 proto tcp from any to any port = 25
> > > > Previously I've had `skip' on athn? as well.
> > > > (I can upload the complete rule set somewhere if it is of interest)
> > > > == dmesg
> > > > OpenBSD 6.4 (GENERIC.MP) #9: Tue Mar 26 19:21:43 CET 2019
> > > > [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> > > > real mem = 2059997184 (1964MB)
> > > > avail mem = 1988288512 (1896MB)
> > > > mpath0 at root
> > > > scsibus0 at mpath0: 256 targets
> > > > mainbus0 at root
> > > > bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xec130 (75 entries)
> > > > bios0: vendor American Megatrends Inc. version "F2" date 08/06/2015
> > > > bios0: Gigabyte Technology Co., Ltd. H81M-DS2
> > > > acpi0 at bios0: rev 2
> > > > acpi0: sleep states S0 S3 S4 S5
> > > > acpi0: tables DSDT FACP APIC FPDT SSDT SSDT MCFG HPET SSDT SSDT
> > > > acpi0: wakeup devices PXSX(S4) RP01(S4) PXSX(S4) PXSX(S4) RP03(S4) PXSX(S4) PXSX(S4) \
> > > > RP05(S4) PXSX(S4) RP06(S4) PXSX(S4) PXSX(S4) GLAN(S4) EHC1(S4) EHC2(S4) XHC_(S4) \
> > > > [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits
> > > > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> > > > cpu0 at mainbus0: apid 0 (boot processor)
> > > > cpu0: Intel(R) Celeron(R) CPU G1840 @ 2.80GHz, 2793.95 MHz, 06-3c-03
> > > > cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,D \
> > > > S,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSS \
> > > > E3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,XSAVE,RDRAND,NXE,PAGE1 \
> > > > GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,ERMS,INVPCID,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> > > > cpu0: 256KB 64b/line 8-way L2 cache
> > > > cpu0: smt 0, core 0, package 0
> > > > mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> > > > cpu0: apic clock running at 99MHz
> > > > cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
> > > > cpu1 at mainbus0: apid 2 (application processor)
> > > > cpu1: Intel(R) Celeron(R) CPU G1840 @ 2.80GHz, 2793.55 MHz, 06-3c-03
> > > > cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,D \
> > > > S,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSS \
> > > > E3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,XSAVE,RDRAND,NXE,PAGE1 \
> > > > GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,ERMS,INVPCID,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> > > > cpu1: 256KB 64b/line 8-way L2 cache
> > > > cpu1: smt 0, core 1, package 0
> > > > ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
> > > > acpimcfg0 at acpi0
> > > > acpimcfg0: addr 0xf8000000, bus 0-63
> > > > acpihpet0 at acpi0: 14318179 Hz
> > > > acpiprt0 at acpi0: bus 0 (PCI0)
> > > > acpiprt1 at acpi0: bus 2 (RP01)
> > > > acpiprt2 at acpi0: bus 3 (RP03)
> > > > acpiprt3 at acpi0: bus 4 (RP05)
> > > > acpiprt4 at acpi0: bus 5 (RP06)
> > > > acpiprt5 at acpi0: bus 1 (PEG0)
> > > > acpiprt6 at acpi0: bus -1 (PEG1)
> > > > acpiprt7 at acpi0: bus -1 (PEG2)
> > > > acpiec0 at acpi0: not present
> > > > acpicpu0 at acpi0: C2(500@67 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> > > > acpicpu1 at acpi0: C2(500@67 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> > > > acpipwrres0 at acpi0: FN00, resource for FAN0
> > > > acpipwrres1 at acpi0: FN01, resource for FAN1
> > > > acpipwrres2 at acpi0: FN02, resource for FAN2
> > > > acpipwrres3 at acpi0: FN03, resource for FAN3
> > > > acpipwrres4 at acpi0: FN04, resource for FAN4
> > > > acpitz0 at acpi0: critical temperature is 105 degC
> > > > acpitz1 at acpi0: critical temperature is 105 degC
> > > > acpicmos0 at acpi0
> > > > acpibtn0 at acpi0: PWRB
> > > > "PNP0C0B" at acpi0 not configured
> > > > "PNP0C0B" at acpi0 not configured
> > > > "PNP0C0B" at acpi0 not configured
> > > > "PNP0C0B" at acpi0 not configured
> > > > "PNP0C0B" at acpi0 not configured
> > > > acpivideo0 at acpi0: GFX0
> > > > acpivout0 at acpivideo0: DD1F
> > > > cpu0: Enhanced SpeedStep 2793 MHz: speeds: 2800, 2700, 2500, 2400, 2300, 2100, 2000, \
> > > > 1900, 1700, 1600, 1500, 1300, 1200, 1100, 900, 800 MHz pci0 at mainbus0 bus 0
> > > > pchb0 at pci0 dev 0 function 0 "Intel Core 4G Host" rev 0x06
> > > > ppb0 at pci0 dev 1 function 0 "Intel Core 4G PCIE" rev 0x06: msi
> > > > pci1 at ppb0 bus 1
> > > > em0 at pci1 dev 0 function 0 "Intel I350" rev 0x01: msi, address
> > > > em1 at pci1 dev 0 function 1 "Intel I350" rev 0x01: msi, address
> > > > em2 at pci1 dev 0 function 2 "Intel I350" rev 0x01: msi, address
> > > > em3 at pci1 dev 0 function 3 "Intel I350" rev 0x01: msi, address
> > > > inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics" rev 0x06
> > > > drm0 at inteldrm0
> > > > inteldrm0: msi
> > > > error: [drm:pid0:drm_edid_block_valid] ERROR EDID checksum is invalid, remainder is \
> > > > 39 Raw EDID:
> > > > 00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> > > > fe 11 01 03 08 25 1e 78 ca bb 05 a3 56 47 99 24
> > > > f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > error: [drm:pid0:drm_edid_block_valid] ERROR EDID checksum is invalid, remainder is \
> > > > 7 Raw EDID:
> > > > 00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> > > > de 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
> > > > d4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > error: [drm:pid0:drm_edid_block_valid] ERROR EDID checksum is invalid, remainder is \
> > > > 39 Raw EDID:
> > > > 00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> > > > de 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
> > > > f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > error: [drm:pid0:drm_edid_block_valid] ERROR EDID checksum is invalid, remainder is \
> > > > 71 Raw EDID:
> > > > 00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> > > > fe 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
> > > > f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > drm:pid0:drm_do_get_edid WARNING VGA-: EDID block 0 invalid.
> > > > inteldrm0: 1024x768, 32bpp
> > > > wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
> > > > wsdisplay0: screen 1-5 added (std, vt100 emulation)
> > > > azalia0 at pci0 dev 3 function 0 "Intel Core 4G HD Audio" rev 0x06: msi
> > > > azalia0: No codecs found
> > > > "Intel 8 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured
> > > > ehci0 at pci0 dev 26 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 16
> > > > usb0 at ehci0: USB revision 2.0
> > > > uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> > > > ppb1 at pci0 dev 28 function 0 "Intel 8 Series PCIE" rev 0xd5: msi
> > > > pci2 at ppb1 bus 2
> > > > ppb2 at pci0 dev 28 function 2 "Intel 8 Series PCIE" rev 0xd5: msi
> > > > pci3 at ppb2 bus 3
> > > > re0 at pci3 dev 0 function 0 "Realtek 8168" rev 0x0c: RTL8168G/8111G (0x4c00), msi, \
> > > > address rgephy0 at re0 phy 7: RTL8251 PHY, rev. 0
> > > > ppb3 at pci0 dev 28 function 4 "Intel 8 Series PCIE" rev 0xd5: msi
> > > > pci4 at ppb3 bus 4
> > > > athn0 at pci4 dev 0 function 0 "Atheros AR9281" rev 0x01: apic 8 int 16
> > > > athn0: AR9280 rev 2 (2T2R), ROM rev 16, address
> > > > ppb4 at pci0 dev 28 function 5 "Intel 8 Series PCIE" rev 0xd5: msi
> > > > pci5 at ppb4 bus 5
> > > > athn1 at pci5 dev 0 function 0 "Atheros AR9281" rev 0x01: apic 8 int 17
> > > > athn1: AR9280 rev 2 (2T2R), ROM rev 22, address
> > > > ehci1 at pci0 dev 29 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 23
> > > > usb1 at ehci1: USB revision 2.0
> > > > uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> > > > pcib0 at pci0 dev 31 function 0 "Intel H81 LPC" rev 0x05
> > > > ahci0 at pci0 dev 31 function 2 "Intel 8 Series AHCI" rev 0x05: msi, AHCI 1.3
> > > > ahci0: port 4: 3.0Gb/s
> > > > ahci0: port 5: 3.0Gb/s
> > > > scsibus1 at ahci0: 32 targets
> > > > sd0 at scsibus1 targ 4 lun 0: <ATA, KINGSTON SHFS37A, 608A> SCSI3 0/direct fixed \
> > > > naa.50026b726304fb22
> > > > sd0: 114473MB, 512 bytes/sector, 234441648 sectors, thin
> > > > sd1 at scsibus1 targ 5 lun 0: <ATA, KINGSTON SHFS37A, 608A> SCSI3 0/direct fixed \
> > > > naa.50026b726304fbe2
> > > > sd1: 114473MB, 512 bytes/sector, 234441648 sectors, thin
> > > > ichiic0 at pci0 dev 31 function 3 "Intel 8 Series SMBus" rev 0x05: apic 8 int 18
> > > > iic0 at ichiic0
> > > > spdmem0 at iic0 addr 0x50: 2GB DDR3 SDRAM PC3-10600
> > > > isa0 at pcib0
> > > > isadma0 at isa0
> > > > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> > > > pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> > > > pckbd0 at pckbc0 (kbd slot)
> > > > wskbd0 at pckbd0: console keyboard, using wsdisplay0
> > > > pcppi0 at isa0 port 0x61
> > > > spkr0 at pcppi0
> > > > uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev \
> > > > 2.00/0.05 addr 2 uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate \
> > > > Matching Hub" rev 2.00/0.05 addr 2 uhidev0 at uhub3 port 2 configuration 1 interface \
> > > > 0 "SINO WEALTH USB KEYBOARD" rev 1.10/1.00 addr 3
> > > > uhidev0: iclass 3/1
> > > > ukbd0 at uhidev0: 8 variable keys, 6 key codes
> > > > wskbd1 at ukbd0 mux 1
> > > > wskbd1: connecting to wsdisplay0
> > > > uhidev1 at uhub3 port 2 configuration 1 interface 1 "SINO WEALTH USB KEYBOARD" rev \
> > > > 1.10/1.00 addr 3
> > > > uhidev1: iclass 3/0, 5 report ids
> > > > uhid0 at uhidev1 reportid 2: input=1, output=0, feature=0
> > > > uhid1 at uhidev1 reportid 3: input=3, output=0, feature=0
> > > > uhid2 at uhidev1 reportid 5: input=0, output=0, feature=5
> > > > uhidev2 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 4
> > > > uhidev2: iclass 3/0, 123 report ids
> > > > upd0 at uhidev2
> > > > vscsi0 at root
> > > > scsibus2 at vscsi0: 256 targets
> > > > softraid0 at root
> > > > scsibus3 at softraid0: 256 targets
> > > > sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006> SCSI2 0/direct fixed
> > > > sd2: 114470MB, 512 bytes/sector, 234435953 sectors
> > > > root on sd2a (59f8b4a610bc10ff.a) swap on sd2b dump on sd2b
> > > > pppoe: GENERIC ERROR: RP-PPPoE: Child pppd process terminated
> > > > wskbd1: disconnecting from wsdisplay0
> > > > wskbd1 detached
> > > > ukbd0 detached
> > > > uhidev0 detached
> > > > uhid0 detached
> > > > uhid1 detached
> > > > uhid2 detached
> > > > uhidev1 detached
> > > > pppoe0: received unexpected PADO
> > > > pppoe0: received unexpected PADO
> > > > pppoe0: received unexpected PADO
> > > > upd0 detached
> > > > uhidev2 detached
> > > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > > uhidev0: iclass 3/0, 123 report ids
> > > > upd0 at uhidev0
> > > > upd0 detached
> > > > uhidev0 detached
> > > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > > uhidev0: iclass 3/0, 123 report ids
> > > > upd0 at uhidev0
> > > > upd0 detached
> > > > uhidev0 detached
> > > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > > uhidev0: iclass 3/0, 123 report ids
> > > > upd0 at uhidev0
> > > > upd0 detached
> > > > uhidev0 detached
> > > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > > uhidev0: iclass 3/0, 123 report ids
> > > > upd0 at uhidev0
> > > > upd0 detached
> > > > uhidev0 detached
> > > > uhub3: device problem, disabling port 3
> > > > uhub3: port error, restarting port 3
> > > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > > uhidev0: iclass 3/0, 123 report ids
> > > > upd0 at uhidev0
> > > > [1] - https://marc.info/?l=openbsd-misc&m=155532485003397&w=2
>
> > > -- 
> > > Lévai, Dániel
>
> > --
>
> > :wq Claudio


Reply | Threaded
Open this post in threaded view
|

Re: Leaking mbufs on 6.4 and 6.5

Lévai, Dániel
So it seems that by flushing the pass/block rules from the bridge(4) the leak has been plugged on this router.

What's interesting is that there's another setup similar to this in another room (another Wi-Fi AP but on an APU1D board, running 6.5 as well), with one athn(4) and three re(4) NICs bridged together with the same amount of pass/block rules, and there's nothing wrong there:
# ifconfig bridge0
bridge0: flags=41<UP,RUNNING>
        index 7 llprio 3
        groups: bridge
        priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp
        designated: id 00:00:00:00:00:00 priority 0
        re2 flags=3<LEARNING,DISCOVER>
                port 3 ifpriority 0 ifcost 0
        re1 flags=3<LEARNING,DISCOVER>
                port 2 ifpriority 0 ifcost 0
        re0 flags=3<LEARNING,DISCOVER>
                port 1 ifpriority 0 ifcost 0
        athn0 flags=3<LEARNING,DISCOVER>
                port 4 ifpriority 0 ifcost 0

# ifconfig bridge0|fgrep pass |wc -l
     124

# uptime
 9:52PM  up 1 day,  8:21, 1 user, load averages: 0.01, 0.07, 0.04

# systat -b mbuf
   1 users Load 0.05 0.08 0.05
IFACE             LIVELOCKS  SIZE ALIVE   LWM   HWM   CWM
System                    0   256   321         104
                             2048    21         149
                             4096    64          12
                             9216    34          44
lo0
re0                          9216    22     2  1023    22
re1                          9216     6     2  1023     6
re2                          9216     5     2  1023     5
athn0
enc0
bridge0


-- 
Lévai, Dániel

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, 2 May 2019 21:58, Lévai, Dániel <[hidden email]> wrote:

> So after about 8 hours running and reintroducing vether(4) into the setup and the bridge(4), the 2048 sized mbufs remained exactly 284, with an occasional bump here and there, but always reset back to that.
> The 256 sized ones grew by about a hundred over this ~8 hour period, I'm not sure if that's normal, even though anywhere else I have at most a couple of hundred of them -- with different types of NICs, however.
>
> 1 users Load 0.09 0.08 0.08
>
> IFACE LIVELOCKS SIZE ALIVE LWM HWM CWM
> System 0 256 763 94
> 2048 284 61
> 2112 178 62
> 4096 128 21
> lo0
> em0 2050 31 10 256 31
> em1 2050 35 10 256 35
> em2 2050 27 10 256 27
> em3 2050 85 10 256 85
> athn0
> athn1
> enc0
> bridge0
> pflog1
> pppoe0
> vether0
>
> -- 
> Lévai, Dániel
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Thursday, 2 May 2019 13:54, Lévai, Dániel [hidden email] wrote:
>
> > Thank you Claudio for taking the time to consider this!
> > Really appreciate the tip on the MAC filters in the bridge(4) -- I'll get to that in a second.
> > First of all, it seems that 6.5 exhibits a tad bit different behaviour then 6.4, and as I concluded I will not be able to reproduce or deduct the exact same thing(s), I just went ahead and retried everything with 6.5. More specifically, 6.5 seems to still leak mbufs without re0 and vether0 in the setup, but with a disabled re0 and a removed vether0, the leak became much slower.
> > Nevertheless, after removing re0 from the setup (disabled in the BIOS, nothing less) and removing the usage of vether0, mbufs for size 256 and 2048 were still growing slowly but steadily. And that's when I issued an `ifconfig bridge0 flushrule athn[01]'. And lo and behold, it's as if the leak had stopped. Still need to test this over more time but while without the block rules both (256, 2048) sized mbufs seemed to stagnate over a 10 minute period, after I reintroduced the block rules, over the same 10 minute period both (256, 2048) mbufs counter increased by ~900.
> > Now I'll try to bring back vether0 into the equation and see if that makes a difference without the bridge(4) block rules. I won't test re0 further, I'll just leave that disabled, because I still have a spare port in the 4 port Intel NIC -- so why bother..
> > Thanks again Claudio!!
> > Dani
> > -- 
> > Lévai, Dániel
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > On Thursday, 2 May 2019 11:32, Claudio Jeker [hidden email] wrote:
> >
> > > On Thu, May 02, 2019 at 07:48:30AM +0000, Lévai, Dániel wrote:
> >
> > > > Hello again,
> > > > I've upgraded to 6.5 and if possible it has gotten worse. Now it gets
> > > > unresponsive after ~half a day. With 6.4 I could get away with one
> > > > reboot per day but it seems I'm going to need two or more with 6.5.
> > > > Does anyone have any suggestions or suspicions? Anyone else noticed this
> > > > or similar throughout the previous two releases? I know a lot of changes
> > > > went into and are underway in the network stack re multi-threading
> > > > etc... is this maybe something related to this leak? I'm trying not to
> > > > push unrelated or unnecessary buttons until there's a good candidate for
> > > > the source of this problem.
> >
> > > Can you try to identify which of the many bits in your setup cause the
> > > mbuf leak? Easiest is to use a second system to split some of the
> > > functionality out. Try to remove vether(4) or the pppoe(4) part from your
> > > setup and check if this changes the behaviour. Also try to remove the MAC
> > > filters on bridge0 and see if that changes behavior.
> > > The leaked mbufs are from the system, re(4) or vether(4). Neither em(4)
> > > nor athn(4) use 2048 byte buffers on the RX side.
> >
> > > There is something special in your setup and you need to help us identify
> > > what it is. I know many people run similar setups without any troubles.
> >
> > > > Thanks in advance,
> > > > Daniel
> >
> > > > > Hi!
> > > > > I have mbufs leaking after introducing a network topology change on a
> > > > > router:
> > > > > After a reboot + 20 minutes:
> > > > > 1 users Load 0.46 0.38 0.33 16:40:43
> > > > > IFACE LIVELOCKS SIZE ALIVE LWM HWM CWM
> > > > > System 0 256 4100 260
> > > > > 2048 3717 468
> > > > > 2112 53 7
> > > > > 4096 128 20
> > > > > 9216 17 12
> > > > > After one day:
> > > > > 1 users Load 0.53 0.40 0.32 16:14:53
> > > > > IFACE LIVELOCKS SIZE ALIVE LWM HWM CWM
> > > > > System 0 256 131K 8456
> > > > > 2048 * 16022
> > > > > 2112 69 23
> > > > > 4096 128 21
> > > > > 9216 17 19
> > > > > After two days:
> > > > > 1 users Load 0.14 0.20 0.22 15:00:00
> > > > > IFACE LIVELOCKS SIZE ALIVE LWM HWM CWM
> > > > > System 0 256 215K 13780
> > > > > 2048 * 26954
> > > > > 2112 66 9
> > > > > 4096 128 25
> > > > > 9216 11 11
> > > > > I think this is either caused by starting to use an on-board re(4)
> > > > > device or starting to use vether(4) in a bridge(4) (dmesg, details
> > > > > below).
> > > > > In any case, 1 gigs worth of memory is consumed in about two days (give
> > > > > or take a couple of hours) with mild internet traffic (pppoe connected
> > > > > to re(4)).
> > > > > I was given the advice to inquire/report here, maybe someone can help
> > > > > track down the root cause.
> > > > > Any help is appreciated!
> > > > > This is my original e-mail [1] sent to misc@ (minus the writing about
> > > > > the processes running), it has a bunch of information, if someone's
> > > > > interested:
> > > > > <TL;DR>
> > > > > After a recent network configuration change (added re(4), vether(4)) I'm
> > > > > experiencing this memory leak from somewhere.
> > > > > How can I check/query how much memory the kernel (or parts of it) is
> > > > > using over time, besides running top(1) with system processes shown --
> > > > > I'm also staring at systat(1)'s `malloc' and`pool' views but I'm not
> > > > > really sure what I'm (or rather what I should be) looking at.
> > > > > </TL;DR>
> > > > > I have an on-board Realtek NIC and an add-on Intel PCIe 4 port NIC.
> > > > > re0 at pci3 dev 0 function 0 "Realtek 8168" rev 0x0c: RTL8168G/8111G (0x4c00), msi
> > > > > em0 at pci1 dev 0 function 0 "Intel I350" rev 0x01: msi
> > > > > em1 at pci1 dev 0 function 1 "Intel I350" rev 0x01: msi
> > > > > em2 at pci1 dev 0 function 2 "Intel I350" rev 0x01: msi
> > > > > em3 at pci1 dev 0 function 3 "Intel I350" rev 0x01: msi
> > > > > Before I started to experience the leak I had the on-board NIC disabled
> > > > > in BIOS and only used two ports of the Intel NIC. My setup looked like
> > > > > this:
> > > > > == Network
> > > > > ISP --- pppoe0 <-> em0 -- OpenBSD 6.4 -- em1(dhcpd) --- [switch] LAN
> > > > > `- athn0 --- LAN`- athn1 --- LAN
> > > > > `- bridge0(em1, athn[01]) Internal IP configured on em1. athn? devices don't have IPs, just bridged to the internal network. dhcpd(8) is running on em1. Now I started to use re0 and one more NIC on the Intel card: ISP --- pppoe0 <-> re0 -- OpenBSD 6.4 -- em0 --- LAN`- em1 --- LAN`- em2 --- LAN`- athn0 --- LAN`- athn1 --- LAN`- vether0 (dhcpd)`- bridge0(em[123], athn[01], vether0)
> > > > > Now the 4 port Intel NIC basically acts as a switch, the em? interfaces
> > > > > don't have IP addresses configured, and the athn? interfaces don't have
> > > > > one either.
> > > > > I've added a vether(4) interface with an internal IP (same as the em1
> > > > > have had before) to the bridge, and dhcpd(8) (and everything else that
> > > > > needs it) now uses that interface.
> >
> > > > > ifconfig bridge0
> >
> > > > > =================
> >
> > > > > bridge0: flags=41<UP,RUNNING>
> > > > > index 10 llprio 3
> > > > > groups: bridge
> > > > > priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto rstp
> > > > > designated: id 00:00:00:00:00:00 priority 0
> >
> > > > >         athn0 flags=3<LEARNING,DISCOVER>
> > > > >                 port 6 ifpriority 0 ifcost 0
> > > > >
> >
> > > > >
> >
> > > > > [bunch of `pass/block in/out on athn0 src/dst <MAC>' rules]
> >
> > > > >         athn1 flags=3<LEARNING,DISCOVER>
> > > > >                 port 7 ifpriority 0 ifcost 0
> > > > >
> >
> > > > >
> >
> > > > > [bunch of `pass/block in/out on athn1 src/dst <MAC>' rules]
> >
> > > > >         em0 flags=3<LEARNING,DISCOVER>
> > > > >                 port 1 ifpriority 0 ifcost 0
> > > > >         em1 flags=3<LEARNING,DISCOVER>
> > > > >                 port 2 ifpriority 0 ifcost 0
> > > > >         em2 flags=3<LEARNING,DISCOVER>
> > > > >                 port 3 ifpriority 0 ifcost 0
> > > > >         em3 flags=3<LEARNING,DISCOVER>
> > > > >                 port 4 ifpriority 0 ifcost 0
> > > > >         vether0 flags=3<LEARNING,DISCOVER>
> > > > >                 port 13 ifpriority 0 ifcost 0
> > > > >         Addresses (max cache: 100, timeout: 240):
> > > > >
> >
> > > > >
> >
> > > > > [...]
> >
> > > > > ifconfig vether0
> >
> > > > > =================
> >
> > > > > vether0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
> > > > > lladdr fe:e1:ba:d0:58:44
> > > > > description: Internal LAN
> > > > > index 13 priority 0 llprio 3
> > > > > groups: vether
> > > > > media: Ethernet autoselect
> > > > > status: active
> > > > > inet 192.168.0.1 netmask 0xffff0000 broadcast 192.168.255.255
> > > > > inet6 <> prefixlen 64 scopeid 0xd
> > > > > inet6 <> prefixlen 64 pltime 573642 vltime 573642
> >
> > > > > ifconfig em
> >
> > > > > ============
> >
> > > > > em0: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
> > > > > lladdr
> > > > > description: Internal LAN - Room1
> > > > > index 1 priority 0 llprio 3
> > > > > media: Ethernet autoselect (100baseTX full-duplex,rxpause,txpause)
> > > > > status: active
> > > > > em1: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
> > > > > lladdr
> > > > > description: Internal LAN - Room2
> > > > > index 2 priority 0 llprio 3
> > > > > media: Ethernet autoselect (1000baseT full-duplex,master,rxpause,txpause)
> > > > > status: active
> > > > > em2: flags=8b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
> > > > > lladdr
> > > > > description: Internal LAN - Room3
> > > > > index 3 priority 0 llprio 3
> > > > > media: Ethernet autoselect (100baseTX full-duplex,rxpause,txpause)
> > > > > status: active
> > > > > em3: flags=8902<BROADCAST,PROMISC,SIMPLEX,MULTICAST> mtu 1500
> > > > > lladdr
> > > > > index 4 priority 0 llprio 3
> > > > > media: Ethernet autoselect (none)
> > > > > status: no carrier
> > > > > Nothing fancy, just basically a "software switch" for connecting
> > > > > separate rooms in the house and adding a vether(4) interface for
> > > > > conveniency.
> > > > > Also, everything else remain unchanged, i.e.: no additional services
> > > > > running, no new software has been installed, no introduction of new pf
> > > > > rules, other than replacing the interface names of course.
> > > > > I'm using /etc/malloc.conf@ -> S on all my machines but never had any
> > > > > problems with it (and still don't anywhere else).
> > > > > == pf(4)
> > > > > pf(4) is used on the box, and the rules that have been changed are the
> > > > > ones related to the interfaces mentioned above:
> > > > > set skip on lo0
> > > > > set skip on bridge0
> > > > > pass on vether0 all flags S/SA allow-opts
> > > > > pass on em0 all flags S/SA allow-opts
> > > > > pass on em1 all flags S/SA allow-opts
> > > > > pass on em2 all flags S/SA allow-opts
> > > > > pass on em3 all flags S/SA allow-opts
> > > > > pass on athn0 all flags S/SA allow-opts
> > > > > pass on athn1 all flags S/SA allow-opts
> > > > > match in on vether0 inet proto tcp from 192.168.0.0/16 port = <port> to any set \
> > > > > (prio(2, 6), queue(ssh_bulk, ssh_rt)) block return on vether0 proto tcp from any to \
> > > > > any port = 25 block return on em0 proto tcp from any to any port = 25
> > > > > block return on em1 proto tcp from any to any port = 25
> > > > > block return on em2 proto tcp from any to any port = 25
> > > > > block return on em3 proto tcp from any to any port = 25
> > > > > block return on athn0 proto tcp from any to any port = 25
> > > > > block return on athn1 proto tcp from any to any port = 25
> > > > > Previously I've had `skip' on athn? as well.
> > > > > (I can upload the complete rule set somewhere if it is of interest)
> > > > > == dmesg
> > > > > OpenBSD 6.4 (GENERIC.MP) #9: Tue Mar 26 19:21:43 CET 2019
> > > > > [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> > > > > real mem = 2059997184 (1964MB)
> > > > > avail mem = 1988288512 (1896MB)
> > > > > mpath0 at root
> > > > > scsibus0 at mpath0: 256 targets
> > > > > mainbus0 at root
> > > > > bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xec130 (75 entries)
> > > > > bios0: vendor American Megatrends Inc. version "F2" date 08/06/2015
> > > > > bios0: Gigabyte Technology Co., Ltd. H81M-DS2
> > > > > acpi0 at bios0: rev 2
> > > > > acpi0: sleep states S0 S3 S4 S5
> > > > > acpi0: tables DSDT FACP APIC FPDT SSDT SSDT MCFG HPET SSDT SSDT
> > > > > acpi0: wakeup devices PXSX(S4) RP01(S4) PXSX(S4) PXSX(S4) RP03(S4) PXSX(S4) PXSX(S4) \
> > > > > RP05(S4) PXSX(S4) RP06(S4) PXSX(S4) PXSX(S4) GLAN(S4) EHC1(S4) EHC2(S4) XHC_(S4) \
> > > > > [...] acpitimer0 at acpi0: 3579545 Hz, 24 bits
> > > > > acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> > > > > cpu0 at mainbus0: apid 0 (boot processor)
> > > > > cpu0: Intel(R) Celeron(R) CPU G1840 @ 2.80GHz, 2793.95 MHz, 06-3c-03
> > > > > cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,D \
> > > > > S,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSS \
> > > > > E3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,XSAVE,RDRAND,NXE,PAGE1 \
> > > > > GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,ERMS,INVPCID,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> > > > > cpu0: 256KB 64b/line 8-way L2 cache
> > > > > cpu0: smt 0, core 0, package 0
> > > > > mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> > > > > cpu0: apic clock running at 99MHz
> > > > > cpu0: mwait min=64, max=64, C-substates=0.2.1.2, IBE
> > > > > cpu1 at mainbus0: apid 2 (application processor)
> > > > > cpu1: Intel(R) Celeron(R) CPU G1840 @ 2.80GHz, 2793.55 MHz, 06-3c-03
> > > > > cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,D \
> > > > > S,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSS \
> > > > > E3,SDBG,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,MOVBE,POPCNT,DEADLINE,XSAVE,RDRAND,NXE,PAGE1 \
> > > > > GB,RDTSCP,LONG,LAHF,ABM,PERF,ITSC,FSGSBASE,ERMS,INVPCID,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> > > > > cpu1: 256KB 64b/line 8-way L2 cache
> > > > > cpu1: smt 0, core 1, package 0
> > > > > ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 20, 24 pins
> > > > > acpimcfg0 at acpi0
> > > > > acpimcfg0: addr 0xf8000000, bus 0-63
> > > > > acpihpet0 at acpi0: 14318179 Hz
> > > > > acpiprt0 at acpi0: bus 0 (PCI0)
> > > > > acpiprt1 at acpi0: bus 2 (RP01)
> > > > > acpiprt2 at acpi0: bus 3 (RP03)
> > > > > acpiprt3 at acpi0: bus 4 (RP05)
> > > > > acpiprt4 at acpi0: bus 5 (RP06)
> > > > > acpiprt5 at acpi0: bus 1 (PEG0)
> > > > > acpiprt6 at acpi0: bus -1 (PEG1)
> > > > > acpiprt7 at acpi0: bus -1 (PEG2)
> > > > > acpiec0 at acpi0: not present
> > > > > acpicpu0 at acpi0: C2(500@67 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> > > > > acpicpu1 at acpi0: C2(500@67 mwait.1@0x10), C1(1000@1 mwait.1), PSS
> > > > > acpipwrres0 at acpi0: FN00, resource for FAN0
> > > > > acpipwrres1 at acpi0: FN01, resource for FAN1
> > > > > acpipwrres2 at acpi0: FN02, resource for FAN2
> > > > > acpipwrres3 at acpi0: FN03, resource for FAN3
> > > > > acpipwrres4 at acpi0: FN04, resource for FAN4
> > > > > acpitz0 at acpi0: critical temperature is 105 degC
> > > > > acpitz1 at acpi0: critical temperature is 105 degC
> > > > > acpicmos0 at acpi0
> > > > > acpibtn0 at acpi0: PWRB
> > > > > "PNP0C0B" at acpi0 not configured
> > > > > "PNP0C0B" at acpi0 not configured
> > > > > "PNP0C0B" at acpi0 not configured
> > > > > "PNP0C0B" at acpi0 not configured
> > > > > "PNP0C0B" at acpi0 not configured
> > > > > acpivideo0 at acpi0: GFX0
> > > > > acpivout0 at acpivideo0: DD1F
> > > > > cpu0: Enhanced SpeedStep 2793 MHz: speeds: 2800, 2700, 2500, 2400, 2300, 2100, 2000, \
> > > > > 1900, 1700, 1600, 1500, 1300, 1200, 1100, 900, 800 MHz pci0 at mainbus0 bus 0
> > > > > pchb0 at pci0 dev 0 function 0 "Intel Core 4G Host" rev 0x06
> > > > > ppb0 at pci0 dev 1 function 0 "Intel Core 4G PCIE" rev 0x06: msi
> > > > > pci1 at ppb0 bus 1
> > > > > em0 at pci1 dev 0 function 0 "Intel I350" rev 0x01: msi, address
> > > > > em1 at pci1 dev 0 function 1 "Intel I350" rev 0x01: msi, address
> > > > > em2 at pci1 dev 0 function 2 "Intel I350" rev 0x01: msi, address
> > > > > em3 at pci1 dev 0 function 3 "Intel I350" rev 0x01: msi, address
> > > > > inteldrm0 at pci0 dev 2 function 0 "Intel HD Graphics" rev 0x06
> > > > > drm0 at inteldrm0
> > > > > inteldrm0: msi
> > > > > error: [drm:pid0:drm_edid_block_valid] ERROR EDID checksum is invalid, remainder is \
> > > > > 39 Raw EDID:
> > > > > 00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> > > > > fe 11 01 03 08 25 1e 78 ca bb 05 a3 56 47 99 24
> > > > > f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > error: [drm:pid0:drm_edid_block_valid] ERROR EDID checksum is invalid, remainder is \
> > > > > 7 Raw EDID:
> > > > > 00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> > > > > de 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
> > > > > d4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > error: [drm:pid0:drm_edid_block_valid] ERROR EDID checksum is invalid, remainder is \
> > > > > 39 Raw EDID:
> > > > > 00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> > > > > de 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
> > > > > f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > error: [drm:pid0:drm_edid_block_valid] ERROR EDID checksum is invalid, remainder is \
> > > > > 71 Raw EDID:
> > > > > 00 ff ff ff ff ff ff 00 fc ad 00 00 00 00 00 00
> > > > > fe 11 01 03 08 25 1e 78 ea bb 05 a3 56 47 99 24
> > > > > f4 50 54 2f ce 00 01 01 e7 ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
> > > > > drm:pid0:drm_do_get_edid WARNING VGA-: EDID block 0 invalid.
> > > > > inteldrm0: 1024x768, 32bpp
> > > > > wsdisplay0 at inteldrm0 mux 1: console (std, vt100 emulation)
> > > > > wsdisplay0: screen 1-5 added (std, vt100 emulation)
> > > > > azalia0 at pci0 dev 3 function 0 "Intel Core 4G HD Audio" rev 0x06: msi
> > > > > azalia0: No codecs found
> > > > > "Intel 8 Series MEI" rev 0x04 at pci0 dev 22 function 0 not configured
> > > > > ehci0 at pci0 dev 26 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 16
> > > > > usb0 at ehci0: USB revision 2.0
> > > > > uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> > > > > ppb1 at pci0 dev 28 function 0 "Intel 8 Series PCIE" rev 0xd5: msi
> > > > > pci2 at ppb1 bus 2
> > > > > ppb2 at pci0 dev 28 function 2 "Intel 8 Series PCIE" rev 0xd5: msi
> > > > > pci3 at ppb2 bus 3
> > > > > re0 at pci3 dev 0 function 0 "Realtek 8168" rev 0x0c: RTL8168G/8111G (0x4c00), msi, \
> > > > > address rgephy0 at re0 phy 7: RTL8251 PHY, rev. 0
> > > > > ppb3 at pci0 dev 28 function 4 "Intel 8 Series PCIE" rev 0xd5: msi
> > > > > pci4 at ppb3 bus 4
> > > > > athn0 at pci4 dev 0 function 0 "Atheros AR9281" rev 0x01: apic 8 int 16
> > > > > athn0: AR9280 rev 2 (2T2R), ROM rev 16, address
> > > > > ppb4 at pci0 dev 28 function 5 "Intel 8 Series PCIE" rev 0xd5: msi
> > > > > pci5 at ppb4 bus 5
> > > > > athn1 at pci5 dev 0 function 0 "Atheros AR9281" rev 0x01: apic 8 int 17
> > > > > athn1: AR9280 rev 2 (2T2R), ROM rev 22, address
> > > > > ehci1 at pci0 dev 29 function 0 "Intel 8 Series USB" rev 0x05: apic 8 int 23
> > > > > usb1 at ehci1: USB revision 2.0
> > > > > uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev 2.00/1.00 addr 1
> > > > > pcib0 at pci0 dev 31 function 0 "Intel H81 LPC" rev 0x05
> > > > > ahci0 at pci0 dev 31 function 2 "Intel 8 Series AHCI" rev 0x05: msi, AHCI 1.3
> > > > > ahci0: port 4: 3.0Gb/s
> > > > > ahci0: port 5: 3.0Gb/s
> > > > > scsibus1 at ahci0: 32 targets
> > > > > sd0 at scsibus1 targ 4 lun 0: <ATA, KINGSTON SHFS37A, 608A> SCSI3 0/direct fixed \
> > > > > naa.50026b726304fb22
> > > > > sd0: 114473MB, 512 bytes/sector, 234441648 sectors, thin
> > > > > sd1 at scsibus1 targ 5 lun 0: <ATA, KINGSTON SHFS37A, 608A> SCSI3 0/direct fixed \
> > > > > naa.50026b726304fbe2
> > > > > sd1: 114473MB, 512 bytes/sector, 234441648 sectors, thin
> > > > > ichiic0 at pci0 dev 31 function 3 "Intel 8 Series SMBus" rev 0x05: apic 8 int 18
> > > > > iic0 at ichiic0
> > > > > spdmem0 at iic0 addr 0x50: 2GB DDR3 SDRAM PC3-10600
> > > > > isa0 at pcib0
> > > > > isadma0 at isa0
> > > > > com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> > > > > pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> > > > > pckbd0 at pckbc0 (kbd slot)
> > > > > wskbd0 at pckbd0: console keyboard, using wsdisplay0
> > > > > pcppi0 at isa0 port 0x61
> > > > > spkr0 at pcppi0
> > > > > uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching Hub" rev \
> > > > > 2.00/0.05 addr 2 uhub3 at uhub1 port 1 configuration 1 interface 0 "Intel Rate \
> > > > > Matching Hub" rev 2.00/0.05 addr 2 uhidev0 at uhub3 port 2 configuration 1 interface \
> > > > > 0 "SINO WEALTH USB KEYBOARD" rev 1.10/1.00 addr 3
> > > > > uhidev0: iclass 3/1
> > > > > ukbd0 at uhidev0: 8 variable keys, 6 key codes
> > > > > wskbd1 at ukbd0 mux 1
> > > > > wskbd1: connecting to wsdisplay0
> > > > > uhidev1 at uhub3 port 2 configuration 1 interface 1 "SINO WEALTH USB KEYBOARD" rev \
> > > > > 1.10/1.00 addr 3
> > > > > uhidev1: iclass 3/0, 5 report ids
> > > > > uhid0 at uhidev1 reportid 2: input=1, output=0, feature=0
> > > > > uhid1 at uhidev1 reportid 3: input=3, output=0, feature=0
> > > > > uhid2 at uhidev1 reportid 5: input=0, output=0, feature=5
> > > > > uhidev2 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 4
> > > > > uhidev2: iclass 3/0, 123 report ids
> > > > > upd0 at uhidev2
> > > > > vscsi0 at root
> > > > > scsibus2 at vscsi0: 256 targets
> > > > > softraid0 at root
> > > > > scsibus3 at softraid0: 256 targets
> > > > > sd2 at scsibus3 targ 1 lun 0: <OPENBSD, SR RAID 1, 006> SCSI2 0/direct fixed
> > > > > sd2: 114470MB, 512 bytes/sector, 234435953 sectors
> > > > > root on sd2a (59f8b4a610bc10ff.a) swap on sd2b dump on sd2b
> > > > > pppoe: GENERIC ERROR: RP-PPPoE: Child pppd process terminated
> > > > > wskbd1: disconnecting from wsdisplay0
> > > > > wskbd1 detached
> > > > > ukbd0 detached
> > > > > uhidev0 detached
> > > > > uhid0 detached
> > > > > uhid1 detached
> > > > > uhid2 detached
> > > > > uhidev1 detached
> > > > > pppoe0: received unexpected PADO
> > > > > pppoe0: received unexpected PADO
> > > > > pppoe0: received unexpected PADO
> > > > > upd0 detached
> > > > > uhidev2 detached
> > > > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > > > uhidev0: iclass 3/0, 123 report ids
> > > > > upd0 at uhidev0
> > > > > upd0 detached
> > > > > uhidev0 detached
> > > > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > > > uhidev0: iclass 3/0, 123 report ids
> > > > > upd0 at uhidev0
> > > > > upd0 detached
> > > > > uhidev0 detached
> > > > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > > > uhidev0: iclass 3/0, 123 report ids
> > > > > upd0 at uhidev0
> > > > > upd0 detached
> > > > > uhidev0 detached
> > > > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > > > uhidev0: iclass 3/0, 123 report ids
> > > > > upd0 at uhidev0
> > > > > upd0 detached
> > > > > uhidev0 detached
> > > > > uhub3: device problem, disabling port 3
> > > > > uhub3: port error, restarting port 3
> > > > > uhidev0 at uhub3 port 3 configuration 1 interface 0 "American Power Conversion \
> > > > > Back-UPS XS 1400U FW:926.T1 .I USB FW:T1" rev 1.10/1.06 addr 3
> > > > > uhidev0: iclass 3/0, 123 report ids
> > > > > upd0 at uhidev0
> > > > > [1] - https://marc.info/?l=openbsd-misc&m=155532485003397&w=2
> >
> > > > -- 
> > > > Lévai, Dániel
> >
> > > --
> >
> > > :wq Claudio


Reply | Threaded
Open this post in threaded view
|

Re: Leaking mbufs on 6.4 and 6.5

Claudio Jeker-3
On Fri, May 03, 2019 at 07:58:04PM +0000, Lévai, Dániel wrote:
> So it seems that by flushing the pass/block rules from the bridge(4) the
> leak has been plugged on this router.
>

Please try the following diff. This is fixing a obvious mem leak in one
special case (this may be why you did not notice it on the other system).

--
:wq Claudio

Index: if_bridge.c
===================================================================
RCS file: /cvs/src/sys/net/if_bridge.c,v
retrieving revision 1.329
diff -u -p -r1.329 if_bridge.c
--- if_bridge.c 3 May 2019 16:53:07 -0000 1.329
+++ if_bridge.c 8 May 2019 22:42:48 -0000
@@ -808,6 +808,10 @@ bridge_enqueue(struct ifnet *ifp, struct
     (m->m_flags & (M_BCAST | M_MCAST)) == 0)
  continue;
 
+ if (bridge_filterrule(&bif->bif_brlout, eh, m) ==
+    BRL_ACTION_BLOCK)
+ continue;
+
  if (SLIST_NEXT(bif, bif_next) == NULL) {
  used = 1;
  mc = m;
@@ -819,10 +823,6 @@ bridge_enqueue(struct ifnet *ifp, struct
  }
  }
 
- if (bridge_filterrule(&bif->bif_brlout, eh, mc) ==
-    BRL_ACTION_BLOCK)
- continue;
-
  error = bridge_ifenqueue(brifp, dst_if, mc);
  if (error)
  continue;
@@ -1249,7 +1249,8 @@ bridge_broadcast(struct bridge_softc *sc
  if (protected != 0 && (protected & bif->bif_protected))
  continue;
 
- if (bridge_filterrule(&bif->bif_brlout, eh, m) == BRL_ACTION_BLOCK)
+ if (bridge_filterrule(&bif->bif_brlout, eh, m) ==
+    BRL_ACTION_BLOCK)
  continue;
 
  /*

Reply | Threaded
Open this post in threaded view
|

Re: Leaking mbufs on 6.4 and 6.5

Lévai, Dániel
Ooh, thank you sir, this is perfect!
Works like a charm on 6.5 with the original setup.


Daniel


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday, 9 May 2019 00:47, Claudio Jeker <[hidden email]> wrote:

> On Fri, May 03, 2019 at 07:58:04PM +0000, Lévai, Dániel wrote:
>
> > So it seems that by flushing the pass/block rules from the bridge(4) the
> > leak has been plugged on this router.
>
> Please try the following diff. This is fixing a obvious mem leak in one
> special case (this may be why you did not notice it on the other system).
>
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> :wq Claudio
>
> Index: if_bridge.c
>
> ================================
>
> RCS file: /cvs/src/sys/net/if_bridge.c,v
> retrieving revision 1.329
> diff -u -p -r1.329 if_bridge.c
> --- if_bridge.c 3 May 2019 16:53:07 -0000 1.329
> +++ if_bridge.c 8 May 2019 22:42:48 -0000
> @@ -808,6 +808,10 @@ bridge_enqueue(struct ifnet *ifp, struct
> (m->m_flags & (M_BCAST | M_MCAST)) == 0)
>
>     continue;
>
>
> -         if (bridge_filterrule(&bif->bif_brlout, eh, m) ==
>
>
> -            BRL_ACTION_BLOCK)
>
>
> -         continue;
>
>
> -         if (SLIST_NEXT(bif, bif_next) == NULL) {
>           used = 1;
>           mc = m;
>
>
>
> @@ -819,10 +823,6 @@ bridge_enqueue(struct ifnet *ifp, struct
> }
> }
>
> -         if (bridge_filterrule(&bif->bif_brlout, eh, mc) ==
>
>
> -            BRL_ACTION_BLOCK)
>
>
> -         continue;
>
>
> -         error = bridge_ifenqueue(brifp, dst_if, mc);
>           if (error)
>           continue;
>
>
>
> @@ -1249,7 +1249,8 @@ bridge_broadcast(struct bridge_softc *sc
> if (protected != 0 && (protected & bif->bif_protected))
>
>     continue;
>
>
> -         if (bridge_filterrule(&bif->bif_brlout, eh, m) == BRL_ACTION_BLOCK)
>
>
>
> -         if (bridge_filterrule(&bif->bif_brlout, eh, m) ==
>
>
> -             BRL_ACTION_BLOCK)
>           continue;
>
>
>
> /*