10Gbit network work only 1Gbit

[Szél Gábor]

Hello @misc,

We have an interesting problem, we run a lot of OpenBSD router/firewalls
in many places.

We have a larger network than our client, 300-400 local wired or
wireless endpoint, 20+ VLAN, 20+ switches.
Network structure:

  * Main switch - 2x Cisco Nexus 3k switch in HA mode (vPC dedicated
    2x40Gbit Peer link, keepalive link)
  * access switch - 10+ Cisco 3750X + C3KX-SM-10G 10Gbit module.
    some 3750x stacked (2 or 3 switch)
  * Main and access switches have redundant 10Gbit fiber link (LACP)
  * when is possible jumbo frame is enabled (mtu 9000)

Firewall/router:

  * 2x Dell 2950 - 2x Xeon X5460 (8 core), 8Gb Memory, 2x10Gbit SFP+
    network card
  * redundant design - CARP, pfsync, ifstated, etc .... master-backup
    configuration
  * HP NC550SFP network card, oce driver (mtu 9000)
  * dual SFP+ port have LACP link to Nexus switches (2x10Gbit access
    link) - use openbsd trunk interface
  * all vlan used openbsd pseudo-device over trunk interface (VLANs not
    have have IP address, only up)
  * all network subnets defined in CARP interfaces, only managment VLAN
    have address on VLAN interface.
  * some vether virtual interface for VPN, DNS, etc ...
  * some tun and tap interface for VPN
  * enc interface for ipsec
  * one bridge interface for openVPN (during termination)
  * OpenBSD 6.3 64bit

PF:

  * global block rule (block all)
  * ruleset-optimization none
  * optimization aggressive
  * reassemble no
  * block-policy drop
  * scrub enabled
  * antispoof enabled
  * regulating traffic between subnets with pf pass in/out rules
  * pf.conf currently 1500+ lines
  * the number of connections during the day in PF 10 000+

Problem:

We see that network traffic is limited to 1Gbit on firewall. Not in one
link, not IP-to-IP, to the whole firewall!

example:

  * i make test traffic form VLAN 2 to VLAN 12 witch iperf.
    test PC-s have 1Gbit ethernet cards.
    Speed is okay, ~800Mbit/sec
  * i make anoter traffic from VLAN 2 to VLan20 with iperf, from another
    PC-s
    (they also have 1gbit ethernet cards)
    speed is not good! ~60-80Mbit/sec
  * if i stopped first speed test (2->12), second test speed is okay!
    (2->20)
  * but i make test from completely different VLANs, 2->12 and 20->30,
    the result is so.

This is firewall (openbsd) limitation, but we don't understand why?

I know openbsd VLAN interface has a speed problem, this is it?
I know it's so difficult to make a mistake from some information, what
should we look at?

--
Üdvözlettel,
Szél Gábor

WanTax Kft.
------------
tel.: [phone]
fax: [phone]
email: [hidden email]
web: http://wantax.hu
web: http://halozatom.hu

[Daniel Melameth]

...
> Firewall/router:
...
>   * OpenBSD 6.3 64bit
...

I recommend updating to a modern version of OpenBSD before spending
time investigating the issue further.

[Hrvoje Popovski]

On 11.11.2019. 13:42, Szél Gábor wrote:
> Hello @misc,
>

Hi,

> We have an interesting problem, we run a lot of OpenBSD router/firewalls
> in many places.
>
> We have a larger network than our client, 300-400 local wired or
> wireless endpoint, 20+ VLAN, 20+ switches.
> Network structure:
>
>  * Main switch - 2x Cisco Nexus 3k switch in HA mode (vPC dedicated
>    2x40Gbit Peer link, keepalive link)
>  * access switch - 10+ Cisco 3750X + C3KX-SM-10G 10Gbit module.
>    some 3750x stacked (2 or 3 switch)
>  * Main and access switches have redundant 10Gbit fiber link (LACP)
>  * when is possible jumbo frame is enabled (mtu 9000)
>
> Firewall/router:
>
>  * 2x Dell 2950 - 2x Xeon X5460 (8 core), 8Gb Memory, 2x10Gbit SFP+
>    network card

hardware is really old, if you can, buy something newer


>  * redundant design - CARP, pfsync, ifstated, etc .... master-backup>    configuration
>  * HP NC550SFP network card, oce driver (mtu 9000)

if you can change oce with ix. ixl is not so bad .. .


>  * dual SFP+ port have LACP link to Nexus switches (2x10Gbit access
>    link) - use openbsd trunk interface
>  * all vlan used openbsd pseudo-device over trunk interface (VLANs not
>    have have IP address, only up)

update to openbsd 6.6 or snapshot and insted of trunk use aggr. why vlan
interfaces don't have ip address ?


>  * all network subnets defined in CARP interfaces, only managment VLAN
>    have address on VLAN interface.

who is parent interface for carp ?

>  * some vether virtual interface for VPN, DNS, etc ...

vether implies that you have bridge? bridge is slow..

>  * some tun and tap interface for VPN
>  * enc interface for ipsec

ipsec is performance killer big time ... even for traffic that doesn't
go through ipsec tunnel ..  if you can move ipsec or any vpn stuff to
other boxes that you speed up your firewalls  ...

>  * one bridge interface for openVPN (during termination)

vether is in that bridge?

>  * OpenBSD 6.3 64bit

please, update boxes regularly.. you have carp and pfsync, you can do
that without any problem ..

>
> PF:
>
>  * global block rule (block all)
>  * ruleset-optimization none
>  * optimization aggressive
>  * reassemble no
>  * block-policy drop
>  * scrub enabled
>  * antispoof enabled
>  * regulating traffic between subnets with pf pass in/out rules
>  * pf.conf currently 1500+ lines
>  * the number of connections during the day in PF 10 000+
>
> Problem:
>
> We see that network traffic is limited to 1Gbit on firewall. Not in one
> link, not IP-to-IP, to the whole firewall!
>

yes ... ipsec, trunk, pf are for whole firewall .. and even if you have
fastest box in the world you will not get performance that you want ..

> example:
>
>  * i make test traffic form VLAN 2 to VLAN 12 witch iperf.
>    test PC-s have 1Gbit ethernet cards.
>    Speed is okay, ~800Mbit/sec
>  * i make anoter traffic from VLAN 2 to VLan20 with iperf, from another
>    PC-s
>    (they also have 1gbit ethernet cards)
>    speed is not good! ~60-80Mbit/sec
>  * if i stopped first speed test (2->12), second test speed is okay!
>    (2->20)
>  * but i make test from completely different VLANs, 2->12 and 20->30,
>    the result is so.

if you disable pf on vlan intefaces (set skip on vlan2/vlan12) do you
get better performace?
and after that for disable ipsec and try testing again... do you see
differences ?

> This is firewall (openbsd) limitation, but we don't understand why?
>
> I know openbsd VLAN interface has a speed problem, this is it?

not in OpenBSD 6.6

> I know it's so difficult to make a mistake from some information, what
> should we look at?
>

OpenBSD is great router and firewall that can do so much for you .. but
please you really need to rethink your hardware and setup ..

[Theo de Raadt-2]

>  * OpenBSD 6.3 64bit

Oh come on.

[Szél Gábor]

Dear Hrvoje, Theo,

Thank you for your answers!

answers to the questions:
-  who is parent interface for carp?  -> vlan  ( carp10 interface parent
vlan10 -> vlan10 interface  parent -> trunk0 )
- why vlan interfaces don't have ip address ? -> it wasn't needed! i
think vlan interface need only tag packages. Carp (over vlan) interface
have IP address.
- vether implies that you have bridge? -> yes whe have only one bridge
for bridget openvpn clients, but  we will eliminate it.


we will do the following:
- refresh our backup firewall to oBSD 6.6
- replace trunk interface with aggr
- remove bridge interface

if there was an update finised, I'll write again!

--
Regars
Szél Gábor

WanTax Kft.
------------
tel.: [phone]
fax: [phone]
email: [hidden email]
web: http://wantax.hu
web: http://halozatom.hu


2019. 11. 11. 23:42 keltezéssel, Hrvoje Popovski írta:

> On 11.11.2019. 13:42, Szél Gábor wrote:
>> Hello @misc,
>>
> Hi,
>
>
>> We have an interesting problem, we run a lot of OpenBSD router/firewalls
>> in many places.
>>
>> We have a larger network than our client, 300-400 local wired or
>> wireless endpoint, 20+ VLAN, 20+ switches.
>> Network structure:
>>
>>   * Main switch - 2x Cisco Nexus 3k switch in HA mode (vPC dedicated
>>     2x40Gbit Peer link, keepalive link)
>>   * access switch - 10+ Cisco 3750X + C3KX-SM-10G 10Gbit module.
>>     some 3750x stacked (2 or 3 switch)
>>   * Main and access switches have redundant 10Gbit fiber link (LACP)
>>   * when is possible jumbo frame is enabled (mtu 9000)
>>
>> Firewall/router:
>>
>>   * 2x Dell 2950 - 2x Xeon X5460 (8 core), 8Gb Memory, 2x10Gbit SFP+
>>     network card
> hardware is really old, if you can, buy something newer
>
>
>>   * redundant design - CARP, pfsync, ifstated, etc .... master-backup>    configuration
>>   * HP NC550SFP network card, oce driver (mtu 9000)
> if you can change oce with ix. ixl is not so bad .. .
>
>
>>   * dual SFP+ port have LACP link to Nexus switches (2x10Gbit access
>>     link) - use openbsd trunk interface
>>   * all vlan used openbsd pseudo-device over trunk interface (VLANs not
>>     have have IP address, only up)
> update to openbsd 6.6 or snapshot and insted of trunk use aggr. why vlan
> interfaces don't have ip address ?
>
>
>>   * all network subnets defined in CARP interfaces, only managment VLAN
>>     have address on VLAN interface.
> who is parent interface for carp ?
>
>>   * some vether virtual interface for VPN, DNS, etc ...
> vether implies that you have bridge? bridge is slow..
>
>>   * some tun and tap interface for VPN
>>   * enc interface for ipsec
> ipsec is performance killer big time ... even for traffic that doesn't
> go through ipsec tunnel ..  if you can move ipsec or any vpn stuff to
> other boxes that you speed up your firewalls  ...
>
>>   * one bridge interface for openVPN (during termination)
> vether is in that bridge?
>
>>   * OpenBSD 6.3 64bit
> please, update boxes regularly.. you have carp and pfsync, you can do
> that without any problem ..
>
>> PF:
>>
>>   * global block rule (block all)
>>   * ruleset-optimization none
>>   * optimization aggressive
>>   * reassemble no
>>   * block-policy drop
>>   * scrub enabled
>>   * antispoof enabled
>>   * regulating traffic between subnets with pf pass in/out rules
>>   * pf.conf currently 1500+ lines
>>   * the number of connections during the day in PF 10 000+
>>
>> Problem:
>>
>> We see that network traffic is limited to 1Gbit on firewall. Not in one
>> link, not IP-to-IP, to the whole firewall!
>>
> yes ... ipsec, trunk, pf are for whole firewall .. and even if you have
> fastest box in the world you will not get performance that you want ..
>
>
>> example:
>>
>>   * i make test traffic form VLAN 2 to VLAN 12 witch iperf.
>>     test PC-s have 1Gbit ethernet cards.
>>     Speed is okay, ~800Mbit/sec
>>   * i make anoter traffic from VLAN 2 to VLan20 with iperf, from another
>>     PC-s
>>     (they also have 1gbit ethernet cards)
>>     speed is not good! ~60-80Mbit/sec
>>   * if i stopped first speed test (2->12), second test speed is okay!
>>     (2->20)
>>   * but i make test from completely different VLANs, 2->12 and 20->30,
>>     the result is so.
> if you disable pf on vlan intefaces (set skip on vlan2/vlan12) do you
> get better performace?
> and after that for disable ipsec and try testing again... do you see
> differences ?
>
>> This is firewall (openbsd) limitation, but we don't understand why?
>>
>> I know openbsd VLAN interface has a speed problem, this is it?
> not in OpenBSD 6.6
>
>> I know it's so difficult to make a mistake from some information, what
>> should we look at?
>>
> OpenBSD is great router and firewall that can do so much for you .. but
> please you really need to rethink your hardware and setup ..
>
>

[Diana Eichert]

Someone else mentioned your underlying system, Dell 2950, is ancient.
It really is ancient, just because it links up @ 10G doesn't mean you
are going to see anywhere near 10G

On Tue, Nov 12, 2019 at 3:10 AM Szél Gábor <[hidden email]> wrote:

>
> Dear Hrvoje, Theo,
>
> Thank you for your answers!
>
> answers to the questions:
> -  who is parent interface for carp?  -> vlan  ( carp10 interface parent
> vlan10 -> vlan10 interface  parent -> trunk0 )
> - why vlan interfaces don't have ip address ? -> it wasn't needed! i
> think vlan interface need only tag packages. Carp (over vlan) interface
> have IP address.
> - vether implies that you have bridge? -> yes whe have only one bridge
> for bridget openvpn clients, but  we will eliminate it.
>
>
> we will do the following:
> - refresh our backup firewall to oBSD 6.6
> - replace trunk interface with aggr
> - remove bridge interface
>
> if there was an update finised, I'll write again!
>
> --
> Regars
> Szél Gábor
>
> WanTax Kft.
> ------------
> tel.: [phone]
> fax: [phone]
> email: [hidden email]
> web: http://wantax.hu
> web: http://halozatom.hu
>
>
> 2019. 11. 11. 23:42 keltezéssel, Hrvoje Popovski írta:
> > On 11.11.2019. 13:42, Szél Gábor wrote:
> >> Hello @misc,
> >>
> > Hi,
> >
> >
> >> We have an interesting problem, we run a lot of OpenBSD router/firewalls
> >> in many places.
> >>
> >> We have a larger network than our client, 300-400 local wired or
> >> wireless endpoint, 20+ VLAN, 20+ switches.
> >> Network structure:
> >>
> >>   * Main switch - 2x Cisco Nexus 3k switch in HA mode (vPC dedicated
> >>     2x40Gbit Peer link, keepalive link)
> >>   * access switch - 10+ Cisco 3750X + C3KX-SM-10G 10Gbit module.
> >>     some 3750x stacked (2 or 3 switch)
> >>   * Main and access switches have redundant 10Gbit fiber link (LACP)
> >>   * when is possible jumbo frame is enabled (mtu 9000)
> >>
> >> Firewall/router:
> >>
> >>   * 2x Dell 2950 - 2x Xeon X5460 (8 core), 8Gb Memory, 2x10Gbit SFP+
> >>     network card
> > hardware is really old, if you can, buy something newer
> >
> >
> >>   * redundant design - CARP, pfsync, ifstated, etc .... master-backup>    configuration
> >>   * HP NC550SFP network card, oce driver (mtu 9000)
> > if you can change oce with ix. ixl is not so bad .. .
> >
> >
> >>   * dual SFP+ port have LACP link to Nexus switches (2x10Gbit access
> >>     link) - use openbsd trunk interface
> >>   * all vlan used openbsd pseudo-device over trunk interface (VLANs not
> >>     have have IP address, only up)
> > update to openbsd 6.6 or snapshot and insted of trunk use aggr. why vlan
> > interfaces don't have ip address ?
> >
> >
> >>   * all network subnets defined in CARP interfaces, only managment VLAN
> >>     have address on VLAN interface.
> > who is parent interface for carp ?
> >
> >>   * some vether virtual interface for VPN, DNS, etc ...
> > vether implies that you have bridge? bridge is slow..
> >
> >>   * some tun and tap interface for VPN
> >>   * enc interface for ipsec
> > ipsec is performance killer big time ... even for traffic that doesn't
> > go through ipsec tunnel ..  if you can move ipsec or any vpn stuff to
> > other boxes that you speed up your firewalls  ...
> >
> >>   * one bridge interface for openVPN (during termination)
> > vether is in that bridge?
> >
> >>   * OpenBSD 6.3 64bit
> > please, update boxes regularly.. you have carp and pfsync, you can do
> > that without any problem ..
> >
> >> PF:
> >>
> >>   * global block rule (block all)
> >>   * ruleset-optimization none
> >>   * optimization aggressive
> >>   * reassemble no
> >>   * block-policy drop
> >>   * scrub enabled
> >>   * antispoof enabled
> >>   * regulating traffic between subnets with pf pass in/out rules
> >>   * pf.conf currently 1500+ lines
> >>   * the number of connections during the day in PF 10 000+
> >>
> >> Problem:
> >>
> >> We see that network traffic is limited to 1Gbit on firewall. Not in one
> >> link, not IP-to-IP, to the whole firewall!
> >>
> > yes ... ipsec, trunk, pf are for whole firewall .. and even if you have
> > fastest box in the world you will not get performance that you want ..
> >
> >
> >> example:
> >>
> >>   * i make test traffic form VLAN 2 to VLAN 12 witch iperf.
> >>     test PC-s have 1Gbit ethernet cards.
> >>     Speed is okay, ~800Mbit/sec
> >>   * i make anoter traffic from VLAN 2 to VLan20 with iperf, from another
> >>     PC-s
> >>     (they also have 1gbit ethernet cards)
> >>     speed is not good! ~60-80Mbit/sec
> >>   * if i stopped first speed test (2->12), second test speed is okay!
> >>     (2->20)
> >>   * but i make test from completely different VLANs, 2->12 and 20->30,
> >>     the result is so.
> > if you disable pf on vlan intefaces (set skip on vlan2/vlan12) do you
> > get better performace?
> > and after that for disable ipsec and try testing again... do you see
> > differences ?
> >
> >> This is firewall (openbsd) limitation, but we don't understand why?
> >>
> >> I know openbsd VLAN interface has a speed problem, this is it?
> > not in OpenBSD 6.6
> >
> >> I know it's so difficult to make a mistake from some information, what
> >> should we look at?
> >>
> > OpenBSD is great router and firewall that can do so much for you .. but
> > please you really need to rethink your hardware and setup ..
> >
> >
>
>

--

-

Past hissy-fits are not a predictor of future hissy-fits.
Nick Holland(06 Dec 2005)

To announce that there must be no criticism of the president,
or that we are to stand by the president, right or wrong, is not
only unpatriotic and servile, but is morally treasonable to
the American public.  - Theodore Roosevelt(1918)

[Szél Gábor]

alright, old hardver, but network speed is limited only 1 gbit! not
more! (not per VLAN, global 1 gbit limit!)
This is not hardvare preformance problem, i think this hardvare have
enough performace to 2x1Gbit parallel network connection.
We monitoring CPU, load, etc .... max load is 2-3, but not permanently

We reinstall backup firewall oBSD 6.6, and make some test.

Thx your reply!

--
Üdvözlettel,
Szél Gábor

WanTax Kft.
------------
tel.: [phone]
fax: [phone]
email: [hidden email]
web: http://wantax.hu
web: http://halozatom.hu



2019. 11. 12. 20:46 keltezéssel, Diana Eichert írta:

> Someone else mentioned your underlying system, Dell 2950, is ancient.
> It really is ancient, just because it links up @ 10G doesn't mean you
> are going to see anywhere near 10G
>
> On Tue, Nov 12, 2019 at 3:10 AM Szél Gábor <[hidden email]> wrote:
>> Dear Hrvoje, Theo,
>>
>> Thank you for your answers!
>>
>> answers to the questions:
>> -  who is parent interface for carp?  -> vlan  ( carp10 interface parent
>> vlan10 -> vlan10 interface  parent -> trunk0 )
>> - why vlan interfaces don't have ip address ? -> it wasn't needed! i
>> think vlan interface need only tag packages. Carp (over vlan) interface
>> have IP address.
>> - vether implies that you have bridge? -> yes whe have only one bridge
>> for bridget openvpn clients, but  we will eliminate it.
>>
>>
>> we will do the following:
>> - refresh our backup firewall to oBSD 6.6
>> - replace trunk interface with aggr
>> - remove bridge interface
>>
>> if there was an update finised, I'll write again!
>>
>> --
>> Regars
>> Szél Gábor
>>
>> WanTax Kft.
>> ------------
>> tel.: [phone]
>> fax: [phone]
>> email: [hidden email]
>> web: http://wantax.hu
>> web: http://halozatom.hu
>>
>>
>> 2019. 11. 11. 23:42 keltezéssel, Hrvoje Popovski írta:
>>> On 11.11.2019. 13:42, Szél Gábor wrote:
>>>> Hello @misc,
>>>>
>>> Hi,
>>>
>>>
>>>> We have an interesting problem, we run a lot of OpenBSD router/firewalls
>>>> in many places.
>>>>
>>>> We have a larger network than our client, 300-400 local wired or
>>>> wireless endpoint, 20+ VLAN, 20+ switches.
>>>> Network structure:
>>>>
>>>>    * Main switch - 2x Cisco Nexus 3k switch in HA mode (vPC dedicated
>>>>      2x40Gbit Peer link, keepalive link)
>>>>    * access switch - 10+ Cisco 3750X + C3KX-SM-10G 10Gbit module.
>>>>      some 3750x stacked (2 or 3 switch)
>>>>    * Main and access switches have redundant 10Gbit fiber link (LACP)
>>>>    * when is possible jumbo frame is enabled (mtu 9000)
>>>>
>>>> Firewall/router:
>>>>
>>>>    * 2x Dell 2950 - 2x Xeon X5460 (8 core), 8Gb Memory, 2x10Gbit SFP+
>>>>      network card
>>> hardware is really old, if you can, buy something newer
>>>
>>>
>>>>    * redundant design - CARP, pfsync, ifstated, etc .... master-backup>    configuration
>>>>    * HP NC550SFP network card, oce driver (mtu 9000)
>>> if you can change oce with ix. ixl is not so bad .. .
>>>
>>>
>>>>    * dual SFP+ port have LACP link to Nexus switches (2x10Gbit access
>>>>      link) - use openbsd trunk interface
>>>>    * all vlan used openbsd pseudo-device over trunk interface (VLANs not
>>>>      have have IP address, only up)
>>> update to openbsd 6.6 or snapshot and insted of trunk use aggr. why vlan
>>> interfaces don't have ip address ?
>>>
>>>
>>>>    * all network subnets defined in CARP interfaces, only managment VLAN
>>>>      have address on VLAN interface.
>>> who is parent interface for carp ?
>>>
>>>>    * some vether virtual interface for VPN, DNS, etc ...
>>> vether implies that you have bridge? bridge is slow..
>>>
>>>>    * some tun and tap interface for VPN
>>>>    * enc interface for ipsec
>>> ipsec is performance killer big time ... even for traffic that doesn't
>>> go through ipsec tunnel ..  if you can move ipsec or any vpn stuff to
>>> other boxes that you speed up your firewalls  ...
>>>
>>>>    * one bridge interface for openVPN (during termination)
>>> vether is in that bridge?
>>>
>>>>    * OpenBSD 6.3 64bit
>>> please, update boxes regularly.. you have carp and pfsync, you can do
>>> that without any problem ..
>>>
>>>> PF:
>>>>
>>>>    * global block rule (block all)
>>>>    * ruleset-optimization none
>>>>    * optimization aggressive
>>>>    * reassemble no
>>>>    * block-policy drop
>>>>    * scrub enabled
>>>>    * antispoof enabled
>>>>    * regulating traffic between subnets with pf pass in/out rules
>>>>    * pf.conf currently 1500+ lines
>>>>    * the number of connections during the day in PF 10 000+
>>>>
>>>> Problem:
>>>>
>>>> We see that network traffic is limited to 1Gbit on firewall. Not in one
>>>> link, not IP-to-IP, to the whole firewall!
>>>>
>>> yes ... ipsec, trunk, pf are for whole firewall .. and even if you have
>>> fastest box in the world you will not get performance that you want ..
>>>
>>>
>>>> example:
>>>>
>>>>    * i make test traffic form VLAN 2 to VLAN 12 witch iperf.
>>>>      test PC-s have 1Gbit ethernet cards.
>>>>      Speed is okay, ~800Mbit/sec
>>>>    * i make anoter traffic from VLAN 2 to VLan20 with iperf, from another
>>>>      PC-s
>>>>      (they also have 1gbit ethernet cards)
>>>>      speed is not good! ~60-80Mbit/sec
>>>>    * if i stopped first speed test (2->12), second test speed is okay!
>>>>      (2->20)
>>>>    * but i make test from completely different VLANs, 2->12 and 20->30,
>>>>      the result is so.
>>> if you disable pf on vlan intefaces (set skip on vlan2/vlan12) do you
>>> get better performace?
>>> and after that for disable ipsec and try testing again... do you see
>>> differences ?
>>>
>>>> This is firewall (openbsd) limitation, but we don't understand why?
>>>>
>>>> I know openbsd VLAN interface has a speed problem, this is it?
>>> not in OpenBSD 6.6
>>>
>>>> I know it's so difficult to make a mistake from some information, what
>>>> should we look at?
>>>>
>>> OpenBSD is great router and firewall that can do so much for you .. but
>>> please you really need to rethink your hardware and setup ..
>>>
>>>
>>
>

[Hrvoje Popovski]

On 12.11.2019. 10:54, Szél Gábor wrote:

> Dear Hrvoje, Theo,
>
> Thank you for your answers!
>
> answers to the questions:
> -  who is parent interface for carp?  -> vlan  ( carp10 interface parent
> vlan10 -> vlan10 interface  parent -> trunk0 )
> - why vlan interfaces don't have ip address ? -> it wasn't needed! i
> think vlan interface need only tag packages. Carp (over vlan) interface
> have IP address.

it's little strange to me to not have ip address on parent carp
interface, but if it works for you ... ok..

> - vether implies that you have bridge? -> yes whe have only one bridge
> for bridget openvpn clients, but  we will eliminate it.
>
>
> we will do the following:
> - refresh our backup firewall to oBSD 6.6
> - replace trunk interface with aggr
> - remove bridge interface

this is nice start to make you setup faster. big performance killer in
your setup is ipsec and old hardware. maybe oce(4) but i never tested
it, so i'm not sure ... if you can, change oce with ix, intel x520 is
not that expensive ..

bridge is slow, but only for traffic that goes through it. with ipsec,
the same second when tunnel is established, forwarding performance will
drop significantly on whole firewall ...

> if there was an update finised, I'll write again!

please do, i would like to hear

[Hrvoje Popovski]

On 13.11.2019. 10:59, Hrvoje Popovski wrote:

> On 12.11.2019. 10:54, Szél Gábor wrote:
>> Dear Hrvoje, Theo,
>>
>> Thank you for your answers!
>>
>> answers to the questions:
>> -  who is parent interface for carp?  -> vlan  ( carp10 interface parent
>> vlan10 -> vlan10 interface  parent -> trunk0 )
>> - why vlan interfaces don't have ip address ? -> it wasn't needed! i
>> think vlan interface need only tag packages. Carp (over vlan) interface
>> have IP address.
>
> it's little strange to me to not have ip address on parent carp
> interface, but if it works for you ... ok..
>
>> - vether implies that you have bridge? -> yes whe have only one bridge
>> for bridget openvpn clients, but  we will eliminate it.
>>
>>
>> we will do the following:
>> - refresh our backup firewall to oBSD 6.6
>> - replace trunk interface with aggr
>> - remove bridge interface
>
> this is nice start to make you setup faster. big performance killer in
> your setup is ipsec and old hardware. maybe oce(4) but i never tested
> it, so i'm not sure ... if you can, change oce with ix, intel x520 is
> not that expensive ..
>
> bridge is slow, but only for traffic that goes through it. with ipsec,
> the same second when tunnel is established, forwarding performance will
> drop significantly on whole firewall ...

i forgot numbers, so i did quick tests ..


forwarding over ix0 and ix1, pf and ipsec disabled
1.35Mpps

forwarding over ix0 and ix1, pf enabled, 500 UDP states
800Kpps

forwarding over ix0 and ix1, ipsec established over em0, pf disabled
800Kpps

forwarding over ix0 and ix1, ipsec established over em0, pf enabled, 500
UDP states
550Kpps



OpenBSD 6.6-current (GENERIC.MP) #453: Mon Nov 11 21:40:31 MST 2019
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 17115840512 (16322MB)
avail mem = 16584790016 (15816MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xcf42c000 (99 entries)
bios0: vendor Dell Inc. version "2.8.0" date 06/26/2019
bios0: Dell Inc. PowerEdge R620
acpi0 at bios0: ACPI 3.0
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP APIC SPCR HPET DMAR MCFG WD__ SLIC ERST HEST
BERT EINJ TCPA PC__ SRAT SSDT
acpi0: wakeup devices PCI0(S5)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 4 (boot processor)
cpu0: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.53 MHz, 06-3e-04
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 2, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 100MHz
cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
cpu1 at mainbus0: apid 6 (application processor)
cpu1: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 3, package 0
cpu2 at mainbus0: apid 8 (application processor)
cpu2: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 4, package 0
cpu3 at mainbus0: apid 16 (application processor)
cpu3: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 8, package 0
cpu4 at mainbus0: apid 18 (application processor)
cpu4: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 0, core 9, package 0
cpu5 at mainbus0: apid 20 (application processor)
cpu5: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 0, core 10, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
ioapic1 at mainbus0: apid 1 pa 0xfec3f000, version 20, 24 pins, remapped
acpihpet0 at acpi0: 14318179 Hz
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 1 (PEX1)
acpiprt2 at acpi0: bus -1 (PE1C)
acpiprt3 at acpi0: bus 3 (PEX2)
acpiprt4 at acpi0: bus 2 (PEX3)
acpiprt5 at acpi0: bus 4 (PEX4)
acpiprt6 at acpi0: bus -1 (PEX5)
acpiprt7 at acpi0: bus 8 (PEX6)
acpiprt8 at acpi0: bus 7 (PEX7)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
acpicpu4 at acpi0: C1(@1 halt!)
acpicpu5 at acpi0: C1(@1 halt!)
acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
acpicmos0 at acpi0
acpipci1 at acpi0 P0B1: 0x00000000 0x00000011 0x00000001
acpipci2 at acpi0 P1B1: 0x00000000 0x00000011 0x00000001
"PNP0C14" at acpi0 not configured
ipmi at mainbus0 not configured
cpu0: using VERW MDS workaround (except on vmm entry)
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel E5 v2 Host" rev 0x04
ppb0 at pci0 dev 1 function 0 "Intel E5 v2 PCIE" rev 0x04
pci1 at ppb0 bus 1
1:0:1: rom address conflict 0xd8000000/0x80000
ix0 at pci1 dev 0 function 0 "Intel 82599" rev 0x01: msi, address
ec:f4:bb:c8:e9:88
ix1 at pci1 dev 0 function 1 "Intel 82599" rev 0x01: msi, address
ec:f4:bb:c8:e9:8a
ppb1 at pci0 dev 2 function 0 "Intel E5 v2 PCIE" rev 0x04: msi
pci2 at ppb1 bus 3
ppb2 at pci0 dev 2 function 2 "Intel E5 v2 PCIE" rev 0x04
pci3 at ppb2 bus 2
mfi0 at pci3 dev 0 function 0 "Symbios Logic MegaRAID SAS2008" rev 0x03:
apic 1 int 10
mfi0: "PERC H310 Mini", firmware 20.13.3-0001
scsibus1 at mfi0: 16 targets
sd0 at scsibus1 targ 0 lun 0: <DELL, PERC H310, 2.12>
naa.6c81f660f14377001b0a11fd0658139c
sd0: 285568MB, 512 bytes/sector, 584843264 sectors
scsibus2 at mfi0: 256 targets
ppb3 at pci0 dev 3 function 0 "Intel E5 v2 PCIE" rev 0x04: msi
pci4 at ppb3 bus 4
4:0:1: rom address conflict 0xda000000/0x80000
4:0:2: rom address conflict 0xda000000/0x80000
4:0:3: rom address conflict 0xda000000/0x80000
ixl0 at pci4 dev 0 function 0 "Intel X710 SFP+" rev 0x01: port 3, FW
7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:68
ixl1 at pci4 dev 0 function 1 "Intel X710 SFP+" rev 0x01: port 2, FW
7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:69
ixl2 at pci4 dev 0 function 2 "Intel X710 SFP+" rev 0x01: port 0, FW
7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:6a
ixl3 at pci4 dev 0 function 3 "Intel X710 SFP+" rev 0x01: port 1, FW
7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:6b
"Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 0 not configured
"Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 1 not configured
"Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 2 not configured
"Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 3 not configured
"Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 4 not configured
"Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 5 not configured
"Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 6 not configured
"Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 7 not configured
"Intel E5 v2 Address Map" rev 0x04 at pci0 dev 5 function 0 not configured
"Intel E5 v2 IIO RAS" rev 0x04 at pci0 dev 5 function 2 not configured
ppb4 at pci0 dev 17 function 0 "Intel C600 Virtual PCIE" rev 0x05
pci5 at ppb4 bus 5
"Intel C600 MEI" rev 0x05 at pci0 dev 22 function 0 not configured
"Intel C600 MEI" rev 0x05 at pci0 dev 22 function 1 not configured
ehci0 at pci0 dev 26 function 0 "Intel C600 USB" rev 0x05: apic 0 int 23
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
ppb5 at pci0 dev 28 function 0 "Intel C600 PCIE" rev 0xb5
pci6 at ppb5 bus 6
ppb6 at pci0 dev 28 function 4 "Intel C600 PCIE" rev 0xb5
pci7 at ppb6 bus 7
em0 at pci7 dev 0 function 0 "Intel I350" rev 0x01: msi, address
ec:f4:bb:c8:e9:8c
em1 at pci7 dev 0 function 1 "Intel I350" rev 0x01: msi, address
ec:f4:bb:c8:e9:8d
ppb7 at pci0 dev 28 function 7 "Intel C600 PCIE" rev 0xb5
pci8 at ppb7 bus 8
ppb8 at pci8 dev 0 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
pci9 at ppb8 bus 9
ppb9 at pci9 dev 0 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
pci10 at ppb9 bus 10
ppb10 at pci10 dev 0 function 0 "Renesas SH7757 PCIE-PCI" rev 0x00
pci11 at ppb10 bus 11
vga1 at pci11 dev 0 function 0 "Matrox MGA G200eR" rev 0x00
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
ppb11 at pci9 dev 1 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
pci12 at ppb11 bus 12
ehci1 at pci0 dev 29 function 0 "Intel C600 USB" rev 0x05: apic 0 int 22
usb1 at ehci1: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
2.00/1.00 addr 1
ppb12 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa5
pci13 at ppb12 bus 13
pcib0 at pci0 dev 31 function 0 "Intel C600 LPC" rev 0x05
ahci0 at pci0 dev 31 function 2 "Intel C600 AHCI" rev 0x05: msi, AHCI 1.3
ahci0: port 4: 1.5Gb/s
scsibus3 at ahci0: 32 targets
cd0 at scsibus3 targ 4 lun 0: <TSSTcorp, DVD+-RW SU-208FB, D150> removable
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com1: console
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
pci14 at mainbus0 bus 63
"Intel E5 v2 QPI Link" rev 0x04 at pci14 dev 8 function 0 not configured
"Intel E5 v2 QPI Link" rev 0x04 at pci14 dev 9 function 0 not configured
"Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 0 not configured
"Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 1 not configured
"Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 2 not configured
"Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 3 not configured
"Intel E5 v2 UBOX" rev 0x04 at pci14 dev 11 function 0 not configured
"Intel E5 v2 UBOX" rev 0x04 at pci14 dev 11 function 3 not configured
"Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 0 not configured
"Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 1 not configured
"Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 2 not configured
"Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 3 not configured
"Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 4 not configured
"Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 0 not configured
"Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 1 not configured
"Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 2 not configured
"Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 3 not configured
"Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 4 not configured
"Intel E5 v2 Home Agent" rev 0x04 at pci14 dev 14 function 0 not configured
"Intel E5 v2 Home Agent" rev 0x04 at pci14 dev 14 function 1 not configured
"Intel E5 v2 TA" rev 0x04 at pci14 dev 15 function 0 not configured
"Intel E5 v2 RAS" rev 0x04 at pci14 dev 15 function 1 not configured
"Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 2 not configured
"Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 3 not configured
"Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 4 not configured
"Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 5 not configured
"Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 0 not configured
"Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 1 not configured
"Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 2 not configured
"Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 3 not configured
"Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 4 not configured
"Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 5 not configured
"Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 7 not configured
"Intel E5 v2 R2PCIE" rev 0x04 at pci14 dev 19 function 0 not configured
"Intel E5 v2 QPI Link Monitor" rev 0x04 at pci14 dev 19 function 1 not
configured
"Intel E5 v2 QPI" rev 0x04 at pci14 dev 19 function 4 not configured
"Intel E5 v2 QPI Link Monitor" rev 0x04 at pci14 dev 19 function 5 not
configured
"Intel E5 v2 SAD" rev 0x04 at pci14 dev 22 function 0 not configured
"Intel E5 v2 Broadcast" rev 0x04 at pci14 dev 22 function 1 not configured
"Intel E5 v2 Broadcast" rev 0x04 at pci14 dev 22 function 2 not configured
pci15 at mainbus0 bus 127
vmm0 at mainbus0: VMX/EPT
uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching
Hub" rev 2.00/0.00 addr 2
uhub3 at uhub2 port 6 configuration 1 interface 0 "no manufacturer
Gadget USB HUB" rev 2.00/0.00 addr 3
uhidev0 at uhub3 port 1 configuration 1 interface 0 "Avocent
Keyboard/Mouse Function" rev 2.00/0.00 addr 4
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub3 port 1 configuration 1 interface 1 "Avocent
Keyboard/Mouse Function" rev 2.00/0.00 addr 4
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
uhidev2 at uhub3 port 1 configuration 1 interface 2 "Avocent
Keyboard/Mouse Function" rev 2.00/0.00 addr 4
uhidev2: iclass 3/1
ums1 at uhidev2: 3 buttons, Z dir
wsmouse1 at ums1 mux 0
uhub4 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching
Hub" rev 2.00/0.00 addr 2
vscsi0 at root
scsibus4 at vscsi0: 256 targets
softraid0 at root
scsibus5 at softraid0: 256 targets
root on sd0a (84c319b65473c5e6.a) swap on sd0b dump on sd0b
ukbd0: was console keyboard
wskbd0 detached
ukbd0 detached
uhidev0 detached
wsmouse0 detached
ums0 detached
uhidev1 detached
wsmouse1 detached
ums1 detached
uhidev2 detached
uhidev0 at uhub3 port 1 configuration 1 interface 0 "Avocent
Keyboard/Mouse Function" rev 2.00/0.00 addr 4
uhidev0: iclass 3/1
ukbd0 at uhidev0: 8 variable keys, 6 key codes
wskbd0 at ukbd0: console keyboard, using wsdisplay0
uhidev1 at uhub3 port 1 configuration 1 interface 1 "Avocent
Keyboard/Mouse Function" rev 2.00/0.00 addr 4
uhidev1: iclass 3/1
ums0 at uhidev1: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
uhidev2 at uhub3 port 1 configuration 1 interface 2 "Avocent
Keyboard/Mouse Function" rev 2.00/0.00 addr 4
uhidev2: iclass 3/1
ums1 at uhidev2: 3 buttons, Z dir
wsmouse1 at ums1 mux 0

[Gregory Edigarov-5]

could you please do one more test:
"forwarding over ix0 and ix1, pf enabled, 5 tcp states"

On 13.11.19 12:52, Hrvoje Popovski wrote:

> On 13.11.2019. 10:59, Hrvoje Popovski wrote:
>> On 12.11.2019. 10:54, Szél Gábor wrote:
>>> Dear Hrvoje, Theo,
>>>
>>> Thank you for your answers!
>>>
>>> answers to the questions:
>>> -  who is parent interface for carp?  -> vlan  ( carp10 interface parent
>>> vlan10 -> vlan10 interface  parent -> trunk0 )
>>> - why vlan interfaces don't have ip address ? -> it wasn't needed! i
>>> think vlan interface need only tag packages. Carp (over vlan) interface
>>> have IP address.
>> it's little strange to me to not have ip address on parent carp
>> interface, but if it works for you ... ok..
>>
>>> - vether implies that you have bridge? -> yes whe have only one bridge
>>> for bridget openvpn clients, but  we will eliminate it.
>>>
>>>
>>> we will do the following:
>>> - refresh our backup firewall to oBSD 6.6
>>> - replace trunk interface with aggr
>>> - remove bridge interface
>> this is nice start to make you setup faster. big performance killer in
>> your setup is ipsec and old hardware. maybe oce(4) but i never tested
>> it, so i'm not sure ... if you can, change oce with ix, intel x520 is
>> not that expensive ..
>>
>> bridge is slow, but only for traffic that goes through it. with ipsec,
>> the same second when tunnel is established, forwarding performance will
>> drop significantly on whole firewall ...
>
> i forgot numbers, so i did quick tests ..
>
>
> forwarding over ix0 and ix1, pf and ipsec disabled
> 1.35Mpps
>
> forwarding over ix0 and ix1, pf enabled, 500 UDP states
> 800Kpps
>
> forwarding over ix0 and ix1, ipsec established over em0, pf disabled
> 800Kpps
>
> forwarding over ix0 and ix1, ipsec established over em0, pf enabled, 500
> UDP states
> 550Kpps
>
>
>
> OpenBSD 6.6-current (GENERIC.MP) #453: Mon Nov 11 21:40:31 MST 2019
>      [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 17115840512 (16322MB)
> avail mem = 16584790016 (15816MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xcf42c000 (99 entries)
> bios0: vendor Dell Inc. version "2.8.0" date 06/26/2019
> bios0: Dell Inc. PowerEdge R620
> acpi0 at bios0: ACPI 3.0
> acpi0: sleep states S0 S4 S5
> acpi0: tables DSDT FACP APIC SPCR HPET DMAR MCFG WD__ SLIC ERST HEST
> BERT EINJ TCPA PC__ SRAT SSDT
> acpi0: wakeup devices PCI0(S5)
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 4 (boot processor)
> cpu0: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.53 MHz, 06-3e-04
> cpu0:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu0: 256KB 64b/line 8-way L2 cache
> cpu0: smt 0, core 2, package 0
> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
> cpu0: apic clock running at 100MHz
> cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
> cpu1 at mainbus0: apid 6 (application processor)
> cpu1: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
> cpu1:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu1: 256KB 64b/line 8-way L2 cache
> cpu1: smt 0, core 3, package 0
> cpu2 at mainbus0: apid 8 (application processor)
> cpu2: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
> cpu2:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu2: 256KB 64b/line 8-way L2 cache
> cpu2: smt 0, core 4, package 0
> cpu3 at mainbus0: apid 16 (application processor)
> cpu3: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
> cpu3:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu3: 256KB 64b/line 8-way L2 cache
> cpu3: smt 0, core 8, package 0
> cpu4 at mainbus0: apid 18 (application processor)
> cpu4: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
> cpu4:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu4: 256KB 64b/line 8-way L2 cache
> cpu4: smt 0, core 9, package 0
> cpu5 at mainbus0: apid 20 (application processor)
> cpu5: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
> cpu5:
> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
> cpu5: 256KB 64b/line 8-way L2 cache
> cpu5: smt 0, core 10, package 0
> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
> ioapic1 at mainbus0: apid 1 pa 0xfec3f000, version 20, 24 pins, remapped
> acpihpet0 at acpi0: 14318179 Hz
> acpimcfg0 at acpi0
> acpimcfg0: addr 0xe0000000, bus 0-255
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpiprt1 at acpi0: bus 1 (PEX1)
> acpiprt2 at acpi0: bus -1 (PE1C)
> acpiprt3 at acpi0: bus 3 (PEX2)
> acpiprt4 at acpi0: bus 2 (PEX3)
> acpiprt5 at acpi0: bus 4 (PEX4)
> acpiprt6 at acpi0: bus -1 (PEX5)
> acpiprt7 at acpi0: bus 8 (PEX6)
> acpiprt8 at acpi0: bus 7 (PEX7)
> acpicpu0 at acpi0: C1(@1 halt!)
> acpicpu1 at acpi0: C1(@1 halt!)
> acpicpu2 at acpi0: C1(@1 halt!)
> acpicpu3 at acpi0: C1(@1 halt!)
> acpicpu4 at acpi0: C1(@1 halt!)
> acpicpu5 at acpi0: C1(@1 halt!)
> acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
> acpicmos0 at acpi0
> acpipci1 at acpi0 P0B1: 0x00000000 0x00000011 0x00000001
> acpipci2 at acpi0 P1B1: 0x00000000 0x00000011 0x00000001
> "PNP0C14" at acpi0 not configured
> ipmi at mainbus0 not configured
> cpu0: using VERW MDS workaround (except on vmm entry)
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel E5 v2 Host" rev 0x04
> ppb0 at pci0 dev 1 function 0 "Intel E5 v2 PCIE" rev 0x04
> pci1 at ppb0 bus 1
> 1:0:1: rom address conflict 0xd8000000/0x80000
> ix0 at pci1 dev 0 function 0 "Intel 82599" rev 0x01: msi, address
> ec:f4:bb:c8:e9:88
> ix1 at pci1 dev 0 function 1 "Intel 82599" rev 0x01: msi, address
> ec:f4:bb:c8:e9:8a
> ppb1 at pci0 dev 2 function 0 "Intel E5 v2 PCIE" rev 0x04: msi
> pci2 at ppb1 bus 3
> ppb2 at pci0 dev 2 function 2 "Intel E5 v2 PCIE" rev 0x04
> pci3 at ppb2 bus 2
> mfi0 at pci3 dev 0 function 0 "Symbios Logic MegaRAID SAS2008" rev 0x03:
> apic 1 int 10
> mfi0: "PERC H310 Mini", firmware 20.13.3-0001
> scsibus1 at mfi0: 16 targets
> sd0 at scsibus1 targ 0 lun 0: <DELL, PERC H310, 2.12>
> naa.6c81f660f14377001b0a11fd0658139c
> sd0: 285568MB, 512 bytes/sector, 584843264 sectors
> scsibus2 at mfi0: 256 targets
> ppb3 at pci0 dev 3 function 0 "Intel E5 v2 PCIE" rev 0x04: msi
> pci4 at ppb3 bus 4
> 4:0:1: rom address conflict 0xda000000/0x80000
> 4:0:2: rom address conflict 0xda000000/0x80000
> 4:0:3: rom address conflict 0xda000000/0x80000
> ixl0 at pci4 dev 0 function 0 "Intel X710 SFP+" rev 0x01: port 3, FW
> 7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:68
> ixl1 at pci4 dev 0 function 1 "Intel X710 SFP+" rev 0x01: port 2, FW
> 7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:69
> ixl2 at pci4 dev 0 function 2 "Intel X710 SFP+" rev 0x01: port 0, FW
> 7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:6a
> ixl3 at pci4 dev 0 function 3 "Intel X710 SFP+" rev 0x01: port 1, FW
> 7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:6b
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 0 not configured
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 1 not configured
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 2 not configured
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 3 not configured
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 4 not configured
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 5 not configured
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 6 not configured
> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 7 not configured
> "Intel E5 v2 Address Map" rev 0x04 at pci0 dev 5 function 0 not configured
> "Intel E5 v2 IIO RAS" rev 0x04 at pci0 dev 5 function 2 not configured
> ppb4 at pci0 dev 17 function 0 "Intel C600 Virtual PCIE" rev 0x05
> pci5 at ppb4 bus 5
> "Intel C600 MEI" rev 0x05 at pci0 dev 22 function 0 not configured
> "Intel C600 MEI" rev 0x05 at pci0 dev 22 function 1 not configured
> ehci0 at pci0 dev 26 function 0 "Intel C600 USB" rev 0x05: apic 0 int 23
> usb0 at ehci0: USB revision 2.0
> uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev
> 2.00/1.00 addr 1
> ppb5 at pci0 dev 28 function 0 "Intel C600 PCIE" rev 0xb5
> pci6 at ppb5 bus 6
> ppb6 at pci0 dev 28 function 4 "Intel C600 PCIE" rev 0xb5
> pci7 at ppb6 bus 7
> em0 at pci7 dev 0 function 0 "Intel I350" rev 0x01: msi, address
> ec:f4:bb:c8:e9:8c
> em1 at pci7 dev 0 function 1 "Intel I350" rev 0x01: msi, address
> ec:f4:bb:c8:e9:8d
> ppb7 at pci0 dev 28 function 7 "Intel C600 PCIE" rev 0xb5
> pci8 at ppb7 bus 8
> ppb8 at pci8 dev 0 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
> pci9 at ppb8 bus 9
> ppb9 at pci9 dev 0 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
> pci10 at ppb9 bus 10
> ppb10 at pci10 dev 0 function 0 "Renesas SH7757 PCIE-PCI" rev 0x00
> pci11 at ppb10 bus 11
> vga1 at pci11 dev 0 function 0 "Matrox MGA G200eR" rev 0x00
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> ppb11 at pci9 dev 1 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
> pci12 at ppb11 bus 12
> ehci1 at pci0 dev 29 function 0 "Intel C600 USB" rev 0x05: apic 0 int 22
> usb1 at ehci1: USB revision 2.0
> uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
> 2.00/1.00 addr 1
> ppb12 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa5
> pci13 at ppb12 bus 13
> pcib0 at pci0 dev 31 function 0 "Intel C600 LPC" rev 0x05
> ahci0 at pci0 dev 31 function 2 "Intel C600 AHCI" rev 0x05: msi, AHCI 1.3
> ahci0: port 4: 1.5Gb/s
> scsibus3 at ahci0: 32 targets
> cd0 at scsibus3 targ 4 lun 0: <TSSTcorp, DVD+-RW SU-208FB, D150> removable
> isa0 at pcib0
> isadma0 at isa0
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
> com1: console
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> pci14 at mainbus0 bus 63
> "Intel E5 v2 QPI Link" rev 0x04 at pci14 dev 8 function 0 not configured
> "Intel E5 v2 QPI Link" rev 0x04 at pci14 dev 9 function 0 not configured
> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 0 not configured
> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 1 not configured
> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 2 not configured
> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 3 not configured
> "Intel E5 v2 UBOX" rev 0x04 at pci14 dev 11 function 0 not configured
> "Intel E5 v2 UBOX" rev 0x04 at pci14 dev 11 function 3 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 0 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 1 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 2 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 3 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 4 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 0 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 1 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 2 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 3 not configured
> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 4 not configured
> "Intel E5 v2 Home Agent" rev 0x04 at pci14 dev 14 function 0 not configured
> "Intel E5 v2 Home Agent" rev 0x04 at pci14 dev 14 function 1 not configured
> "Intel E5 v2 TA" rev 0x04 at pci14 dev 15 function 0 not configured
> "Intel E5 v2 RAS" rev 0x04 at pci14 dev 15 function 1 not configured
> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 2 not configured
> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 3 not configured
> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 4 not configured
> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 5 not configured
> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 0 not configured
> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 1 not configured
> "Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 2 not configured
> "Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 3 not configured
> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 4 not configured
> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 5 not configured
> "Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 7 not configured
> "Intel E5 v2 R2PCIE" rev 0x04 at pci14 dev 19 function 0 not configured
> "Intel E5 v2 QPI Link Monitor" rev 0x04 at pci14 dev 19 function 1 not
> configured
> "Intel E5 v2 QPI" rev 0x04 at pci14 dev 19 function 4 not configured
> "Intel E5 v2 QPI Link Monitor" rev 0x04 at pci14 dev 19 function 5 not
> configured
> "Intel E5 v2 SAD" rev 0x04 at pci14 dev 22 function 0 not configured
> "Intel E5 v2 Broadcast" rev 0x04 at pci14 dev 22 function 1 not configured
> "Intel E5 v2 Broadcast" rev 0x04 at pci14 dev 22 function 2 not configured
> pci15 at mainbus0 bus 127
> vmm0 at mainbus0: VMX/EPT
> uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching
> Hub" rev 2.00/0.00 addr 2
> uhub3 at uhub2 port 6 configuration 1 interface 0 "no manufacturer
> Gadget USB HUB" rev 2.00/0.00 addr 3
> uhidev0 at uhub3 port 1 configuration 1 interface 0 "Avocent
> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 variable keys, 6 key codes
> wskbd0 at ukbd0: console keyboard, using wsdisplay0
> uhidev1 at uhub3 port 1 configuration 1 interface 1 "Avocent
> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
> uhidev1: iclass 3/1
> ums0 at uhidev1: 3 buttons, Z dir
> wsmouse0 at ums0 mux 0
> uhidev2 at uhub3 port 1 configuration 1 interface 2 "Avocent
> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
> uhidev2: iclass 3/1
> ums1 at uhidev2: 3 buttons, Z dir
> wsmouse1 at ums1 mux 0
> uhub4 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching
> Hub" rev 2.00/0.00 addr 2
> vscsi0 at root
> scsibus4 at vscsi0: 256 targets
> softraid0 at root
> scsibus5 at softraid0: 256 targets
> root on sd0a (84c319b65473c5e6.a) swap on sd0b dump on sd0b
> ukbd0: was console keyboard
> wskbd0 detached
> ukbd0 detached
> uhidev0 detached
> wsmouse0 detached
> ums0 detached
> uhidev1 detached
> wsmouse1 detached
> ums1 detached
> uhidev2 detached
> uhidev0 at uhub3 port 1 configuration 1 interface 0 "Avocent
> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
> uhidev0: iclass 3/1
> ukbd0 at uhidev0: 8 variable keys, 6 key codes
> wskbd0 at ukbd0: console keyboard, using wsdisplay0
> uhidev1 at uhub3 port 1 configuration 1 interface 1 "Avocent
> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
> uhidev1: iclass 3/1
> ums0 at uhidev1: 3 buttons, Z dir
> wsmouse0 at ums0 mux 0
> uhidev2 at uhub3 port 1 configuration 1 interface 2 "Avocent
> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
> uhidev2: iclass 3/1
> ums1 at uhidev2: 3 buttons, Z dir
> wsmouse1 at ums1 mux 0
>

[Hrvoje Popovski]

On 13.11.2019. 16:37, Gregory Edigarov wrote:
> could you please do one more test:
> "forwarding over ix0 and ix1, pf enabled, 5 tcp states"

with this generator i can't use tcp. generally pps with 5 or 500000
states are more or less same ... problem with tcp testing is that i
can't get precise pps numbers ...

and.... only for you :)
with iperf3 (8 tcp streams) on client boxes i'm getting this results ...

forwarding over ix0 and ix1, pf and ipsec disabled
9.40Gbps

forwarding over ix0 and ix1, pf enabled, 8 tcp streams
7.40Gbps

forwarding over ix0 and ix1, ipsec established over em0, pf disabled
8.10Gbps

forwarding over ix0 and ix1, ipsec established over em0, pf enabled, 8
TCP streams
5.25Gbps

> On 13.11.19 12:52, Hrvoje Popovski wrote:
>> On 13.11.2019. 10:59, Hrvoje Popovski wrote:
>>> On 12.11.2019. 10:54, Szél Gábor wrote:
>>>> Dear Hrvoje, Theo,
>>>>
>>>> Thank you for your answers!
>>>>
>>>> answers to the questions:
>>>> -  who is parent interface for carp?  -> vlan  ( carp10 interface
>>>> parent
>>>> vlan10 -> vlan10 interface  parent -> trunk0 )
>>>> - why vlan interfaces don't have ip address ? -> it wasn't needed! i
>>>> think vlan interface need only tag packages. Carp (over vlan) interface
>>>> have IP address.
>>> it's little strange to me to not have ip address on parent carp
>>> interface, but if it works for you ... ok..
>>>
>>>> - vether implies that you have bridge? -> yes whe have only one bridge
>>>> for bridget openvpn clients, but  we will eliminate it.
>>>>
>>>>
>>>> we will do the following:
>>>> - refresh our backup firewall to oBSD 6.6
>>>> - replace trunk interface with aggr
>>>> - remove bridge interface
>>> this is nice start to make you setup faster. big performance killer in
>>> your setup is ipsec and old hardware. maybe oce(4) but i never tested
>>> it, so i'm not sure ... if you can, change oce with ix, intel x520 is
>>> not that expensive ..
>>>
>>> bridge is slow, but only for traffic that goes through it. with ipsec,
>>> the same second when tunnel is established, forwarding performance will
>>> drop significantly on whole firewall ...
>>
>> i forgot numbers, so i did quick tests ..
>>
>>
>> forwarding over ix0 and ix1, pf and ipsec disabled
>> 1.35Mpps
>>
>> forwarding over ix0 and ix1, pf enabled, 500 UDP states
>> 800Kpps
>>
>> forwarding over ix0 and ix1, ipsec established over em0, pf disabled
>> 800Kpps
>>
>> forwarding over ix0 and ix1, ipsec established over em0, pf enabled, 500
>> UDP states
>> 550Kpps
>>
>>
>>
>> OpenBSD 6.6-current (GENERIC.MP) #453: Mon Nov 11 21:40:31 MST 2019
>>      [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>> real mem = 17115840512 (16322MB)
>> avail mem = 16584790016 (15816MB)
>> mpath0 at root
>> scsibus0 at mpath0: 256 targets
>> mainbus0 at root
>> bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xcf42c000 (99 entries)
>> bios0: vendor Dell Inc. version "2.8.0" date 06/26/2019
>> bios0: Dell Inc. PowerEdge R620
>> acpi0 at bios0: ACPI 3.0
>> acpi0: sleep states S0 S4 S5
>> acpi0: tables DSDT FACP APIC SPCR HPET DMAR MCFG WD__ SLIC ERST HEST
>> BERT EINJ TCPA PC__ SRAT SSDT
>> acpi0: wakeup devices PCI0(S5)
>> acpitimer0 at acpi0: 3579545 Hz, 24 bits
>> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
>> cpu0 at mainbus0: apid 4 (boot processor)
>> cpu0: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.53 MHz, 06-3e-04
>> cpu0:
>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>>
>> cpu0: 256KB 64b/line 8-way L2 cache
>> cpu0: smt 0, core 2, package 0
>> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
>> cpu0: apic clock running at 100MHz
>> cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
>> cpu1 at mainbus0: apid 6 (application processor)
>> cpu1: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
>> cpu1:
>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>>
>> cpu1: 256KB 64b/line 8-way L2 cache
>> cpu1: smt 0, core 3, package 0
>> cpu2 at mainbus0: apid 8 (application processor)
>> cpu2: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
>> cpu2:
>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>>
>> cpu2: 256KB 64b/line 8-way L2 cache
>> cpu2: smt 0, core 4, package 0
>> cpu3 at mainbus0: apid 16 (application processor)
>> cpu3: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
>> cpu3:
>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>>
>> cpu3: 256KB 64b/line 8-way L2 cache
>> cpu3: smt 0, core 8, package 0
>> cpu4 at mainbus0: apid 18 (application processor)
>> cpu4: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
>> cpu4:
>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>>
>> cpu4: 256KB 64b/line 8-way L2 cache
>> cpu4: smt 0, core 9, package 0
>> cpu5 at mainbus0: apid 20 (application processor)
>> cpu5: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
>> cpu5:
>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>>
>> cpu5: 256KB 64b/line 8-way L2 cache
>> cpu5: smt 0, core 10, package 0
>> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
>> ioapic1 at mainbus0: apid 1 pa 0xfec3f000, version 20, 24 pins, remapped
>> acpihpet0 at acpi0: 14318179 Hz
>> acpimcfg0 at acpi0
>> acpimcfg0: addr 0xe0000000, bus 0-255
>> acpiprt0 at acpi0: bus 0 (PCI0)
>> acpiprt1 at acpi0: bus 1 (PEX1)
>> acpiprt2 at acpi0: bus -1 (PE1C)
>> acpiprt3 at acpi0: bus 3 (PEX2)
>> acpiprt4 at acpi0: bus 2 (PEX3)
>> acpiprt5 at acpi0: bus 4 (PEX4)
>> acpiprt6 at acpi0: bus -1 (PEX5)
>> acpiprt7 at acpi0: bus 8 (PEX6)
>> acpiprt8 at acpi0: bus 7 (PEX7)
>> acpicpu0 at acpi0: C1(@1 halt!)
>> acpicpu1 at acpi0: C1(@1 halt!)
>> acpicpu2 at acpi0: C1(@1 halt!)
>> acpicpu3 at acpi0: C1(@1 halt!)
>> acpicpu4 at acpi0: C1(@1 halt!)
>> acpicpu5 at acpi0: C1(@1 halt!)
>> acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
>> acpicmos0 at acpi0
>> acpipci1 at acpi0 P0B1: 0x00000000 0x00000011 0x00000001
>> acpipci2 at acpi0 P1B1: 0x00000000 0x00000011 0x00000001
>> "PNP0C14" at acpi0 not configured
>> ipmi at mainbus0 not configured
>> cpu0: using VERW MDS workaround (except on vmm entry)
>> pci0 at mainbus0 bus 0
>> pchb0 at pci0 dev 0 function 0 "Intel E5 v2 Host" rev 0x04
>> ppb0 at pci0 dev 1 function 0 "Intel E5 v2 PCIE" rev 0x04
>> pci1 at ppb0 bus 1
>> 1:0:1: rom address conflict 0xd8000000/0x80000
>> ix0 at pci1 dev 0 function 0 "Intel 82599" rev 0x01: msi, address
>> ec:f4:bb:c8:e9:88
>> ix1 at pci1 dev 0 function 1 "Intel 82599" rev 0x01: msi, address
>> ec:f4:bb:c8:e9:8a
>> ppb1 at pci0 dev 2 function 0 "Intel E5 v2 PCIE" rev 0x04: msi
>> pci2 at ppb1 bus 3
>> ppb2 at pci0 dev 2 function 2 "Intel E5 v2 PCIE" rev 0x04
>> pci3 at ppb2 bus 2
>> mfi0 at pci3 dev 0 function 0 "Symbios Logic MegaRAID SAS2008" rev 0x03:
>> apic 1 int 10
>> mfi0: "PERC H310 Mini", firmware 20.13.3-0001
>> scsibus1 at mfi0: 16 targets
>> sd0 at scsibus1 targ 0 lun 0: <DELL, PERC H310, 2.12>
>> naa.6c81f660f14377001b0a11fd0658139c
>> sd0: 285568MB, 512 bytes/sector, 584843264 sectors
>> scsibus2 at mfi0: 256 targets
>> ppb3 at pci0 dev 3 function 0 "Intel E5 v2 PCIE" rev 0x04: msi
>> pci4 at ppb3 bus 4
>> 4:0:1: rom address conflict 0xda000000/0x80000
>> 4:0:2: rom address conflict 0xda000000/0x80000
>> 4:0:3: rom address conflict 0xda000000/0x80000
>> ixl0 at pci4 dev 0 function 0 "Intel X710 SFP+" rev 0x01: port 3, FW
>> 7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:68
>> ixl1 at pci4 dev 0 function 1 "Intel X710 SFP+" rev 0x01: port 2, FW
>> 7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:69
>> ixl2 at pci4 dev 0 function 2 "Intel X710 SFP+" rev 0x01: port 0, FW
>> 7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:6a
>> ixl3 at pci4 dev 0 function 3 "Intel X710 SFP+" rev 0x01: port 1, FW
>> 7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:6b
>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 0 not configured
>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 1 not configured
>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 2 not configured
>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 3 not configured
>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 4 not configured
>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 5 not configured
>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 6 not configured
>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 7 not configured
>> "Intel E5 v2 Address Map" rev 0x04 at pci0 dev 5 function 0 not
>> configured
>> "Intel E5 v2 IIO RAS" rev 0x04 at pci0 dev 5 function 2 not configured
>> ppb4 at pci0 dev 17 function 0 "Intel C600 Virtual PCIE" rev 0x05
>> pci5 at ppb4 bus 5
>> "Intel C600 MEI" rev 0x05 at pci0 dev 22 function 0 not configured
>> "Intel C600 MEI" rev 0x05 at pci0 dev 22 function 1 not configured
>> ehci0 at pci0 dev 26 function 0 "Intel C600 USB" rev 0x05: apic 0 int 23
>> usb0 at ehci0: USB revision 2.0
>> uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev
>> 2.00/1.00 addr 1
>> ppb5 at pci0 dev 28 function 0 "Intel C600 PCIE" rev 0xb5
>> pci6 at ppb5 bus 6
>> ppb6 at pci0 dev 28 function 4 "Intel C600 PCIE" rev 0xb5
>> pci7 at ppb6 bus 7
>> em0 at pci7 dev 0 function 0 "Intel I350" rev 0x01: msi, address
>> ec:f4:bb:c8:e9:8c
>> em1 at pci7 dev 0 function 1 "Intel I350" rev 0x01: msi, address
>> ec:f4:bb:c8:e9:8d
>> ppb7 at pci0 dev 28 function 7 "Intel C600 PCIE" rev 0xb5
>> pci8 at ppb7 bus 8
>> ppb8 at pci8 dev 0 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
>> pci9 at ppb8 bus 9
>> ppb9 at pci9 dev 0 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
>> pci10 at ppb9 bus 10
>> ppb10 at pci10 dev 0 function 0 "Renesas SH7757 PCIE-PCI" rev 0x00
>> pci11 at ppb10 bus 11
>> vga1 at pci11 dev 0 function 0 "Matrox MGA G200eR" rev 0x00
>> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
>> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
>> ppb11 at pci9 dev 1 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
>> pci12 at ppb11 bus 12
>> ehci1 at pci0 dev 29 function 0 "Intel C600 USB" rev 0x05: apic 0 int 22
>> usb1 at ehci1: USB revision 2.0
>> uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
>> 2.00/1.00 addr 1
>> ppb12 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa5
>> pci13 at ppb12 bus 13
>> pcib0 at pci0 dev 31 function 0 "Intel C600 LPC" rev 0x05
>> ahci0 at pci0 dev 31 function 2 "Intel C600 AHCI" rev 0x05: msi, AHCI 1.3
>> ahci0: port 4: 1.5Gb/s
>> scsibus3 at ahci0: 32 targets
>> cd0 at scsibus3 targ 4 lun 0: <TSSTcorp, DVD+-RW SU-208FB, D150>
>> removable
>> isa0 at pcib0
>> isadma0 at isa0
>> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
>> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
>> com1: console
>> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
>> pcppi0 at isa0 port 0x61
>> spkr0 at pcppi0
>> pci14 at mainbus0 bus 63
>> "Intel E5 v2 QPI Link" rev 0x04 at pci14 dev 8 function 0 not configured
>> "Intel E5 v2 QPI Link" rev 0x04 at pci14 dev 9 function 0 not configured
>> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 0 not configured
>> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 1 not configured
>> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 2 not configured
>> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 3 not configured
>> "Intel E5 v2 UBOX" rev 0x04 at pci14 dev 11 function 0 not configured
>> "Intel E5 v2 UBOX" rev 0x04 at pci14 dev 11 function 3 not configured
>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 0 not configured
>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 1 not configured
>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 2 not configured
>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 3 not configured
>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 4 not configured
>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 0 not configured
>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 1 not configured
>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 2 not configured
>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 3 not configured
>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 4 not configured
>> "Intel E5 v2 Home Agent" rev 0x04 at pci14 dev 14 function 0 not
>> configured
>> "Intel E5 v2 Home Agent" rev 0x04 at pci14 dev 14 function 1 not
>> configured
>> "Intel E5 v2 TA" rev 0x04 at pci14 dev 15 function 0 not configured
>> "Intel E5 v2 RAS" rev 0x04 at pci14 dev 15 function 1 not configured
>> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 2 not configured
>> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 3 not configured
>> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 4 not configured
>> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 5 not configured
>> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 0 not configured
>> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 1 not configured
>> "Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 2 not configured
>> "Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 3 not configured
>> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 4 not configured
>> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 5 not configured
>> "Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 7 not configured
>> "Intel E5 v2 R2PCIE" rev 0x04 at pci14 dev 19 function 0 not configured
>> "Intel E5 v2 QPI Link Monitor" rev 0x04 at pci14 dev 19 function 1 not
>> configured
>> "Intel E5 v2 QPI" rev 0x04 at pci14 dev 19 function 4 not configured
>> "Intel E5 v2 QPI Link Monitor" rev 0x04 at pci14 dev 19 function 5 not
>> configured
>> "Intel E5 v2 SAD" rev 0x04 at pci14 dev 22 function 0 not configured
>> "Intel E5 v2 Broadcast" rev 0x04 at pci14 dev 22 function 1 not
>> configured
>> "Intel E5 v2 Broadcast" rev 0x04 at pci14 dev 22 function 2 not
>> configured
>> pci15 at mainbus0 bus 127
>> vmm0 at mainbus0: VMX/EPT
>> uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching
>> Hub" rev 2.00/0.00 addr 2
>> uhub3 at uhub2 port 6 configuration 1 interface 0 "no manufacturer
>> Gadget USB HUB" rev 2.00/0.00 addr 3
>> uhidev0 at uhub3 port 1 configuration 1 interface 0 "Avocent
>> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
>> uhidev0: iclass 3/1
>> ukbd0 at uhidev0: 8 variable keys, 6 key codes
>> wskbd0 at ukbd0: console keyboard, using wsdisplay0
>> uhidev1 at uhub3 port 1 configuration 1 interface 1 "Avocent
>> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
>> uhidev1: iclass 3/1
>> ums0 at uhidev1: 3 buttons, Z dir
>> wsmouse0 at ums0 mux 0
>> uhidev2 at uhub3 port 1 configuration 1 interface 2 "Avocent
>> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
>> uhidev2: iclass 3/1
>> ums1 at uhidev2: 3 buttons, Z dir
>> wsmouse1 at ums1 mux 0
>> uhub4 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching
>> Hub" rev 2.00/0.00 addr 2
>> vscsi0 at root
>> scsibus4 at vscsi0: 256 targets
>> softraid0 at root
>> scsibus5 at softraid0: 256 targets
>> root on sd0a (84c319b65473c5e6.a) swap on sd0b dump on sd0b
>> ukbd0: was console keyboard
>> wskbd0 detached
>> ukbd0 detached
>> uhidev0 detached
>> wsmouse0 detached
>> ums0 detached
>> uhidev1 detached
>> wsmouse1 detached
>> ums1 detached
>> uhidev2 detached
>> uhidev0 at uhub3 port 1 configuration 1 interface 0 "Avocent
>> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
>> uhidev0: iclass 3/1
>> ukbd0 at uhidev0: 8 variable keys, 6 key codes
>> wskbd0 at ukbd0: console keyboard, using wsdisplay0
>> uhidev1 at uhub3 port 1 configuration 1 interface 1 "Avocent
>> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
>> uhidev1: iclass 3/1
>> ums0 at uhidev1: 3 buttons, Z dir
>> wsmouse0 at ums0 mux 0
>> uhidev2 at uhub3 port 1 configuration 1 interface 2 "Avocent
>> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
>> uhidev2: iclass 3/1
>> ums1 at uhidev2: 3 buttons, Z dir
>> wsmouse1 at ums1 mux 0
>>
>

[Gregory Edigarov-5]

On 13.11.19 21:18, Hrvoje Popovski wrote:

> On 13.11.2019. 16:37, Gregory Edigarov wrote:
>> could you please do one more test:
>> "forwarding over ix0 and ix1, pf enabled, 5 tcp states"
> with this generator i can't use tcp. generally pps with 5 or 500000
> states are more or less same ... problem with tcp testing is that i
> can't get precise pps numbers ...
>
> and.... only for you :)
> with iperf3 (8 tcp streams) on client boxes i'm getting this results ...
>
> forwarding over ix0 and ix1, pf and ipsec disabled
> 9.40Gbps
>
> forwarding over ix0 and ix1, pf enabled, 8 tcp streams
> 7.40Gbps
>
> forwarding over ix0 and ix1, ipsec established over em0, pf disabled
> 8.10Gbps
>
> forwarding over ix0 and ix1, ipsec established over em0, pf enabled, 8
> TCP streams
> 5.25Gbps

thanks, Hrvoje

>
>> On 13.11.19 12:52, Hrvoje Popovski wrote:
>>> On 13.11.2019. 10:59, Hrvoje Popovski wrote:
>>>> On 12.11.2019. 10:54, Szél Gábor wrote:
>>>>> Dear Hrvoje, Theo,
>>>>>
>>>>> Thank you for your answers!
>>>>>
>>>>> answers to the questions:
>>>>> -  who is parent interface for carp?  -> vlan  ( carp10 interface
>>>>> parent
>>>>> vlan10 -> vlan10 interface  parent -> trunk0 )
>>>>> - why vlan interfaces don't have ip address ? -> it wasn't needed! i
>>>>> think vlan interface need only tag packages. Carp (over vlan) interface
>>>>> have IP address.
>>>> it's little strange to me to not have ip address on parent carp
>>>> interface, but if it works for you ... ok..
>>>>
>>>>> - vether implies that you have bridge? -> yes whe have only one bridge
>>>>> for bridget openvpn clients, but  we will eliminate it.
>>>>>
>>>>>
>>>>> we will do the following:
>>>>> - refresh our backup firewall to oBSD 6.6
>>>>> - replace trunk interface with aggr
>>>>> - remove bridge interface
>>>> this is nice start to make you setup faster. big performance killer in
>>>> your setup is ipsec and old hardware. maybe oce(4) but i never tested
>>>> it, so i'm not sure ... if you can, change oce with ix, intel x520 is
>>>> not that expensive ..
>>>>
>>>> bridge is slow, but only for traffic that goes through it. with ipsec,
>>>> the same second when tunnel is established, forwarding performance will
>>>> drop significantly on whole firewall ...
>>> i forgot numbers, so i did quick tests ..
>>>
>>>
>>> forwarding over ix0 and ix1, pf and ipsec disabled
>>> 1.35Mpps
>>>
>>> forwarding over ix0 and ix1, pf enabled, 500 UDP states
>>> 800Kpps
>>>
>>> forwarding over ix0 and ix1, ipsec established over em0, pf disabled
>>> 800Kpps
>>>
>>> forwarding over ix0 and ix1, ipsec established over em0, pf enabled, 500
>>> UDP states
>>> 550Kpps
>>>
>>>
>>>
>>> OpenBSD 6.6-current (GENERIC.MP) #453: Mon Nov 11 21:40:31 MST 2019
>>>       [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>>> real mem = 17115840512 (16322MB)
>>> avail mem = 16584790016 (15816MB)
>>> mpath0 at root
>>> scsibus0 at mpath0: 256 targets
>>> mainbus0 at root
>>> bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xcf42c000 (99 entries)
>>> bios0: vendor Dell Inc. version "2.8.0" date 06/26/2019
>>> bios0: Dell Inc. PowerEdge R620
>>> acpi0 at bios0: ACPI 3.0
>>> acpi0: sleep states S0 S4 S5
>>> acpi0: tables DSDT FACP APIC SPCR HPET DMAR MCFG WD__ SLIC ERST HEST
>>> BERT EINJ TCPA PC__ SRAT SSDT
>>> acpi0: wakeup devices PCI0(S5)
>>> acpitimer0 at acpi0: 3579545 Hz, 24 bits
>>> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
>>> cpu0 at mainbus0: apid 4 (boot processor)
>>> cpu0: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.53 MHz, 06-3e-04
>>> cpu0:
>>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>>>
>>> cpu0: 256KB 64b/line 8-way L2 cache
>>> cpu0: smt 0, core 2, package 0
>>> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
>>> cpu0: apic clock running at 100MHz
>>> cpu0: mwait min=64, max=64, C-substates=0.2.1.1, IBE
>>> cpu1 at mainbus0: apid 6 (application processor)
>>> cpu1: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
>>> cpu1:
>>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>>>
>>> cpu1: 256KB 64b/line 8-way L2 cache
>>> cpu1: smt 0, core 3, package 0
>>> cpu2 at mainbus0: apid 8 (application processor)
>>> cpu2: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
>>> cpu2:
>>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>>>
>>> cpu2: 256KB 64b/line 8-way L2 cache
>>> cpu2: smt 0, core 4, package 0
>>> cpu3 at mainbus0: apid 16 (application processor)
>>> cpu3: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
>>> cpu3:
>>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>>>
>>> cpu3: 256KB 64b/line 8-way L2 cache
>>> cpu3: smt 0, core 8, package 0
>>> cpu4 at mainbus0: apid 18 (application processor)
>>> cpu4: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
>>> cpu4:
>>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>>>
>>> cpu4: 256KB 64b/line 8-way L2 cache
>>> cpu4: smt 0, core 9, package 0
>>> cpu5 at mainbus0: apid 20 (application processor)
>>> cpu5: Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz, 3600.01 MHz, 06-3e-04
>>> cpu5:
>>> FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,DEADLINE,AES,XSAVE,AVX,F16C,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,PERF,ITSC,FSGSBASE,SMEP,ERMS,MD_CLEAR,IBRS,IBPB,STIBP,L1DF,SSBD,SENSOR,ARAT,XSAVEOPT,MELTDOWN
>>>
>>> cpu5: 256KB 64b/line 8-way L2 cache
>>> cpu5: smt 0, core 10, package 0
>>> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 20, 24 pins
>>> ioapic1 at mainbus0: apid 1 pa 0xfec3f000, version 20, 24 pins, remapped
>>> acpihpet0 at acpi0: 14318179 Hz
>>> acpimcfg0 at acpi0
>>> acpimcfg0: addr 0xe0000000, bus 0-255
>>> acpiprt0 at acpi0: bus 0 (PCI0)
>>> acpiprt1 at acpi0: bus 1 (PEX1)
>>> acpiprt2 at acpi0: bus -1 (PE1C)
>>> acpiprt3 at acpi0: bus 3 (PEX2)
>>> acpiprt4 at acpi0: bus 2 (PEX3)
>>> acpiprt5 at acpi0: bus 4 (PEX4)
>>> acpiprt6 at acpi0: bus -1 (PEX5)
>>> acpiprt7 at acpi0: bus 8 (PEX6)
>>> acpiprt8 at acpi0: bus 7 (PEX7)
>>> acpicpu0 at acpi0: C1(@1 halt!)
>>> acpicpu1 at acpi0: C1(@1 halt!)
>>> acpicpu2 at acpi0: C1(@1 halt!)
>>> acpicpu3 at acpi0: C1(@1 halt!)
>>> acpicpu4 at acpi0: C1(@1 halt!)
>>> acpicpu5 at acpi0: C1(@1 halt!)
>>> acpipci0 at acpi0 PCI0: 0x00000000 0x00000011 0x00000001
>>> acpicmos0 at acpi0
>>> acpipci1 at acpi0 P0B1: 0x00000000 0x00000011 0x00000001
>>> acpipci2 at acpi0 P1B1: 0x00000000 0x00000011 0x00000001
>>> "PNP0C14" at acpi0 not configured
>>> ipmi at mainbus0 not configured
>>> cpu0: using VERW MDS workaround (except on vmm entry)
>>> pci0 at mainbus0 bus 0
>>> pchb0 at pci0 dev 0 function 0 "Intel E5 v2 Host" rev 0x04
>>> ppb0 at pci0 dev 1 function 0 "Intel E5 v2 PCIE" rev 0x04
>>> pci1 at ppb0 bus 1
>>> 1:0:1: rom address conflict 0xd8000000/0x80000
>>> ix0 at pci1 dev 0 function 0 "Intel 82599" rev 0x01: msi, address
>>> ec:f4:bb:c8:e9:88
>>> ix1 at pci1 dev 0 function 1 "Intel 82599" rev 0x01: msi, address
>>> ec:f4:bb:c8:e9:8a
>>> ppb1 at pci0 dev 2 function 0 "Intel E5 v2 PCIE" rev 0x04: msi
>>> pci2 at ppb1 bus 3
>>> ppb2 at pci0 dev 2 function 2 "Intel E5 v2 PCIE" rev 0x04
>>> pci3 at ppb2 bus 2
>>> mfi0 at pci3 dev 0 function 0 "Symbios Logic MegaRAID SAS2008" rev 0x03:
>>> apic 1 int 10
>>> mfi0: "PERC H310 Mini", firmware 20.13.3-0001
>>> scsibus1 at mfi0: 16 targets
>>> sd0 at scsibus1 targ 0 lun 0: <DELL, PERC H310, 2.12>
>>> naa.6c81f660f14377001b0a11fd0658139c
>>> sd0: 285568MB, 512 bytes/sector, 584843264 sectors
>>> scsibus2 at mfi0: 256 targets
>>> ppb3 at pci0 dev 3 function 0 "Intel E5 v2 PCIE" rev 0x04: msi
>>> pci4 at ppb3 bus 4
>>> 4:0:1: rom address conflict 0xda000000/0x80000
>>> 4:0:2: rom address conflict 0xda000000/0x80000
>>> 4:0:3: rom address conflict 0xda000000/0x80000
>>> ixl0 at pci4 dev 0 function 0 "Intel X710 SFP+" rev 0x01: port 3, FW
>>> 7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:68
>>> ixl1 at pci4 dev 0 function 1 "Intel X710 SFP+" rev 0x01: port 2, FW
>>> 7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:69
>>> ixl2 at pci4 dev 0 function 2 "Intel X710 SFP+" rev 0x01: port 0, FW
>>> 7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:6a
>>> ixl3 at pci4 dev 0 function 3 "Intel X710 SFP+" rev 0x01: port 1, FW
>>> 7.0.50775 API 1.8, msi, address 3c:fd:fe:b3:20:6b
>>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 0 not configured
>>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 1 not configured
>>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 2 not configured
>>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 3 not configured
>>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 4 not configured
>>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 5 not configured
>>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 6 not configured
>>> "Intel E5 v2 I/OAT" rev 0x04 at pci0 dev 4 function 7 not configured
>>> "Intel E5 v2 Address Map" rev 0x04 at pci0 dev 5 function 0 not
>>> configured
>>> "Intel E5 v2 IIO RAS" rev 0x04 at pci0 dev 5 function 2 not configured
>>> ppb4 at pci0 dev 17 function 0 "Intel C600 Virtual PCIE" rev 0x05
>>> pci5 at ppb4 bus 5
>>> "Intel C600 MEI" rev 0x05 at pci0 dev 22 function 0 not configured
>>> "Intel C600 MEI" rev 0x05 at pci0 dev 22 function 1 not configured
>>> ehci0 at pci0 dev 26 function 0 "Intel C600 USB" rev 0x05: apic 0 int 23
>>> usb0 at ehci0: USB revision 2.0
>>> uhub0 at usb0 configuration 1 interface 0 "Intel EHCI root hub" rev
>>> 2.00/1.00 addr 1
>>> ppb5 at pci0 dev 28 function 0 "Intel C600 PCIE" rev 0xb5
>>> pci6 at ppb5 bus 6
>>> ppb6 at pci0 dev 28 function 4 "Intel C600 PCIE" rev 0xb5
>>> pci7 at ppb6 bus 7
>>> em0 at pci7 dev 0 function 0 "Intel I350" rev 0x01: msi, address
>>> ec:f4:bb:c8:e9:8c
>>> em1 at pci7 dev 0 function 1 "Intel I350" rev 0x01: msi, address
>>> ec:f4:bb:c8:e9:8d
>>> ppb7 at pci0 dev 28 function 7 "Intel C600 PCIE" rev 0xb5
>>> pci8 at ppb7 bus 8
>>> ppb8 at pci8 dev 0 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
>>> pci9 at ppb8 bus 9
>>> ppb9 at pci9 dev 0 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
>>> pci10 at ppb9 bus 10
>>> ppb10 at pci10 dev 0 function 0 "Renesas SH7757 PCIE-PCI" rev 0x00
>>> pci11 at ppb10 bus 11
>>> vga1 at pci11 dev 0 function 0 "Matrox MGA G200eR" rev 0x00
>>> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
>>> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
>>> ppb11 at pci9 dev 1 function 0 "Renesas SH7757 PCIE Switch" rev 0x00
>>> pci12 at ppb11 bus 12
>>> ehci1 at pci0 dev 29 function 0 "Intel C600 USB" rev 0x05: apic 0 int 22
>>> usb1 at ehci1: USB revision 2.0
>>> uhub1 at usb1 configuration 1 interface 0 "Intel EHCI root hub" rev
>>> 2.00/1.00 addr 1
>>> ppb12 at pci0 dev 30 function 0 "Intel 82801BA Hub-to-PCI" rev 0xa5
>>> pci13 at ppb12 bus 13
>>> pcib0 at pci0 dev 31 function 0 "Intel C600 LPC" rev 0x05
>>> ahci0 at pci0 dev 31 function 2 "Intel C600 AHCI" rev 0x05: msi, AHCI 1.3
>>> ahci0: port 4: 1.5Gb/s
>>> scsibus3 at ahci0: 32 targets
>>> cd0 at scsibus3 targ 4 lun 0: <TSSTcorp, DVD+-RW SU-208FB, D150>
>>> removable
>>> isa0 at pcib0
>>> isadma0 at isa0
>>> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
>>> com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
>>> com1: console
>>> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
>>> pcppi0 at isa0 port 0x61
>>> spkr0 at pcppi0
>>> pci14 at mainbus0 bus 63
>>> "Intel E5 v2 QPI Link" rev 0x04 at pci14 dev 8 function 0 not configured
>>> "Intel E5 v2 QPI Link" rev 0x04 at pci14 dev 9 function 0 not configured
>>> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 0 not configured
>>> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 1 not configured
>>> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 2 not configured
>>> "Intel E5 v2 PCU" rev 0x04 at pci14 dev 10 function 3 not configured
>>> "Intel E5 v2 UBOX" rev 0x04 at pci14 dev 11 function 0 not configured
>>> "Intel E5 v2 UBOX" rev 0x04 at pci14 dev 11 function 3 not configured
>>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 0 not configured
>>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 1 not configured
>>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 2 not configured
>>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 3 not configured
>>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 12 function 4 not configured
>>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 0 not configured
>>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 1 not configured
>>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 2 not configured
>>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 3 not configured
>>> "Intel E5 v2 Unicast" rev 0x04 at pci14 dev 13 function 4 not configured
>>> "Intel E5 v2 Home Agent" rev 0x04 at pci14 dev 14 function 0 not
>>> configured
>>> "Intel E5 v2 Home Agent" rev 0x04 at pci14 dev 14 function 1 not
>>> configured
>>> "Intel E5 v2 TA" rev 0x04 at pci14 dev 15 function 0 not configured
>>> "Intel E5 v2 RAS" rev 0x04 at pci14 dev 15 function 1 not configured
>>> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 2 not configured
>>> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 3 not configured
>>> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 4 not configured
>>> "Intel E5 v2 TAD" rev 0x04 at pci14 dev 15 function 5 not configured
>>> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 0 not configured
>>> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 1 not configured
>>> "Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 2 not configured
>>> "Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 3 not configured
>>> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 4 not configured
>>> "Intel E5 v2 Thermal" rev 0x04 at pci14 dev 16 function 5 not configured
>>> "Intel E5 v2 Error" rev 0x04 at pci14 dev 16 function 7 not configured
>>> "Intel E5 v2 R2PCIE" rev 0x04 at pci14 dev 19 function 0 not configured
>>> "Intel E5 v2 QPI Link Monitor" rev 0x04 at pci14 dev 19 function 1 not
>>> configured
>>> "Intel E5 v2 QPI" rev 0x04 at pci14 dev 19 function 4 not configured
>>> "Intel E5 v2 QPI Link Monitor" rev 0x04 at pci14 dev 19 function 5 not
>>> configured
>>> "Intel E5 v2 SAD" rev 0x04 at pci14 dev 22 function 0 not configured
>>> "Intel E5 v2 Broadcast" rev 0x04 at pci14 dev 22 function 1 not
>>> configured
>>> "Intel E5 v2 Broadcast" rev 0x04 at pci14 dev 22 function 2 not
>>> configured
>>> pci15 at mainbus0 bus 127
>>> vmm0 at mainbus0: VMX/EPT
>>> uhub2 at uhub0 port 1 configuration 1 interface 0 "Intel Rate Matching
>>> Hub" rev 2.00/0.00 addr 2
>>> uhub3 at uhub2 port 6 configuration 1 interface 0 "no manufacturer
>>> Gadget USB HUB" rev 2.00/0.00 addr 3
>>> uhidev0 at uhub3 port 1 configuration 1 interface 0 "Avocent
>>> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
>>> uhidev0: iclass 3/1
>>> ukbd0 at uhidev0: 8 variable keys, 6 key codes
>>> wskbd0 at ukbd0: console keyboard, using wsdisplay0
>>> uhidev1 at uhub3 port 1 configuration 1 interface 1 "Avocent
>>> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
>>> uhidev1: iclass 3/1
>>> ums0 at uhidev1: 3 buttons, Z dir
>>> wsmouse0 at ums0 mux 0
>>> uhidev2 at uhub3 port 1 configuration 1 interface 2 "Avocent
>>> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
>>> uhidev2: iclass 3/1
>>> ums1 at uhidev2: 3 buttons, Z dir
>>> wsmouse1 at ums1 mux 0
>>> uhub4 at uhub1 port 1 configuration 1 interface 0 "Intel Rate Matching
>>> Hub" rev 2.00/0.00 addr 2
>>> vscsi0 at root
>>> scsibus4 at vscsi0: 256 targets
>>> softraid0 at root
>>> scsibus5 at softraid0: 256 targets
>>> root on sd0a (84c319b65473c5e6.a) swap on sd0b dump on sd0b
>>> ukbd0: was console keyboard
>>> wskbd0 detached
>>> ukbd0 detached
>>> uhidev0 detached
>>> wsmouse0 detached
>>> ums0 detached
>>> uhidev1 detached
>>> wsmouse1 detached
>>> ums1 detached
>>> uhidev2 detached
>>> uhidev0 at uhub3 port 1 configuration 1 interface 0 "Avocent
>>> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
>>> uhidev0: iclass 3/1
>>> ukbd0 at uhidev0: 8 variable keys, 6 key codes
>>> wskbd0 at ukbd0: console keyboard, using wsdisplay0
>>> uhidev1 at uhub3 port 1 configuration 1 interface 1 "Avocent
>>> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
>>> uhidev1: iclass 3/1
>>> ums0 at uhidev1: 3 buttons, Z dir
>>> wsmouse0 at ums0 mux 0
>>> uhidev2 at uhub3 port 1 configuration 1 interface 2 "Avocent
>>> Keyboard/Mouse Function" rev 2.00/0.00 addr 4
>>> uhidev2: iclass 3/1
>>> ums1 at uhidev2: 3 buttons, Z dir
>>> wsmouse1 at ums1 mux 0
>>>

[Hrvoje Popovski]

On 12.11.2019. 10:54, Szél Gábor wrote:

> Dear Hrvoje, Theo,
>
> Thank you for your answers!
>
> answers to the questions:
> -  who is parent interface for carp?  -> vlan  ( carp10 interface parent
> vlan10 -> vlan10 interface  parent -> trunk0 )
> - why vlan interfaces don't have ip address ? -> it wasn't needed! i
> think vlan interface need only tag packages. Carp (over vlan) interface
> have IP address.
> - vether implies that you have bridge? -> yes whe have only one bridge
> for bridget openvpn clients, but  we will eliminate it.
>
>
> we will do the following:
> - refresh our backup firewall to oBSD 6.6
> - replace trunk interface with aggr
> - remove bridge interface
>
> if there was an update finised, I'll write again!
>

Hi,

if you still have bridge and you don't need spanning-tree, try veb
instead. I'm getting 1.95Mpps over veb vs 500Kpps over bridge on 6 x
E5-2643 v2 @ 3.50GHz, 3600.48 MHz.

And of course .. big thanks to dlg@ who wrote it ..

[csszep]

Hi!

I miss something , or veb(4) ifconfig bits not yet commited ?

OpenBSD 6.9-beta (GENERIC.MP) #358: Wed Feb 24 17:11:53 MST 2021
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP


 ifconfig veb0 create
ifconfig: SIOCIFCREATE: Invalid argument





Hrvoje Popovski <[hidden email]> ezt írta (időpont: 2021. febr. 26., P,
8:43):

> On 12.11.2019. 10:54, Szél Gábor wrote:
> > Dear Hrvoje, Theo,
> >
> > Thank you for your answers!
> >
> > answers to the questions:
> > -  who is parent interface for carp?  -> vlan  ( carp10 interface parent
> > vlan10 -> vlan10 interface  parent -> trunk0 )
> > - why vlan interfaces don't have ip address ? -> it wasn't needed! i
> > think vlan interface need only tag packages. Carp (over vlan) interface
> > have IP address.
> > - vether implies that you have bridge? -> yes whe have only one bridge
> > for bridget openvpn clients, but  we will eliminate it.
> >
> >
> > we will do the following:
> > - refresh our backup firewall to oBSD 6.6
> > - replace trunk interface with aggr
> > - remove bridge interface
> >
> > if there was an update finised, I'll write again!
> >
>
> Hi,
>
> if you still have bridge and you don't need spanning-tree, try veb
> instead. I'm getting 1.95Mpps over veb vs 500Kpps over bridge on 6 x
> E5-2643 v2 @ 3.50GHz, 3600.48 MHz.
>
> And of course .. big thanks to dlg@ who wrote it ..
>
>

[Hrvoje Popovski]

On 26.2.2021. 9:00, csszep wrote:

> Hi!
>
> I miss something , or veb(4) ifconfig bits not yet commited ?
>
> OpenBSD 6.9-beta (GENERIC.MP) #358: Wed Feb 24 17:11:53 MST 2021
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
>
>  ifconfig veb0 create
> ifconfig: SIOCIFCREATE: Invalid argument
>
>

it this latest snapshot ?

[csszep]

Hi!

Latest snapshot works. Thx.

OpenBSD 6.9-beta (GENERIC.MP) #360: Thu Feb 25 11:53:45 MST 2021
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

ifconfig veb0
veb0: flags=8802<BROADCAST,SIMPLEX,MULTICAST>
        index 8 llprio 3
        groups: veb
        Addresses (max cache: 100, timeout: 240):




Hrvoje Popovski <[hidden email]> ezt írta (időpont: 2021. febr. 26., P,
9:16):

> On 26.2.2021. 9:00, csszep wrote:
> > Hi!
> >
> > I miss something , or veb(4) ifconfig bits not yet commited ?
> >
> > OpenBSD 6.9-beta (GENERIC.MP) #358: Wed Feb 24 17:11:53 MST 2021
> >     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> >
> >
> >  ifconfig veb0 create
> > ifconfig: SIOCIFCREATE: Invalid argument
> >
> >
>
>
> it this latest snapshot ?
>