Strange behavior when I try to use lladdr

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Strange behavior when I try to use lladdr

Denis Davydov
Hello, all.

I decided to reinstall OpenBSD to a newer version on my VMware ESXi
cluster. So I deleted an old router and start the new one using the old
configuration, except that I add lladdr parameter with the old MAC address
to the external interface to avoid blocking traffic on the port of the
Internet provider.

After applying the configuration and rebooting, I've got packet loss. The
strange thing is that packets started passing when I turned on BPF on
another terminal:

terminal[1]:
router1# uname -a
OpenBSD router1 6.7 GENERIC.MP#182 amd64
router1# pfctl -d
pfctl: pf not enabled
router1# ping -c3 185.1.162.100
PING 185.1.162.100 (185.1.162.100): 56 data bytes

--- 185.1.162.100 ping statistics ---
3 packets transmitted, 0 packets received, 100.0% packet loss
router1# arping -i vmx2 -c3 185.1.162.100
ARPING 185.1.162.100
Timeout
Timeout
Timeout

--- 185.1.162.100 statistics ---
3 packets transmitted, 0 packets received, 100% unanswered (0 extra)

Now, I run a tcpdump on another terminal and don't turn it off:

terminal[2]:
router1# tcpdump -ni vmx2 -e net 185.1.162.0/24
tcpdump: listening on vmx2, link-type EN10MB
07:55:47.123666 68:b5:99:c5:2a:c0 00:50:56:92:d1:18 0800 85:
185.1.162.100.55992 > 185.1.162.2.179: P 1989291079:1989291098(19) ack
284218081 win 229 <nop,nop,timestamp 448227401 881035409>: BGP (KEEPALIVE)
(DF) [tos 0xc0] [ttl 1]
07:55:47.318977 00:50:56:92:d1:18 68:b5:99:c5:2a:c0 0800 66:
185.1.162.2.179 > 185.1.162.100.55992: . ack 19 win 543 <nop,nop,timestamp
881035415 448227401> [tos 0xc0] [ttl 1]
...

Let's back on terminal[1] and run ping:

router1# ping -c3 185.1.162.100
PING 185.1.162.100 (185.1.162.100): 56 data bytes
64 bytes from 185.1.162.100: icmp_seq=0 ttl=64 time=0.357 ms
64 bytes from 185.1.162.100: icmp_seq=1 ttl=64 time=0.376 ms
64 bytes from 185.1.162.100: icmp_seq=2 ttl=64 time=0.399 ms

--- 185.1.162.100 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/std-dev = 0.357/0.377/0.399/0.017 ms
router1# arping -i vmx2 -c3 185.1.162.100
ARPING 185.1.162.100
60 bytes from 68:b5:99:c5:2a:c0 (185.1.162.100): index=0 time=320.700 usec
60 bytes from 68:b5:99:c5:2a:c0 (185.1.162.100): index=1 time=368.795 usec
60 bytes from 68:b5:99:c5:2a:c0 (185.1.162.100): index=2 time=381.975 usec

--- 185.1.162.100 statistics ---
3 packets transmitted, 3 packets received,   0% unanswered (0 extra)
rtt min/avg/max/std-dev = 0.321/0.357/0.382/0.026 ms

Now if I will stop tcpdump on terminal[2] I'll get packet loss again. This
is a weird behavior. What could be wrong?

This router is a virtual machine on VMware ESXi server with a network
adapter type VMXNET3, which is mapped to the vmx2 interface in guest OS
(OpenBSD 6.7):

router1# grep vmx2 /var/run/dmesg.boot | tail -1
vmx2 at pci19 dev 0 function 0 "VMware VMXNET3" rev 0x01: apic 4 int 16,
address 00:50:56:92:02:51
router1# cat /etc/hostname.vmx2
inet 185.1.162.3 255.255.255.0 NONE description "Linx-IX" lladdr
"00:50:56:92:c7:27"
router1# ifconfig vmx2
vmx2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
lladdr 00:50:56:92:c7:27
description: Linx-IX
index 3 priority 0 llprio 3
media: Ethernet autoselect (10GbaseT)
status: active
inet 185.1.162.3 netmask 0xffffff00 broadcast 185.1.162.255
router1#

The VMXNET3 interface belongs to vSwitch which contains allowable policy
ruleset:

[root@esxi5:~] esxcli system version get
   Product: VMware ESXi
   Version: 6.0.0
   Build: Releasebuild-2494585
   Update: 0
   Patch: 0
[root@esxi5:~] esxcli network vswitch standard policy security get -v
vSwitch3
   Allow Promiscuous: true
   Allow MAC Address Change: true
   Allow Forged Transmits: true
[root@esxi5:~]

Don't know if it helps to understand what's going on.
My dmesg.boot is here
https://gist.github.com/dynax60/b1d7e89b0f1b592b2d7bd0fd99b0272b

I removed the lladdr parameter from /etc/hostname.vmx2 and asked the ISP
network engeneer to clear the old MAC address 00:50:56:92:c7:27 on
port-security of the Cisco network switch. After that, I rebooted the
router and it worked as it should.

I think to avoid subsequent contacts with ISP with the request to clear
MAC-address, the best way would be to use lladdr to keep an old MAC address
on interface, because reinstalling the virtual machine on VMware ESXi
server will entail a change in the mac address on the interface. The
strange thing is that this does not work for me. I don't know why.

P.S. offtopic: I turn to the developers: I tearfully ask you to add python
to the base installation. This would allow the configuration to be rolled
automatically using Ansible right after install an OS. It would simplify
the task of configuring OpenBSD on remote hosts.

--
Regards,
Denis
Reply | Threaded
Open this post in threaded view
|

Re: Strange behavior when I try to use lladdr

Markus Wernig
On 5/22/20 12:12 PM, Денис Давыдов wrote:

> I decided to reinstall OpenBSD to a newer version on my VMware ESXi
> cluster. So I deleted an old router and start the new one using the old
> configuration, except that I add lladdr parameter with the old MAC address

Last I looked into it (some years ago) VMware did not allow to manually
set the adapter MAC address in the guest to addresses from some
hardcoded ranges, among which the VMware OUI 00:50:56. According to [1]
this is still the case today, they also specify the range there that can
be used.

> Now if I will stop tcpdump on terminal[2] I'll get packet loss again.
> This is a weird behavior. What could be wrong?

tcpdump by default puts the interface in promiscuous mode, which is why
it picked up frames not addressed to the lladr you set, but also to
00:50:56:92:d1:18, which seems to be the MAC that VMware has assigned to
the adapter.

/m

[1]
https://docs.vmware.com/en/VMware-vSphere/6.0/com.vmware.vsphere.troubleshooting.doc/GUID-7F723748-E7B8-48B9-A773-3822C514684B.html

Reply | Threaded
Open this post in threaded view
|

Re: Strange behavior when I try to use lladdr

Stuart Henderson
In reply to this post by Denis Davydov
On 2020-05-22, Денис Давыдов <[hidden email]> wrote:
> P.S. offtopic: I turn to the developers: I tearfully ask you to add python
> to the base installation. This would allow the configuration to be rolled
> automatically using Ansible right after install an OS. It would simplify
> the task of configuring OpenBSD on remote hosts.

It would be hard or perhaps impossible to deal with Python and its
dependencies in base. I suggest adding it via a combination of siteXX.tgz
and rc.firsttime. For example, create install.site containing

echo 'pkg_add python3 >> /etc/rc.firsttime'

Add it to site67.tgz, place that on a local mirror of the main OpenBSD
distribution in the release directory (/pub/OpenBSD/6.7/amd64, etc),
and add it to index.txt.

When you select that mirror in the installer it will then show up in
the list of file sets to install, it is unpacked and install.site runs
at the end, the entry is added to rc.firsttime, and on first boot
Python is installed. (You can't install packages directly from
install.site due to the restricted environment on the ramdisk kernel).

This can also be worked into autoinstall if you're doing PXE boot (or
you can create a modified CD/USB image with autoinstall added to it).
If you're installing on VMs it might be easier to use Packer to create
them and have it automatically enter keyboard commands to run on first
boot.


Reply | Threaded
Open this post in threaded view
|

Re: Strange behavior when I try to use lladdr

Thomas L.
In reply to this post by Denis Davydov
On Fri, 22 May 2020 13:12:15 +0300
Денис Давыдов <[hidden email]> wrote:

> P.S. offtopic: I turn to the developers: I tearfully ask you to add
> python to the base installation. This would allow the configuration
> to be rolled automatically using Ansible right after install an OS.
> It would simplify the task of configuring OpenBSD on remote hosts.

I solved this by having a task
- name: install python3
  raw: pkg_add python3
at the beginning (raw doesn't need python on the target maschine).

Kind regards,

Thomas