VMs as real hosts on the same network

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

VMs as real hosts on the same network

mabi
Hello,

I am trying out VMM on an OpenBSD 6.4 server which has the following network interfaces defined:

[bnx0]+[bnx1]-->[trunk0]-->[vlan2]
[bnx0]+[bnx1]-->[trunk0]-->[vlan6]-->[bridge6]

The vlan2 is for the internal (management) network and vlan6 for the public (internet) network. I manage my server from vlan2 and would like to have my virtual machines on vlan6 which uses public IP addresses. For that purpose I have setup my /etc/hostname.* files as such:

hostname.bnx0 + hostname.bnx1:
up

hostname.trunk0:
trunkproto failover trunkport bnx0 trunkport bnx1 up

hostname.vlan2:
inet 192.168.1.5 255.255.255.0 192.168.1.255 vnetid 2 parent trunk0 description "private"

hostname.vlan6:
vnetid 6 parent trunk0 description "public" up

hostname.bridge6:
add vlan6

I am actually using Option 4 from the Networking chapter in the  virtualization FAQ (https://www.openbsd.org/faq/faq16.html) just that my setup has a redundant link (trunk0) and a VLAN (vlan6). So in theory that should work but unfortunately when I start a VM to install OpenBSD 6.4 from the bsd.rd boot file I do not have any network connectivity. I tried with DHCP first and in that case on the DHCP server I see the DHCPDISCOVER and DHCPOFFER requests/answer but there is never a DHCPACK. Then I tried assigning a static IP directly but still no network connectivity. I can't ping the default gateway of that public network. Checking with tcpdump on the firewall I can see the ARP who-has request and the ARP reply back the the VM but again it seems like the VM does not get it.

Here is my vm.conf conf file:

switch "uplink_vlan6" {
        interface bridge6
}

vm "example" {
        disable
        memory 2G
        boot "/home/admin/bsd.rd"
        disk "/var/vmm/example.qcow2"

        interface {
                switch "uplink_vlan6"
                lladdr fe:e1:bb:01:01:01
        }
}

I have also totally disabled pf on that OpenBSD VMM server but that did not change anything (I am using the default pf.conf from the installation)

Any ideas what I might be doing wrong or forgetting?

Regards,
Mabi

Reply | Threaded
Open this post in threaded view
|

Re: VMs as real hosts on the same network

Mischa-2

> On 7 Dec 2018, at 11:35, mabi <[hidden email]> wrote:
>
> Hello,
>
> I am trying out VMM on an OpenBSD 6.4 server which has the following network interfaces defined:
>
> [bnx0]+[bnx1]-->[trunk0]-->[vlan2]
> [bnx0]+[bnx1]-->[trunk0]-->[vlan6]-->[bridge6]
>
> The vlan2 is for the internal (management) network and vlan6 for the public (internet) network. I manage my server from vlan2 and would like to have my virtual machines on vlan6 which uses public IP addresses. For that purpose I have setup my /etc/hostname.* files as such:
>
> hostname.bnx0 + hostname.bnx1:
> up
>
> hostname.trunk0:
> trunkproto failover trunkport bnx0 trunkport bnx1 up
>
> hostname.vlan2:
> inet 192.168.1.5 255.255.255.0 192.168.1.255 vnetid 2 parent trunk0 description "private"
>
> hostname.vlan6:
> vnetid 6 parent trunk0 description "public" up
>
> hostname.bridge6:
> add vlan6
>

It might be as easy as adding: up

# cat /etc/hostname.bridge6
add vlan6
up

By default the bridge interface is not brought up.
You can also run: ifconfig bridge6 up

This will most likely be the "problem".

Mischa

> I am actually using Option 4 from the Networking chapter in the  virtualization FAQ (https://www.openbsd.org/faq/faq16.html) just that my setup has a redundant link (trunk0) and a VLAN (vlan6). So in theory that should work but unfortunately when I start a VM to install OpenBSD 6.4 from the bsd.rd boot file I do not have any network connectivity. I tried with DHCP first and in that case on the DHCP server I see the DHCPDISCOVER and DHCPOFFER requests/answer but there is never a DHCPACK. Then I tried assigning a static IP directly but still no network connectivity. I can't ping the default gateway of that public network. Checking with tcpdump on the firewall I can see the ARP who-has request and the ARP reply back the the VM but again it seems like the VM does not get it.
>
> Here is my vm.conf conf file:
>
> switch "uplink_vlan6" {
>        interface bridge6
> }
>
> vm "example" {
>        disable
>        memory 2G
>        boot "/home/admin/bsd.rd"
>        disk "/var/vmm/example.qcow2"
>
>        interface {
>                switch "uplink_vlan6"
>                lladdr fe:e1:bb:01:01:01
>        }
> }
>
> I have also totally disabled pf on that OpenBSD VMM server but that did not change anything (I am using the default pf.conf from the installation)
>
> Any ideas what I might be doing wrong or forgetting?
>
> Regards,
> Mabi
>

Reply | Threaded
Open this post in threaded view
|

Re: VMs as real hosts on the same network

mabi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, December 7, 2018 11:43 AM, Mischa <[hidden email]> wrote:

> It might be as easy as adding: up
>
> cat /etc/hostname.bridge6
>
> ==========================
>
> add vlan6
> up
>
> By default the bridge interface is not brought up.
> You can also run: ifconfig bridge6 up

Good idea and I added "up" to my hostname.bridge6 file but it looks like it was already up (at least by doing an ifconfig bridge6 shows the "UP" flag). Neverthless to be on the safe side I rebooted the server but still not connectivity on the vlan6/bridge6 network for the VMs.

On the bridge6 interface I can see the DHCP request with tcpdump when the OpenBSD installer in the VM tries to fetch an IP address with DHCP:

11:59:35.672258 0.0.0.0.68 > 255.255.255.255.67:  xid:0xbafb375b [|bootp] [tos 0x10]

Then on the DHCP server I can see the following in loop:

Dec  7 12:00:27 dhcpsrv dhcpd[18917]: DHCPDISCOVER from fe:e1:bb:01:01:01 via XXX.XXX.XXX.1
Dec  7 12:00:27 dhcpsrv dhcpd[18917]: DHCPOFFER on XXX.XXX.XXX.101 to fe:e1:bb:01:01:01 via XXX.XXX.XXX.1

The IP address ending with .1 is the gateway on my public network and the one ending with .101 is the IP which should be assigned to my OpenBSD VM.

It seems like the traffic is not flowing back to the VM itself.

I just found a very interesting behaviour by running tcpdump on pretty much all interfaces of my server to analyze the traffic at different levels and BINGO: as soon as I run tcpdump on my trunk0 interface the DHCP request goes through and my VM has network connectivity! But as soon as I stop tcpdump on the trunk interface: no more network connectivity...

Now as far as I know running tcpdump enables promiscous mode (PROMISC flag on the interface) and this should the reason why it works.

But now what does it mean for my setup, do I need to enable promiscuous mode on my trunk interface manually? and if yes how can I do that?

Reply | Threaded
Open this post in threaded view
|

Re: VMs as real hosts on the same network

Mischa-2


> On 7 Dec 2018, at 12:32, mabi <[hidden email]> wrote:
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Friday, December 7, 2018 11:43 AM, Mischa <[hidden email]> wrote:
>
>> It might be as easy as adding: up
>>
>> cat /etc/hostname.bridge6
>>
>> ==========================
>>
>> add vlan6
>> up
>>
>> By default the bridge interface is not brought up.
>> You can also run: ifconfig bridge6 up
>
> Good idea and I added "up" to my hostname.bridge6 file but it looks like it was already up (at least by doing an ifconfig bridge6 shows the "UP" flag). Neverthless to be on the safe side I rebooted the server but still not connectivity on the vlan6/bridge6 network for the VMs.
>
> On the bridge6 interface I can see the DHCP request with tcpdump when the OpenBSD installer in the VM tries to fetch an IP address with DHCP:
>
> 11:59:35.672258 0.0.0.0.68 > 255.255.255.255.67:  xid:0xbafb375b [|bootp] [tos 0x10]
>
> Then on the DHCP server I can see the following in loop:
>
> Dec  7 12:00:27 dhcpsrv dhcpd[18917]: DHCPDISCOVER from fe:e1:bb:01:01:01 via XXX.XXX.XXX.1
> Dec  7 12:00:27 dhcpsrv dhcpd[18917]: DHCPOFFER on XXX.XXX.XXX.101 to fe:e1:bb:01:01:01 via XXX.XXX.XXX.1
>
> The IP address ending with .1 is the gateway on my public network and the one ending with .101 is the IP which should be assigned to my OpenBSD VM.
>
> It seems like the traffic is not flowing back to the VM itself.
>
> I just found a very interesting behaviour by running tcpdump on pretty much all interfaces of my server to analyze the traffic at different levels and BINGO: as soon as I run tcpdump on my trunk0 interface the DHCP request goes through and my VM has network connectivity! But as soon as I stop tcpdump on the trunk interface: no more network connectivity...
>
> Now as far as I know running tcpdump enables promiscous mode (PROMISC flag on the interface) and this should the reason why it works.
>
> But now what does it mean for my setup, do I need to enable promiscuous mode on my trunk interface manually? and if yes how can I do that?
>

The VLAN does require an IP address as far as I am aware.

Mischa



Reply | Threaded
Open this post in threaded view
|

Re: VMs as real hosts on the same network

Martin Sukany
could you post here your /etc/pf.conf rules?

Dne 07. 12. 18 v 12:40 Mischa napsal(a):

>
>> On 7 Dec 2018, at 12:32, mabi <[hidden email]> wrote:
>>
>> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
>> On Friday, December 7, 2018 11:43 AM, Mischa <[hidden email]> wrote:
>>
>>> It might be as easy as adding: up
>>>
>>> cat /etc/hostname.bridge6
>>>
>>> ==========================
>>>
>>> add vlan6
>>> up
>>>
>>> By default the bridge interface is not brought up.
>>> You can also run: ifconfig bridge6 up
>> Good idea and I added "up" to my hostname.bridge6 file but it looks like it was already up (at least by doing an ifconfig bridge6 shows the "UP" flag). Neverthless to be on the safe side I rebooted the server but still not connectivity on the vlan6/bridge6 network for the VMs.
>>
>> On the bridge6 interface I can see the DHCP request with tcpdump when the OpenBSD installer in the VM tries to fetch an IP address with DHCP:
>>
>> 11:59:35.672258 0.0.0.0.68 > 255.255.255.255.67:  xid:0xbafb375b [|bootp] [tos 0x10]
>>
>> Then on the DHCP server I can see the following in loop:
>>
>> Dec  7 12:00:27 dhcpsrv dhcpd[18917]: DHCPDISCOVER from fe:e1:bb:01:01:01 via XXX.XXX.XXX.1
>> Dec  7 12:00:27 dhcpsrv dhcpd[18917]: DHCPOFFER on XXX.XXX.XXX.101 to fe:e1:bb:01:01:01 via XXX.XXX.XXX.1
>>
>> The IP address ending with .1 is the gateway on my public network and the one ending with .101 is the IP which should be assigned to my OpenBSD VM.
>>
>> It seems like the traffic is not flowing back to the VM itself.
>>
>> I just found a very interesting behaviour by running tcpdump on pretty much all interfaces of my server to analyze the traffic at different levels and BINGO: as soon as I run tcpdump on my trunk0 interface the DHCP request goes through and my VM has network connectivity! But as soon as I stop tcpdump on the trunk interface: no more network connectivity...
>>
>> Now as far as I know running tcpdump enables promiscous mode (PROMISC flag on the interface) and this should the reason why it works.
>>
>> But now what does it mean for my setup, do I need to enable promiscuous mode on my trunk interface manually? and if yes how can I do that?
>>
> The VLAN does require an IP address as far as I am aware.
>
> Mischa
>
>
>
--
Martin Sukany
UNIX Engineer - Solaris / Linux / OpenBSD L3 Specialist
+420 776 275 713
www.sukany.cz

Reply | Threaded
Open this post in threaded view
|

Re: VMs as real hosts on the same network

mabi
In reply to this post by Mischa-2
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, December 7, 2018 12:40 PM, Mischa <[hidden email]> wrote:

> The VLAN does require an IP address as far as I am aware.

Thanks that worked. I now have network connectivity on my public VM VLAN. I saw that adding an IP to my VLAN interface automatically set the trunk interface to PROMISC.

I was trying to avoid "wasting" an IP address as there is no real need for an IP on that VLAN interface on the server itself. But if that's the only way I am fine with that :)


Reply | Threaded
Open this post in threaded view
|

Re: VMs as real hosts on the same network

mabi
In reply to this post by Martin Sukany
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, December 7, 2018 12:57 PM, Martin Sukany <[hidden email]> wrote:

> could you post here your /etc/pf.conf rules?

Sure, it's actually the default OpenBSD 6.4 one as you can see below:

#       $OpenBSD: pf.conf,v 1.55 2017/12/03 20:40:04 sthen Exp $
#
# See pf.conf(5) and /etc/examples/pf.conf

set skip on lo

block return log        # block stateless traffic
pass            # establish keep-state

# By default, do not permit remote connections to X11
block return in on ! lo0 proto tcp to port 6000:6010

# Port build user does not need network
block return out log proto {tcp udp} user _pbuild


See my previous mail answering Mischa, his solution of adding an IP to the VLAN interface solves my issue...

Reply | Threaded
Open this post in threaded view
|

Re: VMs as real hosts on the same network

Stuart Henderson
In reply to this post by mabi
On 2018-12-07, mabi <[hidden email]> wrote:

> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Friday, December 7, 2018 12:40 PM, Mischa <[hidden email]> wrote:
>
>> The VLAN does require an IP address as far as I am aware.
>
> Thanks that worked. I now have network connectivity on my public VM VLAN. I saw that adding an IP to my VLAN interface automatically set the trunk interface to PROMISC.
>
> I was trying to avoid "wasting" an IP address as there is no real need for an IP on that VLAN interface on the server itself. But if that's the only way I am fine with that :)
>
>

That sounds like a bug...
>