Weird routing problem on simple CARP setup

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Weird routing problem on simple CARP setup

BARDOU Pierre
Hello,

I have a strange problem with OpenBSD 6.2, which looks like a bug.
Steps to reproduce :

* sh /etc/netstart -> everything works. Routing table :
root@fw-t-wan-chut01:~ # netstat -rnf inet                                                                                                                                                              
Routing tables

Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
default            10.194.119.254     UGS        0       16     -     8 bge0
224/4              127.0.0.1          URS        0      798 32768     8 lo0  
10.194.116/22      10.194.116.29      UCn        1        1     -     4 bge0
10.194.116/22      10.194.116.28      UCn        0        0     -    19 carp0
10.194.116.28      00:00:5e:00:01:0f  UHLl       0        3     -     1 carp0
10.194.116.29      40:a8:f0:36:22:0c  UHLl       0       28     -     1 bge0
10.194.119.254     00:1b:2a:e9:c4:00  UHLch      2        5     -     3 bge0
10.194.119.255     10.194.116.29      UHb        0        0     -     1 bge0
10.194.119.255     10.194.116.28      UHb        0        0     -     1 carp0
127/8              127.0.0.1          UGRS       0        0 32768     8 lo0  
127.0.0.1          127.0.0.1          UHhl       1     1122 32768     1 lo0  
192.168.190/24     192.168.190.1      Cn         0        0     -     4 bge1
192.168.190.1      40:a8:f0:36:22:0d  UHLl       0        0     -     1 bge1
192.168.190.255    192.168.190.1      Hb         0        0     -     1 bge1
root@fw-t-wan-chut01:~ # ifconfig carp0                                                                                                                                                                
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5e:00:01:0f
        description: TL-INT-ADM-WAN
        index 10 priority 15 llprio 3
        carp: MASTER carpdev bge0 vhid 15 advbase 1 advskew 10
        groups: carp
        status: master
        inet 10.194.116.28 netmask 0xfffffc00 broadcast 10.194.119.255

* then sh /etc/netstart carp0 -> routed traffic stops working (ping 10.194.125.120 says "sendmsg: Invalid argument").
Same result if I do ifconfig carp0 10.194.116.28/22.
Routing table and ifconfig look the same :
root@fw-t-wan-chut01:~ # netstat -rnf inet                                                                                                                                                          
Routing tables

Internet:
Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
default            10.194.119.254     UGS        5       58     -     8 bge0
224/4              127.0.0.1          URS        0     3918 32768     8 lo0  
10.194.116/22      10.194.116.29      UCn        1    59014     -     4 bge0
10.194.116/22      10.194.116.28      UCn        0        0     -    19 carp0
10.194.116.28      00:00:5e:00:01:0f  UHLl       0        7     -     1 carp0
10.194.116.29      40:a8:f0:36:22:0c  UHLl       0       40     -     1 bge0
10.194.119.254     00:1b:2a:e9:c4:00  UHLc       0    29528     -     3 bge0
10.194.119.255     10.194.116.29      UHb        0        0     -     1 bge0
10.194.119.255     10.194.116.28      UHb        0        0     -     1 carp0
127/8              127.0.0.1          UGRS       0        0 32768     8 lo0  
127.0.0.1          127.0.0.1          UHhl       1     5498 32768     1 lo0  
192.168.190/24     192.168.190.1      Cn         0        0     -     4 bge1
192.168.190.1      40:a8:f0:36:22:0d  UHLl       0        0     -     1 bge1
192.168.190.255    192.168.190.1      Hb         0        0     -     1 bge1
root@fw-t-wan-chut01:~ # ifconfig carp0                                                                                                                                                                
carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5e:00:01:0f
        description: TL-INT-ADM-WAN
        index 10 priority 15 llprio 3
        carp: MASTER carpdev bge0 vhid 15 advbase 1 advskew 10
        groups: carp
        status: master
        inet 10.194.116.28 netmask 0xfffffc00 broadcast 10.194.119.255

* then again sh /etc/netstart -> everything is working again.
Deleting and readding the default route also does the trick.

If I test something like :
root@fw-t-wan-chut01:~ # sh /etc/netstart
root@fw-t-wan-chut01:~ # ifconfig bge0 10.194.116.29/22
The default route disappears. This is a bit weird, but at least the routing table is consistent with what happens.

I figured a workaround by not using the mygate file, and adding a line in the hostname.bge0 and hostname.carp0 :
!route add default 10.194.119.254 1>/dev/null || route change default 10.194.119.254 1>/dev/null


Additional informations :
Network configuration :
root@fw-t-wan-chut01:~ # cat /etc/hostname.bge0                                                                                                                                                          
10.194.116.29/22 description "Admin"
up -inet6
root@fw-t-wan-chut01:~ # cat /etc/hostname.carp0
10.194.116.28/22 vhid 15 advskew 10 carpdev bge0 pass xxxx description "TL-INT-ADM-WAN"
up -inet6
root@fw-t-wan-chut01:~ # cat /etc/mygate
10.194.119.254

PF is disabled.


--
Cordialement,
Pierre Bardou

Reply | Threaded
Open this post in threaded view
|

Re: Weird routing problem on simple CARP setup

Stefan Sperling-5
On Wed, Jun 27, 2018 at 09:30:16AM +0000, BARDOU Pierre wrote:

> Hello,
>
> I have a strange problem with OpenBSD 6.2, which looks like a bug.
> Steps to reproduce :
>
> * sh /etc/netstart -> everything works. Routing table :
> root@fw-t-wan-chut01:~ # netstat -rnf inet                                                                                                                                                              
> Routing tables
>
> Internet:
> Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
> default            10.194.119.254     UGS        0       16     -     8 bge0
> 224/4              127.0.0.1          URS        0      798 32768     8 lo0  
> 10.194.116/22      10.194.116.29      UCn        1        1     -     4 bge0
> 10.194.116/22      10.194.116.28      UCn        0        0     -    19 carp0
> 10.194.116.28      00:00:5e:00:01:0f  UHLl       0        3     -     1 carp0
> 10.194.116.29      40:a8:f0:36:22:0c  UHLl       0       28     -     1 bge0
> 10.194.119.254     00:1b:2a:e9:c4:00  UHLch      2        5     -     3 bge0
> 10.194.119.255     10.194.116.29      UHb        0        0     -     1 bge0
> 10.194.119.255     10.194.116.28      UHb        0        0     -     1 carp0
> 127/8              127.0.0.1          UGRS       0        0 32768     8 lo0  
> 127.0.0.1          127.0.0.1          UHhl       1     1122 32768     1 lo0  
> 192.168.190/24     192.168.190.1      Cn         0        0     -     4 bge1
> 192.168.190.1      40:a8:f0:36:22:0d  UHLl       0        0     -     1 bge1
> 192.168.190.255    192.168.190.1      Hb         0        0     -     1 bge1
> root@fw-t-wan-chut01:~ # ifconfig carp0                                                                                                                                                                
> carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>         lladdr 00:00:5e:00:01:0f
>         description: TL-INT-ADM-WAN
>         index 10 priority 15 llprio 3
>         carp: MASTER carpdev bge0 vhid 15 advbase 1 advskew 10
>         groups: carp
>         status: master
>         inet 10.194.116.28 netmask 0xfffffc00 broadcast 10.194.119.255
>
> * then sh /etc/netstart carp0 -> routed traffic stops working (ping 10.194.125.120 says "sendmsg: Invalid argument").
> Same result if I do ifconfig carp0 10.194.116.28/22.

Have you tried using a /32 mask on carp0 instead of /22?
That might work around the problem.

I believe this problem is fixed in 6.3. Can you confirm?

Reply | Threaded
Open this post in threaded view
|

Re: Weird routing problem on simple CARP setup

BARDOU Pierre
Hellom

Sorry for the long delay, I've been very busy recently.

Putting the carp in /32 works.
What's the best practice when you have a physical IP + CARP in the same subnet ?
The FAQ here https://www.openbsd.org/faq/pf/carp.html#failover uses the same netmask for the CARP and the physical interface.

I upgraded to 6.3 and it also works.

Thank you for your help

--
Cordialement,
Pierre BARDOU

-----Message d'origine-----
De : Stefan Sperling <[hidden email]>
Envoyé : mardi 3 juillet 2018 13:33
À : BARDOU Pierre <[hidden email]>
Cc : [hidden email]
Objet : Re: Weird routing problem on simple CARP setup

On Wed, Jun 27, 2018 at 09:30:16AM +0000, BARDOU Pierre wrote:

> Hello,
>
> I have a strange problem with OpenBSD 6.2, which looks like a bug.
> Steps to reproduce :
>
> * sh /etc/netstart -> everything works. Routing table :
> root@fw-t-wan-chut01:~ # netstat -rnf inet                                                                                                                                                              
> Routing tables
>
> Internet:
> Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
> default            10.194.119.254     UGS        0       16     -     8 bge0
> 224/4              127.0.0.1          URS        0      798 32768     8 lo0  
> 10.194.116/22      10.194.116.29      UCn        1        1     -     4 bge0
> 10.194.116/22      10.194.116.28      UCn        0        0     -    19 carp0
> 10.194.116.28      00:00:5e:00:01:0f  UHLl       0        3     -     1 carp0
> 10.194.116.29      40:a8:f0:36:22:0c  UHLl       0       28     -     1 bge0
> 10.194.119.254     00:1b:2a:e9:c4:00  UHLch      2        5     -     3 bge0
> 10.194.119.255     10.194.116.29      UHb        0        0     -     1 bge0
> 10.194.119.255     10.194.116.28      UHb        0        0     -     1 carp0
> 127/8              127.0.0.1          UGRS       0        0 32768     8 lo0  
> 127.0.0.1          127.0.0.1          UHhl       1     1122 32768     1 lo0  
> 192.168.190/24     192.168.190.1      Cn         0        0     -     4 bge1
> 192.168.190.1      40:a8:f0:36:22:0d  UHLl       0        0     -     1 bge1
> 192.168.190.255    192.168.190.1      Hb         0        0     -     1 bge1
> root@fw-t-wan-chut01:~ # ifconfig carp0                                                                                                                                                                
> carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>         lladdr 00:00:5e:00:01:0f
>         description: TL-INT-ADM-WAN
>         index 10 priority 15 llprio 3
>         carp: MASTER carpdev bge0 vhid 15 advbase 1 advskew 10
>         groups: carp
>         status: master
>         inet 10.194.116.28 netmask 0xfffffc00 broadcast 10.194.119.255
>
> * then sh /etc/netstart carp0 -> routed traffic stops working (ping 10.194.125.120 says "sendmsg: Invalid argument").
> Same result if I do ifconfig carp0 10.194.116.28/22.

Have you tried using a /32 mask on carp0 instead of /22?
That might work around the problem.

I believe this problem is fixed in 6.3. Can you confirm?

Reply | Threaded
Open this post in threaded view
|

Re: Weird routing problem on simple CARP setup

Tom Smyth
Hi Pierre,

with VRRP on other vendors the IP on the Virtual interface
is recommended to be a /32,


afaik
it prevents ambiguity when it comes to your connected routes
do you route a packet out the carp interface which as an ip on the configured
/24 network or do you route the packet out the physcial interface which also
has a /24 network configured


I note the examples and faq page in openbsd  show ips configured
with a /24 configured
https://man.openbsd.org/carp

and a /24 seems to be the default ip if a subnet mask is not specified


But I would love to hear / learn more experienced OpenBSD Admins
Devs take on it

Thanks
Tom Smyth


On 11 July 2018 at 16:47, BARDOU Pierre <[hidden email]> wrote:

> Hellom
>
> Sorry for the long delay, I've been very busy recently.
>
> Putting the carp in /32 works.
> What's the best practice when you have a physical IP + CARP in the same subnet ?
> The FAQ here https://www.openbsd.org/faq/pf/carp.html#failover uses the same netmask for the CARP and the physical interface.
>
> I upgraded to 6.3 and it also works.
>
> Thank you for your help
>
> --
> Cordialement,
> Pierre BARDOU
>
> -----Message d'origine-----
> De : Stefan Sperling <[hidden email]>
> Envoyé : mardi 3 juillet 2018 13:33
> À : BARDOU Pierre <[hidden email]>
> Cc : [hidden email]
> Objet : Re: Weird routing problem on simple CARP setup
>
> On Wed, Jun 27, 2018 at 09:30:16AM +0000, BARDOU Pierre wrote:
>> Hello,
>>
>> I have a strange problem with OpenBSD 6.2, which looks like a bug.
>> Steps to reproduce :
>>
>> * sh /etc/netstart -> everything works. Routing table :
>> root@fw-t-wan-chut01:~ # netstat -rnf inet
>> Routing tables
>>
>> Internet:
>> Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
>> default            10.194.119.254     UGS        0       16     -     8 bge0
>> 224/4              127.0.0.1          URS        0      798 32768     8 lo0
>> 10.194.116/22      10.194.116.29      UCn        1        1     -     4 bge0
>> 10.194.116/22      10.194.116.28      UCn        0        0     -    19 carp0
>> 10.194.116.28      00:00:5e:00:01:0f  UHLl       0        3     -     1 carp0
>> 10.194.116.29      40:a8:f0:36:22:0c  UHLl       0       28     -     1 bge0
>> 10.194.119.254     00:1b:2a:e9:c4:00  UHLch      2        5     -     3 bge0
>> 10.194.119.255     10.194.116.29      UHb        0        0     -     1 bge0
>> 10.194.119.255     10.194.116.28      UHb        0        0     -     1 carp0
>> 127/8              127.0.0.1          UGRS       0        0 32768     8 lo0
>> 127.0.0.1          127.0.0.1          UHhl       1     1122 32768     1 lo0
>> 192.168.190/24     192.168.190.1      Cn         0        0     -     4 bge1
>> 192.168.190.1      40:a8:f0:36:22:0d  UHLl       0        0     -     1 bge1
>> 192.168.190.255    192.168.190.1      Hb         0        0     -     1 bge1
>> root@fw-t-wan-chut01:~ # ifconfig carp0
>> carp0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>>         lladdr 00:00:5e:00:01:0f
>>         description: TL-INT-ADM-WAN
>>         index 10 priority 15 llprio 3
>>         carp: MASTER carpdev bge0 vhid 15 advbase 1 advskew 10
>>         groups: carp
>>         status: master
>>         inet 10.194.116.28 netmask 0xfffffc00 broadcast 10.194.119.255
>>
>> * then sh /etc/netstart carp0 -> routed traffic stops working (ping 10.194.125.120 says "sendmsg: Invalid argument").
>> Same result if I do ifconfig carp0 10.194.116.28/22.
>
> Have you tried using a /32 mask on carp0 instead of /22?
> That might work around the problem.
>
> I believe this problem is fixed in 6.3. Can you confirm?
>



--
Kindest regards,
Tom Smyth

Mobile: +353 87 6193172
The information contained in this E-mail is intended only for the
confidential use of the named recipient. If the reader of this message
is not the intended recipient or the person responsible for
delivering it to the recipient, you are hereby notified that you have
received this communication in error and that any review,
dissemination or copying of this communication is strictly prohibited.
If you have received this in error, please notify the sender
immediately by telephone at the number above and erase the message
You are requested to carry out your own virus check before
opening any attachment.

Reply | Threaded
Open this post in threaded view
|

Re: Weird routing problem on simple CARP setup

Stuart Henderson
On 2018-07-11, Tom Smyth <[hidden email]> wrote:

> Hi Pierre,
>
> with VRRP on other vendors the IP on the Virtual interface
> is recommended to be a /32,
>
>
> afaik
> it prevents ambiguity when it comes to your connected routes
> do you route a packet out the carp interface which as an ip on the configured
> /24 network or do you route the packet out the physcial interface which also
> has a /24 network configured
>
>
> I note the examples and faq page in openbsd  show ips configured
> with a /24 configured
> https://man.openbsd.org/carp
>
> and a /24 seems to be the default ip if a subnet mask is not specified
>
>
> But I would love to hear / learn more experienced OpenBSD Admins
> Devs take on it

My GBP0.02 (which isn't worth much these days ;)

- generally /32 for addresses on carp

- if there are no IPs from the subnet used in carp on another "real" interface
(i.e. the address is only on the carp interface) then use the full /XX for one
IP in that subnet on carp

- if you're redistributing a subnet on a carp interface into OSPF you need
the full /XX on the carp interface so it can announce the network rather
than a single host (you want to announce it from carp rather than the
"real" interface so the priority changes depending on the carp state)


Reply | Threaded
Open this post in threaded view
|

Re: Weird routing problem on simple CARP setup

BARDOU Pierre
That makes sense.
Thanks for your advices.

--
Cordialement,
Pierre BARDOU


-----Message d'origine-----
De : Stuart Henderson <[hidden email]>
Envoyé : mercredi 11 juillet 2018 23:24
À : [hidden email]
Objet : Re: Weird routing problem on simple CARP setup

On 2018-07-11, Tom Smyth <[hidden email]> wrote:

> Hi Pierre,
>
> with VRRP on other vendors the IP on the Virtual interface is
> recommended to be a /32,
>
>
> afaik
> it prevents ambiguity when it comes to your connected routes do you
> route a packet out the carp interface which as an ip on the configured
> /24 network or do you route the packet out the physcial interface
> which also has a /24 network configured
>
>
> I note the examples and faq page in openbsd  show ips configured with
> a /24 configured https://man.openbsd.org/carp
>
> and a /24 seems to be the default ip if a subnet mask is not specified
>
>
> But I would love to hear / learn more experienced OpenBSD Admins Devs
> take on it

My GBP0.02 (which isn't worth much these days ;)

- generally /32 for addresses on carp

- if there are no IPs from the subnet used in carp on another "real" interface (i.e. the address is only on the carp interface) then use the full /XX for one IP in that subnet on carp

- if you're redistributing a subnet on a carp interface into OSPF you need the full /XX on the carp interface so it can announce the network rather than a single host (you want to announce it from carp rather than the "real" interface so the priority changes depending on the carp state)