carp + 5.1/5.2 woes

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

carp + 5.1/5.2 woes

Toni Mueller-10
Hi,

I have a setup with three machines, all i386, and all plugged into
one switch:

 A: 5.1 (IPv4: master)
 B: 5.0 (IPv4: backup)
 C: 5.2 (IPv4: master, IPv6: backup)

Each host has two IPv4 carp interfaces, all on one interface (carp0 and
carp1), and host C has an additional carp2 with only an IPv6 address (no
IPv4).

Now, A + B work nicely with two carp interfaces (IPv4), but A+C do not.

While the carp interface for IPv6 goes into MASTER mode, as expected, if
I change the advskew on A, the IPv4 interfaces don't go into MASTER
mode, but stay in BACKUP mode instead, no matter what:

Eg. from C:

# cat /etc/hostname.carp*                                                                                                                                    
# carp0:
inet 10.0.0.1 255.255.248.0 10.0.7.255 vhid 1 advskew 100 pass pass1 carpdev em0

# carp1:
inet 10.1.1.1 255.255.255.0 10.1.1.255 vhid 2 advskew 100 pass pass2 carpdev em0

# carp2:
inet6 3ffe:3ffe::1 32 vhid 3 advskew 100 pass pass3 carpdev em0


With this setup, carp1 will stay in BACKUP mode when I say "ifconfig
carp1 advskew 120" on A, while on B, it would go into MASTER
immediately.


I also have trouble taking carp2 down and up again, like in "ifconfig
carp2 down; ifconfig carp2 up". The result is that carp2 does no longer
respond to any packets sent to 3ffe:3ffe::1. Sending to the IPv6 address
bound to em0 continues to work like a charm, though. Saying "ifconfig
carp2 destroy; sh /etc/netstart carp2" - which I thought would re-create
the carp2 pseudy-device from scratch, does also not work, but elicits
the following error message from the kernel:

/bsd: in6_ifloop_request: ADD operation failed for 3ffe:3ffe::0001 (errno=17)


There are error messages related to duplicate IPv6 addresses, mentioning
the link-local auto-generated IPv6 address, which is the same for all
carp interfaces, eg:

/bsd: nd6_na_input: duplicate IP6 address fe80:0008::0200:5eff:fe00:0102

Touring the logs, I also find related error messages that I could not
yet make sense of:

/bsd: arpresolve: 10.0.0.1: route without link local address

The mentioned address is being advertised by A as the master, and
intended to be switched around by the CARP mechanism (works with A+B).

On C, I have pf disabled. On all three systems, I have bgpd enabled. On
A, I have pf enabled with these rules:

# pfctl  -s r                                                                                                                                            

block drop in quick on egress proto tcp from <sshguard> to any ...
pass quick on em0 proto carp all keep state (no-sync)
pass quick on em1 proto carp all keep state (no-sync)


When I reboot the machine, the states of the CARP interface(s) are being
set correctly, but I don't know how to change them thereafter, as
described above.

The desired target state is to have A + C as a pair of CARP'ed routers
for both IPv4 and IPv6.


What am I doing wrong?


TIA!



Kind regards,
--Toni++

Reply | Threaded
Open this post in threaded view
|

Re: carp + 5.1/5.2 woes

Mark Felder-4
On Wed, 2 Jan 2013 13:39:25 +0100
Toni Mueller <[hidden email]> wrote:

>  A: 5.1 (IPv4: master)
>  B: 5.0 (IPv4: backup)
>  C: 5.2 (IPv4: master, IPv6: backup)

Didn't the CARP protocol change between these releases? I don't think it's compatible. I'm sure someone else will chime in with the details, but I believe I remember reading this on the list.

Reply | Threaded
Open this post in threaded view
|

Re: carp + 5.1/5.2 woes

Maxim Bourmistrov-5
Yes, this sounds familiar.

On 2 jan 2013, at 14:37, Mark Felder <[hidden email]> wrote:

> Didn't the CARP protocol change between these releases? I don't think it's
compatible. I'm sure someone else will chime in with the details, but I
believe I remember reading this on the list.

Reply | Threaded
Open this post in threaded view
|

Re: carp + 5.1/5.2 woes

Peter Hessler
Doubtful, CARP has not changed protocol for many years.

You might be thinking of pfsync, but that is mostly forwards compatible
for a couple releases now.


On 2013 Jan 02 (Wed) at 15:30:48 +0100 (+0100), mxb wrote:
:Yes, this sounds familiar.
:
:On 2 jan 2013, at 14:37, Mark Felder <[hidden email]> wrote:
:
:> Didn't the CARP protocol change between these releases? I don't think it's
:compatible. I'm sure someone else will chime in with the details, but I
:believe I remember reading this on the list.
:

--
A candidate is a person who gets money from the rich and votes from the
poor to protect them from each other.

Reply | Threaded
Open this post in threaded view
|

Re: carp + 5.1/5.2 woes

Patrick Lamaiziere
In reply to this post by Toni Mueller-10
Le Wed, 2 Jan 2013 13:39:25 +0100,
Toni Mueller <[hidden email]> a écrit :

Hello,

> With this setup, carp1 will stay in BACKUP mode when I say "ifconfig
> carp1 advskew 120" on A, while on B, it would go into MASTER
> immediately.

Hmm, did you check the value of the carp demote counter?

# ifconfig -g carp

(just a guess, regards)

Reply | Threaded
Open this post in threaded view
|

Re: carp + 5.1/5.2 woes

Toni Mueller-10
Hi,

On Wed, Jan 02, 2013 at 04:53:02PM +0100, Patrick Lamaiziere wrote:
> Le Wed, 2 Jan 2013 13:39:25 +0100, Toni Mueller <[hidden email]> a écrit :
> > With this setup, carp1 will stay in BACKUP mode when I say "ifconfig
> > carp1 advskew 120" on A, while on B, it would go into MASTER
> > immediately.
>
> Hmm, did you check the value of the carp demote counter?
>
> # ifconfig -g carp

I just checked. The result is the same on all three machines:

# ifconfig -g carp
carp: carp demote count 0


Kind regards,
--Toni++

Reply | Threaded
Open this post in threaded view
|

Re: carp + 5.1/5.2 woes

Stuart Henderson
In reply to this post by Toni Mueller-10
On 2013-01-02, Toni Mueller <[hidden email]> wrote:
> Hi,
>
> I have a setup with three machines, all i386, and all plugged into
> one switch:
>
>  A: 5.1 (IPv4: master)
>  B: 5.0 (IPv4: backup)
>  C: 5.2 (IPv4: master, IPv6: backup)

Is this 5.0 release or is it "something close to 5.0"?

----------------------------
revision 1.181
date: 2011/03/08 22:53:28;  author: mpf;  state: Exp;  lines: +6 -8
Fix a subtle carp reconfiguration problem.
Updating the HMAC from the carp_ioctl call does not see the newly
set IP address in the if_addrlist.  The only chance for carp to see
the new address is via the address-hook callback.  This change moves
the detection of address changes entirely into carp_addr_updated.
Furthermore, only call carp_hmac_prepare for the SIOCSVH case.  This
second bug was the reason why the first one went unnoticed for such
a long time.
Problem found and debugging help by camield@.
OK camield@
revision 1.183
date: 2011/04/29 12:36:31;  author: mpf;  state: Exp;  lines: +10 -11
The previous reconfiguration change broke IPv6 only setups.
The address hook was only registered for v4 addresses.
We now call hook_establish at interface creation time.
The hook is now disestablished upon interface destroy,
which plugs a tiny memleak.
While there remove redundancy in carp_set_addr6 and sync
it with carp_set_addr.
Bug noticed by todd@. OK sthen, mikeb (on an earlier version)
OK and some hints by camield@
----------------------------

Reply | Threaded
Open this post in threaded view
|

Re: carp + 5.1/5.2 woes

Toni Mueller-10
Hi,

On Wed, Jan 02, 2013 at 05:47:23PM +0000, Stuart Henderson wrote:
> On 2013-01-02, Toni Mueller <[hidden email]> wrote:
> >  A: 5.1 (IPv4: master)
> >  B: 5.0 (IPv4: backup)
> >  C: 5.2 (IPv4: master, IPv6: backup)
>
> Is this 5.0 release or is it "something close to 5.0"?

the (working!) 5.0 machine runs

# uname -m  -r -s -v  
OpenBSD 5.0 GENERIC#43 i386


The other machines were installed/upgraded from the official CDs.



Kind regards,
--Toni++

Reply | Threaded
Open this post in threaded view
|

Re: carp + 5.1/5.2 woes [PARTIALLY SOLVED]

Toni Mueller-10
In reply to this post by Toni Mueller-10
Hi,


I have just discovered that I made a configuration error that had
resulted in the undesired, but correct, carp behaviour for IPv4. Ie,
OpenBSD operates as desired for this case.

That leaves these questions open:

On Wed, Jan 02, 2013 at 01:39:25PM +0100, Toni Mueller wrote:

> I also have trouble taking carp2 down and up again, like in "ifconfig
> carp2 down; ifconfig carp2 up". The result is that carp2 does no longer
> respond to any packets sent to 3ffe:3ffe::1. Sending to the IPv6 address
> bound to em0 continues to work like a charm, though. Saying "ifconfig
> carp2 destroy; sh /etc/netstart carp2" - which I thought would re-create
> the carp2 pseudy-device from scratch, does also not work, but elicits
> the following error message from the kernel:
>
> /bsd: in6_ifloop_request: ADD operation failed for 3ffe:3ffe::0001 (errno=17)
>
>
> There are error messages related to duplicate IPv6 addresses, mentioning
> the link-local auto-generated IPv6 address, which is the same for all
> carp interfaces, eg:
>
> /bsd: nd6_na_input: duplicate IP6 address fe80:0008::0200:5eff:fe00:0102
>
> Touring the logs, I also find related error messages that I could not
> yet make sense of:
>
> /bsd: arpresolve: 10.0.0.1: route without link local address

I would still be glad to find that I simply configured junk, instead of
running into real bugs...


Kind regards,
--Toni++

Reply | Threaded
Open this post in threaded view
|

Re: carp + 5.1/5.2 woes [PARTIALLY SOLVED]

Stuart Henderson
On 2013-01-02, Toni Mueller <[hidden email]> wrote:
> Hi,
>
>
> I have just discovered that I made a configuration error that had
> resulted in the undesired, but correct, carp behaviour for IPv4. Ie,
> OpenBSD operates as desired for this case.

Ah good :)

> That leaves these questions open:
>
> On Wed, Jan 02, 2013 at 01:39:25PM +0100, Toni Mueller wrote:
>> I also have trouble taking carp2 down and up again, like in "ifconfig
>> carp2 down; ifconfig carp2 up". The result is that carp2 does no longer
>> respond to any packets sent to 3ffe:3ffe::1. Sending to the IPv6 address
>> bound to em0 continues to work like a charm, though. Saying "ifconfig
>> carp2 destroy; sh /etc/netstart carp2" - which I thought would re-create
>> the carp2 pseudy-device from scratch, does also not work, but elicits
>> the following error message from the kernel:
>>
>> /bsd: in6_ifloop_request: ADD operation failed for 3ffe:3ffe::0001 (errno=17)

17 is EEXIST - see errno(2) for a list of these - there's probably
a loopback route hanging around after destroying the interface,
check in netstat -rnfinet6, you could try deleting it..


>> There are error messages related to duplicate IPv6 addresses, mentioning
>> the link-local auto-generated IPv6 address, which is the same for all
>> carp interfaces, eg:
>>
>> /bsd: nd6_na_input: duplicate IP6 address fe80:0008::0200:5eff:fe00:0102

Yes, that happens ;)


>> Touring the logs, I also find related error messages that I could not
>> yet make sense of:
>>
>> /bsd: arpresolve: 10.0.0.1: route without link local address

I've seen this before, I think it was on a router with a (non-/32)
address on both the parent interface and the carp interface, though
I have a few routers doing exactly that which don't see it..
(Normally it's recommended to use /32 on the carp interface, but
that's not going to work if you are announcing it into ospf).

Someone tracked down another situation where this can happen,
http://marc.info/?l=openbsd-misc&m=121455393316796&w=2

Reply | Threaded
Open this post in threaded view
|

Re: carp + 5.1/5.2 woes [PARTIALLY SOLVED]

Toni Mueller-10
Hi,

thanks for the insight.

On Thu, Jan 03, 2013 at 01:37:38AM +0000, Stuart Henderson wrote:
> On 2013-01-02, Toni Mueller <[hidden email]> wrote:
> >> /bsd: in6_ifloop_request: ADD operation failed for 3ffe:3ffe::0001 (errno=17)
>
> 17 is EEXIST - see errno(2) for a list of these - there's probably
> a loopback route hanging around after destroying the interface,
> check in netstat -rnfinet6, you could try deleting it..

this happens exactly the moment when the carp interface that has an
IPv4 address assigned to it, goes into BACKUP state.

> >> /bsd: nd6_na_input: duplicate IP6 address fe80:0008::0200:5eff:fe00:0102
> Yes, that happens ;)

I can I ignore these, and/or can I safely remove the link-local
addresses that seem to be lifted from the physical CARP device?

> >> /bsd: arpresolve: 10.0.0.1: route without link local address
>
> I've seen this before, I think it was on a router with a (non-/32)
> address on both the parent interface and the carp interface, though
> I have a few routers doing exactly that which don't see it..
> (Normally it's recommended to use /32 on the carp interface, but
> that's not going to work if you are announcing it into ospf).

Ok. I do not use OSPF (only BGP), so I set all interfaces to

IP address/ netmask of the connected network (eg. "32" for
the IPv6 network).

> Someone tracked down another situation where this can happen,
> http://marc.info/?l=openbsd-misc&m=121455393316796&w=2

I therefore would expect the problem to show up for the IPv6-only
CARP interface (ie, carp2) after that went down and refused to
come up again, until the next reboot. But the error message specified
the IPv4 address for a carp interface that is actually there, up, and
*should* be working.

Is this a known problem, or is it just me, that CARP interfaces
come up only once?


Kind regards,
--Toni++