Re: Manually-set IPv6 default route disappears after 5 minutes

[Rolf Sommerhalder]

With OpenBSD 5.3-release, I observe the same behavior in a KVM on a
OpenStack cluster.

Although, the v6 default route "disappears" must faster. After being
manually set (again), it does not last longer than a minute or so. My
VM does not run a rtsol, nor is there any rtadvd running in the
connected subnet.

Any ideas? Thanks.

[Rolf Sommerhalder]

FYI, clearing
 sysctl net.inet6.ip6.accept_rtadv=0
which is set to 1 by default, does not help.

[Rolf Sommerhalder]

The v6 default route now persists once I killed dhclient, which was
renewing the v4 address every 54 seconds or so due to a short lease
time!

I am unsure why dhclient touches the v6 routing table. Is this an
unwanted side effect when it eventually refreshes the v4 default
gateway, e.g. a bug?

[Olivier Mehani-5]

On 2013-07-24, Rolf Sommerhalder <[hidden email]> wrote:
> The v6 default route now persists once I killed dhclient, which was
> renewing the v4 address every 54 seconds or so due to a short lease
> time!

OK, I can confirm this reliably:

  $ sudo /sbin/route add -inet6 default -gateway 2001:41D0:1:XX:ff:ff:ff:ff
  $ route -n show -inet6 | grep default
  default                            2001:41d0:1:XX:ff:ff:ff:ff   UGS 0        0     -     8 sis0
  $ sudo pkill dhclient && sudo dhclient sis0
  DHCPREQUEST on sis0 to 255.255.255.255 port 67
  DHCPACK from 91.121.XX.XX (00:30:48:XX:XX:XX)
  Unexpected DHCPACK from 91.121.XX.XX (00:04:4d:XX:XX:XX). State #4 bound to 91.121.XX.XX -- renewal in 300 seconds.
  $ route -n show -inet6 | grep default
  $

> I am unsure why dhclient touches the v6 routing table. Is this an
> unwanted side effect when it eventually refreshes the v4 default
> gateway, e.g. a bug?

Same here. This is odd. Maybe the IPv6 default route gets overwritten at
the same time as the IPv4 one.

I wonder if the very last change to dhclient in CURRENT (1.260 [0])
wouldn't actually fix the problem, at least as a side effect in this
case: it only apply changes when the lease differs from the previously
obtained one.

Prior to this change, bind_lease() unconditionally called
flush_routes(). However, flush_routes() in [1] is documented to be
equivalent to `route -q $rdomain -n flush -inet -iface $interface` (note
the -inet), which might contradict my understanding of the cause.

Anyway, I build dhclient from current today, and tried running it. This
time, I don't seem to lose my route!

  $ sudo /sbin/route add -inet6 default -gateway 2001:41D0:1:XX:ff:ff:ff:ff
  add net default: gateway 2001:41D0:1:XX:ff:ff:ff:ff
  $ route -n show -inet6 | grep default
  default                            2001:41d0:1:XX:ff:ff:ff:ff   UGS 0        0     -     8 sis0
  $ sudo pkill dhclient && sudo ./dhclient sis0
  DHCPREQUEST on sis0 to 255.255.255.255 port 67
  DHCPACK from 91.121.XX.XX (00:30:48:XX:XX:XX)
  bound to 91.121.XX.XX -- renewal in 300 seconds.
  $  route -n show -inet6 | grep default
  default                            2001:41d0:1:XX:ff:ff:ff:ff   UGS 0        0     -     8 sis0
  $

What's better, it doesn't seem to go away anymore:

  $ while : ; do sudo /sbin/route add -inet6 default -gateway 2001:41D0:1:XX:ff:ff:ff:ff -expire 0; time while route show -inet6 | grep -q default; do sleep 1; echo -n .; done; done | tee routelog 2>&1
  route: writing to routing socket: File exists
  add net default: gateway 2001:41D0:1:XX:ff:ff:ff:ff: File exists
  ............... (It keeps going, rather than failing every 5 minutes).

Running dhclient from CURRENT therefore seems to be a good workaround,
but I doubt the lease management is the cause of the problem. Rather, I
still suspect that flush_routes() is closer to the culprit.

I'll file a bug.

[0] http://www.openbsd.org/cgi-bin/cvsweb/src/sbin/dhclient/dhclient.c?rev=1.260
[1] http://www.openbsd.org/cgi-bin/cvsweb/src/sbin/dhclient/kroute.c?rev=1.55

--
Olivier Mehani <[hidden email]>
PGP fingerprint: 4435 CF6A 7C8D DD9B E2DE  F5F9 F012 A6E2 98C6 6655

[Kenneth R Westerback]

On Tue, Aug 06, 2013 at 12:53:38AM +0000, Olivier Mehani wrote:

> On 2013-07-24, Rolf Sommerhalder <[hidden email]> wrote:
> > The v6 default route now persists once I killed dhclient, which was
> > renewing the v4 address every 54 seconds or so due to a short lease
> > time!
>
> OK, I can confirm this reliably:
>
>   $ sudo /sbin/route add -inet6 default -gateway 2001:41D0:1:XX:ff:ff:ff:ff
>   $ route -n show -inet6 | grep default
>   default                            2001:41d0:1:XX:ff:ff:ff:ff   UGS 0        0     -     8 sis0
>   $ sudo pkill dhclient && sudo dhclient sis0
>   DHCPREQUEST on sis0 to 255.255.255.255 port 67
>   DHCPACK from 91.121.XX.XX (00:30:48:XX:XX:XX)
>   Unexpected DHCPACK from 91.121.XX.XX (00:04:4d:XX:XX:XX). State #4 bound to 91.121.XX.XX -- renewal in 300 seconds.
>   $ route -n show -inet6 | grep default
>   $
>
> > I am unsure why dhclient touches the v6 routing table. Is this an
> > unwanted side effect when it eventually refreshes the v4 default
> > gateway, e.g. a bug?

It should not be touching the v6 routing table. If it does, then this is
a bug. priv_flush_routes() assumes that specifying mib[3] = AF_INET will
prevent any IPv6 routes from being reported and thus zapped. If this is
not working ...

>
> Same here. This is odd. Maybe the IPv6 default route gets overwritten at
> the same time as the IPv4 one.

Since add_route() explicitly specifies *.sin_family = AF_INET, the
assumption again is that IPv6 routing table entries should not be
affected.

>
> I wonder if the very last change to dhclient in CURRENT (1.260 [0])
> wouldn't actually fix the problem, at least as a side effect in this
> case: it only apply changes when the lease differs from the previously
> obtained one.
>
> Prior to this change, bind_lease() unconditionally called
> flush_routes(). However, flush_routes() in [1] is documented to be
> equivalent to `route -q $rdomain -n flush -inet -iface $interface` (note
> the -inet), which might contradict my understanding of the cause.
>
> Anyway, I build dhclient from current today, and tried running it. This
> time, I don't seem to lose my route!
>
>   $ sudo /sbin/route add -inet6 default -gateway 2001:41D0:1:XX:ff:ff:ff:ff
>   add net default: gateway 2001:41D0:1:XX:ff:ff:ff:ff
>   $ route -n show -inet6 | grep default
>   default                            2001:41d0:1:XX:ff:ff:ff:ff   UGS 0        0     -     8 sis0
>   $ sudo pkill dhclient && sudo ./dhclient sis0
>   DHCPREQUEST on sis0 to 255.255.255.255 port 67
>   DHCPACK from 91.121.XX.XX (00:30:48:XX:XX:XX)
>   bound to 91.121.XX.XX -- renewal in 300 seconds.
>   $  route -n show -inet6 | grep default
>   default                            2001:41d0:1:XX:ff:ff:ff:ff   UGS 0        0     -     8 sis0
>   $
>
> What's better, it doesn't seem to go away anymore:
>
>   $ while : ; do sudo /sbin/route add -inet6 default -gateway 2001:41D0:1:XX:ff:ff:ff:ff -expire 0; time while route show -inet6 | grep -q default; do sleep 1; echo -n .; done; done | tee routelog 2>&1
>   route: writing to routing socket: File exists
>   add net default: gateway 2001:41D0:1:XX:ff:ff:ff:ff: File exists
>   ............... (It keeps going, rather than failing every 5 minutes).
>
> Running dhclient from CURRENT therefore seems to be a good workaround,
> but I doubt the lease management is the cause of the problem. Rather, I
> still suspect that flush_routes() is closer to the culprit.
>
> I'll file a bug.

Well, if it's working in -current, what's the bug?

.... Ken

>
> [0] http://www.openbsd.org/cgi-bin/cvsweb/src/sbin/dhclient/dhclient.c?rev=1.260
> [1] http://www.openbsd.org/cgi-bin/cvsweb/src/sbin/dhclient/kroute.c?rev=1.55
>
> --
> Olivier Mehani <[hidden email]>
> PGP fingerprint: 4435 CF6A 7C8D DD9B E2DE  F5F9 F012 A6E2 98C6 6655

[Olivier Mehani]

Hey Kenneth,

On Mon, Aug 05, 2013 at 09:07:24PM -0400, Kenneth R Westerback wrote:
> > > I am unsure why dhclient touches the v6 routing table. Is this an
> > > unwanted side effect when it eventually refreshes the v4 default
> > > gateway, e.g. a bug?
> It should not be touching the v6 routing table. If it does, then this is
> a bug. priv_flush_routes() assumes that specifying mib[3] = AF_INET will
> prevent any IPv6 routes from being reported and thus zapped. If this is
> not working ...

AFAIK, priv_flush_routes() is only called on cleanup, not when a new
lease is to be set, nor as a follow-up to flush_routes(). The latter
however is a bit cryptic to me, and I am not sure what it does exactly,
but there surely isn't any AF_INET specification.

> > Same here. This is odd. Maybe the IPv6 default route gets overwritten at
> > the same time as the IPv4 one.
> Since add_route() explicitly specifies *.sin_family = AF_INET, the
> assumption again is that IPv6 routing table entries should not be
> affected.

priv_add_route() does, but add_route() doesn't. I'm not familiar with
how the imsg are handled, so I might be missing something that would
have priv_add_route() be called in response to something add_route()
did. (and s/add/flush/g).

> > Running dhclient from CURRENT therefore seems to be a good workaround,
> > but I doubt the lease management is the cause of the problem. Rather, I
> > still suspect that flush_routes() is closer to the culprit.
> > I'll file a bug.
> Well, if it's working in -current, what's the bug?

I think the fact that it works is a side effect of the change of 1.260,
rather than the potential bug (as you pointed out earlier) being fixed.

--
Olivier Mehani <[hidden email]>
PGP fingerprint: 4435 CF6A 7C8D DD9B E2DE  F5F9 F012 A6E2 98C6 6655

[demime 1.01d removed an attachment of type application/pgp-signature]

[Kenneth R Westerback]

On Tue, Aug 06, 2013 at 11:33:29AM +1000, Olivier Mehani wrote:

> Hey Kenneth,
>
> On Mon, Aug 05, 2013 at 09:07:24PM -0400, Kenneth R Westerback wrote:
> > > > I am unsure why dhclient touches the v6 routing table. Is this an
> > > > unwanted side effect when it eventually refreshes the v4 default
> > > > gateway, e.g. a bug?
> > It should not be touching the v6 routing table. If it does, then this is
> > a bug. priv_flush_routes() assumes that specifying mib[3] = AF_INET will
> > prevent any IPv6 routes from being reported and thus zapped. If this is
> > not working ...
>
> AFAIK, priv_flush_routes() is only called on cleanup, not when a new
> lease is to be set, nor as a follow-up to flush_routes(). The latter
> however is a bit cryptic to me, and I am not sure what it does exactly,
> but there surely isn't any AF_INET specification.

flush_routes() issues IMSG_FLUSH_ROUTES imsg. This is caught in dispatch_imsg(),
where it results in priv_flush_routes() being called. The general mechanism
and naming convention should be consistant. i.e. blah() issues IMSG_BLAH,
which is caught in dispatch_imsg() where the parameters are unpacked and
priv_blah() is called to actually do something. blah() is called by the
non-privileged process and priv_blah() is executed in the privileged
process. The two are connected by an imsg socket.

In some cases the privileged process calls the priv_blah() functions
directly.

>
> > > Same here. This is odd. Maybe the IPv6 default route gets overwritten at
> > > the same time as the IPv4 one.
> > Since add_route() explicitly specifies *.sin_family = AF_INET, the
> > assumption again is that IPv6 routing table entries should not be
> > affected.
>
> priv_add_route() does, but add_route() doesn't. I'm not familiar with
> how the imsg are handled, so I might be missing something that would
> have priv_add_route() be called in response to something add_route()
> did. (and s/add/flush/g).

See above. add_route() issues IMSG_ADD_ROUTE which is caught in
dispatch_imsg(), where priv_add_route() is called.

>
> > > Running dhclient from CURRENT therefore seems to be a good workaround,
> > > but I doubt the lease management is the cause of the problem. Rather, I
> > > still suspect that flush_routes() is closer to the culprit.
> > > I'll file a bug.
> > Well, if it's working in -current, what's the bug?
>
> I think the fact that it works is a side effect of the change of 1.260,
> rather than the potential bug (as you pointed out earlier) being fixed.
>
> --
> Olivier Mehani <[hidden email]>
> PGP fingerprint: 4435 CF6A 7C8D DD9B E2DE  F5F9 F012 A6E2 98C6 6655

You don't mention which version of dhclient you are running.

.... Ken

[Olivier Mehani]

Hey Kenneth,

(Woops I appear to have lost ipv6@ at some point.)

On Mon, Aug 05, 2013 at 11:55:50PM -0400, Kenneth R Westerback wrote:

> > >> > > I'll file a bug.
> > >> > Well, if it's working in -current, what's the bug?
> > >> I think the fact that it works is a side effect of the change of
> > >1.260,
> > >> rather than the potential bug (as you pointed out earlier) being
> > >fixed.
> > >You don't mention which version of dhclient you are running.
> > sendbug took care of it, but it's 5.3 release.
>
> In which case kroute.c r1.44 is not in there. And while that code
> did not survive long, its intent should have been preserved into
> 5.4.

Ah, right. I only based my comments on 1.55, ignoring the fact that it
might not have been the same in 5.3.

5.3 had 1.40 [0], which doesn't have a [priv_]flush_routes(), but only
[priv_]flush_routes_and_arp_cache(). The priv_ variant says nothing
about AF_INET (mib[3] = 0).

This is not the case in 1.55, as it appears to have been fixed with 1.47
[1].

[0]
http://www.openbsd.org/cgi-bin/cvsweb/src/sbin/dhclient/kroute.c?rev=1.40;onl
y_with_tag=OPENBSD_5_3_BASE
[1] http://www.openbsd.org/cgi-bin/cvsweb/src/sbin/dhclient/kroute.c#rev1.47

--
Olivier Mehani <[hidden email]>
PGP fingerprint: 4435 CF6A 7C8D DD9B E2DE  F5F9 F012 A6E2 98C6 6655

[demime 1.01d removed an attachment of type application/pgp-signature]