OpenBSD 6.0 IPv6 issue / kernel crash

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

OpenBSD 6.0 IPv6 issue / kernel crash

J.Sauer
Hello OpenBSD Community,

today i played a little bit with cisco routers and IPv6.
It seems like a plain OpenBSD 6.0 install crashes while
i reduced the IPv6 (cisco) RA parameters to a really short ammount.
This behaviour was for me reproducible three times.

All Screenshots and pcap files captured hopefully helps you
to find/solve the problem.

The crash occurs while i was pinging the R1 router continuosly
on it's IPv6 address 2001:db8:1000::1.
While pinging, i changed on R1 the RA plivetime and vlifetime from
60/30 to 5/3. After a short time OpenBSD crashed!

I hope i have put enough information inside the .zip file for you to
figure out the problem.

Regards

J.Sauer

 -------
Jens Sauer
System Engineer

LinkedIn: http://de.linkedin.com/pub/jens-sauer/a8/a9b/b04/
GitHub:   https://github.com/NanoOps

OpenBSD6-0_Kernel_CRASH_IPv6.zip (318K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD 6.0 IPv6 issue / kernel crash

Martin Pieuchot
On 05/09/16(Mon) 16:39, Jens Sauer wrote:

> Hello OpenBSD Community,
>
> today i played a little bit with cisco routers and IPv6.
> It seems like a plain OpenBSD 6.0 install crashes while
> i reduced the IPv6 (cisco) RA parameters to a really short ammount.
> This behaviour was for me reproducible three times.
>
> All Screenshots and pcap files captured hopefully helps you
> to find/solve the problem.
>
> The crash occurs while i was pinging the R1 router continuosly
> on it's IPv6 address 2001:db8:1000::1.
> While pinging, i changed on R1 the RA plivetime and vlifetime from
> 60/30 to 5/3. After a short time OpenBSD crashed!
>
> I hope i have put enough information inside the .zip file for you to
> figure out the problem.

Could you try a -current snapshot?  A lot of cleanup happened in this
area and I'd like to know if you can still reproduce this crash.

Thanks,
Martin

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD 6.0 IPv6 issue / kernel crash

J.Sauer
Martin Pieuchot wrote
On 05/09/16(Mon) 16:39, Jens Sauer wrote:
> Hello OpenBSD Community,
>
> today i played a little bit with cisco routers and IPv6.
> It seems like a plain OpenBSD 6.0 install crashes while
> i reduced the IPv6 (cisco) RA parameters to a really short ammount.
> This behaviour was for me reproducible three times.
>
> All Screenshots and pcap files captured hopefully helps you
> to find/solve the problem.
>
> The crash occurs while i was pinging the R1 router continuosly
> on it's IPv6 address 2001:db8:1000::1.
> While pinging, i changed on R1 the RA plivetime and vlifetime from
> 60/30 to 5/3. After a short time OpenBSD crashed!
>
> I hope i have put enough information inside the .zip file for you to
> figure out the problem.

Could you try a -current snapshot?  A lot of cleanup happened in this
area and I'd like to know if you can still reproduce this crash.

Thanks,
Martin
Hello Martin,
hello OpenBSD Community,

I have tested it again with OpenBSD 6.0 current #2250 and could reproducible
get the same result (Kernel Crash). Seems like the ping isn't involved
because the kernel crash happens even without it after changing the
RA p-/vlifetime to 5/3. The crash occurs just a couple of seconds after it.

This is a serious bug! An advertising IPv6 Router could crash a connected
OpenBSD6 Host (rtsol) just by changing the p-/vltime to short ammounts!

Snapshot: cd60.iso 2016-09-05
http://ftp.halifax.rwth-aachen.de/openbsd/snapshots/amd64/

Regards

J.Sauer

PS: Attachment: new screenshots( kernel crash/trace/ps), pcap and some more infos.
OpenBSD6_current_snapshot.zip
Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD 6.0 IPv6 issue / kernel crash

J.Sauer
In reply to this post by Martin Pieuchot
Hello Martin,
hello OpenBSD Community,

I have tested it again with OpenBSD 6.0 current #2250 and could reproducible
get the same result (Kernel Crash). Seems like the ping isn't involved
because the kernel crash happens even without it after changing the
RA p-/vlifetime to 5/3. The crash occurs just a couple of seconds after it.

This is a serious bug! An advertising IPv6 Router could crash a connected
OpenBSD6 Host (rtsol) just by changing the p-/vltime to short ammounts!

Snapshot: cd60.iso 2016-09-05
http://ftp.halifax.rwth-aachen.de/openbsd/snapshots/amd64/

Regards

J.Sauer

PS: Attachment: new screenshots( kernel crash/trace/ps), pcap and some more infos.


-------Jens Sauer
System Engineer

LinkedIn: http://de.linkedin.com/pub/jens-sauer/a8/a9b/b04/
GitHub:   https://github.com/NanoOps


----- Urspr√ľngliche Message -----
Von: Martin Pieuchot <[hidden email]>
An: Jens Sauer <[hidden email]>
CC: "[hidden email]" <[hidden email]>
Gesendet: 11:24 Dienstag, 6.September 2016
Betreff: Re: OpenBSD 6.0 IPv6 issue / kernel crash

On 05/09/16(Mon) 16:39, Jens Sauer wrote:

> Hello OpenBSD Community,
>
> today i played a little bit with cisco routers and IPv6.
> It seems like a plain OpenBSD 6.0 install crashes while
> i reduced the IPv6 (cisco) RA parameters to a really short ammount.
> This behaviour was for me reproducible three times.
>
> All Screenshots and pcap files captured hopefully helps you
> to find/solve the problem.
>
> The crash occurs while i was pinging the R1 router continuosly
> on it's IPv6 address 2001:db8:1000::1.
> While pinging, i changed on R1 the RA plivetime and vlifetime from
> 60/30 to 5/3. After a short time OpenBSD crashed!
>
> I hope i have put enough information inside the .zip file for you to
> figure out the problem.
Could you try a -current snapshot?  A lot of cleanup happened in this
area and I'd like to know if you can still reproduce this crash.

Thanks,
Martin

OpenBSD6_current_snapshot.zip (191K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD 6.0 IPv6 issue / kernel crash

Martin Pieuchot
On 06/09/16(Tue) 13:54, Jens Sauer wrote:
> Hello Martin,
> hello OpenBSD Community,
>
> I have tested it again with OpenBSD 6.0 current #2250 and could reproducible
> get the same result (Kernel Crash). Seems like the ping isn't involved
> because the kernel crash happens even without it after changing the
> RA p-/vlifetime to 5/3. The crash occurs just a couple of seconds after it.

Thanks for testing.  Here's a diff that should prevent this panic, could
you confirm it works for you?

Index: netinet6/nd6_rtr.c
===================================================================
RCS file: /cvs/src/sys/netinet6/nd6_rtr.c,v
retrieving revision 1.144
diff -u -p -r1.144 nd6_rtr.c
--- netinet6/nd6_rtr.c 2 Sep 2016 11:51:07 -0000 1.144
+++ netinet6/nd6_rtr.c 7 Sep 2016 11:58:14 -0000
@@ -1972,9 +1972,11 @@ in6_ifadd(struct nd_prefix *pr, int priv
 
  ifra.ifra_flags |= IN6_IFF_AUTOCONF|IN6_IFF_TENTATIVE;
 
- /* allocate ifaddr structure, link into chain, etc. */
+ /* If this address already exists, update it. */
+ ia6 = in6ifa_ifpwithaddr(ifp, &ifra.ifra_addr.sin6_addr);
+
  s = splsoftnet();
- error = in6_update_ifa(ifp, &ifra, NULL);
+ error = in6_update_ifa(ifp, &ifra, ia6);
  splx(s);
 
  if (error != 0) {

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD 6.0 IPv6 issue / kernel crash

J.Sauer
Hello Martin,
hello OpenBSD Community,

I have tested the patch and it works in the way of no more kernel crashs.
But now we have huge ammounts of packet loss (>80%) while changing the
RA p/vltimer from 60/30 to 5/3 and after a while back to original values
the loss stay high. This seems to be a cisco problem or environment based.
I've put some screenshots and the pcap as attachment.

Thx for the fast patch - awesome!

Regards

J.Sauer
 -------
Jens Sauer
System Engineer

LinkedIn: http://de.linkedin.com/pub/jens-sauer/a8/a9b/b04/
GitHub:   https://github.com/NanoOps


----- Urspr√ľngliche Message -----
Von: Martin Pieuchot <[hidden email]>
An: Jens Sauer <[hidden email]>
CC: "[hidden email]" <[hidden email]>
Gesendet: 13:58 Mittwoch, 7.September 2016
Betreff: Re: OpenBSD 6.0 IPv6 issue / kernel crash

On 06/09/16(Tue) 13:54, Jens Sauer wrote:

> Hello Martin,
> hello OpenBSD Community,
>
> I have tested it again with OpenBSD 6.0 current #2250 and could reproducible
> get the same result (Kernel Crash). Seems like the ping isn't involved
> because the kernel crash happens even without it after changing the
> RA p-/vlifetime to 5/3. The crash occurs just a couple of seconds after it.

Thanks for testing.  Here's a diff that should prevent this panic, could
you confirm it works for you?

Index: netinet6/nd6_rtr.c
===================================================================
RCS file: /cvs/src/sys/netinet6/nd6_rtr.c,v
retrieving revision 1.144
diff -u -p -r1.144 nd6_rtr.c
--- netinet6/nd6_rtr.c    2 Sep 2016 11:51:07 -0000    1.144
+++ netinet6/nd6_rtr.c    7 Sep 2016 11:58:14 -0000
@@ -1972,9 +1972,11 @@ in6_ifadd(struct nd_prefix *pr, int priv

    ifra.ifra_flags |= IN6_IFF_AUTOCONF|IN6_IFF_TENTATIVE;

-    /* allocate ifaddr structure, link into chain, etc. */
+    /* If this address already exists, update it. */
+    ia6 = in6ifa_ifpwithaddr(ifp, &ifra.ifra_addr.sin6_addr);
+
    s = splsoftnet();
-    error = in6_update_ifa(ifp, &ifra, NULL);
+    error = in6_update_ifa(ifp, &ifra, ia6);
    splx(s);

    if (error != 0) {

OpenBSD6-0_after_patch.zip (87K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD 6.0 IPv6 issue / kernel crash

J.Sauer
In reply to this post by Martin Pieuchot
Sorry again,

i revisited the pcap 'complete_scenario.pcap' and it seems like after changing to "short RA's" the openbsd multicast address get's a NS but did not respond via NA. Would be pleased if some 'IPv6 gurus' could check it from previous post (attachement). Maybe the reason for paket loss but i am not quit sure ...


pcap from line:165 as example ...
cisco mac: 0000.1111.1111 / IPv6 2001:db8:1000::1

165    61.025848    2001:db8:1000::1    ff02::1:ff01:a5c7    ICMPv6    86    Neighbor Solicitation for 2001:db8:1000:0:c4b1:ad5:8b01:a5c7 from 00:00:11:11:11:11


Regards

J.Sauer

 -------
Jens Sauer
System Engineer

LinkedIn: http://de.linkedin.com/pub/jens-sauer/a8/a9b/b04/
GitHub:   https://github.com/NanoOps

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD 6.0 IPv6 issue / kernel crash

J.Sauer
In reply to this post by Martin Pieuchot
Hi OpenBSD Community,

i have tested the current snapshot again and could confirm that the previous crash doesn't occur anymore. The Ipv6 stack works like expected. Any paket loss 'heals' after a short time when the RA runs in 'normal' conditions again.


Great job!

Regards

J.Sauer