athn panic: invalid key cipher 0x0

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

athn panic: invalid key cipher 0x0

Kevin Chadwick-4

>Synopsis:      <athn0: device timeout - panic: invalid key cipher>
>Category:      <PR Kernel Panic>
>Environment:
        System      : OpenBSD 6.4
        Details     : OpenBSD 6.4-stable i386

        Architecture: OpenBSD.i386 IBM T42 pci athn
        Machine     : i386
>Description:
        <panic: invalid key cipher 0x0>
>How-To-Repeat:
        <Only seen once so far, may have occurred shortly after ifconfig athn0
down ifconfig athn0 up>
>Fix:
        <Restart/Unknown - panic message follows
panic: invalid key cipher 0x0
Stopped at db_enter+0x4: popl %ebp
db_enter() at db_enter+0x4
panic() at panic+0xcc
ieee80211_encrypt(d21ca030,d29c3a00,d2867150) at ieee80211_encrypt+0x62
ar5008_tx(d21ca000,d29c3a00,d2867000,0) at ar5008_tx+0x9b
athn_start(d21ca030) at athn_start+0x111
ar5008_intr(d21ca000) at ar5008_intr+0x1f7
pccbbintr(d2197200) at pccbbintr+0x108
intr_handler(f52b2a58, d218a880) at intr_handler+0x18
Xintr_legacy1_untramp(8,200246,d06f6725,d06f65b0,0) at Xintr_legacy1_untramp+0xf7
cpu_idle_cycle(d0cd9ff8) at cpu_idle_cycle+0xf

Still in ddb for a while if any more info is sought?
>

Reply | Threaded
Open this post in threaded view
|

Re: athn panic: invalid key cipher 0x0

Stefan Sperling-8
On Wed, Nov 28, 2018 at 10:04:52PM +0000, Kevin Chadwick wrote:

>
> >Synopsis:      <athn0: device timeout - panic: invalid key cipher>
> >Category:      <PR Kernel Panic>
> >Environment:
>         System      : OpenBSD 6.4
>         Details     : OpenBSD 6.4-stable i386
>
>         Architecture: OpenBSD.i386 IBM T42 pci athn
>         Machine     : i386
> >Description:
>         <panic: invalid key cipher 0x0>
> >How-To-Repeat:
>         <Only seen once so far, may have occurred shortly after ifconfig athn0
> down ifconfig athn0 up>
> >Fix:
>         <Restart/Unknown - panic message follows
> panic: invalid key cipher 0x0
> Stopped at db_enter+0x4: popl %ebp
> db_enter() at db_enter+0x4
> panic() at panic+0xcc
> ieee80211_encrypt(d21ca030,d29c3a00,d2867150) at ieee80211_encrypt+0x62
> ar5008_tx(d21ca000,d29c3a00,d2867000,0) at ar5008_tx+0x9b
> athn_start(d21ca030) at athn_start+0x111
> ar5008_intr(d21ca000) at ar5008_intr+0x1f7
> pccbbintr(d2197200) at pccbbintr+0x108
> intr_handler(f52b2a58, d218a880) at intr_handler+0x18
> Xintr_legacy1_untramp(8,200246,d06f6725,d06f65b0,0) at Xintr_legacy1_untramp+0xf7
> cpu_idle_cycle(d0cd9ff8) at cpu_idle_cycle+0xf
>
> Still in ddb for a while if any more info is sought?
> >

Similar panics have been reported to me against this driver in private.

No clue what's wrong yet.
I suspect there is a memory (mbuf) corruption bug that triggers this.

Reply | Threaded
Open this post in threaded view
|

Re: athn panic: invalid key cipher 0x0

Kevin Chadwick-4
On 11/29/18 8:12 AM, Stefan Sperling wrote:
> Similar panics have been reported to me against this driver in private.
>
> No clue what's wrong yet.
> I suspect there is a memory (mbuf) corruption bug that triggers this.

Ok, I rebooted, thanks.

Probably doesn't help but just in case.

My /etc/hostname.athn0 has the following possibly relevant lines.

wpaciphers ccmp
wpagroupcipher ccmp
wpa wpaprotos wpa2

Reply | Threaded
Open this post in threaded view
|

Re: athn panic: invalid key cipher 0x0

Stefan Sperling-8
On Thu, Nov 29, 2018 at 10:35:03AM +0000, Kevin Chadwick wrote:
> My /etc/hostname.athn0 has the following possibly relevant lines.
>
> wpaciphers ccmp
> wpagroupcipher ccmp
> wpa wpaprotos wpa2

This matches the default settings. You don't need to set those options.

Reply | Threaded
Open this post in threaded view
|

Re: athn panic: invalid key cipher 0x0

Stefan Sperling-8
In reply to this post by Stefan Sperling-8
On Thu, Nov 29, 2018 at 09:12:10AM +0100, Stefan Sperling wrote:

> On Wed, Nov 28, 2018 at 10:04:52PM +0000, Kevin Chadwick wrote:
> >
> > >Synopsis:      <athn0: device timeout - panic: invalid key cipher>
> > >Category:      <PR Kernel Panic>
> > >Environment:
> >         System      : OpenBSD 6.4
> >         Details     : OpenBSD 6.4-stable i386
> >
> >         Architecture: OpenBSD.i386 IBM T42 pci athn
> >         Machine     : i386
> > >Description:
> >         <panic: invalid key cipher 0x0>
> > >How-To-Repeat:
> >         <Only seen once so far, may have occurred shortly after ifconfig athn0
> > down ifconfig athn0 up>
> > >Fix:
> >         <Restart/Unknown - panic message follows
> > panic: invalid key cipher 0x0
> > Stopped at db_enter+0x4: popl %ebp
> > db_enter() at db_enter+0x4
> > panic() at panic+0xcc
> > ieee80211_encrypt(d21ca030,d29c3a00,d2867150) at ieee80211_encrypt+0x62
> > ar5008_tx(d21ca000,d29c3a00,d2867000,0) at ar5008_tx+0x9b
> > athn_start(d21ca030) at athn_start+0x111
> > ar5008_intr(d21ca000) at ar5008_intr+0x1f7
> > pccbbintr(d2197200) at pccbbintr+0x108
> > intr_handler(f52b2a58, d218a880) at intr_handler+0x18
> > Xintr_legacy1_untramp(8,200246,d06f6725,d06f65b0,0) at Xintr_legacy1_untramp+0xf7
> > cpu_idle_cycle(d0cd9ff8) at cpu_idle_cycle+0xf
> >
> > Still in ddb for a while if any more info is sought?
> > >
>
> Similar panics have been reported to me against this driver in private.
>
> No clue what's wrong yet.
> I suspect there is a memory (mbuf) corruption bug that triggers this.
>

Kevin, can you still reproduce this problem?
Could you try this diff please?

I see a possibility that this could be triggered by QoS data frames
from foreign networks, in which case the bug might not trigger reliably
unless you produce foreign 11n network traffic yourself.

diff 85d5a2d13c62494befd8404b760a73f8146aa70a /usr/src
blob - 24fd9c068968a041fcff497be8fdaa8705995001
file + sys/net80211/ieee80211_input.c
--- sys/net80211/ieee80211_input.c
+++ sys/net80211/ieee80211_input.c
@@ -210,11 +210,29 @@ ieee80211_input(struct ifnet *ifp, struct mbuf *m, str
  tid = 0;
  }
 
- if (type == IEEE80211_FC0_TYPE_DATA && hasqos &&
+ if (ic->ic_state == IEEE80211_S_RUN &&
+    type == IEEE80211_FC0_TYPE_DATA && hasqos &&
     (subtype & IEEE80211_FC0_SUBTYPE_NODATA) == 0 &&
-    !(rxi->rxi_flags & IEEE80211_RXI_AMPDU_DONE)) {
+    !(rxi->rxi_flags & IEEE80211_RXI_AMPDU_DONE)
+#ifndef IEEE80211_STA_ONLY
+    && (ic->ic_opmode == IEEE80211_M_STA || ni != ic->ic_bss)
+#endif
+    ) {
  int ba_state = ni->ni_rx_ba[tid].ba_state;
 
+#ifndef IEEE80211_STA_ONLY
+ if (ic->ic_opmode == IEEE80211_M_HOSTAP) {
+ if (!IEEE80211_ADDR_EQ(wh->i_addr1,
+    ic->ic_bss->ni_bssid)) {
+ ic->ic_stats.is_rx_wrongbss++;
+ goto err;
+ }
+ if (ni->ni_state != IEEE80211_S_ASSOC) {
+ ic->ic_stats.is_rx_notassoc++;
+ goto err;
+ }
+ }
+#endif
  /*
  * If Block Ack was explicitly requested, check
  * if we have a BA agreement for this RA/TID.


Reply | Threaded
Open this post in threaded view
|

Re: athn panic: invalid key cipher 0x0

Kevin Chadwick-4
On 3/20/19 8:58 AM, Stefan Sperling wrote:
>>> I see a possibility that this could be triggered by QoS data frames
>>> from foreign networks, in which case the bug might not trigger reliably
>>> unless you produce foreign 11n network traffic yourself.

>> OK, I shall play around and see if I can trigger it earlier.

> Thanks! Please try without the diff first, and once it has triggered
> try to trigger it again with the diff.


My network is in 11G mode. Is it still possible for n frames to trigger this bug?

Reply | Threaded
Open this post in threaded view
|

Re: athn panic: invalid key cipher 0x0

Stefan Sperling-8
On Thu, Mar 21, 2019 at 10:21:37AM +0000, Kevin Chadwick wrote:

> On 3/20/19 8:58 AM, Stefan Sperling wrote:
> >>> I see a possibility that this could be triggered by QoS data frames
> >>> from foreign networks, in which case the bug might not trigger reliably
> >>> unless you produce foreign 11n network traffic yourself.
>
> >> OK, I shall play around and see if I can trigger it earlier.
>
> > Thanks! Please try without the diff first, and once it has triggered
> > try to trigger it again with the diff.
>
>
> My network is in 11G mode. Is it still possible for n frames to trigger this bug?

I cannot easily tell you that. What I'm doing is I'm going through the
code trying to spot code paths which could make us hit that assertion.
I cannot yet reproduce this problem myself, and I cannot tell you all
the variables which might be involved. I am making blind guesses which
need to be verified by actual tests from people who are seeing this issue.

Reply | Threaded
Open this post in threaded view
|

Re: athn panic: invalid key cipher 0x0

Stefan Sperling-8
On Thu, Mar 21, 2019 at 11:38:44AM +0100, Stefan Sperling wrote:

> On Thu, Mar 21, 2019 at 10:21:37AM +0000, Kevin Chadwick wrote:
> > On 3/20/19 8:58 AM, Stefan Sperling wrote:
> > >>> I see a possibility that this could be triggered by QoS data frames
> > >>> from foreign networks, in which case the bug might not trigger reliably
> > >>> unless you produce foreign 11n network traffic yourself.
> >
> > >> OK, I shall play around and see if I can trigger it earlier.
> >
> > > Thanks! Please try without the diff first, and once it has triggered
> > > try to trigger it again with the diff.
> >
> >
> > My network is in 11G mode. Is it still possible for n frames to trigger this bug?
>
> I cannot easily tell you that. What I'm doing is I'm going through the
> code trying to spot code paths which could make us hit that assertion.
> I cannot yet reproduce this problem myself, and I cannot tell you all
> the variables which might be involved. I am making blind guesses which
> need to be verified by actual tests from people who are seeing this issue.

That said, if anyone is able to unearth more evidence such as packet captures
leading up to the problem, that would of course help me a great deal.

At present, the best method is to listen with a separate iwn(4) device
in monitor mode on the same channel. Example for channel 1:
  ifconfig iwn0 down -nwid
  ifconfig iwn0 mode 11n
  ifconfig iwn0 mediaopt monitor
  ifconfig iwn0 chan 1
  ifconfig iwn0 chan 1 # workaround for channel-setting bug in monitor mode
  ifconfig iwn0 up
  tcpdump -n -i iwn0 -y IEEE802_11_RADIO -s 4096 -w /tmp/iwn.pcap
If you need more help with this, just ask.