SSH on IPv6: RST packet after few inactivity

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

SSH on IPv6: RST packet after few inactivity

Sebastien Marie-3
Hi,

I have a bug since long time on my local network.

My network has IPv6, and I usually use IPv6 addresses to connect to
them. It is globally reachable addresses.

When I connect to ssh server and keep the shell running (without
activity, no tmux with "moving" status bar due to date or loadavg),
after some time (10min seems enough) if I start typing in the terminal,
the first char is sent to server, and the connection reset. the client
doesn't see the echo of the char (I know server has the char because on
simple tmux session, when reattach, the char is present).

This time, I managed to have a tcpdump trace on the client, and on the
server for the problem. After comparing packet per packet the output,
the two hosts saw the same things.

Here the tcpdump output (-vvv). I used sed to rename IPv6 addresses to
names (for better lisibility).

bert is the SSH server  : 2001:41d0:fe39:c05c:afcb:ae83:596f:47e5 (stable soii address)
clyde is the SSH client : 2001:41d0:fe39:c05c:f5eb:676d:ef8f:61f (current active outgoing autoconfprivacy address)

10:39:58.057999 bert.22 > clyde.39234: P 2856085097:2856085165(68) ack 652718852 win 267 <nop,nop,timestamp 2288520555 144012050> [class 0x48] [flowlabel 0x6d55f] (len 100, hlim 64)
10:39:58.058054 clyde.39234 > bert.22: . [tcp sum ok] 1:1(0) ack 68 win 254 <nop,nop,timestamp 144012050 2288520555> [class 0x48] [flowlabel 0x46f7a] (len 32, hlim 64)
10:39:58.058544 bert.22 > clyde.39234: P 68:120(52) ack 1 win 267 <nop,nop,timestamp 2288520555 144012050> [class 0x48] [flowlabel 0x6d55f] (len 84, hlim 64)
10:39:58.249290 clyde.39234 > bert.22: . [tcp sum ok] 1:1(0) ack 120 win 256 <nop,nop,timestamp 144012050 2288520555> [class 0x48] [flowlabel 0x46f7a] (len 32, hlim 64)
10:40:01.090429 bert > clyde: icmp6: neighbor sol: who has clyde(src lladdr: 00:15:c5:0b:8b:7a) [icmp6 cksum ok] (len 32, hlim 255)
10:40:01.090544 2001:41d0:fe39:c05c:9e5f:772e:e1d0:6d94 > bert: icmp6: neighbor adv: tgt is clyde(S) [icmp6 cksum ok] (len 24, hlim 255)
10:52:43.021623 clyde.39234 > bert.22: P 1:37(36) ack 120 win 256 <nop,nop,timestamp 144013579 2288520555> [class 0x48] [flowlabel 0x46f7a] (len 68, hlim 64)
10:52:43.022002 bert.22 > clyde.39234: P 120:164(44) ack 37 win 267 <nop,nop,timestamp 2288522085 144013579> [class 0x48] [flowlabel 0x6d55f] (len 76, hlim 64)
10:52:43.022081 clyde.39234 > bert.22: R [tcp sum ok] 652718888:652718888(0) win 0 (len 20, hlim 64)
10:52:43.022165 bert.22 > clyde.39234: P 164:216(52) ack 37 win 267 <nop,nop,timestamp 2288522085 144013579> [class 0x48] [flowlabel 0x6d55f] (len 84, hlim 64)
10:52:43.022232 clyde.39234 > bert.22: R [tcp sum ok] 652718888:652718888(0) win 0 (len 20, hlim 64)

The connection was already running (I am on X11, st terminal opened,
I ran 'ssh bert'). 10:39:58.249290 is my last interaction. Next, at
10:52:43.021623, I tapped some char on the terminal.

Packet is sent from client (clyde) to server (bert), and the server acks
the packet. Next, the client sent RST.

In the trace, bert asked clyde for neighbor sol, and clyde replied using
soii address that tgt is current-autoconfprivacy.

On the client (clyde), ifconfig was the following:

$ ifconfig bge0
bge0: flags=a08843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6,AUTOCONF4> mtu 1500
        lladdr 00:1b:38:33:97:b0
        index 1 priority 0 llprio 3
        groups: egress
        media: Ethernet autoselect (1000baseT full-duplex,rxpause,txpause)
        status: active
        inet6 fe80::2e75:6f0d:e815:6b0c%bge0 prefixlen 64 scopeid 0x1
        inet 192.168.92.12 netmask 0xffffff00 broadcast 192.168.92.255
        inet6 2001:41d0:fe39:c05c:9e5f:772e:e1d0:6d94 prefixlen 64 autoconf pltime 604635 vltime 2591835
        inet6 2001:41d0:fe39:c05c:e91d:830d:68e:68f2 prefixlen 64 deprecated autoconf autoconfprivacy pltime 0 vltime 363307
        inet6 2001:41d0:fe39:c05c:de9f:bec:d27e:756c prefixlen 64 deprecated autoconf autoconfprivacy pltime 0 vltime 449221
        inet6 2001:41d0:fe39:c05c:f5eb:676d:ef8f:61f prefixlen 64 autoconf autoconfprivacy pltime 16822 vltime 535233

On the server (bert), ifconfig is currently (~1h30 after the tcpdump) the following:
$ ifconfig bce0
bce0: flags=a08a43<UP,BROADCAST,RUNNING,ALLMULTI,SIMPLEX,MULTICAST,AUTOCONF6,AUTOCONF4> mtu 1500
        lladdr 00:15:c5:0b:8b:7a
        index 2 priority 0 llprio 3
        groups: egress
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet6 fe80::1c1e:c211:d802:ceb6%bce0 prefixlen 64 scopeid 0x2
        inet 192.168.92.11 netmask 0xffffff00 broadcast 192.168.92.255
        inet6 2001:41d0:fe39:c05c:afcb:ae83:596f:47e5 prefixlen 64 autoconf pltime 604798 vltime 2591998
        inet6 2001:41d0:fe39:c05c:915e:9dce:91e0:790 prefixlen 64 autoconf autoconfprivacy pltime 16717 vltime 535488


I have reproduced the problem:
- using another server than 'bert' (but still from 'clyde')
- using another tcp protocol (plain tcp stream with nc(1))


bert has default pf.conf configuration.

clyde has a more complex pf.conf, but has 'pass in inet6 proto
ipv6-icmp' as last rule (and no quick rule). I have a 'block in log
all', and nothing in /var/log/pflog at the time.

Any advice on possible fallout is welcome.

Thanks.
--
Sebastien Marie

Reply | Threaded
Open this post in threaded view
|

Re: SSH on IPv6: RST packet after few inactivity

Alexander Bluhm
On Wed, Aug 14, 2019 at 12:33:04PM +0200, Sebastien Marie wrote:
> 10:52:43.021623 clyde.39234 > bert.22: P 1:37(36) ack 120 win 256 <nop,nop,timestamp 144013579 2288520555> [class 0x48] [flowlabel 0x46f7a] (len 68, hlim 64)
> 10:52:43.022002 bert.22 > clyde.39234: P 120:164(44) ack 37 win 267 <nop,nop,timestamp 2288522085 144013579> [class 0x48] [flowlabel 0x6d55f] (len 76, hlim 64)
> 10:52:43.022081 clyde.39234 > bert.22: R [tcp sum ok] 652718888:652718888(0) win 0 (len 20, hlim 64)

So you see TCP resets that make no sense after some time of inactivity.

In my experience this is a bridge(4) somewhere in the network running
pf(4) with a default "block return" rule.  The entries in the bridge
MAC table timeout, the bridge sends a broadcast to all ports, this
packet hits pf on an interface, where it is not expected.  Then pf
generates a TCP reset packet.

Run tcpdump -e and check whether the MAC address of the reset matches
clyde or bert.  If not, you have the bad machine.

bluhm

Reply | Threaded
Open this post in threaded view
|

Re: SSH on IPv6: RST packet after few inactivity

Darren Tucker-3
In reply to this post by Sebastien Marie-3
On Wed, 14 Aug 2019 at 04:02, Sebastien Marie <[hidden email]> wrote:
[...]
> When I connect to ssh server and keep the shell running (without
> activity, no tmux with "moving" status bar due to date or loadavg),
> after some time (10min seems enough) if I start typing in the terminal,
> the first char is sent to server, and the connection reset. the client
> doesn't see the echo of the char (I know server has the char because on
> simple tmux session, when reattach, the char is present).

FWIW if this is caused by some kind inactivity timeout you can
mitigate it by enabling ServerAliveInterval in the ssh client or the
equivalent ClientAliveInterval in the sshdconfiguration.

--
Darren Tucker (dtucker at dtucker.net)
GPG key 11EAA6FA / A86E 3E07 5B19 5880 E860  37F4 9357 ECEF 11EA A6FA (new)
    Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.