kernel/5199: TCP window scale of 7 causes OpenBSD TCP misbehavior

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

kernel/5199: TCP window scale of 7 causes OpenBSD TCP misbehavior

mazieres-tdc4wkyv457zec9ebw8nngy2r2
>Number:         5199
>Category:       kernel
>Synopsis:       TCP window scale of 7 causes OpenBSD TCP misbehavior
>Confidential:   yes
>Severity:       serious
>Priority:       high
>Responsible:    bugs
>State:          open
>Quarter:        
>Keywords:      
>Date-Required:
>Class:          sw-bug
>Submitter-Id:   net
>Arrival-Date:   Sat Aug 12 01:40:01 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator:     David Mazieres
>Release:        3.9 i386
>Organization:
net
>Environment:
       
        System      : OpenBSD 3.9
        Architecture: OpenBSD.i386
        Machine     : i386
>Description:

pf appears to mangle outgoing packets in some circumstances
when the client's TCP window scale option is set to 7.

>How-To-Repeat:

This problem appeared with amd64 servers running Gentoo Linux, between
linux versions 2.6.16 and 2.6.17, when the client's window scale option changed from 2 to 7.

To repeat this, the OpenBSD server must have pf enabled with the
following rule:

pass out keep state

Then, when using a Linux client, various commands will hang instead of
working with the OpenBSD server.  For example, here is a packet trace
taken with bpf on the OpenBSD server when running "rpcinfo -p server"
on the client:

  1   0.000000  Linux -> OpenBSD  TCP 48855 > sunrpc [SYN] Seq=0 Len=0 MSS=1460 TSV=1730873 TSER=0 WS=7
  2   0.000031  OpenBSD -> Linux  TCP sunrpc > 48855 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=0 TSV=3830204155 TSER=1730873
  3   0.000249  Linux -> OpenBSD  TCP 48855 > sunrpc [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=1730873 TSER=3830204155
  4   0.001748  Linux -> OpenBSD  Portmap V2 DUMP Call
  5   0.201004  OpenBSD -> Linux  TCP [TCP Previous segment lost] sunrpc > 48855 [ACK] Seq=313 Ack=45 Win=17376 Len=0 TSV=3830204155 TSER=1730874
  6  32.883021  Linux -> OpenBSD  TCP 48855 > sunrpc [FIN, ACK] Seq=45 Ack=1 Win=5888 Len=0 TSV=1739094 TSER=3830204155
  7  32.883040  OpenBSD -> Linux  TCP sunrpc > 48855 [ACK] Seq=313 Ack=46 Win=17376 Len=0 TSV=3830204221 TSER=1739094
  8  32.883081  OpenBSD -> Linux  TCP sunrpc > 48855 [FIN, ACK] Seq=313 Ack=46 Win=17376 Len=0 TSV=3830204221 TSER=1739094
  9  32.883271  Linux -> OpenBSD  TCP 48855 > sunrpc [RST] Seq=46 Len=0

There are several problems with packet 5.  Most importantly, the
sequence number is wrong.  Second, the Push bit is not set (it
should be as this is the end of a write by portmap).  Finally
note that OpenBSD never retransmits the packet in 30 seconds,
despite receiving an ack, which is weird, but possibly unrelated.

Here is a more detailed dump of the first 5 packets (using tshark -V).
(Linux is the machine with a 10.0.0.13 address, while OpenBSD is
10.0.0.11.)  What seems to trigger this bug is the window scale value of
7 in the first frame, as this is the main difference compared to an
older Linux kernel that doesn't trigger the bug in OpenBSD.


Frame 1 (74 bytes on wire, 74 bytes captured)
    Arrival Time: Aug 11, 2006 16:14:29.163524000
    Time delta from previous packet: 0.000000000 seconds
    Time since reference or first frame: 0.000000000 seconds
    Frame Number: 1
    Packet Length: 74 bytes
    Capture Length: 74 bytes
    Frame is marked: False
    Protocols in frame: eth:ip:tcp
Ethernet II, Src: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41), Dst: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
    Destination: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
        Address: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
        .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
        .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
    Source: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
        Address: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
        .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
        .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
    Type: IP (0x0800)
Internet Protocol, Src: 10.0.0.13 (10.0.0.13), Dst: 10.0.0.11 (10.0.0.11)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 60
    Identification: 0x40f5 (16629)
    Flags: 0x04 (Don't Fragment)
        0... = Reserved bit: Not set
        .1.. = Don't fragment: Set
        ..0. = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: TCP (0x06)
    Header checksum: 0x9d2a [correct]
        Good: True
        Bad : False
    Source: 10.0.0.13 (10.0.0.13)
    Destination: 10.0.0.11 (10.0.0.11)
Transmission Control Protocol, Src Port: 48855 (48855), Dst Port: 111 (111), Seq: 0, Len: 0
    Source port: 48855 (48855)
    Destination port: 111 (111)
    Sequence number: 0    (relative sequence number)
    Header length: 40 bytes
    Flags: 0x0002 (SYN)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...0 .... = Acknowledgment: Not set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..1. = Syn: Set
        .... ...0 = Fin: Not set
    Window size: 5840
    Checksum: 0x4e64 [correct]
    Options: (20 bytes)
        Maximum segment size: 1460 bytes
        SACK permitted
        Time stamp: tsval 1730873, tsecr 0
        NOP
        Window scale: 7 (multiply by 128)

Frame 2 (78 bytes on wire, 78 bytes captured)
    Arrival Time: Aug 11, 2006 16:14:29.163555000
    Time delta from previous packet: 0.000031000 seconds
    Time since reference or first frame: 0.000031000 seconds
    Frame Number: 2
    Packet Length: 78 bytes
    Capture Length: 78 bytes
    Frame is marked: False
    Protocols in frame: eth:ip:tcp
Ethernet II, Src: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8), Dst: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
    Destination: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
        Address: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
        .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
        .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
    Source: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
        Address: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
        .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
        .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
    Type: IP (0x0800)
Internet Protocol, Src: 10.0.0.11 (10.0.0.11), Dst: 10.0.0.13 (10.0.0.13)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 64
    Identification: 0xd020 (53280)
    Flags: 0x00
        0... = Reserved bit: Not set
        .0.. = Don't fragment: Not set
        ..0. = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: TCP (0x06)
    Header checksum: 0x4dfb [correct]
        Good: True
        Bad : False
    Source: 10.0.0.11 (10.0.0.11)
    Destination: 10.0.0.13 (10.0.0.13)
Transmission Control Protocol, Src Port: 111 (111), Dst Port: 48855 (48855), Seq: 0, Ack: 1, Len: 0
    Source port: 111 (111)
    Destination port: 48855 (48855)
    Sequence number: 0    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 44 bytes
    Flags: 0x0012 (SYN, ACK)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...1 .... = Acknowledgment: Set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..1. = Syn: Set
        .... ...0 = Fin: Not set
    Window size: 16384
    Checksum: 0x9ba1 [correct]
    Options: (24 bytes)
        Maximum segment size: 1460 bytes
        NOP
        NOP
        SACK permitted
        NOP
        Window scale: 0 (multiply by 1)
        NOP
        NOP
        Time stamp: tsval 3830204155, tsecr 1730873

Frame 3 (66 bytes on wire, 66 bytes captured)
    Arrival Time: Aug 11, 2006 16:14:29.163773000
    Time delta from previous packet: 0.000218000 seconds
    Time since reference or first frame: 0.000249000 seconds
    Frame Number: 3
    Packet Length: 66 bytes
    Capture Length: 66 bytes
    Frame is marked: False
    Protocols in frame: eth:ip:tcp
Ethernet II, Src: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41), Dst: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
    Destination: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
        Address: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
        .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
        .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
    Source: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
        Address: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
        .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
        .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
    Type: IP (0x0800)
Internet Protocol, Src: 10.0.0.13 (10.0.0.13), Dst: 10.0.0.11 (10.0.0.11)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 52
    Identification: 0x40f6 (16630)
    Flags: 0x04 (Don't Fragment)
        0... = Reserved bit: Not set
        .1.. = Don't fragment: Set
        ..0. = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: TCP (0x06)
    Header checksum: 0x9d31 [correct]
        Good: True
        Bad : False
    Source: 10.0.0.13 (10.0.0.13)
    Destination: 10.0.0.11 (10.0.0.11)
Transmission Control Protocol, Src Port: 48855 (48855), Dst Port: 111 (111), Seq: 1, Ack: 1, Len: 0
    Source port: 48855 (48855)
    Destination port: 111 (111)
    Sequence number: 1    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 32 bytes
    Flags: 0x0010 (ACK)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...1 .... = Acknowledgment: Set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..0. = Syn: Not set
        .... ...0 = Fin: Not set
    Window size: 5888 (scaled)
    Checksum: 0x1c3f [correct]
    Options: (12 bytes)
        NOP
        NOP
        Time stamp: tsval 1730873, tsecr 3830204155

Frame 4 (110 bytes on wire, 110 bytes captured)
    Arrival Time: Aug 11, 2006 16:14:29.165272000
    Time delta from previous packet: 0.001499000 seconds
    Time since reference or first frame: 0.001748000 seconds
    Frame Number: 4
    Packet Length: 110 bytes
    Capture Length: 110 bytes
    Frame is marked: False
    Protocols in frame: eth:ip:tcp:rpc
Ethernet II, Src: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41), Dst: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
    Destination: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
        Address: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
        .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
        .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
    Source: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
        Address: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
        .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
        .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
    Type: IP (0x0800)
Internet Protocol, Src: 10.0.0.13 (10.0.0.13), Dst: 10.0.0.11 (10.0.0.11)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 96
    Identification: 0x40f7 (16631)
    Flags: 0x04 (Don't Fragment)
        0... = Reserved bit: Not set
        .1.. = Don't fragment: Set
        ..0. = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: TCP (0x06)
    Header checksum: 0x9d04 [correct]
        Good: True
        Bad : False
    Source: 10.0.0.13 (10.0.0.13)
    Destination: 10.0.0.11 (10.0.0.11)
Transmission Control Protocol, Src Port: 48855 (48855), Dst Port: 111 (111), Seq: 1, Ack: 1, Len: 44
    Source port: 48855 (48855)
    Destination port: 111 (111)
    Sequence number: 1    (relative sequence number)
    Next sequence number: 45    (relative sequence number)
    Acknowledgement number: 1    (relative ack number)
    Header length: 32 bytes
    Flags: 0x0018 (PSH, ACK)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...1 .... = Acknowledgment: Set
        .... 1... = Push: Set
        .... .0.. = Reset: Not set
        .... ..0. = Syn: Not set
        .... ...0 = Fin: Not set
    Window size: 5888 (scaled)
    Checksum: 0x8a9a [correct]
    Options: (12 bytes)
        NOP
        NOP
        Time stamp: tsval 1730874, tsecr 3830204155
Remote Procedure Call, Type:Call XID:0x7a241079
    Fragment header: Last fragment, 40 bytes
        1... .... .... .... .... .... .... .... = Last Fragment: Yes
        .000 0000 0000 0000 0000 0000 0010 1000 = Fragment Length: 40
    XID: 0x7a241079 (2049183865)
    Message Type: Call (0)
    RPC Version: 2
    Program: Portmap (100000)
    Program Version: 2
    Procedure: DUMP (4)
    Credentials
        Flavor: AUTH_NULL (0)
        Length: 0
    Verifier
        Flavor: AUTH_NULL (0)
        Length: 0
Portmap
    Program Version: 2
    V2 Procedure: DUMP (4)

Frame 5 (66 bytes on wire, 66 bytes captured)
    Arrival Time: Aug 11, 2006 16:14:29.364528000
    Time delta from previous packet: 0.199256000 seconds
    Time since reference or first frame: 0.201004000 seconds
    Frame Number: 5
    Packet Length: 66 bytes
    Capture Length: 66 bytes
    Frame is marked: False
    Protocols in frame: eth:ip:tcp
Ethernet II, Src: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8), Dst: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
    Destination: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
        Address: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
        .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
        .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
    Source: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
        Address: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
        .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
        .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
    Type: IP (0x0800)
Internet Protocol, Src: 10.0.0.11 (10.0.0.11), Dst: 10.0.0.13 (10.0.0.13)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 52
    Identification: 0xce4c (52812)
    Flags: 0x00
        0... = Reserved bit: Not set
        .0.. = Don't fragment: Not set
        ..0. = More fragments: Not set
    Fragment offset: 0
    Time to live: 64
    Protocol: TCP (0x06)
    Header checksum: 0x4fdb [correct]
        Good: True
        Bad : False
    Source: 10.0.0.11 (10.0.0.11)
    Destination: 10.0.0.13 (10.0.0.13)
Transmission Control Protocol, Src Port: 111 (111), Dst Port: 48855 (48855), Seq: 313, Ack: 45, Len: 0
    Source port: 111 (111)
    Destination port: 48855 (48855)
    Sequence number: 313    (relative sequence number)
    Acknowledgement number: 45    (relative ack number)
    Header length: 32 bytes
    Flags: 0x0010 (ACK)
        0... .... = Congestion Window Reduced (CWR): Not set
        .0.. .... = ECN-Echo: Not set
        ..0. .... = Urgent: Not set
        ...1 .... = Acknowledgment: Set
        .... 0... = Push: Not set
        .... .0.. = Reset: Not set
        .... ..0. = Syn: Not set
        .... ...0 = Fin: Not set
    Window size: 17376
    Checksum: 0xd727 [correct]
    Options: (12 bytes)
        NOP
        NOP
        Time stamp: tsval 3830204155, tsecr 1730874
    SEQ/ACK analysis
        TCP Analysis Flags
            A segment before this frame was lost

>Fix:

A workaround is to disable window scaling in clients that experience
this problem when talking to OpenBSD servers, or to fiddle with the pf
rules on the server.


>Release-Note:
>Audit-Trail:
>Unformatted:

Reply | Threaded
Open this post in threaded view
|

Re: kernel/5199: TCP window scale of 7 causes OpenBSD TCP misbehavior

David Krause
The following reply was made to PR kernel/5199; it has been noted by GNATS.

From: David Krause <[hidden email]>
To: [hidden email]
Cc: [hidden email]
Subject: Re: kernel/5199: TCP window scale of 7 causes OpenBSD TCP misbehavior
Date: Sat, 12 Aug 2006 00:15:37 -0500

 Check your ruleset and look for ANY pass rules that do not have 'keep
 state' and add it.  There is some other rule besides the 'pass out keep
 state' rule that your Linux packets are matching and it doesn't have
 'keep state'.
 
 See http://www.benzedrine.cx/pf/msg05117.html for details.
 
 Also, if you cannot immediately fix your pf ruleset, there is no need to
 disable window scaling on the Linux systems.  There is a proc value
 (don't know it off the top of my head) that will revert to the previous
 kernel's behavior.  But you really need to fix the ruleset.
 
 David
 
 * [hidden email] <[hidden email]> [060811 20:51]:
 > >Number:         5199
 > >Category:       kernel
 > >Synopsis:       TCP window scale of 7 causes OpenBSD TCP misbehavior
 > >Confidential:   yes
 > >Severity:       serious
 > >Priority:       high
 > >Responsible:    bugs
 > >State:          open
 > >Quarter:        
 > >Keywords:      
 > >Date-Required:
 > >Class:          sw-bug
 > >Submitter-Id:   net
 > >Arrival-Date:   Sat Aug 12 01:40:01 GMT 2006
 > >Closed-Date:
 > >Last-Modified:
 > >Originator:     David Mazieres
 > >Release:        3.9 i386
 > >Organization:
 > net
 > >Environment:
 >
 > System      : OpenBSD 3.9
 > Architecture: OpenBSD.i386
 > Machine     : i386
 > >Description:
 >
 > pf appears to mangle outgoing packets in some circumstances
 > when the client's TCP window scale option is set to 7.
 >
 > >How-To-Repeat:
 >
 > This problem appeared with amd64 servers running Gentoo Linux, between
 > linux versions 2.6.16 and 2.6.17, when the client's window scale option changed from 2 to 7.
 >
 > To repeat this, the OpenBSD server must have pf enabled with the
 > following rule:
 >
 > pass out keep state
 >
 > Then, when using a Linux client, various commands will hang instead of
 > working with the OpenBSD server.  For example, here is a packet trace
 > taken with bpf on the OpenBSD server when running "rpcinfo -p server"
 > on the client:
 >
 >   1   0.000000  Linux -> OpenBSD  TCP 48855 > sunrpc [SYN] Seq=0 Len=0 MSS=1460 TSV=1730873 TSER=0 WS=7
 >   2   0.000031  OpenBSD -> Linux  TCP sunrpc > 48855 [SYN, ACK] Seq=0 Ack=1 Win=16384 Len=0 MSS=1460 WS=0 TSV=3830204155 TSER=1730873
 >   3   0.000249  Linux -> OpenBSD  TCP 48855 > sunrpc [ACK] Seq=1 Ack=1 Win=5888 Len=0 TSV=1730873 TSER=3830204155
 >   4   0.001748  Linux -> OpenBSD  Portmap V2 DUMP Call
 >   5   0.201004  OpenBSD -> Linux  TCP [TCP Previous segment lost] sunrpc > 48855 [ACK] Seq=313 Ack=45 Win=17376 Len=0 TSV=3830204155 TSER=1730874
 >   6  32.883021  Linux -> OpenBSD  TCP 48855 > sunrpc [FIN, ACK] Seq=45 Ack=1 Win=5888 Len=0 TSV=1739094 TSER=3830204155
 >   7  32.883040  OpenBSD -> Linux  TCP sunrpc > 48855 [ACK] Seq=313 Ack=46 Win=17376 Len=0 TSV=3830204221 TSER=1739094
 >   8  32.883081  OpenBSD -> Linux  TCP sunrpc > 48855 [FIN, ACK] Seq=313 Ack=46 Win=17376 Len=0 TSV=3830204221 TSER=1739094
 >   9  32.883271  Linux -> OpenBSD  TCP 48855 > sunrpc [RST] Seq=46 Len=0
 >
 > There are several problems with packet 5.  Most importantly, the
 > sequence number is wrong.  Second, the Push bit is not set (it
 > should be as this is the end of a write by portmap).  Finally
 > note that OpenBSD never retransmits the packet in 30 seconds,
 > despite receiving an ack, which is weird, but possibly unrelated.
 >
 > Here is a more detailed dump of the first 5 packets (using tshark -V).
 > (Linux is the machine with a 10.0.0.13 address, while OpenBSD is
 > 10.0.0.11.)  What seems to trigger this bug is the window scale value of
 > 7 in the first frame, as this is the main difference compared to an
 > older Linux kernel that doesn't trigger the bug in OpenBSD.
 >
 >
 > Frame 1 (74 bytes on wire, 74 bytes captured)
 >     Arrival Time: Aug 11, 2006 16:14:29.163524000
 >     Time delta from previous packet: 0.000000000 seconds
 >     Time since reference or first frame: 0.000000000 seconds
 >     Frame Number: 1
 >     Packet Length: 74 bytes
 >     Capture Length: 74 bytes
 >     Frame is marked: False
 >     Protocols in frame: eth:ip:tcp
 > Ethernet II, Src: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41), Dst: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
 >     Destination: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
 >         Address: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
 >         .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
 >         .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
 >     Source: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
 >         Address: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
 >         .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
 >         .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
 >     Type: IP (0x0800)
 > Internet Protocol, Src: 10.0.0.13 (10.0.0.13), Dst: 10.0.0.11 (10.0.0.11)
 >     Version: 4
 >     Header length: 20 bytes
 >     Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
 >         0000 00.. = Differentiated Services Codepoint: Default (0x00)
 >         .... ..0. = ECN-Capable Transport (ECT): 0
 >         .... ...0 = ECN-CE: 0
 >     Total Length: 60
 >     Identification: 0x40f5 (16629)
 >     Flags: 0x04 (Don't Fragment)
 >         0... = Reserved bit: Not set
 >         .1.. = Don't fragment: Set
 >         ..0. = More fragments: Not set
 >     Fragment offset: 0
 >     Time to live: 64
 >     Protocol: TCP (0x06)
 >     Header checksum: 0x9d2a [correct]
 >         Good: True
 >         Bad : False
 >     Source: 10.0.0.13 (10.0.0.13)
 >     Destination: 10.0.0.11 (10.0.0.11)
 > Transmission Control Protocol, Src Port: 48855 (48855), Dst Port: 111 (111), Seq: 0, Len: 0
 >     Source port: 48855 (48855)
 >     Destination port: 111 (111)
 >     Sequence number: 0    (relative sequence number)
 >     Header length: 40 bytes
 >     Flags: 0x0002 (SYN)
 >         0... .... = Congestion Window Reduced (CWR): Not set
 >         .0.. .... = ECN-Echo: Not set
 >         ..0. .... = Urgent: Not set
 >         ...0 .... = Acknowledgment: Not set
 >         .... 0... = Push: Not set
 >         .... .0.. = Reset: Not set
 >         .... ..1. = Syn: Set
 >         .... ...0 = Fin: Not set
 >     Window size: 5840
 >     Checksum: 0x4e64 [correct]
 >     Options: (20 bytes)
 >         Maximum segment size: 1460 bytes
 >         SACK permitted
 >         Time stamp: tsval 1730873, tsecr 0
 >         NOP
 >         Window scale: 7 (multiply by 128)
 >
 > Frame 2 (78 bytes on wire, 78 bytes captured)
 >     Arrival Time: Aug 11, 2006 16:14:29.163555000
 >     Time delta from previous packet: 0.000031000 seconds
 >     Time since reference or first frame: 0.000031000 seconds
 >     Frame Number: 2
 >     Packet Length: 78 bytes
 >     Capture Length: 78 bytes
 >     Frame is marked: False
 >     Protocols in frame: eth:ip:tcp
 > Ethernet II, Src: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8), Dst: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
 >     Destination: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
 >         Address: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
 >         .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
 >         .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
 >     Source: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
 >         Address: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
 >         .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
 >         .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
 >     Type: IP (0x0800)
 > Internet Protocol, Src: 10.0.0.11 (10.0.0.11), Dst: 10.0.0.13 (10.0.0.13)
 >     Version: 4
 >     Header length: 20 bytes
 >     Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
 >         0000 00.. = Differentiated Services Codepoint: Default (0x00)
 >         .... ..0. = ECN-Capable Transport (ECT): 0
 >         .... ...0 = ECN-CE: 0
 >     Total Length: 64
 >     Identification: 0xd020 (53280)
 >     Flags: 0x00
 >         0... = Reserved bit: Not set
 >         .0.. = Don't fragment: Not set
 >         ..0. = More fragments: Not set
 >     Fragment offset: 0
 >     Time to live: 64
 >     Protocol: TCP (0x06)
 >     Header checksum: 0x4dfb [correct]
 >         Good: True
 >         Bad : False
 >     Source: 10.0.0.11 (10.0.0.11)
 >     Destination: 10.0.0.13 (10.0.0.13)
 > Transmission Control Protocol, Src Port: 111 (111), Dst Port: 48855 (48855), Seq: 0, Ack: 1, Len: 0
 >     Source port: 111 (111)
 >     Destination port: 48855 (48855)
 >     Sequence number: 0    (relative sequence number)
 >     Acknowledgement number: 1    (relative ack number)
 >     Header length: 44 bytes
 >     Flags: 0x0012 (SYN, ACK)
 >         0... .... = Congestion Window Reduced (CWR): Not set
 >         .0.. .... = ECN-Echo: Not set
 >         ..0. .... = Urgent: Not set
 >         ...1 .... = Acknowledgment: Set
 >         .... 0... = Push: Not set
 >         .... .0.. = Reset: Not set
 >         .... ..1. = Syn: Set
 >         .... ...0 = Fin: Not set
 >     Window size: 16384
 >     Checksum: 0x9ba1 [correct]
 >     Options: (24 bytes)
 >         Maximum segment size: 1460 bytes
 >         NOP
 >         NOP
 >         SACK permitted
 >         NOP
 >         Window scale: 0 (multiply by 1)
 >         NOP
 >         NOP
 >         Time stamp: tsval 3830204155, tsecr 1730873
 >
 > Frame 3 (66 bytes on wire, 66 bytes captured)
 >     Arrival Time: Aug 11, 2006 16:14:29.163773000
 >     Time delta from previous packet: 0.000218000 seconds
 >     Time since reference or first frame: 0.000249000 seconds
 >     Frame Number: 3
 >     Packet Length: 66 bytes
 >     Capture Length: 66 bytes
 >     Frame is marked: False
 >     Protocols in frame: eth:ip:tcp
 > Ethernet II, Src: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41), Dst: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
 >     Destination: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
 >         Address: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
 >         .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
 >         .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
 >     Source: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
 >         Address: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
 >         .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
 >         .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
 >     Type: IP (0x0800)
 > Internet Protocol, Src: 10.0.0.13 (10.0.0.13), Dst: 10.0.0.11 (10.0.0.11)
 >     Version: 4
 >     Header length: 20 bytes
 >     Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
 >         0000 00.. = Differentiated Services Codepoint: Default (0x00)
 >         .... ..0. = ECN-Capable Transport (ECT): 0
 >         .... ...0 = ECN-CE: 0
 >     Total Length: 52
 >     Identification: 0x40f6 (16630)
 >     Flags: 0x04 (Don't Fragment)
 >         0... = Reserved bit: Not set
 >         .1.. = Don't fragment: Set
 >         ..0. = More fragments: Not set
 >     Fragment offset: 0
 >     Time to live: 64
 >     Protocol: TCP (0x06)
 >     Header checksum: 0x9d31 [correct]
 >         Good: True
 >         Bad : False
 >     Source: 10.0.0.13 (10.0.0.13)
 >     Destination: 10.0.0.11 (10.0.0.11)
 > Transmission Control Protocol, Src Port: 48855 (48855), Dst Port: 111 (111), Seq: 1, Ack: 1, Len: 0
 >     Source port: 48855 (48855)
 >     Destination port: 111 (111)
 >     Sequence number: 1    (relative sequence number)
 >     Acknowledgement number: 1    (relative ack number)
 >     Header length: 32 bytes
 >     Flags: 0x0010 (ACK)
 >         0... .... = Congestion Window Reduced (CWR): Not set
 >         .0.. .... = ECN-Echo: Not set
 >         ..0. .... = Urgent: Not set
 >         ...1 .... = Acknowledgment: Set
 >         .... 0... = Push: Not set
 >         .... .0.. = Reset: Not set
 >         .... ..0. = Syn: Not set
 >         .... ...0 = Fin: Not set
 >     Window size: 5888 (scaled)
 >     Checksum: 0x1c3f [correct]
 >     Options: (12 bytes)
 >         NOP
 >         NOP
 >         Time stamp: tsval 1730873, tsecr 3830204155
 >
 > Frame 4 (110 bytes on wire, 110 bytes captured)
 >     Arrival Time: Aug 11, 2006 16:14:29.165272000
 >     Time delta from previous packet: 0.001499000 seconds
 >     Time since reference or first frame: 0.001748000 seconds
 >     Frame Number: 4
 >     Packet Length: 110 bytes
 >     Capture Length: 110 bytes
 >     Frame is marked: False
 >     Protocols in frame: eth:ip:tcp:rpc
 > Ethernet II, Src: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41), Dst: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
 >     Destination: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
 >         Address: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
 >         .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
 >         .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
 >     Source: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
 >         Address: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
 >         .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
 >         .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
 >     Type: IP (0x0800)
 > Internet Protocol, Src: 10.0.0.13 (10.0.0.13), Dst: 10.0.0.11 (10.0.0.11)
 >     Version: 4
 >     Header length: 20 bytes
 >     Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
 >         0000 00.. = Differentiated Services Codepoint: Default (0x00)
 >         .... ..0. = ECN-Capable Transport (ECT): 0
 >         .... ...0 = ECN-CE: 0
 >     Total Length: 96
 >     Identification: 0x40f7 (16631)
 >     Flags: 0x04 (Don't Fragment)
 >         0... = Reserved bit: Not set
 >         .1.. = Don't fragment: Set
 >         ..0. = More fragments: Not set
 >     Fragment offset: 0
 >     Time to live: 64
 >     Protocol: TCP (0x06)
 >     Header checksum: 0x9d04 [correct]
 >         Good: True
 >         Bad : False
 >     Source: 10.0.0.13 (10.0.0.13)
 >     Destination: 10.0.0.11 (10.0.0.11)
 > Transmission Control Protocol, Src Port: 48855 (48855), Dst Port: 111 (111), Seq: 1, Ack: 1, Len: 44
 >     Source port: 48855 (48855)
 >     Destination port: 111 (111)
 >     Sequence number: 1    (relative sequence number)
 >     Next sequence number: 45    (relative sequence number)
 >     Acknowledgement number: 1    (relative ack number)
 >     Header length: 32 bytes
 >     Flags: 0x0018 (PSH, ACK)
 >         0... .... = Congestion Window Reduced (CWR): Not set
 >         .0.. .... = ECN-Echo: Not set
 >         ..0. .... = Urgent: Not set
 >         ...1 .... = Acknowledgment: Set
 >         .... 1... = Push: Set
 >         .... .0.. = Reset: Not set
 >         .... ..0. = Syn: Not set
 >         .... ...0 = Fin: Not set
 >     Window size: 5888 (scaled)
 >     Checksum: 0x8a9a [correct]
 >     Options: (12 bytes)
 >         NOP
 >         NOP
 >         Time stamp: tsval 1730874, tsecr 3830204155
 > Remote Procedure Call, Type:Call XID:0x7a241079
 >     Fragment header: Last fragment, 40 bytes
 >         1... .... .... .... .... .... .... .... = Last Fragment: Yes
 >         .000 0000 0000 0000 0000 0000 0010 1000 = Fragment Length: 40
 >     XID: 0x7a241079 (2049183865)
 >     Message Type: Call (0)
 >     RPC Version: 2
 >     Program: Portmap (100000)
 >     Program Version: 2
 >     Procedure: DUMP (4)
 >     Credentials
 >         Flavor: AUTH_NULL (0)
 >         Length: 0
 >     Verifier
 >         Flavor: AUTH_NULL (0)
 >         Length: 0
 > Portmap
 >     Program Version: 2
 >     V2 Procedure: DUMP (4)
 >
 > Frame 5 (66 bytes on wire, 66 bytes captured)
 >     Arrival Time: Aug 11, 2006 16:14:29.364528000
 >     Time delta from previous packet: 0.199256000 seconds
 >     Time since reference or first frame: 0.201004000 seconds
 >     Frame Number: 5
 >     Packet Length: 66 bytes
 >     Capture Length: 66 bytes
 >     Frame is marked: False
 >     Protocols in frame: eth:ip:tcp
 > Ethernet II, Src: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8), Dst: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
 >     Destination: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
 >         Address: 00:0e:0c:71:3a:41 (00:0e:0c:71:3a:41)
 >         .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
 >         .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
 >     Source: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
 >         Address: 00:0e:0c:84:ed:f8 (00:0e:0c:84:ed:f8)
 >         .... ...0 .... .... .... .... = Multicast: This is a UNICAST frame
 >         .... ..0. .... .... .... .... = Locally Administrated Address: This is a FACTORY DEFAULT address
 >     Type: IP (0x0800)
 > Internet Protocol, Src: 10.0.0.11 (10.0.0.11), Dst: 10.0.0.13 (10.0.0.13)
 >     Version: 4
 >     Header length: 20 bytes
 >     Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
 >         0000 00.. = Differentiated Services Codepoint: Default (0x00)
 >         .... ..0. = ECN-Capable Transport (ECT): 0
 >         .... ...0 = ECN-CE: 0
 >     Total Length: 52
 >     Identification: 0xce4c (52812)
 >     Flags: 0x00
 >         0... = Reserved bit: Not set
 >         .0.. = Don't fragment: Not set
 >         ..0. = More fragments: Not set
 >     Fragment offset: 0
 >     Time to live: 64
 >     Protocol: TCP (0x06)
 >     Header checksum: 0x4fdb [correct]
 >         Good: True
 >         Bad : False
 >     Source: 10.0.0.11 (10.0.0.11)
 >     Destination: 10.0.0.13 (10.0.0.13)
 > Transmission Control Protocol, Src Port: 111 (111), Dst Port: 48855 (48855), Seq: 313, Ack: 45, Len: 0
 >     Source port: 111 (111)
 >     Destination port: 48855 (48855)
 >     Sequence number: 313    (relative sequence number)
 >     Acknowledgement number: 45    (relative ack number)
 >     Header length: 32 bytes
 >     Flags: 0x0010 (ACK)
 >         0... .... = Congestion Window Reduced (CWR): Not set
 >         .0.. .... = ECN-Echo: Not set
 >         ..0. .... = Urgent: Not set
 >         ...1 .... = Acknowledgment: Set
 >         .... 0... = Push: Not set
 >         .... .0.. = Reset: Not set
 >         .... ..0. = Syn: Not set
 >         .... ...0 = Fin: Not set
 >     Window size: 17376
 >     Checksum: 0xd727 [correct]
 >     Options: (12 bytes)
 >         NOP
 >         NOP
 >         Time stamp: tsval 3830204155, tsecr 1730874
 >     SEQ/ACK analysis
 >         TCP Analysis Flags
 >             A segment before this frame was lost
 >
 > >Fix:
 >
 > A workaround is to disable window scaling in clients that experience
 > this problem when talking to OpenBSD servers, or to fiddle with the pf
 > rules on the server.
 >
 >
 > >Release-Note:
 > >Audit-Trail:
 > >Unformatted: