Quantcast

tcp.ts_recent set to 0 after receiving retransmission

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

tcp.ts_recent set to 0 after receiving retransmission

Lauri Tirkkonen-2
>Synopsis: tcp.ts_recent set to 0 after receiving retransmission
>Category: kernel
>Environment:
        System      : OpenBSD 6.1
        Details     : OpenBSD 6.1-current (GENERIC.MP) #67: Mon Apr 17 15:22:46 MDT 2017
                         [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

        Architecture: OpenBSD.amd64
        Machine     : amd64
>Description:

On a congested TCP path between OpenBSD (receiver of data on the
connection) and illumos/OmniOS (sender), lost fast retransmission
segments cause the retransmission timer of the sender to keep growing
until it reaches maximum without ever going down. That's an illumos bug,
but the trigger is the ACK segments sent by OpenBSD always have TSecr=0
after some retransmissions are received; partial packet capture on the
receiver below:

     7183   0.875078     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
     7184   0.876087       sender → receiver     1090 [TCP Previous segment not captured] 9879 → 34480 [ACK] Seq=4512769 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
     7185   0.876089       sender → receiver     1090 9879 → 34480 [ACK] Seq=4513793 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
     7186   0.876090       sender → receiver     1090 9879 → 34480 [ACK] Seq=4514817 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
     7187   0.876090       sender → receiver     1090 9879 → 34480 [ACK] Seq=4515841 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
     7188   0.876091       sender → receiver     1090 9879 → 34480 [ACK] Seq=4516865 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
     7189   0.876093       sender → receiver     1090 9879 → 34480 [ACK] Seq=4517889 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
     7190   0.876102     receiver → sender       66 [TCP Dup ACK 7183#1] 34480 → 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
     7191   0.876107     receiver → sender       66 [TCP Dup ACK 7183#2] 34480 → 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
     7192   0.876110     receiver → sender       66 [TCP Dup ACK 7183#3] 34480 → 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
     7193   0.876114     receiver → sender       66 [TCP Dup ACK 7183#4] 34480 → 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
     7194   0.876117     receiver → sender       66 [TCP Dup ACK 7183#5] 34480 → 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
     7195   0.876120     receiver → sender       66 [TCP Dup ACK 7183#6] 34480 → 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
     7196   0.876198       sender → receiver     1090 9879 → 34480 [ACK] Seq=4518913 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
     7197   0.876202     receiver → sender       66 [TCP Dup ACK 7183#7] 34480 → 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
     7198   0.876326       sender → receiver     1090 9879 → 34480 [ACK] Seq=4519937 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
     7199   0.876327       sender → receiver     1090 9879 → 34480 [ACK] Seq=4520961 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
     7200   0.876331     receiver → sender       66 [TCP Dup ACK 7183#8] 34480 → 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
     7201   0.876335     receiver → sender       66 [TCP Dup ACK 7183#9] 34480 → 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
     7202   0.877872       sender → receiver     1090 [TCP Previous segment not captured] 9879 → 34480 [ACK] Seq=4529153 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
     7203   0.877876     receiver → sender       66 [TCP Dup ACK 7183#10] 34480 → 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
     7204   0.878002       sender → receiver     1090 9879 → 34480 [ACK] Seq=4530177 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
     7205   0.878003       sender → receiver     1090 [TCP Previous segment not captured] 9879 → 34480 [ACK] Seq=4536321 Ack=1 Win=128872 Len=1024 TSval=2677655334 TSecr=1179635542
     7206   0.878007     receiver → sender       66 [TCP Dup ACK 7183#11] 34480 → 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
     7207   0.878011     receiver → sender       66 [TCP Dup ACK 7183#12] 34480 → 9879 [ACK] Seq=1 Ack=4510721 Win=33280 Len=0 TSval=1179635542 TSecr=2677655333
     7208   1.296048       sender → receiver     1514 [TCP Retransmission] 9879 → 34480 [ACK] Seq=4510721 Ack=1 Win=128872 Len=1448 TSval=2677655376 TSecr=1179635542
     7209   1.296064     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4512169 Win=31808 Len=0 TSval=1179635543 TSecr=2677655376
     7210   1.299226       sender → receiver     1514 [TCP Retransmission] 9879 → 34480 [ACK] Seq=4512169 Ack=1 Win=128872 Len=1448 TSval=2677655376 TSecr=1179635543
     7211   1.299228       sender → receiver     1514 [TCP Retransmission] 9879 → 34480 [ACK] Seq=4513617 Ack=1 Win=128872 Len=1448 TSval=2677655376 TSecr=1179635543
     7212   1.299275     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4521985 Win=23488 Len=0 TSval=1179635543 TSecr=2677655376
     7213   1.299282     receiver → sender       66 [TCP Dup ACK 7212#1] 34480 → 9879 [ACK] Seq=1 Ack=4521985 Win=23488 Len=0 TSval=1179635543 TSecr=0
     7214   1.299315     receiver → sender       66 [TCP Window Update] 34480 → 9879 [ACK] Seq=1 Ack=4521985 Win=33280 Len=0 TSval=1179635543 TSecr=0
     7215   1.302160       sender → receiver     1514 [TCP Retransmission] 9879 → 34480 [ACK] Seq=4521985 Ack=1 Win=128872 Len=1448 TSval=2677655376 TSecr=1179635543
     7216   1.302205     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4523433 Win=31808 Len=0 TSval=1179635543 TSecr=0
     7217   1.302365       sender → receiver     1514 [TCP Retransmission] 9879 → 34480 [ACK] Seq=4523433 Ack=1 Win=128872 Len=1448 TSval=2677655376 TSecr=1179635543
     7218   1.302378       sender → receiver     1514 [TCP Retransmission] 9879 → 34480 [ACK] Seq=4524881 Ack=1 Win=128872 Len=1448 TSval=2677655376 TSecr=1179635543
     7219   1.302399     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4524881 Win=31808 Len=0 TSval=1179635543 TSecr=0
     7220   1.302413     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4526329 Win=30400 Len=0 TSval=1179635543 TSecr=0
     7221   1.302423     receiver → sender       66 [TCP Window Update] 34480 → 9879 [ACK] Seq=1 Ack=4526329 Win=33280 Len=0 TSval=1179635543 TSecr=0
     7222   1.305086       sender → receiver     1514 [TCP Retransmission] 9879 → 34480 [ACK] Seq=4526329 Ack=1 Win=128872 Len=1448 TSval=2677655376 TSecr=1179635543
     7223   1.305095     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4527777 Win=31808 Len=0 TSval=1179635543 TSecr=0
     7224   1.305220       sender → receiver     1514 [TCP Retransmission] 9879 → 34480 [ACK] Seq=4527777 Ack=1 Win=128872 Len=1448 TSval=2677655376 TSecr=1179635543
     7225   1.305228     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4531201 Win=29824 Len=0 TSval=1179635543 TSecr=0
     7226   1.305248     receiver → sender       66 [TCP Window Update] 34480 → 9879 [ACK] Seq=1 Ack=4531201 Win=33280 Len=0 TSval=1179635543 TSecr=0
     7227   1.305480       sender → receiver     1514 [TCP Spurious Retransmission] 9879 → 34480 [ACK] Seq=4529225 Ack=1 Win=128872 Len=1448 TSval=2677655376 TSecr=1179635543
     7228   1.305492     receiver → sender       66 [TCP Dup ACK 7225#1] 34480 → 9879 [ACK] Seq=1 Ack=4531201 Win=33280 Len=0 TSval=1179635543 TSecr=0
     7229   1.305605       sender → receiver     1514 [TCP Retransmission] 9879 → 34480 [ACK] Seq=4530673 Ack=1 Win=128872 Len=1448 TSval=2677655376 TSecr=1179635543
     7230   1.305606       sender → receiver     1514 [TCP Retransmission] 9879 → 34480 [ACK] Seq=4532121 Ack=1 Win=128872 Len=1448 TSval=2677655376 TSecr=1179635543
     7231   1.305607       sender → receiver     1514 [TCP Retransmission] 9879 → 34480 [ACK] Seq=4533569 Ack=1 Win=128872 Len=1448 TSval=2677655376 TSecr=1179635543
     7232   1.305616     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4532121 Win=32384 Len=0 TSval=1179635543 TSecr=0
     7233   1.305620     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4533569 Win=30912 Len=0 TSval=1179635543 TSecr=0
     7234   1.305624     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4535017 Win=29440 Len=0 TSval=1179635543 TSecr=0
     7235   1.305638     receiver → sender       66 [TCP Window Update] 34480 → 9879 [ACK] Seq=1 Ack=4535017 Win=33280 Len=0 TSval=1179635543 TSecr=0
     7236   1.308405       sender → receiver     1514 [TCP Retransmission] 9879 → 34480 [ACK] Seq=4535017 Ack=1 Win=128872 Len=1448 TSval=2677655377 TSecr=1179635543
     7237   1.308407       sender → receiver     1514 [TCP Retransmission] 9879 → 34480 [ACK] Seq=4536465 Ack=1 Win=128872 Len=1448 TSval=2677655377 TSecr=1179635543
     7238   1.308408       sender → receiver     1514 9879 → 34480 [ACK] Seq=4537913 Ack=1 Win=128872 Len=1448 TSval=2677655377 TSecr=1179635543
     7239   1.308409       sender → receiver     1514 9879 → 34480 [ACK] Seq=4539361 Ack=1 Win=128872 Len=1448 TSval=2677655377 TSecr=1179635543
     7240   1.308416     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4537345 Win=30976 Len=0 TSval=1179635543 TSecr=0
     7241   1.308421     receiver → sender       66 [TCP ACKed unseen segment] 34480 → 9879 [ACK] Seq=1 Ack=4539361 Win=28928 Len=0 TSval=1179635543 TSecr=0
     7242   1.308432     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4540809 Win=33280 Len=0 TSval=1179635543 TSecr=0
     7243   1.308665       sender → receiver     1514 9879 → 34480 [ACK] Seq=4540809 Ack=1 Win=128872 Len=1448 TSval=2677655377 TSecr=1179635543
     7244   1.308666       sender → receiver     594 9879 → 34480 [ACK] Seq=4542257 Ack=1 Win=128872 Len=528 TSval=2677655377 TSecr=1179635543
     7245   1.308667       sender → receiver     770 9879 → 34480 [ACK] Seq=4542785 Ack=1 Win=128872 Len=704 TSval=2677655377 TSecr=1179635543
     7246   1.308668       sender → receiver     1090 9879 → 34480 [ACK] Seq=4543489 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
     7247   1.308670       sender → receiver     1090 9879 → 34480 [ACK] Seq=4544513 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
     7248   1.308676     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4542785 Win=31296 Len=0 TSval=1179635543 TSecr=0
     7249   1.308680     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4544513 Win=29568 Len=0 TSval=1179635543 TSecr=0
     7250   1.308689     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4545537 Win=33280 Len=0 TSval=1179635543 TSecr=0
     7251   1.308790       sender → receiver     1090 9879 → 34480 [ACK] Seq=4545537 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
     7252   1.308792       sender → receiver     1090 9879 → 34480 [ACK] Seq=4546561 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
     7253   1.308793       sender → receiver     1090 9879 → 34480 [ACK] Seq=4547585 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
     7254   1.308794       sender → receiver     1090 9879 → 34480 [ACK] Seq=4548609 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
     7255   1.308800     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4547585 Win=31232 Len=0 TSval=1179635543 TSecr=0
     7256   1.308805     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4549633 Win=29184 Len=0 TSval=1179635543 TSecr=0
     7257   1.308812     receiver → sender       66 [TCP Window Update] 34480 → 9879 [ACK] Seq=1 Ack=4549633 Win=33280 Len=0 TSval=1179635543 TSecr=0
     7258   1.311403       sender → receiver     1090 9879 → 34480 [ACK] Seq=4549633 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
     7259   1.311404       sender → receiver     1090 9879 → 34480 [ACK] Seq=4550657 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
     7260   1.311405       sender → receiver     1090 9879 → 34480 [ACK] Seq=4551681 Ack=1 Win=128872 Len=1024 TSval=2677655377 TSecr=1179635543
     7261   1.311411     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4551681 Win=31232 Len=0 TSval=1179635543 TSecr=0
     7262   1.311420     receiver → sender       66 34480 → 9879 [ACK] Seq=1 Ack=4552705 Win=33280 Len=0 TSval=1179635543 TSecr=0

The first segment with TSecr=0 is 7216, but even after loss recovery all
further ACKs sent by the receiver have TSecr=0. I could not reproduce
TSecr staying at 0 on a connection between Linux and OpenBSD (though I
could see TSecr=0 ACKs there intermittently), so I'm not sure what
exactly it is that keeps ts_recent at zero (but the diff below makes it
not be set to zero at all).

>How-To-Repeat:

- tcp sender on illumos (OmniOS): nc -l 9879 </dev/null
- for more reliable repro, "disable" fast retransmissions on the sender:
      ndd -set /dev/tcp tcp_dupack_fast_retransmit 1000
- introduce some packet loss on the connection path
- tcp receiver on OpenBSD: nc omnios 9879 | pv >/dev/null
- observe pauses of 0B/s transferred that grow in length during
  connection lifetime

>Fix:

Diff below. It removes the TSecr=0 ACKs from Linux->OpenBSD connections
as well as fixes the OmniOS->OpenBSD stalls.

https://tools.ietf.org/html/rfc7323#section-4.3 states in the algorithm
description (2) that if SEG.TSval is not copied to TS.Recent, then it
should be ignored (ie. not set TS.Recent to 0); in fact RFC 1323 says
the same (but the algorithm changed a little). I don't have a copy of
the book referenced in the comment, but AFAICT RFC7323's algorithm makes
the mentioned fix unnecessary; see (d) in
https://tools.ietf.org/html/rfc7323#appendix-H

Index: tcp_input.c
===================================================================
RCS file: /cvs/src/sys/netinet/tcp_input.c,v
retrieving revision 1.341
diff -u -p -r1.341 tcp_input.c
--- tcp_input.c 19 Apr 2017 15:21:54 -0000 1.341
+++ tcp_input.c 20 Apr 2017 13:35:10 -0000
@@ -1374,15 +1374,10 @@ trimthenstep6:
  /*
  * If last ACK falls within this segment's sequence numbers,
  * record its timestamp if it's more recent.
- * Cf fix from Braden, see Stevens p. 870
  */
  if (opti.ts_present && TSTMP_GEQ(opti.ts_val, tp->ts_recent) &&
     SEQ_LEQ(th->th_seq, tp->last_ack_sent)) {
- if (SEQ_LEQ(tp->last_ack_sent, th->th_seq + tlen +
-    ((tiflags & (TH_SYN|TH_FIN)) != 0)))
- tp->ts_recent = opti.ts_val;
- else
- tp->ts_recent = 0;
+ tp->ts_recent = opti.ts_val;
  tp->ts_recent_age = tcp_now;
  }

--
Lauri Tirkkonen | lotheac @ IRCnet

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: tcp.ts_recent set to 0 after receiving retransmission

Lauri Tirkkonen-2
On Thu, Apr 20 2017 16:47:12 +0300, Lauri Tirkkonen wrote:
> >Synopsis: tcp.ts_recent set to 0 after receiving retransmission

No takers?

--
Lauri Tirkkonen | lotheac @ IRCnet

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: tcp.ts_recent set to 0 after receiving retransmission

Todd C. Miller
In reply to this post by Lauri Tirkkonen-2
Fixed, thanks.  This is actually a case of the code being changed
to match the comment when the comment should have been changed to
match the code :-)

 - todd

Loading...