http/https timeouts with OpenBSD based firewall

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

http/https timeouts with OpenBSD based firewall

Marcin-27
Hi,

I recently upgraded to 5.1, but I was able to reproduce the issue
described below with 4.8, 5.0 and 5.2 snapshot.

After the upgrade I discovered that workstations behind the OpenBSD
firewall experience occasional timeouts
while trying to access web servers running IIS 6.0 on Windows 2003
Server. The firewall itself is not affected.
The problem is rather intermittent and happens with 30%-50%
requests.The workstations are running Windows 7,
Windows XP and Linux.

I was also able to reproduce the issue by installing Windows 2003 R2
server in default configuration,
setting up extremely basic PF rules to redirect port 80 and accessing
the server from the Internet. I was unable to expose
this issue in LAN, which suggests it might happen only on links slower
than 100Mbit. However, it seems to
be hardware independent (although all tests were run on i386 arch) as
I achieve the same results on three
different machines in three different geographic locations connected
via independent ISPs.

This is how the problem can be exposed with curl:

#curl -vI http://www.startvbdotnet.com/
* About to connect() to www.startvbdotnet.com port 80 (#0)
*   Trying 64.79.160.13... connected
> HEAD / HTTP/1.1
> User-Agent: curl/7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: www.startvbdotnet.com
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection #0
curl: (56) Recv failure: Connection reset by peer

I uploaded the tcpdump from machine running curl here:
http://pastebin.com/AkqCeQwW

As far as I can tell, the Win 2008 and Win 2012 are not affected.
Also, the 4.5 seemed to be free from this problem.

Thanks in advance for any suggestions / workarounds!

--
Regards,
Marcin

Reply | Threaded
Open this post in threaded view
|

Re: http/https timeouts with OpenBSD based firewall

James Shupe-4
On 10/22/12 15:16, Marcin wrote:

> Hi,
>
> I recently upgraded to 5.1, but I was able to reproduce the issue
> described below with 4.8, 5.0 and 5.2 snapshot.
>
> After the upgrade I discovered that workstations behind the OpenBSD
> firewall experience occasional timeouts
> while trying to access web servers running IIS 6.0 on Windows 2003
> Server. The firewall itself is not affected.
> The problem is rather intermittent and happens with 30%-50%
> requests.The workstations are running Windows 7,
> Windows XP and Linux.
>
> I was also able to reproduce the issue by installing Windows 2003 R2
> server in default configuration,
> setting up extremely basic PF rules to redirect port 80 and accessing
> the server from the Internet. I was unable to expose
> this issue in LAN, which suggests it might happen only on links slower
> than 100Mbit. However, it seems to
> be hardware independent (although all tests were run on i386 arch) as
> I achieve the same results on three
> different machines in three different geographic locations connected
> via independent ISPs.
>
> This is how the problem can be exposed with curl:
>
> #curl -vI http://www.startvbdotnet.com/
> * About to connect() to www.startvbdotnet.com port 80 (#0)
> *   Trying 64.79.160.13... connected
>> HEAD / HTTP/1.1
>> User-Agent: curl/7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1
zlib/1.2.3.4 libidn/1.23 librtmp/2.3

>> Host: www.startvbdotnet.com
>> Accept: */*
>>
> * Recv failure: Connection reset by peer
> * Closing connection #0
> curl: (56) Recv failure: Connection reset by peer
>
> I uploaded the tcpdump from machine running curl here:
> http://pastebin.com/AkqCeQwW
>
> As far as I can tell, the Win 2008 and Win 2012 are not affected.
> Also, the 4.5 seemed to be free from this problem.
>
> Thanks in advance for any suggestions / workarounds!
>
> --
> Regards,
> Marcin
>
>
>

Please post the following things:

- output of `pfctl -si`
- your pf ruleset
- output of `vmstat -m`

--
James Shupe

[demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: http/https timeouts with OpenBSD based firewall

Joel Sing-3
In reply to this post by Marcin-27
On Tue, 23 Oct 2012, Marcin wrote:

> Hi,
>
> I recently upgraded to 5.1, but I was able to reproduce the issue
> described below with 4.8, 5.0 and 5.2 snapshot.
>
> After the upgrade I discovered that workstations behind the OpenBSD
> firewall experience occasional timeouts
> while trying to access web servers running IIS 6.0 on Windows 2003
> Server. The firewall itself is not affected.
> The problem is rather intermittent and happens with 30%-50%
> requests.The workstations are running Windows 7,
> Windows XP and Linux.
>
> I was also able to reproduce the issue by installing Windows 2003 R2
> server in default configuration,
> setting up extremely basic PF rules to redirect port 80 and accessing
> the server from the Internet. I was unable to expose
> this issue in LAN, which suggests it might happen only on links slower
> than 100Mbit. However, it seems to
> be hardware independent (although all tests were run on i386 arch) as
> I achieve the same results on three
> different machines in three different geographic locations connected
> via independent ISPs.
>
> This is how the problem can be exposed with curl:
>
> #curl -vI http://www.startvbdotnet.com/
> * About to connect() to www.startvbdotnet.com port 80 (#0)
> *   Trying 64.79.160.13... connected
>
> > HEAD / HTTP/1.1
> > User-Agent: curl/7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1
> > zlib/1.2.3.4 libidn/1.23 librtmp/2.3 Host: www.startvbdotnet.com
> > Accept: */*
>
> * Recv failure: Connection reset by peer
> * Closing connection #0
> curl: (56) Recv failure: Connection reset by peer
>
> I uploaded the tcpdump from machine running curl here:
> http://pastebin.com/AkqCeQwW
>
> As far as I can tell, the Win 2008 and Win 2012 are not affected.
> Also, the 4.5 seemed to be free from this problem.
>
> Thanks in advance for any suggestions / workarounds!

Unfortunately we have no idea what firewall rules you have configured, however
I'm going to take random guess and say that you're using a scrub rule
with 'reassemble tcp' - if this is the case you'll probably find that some
TCP connections to Windows-based servers will fail, since they often violate
RFC1323 by using a 0-value timestamp during the three-way handshake, then
increase it by some value between 0 and 2^31 on the first data packet. Note
the TS val fields in the first two packets from 64.79.160.13:

20:49:05.962419 IP (tos 0x0, ttl 117, id 20521, offset 0, flags [none], proto
TCP (6), length 64)
    64.79.160.13.80 > 192.168.1.20.51163: Flags [S.], cksum 0xd698 (correct),
seq 2659727337, ack 77418264, win 16384, options [mss 1440,nop,wscale
0,nop,nop,TS val 0 ecr 0,nop,nop,sackOK], length 0
20:49:06.146338 IP (tos 0x0, ttl 117, id 14230, offset 0, flags [none], proto
TCP (6), length 292)
    64.79.160.13.80 > 192.168.1.20.51163: Flags [P.], cksum 0xd584 (correct),
seq 1:241, ack 173, win 65363, options [nop,nop,TS val 2152972614 ecr
1629278667], length 240

Combined with the way that PF handles timestamp modulation (there is a subtle
and difficult to fix bug here), you can trigger the PAWS checks, which
results in packets being dropped. IIRC these drops will be logged if you run
with 'pfctl -x debug'. Removing 'reassemble tcp' should resolve the issue.
--

    "Reason is not automatic. Those who deny it cannot be conquered by it.
     Do not count on them. Leave them alone." -- Ayn Rand

Reply | Threaded
Open this post in threaded view
|

Re: http/https timeouts with OpenBSD based firewall

Marcin-27
On 23 October 2012 12:53, Joel Sing <[hidden email]> wrote:
> Unfortunately we have no idea what firewall rules you have configured, however
> I'm going to take random guess and say that you're using a scrub rule
> with 'reassemble tcp' - if this is the case you'll probably find that some
> TCP connections to Windows-based servers will fail, since they often violate
> RFC1323 by using a 0-value timestamp during the three-way handshake, then
> increase it by some value between 0 and 2^31 on the first data packet. Note
> the TS val fields in the first two packets from 64.79.160.13:

Joel, your guess was spot on. Removing "reassemble tcp" from scrub rule indeed
resolved the issue. Thank you very much!

Out of interest - it seems like Windows 2008/2012 behave much better
here as I did not
experience such problems with these systems. Is it the case or was I just lucky?

Thanks again!
--
Marcin

Reply | Threaded
Open this post in threaded view
|

Re: http/https timeouts with OpenBSD based firewall

Marcin-27
On 24 October 2012 22:07, Marcin <[hidden email]> wrote:
>
> Joel, your guess was spot on. Removing "reassemble tcp" from scrub rule indeed
> resolved the issue. Thank you very much!
>

Hi,
Just to help people running into similar issues - there is a registry
key on Windows 2003 which can mitigate the problem.
It needs to be created under:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
and is called Tcp1323Opts (DWORD).
Setting it to value 1 seems to solve the problem.

--
Marcin