pfsync too slow ?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

pfsync too slow ?

Loïc Blot-2
Hello,

today i was configuring pfsync on a dual routers (BGP on WAN and CARP on
LAN). Before i run in a stateless mode and it works like a charm.

Now with pfsync state are synchronized but late, then client must launch
2 or 3 TCP connections and when it works it's very slow.
I also have tried defer mode and increasing maxupd but no changes
appear. I also add Is there anything more to do ?

Because of BGP incoming request, and CARP regular configuration, traffic
follows this path:

WAN > Router 2 (BGP related) > HTTP server > Router 1 (CARP related) >
WAN

It seems all WAN incoming requests are blocked on Router 1 when HTTP
server SYN/ACK the request, but they are allowed by Router 2 and pfsync
is working very well.

Here is a pflog on this router (it runs on a block in log all default
rule)

21:16:28.564254 194.XX.XX.196.80 > 92.151.191.92.49189: S
1348056791:1348056791(0) ack 2684884151 win 65535 <mss 1452,nop,wscale
6,sackOK,eol> (DF)
21:16:28.569211 194.XX.XX.196.80 > 92.151.191.92.49190: S
274610014:274610014(0) ack 3399288798 win 65535 <mss 1452,nop,wscale
6,sackOK,eol> (DF)
21:16:28.417129 194.XX.XX.197.443 > 92.141.20.30.52927: S
3247839828:3247839828(0) ack 2174711713 win 65535 <mss 1452,nop,wscale
6,sackOK,timestamp 2749057892 407092137> (DF)
21:16:28.417784 194.XX.XX.197.443 > 92.141.20.30.52928: S
3719844564:3719844564(0) ack 524653966 win 65535 <mss 1452,nop,wscale
6,sackOK,timestamp 2971980508 407092137> (DF)
21:16:28.431525 194.XX.XX.196.443 > 46.193.1.103.61043: S
2792740841:2792740841(0) ack 3070150005 win 65535 <mss 1460,nop,wscale
6,sackOK,eol> (DF)

Here is an extract of my PF rules:
pass in quick inet proto tcp to <http_servers_v4> port { http https }
pass in quick inet6 proto tcp to <http_servers_v6> port { http https }

In stateless configuration this works, here is the working
configuration:
pass in quick inet proto tcp to <http_servers_v4> port { http https } no
state
pass in quick inet proto tcp from <http_servers_v4> port { http https }
no state
pass in quick inet6 proto tcp to <http_servers_v6> port { http https }
no state
pass in quick inet6 proto tcp from <http_servers_v6> port { http https }
no state

Here is a pfsync configuration example:
up syncdev vlanXX5 syncpeer 10.XX.X.129

The latency between the two host is very light, because they are on the
same switch, with a dedicated VLAN

64 bytes from 10.XX.XX.130: icmp_seq=2 ttl=255 time=0.146 ms

For more informations, here is a little extract when i do a tcpdump on
vlanXX5 (each host see the pfsync updated)

21:37:25.031271 10.117.1.129: PFSYNCv6 len 108
    act UPD ST COMP count 1
    ...
 (DF)
21:37:25.036625 10.117.1.130: PFSYNCv6 len 192
    act UPD ST COMP count 2
    ...
 (DF) [tos 0x10]
21:37:25.036753 10.117.1.129: PFSYNCv6 len 108
    act UPD ST COMP count 1
    ...
 (DF)
21:37:25.041298 10.117.1.129: PFSYNCv6 len 108
    act UPD ST COMP count 1
    ...
 (DF)
21:37:25.046578 10.117.1.130: PFSYNCv6 len 192
    act UPD ST COMP count 2
    ...
 (DF) [tos 0x10]
21:37:25.046706 10.117.1.129: PFSYNCv6 len 108
    act UPD ST COMP count 1
    ...
 (DF)
21:37:25.051269 10.117.1.129: PFSYNCv6 len 108
    act UPD ST COMP count 1
    ...
 (DF)
21:37:25.056562 10.117.1.130: PFSYNCv6 len 192
    act UPD ST COMP count 2
    ...

Have you got ideas ?
Thanks in advance

--
Best regards,
Loïc BLOT,
UNIX systems, security and network engineer
http://www.unix-experience.fr

[demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: pfsync too slow ?

Loïc Blot-2
Hmmm
I solved it by removing 'in' from pass in quick <...>

But my PF are configured with the first default rule: pass out all and
there isn't any block out rule... Is this a normal situation ?
On another router (which also do NAT), i use only pass in and pass out
for NAT, and all PF is stateful.
Is there anything i miss ?
--
Best regards,
Loïc BLOT,
UNIX systems, security and network engineer
http://www.unix-experience.fr



Le lundi 07 octobre 2013 à 21:51 +0200, Loïc BLOT a écrit :

> Hello,
>
> today i was configuring pfsync on a dual routers (BGP on WAN and CARP on
> LAN). Before i run in a stateless mode and it works like a charm.
>
> Now with pfsync state are synchronized but late, then client must launch
> 2 or 3 TCP connections and when it works it's very slow.
> I also have tried defer mode and increasing maxupd but no changes
> appear. I also add Is there anything more to do ?
>
> Because of BGP incoming request, and CARP regular configuration, traffic
> follows this path:
>
> WAN > Router 2 (BGP related) > HTTP server > Router 1 (CARP related) >
> WAN
>
> It seems all WAN incoming requests are blocked on Router 1 when HTTP
> server SYN/ACK the request, but they are allowed by Router 2 and pfsync
> is working very well.
>
> Here is a pflog on this router (it runs on a block in log all default
> rule)
>
> 21:16:28.564254 194.XX.XX.196.80 > 92.151.191.92.49189: S
> 1348056791:1348056791(0) ack 2684884151 win 65535 <mss 1452,nop,wscale
> 6,sackOK,eol> (DF)
> 21:16:28.569211 194.XX.XX.196.80 > 92.151.191.92.49190: S
> 274610014:274610014(0) ack 3399288798 win 65535 <mss 1452,nop,wscale
> 6,sackOK,eol> (DF)
> 21:16:28.417129 194.XX.XX.197.443 > 92.141.20.30.52927: S
> 3247839828:3247839828(0) ack 2174711713 win 65535 <mss 1452,nop,wscale
> 6,sackOK,timestamp 2749057892 407092137> (DF)
> 21:16:28.417784 194.XX.XX.197.443 > 92.141.20.30.52928: S
> 3719844564:3719844564(0) ack 524653966 win 65535 <mss 1452,nop,wscale
> 6,sackOK,timestamp 2971980508 407092137> (DF)
> 21:16:28.431525 194.XX.XX.196.443 > 46.193.1.103.61043: S
> 2792740841:2792740841(0) ack 3070150005 win 65535 <mss 1460,nop,wscale
> 6,sackOK,eol> (DF)
>
> Here is an extract of my PF rules:
> pass in quick inet proto tcp to <http_servers_v4> port { http https }
> pass in quick inet6 proto tcp to <http_servers_v6> port { http https }
>
> In stateless configuration this works, here is the working
> configuration:
> pass in quick inet proto tcp to <http_servers_v4> port { http https } no
> state
> pass in quick inet proto tcp from <http_servers_v4> port { http https }
> no state
> pass in quick inet6 proto tcp to <http_servers_v6> port { http https }
> no state
> pass in quick inet6 proto tcp from <http_servers_v6> port { http https }
> no state
>
> Here is a pfsync configuration example:
> up syncdev vlanXX5 syncpeer 10.XX.X.129
>
> The latency between the two host is very light, because they are on the
> same switch, with a dedicated VLAN
>
> 64 bytes from 10.XX.XX.130: icmp_seq=2 ttl=255 time=0.146 ms
>
> For more informations, here is a little extract when i do a tcpdump on
> vlanXX5 (each host see the pfsync updated)
>
> 21:37:25.031271 10.117.1.129: PFSYNCv6 len 108
>     act UPD ST COMP count 1
>     ...
>  (DF)
> 21:37:25.036625 10.117.1.130: PFSYNCv6 len 192
>     act UPD ST COMP count 2
>     ...
>  (DF) [tos 0x10]
> 21:37:25.036753 10.117.1.129: PFSYNCv6 len 108
>     act UPD ST COMP count 1
>     ...
>  (DF)
> 21:37:25.041298 10.117.1.129: PFSYNCv6 len 108
>     act UPD ST COMP count 1
>     ...
>  (DF)
> 21:37:25.046578 10.117.1.130: PFSYNCv6 len 192
>     act UPD ST COMP count 2
>     ...
>  (DF) [tos 0x10]
> 21:37:25.046706 10.117.1.129: PFSYNCv6 len 108
>     act UPD ST COMP count 1
>     ...
>  (DF)
> 21:37:25.051269 10.117.1.129: PFSYNCv6 len 108
>     act UPD ST COMP count 1
>     ...
>  (DF)
> 21:37:25.056562 10.117.1.130: PFSYNCv6 len 192
>     act UPD ST COMP count 2
>     ...
>
> Have you got ideas ?
> Thanks in advance
>
> --
> Best regards,
> Loc BLOT,
> UNIX systems, security and network engineer
> http://www.unix-experience.fr
>
> [demime 1.01d removed an attachment of type application/pgp-signature which
had a name of signature.asc]

[demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: pfsync too slow ?

Loïc Blot-2
Hmm, to precise the last message
after the the: pass out all
There is only:
block return out log quick on { $interco_polytech_v4 $interco_hec_v4 }
inet from <nonwanv4>
block return out log quick on { $interco_polytech_v6 $interco_hec_v6 }
inet6 from <nonwanv6>

and no other out related rule.
<nonwanv4> and <nonwanv6> contain my private IP adresses. And
$interco_xxx are my physical WAN interconnections then those IP
addresses mustn't be there.
In my first problem this is an incoming HTTP connection, then a public
IP to another public IP.


--
Best regards,
Loïc BLOT,
UNIX systems, security and network engineer
http://www.unix-experience.fr



Le lundi 07 octobre 2013 à 22:30 +0200, Loïc BLOT a écrit :

> Hmmm
> I solved it by removing 'in' from pass in quick <...>
>
> But my PF are configured with the first default rule: pass out all and
> there isn't any block out rule... Is this a normal situation ?
> On another router (which also do NAT), i use only pass in and pass out
> for NAT, and all PF is stateful.
> Is there anything i miss ?
> --
> Best regards,
> Loc BLOT,
> UNIX systems, security and network engineer
> http://www.unix-experience.fr
>
>
>
> Le lundi 07 octobre 2013  21:51 +0200, Loc BLOT a crit :
> > Hello,
> >
> > today i was configuring pfsync on a dual routers (BGP on WAN and CARP on
> > LAN). Before i run in a stateless mode and it works like a charm.
> >
> > Now with pfsync state are synchronized but late, then client must launch
> > 2 or 3 TCP connections and when it works it's very slow.
> > I also have tried defer mode and increasing maxupd but no changes
> > appear. I also add Is there anything more to do ?
> >
> > Because of BGP incoming request, and CARP regular configuration, traffic
> > follows this path:
> >
> > WAN > Router 2 (BGP related) > HTTP server > Router 1 (CARP related) >
> > WAN
> >
> > It seems all WAN incoming requests are blocked on Router 1 when HTTP
> > server SYN/ACK the request, but they are allowed by Router 2 and pfsync
> > is working very well.
> >
> > Here is a pflog on this router (it runs on a block in log all default
> > rule)
> >
> > 21:16:28.564254 194.XX.XX.196.80 > 92.151.191.92.49189: S
> > 1348056791:1348056791(0) ack 2684884151 win 65535 <mss 1452,nop,wscale
> > 6,sackOK,eol> (DF)
> > 21:16:28.569211 194.XX.XX.196.80 > 92.151.191.92.49190: S
> > 274610014:274610014(0) ack 3399288798 win 65535 <mss 1452,nop,wscale
> > 6,sackOK,eol> (DF)
> > 21:16:28.417129 194.XX.XX.197.443 > 92.141.20.30.52927: S
> > 3247839828:3247839828(0) ack 2174711713 win 65535 <mss 1452,nop,wscale
> > 6,sackOK,timestamp 2749057892 407092137> (DF)
> > 21:16:28.417784 194.XX.XX.197.443 > 92.141.20.30.52928: S
> > 3719844564:3719844564(0) ack 524653966 win 65535 <mss 1452,nop,wscale
> > 6,sackOK,timestamp 2971980508 407092137> (DF)
> > 21:16:28.431525 194.XX.XX.196.443 > 46.193.1.103.61043: S
> > 2792740841:2792740841(0) ack 3070150005 win 65535 <mss 1460,nop,wscale
> > 6,sackOK,eol> (DF)
> >
> > Here is an extract of my PF rules:
> > pass in quick inet proto tcp to <http_servers_v4> port { http https }
> > pass in quick inet6 proto tcp to <http_servers_v6> port { http https }
> >
> > In stateless configuration this works, here is the working
> > configuration:
> > pass in quick inet proto tcp to <http_servers_v4> port { http https } no
> > state
> > pass in quick inet proto tcp from <http_servers_v4> port { http https }
> > no state
> > pass in quick inet6 proto tcp to <http_servers_v6> port { http https }
> > no state
> > pass in quick inet6 proto tcp from <http_servers_v6> port { http https }
> > no state
> >
> > Here is a pfsync configuration example:
> > up syncdev vlanXX5 syncpeer 10.XX.X.129
> >
> > The latency between the two host is very light, because they are on the
> > same switch, with a dedicated VLAN
> >
> > 64 bytes from 10.XX.XX.130: icmp_seq=2 ttl=255 time=0.146 ms
> >
> > For more informations, here is a little extract when i do a tcpdump on
> > vlanXX5 (each host see the pfsync updated)
> >
> > 21:37:25.031271 10.117.1.129: PFSYNCv6 len 108
> >     act UPD ST COMP count 1
> >     ...
> >  (DF)
> > 21:37:25.036625 10.117.1.130: PFSYNCv6 len 192
> >     act UPD ST COMP count 2
> >     ...
> >  (DF) [tos 0x10]
> > 21:37:25.036753 10.117.1.129: PFSYNCv6 len 108
> >     act UPD ST COMP count 1
> >     ...
> >  (DF)
> > 21:37:25.041298 10.117.1.129: PFSYNCv6 len 108
> >     act UPD ST COMP count 1
> >     ...
> >  (DF)
> > 21:37:25.046578 10.117.1.130: PFSYNCv6 len 192
> >     act UPD ST COMP count 2
> >     ...
> >  (DF) [tos 0x10]
> > 21:37:25.046706 10.117.1.129: PFSYNCv6 len 108
> >     act UPD ST COMP count 1
> >     ...
> >  (DF)
> > 21:37:25.051269 10.117.1.129: PFSYNCv6 len 108
> >     act UPD ST COMP count 1
> >     ...
> >  (DF)
> > 21:37:25.056562 10.117.1.130: PFSYNCv6 len 192
> >     act UPD ST COMP count 2
> >     ...
> >
> > Have you got ideas ?
> > Thanks in advance
> >
> > --
> > Best regards,
> > Loc BLOT,
> > UNIX systems, security and network engineer
> > http://www.unix-experience.fr
> >
> > [demime 1.01d removed an attachment of type application/pgp-signature
which
> had a name of signature.asc]
>
> [demime 1.01d removed an attachment of type application/pgp-signature which
had a name of signature.asc]

[demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: pfsync too slow ?

Stuart Henderson
In reply to this post by Loïc Blot-2
On 2013-10-07, Loïc BLOT <[hidden email]> wrote:
> Now with pfsync state are synchronized but late, then client must launch
> 2 or 3 TCP connections and when it works it's very slow.
> I also have tried defer mode and increasing maxupd but no changes
> appear. I also add Is there anything more to do ?

defer helps, but if your typical scenario is to have a path split
between two routers (rather than just having this happen
occasionally) you may well be better off just using sloppy states.


On 2013-10-07, Lo\xc3\xafc BLOT <[hidden email]> wrote:
> Hmmm
> I solved it by removing 'in' from pass in quick <...>

test that longer connections work ok (or verify that you get wscale
information in all states associated with a connection, pfctl -ss -v
shows this)

> Here is a pfsync configuration example:
> up syncdev vlanXX5 syncpeer 10.XX.X.129
>
> The latency between the two host is very light, because they are on the
> same switch, with a dedicated VLAN

have you tried a direct cable? I find latency significantly lower
that way..

Reply | Threaded
Open this post in threaded view
|

Re: pfsync too slow ?

Loïc Blot-2
Hello Stuart,
thanks for your precisions.
I have tried to download a big matlab.deb on our repositories and it
works like a charm (3GB file). By removing 'in' i also notice a little
more reactivity on the network and the latency.
Now i'll wait tomorrow when my 500 users goes to work to see if router
works well with this configuration, and then i deploy this new type of
rule on all rules and firewalls.
For the cable, i can't. I haven't any more RJ45 slot available (4/4
ports used, LACP trunks).
Thanks for your tips. I the issue is coming when charge is increasing
i'll try it !

Good evening.

--
Best regards,
Loïc BLOT,
UNIX systems, security and network engineer
http://www.unix-experience.fr



Le lundi 07 octobre 2013 à 21:30 +0000, Stuart Henderson a écrit :

> On 2013-10-07, Loïc BLOT <[hidden email]> wrote:
> > Now with pfsync state are synchronized but late, then client must launch
> > 2 or 3 TCP connections and when it works it's very slow.
> > I also have tried defer mode and increasing maxupd but no changes
> > appear. I also add Is there anything more to do ?
>
> defer helps, but if your typical scenario is to have a path split
> between two routers (rather than just having this happen
> occasionally) you may well be better off just using sloppy states.
>
>
> On 2013-10-07, Lo\xc3\xafc BLOT <[hidden email]> wrote:
> > Hmmm
> > I solved it by removing 'in' from pass in quick <...>
>
> test that longer connections work ok (or verify that you get wscale
> information in all states associated with a connection, pfctl -ss -v
> shows this)
>
> > Here is a pfsync configuration example:
> > up syncdev vlanXX5 syncpeer 10.XX.X.129
> >
> > The latency between the two host is very light, because they are on the
> > same switch, with a dedicated VLAN
>
> have you tried a direct cable? I find latency significantly lower
> that way..

[demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]