PF sync doesn't not work very well

classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

PF sync doesn't not work very well

Loïc Blot-2
Hi all
I have a strange issue (or i haven't read pfsync correctly but i don't
think this is the problem :D)

I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site.

Those BGP routers are secure with strong PF in stateful mode, and the
stateful is working very well on each router. Because of my full mesh
BGP configuration, the outgoing layer 7 sessions can leave my network by
one router and responses can income by the other.

To resolve this issue, i have created a dedidated VLAN for the pfsync
traffic and attached pfsync to this VLAN.

Here is a sample output of ifconfig on my first router:

vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr a0:36:9f:10:4a:a6
        priority: 0
        vlan: 995 parent interface: trunk1
        groups: vlan
        status: active
        inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid
0x10
        inet 10.117.1.129 netmask 0xfffffff8 broadcast 10.117.1.135
pfsync0: flags=41<UP,RUNNING> mtu 1500
        priority: 0
        pfsync: syncdev: vlan995 maxupd: 255 defer: off
        groups: carp pfsync

And here on my second router:

vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr a0:36:9f:17:e2:1e
        priority: 0
        vlan: 995 parent interface: trunk1
        groups: vlan
        status: active
        inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid
0x10
        inet 10.117.1.130 netmask 0xfffffff8 broadcast 10.117.1.135
pfsync0: flags=41<UP,RUNNING> mtu 1500
        priority: 0
        pfsync: syncdev: vlan995 maxupd: 255 defer: off
        groups: carp pfsync

As you see in next tcpdump capture, there is some discussions between
the two routers:

# tcpdump -nni vlan995
tcpdump: listening on vlan995, link-type EN10MB
tcpdump: WARNING: compensating for unaligned libpcap packets
23:41:13.699617 10.117.1.130: PFSYNCv6 len 108
    act UPD ST COMP count 1
    ...
 (DF) [tos 0x10]
23:41:14.158500 10.117.1.129: PFSYNCv6 len 108
    act UPD ST COMP count 1
    ...
 (DF) [tos 0x10]
23:41:14.941396 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3
bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
hello=2/0 fwdelay=15/0 pvid=995
23:41:14.949617 10.117.1.130: PFSYNCv6 len 108
    act UPD ST COMP count 1
    ...
 (DF) [tos 0x10]
23:41:15.237655 10.117.1.129: PFSYNCv6 len 640
    act UPD ST COMP count 1
    ...
 (DF) [tos 0x10]
23:41:15.949617 10.117.1.130: PFSYNCv6 len 124
    act UPD ST COMP count 1
    ...
 (DF) [tos 0x10]
23:41:16.255230 10.117.1.129: PFSYNCv6 len 36
    act DEL ST COMP count 1
        id: 51d16a3500006c33 creatorid: a10bbd21
 (DF) [tos 0x10]
23:41:16.946454 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3
bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
hello=2/0 fwdelay=15/0 pvid=995
23:41:16.949619 10.117.1.130: PFSYNCv6 len 1116
    act UPD ST COMP count 13
    ...
 (DF) [tos 0x10]


The problem is simple, when i initiate a stateful connection from one
server, the return (by second router) is blocked by PF (i see the return
with pflog0)

To be precise here is an example (and tested path):

OBSD NTP -> OBSD router 1 -> WAN...ftp.fr.openbsd.org...WAN -> OBSD
router 2 || blocked

PF allow in/out routing traffic from this server but incoming from WAN
is blocked by default

Can you confirm to me that pfsync may add a state for outgoing tcp
connection in the second router when the first router add it ?
Have you got any idea on this issue ?

--
Best regards,
Loïc BLOT,
UNIX systems, security and network expert
http://www.unix-experience.fr

[demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Maxim Bourmistrov-5
pfsync(4) explains this:

"… The pfsync interface will attempt to collapse multiple state updates into
     a single packet where possible.  The maximum number of times a single
     state can be updated before a pfsync packet will be sent out is
     controlled by the maxupd parameter
…"


and

"… Where more than one firewall might actively handle packets, e.g. with
     certain ospfd(8), bgpd(8) or carp(4) configurations, it is beneficial to
     defer transmission of the initial packet of a connection.  The pfsync
     state insert message is sent immediately; the packet is queued until
     either this message is acknowledged by another system, or a timeout has
     expired.  This behaviour is enabled with the defer parameter to
     ifconfig(8).
…"


Eg. "defer: on", yours is "off".

//mxb


On 2 jul 2013, at 21:54, Loïc BLOT <[hidden email]> wrote:

> Hi all
> I have a strange issue (or i haven't read pfsync correctly but i don't
> think this is the problem :D)
>
> I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site.
>
> Those BGP routers are secure with strong PF in stateful mode, and the
> stateful is working very well on each router. Because of my full mesh
> BGP configuration, the outgoing layer 7 sessions can leave my network by
> one router and responses can income by the other.
>
> To resolve this issue, i have created a dedidated VLAN for the pfsync
> traffic and attached pfsync to this VLAN.
>
> Here is a sample output of ifconfig on my first router:
>
> vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>        lladdr a0:36:9f:10:4a:a6
>        priority: 0
>        vlan: 995 parent interface: trunk1
>        groups: vlan
>        status: active
>        inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid
> 0x10
>        inet 10.117.1.129 netmask 0xfffffff8 broadcast 10.117.1.135
> pfsync0: flags=41<UP,RUNNING> mtu 1500
>        priority: 0
>        pfsync: syncdev: vlan995 maxupd: 255 defer: off
>        groups: carp pfsync
>
> And here on my second router:
>
> vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>        lladdr a0:36:9f:17:e2:1e
>        priority: 0
>        vlan: 995 parent interface: trunk1
>        groups: vlan
>        status: active
>        inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid
> 0x10
>        inet 10.117.1.130 netmask 0xfffffff8 broadcast 10.117.1.135
> pfsync0: flags=41<UP,RUNNING> mtu 1500
>        priority: 0
>        pfsync: syncdev: vlan995 maxupd: 255 defer: off
>        groups: carp pfsync
>
> As you see in next tcpdump capture, there is some discussions between
> the two routers:
>
> # tcpdump -nni vlan995
> tcpdump: listening on vlan995, link-type EN10MB
> tcpdump: WARNING: compensating for unaligned libpcap packets
> 23:41:13.699617 10.117.1.130: PFSYNCv6 len 108
>    act UPD ST COMP count 1
>    ...
> (DF) [tos 0x10]
> 23:41:14.158500 10.117.1.129: PFSYNCv6 len 108
>    act UPD ST COMP count 1
>    ...
> (DF) [tos 0x10]
> 23:41:14.941396 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3
> bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
> hello=2/0 fwdelay=15/0 pvid=995
> 23:41:14.949617 10.117.1.130: PFSYNCv6 len 108
>    act UPD ST COMP count 1
>    ...
> (DF) [tos 0x10]
> 23:41:15.237655 10.117.1.129: PFSYNCv6 len 640
>    act UPD ST COMP count 1
>    ...
> (DF) [tos 0x10]
> 23:41:15.949617 10.117.1.130: PFSYNCv6 len 124
>    act UPD ST COMP count 1
>    ...
> (DF) [tos 0x10]
> 23:41:16.255230 10.117.1.129: PFSYNCv6 len 36
>    act DEL ST COMP count 1
>        id: 51d16a3500006c33 creatorid: a10bbd21
> (DF) [tos 0x10]
> 23:41:16.946454 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3
> bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
> hello=2/0 fwdelay=15/0 pvid=995
> 23:41:16.949619 10.117.1.130: PFSYNCv6 len 1116
>    act UPD ST COMP count 13
>    ...
> (DF) [tos 0x10]
>
>
> The problem is simple, when i initiate a stateful connection from one
> server, the return (by second router) is blocked by PF (i see the return
> with pflog0)
>
> To be precise here is an example (and tested path):
>
> OBSD NTP -> OBSD router 1 -> WAN...ftp.fr.openbsd.org...WAN -> OBSD
> router 2 || blocked
>
> PF allow in/out routing traffic from this server but incoming from WAN
> is blocked by default
>
> Can you confirm to me that pfsync may add a state for outgoing tcp
> connection in the second router when the first router add it ?
> Have you got any idea on this issue ?
>
> --
> Best regards,
> Loïc BLOT,
> UNIX systems, security and network expert
> http://www.unix-experience.fr
>
> [demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Loïc Blot-2
Hi,
Thanks for your reply. I wasn't careful about this section.
If i understand i must add defer option to my WAN iface (or i'm wrong i
must add it to my vlan995 iface ?) ?

I will test it this morning, and i return back to misc :)
--
Best regards,
Loïc BLOT,
UNIX systems, security and network expert
http://www.unix-experience.fr


Le mercredi 03 juillet 2013 à 02:02 +0200, mxb a écrit :
> pfsync(4) explains this:
>
> "… The pfsync interface will attempt to collapse multiple state updates
into

>      a single packet where possible.  The maximum number of times a single
>      state can be updated before a pfsync packet will be sent out is
>      controlled by the maxupd parameter
> …"
>
>
> and
>
> "… Where more than one firewall might actively handle packets, e.g. with
>      certain ospfd(8), bgpd(8) or carp(4) configurations, it is beneficial
to
>      defer transmission of the initial packet of a connection.  The pfsync
>      state insert message is sent immediately; the packet is queued until
>      either this message is acknowledged by another system, or a timeout
has

>      expired.  This behaviour is enabled with the defer parameter to
>      ifconfig(8).
> …"
>
>
> Eg. "defer: on", yours is "off".
>
> //mxb
>
>
> On 2 jul 2013, at 21:54, Loïc BLOT <[hidden email]> wrote:
>
> > Hi all
> > I have a strange issue (or i haven't read pfsync correctly but i don't
> > think this is the problem :D)
> >
> > I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site.
> >
> > Those BGP routers are secure with strong PF in stateful mode, and the
> > stateful is working very well on each router. Because of my full mesh
> > BGP configuration, the outgoing layer 7 sessions can leave my network by
> > one router and responses can income by the other.
> >
> > To resolve this issue, i have created a dedidated VLAN for the pfsync
> > traffic and attached pfsync to this VLAN.
> >
> > Here is a sample output of ifconfig on my first router:
> >
> > vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> >        lladdr a0:36:9f:10:4a:a6
> >        priority: 0
> >        vlan: 995 parent interface: trunk1
> >        groups: vlan
> >        status: active
> >        inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid
> > 0x10
> >        inet 10.117.1.129 netmask 0xfffffff8 broadcast 10.117.1.135
> > pfsync0: flags=41<UP,RUNNING> mtu 1500
> >        priority: 0
> >        pfsync: syncdev: vlan995 maxupd: 255 defer: off
> >        groups: carp pfsync
> >
> > And here on my second router:
> >
> > vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> >        lladdr a0:36:9f:17:e2:1e
> >        priority: 0
> >        vlan: 995 parent interface: trunk1
> >        groups: vlan
> >        status: active
> >        inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid
> > 0x10
> >        inet 10.117.1.130 netmask 0xfffffff8 broadcast 10.117.1.135
> > pfsync0: flags=41<UP,RUNNING> mtu 1500
> >        priority: 0
> >        pfsync: syncdev: vlan995 maxupd: 255 defer: off
> >        groups: carp pfsync
> >
> > As you see in next tcpdump capture, there is some discussions between
> > the two routers:
> >
> > # tcpdump -nni vlan995
> > tcpdump: listening on vlan995, link-type EN10MB
> > tcpdump: WARNING: compensating for unaligned libpcap packets
> > 23:41:13.699617 10.117.1.130: PFSYNCv6 len 108
> >    act UPD ST COMP count 1
> >    ...
> > (DF) [tos 0x10]
> > 23:41:14.158500 10.117.1.129: PFSYNCv6 len 108
> >    act UPD ST COMP count 1
> >    ...
> > (DF) [tos 0x10]
> > 23:41:14.941396 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3
> > bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
> > hello=2/0 fwdelay=15/0 pvid=995
> > 23:41:14.949617 10.117.1.130: PFSYNCv6 len 108
> >    act UPD ST COMP count 1
> >    ...
> > (DF) [tos 0x10]
> > 23:41:15.237655 10.117.1.129: PFSYNCv6 len 640
> >    act UPD ST COMP count 1
> >    ...
> > (DF) [tos 0x10]
> > 23:41:15.949617 10.117.1.130: PFSYNCv6 len 124
> >    act UPD ST COMP count 1
> >    ...
> > (DF) [tos 0x10]
> > 23:41:16.255230 10.117.1.129: PFSYNCv6 len 36
> >    act DEL ST COMP count 1
> >        id: 51d16a3500006c33 creatorid: a10bbd21
> > (DF) [tos 0x10]
> > 23:41:16.946454 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3
> > bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
> > hello=2/0 fwdelay=15/0 pvid=995
> > 23:41:16.949619 10.117.1.130: PFSYNCv6 len 1116
> >    act UPD ST COMP count 13
> >    ...
> > (DF) [tos 0x10]
> >
> >
> > The problem is simple, when i initiate a stateful connection from one
> > server, the return (by second router) is blocked by PF (i see the return
> > with pflog0)
> >
> > To be precise here is an example (and tested path):
> >
> > OBSD NTP -> OBSD router 1 -> WAN...ftp.fr.openbsd.org...WAN -> OBSD
> > router 2 || blocked
> >
> > PF allow in/out routing traffic from this server but incoming from WAN
> > is blocked by default
> >
> > Can you confirm to me that pfsync may add a state for outgoing tcp
> > connection in the second router when the first router add it ?
> > Have you got any idea on this issue ?
> >
> > --
> > Best regards,
> > Loïc BLOT,
> > UNIX systems, security and network expert
> > http://www.unix-experience.fr
> >
> > [demime 1.01d removed an attachment of type application/pgp-signature
which had a name of signature.asc]

[demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Loïc Blot-2
Okay, defer is now enabled on pfsync interface (sorry for my last idea,
i haven't the man on me :) ).
It seems the problem isn't resolved.
The transfer starts but blocked at random time.
--
Best regards,

Loïc BLOT, Engineering
UNIX Systems, Security and Networks
http://www.unix-experience.fr


Le mercredi 03 juillet 2013 à 08:12 +0200, Loïc BLOT a écrit :

> Hi,
> Thanks for your reply. I wasn't careful about this section.
> If i understand i must add defer option to my WAN iface (or i'm wrong i
> must add it to my vlan995 iface ?) ?
>
> I will test it this morning, and i return back to misc :)
> --
> Best regards,
> Loc BLOT,
> UNIX systems, security and network expert
> http://www.unix-experience.fr
>
>
> Le mercredi 03 juillet 2013  02:02 +0200, mxb a crit :
> > pfsync(4) explains this:
> >
> > " The pfsync interface will attempt to collapse multiple state updates
> into
> >      a single packet where possible.  The maximum number of times a single
> >      state can be updated before a pfsync packet will be sent out is
> >      controlled by the maxupd parameter
> > "
> >
> >
> > and
> >
> > " Where more than one firewall might actively handle packets, e.g. with
> >      certain ospfd(8), bgpd(8) or carp(4) configurations, it is beneficial
> to
> >      defer transmission of the initial packet of a connection.  The pfsync
> >      state insert message is sent immediately; the packet is queued until
> >      either this message is acknowledged by another system, or a timeout
> has
> >      expired.  This behaviour is enabled with the defer parameter to
> >      ifconfig(8).
> > "
> >
> >
> > Eg. "defer: on", yours is "off".
> >
> > //mxb
> >
> >
> > On 2 jul 2013, at 21:54, Loc BLOT <[hidden email]> wrote:
> >
> > > Hi all
> > > I have a strange issue (or i haven't read pfsync correctly but i don't
> > > think this is the problem :D)
> > >
> > > I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site.
> > >
> > > Those BGP routers are secure with strong PF in stateful mode, and the
> > > stateful is working very well on each router. Because of my full mesh
> > > BGP configuration, the outgoing layer 7 sessions can leave my network by
> > > one router and responses can income by the other.
> > >
> > > To resolve this issue, i have created a dedidated VLAN for the pfsync
> > > traffic and attached pfsync to this VLAN.
> > >
> > > Here is a sample output of ifconfig on my first router:
> > >
> > > vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> > >        lladdr a0:36:9f:10:4a:a6
> > >        priority: 0
> > >        vlan: 995 parent interface: trunk1
> > >        groups: vlan
> > >        status: active
> > >        inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid
> > > 0x10
> > >        inet 10.117.1.129 netmask 0xfffffff8 broadcast 10.117.1.135
> > > pfsync0: flags=41<UP,RUNNING> mtu 1500
> > >        priority: 0
> > >        pfsync: syncdev: vlan995 maxupd: 255 defer: off
> > >        groups: carp pfsync
> > >
> > > And here on my second router:
> > >
> > > vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> > >        lladdr a0:36:9f:17:e2:1e
> > >        priority: 0
> > >        vlan: 995 parent interface: trunk1
> > >        groups: vlan
> > >        status: active
> > >        inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid
> > > 0x10
> > >        inet 10.117.1.130 netmask 0xfffffff8 broadcast 10.117.1.135
> > > pfsync0: flags=41<UP,RUNNING> mtu 1500
> > >        priority: 0
> > >        pfsync: syncdev: vlan995 maxupd: 255 defer: off
> > >        groups: carp pfsync
> > >
> > > As you see in next tcpdump capture, there is some discussions between
> > > the two routers:
> > >
> > > # tcpdump -nni vlan995
> > > tcpdump: listening on vlan995, link-type EN10MB
> > > tcpdump: WARNING: compensating for unaligned libpcap packets
> > > 23:41:13.699617 10.117.1.130: PFSYNCv6 len 108
> > >    act UPD ST COMP count 1
> > >    ...
> > > (DF) [tos 0x10]
> > > 23:41:14.158500 10.117.1.129: PFSYNCv6 len 108
> > >    act UPD ST COMP count 1
> > >    ...
> > > (DF) [tos 0x10]
> > > 23:41:14.941396 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3
> > > bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
> > > hello=2/0 fwdelay=15/0 pvid=995
> > > 23:41:14.949617 10.117.1.130: PFSYNCv6 len 108
> > >    act UPD ST COMP count 1
> > >    ...
> > > (DF) [tos 0x10]
> > > 23:41:15.237655 10.117.1.129: PFSYNCv6 len 640
> > >    act UPD ST COMP count 1
> > >    ...
> > > (DF) [tos 0x10]
> > > 23:41:15.949617 10.117.1.130: PFSYNCv6 len 124
> > >    act UPD ST COMP count 1
> > >    ...
> > > (DF) [tos 0x10]
> > > 23:41:16.255230 10.117.1.129: PFSYNCv6 len 36
> > >    act DEL ST COMP count 1
> > >        id: 51d16a3500006c33 creatorid: a10bbd21
> > > (DF) [tos 0x10]
> > > 23:41:16.946454 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3
> > > bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
> > > hello=2/0 fwdelay=15/0 pvid=995
> > > 23:41:16.949619 10.117.1.130: PFSYNCv6 len 1116
> > >    act UPD ST COMP count 13
> > >    ...
> > > (DF) [tos 0x10]
> > >
> > >
> > > The problem is simple, when i initiate a stateful connection from one
> > > server, the return (by second router) is blocked by PF (i see the return
> > > with pflog0)
> > >
> > > To be precise here is an example (and tested path):
> > >
> > > OBSD NTP -> OBSD router 1 -> WAN...ftp.fr.openbsd.org...WAN -> OBSD
> > > router 2 || blocked
> > >
> > > PF allow in/out routing traffic from this server but incoming from WAN
> > > is blocked by default
> > >
> > > Can you confirm to me that pfsync may add a state for outgoing tcp
> > > connection in the second router when the first router add it ?
> > > Have you got any idea on this issue ?
> > >
> > > --
> > > Best regards,
> > > Loc BLOT,
> > > UNIX systems, security and network expert
> > > http://www.unix-experience.fr
> > >
> > > [demime 1.01d removed an attachment of type application/pgp-signature
> which had a name of signature.asc]
>
> [demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Maxim Bourmistrov-5
How does your CARP setup looks like. On both machines?
Can you send your ifconfig output?

What is your environment/setup for this 2-node CARP?
How interfaces (ext/int) are connected? What switches do you use?


On 3 jul 2013, at 10:23, Loïc Blot <[hidden email]> wrote:

> Okay, defer is now enabled on pfsync interface (sorry for my last idea,
> i haven't the man on me :) ).
> It seems the problem isn't resolved.
> The transfer starts but blocked at random time.
> --
> Best regards,
>
> Loïc BLOT, Engineering
> UNIX Systems, Security and Networks
> http://www.unix-experience.fr
>
>
> Le mercredi 03 juillet 2013 à 08:12 +0200, Loïc BLOT a écrit :
>> Hi,
>> Thanks for your reply. I wasn't careful about this section.
>> If i understand i must add defer option to my WAN iface (or i'm wrong i
>> must add it to my vlan995 iface ?) ?
>>
>> I will test it this morning, and i return back to misc :)
>> --
>> Best regards,
>> Loc BLOT,
>> UNIX systems, security and network expert
>> http://www.unix-experience.fr
>>
>>
>> Le mercredi 03 juillet 2013  02:02 +0200, mxb a crit :
>>> pfsync(4) explains this:
>>>
>>> " The pfsync interface will attempt to collapse multiple state updates
>> into
>>>     a single packet where possible.  The maximum number of times a single
>>>     state can be updated before a pfsync packet will be sent out is
>>>     controlled by the maxupd parameter
>>> "
>>>
>>>
>>> and
>>>
>>> " Where more than one firewall might actively handle packets, e.g. with
>>>     certain ospfd(8), bgpd(8) or carp(4) configurations, it is beneficial
>> to
>>>     defer transmission of the initial packet of a connection.  The pfsync
>>>     state insert message is sent immediately; the packet is queued until
>>>     either this message is acknowledged by another system, or a timeout
>> has
>>>     expired.  This behaviour is enabled with the defer parameter to
>>>     ifconfig(8).
>>> "
>>>
>>>
>>> Eg. "defer: on", yours is "off".
>>>
>>> //mxb
>>>
>>>
>>> On 2 jul 2013, at 21:54, Loc BLOT <[hidden email]> wrote:
>>>
>>>> Hi all
>>>> I have a strange issue (or i haven't read pfsync correctly but i don't
>>>> think this is the problem :D)
>>>>
>>>> I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site.
>>>>
>>>> Those BGP routers are secure with strong PF in stateful mode, and the
>>>> stateful is working very well on each router. Because of my full mesh
>>>> BGP configuration, the outgoing layer 7 sessions can leave my network by
>>>> one router and responses can income by the other.
>>>>
>>>> To resolve this issue, i have created a dedidated VLAN for the pfsync
>>>> traffic and attached pfsync to this VLAN.
>>>>
>>>> Here is a sample output of ifconfig on my first router:
>>>>
>>>> vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>>>>       lladdr a0:36:9f:10:4a:a6
>>>>       priority: 0
>>>>       vlan: 995 parent interface: trunk1
>>>>       groups: vlan
>>>>       status: active
>>>>       inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid
>>>> 0x10
>>>>       inet 10.117.1.129 netmask 0xfffffff8 broadcast 10.117.1.135
>>>> pfsync0: flags=41<UP,RUNNING> mtu 1500
>>>>       priority: 0
>>>>       pfsync: syncdev: vlan995 maxupd: 255 defer: off
>>>>       groups: carp pfsync
>>>>
>>>> And here on my second router:
>>>>
>>>> vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>>>>       lladdr a0:36:9f:17:e2:1e
>>>>       priority: 0
>>>>       vlan: 995 parent interface: trunk1
>>>>       groups: vlan
>>>>       status: active
>>>>       inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid
>>>> 0x10
>>>>       inet 10.117.1.130 netmask 0xfffffff8 broadcast 10.117.1.135
>>>> pfsync0: flags=41<UP,RUNNING> mtu 1500
>>>>       priority: 0
>>>>       pfsync: syncdev: vlan995 maxupd: 255 defer: off
>>>>       groups: carp pfsync
>>>>
>>>> As you see in next tcpdump capture, there is some discussions between
>>>> the two routers:
>>>>
>>>> # tcpdump -nni vlan995
>>>> tcpdump: listening on vlan995, link-type EN10MB
>>>> tcpdump: WARNING: compensating for unaligned libpcap packets
>>>> 23:41:13.699617 10.117.1.130: PFSYNCv6 len 108
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:14.158500 10.117.1.129: PFSYNCv6 len 108
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:14.941396 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3
>>>> bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
>>>> hello=2/0 fwdelay=15/0 pvid=995
>>>> 23:41:14.949617 10.117.1.130: PFSYNCv6 len 108
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:15.237655 10.117.1.129: PFSYNCv6 len 640
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:15.949617 10.117.1.130: PFSYNCv6 len 124
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:16.255230 10.117.1.129: PFSYNCv6 len 36
>>>>   act DEL ST COMP count 1
>>>>       id: 51d16a3500006c33 creatorid: a10bbd21
>>>> (DF) [tos 0x10]
>>>> 23:41:16.946454 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3
>>>> bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
>>>> hello=2/0 fwdelay=15/0 pvid=995
>>>> 23:41:16.949619 10.117.1.130: PFSYNCv6 len 1116
>>>>   act UPD ST COMP count 13
>>>>   ...
>>>> (DF) [tos 0x10]
>>>>
>>>>
>>>> The problem is simple, when i initiate a stateful connection from one
>>>> server, the return (by second router) is blocked by PF (i see the return
>>>> with pflog0)
>>>>
>>>> To be precise here is an example (and tested path):
>>>>
>>>> OBSD NTP -> OBSD router 1 -> WAN...ftp.fr.openbsd.org...WAN -> OBSD
>>>> router 2 || blocked
>>>>
>>>> PF allow in/out routing traffic from this server but incoming from WAN
>>>> is blocked by default
>>>>
>>>> Can you confirm to me that pfsync may add a state for outgoing tcp
>>>> connection in the second router when the first router add it ?
>>>> Have you got any idea on this issue ?
>>>>
>>>> --
>>>> Best regards,
>>>> Loc BLOT,
>>>> UNIX systems, security and network expert
>>>> http://www.unix-experience.fr
>>>>
>>>> [demime 1.01d removed an attachment of type application/pgp-signature
>> which had a name of signature.asc]
>>
>> [demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Loïc Blot-2
Hello,
no carp is used at this time.

My configuration on each router is simple:

em0 + em3 = trunk0
em1 + em2 = trunk1

4 interco vlan (at this time, only 2 are active, 1 for a BGP neighbor
IPv4, 1 for a BGP neighbor IPv6) on trunk0
vlan 50 + vlan 90 + vlan995 on trunk1
pfsync on vlan 995
--
Best regards,

Loïc BLOT, Engineering
UNIX Systems, Security and Networks
http://www.unix-experience.fr


Le mercredi 03 juillet 2013 à 12:47 +0200, mxb a écrit :

> How does your CARP setup looks like. On both machines?
> Can you send your ifconfig output?
>
> What is your environment/setup for this 2-node CARP?
> How interfaces (ext/int) are connected? What switches do you use?
>
>
> On 3 jul 2013, at 10:23, Loïc Blot <[hidden email]> wrote:
>
> > Okay, defer is now enabled on pfsync interface (sorry for my last idea,
> > i haven't the man on me :) ).
> > It seems the problem isn't resolved.
> > The transfer starts but blocked at random time.
> > --
> > Best regards,
> >
> > Loïc BLOT, Engineering
> > UNIX Systems, Security and Networks
> > http://www.unix-experience.fr
> >
> >
> > Le mercredi 03 juillet 2013 à 08:12 +0200, Loïc BLOT a écrit :
> >> Hi,
> >> Thanks for your reply. I wasn't careful about this section.
> >> If i understand i must add defer option to my WAN iface (or i'm wrong i
> >> must add it to my vlan995 iface ?) ?
> >>
> >> I will test it this morning, and i return back to misc :)
> >> --
> >> Best regards,
> >> Loc BLOT,
> >> UNIX systems, security and network expert
> >> http://www.unix-experience.fr
> >>
> >>
> >> Le mercredi 03 juillet 2013  02:02 +0200, mxb a crit :
> >>> pfsync(4) explains this:
> >>>
> >>> " The pfsync interface will attempt to collapse multiple state updates
> >> into
> >>>     a single packet where possible.  The maximum number of times a single
> >>>     state can be updated before a pfsync packet will be sent out is
> >>>     controlled by the maxupd parameter
> >>> "
> >>>
> >>>
> >>> and
> >>>
> >>> " Where more than one firewall might actively handle packets, e.g. with
> >>>     certain ospfd(8), bgpd(8) or carp(4) configurations, it is beneficial
> >> to
> >>>     defer transmission of the initial packet of a connection.  The pfsync
> >>>     state insert message is sent immediately; the packet is queued until
> >>>     either this message is acknowledged by another system, or a timeout
> >> has
> >>>     expired.  This behaviour is enabled with the defer parameter to
> >>>     ifconfig(8).
> >>> "
> >>>
> >>>
> >>> Eg. "defer: on", yours is "off".
> >>>
> >>> //mxb
> >>>
> >>>
> >>> On 2 jul 2013, at 21:54, Loc BLOT <[hidden email]> wrote:
> >>>
> >>>> Hi all
> >>>> I have a strange issue (or i haven't read pfsync correctly but i don't
> >>>> think this is the problem :D)
> >>>>
> >>>> I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site.
> >>>>
> >>>> Those BGP routers are secure with strong PF in stateful mode, and the
> >>>> stateful is working very well on each router. Because of my full mesh
> >>>> BGP configuration, the outgoing layer 7 sessions can leave my network by
> >>>> one router and responses can income by the other.
> >>>>
> >>>> To resolve this issue, i have created a dedidated VLAN for the pfsync
> >>>> traffic and attached pfsync to this VLAN.
> >>>>
> >>>> Here is a sample output of ifconfig on my first router:
> >>>>
> >>>> vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> >>>>       lladdr a0:36:9f:10:4a:a6
> >>>>       priority: 0
> >>>>       vlan: 995 parent interface: trunk1
> >>>>       groups: vlan
> >>>>       status: active
> >>>>       inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid
> >>>> 0x10
> >>>>       inet 10.117.1.129 netmask 0xfffffff8 broadcast 10.117.1.135
> >>>> pfsync0: flags=41<UP,RUNNING> mtu 1500
> >>>>       priority: 0
> >>>>       pfsync: syncdev: vlan995 maxupd: 255 defer: off
> >>>>       groups: carp pfsync
> >>>>
> >>>> And here on my second router:
> >>>>
> >>>> vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> >>>>       lladdr a0:36:9f:17:e2:1e
> >>>>       priority: 0
> >>>>       vlan: 995 parent interface: trunk1
> >>>>       groups: vlan
> >>>>       status: active
> >>>>       inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid
> >>>> 0x10
> >>>>       inet 10.117.1.130 netmask 0xfffffff8 broadcast 10.117.1.135
> >>>> pfsync0: flags=41<UP,RUNNING> mtu 1500
> >>>>       priority: 0
> >>>>       pfsync: syncdev: vlan995 maxupd: 255 defer: off
> >>>>       groups: carp pfsync
> >>>>
> >>>> As you see in next tcpdump capture, there is some discussions between
> >>>> the two routers:
> >>>>
> >>>> # tcpdump -nni vlan995
> >>>> tcpdump: listening on vlan995, link-type EN10MB
> >>>> tcpdump: WARNING: compensating for unaligned libpcap packets
> >>>> 23:41:13.699617 10.117.1.130: PFSYNCv6 len 108
> >>>>   act UPD ST COMP count 1
> >>>>   ...
> >>>> (DF) [tos 0x10]
> >>>> 23:41:14.158500 10.117.1.129: PFSYNCv6 len 108
> >>>>   act UPD ST COMP count 1
> >>>>   ...
> >>>> (DF) [tos 0x10]
> >>>> 23:41:14.941396 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3
> >>>> bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
> >>>> hello=2/0 fwdelay=15/0 pvid=995
> >>>> 23:41:14.949617 10.117.1.130: PFSYNCv6 len 108
> >>>>   act UPD ST COMP count 1
> >>>>   ...
> >>>> (DF) [tos 0x10]
> >>>> 23:41:15.237655 10.117.1.129: PFSYNCv6 len 640
> >>>>   act UPD ST COMP count 1
> >>>>   ...
> >>>> (DF) [tos 0x10]
> >>>> 23:41:15.949617 10.117.1.130: PFSYNCv6 len 124
> >>>>   act UPD ST COMP count 1
> >>>>   ...
> >>>> (DF) [tos 0x10]
> >>>> 23:41:16.255230 10.117.1.129: PFSYNCv6 len 36
> >>>>   act DEL ST COMP count 1
> >>>>       id: 51d16a3500006c33 creatorid: a10bbd21
> >>>> (DF) [tos 0x10]
> >>>> 23:41:16.946454 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3
> >>>> bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
> >>>> hello=2/0 fwdelay=15/0 pvid=995
> >>>> 23:41:16.949619 10.117.1.130: PFSYNCv6 len 1116
> >>>>   act UPD ST COMP count 13
> >>>>   ...
> >>>> (DF) [tos 0x10]
> >>>>
> >>>>
> >>>> The problem is simple, when i initiate a stateful connection from one
> >>>> server, the return (by second router) is blocked by PF (i see the return
> >>>> with pflog0)
> >>>>
> >>>> To be precise here is an example (and tested path):
> >>>>
> >>>> OBSD NTP -> OBSD router 1 -> WAN...ftp.fr.openbsd.org...WAN -> OBSD
> >>>> router 2 || blocked
> >>>>
> >>>> PF allow in/out routing traffic from this server but incoming from WAN
> >>>> is blocked by default
> >>>>
> >>>> Can you confirm to me that pfsync may add a state for outgoing tcp
> >>>> connection in the second router when the first router add it ?
> >>>> Have you got any idea on this issue ?
> >>>>
> >>>> --
> >>>> Best regards,
> >>>> Loc BLOT,
> >>>> UNIX systems, security and network expert
> >>>> http://www.unix-experience.fr
> >>>>
> >>>> [demime 1.01d removed an attachment of type application/pgp-signature
> >> which had a name of signature.asc]
> >>
> >> [demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Mark Felder-4
On Wed, 03 Jul 2013 07:00:02 -0500, Loïc Blot  
<[hidden email]> wrote:

> Hello,
> no carp is used at this time.

pfsync needs to be used with carp... without it you're just playing  
whack-a-mole with your session table.

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Loïc Blot-2
It's not possible to sync pf table without CARP ?

I must use it in some case, then those case will be fixed but the other
(OSPFd routing) may fail i think ?

--
Best regards,

Loïc BLOT, Engineering
UNIX Systems, Security and Networks
http://www.unix-experience.fr


Le mercredi 03 juillet 2013 à 07:11 -0500, Mark Felder a écrit :
> On Wed, 03 Jul 2013 07:00:02 -0500, Loïc Blot  
> <[hidden email]> wrote:
>
> > Hello,
> > no carp is used at this time.
>
> pfsync needs to be used with carp... without it you're just playing  
> whack-a-mole with your session table.

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Mark Felder-4
On Wed, 03 Jul 2013 07:40:08 -0500, Loïc Blot  
<[hidden email]> wrote:

> It's not possible to sync pf table without CARP ?

In order to answer that I'll need to understand what you believe the "pf  
table" is.

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Maxim Bourmistrov-5
In reply to this post by Loïc Blot-2
Sure it syncs, but
node1 has completely different IP addresses than node2(both external and internal ??), if no CARP.
So storing states from node1, which passes/initiated connection to ftp.fr , on node2 does not help.

In your case, you'd probably to decide to ever have MASTER-BACKUP or to have MASTER-MASTER CARP setup.


On 3 jul 2013, at 14:40, Loïc Blot <[hidden email]> wrote:

> It's not possible to sync pf table without CARP ?
>
> I must use it in some case, then those case will be fixed but the other
> (OSPFd routing) may fail i think ?
>
> --
> Best regards,
>
> Loïc BLOT, Engineering
> UNIX Systems, Security and Networks
> http://www.unix-experience.fr
>
>
> Le mercredi 03 juillet 2013 à 07:11 -0500, Mark Felder a écrit :
>> On Wed, 03 Jul 2013 07:00:02 -0500, Loïc Blot  
>> <[hidden email]> wrote:
>>
>>> Hello,
>>> no carp is used at this time.
>>
>> pfsync needs to be used with carp... without it you're just playing  
>> whack-a-mole with your session table.

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Loïc Blot-2
In reply to this post by Mark Felder-4
For me pf table is (sorry for the missing precisions) the pf state
stable for stateful operations
--
Best regards,

Loïc BLOT, Engineering
UNIX Systems, Security and Networks
http://www.unix-experience.fr


Le mercredi 03 juillet 2013 à 08:22 -0500, Mark Felder a écrit :
> On Wed, 03 Jul 2013 07:40:08 -0500, Loïc Blot  
> <[hidden email]> wrote:
>
> > It's not possible to sync pf table without CARP ?
>
> In order to answer that I'll need to understand what you believe the "pf  
> table" is.

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Patrick Lamaiziere
In reply to this post by Mark Felder-4
Le Wed, 03 Jul 2013 07:11:08 -0500,
"Mark Felder" <[hidden email]> a écrit :

> On Wed, 03 Jul 2013 07:00:02 -0500, Loïc Blot  
> <[hidden email]> wrote:
>
> > Hello,
> > no carp is used at this time.
>
> pfsync needs to be used with carp... without it you're just playing  
> whack-a-mole with your session table.

I don't see why as states are not attached on carp interfaces but to
real interfaces (if-bounded) or even by default to no interface at
all ?

Regards

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Mark Felder-4
In reply to this post by Loïc Blot-2
On Wed, 03 Jul 2013 09:24:54 -0500, Loïc Blot  
<[hidden email]> wrote:

> For me pf table is (sorry for the missing precisions) the pf state
> stable for stateful operations

First of all, the states of node 1 being synced to node 2 and vice versa  
is worthless because they have different IP addresses; the states wont  
match anything.

Secondly, you'll probably end up dealing with the nodes fighting each  
other as they sync back and forth. If a state from node1 is synced to  
node2 and node2 decides to expire that session because it hasn't been used  
it will tell node1 to remove that session as well. Now your session that  
was working on node1 has stopped functioning. This is probably the  
hanging/stalling behavior you were experiencing before. I've never even  
attempted to set this up in a lab and I know nothing of the pfsync/pf  
code, but I assume this is what is happening to you. I'm actually quite  
surprised it will even accept any changes to states for IPs that don't  
exist on the server, but I suppose it doesn't seem worthwhile to put such  
strict validation on it.

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Loïc Blot-2
I don't understand why they can't be synced because if i have this
scheme:

server 1 - | Router 1 + Router 2 | remote

server 1 contact remote, outgoing by Router 1 and the return traffic
comes from Router 2.

The state may have "server 1 port A to remote port B", then the virtual
IP is useless in this configuration, no ?
--
Best regards,

Loïc BLOT, Engineering
UNIX Systems, Security and Networks
http://www.unix-experience.fr


Le mercredi 03 juillet 2013 à 09:36 -0500, Mark Felder a écrit :

> On Wed, 03 Jul 2013 09:24:54 -0500, Loïc Blot  
> <[hidden email]> wrote:
>
> > For me pf table is (sorry for the missing precisions) the pf state
> > stable for stateful operations
>
> First of all, the states of node 1 being synced to node 2 and vice versa  
> is worthless because they have different IP addresses; the states wont  
> match anything.
>
> Secondly, you'll probably end up dealing with the nodes fighting each  
> other as they sync back and forth. If a state from node1 is synced to  
> node2 and node2 decides to expire that session because it hasn't been used  
> it will tell node1 to remove that session as well. Now your session that  
> was working on node1 has stopped functioning. This is probably the  
> hanging/stalling behavior you were experiencing before. I've never even  
> attempted to set this up in a lab and I know nothing of the pfsync/pf  
> code, but I assume this is what is happening to you. I'm actually quite  
> surprised it will even accept any changes to states for IPs that don't  
> exist on the server, but I suppose it doesn't seem worthwhile to put such  
> strict validation on it.

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Maxim Bourmistrov-5
States ARE synced.
IPs are not the same on node1 and node2 for external. The you initiated connection to ftp.fr, you done it via node1 with its external IP. On node2 those packets will be DROPPED as those do not belong to external NIC on node2 (IP)



On 3 jul 2013, at 17:16, Loïc Blot <[hidden email]> wrote:

> I don't understand why they can't be synced because if i have this
> scheme:
>
> server 1 - | Router 1 + Router 2 | remote
>
> server 1 contact remote, outgoing by Router 1 and the return traffic
> comes from Router 2.
>
> The state may have "server 1 port A to remote port B", then the virtual
> IP is useless in this configuration, no ?
> --
> Best regards,
>
> Loïc BLOT, Engineering
> UNIX Systems, Security and Networks
> http://www.unix-experience.fr
>
>
> Le mercredi 03 juillet 2013 à 09:36 -0500, Mark Felder a écrit :
>> On Wed, 03 Jul 2013 09:24:54 -0500, Loïc Blot  
>> <[hidden email]> wrote:
>>
>>> For me pf table is (sorry for the missing precisions) the pf state
>>> stable for stateful operations
>>
>> First of all, the states of node 1 being synced to node 2 and vice versa  
>> is worthless because they have different IP addresses; the states wont  
>> match anything.
>>
>> Secondly, you'll probably end up dealing with the nodes fighting each  
>> other as they sync back and forth. If a state from node1 is synced to  
>> node2 and node2 decides to expire that session because it hasn't been used  
>> it will tell node1 to remove that session as well. Now your session that  
>> was working on node1 has stopped functioning. This is probably the  
>> hanging/stalling behavior you were experiencing before. I've never even  
>> attempted to set this up in a lab and I know nothing of the pfsync/pf  
>> code, but I assume this is what is happening to you. I'm actually quite  
>> surprised it will even accept any changes to states for IPs that don't  
>> exist on the server, but I suppose it doesn't seem worthwhile to put such  
>> strict validation on it.

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Loïc Blot-2
The connection is not done by my routers themselves but by DMZ servers
behind them !
--
Best regards,

Loïc BLOT, Engineering
UNIX Systems, Security and Networks
http://www.unix-experience.fr


Le mercredi 03 juillet 2013 à 17:32 +0200, mxb a écrit :

> States ARE synced.
> IPs are not the same on node1 and node2 for external. The you initiated connection to ftp.fr, you done it via node1 with its external IP. On node2 those packets will be DROPPED as those do not belong to external NIC on node2 (IP)
>
>
>
> On 3 jul 2013, at 17:16, Loïc Blot <[hidden email]> wrote:
>
> > I don't understand why they can't be synced because if i have this
> > scheme:
> >
> > server 1 - | Router 1 + Router 2 | remote
> >
> > server 1 contact remote, outgoing by Router 1 and the return traffic
> > comes from Router 2.
> >
> > The state may have "server 1 port A to remote port B", then the virtual
> > IP is useless in this configuration, no ?
> > --
> > Best regards,
> >
> > Loïc BLOT, Engineering
> > UNIX Systems, Security and Networks
> > http://www.unix-experience.fr
> >
> >
> > Le mercredi 03 juillet 2013 à 09:36 -0500, Mark Felder a écrit :
> >> On Wed, 03 Jul 2013 09:24:54 -0500, Loïc Blot  
> >> <[hidden email]> wrote:
> >>
> >>> For me pf table is (sorry for the missing precisions) the pf state
> >>> stable for stateful operations
> >>
> >> First of all, the states of node 1 being synced to node 2 and vice versa  
> >> is worthless because they have different IP addresses; the states wont  
> >> match anything.
> >>
> >> Secondly, you'll probably end up dealing with the nodes fighting each  
> >> other as they sync back and forth. If a state from node1 is synced to  
> >> node2 and node2 decides to expire that session because it hasn't been used  
> >> it will tell node1 to remove that session as well. Now your session that  
> >> was working on node1 has stopped functioning. This is probably the  
> >> hanging/stalling behavior you were experiencing before. I've never even  
> >> attempted to set this up in a lab and I know nothing of the pfsync/pf  
> >> code, but I assume this is what is happening to you. I'm actually quite  
> >> surprised it will even accept any changes to states for IPs that don't  
> >> exist on the server, but I suppose it doesn't seem worthwhile to put such  
> >> strict validation on it.

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

David Gwynne-5
In reply to this post by Mark Felder-4
On 03/07/2013, at 10:11 PM, Mark Felder <[hidden email]> wrote:

> On Wed, 03 Jul 2013 07:00:02 -0500, Loïc Blot <[hidden email]> wrote:
>
>> Hello,
>> no carp is used at this time.
>
> pfsync needs to be used with carp... without it you're just playing whack-a-mole with your session table.

no it doesnt. pfsync just does its best to keep the state table in sync, it in no way relies on carp to achieve that.

however, it does provide feedback to carp to try and avoid the box becoming a master and therefore taking traffic until it either thinks it has the whole state table from a peer or it is alone.

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

David Gwynne-5
In reply to this post by Loïc Blot-2
On 03/07/2013, at 6:23 PM, Loïc Blot <[hidden email]> wrote:

> Okay, defer is now enabled on pfsync interface (sorry for my last idea,
> i haven't the man on me :) ).
> It seems the problem isn't resolved.
> The transfer starts but blocked at random time.

i have hit this too, despite being the person most responsible for trying to make pfsync work in active-active (hi bob!) configurations.

the problem is the tcp window tracking pf does, and how pfsync tries to cope with different routers being responsible for different halves of the packet flow. pfsync tries to merge each side of the tcp windows and tries to detect split paths to exchange updates more rapidly for those states. however, i find at some point the actual tcp windows move too fast for pfsync to keep up and all the real packets fall out of the window, causing the stalls you're talking about.

my solution is to try and prefer one half of the firewalls for all traffic, and use the second for handling failure. the split path handling works well enough that we can support traffic while we change roles (moving master to slave and slave to master) and the upstream hasnt figured it out yet via ospf.

sorry for the bad news. i might try and have a look at the state merge code again and see if there's something obvious i am missing.

cheers,
dlg

> --
> Best regards,
>
> Loïc BLOT, Engineering
> UNIX Systems, Security and Networks
> http://www.unix-experience.fr
>
>
> Le mercredi 03 juillet 2013 à 08:12 +0200, Loïc BLOT a écrit :
>> Hi,
>> Thanks for your reply. I wasn't careful about this section.
>> If i understand i must add defer option to my WAN iface (or i'm wrong i
>> must add it to my vlan995 iface ?) ?
>>
>> I will test it this morning, and i return back to misc :)
>> --
>> Best regards,
>> Loc BLOT,
>> UNIX systems, security and network expert
>> http://www.unix-experience.fr
>>
>>
>> Le mercredi 03 juillet 2013  02:02 +0200, mxb a crit :
>>> pfsync(4) explains this:
>>>
>>> " The pfsync interface will attempt to collapse multiple state updates
>> into
>>>     a single packet where possible.  The maximum number of times a single
>>>     state can be updated before a pfsync packet will be sent out is
>>>     controlled by the maxupd parameter
>>> "
>>>
>>>
>>> and
>>>
>>> " Where more than one firewall might actively handle packets, e.g. with
>>>     certain ospfd(8), bgpd(8) or carp(4) configurations, it is beneficial
>> to
>>>     defer transmission of the initial packet of a connection.  The pfsync
>>>     state insert message is sent immediately; the packet is queued until
>>>     either this message is acknowledged by another system, or a timeout
>> has
>>>     expired.  This behaviour is enabled with the defer parameter to
>>>     ifconfig(8).
>>> "
>>>
>>>
>>> Eg. "defer: on", yours is "off".
>>>
>>> //mxb
>>>
>>>
>>> On 2 jul 2013, at 21:54, Loc BLOT <[hidden email]> wrote:
>>>
>>>> Hi all
>>>> I have a strange issue (or i haven't read pfsync correctly but i don't
>>>> think this is the problem :D)
>>>>
>>>> I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site.
>>>>
>>>> Those BGP routers are secure with strong PF in stateful mode, and the
>>>> stateful is working very well on each router. Because of my full mesh
>>>> BGP configuration, the outgoing layer 7 sessions can leave my network by
>>>> one router and responses can income by the other.
>>>>
>>>> To resolve this issue, i have created a dedidated VLAN for the pfsync
>>>> traffic and attached pfsync to this VLAN.
>>>>
>>>> Here is a sample output of ifconfig on my first router:
>>>>
>>>> vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>>>>       lladdr a0:36:9f:10:4a:a6
>>>>       priority: 0
>>>>       vlan: 995 parent interface: trunk1
>>>>       groups: vlan
>>>>       status: active
>>>>       inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid
>>>> 0x10
>>>>       inet 10.117.1.129 netmask 0xfffffff8 broadcast 10.117.1.135
>>>> pfsync0: flags=41<UP,RUNNING> mtu 1500
>>>>       priority: 0
>>>>       pfsync: syncdev: vlan995 maxupd: 255 defer: off
>>>>       groups: carp pfsync
>>>>
>>>> And here on my second router:
>>>>
>>>> vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>>>>       lladdr a0:36:9f:17:e2:1e
>>>>       priority: 0
>>>>       vlan: 995 parent interface: trunk1
>>>>       groups: vlan
>>>>       status: active
>>>>       inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid
>>>> 0x10
>>>>       inet 10.117.1.130 netmask 0xfffffff8 broadcast 10.117.1.135
>>>> pfsync0: flags=41<UP,RUNNING> mtu 1500
>>>>       priority: 0
>>>>       pfsync: syncdev: vlan995 maxupd: 255 defer: off
>>>>       groups: carp pfsync
>>>>
>>>> As you see in next tcpdump capture, there is some discussions between
>>>> the two routers:
>>>>
>>>> # tcpdump -nni vlan995
>>>> tcpdump: listening on vlan995, link-type EN10MB
>>>> tcpdump: WARNING: compensating for unaligned libpcap packets
>>>> 23:41:13.699617 10.117.1.130: PFSYNCv6 len 108
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:14.158500 10.117.1.129: PFSYNCv6 len 108
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:14.941396 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3
>>>> bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
>>>> hello=2/0 fwdelay=15/0 pvid=995
>>>> 23:41:14.949617 10.117.1.130: PFSYNCv6 len 108
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:15.237655 10.117.1.129: PFSYNCv6 len 640
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:15.949617 10.117.1.130: PFSYNCv6 len 124
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:16.255230 10.117.1.129: PFSYNCv6 len 36
>>>>   act DEL ST COMP count 1
>>>>       id: 51d16a3500006c33 creatorid: a10bbd21
>>>> (DF) [tos 0x10]
>>>> 23:41:16.946454 SSTP STP config root=83e3.0:a:b8:7b:27:80 rootcost=3
>>>> bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
>>>> hello=2/0 fwdelay=15/0 pvid=995
>>>> 23:41:16.949619 10.117.1.130: PFSYNCv6 len 1116
>>>>   act UPD ST COMP count 13
>>>>   ...
>>>> (DF) [tos 0x10]
>>>>
>>>>
>>>> The problem is simple, when i initiate a stateful connection from one
>>>> server, the return (by second router) is blocked by PF (i see the return
>>>> with pflog0)
>>>>
>>>> To be precise here is an example (and tested path):
>>>>
>>>> OBSD NTP -> OBSD router 1 -> WAN...ftp.fr.openbsd.org...WAN -> OBSD
>>>> router 2 || blocked
>>>>
>>>> PF allow in/out routing traffic from this server but incoming from WAN
>>>> is blocked by default
>>>>
>>>> Can you confirm to me that pfsync may add a state for outgoing tcp
>>>> connection in the second router when the first router add it ?
>>>> Have you got any idea on this issue ?
>>>>
>>>> --
>>>> Best regards,
>>>> Loc BLOT,
>>>> UNIX systems, security and network expert
>>>> http://www.unix-experience.fr
>>>>
>>>> [demime 1.01d removed an attachment of type application/pgp-signature
>> which had a name of signature.asc]
>>
>> [demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

BARDOU Pierre
Hello,

I don't know if this may help you, but I have a working BGP setup with two routers active/active.
I don't use pfsync, but keep state (sloppy).

This is less secure according to pf.conf(5), but that's not really a concern for me as those routers are not my border firewalls...
But maybe I am mistaking doing this ?

--
Cordialement,
Pierre BARDOU


-----Message d'origine-----
De : David Gwynne [mailto:[hidden email]]
Envoyé : jeudi 4 juillet 2013 09:47
À : [hidden email]
Cc : [hidden email]
Objet : Re: PF sync doesn't not work very well

On 03/07/2013, at 6:23 PM, Loïc Blot <[hidden email]> wrote:

> Okay, defer is now enabled on pfsync interface (sorry for my last
> idea, i haven't the man on me :) ).
> It seems the problem isn't resolved.
> The transfer starts but blocked at random time.

i have hit this too, despite being the person most responsible for trying to make pfsync work in active-active (hi bob!) configurations.

the problem is the tcp window tracking pf does, and how pfsync tries to cope with different routers being responsible for different halves of the packet flow. pfsync tries to merge each side of the tcp windows and tries to detect split paths to exchange updates more rapidly for those states. however, i find at some point the actual tcp windows move too fast for pfsync to keep up and all the real packets fall out of the window, causing the stalls you're talking about.

my solution is to try and prefer one half of the firewalls for all traffic, and use the second for handling failure. the split path handling works well enough that we can support traffic while we change roles (moving master to slave and slave to master) and the upstream hasnt figured it out yet via ospf.

sorry for the bad news. i might try and have a look at the state merge code again and see if there's something obvious i am missing.

cheers,
dlg

> --
> Best regards,
>
> Loïc BLOT, Engineering
> UNIX Systems, Security and Networks
> http://www.unix-experience.fr
>
>
> Le mercredi 03 juillet 2013 à 08:12 +0200, Loïc BLOT a écrit :
>> Hi,
>> Thanks for your reply. I wasn't careful about this section.
>> If i understand i must add defer option to my WAN iface (or i'm wrong
>> i must add it to my vlan995 iface ?) ?
>>
>> I will test it this morning, and i return back to misc :)
>> --
>> Best regards,
>> Loc BLOT,
>> UNIX systems, security and network expert
>> http://www.unix-experience.fr
>>
>>
>> Le mercredi 03 juillet 2013  02:02 +0200, mxb a crit :
>>> pfsync(4) explains this:
>>>
>>> " The pfsync interface will attempt to collapse multiple state
>>> updates
>> into
>>>     a single packet where possible.  The maximum number of times a single
>>>     state can be updated before a pfsync packet will be sent out is
>>>     controlled by the maxupd parameter "
>>>
>>>
>>> and
>>>
>>> " Where more than one firewall might actively handle packets, e.g. with
>>>     certain ospfd(8), bgpd(8) or carp(4) configurations, it is
>>> beneficial
>> to
>>>     defer transmission of the initial packet of a connection.  The pfsync
>>>     state insert message is sent immediately; the packet is queued until
>>>     either this message is acknowledged by another system, or a
>>> timeout
>> has
>>>     expired.  This behaviour is enabled with the defer parameter to
>>>     ifconfig(8).
>>> "
>>>
>>>
>>> Eg. "defer: on", yours is "off".
>>>
>>> //mxb
>>>
>>>
>>> On 2 jul 2013, at 21:54, Loc BLOT <[hidden email]> wrote:
>>>
>>>> Hi all
>>>> I have a strange issue (or i haven't read pfsync correctly but i
>>>> don't think this is the problem :D)
>>>>
>>>> I'm using 2 OpenBSD as BGP+OSPF routers at the border of one site.
>>>>
>>>> Those BGP routers are secure with strong PF in stateful mode, and
>>>> the stateful is working very well on each router. Because of my
>>>> full mesh BGP configuration, the outgoing layer 7 sessions can
>>>> leave my network by one router and responses can income by the other.
>>>>
>>>> To resolve this issue, i have created a dedidated VLAN for the
>>>> pfsync traffic and attached pfsync to this VLAN.
>>>>
>>>> Here is a sample output of ifconfig on my first router:
>>>>
>>>> vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>>>>       lladdr a0:36:9f:10:4a:a6
>>>>       priority: 0
>>>>       vlan: 995 parent interface: trunk1
>>>>       groups: vlan
>>>>       status: active
>>>>       inet6 fe80::a236:9fff:fe10:4aa6%vlan995 prefixlen 64 scopeid
>>>> 0x10
>>>>       inet 10.117.1.129 netmask 0xfffffff8 broadcast 10.117.1.135
>>>> pfsync0: flags=41<UP,RUNNING> mtu 1500
>>>>       priority: 0
>>>>       pfsync: syncdev: vlan995 maxupd: 255 defer: off
>>>>       groups: carp pfsync
>>>>
>>>> And here on my second router:
>>>>
>>>> vlan995: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>>>>       lladdr a0:36:9f:17:e2:1e
>>>>       priority: 0
>>>>       vlan: 995 parent interface: trunk1
>>>>       groups: vlan
>>>>       status: active
>>>>       inet6 fe80::a236:9fff:fe17:e21e%vlan995 prefixlen 64 scopeid
>>>> 0x10
>>>>       inet 10.117.1.130 netmask 0xfffffff8 broadcast 10.117.1.135
>>>> pfsync0: flags=41<UP,RUNNING> mtu 1500
>>>>       priority: 0
>>>>       pfsync: syncdev: vlan995 maxupd: 255 defer: off
>>>>       groups: carp pfsync
>>>>
>>>> As you see in next tcpdump capture, there is some discussions
>>>> between the two routers:
>>>>
>>>> # tcpdump -nni vlan995
>>>> tcpdump: listening on vlan995, link-type EN10MB
>>>> tcpdump: WARNING: compensating for unaligned libpcap packets
>>>> 23:41:13.699617 10.117.1.130: PFSYNCv6 len 108
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:14.158500 10.117.1.129: PFSYNCv6 len 108
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:14.941396 SSTP STP config root=83e3.0:a:b8:7b:27:80
>>>> rootcost=3
>>>> bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
>>>> hello=2/0 fwdelay=15/0 pvid=995
>>>> 23:41:14.949617 10.117.1.130: PFSYNCv6 len 108
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:15.237655 10.117.1.129: PFSYNCv6 len 640
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:15.949617 10.117.1.130: PFSYNCv6 len 124
>>>>   act UPD ST COMP count 1
>>>>   ...
>>>> (DF) [tos 0x10]
>>>> 23:41:16.255230 10.117.1.129: PFSYNCv6 len 36
>>>>   act DEL ST COMP count 1
>>>>       id: 51d16a3500006c33 creatorid: a10bbd21
>>>> (DF) [tos 0x10]
>>>> 23:41:16.946454 SSTP STP config root=83e3.0:a:b8:7b:27:80
>>>> rootcost=3
>>>> bridge=c3e3.0:17:e:2e:f:80 port=142 ifcost=130 age=1/0 max=20/0
>>>> hello=2/0 fwdelay=15/0 pvid=995
>>>> 23:41:16.949619 10.117.1.130: PFSYNCv6 len 1116
>>>>   act UPD ST COMP count 13
>>>>   ...
>>>> (DF) [tos 0x10]
>>>>
>>>>
>>>> The problem is simple, when i initiate a stateful connection from
>>>> one server, the return (by second router) is blocked by PF (i see
>>>> the return with pflog0)
>>>>
>>>> To be precise here is an example (and tested path):
>>>>
>>>> OBSD NTP -> OBSD router 1 -> WAN...ftp.fr.openbsd.org...WAN -> OBSD
>>>> router 2 || blocked
>>>>
>>>> PF allow in/out routing traffic from this server but incoming from
>>>> WAN is blocked by default
>>>>
>>>> Can you confirm to me that pfsync may add a state for outgoing tcp
>>>> connection in the second router when the first router add it ?
>>>> Have you got any idea on this issue ?
>>>>
>>>> --
>>>> Best regards,
>>>> Loc BLOT,
>>>> UNIX systems, security and network expert
>>>> http://www.unix-experience.fr
>>>>
>>>> [demime 1.01d removed an attachment of type
>>>> application/pgp-signature
>> which had a name of signature.asc]
>>
>> [demime 1.01d removed an attachment of type application/pgp-signature
>> which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: PF sync doesn't not work very well

Henning Brauer
In reply to this post by Mark Felder-4
[pfsync w/o carp]

* Mark Felder <[hidden email]> [2013-07-03 16:37]:
> First of all, the states of node 1 being synced to node 2 and vice
> versa is worthless because they have different IP addresses; the
> states wont match anything.

orly.
have you actually LOOKED at your state table?
pfctl -vvss to the rescue.

a tcp connection from 81.209.180.1 to 129.128.5.194 cares about wether the
intermediate firewall has 80.86.183.252 or 80.86.183.253 how exactly?
how do bridges work at all?

miracles all over the place!

> Secondly, you'll probably end up dealing with the nodes fighting
> each other as they sync back and forth. If a state from node1 is
> synced to node2 and node2 decides to expire that session because it
> hasn't been used it will tell node1 to remove that session as well.

sigh. this is completely wrong, too.

> I've never even attempted to set this up in a lab and I know
> nothing of the pfsync/pf code,

yeah, you made that obvious.

> but I assume

aha.

--
Henning Brauer, [hidden email], [hidden email]
BS Web Services, http://bsws.de, Full-Service ISP
Secure Hosting, Mail and DNS Services. Dedicated Servers, Root to Fully Managed
Henning Brauer Consulting, http://henningbrauer.com/

12