Dual-ISP home router setup problems

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Dual-ISP home router setup problems

Samuel Wagen
Hello,

I'm trying to build a home router with OpenBSD. I have two ISPs, both are
giving me real IPs, one with straight DHCP (ISP_A), the other - via PPPoE
(ISP_B). I've described the topology with more detail in the diagram below.

I wanted to use PF with routing domains instead of multipath forwarding, due
to multipath being very finicky when a link goes down. My current setup is
described below. I have the following issues:

    - Initially I can't pass traffic from the LAN. I think this is due to the
      packets on em0 being dropped before PF has a chance to reach them, due
      to missing default route on rdomain 0. If I execute the following two
      commands:
            # route -T 0 add 198.51.100.0/24 127.0.0.1
            # route -T 0 add 203.0.113.0/24 127.0.0.1
      then traffic starts passing half of the time - if the round-robin
      decides it should go over the PPPoE link (ISP_B) - traffic from the LAN
      flows. If, however, it decides to go through the other link (ISP_A) -
      nothing passes, and I get the following kernel messages:

            arpresolve: 198.51.100.0: route contains no arp information

    - Traffic from the gateway itself to the Internet always fails, unless I
      specify a routing domain manually (route -T 1 exec whatever). Not sure
      what bogus route to add here, so that packets aren't dropped before PF,
      and what to add to PF so that they flow.

In other words, I'm stuck, and need some pointers on how to continue and what
am I doing wrong. I'm running latest snapshot, but also tried with 6.2.

Many thanks in advance.

Here's the info about my config, let me know if you need me to provide some
more. The "internet" networks are from RFC5737 for illustration purposes.

1. Network diagram

   +---------+               +---------+
   |  ISP_A  |               |  ISP_B  |
   +---+-----+               +---+-----+
       |                         |
       |                         |
       |                         |
++-----+-------------------------+----------------++
||  em1                      em2/pppoe0           ||
||  DHCP client              real IP              ||
||  IP: 198.51.100.20        IP: 203.0.113.40     ||
||  Net: 198.51.100.0/24     Net: 203.0.113.0/24  ||
||  GW: 198.51.100.1         GW: 203.0.113.1      ||
||  rdomain 1                rdomain 2            ||
G|  group isp_a              group isp_b          |G
A|                                                |A
T|                                                |T
E+- - - - - - - - - - - NAT- - - - - - - - - - - -+E
W|                                                |W
A|                                                |A
Y|               em0                              |Y
||               DHCP server                      ||
||               IP: 172.16.16.1                  ||
||               Net: 172.16.16.0/24              ||
||               rdomain 0                        ||
||               group lan                        ||
++--------------------+---------------------------++
                      |
                      |
                      |
                   +--+--------+
                   |    LAN    |
                   +-----------+


2. Interface config files

- /etc/hostname.em0

    inet 172.16.16.1 255.255.255.0 172.16.16.255 group lan

- /etc/hostname.em1

    dhcp group isp_a rdomain 1

- /etc/hostname.em2

    up

- /etc/hostname.pppoe0

    inet 0.0.0.0 255.255.255.255 NONE \
        pppoedev em2 authproto chap \
        authname 'user' authkey 'verysecret' \
        group isp_b \
        rdomain 2 \
        up
    dest 0.0.0.1
    !/sbin/route -T 2 add default -ifp pppoe0 0.0.0.1


3. DHCP server config (/etc/dhcpd.conf)

    subnet 172.16.16.0 netmask 255.255.255.0 {
        option domain-name-servers 172.16.16.2, 172.16.16.3;
        option routers 172.16.16.1;
        range 172.16.16.100 172.16.16.199;
    }


4. PF config

    # Need to figure out how avoid hardcoding these
    isp_a_gw = "172.16.18.1"
    isp_b_gw = "192.168.68.1"

    set debug debug

    match in log all scrub (no-df random-id max-mss 1440)

    match out log on em1 from (lan:network) nat-to (em1)
    match out log on pppoe0 from (lan:network) nat-to (pppoe0)

    pass out log on lan to (lan:network)
    pass in log quick on lan from (lan:network) to (lan)

    pass in log on lan from (lan:network) \
        route-to { (em1 $isp_a_gw), (pppoe0  $isp_b_gw) } \
        round-robin

    pass out log on em1 from pppoe0 route-to (pppoe0 $isp_b_gw)
    pass out log on pppoe0 from em1 route-to (em1 $isp_a_gw)

    pass out log quick on em1 inet from (em1) modulate state rtable 1
    pass out log quick on pppoe0 from (pppoe0) modulate state rtable 2


5. Additional issues

- How to avoid hardcoding the ISP defaut routes?
- How to use sticky sessions instead of round-robin?
- How to deal with links going down? E.g. not try to send traffic to a failed
  link.


--
sw

Reply | Threaded
Open this post in threaded view
|

Re: Dual-ISP home router setup problems

Samuel Wagen
And of course, too much copy paste while trying to use documentation
IP ranges. The two gateways in pf.conf above should be

isp_a_gw = "198.51.100.1"
isp_b_gw = "203.0.113.1"

The rest stands.

On Mon, Mar 19, 2018 at 1:40 PM, Samuel Wagen <[hidden email]> wrote:

> Hello,
>
> I'm trying to build a home router with OpenBSD. I have two ISPs, both are
> giving me real IPs, one with straight DHCP (ISP_A), the other - via PPPoE
> (ISP_B). I've described the topology with more detail in the diagram below.
>
> I wanted to use PF with routing domains instead of multipath forwarding, due
> to multipath being very finicky when a link goes down. My current setup is
> described below. I have the following issues:
>
>     - Initially I can't pass traffic from the LAN. I think this is due to the
>       packets on em0 being dropped before PF has a chance to reach them, due
>       to missing default route on rdomain 0. If I execute the following two
>       commands:
>             # route -T 0 add 198.51.100.0/24 127.0.0.1
>             # route -T 0 add 203.0.113.0/24 127.0.0.1
>       then traffic starts passing half of the time - if the round-robin
>       decides it should go over the PPPoE link (ISP_B) - traffic from the LAN
>       flows. If, however, it decides to go through the other link (ISP_A) -
>       nothing passes, and I get the following kernel messages:
>
>             arpresolve: 198.51.100.0: route contains no arp information
>
>     - Traffic from the gateway itself to the Internet always fails, unless I
>       specify a routing domain manually (route -T 1 exec whatever). Not sure
>       what bogus route to add here, so that packets aren't dropped before PF,
>       and what to add to PF so that they flow.
>
> In other words, I'm stuck, and need some pointers on how to continue and what
> am I doing wrong. I'm running latest snapshot, but also tried with 6.2.
>
> Many thanks in advance.
>
> Here's the info about my config, let me know if you need me to provide some
> more. The "internet" networks are from RFC5737 for illustration purposes.
>
> 1. Network diagram
>
>    +---------+               +---------+
>    |  ISP_A  |               |  ISP_B  |
>    +---+-----+               +---+-----+
>        |                         |
>        |                         |
>        |                         |
> ++-----+-------------------------+----------------++
> ||  em1                      em2/pppoe0           ||
> ||  DHCP client              real IP              ||
> ||  IP: 198.51.100.20        IP: 203.0.113.40     ||
> ||  Net: 198.51.100.0/24     Net: 203.0.113.0/24  ||
> ||  GW: 198.51.100.1         GW: 203.0.113.1      ||
> ||  rdomain 1                rdomain 2            ||
> G|  group isp_a              group isp_b          |G
> A|                                                |A
> T|                                                |T
> E+- - - - - - - - - - - NAT- - - - - - - - - - - -+E
> W|                                                |W
> A|                                                |A
> Y|               em0                              |Y
> ||               DHCP server                      ||
> ||               IP: 172.16.16.1                  ||
> ||               Net: 172.16.16.0/24              ||
> ||               rdomain 0                        ||
> ||               group lan                        ||
> ++--------------------+---------------------------++
>                       |
>                       |
>                       |
>                    +--+--------+
>                    |    LAN    |
>                    +-----------+
>
>
> 2. Interface config files
>
> - /etc/hostname.em0
>
>     inet 172.16.16.1 255.255.255.0 172.16.16.255 group lan
>
> - /etc/hostname.em1
>
>     dhcp group isp_a rdomain 1
>
> - /etc/hostname.em2
>
>     up
>
> - /etc/hostname.pppoe0
>
>     inet 0.0.0.0 255.255.255.255 NONE \
>         pppoedev em2 authproto chap \
>         authname 'user' authkey 'verysecret' \
>         group isp_b \
>         rdomain 2 \
>         up
>     dest 0.0.0.1
>     !/sbin/route -T 2 add default -ifp pppoe0 0.0.0.1
>
>
> 3. DHCP server config (/etc/dhcpd.conf)
>
>     subnet 172.16.16.0 netmask 255.255.255.0 {
>         option domain-name-servers 172.16.16.2, 172.16.16.3;
>         option routers 172.16.16.1;
>         range 172.16.16.100 172.16.16.199;
>     }
>
>
> 4. PF config
>
>     # Need to figure out how avoid hardcoding these
>     isp_a_gw = "172.16.18.1"
>     isp_b_gw = "192.168.68.1"
>
>     set debug debug
>
>     match in log all scrub (no-df random-id max-mss 1440)
>
>     match out log on em1 from (lan:network) nat-to (em1)
>     match out log on pppoe0 from (lan:network) nat-to (pppoe0)
>
>     pass out log on lan to (lan:network)
>     pass in log quick on lan from (lan:network) to (lan)
>
>     pass in log on lan from (lan:network) \
>         route-to { (em1 $isp_a_gw), (pppoe0  $isp_b_gw) } \
>         round-robin
>
>     pass out log on em1 from pppoe0 route-to (pppoe0 $isp_b_gw)
>     pass out log on pppoe0 from em1 route-to (em1 $isp_a_gw)
>
>     pass out log quick on em1 inet from (em1) modulate state rtable 1
>     pass out log quick on pppoe0 from (pppoe0) modulate state rtable 2
>
>
> 5. Additional issues
>
> - How to avoid hardcoding the ISP defaut routes?
> - How to use sticky sessions instead of round-robin?
> - How to deal with links going down? E.g. not try to send traffic to a failed
>   link.
>
>
> --
> sw



--
sw

Reply | Threaded
Open this post in threaded view
|

Re: Dual-ISP home router setup problems

sven falempin
i just use route -T X exec dhclient in 6.3

AWESOME

On Mon, Mar 19, 2018 at 7:54 AM, Samuel Wagen <[hidden email]> wrote:

> And of course, too much copy paste while trying to use documentation
> IP ranges. The two gateways in pf.conf above should be
>
> isp_a_gw = "198.51.100.1"
> isp_b_gw = "203.0.113.1"
>
> The rest stands.
>
> On Mon, Mar 19, 2018 at 1:40 PM, Samuel Wagen <[hidden email]> wrote:
>> Hello,
>>
>> I'm trying to build a home router with OpenBSD. I have two ISPs, both are
>> giving me real IPs, one with straight DHCP (ISP_A), the other - via PPPoE
>> (ISP_B). I've described the topology with more detail in the diagram below.
>>
>> I wanted to use PF with routing domains instead of multipath forwarding, due
>> to multipath being very finicky when a link goes down. My current setup is
>> described below. I have the following issues:
>>
>>     - Initially I can't pass traffic from the LAN. I think this is due to the
>>       packets on em0 being dropped before PF has a chance to reach them, due
>>       to missing default route on rdomain 0. If I execute the following two
>>       commands:
>>             # route -T 0 add 198.51.100.0/24 127.0.0.1
>>             # route -T 0 add 203.0.113.0/24 127.0.0.1
>>       then traffic starts passing half of the time - if the round-robin
>>       decides it should go over the PPPoE link (ISP_B) - traffic from the LAN
>>       flows. If, however, it decides to go through the other link (ISP_A) -
>>       nothing passes, and I get the following kernel messages:
>>
>>             arpresolve: 198.51.100.0: route contains no arp information
>>
>>     - Traffic from the gateway itself to the Internet always fails, unless I
>>       specify a routing domain manually (route -T 1 exec whatever). Not sure
>>       what bogus route to add here, so that packets aren't dropped before PF,
>>       and what to add to PF so that they flow.
>>
>> In other words, I'm stuck, and need some pointers on how to continue and what
>> am I doing wrong. I'm running latest snapshot, but also tried with 6.2.
>>
>> Many thanks in advance.
>>
>> Here's the info about my config, let me know if you need me to provide some
>> more. The "internet" networks are from RFC5737 for illustration purposes.
>>
>> 1. Network diagram
>>
>>    +---------+               +---------+
>>    |  ISP_A  |               |  ISP_B  |
>>    +---+-----+               +---+-----+
>>        |                         |
>>        |                         |
>>        |                         |
>> ++-----+-------------------------+----------------++
>> ||  em1                      em2/pppoe0           ||
>> ||  DHCP client              real IP              ||
>> ||  IP: 198.51.100.20        IP: 203.0.113.40     ||
>> ||  Net: 198.51.100.0/24     Net: 203.0.113.0/24  ||
>> ||  GW: 198.51.100.1         GW: 203.0.113.1      ||
>> ||  rdomain 1                rdomain 2            ||
>> G|  group isp_a              group isp_b          |G
>> A|                                                |A
>> T|                                                |T
>> E+- - - - - - - - - - - NAT- - - - - - - - - - - -+E
>> W|                                                |W
>> A|                                                |A
>> Y|               em0                              |Y
>> ||               DHCP server                      ||
>> ||               IP: 172.16.16.1                  ||
>> ||               Net: 172.16.16.0/24              ||
>> ||               rdomain 0                        ||
>> ||               group lan                        ||
>> ++--------------------+---------------------------++
>>                       |
>>                       |
>>                       |
>>                    +--+--------+
>>                    |    LAN    |
>>                    +-----------+
>>
>>
>> 2. Interface config files
>>
>> - /etc/hostname.em0
>>
>>     inet 172.16.16.1 255.255.255.0 172.16.16.255 group lan
>>
>> - /etc/hostname.em1
>>
>>     dhcp group isp_a rdomain 1
>>
>> - /etc/hostname.em2
>>
>>     up
>>
>> - /etc/hostname.pppoe0
>>
>>     inet 0.0.0.0 255.255.255.255 NONE \
>>         pppoedev em2 authproto chap \
>>         authname 'user' authkey 'verysecret' \
>>         group isp_b \
>>         rdomain 2 \
>>         up
>>     dest 0.0.0.1
>>     !/sbin/route -T 2 add default -ifp pppoe0 0.0.0.1
>>
>>
>> 3. DHCP server config (/etc/dhcpd.conf)
>>
>>     subnet 172.16.16.0 netmask 255.255.255.0 {
>>         option domain-name-servers 172.16.16.2, 172.16.16.3;
>>         option routers 172.16.16.1;
>>         range 172.16.16.100 172.16.16.199;
>>     }
>>
>>
>> 4. PF config
>>
>>     # Need to figure out how avoid hardcoding these
>>     isp_a_gw = "172.16.18.1"
>>     isp_b_gw = "192.168.68.1"
>>
>>     set debug debug
>>
>>     match in log all scrub (no-df random-id max-mss 1440)
>>
>>     match out log on em1 from (lan:network) nat-to (em1)
>>     match out log on pppoe0 from (lan:network) nat-to (pppoe0)
>>
>>     pass out log on lan to (lan:network)
>>     pass in log quick on lan from (lan:network) to (lan)
>>
>>     pass in log on lan from (lan:network) \
>>         route-to { (em1 $isp_a_gw), (pppoe0  $isp_b_gw) } \
>>         round-robin
>>
>>     pass out log on em1 from pppoe0 route-to (pppoe0 $isp_b_gw)
>>     pass out log on pppoe0 from em1 route-to (em1 $isp_a_gw)
>>
>>     pass out log quick on em1 inet from (em1) modulate state rtable 1
>>     pass out log quick on pppoe0 from (pppoe0) modulate state rtable 2
>>
>>
>> 5. Additional issues
>>
>> - How to avoid hardcoding the ISP defaut routes?
>> - How to use sticky sessions instead of round-robin?
>> - How to deal with links going down? E.g. not try to send traffic to a failed
>>   link.
>>
>>
>> --
>> sw
>
>
>
> --
> sw
>



--
--
---------------------------------------------------------------------------------------------------------------------
Knowing is not enough; we must apply. Willing is not enough; we must do