Proxy-arp behavior change between 5.9 and 6.0

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Proxy-arp behavior change between 5.9 and 6.0

Andrew Hewus Fresh
I am helping some folks with some OpenBSD stuff, including at some
point being more proactive about updating, but they currently have an
OpenBSD 5.9 machine that has been routing traffic happily for a while.
Unfortunately, they can't currently update because the proxy arp
configuration that they are using on 5.9 doesn't work on anything newer.


Here is the excellent description from someone there, and I was able to
reproduce the issue on my laptop, so I don't believe a dmesg will be
particularly helpful, but here's one for my laptop that's not terribly
old.

https://dmesgd.nycbug.org/index.cgi?do=view&id=5279

As I haven't ever actually used proxy arp on OpenBSD, so don't actually
know whether this was a glitch that was fixed or added.

I do see that 60.html says the routing table is now based on ART, so
that seems a possible culprit.

---


We've found that a proxy-arp configuration we've relied on (which works
perfectly on OpenBSD 5.9 and earlier) no longer works on OpenBSD 6.0 thru 6.7.

On OpenBSD 5.9, we can add an "arp -s" entry for a public IP on our
public-facing gateway router's external interface, followed by a host-route
for that same public IP (pointing to an internal subnet/interface). This
allows us to route public IPs to clients through our internal private subnets,
so a client can have a public IP assigned on their router, despite actually
being several hops deep inside our private network.

With this proxy-arp configuration, from both the client router's perspective
and from an external perspective, it looks as if the client router is in the
public IP subnet.

Unfortunately this method doesn't work in OpenBSD 6.0 and later.


Here's how it looks when you configure this on OpenBSD 5.9
(working as expected with no errors):

        root@openbsd-59-pub-gw-rtr~ # arp -s 1.2.3.4 [PUB-GW_em0_EXT-IF_MAC-ADDR] pub
        root@openbsd-59-pub-gw-rtr~ # route add 1.2.3.4/32 10.0.2.47
        add host 1.2.3.4/32: gateway 10.0.2.47
        root@openbsd-59-pub-gw-rtr~ # netstat -rn -finet|grep 1.2.3.4
        1.2.3.4        10.0.2.47        UGHS       0        0 -     8 em1
        1.2.3.4/32     [PUB-GW_em0_EXT-IF_MAC-ADDR]  ULS2 0        0     -     8 em0
     ##### ^ Success: Routing table shows both the host-route and the "arp -s" entry.


That same setup fails on OpenBSD 6.0 and later
(tested on OpenBSD 6.0, 6.2, 6.6, and 6.7).


If you add an arp -s entry before adding a host route for the same IP, the
"route add" fails with "File exists" error:

        root@openbsd-67-pub-gw-rtr~ # arp -s 1.2.3.4 [PUB-GW_em0_EXT-IF_MAC-ADDR] pub
        root@openbsd-67-pub-gw-rtr~ # route add 1.2.3.4/32 10.0.2.47
    !-> add host 1.2.3.4/32: gateway 10.0.2.47: File exists
        root@openbsd-67-pub-gw-rtr~ # netstat -rn -finet|grep 1.2.3.4
        1.2.3.4/32     [PUB-GW_em0_EXT-IF_MAC-ADDR]  ULS2 0        0     -     8 em0
     ##### ^ Failure: Routing table only shows the "arp -s" entry (failed to add host-route)


If you add the route before the arp -s entry, "arp -s" fails with error
"set: proxy entry exists for non 802 device":

        root@openbsd-67-pub-gw-rtr~ # route add 1.2.3.4/32 10.0.2.47
        add host 1.2.3.4/32: gateway 10.0.2.47
        root@openbsd-67-pub-gw-rtr~ # arp -s 1.2.3.4 [PUB-GW_em0_EXT-IF_MAC-ADDR] pub
    !-> set: proxy entry exists for non 802 device
        root@openbsd-67-pub-gw-rtr~ # netstat -rn -finet|grep 1.2.3.4
        1.2.3.4        10.0.2.47         UGHS       0        0 -     8 em1
     ##### ^ Failure: Routing table only shows the host-route (failed to add arp -s entry)


For most of our clients who request a dedicated public IP, we're able to
assign their router a private IP, which we then bi-NAT to a dedicated public
IP on our gateway router. Unfortunately the bi-NAT method doesn't work for
certain routers' built-in VPNs, so we're looking for a way to re-create the
functionality we had with proxy-arp on OpenBSD v5.9. I imagine there's some
workaround or alternate method to achieve the same thing on OpenBSD 6.x, but I
haven't found it yet.

Reply | Threaded
Open this post in threaded view
|

Re: Proxy-arp behavior change between 5.9 and 6.0

Demi M. Obenour
On 2020-07-29 21:47, Andrew Hewus Fresh wrote:

> I am helping some folks with some OpenBSD stuff, including at some
> point being more proactive about updating, but they currently have an
> OpenBSD 5.9 machine that has been routing traffic happily for a while.
> Unfortunately, they can't currently update because the proxy arp
> configuration that they are using on 5.9 doesn't work on anything newer.
>
>
> Here is the excellent description from someone there, and I was able to
> reproduce the issue on my laptop, so I don't believe a dmesg will be
> particularly helpful, but here's one for my laptop that's not terribly
> old.
>
> https://dmesgd.nycbug.org/index.cgi?do=view&id=5279
>
> As I haven't ever actually used proxy arp on OpenBSD, so don't actually
> know whether this was a glitch that was fixed or added.
>
> I do see that 60.html says the routing table is now based on ART, so
> that seems a possible culprit.
>
> ---
>
>
> We've found that a proxy-arp configuration we've relied on (which works
> perfectly on OpenBSD 5.9 and earlier) no longer works on OpenBSD 6.0 thru 6.7.
>
> On OpenBSD 5.9, we can add an "arp -s" entry for a public IP on our
> public-facing gateway router's external interface, followed by a host-route
> for that same public IP (pointing to an internal subnet/interface). This
> allows us to route public IPs to clients through our internal private subnets,
> so a client can have a public IP assigned on their router, despite actually
> being several hops deep inside our private network.
>
> With this proxy-arp configuration, from both the client router's perspective
> and from an external perspective, it looks as if the client router is in the
> public IP subnet.
>
> Unfortunately this method doesn't work in OpenBSD 6.0 and later.
>
>
> Here's how it looks when you configure this on OpenBSD 5.9
> (working as expected with no errors):
>
>         root@openbsd-59-pub-gw-rtr~ # arp -s 1.2.3.4 [PUB-GW_em0_EXT-IF_MAC-ADDR] pub
>         root@openbsd-59-pub-gw-rtr~ # route add 1.2.3.4/32 10.0.2.47
>         add host 1.2.3.4/32: gateway 10.0.2.47
>         root@openbsd-59-pub-gw-rtr~ # netstat -rn -finet|grep 1.2.3.4
>         1.2.3.4        10.0.2.47        UGHS       0        0 -     8 em1
>         1.2.3.4/32     [PUB-GW_em0_EXT-IF_MAC-ADDR]  ULS2 0        0     -     8 em0
>      ##### ^ Success: Routing table shows both the host-route and the "arp -s" entry.
>
>
> That same setup fails on OpenBSD 6.0 and later
> (tested on OpenBSD 6.0, 6.2, 6.6, and 6.7).
>
>
> If you add an arp -s entry before adding a host route for the same IP, the
> "route add" fails with "File exists" error:
>
>         root@openbsd-67-pub-gw-rtr~ # arp -s 1.2.3.4 [PUB-GW_em0_EXT-IF_MAC-ADDR] pub
>         root@openbsd-67-pub-gw-rtr~ # route add 1.2.3.4/32 10.0.2.47
>     !-> add host 1.2.3.4/32: gateway 10.0.2.47: File exists
>         root@openbsd-67-pub-gw-rtr~ # netstat -rn -finet|grep 1.2.3.4
>         1.2.3.4/32     [PUB-GW_em0_EXT-IF_MAC-ADDR]  ULS2 0        0     -     8 em0
>      ##### ^ Failure: Routing table only shows the "arp -s" entry (failed to add host-route)
Does `route change` do anything different?

> If you add the route before the arp -s entry, "arp -s" fails with error
> "set: proxy entry exists for non 802 device":
>
>         root@openbsd-67-pub-gw-rtr~ # route add 1.2.3.4/32 10.0.2.47
>         add host 1.2.3.4/32: gateway 10.0.2.47
>         root@openbsd-67-pub-gw-rtr~ # arp -s 1.2.3.4 [PUB-GW_em0_EXT-IF_MAC-ADDR] pub
>     !-> set: proxy entry exists for non 802 device
>         root@openbsd-67-pub-gw-rtr~ # netstat -rn -finet|grep 1.2.3.4
>         1.2.3.4        10.0.2.47         UGHS       0        0 -     8 em1
>      ##### ^ Failure: Routing table only shows the host-route (failed to add arp -s entry)
What happens if you delete the check in arp.c that produces the
"proxy entry exists for non 802 device" message, then recompile arp(8)?

> For most of our clients who request a dedicated public IP, we're able to
> assign their router a private IP, which we then bi-NAT to a dedicated public
> IP on our gateway router. Unfortunately the bi-NAT method doesn't work for
> certain routers' built-in VPNs, so we're looking for a way to re-create the
> functionality we had with proxy-arp on OpenBSD v5.9. I imagine there's some
> workaround or alternate method to achieve the same thing on OpenBSD 6.x, but I
> haven't found it yet.

You might want to try using multiple routing tables, and using pf(4)
to shuttle traffic back and forth.  Different routing tables would
avoid this clash.

Sincerely,

Demi




signature.asc (849 bytes) Download Attachment