Getting traffic from rdomain X to talk to a daemon in default rdomain 0

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Getting traffic from rdomain X to talk to a daemon in default rdomain 0

Jiri B-3
Hello,

I'm trying to isolate an app running on OpenBSD on network level and thus I
have started
the app in a specific rdomain.

I can successfully make traffic from the rdomain to reach Internet:

pass out quick on rdomain 1 to any nat-to (egress) rtable 0

But I cannot figure out how to make the app in this rdomain 1 to communicate
which daemons in default rdomain (0).

With above rule I would see something like this on lo0 (rdomain0):

Jan 31 16:04:22.285915 199.195.x.x.60666 > 199.195.x.x.53: 14874+ NS? .(17)

Tested with route -T 1 exec dig @199.195.x.x  www.openbsd.org.
It seems it does not know how to send back replies ?

Without 'nat-to (egress)' the replies would be just send via default gw in
rdomain 0:

mx1# tcpdump -i vio0 -n -e -ttt icmp
tcpdump: listening on vio0, link-type EN10MB
Jan 31 16:08:27.053592 00:16:a1:5d:50:b6 00:12:f2:f2:1a:00 0800 98:
199.195.x.x > 172.16.1.2: icmp: echo reply

(172.16.1.2 was the IP in rdomain 1)

Any idea what would be PF rule to make this working - ie. make an app in
rdomain X talk to daemons in rdomain 0.

I also tried to use pair interfaces but I failed too.

Jiri
Reply | Threaded
Open this post in threaded view
|

Re: Getting traffic from rdomain X to talk to a daemon in default rdomain 0

Sebastian Benoit
Jiri B([hidden email]) on 2019.01.31 22:23:34 +0100:
> Hello,
>
> I'm trying to isolate an app running on OpenBSD on network level and thus I
> have started
> the app in a specific rdomain.
>
> I can successfully make traffic from the rdomain to reach Internet:
>
> pass out quick on rdomain 1 to any nat-to (egress) rtable 0

that rule is only evaluated when the packes pass through a network
interface.

> But I cannot figure out how to make the app in this rdomain 1 to communicate
> which daemons in default rdomain (0).
>
> With above rule I would see something like this on lo0 (rdomain0):
>
> Jan 31 16:04:22.285915 199.195.x.x.60666 > 199.195.x.x.53: 14874+ NS? .(17)
>
> Tested with route -T 1 exec dig @199.195.x.x  www.openbsd.org.
> It seems it does not know how to send back replies ?

yes, because rdomain 0 does not have a route to what network you have in
rdomain 1.

Btw. its hard to talk about this without you giving the actual networks and
IPs used.
 

> Without 'nat-to (egress)' the replies would be just send via default gw in
> rdomain 0:
>
> mx1# tcpdump -i vio0 -n -e -ttt icmp
> tcpdump: listening on vio0, link-type EN10MB
> Jan 31 16:08:27.053592 00:16:a1:5d:50:b6 00:12:f2:f2:1a:00 0800 98:
> 199.195.x.x > 172.16.1.2: icmp: echo reply
>
> (172.16.1.2 was the IP in rdomain 1)
>
> Any idea what would be PF rule to make this working - ie. make an app in
> rdomain X talk to daemons in rdomain 0.
>
> I also tried to use pair interfaces but I failed too.

Try this:

# set up two connected pair interfaces:
ifconfig pair8 inet 192.168.2.8/24 rdomain 8
ifconfig pair1 inet 192.168.2.1/24 rdomain 0
ifconfig pair1 patch pair8

# they now can ping each other:
ping 192.168.2.8
route -T 8 exec ping 192.168.2.1

# my em0 interface in rdomain 0 has the IP 192.168.1.52:
em0: flags=208847<UP,BROADCAST,DEBUG,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6> mtu 1500
        lladdr 44:c6:86:5a:c2:f7
        index 1 priority 0 llprio 3
        groups: egress
        media: Ethernet autoselect
        status: active
        inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255

# add a route to 192.168.1.52 to rdomain 8:
route -T 8 add 192.168.1.52 192.168.2.1
route -T 8 exec ping 192.168.1.52

# the traffic back from rdomain 0 to rdomain 8 works now, because packets
# are send with source ip 192.168.2.8, and rdomain 0 has a route to that IP
# through pair1.

Now run your service on 192.168.1.52.

/Benno

Reply | Threaded
Open this post in threaded view
|

Re: Getting traffic from rdomain X to talk to a daemon in default rdomain 0

Jiri B-3
Thank you, that works fine.


Jiri


On Thu, Jan 31, 2019 at 11:26 PM Sebastian Benoit <[hidden email]> wrote:

>
> Jiri B([hidden email]) on 2019.01.31 22:23:34 +0100:
> > Hello,
> >
> > I'm trying to isolate an app running on OpenBSD on network level and thus I
> > have started
> > the app in a specific rdomain.
> >
> > I can successfully make traffic from the rdomain to reach Internet:
> >
> > pass out quick on rdomain 1 to any nat-to (egress) rtable 0
>
> that rule is only evaluated when the packes pass through a network
> interface.
>
> > But I cannot figure out how to make the app in this rdomain 1 to communicate
> > which daemons in default rdomain (0).
> >
> > With above rule I would see something like this on lo0 (rdomain0):
> >
> > Jan 31 16:04:22.285915 199.195.x.x.60666 > 199.195.x.x.53: 14874+ NS? .(17)
> >
> > Tested with route -T 1 exec dig @199.195.x.x  www.openbsd.org.
> > It seems it does not know how to send back replies ?
>
> yes, because rdomain 0 does not have a route to what network you have in
> rdomain 1.
>
> Btw. its hard to talk about this without you giving the actual networks and
> IPs used.
>
> > Without 'nat-to (egress)' the replies would be just send via default gw in
> > rdomain 0:
> >
> > mx1# tcpdump -i vio0 -n -e -ttt icmp
> > tcpdump: listening on vio0, link-type EN10MB
> > Jan 31 16:08:27.053592 00:16:a1:5d:50:b6 00:12:f2:f2:1a:00 0800 98:
> > 199.195.x.x > 172.16.1.2: icmp: echo reply
> >
> > (172.16.1.2 was the IP in rdomain 1)
> >
> > Any idea what would be PF rule to make this working - ie. make an app in
> > rdomain X talk to daemons in rdomain 0.
> >
> > I also tried to use pair interfaces but I failed too.
>
> Try this:
>
> # set up two connected pair interfaces:
> ifconfig pair8 inet 192.168.2.8/24 rdomain 8
> ifconfig pair1 inet 192.168.2.1/24 rdomain 0
> ifconfig pair1 patch pair8
>
> # they now can ping each other:
> ping 192.168.2.8
> route -T 8 exec ping 192.168.2.1
>
> # my em0 interface in rdomain 0 has the IP 192.168.1.52:
> em0: flags=208847<UP,BROADCAST,DEBUG,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6> mtu 1500
>         lladdr 44:c6:86:5a:c2:f7
>         index 1 priority 0 llprio 3
>         groups: egress
>         media: Ethernet autoselect
>         status: active
>         inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255
>
> # add a route to 192.168.1.52 to rdomain 8:
> route -T 8 add 192.168.1.52 192.168.2.1
> route -T 8 exec ping 192.168.1.52
>
> # the traffic back from rdomain 0 to rdomain 8 works now, because packets
> # are send with source ip 192.168.2.8, and rdomain 0 has a route to that IP
> # through pair1.
>
> Now run your service on 192.168.1.52.
>
> /Benno

Reply | Threaded
Open this post in threaded view
|

Re: Getting traffic from rdomain X to talk to a daemon in default rdomain 0

zach
Apologies for hijacking but how can I redirect other hosts to a pair in PF? I've tried adding a route manually to the hosts via the gateway and adding a rdr-to in PF but I can't reach any hosts in the second rdomain, only the gateway.

Cheers,
Zach

On February 3, 2019 7:20:28 AM GMT+10:00, Jiri B <[hidden email]> wrote:

>Thank you, that works fine.
>
>
>Jiri
>
>
>On Thu, Jan 31, 2019 at 11:26 PM Sebastian Benoit
><[hidden email]> wrote:
>>
>> Jiri B([hidden email]) on 2019.01.31 22:23:34 +0100:
>> > Hello,
>> >
>> > I'm trying to isolate an app running on OpenBSD on network level
>and thus I
>> > have started
>> > the app in a specific rdomain.
>> >
>> > I can successfully make traffic from the rdomain to reach Internet:
>> >
>> > pass out quick on rdomain 1 to any nat-to (egress) rtable 0
>>
>> that rule is only evaluated when the packes pass through a network
>> interface.
>>
>> > But I cannot figure out how to make the app in this rdomain 1 to
>communicate
>> > which daemons in default rdomain (0).
>> >
>> > With above rule I would see something like this on lo0 (rdomain0):
>> >
>> > Jan 31 16:04:22.285915 199.195.x.x.60666 > 199.195.x.x.53: 14874+
>NS? .(17)
>> >
>> > Tested with route -T 1 exec dig @199.195.x.x  www.openbsd.org.
>> > It seems it does not know how to send back replies ?
>>
>> yes, because rdomain 0 does not have a route to what network you have
>in
>> rdomain 1.
>>
>> Btw. its hard to talk about this without you giving the actual
>networks and
>> IPs used.
>>
>> > Without 'nat-to (egress)' the replies would be just send via
>default gw in
>> > rdomain 0:
>> >
>> > mx1# tcpdump -i vio0 -n -e -ttt icmp
>> > tcpdump: listening on vio0, link-type EN10MB
>> > Jan 31 16:08:27.053592 00:16:a1:5d:50:b6 00:12:f2:f2:1a:00 0800 98:
>> > 199.195.x.x > 172.16.1.2: icmp: echo reply
>> >
>> > (172.16.1.2 was the IP in rdomain 1)
>> >
>> > Any idea what would be PF rule to make this working - ie. make an
>app in
>> > rdomain X talk to daemons in rdomain 0.
>> >
>> > I also tried to use pair interfaces but I failed too.
>>
>> Try this:
>>
>> # set up two connected pair interfaces:
>> ifconfig pair8 inet 192.168.2.8/24 rdomain 8
>> ifconfig pair1 inet 192.168.2.1/24 rdomain 0
>> ifconfig pair1 patch pair8
>>
>> # they now can ping each other:
>> ping 192.168.2.8
>> route -T 8 exec ping 192.168.2.1
>>
>> # my em0 interface in rdomain 0 has the IP 192.168.1.52:
>> em0:
>flags=208847<UP,BROADCAST,DEBUG,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6>
>mtu 1500
>>         lladdr 44:c6:86:5a:c2:f7
>>         index 1 priority 0 llprio 3
>>         groups: egress
>>         media: Ethernet autoselect
>>         status: active
>>         inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255
>>
>> # add a route to 192.168.1.52 to rdomain 8:
>> route -T 8 add 192.168.1.52 192.168.2.1
>> route -T 8 exec ping 192.168.1.52
>>
>> # the traffic back from rdomain 0 to rdomain 8 works now, because
>packets
>> # are send with source ip 192.168.2.8, and rdomain 0 has a route to
>that IP
>> # through pair1.
>>
>> Now run your service on 192.168.1.52.
>>
>> /Benno

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Reply | Threaded
Open this post in threaded view
|

Re: Getting traffic from rdomain X to talk to a daemon in default rdomain 0

Sebastian Benoit
Zach Nedwich([hidden email]) on 2019.02.03 07:54:23 +1000:
> Apologies for hijacking but how can I redirect other hosts to a pair in
> PF? I've tried adding a route manually to the hosts via the gateway and
> adding a rdr-to in PF but I can't reach any hosts in the second rdomain,
> only the gateway.

well, i guess you might be missing a route.

But you are not giving any details, just a vague description of your
problem. If i want to read about puzzles, i prefer reading Ready Player One,
not openbsd-misc.

/B


>
> Cheers,
> Zach
>
> On February 3, 2019 7:20:28 AM GMT+10:00, Jiri B <[hidden email]> wrote:
> >Thank you, that works fine.
> >
> >
> >Jiri
> >
> >
> >On Thu, Jan 31, 2019 at 11:26 PM Sebastian Benoit
> ><[hidden email]> wrote:
> >>
> >> Jiri B([hidden email]) on 2019.01.31 22:23:34 +0100:
> >> > Hello,
> >> >
> >> > I'm trying to isolate an app running on OpenBSD on network level
> >and thus I
> >> > have started
> >> > the app in a specific rdomain.
> >> >
> >> > I can successfully make traffic from the rdomain to reach Internet:
> >> >
> >> > pass out quick on rdomain 1 to any nat-to (egress) rtable 0
> >>
> >> that rule is only evaluated when the packes pass through a network
> >> interface.
> >>
> >> > But I cannot figure out how to make the app in this rdomain 1 to
> >communicate
> >> > which daemons in default rdomain (0).
> >> >
> >> > With above rule I would see something like this on lo0 (rdomain0):
> >> >
> >> > Jan 31 16:04:22.285915 199.195.x.x.60666 > 199.195.x.x.53: 14874+
> >NS? .(17)
> >> >
> >> > Tested with route -T 1 exec dig @199.195.x.x  www.openbsd.org.
> >> > It seems it does not know how to send back replies ?
> >>
> >> yes, because rdomain 0 does not have a route to what network you have
> >in
> >> rdomain 1.
> >>
> >> Btw. its hard to talk about this without you giving the actual
> >networks and
> >> IPs used.
> >>
> >> > Without 'nat-to (egress)' the replies would be just send via
> >default gw in
> >> > rdomain 0:
> >> >
> >> > mx1# tcpdump -i vio0 -n -e -ttt icmp
> >> > tcpdump: listening on vio0, link-type EN10MB
> >> > Jan 31 16:08:27.053592 00:16:a1:5d:50:b6 00:12:f2:f2:1a:00 0800 98:
> >> > 199.195.x.x > 172.16.1.2: icmp: echo reply
> >> >
> >> > (172.16.1.2 was the IP in rdomain 1)
> >> >
> >> > Any idea what would be PF rule to make this working - ie. make an
> >app in
> >> > rdomain X talk to daemons in rdomain 0.
> >> >
> >> > I also tried to use pair interfaces but I failed too.
> >>
> >> Try this:
> >>
> >> # set up two connected pair interfaces:
> >> ifconfig pair8 inet 192.168.2.8/24 rdomain 8
> >> ifconfig pair1 inet 192.168.2.1/24 rdomain 0
> >> ifconfig pair1 patch pair8
> >>
> >> # they now can ping each other:
> >> ping 192.168.2.8
> >> route -T 8 exec ping 192.168.2.1
> >>
> >> # my em0 interface in rdomain 0 has the IP 192.168.1.52:
> >> em0:
> >flags=208847<UP,BROADCAST,DEBUG,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6>
> >mtu 1500
> >>         lladdr 44:c6:86:5a:c2:f7
> >>         index 1 priority 0 llprio 3
> >>         groups: egress
> >>         media: Ethernet autoselect
> >>         status: active
> >>         inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255
> >>
> >> # add a route to 192.168.1.52 to rdomain 8:
> >> route -T 8 add 192.168.1.52 192.168.2.1
> >> route -T 8 exec ping 192.168.1.52
> >>
> >> # the traffic back from rdomain 0 to rdomain 8 works now, because
> >packets
> >> # are send with source ip 192.168.2.8, and rdomain 0 has a route to
> >that IP
> >> # through pair1.
> >>
> >> Now run your service on 192.168.1.52.
> >>
> >> /Benno
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>

--

Reply | Threaded
Open this post in threaded view
|

Re: Getting traffic from rdomain X to talk to a daemon in default rdomain 0

zach
Sorry, you're right, I should have included more detail. I've
expiremented a bit more but my solution still isn't 100% functional.

My issue is that I cannot connect to any services e.g. ssh on any hosts
in rdomain2. Ping "works" but the ICMP response is directly from the
router. I think the issue lies in my PF rules below...


router:

# interfaces

#trunk0 - this is my "main" network
rdomain 0
inet 192.168.0.1 255.255.255.0

#vlan130 - this is where i want to access
rdomain 2
inet 192.168.2.1 255.255.255.0 192.168.2.255 vnetid 130 parent trunk0

#pair0 - this is my "main" network pair
rdomain 0
inet 192.168.8.1 255.255.255.0 192.168.8.255
patch pair2
!/sbin/route -T0 add 192.168.2.1/24 192.168.8.2

#pair2 - this is my "rdomain 2" pair
rdomain 2
inet 192.168.8.2 255.255.255.0 192.168.8.255

# route -n show -inet
# Routing tables

Destination        Gateway            Flags   Refs      Use   Mtu  Prio Iface
127/8              127.0.0.1          UGRS       0        0 32768     8 lo0
127.0.0.1          127.0.0.1          UHhl       1        2 32768     1 lo0
192.168.0/24       192.168.0.1        UCn        5      258     -     4 trunk0
192.168.0.1        00:0d:b9:49:b1:10  UHLl       0     8017     -     1 trunk0
192.168.2/24       192.168.8.2        UGS        0        9     -     8 pair0
192.168.8/24       192.168.8.1        UCn        1        0     -     4 pair0
192.168.8.1        fe:e1:ba:d0:f4:c2  UHLl       0        4     -     1 pair0
192.168.8.2        fe:e1:ba:d1:eb:7a  UHLch      1        7     -     3 pair0
192.168.8.255      192.168.8.1        UHb        0        0     -     1 pair0

# pf (important bits)
lan = "trunk0"
guest = "vlan130"
wan ="pppoe0"
wg0 = "tun1"
pair0 = "pair0"
pair2 = "pair2"

pass on $guest rtable 2
pass on $pair2 rtable 2

# block from guest vlan, but redirect internal to pair adapter
block from $guest:network to $lan:network
pass in from $lan:network to $guest:network rdr-to $guest rtable 2


host1:

# route table
default via 192.168.0.1 dev enp0s31f6 onlink
192.168.2.0/24 via 192.168.0.1 dev enp0s31f6


# Partially working example
router:
  route -T2 exec nc -l 4444
  ehlo


host1:
  nc -vvv 192.168.2.1 4444
  Connection to 192.168.2.1 4444 port [tcp/krb524] succeeded!
  ehlo

So I can connect to the router gateway in rdomain 2 but none of the
hosts in rdomain 2

# Example with another host in rdomain 2

# On the router, all good
ssh zach@192.168.2.2 -p 8022
zach@192.168.2.2's password:

# On host1 in rdomain 0
ssh zach@192.168.2.2 -p 8022
ssh: connect to host 192.168.2.2 port 8022: Connection refused

router# tcpdump -i trunk0 'port 8022'
tcpdump: listening on trunk0, link-type EN10MB
10:33:58.082715 192.168.0.110.39518 > 192.168.2.2.8022: S 4146219697:4146219697(0) win 64240 <mss 1460,sackOK,timestamp 793018757 0,nop,wscale 7> (DF)
10:33:58.082821 192.168.2.2.8022 > 192.168.0.110.39518: R 0:0(0) ack 4146219698 win 0 (DF)

Cheers,
Zach

On Sun, Feb 03, 2019 at 12:29:08AM +0100, Sebastian Benoit wrote:

> Zach Nedwich([hidden email]) on 2019.02.03 07:54:23 +1000:
> > Apologies for hijacking but how can I redirect other hosts to a pair in
> > PF? I've tried adding a route manually to the hosts via the gateway and
> > adding a rdr-to in PF but I can't reach any hosts in the second rdomain,
> > only the gateway.
>
> well, i guess you might be missing a route.
>
> But you are not giving any details, just a vague description of your
> problem. If i want to read about puzzles, i prefer reading Ready Player One,
> not openbsd-misc.
>
> /B
>
>
> >
> > Cheers,
> > Zach
> >
> > On February 3, 2019 7:20:28 AM GMT+10:00, Jiri B <[hidden email]> wrote:
> > >Thank you, that works fine.
> > >
> > >
> > >Jiri
> > >
> > >
> > >On Thu, Jan 31, 2019 at 11:26 PM Sebastian Benoit
> > ><[hidden email]> wrote:
> > >>
> > >> Jiri B([hidden email]) on 2019.01.31 22:23:34 +0100:
> > >> > Hello,
> > >> >
> > >> > I'm trying to isolate an app running on OpenBSD on network level
> > >and thus I
> > >> > have started
> > >> > the app in a specific rdomain.
> > >> >
> > >> > I can successfully make traffic from the rdomain to reach Internet:
> > >> >
> > >> > pass out quick on rdomain 1 to any nat-to (egress) rtable 0
> > >>
> > >> that rule is only evaluated when the packes pass through a network
> > >> interface.
> > >>
> > >> > But I cannot figure out how to make the app in this rdomain 1 to
> > >communicate
> > >> > which daemons in default rdomain (0).
> > >> >
> > >> > With above rule I would see something like this on lo0 (rdomain0):
> > >> >
> > >> > Jan 31 16:04:22.285915 199.195.x.x.60666 > 199.195.x.x.53: 14874+
> > >NS? .(17)
> > >> >
> > >> > Tested with route -T 1 exec dig @199.195.x.x  www.openbsd.org.
> > >> > It seems it does not know how to send back replies ?
> > >>
> > >> yes, because rdomain 0 does not have a route to what network you have
> > >in
> > >> rdomain 1.
> > >>
> > >> Btw. its hard to talk about this without you giving the actual
> > >networks and
> > >> IPs used.
> > >>
> > >> > Without 'nat-to (egress)' the replies would be just send via
> > >default gw in
> > >> > rdomain 0:
> > >> >
> > >> > mx1# tcpdump -i vio0 -n -e -ttt icmp
> > >> > tcpdump: listening on vio0, link-type EN10MB
> > >> > Jan 31 16:08:27.053592 00:16:a1:5d:50:b6 00:12:f2:f2:1a:00 0800 98:
> > >> > 199.195.x.x > 172.16.1.2: icmp: echo reply
> > >> >
> > >> > (172.16.1.2 was the IP in rdomain 1)
> > >> >
> > >> > Any idea what would be PF rule to make this working - ie. make an
> > >app in
> > >> > rdomain X talk to daemons in rdomain 0.
> > >> >
> > >> > I also tried to use pair interfaces but I failed too.
> > >>
> > >> Try this:
> > >>
> > >> # set up two connected pair interfaces:
> > >> ifconfig pair8 inet 192.168.2.8/24 rdomain 8
> > >> ifconfig pair1 inet 192.168.2.1/24 rdomain 0
> > >> ifconfig pair1 patch pair8
> > >>
> > >> # they now can ping each other:
> > >> ping 192.168.2.8
> > >> route -T 8 exec ping 192.168.2.1
> > >>
> > >> # my em0 interface in rdomain 0 has the IP 192.168.1.52:
> > >> em0:
> > >flags=208847<UP,BROADCAST,DEBUG,RUNNING,SIMPLEX,MULTICAST,AUTOCONF6>
> > >mtu 1500
> > >>         lladdr 44:c6:86:5a:c2:f7
> > >>         index 1 priority 0 llprio 3
> > >>         groups: egress
> > >>         media: Ethernet autoselect
> > >>         status: active
> > >>         inet 192.168.1.52 netmask 0xffffff00 broadcast 192.168.1.255
> > >>
> > >> # add a route to 192.168.1.52 to rdomain 8:
> > >> route -T 8 add 192.168.1.52 192.168.2.1
> > >> route -T 8 exec ping 192.168.1.52
> > >>
> > >> # the traffic back from rdomain 0 to rdomain 8 works now, because
> > >packets
> > >> # are send with source ip 192.168.2.8, and rdomain 0 has a route to
> > >that IP
> > >> # through pair1.
> > >>
> > >> Now run your service on 192.168.1.52.
> > >>
> > >> /Benno
> >
> > --
> > Sent from my Android device with K-9 Mail. Please excuse my brevity.
> >
>
> --
>