Re: Kerberos SSH routing tables problem

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Kerberos SSH routing tables problem

Predrag Punosevac-2
On 2019-08-02, Stuart Henderson wrote:

> On 2019-07-29, Predrag Punosevac <[hidden email]> wrote:
> > Hi Misc,
> >
> > I am using Edgerouter lite as a firewall/DNS cashing resolver for one of
> > our remote location
> >
> > ubnt1# uname -mrsv
> > OpenBSD 6.5 GENERIC.MP#0 octeon
> >
> > The desktops behind the firewall have to use Kerberised SSH to perform
> > some work on one of .mil servers. I opened egress ports kerberos,
> > klogin, kshell TCP protocol as well as kerberos UDP. After the work is
> > finished and desktops are "logged out" routing tables (dns) are in a bad
> > state on the firewall. A simple
> >
> > pfctl -F all -f /etc/pf.conf
> >
> > fixes the problem and desktops can again do DNS resolving and surfing
> > the Internet.
> >
> > Could somebody give me a head start how to go about further trouble
> > shooting and fixing the problem? Obviously flashing states is not very
> > convenient.
> >
> > Most Kind Regards,
> > Predrag Punosevac
> >
> >
>

Hi Sthen,

I apologize for long silence. I got busy with other stuff and this got
put onto the back burner.

> Can you go into some more details about what the "bad state" is?
>

Please forgive me for repating some things from my original email. After
rereading it I realized that I sounded like a mentally disturb
individual.

As I mentioned in my original email my folks are using Kerberised SSH to
log into some .mil computing nodes. OpenBSD 6.5 release running of
Edgerouter lite provides PF/Unbound DNS caching resolver to our office
computers. In order to reach .mil server I opened in egress direction
bunch of kerberos related ports. Thing worked like a charm. Then one day
somebody start complaining that after the Kerberized ssh session with
.mil server is closed they no longer can connect to the Internet.
Original problem report was complaining about non reachable DNS resolver
on my firewall but after further investigation I tracked down problem to
state of PF tables.

To make things more convoluted I discovered that actually problem was
not caused per se by Kerberized SSH session but a .mil homemade
application called ilauncher session. iluncher provides a web interface
similar to CUPS http://localhost:631 which is used to manage/connect
access to bunch of scientific related software packages (Jupyter
Notebooks, RStudio, etc) in a "user friendly" way. After the ilauncher
session is finished my firewall refuses any network connection to the
workstation from which ilauncher was run. The only remedy is to clear
all tables. Note that I always see that three tables are cleared when I
do

pfctl -F Tables

After that everything works as expected. Right now

ubnt1# pfctl -s Tables

lists no tables while on my home network with similar firewall rules I
see three tables.

minix# pfctl -s Tables  
__automatic_1c228804_0
bruteforce
sshguard


This is our office pf.conf file. I am planning to reproduce one more
time problem but this time to start flashing tables one by one until I
see which one is related to that ilauncher thing.


ext_if="cnmac0"
int_if="cnmac1"
dmz_if="cnmac2"
lan_net = "{192.168.1.0/24}"

broken = "{224.0.0.22, 127.0.0.0/8, 172.16.0.0/12, \
          10.0.0.0/8,  169.254.0.0/16, 192.0.2.0/24, 192.168.100.0/24 \
                198.51.100.0/24, 203.0.113.0/24, \
                0.0.0.0/8,   240.0.0.0/4, 255.255.255.255/32}"
table <bruteforce> persist
table <sshguard> persist


tcp_services = "{ssh, submission, imaps, http, https, 30041, 8080, \
                kerberos, klogin, kshell}"
udp_services= "{domain, ntp, kerberos}"


set block-policy return
set limit states 100000
set loginterface $ext_if
set optimization normal
set ruleset-optimization basic
set skip on lo
set state-policy floating
set timeout interval 10
set timeout frag 30
set timeout src.track 0
set state-defaults pflow


match in all scrub (no-df max-mss 1440)
match out on egress inet from !(egress:network) to any nat-to (egress:0)


block quick from <bruteforce>
block in quick on $ext_if proto tcp from <sshguard> \
        to any port ssh label "ssh bruteforce"
antispoof quick for { $int_if $ext_if }


block drop in quick on egress from {$broken, no-route} to any
block drop in quick from urpf-failed to any
block drop in quick on ! lo0 proto tcp to port 6000:6010

block all


pass inet proto icmp all icmp-type { echoreq, unreach }
pass out on $ext_if inet proto udp to any port $udp_services
pass out on $ext_if inet proto tcp to any port $tcp_services

pass log on $ext_if inet proto tcp from any to any port {ssh} \
        flags S/SA keep state \
            (max-src-conn 100, max-src-conn-rate 15/5, \
             overload <bruteforce> flush global)

pass inet proto tcp from {$lan_net} to any port $tcp_services
pass inet proto udp from {$lan_net} to any port $udp_services





> "routing tables (dns) are in a bad state on the firewall" doesn't
> explain much (and doesn't really make sense, dns has nothing to do with
> routing tables..)


I am not much of a network guy but I have seen fair share of crazy
things. However I am stamped by this. Could you tell me what is this
__automatic_1c228804_0 table? I am guessing this is from $broken IPs.
Any other hints how to troubleshoot this problem. Could you give me an
example of a scenario in which an IP is automatically added to a table
due to the applications run on the client?


Most Kind Regards,
Predrag

Reply | Threaded
Open this post in threaded view
|

Re: Kerberos SSH routing tables problem

Stuart Henderson
The __automatic table is created by pfctl's ruleset optimiser, probably
from $broken.

I suspect this ilauncher program is tripping either sshguard or your
max-src-conn-rate or max-src-conn rules.


--
Sent from a phone, apologies for poor formatting.

On 6 September 2019 03:57:04 Predrag Punosevac <[hidden email]> wrote:

> On 2019-08-02, Stuart Henderson wrote:
>
>> On 2019-07-29, Predrag Punosevac <[hidden email]> wrote:
>>> Hi Misc,
>>>
>>>
>>> I am using Edgerouter lite as a firewall/DNS cashing resolver for one of
>>> our remote location
>>>
>>>
>>> ubnt1# uname -mrsv
>>> OpenBSD 6.5 GENERIC.MP#0 octeon
>>>
>>>
>>> The desktops behind the firewall have to use Kerberised SSH to perform
>>> some work on one of .mil servers. I opened egress ports kerberos,
>>> klogin, kshell TCP protocol as well as kerberos UDP. After the work is
>>> finished and desktops are "logged out" routing tables (dns) are in a bad
>>> state on the firewall. A simple
>>>
>>>
>>> pfctl -F all -f /etc/pf.conf
>>>
>>>
>>> fixes the problem and desktops can again do DNS resolving and surfing
>>> the Internet.
>>>
>>>
>>> Could somebody give me a head start how to go about further trouble
>>> shooting and fixing the problem? Obviously flashing states is not very
>>> convenient.
>>>
>>>
>>> Most Kind Regards,
>>> Predrag Punosevac
>
> Hi Sthen,
>
> I apologize for long silence. I got busy with other stuff and this got
> put onto the back burner.
>
>> Can you go into some more details about what the "bad state" is?
>
> Please forgive me for repating some things from my original email. After
> rereading it I realized that I sounded like a mentally disturb
> individual.
>
> As I mentioned in my original email my folks are using Kerberised SSH to
> log into some .mil computing nodes. OpenBSD 6.5 release running of
> Edgerouter lite provides PF/Unbound DNS caching resolver to our office
> computers. In order to reach .mil server I opened in egress direction
> bunch of kerberos related ports. Thing worked like a charm. Then one day
> somebody start complaining that after the Kerberized ssh session with
> .mil server is closed they no longer can connect to the Internet.
> Original problem report was complaining about non reachable DNS resolver
> on my firewall but after further investigation I tracked down problem to
> state of PF tables.
>
> To make things more convoluted I discovered that actually problem was
> not caused per se by Kerberized SSH session but a .mil homemade
> application called ilauncher session. iluncher provides a web interface
> similar to CUPS http://localhost:631 which is used to manage/connect
> access to bunch of scientific related software packages (Jupyter
> Notebooks, RStudio, etc) in a "user friendly" way. After the ilauncher
> session is finished my firewall refuses any network connection to the
> workstation from which ilauncher was run. The only remedy is to clear
> all tables. Note that I always see that three tables are cleared when I
> do
>
> pfctl -F Tables
>
> After that everything works as expected. Right now
>
> ubnt1# pfctl -s Tables
>
> lists no tables while on my home network with similar firewall rules I
> see three tables.
>
> minix# pfctl -s Tables
> __automatic_1c228804_0
> bruteforce
> sshguard
>
>
> This is our office pf.conf file. I am planning to reproduce one more
> time problem but this time to start flashing tables one by one until I
> see which one is related to that ilauncher thing.
>
>
> ext_if="cnmac0"
> int_if="cnmac1"
> dmz_if="cnmac2"
> lan_net = "{192.168.1.0/24}"
>
> broken = "{224.0.0.22, 127.0.0.0/8, 172.16.0.0/12, \
>          10.0.0.0/8,  169.254.0.0/16, 192.0.2.0/24, 192.168.100.0/24 \
>                198.51.100.0/24, 203.0.113.0/24, \
>                0.0.0.0/8,   240.0.0.0/4, 255.255.255.255/32}"
> table <bruteforce> persist
> table <sshguard> persist
>
>
> tcp_services = "{ssh, submission, imaps, http, https, 30041, 8080, \
>                kerberos, klogin, kshell}"
> udp_services= "{domain, ntp, kerberos}"
>
>
> set block-policy return
> set limit states 100000
> set loginterface $ext_if
> set optimization normal
> set ruleset-optimization basic
> set skip on lo
> set state-policy floating
> set timeout interval 10
> set timeout frag 30
> set timeout src.track 0
> set state-defaults pflow
>
>
> match in all scrub (no-df max-mss 1440)
> match out on egress inet from !(egress:network) to any nat-to (egress:0)
>
>
> block quick from <bruteforce>
> block in quick on $ext_if proto tcp from <sshguard> \
>        to any port ssh label "ssh bruteforce"
> antispoof quick for { $int_if $ext_if }
>
>
> block drop in quick on egress from {$broken, no-route} to any
> block drop in quick from urpf-failed to any
> block drop in quick on ! lo0 proto tcp to port 6000:6010
>
> block all
>
>
> pass inet proto icmp all icmp-type { echoreq, unreach }
> pass out on $ext_if inet proto udp to any port $udp_services
> pass out on $ext_if inet proto tcp to any port $tcp_services
>
> pass log on $ext_if inet proto tcp from any to any port {ssh} \
>        flags S/SA keep state \
>            (max-src-conn 100, max-src-conn-rate 15/5, \
>             overload <bruteforce> flush global)
>
> pass inet proto tcp from {$lan_net} to any port $tcp_services
> pass inet proto udp from {$lan_net} to any port $udp_services
>
>
>
>
>
>> "routing tables (dns) are in a bad state on the firewall" doesn't
>> explain much (and doesn't really make sense, dns has nothing to do with
>> routing tables..)
>
>
> I am not much of a network guy but I have seen fair share of crazy
> things. However I am stamped by this. Could you tell me what is this
> __automatic_1c228804_0 table? I am guessing this is from $broken IPs.
> Any other hints how to troubleshoot this problem. Could you give me an
> example of a scenario in which an IP is automatically added to a table
> due to the applications run on the client?
>
>
> Most Kind Regards,
> Predrag