Bizarre pf/sendmail interaction

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Bizarre pf/sendmail interaction

Tethys
My firewall died recently, so I replaced it with a new machine. Since
I needed to reinstall the OS, I naturally went for 5.4, rather than
whatever obsolete version I'd been using on the old machine. But now I
can't get incoming email. My setup is something like:

public mx -------> firewall -------> internal mail server

My mx server is hosted in a datacentre. It receives mail and forwards
it on to my home mail server. However, it's not working. From my mx
server, I can connect to port 25 on my internal mail server. If I
issue a HELO greeting, everything is fine. If I issue EHLO instead,
the reply never makes it back to the MX server (the reply is being
sent, as I've verified with tcpdump). So clearly something's dropping
it. But nothing's being logged to indicate that. I have two block
rules, both of which should be logging:

block in log
block out log on $ext

I can issue EHLO without problems from other machines on my internal
network, and from the firewall itself. But anything originating
outside of the firewall fails. Any ideas? I'm somewhat stumped. My
previous machine was sufficiently obsolete that the pf syntax has
changed since then, so I wasn't able to just reuse my old pf rules.

Tet

--
"Java is a DSL for taking large XML files and converting them to stack
traces" -- Bulat Shakirzyanov

Reply | Threaded
Open this post in threaded view
|

Re: Bizarre pf/sendmail interaction

Aaron
Did you enable forwarding?

net.inet.ip.forwarding

Aaron

On 12/17/13 11:25, Tethys wrote:

> My firewall died recently, so I replaced it with a new machine. Since
> I needed to reinstall the OS, I naturally went for 5.4, rather than
> whatever obsolete version I'd been using on the old machine. But now I
> can't get incoming email. My setup is something like:
>
> public mx -------> firewall -------> internal mail server
>
> My mx server is hosted in a datacentre. It receives mail and forwards
> it on to my home mail server. However, it's not working. From my mx
> server, I can connect to port 25 on my internal mail server. If I
> issue a HELO greeting, everything is fine. If I issue EHLO instead,
> the reply never makes it back to the MX server (the reply is being
> sent, as I've verified with tcpdump). So clearly something's dropping
> it. But nothing's being logged to indicate that. I have two block
> rules, both of which should be logging:
>
> block in log
> block out log on $ext
>
> I can issue EHLO without problems from other machines on my internal
> network, and from the firewall itself. But anything originating
> outside of the firewall fails. Any ideas? I'm somewhat stumped. My
> previous machine was sufficiently obsolete that the pf syntax has
> changed since then, so I wasn't able to just reuse my old pf rules.
>
> Tet

Reply | Threaded
Open this post in threaded view
|

Re: Bizarre pf/sendmail interaction

Tethys
On Tue, Dec 17, 2013 at 5:30 PM, Aaron <[hidden email]> wrote:

> Did you enable forwarding?
>
> net.inet.ip.forwarding

Yes. Packets are being forwarded without problems, and it's working as
a firewall exactly as you'd expect for outbound traffic. I can browse
the web etc. But something strange is going on. Not only do I get
problems with EHLO vs HELO, but also I can't ssh from the firewall
into my internal mail server and if I ping it, it only works once:

# ping riva
PING riva.astradyne.corp (192.168.8.10): 56 data bytes
64 bytes from 192.168.8.10: icmp_seq=0 ttl=64 time=0.180 ms
ping: sendto: No route to host
ping: wrote riva.astradyne.corp 64 chars, ret=-1
ping: sendto: No route to host
ping: wrote riva.astradyne.corp 64 chars, ret=-1
ping: sendto: No route to host
ping: wrote riva.astradyne.corp 64 chars, ret=-1

It's as if something is allowing through a handful of packets and then
blocking subsequent ones. I wouldn't mind so much if I had a log
telling me what was being blocked and why. At least then I'd have a
clue what was going on and could adjust my pf.conf to fix it. But no,
the packets just seem to disappear into the ether.

Tet

--
"Java is a DSL for taking large XML files and converting them to stack
traces" -- Bulat Shakirzyanov

Reply | Threaded
Open this post in threaded view
|

Re: Bizarre pf/sendmail interaction

Craig Skinner-3
In reply to this post by Tethys
On 2013-12-17 Tue 17:05 PM |, Tethys wrote:

> On Tue, Dec 17, 2013 at 4:43 PM, Craig R. Skinner
> <[hidden email]> wrote:
>
> > I guess you have net.inet<something>.forwarding=1 in /etc/sysctl.conf
>
> Yes, I do. I can browse the web etc from inside the firewall without problems.
>
> > Does the firewall also know where to forward external traffic to your
> > internal mail server? (NON-NAT)
>
> I have:
>
>     pass in on $ext inet proto tcp from $mx to $loki_ext port smtp
> rdr-to $riva port smtp keep state
>
> $ext is the firewall's external interface. $mx expands to the IP
> addresses of my MX servers. $loki_ext is the external IP address of my
> firewall, and $riva is my internal mail server.
>

There might be some other rule later on that's blocking it.

Scan through the output of:
$ sudo pfctl -sr

Reply | Threaded
Open this post in threaded view
|

Re: Bizarre pf/sendmail interaction

Tethys
In reply to this post by Tethys
On Tue, Dec 17, 2013 at 7:51 PM, Jan Stary <[hidden email]> wrote:

>> block in log
>> block out log on $ext
>
> How could anyone help you knowing just these two lines?
> Show your pf.conf

I was trying to show that I only had two block lines and that they
both should log when blocking packets. My rules are actually very
simple:

    match out on $ext from $int_ip to any nat-to $loki_ext

    block in log
    block out log on $ext

    pass in quick on $int flags any

    pass out on $ext from $lokisafe

    pass in on $ext inet proto tcp to port 4334 rdr-to 127.0.0.1 port ssh
    pass in on $ext inet proto tcp from $mx to $loki_ext port smtp
rdr-to $riva port smtp flags any

    pass out on $int inet proto tcp from $mx port smtp flags any

$int and $ext are interfaces on the firewall (loki). $loki_ext is the
external IP, $int_ip is the internal /24. $lokisafe is a selection of
/24s that I've sometimes used, including the internal network. $riva
is my home mail server. $mx is the IP addresses of my hosted MX
servers.

With tcpdump, I can see the response to the EHLO greeting leaving
riva, arriving on $int, but never making it to $ext. Using HELO
instead doesn't prompt the same behaviour.

Tet

--
"Java is a DSL for taking large XML files and converting them to stack
traces" -- Bulat Shakirzyanov

Reply | Threaded
Open this post in threaded view
|

Re: Bizarre pf/sendmail interaction

Aaron
On 12/17/13 21:11, Tethys wrote:

> On Tue, Dec 17, 2013 at 7:51 PM, Jan Stary <[hidden email]> wrote:
>
>>> block in log
>>> block out log on $ext
>> How could anyone help you knowing just these two lines?
>> Show your pf.conf
> I was trying to show that I only had two block lines and that they
> both should log when blocking packets. My rules are actually very
> simple:
>
>      match out on $ext from $int_ip to any nat-to $loki_ext
>
>      block in log
>      block out log on $ext
>
>      pass in quick on $int flags any
>
>      pass out on $ext from $lokisafe
>
>      pass in on $ext inet proto tcp to port 4334 rdr-to 127.0.0.1 port ssh
>      pass in on $ext inet proto tcp from $mx to $loki_ext port smtp
> rdr-to $riva port smtp flags any
>
>      pass out on $int inet proto tcp from $mx port smtp flags any
>
> $int and $ext are interfaces on the firewall (loki). $loki_ext is the
> external IP, $int_ip is the internal /24. $lokisafe is a selection of
> /24s that I've sometimes used, including the internal network. $riva
> is my home mail server. $mx is the IP addresses of my hosted MX
> servers.
>
> With tcpdump, I can see the response to the EHLO greeting leaving
> riva, arriving on $int, but never making it to $ext. Using HELO
> instead doesn't prompt the same behaviour.
>
> Tet
>
this shouldn't be this hard.. can we see output from "netstat -rnf
inet", "pfctl -vvsr", maybe output from dmesg?   You never indicated
what MX server you're running.  postfix, actual sendmail, opensmtpd...
?? Your config from the smtp server would be helpful as well.  The fact
that you're getting different responses from HELO and EHLO would
indicate that something odd is going on with your MX server but the fact
that you get one reply from ping and no more would indicate something else.

A

Reply | Threaded
Open this post in threaded view
|

Re: Bizarre pf/sendmail interaction

Jan Stary
In reply to this post by Tethys
On Dec 18 02:11:55, [hidden email] wrote:

> On Tue, Dec 17, 2013 at 7:51 PM, Jan Stary <[hidden email]> wrote:
>
> >> block in log
> >> block out log on $ext
> >
> > How could anyone help you knowing just these two lines?
> > Show your pf.conf
>
> I was trying to show that I only had two block lines and that they
> both should log when blocking packets. My rules are actually very
> simple:
>
>     match out on $ext from $int_ip to any nat-to $loki_ext
>
>     block in log
>     block out log on $ext
>
>     pass in quick on $int flags any
>
>     pass out on $ext from $lokisafe
>
>     pass in on $ext inet proto tcp to port 4334 rdr-to 127.0.0.1 port ssh
>     pass in on $ext inet proto tcp from $mx to $loki_ext port smtp
> rdr-to $riva port smtp flags any
>
>     pass out on $int inet proto tcp from $mx port smtp flags any

Firstly, why don't you drop the "flags any" everywhere?


> $int and $ext are interfaces on the firewall (loki). $loki_ext is the
> external IP, $int_ip is the internal /24. $lokisafe is a selection of
> /24s that I've sometimes used, including the internal network. $riva
> is my home mail server. $mx is the IP addresses of my hosted MX
> servers.

So $riva is a member of $lokisafe, right?

> With tcpdump, I can see the response to the EHLO greeting leaving
> riva, arriving on $int, but never making it to $ext. Using HELO
> instead doesn't prompt the same behaviour.
>
> Tet
>
> --
> "Java is a DSL for taking large XML files and converting them to stack
> traces" -- Bulat Shakirzyanov

Reply | Threaded
Open this post in threaded view
|

Re: Bizarre pf/sendmail interaction

Tethys
On Wed, Dec 18, 2013 at 7:54 AM, Jan Stary <[hidden email]> wrote:

> So $riva is a member of $lokisafe, right?

Bingo! I knew it would be something trivial that I'd overlooked. All
working now.

Thanks,

Tet

--
"Java is a DSL for taking large XML files and converting them to stack
traces" -- Bulat Shakirzyanov

Reply | Threaded
Open this post in threaded view
|

Re: Bizarre pf/sendmail interaction

Stuart Henderson
In reply to this post by Tethys
On 2013-12-18, Tethys <[hidden email]> wrote:

> On Tue, Dec 17, 2013 at 7:51 PM, Jan Stary <[hidden email]> wrote:
>
>>> block in log
>>> block out log on $ext
>>
>> How could anyone help you knowing just these two lines?
>> Show your pf.conf
>
> I was trying to show that I only had two block lines and that they
> both should log when blocking packets. My rules are actually very
> simple:
>
>     match out on $ext from $int_ip to any nat-to $loki_ext
>
>     block in log
>     block out log on $ext
>
>     pass in quick on $int flags any
>
>     pass out on $ext from $lokisafe
>
>     pass in on $ext inet proto tcp to port 4334 rdr-to 127.0.0.1 port ssh
>     pass in on $ext inet proto tcp from $mx to $loki_ext port smtp
> rdr-to $riva port smtp flags any
>
>     pass out on $int inet proto tcp from $mx port smtp flags any
>
> $int and $ext are interfaces on the firewall (loki). $loki_ext is the
> external IP, $int_ip is the internal /24. $lokisafe is a selection of
> /24s that I've sometimes used, including the internal network. $riva
> is my home mail server. $mx is the IP addresses of my hosted MX
> servers.
>
> With tcpdump, I can see the response to the EHLO greeting leaving
> riva, arriving on $int, but never making it to $ext. Using HELO
> instead doesn't prompt the same behaviour.
>
> Tet
>

Most likely cause is that the firewall state is created by a packet
which is not a SYN. The TCP window-scaling options are *only* sent in
the SYN packet, any state created from a later packet will not have this
information, so the state tracking will reject packets as being "out of
window".

Most likely reason for this happening is that you don't have a general
"block" rule at the start of the ruleset, so some packets (in your case
outbound packets on any interface other than $ext) are passed by the
implicit default rule which is basically "pass flags any no state".

Change your ruleset to start with a rule which just says "block"
or "block log" to ensure that any packets which are passed, have a state
associated with them. Then test things, it may help to monitor output of
'tcpdump -nettipflog0' when doing this to help spot any missing rules.

If it still fails and you think you have sufficient "pass" rules, bump
up pf debug logging ("pfctl -x info" is probably enough) and look at the
console log or /var/log/messages. Return it to normal logging ("pfctl
-x error") afterwards (especially if it is a busy system, and that goes
double if you have the main console on a serial port ;)

Reply | Threaded
Open this post in threaded view
|

Re: Bizarre pf/sendmail interaction

polken
In reply to this post by Craig Skinner-3
i think that u will have to track down the packets
tcpdump can be the solution, or disable blocking while u find the offensive
rule then fix it!


> Date: Tue, 17 Dec 2013 17:56:33 +0000
> To: [hidden email]
> Subject: Re: Bizarre pf/sendmail interaction
> From: [hidden email]
>
> On 2013-12-17 Tue 17:05 PM |, Tethys wrote:
> > On Tue, Dec 17, 2013 at 4:43 PM, Craig R. Skinner
> > <[hidden email]> wrote:
> >
> > > I guess you have net.inet<something>.forwarding=1 in /etc/sysctl.conf
> >
> > Yes, I do. I can browse the web etc from inside the firewall without
problems.

> >
> > > Does the firewall also know where to forward external traffic to your
> > > internal mail server? (NON-NAT)
> >
> > I have:
> >
> >     pass in on $ext inet proto tcp from $mx to $loki_ext port smtp
> > rdr-to $riva port smtp keep state
> >
> > $ext is the firewall's external interface. $mx expands to the IP
> > addresses of my MX servers. $loki_ext is the external IP address of my
> > firewall, and $riva is my internal mail server.
> >
>
> There might be some other rule later on that's blocking it.
>
> Scan through the output of:
> $ sudo pfctl -sr