PF block log all and ddos issue

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

PF block log all and ddos issue

Theron ZORBAS
Hello Misc,

I have an OpenBSD 5.2 i386 firewall. It was running so good till
last night.
We are under a ddos attack(DNS Amplification attack) (ANY? isc.org
requests)
Our firewall freezes. I cant ping to my firewall interfaces even
internal interface. It doesnt answer maybe replies very slowly.
Before this
freezing issue i got these messages at /var/log/messages:

/bsd:
uvm_mapent_alloc: out of static map entries
/bsd: WARNING: mclpools limit
reached; increase kern.maxcluster  


I increased up kern.maxcluster values
but did not work. We had to reboot firewall every 2 hours cause of this ddos
attack.
After that i realized that changing this pf rule worked:

"block log
all" to "block all"

Now we are still under attack but firewall handles it. It
drops udp port 53 attacks and doesnt log any packet.
But this is not what i
want. As default i wanna log which packet my firewall blocked.

So how can i
log all blocked packets and my firewall can be still up and running?

Thanks.
Theron

Reply | Threaded
Open this post in threaded view
|

Re: PF block log all and ddos issue

Peter Nicolai Mathias Hansteen
Theron ZORBAS <[hidden email]> writes:

> I have an OpenBSD 5.2 i386 firewall. It was running so good till
> last night.
> We are under a ddos attack(DNS Amplification attack) (ANY? isc.org
> requests)

First of all, unless you *want* to run an open resolver, reconfigure so
only the ones you want to do recursion for (typically at most clients in
a subset of directly connected networks) will get the data they ask
for. The difference in size between a full answer to the query you quote
and a 'denied' reply is quite significant.

> Our firewall freezes. I cant ping to my firewall interfaces even
> internal interface. It doesnt answer maybe replies very slowly.
> Before this
> freezing issue i got these messages at /var/log/messages:
>
> /bsd:
> uvm_mapent_alloc: out of static map entries
> /bsd: WARNING: mclpools limit
> reached; increase kern.maxcluster  
>
>
> I increased up kern.maxcluster values
> but did not work. We had to reboot firewall every 2 hours cause of this ddos
> attack.
> After that i realized that changing this pf rule worked:
>
> "block log
> all" to "block all"
>
> Now we are still under attack but firewall handles it. It
> drops udp port 53 attacks and doesnt log any packet.
> But this is not what i
> want. As default i wanna log which packet my firewall blocked.
>
> So how can i
> log all blocked packets and my firewall can be still up and running?

If pf logging or not is the difference between your firewall crashing or
not, I'd put a significantly lower priority on collecting statistics
than shutting up the noise makers.

I was in a similar situation a little while back (blagged about it too,
see [1]).  If you do want to run a name service but want to send the
recursion gropers packing, you could do what I did - read the log for
requests denied by named, then blackhole route the offending IP address
to make sure you don't make any noise yourself by sending replies (pfctl
-k and adding to a table you block drop are optional extras).

- P

[1] http://bsdly.blogspot.ca/2012/12/ddos-bots-are-people-or-manned-by-some.html


--
Peter N. M. Hansteen, member of the first RFC 1149 implementation team
http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/
"Remember to set the evil bit on all malicious network traffic"
delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.

Reply | Threaded
Open this post in threaded view
|

Re: PF block log all and ddos issue

Kevin Chadwick-2
On Thu, 27 Dec 2012 18:43:44 +0100
[hidden email] (Peter N. M. Hansteen) wrote:

> As default i wanna log which packet my firewall blocked.
> >
> > So how can i
> > log all blocked packets and my firewall can be still up and
> > running?  

Hopefully I will never need them but I have various pf configs ready
with varying levels of processor usage down to basically drop all but
randomly log a very small percentage of packets simply for upstream
filtering use or to see if increased bandwidth and redundancy is the
only answer.

Reply | Threaded
Open this post in threaded view
|

Re: PF block log all and ddos issue

Theron ZORBAS
In reply to this post by Peter Nicolai Mathias Hansteen
Hello,

@Peter, thanks for your reply. But i have no problem with dns daemon. 
Infact attackers make ddos to ip addresses which have no dns services
listening UDP port 53. 

So i have solved this issue partially with these
rules below:

#Stop pointless udp 53 requests (dont log these packets)
block
drop in  quick on vlan100 inet proto {tcp,udp} from any to $dmz2:network  port
{ 53 }
block drop out  quick on $dmz2 inet proto {tcp,udp} from any to
$dmz2:network  port { 53 }
#default policy block and log all of them
block log
all
# Other ruless
......

But i still wonder why my firewall freezes when
logging all blocked udp 53 requests.
The attack is not too heavy. I had seen
much worse before.

Anyway, thanks.




________________________________
From: Peter N. M. Hansteen <[hidden email]>
To: Theron ZORBAS
<[hidden email]>
Cc: "[hidden email]" <[hidden email]>
Sent:
Thursday, December 27, 2012 7:43 PM
Subject: Re: PF block log all and ddos
issue
 
Theron ZORBAS <[hidden email]> writes:

> I have an OpenBSD
5.2 i386 firewall. It was running so good till
> last night.
> We are under a
ddos attack(DNS Amplification attack) (ANY? isc.org
> requests)

First of all,
unless you *want* to run an open resolver, reconfigure so
only the ones you
want to do recursion for (typically at most clients in
a subset of directly
connected networks) will get the data they ask
for. The difference in size
between a full answer to the query you quote
and a 'denied' reply is quite
significant.

> Our firewall freezes. I cant ping to my firewall interfaces
even
> internal interface. It doesnt answer maybe replies very slowly.
>
Before this
> freezing issue i got these messages at /var/log/messages:
>
>
/bsd:
> uvm_mapent_alloc: out of static map entries
> /bsd: WARNING: mclpools
limit
> reached; increase kern.maxcluster  
>
>
> I increased up
kern.maxcluster values
> but did not work. We had to reboot firewall every 2
hours cause of this ddos
> attack.
> After that i realized that changing this
pf rule worked:
>
> "block log
> all" to "block all"
>
> Now we are still
under attack but firewall handles it. It
> drops udp port 53 attacks and
doesnt log any packet.
> But this is not what i
> want. As default i wanna log
which packet my firewall blocked.
>
> So how can i
> log all blocked packets
and my firewall can be still up and running?

If pf logging or not is the
difference between your firewall crashing or
not, I'd put a significantly
lower priority on collecting statistics
than shutting up the noise makers.

I
was in a similar situation a little while back (blagged about it too,
see
[1]).  If you do want to run a name service but want to send the
recursion
gropers packing, you could do what I did - read the log for
requests denied by
named, then blackhole route the offending IP address
to make sure you don't
make any noise yourself by sending replies (pfctl
-k and adding to a table you
block drop are optional extras).

- P

[1]
http://bsdly.blogspot.ca/2012/12/ddos-bots-are-people-or-manned-by-some.html
--
Peter N. M. Hansteen, member of the first RFC 1149 implementation team
http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/
"Remember
to set the evil bit on all malicious network traffic"
delilah spamd[29949]:
85.152.224.147: disconnected after 42673 seconds.

Reply | Threaded
Open this post in threaded view
|

Re: PF block log all and ddos issue

James Shupe-4
> But i still wonder why my firewall freezes when
> logging all blocked udp 53 requests.
> The attack is not too heavy. I had seen
> much worse before.
>

- Check interrupt usage
- Check states to make sure the reason it seems unresponsive isn't due
to the state table being full

Without more information from the machine, we don't have a lot of advice
we can really give.

--
James Shupe

[demime 1.01d removed an attachment of type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: PF block log all and ddos issue

Theron ZORBAS
Hi again,

Here is the info that i can supply. If need more please tell me how
to do?

PF Options
set timeout { interval 10, frag 30 }
set timeout {
tcp.first 300, tcp.opening 60, tcp.established 86400 }
set timeout {
tcp.closing 900, tcp.finwait 60, tcp.closed 90 }
set timeout { udp.first 120,
udp.single 150, udp.multiple 120 }
set timeout { icmp.first 20, icmp.error 10
}
set timeout { other.first 60, other.single 30, other.multiple 60 }
set
timeout { adaptive.start 0, adaptive.end 0 }
set limit { states 500000, frags
100000 }
set loginterface none
set skip on { lo0 enc0 }
set optimization
normal
set block-policy drop
set fingerprints "/etc/pf.os"

PF states :
root#
pfctl -ss |wc -l
    4765

root# date;vmstat -i
Fri Dec 28 22:57:00 EET 2012
interrupt                       total     rate
irq0/clock                  
91039955      799
irq0/ipi                     17900164      157
irq82/bnx0  
                58237357      511
irq98/bnx1                  215829335    
1896
irq82/bnx2                      59316        0
irq97/bnx4                
   6800293       59
irq80/mfi0                     537214        4
irq82/bnx5
                 125670397     1104
irq84/ehci0                     74177    
   0
Total                       516148208     4534

root# date;vmstat -i
Fri
Dec 28 22:57:05 EET 2012
interrupt                       total     rate
irq0/clock                   91043954      799
irq0/ipi                    
17900210      157
irq82/bnx0                   58237576      511
irq98/bnx1  
               215854554     1896
irq82/bnx2                      59317      
 0
irq97/bnx4                    6800360       59
irq80/mfi0                  
  537232        4
irq82/bnx5                  125684762     1104
irq84/ehci0  
                  74177        0
Total                       516192142    
4535

My egress interface is at bnx1 and my attacked interface is bnx5.
I read
somewhere that intel network cards' (em0 etc.) performance were better. I can
try to get a new nic to see difference.
I have taken these outputs when i am
not logging udp 53 requests which are just attack.

Thanks.
________________________________
 From: James Shupe <[hidden email]>
To:
[hidden email]
Sent: Friday, December 28, 2012 8:11 PM
Subject: Re: PF
block log all and ddos issue
 
> But i still wonder why my firewall freezes
when
> logging all blocked udp 53 requests.
> The attack is not too heavy. I
had seen
> much worse before.
>

- Check interrupt usage
- Check states to
make sure the reason it seems unresponsive isn't due
to the state table being
full

Without more information from the machine, we don't have a lot of advice
we can really give.

--
James Shupe

[demime 1.01d removed an attachment of
type application/pgp-signature which had a name of signature.asc]

Reply | Threaded
Open this post in threaded view
|

Re: PF block log all and ddos issue

Theron ZORBAS
Sorry my last post is broken:
You can see my outputs at :
http://pastebin.com/FtbfHXf8


Thanks.


________________________________
From: Theron ZORBAS <[hidden email]>
To: James Shupe
<[hidden email]>; "[hidden email]" <[hidden email]>
Sent: Friday,
December 28, 2012 11:00 PM
Subject: Re: PF block log all and ddos issue
 
Hi
again,

Here is the info that i can supply. If need more please tell me how
to
do?

PF Options
set timeout { interval 10, frag 30 }
set timeout {
tcp.first
300, tcp.opening 60, tcp.established 86400 }
set timeout {
tcp.closing 900,
tcp.finwait 60, tcp.closed 90 }
set timeout { udp.first 120,
udp.single 150,
udp.multiple 120 }
set timeout { icmp.first 20, icmp.error 10
}
set timeout {
other.first 60, other.single 30, other.multiple 60 }
set
timeout {
adaptive.start 0, adaptive.end 0 }
set limit { states 500000, frags
100000 }
set loginterface none
set skip on { lo0 enc0 }
set optimization
normal
set
block-policy drop
set fingerprints "/etc/pf.os"

PF states :
root#
pfctl -ss
|wc -l
    4765

root# date;vmstat -i
Fri Dec 28 22:57:00 EET 2012
interrupt  
                    total     rate
irq0/clock                  
91039955    
 799
irq0/ipi                     17900164      157
irq82/bnx0  
             
  58237357      511
irq98/bnx1                  215829335    
1896
irq82/bnx2
                     59316        0
irq97/bnx4                
   6800293    
  59
irq80/mfi0                     537214        4
irq82/bnx5
               
 125670397     1104
irq84/ehci0                     74177    
   0
Total      
                516148208     4534

root# date;vmstat -i
Fri
Dec 28 22:57:05
EET 2012
interrupt                       total     rate
irq0/clock            
      91043954      799
irq0/ipi                    
17900210      157
irq82/bnx0                   58237576      511
irq98/bnx1  
             
 215854554     1896
irq82/bnx2                      59317      
 0
irq97/bnx4
                   6800360       59
irq80/mfi0                  
  537232    
   4
irq82/bnx5                  125684762     1104
irq84/ehci0  
           
      74177        0
Total                       516192142    
4535

My egress
interface is at bnx1 and my attacked interface is bnx5.
I read
somewhere that
intel network cards' (em0 etc.) performance were better. I can
try to get a
new nic to see difference.
I have taken these outputs when i am
not logging
udp 53 requests which are just attack.

Thanks.
________________________________
From: James Shupe <[hidden email]>
To:
[hidden email]
Sent: Friday, December 28, 2012 8:11 PM
Subject: Re: PF
block log all and ddos issue

> But i still wonder why my firewall freezes
when
> logging all blocked udp 53 requests.
> The attack is not too heavy. I
had seen
> much worse before.
>

- Check interrupt usage
- Check states to
make sure the reason it seems unresponsive isn't due
to the state table being
full

Without more information from the machine, we don't have a lot of advice
we can really give.

--
James Shupe

[demime 1.01d removed an attachment of
type application/pgp-signature which had a name of signature.asc]