How to deal with DDoS ?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

How to deal with DDoS ?

Roger S.-3
Greetings misc@

I am facing regular and consequent DDoS, and I would like to know how
the OpenBSD community deal with these. Hints and inputs welcome.

The obvious first : my input pipes are not filled, there is plenty of
bandwith available for my regular users. As OpenBSD is not enough (in
my setup, I am sure there is a solution) to mitigate such attacks we
use a proprietary product, but this solution has some undesirable
side-effects and is not a viable long term solution for us.

Methodology is more or less always the same :
        - massive UDP flood           :   2 Gbps / 150 Kpps -> dropped
directly on the router, not a problem
        - moderate ICMP flood         :  10 Mbps /  12 Kpps
        - moderate IP fragments flood : 380 Mbps /  57 Kpps
        - moderate TCP RST flood      :  10 Mbps /  30 Kpps
        - massive TCP SYN flood       : 640 Mbps /   2 Mpps -> yup, that hurts

So, UDP never ever reaches my OpenBSD box. The SYN are made with a
very vicious method : each used IP send exactly one SYN, but there are
millions of them (traffic probably spoofed, but can not use uRPF as we
have asymmetric traffic and routes). I tried to set limit states with
1M entries, and it was quickly filled (tried 5M but the box collapses
way before that). So in the end, the state table collapses and no
traffic can pass, even for regular users with already established
connections.

I ran some experiments in a lab trying to reproduce this, with a box
roughly identical to what I have in production (but much weaker, of
course). The box collapses at 600 Kpps SYN (100% interrupts), but
handles everything very gently (less than 50% interrupts and no packet
loss) if the first rule evaluated is block drop in quick from !
<whitelisted_users>. So it seems that my bottleneck is PF here, not
the hardware. A consequence of this saturation : both my main firewall
and my backup claims MASTER ownership of the CARP (split brain
syndrome). CARP works just fine when I add the block rule, though.

Some configuration details :
        - OS  : OpenBSD 5.0/amd64 box, using GENERIC.MP
        - CPU : Intel X3460 CPU (4 cores, 2.80GHz)
        - RAM : 4GB
        - NIC : 2x Intel 82576 (2 ports each)

Each network card has the following setup : one port to the LAN, one
port to the WAN. Each pair (LAN1/LAN2 and WAN1/WAN2) is trunked using
LACP. Already bumped net.inet.ip.ifq.maxlen, as all NICs are
supported. My benchmarks did highlight two interesting things : amd64
has better performance than i386 (roughly 5-10% less interrupts, with
same rules and traffic), but the difference between GENERIC and
GENERIC.MP is insignificant.

My current idea is to hack a daemon to track established connections
(extracting them ` la netstat), and inject my block rule in an anchor
(` la relayd) when needed (watching some stats from pf, with its ioctl
interface). Pros: regular users the firewall saw before the attack can
still use the service. Cons: no new users are allowed until the
removal of the rule, obviously. Better than nothing, but I welcome any
other hints :)

One other solution may be to add boxes. I tried a carpnodes cluster,
but at 600 Kpps I got a "split brain" with both nodes claiming MASTER
for each carpnode. Maybe if I configure ALTQ it could help this ? As I
have more boxes, I could deal with the performance impact of ALTQ.

I am willing to test any patch/suggestion you may have, of course.
Even just hints about kernel code, as I am currently messing with PF
code myself. I did compile a profiled kernel, I must now check the
results but that will be another story.

To finish, here is the typical load on the box (errors are from
various DDoS, not related to normal use) :

Status: Enabled for 77 days 02:17:58             Debug: err

Interface Stats for trunk1            IPv4             IPv6
  Bytes In                   8885330383273                0
  Bytes Out                 72449316050298            20224
  Packets In
    Passed                     48738702875                0
    Blocked                    10152865611                0
  Packets Out
    Passed                     67293792876              281
    Blocked                     4557637133                0

State Table                          Total             Rate
  current entries                    37135
  searches                    130771929548        19632.2/s
  inserts                       4718030394          708.3/s
  removals                      4717993259          708.3/s
Source Tracking Table
  current entries                     7455
  searches                      4951426366          743.3/s
  inserts                        623672861           93.6/s
  removals                       623665406           93.6/s
Counters
  match                         5600111978          840.7/s
  bad-offset                             0            0.0/s
  fragment                         3591379            0.5/s
  short                            2500133            0.4/s
  normalize                          10968            0.0/s
  memory                             71750            0.0/s
  bad-timestamp                          0            0.0/s
  congestion                       3863476            0.6/s
  ip-option                              0            0.0/s
  proto-cksum                            0            0.0/s
  state-mismatch                   3722058            0.6/s
  state-insert                           0            0.0/s
  state-limit                            0            0.0/s
  src-limit                      234360390           35.2/s
  synproxy                     13817759263         2074.4/s
Limit Counters
  max states per rule                    0            0.0/s
  max-src-states                     89727            0.0/s
  max-src-nodes                          0            0.0/s
  max-src-conn                           0            0.0/s
  max-src-conn-rate                      0            0.0/s
  overload table insertion               0            0.0/s
  overload flush states                  0            0.0/s

Reply | Threaded
Open this post in threaded view
|

Re: How to deal with DDoS ?

Mindless Gr
I think the quickest and most effective solution is to blackhole the target ip
in your upstream's network, if you are running BGP, most of the ISPs have
blackhole communities that accepts /32 prefixes. All the other solution are
very complex and involves "super" hardware.
If you cannot afford to lose the
target ip's visibility during the attack there are several methods that
lessens the traffic, example, if your site is targeted to national customers
you can use DNS answer different ip's based on request's country the you can
direct all requests from china/north korea etc to a vps that you can rent for
$50
Commercial solutions exists, but they are just large clusters of firewalls
and load-balancers in which they are doing something like you do in your box.
there is no horizontal solution to these attacks, just per user case.

HTH
mindlessgr
blog: http://wp.mindless.gr


________________________________
From: Roger S. <[hidden email]>
To: [hidden email]
Sent: Monday,
February 20, 2012 6:57 PM
Subject: How to deal with DDoS ?
 
Greetings misc@
I am facing regular and consequent DDoS, and I would like to know how
the
OpenBSD community deal with these. Hints and inputs welcome.

The obvious
first : my input pipes are not filled, there is plenty of
bandwith available
for my regular users. As OpenBSD is not enough (in
my setup, I am sure there
is a solution) to mitigate such attacks we
use a proprietary product, but this
solution has some undesirable
side-effects and is not a viable long term
solution for us.

Methodology is more or less always the same :
    - massive
UDP flood           :   2 Gbps / 150 Kpps -> dropped
directly on the router,
not a problem
    - moderate ICMP flood         :  10 Mbps /  12 Kpps
    -
moderate IP fragments flood : 380 Mbps /  57 Kpps
    - moderate TCP RST
flood      :  10 Mbps /  30 Kpps
    - massive TCP SYN flood       : 640 Mbps
/   2 Mpps -> yup, that hurts

So, UDP never ever reaches my OpenBSD box. The
SYN are made with a
very vicious method : each used IP send exactly one SYN,
but there are
millions of them (traffic probably spoofed, but can not use uRPF
as we
have asymmetric traffic and routes). I tried to set limit states with
1M
entries, and it was quickly filled (tried 5M but the box collapses
way before
that). So in the end, the state table collapses and no
traffic can pass, even
for regular users with already established
connections.

I ran some
experiments in a lab trying to reproduce this, with a box
roughly identical to
what I have in production (but much weaker, of
course). The box collapses at
600 Kpps SYN (100% interrupts), but
handles everything very gently (less than
50% interrupts and no packet
loss) if the first rule evaluated is block drop
in quick from !
<whitelisted_users>. So it seems that my bottleneck is PF
here, not
the hardware. A consequence of this saturation : both my main
firewall
and my backup claims MASTER ownership of the CARP (split brain
syndrome). CARP works just fine when I add the block rule, though.

Some
configuration details :
    - OS  : OpenBSD 5.0/amd64 box, using GENERIC.MP
    - CPU : Intel X3460 CPU (4 cores, 2.80GHz)
    - RAM : 4GB
    - NIC : 2x
Intel 82576 (2 ports each)

Each network card has the following setup : one
port to the LAN, one
port to the WAN. Each pair (LAN1/LAN2 and WAN1/WAN2) is
trunked using
LACP. Already bumped net.inet.ip.ifq.maxlen, as all NICs are
supported. My benchmarks did highlight two interesting things : amd64
has
better performance than i386 (roughly 5-10% less interrupts, with
same rules
and traffic), but the difference between GENERIC and
GENERIC.MP is
insignificant.

My current idea is to hack a daemon to track established
connections
(extracting them ` la netstat), and inject my block rule in an
anchor
(` la relayd) when needed (watching some stats from pf, with its ioctl
interface). Pros: regular users the firewall saw before the attack can
still
use the service. Cons: no new users are allowed until the
removal of the rule,
obviously. Better than nothing, but I welcome any
other hints :)

One other
solution may be to add boxes. I tried a carpnodes cluster,
but at 600 Kpps I
got a "split brain" with both nodes claiming MASTER
for each carpnode. Maybe
if I configure ALTQ it could help this ? As I
have more boxes, I could deal
with the performance impact of ALTQ.

I am willing to test any
patch/suggestion you may have, of course.
Even just hints about kernel code,
as I am currently messing with PF
code myself. I did compile a profiled
kernel, I must now check the
results but that will be another story.

To
finish, here is the typical load on the box (errors are from
various DDoS, not
related to normal use) :

Status: Enabled for 77 days 02:17:58          
Debug: err

Interface Stats for trunk1            IPv4             IPv6
 
Bytes In                   8885330383273                0
  Bytes Out        
       72449316050298            20224
  Packets In
    Passed              
     48738702875                0
    Blocked                    10152865611
              0
  Packets Out
    Passed                     67293792876    
        281
    Blocked                     4557637133                0

State
Table                          Total             Rate
  current entries      
            37135
  searches                    130771929548        19632.2/s
  inserts                       4718030394          708.3/s
  removals      
              4717993259          708.3/s
Source Tracking Table
  current
entries                     7455
  searches                      4951426366  
      743.3/s
  inserts                        623672861           93.6/s
 
removals                       623665406           93.6/s
Counters
  match  
                     5600111978          840.7/s
  bad-offset                
           0            0.0/s
  fragment                         3591379    
      0.5/s
  short                            2500133            0.4/s
 
normalize                          10968            0.0/s
  memory          
                 71750            0.0/s
  bad-timestamp                      
  0            0.0/s
  congestion                       3863476          
0.6/s
  ip-option                              0            0.0/s
 
proto-cksum                            0            0.0/s
  state-mismatch  
               3722058            0.6/s
  state-insert                      
   0            0.0/s
  state-limit                            0          
0.0/s
  src-limit                      234360390           35.2/s
  synproxy
                   13817759263         2074.4/s
Limit Counters
  max states
per rule                    0            0.0/s
  max-src-states              
     89727            0.0/s
  max-src-nodes                          0      
    0.0/s
  max-src-conn                           0            0.0/s
 
max-src-conn-rate                      0            0.0/s
  overload table
insertion               0            0.0/s
  overload flush states          
      0            0.0/s

Reply | Threaded
Open this post in threaded view
|

Re: How to deal with DDoS ?

Joachim Schipper-2
In reply to this post by Roger S.-3
On Mon, Feb 20, 2012 at 05:57:05PM +0100, Roger S. wrote:

> I am facing regular and consequent DDoS, and I would like to know how
> the OpenBSD community deal with these. Hints and inputs welcome.
>
> The obvious first : my input pipes are not filled, there is plenty of
> bandwith available for my regular users. (...)
>
> Methodology is more or less always the same :
> - massive UDP flood           :   2 Gbps / 150 Kpps -> dropped
> directly on the router, not a problem
> - moderate ICMP flood         :  10 Mbps /  12 Kpps
> - moderate IP fragments flood : 380 Mbps /  57 Kpps
> - moderate TCP RST flood      :  10 Mbps /  30 Kpps
> - massive TCP SYN flood       : 640 Mbps /   2 Mpps -> yup, that hurts
>
> So, UDP never ever reaches my OpenBSD box. The SYN are made with a
> very vicious method : each used IP send exactly one SYN, but there are
> millions of them (traffic probably spoofed, but can not use uRPF as we
> have asymmetric traffic and routes). I tried to set limit states with
> 1M entries, and it was quickly filled (tried 5M but the box collapses
> way before that). So in the end, the state table collapses and no
> traffic can pass, even for regular users with already established
> connections.
>
> I ran some experiments in a lab trying to reproduce this, with a box
> roughly identical to what I have in production (but much weaker, of
> course). The box collapses at 600 Kpps SYN (100% interrupts), but
> handles everything very gently (less than 50% interrupts and no packet
> loss) if the first rule evaluated is block drop in quick from !
> <whitelisted_users>. So it seems that my bottleneck is PF here, not
> the hardware. A consequence of this saturation : both my main firewall
> and my backup claims MASTER ownership of the CARP (split brain
> syndrome). CARP works just fine when I add the block rule, though.
>
> Some configuration details :
> - OS  : OpenBSD 5.0/amd64 box, using GENERIC.MP
> - CPU : Intel X3460 CPU (4 cores, 2.80GHz)
> - RAM : 4GB
> - NIC : 2x Intel 82576 (2 ports each)
>
> Each network card has the following setup : one port to the LAN, one
> port to the WAN. Each pair (LAN1/LAN2 and WAN1/WAN2) is trunked using
> LACP. Already bumped net.inet.ip.ifq.maxlen, as all NICs are
> supported. My benchmarks did highlight two interesting things : amd64
> has better performance than i386 (roughly 5-10% less interrupts, with
> same rules and traffic), but the difference between GENERIC and
> GENERIC.MP is insignificant.
>
> My current idea is to hack a daemon to track established connections
> (extracting them ` la netstat), and inject my block rule in an anchor
> (` la relayd) when needed (watching some stats from pf, with its ioctl
> interface). Pros: regular users the firewall saw before the attack can
> still use the service. Cons: no new users are allowed until the
> removal of the rule, obviously. Better than nothing, but I welcome any
> other hints :)
>
> One other solution may be to add boxes. I tried a carpnodes cluster,
> but at 600 Kpps I got a "split brain" with both nodes claiming MASTER
> for each carpnode. Maybe if I configure ALTQ it could help this ? As I
> have more boxes, I could deal with the performance impact of ALTQ.
>
> I am willing to test any patch/suggestion you may have, of course.
> Even just hints about kernel code, as I am currently messing with PF
> code myself. I did compile a profiled kernel, I must now check the
> results but that will be another story.

Just the most obvious idea, since you mention that this sort-of-works if
you put "block drop in quick from !<whitelisted_users>": does it handle
this load if you turn off pf, or only include one or two trivial rules?
It certainly suggests that you may be well-served by optimizing your
pf.conf... (also, you've probably found the "synproxy" directive? If
not, try that too.)

Also, state tracking is apparently faster than stateless pf for normal
firewalls. I'd double-check if this is still true in your case, though;
if nothing else, stateless pf makes a CARP'ed setup easier.

I'm pretty sure you can muck with the rules without dropping existing
connections. (pf essentially does "does this packet match a known state?
If not, look at pf.conf".) This is almost certainly easier than your
proposed daemon.

A final, rather hackish, idea that probably does need a bit of
programming: greylisting for SYNs. Legitimate users will send you a
second SYN, so you could do something like (this has not even been
syntax-checked!)

  block drop log in quick from !<syn_seen> no state flags S/SA

and then add every logged IP to syn_seen. Obviously, this will slow down
access to the service for legitimate users, which may or may not be
acceptable.

        Joachim

--
PotD: www/squid,ntlm - WWW and FTP proxy cache and accelerator
http://www.joachimschipper.nl/

Reply | Threaded
Open this post in threaded view
|

Re: How to deal with DDoS ?

Hassan Monfared
Hi,
have you tried to set some tuning options in pf.conf & sysctl.conf ?
eg:
for sysctl.conf:
net.inet.ip.ifq.maxlen=512     # Maximum allowed input queue length
(256*number of physical interfaces)
kern.bufcachepercent=90        # Allow the kernel to use up to 90% of the
RAM for cache (default 10%)
net.inet.udp.recvspace=131072 # Increase based on your memory
net.inet.udp.sendspace=131072 # Increase based on your memory
ddb.panic=0                    # do not enter ddb console on kernel panic,
reboot if possible , this reduces headache

for pf.conf :
set optimization aggressive
...


On Wed, Feb 22, 2012 at 12:21 AM, Joachim Schipper <
[hidden email]> wrote:

> On Mon, Feb 20, 2012 at 05:57:05PM +0100, Roger S. wrote:
> > I am facing regular and consequent DDoS, and I would like to know how
> > the OpenBSD community deal with these. Hints and inputs welcome.
> >
> > The obvious first : my input pipes are not filled, there is plenty of
> > bandwith available for my regular users. (...)
> >
> > Methodology is more or less always the same :
> >       - massive UDP flood           :   2 Gbps / 150 Kpps -> dropped
> > directly on the router, not a problem
> >       - moderate ICMP flood         :  10 Mbps /  12 Kpps
> >       - moderate IP fragments flood : 380 Mbps /  57 Kpps
> >       - moderate TCP RST flood      :  10 Mbps /  30 Kpps
> >       - massive TCP SYN flood       : 640 Mbps /   2 Mpps -> yup, that
> hurts
> >
> > So, UDP never ever reaches my OpenBSD box. The SYN are made with a
> > very vicious method : each used IP send exactly one SYN, but there are
> > millions of them (traffic probably spoofed, but can not use uRPF as we
> > have asymmetric traffic and routes). I tried to set limit states with
> > 1M entries, and it was quickly filled (tried 5M but the box collapses
> > way before that). So in the end, the state table collapses and no
> > traffic can pass, even for regular users with already established
> > connections.
> >
> > I ran some experiments in a lab trying to reproduce this, with a box
> > roughly identical to what I have in production (but much weaker, of
> > course). The box collapses at 600 Kpps SYN (100% interrupts), but
> > handles everything very gently (less than 50% interrupts and no packet
> > loss) if the first rule evaluated is block drop in quick from !
> > <whitelisted_users>. So it seems that my bottleneck is PF here, not
> > the hardware. A consequence of this saturation : both my main firewall
> > and my backup claims MASTER ownership of the CARP (split brain
> > syndrome). CARP works just fine when I add the block rule, though.
> >
> > Some configuration details :
> >       - OS  : OpenBSD 5.0/amd64 box, using GENERIC.MP
> >       - CPU : Intel X3460 CPU (4 cores, 2.80GHz)
> >       - RAM : 4GB
> >       - NIC : 2x Intel 82576 (2 ports each)
> >
> > Each network card has the following setup : one port to the LAN, one
> > port to the WAN. Each pair (LAN1/LAN2 and WAN1/WAN2) is trunked using
> > LACP. Already bumped net.inet.ip.ifq.maxlen, as all NICs are
> > supported. My benchmarks did highlight two interesting things : amd64
> > has better performance than i386 (roughly 5-10% less interrupts, with
> > same rules and traffic), but the difference between GENERIC and
> > GENERIC.MP is insignificant.
> >
> > My current idea is to hack a daemon to track established connections
> > (extracting them ` la netstat), and inject my block rule in an anchor
> > (` la relayd) when needed (watching some stats from pf, with its ioctl
> > interface). Pros: regular users the firewall saw before the attack can
> > still use the service. Cons: no new users are allowed until the
> > removal of the rule, obviously. Better than nothing, but I welcome any
> > other hints :)
> >
> > One other solution may be to add boxes. I tried a carpnodes cluster,
> > but at 600 Kpps I got a "split brain" with both nodes claiming MASTER
> > for each carpnode. Maybe if I configure ALTQ it could help this ? As I
> > have more boxes, I could deal with the performance impact of ALTQ.
> >
> > I am willing to test any patch/suggestion you may have, of course.
> > Even just hints about kernel code, as I am currently messing with PF
> > code myself. I did compile a profiled kernel, I must now check the
> > results but that will be another story.
>
> Just the most obvious idea, since you mention that this sort-of-works if
> you put "block drop in quick from !<whitelisted_users>": does it handle
> this load if you turn off pf, or only include one or two trivial rules?
> It certainly suggests that you may be well-served by optimizing your
> pf.conf... (also, you've probably found the "synproxy" directive? If
> not, try that too.)
>
> Also, state tracking is apparently faster than stateless pf for normal
> firewalls. I'd double-check if this is still true in your case, though;
> if nothing else, stateless pf makes a CARP'ed setup easier.
>
> I'm pretty sure you can muck with the rules without dropping existing
> connections. (pf essentially does "does this packet match a known state?
> If not, look at pf.conf".) This is almost certainly easier than your
> proposed daemon.
>
> A final, rather hackish, idea that probably does need a bit of
> programming: greylisting for SYNs. Legitimate users will send you a
> second SYN, so you could do something like (this has not even been
> syntax-checked!)
>
>  block drop log in quick from !<syn_seen> no state flags S/SA
>
> and then add every logged IP to syn_seen. Obviously, this will slow down
> access to the service for legitimate users, which may or may not be
> acceptable.
>
>        Joachim
>
> --
> PotD: www/squid,ntlm - WWW and FTP proxy cache and accelerator
> http://www.joachimschipper.nl/

Reply | Threaded
Open this post in threaded view
|

Re: How to deal with DDoS ?

Ville Valkonen
On 22 February 2012 01:22, Hassan Monfared <[hidden email]> wrote:
> net.inet.udp.recvspace=131072 # Increase based on your memory
> net.inet.udp.sendspace=131072 # Increase based on your memory

System is running on obsd 5.0, thus these are obsolete.

Reply | Threaded
Open this post in threaded view
|

Re: How to deal with DDoS ?

Jan Stary
On Feb 22 02:21:04, Ville Valkonen wrote:
> On 22 February 2012 01:22, Hassan Monfared <[hidden email]> wrote:
> > net.inet.udp.recvspace=131072 # Increase based on your memory
> > net.inet.udp.sendspace=131072 # Increase based on your memory
>
> System is running on obsd 5.0, thus these are obsolete.

$ uname -a
OpenBSD biblio.stare.cz 5.1 GENERIC.MP#167 i386

$ sysctl net.inet.udp.{recvspace,sendspace}
net.inet.udp.recvspace=131072
net.inet.udp.sendspace=131072

I don't think it's gonna help with handling a DDOS, anyway.

Reply | Threaded
Open this post in threaded view
|

Re: How to deal with DDoS ?

Rudolf Leitgeb
Am Mittwoch, 22. Februar 2012, 08:36:49 schrieb Jan Stary:
> > $ sysctl net.inet.udp.{recvspace,sendspace}
> > net.inet.udp.recvspace=131072
> > net.inet.udp.sendspace=131072
>
> I don't think it's gonna help with handling a DDOS, anyway.

Especially not in this particular case. He drops UDP anyway and
reportedly fights a SYN flood attack.

Reply | Threaded
Open this post in threaded view
|

Re: How to deal with DDoS ?

Stuart Henderson
In reply to this post by Hassan Monfared
On 2012-02-21, Hassan Monfared <[hidden email]> wrote:

> Hi,
> have you tried to set some tuning options in pf.conf & sysctl.conf ?
> eg:
> for sysctl.conf:
> net.inet.ip.ifq.maxlen=512     # Maximum allowed input queue length
> (256*number of physical interfaces)
> kern.bufcachepercent=90        # Allow the kernel to use up to 90% of the
> RAM for cache (default 10%)
> net.inet.udp.recvspace=131072 # Increase based on your memory
> net.inet.udp.sendspace=131072 # Increase based on your memory
> ddb.panic=0                    # do not enter ddb console on kernel panic,
> reboot if possible , this reduces headache

These have nothing to do with state overflow (except raising
bufcachepercent will leave less space for states..)

> for pf.conf :
> set optimization aggressive

May possibly help (or you can set state limits per-rule; *very*
tight ones might be appropriate for the attack traffic).

Reply | Threaded
Open this post in threaded view
|

Re: How to deal with DDoS ?

Roger S.-3
In reply to this post by Joachim Schipper-2
On Tue, Feb 21, 2012 at 9:51 PM, Joachim Schipper
<[hidden email]> wrote:
> Just the most obvious idea, since you mention that this sort-of-works if
> you put "block drop in quick from !<whitelisted_users>": does it handle
> this load if you turn off pf, or only include one or two trivial rules?

Did not try to turn off pf (I need it anyway), and my pf.conf is very
simple and already optimized following the good book of pf and some
undeadly posts.

> It certainly suggests that you may be well-served by optimizing your
> pf.conf... (also, you've probably found the "synproxy" directive? If
> not, try that too.)

I already use synproxy, the problem is that I get so much SYN that
pf/state table collapses.

> Also, state tracking is apparently faster than stateless pf for normal
> firewalls. I'd double-check if this is still true in your case, though;
> if nothing else, stateless pf makes a CARP'ed setup easier.

I am not sure to understand here. I want to use synproxy to protect my
backend servers, so I need state stracking.

> I'm pretty sure you can muck with the rules without dropping existing
> connections. (pf essentially does "does this packet match a known state?
> If not, look at pf.conf".) This is almost certainly easier than your
> proposed daemon.

Sure thing, the daemon is only a workaround to provide degraded but
working service when under attack.

> A final, rather hackish, idea that probably does need a bit of
> programming: greylisting for SYNs. Legitimate users will send you a
> second SYN, so you could do something like (this has not even been
> syntax-checked!)
>  block drop log in quick from !<syn_seen> no state flags S/SA

I like the idea. This may need some programming indeed, but it seems
even better than my idea. Thanks, I'll take a look at this.

> and then add every logged IP to syn_seen. Obviously, this will slow down
> access to the service for legitimate users, which may or may not be
> acceptable.

We are speaking of a slower but working service, or no service at all.
I prefer the first alternative :)

Reply | Threaded
Open this post in threaded view
|

Re: How to deal with DDoS ?

Henning Brauer
In reply to this post by Hassan Monfared
can people please stop suggesting to push random buttons they don't
understand?
this is a prime ewxample.

* Hassan Monfared <[hidden email]> [2012-02-22 00:22]:
> Hi,
> have you tried to set some tuning options in pf.conf & sysctl.conf ?
> eg:
> for sysctl.conf:
> net.inet.ip.ifq.maxlen=512     # Maximum allowed input queue length
> (256*number of physical interfaces)

that rule of thumb is at least inaccurate. i'm pretty certain i
explained the details before and am getting tired of repeating myself
over and over.

> kern.bufcachepercent=90        # Allow the kernel to use up to 90% of the
> RAM for cache (default 10%)

that is entirely useless on a firewall.

> net.inet.udp.recvspace=131072 # Increase based on your memory
> net.inet.udp.sendspace=131072 # Increase based on your memory

that is
a) obsoleted by the autosizing
b) entirely useless for not locally terminated connections anyway

I gave the OP some input in private mail which I don't think belongs in
public. There is no one-size-fits-all recipe for dealing with DDoS.

And I certainly don't want to teach people how to make better DDoS
attacks.

--
Henning Brauer, [hidden email], [hidden email]
BS Web Services, http://bsws.de, Full-Service ISP
Secure Hosting, Mail and DNS Services. Dedicated Servers, Root to Fully Managed
Henning Brauer Consulting, http://henningbrauer.com/

Reply | Threaded
Open this post in threaded view
|

Re: How to deal with DDoS ?

Stuart Henderson
In reply to this post by Stuart Henderson
On 2012-02-22, Stuart Henderson <[hidden email]> wrote:

> On 2012-02-21, Hassan Monfared <[hidden email]> wrote:
>> Hi,
>> have you tried to set some tuning options in pf.conf & sysctl.conf ?
>> eg:
>> for sysctl.conf:
>> net.inet.ip.ifq.maxlen=512     # Maximum allowed input queue length
>> (256*number of physical interfaces)
>> kern.bufcachepercent=90        # Allow the kernel to use up to 90% of the
>> RAM for cache (default 10%)
>> net.inet.udp.recvspace=131072 # Increase based on your memory
>> net.inet.udp.sendspace=131072 # Increase based on your memory
>> ddb.panic=0                    # do not enter ddb console on kernel panic,
>> reboot if possible , this reduces headache
>
> These have nothing to do with state overflow

> (except raising bufcachepercent will leave less space for states..)

it was pointed out offlist that this may be incorrect, the theory is
that it should shrink when you need the space; that said it won't help
anyway and if for some reason it doesn't shrink you'll have problems.

Reply | Threaded
Open this post in threaded view
|

Re: How to deal with DDoS ?

Stuart Henderson
My followup mail was just about bufcachepercent. Auto-sizing socket
buffers is pointless on a firewall. Even if it were useful, if you are
running into resource starvation you want to *DECREASE* resource use
not increase it.

"aggressive" sets tcp.first to 30s. 2M SYNs per second * 30s = 60M states;
Roger said that 5M states is too much for the box.


On 2012/02/22 13:11, Hassan Monfared wrote:

> 1- auto-sizing in obsd5.0 is for tcp not udp.
> 2- I think setting option to aggressive will help.
>
>
> On 2/22/12, Stuart Henderson <[hidden email]> wrote:
> > On 2012-02-22, Stuart Henderson <[hidden email]> wrote:
> >> On 2012-02-21, Hassan Monfared <[hidden email]> wrote:
> >>> Hi,
> >>> have you tried to set some tuning options in pf.conf & sysctl.conf ?
> >>> eg:
> >>> for sysctl.conf:
> >>> net.inet.ip.ifq.maxlen=512     # Maximum allowed input queue length
> >>> (256*number of physical interfaces)
> >>> kern.bufcachepercent=90        # Allow the kernel to use up to 90% of the
> >>> RAM for cache (default 10%)
> >>> net.inet.udp.recvspace=131072 # Increase based on your memory
> >>> net.inet.udp.sendspace=131072 # Increase based on your memory
> >>> ddb.panic=0                    # do not enter ddb console on kernel
> >>> panic,
> >>> reboot if possible , this reduces headache
> >>
> >> These have nothing to do with state overflow
> >
> >> (except raising bufcachepercent will leave less space for states..)
> >
> > it was pointed out offlist that this may be incorrect, the theory is
> > that it should shrink when you need the space; that said it won't help
> > anyway and if for some reason it doesn't shrink you'll have problems.

Reply | Threaded
Open this post in threaded view
|

Re: How to deal with DDoS ?

mehma sarja
In reply to this post by Roger S.-3
On 2/22/12 12:39 AM, Roger S. wrote:
> On Tue, Feb 21, 2012 at 9:51 PM, Joachim Schipper
> <[hidden email]>  wrote:
>> Just the most obvious idea, since you mention that this sort-of-works if
>> you put "block drop in quick from !<whitelisted_users>": does it handle
>> this load if you turn off pf, or only include one or two trivial rules?
Hi,

I don't know nothing about nothing but someone once said as I was
struggling with a Snort and country block setup, "why don't you put them
on different machines?" As I am sure you have thought about this, can
you reduce the volume of attacks with a different machine so your pf
machine can handle the rest?

Mehma