Ways to handle DNS amplification attacks with OpenBSD

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Ways to handle DNS amplification attacks with OpenBSD

Kostas Zorbadelos
Hello all,

there is a need to restrict a specific type of DNS queries (ANY queries)
in our nameservers. We faced a DDoS attack in our resolvers and the
thing is that we could not simply cut access to DNS resolution to
specific client IPs, the queries came from our own unsuspecting
customers.  

The situation is similar but not the same as the one discribed here:

https://isc.sans.edu/diary.html?storyid=13261

We used IPtables and the string module to match a specific signature of
the problematic queries and it worked quite well (in our attack case the
problematic queries had a very specific and simple pattern).
The question is, if we had OpenBSD and PF as a
firewall what could we do to address this? From searching the archives I
saw this quite old post

http://www.monkey.org/openbsd/archive/misc/0207/msg00743.html

I haven't seen any string matching capability in PF for the packet
payload. Unless I am missing something, what would your suggestions be
in such a scenario? I am interested to hear possible solutions in other
layers as well.

Regards,
Kostas

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Henning Brauer
* Kostas Zorbadelos <[hidden email]> [2012-06-09 13:12]:

> We used IPtables and the string module to match a specific signature of
> the problematic queries and it worked quite well (in our attack case the
> problematic queries had a very specific and simple pattern).
> The question is, if we had OpenBSD and PF as a
> firewall what could we do to address this? From searching the archives I
> saw this quite old post
>
> http://www.monkey.org/openbsd/archive/misc/0207/msg00743.html
>
> I haven't seen any string matching capability in PF for the packet
> payload. Unless I am missing something, what would your suggestions be
> in such a scenario? I am interested to hear possible solutions in other
> layers as well.

string matching to more or less random packets' payload in the kernel?
that is beyond insane.

the proper solution is a small userland helper process, using divert-to
and maybe socket splicing.

--
Henning Brauer, [hidden email], [hidden email]
BS Web Services, http://bsws.de, Full-Service ISP
Secure Hosting, Mail and DNS Services. Dedicated Servers, Root to Fully Managed
Henning Brauer Consulting, http://henningbrauer.com/

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Jérémie Courrèges-Anglas-2
In reply to this post by Kostas Zorbadelos
Kostas Zorbadelos <[hidden email]> writes:

> Hello all,

Hi

> there is a need to restrict a specific type of DNS queries (ANY queries)
> in our nameservers. We faced a DDoS attack in our resolvers and the
> thing is that we could not simply cut access to DNS resolution to
> specific client IPs, the queries came from our own unsuspecting
> customers.

So you run resolvers for your clients.  I will assume you're an ISP.
In that case, you should be checking that the DNS queries that seem to
come from your clients *actually* come from your clients, not out of
nowhere, from spoofed IPs.  This could be done very easily with PF, *if*
your current architecture allows it (if you have a way to distinguish
network flow coming from your clients from spoofed requests coming from
the Internet).

Of course, if you're not an ISP, then forget what I said.

> The situation is similar but not the same as the one discribed here:
>
> https://isc.sans.edu/diary.html?storyid=13261

Indeed, that involves authoritative nameservers flooded with requests
that can come from anywhere.

[...]

--
Jérémie Courrèges-Anglas
GPG fingerprint: 61DB D9A0 00A4 67CF 2A90 8961 6191 8FBF 06A1 1494

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Peter Nicolai Mathias Hansteen
In reply to this post by Kostas Zorbadelos
Kostas Zorbadelos <[hidden email]> writes:

> there is a need to restrict a specific type of DNS queries (ANY queries)
> in our nameservers. We faced a DDoS attack in our resolvers and the
> thing is that we could not simply cut access to DNS resolution to
> specific client IPs, the queries came from our own unsuspecting
> customers.  

My first impulse when reading the sans diary item was to rate-limit,
possibly via the overload table mechanism, and if not blocking them
outright perhaps put the DNS requests from the overloads in a
minimal-bandwidth queue.  That may or may not be appropriate to your
context, and I suspect detection may be the main priority.  

While string matching in PF is not an option, I vaguely remember snort
users coming up with patterns to match earlier DNS tomfoolery, so
there's a chance you may be able to get useful info and possibly even a
working snort setup to deal with this one.

--
Peter N. M. Hansteen, member of the first RFC 1149 implementation team
http://bsdly.blogspot.com/ http://www.bsdly.net/ http://www.nuug.no/
"Remember to set the evil bit on all malicious network traffic"
delilah spamd[29949]: 85.152.224.147: disconnected after 42673 seconds.

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Tomasz Marszal
In reply to this post by Jérémie Courrèges-Anglas-2
On Sat, 09 Jun 2012 13:51:00 +0200, [hidden email] (Jérémie
Courrèges-Anglas) wrote:

> Kostas Zorbadelos <[hidden email]> writes:
>
>> Hello all,
>
> Hi
>
>> there is a need to restrict a specific type of DNS queries (ANY queries)
>> in our nameservers. We faced a DDoS attack in our resolvers and the
>> thing is that we could not simply cut access to DNS resolution to
>> specific client IPs, the queries came from our own unsuspecting
>> customers.
>
> So you run resolvers for your clients.  I will assume you're an ISP.
> In that case, you should be checking that the DNS queries that seem to
> come from your clients *actually* come from your clients, not out of
> nowhere, from spoofed IPs.  This could be done very easily with PF, *if*
> your current architecture allows it (if you have a way to distinguish
> network flow coming from your clients from spoofed requests coming from
> the Internet).
Does it affect cashing name server only or the one with zones to i know its
stupid question because the authoritative server have to be open for all to
redistribute domain ( or not for example we do not want some regions to
access our domain ?)
>

> Of course, if you're not an ISP, then forget what I said.
>
>> The situation is similar but not the same as the one discribed here:
>>
>> https://isc.sans.edu/diary.html?storyid=13261
>
> Indeed, that involves authoritative nameservers flooded with requests
> that can come from anywhere.
>
> [...]
>
> --
> Jérémie Courrèges-Anglas
> GPG fingerprint: 61DB D9A0 00A4 67CF 2A90 8961 6191 8FBF 06A1 1494

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Kevin Chadwick-2
In reply to this post by Peter Nicolai Mathias Hansteen
On Sat, 09 Jun 2012 14:08:58 +0200
Peter N. M. Hansteen wrote:

> While string matching in PF is not an option, I vaguely remember snort
> users coming up with patterns to match earlier DNS tomfoolery, so
> there's a chance you may be able to get useful info and possibly even a
> working snort setup to deal with this one.

I've made custom rules scanning for user names with Snort and it was
pretty easy. I had little performance concerns though so if possible,
minimising the packet percentage handed to Snort or analysed would
obviously be important.

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Rudolf Leitgeb
In reply to this post by Kostas Zorbadelos
Am Samstag, den 09.06.2012, 14:11 +0300 schrieb Kostas Zorbadelos:
> The situation is similar but not the same as the one discribed here:
>
> https://isc.sans.edu/diary.html?storyid=13261
>
> We used IPtables and the string module to match a specific signature of
> the problematic queries and it worked quite well (in our attack case the
> problematic queries had a very specific and simple pattern).

Mitigating this with snort looks much uglier than the beautiful and
elegant iptables counter measure posted in this list. Not sure how it
holds up under load, though.

Since the attacker uses fixed patterns, he/she seems to be a script
kiddy, and there is a good chance that the TTL can be used to identify
his/her packets. My approach would be to check what TTLs the packets
have vs. those from your clients and see whether you can filter based
on that.

Rudi

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Kostas Zorbadelos
In reply to this post by Peter Nicolai Mathias Hansteen
Hi, will try to comment to many posts at once :)

> Kostas Zorbadelos <[hidden email]> writes:
>
>> there is a need to restrict a specific type of DNS queries (ANY queries)
>> in our nameservers. We faced a DDoS attack in our resolvers and the
>> thing is that we could not simply cut access to DNS resolution to
>> specific client IPs, the queries came from our own unsuspecting
>> customers.

[hidden email] (Peter N. M. Hansteen) writes:

>
> My first impulse when reading the sans diary item was to rate-limit,
> possibly via the overload table mechanism, and if not blocking them
> outright perhaps put the DNS requests from the overloads in a
> minimal-bandwidth queue.  That may or may not be appropriate to your
> context, and I suspect detection may be the main priority.
>

I also thought about this one, but unfortunately it would affect service
for the customers and it would be unacceptable in this case. We would
also need to automate detection as you mentioned.

> While string matching in PF is not an option, I vaguely remember snort
> users coming up with patterns to match earlier DNS tomfoolery, so
> there's a chance you may be able to get useful info and possibly even a
> working snort setup to deal with this one.

From the little I know about snort, I am not sure what would the action
be in case of detection. The desired outcome is to filter the problem
traffic not allowing it to reach the nameservers. Perhaps a working
setup could be achieved, rearranging however the network setup.

[hidden email] (Jérémie Courrèges-Anglas) writes:
>
> So you run resolvers for your clients.  I will assume you're an ISP.

Yes.

> In that case, you should be checking that the DNS queries that seem to
> come from your clients *actually* come from your clients, not out of
> nowhere, from spoofed IPs.  This could be done very easily with PF, *if*
> your current architecture allows it (if you have a way to distinguish
> network flow coming from your clients from spoofed requests coming from
> the Internet).
>

Ohh, they do and that is the problem. We can't just cut the "offending"
clients because they will have no Internet service :)
Also we do not accept packets with sources in our network ranges "from
the outside" in our border ACLs (I guess this is common antispoof
practice).

> Of course, if you're not an ISP, then forget what I said.
>
>> The situation is similar but not the same as the one discribed here:
>>
>> https://isc.sans.edu/diary.html?storyid=13261
>
> Indeed, that involves authoritative nameservers flooded with requests
> that can come from anywhere.
>

Exactly.

Henning Brauer <[hidden email]> writes:

>
> string matching to more or less random packets' payload in the kernel?
> that is beyond insane.
>

I am interested to know if this has caused problems in IPtables'
setups. It sounds dangerous, however Linux systems provide the
capability. I guess they have thought about consequences and hopefully
somehow addressed them.

> the proper solution is a small userland helper process, using divert-to
> and maybe socket splicing.

I am not sure we are talking about the same thing (you must have an
implementation clearly in your mind ;-) ), but
my feeling for a proper way to address this problem is via a
userland application in a proxy or intercepting mode. This could filter
the offending traffic and give to the nameserver the rest to
service.
I think you also talk about this (correct me if I am wrong). The main
problem with it is that it needs to be developed :)
Perhaps relayd is a good match for this one.

IPtables' string module however is here now and it provided a crude but
working solution.

Thanks for all the comments.

Kostas

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Henning Brauer
* Kostas Zorbadelos <[hidden email]> [2012-06-09 18:02]:
> Henning Brauer <[hidden email]> writes:
> > string matching to more or less random packets' payload in the kernel?
> > that is beyond insane.
> I am interested to know if this has caused problems in IPtables'
> setups. It sounds dangerous, however Linux systems provide the
> capability. I guess they have thought about consequences and hopefully
> somehow addressed them.

your guess is wrong... they might have been lucky so far, or not, I
don't follow all the itables bugs.

> > the proper solution is a small userland helper process, using divert-to
> > and maybe socket splicing.
> I am not sure we are talking about the same thing (you must have an
> implementation clearly in your mind ;-) ), but
> my feeling for a proper way to address this problem is via a
> userland application in a proxy or intercepting mode. This could filter
> the offending traffic and give to the nameserver the rest to
> service.

that is pretty much what it comes down to, tho writing these proxies
is very easy these days, using the techniques i mentioned above.

> I think you also talk about this (correct me if I am wrong). The main
> problem with it is that it needs to be developed :)

right.

--
Henning Brauer, [hidden email], [hidden email]
BS Web Services, http://bsws.de, Full-Service ISP
Secure Hosting, Mail and DNS Services. Dedicated Servers, Root to Fully Managed
Henning Brauer Consulting, http://henningbrauer.com/

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Kostas Zorbadelos
In reply to this post by Rudolf Leitgeb
Rudolf Leitgeb <[hidden email]> writes:

> Am Samstag, den 09.06.2012, 14:11 +0300 schrieb Kostas Zorbadelos:
>> The situation is similar but not the same as the one discribed here:
>>
>> https://isc.sans.edu/diary.html?storyid=13261
>>
>> We used IPtables and the string module to match a specific signature of
>> the problematic queries and it worked quite well (in our attack case the
>> problematic queries had a very specific and simple pattern).
>
> Mitigating this with snort looks much uglier than the beautiful and
> elegant iptables counter measure posted in this list. Not sure how it
> holds up under load, though.
>

In our case (nameservers handling thousands of queries per second) and
during the time of attack multiple times that, it worked with negligible
performance impact. The actual network traffic however was in the order
of 40-50 Mbps per server.
 
> Since the attacker uses fixed patterns, he/she seems to be a script
> kiddy, and there is a good chance that the TTL can be used to identify
> his/her packets. My approach would be to check what TTLs the packets
> have vs. those from your clients and see whether you can filter based
> on that.
>

What do you mean identify and filter based on TTL? In our case the
attacker used a specific query for a single domain.

> Rudi
>

Kostas

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Rudolf Leitgeb
Am Samstag, den 09.06.2012, 19:17 +0300 schrieb Kostas Zorbadelos:
> What do you mean identify and filter based on TTL? In our case the
> attacker used a specific query for a single domain.

I mean the TTL field from the IP header of these packets. While the
attacker's packets spoof the sender address, they might not spoof
the TTL, and probably being away more hops from your servers than
your clients, their packets should have lower TTL values.

A network traffic dump could show quickly whether this approach
could possibly work.

Cheers,

Rudi

PS: Obviously a skilled attacker can also crank up TTL values to
compensate for their longer route, but "fixed pattern" indicates
to me that you deal with a script kiddie here.

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Stuart Henderson
In reply to this post by Kostas Zorbadelos
On 2012-06-09, Kostas Zorbadelos <[hidden email]> wrote:
> I am interested to hear possible solutions in other layers as well.

http://fanf.livejournal.com/122111.html seems a nice approach...

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Rudolf Leitgeb
Am Sonntag, den 10.06.2012, 00:37 +0000 schrieb Stuart Henderson:
> On 2012-06-09, Kostas Zorbadelos <[hidden email]> wrote:
> > I am interested to hear possible solutions in other layers as well.
>
> http://fanf.livejournal.com/122111.html seems a nice approach...

This seems to work nicely if the attacker spoofs random addresses or
if the real target is not the DNS server but the endpoint receiving
its replies (therefore the term "amplification attack").

In Kostas's case it appears the attacker spoofs legit client addresses,
which means rate limiting would likely cut off these clients.

Rudi

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Kapetanakis Giannis
In reply to this post by Kostas Zorbadelos
On 09/06/12 18:58, Kostas Zorbadelos wrote:

>> Kostas Zorbadelos<[hidden email]>  writes:
>>
>>> there is a need to restrict a specific type of DNS queries (ANY queries)
>>> in our nameservers. We faced a DDoS attack in our resolvers and the
>>> thing is that we could not simply cut access to DNS resolution to
>>> specific client IPs, the queries came from our own unsuspecting
>>> customers.
>>>
> Ohh, they do and that is the problem. We can't just cut the
> "offending" clients because they will have no Internet service :) Also
> we do not accept packets with sources in our network ranges "from the
> outside" in our border ACLs (I guess this is common antispoof practice).

Hi,

My understanding so far is that the queries hit your DNS servers from
your ISP network/clients
and are not spoofed. Also those queries hit the recursive/caching DNS
servers (open only to ISP network)
and they are of type ANY for a specific domain (example.com).

If this is true then why not try to 'filter' on the application layer
than on the network?

As far as I know BIND cannot block specific queries. You might want to
check unbound and local-data.

Maybe you could try some kind of DNS-proxy to filter out the unwanted
queries, since there is a pattern.
Check http://thesprawl.org/projects/dnschef/ (haven't tested it).

Alternatively you would want to implement that pf helper/'proxy' as
Henning suggested
which without doubt would be faster but you have to develop it.

Last, run the iptables matching filter on the DNS's firewall itself (if
they run linux)
and not on the external firewalls.

good luck,

Giannis

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Stuart Henderson
In reply to this post by Rudolf Leitgeb
On 2012-06-10, Rudolf Leitgeb <[hidden email]> wrote:
> Am Sonntag, den 10.06.2012, 00:37 +0000 schrieb Stuart Henderson:
>> On 2012-06-09, Kostas Zorbadelos <[hidden email]> wrote:
>> > I am interested to hear possible solutions in other layers as well.
>>
>> http://fanf.livejournal.com/122111.html seems a nice approach...
>
> This seems to work nicely if the attacker spoofs random addresses or
> if the real target is not the DNS server but the endpoint receiving
> its replies (therefore the term "amplification attack").

Yes that's kind-of implied by "amplification attack". Queries are sourced
from a false address (the actual target of the attack), with the intention
to direct larger DNS responses at the target.

> In Kostas's case it appears the attacker spoofs legit client addresses,
> which means rate limiting would likely cut off these clients.

From the followup mail it sounds like these are queries being generated
directly from client machines on the network i.e. DDOS rather than amplification.
So what is causing the queries? Client PCs under the control of attackers?
CPE routers with broken configuration allowing global access to DNS forwarders?

Reply | Threaded
Open this post in threaded view
|

Re: Ways to handle DNS amplification attacks with OpenBSD

Kostas Zorbadelos
In reply to this post by Kapetanakis Giannis
Kapetanakis Giannis <[hidden email]> writes:

> On 09/06/12 18:58, Kostas Zorbadelos wrote:
>
> Hi,
>

Hi Giannis,

> My understanding so far is that the queries hit your DNS servers from
> your ISP network/clients

Yes.

> and are not spoofed.

I didn't say that.

> Also those queries hit the recursive/caching DNS
> servers (open only to ISP network)
> and they are of type ANY for a specific domain (example.com).
>

Yes.

> If this is true then why not try to 'filter' on the application layer
> than on the network?
>

I agree that this is the proper layer to address the issue. However this
is not easy.

> As far as I know BIND cannot block specific queries. You might want to
> check unbound and local-data.
>

You can't just reimplement a very busy commercial setup overnight,
especially in such a critical service.

> Maybe you could try some kind of DNS-proxy to filter out the unwanted
> queries, since there is a pattern.
> Check http://thesprawl.org/projects/dnschef/ (haven't tested it).
>

I will take a look at it.

> Alternatively you would want to implement that pf helper/'proxy' as
> Henning suggested
> which without doubt would be faster but you have to develop it.
>

Yes.

> Last, run the iptables matching filter on the DNS's firewall itself (if
> they run linux)
> and not on the external firewalls.
>

This is what we did and what we will do again.

> good luck,
>
> Giannis
>

Kostas