Openbgpd & kernel tuning

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Openbgpd & kernel tuning

Marcel Prisi
Hi !

I am in the process of setting up an OpenBSD / OpenBGPD core router for
a small local ISP (two 20mbps upstreams, simple setup).

OpenBGPD's config seems OK, but I need some help about OpenBSD's tunable
parameters using sysctl.

I have the following :

net.inet.tcp.recvspace=65536
net.inet.tcp.sendspace=65536
kern.ipc.somaxconn=1024
net.inet.icmp.drop_redirect=1
net.inet.icmp.log_redirect=1
net.inet.ip.redirect=0
net.inet.ip.sourceroute=0
net.inet.icmp.bmcastecho=0
net.inet.icmp.maskrepl=0

Are these OK ? Should I also do something for udp ? Do I miss some ?

Thanks for your help.

Reply | Threaded
Open this post in threaded view
|

Re: Openbgpd & kernel tuning

Theo de Raadt
> I am in the process of setting up an OpenBSD / OpenBGPD core router for
> a small local ISP (two 20mbps upstreams, simple setup).
>
> OpenBGPD's config seems OK, but I need some help about OpenBSD's tunable
> parameters using sysctl.

The idea is that you shouldn't need to change any options.

Reply | Threaded
Open this post in threaded view
|

Re: Openbgpd & kernel tuning

Marcel Prisi
Theo de Raadt a icrit :

>
>The idea is that you shouldn't need to change any options.
>
>  
>
Well then it will be easier than I thought :-)

I read some old threads about too small tcp.sendspace / tcp.recvspace in
3.4 time that used to hit performance so I thought it would be useful.

The others were about DOS prevention.

Thanks for your help.

Reply | Threaded
Open this post in threaded view
|

Re: Openbgpd & kernel tuning

Stuart Henderson
In reply to this post by Marcel Prisi
On 2006/03/08 16:37, Marcel Prisi wrote:
> OpenBGPD's config seems OK, but I need some help about OpenBSD's tunable
> parameters using sysctl.

> net.inet.tcp.recvspace=65536
> net.inet.tcp.sendspace=65536
> kern.ipc.somaxconn=1024
> net.inet.icmp.drop_redirect=1
> net.inet.icmp.log_redirect=1
> net.inet.ip.redirect=0
> net.inet.ip.sourceroute=0
> net.inet.icmp.bmcastecho=0
> net.inet.icmp.maskrepl=0

Half of these aren't even for OpenBSD. Are these settings from some
guide to tuning another OS for use as a webserver or something like
that?

> Are these OK ? Should I also do something for udp ? Do I miss some ?

I think you should remove them all and only touch the defaults if you
encounter a specific problem and have understood how the change that
you're making will help. The defaults are pretty sane. The thing you
might want to monitor on a busy router is mbuf use (netstat -m) but
that's monitoring, not tweaking, unless you start having a problem.

Reply | Threaded
Open this post in threaded view
|

Re: Openbgpd & kernel tuning

Henning Brauer
In reply to this post by Marcel Prisi
* Marcel Prisi <[hidden email]> [2006-03-08 16:42]:
> OpenBGPD's config seems OK, but I need some help about OpenBSD's tunable
> parameters using sysctl.

the only thing you might want to change is
  net.inet.ip.ifq.maxlen
the default is a little low for routing at higher speeds. 250 seems
a good compromise for many higher-bandwidth routers.

--
BS Web Services, http://www.bsws.de/
OpenBSD-based Webhosting, Mail Services, Managed Servers, ...
Unix is very simple, but it takes a genius to understand the simplicity.
(Dennis Ritchie)

Reply | Threaded
Open this post in threaded view
|

Re: Openbgpd & kernel tuning

Will H. Backman
Henning Brauer wrote:

> * Marcel Prisi <[hidden email]> [2006-03-08 16:42]:
>
>>OpenBGPD's config seems OK, but I need some help about OpenBSD's tunable
>>parameters using sysctl.
>
>
> the only thing you might want to change is
>   net.inet.ip.ifq.maxlen
> the default is a little low for routing at higher speeds. 250 seems
> a good compromise for many higher-bandwidth routers.
>

What is the easiest way to know when you are hitting the limit?  Does it
just drop new connections?

Reply | Threaded
Open this post in threaded view
|

Re: Openbgpd & kernel tuning

Henning Brauer
* Will H. Backman <[hidden email]> [2006-03-08 19:17]:

> Henning Brauer wrote:
> >* Marcel Prisi <[hidden email]> [2006-03-08 16:42]:
> >>OpenBGPD's config seems OK, but I need some help about OpenBSD's tunable
> >>parameters using sysctl.
> >the only thing you might want to change is
> >  net.inet.ip.ifq.maxlen
> >the default is a little low for routing at higher speeds. 250 seems
> >a good compromise for many higher-bandwidth routers.
> What is the easiest way to know when you are hitting the limit?  Does it
> just drop new connections?

monitoring the congestion counter in pfctl -si helps a lot.

you don't want too long queues tho, that is contraproductive.

sorry, if there was an easy rule, we'd do it automagically.

--
BS Web Services, http://www.bsws.de/
OpenBSD-based Webhosting, Mail Services, Managed Servers, ...
Unix is very simple, but it takes a genius to understand the simplicity.
(Dennis Ritchie)

Reply | Threaded
Open this post in threaded view
|

Re: Openbgpd & kernel tuning

Chris Cappuccio
In reply to this post by Marcel Prisi
Marcel Prisi [[hidden email]] wrote:
>
> I read some old threads about too small tcp.sendspace / tcp.recvspace in
> 3.4 time that used to hit performance so I thought it would be useful.
>

These settings only affect TCP sessions that connect directly to that system.

In other words, they don't do anything on a router.

> The others were about DOS prevention.
>

If the box isn't completely livelocked, you can Use tcpdump to figure out
which IPs you need your upstream to block traffic from or to in the event
of a DoS

If you're lucky, most of the traffic will either come from one network
or most of it will go to a small number of IP addresses on your side.  If
your upstream provider blocks that traffic, then your pipe isn't full
anymore.  If you're not lucky, you're screwed, and you need to have
more bandwidth than your attacker to sustain an attack.  

Reply | Threaded
Open this post in threaded view
|

Re: Openbgpd & kernel tuning

Matt Rowley
In reply to this post by Henning Brauer
> monitoring the congestion counter in pfctl -si helps a lot.
>
> you don't want too long queues tho, that is contraproductive.

What are the consequences of ifq set too large?

--Matt

Reply | Threaded
Open this post in threaded view
|

Re: Openbgpd & kernel tuning

Ted Unangst-2
On 3/8/06, Matt Rowley <[hidden email]> wrote:
> > monitoring the congestion counter in pfctl -si helps a lot.
> >
> > you don't want too long queues tho, that is contraproductive.
>
> What are the consequences of ifq set too large?

packets go in the queue and don't come out.

Reply | Threaded
Open this post in threaded view
|

Re: Openbgpd & kernel tuning

jared r r spiegel
In reply to this post by Marcel Prisi
On Wed, Mar 08, 2006 at 06:24:02PM +0100, Marcel Prisi wrote:
>
> I read some old threads about too small tcp.sendspace / tcp.recvspace in
> 3.4 time that used to hit performance so I thought it would be useful.

  of all the times i dicked with those, my results have been that
  any performance gain (*if* any) i get in throughput for long-lived connections
  which ride on an essentially trouble-free link between me and the
  other host i'm testing on are offset by an increased latency i
  see between me and my default gateway.

  shorter connections, i've found, tend to appreciate not increasing those values.

  eg, i start a ping to default my gateway, observe the RTT; observe
  my tcp sysctl crappers, start a large tcp xfer to someone else in town
  who is at most one router (and a few ATM hops) away; observe
  the throughput of the xfer and observe the current RTT to the gateway
  while that is happening; dick with sysctl {send,recv}space; repeat.

  don't have hard records of the results, but iirc, where normally my
  RTT is on the order of 12ms, with default tcp sizes, under full
  congestion (tested up and down streams individually) i was seeing
  something like a jump to 600ms while downstream saturation and
  2200ms while full upstream saturation; jacking the tcp to 64k like
  everyone uses increased the throughput of the transfer by some 8-15%,
  but sent the RTT to something like 1500ms/8000ms respectively.

  ALTQ was not in use.

  one may say 'tcp window size has nothing to do with ping times',
  but in my non-expert experience it certainly does when the pings and
  the tcp are sharing the same congested pipe; or at least when an
  encapsulation from IP to ATM (dsl) is involved.

  there may be other factors too, and i am actually at a loss to explain
  it beyond the conclusions that one can draw based on watching the results,
  but the delightful thing is it doesn't particularly matter why to me
  since the default settings have proven themselves (to me) to be just
  fine in all cases (of my connection), and thus far i've found that
  changing them usually degrades one trait of my link more than it improves
  another one that i value less.

  profile at the dslam is something similar to 3Mb down / 768Kb up, interleaved.

--

  jared

[ openbsd 3.9-beta GENERIC ( jan 30 ) // i386 ]

Reply | Threaded
Open this post in threaded view
|

Re: Openbgpd & kernel tuning

Henning Brauer
In reply to this post by Ted Unangst-2
* Ted Unangst <[hidden email]> [2006-03-08 23:28]:
> On 3/8/06, Matt Rowley <[hidden email]> wrote:
> > > monitoring the congestion counter in pfctl -si helps a lot.
> > > you don't want too long queues tho, that is contraproductive.
> > What are the consequences of ifq set too large?
> packets go in the queue and don't come out.

it is not that simple.

first, it burns kernel memory. tho unless you set the queue length to
totally insane high values that should not be much of an issue.

then it can increase latency quite a bit.
packets do not get dropped early enough but just sit in the queue. tcp
relies on packets to be dropped to adjust to available bandwidth.

last not least, it can prevent the congestion indicator to be set in
time and thus hurt your machine badly. details about taht are on
http://bulabula.org/papers/opencon05/mgp00027.html and following slides.

there's more, but that should scare you enough :)

--
BS Web Services, http://www.bsws.de/
OpenBSD-based Webhosting, Mail Services, Managed Servers, ...
Unix is very simple, but it takes a genius to understand the simplicity.
(Dennis Ritchie)