Load balancing with DSR

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Load balancing with DSR

Linden Varley
Hi,

Anyone know of any load balancing software for OpenBSD that can do
direct-server return? (our load balancers (openbsd boxes) are co-located
and we pay for all data bandwidth).

Something like BalanceNG (which unfortunately doesnt run on OpenBSD)
woudl be ideal.

It is generally for http layer requests but I don't think apache
re-directs will suffice.

Cheers,
Linden.

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

bofh-6
On 6/12/07, Linden Varley <[hidden email]> wrote:
> It is generally for http layer requests but I don't think apache
> re-directs will suffice.

You may want to look at pound.  A lot of people seem to like it.

--
"This officer's men seem to follow him merely out of idle curiosity."
-- Sandhurst officer cadet evaluation.

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

Lars Hansson
In reply to this post by Linden Varley
Linden Varley wrote:
> Anyone know of any load balancing software for OpenBSD that can do
> direct-server return? (our load balancers (openbsd boxes) are co-located
> and we pay for all data bandwidth).

hoststated?

---
Lars Hansson

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

Pierre-Yves Ritschard-4
On Wed, 13 Jun 2007 10:54:58 +0800
Lars Hansson <[hidden email]> wrote:

> Linden Varley wrote:
> > Anyone know of any load balancing software for OpenBSD that can do
> > direct-server return? (our load balancers (openbsd boxes) are
> > co-located and we pay for all data bandwidth).
>
> hoststated?
>
No, hoststated won't do DSR yet, neither will any load balancers on
OpenBSD.
DSR needs Layer 2 trickery that is not possible with OpenBSD.
Maybe someday, it is on my todo-list if I find a clean way to do it.

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

Pierre-Yves Ritschard-4
On Wed, 13 Jun 2007 15:40:36 +1000
Darren Tucker <[hidden email]> wrote:

> Would it be possible to to this the way the IBM eNetwork dispatchers
> used to do this?  Put all of the machines on the same broadcast
> domain, then:
>
> 1. add a static published arp entry for the cluster address on the
> balancer with its own mac address so packets aimed at the cluster
> address will go to the balancer.
>
> 2. configure all cluster members with a loopback interface with the
> cluster address.
>
> 3. use route-to pf rules with a next-hop to punt incoming packets to
> various nodes in the cluster

I think all load balancers implementing direct server return / direct
routing use this trick.
You're not going to be able to get away without messing with arp so
you're bound to a single broadcast domain.
 
Your scenario should be tried out, yes, but it is still just a ugly
hack if you ask me :)

Now you still can't really make this work with hoststated or any
other LB on OpenBSD. I'd still like to find an elegant way to do this
and integrate it with hoststated.

And just for the record what you said maps to:

pass in on $ext_if route-to { $webh1, $webh2 } round-robin proto tcp \
 from any to $virt_ip port http no state
pass out on $int_if from any to $virt_ip port http no state

If I get the occasion I'll try it out and see how that works.
I also wonder how it would behave when setting the arp entry to that of
a carp interface.

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

Darren Tucker
Pierre-Yves Ritschard wrote:
> On Wed, 13 Jun 2007 15:40:36 +1000
> Darren Tucker <[hidden email]> wrote:
[...]

>> 1. add a static published arp entry for the cluster address on the
>> balancer with its own mac address so packets aimed at the cluster
>> address will go to the balancer.
>>
>> 2. configure all cluster members with a loopback interface with the
>> cluster address.
>>
>> 3. use route-to pf rules with a next-hop to punt incoming packets to
>> various nodes in the cluster
>
> I think all load balancers implementing direct server return / direct
> routing use this trick.
> You're not going to be able to get away without messing with arp so
> you're bound to a single broadcast domain.

As long as you get the route-to right, all you need for this to work is
for the incoming packets to be routed to the balancer.  What if, eg,
bgpd was configured to advertise a route to the /32 containing the
cluster address via the balancer's real IP?

> Your scenario should be tried out, yes, but it is still just a ugly
> hack if you ask me :)
>
> Now you still can't really make this work with hoststated or any
> other LB on OpenBSD. I'd still like to find an elegant way to do this
> and integrate it with hoststated.
>
> And just for the record what you said maps to:
>
> pass in on $ext_if route-to { $webh1, $webh2 } round-robin proto tcp \
>  from any to $virt_ip port http no state
> pass out on $int_if from any to $virt_ip port http no state

Wouldn't you need some kind of state here?  Otherwise there's no
guarantee of the packets for a given connection always being routed to
the same physical server.

> If I get the occasion I'll try it out and see how that works.
> I also wonder how it would behave when setting the arp entry to that of
> a carp interface.

--
Darren Tucker (dtucker at zip.com.au)
GPG key 8FF4FA69 / D9A3 86E9 7EEE AF4B B2D4  37C9 C982 80C7 8FF4 FA69
     Good judgement comes with experience. Unfortunately, the experience
usually comes from bad judgement.

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

Pierre-Yves Ritschard-4
> > pass in on $ext_if route-to { $webh1, $webh2 } round-robin proto
> > tcp \ from any to $virt_ip port http no state
> > pass out on $int_if from any to $virt_ip port http no state
>
> Wouldn't you need some kind of state here?  Otherwise there's no
> guarantee of the packets for a given connection always being routed
> to the same physical server.
>
State is useless here as packets won't come back through this router.
Moreover, you're going to block some packets in some cases since you
only see half of the trafic.
If you want a sticky behavior you can change the pool type to
source-hash.

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

Reyk Floeter-2
In reply to this post by Pierre-Yves Ritschard-4
On Wed, Jun 13, 2007 at 06:42:24AM +0200, Pierre-Yves Ritschard wrote:

> On Wed, 13 Jun 2007 10:54:58 +0800
> Lars Hansson <[hidden email]> wrote:
>
> > Linden Varley wrote:
> > > Anyone know of any load balancing software for OpenBSD that can do
> > > direct-server return? (our load balancers (openbsd boxes) are
> > > co-located and we pay for all data bandwidth).
> >
> > hoststated?
> >
> No, hoststated won't do DSR yet, neither will any load balancers on
> OpenBSD.
> DSR needs Layer 2 trickery that is not possible with OpenBSD.
> Maybe someday, it is on my todo-list if I find a clean way to do it.
>

i don't like the idea about "DSR", it sounds like an evil hack to get
some performance at the wrong place. it is better to focus on
improving the pf/network stack performance itself and to be able to do
traffic filtering and normalization on the loadbalancers.

reyk

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

Pierre-Yves Ritschard-4
On Wed, 13 Jun 2007 17:05:44 +0200
Reyk Floeter <[hidden email]> wrote:
>
> i don't like the idea about "DSR", it sounds like an evil hack to get
> some performance at the wrong place. it is better to focus on
> improving the pf/network stack performance itself and to be able to do
> traffic filtering and normalization on the loadbalancers.
>
>
There are scenari where DSR/DR really solves problem, some that the
best pf network stack cannot solve.
If DSR/DR ever finds it way into OpenBSD (through pf, hoststated, ...)
it will be because we find a clean way to do it.

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

bseklecki-2
In reply to this post by Linden Varley
This is like "Local Triangulation" in Radware-speak? (Don't know what
F5) calls it.  Basically you bring up an alias on lo0 or lo1 primary as
the inet4 of your HAL4 address and as long as everything is in the same
subnet...

~BAS

On Wed, 2007-06-13 at 12:25 +1000, Linden Varley wrote:

> Hi,
>
> Anyone know of any load balancing software for OpenBSD that can do
> direct-server return? (our load balancers (openbsd boxes) are co-located
> and we pay for all data bandwidth).
>
> Something like BalanceNG (which unfortunately doesnt run on OpenBSD)
> woudl be ideal.
>
> It is generally for http layer requests but I don't think apache
> re-directs will suffice.
>
> Cheers,
> Linden.
>
--
Brian A. Seklecki <[hidden email]>
Collaborative Fusion, Inc.




IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited.  Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

bseklecki-2
In reply to this post by Pierre-Yves Ritschard-4
Such as Distributed computing environments where you have your HAL4
service VIP on the same segment/subnet as your distributed server farm.

Or HA databses

~BAS

On Wed, 2007-06-13 at 17:49 +0200, Pierre-Yves Ritschard wrote:
> best pf network stack cannot solve.
--
Brian A. Seklecki <[hidden email]>
Collaborative Fusion, Inc.




IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited.  Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

Reyk Floeter-2
On Wed, Jun 13, 2007 at 12:36:33PM -0400, Brian A. Seklecki wrote:
> Such as Distributed computing environments where you have your HAL4
> service VIP on the same segment/subnet as your distributed server farm.
>

so they should redesign their network instead of inventing crazy
features. this DSR sounds like a hack, an evil workaround which can be
sold as a $ feature by the named companies.

i still do not believe in it, because it bypasses the main benefit of
OpenBSD-based loadbalancing: running a good firewall and "network
optimizer" in front of the loadbalanced servers. and it does some very
strange tricks with the network stacks.

> Or HA databses
>

and there is no better way to connect them?

> ~BAS
>
> On Wed, 2007-06-13 at 17:49 +0200, Pierre-Yves Ritschard wrote:
> > best pf network stack cannot solve.
> --
> Brian A. Seklecki <[hidden email]>
> Collaborative Fusion, Inc.
>
>
>
>
> IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited.  Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

Linden Varley
In reply to this post by Reyk Floeter-2
The only reason we need DSR is our load-balancers are co-located and we
have a limit on data usage so the connection needs to be offloaded to
the server/client and not proxied as this would get quite expensive with
the traffic flowing through our co-location pipe.

Might have to move to Linux with BalanceNG for the time-being.

Cheers,
Linden.

Reyk Floeter wrote:

> On Wed, Jun 13, 2007 at 06:42:24AM +0200, Pierre-Yves Ritschard wrote:
>  
>> On Wed, 13 Jun 2007 10:54:58 +0800
>> Lars Hansson <[hidden email]> wrote:
>>
>>    
>>> Linden Varley wrote:
>>>      
>>>> Anyone know of any load balancing software for OpenBSD that can do
>>>> direct-server return? (our load balancers (openbsd boxes) are
>>>> co-located and we pay for all data bandwidth).
>>>>        
>>> hoststated?
>>>
>>>      
>> No, hoststated won't do DSR yet, neither will any load balancers on
>> OpenBSD.
>> DSR needs Layer 2 trickery that is not possible with OpenBSD.
>> Maybe someday, it is on my todo-list if I find a clean way to do it.
>>
>>    
>
> i don't like the idea about "DSR", it sounds like an evil hack to get
> some performance at the wrong place. it is better to focus on
> improving the pf/network stack performance itself and to be able to do
> traffic filtering and normalization on the loadbalancers.
>
> reyk

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

Adam-29
Linden Varley <[hidden email]> wrote:

> The only reason we need DSR is our load-balancers are co-located and we
> have a limit on data usage so the connection needs to be offloaded to
> the server/client and not proxied as this would get quite expensive with
> the traffic flowing through our co-location pipe.

Are you actually *stuck* with this messed up setup for some reason?  Why
can't you just move your web servers behind the load balancers where they
belong?

Adam

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

Linden Varley
Load-balancers were co-located for redundancy reasons I believe. Its
just a shame traffic in/out is paid-for so even if web-servers were also
co-located then traffic will still be metered.

We could bring the load-balancers into our network to stop this problem
but we have two-sites on different subnets and I need the load-balancers
to have a common-ip to which they could then proxy to one of the two sites.

- Linden

Adam wrote:

> Linden Varley <[hidden email]> wrote:
>
>  
>> The only reason we need DSR is our load-balancers are co-located and we
>> have a limit on data usage so the connection needs to be offloaded to
>> the server/client and not proxied as this would get quite expensive with
>> the traffic flowing through our co-location pipe.
>>    
>
> Are you actually *stuck* with this messed up setup for some reason?  Why
> can't you just move your web servers behind the load balancers where they
> belong?
>
> Adam

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

Adam-29
Linden Varley <[hidden email]> wrote:

> Load-balancers were co-located for redundancy reasons I believe. Its
> just a shame traffic in/out is paid-for so even if web-servers were also
> co-located then traffic will still be metered.

If your web servers and load balancers aren't on the same network then
you can't do DSR anyways.  ISP_A where your web servers are is
(hopefully) not going to let you send out traffic with a source IP
belonging to ISP_B (where your load balancers are).

Adam

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

Linden Varley
We host our own web-servers so DSR shouldn't be a problem. Will probably
get rid of the co-located balancers and bring them inside our network as
we dont really gain anything from co-locating. Might just use something
simple like lbnamed !

Adam wrote:

> Linden Varley <[hidden email]> wrote:
>
>  
>> Load-balancers were co-located for redundancy reasons I believe. Its
>> just a shame traffic in/out is paid-for so even if web-servers were also
>> co-located then traffic will still be metered.
>>    
>
> If your web servers and load balancers aren't on the same network then
> you can't do DSR anyways.  ISP_A where your web servers are is
> (hopefully) not going to let you send out traffic with a source IP
> belonging to ISP_B (where your load balancers are).
>
> Adam

Reply | Threaded
Open this post in threaded view
|

Re: Load balancing with DSR

bseklecki-2
In reply to this post by Reyk Floeter-2
No argument there on the pragmatics.

But it does work, and a lot of places use it.  

~BAS

FYI I wasn't advocating implementing it; just providing background.  If
you want it, shell the $500k for the hardware L.B.

On Thu, 2007-06-14 at 00:07 +0200, Reyk Floeter wrote:

> On Wed, Jun 13, 2007 at 12:36:33PM -0400, Brian A. Seklecki wrote:
> > Such as Distributed computing environments where you have your HAL4
> > service VIP on the same segment/subnet as your distributed server farm.
> >
>
> so they should redesign their network instead of inventing crazy
> features. this DSR sounds like a hack, an evil workaround which can be
> sold as a $ feature by the named companies.
>
> i still do not believe in it, because it bypasses the main benefit of
> OpenBSD-based loadbalancing: running a good firewall and "network
> optimizer" in front of the loadbalanced servers. and it does some very
> strange tricks with the network stacks.
>
> > Or HA databses
> >
>
> and there is no better way to connect them?
>
> > ~BAS
> >
> > On Wed, 2007-06-13 at 17:49 +0200, Pierre-Yves Ritschard wrote:
> > > best pf network stack cannot solve.
> > --
> > Brian A. Seklecki <[hidden email]>
> > Collaborative Fusion, Inc.
> >
> >
> >
> >
> > IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited.  Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.
> >
>
>
>
>
>
>
--
Brian A. Seklecki <[hidden email]>
Collaborative Fusion, Inc.




IMPORTANT: This message contains confidential information and is intended only for the individual named. If the reader of this message is not an intended recipient (or the individual responsible for the delivery of this message to an intended recipient), please be advised that any re-use, dissemination, distribution or copying of this message is prohibited.  Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system.