HA: pair of firewalls, 2 switches and 1 server

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
27 messages Options
12
Reply | Threaded
Open this post in threaded view
|

HA: pair of firewalls, 2 switches and 1 server

Axel Rau
Hi all,

I have a pair of redundant firewalls (obsd 4.6) and a server (fbsd 8.0):

        +---+                      +------+
        |   |                      |      |
    ----+fw1+----------+ +---------+      |
   carp0|   |carp1     | |      em0|      |
        |   |          | |         |      |
        +-+-+        +-+-+-+       |      |
          |          | sw  |       |Server|
        +-+-+        +-+-+-+       | fbsd |
        |   |          | |         |      |
    ----+fw2+----------+ +---------+      |
   carp0|   |carp1              em1|      |
        |   |                      |      |
        +---+          DMZ         +------+

We all know, the switch is the sigle point of failure.
Even worse, when it fails the carp0 pair starts flapping, disturbing
other firewall traffic.
So, how to resolve this?

Trunking would only be possible between 2 boxes, not 3.
Carp on top of trunk?
2 Carp pairs on the firewalls and 1 pair at the server?

If I get it right, the physical LAN should look like this:

        +---+                      +------+
        |   |        +-----+       |      |
    ----+fw1+--------+ sw1 +-------+      |
   carp0|   +--+     +-+-+-+    em0|      |
        |   |  |       |           |      |
        +-+-+  |  +----+           |      |
          |    |  |                |Server|
        +-+-+  +--|------+         | fbsd |
        |   |     |      |         |      |
        |   +-----+  +-+-+-+       |      |
    ----+fw2+--------+ sw2 +-------+      |
   carp0|   |        +-----+    em1|      |
        +---+                      +------+

Switches must have Spanning Tree support (RSTP), so I hope a pair of
Netgear GS108T can do this.

Any proposals highly appreciated,
Axel
---
[hidden email]  PGP-Key:29E99DD6  +49 151 2300 9283  computing @
chaos claudius

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Guido Tschakert
Axel Rau schrieb:

> Hi all,
>
> I have a pair of redundant firewalls (obsd 4.6) and a server (fbsd 8.0):
>
>        +---+                      +------+
>        |   |                      |      |
>    ----+fw1+----------+ +---------+      |
>   carp0|   |carp1     | |      em0|      |
>        |   |          | |         |      |
>        +-+-+        +-+-+-+       |      |
>          |          | sw  |       |Server|
>        +-+-+        +-+-+-+       | fbsd |
>        |   |          | |         |      |
>    ----+fw2+----------+ +---------+      |
>   carp0|   |carp1              em1|      |
>        |   |                      |      |
>        +---+          DMZ         +------+
>
> We all know, the switch is the sigle point of failure.

Hi,

I would say your Server is __the__ single point of failure (sure the
switch is also a spof but normally I'm more worried about servers then
switches)

guido

> Even worse, when it fails the carp0 pair starts flapping, disturbing
> other firewall traffic.
> So, how to resolve this?
>
> Trunking would only be possible between 2 boxes, not 3.
> Carp on top of trunk?
> 2 Carp pairs on the firewalls and 1 pair at the server?
>
> If I get it right, the physical LAN should look like this:
>
>        +---+                      +------+
>        |   |        +-----+       |      |
>    ----+fw1+--------+ sw1 +-------+      |
>   carp0|   +--+     +-+-+-+    em0|      |
>        |   |  |       |           |      |
>        +-+-+  |  +----+           |      |
>          |    |  |                |Server|
>        +-+-+  +--|------+         | fbsd |
>        |   |     |      |         |      |
>        |   +-----+  +-+-+-+       |      |
>    ----+fw2+--------+ sw2 +-------+      |
>   carp0|   |        +-----+    em1|      |
>        +---+                      +------+
>
> Switches must have Spanning Tree support (RSTP), so I hope a pair of
> Netgear GS108T can do this.
>
> Any proposals highly appreciated,
> Axel
> ---
> [hidden email]  PGP-Key:29E99DD6  +49 151 2300 9283  computing @
> chaos claudius

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Leonardo Carneiro - Veltrac
In reply to this post by Axel Rau
Axel Rau wrote:

> Hi all,
>
> I have a pair of redundant firewalls (obsd 4.6) and a server (fbsd 8.0):
>
>        +---+                      +------+
>        |   |                      |      |
>    ----+fw1+----------+ +---------+      |
>   carp0|   |carp1     | |      em0|      |
>        |   |          | |         |      |
>        +-+-+        +-+-+-+       |      |
>          |          | sw  |       |Server|
>        +-+-+        +-+-+-+       | fbsd |
>        |   |          | |         |      |
>    ----+fw2+----------+ +---------+      |
>   carp0|   |carp1              em1|      |
>        |   |                      |      |
>        +---+          DMZ         +------+
>
> We all know, the switch is the sigle point of failure.
> Even worse, when it fails the carp0 pair starts flapping, disturbing
> other firewall traffic.
> So, how to resolve this?
>
> Trunking would only be possible between 2 boxes, not 3.
> Carp on top of trunk?
> 2 Carp pairs on the firewalls and 1 pair at the server?
>
> If I get it right, the physical LAN should look like this:
>
>        +---+                      +------+
>        |   |        +-----+       |      |
>    ----+fw1+--------+ sw1 +-------+      |
>   carp0|   +--+     +-+-+-+    em0|      |
>        |   |  |       |           |      |
>        +-+-+  |  +----+           |      |
>          |    |  |                |Server|
>        +-+-+  +--|------+         | fbsd |
>        |   |     |      |         |      |
>        |   +-----+  +-+-+-+       |      |
>    ----+fw2+--------+ sw2 +-------+      |
>   carp0|   |        +-----+    em1|      |
>        +---+                      +------+
>
> Switches must have Spanning Tree support (RSTP), so I hope a pair of
> Netgear GS108T can do this.
>
> Any proposals highly appreciated,
> Axel
> ---
> [hidden email]  PGP-Key:29E99DD6  +49 151 2300 9283  computing @
> chaos claudius
IMHO, the second scenario you draw solves the problem in a very elegant
way. Beside, STP and RSTP-enabled switches are becoming less expansive
in the last years.

Best regards.

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Axel Rau
Am 18.05.2010 um 14:20 schrieb Leonardo Carneiro - Veltrac:

> IMHO, the second scenario you draw solves the problem in a very elegant way.
Beside, STP and RSTP-enabled switches are becoming less expansive in the last
years.
Yes, but what carps/trunks do I need?

Axel
---
[hidden email]  PGP-Key:29E99DD6  +49 151 2300 9283  computing @ chaos
claudius

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Axel Rau
In reply to this post by Guido Tschakert
Am 18.05.2010 um 14:11 schrieb Guido Tschakert:

> I would say your Server is __the__ single point of failure (sure the
> switch is also a spof but normally I'm more worried about servers then
> switches)
Yes, but it has 2 power supplies and redundant disks. If the mini pwr supply
of the single switch dies, I'm loosing.
Also a 2nd server is in the pipeline...

Axel
---
[hidden email]  PGP-Key:29E99DD6  +49 151 2300 9283  computing @ chaos
claudius

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Guido Tschakert
Axel Rau schrieb:
> Am 18.05.2010 um 14:11 schrieb Guido Tschakert:
>
>> I would say your Server is __the__ single point of failure (sure the
>> switch is also a spof but normally I'm more worried about servers then
>> switches)
> Yes, but it has 2 power supplies and redundant disks. If the mini pwr supply of the single switch dies, I'm loosing.
Oh, yes and it has two mainboards and you have two ups for all the
things. ;-)
Have you thought of two internet connections from two different providers?

Sorry I don't want to bother you, I just want to say that achieving
redundancy is not that easy like you described it in your first message.

The first question is:
What problem are you trying to resolve?
Or in your case: How much redundancy do you want/need?

> Also a 2nd server is in the pipeline...

Ok, that's fine.

guido

Btw: it would be great for the archive, if you got it working, that you
send a message to the list, describing your configuration.

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Axel Rau
Am 19.05.2010 um 07:59 schrieb Guido Tschakert:

> What problem are you trying to resolve?


I will clarify:

       +---+                      +------+
       |   |        +-----+       |      |
   ----+fw1+--------+ sw1 +-------+      |
  carp0|   +--+     +-+-+-+    em0|      |
       |   |  |       |           |      |
       +-+-+  |  +----+           |      |
         |    |  |                |Server|
       +-+-+  +--|------+         | fbsd |
       |   |     |      |         |      |
       |   +-----+  +-+-+-+       |      |
   ----+fw2+--------+ sw2 +-------+      |
  carp0|   |        +-----+    em1|      |
       +---+                      +------+

Server uses fw1/fw2 as default gateway(s).
Server has a bunch of IPs. I can't add these as aliases to either em0
or em1 (would be single point o failure).
I need a virtual interface, like a trunk, to which I can tie the IPs.

A trunk connects 2 hosts (AFAIK), in my case, I have 3.
I could reduce the pair fw1/fw2 to one virtual system, using 2 carp
interfaces.
This way, I would have a valid configuration of 2 hosts for the trunk,
with 2 interfaces on each side.

Now the question: Can I put a trunk on top of a carp?
AFAIK No.
What do you mean?

Are there other possibilities to connect the boxes with the above
functionality?

Axel
---
[hidden email]  PGP-Key:29E99DD6  +49 151 2300 9283  computing @
chaos claudius

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Henning Brauer
* Axel Rau <[hidden email]> [2010-05-19 10:34]:
> Now the question: Can I put a trunk on top of a carp?

you put carp on top of the trunk of course.

--
Henning Brauer, [hidden email], [hidden email]
BS Web Services, http://bsws.de
Full-Service ISP - Secure Hosting, Mail and DNS Services
Dedicated Servers, Rootservers, Application Hosting

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Axel Rau
Am 20.05.2010 um 00:04 schrieb Henning Brauer:

> * Axel Rau <[hidden email]> [2010-05-19 10:34]:
>> Now the question: Can I put a trunk on top of a carp?
>
> you put carp on top of the trunk of course.
OK.
Can I have a trunk connected to 2 different switches then?

Axel
---
[hidden email]  PGP-Key:29E99DD6  +49 151 2300 9283  computing @
chaos claudius

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Graham Allan
On Thu, May 20, 2010 at 07:02:23PM +0200, Axel Rau wrote:
> Am 20.05.2010 um 00:04 schrieb Henning Brauer:
>
> >* Axel Rau <[hidden email]> [2010-05-19 10:34]:
> >>Now the question: Can I put a trunk on top of a carp?
> >
> >you put carp on top of the trunk of course.
> OK.
> Can I have a trunk connected to 2 different switches then?
 
Not normally. Some higher-end switches can support this, eg the
HP Procurve switches running their K-series software can do something
they call distributed trunking (and no doubt Cisco and other vendors all
call it something else). But as I think you were talking about using
cheapish Netgear switches it's unlikely to be possible.

--
-------------------------------------------------------------------------
Graham Allan
School of Physics and Astronomy - University of Minnesota
-------------------------------------------------------------------------

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Henning Brauer
* Graham Allan <[hidden email]> [2010-05-20 19:23]:

> On Thu, May 20, 2010 at 07:02:23PM +0200, Axel Rau wrote:
> > Am 20.05.2010 um 00:04 schrieb Henning Brauer:
> >
> > >* Axel Rau <[hidden email]> [2010-05-19 10:34]:
> > >>Now the question: Can I put a trunk on top of a carp?
> > >
> > >you put carp on top of the trunk of course.
> > OK.
> > Can I have a trunk connected to 2 different switches then?
>  
> Not normally. Some higher-end switches can support this, eg the
> HP Procurve switches running their K-series software can do something
> they call distributed trunking (and no doubt Cisco and other vendors all
> call it something else). But as I think you were talking about using
> cheapish Netgear switches it's unlikely to be possible.

well, lacp usually doesn't work across switches. but lacp is not the
only mode trunk supports. roundrobin definately works across switches
- how well might depend on your switches. works well for me on
procurve with E-series software which doesn't do distributed trunking
afair.<

--
Henning Brauer, [hidden email], [hidden email]
BS Web Services, http://bsws.de
Full-Service ISP - Secure Hosting, Mail and DNS Services
Dedicated Servers, Rootservers, Application Hosting

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Jussi Peltola
On Thu, May 20, 2010 at 07:28:55PM +0200, Henning Brauer wrote:

> * Graham Allan <[hidden email]> [2010-05-20 19:23]:
> > On Thu, May 20, 2010 at 07:02:23PM +0200, Axel Rau wrote:
> > > Am 20.05.2010 um 00:04 schrieb Henning Brauer:
> > >
> > > >* Axel Rau <[hidden email]> [2010-05-19 10:34]:
> > > >>Now the question: Can I put a trunk on top of a carp?
> > > >
> > > >you put carp on top of the trunk of course.
> > > OK.
> > > Can I have a trunk connected to 2 different switches then?
> >  
> > Not normally. Some higher-end switches can support this, eg the
> > HP Procurve switches running their K-series software can do something
> > they call distributed trunking (and no doubt Cisco and other vendors all
> > call it something else). But as I think you were talking about using
> > cheapish Netgear switches it's unlikely to be possible.
>
> well, lacp usually doesn't work across switches. but lacp is not the
> only mode trunk supports. roundrobin definately works across switches
> - how well might depend on your switches. works well for me on
> procurve with E-series software which doesn't do distributed trunking
> afair.<
 
How about the warnings about packet reordering and interactions with
TCP? I'd guess it's not really such a big issue if you have two
identical switches and routers. But shouldn't the hash based trunk modes
work just fine, too (with the caveat that some flows will stop working
completely if the other switch fails in some ways while roundrobin will
cause half of the packets to be blackholed, keeping badly degraded
connectivity)

Also, the switches need to be separate; connecting them directly may
cause learned MACs to flap between the real host port and the cable
between the switches and make the trunk receive its own traffic on the
other port.

Fail-over trunk should work just fine, too. But see the following
paragraphs...

If you want reliability, do not use cheap switches. Switch power
supplies are not the failure mode you want to avoid. I don't remember
seeing very many at all, however I've seen lots of crappy ones lose
their config or stop forwarding completely while keeping the link up.

I have two identical "core" switches in one (not really so critical at
all) place running OSPF, with a bunch of routers connecting to both
switches for redundancy. Works pretty well and there has even been a
config reset incident, which didn't break anything - because OSPF can
detect link failures. Trying to do the same all the way to the end hosts
(i.e.  without a routing protocol) is pretty difficult.

One pseudo solution is to run a bridge instead of trunk on the 2
interfaces and use STP for fail-over; I find that too yucky to solve a
problem that doesn't really exist (just buy a reliable switch with a
redundant power supply or connect the single one to a good UPS)

However, if you need to ask if you can run a trunk on top of a carp, do
yourself a favor and use a single switch. There will be less downtime.

Jussi Peltola

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Henning Brauer
* Jussi Peltola <[hidden email]> [2010-05-20 20:07]:

> On Thu, May 20, 2010 at 07:28:55PM +0200, Henning Brauer wrote:
> > * Graham Allan <[hidden email]> [2010-05-20 19:23]:
> > > On Thu, May 20, 2010 at 07:02:23PM +0200, Axel Rau wrote:
> > > > Am 20.05.2010 um 00:04 schrieb Henning Brauer:
> > > >
> > > > >* Axel Rau <[hidden email]> [2010-05-19 10:34]:
> > > > >>Now the question: Can I put a trunk on top of a carp?
> > > > >
> > > > >you put carp on top of the trunk of course.
> > > > OK.
> > > > Can I have a trunk connected to 2 different switches then?
> > >  
> > > Not normally. Some higher-end switches can support this, eg the
> > > HP Procurve switches running their K-series software can do something
> > > they call distributed trunking (and no doubt Cisco and other vendors all
> > > call it something else). But as I think you were talking about using
> > > cheapish Netgear switches it's unlikely to be possible.
> >
> > well, lacp usually doesn't work across switches. but lacp is not the
> > only mode trunk supports. roundrobin definately works across switches
> > - how well might depend on your switches. works well for me on
> > procurve with E-series software which doesn't do distributed trunking
> > afair.<
>  
> How about the warnings about packet reordering and interactions with
> TCP?

never ran into such issues. too lazy right now to check wether trunk
deals with that in roundrobin or wether i just got lucky.

> I'd guess it's not really such a big issue if you have two
> identical switches and routers. But shouldn't the hash based trunk modes
> work just fine, too (with the caveat that some flows will stop working
> completely if the other switch fails in some ways while roundrobin will
> cause half of the packets to be blackholed, keeping badly degraded
> connectivity)

err. wait. if the switch fails for real the link goes down and the
port is just taken out of the active ports on the trunk.

now there are of course more subtle ways of failure that could lead to
the above scenario. but how likely is that really? and would this
issue be your real problem then?
 
> Also, the switches need to be separate; connecting them directly may
> cause learned MACs to flap between the real host port and the cable
> between the switches and make the trunk receive its own traffic on the
> other port.

that is the "may depend on your switch" part. I have not seen any
problems with interconnected procurves, 5300XL series.

> Fail-over trunk should work just fine, too.

indeed.

> If you want reliability, do not use cheap switches. Switch power
> supplies are not the failure mode you want to avoid. I don't remember
> seeing very many at all, however I've seen lots of crappy ones lose
> their config or stop forwarding completely while keeping the link up.

guess i lack the cheap shit switch experience.

i do have experience with expensive shit switches tho. they suck in
many different ways, never seen the behaviour you describe above tho.

but then, ever since using said procurves, that is history.

> I have two identical "core" switches in one (not really so critical at
> all) place running OSPF, with a bunch of routers connecting to both
> switches for redundancy. Works pretty well and there has even been a
> config reset incident, which didn't break anything - because OSPF can
> detect link failures. Trying to do the same all the way to the end hosts
> (i.e.  without a routing protocol) is pretty difficult.

i would never ever run any L3 on switches.

> However, if you need to ask if you can run a trunk on top of a carp, do
> yourself a favor and use a single switch. There will be less downtime.

that is something i could subscribe to :)

--
Henning Brauer, [hidden email], [hidden email]
BS Web Services, http://bsws.de
Full-Service ISP - Secure Hosting, Mail and DNS Services
Dedicated Servers, Rootservers, Application Hosting

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Henning Brauer
* Henning Brauer <[hidden email]> [2010-05-20 20:23]:

> * Jussi Peltola <[hidden email]> [2010-05-20 20:07]:
> > On Thu, May 20, 2010 at 07:28:55PM +0200, Henning Brauer wrote:
> > > * Graham Allan <[hidden email]> [2010-05-20 19:23]:
> > > > On Thu, May 20, 2010 at 07:02:23PM +0200, Axel Rau wrote:
> > > > > Am 20.05.2010 um 00:04 schrieb Henning Brauer:
> > > > >
> > > > > >* Axel Rau <[hidden email]> [2010-05-19 10:34]:
> > > > > >>Now the question: Can I put a trunk on top of a carp?
> > > > > >
> > > > > >you put carp on top of the trunk of course.
> > > > > OK.
> > > > > Can I have a trunk connected to 2 different switches then?
> > > >  
> > > > Not normally. Some higher-end switches can support this, eg the
> > > > HP Procurve switches running their K-series software can do something
> > > > they call distributed trunking (and no doubt Cisco and other vendors all
> > > > call it something else). But as I think you were talking about using
> > > > cheapish Netgear switches it's unlikely to be possible.
> > >
> > > well, lacp usually doesn't work across switches. but lacp is not the
> > > only mode trunk supports. roundrobin definately works across switches
> > > - how well might depend on your switches. works well for me on
> > > procurve with E-series software which doesn't do distributed trunking
> > > afair.<
> >  
> > How about the warnings about packet reordering and interactions with
> > TCP?
>
> never ran into such issues. too lazy right now to check wether trunk
> deals with that in roundrobin or wether i just got lucky.

uh, I just checked and... I am actually running failover. oups.

--
Henning Brauer, [hidden email], [hidden email]
BS Web Services, http://bsws.de
Full-Service ISP - Secure Hosting, Mail and DNS Services
Dedicated Servers, Rootservers, Application Hosting

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Axel Rau
In reply to this post by Henning Brauer
Am 20.05.2010 um 20:17 schrieb Henning Brauer:

>>
>> However, if you need to ask if you can run a trunk on top of a carp,
This was an academic question to keep the thread running (-;
>> do
>> yourself a favor and use a single switch. There will be less
>> downtime.
>
> that is something i could subscribe to :)
I try to keep things simple usually. Thanks to all for the advice.

Axel
---
[hidden email]  PGP-Key:29E99DD6  +49 151 2300 9283  computing @
chaos claudius

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Jussi Peltola
In reply to this post by Henning Brauer
On Thu, May 20, 2010 at 08:17:48PM +0200, Henning Brauer wrote:
> > I have two identical "core" switches in one (not really so critical at
> > all) place running OSPF, with a bunch of routers connecting to both
> > switches for redundancy. Works pretty well and there has even been a
> > config reset incident, which didn't break anything - because OSPF can
> > detect link failures. Trying to do the same all the way to the end hosts
> > (i.e.  without a routing protocol) is pretty difficult.
>
> i would never ever run any L3 on switches.
 
Bad wording on my part, the routers run OSPF and the switches are dumb
L2 devices.

Still, without OSPF et al there would be no way to detect a crappy
switch failing in funny ways, which was my point.

As an extra note, if you do get a crappy switch, be very careful with
its management interface. The cheapest ones have unbelievably slow CPUs
that are easily overloaded by broadcasts making the whole thing stop
responding. Even worse, the interrupt load seems to trigger some other
bugs, like LACP mysteriously failing and disabling one port on a trunk
and blackholing half of your traffic (this happened on a ZyXEL GS-4024,
which has otherwise totally Just Worked as a L2 switch for years) or
even the whole switch ASIC "crashing" after a broadcast storm and
requiring a reboot (though the management CPU was still responding
through the out of band ether and serial port after the storm was gone)

Also, it's a very obvious DoS; a malicious person needs to send a rather
small amount of BPDUs to overload the tiny CPU and the cheap switches
obviously have no rate limiting for packets going to the CPU (only on
all broadcasts). So, blocking BPDUs from non-trusted devices should be
enabled (but that should probably be done anyway.)

Even among "trusted" devices STP and LACP involve the shitty code
running on the underpowered management CPU, and that is not the part
that shines in the cheap switches. Static link aggregation works OK.

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Graham Allan
In reply to this post by Henning Brauer
On Thu, May 20, 2010 at 08:17:48PM +0200, Henning Brauer wrote:

> * Jussi Peltola <[hidden email]> [2010-05-20 20:07]:
>
> > If you want reliability, do not use cheap switches. Switch power
> > supplies are not the failure mode you want to avoid. I don't remember
> > seeing very many at all, however I've seen lots of crappy ones lose
> > their config or stop forwarding completely while keeping the link up.
>
> guess i lack the cheap shit switch experience.
>
> i do have experience with expensive shit switches tho. they suck in
> many different ways, never seen the behaviour you describe above tho.
>
> but then, ever since using said procurves, that is history.

I agree with the "don't use cheap switches" statement. If you look at
the price of (eg) used procurve 2824's then I don't see why anyone would
use Netgear or suchlike.

It's also good to have a switch with a real management interface that can
help you tell what's going on.

Graham

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Reyk Floeter-2
In reply to this post by Axel Rau
On Thu, May 20, 2010 at 07:02:23PM +0200, Axel Rau wrote:
> >>Now the question: Can I put a trunk on top of a carp?
> >
> >you put carp on top of the trunk of course.
> OK.
> Can I have a trunk connected to 2 different switches then?
>

yes, i did this many times using trunk in failover mode.  this is
actually the main reason why i implemented failover mode: for l2
redundancy.  i even normally use it in combination with VLANs.

to explain it using your artwork:

      +---+                      +------+                                                                                                                                          
      |   |        +-----+       |      |                                                                                                                                          
  ----+fw1+--------+ sw1 +-------+      |                                                                                                                                          
 carp0|   +--+     +-+-+-+    em0|      |                                                                                                                                          
      |   |  |       |           |      |                                                                                                                                          
      +-+-+  |  +----+           |      |                                                                                                                                          
        |    |  |                |Server|                                                                                                                                          
      +-+-+  +--|------+         | fbsd |                                                                                                                                          
      |   |     |      |         |      |                                                                                                                                          
      |   +-----+  +-+-+-+       |      |                                                                                                                                          
  ----+fw2+--------+ sw2 +-------+      |                                                                                                                                          
 carp0|   |        +-----+    em1|      |                                                                                                                                          
      +---+                      +------+                                                                                                                                          

let's assume that fw1 and fw2 are connected with em1 and em2, em1 is
connected to sw1 and em2 is connected to sw2 on each fw.  fbsd server
sits in vlan2, the uplink is in vlan1 connected to the same switches
(you might also have other physical switches for the uplink, which is
also fairly common, which would just require to move vlan1 to another
trunk or physical iface).

the switches don't need any special configuration, no trunks on the
switch and no stacking or similar.  they just need to be in the same
VLANs, so a simple interlink between them is all you need.  failover
mode means that the trunk only uses one active link at a time (the
first trunkport you add and so on) as long as the link is up.  this is
works nicely with any kind of switches, is safe to use and doesn't
cause any loops, address conflicts etc..  i use procurve switches
(now: hp networking e-series), but there is no need for distributed
trunking or tricks like this with failover mode.

fw1# ifconfig em0 up
fw1# ifconfig em1 up
fw1# ifconfig trunk0 trunkport em0 trunkport em1 trunkproto failover up
fw1# ifconfig vlan1 vlandev trunk0 descr UPLINK 10.1.1.2/24
fw1# ifconfig vlan2 vlandev trunk0 descr SERVERLAN 10.1.2.2/24
fw1# ifconfig carp1 vhid 1 carpdev vlan1 10.1.1.1/24
fw1# ifconfig carp2 vhid 2 carpdev vlan2 10.1.2.1/24

fw2# ifconfig em0 up
fw2# ifconfig em1 up
fw2# ifconfig trunk0 trunkport em0 trunkport em1 trunkproto failover up
fw2# ifconfig vlan1 vlandev trunk0 descr UPLINK 10.1.1.3/24
fw2# ifconfig vlan2 vlandev trunk0 descr SERVERLAN 10.1.2.3/24
fw2# ifconfig carp1 vhid 1 carpdev vlan1 advskew 100 10.1.1.1/24
fw2# ifconfig carp2 vhid 2 carpdev vlan2 advskew 100 10.1.2.1/24

and you can also move the pfsync traffic over the same trunk:

fw1# ifconfig vlan240 vlandev trunk0 192.168.240.2/24 up
fw1# ifconfig pfsync0 syncdev vlan240 up

fw2# ifconfig vlan240 vlandev trunk0 192.168.240.3/24 up
fw2# ifconfig pfsync0 syncdev vlan240 up

reyk

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Jussi Peltola
I do this too. In addition to the previously mentioned problems with
cheap switches losing their configs (and vlans) you should make sure the
active interfaces are all on one switch so that the link between them
isn't uselessly used; this will also avoid an unpleasant split brain
event if that link ever happens to fail. But in this case you will also
have to very carefully check the other switch stays properly configured so
the backup interfaces will actually pass the traffic you want.

Linux's bonding module has an arp monitor which solves some of these
problems, but the implementation is so hackish (as usual there...) that
I'd rather not use it in production. arping and ifstated might do the
same on openbsd, but I'm not sure if that will work when the interfaces
are trunk ports. I'll need to check this when I have time.

Reply | Threaded
Open this post in threaded view
|

Re: HA: pair of firewalls, 2 switches and 1 server

Axel Rau
In reply to this post by Reyk Floeter-2
Thanks for this detailed elaboration, Reyk.
A few questions:

Am 20.05.2010 um 22:07 schrieb Reyk Floeter:

> On Thu, May 20, 2010 at 07:02:23PM +0200, Axel Rau wrote:
>>>> Now the question: Can I put a trunk on top of a carp?
>>>
>>> you put carp on top of the trunk of course.
>> OK.
>> Can I have a trunk connected to 2 different switches then?
>>
>
> yes, i did this many times using trunk in failover mode.  this is
> actually the main reason why i implemented failover mode: for l2
> redundancy.  i even normally use it in combination with VLANs.
>
> to explain it using your artwork:
>
>      +---+                      +------+
>      |   |        +-----+       |      |
>  ----+fw1+--------+ sw1 +-------+      |
> carp0|   +--+     +-+-+-+    em0|      |
>      |   |  |       |           |      |
>      +-+-+  |  +----+           |      |
>        |    |  |                |Server|
>      +-+-+  +--|------+         | fbsd |
>      |   |     |      |         |      |
>      |   +-----+  +-+-+-+       |      |
>  ----+fw2+--------+ sw2 +-------+      |
> carp0|   |        +-----+    em1|      |
>      +---+                      +------+
>
> let's assume that fw1 and fw2 are connected with em1 and em2, em1 is
> connected to sw1 and em2 is connected to sw2 on each fw.  fbsd server
> sits in vlan2, the uplink is in vlan1 connected to the same switches
> (you might also have other physical switches for the uplink, which is
> also fairly common, which would just require to move vlan1 to another
> trunk or physical iface).
>
> the switches don't need any special configuration, no trunks on the
> switch and no stacking or similar.  they just need to be in the same
> VLANs, so a simple interlink between them is all you need.
You mean a physical connection between sw1 and sw2?

>  failover
> mode means that the trunk only uses one active link at a time (the
> first trunkport you add and so on) as long as the link is up.  this is
> works nicely with any kind of switches, is safe to use and doesn't
> cause any loops, address conflicts etc..  i use procurve switches
> (now: hp networking e-series), but there is no need for distributed
> trunking or tricks like this with failover mode.
>
> fw1# ifconfig em0 up
> fw1# ifconfig em1 up
> fw1# ifconfig trunk0 trunkport em0 trunkport em1 trunkproto failover
> up
> fw1# ifconfig vlan1 vlandev trunk0 descr UPLINK 10.1.1.2/24
> fw1# ifconfig vlan2 vlandev trunk0 descr SERVERLAN 10.1.2.2/24
> fw1# ifconfig carp1 vhid 1 carpdev vlan1 10.1.1.1/24
> fw1# ifconfig carp2 vhid 2 carpdev vlan2 10.1.2.1/24
>
> fw2# ifconfig em0 up
> fw2# ifconfig em1 up
> fw2# ifconfig trunk0 trunkport em0 trunkport em1 trunkproto failover
> up
> fw2# ifconfig vlan1 vlandev trunk0 descr UPLINK 10.1.1.3/24
> fw2# ifconfig vlan2 vlandev trunk0 descr SERVERLAN 10.1.2.3/24
> fw2# ifconfig carp1 vhid 1 carpdev vlan1 advskew 100 10.1.1.1/24
> fw2# ifconfig carp2 vhid 2 carpdev vlan2 advskew 100 10.1.2.1/24

On fbsd, I set default gw to 10.1.1.1 ?

But a trunk would have no counter parts. How does this fit in?

fbsd# ifconfig em0 up
fbsd# ifconfig em1 up
fbsd# ifconfig lagg0 laggproto failover laggport em0 laggport em1 \
                10.1.2.10 netmask 255.255.255.0
?

Axel
---
[hidden email]  PGP-Key:29E99DD6  +49 151 2300 9283  computing @
chaos claudius

12