Hardware for a PF box

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Hardware for a PF box

BARDOU Pierre
Hello,

I'm going to buy hardware to create 4 PF/relayd/openVPN boxes (2 active, 2
passive).
I have an average of 500 new connections/s, 40k states and 40kpps in PF, 20
remote concurrent accesses on OpenVPN.

What CPU would you recommend between Intel and AMD ?
Since PF is mono threaded, I think more than 2 CPU cores are useless. Am I
right ?
For the same reason, I think that the CPU with the highest frequency will be
the best ?
Would it be useful to replace 15ktpm SAS HDDs by SSDs ?

Thank you.

--
Cordialement,

Pierre BARDOU
CSIM - Bureau 002


[cid:image001.jpg@01CAF064.EC6665D0]

12 rue Michel Labrousse
BP93668
F-31036 Toulouse CEDEX 1

Til : 05 67 69 71 84
Fax : 05 34 61 51 00
Mail : [hidden email]<mailto:[hidden email]>

[demime 1.01d removed an attachment of type image/jpeg which had a name of image001.jpg]

Reply | Threaded
Open this post in threaded view
|

Re: Hardware for a PF box

Henning Brauer
* BARDOU Pierre <[hidden email]> [2010-05-10 17:27]:
> Hello,
>
> I'm going to buy hardware to create 4 PF/relayd/openVPN boxes (2 active, 2
> passive).
> I have an average of 500 new connections/s, 40k states and 40kpps in PF, 20
> remote concurrent accesses on OpenVPN.

that's not much. a PIII @ 1GHz probably easily suffices.

> What CPU would you recommend between Intel and AMD ?

doesn't matter all that much.

> Since PF is mono threaded, I think more than 2 CPU cores are useless. Am I
> right ?
> For the same reason, I think that the CPU with the highest frequency will be
> the best ?

you want to run GENERIC, not GENERIC.MP, unless you also do lots of
stuff in userland on the pf box, then MP might pay out.
and since you'll be using one core only anyway you want as few and as
fast cores you can.

> Would it be useful to replace 15ktpm SAS HDDs by SSDs ?

yes.
harddisks don't matter on pure firewalls. what is written to disks?
logs. not all that much. read? after boot, not much.
so using your expensive SAS-disks elsewhere is a good idea. a cheap
40..64G SSD will do fine.

--
Henning Brauer, [hidden email], [hidden email]
BS Web Services, http://bsws.de
Full-Service ISP - Secure Hosting, Mail and DNS Services
Dedicated Servers, Rootservers, Application Hosting

Reply | Threaded
Open this post in threaded view
|

Re: Hardware for a PF box

Stuart Henderson
In reply to this post by BARDOU Pierre
On 2010-05-10, BARDOU Pierre <[hidden email]> wrote:
> I'm going to buy hardware to create 4 PF/relayd/openVPN boxes (2 active, 2
> passive).
> I have an average of 500 new connections/s, 40k states and 40kpps in PF, 20
> remote concurrent accesses on OpenVPN.
>
> What CPU would you recommend between Intel and AMD ?

This question is silly, the CPU manufacturer doesn't matter.
There is a lot more difference between the various CPUs made by a
manufacturer (486, atom, p4, p3, core2, nehalem, ...) than the
difference between AMD's fastest CPU and Intel's fastest CPU.

> Since PF is mono threaded, I think more than 2 CPU cores are useless. Am I
> right ?

More than 1 core might be useful for OpenVPN (especially if you can run
multiple openvpn processes, for example maybe listening to different
ports and distributing between them using a rdr pool).

Since any new CPU you get is likely to be multi-core I suggest you
benchmark both GENERIC and GENERIC.MP...

> For the same reason, I think that the CPU with the highest frequency will be
> the best ?

Generally yes (but some arch like P4, Atom are much slower for a
given clock speed than P3-based arch, for example).

Also consider memory bandwidth and cache size.

> Would it be useful to replace 15ktpm SAS HDDs by SSDs ?

Depends what you're writing to disk. Presumably you won't be doing much in
the way of random disk access, but might be doing some sequential writes for
logging, so in that case SSDs are more likely to hurt than help.

You will probably do better to propose some specific system and ask if
anyone knows of problems with that machine.

Reply | Threaded
Open this post in threaded view
|

Re: Hardware for a PF box

gwes-2
In reply to this post by BARDOU Pierre
BARDOU Pierre <[hidden email]> wrote on Mon, 10 May 2010 17:24:21
>Subject: Hardware for a PF box

>I'm going to buy hardware to create 4 PF/relayd/openVPN boxes
>(2 active, 2 passive).
>I have an average of 500 new connections/s,
>40k states and 40kpps in PF, 20
>remote concurrent accesses on OpenVPN.

>What CPU would you recommend between Intel and AMD ?

As other people have said, models/versions vary much more over
each vendor than overall between vendors.
>....

>For the same reason, I think that the CPU with the
>highest frequency will be the best ?

As other people have said, memory access time, cache size,
and integer arithmetic performance matter.
For any specific CPU version/architecture, faster clocks are
better up to the point where CPU utilization is under
(for instance) 50%.
Choice of memory speed is also important.
There are non-intuitive interactions between CPU clocks
and RAM clocks - sometimes lower clock speeds can mean
fewer clock cycles. If you lower the clock speed 10%
and reduce access time from 6 cycles to 5, you get
6% improvement.

Choice of network interfaces can make as much impact
as CPU choice. Many of the gigabit chips have better
performance and better driver interaction than older
10/100 chips. I use the gigabit RE (Realtek) because
they're very cheap and quite fast. I can't say which
other gigabit ones are as good or better but as a rule
the 10/100 interfaces are expensive in CPU time.

>Would it be useful to replace 15ktpm SAS HDDs by SSDs ?

If there are local servers available, what about running
the firewalls as diskless machines? Cheaper, cooler, and
if you are running a backed up RAID on your servers,
more reliable.

I currently run a lightly loaded firewall on a 1.5 GHz
VIA CPU with 3 interfaces - most packets traverse 2 bridged
interfaces. Running 20 Mbit/sec the CPU loading is 25%.
There are usually 500 states or so with a moderately complex
(200+ lines) pf rule set and 20-50 connections/sec.

The VIA is very slow but also runs quite cool & low power.
Total power with a local SATA laptop disk is 24W.

I have run that system with a USB flash stick as the only
local disk for more than a year with no problems.

I hope this helps.

geoff steckel
omnivore technology

Reply | Threaded
Open this post in threaded view
|

Re: Hardware for a PF box

Chris Smith-32
On Mon, May 10, 2010 at 1:57 PM, Geoff <[hidden email]> wrote:
> If there are local servers available, what about running
> the firewalls as diskless machines?

What about logging in this case? Can PF logs be sent to another system
running a syslog daemon?

Chris

Reply | Threaded
Open this post in threaded view
|

Re: Hardware for a PF box

Lars Nooden-2
On Mon, 10 May 2010, Chris Smith wrote:
> What about logging in this case? Can PF logs be sent to another system
> running a syslog daemon?

You answered your own question. ;)  Look at the 'action' field explanation
in the manual page for syslog.conf(5)

About the diskless machine, many of the so-called diskless machines
actually use flash or ssd instead of a spinning magnetic platter.  The
base installation of openbsd is still quite small.  If you are only
running PF, you will have a lot of space left over on a 1GB CF to make a
logging partition.  Flash can be very slow, so volitile caches can be
stored in an mfs partition.

/Lars

Reply | Threaded
Open this post in threaded view
|

Re: Hardware for a PF box

Aaron Mason
On Tue, May 11, 2010 at 4:56 PM, Lars Nooden <[hidden email]> wrote:

> On Mon, 10 May 2010, Chris Smith wrote:
>>
>> What about logging in this case? Can PF logs be sent to another system
>> running a syslog daemon?
>
> You answered your own question. ;)  Look at the 'action' field explanation
> in the manual page for syslog.conf(5)
>
> About the diskless machine, many of the so-called diskless machines actually
> use flash or ssd instead of a spinning magnetic platter.  The base
> installation of openbsd is still quite small.  If you are only running PF,
> you will have a lot of space left over on a 1GB CF to make a logging
> partition.  Flash can be very slow, so volitile caches can be stored in an
> mfs partition.
>
> /Lars
>
>

OpenBSD will happily fit into about 160mb by installing only base and
etc which provide plenty for a firewall.  My 1.4GHz Toshiba laptop
acting as a wireless-wired gateway runs OpenBSD 4.6 on a 512mb USB
drive (which I'd like to replace with a CF disk on a 2.5" compatible
adapter) with space to spare.  Sure it doesn't do anywhere near as
many packets as you propose, but it handles a constantly-running
seedbox and my gaming together without skipping a beat, which is more
than I can ask for.

--
Aaron Mason - Programmer, open source addict
I've taken my software vows - for beta or for worse

Reply | Threaded
Open this post in threaded view
|

Re: Hardware for a PF box

BARDOU Pierre
Hello,



I'll try to answer every suggestion...



I'm going to buy brand new HP servers, DL360 G5 or DL165 G7. So the choice for CPU is between AMD Opteron 24xx or Intel Xeon 55xx.

I've read that a PIII would be sufficient : I have performance issues actually, running on a Xeon 2.8GHz (monocore, FSB 800, socket 604). I don't think they come from PF BTW, it should be logging/relayd/OpenVPN which makes the box lag.



I'm actually on a test with dual xeon E5420 on GEMERIC.MP, it runs like a charm. But it's borrowed hardware, I have to give it back :)



I'm very interested in separated log machine, I think I'll do that. Could you give me an estimation on how many Mbps I need on the log server ?

I think I'll put this on a VM, we have an ESX cluster connected to a CX3-40 SAN which should give enough disk I/O...



Installing SSD on the machines is way more expensive with HP hardware : 72 GB SAS 15Ktpm costs 260b,, 60 GB SSD costs 950b,.

HP offers no way to install a compact flash as disk drive.



Networks cards are Intel Gb, using em(4) driver.



So, with all your considerations, here's my actual setup :

* Xeon E5504 quad core @2Ghz (don't need AMD's 6 cores, and costs nearly the same prize than the only dual core remaining, E5502 @1.86GHz)

* 3*1GB memory (Xeon are triple channel, so I need three DIMM for maximal memory bandwidth)

* 2x72 Gb SAS drives on raid0



Does it sound correct to you ?

Do you have any suggestion/modification ?



Thank you very much for the help.



--

Cordialement,

Pierre BARDOU





-----Message d'origine-----

DeB : Aaron Mason [mailto:[hidden email]]

EnvoyC)B : mardi 11 mai 2010 14:01

CB : Lars Nooden

CcB : [hidden email]

ObjetB : Re: Hardware for a PF box



On Tue, May 11, 2010 at 4:56 PM, Lars Nooden <[hidden email]> wrote:

> On Mon, 10 May 2010, Chris Smith wrote:

>>

>> What about logging in this case? Can PF logs be sent to another system

>> running a syslog daemon?

>

> You answered your own question. ;)  Look at the 'action' field explanation

> in the manual page for syslog.conf(5)

>

> About the diskless machine, many of the so-called diskless machines actually

> use flash or ssd instead of a spinning magnetic platter.  The base

> installation of openbsd is still quite small.  If you are only running PF,

> you will have a lot of space left over on a 1GB CF to make a logging

> partition.  Flash can be very slow, so volitile caches can be stored in an

> mfs partition.

>

> /Lars

>

>



OpenBSD will happily fit into about 160mb by installing only base and

etc which provide plenty for a firewall.  My 1.4GHz Toshiba laptop

acting as a wireless-wired gateway runs OpenBSD 4.6 on a 512mb USB

drive (which I'd like to replace with a CF disk on a 2.5" compatible

adapter) with space to spare.  Sure it doesn't do anywhere near as

many packets as you propose, but it handles a constantly-running

seedbox and my gaming together without skipping a beat, which is more

than I can ask for.



--

Aaron Mason - Programmer, open source addict

I've taken my software vows - for beta or for worse


Reply | Threaded
Open this post in threaded view
|

Re: Hardware for a PF box

BARDOU Pierre
In reply to this post by Aaron Mason
Sorry, typo :

SAS drives would be on RAID1.



So the config would be :

* Xeon E5504 quad core @2Ghz (don't need AMD's 6 cores, and costs nearly the same prize than the only dual core remaining, E5502 @1.86GHz)

* 3*1GB memory (Xeon are triple channel, so I need three DIMM for maximal memory bandwidth)

* 2x72 Gb SAS drives on raid1

* GENERIC.MP kernel



--

Cordialement,

Pierre BARDOU





-----Message d'origine-----

DeB : BARDOU Pierre

EnvoyC)B : mardi 11 mai 2010 15:40

CB : '[hidden email]'

ObjetB : RE: Hardware for a PF box



Hello,



I'll try to answer every suggestion...



I'm going to buy brand new HP servers, DL360 G5 or DL165 G7. So the choice for CPU is between AMD Opteron 24xx or Intel Xeon 55xx.

I've read that a PIII would be sufficient : I have performance issues actually, running on a Xeon 2.8GHz (monocore, FSB 800, socket 604). I don't think they come from PF BTW, it should be logging/relayd/OpenVPN which makes the box lag.



I'm actually on a test with dual xeon E5420 on GEMERIC.MP, it runs like a charm. But it's borrowed hardware, I have to give it back :)



I'm very interested in separated log machine, I think I'll do that. Could you give me an estimation on how many Mbps I need on the log server ?

I think I'll put this on a VM, we have an ESX cluster connected to a CX3-40 SAN which should give enough disk I/O...



Installing SSD on the machines is way more expensive with HP hardware : 72 GB SAS 15Ktpm costs 260b,, 60 GB SSD costs 950b,.

HP offers no way to install a compact flash as disk drive.



Networks cards are Intel Gb, using em(4) driver.



So, with all your considerations, here's my actual setup :

* Xeon E5504 quad core @2Ghz (don't need AMD's 6 cores, and costs nearly the same prize than the only dual core remaining, E5502 @1.86GHz)

* 3*1GB memory (Xeon are triple channel, so I need three DIMM for maximal memory bandwidth)

* 2x72 Gb SAS drives on raid0



Does it sound correct to you ?

Do you have any suggestion/modification ?



Thank you very much for the help.



--

Cordialement,

Pierre BARDOU





-----Message d'origine-----

DeB : Aaron Mason [mailto:[hidden email]]

EnvoyC)B : mardi 11 mai 2010 14:01

CB : Lars Nooden

CcB : [hidden email]

ObjetB : Re: Hardware for a PF box



On Tue, May 11, 2010 at 4:56 PM, Lars Nooden <[hidden email]> wrote:

> On Mon, 10 May 2010, Chris Smith wrote:

>>

>> What about logging in this case? Can PF logs be sent to another system

>> running a syslog daemon?

>

> You answered your own question. ;)  Look at the 'action' field explanation

> in the manual page for syslog.conf(5)

>

> About the diskless machine, many of the so-called diskless machines actually

> use flash or ssd instead of a spinning magnetic platter.  The base

> installation of openbsd is still quite small.  If you are only running PF,

> you will have a lot of space left over on a 1GB CF to make a logging

> partition.  Flash can be very slow, so volitile caches can be stored in an

> mfs partition.

>

> /Lars

>

>



OpenBSD will happily fit into about 160mb by installing only base and

etc which provide plenty for a firewall.  My 1.4GHz Toshiba laptop

acting as a wireless-wired gateway runs OpenBSD 4.6 on a 512mb USB

drive (which I'd like to replace with a CF disk on a 2.5" compatible

adapter) with space to spare.  Sure it doesn't do anywhere near as

many packets as you propose, but it handles a constantly-running

seedbox and my gaming together without skipping a beat, which is more

than I can ask for.



--

Aaron Mason - Programmer, open source addict

I've taken my software vows - for beta or for worse


Reply | Threaded
Open this post in threaded view
|

Re: Hardware for a PF box

Lars Nooden-2
In reply to this post by BARDOU Pierre
On Tue, 11 May 2010, BARDOU Pierre wrote:
> ... I don't think they come from PF BTW, it should be
> logging/relayd/OpenVPN which makes the box lag.

Verify before you flush money.  Tools like iostat, vmstat and pftop might
help show where the load is.  Does the load you have from OpenVPN suggest
the need for hardware random number generator?

> I'm very interested in separated log machine, I think I'll do that.
> Could you give me an estimation on how many Mbps I need on the log
> server ?

It depends on what you have chosen to log, the level of detail you have
chosen to log at and how much that service is actually used.  Try set up
the logging rules and use tcpdump or pftop to track the connection to the
log server to see.

> Does it sound correct to you ?

It could be overkill on the hardware.

> Do you have any suggestion/modification ?

Several have already mentioned that a diskless set up would work.  For
PF,relayd,OpenVPN you do not need much of a hard drive.

You boot from a 1GB CF and fit base in way less than 250MB of it.  The
rest could be used for short-term logging with copies sent to a log
server.

If you are running squid or another cache, then the RAID set up might be
useful.  Or it might not be.  If you have a lot of RAM, then you can put
the cache onto a ramdisk using mfs, if the size is right.

/Lars

Reply | Threaded
Open this post in threaded view
|

Re: Hardware for a PF box

Chris Smith-32
In reply to this post by Lars Nooden-2
On Tue, May 11, 2010 at 2:56 AM, Lars Nooden <[hidden email]> wrote:
> You answered your own question. ;) B Look at the 'action' field explanation
> in the manual page for syslog.conf(5)

Maybe I'm missing something:

I can send normal syslog data to a remote logging server without
writing log files but not PF log entries - there is no entry in
syslog.conf for pflog. There's a neat trick listed here:
http://www.openbsd.org/faq/pf/logging.html but the PF logs first have
to be written locally to a the pflog file. The concern is repeated
writing to the SSD or CF which apparently tends to shorten their life.

If PF could write directly to syslog this problem would be ameliorated.

Chris

Reply | Threaded
Open this post in threaded view
|

Re: Hardware for a PF box

Martin Pelikan
2010/5/11, Chris Smith <[hidden email]>:
> Maybe I'm missing something:

You might want something like this:
# mkdir /var/log/rd ; chmod 700 /var/log/rd ; chown _pflogd:_pflogd
/var/log/rd
# echo 'pflogd_flags="-f /var/log/rd/pflog" ' >> /etc/rc.conf.local
# echo 'swap /var/log/rd/ mfs rw,nodev,nosuid,-s=67108864 0 0' >> /etc/fstab
# mount /var/log/rd/
# pkill pflogd ; sleep 1 ; pflogd -f /var/log/rd/pflog

Filesystems in RAM are extremely handy, but make sure the remote
logging works, because umount makes the data disappear - see mfs(8).
Does anyone know neater solution?

--
Martin PelikC!n, Steadynet
Jabber: [hidden email]
web: http://cap.potazmo.cz/

Reply | Threaded
Open this post in threaded view
|

Re: Hardware for a PF box

Lars Nooden-2
In reply to this post by Chris Smith-32
On Tue, 11 May 2010, Chris Smith wrote:
> ...http://www.openbsd.org/faq/pf/logging.html but the PF logs first have
> to be written locally to a the pflog file.

Or you can pipe to logger(1) directly or go via a FIFO

/Lars

Reply | Threaded
Open this post in threaded view
|

Re: Hardware for a PF box

Rod Whitworth-3
In reply to this post by Chris Smith-32
On Tue, 11 May 2010 12:43:17 -0400, Chris Smith wrote:

>On Tue, May 11, 2010 at 2:56 AM, Lars Nooden <[hidden email]> wrote:
>> You answered your own question. ;) B Look at the 'action' field explanation
>> in the manual page for syslog.conf(5)
>
>Maybe I'm missing something:
>
>I can send normal syslog data to a remote logging server without
>writing log files but not PF log entries - there is no entry in
>syslog.conf for pflog. There's a neat trick listed here:
>http://www.openbsd.org/faq/pf/logging.html but the PF logs first have
>to be written locally to a the pflog file. The concern is repeated
>writing to the SSD or CF which apparently tends to shorten their life.

I have tried to kill a CF for years. For more than a year it was
running spamd with the most verbose logging possible and lots of other
read/writes the system could live without.

It is still going.

I suggest that you use CF and when upgrade time comes around you
program a new one and then have a halt-swap-reboot event and send me
the one you don't think has much life left. I'll try wearing it out for
you.

My clients have lost more hard drives last year (3) than CFs in my
lifetime (0) and I've been using them since they were exorbitantly
priced.

Some of that is good luck but they sure are not easily worn out.


*** NOTE *** Please DO NOT CC me. I <am> subscribed to the list.
Mail to the sender address that does not originate at the list server is tarpitted. The reply-to: address is provided for those who feel compelled to reply off list. Thankyou.

Rod/
---
This life is not the real thing.
It is not even in Beta.
If it was, then OpenBSD would already have a man page for it.

Reply | Threaded
Open this post in threaded view
|

Re: Hardware for a PF box

Bryan Vyhmeister-2
On May 11, 2010, at 17:18, "Rod Whitworth" <[hidden email]> wrote:

> On Tue, 11 May 2010 12:43:17 -0400, Chris Smith
>
> I have tried to kill a CF for years. For more than a year it was
> running spamd with the most verbose logging possible and lots of other
> read/writes the system could live without.
>
> It is still going.
>
> I suggest that you use CF and when upgrade time comes around you
> program a new one and then have a halt-swap-reboot event and send me
> the one you don't think has much life left. I'll try wearing it out for
> you.
>
> My clients have lost more hard drives last year (3) than CFs in my
> lifetime (0) and I've been using them since they were exorbitantly
> priced.
>
> Some of that is good luck but they sure are not easily worn out.

I'd have to agree there. I had one CF fail after three years of heavy DNS
logging and I had a brand new card fail immediately as well. I've had many
more times the hard drives fail.

I would also suggest looking at the flashrd project.

http://www.nmedia.net/flashrd/

I just recently started using it on some individual firewalls as well as
several clusters. The whole point of the setup is to mount everything possible
as read only and the rest to mfs.

Bryan