Defending OpenBSD Performance

classic Classic list List threaded Threaded
149 messages Options
1234 ... 8
Reply | Threaded
Open this post in threaded view
|

Defending OpenBSD Performance

ts8807385
Hi Misc,

Even though this article: http://bulk.fefe.de/scalability was many years ago
and performance in OpenBSD had improved greatly since that time, I still
hear people (mostly younger people) complain about OpenBSD performance. They
cite poor threading, unused cores, no bigmem support, etc. Yet, when asked
outright to demonstrate their issue, no one can show numbers or reproduce a
performance issue. How do others defend OpenBSD in these conversations? I
normally cite the things I admire most about OpenBSD:

1. Simplicty - (IMO, this is by far its greatest attribute... simple is
secure)
2. OpenSSH
3. pf, carp, OpenBGPD
4. built-in security
5. ports collection

But, I'd like to have hard technicaly data to demonstrate that while Linux
and FreeBSD may scale to a gazillion CPUs and PetaBytes of Memory that
OpenBSD makes a fine firewall or desktop or mail server, etc and point out
that the old article so many people cite is indeed *old*.

Thanks for any suggestions. I hate seeing such a fine OS so easily dismissed
by folks (many of whom) have never even tried it!

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

STeve Andre'
On Monday 14 September 2009 13:39:46 Tom Smith wrote:

> Hi Misc,
>
> Even though this article: http://bulk.fefe.de/scalability was many years
> ago and performance in OpenBSD had improved greatly since that time, I
> still hear people (mostly younger people) complain about OpenBSD
> performance. They cite poor threading, unused cores, no bigmem support,
> etc. Yet, when asked outright to demonstrate their issue, no one can show
> numbers or reproduce a performance issue. How do others defend OpenBSD in
> these conversations? I normally cite the things I admire most about
> OpenBSD:
>
> 1. Simplicty - (IMO, this is by far its greatest attribute... simple is
> secure)
> 2. OpenSSH
> 3. pf, carp, OpenBGPD
> 4. built-in security
> 5. ports collection
>
> But, I'd like to have hard technicaly data to demonstrate that while Linux
> and FreeBSD may scale to a gazillion CPUs and PetaBytes of Memory that
> OpenBSD makes a fine firewall or desktop or mail server, etc and point out
> that the old article so many people cite is indeed *old*.
>
> Thanks for any suggestions. I hate seeing such a fine OS so easily
> dismissed by folks (many of whom) have never even tried it!

Attempting to prove the worth of OpenBSD to folks who are not able to
figure things out for themsevles is much like trying to teach butterflies
Calculus.

It doesn't work and wastes your time.

--STeve Andre'

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

Lars Nooden-2
In reply to this post by ts8807385
Tom Smith wrote:

> 3. pf, carp, OpenBGPD

pfsync


Regards,
-Lars

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

Tico Hannan
In reply to this post by STeve Andre'
STeve Andre' wrote:

> On Monday 14 September 2009 13:39:46 Tom Smith wrote:
>  
>> Hi Misc,
>>
>> Even though this article: http://bulk.fefe.de/scalability was many years
>> ago and performance in OpenBSD had improved greatly since that time, I
>> still hear people (mostly younger people) complain about OpenBSD
>> performance. They cite poor threading, unused cores, no bigmem support,
>> etc. Yet, when asked outright to demonstrate their issue, no one can show
>> numbers or reproduce a performance issue. How do others defend OpenBSD in
>> these conversations? I normally cite the things I admire most about
>> OpenBSD:
>>
>> 1. Simplicty - (IMO, this is by far its greatest attribute... simple is
>> secure)
>> 2. OpenSSH
>> 3. pf, carp, OpenBGPD
>> 4. built-in security
>> 5. ports collection
>>
>> But, I'd like to have hard technicaly data to demonstrate that while Linux
>> and FreeBSD may scale to a gazillion CPUs and PetaBytes of Memory that
>> OpenBSD makes a fine firewall or desktop or mail server, etc and point out
>> that the old article so many people cite is indeed *old*.
>>
>> Thanks for any suggestions. I hate seeing such a fine OS so easily
>> dismissed by folks (many of whom) have never even tried it!
>>    
>
> Attempting to prove the worth of OpenBSD to folks who are not able to
> figure things out for themsevles is much like trying to teach butterflies
> Calculus.
>
> It doesn't work and wastes your time.
>
> --STeve Andre'
>
>  
Ditto.

Furthermore, OpenBSD is not a religion (not for me, at least). The only
things OpenBSD *itself* needs is code and donations, not more devotees
unless I'm severely mistaken.

If someone wants to use inferior tools to for a given project's
requirements I'm more than happy to let them do so (unless they're
paying me for consulting).

Cheers,
Tico

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

Jose Quinteiro-2
In reply to this post by STeve Andre'
I've heard a different version of that one: "...is like teaching a pig to
sing, it wastes your time and it annoys the pig."

Saludos,
Jose.

On Mon, 14 Sep 2009 13:44:55 -0400, "STeve Andre'" <[hidden email]> wrote:
>
> Attempting to prove the worth of OpenBSD to folks who are not able to
> figure things out for themsevles is much like trying to teach
butterflies
> Calculus.
>
> It doesn't work and wastes your time.
>
> --STeve Andre'

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

Marco Peereboom
In reply to this post by ts8807385
If you think micro benchmarks are worth anything you have a micro
understanding of the problem.

On Mon, Sep 14, 2009 at 01:39:46PM -0400, Tom Smith wrote:

> Hi Misc,
>
> Even though this article: http://bulk.fefe.de/scalability was many years ago
> and performance in OpenBSD had improved greatly since that time, I still
> hear people (mostly younger people) complain about OpenBSD performance. They
> cite poor threading, unused cores, no bigmem support, etc. Yet, when asked
> outright to demonstrate their issue, no one can show numbers or reproduce a
> performance issue. How do others defend OpenBSD in these conversations? I
> normally cite the things I admire most about OpenBSD:
>
> 1. Simplicty - (IMO, this is by far its greatest attribute... simple is
> secure)
> 2. OpenSSH
> 3. pf, carp, OpenBGPD
> 4. built-in security
> 5. ports collection
>
> But, I'd like to have hard technicaly data to demonstrate that while Linux
> and FreeBSD may scale to a gazillion CPUs and PetaBytes of Memory that
> OpenBSD makes a fine firewall or desktop or mail server, etc and point out
> that the old article so many people cite is indeed *old*.
>
> Thanks for any suggestions. I hate seeing such a fine OS so easily dismissed
> by folks (many of whom) have never even tried it!

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

Steven Shockley
On 9/14/2009 2:53 PM, Marco Peereboom wrote:
> If you think micro benchmarks are worth anything you have a micro
> understanding of the problem.

You shouldn't make generalizations like that.  What if his primary
workload is micro benchmarks?

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

Marco Peereboom
Then it is of micro importance.

On Mon, Sep 14, 2009 at 03:46:13PM -0400, Steve Shockley wrote:
> On 9/14/2009 2:53 PM, Marco Peereboom wrote:
> > If you think micro benchmarks are worth anything you have a micro
> > understanding of the problem.
>
> You shouldn't make generalizations like that.  What if his primary
> workload is micro benchmarks?

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

Claudio Jeker
In reply to this post by ts8807385
On Mon, Sep 14, 2009 at 01:39:46PM -0400, Tom Smith wrote:

> Hi Misc,
>
> Even though this article: http://bulk.fefe.de/scalability was many years ago
> and performance in OpenBSD had improved greatly since that time, I still
> hear people (mostly younger people) complain about OpenBSD performance. They
> cite poor threading, unused cores, no bigmem support, etc. Yet, when asked
> outright to demonstrate their issue, no one can show numbers or reproduce a
> performance issue. How do others defend OpenBSD in these conversations? I
> normally cite the things I admire most about OpenBSD:
>

Yeah, it is always the same crap that is cited everywhere.
The performance problems his benchmarks have shown on connect, bind and
accept were solved in 2003 (quickly after he published his rant).
Since then OpenBSD has an approximate O(1) behaviour like all the other
OSs.
The biggest flaw with all those great benchmark articles is that their
often flawed and the people behind them are biased to show whatever they
like to prove. In the end many of fefe's test programs did not actually
measure what he assumed they would.

--
:wq Claudio

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

frantisek holop
hmm, on Mon, Sep 14, 2009 at 10:23:58PM +0200, Claudio Jeker said that
> like to prove. In the end many of fefe's test programs did not actually
> measure what he assumed they would.

and he was open to get patches to remedy those problems.

general dislike of any benchmark in the world is also part of the
openbsd culture just like some qualities of misc@ (although it's been
quite quiet lately).

if the numbers were better, the general sentiment would
be rather different i believe.

linux is faster in many respects (just look at zaurus) so what?
i dont use openbsd for its speed, but on the other hand i dont
downplay the importance of measuring things up and comparing it
with the others once in a while.  i am sure speed in the end is
of councern, otherwise the os woudln't be in C but, whatchamacallit,
python.

some things can be measured actually quite easily: how much content
a web server serves (not that much without sendfile()), how do the
databases perform, etc, this is all benchmark in the end, and the
programs doing the benchmarking are actually the daemons themselves.
so there, everyone is benchmarking 24/7 :]

-f
--
dragon riders make good first impressions.

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

roberth-5
On Tue, 15 Sep 2009 01:15:27 +0200
frantisek holop <[hidden email]> wrote:

> hmm, on Mon, Sep 14, 2009 at 10:23:58PM +0200, Claudio Jeker said that
> > like to prove. In the end many of fefe's test programs did not
> > actually measure what he assumed they would.
>
> and he was open to get patches to remedy those problems.
>
> general dislike of any benchmark in the world is also part of the
> openbsd culture just like some qualities of misc@ (although it's been
> quite quiet lately).
>
> if the numbers were better, the general sentiment would
> be rather different i believe.
>
> linux is faster in many respects (just look at zaurus) so what?
> i dont use openbsd for its speed, but on the other hand i dont
> downplay the importance of measuring things up and comparing it
> with the others once in a while.  i am sure speed in the end is
> of councern, otherwise the os woudln't be in C but, whatchamacallit,
> python.
>
> some things can be measured actually quite easily: how much content
> a web server serves (not that much without sendfile()), how do the
> databases perform, etc, this is all benchmark in the end, and the
> programs doing the benchmarking are actually the daemons themselves.
> so there, everyone is benchmarking 24/7 :]
>
> -f

In the end it boils down to measuring the different OS on the hardware
you will use for the task they should fullfill, nothing else matters in
the end.

- Robert

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

Nick Holland
In reply to this post by frantisek holop
frantisek holop wrote:
> hmm, on Mon, Sep 14, 2009 at 10:23:58PM +0200, Claudio Jeker said that
>> like to prove. In the end many of fefe's test programs did not actually
>> measure what he assumed they would.
>
> and he was open to get patches to remedy those problems.

<sarcasm>and always showed total non-bias.</sarcasm>

however, if you look at the tree, OpenBSD developers DID look at his test
programs and for the ones that pointed out real issues, made considerable
improve the performance in those areas.  Oh, also in a lot of areas he
didn't test.  However, the goal was to do it better (in the broad sense of
the term), not to top out on anyone's benchmark.

I'm sure he updated his page to show the new results, right?

> general dislike of any benchmark in the world is also part of the
> openbsd culture just like some qualities of misc@ (although it's been
> quite quiet lately).

I think dislike of benchmarks is a general attribute of most experienced
people on the technical side of things.  Love of and trust in benchmarks
is a general attribute of managers who want to pretend they made a good
decision rather than a wild-ass guess based on who has the cuter sales
rep and/or buys the better lunch (or sells products missing on their
resume).

> if the numbers were better, the general sentiment would
> be rather different i believe.

naw.
Once you learn to loath benchmarks and how people use them, you
can't start to cite them without your tongue ripping itself out of
your mouth and beating you senseless.  Even when you find one that
sorta looks good, you start thinking about all the edge (or straight
down the middle) cases it misses.

One of my favorite benchmarks:  I was approached by a network
consultant who told me he was going to need an emergency DHCP server
for an office and asked if I could have it done by the next day.
I told him I could have it done in 30 minutes.  He couldn't believe
me when I told him that, so I invited him to watch me.  So, I walked
him through an OpenBSD install, and twenty minutes after the target
machine (a very modest 400MHz Celeron) was first powered on, it was
ready to be plugged in and serve DHCP.  Still doesn't mean much. :)
See the flaws in my "benchmark"?  Of course you do.  So do I.

(got cocky a couple weeks later, told someone I could do a basic
OpenBSD install on one of these things in "about five minutes".  The
machine I picked turned out to have a bad hard disk and did a
massive number of disk retries before it finished loading.  Ended up
taking about seven minutes..and DID boot successfully.  While I
didn't hit my target, the spectator was still quite impressed :)

> linux is faster in many respects (just look at zaurus) so what?
> i dont use openbsd for its speed, but on the other hand i dont
> downplay the importance of measuring things up and comparing it
> with the others once in a while.  i am sure speed in the end is
> of councern, otherwise the os woudln't be in C but, whatchamacallit,
> python.

of course speed is a concern...to use one of my infamous car analogies,
though...you need enough power and performance to get by safely.
You don't want to drive a kiddie go-cart on the interstate freeway.
However, once you can go twice the speed you ever need to go, what's
the point in building a louder, more expensive and uncomfortable
vehicle?  Sure, some people go for that kinda thing, but that's why
there are more than one kind of car on the market.  (speaking of
hypocracy, I do my darnedest to avoid being seen in a dull vehicle,
though "speed" is only one definition of "not dull" :)

Speed matters.  Almost as much as some things, and nowhere near as
much as others.

> some things can be measured actually quite easily: how much content
> a web server serves (not that much without sendfile()), how do the
> databases perform, etc, this is all benchmark in the end, and the
> programs doing the benchmarking are actually the daemons themselves.
> so there, everyone is benchmarking 24/7 :]

What is also worth measuring is what the ultimate bottleneck is.

It is entertaining to hear people discuss benchmarks and performance,
and make decisions based on those criteria, then realize they are
sitting behind a pipe so skinny that a ten year old version of OpenBSD
on a 17 year old 486 could saturate the link many times over, with no
prospects of upgrading past what a Pentium II could handle in the
reasonable future.

SOME people DO need to filter or host or otherwise use multi-gigabit
links.  However making bad decisions based on unlikely scaling
concerns is silly.  As I've told people many times, "If you outgrow
this, replacing it will be among the LEAST of your problems, look
forward to it".

Practically speaking, the people who need the performance at the
edge of what OpenBSD can deliver usually are too busy to argue
benchmarks.

Nick.

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

Paul de Weerd
In reply to this post by roberth-5
On Tue, Sep 15, 2009 at 01:43:27AM +0200, Robert wrote:
| On Tue, 15 Sep 2009 01:15:27 +0200
| frantisek holop <[hidden email]> wrote:
|
| > hmm, on Mon, Sep 14, 2009 at 10:23:58PM +0200, Claudio Jeker said that
| > > like to prove. In the end many of fefe's test programs did not
| > > actually measure what he assumed they would.
| >
| > and he was open to get patches to remedy those problems.
| >
| > general dislike of any benchmark in the world is also part of the
| > openbsd culture just like some qualities of misc@ (although it's been
| > quite quiet lately).
| >
| > if the numbers were better, the general sentiment would
| > be rather different i believe.
| >
| > linux is faster in many respects (just look at zaurus) so what?
| > i dont use openbsd for its speed, but on the other hand i dont
| > downplay the importance of measuring things up and comparing it
| > with the others once in a while.  i am sure speed in the end is
| > of councern, otherwise the os woudln't be in C but, whatchamacallit,
| > python.
| >
| > some things can be measured actually quite easily: how much content
| > a web server serves (not that much without sendfile()), how do the
| > databases perform, etc, this is all benchmark in the end, and the
| > programs doing the benchmarking are actually the daemons themselves.
| > so there, everyone is benchmarking 24/7 :]
| >
| > -f
|
| In the end it boils down to measuring the different OS on the hardware
| you will use for the task they should fullfill, nothing else matters in
| the end.

I would say the most important thing is whether or not your solution
is up to the task you hand it. Be it with MS-DOS 3.1 or with OpenBSD,
I tend to want things to work and have enough scalability for (a bit
more than) estimated future growth.

Benchmarks are great to measure the effects of patches designed to
improve performance, not to pick the tool for the job at hand.

Paul 'WEiRD' de Weerd

--
>++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
+++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
                 http://www.weirdnet.nl/                 

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

Claudio Jeker
In reply to this post by frantisek holop
On Tue, Sep 15, 2009 at 01:15:27AM +0200, frantisek holop wrote:
> hmm, on Mon, Sep 14, 2009 at 10:23:58PM +0200, Claudio Jeker said that
> > like to prove. In the end many of fefe's test programs did not actually
> > measure what he assumed they would.
>
> and he was open to get patches to remedy those problems.
>

Hah. That's why he did not update his site since 2003. Do you realy think
that OpenBSD 3.4 and 4.6 are the same?

> general dislike of any benchmark in the world is also part of the
> openbsd culture just like some qualities of misc@ (although it's been
> quite quiet lately).
>
> if the numbers were better, the general sentiment would
> be rather different i believe.
>

Actually, I think that bad sentiment comes from the article itself:
    OpenBSD 3.4 was a real stinker in these tests. The installation
    routine sucks, the disk performance sucks, the kernel was unstable,
    and in the network scalability department it was even outperformed
    by it's father, NetBSD. OpenBSD also gets points deducted for the
    sabotage they did to their IPv6 stack. If you are using OpenBSD,
    you should move away now.

With this he proofed himself as non credible and uninterested in serious
measuring.

> linux is faster in many respects (just look at zaurus) so what?

and in many it is slower or plain unusable without further hacks.
It mostly depends on what you need, so choose your tool wisely.

> i dont use openbsd for its speed, but on the other hand i dont
> downplay the importance of measuring things up and comparing it
> with the others once in a while.  i am sure speed in the end is
> of councern, otherwise the os woudln't be in C but, whatchamacallit,
> python.
>

The reason for C has nothing to do with speed.

> some things can be measured actually quite easily: how much content
> a web server serves (not that much without sendfile()), how do the
> databases perform, etc, this is all benchmark in the end, and the
> programs doing the benchmarking are actually the daemons themselves.
> so there, everyone is benchmarking 24/7 :]
>

And here again comes this style of uninformed dumb rant. Why do you think
a web server will not do that much without sendfile()? Honestly it is
exactly the opposide, a web server that never touches the disk for content
delivery will outperform all others and can server enough data to fill a
gigabit link. sendfile() is no magic pill, sure it saves work and helps
increasing the performance but it still needs to get the data from the
disk at one point which is very slow.

--
:wq Claudio

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

cognacc
So since benchmarking is out, how do we then find out where
potential problems are.
What does OpenBSD developers do, since surely they don't benchmark :)

Maybe we should  profile instead ?

I'm not very experienced with webservers, but here
how i would approach it.

1. i have a problem, i think about it where/what the problem could be
2 i check the logs - test my equipment
3. I create 1 or a few profiling tests / micro benchmarks to test my assumptions
    or make certain i haven't misinterpreted my problems.
4.  Step back and interpret results
5. think of other tests / micro benchmarks that could further enlighten me
   and confirm/unconfirm(?) my "findings"

What i wouldnt  do, is "design" a mother of a benchmark that covers
all the bases.
It's to hard to get right. It would take to much time.


How would OpenBSD dev's approach a issue.
How are issues generally searched for/ found out?

I imagine something like
OpenBSD dev works on the httpd daemon - asks for testing.
I find a problem,  ex: it'd slow like heck - check configuration -
interfaces - logs
What now - i write back to dev.
dev asks me to do what?
What does the dev do behind the scenes?


regards mic

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

ttw+bsd-2
In reply to this post by Nick Holland
On 14.09-20:43, Nick Holland wrote:
> [ ... ]
> Speed matters.  Almost as much as some things, and nowhere near as
> much as others.

beautifully specific and vague, i'd challenge anyone to sum up
benchmarking better.  if that's not a quote, it is now; i'm writing
it down and sticking it to my wall.

> [ ... ]
> Practically speaking, the people who need the performance at the
> edge of what OpenBSD can deliver usually are too busy to argue
> benchmarks.

careful, that could be seen as an admission
        ;-)

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

Nick Holland
[hidden email] wrote:
> On 14.09-20:43, Nick Holland wrote:
>> [ ... ]
>> Speed matters.  Almost as much as some things, and nowhere near as
>> much as others.
>
> beautifully specific and vague, i'd challenge anyone to sum up
> benchmarking better.  if that's not a quote, it is now; i'm writing
> it down and sticking it to my wall.

Original material, as far as I know. :)

>> [ ... ]
>> Practically speaking, the people who need the performance at the
>> edge of what OpenBSD can deliver usually are too busy to argue
>> benchmarks.
>
> careful, that could be seen as an admission
> ;-)

Yep.  Most performance-oriented thing I've done with OpenBSD was
firewalling a 45Mbps T3 line.  It did tax the machine a little bit,
but the primary firewall was a Celeron 600, about five years old at
the time it was put into service (failover was a PIII-750, which
showed a lot lower load, I think it was more the cache than the MHz).

Lame. :)

Nick.

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

Bob Beck-4
In reply to this post by Nick Holland
> Practically speaking, the people who need the performance at the
> edge of what OpenBSD can deliver usually are too busy to argue
> benchmarks.
>
>
Precisely.

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

Henning Brauer
In reply to this post by Nick Holland
* Nick <[hidden email]> [2009-09-15 13:52]:
> Yep.  Most performance-oriented thing I've done with OpenBSD was
> firewalling a 45Mbps T3 line.  It did tax the machine a little bit,
> but the primary firewall was a Celeron 600, about five years old at
> the time it was put into service (failover was a PIII-750, which
> showed a lot lower load, I think it was more the cache than the MHz).

i have a bgp machine forwarding 800MBit/s of real world generic
internet traffic. can handle at least twice that. enough of a
benchmark?

--
Henning Brauer, [hidden email], [hidden email]
BS Web Services, http://bsws.de
Full-Service ISP - Secure Hosting, Mail and DNS Services
Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam

Reply | Threaded
Open this post in threaded view
|

Re: Defending OpenBSD Performance

Florian Fuessl
Hi Henning,

> -----Original Message-----
> From: owner-misc@[...] on Behalf
> Of Henning Brauer
> Sent: Tuesday, September 15, 2009 2:39 PM
> Subject: Re: Defending OpenBSD Performance
>
> * Nick <[hidden email]> [2009-09-15 13:52]:
> > [...]
>
> i have a bgp machine forwarding 800MBit/s of real world generic
> internet traffic. can handle at least twice that. enough of a
> benchmark?

what hardware (CPU, NICs, MEM) and OpenBSD release (i386, GENERIC?) are you
running on your bgp machine(s)?

> --
> Henning Brauer, [hidden email], [hidden email]
> BS Web Services, http://bsws.de
> Full-Service ISP - Secure Hosting, Mail and DNS Services
> Dedicated Servers, Rootservers, Application Hosting - Hamburg &
> Amsterdam

-Florian

1234 ... 8