Coming from FreeBSD, lower networking performance

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Coming from FreeBSD, lower networking performance

Sijmen J. Mulder
Hi all,

After migrating a VPS from FreeBSD to OpenBSD I noticed reduced networking performance. Both incoming and outgoing traffic seems to be 2-3 times slower on average. By testing 100MB file transfers I've mostly eliminated the following factors:

 - Protocol and ciphers (tested SCP, SFTP, FTP+TLS, HTTP, HTTPS)
 - Client software
 - Peer host
 - VM provider/platform

The easiest way to show and test this difference is as such:

 1. Install either FreeBSD 11.2 or OpenBSD 6.3 on a machine or VM
 2. Install curl, then `time curl -O http://download.thinkbroadband.com/100MB.zip`

On both my VPS provider and my own PC with VirtualBox VMs the difference is about 3x for the above test. Similar results happen when scp-ing a file to the machine.

Any ideas on the cause of this? Any additional tests or tweaks I could try?

Sijmen

Reply | Threaded
Open this post in threaded view
|

Re: Coming from FreeBSD, lower networking performance

bit shifter
On Fri, Jul 13, 2018 at 5:20 PM, Sijmen J. Mulder <[hidden email]> wrote:
> Any ideas on the cause of this? Any additional tests or tweaks I could try?

You could try running iperf to eliminate disk IO from the equation and
narrow down the potential sources of the performance deficit you're
seeing.

Reply | Threaded
Open this post in threaded view
|

Re: Coming from FreeBSD, lower networking performance

Henry Bonath
In reply to this post by Sijmen J. Mulder
Have you looked into IPerf? https://iperf.fr/

This is what I typically use for testing network throughput.
Downloading a file is a bit more complex and involves things
like the source server/latency/etc. as well as disk performance.
(I know a 100MB file isnt much but still...)

IPerf has a lot of knobs to tweak, like number of threads, TCP window size,
etc.
You will need 2 hosts, one to act as server and one as client.
I recommend the latest stable version of iperf3, which is available as a
package:
$ doas pkg_add iperf3

Hope this helps.

On Fri, Jul 13, 2018 at 7:20 PM, Sijmen J. Mulder <[hidden email]> wrote:

> Hi all,
>
> After migrating a VPS from FreeBSD to OpenBSD I noticed reduced networking
> performance. Both incoming and outgoing traffic seems to be 2-3 times
> slower on average. By testing 100MB file transfers I've mostly eliminated
> the following factors:
>
>  - Protocol and ciphers (tested SCP, SFTP, FTP+TLS, HTTP, HTTPS)
>  - Client software
>  - Peer host
>  - VM provider/platform
>
> The easiest way to show and test this difference is as such:
>
>  1. Install either FreeBSD 11.2 or OpenBSD 6.3 on a machine or VM
>  2. Install curl, then `time curl -O http://download.
> thinkbroadband.com/100MB.zip`
>
> On both my VPS provider and my own PC with VirtualBox VMs the difference
> is about 3x for the above test. Similar results happen when scp-ing a file
> to the machine.
>
> Any ideas on the cause of this? Any additional tests or tweaks I could try?
>
> Sijmen
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Coming from FreeBSD, lower networking performance

Marcus MERIGHI
In reply to this post by Sijmen J. Mulder
Hello Sijmen,

this is not something I remember seeing often on the list; to improve
your chances of replies beyond "use iperf to test" (there's tcpbench(1)
in base) you should at least provide a dmesg(8).

Maybe even use sendbug(1) and have the report go to bugs@ [1].

Could you run -current [2] to see if the problem is still there?

[1] http://www.openbsd.org/report.html
[2] http://www.openbsd.org/faq/current.html

Marcus

[hidden email] (Sijmen J. Mulder), 2018.07.14 (Sat) 01:20 (CEST):

> Hi all,
>
> After migrating a VPS from FreeBSD to OpenBSD I noticed reduced
> networking performance. Both incoming and outgoing traffic seems to be
> 2-3 times slower on average. By testing 100MB file transfers I've
> mostly eliminated the following factors:
>
>  - Protocol and ciphers (tested SCP, SFTP, FTP+TLS, HTTP, HTTPS)
>  - Client software
>  - Peer host
>  - VM provider/platform
>
> The easiest way to show and test this difference is as such:
>
>  1. Install either FreeBSD 11.2 or OpenBSD 6.3 on a machine or VM
>  2. Install curl, then `time curl -O http://download.thinkbroadband.com/100MB.zip`
>
> On both my VPS provider and my own PC with VirtualBox VMs the
> difference is about 3x for the above test. Similar results happen when
> scp-ing a file to the machine.
>
> Any ideas on the cause of this? Any additional tests or tweaks I could
> try?
>
> Sijmen
>

Reply | Threaded
Open this post in threaded view
|

Re: Coming from FreeBSD, lower networking performance

Jordan Geoghegan
In reply to this post by Sijmen J. Mulder
Hello Sijmen,


On 07/13/18 16:20, Sijmen J. Mulder wrote:
> Hi all,
>
> After migrating a VPS from FreeBSD to OpenBSD I noticed reduced networking performance. Both incoming and outgoing traffic seems to be 2-3 times slower on average. By testing 100MB file transfers I've mostly eliminated the following factors:

> The easiest way to show and test this difference is as such:
>
>   1. Install either FreeBSD 11.2 or OpenBSD 6.3 on a machine or VM
>   2. Install curl, then `time curl -O http://download.thinkbroadband.com/100MB.zip`
>
> On both my VPS provider and my own PC with VirtualBox VMs the difference is about 3x for the above test. Similar results happen when scp-ing a file to the machine.

Please do some benchmarks using -current and send in the results. The
OpenBSD devs don't have time to do benchmarking, and always appreciate
proper benchmarking comparisons. For your networking tests, please don't
do any tests that involve disk writes. Disk speed is where you may have
issues, as most disk operations/drivers are not multithreaded and/or
kernel biglocked still.  For example, when using identical hardware (an
old core 2 duo Dell workstation), I can read from my Samsung 860 SSD at
~325MB/s on CentOS. On OpenBSD, I get more like ~130MB/s.  if you want
to test ssh/scp speed, make sure you write/output all ssh/scp data
transfers to /dev/null. What sort of data pipe are you working with? I
have found myself able to saturate a gigabit line fairly easily over ssh
using modest hardware.

Please reply back with some proper benchmarks as I'm sure many of us
would love to take a look.

Cheers,

Jordan

Reply | Threaded
Open this post in threaded view
|

Re: Coming from FreeBSD, lower networking performance

Bodie-3
In reply to this post by Sijmen J. Mulder


On 14.7.2018 01:20, Sijmen J. Mulder wrote:

> Hi all,
>
> After migrating a VPS from FreeBSD to OpenBSD I noticed reduced
> networking performance. Both incoming and outgoing traffic seems to be
> 2-3 times slower on average. By testing 100MB file transfers I've
> mostly eliminated the following factors:
>
>  - Protocol and ciphers (tested SCP, SFTP, FTP+TLS, HTTP, HTTPS)
>  - Client software
>  - Peer host
>  - VM provider/platform
>
> The easiest way to show and test this difference is as such:
>
>  1. Install either FreeBSD 11.2 or OpenBSD 6.3 on a machine or VM
>  2. Install curl, then `time curl -O
> http://download.thinkbroadband.com/100MB.zip`
>

1. What machine HW and VM are you testing? dmesg?
2. That page/file does not exist, but is that delay consistent
with eg. ftp(1) use instead of curl?

> On both my VPS provider and my own PC with VirtualBox VMs the
> difference is about 3x for the above test. Similar results happen when
> scp-ing a file to the machine.
>

VirtualBox is real crap when it comes to xBSD (not that it's much
better for others anyway). No matter what you set especially
I/O will be terrible

> Any ideas on the cause of this? Any additional tests or tweaks I could
> try?
>

systat, vmstat, top....... to find where is it showing bottleneck

As well do not tweak anything unless you can be sure what is the
weakpoint.
And then you may find that there is however nothing to tweak.

Most easy tweak for start is to install -current and test there
> Sijmen

Reply | Threaded
Open this post in threaded view
|

Re: Coming from FreeBSD, lower networking performance

Bodie-3
In reply to this post by Sijmen J. Mulder


On 14.7.2018 01:20, Sijmen J. Mulder wrote:
> Hi all,
>
> After migrating a VPS from FreeBSD to OpenBSD I noticed reduced
> networking performance. Both incoming and outgoing traffic seems to be
> 2-3 times slower on average. By testing 100MB file transfers I've
> mostly eliminated the following factors:

As well "define" slow.....

$ time curl -O
http://ftp.fi.netbsd.org/pub/NetBSD/NetBSD-current/tar_files/xsrc.tar.gz
   % Total    % Received % Xferd  Average Speed   Time    Time     Time  
Current
                                  Dload  Upload   Total   Spent    Left  
Speed
100  123M  100  123M    0     0  7646k      0  0:00:16  0:00:16 --:--:--
9242k
     0m16.58s real     0m00.09s user     0m03.24s system
$

That is OpenBSD 64bit snapshot, running as guest in VirtualBox on
Windows 10,
2 years old laptop, connected via WiFi in crowded space (at least 15
other networks
around) to not-perfect Euro DOCSIS 3.0 CPE via provider structure and
from
Czech Republic to Finland mirror of NetBSD

Theoretical maximum on my CPE is 18,75MB/s so having those speeds above
is
pretty nice in such environment.

BTW just for fun let's see how Windows 10 host deal with that.....


λ curl -O
http://ftp.fi.netbsd.org/pub/NetBSD/NetBSD-current/tar_files/xsrc.tar.gz
   % Total    % Received % Xferd  Average Speed   Time    Time     Time  
Current
                                  Dload  Upload   Total   Spent    Left  
Speed
100  123M  100  123M    0     0   661k      0  0:03:11  0:03:11 --:--:--
  704k


WHOAAAAAAA :-)




>
>  - Protocol and ciphers (tested SCP, SFTP, FTP+TLS, HTTP, HTTPS)
>  - Client software
>  - Peer host
>  - VM provider/platform
>
> The easiest way to show and test this difference is as such:
>
>  1. Install either FreeBSD 11.2 or OpenBSD 6.3 on a machine or VM
>  2. Install curl, then `time curl -O
> http://download.thinkbroadband.com/100MB.zip`
>
> On both my VPS provider and my own PC with VirtualBox VMs the
> difference is about 3x for the above test. Similar results happen when
> scp-ing a file to the machine.
>
> Any ideas on the cause of this? Any additional tests or tweaks I could
> try?
>
> Sijmen

Reply | Threaded
Open this post in threaded view
|

Re: Coming from FreeBSD, lower networking performance

Alarig Le Lay
In reply to this post by bit shifter
Hi,

On ven. 13 juil. 21:46:24 2018, bit shifter wrote:
> You could try running iperf to eliminate disk IO from the equation and
> narrow down the potential sources of the performance deficit you're
> seeing.

By curiosity, I ran a test here, on a already production running infra.
FreeBSD 11.1 is 20 times faster than OpenBSD 6.3; 10Gbps vs. 500M.

My raw terminal output is here: https://paste.swordarmor.fr/raw/vlix

--
alarig

Reply | Threaded
Open this post in threaded view
|

Re: Coming from FreeBSD, lower networking performance

Stuart Henderson
In reply to this post by Sijmen J. Mulder
On 2018-07-13, Sijmen J. Mulder <[hidden email]> wrote:
> After migrating a VPS from FreeBSD to OpenBSD I noticed reduced networking performance.

Yes, that's not a huge surprise, FreeBSD has taken speed as quite a high
priority, OpenBSD has concentrated more on other areas.

As more of the kernel becomes parallelised between multiple cpu cores
and bottlenecks are identified and fixed, this should improve.


Reply | Threaded
Open this post in threaded view
|

Re: Coming from FreeBSD, lower networking performance

Selveste1
Funny, cause I have almost identical speeds to/from my OpenBSD to/from
my Debian, but different results between OpenBSD to OpenBSD and Debian
to Debian.

All servers are sitting on a 4x1GB LACP-link in the same network switch

*OpenBSD 6.3 to OpenBSD 6.3*

**# iperf3 -4 -c ns1.semarkit.net
Connecting to host ns1.semarkit.net, port 5201
[  4] local 192.168.245.1 port 47049 connected to 192.168.245.9 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  56.3 MBytes   472 Mbits/sec                 
[  4]   1.00-2.00   sec  64.0 MBytes   537 Mbits/sec                 
[  4]   2.00-3.00   sec  64.8 MBytes   543 Mbits/sec                 
[  4]   3.00-4.00   sec  66.0 MBytes   554 Mbits/sec                 
[  4]   4.00-5.00   sec  64.8 MBytes   544 Mbits/sec                 
[  4]   5.00-6.00   sec  67.1 MBytes   562 Mbits/sec                 
[  4]   6.00-7.00   sec  66.7 MBytes   558 Mbits/sec                 
[  4]   7.00-8.00   sec  63.3 MBytes   533 Mbits/sec                 
[  4]   8.00-9.00   sec  65.2 MBytes   547 Mbits/sec                 
[  4]   9.00-10.00  sec  66.5 MBytes   558 Mbits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec   645 MBytes   541 Mbits/sec                  sender
[  4]   0.00-10.00  sec   645 MBytes   541 Mbits/sec                 
receiver

iperf Done.

*OpenBSD 6.3 to Debian Stretch*

# iperf3 -4 -c 192.168.245.7
Accepted connection from 192.168.245.7, port 55458
[  5] local 192.168.245.9 port 5201 connected to 192.168.245.7 port 55460
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  96.0 MBytes   805 Mbits/sec                 
[  5]   1.00-2.00   sec   136 MBytes  1.14 Gbits/sec                 
[  5]   2.00-3.00   sec   148 MBytes  1.25 Gbits/sec                 
[  5]   3.00-4.00   sec   163 MBytes  1.37 Gbits/sec                 
[  5]   4.00-5.00   sec   179 MBytes  1.50 Gbits/sec                 
[  5]   5.00-6.00   sec   178 MBytes  1.49 Gbits/sec                 
[  5]   6.00-7.00   sec   185 MBytes  1.55 Gbits/sec                 
[  5]   7.00-8.00   sec   179 MBytes  1.50 Gbits/sec                 
[  5]   8.00-9.00   sec   183 MBytes  1.54 Gbits/sec                 
[  5]   9.00-10.00  sec   183 MBytes  1.53 Gbits/sec                 
[  5]  10.00-10.00  sec  0.00 Bytes  0.00 bits/sec                 
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.00  sec  1.59 GBytes  1.37 Gbits/sec                 
receiver
-----------------------------------------------------------

*Debian Stretch to OpenBSD 6.3
*

# iperf3 -4 -c ns1.semarkit.net
Connecting to host ns1.semarkit.net, port 5201
[  4] local 192.168.245.7 port 55460 connected to 192.168.245.9 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  96.3 MBytes   808 Mbits/sec    0   55.1
KBytes      
[  4]   1.00-2.00   sec   136 MBytes  1.14 Gbits/sec    0   90.5
KBytes      
[  4]   2.00-3.00   sec   149 MBytes  1.25 Gbits/sec    0    124
KBytes      
[  4]   3.00-4.00   sec   163 MBytes  1.37 Gbits/sec    0    157
KBytes      
[  4]   4.00-5.00   sec   179 MBytes  1.50 Gbits/sec    0    191
KBytes      
[  4]   5.00-6.00   sec   178 MBytes  1.49 Gbits/sec    0    225
KBytes      
[  4]   6.00-7.00   sec   185 MBytes  1.56 Gbits/sec    0    266
KBytes      
[  4]   7.00-8.00   sec   179 MBytes  1.51 Gbits/sec    0    296
KBytes      
[  4]   8.00-9.00   sec   184 MBytes  1.54 Gbits/sec    0    327
KBytes      
[  4]   9.00-10.00  sec   183 MBytes  1.54 Gbits/sec    0    375
KBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  1.59 GBytes  1.37 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  1.59 GBytes  1.37 Gbits/sec                 
receiver

iperf Done.

*Debian Stretch to Debian Stretch*

# iperf3 -4 -c 192.168.245.231
Connecting to host 192.168.245.231, port 5201
[  4] local 192.168.245.7 port 55028 connected to 192.168.245.231 port 5201
[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec   269 MBytes  2.26 Gbits/sec    0   1.72
MBytes      
[  4]   1.00-2.00   sec   265 MBytes  2.22 Gbits/sec    0   2.17
MBytes      
[  4]   2.00-3.00   sec   232 MBytes  1.95 Gbits/sec  494   1.13
MBytes      
[  4]   3.00-4.00   sec   254 MBytes  2.13 Gbits/sec    0   1.17
MBytes      
[  4]   4.00-5.00   sec   291 MBytes  2.44 Gbits/sec    0   1.17
MBytes      
[  4]   5.00-6.00   sec   287 MBytes  2.40 Gbits/sec    0   1.17
MBytes      
[  4]   6.00-7.00   sec   297 MBytes  2.49 Gbits/sec    0   1.17
MBytes      
[  4]   7.00-8.00   sec   289 MBytes  2.42 Gbits/sec    0   1.17
MBytes      
[  4]   8.00-9.00   sec   301 MBytes  2.53 Gbits/sec    0   1.17
MBytes      
[  4]   9.00-10.00  sec   296 MBytes  2.48 Gbits/sec    0   1.42
MBytes      
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  2.72 GBytes  2.33 Gbits/sec  494             sender
[  4]   0.00-10.00  sec  2.71 GBytes  2.33 Gbits/sec                 
receiver

iperf Done.

Med Venlig Hilsen / Best Regards
Henrik Dige Semark

On 2018-07-14 14:09, Stuart Henderson wrote:
> On 2018-07-13, Sijmen J. Mulder <[hidden email]> wrote:
>> After migrating a VPS from FreeBSD to OpenBSD I noticed reduced networking performance.
> Yes, that's not a huge surprise, FreeBSD has taken speed as quite a high
> priority, OpenBSD has concentrated more on other areas.
>
> As more of the kernel becomes parallelised between multiple cpu cores
> and bottlenecks are identified and fixed, this should improve.
>
>