OpenBSD as a router on Oracle T5120

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

OpenBSD as a router on Oracle T5120

Eduardo Meyer
hello,

I am doing some basic testings on the above mentioned scenario and I am
stuck on some limits which I consider to be very low: I cannot get more
than 27Kpps and 200Mbit/s routing performance without starting to loose
packets.

System is:

# uname -srm

OpenBSD 5.4 sparc64

# sysctl hw

hw.machine=sparc64

hw.model=SUNW,UltraSPARC-T2 (rev 0.0) @ 1415.103 MHz

hw.ncpu=32

hw.byteorder=4321

hw.pagesize=8192

hw.disknames=sd0:dc8022901cadee32,sd1:,cd0:

hw.diskcount=3

hw.cpuspeed=1415

hw.vendor=Sun

hw.product=SUNW,SPARC-Enterprise-T5120

hw.physmem=8455716864

hw.usermem=8455700480

hw.ncpufound=32

hw.allowpowerdown=1

No tuning, and no firewall to (pfctl -d).

I am routing from em0 to em1 but also tried from em0 to em5 and em4 with
em5 mixing onboard and PCI ports and results are the very same.

Output from top points the bottleneck:

load averages:  0.17,  0.21,  0.12               bgp.newtelecom.net.br18:06:20

9 processes: 8 idle, 1 on processor

CPU00:  0.0% user,  0.0% nice,  0.0% system, 98.2% interrupt,  1.8% idle

CPU01:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU02:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU03:  0.2% user,  0.0% nice,  0.0% system,  0.0% interrupt, 99.8% idle

CPU04:  0.2% user,  0.0% nice,  0.2% system,  0.0% interrupt, 99.6% idle

CPU05:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU06:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU07:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU08:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU09:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU10:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU11:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU12:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU13:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU14:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU15:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU16:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU17:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU18:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU19:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU20:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU21:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU22:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU23:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU24:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU25:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU26:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU27:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU28:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU29:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle

CPU30:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle


All my NICs are getting interrupted on CPU0.

All 6 network cards are Intel 82571EB which support MSI-X and should, in
theory support IRQ balance.

So my question is, is there anything I can do to allow OpenBSD use more
than one CPU or at least choose which CPU will be used for each NIC?

What other tunings and settings and tweaks should I look for?

Is this performance expected to be so low on this machine? I got much
better numbers w/ OpenBSD on i386 servers.

Thank you for any hint ]:)

--
===========
Eduardo Meyer

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD as a router on Oracle T5120

Chris Cappuccio
Eduardo Meyer [[hidden email]] wrote:

> hello,
>
> I am doing some basic testings on the above mentioned scenario and I am
> stuck on some limits which I consider to be very low: I cannot get more
> than 27Kpps and 200Mbit/s routing performance without starting to loose
> packets.
>
> System is:
>
> # uname -srm
>
> OpenBSD 5.4 sparc64
...
> CPU00:  0.0% user,  0.0% nice,  0.0% system, 98.2% interrupt,  1.8% idle
...
> CPU30:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
>
> All my NICs are getting interrupted on CPU0.
>

OpenBSD doesn't yet support any other mode of operation, although you
may be seeing improvements here shortly.

> Thank you for any hint ]:)
>

You may wish to try a 5.5-beta snapshot which will improve the single-core
performance slightly, although the significant improvement of distributing
across all cores it not yet available.

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD as a router on Oracle T5120

Eduardo Meyer
On Mon, Jan 20, 2014 at 8:34 PM, Chris Cappuccio <[hidden email]> wrote:

> Eduardo Meyer [[hidden email]] wrote:
> > hello,
> >
> > I am doing some basic testings on the above mentioned scenario and I am
> > stuck on some limits which I consider to be very low: I cannot get more
> > than 27Kpps and 200Mbit/s routing performance without starting to loose
> > packets.
> >
> > System is:
> >
> > # uname -srm
> >
> > OpenBSD 5.4 sparc64
> ...
> > CPU00:  0.0% user,  0.0% nice,  0.0% system, 98.2% interrupt,  1.8% idle
> ...
> > CPU30:  0.0% user,  0.0% nice,  0.0% system,  0.0% interrupt,  100% idle
> >
> > All my NICs are getting interrupted on CPU0.
> >
>
> OpenBSD doesn't yet support any other mode of operation, although you
> may be seeing improvements here shortly.
>
> > Thank you for any hint ]:)
> >
>
> You may wish to try a 5.5-beta snapshot which will improve the single-core
> performance slightly, although the significant improvement of distributing
> across all cores it not yet available.
>

Dear Chris, thank you for your kind reply,

I will try
ftp://ftp.openbsd.org//pub/OpenBSD/snapshots/sparc64/install55.iso right
now. Other than simply running it is there anything else I should look at,
or any new command line tool to play around?

Thank you.



--
===========
Eduardo Meyer
pessoal: [hidden email]
profissional: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD as a router on Oracle T5120

Chris Cappuccio
Eduardo Meyer [[hidden email]] wrote:
>
> I will try
> ftp://ftp.openbsd.org//pub/OpenBSD/snapshots/sparc64/install55.iso right
> now. Other than simply running it is there anything else I should look at,
> or any new command line tool to play around?
>

Nope. All improvements here are going to be in the kernel. There is
work going on in the myx driver, for instance this change by dlg:

"introduce fine grained locking around the lists of packet handlers
myx maintains. this moves it away from relying on splnet to protect
them."

http://www.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pci/if_myx.c.diff?r1=1.47;r2=1.48

Nobody has done anything like this in if_em yet, but perhaps someone
will step up. For that matter, myx interrupts aren't yet distributed
across cores, but appear to be getting closer to that possibility.

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD as a router on Oracle T5120

Jeremie Courreges-Anglas-2
In reply to this post by Eduardo Meyer
Eduardo Meyer <[hidden email]> writes:

[...]

> I will try
> ftp://ftp.openbsd.org//pub/OpenBSD/snapshots/sparc64/install55.iso right
> now. Other than simply running it is there anything else I should look at,
> or any new command line tool to play around?

Disabling pf and kern.pool_debug would save some CPU cycles.

--
jca | PGP: 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE
(previous: 0x06A11494 / 61DB D9A0 00A4 67CF 2A90  8961 6191 8FBF 06A1 1494)

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD as a router on Oracle T5120

Claudio Jeker
In reply to this post by Chris Cappuccio
On Mon, Jan 20, 2014 at 04:00:05PM -0800, Chris Cappuccio wrote:

> Eduardo Meyer [[hidden email]] wrote:
> >
> > I will try
> > ftp://ftp.openbsd.org//pub/OpenBSD/snapshots/sparc64/install55.iso right
> > now. Other than simply running it is there anything else I should look at,
> > or any new command line tool to play around?
> >
>
> Nope. All improvements here are going to be in the kernel. There is
> work going on in the myx driver, for instance this change by dlg:
>
> "introduce fine grained locking around the lists of packet handlers
> myx maintains. this moves it away from relying on splnet to protect
> them."
>
> http://www.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pci/if_myx.c.diff?r1=1.47;r2=1.48
>
> Nobody has done anything like this in if_em yet, but perhaps someone
> will step up. For that matter, myx interrupts aren't yet distributed
> across cores, but appear to be getting closer to that possibility.

Don't expect too much from this yet. This is more of a test balloon to see
what is neccessary also I guess at first the performance will degrade.

--
:wq Claudio

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD as a router on Oracle T5120

Patrick Lamaiziere
In reply to this post by Eduardo Meyer
Le Mon, 20 Jan 2014 18:59:02 -0200,
Eduardo Meyer <[hidden email]> a écrit :

> hello,
>
> I am doing some basic testings on the above mentioned scenario and I
> am stuck on some limits which I consider to be very low: I cannot get
> more than 27Kpps and 200Mbit/s routing performance without starting
> to loose packets.
> ...
> All 6 network cards are Intel 82571EB which support MSI-X and should,
> in theory support IRQ balance.

MSI are disabled on this chipset since OpenBSD 5.2...
You can try to renabled MSI in em(4), here this helps a lot (on amd64).

Check the thread "(5.3) load problem on em(4) MSI / interrupt ?" on
misc@

https://www.mail-archive.com/misc@.../msg123743.html

Regards,

patch on 5.3:
--- /usr/src/sys/dev/pci/if_em.c.orig   Tue Oct  1 14:45:36 2013
+++ /usr/src/sys/dev/pci/if_em.c        Tue Oct  1 14:48:52 2013
@@ -337,7 +337,7 @@
         * Only use MSI on the newer PCIe parts, with the exception
         * of 82571/82572 due to "Byte Enables 2 and 3 Are Not Set"
errata */
-       if (sc->hw.mac_type <= em_82572)
+       if (sc->hw.mac_type < em_82571)
                sc->osdep.em_pa.pa_flags &= ~PCI_FLAGS_MSI_ENABLED;
 
        /* Parameters (to be read from user) */

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD as a router on Oracle T5120

Eduardo Meyer
On Tue, Jan 21, 2014 at 5:32 AM, Patrick Lamaiziere
<[hidden email]>wrote:

> Le Mon, 20 Jan 2014 18:59:02 -0200,
> Eduardo Meyer <[hidden email]> a écrit :
>
> > hello,
> >
> > I am doing some basic testings on the above mentioned scenario and I
> > am stuck on some limits which I consider to be very low: I cannot get
> > more than 27Kpps and 200Mbit/s routing performance without starting
> > to loose packets.
> > ...
> > All 6 network cards are Intel 82571EB which support MSI-X and should,
> > in theory support IRQ balance.
>
> MSI are disabled on this chipset since OpenBSD 5.2...
> You can try to renabled MSI in em(4), here this helps a lot (on amd64).
>
> Check the thread "(5.3) load problem on em(4) MSI / interrupt ?" on
> misc@
>
> https://www.mail-archive.com/misc@.../msg123743.html
>
> Regards,
>
> patch on 5.3:
> --- /usr/src/sys/dev/pci/if_em.c.orig   Tue Oct  1 14:45:36 2013
> +++ /usr/src/sys/dev/pci/if_em.c        Tue Oct  1 14:48:52 2013
> @@ -337,7 +337,7 @@
>          * Only use MSI on the newer PCIe parts, with the exception
>          * of 82571/82572 due to "Byte Enables 2 and 3 Are Not Set"
> errata */
> -       if (sc->hw.mac_type <= em_82572)
> +       if (sc->hw.mac_type < em_82571)
>                 sc->osdep.em_pa.pa_flags &= ~PCI_FLAGS_MSI_ENABLED;
>
>         /* Parameters (to be read from user) */
>

Thank you everyone, here we go with the results.

Disabling PF caused a 15% performance improvement. OpenBSD 5.5 made the
system more responsive under this load but made no real difference on pps
or bps results. MSIX did not cause any trouble and it helped to raise pps
up to 30K and bps up to 240M which is better but still very low :-(

Thank you all :-)