Small pauses with a trunk(4) interface

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Small pauses with a trunk(4) interface

Bruno Carnazzi
  Hi all,

I use OpenBSD/i386 3.8 with GENERIC.MP.
I use a trunk interface based on 2 physical devices as you can see here :

$ ifconfig -a
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 33224
        groups: lo
        inet 127.0.0.1 netmask 0xff000000
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
fxp0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:02:a5:29:15:f0
        trunk: trunkdev trunk0
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet6 fe80::202:a5ff:fe29:15f0%fxp0 prefixlen 64 scopeid 0x1
xl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:0a:5e:5a:c9:a4
        trunk: trunkdev trunk0
        media: Ethernet autoselect (100baseTX full-duplex)
        status: active
        inet6 fe80::20a:5eff:fe5a:c9a4%xl0 prefixlen 64 scopeid 0x2
pflog0: flags=0<> mtu 33224
pfsync0: flags=0<> mtu 1348
enc0: flags=0<> mtu 1536
trunk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:02:a5:29:15:f0
        trunk: trunkproto roundrobin
                trunkport xl0
                trunkport fxp0 master
        groups: trunk egress
        media: Ethernet autoselect
        status: active
        inet 172.20.3.100 netmask 0xfffffe00 broadcast 172.20.3.255

Here's the dmesg (notice the xl0 strange message) :
OpenBSD 3.8 (GENERIC.MP) #298: Sat Sep 10 15:51:54 MDT 2005
    [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC.MP
cpu0: Intel Pentium III ("GenuineIntel" 686-class) 864 MHz
cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE
real mem  = 2147049472 (2096728K)
avail mem = 1953083392 (1907308K)
using 4278 buffers containing 107454464 bytes (104936K) of memory
mainbus0 (root)
bios0 at mainbus0: AT/286+(00) BIOS, date 12/31/99, BIOS32 rev. 0 @ 0xf0000
pcibios0 at bios0: rev 2.1 @ 0xf0000/0x2000
pcibios0: PCI BIOS has 7 Interrupt Routing table entries
pcibios0: PCI Interrupt Router at 000:15:0 ("ServerWorks ROSB4
SouthBridge" rev 0x00)
pcibios0: PCI bus #0 is the last bus
bios0: ROM list: 0xc0000/0x8000 0xc8000/0x4000! 0xcc000/0x800
0xe8000/0x6000 0xee000/0x2000!
mainbus0: Intel MP Specification (Version 1.4) (COMPAQ   PROLIANT    )
cpu0 at mainbus0: apid 1 (boot processor)
cpu0: apic clock running at 132 MHz
cpu1 at mainbus0: apid 0 (application processor)
cpu1: Intel Pentium III ("GenuineIntel" 686-class) 864 MHz
cpu1: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE
mainbus0: bus 0 is type PCI
mainbus0: bus 3 is type PCI
mainbus0: bus 9 is type ISA
ioapic0 at mainbus0: apid 8 pa 0xfec00000, version 11, 35 pins
ioapic0: misconfigured as apic 0, remapped to apic 8
pci0 at mainbus0 bus 0: configuration mode 1 (no bios)
pchb0 at pci0 dev 0 function 0 "ServerWorks CNB20LE Host" rev 0x05
pchb1 at pci0 dev 0 function 1 "ServerWorks CNB20LE Host" rev 0x05
pci1 at pchb1 bus 3
cac0 at pci0 dev 1 function 0 "Symbios Logic 53c1510" rev 0x02: apic 8
int 10 (irq 10) Compaq Integrated Array
scsibus0 at cac0: 1 targets
sd0 at scsibus0 targ 0 lun 0: <Compaq, RAID5 volume #, > SCSI2 0/direct fixed
sd0: 69455MB, 17432 cyl, 255 head, 32 sec, 512 bytes/sec, 142245120 sec total
fxp0 at pci0 dev 2 function 0 "Intel 82557" rev 0x08, i82559: apic 8
int 11 (irq 11), address 00:02:a5:29:15:f0
inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4
vga1 at pci0 dev 3 function 0 "ATI Mach64 GV" rev 0x7a
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
"Compaq Netelligent ASMC" rev 0x00 at pci0 dev 4 function 0 not configured
xl0 at pci0 dev 5 function 0 "3Com 3c905C 100Base-TX" rev 0x78: apic 8
int 5 (irq 5)xl0: reset didn't complete
, address 00:0a:5e:5a:c9:a4
exphy0 at xl0 phy 24: 3Com internal media interface
xl0: reset didn't complete
pcib0 at pci0 dev 15 function 0 "ServerWorks ROSB4 SouthBridge" rev 0x4f
pciide0 at pci0 dev 15 function 1 "ServerWorks OSB4 IDE" rev 0x00: DMA
atapiscsi0 at pciide0 channel 0 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <COMPAQ, CD-224E, 9.0B> SCSI0 5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, DMA mode 2
isa0 at pcib0
isadma0 at isa0
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
sysbeep0 at pcppi0
npx0 at isa0 port 0xf0/16: using exception 16
pccom0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pccom1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
fd0 at fdc0 drive 0: 1.44MB 80 cyl, 2 head, 18 sec
biomask 0 netmask 0 ttymask 0
pctr: 686-class user-level performance counters enabled
mtrr: Pentium Pro MTRR support
dkcsum: sd0 matches BIOS drive 0x80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02

With this configuration, I can see small network pauses  (1s) when
SSHing or downloading something through the local running squid. I
figure the trunk works this way : packet are sent through the member
interfaces with the trunk IP address and with the master NIC MAC@ and
packet are received on all trunk members thanks to the promiscuous
mode. As my NIC are plugged on the same L2 switch (for testing
purpose, maybe I'll use 2 different switch on a production
configuration), I suggest it bothers the switch with the same MAC@ on
2 ports. All error counters are null on this switch. I didn't
configure something like etherchannel or 802.3ad on these interface.

Does trunk operate this way ? Is there a way to avoid this phenomenon ?

Thank you,

Bruno.

Reply | Threaded
Open this post in threaded view
|

Re: Small pauses with a trunk(4) interface

Reyk Floeter-2
hi,

On Wed, Feb 01, 2006 at 06:42:47PM +0400, Bruno Carnazzi wrote:
> I use OpenBSD/i386 3.8 with GENERIC.MP.

trunk(4) support in OpenBSD 3.8 was quite new and there were some
bugfixes and improvements during the last development cycle. these
fixes, like trunkproto failover and multicast support for things like
pfsync or carp, will be available in OpenBSD 3.9.

> I use a trunk interface based on 2 physical devices as you can see here :
>
-//-

> trunk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>         lladdr 00:02:a5:29:15:f0
>         trunk: trunkproto roundrobin
>                 trunkport xl0
>                 trunkport fxp0 master
>         groups: trunk egress
>         media: Ethernet autoselect
>         status: active
>         inet 172.20.3.100 netmask 0xfffffe00 broadcast 172.20.3.255
>
-//-
> fxp0 at pci0 dev 2 function 0 "Intel 82557" rev 0x08, i82559: apic 8
> int 11 (irq 11), address 00:02:a5:29:15:f0
> inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4
-//-
> xl0 at pci0 dev 5 function 0 "3Com 3c905C 100Base-TX" rev 0x78: apic 8
> int 5 (irq 5)xl0: reset didn't complete
> , address 00:0a:5e:5a:c9:a4
> exphy0 at xl0 phy 24: 3Com internal media interface
> xl0: reset didn't complete

there's a problem with your 3com nic which is not related to trunk,
you should replace the nic ie. with a second fxp(4).

> With this configuration, I can see small network pauses  (1s) when
> SSHing or downloading something through the local running squid. I
> figure the trunk works this way : packet are sent through the member
> interfaces with the trunk IP address and with the master NIC MAC@ and
> packet are received on all trunk members thanks to the promiscuous
> mode. As my NIC are plugged on the same L2 switch (for testing
> purpose, maybe I'll use 2 different switch on a production
> configuration), I suggest it bothers the switch with the same MAC@ on
> 2 ports. All error counters are null on this switch. I didn't
> configure something like etherchannel or 802.3ad on these interface.
>

trunk(4) operates on layer 2, it is (not yet) related to any IP stuff.
it uses one MAC address borrowed from the master interface (currently
the first trunkport you add). in roundrobin mode, it distributes the
outgoing packets through all attached and active interfaces and
receives packets for this MAC from any active port. as you noticed,
your switch will get confused if you don't configure a trunk group
(cizzco calls it etherchannel or port group; hp procurve calls it
trunk).

OpenBSD 3.8 only supported trunk(4) in roundrobin mode, but i added a
failover mode in 3.8-current for the next release. failover mode only
uses the first active interface (primarily the master interface) for
packet distribution and does a failover to the next active interface
if the port's state link goes down. this works with "stupid" switches
or even hubs. the switch will probably take some time to learn the mac
address on a new port but only once in case of a failover.

IEEE 802.3ad is not supported at the moment, it just works fine in
simple roundrobin or in failover mode without any dynamic link
aggregation protocols (i don't really see the point for 802.3ad at the
moment, it adds no benefit just some dynamic "plug and play" foo).

> Does trunk operate this way ? Is there a way to avoid this phenomenon ?
>

- try again with enabling a trunk group on your switch
- try again with 3.9-beta
- try again in failover mode

my development is focussed on hp procurve switches because their trunk
implementation works pretty well and i can probably play with
802.3ad/LACP later (and they use openssh in their switches ;-)).

examples:

(1) openbsd trunkproto roundrobin <-> hp trunk

HPswitch(config)# no interface c4-c5 lacp
HPswitch(config)# interface c4-c5 trk1 trunk

openbsd# ifconfig trunk0
trunk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5a:9f:31:d6
        trunk: trunkproto roundrobin
                trunkport sk1
                trunkport sk0 master
        groups: trunk
        media: Ethernet autoselect
        status: active
        inet6 fe80::200:5aff:fe9f:31d6%trunk0 prefixlen 64 scopeid 0x9
        inet 172.23.5.70 netmask 0xffffff00 broadcast 172.23.5.255

the cizzco world is a bit more complex, ie.:

(2) openbsd trunkproto failover <-> plain cizzco 3750 switch

cizzco's 'trunk' is not related to port trunking, it's the description
for interfaces with multiple tagged and untagged vlans... don't get
confused by these terms! cizzco's trunk mode is _not_ required for
openbsd trunk(4) operation, but it's required for using tagged and
untagged 802.1q vlan(4)s on the switch ports.

cizzco3750#
interface GigabitEthernet2/0/19
 description openbsdtrunkmaster
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 300
 switchport trunk allowed vlan 300-699
 switchport mode trunk
 spanning-tree portfast trunk
!
interface GigabitEthernet2/0/20
 description openbsdtrunkfailover
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 300
 switchport trunk allowed vlan 300-699
 switchport mode trunk
 spanning-tree portfast trunk
!

openbsd# ifconfig trunk0
trunk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5a:9f:31:d6
        trunk: trunkproto failover
                trunkport sk1
                trunkport sk0 master
        groups: trunk
        media: Ethernet autoselect
        status: active
        inet6 fe80::200:5aff:fe9f:31d6%trunk0 prefixlen 64 scopeid 0x9
        inet 172.23.5.70 netmask 0xffffff00 broadcast 172.23.5.255
openbsd# ifconfig vlan301
vlan301: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:00:5a:9f:31:d6
        vlan: 300 parent interface: trunk0
        groups: vlan
        inet6 fe80::200:5aff:fe9f:31d6%vlan300 prefixlen 64 scopeid 0xc

(3) openbsd trunkproto roundrobin <-> cizzco 3512xl (old IOS 12.0)

cizzco3512#
interface FastEthernet0/1
 port group 1
 switchport access vlan 2
 spanning-tree portfast
!
interface FastEthernet0/2
 port group 1
 switchport access vlan 2
 spanning-tree portfast
!

openbsd# ifconfig trunk0
trunk0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
        lladdr 00:03:1d:01:a3:4e
        trunk: trunkproto roundrobin
                trunkport fxp1 active
                trunkport fxp0 master,active
        groups: trunk
        media: Ethernet autoselect
        status: active
        inet 172.23.6.252 netmask 0xffffff00 broadcast 172.23.6.255
        inet6 fe80::203:1dff:fe01:a34e%trunk0 prefixlen 64 scopeid 0x9

easy stuff!

i didn't test any other vendors. switch donations are welcome! ;)

i'll eventually add some new trunkprotos after the lock (like MAC
SA/DA, IP SA/DA, IP port) for load balancing issues. and probably
802.3ad/LACP in the far future... nevertheless, trunk(4) support in
OpenBSD 3.9 will work very well especially as a layer 2 addition for
failover scenarios with CARP+pfsync.

reyk

--
/* .vantronix|secure systems - (research & development)
 * reyk floeter - friendly known free software engineer
 * [hidden email] - http://team.vantronix.net/reyk/
 */

Reply | Threaded
Open this post in threaded view
|

Re: Small pauses with a trunk(4) interface

Bruno Carnazzi
Yeah !

Using your third suggested configuration on my old cisco 2950, I now
have a very responsive system...
Thank you for your fantastic work on trunk and these information (the
need for a switch configuration (etherchannel, HP trunking, etc.)
should be somewhere in manpages, in my mind...)

Best regards,

Bruno.

On 2/1/06, Reyk Floeter <[hidden email]> wrote:

> hi,
>
> On Wed, Feb 01, 2006 at 06:42:47PM +0400, Bruno Carnazzi wrote:
> > I use OpenBSD/i386 3.8 with GENERIC.MP.
>
> trunk(4) support in OpenBSD 3.8 was quite new and there were some
> bugfixes and improvements during the last development cycle. these
> fixes, like trunkproto failover and multicast support for things like
> pfsync or carp, will be available in OpenBSD 3.9.
>
> > I use a trunk interface based on 2 physical devices as you can see here :
> >
> -//-
> > trunk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> >         lladdr 00:02:a5:29:15:f0
> >         trunk: trunkproto roundrobin
> >                 trunkport xl0
> >                 trunkport fxp0 master
> >         groups: trunk egress
> >         media: Ethernet autoselect
> >         status: active
> >         inet 172.20.3.100 netmask 0xfffffe00 broadcast 172.20.3.255
> >
> -//-
> > fxp0 at pci0 dev 2 function 0 "Intel 82557" rev 0x08, i82559: apic 8
> > int 11 (irq 11), address 00:02:a5:29:15:f0
> > inphy0 at fxp0 phy 1: i82555 10/100 PHY, rev. 4
> -//-
> > xl0 at pci0 dev 5 function 0 "3Com 3c905C 100Base-TX" rev 0x78: apic 8
> > int 5 (irq 5)xl0: reset didn't complete
> > , address 00:0a:5e:5a:c9:a4
> > exphy0 at xl0 phy 24: 3Com internal media interface
> > xl0: reset didn't complete
>
> there's a problem with your 3com nic which is not related to trunk,
> you should replace the nic ie. with a second fxp(4).
>
> > With this configuration, I can see small network pauses  (1s) when
> > SSHing or downloading something through the local running squid. I
> > figure the trunk works this way : packet are sent through the member
> > interfaces with the trunk IP address and with the master NIC MAC@ and
> > packet are received on all trunk members thanks to the promiscuous
> > mode. As my NIC are plugged on the same L2 switch (for testing
> > purpose, maybe I'll use 2 different switch on a production
> > configuration), I suggest it bothers the switch with the same MAC@ on
> > 2 ports. All error counters are null on this switch. I didn't
> > configure something like etherchannel or 802.3ad on these interface.
> >
>
> trunk(4) operates on layer 2, it is (not yet) related to any IP stuff.
> it uses one MAC address borrowed from the master interface (currently
> the first trunkport you add). in roundrobin mode, it distributes the
> outgoing packets through all attached and active interfaces and
> receives packets for this MAC from any active port. as you noticed,
> your switch will get confused if you don't configure a trunk group
> (cizzco calls it etherchannel or port group; hp procurve calls it
> trunk).
>
> OpenBSD 3.8 only supported trunk(4) in roundrobin mode, but i added a
> failover mode in 3.8-current for the next release. failover mode only
> uses the first active interface (primarily the master interface) for
> packet distribution and does a failover to the next active interface
> if the port's state link goes down. this works with "stupid" switches
> or even hubs. the switch will probably take some time to learn the mac
> address on a new port but only once in case of a failover.
>
> IEEE 802.3ad is not supported at the moment, it just works fine in
> simple roundrobin or in failover mode without any dynamic link
> aggregation protocols (i don't really see the point for 802.3ad at the
> moment, it adds no benefit just some dynamic "plug and play" foo).
>
> > Does trunk operate this way ? Is there a way to avoid this phenomenon ?
> >
>
> - try again with enabling a trunk group on your switch
> - try again with 3.9-beta
> - try again in failover mode
>
> my development is focussed on hp procurve switches because their trunk
> implementation works pretty well and i can probably play with
> 802.3ad/LACP later (and they use openssh in their switches ;-)).
>
> examples:
>
> (1) openbsd trunkproto roundrobin <-> hp trunk
>
> HPswitch(config)# no interface c4-c5 lacp
> HPswitch(config)# interface c4-c5 trk1 trunk
>
> openbsd# ifconfig trunk0
> trunk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>         lladdr 00:00:5a:9f:31:d6
>         trunk: trunkproto roundrobin
>                 trunkport sk1
>                 trunkport sk0 master
>         groups: trunk
>         media: Ethernet autoselect
>         status: active
>         inet6 fe80::200:5aff:fe9f:31d6%trunk0 prefixlen 64 scopeid 0x9
>         inet 172.23.5.70 netmask 0xffffff00 broadcast 172.23.5.255
>
> the cizzco world is a bit more complex, ie.:
>
> (2) openbsd trunkproto failover <-> plain cizzco 3750 switch
>
> cizzco's 'trunk' is not related to port trunking, it's the description
> for interfaces with multiple tagged and untagged vlans... don't get
> confused by these terms! cizzco's trunk mode is _not_ required for
> openbsd trunk(4) operation, but it's required for using tagged and
> untagged 802.1q vlan(4)s on the switch ports.
>
> cizzco3750#
> interface GigabitEthernet2/0/19
>  description openbsdtrunkmaster
>  switchport trunk encapsulation dot1q
>  switchport trunk native vlan 300
>  switchport trunk allowed vlan 300-699
>  switchport mode trunk
>  spanning-tree portfast trunk
> !
> interface GigabitEthernet2/0/20
>  description openbsdtrunkfailover
>  switchport trunk encapsulation dot1q
>  switchport trunk native vlan 300
>  switchport trunk allowed vlan 300-699
>  switchport mode trunk
>  spanning-tree portfast trunk
> !
>
> openbsd# ifconfig trunk0
> trunk0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>         lladdr 00:00:5a:9f:31:d6
>         trunk: trunkproto failover
>                 trunkport sk1
>                 trunkport sk0 master
>         groups: trunk
>         media: Ethernet autoselect
>         status: active
>         inet6 fe80::200:5aff:fe9f:31d6%trunk0 prefixlen 64 scopeid 0x9
>         inet 172.23.5.70 netmask 0xffffff00 broadcast 172.23.5.255
> openbsd# ifconfig vlan301
> vlan301: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
>         lladdr 00:00:5a:9f:31:d6
>         vlan: 300 parent interface: trunk0
>         groups: vlan
>         inet6 fe80::200:5aff:fe9f:31d6%vlan300 prefixlen 64 scopeid 0xc
>
> (3) openbsd trunkproto roundrobin <-> cizzco 3512xl (old IOS 12.0)
>
> cizzco3512#
> interface FastEthernet0/1
>  port group 1
>  switchport access vlan 2
>  spanning-tree portfast
> !
> interface FastEthernet0/2
>  port group 1
>  switchport access vlan 2
>  spanning-tree portfast
> !
>
> openbsd# ifconfig trunk0
> trunk0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> mtu 1500
>         lladdr 00:03:1d:01:a3:4e
>         trunk: trunkproto roundrobin
>                 trunkport fxp1 active
>                 trunkport fxp0 master,active
>         groups: trunk
>         media: Ethernet autoselect
>         status: active
>         inet 172.23.6.252 netmask 0xffffff00 broadcast 172.23.6.255
>         inet6 fe80::203:1dff:fe01:a34e%trunk0 prefixlen 64 scopeid 0x9
>
> easy stuff!
>
> i didn't test any other vendors. switch donations are welcome! ;)
>
> i'll eventually add some new trunkprotos after the lock (like MAC
> SA/DA, IP SA/DA, IP port) for load balancing issues. and probably
> 802.3ad/LACP in the far future... nevertheless, trunk(4) support in
> OpenBSD 3.9 will work very well especially as a layer 2 addition for
> failover scenarios with CARP+pfsync.
>
> reyk
>
> --
> /* .vantronix|secure systems - (research & development)
>  * reyk floeter - friendly known free software engineer
>  * [hidden email] - http://team.vantronix.net/reyk/
>  */