6.4 - RX not working on new supported BCM574xx (bnxt)

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

6.4 - RX not working on new supported BCM574xx (bnxt)

Luthing
Hello there,

I am facing an issue with a Broadcom NIC (specs here
https://www.broadcom.com/products/ethernet-connectivity/controllers/bcm57416/#specifications).

After some troubleshooting, I am not able to resolve listen ARP request from
my connected switch.
The NIC has negociated 10G with auto neg enabled on the switch.

The switch is able to see mac address on the port where the BCM NIC is
connected to. The switch can see ARP request going though him from the NIC.
But, my OBSD cannot receive packets.

I checked with a tcpdump -i bnxt0, and I see that I am sending ARP requests,
but not receiving any response.

A arp -a shows an incomplete mac address.


"dmesg | grep bn" output is :
bnxt0 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
20.8.163, apic 10 int 2, address 01:23:45:67:89
bnxt1 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
20.8.163, apic 10 int 5, address 01:23:45:67:8a
bnxt0 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
20.8.163, apic 10 int 2, address 01:23:45:67:89
bnxt1 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
20.8.163, apic 10 int 5, address 01:23:45:67:8a


Does anyone is using this NIC?
Can you help me on this?

Thanks
Luthing




--
Sent from: http://openbsd-archive.7691.n7.nabble.com/openbsd-dev-tech-f151936.html

Reply | Threaded
Open this post in threaded view
|

Re: 6.4 - RX not working on new supported BCM574xx (bnxt)

Carlos Cardenas
On Fri, Nov 09, 2018 at 04:35:38AM -0700, Luthing wrote:

> Hello there,
>
> I am facing an issue with a Broadcom NIC (specs here
> https://www.broadcom.com/products/ethernet-connectivity/controllers/bcm57416/#specifications).
>
> After some troubleshooting, I am not able to resolve listen ARP request from
> my connected switch.
> The NIC has negociated 10G with auto neg enabled on the switch.
>
> The switch is able to see mac address on the port where the BCM NIC is
> connected to. The switch can see ARP request going though him from the NIC.
> But, my OBSD cannot receive packets.
>
> I checked with a tcpdump -i bnxt0, and I see that I am sending ARP requests,
> but not receiving any response.
>
> A arp -a shows an incomplete mac address.
>
>
> "dmesg | grep bn" output is :
> bnxt0 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> 20.8.163, apic 10 int 2, address 01:23:45:67:89
> bnxt1 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> 20.8.163, apic 10 int 5, address 01:23:45:67:8a
> bnxt0 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> 20.8.163, apic 10 int 2, address 01:23:45:67:89
> bnxt1 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> 20.8.163, apic 10 int 5, address 01:23:45:67:8a
>
>
> Does anyone is using this NIC?
> Can you help me on this?
>
> Thanks
> Luthing

With a BCM57404 (10/25 SFP28), there are times I observe this after
boot.  When this happens, I power off the board and then do a cold boot.

We're still trying to track down root cause on this as it doesn't happen
on all NICs supported by this driver.

+--+
Carlos

Reply | Threaded
Open this post in threaded view
|

Re: 6.4 - RX not working on new supported BCM574xx (bnxt)

Jonathan Matthew-4
In reply to this post by Luthing
On Fri, Nov 09, 2018 at 04:35:38AM -0700, Luthing wrote:

> Hello there,
>
> I am facing an issue with a Broadcom NIC (specs here
> https://www.broadcom.com/products/ethernet-connectivity/controllers/bcm57416/#specifications).
>
> After some troubleshooting, I am not able to resolve listen ARP request from
> my connected switch.
> The NIC has negociated 10G with auto neg enabled on the switch.
>
> The switch is able to see mac address on the port where the BCM NIC is
> connected to. The switch can see ARP request going though him from the NIC.
> But, my OBSD cannot receive packets.
>
> I checked with a tcpdump -i bnxt0, and I see that I am sending ARP requests,
> but not receiving any response.

Send the full dmesg from your machine, output of 'ifconfig bnxt', 'vmstat -zi' and
probably some tcpdump output if anything shows up there.

Have you tried both ports, or just bnxt0?

>
> A arp -a shows an incomplete mac address.
>
>
> "dmesg | grep bn" output is :
> bnxt0 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> 20.8.163, apic 10 int 2, address 01:23:45:67:89
> bnxt1 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> 20.8.163, apic 10 int 5, address 01:23:45:67:8a
> bnxt0 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> 20.8.163, apic 10 int 2, address 01:23:45:67:89
> bnxt1 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> 20.8.163, apic 10 int 5, address 01:23:45:67:8a

Are those the real mac addresses?  They don't look real.

Reply | Threaded
Open this post in threaded view
|

Re: 6.4 - RX not working on new supported BCM574xx (bnxt)

David Gwynne-5


> On 10 Nov 2018, at 12:22 pm, Jonathan Matthew <[hidden email]> wrote:
>
> On Fri, Nov 09, 2018 at 04:35:38AM -0700, Luthing wrote:
>> Hello there,
>>
>> I am facing an issue with a Broadcom NIC (specs here
>> https://www.broadcom.com/products/ethernet-connectivity/controllers/bcm57416/#specifications).
>>
>> After some troubleshooting, I am not able to resolve listen ARP request from
>> my connected switch.
>> The NIC has negociated 10G with auto neg enabled on the switch.
>>
>> The switch is able to see mac address on the port where the BCM NIC is
>> connected to. The switch can see ARP request going though him from the NIC.
>> But, my OBSD cannot receive packets.
>>
>> I checked with a tcpdump -i bnxt0, and I see that I am sending ARP requests,
>> but not receiving any response.
>
> Send the full dmesg from your machine, output of 'ifconfig bnxt', 'vmstat -zi' and
> probably some tcpdump output if anything shows up there.
>
> Have you tried both ports, or just bnxt0?
>
>>
>> A arp -a shows an incomplete mac address.
>>
>>
>> "dmesg | grep bn" output is :
>> bnxt0 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
>> 20.8.163, apic 10 int 2, address 01:23:45:67:89
>> bnxt1 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
>> 20.8.163, apic 10 int 5, address 01:23:45:67:8a
>> bnxt0 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
>> 20.8.163, apic 10 int 2, address 01:23:45:67:89
>> bnxt1 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
>> 20.8.163, apic 10 int 5, address 01:23:45:67:8a
>
> Are those the real mac addresses?  They don't look real.

the low bit of the first byte is especially concerning, as that bit indicates the address is multicast. definitely looks like a placeholder value though, esp as it used on all ports.

Reply | Threaded
Open this post in threaded view
|

Re: 6.4 - RX not working on new supported BCM574xx (bnxt)

Tom Smyth
Does That card need specific firmware loaded?
like the older bnx cards ?
I dont see a reference to it in the bnxt man page...


On Sat, 10 Nov 2018 at 05:10, David Gwynne <[hidden email]> wrote:

>
>
>
> > On 10 Nov 2018, at 12:22 pm, Jonathan Matthew <[hidden email]> wrote:
> >
> > On Fri, Nov 09, 2018 at 04:35:38AM -0700, Luthing wrote:
> >> Hello there,
> >>
> >> I am facing an issue with a Broadcom NIC (specs here
> >> https://www.broadcom.com/products/ethernet-connectivity/controllers/bcm57416/#specifications).
> >>
> >> After some troubleshooting, I am not able to resolve listen ARP request from
> >> my connected switch.
> >> The NIC has negociated 10G with auto neg enabled on the switch.
> >>
> >> The switch is able to see mac address on the port where the BCM NIC is
> >> connected to. The switch can see ARP request going though him from the NIC.
> >> But, my OBSD cannot receive packets.
> >>
> >> I checked with a tcpdump -i bnxt0, and I see that I am sending ARP requests,
> >> but not receiving any response.
> >
> > Send the full dmesg from your machine, output of 'ifconfig bnxt', 'vmstat -zi' and
> > probably some tcpdump output if anything shows up there.
> >
> > Have you tried both ports, or just bnxt0?
> >
> >>
> >> A arp -a shows an incomplete mac address.
> >>
> >>
> >> "dmesg | grep bn" output is :
> >> bnxt0 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> >> 20.8.163, apic 10 int 2, address 01:23:45:67:89
> >> bnxt1 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> >> 20.8.163, apic 10 int 5, address 01:23:45:67:8a
> >> bnxt0 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> >> 20.8.163, apic 10 int 2, address 01:23:45:67:89
> >> bnxt1 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> >> 20.8.163, apic 10 int 5, address 01:23:45:67:8a
> >
> > Are those the real mac addresses?  They don't look real.
>
> the low bit of the first byte is especially concerning, as that bit indicates the address is multicast. definitely looks like a placeholder value though, esp as it used on all ports.
>


--
Kindest regards,
Tom Smyth

Mobile: +353 87 6193172
The information contained in this E-mail is intended only for the
confidential use of the named recipient. If the reader of this message
is not the intended recipient or the person responsible for
delivering it to the recipient, you are hereby notified that you have
received this communication in error and that any review,
dissemination or copying of this communication is strictly prohibited.
If you have received this in error, please notify the sender
immediately by telephone at the number above and erase the message
You are requested to carry out your own virus check before
opening any attachment.

Reply | Threaded
Open this post in threaded view
|

Re: 6.4 - RX not working on new supported BCM574xx (bnxt)

Jonathan Matthew-4
On Sat, Nov 10, 2018 at 12:32:01PM +0000, Tom Smyth wrote:
> Does That card need specific firmware loaded?
> like the older bnx cards ?
> I dont see a reference to it in the bnxt man page...

The firmware is stored on the card.  You can see the firmware package version
printed when the driver attaches:

bnxt0 at pci16 dev 0 function 0 "Broadcom BCM57404" rev 0x01: fw ver 20.8.171, apic 2 int 8, address 00:0a:f7:ae:e6:c8

or

bnxt0 at pci10 dev 0 function 0 "Broadcom BCM57416" rev 0x00: fw ver 20.6.151, apic 131 int 12, address d0:94:66:45:11:d0

>
>
> On Sat, 10 Nov 2018 at 05:10, David Gwynne <[hidden email]> wrote:
> >
> >
> >
> > > On 10 Nov 2018, at 12:22 pm, Jonathan Matthew <[hidden email]> wrote:
> > >
> > > On Fri, Nov 09, 2018 at 04:35:38AM -0700, Luthing wrote:
> > >> Hello there,
> > >>
> > >> I am facing an issue with a Broadcom NIC (specs here
> > >> https://www.broadcom.com/products/ethernet-connectivity/controllers/bcm57416/#specifications).
> > >>
> > >> After some troubleshooting, I am not able to resolve listen ARP request from
> > >> my connected switch.
> > >> The NIC has negociated 10G with auto neg enabled on the switch.
> > >>
> > >> The switch is able to see mac address on the port where the BCM NIC is
> > >> connected to. The switch can see ARP request going though him from the NIC.
> > >> But, my OBSD cannot receive packets.
> > >>
> > >> I checked with a tcpdump -i bnxt0, and I see that I am sending ARP requests,
> > >> but not receiving any response.
> > >
> > > Send the full dmesg from your machine, output of 'ifconfig bnxt', 'vmstat -zi' and
> > > probably some tcpdump output if anything shows up there.
> > >
> > > Have you tried both ports, or just bnxt0?
> > >
> > >>
> > >> A arp -a shows an incomplete mac address.
> > >>
> > >>
> > >> "dmesg | grep bn" output is :
> > >> bnxt0 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> > >> 20.8.163, apic 10 int 2, address 01:23:45:67:89
> > >> bnxt1 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> > >> 20.8.163, apic 10 int 5, address 01:23:45:67:8a
> > >> bnxt0 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> > >> 20.8.163, apic 10 int 2, address 01:23:45:67:89
> > >> bnxt1 at pci6 dev 0 function 0 "Broadcom BCM57416" rev 0x01: fw ver
> > >> 20.8.163, apic 10 int 5, address 01:23:45:67:8a
> > >
> > > Are those the real mac addresses?  They don't look real.
> >
> > the low bit of the first byte is especially concerning, as that bit indicates the address is multicast. definitely looks like a placeholder value though, esp as it used on all ports.
> >
>
>
> --
> Kindest regards,
> Tom Smyth
>
> Mobile: +353 87 6193172
> The information contained in this E-mail is intended only for the
> confidential use of the named recipient. If the reader of this message
> is not the intended recipient or the person responsible for
> delivering it to the recipient, you are hereby notified that you have
> received this communication in error and that any review,
> dissemination or copying of this communication is strictly prohibited.
> If you have received this in error, please notify the sender
> immediately by telephone at the number above and erase the message
> You are requested to carry out your own virus check before
> opening any attachment.
>

Reply | Threaded
Open this post in threaded view
|

Re: 6.4 - RX not working on new supported BCM574xx (bnxt)

Luthing
Hello there,

Sorry I haven't access to my server until today.

Please find the requested output here : https://pastebin.com/V5ayBfyS
(dmesg, ifconfig bnxt, vmstat -zi)

I modified mac addresses on my different outputs, the mac addresses are
normal.

Cheers
Luthing



--
Sent from: http://openbsd-archive.7691.n7.nabble.com/openbsd-dev-tech-f151936.html

Reply | Threaded
Open this post in threaded view
|

Re: 6.4 - RX not working on new supported BCM574xx (bnxt)

Luthing
FYI, the NIC is working well on Centos/Vmware OS.

From all of your findings, does the driver need to be updated ?
Can anyone has resolve this issue?

Many thanks
Luthing



--
Sent from: http://openbsd-archive.7691.n7.nabble.com/openbsd-dev-tech-f151936.html