Hardware fault on M3000

[michael-2]

Hello,

I have a Fujitsu SPARC Enterprise M3000 that appears to have been
rendered inoperable by the OpenBSD 6.2 installer. After partitioning
and during the installation of sets, the machine abruptly shut down and
faulted its motherboard. From my limited research, it appears the only
way to clear this fault is to have Oracle dispatch a field technician,
which is out of my reach as a hobbyist. Even a full factory reset of
the SCF did not successfully clear the fault, and the machine now
refuses to power on.

Somone named Naruaki Etomi posted a message to this list on March 28,
2015, complaining of the same issue with OpenBSD 5.6. The description
of the issue in that post is virtually identical to what happened to
my machine, down to the same fault code (SCF-8003-HA), and the same
timing (during the expansion of the sets). If I hadn't found that post,
I would have dismissed this as an unlucky hardware issue.

I'm curious whether anyone is successfully using OpenBSD on an M3000.
If not, perhaps some kind of warning could be added to the SPARC64
port's website to discourage further testing?

Below is some output from the XSCF showing the fault that occurred.
It's very similar to the Etomi post from 2015.

XSCF> fmdump -v -u 97a2854b-5603-4efa-b609-127469fc445d
TIME UUID MSG-ID
Jan 10 17:28:26.7967 97a2854b-5603-4efa-b609-127469fc445d SCF-8003-HA
100% fault.chassis.SPARC-Enterprise.asic.mbc.fe

Problem in: hc:///chassis=0/cmu=0/mbc=0
Affects: hc:///chassis=0/cmu=0/xsb=0
FRU: hc://:product-id=SPARC Enterprise M3000:chassis-id=PX61011015:
server-id=brad:serial=PP101000WS:part=CA07082-D051 D1 \541-4281-04:
revision=0301/component=/MBU_A
Location: /MBU_A

XSCF> showstatus
* MBU_A Status:Faulted;
* CPU Status:Deconfigured;
* MEM#0A Status:Deconfigured;
* MEM#0B Status:Deconfigured;
* MEM#1A Status:Deconfigured;
* MEM#1B Status:Deconfigured;
* MEM#2A Status:Deconfigured;
* MEM#2B Status:Deconfigured;
* MEM#3A Status:Deconfigured;
* MEM#3B Status:Deconfigured;

XSCF> poweron -y -d 0
DomainIDs to power on:00
Continue? [y|n] :y
00 :Not powering on :Poweron canceled due to missing component.

I'd be glad to collect any other information that might be useful
to the community.

Thanks
Michael Proctor
[hidden email]

[tinkr]

> Hello,
>
> I have a Fujitsu SPARC Enterprise M3000 that appears to have been
> rendered inoperable by the OpenBSD 6.2 installer. After partitioning
> and during the installation of sets, the machine abruptly shut down and
> faulted its motherboard. From my limited research, it appears the only
> way to clear this fault is to have Oracle dispatch a field technician,
> which is out of my reach as a hobbyist. Even a full factory reset of
> the SCF did not successfully clear the fault, and the machine now
> refuses to power on.

Offlist 2016-10-23 Theo said "We have had lots of them [M3000].  And it does not work
because of something we didn't figure out."

As you see on https://www.openbsd.org/sparc64.html M3000 is not listed as supported.

Sorry to hear you had this issue.

[Jeff Veiss]

Hi,

I'd be surprised if OpenBSD caused that. I'm pretty sure it doesn't touch the firmware particularly the system controller (XSCF). It's more likely that your system board has simply failed.  If this was an M4000 or M5000, I would suggest reseating the board. In this case, you can try doing a full power reset (pull the power cords for at least a minute) or possibly doing a full reset of the XSCF.

Add a last resort, if you don't pay Oracle for support, you might be able to find a replacement cheaply on eBay. I know sparc hardware has been going cheap there recently.
--
-Jeff

On January 13, 2018 10:03:11 AM MST, [hidden email] wrote:

>Hello,
>
>I have a Fujitsu SPARC Enterprise M3000 that appears to have been
>rendered inoperable by the OpenBSD 6.2 installer. After partitioning
>and during the installation of sets, the machine abruptly shut down and
>faulted its motherboard. From my limited research, it appears the only
>way to clear this fault is to have Oracle dispatch a field technician,
>which is out of my reach as a hobbyist. Even a full factory reset of
>the SCF did not successfully clear the fault, and the machine now
>refuses to power on.
>
>Somone named Naruaki Etomi posted a message to this list on March 28,
>2015, complaining of the same issue with OpenBSD 5.6. The description
>of the issue in that post is virtually identical to what happened to
>my machine, down to the same fault code (SCF-8003-HA), and the same
>timing (during the expansion of the sets). If I hadn't found that post,
>I would have dismissed this as an unlucky hardware issue.
>
>I'm curious whether anyone is successfully using OpenBSD on an M3000.
>If not, perhaps some kind of warning could be added to the SPARC64
>port's website to discourage further testing?
>
>Below is some output from the XSCF showing the fault that occurred.
>It's very similar to the Etomi post from 2015.
>
>XSCF> fmdump -v -u 97a2854b-5603-4efa-b609-127469fc445d
>TIME UUID MSG-ID
>Jan 10 17:28:26.7967 97a2854b-5603-4efa-b609-127469fc445d SCF-8003-HA
>100% fault.chassis.SPARC-Enterprise.asic.mbc.fe
>
>Problem in: hc:///chassis=0/cmu=0/mbc=0
>Affects: hc:///chassis=0/cmu=0/xsb=0
>FRU: hc://:product-id=SPARC Enterprise M3000:chassis-id=PX61011015:
>server-id=brad:serial=PP101000WS:part=CA07082-D051 D1 \541-4281-04:
>revision=0301/component=/MBU_A
>Location: /MBU_A
>
>XSCF> showstatus
>* MBU_A Status:Faulted;
>* CPU Status:Deconfigured;
>* MEM#0A Status:Deconfigured;
>* MEM#0B Status:Deconfigured;
>* MEM#1A Status:Deconfigured;
>* MEM#1B Status:Deconfigured;
>* MEM#2A Status:Deconfigured;
>* MEM#2B Status:Deconfigured;
>* MEM#3A Status:Deconfigured;
>* MEM#3B Status:Deconfigured;
>
>XSCF> poweron -y -d 0
>DomainIDs to power on:00
>Continue? [y|n] :y
>00 :Not powering on :Poweron canceled due to missing component.
>
>I'd be glad to collect any other information that might be useful
>to the community.
>
>Thanks
>Michael Proctor
>[hidden email]

[michael-2]

> Offlist 2016-10-23 Theo said "We have had lots of them [M3000].  And it
> does not work
> because of something we didn't figure out."
>
> As you see on https://www.openbsd.org/sparc64.html M3000 is not listed as
> supported.
>
> Sorry to hear you had this issue.

Thanks, it's good to know this isn't a new problem. If this is a known
failure mode, maybe the M3000 could be moved from "untested" to
"unsupported" on sparc64.html, to prevent anyone else from making
the same mistake.

[Tijs Michels]

I bought the same machine, hoping I could put OpenBSD on it.  Since I have
OpenBSD running on my PrimePower 250, I did not expect serious problems.
Luckily, I read just in time, in two separate threads, that any attempt to
install OpenBSD on it will actually brick the M3000.  So now it sits in a
corner, unused, and the PrimePower has to soldier on.  Such a shame.

On Sat, Jan 13, 2018 at 7:47 PM, <[hidden email]> wrote:

> > Offlist 2016-10-23 Theo said "We have had lots of them [M3000].  And it
> > does not work
> > because of something we didn't figure out."
> >
> > As you see on https://www.openbsd.org/sparc64.html M3000 is not listed
> as
> > supported.
> >
> > Sorry to hear you had this issue.
>
> Thanks, it's good to know this isn't a new problem. If this is a known
> failure mode, maybe the M3000 could be moved from "untested" to
> "unsupported" on sparc64.html, to prevent anyone else from making
> the same mistake.
>
>
>
>

[amcwhirter]

Is this specific to the M3000? Because I run 6.1 on a M4000 without issue
which is nearly identical architecture wise.
IIRC if you do a full xscf reset and get to the initial user setup dialog
you can add field engineer status to any account.

[michael-2]

> Is this specific to the M3000? Because I run 6.1 on a M4000 without issue
> which is nearly identical architecture wise.
> IIRC if you do a full xscf reset and get to the initial user setup dialog
> you can add field engineer status to any account.

I was able to assign the fieldeng and mode privileges to my user
after performing a full reset of the XSCF, but trying to use
enableservice requires a service password from Oracle support.

On the M3000, addboard and deleteboard are not supported, and
replacefru only allows me to replace fans and PSUs -- I guess
because there is only one system board and it's not considered
field-replaceable.

[Tijs Michels]

It is not just this one system board.  Last year I found multiple reports
of people who had bricked their M3000 when they tried to install OpenBSD.
Here are three:

https://www.mail-archive.com/sparc@.../msg00523.html
https://anindito.my.id/openbsd-bricks-sun-enterprise-m3000/
http://www.abclinuxu.cz/poradna/unix/show/383354

That second link seems dead now.  Still, it's a known issue.

On Sat, Jan 13, 2018 at 7:31 PM, Jeff Veiss <[hidden email]> wrote:

> Hi,
>
> I'd be surprised if OpenBSD caused that. I'm pretty sure it doesn't touch
> the firmware particularly the system controller (XSCF). It's more likely
> that your system board has simply failed.  If this was an M4000 or M5000, I
> would suggest reseating the board. In this case, you can try doing a full
> power reset (pull the power cords for at least a minute) or possibly doing
> a full reset of the XSCF.
>
> Add a last resort, if you don't pay Oracle for support, you might be able
> to find a replacement cheaply on eBay. I know sparc hardware has been going
> cheap there recently.
> --
> -Jeff
>
> On January 13, 2018 10:03:11 AM MST, [hidden email] wrote:
> >Hello,
> >
> >I have a Fujitsu SPARC Enterprise M3000 that appears to have been
> >rendered inoperable by the OpenBSD 6.2 installer. After partitioning
> >and during the installation of sets, the machine abruptly shut down and
> >faulted its motherboard. From my limited research, it appears the only
> >way to clear this fault is to have Oracle dispatch a field technician,
> >which is out of my reach as a hobbyist. Even a full factory reset of
> >the SCF did not successfully clear the fault, and the machine now
> >refuses to power on.
> >
> >Somone named Naruaki Etomi posted a message to this list on March 28,
> >2015, complaining of the same issue with OpenBSD 5.6. The description
> >of the issue in that post is virtually identical to what happened to
> >my machine, down to the same fault code (SCF-8003-HA), and the same
> >timing (during the expansion of the sets). If I hadn't found that post,
> >I would have dismissed this as an unlucky hardware issue.
> >
> >I'm curious whether anyone is successfully using OpenBSD on an M3000.
> >If not, perhaps some kind of warning could be added to the SPARC64
> >port's website to discourage further testing?
> >
> >Below is some output from the XSCF showing the fault that occurred.
> >It's very similar to the Etomi post from 2015.
> >
> >XSCF> fmdump -v -u 97a2854b-5603-4efa-b609-127469fc445d
> >TIME UUID MSG-ID
> >Jan 10 17:28:26.7967 97a2854b-5603-4efa-b609-127469fc445d SCF-8003-HA
> >100% fault.chassis.SPARC-Enterprise.asic.mbc.fe
> >
> >Problem in: hc:///chassis=0/cmu=0/mbc=0
> >Affects: hc:///chassis=0/cmu=0/xsb=0
> >FRU: hc://:product-id=SPARC Enterprise M3000:chassis-id=PX61011015:
> >server-id=brad:serial=PP101000WS:part=CA07082-D051 D1 \541-4281-04:
> >revision=0301/component=/MBU_A
> >Location: /MBU_A
> >
> >XSCF> showstatus
> >* MBU_A Status:Faulted;
> >* CPU Status:Deconfigured;
> >* MEM#0A Status:Deconfigured;
> >* MEM#0B Status:Deconfigured;
> >* MEM#1A Status:Deconfigured;
> >* MEM#1B Status:Deconfigured;
> >* MEM#2A Status:Deconfigured;
> >* MEM#2B Status:Deconfigured;
> >* MEM#3A Status:Deconfigured;
> >* MEM#3B Status:Deconfigured;
> >
> >XSCF> poweron -y -d 0
> >DomainIDs to power on:00
> >Continue? [y|n] :y
> >00 :Not powering on :Poweron canceled due to missing component.
> >
> >I'd be glad to collect any other information that might be useful
> >to the community.
> >
> >Thanks
> >Michael Proctor
> >[hidden email]
>

 
I bricked an M3000 installing 6.1, last year.
 
 
 
Error on the console when the orange fault light went on:
 
 
"May  3 01:49:10 localhost fmd: SOURCE: sde, REV: 1.16, CSN: PX61212030
  EVENT-ID: d6245ec0-3f56-4ab1-8de1-adc440ce3a81 Refer to [ http://www ]( http://www ).
sun.com/msg/SCF-8002-12 for detailed information."
 
 
Tried to find a way to clear the fault for days, before I got
so angry, I pulled my drives and dumped it at a recycler.

 
Meghan
 
 

>Offlist 2016-10-23 Theo said "We have had lots of them [M3000]. And it does not work
>because of something we didn't figure out."

That would have been great to know before I blew US$400 in May of 2017.
 
"As you see on https://www.openbsd.org/sparc64.html M3000 is not listed as supported."
It's listed under systems that are untested, but:
!!!! ". . . most of these machines will almost certainly just work." !!!!
 
This machine is still costly on eBay. Please update the the page before
anyone else finds out the hard way what was already known to devs.
 
Meghan

[Kyle K]

Wish I searched about this before trying to install OpenBSD 6.2. I just
bricked my M3000 too few hours ago, sucks, I tried resetting the XSCF using:

XSCF> restoredefaults -c factory

but no avail. I'm not giving up yet though, I have a Sun XVR-300 Graphics
Adapter in my M3000 and I was using it as a desktop running Solaris with X11
up to this point by having modified intake and PSU fans to quiet them down
and make server bearble to use. Plus I invested in 2HDDs and 32GB of RAM, I
don't want to lose this.

I was searching and reading that there's a way to obtain CUA number that
will allow you to sign up for support at Oracle site. You have to purchase a
random software from their online shop (preferably cheap one) and buy
support along the side. Once you make a purchase you should get welcome
letter from Oracle along with CUA number, which then will give you SI
(Support Identifier). Then you should be able to complete your Account
Registration at on support.oracle.com and create tickets and submit a case
asking for service password that will unlock command to clear the fault
codes. I have found "Oracle Configuration Management Pack for Applications
(Named User; 1 Year)" for $20 and $22 for 1 year support for that, totaling
$42, not bad?
https://shop.oracle.com/apex/f?p=DSTORE:6:19266001435453:::6:P6_LPI,P6_PPI,P6_METRIC,P6_TERM:1621014036861810302734,3415207466896555108,Named%20User,1_Year

If that works out I could help some of you to unbrick your M3000 as well.
Let me know. Hoepfully you didn't trash it yet.





--
Sent from: http://openbsd-archive.7691.n7.nabble.com/openbsd-port-sparc-f200320.html

[Mark Kettenis]

> Date: Sat, 27 Jan 2018 06:18:09 -0700 (MST)
> From: Kyle K <[hidden email]>
>
> Wish I searched about this before trying to install OpenBSD 6.2. I just
> bricked my M3000 too few hours ago, sucks, I tried resetting the XSCF using:
>
> XSCF> restoredefaults -c factory
>
> but no avail. I'm not giving up yet though, I have a Sun XVR-300 Graphics
> Adapter in my M3000 and I was using it as a desktop running Solaris with X11
> up to this point by having modified intake and PSU fans to quiet them down
> and make server bearble to use. Plus I invested in 2HDDs and 32GB of RAM, I
> don't want to lose this.
>
> I was searching and reading that there's a way to obtain CUA number that
> will allow you to sign up for support at Oracle site. You have to purchase a
> random software from their online shop (preferably cheap one) and buy
> support along the side. Once you make a purchase you should get welcome
> letter from Oracle along with CUA number, which then will give you SI
> (Support Identifier). Then you should be able to complete your Account
> Registration at on support.oracle.com and create tickets and submit a case
> asking for service password that will unlock command to clear the fault
> codes. I have found "Oracle Configuration Management Pack for Applications
> (Named User; 1 Year)" for $20 and $22 for 1 year support for that, totaling
> $42, not bad?
> https://shop.oracle.com/apex/f?p=DSTORE:6:19266001435453:::6:P6_LPI,P6_PPI,P6_METRIC,P6_TERM:1621014036861810302734,3415207466896555108,Named%20User,1_Year
>
> If that works out I could help some of you to unbrick your M3000 as well.
> Let me know. Hoepfully you didn't trash it yet.

Hope that'll work for you.

Note that if you're running a recent-enough version of the XCP
firmware, you can clear the fault yourself.  XCP 1116 is known to
allow this.  I'm not sure what the earliest version version is that no
longer requires the help of a field engineer to clear spurious
hardware faults like this.

There is no evidence that OpenBSD actually damages the hardware.  All
evidence points at the firmware not being able to handle the way
OpenBSD outputs characters over the serial console; the same issue
that plagued the M4000/M5000 (and presumable the M8000/M9000).  I
thought a workaround was in place but apparently it doesn't work for
the M3000.

Apart from this issue, OpenBSD would run just fine on these machines
:(.  In my opinion Oracle should not require customers to sign up for
support to make machines work again that have been a victim of
firmware bugs.  But apparently the "Enterprise" in the name of these
machines means that normal consumer protection doesn't apply...

[amcwhirter]

I'm curious if updating the firmware after the fault has occurred will
allow users to reset it? I'm sure I have some m3000 firmware
somewhere...

[Mark Kettenis]

> I'm curious if updating the firmware after the fault has occurred
> will allow users to reset it? I'm sure I have some m3000 firmware
> somewhere...

Unfortunately not.  You can't update the firmware while the machine is
in the faulted state.

[Kyle K]

Hello guys,

I went ahead and purchased the "Oracle Configuration Management Pack for
Applications" for $20 along with 1-year of support for that software for $22.
With this I was able to gain access to Oracle's internal portal and ability
to file SR (service request) which is more or less a support ticket.
Problem was that I couldn't file a ticket regarding my M3000 server since I
don't have support contract (probably costs thousands?) but I filed a
ticket anyway for the software that I bought but used the title&body of the
ticket and explained my M3000 fault code situation. Well the ticket was
promptly left aside due to being an invalid request, however by this point
I had enough information to call the support center and explain things
further (you need SR # of your ticket to past automatic machine prompt to
be able to reach a person once you call). Oracle gentlemen was super
helpful and transferred the ticket to M3000/M4000/M5000 team within minutes.

I have been told that they don't provide service passwords anymore and that
there are 2 options, either dispatch a Field Engineer to solve the problem
or engage in web conference. I went with latter and I'm going to have a
conference call later today and I should be able to collaborate with him
1-on-1 and give SSH access to the XSCF from where he can do the magic. I
will provide you guys with an update. I might unbrick my M3000, but I don't
believe I will be able to help any of you, if they provided service
passwords I could have just play along and make 5 more SRs with serial
numbers of your M3000s and help you out.

Kyle.

On Sat, Jan 27, 2018 at 3:53 PM, Mark Kettenis <[hidden email]>
wrote:

> > I'm curious if updating the firmware after the fault has occurred
> > will allow users to reset it? I'm sure I have some m3000 firmware
> > somewhere...
>
> Unfortunately not.  You can't update the firmware while the machine is
> in the faulted state.
>

[Paolo Aglialoro]

Unless u find in this little time how to keylog your xscf ssh session...
some box in the middle?

Il 31 gen 2018 10:59 AM, "Kyle K" <[hidden email]> ha scritto:

> Hello guys,
>
> I went ahead and purchased the "Oracle Configuration Management Pack for
> Applications" for $20 along with 1-year of support for that software for
> $22.
> With this I was able to gain access to Oracle's internal portal and ability
> to file SR (service request) which is more or less a support ticket.
> Problem was that I couldn't file a ticket regarding my M3000 server since I
> don't have support contract (probably costs thousands?) but I filed a
> ticket anyway for the software that I bought but used the title&body of the
> ticket and explained my M3000 fault code situation. Well the ticket was
> promptly left aside due to being an invalid request, however by this point
> I had enough information to call the support center and explain things
> further (you need SR # of your ticket to past automatic machine prompt to
> be able to reach a person once you call). Oracle gentlemen was super
> helpful and transferred the ticket to M3000/M4000/M5000 team within
> minutes.
>
> I have been told that they don't provide service passwords anymore and that
> there are 2 options, either dispatch a Field Engineer to solve the problem
> or engage in web conference. I went with latter and I'm going to have a
> conference call later today and I should be able to collaborate with him
> 1-on-1 and give SSH access to the XSCF from where he can do the magic. I
> will provide you guys with an update. I might unbrick my M3000, but I don't
> believe I will be able to help any of you, if they provided service
> passwords I could have just play along and make 5 more SRs with serial
> numbers of your M3000s and help you out.
>
> Kyle.
>
> On Sat, Jan 27, 2018 at 3:53 PM, Mark Kettenis <[hidden email]>
> wrote:
>
> > > I'm curious if updating the firmware after the fault has occurred
> > > will allow users to reset it? I'm sure I have some m3000 firmware
> > > somewhere...
> >
> > Unfortunately not.  You can't update the firmware while the machine is
> > in the faulted state.
> >
>

[Kyle K]

Would you be able to share the XCP firmware that you have for M3000? I was
unable to obtain updated firmware from Oracle,

I wanted to post an update to you guys, I had an Oracle tech today SSH into
my XSCF and clear the fault code. He took serial number off my server and
generated a service password and his end and then pasted it into the
console, so this means that every service password is different plus they
expire after 48 hours if I'm not mistaken. My M3000 is back up and running
again! If any of you is thinking of reviving your M3000, you might try
doing what I did. It is a bit of hassle but it might be worth it for some
of you. I myself have M3000, 32GB of RAM, the XVR3000 graphics card, 2x SAS
HDDs, Sun type6 keyboard, Sun mouse, and top of the line CRT monitor from
2000s. I can basically use it as desktop, running Solaris with X11. I have
replaced the 2 super loud intake fans with typical 80mm desktop fans and
built a small circuit with 555 IC that generates just the PWM signal to
fool the server that fans are running at high speed when indeed they're
running quiet. Also did same for PSU, the 555 circuit fitted inside of it
nicely.

I still haven't figured whether Linux supports SPARC64 VII CPU? Anyone
knows? I have asked about and posted logs here:
https://oss.oracle.com/pipermail/linux-sparc-users/2017-October/000027.html

Kyle.

On Sat, Jan 27, 2018 at 3:41 PM, <[hidden email]> wrote:

> I'm curious if updating the firmware after the fault has occurred will
> allow users to reset it? I'm sure I have some m3000 firmware somewhere...
>

[Meelis Roos]

> I still haven't figured whether Linux supports SPARC64 VII CPU? Anyone
> knows? I have asked about and posted logs here:
> https://oss.oracle.com/pipermail/linux-sparc-users/2017-October/000027.html

No.

To the best of my knowledge, sparclinux does not support any Fujitsu
SPARC64 CPU-s.

I have a couple of PrimePowers with SPARC64 IV and V CPUs and I tried to
get a master student to add support for SPARC64 V and one corresponding
PCI root complex, but we have not done anything yet and may never get to
it as the hardware ages and interest lessens.

If someone wants to do it. I still have the PrimePower 450 hooped up to
power, net and scf in DMZ (remote access is possible).

Does VII also have specs of CPU + chipset available?

--
Meelis Roos ([hidden email])