Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Rolf Sommerhalder
Hello list

While the Go 1.4.1 package works fine on OpenBSD 5.7 amd64 (Lenovo
X230 laptop), it fails with i386 (PCengines ALIX with Geode
processor), see output below.

I have put my user into the staff login class, and increased ulimits,
as hinted in
 https://github.com/golang/go/wiki/OpenBSD
and
 http://homing-on-code.blogspot.ch/2015/03/go-development-on-openbsd.html

Googling reveals some PRs regarding run-time error at Go load-time due
to missing instructions on some i386 processors:
  https://code.google.com/p/go/issues/detail?id=4798
  https://github.com/golang/go/issues/4798

Fixes #4798, 14 Feb 2013
cmd/8c: disable use of prefetch with GO386=387
 https://github.com/golang/go/commit/594360cb1b31a99a349ba03294f5459aff0bc33d

Can anyone confirm that this package should indeed work on i386 as
well, or if there is some more knobs to turn (dis-/enable options in
the BIOS or kernel, etc) that I have missed?

Thanks,
Rolf


$ dmesg
OpenBSD 5.7 (GENERIC) #738: Sun Mar  8 10:59:31 MDT 2015
    [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC
RTC BIOS diagnostic error 80<clock_battery>
cpu0: Geode(TM) Integrated Processor by AMD PCS ("AuthenticAMD"
586-class) 499 MHz
cpu0: FPU,DE,PSE,TSC,MSR,CX8,SEP,PGE,CMOV,CFLUSH,MMX,MMXX,3DNOW2,3DNOW
real mem  = 267931648 (255MB)
avail mem = 251179008 (239MB)
...


$ pkg_info | grep go
                                                go-1.4.1            Go
programming language


$ go version
SIGILL: illegal instruction
PC=0x80a58bf

goroutine 1 [running, locked to thread]:
runtime.mapassign1(0x82f5f80, 0x18738020, 0x1872ff60, 0x1872ff5c)
        /usr/local/go/src/runtime/hashmap.go:455 +0x23f fp=0x1872ff4c
sp=0x1872fef0
unicode.init()
        /usr/local/go/src/unicode/tables.go:52 +0xaf fp=0x1872ff6c sp=0x1872ff4c
go/parser.init()
        /usr/local/go/src/go/parser/parser.go:2462 +0x3f fp=0x1872ff78
sp=0x1872ff6c
main.init()
        /usr/local/go/src/cmd/go/zdefaultcc.go:6 +0x3f fp=0x1872ffcc
sp=0x1872ff78
runtime.main()
        /usr/local/go/src/runtime/proc.go:58 +0xb7 fp=0x1872fff0 sp=0x1872ffcc
runtime.goexit()
        /usr/local/go/src/runtime/asm_386.s:2287 +0x1 fp=0x1872fff4
sp=0x1872fff0

goroutine 2 [runnable]:
runtime.forcegchelper()
        /usr/local/go/src/runtime/proc.go:90
runtime.goexit()
        /usr/local/go/src/runtime/asm_386.s:2287 +0x1

goroutine 3 [runnable]:
runtime.bgsweep()
        /usr/local/go/src/runtime/mgc0.go:82
runtime.goexit()
        /usr/local/go/src/runtime/asm_386.s:2287 +0x1

eax     0x0
ebx     0x0
ecx     0x1870c070
edx     0x18738020
edi     0x1b
esi     0x1870c077
ebp     0x0
esp     0x1872fef0
eip     0x80a58bf
eflags  0x10246
cs      0x2b
fs      0x5b
gs      0x63

Reply | Threaded
Open this post in threaded view
|

Re: Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Stuart Henderson-6
On 2015/08/05 13:41, Rolf Sommerhalder wrote:
> While the Go 1.4.1 package works fine on OpenBSD 5.7 amd64 (Lenovo
> X230 laptop), it fails with i386 (PCengines ALIX with Geode
> processor), see output below.

Go has cpuid autodetection at runtime which seems to be correct to
me, but just in case there's a problem with that, can you try setting
GO386=387 in your environment to disable the cpuid autodetection and see
if it still does the same?

> I have put my user into the staff login class, and increased ulimits,

These won't affect "illegal instruction".

> Googling reveals some PRs regarding run-time error at Go load-time due
> to missing instructions on some i386 processors:
>   https://code.google.com/p/go/issues/detail?id=4798
>   https://github.com/golang/go/issues/4798
>
> Fixes #4798, 14 Feb 2013
> cmd/8c: disable use of prefetch with GO386=387
>  https://github.com/golang/go/commit/594360cb1b31a99a349ba03294f5459aff0bc33d
>
> Can anyone confirm that this package should indeed work on i386 as
> well, or if there is some more knobs to turn (dis-/enable options in
> the BIOS or kernel, etc) that I have missed?

I occasionally run the regression test suite on i386 and amd64 and those
usually come up OK, however I wouldn't normally do that on a non-SSE2
machine.

Reply | Threaded
Open this post in threaded view
|

Re: Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Rolf Sommerhalder
Stuart, many thanks for your reply, assertions and suggestions!

On Wed, Aug 5, 2015 at 4:04 PM, Stuart Henderson <[hidden email]> wrote:
> ... can you try setting
> GO386=387 in your environment to disable the cpuid autodetection and see
> if it still does the same?

The problem persists even after adding this environment variable.
However, from [1, 2, 3] I understand, that I need to rebuild the
entire tool-chain, not only re-run the binary from the existing
package.
Thus, I will now re-build the entire package with this option, re-try,
and report the outcome...

> I occasionally run the regression test suite on i386 and amd64 and those
> usually come up OK, however I wouldn't normally do that on a non-SSE2
> machine.

Indeed, it works fine on another i386 machine that has SSE2/3 :-)
$ dmesg
OpenBSD 5.7 (GENERIC) #738: Sun Mar  8 10:59:31 MDT 2015
    [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC
RTC BIOS diagnostic error 80<clock_battery>
cpu0: Intel(R) Core(TM)2 Solo CPU U3500 @ 1.40GHz ("GenuineIntel"
686-class) 1.40 GHz
cpu0: FPU,V86,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,TM,PBE,NXE,LONG,SSE3,DTES64,MWAIT,DS-CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,XSAVE,LAHF,PERF
real mem  = 3147714560 (3001MB)
avail mem = 3083931648 (2941MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: date 08/10/10, SMBIOS rev. 2.6 @ 0xe8180 (28 entries)
bios0: vendor Acer version "V1.28" date 08/10/2010
bios0: Acer Aspire 3810T
...


$ go env
GOARCH="386"
GOBIN=""
GOCHAR="8"
GOEXE=""
GOHOSTARCH="386"
GOHOSTOS="openbsd"
GOOS="openbsd"
GOPATH="/home/rs/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/openbsd_386"
CC="cc"
GOGCCFLAGS="-fPIC -m32 -pthread -fmessage-length=0"
CXX="c++"
CGO_ENABLED="1"


[1]  go1.1beta2 app - illegal instruction on LX800 processor Alix Board
 https://groups.google.com/forum/#!msg/golang-nuts/v4MYW6sDrRo/nLP8YpiObLUJ

[2] code.google.com go 386 binaries now requiring SSE2?
 https://groups.google.com/forum/#!topic/golang-nuts/NsROxh3Dc6M

[3] 386 woes
 https://groups.google.com/forum/#!msg/golang-dev/oPOvRr7JU2Y/DAn2jwS38E8J

Reply | Threaded
Open this post in threaded view
|

Re: Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Rolf Sommerhalder
On Thu, Aug 6, 2015 at 10:49 AM, Rolf Sommerhalder > Thus, I will now
re-build the entire package with this option, re-try,
> and report the outcome...

Being ecouraged by [1] too, I re-build go-1.4.1 package with an
additional line in its Makefile:
.elif ${MACHINE_ARCH} == "i386"
GOARCH =                386
GO386 =                 387
EXEPREFIX =             8

Unfortunately, the resulting package still aborts with SIGILL on ALIX
with AMD Geode LX.

Thus, I will dig into the package build in oder to verify that the
GO386 parameter is indeed taking effect throughout the build
process...

[1] Golang on the Geode processor
 http://blog.nella.org/golang-on-the-geode-processor/

Reply | Threaded
Open this post in threaded view
|

Re: Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Stuart Henderson-6
On 2015/08/06 11:54, Rolf Sommerhalder wrote:

> On Thu, Aug 6, 2015 at 10:49 AM, Rolf Sommerhalder > Thus, I will now
> re-build the entire package with this option, re-try,
> > and report the outcome...
>
> Being ecouraged by [1] too, I re-build go-1.4.1 package with an
> additional line in its Makefile:
> .elif ${MACHINE_ARCH} == "i386"
> GOARCH =                386
> GO386 =                 387
> EXEPREFIX =             8
>
> Unfortunately, the resulting package still aborts with SIGILL on ALIX
> with AMD Geode LX.
>
> Thus, I will dig into the package build in oder to verify that the
> GO386 parameter is indeed taking effect throughout the build
> process...
>
> [1] Golang on the Geode processor
>  http://blog.nella.org/golang-on-the-geode-processor/
>

That's a makefile variable only, not passed to the build. Try this?

Index: Makefile
===================================================================
RCS file: /cvs/ports/lang/go/Makefile,v
retrieving revision 1.25
diff -u -p -r1.25 Makefile
--- Makefile 6 May 2015 08:28:26 -0000 1.25
+++ Makefile 6 Aug 2015 10:06:49 -0000
@@ -44,6 +44,8 @@ GOARCH = 386
 EXEPREFIX = 8
 PKG_ARGS += -Damd64=0
 PKG_ARGS += -Di386=1
+# avoid SSE
+MAKE_ENV += GO386=387
 .else
 PKG_ARGS += -Damd64=0 -Di386=0
 .endif

Reply | Threaded
Open this post in threaded view
|

Re: Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Rolf Sommerhalder
On Thu, Aug 6, 2015 at 12:07 PM, Stuart Henderson <[hidden email]> wrote:
> That's a makefile variable only, not passed to the build. Try this?

Thanks for your diff. Unfortunately, it still does not solve the problem.

Is there any way I can check that the environment variable is indeed
applied to all steps while ("cross-")building the package on my Acer
laptop which has SSE2/SSE3 for my ALIX which has only MMX?

Reply | Threaded
Open this post in threaded view
|

Re: Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Rolf Sommerhalder
Despite the make_env, the go binary still has a ton of SSE2 code using
XMM registers:
  objdump -d /usr/obj/ports/go-1.4.1/fake-i386/usr/local/bin/go | grep xmm
 808a559:       f2 0f 10 05 e8 3e 45    movsd  0x8453ee8,%xmm0
 808a561:       f2 0f 11 44 24 38       movsd  %xmm0,0x38(%esp)
 808a61a:       f2 0f 10 44 24 60       movsd  0x60(%esp),%xmm0
 808a620:       f2 0f 11 44 24 60       movsd  %xmm0,0x60(%esp)
 808a62e:       f2 0f 10 44 24 58       movsd  0x58(%esp),%xmm0
 808a634:       f2 0f 11 44 24 58       movsd  %xmm0,0x58(%esp)
 808a63a:       f2 0f 10 44 24 58       movsd  0x58(%esp),%xmm0
 808a640:       f2 0f 59 05 00 3f 45    mulsd  0x8453f00,%xmm0
 808a648:       f2 0f 11 44 24 58       movsd  %xmm0,0x58(%esp)
 808a64e:       f2 0f 10 44 24 58       movsd  0x58(%esp),%xmm0
 808a654:       f2 0f 10 4c 24 60       movsd  0x60(%esp),%xmm1
 808a65a:       f2 0f 58 c8             addsd  %xmm0,%xmm1
 ...

I'll try to build go "manually" as per [1] now, just to compare its
binary output with the package build process, and see if GO386=387 has
an effect.

[1] https://golang.org/doc/install/source

Reply | Threaded
Open this post in threaded view
|

Re: Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Rolf Sommerhalder
> I'll try to build go "manually" as per [1] now, just to compare its
> binary output with the package build process, and see if GO386=387 has
> an effect.
>
> [1] https://golang.org/doc/install/source

The resulting binary of this manual build still has some SSE2 code
with XMM registers, though much less than with the package build with
'MAKE_ENV += GO386=387' :

git clone https://go.googlesource.com/go
cd go
git checkout go1.4.1
cd src
GO386=387 GOARCH=386 ./make.bash
objdump -d /home/rs/goFromSource/go/bin/go | grep xmm

So 'GO386=387' appears to have an impact in the manual build, and most
checks still pass ok.

The good news is, that simply scp /home/rs/goFromSource/go/bin/go from
the i386 build host with SSE2/3 over to the ALIX, the go binary works
without SIGILL :-)

Will need to figure out what I still miss in the package build (I had
done 'make clean=all' before 'make package') ...

Reply | Threaded
Open this post in threaded view
|

Re: Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Stuart Henderson-6
On 2015/08/06 13:58, Rolf Sommerhalder wrote:
> > I'll try to build go "manually" as per [1] now, just to compare its
> > binary output with the package build process, and see if GO386=387 has
> > an effect.
> >
> > [1] https://golang.org/doc/install/source
>
> The resulting binary of this manual build still has some SSE2 code
> with XMM registers, though much less than with the package build with
> 'MAKE_ENV += GO386=387' :

I would expect some, as there's definitely code to do runtime detection.

> git clone https://go.googlesource.com/go
> cd go
> git checkout go1.4.1
> cd src
> GO386=387 GOARCH=386 ./make.bash
> objdump -d /home/rs/goFromSource/go/bin/go | grep xmm
>
> So 'GO386=387' appears to have an impact in the manual build, and most
> checks still pass ok.
>
> The good news is, that simply scp /home/rs/goFromSource/go/bin/go from
> the i386 build host with SSE2/3 over to the ALIX, the go binary works
> without SIGILL :-)
>
> Will need to figure out what I still miss in the package build (I had
> done 'make clean=all' before 'make package') ...

I'll see if I can figure something out.

Reply | Threaded
Open this post in threaded view
|

Re: Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Rolf Sommerhalder
On Thu, Aug 6, 2015 at 5:54 PM, Stuart Henderson <[hidden email]> wrote:
> I'll see if I can figure something out.

I could get go-1.4.1 and also a "backport" of -1.4.2 which got just
tagged for OpenBSD 5.8 to work with the patches to Makefile below.
Note that no MAKE_ENV is set. though.

> I would expect some, as there's definitely code to do runtime detection.

Indeed, the go binary still includes some SSE instructions that uses
XMM registers, as you indicated probably for detection at run-time.

Today, I want to do more testing using the gopacket library and its
new bsdbpf extension on the PC Engines ALIX, and on APU too.

Regarding the Go port, I am unsure what would be the best way to
proceed, e.g. if a flavor with GO386=387 for i386 without SSE would be
justified for example.


diff for Makefile go-1.4.1 :

@@ -64,7 +67,8 @@
                CXX="${CXX}" \
                GOROOT=${WRKDIST} \
                GOBIN=${WRKDIST}/bin \
-               GOROOT_FINAL=${GOROOT} ./make.bash --no-banner
+               GOROOT_FINAL=${GOROOT} \
+               GO386=387 ./make.bash --no-banner

 do-test:
        @cd ${WRKSRC} && \


diff for Makefile go-1.4.2 :

@@ -1,6 +1,7 @@
 # $OpenBSD: Makefile,v 1.25 2015/05/06 08:28:26 jasper Exp $

-ONLY_FOR_ARCHS =       ${GO_ARCHS}
+#ONLY_FOR_ARCHS =      ${GO_ARCHS}
+ONLY_FOR_ARCHS =        amd64 i386

 COMMENT =              Go programming language

@@ -64,7 +67,8 @@
                CXX="${CXX}" \
                GOROOT=${WRKDIST} \
                GOBIN=${WRKDIST}/bin \
-               GOROOT_FINAL=${GOROOT} ./make.bash --no-banner
+               GOROOT_FINAL=${GOROOT} \
+               GO386=387 ./make.bash --no-banner

 do-test:
        @cd ${WRKSRC} && \

Reply | Threaded
Open this post in threaded view
|

Re: Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Stuart Henderson
On 2015/08/07 08:38, Rolf Sommerhalder wrote:
> On Thu, Aug 6, 2015 at 5:54 PM, Stuart Henderson <[hidden email]> wrote:
> > I'll see if I can figure something out.
>
> I could get go-1.4.1 and also a "backport" of -1.4.2 which got just
> tagged for OpenBSD 5.8 to work with the patches to Makefile below.
> Note that no MAKE_ENV is set. though.

Ah good, that method looks reasonable.

> > I would expect some, as there's definitely code to do runtime detection.
>
> Indeed, the go binary still includes some SSE instructions that uses
> XMM registers, as you indicated probably for detection at run-time.
>
> Today, I want to do more testing using the gopacket library and its
> new bsdbpf extension on the PC Engines ALIX, and on APU too.
>
> Regarding the Go port, I am unsure what would be the best way to
> proceed, e.g. if a flavor with GO386=387 for i386 without SSE would be
> justified for example.

I don't think a flavour is warranted, i386 packages are meant to run
on all supported CPUs so if the runtime detection is insufficient
we should just always set this variable.

Reply | Threaded
Open this post in threaded view
|

Re: Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Ted Unangst-6
Stuart Henderson wrote:
>
> I don't think a flavour is warranted, i386 packages are meant to run
> on all supported CPUs so if the runtime detection is insufficient
> we should just always set this variable.

talk to upstream? this sounds like a bug if the runtime detection is
insufficient. unconditionally disabling sse would be a pessimization for most
users.

Reply | Threaded
Open this post in threaded view
|

Re: Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Matthew Dempsky-3
I seem to recall a bug where some CPU models don't properly set
registers after a CPUID instruction (e.g.,
http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/arch/i386/i386/locore.s#rev1.150).
Is it possible this CPU is one of those?  Maybe Go needs similar
workarounds?

A way to test that hypothesis would be to add

    MOVL $0, BX
    MOVL $0, CX
    MOVL $0, DX

before each of these CPUID instructions:

    src/cmd/dist/cpuid_386.s:9: CPUID
    src/runtime/asm_386.s:30: CPUID
    src/runtime/asm_386.s:47: CPUID

On Fri, Aug 7, 2015 at 3:22 PM, Ted Unangst <[hidden email]> wrote:

> Stuart Henderson wrote:
>>
>> I don't think a flavour is warranted, i386 packages are meant to run
>> on all supported CPUs so if the runtime detection is insufficient
>> we should just always set this variable.
>
> talk to upstream? this sounds like a bug if the runtime detection is
> insufficient. unconditionally disabling sse would be a pessimization for most
> users.
>

Reply | Threaded
Open this post in threaded view
|

Re: Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Rolf Sommerhalder
In reply to this post by Ted Unangst-6
On Sat, Aug 8, 2015 at 12:22 AM, Ted Unangst <[hidden email]> wrote:
Thanks everyone to your follow-ups.

> talk to upstream? this sounds like a bug if the runtime detection is
> insufficient.

From what I gathered about Go so far, upstream had already fixed that
(see references in my previous posts). The problem might be more
subtle, as I am "cross-"building the Go package on a SSE machine, but
then install and run it on a non-SSE machine. At build time, the Go
system will correctly detect my build host's capabilities, which
results in Go system binaries that include SSE instructions. However,
these system binaries fail with SIGILL on my non-SSE target as the
initial Go loader, linker and other Go system libraries (try to) use
SSE instructions at startup time of the Go system, before it actually
detects the less capable CPU of my run-time target host.

Thus my hypothesis is that the Go package would run fine if I would
build it directly on the non-SSE target. But currently, I have no
build environment on a ALIX host. Still, I will dig a bit further into
the implementation of the Go system while learning it, and while
experimenting with gopacket and its sub-libraries such as the recent
bsdbpf.

> unconditionally disabling sse would be a pessimization for most
> users.

As I share your concern, I thought that a separate flavour might be a
solution. Another much more pragmatic and lighther solutions would be
to just add a short note (in the Makefile or package info?) that shows
how to rebuild the package for those who want to run Go on non-SSE CPU
like Geode LX on ALIX.

Reply | Threaded
Open this post in threaded view
|

Re: Go-1.4.1 on OpenBSD 5.7 i386 fails with SIGILL: illegal instruction

Stuart Henderson-6
In reply to this post by Ted Unangst-6
On 2015/08/07 18:22, Ted Unangst wrote:
> Stuart Henderson wrote:
> >
> > I don't think a flavour is warranted, i386 packages are meant to run
> > on all supported CPUs so if the runtime detection is insufficient
> > we should just always set this variable.
>
> talk to upstream? this sounds like a bug if the runtime detection is
> insufficient. unconditionally disabling sse would be a pessimization for most
> users.

Won't "most users" be running amd64 anyway?