statethreads crashes in ld on 6.4

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

statethreads crashes in ld on 6.4

Claus Assmann-4
statethreads (http://state-threads.sourceforge.net/) crashes on
OpenBSD 6.4/amd64 (release) with an error in ld (see below); it
works fine on previous OpenBSD versions.  Do I have to set some
"special" cc/ld options to make this work? Or are patches to
statehreads required (there doesn't seem to be a port for it,
otherwise I would try that)?

#0  0x00000c0b0980db08 in _dl_bind (object=0xc0a85cff400, index=)
   from /usr/libexec/ld.so
(gdb) where
#0  0x00000c0b0980db08 in _dl_bind (object=0xc0a85cff400, index=)
   from /usr/libexec/ld.so
#1  0x00000c0b098080dd in _dl_bind_start ()
    at /usr/src/libexec/ld.so/amd64/ldasm.S:108
#2  0x00000c08448058bc in st_accept (fd=0xc0a634981c0, addr=0xc0ae5a31dd0,
    addrlen=0xc0ae5a31dcc, timeout=18446744073709551615)
    at ../../mta/statethreads/io.c:258
#3  0x00000c0844802928 in handle_connections (arg=0x0)
    at ../../../mta/statethreads/examples/server.c:880
#4  0x00000c0844803e4a in st_thread_create (start=0, arg=)
    at ../../mta/statethreads/sched.c:329
#5  0x00000c0844801e38 in start_threads ()
    at ../../../mta/statethreads/examples/server.c:855
#6  0x00000c0844800b17 in main (argc=6, argv=0x7f7ffffd4ec8)
    at ../../../mta/statethreads/examples/server.c:266
(gdb) print *object
$2 = {obj_base = 13229648510976, load_name = 0xc0b0ee98d90 "./server",
  load_dyn = 0xc0844a0dbc8, next = 0xc0a4aacb800, prev = 0x0,
  load_base = 13229648510976, load_list = 0xc0a64e9c400, load_size = 2158592,
  Dyn = {info = 0xc0a85cff440, u = {null = 0, needed = 1, pltrelsz = 1872,
      pltgot = 0xc0844a0dd28, hash = 0xc0844908960, strtab = 0xc0844909650 "",
      symtab = 0xc0844908c90, rela = 0xc0844909940, relasz = 1152,
      relaent = 24, strsz = 748, syment = 24, init = 0, fini = 0,
      soname = 0x0, rpath = 0x0, symbolic = 0, rel = 0x0, relsz = 0,
      relent = 0, pltrel = 7, debug = 0, textrel = 0, jmprel = 13229649599936,
      bind_now = 0, init_array = 0x0, fini_array = 0x0, init_arraysz = 0,
      fini_arraysz = 0, runpath = 0x0, flags = 0, encoding = 0,
      preinit_array = 0x0, preinit_arraysz = 0}}, relacount = 45,
  relcount = 0, status = 5, phdrp = 0xc0844800040, phdrc = 10, obj_type = 2,
  obj_flags = 2, buckets = 0xc0844908968, nbuckets = 97,
  chains = 0xc0844908aec, nchains = 104, dynamic = 0x0, child_list = {
    tqh_first = 0xc0ab8651740, tqh_last = 0xc0ab8651740}, grpsym_list = {
    tqh_first = 0xc0aab280e40, tqh_last = 0xc0ad21a8bc0}, grpref_list = {
    tqh_first = 0x0, tqh_last = 0xc0a85cff5c8}, refcount = 0, opencount = 1,
  grprefcount = 0, load_object = 0xc0a85cff400, sod = {
    sod_name = 13240289961776, sod_library = 0, sod_reserved = 0,
    sod_major = 0, sod_minor = 0, sod_next = 0}, dev = 0, inode = 0,
  tls_fsize = 0, tls_msize = 0, tls_align = 0, tls_static_data = 0x0,
  tls_offset = 0, relro_addr = 13229650663352, relro_size = 2120,
  grpsym_gen = 0, rpath = 0x0, runpath = 0x0, traced = 0}


$ ldd server
server:
        Start            End              Type  Open Ref GrpRef Name
        0000004e8ca00000 0000004e8cc0f000 exe   1    0   0      server
        00000051672ec000 00000051675db000 rlib  0    1   0      /usr/lib/libc.so.92.5
        0000005183400000 0000005183400000 ld.so 0    1   0      /usr/libexec/ld.so

ld doesn't seem to be mentioned in errata64.html, so I guess there's
no relevant change for it.

The test program (as well as statethreds) is part of MeTA1 in case
someone can take a look at it.

Reply | Threaded
Open this post in threaded view
|

Re: statethreads crashes in ld on 6.4

Philip Guenther-2
On Sat, Dec 1, 2018 at 6:34 AM Claus Assmann <[hidden email]>
wrote:

> statethreads (http://state-threads.sourceforge.net/) crashes on
> OpenBSD 6.4/amd64 (release) with an error in ld (see below); it
> works fine on previous OpenBSD versions.  Do I have to set some
> "special" cc/ld options to make this work?


That'll depend on what the problem turns out to be, of course...


> Or are patches to
> statehreads required (there doesn't seem to be a port for it,
> otherwise I would try that)?
>

Not that I know of.



> #0  0x00000c0b0980db08 in _dl_bind (object=0xc0a85cff400, index=)
>    from /usr/libexec/ld.so
> (gdb)
>

Since ld.so is relinked on each boot, just an address doesn't really show
what died.  The disassembly up to that address would help.
More important is knowing what signal killed the process.  ktracing it and
seeing what the syscalls leading up to signal were (and what extra info was
in the signal) tells a lot.


Philip Guenther
Reply | Threaded
Open this post in threaded view
|

Re: statethreads crashes in ld on 6.4

Edgar Pettijohn III-2
In reply to this post by Claus Assmann-4
I downloaded state threads from sourceforge and had to make the following
change to get it to build. I didn't test further than just compiling though.
Not sure what you would need to change to get your `autotools' makefiles to
work.

--- ./st-1.9/Makefile Thu Oct  1 17:55:03 2009
+++ Makefile Sun Dec  2 21:35:22 2018
@@ -200,6 +200,7 @@
 SFLAGS      = -fPIC
 LDFLAGS     = -shared -soname=$(SONAME) -lc
 OTHER_FLAGS = -Wall
+LD    = gcc -rdynamic -Wl,-rpath
 ifeq ($(shell test -f /usr/include/sys/event.h && echo yes), yes)
 DEFINES     += -DMD_HAVE_KQUEUE
 endif

Reply | Threaded
Open this post in threaded view
|

Re: statethreads crashes in ld on 6.4

Edgar Pettijohn III-2
Sorry just saw it came with some examples. Testing with the `lookupdns' program
ended with a Bus error (core dumped). Here is gdb output:

Core was generated by `lookupdns'.
Program terminated with signal SIGBUS, Bus error.
#0  _longjmp () at /usr/src/lib/libc/arch/amd64/gen/_setjmp.S:99
99      1:      movq    %r11,0(%rsp)
(gdb) bt
#0  _longjmp () at /usr/src/lib/libc/arch/amd64/gen/_setjmp.S:99
Backtrace stopped: Cannot access memory at address 0xb044815db732800f

Reply | Threaded
Open this post in threaded view
|

Re: statethreads crashes in ld on 6.4

Philip Guenther-2
On Sun, Dec 2, 2018 at 7:51 PM Edgar Pettijohn <[hidden email]>
wrote:

> Sorry just saw it came with some examples. Testing with the `lookupdns'
> program
> ended with a Bus error (core dumped). Here is gdb output:
>
> Core was generated by `lookupdns'.
> Program terminated with signal SIGBUS, Bus error.
> #0  _longjmp () at /usr/src/lib/libc/arch/amd64/gen/_setjmp.S:99
> 99      1:      movq    %r11,0(%rsp)
> (gdb) bt
> #0  _longjmp () at /usr/src/lib/libc/arch/amd64/gen/_setjmp.S:99
> Backtrace stopped: Cannot access memory at address 0xb044815db732800f
>

Crashing on _longjmp() would suggest it's not happy with OpenBSD's
setjmp/longjmp XOR cookies, but those have been in for a while.  If
statethreads were working for Claus with 6.3 then he's hitting something
different.


Philip
Reply | Threaded
Open this post in threaded view
|

Re: statethreads crashes in ld on 6.4

Claus Assmann-4
In reply to this post by Claus Assmann-4
Thanks for the replies! Here's an update what I tried so far
-- I will follow the suggestions next.

About the
- signal: it's "segmentation violation".
- syscall: it's accept(2): st_accept invokes that function.

----------
I checked setjmp: no change from 6.3 to 6.4 (cvs diff, comparing
files, etc), and also looked at the current version (4.20) of nspr
(statethreads started out from nspr AFAIR): it has the same definitions
wrt stack layout and no "recent" changes.

----------
I found this which might be relevant:
  * RETGUARD is a replacement for the stack-protector which uses a
    per-function random cookie (located in the ELF .openbsd.randomdata
    section) to consistency check the return address on the stack.
but turning it off using -fno-ret-protector did not help.

----------
Next I tried static linking (because the crash happened in the
dynamic linker): that did not help either
$ ldd server
server:
        Start            End              Type  Open Ref GrpRef Name
        00000b09cff7b000 00000b09d01d9000 dlib  1    0   0      /home/ca/sm-9/obj.OpenBSD/statethreads/examples/server
$ ./server -l. -b localhost:1234 -a -i
[02/Dec/2018:08:51:23] INFO: process 0 (pid 97666): configuration loaded
[02/Dec/2018:08:51:23] INFO: process 0 (pid 97666): starting 8 threads on localhost:1234
Segmentation fault (core dumped)

neec.esmtp.org$ gdb server{,.core}
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-unknown-openbsd6.4"...
Core was generated by `server'.
Program terminated with signal 11, Segmentation fault.
#0  0x0000104bd8b1190a in ?? ()
(gdb) where
#0  0x0000104bd8b1190a in ?? ()
#1  0x0000104bd8b118be in ?? ()
#2  0x00007f7ffffbe5f8 in ?? ()
#3  0xda20e8d18db78ebb in ?? ()
#4  0x0000104e53b14d70 in ?? ()
#5  0x0000104bd8b0590b in ?? ()
#6  0x0000000000000000 in ?? ()
(gdb) quit

----------
I also tried whether this could be a problem:
  The Retpoline mitigation against Spectre Variant 2 has been enabled
  in clang(1) and in assembly files on amd64 and i386.
but using -mno-retpoline did not help either.

Reply | Threaded
Open this post in threaded view
|

Re: statethreads crashes in ld on 6.4

Claus Assmann-4
In reply to this post by Edgar Pettijohn III-2
On Sun, Dec 02, 2018, Edgar Pettijohn wrote:
> Sorry just saw it came with some examples. Testing with the `lookupdns' program
> ended with a Bus error (core dumped). Here is gdb output:

You might want to download MeTA1 and use its statethreads version,
I'm not sure all of my fixes made it back into the distribution.

Reply | Threaded
Open this post in threaded view
|

Re: statethreads crashes in ld on 6.4

Claus Assmann-4
In reply to this post by Philip Guenther-2
On Sun, Dec 02, 2018, Philip Guenther wrote:

> Since ld.so is relinked on each boot, just an address doesn't really show
> what died.  The disassembly up to that address would help.
> More important is knowing what signal killed the process.  ktracing it and
> seeing what the syscalls leading up to signal were (and what extra info was
> in the signal) tells a lot.


Here's the dissambler output and the ktrace output follows.
Unfortunately I don't know enough about this to figure out
what is wrong, hopefully someone else can (or tell me which
other information is still needed). TIA!

$ gdb server server.core
GNU gdb 6.3
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-unknown-openbsd6.4"...
Core was generated by `server'.
Program terminated with signal 11, Segmentation fault.
Loaded symbols for /home/ca/sm-9/obj.OpenBSD/statethreads/examples/server
Reading symbols from /usr/lib/libc.so.92.5...done.
Loaded symbols for /usr/lib/libc.so.92.5
Reading symbols from /usr/libexec/ld.so...done.
Loaded symbols for /usr/libexec/ld.so
#0  0x00000987baf0d488 in _dl_bind (object=0x987f117d800, index=Variable "index" is not available.
) from /usr/libexec/ld.so
(gdb) where
#0  0x00000987baf0d488 in _dl_bind (object=0x987f117d800, index=Variable "index" is not available.
) from /usr/libexec/ld.so
#1  0x00000987baf0681d in _dl_bind_start () at /usr/src/libexec/ld.so/amd64/ldasm.S:108
#2  0x00000985b8e06618 in st_accept (fd=0x9885b802800, addr=0x98840217db0, addrlen=0x98840217dac,
    timeout=18446744073709551615) at ../../mta/statethreads/io.c:258
#3  0x00000985b8e02928 in handle_connections (arg=0x0) at ../../../mta/statethreads/examples/server.c:880
#4  0x00000985b8e0449f in _st_thread_main () at ../../mta/statethreads/sched.c:329
#5  0x00000985b8e03f05 in st_thread_create (start=0x36a12ea863f81b25, arg=0x0, joinable=2440, stk_size=230505664)
    at ../../mta/statethreads/sched.c:593
#6  0x00000985b8e01e38 in start_threads () at ../../../mta/statethreads/examples/server.c:855
#7  0x00000985b8e00b17 in main (argc=5, argv=0x7f7fffffa468) at ../../../mta/statethreads/examples/server.c:266
Current language:  auto; currently minimal
(gdb) disassemble
Dump of assembler code for function _dl_bind:
0x00000987baf0d3c0 <_dl_bind+0>:        mov    2112049(%rip),%r11        # 0x987bb110df8 <__retguard_3683>
0x00000987baf0d3c7 <_dl_bind+7>:        xor    (%rsp),%r11
0x00000987baf0d3cb <_dl_bind+11>:       push   %rbp
0x00000987baf0d3cc <_dl_bind+12>:       mov    %rsp,%rbp
0x00000987baf0d3cf <_dl_bind+15>:       push   %r11
0x00000987baf0d3d1 <_dl_bind+17>:       push   %r15
0x00000987baf0d3d3 <_dl_bind+19>:       push   %r14
0x00000987baf0d3d5 <_dl_bind+21>:       push   %r13
0x00000987baf0d3d7 <_dl_bind+23>:       push   %r12
0x00000987baf0d3d9 <_dl_bind+25>:       push   %rbx
0x00000987baf0d3da <_dl_bind+26>:       sub    $0x30,%rsp
0x00000987baf0d3de <_dl_bind+30>:       mov    %rdi,%r15
0x00000987baf0d3e1 <_dl_bind+33>:       mov    2112032(%rip),%rax        # 0x987bb110e08 <__guard_local>
0x00000987baf0d3e8 <_dl_bind+40>:       mov    %rax,0xffffffffffffffc8(%rbp)
0x00000987baf0d3ec <_dl_bind+44>:       mov    2111981(%rip),%r14        # 0x987bb110de0 <pcookie>
0x00000987baf0d3f3 <_dl_bind+51>:       mov    0x70(%r15),%rax
0x00000987baf0d3f7 <_dl_bind+55>:       mov    0xf8(%r15),%r13
0x00000987baf0d3fe <_dl_bind+62>:       movslq %esi,%rcx
0x00000987baf0d401 <_dl_bind+65>:       lea    (%rcx,%rcx,2),%rbx
0x00000987baf0d405 <_dl_bind+69>:       mov    0xc(%r13,%rbx,8),%ecx
0x00000987baf0d40a <_dl_bind+74>:       lea    (%rcx,%rcx,2),%rdx
0x00000987baf0d40e <_dl_bind+78>:       lea    (%rax,%rdx,8),%rcx
0x00000987baf0d412 <_dl_bind+82>:       mov    (%rax,%rdx,8),%r12d
0x00000987baf0d416 <_dl_bind+86>:       add    0x68(%r15),%r12
0x00000987baf0d41a <_dl_bind+90>:       movq   $0x0,0xffffffffffffffc0(%rbp)
0x00000987baf0d422 <_dl_bind+98>:       lea    0xffffffffffffffc0(%rbp),%rsi
0x00000987baf0d426 <_dl_bind+102>:      lea    0xffffffffffffffb8(%rbp),%r9
0x00000987baf0d42a <_dl_bind+106>:      mov    $0x30,%edx
0x00000987baf0d42f <_dl_bind+111>:      mov    %r12,%rdi
0x00000987baf0d432 <_dl_bind+114>:      mov    %r15,%r8
0x00000987baf0d435 <_dl_bind+117>:      callq  0x987baf07e60 <_dl_find_symbol>
0x00000987baf0d43a <_dl_bind+122>:      mov    0xffffffffffffffc0(%rbp),%rcx
0x00000987baf0d43e <_dl_bind+126>:      test   %rcx,%rcx
0x00000987baf0d441 <_dl_bind+129>:      je     0x987baf0d4c7 <_dl_bind+263>
0x00000987baf0d447 <_dl_bind+135>:      add    0x8(%rcx),%rax
0x00000987baf0d44b <_dl_bind+139>:      add    0x10(%r13,%rbx,8),%rax
0x00000987baf0d450 <_dl_bind+144>:      mov    %rax,0xffffffffffffffb0(%rbp)
0x00000987baf0d454 <_dl_bind+148>:      mov    0xffffffffffffffb8(%rbp),%rdi
0x00000987baf0d458 <_dl_bind+152>:      cmpl   $0x0,0x268(%rdi)
0x00000987baf0d45f <_dl_bind+159>:      jne    0x987baf0d4b9 <_dl_bind+249>
0x00000987baf0d461 <_dl_bind+161>:      mov    0x0(%r13,%rbx,8),%rax
0x00000987baf0d466 <_dl_bind+166>:      add    (%r15),%rax
0x00000987baf0d469 <_dl_bind+169>:      mov    %rax,0xffffffffffffffa0(%rbp)
0x00000987baf0d46d <_dl_bind+173>:      movq   $0x8,0xffffffffffffffa8(%rbp)
0x00000987baf0d475 <_dl_bind+181>:      lea    0xffffffffffffffa0(%rbp),%rdi
0x00000987baf0d479 <_dl_bind+185>:      mov    $0x18,%esi
0x00000987baf0d47e <_dl_bind+190>:      mov    $0x56,%eax
0x00000987baf0d483 <_dl_bind+195>:      mov    %r14,%rdx
0x00000987baf0d486 <_dl_bind+198>:      syscall
0x00000987baf0d488 <_dl_bind+200>:      mov    0xffffffffffffffb0(%rbp),%rax
0x00000987baf0d48c <_dl_bind+204>:      mov    2111861(%rip),%rcx        # 0x987bb110e08 <__guard_local>
0x00000987baf0d493 <_dl_bind+211>:      cmp    0xffffffffffffffc8(%rbp),%rcx
0x00000987baf0d497 <_dl_bind+215>:      jne    0x987baf0d4d5 <_dl_bind+277>
0x00000987baf0d499 <_dl_bind+217>:      add    $0x30,%rsp
0x00000987baf0d49d <_dl_bind+221>:      pop    %rbx
0x00000987baf0d49e <_dl_bind+222>:      pop    %r12
0x00000987baf0d4a0 <_dl_bind+224>:      pop    %r13
0x00000987baf0d4a2 <_dl_bind+226>:      pop    %r14
0x00000987baf0d4a4 <_dl_bind+228>:      pop    %r15
0x00000987baf0d4a6 <_dl_bind+230>:      pop    %r11
0x00000987baf0d4a8 <_dl_bind+232>:      pop    %rbp
0x00000987baf0d4a9 <_dl_bind+233>:      xor    (%rsp),%r11
0x00000987baf0d4ad <_dl_bind+237>:      cmp    2111812(%rip),%r11        # 0x987bb110df8 <__retguard_3683>
0x00000987baf0d4b4 <_dl_bind+244>:      je     0x987baf0d4b8 <_dl_bind+248>
0x00000987baf0d4b6 <_dl_bind+246>:      int3  
0x00000987baf0d4b7 <_dl_bind+247>:      int3  
0x00000987baf0d4b8 <_dl_bind+248>:      retq  
0x00000987baf0d4b9 <_dl_bind+249>:      mov    %r12,%rsi
0x00000987baf0d4bc <_dl_bind+252>:      callq  0x987baf07270 <_dl_trace_plt>
0x00000987baf0d4c1 <_dl_bind+257>:      test   %eax,%eax
0x00000987baf0d4c3 <_dl_bind+259>:      jne    0x987baf0d488 <_dl_bind+200>
0x00000987baf0d4c5 <_dl_bind+261>:      jmp    0x987baf0d461 <_dl_bind+161>
0x00000987baf0d4c7 <_dl_bind+263>:      lea    1057182(%rip),%rdi        # 0x987bb00f66c <_dl_pledge+1054076>
0x00000987baf0d4ce <_dl_bind+270>:      xor    %eax,%eax
0x00000987baf0d4d0 <_dl_bind+272>:      callq  0x987baf03580 <_dl_die>
0x00000987baf0d4d5 <_dl_bind+277>:      lea    1057224(%rip),%rdi        # 0x987bb00f6a4 <_dl_pledge+1054132>
0x00000987baf0d4dc <_dl_bind+284>:      callq  0x987baf0d5e0 <__stack_smash_handler>
End of assembler dump.
(gdb) i all-registers
rax            0x1      1
rbx            0x4e     78
rcx            0x987baf0d488    10478561580168
rdx            0x10f9fba15af66a80       1223285444242205312
rsi            0x18     24
rdi            0x98840217c48    10480796138568
rbp            0x98840217ca8    0x98840217ca8
rsp            0x98840217c48    0x98840217c48
r8             0x985b8f0b600    10469938083328
r9             0x6799c74        108633204
r10            0x1b     27
r11            0x202    514
r12            0x985b8f0b600    10469938083328
r13            0x985b8f0bc90    10469938085008
r14            0x10f9fba15af66a80       1223285444242205312
r15            0x987f117d800    10479470106624
rip            0x987baf0d488    0x987baf0d488 <_dl_bind+200>
eflags         0x203    515
cs             0x2b     43
ss             0x23     35
ds             0x23     35
es             0x23     35
fs             0x23     35
gs             0x23     35
st0            0        (raw 0x00000000000000000000)
st1            0        (raw 0x00000000000000000000)
st2            0        (raw 0x00000000000000000000)
st3            0        (raw 0x00000000000000000000)
st4            0        (raw 0x00000000000000000000)
st5            0        (raw 0x00000000000000000000)
st6            0        (raw 0x00000000000000000000)
st7            0        (raw 0x00000000000000000000)
fctrl          0x37f    895
fstat          0x0      0
ftag           0xffff   65535
fiseg          0x0      0
fioff          0x0      0
foseg          0x0      0
fooff          0x0      0
fop            0x0      0
xmm0           {f = {0x0, 0x8275e100, 0x0, 0x0}}        {f = {-1.78805571e-16, -2.1062039e+09, 2.98031811e-32,
    6.83553471e-15}}
xmm1           {f = {0x0, 0x0, 0x0, 0x0}}       {f = {3.62845661e-35, 5.21596746e+35, -7.55546104e-15, -6.37945926e+19}}
xmm2           {f = {0x0, 0x0, 0x0, 0x0}}       {f = {0, 0, 0, 0}}
xmm3           {f = {0x0, 0x0, 0x0, 0x0}}       {f = {0, 0, 0, 0}}
xmm4           {f = {0x0, 0x0, 0x0, 0x0}}       {f = {0, 0, 0, 0}}
xmm5           {f = {0x0, 0x0, 0x0, 0x0}}       {f = {0, 0, 0, 0}}
xmm6           {f = {0x0, 0x0, 0x0, 0x0}}       {f = {0, 0, 0, 0}}
xmm7           {f = {0x0, 0x0, 0x0, 0x0}}       {f = {0, 0, 0, 0}}
xmm8           {f = {0x0, 0x0, 0x0, 0x0}}       {f = {0, 0, 0, 0}}
xmm9           {f = {0x0, 0x0, 0x0, 0x0}}       {f = {0, 0, 0, 0}}
xmm10          {f = {0x0, 0x0, 0x0, 0x0}}       {f = {0, 0, 0, 0}}
xmm11          {f = {0x0, 0x0, 0x0, 0x0}}       {f = {0, 0, 0, 0}}
xmm12          {f = {0x0, 0x0, 0x0, 0x0}}       {f = {0, 0, 0, 0}}
xmm13          {f = {0x0, 0x0, 0x0, 0x0}}       {f = {0, 0, 0, 0}}
xmm14          {f = {0x0, 0x0, 0x0, 0x0}}       {f = {0, 0, 0, 0}}
xmm15          {f = {0x0, 0x0, 0x0, 0x0}}       {f = {0, 0, 0, 0}}
mxcsr          0x1fa0   8096
(gdb) quit


[full output is hopefully attached which include the binary data
read from various files]

 65554 ktrace   RET   ktrace 0
 65554 ktrace   CALL  execve(0x7f7ffffe9aff,0x7f7ffffe9890,0x7f7ffffe98c0)
 65554 ktrace   NAMI  "./server"
 65554 ktrace   ARGS  
        [0] = "./server"
        [1] = "-b"
        [2] = "localhost:1234"
        [3] = "-a"
        [4] = "-i"
 65554 server   NAMI  "/usr/libexec/ld.so"
 65554 server   RET   execve 0
 65554 server   CALL  mprotect(0x13cd36e10ae8,0x518,0x1<PROT_READ>)
 65554 server   RET   mprotect 0
 65554 server   CALL  issetugid()
 65554 server   RET   issetugid 0
 65554 server   CALL  getentropy(0x7f7ffffd4f50,40)
 65554 server   RET   getentropy 0
 65554 server   CALL  getentropy(0x7f7ffffd4f50,40)
 65554 server   RET   getentropy 0
 65554 server   CALL  mmap(0,0x4000,0<PROT_NONE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21775259197440/0x13cdf296e000
 65554 server   CALL  mprotect(0x13cdf296f000,0x2000,0x3<PROT_READ|PROT_WRITE>)
 65554 server   RET   mprotect 0
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21772035788800/0x13cd32759000
 65554 server   CALL  mprotect(0x13cd36e13000,0x1000,0x1<PROT_READ>)
 65554 server   RET   mprotect 0
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21774128291840/0x13cdaf2ea000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21772793835520/0x13cd5fa47000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21772857724928/0x13cd63735000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773130874880/0x13cd73bb4000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21775204900864/0x13cdef5a6000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21774841597952/0x13cdd9b2d000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773520191488/0x13cd8aefc000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773081993216/0x13cd70d16000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21772176011264/0x13cd3ad13000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21771920187392/0x13cd2b91a000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773814706176/0x13cd9c7db000
 65554 server   CALL  open(0x13cd36d0e6a0,0x10000<O_RDONLY|O_CLOEXEC>)
 65554 server   NAMI  "/var/run/ld.so.hints"
 65554 server   RET   open 3
 65554 server   CALL  fstat(3,0x7f7ffffd4d28)
 65554 server   STRU  struct stat { dev=4, ino=155921, mode=-r--r--r-- , nlink=1, uid=0<"root">, gid=0<"wheel">, rdev=623688, atime=1543854086<"Dec  3 08:21:26 2018">.568456239, mtime=1543854071<"Dec  3 08:21:11 2018">.430119828, ctime=1543854071<"Dec  3 08:21:11 2018">.430119828, size=49194, blocks=100, blksize=16384, flags=0x0, gen=0x0 }
 65554 server   RET   fstat 0
 65554 server   CALL  mmap(0,0xc02a,0x1<PROT_READ>,0x2<MAP_PRIVATE>,3,0)
 65554 server   RET   mmap 21774911295488/0x13cdddda5000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21771557994496/0x13cd15fb0000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773360447488/0x13cd816a4000
 65554 server   CALL  close(3)
 65554 server   RET   close 0
 65554 server   CALL  open(0x13cddddab55e,0x10000<O_RDONLY|O_CLOEXEC>)
 65554 server   NAMI  "/usr/lib/libc.so.92.5"
 65554 server   RET   open 3
 65554 server   CALL  fstat(3,0x7f7ffffd3e80)
 65554 server   STRU  struct stat { dev=5, ino=77957, mode=-r--r--r-- , nlink=1, uid=0<"root">, gid=7<"bin">, rdev=317872, atime=1543854500<"Dec  3 08:28:20 2018">.389719893, mtime=1543854065<"Dec  3 08:21:05 2018">.936743077, ctime=1543854065<"Dec  3 08:21:05 2018">.967398929, size=3641068, blocks=7168, blksize=16384, flags=0x0, gen=0x0 }
 65554 server   RET   fstat 0
 65554 server   CALL  read(3,0x7f7ffffd3f00,0x1000)
 65554 server   GIO   fd 3 read 4096 bytes
       [[... see attachment ...]]
 65554 server   RET   read 4096/0x1000
 65554 server   CALL  mmap(0,0x2f0000,0<PROT_NONE>,0x2<MAP_PRIVATE>,3,0)
 65554 server   RET   mmap 21773823963136/0x13cd9d0af000
 65554 server   CALL  mmap(0x13cd9d0af000,0xa3000,0x5<PROT_READ|PROT_EXEC>,0x12<MAP_PRIVATE|MAP_FIXED>,3,0)
 65554 server   RET   mmap 21773823963136/0x13cd9d0af000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21771790139392/0x13cd23d14000
 65554 server   CALL  mmap(0x13cd9d251000,0x36000,0x1<PROT_READ>,0x12<MAP_PRIVATE|MAP_FIXED>,3,0xa2000)
 65554 server   RET   mmap 21773825675264/0x13cd9d251000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773143207936/0x13cd74777000
 65554 server   CALL  mmap(0x13cd9d387000,0x8000,0x3<PROT_READ|PROT_WRITE>,0x12<MAP_PRIVATE|MAP_FIXED>,3,0xd8000)
 65554 server   RET   mmap 21773826945024/0x13cd9d387000
 65554 server   CALL  mmap(0x13cd9d38f000,0x10000,0x3<PROT_READ|PROT_WRITE>,0x1012<MAP_PRIVATE|MAP_FIXED|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773826977792/0x13cd9d38f000
 65554 server   CALL  close(3)
 65554 server   RET   close 0
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21775081865216/0x13cde8050000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21774905176064/0x13cddd7cf000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21772561113088/0x13cd51c56000
 65554 server   CALL  mmap(0,0xa000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21775384383488/0x13cdfa0d1000
 65554 server   CALL  mprotect(0x13cd9d387000,0x6000,0x1<PROT_READ>)
 65554 server   RET   mprotect 0
 65554 server   CALL  munmap(0x13cdfa0d1000,0xa000)
 65554 server   RET   munmap 0
 65554 server   CALL  mprotect(0x13cb0ea0f000,0x1000,0x1<PROT_READ>)
 65554 server   RET   mprotect 0
 65554 server   CALL  getthrid()
 65554 server   RET   getthrid 356436/0x57054
 65554 server   CALL  __set_tcb(0x13cdaf2ea2c0)
 65554 server   RET   __set_tcb 0
 65554 server   CALL  kbind(0x7f7ffffd5078,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21774328913920/0x13cdbb23e000
 65554 server   CALL  mprotect(0x13cdbb23e000,0x1000,0x1<PROT_READ>)
 65554 server   RET   mprotect 0
 65554 server   CALL  kbind(0x7f7ffffd5068,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  mprotect(0x13cdbb23e000,0x1000,0x3<PROT_READ|PROT_WRITE>)
 65554 server   RET   mprotect 0
 65554 server   CALL  mprotect(0x13cdbb23e000,0x1000,0x1<PROT_READ>)
 65554 server   RET   mprotect 0
 65554 server   CALL  kbind(0x7f7ffffd4fd8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd4fd8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd4fa8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  readlink(0x13cd9d26da20,0x7f7ffffd4fb0,63)
 65554 server   NAMI  "/etc/malloc.conf"
 65554 server   RET   readlink -1 errno 2 No such file or directory
 65554 server   CALL  issetugid()
 65554 server   RET   issetugid 0
 65554 server   CALL  getentropy(0x7f7ffffd4f40,40)
 65554 server   RET   getentropy 0
 65554 server   CALL  mmap(0,0x450,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773261520896/0x13cd7b84c000
 65554 server   CALL  minherit(0x13cd7b84c000,0x450,MAP_INHERIT_ZERO)
 65554 server   RET   minherit 0
 65554 server   CALL  kbind(0x7f7ffffd4e68,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  mprotect(0x13cd9d395000,0x1000,0x3<PROT_READ|PROT_WRITE>)
 65554 server   RET   mprotect 0
 65554 server   CALL  mmap(0,0x4000,0<PROT_NONE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21775003488256/0x13cde3591000
 65554 server   CALL  mprotect(0x13cde3592000,0x2000,0x3<PROT_READ|PROT_WRITE>)
 65554 server   RET   mprotect 0
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773550137344/0x13cd8cb8b000
 65554 server   CALL  mprotect(0x13cd9d395000,0x1000,0x1<PROT_READ>)
 65554 server   RET   mprotect 0
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773304700928/0x13cd7e17a000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773934125056/0x13cda39be000
 65554 server   CALL  kbind(0x7f7ffffd4fd8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  getuid()
 65554 server   RET   getuid 1503<"ca">
 65554 server   CALL  kbind(0x7f7ffffd4fb8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  sysctl(6.25<hw.ncpuonline>,0x7f7ffffd4ef0,0x7f7ffffd5048,0,0)
 65554 server   RET   sysctl 0
 65554 server   CALL  kbind(0x7f7ffffd5038,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd4f98,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd4f98,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  sigaction(SIGPIPE,0x7f7ffffd5090,0)
 65554 server   STRU  struct sigaction { handler=SIG_IGN, mask=0<>, flags=0<> }
 65554 server   RET   sigaction 0
 65554 server   CALL  kbind(0x7f7ffffd4f98,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  getrlimit(RLIMIT_NOFILE,0x7f7ffffd5080)
 65554 server   STRU  struct rlimit { cur=512, max=1024 }
 65554 server   RET   getrlimit 0
 65554 server   CALL  kbind(0x7f7ffffd4f98,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  setrlimit(RLIMIT_NOFILE,0x7f7ffffd5080)
 65554 server   STRU  struct rlimit { cur=1024, max=1024 }
 65554 server   RET   setrlimit 0
 65554 server   CALL  kbind(0x7f7ffffd4fe8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd4fc8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  mmap(0,0x4000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21771835613184/0x13cd26872000
 65554 server   CALL  kbind(0x7f7ffffd4fe8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd4f98,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  gettimeofday(0x7f7ffffd5090,0)
 65554 server   STRU  struct timeval { 1543854509<"Dec  3 08:28:29 2018">.403448 }
 65554 server   RET   gettimeofday 0
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21774308524032/0x13cdb9ecc000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21772308422656/0x13cd42b5a000
 65554 server   CALL  kbind(0x7f7ffffd4ee8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21772607021056/0x13cd5481e000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21771974471680/0x13cd2ecdf000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21774738231296/0x13cdd3899000
 65554 server   CALL  kbind(0x7f7ffffd4fd8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd4fd8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd4fd8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  socket(AF_INET,0x1<SOCK_STREAM>,0)
 65554 server   RET   socket 3
 65554 server   CALL  kbind(0x7f7ffffd4fd8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  setsockopt(3,SOL_SOCKET,SO_REUSEADDR,0x7f7ffffd50dc,4)
 65554 server   RET   setsockopt 0
 65554 server   CALL  kbind(0x7f7ffffd4fd8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd4fd8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd4f58,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21771807305728/0x13cd24d73000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773593063424/0x13cd8f47b000
 65554 server   CALL  getpid()
 65554 server   RET   getpid 65554/0x10012
 65554 server   CALL  clock_gettime(CLOCK_MONOTONIC,0x7f7ffffd3f50)
 65554 server   STRU  struct timespec { 454.083406774 }
 65554 server   RET   clock_gettime 0
 65554 server   CALL  stat(0x13cd9d27098a,0x7f7ffffd3f60)
 65554 server   NAMI  "/etc/resolv.conf"
 65554 server   STRU  struct stat { dev=0, ino=1585721, mode=-rw-r--r-- , nlink=1, uid=0<"root">, gid=0<"wheel">, rdev=6328714, atime=1543854358<"Dec  3 08:25:58 2018">.665661463, mtime=1543008951<"Nov 23 13:35:51 2018">.259678741, ctime=1543009112<"Nov 23 13:38:32 2018">.486958401, size=38, blocks=4, blksize=16384, flags=0x0, gen=0x0 }
 65554 server   RET   stat 0
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21771602399232/0x13cd18a09000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773329338368/0x13cd7f8f9000
 65554 server   CALL  mprotect(0x13cdbb23e000,0x1000,0x3<PROT_READ|PROT_WRITE>)
 65554 server   RET   mprotect 0
 65554 server   CALL  mprotect(0x13cdbb23e000,0x1000,0x1<PROT_READ>)
 65554 server   RET   mprotect 0
 65554 server   CALL  open(0x13cd9d27098a,0x10000<O_RDONLY|O_CLOEXEC>)
 65554 server   NAMI  "/etc/resolv.conf"
 65554 server   RET   open 4
 65554 server   CALL  fstat(4,0x7f7ffffd3e08)
 65554 server   STRU  struct stat { dev=0, ino=1585721, mode=-rw-r--r-- , nlink=1, uid=0<"root">, gid=0<"wheel">, rdev=6328714, atime=1543854358<"Dec  3 08:25:58 2018">.665661463, mtime=1543008951<"Nov 23 13:35:51 2018">.259678741, ctime=1543009112<"Nov 23 13:38:32 2018">.486958401, size=38, blocks=4, blksize=16384, flags=0x0, gen=0x0 }
 65554 server   RET   fstat 0
 65554 server   CALL  mmap(0,0x4000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773363265536/0x13cd81954000
 65554 server   CALL  read(4,0x13cd81954000,0x4000)
 65554 server   GIO   fd 4 read 38 bytes
       "lookup file bind
        nameserver 10.1.1.12
       "
 65554 server   RET   read 38/0x26
 65554 server   CALL  read(4,0x13cd81954000,0x4000)
 65554 server   RET   read 0
 65554 server   CALL  close(4)
 65554 server   RET   close 0
 65554 server   CALL  kbind(0x7f7ffffd3e48,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  sysctl(1.10<kern.hostname>,0x7f7ffffd3910,0x7f7ffffd38e0,0,0)
 65554 server   RET   sysctl 0
 65554 server   CALL  issetugid()
 65554 server   RET   issetugid 0
 65554 server   CALL  getpid()
 65554 server   RET   getpid 65554/0x10012
 65554 server   CALL  clock_gettime(CLOCK_MONOTONIC,0x7f7ffffd3f50)
 65554 server   STRU  struct timespec { 454.083712129 }
 65554 server   RET   clock_gettime 0
 65554 server   CALL  getpid()
 65554 server   RET   getpid 65554/0x10012
 65554 server   CALL  clock_gettime(CLOCK_MONOTONIC,0x7f7ffffd3de0)
 65554 server   STRU  struct timespec { 454.083739228 }
 65554 server   RET   clock_gettime 0
 65554 server   CALL  getpid()
 65554 server   RET   getpid 65554/0x10012
 65554 server   CALL  clock_gettime(CLOCK_MONOTONIC,0x7f7ffffd3dd0)
 65554 server   STRU  struct timespec { 454.083764930 }
 65554 server   RET   clock_gettime 0
 65554 server   CALL  open(0x13cd9d26edb0,0x10000<O_RDONLY|O_CLOEXEC>)
 65554 server   NAMI  "/etc/hosts"
 65554 server   RET   open 4
 65554 server   CALL  fstat(4,0x7f7ffffd2fa8)
 65554 server   STRU  struct stat { dev=0, ino=1585415, mode=-rw-r--r-- , nlink=1, uid=0<"root">, gid=0<"wheel">, rdev=6328711, atime=1543854358<"Dec  3 08:25:58 2018">.665661463, mtime=1543009112<"Nov 23 13:38:32 2018">.476958251, ctime=1543009112<"Nov 23 13:38:32 2018">.476958251, size=65, blocks=4, blksize=16384, flags=0x0, gen=0x0 }
 65554 server   RET   fstat 0
 65554 server   CALL  mmap(0,0x4000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21771571834880/0x13cd16ce3000
 65554 server   CALL  read(4,0x13cd16ce3000,0x4000)
 65554 server   GIO   fd 4 read 65 bytes
       "127.0.0.1 localhost
        ::1 localhost
        10.1.1.53 neec.esmtp.org neec
       "
 65554 server   RET   read 65/0x41
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21772399865856/0x13cd4828f000
 65554 server   CALL  mmap(0,0x1000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773493972992/0x13cd895fb000
 65554 server   CALL  close(4)
 65554 server   RET   close 0
 65554 server   CALL  kbind(0x7f7ffffd4fd8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd4fd8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  bind(3,0x7f7ffffd50e0,16)
 65554 server   STRU  struct sockaddr { AF_INET, 127.0.0.1:1234 }
 65554 server   RET   bind 0
 65554 server   CALL  kbind(0x7f7ffffd4fd8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  listen(3,256)
 65554 server   RET   listen 0
 65554 server   CALL  kbind(0x7f7ffffd4f68,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  ioctl(3,FIONBIO,0x7f7ffffd506c)
 65554 server   RET   ioctl 0
 65554 server   CALL  kbind(0x7f7ffffd4fe8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  getpid()
 65554 server   RET   getpid 65554/0x10012
 65554 server   CALL  kbind(0x7f7ffffd5018,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  gettimeofday(0x7f7ffffd50b8,0)
 65554 server   STRU  struct timeval { 1543854509<"Dec  3 08:28:29 2018">.404533 }
 65554 server   RET   gettimeofday 0
 65554 server   CALL  gettimeofday(0x7f7ffffd50c0,0)
 65554 server   STRU  struct timeval { 1543854509<"Dec  3 08:28:29 2018">.404550 }
 65554 server   RET   gettimeofday 0
 65554 server   CALL  kbind(0x7f7ffffd4ff8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  pipe(0x7f7ffffd50e8)
 65554 server   STRU  fd[0] = 4, fd[1] = 5
 65554 server   RET   pipe 0
 65554 server   CALL  kbind(0x7f7ffffd4f88,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  fcntl(4,F_GETFL)
 65554 server   RET   fcntl 2
 65554 server   CALL  fcntl(4,F_SETFL,0x6<O_RDWR|O_NONBLOCK>)
 65554 server   RET   fcntl 0
 65554 server   CALL  fcntl(5,F_GETFL)
 65554 server   RET   fcntl 2
 65554 server   CALL  fcntl(5,F_SETFL,0x6<O_RDWR|O_NONBLOCK>)
 65554 server   RET   fcntl 0
 65554 server   CALL  sigaction(SIGTERM,0x7f7ffffd50a0,0)
 65554 server   STRU  struct sigaction { handler=0x13cb0e802600, mask=0<>, flags=0<> }
 65554 server   RET   sigaction 0
 65554 server   CALL  sigaction(SIGHUP,0x7f7ffffd50a0,0)
 65554 server   STRU  struct sigaction { handler=0x13cb0e802600, mask=0<>, flags=0<> }
 65554 server   RET   sigaction 0
 65554 server   CALL  sigaction(SIGUSR1,0x7f7ffffd50a0,0)
 65554 server   STRU  struct sigaction { handler=0x13cb0e802600, mask=0<>, flags=0<> }
 65554 server   RET   sigaction 0
 65554 server   CALL  kbind(0x7f7ffffd4ff8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd4ff8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  sigprocmask(SIG_UNBLOCK,0x20004001<SIGHUP|SIGTERM|SIGUSR1>)
 65554 server   RET   sigprocmask 0<>
 65554 server   CALL  kbind(0x7f7ffffd3da8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd3d38,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  mmap(0,0x5000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21772136759296/0x13cd387a4000
 65554 server   CALL  mmap(0,0xb000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21775349891072/0x13cdf7fec000
 65554 server   CALL  open(0x13cd9d26e08e,0<O_RDONLY>)
 65554 server   NAMI  "/etc/localtime"
 65554 server   RET   open 6
 65554 server   CALL  read(6,0x13cdf7fec000,0xa1e8)
 65554 server   GIO   fd 6 read 2819 bytes
       [[... see attachment ...]]
 65554 server   RET   read 2819/0xb03
 65554 server   CALL  close(6)
 65554 server   RET   close 0
 65554 server   CALL  mmap(0,0xb000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21774069030912/0x13cdaba66000
 65554 server   CALL  issetugid()
 65554 server   RET   issetugid 0
 65554 server   CALL  open(0x7f7ffffcec90,0<O_RDONLY>)
 65554 server   NAMI  "/usr/share/zoneinfo/posixrules"
 65554 server   RET   open 6
 65554 server   CALL  read(6,0x13cdaba66000,0xa1e8)
 65554 server   GIO   fd 6 read 2819 bytes
       [[... see attachment ...]]
        PST8PDT,M3.2.0,M11.1.0
       "
 65554 server   RET   read 2819/0xb03
 65554 server   CALL  close(6)
 65554 server   RET   close 0
 65554 server   CALL  kbind(0x7f7ffffd3d38,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd3da8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd3da8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd3da8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd3da8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  kbind(0x7f7ffffd3da8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  write(2,0x7f7ffffd3f10,0x49)
 65554 server   GIO   fd 2 wrote 73 bytes
       "[03/Dec/2018:08:28:29] INFO: process 0 (pid 65554): configuration loaded
       "
 65554 server   RET   write 73/0x49
 65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21772703744000/0x13cd5a45c000
 65554 server   CALL  write(2,0x7f7ffffd3f00,0x59)
 65554 server   GIO   fd 2 wrote 89 bytes
       "[03/Dec/2018:08:28:29] INFO: process 0 (pid 65554): starting 8 threads on localhost:1234
       "
 65554 server   RET   write 89/0x59
 65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21771804393472/0x13cd24aac000
 65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21771451404288/0x13cd0fa09000
 65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773345935360/0x13cd808cd000
 65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21774756491264/0x13cdd4a03000
 65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21774604423168/0x13cdcb8fd000
 65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773142749184/0x13cd74707000
 65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21773994246144/0x13cda7314000
 65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
 65554 server   RET   mmap 21774606540800/0x13cdcbb02000
 65554 server   CALL  kbind(0x7f7ffffd4fa8,24,0x8a4abe18ba78cb4a)
 65554 server   RET   kbind 0
 65554 server   CALL  read(4,0x7f7ffffd50ec,0x4)
 65554 server   RET   read -1 errno 35 Resource temporarily unavailable
 65554 server   CALL  kbind(0x13cd24abcc48,24,0x8a4abe18ba78cb4a)
 65554 server   PSIG  SIGSEGV SIG_DFL addr=0x0 trapno=0
 65554 server   NAMI  "server.core"

ktrace.txt.gz (11K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: statethreads crashes in ld on 6.4

Philip Guenther-2
On Mon, 3 Dec 2018, Claus Assmann wrote:
> Here's the dissambler output and the ktrace output follows.
> Unfortunately I don't know enough about this to figure out
> what is wrong, hopefully someone else can (or tell me which
> other information is still needed). TIA!

A close read of the ktrace output points to the problem:

...
>  65554 server   GIO   fd 2 wrote 89 bytes
>        "[03/Dec/2018:08:28:29] INFO: process 0 (pid 65554): starting 8 threads on localhost:1234
>        "

So it's just about to create its eight (userspace) threads...


>  65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
>  65554 server   RET   mmap 21771804393472/0x13cd24aac000
>  65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
>  65554 server   RET   mmap 21771451404288/0x13cd0fa09000
>  65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
>  65554 server   RET   mmap 21773345935360/0x13cd808cd000
>  65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
>  65554 server   RET   mmap 21774756491264/0x13cdd4a03000
>  65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
>  65554 server   RET   mmap 21774604423168/0x13cdcb8fd000
>  65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
>  65554 server   RET   mmap 21773142749184/0x13cd74707000
>  65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
>  65554 server   RET   mmap 21773994246144/0x13cda7314000
>  65554 server   CALL  mmap(0,0x12000,0x3<PROT_READ|PROT_WRITE>,0x1002<MAP_PRIVATE|MAP_ANON>,-1,0)
>  65554 server   RET   mmap 21774606540800/0x13cdcbb02000

Eight mmaps, presumably one per thread...


>  65554 server   CALL  kbind(0x7f7ffffd4fa8,24,0x8a4abe18ba78cb4a)
>  65554 server   RET   kbind 0

Okay, so this kbind() is by the original thread.  The first argument to
kbind() happens to be a buffer which is always on the current thread's
stack.  All is good here.

...
>  65554 server   CALL  kbind(0x13cd24abcc48,24,0x8a4abe18ba78cb4a)
>  65554 server   PSIG  SIGSEGV SIG_DFL addr=0x0 trapno=0
>  65554 server   NAMI  "server.core"

And now this kbind() call blows up: the address is not on the original
thread's stack but in one of those mmap()s...but those mmap()s were not
marked as stacks by including MAP_STACK.  To quote the "Security
improvements" section of https://www.openbsd.org/64.html

    * Implemented MAP_STACK option for mmap(2). At pagefaults and
      syscalls the kernel will check that the stack pointer points
      to MAP_STACK memory, which mitigates against attacks using
      stack pivots.


To confirm, if you check your dmesg(8) or /var/log/messages you should
find the kernel complaining something like
   syscall [server]65554/### sp 13cd24a## not inside 0x7f7f###-0x7f7f###


Philip Guenther

Reply | Threaded
Open this post in threaded view
|

Re: statethreads crashes in ld on 6.4

Claus Assmann-4
On Mon, Dec 03, 2018, Philip Guenther wrote:

[thanks for the analysis/explanation!]

> And now this kbind() call blows up: the address is not on the original
> thread's stack but in one of those mmap()s...but those mmap()s were not
> marked as stacks by including MAP_STACK.  To quote the "Security
> improvements" section of https://www.openbsd.org/64.html

>     * Implemented MAP_STACK option for mmap(2). At pagefaults and
>       syscalls the kernel will check that the stack pointer points
>       to MAP_STACK memory, which mitigates against attacks using
>       stack pivots.

Hmm, I read that and it seems I misunderstood it -- I will give
this a try.
However, here's the weird part: there's a compile time switch not
to use mmap(2) but malloc(2) and I selected that option in one of
my test because of that note: that version also crashed, hence I
was under the impression that MAP_STACK couldn't be the problem.


static char *_st_new_stk_segment(int size)
{
#ifdef MALLOC_STACK
  void *vaddr = malloc(size);
#else
  int mmap_flags = MAP_PRIVATE;
  void *vaddr;

  mmap_flags |= MAP_ANON;
  vaddr = mmap(NULL, size, PROT_READ | PROT_WRITE, mmap_flags, zero_fd, 0);
  if (vaddr == (void *)MAP_FAILED)
    return NULL;
#endif /* MALLOC_STACK */
  return (char *)vaddr;
}

Reply | Threaded
Open this post in threaded view
|

Re: statethreads crashes in ld on 6.4

Otto Moerbeek
On Tue, Dec 04, 2018 at 02:33:40AM -0800, Claus Assmann wrote:

> On Mon, Dec 03, 2018, Philip Guenther wrote:
>
> [thanks for the analysis/explanation!]
>
> > And now this kbind() call blows up: the address is not on the original
> > thread's stack but in one of those mmap()s...but those mmap()s were not
> > marked as stacks by including MAP_STACK.  To quote the "Security
> > improvements" section of https://www.openbsd.org/64.html
>
> >     * Implemented MAP_STACK option for mmap(2). At pagefaults and
> >       syscalls the kernel will check that the stack pointer points
> >       to MAP_STACK memory, which mitigates against attacks using
> >       stack pivots.
>
> Hmm, I read that and it seems I misunderstood it -- I will give
> this a try.
> However, here's the weird part: there's a compile time switch not
> to use mmap(2) but malloc(2) and I selected that option in one of
> my test because of that note: that version also crashed, hence I
> was under the impression that MAP_STACK couldn't be the problem.

malloc(3) uses mmap without MAP_STACK flag, so you'll end up with memory
not marked MAP_STACK in both cases.

Define MALLOC_STACK and add MAP_STACK to the flags,

        -Otto

>
>
> static char *_st_new_stk_segment(int size)
> {
> #ifdef MALLOC_STACK
>   void *vaddr = malloc(size);
> #else
>   int mmap_flags = MAP_PRIVATE;
>   void *vaddr;
>
>   mmap_flags |= MAP_ANON;
>   vaddr = mmap(NULL, size, PROT_READ | PROT_WRITE, mmap_flags, zero_fd, 0);
>   if (vaddr == (void *)MAP_FAILED)
>     return NULL;
> #endif /* MALLOC_STACK */
>   return (char *)vaddr;
> }
>

Reply | Threaded
Open this post in threaded view
|

Re: statethreads crashes in ld on 6.4

Claus Assmann-4
On Tue, Dec 04, 2018, Otto Moerbeek wrote:

> malloc(3) uses mmap without MAP_STACK flag, so you'll end up with memory
> not marked MAP_STACK in both cases.

Thanks for the information.

> Define MALLOC_STACK and add MAP_STACK to the flags,

You mean "undefine MALLOC_STACK", right? I don't see a way (in the
man page) to add MAP_STACK to malloc(2) memory, but I might be
missing something (again).

statethreads works when adding MAP_STACK:

...
#if defined (MAP_STACK)
  mmap_flags |= MAP_STACK;
#endif
  vaddr = mmap(NULL, size, PROT_READ | PROT_WRITE, mmap_flags, zero_fd, 0);


Thanks!

--
Address is valid for this mailing list only.

Reply | Threaded
Open this post in threaded view
|

Re: statethreads crashes in ld on 6.4

Edgar Pettijohn III-2
In reply to this post by Claus Assmann-4

On Dec 4, 2018 5:41 AM, Claus Assmann <[hidden email]> wrote:

>
> On Tue, Dec 04, 2018, Otto Moerbeek wrote:
>
> > malloc(3) uses mmap without MAP_STACK flag, so you'll end up with memory
> > not marked MAP_STACK in both cases.
>
> Thanks for the information.
>
> > Define MALLOC_STACK and add MAP_STACK to the flags,
>
> You mean "undefine MALLOC_STACK", right? I don't see a way (in the
> man page) to add MAP_STACK to malloc(2) memory, but I might be
> missing something (again).
>
> statethreads works when adding MAP_STACK:
>
> ...
> #if defined (MAP_STACK)
>   mmap_flags |= MAP_STACK;
> #endif
>   vaddr = mmap(NULL, size, PROT_READ | PROT_WRITE, mmap_flags, zero_fd, 0);
>
>
> Thanks!
>

Cool. I enjoy playing with meta1 from time to time. Haven't been entirely successful yet, but every mta has a learning curve.

Edgar
> --
> Address is valid for this mailing list only.
>

Reply | Threaded
Open this post in threaded view
|

Re: statethreads crashes in ld on 6.4

Otto Moerbeek
In reply to this post by Claus Assmann-4
On Tue, Dec 04, 2018 at 03:41:48AM -0800, Claus Assmann wrote:

> On Tue, Dec 04, 2018, Otto Moerbeek wrote:
>
> > malloc(3) uses mmap without MAP_STACK flag, so you'll end up with memory
> > not marked MAP_STACK in both cases.
>
> Thanks for the information.
>
> > Define MALLOC_STACK and add MAP_STACK to the flags,
>
> You mean "undefine MALLOC_STACK", right? I don't see a way (in the

Yes of course, you want to call mmap(2) yourself.

> man page) to add MAP_STACK to malloc(2) memory, but I might be
> missing something (again).
>
> statethreads works when adding MAP_STACK:
>
> ..
> #if defined (MAP_STACK)
>   mmap_flags |= MAP_STACK;
> #endif
>   vaddr = mmap(NULL, size, PROT_READ | PROT_WRITE, mmap_flags, zero_fd, 0);
>
>
> Thanks!
>
> --
> Address is valid for this mailing list only.
>