SIGSEGV in libunwind::UnwindInfoSectionsCache::CacheTree_RB_INSERT_COLOR

[stefan-2]

>Synopsis: SIGSEGV in libunwind::UnwindInfoSectionsCache::CacheTree_RB_INSERT_COLOR
>Category: library
>Environment:
        System      : OpenBSD 6.7
        Details     : OpenBSD 6.7 (GENERIC.MP) #182: Thu May  7 11:11:58 MDT 2020
                         [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP

        Architecture: OpenBSD.amd64
        Machine     : amd64
>Description:
[clang++] During a long-running stress test for exceptions inside threads
a segfault occurs in libunwind.  eg++ does not exhibit the problem.

(gdb) bt
#0  0x00000c75b159cca4 in libunwind::UnwindInfoSectionsCache::CacheTree_RB_INSERT_COLOR (this=0xc75b15c1248 <libunwind::uwis_cache>, head=0xc75b15c1250 <libunwind::uwis_cache+8>, elm=0x0)
    at /usr/src/lib/libcxxabi/../libunwind/src/AddressSpace.hpp:234
#1  0x00000c75b1599eaa in libunwind::UnwindInfoSectionsCache::CacheTree_RB_INSERT (this=<optimized out>, head=<optimized out>, elm=<optimized out>)
    at /usr/src/lib/libcxxabi/../libunwind/src/AddressSpace.hpp:234
#2  libunwind::UnwindInfoSectionsCache::setUnwindInfoSectionsForPC (this=<optimized out>, key=<optimized out>, uis=...) at /usr/src/lib/libcxxabi/../libunwind/src/AddressSpace.hpp:228
#3  libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::setInfoBasedOnIPRegister (this=0xc767ebd50c8, isReturnAddress=<optimized out>)
    at /usr/src/lib/libcxxabi/../libunwind/src/UnwindCursor.hpp:1828
#4  0x00000c75b1599b7d in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::step (this=0xc767ebd50c8)
    at /usr/src/lib/libcxxabi/../libunwind/src/UnwindCursor.hpp:1968
#5  0x00000c75b1598204 in unwind_phase1 (uc=0xc75b15a0af0 <__cxxabiv1::exception_cleanup_func(_Unwind_Reason_Code, _Unwind_Exception*)>,
    cursor=0xc75b15bcdd8 <vtable for libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>+16>, exception_object=0xc7638cc8960)
    at /usr/src/lib/libcxxabi/../libunwind/src/UnwindLevel1.c:46
#6  _Unwind_RaiseException (exception_object=0xc7638cc8960) at /usr/src/lib/libcxxabi/../libunwind/src/UnwindLevel1.c:357
#7  0x00000c75b15a0ae3 in __cxa_throw (thrown_object=0xc7638cc8980, tinfo=0xc739df31078 <typeinfo for suntime_error>, dest=<optimized out>) at /usr/src/lib/libcxxabi/src/cxa_exception.cpp:281
#8  0x00000c739df2f4c7 in doit () at test.cc:14
#9  0x00000c739df2f639 in std::__1::__invoke<void (*)()> (__f=<optimized out>) at /usr/include/c++/v1/type_traits:4353
#10 std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)()>&, std::__1::__tuple_indices<>) (__t=...) at /usr/include/c++/v1/thread:342
#11 std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)()> > (__vp=0xc760f757a40)
    at /usr/include/c++/v1/thread:352
#12 0x00000c768afa30d1 in _rthread_start (v=<optimized out>) at /usr/src/lib/librthread/rthread.c:96
#13 0x00000c7687359748 in __tfork_thread () at /usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:77
#14 0x0000000000000000 in ?? ()


>How-To-Repeat:

test.cc:
===============================================================================
#include <stdexcept>
#include <thread>
#include <vector>


class suntime_error : public std::runtime_error {
  using std::runtime_error::runtime_error;
};

static void
doit()
{
    try {
        throw suntime_error("this is an error");
    }
    catch (const suntime_error& err) {
        (void)err;
    }
}

static void
do_threads()
{
    const size_t n = 1000;
    std::vector<std::thread> t(n);

    for (size_t k = 0; k < n; k++) {
        t[k] = std::thread(doit);
    }

    for (size_t k = 0; k < n; k++) {
        t[k].join();
    }
}

int
main()
{
    do_threads();
    return 0;
}
===============================================================================


doit.sh:
===============================================================================
#!/bin/sh

while ./test; do
    :
done
===============================================================================


clang++ -pthread -Wall -Wextra -std=c++11 -pedantic -DNDEBUG -O2 -g -o test test.cc
./doit.sh


Note that it can take between 1min and several hours to reproduce this issue.



dmesg:
OpenBSD 6.7 (GENERIC.MP) #182: Thu May  7 11:11:58 MDT 2020
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 1056833536 (1007MB)
avail mem = 1012244480 (965MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf64b0 (9 entries)
bios0: vendor SeaBIOS version "Ubuntu-1.8.2-1ubuntu1" date 04/01/2014
bios0: QEMU Standard PC (i440FX + PIIX, 1996)
acpi0 at bios0: ACPI 1.0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC HPET
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: QEMU Virtual CPU version 2.5+, 4320.47 MHz, 06-06-03
cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,VMX,CX16,x2APIC,HV,NXE,LONG,LAHF,MELTDOWN
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 1230MHz
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpihpet0 at acpi0: 100000000 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: _OSC failed
acpicmos0 at acpi0
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
"PNP0A06" at acpi0 not configured
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
wd0 at pciide0 channel 0 drive 0: <QEMU HARDDISK>
wd0: 16-sector PIO, LBA48, 61440MB, 125829120 sectors
wd0(pciide0:0:0): using PIO mode 4, DMA mode 2
atapiscsi0 at pciide0 channel 1 drive 0
scsibus1 at atapiscsi0: 2 targets
cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
iic0 at piixpm0
vga1 at pci0 dev 2 function 0 "Bochs VGA" rev 0x02
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
em0 at pci0 dev 3 function 0 "Intel 82540EM" rev 0x03: apic 0 int 11, address 52:54:00:12:34:56
isa0 at pcib0
isadma0 at isa0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
lpt0 at isa0 port 0x378/4 irq 7
vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation)
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on wd0a (8da2a562c9eb4344.a) swap on wd0b dump on wd0b

usbdevs:
usbdevs: no USB controllers found

[Robert Nagy]

I am CC'ing pirofti@ on this.

On 18/12/20 15:19 +0100, [hidden email] wrote:

> >Synopsis: SIGSEGV in libunwind::UnwindInfoSectionsCache::CacheTree_RB_INSERT_COLOR
> >Category: library
> >Environment:
> System      : OpenBSD 6.7
> Details     : OpenBSD 6.7 (GENERIC.MP) #182: Thu May  7 11:11:58 MDT 2020
> [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> Architecture: OpenBSD.amd64
> Machine     : amd64
> >Description:
> [clang++] During a long-running stress test for exceptions inside threads
> a segfault occurs in libunwind.  eg++ does not exhibit the problem.
>
> (gdb) bt
> #0  0x00000c75b159cca4 in libunwind::UnwindInfoSectionsCache::CacheTree_RB_INSERT_COLOR (this=0xc75b15c1248 <libunwind::uwis_cache>, head=0xc75b15c1250 <libunwind::uwis_cache+8>, elm=0x0)
>     at /usr/src/lib/libcxxabi/../libunwind/src/AddressSpace.hpp:234
> #1  0x00000c75b1599eaa in libunwind::UnwindInfoSectionsCache::CacheTree_RB_INSERT (this=<optimized out>, head=<optimized out>, elm=<optimized out>)
>     at /usr/src/lib/libcxxabi/../libunwind/src/AddressSpace.hpp:234
> #2  libunwind::UnwindInfoSectionsCache::setUnwindInfoSectionsForPC (this=<optimized out>, key=<optimized out>, uis=...) at /usr/src/lib/libcxxabi/../libunwind/src/AddressSpace.hpp:228
> #3  libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::setInfoBasedOnIPRegister (this=0xc767ebd50c8, isReturnAddress=<optimized out>)
>     at /usr/src/lib/libcxxabi/../libunwind/src/UnwindCursor.hpp:1828
> #4  0x00000c75b1599b7d in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::step (this=0xc767ebd50c8)
>     at /usr/src/lib/libcxxabi/../libunwind/src/UnwindCursor.hpp:1968
> #5  0x00000c75b1598204 in unwind_phase1 (uc=0xc75b15a0af0 <__cxxabiv1::exception_cleanup_func(_Unwind_Reason_Code, _Unwind_Exception*)>,
>     cursor=0xc75b15bcdd8 <vtable for libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>+16>, exception_object=0xc7638cc8960)
>     at /usr/src/lib/libcxxabi/../libunwind/src/UnwindLevel1.c:46
> #6  _Unwind_RaiseException (exception_object=0xc7638cc8960) at /usr/src/lib/libcxxabi/../libunwind/src/UnwindLevel1.c:357
> #7  0x00000c75b15a0ae3 in __cxa_throw (thrown_object=0xc7638cc8980, tinfo=0xc739df31078 <typeinfo for suntime_error>, dest=<optimized out>) at /usr/src/lib/libcxxabi/src/cxa_exception.cpp:281
> #8  0x00000c739df2f4c7 in doit () at test.cc:14
> #9  0x00000c739df2f639 in std::__1::__invoke<void (*)()> (__f=<optimized out>) at /usr/include/c++/v1/type_traits:4353
> #10 std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)()>&, std::__1::__tuple_indices<>) (__t=...) at /usr/include/c++/v1/thread:342
> #11 std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)()> > (__vp=0xc760f757a40)
>     at /usr/include/c++/v1/thread:352
> #12 0x00000c768afa30d1 in _rthread_start (v=<optimized out>) at /usr/src/lib/librthread/rthread.c:96
> #13 0x00000c7687359748 in __tfork_thread () at /usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:77
> #14 0x0000000000000000 in ?? ()
>
>
> >How-To-Repeat:
>
> test.cc:
> ===============================================================================
> #include <stdexcept>
> #include <thread>
> #include <vector>
>
>
> class suntime_error : public std::runtime_error {
>   using std::runtime_error::runtime_error;
> };
>
> static void
> doit()
> {
>     try {
>         throw suntime_error("this is an error");
>     }
>     catch (const suntime_error& err) {
>         (void)err;
>     }
> }
>
> static void
> do_threads()
> {
>     const size_t n = 1000;
>     std::vector<std::thread> t(n);
>
>     for (size_t k = 0; k < n; k++) {
>         t[k] = std::thread(doit);
>     }
>
>     for (size_t k = 0; k < n; k++) {
>         t[k].join();
>     }
> }
>
> int
> main()
> {
>     do_threads();
>     return 0;
> }
> ===============================================================================
>
>
> doit.sh:
> ===============================================================================
> #!/bin/sh
>
> while ./test; do
>     :
> done
> ===============================================================================
>
>
> clang++ -pthread -Wall -Wextra -std=c++11 -pedantic -DNDEBUG -O2 -g -o test test.cc
> ./doit.sh
>
>
> Note that it can take between 1min and several hours to reproduce this issue.
>
>
>
> dmesg:
> OpenBSD 6.7 (GENERIC.MP) #182: Thu May  7 11:11:58 MDT 2020
>     [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 1056833536 (1007MB)
> avail mem = 1012244480 (965MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf64b0 (9 entries)
> bios0: vendor SeaBIOS version "Ubuntu-1.8.2-1ubuntu1" date 04/01/2014
> bios0: QEMU Standard PC (i440FX + PIIX, 1996)
> acpi0 at bios0: ACPI 1.0
> acpi0: sleep states S3 S4 S5
> acpi0: tables DSDT FACP SSDT APIC HPET
> acpi0: wakeup devices
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: QEMU Virtual CPU version 2.5+, 4320.47 MHz, 06-06-03
> cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,VMX,CX16,x2APIC,HV,NXE,LONG,LAHF,MELTDOWN
> cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
> cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
> cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 1230MHz
> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
> acpihpet0 at acpi0: 100000000 Hz
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpicpu0 at acpi0: C1(@1 halt!)
> "ACPI0006" at acpi0 not configured
> acpipci0 at acpi0 PCI0: _OSC failed
> acpicmos0 at acpi0
> "PNP0A06" at acpi0 not configured
> "PNP0A06" at acpi0 not configured
> "PNP0A06" at acpi0 not configured
> pvbus0 at mainbus0: KVM
> pvclock0 at pvbus0
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
> pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
> pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
> wd0 at pciide0 channel 0 drive 0: <QEMU HARDDISK>
> wd0: 16-sector PIO, LBA48, 61440MB, 125829120 sectors
> wd0(pciide0:0:0): using PIO mode 4, DMA mode 2
> atapiscsi0 at pciide0 channel 1 drive 0
> scsibus1 at atapiscsi0: 2 targets
> cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
> cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
> piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
> iic0 at piixpm0
> vga1 at pci0 dev 2 function 0 "Bochs VGA" rev 0x02
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> em0 at pci0 dev 3 function 0 "Intel 82540EM" rev 0x03: apic 0 int 11, address 52:54:00:12:34:56
> isa0 at pcib0
> isadma0 at isa0
> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pms0 at pckbc0 (aux slot)
> wsmouse0 at pms0 mux 0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> lpt0 at isa0 port 0x378/4 irq 7
> vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation)
> vscsi0 at root
> scsibus2 at vscsi0: 256 targets
> softraid0 at root
> scsibus3 at softraid0: 256 targets
> root on wd0a (8da2a562c9eb4344.a) swap on wd0b dump on wd0b
>
> usbdevs:
> usbdevs: no USB controllers found
>

--
Regards,
Robert Nagy

[Paul Irofti-4]

Hi,

First of all thank you for the report!

I looked at this last night when you sent it. Nothing obvious pops up. I
will try to try it on an OpenBSD machine soon. Currently away.

Not sure caching is the culprit, but will definitely investigate this
when I get back. If you don't mind asking, how did you trigger this
issue in real life?

Cheers,
Paul

On 18.12.2020 16:19, [hidden email] wrote:

>> Synopsis: SIGSEGV in libunwind::UnwindInfoSectionsCache::CacheTree_RB_INSERT_COLOR
>> Category: library
>> Environment:
> System      : OpenBSD 6.7
> Details     : OpenBSD 6.7 (GENERIC.MP) #182: Thu May  7 11:11:58 MDT 2020
> [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
>
> Architecture: OpenBSD.amd64
> Machine     : amd64
>> Description:
> [clang++] During a long-running stress test for exceptions inside threads
> a segfault occurs in libunwind.  eg++ does not exhibit the problem.
>
> (gdb) bt
> #0  0x00000c75b159cca4 in libunwind::UnwindInfoSectionsCache::CacheTree_RB_INSERT_COLOR (this=0xc75b15c1248 <libunwind::uwis_cache>, head=0xc75b15c1250 <libunwind::uwis_cache+8>, elm=0x0)
>      at /usr/src/lib/libcxxabi/../libunwind/src/AddressSpace.hpp:234
> #1  0x00000c75b1599eaa in libunwind::UnwindInfoSectionsCache::CacheTree_RB_INSERT (this=<optimized out>, head=<optimized out>, elm=<optimized out>)
>      at /usr/src/lib/libcxxabi/../libunwind/src/AddressSpace.hpp:234
> #2  libunwind::UnwindInfoSectionsCache::setUnwindInfoSectionsForPC (this=<optimized out>, key=<optimized out>, uis=...) at /usr/src/lib/libcxxabi/../libunwind/src/AddressSpace.hpp:228
> #3  libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::setInfoBasedOnIPRegister (this=0xc767ebd50c8, isReturnAddress=<optimized out>)
>      at /usr/src/lib/libcxxabi/../libunwind/src/UnwindCursor.hpp:1828
> #4  0x00000c75b1599b7d in libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>::step (this=0xc767ebd50c8)
>      at /usr/src/lib/libcxxabi/../libunwind/src/UnwindCursor.hpp:1968
> #5  0x00000c75b1598204 in unwind_phase1 (uc=0xc75b15a0af0 <__cxxabiv1::exception_cleanup_func(_Unwind_Reason_Code, _Unwind_Exception*)>,
>      cursor=0xc75b15bcdd8 <vtable for libunwind::UnwindCursor<libunwind::LocalAddressSpace, libunwind::Registers_x86_64>+16>, exception_object=0xc7638cc8960)
>      at /usr/src/lib/libcxxabi/../libunwind/src/UnwindLevel1.c:46
> #6  _Unwind_RaiseException (exception_object=0xc7638cc8960) at /usr/src/lib/libcxxabi/../libunwind/src/UnwindLevel1.c:357
> #7  0x00000c75b15a0ae3 in __cxa_throw (thrown_object=0xc7638cc8980, tinfo=0xc739df31078 <typeinfo for suntime_error>, dest=<optimized out>) at /usr/src/lib/libcxxabi/src/cxa_exception.cpp:281
> #8  0x00000c739df2f4c7 in doit () at test.cc:14
> #9  0x00000c739df2f639 in std::__1::__invoke<void (*)()> (__f=<optimized out>) at /usr/include/c++/v1/type_traits:4353
> #10 std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)()>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)()>&, std::__1::__tuple_indices<>) (__t=...) at /usr/include/c++/v1/thread:342
> #11 std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void (*)()> > (__vp=0xc760f757a40)
>      at /usr/include/c++/v1/thread:352
> #12 0x00000c768afa30d1 in _rthread_start (v=<optimized out>) at /usr/src/lib/librthread/rthread.c:96
> #13 0x00000c7687359748 in __tfork_thread () at /usr/src/lib/libc/arch/amd64/sys/tfork_thread.S:77
> #14 0x0000000000000000 in ?? ()
>
>
>> How-To-Repeat:
>
> test.cc:
> ===============================================================================
> #include <stdexcept>
> #include <thread>
> #include <vector>
>
>
> class suntime_error : public std::runtime_error {
>    using std::runtime_error::runtime_error;
> };
>
> static void
> doit()
> {
>      try {
>          throw suntime_error("this is an error");
>      }
>      catch (const suntime_error& err) {
>          (void)err;
>      }
> }
>
> static void
> do_threads()
> {
>      const size_t n = 1000;
>      std::vector<std::thread> t(n);
>
>      for (size_t k = 0; k < n; k++) {
>          t[k] = std::thread(doit);
>      }
>
>      for (size_t k = 0; k < n; k++) {
>          t[k].join();
>      }
> }
>
> int
> main()
> {
>      do_threads();
>      return 0;
> }
> ===============================================================================
>
>
> doit.sh:
> ===============================================================================
> #!/bin/sh
>
> while ./test; do
>      :
> done
> ===============================================================================
>
>
> clang++ -pthread -Wall -Wextra -std=c++11 -pedantic -DNDEBUG -O2 -g -o test test.cc
> ./doit.sh
>
>
> Note that it can take between 1min and several hours to reproduce this issue.
>
>
>
> dmesg:
> OpenBSD 6.7 (GENERIC.MP) #182: Thu May  7 11:11:58 MDT 2020
>      [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
> real mem = 1056833536 (1007MB)
> avail mem = 1012244480 (965MB)
> mpath0 at root
> scsibus0 at mpath0: 256 targets
> mainbus0 at root
> bios0 at mainbus0: SMBIOS rev. 2.8 @ 0xf64b0 (9 entries)
> bios0: vendor SeaBIOS version "Ubuntu-1.8.2-1ubuntu1" date 04/01/2014
> bios0: QEMU Standard PC (i440FX + PIIX, 1996)
> acpi0 at bios0: ACPI 1.0
> acpi0: sleep states S3 S4 S5
> acpi0: tables DSDT FACP SSDT APIC HPET
> acpi0: wakeup devices
> acpitimer0 at acpi0: 3579545 Hz, 24 bits
> acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
> cpu0 at mainbus0: apid 0 (boot processor)
> cpu0: QEMU Virtual CPU version 2.5+, 4320.47 MHz, 06-06-03
> cpu0: FPU,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SSE3,VMX,CX16,x2APIC,HV,NXE,LONG,LAHF,MELTDOWN
> cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 512KB 64b/line 16-way L2 cache
> cpu0: ITLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
> cpu0: DTLB 255 4KB entries direct-mapped, 255 4MB entries direct-mapped
> cpu0: smt 0, core 0, package 0
> mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
> cpu0: apic clock running at 1230MHz
> ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
> acpihpet0 at acpi0: 100000000 Hz
> acpiprt0 at acpi0: bus 0 (PCI0)
> acpicpu0 at acpi0: C1(@1 halt!)
> "ACPI0006" at acpi0 not configured
> acpipci0 at acpi0 PCI0: _OSC failed
> acpicmos0 at acpi0
> "PNP0A06" at acpi0 not configured
> "PNP0A06" at acpi0 not configured
> "PNP0A06" at acpi0 not configured
> pvbus0 at mainbus0: KVM
> pvclock0 at pvbus0
> pci0 at mainbus0 bus 0
> pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
> pcib0 at pci0 dev 1 function 0 "Intel 82371SB ISA" rev 0x00
> pciide0 at pci0 dev 1 function 1 "Intel 82371SB IDE" rev 0x00: DMA, channel 0 wired to compatibility, channel 1 wired to compatibility
> wd0 at pciide0 channel 0 drive 0: <QEMU HARDDISK>
> wd0: 16-sector PIO, LBA48, 61440MB, 125829120 sectors
> wd0(pciide0:0:0): using PIO mode 4, DMA mode 2
> atapiscsi0 at pciide0 channel 1 drive 0
> scsibus1 at atapiscsi0: 2 targets
> cd0 at scsibus1 targ 0 lun 0: <QEMU, QEMU DVD-ROM, 2.5+> removable
> cd0(pciide0:1:0): using PIO mode 4, DMA mode 2
> piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: apic 0 int 9
> iic0 at piixpm0
> vga1 at pci0 dev 2 function 0 "Bochs VGA" rev 0x02
> wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
> wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
> em0 at pci0 dev 3 function 0 "Intel 82540EM" rev 0x03: apic 0 int 11, address 52:54:00:12:34:56
> isa0 at pcib0
> isadma0 at isa0
> fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
> com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
> pckbc0 at isa0 port 0x60/5 irq 1 irq 12
> pckbd0 at pckbc0 (kbd slot)
> wskbd0 at pckbd0: console keyboard, using wsdisplay0
> pms0 at pckbc0 (aux slot)
> wsmouse0 at pms0 mux 0
> pcppi0 at isa0 port 0x61
> spkr0 at pcppi0
> lpt0 at isa0 port 0x378/4 irq 7
> vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation)
> vscsi0 at root
> scsibus2 at vscsi0: 256 targets
> softraid0 at root
> scsibus3 at softraid0: 256 targets
> root on wd0a (8da2a562c9eb4344.a) swap on wd0b dump on wd0b
>
> usbdevs:
> usbdevs: no USB controllers found
>

[stefan-2]

Hi,

thanks for taking a look!

On Sat, Dec 19, 2020 at 10:37:49PM +0200, Paul Irofti wrote:
> Not sure caching is the culprit, but will definitely investigate this when I
> get back. If you don't mind asking, how did you trigger this issue in real
> life?

This was found by the mpdecimal-2.5.1 release candidate test suite. libmpdec++
(apitest.cc) has many tests of the form:

    assertRaises(ConversionSyntax, [&](){ Decimal("xyz", xc); });

assertRaises() does pretty much the same as the doit() function in test.cc
in this issue, and apitest.cc runs the tests in parallel in 35 threads.


A single run of the tests usually does not trigger this.  I wrote a script
(initially for AIX/xlc) that runs the tests with all libmpdec configs, all
optimization levels and all test suite flags.

This boils down to > 400 cases and the script is stress testing the OS and
tool chain for a week.


On AIX it found that errno still needs -D_THREAD_SAFE to be actually thread
safe, and on OpenBSD this issue.  Both issues were only discovered after
several days`of running the tests.



Cheers,

Stefan Krah

[stefan-2]

The upgrade to clang 10.0.1 is probably still a moving target, but I can
still reproduce this issue with the latest snapshot.


Stefan Krah

[Paul Irofti-4]

I think this was fixed by otto@ with the commit below:

http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/gnu/llvm/libunwind/src/UnwindCursor.hpp.diff?r1=1.2&r2=1.3

[stefan-2]

On Mon, Feb 22, 2021 at 07:16:07PM +0200, Paul Irofti wrote:
> I think this was fixed by otto@ with the commit below:
>
> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/gnu/llvm/libunwind/src/UnwindCursor.hpp.diff?r1=1.2&r2=1.3

Thanks, I can confirm that it is fixed in the latest snapshot (I ran the tests for
4 hours).


Stefan Krah