option kcov + GENERIC.MP -> silent crash

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

option kcov + GENERIC.MP -> silent crash

Greg Steuck
Hi Anton,

I tried to boot a kernel with kcov based on GENERIC.MP and the machine
reboots without a peep immediately after

vmm0 at mainbus0: VMX (using slow L1TF mitigation)

Switching off either of kcov or MP results in normally working kernels. I'm
attaching two concatenated dmesgs. The effect is reproducible on real HW
and on GCE VM. Broken config is just:
$ cat /sys/arch/amd64/conf/SYZKALLER
include "arch/amd64/conf/GENERIC.MP"
pseudo-device kcov 1

Disabling either vmm or kcov in broken kernel UKC doesn't prevent crashes.

Thanks
Greg

--
nest.cx is Gmail hosted, use PGP for anything private. Key:
http://goo.gl/6dMsr
Fingerprint: 5E2B 2D0E 1E03 2046 BEC3  4D50 0B15 42BD 8DF5 A1B0

kcov-mp-crash.txt (15K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: option kcov + GENERIC.MP -> silent crash

Anton Lindqvist-2
Hi Greg,

On Sun, Nov 25, 2018 at 10:13:52AM -0800, Greg Steuck wrote:

> Hi Anton,
>
> I tried to boot a kernel with kcov based on GENERIC.MP and the machine
> reboots without a peep immediately after
>
> vmm0 at mainbus0: VMX (using slow L1TF mitigation)
>
> Switching off either of kcov or MP results in normally working kernels. I'm
> attaching two concatenated dmesgs. The effect is reproducible on real HW
> and on GCE VM. Broken config is just:
> $ cat /sys/arch/amd64/conf/SYZKALLER
> include "arch/amd64/conf/GENERIC.MP"
> pseudo-device kcov 1
>
> Disabling either vmm or kcov in broken kernel UKC doesn't prevent crashes.

Known limitation, I haven't spent much time on making kcov MP-safe.
Especially since it's primarily used inside a VM through vmm which
currently is limited to a single CPU.

However, I did some investigation before and concluded that the problem
resides in the trace routine which is called from
cpu_boot_secondary_processors() before the secondary CPU is accessible
through curcpu(). I came up with a hackish solution to this problem (see
diff below) that got rejected; kettenis@ mentioned that we instead
should set MSR_GSBASE earlier in cpu_hatch() but I never managed to get
the right people involved with knowledge in this area. I might take a
look myself.

In the meantime, you could give the diff a try. It might be the case
that more functions are not eligible for tracing. OpenBSD as no method
of turning of tracing for a given source file like Linux does. This
might become necessary since I fear many more functions will not cope
with tracing.

Index: dev/kcov.c
===================================================================
RCS file: /cvs/src/sys/dev/kcov.c,v
retrieving revision 1.4
diff -u -p -r1.4 kcov.c
--- dev/kcov.c 27 Aug 2018 15:57:39 -0000 1.4
+++ dev/kcov.c 8 Sep 2018 21:51:20 -0000
@@ -49,6 +49,7 @@ struct kcov_dev {
 };
 
 void kcovattach(int);
+void kcov_attachhook(struct device *);
 
 int kd_alloc(struct kcov_dev *, unsigned long);
 void kd_free(struct kcov_dev *);
@@ -57,6 +58,7 @@ struct kcov_dev *kd_lookup(int);
 static inline int inintr(void);
 
 TAILQ_HEAD(, kcov_dev) kd_list = TAILQ_HEAD_INITIALIZER(kd_list);
+int kcov_attached = 0;
 
 #ifdef KCOV_DEBUG
 int kcov_debug = 1;
@@ -76,12 +78,11 @@ int kcov_debug = 1;
 void
 __sanitizer_cov_trace_pc(void)
 {
- extern int cold;
  struct kcov_dev *kd;
  uint64_t idx;
 
- /* Do not trace during boot. */
- if (cold)
+ /* Do not trace before the root file system is mounted. */
+ if (!kcov_attached)
  return;
 
  /* Do not trace in interrupts to prevent noisy coverage. */
@@ -102,6 +103,13 @@ __sanitizer_cov_trace_pc(void)
 void
 kcovattach(int count)
 {
+ config_mountroot(NULL, kcov_attachhook);
+}
+
+void
+kcov_attachhook(struct device *dev)
+{
+ kcov_attached = 1;
 }
 
 int

Reply | Threaded
Open this post in threaded view
|

Re: option kcov + GENERIC.MP -> silent crash

Greg Steuck
Thanks for the patch, I'll give it a go. Should I make up a patch reporting
#error when trying to build kcov with MP in the meantime? The next person
won't have to find it the hard way...

On Sun, Nov 25, 2018 at 11:21 PM Anton Lindqvist <[hidden email]> wrote:

> Hi Greg,
>
> On Sun, Nov 25, 2018 at 10:13:52AM -0800, Greg Steuck wrote:
> > Hi Anton,
> >
> > I tried to boot a kernel with kcov based on GENERIC.MP and the machine
> > reboots without a peep immediately after
> >
> > vmm0 at mainbus0: VMX (using slow L1TF mitigation)
> >
> > Switching off either of kcov or MP results in normally working kernels.
> I'm
> > attaching two concatenated dmesgs. The effect is reproducible on real HW
> > and on GCE VM. Broken config is just:
> > $ cat /sys/arch/amd64/conf/SYZKALLER
> > include "arch/amd64/conf/GENERIC.MP"
> > pseudo-device kcov 1
> >
> > Disabling either vmm or kcov in broken kernel UKC doesn't prevent
> crashes.
>
> Known limitation, I haven't spent much time on making kcov MP-safe.
> Especially since it's primarily used inside a VM through vmm which
> currently is limited to a single CPU.
>
> However, I did some investigation before and concluded that the problem
> resides in the trace routine which is called from
> cpu_boot_secondary_processors() before the secondary CPU is accessible
> through curcpu(). I came up with a hackish solution to this problem (see
> diff below) that got rejected; kettenis@ mentioned that we instead
> should set MSR_GSBASE earlier in cpu_hatch() but I never managed to get
> the right people involved with knowledge in this area. I might take a
> look myself.
>
> In the meantime, you could give the diff a try. It might be the case
> that more functions are not eligible for tracing. OpenBSD as no method
> of turning of tracing for a given source file like Linux does. This
> might become necessary since I fear many more functions will not cope
> with tracing.
>
> Index: dev/kcov.c
> ===================================================================
> RCS file: /cvs/src/sys/dev/kcov.c,v
> retrieving revision 1.4
> diff -u -p -r1.4 kcov.c
> --- dev/kcov.c  27 Aug 2018 15:57:39 -0000      1.4
> +++ dev/kcov.c  8 Sep 2018 21:51:20 -0000
> @@ -49,6 +49,7 @@ struct kcov_dev {
>  };
>
>  void kcovattach(int);
> +void kcov_attachhook(struct device *);
>
>  int kd_alloc(struct kcov_dev *, unsigned long);
>  void kd_free(struct kcov_dev *);
> @@ -57,6 +58,7 @@ struct kcov_dev *kd_lookup(int);
>  static inline int inintr(void);
>
>  TAILQ_HEAD(, kcov_dev) kd_list = TAILQ_HEAD_INITIALIZER(kd_list);
> +int kcov_attached = 0;
>
>  #ifdef KCOV_DEBUG
>  int kcov_debug = 1;
> @@ -76,12 +78,11 @@ int kcov_debug = 1;
>  void
>  __sanitizer_cov_trace_pc(void)
>  {
> -       extern int cold;
>         struct kcov_dev *kd;
>         uint64_t idx;
>
> -       /* Do not trace during boot. */
> -       if (cold)
> +       /* Do not trace before the root file system is mounted. */
> +       if (!kcov_attached)
>                 return;
>
>         /* Do not trace in interrupts to prevent noisy coverage. */
> @@ -102,6 +103,13 @@ __sanitizer_cov_trace_pc(void)
>  void
>  kcovattach(int count)
>  {
> +       config_mountroot(NULL, kcov_attachhook);
> +}
> +
> +void
> +kcov_attachhook(struct device *dev)
> +{
> +       kcov_attached = 1;
>  }
>
>  int
>


--
nest.cx is Gmail hosted, use PGP for anything private. Key:
http://goo.gl/6dMsr
Fingerprint: 5E2B 2D0E 1E03 2046 BEC3  4D50 0B15 42BD 8DF5 A1B0
Reply | Threaded
Open this post in threaded view
|

Re: option kcov + GENERIC.MP -> silent crash

Anton Lindqvist-2
On Mon, Nov 26, 2018 at 08:43:12AM -0800, Greg Steuck wrote:
> Thanks for the patch, I'll give it a go. Should I make up a patch reporting
> #error when trying to build kcov with MP in the meantime? The next person
> won't have to find it the hard way...

Please try out the diff first. I'd rather try coming up with a proper
fix before adding any #error directives.

>
> On Sun, Nov 25, 2018 at 11:21 PM Anton Lindqvist <[hidden email]> wrote:
>
> > Hi Greg,
> >
> > On Sun, Nov 25, 2018 at 10:13:52AM -0800, Greg Steuck wrote:
> > > Hi Anton,
> > >
> > > I tried to boot a kernel with kcov based on GENERIC.MP and the machine
> > > reboots without a peep immediately after
> > >
> > > vmm0 at mainbus0: VMX (using slow L1TF mitigation)
> > >
> > > Switching off either of kcov or MP results in normally working kernels.
> > I'm
> > > attaching two concatenated dmesgs. The effect is reproducible on real HW
> > > and on GCE VM. Broken config is just:
> > > $ cat /sys/arch/amd64/conf/SYZKALLER
> > > include "arch/amd64/conf/GENERIC.MP"
> > > pseudo-device kcov 1
> > >
> > > Disabling either vmm or kcov in broken kernel UKC doesn't prevent
> > crashes.
> >
> > Known limitation, I haven't spent much time on making kcov MP-safe.
> > Especially since it's primarily used inside a VM through vmm which
> > currently is limited to a single CPU.
> >
> > However, I did some investigation before and concluded that the problem
> > resides in the trace routine which is called from
> > cpu_boot_secondary_processors() before the secondary CPU is accessible
> > through curcpu(). I came up with a hackish solution to this problem (see
> > diff below) that got rejected; kettenis@ mentioned that we instead
> > should set MSR_GSBASE earlier in cpu_hatch() but I never managed to get
> > the right people involved with knowledge in this area. I might take a
> > look myself.
> >
> > In the meantime, you could give the diff a try. It might be the case
> > that more functions are not eligible for tracing. OpenBSD as no method
> > of turning of tracing for a given source file like Linux does. This
> > might become necessary since I fear many more functions will not cope
> > with tracing.
> >
> > Index: dev/kcov.c
> > ===================================================================
> > RCS file: /cvs/src/sys/dev/kcov.c,v
> > retrieving revision 1.4
> > diff -u -p -r1.4 kcov.c
> > --- dev/kcov.c  27 Aug 2018 15:57:39 -0000      1.4
> > +++ dev/kcov.c  8 Sep 2018 21:51:20 -0000
> > @@ -49,6 +49,7 @@ struct kcov_dev {
> >  };
> >
> >  void kcovattach(int);
> > +void kcov_attachhook(struct device *);
> >
> >  int kd_alloc(struct kcov_dev *, unsigned long);
> >  void kd_free(struct kcov_dev *);
> > @@ -57,6 +58,7 @@ struct kcov_dev *kd_lookup(int);
> >  static inline int inintr(void);
> >
> >  TAILQ_HEAD(, kcov_dev) kd_list = TAILQ_HEAD_INITIALIZER(kd_list);
> > +int kcov_attached = 0;
> >
> >  #ifdef KCOV_DEBUG
> >  int kcov_debug = 1;
> > @@ -76,12 +78,11 @@ int kcov_debug = 1;
> >  void
> >  __sanitizer_cov_trace_pc(void)
> >  {
> > -       extern int cold;
> >         struct kcov_dev *kd;
> >         uint64_t idx;
> >
> > -       /* Do not trace during boot. */
> > -       if (cold)
> > +       /* Do not trace before the root file system is mounted. */
> > +       if (!kcov_attached)
> >                 return;
> >
> >         /* Do not trace in interrupts to prevent noisy coverage. */
> > @@ -102,6 +103,13 @@ __sanitizer_cov_trace_pc(void)
> >  void
> >  kcovattach(int count)
> >  {
> > +       config_mountroot(NULL, kcov_attachhook);
> > +}
> > +
> > +void
> > +kcov_attachhook(struct device *dev)
> > +{
> > +       kcov_attached = 1;
> >  }
> >
> >  int
> >
>
>
> --
> nest.cx is Gmail hosted, use PGP for anything private. Key:
> http://goo.gl/6dMsr
> Fingerprint: 5E2B 2D0E 1E03 2046 BEC3  4D50 0B15 42BD 8DF5 A1B0

Reply | Threaded
Open this post in threaded view
|

Re: option kcov + GENERIC.MP -> silent crash

Greg Steuck
I booted the patched kernel and it seems to have gone farther and I believe
reached init before crashing.

boot> b bsd.anton
booting hd0a:bsd.anton: 12380226+2360336+270368+0+675840
[684182+128+754752+529898]=0x10d8f48
entry point at 0x1001000
[ using 1969992 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
        The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2018 OpenBSD. All rights reserved.
https://www.OpenBSD.org
OpenBSD 6.4-current (SYZKALLER) #0: Tue Nov 27 17:40:55 PST 2018
    [hidden email]
:/home/syzkaller/src/sys/arch/amd64/compile/SYZKALLER
real mem = 17163079680 (16367MB)
avail mem = 16632164352 (15861MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xbffffcf0 (20 entries)
bios0: vendor Google version "Google" date 01/01/2011
bios0: Google Google Compute Engine
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC WAET SRAT
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU @ 2.30GHz, 2300.42 MHz, 06-3f-00
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.
2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 999MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU @ 2.30GHz, 2299.55 MHz, 06-3f-00
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.
2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) CPU @ 2.30GHz, 2299.53 MHz, 06-3f-00
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.
2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU @ 2.30GHz, 2299.54 MHz, 06-3f-00
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.
2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 1 (application processor)
cpu4: Intel(R) Xeon(R) CPU @ 2.30GHz, 2299.57 MHz, 06-3f-00
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.
2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 1, core 0, package 0
cpu5 at mainbus0: apid 3 (application processor)
cpu5: Intel(R) Xeon(R) CPU @ 2.30GHz, 2299.56 MHz, 06-3f-00
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.
2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 1, core 1, package 0
cpu6 at mainbus0: apid 5 (application processor)
cpu6: Intel(R) Xeon(R) CPU @ 2.30GHz, 2299.54 MHz, 06-3f-00
cpu6:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.
2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu6: 256KB 64b/line 8-way L2 cache
cpu6: smt 1, core 2, package 0
cpu7 at mainbus0: apid 7 (application processor)
cpu7: Intel(R) Xeon(R) CPU @ 2.30GHz, 2299.57 MHz, 06-3f-00
cpu7:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.
2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu7: 256KB 64b/line 8-way L2 cache
cpu7: smt 1, core 3, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
acpicpu4 at acpi0: C1(@1 halt!)
acpicpu5 at acpi0: C1(@1 halt!)
acpicpu6 at acpi0: C1(@1 halt!)
acpicpu7 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: _OSC failed
acpicmos0 at acpi0
"QEMU0001" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371AB PIIX4 ISA" rev 0x03
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: SMBus
disabled
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio SCSI" rev 0x00
vioscsi0 at virtio0: qsize 8192
scsibus1 at vioscsi0: 253 targets
sd0 at scsibus1 targ 1 lun 0: <Google, PersistentDisk, 1> SCSI4 0/direct
fixed serial.Google_PersistentDisk_
sd0: 10240MB, 512 bytes/sector, 20971520 sectors, thin
virtio0: msix shared
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio1: address 42:01:0a:80:00:2f
virtio1: msix per-VQ
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0 mux 1
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation)
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (cea3ae3572e2c76c.a) swap on sd0b dump on sd0b

SeaBIOS (version 1.8.2-20181014_101610-google)
Total RAM Size = 0x0000000400000000 = 16384 MiB

On Mon, Nov 26, 2018 at 10:56 AM Anton Lindqvist <[hidden email]> wrote:

> On Mon, Nov 26, 2018 at 08:43:12AM -0800, Greg Steuck wrote:
> > Thanks for the patch, I'll give it a go. Should I make up a patch
> reporting
> > #error when trying to build kcov with MP in the meantime? The next person
> > won't have to find it the hard way...
>
> Please try out the diff first. I'd rather try coming up with a proper
> fix before adding any #error directives.
>
> >
> > On Sun, Nov 25, 2018 at 11:21 PM Anton Lindqvist <[hidden email]>
> wrote:
> >
> > > Hi Greg,
> > >
> > > On Sun, Nov 25, 2018 at 10:13:52AM -0800, Greg Steuck wrote:
> > > > Hi Anton,
> > > >
> > > > I tried to boot a kernel with kcov based on GENERIC.MP and the
> machine
> > > > reboots without a peep immediately after
> > > >
> > > > vmm0 at mainbus0: VMX (using slow L1TF mitigation)
> > > >
> > > > Switching off either of kcov or MP results in normally working
> kernels.
> > > I'm
> > > > attaching two concatenated dmesgs. The effect is reproducible on
> real HW
> > > > and on GCE VM. Broken config is just:
> > > > $ cat /sys/arch/amd64/conf/SYZKALLER
> > > > include "arch/amd64/conf/GENERIC.MP"
> > > > pseudo-device kcov 1
> > > >
> > > > Disabling either vmm or kcov in broken kernel UKC doesn't prevent
> > > crashes.
> > >
> > > Known limitation, I haven't spent much time on making kcov MP-safe.
> > > Especially since it's primarily used inside a VM through vmm which
> > > currently is limited to a single CPU.
> > >
> > > However, I did some investigation before and concluded that the problem
> > > resides in the trace routine which is called from
> > > cpu_boot_secondary_processors() before the secondary CPU is accessible
> > > through curcpu(). I came up with a hackish solution to this problem
> (see
> > > diff below) that got rejected; kettenis@ mentioned that we instead
> > > should set MSR_GSBASE earlier in cpu_hatch() but I never managed to get
> > > the right people involved with knowledge in this area. I might take a
> > > look myself.
> > >
> > > In the meantime, you could give the diff a try. It might be the case
> > > that more functions are not eligible for tracing. OpenBSD as no method
> > > of turning of tracing for a given source file like Linux does. This
> > > might become necessary since I fear many more functions will not cope
> > > with tracing.
> > >
> > > Index: dev/kcov.c
> > > ===================================================================
> > > RCS file: /cvs/src/sys/dev/kcov.c,v
> > > retrieving revision 1.4
> > > diff -u -p -r1.4 kcov.c
> > > --- dev/kcov.c  27 Aug 2018 15:57:39 -0000      1.4
> > > +++ dev/kcov.c  8 Sep 2018 21:51:20 -0000
> > > @@ -49,6 +49,7 @@ struct kcov_dev {
> > >  };
> > >
> > >  void kcovattach(int);
> > > +void kcov_attachhook(struct device *);
> > >
> > >  int kd_alloc(struct kcov_dev *, unsigned long);
> > >  void kd_free(struct kcov_dev *);
> > > @@ -57,6 +58,7 @@ struct kcov_dev *kd_lookup(int);
> > >  static inline int inintr(void);
> > >
> > >  TAILQ_HEAD(, kcov_dev) kd_list = TAILQ_HEAD_INITIALIZER(kd_list);
> > > +int kcov_attached = 0;
> > >
> > >  #ifdef KCOV_DEBUG
> > >  int kcov_debug = 1;
> > > @@ -76,12 +78,11 @@ int kcov_debug = 1;
> > >  void
> > >  __sanitizer_cov_trace_pc(void)
> > >  {
> > > -       extern int cold;
> > >         struct kcov_dev *kd;
> > >         uint64_t idx;
> > >
> > > -       /* Do not trace during boot. */
> > > -       if (cold)
> > > +       /* Do not trace before the root file system is mounted. */
> > > +       if (!kcov_attached)
> > >                 return;
> > >
> > >         /* Do not trace in interrupts to prevent noisy coverage. */
> > > @@ -102,6 +103,13 @@ __sanitizer_cov_trace_pc(void)
> > >  void
> > >  kcovattach(int count)
> > >  {
> > > +       config_mountroot(NULL, kcov_attachhook);
> > > +}
> > > +
> > > +void
> > > +kcov_attachhook(struct device *dev)
> > > +{
> > > +       kcov_attached = 1;
> > >  }
> > >
> > >  int
> > >
> >
> >
> > --
> > nest.cx is Gmail hosted, use PGP for anything private. Key:
> > http://goo.gl/6dMsr
> > Fingerprint: 5E2B 2D0E 1E03 2046 BEC3  4D50 0B15 42BD 8DF5 A1B0
>


--
nest.cx is Gmail hosted, use PGP for anything private. Key:
http://goo.gl/6dMsr
Fingerprint: 5E2B 2D0E 1E03 2046 BEC3  4D50 0B15 42BD 8DF5 A1B0
Reply | Threaded
Open this post in threaded view
|

Re: option kcov + GENERIC.MP -> silent crash

Anton Lindqvist-2
On Tue, Nov 27, 2018 at 05:52:15PM -0800, Greg Steuck wrote:
> I booted the patched kernel and it seems to have gone farther and I believe
> reached init before crashing.

By performing a semi-automated bisect I was able to identify the source
files that are incompatible with tracing. Common for all source files
seems to be that they define routines called early on in the boot
process before curcpu() is usable.

I do not have any plans on committing the diff below but please give it
a try. Instead, I'm working on extending the files.conf(5) grammar in
order to infer a different make target for the files.

Index: arch/amd64/conf/Makefile.amd64
===================================================================
RCS file: /cvs/src/sys/arch/amd64/conf/Makefile.amd64,v
retrieving revision 1.106
diff -u -p -r1.106 Makefile.amd64
--- arch/amd64/conf/Makefile.amd64 30 Oct 2018 11:08:30 -0000 1.106
+++ arch/amd64/conf/Makefile.amd64 1 Dec 2018 15:32:58 -0000
@@ -151,7 +151,31 @@ vers.o: ${SYSTEM_DEP:Ngap.o}
  ${CC} ${CFLAGS} ${CPPFLAGS} ${PROF} -c vers.c
 
 .if ${SYSTEM_OBJ:Mkcov.o} && ${COMPILER_VERSION:Mclang}
+amd64_mem.o: $S/arch/amd64/amd64/amd64_mem.c
+ ${NORMAL_C} -fno-sanitize-coverage=trace-pc
+cpu.o: $S/arch/amd64/amd64/cpu.c
+ ${NORMAL_C} -fno-sanitize-coverage=trace-pc
+fpu.o: $S/arch/amd64/amd64/fpu.c
+ ${NORMAL_C} -fno-sanitize-coverage=trace-pc
+gdt.o: $S/arch/amd64/amd64/gdt.c
+ ${NORMAL_C} -fno-sanitize-coverage=trace-pc
+intr.o: $S/arch/amd64/amd64/intr.c
+ ${NORMAL_C} -fno-sanitize-coverage=trace-pc
+lapic.o: $S/arch/amd64/amd64/lapic.c
+ ${NORMAL_C} -fno-sanitize-coverage=trace-pc
+machdep.o: $S/arch/amd64/amd64/machdep.c
+ ${NORMAL_C} -fno-sanitize-coverage=trace-pc
+tsc.o: $S/arch/amd64/amd64/tsc.c
+ ${NORMAL_C} -fno-sanitize-coverage=trace-pc
 kcov.o: $S/dev/kcov.c
+ ${NORMAL_C} -fno-sanitize-coverage=trace-pc
+pvbus.o: $S/dev/pv/pvbus.c
+ ${NORMAL_C} -fno-sanitize-coverage=trace-pc
+kern_lock.o: $S/kern/kern_lock.c
+ ${NORMAL_C} -fno-sanitize-coverage=trace-pc
+kern_sched.o: $S/kern/kern_sched.c
+ ${NORMAL_C} -fno-sanitize-coverage=trace-pc
+kern_tc.o: $S/kern/kern_tc.c
  ${NORMAL_C} -fno-sanitize-coverage=trace-pc
 .endif
 

Reply | Threaded
Open this post in threaded view
|

Re: option kcov + GENERIC.MP -> silent crash

Martin Pieuchot
On 01/12/18(Sat) 16:34, Anton Lindqvist wrote:

> On Tue, Nov 27, 2018 at 05:52:15PM -0800, Greg Steuck wrote:
> > I booted the patched kernel and it seems to have gone farther and I believe
> > reached init before crashing.
>
> By performing a semi-automated bisect I was able to identify the source
> files that are incompatible with tracing. Common for all source files
> seems to be that they define routines called early on in the boot
> process before curcpu() is usable.
>
> I do not have any plans on committing the diff below but please give it
> a try. Instead, I'm working on extending the files.conf(5) grammar in
> order to infer a different make target for the files.

Is it possible to mark incompatible functions using __attribute__ and
the preprocessor?  For example offending code with GPROF is marked with:

#define __noprof __attribute__((__no_instrument_function__))

> Index: arch/amd64/conf/Makefile.amd64
> ===================================================================
> RCS file: /cvs/src/sys/arch/amd64/conf/Makefile.amd64,v
> retrieving revision 1.106
> diff -u -p -r1.106 Makefile.amd64
> --- arch/amd64/conf/Makefile.amd64 30 Oct 2018 11:08:30 -0000 1.106
> +++ arch/amd64/conf/Makefile.amd64 1 Dec 2018 15:32:58 -0000
> @@ -151,7 +151,31 @@ vers.o: ${SYSTEM_DEP:Ngap.o}
>   ${CC} ${CFLAGS} ${CPPFLAGS} ${PROF} -c vers.c
>  
>  .if ${SYSTEM_OBJ:Mkcov.o} && ${COMPILER_VERSION:Mclang}
> +amd64_mem.o: $S/arch/amd64/amd64/amd64_mem.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +cpu.o: $S/arch/amd64/amd64/cpu.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +fpu.o: $S/arch/amd64/amd64/fpu.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +gdt.o: $S/arch/amd64/amd64/gdt.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +intr.o: $S/arch/amd64/amd64/intr.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +lapic.o: $S/arch/amd64/amd64/lapic.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +machdep.o: $S/arch/amd64/amd64/machdep.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +tsc.o: $S/arch/amd64/amd64/tsc.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
>  kcov.o: $S/dev/kcov.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +pvbus.o: $S/dev/pv/pvbus.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +kern_lock.o: $S/kern/kern_lock.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +kern_sched.o: $S/kern/kern_sched.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +kern_tc.o: $S/kern/kern_tc.c
>   ${NORMAL_C} -fno-sanitize-coverage=trace-pc
>  .endif
>  
>

Reply | Threaded
Open this post in threaded view
|

Re: option kcov + GENERIC.MP -> silent crash

Greg Steuck
In reply to this post by Anton Lindqvist-2
Hi Anton,

Unfortunately it's still crashing. The log is below, but to make
sure I'm not deluding myself, the source tree is
https://github.com/blackgnezdo/src/tree/anton-kcov-dec1

This is the workdir where I'm building:

commit fea58d64a837907fd3b5c45eb2b77351ac105d5f (HEAD -> anton-kcov-dec1)

    SYZKALLER.MP conf

commit 583b85f9857e44ee3339d9bb74e2780e435e3937 (origin/anton-kcov-dec1)

    anton@: disable kcov in files incompatible with tracing

    By performing a semi-automated bisect I was able to identify the source
    files that are incompatible with tracing. Common for all source files
    seems to be that they define routines called early on in the boot
    process before curcpu() is usable.

commit 3f7c3e6a6fe644f1ab7c92ea63819fb056a99f6d

    regen

Happy to test more patches. The easiest for me would be to merge
them as PRs into my github tree, but I'm happy to keep applying
them manually if it's more convenient for you.

Logs:

SeaBIOS (version 1.8.2-20181014_101610-google)
Total RAM Size = 0x0000000400000000 = 16384 MiB
CPUs found: 16     Max CPUs supported: 16
found virtio-scsi at 0:3
virtio-scsi vendor='Google' product='PersistentDisk' rev='1' type=0
removable=0
virtio-scsi blksize=512 sectors=20971520 = 10240 MiB
drive 0x000f2c00: PCHS=0/0/0 translation=lba LCHS=1024/255/63 s=20971520
Booting from Hard Disk 0...
>> OpenBSD/amd64 BOOT 3.41
boot> b bsd.anton

[ using 1964296 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2018 OpenBSD. All rights reserved.
https://www.OpenBSD.org

OpenBSD 6.4-current (SYZKALLER.MP) #0: Sat Dec  1 10:27:33 PST 2018
    [hidden email]
:/home/syzkaller/s/src/sys/arch/amd64/compile/SYZKALLER.MP
real mem = 17163079680 (16367MB)
avail mem = 16632180736 (15861MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xbffffb80 (28 entries)
bios0: vendor Google version "Google" date 01/01/2011
bios0: Google Google Compute Engine
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC WAET SRAT
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU @ 2.30GHz, 2300.63 MHz, 06-3f-00
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 989MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.74 MHz, 06-3f-00
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.75 MHz, 06-3f-00
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.74 MHz, 06-3f-00
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 8 (application processor)
cpu4: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.74 MHz, 06-3f-00
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 0, core 4, package 0
cpu5 at mainbus0: apid 10 (application processor)
cpu5: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.76 MHz, 06-3f-00
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 0, core 5, package 0
cpu6 at mainbus0: apid 12 (application processor)
cpu6: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.73 MHz, 06-3f-00
cpu6:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu6: 256KB 64b/line 8-way L2 cache
cpu6: smt 0, core 6, package 0
cpu7 at mainbus0: apid 14 (application processor)
cpu7: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.74 MHz, 06-3f-00
cpu7:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu7: 256KB 64b/line 8-way L2 cache
cpu7: smt 0, core 7, package 0
cpu8 at mainbus0: apid 1 (application processor)
cpu8: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.76 MHz, 06-3f-00
cpu8:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu8: 256KB 64b/line 8-way L2 cache
cpu8: smt 1, core 0, package 0
cpu9 at mainbus0: apid 3 (application processor)
cpu9: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.74 MHz, 06-3f-00
cpu9:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu9: 256KB 64b/line 8-way L2 cache
cpu9: smt 1, core 1, package 0
cpu10 at mainbus0: apid 5 (application processor)
cpu10: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.75 MHz, 06-3f-00
cpu10:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu10: 256KB 64b/line 8-way L2 cache
cpu10: smt 1, core 2, package 0
cpu11 at mainbus0: apid 7 (application processor)
cpu11: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.75 MHz, 06-3f-00
cpu11:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu11: 256KB 64b/line 8-way L2 cache
cpu11: smt 1, core 3, package 0
cpu12 at mainbus0: apid 9 (application processor)
cpu12: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.76 MHz, 06-3f-00
cpu12:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu12: 256KB 64b/line 8-way L2 cache
cpu12: smt 1, core 4, package 0
cpu13 at mainbus0: apid 11 (application processor)
cpu13: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.76 MHz, 06-3f-00
cpu13:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu13: 256KB 64b/line 8-way L2 cache
cpu13: smt 1, core 5, package 0
cpu14 at mainbus0: apid 13 (application processor)
cpu14: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.76 MHz, 06-3f-00
cpu14:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu14: 256KB 64b/line 8-way L2 cache
cpu14: smt 1, core 6, package 0
cpu15 at mainbus0: apid 15 (application processor)
cpu15: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.76 MHz, 06-3f-00
cpu15:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu15: 256KB 64b/line 8-way L2 cache
cpu15: smt 1, core 7, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
acpicpu4 at acpi0: C1(@1 halt!)
acpicpu5 at acpi0: C1(@1 halt!)
acpicpu6 at acpi0: C1(@1 halt!)
acpicpu7 at acpi0: C1(@1 halt!)
acpicpu8 at acpi0: C1(@1 halt!)
acpicpu9 at acpi0: C1(@1 halt!)
acpicpu10 at acpi0: C1(@1 halt!)
acpicpu11 at acpi0: C1(@1 halt!)
acpicpu12 at acpi0: C1(@1 halt!)
acpicpu13 at acpi0: C1(@1 halt!)
acpicpu14 at acpi0: C1(@1 halt!)
acpicpu15 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: _OSC failed
acpicmos0 at acpi0
"QEMU0001" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371AB PIIX4 ISA" rev 0x03
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: SMBus
disabled
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio SCSI" rev 0x00
vioscsi0 at virtio0: qsize 8192
scsibus1 at vioscsi0: 253 targets
sd0 at scsibus1 targ 1 lun 0: <Google, PersistentDisk, 1> SCSI4 0/direct
fixed serial.Google_PersistentDisk_
sd0: 10240MB, 512 bytes/sector, 20971520 sectors, thin
virtio0: msix shared
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio1: address 42:01:0a:80:0a:1d
virtio1: msix per-VQ
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0 mux 1
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation)
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (96b3ced08079998e.a) swap on sd0b dump on sd0b

SeaBIOS (version 1.8.2-20181014_101610-google)
Total RAM Size = 0x0000000400000000 = 16384 MiB
CPUs found: 16     Max CPUs supported: 16
found virtio-scsi at 0:3
virtio-scsi vendor='Google' product='PersistentDisk' rev='1' type=0
removable=0
virtio-scsi blksize=512 sectors=20971520 = 10240 MiB
drive 0x000f2c00: PCHS=0/0/0 translation=lba LCHS=1024/255/63 s=20971520
Booting from Hard Disk 0...
>> OpenBSD/amd64 BOOT 3.41
boot>

[ using 2125472 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2018 OpenBSD. All rights reserved.
https://www.OpenBSD.org

OpenBSD 6.4-current (GENERIC.MP) #479: Tue Nov 27 01:23:55 MST 2018
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 17163079680 (16367MB)
avail mem = 16633638912 (15863MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.4 @ 0xbffffb80 (28 entries)
bios0: vendor Google version "Google" date 01/01/2011
bios0: Google Google Compute Engine
acpi0 at bios0: rev 0
acpi0: sleep states S3 S4 S5
acpi0: tables DSDT FACP SSDT APIC WAET SRAT
acpi0: wakeup devices
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: Intel(R) Xeon(R) CPU @ 2.30GHz, 2300.80 MHz, 06-3f-00
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu0: 256KB 64b/line 8-way L2 cache
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 990MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.89 MHz, 06-3f-00
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu1: 256KB 64b/line 8-way L2 cache
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 4 (application processor)
cpu2: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.91 MHz, 06-3f-00
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu2: 256KB 64b/line 8-way L2 cache
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 6 (application processor)
cpu3: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.91 MHz, 06-3f-00
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu3: 256KB 64b/line 8-way L2 cache
cpu3: smt 0, core 3, package 0
cpu4 at mainbus0: apid 8 (application processor)
cpu4: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.90 MHz, 06-3f-00
cpu4:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu4: 256KB 64b/line 8-way L2 cache
cpu4: smt 0, core 4, package 0
cpu5 at mainbus0: apid 10 (application processor)
cpu5: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.89 MHz, 06-3f-00
cpu5:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu5: 256KB 64b/line 8-way L2 cache
cpu5: smt 0, core 5, package 0
cpu6 at mainbus0: apid 12 (application processor)
cpu6: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.89 MHz, 06-3f-00
cpu6:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu6: 256KB 64b/line 8-way L2 cache
cpu6: smt 0, core 6, package 0
cpu7 at mainbus0: apid 14 (application processor)
cpu7: Intel(R) Xeon(R) CPU @ 2.30GHz, 2277.00 MHz, 06-3f-00
cpu7:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu7: 256KB 64b/line 8-way L2 cache
cpu7: smt 0, core 7, package 0
cpu8 at mainbus0: apid 1 (application processor)
cpu8: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.92 MHz, 06-3f-00
cpu8:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu8: 256KB 64b/line 8-way L2 cache
cpu8: smt 1, core 0, package 0
cpu9 at mainbus0: apid 3 (application processor)
cpu9: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.93 MHz, 06-3f-00
cpu9:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu9: 256KB 64b/line 8-way L2 cache
cpu9: smt 1, core 1, package 0
cpu10 at mainbus0: apid 5 (application processor)
cpu10: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.90 MHz, 06-3f-00
cpu10:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu10: 256KB 64b/line 8-way L2 cache
cpu10: smt 1, core 2, package 0
cpu11 at mainbus0: apid 7 (application processor)
cpu11: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.93 MHz, 06-3f-00
cpu11:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu11: 256KB 64b/line 8-way L2 cache
cpu11: smt 1, core 3, package 0
cpu12 at mainbus0: apid 9 (application processor)
cpu12: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.89 MHz, 06-3f-00
cpu12:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu12: 256KB 64b/line 8-way L2 cache
cpu12: smt 1, core 4, package 0
cpu13 at mainbus0: apid 11 (application processor)
cpu13: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.93 MHz, 06-3f-00
cpu13:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu13: 256KB 64b/line 8-way L2 cache
cpu13: smt 1, core 5, package 0
cpu14 at mainbus0: apid 13 (application processor)
cpu14: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.92 MHz, 06-3f-00
cpu14:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu14: 256KB 64b/line 8-way L2 cache
cpu14: smt 1, core 6, package 0
cpu15 at mainbus0: apid 15 (application processor)
cpu15: Intel(R) Xeon(R) CPU @ 2.30GHz, 2276.91 MHz, 06-3f-00
cpu15:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,SS,HTT,SSE3,PCLMUL,VMX,SSSE3,FMA3,CX16,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,RDRAND,HV,NXE,PAGE1GB,RDTSCP,LONG,LAHF,ABM,ITSC,FSGSBASE,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,ARAT,XSAVEOPT,MELTDOWN
cpu15: 256KB 64b/line 8-way L2 cache
cpu15: smt 1, core 7, package 0
ioapic0 at mainbus0: apid 0 pa 0xfec00000, version 11, 24 pins
acpiprt0 at acpi0: bus 0 (PCI0)
acpicpu0 at acpi0: C1(@1 halt!)
acpicpu1 at acpi0: C1(@1 halt!)
acpicpu2 at acpi0: C1(@1 halt!)
acpicpu3 at acpi0: C1(@1 halt!)
acpicpu4 at acpi0: C1(@1 halt!)
acpicpu5 at acpi0: C1(@1 halt!)
acpicpu6 at acpi0: C1(@1 halt!)
acpicpu7 at acpi0: C1(@1 halt!)
acpicpu8 at acpi0: C1(@1 halt!)
acpicpu9 at acpi0: C1(@1 halt!)
acpicpu10 at acpi0: C1(@1 halt!)
acpicpu11 at acpi0: C1(@1 halt!)
acpicpu12 at acpi0: C1(@1 halt!)
acpicpu13 at acpi0: C1(@1 halt!)
acpicpu14 at acpi0: C1(@1 halt!)
acpicpu15 at acpi0: C1(@1 halt!)
"ACPI0006" at acpi0 not configured
acpipci0 at acpi0 PCI0: _OSC failed
acpicmos0 at acpi0
"QEMU0001" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
"ACPI0007" at acpi0 not configured
pvbus0 at mainbus0: KVM
pvclock0 at pvbus0
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "Intel 82441FX" rev 0x02
pcib0 at pci0 dev 1 function 0 "Intel 82371AB PIIX4 ISA" rev 0x03
piixpm0 at pci0 dev 1 function 3 "Intel 82371AB Power" rev 0x03: SMBus
disabled
virtio0 at pci0 dev 3 function 0 "Qumranet Virtio SCSI" rev 0x00
vioscsi0 at virtio0: qsize 8192
scsibus1 at vioscsi0: 253 targets
sd0 at scsibus1 targ 1 lun 0: <Google, PersistentDisk, 1> SCSI4 0/direct
fixed serial.Google_PersistentDisk_
sd0: 10240MB, 512 bytes/sector, 20971520 sectors, thin
virtio0: msix shared
virtio1 at pci0 dev 4 function 0 "Qumranet Virtio Network" rev 0x00
vio0 at virtio1: address 42:01:0a:80:0a:1d
virtio1: msix per-VQ
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com0: console
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com2 at isa0 port 0x3e8/8 irq 5: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pckbd0 at pckbc0 (kbd slot)
wskbd0 at pckbd0 mux 1
pms0 at pckbc0 (aux slot)
wsmouse0 at pms0 mux 0
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT (using slow L1TF mitigation)
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
root on sd0a (96b3ced08079998e.a) swap on sd0b dump on sd0b
Automatic boot in progress: starting file system checks.
/dev/sd0a (96b3ced08079998e.a): file system is clean; not checking
setting tty flags
pf enabled
hw.smt: 0 -> 1
starting network
vio0: bound to 10.128.10.29 from 169.254.169.254 (42:01:0a:80:00:01)
reordering libraries: done.
starting early daemons: syslogd pflogd ntpd.
starting RPC daemons:.
savecore: no core dump
checking quotas: done.
clearing /tmp
kern.securelevel: 0 -> 1
creating runtime link editor directory cache.
preserving editor files.
starting network daemons: sshd.
+ echo starting syz-ci
starting syz-ci
+ fsck -y /dev/sd1a
Can't open /dev/rsd1a: Device not configured
+ mount /syzkaller
mount_ffs: /dev/sd1a on /syzkaller: Device not configured
starting local daemons: cron.
Sat Dec  1 10:30:19 PST 2018

OpenBSD/amd64 (ci-openbsd.syzkaller) (tty00)

login:
Reply | Threaded
Open this post in threaded view
|

Re: option kcov + GENERIC.MP -> silent crash

Anton Lindqvist-2
In reply to this post by Anton Lindqvist-2
On Sat, Dec 01, 2018 at 04:34:57PM +0100, Anton Lindqvist wrote:

> On Tue, Nov 27, 2018 at 05:52:15PM -0800, Greg Steuck wrote:
> > I booted the patched kernel and it seems to have gone farther and I believe
> > reached init before crashing.
>
> By performing a semi-automated bisect I was able to identify the source
> files that are incompatible with tracing. Common for all source files
> seems to be that they define routines called early on in the boot
> process before curcpu() is usable.
>
> I do not have any plans on committing the diff below but please give it
> a try. Instead, I'm working on extending the files.conf(5) grammar in
> order to infer a different make target for the files.
>
> Index: arch/amd64/conf/Makefile.amd64
> ===================================================================
> RCS file: /cvs/src/sys/arch/amd64/conf/Makefile.amd64,v
> retrieving revision 1.106
> diff -u -p -r1.106 Makefile.amd64
> --- arch/amd64/conf/Makefile.amd64 30 Oct 2018 11:08:30 -0000 1.106
> +++ arch/amd64/conf/Makefile.amd64 1 Dec 2018 15:32:58 -0000
> @@ -151,7 +151,31 @@ vers.o: ${SYSTEM_DEP:Ngap.o}
>   ${CC} ${CFLAGS} ${CPPFLAGS} ${PROF} -c vers.c
>  
>  .if ${SYSTEM_OBJ:Mkcov.o} && ${COMPILER_VERSION:Mclang}
> +amd64_mem.o: $S/arch/amd64/amd64/amd64_mem.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +cpu.o: $S/arch/amd64/amd64/cpu.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +fpu.o: $S/arch/amd64/amd64/fpu.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +gdt.o: $S/arch/amd64/amd64/gdt.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +intr.o: $S/arch/amd64/amd64/intr.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +lapic.o: $S/arch/amd64/amd64/lapic.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +machdep.o: $S/arch/amd64/amd64/machdep.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +tsc.o: $S/arch/amd64/amd64/tsc.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
>  kcov.o: $S/dev/kcov.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +pvbus.o: $S/dev/pv/pvbus.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +kern_lock.o: $S/kern/kern_lock.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +kern_sched.o: $S/kern/kern_sched.c
> + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> +kern_tc.o: $S/kern/kern_tc.c
>   ${NORMAL_C} -fno-sanitize-coverage=trace-pc
>  .endif
>  

Here's a new diff taking a different approach. Keeping tracing off until
all secondary CPUs have booted solves the issue of accessing curcpu()
too early. Another issue was then discovered, curproc can be NULL before
the idle thread tied the current CPU has started. Currently running with
this diff applied on my laptop (MP) and positive results from Greg. The
diff will be further exercised in the actual syzkaller setup before
committing.

Comments? OK?

diff --git sys/dev/kcov.c sys/dev/kcov.c
index 8e36bc8b8ef..c97aae4ed5d 100644
--- sys/dev/kcov.c
+++ sys/dev/kcov.c
@@ -58,6 +58,8 @@ static inline int inintr(void);
 
 TAILQ_HEAD(, kcov_dev) kd_list = TAILQ_HEAD_INITIALIZER(kd_list);
 
+int kcov_cold = 1;
+
 #ifdef KCOV_DEBUG
 int kcov_debug = 1;
 #endif
@@ -76,19 +78,31 @@ int kcov_debug = 1;
 void
 __sanitizer_cov_trace_pc(void)
 {
- extern int cold;
  struct kcov_dev *kd;
+ struct proc *p;
  uint64_t idx;
 
- /* Do not trace during boot. */
- if (cold)
+ /*
+ * Do not trace before all secondary CPUs have booted.
+ * Accessing the current CPU during boot causes a subtle crash since its
+ * GSBASE register has not yet been written.
+ */
+ if (kcov_cold)
  return;
 
  /* Do not trace in interrupts to prevent noisy coverage. */
  if (inintr())
  return;
 
- kd = curproc->p_kd;
+ /*
+ * Protect against when the idle thread for the current CPU has not yet
+ * started and curproc is absent.
+ */
+ p = curproc;
+ if (p == NULL)
+ return;
+
+ kd = p->p_kd;
  if (kd == NULL || kd->kd_mode != KCOV_MODE_TRACE_PC)
  return;
 
@@ -226,6 +240,12 @@ kcov_exit(struct proc *p)
  p->p_kd = NULL;
 }
 
+void
+kcov_init(void)
+{
+ kcov_cold = 0;
+}
+
 struct kcov_dev *
 kd_lookup(int unit)
 {
diff --git sys/kern/init_main.c sys/kern/init_main.c
index 91070090bb1..25b71fd89ce 100644
--- sys/kern/init_main.c
+++ sys/kern/init_main.c
@@ -103,6 +103,11 @@ extern void nfs_init(void);
 #include "vscsi.h"
 #include "softraid.h"
 
+#include "kcov.h"
+#if NKCOV > 0
+#include <sys/kcov.h>
+#endif
+
 const char copyright[] =
 "Copyright (c) 1982, 1986, 1989, 1991, 1993\n"
 "\tThe Regents of the University of California.  All rights reserved.\n"
@@ -555,6 +560,10 @@ main(void *framep)
 
  config_process_deferred_mountroot();
 
+#if NKCOV > 0
+ kcov_init();
+#endif
+
  /*
  * Okay, now we can let init(8) exec!  It's off to userland!
  */
diff --git sys/sys/kcov.h sys/sys/kcov.h
index 752b290e615..3ff32b330e4 100644
--- sys/sys/kcov.h
+++ sys/sys/kcov.h
@@ -30,6 +30,7 @@
 #define KCOV_BUF_MAX_NMEMB (256 << 10)
 
 void kcov_exit(struct proc *);
+void kcov_init(void);
 
 #endif /* _KERNEL */
 

Reply | Threaded
Open this post in threaded view
|

Re: option kcov + GENERIC.MP -> silent crash

Greg Steuck
> Here's a new diff taking a different approach. Keeping tracing off until
> all secondary CPUs have booted solves the issue of accessing curcpu()
> too early. Another issue was then discovered, curproc can be NULL before
> the idle thread tied the current CPU has started. Currently running with
> this diff applied on my laptop (MP) and positive results from Greg. The
> diff will be further exercised in the actual syzkaller setup before
> committing.

Thanks Anton. This diff is running now on
https://syzkaller.appspot.com/#openbsd as
openbsd/ci-openbsd-multicore. Looking great so far.

Reply | Threaded
Open this post in threaded view
|

Re: option kcov + GENERIC.MP -> silent crash

Anton Lindqvist-2
In reply to this post by Anton Lindqvist-2
On Wed, Dec 05, 2018 at 10:03:38PM +0100, Anton Lindqvist wrote:

> On Sat, Dec 01, 2018 at 04:34:57PM +0100, Anton Lindqvist wrote:
> > On Tue, Nov 27, 2018 at 05:52:15PM -0800, Greg Steuck wrote:
> > > I booted the patched kernel and it seems to have gone farther and I believe
> > > reached init before crashing.
> >
> > By performing a semi-automated bisect I was able to identify the source
> > files that are incompatible with tracing. Common for all source files
> > seems to be that they define routines called early on in the boot
> > process before curcpu() is usable.
> >
> > I do not have any plans on committing the diff below but please give it
> > a try. Instead, I'm working on extending the files.conf(5) grammar in
> > order to infer a different make target for the files.
> >
> > Index: arch/amd64/conf/Makefile.amd64
> > ===================================================================
> > RCS file: /cvs/src/sys/arch/amd64/conf/Makefile.amd64,v
> > retrieving revision 1.106
> > diff -u -p -r1.106 Makefile.amd64
> > --- arch/amd64/conf/Makefile.amd64 30 Oct 2018 11:08:30 -0000 1.106
> > +++ arch/amd64/conf/Makefile.amd64 1 Dec 2018 15:32:58 -0000
> > @@ -151,7 +151,31 @@ vers.o: ${SYSTEM_DEP:Ngap.o}
> >   ${CC} ${CFLAGS} ${CPPFLAGS} ${PROF} -c vers.c
> >  
> >  .if ${SYSTEM_OBJ:Mkcov.o} && ${COMPILER_VERSION:Mclang}
> > +amd64_mem.o: $S/arch/amd64/amd64/amd64_mem.c
> > + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> > +cpu.o: $S/arch/amd64/amd64/cpu.c
> > + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> > +fpu.o: $S/arch/amd64/amd64/fpu.c
> > + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> > +gdt.o: $S/arch/amd64/amd64/gdt.c
> > + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> > +intr.o: $S/arch/amd64/amd64/intr.c
> > + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> > +lapic.o: $S/arch/amd64/amd64/lapic.c
> > + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> > +machdep.o: $S/arch/amd64/amd64/machdep.c
> > + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> > +tsc.o: $S/arch/amd64/amd64/tsc.c
> > + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> >  kcov.o: $S/dev/kcov.c
> > + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> > +pvbus.o: $S/dev/pv/pvbus.c
> > + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> > +kern_lock.o: $S/kern/kern_lock.c
> > + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> > +kern_sched.o: $S/kern/kern_sched.c
> > + ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> > +kern_tc.o: $S/kern/kern_tc.c
> >   ${NORMAL_C} -fno-sanitize-coverage=trace-pc
> >  .endif
> >  
>
> Here's a new diff taking a different approach. Keeping tracing off until
> all secondary CPUs have booted solves the issue of accessing curcpu()
> too early. Another issue was then discovered, curproc can be NULL before
> the idle thread tied the current CPU has started. Currently running with
> this diff applied on my laptop (MP) and positive results from Greg. The
> diff will be further exercised in the actual syzkaller setup before
> committing.

Yet another iteration and hopefully the last one. Idea from visa@ to
delay tracing until /dev/kcov has been successfully opened at least
once. At this point, accessing curcpu() is safe. Sort of a hack but it
removes the need to hook into init_main.c which is favorable.

Comments? OK?

Index: dev/kcov.c
===================================================================
RCS file: /cvs/src/sys/dev/kcov.c,v
retrieving revision 1.4
diff -u -p -r1.4 kcov.c
--- dev/kcov.c 27 Aug 2018 15:57:39 -0000 1.4
+++ dev/kcov.c 8 Dec 2018 16:40:41 -0000
@@ -58,6 +58,8 @@ static inline int inintr(void);
 
 TAILQ_HEAD(, kcov_dev) kd_list = TAILQ_HEAD_INITIALIZER(kd_list);
 
+int kcov_cold = 1;
+
 #ifdef KCOV_DEBUG
 int kcov_debug = 1;
 #endif
@@ -76,12 +78,15 @@ int kcov_debug = 1;
 void
 __sanitizer_cov_trace_pc(void)
 {
- extern int cold;
  struct kcov_dev *kd;
  uint64_t idx;
 
- /* Do not trace during boot. */
- if (cold)
+ /*
+ * Do not trace before kcovopen() has been called at least once.
+ * At this point, all secondary CPUs have booted and accessing curcpu()
+ * is safe.
+ */
+ if (kcov_cold)
  return;
 
  /* Do not trace in interrupts to prevent noisy coverage. */
@@ -111,6 +116,9 @@ kcovopen(dev_t dev, int flag, int mode,
 
  if (kd_lookup(minor(dev)) != NULL)
  return (EBUSY);
+
+ if (kcov_cold)
+ kcov_cold = 0;
 
  DPRINTF("%s: unit=%d\n", __func__, minor(dev));