6.6-current stutters after heavy disk loads

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

6.6-current stutters after heavy disk loads

Martin Got
Description

After heavy disk loads (copying large files, dd urandom data to a disk) especially USB3.0 attached. OpenBSD 6.6-current stutters every 1-3s even after copying is completed till next reboot. Disk was encrypted by bioctl with -r64 option. It looks like system wide disk I/O problem, maybe USB related.

How to reproduce

Connect external HDD and dd if=/dev/urandom of=/dev/rsd0c bs=1M or copy large file with some (possible heavy) CPU load by any background process. During the urandom writing or copying a large file procedure 6.6-current stutters. It stutters anyway even all copy operations ended, until reboot. From user's side it looks as missed symbols during typing, any realtime traffic like VoIP RTP interrupts every stutter.

Some tests detail have been described here:
https://marc.info/?l=openbsd-misc&m=158403485529894&w=2

dmesg of affected machine has been sent to [hidden email]

Martin
Reply | Threaded
Open this post in threaded view
|

Re: 6.6-current stutters after heavy disk loads

Stuart Henderson
On 2020/03/13 17:44, Martin wrote:

> Description
>
> After heavy disk loads (copying large files, dd urandom data to a disk) especially USB3.0 attached. OpenBSD 6.6-current stutters every 1-3s even after copying is completed till next reboot. Disk was encrypted by bioctl with -r64 option. It looks like system wide disk I/O problem, maybe USB related.
>
> How to reproduce
>
> Connect external HDD and dd if=/dev/urandom of=/dev/rsd0c bs=1M or copy large file with some (possible heavy) CPU load by any background process. During the urandom writing or copying a large file procedure 6.6-current stutters. It stutters anyway even all copy operations ended, until reboot. From user's side it looks as missed symbols during typing, any realtime traffic like VoIP RTP interrupts every stutter.
>
> Some tests detail have been described here:
> https://marc.info/?l=openbsd-misc&m=158403485529894&w=2
>
> dmesg of affected machine has been sent to [hidden email]

It's unhelpful to scatter information amongst about 5 different mails,
one of which is to an address that is not visible to normal bugs@ readers
(and is a pain to locate for developers). Since you mention you have
tweaked -r for bioctl softraid, I bet there are other things you tweaked
too.

Please write everything up that you can about your configuration into a
single self-contained mail.

Note that USB support is not a strong point of OpenBSD.

Reply | Threaded
Open this post in threaded view
|

Re: 6.6-current stutters after heavy disk loads

Martin Got
I reinstall 6.6 from scratch to test the issue without any possible tweaks affected. Stutters didn't appear on 6.4 and 6.5-current at all.

Description:

1. Fresh install 6.6 from cd using bioctl full disk enc. with default rounds for password onto AMD SOC based PC.
  1.1 No visible stutters present after reinstall (apm -A and CPU in Low freq. mode).
2. Emulate heavy disk load by reindexing 150+Gb database without any USB connected devices (apm -A CPU in High freq.mode).
3. While indexing stutters are present and its behavior is the same as on previously tested 6.6-current.
4. Run: First stutter long is 1.2s -> next stutter appears in 2s and long 1.2s -> next stutter after 1s with 1.2s long -> next stutter after 3s with 1.2s long -> return of beginning of cycle (apm -A CPU in High freq. mode).
5. Re-indexing is stopped completely: Stutters are still present till the next reboot even without any HDD activity or any CPU load (apm -A CPU in Low freq. mode).
First stutter long is 0.7s -> next stutter appears in 2s and long .7s -> next stutter after 1s with .7s long -> next stutter after 3s with .7s long -> return of beginning of cycle
6. Stutters can't be determined by top command. But can be seen when user input text from keyboard, VoIP RTP traffic, or mpv music or video playing etc.

It seems the same issue has been reported by "Peter J. Philipp" <pjp () centroid ! eu> in reply to my question in misc@.
He has significantly different HW configuration and different services run but 6.6 stutters the same way.

Peter wrote:
"I get that too, it started when I moved cyrus imapd to my local system a year
or more ago.  I've learned to live with it.  To elaborate, something cyrus
imapd does is when a mail comes in via fetchmail that mplayer which plays
flac's or streams from an icecast starts stuttering.  The combination of
cyrus, thunderbird, and fetchmail may be to blame too, dunno.  It may be
disk related?  I have a Samsung SSD.  My workstation is a 2014 Xeon E3-1275
with 32 GB RAM.

pkg_scripts="isc_named cyrus_imapd saslauthd"

along with apmd -A, vmd, sndiod -f rsnd/0 -f rsnd/1, portmap, nfsd, mountd,
unwind, rad, tftpd, xenodm, dhcpd

It's just the workstation I power up every day when I'm home."

In most cases of testing, it looks like some system process(es), possible disk I/O procedures case bumbles.

How to reproduce:

Boot 6.6 from internal HDD and dd if=/dev/urandom of=/dev/the_same_disk_test_partion bs=1M or copy large file with some (possible heavy) CPU load by any background process. Or reindex any *big* database (it can produce heavy CPU load with HDD writing simultaneously). Stutters still present after HDD & CPU load removed till next reboot.

Testing machine:

OpenBSD 6.6-stable (GENERIC.MP) #1: Tue Mar 7 14:27:12 UTC 2020
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 16554262528 (15787MB)
avail mem = 16044179456 (15300MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 2.7 @ 0xe0840 (39 entries)
bios0: vendor Phoenix Technologies Ltd. version "FP4.3.0.0.312.13 X64" date 03/02/2018
bios0: CompuLab fit-PC4
acpi0 at bios0: ACPI 5.0
acpi0: sleep states S0 S3 S4 S5
acpi0: tables DSDT FACP HPET APIC MCFG FPDT UEFI POAT BATB SSDT SSDT UEFI
acpi0: wakeup devices GPP0(S4) GPP1(S4) GPP2(S4) GPP3(S4) GFX_(S4) XHC0(S4) OHC1(S4) EHC1(S4) OHC2(S4) EHC2(S4) SBAZ(S4) UAR1(S3)
acpitimer0 at acpi0: 3579545 Hz, 32 bits
acpihpet0 at acpi0: 14318180 Hz
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD GX-420CA SOC with Radeon(tm) HD Graphics, 1996.51 MHz, 16-00-01
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu0: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu0: smt 0, core 0, package 0
mtrr: Pentium Pro MTRR support, 8 var ranges, 88 fixed ranges
cpu0: apic clock running at 99MHz
cpu0: mwait min=64, max=64, IBE
cpu1 at mainbus0: apid 1 (application processor)
cpu1: AMD GX-420CA SOC with Radeon(tm) HD Graphics, 1996.26 MHz, 16-00-01
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu1: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu1: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu1: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu1: smt 0, core 1, package 0
cpu2 at mainbus0: apid 2 (application processor)
cpu2: AMD GX-420CA SOC with Radeon(tm) HD Graphics, 1996.26 MHz, 16-00-01
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu2: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu2: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu2: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu2: smt 0, core 2, package 0
cpu3 at mainbus0: apid 3 (application processor)
cpu3: AMD GX-420CA SOC with Radeon(tm) HD Graphics, 1996.26 MHz, 16-00-01
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,PCLMUL,MWAIT,SSSE3,CX16,SSE4.1,SSE4.2,MOVBE,POPCNT,AES,XSAVE,AVX,F16C,NXE,MMXX,FFXSR,PAGE1GB,RDTSCP,LONG,LAHF,CMPLEG,SVM,EAPICSP,AMCR8,ABM,SSE4A,MASSE,3DNOWP,OSVW,IBS,SKINIT,TOPEXT,DBKP,PCTRL3,ITSC,BMI1,XSAVEOPT
cpu3: 32KB 64b/line 2-way I-cache, 32KB 64b/line 8-way D-cache, 2MB 64b/line 16-way L2 cache
cpu3: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu3: DTLB 40 4KB entries fully associative, 8 4MB entries fully associative
cpu3: smt 0, core 3, package 0
ioapic0 at mainbus0: apid 4 pa 0xfec00000, version 21, 24 pins, remapped
ioapic1 at mainbus0: apid 5 pa 0xfec01000, version 21, 32 pins, remapped
acpimcfg0 at acpi0
acpimcfg0: addr 0xf8000000, bus 0-63
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 2 (GPP0)
acpiprt2 at acpi0: bus -1 (GPP1)
acpiprt3 at acpi0: bus 4 (GPP2)
acpiprt4 at acpi0: bus 5 (GPP3)
acpiprt5 at acpi0: bus 1 (GFX_)
acpicpu0 at acpi0: C2(0@400 io@0x841), C1(@1 halt!), PSS
acpicpu1 at acpi0: C2(0@400 io@0x841), C1(@1 halt!), PSS
acpicpu2 at acpi0: C2(0@400 io@0x841), C1(@1 halt!), PSS
acpicpu3 at acpi0: C2(0@400 io@0x841), C1(@1 halt!), PSS
acpibtn0 at acpi0: PWRB
acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
acpicmos0 at acpi0
"PNP0A05" at acpi0 not configured
acpivideo0 at acpi0: VGA_
cpu0: 1996 MHz: speeds: 2000 1800 1600 1200 1000 800 MHz
pci0 at mainbus0 bus 0
pchb0 at pci0 dev 0 function 0 "AMD AMD64 16h Host" rev 0x00
radeondrm0 at pci0 dev 1 function 0 "ATI Kabini" rev 0x00
drm0 at radeondrm0
radeondrm0: msi
azalia0 at pci0 dev 1 function 1 "ATI Radeon HD Audio" rev 0x00: msi
azalia0: no supported codecs
pchb1 at pci0 dev 2 function 0 vendor "AMD", unknown product 0x1538 rev 0x00
ppb0 at pci0 dev 2 function 1 "AMD AMD64 16h PCIE" rev 0x00: msi
pci1 at ppb0 bus 1
ppb1 at pci0 dev 2 function 2 "AMD AMD64 16h PCIE" rev 0x00: msi
pci2 at ppb1 bus 2
ppb2 at pci0 dev 2 function 4 "AMD AMD64 16h PCIE" rev 0x00: msi
pci3 at ppb2 bus 4
em0 at pci3 dev 0 function 0 "Intel I211" rev 0x03: msi, address 00:01:c0:28:6b:12
ppb3 at pci0 dev 2 function 5 "AMD AMD64 16h PCIE" rev 0x00: msi
pci4 at ppb3 bus 5
em1 at pci4 dev 0 function 0 "Intel I211" rev 0x03: msi, address 00:01:c0:28:6b:15
xhci0 at pci0 dev 16 function 0 "AMD Bolton xHCI" rev 0x01: msi, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "AMD xHCI root hub" rev 3.00/1.00 addr 1
ahci0 at pci0 dev 17 function 0 "AMD Hudson-2 SATA" rev 0x40: msi, AHCI 1.3
ahci0: port 0: 6.0Gb/s
scsibus1 at ahci0: 32 targets
sd0 at scsibus1 targ 0 lun 0: <ATA, ST1000LM049-2GH1, SDM1> naa.5000c500b8d81a04
sd0: 953869MB, 512 bytes/sector, 1953525168 sectors, thin
ohci0 at pci0 dev 18 function 0 "AMD Hudson-2 USB" rev 0x39: apic 4 int 18, version 1.0, legacy support
ehci0 at pci0 dev 18 function 2 "AMD Hudson-2 USB2" rev 0x39: apic 4 int 17
usb1 at ehci0: USB revision 2.0
uhub1 at usb1 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
ohci1 at pci0 dev 19 function 0 "AMD Hudson-2 USB" rev 0x39: apic 4 int 18, version 1.0, legacy support
ehci1 at pci0 dev 19 function 2 "AMD Hudson-2 USB2" rev 0x39: apic 4 int 17
usb2 at ehci1: USB revision 2.0
uhub2 at usb2 configuration 1 interface 0 "AMD EHCI root hub" rev 2.00/1.00 addr 1
"AMD Hudson-2 SMBus" rev 0x3a at pci0 dev 20 function 0 not configured
azalia1 at pci0 dev 20 function 2 "AMD Hudson-2 HD Audio" rev 0x02: apic 4 int 16
azalia1: codecs: Realtek ALC888
audio0 at azalia1
pcib0 at pci0 dev 20 function 3 "AMD Hudson-2 LPC" rev 0x11
sdhc0 at pci0 dev 20 function 7 "AMD Bolton SD/MMC" rev 0x01: apic 4 int 16
sdhc0: SDHC 2.0, 50 MHz base clock
sdmmc0 at sdhc0: 4-bit, sd high-speed, mmc high-speed, dma
pchb2 at pci0 dev 24 function 0 "AMD AMD64 16h Link Cfg" rev 0x00
pchb3 at pci0 dev 24 function 1 "AMD AMD64 16h Address Map" rev 0x00
pchb4 at pci0 dev 24 function 2 "AMD AMD64 16h DRAM Cfg" rev 0x00
pchb5 at pci0 dev 24 function 3 "AMD AMD64 16h Misc Cfg" rev 0x00
pchb6 at pci0 dev 24 function 4 "AMD AMD64 16h CPU Power" rev 0x00
pchb7 at pci0 dev 24 function 5 vendor "AMD", unknown product 0x1535 rev 0x00
usb3 at ohci0: USB revision 1.0
uhub3 at usb3 configuration 1 interface 0 "AMD OHCI root hub" rev 1.00/1.00 addr 1
usb4 at ohci1: USB revision 1.0
uhub4 at usb4 configuration 1 interface 0 "AMD OHCI root hub" rev 1.00/1.00 addr 1
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5 irq 1 irq 12
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: SVM/RVI
sdmmc0: can't enable card
vscsi0 at root
scsibus2 at vscsi0: 256 targets
softraid0 at root
scsibus3 at softraid0: 256 targets
sd1 at scsibus3 targ 1 lun 0: <OPENBSD, SR CRYPTO, 006>
sd1: 953866MB, 512 bytes/sector, 1953519473 sectors
root on sd1a (f4ead2b60254f2df.a) swap on sd1b dump on sd1b
initializing kernel modesetting (KABINI 0x1002:0x9831 0x1022:0x1234 0x00).
radeondrm0: 1920x1200, 32bpp
wsdisplay0 at radeondrm0 mux 1: console (std, vt100 emulation), using wskbd0
wsdisplay0: screen 1-5 added (std, vt100 emulation)

Martin

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, March 13, 2020 10:12 PM, Stuart Henderson <[hidden email]> wrote:

> On 2020/03/13 17:44, Martin wrote:
>
> > Description
> > After heavy disk loads (copying large files, dd urandom data to a disk) especially USB3.0 attached. OpenBSD 6.6-current stutters every 1-3s even after copying is completed till next reboot. Disk was encrypted by bioctl with -r64 option. It looks like system wide disk I/O problem, maybe USB related.
> > How to reproduce
> > Connect external HDD and dd if=/dev/urandom of=/dev/rsd0c bs=1M or copy large file with some (possible heavy) CPU load by any background process. During the urandom writing or copying a large file procedure 6.6-current stutters. It stutters anyway even all copy operations ended, until reboot. From user's side it looks as missed symbols during typing, any realtime traffic like VoIP RTP interrupts every stutter.
> > Some tests detail have been described here:
> > https://marc.info/?l=openbsd-misc&m=158403485529894&w=2
> > dmesg of affected machine has been sent to [hidden email]
>
> It's unhelpful to scatter information amongst about 5 different mails,
> one of which is to an address that is not visible to normal bugs@ readers
> (and is a pain to locate for developers). Since you mention you have
> tweaked -r for bioctl softraid, I bet there are other things you tweaked
> too.
>
> Please write everything up that you can about your configuration into a
> single self-contained mail.
>
> Note that USB support is not a strong point of OpenBSD.