H5 SOC system instability - panic: amap_pp_adjref: negative reference count

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

H5 SOC system instability - panic: amap_pp_adjref: negative reference count

s_graf
My orangepi pc2 will eventually crash with always the same symptom " panic:
amap_pp_adjref: negative reference count" if pushed with enough work and
network traffic.
I can compile the kernel successfully as long as I stay off the lan during
the processing. I have also tested the system with stress.
I have tested with dwxe and run (WIFI) lan connections with the same crash
results.
I tested with a port build of nano.  I used to test with a pkg_add, but the
packages are currently out of date. Ports is on a separate file system
(/test) so the crashes don't mess up the root directory as much.

It is my hope that someone will take some interest in this problem.  I saw
that it was reported in mid 2018 also.
I have my system set up to do testing and I can compile the kernel and do
whatever I am directed but the I just don't know where to start with the
code.

oppc2obsd# sysctl hw
hw.machine=arm64
hw.model=ARM Cortex-A53 r0p4
hw.ncpu=4
hw.byteorder=1234
hw.pagesize=4096
hw.disknames=sd0:ca2aaaf77eda0cbd,sd1:,sd2:,sd3:,sd4:,sd5:4f70a834aa7afa60
hw.diskcount=6
hw.sensors.sxitemp0.temp0=36.61 degC (CPU)
hw.sensors.sxitemp0.temp1=37.44 degC (GPU)
hw.cpuspeed=1296
hw.setperf=95
hw.product=Xunlong Orange Pi PC 2
hw.physmem=944377856
hw.usermem=944365568
hw.ncpufound=4
hw.allowpowerdown=1
hw.perfpolicy=manual
hw.ncpuonline=4

oppc2obsd# ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 32768
        index 3 priority 0 llprio 3
        groups: lo
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
dwxe0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
        lladdr 02:01:81:8e:44:f6
        index 1 priority 0 llprio 3
        groups: egress
        media: Ethernet 1000baseT full-duplex
        status: active
        inet 192.168.1.28 netmask 0xffffff00 broadcast 192.168.1.255
enc0: flags=0<>
        index 2 priority 0 llprio 3
        groups: enc
        status: active
pflog0: flags=141<UP,RUNNING,PROMISC> mtu 33136
        index 4 priority 0 llprio 3
        groups: pflog

oppc2obsd# df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd0a      6.9G    873M    5.7G    13%    /
/dev/sd0d      6.7G    433M    5.9G     7%    /test

****
The crash:

Making install in tests
 mkdir -p '/usr/ports/pobj/xz-5.2.4/fake-arm64/usr/local/share/doc/xz'
 /usr/ports/pobj/xz-5.2.4/bin/install -c -m 644
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/AUTHORS
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/COPYING
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/COPYING.GPLv2
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/NEWS
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/README
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/THANKS
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/TODO
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/doc/faq.txt
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/doc/history.txt
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/doc/xz-file-format.txt
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/doc/lzma-file-format.txt
'/usr/ports/pobj/xz-5.2.4/fake-arm64/usr/local/share/doc/xz'
 mkdir -p
'/usr/ports/pobj/xz-5.2.4/fake-arm64/usr/local/share/doc/xz/examples'
 /usr/ports/pobj/xz-5.2.4/bin/install -c -m 644
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/doc/examples/00_README.txt
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/doc/examples/01_compress_easy.c
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/doc/examples/02_decompress.c
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/doc/examples/03_compress_custom.c
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/doc/examples/04_compress_easy_mt.c
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/doc/examples/Makefile
'/usr/ports/pobj/xz-5.2.4/fake-arm64/usr/local/share/doc/xz/examples'
 mkdir -p
'/usr/ports/pobj/xz-5.2.4/fake-arm64/usr/local/share/doc/xz/examples_old'
 /usr/ports/pobj/xz-5.2.4/bin/install -c -m 644
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/doc/examples_old/xz_pipe_comp.c
/usr/ports/pobj/xz-5.2.4/xz-5.2.4/doc/examples_old/xz_pipe_decomp.c
'/usr/ports/pobj/xz-5.2.4/fake-arm64/usr/local/share/doc/xz/examples_old'
===>  Building package for xz-5.2.4
Create /usr/ports/packages/aarch64/all/xz-5.2.4.tgz
Creating package xz-5.2.4


panic: amap_pp_adjref: negative reference count
Stopped at      panic+0x148:        TID    PID    UID     PRFLAGS     PFLAGS
C
PU  COMMAND
*116910  76189      0      0x1005          0    3K perl
db_enter() at panic+0x144
panic() at uvm_unmap_detach+0x7c
uvm_unmap_detach() at uvmspace_exec+0x1d0
uvmspace_exec() at sys_execve+0x5a8
sys_execve() at svc_handler+0x264
svc_handler() at do_el0_sync+0x1b0
do_el0_sync() at handle_el0_sync+0x74
https://www.openbsd.org/ddb.html describes the minimum info required in bug
reports.  Insufficient info makes it difficult to find and fix bugs.

ddb{3}> ps
   PID     TID   PPID    UID  S       FLAGS  WAIT          COMMAND
*76189  116910  10639      0  7      0x1005                perl
 10639  496153  25226      0  3        0x83  wait          perl
 25226  410714   9287      0  3    0x10008b  pause         sh
  9287  408804  27550      0  3    0x10008b  pause         make
 27550  143039  18098      0  3    0x10008b  pause         sh
 18098   59524  74326      0  3    0x10008b  pause         make
 74326  323729  59029      0  3    0x10008b  pause         sh
 59029  178400  58990      0  3    0x10008b  pause         make
 58990  457091   9923      0  3    0x10008b  pause         make
  9923  520778  77050      0  3    0x10008b  pause         sh
 77050   55917  43114      0  3    0x10008b  pause         make
 43114  431026  47367      0  3    0x100089  pause         sh
 47367  409167   2319      0  3    0x10008b  pause         sh
  2319  417637  47400      0  3    0x10008b  pause         make
 47400   84721  77653      0  3    0x10008b  pause         sh
 77653  195113  10716      0  3    0x10008b  pause         make
 10716   10839      1      0  3    0x10008b  pause         ksh
 99049  301960      1      0  3    0x100098  poll          cron
  6516  318235      1     99  3    0x100090  poll          sndiod
 89902  407872      1    110  3    0x100090  poll          sndiod
 83044  197006  20229     95  3    0x100092  kqread        smtpd
 26353  437519  20229    103  3    0x100092  kqread        smtpd
 78935  389339  20229     95  3    0x100092  kqread        smtpd
 48322   39591  20229     95  3    0x100092  kqread        smtpd
  4514  347011  20229     95  3    0x100092  kqread        smtpd
 75132  422156  20229     95  3    0x100092  kqread        smtpd
 20229  113058      1      0  3    0x100080  kqread        smtpd
 56663  191266      1      0  3        0x80  select        sshd
 11112  413669      1      0  3    0x100080  poll          ntpd
 83735   80478  69664     83  3    0x100092  poll          ntpd
 69664  168846      1     83  3    0x100092  poll          ntpd
 60772  470329  27861     74  3    0x100092  bpf           pflogd
 27861  201089      1      0  3        0x80  netio         pflogd
  6731  309840  70141     73  3    0x100090  kqread        syslogd
 70141   72517      1      0  3    0x100082  netio         syslogd
 39772  400464   3688    115  3    0x100092  kqread        slaacd
 49085  170108   3688    115  3    0x100092  kqread        slaacd
  3688  412995      1      0  3    0x100080  kqread        slaacd
 43510  399669      0      0  3     0x14200  pgzero        zerothread
 47093  226254      0      0  3     0x14200  aiodoned      aiodoned
 84947   87734      0      0  3     0x14200  syncer        update
 29707   68630      0      0  3     0x14200  cleaner       cleaner
 58044  156673      0      0  3     0x14200  reaper        reaper
 16151  435974      0      0  3     0x14200  pgdaemon      pagedaemon
 17118  305370      0      0  3     0x14200  bored         crynlk
 79904  181968      0      0  3     0x14200  bored         crypto
 37015  263878      0      0  3  0x40014200                idle3
 98338  248545      0      0  7  0x40014200                idle2
 32681  339086      0      0  7  0x40014200                idle1
 47904  311340      0      0  3     0x14200  bored         sensors
  6922  113453      0      0  3     0x14200  usbtsk        usbtask
 53385   13742      0      0  3     0x14200  usbatsk       usbatsk
 71035  217622      0      0  3     0x14200  idma          sdmmc0
 42437  273161      0      0  3     0x14200  bored         softnet
 97311   74791      0      0  3     0x14200  bored         systqmp
 16571  517702      0      0  3     0x14200  bored         systq
  4124  371062      0      0  3  0x40014200  bored         softclock
 82329  177746      0      0  7  0x40014200                idle0
 71880   45469      0      0  3     0x14200  kmalloc       kmthread
     1  415146      0      0  3        0x82  wait          init
     0       0     -1      0  3     0x10200  scheduler     swapper

ddb{3}> show uvm
Current UVM status:
  pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
  216437 VM pages: 11143 active, 34969 inactive, 0 wired, 112459 free (14091
ze
ro)
  min  10% (25) anon, 10% (25) vnode, 5% (12) vtext
  freemin=7214, free-target=9618, inactive-target=15303, wired-max=72145
  faults=5204454, traps=0, intrs=0, ctxswitch=142569320 fpuswitch=0
  softint=369167, syscalls=4515133, kmapent=13
  fault counts:
    noram=0, noanon=0, noamap=0, pgwait=0, pgrele=0
    ok relocks(total)=62417(62418), anget(retries)=1885193(0),
amapcopy=919374
    neighbor anon/obj pg=132157/2787345, gets(lock/unlock)=2607228/62418
    cases: anon=1763088, anoncow=122105, obj=2324309, prcopy=282918,
przero=712
037
  daemon and swap counts:
    woke=3, revs=0, scans=0, obscans=0, anscans=0
    busy=0, freed=0, reactivate=0, deactivate=0
    pageouts=0, pending=0, nswget=0
    nswapdev=1
    swpages=263967, swpginuse=0, swpgonly=0 paging=0
  kernel pointers:
    objs(kern)=0xffffff8000ae20e0

ddb{3}> show bcstats
Current Buffer Cache status:
numbufs 10555 busymapped 1, delwri 167
kvaslots 2705 avail kva slots 2704
bufpages 43220, dmapages 43220, dirtypages 668
pendingreads 3, pendingwrites 0
highflips 0, highflops 0, dmaflips 0
ddb{3}> machine ddbcpu 1
Stopped at      ampintc_ipi_ddb+0x1c:   db_enter() at ampintc_ipi_ddb+0x18
ampintc_ipi_ddb() at handle_el1h_irq+0x6c
handle_el1h_irq() at sched_idle+0x224
sched_idle() at proc_trampoline+0x10

ddb{1}> trace
db_enter() at ampintc_ipi_ddb+0x18
ampintc_ipi_ddb() at handle_el1h_irq+0x6c
handle_el1h_irq() at sched_idle+0x224
sched_idle() at proc_trampoline+0x10
ddb{1}> machine ddbcpu 2
Stopped at      ampintc_ipi_ddb+0x1c:   db_enter() at ampintc_ipi_ddb+0x18
ampintc_ipi_ddb() at handle_el1h_irq+0x6c
handle_el1h_irq() at sched_idle+0x224
sched_idle() at proc_trampoline+0x10

ddb{2}> trace
db_enter() at ampintc_ipi_ddb+0x18
ampintc_ipi_ddb() at handle_el1h_irq+0x6c
handle_el1h_irq() at sched_idle+0x224
sched_idle() at proc_trampoline+0x10
ddb{2}> machine ddbcpu 3
Stopped at      panic+0x148:    db_enter() at panic+0x144
panic() at uvm_unmap_detach+0x7c
uvm_unmap_detach() at uvmspace_exec+0x1d0
uvmspace_exec() at sys_execve+0x5a8
sys_execve() at svc_handler+0x264
svc_handler() at do_el0_sync+0x1b0
do_el0_sync() at handle_el0_sync+0x74

ddb{3}> trace
db_enter() at panic+0x144
panic() at uvm_unmap_detach+0x7c
uvm_unmap_detach() at uvmspace_exec+0x1d0
uvmspace_exec() at sys_execve+0x5a8
sys_execve() at svc_handler+0x264
svc_handler() at do_el0_sync+0x1b0
do_el0_sync() at handle_el0_sync+0x74
handle_el0_sync() at 0x206689f7d4
address 0x7ffffee5b8 is invalid
--- trap ---
ddb{3}> machine ddbcpu 0

******
No output and no longer responsive.