panic: init died (signal 10, exit 0) on arm64 on Rock64 due to full file system?

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

panic: init died (signal 10, exit 0) on arm64 on Rock64 due to full file system?

Sebastian Reitenbach
Hi,

I ran into the very same panic multiple times now, i.e. once trying to install kibana on that node,
but not exactly sure if that's the reason, ther's also lots of stuff running in the background.



U-Boot TPL 2020.01 (Mar 30 2020 - 19:56:30)
LPDDR3, 800MHz
BW=32 Col=11 Bk=8 CS0 Row=15 CS1 Row=15 CS=2 Die BW=16 Size=4096MB
Trying to boot from BOOTROM
Returning to boot ROM...

U-Boot SPL 2020.01 (Mar 30 2020 - 19:56:30 -0600)
Trying to boot from MMC2
NOTICE:  BL31: v2.2(debug):2.2
NOTICE:  BL31: Built : 19:47:49, Mar 30 2020
ERROR:   over or zero region, nr=4188232, max=10
NOTICE:  BL31:Rockchip release version: v1.2
INFO:    ARM GICv2 driver initialized
INFO:    plat_rockchip_pmu_init: pd status 0xe
INFO:    BL31: Initializing runtime services
WARNING: BL31: cortex_a53: CPU workaround for 855873 was missing!
INFO:    BL31: Preparing for EL3 exit to normal world
INFO:    Entry point address = 0x200000
INFO:    SPSR = 0x3c9


U-Boot 2020.01 (Mar 30 2020 - 19:56:30 -0600)

Model: Pine64 Rock64
DRAM:  4 GiB
PMIC:  RK8050 (on=0x40, off=0x00)
MMC:   rksdmmc@ff500000: 1, rksdmmc@ff520000: 0
Loading Environment from MMC... Card did not respond to voltage select!
*** Warning - No block device, using default environment

In:    serial@ff130000
Out:   serial@ff130000
Err:   serial@ff130000
Model: Pine64 Rock64
Net:   eth0: ethernet@ff540000
Hit any key to stop autoboot:  0
switch to partitions #0, OK
mmc0(part 0) is current device
Scanning mmc 0:1...
Found EFI removable media binary efi/boot/bootaa64.efi
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
Card did not respond to voltage select!
Scanning disk [hidden email]...
Disk [hidden email] not ready
Scanning disk [hidden email]...
** Unrecognized filesystem type **
Found 3 disks
BootOrder not defined
EFI boot manager: Cannot load any image
169352 bytes read in 10 ms (16.2 MiB/s)
libfdt fdt_check_header(): FDT_ERR_BADMAGIC
disks: sd0*
>> OpenBSD/arm64 BOOTAA64 1.2
boot>
NOTE: random seed is being reused.
booting sd0a:/bsd: 8499816+1717104+554040+791336 [1076136+99+628796]=0xee8f30
type 0x2 pa 0x200000 va 0x200000 pages 0x4000 attr 0x8
type 0x7 pa 0x4200000 va 0x4200000 pages 0x3ef3 attr 0x8
type 0x4 pa 0x80f3000 va 0x80f3000 pages 0x1a attr 0x8
type 0x7 pa 0x810d000 va 0x810d000 pages 0xf30b3 attr 0x8
type 0x2 pa 0xfb1c0000 va 0xfb1c0000 pages 0xc27 attr 0x8
type 0x4 pa 0xfbde7000 va 0xfbde7000 pages 0x1 attr 0x8
type 0x2 pa 0xfbde8000 va 0xfbde8000 pages 0x3 attr 0x8
type 0x7 pa 0xfbdeb000 va 0xfbdeb000 pages 0x1 attr 0x8
type 0x2 pa 0xfbdec000 va 0xfbdec000 pages 0x100 attr 0x8
type 0x1 pa 0xfbeec000 va 0xfbeec000 pages 0x2a attr 0x8
type 0x0 pa 0xfbf16000 va 0xfbf16000 pages 0x7 attr 0x8
type 0x4 pa 0xfbf1d000 va 0xfbf1d000 pages 0x1 attr 0x8
type 0x6 pa 0xfbf1e000 va 0x5e70324000 pages 0x1 attr 0x8000000000000008
type 0x4 pa 0xfbf1f000 va 0xfbf1f000 pages 0x2 attr 0x8
type 0x0 pa 0xfbf21000 va 0xfbf21000 pages 0x4 attr 0x8
type 0x4 pa 0xfbf25000 va 0xfbf25000 pages 0x2 attr 0x8
type 0x6 pa 0xfbf27000 va 0x5e7032d000 pages 0x1 attr 0x8000000000000008
type 0x2 pa 0xfbf28000 va 0xfbf28000 pages 0x3008 attr 0x8
type 0x5 pa 0xfef30000 va 0x5e73336000 pages 0x10 attr 0x8000000000000008
type 0x2 pa 0xfef40000 va 0xfef40000 pages 0xc0 attr 0x8
[ using 1705936 bytes of bsd ELF symbol table ]
Copyright (c) 1982, 1986, 1989, 1991, 1993
        The Regents of the University of California.  All rights reserved.
Copyright (c) 1995-2020 OpenBSD. All rights reserved.  https://www.OpenBSD.org

OpenBSD 6.7-current (GENERIC.MP) #696: Wed Jul  1 12:40:57 MDT 2020
    [hidden email]:/usr/src/sys/arch/arm64/compile/GENERIC.MP
real mem  = 4210712576 (4015MB)
avail mem = 4005163008 (3819MB)
random: boothowto does not indicate good seed
mainbus0 at root: Pine64 Rock64
psci0 at mainbus0: PSCI 1.1, SMCCC 1.1
cpu0 at mainbus0 mpidr 0: ARM Cortex-A53 r0p4
cpu0: 32KB 64b/line 2-way L1 VIPT I-cache, 32KB 64b/line 4-way L1 D-cache
cpu0: 256KB 64b/line 16-way L2 cache
cpu1 at mainbus0 mpidr 1: ARM Cortex-A53 r0p4
cpu1: 32KB 64b/line 2-way L1 VIPT I-cache, 32KB 64b/line 4-way L1 D-cache
cpu1: 256KB 64b/line 16-way L2 cache
cpu2 at mainbus0 mpidr 2: ARM Cortex-A53 r0p4
cpu2: 32KB 64b/line 2-way L1 VIPT I-cache, 32KB 64b/line 4-way L1 D-cache
cpu2: 256KB 64b/line 16-way L2 cache
cpu3 at mainbus0 mpidr 3: ARM Cortex-A53 r0p4
cpu3: 32KB 64b/line 2-way L1 VIPT I-cache, 32KB 64b/line 4-way L1 D-cache
cpu3: 256KB 64b/line 16-way L2 cache
efi0 at mainbus0: UEFI 2.8
efi0: Das U-Boot rev 0x20200100
apm0 at mainbus0
syscon0 at mainbus0: "syscon"
"io-domains" at syscon0 not configured
syscon1 at mainbus0: "power-management"
rkclock0 at mainbus0
rkclock_set_frequency: 0x00000145
rkclock_set_frequency: 0x00000045
rkclock_set_frequency: 0x0000003e
rkclock_set_frequency: 0x000000e5
rkclock_set_frequency: 0x00000092
rkclock_set_frequency: 0x000000dc
rkclock_set_frequency: 0x00000061
ampintc0 at mainbus0 nirq 160, ncpu 4 ipi: 0, 1: "interrupt-controller"
rkpinctrl0 at mainbus0: "pinctrl"
rkgpio0 at rkpinctrl0
rkgpio1 at rkpinctrl0
rkgpio2 at rkpinctrl0
rkgpio3 at rkpinctrl0
"fit-images" at mainbus0 not configured
"opp_table0" at mainbus0 not configured
"arm-pmu" at mainbus0 not configured
agtimer0 at mainbus0: tick rate 24000 KHz
"xin24m" at mainbus0 not configured
com0 at mainbus0: ns16550, no working fifo
com0: console
rkiic0 at mainbus0
iic0 at rkiic0
rkpmic0 at iic0 addr 0x18: RK805
"spi" at mainbus0 not configured
simplebus0 at mainbus0: "amba"
"dmac" at simplebus0 not configured
dwmmc0 at mainbus0: 50 MHz base clock
sdmmc0 at dwmmc0: 4-bit, sd high-speed, mmc high-speed, dma
dwmmc1 at mainbus0: 50 MHz base clock
sdmmc1 at dwmmc1: 8-bit, mmc high-speed, dma
dwge0 at mainbus0: address 82:54:08:75:8c:6d
rgephy0 at dwge0 phy 0: RTL8169S/8110S/8211 PHY, rev. 6
ehci0 at mainbus0
usb0 at ehci0: USB revision 2.0
uhub0 at usb0 configuration 1 interface 0 "Generic EHCI root hub" rev 2.00/1.00 addr 1
ohci0 at mainbus0: version 1.0
"usb" at mainbus0 not configured
"external-gmac-clock" at mainbus0 not configured
"sdmmc-regulator" at mainbus0 not configured
"vcc-host-5v-regulator" at mainbus0 not configured
"vcc-sys" at mainbus0 not configured
"dmc" at mainbus0 not configured
"usb" at mainbus0 not configured
usb1 at ohci0: USB revision 1.0
uhub1 at usb1 configuration 1 interface 0 "Generic OHCI root hub" rev 1.00/1.00 addr 1
scsibus0 at sdmmc1: 2 targets, initiator 0
sd0 at scsibus0 targ 1 lun 0: <SD/MMC, SLD32G, 0000> removable
sd0: 29600MB, 512 bytes/sector, 60620800 sectors
vscsi0 at root
scsibus1 at vscsi0: 256 targets
softraid0 at root
scsibus2 at softraid0: 256 targets
bootfile: sd0a:/bsd
boot device: sd0
root on sd0a (aeb15307a25f954d.a) swap on sd0b dump on sd0b
cpu0: clock not implemented
Automatic boot in progress: starting file system checks.
/dev/sd0a (aeb15307a25f954d.a): file system is clean; not checking
/dev/sd0h (aeb15307a25f954d.h): file system is clean; not checking
/dev/sd0g (aeb15307a25f954d.g): file system is clean; not checking
/dev/sd0d (aeb15307a25f954d.d): file system is clean; not checking
/dev/sd0f (aeb15307a25f954d.f): file system is clean; not checking
/dev/sd0e (aeb15307a25f954d.e): file system is clean; not checking
/dev/sd0j (aeb15307a25f954d.j): file system is clean; not checking
pf enabled
kern.maxfiles: 7030 -> 65536
kern.bufcachepercent: 20 -> 80
net.inet.ip.forwarding: 0 -> 0
net.inet6.ip6.forwarding: 0 -> 0
starting network
dwge0: no link....
dwge0: 10.0.0.61 lease accepted from 10.0.0.60 (02:ba:3a:e3:12:3c)
reordering libraries: done.
starting early daemons: syslogd pflogd ntpd.
starting RPC daemons:.
savecore: no core dump
checking quotas: done.
clearing /tmp
kern.securelevel: 0 -> 1
creating runtime link editor directory cache.
preserving editor files.
starting network daemons: sshd snmpd smtpd sndiod.
starting package daemons: puppet filebeat icinga2 nginx mcollectived.
starting local daemons: apmd cron.
Thu Jul  2 23:17:17 CEST 2020

mount
/dev/sd0a on / type ffs (local)
/dev/sd0h on /home type ffs (local, nodev, nosuid, wxallowed)
/dev/sd0g on /tmp type ffs (local, nodev, nosuid)
/dev/sd0d on /usr type ffs (local, nodev)
/dev/sd0f on /usr/X11R6 type ffs (local, nodev)
/dev/sd0e on /usr/local type ffs (local, nodev, wxallowed)
/dev/sd0j on /var type ffs (local, nodev, nosuid)

PKG_PATH=/home pkg_add kibana

uid 0 on /: out of inodes
uid 0 on /: out of inodes
uid 0 on /: file system full
uid 0 on /: file system full
uid 0 on /: out of inodes
uid 0 on /: out of inodes
uid 0 on /: out of inodes
uid 748 on /var: out of inodes
uid 748 on /var: out of inodes
...

uid 784 belongs to _elasticsearch user.

panic: init died (signal 10, exit 0)

Stopped at      panic+0x150:        TID    PID    UID     PRFLAGS     PFLAGS  C
PU  COMMAND
*128013      1      0       0x802     0x2000    3K init
db_enter() at panic+0x14c
panic() at trapsignal+0x164
trapsignal() at data_abort+0x90
data_abort() at do_el0_sync+0x140
do_el0_sync() at handle_el0_sync+0x74
handle_el0_sync() at 0x1eab6724c0

ddb{3}> ps
   PID     TID   PPID    UID  S       FLAGS  WAIT          COMMAND
 31538   69205   8860   1000  3    0x100083  ttyin         ksh
  8860  407736  33843   1000  3        0x90  select        sshd
 33843  498695  50495      0  3        0x92  poll          sshd
 83692  340239      1      0  3    0x100098  poll          cron
 19397  351065      1      0  3        0x80  kqread        apmd
 87980  165016  12968     67  3        0x90  kqread        nginx
 73630   77613  12968     67  3        0x90  kqread        nginx
  6309  129765  12968     67  3        0x90  kqread        nginx
  9174  347765  12968     67  3        0x90  kqread        nginx
 12968  249655      1      0  3        0x88  pause         nginx
 52280   53594      1     99  3    0x100090  poll          sndiod
  3641  117391      1    110  3    0x100090  poll          sndiod
 50495  488355      1      0  3        0x80  select        sshd
 34703  130549  44951     74  3    0x100092  bpf           pflogd
 44951  168978      1      0  3        0x80  netio         pflogd
 63492  292899  85800    115  3    0x100092  kqread        slaacd
 18705  194033  85800    115  3    0x100092  kqread        slaacd
 85800  159406      1      0  3    0x100080  kqread        slaacd
 58927  332919      0      0  3     0x14200  bored         smr
 70183  382734      0      0  3     0x14200  pgzero        zerothread
 85384  369869      0      0  3     0x14200  aiodoned      aiodoned
 81872  352159      0      0  3     0x14200  syncer        update
  9887   25509      0      0  3     0x14200  cleaner       cleaner
 15634  310727      0      0  3     0x14200  reaper        reaper
 30651  301266      0      0  3     0x14200  pgdaemon      pagedaemon
 94627  418391      0      0  3     0x14200  bored         crynlk
 39098  396675      0      0  3     0x14200  bored         crypto
 86928   21808      0      0  3     0x14200  usbtsk        usbtask
 64901  285020      0      0  3     0x14200  usbatsk       usbatsk
 45962  207407      0      0  3     0x14200  mmctsk        sdmmc1
   283  214036      0      0  3     0x14200  mmctsk        sdmmc0
 69427  520025      0      0  3  0x40014200                idle3
 26744   33679      0      0  7  0x40014200                idle2
 90416  507212      0      0  7  0x40014200                idle1
 97471  120227      0      0  3     0x14200  bored         softnet
 86849   20498      0      0  3     0x14200  bored         systqmp
 73242   24650      0      0  3     0x14200  bored         systq
 42355  465745      0      0  3  0x40014200  bored         softclock
 26152  245965      0      0  7  0x40014200                idle0
 21900  216727      0      0  3     0x14200  kmalloc       kmthread
*    1  128013      0      0  7      0x2802                init
     0       0     -1      0  3     0x10200  scheduler     swapper

machine ddbcpu 1
Stopped at      ampintc_ipi_ddb+0x1c:   db_enter() at ampintc_ipi_ddb+0x18
ampintc_ipi_ddb() at arm_cpu_intr+0x30
arm_cpu_intr() at handle_el1h_irq+0x6c
handle_el1h_irq() at sched_idle+0x220
sched_idle() at proc_trampoline+0x10

ddb{1}> machine ddbcpu 2
Stopped at      ampintc_ipi_ddb+0x1c:   db_enter() at ampintc_ipi_ddb+0x18
ampintc_ipi_ddb() at arm_cpu_intr+0x30
arm_cpu_intr() at handle_el1h_irq+0x6c
handle_el1h_irq() at sched_idle+0x220
sched_idle() at proc_trampoline+0x10

ddb{2}> machine ddbcpu 3
Stopped at      panic+0x150:    db_enter() at panic+0x14c
panic() at trapsignal+0x164
trapsignal() at data_abort+0x90
data_abort() at do_el0_sync+0x140
do_el0_sync() at handle_el0_sync+0x74
handle_el0_sync() at 0x1eab6724c0
--- trap ---

ddb{3}> show uvm
Current UVM status:
  pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12
  988716 VM pages: 1584 active, 9729 inactive, 0 wired, 671897 free (84001 zero
)
  min  10% (25) anon, 10% (25) vnode, 5% (12) vtext
  freemin=32957, free-target=43942, inactive-target=200180, wired-max=329572
  faults=5120832, traps=0, intrs=0, ctxswitch=13843278 fpuswitch=0
  softint=1055809, syscalls=12491465, kmapent=19
  fault counts:
    noram=0, noanon=0, noamap=0, pgwait=35, pgrele=0
    ok relocks(total)=70330(70790), anget(retries)=1073522(4343), amapcopy=1008
770
    neighbor anon/obj pg=163509/726421, gets(lock/unlock)=623190/66333
    cases: anon=963489, anoncow=109873, obj=583514, prcopy=39288, przero=342440
7
  daemon and swap counts:
    woke=242, revs=193, scans=1943598, obscans=42634, anscans=1044671
    busy=0, freed=99260, reactivate=856293, deactivate=1010536
    pageouts=61791, pending=3665, nswget=4468
    nswapdev=1
    swpages=524287, swpginuse=1460, swpgonly=1418 paging=0
  kernel pointers:
    objs(kern)=0xffffff8000ebeb08
ddb{3}> show bcstats
Current Buffer Cache status:
numbufs 60357 busymapped 0, delwri 0
kvaslots 12358 avail kva slots 12358
bufpages 241428, dmapages 241428, dirtypages 0
pendingreads 0, pendingwrites 0
highflips 0, highflops 0, dmaflips 0

ddb{3}> machine ddbcpu 0
^^^^^^
finally hangs ddb


however, after reboot:
df -kh
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd0a      242M   94.7M    135M    41%    /
/dev/sd0h      1.9G    1.6G    204M    89%    /home
/dev/sd0g     96.7M   10.0K   91.8M     0%    /tmp
/dev/sd0d      2.9G    745M    2.0G    26%    /usr
/dev/sd0f      986M    195M    741M    21%    /usr/X11R6
/dev/sd0e      2.9G    2.0G    791M    72%    /usr/local
/dev/sd0j     17.0G   35.2M   16.1G     0%    /var
root@daedalus:~# df -i
Filesystem  512-blocks      Used     Avail Capacity iused   ifree  %iused  Mounted on
/dev/sd0a       495580    193856    276948    41%    3137   28861    10%   /
/dev/sd0h      4050876   3431596    416740    89%  274198   10920    96%   /home
/dev/sd0g       197980        20    188064     0%       6   12792     0%   /tmp
/dev/sd0d      6082908   1525800   4252964    26%   14918  399800     4%   /usr
/dev/sd0f      2018844    399760   1518144    21%    9070  146448     6%   /usr/X11R6
/dev/sd0e      6082908   4158968   1619796    72%  163020  251698    39%   /usr/local
/dev/sd0j     35610492     72160  33757808     0%    2030 2304848     0%   /var

doesn't really look like / or /var is full?

if I see it again, anything else to check?

Sebastian