aarch64 panic() at do_el1h_sync+0x21c
Locked
aarch64 panic() at do_el1h_sync+0x21c
 
|
Hi,
Yesterday I kicked off a dpb(1) run on a Raspberry Pi 4. When I came back later, the following crash was on the serial console. I don't know what triggers it, I'm afraid. I can try things, if anyone has ideas. ``` Stopped at panic+0x158: mov w0, w20 TID PID UID PRFLAGS PFLAGS CPU COMMAND *520912 45342 55 0x100000 0 0K sh 271552 57609 55 0x2 0 1 perl 246287 17436 55 0x2 0 3 cc 297670 78204 55 0x2 0 2 c++ db_enter() at panic+0x154 panic() at do_el1h_sync+0x21c do_el0_sync() at handle_el1h_sync+0x6c handle_el1h_sync() at pmap_copy_page+0x98 pmap_copy_page() at pmap_copy_page+0x98 pmap_copy_page() at uvm_fault_upper+0x144 uvm_fault_upper() at uvm_fault+0x100 https://www.openbsd.org/ddb.html describes the minimum info required in bug reports. Insufficient info makes it difficult to find and fix bugs. ddb{0}> trace db_enter() at panic+0x154 panic() at do_el1h_sync+0x21c do_el0_sync() at handle_el1h_sync+0x6c handle_el1h_sync() at pmap_copy_page+0x98 pmap_copy_page() at pmap_copy_page+0x98 pmap_copy_page() at uvm_fault_upper+0x144 uvm_fault_upper() at uvm_fault+0x100 uvm_fault() at udata_abort+0x12c udata_abort() at do_el0_sync+0x13c do_el0_sync() at handle_el0_sync+0x74 handle_el0_sync() at 0x1a5de72398 --- trap --- ddb{0}> machine ddbcpu 1 Stopped at ampintc_ipi_ddb+0x1c: ldr x15, [sp,#16] db_enter() at ampintc_ipi_ddb+0x18 ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 ampintc_irq_handler() at arm_cpu_intr+0x30 arm_cpu_intr() at handle_el1h_irq+0x6c handle_el1h_irq() at svc_handler+0x1e8 svc_handler() at svc_handler+0x1e8 svc_handler() at do_el0_sync+0xf4 ddb{1}> trace db_enter() at ampintc_ipi_ddb+0x18 ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 ampintc_irq_handler() at arm_cpu_intr+0x30 arm_cpu_intr() at handle_el1h_irq+0x6c handle_el1h_irq() at svc_handler+0x1e8 svc_handler() at svc_handler+0x1e8 svc_handler() at do_el0_sync+0xf4 do_el0_sync() at handle_el0_sync+0x74 handle_el0_sync() at 0xbe346549c --- trap --- ddb{1}> machine ddbcpu 2 Stopped at ampintc_ipi_ddb+0x1c: ldr x15, [sp,#16] db_enter() at ampintc_ipi_ddb+0x18 ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 ampintc_irq_handler() at arm_cpu_intr+0x30 arm_cpu_intr() at handle_el1h_irq+0x6c handle_el1h_irq() at svc_handler+0x1e8 svc_handler() at svc_handler+0x1e8 svc_handler() at do_el0_sync+0xf4 ddb{2}> trace db_enter() at ampintc_ipi_ddb+0x18 ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 ampintc_irq_handler() at arm_cpu_intr+0x30 arm_cpu_intr() at handle_el1h_irq+0x6c handle_el1h_irq() at svc_handler+0x1e8 svc_handler() at svc_handler+0x1e8 svc_handler() at do_el0_sync+0xf4 do_el0_sync() at handle_el0_sync+0x74 handle_el0_sync() at 0x49c136c84 --- trap --- ddb{2}> machine ddbcpu 3 Stopped at ampintc_ipi_ddb+0x1c: ldr x15, [sp,#16] db_enter() at ampintc_ipi_ddb+0x18 ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 ampintc_irq_handler() at arm_cpu_intr+0x30 arm_cpu_intr() at handle_el1h_irq+0x6c handle_el1h_irq() at svc_handler+0x1e8 svc_handler() at svc_handler+0x1e8 svc_handler() at do_el0_sync+0xf4 ddb{3}> trace db_enter() at ampintc_ipi_ddb+0x18 ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 ampintc_irq_handler() at arm_cpu_intr+0x30 arm_cpu_intr() at handle_el1h_irq+0x6c handle_el1h_irq() at svc_handler+0x1e8 svc_handler() at svc_handler+0x1e8 svc_handler() at do_el0_sync+0xf4 do_el0_sync() at handle_el0_sync+0x74 handle_el0_sync() at 0x42749a2f4 --- trap --- ddb{0}> ps PID TID PPID UID S FLAGS WAIT COMMAND *45342 520912 37571 55 7 0x100000 sh 57609 271552 21291 55 7 0x2 perl 17436 246287 35320 55 7 0x2 cc 35320 135299 39411 55 3 0x100088 sigsusp sh 78204 297670 17054 55 7 0x2 c++ 17054 63352 35973 55 3 0x82 wait perl 35973 95003 91433 55 3 0x10008a sigsusp sh 10647 494721 94245 0 3 0x91 nanoslp perl 91433 191032 8978 55 3 0x82 wait gmake 8978 33444 94837 55 3 0x100088 sigsusp sh 37571 49113 82738 55 3 0x10008a sigsusp sh 82738 287706 6798 55 3 0x10008a sigsusp sh 6798 273621 90412 55 3 0x10008a sigsusp make 90412 356203 30174 55 3 0x10008a sigsusp make 30174 496473 88029 55 3 0x10008a sigsusp sh 88029 445633 29311 55 3 0x10008a sigsusp make 29311 152572 18553 55 3 0x10008a sigsusp sh 18553 202605 94245 55 3 0x10008a sigsusp make 94837 40263 98003 55 3 0x10008a sigsusp sh 98003 309681 6187 55 3 0x82 wait gmake 6187 390103 12422 55 3 0x100088 sigsusp sh 12422 86000 52828 55 3 0x10008a sigsusp sh 52828 335301 58810 55 3 0x82 wait gmake 58810 344911 45510 55 3 0x82 wait gmake 45510 159573 81731 55 3 0x10008a sigsusp make 81731 466687 26402 55 3 0x10008a sigsusp sh 26402 437636 13909 55 3 0x10008a sigsusp make 13909 515569 16990 55 3 0x10008a sigsusp sh 16990 309064 94245 55 3 0x10008a sigsusp make 39411 403349 34554 55 3 0x10008a sigsusp sh 34554 485373 92598 55 3 0x10008a sigsusp sh 92598 319508 58962 55 3 0x10008a sigsusp make 21291 168268 57968 55 3 0x10008a sigsusp sh 57968 136689 86283 55 3 0x10008a sigsusp make 86283 277561 53841 55 3 0x10008a sigsusp make 53841 194835 29897 55 3 0x10008a sigsusp sh 29897 468463 76592 55 3 0x10008a sigsusp make 76592 489130 75569 55 3 0x10008a sigsusp sh 75569 76155 94245 55 3 0x10008a sigsusp make 58962 341713 28611 55 3 0x10008a sigsusp make 28611 475922 51233 55 3 0x10008a sigsusp sh 51233 445846 97944 55 3 0x10008a sigsusp make 97944 513430 14018 55 3 0x10008a sigsusp sh 14018 517058 94245 55 3 0x10008a sigsusp make 16586 129898 62476 1000 3 0x100083 kqread tmux 62476 185241 21106 1000 3 0x10008b sigsusp ksh 21106 289984 65923 1000 3 0x90 select sshd 65923 47701 28035 0 3 0x92 poll sshd 94245 111757 57716 0 3 0x93 wait perl 57716 95218 79063 0 3 0x10008b sigsusp ksh 79063 397105 69092 1000 3 0x10008b sigsusp ksh 69092 40715 1 1000 3 0x100080 kqread tmux 23034 519794 1 0 3 0x100083 ttyin getty 31212 509801 1 0 3 0x100098 poll cron 77120 80514 1 99 3 0x100090 poll sndiod 58244 148661 1 110 3 0x100090 poll sndiod 34557 422053 56929 95 3 0x100092 kqread smtpd 1545 377516 56929 103 3 0x100092 kqread smtpd 44915 168241 56929 95 3 0x100092 kqread smtpd 6911 462094 56929 95 3 0x100092 kqread smtpd 67181 271492 56929 95 3 0x100092 kqread smtpd 44292 358201 56929 95 3 0x100092 kqread smtpd 56929 178617 1 0 3 0x100080 kqread smtpd 28035 123536 1 0 3 0x80 select sshd 12202 236134 1 0 3 0x100080 poll ntpd 47011 442315 65316 83 3 0x100092 poll ntpd 65316 352793 1 83 3 0x100092 poll ntpd 26910 507945 53602 74 3 0x100092 bpf pflogd 53602 452866 1 0 3 0x80 netio pflogd 92354 409382 54329 73 3 0x100090 kqread syslogd 54329 35524 1 0 3 0x100082 netio syslogd 76882 296612 66351 115 3 0x100092 kqread slaacd 86076 447263 66351 115 3 0x100092 kqread slaacd 66351 51769 1 0 3 0x100080 kqread slaacd 95452 319671 0 0 3 0x14200 bored smr 33055 34426 0 0 2 0x14200 zerothread 92693 399241 0 0 3 0x14200 aiodoned aiodoned 896 329262 0 0 3 0x14200 syncer update 593 301341 0 0 3 0x14200 cleaner cleaner 46671 460752 0 0 3 0x14200 reaper reaper 24787 324303 0 0 3 0x14200 pgdaemon pagedaemon 96276 337550 0 0 3 0x14200 bored crynlk 97434 489378 0 0 3 0x14200 bored crypto 32406 192127 0 0 3 0x40014200 acpi0 acpi0 25685 47102 0 0 3 0x14200 usbtsk usbtask 28401 342245 0 0 3 0x14200 usbatsk usbatsk 83381 68268 0 0 3 0x40014200 idle3 30770 291130 0 0 3 0x40014200 idle2 79959 328443 0 0 3 0x40014200 idle1 53482 511823 0 0 3 0x14200 bored softnet 40995 325138 0 0 3 0x14200 bored systqmp 15091 285486 0 0 3 0x14200 bored systq 84205 230094 0 0 3 0x40014200 bored softclock 52567 49791 0 0 3 0x40014200 idle0 95095 182698 0 0 3 0x14200 kmalloc kmthread 1 492182 0 0 3 0x82 wait init 0 0 -1 0 3 0x10200 scheduler swapper ddb{0}> show uvm Current UVM status: pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12 1770335 VM pages: 27884 active, 36707 inactive, 1 wired, 216902 free (26816 z ero) min 10% (25) anon, 10% (25) vnode, 5% (12) vtext freemin=59011, free-target=78681, inactive-target=78682, wired-max=590111 faults=242619496, traps=0, intrs=0, ctxswitch=14378837 fpuswitch=0 softint=10915677, syscalls=168677618, kmapent=34 fault counts: noram=0, noanon=0, noamap=0, pgwait=0, pgrele=0 ok relocks(total)=329274(329893), anget(retries)=92181088(0), amapcopy=5141 2249 neighbor anon/obj pg=9678591/131558139, gets(lock/unlock)=103017533/329893 cases: anon=83775542, anoncow=8405546, obj=100492731, prcopy=2524183, przer o=47421495 daemon and swap counts: woke=148, revs=0, scans=0, obscans=0, anscans=0 busy=0, freed=0, reactivate=0, deactivate=0 pageouts=0, pending=0, nswget=0 nswapdev=1 swpages=263063, swpginuse=0, swpgonly=0 paging=0 kernel pointers: objs(kern)=0xffffff8000e95b48 ddb{0}> show bcstats Current Buffer Cache status: numbufs 102448 busymapped 0, delwri 69 kvaslots 235 avail kva slots 235 bufpages 1422246, dmapages 3702, dirtypages 584 pendingreads 0, pendingwrites 0 highflips 638606, highflops 0, dmaflips 71986 ``` dmesg: ``` OpenBSD 6.8-current (GENERIC.MP) #999: Mon Feb 1 15:05:58 MST 2021 [hidden email]:/usr/src/sys/arch/arm64/compile/GENERIC.MP real mem = 7514284032 (7166MB) avail mem = 7251083264 (6915MB) random: good seed from bootblocks mainbus0 at root: ACPI psci0 at mainbus0: PSCI 1.1, SMCCC 1.2 cpu0 at mainbus0 mpidr 0: ARM Cortex-A72 r0p3 cpu0: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu0: 1024KB 64b/line 16-way L2 cache cpu0: CRC32 cpu1 at mainbus0 mpidr 1: ARM Cortex-A72 r0p3 cpu1: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu1: 1024KB 64b/line 16-way L2 cache cpu1: CRC32 cpu2 at mainbus0 mpidr 2: ARM Cortex-A72 r0p3 cpu2: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu2: 1024KB 64b/line 16-way L2 cache cpu2: CRC32 cpu3 at mainbus0 mpidr 3: ARM Cortex-A72 r0p3 cpu3: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache cpu3: 1024KB 64b/line 16-way L2 cache cpu3: CRC32 efi0 at mainbus0: UEFI 2.7 efi0: https://github.com/pftf/RPi4 rev 0x10000 smbios0 at efi0: SMBIOS 3.3.0 smbios0: vendor https://github.com/pftf/RPi4 version "UEFI Firmware v1.19" date 08/13/2020 smbios0: Raspberry Pi Foundation Raspberry Pi 4 Model B apm0 at mainbus0 ampintc0 at mainbus0 nirq 256, ncpu 4 ipi: 0, 1: "interrupt-controller" agtimer0 at mainbus0: 54000 kHz acpi0 at mainbus0: ACPI 6.3 acpi0: sleep states acpi0: tables DSDT FACP CSRT DBG2 GTDT APIC PPTT SPCR acpi0: wakeup devices "BCM2849" at acpi0 not configured "BCM2835" at acpi0 not configured "BCM2854" at acpi0 not configured "ACPI0004" at acpi0 not configured xhci0 at acpi0 XHC0 addr 0x600000000/0x1000 irq 175, xHCI 1.0 usb0 at xhci0: USB revision 3.0 uhub0 at usb0 configuration 1 interface 0 "Generic xHCI root hub" rev 3.00/1.00 addr 1 "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0007" at acpi0 not configured "ACPI0004" at acpi0 not configured "BCM2848" at acpi0 not configured "BCM2850" at acpi0 not configured "BCM2856" at acpi0 not configured "BCM2845" at acpi0 not configured "BCM2841" at acpi0 not configured "BCM2841" at acpi0 not configured "BCM2838" at acpi0 not configured "BCM2839" at acpi0 not configured "BCM2844" at acpi0 not configured pluart0 at acpi0 URT0 addr 0xfe201000/0x1000 irq 153: console "BCM2836" at acpi0 not configured "BCM2EA6" at acpi0 not configured "MSFT8000" at acpi0 not configured "BCM2847" at acpi0 not configured "BCM2855" at acpi0 not configured bse0 at acpi0 ETH0 addr 0xfd580000/0x10000 irq 189: address dc:a6:32:c4:5b:99 brgphy0 at bse0 phy 1: BCM54210E 10/100/1000baseT PHY, rev. 2 uhub1 at uhub0 port 1 configuration 1 interface 0 "VIA Labs USB2.0 Hub" rev 2.10/4.21 addr 2 umass0 at uhub1 port 2 configuration 1 interface 0 "JMicron USB to ATA/ATAPI Bridge" rev 2.10/0.00 addr 3 umass0: using SCSI over Bulk-Only scsibus0 at umass0: 2 targets, initiator 0 sd0 at scsibus0 targ 1 lun 0: <TOSHIBA, MQ01ABD100, 0100> serial.357d7788000012345678 sd0: 948748MB, 512 bytes/sector, 1943035904 sectors vscsi0 at root scsibus1 at vscsi0: 256 targets softraid0 at root scsibus2 at softraid0: 256 targets root on sd0a (eeae950571c974c2.a) swap on sd0b dump on sd0b ``` mounts: ``` /dev/sd0a on / type ffs (local, wxallowed) /dev/sd0d on /usr/ports type ffs (local, nodev, nosuid) ``` -- Best Regards Edd Barrett http://www.theunixzoo.co.uk |
Re: aarch64 panic() at do_el1h_sync+0x21c
|
|
> Date: Wed, 3 Feb 2021 21:59:11 +0000
> From: Edd Barrett <[hidden email]> > > Hi, > > Yesterday I kicked off a dpb(1) run on a Raspberry Pi 4. When I came > back later, the following crash was on the serial console. > > I don't know what triggers it, I'm afraid. > > I can try things, if anyone has ideas. The actual panic message would be helpful... > ``` > Stopped at panic+0x158: mov w0, w20 > TID PID UID PRFLAGS PFLAGS CPU COMMAND > *520912 45342 55 0x100000 0 0K sh > 271552 57609 55 0x2 0 1 perl > 246287 17436 55 0x2 0 3 cc > 297670 78204 55 0x2 0 2 c++ > db_enter() at panic+0x154 > panic() at do_el1h_sync+0x21c > do_el0_sync() at handle_el1h_sync+0x6c > handle_el1h_sync() at pmap_copy_page+0x98 > pmap_copy_page() at pmap_copy_page+0x98 > pmap_copy_page() at uvm_fault_upper+0x144 > uvm_fault_upper() at uvm_fault+0x100 > https://www.openbsd.org/ddb.html describes the minimum info required in bug > reports. Insufficient info makes it difficult to find and fix bugs. > ddb{0}> trace > db_enter() at panic+0x154 > panic() at do_el1h_sync+0x21c > do_el0_sync() at handle_el1h_sync+0x6c > handle_el1h_sync() at pmap_copy_page+0x98 > pmap_copy_page() at pmap_copy_page+0x98 > pmap_copy_page() at uvm_fault_upper+0x144 > uvm_fault_upper() at uvm_fault+0x100 > uvm_fault() at udata_abort+0x12c > udata_abort() at do_el0_sync+0x13c > do_el0_sync() at handle_el0_sync+0x74 > handle_el0_sync() at 0x1a5de72398 > --- trap --- > ddb{0}> machine ddbcpu 1 > Stopped at ampintc_ipi_ddb+0x1c: ldr x15, [sp,#16] > db_enter() at ampintc_ipi_ddb+0x18 > ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 > ampintc_irq_handler() at arm_cpu_intr+0x30 > arm_cpu_intr() at handle_el1h_irq+0x6c > handle_el1h_irq() at svc_handler+0x1e8 > svc_handler() at svc_handler+0x1e8 > svc_handler() at do_el0_sync+0xf4 > ddb{1}> trace > db_enter() at ampintc_ipi_ddb+0x18 > ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 > ampintc_irq_handler() at arm_cpu_intr+0x30 > arm_cpu_intr() at handle_el1h_irq+0x6c > handle_el1h_irq() at svc_handler+0x1e8 > svc_handler() at svc_handler+0x1e8 > svc_handler() at do_el0_sync+0xf4 > do_el0_sync() at handle_el0_sync+0x74 > handle_el0_sync() at 0xbe346549c > --- trap --- > ddb{1}> machine ddbcpu 2 > Stopped at ampintc_ipi_ddb+0x1c: ldr x15, [sp,#16] > db_enter() at ampintc_ipi_ddb+0x18 > ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 > ampintc_irq_handler() at arm_cpu_intr+0x30 > arm_cpu_intr() at handle_el1h_irq+0x6c > handle_el1h_irq() at svc_handler+0x1e8 > svc_handler() at svc_handler+0x1e8 > svc_handler() at do_el0_sync+0xf4 > ddb{2}> trace > db_enter() at ampintc_ipi_ddb+0x18 > ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 > ampintc_irq_handler() at arm_cpu_intr+0x30 > arm_cpu_intr() at handle_el1h_irq+0x6c > handle_el1h_irq() at svc_handler+0x1e8 > svc_handler() at svc_handler+0x1e8 > svc_handler() at do_el0_sync+0xf4 > do_el0_sync() at handle_el0_sync+0x74 > handle_el0_sync() at 0x49c136c84 > --- trap --- > ddb{2}> machine ddbcpu 3 > Stopped at ampintc_ipi_ddb+0x1c: ldr x15, [sp,#16] > db_enter() at ampintc_ipi_ddb+0x18 > ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 > ampintc_irq_handler() at arm_cpu_intr+0x30 > arm_cpu_intr() at handle_el1h_irq+0x6c > handle_el1h_irq() at svc_handler+0x1e8 > svc_handler() at svc_handler+0x1e8 > svc_handler() at do_el0_sync+0xf4 > ddb{3}> trace > db_enter() at ampintc_ipi_ddb+0x18 > ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 > ampintc_irq_handler() at arm_cpu_intr+0x30 > arm_cpu_intr() at handle_el1h_irq+0x6c > handle_el1h_irq() at svc_handler+0x1e8 > svc_handler() at svc_handler+0x1e8 > svc_handler() at do_el0_sync+0xf4 > do_el0_sync() at handle_el0_sync+0x74 > handle_el0_sync() at 0x42749a2f4 > --- trap --- > ddb{0}> ps > PID TID PPID UID S FLAGS WAIT COMMAND > *45342 520912 37571 55 7 0x100000 sh > 57609 271552 21291 55 7 0x2 perl > 17436 246287 35320 55 7 0x2 cc > 35320 135299 39411 55 3 0x100088 sigsusp sh > 78204 297670 17054 55 7 0x2 c++ > 17054 63352 35973 55 3 0x82 wait perl > 35973 95003 91433 55 3 0x10008a sigsusp sh > 10647 494721 94245 0 3 0x91 nanoslp perl > 91433 191032 8978 55 3 0x82 wait gmake > 8978 33444 94837 55 3 0x100088 sigsusp sh > 37571 49113 82738 55 3 0x10008a sigsusp sh > 82738 287706 6798 55 3 0x10008a sigsusp sh > 6798 273621 90412 55 3 0x10008a sigsusp make > 90412 356203 30174 55 3 0x10008a sigsusp make > 30174 496473 88029 55 3 0x10008a sigsusp sh > 88029 445633 29311 55 3 0x10008a sigsusp make > 29311 152572 18553 55 3 0x10008a sigsusp sh > 18553 202605 94245 55 3 0x10008a sigsusp make > 94837 40263 98003 55 3 0x10008a sigsusp sh > 98003 309681 6187 55 3 0x82 wait gmake > 6187 390103 12422 55 3 0x100088 sigsusp sh > 12422 86000 52828 55 3 0x10008a sigsusp sh > 52828 335301 58810 55 3 0x82 wait gmake > 58810 344911 45510 55 3 0x82 wait gmake > 45510 159573 81731 55 3 0x10008a sigsusp make > 81731 466687 26402 55 3 0x10008a sigsusp sh > 26402 437636 13909 55 3 0x10008a sigsusp make > 13909 515569 16990 55 3 0x10008a sigsusp sh > 16990 309064 94245 55 3 0x10008a sigsusp make > 39411 403349 34554 55 3 0x10008a sigsusp sh > 34554 485373 92598 55 3 0x10008a sigsusp sh > 92598 319508 58962 55 3 0x10008a sigsusp make > 21291 168268 57968 55 3 0x10008a sigsusp sh > 57968 136689 86283 55 3 0x10008a sigsusp make > 86283 277561 53841 55 3 0x10008a sigsusp make > 53841 194835 29897 55 3 0x10008a sigsusp sh > 29897 468463 76592 55 3 0x10008a sigsusp make > 76592 489130 75569 55 3 0x10008a sigsusp sh > 75569 76155 94245 55 3 0x10008a sigsusp make > 58962 341713 28611 55 3 0x10008a sigsusp make > 28611 475922 51233 55 3 0x10008a sigsusp sh > 51233 445846 97944 55 3 0x10008a sigsusp make > 97944 513430 14018 55 3 0x10008a sigsusp sh > 14018 517058 94245 55 3 0x10008a sigsusp make > 16586 129898 62476 1000 3 0x100083 kqread tmux > 62476 185241 21106 1000 3 0x10008b sigsusp ksh > 21106 289984 65923 1000 3 0x90 select sshd > 65923 47701 28035 0 3 0x92 poll sshd > 94245 111757 57716 0 3 0x93 wait perl > 57716 95218 79063 0 3 0x10008b sigsusp ksh > 79063 397105 69092 1000 3 0x10008b sigsusp ksh > 69092 40715 1 1000 3 0x100080 kqread tmux > 23034 519794 1 0 3 0x100083 ttyin getty > 31212 509801 1 0 3 0x100098 poll cron > 77120 80514 1 99 3 0x100090 poll sndiod > 58244 148661 1 110 3 0x100090 poll sndiod > 34557 422053 56929 95 3 0x100092 kqread smtpd > 1545 377516 56929 103 3 0x100092 kqread smtpd > 44915 168241 56929 95 3 0x100092 kqread smtpd > 6911 462094 56929 95 3 0x100092 kqread smtpd > 67181 271492 56929 95 3 0x100092 kqread smtpd > 44292 358201 56929 95 3 0x100092 kqread smtpd > 56929 178617 1 0 3 0x100080 kqread smtpd > 28035 123536 1 0 3 0x80 select sshd > 12202 236134 1 0 3 0x100080 poll ntpd > 47011 442315 65316 83 3 0x100092 poll ntpd > 65316 352793 1 83 3 0x100092 poll ntpd > 26910 507945 53602 74 3 0x100092 bpf pflogd > 53602 452866 1 0 3 0x80 netio pflogd > 92354 409382 54329 73 3 0x100090 kqread syslogd > 54329 35524 1 0 3 0x100082 netio syslogd > 76882 296612 66351 115 3 0x100092 kqread slaacd > 86076 447263 66351 115 3 0x100092 kqread slaacd > 66351 51769 1 0 3 0x100080 kqread slaacd > 95452 319671 0 0 3 0x14200 bored smr > 33055 34426 0 0 2 0x14200 zerothread > 92693 399241 0 0 3 0x14200 aiodoned aiodoned > 896 329262 0 0 3 0x14200 syncer update > 593 301341 0 0 3 0x14200 cleaner cleaner > 46671 460752 0 0 3 0x14200 reaper reaper > 24787 324303 0 0 3 0x14200 pgdaemon pagedaemon > 96276 337550 0 0 3 0x14200 bored crynlk > 97434 489378 0 0 3 0x14200 bored crypto > 32406 192127 0 0 3 0x40014200 acpi0 acpi0 > 25685 47102 0 0 3 0x14200 usbtsk usbtask > 28401 342245 0 0 3 0x14200 usbatsk usbatsk > 83381 68268 0 0 3 0x40014200 idle3 > 30770 291130 0 0 3 0x40014200 idle2 > 79959 328443 0 0 3 0x40014200 idle1 > 53482 511823 0 0 3 0x14200 bored softnet > 40995 325138 0 0 3 0x14200 bored systqmp > 15091 285486 0 0 3 0x14200 bored systq > 84205 230094 0 0 3 0x40014200 bored softclock > 52567 49791 0 0 3 0x40014200 idle0 > 95095 182698 0 0 3 0x14200 kmalloc kmthread > 1 492182 0 0 3 0x82 wait init > 0 0 -1 0 3 0x10200 scheduler swapper > ddb{0}> show uvm > Current UVM status: > pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12 > 1770335 VM pages: 27884 active, 36707 inactive, 1 wired, 216902 free (26816 z > ero) > min 10% (25) anon, 10% (25) vnode, 5% (12) vtext > freemin=59011, free-target=78681, inactive-target=78682, wired-max=590111 > faults=242619496, traps=0, intrs=0, ctxswitch=14378837 fpuswitch=0 > softint=10915677, syscalls=168677618, kmapent=34 > fault counts: > noram=0, noanon=0, noamap=0, pgwait=0, pgrele=0 > ok relocks(total)=329274(329893), anget(retries)=92181088(0), amapcopy=5141 > 2249 > neighbor anon/obj pg=9678591/131558139, gets(lock/unlock)=103017533/329893 > cases: anon=83775542, anoncow=8405546, obj=100492731, prcopy=2524183, przer > o=47421495 > daemon and swap counts: > woke=148, revs=0, scans=0, obscans=0, anscans=0 > busy=0, freed=0, reactivate=0, deactivate=0 > pageouts=0, pending=0, nswget=0 > nswapdev=1 > swpages=263063, swpginuse=0, swpgonly=0 paging=0 > kernel pointers: > objs(kern)=0xffffff8000e95b48 > ddb{0}> show bcstats > Current Buffer Cache status: > numbufs 102448 busymapped 0, delwri 69 > kvaslots 235 avail kva slots 235 > bufpages 1422246, dmapages 3702, dirtypages 584 > pendingreads 0, pendingwrites 0 > highflips 638606, highflops 0, dmaflips 71986 > ``` > > dmesg: > ``` > OpenBSD 6.8-current (GENERIC.MP) #999: Mon Feb 1 15:05:58 MST 2021 > [hidden email]:/usr/src/sys/arch/arm64/compile/GENERIC.MP > real mem = 7514284032 (7166MB) > avail mem = 7251083264 (6915MB) > random: good seed from bootblocks > mainbus0 at root: ACPI > psci0 at mainbus0: PSCI 1.1, SMCCC 1.2 > cpu0 at mainbus0 mpidr 0: ARM Cortex-A72 r0p3 > cpu0: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache > cpu0: 1024KB 64b/line 16-way L2 cache > cpu0: CRC32 > cpu1 at mainbus0 mpidr 1: ARM Cortex-A72 r0p3 > cpu1: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache > cpu1: 1024KB 64b/line 16-way L2 cache > cpu1: CRC32 > cpu2 at mainbus0 mpidr 2: ARM Cortex-A72 r0p3 > cpu2: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache > cpu2: 1024KB 64b/line 16-way L2 cache > cpu2: CRC32 > cpu3 at mainbus0 mpidr 3: ARM Cortex-A72 r0p3 > cpu3: 48KB 64b/line 3-way L1 PIPT I-cache, 32KB 64b/line 2-way L1 D-cache > cpu3: 1024KB 64b/line 16-way L2 cache > cpu3: CRC32 > efi0 at mainbus0: UEFI 2.7 > efi0: https://github.com/pftf/RPi4 rev 0x10000 > smbios0 at efi0: SMBIOS 3.3.0 > smbios0: vendor https://github.com/pftf/RPi4 version "UEFI Firmware v1.19" date 08/13/2020 > smbios0: Raspberry Pi Foundation Raspberry Pi 4 Model B > apm0 at mainbus0 > ampintc0 at mainbus0 nirq 256, ncpu 4 ipi: 0, 1: "interrupt-controller" > agtimer0 at mainbus0: 54000 kHz > acpi0 at mainbus0: ACPI 6.3 > acpi0: sleep states > acpi0: tables DSDT FACP CSRT DBG2 GTDT APIC PPTT SPCR > acpi0: wakeup devices > "BCM2849" at acpi0 not configured > "BCM2835" at acpi0 not configured > "BCM2854" at acpi0 not configured > "ACPI0004" at acpi0 not configured > xhci0 at acpi0 XHC0 addr 0x600000000/0x1000 irq 175, xHCI 1.0 > usb0 at xhci0: USB revision 3.0 > uhub0 at usb0 configuration 1 interface 0 "Generic xHCI root hub" rev 3.00/1.00 addr 1 > "ACPI0007" at acpi0 not configured > "ACPI0007" at acpi0 not configured > "ACPI0007" at acpi0 not configured > "ACPI0007" at acpi0 not configured > "ACPI0004" at acpi0 not configured > "BCM2848" at acpi0 not configured > "BCM2850" at acpi0 not configured > "BCM2856" at acpi0 not configured > "BCM2845" at acpi0 not configured > "BCM2841" at acpi0 not configured > "BCM2841" at acpi0 not configured > "BCM2838" at acpi0 not configured > "BCM2839" at acpi0 not configured > "BCM2844" at acpi0 not configured > pluart0 at acpi0 URT0 addr 0xfe201000/0x1000 irq 153: console > "BCM2836" at acpi0 not configured > "BCM2EA6" at acpi0 not configured > "MSFT8000" at acpi0 not configured > "BCM2847" at acpi0 not configured > "BCM2855" at acpi0 not configured > bse0 at acpi0 ETH0 addr 0xfd580000/0x10000 irq 189: address dc:a6:32:c4:5b:99 > brgphy0 at bse0 phy 1: BCM54210E 10/100/1000baseT PHY, rev. 2 > uhub1 at uhub0 port 1 configuration 1 interface 0 "VIA Labs USB2.0 Hub" rev 2.10/4.21 addr 2 > umass0 at uhub1 port 2 configuration 1 interface 0 "JMicron USB to ATA/ATAPI Bridge" rev 2.10/0.00 addr 3 > umass0: using SCSI over Bulk-Only > scsibus0 at umass0: 2 targets, initiator 0 > sd0 at scsibus0 targ 1 lun 0: <TOSHIBA, MQ01ABD100, 0100> serial.357d7788000012345678 > sd0: 948748MB, 512 bytes/sector, 1943035904 sectors > vscsi0 at root > scsibus1 at vscsi0: 256 targets > softraid0 at root > scsibus2 at softraid0: 256 targets > root on sd0a (eeae950571c974c2.a) swap on sd0b dump on sd0b > ``` > > mounts: > ``` > /dev/sd0a on / type ffs (local, wxallowed) > /dev/sd0d on /usr/ports type ffs (local, nodev, nosuid) > ``` > > -- > Best Regards > Edd Barrett > > http://www.theunixzoo.co.uk > > |
Re: aarch64 panic() at do_el1h_sync+0x21c
|
|
On Wed, Feb 03, 2021 at 11:22:53PM +0100, Mark Kettenis wrote:
> The actual panic message would be helpful... Doh. If it aint there, then I fudged the copy-paste. I can try to reproduce it... -- Best Regards Edd Barrett http://www.theunixzoo.co.uk |
Re: aarch64 panic() at do_el1h_sync+0x21c
|
|
On Wed, Feb 03, 2021 at 10:34:48PM +0000, Edd Barrett wrote:
> On Wed, Feb 03, 2021 at 11:22:53PM +0100, Mark Kettenis wrote: > > The actual panic message would be helpful... > > Doh. If it aint there, then I fudged the copy-paste. I can try to > reproduce it... Got it! See below. Note that I upddated my snapshot since last time. I can provide a new dmesg if required, but I'd need to reboot. First let's confirm that we have everything we need from ddb (i still have it at the prompt on the serial line). panic: uvm_fault failed: ffffff8000422080 esr 96000007 far ffffff8001fc3380 Stopped at panic+0x158: mov w0, w20 TID PID UID PRFLAGS PFLAGS CPU COMMAND * 20589 81386 0 0x11 0 0K perl 506484 33678 0 0x13 0 2 perl db_enter() at panic+0x154 panic() at do_el1h_sync+0x21c do_el0_sync() at handle_el1h_sync+0x6c handle_el1h_sync() at pmap_copy_page+0x98 pmap_copy_page() at pmap_copy_page+0x98 pmap_copy_page() at uvm_fault_upper+0x144 uvm_fault_upper() at uvm_fault+0x100 https://www.openbsd.org/ddb.html describes the minimum info required in bug reports. Insufficient info makes it difficult to find and fix bugs. ddb{0}> show panic uvm_fault failed: ffffff8000422080 esr 96000007 far ffffff8001fc3380 ddb{0}> show reg x0 0x1 x1 0xffffff8028413000 x2 0x4 x3 0xb0 $d.1+0x9d x4 0xffffff80286238c0 x5 0 x6 0x2 x7 0xffffff8028623c10 x8 0x1 x9 0x92178fc3ac5bcdfb x10 0xb x11 0x1 x12 0xffffffc8 x13 0x61781003 x14 0xffffffff x15 0x4d4223665ac6ecd x16 0x901933ca8 x17 0x8f7094788 x18 0xffffff8028623920 x19 0xffffff8000c459b5 $d.2+0x27b x20 0x100 $d.1+0xed x21 0xffffff8000e1ca90 kprintf_mutex x22 0xffffff8000eee000 rttimer_pool+0x148 x23 0xffffff8028623970 x24 0xffffff8000ee50c0 kernel_map_store x25 0xffffff8001fc3000 _end+0x106e210 x26 0xffffff802861f000 x27 0x20 $d.1+0xd x28 0x9b $d.1+0x88 x29 0xffffff8028623920 x30 0 sp 0xffffff8028623920 spsr 0x60000345 elr 0xffffff80009ef9d8 db_enter+0x14 lr 0xffffff80006821cc panic+0x158 panic+0x158: mov w0, w20 ddb{0}> machine ddbcpu 1 Stopped at ampintc_ipi_ddb+0x1c: ldr x15, [sp,#16] db_enter() at ampintc_ipi_ddb+0x18 ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 ampintc_irq_handler() at arm_cpu_intr+0x30 arm_cpu_intr() at handle_el1h_irq+0x6c handle_el1h_irq() at sched_idle+0x290 sched_idle() at sched_idle+0x290 sched_idle() at proc_trampoline+0x10 ddb{1}> show reg x0 0xffffff8011b3f400 x1 0xffffff8028413000 x2 0x4 x3 0xfffffff0 x4 0xffffff800096b2a0 smc_call x5 0xffffff8028471000 x6 0x62 $d.1+0x4f x7 0x1 x8 0xffffff8011b3f400 x9 0xffffff800061db50 ampintc_ipi_ddb x10 0xb0 $d.1+0x9d x11 0xffffff8011b3a800 x12 0xffffff8011b3f400 x13 0 x14 0xffffff8000cfd510 ap_bits_kern x15 0x4d4223665a5946d x16 0xffffff8000f0c000 uvm_map_entry_pool+0x110 x17 0xffffff8000f0c000 uvm_map_entry_pool+0x110 x18 0xffffff8028475c20 x19 0 x20 0x1 x21 0xffffff8028475cd0 x22 0xffffff8000f0b000 pfr_mask+0x18 x23 0 x24 0xffffff8011b41580 x25 0 x26 0x28 $d.1+0x15 x27 0x3 x28 0x1 x29 0xffffff8028475c20 x30 0 sp 0xffffff8028475c20 spsr 0xa0000345 elr 0xffffff80009ef9d8 db_enter+0x14 lr 0xffffff800061db6c ampintc_ipi_ddb+0x1c ampintc_ipi_ddb+0x1c: ldr x15, [sp,#16] ddb{1}> machine ddbcpu 2 Stopped at ampintc_ipi_ddb+0x1c: ldr x15, [sp,#16] db_enter() at ampintc_ipi_ddb+0x18 ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 ampintc_irq_handler() at arm_cpu_intr+0x30 arm_cpu_intr() at handle_el1h_irq+0x6c handle_el1h_irq() at udata_abort+0x118 udata_abort() at udata_abort+0x118 udata_abort() at do_el0_sync+0x13c ddb{2}> show reg x0 0xffffff8011b3f400 x1 0xffffff8028413000 x2 0x4 x3 0xfffffff0 x4 0xffffff8000626a94 uvm_map_inentry_sp x5 0x2 x6 0 x7 0 x8 0xffffff8011b3f400 x9 0xffffff800061db50 ampintc_ipi_ddb x10 0xb0 $d.1+0x9d x11 0xffffff8011b44000 x12 0xffffff8011b3f400 x13 0xc0000 $d.1+0xbffed x14 0xfc000 $d.1+0xfbfed x15 0x4d4223665a5946d x16 0x901933e58 x17 0x8f708f7c4 x18 0xffffff802a4a7bf0 x19 0 x20 0x1 x21 0xffffff802a4a7ca0 x22 0xffffff8000f0b000 pfr_mask+0x18 x23 0 x24 0xffffff8011b41580 x25 0 x26 0x28 $d.1+0x15 x27 0x85f7c1000 x28 0x40 $d.1+0x2d x29 0xffffff802a4a7bf0 x30 0 sp 0xffffff802a4a7bf0 spsr 0xa0000345 elr 0xffffff80009ef9d8 db_enter+0x14 lr 0xffffff800061db6c ampintc_ipi_ddb+0x1c ampintc_ipi_ddb+0x1c: ldr x15, [sp,#16] ddb{2}> machine ddbcpu 3 Stopped at ampintc_ipi_ddb+0x1c: ldr x15, [sp,#16] db_enter() at ampintc_ipi_ddb+0x18 ampintc_ipi_ddb() at ampintc_irq_handler+0x1c4 ampintc_irq_handler() at arm_cpu_intr+0x30 arm_cpu_intr() at handle_el1h_irq+0x6c handle_el1h_irq() at sched_idle+0x290 sched_idle() at sched_idle+0x290 sched_idle() at proc_trampoline+0x10 ddb{3}> show reg x0 0xffffff8011b3f400 x1 0xffffff8028413000 x2 0x4 x3 0xfffffff0 x4 0xffffff800096b2a0 smc_call x5 0xffffff802847d000 x6 0xb02bb6d38bee4a07 x7 0 x8 0xffffff8011b3f400 x9 0xffffff800061db50 ampintc_ipi_ddb x10 0xb0 $d.1+0x9d x11 0xffffff8011b44800 x12 0xffffff8011b3f400 x13 0 x14 0x3b9aca00 x15 0x4d4223665a5946d x16 0x14 $d.1+0x1 x17 0x8f70b0430 x18 0xffffff8028481c20 x19 0 x20 0x1 x21 0xffffff8028481cd0 x22 0xffffff8000f0b000 pfr_mask+0x18 x23 0 x24 0xffffff8011b41580 x25 0 x26 0x28 $d.1+0x15 x27 0x3 x28 0x1 x29 0xffffff8028481c20 x30 0 sp 0xffffff8028481c20 spsr 0xa0000345 elr 0xffffff80009ef9d8 db_enter+0x14 lr 0xffffff800061db6c ampintc_ipi_ddb+0x1c ampintc_ipi_ddb+0x1c: ldr x15, [sp,#16] ddb{0}> ps PID TID PPID UID S FLAGS WAIT COMMAND *81386 20589 33678 0 7 0x11 perl 43340 231344 34457 55 3 0x2 bqwait ld.lld 43340 359315 34457 55 3 0x4000082 fsleep ld.lld 43340 38601 34457 55 3 0x4000082 fsleep ld.lld 43340 231750 34457 55 3 0x4000082 fsleep ld.lld 43340 377913 34457 55 3 0x4000082 fsleep ld.lld 34457 462205 27690 55 3 0x82 wait cc 27690 401886 15933 55 3 0x200082 poll miniruby 27690 453437 15933 55 3 0x4200082 poll miniruby 15933 459398 75514 55 3 0x10008a sigsusp sh 75514 78812 48167 55 3 0x10008a sigsusp make 48167 96186 58908 55 3 0x10008a sigsusp sh 58908 454846 6135 55 3 0x10008a sigsusp make 6135 50049 96543 55 3 0x10008a sigsusp make 96543 478499 46118 55 3 0x10008a sigsusp sh 46118 166608 33755 55 3 0x10008a sigsusp make 33755 239357 94329 55 3 0x10008a sigsusp sh 94329 475508 33678 55 3 0x10008a sigsusp make 7568 16158 56753 1000 3 0x100083 poll top 56753 279171 64123 1000 3 0x10008b sigsusp ksh 33678 506484 69489 0 7 0x13 perl 69489 40952 71215 0 3 0x10008b sigsusp ksh 71215 352113 64123 1000 3 0x10008b sigsusp ksh 64123 446796 1 1000 3 0x100080 kqread tmux 7254 474451 1 0 3 0x100083 ttyin getty 65057 5194 1 0 3 0x100098 poll cron 3381 54908 1 99 3 0x100090 poll sndiod 15623 29370 1 110 3 0x100090 poll sndiod 48070 373922 93618 95 3 0x100092 kqread smtpd 5392 63995 93618 103 3 0x100092 kqread smtpd 2129 47763 93618 95 3 0x100092 kqread smtpd 62156 75475 93618 95 3 0x100092 kqread smtpd 58977 471444 93618 95 3 0x100092 kqread smtpd 43789 417562 93618 95 3 0x100092 kqread smtpd 93618 341463 1 0 3 0x100080 kqread smtpd 3471 475173 1 0 3 0x80 select sshd 94214 42281 1 0 3 0x100080 poll ntpd 40001 291382 23730 83 3 0x100092 poll ntpd 23730 5990 1 83 3 0x100092 poll ntpd 15283 133676 57214 74 3 0x100092 bpf pflogd 57214 285694 1 0 3 0x80 netio pflogd 62987 308017 9962 73 3 0x100090 kqread syslogd 9962 283510 1 0 3 0x100082 netio syslogd 97453 382325 77967 115 3 0x100092 kqread slaacd 84901 495061 77967 115 3 0x100092 kqread slaacd 77967 283269 1 0 3 0x100080 kqread slaacd 8247 475743 0 0 3 0x14200 bored smr 41206 421626 0 0 3 0x14200 pgzero zerothread 47538 512418 0 0 3 0x14200 aiodoned aiodoned 86063 391830 0 0 3 0x14200 bqwait update 61385 219501 0 0 3 0x14200 cleaner cleaner 9044 244147 0 0 3 0x14200 reaper reaper 40386 229370 0 0 3 0x14200 pgdaemon pagedaemon 64029 172554 0 0 3 0x14200 bored crynlk 99225 105130 0 0 3 0x14200 bored crypto 92412 24463 0 0 3 0x40014200 acpi0 acpi0 27015 467716 0 0 3 0x14200 usbtsk usbtask 61051 95902 0 0 3 0x14200 usbatsk usbatsk 99634 450063 0 0 7 0x40014200 idle3 6801 268986 0 0 3 0x40014200 idle2 90550 464913 0 0 7 0x40014200 idle1 67752 457826 0 0 3 0x14200 bored softnet 48397 377854 0 0 3 0x14200 bored systqmp 88690 230054 0 0 3 0x14200 bored systq 14185 429422 0 0 3 0x40014200 bored softclock 88850 59880 0 0 3 0x40014200 idle0 2994 322564 0 0 3 0x14200 kmalloc kmthread 1 362803 0 0 3 0x82 wait init 0 0 -1 0 3 0x10200 scheduler swapper ddb{3}> show uvm Current UVM status: pagesize=4096 (0x1000), pagemask=0xfff, pageshift=12 1770326 VM pages: 35282 active, 85064 inactive, 1 wired, 1126150 free (141726 zero) min 10% (25) anon, 10% (25) vnode, 5% (12) vtext freemin=59010, free-target=78680, inactive-target=78681, wired-max=590108 faults=161531384, traps=0, intrs=0, ctxswitch=5770326 fpuswitch=0 softint=4167971, syscalls=108978586, kmapent=16 fault counts: noram=0, noanon=0, noamap=0, pgwait=0, pgrele=0 ok relocks(total)=318962(319600), anget(retries)=60066360(0), amapcopy=3648 0996 neighbor anon/obj pg=5542462/87231636, gets(lock/unlock)=68211676/319600 cases: anon=54148894, anoncow=5917466, obj=66495873, prcopy=1715165, przero =33254007 daemon and swap counts: woke=35, revs=0, scans=0, obscans=0, anscans=0 busy=0, freed=0, reactivate=0, deactivate=0 pageouts=0, pending=0, nswget=0 nswapdev=1 swpages=263063, swpginuse=0, swpgonly=0 paging=0 kernel pointers: objs(kern)=0xffffff8000e912c0 ddb{3}> show bcstats Current Buffer Cache status: numbufs 52574 busymapped 0, delwri 37 kvaslots 235 avail kva slots 235 bufpages 484414, dmapages 3702, dirtypages 320 pendingreads 0, pendingwrites 0 highflips 217185, highflops 0, dmaflips 22564 -- Best Regards Edd Barrett http://www.theunixzoo.co.uk |
«
Return to openbsd dev - bugs
|
1 view|%1 views
| Free forum by Nabble | Edit this page |