Rsync is too slow

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Rsync is too slow

Rupert Gallagher
Latest obsd with new 4TB wd red ssd disk copying from 2TB seagate exos returns 80GB in 8 hours with zero activity by other tasks. The server has 12GB ecc ram cache. Copying 1.4 TB from a nas to the same exos took 2.5 hours shy. Is there a problem with how obsd handles internal storage? Or a problem with the default kernel sysconfig and staff defaults?
Reply | Threaded
Open this post in threaded view
|

Re: Rsync is too slow

Chris Cappuccio
Rupert Gallagher [[hidden email]] wrote:
> Latest obsd with new 4TB wd red ssd disk copying from 2TB seagate exos returns 80GB in 8 hours with zero activity by other tasks. The server has 12GB ecc ram cache. Copying 1.4 TB from a nas to the same exos took 2.5 hours shy. Is there a problem with how obsd handles internal storage? Or a problem with the default kernel sysconfig and staff defaults?

sounds like you are copying from SATA to USB ? first, send a dmesg so
people get a better idea. second, try -current. several improvements
have been made for USB to be faster.

Reply | Threaded
Open this post in threaded view
|

Re: Rsync is too slow

Rupert Gallagher
Attached.

dmesg.txt (8K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Rsync is too slow

Rupert Gallagher
In reply to this post by Chris Cappuccio
No, I am not using USB.

Reply | Threaded
Open this post in threaded view
|

Re: Rsync is too slow

Chris Cappuccio
Rupert Gallagher [[hidden email]] wrote:
> No, I am not using USB.

rsync between disks should be very fast. you are going from the sata to the
nvme ? NetBSD or FreeBSD or somebody made some speed improvements to nvme
that we should review. i can't remember right now. anyways, 10GB/hour sounds
extremely slow for an nvme SSD, way way way too slow for anything I have
experienced in recent memory.

it might be interesting to try using cp between filesystems, or tar

such as: cp -r /usr/bin /mnt/usr/bin
or: tar cf - -C /usr/bin . | tar xpf - -C /mnt/usr/bin

also what speeds are you getting on the destination filesystem?

dd count=1 bs=1G if=/dev/zero of=/mnt/test conv=fsync

might give you some rough idea of what 1G write costs.

here's 1G write on my Samsung 845DC Pro which is one of my all-time favorite
SATA SSDs for reliability

# dd count=1 bs=1G if=/dev/zero of=test conv=fsync
1+0 records in
1+0 records out
1073741824 bytes transferred in 2.906 secs (369450372 bytes/sec)

here's the same for a Crucial M500

# dd count=1 bs=1G if=/dev/zero of=test conv=fsync
1+0 records in
1+0 records out
1073741824 bytes transferred in 4.356 secs (246484472 bytes/sec)

it's not clear to me how much the buffer cache affects this but i'm hoping
here that conv=fsync helps. in a wierd twist, tests like this with conv=fsync
run consistently faster than without, so my understanding isn't that great.

Reply | Threaded
Open this post in threaded view
|

Re: Rsync is too slow

Chris Cappuccio
In reply to this post by Rupert Gallagher
Rupert Gallagher [[hidden email]] wrote:
> No, I am not using USB.

your dmesg didn't make it to the list because you are attaching a text file
and attachments are not allowed on misc.

please put it inline with the message.

Reply | Threaded
Open this post in threaded view
|

Re: Rsync is too slow

Todd C. Miller-3
On Thu, 30 Jul 2020 13:37:39 -0700, Chris Cappuccio wrote:

> Rupert Gallagher [[hidden email]] wrote:
> > No, I am not using USB.
>
> your dmesg didn't make it to the list because you are attaching a text file
> and attachments are not allowed on misc.

Actually, these days they are allowed.  I didn't have any problem
reading the attached dmesg.

 - todd

Reply | Threaded
Open this post in threaded view
|

Re: Rsync is too slow

Chris Cappuccio
In reply to this post by Chris Cappuccio
Chris Cappuccio [[hidden email]] wrote:
>
> such as: cp -r /usr/bin /mnt/usr/bin
> or: tar cf - -C /usr/bin . | tar xpf - -C /mnt/usr/bin
>

also the destination filesystem should be mounted with async (dangerous on
power loss) or softdep (not very dangerous on power loss) to avoid huge
amounts of metadata updates slowing your action.

Reply | Threaded
Open this post in threaded view
|

Re: Rsync is too slow

Rupert Gallagher
In reply to this post by Chris Cappuccio
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday 30 July 2020 22:36, Chris Cappuccio <[hidden email]> wrote:

> Rupert Gallagher [[hidden email]] wrote:
>
> > No, I am not using USB.
>
> rsync between disks should be very fast.

Right.

> you are going from the sata to the nvme ?

No. It is SATA to SATA, using a M14TQC with a Mini-SAS HD to 4 SATA cable:

https://www.supermicro.com/en/products/accessories/mobilerack/CSE-M14TQC.php

> it might be interesting to try using cp between filesystems, or tar
>
> such as: cp -r /usr/bin /mnt/usr/bin
> or: tar cf - -C /usr/bin . | tar xpf - -C /mnt/usr/bin
>
> also what speeds are you getting on the destination filesystem?
>
> dd count=1 bs=1G if=/dev/zero of=/mnt/test conv=fsync
>
> might give you some rough idea of what 1G write costs.
>
> here's 1G write on my Samsung 845DC Pro which is one of my all-time favorite
> SATA SSDs for reliability
>
> dd count=1 bs=1G if=/dev/zero of=test conv=fsync
>
> =================================================
>
> 1+0 records in
> 1+0 records out
> 1073741824 bytes transferred in 2.906 secs (369450372 bytes/sec)
>
> here's the same for a Crucial M500
>
> dd count=1 bs=1G if=/dev/zero of=test conv=fsync
>
> =================================================
>
> 1+0 records in
> 1+0 records out
> 1073741824 bytes transferred in 4.356 secs (246484472 bytes/sec)
>
> it's not clear to me how much the buffer cache affects this but i'm hoping
> here that conv=fsync helps. in a wierd twist, tests like this with conv=fsync
> run consistently faster than without, so my understanding isn't that great.

Yours are NVMe. I have an SSD on a SATA bus.

This is my result:

>doas dd count=1 bs=1G if=/dev/zero of=/archive2/test conv=fsync
1+0 records in
1+0 records out
1073741824 bytes transferred in 8.118 secs (132261121 bytes/sec)

>grep archive2 /etc/fstab
[label].a /archive2 ffs rw,nodev,nosuid,softdep,noatime 1 2

However, 1G is not enough to go past the cache in ram.

This is what I do:

write test
doas /bin/dd if=/dev/zero of=$testfile bs=$(( $bs * 1024 )) count=$count conv=sync

read test
doas /bin/dd if=$testfile of=/dev/null bs=$(( $bs * 1024 )) conv=sync

where

$testfile = /archive2/test for example
$bs=$(( stat -f "%k" /dev/sd1a )) where sd1a is the device of /archive2

count=$(( $ram / ( $bs * 1024 ) )); # ram expressed in blocks
count=$(( $count + 1 )); # exceed ram by 1 block

This is the speed test on WDS400T1R0A (WD RED SSD 4TB)

Free disk space    : 3151013175296 bytes
RAM                : 17125511168 bytes
fs block size      : 8192 bytes
Size of test file  : 17129537536 bytes = 2042 block(s) of 8192K
Test file          : /archive2/disk-speed-test.raw

Writing speed      : 182 MB/s
Reading speed      : 109 MB/s

The product brief of WD RED SSD says "560MB/s read" and "530MB/s write".

By comparison, this is the speed test on the ST2000NX0403 (Seagate Exos 2TB)

Writing speed      : 117 MB/s
Reading speed      : 99 MB/s

The product brief of the Exos says "136MB/s" max transfer.

Both the exos and the wd red have hardware bytes/sector of 512.

This is how I prepared both, with details for the wd red ssd only:

> fdisk -iy -g sd1

>echo "/  1G-*  100%" >/tmp/my_disk_label
>disklabel -w -A -T /tmp/my_disk_label sd1

> disklabel -hn sd1
# /dev/rsd1c:
type: SCSI
disk: SCSI disk
label: WDC  WDS400T1R0A
duid: b8d30be7c118b250
flags:
bytes/sector: 512
sectors/track: 255
tracks/cylinder: 511
sectors/cylinder: 130305
cylinders: 59967
total sectors: 7814037168 # total bytes: 3.6T
boundstart: 64
boundend: 7814037105
drivedata: 0

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  a:             3.6T               64  4.2BSD   8192 65536     1
  c:             3.6T                0  unused

> newfs -O2 sd1a
/dev/rsd1a: 3815447.8MB in 7814036928 sectors of 512 bytes
1168 cylinder groups of 3266.88MB, 52270 blocks, 104704 inodes each
super-block backups (for fsck -b #) at:
[omissis]

> dumpfs /dev/sd1a | head -19

magic   19540119 (FFS2) time    Wed Jul 29 18:41:40 2020
superblock location     65536   id      [ 5f21a6c4 bb9dec49 ]
ncg     1168    size    488377308       blocks  484536905
bsize   65536   shift   16      mask    0xffff0000
fsize   8192    shift   13      mask    0xffffe000
frag    8       shift   3       fsbtodb 4
minfree 5%      optim   time    symlinklen 120
maxbsize 0      maxbpg  8192    maxcontig 1     contigsumsize 0
nbfree  60567111        ndir    1       nifree  122294269       nffree  16
bpg     52270   fpg     418160  ipg     104704
nindir  8192    inopb   256     maxfilesize     36033195603132415
sbsize  8192    cgsize  65536   csaddr  3304    cssize  24576
sblkno  16      cblkno  24      iblkno  32      dblkno  3304
cgrotor 0       fmod    0       ronly   0       clean   1
avgfpdir 64     avgfilesize 16384
flags   none
fsmnt
volname         swuid   0

Finally, this is how I use rsync:

#!/bin/sh
from="$1";
to="$2";
if [[ "$from" == "" || "$to" == "" ]]; then
   echo "usage: copy /from /to":
   exit 1;
fi
doas /usr/bin/nice -n 11 rsync --recursive --links --times --modify-window=1 -O -J --devices --specials --update --super --owner --group --perms --delete --delete-before --delete-excluded --exclude-from=/etc/excluded_from_backup.conf --numeric-ids --compress-level=2 --outbuf=Block --inplace $from/ $to/;
doas rm gmon.out

Again, the SSD is brand new, just prepared, and writing on it for the very first time, so write amplification should not happen.

Reply | Threaded
Open this post in threaded view
|

Re: Rsync is too slow

Rupert Gallagher
In reply to this post by Chris Cappuccio
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Thursday 30 July 2020 22:37, Chris Cappuccio <[hidden email]> wrote:

> Rupert Gallagher [[hidden email]] wrote:
>
> > No, I am not using USB.
>
> your dmesg didn't make it to the list because you are attaching a text file
> and attachments are not allowed on misc.
>
> please put it inline with the message.


OK


OpenBSD 6.7 (GENERIC.MP) #2: Thu Jun  4 09:55:08 MDT 2020
    [hidden email]:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 17125511168 (16332MB)
avail mem = 16593870848 (15825MB)
mpath0 at root
scsibus0 at mpath0: 256 targets
mainbus0 at root
bios0 at mainbus0: SMBIOS rev. 3.0 @ 0x7f0c3000 (34 entries)
bios0: vendor American Megatrends Inc. version "1.2" date 11/05/2019
bios0: Supermicro A2SDi-4C-HLN4F
acpi0 at bios0: ACPI 6.1
acpi0: sleep states S0 S4 S5
acpi0: tables DSDT FACP FPDT FIDT SPMI MCFG WDAT APIC BDAT HPET UEFI SSDT SSDT SSDT DMAR SPCR HEST BERT ERST EINJ WSMT
acpi0: wakeup devices XHC1(S4) OBL1(S4) LAN1(S4) PEX0(S4) LAN2(S4) LAN3(S4) PEX1(S4) PEX6(S4) PEX7(S4)
acpitimer0 at acpi0: 3579545 Hz, 24 bits
acpimcfg0 at acpi0
acpimcfg0: addr 0xe0000000, bus 0-255
acpimadt0 at acpi0 addr 0xfee00000: PC-AT compat
cpu0 at mainbus0: apid 4 (boot processor)
cpu0: Intel(R) Atom(TM) CPU C3558 @ 2.20GHz, 1104.90 MHz, 06-5f-01
cpu0: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SMEP,ERMS,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,SHA,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu0: 2MB 64b/line 16-way L2 cache
cpu0: cannot disable silicon debug
cpu0: smt 0, core 2, package 0
mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
cpu0: apic clock running at 25MHz
cpu0: mwait min=64, max=64, C-substates=0.2.0.2, IBE
cpu1 at mainbus0: apid 12 (application processor)
cpu1: Intel(R) Atom(TM) CPU C3558 @ 2.20GHz, 1100.10 MHz, 06-5f-01
cpu1: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SMEP,ERMS,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,SHA,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu1: 2MB 64b/line 16-way L2 cache
cpu1: cannot disable silicon debug
cpu1: smt 0, core 6, package 0
cpu2 at mainbus0: apid 16 (application processor)
cpu2: Intel(R) Atom(TM) CPU C3558 @ 2.20GHz, 1100.10 MHz, 06-5f-01
cpu2: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SMEP,ERMS,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,SHA,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu2: 2MB 64b/line 16-way L2 cache
cpu2: cannot disable silicon debug
cpu2: smt 0, core 8, package 0
cpu3 at mainbus0: apid 24 (application processor)
cpu3: Intel(R) Atom(TM) CPU C3558 @ 2.20GHz, 1100.10 MHz, 06-5f-01
cpu3: FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,DS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE,SSE3,PCLMUL,DTES64,MWAIT,DS-CPL,VMX,EST,TM2,SSSE3,SDBG,CX16,xTPR,PDCM,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,DEADLINE,AES,XSAVE,RDRAND,NXE,PAGE1GB,RDTSCP,LONG,LAHF,3DNOWP,PERF,ITSC,FSGSBASE,TSC_ADJUST,SMEP,ERMS,MPX,RDSEED,SMAP,CLFLUSHOPT,PT,SHA,MD_CLEAR,IBRS,IBPB,STIBP,SSBD,SENSOR,ARAT,XSAVEOPT,XSAVEC,XGETBV1,XSAVES
cpu3: 2MB 64b/line 16-way L2 cache
cpu3: cannot disable silicon debug
cpu3: smt 0, core 12, package 0
ioapic0 at mainbus0: apid 2 pa 0xfec00000, version 20, 24 pins
acpihpet0 at acpi0: 23999999 Hz
acpiprt0 at acpi0: bus 0 (PCI0)
acpiprt1 at acpi0: bus 6 (VRP0)
acpiprt2 at acpi0: bus 2 (PEX0)
acpiprt3 at acpi0: bus 1 (VRP2)
acpiprt4 at acpi0: bus 7 (VRP1)
acpiprt5 at acpi0: bus -1 (PEX1)
acpiprt6 at acpi0: bus 3 (PEX6)
acpiprt7 at acpi0: bus 4 (PEX7)
acpiprt8 at acpi0: bus 5 (BR28)
acpicpu0 at acpi0: C2(10@50 mwait.1@0x21), C1(1000@1 mwait.1@0x1), PSS
acpicpu1 at acpi0: C2(10@50 mwait.1@0x21), C1(1000@1 mwait.1@0x1), PSS
acpicpu2 at acpi0: C2(10@50 mwait.1@0x21), C1(1000@1 mwait.1@0x1), PSS
acpicpu3 at acpi0: C2(10@50 mwait.1@0x21), C1(1000@1 mwait.1@0x1), PSS
acpipci0 at acpi0 PCI0: 0x00000010 0x00000011 0x00000000
"PNP0003" at acpi0 not configured
acpicmos0 at acpi0
"IPI0001" at acpi0 not configured
"PNP0C33" at acpi0 not configured
ipmi at mainbus0 not configured
cpu0: Enhanced SpeedStep 1104 MHz: speeds: 2200, 2100, 2000, 1900, 1800, 1700, 1600, 1500, 1400, 1300, 1200, 1100, 1000, 900, 800 MHz
pci0 at mainbus0 bus 0
0:31:5: mem address conflict 0xfe010000/0x1000
pchb0 at pci0 dev 0 function 0 "Intel C3000 Host" rev 0x11
pchb1 at pci0 dev 4 function 0 "Intel C3000 GLREG" rev 0x11
"Intel C3000 RCEC" rev 0x11 at pci0 dev 5 function 0 not configured
ppb0 at pci0 dev 6 function 0 "Intel C3000 PCIE" rev 0x11
pci1 at ppb0 bus 1
"Intel C3000 QAT" rev 0x11 at pci1 dev 0 function 0 not configured
ppb1 at pci0 dev 9 function 0 "Intel C3000 PCIE" rev 0x11: msi
pci2 at ppb1 bus 2
ppb2 at pci0 dev 16 function 0 "Intel C3000 PCIE" rev 0x11
pci3 at ppb2 bus 3
nvme0 at pci3 dev 0 function 0 "SanDisk WD Black NVMe" rev 0x00: msix, NVMe 1.3
nvme0: WDS250G2X0C-00L350, firmware 101110WD, serial 181644433343
scsibus1 at nvme0: 2 targets, initiator 0
sd0 at scsibus1 targ 1 lun 0: <NVMe, WDS250G2X0C-00L3, 1011>
sd0: 238475MB, 512 bytes/sector, 488397168 sectors
ppb3 at pci0 dev 17 function 0 "Intel C3000 PCIE" rev 0x11
pci4 at ppb3 bus 4
ppb4 at pci4 dev 0 function 0 "ASPEED Technology AST1150 PCI" rev 0x03
pci5 at ppb4 bus 5
"ASPEED Technology AST2000" rev 0x30 at pci5 dev 0 function 0 not configured
"Intel C3000 SMBus" rev 0x11 at pci0 dev 18 function 0 not configured
ahci0 at pci0 dev 19 function 0 "Intel C3000 AHCI" rev 0x11: msi, AHCI 1.3.1
ahci0: port 0: 6.0Gb/s
ahci0: port 1: 6.0Gb/s
ahci0: port 2: 6.0Gb/s
ahci0: PHY offline on port 3
scsibus2 at ahci0: 32 targets
sd1 at scsibus2 targ 0 lun 0: <ATA, WDC WDS400T1R0A, 4110> naa.5001b444a777b750
sd1: 3815447MB, 512 bytes/sector, 7814037168 sectors, thin
sd2 at scsibus2 targ 1 lun 0: <ATA, ST2000NX0403, TN04> naa.5000c500b98ef9ce
sd2: 1907729MB, 512 bytes/sector, 3907029168 sectors
sd3 at scsibus2 targ 2 lun 0: <ATA, ST5000LM000-2AN1, 0001> naa.5000c500c3ad5c90
sd3: 4769307MB, 512 bytes/sector, 9767541168 sectors
xhci0 at pci0 dev 21 function 0 "Intel C3000 xHCI" rev 0x11: msi, xHCI 1.0
usb0 at xhci0: USB revision 3.0
uhub0 at usb0 configuration 1 interface 0 "Intel xHCI root hub" rev 3.00/1.00 addr 1
ppb5 at pci0 dev 22 function 0 "Intel C3000 PCIE" rev 0x11
pci6 at ppb5 bus 6
ix0 at pci6 dev 0 function 0 "Intel X553 SGMII" rev 0x11: msi, address ac:1f:6b:49:6a:30
ix1 at pci6 dev 0 function 1 "Intel X553 SGMII" rev 0x11: msi, address ac:1f:6b:49:6a:31
ppb6 at pci0 dev 23 function 0 "Intel C3000 PCIE" rev 0x11
pci7 at ppb6 bus 7
ix2 at pci7 dev 0 function 0 "Intel X553 SGMII" rev 0x11: msi, address ac:1f:6b:49:6a:32
ix3 at pci7 dev 0 function 1 "Intel X553 SGMII" rev 0x11: msi, address ac:1f:6b:49:6a:33
"Intel C3000 ME HECI" rev 0x11 at pci0 dev 24 function 0 not configured
pcib0 at pci0 dev 31 function 0 "Intel C3000 LPC" rev 0x11
"Intel C3000 PMC" rev 0x11 at pci0 dev 31 function 2 not configured
"Intel C3000 SMBus" rev 0x11 at pci0 dev 31 function 4 not configured
"Intel C3000 SPI" rev 0x11 at pci0 dev 31 function 5 not configured
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
com1: console
pcppi0 at isa0 port 0x61
spkr0 at pcppi0
vmm0 at mainbus0: VMX/EPT
efifb0 at mainbus0: 1024x768, 32bpp
wsdisplay0 at efifb0 mux 1
wsdisplay0: screen 0-5 added (std, vt100 emulation)
uhidev0 at uhub0 port 1 configuration 1 interface 0 "EATON Ellipse ECO" rev 2.00/1.00 addr 2
uhidev0: iclass 3/0, 32 report ids
upd0 at uhidev0
uhub1 at uhub0 port 3 configuration 1 interface 0 "Genesys Logic USB2.0 Hub" rev 2.00/32.98 addr 3
uhub2 at uhub1 port 1 configuration 1 interface 0 "ATEN International product 0x7000" rev 2.00/0.00 addr 4
uhidev1 at uhub2 port 1 configuration 1 interface 0 "ATEN International product 0x2419" rev 1.10/1.00 addr 5
uhidev1: iclass 3/1
ukbd0 at uhidev1: 8 variable keys, 6 key codes
wskbd0 at ukbd0 mux 1
wskbd0: connecting to wsdisplay0
uhidev2 at uhub2 port 1 configuration 1 interface 1 "ATEN International product 0x2419" rev 1.10/1.00 addr 5
uhidev2: iclass 3/1
ums0 at uhidev2: 3 buttons, Z dir
wsmouse0 at ums0 mux 0
uhub3 at uhub1 port 2 configuration 1 interface 0 "Genesys Logic USB2.0 Hub" rev 2.00/32.98 addr 6
vscsi0 at root
scsibus3 at vscsi0: 256 targets
softraid0 at root
scsibus4 at softraid0: 256 targets
root on sd0a (e8bf1d24214b85cb.a) swap on sd0b dump on sd0b