Installing OpenBSD 6.5 sparc64 onto RAID1

[Chris Ross]

Hello all.  I have a Sparc T2 server that I was able to install OpenBSD 6.5
onto, but wanted to install it on a RAID1 softraid instead.  Unfortunately,
I have not found enough conclusive information in searching the interwebs.

I followed https://www.openbsd.org/faq/faq14.html#softraidDI and other
than ignoring the "fdisk" commands which I didn't have, was able to get an
sd4 softraid configured to cover two internal disks (each one big partition).
An install to sd4 completed sucessfully, but when I reboot, trying to
boot off of either of the disks the softraid spans yields:


{0} ok boot disk3
Boot device: /pci@0/pci@0/pci@2/scsi@0/disk@3  File and args:
OpenBSD IEEE 1275 Bootblock 1.4
>> OpenBSD BOOT 1.12
ERROR: /iscsi-hba: No iscsi-network-bootpath property
Trying bsd...
open /pci@0/pci@0/pci@2/scsi@0/disk@3,0:a/etc/random.seed: Invalid argument
open /pci@0/pci@0/pci@2/scsi@0/disk@3,0:a/bsd: Invalid argument

Boot:


I looked around a little more, and found
http://brycv.com/blog/2012/openbsd-sparc64-and-root-on-softraid/ but that
is quite old, and sounds like it's reflective of the time that even 386/amd64
installs recommended separate partitions for root and swap aside the RAID,
which is not true currently.

So, does anyone have a recent walkthrough of how to install OpenBSD 6 onto
a sparc RAID1?  Thanks much.

                     - Chris

[Chris Ross]

On Sat, Jun 29, 2019 at 01:35:52PM -0400, Gao-Mi Baohao wrote:
> On openbsd sparc64, your RAID slice must be slice 'a' on your physical
> disks, unless you use a separate boot partition (ref: source for the
> sparc64 stage0).

Sorry if I didn't say that, but it is.  As mentioned at the FAQ, I did

bioctl -c 1 -l sd2a,sd3a softraid0

And then installed onto my softraid, sd4.

> Once you've got the system installed, you can boot with e.g. "boot disk0
> sr0a:/bsd" (slice a of the consolidated softraid volume) and to make the
> config persistent, set openboot env boot-device = disk0 and boot-file =
> sr0a:/bsd .

I saw this, but presumed sr0a wasn't literal.  I did try that, but it
says:

{0} ok boot disk2 sr0a:/bsd
Boot device: /pci@0/pci@0/pci@2/scsi@0/disk@2  File and args: sr0a:/bsd
OpenBSD IEEE 1275 Bootblock 1.4
>> OpenBSD BOOT 1.12
ERROR: /iscsi-hba: No iscsi-network-bootpath property
Unknown device: sr0
Cannot boot from softraid: Unknown error: code 19
Program terminated
{0} ok

If I go back to trying to boot off of disk2 or disk3, specifying
sd2a:/bsd, sd3a:/bsd, or sd4a:/bsd as boot-file, I simply am seeing:


OpenBSD BOOT 1.12
ERROR: /iscsi-hba: No iscsi-network-bootpath property
open /pci@0/pci@0/pci@2/scsi@0/disk@2,0:a/etc/random.seed: Invalid argument
open /pci@0/pci@0/pci@2/scsi@0/disk@2,0:a/sd4a:/bsd: Invalid argument

or

open /pci@0/pci@0/pci@2/scsi@0/disk@2,0:a/sd2a:/bsd: Invalid argument

or whatever I give as a boot-file value.

So whatever the bootloader needs to see to find the filesystem seems not to
be working.  The RAID filesystem on sd2a and sd3a, on which the softraid is
built, seems not to be able to be loaded from.

Apologies if I'm missing something in the advice you responded with.

> > So, does anyone have a recent walkthrough of how to install OpenBSD 6 onto
> > a sparc RAID1?  Thanks much.

        - Chris

[Chris Ross]

On Sun, Jun 30, 2019 at 01:34:53AM -0400, Gao-Mi Baohao wrote:
> "bioctl -c 1 -l sd2a,sd3a softraid0"
>
> ... And those were both set to slices of type "RAID" when you ran disklabel
> (which is needed for the loader to discover them) as opposed to type
> "4.2BSD" or such?

I'm sure they were once, but I initialized the disks multiple times.
Booting it back up to verify now.  Yup, both drives show:

# disklabel sd2
# /dev/rsd2c:
type: SCSI
[...]
total sectors: 286739329

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  a:        286739328                0    RAID                    
  c:        286739329                0  unused                    
#

> If so, then I'm officially stumped.  Running a virtually identical setup
> successfully on my T5140 and Blade 1000.  Clearly I've missed something,
> but what that might be does not occur to me.

:-/

So after experimenting yesterday, I think my dated recollection of "srN"
meaning "SCSI removable" caused me to presume it incorrect.  I was trying
"sd2a/BSD" and "sd4a/bsd", but those give me unusual errors.  If "srN"
instead means "soft RAID N", then "sr0a" is correct, and I hadn't originally
tried that.  But, as noted, that gives me:

{0} ok boot disk2 sr0a/bsd
Boot device: /pci@0/pci@0/pci@2/scsi@0/disk@2  File and args: sr0a/bsd
OpenBSD IEEE 1275 Bootblock 1.4
>> OpenBSD BOOT 1.12
ERROR: /iscsi-hba: No iscsi-network-bootpath property
Unknown device: sr0
Cannot boot from softraid: Unknown error: code 19
Program terminated
{0} ok

So, maybe I should look at that error? Let me know if you can confirm
that interpretation of "sr0a", and if that's for sure what I should
be using with this raid across sd2a and sd3a.  Thanks!

       - Chris

[Gao-Mi Baohao]

> Let me know if you can confirm that interpretation of "sr0a"

Yes, "sr" == "softraid," but when I pulled the working config from my
systems, I discovered that there's an unintuitive syntax that I had
forgotten about, which is probeblty your missing piece.

It looks like I actually set boot-device = "disk0" and boot-file =
"sr0" (notice the omitted file and slice into), and the loader
auto-selected "sr0:a/bsd" (no that's not a typo... "sr" + [softraid
number] + ":" + [root slice on softraid dev] + "/bsd").

Give that a shot.  That's jogging my memory about having to dig
through source for the sparc64 loader to discover that very unintutive
syntax.

Here's my working config.

>
> ===PHYSICAL DISK PARTITION INFO===
> glasloc-obsd# disklabel -h /dev/rsd0c
> # /dev/rsd0c:
> [...]
> 16 partitions:
> #                size           offset  fstype [fsize bsize   cpg]
>   a:           931.5G                0    RAID
>   c:           931.5G                0  unused
> glasloc-obsd# bioctl softraid0
> Volume      Status               Size Device
> softraid0 0 Online      1000190238720 sd2     RAID1
>           0 Online      1000190238720 0:0.0   noencl <sd0a>
>           1 Online      1000190238720 0:1.0   noencl <sd1a>
>
> ===SOFTRAID DISK PARTITION INFO===
> glasloc-obsd# disklabel -h /dev/rsd2c
> # /dev/rsd2c:
> [...]
> 16 partitions:
> #                size           offset  fstype [fsize bsize   cpg]
>   a:             1.0G                0  4.2BSD   2048 16384 12958 # /
>   b:            28.3G          2104515    swap                    # none
>   c:           931.5G                0  unused
>   d:             4.0G         61352235  4.2BSD   2048 16384 12958 # /tmp
>   e:            60.0G         69754230  4.2BSD   2048 16384 12958 # /var
>   f:             2.0G        195591375  4.2BSD   2048 16384 12958 # /usr
>   g:             1.0G        199800405  4.2BSD   2048 16384 12958 # /usr/X11R6
>   h:            20.0G        201904920  4.2BSD   2048 16384 12958 # /usr/local
>   i:             2.0G        243850635  4.2BSD   2048 16384 12958 # /usr/src
>   j:            40.0G        248059665  4.2BSD   2048 16384 12958 # /usr/obj
>   k:            10.0G        331951095  4.2BSD   2048 16384 12958 # /usr/ports
>   l:             8.0G        352931985  4.2BSD   2048 16384 12958 # /usr/xenocara
>   m:           755.2G        369719910  4.2BSD   8192 65536 52238 # /home
>
> ===OPENBOOT VARIABLES===
> glasloc-obsd# eeprom boot-device
> boot-device=vdisk0
> glasloc-obsd# eeprom boot-file
> boot-file=sr0
>
> ===BOOT MESSAGES===
> T5140, No Keyboard
> Copyright (c) 1998, 2012, Oracle and/or its affiliates. All rights reserved.
> OpenBoot 4.33.6, <<OMITTED>> MB memory available, Serial #<<OMITTED>>.
> Ethernet address <<OMITTED>>, Host ID: <<OMITTED>>.
>
>
>
> Boot device: disk0  File and args: sr0
> OpenBSD IEEE 1275 Bootblock 1.4
> >> OpenBSD BOOT 1.12
> ERROR: /iscsi-hba: No iscsi-network-bootpath property
> sr0*
> Booting sr0:a/bsd
> <<OMITTED>>
> <<OMITTED>>
> [ using 1036112 bytes of bsd ELF symbol table ]
> console is /virtual-devices@100/console@1
> Copyright (c) 1982, 1986, 1989, 1991, 1993
>         The Regents of the University of California.  All rights reserved.
> Copyright (c) 1995-2019 OpenBSD. All rights reserved.  https://www.OpenBSD.org
>
> OpenBSD 6.5 (GENERIC.MP) #0: Tue Jun 18 18:13:03 EDT 2019

On Sun, Jun 30, 2019 at 10:55 AM Chris Ross <[hidden email]> wrote:

>
> On Sun, Jun 30, 2019 at 01:34:53AM -0400, Gao-Mi Baohao wrote:
> > "bioctl -c 1 -l sd2a,sd3a softraid0"
> >
> > ... And those were both set to slices of type "RAID" when you ran disklabel
> > (which is needed for the loader to discover them) as opposed to type
> > "4.2BSD" or such?
>
> I'm sure they were once, but I initialized the disks multiple times.
> Booting it back up to verify now.  Yup, both drives show:
>
> # disklabel sd2
> # /dev/rsd2c:
> type: SCSI
> [...]
> total sectors: 286739329
>
> 16 partitions:
> #                size           offset  fstype [fsize bsize   cpg]
>   a:        286739328                0    RAID
>   c:        286739329                0  unused
> #
>
> > If so, then I'm officially stumped.  Running a virtually identical setup
> > successfully on my T5140 and Blade 1000.  Clearly I've missed something,
> > but what that might be does not occur to me.
>
> :-/
>
> So after experimenting yesterday, I think my dated recollection of "srN"
> meaning "SCSI removable" caused me to presume it incorrect.  I was trying
> "sd2a/BSD" and "sd4a/bsd", but those give me unusual errors.  If "srN"
> instead means "soft RAID N", then "sr0a" is correct, and I hadn't originally
> tried that.  But, as noted, that gives me:
>
> {0} ok boot disk2 sr0a/bsd
> Boot device: /pci@0/pci@0/pci@2/scsi@0/disk@2  File and args: sr0a/bsd
> OpenBSD IEEE 1275 Bootblock 1.4
> >> OpenBSD BOOT 1.12
> ERROR: /iscsi-hba: No iscsi-network-bootpath property
> Unknown device: sr0
> Cannot boot from softraid: Unknown error: code 19
> Program terminated
> {0} ok
>
> So, maybe I should look at that error? Let me know if you can confirm
> that interpretation of "sr0a", and if that's for sure what I should
> be using with this raid across sd2a and sd3a.  Thanks!
>
>        - Chris

[Chris Ross]

On Sun, Jun 30, 2019 at 12:12:46PM -0400, Gao-Mi Baohao wrote:

> > Let me know if you can confirm that interpretation of "sr0a"
>
> Yes, "sr" == "softraid," but when I pulled the working config from my
> systems, I discovered that there's an unintuitive syntax that I had
> forgotten about, which is probeblty your missing piece.
>
> It looks like I actually set boot-device = "disk0" and boot-file =
> "sr0" (notice the omitted file and slice into), and the loader
> auto-selected "sr0:a/bsd" (no that's not a typo... "sr" + [softraid
> number] + ":" + [root slice on softraid dev] + "/bsd").
>
> Give that a shot.  That's jogging my memory about having to dig
> through source for the sparc64 loader to discover that very unintutive
> syntax.

Okay.  Thanks.  Well, I tried using "sr0", but am still getting the same
error.  Below.  Note, I checked my "bioctl softraid0" which looks like yours,
and the disklabel of my sd4 (the softraid disk) looks about like yours too.
a 1g 4.2BSD filesystem at offset 0.

Something is still wrong on my end.  So, before the difference in the output
of OBP boot command, I notice you said "disk0" above, but your eeprom output
below shows "vdisk0".  Do you have a vdisk0 devalias on your system?

After that, there's just the different response I see compared you yours.
I get:

Boot device: /pci@0/pci@0/pci@2/scsi@0/disk@2  File and args: sr0
OpenBSD IEEE 1275 Bootblock 1.4
>> OpenBSD BOOT 1.12
ERROR: /iscsi-hba: No iscsi-network-bootpath property
Unknown device: sr0
Cannot boot from softraid: Unknown error: code 19

And, you see:

sr0*
Booting sr0:a/bsd

where I see:

Unknown device: sr0
Cannot boot from softraid: Unknown error: code 19

Any idea why the bootloader says "sr0*" for you, and "Unknown device" for me?

Thanks...

            - Chris

> Here's my working config.
> >
> > ===OPENBOOT VARIABLES===
> > glasloc-obsd# eeprom boot-device
> > boot-device=vdisk0
> > glasloc-obsd# eeprom boot-file
> > boot-file=sr0
> >
> > ===BOOT MESSAGES===
> > T5140, No Keyboard
> > Copyright (c) 1998, 2012, Oracle and/or its affiliates. All rights reserved.
> > OpenBoot 4.33.6, <<OMITTED>> MB memory available, Serial #<<OMITTED>>.
> > Ethernet address <<OMITTED>>, Host ID: <<OMITTED>>.
> >
> >
> >
> > Boot device: disk0  File and args: sr0
> > OpenBSD IEEE 1275 Bootblock 1.4
> > >> OpenBSD BOOT 1.12
> > ERROR: /iscsi-hba: No iscsi-network-bootpath property
> > sr0*
> > Booting sr0:a/bsd
> > <<OMITTED>>
> > <<OMITTED>>
> > [ using 1036112 bytes of bsd ELF symbol table ]
> > console is /virtual-devices@100/console@1
> > Copyright (c) 1982, 1986, 1989, 1991, 1993
> >         The Regents of the University of California.  All rights reserved.
> > Copyright (c) 1995-2019 OpenBSD. All rights reserved.  https://www.OpenBSD.org
> >
> > OpenBSD 6.5 (GENERIC.MP) #0: Tue Jun 18 18:13:03 EDT 2019

[Gao-Mi Baohao]

The disk aliasing is a bit complicated, but short answer is yes- that's an
alias.  I have a bit of config in place so that I can boot the same disks
as both an LDom and on bare metal.  Results are generally the same either
way, as far as softraid is concerned.

As far as the loader messages go, I suspect that the loader simply isn't
detecting the softraid.  The final thing that occurs to me is installboot.
Have you run installboot on the softraid device itself?  Here's an example
of the output from my working setup (you'll likely need to specify paths
explicitly if you are working off the install image):

> glasloc-obsd# installboot -v sd2
> Using / as root
> installing bootstrap on /dev/rsd2c
> using first-stage /usr/mdec/bootblk, second-stage /usr/mdec/ofwboot
> boot block is 5840 bytes (12 blocks @ 512 bytes = 6144 bytes)
> sd2: softraid volume with 2 disk(s)
> sd2: installing boot loader on softraid volume
> sd0a: installing boot blocks on /dev/rsd0c
> writing boot block to disk /dev/rsd0c
> sd1a: installing boot blocks on /dev/rsd1c
> writing boot block to disk /dev/rsd1c
>

I thinks that's about the end of what I've got suggestion-wise.


On Sun, Jun 30, 2019 at 12:46 PM Chris Ross <[hidden email]>
wrote:

> On Sun, Jun 30, 2019 at 12:12:46PM -0400, Gao-Mi Baohao wrote:
> > > Let me know if you can confirm that interpretation of "sr0a"
> >
> > Yes, "sr" == "softraid," but when I pulled the working config from my
> > systems, I discovered that there's an unintuitive syntax that I had
> > forgotten about, which is probeblty your missing piece.
> >
> > It looks like I actually set boot-device = "disk0" and boot-file =
> > "sr0" (notice the omitted file and slice into), and the loader
> > auto-selected "sr0:a/bsd" (no that's not a typo... "sr" + [softraid
> > number] + ":" + [root slice on softraid dev] + "/bsd").
> >
> > Give that a shot.  That's jogging my memory about having to dig
> > through source for the sparc64 loader to discover that very unintutive
> > syntax.
>
> Okay.  Thanks.  Well, I tried using "sr0", but am still getting the same
> error.  Below.  Note, I checked my "bioctl softraid0" which looks like
> yours,
> and the disklabel of my sd4 (the softraid disk) looks about like yours too.
> a 1g 4.2BSD filesystem at offset 0.
>
> Something is still wrong on my end.  So, before the difference in the
> output
> of OBP boot command, I notice you said "disk0" above, but your eeprom
> output
> below shows "vdisk0".  Do you have a vdisk0 devalias on your system?
>
> After that, there's just the different response I see compared you yours.
> I get:
>
> Boot device: /pci@0/pci@0/pci@2/scsi@0/disk@2  File and args: sr0
> OpenBSD IEEE 1275 Bootblock 1.4
> >> OpenBSD BOOT 1.12
> ERROR: /iscsi-hba: No iscsi-network-bootpath property
> Unknown device: sr0
> Cannot boot from softraid: Unknown error: code 19
>
> And, you see:
>
> sr0*
> Booting sr0:a/bsd
>
> where I see:
>
> Unknown device: sr0
> Cannot boot from softraid: Unknown error: code 19
>
> Any idea why the bootloader says "sr0*" for you, and "Unknown device" for
> me?
>
> Thanks...
>
>             - Chris
>
> > Here's my working config.
> > >
> > > ===OPENBOOT VARIABLES===
> > > glasloc-obsd# eeprom boot-device
> > > boot-device=vdisk0
> > > glasloc-obsd# eeprom boot-file
> > > boot-file=sr0
> > >
> > > ===BOOT MESSAGES===
> > > T5140, No Keyboard
> > > Copyright (c) 1998, 2012, Oracle and/or its affiliates. All rights
> reserved.
> > > OpenBoot 4.33.6, <<OMITTED>> MB memory available, Serial #<<OMITTED>>.
> > > Ethernet address <<OMITTED>>, Host ID: <<OMITTED>>.
> > >
> > >
> > >
> > > Boot device: disk0  File and args: sr0
> > > OpenBSD IEEE 1275 Bootblock 1.4
> > > >> OpenBSD BOOT 1.12
> > > ERROR: /iscsi-hba: No iscsi-network-bootpath property
> > > sr0*
> > > Booting sr0:a/bsd
> > > <<OMITTED>>
> > > <<OMITTED>>
> > > [ using 1036112 bytes of bsd ELF symbol table ]
> > > console is /virtual-devices@100/console@1
> > > Copyright (c) 1982, 1986, 1989, 1991, 1993
> > >         The Regents of the University of California.  All rights
> reserved.
> > > Copyright (c) 1995-2019 OpenBSD. All rights reserved.
> https://www.OpenBSD.org
> > >
> > > OpenBSD 6.5 (GENERIC.MP) #0: Tue Jun 18 18:13:03 EDT 2019
>

[Gao-Mi Baohao]

P.S. looks like there is some kung-fu going on in
src/usr.sbin/installboot/sparc64_softraid.c with the install location of
the ofwboot image that only occurs if you do installboot on the softraid
itself.

On Sun, Jun 30, 2019 at 1:05 PM Gao-Mi Baohao <[hidden email]>
wrote:

> The disk aliasing is a bit complicated, but short answer is yes- that's an
> alias.  I have a bit of config in place so that I can boot the same disks
> as both an LDom and on bare metal.  Results are generally the same either
> way, as far as softraid is concerned.
>
> As far as the loader messages go, I suspect that the loader simply isn't
> detecting the softraid.  The final thing that occurs to me is installboot.
> Have you run installboot on the softraid device itself?  Here's an example
> of the output from my working setup (you'll likely need to specify paths
> explicitly if you are working off the install image):
>
>> glasloc-obsd# installboot -v sd2
>> Using / as root
>> installing bootstrap on /dev/rsd2c
>> using first-stage /usr/mdec/bootblk, second-stage /usr/mdec/ofwboot
>> boot block is 5840 bytes (12 blocks @ 512 bytes = 6144 bytes)
>> sd2: softraid volume with 2 disk(s)
>> sd2: installing boot loader on softraid volume
>> sd0a: installing boot blocks on /dev/rsd0c
>> writing boot block to disk /dev/rsd0c
>> sd1a: installing boot blocks on /dev/rsd1c
>> writing boot block to disk /dev/rsd1c
>>
>
> I thinks that's about the end of what I've got suggestion-wise.
>
>
> On Sun, Jun 30, 2019 at 12:46 PM Chris Ross <[hidden email]>
> wrote:
>
>> On Sun, Jun 30, 2019 at 12:12:46PM -0400, Gao-Mi Baohao wrote:
>> > > Let me know if you can confirm that interpretation of "sr0a"
>> >
>> > Yes, "sr" == "softraid," but when I pulled the working config from my
>> > systems, I discovered that there's an unintuitive syntax that I had
>> > forgotten about, which is probeblty your missing piece.
>> >
>> > It looks like I actually set boot-device = "disk0" and boot-file =
>> > "sr0" (notice the omitted file and slice into), and the loader
>> > auto-selected "sr0:a/bsd" (no that's not a typo... "sr" + [softraid
>> > number] + ":" + [root slice on softraid dev] + "/bsd").
>> >
>> > Give that a shot.  That's jogging my memory about having to dig
>> > through source for the sparc64 loader to discover that very unintutive
>> > syntax.
>>
>> Okay.  Thanks.  Well, I tried using "sr0", but am still getting the same
>> error.  Below.  Note, I checked my "bioctl softraid0" which looks like
>> yours,
>> and the disklabel of my sd4 (the softraid disk) looks about like yours
>> too.
>> a 1g 4.2BSD filesystem at offset 0.
>>
>> Something is still wrong on my end.  So, before the difference in the
>> output
>> of OBP boot command, I notice you said "disk0" above, but your eeprom
>> output
>> below shows "vdisk0".  Do you have a vdisk0 devalias on your system?
>>
>> After that, there's just the different response I see compared you yours.
>> I get:
>>
>> Boot device: /pci@0/pci@0/pci@2/scsi@0/disk@2  File and args: sr0
>> OpenBSD IEEE 1275 Bootblock 1.4
>> >> OpenBSD BOOT 1.12
>> ERROR: /iscsi-hba: No iscsi-network-bootpath property
>> Unknown device: sr0
>> Cannot boot from softraid: Unknown error: code 19
>>
>> And, you see:
>>
>> sr0*
>> Booting sr0:a/bsd
>>
>> where I see:
>>
>> Unknown device: sr0
>> Cannot boot from softraid: Unknown error: code 19
>>
>> Any idea why the bootloader says "sr0*" for you, and "Unknown device" for
>> me?
>>
>> Thanks...
>>
>>             - Chris
>>
>> > Here's my working config.
>> > >
>> > > ===OPENBOOT VARIABLES===
>> > > glasloc-obsd# eeprom boot-device
>> > > boot-device=vdisk0
>> > > glasloc-obsd# eeprom boot-file
>> > > boot-file=sr0
>> > >
>> > > ===BOOT MESSAGES===
>> > > T5140, No Keyboard
>> > > Copyright (c) 1998, 2012, Oracle and/or its affiliates. All rights
>> reserved.
>> > > OpenBoot 4.33.6, <<OMITTED>> MB memory available, Serial #<<OMITTED>>.
>> > > Ethernet address <<OMITTED>>, Host ID: <<OMITTED>>.
>> > >
>> > >
>> > >
>> > > Boot device: disk0  File and args: sr0
>> > > OpenBSD IEEE 1275 Bootblock 1.4
>> > > >> OpenBSD BOOT 1.12
>> > > ERROR: /iscsi-hba: No iscsi-network-bootpath property
>> > > sr0*
>> > > Booting sr0:a/bsd
>> > > <<OMITTED>>
>> > > <<OMITTED>>
>> > > [ using 1036112 bytes of bsd ELF symbol table ]
>> > > console is /virtual-devices@100/console@1
>> > > Copyright (c) 1982, 1986, 1989, 1991, 1993
>> > >         The Regents of the University of California.  All rights
>> reserved.
>> > > Copyright (c) 1995-2019 OpenBSD. All rights reserved.
>> https://www.OpenBSD.org
>> > >
>> > > OpenBSD 6.5 (GENERIC.MP) #0: Tue Jun 18 18:13:03 EDT 2019
>>
>

[Chris Ross]

On Sun, Jun 30, 2019 at 01:05:50PM -0400, Gao-Mi Baohao wrote:

> As far as the loader messages go, I suspect that the loader simply isn't
> detecting the softraid.  The final thing that occurs to me is installboot.
> Have you run installboot on the softraid device itself?  Here's an example
> of the output from my working setup (you'll likely need to specify paths
> explicitly if you are working off the install image):
>
> > glasloc-obsd# installboot -v sd2
> > Using / as root
> > installing bootstrap on /dev/rsd2c
> > using first-stage /usr/mdec/bootblk, second-stage /usr/mdec/ofwboot
> > boot block is 5840 bytes (12 blocks @ 512 bytes = 6144 bytes)
> > sd2: softraid volume with 2 disk(s)
> > sd2: installing boot loader on softraid volume
> > sd0a: installing boot blocks on /dev/rsd0c
> > writing boot block to disk /dev/rsd0c
> > sd1a: installing boot blocks on /dev/rsd1c
> > writing boot block to disk /dev/rsd1c
> >
>
> I thinks that's about the end of what I've got suggestion-wise.

Thanks.  I'm pretty sure I had done this, but to verify, I booted the
install ISO, then mounted and chroot'd into the softraid.  The following
was run:

# chroot /mnt
# installboot -v sd4
Using / as root
installing bootstrap on /dev/rsd4c
using first-stage /usr/mdec/bootblk, second-stage /usr/mdec/ofwboot
boot block is 5840 bytes (12 blocks @ 512 bytes = 6144 bytes)
sd4: softraid volume with 2 disk(s)
sd4: installing boot loader on softraid volume
sd2a: installing boot blocks on /dev/rsd2c
writing boot block to disk /dev/rsd2c
sd3a: installing boot blocks on /dev/rsd3c
writing boot block to disk /dev/rsd3c
# halt

After that a "boot disk2 sr0" produces the same results.  "Unknown device: sr0"

> P.S. looks like there is some kung-fu going on in
> src/usr.sbin/installboot/sparc64_softraid.c with the install location of
> the ofwboot image that only occurs if you do installboot on the softraid
> itself.

Well, unless the kung-fu knows something different about sd0 vs sd2/sd3,
I don't know what it might not be doing for me.  :-/

Thanks.  The next thing I'll have to trace is why ofwboot could be
saying "Unknown device", and/or "Unknown error: code 19".  Let me know
if anyone has any other thoughts.  I'm clearly getting close. :-)

           - Chris

[Chris Ross]

On Sun, Jun 30, 2019 at 01:17:11PM -0400, Gao-Mi Baohao wrote:
> P.S. looks like there is some kung-fu going on in
> src/usr.sbin/installboot/sparc64_softraid.c with the install location of
> the ofwboot image that only occurs if you do installboot on the softraid
> itself.

Looking at src/sys/arch/sparc64/stand/ofwboot/diskprobe.c, I see a couple
of things that might help.  Is there a way to turn on the output of
DPRINTF() at runtime, or is that compile-time?  And, there is some
code inside of #ifdef BOOT_DEBUG.  If someone could provide me with
an ofwboot that was compiled with this extra debugging, I could use
installboot to install it, and maybe get a little more idea what the
code is failing to find on my server.

Thanks for the info and help.

                - Chris

[Gao-Mi Baohao]

Don't forget about e.g. "boot disk2 -V" at your OpenBoot prompt.  Have a
look at
https://github.com/openbsd/src/blob/master/sys/arch/sparc64/stand/bootblk/bootblk.fth
.

Match the verbose boot output with the debug messages in that bootblk.fth
(look for statements in the source like "boot-debug? if") and see where it
chokes.  That will at least tell you if bootblk is getting the softraid
okay, or if it's choking before you ever get to ofwboot.  And that Forth
bootblk doesn't need to be recompiled for debugging.

On Sun, Jun 30, 2019 at 2:06 PM Chris Ross <[hidden email]> wrote:

> On Sun, Jun 30, 2019 at 01:17:11PM -0400, Gao-Mi Baohao wrote:
> > P.S. looks like there is some kung-fu going on in
> > src/usr.sbin/installboot/sparc64_softraid.c with the install location of
> > the ofwboot image that only occurs if you do installboot on the softraid
> > itself.
>
> Looking at src/sys/arch/sparc64/stand/ofwboot/diskprobe.c, I see a couple
> of things that might help.  Is there a way to turn on the output of
> DPRINTF() at runtime, or is that compile-time?  And, there is some
> code inside of #ifdef BOOT_DEBUG.  If someone could provide me with
> an ofwboot that was compiled with this extra debugging, I could use
> installboot to install it, and maybe get a little more idea what the
> code is failing to find on my server.
>
> Thanks for the info and help.
>
>                 - Chris
>
>

[Chris Ross]

On Sun, Jun 30, 2019 at 02:14:41PM -0400, Gao-Mi Baohao wrote:

> Don't forget about e.g. "boot disk2 -V" at your OpenBoot prompt.  Have a
> look at
> https://github.com/openbsd/src/blob/master/sys/arch/sparc64/stand/bootblk/bootblk.fth
> .
>
> Match the verbose boot output with the debug messages in that bootblk.fth
> (look for statements in the source like "boot-debug? if") and see where it
> chokes.  That will at least tell you if bootblk is getting the softraid
> okay, or if it's choking before you ever get to ofwboot.  And that Forth
> bootblk doesn't need to be recompiled for debugging.

Thanks.  That looks like it's getting the softraid data, and I presume loading
ofwboot from there:

Boot device: /pci@0/pci@0/pci@2/scsi@0/disk@2  File and args: sr0 -V
OpenBSD IEEE 1275 Bootblock 1.4
Booting from device /pci@0/pci@0/pci@2/scsi@0/disk@2,0:a
found softraid metadata
softraid-boot 28000 50
>> OpenBSD BOOT 1.12
ERROR: /iscsi-hba: No iscsi-network-bootpath property
Unknown device: sr0
Cannot boot from softraid: Unknown error: code 19

I don't know if I should see anything after the message from softraid-boot,
but assume "OpenBSD BOOT 1.12" is coming from the ofwboot that's loaded
from the softraid.

If you think more debugging in bootblk.fth would be valuable, I'm happy to
put more in.  Just let me know how to reload that, because I am not
aware right now of how that gets stuck into place.  installboot loads
/usr/mdev/bootblk, should I use fgen to build a new one?  I can do that
when booted up from the ISO and running in chroot on the disks I think.
But, if we're already beyond that, I'll want to look into ofwboot...

      - Chris

[Gao-Mi Baohao]

Looks like you're definitely past bootblk and into ofwboot.  That's good.

Unfortunately troubleshooting ofwboot any further will probably need a
recompile, as you correctly noted earlier.  I'll get a debug build of that
for you in a bit.

In the meantime, out of curiosity, have you tried other numbered sr
devices, like "boot disk2 sr1" (and sr2, etc.)?  That's a shot in the dark,
and I don't expect it to work (because ofwboot didn't t seem to detect any
softraid at all), but it's worth a shot.

On Sun, Jun 30, 2019, 2:39 PM Chris Ross <[hidden email]> wrote:

> On Sun, Jun 30, 2019 at 02:14:41PM -0400, Gao-Mi Baohao wrote:
> > Don't forget about e.g. "boot disk2 -V" at your OpenBoot prompt.  Have a
> > look at
> >
> https://github.com/openbsd/src/blob/master/sys/arch/sparc64/stand/bootblk/bootblk.fth
> > .
> >
> > Match the verbose boot output with the debug messages in that bootblk.fth
> > (look for statements in the source like "boot-debug? if") and see where
> it
> > chokes.  That will at least tell you if bootblk is getting the softraid
> > okay, or if it's choking before you ever get to ofwboot.  And that Forth
> > bootblk doesn't need to be recompiled for debugging.
>
> Thanks.  That looks like it's getting the softraid data, and I presume
> loading
> ofwboot from there:
>
> Boot device: /pci@0/pci@0/pci@2/scsi@0/disk@2  File and args: sr0 -V
> OpenBSD IEEE 1275 Bootblock 1.4
> Booting from device /pci@0/pci@0/pci@2/scsi@0/disk@2,0:a
> found softraid metadata
> softraid-boot 28000 50
> >> OpenBSD BOOT 1.12
> ERROR: /iscsi-hba: No iscsi-network-bootpath property
> Unknown device: sr0
> Cannot boot from softraid: Unknown error: code 19
>
> I don't know if I should see anything after the message from softraid-boot,
> but assume "OpenBSD BOOT 1.12" is coming from the ofwboot that's loaded
> from the softraid.
>
> If you think more debugging in bootblk.fth would be valuable, I'm happy to
> put more in.  Just let me know how to reload that, because I am not
> aware right now of how that gets stuck into place.  installboot loads
> /usr/mdev/bootblk, should I use fgen to build a new one?  I can do that
> when booted up from the ISO and running in chroot on the disks I think.
> But, if we're already beyond that, I'll want to look into ofwboot...
>
>       - Chris
>

[Chris Ross]

On Sun, Jun 30, 2019 at 02:45:42PM -0400, Gao-Mi Baohao wrote:
> Looks like you're definitely past bootblk and into ofwboot.  That's good.
>
> Unfortunately troubleshooting ofwboot any further will probably need a
> recompile, as you correctly noted earlier.  I'll get a debug build of that
> for you in a bit.

Thanks!  No rush, I'm offline for a few hours shortly, and I appreciate
all of your help so far!

> In the meantime, out of curiosity, have you tried other numbered sr
> devices, like "boot disk2 sr1" (and sr2, etc.)?  That's a shot in the dark,
> and I don't expect it to work (because ofwboot didn't t seem to detect any
> softraid at all), but it's worth a shot.

I know I tried "sr2a" earlier, on the incorrect assuption of what "sr" meant.
It produced the same "Unknown device:" error.  Trying just now "boot disk2 sr1"
and "boot disk2 sr2" just produce "Unknown device: srN" for each.  I tried sr3
and sr4, just to cover bases. :-)

           - Chris

> On Sun, Jun 30, 2019, 2:39 PM Chris Ross <[hidden email]> wrote:
>
> > On Sun, Jun 30, 2019 at 02:14:41PM -0400, Gao-Mi Baohao wrote:
> > > Don't forget about e.g. "boot disk2 -V" at your OpenBoot prompt.  Have a
> > > look at
> > >
> > https://github.com/openbsd/src/blob/master/sys/arch/sparc64/stand/bootblk/bootblk.fth
> > > .
> > >
> > > Match the verbose boot output with the debug messages in that bootblk.fth
> > > (look for statements in the source like "boot-debug? if") and see where
> > it
> > > chokes.  That will at least tell you if bootblk is getting the softraid
> > > okay, or if it's choking before you ever get to ofwboot.  And that Forth
> > > bootblk doesn't need to be recompiled for debugging.
> >
> > Thanks.  That looks like it's getting the softraid data, and I presume
> > loading
> > ofwboot from there:
> >
> > Boot device: /pci@0/pci@0/pci@2/scsi@0/disk@2  File and args: sr0 -V
> > OpenBSD IEEE 1275 Bootblock 1.4
> > Booting from device /pci@0/pci@0/pci@2/scsi@0/disk@2,0:a
> > found softraid metadata
> > softraid-boot 28000 50
> > >> OpenBSD BOOT 1.12
> > ERROR: /iscsi-hba: No iscsi-network-bootpath property
> > Unknown device: sr0
> > Cannot boot from softraid: Unknown error: code 19
> >
> > I don't know if I should see anything after the message from softraid-boot,
> > but assume "OpenBSD BOOT 1.12" is coming from the ofwboot that's loaded
> > from the softraid.
> >
> > If you think more debugging in bootblk.fth would be valuable, I'm happy to
> > put more in.  Just let me know how to reload that, because I am not
> > aware right now of how that gets stuck into place.  installboot loads
> > /usr/mdev/bootblk, should I use fgen to build a new one?  I can do that
> > when booted up from the ISO and running in chroot on the disks I think.
> > But, if we're already beyond that, I'll want to look into ofwboot...
> >
> >       - Chris
> >

[Chris Ross]

On Sun, Jun 30, 2019 at 09:23:56PM -0400, Chris Ross wrote:
> On Sun, Jun 30, 2019 at 03:12:22PM -0400, Gao-Mi Baohao wrote:
> > Here's your debug build of ofwboot.  
>
> Thanks.  Unfortunately, I can't install that.  Maybe a size issue?
>
> I'll look into the installboot code, but let me know if you have an idea.

Okay, looking at installboot, BIOCINSTALLBOOT, and sys/dev/softraid.c, I
see that sr_ioctl_installboot() returns an error without explanatory output
if the bootblk or bootloader provided to it are "too big".  For boot loader,
that's:

        if (bb->bb_bootldr_size > SR_BOOT_LOADER_SIZE * DEV_BSIZE)

and SR_BOOTLOADER_SIZE is 320, so assuming DEV_BSIZE is 512 bytes, that's
160kb.  The debugging ofwboot was 250, so too big.

Anyone else have an idea what I could try to see why ofwboot seems not to
be able to load and boot my system from my RAID1 softraid on sparc64?


                       - Chris

(bits of earlier off-list email showing the failure to install the build of
ofwboot compiled with debugging:)

> # ls -l
> total 972
> -rw-r--r--  1 root  bin      5840 Apr 14 05:37 bootblk
> -r--r--r--  1 root  bin    102344 Apr 14 05:37 ofwboot
> -rw-------  1 root  wheel  253802 Jun 30 21:20 ofwboot.debug
> -r--r--r--  1 root  bin     54416 Apr 14 05:37 ofwboot.net
> -r--r--r--  1 root  bin     54160 Apr 14 05:37 ofwbootfd
> # chmod 644 ofwboot.debug
> # installboot -v sd4
> Using / as root
> installing bootstrap on /dev/rsd4c
> using first-stage /usr/mdec/bootblk, second-stage /usr/mdec/ofwboot
> boot block is 5840 bytes (12 blocks @ 512 bytes = 6144 bytes)
> sd4: softraid volume with 2 disk(s)
> sd4: installing boot loader on softraid volume
> sd2a: installing boot blocks on /dev/rsd2c
> writing boot block to disk /dev/rsd2c
> sd3a: installing boot blocks on /dev/rsd3c
> writing boot block to disk /dev/rsd3c
> # installboot -v sd4 /usr/mdec/bootblk /usr/mdec/ofwboot.debug
> Using / as root
> installing bootstrap on /dev/rsd4c
> using first-stage /usr/mdec/bootblk, second-stage /usr/mdec/ofwboot.debug
> boot block is 5840 bytes (12 blocks @ 512 bytes = 6144 bytes)
> sd4: softraid volume with 2 disk(s)
> sd4: installing boot loader on softraid volume
> installboot: softraid installboot failed
> # installboot -v sd4 /usr/mdec/bootblk /usr/mdec/ofwboot.debug
> Using / as root
> installing bootstrap on /dev/rsd4c
> using first-stage /usr/mdec/bootblk, second-stage /usr/mdec/ofwboot.debug
> boot block is 5840 bytes (12 blocks @ 512 bytes = 6144 bytes)
> sd4: softraid volume with 2 disk(s)
> sd4: installing boot loader on softraid volume
> installboot: softraid installboot failed
> # file *
> bootblk:       data
> ofwboot:       ELF 64-bit MSB executable, SPARC V9, version 1
> ofwboot.debug: ELF 64-bit MSB executable, SPARC V9, version 1
> ofwboot.net:   ELF 64-bit MSB executable, SPARC V9, version 1
> ofwbootfd:     ELF 64-bit MSB executable, SPARC V9, version 1
> # ls -l *
> -rw-r--r--  1 root  bin      5840 Apr 14 05:37 bootblk
> -r--r--r--  1 root  bin    102344 Apr 14 05:37 ofwboot
> -rw-r--r--  1 root  wheel  253802 Jun 30 21:20 ofwboot.debug
> -r--r--r--  1 root  bin     54416 Apr 14 05:37 ofwboot.net
> -r--r--r--  1 root  bin     54160 Apr 14 05:37 ofwbootfd
> #

[Alex McWhirter-2]

It's been a while since i last installed on sparc64,

but i remember putting the rootfs and subvolumes on raid1, but i made a
separate boot partition on each drive, then used a shutdown script that
kept the two boot partitions in sync.

IIRC, ofwboot will just look for the kernel on the first boot partition
it finds, so as long as the two partitions stay synced it should boot
fine after disk removal / failure.

On 2019-07-01 13:02, Chris Ross wrote:

> On Sun, Jun 30, 2019 at 09:23:56PM -0400, Chris Ross wrote:
>> On Sun, Jun 30, 2019 at 03:12:22PM -0400, Gao-Mi Baohao wrote:
>> > Here's your debug build of ofwboot.
>>
>> Thanks.  Unfortunately, I can't install that.  Maybe a size issue?
>>
>> I'll look into the installboot code, but let me know if you have an
>> idea.
>
> Okay, looking at installboot, BIOCINSTALLBOOT, and sys/dev/softraid.c,
> I
> see that sr_ioctl_installboot() returns an error without explanatory
> output
> if the bootblk or bootloader provided to it are "too big".  For boot
> loader,
> that's:
>
>         if (bb->bb_bootldr_size > SR_BOOT_LOADER_SIZE * DEV_BSIZE)
>
> and SR_BOOTLOADER_SIZE is 320, so assuming DEV_BSIZE is 512 bytes,
> that's
> 160kb.  The debugging ofwboot was 250, so too big.
>
> Anyone else have an idea what I could try to see why ofwboot seems not
> to
> be able to load and boot my system from my RAID1 softraid on sparc64?
>
>
>                        - Chris
>
> (bits of earlier off-list email showing the failure to install the
> build of
> ofwboot compiled with debugging:)
>
>> # ls -l
>> total 972
>> -rw-r--r--  1 root  bin      5840 Apr 14 05:37 bootblk
>> -r--r--r--  1 root  bin    102344 Apr 14 05:37 ofwboot
>> -rw-------  1 root  wheel  253802 Jun 30 21:20 ofwboot.debug
>> -r--r--r--  1 root  bin     54416 Apr 14 05:37 ofwboot.net
>> -r--r--r--  1 root  bin     54160 Apr 14 05:37 ofwbootfd
>> # chmod 644 ofwboot.debug
>> # installboot -v sd4
>> Using / as root
>> installing bootstrap on /dev/rsd4c
>> using first-stage /usr/mdec/bootblk, second-stage /usr/mdec/ofwboot
>> boot block is 5840 bytes (12 blocks @ 512 bytes = 6144 bytes)
>> sd4: softraid volume with 2 disk(s)
>> sd4: installing boot loader on softraid volume
>> sd2a: installing boot blocks on /dev/rsd2c
>> writing boot block to disk /dev/rsd2c
>> sd3a: installing boot blocks on /dev/rsd3c
>> writing boot block to disk /dev/rsd3c
>> # installboot -v sd4 /usr/mdec/bootblk /usr/mdec/ofwboot.debug
>> Using / as root
>> installing bootstrap on /dev/rsd4c
>> using first-stage /usr/mdec/bootblk, second-stage
>> /usr/mdec/ofwboot.debug
>> boot block is 5840 bytes (12 blocks @ 512 bytes = 6144 bytes)
>> sd4: softraid volume with 2 disk(s)
>> sd4: installing boot loader on softraid volume
>> installboot: softraid installboot failed
>> # installboot -v sd4 /usr/mdec/bootblk /usr/mdec/ofwboot.debug
>> Using / as root
>> installing bootstrap on /dev/rsd4c
>> using first-stage /usr/mdec/bootblk, second-stage
>> /usr/mdec/ofwboot.debug
>> boot block is 5840 bytes (12 blocks @ 512 bytes = 6144 bytes)
>> sd4: softraid volume with 2 disk(s)
>> sd4: installing boot loader on softraid volume
>> installboot: softraid installboot failed
>> # file *
>> bootblk:       data
>> ofwboot:       ELF 64-bit MSB executable, SPARC V9, version 1
>> ofwboot.debug: ELF 64-bit MSB executable, SPARC V9, version 1
>> ofwboot.net:   ELF 64-bit MSB executable, SPARC V9, version 1
>> ofwbootfd:     ELF 64-bit MSB executable, SPARC V9, version 1
>> # ls -l *
>> -rw-r--r--  1 root  bin      5840 Apr 14 05:37 bootblk
>> -r--r--r--  1 root  bin    102344 Apr 14 05:37 ofwboot
>> -rw-r--r--  1 root  wheel  253802 Jun 30 21:20 ofwboot.debug
>> -r--r--r--  1 root  bin     54416 Apr 14 05:37 ofwboot.net
>> -r--r--r--  1 root  bin     54160 Apr 14 05:37 ofwbootfd
>> #

[Chris Ross]

On Mon, Jul 01, 2019 at 01:34:23PM -0400, Alex McWhirter wrote:
> It's been a while since i last installed on sparc64,
>
> but i remember putting the rootfs and subvolumes on raid1, but i made a
> separate boot partition on each drive, then used a shutdown script that
> kept the two boot partitions in sync.
>
> IIRC, ofwboot will just look for the kernel on the first boot partition
> it finds, so as long as the two partitions stay synced it should boot
> fine after disk removal / failure.

Yeah.  And, some of the older descriptions of how to do this required that.
But, since the bootloader can do this now, and is for others, I hope to
avoid that.

FMI, in your case, is the boot partition you put on your sparc normally
mounted, or just a small only-mounted-to-be-updated partition that only
ofwboot uses?  In either case, what files need to be in it?

Thanks.

           - Chris