Alix 2d13 and OpenBSD 6.5 Problems

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Alix 2d13 and OpenBSD 6.5 Problems

Sean Kamath-5
Hi.

I’m hoping someone either has a cluebat or some helpful suggestions beyond “reinstall”.

I had an alix 2d13 running OpenBSD 6.3.  I finally got around to upgrading to 6.4 (via https://www.openbsd.org/faq/upgrade64.html), and that seemed to go just fine (I used the Upgrading Manually section, since I don’t have (easy) access to the console).

I let that run for a day, just to make sure all was well, and then attempted an upgrade to 6.5 (via https://www.openbsd.org/faq/upgrade65.html), again using the “Upgrading Manually” section.

This time, between smtpd and relinking the kernel, it appears my Alix board is quickly running out of memory.  Within a few seconds the sr rate is in the 20K range.  I stopped the ld for relinking, and killed SMTPD in order to finish the install (the makedev ALL, sysmerge, pkg_update -u bits), and that all ran fine.  But about 15-20 minutes after a reboot, the box just goes off the network, and there’s not much I can do.

I can download and reinstall 6.5, but was hoping to avoid that pain, but I just want to make sure 6.5 has no issues on the Alix boards. . .

Thanks!  I’d attach dmesg, but the box is dead again. . .  If anyone wants to dive into what’s going on, just let me know what info you want to see.

Sean

Reply | Threaded
Open this post in threaded view
|

Re: Alix 2d13 and OpenBSD 6.5 Problems

Stefan Sperling-5
On Tue, Oct 01, 2019 at 10:46:50PM -0700, Sean Kamath wrote:
> Hi.
>
> I’m hoping someone either has a cluebat or some helpful suggestions beyond “reinstall”.

Try adding swap space.
I have added 2GB of swap space on my alix and it has been running fine ever since.

I avoided a reinstall by repurposing unused /usr/src and /usr/obj partitions.
Snippet from /etc/fstab:

#/dev/wd0i /usr/src ffs rw,nodev,nosuid,softdep 1 2
#/dev/wd0j /usr/obj ffs rw,nodev,nosuid,softdep 1 2
/dev/wd0i none swap sw 0 0
/dev/wd0j none swap sw 0 0

Reply | Threaded
Open this post in threaded view
|

Re: Alix 2d13 and OpenBSD 6.5 Problems

Joe Barnett-2
In reply to this post by Sean Kamath-5
On 2019-10-01 22:46, Sean Kamath wrote:

> Hi.
>
> I’m hoping someone either has a cluebat or some helpful suggestions
> beyond “reinstall”.
>
> I had an alix 2d13 running OpenBSD 6.3.  I finally got around to
> upgrading to 6.4 (via https://www.openbsd.org/faq/upgrade64.html), and
> that seemed to go just fine (I used the Upgrading Manually section,
> since I don’t have (easy) access to the console).
>
> I let that run for a day, just to make sure all was well, and then
> attempted an upgrade to 6.5 (via
> https://www.openbsd.org/faq/upgrade65.html), again using the
> “Upgrading Manually” section.
>
> This time, between smtpd and relinking the kernel, it appears my Alix
> board is quickly running out of memory.  Within a few seconds the sr
> rate is in the 20K range.  I stopped the ld for relinking, and killed
> SMTPD in order to finish the install (the makedev ALL, sysmerge,
> pkg_update -u bits), and that all ran fine.  But about 15-20 minutes
> after a reboot, the box just goes off the network, and there’s not
> much I can do.
>
> I can download and reinstall 6.5, but was hoping to avoid that pain,
> but I just want to make sure 6.5 has no issues on the Alix boards. . .

I cannot comment on the upgrade process, but I have had zero fatal
issues running 6.5 on my alix2d13 boards.  That said, memory has been
getting tighter with more recent OpenBSD versions, and swap (as someone
else suggested) should help.  I love these reliable boards, but they are
starting to show their age (at least relative to how I use them with
OpenBSD).

>
> Thanks!  I’d attach dmesg, but the box is dead again. . .  If anyone
> wants to dive into what’s going on, just let me know what info you
> want to see.
>
> Sean

Reply | Threaded
Open this post in threaded view
|

Re: Alix 2d13 and OpenBSD 6.5 Problems

Olivier Cherrier
In reply to this post by Stefan Sperling-5
On Wed, Oct 02, 2019 at 11:16:21AM +0200, [hidden email] wrote:
> Try adding swap space.
> I have added 2GB of swap space on my alix and it has been running fine ever since.
 
On mine (only 32 MB of swap), I had to disable kernel relinking.
Otherwise, the system more or less collapses at boot time.

--
Olivier Cherrier
mailto:[hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Alix 2d13 and OpenBSD 6.5 Problems

Theo de Raadt-2
Olivier Cherrier <[hidden email]> wrote:

> On Wed, Oct 02, 2019 at 11:16:21AM +0200, [hidden email] wrote:
> > Try adding swap space.
> > I have added 2GB of swap space on my alix and it has been running fine ever since.
>  
> On mine (only 32 MB of swap), I had to disable kernel relinking.
> Otherwise, the system more or less collapses at boot time.

You must be at least this tall to go onto the big boy rides.

Reply | Threaded
Open this post in threaded view
|

Re: Alix 2d13 and OpenBSD 6.5 Problems

Stuart Henderson
In reply to this post by Sean Kamath-5
On 2019-10-02, Sean Kamath <[hidden email]> wrote:

> Hi.
>
> I’m hoping someone either has a cluebat or some helpful suggestions beyond “reinstall”.
>
> I had an alix 2d13 running OpenBSD 6.3.  I finally got around to upgrading to 6.4 (via https://www.openbsd.org/faq/upgrade64.html), and that seemed to go just fine (I used the Upgrading Manually section, since I don’t have (easy) access to the console).
>
> I let that run for a day, just to make sure all was well, and then attempted an upgrade to 6.5 (via https://www.openbsd.org/faq/upgrade65.html), again using the “Upgrading Manually” section.
>
> This time, between smtpd and relinking the kernel, it appears my Alix board is quickly running out of memory.  Within a few seconds the sr rate is in the 20K range.  I stopped the ld for relinking, and killed SMTPD in order to finish the install (the makedev ALL, sysmerge, pkg_update -u bits), and that all ran fine.  But about 15-20 minutes after a reboot, the box just goes off the network, and there’s not much I can do.
>
> I can download and reinstall 6.5, but was hoping to avoid that pain, but I just want to make sure 6.5 has no issues on the Alix boards. . .
>
> Thanks!  I’d attach dmesg, but the box is dead again. . .  If anyone wants to dive into what’s going on, just let me know what info you want to see.
>
> Sean
>
>

After boot, the kernel is relinked in a random order in the background
("/usr/libexec/reorder_kernel &" in /etc/rc). This is done so that
there will be a different memory layout on different boots, making
it harder to carry out types of attack that rely on knowing where
things are in the kernel.

Unfortunately the Alix doesn't have much RAM and if you have pretty
much anything other than a minimal set of daemons running it won't
cope well.

You can disable the reordering by removing /var/db/kernel.SHA256
but be aware that syspatch relies on the reorder_kernel mechanism in
order to apply kernel patches. So if you do this and need to apply
such patches, re-enable it temporarily before running syspatch:
"sha256 -h /var/db/kernel.SHA256 /bsd" - stop any unnecessary
processes - then run syspatch. After syspatch has finished
you can remove kernel.SHA256 again before rebooting.


Reply | Threaded
Open this post in threaded view
|

Re: Alix 2d13 and OpenBSD 6.5 Problems

Sean Kamath-5
Just wanted to say a thank you for everyone’s comments.  I’ve combined all my replies into one mostly to sum everything up.

> On Oct 2, 2019, at 02:16, Stefan Sperling <[hidden email]> wrote:
> Try adding swap space.
> I have added 2GB of swap space on my alix and it has been running fine ever since.

My “disk” is 2GB. I don’t even have any X sets loaded as they won’t fit.

> On Oct 2, 2019, at 09:03, Olivier Cherrier <[hidden email]> wrote:
> On mine (only 32 MB of swap), I had to disable kernel relinking.
> Otherwise, the system more or less collapses at boot time.


Yeah, I believe I only have 32MB of swap (I chose the default disk layout oh so long ago).


> On Oct 2, 2019, at 08:34, Joe Barnett <[hidden email]> wrote:
> I cannot comment on the upgrade process, but I have had zero fatal issues running 6.5 on my alix2d13 boards.  That said, memory has been getting tighter with more recent OpenBSD versions, and swap (as someone else suggested) should help.  I love these reliable boards, but they are starting to show their age (at least relative to how I use them with OpenBSD).

Yeah, so I’m wondering if I want to get a larger CompactFlash card and reinstall, try and be clever and go off the reservation, or just pack in the 9 of these things I have and get an APU.

> On Oct 2, 2019, at 09:15, Stuart Henderson <[hidden email]> wrote:
> After boot, the kernel is relinked in a random order in the background
> ("/usr/libexec/reorder_kernel &" in /etc/rc).

Yes, I’m familiar with why it’s done.  I was mostly wondering if I broke something because I’ve not had this problem since I got these things (I don’t even know how long ago), and 6.5 just killed it.

> Unfortunately the Alix doesn't have much RAM and if you have pretty
> much anything other than a minimal set of daemons running it won't
> cope well.

I’m running nsd and unbound.  I can turn off smtpd. . . What I would be nice to do is delay starting daemons until relinking is done.  Regardless, I think I have my answer about why it’s falling over.

> You can disable the reordering by removing /var/db/kernel.SHA256
> but be aware that syspatch relies on the reorder_kernel mechanism in
> order to apply kernel patches.

Good to know.  I’m going to do everything I can to avoid turning off relinking, because I want to go on the big boy rides! :-)

Sean

Reply | Threaded
Open this post in threaded view
|

Re: Alix 2d13 and OpenBSD 6.5 Problems

Aaron Mason
Hi Sean

Maybe plug in a spare USB drive and format it for swap?  Swap on CF is
rarely a good idea, especially if you swap often.

On Thu, Oct 3, 2019 at 2:01 PM Sean Kamath <[hidden email]> wrote:

>
> Just wanted to say a thank you for everyone’s comments.  I’ve combined all my replies into one mostly to sum everything up.
>
> > On Oct 2, 2019, at 02:16, Stefan Sperling <[hidden email]> wrote:
> > Try adding swap space.
> > I have added 2GB of swap space on my alix and it has been running fine ever since.
>
> My “disk” is 2GB. I don’t even have any X sets loaded as they won’t fit.
>
> > On Oct 2, 2019, at 09:03, Olivier Cherrier <[hidden email]> wrote:
> > On mine (only 32 MB of swap), I had to disable kernel relinking.
> > Otherwise, the system more or less collapses at boot time.
>
>
> Yeah, I believe I only have 32MB of swap (I chose the default disk layout oh so long ago).
>
>
> > On Oct 2, 2019, at 08:34, Joe Barnett <[hidden email]> wrote:
> > I cannot comment on the upgrade process, but I have had zero fatal issues running 6.5 on my alix2d13 boards.  That said, memory has been getting tighter with more recent OpenBSD versions, and swap (as someone else suggested) should help.  I love these reliable boards, but they are starting to show their age (at least relative to how I use them with OpenBSD).
>
> Yeah, so I’m wondering if I want to get a larger CompactFlash card and reinstall, try and be clever and go off the reservation, or just pack in the 9 of these things I have and get an APU.
>
> > On Oct 2, 2019, at 09:15, Stuart Henderson <[hidden email]> wrote:
> > After boot, the kernel is relinked in a random order in the background
> > ("/usr/libexec/reorder_kernel &" in /etc/rc).
>
> Yes, I’m familiar with why it’s done.  I was mostly wondering if I broke something because I’ve not had this problem since I got these things (I don’t even know how long ago), and 6.5 just killed it.
>
> > Unfortunately the Alix doesn't have much RAM and if you have pretty
> > much anything other than a minimal set of daemons running it won't
> > cope well.
>
> I’m running nsd and unbound.  I can turn off smtpd. . . What I would be nice to do is delay starting daemons until relinking is done.  Regardless, I think I have my answer about why it’s falling over.
>
> > You can disable the reordering by removing /var/db/kernel.SHA256
> > but be aware that syspatch relies on the reorder_kernel mechanism in
> > order to apply kernel patches.
>
> Good to know.  I’m going to do everything I can to avoid turning off relinking, because I want to go on the big boy rides! :-)
>
> Sean
>


--
Aaron Mason - Programmer, open source addict
I've taken my software vows - for beta or for worse

Reply | Threaded
Open this post in threaded view
|

Re: Alix 2d13 and OpenBSD 6.5 Problems

Stuart Henderson
In reply to this post by Sean Kamath-5
On 2019-10-03, Sean Kamath <[hidden email]> wrote:
>> You can disable the reordering by removing /var/db/kernel.SHA256
>> but be aware that syspatch relies on the reorder_kernel mechanism in
>> order to apply kernel patches.
>
> Good to know.  I’m going to do everything I can to avoid turning off relinking, because I want to go on the big boy rides! :-)

Even if you only occasionally trigger the relinking by hand when you have
shutdown other daemons,, it's still better than not at all.


Reply | Threaded
Open this post in threaded view
|

Re: Alix 2d13 and OpenBSD 6.5 Problems

Sean Kamath-5
On Oct 4, 2019, at 16:28, Stuart Henderson <[hidden email]> wrote:

>
> On 2019-10-03, Sean Kamath <[hidden email]> wrote:
>>> You can disable the reordering by removing /var/db/kernel.SHA256
>>> but be aware that syspatch relies on the reorder_kernel mechanism in
>>> order to apply kernel patches.
>>
>> Good to know.  I’m going to do everything I can to avoid turning off relinking, because I want to go on the big boy rides! :-)
>
> Even if you only occasionally trigger the relinking by hand when you have
> shutdown other daemons,, it's still better than not at all.

Agreed, but not necessary.

For the archives and anyone who might google this:

I installed fresh OBSD6.5 on another box (I have like 6 of these — this particular one had 4.7 on it.  Even getting bsd.rd from 6.5 to boot on it took installing a new bootbios :-)).  It took a while to relink the kernel before the reboot, but it worked just fine.  Reboots were also fine.  OK ,so a stock 6.5 on the Alix works.

I thought perhaps the disk layout was updated in 6.5.  Nope (in fact, the other machine had a slightly larger swap partition).  OK.

Time to just try adding swap: I added progressively larger swap files until it worked, then I did some math.  I think I got down to the lowest reliable swap size that allows me to reboot and relink:  About 185M.

So, this seems kinda nuts, because literally the only non-stock thing is nsd and unbound, and they’re taking up 137M of VM, but whatever.  They’re tiny little boxes and someday just won’t work.  One itty bitty box per thingie, I guess (my primary reason for upgrading was to install smokeping to be able to bitch at AT&T about my DSL line.  I’ll do that on the box I just rebuilt.).

Just want to say thanks for all the sage advice.  I really do appreciate it.

Sean