i386 kernel relinking

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

i386 kernel relinking

Nick Holland
Question about kernel randomization and relinking...

It seems to take a fair amount of RAM, at least for systems that
are forced to run i386.  And I mean real RAM -- swap doesn't seem
to cut it.  

I discovered that several machines I was intending on using for
minimal purposes just couldn't complete relinking.  So I built a
VM and started playing with the RAM.

Built with 1G RAM, default was a 1.2G swap, worked fine.
Reduced to 256MB RAM, Kernel failed to relink.  As with my old
junk.

The magic number seemed to be between 320MB (failed) and 384MB
(worked) of RAM.  Ok, fine.  

Kernel relinking is important, I get that.  Probably time to
start tossing old junk.  I get that, too.  I'm not complaining
about the forcible retirement of some of my old junk.  I'm just
curious why swap didn't "fix" this problem.

But that VM failed at 320MB RAM, even though it had 1.2G of swap,
mostly unused (MOSTLY.  Yes, it was going into swap).  Is there a
semi-layperson's explanation of this?  Or is this a "if you got
to ask, you won't understand" kind of thing?

And here's the relink log from my VM, but the ones from my physical
boxes looked pretty similiar.

$ cat relink.log  
(SHA256) /bsd: OK
LD="ld" LDFLAGS="-g" sh makegap.sh 0xcccccccc gapdummy.o
ld -T ld.script -X --warn-common -nopie -o newbsd ${SYSTEM_HEAD} vers.o ${OBJS}
text    data    bss     dec     hex
0       0       0       0       0
mv newbsd newbsd.gdb
ctfstrip -S -o newbsd newbsd.gdb
strip: there are no sections to be copied!
rm -f bsd.gdb
mv -f newbsd bsd
mv: newbsd: No such file or directory
*** Error 1 in /usr/share/relink/kernel/GENERIC.MP (Makefile:1131 'newbsd')

I also found that a 320MB machine could not build the kernel from scratch.
Nothing used much memory until the ld step, which started using large amounts
of memory and some swap, and errored out the same way:

LD="ld" LDFLAGS="-g" sh makegap.sh 0xcccccccc gapdummy.o
ld -T ld.script -X --warn-common -nopie -o bsd ${SYSTEM_HEAD} vers.o ${OBJS}
text    data    bss     dec     hex
0       0       0       0       0
mv bsd bsd.gdb
ctfstrip -S -o bsd bsd.gdb
strip: there are no sections to be copied!

Thanks!

Nick.

Reply | Threaded
Open this post in threaded view
|

Re: i386 kernel relinking

Stefan Sperling-5
On Fri, Apr 10, 2020 at 09:35:16AM -0400, Nick Holland wrote:

> Question about kernel randomization and relinking...
>
> It seems to take a fair amount of RAM, at least for systems that
> are forced to run i386.  And I mean real RAM -- swap doesn't seem
> to cut it.  
>
> I discovered that several machines I was intending on using for
> minimal purposes just couldn't complete relinking.  So I built a
> VM and started playing with the RAM.
>
> Built with 1G RAM, default was a 1.2G swap, worked fine.
> Reduced to 256MB RAM, Kernel failed to relink.  As with my old
> junk.
>
> The magic number seemed to be between 320MB (failed) and 384MB
> (worked) of RAM.  Ok, fine.  

FWIW, my soekris net5501 with 256MB of RAM and 512MB swap does manage
to relink a kernel (on 6.6 + syspatches).

# ls -l relink.log
-rw-r--r--  1 root  wheel  -  507B Apr 10 13:33 relink.log
# cat relink.log                              
(SHA256) /bsd: OK
LD="ld" LDFLAGS="-g" sh makegap.sh 0xcccccccc gapdummy.o
ld -T ld.script -X --warn-common -nopie -o newbsd ${SYSTEM_HEAD} vers.o ${OBJS}
text    data    bss     dec     hex
11815507        267748  1101824 13185079        c93037
mv newbsd newbsd.gdb
ctfstrip -S -o newbsd newbsd.gdb
rm -f bsd.gdb
mv -f newbsd bsd
install -F -m 700 bsd /bsd && sha256 -h /var/db/kernel.SHA256 /bsd

Kernel has been relinked and is active on next reboot.

SHA256 (/bsd) = a940ce989d708e5b87a1186ee81bd624066baeabe67b8405b52e4fa2988b565


# dislabel -pm wd0
#                size           offset  fstype [fsize bsize   cpg]
  a:           353.0M               64  4.2BSD   2048 16384  5624 # /
  b:           511.1M           722944    swap                    # none
  c:         15280.0M                0  unused                    
  d:           444.8M          1769728  4.2BSD   2048 16384  7116 # /tmp
  e:           607.7M          2680576  4.2BSD   2048 16384  9685 # /var
  f:          1703.0M          3925216  4.2BSD   2048 16384 12958 # /usr
  g:           505.8M          7412896  4.2BSD   2048 16384  8060 # /usr/X11R6
  h:          1632.9M          8448736  4.2BSD   2048 16384 12958 # /usr/local
  i:          1381.2M         11792960  4.2BSD   2048 16384 12958 # /usr/src
  j:          5282.4M         14621632  4.2BSD   2048 16384 12958 # /usr/obj
  k:          2850.9M         25439936  4.2BSD   2048 16384 12958 # /home

Reply | Threaded
Open this post in threaded view
|

Re: i386 kernel relinking

Nick Holland


On 2020-04-10 10:10, Stefan Sperling wrote:

> On Fri, Apr 10, 2020 at 09:35:16AM -0400, Nick Holland wrote:
>> Question about kernel randomization and relinking...
>>
>> It seems to take a fair amount of RAM, at least for systems that
>> are forced to run i386.  And I mean real RAM -- swap doesn't seem
>> to cut it.  
>>
>> I discovered that several machines I was intending on using for
>> minimal purposes just couldn't complete relinking.  So I built a
>> VM and started playing with the RAM.
>>
>> Built with 1G RAM, default was a 1.2G swap, worked fine.
>> Reduced to 256MB RAM, Kernel failed to relink.  As with my old
>> junk.
>>
>> The magic number seemed to be between 320MB (failed) and 384MB
>> (worked) of RAM.  Ok, fine.  
>
> FWIW, my soekris net5501 with 256MB of RAM and 512MB swap does manage
> to relink a kernel (on 6.6 + syspatches).

Whoops.  Guess I should have mentioned, that was -current, as of
yesterday
OpenBSD 6.7-beta (GENERIC.MP) #110: Thu Apr  9 01:20:52 MDT 2020
    [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC.MP
real mem  = 334970880 (319MB)
avail mem = 313077760 (298MB)

and probably a couple weeks ago for the real (old) hw.

I'm curious if your Soekris can handle 6.7-beta.

Nick.


>
> # ls -l relink.log
> -rw-r--r--  1 root  wheel  -  507B Apr 10 13:33 relink.log
> # cat relink.log                              
> (SHA256) /bsd: OK
> LD="ld" LDFLAGS="-g" sh makegap.sh 0xcccccccc gapdummy.o
> ld -T ld.script -X --warn-common -nopie -o newbsd ${SYSTEM_HEAD} vers.o ${OBJS}
> text    data    bss     dec     hex
> 11815507        267748  1101824 13185079        c93037
> mv newbsd newbsd.gdb
> ctfstrip -S -o newbsd newbsd.gdb
> rm -f bsd.gdb
> mv -f newbsd bsd
> install -F -m 700 bsd /bsd && sha256 -h /var/db/kernel.SHA256 /bsd
>
> Kernel has been relinked and is active on next reboot.
>
> SHA256 (/bsd) = a940ce989d708e5b87a1186ee81bd624066baeabe67b8405b52e4fa2988b565
>
>
> # dislabel -pm wd0
> #                size           offset  fstype [fsize bsize   cpg]
>   a:           353.0M               64  4.2BSD   2048 16384  5624 # /
>   b:           511.1M           722944    swap                    # none
>   c:         15280.0M                0  unused                    
>   d:           444.8M          1769728  4.2BSD   2048 16384  7116 # /tmp
>   e:           607.7M          2680576  4.2BSD   2048 16384  9685 # /var
>   f:          1703.0M          3925216  4.2BSD   2048 16384 12958 # /usr
>   g:           505.8M          7412896  4.2BSD   2048 16384  8060 # /usr/X11R6
>   h:          1632.9M          8448736  4.2BSD   2048 16384 12958 # /usr/local
>   i:          1381.2M         11792960  4.2BSD   2048 16384 12958 # /usr/src
>   j:          5282.4M         14621632  4.2BSD   2048 16384 12958 # /usr/obj
>   k:          2850.9M         25439936  4.2BSD   2048 16384 12958 # /home
>

Reply | Threaded
Open this post in threaded view
|

Re: i386 kernel relinking

Theo de Raadt-2
I am succesfully relinking kernels on a landisk with 128MB of ram.

I think this conversation is ridiculous:

If a machine is too small, then it is too small.  Do I have to paypal
a $0.05 to some users?

Nick Holland <[hidden email]> wrote:

>
>
>
> On 2020-04-10 10:10, Stefan Sperling wrote:
> > On Fri, Apr 10, 2020 at 09:35:16AM -0400, Nick Holland wrote:
> >> Question about kernel randomization and relinking...
> >>
> >> It seems to take a fair amount of RAM, at least for systems that
> >> are forced to run i386.  And I mean real RAM -- swap doesn't seem
> >> to cut it.  
> >>
> >> I discovered that several machines I was intending on using for
> >> minimal purposes just couldn't complete relinking.  So I built a
> >> VM and started playing with the RAM.
> >>
> >> Built with 1G RAM, default was a 1.2G swap, worked fine.
> >> Reduced to 256MB RAM, Kernel failed to relink.  As with my old
> >> junk.
> >>
> >> The magic number seemed to be between 320MB (failed) and 384MB
> >> (worked) of RAM.  Ok, fine.  
> >
> > FWIW, my soekris net5501 with 256MB of RAM and 512MB swap does manage
> > to relink a kernel (on 6.6 + syspatches).
>
> Whoops.  Guess I should have mentioned, that was -current, as of
> yesterday
> OpenBSD 6.7-beta (GENERIC.MP) #110: Thu Apr  9 01:20:52 MDT 2020
>     [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC.MP
> real mem  = 334970880 (319MB)
> avail mem = 313077760 (298MB)
>
> and probably a couple weeks ago for the real (old) hw.
>
> I'm curious if your Soekris can handle 6.7-beta.
>
> Nick.
>
>
> >
> > # ls -l relink.log
> > -rw-r--r--  1 root  wheel  -  507B Apr 10 13:33 relink.log
> > # cat relink.log                              
> > (SHA256) /bsd: OK
> > LD="ld" LDFLAGS="-g" sh makegap.sh 0xcccccccc gapdummy.o
> > ld -T ld.script -X --warn-common -nopie -o newbsd ${SYSTEM_HEAD} vers.o ${OBJS}
> > text    data    bss     dec     hex
> > 11815507        267748  1101824 13185079        c93037
> > mv newbsd newbsd.gdb
> > ctfstrip -S -o newbsd newbsd.gdb
> > rm -f bsd.gdb
> > mv -f newbsd bsd
> > install -F -m 700 bsd /bsd && sha256 -h /var/db/kernel.SHA256 /bsd
> >
> > Kernel has been relinked and is active on next reboot.
> >
> > SHA256 (/bsd) = a940ce989d708e5b87a1186ee81bd624066baeabe67b8405b52e4fa2988b565
> >
> >
> > # dislabel -pm wd0
> > #                size           offset  fstype [fsize bsize   cpg]
> >   a:           353.0M               64  4.2BSD   2048 16384  5624 # /
> >   b:           511.1M           722944    swap                    # none
> >   c:         15280.0M                0  unused                    
> >   d:           444.8M          1769728  4.2BSD   2048 16384  7116 # /tmp
> >   e:           607.7M          2680576  4.2BSD   2048 16384  9685 # /var
> >   f:          1703.0M          3925216  4.2BSD   2048 16384 12958 # /usr
> >   g:           505.8M          7412896  4.2BSD   2048 16384  8060 # /usr/X11R6
> >   h:          1632.9M          8448736  4.2BSD   2048 16384 12958 # /usr/local
> >   i:          1381.2M         11792960  4.2BSD   2048 16384 12958 # /usr/src
> >   j:          5282.4M         14621632  4.2BSD   2048 16384 12958 # /usr/obj
> >   k:          2850.9M         25439936  4.2BSD   2048 16384 12958 # /home
> >
>

Reply | Threaded
Open this post in threaded view
|

on "underpowered" machines (was: Re: i386 kernel relinking)

zeurkous
theo wrote:
> I think this conversation is ridiculous:
>
> If a machine is too small, then it is too small. Do I have to paypal
> a $0.05 to some users?

Yes, me! Pretty please? *kitty eyes*

Seriously: sometimes, a machine is perfectly adequate, except for one
task that eats disproportionate amounts of resources. Questioning the
apropriateness of that task would seem to be a reasonable step to
undertake, before running to the store waving your credit card (not to
speak of the waste problem).

Apart from the fact that many people on this planet *are*
really-terribly-out-of-pocket, there are still other issues than money.

       --zeurkous.

--
Friggin' Machines!

Reply | Threaded
Open this post in threaded view
|

Re: i386 kernel relinking

Josh Grosse-3
In reply to this post by Theo de Raadt-2
FWIW, the GNU linker can reorder the kernel on i386-current
with 256MB RAM:

   # env LD=ld.bfd /usr/libexec/reorder_kernel

---

relink.log:

(SHA256) /bsd: OK
LD="ld.bfd" LDFLAGS="-g" sh makegap.sh 0xcccccccc gapdummy.o
ld.bfd -T ld.script -X --warn-common -nopie -o newbsd ${SYSTEM_HEAD} vers.o ${OBJS}
text    data    bss     dec     hex
11916299        262796  1101824 13280919        caa697
mv newbsd newbsd.gdb
ctfstrip -S -o newbsd newbsd.gdb
rm -f bsd.gdb
mv -f newbsd bsd
install -F -m 700 bsd /bsd && sha256 -h /var/db/kernel.SHA256 /bsd

Kernel has been relinked and is active on next reboot.

SHA256 (/bsd) = 3c9cca9da06acdc92270f6f0e68b57447881a01e3e2584a0086291efb5033ba7

---

On Fri, Apr 10, 2020 at 10:52:37AM -0600, Theo de Raadt wrote:

> I am succesfully relinking kernels on a landisk with 128MB of ram.
>
> I think this conversation is ridiculous:
>
> If a machine is too small, then it is too small.  Do I have to paypal
> a $0.05 to some users?
>
> Nick Holland <[hidden email]> wrote:
>
> >
> >
> >
> > On 2020-04-10 10:10, Stefan Sperling wrote:
> > > On Fri, Apr 10, 2020 at 09:35:16AM -0400, Nick Holland wrote:
> > >> Question about kernel randomization and relinking...
> > >>
> > >> It seems to take a fair amount of RAM, at least for systems that
> > >> are forced to run i386.  And I mean real RAM -- swap doesn't seem
> > >> to cut it.  
> > >>
> > >> I discovered that several machines I was intending on using for
> > >> minimal purposes just couldn't complete relinking.  So I built a
> > >> VM and started playing with the RAM.
> > >>
> > >> Built with 1G RAM, default was a 1.2G swap, worked fine.
> > >> Reduced to 256MB RAM, Kernel failed to relink.  As with my old
> > >> junk.
> > >>
> > >> The magic number seemed to be between 320MB (failed) and 384MB
> > >> (worked) of RAM.  Ok, fine.  
> > >
> > > FWIW, my soekris net5501 with 256MB of RAM and 512MB swap does manage
> > > to relink a kernel (on 6.6 + syspatches).
> >
> > Whoops.  Guess I should have mentioned, that was -current, as of
> > yesterday
> > OpenBSD 6.7-beta (GENERIC.MP) #110: Thu Apr  9 01:20:52 MDT 2020
> >     [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC.MP
> > real mem  = 334970880 (319MB)
> > avail mem = 313077760 (298MB)
> >
> > and probably a couple weeks ago for the real (old) hw.
> >
> > I'm curious if your Soekris can handle 6.7-beta.
> >
> > Nick.
> >
> >
> > >
> > > # ls -l relink.log
> > > -rw-r--r--  1 root  wheel  -  507B Apr 10 13:33 relink.log
> > > # cat relink.log                              
> > > (SHA256) /bsd: OK
> > > LD="ld" LDFLAGS="-g" sh makegap.sh 0xcccccccc gapdummy.o
> > > ld -T ld.script -X --warn-common -nopie -o newbsd ${SYSTEM_HEAD} vers.o ${OBJS}
> > > text    data    bss     dec     hex
> > > 11815507        267748  1101824 13185079        c93037
> > > mv newbsd newbsd.gdb
> > > ctfstrip -S -o newbsd newbsd.gdb
> > > rm -f bsd.gdb
> > > mv -f newbsd bsd
> > > install -F -m 700 bsd /bsd && sha256 -h /var/db/kernel.SHA256 /bsd
> > >
> > > Kernel has been relinked and is active on next reboot.
> > >
> > > SHA256 (/bsd) = a940ce989d708e5b87a1186ee81bd624066baeabe67b8405b52e4fa2988b565
> > >
> > >
> > > # dislabel -pm wd0
> > > #                size           offset  fstype [fsize bsize   cpg]
> > >   a:           353.0M               64  4.2BSD   2048 16384  5624 # /
> > >   b:           511.1M           722944    swap                    # none
> > >   c:         15280.0M                0  unused                    
> > >   d:           444.8M          1769728  4.2BSD   2048 16384  7116 # /tmp
> > >   e:           607.7M          2680576  4.2BSD   2048 16384  9685 # /var
> > >   f:          1703.0M          3925216  4.2BSD   2048 16384 12958 # /usr
> > >   g:           505.8M          7412896  4.2BSD   2048 16384  8060 # /usr/X11R6
> > >   h:          1632.9M          8448736  4.2BSD   2048 16384 12958 # /usr/local
> > >   i:          1381.2M         11792960  4.2BSD   2048 16384 12958 # /usr/src
> > >   j:          5282.4M         14621632  4.2BSD   2048 16384 12958 # /usr/obj
> > >   k:          2850.9M         25439936  4.2BSD   2048 16384 12958 # /home
> > >
> >

Reply | Threaded
Open this post in threaded view
|

Re: i386 kernel relinking

Stefan Sperling-5
In reply to this post by Nick Holland
On Fri, Apr 10, 2020 at 10:25:14AM -0400, Nick Holland wrote:

> On 2020-04-10 10:10, Stefan Sperling wrote:
> > On Fri, Apr 10, 2020 at 09:35:16AM -0400, Nick Holland wrote:
> > FWIW, my soekris net5501 with 256MB of RAM and 512MB swap does manage
> > to relink a kernel (on 6.6 + syspatches).
>
> Whoops.  Guess I should have mentioned, that was -current, as of
> yesterday
> OpenBSD 6.7-beta (GENERIC.MP) #110: Thu Apr  9 01:20:52 MDT 2020
>     [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC.MP
> real mem  = 334970880 (319MB)
> avail mem = 313077760 (298MB)
>
> and probably a couple weeks ago for the real (old) hw.
>
> I'm curious if your Soekris can handle 6.7-beta.
>
> Nick.

It's been fixed in 6.7 release!

This was indeed a problem on my alix and soekris machines for a while.
And I could also reproduce it in a 6.7-beta low memory VM as you did.

I attempted to debug ld.lld for a while but eventually gave up (debugging
the linker is quite painful).

I did not come to any conclusion but did notice that ld.lld uses mmap to
write ELF data to the output file. That file always ended up being corrupt
under low memory conditions. The old ld.bfd linker uses stdio and worked OK.

With 6.7 kernel relinking with little amounts of memory suddenly started
working again. To make sure I'm not dreaming I bisected the commit which
fixed it.

Turns out this problem was fixed by beck@ in the commit below.
I can even relink kernels on i386 in 64MB of RAM now. Didn't test lower
than that.  Case closed. Just upgrade to 6.7 and it'll be fine :-)

CVSROOT:        /cvs
Module name:    src
Changes by:     [hidden email]    2020/04/28 20:25:48

Modified files:
        sys/kern       : vfs_bio.c

Log message:
Ensure that if we are doing a delayed write with a NOCACHE buffer, we
clear the NOCACHE flag, since if we are doing a delayed write the buffer
must be cached or it is thrown away when the "write" is done.
fixes vnd on mfs regress tests.

ok kettenis@ deraadt@

Reply | Threaded
Open this post in threaded view
|

Re: i386 kernel relinking

Josh Grosse-3
On Tue, May 19, 2020 at 05:21:59PM +0200, Stefan Sperling wrote:

> On Fri, Apr 10, 2020 at 10:25:14AM -0400, Nick Holland wrote:
> > On 2020-04-10 10:10, Stefan Sperling wrote:
> > > On Fri, Apr 10, 2020 at 09:35:16AM -0400, Nick Holland wrote:
> > > FWIW, my soekris net5501 with 256MB of RAM and 512MB swap does manage
> > > to relink a kernel (on 6.6 + syspatches).
> >
> > Whoops.  Guess I should have mentioned, that was -current, as of
> > yesterday
> > OpenBSD 6.7-beta (GENERIC.MP) #110: Thu Apr  9 01:20:52 MDT 2020
> >     [hidden email]:/usr/src/sys/arch/i386/compile/GENERIC.MP
> > real mem  = 334970880 (319MB)
> > avail mem = 313077760 (298MB)
> >
> > and probably a couple weeks ago for the real (old) hw.
> >
> > I'm curious if your Soekris can handle 6.7-beta.
> >
> > Nick.
>
> It's been fixed in 6.7 release!

Thank you!!!  Awesome!!!