Best Practices for growing disk partitions on a server

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

Best Practices for growing disk partitions on a server

Lev Lazinskiy
Hi folks,

I am new to openBSD, so forgive me if I am missing something obvious.

I recently installed openBSD on a server using the auto-partition layout
during installation and am quickly starting to run out of disk space.

I have read the section in the FAQ [1] regarding how to grow a disk
partition, but I am confused on the best way to actually do this.

Specifically, I am trying to grow /usr and /home but they are "busy"
when I try to follow these steps on a running server.

Is the assumption that you are supposed to reboot the server with the
ISO attached and pop into a shell to complete these steps?

[1] https://www.openbsd.org/faq/faq14.html#GrowPartition

--
Lev Lazinskiy

Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for growing disk partitions on a server

Brian Brombacher
Boot into single user mode.  At the boot loader prompt, type boot -s.  This will drop you to a root shell.



> On Nov 17, 2019, at 3:39 PM, Lev Lazinskiy <[hidden email]> wrote:
>
> Hi folks,
>
> I am new to openBSD, so forgive me if I am missing something obvious.
>
> I recently installed openBSD on a server using the auto-partition layout
> during installation and am quickly starting to run out of disk space.
>
> I have read the section in the FAQ [1] regarding how to grow a disk
> partition, but I am confused on the best way to actually do this.
>
> Specifically, I am trying to grow /usr and /home but they are "busy"
> when I try to follow these steps on a running server.
>
> Is the assumption that you are supposed to reboot the server with the
> ISO attached and pop into a shell to complete these steps?
>
> [1] https://www.openbsd.org/faq/faq14.html#GrowPartition
>
> --
> Lev Lazinskiy
>

Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for growing disk partitions on a server

Lev Lazinskiy
Hi Brian,

Thank you very much, I really appreciate the help.
On Sun, Nov 17, 2019 at 03:46:27PM -0500, Brian Brombacher wrote:

> Boot into single user mode.  At the boot loader prompt, type boot -s.  This will drop you to a root shell.
>
>
>
> > On Nov 17, 2019, at 3:39 PM, Lev Lazinskiy <[hidden email]> wrote:
> >
> > Hi folks,
> >
> > I am new to openBSD, so forgive me if I am missing something obvious.
> >
> > I recently installed openBSD on a server using the auto-partition layout
> > during installation and am quickly starting to run out of disk space.
> >
> > I have read the section in the FAQ [1] regarding how to grow a disk
> > partition, but I am confused on the best way to actually do this.
> >
> > Specifically, I am trying to grow /usr and /home but they are "busy"
> > when I try to follow these steps on a running server.
> >
> > Is the assumption that you are supposed to reboot the server with the
> > ISO attached and pop into a shell to complete these steps?
> >
> > [1] https://www.openbsd.org/faq/faq14.html#GrowPartition
> >
> > --
> > Lev Lazinskiy
> >
>

Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for growing disk partitions on a server

Edgar Pettijohn III-2
In reply to this post by Lev Lazinskiy

On Nov 17, 2019 2:35 PM, Lev Lazinskiy <[hidden email]> wrote:

>
> Hi folks,
>
> I am new to openBSD, so forgive me if I am missing something obvious.
>
> I recently installed openBSD on a server using the auto-partition layout
> during installation and am quickly starting to run out of disk space.
>
> I have read the section in the FAQ [1] regarding how to grow a disk
> partition, but I am confused on the best way to actually do this.
>
> Specifically, I am trying to grow /usr and /home but they are "busy"
> when I try to follow these steps on a running server.
>

You must umount them first.


> Is the assumption that you are supposed to reboot the server with the
> ISO attached and pop into a shell to complete these steps?
>
> [1] https://www.openbsd.org/faq/faq14.html#GrowPartition
>
> --
> Lev Lazinskiy
>

If it's a fresh install probably be easier to just reinstall and size partitions how you need them.

Edgar

Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for growing disk partitions on a server

Lev Lazinskiy
In reply to this post by Lev Lazinskiy
Hi Edgar,

Thanks for the response.
On Sun, Nov 17, 2019 at 04:06:18PM -0600, Edgar Pettijohn wrote:

>
> On Nov 17, 2019 2:35 PM, Lev Lazinskiy <[hidden email]> wrote:
> >
> > Hi folks,
> >
> > I am new to openBSD, so forgive me if I am missing something obvious.
> >
> > I recently installed openBSD on a server using the auto-partition layout
> > during installation and am quickly starting to run out of disk space.
> >
> > I have read the section in the FAQ [1] regarding how to grow a disk
> > partition, but I am confused on the best way to actually do this.
> >
> > Specifically, I am trying to grow /usr and /home but they are "busy"
> > when I try to follow these steps on a running server.
> >
>
> You must umount them first.

Right, the issue is that I am unable to unmount them because they are
actively being used.

>
>
> > Is the assumption that you are supposed to reboot the server with the
> > ISO attached and pop into a shell to complete these steps?
> >
> > [1] https://www.openbsd.org/faq/faq14.html#GrowPartition
> >
> > --
> > Lev Lazinskiy
> >
>
> If it's a fresh install probably be easier to just reinstall and size partitions how you need them.

This makes sense, but I was curious what the recommended approach is for
a server that you cannot simply reinstall.

Someone else suggested using single user mode on boot and that approach
seems to work very well.

>
> Edgar

Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for growing disk partitions on a server

Joseph Borg
you have to boot in single user mode:

https://www.openbsd.org/faq/faq8.html


> On 18 Nov 2019, at 00:12, Lev Lazinskiy <[hidden email]> wrote:
>
> Hi Edgar,
>
> Thanks for the response.
> On Sun, Nov 17, 2019 at 04:06:18PM -0600, Edgar Pettijohn wrote:
>>
>> On Nov 17, 2019 2:35 PM, Lev Lazinskiy <[hidden email]> wrote:
>>>
>>> Hi folks,
>>>
>>> I am new to openBSD, so forgive me if I am missing something obvious.
>>>
>>> I recently installed openBSD on a server using the auto-partition layout
>>> during installation and am quickly starting to run out of disk space.
>>>
>>> I have read the section in the FAQ [1] regarding how to grow a disk
>>> partition, but I am confused on the best way to actually do this.
>>>
>>> Specifically, I am trying to grow /usr and /home but they are "busy"
>>> when I try to follow these steps on a running server.
>>>
>>
>> You must umount them first.
>
> Right, the issue is that I am unable to unmount them because they are
> actively being used.
>>
>>
>>> Is the assumption that you are supposed to reboot the server with the
>>> ISO attached and pop into a shell to complete these steps?
>>>
>>> [1] https://www.openbsd.org/faq/faq14.html#GrowPartition
>>>
>>> --
>>> Lev Lazinskiy
>>>
>>
>> If it's a fresh install probably be easier to just reinstall and size partitions how you need them.
>
> This makes sense, but I was curious what the recommended approach is for
> a server that you cannot simply reinstall.
>
> Someone else suggested using single user mode on boot and that approach
> seems to work very well.
>
>>
>> Edgar
>

Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for growing disk partitions on a server

Stuart Henderson
In reply to this post by Lev Lazinskiy
On 2019-11-17, Lev Lazinskiy <[hidden email]> wrote:

> Hi folks,
>
> I am new to openBSD, so forgive me if I am missing something obvious.
>
> I recently installed openBSD on a server using the auto-partition layout
> during installation and am quickly starting to run out of disk space.
>
> I have read the section in the FAQ [1] regarding how to grow a disk
> partition, but I am confused on the best way to actually do this.
>
> Specifically, I am trying to grow /usr and /home but they are "busy"
> when I try to follow these steps on a running server.
>
> Is the assumption that you are supposed to reboot the server with the
> ISO attached and pop into a shell to complete these steps?
>
> [1] https://www.openbsd.org/faq/faq14.html#GrowPartition
>

This faq entry tells you to use growfs, which won't work for /usr
from a standard auto-defaults install because it requires empty space
immediately following the partition to grow into.

On a larger disk it often will work for /home because auto-defaults
place it at the end of the disk, size it at max 300G, and leave the space
following it empty.

Sometimes you can shunt things around - if you have free space at the end
of the disk you can move files from the filesystem immediately following
/usr there, and growfs into the now-vacated space - sometimes you have
another partition that is A) larger than /usr and B) that is larger than
you need, in which case you copy files so you can swap them around.
Or again if you have free space at the end of the disk, after what you'd
want to grow /home into, you can make a new partition there, copy the
files, and leave the former /usr partition empty. But it's quite delicate
work and is often easier to reinstall.

Interested what you are bumping into on /usr, on a recently installed
system the default size is usually ok for most uses unless it's a small
disk or unless you are building ports and don't have a separate /usr/ports
partition. (If you *are* using ports and have free space at the end -
as above - you could create a new partition to hold /usr/ports in there
and move *those* files across - that's a much easier proposition than
moving the whole of /usr.



Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for growing disk partitions on a server

Theo de Raadt-2
Stuart Henderson <[hidden email]> wrote:

> On 2019-11-17, Lev Lazinskiy <[hidden email]> wrote:
> > Hi folks,
> >
> > I am new to openBSD, so forgive me if I am missing something obvious.
> >
> > I recently installed openBSD on a server using the auto-partition layout
> > during installation and am quickly starting to run out of disk space.
> >
> > I have read the section in the FAQ [1] regarding how to grow a disk
> > partition, but I am confused on the best way to actually do this.
> >
> > Specifically, I am trying to grow /usr and /home but they are "busy"
> > when I try to follow these steps on a running server.
> >
> > Is the assumption that you are supposed to reboot the server with the
> > ISO attached and pop into a shell to complete these steps?
> >
> > [1] https://www.openbsd.org/faq/faq14.html#GrowPartition
> >
>
> This faq entry tells you to use growfs, which won't work for /usr
> from a standard auto-defaults install because it requires empty space
> immediately following the partition to grow into.
>
> On a larger disk it often will work for /home because auto-defaults
> place it at the end of the disk, size it at max 300G, and leave the space
> following it empty.
>
> Sometimes you can shunt things around - if you have free space at the end
> of the disk you can move files from the filesystem immediately following
> /usr there, and growfs into the now-vacated space - sometimes you have
> another partition that is A) larger than /usr and B) that is larger than
> you need, in which case you copy files so you can swap them around.
> Or again if you have free space at the end of the disk, after what you'd
> want to grow /home into, you can make a new partition there, copy the
> files, and leave the former /usr partition empty. But it's quite delicate
> work and is often easier to reinstall.
>
> Interested what you are bumping into on /usr, on a recently installed
> system the default size is usually ok for most uses unless it's a small
> disk or unless you are building ports and don't have a separate /usr/ports
> partition. (If you *are* using ports and have free space at the end -
> as above - you could create a new partition to hold /usr/ports in there
> and move *those* files across - that's a much easier proposition than
> moving the whole of /usr.

Which is to say, there are all sorts of complexities, and you'll have to
work through them.

On the other hand learning how to save all your files, and then do a
fresh install, is like gaining a superpower.

Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for growing disk partitions on a server

Dumitru Moldovan-2
In reply to this post by Lev Lazinskiy
On Sun, Nov 17, 2019 at 03:12:06PM -0800, Lev Lazinskiy wrote:
>
>This makes sense, but I was curious what the recommended approach is for
>a server that you cannot simply reinstall.

A humble piece of advice from a fellow system admin...  Never ever build
a system that "you cannot simply reinstall."  There should be at least
two ways to redo everything:

  1. From scratch using documentation or preferably a modern deployment
  system such as Ansible, Salt, etc.

  2. From backup.  This should be drilled from time to time, even
  without a practical need, just to make sure things work as expected.

Yes, practice is not as simple as theory and I admit being guilty at
times of every possible mistake in not following the above.  :-]

But, realistically speaking, as long as a system still functions, it
should be reproducible.  Document the setup, backup its configs and
saved data, etc.

Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for growing disk partitions on a server

Dumitru Moldovan-2
On Tue, Nov 19, 2019 at 12:31:25PM +0200, Dumitru Moldovan wrote:

>On Sun, Nov 17, 2019 at 03:12:06PM -0800, Lev Lazinskiy wrote:
>>
>>This makes sense, but I was curious what the recommended approach is for
>>a server that you cannot simply reinstall.
>
>A humble piece of advice from a fellow system admin...  Never ever build
>a system that "you cannot simply reinstall."  There should be at least
>two ways to redo everything:
>
> 1. From scratch using documentation or preferably a modern deployment
> system such as Ansible, Salt, etc.
>
> 2. From backup.  This should be drilled from time to time, even
> without a practical need, just to make sure things work as expected.
>
>Yes, practice is not as simple as theory and I admit being guilty at
>times of every possible mistake in not following the above.  :-]
>
>But, realistically speaking, as long as a system still functions, it
>should be reproducible.  Document the setup, backup its configs and
>saved data, etc.

May be bad advice for your situation, but this is what I used to do
when migrating setups that were not documented or easy to redeploy:

  1. Install the same OS on new hardware, in your case with a better
  partitioned drive.

  2. rsync everything relevant to the new machine (but make sure it
  still boots afterwards and functions as expected, so amend boot
  manager files, hardware-dependant configs etc.).

  3. Update relevant DNS records for affected services AND set packet
  redirection for services on the old machines until DNS propagation
  does its thing (which is usually much longer than expected, esp.
  for public services, even if you lower TTLs well in advance).

If you don't have spare hardware for the migration, I guess you could
use a spare VM until you repartition drives in your old hardware as
needed.  If you don't care about service disruption, I guess you could
skip redirection.

Hope that helps!

Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for growing disk partitions on a server

Steve Litt
In reply to this post by Stuart Henderson
On Mon, 18 Nov 2019 18:28:55 -0000 (UTC)
Stuart Henderson <[hidden email]> wrote:

> On 2019-11-17, Lev Lazinskiy <[hidden email]> wrote:
> > Hi folks,
> >
> > I am new to openBSD, so forgive me if I am missing something
> > obvious.
> >
> > I recently installed openBSD on a server using the auto-partition
> > layout during installation and am quickly starting to run out of
> > disk space.
> >
> > I have read the section in the FAQ [1] regarding how to grow a disk
> > partition, but I am confused on the best way to actually do this.
> >
> > Specifically, I am trying to grow /usr and /home but they are "busy"
> > when I try to follow these steps on a running server.
> >
> > Is the assumption that you are supposed to reboot the server with
> > the ISO attached and pop into a shell to complete these steps?
> >
> > [1] https://www.openbsd.org/faq/faq14.html#GrowPartition
> >  
>
> This faq entry tells you to use growfs, which won't work for /usr
> from a standard auto-defaults install because it requires empty space
> immediately following the partition to grow into.
>
> On a larger disk it often will work for /home because auto-defaults
> place it at the end of the disk, size it at max 300G, and leave the
> space following it empty.
>
> Sometimes you can shunt things around - if you have free space at the
> end of the disk you can move files from the filesystem immediately
> following /usr there, and growfs into the now-vacated space -
> sometimes you have another partition that is A) larger than /usr and
> B) that is larger than you need, in which case you copy files so you
> can swap them around. Or again if you have free space at the end of
> the disk, after what you'd want to grow /home into, you can make a
> new partition there, copy the files, and leave the former /usr
> partition empty. But it's quite delicate work and is often easier to
> reinstall.

In OpenBSD is there such a thing as a bind mount like they have in
Linux?
 
SteveT

Steve Litt
November 2019 featured book: Manager's Guide to Technical
Troubleshooting Second edition
http://www.troubleshooters.com/mgr

Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for growing disk partitions on a server

Steve Litt
In reply to this post by Dumitru Moldovan-2
On Tue, 19 Nov 2019 12:52:52 +0200
Dumitru Moldovan <[hidden email]> wrote:


>
>   1. Install the same OS on new hardware, in your case with a better
>   partitioned drive.
>
>   2. rsync everything relevant to the new machine (but make sure it
>   still boots afterwards and functions as expected, so amend boot
>   manager files, hardware-dependant configs etc.).
>
>   3. Update relevant DNS records for affected services AND set packet
>   redirection for services on the old machines until DNS propagation
>   does its thing (which is usually much longer than expected, esp.
>   for public services, even if you lower TTLs well in advance).
>
> If you don't have spare hardware for the migration, I guess you could
> use a spare VM until you repartition drives in your old hardware as
> needed.

Also rsync to a USB hard drive, so even if everything including your VM
goes to hell you can still restore. I just bought a 5TB USB drive for
$99 at Costco.
 
SteveT

Steve Litt
November 2019 featured book: Manager's Guide to Technical
Troubleshooting Second edition
http://www.troubleshooters.com/mgr

Reply | Threaded
Open this post in threaded view
|

Re: Best Practices for growing disk partitions on a server

Stuart Henderson
In reply to this post by Steve Litt
On 2019-11-19, Steve Litt <[hidden email]> wrote:
> In OpenBSD is there such a thing as a bind mount like they have in
> Linux?

No. The closest is probably "mount from 127.0.0.1 over NFS".