Reboot and re-link

classic Classic list List threaded Threaded
37 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

chohag
Lyndon Nerenberg writes:
> We are looking forward to that.  *However*, there is a lot to be
> said for regularly re-installing your hosts from scratch.  This
> ensures your installer scripts don't rot as host system "features"
> accrete over time.  This is prone to happen when you Ansible- or

Or as I like to put it: Reboot* often, to ensure that you can. Uptime is
overrated.

Matthew

[*] Or, as in this case, reinstall. It's more or less the same if you're
set up right.

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

Frank Beuth
In reply to this post by Lyndon Nerenberg (VE6BBM/VE7TFX)
On Sat, Jun 22, 2019 at 10:28:53AM -0700, Lyndon Nerenberg wrote:
>
>We are looking forward to that.  *However*, there is a lot to be
>said for regularly re-installing your hosts from scratch.  This
>ensures your installer scripts don't rot as host system "features"
>accrete over time.  This is prone to happen when you Ansible- or
>Puppet-manage servers.  Things get added, things get removed.  Often
>you miss hidden dependencies that sneak in; you don't want to be
>discovering those when you're trying to reinstall a production host
>after a catastrophic failure.

Yes, and being able to Ansible-manage even the re-installation would make the
whole process that much nicer :)

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

Lyndon Nerenberg (VE6BBM/VE7TFX)
Frank Beuth writes:

> Yes, and being able to Ansible-manage even the re-installation would make the
> whole process that much nicer :)

I started writing a rebuttal to this, but it quickly turned into writing
our design document for how we handle this internally across he data-
centre.  That's not something I can share.

Suffice to say this is not as simple a process as you might think it is.

--lyndon

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

chohag
In reply to this post by Frank Beuth
Frank Beuth writes:
> Yes, and being able to Ansible-manage even the re-installation would make the
> whole process that much nicer :)

Ansible is not the correct tool for this job; it can only configure and
maintain an _extant_ system.

None of the recent plethora of configuration management tools have
considered the scenario *before* an operating system has been
installed. All of them expect the server to exist and for secured
communication channels to have been established between it and the
master control system before they are operable. The vast majory seem to
solve this problem with the moral equivalent of blindly saying "yes" to
ssh's unexpected-fingerprint prompt.

If you wish to head down that rabbit-hole then best of luck to you.

FWIW I'm working on-and-off on a tool which specifically automates
*that* problem (build a new server/vm/chroot with zero human
interaction so Ansible et al. can subsequently and safely take over)
but what I've released so far is alpha quality at best.

Conveniently if you're only targetting OpenBSD then it's entirely
useless because, provided you can use PXE*, the OpenBSD developers have
already solved it.

Without Ansible.

Matthew

[*] The autoinstall/siteXX.tgz/etc. solution provided by the OpenBSD
developers is very good but there are some questions I have around
integrity on a potentially untrusted network. However as I'm trying to
target more than just OpenBSD, and I don't trust any network, I've
simply abandoned the idea of using PXE in my own environments so I
haven't looked into the answers to them. YMMV.

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

Frank Beuth
On Sat, Jun 22, 2019 at 10:29:22PM +0300, [hidden email] wrote:
>
>Ansible is not the correct tool for this job; it can only configure and
>maintain an _extant_ system.
>
>None of the recent plethora of configuration management tools have
>considered the scenario *before* an operating system has been
>installed. All of them expect the server to exist and for secured
>communication channels to have been established between it and the
>master control system before they are operable.

That's the interesting thing in my case (at least)... the system *IS* already
extant!

It has a nice shiny new Ubuntu/Debian/Fedora/centOS install that has just been
imaged onto it using the hosting provider's default tooling, and SSH is already
configured. (without blindly saying "yes" to the unexpected-fingerprint prompt)

Normally in this situation one would just use Ansible to harden the default
Linux install and configure whatever applications are needed. But in this case
I feel like hardening the Linux install even more, by replacing it with OpenBSD
:)

Maybe I'm wrong, but it seems like if this problem were well-solved then it
would make easier to use OpenBSD in many more applications and situations.

>FWIW I'm working on-and-off on a tool which specifically automates
>*that* problem (build a new server/vm/chroot with zero human
>interaction so Ansible et al. can subsequently and safely take over)
>but what I've released so far is alpha quality at best.
>
>Conveniently if you're only targetting OpenBSD then it's entirely
>useless because, provided you can use PXE*, the OpenBSD developers have
>already solved it.
>
>Without Ansible.
>
>Matthew
>
>[*] The autoinstall/siteXX.tgz/etc. solution provided by the OpenBSD
>developers is very good but there are some questions I have around
>integrity on a potentially untrusted network. However as I'm trying to
>target more than just OpenBSD, and I don't trust any network, I've
>simply abandoned the idea of using PXE in my own environments so I
>haven't looked into the answers to them. YMMV.

I'd love to see your tool. PXE is mostly not available for this case (in
general I am trying to target the most generic possible situation).

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

chohag
Frank Beuth writes:
> That's the interesting thing in my case (at least)... the system *IS* already
> extant!

And how have you introduced it to your command-and-control system? That
is, ultimately, the key.

> It has a nice shiny new Ubuntu/Debian/Fedora/centOS install that has just been
> imaged onto it using the hosting provider's default tooling, and SSH is already
> configured. (without blindly saying "yes" to the unexpected-fingerprint prompt)

That is what these tools depend on, but how is such a state of "already
configured" achieved before the tool that does the configuration gets
involved? This is why these are not the right tool for the job.

> Normally in this situation one would just use Ansible to harden the default
> Linux install and configure whatever applications are needed. But in this case
> I feel like hardening the Linux install even more, by replacing it with OpenBSD
> :)

If you're hardening a system you've already lost. Systems should be hard
by default.

> Maybe I'm wrong, but it seems like if this problem were well-solved then it
> would make easier to use OpenBSD in many more applications and situations.

It's not well-solved because nobody tries to solve it. Installing
systems in the first instance is assumed to be a manual process and no
further thought is applied because you've got your clonable image, right?

Actually that's not entirely true but I've yet to find a *portable* solution.

> I'd love to see your tool.

Oooh sir!

> PXE is mostly not available for this case (in
> general I am trying to target the most generic possible situation).

PXE is only applicable in situations where the network can be guaranteed
to be trusted; you're letting your DHCP server give you unverifiable
code to execute and if you can't trust the network you can't trust the
DHCP response.

I wrote stash so that I could deploy my own servers without trust being
an issue. It got as far as that and I (temporarily; I'll get back to it)
lost interest. Nobody is paying me for this, I'm just bored. The
documentation is ... poor. But it works. In my little network there are
currently 6 distinct servers, all built using it with zero manual
interaction.

https://github.com/chohag/stash

Enjoy.

Happy to answer questions (I need some critical feedback). I plan to make
more out of this but for the time being it's little more than a toy.

Matthew

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

Frank Beuth
In reply to this post by Frank Beuth
On Sat, Jun 22, 2019 at 03:06:30AM +0100, Andrew Luke Nesbit wrote:
>On 21/06/2019 19:02, Frank Beuth wrote:
>> I don't want to re-open the hostilities, but installing OpenBSD via
>> Ansible is very relevant to my interests.
>
>I feel exactly the same way and am surprised that Ansible caused
>hostilities.  Can you send me a link to the thread where this happened
>please?  I want to know why, i.e., pros and cons.

It doesn't look to me like Ansible as such caused any trouble, it was someone's
use of Ansible in an unsupported way (and probably many other configuration
choices), leading to further problems, and then people got angry.

For details search the misc@ archives for "Reboot and re-link" (the subject
line), things got spread across multiple threads:
https://marc.info/?l=openbsd-misc&w=2&r=1&s=Reboot+and+re-link&q=t

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

Brian Brombacher
Using Ansible to reinstall the operating system is like trying to turn a four door sedan into a monster truck with a hammer.

Wrong tool for the job.

> On Jun 22, 2019, at 6:46 PM, Frank Beuth <[hidden email]> wrote:
>
>> On Sat, Jun 22, 2019 at 03:06:30AM +0100, Andrew Luke Nesbit wrote:
>>> On 21/06/2019 19:02, Frank Beuth wrote:
>>> I don't want to re-open the hostilities, but installing OpenBSD via
>>> Ansible is very relevant to my interests.
>>
>> I feel exactly the same way and am surprised that Ansible caused
>> hostilities.  Can you send me a link to the thread where this happened
>> please?  I want to know why, i.e., pros and cons.
>
> It doesn't look to me like Ansible as such caused any trouble, it was someone's use of Ansible in an unsupported way (and probably many other configuration choices), leading to further problems, and then people got angry.
>
> For details search the misc@ archives for "Reboot and re-link" (the subject line), things got spread across multiple threads:
> https://marc.info/?l=openbsd-misc&w=2&r=1&s=Reboot+and+re-link&q=t
>

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

Andrew Luke Nesbit-2
In reply to this post by chohag
On 22/06/2019 19:52, [hidden email] wrote:
> Lyndon Nerenberg writes:
>> We are looking forward to that.  *However*, there is a lot to be
>> said for regularly re-installing your hosts from scratch.  This
>> ensures your installer scripts don't rot as host system "features"
>> accrete over time.  This is prone to happen when you Ansible- or
>
> Or as I like to put it: Reboot* often, to ensure that you can. Uptime is
> overrated.

In my experience, there are indeed benefits to rebooting production
servers on a scheduled maintenance basis.

If long-running processes are running then there is some chance that the
system is suffering memory fragmentation.  This will make your server
slower.  I think it could also/either trigger an OOM.

Untested changes could have been deployed since last reboot.  They might
have unpredictable side-effects on the startup scripts.

Some benefits of a regular, scheduled reboot cycle:d

1.  Rebooting will clear up memory fragmentation.

2.  Rebooting will improve confidence that it is possible to reboot the
server and in a clean way and improve confidence that the startup
scripts still work.  After initial boot it will progress to its intended
runtime state.  ("Have you tried turning it off and then back on again?")

    This is particularly important in a situation where a server
crashes, needs unscheduled maintenance, or you need to decide whether it
is safe to reboot

  (A thought just occurred to me that the following reasons might be a
part of chaos engineering, which I have been meaning to investigate but
haven't found time yet.)

-- OpenPGP key: EB28 0338 28B7 19DA DAB0  B193 D21D 996E 883B E5B9

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link (ignore previously sent message)

Andrew Luke Nesbit-2
In reply to this post by chohag
[Please ignore the previous message I sent on this topic.  I
accidentally pressed 'Send' before my message was complete.]

On 22/06/2019 19:52, [hidden email] wrote:
> Lyndon Nerenberg writes:
>> We are looking forward to that.  *However*, there is a lot to be
>> said for regularly re-installing your hosts from scratch.  This
>> ensures your installer scripts don't rot as host system "features"
>> accrete over time.  This is prone to happen when you Ansible- or
>
> Or as I like to put it: Reboot* often, to ensure that you can. Uptime is
> overrated.

In my experience, there are indeed benefits to rebooting production
servers on a scheduled maintenance basis.  Here are two example problems
that it could help with:

1.  If long-running processes are running then there is some chance that
the system is suffering memory fragmentation.  This will make your
server slower.  I think it could also/either trigger an OOM.

2.  Untested changes could have been deployed since last reboot.  They
might have unpredictable effects on the startup scripts.

3.  The startup scripts might no longer work _at all_ if the server has
been in continual operation for a long time, such as five years.  This
can happen due to the phenomenon known as "bit rot".

Some benefits of a regular, scheduled reboot cycle:

1.  Rebooting will clear up memory fragmentation.

2.  Rebooting will improve confidence that it is possible to reboot the
server in a clean way and that the startup scripts still work.  After
initial boot the server will progress to its intended runtime state.
("Have you tried turning it off and then back on again?")

    Having this kind of confidence is particularly important when a
server crashes or when you need to perform unscheduled maintenance to
deploy to urgent hotfix.

    Another thought literally just occurred to me.  Regular
_unscheduled_ reboots seem like a typical chaos engineering technique.
I haven't investigated chaos engineering closely but I'd be surprised if
it isn't.

Andrew
--
OpenPGP key: EB28 0338 28B7 19DA DAB0  B193 D21D 996E 883B E5B9

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

Frank Beuth
In reply to this post by chohag
On Sun, Jun 23, 2019 at 01:37:35AM +0300, [hidden email] wrote:

>Frank Beuth writes:
>> That's the interesting thing in my case (at least)... the system *IS* already
>> extant!
>
>And how have you introduced it to your command-and-control system? That
>is, ultimately, the key.
>
>> It has a nice shiny new Ubuntu/Debian/Fedora/centOS install that has just been
>> imaged onto it using the hosting provider's default tooling, and SSH is already
>> configured. (without blindly saying "yes" to the unexpected-fingerprint prompt)
>
>That is what these tools depend on, but how is such a state of "already
>configured" achieved before the tool that does the configuration gets
>involved? This is why these are not the right tool for the job.

I can see that you've never worked with a VPS provider, which is understandable
since a VPS is going to be a bit less secure than a dedicated server.

How this works is fairly simple:

the provider has a web interface where you can choose whether you want Debian
or Ubuntu on your server (often not even the latest version) and then you click
the button and the automated system images your server with that OS.

Then once the imaging and set-up is done, they email you the root password and
IP address so you can log in for the first time over SSH.

Sometimes other things are possible (VNC, uploading custom ISOs to image) but
what I described is all you can count on from provider to provider.

>I wrote stash so that I could deploy my own servers without trust being
>an issue. It got as far as that and I (temporarily; I'll get back to it)
>lost interest. Nobody is paying me for this, I'm just bored. The
>documentation is ... poor. But it works. In my little network there are
>currently 6 distinct servers, all built using it with zero manual
>interaction.
>
>https://github.com/chohag/stash
>
>Enjoy.
>
>Happy to answer questions (I need some critical feedback). I plan to make
>more out of this but for the time being it's little more than a toy.

Huh. How does the ISO actually end up in the server -- is the user expected to
manually mount the ISO in the VM system?

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

chohag
Frank Beuth writes:
> <some patronising bullshit about how today's crop of ISPs work>

You go ahead and continue to trust your VPS without taking any care to
consider where your software comes from.

It's choices like that which make "hardening" even be a thing. Have you
considered _not_ building a system on a foundation made of cheese?

Have fun with that.

Matthew

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

Frank Beuth
On Sun, Jun 23, 2019 at 10:49:22AM +0300, [hidden email] wrote:
>Frank Beuth writes:
>> <some patronising bullshit about how today's crop of ISPs work>
>
>You go ahead and continue to trust your VPS without taking any care to
>consider where your software comes from.
>
>It's choices like that which make "hardening" even be a thing. Have you
>considered _not_ building a system on a foundation made of cheese?

Of course, but those are the constraints I have to deal with.

Naturally a dedicated bare-metal server would be preferable, but that is not
always possible. Given the tremendous popularity of VPS hosting, it does seem
like I am not alone.

Just because there is risk on the back-end of the system does not mean we
should be careless in other respects.

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

Gregory Edigarov-5
In reply to this post by Frank Beuth

On 21.06.19 21:02, Frank Beuth wrote:

> On Wed, Jun 19, 2019 at 11:29:32PM +0200, Maxim Bourmistrov wrote:
>> Installing via NOT RECOMMENDED WAY(following upgrade65.html) -
>> scripting on
>> steroides (ansible).
>
> I don't want to re-open the hostilities, but installing OpenBSD via
> Ansible is very relevant to my interests. Previously discussed on this
> list was a very roundabout approach using Qemu -- is there a better
> way now?
>
it's all easy given it is some IaaS provider, just use terraform to
create the ground, (terraform could also be used to upload keys, and do
some preconfiguration) then call ansible.

my worst timing on AWS is ~20 minutes.

baremetal servers are more interesting beasts here but if your
colocation/infrastructure provider allows for boot image uploads that's
also quite doable with existing tools.


Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

David Sastre
In reply to this post by Frank Beuth
On Fri, Jun 21, 2019 at 11:50 PM Frank Beuth <[hidden email]> wrote:

>
> On Fri, Jun 21, 2019 at 12:36:22PM -0700, Misc User wrote:
> >I use PXE + install.conf + siteXX.tgz + siteXX-%hostname%.tgz for my
> >installs.  I also have an rc.firsttime to download and install the
> >required packages.
>
> Thanks, but neither this nor the autoinstall suggestion seem applicable for my
> use case.
>
> I am dealing with virtualized servers which usually start out as
> Ubuntu/Debian/Fedora images, then the hosting provider supplies the IP address
> and root password for a first-time SSH login.
>
> In many cases it is not possible to upload an ISO to be used as server
> installation media, and VNC consoles (if available) are often not even
> encrypted. (How would you feel about installing OpenBSD and then having your
> root password sent in plaintext at the very beginning?)
>
> I realize installing OpenBSD under these constraints is rather like installing
> a ship in a bottle, but it seemed worth it to ask...

Apologies for a late reply to this thread.

I would not consider ansible as the right tool to provision a system
from scratch (as in PXE booting, etc...).
Ansible is better used on a system you can connect to using SSH and
perform actions as required, with or without doas, as you surely know.
You don't mention cloud providers/VPS you are trying to bootstrap
OpenBSD to, but the way I'd tackle this situation, if I have
understood your use case correctly, is as follows:

- Find out if the specific cloud provider is supported by packer [1]
(packer itself can be run in OpenBSD[2]).
  Custom builders can be written, but might be overkill for the task at hand.
- If the answer is yes, create a template to bootstrap an OpenBSD image.
  You can find many examples online[3]. The specifics of the packer
template vary depending on the cloud provider,
  but usually you can bootstrap the system from an ISO (or an existing
AMI, if in AWS), and finish provisioning
  the configuration using ansible.
- From that point onwards, use ansible to further modify the settings
in the managed system, prevent configuration drift, if your use case
includes the eventual manual actions
  and/or reflect in ansible tasks/playbooks any modifications applied
to the system, so as to be able to reproduce them again if required.
- One further option if the OS provided by the hosting service has to
be a Linux system would be to consider using it as an hypervisor for
OpenBSD VMs.

Do let me know if this is the type of provisioning you are looking into.

Regards.

[1] https://packer.io/docs/builders/index.html
[2] https://packer.io/downloads.html
[3] https://github.com/kaorimatz/packer-templates/blob/master/openbsd-6.3-amd64.json
# there are many other examples online, this is just one of them, I
haven't tested it

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

Frank Beuth
In reply to this post by Gregory Edigarov-5
On Mon, Jun 24, 2019 at 11:43:36AM +0300, Gregory Edigarov wrote:
>>I don't want to re-open the hostilities, but installing OpenBSD via
>>Ansible is very relevant to my interests. Previously discussed on
>>this list was a very roundabout approach using Qemu -- is there a
>>better way now?
>>
>it's all easy given it is some IaaS provider, just use terraform to
>create the ground, (terraform could also be used to upload keys, and
>do some preconfiguration) then call ansible.

Terraform looks great, but while some of the providers I need to support are
listed (https://www.terraform.io/docs/providers/index.html ) that's not true of
all of them, and probably never will be. In general, being bound to Big Cloud
(AWS / DigitalOcean / et al) is not desirable.

On top of this, my objective for this project is to support the most generic
and standardised possible interface ("image with the provider's web interface
and SSH in after") rather than develop a system that implicitly encourages
lock-in.

Nevertheless looks like a superb tool if it fits.

Reply | Threaded
Open this post in threaded view
|

Re: Ansible install Re: Reboot and re-link

Frank Beuth
In reply to this post by David Sastre
On Mon, Jun 24, 2019 at 10:59:44AM +0200, David Sastre wrote:

>I would not consider ansible as the right tool to provision a system
>from scratch (as in PXE booting, etc...).
>Ansible is better used on a system you can connect to using SSH and
>perform actions as required, with or without doas, as you surely know.
>You don't mention cloud providers/VPS you are trying to bootstrap
>OpenBSD to, but the way I'd tackle this situation, if I have
>understood your use case correctly, is as follows:
>
>- Find out if the specific cloud provider is supported by packer [1]
>(packer itself can be run in OpenBSD[2]).
>  Custom builders can be written, but might be overkill for the task at hand.
>- If the answer is yes, create a template to bootstrap an OpenBSD image.
>  You can find many examples online[3]. The specifics of the packer
>template vary depending on the cloud provider,
>  but usually you can bootstrap the system from an ISO (or an existing
>AMI, if in AWS), and finish provisioning
>  the configuration using ansible.

And what if the answer is no? You didn't mention that :)

Yes, Ansible is not really the right tool for installing new images onto
machines or re-imaging server. Yes, Packer and Terraform (both from the same
company) are superb and wonderful ways of managing machines on AWS/Azure/the
rest of the API-enabled IaaS crowd.

However great the Big Cloud providers are, though, sometimes they are not
suitable for a project, and instead one is left in the position of bartering
a cow in exchange for a VPS instance at HostElbonia, where you're lucky if the
API is RFC 2549 compliant.

And in general, one of the things I like most about OpenBSD is the design for
simplicity, emphasizing standard, boring interfaces that are extremely
reliable and which don't require the "hot new shiny object" du jour.

So while yes, automated provisioning of AWS over the API is an option, as is
setting up the Linux VPS as a hypervisor running OpenBSD... it doesn't quite
feel right.

It sounds like a custom bsd.rd with auto_install.conf, dropped onto the /boot
partition by Ansible (or some other script or management tool!) is the way to
go for this.

Ihave a few other projects to deal with, but it is on my to-do list, and if I
come up with something potentially useful to others then I will try and write
it up in some form.

12