OpenBSD on VMware ESXi

classic Classic list List threaded Threaded
29 messages Options
12
Reply | Threaded
Open this post in threaded view
|

OpenBSD on VMware ESXi

Roderick

Hallo!

As far as I read in WWW, OpenBSD do run on VMware ESXi out of the box.

What does run better on amd64 virtual machine? i386 or amd64?
Are there reasons to preffer one to the other?

Any recommendations in general? Current or stable?

I have a virtual server, just for testing, at the moment with debian
and I find it awfull. Is there any reasong to keep it with linux?

A detail: the console is in WWW, almost unreadable small fonts,
unstable, high latency (result of low price :). The best would
be a short installation path to get a listening sshd and end the
installation with shell login.

Thanks for any hint
Rodrigo

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD on VMware ESXi

Tom Smyth
Hi Roderick,

use amd64 ... as  it offers better mitigation's and has better support
overall

also if your hypervisor / machine offers nested virtualisation you would
be able to run vmm inside your machine...
i386 does not have support for vmm anymore

use stable if you want to run in production, and want to avoid rebooting
use current if you want to run and test latest features,  ( and you dont
mind rebooting to
upgrade regularly)
sysupgrade in current makes running current much easier,

Vmxnet worked fine for me  in the past.

I have had issues with vmware 6.0 ... But these were solved with
vmware6.0 Update 2
Thanks





On Wed, 22 May 2019 at 12:09, Roderick <[hidden email]> wrote:

>
> Hallo!
>
> As far as I read in WWW, OpenBSD do run on VMware ESXi out of the box.
>
> What does run better on amd64 virtual machine? i386 or amd64?
> Are there reasons to preffer one to the other?
>
> Any recommendations in general? Current or stable?
>
> I have a virtual server, just for testing, at the moment with debian
> and I find it awfull. Is there any reasong to keep it with linux?
>
> A detail: the console is in WWW, almost unreadable small fonts,
> unstable, high latency (result of low price :). The best would
> be a short installation path to get a listening sshd and end the
> installation with shell login.
>
> Thanks for any hint
> Rodrigo
>
>

--
Kindest regards,
Tom Smyth.
Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD on VMware ESXi

Janne Johansson-3
In reply to this post by Roderick
Den ons 22 maj 2019 kl 12:52 skrev Roderick <[hidden email]>:

> Hallo!
> As far as I read in WWW, OpenBSD do run on VMware ESXi out of the box.
> What does run better on amd64 virtual machine? i386 or amd64?
> Are there reasons to preffer one to the other?
>

The ESX template for 64-bit comes with more recent "hardware" in the
environment IIRC, so it will be less tweaking the supplied virtualized
hardware if you select 64bit guest instead of 32bit.
Apart from that, 64bit is better on both virtual and real hw.

--
May the most significant bit of your life be positive.
Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD on VMware ESXi

Reyk Floeter-2
On Wed, May 22, 2019 at 01:43:35PM +0200, Janne Johansson wrote:

> Den ons 22 maj 2019 kl 12:52 skrev Roderick <[hidden email]>:
>
> > Hallo!
> > As far as I read in WWW, OpenBSD do run on VMware ESXi out of the box.
> > What does run better on amd64 virtual machine? i386 or amd64?
> > Are there reasons to preffer one to the other?
> >
>
> The ESX template for 64-bit comes with more recent "hardware" in the
> environment IIRC, so it will be less tweaking the supplied virtualized
> hardware if you select 64bit guest instead of 32bit.
> Apart from that, 64bit is better on both virtual and real hw.
>

But unfortunately, there is no openbsd template.  So use "Other 64bit"
and enable vmxnet3 manually, as mentioned in vmx(4):

     The following entry must be added to the VMware configuration file to
     provide the vmx device:

           ethernet0.virtualDev = "vmxnet3"

This is much better than the e1000 emulation.

Reyk

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD on VMware ESXi

Maxim Bourmistrov-5
I think FreeBSD or any Linux template will work just fine and add vmxnet3.
However, last I checked (1year ago) vmxnet3 been less stable than e1000 under pressure.

Sent from my iDevice

> 22 мая 2019 г., в 13:47, Reyk Floeter <[hidden email]> написал(а):
>
>> On Wed, May 22, 2019 at 01:43:35PM +0200, Janne Johansson wrote:
>> Den ons 22 maj 2019 kl 12:52 skrev Roderick <[hidden email]>:
>>
>>> Hallo!
>>> As far as I read in WWW, OpenBSD do run on VMware ESXi out of the box.
>>> What does run better on amd64 virtual machine? i386 or amd64?
>>> Are there reasons to preffer one to the other?
>>>
>>
>> The ESX template for 64-bit comes with more recent "hardware" in the
>> environment IIRC, so it will be less tweaking the supplied virtualized
>> hardware if you select 64bit guest instead of 32bit.
>> Apart from that, 64bit is better on both virtual and real hw.
>>
>
> But unfortunately, there is no openbsd template.  So use "Other 64bit"
> and enable vmxnet3 manually, as mentioned in vmx(4):
>
>     The following entry must be added to the VMware configuration file to
>     provide the vmx device:
>
>           ethernet0.virtualDev = "vmxnet3"
>
> This is much better than the e1000 emulation.
>
> Reyk
>

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD on VMware ESXi

R0me0 ***
Vmware ESXI detects as FreeBSD 32bit.

Set network interface to vmxnet3.

Also you can use pvscsi driver ( I had some issues with filesystem
corruption,
there is a weird bug, but there is a workaround.)

In general buslogic is more resilient.

Regards,


Em qua, 22 de mai de 2019 às 14:26, mxb <[hidden email]> escreveu:

> I think FreeBSD or any Linux template will work just fine and add vmxnet3.
> However, last I checked (1year ago) vmxnet3 been less stable than e1000
> under pressure.
>
> Sent from my iDevice
>
> > 22 мая 2019 г., в 13:47, Reyk Floeter <[hidden email]> написал(а):
> >
> >> On Wed, May 22, 2019 at 01:43:35PM +0200, Janne Johansson wrote:
> >> Den ons 22 maj 2019 kl 12:52 skrev Roderick <[hidden email]>:
> >>
> >>> Hallo!
> >>> As far as I read in WWW, OpenBSD do run on VMware ESXi out of the box.
> >>> What does run better on amd64 virtual machine? i386 or amd64?
> >>> Are there reasons to preffer one to the other?
> >>>
> >>
> >> The ESX template for 64-bit comes with more recent "hardware" in the
> >> environment IIRC, so it will be less tweaking the supplied virtualized
> >> hardware if you select 64bit guest instead of 32bit.
> >> Apart from that, 64bit is better on both virtual and real hw.
> >>
> >
> > But unfortunately, there is no openbsd template.  So use "Other 64bit"
> > and enable vmxnet3 manually, as mentioned in vmx(4):
> >
> >     The following entry must be added to the VMware configuration file to
> >     provide the vmx device:
> >
> >           ethernet0.virtualDev = "vmxnet3"
> >
> > This is much better than the e1000 emulation.
> >
> > Reyk
> >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD on VMware ESXi

Adam Thompson
In reply to this post by Maxim Bourmistrov-5
On 2019-05-22 09:25, mxb wrote:
> I think FreeBSD or any Linux template will work just fine and add
> vmxnet3.
> However, last I checked (1year ago) vmxnet3 been less stable than
> e1000 under pressure.

Don't use the Linux templates.  I would recommend against using the
FreeBSD templates, and go with "Other (64-bit)" instead.  YMMV on using
FreeBSD vs Other, I haven't seen consistent results here yet... just
don't pick Linux, or DOS, or Windows - in some situations, that allows
VMware to take certain shortcuts that are based on assumptions about the
Linux/Win/etc. kernel & device drivers that (probably) aren't valid
under OpenBSD.

Various people have reported different problems with vmxnet3; I'm aware
of at least 4 or 5 different environment-specific issues (i.e. can't be
reproduced on any other vSphere/ESXi system).  I have some of those
problems, and I cannot reproduce them outside my production environment,
but they don't prevent me from running OpenBSD.

Workarounds:
* use vmxnet2
* use e1000

If vmxnet3 and pvscsi work for you (you'll know pretty darn fast!), use
them.  When they work, which is usually (in my experience), they're
generally very stable and high-performing compared to the emulated h/w
(e1000, lsisas, lsiscsi, buslogic).

I also experience timer issues, and I've had to specify
kern.timecounter.hardware=i8254 in sysctl.conf.  This is likely a VMware
problem, not an OpenBSD problem, but it's non-trivial to diagnose.  
(Even i8254 doesn't work perfectly: e.g. usleep() isn't very accurate in
my VMs!)  I'm also running these VMs on very heavily-loaded hosts, which
is probably a factor.

My disk write throughput is horrible, but that's an interaction between
how OpenBSD does writes, how VMware handles writes into thin-provisioned
disks, and how my NFS storage handles writes on thin-provisioned
volumes; it's not an OpenBSD problem, strictly speaking, although that's
the only place it's normally visible.

Overall, OpenBSD works well under ESXi, but there are semi-random
problems that do have workarounds.

Several years ago, Theo noted (approximately, I'm going from memory here
AND paraphrasing) that it was hard enough for OpenBSD to handle broken
hardware implementations, it's exponentially harder to handle an
incorrect software emulation of hardware that was incorrect in the first
place.  This has proven accurate, and VMware doesn't really care much
about OpenBSD, since I doubt it even registers on their radar so they're
not terribly interested in fixing VMware bugs that are only visible
under OpenBSD.  (If you have a support contract, please submit bug
reports to VMware.  If enough of us do so, they might start fixing some
of the problems.)

-Adam

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD on VMware ESXi

Stuart Henderson
In reply to this post by Reyk Floeter-2
On 2019-05-22, Reyk Floeter <[hidden email]> wrote:

>
> But unfortunately, there is no openbsd template.  So use "Other 64bit"
> and enable vmxnet3 manually, as mentioned in vmx(4):
>
>      The following entry must be added to the VMware configuration file to
>      provide the vmx device:
>
>            ethernet0.virtualDev = "vmxnet3"
>
> This is much better than the e1000 emulation.

vmxnet3 suffers kernel panics under some conditions, e1000 is rock solid
for me.


Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD on VMware ESXi

Igor Podlesny-2
On Wed, 22 May 2019 at 23:06, Stuart Henderson <[hidden email]> wrote:
[...]
> vmxnet3 suffers kernel panics under some conditions, e1000 is rock solid
> for me.

Any known similarities in regards of vio -- VirtIO network device
(bhyve, KVM, QEMU, and VirtualBox)?

--
End of message. Next message?

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD on VMware ESXi

Roderick
In reply to this post by Adam Thompson

I thanks a lot! I need a litle more help, I do not want to give up. :)

I use now the terms of the provider, I do not like virtual machines
and have no experience with them.

I loaded cd65.iso in the providers platform as "other 64-bit
OS image". Then I loaded it in the (virtual) DVD rom and started the
server. bsd.rd booted, vmx0 appeared and did work, I installed the
system with internet connection.

The installed system seems to boot. I can start and stop it clicking
in the web, with the offered "software method" that supposedly does
not work without the "VMWare tools". But it complains that
without "VMWare tools" there is no console, and I do not get the damned
WWW console, called "KVM console" by the provider.

And the installed and booted system do not react to ping and
ssh, probably a net configuration is necessary. This means: I
must use c65.iso as rescue system.

c65.iso as (virtual) DVD do boot and I get the console, although
I also get the same messages that without the  "VMWare tools" that
is not possible. Whay the difference?

During installation there was something strange, but I think that
it is not the origin of the problem. The automatic partition offered me:

a  2048 on /
c  41943040
i 997376  extfs (?!) to be mounted nowhere
j 40943616 unknown to be mounted nowhere

Although this was very suspicious, I answered yes for not dealing long
with the console. It seems, it did no do partition and fs. It
did not load the sets, it complained "cannot determine prefetch
area ...". Perhaps a bug?

I did the partition manually (what I always use to do). 200 times more
than that a on /, 512 times more that that a as b (swap), and the rest
as e on /usr. There was then no problem with the installation.

But was that ext2fs that appeared in the automatic partition
necessary for something?!

Thanks again for any hint
Rodrigo

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD on VMware ESXi

Stuart Henderson
In reply to this post by Igor Podlesny-2
No idea, I don't run those.

--
Sent from a phone, apologies for poor formatting.

On 22 May 2019 17:20:16 Igor Podlesny <[hidden email]> wrote:

> On Wed, 22 May 2019 at 23:06, Stuart Henderson <[hidden email]> wrote:
> [...]
>> vmxnet3 suffers kernel panics under some conditions, e1000 is rock solid
>> for me.
>
> Any known similarities in regards of vio -- VirtIO network device
> (bhyve, KVM, QEMU, and VirtualBox)?
>
> --
> End of message. Next message?



Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD on VMware ESXi

Roderick

On Wed, 22 May 2019, Roderick wrote:

> The installed system seems to boot.

Or perhaps not. I put in /etc/hostname.vmx0 with the help of cd65.iso:
dhcp. But got no connection. Dificult to know without console.

In the rescue disk was "dhclient vmx0" enough for getting connection.

I did MBR, no gpt. I suppose that is not the problem.

Rodrigo

Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD on VMware ESXi

Roderick

Of course never booted: /var/log/messages is empty. :)

I was too sleepy and optimistic.


Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD on VMware ESXi

Rudy Baker
There's a bug in ESXI 6.5 specifically with vmxnet 3. We we're using Linux
when it was noticed but anytime one of our floating ips (haproxy,
keepalived) would switch to the node with vmxnet 3, instant kernel panic. I
wonder if the problem is happening to you.

https://kb.vmware.com/s/article/2151480

Like others have mentioned, E1000 doesn't have the problem and the issue
also goes away after upgrading to 6.7.



On Wed, May 22, 2019, 4:24 PM Roderick, <[hidden email]> wrote:

>
> Of course never booted: /var/log/messages is empty. :)
>
> I was too sleepy and optimistic.
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: OpenBSD on VMware ESXi

Steven Shockley
In reply to this post by Roderick
On 5/22/2019 6:46 AM, Roderick wrote:
> Any recommendations in general? Current or stable?

I've had bad luck with softupdates and OpenBSD on ESXi when the ESXi
datastore is on nfs.  (Encountered on ESX 5.0, 5.1, and 5.5; I must not
learn from my mistakes.)  From what I can tell, if the nfs datastore
takes too long to respond OpenBSD thinks the disk has gone away and
panics.  It's not OpenBSD's fault; if a real disk stopped responding a
panic is probably the best result.

Probably some tuning could be done to make the ESXi nfs timeout match
the OpenBSD timeout, but it's easier to just not use softupdates and
match the backing disk to my performance requirements.

Note that softupdates are not enabled by default.

Reply | Threaded
Open this post in threaded view
|

Installer sucks ! (Re: OpenBSD on VMware ESXi)

Roderick

Please, delete the cc to [hidden email] in any answer.

I am now, after hours typing in the damned web console and dealing
with the buggy installer, a little bit furious. This is definitively
not the OpenBSD I know!

I did manage to install OpenBSD in VMWare, with the "autopartition", but
I want a custom partition due to my spare resources. I will now do a
pause to calm myself and then continue trying. I appreciate any hint
and thank for it very much.

My remarks:

(1) There no way to write a custom disklabel. After doing it, after giving the
     command w and leaving with x, or after leaving with q, the installer
     overwrites it with something arbitrary that may be unusable (then
     one will note it no later than when loading the sets due to error
     "cannot determine prefetch ...").

(2) "disklabel -E" in cd65.iso puts a cpg=1 for all partitions: is that
      correct?

(4) No way to leave the disklabel as it is. I wrote one with the shell
     of cd65.iso (and cpg=1), also did nefwfs on the partitions, but
     the installer do not show the partition it anywhere. One is compelled
     to type again, and again comes the unusable arbitrary modification.

(5) Auto allocaton puts partitions for X11 even if one selects
     that one will run no X11.

(6) No vi in cd65.iso (but at least ed).

Rodrigo

Reply | Threaded
Open this post in threaded view
|

Re: Installer sucks ! (Re: OpenBSD on VMware ESXi)

Roderick

On Thu, 23 May 2019, Otto Moerbeek wrote:

> You must be doing something wrong. Since it installer surely leats you
> use a custom label. But since you are not showgin waht you did and you
> start insulting remarks, you won't get much help.

Excuse me, although my words was not flowers, they were no insult.
And there was no intention to insult, but it is realy nerving to deal
with this web console. I cannot even do "copy and paste" in it for
showing what I did. That is why I described it. It is sure a bug there.
As said: it is not my previous experience with OpenBSD.

There is sure a bug there:

(1) if you installed a label before, the installer does not offer
     it to you.

(2) it changes the "custom label" you write in the process. Really
     changes it. That can never be something else than a bug.

(3) "disklabel -E /dev/sd0" puts alway cpg=1. The installer in
     the autoinstall puts numbers like 10277, 16384.

Rodrigo

Reply | Threaded
Open this post in threaded view
|

Re: Installer buggy (Re: OpenBSD on VMware ESXi)

Roderick

Some more details about my experience. Excuseme that I dont take
fotos of the screen and just describe, with the data y wrote down.

As I wrote, yesterday my instalation did not work. Today morning
I began to inspect with the shell of cd65.iso the partitions.

fdisk gave me  41943040 sectors divided as follows:

   start    size
0*  2048     997376  Linux files
1 999424   40943616  Linux LVM

disklabel gave me the sizes I typed:

a   409600   0       4.2BSD
b  1048576   409600  swap
c 41943040   0
e 40484864  1458176  4.2BSD

you see there the offset zero, no MBR. That is why the system did not
boot. I went to the shell and did "fdisk -iy sd0". Now I have a partition
3 with the whole, 64 till 41942976. I started the installer and
got an installation, as described, only with the automatic set values.

The server hoster offers two kinds of shutting down the system by
clicking on their web plattform: software and hardware. If I restart
with "hardware", the system does not go up anymore, unless I reinstall
the whole system. I though the hoster spoiled my disk because it
writes in the first 2048 sectors. I went to install again, but with
an offset of 2048 in the OpenBSD fdisk partition: no way, the installer
ignores my settings.

Rodrigo

Reply | Threaded
Open this post in threaded view
|

Re: Installer sucks ! (Re: OpenBSD on VMware ESXi)

Arnaud BRAND
In reply to this post by Roderick
The web console copy/paste functionnality is a VMWare limitation.
I don't think it ever worked.
It think would require the console to emulate/simulate key presses
depending on what is pasted and somehow assuming what the VM keymap is.

I didn't try to install 6.5 on ESXI yet, but I definitely installed 6.4.
On a lot of ESXi versions from 5.5 to 6.7.
So you could try to install 6.4 to see if you have the same problems ?

Never experienced your problems, although I experienced some strange
behaviors with disklabeling (if I remember well sometimes it couldn't
install the bootloader or wouldn't boot after install).

They were resolved by :
- dropping to the shell at the start of the install
- fdisk -i <the disk>
- return to the install and proceed normally

Never tried to install a custom label on ESXi, but did it sucessfully
on virtualbox with 6.5 without any issue (not even needed to fdisk -i).

What ESXi version are you running ?
What disk controller are you showing to OpenBSD ?

AB

Le 2019-05-23 12:17, Roderick a écrit :

> On Thu, 23 May 2019, Otto Moerbeek wrote:
>
>> You must be doing something wrong. Since it installer surely leats you
>> use a custom label. But since you are not showgin waht you did and you
>> start insulting remarks, you won't get much help.
>
> Excuse me, although my words was not flowers, they were no insult.
> And there was no intention to insult, but it is realy nerving to deal
> with this web console. I cannot even do "copy and paste" in it for
> showing what I did. That is why I described it. It is sure a bug there.
> As said: it is not my previous experience with OpenBSD.
>
> There is sure a bug there:
>
> (1) if you installed a label before, the installer does not offer
>     it to you.
>
> (2) it changes the "custom label" you write in the process. Really
>     changes it. That can never be something else than a bug.
>
> (3) "disklabel -E /dev/sd0" puts alway cpg=1. The installer in
>     the autoinstall puts numbers like 10277, 16384.
>
> Rodrigo

Reply | Threaded
Open this post in threaded view
|

Re: Installer buggy (Re: OpenBSD on VMware ESXi)

janus
In reply to this post by Roderick
Hi Rodrigo,

IIRC, pvscsi used to eat up first write to the paravirtual storage
device with VMware. Not sure what's the current situation as I tend to
use LSI Logic SAS.

Also, the first eaten-up write would explain why you're still seeing
Linux partitions instead of OpenBSD.


# fdisk sd0
Disk: sd0       geometry: 14593/255/63 [234441648 Sectors]
Offset: 0       Signature: 0xAA55
            Starting         Ending         LBA Info:
 #: id      C   H   S -      C   H   S [       start:        size ]
-------------------------------------------------------------------------------
 0: 00      0   0   0 -      0   0   0 [           0:           0 ]
unused
 1: 00      0   0   0 -      0   0   0 [           0:           0 ]
unused
 2: 00      0   0   0 -      0   0   0 [           0:           0 ]
unused
*3: A6      0   1   2 -  14592 254  63 [          64:   234436481 ]
OpenBSD

jvl

P.S. Description or even screenshots would help to know where you got
stuck.

12