struggling with qemu guests using >256M memory

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

struggling with qemu guests using >256M memory

David Cantrell-3
I am unable to start a qemu guest with more than 256M of memory, despite
having 4G in my system.  I get ENOMEM and qemu exits.  Using 256M or
less works just fine.

After some debugging, I see that qemu uses valloc(), which then uses
malloc() and that's preventing anything larger than 256M.

Out of curiosity, I wrote a simple test program that used valloc() to
ask for 256M.  Then I tried 512M, and got the failure I expected.
Debugging passed NKMEMPAGES in front of my eyes and I was curious what
changing that would have.  I built a kernel with:

     option NKMEMPAGES=131072

My test program was able to request 536678400 bytes and no more, short
of 512M by 192512 bytes, or 47 pages on my system.

Despite that test, qemu still wouldn't have any of it and refused to
start guests with more than 256M.  I really am at a loss.  Anyone have
any suggestions?

I am back to running GENERIC.MP unmodified on OpenBSD 4.9.

Thanks,

--
David Cantrell <[hidden email]>
WH6DSN | http://blog.burdell.org/

Reply | Threaded
Open this post in threaded view
|

Re: struggling with qemu guests using >256M memory

Ryan Freeman
On Sun, Aug 21, 2011 at 10:09:30PM -0400, David Cantrell wrote:

> I am unable to start a qemu guest with more than 256M of memory,
> despite having 4G in my system.  I get ENOMEM and qemu exits.  Using
> 256M or less works just fine.
>
> After some debugging, I see that qemu uses valloc(), which then uses
> malloc() and that's preventing anything larger than 256M.
>
> Out of curiosity, I wrote a simple test program that used valloc()
> to ask for 256M.  Then I tried 512M, and got the failure I expected.
> Debugging passed NKMEMPAGES in front of my eyes and I was curious
> what changing that would have.  I built a kernel with:
>
>     option NKMEMPAGES=131072
>
> My test program was able to request 536678400 bytes and no more,
> short of 512M by 192512 bytes, or 47 pages on my system.
>
> Despite that test, qemu still wouldn't have any of it and refused to
> start guests with more than 256M.  I really am at a loss.  Anyone
> have any suggestions?
>
> I am back to running GENERIC.MP unmodified on OpenBSD 4.9.

have you tried increasing ulimit -d?

-ryan

>
> Thanks,
>
> --
> David Cantrell <[hidden email]>
> WH6DSN | http://blog.burdell.org/
>

Reply | Threaded
Open this post in threaded view
|

Re: struggling with qemu guests using >256M memory

Todd T. Fries-2
In reply to this post by David Cantrell-3
Try checking login.conf(5), specifically your default limits.  ksh(1)
shows these via 'ulimit -a'.

Penned by David Cantrell on 20110821 21:09.30, we have:
| I am unable to start a qemu guest with more than 256M of memory,
| despite having 4G in my system.  I get ENOMEM and qemu exits.  Using
| 256M or less works just fine.
|
| After some debugging, I see that qemu uses valloc(), which then uses
| malloc() and that's preventing anything larger than 256M.
|
| Out of curiosity, I wrote a simple test program that used valloc() to
| ask for 256M.  Then I tried 512M, and got the failure I expected.
| Debugging passed NKMEMPAGES in front of my eyes and I was curious
| what changing that would have.  I built a kernel with:
|
|     option NKMEMPAGES=131072
|
| My test program was able to request 536678400 bytes and no more,
| short of 512M by 192512 bytes, or 47 pages on my system.
|
| Despite that test, qemu still wouldn't have any of it and refused to
| start guests with more than 256M.  I really am at a loss.  Anyone
| have any suggestions?
|
| I am back to running GENERIC.MP unmodified on OpenBSD 4.9.
|
| Thanks,
|
| --
| David Cantrell <[hidden email]>
| WH6DSN | http://blog.burdell.org/

--
Todd Fries .. [hidden email]

 _____________________________________________
|                                             \  1.636.410.0632 (voice)
| Free Daemon Consulting, LLC                 \  1.405.227.9094 (voice)
| http://FreeDaemonConsulting.com             \  1.866.792.3418 (FAX)
| 2525 NW Expy #525, Oklahoma City, OK 73112  \  sip:[hidden email]
| "..in support of free software solutions."  \  sip:[hidden email]
 \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\
                                                 
              37E7 D3EB 74D0 8D66 A68D  B866 0326 204E 3F42 004A
                        http://todd.fries.net/pgp.txt

Reply | Threaded
Open this post in threaded view
|

Re: struggling with qemu guests using >256M memory

Amit Kulkarni-5
In reply to this post by David Cantrell-3
> I am unable to start a qemu guest with more than 256M of memory, despite
> having 4G in my system.  I get ENOMEM and qemu exits.  Using 256M or less
> works just fine.
>
> After some debugging, I see that qemu uses valloc(), which then uses
> malloc() and that's preventing anything larger than 256M.
>
> Out of curiosity, I wrote a simple test program that used valloc() to ask
> for 256M.  Then I tried 512M, and got the failure I expected. Debugging
> passed NKMEMPAGES in front of my eyes and I was curious what changing that
> would have.  I built a kernel with:
>
>    option NKMEMPAGES=131072
>
> My test program was able to request 536678400 bytes and no more, short of
> 512M by 192512 bytes, or 47 pages on my system.
>
> Despite that test, qemu still wouldn't have any of it and refused to start
> guests with more than 256M.  I really am at a loss.  Anyone have any
> suggestions?
>
> I am back to running GENERIC.MP unmodified on OpenBSD 4.9.

I have tried qemu-13 with more than 1G memory just fine on amd64. So
this is a recent regression. what qemu version?

If you increase pagecache size using malloc.conf, does it allow you to
go higher? (man malloc.conf)

Reply | Threaded
Open this post in threaded view
|

Re: struggling with qemu guests using >256M memory

David Cantrell-3
In reply to this post by Ryan Freeman
On 08/22/11 01:21, Ryan Freeman wrote:

> On Sun, Aug 21, 2011 at 10:09:30PM -0400, David Cantrell wrote:
>> I am unable to start a qemu guest with more than 256M of memory,
>> despite having 4G in my system.  I get ENOMEM and qemu exits.  Using
>> 256M or less works just fine.
>>
>> After some debugging, I see that qemu uses valloc(), which then uses
>> malloc() and that's preventing anything larger than 256M.
>>
>> Out of curiosity, I wrote a simple test program that used valloc()
>> to ask for 256M.  Then I tried 512M, and got the failure I expected.
>> Debugging passed NKMEMPAGES in front of my eyes and I was curious
>> what changing that would have.  I built a kernel with:
>>
>>      option NKMEMPAGES=131072
>>
>> My test program was able to request 536678400 bytes and no more,
>> short of 512M by 192512 bytes, or 47 pages on my system.
>>
>> Despite that test, qemu still wouldn't have any of it and refused to
>> start guests with more than 256M.  I really am at a loss.  Anyone
>> have any suggestions?
>>
>> I am back to running GENERIC.MP unmodified on OpenBSD 4.9.
>
> have you tried increasing ulimit -d?

I did that initially, but it failed for me.  Further investigation today
revealed I should have dug in to that more.  More details in another
followup.

--
David Cantrell <[hidden email]>
WH6DSN | http://blog.burdell.org/

Reply | Threaded
Open this post in threaded view
|

Re: struggling with qemu guests using >256M memory

David Cantrell-3
In reply to this post by Amit Kulkarni-5
On 08/22/11 10:20, Amit Kulkarni wrote:

>> I am unable to start a qemu guest with more than 256M of memory, despite
>> having 4G in my system.  I get ENOMEM and qemu exits.  Using 256M or less
>> works just fine.
>>
>> After some debugging, I see that qemu uses valloc(), which then uses
>> malloc() and that's preventing anything larger than 256M.
>>
>> Out of curiosity, I wrote a simple test program that used valloc() to ask
>> for 256M.  Then I tried 512M, and got the failure I expected. Debugging
>> passed NKMEMPAGES in front of my eyes and I was curious what changing that
>> would have.  I built a kernel with:
>>
>>     option NKMEMPAGES=131072
>>
>> My test program was able to request 536678400 bytes and no more, short of
>> 512M by 192512 bytes, or 47 pages on my system.
>>
>> Despite that test, qemu still wouldn't have any of it and refused to start
>> guests with more than 256M.  I really am at a loss.  Anyone have any
>> suggestions?
>>
>> I am back to running GENERIC.MP unmodified on OpenBSD 4.9.
>
> I have tried qemu-13 with more than 1G memory just fine on amd64. So
> this is a recent regression. what qemu version?

qemu is working fine for me now, though I think there is a problem in
another port worth investigating.  See below.

> If you increase pagecache size using malloc.conf, does it allow you to
> go higher? (man malloc.conf)

As recommended by the first two replies to my original message
suggested, increase the data segment size.  I tried that initially, but
it never worked.  So I started looking elsewhere for a solution.  I
skipped over a key thing I should have eliminated during debugging.

The problem appears to be in bash's ulimit builtin.  My user account is
in the default login class.  I increased the datasize-max value, logged
out, logged back in.  When trying to increase ulimit -d, bash insisted
it couldn't be done.  I ran /bin/ksh and tried ulimit there and it
worked.  I was able to start up a qemu guest with 384M and another with
512M.  Tried under root's account (which I keep /bin/ksh) and it worked
fine.  Go back to my account using bash and ulimit -d won't go up past
the datasize-cur value.

I created a new login class for myself and set datasize=infinity:,
rather than setting both a -cur and -max datasize value.  Logging in
with bash and checking ulimit -d reported unlimited.  qemu fired up just
fine.  So it was the data segment size after all, the problem appears to
be in bash's ulimit builtin.

Might be worth noting for people in qemu's MESSAGE.  I use bash for my
user account because at work we have standardized on bash across all
platforms.  It's just easier for me to have the same shell everywhere.
Workarounds are easy, either make a change in /etc/login.conf similar to
what described or don't use bash and expect ulimit to behave.

When I get some spare time, I want to dig in to bash and see why it's
ulimit builtin is behaving incorrectly.

Thanks,

--
David Cantrell <[hidden email]>
WH6DSN | http://blog.burdell.org/