2013/9/26 Joe Gordon :
>> Yes, when moving beyond simple flavours, the idea as initially proposed
>> falls apart. I see two ways to fix that:
>>
>> * Don't move beyond simple flavours. Seriously. Amazon have been pretty
>>darn succesful with just their simple instance types.
> Who says we hav
On Thu, Sep 26, 2013 at 1:53 PM, Soren Hansen wrote:
> Hey, sorry for necroposting. I completely missed this thread when it was
> active, but Russel just pointed it out to me on Twitter earlier today and
> I couldn't help myself.
>
>
> 2013/7/19 Sandy Walsh :
> > On 07/19/2013 05:01 PM, Boris Pav
Hey, sorry for necroposting. I completely missed this thread when it was
active, but Russel just pointed it out to me on Twitter earlier today and
I couldn't help myself.
2013/7/19 Sandy Walsh :
> On 07/19/2013 05:01 PM, Boris Pavlovic wrote:
> Sorry, I was commenting on Soren's suggestion from w
c.com<mailto:harlo...@yahoo-inc.com>]
Sent: Thursday, July 25, 2013 5:36 AM
To: OpenStack Development Mailing List; Boris Pavlovic
Subject: Re: [openstack-dev] A simple way to improve nova scheduler
As far as the send only when you have to. That reminds me of this piece of work
that could
do you think?
>
> ** **
>
> Thanks.
>
> --
>
> Shane
>
> *From:* Joshua Harlow [mailto:harlo...@yahoo-inc.com]
> *Sent:* Thursday, July 25, 2013 5:36 AM
> *To:* OpenStack Development Mailing List; Boris Pavlovic
>
> *Subject:* Re: [openstack-d
.
--
Shane
From: Joshua Harlow [mailto:harlo...@yahoo-inc.com]
Sent: Thursday, July 25, 2013 5:36 AM
To: OpenStack Development Mailing List; Boris Pavlovic
Subject: Re: [openstack-dev] A simple way to improve nova scheduler
As far as the send only when you have to. That reminds me of this piec
Excerpts from Joe Gordon's message of 2013-07-24 11:43:46 -0700:
> On Wed, Jul 24, 2013 at 12:24 PM, Russell Bryant wrote:
>
> > On 07/23/2013 06:00 PM, Clint Byrum wrote:
> > > This is really interesting work, thanks for sharing it with us. The
> > > discussion that has followed has brought up s
[openstack-dev] A simple way to improve nova scheduler
Hi Mike,
On Wed, Jul 24, 2013 at 1:01 AM, Mike Wilson
mailto:geekinu...@gmail.com>> wrote:
Again I can only speak for qpid, but it's not really a big load on the qpidd
server itself. I think the issue is that the updates come in
Hi Mike,
On Wed, Jul 24, 2013 at 1:01 AM, Mike Wilson wrote:
> Again I can only speak for qpid, but it's not really a big load on the
> qpidd server itself. I think the issue is that the updates come in serially
> into each scheduler that you have running. We don't process those quickly
> enoug
On Wed, Jul 24, 2013 at 12:24 PM, Russell Bryant wrote:
> On 07/23/2013 06:00 PM, Clint Byrum wrote:
> > This is really interesting work, thanks for sharing it with us. The
> > discussion that has followed has brought up some thoughts I've had for
> > a while about this choke point in what is sup
On 07/23/2013 06:00 PM, Clint Byrum wrote:
> This is really interesting work, thanks for sharing it with us. The
> discussion that has followed has brought up some thoughts I've had for
> a while about this choke point in what is supposed to be an extremely
> scalable cloud platform (OpenStack).
>
I like the idea clint.
It appears to me that the kind of scheduler 'buckets' that are being
established allow for different kind of policies around how accurate and
how 'global' the deployer wants scheduling to be (which might be a
differing policies depending on the deployer). All of these kind o
Excerpts from Boris Pavlovic's message of 2013-07-19 07:52:55 -0700:
> Hi all,
>
>
>
> In Mirantis Alexey Ovtchinnikov and me are working on nova scheduler
> improvements.
>
> As far as we can see the problem, now scheduler has two major issues:
>
>
> 1) Scalability. Factors that contribute t
Again I can only speak for qpid, but it's not really a big load on the
qpidd server itself. I think the issue is that the updates come in serially
into each scheduler that you have running. We don't process those quickly
enough for it to do any good, which is why the lookup from db. You can see
thi
Joe,
Sure we will.
Mike,
Thanks for sharing information about scalability problems, presentation was
great.
Also could you say what do you think is 150 req/sec is it big load for qpid
or rabbit? I think it is just nothing..
Best regards,
Boris Pavlovic
---
Mirantis Inc.
On Wed, Jul 24, 2013 a
On Tue, Jul 23, 2013 at 1:09 PM, Boris Pavlovic wrote:
> Ian,
>
> There are serious scalability and performance problems with DB usage in
> current scheduler.
> Rapid Updates + Joins makes current solution absolutely not scalable.
>
> Bleuhost example just shows personally for me just a trivial t
Just some added info for that talk, we are using qpid as our messaging
backend. I have no data for RabbitMQ, but our schedulers are _always_
behind on processing updates. It may be different with rabbit.
-Mike
On Tue, Jul 23, 2013 at 1:56 PM, Joe Gordon wrote:
>
> On Jul 23, 2013 3:44 PM, "Ian
Ian,
There are serious scalability and performance problems with DB usage in
current scheduler.
Rapid Updates + Joins makes current solution absolutely not scalable.
Bleuhost example just shows personally for me just a trivial thing. (It
just won't work)
We will add tomorrow antother graphic:
Av
On Jul 23, 2013 3:44 PM, "Ian Wells" wrote:
>
> > * periodic updates can overwhelm things. Solution: remove unneeded
updates,
> > most scheduling data only changes when an instance does some state
change.
>
> It's not clear that periodic updates do overwhelm things, though.
> Boris ran the tests.
> * periodic updates can overwhelm things. Solution: remove unneeded updates,
> most scheduling data only changes when an instance does some state change.
It's not clear that periodic updates do overwhelm things, though.
Boris ran the tests. Apparently 10k nodes updating once a minute
extend the
penstack-dev@lists.openstack.org>
> Date: Monday, July 22, 2013 3:47 PM
> To: OpenStack Development Mailing List
>
> Subject: Re: [openstack-dev] A simple way to improve nova scheduler
>
>
>
>
> On Mon, Jul 22, 2013 at 5:16 AM, Boris Pavlovic wrote:
>>
>>
ing List
mailto:openstack-dev@lists.openstack.org>>
Date: Monday, July 22, 2013 4:12 PM
To: OpenStack Development Mailing List
mailto:openstack-dev@lists.openstack.org>>,
Joe Gordon mailto:joe.gord...@gmail.com>>
Subject: Re: [openstack-dev] A simple way to improve nova scheduler
An
mail.com>>
Reply-To: OpenStack Development Mailing List
mailto:openstack-dev@lists.openstack.org>>
Date: Monday, July 22, 2013 3:47 PM
To: OpenStack Development Mailing List
mailto:openstack-dev@lists.openstack.org>>
Subject: Re: [openstack-dev] A simple way to improve nova sche
On Mon, Jul 22, 2013 at 5:16 AM, Boris Pavlovic wrote:
> Joe,
>
> >> Speaking of Chris Beherns "Relying on anything but the DB for current
> memory free, etc, is just too laggy… so we need to stick with it, IMO."
> http://lists.openstack.org/pipermail/openstack-dev/2013-June/010485.html
>
> It d
On 07/22/2013 12:51 PM, John Garbutt wrote:
> On 22 July 2013 13:23, Boris Pavlovic wrote:
>> I see only one race condition. (in current solution we have the same
>> situtaiton)
>> Between request to compute node and data is updated in DB, we could use
>> wrong state of compute node.
>> By the way
On 22 July 2013 13:23, Boris Pavlovic wrote:
> I see only one race condition. (in current solution we have the same
> situtaiton)
> Between request to compute node and data is updated in DB, we could use
> wrong state of compute node.
> By the way it is fixed by retry.
This race turns out to be a
controller, as stated in
http://lists.openstack.org/pipermail/openstack-dev/2013-June/010490.html .
-jyh
From: Boris Pavlovic [mailto:bo...@pavlovic.me]
Sent: Monday, July 22, 2013 5:17 AM
To: OpenStack Development Mailing List
Subject: Re: [openstack-dev] A simple way to improve nova scheduler
On 07/22/2013 10:43 AM, Boris Pavlovic wrote:
> Russell,
>
> To get information about "all" compute nodes we should wait one periodic
> task (60 seconds by default).
> So starting will take a while.
>
> But I don't think that this is a big problem:
> 1) if we are already able to wait each time fo
Russell,
To get information about "all" compute nodes we should wait one periodic
task (60 seconds by default).
So starting will take a while.
But I don't think that this is a big problem:
1) if we are already able to wait each time for heavy and long (> few
seconds) db querie
2) if we have more
On 07/22/2013 08:16 AM, Boris Pavlovic wrote:
>>> * How do you bring a new scheduler up in an existing deployment and make it
>>> get the full state of the system?
>
> You should wait for a one periodic task time. And you will get full
> information about all compute nodes.
This also affects up
Sandy,
I see only one race condition. (in current solution we have the same
situtaiton)
Between request to compute node and data is updated in DB, we could use
wrong state of compute node.
By the way it is fixed by retry.
I don't see any new races that are produces by new approach without DB.
Cou
Joe,
>> Speaking of Chris Beherns "Relying on anything but the DB for current
memory free, etc, is just too laggy… so we need to stick with it, IMO."
http://lists.openstack.org/pipermail/openstack-dev/2013-June/010485.html
It doesn't scale, use tons of resources, works slow and is hard to extend
On Fri, Jul 19, 2013 at 3:13 PM, Sandy Walsh wrote:
>
>
> On 07/19/2013 05:36 PM, Boris Pavlovic wrote:
> > Sandy,
> >
> > I don't think that we have such problems here.
> > Because scheduler doesn't pool compute_nodes.
> > The situation is another compute_nodes notify scheduler about their
> > st
On 07/19/2013 05:36 PM, Boris Pavlovic wrote:
> Sandy,
>
> I don't think that we have such problems here.
> Because scheduler doesn't pool compute_nodes.
> The situation is another compute_nodes notify scheduler about their
> state. (instead of updating their state in DB)
>
> So for example if
On 07/19/2013 04:25 PM, Brian Schott wrote:
> I think Soren suggested this way back in Cactus to use MQ for compute
> node state rather than database and it was a good idea then.
The problem with that approach was the number of queues went exponential
as soon as you went beyond simple flavors.
Sandy,
Hm I don't know that algorithm. But our approach doesn't have exponential
exchange.
I don't think that in 10k nodes cloud we will have a problems with 150 RPC
call/sec. Even in 100k we will have only 1.5k RPC call/sec.
More then (compute nodes update their state in DB through conductor whic
Sandy,
I don't think that we have such problems here.
Because scheduler doesn't pool compute_nodes.
The situation is another compute_nodes notify scheduler about their state.
(instead of updating their state in DB)
So for example if scheduler send request to compute_node, compute_node is
able to
On 07/19/2013 05:01 PM, Boris Pavlovic wrote:
> Sandy,
>
> Hm I don't know that algorithm. But our approach doesn't have
> exponential exchange.
> I don't think that in 10k nodes cloud we will have a problems with 150
> RPC call/sec. Even in 100k we will have only 1.5k RPC call/sec.
> More then
I think Soren suggested this way back in Cactus to use MQ for compute node
state rather than database and it was a good idea then.
On Jul 19, 2013, at 10:52 AM, Boris Pavlovic wrote:
> Hi all,
>
>
> In Mirantis Alexey Ovtchinnikov and me are working on nova scheduler
> improvements.
>
> A
39 matches
Mail list logo
