[Openstack] A single cross-zone database?

[Sandy Walsh]

Hi y'all, getting any sleep before Feature Freeze?

As you know, one of the main design tenants of OpenStack is Share Nothing 
(where possible). http://wiki.openstack.org/BasicDesignTenets

That's the mantra we've been chanting with Zones. But it does cause a problem 
with a particular Use Case:

"Show me all Customer X Instances, across all Zones."

This is an expensive request. We have to poll all zones and ask them to return 
a list of matching instances.

There has been some water cooler chat about some things we could do to make 
this more efficient in the near term. One proposal has been to assume a single 
database, replicated across zones. I'll call it SDB for short. With SDB we can 
have a join table that links Zone to Instance ... keeping a record of all 
instances across zones. Maybe it's a completely separate set of tables? Maybe 
it's a separate replicated db? The intention is to let us talk to the 
appropriate zone directly.

Sure, there are a ton more optimizations we could make if we go further with 
SDB. We could store all the Zone capabilities in the db to make Zone selection 
faster. We could store all the customers in the db to make multi-tenant easier. 
But that's not what we're talking about here. We're talking about the bare 
minimum required to make the get_instances query fast.

Conversely, there are issues with a single DB. The largest being the 
implication it has on Bursting (Hybrid Private/Public clouds) ... a pretty 
funky feature imho.

Personally, I think the same query gains can be obtained by creating a separate 
db using off-the-shelf ETL tools to create cache/read-only db's. 
http://en.wikipedia.org/wiki/Extract,_transform,_load

I was considering SDB for Zones (phase 4), but for now, I'm going to stick with 
the original plan of separate databases (1 per zone) and see what the 
performance implications are.

What are your thoughts on this issue?

... let the games begin!

-S




Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace.
Any dissemination, distribution or copying of the enclosed material is 
prohibited.
If you receive this transmission in error, please notify us immediately by 
e-mail
at ab...@rackspace.com, and delete the original message.
Your cooperation is appreciated.

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Justin Santa Barbara]

Thanks for raising this Sandy: +1 on keeping separate DBs until a problem
arises.

I don't see a performance problem with recursively querying child zones.  I
guess this will partially depend on our zone topology: if the intent is to
have child zones that are geographically distributed where the latency
becomes an issue (and it's not clear to me that even this will be a
problem), then we could use a 'caching' approach rather than the 'total
consistency' approach that a database implies.

It does seem like there's a separate issue, which is finding the appropriate
zone for a given instance.  Again, we could easily use caching here, and
simply retry on cache miss.

What sort of complexity of zones hierarchies are we contemplating?  Even at
our full target scales, I don't believe we're talking about more than a
dozen zones (?), and I believe that even the naive implementation which
simply recursively queries child zones with no caching will be more than
good enough.

Justin



On Wed, Mar 16, 2011 at 7:53 AM, Sandy Walsh wrote:

>  Hi y'all, getting any sleep before Feature Freeze?
>
>  As you know, one of the main design tenants of OpenStack is Share Nothing
> (where possible). http://wiki.openstack.org/BasicDesignTenets
>
>  That's the mantra we've been chanting with Zones. But it does cause a
> problem with a particular Use Case:
>
>  *"Show me all Customer X Instances, across all Zones."*
>
>  This is an expensive request. We have to poll all zones and ask them to
> return a list of matching instances.
>
>  There has been some water cooler chat about some things we could do to
> make this more efficient in the near term. One proposal has been to assume a
> single database, replicated across zones. I'll call it SDB for short. With
> SDB we can have a join table that links Zone to Instance ... keeping a
> record of all instances across zones. Maybe it's a completely separate set
> of tables? Maybe it's a separate replicated db? The intention is to let us
> talk to the appropriate zone directly.
>
>  Sure, there are a ton more optimizations we could make if we go further
> with SDB. We could store all the Zone capabilities in the db to make Zone
> selection faster. We could store all the customers in the db to make
> multi-tenant easier. But that's not what we're talking about here. We're
> talking about the* bare minimum *required to make the get_instances query
> fast.
>
>  Conversely, there are issues with a single DB. The largest being the
> implication it has on Bursting (Hybrid Private/Public clouds) ... a pretty
> funky feature imho.
>
>  Personally, I think the same query gains can be obtained by creating a
> separate db using off-the-shelf ETL tools to create cache/read-only db's.
> http://en.wikipedia.org/wiki/Extract,_transform,_load
>
>  I was considering SDB for Zones (phase 4), but for now, I'm going to
> stick with the original plan of separate databases (1 per zone) and see what
> the performance implications are.
>
>  What are your thoughts on this issue?
>
>  ... let the games begin!
>
>  -S
>
>
>  Confidentiality Notice: This e-mail message (including any attached or
> embedded documents) is intended for the exclusive and confidential use of the
> individual or entity to which this message is addressed, and unless otherwise
> expressly indicated, is confidential and privileged information of Rackspace.
> Any dissemination, distribution or copying of the enclosed material is 
> prohibited.
> If you receive this transmission in error, please notify us immediately by 
> e-mail
> at ab...@rackspace.com, and delete the original message.
> Your cooperation is appreciated.
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Paul Voccio]

Sandy,

Not only is this expensive, but there is no way I can see at the moment to do 
pagination, which is what makes this really expensive. If someone asked for an 
entire list of all their instances and it was > 10,000 then I would think 
they're ok with waiting while that response is gathered and returned. However, 
since the API spec says we should be able to do pagination, this is where 
asking each zone for all its children every time gets untenable.

Looking forward to the discussion. More below.

From: Sandy Walsh mailto:sandy.wa...@rackspace.com>>
Date: Wed, 16 Mar 2011 14:53:37 +
To: "openstack@lists.launchpad.net" 
mailto:openstack@lists.launchpad.net>>
Subject: [Openstack] A single cross-zone database?

Hi y'all, getting any sleep before Feature Freeze?

As you know, one of the main design tenants of OpenStack is Share Nothing 
(where possible). http://wiki.openstack.org/BasicDesignTenets

That's the mantra we've been chanting with Zones. But it does cause a problem 
with a particular Use Case:

"Show me all Customer X Instances, across all Zones."

This is an expensive request. We have to poll all zones and ask them to return 
a list of matching instances.

There has been some water cooler chat about some things we could do to make 
this more efficient in the near term. One proposal has been to assume a single 
database, replicated across zones. I'll call it SDB for short. With SDB we can 
have a join table that links Zone to Instance ... keeping a record of all 
instances across zones. Maybe it's a completely separate set of tables? Maybe 
it's a separate replicated db? The intention is to let us talk to the 
appropriate zone directly.

Sure, there are a ton more optimizations we could make if we go further with 
SDB. We could store all the Zone capabilities in the db to make Zone selection 
faster. We could store all the customers in the db to make multi-tenant easier. 
But that's not what we're talking about here. We're talking about the bare 
minimum required to make the get_instances query fast.

Conversely, there are issues with a single DB. The largest being the 
implication it has on Bursting (Hybrid Private/Public clouds) ... a pretty 
funky feature imho.




Personally, I think the same query gains can be obtained by creating a separate 
db using off-the-shelf ETL tools to create cache/read-only db's. 
http://en.wikipedia.org/wiki/Extract,_transform,_load

Isn't the hard part keeping this in sync with what the zones have?


I was considering SDB for Zones (phase 4), but for now, I'm going to stick with 
the original plan of separate databases (1 per zone) and see what the 
performance implications are.

What are your thoughts on this issue?

... let the games begin!

-S



Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace.
Any dissemination, distribution or copying of the enclosed material is 
prohibited.
If you receive this transmission in error, please notify us immediately by 
e-mail
at ab...@rackspace.com, and delete the original 
message.
Your cooperation is appreciated.


___ Mailing list: 
https://launchpad.net/~openstack Post to : 
openstack@lists.launchpad.net Unsubscribe 
: https://launchpad.net/~openstack More help : 
https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Justin Santa Barbara]

Good point that pagination makes this harder.  However, thankfully the limit
is implemented using a token (the last ID seen), not an absolute offset, so
I believe we can still do pagination even in loosely coordinated DBs.  Good
job whoever dodged that bullet (Jorge?)

(Aside #1: Sorting by uptime time - now _that's_ expensive, and has to be
done for every page.)
(Aside #2: What is the point of pagination really?  I mean how many API
users that are getting a list actually stop before fetching them all?)

Justin


On Wed, Mar 16, 2011 at 9:23 AM, Paul Voccio wrote:

>  Sandy,
>
>  Not only is this expensive, but there is no way I can see at the moment
> to do pagination, which is what makes this really expensive. If someone
> asked for an entire list of all their instances and it was > 10,000 then I
> would think they're ok with waiting while that response is gathered and
> returned. However, since the API spec says we should be able to do
> pagination, this is where asking each zone for all its children every time
> gets untenable.
>
>  Looking forward to the discussion. More below.
>
>   From: Sandy Walsh 
> Date: Wed, 16 Mar 2011 14:53:37 +
> To: "openstack@lists.launchpad.net" 
> Subject: [Openstack] A single cross-zone database?
>
>   Hi y'all, getting any sleep before Feature Freeze?
>
>  As you know, one of the main design tenants of OpenStack is Share Nothing
> (where possible). http://wiki.openstack.org/BasicDesignTenets
>
>  That's the mantra we've been chanting with Zones. But it does cause a
> problem with a particular Use Case:
>
>  *"Show me all Customer X Instances, across all Zones."*
>
>  This is an expensive request. We have to poll all zones and ask them to
> return a list of matching instances.
>
>  There has been some water cooler chat about some things we could do to
> make this more efficient in the near term. One proposal has been to assume a
> single database, replicated across zones. I'll call it SDB for short. With
> SDB we can have a join table that links Zone to Instance ... keeping a
> record of all instances across zones. Maybe it's a completely separate set
> of tables? Maybe it's a separate replicated db? The intention is to let us
> talk to the appropriate zone directly.
>
>  Sure, there are a ton more optimizations we could make if we go further
> with SDB. We could store all the Zone capabilities in the db to make Zone
> selection faster. We could store all the customers in the db to make
> multi-tenant easier. But that's not what we're talking about here. We're
> talking about the* bare minimum *required to make the get_instances query
> fast.
>
>  Conversely, there are issues with a single DB. The largest being the
> implication it has on Bursting (Hybrid Private/Public clouds) ... a pretty
> funky feature imho.
>
>
>
>
>
>  Personally, I think the same query gains can be obtained by creating a
> separate db using off-the-shelf ETL tools to create cache/read-only db's.
> http://en.wikipedia.org/wiki/Extract,_transform,_load
>
>
>  Isn't the hard part keeping this in sync with what the zones have?
>
>
>  I was considering SDB for Zones (phase 4), but for now, I'm going to
> stick with the original plan of separate databases (1 per zone) and see what
> the performance implications are.
>
>  What are your thoughts on this issue?
>
>  ... let the games begin!
>
>  -S
>
>
>  Confidentiality Notice: This e-mail message (including any attached or
> embedded documents) is intended for the exclusive and confidential use of the
> individual or entity to which this message is addressed, and unless otherwise
> expressly indicated, is confidential and privileged information of Rackspace.
> Any dissemination, distribution or copying of the enclosed material is 
> prohibited.
> If you receive this transmission in error, please notify us immediately by 
> e-mail
> at ab...@rackspace.com, and delete the original message.
> Your cooperation is appreciated.
>
>  ___ Mailing list:
> https://launchpad.net/~openstack Post to : 
> openstack@lists.launchpad.netUnsubscribe :
> https://launchpad.net/~openstack More help :
> https://help.launchpad.net/ListHelp
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Ed Leafe]

On Mar 16, 2011, at 12:23 PM, Paul Voccio wrote:

> Not only is this expensive, but there is no way I can see at the moment to do 
> pagination, which is what makes this really expensive. If someone asked for 
> an entire list of all their instances and it was > 10,000 then I would think 
> they're ok with waiting while that response is gathered and returned. 
> However, since the API spec says we should be able to do pagination, this is 
> where asking each zone for all its children every time gets untenable.

This gets us into the caching issues that were discussed at the last 
summit. We could run the query and then cache the results at the endpoint, but 
this would require accepting some level of staleness of the results. The cache 
would handle the paging, and some sort of TTL would have to be established as a 
balance between performance and staleness.



-- Ed Leafe


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Paul Voccio]

Ed,

I would agree. The caching would go with the design tenet #7: Accept
eventual consistency and use it where it is appropriate.

If we're ok with accepting that the list may or may not always be up to
date and feel its appropriate, we should be good with the caching.

pvo


On 3/16/11 11:45 AM, "Ed Leafe"  wrote:

>On Mar 16, 2011, at 12:23 PM, Paul Voccio wrote:
>
>> Not only is this expensive, but there is no way I can see at the moment
>>to do pagination, which is what makes this really expensive. If someone
>>asked for an entire list of all their instances and it was > 10,000 then
>>I would think they're ok with waiting while that response is gathered
>>and returned. However, since the API spec says we should be able to do
>>pagination, this is where asking each zone for all its children every
>>time gets untenable.
>
>This gets us into the caching issues that were discussed at the last
>summit. We could run the query and then cache the results at the
>endpoint, but this would require accepting some level of staleness of the
>results. The cache would handle the paging, and some sort of TTL would
>have to be established as a balance between performance and staleness.
>
>
>
>-- Ed Leafe
>


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Andrew Shafer]

Global temporal consistency is a myth.

If you decide not to cache and support pagination then querying every zone
for every page is potentially as misleading as caching because what should
be on each page could change for every request.

+1 for cache with ttl


On Wed, Mar 16, 2011 at 11:58 AM, Paul Voccio wrote:

> Ed,
>
> I would agree. The caching would go with the design tenet #7: Accept
> eventual consistency and use it where it is appropriate.
>
> If we're ok with accepting that the list may or may not always be up to
> date and feel its appropriate, we should be good with the caching.
>
> pvo
>
>
> On 3/16/11 11:45 AM, "Ed Leafe"  wrote:
>
> >On Mar 16, 2011, at 12:23 PM, Paul Voccio wrote:
> >
> >> Not only is this expensive, but there is no way I can see at the moment
> >>to do pagination, which is what makes this really expensive. If someone
> >>asked for an entire list of all their instances and it was > 10,000 then
> >>I would think they're ok with waiting while that response is gathered
> >>and returned. However, since the API spec says we should be able to do
> >>pagination, this is where asking each zone for all its children every
> >>time gets untenable.
> >
> >This gets us into the caching issues that were discussed at the last
> >summit. We could run the query and then cache the results at the
> >endpoint, but this would require accepting some level of staleness of the
> >results. The cache would handle the paging, and some sort of TTL would
> >have to be established as a balance between performance and staleness.
> >
> >
> >
> >-- Ed Leafe
> >
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Justin Santa Barbara]

Can someone explain _why_ we need caching?

With our approach to pagination, without caching, the answer is always
correct: each query always returns the next {limit} values whose ID is >=
{start-id}.

I agree that in practice this means that there's no way to guarantee you get
all values while they're changing behind the scenes, but this is a
shortcoming of pagination, not caching.  Caching doesn't solve this, it just
creates thornier edge cases.  The solution here is a more sensible ordering
than 'last modified', and I question the value of pagination (other than
for compatibility)

Justin

---

Justin Santa Barbara
Founder, FathomDB





On Wed, Mar 16, 2011 at 11:14 AM, Andrew Shafer wrote:

>
> Global temporal consistency is a myth.
>
> If you decide not to cache and support pagination then querying every zone
> for every page is potentially as misleading as caching because what should
> be on each page could change for every request.
>
> +1 for cache with ttl
>
>
> On Wed, Mar 16, 2011 at 11:58 AM, Paul Voccio 
> wrote:
>
>> Ed,
>>
>> I would agree. The caching would go with the design tenet #7: Accept
>> eventual consistency and use it where it is appropriate.
>>
>> If we're ok with accepting that the list may or may not always be up to
>> date and feel its appropriate, we should be good with the caching.
>>
>> pvo
>>
>>
>> On 3/16/11 11:45 AM, "Ed Leafe"  wrote:
>>
>> >On Mar 16, 2011, at 12:23 PM, Paul Voccio wrote:
>> >
>> >> Not only is this expensive, but there is no way I can see at the moment
>> >>to do pagination, which is what makes this really expensive. If someone
>> >>asked for an entire list of all their instances and it was > 10,000 then
>> >>I would think they're ok with waiting while that response is gathered
>> >>and returned. However, since the API spec says we should be able to do
>> >>pagination, this is where asking each zone for all its children every
>> >>time gets untenable.
>> >
>> >This gets us into the caching issues that were discussed at the last
>> >summit. We could run the query and then cache the results at the
>> >endpoint, but this would require accepting some level of staleness of the
>> >results. The cache would handle the paging, and some sort of TTL would
>> >have to be established as a balance between performance and staleness.
>> >
>> >
>> >
>> >-- Ed Leafe
>> >
>>
>>
>> ___
>> Mailing list: https://launchpad.net/~openstack
>> Post to : openstack@lists.launchpad.net
>> Unsubscribe : https://launchpad.net/~openstack
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Glen Campbell]

Instead of building data-specific caching (which always worries me), you
could simply build the service to return the data directly, then add a
"Cach-Control: max-age=NNN" header to the result. That way, users who
wanted to improve their performance could add a squid layer (or other
caching HTTP proxy) to their endpoints and ensure consistency within the
cache max-age range (for example, max-age=300 would ensure consistency as
of 5 minutes ago). If the user prefers consistency over performance,
simply bypass the cache.


In that manner, the "consistency vs. performance" decision is left up to
the cloud administrator.

On 3/16/11 12:58 PM, "Paul Voccio"  wrote:

>Ed,
>
>I would agree. The caching would go with the design tenet #7: Accept
>eventual consistency and use it where it is appropriate.
>
>If we're ok with accepting that the list may or may not always be up to
>date and feel its appropriate, we should be good with the caching.
>
>pvo
>
>
>On 3/16/11 11:45 AM, "Ed Leafe"  wrote:
>
>>On Mar 16, 2011, at 12:23 PM, Paul Voccio wrote:
>>
>>> Not only is this expensive, but there is no way I can see at the moment
>>>to do pagination, which is what makes this really expensive. If someone
>>>asked for an entire list of all their instances and it was > 10,000 then
>>>I would think they're ok with waiting while that response is gathered
>>>and returned. However, since the API spec says we should be able to do
>>>pagination, this is where asking each zone for all its children every
>>>time gets untenable.
>>
>>This gets us into the caching issues that were discussed at the last
>>summit. We could run the query and then cache the results at the
>>endpoint, but this would require accepting some level of staleness of the
>>results. The cache would handle the paging, and some sort of TTL would
>>have to be established as a balance between performance and staleness.
>>
>>
>>
>>-- Ed Leafe
>>
>
>
>___
>Mailing list: https://launchpad.net/~openstack
>Post to : openstack@lists.launchpad.net
>Unsubscribe : https://launchpad.net/~openstack
>More help   : https://help.launchpad.net/ListHelp



Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace.
Any dissemination, distribution or copying of the enclosed material is 
prohibited.
If you receive this transmission in error, please notify us immediately by 
e-mail
at ab...@rackspace.com, and delete the original message.
Your cooperation is appreciated.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Eric Day]

We can handle pagination whether we have a single database, multiple
databases with cache, or query each zone on each request. In the last
case an instance would be identified with the zone it exists in (for
example, the marker would be a fully qualified zone:instance name)
and we can just pick up where we left off, using a deterministic order
of zones/instances for all API frontends. I don't think we need this,
we need an active cache with one db per zone (same thing I've been
saying since the Austin summit).

I have a number of issues with using a central DB for this application,
but I'll save my usual rant and focus on a main issue you already
mentioned: hybrid clouds. If someone stands up a large public cloud,
lets say dozens of zones, and customers are allowed to connect their
private cloud to their account (possibly thousands of zones), do folks
expect to use a central db? If so, please explain in detail with how
this will work focusing on scalability and security.

I propose we stick with the original proposal of each zone having
it's own DB and ability for active caching for zones that need it
(aggregate zones). We should be doing active caching so we don't have
staleness issues that Ed mentions. All records should be timestamped
(and indexed) so parent zones can efficiently ask for "all updates
since X" if they need to resync. Child zones will push updates to any
subscribed parent zones which can keep a list that should hardly ever
be out of sync (for listing/pagination/etc.). We should batch updates
between each zone level to ensure efficient data flow.

-Eric

On Wed, Mar 16, 2011 at 04:45:46PM +, Ed Leafe wrote:
> On Mar 16, 2011, at 12:23 PM, Paul Voccio wrote:
> 
> > Not only is this expensive, but there is no way I can see at the moment to 
> > do pagination, which is what makes this really expensive. If someone asked 
> > for an entire list of all their instances and it was > 10,000 then I would 
> > think they're ok with waiting while that response is gathered and returned. 
> > However, since the API spec says we should be able to do pagination, this 
> > is where asking each zone for all its children every time gets untenable.
> 
>   This gets us into the caching issues that were discussed at the last 
> summit. We could run the query and then cache the results at the endpoint, 
> but this would require accepting some level of staleness of the results. The 
> cache would handle the paging, and some sort of TTL would have to be 
> established as a balance between performance and staleness.
> 
> 
> 
> -- Ed Leafe
> 
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Eric Day]

Justin,

You don't need a cache, but large installations will probably want it
for the usual reasons (improve response time, reduce load, etc). Since
this is one of our primary use cases, we need to do it. Also, to do
caching correctly, it's not just something you tack on. Efficient
caching systems that yield the best user experiences usually need to
be a bit more integrated.

-Eric

On Wed, Mar 16, 2011 at 11:24:08AM -0700, Justin Santa Barbara wrote:
>Can someone explain _why_ we need caching?
>With our approach to pagination, without caching, the answer is always
>correct: each query always returns the next {limit} values whose ID is >=
>{start-id}.
>I agree that in practice this means that there's no way to guarantee you
>get all values while they're changing behind the scenes, but this is a
>shortcoming of pagination, not caching.  Caching doesn't solve this, it
>just creates thornier edge cases.  The solution here is a more sensible
>ordering than 'last modified', and I question the value of pagination
>(other than for compatibility)
>Justin
> 
>---
> 
>Justin Santa Barbara
>Founder, FathomDB
> 
>On Wed, Mar 16, 2011 at 11:14 AM, Andrew Shafer 
>wrote:
> 
>  Global temporal consistency is a myth.
>  If you decide not to cache and support pagination then querying every
>  zone for every page is potentially as misleading as caching because what
>  should be on each page could change for every request.
>  +1 for cache with ttl
>  On Wed, Mar 16, 2011 at 11:58 AM, Paul Voccio
>   wrote:
> 
>Ed,
> 
>I would agree. The caching would go with the design tenet #7: Accept
>eventual consistency and use it where it is appropriate.
> 
>If we're ok with accepting that the list may or may not always be up
>to
>date and feel its appropriate, we should be good with the caching.
> 
>pvo
> 
>On 3/16/11 11:45 AM, "Ed Leafe"  wrote:
> 
>>On Mar 16, 2011, at 12:23 PM, Paul Voccio wrote:
>>
>>> Not only is this expensive, but there is no way I can see at the
>moment
>>>to do pagination, which is what makes this really expensive. If
>someone
>>>asked for an entire list of all their instances and it was > 10,000
>then
>>>I would think they're ok with waiting while that response is
>gathered
>>>and returned. However, since the API spec says we should be able to
>do
>>>pagination, this is where asking each zone for all its children
>every
>>>time gets untenable.
>>
>>This gets us into the caching issues that were discussed at the
>last
>>summit. We could run the query and then cache the results at the
>>endpoint, but this would require accepting some level of staleness of
>the
>>results. The cache would handle the paging, and some sort of TTL
>would
>>have to be established as a balance between performance and
>staleness.
>>
>>
>>
>>-- Ed Leafe
>>
> 
>___
>Mailing list: https://launchpad.net/~openstack
>Post to : openstack@lists.launchpad.net
>Unsubscribe : https://launchpad.net/~openstack
>More help   : https://help.launchpad.net/ListHelp
> 
>  ___
>  Mailing list: https://launchpad.net/~openstack
>  Post to : openstack@lists.launchpad.net
>  Unsubscribe : https://launchpad.net/~openstack
>  More help   : https://help.launchpad.net/ListHelp

> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Ed Leafe]

On Mar 16, 2011, at 2:24 PM, Justin Santa Barbara wrote:

> Can someone explain _why_ we need caching?

We don't *need* caching - it is simply the most direct way to avoid 
multiple expensive calls.

> With our approach to pagination, without caching, the answer is always 
> correct: each query always returns the next {limit} values whose ID is >= 
> {start-id}.

But for this example, you have to traverse *all* the zones in order to 
get the list of instances for a customer. How else would you define "instances 
101-199 of 823 total instances"? How would you know where #101 is?

> I agree that in practice this means that there's no way to guarantee you get 
> all values while they're changing behind the scenes, but this is a 
> shortcoming of pagination, not caching.  Caching doesn't solve this, it just 
> creates thornier edge cases.  The solution here is a more sensible ordering 
> than 'last modified', and I question the value of pagination (other than for 
> compatibility)

The question is performance in this particular use case. As pvo said, 
you can accept a certain level of inconsistency as a trade-off for scaling. We 
would then allow the TTL to be controlled so that different OpenStack 
deployments can adjust this to their needs.



-- Ed Leafe



Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace.
Any dissemination, distribution or copying of the enclosed material is 
prohibited.
If you receive this transmission in error, please notify us immediately by 
e-mail
at ab...@rackspace.com, and delete the original message.
Your cooperation is appreciated.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Eric Day]

It's always a trade-off with generic vs context-specific caching. You
can usually be more efficient and provide a better UX with
context-specific caching, but it can be a bit more work.

For Nova (and any distributed application along these lines) I think
we need to use context-specific active caching. Punting on it doesn't
give the large deployments many options to scale, you are forced into
generic caching and the UX may always suffer from it. By providing
context-specific active caching hooks we can greatly increase our
usability and efficiency.

-Eric

On Wed, Mar 16, 2011 at 06:33:35PM +, Glen Campbell wrote:
> Instead of building data-specific caching (which always worries me), you
> could simply build the service to return the data directly, then add a
> "Cach-Control: max-age=NNN" header to the result. That way, users who
> wanted to improve their performance could add a squid layer (or other
> caching HTTP proxy) to their endpoints and ensure consistency within the
> cache max-age range (for example, max-age=300 would ensure consistency as
> of 5 minutes ago). If the user prefers consistency over performance,
> simply bypass the cache.
> 
> 
> In that manner, the "consistency vs. performance" decision is left up to
> the cloud administrator.
> 
> On 3/16/11 12:58 PM, "Paul Voccio"  wrote:
> 
> >Ed,
> >
> >I would agree. The caching would go with the design tenet #7: Accept
> >eventual consistency and use it where it is appropriate.
> >
> >If we're ok with accepting that the list may or may not always be up to
> >date and feel its appropriate, we should be good with the caching.
> >
> >pvo
> >
> >
> >On 3/16/11 11:45 AM, "Ed Leafe"  wrote:
> >
> >>On Mar 16, 2011, at 12:23 PM, Paul Voccio wrote:
> >>
> >>> Not only is this expensive, but there is no way I can see at the moment
> >>>to do pagination, which is what makes this really expensive. If someone
> >>>asked for an entire list of all their instances and it was > 10,000 then
> >>>I would think they're ok with waiting while that response is gathered
> >>>and returned. However, since the API spec says we should be able to do
> >>>pagination, this is where asking each zone for all its children every
> >>>time gets untenable.
> >>
> >>This gets us into the caching issues that were discussed at the last
> >>summit. We could run the query and then cache the results at the
> >>endpoint, but this would require accepting some level of staleness of the
> >>results. The cache would handle the paging, and some sort of TTL would
> >>have to be established as a balance between performance and staleness.
> >>
> >>
> >>
> >>-- Ed Leafe
> >>
> >
> >
> >___
> >Mailing list: https://launchpad.net/~openstack
> >Post to : openstack@lists.launchpad.net
> >Unsubscribe : https://launchpad.net/~openstack
> >More help   : https://help.launchpad.net/ListHelp
> 
> 
> 
> Confidentiality Notice: This e-mail message (including any attached or
> embedded documents) is intended for the exclusive and confidential use of the
> individual or entity to which this message is addressed, and unless otherwise
> expressly indicated, is confidential and privileged information of Rackspace.
> Any dissemination, distribution or copying of the enclosed material is 
> prohibited.
> If you receive this transmission in error, please notify us immediately by 
> e-mail
> at ab...@rackspace.com, and delete the original message.
> Your cooperation is appreciated.
> 
> 
> ___
> Mailing list: https://launchpad.net/~openstack
> Post to : openstack@lists.launchpad.net
> Unsubscribe : https://launchpad.net/~openstack
> More help   : https://help.launchpad.net/ListHelp

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Justin Santa Barbara]

Inline...

> Can someone explain _why_ we need caching?
>
> We don't *need* caching - it is simply the most direct way to avoid
> multiple expensive calls.


So if we don't need it...?  You cite avoiding expensive calls, but I think
it's entirely unproven that those call are too expensive.

If it makes you happier, think of the child zones as a cache that just
happens to not have any inconsistency issues.  I see Eric's point that it
would be better to integrate it up-front, but let's get ten zones working
first, before we worry out how we can scale to ten thousand zones.

I think this should be Sandy's call.  My interpretation is that he'd like to
implement a correct algorithm first, to understand where the bottlenecks
are.  I think that's the right way.  Equally, if Sandy says he wants to do
caching, then that's fine with me also.



> > With our approach to pagination, without caching, the answer is always
> correct: each query always returns the next {limit} values whose ID is >=
> {start-id}.
>
> But for this example, you have to traverse *all* the zones in order
> to get the list of instances for a customer. How else would you define
> "instances 101-199 of 823 total instances"? How would you know where #101
> is?
>

You do need to traverse all zones, correct.  (Though note that you don't
need to know where #101 is)  I think you may have missed the point about
token-based pagination: the query is not "instances at indexes 101-199 of
823", it is e.g. "the next 100 instances whose id > 'i-198273'.

But how is this a problem?  How many zones are we talking about?  If this is
bigger than a few dozen, why?


>
> > I agree that in practice this means that there's no way to guarantee you
> get all values while they're changing behind the scenes, but this is a
> shortcoming of pagination, not caching.  Caching doesn't solve this, it just
> creates thornier edge cases.  The solution here is a more sensible ordering
> than 'last modified', and I question the value of pagination (other than for
> compatibility)
>
> The question is performance in this particular use case. As pvo
> said, you can accept a certain level of inconsistency as a trade-off for
> scaling. We would then allow the TTL to be controlled so that different
> OpenStack deployments can adjust this to their needs.
>

Except every binding to the CloudServers API has a hacky way to bypass
caching.  We have direct evidence that people don't want to accept these
caching trade-offs / edge cases.

It sounds to me like you're saying: "But caching is web-scale" :-)


Justin
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Eric Day]

On Wed, Mar 16, 2011 at 06:47:03PM +, Ed Leafe wrote:
> > With our approach to pagination, without caching, the answer is always 
> > correct: each query always returns the next {limit} values whose ID is >= 
> > {start-id}.
> 
>   But for this example, you have to traverse *all* the zones in order to 
> get the list of instances for a customer. How else would you define 
> "instances 101-199 of 823 total instances"? How would you know where #101 is?

The marker should be the canonical zone:instance, so you just query
that zone with instance marker and limit, and move on to the next
zone(s) if your limit is not reached. You never need to build the
full list.

-Eric

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

[Openstack] Ubuntu Cloud days, call for session

[Ahmed Kamal]

Hi everyone,

From March 23rd 2011 to March 24th 2011 Ubuntu is hosting the very 
first Ubuntu Cloud Days. This is an event of IRC tutorials and sessions. 
It would be awesome if someone would volunteer to hold an openstack irc 
session. Sessions are usually not too hard to prepare, if you're 
considering it contact me and I'll try to help in any way possible


You can find the preliminary schedule at
https://wiki.ubuntu.com/UbuntuCloudDays

If you are considering to give a session, please get in touch with me or 
directly edit the timeTable then ping me

https://wiki.ubuntu.com/UbuntuCloudDays/Timetable

Thanks!

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Justin Santa Barbara]

I agree that we could have a better marker, but I'm just going off the spec
at the moment.

I've checked the agreed blueprint, and caching in zones is out of scope for
Cactus.

Please propose a discussion topic for the Design Summit.

Justin





On Wed, Mar 16, 2011 at 12:21 PM, Eric Day  wrote:

> On Wed, Mar 16, 2011 at 06:47:03PM +, Ed Leafe wrote:
> > > With our approach to pagination, without caching, the answer is always
> correct: each query always returns the next {limit} values whose ID is >=
> {start-id}.
> >
> >   But for this example, you have to traverse *all* the zones in order
> to get the list of instances for a customer. How else would you define
> "instances 101-199 of 823 total instances"? How would you know where #101
> is?
>
> The marker should be the canonical zone:instance, so you just query
> that zone with instance marker and limit, and move on to the next
> zone(s) if your limit is not reached. You never need to build the
> full list.
>
> -Eric
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Ed Leafe]

On Mar 16, 2011, at 3:39 PM, Justin Santa Barbara wrote:

> I agree that we could have a better marker, but I'm just going off the spec 
> at the moment.
> 
> I've checked the agreed blueprint, and caching in zones is out of scope for 
> Cactus.
> 
> Please propose a discussion topic for the Design Summit.

Can we get back to the original topic? The only reason caching came up 
was as an alternative to a single DB to hold all instance information. That was 
an implementation solution suggested for multi-cluster/zones, so it is 
definitely in scope for Cactus.



-- Ed Leafe



Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace. 
Any dissemination, distribution or copying of the enclosed material is 
prohibited.
If you receive this transmission in error, please notify us immediately by 
e-mail
at ab...@rackspace.com, and delete the original message. 
Your cooperation is appreciated.


___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Justin Santa Barbara]

Seems that the person writing the code (Sandy) wants _not_ to do a single DB
initially.  It sounds like there are back channels where a single DB is
being pushed on Sandy.

To me, it sounds like we have these choices:

   1. We can have a zones implementation in Cactus.  As specified in the
   blueprint, it will use recursive querying, and there will be no caching
   initially, nor will there be a single DB.
   2. We can go 'off blueprint', and simply not have a multi-zones
   implementation in Cactus.


Given that, I don't see why we would deviate from what we've agreed (and I'm
normally all for flexibility); let's get a baseline implementation into
Cactus.  People that want to add caching or a single DB are then free to do
so in their own branches, but at least those enhancements will be starting
from a common base.  I'm not against adding caching / a single DB if it
proves necessary / good later.

Hopefully we'll actually learn of any real-world issues with the simple
approach by running Sandy's code, and we can discuss those facts at the
design conference, rather than talking in hypotheticals.

Sandy: Have I got the wrong end of the stick here?  Are these our choices?

Justin



On Wed, Mar 16, 2011 at 1:13 PM, Ed Leafe  wrote:

> On Mar 16, 2011, at 3:39 PM, Justin Santa Barbara wrote:
>
> > I agree that we could have a better marker, but I'm just going off the
> spec at the moment.
> >
> > I've checked the agreed blueprint, and caching in zones is out of scope
> for Cactus.
> >
> > Please propose a discussion topic for the Design Summit.
>
> Can we get back to the original topic? The only reason caching came
> up was as an alternative to a single DB to hold all instance information.
> That was an implementation solution suggested for multi-cluster/zones, so it
> is definitely in scope for Cactus.
>
>
>
> -- Ed Leafe
>
>
>
> Confidentiality Notice: This e-mail message (including any attached or
> embedded documents) is intended for the exclusive and confidential use of
> the
> individual or entity to which this message is addressed, and unless
> otherwise
> expressly indicated, is confidential and privileged information of
> Rackspace.
> Any dissemination, distribution or copying of the enclosed material is
> prohibited.
> If you receive this transmission in error, please notify us immediately by
> e-mail
> at ab...@rackspace.com, and delete the original message.
> Your cooperation is appreciated.
>
>
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Paul Voccio]

For Cactus, I'm with Justin and Sandy on #1 to get something working.

Justin — you said earlier that you're not sure this is going to be a problem. 
From experience, this is a problem with trying to query all the instances 
across zones for Rackspace now. Sandy and others including myself have talked 
though this a few times to try to solve this early, but I don't think we can. 
I, like you, want to get something working with zones asap. I'd rather retreat 
from trying to solve this now to get something working. We'll obviously tackle 
the list instances once something works.

From: Justin Santa Barbara mailto:jus...@fathomdb.com>>
Date: Wed, 16 Mar 2011 13:37:50 -0700
To: Ed Leafe mailto:ed.le...@rackspace.com>>
Cc: "openstack@lists.launchpad.net" 
mailto:openstack@lists.launchpad.net>>
Subject: Re: [Openstack] A single cross-zone database?

Seems that the person writing the code (Sandy) wants _not_ to do a single DB 
initially.  It sounds like there are back channels where a single DB is being 
pushed on Sandy.

To me, it sounds like we have these choices:

  1.  We can have a zones implementation in Cactus.  As specified in the 
blueprint, it will use recursive querying, and there will be no caching 
initially, nor will there be a single DB.
  2.  We can go 'off blueprint', and simply not have a multi-zones 
implementation in Cactus.

Given that, I don't see why we would deviate from what we've agreed (and I'm 
normally all for flexibility); let's get a baseline implementation into Cactus. 
 People that want to add caching or a single DB are then free to do so in their 
own branches, but at least those enhancements will be starting from a common 
base.  I'm not against adding caching / a single DB if it proves necessary / 
good later.

Hopefully we'll actually learn of any real-world issues with the simple 
approach by running Sandy's code, and we can discuss those facts at the design 
conference, rather than talking in hypotheticals.

Sandy: Have I got the wrong end of the stick here?  Are these our choices?

Justin



On Wed, Mar 16, 2011 at 1:13 PM, Ed Leafe 
mailto:ed.le...@rackspace.com>> wrote:
On Mar 16, 2011, at 3:39 PM, Justin Santa Barbara wrote:

> I agree that we could have a better marker, but I'm just going off the spec 
> at the moment.
>
> I've checked the agreed blueprint, and caching in zones is out of scope for 
> Cactus.
>
> Please propose a discussion topic for the Design Summit.

   Can we get back to the original topic? The only reason caching came up 
was as an alternative to a single DB to hold all instance information. That was 
an implementation solution suggested for multi-cluster/zones, so it is 
definitely in scope for Cactus.



-- Ed Leafe



Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace.
Any dissemination, distribution or copying of the enclosed material is 
prohibited.
If you receive this transmission in error, please notify us immediately by 
e-mail
at ab...@rackspace.com, and delete the original 
message.
Your cooperation is appreciated.


___ Mailing list: 
https://launchpad.net/~openstack Post to : 
openstack@lists.launchpad.net Unsubscribe 
: https://launchpad.net/~openstack More help : 
https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp

Re: [Openstack] A single cross-zone database?

[Sandy Walsh]

Yup, that's a fair assessment.

That said, even without SDB and caching it's going to be tight for Cactus 
anyway. There are lots of little issues that are cropping up once I got down to 
100'.

-S


From: Justin Santa Barbara mailto:jus...@fathomdb.com>>

Sandy: Have I got the wrong end of the stick here?  Are these our choices?

Justin



Confidentiality Notice: This e-mail message (including any attached or
embedded documents) is intended for the exclusive and confidential use of the
individual or entity to which this message is addressed, and unless otherwise
expressly indicated, is confidential and privileged information of Rackspace.
Any dissemination, distribution or copying of the enclosed material is 
prohibited.
If you receive this transmission in error, please notify us immediately by 
e-mail
at ab...@rackspace.com, and delete the original message.
Your cooperation is appreciated.

___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help   : https://help.launchpad.net/ListHelp