On Fri, May 27, 2011 at 10:45 AM, Thorsten von Eicken
wrote:
> On 5/27/2011 6:05 AM, Jay Pipes wrote:
>>
>> But, it sounds like folks aren't really concerned about the
>> consistency of the view as much as the scalability concerns, and I'm
>> perfectly cool with ditching OFFSET in favour of a last
On 5/27/2011 6:05 AM, Jay Pipes wrote:
>
> But, it sounds like folks aren't really concerned about the
> consistency of the view as much as the scalability concerns, and I'm
> perfectly cool with ditching OFFSET in favour of a last-record marker.
>
THere's consistency and consistency. We need some
On Thu, May 26, 2011 at 9:56 AM, Michael Barton
wrote:
> On Wed, May 25, 2011 at 2:40 PM, Jay Pipes wrote:
>
>> The pagination in Swift is not consistent. Inserts into the Swift
>> databases in between the time of the initial query and the requesting
>> the "next page" can result in rows from the
On Wed, May 25, 2011 at 2:40 PM, Jay Pipes wrote:
> The pagination in Swift is not consistent. Inserts into the Swift
> databases in between the time of the initial query and the requesting
> the "next page" can result in rows from the original first page
> getting on the new second page.
No, yo
Hi Jay,
On Wed, May 25, 2011 at 03:40:27PM -0400, Jay Pipes wrote:
> On Wed, May 25, 2011 at 3:32 PM, Greg Holt wrote:
> > select w from x where y > marker order by y limit z
> >
> > This gives you consistent pagination, means the database doesn't have to
> > count matching rows to find an offse
On Wed, 2011-05-25 at 16:27 -0400, William Wolf wrote:
> I personally like Jay's proposal (except maybe keeping 'pages' out for
> now in favor of just having 1 way to do things, rather than many ways
> to do the same thing), but feel that the term 'marker' should maybe be
> renamed. Maybe 'timesta
er/limit but still move inefficient logic out of python to db
layer, and have glance support marker/limit as well.
Thoughts on these two paths of moving forward? Anyone have ideas for other
routes we could take?
-Original Message-
From: "Jay Pipes"
Sent: Wednesday, May 25,
On Wed, May 25, 2011 at 3:43 PM, Greg Holt wrote:
> Okay, I give up then. Not sure what's different with what you have vs. Swift
> dbs. Just trying to offer up what we do and have been doing for a while now.
The pagination in Swift is not consistent. Inserts into the Swift
databases in between t
Okay, I give up then. Not sure what's different with what you have vs. Swift
dbs. Just trying to offer up what we do and have been doing for a while now.
On May 25, 2011, at 2:40 PM, Jay Pipes wrote:
> On Wed, May 25, 2011 at 3:32 PM, Greg Holt wrote:
>> select w from x where y > marker order b
On Wed, May 25, 2011 at 3:32 PM, Greg Holt wrote:
> select w from x where y > marker order by y limit z
>
> This gives you consistent pagination, means the database doesn't have to
> count matching rows to find an offset, and means you could shard by y later
> in life if you need to scale that m
select w from x where y > marker order by y limit z
This gives you consistent pagination, means the database doesn't have to count
matching rows to find an offset, and means you could shard by y later in life
if you need to scale that much.
On May 25, 2011, at 2:28 PM, Jay Pipes wrote:
> On W
On Wed, May 25, 2011 at 3:10 PM, Greg Holt wrote:
> Okay cool. Just stop using the term offset anywhere in your examples then and
> drop the whole page= thing as well. :)
Sorry, I'm not understanding what you're getting at here. The offset
is required to pass to the database query as the OFFSET
Okay cool. Just stop using the term offset anywhere in your examples then and
drop the whole page= thing as well. :)
On May 25, 2011, at 1:28 PM, Jay Pipes wrote:
> I'm not proposing using marker/limit over offset/limit. I'm proposing
> using marker/limit/offset over searching manually through a
I'm not proposing using marker/limit over offset/limit. I'm proposing
using marker/limit/offset over searching manually through a giant set
of results looking for a marker and then grabbing the next few
results. ;)
On Wed, May 25, 2011 at 2:26 PM, Greg Holt wrote:
> Also, using marker/limit rathe
Also, using marker/limit rather than offset/limit makes it much easier to shard
for scale if desired. One of the reasons we chose that route with Swift.
Regular databases with indexes work better with marker/limit than offset/limit
too.
On May 25, 2011, at 12:38 PM, Justin Santa Barbara wrote:
This is what happens when I braindump a bunch of stuff into an
email... I get myself confused :)
I'm proposing the following. Please pick the strategy apart:
1) Pass the LIMIT and OFFSET parameters down into the database API queries
2) Add a default ORDER BY for all queries returning result sets
On Wed, May 25, 2011 at 1:38 PM, Justin Santa Barbara
wrote:
> The SQL query using the PK would look like this (and there's no need for an
> OFFSET clause):
> SELECT *
> FROM
> ORDER BY
> WHERE > @marker
> AND ()
> LIMIT
> Without additional filters, that's a _very_ efficient query for a sensi
I misspoke in my original email.
Marker should be the timestamp of the initial query.
-jay
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https
I definitely agree that the paging logic is questionable; we definitely
should be specifying the sort order if we're relying on it, as I believe a
database is otherwise free to return results however it sees fit.
I'm not convinced that the proposed logic around the queries and update
timestamps is
When a user issues a GET request for a resource collection, it's clear
that you don't want to return ALL results in the collection. Returning
hundreds of thousands of records per request isn't particularly a good
idea.
Therefore, we use the concept of "pagination" -- returning a subset of
records
20 matches
Mail list logo
