Ryan, I just made the pagination code work using cursors. One thing I noticed in the documentation is this:
- A query with a cursor does not always work as expected when the query uses an inequality filter or a sort order on a property with multiple values. Logic that de-duplicates the multivalued property for the query does not persist between queries, and may return a result more than once. I am sorting the results of the query (simple query that returns data below a threshold value which is a long) by post date. The sort order is post date and might have duplicate values. Does this mean that the cursors will return duplicates? Manny On Tue, Feb 16, 2010 at 4:01 PM, Manny S <[email protected]> wrote: > Ryan - I will check out query cursors. Took a quick look and I think this > would solve my problems. Thanks. > Jeff - thanks for the input. I have included a createdDate and a > modifiedDate in all my entities. > > manny > > On Tue, Feb 16, 2010 at 12:11 AM, Jeff Schnitzer <[email protected]>wrote: > >> You will likely find it good practice to keep a createdDate (and >> possibly a modifiedDate) in all of your entities. Makes debugging >> easier, and if you're looking for this kind of pagination, there you >> go. Trying to abuse the key system for these purposes seems a bit >> hazardous. >> >> That said, you can always generate your own String name as part of the >> key. This is not a trivial task given the distributed nature of >> appengine. >> >> Jeff >> >> On Sat, Feb 13, 2010 at 8:12 AM, Manny S <[email protected]> wrote: >> > Hi Tim, >> > Thanks again Tim. A naive question - if I use the datastore generated >> keys >> > and sort by the keys desc (for pagination) the key value of new data >> > inserted will be greater than that of previously entered data (if I do a >> > keytostring and a string compare?). >> > In other words does the datastore generate keys in a order or at random. >> I >> > did some experiments and I found that pagination works according to >> > date/time of insert if I just sort keys by descending but just wanted >> to >> > make sure. >> > Regards, >> > Manny >> > On Sat, Feb 13, 2010 at 1:17 PM, Tim Hoffman <[email protected]> >> wrote: >> >> >> >> Hi Manny >> >> >> >> Understood. >> >> >> >> You know you can let the system generate the entity identifiers for >> >> you, and pagination will >> >> just work (by __key__) any way. So you don't have to the key_name or >> >> id at all >> >> if you don't care what the key is. >> >> >> >> Rgds >> >> >> >> T >> >> >> >> On Feb 13, 1:00 am, Manny S <[email protected]> wrote: >> >> > Thanks a bunch Tim for your inputs, >> >> > >> >> > My rationale for adding the date to the appstore generated key is to >> >> > make >> >> > pagination easier. I would do pagination on the key and not add a >> >> > separate >> >> > column for that. (Pagination by date alone will also not solve my >> >> > problem >> >> > since it can have duplicates and hence couple it with the key) >> >> > >> >> > I understand fetching by keys is much faster. Though I don't see a >> >> > scenario >> >> > where I would have to do that now I would like to architect my app >> where >> >> > that would be possible. However, I do not have anything unique in my >> >> > record >> >> > with which I can set the key. It just contains city name, locality >> >> > details >> >> > and a series of other fields all of which could have duplicates. Any >> >> > ideas >> >> > as to how I can generate unique ids from these or any general >> pointers >> >> > towards generating unique Ids from data where the data itself does >> not >> >> > have >> >> > a unique field? >> >> > >> >> > Manny >> >> > >> >> > On Thu, Feb 11, 2010 at 4:49 PM, Tim Hoffman <[email protected]> >> wrote: >> >> > > Hi Manny >> >> > >> >> > > Do you really want to do that for a key. One if the big advantages >> of >> >> > > creating your own keys >> >> > > is being able to explicitly get entities by key (1 or more with >> >> > > db.get(list_of_keys) which is much >> >> > > quicker than a gql or filter. Making your keys include dates mean >> you >> >> > > will be unlikely to >> >> > > guess/know what the keys are in advance. >> >> > >> >> > > This of course may not be useful for what you are doing, but worth >> >> > > keeping in mind. >> >> > >> >> > > Rgds >> >> > >> >> > > T >> >> > >> >> > > On Feb 11, 2:12 pm, Manny S <[email protected]> wrote: >> >> > > > Hi Ikai, >> >> > >> >> > > > I did read the documentation and now I have my data structures in >> >> > > > place. >> >> > > One >> >> > > > thing I wanted to do and that was not clear from my previous post >> >> > > > was to >> >> > > > append a app generated string (not unique) as a prefix to a >> >> > > > datastore >> >> > > > generated key. For instance, I want to generate a key that has >> the >> >> > > > date >> >> > > (of >> >> > > > record creation) as a prefix to the datastore generated unique >> key. >> >> > > > Is >> >> > > > there a way to do this? I do not want my application to generate >> >> > > > unique >> >> > > Ids. >> >> > >> >> > > > From reading through the literature so far, I am guessing that >> will >> >> > > > not >> >> > > be >> >> > > > possible since the datastore keys are generated only at the time >> >> > > > when the >> >> > > > objects are being made persistent. >> >> > >> >> > > > Manny >> >> > >> >> > > > On Wed, Feb 10, 2010 at 3:30 AM, Ikai L (Google) < >> [email protected]> >> >> > > wrote: >> >> > >> >> > > > > Have you read our documentation on KeyFactory? >> >> > >> >> > > > >> >> > > > > > >> http://code.google.com/appengine/docs/java/datastore/relationships.html >> >> > >> >> > > > > < >> >> > > >> http://code.google.com/appengine/docs/java/datastore/relationships.html >> >> > > >I'd >> >> > > > > try to understand what's going on there. It sounds like you're >> >> > > > > doing it >> >> > > the >> >> > > > > right way, but it's up to you to benchmark and find the best >> >> > > > > approach >> >> > > for >> >> > > > > what works for you. The usage characteristics of your >> application >> >> > > should >> >> > > > > determine the way your store your data. >> >> > >> >> > > > > On Wed, Feb 3, 2010 at 3:42 AM, Manny S <[email protected]> >> >> > > > > wrote: >> >> > >> >> > > > >> Ikai, >> >> > > > >> Based on your inputs I created two data classes that have a >> >> > > unidirectional >> >> > > > >> one-to-one relationship >> >> > > > >> Now, I have two data classes simpledata and detailscol. >> >> > > > >> simpledata contains fields A, B, C (and a Key field) >> >> > > > >> detailscol just contains field D. >> >> > >> >> > > > >> simpledata imports detailscol that contains field D (and a Key >> >> > > > >> field). >> >> > > It >> >> > > > >> also contains an accessor for the detailscol. >> >> > > > >> Code: >> >> > > > >> simpledata sdata = new simpledata(A,B,C); >> >> > > > >> sdata.setKey(null); >> >> > > > >> detailscol obj = new detailscol(D); >> >> > > > >> sdata.setD(obj); >> >> > >> >> > > > >> The keys are generated by the application and then I make the >> >> > > > >> data >> >> > > > >> persistent. >> >> > >> >> > > > >> Now, I display just the data in simpledata and if the user >> clicks >> >> > > > >> on a >> >> > > > >> details link I get the data stored in detailscol >> >> > > > >> To get to that data I just do >> >> > >> >> > > > >> detailscol d = sdata.getDetails(); >> >> > >> >> > > > >> Two questions: >> >> > >> >> > > > >> 1) Is this the right approach? >> >> > >> >> > > > >> 2) If I want to get the child data using just the parent >> keyhow >> >> > > > >> do I >> >> > > go >> >> > > > >> about it? >> >> > >> >> > > > >> E.g, user clicks details and I use some AJAX to redirect to a >> >> > > different >> >> > > > >> servlet with just parent key as a parameter (since I don't >> access >> >> > > > >> the >> >> > > child >> >> > > > >> object yet). I get the parent key using >> >> > > > >> KeyFactory.keyToString(sdata.getKey()); >> >> > >> >> > > > >> Now, that I have the parent's key should I do a getObjectbyID >> on >> >> > > > >> the >> >> > > > >> parent data again using this and then get the child using the >> >> > > > >> accessor >> >> > > > >> method or is there a direct way to construct the child key and >> >> > > > >> get to >> >> > > the >> >> > > > >> child data. >> >> > >> >> > > > >> Due to the nature of my application I would like to have the >> key >> >> > > generated >> >> > > > >> automatically (using setKey(null)). >> >> > >> >> > > > >> Apologies for the confusion in advance :) >> >> > >> >> > > > >> Manny >> >> > >> >> > > > >> On Sat, Jan 30, 2010 at 12:16 AM, Ikai L (Google) >> >> > > > >> <[email protected] >> >> > > >wrote: >> >> > >> >> > > > >>> Hi Manny, >> >> > >> >> > > > >>> A few things to first remember - App Engine's datastore is >> not a >> >> > > > >>> database, but a distributed key value store with additional >> >> > > > >>> features. >> >> > > Thus, >> >> > > > >>> we should be careful not to frame our thinking in terms of >> RDBMS >> >> > > schemas. >> >> > > > >>> For this reason, I like to avoid using database terminology >> that >> >> > > > >>> can >> >> > > > >>> confound the design process like "table" or "column". App >> Engine >> >> > > stores >> >> > > > >>> objects serialized ("entities") and indexes on the values. >> It'd >> >> > > > >>> be >> >> > > similar >> >> > > > >>> to an approach of creating a MySQL table with a String ID and >> a >> >> > > > >>> blob >> >> > > value, >> >> > > > >>> storing serialized Objects in the blob column, or using >> Memcache >> >> > > > >>> and >> >> > > storing >> >> > > > >>> JSON values. >> >> > >> >> > > > >>> When you retrieve a single value from the key value store, we >> >> > > > >>> have to >> >> > > > >>> retrieve everything at once. In most scenarios, unlike SQL >> >> > > > >>> databases >> >> > > you may >> >> > > > >>> be used to, retrieving large binary or text data does not add >> >> > > > >>> serious >> >> > > > >>> overhead. Of course, this changes if you start storing data >> on >> >> > > > >>> the >> >> > > scale of >> >> > > > >>> 1mb and are retrieving it unnecessarily. How large is the >> data >> >> > > > >>> you >> >> > > are >> >> > > > >>> retrieving? >> >> > >> >> > > > >>> Here's the way I would model your scenario if I was positive >> the >> >> > > > >>> text/binary field had a 1:1 relationship with the parent >> class: >> >> > >> >> > > > >>> * on your main entity, define the properties. >> >> > > > >>> * define a new entity with a text/binary field, and encode >> the >> >> > > > >>> parent >> >> > > key >> >> > > > >>> information in this key such that generating the key for this >> >> > > > >>> child >> >> > > field is >> >> > > > >>> very cheap. KeyFactory.stringToKey and KeyFactory.keyToString >> >> > > > >>> are >> >> > > crucial >> >> > > > >>> here. Read more about them here: >> >> > >> >> > >> >> > > > >> http://code.google.com/appengine/docs/java/javadoc/com/google/appengi.... >> >> > > > >>> You can call your child property "parent_id:additional_info" >> or >> >> > > whatever >> >> > > > >>> makes sense to you. >> >> > >> >> > > > >>> Robert's solution of using a child key is basically just a >> >> > > > >>> variation >> >> > > on >> >> > > > >>> this, as parent key information is encoded in a child key. >> >> > >> >> > > > >>> A lot of this stuff can be a bit different to get used to. I >> >> > > > >>> suggest >> >> > > > >>> becoming familiar with keys and how they are used in App >> Engine: >> >> > >> >> > > > >>> Basic documentation about relationships: >> >> > >> >> > > >> http://code.google.com/appengine/docs/java/datastore/relationships.html >> >> > > > >>> A more advanced article: >> >> > > > >>> >> http://code.google.com/appengine/articles/storage_breakdown.html >> >> > >> >> > > > >>> On Thu, Jan 28, 2010 at 10:28 PM, Manny S >> >> > > > >>> <[email protected] >> >> > > >wrote: >> >> > >> >> > > > >>>> Hi All, >> >> > >> >> > > > >>>> First off, thanks for your time. A quick noob question on >> the >> >> > > > >>>> right >> >> > > way >> >> > > > >>>> to model data. >> >> > >> >> > > > >>>> I have a table with four columns A,B,C, D. D - the fourth >> is >> >> > > > >>>> of >> >> > > type >> >> > > > >>>> text (contains quite a bit of data). >> >> > >> >> > > > >>>> I wanted to ensure that the contents of the details column >> 'D' >> >> > > > >>>> is >> >> > > not >> >> > > > >>>> fetched during a query. A sample scenario >> >> > > > >>>> User does a search. Sees Columns A,B,C. If they need more >> >> > > > >>>> details >> >> > > for >> >> > > > >>>> that particular record Click on a link that fetches D for >> that >> >> > > particular >> >> > > > >>>> record. >> >> > >> >> > > > >>>> So I tried to do something like - Select A, B, C from >> >> > > > >>>> tablename. >> >> > >> >> > > > >>>> I found from the documentation that the GQL query returns >> full >> >> > > > >>>> data >> >> > > > >>>> objects and so all queries start with SELECT *. Is this >> true >> >> > > > >>>> for >> >> > > JDOQL on >> >> > > > >>>> the datastore as well? Does this mean everytime I query the >> >> > > > >>>> data >> >> > > store its >> >> > > > >>>> going to return all columns consuming bandwidth? >> >> > >> >> > > > >>>> Also since I want the content of COlumn D to be fetched on >> >> > > subsequent >> >> > > > >>>> user action so should I instead create two tables one with >> >> > >> >> > > > >>>> ID_TB1, A, B, C >> >> > >> >> > > > >>>> and the other one with >> >> > >> >> > > > >>>> ID, ID_TB1, D? >> >> > >> >> > > > >>>> Manny >> >> > >> >> > > > >>>> -- >> >> > > > >>>> You received this message because you are subscribed to the >> >> > > > >>>> Google >> >> > > > >>>> Groups "Google App Engine" group. >> >> > > > >>>> To post to this group, send email to >> >> > > [email protected]. >> >> > > > >>>> To unsubscribe from this group, send email to >> >> > > > >>>> >> >> > > > >>>> [email protected]<google-appengine%[email protected]> >> <google-appengine%[email protected]<google-appengine%[email protected]> >> > >> >> > > >> >> > > <google-appengine%[email protected]<google-appengine%[email protected]> >> <google-appengine%[email protected]<google-appengine%[email protected]> >> > >> >> > >> >> > > > >>>> . >> >> > > > >>>> For more options, visit this group at >> >> > > > >>>>http://groups.google.com/group/google-appengine?hl=en. >> >> > >> >> > > > >>> -- >> >> > > > >>> Ikai Lan >> >> > > > >>> Developer Programs Engineer, Google App Engine >> >> > > > >>>http://googleappengine.blogspot.com| >> http://twitter.com/app_engine >> >> > >> >> > > > >>> -- >> >> > > > >>> You received this message because you are subscribed to the >> >> > > > >>> Google >> >> > > > >>> Groups "Google App Engine" group. >> >> > > > >>> To post to this group, send email to >> >> > > [email protected]. >> >> > > > >>> To unsubscribe from this group, send email to >> >> > > > >>> >> >> > > > >>> [email protected]<google-appengine%[email protected]> >> <google-appengine%[email protected]<google-appengine%[email protected]> >> > >> >> > > >> >> > > <google-appengine%[email protected]<google-appengine%[email protected]> >> <google-appengine%[email protected]<google-appengine%[email protected]> >> > >> >> > >> >> > > > >>> . >> >> > > > >>> For more options, visit this group at >> >> > > > >>>http://groups.google.com/group/google-appengine?hl=en. >> >> > >> >> > > > >> -- >> >> > > > >> You received this message because you are subscribed to the >> >> > > > >> Google >> >> > > Groups >> >> > > > >> "Google App Engine" group. >> >> > > > >> To post to this group, send >> >> > >> >> > ... >> >> > >> >> > read more ยป >> >> >> >> -- >> >> You received this message because you are subscribed to the Google >> Groups >> >> "Google App Engine" group. >> >> To post to this group, send email to [email protected] >> . >> >> To unsubscribe from this group, send email to >> >> [email protected]<google-appengine%[email protected]> >> . >> >> For more options, visit this group at >> >> http://groups.google.com/group/google-appengine?hl=en. >> >> >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups >> > "Google App Engine" group. >> > To post to this group, send email to [email protected]. >> > To unsubscribe from this group, send email to >> > [email protected]<google-appengine%[email protected]> >> . >> > For more options, visit this group at >> > http://groups.google.com/group/google-appengine?hl=en. >> > >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Google App Engine" group. >> To post to this group, send email to [email protected]. >> To unsubscribe from this group, send email to >> [email protected]<google-appengine%[email protected]> >> . >> For more options, visit this group at >> http://groups.google.com/group/google-appengine?hl=en. >> >> > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
