Massimo, Scott, i filed https://code.google.com/p/web2py/issues/detail?id=1373 with a patch against trunk. Looks like my first patch on this thread made it to the latest release, but the second patch did not. If that can be applied i think we'll be all set.
perhaps i now need to start submitting book patches to write up these features! cfh On Thursday, February 28, 2013 9:35:33 AM UTC-8, Scott Hunter wrote: > > Sorry, I thought I had -- it works like a champ. Would be nice if there > were a way to mark the signature fields as unindexed w/o having to recreate > them (not as simple as cut and paste, as the table def uses a bunch of > local definitions, but not hard), but definitely workable. And lowers the > index overhead a LOT; in my case, from 15K to <3K. > > - Scott > > On Thursday, February 28, 2013 11:12:52 AM UTC-5, howesc wrote: >> >> can you let me know if it works as desired now? if so, i'll submit the >> patch to massimo for review and possible inclusion in the next release of >> web2py. >> >> thanks, >> >> cfh >> >> On Tuesday, February 26, 2013 6:31:52 PM UTC-8, Scott Hunter wrote: >>> >>> With the latest patch, I was able to disable indices on the string >>> fields I wanted to. Unfortunately (for me), I've just about used up my >>> quota, so I'll have to wait until tomorrow to see how much doing so saves >>> me. >>> >>> Thanks, >>> Scott >>> >>> On Tuesday, February 26, 2013 8:50:29 PM UTC-5, howesc wrote: >>>> >>>> here's an updated DAL patch to try..... >>>> >>>> thanks for trudging through this with us! >>>> >>>> cfh >>>> >>>> On Tuesday, February 26, 2013 3:36:30 PM UTC-8, Scott Hunter wrote: >>>>> >>>>> I'm working from Version 2.3.2 (2012-12-17 15:03:30) stable >>>>> >>>>> Here's the table in question (I've commented-out the custom_qualifier >>>>> for the string fields): >>>>> >>>>> db.define_table('t_run', >>>>> Field('f_trial', type='reference t_trial', >>>>> label=T('Trial')), >>>>> Field('f_when', type='date', >>>>> label=T('When')), >>>>> Field('f_dog', type='reference t_dog', >>>>> label=T('Dog')), >>>>> Field('f_name', type='string',#custom_qualifier={'indexed':False}, >>>>> label=T('Name')), >>>>> Field('f_breed', type='string', >>>>> #custom_qualifier={'indexed':False}, >>>>> label=T('Breed')), >>>>> Field('f_jump_height', type='integer',custom_qualifier={'indexed': >>>>> False}, >>>>> label=T('Jump Height')), >>>>> Field('f_level', type='string', >>>>> #custom_qualifier={'indexed':False}, >>>>> label=T('Level')), >>>>> Field('f_class', type='string', >>>>> #custom_qualifier={'indexed':False}, >>>>> label=T('Class')), >>>>> Field('f_pref', type='boolean',custom_qualifier={'indexed':False}, >>>>> label=T('Preferred')), >>>>> Field('f_armband', type='string', >>>>> #custom_qualifier={'indexed':False}, >>>>> label=T('Armband')), >>>>> Field('f_yards', type='integer',custom_qualifier={'indexed':False >>>>> }, >>>>> label=T('Yards')), >>>>> Field('f_sct', type='integer',custom_qualifier={'indexed':False}, >>>>> label=T('SCT')), >>>>> Field('f_judge', type='string', >>>>> #custom_qualifier={'indexed':False}, >>>>> label=T('Judge')), >>>>> Field('f_score', type='integer',custom_qualifier={'indexed':False >>>>> }, >>>>> label=T('Score')), >>>>> Field('f_time', type='double',custom_qualifier={'indexed':False}, >>>>> label=T('Time')), >>>>> Field('f_faults', type='string', >>>>> #custom_qualifier={'indexed':False}, >>>>> label=T('Faults')), >>>>> Field('f_handler', type='string', >>>>> #custom_qualifier={'indexed':False}, >>>>> label=T('Handler')), >>>>> Field('f_order', type='integer', >>>>> label=T('Order')), >>>>> auth.signature, >>>>> format='%(f_key)s', >>>>> migrate=settings.migrate) >>>>> >>>>> >>>>> P.S. Turning off the indexes DOES make a difference; with the changes >>>>> above, instead of using up over 30% of my quota, I'm "only" using 23%. >>>>> If >>>>> I can get the strings unindexed, and take out the web2py-supplied fields >>>>> (created_by & _on, modified_by & _on, and maybe is_active), that should >>>>> get >>>>> it down to something manageable. (Not clear on how to handle the >>>>> web2py-supplied fields, as I don't know what parameters were using in >>>>> making them, making it difficult to know just how to "supply my own >>>>> definitions".) >>>>> >>>>> On Tuesday, February 26, 2013 6:24:21 PM UTC-5, howesc wrote: >>>>>> >>>>>> your line numbers are off from mine so i'm having trouble making >>>>>> sense of this. :( >>>>>> >>>>>> can you send your model definition so i can see what you are working >>>>>> with? i think then i can line it up with the dal version i have here >>>>>> (which was trunk from HG as of saturday AM PST) >>>>>> >>>>>> cfh >>>>>> >>>>>> On Tuesday, February 26, 2013 3:18:13 PM UTC-8, Scott Hunter wrote: >>>>>>> >>>>>>> Sorry about that. I've fixed it, and now I get the following: >>>>>>> >>>>>>> In FILE: /base/data/home/apps/s~sbhweb2py/1.365574604253984974/ >>>>>>> applications/ppt_demo/models/db_wizard.py >>>>>>> >>>>>>> >>>>>>> Traceback (most recent call last): >>>>>>> File >>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/restricted.py" >>>>>>> , line 212, in restricted >>>>>>> exec ccode in environment >>>>>>> File >>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/applications/ppt_demo/models/db_wizard.py" >>>>>>> , line 198, in <module> >>>>>>> migrate=settings.migrate) >>>>>>> File >>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/dal.py" >>>>>>> , line 7189, in define_table >>>>>>> table = self.lazy_define_table(tablename,*fields,**args) >>>>>>> File >>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/dal.py" >>>>>>> , line 7225, in lazy_define_table >>>>>>> polymodel=polymodel) >>>>>>> File >>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/dal.py" >>>>>>> , line 4368, in create_table >>>>>>> ftype = self.types[field_type](**attr) >>>>>>> TypeError: <lambda>() takes no arguments (1 given) >>>>>>> >>>>>>> >>>>>>> This only happens if one of the fields in question is a string; I >>>>>>> get no error when the unindexed fields are all integer, boolean or >>>>>>> double. >>>>>>> >>>>>>> - Scott >>>>>>> >>>>>>> >>>>>>> On Tuesday, February 26, 2013 1:04:12 AM UTC-5, howesc wrote: >>>>>>>> >>>>>>>> it looks like you have a typo.... "custom_qualifer" vs >>>>>>>> "custom_qualifier" >>>>>>>> >>>>>>>> On Monday, February 25, 2013 6:44:14 PM UTC-8, Scott Hunter wrote: >>>>>>>>> >>>>>>>>> I applied the patch, and added custom_qualifiers like so: >>>>>>>>> >>>>>>>>> Field('f_name', type='string',custom_qualifer={'indexed':False >>>>>>>>> }, >>>>>>>>> label=T('Name')), >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> and this is the error I got: >>>>>>>>> >>>>>>>>> In FILE: /base/data/home/apps/s~sbhweb2py/1.365567821359373728/ >>>>>>>>> applications/ppt_demo/models/db_wizard.py >>>>>>>>> >>>>>>>>> >>>>>>>>> Traceback (most recent call last): >>>>>>>>> File >>>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365567821359373728/gluon/restricted.py" >>>>>>>>> , line 212, in restricted >>>>>>>>> exec ccode in environment >>>>>>>>> File >>>>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365567821359373728/applications/ppt_demo/models/db_wizard.py" >>>>>>>>> , line 165, in <module> >>>>>>>>> label=T('Name')), >>>>>>>>> TypeError: __init__() got an unexpected keyword argument >>>>>>>>> 'custom_qualifer' >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Saturday, February 23, 2013 12:30:48 PM UTC-5, howesc wrote: >>>>>>>>>> >>>>>>>>>> Scott, >>>>>>>>>> >>>>>>>>>> this is *completely* untested, but here's a proposal: >>>>>>>>>> - use the (undocumented) field.custom_qualifier property in GAE >>>>>>>>>> field definitions >>>>>>>>>> - if you want a field to be unindexed set >>>>>>>>>> custom_qualifier={'indexed':False} >>>>>>>>>> - for "properties that don't get listed explicitly"....override >>>>>>>>>> those default properties with what you want (there are a few >>>>>>>>>> techniques for >>>>>>>>>> not explicitly listing fields, depending on which technique you are >>>>>>>>>> using >>>>>>>>>> the answer here is different. >>>>>>>>>> >>>>>>>>>> this patch against HG trunk this AM might work (i say might cause >>>>>>>>>> i have not tested it). are you willing to experiment with it and >>>>>>>>>> let us >>>>>>>>>> know? >>>>>>>>>> >>>>>>>>>> thanks, >>>>>>>>>> >>>>>>>>>> christian >>>>>>>>>> >>>>>>>>>> On Friday, February 22, 2013 3:53:33 PM UTC-8, Scott Hunter wrote: >>>>>>>>>>> >>>>>>>>>>> If I had to guess, I'd say a patch is needed in the loop over >>>>>>>>>>> the fields in create_table, that would add a new entry to the dict >>>>>>>>>>> for the >>>>>>>>>>> field definition based on a new attribute (which would be ignored >>>>>>>>>>> for >>>>>>>>>>> anything other than Google's Datastore) which gets added to >>>>>>>>>>> sql_fields; >>>>>>>>>>> then migrate_table, which seems to build the table >>>>>>>>>>> building/altering >>>>>>>>>>> commands would need to be made to recognize the new entry & add the >>>>>>>>>>> disable >>>>>>>>>>> index command. There's not a lot of comments in the code, and I'd >>>>>>>>>>> be >>>>>>>>>>> afraid to break something, but might give it a try. >>>>>>>>>>> >>>>>>>>>>> - Scott >>>>>>>>>>> >>>>>>>>>>> P.S. Not sure of the best way to handle this (what's describe >>>>>>>>>>> above wouldn't handle it), but it would be nice to be able to turn >>>>>>>>>>> off the >>>>>>>>>>> indexes for the fields that don't get listed explicitly in >>>>>>>>>>> models.db >>>>>>>>>>> (things like created by & when, modified by & when). >>>>>>>>>>> >>>>>>>>>>> On Friday, February 22, 2013 6:14:28 PM UTC-5, howesc wrote: >>>>>>>>>>>> >>>>>>>>>>>> thanks for the link. i'll try and take a look this weekend and >>>>>>>>>>>> see if there is a place for that in the DAL (feel free to open >>>>>>>>>>>> gluon/dal.py yourself too). >>>>>>>>>>>> >>>>>>>>>>>> yup, i'm paying more for writes right now then i am for >>>>>>>>>>>> instance hours per day on my largest paid application. :) >>>>>>>>>>>> >>>>>>>>>>>> cfh >>>>>>>>>>>> >>>>>>>>>>>> On Thursday, February 21, 2013 5:59:06 PM UTC-8, Scott Hunter >>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> According to >>>>>>>>>>>>> https://developers.google.com/appengine/docs/python/datastore/indexes#Unindexed_Properties, >>>>>>>>>>>>> >>>>>>>>>>>>> "You declare a property unindexed by setting indexed=False in the >>>>>>>>>>>>> property >>>>>>>>>>>>> constructor". >>>>>>>>>>>>> >>>>>>>>>>>>> One incurs the cost of a write (or 2?) for every property of a >>>>>>>>>>>>> record that gets written; so, for a table with about 25 fields >>>>>>>>>>>>> (once you >>>>>>>>>>>>> include all of the ones web2py adds), inserting 300 records >>>>>>>>>>>>> incurs the cost >>>>>>>>>>>>> of 14K+ writes; as the free account has a limit of 50K writes per >>>>>>>>>>>>> day, that >>>>>>>>>>>>> is rather limiting. It is easy to turn such indicies back on, >>>>>>>>>>>>> but you have >>>>>>>>>>>>> to re-write every record in order to repopulate them. >>>>>>>>>>>>> >>>>>>>>>>>>> - Scott >>>>>>>>>>>>> >>>>>>>>>>>>> On Thursday, February 21, 2013 4:51:59 PM UTC-5, howesc wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> i'm being lazy here....do you have the instructions (or link >>>>>>>>>>>>>> to instructions) from GAE for disabling indexes? i don't have >>>>>>>>>>>>>> it handy >>>>>>>>>>>>>> right now.... >>>>>>>>>>>>>> >>>>>>>>>>>>>> we can check if the DAL has a secret way to handle it, and/or >>>>>>>>>>>>>> create a patch to allow it. >>>>>>>>>>>>>> >>>>>>>>>>>>>> may i ask what advantage you are hoping to achieve by >>>>>>>>>>>>>> skipping those indexes? i've left them on for fear of making a >>>>>>>>>>>>>> bad choice >>>>>>>>>>>>>> that i can't revert. >>>>>>>>>>>>>> >>>>>>>>>>>>>> christian >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thursday, February 21, 2013 5:56:07 AM UTC-8, Scott Hunter >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> As I understand it, the GAE datastore will automatically >>>>>>>>>>>>>>> create an index for every field of every table (to facilitate >>>>>>>>>>>>>>> queries using >>>>>>>>>>>>>>> single fields); those requiring multiple fields also get >>>>>>>>>>>>>>> generated, but >>>>>>>>>>>>>>> will get added to index.yaml by the SDK when tested locally. >>>>>>>>>>>>>>> Since these >>>>>>>>>>>>>>> are created outside of the DAL, I wouldn't expect to be able to >>>>>>>>>>>>>>> remove them >>>>>>>>>>>>>>> using it. I also understand that there is a way, with the GAE >>>>>>>>>>>>>>> SDK, to >>>>>>>>>>>>>>> disable the index for any given field (by a parameter in the >>>>>>>>>>>>>>> field >>>>>>>>>>>>>>> definition); is there any way to accomplish this through the >>>>>>>>>>>>>>> DAL, and if >>>>>>>>>>>>>>> so, what is it? >>>>>>>>>>>>>> >>>>>>>>>>>>>> -- --- You received this message because you are subscribed to the Google Groups "web2py-users" group. To unsubscribe from this group and stop receiving emails from it, send an email to web2py+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.