With the latest patch, I was able to disable indices on the string fields I
wanted to. Unfortunately (for me), I've just about used up my quota, so
I'll have to wait until tomorrow to see how much doing so saves me.
Thanks,
Scott
On Tuesday, February 26, 2013 8:50:29 PM UTC-5, howesc wrote:
>
> here's an updated DAL patch to try.....
>
> thanks for trudging through this with us!
>
> cfh
>
> On Tuesday, February 26, 2013 3:36:30 PM UTC-8, Scott Hunter wrote:
>>
>> I'm working from Version 2.3.2 (2012-12-17 15:03:30) stable
>>
>> Here's the table in question (I've commented-out the custom_qualifier for
>> the string fields):
>>
>> db.define_table('t_run',
>> Field('f_trial', type='reference t_trial',
>> label=T('Trial')),
>> Field('f_when', type='date',
>> label=T('When')),
>> Field('f_dog', type='reference t_dog',
>> label=T('Dog')),
>> Field('f_name', type='string',#custom_qualifier={'indexed':False},
>> label=T('Name')),
>> Field('f_breed', type='string',#custom_qualifier={'indexed':False},
>> label=T('Breed')),
>> Field('f_jump_height', type='integer',custom_qualifier={'indexed':
>> False},
>> label=T('Jump Height')),
>> Field('f_level', type='string',#custom_qualifier={'indexed':False},
>> label=T('Level')),
>> Field('f_class', type='string',#custom_qualifier={'indexed':False},
>> label=T('Class')),
>> Field('f_pref', type='boolean',custom_qualifier={'indexed':False},
>> label=T('Preferred')),
>> Field('f_armband', type='string',#custom_qualifier={'indexed':False},
>> label=T('Armband')),
>> Field('f_yards', type='integer',custom_qualifier={'indexed':False},
>> label=T('Yards')),
>> Field('f_sct', type='integer',custom_qualifier={'indexed':False},
>> label=T('SCT')),
>> Field('f_judge', type='string',#custom_qualifier={'indexed':False},
>> label=T('Judge')),
>> Field('f_score', type='integer',custom_qualifier={'indexed':False},
>> label=T('Score')),
>> Field('f_time', type='double',custom_qualifier={'indexed':False},
>> label=T('Time')),
>> Field('f_faults', type='string',#custom_qualifier={'indexed':False},
>> label=T('Faults')),
>> Field('f_handler', type='string',#custom_qualifier={'indexed':False},
>> label=T('Handler')),
>> Field('f_order', type='integer',
>> label=T('Order')),
>> auth.signature,
>> format='%(f_key)s',
>> migrate=settings.migrate)
>>
>>
>> P.S. Turning off the indexes DOES make a difference; with the changes
>> above, instead of using up over 30% of my quota, I'm "only" using 23%. If
>> I can get the strings unindexed, and take out the web2py-supplied fields
>> (created_by & _on, modified_by & _on, and maybe is_active), that should get
>> it down to something manageable. (Not clear on how to handle the
>> web2py-supplied fields, as I don't know what parameters were using in
>> making them, making it difficult to know just how to "supply my own
>> definitions".)
>>
>> On Tuesday, February 26, 2013 6:24:21 PM UTC-5, howesc wrote:
>>>
>>> your line numbers are off from mine so i'm having trouble making sense
>>> of this. :(
>>>
>>> can you send your model definition so i can see what you are working
>>> with? i think then i can line it up with the dal version i have here
>>> (which was trunk from HG as of saturday AM PST)
>>>
>>> cfh
>>>
>>> On Tuesday, February 26, 2013 3:18:13 PM UTC-8, Scott Hunter wrote:
>>>>
>>>> Sorry about that. I've fixed it, and now I get the following:
>>>>
>>>> In FILE: /base/data/home/apps/s~sbhweb2py/1.365574604253984974/
>>>> applications/ppt_demo/models/db_wizard.py
>>>>
>>>>
>>>> Traceback (most recent call last):
>>>> File
>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/restricted.py"
>>>> , line 212, in restricted
>>>> exec ccode in environment
>>>> File
>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/applications/ppt_demo/models/db_wizard.py"
>>>> , line 198, in <module>
>>>> migrate=settings.migrate)
>>>> File
>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/dal.py",line
>>>> 7189, in define_table
>>>> table = self.lazy_define_table(tablename,*fields,**args)
>>>> File
>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/dal.py",line
>>>> 7225, in lazy_define_table
>>>> polymodel=polymodel)
>>>> File
>>>> "/base/data/home/apps/s~sbhweb2py/1.365574604253984974/gluon/dal.py",line
>>>> 4368, in create_table
>>>> ftype = self.types[field_type](**attr)
>>>> TypeError: <lambda>() takes no arguments (1 given)
>>>>
>>>>
>>>> This only happens if one of the fields in question is a string; I get
>>>> no error when the unindexed fields are all integer, boolean or double.
>>>>
>>>> - Scott
>>>>
>>>>
>>>> On Tuesday, February 26, 2013 1:04:12 AM UTC-5, howesc wrote:
>>>>>
>>>>> it looks like you have a typo.... "custom_qualifer" vs
>>>>> "custom_qualifier"
>>>>>
>>>>> On Monday, February 25, 2013 6:44:14 PM UTC-8, Scott Hunter wrote:
>>>>>>
>>>>>> I applied the patch, and added custom_qualifiers like so:
>>>>>>
>>>>>> Field('f_name', type='string',custom_qualifer={'indexed':False},
>>>>>> label=T('Name')),
>>>>>>
>>>>>>
>>>>>>
>>>>>> and this is the error I got:
>>>>>>
>>>>>> In FILE: /base/data/home/apps/s~sbhweb2py/1.365567821359373728/
>>>>>> applications/ppt_demo/models/db_wizard.py
>>>>>>
>>>>>>
>>>>>> Traceback (most recent call last):
>>>>>> File
>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365567821359373728/gluon/restricted.py"
>>>>>> , line 212, in restricted
>>>>>> exec ccode in environment
>>>>>> File
>>>>>> "/base/data/home/apps/s~sbhweb2py/1.365567821359373728/applications/ppt_demo/models/db_wizard.py"
>>>>>> , line 165, in <module>
>>>>>> label=T('Name')),
>>>>>> TypeError: __init__() got an unexpected keyword argument
>>>>>> 'custom_qualifer'
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Saturday, February 23, 2013 12:30:48 PM UTC-5, howesc wrote:
>>>>>>>
>>>>>>> Scott,
>>>>>>>
>>>>>>> this is *completely* untested, but here's a proposal:
>>>>>>> - use the (undocumented) field.custom_qualifier property in GAE
>>>>>>> field definitions
>>>>>>> - if you want a field to be unindexed set
>>>>>>> custom_qualifier={'indexed':False}
>>>>>>> - for "properties that don't get listed explicitly"....override
>>>>>>> those default properties with what you want (there are a few techniques
>>>>>>> for
>>>>>>> not explicitly listing fields, depending on which technique you are
>>>>>>> using
>>>>>>> the answer here is different.
>>>>>>>
>>>>>>> this patch against HG trunk this AM might work (i say might cause i
>>>>>>> have not tested it). are you willing to experiment with it and let us
>>>>>>> know?
>>>>>>>
>>>>>>> thanks,
>>>>>>>
>>>>>>> christian
>>>>>>>
>>>>>>> On Friday, February 22, 2013 3:53:33 PM UTC-8, Scott Hunter wrote:
>>>>>>>>
>>>>>>>> If I had to guess, I'd say a patch is needed in the loop over the
>>>>>>>> fields in create_table, that would add a new entry to the dict for the
>>>>>>>> field definition based on a new attribute (which would be ignored for
>>>>>>>> anything other than Google's Datastore) which gets added to
>>>>>>>> sql_fields;
>>>>>>>> then migrate_table, which seems to build the table building/altering
>>>>>>>> commands would need to be made to recognize the new entry & add the
>>>>>>>> disable
>>>>>>>> index command. There's not a lot of comments in the code, and I'd be
>>>>>>>> afraid to break something, but might give it a try.
>>>>>>>>
>>>>>>>> - Scott
>>>>>>>>
>>>>>>>> P.S. Not sure of the best way to handle this (what's describe above
>>>>>>>> wouldn't handle it), but it would be nice to be able to turn off the
>>>>>>>> indexes for the fields that don't get listed explicitly in models.db
>>>>>>>> (things like created by & when, modified by & when).
>>>>>>>>
>>>>>>>> On Friday, February 22, 2013 6:14:28 PM UTC-5, howesc wrote:
>>>>>>>>>
>>>>>>>>> thanks for the link. i'll try and take a look this weekend and
>>>>>>>>> see if there is a place for that in the DAL (feel free to open
>>>>>>>>> gluon/dal.py yourself too).
>>>>>>>>>
>>>>>>>>> yup, i'm paying more for writes right now then i am for instance
>>>>>>>>> hours per day on my largest paid application. :)
>>>>>>>>>
>>>>>>>>> cfh
>>>>>>>>>
>>>>>>>>> On Thursday, February 21, 2013 5:59:06 PM UTC-8, Scott Hunter
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> According to
>>>>>>>>>> https://developers.google.com/appengine/docs/python/datastore/indexes#Unindexed_Properties,
>>>>>>>>>>
>>>>>>>>>> "You declare a property unindexed by setting indexed=False in the
>>>>>>>>>> property
>>>>>>>>>> constructor".
>>>>>>>>>>
>>>>>>>>>> One incurs the cost of a write (or 2?) for every property of a
>>>>>>>>>> record that gets written; so, for a table with about 25 fields (once
>>>>>>>>>> you
>>>>>>>>>> include all of the ones web2py adds), inserting 300 records incurs
>>>>>>>>>> the cost
>>>>>>>>>> of 14K+ writes; as the free account has a limit of 50K writes per
>>>>>>>>>> day, that
>>>>>>>>>> is rather limiting. It is easy to turn such indicies back on, but
>>>>>>>>>> you have
>>>>>>>>>> to re-write every record in order to repopulate them.
>>>>>>>>>>
>>>>>>>>>> - Scott
>>>>>>>>>>
>>>>>>>>>> On Thursday, February 21, 2013 4:51:59 PM UTC-5, howesc wrote:
>>>>>>>>>>>
>>>>>>>>>>> i'm being lazy here....do you have the instructions (or link to
>>>>>>>>>>> instructions) from GAE for disabling indexes? i don't have it
>>>>>>>>>>> handy right
>>>>>>>>>>> now....
>>>>>>>>>>>
>>>>>>>>>>> we can check if the DAL has a secret way to handle it, and/or
>>>>>>>>>>> create a patch to allow it.
>>>>>>>>>>>
>>>>>>>>>>> may i ask what advantage you are hoping to achieve by skipping
>>>>>>>>>>> those indexes? i've left them on for fear of making a bad choice
>>>>>>>>>>> that i
>>>>>>>>>>> can't revert.
>>>>>>>>>>>
>>>>>>>>>>> christian
>>>>>>>>>>>
>>>>>>>>>>> On Thursday, February 21, 2013 5:56:07 AM UTC-8, Scott Hunter
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> As I understand it, the GAE datastore will automatically create
>>>>>>>>>>>> an index for every field of every table (to facilitate queries
>>>>>>>>>>>> using single
>>>>>>>>>>>> fields); those requiring multiple fields also get generated, but
>>>>>>>>>>>> will get
>>>>>>>>>>>> added to index.yaml by the SDK when tested locally. Since these
>>>>>>>>>>>> are
>>>>>>>>>>>> created outside of the DAL, I wouldn't expect to be able to remove
>>>>>>>>>>>> them
>>>>>>>>>>>> using it. I also understand that there is a way, with the GAE
>>>>>>>>>>>> SDK, to
>>>>>>>>>>>> disable the index for any given field (by a parameter in the field
>>>>>>>>>>>> definition); is there any way to accomplish this through the DAL,
>>>>>>>>>>>> and if
>>>>>>>>>>>> so, what is it?
>>>>>>>>>>>
>>>>>>>>>>>
--
---
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to web2py+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
