>From what I've read enabling billing doesn't increase the CPU time
just some of the other quotas. It's CPU time that gets eaten up so
badly when you remove/add items to the datastore and I'm not keen on
paying for extra CPU time until I at least have some sort of working
app to expand on.
As far as the duplicate entries, I removed the "ID" field before
uploading since I knew the datastore would assign a different ID
number than a SQL based web2py table. Is that the reason re-uploading
data didn't overwrite old data? I would have preferred it to have
over written it, but with out knowing the unique identifier that
didn't happen. I'm sure I could use the remote_api_shell to recurse
through the database and remove duplicates it's just that I don't see
how that could be the problem.
I've tried uploading this same data to the development (local) copy of
the GAE datastore and the table finds the entities as expected. So
where do I go now since that's not the same thing I'm seeing on the
production Datastore? Do I need to delete all enteries and re-upload
all of the data again? I sure hope not.
Thanks for pointing out where I can find info on what the web2py
fields translate to in datastore terms. That will help me insure I
don't have an issue with that in the future.
On Sep 19, 5:09 pm, Robin B <robi...@gmail.com> wrote:
> > Looking over two tables here are the things I've identified as
> > possible problems:
> > 1 - This table has a 'double' in it. Not just int/string like the
> > working one. For GAE uploading I defined the double field as a GAE
> > "float" property. Is it possible web2py defines a 'double' as
> > something else on GAE? If so is there a mapping that would tell me
> > how to map web2py fields into GAE fields for uploading data?
>
> On GAE, 'double' is a float looking at the top of gql.py.
>
>
>
> > 2 - Searching the Data Viewer on GAE shows that due to a previously
> > failed data upload I now have multiple copies of some of the
> > entities. These copies the exact same data in the except for the ID/
> > Name fields. I would like to remove the duplicates sometime but that
> > wasn't my first concern. Would duplicate copies cause this problem?
> > I figured since the ID is unique it would just show up as another
> > entery in the table. I haven't set any of the entries to 'unique'.
>
> It depends on your app, but you probably want to remove any duplicates
> before reloading the data. I read that the bulk loader will over-
> write records using the IDs given in the .csv, which is what you want.
>
>
>
> > 3 - Both tables are set with migrate=False, specifically b/c I've
> > seperated the data upload and process from the data reading process.
> > I'm not even sure what web2py would do if I removed the migrate=false
> > flag. What is the recommendation here?
>
> AFAIK, migrate should have no effect on GAE.
>
>
>
> > Inserting/deleting items from the GAE Datastore is very expensive in
> > CPU time, I messed with this yesterday and in a matter of a few
> > minutes (delete all, re-upload) was over my quota. Because of that
> > I'd really like to avoid having to change data in the Datastore as
> > much as I can. With 20K+ items you don't get a lot of 'do-overs'.
>
> Have you enabled billing on your account? Enabiling billing greatly
> increases your free quota.
>
> Robin
>
> On Sep 19, 2:45 pm, Chris S <sanders.ch...@gmail.com> wrote:
>
> > Background:
> > Working with an app on GAE currently I've run across a slight issue
> > with uploading data into my app. I initially tried to download my
> > tables to CSV from my development machine then use simply re-upload
> > them into GAE. It's been a while since I tried that but it was timing
> > out.
>
> > Problem:
> > Because of that I decided to upload with bulkloader.py. I did my
> > first table (~2K entries) with out a hitch. My second Table is where
> > I'm having an issue. This table is about 24K entities. After
> > uploading all of the data trying to view it on the appadmin returns
> > the following error (web2py error ticket):
> > The table is "sqldb.item" if it matters.
> > -------------
> > Traceback (most recent call last):
> > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
> > main.py", line 396, in wsgibase
> > session._try_store_in_db(request, response)
> > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
> > globals.py", line 342, in _try_store_in_db
> > table._db(table.id == record_id).update(**dd)
> > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
> > contrib/gql.py", line 698, in update
> > (item, tablename, fields) = self._getitem_exception()
> > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/
> > contrib/gql.py", line 632, in _getitem_exception
> > item = self._db[tablename]._tableobj.get_by_id(long(id))
> > File "/base/python_lib/versions/1/google/appengine/ext/db/
> > __init__.py", line 943, in get_by_id
> > return get(*keys)
> > File "/base/python_lib/versions/1/google/appengine/ext/db/
> > __init__.py", line 1123, in get
> > entities = datastore.Get(keys)
> > File "/base/python_lib/versions/1/google/appengine/api/
> > datastore.py", line 224, in Get
> > apiproxy_stub_map.MakeSyncCall('datastore_v3', 'Get', req, resp)
> > File "/base/python_lib/versions/1/google/appengine/api/
> > apiproxy_stub_map.py", line 72, in MakeSyncCall
> > apiproxy.MakeSyncCall(service, call, request, response)
> > File "/base/python_lib/versions/1/google/appengine/api/
> > apiproxy_stub_map.py", line 255, in MakeSyncCall
> > rpc.CheckSuccess()
> > File "/base/python_lib/versions/1/google/appengine/api/
> > apiproxy_rpc.py", line 111, in CheckSuccess
> > raise self.exception
> > CancelledError: The API call datastore_v3.Get() was explicitly
> > cancelled.
> > -----------------------------
>
> > Looking over two tables here are the things I've identified as
> > possible problems:
> > 1 - This table has a 'double' in it. Not just int/string like the
> > working one. For GAE uploading I defined the double field as a GAE
> > "float" property. Is it possible web2py defines a 'double' as
> > something else on GAE? If so is there a mapping that would tell me
> > how to map web2py fields into GAE fields for uploading data?
>
> > 2 - Searching the Data Viewer on GAE shows that due to a previously
> > failed data upload I now have multiple copies of some of the
> > entities. These copies the exact same data in the except for the ID/
> > Name fields. I would like to remove the duplicates sometime but that
> > wasn't my first concern. Would duplicate copies cause this problem?
> > I figured since the ID is unique it would just show up as another
> > entery in the table. I haven't set any of the entries to 'unique'.
>
> > 3 - Both tables are set with migrate=False, specifically b/c I've
> > seperated the data upload and process from the data reading process.
> > I'm not even sure what web2py would do if I removed the migrate=false
> > flag. What is the recommendation here?
>
> > Inserting/deleting items from the GAE Datastore is very expensive in
> > CPU time, I messed with this yesterday and in a matter of a few
> > minutes (delete all, re-upload) was over my quota. Because of that
> > I'd really like to avoid having to change data in the Datastore as
> > much as I can. With 20K+ items you don't get a lot of 'do-overs'.
--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups
"web2py-users" group.
To post to this group, send email to web2py@googlegroups.com
To unsubscribe from this group, send email to
web2py+unsubscr...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/web2py?hl=en
-~----------~----~----~----~------~----~------~--~---
