I have not tried the appadmin on GAE, but I wonder if it uses count and or offset which will not work beyond 1000 records. You have to roll your own pagination on GAE by giving each field a unique composite sorted value or the ID, and then pass a next=key url query string parameter that points to the first item of the next page, and then query for it, not offset to it.
I would try to upload your data to a new table. I saw you can specify the --kind to the bulk uploader, so maybe just pick a new table name and upload the data there. To delete the old table, you could try the remote api and slowly delete the data locally. Another way is to make a cron task that periodically removes rows from the first table. Or if you want to use the task queue, make a task that removes 10 items and then spawns another task, if there are items remaining, that executes after some delay. Eventually all the items will be deleted. Robin On Sep 19, 9:49 pm, Chris S <sanders.ch...@gmail.com> wrote: > I was trying to move a MySQL based app to the GAE. The MySQL was > built by inserting values into the SQL table with a python script, so > it made since (to me) to do the same on the GAE. I'm guessing the > easiest way to avoid this would have been to download a CSV of my data > and use the appadmin to upload it again, however the larger of the two > tables returns 'can't parse file' when I tried that. So I was trying > to upload the data with a script. > > However, it seems the error I'm seeing here might actually just be > related to the appadmin displaying very large tables. I tried doing a > serach for just one of my uploaded entries and it's actually there I > just have to put in the URL to the appadmin manually b/c the default > is to display all values. > > I'm uploading a version of my code now that I think will work since > the data appears to be there and it's just an appadmin problem with > the display. > > As for removing the duplicates it's not a big deal but anyone have a > suggestion as to how I go about that? With the 1000 entity limit on > selects I'm trying to think now how to best find only duplicates. > Unfortunately I didn't put any 'time' variables in the table. I'm > open to suggestions. > > On Sep 19, 8:02 pm, Robin B <robi...@gmail.com> wrote: > > > On Sep 19, 6:13 pm, Chris S <sanders.ch...@gmail.com> wrote: > > > > From what I've read enabling billing doesn't increase the CPU time > > > just some of the other quotas. It's CPU time that gets eaten up so > > > badly when you remove/add items to the datastore and I'm not keen on > > > paying for extra CPU time until I at least have some sort of working > > > app to expand on. > > > It is true that enabling billing does not increase the total free > > monthly CPU, it does however increase the max rate you can consume CPU > > by 4.8X, so you won't hit your over quota so quickly in the short > > term. > > >http://code.google.com/appengine/docs/quotas.html#Resources > > > > As far as the duplicate entries, I removed the "ID" field before > > > uploading since I knew the datastore would assign a different ID > > > number than a SQL based web2py table. Is that the reason re-uploading > > > data didn't overwrite old data? I would have preferred it to have > > > over written it, but with out knowing the unique identifier that > > > didn't happen. I'm sure I could use the remote_api_shell to recurse > > > through the database and remove duplicates it's just that I don't see > > > how that could be the problem. > > > Removing the IDs certainly created the duplicate entries, and if you > > had references between entries, the data is probably invalid. > > > > I've tried uploading this same data to the development (local) copy of > > > the GAE datastore and the table finds the entities as expected. So > > > where do I go now since that's not the same thing I'm seeing on the > > > production Datastore? Do I need to delete all enteries and re-upload > > > all of the data again? I sure hope not. > > > I would try to remove the table entries. If you can't remove the > > entries, you might instead create a brand new table with a different > > name somehow. > > > > Thanks for pointing out where I can find info on what the web2py > > > fields translate to in datastore terms. That will help me insure I > > > don't have an issue with that in the future. > > > What were you trying to do? Backup/Restore? Schema migration? > > > Robin > > > > On Sep 19, 5:09 pm, Robin B <robi...@gmail.com> wrote: > > > > > > Looking over two tables here are the things I've identified as > > > > > possible problems: > > > > > 1 - This table has a 'double' in it. Not just int/string like the > > > > > working one. For GAE uploading I defined the double field as a GAE > > > > > "float" property. Is it possible web2py defines a 'double' as > > > > > something else on GAE? If so is there a mapping that would tell me > > > > > how to map web2py fields into GAE fields for uploading data? > > > > > On GAE, 'double' is a float looking at the top of gql.py. > > > > > > 2 - Searching the Data Viewer on GAE shows that due to a previously > > > > > failed data upload I now have multiple copies of some of the > > > > > entities. These copies the exact same data in the except for the ID/ > > > > > Name fields. I would like to remove the duplicates sometime but that > > > > > wasn't my first concern. Would duplicate copies cause this problem? > > > > > I figured since the ID is unique it would just show up as another > > > > > entery in the table. I haven't set any of the entries to 'unique'. > > > > > It depends on your app, but you probably want to remove any duplicates > > > > before reloading the data. I read that the bulk loader will over- > > > > write records using the IDs given in the .csv, which is what you want. > > > > > > 3 - Both tables are set with migrate=False, specifically b/c I've > > > > > seperated the data upload and process from the data reading process. > > > > > I'm not even sure what web2py would do if I removed the migrate=false > > > > > flag. What is the recommendation here? > > > > > AFAIK, migrate should have no effect on GAE. > > > > > > Inserting/deleting items from the GAE Datastore is very expensive in > > > > > CPU time, I messed with this yesterday and in a matter of a few > > > > > minutes (delete all, re-upload) was over my quota. Because of that > > > > > I'd really like to avoid having to change data in the Datastore as > > > > > much as I can. With 20K+ items you don't get a lot of 'do-overs'. > > > > > Have you enabled billing on your account? Enabiling billing greatly > > > > increases your free quota. > > > > > Robin > > > > > On Sep 19, 2:45 pm, Chris S <sanders.ch...@gmail.com> wrote: > > > > > > Background: > > > > > Working with an app on GAE currently I've run across a slight issue > > > > > with uploading data into my app. I initially tried to download my > > > > > tables to CSV from my development machine then use simply re-upload > > > > > them into GAE. It's been a while since I tried that but it was timing > > > > > out. > > > > > > Problem: > > > > > Because of that I decided to upload with bulkloader.py. I did my > > > > > first table (~2K entries) with out a hitch. My second Table is where > > > > > I'm having an issue. This table is about 24K entities. After > > > > > uploading all of the data trying to view it on the appadmin returns > > > > > the following error (web2py error ticket): > > > > > The table is "sqldb.item" if it matters. > > > > > ------------- > > > > > Traceback (most recent call last): > > > > > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/ > > > > > main.py", line 396, in wsgibase > > > > > session._try_store_in_db(request, response) > > > > > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/ > > > > > globals.py", line 342, in _try_store_in_db > > > > > table._db(table.id == record_id).update(**dd) > > > > > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/ > > > > > contrib/gql.py", line 698, in update > > > > > (item, tablename, fields) = self._getitem_exception() > > > > > File "/base/data/home/apps/aion-info/2.336452968225099838/gluon/ > > > > > contrib/gql.py", line 632, in _getitem_exception > > > > > item = self._db[tablename]._tableobj.get_by_id(long(id)) > > > > > File "/base/python_lib/versions/1/google/appengine/ext/db/ > > > > > __init__.py", line 943, in get_by_id > > > > > return get(*keys) > > > > > File "/base/python_lib/versions/1/google/appengine/ext/db/ > > > > > __init__.py", line 1123, in get > > > > > entities = datastore.Get(keys) > > > > > File "/base/python_lib/versions/1/google/appengine/api/ > > > > > datastore.py", line 224, in Get > > > > > apiproxy_stub_map.MakeSyncCall('datastore_v3', 'Get', req, resp) > > > > > File "/base/python_lib/versions/1/google/appengine/api/ > > > > > apiproxy_stub_map.py", line 72, in MakeSyncCall > > > > > apiproxy.MakeSyncCall(service, call, request, response) > > > > > File "/base/python_lib/versions/1/google/appengine/api/ > > > > > apiproxy_stub_map.py", line 255, in MakeSyncCall > > > > > rpc.CheckSuccess() > > > > > File "/base/python_lib/versions/1/google/appengine/api/ > > > > > apiproxy_rpc.py", line 111, in CheckSuccess > > > > > raise self.exception > > > > > CancelledError: The API call datastore_v3.Get() was explicitly > > > > > cancelled. > > > > > ----------------------------- > > > > > > Looking over two tables here are the things I've identified as > > > > > possible problems: > > > > > 1 - This table has a 'double' in it. Not just int/string like the > > > > > working one. For GAE uploading I defined the double field as a GAE > > > > > "float" property. Is it possible web2py defines a 'double' as > > > > > something else on GAE? If so is there a mapping that would tell me > > > > > how to map web2py fields into GAE fields for uploading data? > > > > > > 2 - Searching the Data Viewer on GAE shows that due to a previously > > > > > failed data upload I now have multiple copies of some of the > > > > > entities. These copies the exact same data in the except for the ID/ > > > > > Name fields. I would like to remove the duplicates sometime but that > > > > > wasn't my first concern. Would duplicate copies cause this problem? > > > > > I figured since the ID is unique it would just show up as another > > > > > entery in the table. I haven't set any of the entries to 'unique'. > > > > > > 3 - Both tables are set with migrate=False, specifically b/c I've > > > > > seperated the data upload and process from the data reading process. > > > > > I'm not even sure what web2py would do if I removed the migrate=false > > > > > flag. What is the recommendation here? > > > > > > Inserting/deleting items from the GAE Datastore is very expensive in > > > > > CPU time, I messed with this yesterday and in a matter of a few > > > > > minutes > > ... > > read more » --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "web2py-users" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~----------~----~----~----~------~----~------~--~---
