Thank Robin for the insight. I guess you are saying: Although writing only one record is somewhat slow in GAE, writing multiple (dozens/ hundreds/thousands/or even more) records can be finished in similar time due to the scalability of GAE, if using the batch get/put/delete properly. That is cool. And hopefully web2py's GQLDB will have built- in batch operation soon. Only then, web2py fully deserves the reputation which "write same code once, then the application can run both with or without GAE".
On the other hand, I think using bulk uploader is helpful but not an elegant way from web2py's point of view, because it forces web2py developers to define a duplicated data model in both web2py and in bulk load handler, via different syntax. So I perfer to solve this problem in web2py's way, perhaps a functional admin app with its csv import function will do. Besides, batch delete is equally important. Currently (without batch delete yet) web2py on GAE can do 80~100 deletion per second, that is not fast enough if you are going to do a db reset on GAE which invokes thousands of records. By the way, if I upload only one web2py as my "GAE application" (such as web2py.appspot.com), then I upload/deploy many "web2py applications" (say welcome, demo, t3), are they sharing the same Google DB name space therefore we must be careful not to have same table name? On Jan 28, 12:32 am, Robin B <robi...@gmail.com> wrote: > Hey, > > > Did I miss something? Do you guys encounter same problem when playing > > with GAE? Does your application successfully insert hundreds of > > records during one request? Any feedback will be highly appreciated! > > On GAE, writes are very expensive because each record is actually > written to disk, indexes are updated and written to disk and all this > happens synchronously, so it takes a long time to serially write > records. > > For GAE, take a look at batch get,put,delete which let you do > operations in parallel as opposed to serially. These batch operations > will be added to GQLDB eventually. > > Either way, to prepopulate a db on GAE, you have to do it over a > series of requests, because each request can only be 10 seconds max. > See bulk uploader:http://code.google.com/appengine/articles/bulkload.html > > Robin > > On Jan 27, 10:13 am, Iceberg <iceb...@21cn.com> wrote: > > > Hi pals, > > > Don't know whether this is a web2py problem or a GAE problem, anyway I > > still post it here and hope to get some kindly hints. > > > I started my GAE experience just today. I managed to upload web2py > > 1.55.2 and its welcome application and later one of my homebrew "no-db- > > required" application, all of them work fine. Then I tried to port my > > another "serious" application to GAE. The application requires some > > initial data (20000+ records, about 500KB total) before it can do some > > real job. > > > In my local developing environment, I can easily setup an SQLite db, > > then visit something like "/myapp/default/putDataIntoSQLDB" to trigger > > the data initialization without any problem. > > > But when talking about GAE, I guess there is no way to supply these > > data in a predefined storage.db file, probably because GAE doesn't > > support file operation, am I correct? In fact, I tried defining > > "db=SQLDB('sqlite://storage.db')" in my db.py but it caused an error > > in GAE: > > NameError: global name 'sqlite3' is not defined > > > So I used "db=GQLDB()" instead. Then the applications can start, BUT, > > when I visit "/myapp/default/putDataIntoGQLDB", my application ALWAYS > > results in "DeadlineExceededError" after about 8 seconds of execution, > > during this time only several dozens of records are inserted into > > GQLDB. If this is because I called db.mytable.insert(...) for too many > > times, may I have some other way to do batch insert()? > > > By the way, later when I tried to reset the faulty db data, I found > > out that there are more than 50% chances that GAE complains: > > Timeout: datastore timeout: operation took too long. > > even when I simply invoke: > > db(db.mytable.id>0).delete() > > > I search the web and found some rumors, such as: > > http://highscalability.com/google-appengine-second-look > > But I still can't believe the performance is as bad as what I > > mentioned above. > > > Did I miss something? Do you guys encounter same problem when playing > > with GAE? Does your application successfully insert hundreds of > > records during one request? Any feedback will be highly appreciated! > > > Sincerely, > > Iceberg --~--~---------~--~----~------------~-------~--~----~ You received this message because you are subscribed to the Google Groups "web2py Web Framework" group. To post to this group, send email to web2py@googlegroups.com To unsubscribe from this group, send email to web2py+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/web2py?hl=en -~----------~----~----~----~------~----~------~--~---
