Essentially you need to buy time for the data to fully move itself from App Engine servers to Cloud Storage servers. You also need to hint to App Engine to move the data in smaller chunks.
There's a couple of ways to accomplish this. In a previous email you said that you're using the waitForOutstandingWrites method every 100 records. Try using it more frequently, and in combination, decreasing the size of the ByteBuffer being written to GCS. If you feel comfortable using the JSON API, what you can also try is writing multiple smaller files, then using compose<https://developers.google.com/storage/docs/json_api/v1/objects/compose>to combine them into a single object (or just leave them as smaller files, there's nothing wrong with that). ----------------- -Vinny P Technology & Media Advisor Chicago, IL App Engine Code Samples: http://www.learntogoogleit.com On Tue, Apr 15, 2014 at 8:58 AM, Evan Ruff <[email protected]>wrote: > Thanks for the very detailed response Vinny. I am attempting to alleviate > the issue with mixed results by clearing out the objectify cache every few > runs. While this has helped a little bit, I am still seeing all sorts of > weirdness with it. > > Could you go into more detail about allocating new URLFetches? While I > understand I can use the builder to fine tune the options, I don't see how > that could be utilized with the GCS implementation? > -- You received this message because you are subscribed to the Google Groups "Google App Engine" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/google-appengine. For more options, visit https://groups.google.com/d/optout.
