For the import part if it has not to be "atomic" you could create another app and use it to feed via a web service the data on the production database while the main app is live.
To download only the main dataset create a boundle, animate it on a new application, delete the huge data with a migration or with a sql query in the console and the pull the database locally. By the way I'm investigating about how to work with big db too. Coming from a standard server world I love Heroku but some steps could be a bit less flexible. Nothing is perfect :) On 12 Mar, 07:29, Mike <[email protected]> wrote: > I'm going to be adding a number of discrete, but enormous (maybe many > gigs each), datasets to my Heroku app's database. In many ways, I'm > in a similar situation faced by Tobin in another current post, but > with a different > question:http://groups.google.com/group/heroku/browse_thread/thread/141c3ef84b... > > Right now I still haven't merged the datasets into my database yet. > What's the best way for me to approach this? > > The lack of ability to push individual tables with taps suggests to me > I'm going to want to do this probably as a one shot deal, rather than > doing each dataset sequentially and testing that one before proceeding > to the next. I'm thinking about doing a db:pull to get the current > state of my database, and then shutting down my application in > maintenance mode, running a local merge of the datasets (maybe taking > days I'm guessing just to process the enormous things), doing some > exhaustive local testing on the result, and then doing a push back to > Heroku (maybe taking days again), before reactivating my app. Because > of their massive size, it seems like after I've done one, doing any > further db:pulls is going to be basically impossible. Just the idea > of possibly having made a mistake in merging the datasets that I don't > catch until after it's been pushed to the site gives me the shivers. > Overall, I wonder if there could be a better way that I'm overlooking. > > One possible alternative I thought of is would it be possible to do > something involving creating a local bundle from my database using > YamlDB? But then I'm not sure how to get the bundle back onto the > server and then to restore from it? The documentation on Heroku > doesn't seem to really talk about that possibility. > > Also, in my case this data is integral to the application, so I'm not > going to be able to split it up into a separate Heroku application > like in Tobin's case. Is there going to be any practical way for me > to be pulling just the non-dataset data from the server in order to > use on a development machine? > > Does anyone have any ideas on how they would approach this problem? > If so, I'd be filled with gratitude. > > Mike -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
