Yep good point Carl. I could see staving off the issue a little longer by altering the script to backup one table per file. I believe the pg_dump utility lets you specify the table to dump. Ultimately, we'd run into the same issue though. I guess I'll figure this out when I start getting close to the GB limit :)
You could give some derivation of the pg_restore task a shot along with your bundle/extract backup, but I suppose you'd likely have the same timeout issue. Jason On May 12, 8:49 pm, Mike <[email protected]> wrote: > This looks really great, thanks for sharing. > > Only thing I see that I'd be concerned about are reliability on larger > databases. > > You have a step where it's downloading the image from S3 onto the > Heroku node. If you have a database that's more than a GB, I wonder if > you could run into any problems there. > > Also, I'd previously used a solution where I use pg_dump for my > backups. Unfortunately, I ran into a problem there where my database > (about 1GB) was taking longer than the Heroku database query execution > limit, and failing. Normal commands to increase the query execution > timeout don't work for pg_dump since it runs in a separate > environment. I conferred with Heroku's customer service about this, > but there didn't seem to be a rectify this problem. > > The way I ended up doing my backups is to create a cron job that > deletes my old bundle, and creates a new bundle. Bundles contain > within them a pg_dump, but run in a special environment I guess that > isn't subject to the limits. I then have the cron job push that to > Amazon. > > Unfortunately, restores are much more of a pain, and I would have to > wipe the Heroku database, download the bundle locally, pg_restore it > locally, and then push it up with taps. > > On May 10, 8:50 pm, Carl Anderson <[email protected]> wrote: > > > > > Excellent to hear. > > > Carl > > > On Mon, May 10, 2010 at 5:48 PM, Jason <[email protected]> wrote: > > > I sure have. I've run it to transfer between environments and as a > > > cron. It will be running nightly for me from now on. > > > > Jason > > > > On May 10, 7:43 pm, Carl Anderson <[email protected]> wrote: > > > > That looks pretty helpful. Have you tested it with any production > > > datasets > > > > yet? > > > > > Carl > > > > > On Mon, May 10, 2010 at 5:17 PM, Jason <[email protected]> wrote: > > > > > Just wanted to let you guys/girls know that I've created a project up > > > > > on github with backup and restore tasks using pg_dump. I've wanted > > > > > this for a long time, and I figured some of you might too. I've seen > > > > > a lot of backup solutions (which I've borrowed heavily from), but very > > > > > few restores! I could just imagine scrambling to restore the > > > > > database, having not tested it before. > > > > > > Anyway, hope some people get some use out of this. > > > > > >http://github.com/jpearl/heroku_backups > > > > > > From the readme: > > > > > > These rake tasks are basically a cleaned up and consolidated version > > > > > of everything good I could find related to backup of heroku > > > > > databases. It essentially takes advantage of pg_dump and pg_restore > > > > > capailities of postgres. > > > > > > You can use it to run a nightly backup cron, or to transfer between > > > > > environments. > > > > > > heroku rake db:backup --app from_application > > > > > heroku rake db:restore -app to_application > > > > > > Notes: > > > > > * You'll need to place your backup bucket name where it indicates in > > > > > database.rake > > > > > * You will need aws_s3 and yaml_db, since we're dumping both a > > > > > database agnostic yaml file and a pg_dump compressed > > > > > backup > > > > > > Things that could be done: > > > > > * Most of you may not care about the yaml_db portion. I like it > > > > > because I use mysql locally, even though I'm running on postgres at > > > > > Heroku. You could easily rip this part out. > > > > > > Hope this helps some folks. > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > Groups > > > > > "Heroku" group. > > > > > To post to this group, send email to [email protected]. > > > > > To unsubscribe from this group, send email to > > > > > [email protected]<heroku%[email protected]> > > > <heroku%[email protected]<heroku%[email protected]> > > > > > > . > > > > > For more options, visit this group at > > > > >http://groups.google.com/group/heroku?hl=en. > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups > > > "Heroku" group. > > > > To post to this group, send email to [email protected]. > > > > To unsubscribe from this group, send email to > > > [email protected]<heroku%[email protected]> > > > . > > > > For more options, visit this group athttp:// > > > groups.google.com/group/heroku?hl=en. > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "Heroku" group. > > > To post to this group, send email to [email protected]. > > > To unsubscribe from this group, send email to > > > [email protected]<heroku%[email protected]> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/heroku?hl=en. > > > -- > > You received this message because you are subscribed to the Google Groups > > "Heroku" group. > > To post to this group, send email to [email protected]. > > To unsubscribe from this group, send email to > > [email protected]. > > For more options, visit this group > > athttp://groups.google.com/group/heroku?hl=en. > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]. > For more options, visit this group > athttp://groups.google.com/group/heroku?hl=en. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
