Re: Inherited an 18TB DB & need to backup

Thanks James... that is reassuring, will be working on it this week. On Mon, 18 May 2020, 04:33 James Sewell, wrote: > > > On Fri, 15 May 2020 at 17:09, Suhail Bamzena > wrote: > >> Hello All, >> I have very recently inherited an 18 TB DB that is running version 9.2. >> Apparently this database

Re: Inherited an 18TB DB & need to backup

On Fri, 15 May 2020 15:08:05 +0200 Christoph Berg wrote: > Re: Rory Campbell-Lange > > On 15/05/20, Suhail Bamzena (suhailsa...@gmail.com) wrote: > > > Hello All, > > > I have very recently inherited an 18 TB DB that is running > > > version 9.2. > > Push hard to get that upgraded to a suppo

Re: Inherited an 18TB DB & need to backup

On Fri, 15 May 2020 at 17:09, Suhail Bamzena wrote: > Hello All, > I have very recently inherited an 18 TB DB that is running version 9.2. > Apparently this database has never been backed up and I have been tasked to > set in a periodic backup routine (weekly full & daily incremental) and dump >

Re: Inherited an 18TB DB & need to backup

On 5/16/20 3:30 PM, Gavin Flower wrote: On 17/05/2020 08:12, Ron wrote: On 5/16/20 7:18 AM, Rob Sargent wrote: O Another problem is storage devices fail.  S3 storage lakes _should_ be checking your data integrity on a regular basis and possibly maintaining copies of it iin multiple locations

Re: Inherited an 18TB DB & need to backup

On 17/05/2020 08:12, Ron wrote: On 5/16/20 7:18 AM, Rob Sargent wrote: O Another problem is storage devices fail.  S3 storage lakes _should_ be checking your data integrity on a regular basis and possibly maintaining copies of it iin multiple locations so you're not vulnerable to a site disas

Re: Inherited an 18TB DB & need to backup

On 5/16/20 7:18 AM, Rob Sargent wrote: O Another problem is storage devices fail. S3 storage lakes _should_ be checking your data integrity on a regular basis and possibly maintaining copies of it iin multiple locations so you're not vulnerable to a site disaster. Tape FTW!! Or WTF Tape??

Re: Inherited an 18TB DB & need to backup

Hi Peter Thanks for the info & the entire forum for their inputs i did fireup a pg_dump last night pairing it with gzip & split it to 1TB size.. will let you all know how it goes. On Sat, 16 May 2020, 18:12 Peter J. Holzer, wrote: > On 2020-05-15 14:02:46 +0100, Rory Campbell-Lange wrote:

Re: Inherited an 18TB DB & need to backup

On 2020-05-15 14:02:46 +0100, Rory Campbell-Lange wrote: > On 15/05/20, Suhail Bamzena (suhailsa...@gmail.com) wrote: > > I have very recently inherited an 18 TB DB that is running version 9.2. > > Apparently this database has never been backed up [...] > A very simple solution could be just to dum

Re: Inherited an 18TB DB & need to backup

> O >> Another problem is storage devices fail. S3 storage lakes _should_ be >> checking your data integrity on a regular basis and possibly maintaining >> copies of it iin multiple locations so you're not vulnerable to a site >> disaster. > > Tape FTW!! > > -- Or WTF Tape?? :) > Angula

Re: Inherited an 18TB DB & need to backup

On 5/15/20 6:01 PM, Michael Nolan wrote: On Fri, May 15, 2020 at 12:51 PM Ravi Krishna > wrote: Why should the backup land in S3, and not local somewhere? Any good reason why one should pay for the additional storage and transfer costs? Good q

Re: Inherited an 18TB DB & need to backup

Thanks Ron.. pgbackrest and barman seem to b good options.. On Sat, 16 May 2020, 02:26 Ron, wrote: > For a database that size, I'd install pgbackrest, since it features > parallel backups and compression. With it, I'd do monthly full backups > with daily differential backups. > > (If it's mostl

Re: Inherited an 18TB DB & need to backup

On Fri, May 15, 2020 at 12:51 PM Ravi Krishna wrote: > > Why should the backup land in S3, and not local somewhere? > Any good reason why one should pay for the additional storage and transfer > costs? > > Good question. The key point in my statement was "db of this size". > > The problem with lo

Re: Inherited an 18TB DB & need to backup

For a database that size, I'd install pgbackrest, since it features parallel backups and compression.  With it, I'd do monthly full backups with daily differential backups. (If it's mostly historical data, I'd split the database into multiple instances, so that older data rarely needs to be ba

Re: Inherited an 18TB DB & need to backup

> On May 15, 2020, at 12:01, Scottix wrote: > > Also when you get in the multi TB data storage the bill gets a little harder > to digest in S3. Indeed. Right now, just buying off of Amazon, a 12TB Seagate IronWolf drive is $0.03/GB. S3 infrequent access is $0.0125/GB/month, so the drive p

Re: Inherited an 18TB DB & need to backup

Also when you get in the multi TB data storage the bill gets a little harder to digest in S3. On Fri, May 15, 2020 at 11:49 Andreas 'ads' Scherbaum wrote: > > > On Fri, May 15, 2020 at 7:52 PM Ravi Krishna > wrote: > >> >> Why should the backup land in S3, and not local somewhere? >> Any good r

Re: Inherited an 18TB DB & need to backup

On Fri, May 15, 2020 at 7:52 PM Ravi Krishna wrote: > > Why should the backup land in S3, and not local somewhere? > Any good reason why one should pay for the additional storage and transfer > costs? > > Good question. The key point in my statement was "db of this size". > > The problem with loc

Re: Inherited an 18TB DB & need to backup

Why should the backup land in S3, and not local somewhere? Any good reason why one should pay for the additional storage and transfer costs? Good question. The key point in my statement was "db of this size". The problem with local backup is that space is not infinite. If your business requ

Re: Inherited an 18TB DB & need to backup

On Fri, May 15, 2020 at 3:49 PM Ravi Krishna wrote: > IMO a database of this size should only be backed up in s3. pgbackrest > has support for backup to s3. > Why should the backup land in S3, and not local somewhere? Any good reason why one should pay for the additional storage and transfer co

Re: Inherited an 18TB DB & need to backup

Thanks Gabriele... i will review this in detail. On Fri, 15 May 2020, 18:41 Gabriele Bartolini, wrote: > Hi Suhail, > > We developed Barman (www.pgbarman.org), which works with older versions > of PostgreSQL too. It works with very large databases and is open source > (GPL). > > This is an old a

Re: Inherited an 18TB DB & need to backup

Hi Suhail, We developed Barman (www.pgbarman.org), which works with older versions of PostgreSQL too. It works with very large databases and is open source (GPL). This is an old article in which I could publicly mention the size of the database: https://www.2ndquadrant.com/en/blog/incremental-ba

Re: Inherited an 18TB DB & need to backup

IMO a database of this size should only be backed up in s3. pgbackrest has support for backup to s3.

Re: Inherited an 18TB DB & need to backup

Yeah Rory want to pull one asap..hopefully by COB tonight.. On Fri, 15 May 2020, 17:35 Rory Campbell-Lange, wrote: > On 15/05/20, Suhail Bamzena (suhailsa...@gmail.com) wrote: > > Thanks Rory, the machine has the capacity to pull through pg_dumps but > like > > u rightly mentioned incremental ba

Re: Inherited an 18TB DB & need to backup

On 15/05/20, Suhail Bamzena (suhailsa...@gmail.com) wrote: > Thanks Rory, the machine has the capacity to pull through pg_dumps but like > u rightly mentioned incremental backups mean that we will need to work with > the wal's.. 18TB is what is the scary part and with compression I dont see > it be

Re: Inherited an 18TB DB & need to backup

Thanks Jan.. would appreciate all the info I can get. On Fri, 15 May 2020, 17:32 Jan Karremans, wrote: > Hi Suhail, > > I do not know personally, but I can figure out / get you setup with > someone who can tell you more. > > Mit freundlichem Gruß, kind regards, > > > > > *Jan Karremans*Director

Re: Inherited an 18TB DB & need to backup

l Bamzena > *Sent:* Friday, May 15, 2020 7:23 AM > *To:* Christoph Berg > *Cc:* Rory Campbell-Lange ; > pgsql-general@lists.postgresql.org; pgeu-gene...@lists.postgresql.org > *Subject:* EXTERNAL: Re: Inherited an 18TB DB & need to backup > > > > Hi Christoph > >

Re: Inherited an 18TB DB & need to backup

Hi Suhail, I do not know personally, but I can figure out / get you setup with someone who can tell you more. Mit freundlichem Gruß, kind regards, Jan Karremans Director of Sales Engineering, EMEA Senior Sales Engineer DACH-Region EDB Postgres Advanced Server Professional -- Postgres Everywhe

Re: Inherited an 18TB DB & need to backup

Thanks Jan.. just to know more is it using the native pg_dump or another tool like pgbackrest or barman ?? On Fri, 15 May 2020, 17:26 Jan Karremans, wrote: > Hi Suhail, > > That is not an issue. We have customers backing up Postgres databases up > to 80 TB. > > Mit freundlichem Gruß, kind regard

Re: Inherited an 18TB DB & need to backup

Thanks Rory, the machine has the capacity to pull through pg_dumps but like u rightly mentioned incremental backups mean that we will need to work with the wal's.. 18TB is what is the scary part and with compression I dont see it being less than 2TB a day... On Fri, 15 May 2020, 17:02 Rory Campbel

Re: Inherited an 18TB DB & need to backup

...@lists.postgresql.org Subject: EXTERNAL: Re: Inherited an 18TB DB & need to backup Hi Christoph Thats very high on my agenda.. but need to make sure i can backup this beast to start with.. On Fri, 15 May 2020, 17:08 Christoph Berg, mailto:christoph.b...@credativ.de>> wrote: Re: Rory Camp

Re: Inherited an 18TB DB & need to backup

Hi Suhail, That is not an issue. We have customers backing up Postgres databases up to 80 TB. Mit freundlichem Gruß, kind regards, Jan Karremans Director of Sales Engineering, EMEA Senior Sales Engineer DACH-Region EDB Postgres Advanced Server Professional -- Postgres Everywhere -- Oracle A

Re: Inherited an 18TB DB & need to backup

Hi Christoph Thats very high on my agenda.. but need to make sure i can backup this beast to start with.. On Fri, 15 May 2020, 17:08 Christoph Berg, wrote: > Re: Rory Campbell-Lange > > On 15/05/20, Suhail Bamzena (suhailsa...@gmail.com) wrote: > > > Hello All, > > > I have very recently inherit

Re: Inherited an 18TB DB & need to backup

Re: Rory Campbell-Lange > On 15/05/20, Suhail Bamzena (suhailsa...@gmail.com) wrote: > > Hello All, > > I have very recently inherited an 18 TB DB that is running version 9.2. Push hard to get that upgraded to a supported version. Christoph

Re: Inherited an 18TB DB & need to backup

On 15/05/20, Suhail Bamzena (suhailsa...@gmail.com) wrote: > Hello All, > I have very recently inherited an 18 TB DB that is running version 9.2. > Apparently this database has never been backed up and I have been tasked to > set in a periodic backup routine (weekly full & daily incremental) and du

Inherited an 18TB DB & need to backup

Hello All, I have very recently inherited an 18 TB DB that is running version 9.2. Apparently this database has never been backed up and I have been tasked to set in a periodic backup routine (weekly full & daily incremental) and dump it into a NAS. What is the best way to go about this? Did some r