Re: Rearchitecting for storage

2019-07-19 Thread Matthew Pounsett
On Fri, 19 Jul 2019 at 11:25, Peter J. Holzer wrote: > On 2019-07-19 10:41:31 -0400, Matthew Pounsett wrote: > > Okay. So I guess the short answer is no, nobody really knows how to > > judge how much space is required for an upgrade? :) > > As I understand it, a pg_upg

Re: Rearchitecting for storage

2019-07-19 Thread Matthew Pounsett
On Thu, 18 Jul 2019 at 09:44, Matthew Pounsett wrote: > > I've recently inherited a database that is dangerously close to outgrowing > the available storage on its existing hardware. I'm looking for (pointers > to) advice on scaling the storage in a financially constr

Re: Rearchitecting for storage

2019-07-19 Thread Matthew Pounsett
On Fri, 19 Jul 2019 at 04:21, Luca Ferrari wrote: > > This could be trivial, but any chance you can partition the table > and/or archive unused records (at least temporarly)? A 18 TB table > quite frankly sounds a good candidate to contain records no one is > interested in the near future. > Par

Re: Rearchitecting for storage

2019-07-19 Thread Matthew Pounsett
On Thu, 18 Jul 2019 at 21:59, Andy Colson wrote: > > > > Now might be a good time to consider splitting the database onto multiple > computers. Might be simpler with a mid-range database, then your plan for > the future is "add more computers". > Hmm... yes. Range partitioning seems like a pos

Re: Rearchitecting for storage

2019-07-19 Thread Matthew Pounsett
On Thu, 18 Jul 2019 at 19:53, Rob Sargent wrote: > > > > > That would likely keep the extra storage requirements small, but still > non-zero. Presumably the upgrade would be unnecessary if it could be done > without rewriting files. Is there any rule of thumb for making sure one > has enough sp

Re: Rearchitecting for storage

2019-07-18 Thread Matthew Pounsett
On Thu, 18 Jul 2019 at 13:34, Kenneth Marshall wrote: > Hi Matt, > Hi! Thanks for your reply. > Have you considered using the VDO compression for tables that are less > update intensive. Using just compression you can get almost 4X size > reduction. For a database, I would forgo the deduplica

Rearchitecting for storage

2019-07-18 Thread Matthew Pounsett
I've recently inherited a database that is dangerously close to outgrowing the available storage on its existing hardware. I'm looking for (pointers to) advice on scaling the storage in a financially constrained not-for-profit. The current size of the DB's data directory is just shy of 23TB. Whe

Re: [External] LIMIT not showing all results

2019-03-05 Thread Matthew Pounsett
On Tue, 5 Mar 2019 at 18:39, Adrian Klaver wrote: > > > > The replication documentation, and more specifically the pg_basebackup > > documentation, makes no mention of cross-OS replication as being a > > problem for any reason. If that is expected to be a problem, then > > perhaps that should be

Re: [External] LIMIT not showing all results

2019-03-05 Thread Matthew Pounsett
On Tue, 5 Mar 2019 at 18:09, Tom Lane wrote: > > If you're planninng to install (the same version of) FreeBSD on > the original server hardware, then rsync'ing back from the new > system should be fine. But Debian<->FreeBSD is gonna be trouble > in either direction. > But I'm specifically NOT t

Re: [External] LIMIT not showing all results

2019-03-05 Thread Matthew Pounsett
On Tue, 5 Mar 2019 at 13:55, Tom Lane wrote: > > Yeah, that would fit the theory :-(. Debian would be using glibc > and FreeBSD would not be. If you were using C collation in the > database, you'd be all right because that's standardized, but I'll > bet you were using something else. What does

Re: [External] LIMIT not showing all results

2019-03-05 Thread Matthew Pounsett
On Tue, 5 Mar 2019 at 12:54, Tom Lane wrote: > Andreas Kretschmer writes: > > the other thing is, it would be nice to to know why the index is corrupt. > > Given that (a) this was triggered by a server migration and (b) > the leading column of the index looks like it's probably varchar, > I'm su