Re: Multitenent architecture

2020-06-05 Thread Rob Sargent
> On Jun 5, 2020, at 9:57 PM, Vasu Madhineni wrote: > >  > Hi Rob, > > Our environment is medical clinical data, so each clinic as a tenant. > Approximately 500+ tenants with 6TB data. > . How quickly are both those numbers growing? What are your cross clinic query requirements ? Any sin

Logical replication - ERROR: could not send data to WAL stream: cannot allocate memory for input buffer

2020-06-05 Thread Aleš Zelený
Hello, we are using logical replication for more than 2 years and today I've found new not yet know error message from wal receiver. The replication was in catchup mode (on publisher side some new tables were created and added to publication, on subscriber side they were missing). RDBMS version:P

Re: Logical Replication and table bloat

2020-06-05 Thread Peter Eisentraut
On 2020-06-05 21:53, Martín Fernández wrote: Yesterday we stumbled upon a performance issue that we were not expecting. We are replicating our database using AWS DMS which uses logical replication to capture changes. We have some hot tables that get updated very regularly and with the DMS turn

Re: Possible improvement

2020-06-05 Thread Paul van der Linden
Ok, as always there's a lot more to take into account then when just superficially looking at it. And indeed your counterexample shows that you'd have to include all the previous when-conditions too as false WHERE x=0 IS DISTINCT FROM true AND 1/x > 100, which could become quite messy (especially w

Re: Possible improvement

2020-06-05 Thread Paul van der Linden
Thanks for your thoughts. For the case where it isn't known if the case expression itself is indexed, technically that should be added as a decision-node in the query planner. After all there are 2 possibilities to handle that so it should be up to the planner to choose the cheapest. Having said

Re: Logical Replication and table bloat

2020-06-05 Thread Adrian Klaver
On 6/5/20 12:53 PM, Martín Fernández wrote: Hello, Yesterday we stumbled upon a performance issue that we were not expecting. We are replicating our database using AWS DMS which uses logical replication to capture changes. We have some hot tables that get updated very regularly and with the D

Re: Oracle vs. PostgreSQL - a comment

2020-06-05 Thread Ron
On 6/5/20 10:02 AM, Achilleas Mantzios wrote: On 5/6/20 5:19 μ.μ., Thomas Kellerer wrote: Achilleas Mantzios schrieb am 05.06.2020 um 14:05: Plus PG does not directly support cross database queries using 3 part name, something sqlserver excels at. Maybe because SQL server does not have real d

Logical Replication and table bloat

2020-06-05 Thread Martín Fernández
Hello, Yesterday we stumbled upon a performance issue that we were not expecting. We are replicating our database using AWS DMS which uses logical replication to capture changes. We have some hot tables that get updated very regularly and with the DMS turned on we started noticing that in those

Re: Index no longer being used, destroying and recreating it restores use.

2020-06-05 Thread Michael Lewis
Those row estimates are pretty far off. Standard indexes and partial indexes don't get custom statistics created on them, but functional indexes do. I wonder if a small function needs_backup( shouldbebackedup, backupperformed ) and an index created on that function would nicely alleviate the pain.

Re: Possible improvement

2020-06-05 Thread Tom Lane
Paul van der Linden writes: > For the case where it isn't known if the case expression itself is indexed, > technically that should be added as a decision-node in the query planner. That'd be fairly hard to do, if we're regarding this as an expression simplification step, since expression simplif

Re: Postgres12 - Confusion with pg_restore

2020-06-05 Thread Tom Lane
Laura Smith writes: > But doesn't the second half of my original post demonstrate that I tried that > very thing ? I did try creating the database first, but pg_restore just > complained even more ? There are two ways you can do this: 1. Create the new database by hand (with CREATE DATABASE)

Re: Postgres12 - Confusion with pg_restore

2020-06-05 Thread Laura Smith
Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Friday, 5 June 2020 19:23, Christophe Pettus wrote: > > On Jun 5, 2020, at 11:20, Laura Smith n5d9xq3ti233xiyif...@protonmail.ch > > wrote: > > sudo -u postgres pg_restore -v -C -d foobar 4_foobar_pgdump_Fc > > You need

Re: Postgres12 - Confusion with pg_restore

2020-06-05 Thread Christophe Pettus
> On Jun 5, 2020, at 11:20, Laura Smith > wrote: > sudo -u postgres pg_restore -v -C -d foobar 4_foobar_pgdump_Fc You need to connect to a database that already exists (such as "postgres"); it then creates the database you are restoring and switches to it. The relevant manual line is:

Postgres12 - Confusion with pg_restore

2020-06-05 Thread Laura Smith
According to the all-mighty manual (https://www.postgresql.org/docs/current/app-pgrestore.html), life is supposed to be as simple as: "To drop the database and recreate it from the dump: $ dropdb mydb $ pg_restore -C -d postgres db.dump" The reality seems to be somewhat different ? sudo -u pos

Re: Index no longer being used, destroying and recreating it restores use.

2020-06-05 Thread Koen De Groote
The requested result: https://explain.depesz.com/s/G7mU Also, the data from the statistic itself: => SELECT stxname, stxkeys, stxdependencies -> FROM pg_statistic_ext -> WHERE stxname = 's1'; stxname | stxkeys | stxdependencies -+-+- s1 | 29 35 |

Re: Binary encoding of TIMESTAMP WITH TIME ZONE

2020-06-05 Thread Peter J. Holzer
On 2020-06-04 20:32:51 -0400, Tom Lane wrote: > Joe Abbate writes: > > However, when using the same query using the Rust adapter the transition > > to a new row started showing up after midgnight GMT. I opened an issue > > on Github (https://github.com/sfackler/rust-postgres/issues/608 ) and >

Re: Oracle vs. PostgreSQL - a comment

2020-06-05 Thread Achilleas Mantzios
On 5/6/20 5:19 μ.μ., Thomas Kellerer wrote: Achilleas Mantzios schrieb am 05.06.2020 um 14:05: Plus PG does not directly support cross database queries using 3 part name, something sqlserver excels at. Maybe because SQL server does not have real databases but schemas instead ? This sucks secur

Re: Possible improvement

2020-06-05 Thread Tom Lane
David Rowley writes: > On Fri, 5 Jun 2020 at 14:41, Paul van der Linden > wrote: >> If I have a query like: >> >> SELECT * FROM ( >> SELECT >> CASE >> WHEN field='value1' THEN 1 >> WHEN field='value2' THEN 2 >> END AS category >> FROM table1 >> ) AS foo >> WHERE category=1 >> >> doesn't use the

Re: Oracle vs. PostgreSQL - a comment

2020-06-05 Thread Thomas Kellerer
Achilleas Mantzios schrieb am 05.06.2020 um 14:05: >> Plus PG does not directly support cross database queries using 3 part name, >> something >> sqlserver excels at. > > Maybe because SQL server does not have real databases but schemas instead ? > This sucks security wise. That is wrong. SQL Se

Re: Index no longer being used, destroying and recreating it restores use.

2020-06-05 Thread Adrian Klaver
On 6/5/20 7:05 AM, Koen De Groote wrote: I've collected all relevant info(I think so at least) and put it here: The table in question is used to keep filepath data, of files on a harddrive. The query in question is used to retrieve items which should be backed up, but have not yet been. The

Re: Index no longer being used, destroying and recreating it restores use.

2020-06-05 Thread Koen De Groote
I've collected all relevant info(I think so at least) and put it here: The table in question is used to keep filepath data, of files on a harddrive. The query in question is used to retrieve items which should be backed up, but have not yet been. The relevant columns of the table:

Re: When to use PARTITION BY HASH?

2020-06-05 Thread Jeff Janes
On Fri, Jun 5, 2020 at 6:12 AM Oleksandr Shulgin < oleksandr.shul...@zalando.de> wrote: > On Thu, Jun 4, 2020 at 4:32 PM Jeff Janes wrote: > >> On Wed, Jun 3, 2020 at 7:55 AM Oleksandr Shulgin < >> oleksandr.shul...@zalando.de> wrote: >> >> With hash partitioning you are not expected, in general,

Re: Oracle vs. PostgreSQL - a comment

2020-06-05 Thread Achilleas Mantzios
On 5/6/20 3:33 μ.μ., Ravi Krishna wrote: Plus PG does not directly support cross database queries using 3 part name, something sqlserver excels at. Maybe because SQL server does not have real databases but schemas instead ? This sucks security wise. SQLServer has real databases with its own tr

Re: Oracle vs. PostgreSQL - a comment

2020-06-05 Thread Ravi Krishna
>> Plus PG does not directly support cross database queries using 3 part name, >> something >> sqlserver excels at. >Maybe because SQL server does not have real databases but schemas instead ? >This sucks security wise. SQLServer has real databases with its own transaction log files. You can r

Re: Oracle vs. PostgreSQL - a comment

2020-06-05 Thread Achilleas Mantzios
On 2/6/20 10:45 μ.μ., Ravi Krishna wrote: Generally speaking, I discourage having lots of databases under one PG cluster for exactly these kinds of reasons. PG's individual clusters are relatively lightweight, after all. Plus PG does not directly support cross database queries using 3 part nam

Re: Potential optimisation for the creation of a partial index condition over a newly created nullable, non-default-valued column?

2020-06-05 Thread David Rowley
On Thu, 4 Jun 2020 at 17:59, Tim Dawborn wrote: > tmp=> \timing on > Timing is on. > tmp=> BEGIN; > BEGIN > Time: 1.333 ms > tmp=> ALTER TABLE foo ADD COLUMN d integer NULL; > ALTER TABLE > Time: 1.581 ms > tmp=> CREATE UNIQUE INDEX myindex ON foo (a, b, c) where d = 2; > CREATE INDEX > Time: 3775

Re: Multitenent architecture

2020-06-05 Thread Ravi Krishna
> > If the data size is more than 6TB, which approach better? Do you require cross tenants queries? If yes, then schemas are a better solution.

Re: When to use PARTITION BY HASH?

2020-06-05 Thread Oleksandr Shulgin
On Thu, Jun 4, 2020 at 4:32 PM Jeff Janes wrote: > On Wed, Jun 3, 2020 at 7:55 AM Oleksandr Shulgin < > oleksandr.shul...@zalando.de> wrote: > > With hash partitioning you are not expected, in general, to end up with a >> small number of partitions being accessed more heavily than the rest. So >

Re: Multitenent architecture

2020-06-05 Thread Rob Sargent
> On Jun 5, 2020, at 2:54 AM, Vasu Madhineni wrote: > >  > If the data size is more than 6TB, which approach better? > >> On Fri, Jun 5, 2020 at 2:57 PM Laurenz Albe wrote: >> On Thu, 2020-06-04 at 23:52 +0800, Vasu Madhineni wrote: >> > We are planning a POC on multitenant architecture in P

Re: Multitenent architecture

2020-06-05 Thread Vasu Madhineni
If the data size is more than 6TB, which approach better? On Fri, Jun 5, 2020 at 2:57 PM Laurenz Albe wrote: > On Thu, 2020-06-04 at 23:52 +0800, Vasu Madhineni wrote: > > We are planning a POC on multitenant architecture in Postgres, Could you > please > > help us with steps for multitenant usi

Re: Possible improvement

2020-06-05 Thread David Rowley
On Fri, 5 Jun 2020 at 14:41, Paul van der Linden wrote: > If I have a query like: > > SELECT * FROM ( > SELECT > CASE > WHEN field='value1' THEN 1 > WHEN field='value2' THEN 2 > END AS category > FROM table1 > ) AS foo > WHERE category=1 > > doesn't use the index on field, while te