Re: [GENERAL] vacuum on streaming replication

2017-07-31 Thread Michael Paquier
On Mon, Jul 31, 2017 at 7:28 AM, Andreas Kretschmer wrote: > The standby is read only, vacuum runs on the master and replicated to the > standby. Analyse as well. Please note as well that if hot_standby_feedback is enabled, the cleanup done by VACUUM on the primary is influenced as well so as tu

Re: [GENERAL] vacuum on streaming replication

2017-07-31 Thread Chris Travers
On Mon, Jul 31, 2017 at 10:08 AM, Michael Paquier wrote: > On Mon, Jul 31, 2017 at 7:28 AM, Andreas Kretschmer > wrote: > > The standby is read only, vacuum runs on the master and replicated to > the standby. Analyse as well. > > Please note as well that if hot_standby_feedback is enabled, the >

Re: [GENERAL] vacuum on streaming replication

2017-07-31 Thread Alex Samad
Thanks On 31 July 2017 at 18:11, Chris Travers wrote: > > > On Mon, Jul 31, 2017 at 10:08 AM, Michael Paquier < > michael.paqu...@gmail.com> wrote: > >> On Mon, Jul 31, 2017 at 7:28 AM, Andreas Kretschmer >> wrote: >> > The standby is read only, vacuum runs on the master and replicated to >> th

[GENERAL] Question about loading up a table

2017-07-31 Thread Alex Samad
Hi I am using pg_dump | psql to transfer data from my old 9.2 psql into a 9.6 psql. The new DB server is setup as master replicating to a hot standby server. What I have noticed is that the rows don't get replicated over until the copy from stdin is finished... hard to test when you have M+ line

Re: [GENERAL] Question about loading up a table

2017-07-31 Thread vinny
On 2017-07-31 11:02, Alex Samad wrote: Hi I am using pg_dump | psql to transfer data from my old 9.2 psql into a 9.6 psql. The new DB server is setup as master replicating to a hot standby server. What I have noticed is that the rows don't get replicated over until the copy from stdin is finis

Re: [GENERAL] Row based permissions: at DB or at Application level?

2017-07-31 Thread Thomas Güttler
Am 27.07.2017 um 12:05 schrieb vinny: On 2017-07-27 10:27, Thomas Güttler wrote: Am 25.07.2017 um 12:59 schrieb vinny: On 2017-07-25 11:40, Thomas Güttler wrote: I would like to reduce the "ifing and elsing" in my python code (less conditions, less bugs, more SQL, more performance) Wh

Re: [GENERAL] Perfomance of IN-clause with many elements and possible solutions

2017-07-31 Thread Jeff Janes
On Mon, Jul 24, 2017 at 3:12 PM, Dmitry Lazurkin wrote: > On 25.07.2017 00:31, David G. Johnston wrote: > > > Basically you want to write something like: > > SELECT * > FROM ids > JOIN ( :values_clause ) vc (vid) ON (vc.vid = ids.id)​ > > or > > WITH vc AS (SELECT vid FROM ORDER BY ... LIMIT

Re: [GENERAL] Perfomance of IN-clause with many elements and possible solutions

2017-07-31 Thread Jeff Janes
On Tue, Jul 25, 2017 at 2:03 AM, Dmitry Lazurkin wrote: > On 25.07.2017 05:50, Jeff Janes wrote: > >> It isn't either-or. It is the processing of millions of rows over the >> large in-list which is taking the time. Processing an in-list as a hash >> table would be great, but no one has gotten ar

Re: [GENERAL] Perfomance of IN-clause with many elements and possible solutions

2017-07-31 Thread Jeff Janes
On Mon, Jul 24, 2017 at 3:46 PM, Tom Lane wr > > > regression=# EXPLAIN (ANALYZE, BUFFERS) SELECT count(*) FROM ids WHERE id > IN > :values_clause; > QUERY PLAN > >

Re: [GENERAL] Perfomance of IN-clause with many elements and possible solutions

2017-07-31 Thread Jeff Janes
On Mon, Jul 24, 2017 at 8:03 PM, David G. Johnston < david.g.johns...@gmail.com> wrote: > On Mon, Jul 24, 2017 at 7:58 PM, David G. Johnston < > david.g.johns...@gmail.com> wrote: > >> On Mon, Jul 24, 2017 at 3:46 PM, Tom Lane wrote: >> >>> >>> The cost to form the inner hash is basically negligi

Re: [GENERAL] Perfomance of IN-clause with many elements and possible solutions

2017-07-31 Thread Dmitry Lazurkin
On 31.07.2017 19:42, Jeff Janes wrote: > I think it is simply because no one has gotten around to implementing > it that way. When you can just write it as a values list instead, the > incentive to make the regular in-list work better is not all that strong. > > Cheers, > > Jeff I see from explai

[GENERAL] Partitioned TEMP tables

2017-07-31 Thread Ed Behn
I have an issue regarding partitioned TEMP tables. I have a database with a number of families of tables partitioned by day as described in section 5.10 of the User's Manual. I have an empty parent tables each with a number of child tables containing data partitioned by date. Each child has a CHEC

Re: [GENERAL] Dealing with ordered hierarchies

2017-07-31 Thread Peter J. Holzer
On 2017-07-25 01:15:56 +1200, Tim Uckun wrote: > I don't like the approach with a large increment. It would mean complicated > logic to see if you filled the gap and then update all the other peers if you > did. It sounds like the re-order is going to be expensive no matter what. My > primary conce

Re: [GENERAL] Partitioned TEMP tables

2017-07-31 Thread Tom Lane
Ed Behn writes: > Does partitioning of TEMP tables not work like non-TEMP tables? Should be the same ... but you don't get any auto-analyze support on a temp table. I wonder if you're remembering to ANALYZE the temp tables after you've populated them. regards, tom lane

Re: [GENERAL] Question about loading up a table

2017-07-31 Thread Scott Marlowe
On Mon, Jul 31, 2017 at 2:31 AM, vinny wrote: > On 2017-07-31 11:02, Alex Samad wrote: >> >> Hi >> >> I am using pg_dump | psql to transfer data from my old 9.2 psql into a >> 9.6 psql. Note that you should be doing pg_dump with 9.6's pg_dump, as it's possible for 9.2's pg_dump to not know about

Re: [GENERAL] partitioning question

2017-07-31 Thread Scott Marlowe
On Sun, Jul 30, 2017 at 7:13 PM, Alex Samad wrote: > How expensive is dynamic over static. I'm looking at storing yearly now, so > I figure if my if then clause has the latest year at the top it should be > very quick. Assuming you're not doing anything particularly crazy it's minimal. But what

Re: [GENERAL] Schemas and serials

2017-07-31 Thread Scott Marlowe
On Sat, Jul 29, 2017 at 1:17 PM, Melvin Davidson wrote: > > On Sat, Jul 29, 2017 at 3:38 PM, tel medola wrote: > >> Depends. >> When you create your tables in new schema, the script was the same from >> "qa"? >> Sequences, tables, etc.. belong to the schema where was created. >> >> Roberto. >> >

[GENERAL] standby database crash

2017-07-31 Thread Seong Son (US)
I have a client who has streaming replication setup with the primary in one city and standby in another city both identical servers with Postgresql 9.6 on Windows Server 2012. They have some network issues, which is causing the connection from the primary to standby to drop sometimes. And rece

[GENERAL] upsert and update filtering

2017-07-31 Thread armand pirvu
Hi create table dimc1 ( col1 integer not null, col2 char(10), primary key (col1) ); create table dimc2 ( col1 integer not null, col2 char(10), primary key (col1) ); testdb=# select * from dimc1 order by 1; col1 |col2 --+ 111 | foo111 112 | foo112 (2 row

Re: [GENERAL] upsert and update filtering

2017-07-31 Thread Peter Geoghegan
armand pirvu wrote: But what if in the conflict situation I want to performa the update ONLY if the record is different. The update seems to happen no matter what In other words is there anyway I can filter the update to happen (based on the sample date) only for 112 since col2 is different ?

Re: [GENERAL] upsert and update filtering

2017-07-31 Thread armand pirvu
That's because the access on this case is done to the existing row using the table's name / alias , and to the rows that we attempt to insert using the excluded Thank you Peter Armand > On Jul 31, 2017, at 4:31 PM, Peter Geoghegan wrote: > > armand pirvu wrote: >> But what if in the

Re: [GENERAL] Manage slot in logical/pglogical replication

2017-07-31 Thread Scott Marlowe
On Mon, Jul 17, 2017 at 7:08 AM, dpat wrote: > yes i have estimated pg_xlog but server, some time, writes a lot of WAL file. > there are some robust alternative to replicate partial database in > Postgresql over WAN? > or, what's the best way to realign pglogical replica? pg_dump/restore?. > i hav

Re: [GENERAL] Partitioned TEMP tables

2017-07-31 Thread Tom Lane
Ed Behn writes: > I tried that. I didn't seem to help. Well, it works in a simple test case for me. You'll need to post a self-contained example that's not working if you want much help ... regression=# create temp table pp(f1 int, f2 text); CREATE TABLE regression=# create temp table c1(check(

[GENERAL] PostgreSQL and Data Warehouse optimizations

2017-07-31 Thread Daniel Franco
Recently, I've been doing a research about data warehouse optimizations and techniques. For example, Oracle uses parallel queries, materialized views with different refresh modes, star transformations in queries, etc. Since I've never worked with PostgreSQL in an OLAP environment, I thought maybe t

Re: [GENERAL] Question about loading up a table

2017-07-31 Thread Alex Samad
Hi I'm using pg_dump 9.6 to do the dumps. I'm also pretty sure no data is being replicated until the end of the copy stdin as I was watching tcpdump output and I can see data from the orig master to the new master and no traffic between new master and the standby, pretty sure my replication is wo

Re: [GENERAL] partitioning question

2017-07-31 Thread Alex Samad
The comparison would be a if then else end if .. about 8 of them 2013-> and a static insert into v's making a dynamic string and using execute, my presumption would be the execute would be expensive verses a INSERT command A On 1 August 2017 at 07:04, Scott Marlowe wrote: > On Sun, Jul 30, 2

Re: [GENERAL] Question about loading up a table

2017-07-31 Thread Alex Samad
Hi I double checked and there is data going over, thought I would correct that. But it seems to be very slow. Having said that how do I / what tools do I use to check through put A On 1 August 2017 at 08:56, Alex Samad wrote: > Hi > > I'm using pg_dump 9.6 to do the dumps. > > I'm also pret