[HACKERS] Inheritance and indexes

2014-01-14 Thread knizhnik
From PostgreSQL manual: "A serious limitation of the inheritance feature is that indexes (including unique constraints) and foreign key constraints only apply to single tables, not to their inheritance children." But is it possible to use index for derived table at all? Why sequential search i

Re: [Lsf-pc] [HACKERS] Linux kernel impact on PostgreSQL performance

2014-01-16 Thread knizhnik
I wonder if kernel can sometimes provide weaker version of fsync() which is not enforcing all pending data to be written immediately but just servers as write barrier, guaranteeing that all write operations preceding fsync() will be completed before any of subsequent operations. It will allow

Re: [HACKERS] Memory ordering issue in LWLockRelease, WakeupWaiters, WALInsertSlotRelease

2014-02-12 Thread knizhnik
On 02/12/2014 05:42 PM, Florian Pflug wrote: On Feb12, 2014, at 12:55 , MauMau wrote: From: "Andres Freund" It's x86, right? Then it's unlikely to be actual unordered memory accesses, but if the compiler reordered: LOG_LWDEBUG("LWLockRelease", T_NAME(l), T_ID(l), "release waiter"); proc

Re: [HACKERS] Memory ordering issue in LWLockRelease, WakeupWaiters, WALInsertSlotRelease

2014-02-13 Thread knizhnik
On 02/12/2014 06:10 PM, Ants Aasma wrote: On Wed, Feb 12, 2014 at 4:04 PM, knizhnik wrote: Even if reordering was not done by compiler, it still can happen while execution. There is no warranty that two subsequent assignments will be observed by all CPU cores in the same order. The x86

Re: [HACKERS] Memory ordering issue in LWLockRelease, WakeupWaiters, WALInsertSlotRelease

2014-02-14 Thread knizhnik
On 02/14/2014 07:51 PM, Andres Freund wrote: On 2014-02-14 15:03:16 +0100, Florian Pflug wrote: On Feb14, 2014, at 14:07 , Andres Freund wrote: On 2014-02-14 13:52:45 +0100, Florian Pflug wrote: I agree we should do that, but imo not in the backbranches. Anything more than than the minimal fi

Re: [HACKERS] Memory ordering issue in LWLockRelease, WakeupWaiters, WALInsertSlotRelease

2014-02-14 Thread knizhnik
On 02/14/2014 08:28 PM, Andres Freund wrote: On 2014-02-14 20:23:32 +0400, knizhnik wrote: we'll trade correctness for cleanliness if we continue to reset lwWaitLink without protecting against the race. That's a bit of a weird trade-off to make. It's not just cleanliness, it&

[HACKERS] In-Memory Columnar Store

2013-12-09 Thread knizhnik
Hello! I want to annouce my implementation of In-Memory Columnar Store extension for PostgreSQL: Documentation: http://www.garret.ru/imcs/user_guide.html Sources: http://www.garret.ru/imcs-1.01.tar.gz Any feedbacks, bug reports and suggestions are welcome. Vertical representation o

Re: [HACKERS] In-Memory Columnar Store

2013-12-11 Thread knizhnik
ет всего 7 секунд... On 12/11/2013 06:33 PM, Merlin Moncure wrote: On Mon, Dec 9, 2013 at 1:40 PM, knizhnik wrote: Hello! I want to annouce my implementation of In-Memory Columnar Store extension for PostgreSQL: Documentation: http://www.garret.ru/imcs/user_guide.html Sou

Re: [HACKERS] In-Memory Columnar Store

2013-12-11 Thread knizhnik
ec 09, 2013 at 11:40:41PM +0400, knizhnik wrote: Hello! I want to annouce my implementation of In-Memory Columnar Store extension for PostgreSQL: Documentation: http://www.garret.ru/imcs/user_guide.html Sources: http://www.garret.ru/imcs-1.01.tar.gz Any feedbacks, bug reports an

Re: [HACKERS] In-Memory Columnar Store

2013-12-11 Thread knizhnik
, the code falls into infinite recursion. Patched version of IMCS is available at http://www.garret.ru/imcs-1.01.tar.gz I am going to place IMCS under version control now. Just looking for proper place for repository... On 12/12/2013 04:06 AM, desmodemone wrote: 2013/12/9 knizhnik

Re: [HACKERS] In-Memory Columnar Store

2013-12-12 Thread knizhnik
rom default 1Mb, then time of query execution is reduced to 7107.146 ms. So the real difference is ten times, not 1000 times. Yesterday we did a some tests, that shows so for large tables (5G)a our hashing is not effective. Disabling hash join and using merge join increased speed 2x

Re: [HACKERS] In-Memory Columnar Store

2013-12-12 Thread knizhnik
On 12/12/2013 07:03 PM, Merlin Moncure wrote: On Thu, Dec 12, 2013 at 4:02 AM, knizhnik wrote: Yeah. It's not fair to compare vs an implementation that is constrained to use only 1mb. For analytics work huge work mem is pretty typical setting. 10x improvement is believable considering y

[HACKERS] Polymorphic function calls

2013-12-29 Thread knizhnik
Is there any chance to implement polymorphic calls in PostgreSQL? Consider the following definitions: create table base_table (x integer, y integer); create table derived_table (z integer) inherits (base_table); create function volume(r base_table) returns integer as $$ begin return r.x*r.y; end

Re: [HACKERS] Polymorphic function calls

2013-12-30 Thread knizhnik
On 12/30/2013 01:22 AM, Sergey Konoplev wrote: On Sun, Dec 29, 2013 at 8:44 AM, knizhnik wrote: create function volume(r base_table) returns integer as $$ begin return r.x*r.y; end; $$ language plpgsql strict stable; create function volume(r derived_table) returns integer as $$ begin return

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-03 Thread knizhnik
ht that I have done everything in legal way. On 01/04/2014 03:21 AM, David Fetter wrote: On Thu, Jan 02, 2014 at 08:48:24PM +0400, knizhnik wrote: I want to announce implementation of In-Memory Columnar Store extension for PostgreSQL. Vertical representation of data is stored in PostgreSQL sh

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-04 Thread knizhnik
t somehow interleave with shared memory used for PostgreSQL shared buffers. And the only limitation is this 2567Gb limit at Linux, which can be resolved using the patch included in IMCS distributive. Cheers, David. On Sat, Jan 04, 2014 at 11:46:25AM +0400, knizhnik wrote: Hi David, Sorry

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-04 Thread knizhnik
On 01/04/2014 11:11 PM, Tom Lane wrote: knizhnik writes: On 01/04/2014 12:05 PM, David Fetter wrote: Is there some way not to use shared memory for it? No, IMCS ("In-Memory Columnar Store") is storing data in shared memory. It would probably be better if it made use of the dyna

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-05 Thread knizhnik
e how dynamically allocated DSM segments will be shared by all PostgreSQL processes? On 01/05/2014 08:50 PM, Robert Haas wrote: On Sat, Jan 4, 2014 at 3:27 PM, knizhnik wrote: 1. I want IMCS to work with PostgreSQL versions not supporting DSM (dynamic shared memory), like 9.2, 9.3.1,... Yeah. If

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-08 Thread knizhnik
On 01/08/2014 10:51 PM, Robert Haas wrote: On Tue, Jan 7, 2014 at 10:20 PM, Amit Kapila wrote: On Tue, Jan 7, 2014 at 2:46 AM, Robert Haas wrote: On Mon, Jan 6, 2014 at 4:04 PM, james wrote: The point remains that you need to duplicate it into every process that might want to use it subsequ

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-09 Thread knizhnik
On 01/09/2014 09:22 PM, Robert Haas wrote: On Wed, Jan 8, 2014 at 2:39 PM, knizhnik wrote: I wonder what is the intended use case of dynamic shared memory? Is is primarly oriented on PostgreSQL extensions or it will be used also in PosatgreSQL core? My main motivation is that I want to use it

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-09 Thread knizhnik
On 01/09/2014 09:46 PM, Claudio Freire wrote: On Thu, Jan 9, 2014 at 2:22 PM, Robert Haas wrote: It would be nice to have better operating system support for this. For example, IIUC, 64-bit Linux has 128TB of address space available for user processes. When you clone(), it can either share the

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-09 Thread knizhnik
On 01/09/2014 11:09 PM, Amit Kapila wrote: On Thu, Jan 9, 2014 at 12:21 AM, Robert Haas wrote: On Tue, Jan 7, 2014 at 10:20 PM, Amit Kapila wrote: On Tue, Jan 7, 2014 at 2:46 AM, Robert Haas wrote: Well, right now we just reopen the same object from all of the processes, which seems to work

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-09 Thread knizhnik
On 01/09/2014 11:30 PM, Claudio Freire wrote: On Thu, Jan 9, 2014 at 4:24 PM, knizhnik wrote: On 01/09/2014 09:46 PM, Claudio Freire wrote: On Thu, Jan 9, 2014 at 2:22 PM, Robert Haas wrote: It would be nice to have better operating system support for this. For example, IIUC, 64-bit Linux

Re: [HACKERS] [ANNOUNCE] IMCS: In Memory Columnar Store for PostgreSQL

2014-01-09 Thread knizhnik
On 01/09/2014 11:48 PM, Claudio Freire wrote: On Thu, Jan 9, 2014 at 4:39 PM, knizhnik wrote: At fork time I only wrote about reserving the address space. After reserving it, all you have to do is implement an allocator that works in shared memory (protected by a lwlock of course). In essence

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Konstantin Knizhnik
On 02/27/2016 06:57 AM, Robert Haas wrote: On Sat, Feb 27, 2016 at 1:49 AM, Konstantin Knizhnik wrote: pg_tsdtm is based on another approach: it is using system time as CSN and doesn't require arbiter. In theory there is no limit for scalability. But differences in system time and nece

Re: [HACKERS] Relation cache invalidation on replica

2016-02-26 Thread Konstantin Knizhnik
On 02/27/2016 04:16 AM, Simon Riggs wrote: On 27 February 2016 at 00:33, Simon Riggs mailto:si...@2ndquadrant.com>> wrote: On 27 February 2016 at 00:29, Andres Freund mailto:and...@anarazel.de>> wrote: On 2016-02-26 18:05:55 +0300, Konstantin Knizhnik wrote: >

Re: [HACKERS] The plan for FDW-based sharding

2016-02-27 Thread Konstantin Knizhnik
On 02/27/2016 06:54 AM, Robert Haas wrote: On Fri, Feb 26, 2016 at 10:56 PM, Konstantin Knizhnik wrote: We do not have formal prove that proposed XTM is "general enough" to handle all possible transaction manager implementations. But there are two general ways of dealing with

Re: [HACKERS] The plan for FDW-based sharding

2016-02-27 Thread Konstantin Knizhnik
visibility check. On 02/27/2016 01:48 AM, Kevin Grittner wrote: On Fri, Feb 26, 2016 at 2:19 PM, Konstantin Knizhnik wrote: pg_tsdtm is based on another approach: it is using system time as CSN Which brings up an interesting point, if we want logical replication to be free of serialization ano

Re: [HACKERS] The plan for FDW-based sharding

2016-02-28 Thread Konstantin Knizhnik
have not (yet) reviewed the literature for it. The reference to the article is at our WiKi pages explaining our DTM: https://wiki.postgresql.org/wiki/DTM http://research.microsoft.com/en-us/people/samehe/clocksi.srds2013.pdf -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Konstantin Knizhnik
Thank you very much for you comments. On 01.03.2016 18:19, Robert Haas wrote: On Sat, Feb 27, 2016 at 2:29 AM, Konstantin Knizhnik wrote: How do you prevent clock skew from causing serialization anomalies? If node receives message from "feature" it just needs to wait until this fut

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Konstantin Knizhnik
ion manager API would end up similarly situated. IMHO non-stable API is better than lack of API. Just because it makes it possible to implement features in modular way. And refactoring of API is not so difficult thing... -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The R

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Konstantin Knizhnik
pg_tsdtm provides mapping between local XIDs and global CSNs. Visibility checking rules looks on CSNs, not on XIDs. In both cases if system is for some reasons restarted and DTM plugin failed to be loaded, you can still access database locally. No data can be lost. -- Konstantin Knizhnik Po

Re: [HACKERS] PROPOSAL: Fast temporary tables

2016-03-01 Thread Konstantin Knizhnik
t's tables. Actually translating that into relcache and everything else would be a serious amount of work. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] The plan for FDW-based sharding

2016-03-02 Thread Konstantin Knizhnik
P and simple OLAP. For OLTP we definitely need transaction manager to provide global consistency. And we have actually prototype of integration postgres_fdw with out pg_dtm and pg_tsdtm transaction managers. The results are quite IMHO promising (see attached diagram). -- Konstantin Knizhnik Pos

Re: [HACKERS] The plan for FDW-based sharding

2016-03-07 Thread Konstantin Knizhnik
mprovements in this area to be part of core. None of that means I would support any particular hook proposal, of course. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] Optimizer questions

2016-03-08 Thread Konstantin Knizhnik
On 03/08/2016 07:01 AM, Tom Lane wrote: Konstantin Knizhnik writes: Attached please find improved version of the optimizer patch for LIMIT clause. This patch isn't anywhere close to working after 3fc6e2d7f5b652b4. (TBH, the reason I was negative about this upthread is that I had that o

Re: [HACKERS] Optimizer questions

2016-03-09 Thread Konstantin Knizhnik
construction of the path instead of comparing costs of full paths. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.post

Re: [HACKERS] Optimizer questions

2016-03-10 Thread konstantin knizhnik
On Mar 10, 2016, at 1:56 AM, Tom Lane wrote: > Konstantin Knizhnik writes: >> I think that the best approach is to generate two different paths: >> original one, when projection is always done before sort and another one >> with postponed projection of non-trivial col

Re: [HACKERS] eXtensible Transaction Manager API (v2)

2016-03-11 Thread Konstantin Knizhnik
committable during this 'fest. I think we'd be well advised to boot it to the 2016-09 CF and focus our efforts on other stuff that has a better chance of getting finished this month. regards, tom lane -- Konstantin Knizhnik Postgres Professional: http://

Re: [HACKERS] Batch update of indexes

2016-03-14 Thread Konstantin Knizhnik
Hi David, Rebased patch is attached. On 14.03.2016 15:09, David Steele wrote: Hi Konstantin, On 2/3/16 11:47 AM, Konstantin Knizhnik wrote: Attached please find patch for "ALTER INDEX ... WHERE ..." clause. It is now able to handle all three possible situations: 1. Making index pa

[HACKERS] Applying logical replication changes by more than one process

2016-03-19 Thread Konstantin Knizhnik
done using current logical replication mechanism when changes of each slot are applied by more than one process? Or the only alternative is to write/read origin LSNs in WAL myself, for example using custom WAL records? Thanks in advance! -- Konstantin Knizhnik Postgres Professional: http://www.

Re: [HACKERS] Applying logical replication changes by more than one process

2016-03-21 Thread Konstantin Knizhnik
On 21.03.2016 15:10, Petr Jelinek wrote: Hi, On 19/03/16 11:46, Konstantin Knizhnik wrote: Hi, I am trying to use logical replication mechanism in implementation of PostgreSQL multimaster and faced with one conceptual problem. Originally logical replication was intended to support

Re: [HACKERS] Applying logical replication changes by more than one process

2016-03-21 Thread konstantin knizhnik
On Mar 21, 2016, at 4:30 PM, Petr Jelinek wrote: > On 21/03/16 14:25, Andres Freund wrote: >> On 2016-03-21 14:18:27 +0100, Petr Jelinek wrote: >>> On 21/03/16 14:15, Andres Freund wrote: > Only when the origin is actually setup for the current session. You > need > to call the replor

Re: [HACKERS] Applying logical replication changes by more than one process

2016-03-22 Thread konstantin knizhnik
On Mar 22, 2016, at 10:10 AM, Craig Ringer wrote: > On 22 March 2016 at 14:32, konstantin knizhnik > wrote: > >> Ah you mean because with wal_log=true the origin advance is in different WAL >> record than commit? OK yeah you might be one transaction behind then, true. &

Re: [HACKERS] Applying logical replication changes by more than one process

2016-03-22 Thread konstantin knizhnik
On Mar 22, 2016, at 11:14 AM, Petr Jelinek wrote: > > And each slot means connection with logical decoding attached to it so you > don't really want to have thousands of those anyway. I think you'll hit other > problems faster than loop over slots becomes problem if you plan to keep all > of t

[HACKERS] avg,first,last,median in one query

2016-03-24 Thread Konstantin Knizhnik
er-defined aggregate function which uses array_appendand so materialize all values in memory: https://wiki.postgresql.org/wiki/Aggregate_Median 3. Using percentile aggregate: http://blog.jooq.org/2015/01/06/how-to-emulate-the-median-aggregate-function-using-inverse-distribution-functions/ Th

[HACKERS] Very small patch for decode.c

2016-03-30 Thread Konstantin Knizhnik
i; -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

[HACKERS] Limit and inherited tables

2016-01-15 Thread Konstantin Knizhnik
er patch for it. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Limit and inherited tables

2016-01-15 Thread Konstantin Knizhnik
ows=1 loops=1) But if sort is performed by non-indexed fields, then current behaviour will be inefficient and can be significantly improved by pushing limits to remote hosts. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via p

Re: [HACKERS] Optimizer questions

2016-01-18 Thread Konstantin Knizhnik
onstantin knizhnik wrote: Hi hackers, I want to ask two questions about PostgreSQL optimizer. I have the following query: SELECT o.id as id,s.id as sid,o.owner,o.creator,o.parent_id as dir_id,s.mime_id,m.c_type,s.p_file,s.mtime,o.ctime,o.name ,o.title,o.size,o.deleted,la.otime,la.etime,uo

[HACKERS] Batch update of indexes

2016-01-20 Thread Konstantin Knizhnik
ase: search condition is exactly the same as partial index condition. Optimal plan should be: Index Scan using idx1 on t (cost=0.00..4.13 rows=12263 width=0) Index Cond: (c1 < '10'::double precision) What do you think about this approach? Will it be useful to work in this direction? Or there are some better solutions for the problem? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Batch update of indexes

2016-01-20 Thread Konstantin Knizhnik
nk it'll be interesting to you. So small patch... Why it was not accepted? I do no see any problems with it... -- Anastasia Lubennikova Postgres Professional:http://www.postgrespro.com The Russian Postgres Company -- Konstantin Knizhnik Postgres Professional: http://www.postgrespr

Re: [HACKERS] Proposal for UPDATE: do not insert new tuple on heap if update does not change data

2016-01-20 Thread Konstantin Knizhnik
r any further help with testing and ideas. Attached is a script with minimal test case. Kind regards, Gasper Zejn -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Batch update of indexes

2016-01-20 Thread konstantin knizhnik
On Jan 21, 2016, at 5:14 AM, Simon Riggs wrote: > On 20 January 2016 at 14:55, Konstantin Knizhnik > wrote: > Hi, > > Hi, I glad to see that you interested in that too. > I think this is a good feature and I think it will be very useful to have. > I have already m

Re: [HACKERS] Batch update of indexes

2016-01-21 Thread Konstantin Knizhnik
On 21.01.2016 10:14, Simon Riggs wrote: On 21 January 2016 at 06:41, konstantin knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Certainly for B-Tree we can organize insert buffer (or pending list) as sorted array or also as a tree. But in both case complexity of search i

Re: [HACKERS] Batch update of indexes

2016-01-21 Thread Konstantin Knizhnik
es during index scan and provide proper recovery of main index in case of failure (assuming that pending list is maintained in memory and is lost after the fault). -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers ma

Re: [HACKERS] Batch update of indexes

2016-01-26 Thread Konstantin Knizhnik
ed to be interesting by community, I will try to address these issues. On 20.01.2016 12:28, Konstantin Knizhnik wrote: Hi hackers, I want to know opinion of community about possible ways of solving quite common problem: increasing insert speed while still providing indexes for efficient executi

Re: [HACKERS] Optimizer questions

2016-01-30 Thread Konstantin Knizhnik
34 rows=1 width=8) (actual time=0.007..0.007 rows=1 loops=10) Index Cond: (t1.k = k) Planning time: 0.537 ms Execution time: 0.241 ms (7 rows) On 01/30/2016 01:01 AM, Alexander Korotkov wrote: On Fri, Jan 8, 2016 at 11:58 AM, Konstantin Knizhnik mailto:k.knizh...@postgrespro.r

Re: [HACKERS] PATCH: index-only scans with partial indexes

2016-01-31 Thread Konstantin Knizhnik
this point I'm not sure if either Kyotaro or Tomas should be considered the patch author ... maybe both?) -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgres

Re: [HACKERS] PATCH: index-only scans with partial indexes

2016-02-02 Thread Konstantin Knizhnik
On 01.02.2016 01:11, Alvaro Herrera wrote: Konstantin Knizhnik wrote: I am very interested in this patch because it allows to use partial indexes to ... speed up inserts. I have implemented "ALTER INDEX ... WHERE ..." construction which allows to change predicate of partial index without

Re: [HACKERS] Batch update of indexes

2016-02-03 Thread Konstantin Knizhnik
ether with "index-only scans with partial indexes" patch: http://www.postgresql.org/message-id/560c7213.3010...@2ndquadrant.com only in this case regression test will produce expected output. On 27.01.2016 23:15, Robert Haas wrote: On Wed, Jan 20, 2016 at 4:28 AM, Konstantin Kniz

Re: [HACKERS] Batch update of indexes

2016-02-03 Thread konstantin knizhnik
On Feb 4, 2016, at 2:00 AM, Jim Nasby wrote: > > My suspicion is that it would be useful to pre-order the new data before > trying to apply it to the indexes. Sorry, but ALTER INDEX is expected to work for all indexes, not only B-Tree, and for them sorting may not be possible... But for B-Tre

[HACKERS] eXtensible Transaction Manager API (v2)

2016-02-10 Thread Konstantin Knizhnik
age. The complete PostgreSQL branch with all our changes can be found here: https://github.com/postgrespro/postgres_cluster -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/contrib/pg_tsdtm/Makefile b/contrib/pg_tsdtm/Makefil

[HACKERS] Clock with Adaptive Replacement

2016-02-11 Thread Konstantin Knizhnik
ARC (CAR is inspired by ARC, but it is different algorithm). As far as I know there are several problems with current clock-sweep algorithm in PostgreSQL, especially for very large caches. May be CAR can address some of them? -- Konstantin Knizhnik Postgres Professional: http

Re: [HACKERS] Clock with Adaptive Replacement

2016-02-12 Thread Konstantin Knizhnik
T can address this problem because there are not counters - justs single bit per page. On 12.02.2016 18:55, Robert Haas wrote: On Thu, Feb 11, 2016 at 4:02 PM, Konstantin Knizhnik wrote: What do you think about improving cache replacement clock-sweep algorithm in PostgreSQL with adaptive ve

Re: [HACKERS] pglogical - logical replication contrib module

2016-02-17 Thread Konstantin Knizhnik
are a lot simpler than the sort of gymnastics done by ALTER TABLE, etc. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] pglogical - logical replication contrib module

2016-02-17 Thread Konstantin Knizhnik
. On 17.02.2016 12:16, Craig Ringer wrote: On 17 February 2016 at 16:24, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Thanks for your explanation. I have to agree with your arguments that in general case replication of DDL statement using logical decoding seems

Re: [HACKERS] The plan for FDW-based sharding

2016-02-24 Thread Konstantin Knizhnik
ostgres XC/XL features, Postgres XC/XL will probably remain a separate fork of Postgres. I don't think anyone knows the answer to this question, and I don't know how to find the answer except to keep going with our current FDW sharding approach. -- Konstantin Knizhnik Postgres Profess

Re: [HACKERS] Relation cache invalidation on replica

2016-02-26 Thread Konstantin Knizhnik
tgreSQL 9.5.1. --- Dmitry Vasilyev Postgres Professional: http://www.postgrespro.com Russian Postgres Company -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Konstantin Knizhnik
n't think about distributed transactions support doesn't mean there no "other people", who has different ideas on postgres future. That's why we propose this patch, let's play the game ! I don't like to play games with the architecture of PostgreSQL. -- Konstantin

Re: [HACKERS] The plan for FDW-based sharding

2016-02-26 Thread Konstantin Knizhnik
On 02/26/2016 09:30 PM, Alvaro Herrera wrote: Konstantin Knizhnik wrote: Yes, it is certainly possible to develop cluster by cloning PostgreSQL. But it cause big problems both for developers, which have to permanently synchronize their branch with master, and, what is more important, for

[HACKERS] Logical decoding restart problems

2016-08-19 Thread konstantin knizhnik
r recreation of the slot? Is there any mechanism in xlog which can enforce consistent decoding of transaction (so that no transaction records are missed)? May be I missed something but I didn't find any "record_number" or something else which can identify first record of transaction.

Re: [HACKERS] Logical decoding restart problems

2016-08-19 Thread Konstantin Knizhnik
a. I started investigation of logical decoding code and found several things which I do not understand. Never seen this happen. Do you have more details about what exactly is happening? This is transaction at primary node: root@knizhnik:/home/knizhnik/postgres_cluster/contrib/mmts# docker exec

Re: [HACKERS] Logical decoding restart problems

2016-08-20 Thread konstantin knizhnik
Thank you for answers. > No, you don't need to recreate them. Just advance your replication identifier > downstream and request a replay position in the future. Let the existing slot > skip over unwanted data and resume where you want to start replay. > > You can advance the replication origins

Re: [HACKERS] UPSERT strange behavior

2016-08-25 Thread Konstantin Knizhnik
e correct, what is the best workaround for the problem if we really need to have to separate indexes and want to enforce unique constraint for both keys? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list

[HACKERS] Handling dropped attributes in pglogical_proto

2016-09-28 Thread Konstantin Knizhnik
t->atttypid == InvalidOid) { + continue; + } + kind = pq_getmsgbyte(in); + switch (kind) { case 'n': /* null */ -- Konstantin Knizhnik Postgres Professional: http://www.postgr

[HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-06 Thread Konstantin Knizhnik
"shardman_copy_t_10_3_4_17307_sync_17302\" TEMPORARY LOGICAL pgoutput USE_SNAPSHOT") at walsender.c:1515 #26 0x0088eccc in PostgresMain (argc=1, argv=0x24f0b28, dbname=0x24f0948 "postgres", username=0x24bf7f0 "knizhnik") at postgres.c:4086 #27 0x00

Re: [HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-06 Thread Konstantin Knizhnik
On 06.10.2017 15:29, Petr Jelinek wrote: On 06/10/17 12:16, Konstantin Knizhnik wrote: When creating logical replication slots we quite often get the following error: ERROR: cannot build an initial slot snapshot when MyPgXact->xmin already is valid which cause restart of WAL sender.

Re: [HACKERS] Issue with logical replication: MyPgXact->xmin already is valid

2017-10-07 Thread Konstantin Knizhnik
at logical decoding snapshot is being built and using that to skip catalog access which does not seem very pretty. It is not quite clear from the comment why it is not possible to overwrite MyPgXact->xmin or just use existed value. -- Konstantin Knizhnik Postgres Professional: http://www.postgr

[HACKERS] Slow synchronous logical replication

2017-10-07 Thread konstantin knizhnik
In our sharded cluster project we are trying to use logical relication for providing HA (maintaining redundant shard copies). Using asynchronous logical replication has not so much sense in context of HA. This is why we try to use synchronous logical replication. Unfortunately it shows very bad p

Re: [HACKERS] Slow synchronous logical replication

2017-10-07 Thread Konstantin Knizhnik
On 10/07/2017 10:42 PM, Andres Freund wrote: Hi, On 2017-10-07 22:39:09 +0300, konstantin knizhnik wrote: In our sharded cluster project we are trying to use logical relication for providing HA (maintaining redundant shard copies). Using asynchronous logical replication has not so much sense

Re: [HACKERS] Slow synchronous logical replication

2017-10-09 Thread Konstantin Knizhnik
rarely than other or is not updated at all (for example because communication channels between this node is broken), then all backens will stuck. Also all backends are competing for the single SyncRepLock, which also can be a contention point. -- Konstantin Knizhnik Postgres Professional: http:/

Re: [HACKERS] Columnar storage support

2017-10-10 Thread Konstantin Knizhnik
tgres: IMCS (In-Memory-Columnar-Store): https://github.com/knizhnik/imcs.git VOPS (Vectorized Operations): https://github.com/postgrespro/vops.git First one is more oriented on in-memory databases (although support spilling data to the disk) and requires to use special functions to manipulate

Re: [HACKERS] Slow synchronous logical replication

2017-10-11 Thread Konstantin Knizhnik
On 11.10.2017 10:07, Craig Ringer wrote: On 9 October 2017 at 15:37, Konstantin Knizhnik wrote: Thank you for explanations. On 08.10.2017 16:00, Craig Ringer wrote: I think it'd be helpful if you provided reproduction instructions, test programs, etc, making it very clear when thing

Re: [HACKERS] Slow synchronous logical replication

2017-10-12 Thread Konstantin Knizhnik
On 12.10.2017 04:23, Craig Ringer wrote: On 12 October 2017 at 00:57, Konstantin Knizhnik wrote: The reason of such behavior is obvious: wal sender has to decode huge transaction generate by insert although it has no relation to this publication. It does. Though I wouldn't expect any

[HACKERS] Deadlock in ALTER SUBSCRIPTION REFRESH PUBLICATION

2017-10-24 Thread Konstantin Knizhnik
Parallel execution of ALTER SUBSCRIPTION REFRESH PUBLICATION at several nodes may cause deadlock: knizhnik  1480  0.0  0.1 1417532 16496 ?   Ss   20:01   0:00 postgres: bgworker: logical replication worker for subscription 16589 sync 16720    waiting knizhnik  1481  0.0  0.1 1417668 17668

Re: [HACKERS] Secondary index access optimizations

2017-11-06 Thread Konstantin Knizhnik
On 11/06/2017 04:27 AM, Thomas Munro wrote: On Fri, Sep 8, 2017 at 3:58 AM, Konstantin Knizhnik wrote: Updated version of the patch is attached to this mail. Also I added support of date type to operator_predicate_proof to be able to imply (logdate <= '2017-03-31') from (logdate

Re: [HACKERS] SQL procedures

2017-11-08 Thread Konstantin Knizhnik
have not done it yet, because there seems to be no chances to push this patch to community. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

[HACKERS] Aggregates push-down to partitions

2017-11-09 Thread Konstantin Knizhnik
estions about the best approach to implement this feature? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Aggregates push-down to partitions

2017-11-10 Thread Konstantin Knizhnik
->  Seq Scan on derived2  (cost=0.00..14425.00 rows=100 width=0) ->  Foreign Scan on derived_fdw  (cost=100.00..212.39 rows=3413 width=0) (6 rows) Are there some principle problems? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgre

Re: [HACKERS] Partition-wise aggregation/grouping

2017-11-11 Thread Konstantin Knizhnik
d approach is easier for implementation. But in case of sharded table, distributed query may need to traverse both remote and local shards and this approach doesn't allow to processed several local shards in parallel. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The R

Re: [HACKERS] Partition-wise aggregation/grouping

2017-11-13 Thread Konstantin Knizhnik
On 11.11.2017 23:29, Konstantin Knizhnik wrote: On 10/27/2017 02:01 PM, Jeevan Chalke wrote: Hi, Attached new patch-set here. Changes include: 1. Added separate patch for costing Append node as discussed up-front in the patch-set. 2. Since we now cost Append node, we don't

[HACKERS] On conflict update & hint bits

2016-09-30 Thread Konstantin Knizhnik
s content lock. and we get assertion failure in /* here, either share or exclusive lock is OK */ Assert(LWLockHeldByMe(BufferDescriptorGetContentLock(bufHdr))); So the question is whether it is correct that ExecOnConflictUpdate tries to access and update tuple without hol

Re: [HACKERS] On conflict update & hint bits

2016-09-30 Thread Konstantin Knizhnik
On 30.09.2016 19:37, Peter Geoghegan wrote: On Fri, Sep 30, 2016 at 5:33 PM, Konstantin Knizhnik wrote: Later we try to check tuple visibility: ExecCheckHeapTupleVisible(estate, &tuple, buffer); and inside HeapTupleSatisfiesMVCC try to set hint bit. So, you're using repeat

Re: [HACKERS] On conflict update & hint bits

2016-10-24 Thread Konstantin Knizhnik
te->mt_existing, buffer, false); +LockBuffer(buffer, BUFFER_LOCK_UNLOCK); + /* * Make tuple and any needed join variables available to ExecQual and * ExecProject. The EXCLUDED tuple is installed in ecxt_innertuple, while -- Konstantin Knizhnik Postgres Professional: htt

[HACKERS] Unlogged tables cleanup

2016-11-09 Thread Konstantin Knizhnik
Hi, hackers I wonder if such behavior can be considered as a bug: knizhnik@knizhnik:~/dtm-data$ psql postgres psql (10devel) Type "help" for help. postgres=# create tablespace fs location '/home/knizhnik/dtm-data/fs'; CREATE TABLESPACE postgres=# set default_tablesp

Re: [HACKERS] Unlogged tables cleanup

2016-11-09 Thread konstantin knizhnik
abase and tablespace to reproduce this issue. So actually the whole sequence is mkdir fs initdb -D pgsql pg_ctl -D pgsql -l logfile start psql postgres # create tablespace fs location '/home/knizhnik/dtm-data/fs'; # set default_tablespace=fs; # create unlogged table foo(x integer); # ins

[HACKERS] Two questions about Postgres parser

2017-02-27 Thread Konstantin Knizhnik
::float8) from foo; select coalesce(c, 0.0::float8::complex) from foo; coalesce -- (0 rows) -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] WIP: Faster Expression Processing v4

2017-03-13 Thread Konstantin Knizhnik
1171 ms Your patch 6420 ms 1034 ms VOPS 396 ms 249 ms VOPS + patch367 ms 233 ms -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

[HACKERS] Parallel query execution with SPI

2017-03-31 Thread Konstantin Knizhnik
state->portal = SPI_cursor_open_with_args(NULL, fsstate->query, fsstate->numParams, argtypes, values, nulls, true, CURSOR_OPT_PARALLEL_OK); ... SPI_cursor_fetch(fsstate->portal, true, 1); Thanks in advance, -- Konstantin Knizhnik Postgres Professional: http://www.postgrespr

  1   2   3   >