Re: [Proposal] Global temporary tables

2019-10-28 Thread Konstantin Knizhnik
On 28.10.2019 15:13, Robert Haas wrote: On Fri, Oct 25, 2019 at 12:22 PM Konstantin Knizhnik wrote: On 25.10.2019 18:01, Robert Haas wrote: On Fri, Oct 11, 2019 at 9:50 AM Konstantin Knizhnik wrote: Just to clarify. I have now proposed several different solutions for GTT: Shared vs

Re: [Proposal] Global temporary tables

2019-10-29 Thread Konstantin Knizhnik
agree on this point, perhaps it isn't necessary to argue about those things right now. Ok. I attached new patch for GTT with local (private) buffer and no replica access. It provides GTT for all built-in indexes -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The

Re: [Proposal] Global temporary tables

2019-11-01 Thread Konstantin Knizhnik
not in pg_statistic table itself. To do it I have to add InsertSysCache/InsertCatCache functions which insert pinned entry in the correspondent cache. I wonder if there are some pitfalls of such approach? New patch for GTT is attached. -- Konstantin Knizhnik Postgres Professional: http://

Re: [Proposal] Global temporary tables

2019-11-01 Thread Konstantin Knizhnik
On 01.11.2019 18:26, Robert Haas wrote: On Fri, Nov 1, 2019 at 11:15 AM Konstantin Knizhnik wrote: It seems to me that I have found quite elegant solution for per-backend statistic for GTT: I just inserting it in backend's catalog cache, but not in pg_statistic table itself. To do

Re: [Proposal] Global temporary tables

2019-11-02 Thread Konstantin Knizhnik
On 02.11.2019 10:19, Julien Rouhaud wrote: On Sat, Nov 2, 2019 at 6:31 AM Pavel Stehule wrote: pá 1. 11. 2019 v 17:09 odesílatel Konstantin Knizhnik napsal: On 01.11.2019 18:26, Robert Haas wrote: On Fri, Nov 1, 2019 at 11:15 AM Konstantin Knizhnik wrote: It seems to me that I have

Re: [Proposal] Global temporary tables

2019-11-02 Thread Konstantin Knizhnik
On 02.11.2019 8:30, Pavel Stehule wrote: pá 1. 11. 2019 v 17:09 odesílatel Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> napsal: On 01.11.2019 18:26, Robert Haas wrote: > On Fri, Nov 1, 2019 at 11:15 AM Konstantin Knizhnik > mailto:k.knizh...@po

Re: [Proposal] Global temporary tables

2019-11-06 Thread Konstantin Knizhnik
insert into gtt values (generate_series(1,10), generate_series(1,10)); INSERT 0 10 postgres=# create index on gtt(y); ERROR:  can not create index when have one or more backend attached this global temp table I wonder why do you need such restriction? -- Konstantin Knizhnik Postgr

Re: [Proposal] Global temporary tables

2019-11-07 Thread Konstantin Knizhnik
information about GTT in shared memory. Looks like the only information we really need to share is table's metadata. But it is already shared though catalog. All other GTT related information is private to backend so I do not see reasons to place it in shared memory. -- Konstantin Knizhn

Re: Global temporary tables

2019-11-08 Thread Konstantin Knizhnik
proposed by Pavel). I afraid that it may break compatibility with some existed extensions and applications. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c index ae7b729

Re: [Proposal] Global temporary tables

2019-11-08 Thread Konstantin Knizhnik
ementation? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Why overhead of SPI is so large?

2019-11-08 Thread Konstantin Knizhnik
á 23. 8. 2019 v 16:32 odesílatel Konstantin Knizhnik < > k.knizh...@postgrespro.ru <mailto:k.knizh...@postgrespro.ru>> napsal: > > > > > > > On 23.08.2019 14:42, Pavel Stehule wrote: > > > > > > In

Re: Global temporary tables

2019-11-11 Thread Konstantin Knizhnik
Yet another version of my GTT patch addressing issues reported by 曾文旌(义从) * Bug in TRUNCATE is fixed, * ON COMMIT DELETE ROWS option is supported * ALTER TABLE is correctly handled -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git

Re: [Proposal] Global temporary tables

2019-11-11 Thread Konstantin Knizhnik
nciple, I have also implemented special visibility rules for GTT, but only for the case when them are accessed at replica. And it is not included in this patch, because everybody think that access to GTT replica should be considered in separate patch. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Why overhead of SPI is so large?

2019-11-12 Thread Konstantin Knizhnik
that it "proves nothing" if extension can put stuff in pg_catalog. I can replace it with comparison with FirstGenbkiObjectId. But all this makes sense only if using contain_mutable_function() is not acceptable. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Built-in connection pooler

2019-11-12 Thread Konstantin Knizhnik
Hi On 12.11.2019 10:50, ideriha.take...@fujitsu.com wrote: Hi. From: Konstantin Knizhnik [mailto:k.knizh...@postgrespro.ru] New version of builtin connection pooler fixing handling messages of extended protocol. Here are things I've noticed. 1. Is adding guc to postgresql.conf.s

Re: Built-in connection pooler

2019-11-13 Thread Konstantin Knizhnik
not imagine that unlike standard itoa it accepts int16 parameter instead of int). Attached please find rebased patch with this bug fixed. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/contrib/spi/refint.c b/contrib/spi/ref

Re: Replication & recovery_min_apply_delay

2019-11-15 Thread Konstantin Knizhnik
with "wal_receiver_start_condition" GUC added (preserving by default original behavior). -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml index f837703..bf4e9d4 100644 --- a/doc/src/sgml/co

Internal function returning pg_statistic

2019-11-20 Thread Konstantin Knizhnik
_statistic, then reported error is "cannot cast type record to pg_statistic". So the only possible way I found is to create extension and define function in this extension. I wonder if there is some better solution? Thanks in advance, -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Global temporary tables

2019-11-20 Thread Konstantin Knizhnik
Now pg_gtt_statistic view is provided for global temp tables. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/backend/access/brin/brin.c b/src/backend/access/brin/brin.c index ae7b729..485c068 100644 --- a/src/backend/access

Re: Why overhead of SPI is so large?

2019-11-21 Thread Konstantin Knizhnik
I've set the CF entry to "Waiting on Author" pending a new patch that does it like that. With contain_mutable_functions the patch becomes trivial. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/pl/plpgsq

Re: Why overhead of SPI is so large?

2019-11-21 Thread Konstantin Knizhnik
ons. Notes, comments? I think that even current model with "volatile", "immutable" and "stable" is complex enough. Adding more qualifiers will make it even more obscure and error-prone. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Why overhead of SPI is so large?

2019-11-22 Thread Konstantin Knizhnik
optimization Postgres can produce incorrect result in case of incorrectly specidied immutable or stable qualifiers. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Why JIT speed improvement is so modest?

2019-11-25 Thread Konstantin Knizhnik
VOPS shows that used aggregation algorithm itself is not a bottleneck. Profile also give no answer for this question. Any ideas? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

How to prohibit parallel scan through tableam?

2019-11-27 Thread Konstantin Knizhnik
local to the backend)? Thanks in advance, -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: How to prohibit parallel scan through tableam?

2019-11-27 Thread Konstantin Knizhnik
On 27.11.2019 15:12, Rafia Sabih wrote: On Wed, 27 Nov 2019 at 12:33, Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> wrote: Hi hackers, I wonder how it is possible to prohibit parallel scan for the external storage accessed through tableam? For exampl

Re: Why JIT speed improvement is so modest?

2019-11-27 Thread Konstantin Knizhnik
On 25.11.2019 18:24, Merlin Moncure wrote: On Mon, Nov 25, 2019 at 9:09 AM Konstantin Knizhnik wrote: JIT was not able to significantly (times) increase speed on Q1 query? Experiment with VOPS shows that used aggregation algorithm itself is not a bottleneck. Profile also give no answer for

Re: Why JIT speed improvement is so modest?

2019-11-27 Thread Konstantin Knizhnik
On 27.11.2019 19:05, Tomas Vondra wrote: On Wed, Nov 27, 2019 at 06:38:45PM +0300, Konstantin Knizhnik wrote: On 25.11.2019 18:24, Merlin Moncure wrote: On Mon, Nov 25, 2019 at 9:09 AM Konstantin Knizhnik wrote: JIT was not able to significantly (times) increase speed on Q1 query

Re: Why JIT speed improvement is so modest?

2019-11-28 Thread Konstantin Knizhnik
y checks I  want to exclude this overhead and reach almost the same speed as VOPS. But it doesn't happen. Konstantin Knizhnik <mailto:k.knizh...@postgrespro.ru>> 于2019年11月28日周四 下午3:08写道: On 27.11.2019 19:05, Tomas Vondra wrote: > On Wed, Nov 27, 2019 at 06:38:

Re: Yet another vectorized engine

2019-11-28 Thread Konstantin Knizhnik
think about format of storing data in VectorTupleTableSlot? Should it be array of Datum? Or we need to represent vector in more low level format (for example as array of floats for real4 type)? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Global temporary tables

2019-12-02 Thread Konstantin Knizhnik
On 01.12.2019 4:56, Michael Paquier wrote: On Wed, Nov 20, 2019 at 07:32:14PM +0300, Konstantin Knizhnik wrote: Now pg_gtt_statistic view is provided for global temp tables. Latest patch fails to apply, per Mr Robot's report. Could you please rebase and send an updated version? For

Re: [HACKERS] Cached plans and statement generalization

2019-12-02 Thread Konstantin Knizhnik
On 01.12.2019 6:26, Michael Paquier wrote: On Thu, Sep 26, 2019 at 10:23:38AM +0300, Konstantin Knizhnik wrote: Sorry, New version of the patch with corrected expected output for rules test is attached. It looks like the documentation is failing to build. Could you fix that? There may be

Re: Yet another vectorized engine

2019-12-03 Thread Konstantin Knizhnik
rized scan can be combined with parallel execution (it is already supported in9.6, isn't it?) -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Session WAL activity

2019-12-03 Thread Konstantin Knizhnik
h from my point of view is adding 8 bytes to PGPROC. But there are already so many fields in this structure (sizeof(PGPROC)=816), that adding yet another 8 bytes should not be noticeable. Comments are welcome. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russia

Re: Session WAL activity

2019-12-04 Thread Konstantin Knizhnik
. Will fix it. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Yet another vectorized engine

2019-12-04 Thread Konstantin Knizhnik
or parallel search so if we want to provide parallel execution for vectorized operations we need also to substitute this nodes with custom nodes. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Why JIT speed improvement is so modest?

2019-12-05 Thread Konstantin Knizhnik
Hi, Thank you for your replay and explanations. My comments are inside. On 04.12.2019 22:43, Andres Freund wrote: Hi, On 2019-11-25 18:09:29 +0300, Konstantin Knizhnik wrote: I wonder why even at this query, which seems to be ideal use case for JIT, we get such modest improvement? I think

Re: Session WAL activity

2019-12-05 Thread Konstantin Knizhnik
that race conditions here are so critical. Right now pg_stat_activity also accessing PGPROC to obtain wait event information and also not taking any locks. So it can wrongly report backend status. But I never heard that somebody complains about it. -- Konstantin Knizhnik Postgres Profession

Re: Session WAL activity

2019-12-06 Thread Konstantin Knizhnik
On 06.12.2019 4:57, Michael Paquier wrote: On Thu, Dec 05, 2019 at 12:23:40PM +0300, Konstantin Knizhnik wrote: Concerning keeping PGPROC size as small as possible, I agree that it is reasonable argument. But even now it is very large (816 bytes) and adding extra 8 bytes will increase it on

Re: Why JIT speed improvement is so modest?

2019-12-06 Thread Konstantin Knizhnik
On 06.12.2019 18:53, Robert Haas wrote: On Thu, Nov 28, 2019 at 2:08 AM Konstantin Knizhnik wrote: calls float4_accum for each row of T, the same query in VOPS will call vops_float4_avg_accumulate for each tile which contains 64 elements. So vops_float4_avg_accumulate is called 64 times

Re: Why JIT speed improvement is so modest?

2019-12-06 Thread Konstantin Knizhnik
On 06.12.2019 19:52, Konstantin Knizhnik wrote: On 06.12.2019 18:53, Robert Haas wrote: On Thu, Nov 28, 2019 at 2:08 AM Konstantin Knizhnik wrote: calls float4_accum for each row of T, the same query in VOPS will call vops_float4_avg_accumulate for each tile which contains 64 elements

Re: Session WAL activity

2019-12-11 Thread Konstantin Knizhnik
walWritten field. As far as at 64-bit systems, pg_atomic_write_u64and pg_atomic_read_u64 are translated to ordinary memory access, them should not have some negative impact on performance. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/backe

Re: Read Uncommitted

2019-12-18 Thread Konstantin Knizhnik
d by two transactions both of which were started before us and committed during table traversal by transaction with "read uncommitted" policy. Certainly "read uncommitted" means that we are ready to get inconsistent results, but is it really acceptable to multiple versions

Re: Columns correlation and adaptive query optimization

2019-12-24 Thread Konstantin Knizhnik
). -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/contrib/auto_explain/auto_explain.c b/contrib/auto_explain/auto_explain.c index a9536c2..3f9d6ef 100644 --- a/contrib/auto_explain/auto_explain.c +++ b/contrib/auto_explain/auto_explain.c

Re: [Proposal] Global temporary tables

2020-01-09 Thread Konstantin Knizhnik
porary tables is overkill from my point of view. I do not understand why do we need to maintain hash with some extra information for GTT in backends memory (as it was done in Wenjing patch). Also idea to use create extension for accessing this information seems to be dubious. -- Kon

Re: [Proposal] Global temporary tables

2020-01-09 Thread Konstantin Knizhnik
here? The same problem takes place for normal (local) temp tables, doesn't it? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [Proposal] Global temporary tables

2020-01-10 Thread Konstantin Knizhnik
if you can point me on what is wrong with this approach, I will think about alternative solution. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [Proposal] Global temporary tables

2020-01-10 Thread Konstantin Knizhnik
o me that nobody complains about it. We discuss  many issues related with temp tables (statistic, parallel queries,...) which seems to be less critical. But this problem is not specific to GTT - it can be reproduced with normal (local) temp tables. This is why I wonder why do we need to solve

Re: [Proposal] Global temporary tables

2020-01-13 Thread Konstantin Knizhnik
On 12.01.2020 4:51, Tomas Vondra wrote: On Fri, Jan 10, 2020 at 11:47:42AM +0300, Konstantin Knizhnik wrote: On 09.01.2020 19:48, Tomas Vondra wrote: The most complex and challenged task is to support GTT for all kind of indexes. Unfortunately I can not proposed some good universal

Create/alter policy and exclusive table lock

2020-01-14 Thread Konstantin Knizhnik
in pg_policy table and perform RLS checks according to old policies. Once transaction is committed, everybody will switch to new policies. I wonder if we it is possible to replace AccessExclusiveLock with AccessSharedLock in RangeVarGetRelidExtended in CreatePolicy and AlterPolicy? -- Konstan

Re: Create/alter policy and exclusive table lock

2020-01-14 Thread Konstantin Knizhnik
On 14.01.2020 17:40, Tom Lane wrote: Konstantin Knizhnik writes: Right now changing policies (create/alter policy statements) requires exclusive lock of target table: Yup. I wonder why do we really need exclusive lock here? Because it affects the behavior of a SELECT. May be I missed

Re: [Proposal] Global temporary tables

2020-01-15 Thread Konstantin Knizhnik
sting maintenance_work_mem, so this solution is more flexible. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [Proposal] Global temporary tables

2020-01-24 Thread Konstantin Knizhnik
acumm will produce this warnings (which will ton be visible by end user and only appended to the log). And at some moment of time wrap around happen and if there still some old active GTT, we will get incorrect results. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [Proposal] Global temporary tables

2020-01-24 Thread Konstantin Knizhnik
ing table schema. My current solution is to allow creation/droping index on GTT and dropping table itself, while prohibit alter schema at all for GTT. Wenjing's approach is to prohibit any DDL if GTT is used by more than one backend. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [Proposal] Global temporary tables

2020-01-24 Thread Konstantin Knizhnik
On 24.01.2020 12:09, Pavel Stehule wrote: pá 24. 1. 2020 v 9:39 odesílatel Konstantin Knizhnik mailto:k.knizh...@postgrespro.ru>> napsal: On 23.01.2020 23:47, Robert Haas wrote: > On Sat, Jan 11, 2020 at 8:51 PM Tomas Vondra > mailto:tomas.von...@2ndquadrant

Re: [Proposal] Global temporary tables

2020-01-24 Thread Konstantin Knizhnik
nitialize index and populates it with data. So the solution proposed for me is most natural, convenient and simplest solution at the same time. And compatible with Oracle. Regards Pavel -- Konstantin Knizhnik Postgres Professional:http://www.postgrespro.com The Russian P

Re: [Proposal] Global temporary tables

2020-01-27 Thread Konstantin Knizhnik
is to make it compatible with normal tables. And do not forget about compatibility with Oracle. Simplifying of porting existed applications from Oracle to Postgres  may be the main motivation of adding GTT to Postgres. And making them incompatible with Oracle will be very strange. -- Konstantin

Re: [Proposal] Global temporary tables

2020-01-27 Thread Konstantin Knizhnik
expected to live long time. Certainly it is possible to imagine situation when session use GTT to store some local data which is valid during all session life time (which can be large enough). But I am not sure that it is popular scenario. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [Proposal] Global temporary tables

2020-01-29 Thread Konstantin Knizhnik
can be changed... Sorry, but I do not consider proposals to create indexes locally for each session (i.e. global tables but private indexes) or use some special complicated SQL syntax constructions like CREATE INDEX IMMEDIATELY FOR ALL SESSION as some real alternatives which have to be discussed.

Re: [HACKERS] Cached plans and statement generalization

2019-04-03 Thread Konstantin Knizhnik
New version of the patch with fixed error of autopreparing CTE queries. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/doc/src/sgml/autoprepare.sgml b/doc/src/sgml/autoprepare.sgml new file mode 100644 index 000..b3309bd

Inheritance, invalidations and prepared statements.

2019-04-04 Thread Konstantin Knizhnik
permutation: s1b s1delc1 s2prep s2sel s1c s2sel   step s1b: BEGIN;   step s1delc1: ALTER TABLE c1 NO INHERIT p; ! step s2prep: PREPARE summa as SELECT SUM(a) FROM p; ! step s2sel: EXECUTE summa;   step s1c: COMMIT;   step s2sel: <... completed>   sum   11 ! step s2sel: EXECUTE summ

Re: [HACKERS] Cached plans and statement generalization

2019-04-09 Thread Konstantin Knizhnik
New version of the patching disabling autoprepare for rules and handling planner error. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/doc/src/sgml/autoprepare.sgml b/doc/src/sgml/autoprepare.sgml new file mode 100644 index

Re: Zedstore - compressed in-core columnar storage

2019-04-09 Thread Konstantin Knizhnik
in zedstore table... Actually the main question is why this table is not empty if INSERT statement was failed? Please let me know if I can somehow help you to reproduce and investigate the problem. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company vstore_bench.sql Description: application/sql

Re: Zedstore - compressed in-core columnar storage

2019-04-09 Thread Konstantin Knizhnik
On 09.04.2019 17:09, Konstantin Knizhnik wrote: Hi, On 09.04.2019 3:27, Ashwin Agrawal wrote: Heikki and I have been hacking recently for few weeks to implement in-core columnar storage for PostgreSQL. Here's the design and initial implementation of Zedstore, compressed in-core col

Re: Zedstore - compressed in-core columnar storage

2019-04-09 Thread Konstantin Knizhnik
On 09.04.2019 18:08, Heikki Linnakangas wrote: On 09/04/2019 18:00, Konstantin Knizhnik wrote: On 09.04.2019 17:09, Konstantin Knizhnik wrote: standard Postgres heap and my VOPS extension. As test data I used TPC-H benchmark (actually only one lineitem table generated with tpch-dbgen

Re: Zedstore - compressed in-core columnar storage

2019-04-09 Thread Konstantin Knizhnik
On 09.04.2019 18:51, Alvaro Herrera wrote: On 2019-Apr-09, Konstantin Knizhnik wrote: On 09.04.2019 3:27, Ashwin Agrawal wrote: Heikki and I have been hacking recently for few weeks to implement in-core columnar storage for PostgreSQL. Here's the design and initial implementati

Re: Zedstore - compressed in-core columnar storage

2019-04-09 Thread Konstantin Knizhnik
On 09.04.2019 19:19, Heikki Linnakangas wrote: On 09/04/2019 18:00, Konstantin Knizhnik wrote: Looks like the original problem was caused by internal postgres compressor: I have not configured Postgres to use lz4. When I configured Postgres --with-lz4, data was correctly inserted in zedstore

Re: Zedstore - compressed in-core columnar storage

2019-04-10 Thread Konstantin Knizhnik
ize of original Postgres table. It seems to be very strange. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: block-level incremental backup

2019-04-10 Thread Konstantin Knizhnik
l (as pg_basebackup or pg_probackup). -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c index 61a8f11..f4b8506 100644 --- a/src/backend/storage/smgr/md.c +++ b/src/ba

Re: block-level incremental backup

2019-04-10 Thread Konstantin Knizhnik
On 10.04.2019 19:51, Robert Haas wrote: On Wed, Apr 10, 2019 at 10:22 AM Konstantin Knizhnik wrote: Some times ago I have implemented alternative version of ptrack utility (not one used in pg_probackup) which detects updated block at file level. It is very simple and may be it can be

Re: Zedstore - compressed in-core columnar storage

2019-04-11 Thread Konstantin Knizhnik
b) Time (sec) no compression 15.31 92 zlib (default level)2.37284 zlib (best speed) 2.43191 postgres internal lz3.89214 lz4 4.12 95 snappy 5.1899 lzfse 2.801099 (apple) 2.80 1099 1.69125 You see that zstd provide

Re: Zedstore - compressed in-core columnar storage

2019-04-11 Thread Konstantin Knizhnik
On 11.04.2019 16:18, Andreas Karlsson wrote: On 4/11/19 10:46 AM, Konstantin Knizhnik wrote: This my results of compressing pbench data using different compressors: Configuration Size (Gb) Time (sec) no compression 15.31 92 zlib (default level)2.37284 zlib (best

Re: block-level incremental backup

2019-04-22 Thread Konstantin Knizhnik
probackup supports both features: parallel and incremental backups and it is up to user how to use it in more efficient way for particular configuration. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: PROXY protocol support

2019-05-20 Thread Konstantin Knizhnik
d without inventing new protocol. There is my prototype implementation of built-in connection pooler on commit-fest: https://commitfest.postgresql.org/23/2067/ -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Read-only access to temp tables for 2PC transactions

2019-05-24 Thread Konstantin Knizhnik
then most likely writing to temporary table also has to be wrapped in 2PC, which is not possible with the proposed solution. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Pinned files at Windows

2019-05-27 Thread Konstantin Knizhnik
rare. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: Pinned files at Windows

2019-05-27 Thread Konstantin Knizhnik
On 27.05.2019 12:26, Konstantin Knizhnik wrote: Hi, hackers. There is the following problem with Postgres at Windows: files of dropped relation can be blocked for arbitrary long amount of time. Such behavior is caused by two factors: 1. Windows doesn't allow deletion of opened fi

Re: Pinned files at Windows

2019-05-30 Thread Konstantin Knizhnik
On 29.05.2019 22:20, Michael Paquier wrote: On Mon, May 27, 2019 at 05:52:13PM +0300, Konstantin Knizhnik wrote: Postgres is opening file with FILE_SHARE_DELETE  flag which makes it possible to unlink opened file. But unlike Unixes, the file is not actually deleted. You can see it using &quo

Re: Pinned files at Windows

2019-06-03 Thread Konstantin Knizhnik
On 03.06.2019 22:15, Robert Haas wrote: On Thu, May 30, 2019 at 3:25 AM Konstantin Knizhnik wrote: If call of stat() is succeed, then my assumption is that the only reason of GetFileAttributesEx failure is that file is deleted and returning ENOENT error code in this case is correct behavior

out-of-order XID insertion in KnownAssignedXids

2018-10-05 Thread Konstantin Knizhnik
Hi hackers, Looks like there is a bug with logging running transactions XIDs and prepared transactions. One of our customers get error "FATAL: out-of-order XID insertion in KnownAssignedXids" trying to apply backup. WAL contains the following record: rmgr: Standby len (rec/tot): 98/  

Re: out-of-order XID insertion in KnownAssignedXids

2018-10-05 Thread Konstantin Knizhnik
On 05.10.2018 11:04, Michael Paquier wrote: On Fri, Oct 05, 2018 at 10:06:45AM +0300, Konstantin Knizhnik wrote: As you can notice, XID 2004495308 is encountered twice which cause error in KnownAssignedXidsAdd:     if (head > tail &&         TransactionIdFollowsOrEquals(KnownA

now() vs transaction_timestamp()

2018-10-05 Thread Konstantin Knizhnik
ime (1 row) As a result using now() in query disable parallel execution while transaction_timestamp allows it. Was it done intentionally or it is just a bug? -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: now() vs transaction_timestamp()

2018-10-06 Thread Konstantin Knizhnik
On 06.10.2018 00:25, Tom Lane wrote: I wrote: Konstantin Knizhnik writes: Postgres documentation says that |"now()| is a traditional PostgreSQL equivalent to |transaction_timestamp()|". Also both use the same implementation. Right. But them have different parallel safety prope

Re: now() vs transaction_timestamp()

2018-10-06 Thread Konstantin Knizhnik
On 07.10.2018 07:58, Amit Kapila wrote: On Sat, Oct 6, 2018 at 9:40 PM Tom Lane wrote: Konstantin Knizhnik writes: On 06.10.2018 00:25, Tom Lane wrote: So maybe the right answer is to change the parallel mode infrastructure so it transmits xactStartTimestamp, making transaction_timestamp

Re: out-of-order XID insertion in KnownAssignedXids

2018-10-08 Thread Konstantin Knizhnik
On 05.10.2018 11:04, Michael Paquier wrote: On Fri, Oct 05, 2018 at 10:06:45AM +0300, Konstantin Knizhnik wrote: As you can notice, XID 2004495308 is encountered twice which cause error in KnownAssignedXidsAdd:     if (head > tail &&         TransactionIdFollowsOrEquals(KnownA

Re: out-of-order XID insertion in KnownAssignedXids

2018-10-08 Thread Konstantin Knizhnik
On 08.10.2018 12:14, Michael Paquier wrote: On Mon, Oct 08, 2018 at 12:04:28PM +0300, Konstantin Knizhnik wrote: The simplest way to fix the problem is to ignore duplicates before adding them to KnownAssignedXids. We in any case perform sort i this place... I may of course be missing

Re: out-of-order XID insertion in KnownAssignedXids

2018-10-08 Thread Konstantin Knizhnik
On 08.10.2018 18:24, Andres Freund wrote: On October 8, 2018 2:04:28 AM PDT, Konstantin Knizhnik wrote: On 05.10.2018 11:04, Michael Paquier wrote: On Fri, Oct 05, 2018 at 10:06:45AM +0300, Konstantin Knizhnik wrote: As you can notice, XID 2004495308 is encountered twice which cause

Race condition in create table

2018-10-09 Thread Konstantin Knizhnik
t_1" already exists, skipping NOTICE:  relation "t_1" already exists, skipping NOTICE:  relation "t_1" already exists, skipping client 2 aborted in command 0 (SQL) of script 0; ERROR:  relation "t_1" already exists NOTICE:  relation "t_1" already exists, skipping NOTICE:  relation "t_1" already exists, skipping NOTICE:  relation "t_1" already exists, skipping client 5 aborted in command 0 (SQL) of script 0; ERROR:  relation "t_1" already exists client 4 aborted in command 0 (SQL) of script 0; ERROR:  relation "t_1" already exists -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company create_table.sql Description: application/sql create_part.sql Description: application/sql

Re: out-of-order XID insertion in KnownAssignedXids

2018-10-09 Thread Konstantin Knizhnik
is seems to be the most efficient alternative). But I also do not fill that moving sort to GetRunningTransactionData and elimination of duplicates here (requires one more scan and copying of the whole array) is principally better... -- Konstantin Knizhnik Postgres Professional: http

Few remarks on JIT , parallel query execution and columnar store...

2018-10-11 Thread Konstantin Knizhnik
cond it has noticeable impact on total performance. In some other my prototype DBMS with vertical data representation and multhreaded execution time of execution of this query is 195 msec. So there is still scope for improvements:) -- Konstantin Knizhnik Postgres Professional: http://www.post

Re: out-of-order XID insertion in KnownAssignedXids

2018-10-11 Thread Konstantin Knizhnik
that GetRunningTransactionData may return duplicated XIDs because of the dummy 2PC entries which overlap with the active ones, and also add a proper comment in ProcArrayApplyRecoveryInfo(). Konstantin, do you want to give it a try with a patch? Or should I? -- Michael Proposed patch is attached. -- Konst

Re: [HACKERS] Secondary index access optimizations

2018-10-12 Thread Konstantin Knizhnik
On 08.10.2018 00:16, David Rowley wrote: On 5 October 2018 at 04:45, Konstantin Knizhnik wrote: Will the following test be enough: -- check that columns for parent table are correctly mapped to child partition of their order doesn't match create table paren (a int, b text) partiti

Estimating number of distinct values.

2018-10-24 Thread Konstantin Knizhnik
Hello hackers, I will be pleased if somebody (first of all Robert) can comment me "strange" results of distinct values estimation. There is the following code in analyze.c:        /*--              * Estimate the number of distinct values using the estimator              * propos

Re: Parallel threads in query

2018-11-01 Thread Konstantin Knizhnik
allel capabilities of video cards). -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Surjective functional indexes

2018-11-08 Thread Konstantin Knizhnik
at "projection index" is an appropriate term at all, nor do I think that "recheck_on_update" is a particularly helpful option name, though we may be stuck with the latter at this point. As I am not native english speaking, I will agree with any proposed terminology. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Surjective functional indexes

2018-11-08 Thread Konstantin Knizhnik
ion indexes I want to attach small bug fix patch which fixes the original problem (SIGSEGV) and also disables recheck_on_update by default. As Laurenz has suggested, I replaced boolean recheck_on_update option with "on","auto,"off" (default). -- Konstantin Knizhnik Post

Re: [HACKERS] Surjective functional indexes

2018-11-09 Thread Konstantin Knizhnik
t;'title')); Document description may include many attributes which are updated quite frequently, like "comments", "keywords",... But "title" is rarely changed, so this optimization will be very useful for such index. -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company

Re: [HACKERS] Surjective functional indexes

2018-11-09 Thread Konstantin Knizhnik
On 08.11.2018 22:05, Alvaro Herrera wrote: On 2018-Nov-08, Konstantin Knizhnik wrote: Before doing any other refactoring of projection indexes I want to attach small bug fix patch which fixes the original problem (SIGSEGV) and also disables recheck_on_update by default. As Laurenz has

Index-only scan is slower than Index scan.

2018-11-23 Thread Konstantin Knizhnik
mail. After applying it index-only scan takes almost the same time as index scan: index scan 1.995 indexonly scan (original) 2.686 indexonly scan (patch) 2.005 -- Konstantin Knizhnik Postgres Professional: http://www.postgrespro.com The Russian Postgres Company diff --git a/src

Re: VOPS-2.0

2018-11-28 Thread Konstantin Knizhnik
On 28.11.2018 16:18, Bruce Momjian wrote: On Wed, Nov 28, 2018 at 01:01:03PM +0300, Konstantin Knizhnik wrote: Hi, I want to introduce new version of VOPS extension for Postgres (vectorized operations) providing new, more convenient way of usage: auto-substitution of projections. [Announce

Re: VOPS-2.0

2018-11-28 Thread Konstantin Knizhnik
On 28.11.2018 16:45, Alvaro Herrera wrote: On 2018-Nov-28, Bruce Momjian wrote: On Wed, Nov 28, 2018 at 01:01:03PM +0300, Konstantin Knizhnik wrote: Hi, I want to introduce new version of VOPS extension for Postgres (vectorized operations) providing new, more convenient way of usage: auto

<    1   2   3   4   5   6   7   8   >