Re: Defer selection of asynchronous subplans until the executor initialization stage

2022-04-04 Thread Andrey V. Lepikhov
On 4/3/22 15:29, Etsuro Fujita wrote: On Sun, Mar 13, 2022 at 6:39 PM Etsuro Fujita wrote: On Wed, Sep 15, 2021 at 3:40 PM Alexander Pyhalov wrote: The patch looks good to me and seems to work as expected. I’m planning to commit the patch. I polished the patch a bit: * Reordered a bit of

Re: Removing unneeded self joins

2022-04-04 Thread Andrey V. Lepikhov
just memorialize an incorrect output. Btw, it's the last week before feature freeze so time is of the essence.Thanks, patch in attachment rebased on current master. Sorry for late answer. -- regards, Andrey Lepikhov Postgres ProfessionalFrom d5fab52bd7e7124d0e557f1eec075a9543c67d29 Mon Sep 17 00:0

Re: Fast COPY FROM based on batch insert

2022-03-23 Thread Andrey V. Lepikhov
t. I didn’t finish my review, but I’ll mark this as “Waiting on Author”. I rebased the patch onto current master. Now it works correctly. I'll mark it as "Waiting for review". -- regards, Andrey Lepikhov Postgres ProfessionalFrom 2d51d0f5d94a3e4b3400714b5841228d1896fb56 Mon Se

Re: Removing unneeded self joins

2022-03-23 Thread Andrey V. Lepikhov
ep 17 00:00:00 2001 From: "Andrey V. Lepikhov" Date: Thu, 15 Jul 2021 15:26:13 +0300 Subject: [PATCH] Remove self-joins. Remove inner joins of a relation to itself if could prove that the join can be replaced with a scan. We can prove the uniqueness using the existing innerrel_is_unique

Re: POC: GROUP BY optimization

2022-03-18 Thread Andrey V. Lepikhov
On 3/15/22 13:26, Tomas Vondra wrote: Thanks for the rebase. The two proposed changes (tweaked costing and simplified fake_var handling) seem fine to me. I think the last thing that needs to be done is cleanup of the debug GUCs, which I added to allow easier experimentation with the patch. Thank

Re: Merging statistics from children instead of re-sampling everything

2022-02-18 Thread Andrey V. Lepikhov
On 2/14/22 20:16, Tomas Vondra wrote: On 2/14/22 11:22, Andrey V. Lepikhov wrote: On 2/11/22 20:12, Tomas Vondra wrote: On 2/11/22 05:29, Andrey V. Lepikhov wrote: On 2/11/22 03:37, Tomas Vondra wrote: That being said, this thread was not really about foreign partitions, but about re

Re: Merging statistics from children instead of re-sampling everything

2022-02-14 Thread Andrey V. Lepikhov
On 2/11/22 20:12, Tomas Vondra wrote: On 2/11/22 05:29, Andrey V. Lepikhov wrote: On 2/11/22 03:37, Tomas Vondra wrote: That being said, this thread was not really about foreign partitions, but about re-analyzing inheritance trees in general. And sampling foreign partitions doesn't r

Re: Merging statistics from children instead of re-sampling everything

2022-02-10 Thread Andrey V. Lepikhov
On 2/11/22 03:37, Tomas Vondra wrote: That being said, this thread was not really about foreign partitions, but about re-analyzing inheritance trees in general. And sampling foreign partitions doesn't really solve that - we'll still do the sampling over and over. IMO, to solve the problem we sho

Re: POC: GROUP BY optimization

2022-02-10 Thread Andrey V. Lepikhov
On 1/22/22 01:34, Tomas Vondra wrote: I rebased (with minor fixes) this patch onto current master. Also, second patch dedicated to a problem of "varno 0" (fake_var). I think, this case should make the same estimations as in the case of varno != 0, but no any stats found. So I suppose to rest

Re: POC: GROUP BY optimization

2022-02-01 Thread Andrey V. Lepikhov
On 7/22/21 3:58 AM, Tomas Vondra wrote: I've simplified the costing a bit, and the attached version actually undoes all the "suspicious" plan changes in postgres_fdw. It changes one new plan, but that seems somewhat reasonable, as it pushes sort to the remote side. I tried to justify heap-sort p

Re: Multiple Query IDs for a rewritten parse tree

2022-01-31 Thread Andrey V. Lepikhov
On 1/28/22 9:51 PM, Dmitry Dolgov wrote: On Fri, Jan 21, 2022 at 11:33:22AM +0500, Andrey V. Lepikhov wrote: Registration of an queryId generator implemented by analogy with extensible methods machinery. Why not more like suggested with stakind and slots in some data structure? All of those

Re: Multiple Query IDs for a rewritten parse tree

2022-01-20 Thread Andrey V. Lepikhov
On 1/9/22 5:49 AM, Tom Lane wrote: The idea I'd been vaguely thinking about is to allow attaching a list of query-hash nodes to a Query, where each node would contain a "tag" identifying the specific hash calculation method, and also the value of the query's hash calculated according to that meth

Re: POC: GROUP BY optimization

2022-01-19 Thread Andrey V. Lepikhov
I keep work on this patch. Here is intermediate results. On 7/22/21 3:58 AM, Tomas Vondra wrote: in the first loop. Which seems pretty bogus - why would there be just two groups? When processing the first expression, it's as if there was one big "prev group" with all the tuples, so why not to ju

Re: Multiple Query IDs for a rewritten parse tree

2022-01-09 Thread Andrey V. Lepikhov
On 1/10/22 9:51 AM, Julien Rouhaud wrote: On Mon, Jan 10, 2022 at 09:10:59AM +0500, Andrey V. Lepikhov wrote: I can add one more use case. Our extension for freezing query plan uses query tree comparison technique to prove, that the plan can be applied (and we don't need to execute pla

Re: Multiple Query IDs for a rewritten parse tree

2022-01-09 Thread Andrey V. Lepikhov
On 1/9/22 5:13 PM, Julien Rouhaud wrote: For now the queryid mixes two different things: fingerprinting and query text normalization. Should each calculation method be allowed to do a different normalization too, and if yes where should be stored the state data needed for that? If not, we would

Re: pg_stat_statements and "IN" conditions

2022-01-04 Thread Andrey V. Lepikhov
On 1/5/22 4:02 AM, Tom Lane wrote: Dmitry Dolgov <9erthali...@gmail.com> writes: And now for something completely different, here is a new patch version. It contains a small fix for one problem we've found during testing (one path code was incorrectly assuming find_const_walker results). I've

Re: Clarify planner_hook calling convention

2022-01-04 Thread Andrey V. Lepikhov
On 1/3/22 8:59 PM, Tom Lane wrote: "Andrey V. Lepikhov" writes: planner hook is frequently used in monitoring and advising extensions. Yeah. The call to this hook is implemented in the way, that the standard_planner routine must be called at least once in the hook's call c

Clarify planner_hook calling convention

2022-01-02 Thread Andrey V. Lepikhov
Hi, planner hook is frequently used in monitoring and advising extensions. The call to this hook is implemented in the way, that the standard_planner routine must be called at least once in the hook's call chain. But, as I see in [1], it should allow us "... replace the planner altogether".

Re: Look at all paths?

2021-12-28 Thread Andrey V. Lepikhov
On 12/29/21 5:07 AM, Chris Cleveland wrote: I'm developing a new index access method. Sometimes the planner uses it and sometimes it doesn't. I'm trying to debug the process to understand why the index does or doesn't get picked up. Is there a way to dump all of the query plans that the planne

Re: POC: GROUP BY optimization

2021-12-28 Thread Andrey V. Lepikhov
On 9/2/20 9:12 PM, Tomas Vondra wrote: > We could simply use the input "tuples" value here, and then divide the > current and previous estimate to calculate the number of new groups. Performing a review of this patch I made a number of changes (see cleanup.txt). Maybe it will be useful. As I see

Re: Global snapshots

2021-11-19 Thread Andrey V. Lepikhov
Patch in the previous letter is full of faulties. Please, use new version. Also, here we fixed the problem with loosing CSN value in a parallel worker (TAP test 003_parallel_safe.pl). Thanks for a.pyhalov for the problem detection and a bugfix. -- regards, Andrey Lepikhov Postgres Professional

Re: Global snapshots

2021-11-17 Thread Andrey V. Lepikhov
Next version of CSN implementation in snapshots to achieve a proper snapshot isolation in the case of a cross-instance distributed transaction. -- regards, Andrey Lepikhov Postgres Professional >From bbb7dd1d7621c091f11e697d3d894fe7a36918a6 Mon Sep 17 00:00:00 2001 From: Andrey Lepikhov Date: W

Make query ID more portable

2021-10-12 Thread Andrey V. Lepikhov
Hi, QueryID is good tool for query analysis. I want to improve core jumbling machinery in two ways: 1. QueryID value should survive dump/restore of a database (use fully qualified name of table instead of relid). 2. QueryID could represent more general class of queries: for example, it can be

Re: Asymmetric partition-wise JOIN

2021-09-13 Thread Andrey V. Lepikhov
On 9/9/21 8:38 PM, Jaime Casanova wrote: On Thu, Sep 09, 2021 at 09:50:46AM +, Aleksander Alekseev wrote: It looks like this patch needs to be updated. According to http://cfbot.cputube.org/ it applies but doesn't pass any tests. Changing the status to save time for reviewers. The new sta

Re: Increase value of OUTER_VAR

2021-09-13 Thread Andrey V. Lepikhov
On 9/11/21 10:37 PM, Tom Lane wrote: Aleksander Alekseev writes: (v2 below is a rebase up to HEAD; no actual code changes except for adjusting the definition of IS_SPECIAL_VARNO.) I have looked at this code. No problems found. Also, as a test, I used two tables with 1E5 partitions each. I tried

Re: Defer selection of asynchronous subplans until the executor initialization stage

2021-08-30 Thread Andrey V. Lepikhov
On 8/23/21 2:18 PM, Etsuro Fujita wrote: To just execute what was planned at execution time, I think we should return to the patch in [1]. The patch was created for Horiguchi-san’s async-execution patch, so I modified it to work with HEAD, and added a simplified version of your test cases. Plea

Re: Postgres picks suboptimal index after building of an extended statistics

2021-08-13 Thread Andrey V. Lepikhov
On 8/12/21 4:26 AM, Tomas Vondra wrote: On 8/11/21 2:48 AM, Peter Geoghegan wrote: On Wed, Jun 23, 2021 at 7:19 AM Andrey V. Lepikhov wrote: Ivan Frolkov reported a problem with choosing a non-optimal index during a query optimization. This problem appeared after building of an extended

Extra code in commit_ts.h

2021-08-03 Thread Andrey V. Lepikhov
Hi, I found two extra code lines in commit_ts.h (see attachment). They confused me during exploring of the code. If they still needed, may be add some comments? -- regards, Andrey Lepikhov Postgres Professional diff --git a/src/include/access/commit_ts.h b/src/include/access/commit_ts.h index

Re: Removing unneeded self joins

2021-07-26 Thread Andrey V. Lepikhov
, root->rowMarks) It seems once imark and omark are set, we can come out of the loop. Maybe you right. fixed. -- regards, Andrey Lepikhov Postgres Professional >From e8b4047aa71c808fa5799b2739b2ae0ab7b6d7e3 Mon Sep 17 00:00:00 2001 From: "Andrey V. Lepikhov" Date: Thu, 15 Jul 2021

Postgres picks suboptimal index after building of an extended statistics

2021-06-23 Thread Andrey V. Lepikhov
Hi, Ivan Frolkov reported a problem with choosing a non-optimal index during a query optimization. This problem appeared after building of an extended statistics. I prepared the test case (see t.sql in attachment). For reproduction of this case we need to have a composite primary key index a

Re: Removing unneeded self joins

2021-05-26 Thread Andrey V. Lepikhov
look further */ } if (best_score < 0) >From 836049b1467ded2f257ffe1844e5656b3f273d6c Mon Sep 17 00:00:00 2001 From: "Andrey V. Lepikhov" Date: Wed, 28 Apr 2021 18:27:53 +0500 Subject: [PATCH] Remove self-joins. Remove inner joins of a relation to itself if could prove that

Re: Asymmetric partition-wise JOIN

2021-04-29 Thread Andrey V. Lepikhov
On 11/30/20 7:43 PM, Anastasia Lubennikova wrote: This entry was inactive during this CF, so I've marked it as returned with feedback. Feel free to resubmit an updated version to a future commitfest. I return the patch to commitfest. My current reason differs from reason of origin author. This

Re: Asynchronous Append on postgres_fdw nodes.

2021-04-26 Thread Andrey V. Lepikhov
On 4/23/21 8:12 AM, Etsuro Fujita wrote: On Thu, Apr 22, 2021 at 12:30 PM Etsuro Fujita wrote: I have committed the patch. One more question. Append choose async plans at the stage of the Append plan creation. Later, the planner performs some optimizations, such as eliminating trivial Subque

Re: Asynchronous Append on postgres_fdw nodes.

2021-04-26 Thread Andrey V. Lepikhov
On 4/23/21 8:12 AM, Etsuro Fujita wrote: I have committed the patch. Small mistake i found. If no tuple was received from a foreign partition, explain shows that we never executed node. For example, if we have 0 tuples in f1 and 100 tuples in f2: Query: EXPLAIN (ANALYZE, VERBOSE, TIMING OFF, C

Re: Asynchronous Append on postgres_fdw nodes.

2021-04-25 Thread Andrey V. Lepikhov
On 4/23/21 8:12 AM, Etsuro Fujita wrote: I have committed the patch. While studying the capabilities of AsyncAppend, I noticed an inconsistency with the cost model of the optimizer: async_capable = off: Append (cost=100.00..695.00 ...) -> Foreign Scan on f1 part_1 (c

Re: Asymmetric partition-wise JOIN

2021-04-09 Thread Andrey V. Lepikhov
On 11/30/20 7:43 PM, Anastasia Lubennikova wrote: This entry was inactive during this CF, so I've marked it as returned with feedback. Feel free to resubmit an updated version to a future commitfest. Attached version is rebased on current master and fixes problems with complex parameterized p

Re: Increase value of OUTER_VAR

2021-04-07 Thread Andrey V. Lepikhov
On 4/8/21 8:13 AM, Tom Lane wrote: I wrote: Peter Eisentraut writes: Can we move forward with this? We could just push the change and see what happens. But I was thinking more in terms of doing that early in the v15 cycle. I remain skeptical that we need a near-term fix. To make sure we

Re: Global snapshots

2021-02-25 Thread Andrey V. Lepikhov
On 1/1/21 8:14 AM, tsunakawa.ta...@fujitsu.com wrote: -- 11. A method comprising: receiving information relating to a distributed database transaction operating on data in data stores associated with respective participating nodes associated with

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-16 Thread Andrey V. Lepikhov
On 2/15/21 1:31 PM, Amit Langote wrote: Tsunakawa-san, Andrey, +static void +postgresBeginForeignCopy(ModifyTableState *mtstate, + ResultRelInfo *resultRelInfo) +{ ... + if (resultRelInfo->ri_RangeTableIndex == 0) + { + ResultRelInfo *rootResultRelInfo = resultR

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-09 Thread Andrey V. Lepikhov
On 2/9/21 12:47 PM, tsunakawa.ta...@fujitsu.com wrote: From: Andrey V. Lepikhov I guess you used many hash partitions. Sadly, The current COPY implementation only accumulates either 1,000 rows or 64 KB of input data (very small!) before flushing all CopyMultiInsertBuffers. One

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-08 Thread Andrey V. Lepikhov
On 2/9/21 9:35 AM, tsunakawa.ta...@fujitsu.com wrote: From: tsunakawa.ta...@fujitsu.com From: Andrey Lepikhov Also, I might defer working on the extended part (v9 0003 and 0004) and further separate them in a different thread, if it seems to take longer. I reviewed them but haven't r

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-01-11 Thread Andrey V. Lepikhov
On 1/11/21 11:16 PM, Tomas Vondra wrote: Hi Andrey, Unfortunately, this no longer applies :-( I tried to apply this on top of c532d15ddd (from 2020/12/30) but even that has non-trivial conflicts. Can you send a rebased version? regards Applied on 044aa9e70e. -- regards, Andrey Lepikhov Post

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2021-01-11 Thread Andrey V. Lepikhov
On 1/11/21 4:59 PM, Tang, Haiying wrote: Hi Andrey, I had a general look at this extension feature, I think it's beneficial for some application scenarios of PostgreSQL. So I did 7 performance cases test on your patch(v13). The results are really good. As you can see below we can get 7-10 tim

Re: Removing unneeded self joins

2021-01-10 Thread Andrey V. Lepikhov
On 1/7/21 7:08 PM, Masahiko Sawada wrote: On Mon, Nov 30, 2020 at 2:51 PM Andrey V. Lepikhov wrote: Thanks, it is my fault. I tried to extend this patch with foreign key references and made a mistake. Currently I rollback this new option (see patch in attachment), but will be working for a

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-12-23 Thread Andrey V. Lepikhov
On 12/22/20 12:04 PM, Tang, Haiying wrote: Hi Andrey, There is an error report in your patch as follows. Please take a check. https://travis-ci.org/github/postgresql-cfbot/postgresql/jobs/750682857#L1519 copyfrom.c:374:21: error: ‘save_cur_lineno’ is used uninitialized in this function [-Wer

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-12-14 Thread Andrey V. Lepikhov
On 12/1/20 2:02 PM, Amit Langote wrote: On Tue, Dec 1, 2020 at 2:40 PM tsunakawa.ta...@fujitsu.com wrote: From: Amit Langote >> The code appears to require both BeginForeignCopy and EndForeignCopy, >> while the following documentation says they are optional. Which is >> correct? (I suppose

Re: Asynchronous Append on postgres_fdw nodes.

2020-12-09 Thread Andrey V. Lepikhov
On 11/17/20 2:56 PM, Etsuro Fujita wrote: On Mon, Oct 5, 2020 at 3:35 PM Etsuro Fujita wrote: Comments welcome! The attached is still WIP and maybe I'm missing something, though. I reviewed your patch and used it in my TPC-H benchmarks. It is still WIP. Will you improve this patch? I also wa

Re: Cost overestimation of foreign JOIN

2020-12-02 Thread Andrey V. Lepikhov
On 12/1/20 6:17 PM, Ashutosh Bapat wrote: On Mon, Nov 30, 2020 at 11:56 PM Andrey Lepikhov wrote: On 30.11.2020 22:38, Tom Lane wrote: Andrey Lepikhov writes: If you're unhappy with the planning results you get for this, why don't you have use_remote_estimate turned on? I have a mixed load

Re: Removing unneeded self joins

2020-11-29 Thread Andrey V. Lepikhov
On 11/29/20 10:10 PM, Heikki Linnakangas wrote: On 28/11/2020 19:21, Andrey Lepikhov wrote: On 27.11.2020 21:49, Heikki Linnakangas wrote: CREATE TABLE a(x int, y int); CREATE UNIQUE INDEX ON a(x); SELECT a1.* FROM a a1, a a2 WHERE a1.x = a2.x;  -- self-join CREATE UNIQUE INDEX ON a(y); SELECT a

Re: Removing unneeded self joins

2020-10-31 Thread Andrey V. Lepikhov
Thank you for this partial review, I included your changes: On 9/23/20 9:23 AM, David Rowley wrote: On Fri, 3 Apr 2020 at 17:43, Andrey Lepikhov wrote: Doing thing the way I describe will allow you to get rid of all the UniqueRelInfo stuff. Thanks for the review and sorry for the late reply. I

Re: Asynchronous Append on postgres_fdw nodes.

2020-10-08 Thread Andrey V. Lepikhov
On 10/5/20 11:35 AM, Etsuro Fujita wrote: Hi, I found a small problem. If we have a mix of async and sync subplans when we catch an assertion on a busy connection. Just for example: PLAN Nested Loop (cost=100.00..174316.95 rows=975 width=8) (actual time=5.191..9.262 rows=9 loops=1) J

Re: Adding Support for Copy callback functionality on COPY TO api

2020-09-30 Thread Andrey V. Lepikhov
On 7/2/20 2:41 AM, Sanaba, Bilva wrote: Hi hackers, Currently, the COPY TO api does not support callback functions, while the COPY FROM api does. The COPY TO code does, however, include placeholders for supporting callbacks in the future. Rounding out the support of callback functions to bot

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-10 Thread Andrey V. Lepikhov
On 9/9/20 5:51 PM, Amit Langote wrote: On Wed, Sep 9, 2020 at 6:42 PM Alexey Kondratov wrote: On 2020-09-09 11:45, Andrey V. Lepikhov wrote: This does not seem very convenient and will lead to errors in the future. So, I agree with Amit. And InitResultRelInfo() may set ri_usesMultiInsert to

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-09 Thread Andrey V. Lepikhov
Version 8 split into two patches (in accordance with Amit suggestion). Also I eliminate naming inconsistency (thanks to Alexey). Based on master, f481d28232. -- regards, Andrey Lepikhov Postgres Professional >From 21b11f4ec0bec71bc7226014ef15c58dee9002da Mon Sep 17 00:00:00 2001 From: amitlan Da

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-09 Thread Andrey V. Lepikhov
On 9/8/20 8:34 PM, Alexey Kondratov wrote: On 2020-09-08 17:00, Amit Langote wrote: wrote: On 2020-09-08 10:34, Amit Langote wrote: Another ambiguous part of the refactoring was in changing InitResultRelInfo() arguments: @@ -1278,6 +1280,7 @@ InitResultRelInfo(ResultRelInfo *resultRelInfo,   

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-08 Thread Andrey V. Lepikhov
On 9/8/20 12:34 PM, Amit Langote wrote: Hi Andrey, On Mon, Sep 7, 2020 at 7:31 PM Andrey V. Lepikhov wrote: On 9/7/20 12:26 PM, Michael Paquier wrote: While on it, the CF bot is telling that the documentation of the patch fails to compile. This needs to be fixed. -- Michael v.7 (in

Re: Ideas about a better API for postgres_fdw remote estimates

2020-09-07 Thread Andrey V. Lepikhov
On 9/4/20 6:23 PM, Ashutosh Bapat wrote: On Thu, 3 Sep 2020 at 10:44, Andrey V. Lepikhov mailto:a.lepik...@postgrespro.ru>> wrote: On 8/31/20 6:19 PM, Ashutosh Bapat wrote: > On Mon, Aug 31, 2020 at 3:36 PM Andrey V. Lepikhov > mailto:a.lepik...@postgrespr

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-07 Thread Andrey V. Lepikhov
On 9/7/20 12:26 PM, Michael Paquier wrote: On Mon, Aug 24, 2020 at 06:19:28PM +0900, Amit Langote wrote: On Mon, Aug 24, 2020 at 4:18 PM Amit Langote wrote: I would Oops, thought I'd continue writing, but hit send before actually doing that. Please ignore. I have some comments on v6, which

Re: Ideas about a better API for postgres_fdw remote estimates

2020-09-02 Thread Andrey V. Lepikhov
On 8/31/20 6:19 PM, Ashutosh Bapat wrote: On Mon, Aug 31, 2020 at 3:36 PM Andrey V. Lepikhov wrote: Thanks for this helpful feedback. I think the patch has some other problems like it works only for regular tables on foreign server but a foreign table can be pointing to any relation like a

Re: Ideas about a better API for postgres_fdw remote estimates

2020-08-31 Thread Andrey V. Lepikhov
On 8/29/20 9:50 PM, Tom Lane wrote: Years ago (when I was still at Salesforce, IIRC, so ~5 years) we had some discussions about making it possible for pg_dump and/or pg_upgrade to propagate stats data forward to the new database. There is at least one POC patch in the archives for doing that by

Re: Asymmetric partition-wise JOIN

2020-08-20 Thread Andrey V. Lepikhov
On 7/1/20 2:10 PM, Daniel Gustafsson wrote: On 27 Dec 2019, at 08:34, Kohei KaiGai wrote: The attached v2 fixed the problem, and regression test finished correctly. This patch no longer applies to HEAD, please submit an rebased version. Marking the entry Waiting on Author in the meantime.

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-29 Thread Andrey V. Lepikhov
On 7/29/20 1:03 PM, Amit Langote wrote: Hi Andrey, Thanks for updating the patch. I will try to take a look later. On Wed, Jul 22, 2020 at 6:09 PM Andrey V. Lepikhov wrote: On 7/16/20 2:14 PM, Amit Langote wrote: * Why the "In" in these API names? + /* COPY a bulk of tup

Re: Global snapshots

2020-07-26 Thread Andrey V. Lepikhov
On 7/27/20 11:22 AM, tsunakawa.ta...@fujitsu.com wrote: Hi Andrey san, Movead san, From: tsunakawa.ta...@fujitsu.com While Clock-SI seems to be considered the best promising for global serializability here, * Why does Clock-SI gets so much attention? How did Clock-SI become the only choice?

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-22 Thread Andrey V. Lepikhov
On 7/16/20 2:14 PM, Amit Langote wrote: Amit Langote EnterpriseDB: http://www.enterprisedb.com Version 5 of the patch. With changes caused by Amit's comments. -- regards, Andrey Lepikhov Postgres Professional >From 24465d61d6f0ec6a45578d252bda1690ac045543 Mon Sep 17 00:00:00 2001 From: Andrey

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-22 Thread Andrey V. Lepikhov
On 7/16/20 2:14 PM, Amit Langote wrote: Hi Andrey, Thanks for this work. I have been reading through your patch and here's a what I understand it does and how: The patch aims to fix the restriction that COPYing into a foreign table can't use multi-insert buffer mechanism effectively. That's b

Re: Partitioning and postgres_fdw optimisations for multi-tenancy

2020-07-16 Thread Andrey V. Lepikhov
On 7/16/20 9:35 PM, Etsuro Fujita wrote: On Thu, Jul 16, 2020 at 8:56 PM Andrey Lepikhov wrote: On 7/16/20 9:55 AM, Etsuro Fujita wrote: On Tue, Jul 14, 2020 at 12:48 AM Alexey Kondratov wrote: Some real-life test queries show, that all single-node queries aren't pushed-down to the require

Re: POC and rebased patch for CSN based snapshots

2020-07-15 Thread Andrey V. Lepikhov
On 7/13/20 11:46 AM, movead...@highgo.ca wrote: I continue to see your patch. Some code improvements see at the attachment. Questions: * csnSnapshotActive is the only member of the CSNshapshotShared struct. * The WriteAssignCSNXlogRec() routine. I din't understand why you add 20 nanosec to curr

Re: POC and rebased patch for CSN based snapshots

2020-07-12 Thread Andrey V. Lepikhov
On 7/4/20 7:56 PM, movead...@highgo.ca wrote: As far as I know about Clock-SI, left part of the blue line will setup as a snapshot if master require a snapshot at time t1. But in fact data A should in snapshot but not and data B should out of snapshot but not. If thi

Re: POC: postgres_fdw insert batching

2020-07-09 Thread Andrey V. Lepikhov
On 6/28/20 8:10 PM, Tomas Vondra wrote: Now, the primary reason why the performance degrades like this is that while FDW has batching for SELECT queries (i.e. we read larger chunks of data from the cursors), we don't have that for INSERTs (or other DML). Every time you insert a row, it has to go

Re: Asymmetric partition-wise JOIN

2020-07-06 Thread Andrey V. Lepikhov
On 12/27/19 12:34 PM, Kohei KaiGai wrote: The attached v2 fixed the problem, and regression test finished correctly. Using your patch I saw incorrect value of predicted rows at the top node of the plan: "Append (cost=270.02..35165.37 rows=40004 width=16)" Full explain of the query plan see in a

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-07-02 Thread Andrey V. Lepikhov
On 6/22/20 5:11 PM, Ashutosh Bapat wrote: mailto:a.lepik...@postgrespro.ru>> wrote: It looks like we call BeginForeignInsert and EndForeignInsert even though actual copy is performed using BeginForeignCopy, ExecForeignCopy and EndForeignCopy. BeginForeignInsert constructs the INSERT query whic

Re: POC and rebased patch for CSN based snapshots

2020-07-02 Thread Andrey V. Lepikhov
On 7/2/20 7:31 PM, Movead Li wrote: Thanks for the remarks, >Some remarks on your patch: >1. The variable last_max_csn can be an atomic variable. Yes will consider. >2. GenerateCSN() routine: in the case than csn < csnState->last_max_csn >This is the case when someone changed the value of t

Re: POC and rebased patch for CSN based snapshots

2020-06-29 Thread Andrey V. Lepikhov
On 6/12/20 2:41 PM, movead...@highgo.ca wrote: Hello hackers, Currently, I do some changes based on the last version: 1. Catch up to the current  commit (c2bd1fec32ab54). 2. Add regression and document. 3. Add support to switch from xid-base snapshot to csn-base snapshot, and the same with stand

Re: Global snapshots

2020-06-19 Thread Andrey V. Lepikhov
On 6/19/20 11:48 AM, Amit Kapila wrote: On Wed, Jun 10, 2020 at 8:36 AM Andrey V. Lepikhov wrote: On 09.06.2020 11:41, Fujii Masao wrote: The patches seem not to be registered in CommitFest yet. Are you planning to do that? Not now. It is a sharding-related feature. I'm not sure that

Re: Asynchronous Append on postgres_fdw nodes.

2020-06-17 Thread Andrey V. Lepikhov
On 6/16/20 1:30 PM, Kyotaro Horiguchi wrote: They return 25056 rows, which is far more than 9741 rows. So remote join won. Of course the number of returning rows is not the only factor of the cost change but is the most significant factor in this case. Thanks for the attention. I see one sligh

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-06-16 Thread Andrey V. Lepikhov
On 6/15/20 10:26 AM, Ashutosh Bapat wrote: Thanks Andrey for the patch. I am glad that the patch has taken care of some corner cases already but there exist still more. COPY command constructed doesn't take care of dropped columns. There is code in deparseAnalyzeSql which constructs list of colu

Re: Asynchronous Append on postgres_fdw nodes.

2020-06-15 Thread Andrey V. Lepikhov
On 6/15/20 1:29 PM, Kyotaro Horiguchi wrote: Thanks for testing, but.. At Mon, 15 Jun 2020 08:51:23 +0500, "Andrey V. Lepikhov" wrote in The patch has a problem with partitionwise aggregates. Asynchronous append do not allow the planner to use partial aggregates. Example you

Re: Asynchronous Append on postgres_fdw nodes.

2020-06-14 Thread Andrey V. Lepikhov
The patch has a problem with partitionwise aggregates. Asynchronous append do not allow the planner to use partial aggregates. Example you can see in attachment. I can't understand why: costs of partitionwise join are less. Initial script and explains of the query with and without the patch you

Re: Asynchronous Append on postgres_fdw nodes.

2020-06-11 Thread Andrey V. Lepikhov
On 6/10/20 8:05 AM, Kyotaro Horiguchi wrote: Hello, Andrey. At Tue, 9 Jun 2020 14:20:42 +0500, Andrey Lepikhov wrote in On 6/4/20 11:00 AM, Kyotaro Horiguchi wrote: 2. Total cost of an Append node is a sum of the subplans. Maybe in the case of asynchronous append we need to use some reduce fa

Re: Global snapshots

2020-06-09 Thread Andrey V. Lepikhov
On 09.06.2020 11:41, Fujii Masao wrote: On 2020/05/12 19:24, Andrey Lepikhov wrote: Rebased onto current master (fb544735f1). Thanks for the patches! These patches are no longer applied cleanly and caused the compilation failure. So could you rebase and update them? Rebased onto 57cb806

Re: [PATCH] Timestamp for a XLOG_BACKUP_END WAL-record

2018-07-12 Thread Andrey V. Lepikhov
On 10.07.2018 22:26, Fujii Masao wrote: On Tue, Jul 10, 2018 at 6:41 PM, Andrey V. Lepikhov wrote: On 10.07.2018 06:45, Andres Freund wrote: Hi, On 2018-07-10 06:41:32 +0500, Andrey V. Lepikhov wrote: This functionality is needed in practice when we have to determine a recovery time

Re: [PATCH] Timestamp for a XLOG_BACKUP_END WAL-record

2018-07-10 Thread Andrey V. Lepikhov
On 10.07.2018 06:45, Andres Freund wrote: Hi, On 2018-07-10 06:41:32 +0500, Andrey V. Lepikhov wrote: This functionality is needed in practice when we have to determine a recovery time of specific backup. What do you mean by "recovery time of specific backup"? recovery time -

[PATCH] Timestamp for a XLOG_BACKUP_END WAL-record

2018-07-09 Thread Andrey V. Lepikhov
record. -- Andrey Lepikhov Postgres Professional: https://postgrespro.com The Russian Postgres Company >From 8852a64156e7726bae11c1904b142c9b157cf654 Mon Sep 17 00:00:00 2001 From: "Andrey V. Lepikhov" Date: Mon, 9 Jul 2018 10:57:10 +0500 Subject: [PATCH] BACKUP_END timestamp addit

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-07-03 Thread Andrey V. Lepikhov
ZtP4e%2BW64HtVTdOenqd1d7HjJL3xZQ%40mail.gmail.com -- Andrey Lepikhov Postgres Professional: https://postgrespro.com The Russian Postgres Company >From 1c8569abe9479e547911ec3079633f79056eff96 Mon Sep 17 00:00:00 2001 From: "Andrey V. Lepikhov" Date: Tue, 3 Jul 2018 16:54:46 +0500 Subjec

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-07-02 Thread Andrey V. Lepikhov
On 29.06.2018 14:07, Юрий Соколов wrote: чт, 28 июн. 2018 г., 8:37 Andrey V. Lepikhov <mailto:a.lepik...@postgrespro.ru>>: On 28.06.2018 05:00, Peter Geoghegan wrote: > On Tue, Jun 26, 2018 at 11:40 PM, Andrey V. Lepikhov > mailto:a.lepik...@postgrespro.ru>

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-06-28 Thread Andrey V. Lepikhov
On 29.06.2018 10:00, Kuntal Ghosh wrote: On Wed, Jun 27, 2018 at 12:10 PM, Andrey V. Lepikhov wrote: I prepare third version of the patches. Summary: 1. Mask DEAD tuples at a page during consistency checking (See comments for the mask_dead_tuples() function). - ItemIdSetDead(lp); + if

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-06-27 Thread Andrey V. Lepikhov
On 28.06.2018 05:00, Peter Geoghegan wrote: On Tue, Jun 26, 2018 at 11:40 PM, Andrey V. Lepikhov wrote: I still believe that the patch for physical TID ordering in btree: 1) has its own value, not only for target deletion, 2) will require only a few local changes in my code, and this

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-06-26 Thread Andrey V. Lepikhov
On 23.06.2018 00:43, Peter Geoghegan wrote: On Fri, Jun 22, 2018 at 4:24 AM, Andrey V. Lepikhov wrote: According to your feedback, i develop second version of the patch. In this version: 1. High-level functions index_beginscan(), index_rescan() not used. Tree descent made by _bt_search

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-06-26 Thread Andrey V. Lepikhov
On 26.06.2018 15:31, Masahiko Sawada wrote: On Fri, Jun 22, 2018 at 8:24 PM, Andrey V. Lepikhov wrote: Hi, According to your feedback, i develop second version of the patch. In this version: 1. High-level functions index_beginscan(), index_rescan() not used. Tree descent made by _bt_search

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-06-25 Thread Andrey V. Lepikhov
On 23.06.2018 01:14, Peter Geoghegan wrote: On Fri, Jun 22, 2018 at 12:43 PM, Peter Geoghegan wrote: On Fri, Jun 22, 2018 at 4:24 AM, Andrey V. Lepikhov wrote: According to your feedback, i develop second version of the patch. In this version: 1. High-level functions index_beginscan

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-06-22 Thread Andrey V. Lepikhov
Hi, According to your feedback, i develop second version of the patch. In this version: 1. High-level functions index_beginscan(), index_rescan() not used. Tree descent made by _bt_search(). _bt_binsrch() used for positioning on the page. 2. TID list introduced in amtargetdelete() interface. Now

Re: [WIP] [B-Tree] Retail IndexTuple deletion

2018-06-19 Thread Andrey V. Lepikhov
On 19.06.2018 04:05, Peter Geoghegan wrote: On Mon, Jun 18, 2018 at 2:54 PM, Peter Geoghegan wrote: On Sun, Jun 17, 2018 at 9:39 PM, Andrey V. Lepikhov wrote: Patch '0001-retail-indextuple-deletion' introduce new function amtargetdelete() in access method interface. Patch &

[WIP] [B-Tree] Retail IndexTuple deletion

2018-06-17 Thread Andrey V. Lepikhov
Hi, I have written a code for quick indextuple deletion from an relation by heap tuple TID. The code relate to "Retail IndexTuple deletion" enhancement of btree index on postgresql wiki [1]. Briefly, it includes three steps: 1. Key generation for index tuple searching. 2. Index relation search