Farewell greeting

2021-06-27 Thread tsunakawa.ta...@fujitsu.com
Hello all, I'm moving to another company from July 1st. I may possibly not be allowed to do open source activities there, so let me say goodbye once. Thank you all for your help. I really enjoyed learning PostgreSQL and participating in its development. It's a pity that I may not be able t

RE: Transactions involving multiple postgres foreign servers, take 2

2021-06-17 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > On Tue, Jun 15, 2021 at 5:51 AM tsunakawa.ta...@fujitsu.com > wrote: > > Postgres can do that, but other implementations can not necessaily do it, > > I'm > afraid. But before that, the FDW interface documentation doesn't describe

RE: Transactions involving multiple postgres foreign servers, take 2

2021-06-15 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > Well, we're talking about running this commit routine from within > CommitTransaction(), right? So I think it is in fact running in the > server. And if that's so, then you have to worry about how to make it > respond to interrupts. You can't just call some functions > DBMSX_xa

RE: Transactions involving multiple postgres foreign servers, take 2

2021-06-13 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > On Thu, Jun 10, 2021 at 9:58 PM tsunakawa.ta...@fujitsu.com > wrote: > > The question I have been asking is how. With that said, we should only have > two options; one is the return value of the FDW commit routine, and the other > is > via ereport(ERR

RE: Transactions involving multiple postgres foreign servers, take 2

2021-06-10 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > That is completely unrealistic. As Sawada-san has pointed out > repeatedly, there are tons of things that can go wrong even after the > remote side has prepared the transaction. Preparing a transaction only > promises that the remote side will let you commit the transaction upo

RE: Transactions involving multiple postgres foreign servers, take 2

2021-06-10 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > If we accept each elementary-commit (via FDW connection) to fail, the > parent(?) there's no way the root 2pc-commit can succeed. How can we > ignore the fdw-error in that case? No, we don't ignore the error during FDW commit. As mentioned at the end of this mail, the

RE: Transactions involving multiple postgres foreign servers, take 2

2021-06-09 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > Maybe it's better to start a new thread to discuss this topic. If your > idea is good, we can lower all error that happened after writing the > commit record to warning, reducing the cases where the client gets > confusion by receiving an error after the commit. No. It's

RE: Transactions involving multiple postgres foreign servers, take 2

2021-06-09 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > On Tue, Jun 8, 2021 at 5:28 PM tsunakawa.ta...@fujitsu.com > wrote: > > Then, in what kind of scenario are we talking about the difficulty, and how > > is > it difficult to handle, when we adopt either the method 1 or 2? (I'd just >

RE: Transactions involving multiple postgres foreign servers, take 2

2021-06-08 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > I think the discussion is based the behavior that any process that is > responsible for finishing the 2pc-commit continue retrying remote > commits until all of the remote-commits succeed. Thank you for coming back. We're talking about the first attempt to prepare and

RE: Transactions involving multiple postgres foreign servers, take 2

2021-06-08 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > On Tue, Jun 8, 2021 at 9:47 AM tsunakawa.ta...@fujitsu.com > wrote: > > Why does the client have to know the error on a remote server, whereas the > global transaction itself is destined to commit? > > It's not necessarily on a remote server.

RE: Transactions involving multiple postgres foreign servers, take 2

2021-06-07 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > I think we should not reinterpret the severity of the error and lower > it. Especially, in this case, any kind of errors can be thrown. It > could be such a serious error that FDW developer wants to report to > the client. Do we lower even PANIC to a lower severity such as

RE: Transactions involving multiple postgres foreign servers, take 2

2021-06-04 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > On Fri, Jun 4, 2021 at 5:04 PM tsunakawa.ta...@fujitsu.com > wrote: > > Why does the user have to get an error? Once the local transaction has been > prepared, which means all remote ones also have been prepared, the whole > transaction is determined t

RE: Fast COPY FROM based on batch insert

2021-06-04 Thread tsunakawa.ta...@fujitsu.com
From: Andrey Lepikhov > We still have slow 'COPY FROM' operation for foreign tables in current master. > Now we have a foreign batch insert operation And I tried to rewrite the patch > [1] > with this machinery. I haven't looked at the patch, but nice performance. However, I see the following p

RE: Transactions involving multiple postgres foreign servers, take 2

2021-06-04 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada 1. the backend continues attempting to commit all prepared foreign > transactions until all of them are committed. > 2. the backend attempts to commit all prepared foreign transactions > once. If an error happens, leave them for the resolver. > 3. the backend asks the resolve

RE: Parallel Inserts in CREATE TABLE AS

2021-05-27 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > I'm still not sure why the execution time with 0 workers (or serial execution > or > no parallelism involved) on my testing system is 112 sec compared to 58 sec on > Hou-San's system for the same use case. Maybe the testing system I'm using > is not of the latest configu

RE: Parallel Inserts in CREATE TABLE AS

2021-05-27 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > I think we can discuss this in a separate thread and see what other > hackers think. OK, unless we won't get stuck in the current direction. (Our goal is to not degrade in performance, but to outperform serial execution, isn't it?) > If the idea is to give the user c

RE: Parallel Inserts in CREATE TABLE AS

2021-05-27 Thread tsunakawa.ta...@fujitsu.com
From: houzj.f...@fujitsu.com > Although, the 4 workers case still has performance degradation compared to > serial case. > > SERIAL: 58759.213 ms > PARALLEL 2 WORKER [NOT SKIP FSM]: 68390.221 ms [SKIP FSM]: > 58633.924 ms > PARALLEL 4 WORKER [NOT SKIP FSM]: 67448.142 ms [SKIP FSM]: > 66,960.30

RE: Parallel Inserts in CREATE TABLE AS

2021-05-27 Thread tsunakawa.ta...@fujitsu.com
From: Dilip Kumar Basically you are creating a new table and loading data to it and that means you will be less likely to access those data soon so for such thing spoiling buffer cache may not be a good idea. -- Some people, including me, would s

RE: Parallel Inserts in CREATE TABLE AS

2021-05-26 Thread tsunakawa.ta...@fujitsu.com
From: Dilip Kumar > I think some other cause of contention on relation extension locks are > 1. CTAS is using a buffer strategy and due to that, it might need to > evict out the buffer frequently for getting the new block in. Maybe > we can identify by turning off the buffer strategy for CTAS and

RE: Parallel Inserts in CREATE TABLE AS

2021-05-26 Thread tsunakawa.ta...@fujitsu.com
Thank you for the detailed analysis, I'll look into it too. (The times have changed...) From: Bharath Rupireddy > Well, one might think to add more blocks at a time, say > Min(1024, lockWaiters * 128/256/512) than currently extraBlocks = > Min(512, lockWaiters * 20);. This will work (i.e. we do

RE: Batch insert in CTAS/MatView code

2021-05-26 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > I think the "New Table Access Methods for Multi and Single Inserts" > patches at [1] make multi insert usage easy for COPY, CTAS/Create Mat > View, Refresh Mat View and so on. It also has a patch for multi > inserts in CTAS and Refresh Mat View > (v6-0002-CTAS-and-REFRESH

RE: Batch insert in CTAS/MatView code

2021-05-25 Thread tsunakawa.ta...@fujitsu.com
Hello Paul-san, From: Daniel Gustafsson > In an off-list discussion with Paul, we decided to withdraw this patch for now > and instead create a new entry when there is a re-worked patch. This has > now > been done in the CF app. Would you mind if I take over this patch for PG 15? I find this p

RE: Parallel Inserts in CREATE TABLE AS

2021-05-25 Thread tsunakawa.ta...@fujitsu.com
From: houzj.f...@fujitsu.com > + /* > + * We don't need to skip contacting FSM while inserting tuples > for > + * parallel mode, while extending the relations, workers > instead of > + * blocking on a page while another worker is inserting, can >

RE: Parallel Inserts in CREATE TABLE AS

2021-05-25 Thread tsunakawa.ta...@fujitsu.com
Bharath-san, all, Hmm, I didn't experience performance degradation on my poor-man's Linux VM (4 CPU, 4 GB RAM, HDD)... [benchmark preparation] autovacuum = off shared_buffers = 1GB checkpoint_timeout = 1h max_wal_size = 8GB min_wal_size = 8GB (other settings to enable parallelism) CREATE UNLOGG

RE: Skip partition tuple routing with constant partition key

2021-05-24 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > Ah, Maybe I found the issue. > Attaching a new patch, please have a try on this patch. Thanks, it has compiled perfectly without any warning. Regards Takayuki Tsunakawa

RE: Skip partition tuple routing with constant partition key

2021-05-24 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > It seems a little strange, I have compiled it alone in two different linux > machine > and did > not find such an error. Did you compile it on a windows machine ? On Linux, it produces: gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-s\ tat

RE: Skip partition tuple routing with constant partition key

2021-05-24 Thread tsunakawa.ta...@fujitsu.com
From: houzj.f...@fujitsu.com > I think this patch can solve the performance degradation of key expression > after > applying the [Save the last partition] patch. > Besides, this could be a separate patch which can improve some more cases. > Thoughts ? Thank you for proposing an impressive improv

RE: Bug in query rewriter - hasModifyingCTE not getting set

2021-05-20 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > "tsunakawa.ta...@fujitsu.com" writes: > > Finally, I think I've understood what you meant. Yes, the current code > > seems > to be wrong. > > I'm fairly skeptical of this claim, because that code has stood for a > long t

RE: Bug in query rewriter - hasModifyingCTE not getting set

2021-05-20 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > I think either the bit about rule_action is unnecessary, or most of > the code immediately above this is wrong, because it's only updating > flags in sub_action. Why do you think it's necessary to change > rule_action in addition to sub_action? Finally, I think I've understood w

RE: Skip partition tuple routing with constant partition key

2021-05-19 Thread tsunakawa.ta...@fujitsu.com
From: Amit Langote > On Tue, May 18, 2021 at 11:11 AM houzj.f...@fujitsu.com > wrote: > > For some big data scenario, we sometimes transfer data from one table(only > store not expired data) > > to another table(historical data) for future analysis. > > In this case, we import data into historica

RE: Bug in query rewriter - hasModifyingCTE not getting set

2021-05-17 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > In view of this, maybe the right thing is to disallow modifying CTEs > in rule actions in the first place. I see we already do that for > views (i.e. ON SELECT rules), but they're not really any safer in > other types of rules. You meant by views something like the following, di

RE: Support for VACUUMing Foreign Tables

2021-05-16 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > This can be useful in situations like where there are many remote > postgres servers that are connected to a single coordinator on which > foreign tables are defined for each of the remote tables. In this > case, the DBA (or whoever is responsible to do that job) doesn't

RE: FDW and connections

2021-05-16 Thread tsunakawa.ta...@fujitsu.com
From: Phil Godfrin My question is - how does the call to GetConnection() know what port to use? Lets say we're using PGBouncer to connect on the local server at port 6432, but there is no pgbouncer listening at the foreign server, what port gets passed? My first thought is whatever the client c

RE: Support for VACUUMing Foreign Tables

2021-05-13 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > I think it will be useful to allow foreign tables to be VACUUMed if > the underlying FDW supports, currently VACUUM doesn't support foreign > tables, see [1]. Could you let us imagine more concretely how useful it will be? While TRUNCATE can be part of an application's

RE: Inaccurate error message when set fdw batch_size to 0

2021-05-09 Thread tsunakawa.ta...@fujitsu.com
From: houzj.f...@fujitsu.com > So, is it better to change the error message to “fetch_size requires a > positive integer value” ? > I also found fetch_size has the similar issue, attaching a patch to fix this. I have a faint memory that I fixed them after receiving the same feedback from someo

RE: batch fdw insert bug (Postgres 14)

2021-05-09 Thread tsunakawa.ta...@fujitsu.com
From: Tomas Vondra > I think the simplest fix is simply to pstrdup() the query when building > fmstate in create_foreign_modify. We've already been doing that for > orig_query anyway. That seems cleaner than printing the last query we > build (which would be confusing I think). > > I've pushed a

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-05-06 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > On Wed, May 5, 2021 at 10:54 PM tsunakawa.ta...@fujitsu.com > wrote: > > As proposed in this thread and/or "Parallel INSERT SELECT take 2", we > thought of detecting parallel unsafe function execution during SQL statement > execution, instead

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-05-06 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > On Wed, Apr 28, 2021 at 9:42 PM houzj.f...@fujitsu.com > wrote: > > So, If we do not want to lock down the parallel safety of built-in > > functions. > > It seems we can try to fetch the proparallel from pg_proc for built-in > > function > > in fmgr_info_cxt_security too. To

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-05-05 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > On Tue, May 4, 2021 at 11:47 PM Greg Nancarrow > wrote: > > Problem is, for built-in functions, the changes are allowed, but for > > some properties (like strict) the allowed changes don't actually take > > effect (this is what Amit was referring to - so why allow those > > ch

RE: A test for replay of regression tests

2021-04-22 Thread tsunakawa.ta...@fujitsu.com
From: Thomas Munro > We have automated tests for many specific replication and recovery scenarios, > but nothing that tests replay of a wide range of records. > People working on recovery code often use installcheck (presumably along > with other custom tests) to exercise it, sometimes with > wal_

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-04-22 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > For approach 1): I think it could result in infinite recursion. > > For example: > If we first access one built-in function A which have not been cached, > it need access the pg_proc, When accessing the pg_proc, it internally still > need > some built-in function B to sc

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-04-22 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > I agree that it's better to mark the function with correct parallel safety > lable. > Especially for the above functions which will be executed in parallel mode. > It will be friendly to developer and user who is working on something related > to > parallel test. > > So

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-04-21 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > Amit Kapila writes: > > IIUC, the idea here is to check for parallel safety of functions at > > someplace in the code during function invocation so that if we execute > > any parallel unsafe/restricted function via parallel worker then we > > error out. If so, isn't it possible t

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-04-20 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > [ raised eyebrow... ] I find it very hard to understand why that would > be necessary, or even a good idea. Not least because there's no spare > room there; you'd have to incur a substantial enlargement of the > array to add another flag. But also, that would indeed lock down >

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-04-20 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > Bharath Rupireddy writes: > > IMO, the reason for not checking the parallel safety of the support > > functions is that the functions themselves can have whole lot of other > > functions (can be nested as well) which might be quite hard to check > > at the planning time. That is

[bug?] Missed parallel safety checks, and wrong parallel safety

2021-04-20 Thread tsunakawa.ta...@fujitsu.com
Hello, I think we've found a few existing problems with handling the parallel safety of functions while doing an experiment. Could I hear your opinions on what we should do? I'd be willing to create and submit a patch to fix them. The experiment is to add a parallel safety check in FunctionC

RE: Schema variables - new implementation for Postgres 15

2021-04-16 Thread tsunakawa.ta...@fujitsu.com
From: Pavel Stehule -- I am sorry, but in this topic we have different opinions. The variables in PLpgSQL are not transactional too (same is true in Perl, Python, ...). Session variables in Oracle, MS SQL, DB2, MySQL are not transactional too. My p

RE: Schema variables - new implementation for Postgres 15

2021-04-15 Thread tsunakawa.ta...@fujitsu.com
From: Pavel Stehule -- do $$ declare x int ; begin for i in 1..100 loop let ooo = i; end loop; end; $$; variant 1 .. 1500 ms variant 2 with PLpgSQL support .. 140 ms variant 2 without PLpgSQL support 9000 ms ---

Parallel INSERT SELECT take 2

2021-04-11 Thread tsunakawa.ta...@fujitsu.com
This is another try of [1]. BACKGROUND We want to realize parallel INSERT SELECT in the following steps: 1) INSERT + parallel SELECT 2) Parallel INSERT + parallel SELECT Below are example use cases. We don't expect high concurrency or an empty data sou

RE: 2019-03 CF now in progress

2021-04-08 Thread tsunakawa.ta...@fujitsu.com
From: David Steele > Overall, 118 of 262 entries were closed during this commitfest (45%). > That includes 91 patches committed since March 1, which is pretty > fantastic. Thank you to everyone, especially the committers, for your > hard work! The number of committed patches in the last CF is rec

RE: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > I didn't see that, but found the following article. > > https://stackoverflow.com/questions/2590794/gcov-warning-merge-mismat > ch-for-summaries ... > It seems like your working directory needs some cleanup. That could very well be. It'd be safer to run "make coverage-

RE: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread tsunakawa.ta...@fujitsu.com
From: osumi.takami...@fujitsu.com > By the way, when I build postgres with this patch and enable-coverage option, > the results of RT becomes unstable. Does someone know the reason ? > When it fails, I get stderr like below > > t/001_start_stop.pl .. 10/24 > # Failed test 'pg_ctl start: no stde

RE: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2021-04-01 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > Attaching a small patch that emits a warning when the persistence setting of a > partitioned table is changed and also added a note into the docs. Please have > a > look at it. Thank you. However, I think I'll withdraw this CF entry since I'm not sure I can take time

RE: Add client connection check during the execution of the query

2021-04-01 Thread tsunakawa.ta...@fujitsu.com
From: Thomas Munro > > Following PostmasterIsAlive(), maybe it's better to name it as > pq_connection_is_alive() pq_client_is_alive(), or pq_frontend_is_alive() like > the pqcomm.c's head comment uses the word frontend? > > I think it's OK, because it matches the name of the GUC. I'm more concer

RE: Add client connection check during the execution of the query

2021-04-01 Thread tsunakawa.ta...@fujitsu.com
From: Thomas Munro > I changed my mind. Let's commit the pleasingly simple Linux-only feature for > now, and extend to it to send some kind of no-op message in a later release. > So this is the version I'd like to go with. > Objections? +1, as some of our users experienced the problem that the s

RE: libpq debug log

2021-03-30 Thread tsunakawa.ta...@fujitsu.com
From: 'alvhe...@alvh.no-ip.org' > > got expected ERROR: cannot insert multiple commands into a prepared > statement > > got expected division-by-zero > > Eh? this is not at all expected, of course, but clearly not PQtrace's > fault. I'll look tomorrow. Iwata-san and I were starting to investi

RE: libpq debug log

2021-03-30 Thread tsunakawa.ta...@fujitsu.com
From: 'alvhe...@alvh.no-ip.org' > Okay, pushed this patch and the new testing for it based on > libpq_pipeline. We'll see how the buildfarm likes it. Thank you very much! I appreciate you taking your valuable time while I imagine you must be pretty busy with taking care of other (possibly more

RE: libpq debug log

2021-03-28 Thread tsunakawa.ta...@fujitsu.com
From: 'alvhe...@alvh.no-ip.org' > > > (Hey, what the heck is that "Z" at the end of the message? I thought > > > the error ended right at the \x00 ...) > > > > We'll investigate these issues. > > For what it's worth, I did fix this problem in patch 0005 that I > attached. The problem was that o

RE: libpq debug log

2021-03-28 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > It's better to be short as far as it is clear enough. Actually '<' to > 'F' and '>' to 'B' is clear enough to me. So I don't need a longer > notation. Agreed, because the message format description in the PG manual uses F and B. Regards Takayuki Tsunakawa

RE: libpq debug log

2021-03-28 Thread tsunakawa.ta...@fujitsu.com
From: alvhe...@alvh.no-ip.org > > Proposed changes on top of v29. > > This last one uses libpq_pipeline -t and verifies the output against an > expected > trace file. Applies on top of all the previous patches. I attach the whole > lot, > so that the CF bot has a chance to run it. Thank you

RE: libpq debug log

2021-03-28 Thread tsunakawa.ta...@fujitsu.com
From: alvhe...@alvh.no-ip.org > I added an option to the new libpq_pipeline program that it activates > libpq trace. It works nicely and I think we can add that to the > regression tests. That's good news. Thank you. > 1. The trace output for the error message won't be very nice, because it

RE: Global snapshots

2021-03-24 Thread tsunakawa.ta...@fujitsu.com
From: Andrey V. Lepikhov > Current state of the patch set rebased on master, 5aed6a1fc2. > > It is development version. Here some problems with visibility still detected > in > two tests: > 1. CSN Snapshot module - TAP test on time skew. > 2. Clock SI implementation - TAP test on emulation of ba

RE: Disable WAL logging to speed up data loading

2021-03-24 Thread tsunakawa.ta...@fujitsu.com
From: Stephen Frost > * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > > As Laurenz-san kindly replied, the database server refuses to start with a > clear message. So, it's similarly very clear what happens. The user will > never > unknowingl

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-22 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > On Mon, Mar 22, 2021 at 08:18:56PM -0700, Zhihong Yu wrote: > > with data_dest_cb callback. It is used for send text representation of a > > tuple to a custom destination. > > > > send text -> sending text > > I would say "It is used to send the text representation ..." I t

RE: Disable WAL logging to speed up data loading

2021-03-22 Thread tsunakawa.ta...@fujitsu.com
From: Stephen Frost > First- what are you expecting would actually happen during crash recovery in > this specific case with your proposed new WAL level? ... > I'm not suggesting it's somehow more crash safe- but it's at least very clear > what happens in such a case, to wit: the entire table is c

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-22 Thread tsunakawa.ta...@fujitsu.com
From: Andrey Lepikhov > Macros _() at the postgresExecForeignCopy routine: > if (PQputCopyEnd(conn, OK ? NULL : _("canceled by server")) <= 0) > > uses gettext. Under linux it is compiled ok, because (as i understood) > uses standard implementation of gettext: > objdump -t contrib/postgres_fdw/po

RE: Disable WAL logging to speed up data loading

2021-03-21 Thread tsunakawa.ta...@fujitsu.com
From: Stephen Frost > The argument here seems to stem from the idea that issueing a 'TRUNCATE' > inside the transaction before starting the 'COPY' command is 'too hard'. > I could be wrong and perhaps opinions will change in the future, but it really > doesn't seem like the there's much support

RE: Disable WAL logging to speed up data loading

2021-03-19 Thread tsunakawa.ta...@fujitsu.com
From: Laurenz Albe > Now I'm not saying that this feature should not go in (I set it to > "ready for committer", because I see no technical flaw with the > implementation), but it remains debatable if we want the feature or not. Oh, yes, thank you very much for supporting this and other relevant

RE: libpq debug log

2021-03-19 Thread tsunakawa.ta...@fujitsu.com
Alvaro-san, Iwata-san, cc: Tom-san, Horiguchi-san, Thank you, it finally looks complete to me! Alvaro-san, We appreciate your help and patience, sometimes rewriting large part of the patch. Could you do the final check (and possibly tweak) for commit? Regards Takayuki Tsunakawa

RE: Disable WAL logging to speed up data loading

2021-03-18 Thread tsunakawa.ta...@fujitsu.com
From: David Steele > After reading through the thread (but not reading the patch) I am -1 on > this proposal. > > The feature seems ripe for abuse and misunderstanding, and as has been > noted in the thread, there are a variety of alternatives that can > provide a similar effect. > > It doesn't

RE: libpq debug log

2021-03-18 Thread tsunakawa.ta...@fujitsu.com
Hello Iwata-san, Thanks to removing the static arrays, the code got much smaller. I'm happy with this result. (1) + (> for messages from client to server, + and < for messages from server to client), I think this was meant to say "> or <". So, maybe you should remove "," at the e

RE: libpq debug log

2021-03-18 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > + pqTraceOutputR(const char *message, FILE *f) > + { > + int cursor = 0; > + > + pqTraceOutputInt32(message, &cursor, f); > > I don't understand the reason for spliting message and &cursor here. > > + pqTraceOutputR(const char *message, FILE *f) > + { > +

RE: libpq debug log

2021-03-18 Thread tsunakawa.ta...@fujitsu.com
From: Iwata, Aya/岩田 彩 > > Yes, precisely, 2 bytes for the double quotes needs to be subtracted > > as > > follows: > > > > len = fprintf(...); > > *cursor += (len - 2); > > Thank you for your advice. I changed pqTraceOutputString set cursor to fprintf > return -2. > And I removed cursor m

RE: libpq debug log

2021-03-17 Thread tsunakawa.ta...@fujitsu.com
From: Alvaro Herrera > In pqTraceOutputString(), you can use the return value from fprintf to > move the cursor -- no need to count chars. Yes, precisely, 2 bytes for the double quotes needs to be subtracted as follows: len = fprintf(...); *cursor += (len - 2); > I still think

RE: libpq debug log

2021-03-17 Thread tsunakawa.ta...@fujitsu.com
I'll look at the comments from Alvaro-san and Horiguchi-san. Here are my review comments: (23) + /* Trace message only when there is first 1 byte */ + if (conn->Pfdebug && conn->outCount < conn->outMsgStart) + { + if (conn->outCount < conn->outMsgStart) +

RE: libpq debug log

2021-03-16 Thread tsunakawa.ta...@fujitsu.com
Alvaro-san, Thank you for taking your time to take a look at an incomplete patch. I thought I would ask you for final check for commit after Iwata-san has reflected my review comments. I discussed with Iwata-sn your below comments. Let me convey her opinions. (She is now focusing on fixing

RE: libpq debug log

2021-03-16 Thread tsunakawa.ta...@fujitsu.com
I've finished the review. Here are the last set of comments. (16) (15) is also true for the processing of 'H' message. (17) pqTraceOutputD + for (i = 0; i < nfields; i++) + { + len = pqTraceOutputInt32(message + cursor, f); +

RE: libpq debug log

2021-03-15 Thread tsunakawa.ta...@fujitsu.com
I've not finished reviewing yet, but there seems to be many mistakes. I'm sending second set of review comments now so you can fix them in parallel. (8) + charid = '\0'; This initialization is not required because id will always be assigned a value shortly. (9) +static int

RE: libpq debug log

2021-03-15 Thread tsunakawa.ta...@fujitsu.com
I'm looking at the last file libpq-trace.c. I'll continue the review after lunch. Below are some comments so far. (1) - Enables tracing of the client/server communication to a debugging file stream. + Enables tracing of the client/server communication to a debugging file + str

RE: Enhance traceability of wal_level changes for backup management

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: David Steele > As a backup software author, I don't see this feature as very useful. > > The problem is that there are lots of ways for WAL to go missing so > monitoring the WAL archive for gaps is essential and this feature would > not replace that requirement. The only extra information y

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > About the patch, it would be better to change the type of > BUF_DROP_FULL_SCAN_THRESHOLD to uint64, even though the current > value > doesn't harm. OK, attached, to be prepared for the distant future when NBuffers becomes 64-bit. Regards Takayuki Tsunakawa

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > Could you rebase again and send an updated patch ? > I could do it if you want. Rebased and attached. Fortunately, there was no rebase conflict this time. make check passed for PG core and postgres_fdw. Regards Takayuki Tsunakawa v20-0001-Fast-COPY-FROM-into-

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: Thomas Munro > > uint64 > > +1 Thank you, the patch is attached (we tend to forget how large our world is... 64-bit) We're sorry to cause you trouble. Regards Takayuki Tsunakawa v1-0001-Fix-overflow-when-counting-the-number-of-buffers-.patch Description: v1-0001-Fix-overflo

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: Thomas Munro > On Fri, Mar 12, 2021 at 5:20 PM Amit Kapila wrote: > > uint64 > > +1 +1 I'll send a patch later. Regards Takayuki Tsunakawa

RE: Parallel INSERT (INTO ... SELECT ...)

2021-03-10 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > Now, coming back to Hou-San's patch to introduce a GUC and reloption > for this feature, I think both of those make sense to me because when > the feature is enabled via GUC, one might want to disable it for > partitioned tables? Do we agree on that part or someone thinks > oth

RE: libpq debug log

2021-03-10 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > Right. So something like this? > > unsigned char p; > > p = buf + *cursor; > result = (uint32) (*p << 24) + (*(p + 1)) << 16 + ...); Yes, that would work (if p is a pointer), but I think memcpy() is enough like pqGetInt() does. Regards Takayuki Tsunakawa

RE: libpq debug log

2021-03-10 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > The output functions copy message bytes into local variable but the > same effect can be obtained by just casing via typed pointer type. > > uint32 tmp4; > .. > memcpy(&tmp4, buf + *cursor, 4); > result = (int) pg_ntoh32(tmp4); > > can be written as > > resul

RE: libpq debug log

2021-03-10 Thread tsunakawa.ta...@fujitsu.com
Alvaro-san, Tom-san, Horiguchi-san, From: Kyotaro Horiguchi > +1 for the thanks for the quick work. I have some random comments > after a quick look on it. Thank you very much for giving many comments. And We're sorry to have caused you trouble. I told Iwata-san yesterday to modify the patch

RE: Avoid CommandCounterIncrement in RI trigger when INSERT INTO referencing table

2021-03-10 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > After some more on how to support parallel insert into fk relation. > It seems we do not have a cheap way to implement this feature. > > In RI_FKey_check, Currently, postgres execute "select xx for key share" to > check that foreign key exists in PK table. > However "sele

RE: POC: Cleaning up orphaned files using undo logs

2021-03-09 Thread tsunakawa.ta...@fujitsu.com
I'm crawling like a snail to read the patch set. Below are my first set of review comments, which are all minor. (1) + tablespacetemporary temporary -> undo (2) undo_tablespaces (string) + ... +The value is a list of names of tablespaces. When there is more than +

RE: [PoC] Non-volatile WAL buffer

2021-03-08 Thread tsunakawa.ta...@fujitsu.com
From: Takashi Menjo > > The other question is whether simply placing WAL on DAX (without any > > code changes) is safe. If it's not, then all the "speedups" are > > computed with respect to unsafe configuration and so are useless. And > > BTT should be used instead, which would of course produce v

RE: Implementing Incremental View Maintenance

2021-03-08 Thread tsunakawa.ta...@fujitsu.com
From: Thomas Munro > It's probably time to move forward with the plan of pushing the > results into a commitfest.postgresql.org API, and then making Magnus > et al write the email spam code with a preferences screen linked to > your community account :-D +1 I wish to see all the patch status info

RE: Parallel INSERT (INTO ... SELECT ...)

2021-03-07 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > For now, I have left 0005 and 0006 patches, we can come back to those once we > are done with the first set of patches. The first patch looks good to me and I > think we can commit it and then bikeshed about GUC/reloption patch. Agreed, it looks good to me, too. Regards Taka

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-05 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > I think this change to the regression tests is suspicous: > > -CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES > ($1, $2) > -COPY rem2, line 1: "-1 xyzzy" > +CONTEXT: COPY loc2, line 1: "-1 xyzzy" > +remote SQL command: COPY public.loc2(f1, f2) FR

RE: Avoid CommandCounterIncrement in RI trigger when INSERT INTO referencing table

2021-03-04 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > From the wiki[1], CCI is to let statements can not see the rows they modify. > > Here is an example of the case 1): > (Note table referenced and referencing are both empty) > - > postgres=# with cte as (insert into referenced values(1)) insert into > referencing value

RE: libpq debug log

2021-03-04 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > But I think passing the message start address explicitly might be better than > having it understand the buffering behavior in enough detail to know where to > find the message. Part of the point here > (IMO) is to decouple the tracing logic from the core libpq logic, in hopes of

RE: libpq debug log

2021-03-04 Thread tsunakawa.ta...@fujitsu.com
From: tsunakawa.ta...@fujitsu.com > I understood that the former is pqParseInput3() and the latter is > pqPutMsgEnd(). They call the logging function wne conn->pfdebug is not > NULL. Its signature is like this (that will be defined in libpq-trace.c): > > void pqLog

RE: libpq debug log

2021-03-04 Thread tsunakawa.ta...@fujitsu.com
Tom-san, Alvaro-san, From: Tom Lane > I took a quick look through the v22 patch, and TBH I don't like much of > anything > at all about the proposed architecture. It's retained most of the flavor of > the > way it was done before, which was a hangover from the old process-on-the-fly > scheme.

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-03 Thread tsunakawa.ta...@fujitsu.com
From: Zhihong Yu > This feature enables bulk COPY into foreign table in the case of > multi inserts is possible > > 'is possible' -> 'if possible' > > FDWAPI was extended by next routines: > > next routines -> the following routines Thank you, fixed slightly differently. (I feel the need for

RE: Avoid CommandCounterIncrement in RI trigger when INSERT INTO referencing table

2021-03-03 Thread tsunakawa.ta...@fujitsu.com
Excuse me for asking probably stupid questions... From: houzj.f...@fujitsu.com > As far as I can see, it’s only necessary to increment command id when the > INSERT command modified the referenced table. Why do we have to increment the command ID when the INSERT's target table is a referenced ta

  1   2   3   4   5   >