RE: libpq debug log

2021-03-18 Thread tsunakawa.ta...@fujitsu.com
From: Iwata, Aya/岩田 彩 > > Yes, precisely, 2 bytes for the double quotes needs to be subtracted > > as > > follows: > > > > len = fprintf(...); > > *cursor += (len - 2); > > Thank you for your advice. I changed pqTraceOutputString set cursor to fprintf > return -2. > And I removed cursor m

RE: libpq debug log

2021-03-18 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > + pqTraceOutputR(const char *message, FILE *f) > + { > + int cursor = 0; > + > + pqTraceOutputInt32(message, &cursor, f); > > I don't understand the reason for spliting message and &cursor here. > > + pqTraceOutputR(const char *message, FILE *f) > + { > +

RE: libpq debug log

2021-03-18 Thread tsunakawa.ta...@fujitsu.com
Hello Iwata-san, Thanks to removing the static arrays, the code got much smaller. I'm happy with this result. (1) + (> for messages from client to server, + and < for messages from server to client), I think this was meant to say "> or <". So, maybe you should remove "," at the e

RE: Disable WAL logging to speed up data loading

2021-03-18 Thread tsunakawa.ta...@fujitsu.com
From: David Steele > After reading through the thread (but not reading the patch) I am -1 on > this proposal. > > The feature seems ripe for abuse and misunderstanding, and as has been > noted in the thread, there are a variety of alternatives that can > provide a similar effect. > > It doesn't

RE: libpq debug log

2021-03-19 Thread tsunakawa.ta...@fujitsu.com
Alvaro-san, Iwata-san, cc: Tom-san, Horiguchi-san, Thank you, it finally looks complete to me! Alvaro-san, We appreciate your help and patience, sometimes rewriting large part of the patch. Could you do the final check (and possibly tweak) for commit? Regards Takayuki Tsunakawa

RE: Disable WAL logging to speed up data loading

2021-03-19 Thread tsunakawa.ta...@fujitsu.com
From: Laurenz Albe > Now I'm not saying that this feature should not go in (I set it to > "ready for committer", because I see no technical flaw with the > implementation), but it remains debatable if we want the feature or not. Oh, yes, thank you very much for supporting this and other relevant

RE: Disable WAL logging to speed up data loading

2021-03-21 Thread tsunakawa.ta...@fujitsu.com
From: Stephen Frost > The argument here seems to stem from the idea that issueing a 'TRUNCATE' > inside the transaction before starting the 'COPY' command is 'too hard'. > I could be wrong and perhaps opinions will change in the future, but it really > doesn't seem like the there's much support

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-22 Thread tsunakawa.ta...@fujitsu.com
From: Andrey Lepikhov > Macros _() at the postgresExecForeignCopy routine: > if (PQputCopyEnd(conn, OK ? NULL : _("canceled by server")) <= 0) > > uses gettext. Under linux it is compiled ok, because (as i understood) > uses standard implementation of gettext: > objdump -t contrib/postgres_fdw/po

RE: Disable WAL logging to speed up data loading

2021-03-22 Thread tsunakawa.ta...@fujitsu.com
From: Stephen Frost > First- what are you expecting would actually happen during crash recovery in > this specific case with your proposed new WAL level? ... > I'm not suggesting it's somehow more crash safe- but it's at least very clear > what happens in such a case, to wit: the entire table is c

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-22 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > On Mon, Mar 22, 2021 at 08:18:56PM -0700, Zhihong Yu wrote: > > with data_dest_cb callback. It is used for send text representation of a > > tuple to a custom destination. > > > > send text -> sending text > > I would say "It is used to send the text representation ..." I t

RE: Disable WAL logging to speed up data loading

2021-03-24 Thread tsunakawa.ta...@fujitsu.com
From: Stephen Frost > * tsunakawa.ta...@fujitsu.com (tsunakawa.ta...@fujitsu.com) wrote: > > As Laurenz-san kindly replied, the database server refuses to start with a > clear message. So, it's similarly very clear what happens. The user will > never > unknowingl

RE: Global snapshots

2021-03-24 Thread tsunakawa.ta...@fujitsu.com
From: Andrey V. Lepikhov > Current state of the patch set rebased on master, 5aed6a1fc2. > > It is development version. Here some problems with visibility still detected > in > two tests: > 1. CSN Snapshot module - TAP test on time skew. > 2. Clock SI implementation - TAP test on emulation of ba

RE: libpq debug log

2021-03-28 Thread tsunakawa.ta...@fujitsu.com
From: alvhe...@alvh.no-ip.org > I added an option to the new libpq_pipeline program that it activates > libpq trace. It works nicely and I think we can add that to the > regression tests. That's good news. Thank you. > 1. The trace output for the error message won't be very nice, because it

RE: libpq debug log

2021-03-28 Thread tsunakawa.ta...@fujitsu.com
From: alvhe...@alvh.no-ip.org > > Proposed changes on top of v29. > > This last one uses libpq_pipeline -t and verifies the output against an > expected > trace file. Applies on top of all the previous patches. I attach the whole > lot, > so that the CF bot has a chance to run it. Thank you

RE: libpq debug log

2021-03-28 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > It's better to be short as far as it is clear enough. Actually '<' to > 'F' and '>' to 'B' is clear enough to me. So I don't need a longer > notation. Agreed, because the message format description in the PG manual uses F and B. Regards Takayuki Tsunakawa

RE: libpq debug log

2021-03-28 Thread tsunakawa.ta...@fujitsu.com
From: 'alvhe...@alvh.no-ip.org' > > > (Hey, what the heck is that "Z" at the end of the message? I thought > > > the error ended right at the \x00 ...) > > > > We'll investigate these issues. > > For what it's worth, I did fix this problem in patch 0005 that I > attached. The problem was that o

RE: libpq debug log

2021-03-30 Thread tsunakawa.ta...@fujitsu.com
From: 'alvhe...@alvh.no-ip.org' > Okay, pushed this patch and the new testing for it based on > libpq_pipeline. We'll see how the buildfarm likes it. Thank you very much! I appreciate you taking your valuable time while I imagine you must be pretty busy with taking care of other (possibly more

RE: libpq debug log

2021-03-30 Thread tsunakawa.ta...@fujitsu.com
From: 'alvhe...@alvh.no-ip.org' > > got expected ERROR: cannot insert multiple commands into a prepared > statement > > got expected division-by-zero > > Eh? this is not at all expected, of course, but clearly not PQtrace's > fault. I'll look tomorrow. Iwata-san and I were starting to investi

RE: Add client connection check during the execution of the query

2021-04-01 Thread tsunakawa.ta...@fujitsu.com
From: Thomas Munro > I changed my mind. Let's commit the pleasingly simple Linux-only feature for > now, and extend to it to send some kind of no-op message in a later release. > So this is the version I'd like to go with. > Objections? +1, as some of our users experienced the problem that the s

RE: Add client connection check during the execution of the query

2021-04-01 Thread tsunakawa.ta...@fujitsu.com
From: Thomas Munro > > Following PostmasterIsAlive(), maybe it's better to name it as > pq_connection_is_alive() pq_client_is_alive(), or pq_frontend_is_alive() like > the pqcomm.c's head comment uses the word frontend? > > I think it's OK, because it matches the name of the GUC. I'm more concer

RE: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2021-04-01 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > Attaching a small patch that emits a warning when the persistence setting of a > partitioned table is changed and also added a note into the docs. Please have > a > look at it. Thank you. However, I think I'll withdraw this CF entry since I'm not sure I can take time

RE: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread tsunakawa.ta...@fujitsu.com
From: osumi.takami...@fujitsu.com > By the way, when I build postgres with this patch and enable-coverage option, > the results of RT becomes unstable. Does someone know the reason ? > When it fails, I get stderr like below > > t/001_start_stop.pl .. 10/24 > # Failed test 'pg_ctl start: no stde

RE: Stronger safeguard for archive recovery not to miss data

2021-04-05 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > I didn't see that, but found the following article. > > https://stackoverflow.com/questions/2590794/gcov-warning-merge-mismat > ch-for-summaries ... > It seems like your working directory needs some cleanup. That could very well be. It'd be safer to run "make coverage-

RE: 2019-03 CF now in progress

2021-04-08 Thread tsunakawa.ta...@fujitsu.com
From: David Steele > Overall, 118 of 262 entries were closed during this commitfest (45%). > That includes 91 patches committed since March 1, which is pretty > fantastic. Thank you to everyone, especially the committers, for your > hard work! The number of committed patches in the last CF is rec

Parallel INSERT SELECT take 2

2021-04-11 Thread tsunakawa.ta...@fujitsu.com
This is another try of [1]. BACKGROUND We want to realize parallel INSERT SELECT in the following steps: 1) INSERT + parallel SELECT 2) Parallel INSERT + parallel SELECT Below are example use cases. We don't expect high concurrency or an empty data sou

RE: Global snapshots

2020-07-22 Thread tsunakawa.ta...@fujitsu.com
Hello, While I'm thinking of the following issues of the current approach Andrey raised, I'm getting puzzled and can't help asking certain things. Please forgive me if I'm missing some discussions in the past. > 1. Dependency on clocks synchronization > 2. Needs guarantees of monotonically inc

RE: Global snapshots

2020-07-26 Thread tsunakawa.ta...@fujitsu.com
Hi Andrey san, Movead san, From: tsunakawa.ta...@fujitsu.com > While Clock-SI seems to be considered the best promising for global > serializability here, > > * Why does Clock-SI gets so much attention? How did Clock-SI become the > only choice? > > * Clock-SI was

RE: Can I test Extended Query in core test framework

2020-08-11 Thread tsunakawa.ta...@fujitsu.com
Tatsuo Ishii san, a committer, proposed this to test extended query protocol. Can it be included in Postgres core? A toool to test programs by issuing frontend/backend protocol messages https://github.com/tatsuo-ishii/pgproto Regards Takayuki Tsunakawa

Autonomous database is coming to Postgres?

2020-08-12 Thread tsunakawa.ta...@fujitsu.com
Hello, I'm not sure if I should have posted this to pgsql-advocacy, but this is being developed so I posted here. Does anyone know if this development come to open source Postgres, or only to the cloud services of Microsoft and Google? (I wonder this will become another reason that Postgres wo

RE: Libpq support to connect to standby server as priority

2020-08-13 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > I think it would be better to have read-write and read-only check > trnasaction_read_only, and primary and standby can check the new > thing. There can never be any real advantage in having synonyms for > the same thing, but there can be an advantage to letting users choose > t

RE: New statistics for tuning WAL buffer size

2020-08-18 Thread tsunakawa.ta...@fujitsu.com
From: Masahiro Ikeda > It's important to provide the metrics for tuning the size of WAL buffers. > For now, it's lack of the statistics how often processes wait to write WAL > because WAL buffer is full. > > If those situation are often occurred, WAL buffer is too small for the > workload. > DBA

RE: New statistics for tuning WAL buffer size

2020-08-18 Thread tsunakawa.ta...@fujitsu.com
From: Masahiro Ikeda > If my understanding is correct, we have to measure the performance > impact first. > Do you know HariBabu is now trying to solve it? If not, I will try to > modify patches to apply HEAD. No, he's not doing it anymore. It'd be great if you could resume it. However, I reco

RE: New statistics for tuning WAL buffer size

2020-08-20 Thread tsunakawa.ta...@fujitsu.com
From: Fujii Masao > I agree to expose the number of WAL write caused by full of WAL buffers. > It's helpful when tuning wal_buffers size. Haribabu separated that number > into two fields in his patch; one is the number of WAL write by backend, > and another is by background processes and workers.

RE: New statistics for tuning WAL buffer size

2020-08-20 Thread tsunakawa.ta...@fujitsu.com
From: Fujii Masao > Just idea; it may be worth exposing the number of when new WAL file is > created and zero-filled. This initialization may have impact on > the performance of write-heavy workload generating lots of WAL. If this > number is reported high, to reduce the number of this initializat

RE: Implement UNLOGGED clause for COPY FROM

2020-08-26 Thread tsunakawa.ta...@fujitsu.com
Hello, I think it's worth thinking about a sophisticated feature like Oracle's UNRECOVERABLE data loading (because SQL Server's BCP load utility also has such a feature, but for an empty table), how about an easier approach like MySQL? I expect this won't complicate Postgres code much. The c

RE: Implement UNLOGGED clause for COPY FROM

2020-08-26 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > So you want your users to shutdown and restart the server before Copy > because that would be required if you want to change the wal_level. Yes. They seem to be fine with it, as far as I heard from a person who is involved in the system design. > However, even if we do tha

RE: Implement UNLOGGED clause for COPY FROM

2020-08-26 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > Sure, but on a daily basis, one requires only incremental WAL to > complete the backup but in this case, it would require the entire > database back up unless we have some form of block-level incremental > backup method. Regarding the backup time, I think users can shorten it

RE: On login trigger: take three

2020-09-03 Thread tsunakawa.ta...@fujitsu.com
From: Konstantin Knizhnik > Recently I have asked once again by one of our customers about login trigger > in > postgres. People are migrating to Postgres from Oracle and looking for > Postgres > analog of this Oracle feature. > This topic is not new: > I attached my prototype implementation of

RE: New statistics for tuning WAL buffer size

2020-09-03 Thread tsunakawa.ta...@fujitsu.com
From: Fujii Masao > > I changed the view name from pg_stat_walwrites to pg_stat_walwriter. > > I think it is better to match naming scheme with other views like > pg_stat_bgwriter, > > which is for bgwriter statistics but it has the statistics related to > > backend. > > I prefer the view name p

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-07 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > I intend to say that the global-visibility work can impact this in a > major way and we have analyzed that to some extent during a discussion > on the other thread. So, I think without having a complete > design/solution that addresses both the 2PC and global-visibility, it > i

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-07 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > if (RelFileNodeEquals(bufHdr->tag.rnode, rnode.node) && > + bufHdr->tag.forkNum == forkNum[j] && > + bufHdr->tag.blockNum >= firstDelBlock[j]) > > Here, I think you need to use 'i' not 'j' for forkNum and > firstDelBlock as those are arrays w.r.t forks. That might fix

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-07 Thread tsunakawa.ta...@fujitsu.com
From: tsunakawa.ta...@fujitsu.com > (1) > + INIT_BUFFERTAG(newTag, > rnode.node, forkNum[j], firstDelBlock[j]); > > And you need to use i here, too. I remember the books "Code Complete" and/or "Readable Code" suggest to

RE: Inconsistent Japanese name order in v13 contributors list

2020-09-08 Thread tsunakawa.ta...@fujitsu.com
From: Fujii Masao > On 2020/09/09 14:15, Etsuro Fujita wrote: > > Attached is a patch to standardize Japanese names as given-name-first > > in the v13 contributors list as before. > > Using given-name-first order is our consensus? I was thinking we have not > reached that yet and our "vague" cons

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-09 Thread tsunakawa.ta...@fujitsu.com
Alexey-san, Sawada-san, cc: Fujii-san, From: Fujii Masao > But if we > implement 2PC as the improvement on FDW independently from PostgreSQL > sharding, I think that it's necessary to support other FDW. And this is our > direction, isn't it? I understand the same way as Fujii san. 2PC FDW is i

RE: Global snapshots

2020-09-09 Thread tsunakawa.ta...@fujitsu.com
Hi Andrey san, From: Andrey V. Lepikhov > > From: tsunakawa.ta...@fujitsu.com > >> While Clock-SI seems to be considered the best promising for global > >>> > Could you take a look at this patent? I'm afraid this is the Clock-SI > >>> > for MV

RE: SIGQUIT handling, redux

2020-09-09 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > This is straying a bit from the stated topic of this thread, but ... > I did some further looking around to see whether there were any > unsafe signal handlers besides SIGQUIT ones. The situation is not > too awful, but I did find several issues not already mentioned > in this th

RE: Global snapshots

2020-09-10 Thread tsunakawa.ta...@fujitsu.com
From: Fujii Masao > But I'm concerned about that it's really hard to say there is no patent risk > around that. I'm not sure who can judge there is no patent risk, > in the community. Maybe no one? Anyway, I was thinking that Google Spanner, > YugabyteDB, etc use the global transaction approach ba

RE: Implement UNLOGGED clause for COPY FROM

2020-09-10 Thread tsunakawa.ta...@fujitsu.com
From: Peter Smith > Earlier, Osumi-san was rejecting the idea of using ALTER TABLE tbl SET > UNLOGGED on basis that it is too time consuming for large data to > switch the table modes [1]. > Doesn't wal_level=none essentially just behave as if every table was > UNLOGGED; not just the ones we are

RE: Implement UNLOGGED clause for COPY FROM

2020-09-10 Thread tsunakawa.ta...@fujitsu.com
From: Peter Smith On Thu, Sep 10, 2020 at 7:16 PM tsunakawa.ta...@fujitsu.com > wrote: > > ALTER TABLE takes long time proportional to the amount of existing data, > while wal_level = none doesn't. > > Right, but if wal_level=none is considered OK for that table with &g

Fix of fake unlogged LSN initialization

2019-10-19 Thread tsunakawa.ta...@fujitsu.com
Hello, The attached trivial patch fixes the initialization of the fake unlogged LSN. Currently, BootstrapXLOG() in initdb sets the initial fake unlogged LSN to FirstNormalUnloggedLSN (=1000), but the recovery and pg_resetwal sets it to 1. The patch modifies the latter two cases to match init

RE: Fix of fake unlogged LSN initialization

2019-10-25 Thread tsunakawa.ta...@fujitsu.com
From: Simon Riggs > From xlogdefs.h added by 9155580: > /* >* First LSN to use for "fake" LSNs. >* >* Values smaller than this can be used for special per-AM purposes. >*/ > #define FirstNormalUnloggedLSN ((XLogRecPtr) 1000) Yeah, I had seen it,

RE: Fix of fake unlogged LSN initialization

2019-10-25 Thread tsunakawa.ta...@fujitsu.com
From: Dilip Kumar > I have noticed that in StartupXlog also we reset it with 1, you might > want to fix that as well? > > StartupXLOG > { > ... > /* > * Initialize unlogged LSN. On a clean shutdown, it's restored from the > * control file. On recovery, all unlogged relations are blown away, so >

RE: get_database_name() from background worker

2019-12-10 Thread tsunakawa.ta...@fujitsu.com
From: Koichi Suzuki > I'm writing an extension running on background workers and found > get_database_name() causes SEGV and found internally resource owner was > wet to NULL. Could anybody let me know how it happens and how I can use > this function. Argument to get_database_name() looks corr

RE: get_database_name() from background worker

2019-12-11 Thread tsunakawa.ta...@fujitsu.com
From: Koichi Suzuki > I'm not using this. Is this the must to use get_database_name()? I don't think pg_background is a must, but the system catalog access by get_database_name() should require database connection and transaction. See src/test/modules/worker_spi/worker_spi.c for an example o

RE: reducing memory usage by using "proxy" memory contexts?

2019-12-16 Thread tsunakawa.ta...@fujitsu.com
From: Andres Freund > We waste a lot of space due to all these small contexts. Even leaving > aside the overhead of the context and its blocks - not insignificant - > they are mostly between ~1/2 a ~1/4 empty. > > > But what if we had a new type of memory context that did not itself > manage mem

RE: Libpq support to connect to standby server as priority

2019-12-19 Thread tsunakawa.ta...@fujitsu.com
From: Greg Nancarrow > With the permission of the original patch author, Haribabu Kommi, I’ve > rationalized the existing 8 patches into 3 patches, merging patches > 1-5 and 6-7, and tidying up some documentation and code comments. I > also rebased them to the latest PG12 source code (as of Octobe

RE: Hooks for session start and end, take two

2019-12-19 Thread tsunakawa.ta...@fujitsu.com
From: Michael Paquier > Adding extra custom logging information, or plug that information > elsewhere than Postgres. I have use cases for that when it comes to > store external telemetry data or audit things for events happening > specifically in Postgres. > > Well, hook authors can do a lot of

RE: Implementing Incremental View Maintenance

2019-12-19 Thread tsunakawa.ta...@fujitsu.com
Hello, I'm starting to take a closer look at this feature. I've just finished reading the discussion, excluding other referenced materials. The following IVM wiki page returns an error. Does anybody know what's wrong? https://wiki.postgresql.org/wiki/Incremental_View_Maintenance [screen] --

RE: Implementing Incremental View Maintenance

2019-12-22 Thread tsunakawa.ta...@fujitsu.com
From: Tatsuo Ishii > > The following IVM wiki page returns an error. Does anybody know what's > wrong? > > > > https://wiki.postgresql.org/wiki/Incremental_View_Maintenance > > I don't have any problem with the page. Maybe temporary error? Yeah, I can see it now. I could see it on the weekend.

RE: Implementing Incremental View Maintenance

2019-12-22 Thread tsunakawa.ta...@fujitsu.com
From: legrand legrand > For each insert into a base table there are 3 statements: > - ANALYZE pg_temp_3.pg_temp_81976 > - WITH updt AS ( UPDATE public.mv1 AS mv SET __ivm_count__ = ... > - DROP TABLE pg_temp_3.pg_temp_81976 Does it also include CREATE TEMPORARY TABLE, because there's DROP? I re

RE: Implementing Incremental View Maintenance

2019-12-22 Thread tsunakawa.ta...@fujitsu.com
From: Tatsuo Ishii > the target data? In the discussion, someone referred to master data with low > update frequency, because the proposed IVM implementation adds triggers on > source tables, which limits the applicability to update-heavy tables. > > But if you want to get always up-to-data you

RE: Implementing Incremental View Maintenance

2019-12-23 Thread tsunakawa.ta...@fujitsu.com
From: Yugo Nagata > 1. Create a temporary table only once at the first view maintenance in > this session. This is possible if we store names or oid of temporary > tables used for each materialized view in memory. However, users may > access to these temptables whenever during the session. > > 2.

RE: Implementing Incremental View Maintenance

2019-12-23 Thread tsunakawa.ta...@fujitsu.com
From: Tatsuo Ishii > First of all, we do not think that current approach is the final > one. Instead we want to implement IVM feature one by one: i.e. we > start with "immediate update" approach, because it's simple and easier > to implement. Then we will add "deferred update" mode later on. I ag

RE: Implementing Incremental View Maintenance

2019-12-23 Thread tsunakawa.ta...@fujitsu.com
From: Yugo Nagata > On Mon, 23 Dec 2019 08:08:53 + > "tsunakawa.ta...@fujitsu.com" wrote: > > How about unlogged tables ? I thought the point of using a temp table is to > avoid WAL overhead. > > Hmm... this might be another option. However, if we use unl

RE: Implementing Incremental View Maintenance

2019-12-24 Thread tsunakawa.ta...@fujitsu.com
From: Tatsuo Ishii > AFAIK benefit of ON STATEMENT is the transaction can see the result of > update to the base tables. With ON COMMIT, the transaction does not > see the result until the transaction commits. > > Well, I can see use cases of IVM in both DWH and OLTP. > > For example, a user cre

RE: Libpq support to connect to standby server as priority

2019-12-26 Thread tsunakawa.ta...@fujitsu.com
From: Alvaro Herrera > I'm not sure I understand why we end up with "prefer-read" in addition > to "prefer-standby" (and similar seeming redundancy between "primary" > and "read-write"). Do we really need more than one way to identify > hosts' roles? It seems 0001 adds the "prefer-read" modes by

RE: Libpq support to connect to standby server as priority

2020-01-05 Thread tsunakawa.ta...@fujitsu.com
From: Alvaro Herrera > So, we can know whether server is primary/standby by checking > in_recovery, as opposed to knowing whether read-write which is done by > checking transaction_read_only. So we can keep read-write as a synonym > for "primary", and check in_recovery when used in servers that s

RE: Let people set host(no)ssl settings from initdb

2020-01-07 Thread tsunakawa.ta...@fujitsu.com
From: David Fetter > > But I see two problems with the proposed approach: (1) initdb > > doesn't support setting up SSL, so the only thing you can achieve > > here is to reject all TCP/IP connections, until you have set up SSL. > > I don't believe any special setup is needed to require TLS for th

RE: SLRU statistics

2020-01-19 Thread tsunakawa.ta...@fujitsu.com
From: Tomas Vondra > One of the stats I occasionally wanted to know are stats for the SLRU > stats (we have couple of those - clog, subtrans, ...). So here is a WIP > version of a patch adding that. How can users take advantage of this information? I think we also need the ability to set the si

RE: SLRU statistics

2020-01-20 Thread tsunakawa.ta...@fujitsu.com
From: Tomas Vondra > You're right the users can't really take advantage of this - my primary > motivation was providing a feedback for devs, benchmarking etc. That > might have been done with DEBUG messages or something, but this seems > more convenient. Understood. I'm in favor of adding perfor

RE: Schema variables - new implementation for Postgres 15

2021-04-15 Thread tsunakawa.ta...@fujitsu.com
From: Pavel Stehule -- do $$ declare x int ; begin for i in 1..100 loop let ooo = i; end loop; end; $$; variant 1 .. 1500 ms variant 2 with PLpgSQL support .. 140 ms variant 2 without PLpgSQL support 9000 ms ---

RE: Schema variables - new implementation for Postgres 15

2021-04-16 Thread tsunakawa.ta...@fujitsu.com
From: Pavel Stehule -- I am sorry, but in this topic we have different opinions. The variables in PLpgSQL are not transactional too (same is true in Perl, Python, ...). Session variables in Oracle, MS SQL, DB2, MySQL are not transactional too. My p

[bug?] Missed parallel safety checks, and wrong parallel safety

2021-04-20 Thread tsunakawa.ta...@fujitsu.com
Hello, I think we've found a few existing problems with handling the parallel safety of functions while doing an experiment. Could I hear your opinions on what we should do? I'd be willing to create and submit a patch to fix them. The experiment is to add a parallel safety check in FunctionC

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-04-20 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > Bharath Rupireddy writes: > > IMO, the reason for not checking the parallel safety of the support > > functions is that the functions themselves can have whole lot of other > > functions (can be nested as well) which might be quite hard to check > > at the planning time. That is

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-04-20 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > [ raised eyebrow... ] I find it very hard to understand why that would > be necessary, or even a good idea. Not least because there's no spare > room there; you'd have to incur a substantial enlargement of the > array to add another flag. But also, that would indeed lock down >

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-04-21 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > Amit Kapila writes: > > IIUC, the idea here is to check for parallel safety of functions at > > someplace in the code during function invocation so that if we execute > > any parallel unsafe/restricted function via parallel worker then we > > error out. If so, isn't it possible t

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-04-22 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > I agree that it's better to mark the function with correct parallel safety > lable. > Especially for the above functions which will be executed in parallel mode. > It will be friendly to developer and user who is working on something related > to > parallel test. > > So

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-04-22 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > For approach 1): I think it could result in infinite recursion. > > For example: > If we first access one built-in function A which have not been cached, > it need access the pg_proc, When accessing the pg_proc, it internally still > need > some built-in function B to sc

RE: A test for replay of regression tests

2021-04-22 Thread tsunakawa.ta...@fujitsu.com
From: Thomas Munro > We have automated tests for many specific replication and recovery scenarios, > but nothing that tests replay of a wide range of records. > People working on recovery code often use installcheck (presumably along > with other custom tests) to exercise it, sometimes with > wal_

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-05-05 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > On Tue, May 4, 2021 at 11:47 PM Greg Nancarrow > wrote: > > Problem is, for built-in functions, the changes are allowed, but for > > some properties (like strict) the allowed changes don't actually take > > effect (this is what Amit was referring to - so why allow those > > ch

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-05-06 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > On Wed, Apr 28, 2021 at 9:42 PM houzj.f...@fujitsu.com > wrote: > > So, If we do not want to lock down the parallel safety of built-in > > functions. > > It seems we can try to fetch the proparallel from pg_proc for built-in > > function > > in fmgr_info_cxt_security too. To

RE: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-05-06 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > On Wed, May 5, 2021 at 10:54 PM tsunakawa.ta...@fujitsu.com > wrote: > > As proposed in this thread and/or "Parallel INSERT SELECT take 2", we > thought of detecting parallel unsafe function execution during SQL statement > execution, instead

RE: batch fdw insert bug (Postgres 14)

2021-05-09 Thread tsunakawa.ta...@fujitsu.com
From: Tomas Vondra > I think the simplest fix is simply to pstrdup() the query when building > fmstate in create_foreign_modify. We've already been doing that for > orig_query anyway. That seems cleaner than printing the last query we > build (which would be confusing I think). > > I've pushed a

RE: Inaccurate error message when set fdw batch_size to 0

2021-05-09 Thread tsunakawa.ta...@fujitsu.com
From: houzj.f...@fujitsu.com > So, is it better to change the error message to “fetch_size requires a > positive integer value” ? > I also found fetch_size has the similar issue, attaching a patch to fix this. I have a faint memory that I fixed them after receiving the same feedback from someo

RE: Support for VACUUMing Foreign Tables

2021-05-13 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > I think it will be useful to allow foreign tables to be VACUUMed if > the underlying FDW supports, currently VACUUM doesn't support foreign > tables, see [1]. Could you let us imagine more concretely how useful it will be? While TRUNCATE can be part of an application's

RE: FDW and connections

2021-05-16 Thread tsunakawa.ta...@fujitsu.com
From: Phil Godfrin My question is - how does the call to GetConnection() know what port to use? Lets say we're using PGBouncer to connect on the local server at port 6432, but there is no pgbouncer listening at the foreign server, what port gets passed? My first thought is whatever the client c

RE: Support for VACUUMing Foreign Tables

2021-05-16 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > This can be useful in situations like where there are many remote > postgres servers that are connected to a single coordinator on which > foreign tables are defined for each of the remote tables. In this > case, the DBA (or whoever is responsible to do that job) doesn't

RE: Bug in query rewriter - hasModifyingCTE not getting set

2021-05-17 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > In view of this, maybe the right thing is to disallow modifying CTEs > in rule actions in the first place. I see we already do that for > views (i.e. ON SELECT rules), but they're not really any safer in > other types of rules. You meant by views something like the following, di

RE: Skip partition tuple routing with constant partition key

2021-05-19 Thread tsunakawa.ta...@fujitsu.com
From: Amit Langote > On Tue, May 18, 2021 at 11:11 AM houzj.f...@fujitsu.com > wrote: > > For some big data scenario, we sometimes transfer data from one table(only > store not expired data) > > to another table(historical data) for future analysis. > > In this case, we import data into historica

RE: Bug in query rewriter - hasModifyingCTE not getting set

2021-05-20 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > I think either the bit about rule_action is unnecessary, or most of > the code immediately above this is wrong, because it's only updating > flags in sub_action. Why do you think it's necessary to change > rule_action in addition to sub_action? Finally, I think I've understood w

RE: Bug in query rewriter - hasModifyingCTE not getting set

2021-05-20 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > "tsunakawa.ta...@fujitsu.com" writes: > > Finally, I think I've understood what you meant. Yes, the current code > > seems > to be wrong. > > I'm fairly skeptical of this claim, because that code has stood for a > long t

RE: Skip partition tuple routing with constant partition key

2021-05-24 Thread tsunakawa.ta...@fujitsu.com
From: houzj.f...@fujitsu.com > I think this patch can solve the performance degradation of key expression > after > applying the [Save the last partition] patch. > Besides, this could be a separate patch which can improve some more cases. > Thoughts ? Thank you for proposing an impressive improv

RE: Skip partition tuple routing with constant partition key

2021-05-24 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > It seems a little strange, I have compiled it alone in two different linux > machine > and did > not find such an error. Did you compile it on a windows machine ? On Linux, it produces: gcc -std=gnu99 -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-s\ tat

RE: Skip partition tuple routing with constant partition key

2021-05-24 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > Ah, Maybe I found the issue. > Attaching a new patch, please have a try on this patch. Thanks, it has compiled perfectly without any warning. Regards Takayuki Tsunakawa

RE: Parallel Inserts in CREATE TABLE AS

2021-05-25 Thread tsunakawa.ta...@fujitsu.com
Bharath-san, all, Hmm, I didn't experience performance degradation on my poor-man's Linux VM (4 CPU, 4 GB RAM, HDD)... [benchmark preparation] autovacuum = off shared_buffers = 1GB checkpoint_timeout = 1h max_wal_size = 8GB min_wal_size = 8GB (other settings to enable parallelism) CREATE UNLOGG

RE: Parallel Inserts in CREATE TABLE AS

2021-05-25 Thread tsunakawa.ta...@fujitsu.com
From: houzj.f...@fujitsu.com > + /* > + * We don't need to skip contacting FSM while inserting tuples > for > + * parallel mode, while extending the relations, workers > instead of > + * blocking on a page while another worker is inserting, can >

RE: Batch insert in CTAS/MatView code

2021-05-25 Thread tsunakawa.ta...@fujitsu.com
Hello Paul-san, From: Daniel Gustafsson > In an off-list discussion with Paul, we decided to withdraw this patch for now > and instead create a new entry when there is a re-worked patch. This has > now > been done in the CF app. Would you mind if I take over this patch for PG 15? I find this p

RE: Batch insert in CTAS/MatView code

2021-05-26 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > I think the "New Table Access Methods for Multi and Single Inserts" > patches at [1] make multi insert usage easy for COPY, CTAS/Create Mat > View, Refresh Mat View and so on. It also has a patch for multi > inserts in CTAS and Refresh Mat View > (v6-0002-CTAS-and-REFRESH

RE: Parallel Inserts in CREATE TABLE AS

2021-05-26 Thread tsunakawa.ta...@fujitsu.com
Thank you for the detailed analysis, I'll look into it too. (The times have changed...) From: Bharath Rupireddy > Well, one might think to add more blocks at a time, say > Min(1024, lockWaiters * 128/256/512) than currently extraBlocks = > Min(512, lockWaiters * 20);. This will work (i.e. we do

RE: Parallel Inserts in CREATE TABLE AS

2021-05-26 Thread tsunakawa.ta...@fujitsu.com
From: Dilip Kumar > I think some other cause of contention on relation extension locks are > 1. CTAS is using a buffer strategy and due to that, it might need to > evict out the buffer frequently for getting the new block in. Maybe > we can identify by turning off the buffer strategy for CTAS and

<    1   2   3   4   5   >