Re: Improve tab completion for COPY

2025-06-09 Thread torikoshia
On 2025-06-03 17:58, Yugo Nagata wrote: Thank you for updating the patch. It looks good and I confirmed that this works as expected. Thanks for your review! BTW this is a small patch, but it does two things: (1) adds tab completion support for the REJECT_LIMIT option, which was introduced in

Re: expand on_error ignore error handling scope

2025-06-04 Thread torikoshia
On 2025-04-30 09:00, jian he wrote: hi. Currently, (on_error ignore) only handles data type incompatibility errors. However, we can extend its functionality to also handle errors caused by extra data (additional columns) or missing data (fewer columns), I think. on_error is only applicable t

Re: RFC: Logging plan of the running query

2025-06-02 Thread torikoshia
On 2025-05-23 17:50, Atsushi Torikoshi wrote: Thanks for the idea and the sample patch! Agreed. I’ll go ahead and implement a new patch based on this approach. Updated the patch. Here are some notes: As with the previous patches, this one wraps not only the currently executing plan node but

Minor adjustment to pg_aios output naming

2025-05-20 Thread torikoshia
Hi, I've noticed a minor inconsistency in the output of pg_aios. According to the documentation, the values for 'operation' are described as: readv, a vectored read ... writev, a vectored write However, in actual output, they appear as read and write -- without the trailing v: =#

Re: RFC: Logging plan of the running query

2025-05-20 Thread torikoshia
On Sat, Apr 5, 2025 at 3:14 PM Atsushi Torikoshi wrote: On Thu, Apr 3, 2025 at 11:10 PM Robert Haas wrote: Do we really need ExecProcNodeOriginal? Can we find some way to reuse ExecProcNodeReal instead of making the structure bigger? I also wanted to implement this without adding elements to

Re: RFC: Allow EXPLAIN to Output Page Fault Information

2025-05-08 Thread torikoshia
On 2025-04-11 22:18, torikoshia wrote: On 2025-03-25 10:27, torikoshia wrote: On 2025-03-22 20:23, Jelte Fennema-Nio wrote: On Wed, 19 Mar 2025 at 14:15, torikoshia wrote: BTW based on your discussion, I thought this patch could not be merged anytime soon. Does that align with your

Re: Improve tab completion for COPY

2025-05-08 Thread torikoshia
Thanks for your comments! On 2025-05-08 08:53, Yugo Nagata wrote: On Wed, 7 May 2025 14:36:35 -0700 Masahiko Sawada wrote: On Wed, May 7, 2025 at 6:23 AM Yugo Nagata wrote: > > On Wed, 7 May 2025 15:39:26 +0900 > torikoshia wrote: > > > Hi, > > > > I notice

Improve tab completion for COPY

2025-05-06 Thread torikoshia
Hi, I noticed that REJECT_LIMIT, an option available for COPY FROM, is not currently supported in psql's tab completion. Additionally, some options are only valid for COPY FROM or COPY TO, i.e. FREEZE, ON_ERROR, FORCE_QUOTE, but psql currently suggests them for both COPY FROM and COPY TO. As

Re: pgsql: Add function to log the memory contexts of specified backend pro

2025-05-01 Thread torikoshia
On 2025-05-02 09:02, Fujii Masao wrote: On 2025/05/02 2:27, Fujii Masao wrote: On 2025/05/01 21:42, Robert Haas wrote: On Thu, May 1, 2025 at 3:53 AM Fujii Masao wrote: Just idea, what do you think about adding a flag to indicate whether ProcessLogMemoryContextInterrupt() is currently runni

Re: RFC: Logging plan of the running query

2025-04-30 Thread torikoshia
Hi, On 2025-04-28 08:55, Hannu Krosing wrote: Have you also checked out https://github.com/postgrespro/pg_query_state which logs running query plan AND collected counts and timings as a response to a signal? Yes. For example, see the discussion: https://www.postgresql.org/message-id/d68c3ae316

Re: RFC: Logging plan of the running query

2025-04-24 Thread torikoshia
Hi, Attached a rebased version of the patch. -- Regards, -- Atsushi Torikoshi Seconded from NTT DATA GROUP CORPORATION to SRA OSS K.K.From 112467238f585bb3398d86ac6e1a71caa6549fb4 Mon Sep 17 00:00:00 2001 From: Atsushi Torikoshi Date: Thu, 24 Apr 2025 20:50:41 +0900 Subject: [PATCH v44] Add fu

Re: Align memory context level numbering in pg_log_backend_memory_contexts()

2025-04-17 Thread torikoshia
Thanks for your review, Melih and Rahila. On 2025-04-17 21:25, Rahila Syed wrote: Hi, The attached patch is how I think we should do it. Thank you for the patch. I tested this patch and it works fine. I agree with the changes made in it. Regarding v2 patch, - int

Re: Align memory context level numbering in pg_log_backend_memory_contexts()

2025-04-17 Thread torikoshia
On 2025-04-17 07:31, David Rowley wrote: On Thu, 17 Apr 2025 at 02:20, torikoshia wrote: Regarding the implementation: In the initial patch attached, I naïvely incremented the level just before emitting the log line. However, it might be cleaner to simply initialize the level variable to 1

Re: Align memory context level numbering in pg_log_backend_memory_contexts()

2025-04-16 Thread torikoshia
On 2025-04-16 06:18, Daniel Gustafsson wrote: On 15 Apr 2025, at 23:03, David Rowley wrote: My vote is to make the levels 1-based in all locations where we output the context information. I agree with this, pg_get_process_memory_contexts() also use 1-based levels fwiw. +1. I believe the

Align memory context level numbering in pg_log_backend_memory_contexts()

2025-04-15 Thread torikoshia
Hi, With commit 32d3ed81, pg_backend_memory_contexts view will start numbering memory context levels from 1 instead of 0 in PostgreSQL 18. For example: =# select name, level from pg_backend_memory_contexts; name | level +--- TopM

Re: RFC: Allow EXPLAIN to Output Page Fault Information

2025-04-11 Thread torikoshia
On 2025-03-25 10:27, torikoshia wrote: On 2025-03-22 20:23, Jelte Fennema-Nio wrote: On Wed, 19 Mar 2025 at 14:15, torikoshia wrote: BTW based on your discussion, I thought this patch could not be merged anytime soon. Does that align with your understanding? Yeah, that aligns with my

Re: Change log level for notifying hot standby is waiting non-overflowed snapshot

2025-04-05 Thread torikoshia
On 2025-03-31 12:51, Fujii Masao wrote: I had the same thought during our off-list discussion. However, after reviewing the recovery code - such as recoveryStopsBefore(), which checks whether a consistent state is reached - I now believe the manual’s definition of a consistent state may be incorr

Re: Change log level for notifying hot standby is waiting non-overflowed snapshot

2025-04-04 Thread torikoshia
On 2025-04-01 01:12, Fujii Masao wrote: On 2025/03/31 22:45, Yugo Nagata wrote: I prefer this approach clarifying that consistency and subtransaction overflow are separate concepts in the documentation. Here are minor comments on the patch: Thanks for the review! - case CAC_NOTC

Re: RFC: Logging plan of the running query

2025-04-04 Thread torikoshia
On 2025-04-02 03:52, Robert Haas wrote: Thank you for review! On Fri, Mar 21, 2025 at 8:40 AM torikoshia wrote: Rebased it again. On 2025-03-10 14:10, torikoshia wrote: > BTW the patch adds about 400 lines to explain.c and it may be better > to split the file as well as 9173e8b6046,

Re: Proposal: Progressive explain

2025-04-04 Thread torikoshia
On Fri, Mar 7, 2025 at 6:43 AM Rafael Thofehrn Castro wrote: The wrapper code was implemented by torikoshia (torikoshia(at)oss(dot)nttdata(dot)com), so adding the credits here. On Thu, Mar 20, 2025 at 5:35 AM Robert Haas wrote: Without having the prior discussion near to hand, I *think

Re: RFC: Logging plan of the running query

2025-04-02 Thread torikoshia
On 2025-04-03 01:22, Sami Imseih wrote: > FWIW, I have been thinking about auto_explain for another task, > remote plans for fdw [0], and perhaps there are now other good > reasons, some that you mention, that can be simplified if "auto_explain" > becomes a core feature. This could be a proposal

Re: RFC: Logging plan of the running query

2025-04-02 Thread torikoshia
On 2025-04-01 03:51, Sadeq Dousti wrote: Hi, Sergey and I reviewed this patch and have a few comments, listed below. Thanks for your review! BTW the patch adds about 400 lines to explain.c and it may be better to split the file as well as 9173e8b6046, but I leave it as it is for now. 1. A

Re: Change log level for notifying hot standby is waiting non-overflowed snapshot

2025-04-02 Thread torikoshia
On 2025-04-02 15:21, Fujii Masao wrote: On 2025/04/01 20:54, torikoshia wrote: Thank you for updating the patch! LGTM. I've pushed the patch. Thanks! Thanks a lot! I feel like changing the status to 'Ready for Committer', but since Nagata-san may have additional comments,

Re: Proposal: Progressive explain

2025-04-01 Thread torikoshia
On 2025-04-01 15:23, Rafael Thofehrn Castro wrote: Hello again, ERROR: could not attach to dynamic shared area In addition to that refactoring issue, the current patch had a race condition in pg_stat_progress_explain to access the DSA of a process running a query that gets aborted. While di

Re: RFC: Logging plan of the running query

2025-04-01 Thread torikoshia
On 2025-04-01 21:32, Robert Haas wrote: Thanks for your comment. On Mon, Mar 31, 2025 at 9:43 PM torikoshia wrote: Previously, Rafael proposed a patch in this thread that added execution progress tracking. However, I felt that expanding the scope could make it harder to get the patch

Re: RFC: Logging plan of the running query

2025-03-31 Thread torikoshia
On 2025-04-01 04:24, Robert Haas wrote: On Fri, Mar 21, 2025 at 8:40 AM torikoshia wrote: Rebased it again. Hi, I apologize for not having noticed this thread sooner. Thank you for your checking! No worries. I just became aware of it as a result of a discussion in the hacking Discord

Re: Change log level for notifying hot standby is waiting non-overflowed snapshot

2025-03-27 Thread torikoshia
On 2025-03-27 11:06, torikoshia wrote: Hi, I had another off-list discussion with Fujii-san, and according to the following manual[1], it seems that a transaction with an overflowed subtransaction is already considered inconsistent: Reaching a consistent state can also be delayed in the

Re: Change log level for notifying hot standby is waiting non-overflowed snapshot

2025-03-26 Thread torikoshia
Hi, I had another off-list discussion with Fujii-san, and according to the following manual[1], it seems that a transaction with an overflowed subtransaction is already considered inconsistent: Reaching a consistent state can also be delayed in the presence of both of these conditions:

Re: RFC: Allow EXPLAIN to Output Page Fault Information

2025-03-24 Thread torikoshia
On 2025-03-22 20:23, Jelte Fennema-Nio wrote: On Wed, 19 Mar 2025 at 14:15, torikoshia wrote: BTW based on your discussion, I thought this patch could not be merged anytime soon. Does that align with your understanding? Yeah, that aligns with my understanding. I don't think it's

Re: Change log level for notifying hot standby is waiting non-overflowed snapshot

2025-03-24 Thread torikoshia
On 2025-03-24 00:08, Fujii Masao wrote: Do you also think the errhint message is unnecessary? I agree with your idea to add a description of the overflowed subtransaction in the manual, but I'm not sure all users will be able to find it. Some people may not understand what needs to be done to

Re: RFC: Logging plan of the running query

2025-03-21 Thread torikoshia
Hi, Rebased it again. On 2025-03-10 14:10, torikoshia wrote: BTW the patch adds about 400 lines to explain.c and it may be better to split the file as well as 9173e8b6046, but I leave it as it is for now. This part remains unchanged. -- Regards, -- Atsushi Torikoshi Seconded from NTT DATA

Re: Change log level for notifying hot standby is waiting non-overflowed snapshot

2025-03-21 Thread torikoshia
Hi, On 2025-03-21 02:15, Fujii Masao wrote: Thanks for your review! Personally, I feel 1st patch may be sufficient, but I would appreciate any feedback. Agreed. - errdetail("Consistent recovery state has not been yet reached."))); +

Re: RFC: Allow EXPLAIN to Output Page Fault Information

2025-03-19 Thread torikoshia
Hi, On 2025-03-18 08:52, Jelte Fennema-Nio wrote: On Mon, 10 Feb 2025 at 14:23, torikoshia wrote: Thanks for reviewing the patch and comments! Fixed issues you pointed out and attached v2 patch. This patch needs a rebase, because it's failing to compile currently. So I marked th

Re: Change log level for notifying hot standby is waiting non-overflowed snapshot

2025-03-12 Thread torikoshia
Hi, After an off-list discussion with Fujii-san, I'm now trying to modify the following message that is output when a client attempts to connect instead of changing the log level as the original proposal: $ psql: error: connection to server at "localhost" (::1), port 5433 failed: FATAL: t

Re: RFC: Logging plan of the running query

2025-03-09 Thread torikoshia
On 2025-03-09 00:42, Akshat Jaimini wrote: Hi, I think there is still some problem with the patch. I am not able to apply it to the master branch. Can you please take another look at it? Thanks for pointing it out! Modified it. BTW the patch adds about 400 lines to explain.c and it may be bet

Re: Change log level for notifying hot standby is waiting non-overflowed snapshot

2025-03-03 Thread torikoshia
On 2025-03-04 03:17, Fujii Masao wrote: Agreed and I feel that a message suggesting something like "check if there are any overflowing transactions on the primary side" would make it useful. I’m wondering if this message might still be confusing for users. Would they immediately understand wh

Re: Change log level for notifying hot standby is waiting non-overflowed snapshot

2025-03-03 Thread torikoshia
On 2025-03-03 13:10, Fujii Masao wrote: Thanks for your comments! On 2025/02/03 22:35, torikoshia wrote: Hi, When a hot standby is restarted in a state where subtransactions have overflowed, it may become inaccessible:   $ psql: error: connection to server at "localhost" (::1),

Re: RFC: Allow EXPLAIN to Output Page Fault Information

2025-02-10 Thread torikoshia
On 2025-02-10 05:06, Jelte Fennema-Nio wrote: Thanks for reviewing the patch and comments! Fixed issues you pointed out and attached v2 patch. On Sun, 9 Feb 2025 at 19:05, Tom Lane wrote: Andres Freund writes: > I'm somewhat against this patch, as it's fairly fundamentally incompatible > wi

Re: Enhancing Memory Context Statistics Reporting

2025-02-10 Thread torikoshia
On 2025-02-03 21:47, Rahila Syed wrote: Hi, Just idea; as an another option, how about blocking new requests to the target process (e.g., causing them to fail with an error or returning NULL with a warning) if a previous request is still pending? Users can simply retry the request if it

Change log level for notifying hot standby is waiting non-overflowed snapshot

2025-02-03 Thread torikoshia
Hi, When a hot standby is restarted in a state where subtransactions have overflowed, it may become inaccessible: $ psql: error: connection to server at "localhost" (::1), port 5433 failed: FATAL: the database system is not yet accepting connections DETAIL: Consistent recovery

Re: RFC: Logging plan of the running query

2025-02-03 Thread torikoshia
Hi, Rebased the patch since it couldn't be applied anymore. -- Regards, -- Atsushi Torikoshi Seconded from NTT DATA GROUP CORPORATION to SRA OSS K.K.From 3f580c17214595eb8d6013674f5f054ec352ab7a Mon Sep 17 00:00:00 2001 From: Atsushi Torikoshi Date: Mon, 3 Feb 2025 21:22:40 +0900 Subject: [PAT

Re: RFC: Allow EXPLAIN to Output Page Fault Information

2025-01-27 Thread torikoshia
Hi, On Tue, Jan 7, 2025 at 5:09 AM Jelte Fennema-Nio wrote: The core functionality works well in my opinion. I think it makes sense to spend the effort to move this from PoC quality to something committable. Below some of the things that are necessary to do that after an initial pass over the

Re: RFC: Allow EXPLAIN to Output Page Fault Information

2025-01-06 Thread torikoshia
On Tue, Dec 31, 2024 at 1:39 AM Tom Lane wrote: Bruce Momjian writes: I certainly would love to see storage I/O numbers as distinct from kernel read I/O numbers. Me too, but I think it is 100% wishful thinking to imagine that page fault counts match up with that. Maybe there are filesyste

Re: RFC: Allow EXPLAIN to Output Page Fault Information

2024-12-25 Thread torikoshia
On 2024-12-25 00:52, Tom Lane wrote: torikoshia writes: I have attached a PoC patch that modifies EXPLAIN to include page fault information during both the planning and execution phases of a query. Surely these numbers would be too unstable to be worth anything. Thanks for your comment

RFC: Allow EXPLAIN to Output Page Fault Information

2024-12-24 Thread torikoshia
Hi, When reading the output of EXPLAIN (ANALYZE) to diagnose slow queries for our customers, I often want to know how many page faults occurred, especially major page faults, which involve disk access. Currently, the BUFFERS option in EXPLAIN provides information on whether a page was found

Re: Enhancing Memory Context Statistics Reporting

2024-12-16 Thread torikoshia
Hi, Thanks for updating the patch and here are some comments: 'path' column of pg_get_process_memory_contexts() begins with 0, but that column of pg_backend_memory_contexts view begins with 1: =# select path FROM pg_get_process_memory_contexts('20271', false); path --- {0} {0,

Re: Add reject_limit option to file_fdw

2024-11-20 Thread torikoshia
On 2024-11-21 00:43, Fujii Masao wrote: On 2024/11/19 21:40, torikoshia wrote: These messages may be unexpected for some users because the documentation of fild_fdw does not explicitly describe that file_fdw uses COPY internally. (I can find several wordings like "as COPY", though

Re: Add reject_limit option to file_fdw

2024-11-19 Thread torikoshia
On 2024-11-13 23:17, Yugo NAGATA wrote: On Wed, 13 Nov 2024 21:48:10 +0900 torikoshia wrote: On 2024-11-12 14:51, Yugo Nagata wrote: Thanks for your review! > On Tue, 12 Nov 2024 10:16:50 +0900 > torikoshia wrote: > >> On 2024-11-12 01:49, Fujii Masao wrote: >>

Re: Change COPY ... ON_ERROR ignore to ON_ERROR ignore_row

2024-11-14 Thread torikoshia
On 2024-11-13 22:02, Yugo NAGATA wrote: On Tue, 12 Nov 2024 17:38:25 +0900 torikoshia wrote: On 2024-11-12 14:17, Yugo Nagata wrote: > On Tue, 12 Nov 2024 14:03:50 +0900 > Yugo Nagata wrote: > >> On Tue, 12 Nov 2024 01:27:53 +0500 >> Kirill Reshke wrote: >> >&g

Re: Add reject_limit option to file_fdw

2024-11-13 Thread torikoshia
On 2024-11-12 14:51, Yugo Nagata wrote: Thanks for your review! On Tue, 12 Nov 2024 10:16:50 +0900 torikoshia wrote: On 2024-11-12 01:49, Fujii Masao wrote: > On 2024/11/11 21:45, torikoshia wrote: >>> Thanks for adding the comment. It clearly states that REJECT_LIMIT >

Re: Add reject_limit option to file_fdw

2024-11-13 Thread torikoshia
On 2024-11-12 15:23, Kirill Reshke wrote: Thanks for your review! On Tue, 12 Nov 2024 at 06:17, torikoshia wrote: On 2024-11-12 01:49, Fujii Masao wrote: > On 2024/11/11 21:45, torikoshia wrote: >>> Thanks for adding the comment. It clearly states that REJECT_LIMIT >>>

Re: Change COPY ... ON_ERROR ignore to ON_ERROR ignore_row

2024-11-12 Thread torikoshia
On 2024-11-12 14:17, Yugo Nagata wrote: On Tue, 12 Nov 2024 14:03:50 +0900 Yugo Nagata wrote: On Tue, 12 Nov 2024 01:27:53 +0500 Kirill Reshke wrote: > On Mon, 11 Nov 2024 at 16:11, torikoshia wrote: > > > > On 2024-11-09 21:55, Kirill Reshke wrote: > > > >

Re: Add reject_limit option to file_fdw

2024-11-11 Thread torikoshia
On 2024-11-12 01:49, Fujii Masao wrote: On 2024/11/11 21:45, torikoshia wrote: Thanks for adding the comment. It clearly states that REJECT_LIMIT can be a single-quoted string. However, it might also be helpful to mention that it can be provided as an int64 in the COPY command option. How

Re: Add reject_limit option to file_fdw

2024-11-11 Thread torikoshia
On 2024-11-08 01:44, Fujii Masao wrote: Thanks for your review! On 2024/11/05 22:30, torikoshia wrote: Thanks for the patch! Could you add it to the next CommitFest? Added an entry for this patch: https://commitfest.postgresql.org/50/5331/ Thanks! +ALTER FOREIGN TABLE agg_bad OPTIONS

Re: Change COPY ... ON_ERROR ignore to ON_ERROR ignore_row

2024-11-11 Thread torikoshia
On 2024-11-09 21:55, Kirill Reshke wrote: Thanks for working on this! On Thu, 7 Nov 2024 at 23:00, Fujii Masao wrote: On 2024/10/26 6:03, Kirill Reshke wrote: > when the REJECT LIMIT is set to some non-zero number and the number of > row NULL replacements exceeds the limit, is it OK to fai

Re: Add reject_limit option to file_fdw

2024-11-05 Thread torikoshia
On 2024-10-18 01:51, Fujii Masao wrote: Thanks for your review and sorry for my late reply. On 2024/10/17 22:45, torikoshia wrote: Hi, 4ac2a9bec introduced reject_limit option to the COPY command, and I was wondering if it might be beneficial to add the same option to file_fdw. Although

Re: RFC: Logging plan of the running query

2024-10-28 Thread torikoshia
Hi I've recently resumed work on this proposal. I was revising tests that use the injection point, but given that there was a recent talk at PGConf.EU about further features based on this proposal [1], I thought it might be of interest to some. Therefore, I'll share a patch that removes the i

Re: Enhancing Memory Context Statistics Reporting

2024-10-24 Thread torikoshia
On 2024-10-24 14:59, Rahila Syed wrote: Hi Torikoshia, Thank you for reviewing the patch! On Wed, Oct 23, 2024 at 9:28 AM torikoshia wrote: On 2024-10-22 03:24, Rahila Syed wrote: Hi, PostgreSQL provides following capabilities for reporting memory contexts statistics. 1

Re: Enhancing Memory Context Statistics Reporting

2024-10-22 Thread torikoshia
On 2024-10-22 03:24, Rahila Syed wrote: Hi, PostgreSQL provides following capabilities for reporting memory contexts statistics. 1. pg_get_backend_memory_contexts(); [1] 2. pg_log_backend_memory_contexts(pid); [2] [1] provides a view of memory context statistics for a local backend, while [2]

Add reject_limit option to file_fdw

2024-10-17 Thread torikoshia
Hi, 4ac2a9bec introduced reject_limit option to the COPY command, and I was wondering if it might be beneficial to add the same option to file_fdw. Although there may be fewer practical use cases compared to COPY, it could still be useful in situations where the file being read via file_fdw

Re: Add new COPY option REJECT_LIMIT

2024-10-08 Thread torikoshia
On 2024-10-08 18:39, Fujii Masao wrote: On 2024/10/07 21:51, torikoshia wrote: While reviewing, I also noticed that the check for "opts_out->binary && opts_out->on_error != COPY_ON_ERROR_STOP" is similarly placed before setting the defaults, which might not be co

Re: Add new COPY option REJECT_LIMIT

2024-10-07 Thread torikoshia
Thanks for your review! On Thu, Oct 3, 2024 at 4:27 PM jian he wrote: mentioning maxerror is a bigint type or explicitly mentioning the maximum allowed value of "maxerror" would be great. Added a description that it allows positive bigint. On Thu, Oct 3, 2024 at 11:43 PM Fujii Masao w

Re: Add on_error and log_verbosity options to file_fdw

2024-10-03 Thread torikoshia
On 2024-10-03 18:03, Fujii Masao wrote: On 2024/10/03 13:23, Masahiko Sawada wrote: On Tue, Oct 1, 2024 at 11:34 PM Fujii Masao wrote: On 2024/10/02 9:27, Masahiko Sawada wrote: Sorry for being late in joining the review of this patch. Both 0001 and 0003 look good to me. I have two comments

Re: Add new COPY option REJECT_LIMIT

2024-09-30 Thread torikoshia
On 2024-09-28 10:48, Jian he wrote: Thanks for your comments! +/* + * Extract REJECT_LIMIT value from a DefElem. + */ +static int64 +defGetCopyRejectLimitOptions(DefElem *def) +{ + int64 reject_limit; + + if (def->arg == NULL) + ereport(ERROR, + (errcode(ERRCODE_SYNTAX_ERROR), + errmsg("REJECT_

Re: Add new COPY option REJECT_LIMIT

2024-09-26 Thread torikoshia
On 2024-09-25 02:22, Fujii Masao wrote: Thanks for your review! Attached v5 patch. On 2024/09/24 14:25, torikoshia wrote: Updated the patch. Thanks for updating the patch! +REJECT_LIMIT { class="parameter">integer } The curly braces {} seem unnecessary here. + Wh

Re: Add on_error and log_verbosity options to file_fdw

2024-09-26 Thread torikoshia
On 2024-09-25 00:46, Fujii Masao wrote: Thanks for the comments! On 2024/09/24 20:08, torikoshia wrote: Thanks for the explanation and suggestion. Since there is almost the same code in copyfrom.c, attached 0003 patch for refactoring both. Thanks for updating the patches! Regarding 0002

Re: Add on_error and log_verbosity options to file_fdw

2024-09-24 Thread torikoshia
On 2024-09-20 11:27, Fujii Masao wrote: Thanks for your comments! On 2024/09/19 23:16, torikoshia wrote: -   COPY_LOG_VERBOSITY_DEFAULT = 0, /* logs no additional messages, default */ -   COPY_LOG_VERBOSITY_VERBOSE, /* logs additional messages */ +   COPY_LOG_VERBOSITY_SILENT

Re: Add new COPY option REJECT_LIMIT

2024-09-23 Thread torikoshia
Updated the patch. On 2024-07-23 23:06, torikoshia wrote: On Tue, Jul 23, 2024 at 1:35 PM Fujii Masao wrote: Thanks for your review. On 2024/07/22 21:37, torikoshia wrote: On Fri, Jul 19, 2024 at 11:48 PM Junwang Zhao wrote: Thanks for the comment. In patch 0002, the ratio is

Re: Add on_error and log_verbosity options to file_fdw

2024-09-19 Thread torikoshia
On 2024-09-11 23:50, Fujii Masao wrote: Thanks for your comments! Here are the review comments for 0001 patch. + silent excludes verbose messages. This should clarify that in silent mode, not only verbose messages but also default ones are suppressed? Agreed. + cstate-

Re: Using per-transaction memory contexts for storing decoded tuples

2024-09-12 Thread torikoshia
On 2024-09-12 07:32, Masahiko Sawada wrote: Thanks a lot for working on this! Hi all, We have several reports that logical decoding uses memory much more than logical_decoding_work_mem[1][2][3]. For instance in one of the reports[1], even though users set logical_decoding_work_mem to '256MB',

Re: ANALYZE ONLY

2024-09-12 Thread torikoshia
On 2024-09-11 16:47, Michael Harris wrote: Thanks for the feedback. On Tue, 10 Sept 2024 at 22:03, torikoshia wrote: Regarding the addition of partition descendant tables, should we also update the below comment on expand_vacuum_rel? Currently it refers only partitions: | * Given a

Re: ANALYZE ONLY

2024-09-10 Thread torikoshia
On 2024-09-09 16:56, Michael Harris wrote: Thanks for updating the patch. Here is a minor comment. @@ -944,20 +948,32 @@ expand_vacuum_rel(VacuumRelation *vrel, MemoryContext vac_context, MemoryContextSwitchTo(oldcontext); } .. +* Unless the user has specified ONLY,

Re: ANALYZE ONLY

2024-09-01 Thread torikoshia
t;include_children" abode). I think running pgindent would resolve such issue in most places. I fixed that extra space, and then ran pgindent. It did not report any more problems. On Fri, 30 Aug 2024 at 16:45, torikoshia wrote: -- https://www.postgresql.org/docs/devel/progress-reporting.

Re: ANALYZE ONLY

2024-08-29 Thread torikoshia
Hi Michael, On 2024-08-23 19:01, Michael Harris wrote: V2 of the patch is attached. Thanks for the proposal and the patch. You said this patch was a first draft version, so these may be too minor comments, but I will share them: -- https://www.postgresql.org/docs/devel/progress-reporting

Re: Add on_error and log_verbosity options to file_fdw

2024-08-08 Thread torikoshia
On 2024-07-24 19:43, torikoshia wrote: On 2024-07-23 08:57, Michael Paquier wrote: On Mon, Jul 22, 2024 at 03:07:46PM -0700, Masahiko Sawada wrote: I'm slightly concerned that users might not want to see the NOTICE message for every scan. Unlike COPY FROM, scanning a file via file_fdw

Re: Add on_error and log_verbosity options to file_fdw

2024-07-24 Thread torikoshia
On 2024-07-23 08:57, Michael Paquier wrote: On Mon, Jul 22, 2024 at 03:07:46PM -0700, Masahiko Sawada wrote: I'm slightly concerned that users might not want to see the NOTICE message for every scan. Unlike COPY FROM, scanning a file via file_fdw could be frequent. Agreed. Yeah, I also have

Re: Add new COPY option REJECT_LIMIT

2024-07-23 Thread torikoshia
On 2024-07-23 02:06, Kirill Reshke wrote: Thanks for your review. Few comments: + When a positive integer value is specified, COPY limits + the maximum tolerable number of errors while converting a column's input + value into its data type. If nothing is specified, then the

Re: Add new COPY option REJECT_LIMIT

2024-07-23 Thread torikoshia
On Tue, Jul 23, 2024 at 1:35 PM Fujii Masao wrote: Thanks for your review. On 2024/07/22 21:37, torikoshia wrote: On Fri, Jul 19, 2024 at 11:48 PM Junwang Zhao wrote: Thanks for the comment. In patch 0002, the ratio is calculated by the already skipped/processed rows, but what if a user

Re: Add new COPY option REJECT_LIMIT

2024-07-22 Thread torikoshia
mn i: "a" NOTICE: skipping row due to data type incompatibility at line 18 for column i: "a" NOTICE: skipping row due to data type incompatibility at line 19 for column i: "a" NOTICE: 10 rows were skipped due to data type incompatibility COPY 990 On 2024-07-2

Re: Add on_error and log_verbosity options to file_fdw

2024-07-18 Thread torikoshia
On 2024-07-05 00:27, torikoshia wrote: Hi, With the current file_fdw, if even one line of data conversion fails, the contents of the file cannot be referenced at all: =# \! cat data/test.data 1,a 2,b a,c =# create foreign table f_fdw_test_1 (i int, t text) server f_fdw options

Re: Add new COPY option REJECT_LIMIT

2024-07-17 Thread torikoshia
d this time is to specify an upper limit on the number or ratio of errors, the name of this option like "reject_limit" seems better than "ignore_errors". On Fri, Jul 5, 2024 at 4:13 PM torikoshia wrote: On 2024-07-05 12:59, Fujii Masao wrote: On 2024/07/04 12:05, t

Re: Add new COPY option REJECT_LIMIT

2024-07-05 Thread torikoshia
On 2024-07-05 12:59, Fujii Masao wrote: On 2024/07/04 12:05, torikoshia wrote: I'm going to update it after discussing the option format as described below. Thanks! I agree that it's possible to use only REJECT_LIMIT without ON_ERROR. I also think it's easy to understand that

Add on_error and log_verbosity options to file_fdw

2024-07-04 Thread torikoshia
Hi, With the current file_fdw, if even one line of data conversion fails, the contents of the file cannot be referenced at all: =# \! cat data/test.data 1,a 2,b a,c =# create foreign table f_fdw_test_1 (i int, t text) server f_fdw options (filename 'test.data', format 'csv'); CRE

Re: Add new COPY option REJECT_LIMIT

2024-07-03 Thread torikoshia
On 2024-07-03 02:07, Fujii Masao wrote: Thanks for your comments! On 2024/01/26 18:49, torikoshia wrote: Hi, 9e2d870 enabled the COPY command to skip soft error, and I think we can add another option which specifies the maximum tolerable number of soft errors. I remember this was

Re: First draft of PG 17 release notes

2024-05-22 Thread torikoshia
On 2024-05-09 13:03, Bruce Momjian wrote: I have committed the first draft of the PG 17 release notes; you can see the results here: https://momjian.us/pgsql_docs/release-17.html It will be improved until the final release. The item count is 188, which is similar to recent releases:

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2024-04-17 Thread torikoshia
On 2024-04-16 13:16, Masahiko Sawada wrote: On Tue, Apr 2, 2024 at 7:34 PM torikoshia wrote: On 2024-04-01 11:31, Masahiko Sawada wrote: > On Fri, Mar 29, 2024 at 11:54 AM torikoshia > wrote: >> >> On 2024-03-28 21:54, Masahiko Sawada wrote: >> > On Thu, Mar 28,

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2024-04-02 Thread torikoshia
On 2024-04-01 11:31, Masahiko Sawada wrote: On Fri, Mar 29, 2024 at 11:54 AM torikoshia wrote: On 2024-03-28 21:54, Masahiko Sawada wrote: > On Thu, Mar 28, 2024 at 9:38 PM torikoshia > wrote: >> >> On 2024-03-28 10:20, Masahiko Sawada wrote: >> > Hi, >>

Re: Add new error_action COPY ON_ERROR "log"

2024-03-29 Thread torikoshia
On 2024-03-28 21:36, torikoshia wrote: On 2024-03-28 17:27, Bharath Rupireddy wrote: On Thu, Mar 28, 2024 at 1:43 PM torikoshia wrote: Attached patch fixes the doc, May I know which patch you are referring to? And, what do you mean by "fixes the doc"? Ugh, I replied to the wr

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2024-03-28 Thread torikoshia
On 2024-03-28 21:54, Masahiko Sawada wrote: On Thu, Mar 28, 2024 at 9:38 PM torikoshia wrote: On 2024-03-28 10:20, Masahiko Sawada wrote: > Hi, > > On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada > wrote: >> >> On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov >

Re: POC PATCH: copy from ... exceptions to: (was Re: VLDB Features)

2024-03-28 Thread torikoshia
On 2024-03-28 10:20, Masahiko Sawada wrote: Hi, On Thu, Jan 18, 2024 at 5:33 PM Masahiko Sawada wrote: On Thu, Jan 18, 2024 at 4:59 PM Alexander Korotkov wrote: > > On Thu, Jan 18, 2024 at 4:16 AM torikoshia wrote: > > On 2024-01-18 10:10, jian he wrote: > > > On T

Re: Add new error_action COPY ON_ERROR "log"

2024-03-28 Thread torikoshia
On 2024-03-28 17:27, Bharath Rupireddy wrote: On Thu, Mar 28, 2024 at 1:43 PM torikoshia wrote: Attached patch fixes the doc, May I know which patch you are referring to? And, what do you mean by "fixes the doc"? Ugh, I replied to the wrong thread. Sorry for making you confused

Re: Add new error_action COPY ON_ERROR "log"

2024-03-28 Thread torikoshia
On 2024-03-26 17:16, Masahiko Sawada wrote: On Tue, Mar 26, 2024 at 3:04 PM Bharath Rupireddy wrote: On Tue, Mar 26, 2024 at 9:56 AM Masahiko Sawada wrote: > > > > errmsg("data type incompatibility at line %llu for column %s: \"%s\"", > > > > I guess it would be better to make the log messa

Re: RFC: Logging plan of the running query

2024-03-17 Thread torikoshia
On 2024-03-14 04:33, Robert Haas wrote: Thanks for your review! On Wed, Mar 13, 2024 at 1:28 AM torikoshia wrote: - I saw no way to find the next node to be executed from the planstate tree, so the patch wraps all the ExecProcNode of the planstate tree at CHECK_FOR_INTERRUPTS(). I don&#

Re: RFC: Logging plan of the running query

2024-03-12 Thread torikoshia
On Fri, Feb 16, 2024 at 11:42 PM torikoshia wrote: I'm not so sure about the implementation now, i.e. finding the next node to be executed from the planstate tree, but I'm going to try this approach. Attached a patch which takes this approach. - I saw no way to find the next

Re: Add new error_action COPY ON_ERROR "log"

2024-03-06 Thread torikoshia
On 2024-03-07 13:00, Michael Paquier wrote: On Wed, Mar 06, 2024 at 07:32:28PM +0530, Bharath Rupireddy wrote: Please see the attached v4 patch. If it looks good, I can pull LOG_VERBOSITY changes out into 0001 and with 0002 containing the detailed messages for discarded rows. The approach look

Re: Add new error_action COPY ON_ERROR "log"

2024-02-26 Thread torikoshia
On 2024-02-20 17:22, torikoshia wrote: On 2024-02-17 15:00, Bharath Rupireddy wrote: On Fri, Feb 16, 2024 at 8:17 PM torikoshia wrote: I may be wrong since I seldom do data loading tasks, but I greed with you. I also a little concerned about the case where there are many malformed data

Re: RFC: Logging plan of the running query

2024-02-26 Thread torikoshia
On 2024-02-24 00:23, Robert Haas wrote: On Fri, Feb 23, 2024 at 7:50 PM Julien Rouhaud wrote: On Fri, Feb 23, 2024 at 10:22:32AM +0530, Robert Haas wrote: > On Thu, Feb 22, 2024 at 6:25 AM James Coleman wrote: > > This is potentially a bit of a wild idea, but I wonder if having some > > kind o

Re: Add new error_action COPY ON_ERROR "log"

2024-02-20 Thread torikoshia
On 2024-02-17 15:00, Bharath Rupireddy wrote: On Fri, Feb 16, 2024 at 8:17 PM torikoshia wrote: I may be wrong since I seldom do data loading tasks, but I greed with you. I also a little concerned about the case where there are many malformed data and it causes lots of messages, but the

Re: Add new error_action COPY ON_ERROR "log"

2024-02-16 Thread torikoshia
On 2024-02-16 17:15, Bharath Rupireddy wrote: On Wed, Feb 14, 2024 at 5:04 PM torikoshia wrote: [] let the users know what line numbers are > containing the errors that COPY ignored something like [1] with a > simple change like [2]. Agreed. Unlike my patch, it hides the

Re: RFC: Logging plan of the running query

2024-02-16 Thread torikoshia
On Thu, Feb 15, 2024 at 6:12 PM Robert Haas wrote: Hi, I've just been catching up on this thread. + if (MyProc->heldLocks) + { + ereport(LOG_SERVER_ONLY, + errmsg("ignored request for logging query plan due to lock conflicts"), + errdetail("You can try again in a moment.")); + return; + } I

  1   2   3   4   >