RE: Improve error handling for invalid slots and ensure a same 'inactive_since' time for inactive slots

2025-01-29 Thread Hayato Kuroda (Fujitsu)
Dear Nisha, Thanks for creating a patch! > Removed patch v1-0001. Please find the attached version 2 of 0002, > which is now v2-0001. ISMT error_if_invalid is set to true when the slot is using and set to false when dropping. One exception is the slot_sync, but it has already had mechanism to h

RE: Adding a '--clean-publisher-objects' option to 'pg_createsubscriber' utility.

2025-01-28 Thread Hayato Kuroda (Fujitsu)
Dear Shubham, > I propose adding the --clean-publisher-objects option to the > pg_createsubscriber utility. As discussed in [1], this feature ensures > a clean and streamlined setup of logical replication by removing stale > or unnecessary publications from the subscriber node. These > publication

RE: POC: enable logical decoding when wal_level = 'replica' without a server restart

2025-01-28 Thread Hayato Kuroda (Fujitsu)
Dear Sawada-san, I love the idea. I've roughly tested the patch and worked on my env. Here are initial comments... 1. xloglevelworker.c ``` +#include "replication/logicalxlog.h" ``` xloglevelworker.c includes replication/logicalxlog.h, but it does not exist. The line had to be removed to build a

RE: pgbench without dbname worked differently with psql and pg_dump

2025-01-21 Thread Hayato Kuroda (Fujitsu)
Dear David, > As none of those programs "create" a database the wording "The default is to > create a database" seems quite a poor choice to standardize on. Oh, I misunderstood. Sorry. I wanted to suggest like below: ``` The default is to connect to a database with the same name as the current s

RE: pgbench without dbname worked differently with psql and pg_dump

2025-01-21 Thread Hayato Kuroda (Fujitsu)
Dear Ashutosh, > I see the same behaviour. The code in pgbench which decided the > database to connect to, looks similar to the relevant code in > createdb. But createdb documentation is in agreement with the code. > [1] > ``` > dbname > > Specifies the name of the database to be created. The nam

pgbench without dbname worked differently with psql and pg_dump

2025-01-20 Thread Hayato Kuroda (Fujitsu)
ABASE is used. If that is not set, the user name specified for the connection is used. ``` Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Conflict detection for update_deleted in logical replication

2025-01-19 Thread Hayato Kuroda (Fujitsu)
ound that step 6 (Wait until all the changes on the publisher is replicated to the subscriber) was necessary. If it was skipped, the slot would be invalidated soon. This is because the remained changes are not replicated to the subscriber side yet and the catchup is delayed due to them. Best regards, Hayato Kuroda FUJITSU LIMITED <>

RE: Replace current implementations in crypt() and gen_salt() to OpenSSL

2025-01-14 Thread Hayato Kuroda (Fujitsu)
ssage-id/1f32ff67-255d-4c0c-8433-c8c721842aa3%40eisentraut.org Best regards, Hayato Kuroda FUJITSU LIMITED

RE: ecpg command does not warn COPY ... FROM STDIN;

2025-01-08 Thread Hayato Kuroda (Fujitsu)
obably don't need to bother with adjusting the check > to allow that. To confirm - I have no objections for the decision. I'm also think it is not an usual grammar. I checked the doc [1] and I could not find "COPY FROM STDOUT". I.e., first version is enough. [1]: https://www.postgresql.org/docs/devel/sql-copy.html Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Log a warning in pg_createsubscriber for max_slot_wal_keep_size

2025-01-08 Thread Hayato Kuroda (Fujitsu)
keep_size = 1'); +$node_p->reload; ``` Can you use 'max_slot_wal_keep_size = 10MB' or something to test the handling is correct? Best regards, Hayato Kuroda FUJITSU LIMITED

RE: ecpg command does not warn COPY ... FROM STDIN;

2025-01-08 Thread Hayato Kuroda (Fujitsu)
ve a comment about the fix. The parser accepts a statement like "COPY ... FROM STDOUT", and ISTM it is not either implemented yet. However, the warning message only mentions STDIN case even STDOUT is specified. EXEC SQL COPY foo FROM STDOUT; -> WARNING: COPY FROM STDIN is not implemente

RE: Conflict detection for update_deleted in logical replication

2025-01-07 Thread Hayato Kuroda (Fujitsu)
ete_origin_differs as well for the same reason? Right. I've also tested with almost the same way as above, and I got the same result. You can confirm by modifying the last statement in the attached script. Best regards, Hayato Kuroda FUJITSU LIMITED test_origin_differ.sh Description: test_origin_differ.sh

RE: initdb -c "track_commit_timestamp=on" crashes in case of debug build

2025-01-06 Thread Hayato Kuroda (Fujitsu)
ccording to the post [1], there are no similar failures for boolean GUCs. From this perspective, we do not have to do some fundamental ways to fix this bug, i.e., my patch is enough. [1]: https://www.postgresql.org/message-id/0307b599-9528-643d-934c-a7e76d376e94%40enterprisedb.com Best regard

initdb -c "track_commit_timestamp=on" crashes in case of debug build

2025-01-06 Thread Hayato Kuroda (Fujitsu)
in main (argc=8, argv=0x1be1040) at main.c:21 ``` Best regards, Hayato Kuroda FUJITSU LIMITED 0001-Avoid-activating-commit_ts-while-bootstrap.patch Description: 0001-Avoid-activating-commit_ts-while-bootstrap.patch

RE: Log a warning in pg_createsubscriber for max_slot_wal_keep_size

2025-01-05 Thread Hayato Kuroda (Fujitsu)
"10", but this is not clean. I suggest either of 1) accepting the value as the string, or 2) using an SQL function pg_size_bytes() to get max_slot_wal_keep_size. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Logical Replication of sequences

2024-12-27 Thread Hayato Kuroda (Fujitsu)
uence_state' }, ``` I think this is not wrong, but according to the src/include/catalog/unused_oids, the oid should be at 8000- while developing: ``` $ ./unused_oids ... Patches should use a more-or-less consecutive range of OIDs. Best practice is to start with a random choice in the range 8000-. Suggested random unused OID: 9293 (415 consecutive OID(s) available starting here) ``` Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Logical replication timeout

2024-12-25 Thread Hayato Kuroda (Fujitsu)
resql.org/message-id/OSCPR01MB14966B646506E0C9B81B3A4CFF5022%40OSCPR01MB14966.jpnprd01.prod.outlook.com Best regards, Hayato Kuroda FUJITSU LIMITED 0001-WIP-track-wal-segments.patch Description: 0001-WIP-track-wal-segments.patch

RE: Object identifier types in logical replication binary mode

2024-12-24 Thread Hayato Kuroda (Fujitsu)
iption.html#SQL-CREATESUBSCRIPTION-PARAMS-WITH-BINARY Best regards, Hayato Kuroda FUJITSU LIMITED test_1225.sh Description: test_1225.sh

RE: Parallel heap vacuum

2024-12-23 Thread Hayato Kuroda (Fujitsu)
num_index_scans; + ``` I feel this is bit confusing because LVRelState also has "num_index_scans". How about "num_parallel_index_scans"? Attached patch contains above changes. Best regards, Hayato Kuroda FUJITSU LIMITED kuroda.diffs Description: kuroda.diffs

RE: Logical replication timeout

2024-12-23 Thread Hayato Kuroda (Fujitsu)
ment is accepted, I feel you can use foreach_delete_current(). === Also, even when we optimize the truncation of files, there is a possibility that replication is timed out. Can you also create a patch which implements [1]? [1]: https://www.postgresql.org/message-id/CAExHW5s2_T9mULDQRKsdV72wpnA%2BNLT63cX51b51QQVEV4sG5g%40mail.gmail.com Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Logical replication timeout

2024-12-23 Thread Hayato Kuroda (Fujitsu)
not solve it. Oh, I understood that you've already increased the parameter to the appropriate value on your env. Is it right? Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Logical replication timeout

2024-12-23 Thread Hayato Kuroda (Fujitsu)
ttings can optimize your system. [1]: https://www.postgresql.org/docs/14/sql-createsubscription.html [2]: https://www.postgresql.org/docs/14/runtime-config-resource.html#GUC-LOGICAL-DECODING-WORK-MEM Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Conflict detection for update_deleted in logical replication

2024-12-22 Thread Hayato Kuroda (Fujitsu)
t; section in logical-replication.sgml because an additional slot is required? 05. check_remote_recovery() Can we add a test case related with this? Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Adding a '--two-phase' option to 'pg_createsubscriber' utility.

2024-12-11 Thread Hayato Kuroda (Fujitsu)
T count(1) = 0 FROM pg_subscription WHERE subtwophasestate NOT IN ('e'); Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Conflict detection for update_deleted in logical replication

2024-12-11 Thread Hayato Kuroda (Fujitsu)
59443 5 59541 median 59536 ## Apendix [5] - shorter wal_receiver_status_interval pached # run interval=10ms 1 58081 2 57876 3 58083 4 57915 5 57933 median 57933 head # run interval=10ms 1 57595 2 57322 3 57271 4 57421 5

RE: Parallel heap vacuum

2024-12-10 Thread Hayato Kuroda (Fujitsu)
I think it would be good to have some sort of README explaining how > the parallel heap vacuum works, i.e. how it's driven by FSM. Took me a > while to realize how the workers coordinate which blocks to scan. I love the idea, it is quite helpful for reviewers like me. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Adding a '--two-phase' option to 'pg_createsubscriber' utility.

2024-12-09 Thread Hayato Kuroda (Fujitsu)
the argument. I think it is enough to provide "enable-twophase" and "T" option which specify the two_phase to true. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Introduce XID age and inactive timeout based replication slot invalidation

2024-12-02 Thread Hayato Kuroda (Fujitsu)
usleep() is not used in this file. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Rework subscription-related code for psql and pg_dump

2024-12-02 Thread Hayato Kuroda (Fujitsu)
Dear Michael, > I've looked at that again and let this as-is, then applied all this > cleanup. I missed your e-mail but it's ok to keep the output style for two_phase. Thanks a lot for working on it! Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Conflict detection for update_deleted in logical replication

2024-11-29 Thread Hayato Kuroda (Fujitsu)
, next xid and their epoch. Or, it is more important to reduce the amount of calc. on publisher side? Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Rework subscription-related code for psql and pg_dump

2024-11-28 Thread Hayato Kuroda (Fujitsu)
Kuroda FUJITSU LIMITED

RE: Introduce XID age and inactive timeout based replication slot invalidation

2024-11-28 Thread Hayato Kuroda (Fujitsu)
eter is handled as the small value. - solution I think there are two possible solutions. You can choose one of them: a. Make the maximum INT_MAX/1000, or b. Change the unit to millisecond. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Parallel heap vacuum

2024-11-25 Thread Hayato Kuroda (Fujitsu)
od to me. I imagine you will add new APIs for tableam like parallel_vacuum_compute_workers_for_scaning and parallel_vacuum_compute_workers_for_vacuuming. If other tableam developers want to use the same number of workers as scanning, they can pass the same function to the pointer. Is it right? Bes

RE: Improve the error message for logical replication of regular column to generated column.

2024-11-14 Thread Hayato Kuroda (Fujitsu)
issing_attrs, some of changes are not needed anymore. You can just add comment in logicalrep_rel_open and modify the message in logicalrep_report_missing_attrs. [1]: https://www.postgresql.org/message-id/CAHut%2BPumbPEqk6v2XVjT7vKWKzQNBjMHXByWJ5%3DFmjEfk1v_pQ%40mail.gmail.com Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Fix for pageinspect bug in PG 17

2024-11-13 Thread Hayato Kuroda (Fujitsu)
der pageinspect extension. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Parallel heap vacuum

2024-11-13 Thread Hayato Kuroda (Fujitsu)
ble_block_parallelscan_estimate() and mul_size() from here? 05. Idea Can you update documentations? 06. Idea AFAICS pg_stat_progress_vacuum does not contain information related with the parallel execution. How do you think adding an attribute which shows a list of pids? Not sure it is helpful for users but it can show the parallelism. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Commit Timestamp and LSN Inversion issue

2024-11-12 Thread Hayato Kuroda (Fujitsu)
n(). Just in case I want to clarify what I used. My head was bfeeb06 and I applied [1] atop it. [1]: https://www.postgresql.org/message-id/OS0PR01MB57162A227EC357482FEB4470945D2%40OS0PR01MB5716.jpnprd01.prod.outlook.com Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Commit Timestamp and LSN Inversion issue

2024-11-12 Thread Hayato Kuroda (Fujitsu)
0094a56f in pg_logical_emit_message_text (fcinfo=0x23a65d0) at logicalfuncs.c:385 ``` 6. confirmed xlcommitrec was NULL ``` (gdb) p xlcommitrec $1 = (xl_xact_commit *) 0x0 ``` Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Fix for pageinspect bug in PG 17

2024-11-12 Thread Hayato Kuroda (Fujitsu)
>setDesc->natts`. Can you use the boolean and add comments, something like ``` +/* Support older API version */ +booloutput_empty_attr = (rsinfo->setDesc->natts >= 8); ``` 3. Do we have to add the test for the fix? [1]: https://www.postgresql.org/message-id/VI1PR02MB63331C3D90E2104FD12399D38A5D2%40VI1PR02MB6333.eurprd02.prod.outlook.com Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Parallel heap vacuum

2024-11-11 Thread Hayato Kuroda (Fujitsu)
another reason that lazy_vacuum_heap_rel() uses TidStoreBeginIterateShared()? Another approach is to restrict TidStoreBeginIterate() to support only the local one. How do you think? Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Commit Timestamp and LSN Inversion issue

2024-11-11 Thread Hayato Kuroda (Fujitsu)
the default were: ``` autovacuum = false checkpoint_timeout = 1h shared_buffers = '30GB' max_wal_size = 20GB min_wal_size = 10GB ``` Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Pgoutput not capturing the generated columns

2024-11-01 Thread Hayato Kuroda (Fujitsu)
n the sub but it was reported as missing column. I think we should somehow report like "generated column on publisher is replicated the generated column on the subscriber". Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Conflict detection for update_deleted in logical replication

2024-10-30 Thread Hayato Kuroda (Fujitsu)
ead tuples. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Conflict detection for update_deleted in logical replication

2024-10-30 Thread Hayato Kuroda (Fujitsu)
t it seemed that entries can be removed when it behinds GetOldestNonRemovableTransactionId(NULL), i.e., horizons.shared_oldest_nonremovable. The value is affected by the replication slots so that interesting commit_ts entries for us are not removed. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Conflict detection for update_deleted in logical replication

2024-10-29 Thread Hayato Kuroda (Fujitsu)
is topic introduces huge complexity and is not mandatory for update_deleted detection. We can work on it in later versions based on the needs. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Conflict detection for update_deleted in logical replication

2024-10-29 Thread Hayato Kuroda (Fujitsu)
s much as possible. This feature is related with the replication so that changes should be closed within the replication subdir. [1]: https://www.postgresql.org/message-id/TYAPR01MB5692541820BCC365C69442FFF54F2%40TYAPR01MB5692.jpnprd01.prod.outlook.com Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Pgoutput not capturing the generated columns

2024-10-28 Thread Hayato Kuroda (Fujitsu)
I'm not sure it is correct. Why do you skip the generated column even when it is in the column list? Also, can you add comments what you want to do? Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Pgoutput not capturing the generated columns

2024-10-27 Thread Hayato Kuroda (Fujitsu)
t. ``` Based on the comment, this case does not test the behavior of generated columns anymore. So, I felt column 'd' could be removed from the case. 03. 031_column_list.pl Can we test that generated columns won't be replaced if it does not included in the column list? [1]:

RE: Pgoutput not capturing the generated columns

2024-10-27 Thread Hayato Kuroda (Fujitsu)
)" ``` I think no need to separate lines and add bracket. How about like below? ``` "SELECT DISTINCT gpt.attrs" ``` Best regards, Hayato Kuroda FUJITSU LIMITED diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 1

RE: Conflict detection for update_deleted in logical replication

2024-10-24 Thread Hayato Kuroda (Fujitsu)
gresql.org/message-id/OS0PR01MB5716A78EE3D6F2F4754E1209946C2%40OS0PR01MB5716.jpnprd01.prod.outlook.com Best regards, Hayato Kuroda FUJITSU LIMITED test_1025.sh Description: test_1025.sh

RE: Make default subscription streaming option as Parallel

2024-10-22 Thread Hayato Kuroda (Fujitsu)
ns with streaming=off, it was chosen because it was a default. I cannot find strong reasons to keep current behavior. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: incorrect wal removal due to max_slot_wal_keep_size

2024-10-10 Thread Hayato Kuroda (Fujitsu)
dXlogFiles() and RemoveXlogFile(). I want to see the log when you can reproduce. They are inspired by [1]. I doubt the thread and yours are the same issue or not. [1]: https://www.postgresql.org/message-id/flat/Yz2hivgyjS1RfMKs%40depesz.com Best regards, Hayato Kuroda FUJITSU LIMITED

RE: long-standing data loss bug in initial sync of logical replication

2024-10-07 Thread Hayato Kuroda (Fujitsu)
o pgoutput layer as "relation". I.e., I think it is enough to invalidate relcache of leaf. > I have also modified the tests in 0001 patch. These changes are only > related to syntax of writing tests. LGTM. I found small improvements, please find the attached. Best regard

RE: Make default subscription streaming option as Parallel

2024-10-06 Thread Hayato Kuroda (Fujitsu)
ould be avoided in any case, and everyone knows it, I agree with your point. One comment for your patch; Shouldn't we add a streaming=off case for pg_dump test? You added lines but no one passes it. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: long-standing data loss bug in initial sync of logical replication

2024-10-03 Thread Hayato Kuroda (Fujitsu)
think? Best regards, Hayato Kuroda FUJITSU LIMITED add_invalidations.diffs Description: add_invalidations.diffs

RE: Using per-transaction memory contexts for storing decoded tuples

2024-10-02 Thread Hayato Kuroda (Fujitsu)
ust to confirm - you do not push rb_mem_block_size patch and just replace SLAB_LARGE_BLOCK_SIZE -> SLAB_DEFAULT_BLOCK_SIZE, right? It seems that only reorderbuffer.c uses the LARGE macro so that it can be removed. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Conflict detection for update_deleted in logical replication

2024-10-01 Thread Hayato Kuroda (Fujitsu)
//www.postgresql.org/message-id/OS0PR01MB5716E0A283D1B66954CDF5A694682%40OS0PR01MB5716.jpnprd01.prod.outlook.com Best regards, Hayato Kuroda FUJITSU LIMITED

RE: long-standing data loss bug in initial sync of logical replication

2024-09-30 Thread Hayato Kuroda (Fujitsu)
ww.postgresql.org/docs/devel/logical-replication-security.html Best regards, Hayato Kuroda FUJITSU LIMITED

RE: long-standing data loss bug in initial sync of logical replication

2024-09-30 Thread Hayato Kuroda (Fujitsu)
n and must invalidate relcaches there. 2. Similarly with above, the relcache won't be invalidated when ALTER PUBLICATION OWNER TO is executed. This means that privilage checks may be ignored if the entry is still valid. Not sure, but is there a possibility this causes an inconsistency? Best rega

RE: long-standing data loss bug in initial sync of logical replication

2024-09-26 Thread Hayato Kuroda (Fujitsu)
as shared in [1] and observed a significant > decrease in the degradation with the new changes. With selective > invalidation degradation is around ~5%. This results are an average of > 3 runs. IIUC, the executed workload did not contain ALTER SCHEMA command, so third improvement did not contribute this improvement. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Clock-skew management in logical replication

2024-09-25 Thread Hayato Kuroda (Fujitsu)
ration-checklist-for-oracle-grid-infrastructure.html Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Using per-transaction memory contexts for storing decoded tuples

2024-09-20 Thread Hayato Kuroda (Fujitsu)
total: 721420288 bytes in 86 blocks; 1415344 free (0 chunks); 720004944 use Best regards, Hayato Kuroda FUJITSU LIMITED test.sh Description: test.sh

RE: [Proposal] Add foreign-server health checks infrastructure

2024-09-19 Thread Hayato Kuroda (Fujitsu)
Thanks everyone for many efforts! [1]: https://github.com/postgres/postgres/commit/4f08ab55457751308ffd8d33e82155758cd0e304 Best regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2024-09-16 Thread Hayato Kuroda (Fujitsu)
s NULL, invalid > > LGTM. > Also I added the following to the example for clarity: > > postgres=# SELECT * FROM postgres_fdw_get_connections(true); +1. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Using per-transaction memory contexts for storing decoded tuples

2024-09-16 Thread Hayato Kuroda (Fujitsu)
more > than 4GB memory. I appreciate your work on logical replication and am interested in the thread. I've heard this issue from others, and this has been the barrier to using logical replication. Please let me know if I can help with benchmarking, other measurements, etc. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Conflict detection and logging in logical replication

2024-08-20 Thread Hayato Kuroda (Fujitsu)
name is quoted. 0002: 03. ``` -#include "replication/logicalrelation.h" ``` Just to confirm - this removal is not related with the feature but just the improvement, right? Best regards, Hayato Kuroda FUJITSU LIMITED

RE: [bug fix] prepared transaction might be lost when max_prepared_transactions is zero on the subscriber

2024-08-09 Thread Hayato Kuroda (Fujitsu)
EANUP()/PG_END_ENSURE_ERROR_CLEANUP macros here. According to codes, it assumes that any before-shmem callbacks are not registered within the block because the cleanup function is registered and canceled within the macro. LogicalRepApplyLoop() can register the function when it handles COMMIT PREPAR

RE: Found issues related with logical replication and 2PC

2024-08-08 Thread Hayato Kuroda (Fujitsu)
`` @@ -3297,6 +3297,8 @@ apply_dispatch(StringInfo s) saved_command = apply_error_callback_arg.command; apply_error_callback_arg.command = action; +elog(LOG, "XXX got message %d", action); ``` Best regards, Hayato Kuroda FUJITSU LIMITED test_0809.sh Description: test_0809.sh

RE: Found issues related with logical replication and 2PC

2024-08-08 Thread Hayato Kuroda (Fujitsu)
same result as when the stream option is not "parallel." - XactLastCommitEnd was replaced even ROLLBACK PREPARED case. Since the COMMIT PREPARED record is flushed in RecordTransactionAbortPrepared(), there is no need to ensure the WAL must be sent. Best regards, Hayato Kuroda FUJITS

RE: Found issues related with logical replication and 2PC

2024-08-07 Thread Hayato Kuroda (Fujitsu)
nd; > Opps. Fixed version is posted in [1]. [1]: https://www.postgresql.org/message-id/TYAPR01MB5692FAC23BE40C69DA8ED4AFF5B92%40TYAPR01MB5692.jpnprd01.prod.outlook.com Best regards, Hayato Kuroda FUJITSU LIMITED

[bug fix] prepared transaction might be lost when max_prepared_transactions is zero on the subscriber

2024-08-07 Thread Hayato Kuroda (Fujitsu)
ql.org/message-id/CAA4eK1L-r8OKGdBwC6AeXSibrjr9xKsg8LjGpX_PDR5Go-A9TA%40mail.gmail.com Best regards, Hayato Kuroda FUJITSU LIMITED v2-0001-Prevent-origin-progress-advancement-if-failed-to-.patch Description: v2-0001-Prevent-origin-progress-advancement-if-failed-to-.patch test_2pc.sh Description: test_2pc.sh

RE: [Proposal] Add foreign-server health checks infrastructure

2024-08-07 Thread Hayato Kuroda (Fujitsu)
eign table with the given OID. A > - ForeignTable object contains properties of > the > - foreign table (see foreign/foreign.h for details). > - > - > - > - > > Why did you remove these code? Just mistake? Oh, my fault. I tried to remove GetUserMappingByOid() and the entry was also Removed at that time. Restored. Best regards, Hayato Kuroda FUJITSU LIMITED v3-0001-Extend-postgres_fdw_get_connections-to-return-use.patch Description: v3-0001-Extend-postgres_fdw_get_connections-to-return-use.patch

RE: Conflict detection and logging in logical replication

2024-08-07 Thread Hayato Kuroda (Fujitsu)
uce a global variable which is turned on when the conflict is found. Thought? Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Conflict detection and logging in logical replication

2024-08-06 Thread Hayato Kuroda (Fujitsu)
ix patch. Feel free to include in the next version. Best regards, Hayato Kuroda FUJITSU LIMITED fixes_for_v11.patch Description: fixes_for_v11.patch

RE: [Proposal] Add foreign-server health checks infrastructure

2024-08-01 Thread Hayato Kuroda (Fujitsu)
hat we can keep the code consistent with the server case. The part does not read data from the entry, and we can follow. Best regards, Hayato Kuroda FUJITSU LIMITED v2-0001-Extend-postgres_fdw_get_connections-to-return-use.patch Description: v2-0001-Extend-postgres_fdw_get_connections-to-return-use.patch

RE: Conflict detection and logging in logical replication

2024-08-01 Thread Hayato Kuroda (Fujitsu)
ublic.tab" in transaction 740, finished at 0/1543940 ``` = Can we support the type of conflict? [1]: https://www.postgresql.org/docs/devel/sql-createtable.html#SQL-CREATETABLE-EXCLUDE Best regards, Hayato Kuroda FUJITSU LIMITED

RE: [BUG?] check_exclusion_or_unique_constraint false negative

2024-07-31 Thread Hayato Kuroda (Fujitsu)
ments for parallel apply. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Remove duplicate table scan in logical apply worker and code refactoring

2024-07-31 Thread Hayato Kuroda (Fujitsu)
120 Model name:Intel(R) Xeon(R) CPU E7-4890 v2 @ 2.80GHz Core(s) per socket:15 Socket(s): 4 Best regards, Hayato Kuroda FUJITSU LIMITED test.sh Description: test.sh

RE: make pg_createsubscriber option names more consistent

2024-07-31 Thread Hayato Kuroda (Fujitsu)
nology and > a similar operating mode, so it would make sense to keep this consistent. +1. If so, should we say "default current dir." instead of "default current directory" in usage() because pg_upgrade says like that? Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Remove duplicate table scan in logical apply worker and code refactoring

2024-07-31 Thread Hayato Kuroda (Fujitsu)
nfo; + targetrel = rel; + ``` Here I can say the same thing as 1. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: speed up a logical replica setup

2024-07-30 Thread Hayato Kuroda (Fujitsu)
-e198dded0bac%40gmail.com Best regards, Hayato Kuroda FUJITSU LIMITED

RE: speed up a logical replica setup

2024-07-29 Thread Hayato Kuroda (Fujitsu)
ct that users will use. Overall, I could agree to use pg_log_standby_snapshot(). This flushes the WAL asynchronously but ensures the timing is not so delayed. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2024-07-28 Thread Hayato Kuroda (Fujitsu)
`GetUserMappingFromOid` to `GetUserMappingByOid` to keep the name consistent with others. - Comments and docs were updated. Best regards, Hayato Kuroda FUJITSU LIMITED 0001-Extend-postgres_fdw_get_connections-to-return-user-n.patch Description: 0001-Extend-postgres_fdw_get_connections-to-return-user-n.patch

RE: [Proposal] Add foreign-server health checks infrastructure

2024-07-28 Thread Hayato Kuroda (Fujitsu)
fd.revents & > + (POLLRDHUP | POLLHUP | POLLERR | > POLLNVAL)) ? 1 : 0; I think you are right. According to the man page, the input socket is invalid or disconnected if revents has such bits. So such cases should be also regarded as *failure*. After the fix, th

RE: [Proposal] Add foreign-server health checks infrastructure

2024-07-26 Thread Hayato Kuroda (Fujitsu)
://www.postgresql.org/message-id/768032ee-fb57-494b-b674-1ccb65b6f969%40oss.nttdata.com Best regards, Hayato Kuroda FUJITSU LIMITED v44-0001-doc-Enhance-documentation-for-postgres_fdw_get_c.patch Description: v44-0001-doc-Enhance-documentation-for-postgres_fdw_get_c.patch v44-0002-postgres_fdw-Add

RE: [Proposal] Add foreign-server health checks infrastructure

2024-07-25 Thread Hayato Kuroda (Fujitsu)
ng, but can you clarify why (Datun) 0 is returned instead of PG_RETURN_VOID()? [1]: https://www.postgresql.org/message-id/3ee79f2c-e8b3-4342-857c-a31b87e1afda%40eisentraut.org Best regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2024-07-25 Thread Hayato Kuroda (Fujitsu)
Dear Fujii-san, > > Thanks for updating the patches! > > I’ve created a new base patch and split the v42-0001 patch into two parts > to implement the feature and improvements step by step. After pushing these > patches, I’ll focus on the v42-0002 patch next. Best regards, Hay

RE: Parallel heap vacuum

2024-07-25 Thread Hayato Kuroda (Fujitsu)
PI for table scan as well?) [1]: ``` vacuumlazy.c: In function ‘lazy_scan_prune’: vacuumlazy.c:1666:34: error: ‘LVRelState’ {aka ‘struct LVRelState’} has no member named ‘NewRelminMxid’ Assert(MultiXactIdIsValid(vacrel->NewRelminMxid)); ^~ ``` Best regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2024-07-24 Thread Hayato Kuroda (Fujitsu)
Dear Fujii-san, > On 2024/07/18 19:49, Hayato Kuroda (Fujitsu) wrote: > >> Shouldn't this test also check if the returned user_name is valid? > > > > You meant to say that we should print the user_name, right? Done. > > Yes, I think it's better to test if

Found issues related with logical replication and 2PC

2024-07-23 Thread Hayato Kuroda (Fujitsu)
R or 2) apply worker exits. 0002 patch fixes the issue. How do you think? [1]: https://www.postgresql.org/message-id/flat/8fab8-65d74c80-1-2f28e880@39088166 Best regards, Hayato Kuroda FUJITSU LIMITED test_2pc.sh Description: test_2pc.sh 0001-Add-XactLastPrepareEnd-to-indicate-the-last-PREPARE

RE: pg_upgrade and logical replication

2024-07-22 Thread Hayato Kuroda (Fujitsu)
so, in create_logical_replication_slots(), we may have to check the located database for every slot and connect to the appropriate database. These changes make it difficult to parallelize the operation. [1]: https://www.postgresql.org/message-id/flat/20240516211638.GA1688936@nathanxps13 Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Slow catchup of 2PC (twophase) transactions on replica in LR

2024-07-21 Thread Hayato Kuroda (Fujitsu)
blisher cannot be modified if the slot is currently acquired by the * existing walsender. + * + * XXX: when toggling two_phase from "false" to "true", the slot parameter + * is not modified by the backend process so that the lock conflict won't + * occur. The restarted walsender will do the alternation. Therefore, we + * can allow to switch without the restriction. This can be changed in + * the future based on the requirement. ``` Best regards, Hayato Kuroda FUJITSU LIMITED

RE: [Proposal] Add foreign-server health checks infrastructure

2024-07-18 Thread Hayato Kuroda (Fujitsu)
found an inconsistency of name between source and doc, so I unified to postgres_fdw_can_verify_connection(). - Also, test patch (0002) was combined into this. 0002: - user_name was added after the `valid` to preserve ordering of the old version. - GetUserMappingFromOid() is allowed to miss a tuple. [1

RE: Slow catchup of 2PC (twophase) transactions on replica in LR

2024-07-17 Thread Hayato Kuroda (Fujitsu)
, I started to think patches can be merged in future versions because they must be included at once and codes are not so long. Thought? Best regards, Hayato Kuroda FUJITSU LIMITED v20-0001-Allow-altering-of-two_phase-option-of-a-SUBSCRIP.patch Description: v20-0001-Allow-altering-of-two_phase

RE: speed up a logical replica setup

2024-07-17 Thread Hayato Kuroda (Fujitsu)
erted just after the restart_lsn but before the RUNNING_XACT record? In this case, yes, the streaming replication finishes before replicating tuples but logical replication will skip them. Euler's approach cannot be used as-is. Best regards, Hayato Kuroda FUJITSU LIMITED

RE: speed up a logical replica setup

2024-07-17 Thread Hayato Kuroda (Fujitsu)
er. How do you think? [1]: https://www.postgresql.org/message-id/osbpr01mb25521b15bf950d2523bbe143f5...@osbpr01mb2552.jpnprd01.prod.outlook.com [2]: https://www.postgresql.org/message-id/b1f0f8c7-8f01-4950-af77-339df3dc4684%40app.fastmail.com Best regards, Hayato Kuroda FUJITSU LIMITED

RE: Slow catchup of 2PC (twophase) transactions on replica in LR

2024-07-16 Thread Hayato Kuroda (Fujitsu)
s > function simpler. This style was needed to preserve error condition for failover option. Not changed. [1]: https://www.postgresql.org/message-id/OS3PR01MB571834FBD3E6D3804484038F94A32%40OS3PR01MB5718.jpnprd01.prod.outlook.com Best regards, Hayato Kuroda FUJITSU LIMITED v19-0001-Allow-alter

RE: Slow catchup of 2PC (twophase) transactions on replica in LR

2024-07-15 Thread Hayato Kuroda (Fujitsu)
gresql.org/message-id/OS3PR01MB57184E0995521300AC06CB4B94A72%40OS3PR01MB5718.jpnprd01.prod.outlook.com Best regards, Hayato Kuroda FUJITSU LIMITED v18-0001-Allow-altering-of-two_phase-option-of-a-SUBSCRIP.patch Description: v18-0001-Allow-altering-of-two_phase-option-of-a-SUBSCRIP.patch v18-0002-

RE: Slow catchup of 2PC (twophase) transactions on replica in LR

2024-07-09 Thread Hayato Kuroda (Fujitsu)
quired in [2]. Previous patch set could not be accepted due to the initialization miss. PSA new version. Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/ v17-0001-Allow-altering-of-two_phase-option-of-a-SUBSCRIP.patch Description: v17-0001-Allow-altering-of-two_phase-option-of-a-SU

RE: Slow catchup of 2PC (twophase) transactions on replica in LR

2024-07-09 Thread Hayato Kuroda (Fujitsu)
https://www.postgresql.org/message-id/CAA4eK1%2BFRrL_fLWLsWQGHZRESg39ixzDX_S9hU8D7aFtU%2Ba8uQ%40mail.gmail.com [2]: https://www.postgresql.org/message-id/CAA4eK1Khy_YWFoQ1HOF_tGtiixD8YoTg86coX1-ckxt8vK3U%3DQ%40mail.gmail.com Best Regards, Hayato Kuroda FUJITSU LIMITED https://www.fujitsu.com/ v17

  1   2   3   4   5   6   7   >