RE: Disable WAL logging to speed up data loading

2020-11-30 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > We're emitting only redo logs. So I think theoretically we don't need > anything other than the shutdown checkpoint record because we don't > perform recovery and checkpoint record is required at startup. > > RM_XLOG_ID: > XLOG_FPI_FOR_HINT - not needed? > XLOG_FPI

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-11-30 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > We are relying on the "fact" that the first lseek() call of a > (startup) process tells the truth. We added an assertion so that we > make sure that the cached value won't be cleared during recovery. A > possible remaining danger would be closing of an smgr object of a

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2020-11-30 Thread tsunakawa.ta...@fujitsu.com
From: Amit Langote > Andrey's original patch had the flag to, as I understand it, make the > partitioning case work correctly. When inserting into a > non-partitioned table, there's only one relation to care about. In > that case, CopyFrom() can use either the new COPY interface or the > INSERT

RE: Disable WAL logging to speed up data loading

2020-12-01 Thread tsunakawa.ta...@fujitsu.com
From: Osumi, Takamichi/大墨 昂道 > I executed each wal_level three times and calculated the average time > and found that disabling WAL logging reduced about 73 % of the minimal's > loading speed > in this test. This speed-up came from the difference of generated WAL sizes. So, it's 4x speedup when t

[bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2020-12-02 Thread tsunakawa.ta...@fujitsu.com
Hello, ALTER TABLE SET LOGGED/UNLOGED on a partitioned table completes successfully, but nothing changes. I expected the underlying partitions are changed accordingly. The attached patch fixes this. Regards Takayuki Tsunakawa v1-0001-Make-ALTER-TABLE-SET-LOGGED-UNLOGGED-on-a-partiti.pat

RE: Deprecate custom encoding conversions

2020-12-02 Thread tsunakawa.ta...@fujitsu.com
From: Heikki Linnakangas > I propose that we add a notice to the CREATE CONVERSION docs to say that > it is deprecated, and remove it in a few years. > > Any objections? Anyone using custom encoding conversions in production? I can't answer deeper questions because I'm not familiar with characte

RE: Deprecate custom encoding conversions

2020-12-02 Thread tsunakawa.ta...@fujitsu.com
From: Michael Paquier > Tsunakawa-san, could you post a link to this article, if possible? I am > curious > about their problem and why they used CREATE CONVERSION as a way to > solve it. That's fine even if it is in Japanese. I just pulled info from my old memory in my previous mail. Now the

RE: Disable WAL logging to speed up data loading

2020-12-02 Thread tsunakawa.ta...@fujitsu.com
From: Osumi, Takamichi/大墨 昂道 > I've made a new patch v05 that took in comments to filter out WALs more > strictly > and addressed some minor fixes that were discussed within past few days. > Also, I changed the documentations, considering those modifications. The code looks good,

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-02 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > Apologies for the delay, but attached are the updated versions to simplify the > patches. Looks good for me. Thanks to Horiguchi-san and Andres-san, the code bebecame further compact and easier to read. I've marked this ready for committer. To the committer: I

RE: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2020-12-03 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > 1) What happens if a partitioned table has a foreign partition along > with few other local partitions[1]? Currently, if we try to set > logged/unlogged of a foreign table, then an "ERROR: "" is not a > table" is thrown. This makes sense. With your patch also we see

RE: In-placre persistance change of a relation

2020-12-03 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > > No, not really. The issue is more around what happens if we crash > > part way through. At crash recovery time, the system catalogs are not > > available, because the database isn't consistent yet and, anyway, the > > startup process can't be bound to a database, let

RE: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2020-12-06 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > IMHO, we should also change the parent table. Say, I have 2 local > partitions for a logged table, then I alter that table to > unlogged(with your patch, parent table doesn't become unlogged whereas > the partitions will), and I detach all the partitions for some reason.

RE: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2020-12-07 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > Do you mean to say that if we detach all the partitions(assuming they > are all unlogged) then the parent table(assuming logged) gets changed > to unlogged? Does it happen on master? Am I missing something here? No, the parent remains logged in that case both on master a

RE: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2020-12-07 Thread tsunakawa.ta...@fujitsu.com
From: Alvaro Herrera > Does "ALTER TABLE ONLY parent" work correctly? Namely, do not affect > existing partitions, but cause future partitions to acquire the new > setting. Yes, it works correctly in the sense that ALTER TABLE ONLY on a partitioned table does nothing because it has no storage a

RE: Blocking I/O, async I/O and io_uring

2020-12-07 Thread tsunakawa.ta...@fujitsu.com
From: Andres Freund > Especially with direct IO > checkpointing can be a lot faster *and* less impactful on the "regular" > load. I'm looking forward to this from the async+direct I/O, since the throughput of some write-heavy workload decreased by half or more during checkpointing (due to fsync

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-07 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > Because one of the rel's cached value was false, it forced the > full-scan path for TRUNCATE. > Is there a possible workaround for this? Hmm, the other two relfilenodes are for the TOAST table and index of the target table. I think the INSERT didn't access those

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-08 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > At Tue, 8 Dec 2020 16:28:41 +0530, Amit Kapila > wrote in > I also can't think of a way to use an optimized path for such cases > > but I don't agree with your comment on if it is common enough that we > > leave this optimization entirely for the truncate path. > > An

RE: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2020-12-08 Thread tsunakawa.ta...@fujitsu.com
From: Alvaro Herrera > But what happens when you create another partition after you change the > "loggedness" of the partitioned table? The new partition will have a property specified when the user creates it. That is, while the storage property of each storage unit (=partition) is basically

RE: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2020-12-09 Thread tsunakawa.ta...@fujitsu.com
From: Alvaro Herrera > Well, that definition seems unfriendly to me. I prefer the stance that > if you change the value for the parent, then future partitions inherit > that value. That would be right when the storage property is an optional specification such as fillfactor. For example, when

RE: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2020-12-09 Thread tsunakawa.ta...@fujitsu.com
From: Bharath Rupireddy > I'm not sure how many more of such commands exist which require changes. > > How about doing it this way? > > 1) Have a separate common thread listing the commands and specifying > clearly the agreed behaviour for such commands. > 2) Whenever the separate patches are su

RE: Parallel INSERT (INTO ... SELECT ...)

2020-12-09 Thread tsunakawa.ta...@fujitsu.com
From: Greg Nancarrow > Firstly, in order to perform parallel-safety checks in the case of > partitions, the > patch currently recursively locks/unlocks > (AccessShareLock) each partition during such checks (as each partition may > itself be a partitioned table). Is there a better way of performin

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-09 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > Oh, sorry. I wrongly looked to non-recovery path. smgrnblocks is > called during buffer loading while recovery. So, smgrnblock is called > for indexes if any update happens on the heap relation. I misunderstood that you said there's no problem with the TOAST index becaus

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-10 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > I added comment in 0004 the limitation of optimization when there are TOAST > relations that use NON-PLAIN strategy. i.e. The optimization works if the data > types used are integers, OID, bytea, etc. But for TOAST-able data types like > text, > the optimization wi

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-10 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > On Thursday, December 10, 2020 8:12 PM, Amit Kapila wrote: > > AFAIU, it won't take optimization path only when we have TOAST relation but > > there is no insertion corresponding to it. If so, then we don't need to > > mention > > it specifically because there are

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-10 Thread tsunakawa.ta...@fujitsu.com
From: tsunakawa.ta...@fujitsu.com > What's valuable as a code comment to describe the remaining issue is that the You can attach XXX or FIXME in front of the issue description for easier search. (XXX appears to be used much more often in Postgres.) Regards Takayuki Tsunakawa

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-13 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > Attached are the final updated patches. Looks good, and the patch remains ready for committer. (Personally, I wanted the code comment to touch upon the TOAST and FSM/VM for the reader, because we couldn't think of those possibilities and took some time to find w

RE: libpq debug log

2020-12-21 Thread tsunakawa.ta...@fujitsu.com
From: k.jami...@fujitsu.com > I understand that protocol 2.0 is still supported, but it is only used for > PostgreSQL versions 7.3 and earlier, which is not updated by fixes anymore > since we only backpatch up to previous 5 versions. However I am not sure if > it's a good idea, but how about if w

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-21 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > This answers the second part of the question but what about the first > part (We hold a buffer partition lock, and have done a lookup in th > mapping table. Why are we then rechecking the > relfilenode/fork/blocknum?) > > I think we don't need such a check, rather we can have

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-21 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > Why would all client backends wait for AccessExclusive lock on this > relation? Say, a client needs a buffer for some other relation and > that might evict this buffer after we release the lock on the > partition. In StrategyGetBuffer, it is important to either have a pin > on

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-21 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > Mmm. If that is true, doesn't the unoptimized path also need the > rechecking? Yes, the traditional processing does the recheck after acquiring the buffer header spinlock. Regards Takayuki Tsunakawa

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-22 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > + /* Get the number of blocks for a relation's fork */ > + block[i][j] = smgrnblocks(smgr_reln[i], j, &cached); > + > + if (!cached) > + goto buffer_full_scan; > > Why do we need to use goto here? We can simply break from the loop and > then check if (cached && nBlocksToInvali

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-12-23 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク compiles. Passes the regression tests too. > Your feedbacks are definitely welcome. The code looks correct and has become further compact. Remains ready for committer. Regards Takayuki Tsunakawa

RE: [bug fix] ALTER TABLE SET LOGGED/UNLOGGED on a partitioned table does nothing silently

2020-12-25 Thread tsunakawa.ta...@fujitsu.com
From: Michael Paquier > On Wed, Dec 09, 2020 at 09:52:17AM -0300, Alvaro Herrera wrote: > > Well, that definition seems unfriendly to me. I prefer the stance > > that if you change the value for the parent, then future partitions > > inherit that value. > > That's indeed more interesting from th

RE: Global snapshots

2020-12-31 Thread tsunakawa.ta...@fujitsu.com
Hello, Fujii-san and I discussed how to move the scale-out development forward. We are both worried that Clock-SI is (highly?) likely to infringe the said Microsoft's patent. So we agreed we are going to investigate the Clock-SI and the patent, and if we have to conclude that we cannot embra

RE: Disable WAL logging to speed up data loading

2020-12-31 Thread tsunakawa.ta...@fujitsu.com
From: Michael Paquier > Something that has not been mentioned on this thread is that if you could also > put pg_wal/ on a RAM disk. That's similarly unsafe, of course, but it does > not > require any extra upstream patching, and that should be really fast for the > case > of this thread. If yo

RE: Disable WAL logging to speed up data loading

2020-12-31 Thread tsunakawa.ta...@fujitsu.com
From: Simon Riggs > Agreed, it is a footgun. -1 to commit the patch as-is. > > The patch to avoid WAL is simple but it is dangerous for both the user > and the PostgreSQL project. > > In my experience, people will use this option and when it crashes and > they lose their data, they will claim Po

When (and whether) should we improve the chapter on parallel query to accommodate parallel data updates?

2021-01-05 Thread tsunakawa.ta...@fujitsu.com
Hello, While I'm reviewing patches for parallel operations, it occured to me that we may need to correct the words and enrich content in "15. Parallel Query". I don't have a concrete image of how the content should be yet, and I think the documentation may be improved after the feature freeze

RE: When (and whether) should we improve the chapter on parallel query to accommodate parallel data updates?

2021-01-06 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > I think each feature should develop the docs as part of feature > development but if we want to see some additional work like improving > overall docs for parallel execution as you are suggesting then it can > be done separately as well. > > I think you have a valid point but

RE: Enhance traceability of wal_level changes for backup management

2021-01-06 Thread tsunakawa.ta...@fujitsu.com
From: osumi.takami...@fujitsu.com > I wondered, couldn't backup management tools utilize the information > in the backup that is needed to be made when wal_level is changed to "none" > for example ? IIRC, someone proposed in the original thread that the change count can be recorded in pg_control

RE: Disable WAL logging to speed up data loading

2021-01-07 Thread tsunakawa.ta...@fujitsu.com
From: Robert Haas > Were the issues that I mentioned regarding GIST (and maybe other AMs) > in the last paragraph of > http://postgr.es/m/CA+TgmoZEZ5RONS49C7mEpjhjndqMQtVrz_LCQUkpRW > dmrev...@mail.gmail.com > addressed in some way? That seems like a pretty hard engineering > problem to me, and I

RE: data_checksums enabled by default (was: Move --data-checksums to common options in initdb --help)

2021-01-07 Thread tsunakawa.ta...@fujitsu.com
From: Stephen Frost > I hadn't intended to make an argument that enabling checksums was > equivilant to enabling or disabling fsync- I said it was 'akin', by which I > meant it > was similar in character, as in, as I said previously, a way for PG to hedge > against certain external-to-PG risks (t

RE: libpq debug log

2021-02-11 Thread tsunakawa.ta...@fujitsu.com
(48) void PQtrace(PGconn *conn, FILE *stream); + + Calls PQtraceSetFlags to output with or without a timestamp. + + Why is this necessary? Even if you decide to remove this change, can you share your idea on why you added this just in case? (49) + Dete

RE: Parallel INSERT (INTO ... SELECT ...)

2021-02-11 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > > What would the result look like if you turn off > > parallel_leader_participation? If the leader is freed from > > reading/writing the table and index, the index page splits and > > internal lock contention may decrease enough to recover part of the loss. > > > > https:

RE: Parallel INSERT (INTO ... SELECT ...)

2021-02-11 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > If we diable bitmapscan, the performance degradation seems will not happen. Yes, but that's because the hundreds of times slower sequential scan hides the insert time. Furthermore, as an aside, Worker 3 does much of the work in the parallel sequential scan + parallel i

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-11 Thread tsunakawa.ta...@fujitsu.com
From: Andrey V. Lepikhov > On 2/9/21 12:47 PM, tsunakawa.ta...@fujitsu.com wrote: > > As Tang-san and you showed, the basic part already demonstrated > impressive improvement. If there's no objection, I'd like to make this ready > for > committer in a few days.

RE: [PoC] Non-volatile WAL buffer

2021-02-14 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > I've done some performance benchmarks with the master and NTT v4 > patch. Let me share the results. > ... > master NTT master-unlogged > 32 113209 67107 154298 > 64 144880 54289 178883 > 96 151405 50562 180018 > > "master-unlogged" is

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-14 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > This is crashing during fdw check. > http://cfbot.cputube.org/andrey-lepikhov.html > > Maybe it's related to this patch: > |commit 6214e2b2280462cbc3aa1986e350e167651b3905 > |Fix permission checks on constraint violation errors on partitions. > |Security: CVE-2021-33

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-02-15 Thread tsunakawa.ta...@fujitsu.com
From: Amit Langote > Think I have mentioned upthread that this looks better as: > > if (rootResultRelInfo->ri_usesMultiInsert) > leaf_part_rri->ri_usesMultiInsert = ExecMultiInsertAllowed(leaf_part_rri); > > This keeps the logic confined to ExecInitPartitionInfo() where it > belongs. No poi

RE: [PoC] Non-volatile WAL buffer

2021-02-16 Thread tsunakawa.ta...@fujitsu.com
From: Takashi Menjo > I made a new page at PostgreSQL Wiki to gather and summarize information and > discussion about PMEM-backed WAL designs and implementations. Some parts of > the page are TBD. I will continue to maintain the page. Requests are welcome. > > Persistent Memory for WAL > https

RE: Parallel INSERT (INTO ... SELECT ...)

2021-02-18 Thread tsunakawa.ta...@fujitsu.com
From: Greg Nancarrow -- On Mon, Jan 25, 2021 at 10:23 AM tsunakawa.ta...@fujitsu.com wrote: > (8) > + /* > +* If the trigger type is RI_TRIGGER_FK, this indicates a FK > exists in > +

RE: Parallel INSERT (INTO ... SELECT ...)

2021-02-21 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > It is quite possible what you are saying is correct but I feel that is > not this patch's fault. So, won't it better to discuss this in a > separate thread? > > Good use case but again, I think this can be done as a separate patch. Agreed. I think even the current patch offer

RE: libpq debug log

2021-02-21 Thread tsunakawa.ta...@fujitsu.com
From: Iwata, Aya/岩田 彩 > I update patch to v18. It has been fixed in response to Tsunakawa san's > review. (52) + of tracing. If (flags contains PQTRACE_SUPPRESS_TIMESTAMPS), () can be removed? (53) + int inLogging; /* next byte of logging */ I u

RE: libpq debug log

2021-02-22 Thread tsunakawa.ta...@fujitsu.com
Alvaro-san, Iwata-san, From: Iwata, Aya/岩田 彩 > I update patch to v19. ... > And 3 bugs I noted February 2nd email are all fixed. > > 1. fix 3 bugs > > 1.1 -1 output in "Query" message > > 1.2 two message output in "ReadyForQuery" message > > 1.3 "StartupMessage" output as " UnknownMessage "

RE: libpq debug log

2021-02-23 Thread tsunakawa.ta...@fujitsu.com
From: alvhe...@alvh.no-ip.org > I'll give this another look tomorrow, but I wanted to pass along that I prefer > libpq-trace.{c,h} instead of libpq-logging. I also renamed variable "pin" and > pgindented. Ah, you're right, because the function names are PQtrace() and PQuntrace(). > I don't hav

RE: libpq debug log

2021-02-25 Thread tsunakawa.ta...@fujitsu.com
From: Álvaro Herrera > It appears that something is still wrong. I applied lipq pipeline v27 from > [1] and ran src/test/modules/test_libpq/pipeline singlerow, after patching it > to do PQtrace() after PQconn(). Below is the output I get from that. The > noteworthy point is that "ParseComplete

RE: Global snapshots

2021-02-25 Thread tsunakawa.ta...@fujitsu.com
From: Andrey V. Lepikhov > After researching in depth, I think this is the real problem. > My idea was that we are not using real clocks, we only use clock ticks to > measure time intervals. It can also be interpreted as a kind of clock. Yes, patent claims tend to be written to cover broad interp

RE: Parallel INSERT (INTO ... SELECT ...)

2021-02-26 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > After doing some more tests on it (performance degradation will not happen > when source table is out of order). > I think we can say the performance degradation is related to the order of the > data in source table. ... > So, the order of data 's influence seems a normal

RE: libpq debug log

2021-02-26 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > Using (inCursor - inStart) as logCursor doesn't work correctly if tracing > state > desyncs. Once desync happens inStart can be moved at the timing that the > tracing code doesn't expect. This requires (as I mentioned upthread) > pqReadData to actively reset logCursor,

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-03 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > Find attached some language fixes. Thanks a lot! (I wish there will be some tool like "pgEnglish" that corrects English in code comments and docs.) > |/* Do this to ensure we've pumped libpq back to idle state */ > > I don't know why you mean by "pumped"? I cha

RE: Avoid CommandCounterIncrement in RI trigger when INSERT INTO referencing table

2021-03-03 Thread tsunakawa.ta...@fujitsu.com
Excuse me for asking probably stupid questions... From: houzj.f...@fujitsu.com > As far as I can see, it’s only necessary to increment command id when the > INSERT command modified the referenced table. Why do we have to increment the command ID when the INSERT's target table is a referenced ta

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-03 Thread tsunakawa.ta...@fujitsu.com
From: Zhihong Yu > This feature enables bulk COPY into foreign table in the case of > multi inserts is possible > > 'is possible' -> 'if possible' > > FDWAPI was extended by next routines: > > next routines -> the following routines Thank you, fixed slightly differently. (I feel the need for

RE: libpq debug log

2021-03-04 Thread tsunakawa.ta...@fujitsu.com
Tom-san, Alvaro-san, From: Tom Lane > I took a quick look through the v22 patch, and TBH I don't like much of > anything > at all about the proposed architecture. It's retained most of the flavor of > the > way it was done before, which was a hangover from the old process-on-the-fly > scheme.

RE: libpq debug log

2021-03-04 Thread tsunakawa.ta...@fujitsu.com
From: tsunakawa.ta...@fujitsu.com > I understood that the former is pqParseInput3() and the latter is > pqPutMsgEnd(). They call the logging function wne conn->pfdebug is not > NULL. Its signature is like this (that will be defined in libpq-trace.c): > > void pqLog

RE: libpq debug log

2021-03-04 Thread tsunakawa.ta...@fujitsu.com
From: Tom Lane > But I think passing the message start address explicitly might be better than > having it understand the buffering behavior in enough detail to know where to > find the message. Part of the point here > (IMO) is to decouple the tracing logic from the core libpq logic, in hopes of

RE: Avoid CommandCounterIncrement in RI trigger when INSERT INTO referencing table

2021-03-04 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > From the wiki[1], CCI is to let statements can not see the rows they modify. > > Here is an example of the case 1): > (Note table referenced and referencing are both empty) > - > postgres=# with cte as (insert into referenced values(1)) insert into > referencing value

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-05 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > I think this change to the regression tests is suspicous: > > -CONTEXT: remote SQL command: INSERT INTO public.loc2(f1, f2) VALUES > ($1, $2) > -COPY rem2, line 1: "-1 xyzzy" > +CONTEXT: COPY loc2, line 1: "-1 xyzzy" > +remote SQL command: COPY public.loc2(f1, f2) FR

RE: Parallel INSERT (INTO ... SELECT ...)

2021-03-07 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > For now, I have left 0005 and 0006 patches, we can come back to those once we > are done with the first set of patches. The first patch looks good to me and I > think we can commit it and then bikeshed about GUC/reloption patch. Agreed, it looks good to me, too. Regards Taka

RE: Implementing Incremental View Maintenance

2021-03-08 Thread tsunakawa.ta...@fujitsu.com
From: Thomas Munro > It's probably time to move forward with the plan of pushing the > results into a commitfest.postgresql.org API, and then making Magnus > et al write the email spam code with a preferences screen linked to > your community account :-D +1 I wish to see all the patch status info

RE: [PoC] Non-volatile WAL buffer

2021-03-08 Thread tsunakawa.ta...@fujitsu.com
From: Takashi Menjo > > The other question is whether simply placing WAL on DAX (without any > > code changes) is safe. If it's not, then all the "speedups" are > > computed with respect to unsafe configuration and so are useless. And > > BTT should be used instead, which would of course produce v

RE: POC: Cleaning up orphaned files using undo logs

2021-03-09 Thread tsunakawa.ta...@fujitsu.com
I'm crawling like a snail to read the patch set. Below are my first set of review comments, which are all minor. (1) + tablespacetemporary temporary -> undo (2) undo_tablespaces (string) + ... +The value is a list of names of tablespaces. When there is more than +

RE: Avoid CommandCounterIncrement in RI trigger when INSERT INTO referencing table

2021-03-10 Thread tsunakawa.ta...@fujitsu.com
From: Hou, Zhijie/侯 志杰 > After some more on how to support parallel insert into fk relation. > It seems we do not have a cheap way to implement this feature. > > In RI_FKey_check, Currently, postgres execute "select xx for key share" to > check that foreign key exists in PK table. > However "sele

RE: libpq debug log

2021-03-10 Thread tsunakawa.ta...@fujitsu.com
Alvaro-san, Tom-san, Horiguchi-san, From: Kyotaro Horiguchi > +1 for the thanks for the quick work. I have some random comments > after a quick look on it. Thank you very much for giving many comments. And We're sorry to have caused you trouble. I told Iwata-san yesterday to modify the patch

RE: libpq debug log

2021-03-10 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > The output functions copy message bytes into local variable but the > same effect can be obtained by just casing via typed pointer type. > > uint32 tmp4; > .. > memcpy(&tmp4, buf + *cursor, 4); > result = (int) pg_ntoh32(tmp4); > > can be written as > > resul

RE: libpq debug log

2021-03-10 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > Right. So something like this? > > unsigned char p; > > p = buf + *cursor; > result = (uint32) (*p << 24) + (*(p + 1)) << 16 + ...); Yes, that would work (if p is a pointer), but I think memcpy() is enough like pqGetInt() does. Regards Takayuki Tsunakawa

RE: Parallel INSERT (INTO ... SELECT ...)

2021-03-10 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > Now, coming back to Hou-San's patch to introduce a GUC and reloption > for this feature, I think both of those make sense to me because when > the feature is enabled via GUC, one might want to disable it for > partitioned tables? Do we agree on that part or someone thinks > oth

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: Thomas Munro > On Fri, Mar 12, 2021 at 5:20 PM Amit Kapila wrote: > > uint64 > > +1 +1 I'll send a patch later. Regards Takayuki Tsunakawa

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: Thomas Munro > > uint64 > > +1 Thank you, the patch is attached (we tend to forget how large our world is... 64-bit) We're sorry to cause you trouble. Regards Takayuki Tsunakawa v1-0001-Fix-overflow-when-counting-the-number-of-buffers-.patch Description: v1-0001-Fix-overflo

RE: [POC] Fast COPY FROM command for the table with foreign partitions

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: Justin Pryzby > Could you rebase again and send an updated patch ? > I could do it if you want. Rebased and attached. Fortunately, there was no rebase conflict this time. make check passed for PG core and postgres_fdw. Regards Takayuki Tsunakawa v20-0001-Fast-COPY-FROM-into-

RE: [Patch] Optimize dropping of relation buffers using dlist

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > About the patch, it would be better to change the type of > BUF_DROP_FULL_SCAN_THRESHOLD to uint64, even though the current > value > doesn't harm. OK, attached, to be prepared for the distant future when NBuffers becomes 64-bit. Regards Takayuki Tsunakawa

RE: Enhance traceability of wal_level changes for backup management

2021-03-11 Thread tsunakawa.ta...@fujitsu.com
From: David Steele > As a backup software author, I don't see this feature as very useful. > > The problem is that there are lots of ways for WAL to go missing so > monitoring the WAL archive for gaps is essential and this feature would > not replace that requirement. The only extra information y

RE: libpq debug log

2021-03-15 Thread tsunakawa.ta...@fujitsu.com
I'm looking at the last file libpq-trace.c. I'll continue the review after lunch. Below are some comments so far. (1) - Enables tracing of the client/server communication to a debugging file stream. + Enables tracing of the client/server communication to a debugging file + str

RE: libpq debug log

2021-03-15 Thread tsunakawa.ta...@fujitsu.com
I've not finished reviewing yet, but there seems to be many mistakes. I'm sending second set of review comments now so you can fix them in parallel. (8) + charid = '\0'; This initialization is not required because id will always be assigned a value shortly. (9) +static int

RE: libpq debug log

2021-03-16 Thread tsunakawa.ta...@fujitsu.com
I've finished the review. Here are the last set of comments. (16) (15) is also true for the processing of 'H' message. (17) pqTraceOutputD + for (i = 0; i < nfields; i++) + { + len = pqTraceOutputInt32(message + cursor, f); +

RE: libpq debug log

2021-03-16 Thread tsunakawa.ta...@fujitsu.com
Alvaro-san, Thank you for taking your time to take a look at an incomplete patch. I thought I would ask you for final check for commit after Iwata-san has reflected my review comments. I discussed with Iwata-sn your below comments. Let me convey her opinions. (She is now focusing on fixing

RE: extension patch of CREATE OR REPLACE TRIGGER

2020-10-26 Thread tsunakawa.ta...@fujitsu.com
From: osumi.takami...@fujitsu.com > > > * I don't think that you've fully thought through the implications > > > of replacing a trigger for a table that the current transaction has > > > already modified. Is it really sufficient, or even useful, to do > > > this: > > > > > > +/* > > >

RE: Multiple hosts in connection string failed to failover in non-hot standby mode

2020-10-27 Thread tsunakawa.ta...@fujitsu.com
Please send emails in text format. Your email was in HTML, and I changed this reply to text format. From: Hubert Zhang > Libpq has supported to specify multiple hosts in connection string and enable > auto failover when the previous PostgreSQL instance cannot be accessed. > But when I tried

RE: Global snapshots

2020-10-28 Thread tsunakawa.ta...@fujitsu.com
Fujii-san, Sawada-san, all, From: Fujii Masao > Yeah, so if we need to guarantee the transaction linearizability even > in distributed env (probably this is yes. Right?), using only Clock-SI > is not enough. We would need to implement something more > in addition to Clock-SI or adopt the differen

RE: Disable WAL logging to speed up data loading

2020-10-28 Thread tsunakawa.ta...@fujitsu.com
From: Osumi, Takamichi/大墨 昂道 > I wrote and attached the first patch to disable WAL logging. > This patch passes the regression test of check-world already and is formatted I think make check-world uses the default setting for wal_level. You need to set wal_level = none and do make installcheck-

RE: Multiple hosts in connection string failed to failover in non-hot standby mode

2020-10-28 Thread tsunakawa.ta...@fujitsu.com
From: Hubert Zhang > Hao Wu and I wrote a patch to fix this problem. Client side libpq should try > another hosts in connection string when it is rejected by a non-hot standby, > or the first host encounter some n/w problems during the libpq handshake. Thank you. Please add it to the November

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-10-28 Thread tsunakawa.ta...@fujitsu.com
The patch looks almost good except for the minor ones: (1) + for (i = 0; i < nnodes; i++) + { + RelFileNodeBackend rnode = smgr_reln[i]->smgr_rnode; + + rnodes[i] = rnode; + } You can write: + for (i = 0; i < nnodes; i++) + rnodes

RE: POC: postgres_fdw insert batching

2020-11-09 Thread tsunakawa.ta...@fujitsu.com
Hello, The attached patch implements the new bulk insert routine for postgres_fdw and the executor utilizing it. It passes make check-world. I measured performance in a basic non-partitioned case by modifying Tomas-san's scripts. They perform an INSERT SELECT statement that copies one millio

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-11-09 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > So I proceeded to update the patches using the "cached" parameter and > updated the corresponding comments to it in 0002. OK, I'm in favor of the name "cached" now, although I first agreed with Horiguchi-san in that it's better to use a name that represents the na

RE: POC: postgres_fdw insert batching

2020-11-10 Thread tsunakawa.ta...@fujitsu.com
From: Tomas Vondra > I see the patch builds the "bulk" query in execute_foreign_modify. IMO > that's something we should do earlier, when we're building the simple > query (for 1-row inserts). I'd understand if you were concerned about > overhead in case of 1-row inserts, trying to not plan the bu

RE: Disable WAL logging to speed up data loading

2020-11-11 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > Thanks! Since this feature is different from the feature that is > being proposed in this thhead, I started another thread for this > feature. > > https://www.postgresql.org/message-id/2020.173317.460890039962481 > 381.horikyota@gmail.com Thank you, Horiguchi-s

RE: POC: postgres_fdw insert batching

2020-11-11 Thread tsunakawa.ta...@fujitsu.com
From: t...@corona.is.ed.ac.uk On Behalf Of > Does this patch affect trigger semantics on the base table? > > At the moment when I insert 1000 rows into a postgres_fdw table using a > single insert statement (e.g. INSERT INTO fdw_foo SELECT ... FROM bar) I > naively expect a "statement level" trig

RE: Disable WAL logging to speed up data loading

2020-11-11 Thread tsunakawa.ta...@fujitsu.com
From: Stephen Frost > I'm not sure that I see this as really being much of an issue. Perhaps there > are > some things we can do, as I mentioned before, to make it easier for users to > have tables be created as unlogged from the start, or to be able to ALTER > TABLE a bunch of tables at once (u

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-11-11 Thread tsunakawa.ta...@fujitsu.com
The patch looks OK. I think as Thomas-san suggested, we can remove the modification to smgrnblocks() and don't care wheter the size is cached or not. But I think the current patch is good too, so I'd like to leave it up to a committer to decide which to choose. I measured performance in a dif

RE: Detecting File Damage & Inconsistencies

2020-11-11 Thread tsunakawa.ta...@fujitsu.com
From: Simon Riggs > I would like to propose a few points that will help us detect file > damage, inconsistencies in files and track actions of users. Hello, Simon san. Long time no see. I'm happy to see you be back here recently. What kind of improvement do you expect? What problems would th

RE: Detecting File Damage & Inconsistencies

2020-11-12 Thread tsunakawa.ta...@fujitsu.com
From: Simon Riggs > If a rogue user/process is suspected, this would allow you to identify > more easily the changes made by specific sessions/users. Isn't that kind of auditing a job of pgAudit or log_statement = mod? Or, does "more easily" mean that you find pgAudit complex to use and/or log_

RE: In-placre persistance change of a relation

2020-11-12 Thread tsunakawa.ta...@fujitsu.com
Hi Horiguchi-san, Thank you for making a patch so quickly. I've started looking at it. What makes you think this is a PoC? Documentation and test cases? If there's something you think that doesn't work or are concerned about, can you share it? Do you know the reason why data copy was done b

  1   2   3   4   5   >