On 2/9/21 12:47 PM, tsunakawa.ta...@fujitsu.com wrote:
From: Andrey V. Lepikhov <a.lepik...@postgrespro.ru>
I guess you used many hash partitions. Sadly, The current COPY implementation
only accumulates either 1,000 rows or 64 KB of input data (very small!) before
flushing all CopyMultiInsertBuffers. One CopyMultiInsertBuffer corresponds to
one partition. Flushing a CopyMultiInsertBuffer calls ExecForeignCopy() once,
which connects to a remote database, runs COPY FROM STDIN, and disconnects.
Here, the flushing trigger (1,000 rows or 64 KB input data, whichever comes
first) is so small that if there are many target partitions, the amount of data
for each partition is small.
I tried to use 1E4 - 1E8 rows in a tuple buffer. But the results weren't
impressive.
We can use one more GUC instead of a precompiled constant.
Why don't we focus on committing the basic part and addressing the extended
part (0003 and 0004) separately later?
I focused only on the 0001 and 0002 patches.
As Tang-san and you showed, the basic part already demonstrated impressive
improvement. If there's no objection, I'd like to make this ready for
committer in a few days.
Good.
--
regards,
Andrey Lepikhov
Postgres Professional
