On 2018-Aug-06, Kohei KaiGai wrote:
> 2018-08-06 1:50 GMT+09:00 Alvaro Herrera :
> >> Now, it consumed about 60MB rss at the beginning of COPY FROM, and it
> >> grows up very slowly during the COPY FROM execution, then grew up to
> >> 250MB before completion.
> >> We may have another memory blocks
2018-08-06 1:50 GMT+09:00 Alvaro Herrera :
>> Now, it consumed about 60MB rss at the beginning of COPY FROM, and it
>> grows up very slowly during the COPY FROM execution, then grew up to
>> 250MB before completion.
>> We may have another memory blocks which are not released during
>> execution, ho
On 2018-Aug-04, Kohei KaiGai wrote:
> I could load the same data (544GB csv, 789GB heap) using COPY FROM
> successfully.
> When I reported the problem, rss usage of postgresql process increased
> about 10MB/s ratio, then OOM killer eliminated after a few hours.
OK, I think we can consider this p
2018-08-03 12:38 GMT+09:00 Alvaro Herrera :
> On 2018-Aug-03, Kohei KaiGai wrote:
>
>> 2018-08-02 5:38 GMT+09:00 Alvaro Herrera :
>> > On 2018-Aug-01, Alvaro Herrera wrote:
>> >
>> >> Right, makes sense. Pushed that way.
>> >
>> > KaiGai, if you can please confirm that the pushed change fixes your
On 2018-Aug-03, Kohei KaiGai wrote:
> 2018-08-02 5:38 GMT+09:00 Alvaro Herrera :
> > On 2018-Aug-01, Alvaro Herrera wrote:
> >
> >> Right, makes sense. Pushed that way.
> >
> > KaiGai, if you can please confirm that the pushed change fixes your test
> > case, I'd appreciate it.
>
> Can you wait f
2018-08-02 5:38 GMT+09:00 Alvaro Herrera :
> On 2018-Aug-01, Alvaro Herrera wrote:
>
>> Right, makes sense. Pushed that way.
>
> KaiGai, if you can please confirm that the pushed change fixes your test
> case, I'd appreciate it.
>
Can you wait for a few days? I can drop the test dataset and reuse
On 2018-Aug-02, Amit Langote wrote:
> On 2018/08/02 6:03, Alvaro Herrera wrote:
> > Hmm, variable shadowing ...
>
> Crap, sorry about forgetting to remove that and thanks for taking care of
> that.
I think it was my bug actually, while rebasing to branch master after
renaming the variable.
--
On 2018/08/02 6:03, Alvaro Herrera wrote:
> On 2018-Aug-01, Andres Freund wrote:
>
>> On 2018-08-01 16:38:11 -0400, Alvaro Herrera wrote:
>>> On 2018-Jul-24, Amit Langote wrote:
>>>
Your patch takes care of allocation happening inside
get_partition_for_tuple, but as you mention there mig
On 2018/08/02 5:38, Alvaro Herrera wrote:
> On 2018-Jul-24, Amit Langote wrote:
>
>> Your patch takes care of allocation happening inside
>> get_partition_for_tuple, but as you mention there might be others in its
>> caller ExecFindPartition. So, I think we should switch to the per-tuple
>> conte
On 2018-Aug-01, Andres Freund wrote:
> On 2018-08-01 16:38:11 -0400, Alvaro Herrera wrote:
> > On 2018-Jul-24, Amit Langote wrote:
> >
> > > Your patch takes care of allocation happening inside
> > > get_partition_for_tuple, but as you mention there might be others in its
> > > caller ExecFindPar
On 2018-08-01 16:38:11 -0400, Alvaro Herrera wrote:
> On 2018-Jul-24, Amit Langote wrote:
>
> > Your patch takes care of allocation happening inside
> > get_partition_for_tuple, but as you mention there might be others in its
> > caller ExecFindPartition. So, I think we should switch to the per-t
On 2018-Aug-01, Alvaro Herrera wrote:
> Right, makes sense. Pushed that way.
KaiGai, if you can please confirm that the pushed change fixes your test
case, I'd appreciate it.
--
Álvaro Herrerahttps://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Trainin
On 2018-Jul-24, Amit Langote wrote:
> Your patch takes care of allocation happening inside
> get_partition_for_tuple, but as you mention there might be others in its
> caller ExecFindPartition. So, I think we should switch to the per-tuple
> context in ExecFindPartition.
Right, makes sense. Pus
Hi Kaigai-san,
Thanks for the report and the patch.
On 2018/07/24 11:43, Kohei KaiGai wrote:
> Further investigation I did:
>
> CopyFrom() calls ExecFindPartition() to identify the destination child
> table of partitioned table.
> Then, it internally calls get_partition_for_tuple() to get partit
Further investigation I did:
CopyFrom() calls ExecFindPartition() to identify the destination child
table of partitioned table.
Then, it internally calls get_partition_for_tuple() to get partition
index according to the key value.
This invocation is not under the per-tuple context.
In case of has
On Tue, Jul 24, 2018 at 09:41:52AM +0900, Kohei KaiGai wrote:
> In PG11beta2, my backend process gets terminated during COPY FROM of
> large text file (544GB) on partitioned table.
> The kernel log says OOM Killer send SIGKILL due to memory pressure.
> In fact, 63GB of system physical 64GB was cons
16 matches
Mail list logo
