Re: JSON out of memory error on PostgreSQL 9.6.x

2017-12-03 Thread Michael Paquier
On Mon, Dec 4, 2017 at 9:39 AM, Tom Lane wrote: > Ah. The problem here is that "json_rmq->>'totalSize'" leaks some memory > on each execution, and it's executed again for each row produced by the > json_array_elements() SRF, and the memory can't be reclaimed until we've > finished the full output

Re: JSON out of memory error on PostgreSQL 9.6.x

2017-12-03 Thread Yuri Budilov
thank you, I will look into the work-around ! From: Tom Lane Sent: Monday, 4 December 2017 11:39 AM To: Yuri Budilov Cc: rob stone; pgsql-general@lists.postgresql.org Subject: Re: JSON out of memory error on PostgreSQL 9.6.x Yuri Budilov writes: > The

Re: JSON out of memory error on PostgreSQL 9.6.x

2017-12-03 Thread Tom Lane
Yuri Budilov writes: > The out-of-memory error happens if I also retrieve another JSON Column like > so: > CREATE TABLE csnbi_stg.junk4 > AS > SELECT >json_rmq->>'totalSize' as totalSize, -- this plus array below causes > out of memory error >json_array_elements(json_rmq -> 'ord

Re: JSON out of memory error on PostgreSQL 9.6.x

2017-12-03 Thread John R Pierce
On 12/3/2017 3:18 PM, Yuri Budilov wrote: |CREATETABLEX ASSELECTjson_array_elements(json_rmq ->'orders'::text)ASorderFROMtable_name WHEREblah;| I get out of memory error. are you sure thats a postgres error ?  are you doing this in psql, or what sort of application environment ? how many

Re: JSON out of memory error on PostgreSQL 9.6.x

2017-12-03 Thread Yuri Budilov
raw_jmcn stg_rmq_json_raw WHERE rmq_exchange_name = 'Staging.Salesforce.Order' AND rmq_message_id = 'd1200de2-30b0-4599-bb17-64405f45ca19'; THANK YOU From: Yuri Budilov Sent: Monday, 4 December 2017 11:14 AM To: rob stone; pgsql-general@lists

Re: JSON out of memory error on PostgreSQL 9.6.x

2017-12-03 Thread John R Pierce
On 12/3/2017 3:18 PM, Yuri Budilov wrote: Posted on Stack Overflow, sadly no replies, so trying here ... ,,, why did you email me personally ? -- john r pierce, recycling bits in santa cruz

Re: JSON out of memory error on PostgreSQL 9.6.x

2017-12-03 Thread Yuri Budilov
FWIW. It takes well under 1 min elapsed time to fail. best regards and many thanks for trying to help me From: rob stone Sent: Monday, 4 December 2017 11:01 AM To: Yuri Budilov; John R Pierce; pgsql-general@lists.postgresql.org Subject: Re: JSON out of memor

Re: JSON out of memory error on PostgreSQL 9.6.x

2017-12-03 Thread rob stone
On Sun, 2017-12-03 at 23:18 +, Yuri Budilov wrote: > Posted on Stack Overflow, sadly no replies, so trying here > > CREATE TABLE X AS > SELECT json_array_elements(json_rmq -> 'orders'::text) AS order > FROM table_name > WHERE blah; > I get out of memory error. > > Is there anything I c

Re: JSON out of memory error on PostgreSQL 9.6.x

2017-12-03 Thread Tom Lane
Yuri Budilov writes: > Posted on Stack Overflow, sadly no replies, so trying here > CREATE TABLE X AS > SELECT json_array_elements(json_rmq -> 'orders'::text) AS order > FROM table_name > WHERE blah; > I get out of memory error. > The JSON column is about ~5 MB and it has about ~150,000 array

JSON out of memory error on PostgreSQL 9.6.x

2017-12-03 Thread Yuri Budilov
Posted on Stack Overflow, sadly no replies, so trying here CREATE TABLE X AS SELECT json_array_elements(json_rmq -> 'orders'::text) AS order FROM table_name WHERE blah; I get out of memory error. Is there anything I can do to unpack the above? The JSON column is about ~5 MB and it has abou