Re: [GENERAL] bytea columns and large values

2011-10-01 Thread Dmitriy Igrishin
2011/10/2 Merlin Moncure > On Sat, Oct 1, 2011 at 4:27 AM, Dmitriy Igrishin > wrote: > > Hey Merlin, > > > >> The lo interface sucks but it's slightly better on resources for > >> really huge bytea and tends to be more consistently implemented in > >> database drivers. If I was doing this, I wo

Re: [GENERAL] bytea columns and large values

2011-10-01 Thread Merlin Moncure
On Sat, Oct 1, 2011 at 4:27 AM, Dmitriy Igrishin wrote: > Hey Merlin, > >> The lo interface sucks but it's slightly better on resources for >> really huge bytea and tends to be more consistently implemented in >> database drivers.  If I was doing this, I would of course be crafting >> a carefully

Re: [GENERAL] bytea columns and large values

2011-10-01 Thread Dmitriy Igrishin
Hey Merlin, The lo interface sucks but it's slightly better on resources for > really huge bytea and tends to be more consistently implemented in > database drivers. If I was doing this, I would of course be crafting > a carefully generated client in C, using libpqtypes, which is the gold > stand

Re: [GENERAL] bytea columns and large values

2011-09-29 Thread Merlin Moncure
On Thu, Sep 29, 2011 at 10:54 AM, Jon Nelson wrote: > On Thu, Sep 29, 2011 at 10:51 AM, David North wrote: > >> I'll also apply GZip to keep the amount of data in any given bytea/lob as >> small as possible. > > Aren't bytea fields compressed by postgresql anyway (when EXTENDED or > MAIN is used)

Re: [GENERAL] bytea columns and large values

2011-09-29 Thread Jon Nelson
On Thu, Sep 29, 2011 at 10:51 AM, David North wrote: > I'll also apply GZip to keep the amount of data in any given bytea/lob as > small as possible. Aren't bytea fields compressed by postgresql anyway (when EXTENDED or MAIN is used) (by default). http://www.postgresql.org/docs/8.4/static/stora

Re: [GENERAL] bytea columns and large values

2011-09-29 Thread David North
On 29/09/11 14:55, Merlin Moncure wrote: On Thu, Sep 29, 2011 at 6:12 AM, Radosław Smogura wrote: On Wed, 28 Sep 2011 10:18:27 -0500, Merlin Moncure wrote: sending ~1GB bytea values is borderline crazy, and is completely crazy if you are not absolutely sure the transmission is not 100% binary

Re: [GENERAL] bytea columns and large values

2011-09-29 Thread Marti Raudsepp
On Thu, Sep 29, 2011 at 15:48, Jon Nelson wrote: > especially if my attempt > to do so kills the backend I am using (which triggers a shutdown of > all other backends, no?). No, this is just an ereport(ERROR) that's handled gracefully by rolling back the transaction. Regards, Marti -- Sent via

Re: [GENERAL] bytea columns and large values

2011-09-29 Thread Merlin Moncure
On Thu, Sep 29, 2011 at 6:12 AM, Radosław Smogura wrote: > On Wed, 28 Sep 2011 10:18:27 -0500, Merlin Moncure wrote: >> sending ~1GB bytea values is borderline crazy, and is completely crazy >> if you are not absolutely sure the transmission is not 100% binary.  I >> don't know if the JDBC sends/r

Re: [GENERAL] bytea columns and large values

2011-09-29 Thread Jon Nelson
On Thu, Sep 29, 2011 at 7:30 AM, Alban Hertroys wrote: > On 29 September 2011 13:12, Radosław Smogura wrote: >>> sending ~1GB bytea values is borderline crazy, and is completely crazy >>> if you are not absolutely sure the transmission is not 100% binary.  I >>> don't know if the JDBC sends/recei

Re: [GENERAL] bytea columns and large values

2011-09-29 Thread Alban Hertroys
On 29 September 2011 13:12, Radosław Smogura wrote: >> sending ~1GB bytea values is borderline crazy, and is completely crazy >> if you are not absolutely sure the transmission is not 100% binary.  I >> don't know if the JDBC sends/receives bytea as binary, but it may not. >>  If not, you might ha

Re: [GENERAL] bytea columns and large values

2011-09-29 Thread Radosław Smogura
On Wed, 28 Sep 2011 10:18:27 -0500, Merlin Moncure wrote: On Wed, Sep 28, 2011 at 3:28 AM, David North wrote: On 28/09/11 01:50, Craig Ringer wrote: On 09/28/2011 01:01 AM, David North wrote: testdb=# select * from problem_table; ERROR: invalid memory alloc request size 2003676411 What Pg

Re: [GENERAL] bytea columns and large values

2011-09-29 Thread Marti Raudsepp
On Tue, Sep 27, 2011 at 20:01, David North wrote: > testdb=# select * from problem_table; > ERROR:  invalid memory alloc request size 2003676411 > Is there some reason why my data can be stored in <1GB but triggers the > allocation of 2GB of memory when I try to read it back? Is there any setting

Re: [GENERAL] bytea columns and large values

2011-09-28 Thread Merlin Moncure
On Wed, Sep 28, 2011 at 3:28 AM, David North wrote: > On 28/09/11 01:50, Craig Ringer wrote: >> >> On 09/28/2011 01:01 AM, David North wrote: >> >>> testdb=# select * from problem_table; >>> ERROR: invalid memory alloc request size 2003676411 >> >> What Pg version are you using? >> >> On which pla

Re: [GENERAL] bytea columns and large values

2011-09-28 Thread David North
On 28/09/11 01:50, Craig Ringer wrote: On 09/28/2011 01:01 AM, David North wrote: testdb=# select * from problem_table; ERROR: invalid memory alloc request size 2003676411 What Pg version are you using? On which platform? 32-bit or 64-bit OS? If 64-bit, with a 32-bit or 64-bit build of Pos

Re: [GENERAL] bytea columns and large values

2011-09-27 Thread Craig Ringer
On 09/28/2011 01:01 AM, David North wrote: testdb=# select * from problem_table; ERROR: invalid memory alloc request size 2003676411 What Pg version are you using? On which platform? 32-bit or 64-bit OS? If 64-bit, with a 32-bit or 64-bit build of PostgreSQL? -- Craig Ringer -- Sent via pg

[GENERAL] bytea columns and large values

2011-09-27 Thread David North
My application uses a bytea column to store some fairly large binary values (hundreds of megabytes). Recently I've run into a problem as my values start to approach the 1GB limit on field size: When I write a 955MB byte array from Java into my table from JDBC, the write succeeds and the numb