Re: xmin and very high number of concurrent transactions

2019-03-13 Thread Laurenz Albe
Vijaykumar Jain wrote: > I was asked this question in one of my demos, and it was interesting one. > > we update xmin for new inserts with the current txid. > now in a very high concurrent scenario where there are more than 2000 > concurrent users trying to insert new data, > will updating xmin va

Re: xmin and very high number of concurrent transactions

2019-03-13 Thread Julien Rouhaud
On Wed, Mar 13, 2019 at 9:50 AM Laurenz Albe wrote: > > Vijaykumar Jain wrote: > > I was asked this question in one of my demos, and it was interesting one. > > > > we update xmin for new inserts with the current txid. > > now in a very high concurrent scenario where there are more than 2000 > > c

ERROR: XX000: cannot update SecondarySnapshot during a parallel operation

2019-03-13 Thread fuzk
Dear Sir/Madam I got an error when I execute the following select sentence. Would you please solve the problem for me? Thank you . Alan Fu. postgres=# \set VERBOSITY verbose postgres=# SELECT round(cast(coalesce(sum(ST_length(geography(geometry)))/1000,0) as NUMERIC),4)||'KM' field_value fr

PostgreSQL temp table blues

2019-03-13 Thread Jahwan Kim
Hi all, I'd like to share my (painful) experience, in which temp tables caused PostgreSQL shutdown. TL;DR. Do not use temp tables in PostgreSQL with connection pool. * My app uses connection pool AND temp tables, with default setting of ON COMMIT PRESERVE ROWS. * I found out later that autovacuu

PG 10 vs. 11: Large increase in memory usage when selecting BYTEA data (actually out of memory with PG11)

2019-03-13 Thread Matthias Otterbach
Dear mailing list, I am currently testing an application for which I previously used PostgreSQL 10 with the current PostgreSQL 11.2 release. During the tests I experienced out of memory errors of my database which I could not explain, they seem to affect tables containing large BYTEA data. I w

Permission to refresh materialized view

2019-03-13 Thread Johann Spies
We did run this query: *GRANT ALL ON ALL TABLES IN SCHEMA X TO USER Y;* But user Y gets the message that he has to be the owner of a materialized view to be able to refresh it. Is that intended behaviour? Is there a way to enable the user to refresh materialized views in that schema? Regards J

Re: Permission to refresh materialized view

2019-03-13 Thread Adrian Klaver
On 3/13/19 6:27 AM, Johann Spies wrote: We did run this query: /GRANT ALL ON ALL TABLES IN SCHEMA X TO USER Y;/ / / But user Y gets the message that he has to be the owner of a materialized view to be able to refresh it. What is the exact message? Is that intended behaviour?  Is there a wa

Where to store Blobs?

2019-03-13 Thread Thomas Güttler
Some days ago I asked "Where **not** to use PostgreSQL?" on this mailing list. Now I realized: Nobody talked about Blobs. I guess most people do not store Blobs in PostgresSQL. Where do you store Blobs? (In my case Blobs are PDF/image files with size up to 20 MByte. I do not talk about very bi

Re: ERROR: XX000: cannot update SecondarySnapshot during a parallel operation

2019-03-13 Thread Adrian Klaver
On 3/12/19 7:54 PM, fuzk wrote: Dear Sir/Madam I got an error when I execute the following select sentence. Would you please solve the problem for me? What version of Postgres? Thank you . Alan Fu. postgres=# \set VERBOSITY verbose postgres=# SELECT round(cast(coalesce(sum(ST_length(geogr

Re: Where to store Blobs?

2019-03-13 Thread Adrian Klaver
On 3/13/19 7:28 AM, Thomas Güttler wrote: Some days ago I asked "Where **not** to use PostgreSQL?" on this mailing list. Now I realized: Nobody talked about Blobs. I guess most people do not store Blobs in PostgresSQL. Where do you store Blobs? Probably the preferred method: https://www.pos

Re: Where to store Blobs?

2019-03-13 Thread Laurenz Albe
Thomas Güttler wrote: > Now I realized: Nobody talked about Blobs. > > I guess most people do not store Blobs in PostgresSQL. > > Where do you store Blobs? > > (In my case Blobs are PDF/image files with size up to 20 MByte. > I do not talk about very big blobs which are several hundret MByte) I

Re: Where to store Blobs?

2019-03-13 Thread Ron
On 3/13/19 9:28 AM, Thomas Güttler wrote: Some days ago I asked "Where **not** to use PostgreSQL?" on this mailing list. Now I realized: Nobody talked about Blobs. I guess most people do not store Blobs in PostgresSQL. Where do you store Blobs? (In my case Blobs are PDF/image files with size

Re: Where to store Blobs?

2019-03-13 Thread Chuck Martin
I store them as bytea in the database despite the fact that there are benefits to storing them in the file system. The reason is that it is easier to secure access to the database than to secure both the database and provide secure access to the file system. Chuck Martin Avondale Software On Wed

Re: Where to store Blobs?

2019-03-13 Thread Christopher Browne
On Wed, 13 Mar 2019 at 10:27, Thomas Güttler wrote: > I guess most people do not store Blobs in PostgresSQL. > > Where do you store Blobs? Things have changed, but at one time, we were using RT as our ticketing system (https://bestpractical.com/request-tracker) and it would capture documents as d

Re: Where to store Blobs?

2019-03-13 Thread Karsten Hilbert
On Wed, Mar 13, 2019 at 11:50:37AM -0400, Christopher Browne wrote: >> I guess most people do not store Blobs in PostgresSQL. - BYTEA puts practical limits on size - LO storage happens inside the system (!) table Nowadays, there are Foreign Data Wrappers which might encapsulate files as if they

Re: Where to store Blobs?

2019-03-13 Thread Achilleas Mantzios
On 13/3/19 4:28 μ.μ., Thomas Güttler wrote: Some days ago I asked "Where **not** to use PostgreSQL?" on this mailing list. Where do you store Blobs? (In my case Blobs are PDF/image files with size up to 20 MByte. I do not talk about very big blobs which are several hundret MByte) bytea. Keepin

RE: Permission to refresh materialized view

2019-03-13 Thread Day, David
-Original Message- From: Adrian Klaver [mailto:adrian.kla...@aklaver.com] Sent: Wednesday, March 13, 2019 10:24 AM To: Johann Spies ; pgsql-gene...@postgresql.org Subject: Re: Permission to refresh materialized view On 3/13/19 6:27 AM, Johann Spies wrote: > We did run this query: > > /

Re: Where to store Blobs?

2019-03-13 Thread Hannes Erven
Hi, Am 13.03.19 um 15:28 schrieb Thomas Güttler: Where do you store Blobs? Within PostgreSQL, of course. The system I have in mind stores ZIP and PDF files, usually a few MBs each; we're currently at a total of about 100 GB and there are no evident problems. For this application, it is

Re: Where to store Blobs?

2019-03-13 Thread Jamesie Pic
Make dump/restore of database data unnecessarily expensive in terms of time and space imho.

varlena objects greater than 1GB

2019-03-13 Thread Michel Pelletier
Hello, I have read through this thread started by pg-strom's Kohei KaiGai: https://www.postgresql.org/message-id/CADyhKSUP0PM6odyoV27q8CFG6mxMQFKY4R7XLz2NsXCHuwRZKA%40mail.gmail.com and have a similar need to Kohei for varlena objects greater than 1GB, in my case, also vector/matrix objects stor

Re: [External] Re: xmin and very high number of concurrent transactions

2019-03-13 Thread Vijaykumar Jain
Thank you everyone for responding. Appreciate your help. Looks like I need to understand the concepts a little more in detail , to be able to ask the right questions, but atleast now I can look at the relevant docs. On Wed, 13 Mar 2019 at 2:44 PM Julien Rouhaud wrote: > On Wed, Mar 13, 2019 a

Re: Autovacuum Transaction Wraparound

2019-03-13 Thread Adrian Klaver
On 3/11/19 1:24 PM, Perumal Raj wrote: Hi Adrian What was the full message?            autovacuum: VACUUM (to prevent wraparound) Though i am running vacuum manually (nowadays) and autovacuum is running perfectly once its threshold reaches. What will happen if my DB reaches 200M transactio

Re: Where to store Blobs?

2019-03-13 Thread Benedict Holland
I store large models in the database because I need to have a historical data to compare to. That said, I could probably also automate a git repo but it will be just that much more work and git with binary files really doesn't make sense. Storage is really cheap and I assume the database stores byt

Re: Where to store Blobs?

2019-03-13 Thread Tim Cross
I don't think there is a suitable 'one size fits all' answer to this question. A lot will depend on how you intend to use the blobs and what sort of hardware architecture, especially storage systems, you have. At first glance, sticking everything in the DB seems like an easy choice. However, tha

Re: PostgreSQL temp table blues

2019-03-13 Thread Rene Romero Benavides
Wow, thanks for sharing your experience. What kind of connection pooling are we talking about? some connection pools implement a DISCARD ALL statement after a session close, that may help if possible to configure. On Wed, Mar 13, 2019 at 4:21 AM Jahwan Kim wrote: > Hi all, > > > I'd like to shar

Re: Notification or action when WAL archives fully restored and streaming replication started

2019-03-13 Thread Michael Cassaniti
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi, I've got master/slave replication setup between a few hosts. At any point a slave could become a master. I've got appropriate locking in place using an external system so that only one master can exist at a time. I'm having trouble determining

Re: Notification or action when WAL archives fully restored and streaming replication started

2019-03-13 Thread Michael Paquier
On Thu, Mar 14, 2019 at 02:59:38PM +1100, Michael Cassaniti wrote: > I've got master/slave replication setup between a few hosts. At any > point a slave could become a master. I've got appropriate locking in > place using an external system so that only one master can exist at a > time. I'm having

Re: LDAP authenticated session terminated by signal 11: Segmentation fault, PostgresSQL server terminates other active server processes

2019-03-13 Thread Thomas Munro
On Thu, Mar 7, 2019 at 4:19 PM Noah Misch wrote: > Has anyone else reproduced this? I tried, but could not reproduce this problem on "CentOS Linux release 7.6.1810 (Core)" using OpenLDAP "2.4.44-21.el7_6" (same as Mike reported, what yum install is currently serving up). I tried "make check" in

Re: LDAP authenticated session terminated by signal 11: Segmentation fault, PostgresSQL server terminates other active server processes

2019-03-13 Thread Noah Misch
On Thu, Mar 14, 2019 at 05:18:49PM +1300, Thomas Munro wrote: > On Thu, Mar 7, 2019 at 4:19 PM Noah Misch wrote: > > Has anyone else reproduced this? > > I tried, but could not reproduce this problem on "CentOS Linux release > 7.6.1810 (Core)" using OpenLDAP "2.4.44-21.el7_6" (same as Mike > repo

Re: PostgreSQL temp table blues

2019-03-13 Thread Rene Romero Benavides
In conjunction with some parameter to renew idle connections and those that have been opened for too long will help you prevent this in the future, this also helps prevent server processes from becoming too big memory wise. On Wed, Mar 13, 2019 at 4:32 PM Rene Romero Benavides < rene.romer...@gma

Re: Notification or action when WAL archives fully restored and streaming replication started

2019-03-13 Thread Michael Cassaniti
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 14/3/19 3:10 pm, Michael Paquier wrote: > On Thu, Mar 14, 2019 at 02:59:38PM +1100, Michael Cassaniti wrote: >> I've > got master/slave replication setup between a few hosts. At any >> point a slave could become a master. I've got appropriate l

Re: [External] Re: PostgreSQL temp table blues

2019-03-13 Thread Vijaykumar Jain
May be I am wrong here, but is it not the classic case of connections open too long idle in TX and xid wraparound ? How is connection pool (and which one ?) adding to the woes? I mean the same can be a problem with direct connections too right ? We use pgbouncer with mostly TX level pooling which