PostgreSQL Database Upgrade

2020-12-07 Thread github kran
Hello PostgreSQL Team, We have recently upgraded the postgreSQL version from V9 to V10. - Is the below command right to update the database statistics after the database upgrade ?. - How do I verify in PG_STATS if the Analyze ran successfully? Does it show any time when the last Analy

Re: AutoVacuum and growing transaction XID's

2020-05-12 Thread github kran
Thanks for yous suggestions Michael and David. On Fri, May 8, 2020 at 4:11 PM Michael Lewis wrote: > autovacuum_naptime being only 5 seconds seems too frequent. A lock_timeout > might be 1-5 seconds depending on your system. Usually, DDL can fail and > wait a little time rather than lock the ta

Re: AutoVacuum and growing transaction XID's

2020-05-07 Thread github kran
On Thu, May 7, 2020 at 11:04 PM David Rowley wrote: > On Fri, 8 May 2020 at 13:51, github kran wrote: > > I can't either DROP or ALTER any other tables ( REMOVE Inheritance > for any of old tables where the WRITES are not getting written to). Any of > the ALTER TABLE

Re: AutoVacuum and growing transaction XID's

2020-05-07 Thread github kran
Thanks David for your replies. On Thu, May 7, 2020 at 11:01 PM David Rowley wrote: > On Fri, 8 May 2020 at 09:18, github kran wrote: > > 1) We haven't changed anything related to autovacuum except a work_mem > parameter which was increased to 4 GB which I believe

Re: AutoVacuum and growing transaction XID's

2020-05-07 Thread github kran
On Thu, May 7, 2020 at 4:18 PM github kran wrote: > > > On Thu, May 7, 2020 at 1:33 PM Michael Lewis wrote: > >> It is trying to do a vacuum freeze. Do you have autovacuum turned off? >> Any settings changed from default related to autovacuum? >> >> https:/

Re: AutoVacuum and growing transaction XID's

2020-05-07 Thread github kran
On Thu, May 7, 2020 at 1:33 PM Michael Lewis wrote: > It is trying to do a vacuum freeze. Do you have autovacuum turned off? Any > settings changed from default related to autovacuum? > > https://www.postgresql.org/docs/9.6/routine-vacuuming.html > Read 24.1.5. Preventing Transaction ID Wraparoun

AutoVacuum and growing transaction XID's

2020-05-07 Thread github kran
Hello Team, We are using a PostgreSQL version -9.6.12 version and from last 4 weeks our Transaction ID's (XID's) have increased by 195 million to 341 million transactions. I see the below from pg_stat_activity from the postGreSQL DB. 1) Viewing the pg_stat-activity I noticed that the vacuum qu

Re: Need support on tuning at the time of index creation

2020-01-27 Thread github kran
On Mon, Jan 27, 2020 at 5:27 AM Ron wrote: > On 1/27/20 5:10 AM, Sandip Pradhan wrote: > > Dear Sir/Madam, > > > > One of our ericsson product used backend db as postgresql 9. We are > facing > > following performance issues where we need some support from your side. > > We are having 10 tables a

Re: Upgrade PostgreSQL 9.6 to 10.6

2020-01-09 Thread github kran
On Wed, Jan 8, 2020 at 11:03 PM Michael Lewis wrote: > On Wed, Jan 8, 2020 at 8:52 PM github kran wrote: > >> You are right on RDS but I believe the problem is on Aurora PostgreSQL >> where the pglogical throws an error during installation. Are you aware if >> this wor

Re: Upgrade PostgreSQL 9.6 to 10.6

2020-01-08 Thread github kran
On Wed, Jan 8, 2020 at 5:57 PM Christophe Pettus wrote: > > > > On Jan 8, 2020, at 15:56, github kran wrote: > > > > Great I see its supported based on the link but the problem is we are > locked into a Aurora RDS and we can't use logical replication on that >

Re: Upgrade PostgreSQL 9.6 to 10.6

2020-01-08 Thread github kran
er we do the pg_dump and pg_restore ?. On Wed, Jan 8, 2020 at 10:25 AM Michael Lewis wrote: > On Wed, Jan 8, 2020 at 7:20 AM github kran wrote: > >> Sorry for the confusion, I tried to install this extension on 9.6 and it >> seems to be not working. Does 9.6 PostGreSQL suppo

Re: Upgrade PostgreSQL 9.6 to 10.6

2020-01-08 Thread github kran
Sorry for the confusion, I tried to install this extension on 9.6 and it seems to be not working. Does 9.6 PostGreSQL supports logical replication ? On Tue, Jan 7, 2020 at 4:56 PM github kran wrote: > > On Tue, Dec 10, 2019 at 2:53 AM Andreas Kretschmer < > andr...@a-kretschm

Re: Upgrade PostgreSQL 9.6 to 10.6

2020-01-07 Thread github kran
On Tue, Dec 10, 2019 at 2:53 AM Andreas Kretschmer wrote: > > > Am 09.12.19 um 23:37 schrieb github kran: > > Great, thanks Andreas, So this seems to be a good feature using the > > core concept of replication. Can I use this extension and do the major > > upgrade with

Re: Upgrade PostgreSQL 9.6 to 10.6

2019-12-09 Thread github kran
Great, thanks Andreas, So this seems to be a good feature using the core concept of replication. Can I use this extension and do the major upgrade without paying ?. On Mon, Dec 9, 2019 at 10:02 AM Andreas Kretschmer wrote: > > > Am 09.12.19 um 16:43 schrieb github kran: > > H

Upgrade PostgreSQL 9.6 to 10.6

2019-12-09 Thread github kran
Hello PostgreSQL Team, I would like to know what would be the best way to do Database migration from PostgreSQL 9.6 engine to 10.6 by creating a new cluster in 10.6 and then copy data. Size of the cluster is 3.8 TB. 1) It would be a new cluster we want to create on 10.6 and then copy data form 9.

Re: PostGreSQL Replication and question on maintenance

2019-11-26 Thread github kran
me few resources if there is a better solution for this problem. Thanks githubkran On Sat, Nov 16, 2019 at 11:13 AM Jeff Janes wrote: > On Thu, Nov 14, 2019 at 12:23 PM github kran wrote: > >> >>> >>> *Problem what we have right now. * >>> >>&g

Re: PostGreSQL Replication and question on maintenance

2019-11-16 Thread github kran
Any reply on this please ?. On Fri, Nov 15, 2019 at 9:10 AM github kran wrote: > > > On Thu, Nov 14, 2019 at 11:42 PM Pavel Stehule > wrote: > >> these numbers looks crazy high - how much memory has your server - more >> than 1TB? >> > > The cluster go

Re: PostGreSQL Replication and question on maintenance

2019-11-15 Thread github kran
On Thu, Nov 14, 2019 at 11:42 PM Pavel Stehule wrote: > these numbers looks crazy high - how much memory has your server - more > than 1TB? > The cluster got 244 GB of RAM and storage capacity it has is 64 TB. > > > pá 15. 11. 2019 v 6:26 odesílatel github kran >

Fwd: PostGreSQL Replication and question on maintenance

2019-11-14 Thread github kran
> Hello postGreSQL Community , > > > > Hope everyone is doing great !!. > > > *Background* > > We use PostgreSQL Version 10.6 version and heavily use PostgreSQL for our > day to day activities to write and read data. We have 2 clusters running > PostgreSQL engine , one cluster > > keeps data up to

PostGreSQL Replication and question on maintenance

2019-11-14 Thread github kran
sorry changing the subject line. On Thu, Nov 14, 2019 at 11:21 AM github kran wrote: > Hello Team, > > > > Hope everyone is doing great !!. > > > *Background* > > We use PostgreSQL Version 10.6 version and heavily use PostgreSQL for our > day to day activities

Re: Postgresql RDS DB Latency Chossing Hash join Plan

2019-11-14 Thread github kran
Hello Team, Hope everyone is doing great !!. *Background* We use PostgreSQL Version 10.6 version and heavily use PostgreSQL for our day to day activities to write and read data. We have 2 clusters running PostgreSQL engine , one cluster keeps data up to 60 days and another cluster retains da

Archival process of partition tables with filtering few rows from tables.

2019-03-28 Thread github kran
Hello Team, We are using PostgreSQL Version 9.6 and planning to archive our partition tables containing about 300 - 500 million rows . We have around ~ 50 partition tables to be archived to a new cold path PostgreSQL database , version 10.6. We have a requirement to filter few rows before exportin

Re: Tools to migrate data from Json files to PostgreSQL DB.

2019-03-11 Thread github kran
Not terribly difficult to do, especially if you are not navigating complex > JSON structures in pl/pgsql to strip out the data. > > Plenty of python JSON examples out there. Less so for PG :) > > On Thu, 7 Mar 2019 at 23:21, github kran wrote: > >> Hello PostgreSQL Team, >

Tools to migrate data from Json files to PostgreSQL DB.

2019-03-07 Thread github kran
Hello PostgreSQL Team, Are there are any tools to migrate data present in the json files ? to the postgreSQL database. We have data in flat files about 2 billion records across multiple files. 1) What is the easiest way I can transfer this data to relational database ?. 2) Any tools I can use ?.

Re: cannot execute VACUUM during recovery

2019-03-02 Thread github kran
erName This option had a .pgpass file at the root in this linux instance and I did not pass any password here, the *VACUUM on the table Iam running works.* *pgpass is setup here and contains -> clusterName:5432:myPostgresDB:myUserName:myDBPassword* Thanks - Kran. > On Feb 27, 2019, 6:43

cannot execute VACUUM during recovery

2019-02-27 Thread github kran
Hello Team, We are using a PostgreSQL 9.6 and seeing the below error while trying to run a VACUUM on one of our live tables running in Production. We wanted to clean up some DEAD tuples on the table. *Command*: VACUUM (ANALYZE,VERBOSE) table_name. ERROR: cannot execute VACUUM during recovery

PostgreSQL DB Maintenance and Partitioning to keep data longer.

2019-02-25 Thread github kran
Hello pgsql-General Team, We have currently using Postgresql DB which is growing about 1.4 billion records/month with a total of about 16 to 17 billion records/year. The DB storage is growing about 6.8 TB/year including all indexes and data. Current total DB Storage we got is 60 TB. *Use ca

Re: How many billion rows of data I can store in PostgreSQL RDS.

2019-02-22 Thread github kran
On Fri, Feb 22, 2019 at 5:48 AM Samuel Teixeira Santos wrote: > Just adding that my case it's not a Amazon RDS, it's common server, if I > can say like that... > Aplologies I missed the point to mention that this is a question to PostgreSQL community. We are currently using PostgreSQL. ( Aurora

How many billion rows of data I can store in PostgreSQL RDS.

2019-02-21 Thread github kran
Hello Pgsql-General, We have currently have around 6 TB of data and have plans to move some historic datainto RDS of about close to 1 TB of data. The total rows in partitioned tables is around 6 billion rows today and have plans to keep the data long term which would be around 5-8 billion rows per

Re: Postgresql RDS DB Latency Chossing Hash join Plan

2019-02-21 Thread github kran
Your_PostgreSQL_Server > > https://www.postgresql.org/docs/current/runtime-config-resource.html#GUC-WORK-MEM > > I'm glad your performance is improved in AWS/dev environment. It can be a > big hassle to test things in an environment that performs significantly > different. > &

Re: Postgresql RDS DB Latency Chossing Hash join Plan

2019-02-17 Thread github kran
?. 2) Apart from analyzing the tables do I need to do any other changes with the statistics setting ? 3) Also the current work mem is set to 4 MB and we didnt play with this value so far. For future needs can I increase the WORK MEM setting ?. Appreciate your reply. Thanks > On Thu,

Re: Postgresql RDS DB Latency Chossing Hash join Plan

2019-02-14 Thread github kran
the data longer if there is a regular maintenance like VACUUM and other maintenace activities > > On Thu, Feb 14, 2019 at 8:48 AM github kran wrote: > >> >> >> On Wed, Feb 13, 2019 at 11:38 AM Michael Lewis >> wrote: >> >>> I didn't see yo

Re: Postgresql RDS DB Latency Chossing Hash join Plan

2019-02-14 Thread github kran
illion rows and then did VACUUM and re-indexed the tables. I cant think of anything we can do here but let us know if you need any more details on this problem. Iam happy to share more details. > > On Wed, Feb 13, 2019 at 8:49 AM github kran wrote: > >> >> >> On Tue, F

Re: Postgresql RDS DB Latency Chossing Hash join Plan

2019-02-13 Thread github kran
On Tue, Feb 12, 2019 at 12:55 PM github kran wrote: > > > On Tue, Feb 12, 2019 at 12:33 PM Michael Lewis wrote: > >> Did you update the stats by running ANALYZE on the tables involved, or >> perhaps the entire database on the 'Non prod Aurora RDS instance'?

Postgresql RDS DB Latency Chossing Hash join Plan

2019-02-12 Thread github kran
On Tue, Feb 12, 2019 at 7:07 AM github kran wrote: > > > On Mon, Feb 11, 2019 at 6:00 PM github kran wrote: > >> >> >> On Mon, Feb 11, 2019 at 3:29 PM Michael Lewis wrote: >> >>> Are default statistics target the same on both prod and AWS? Have y

Re: Aurora Postgresql RDS DB Latency

2019-02-12 Thread github kran
On Mon, Feb 11, 2019 at 6:00 PM github kran wrote: > > > On Mon, Feb 11, 2019 at 3:29 PM Michael Lewis wrote: > >> Are default statistics target the same on both prod and AWS? Have you >> analyzed all tables being used in this query to ensure stats are up proper? >>

Re: Aurora Postgresql RDS DB Latency

2019-02-11 Thread github kran
databases ?. ( Also the DB instance size is same but there is little more data in the Non prod Aurora RDS instance compared to Prod instance). Query used. >select * from pg_stats where tablename = 'tableName' > > > On Mon, Feb 11, 2019 at 2:15 PM github kran wrote: >

Aurora Postgresql RDS DB Latency

2019-02-11 Thread github kran
Hi Postgres Team, We are using AWS Aurora RDS Postgres DB 9.6.8 Engine. We recently deleted few million rows from the database and ran into a issue in one of our dev account where the DB was not normal after this deletion. We did re index, vacuuming entire database but we couldnt bring it to the sa

Re: Pulling data from Postgres DB table for every 5 seconds.

2019-01-09 Thread github kran
On Wed, Jan 9, 2019 at 12:36 PM Mark Fletcher wrote: > On Wed, Jan 9, 2019 at 10:10 AM github kran wrote: > >> Mark - We are currently on 9.6 version of postgres and cant use this >> feature of logical replication.Answering to your question we are looking >> for any cha

Re: Pulling data from Postgres DB table for every 5 seconds.

2019-01-09 Thread github kran
On Wed, Jan 9, 2019 at 12:26 PM Rob Sargent wrote: > > > On Jan 9, 2019, at 11:11 AM, github kran wrote: > > Rob - It's a Java based application. We dont have triggers yet on the > table and is trigger a only option in 9.6 version ?. > > On Wed, Jan 9, 2019 a

Re: Pulling data from Postgres DB table for every 5 seconds.

2019-01-09 Thread github kran
Rob - It's a Java based application. We dont have triggers yet on the table and is trigger a only option in 9.6 version ?. On Wed, Jan 9, 2019 at 12:01 PM Rob Sargent wrote: > > On 1/9/19 10:21 AM, github kran wrote: > > Thanks for your reply Rob. Reading the below documentat

Re: Pulling data from Postgres DB table for every 5 seconds.

2019-01-09 Thread github kran
a specific product entity). Any other alternatives ?. On Wed, Jan 9, 2019 at 11:24 AM Mark Fletcher wrote: > On Wed, Jan 9, 2019 at 9:02 AM github kran wrote: > >> >>> Hi Postgres Team, >>> >>> I have an application using RDS Aurora Postgresql 9.6 version

Re: Pulling data from Postgres DB table for every 5 seconds.

2019-01-09 Thread github kran
wrote: > > > On Jan 9, 2019, at 10:02 AM, github kran wrote: > > >> Hi Postgres Team, >> >> I have an application using RDS Aurora Postgresql 9.6 version having 4 TB >> of DB size. In this DB we have a table PRODUCT_INFO with around 1 million >> r

Re: Pulling data from Postgres DB table for every 5 seconds.

2019-01-09 Thread github kran
> > > Hi Postgres Team, > > I have an application using RDS Aurora Postgresql 9.6 version having 4 TB > of DB size. In this DB we have a table PRODUCT_INFO with around 1 million > rows and table size of 1 GB. > We are looking for a implementation where we want to pull the data in real > time for e