Re: Which partition scheme makes sense for my time based IoT-datagrams?

2021-02-10 Thread Laurenz Albe
On Wed, 2021-02-10 at 18:59 +0100, Thorsten Schöning wrote: > > They will scan all partitions. You have to be prepared that most queries > > will become at least slightly slower with partitioning. That is > > expected. > > Does "most queries" address thos efor IDs scanning all partitions or > th

Re: Unable to execute Query in parallel for partitioned table

2021-02-10 Thread Laurenz Albe
On Thu, 2021-02-11 at 05:09 +0530, Brajendra Pratap wrote: > I am unable to execute the below in parallel plz suggest how can I achieve > parallelism here. > > select count(*) over () VIEWALLROWCOUNT,abc.*,ROW_NUMBER() OVER (order by > trn_transaction_date desc ) AS RowNumber from ( > select *

Re: Encryption of Data Specific to a Tenant in PostgreSQL database | General Idea

2021-02-10 Thread Jagmohan Kaintura
Hi Tim, Yeh datatype is even not supported right now. As tenantId is numeric and encryption/decryption would go through text/bytea , so even tougher to encrypt that column data. On Thu, Feb 11, 2021 at 7:38 AM Tim Cross wrote: > > Jagmohan Kaintura writes: > > > HI All, > > > > For POstgreSQL

Re: Encryption of Data Specific to a Tenant in PostgreSQL database | General Idea

2021-02-10 Thread Jagmohan Kaintura
Hi Guyren, Yup right now data is being accessed in this manner only. application access using tenant user only who have specific tenantId in that session and can see its own data only. It doesn't know about anyone else's data and neither can get/fetch. So isolation is 100% guaranteed right now.

Re: Encryption of Data Specific to a Tenant in PostgreSQL database | General Idea

2021-02-10 Thread Guyren Howe
An interesting option is to make your tenants work through views, and have the views work through a variable that contains the tenant’s id. There would be a bit of coding, but it would be the same for every table, so you could automate it easy enough. When you’re done, client software just sets

Re: Encryption of Data Specific to a Tenant in PostgreSQL database | General Idea

2021-02-10 Thread Rob Sargent
> On Feb 10, 2021, at 6:45 PM, Tim Cross wrote: > > > Jagmohan Kaintura mailto:jagmo...@tecorelabs.com>> > writes: > >> HI All, >> >> For POstgreSQL database to store data for multiple tenants, the approach >> decided was to have >> Shared Database (Holding data for all tenants) >> =>

Re: Encryption of Data Specific to a Tenant in PostgreSQL database | General Idea

2021-02-10 Thread Tim Cross
Jagmohan Kaintura writes: > HI All, > > For POstgreSQL database to store data for multiple tenants, the approach > decided was to have > Shared Database (Holding data for all tenants) > => Data would be segregated on basis of some additional column > (tennatid,different tenants having dif

Encryption of Data Specific to a Tenant in PostgreSQL database | General Idea

2021-02-10 Thread Jagmohan Kaintura
HI All, For POstgreSQL database to store data for multiple tenants, the approach decided was to have Shared Database (Holding data for all tenants) => Data would be segregated on basis of some additional column (tennatid,different tenants having different tenantId) => Data would b

PostgreSQL occasionally unable to rename WAL files (NTFS)

2021-02-10 Thread Guy Burgess
Hello, Running 13.1 on Windows Server 2019, I am getting the following log entries occasionally:     2021-02-11 12:34:10.149 NZDT [6072] LOG:  could not rename file "pg_wal/0001009900D3": Permission denied     2021-02-11 12:40:31.377 NZDT [6072] LOG:  could not rename file "pg_wa

Unable to execute Query in parallel for partitioned table

2021-02-10 Thread Brajendra Pratap
Hi, I am unable to execute the below in parallel plz suggest how can I achieve parallelism here. select count(*) over () VIEWALLROWCOUNT,abc.*,ROW_NUMBER() OVER (order by trn_transaction_date desc ) AS RowNumber from ( select * from transactions where trn_store_date_id=20201202) abc; Query plan

Compiler warnings on Debian 10

2021-02-10 Thread Daniel Westermann (DWE)
HI, I was wondering if someone already has seen these warnings on Debian 10 (PostgreSQL 13.1): ../../../../src/include/port.h:176:70: warning: 'format' attribute argument not supported: gnu_printf [-Wignored-attributes] extern int pg_snprintf(char *str, size_t count, const char *fmt,

Re: Which partition scheme makes sense for my time based IoT-datagrams?

2021-02-10 Thread Thorsten Schöning
Guten Tag Laurenz Albe, am Mittwoch, 10. Februar 2021 um 17:45 schrieben Sie: > They will scan all partitions. You have to be prepared that most queries > will become at least slightly slower with partitioning. That is > expected. Does "most queries" address thos efor IDs scanning all partition

Connecting to database through username,passphrase and private key on PgAdmin

2021-02-10 Thread Yambu
Hello May i please know how i can connect to a db server using pgAdmin. How do i use the combination of username,passphrase and private key . Below is when i connect using putty. I then login as postgres user and connect to the database. login as: user1 Authenticating with public key "*

Re: Which partition scheme makes sense for my time based IoT-datagrams?

2021-02-10 Thread Laurenz Albe
On Wed, 2021-02-10 at 16:09 +0100, Thorsten Schöning wrote: > I have a table storing datagrams from some IoT-devices, with one > datagram per device per day most likely for around 75'000 devices > currently. I want to test query performance with a partitioned table > and am interested in the follow

Which partition scheme makes sense for my time based IoT-datagrams?

2021-02-10 Thread Thorsten Schöning
Hi all, I have a table storing datagrams from some IoT-devices, with one datagram per device per day most likely for around 75'000 devices currently. I want to test query performance with a partitioned table and am interested in the following queries mostly: * querying arbitrary datagrams by the

Re: How does Postgres decide if to use additional workers?

2021-02-10 Thread Philip Semanchuk
> On Feb 9, 2021, at 10:52 AM, Thorsten Schöning wrote: > > So, based on which facts does Postgres decide if to use aadditional > workers or not? Can I see those decisions explained somewhere? I don't > see anything in the query plan. Thanks! Hi Thorsten, This is an interesting topic for me t

Re: Increased size of database dump even though LESS consumed storage

2021-02-10 Thread Thorsten Schöning
Guten Tag Thorsten Schöning, am Mittwoch, 10. Februar 2021 um 09:58 schrieben Sie: >> unpartitioned: 6,4 GiB >> half-yearly parts: 4,8 GiB >> yearly parts: 4,8 GiB The above number for `unpartitioned` might be wrong: I've re-created the same database, applied the same data migration like

Re: Increased size of database dump even though LESS consumed storage

2021-02-10 Thread Francisco Olarte
Thorsten: On Wed, Feb 10, 2021 at 12:46 PM Thorsten Schöning wrote: UUppzz, completely missed the second message. > > CREATE INDEX idx_datagram_for_time_window ON datagram USING btree > > (src_meter, captured_at DESC); Depending on the query and data "shape", this may be chosen or not for

Re: Increased size of database dump even though LESS consumed storage

2021-02-10 Thread Francisco Olarte
Thorsten: On Wed, Feb 10, 2021 at 12:36 PM Thorsten Schöning wrote: > Guten Tag Francisco Olarte, > am Mittwoch, 10. Februar 2021 um 10:38 schrieben Sie: > > [...]and partitioning sorts partially > > by date ( or fully if you have made the partitions by range-querying > > via index scan ).[...] >

Re: Insertion time is very high for inserting data in postgres

2021-02-10 Thread Dave Cramer
On Wed, 10 Feb 2021 at 06:11, Niels Jespersen wrote: > >Fra: prachi surangalikar > > > > > >Hello Team, > > >Greetings! > > > > > >We are using Postgres 12.2.1 for fetching per minute data for about 25 > machines but running parallely via a single thread in python. > > >But suddenly the insertio

Re: Increased size of database dump even though LESS consumed storage

2021-02-10 Thread Thorsten Schöning
Guten Tag Francisco Olarte, am Mittwoch, 10. Februar 2021 um 10:38 schrieben Sie: > [...]and partitioning sorts partially > by date ( or fully if you have made the partitions by range-querying > via index scan ).[...] That statement is especially interesting not only because of my dump-size, but

Re: Increased size of database dump even though LESS consumed storage

2021-02-10 Thread Thorsten Schöning
Guten Tag Francisco Olarte, am Mittwoch, 10. Februar 2021 um 10:38 schrieben Sie: > [...]and partitioning sorts partially > by date ( or fully if you have made the partitions by range-querying > via index scan ).[...] That statement is especially interesting not only because of my dump-size, but

SV: Insertion time is very high for inserting data in postgres

2021-02-10 Thread Niels Jespersen
>Fra: prachi surangalikar > >Hello Team, >Greetings! > >We are using Postgres 12.2.1 for fetching per minute data for about 25 >machines but running parallely via a single thread in python. >But suddenly the insertion time has increased to a very high level, about 30 >second for one machine. >We

Re: Regarding Postgres - Insertion Time Getting Increased As Data Volume is getting increased

2021-02-10 Thread cen
On 10. 02. 21 09:14, Rajnish Vishwakarma wrote: Hi Postgres Team, The below are the scenarios which we are dealing with. 1) There are 20 Tables - On an average each having 150 columns. 2) There are 20 Threads Handled by Thread Pool Executor ( here we are using Python's - psycopg2 module / l

Re: Increased size of database dump even though LESS consumed storage

2021-02-10 Thread Francisco Olarte
Thorsten: On Wed, Feb 10, 2021 at 9:58 AM Thorsten Schöning wrote: ... > I've changed the new table "datagram" to be a partitioned one with > partitions containing rows per year and per half-year. Each partition > contains far less rows than before this way and while I only tested > with "--compr

Re: Regarding Postgres - Insertion Time Getting Increased As Data Volume is getting increased

2021-02-10 Thread Daniele Varrazzo
On Wed, 10 Feb 2021, 09:26 Rajnish Vishwakarma, < rajnish.nationfi...@gmail.com> wrote: > Hi Postgres Team, > > The below are the scenarios which we are dealing with. > > 1) There are 20 Tables - On an average each having 150 columns. > > 2) There are 20 Threads Handled by Thread Pool Executor ( h

Re: Increased size of database dump even though LESS consumed storage

2021-02-10 Thread Thorsten Schöning
Guten Tag Thorsten Schöning, am Dienstag, 9. Februar 2021 um 10:56 schrieben Sie: > Any explanation for the increased dump size? Thanks! While I don't have a detailed explanation myself, there's the following hint [in the docs][1]: > For the custom archive format, this specifies compression of

Re: Insertion time is very high for inserting data in postgres

2021-02-10 Thread Ganesh Korde
On Wed, 10 Feb 2021, 1:56 pm prachi surangalikar, < surangalikarprachi...@gmail.com> wrote: > Hello Team, > Greetings! > > We are using Postgres 12.2.1 for fetching per minute data for about 25 > machines but running parallely via a single thread in python. > But suddenly the insertion time has in

Regarding Postgres - Insertion Time Getting Increased As Data Volume is getting increased

2021-02-10 Thread Rajnish Vishwakarma
Hi Postgres Team, The below are the scenarios which we are dealing with. 1) There are 20 Tables - On an average each having 150 columns. 2) There are 20 Threads Handled by Thread Pool Executor ( here we are using Python's - psycopg2 module / library to fetch the data .) 3) I am using the below

Insertion time is very high for inserting data in postgres

2021-02-10 Thread prachi surangalikar
Hello Team, Greetings! We are using Postgres 12.2.1 for fetching per minute data for about 25 machines but running parallely via a single thread in python. But suddenly the insertion time has increased to a very high level, about 30 second for one machine. We are in so much problem as the data fet