[HACKERS] WAL consistency check facility

2016-08-22 Thread Kuntal Ghosh
r4vxdkijp+du82vocongmvutq-gfqiu2dsh4bsm77...@mail.gmail.com Please let me know your thoughts on this. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index f13f9c1..9380079 100644 --- a/src/

Re: [HACKERS] WAL consistency check facility

2016-08-22 Thread Kuntal Ghosh
Yes, I've verified the outputs and log contents after running gmake installcheck and gmake installcheck-world. The status of the test was marked as pass for all the testcases. On Mon, Aug 22, 2016 at 9:26 PM, Simon Riggs wrote: > On 22 August 2016 at 13:44, Kuntal Ghosh wrote: > &g

Re: [HACKERS] WAL consistency check facility

2016-08-25 Thread Kuntal Ghosh
ion technique. On Wed, Aug 24, 2016 at 2:14 PM, Simon Riggs wrote: > On 22 August 2016 at 16:56, Simon Riggs wrote: >> On 22 August 2016 at 13:44, Kuntal Ghosh wrote: >> >>> Please let me know your thoughts on this. >> >> Do the regression tests pass with t

Re: [HACKERS] WAL consistency check facility

2016-08-25 Thread Kuntal Ghosh
set to blkno and xlrec->offnum. I think this is why I was getting the above warning. On Thu, Aug 25, 2016 at 10:33 PM, Alvaro Herrera wrote: > Kuntal Ghosh wrote: > >> 4. For Speculative Heap tuple insert operation, there was >> inconsistency in t_ctid value. So, I've mod

Re: [HACKERS] WAL consistency check facility

2016-08-27 Thread Kuntal Ghosh
ze and page ID to identify page type. But, I've noticed some cases where the entire page is initialized to zero (Ex: hash_xlog_squeeze_page). RmgrID and info bit can help us to identify those pages. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-

Re: [HACKERS] WAL consistency check facility

2016-08-28 Thread Kuntal Ghosh
gs that are > masked in the patch shouldn't be. > > > -- > Peter Geoghegan -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL consistency check facility

2016-09-07 Thread Kuntal Ghosh
numbers (in rm_tid[0]) are different in REVMAP page. This happens only for two cases. I'm not sure what the reason can be. I haven't done sufficient tests yet to measure the overhead of this modification. I'll do that next. Thanks to Amit Kapila, Dilip Kumar and Robert Haas for the

Re: [HACKERS] WAL consistency check facility

2016-09-07 Thread Kuntal Ghosh
On Wed, Sep 7, 2016 at 3:52 PM, Kuntal Ghosh wrote: > Hello, > > As per the earlier discussions, I've attached the updated patch for > WAL consistency check feature. This is how the patch works: > The earlier patch (wal_consistency_v6.patch) was based on the commit id 67e

Re: [HACKERS] WAL consistency check facility

2016-09-09 Thread Kuntal Ghosh
tter change the > definition list of rmgrs in rmgr.h and get something closer to > RmgrDescData that pg_xlogdump has to avoid all this stanza by > completing it with the name of the rmgr. The only special cases that > this code path would need to take care of would be then 'none

Re: [HACKERS] WAL consistency check facility

2016-09-11 Thread Kuntal Ghosh
G message is going to be >useless for the buildfarm. If we want to detect errors, we could for >example have an additional GUC to trigger an ERROR or a FATAL, taking >down the cluster, and allowing things to show in red on a platform. For now, I've kept this as a WARNING message to detec

Re: [HACKERS] WAL consistency check facility

2016-09-13 Thread Kuntal Ghosh
the first versions >(Heikki, Simon and I?) is going to have a hard time to review this >patch in details moving on if there is no reference to tell what this >feature does for the user... > >This patch is going to the good direction, but I don't think it's far >from being

Re: [HACKERS] WAL consistency check facility

2016-09-13 Thread Kuntal Ghosh
the backup image. - In checkConsistency, we only check if XLogRecHasBlockImage() returns true when wal_consistency check is enabled for this rmid. *XLogRecHasBlockImageForRedo() checks whether bimg_info is set with BKPIMAGE_IS_REQUIRED_FOR_REDO. Thoughts? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL consistency check facility

2016-09-14 Thread Kuntal Ghosh
On Wed, Sep 14, 2016 at 11:31 AM, Michael Paquier wrote: > On Wed, Sep 14, 2016 at 2:56 PM, Kuntal Ghosh > wrote: >> Master >> --- >> - If wal_consistency check is enabled or needs_backup is set in >> XLogRecordAssemble(), we do a fpw. >> - If a f

Re: [HACKERS] WAL consistency check facility

2016-09-14 Thread Kuntal Ghosh
old offnum 11, new offnum 1 thoughts? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL consistency check facility

2016-09-15 Thread Kuntal Ghosh
s warning because of some inconsistency in BRIN VACUUM during gmake check. In recovery tests, I've enabled this feature in PostgresNode.pm. Thanks to Amit, Dilip, Michael, Simon and Robert for their valuable feedbacks. Thoughts? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.ent

Re: [HACKERS] pgbench - compute & show latency consistently

2016-09-21 Thread Kuntal Ghosh
4 ms Suggestion-> SQL script 1: so.sql - weight = 1 (targets 50.0% of total) - 10010 transactions (50.1% of total) - tps = 100.101872 - latency average = 1.878 ms - latency stddev = 3.614 ms Apart from that, pgbench.sgml should be updated to reflect latency average in the outpu

Re: [HACKERS] pgbench - compute & show latency consistently

2016-09-21 Thread Kuntal Ghosh
0 number of threads: 1 number of transactions per client: 1000 number of transactions actually processed: 1/1 tps = 85.184871 (including connections establishing) tps = 85.296346 (excluding connections establishing) Shouldn't we include latency average here as well and explain

Re: [HACKERS] pgbench - compute & show latency consistently

2016-09-22 Thread Kuntal Ghosh
) > tps = 1049.890194 (excluding connections establishing) > > Which is about 10 times better. Yes, you are right. In the documentation, the above example has not been updated across different pg versions. Although this is just an example, it should reflect the current performance.

Re: [HACKERS] wal_segment size vs max_wal_size

2016-09-26 Thread Kuntal Ghosh
buffer gets flushed. (Not a good solution) 3. In CreateRestartPoint() method, we can force a XLogFlush to update minRecoveryPoint. Thoughts? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] wal_segment size vs max_wal_size

2016-09-26 Thread Kuntal Ghosh
lab.ntt.co.jp > +1. I've tested after applying the patch. This clearly solves the problem. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] Pluggable storage

2017-10-13 Thread Kuntal Ghosh
uldFree=true for a tuple on a disk page is not sane */ Assert(BufferIsValid(buffer) ? (!shouldFree) : true); For some storage engine, isn't it possible that the buffer is valid and the tuple to be stored is formed in memory (for example, tuple formed from UNDO, in-memory decrypted tuple e

Re: [HACKERS] Implementing pg_receivewal --no-sync

2017-10-24 Thread Kuntal Ghosh
waitings'? + [ 'pg_receivewal', '-D', $stream_dir, '--synchronous', '--no-sync' ], + 'failure if --synchronous specified without --no-sync'); s/without/with -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com --

Re: [HACKERS] Mention column name in error messages

2016-10-25 Thread Kuntal Ghosh
name. > Imagine for example the case of a CTE using an INSERT query... > Providing the query type would be also useful I think. You can look at > state->p_is_insert for that. In short the context message could just > have this shape: > CONTEXT { INSERT | UPDATE } relname,

Re: [HACKERS] [bug fix] Stats collector is not restarted on the standby

2016-10-26 Thread Kuntal Ghosh
ted with the patch. The patch doesn't solve the problem completely. In standby, after forcible termination, statistics collector process is taking some time to get restarted. In between, if somebody SELECTs against the statistics views, he will still get stale data with the above LOG message.

Re: [HACKERS] [bug fix] Stats collector is not restarted on the standby

2016-10-26 Thread Kuntal Ghosh
NDBY state restarts it. Yes, you are right. Master also has some delay for restarting the process. Otherwise, the patch solves the problem. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To ma

Re: [HACKERS] WAL consistency check facility

2016-10-31 Thread Kuntal Ghosh
ting > manually some inconsistencies in the REDO routines to trigger failures > on standbys. And that was sort of fun to break things intentionally. I know the feeling. :) -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (p

Re: [HACKERS] WAL consistency check facility

2016-11-02 Thread Kuntal Ghosh
aims to validate whether wal replay operation is happening correctly or not. To achieve that aim, we should not alter the wal replay operation itself. Rest of the suggestions are well-taken. I'll update the patch accordingly. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.en

Re: [HACKERS] WAL consistency check facility

2016-11-02 Thread Kuntal Ghosh
eds to be applied at redo. Or BKPIMAGE_IGNORE, to bypass it > when replaying it. IS_REQUIRED_FOR_REDO is quite confusing. BKPIMAGE_APPLY seems reasonable. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] WAL consistency check facility

2016-11-02 Thread Kuntal Ghosh
On Wed, Nov 2, 2016 at 1:11 PM, Kuntal Ghosh wrote: > Rest of the suggestions are well-taken. I'll update the patch accordingly. I've updated the last submitted patch(v10) with the following changes: - added a block level flag BKPIMAGE_APPLY to distinguish backup image blocks whic

Re: [HACKERS] split up psql \d Modifiers column

2016-11-02 Thread Kuntal Ghosh
> + name | text| | | ''::text > +1. psql-ref.sgml(line 4085) needs to be modified. Otherwise, the patch looks good to me. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-h

Re: [HACKERS] WAL consistency check facility

2016-11-02 Thread Kuntal Ghosh
e consistent to have a boolean flag to treat > BKPIMAGE_APPLY in xlogreader.c. Have a look at has_image for example. For flags in bimg_info, we directly check if the mask bit is set in bimg_info (ex: hole, compressed). Besides, we use this flag only at XLogReadBufferForRedoExtended. -- Thanks & Reg

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Kuntal Ghosh
On Thu, Nov 3, 2016 at 12:34 PM, Michael Paquier wrote: > On Thu, Nov 3, 2016 at 3:24 PM, Kuntal Ghosh > wrote: >> On Thu, Nov 3, 2016 at 2:35 AM, Michael Paquier >>> -/* If it's a full-page image, restore it. */ >>> -if (XLogRecHasBlockImage(record,

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Kuntal Ghosh
ockImage(record, block_id) || !XLogRecBlockImageApply(record, block_id)). Thoughts?? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Kuntal Ghosh
On Thu, Nov 3, 2016 at 2:52 PM, Michael Paquier wrote: > On Thu, Nov 3, 2016 at 6:15 PM, Kuntal Ghosh > wrote: >> Actually, I just verified that bimg_info is not even valid if >> has_image is not set. >> In DecodeXLogRecord, we initialize bimg_info only when has_

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Kuntal Ghosh
I've updated the patch for review. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com walconsistency_v12.patch Description: application/download -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscript

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Kuntal Ghosh
On Thu, Nov 3, 2016 at 7:47 PM, Kuntal Ghosh wrote: > I've updated the patch for review. > If an inconsistency is found, it'll just log it for now. Once, the patch is finalized, we can change it to FATAL as before. I was making sure that all regression tests should pass with the

Re: [HACKERS] WAL consistency check facility

2016-11-03 Thread Kuntal Ghosh
On Thu, Nov 3, 2016 at 8:24 PM, Robert Haas wrote: > On Wed, Nov 2, 2016 at 10:30 AM, Kuntal Ghosh > wrote: >> - Another suggestion was to remove wal_consistency from PostgresNode.pm >> because small buildfarm machines may suffer on it. Although I've no >> experience

[HACKERS] Incorrect overflow check condition for WAL segment size

2016-11-07 Thread Kuntal Ghosh
ent is wrongly written or the check for overflow condition has to be fixed. Assuming the overflow check condition to be erroneous, I've attached a patch to fix this. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com wrong_wal_segment_size_upper_limit.patch Descripti

[HACKERS] Incorrect XLogRegisterBuffer flag for revmapbuf in brin

2016-11-09 Thread Kuntal Ghosh
erBuffer(1, revmapbuf, 0); Or, we can update the pd_upper and pd_lower in brin_page_init() as follows: if (BRIN_IS_REVMAP_PAGE(page)) p->pd_lower = p->upper. To fix this, I've attached a small patch which follows the first approach. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: ht

Re: [HACKERS] WAL consistency check facility

2016-11-09 Thread Kuntal Ghosh
ls the space between pd_upper and pd_lower(page hole) with zero. I've posted this as a separate thread. https://www.postgresql.org/message-id/flat/CAGz5QCJ%3D00UQjScSEFbV%3D0qO5ShTZB9WWz_Fm7%2BWd83zPs9Geg%40mail.gmail.com#CAGz5QCJ=00UQjScSEFbV=0qo5shtzb9wwz_fm7+wd83zps9...@mail.gmail.com -- Than

Re: [HACKERS] Unlogged tables cleanup

2016-11-10 Thread Kuntal Ghosh
other *buildempty() functions as well. For example, if the table has a primary key, we'll encounter the error again for btree index. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL consistency check facility

2016-11-10 Thread Kuntal Ghosh
On Thu, Nov 10, 2016 at 10:25 AM, Michael Paquier wrote: > On Wed, Nov 9, 2016 at 11:32 PM, Kuntal Ghosh >> Thanks a lot for reviewing the patch. Based on your review, I've attached the >> I've done a fair amount of testing which includes regression tests

Re: [HACKERS] WAL consistency check facility

2016-11-11 Thread Kuntal Ghosh
patch, but I let that up to your judgement. > Kuntai is definitely an author. Although lot of changes have been done later, but I've started with the patch attached in the above thread. Hence, I feel the author of that patch should also get the credit. -- Thanks & Regards, Kuntal G

Re: [HACKERS] WAL consistency check facility

2016-11-15 Thread Kuntal Ghosh
mitted a patch for that. Following is the thread for the same: https://www.postgresql.org/message-id/flat/CAGz5QCJ%3D00UQjScSEFbV%3D0qO5ShTZB9WWz_Fm7%2BWd83zPs9Geg%40mail.gmail.com#CAGz5QCJ=00UQjScSEFbV=0qo5shtzb9wwz_fm7+wd83zps9...@mail.gmail.com -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http:/

Re: [HACKERS] Passing query string to workers

2017-02-20 Thread Kuntal Ghosh
xt is a const char* variable. Assigning this const pointer to a non-const pointer violates the rules constant-correctness. So, either you should change query_data as const char*, or as Robert suggested, you can directly use estate->es_sourceText. -- Thanks & Regards, Kuntal Ghosh Enterprise

[HACKERS] Performance degradation in TPC-H Q18

2017-02-21 Thread Kuntal Ghosh
sh expansion. The problem with the patch is that it triggers a hash expansion even when the filled percentage is pretty low, causing unnecessary memory consumption. Thoughts? [1] https://www.postgresql.org/message-id/20161123083351.5vramz52nmdokhzz%40alap3.anarazel.de 0001-Resize-sim

Re: [HACKERS] increasing the default WAL segment size

2017-02-24 Thread Kuntal Ghosh
umentation describing the scope and limitations of each approach? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] increasing the default WAL segment size

2017-02-27 Thread Kuntal Ghosh
On Tue, Feb 28, 2017 at 9:45 AM, Jim Nasby wrote: > On 2/24/17 6:30 AM, Kuntal Ghosh wrote: >> >> * You're considering any WAL file with a power of 2 as valid. Suppose, >> the correct WAL seg size is 64mb. For some reason, the server >> generated a 16mb invalid

[HACKERS] WAL Consistency checking for hash indexes

2017-02-27 Thread Kuntal Ghosh
https://www.postgresql.org/message-id/CAA4eK1%2B%2BP%2BjVZC7MC3xzw5uLCpva9%2BgsBpd3semuWffWAftr5Q%40mail.gmail.com -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com 0001-wal_consistency_checking-for-hash-index.patch Description: application/download -- Sent via pgsq

Re: [HACKERS] Performance degradation in TPC-H Q18

2017-02-28 Thread Kuntal Ghosh
or the existing element is less than the current probe length for the element being inserted, swap the two elements and keep going. It leads to a low variance for the chain lengths rather than the last one. Is this approach already considered? -- Thanks & Regards, Kuntal Ghosh Enterprise

Re: [HACKERS] Performance degradation in TPC-H Q18

2017-02-28 Thread Kuntal Ghosh
On Wed, Mar 1, 2017 at 9:19 AM, Andres Freund wrote: > On 2017-03-01 09:13:15 +0530, Kuntal Ghosh wrote: >> On Wed, Mar 1, 2017 at 8:48 AM, Andres Freund wrote: >> >> BTW, another option to consider is lowering the target fillfactor. >> >> IIRC, Kuntal ment

Re: [HACKERS] Performance degradation in TPC-H Q18

2017-02-28 Thread Kuntal Ghosh
On Wed, Mar 1, 2017 at 9:33 AM, Kuntal Ghosh wrote: > On Wed, Mar 1, 2017 at 9:19 AM, Andres Freund wrote: >> That's without the patch in >> http://archives.postgresql.org/message-id/20161123083351.5vramz52nmdokhzz%40alap3.anarazel.de >> ? With that patch it should co

Re: [HACKERS] Performance degradation in TPC-H Q18

2017-02-28 Thread Kuntal Ghosh
On Wed, Mar 1, 2017 at 9:38 AM, Andres Freund wrote: > Hi, > > On 2017-03-01 09:33:07 +0530, Kuntal Ghosh wrote: >> On Wed, Mar 1, 2017 at 9:19 AM, Andres Freund wrote: >> >> So, I was looking for other alternatives and I've found one called >> >&g

Re: [HACKERS] Performance degradation in TPC-H Q18

2017-02-28 Thread Kuntal Ghosh
On Wed, Mar 1, 2017 at 10:53 AM, Andres Freund wrote: > On 2017-03-01 10:47:45 +0530, Kuntal Ghosh wrote: >> if (insertdist > curdist) >> { >> swap the entry to be inserted with the current entry. >> Try to insert the current entry in the same logic. >> } &g

Re: [HACKERS] Performance degradation in TPC-H Q18

2017-03-02 Thread Kuntal Ghosh
ould require an extra shift op every time we want to find the next or prev location during a collision. [1] https://www.postgresql.org/message-id/caeepm%3d3rdgjfxw4ckvj0oemya2-34b0qhng1xv0vk7tgpjg...@mail.gmail.com -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Performance degradation in TPC-H Q18

2017-03-05 Thread Kuntal Ghosh
usy system; I'm fairly strongly of the opinion > that you ought to downgrade that by a couple of levels, say to DEBUG3 > or so. +1 I've tested with TPC-H query 18 and it's working fine. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-08 Thread Kuntal Ghosh
On Fri, Mar 3, 2017 at 9:44 AM, Amit Kapila wrote: > On Tue, Feb 28, 2017 at 11:06 AM, Kuntal Ghosh > wrote: >> Hello everyone, >> >> I've attached a patch which implements WAL consistency checking for >> hash indexes. This feature is going to be useful f

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-09 Thread Kuntal Ghosh
t; > @@ -248,6 +248,9 @@ BackgroundWriterMain(void) > */ > prev_hibernate = false; > > +/* report walwriter process in the PgBackendStatus array */ > +pgstat_procstart(); > + > > s/walwriter/writer/g Fixed. > Patch 0004 should update monitoring.sgml. A

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-09 Thread Kuntal Ghosh
er Activity| WalSenderMain | idle | wal sender | | active | client backend Activity| BgWriterMain| idle | writer Activity| CheckpointerMain| idle | checkpointer Activity| WalWriterMain | idle | wa

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-14 Thread Kuntal Ghosh
ndex. This is useful for generating a set of currently active client backend ID numbers (from 1 to the number of active client backends). These IDs are used for some pgstat_* functions relevant to client processes, e.g., pg_stat_get_backend_activity, pg_stat_get_backend_client_port etc. Any sugg

Re: [HACKERS] WAL Consistency checking for hash indexes

2017-03-14 Thread Kuntal Ghosh
pgist, >> +brin, and generic. Only > > Did that, committed this. Also ran pgindent over hash_mask() and > fixed an instance of dubious capitalization. Thanks Robert for the commit. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-15 Thread Kuntal Ghosh
ELECT pg_stat_get_backend_pid(s.backendid) AS pid, + |pg_stat_get_backend_activity(s.backendid) AS query + | FROM (SELECT pg_stat_get_backend_idset() AS backendid) AS s; 16925 | 16927 | 16926 | 16929 | IMHO, this scenario can be easily avoided by fil

Re: [HACKERS] parallelize queries containing initplans

2017-03-15 Thread Kuntal Ghosh
open up > those cases if required in a separate patch. +1. Unfortunately, this patch doesn't enable parallelism for all possible cases with InitPlans. Our objective is to keep things simple and clean. Still, TPC-H q22 runs 2.5~3 times faster with this patch. -- Thanks & Regards, Kuntal

Re: [HACKERS] Two phase commit in ECPG

2017-03-17 Thread Kuntal Ghosh
e issue. But, I'm not sure whether this test should be performed by installcheck by default or should only be performed by make installcheck-prepared-txns. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@p

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-17 Thread Kuntal Ghosh
On Thu, Mar 16, 2017 at 1:18 PM, Michael Paquier wrote: > On Wed, Mar 15, 2017 at 9:14 PM, Kuntal Ghosh > wrote: >> I've attached the updated patches. > > Thanks for the new versions. This begins to look really clear. Thanks again for the review. > Having some acti

Re: [HACKERS] Two phase commit in ECPG

2017-03-17 Thread Kuntal Ghosh
On Fri, Mar 17, 2017 at 4:34 PM, Masahiko Sawada wrote: > On Fri, Mar 17, 2017 at 12:17 PM, Kuntal Ghosh > wrote: >> On Tue, Mar 14, 2017 at 1:35 AM, Michael Meskes >> wrote: >>>> Previous 002 patch lacked to add describing PREPARE TRANSACTION. >>>>

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-21 Thread Kuntal Ghosh
7;ve made >> it visible to all user for now. >> >> Please find the updated patches in the attachment. > > Yeah, hiding it may make sense... Modified. > /* The autovacuum launcher is done here */ > if (IsAutoVacuumLauncherProcess()) > + { > return; >

Re: [HACKERS] increasing the default WAL segment size

2017-03-22 Thread Kuntal Ghosh
n main () I think that the problem is in following code: /* parse files as start/end boundaries, extract path if not specified */ if (optind < argc) { + if (!RetrieveXLogSegSize(full_path)) ... } In this case, RetrieveXLogSegSize is conditionally called. So, if the condition is false

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-23 Thread Kuntal Ghosh
iterMain() walreceiver WalReceiverMain() walsender InitWalSender() Hence, to be consistent with others, bgworker processes can be initialized from BackgroundWorkerInitializeConnectionBy[Oid]. I've attached the updated patches which reflect the above change. PFA. -- Thanks & Regards, Kunt

[HACKERS] BUG: pg_dump generates corrupted gzip file in Windows

2017-03-23 Thread Kuntal Ghosh
ain-text format, fmt is set to archNull. In that case, the binary mode will not be forced(I think). To fix this, I've attached a patch which adds one extra check in the 'if condition' to check the compression level. PFA. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www

Re: [HACKERS] BUG: pg_dump generates corrupted gzip file in Windows

2017-03-23 Thread Kuntal Ghosh
On Fri, Mar 24, 2017 at 11:28 AM, Kuntal Ghosh wrote: > Hello, > In Windows, if one needs to take a dump in plain text format (this is > the default option, or can be specified using -Fp) with some level of > compression (-Z[0-9]), an output file has to > be specified. Otherwise, i

Re: [HACKERS] BUG: pg_dump generates corrupted gzip file in Windows

2017-03-24 Thread Kuntal Ghosh
On Fri, Mar 24, 2017 at 12:35 PM, Craig Ringer wrote: > On 24 March 2017 at 14:07, Kuntal Ghosh wrote: >> On Fri, Mar 24, 2017 at 11:28 AM, Kuntal Ghosh >> wrote: >>> Hello, >>> In Windows, if one needs to take a dump in plain text format (this is >>&g

Re: [HACKERS] BUG: pg_dump generates corrupted gzip file in Windows

2017-03-24 Thread Kuntal Ghosh
On Fri, Mar 24, 2017 at 2:17 PM, Kuntal Ghosh wrote: > On Fri, Mar 24, 2017 at 12:35 PM, Craig Ringer wrote: >> On 24 March 2017 at 14:07, Kuntal Ghosh wrote: >>> On Fri, Mar 24, 2017 at 11:28 AM, Kuntal Ghosh >>> wrote: >>>> Hello, >>>> In

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-25 Thread Kuntal Ghosh
On Fri, Mar 24, 2017 at 9:23 PM, Robert Haas wrote: > On Thu, Mar 23, 2017 at 7:29 AM, Michael Paquier > wrote: >> On Thu, Mar 23, 2017 at 8:19 PM, Kuntal Ghosh >> wrote: >>> Hence, to be consistent with others, bgworker processes

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-27 Thread Kuntal Ghosh
Thank you Robert for committing the patch. commit fc70a4b0df38bda6a13941f1581f25fbb643c7f3 I've changed the status to Committed. On Mon, Mar 27, 2017 at 6:09 AM, Michael Paquier wrote: > On Sat, Mar 25, 2017 at 5:26 PM, Kuntal Ghosh > wrote: >> On Fri, Mar 24, 2017 at 9:2

Re: [HACKERS] increasing the default WAL segment size

2017-03-30 Thread Kuntal Ghosh
db_tests.patch adds tap tests to initialize cluster with different > wal_segment_size and then check the config values. What other tests do you > have in mind? Checking the various tools? Nothing from me to add here. I've nothing to add here for the attached set of patches. On top of the

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-03-30 Thread Kuntal Ghosh
) >= max_parallel_workers) DO NOT launch any parallel worker. Hence, nworkers = n nworkers_launched = 0. I thought because of my stupid mistake the parallel worker is crashing, so, this is supposed to happen. Sorry for that. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-03-30 Thread Kuntal Ghosh
On Fri, Mar 31, 2017 at 2:05 AM, Kuntal Ghosh wrote: > > 1. Put an Assert(0) in ParallelQueryMain(), start server and execute > any parallel query. > In LaunchParallelWorkers, you can see >nworkers = n nworkers_launched = n (n>0) > But, all the workers will crash

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-03-30 Thread Kuntal Ghosh
On Fri, Mar 31, 2017 at 5:43 AM, Neha Khatri wrote: > > On Fri, Mar 31, 2017 at 8:29 AM, Kuntal Ghosh > wrote: >> >> On Fri, Mar 31, 2017 at 2:05 AM, Kuntal Ghosh >> wrote: >> > >> > 1. Put an Assert(0) in ParallelQueryMain(), start server

Re: [HACKERS] increasing the default WAL segment size

2017-03-30 Thread Kuntal Ghosh
On Fri, Mar 31, 2017 at 10:40 AM, Beena Emerson wrote: > On 30 Mar 2017 15:10, "Kuntal Ghosh" wrote: >> 03-modify-tools.patch - Makes XLogSegSize into a variable, currently set >> as >> XLOG_SEG_SIZE and modifies the tools to fetch the size instead of using >

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-03 Thread Kuntal Ghosh
On Fri, Mar 31, 2017 at 6:50 PM, Robert Haas wrote: > On Thu, Mar 30, 2017 at 4:35 PM, Kuntal Ghosh > wrote: >> 2. the server restarts automatically, initialize >> BackgroundWorkerData->parallel_register_count and >> BackgroundWorkerData->parallel_terminate_count

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-05 Thread Kuntal Ghosh
On Tue, Apr 4, 2017 at 11:22 PM, Tomas Vondra wrote: > On 04/04/2017 06:52 PM, Robert Haas wrote: >> >> On Mon, Apr 3, 2017 at 6:08 AM, Kuntal Ghosh >> wrote: >>> >>> On Fri, Mar 31, 2017 at 6:50 PM, Robert Haas >>> wrote: >>>> &g

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-05 Thread Kuntal Ghosh
ver unnecessarily while executing q2. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-05 Thread Kuntal Ghosh
On Wed, Apr 5, 2017 at 3:07 PM, Tomas Vondra wrote: > > > On 04/05/2017 09:05 AM, Kuntal Ghosh wrote: >> >> AFAICU, during crash recovery, we wait for all non-syslogger children >> to exit, then reset shmem(call BackgroundWorkerShmemInit) and perform >> StartupDat

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-05 Thread Kuntal Ghosh
WorkerArray claims this > is the case. > Agreed on the overflowed case. But, my concern is when an overflow has not yet happened: Suppose, uint parallel_register_count = 1; /* Didn't overflow* / uint parallel_terminate_count = 2; /* Didn't overflow */ Assert(parallel_register_count - par

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-05 Thread Kuntal Ghosh
On Wed, Apr 5, 2017 at 7:31 PM, Robert Haas wrote: > On Wed, Apr 5, 2017 at 6:36 AM, Kuntal Ghosh > wrote: >> Yes. But, as Robert suggested up in the thread, we should not use >> (parallel_register_count = 0) as an alternative to define a bgworker >> crash. Hence, I&#x

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-05 Thread Kuntal Ghosh
On Wed, Apr 5, 2017 at 7:45 PM, Robert Haas wrote: > On Wed, Apr 5, 2017 at 10:09 AM, Kuntal Ghosh > wrote: >>> Did you intend to attach that patch to this email? >>> >> Actually, I'm confused how we should ensure (register_count > >> terminate_cou

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-06 Thread Kuntal Ghosh
On Wed, Apr 5, 2017 at 6:49 PM, Amit Kapila wrote: > On Wed, Apr 5, 2017 at 12:35 PM, Kuntal Ghosh > wrote: >> On Tue, Apr 4, 2017 at 11:22 PM, Tomas Vondra >>> I'm probably missing something, but I don't quite understand how these >>> values actually su

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-10 Thread Kuntal Ghosh
ed integer will be near UINT32_MAX. Hence, we need the absolute difference after typecasting the same to integer. This value should be less than the max_parallel_workers upper limit. I've attached both the patches for better accessibility. PFA. -- Thanks & Regards, Kuntal Ghosh Enterpri

Re: [HACKERS] strange parallel query behavior after OOM crashes

2017-04-11 Thread Kuntal Ghosh
r the code > you're talking about can be reached in those cases, but I wouldn't bet > against it. I think that for above-mentioned background workers, we follow a different path to call ForgetBackgroundWorker(). CleanupBackgroundWorker() - ReportBackgroundWorkerExit()

Re: [HACKERS] Why does logical replication launcher set application_name?

2017-04-11 Thread Kuntal Ghosh
two ways? > For backend_type=background worker, application_name shows the name of the background worker (BackgroundWorker->bgw_name). I think we need some way to distinguish among different background workers. But, application_name may not be the correct field to show the information. -- Th

Re: [HACKERS] Autovacuum launcher occurs error when cancelled by SIGINT

2017-06-21 Thread Kuntal Ghosh
, } - else + else if(!dsm_find_mapping(AutoVacuumShmem->av_dsa_handle)) { AutoVacuumDSA = dsa_attach(AutoVacuumShmem->av_dsa_handle); dsa_pin_mapping(AutoVacuumDSA); Thoughts? -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://w

Re: [HACKERS] Incorrect documentation about pg_stat_activity

2017-06-21 Thread Kuntal Ghosh
On Wed, Jun 21, 2017 at 6:05 PM, Yugo Nagata wrote: > > Attached is a patch for the documentation fix. > Please attach the patch as well. :-) -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postg

Re: [HACKERS] Autovacuum launcher occurs error when cancelled by SIGINT

2017-06-21 Thread Kuntal Ghosh
On Wed, Jun 21, 2017 at 7:07 PM, Dilip Kumar wrote: > On Wed, Jun 21, 2017 at 6:50 PM, Kuntal Ghosh > wrote: >> I think we can just check dsm_find_mapping() to check whether the dsm >> handle is already attached. Something like, >> >> } >

Re: [HACKERS] Autovacuum launcher occurs error when cancelled by SIGINT

2017-06-21 Thread Kuntal Ghosh
On Wed, Jun 21, 2017 at 7:52 PM, Dilip Kumar wrote: > On Wed, Jun 21, 2017 at 7:47 PM, Kuntal Ghosh > wrote: >>> IMHO, It's not a good idea to use DSM call to verify the DSA handle. >>> >> Okay. Is there any particular scenario you've in mind where this m

Re: [HACKERS] An attempt to reduce WALWriteLock contention

2017-06-21 Thread Kuntal Ghosh
67193%402ndquadrant.com > Hello Tomas, I'm really sorry for this late reply. I've somehow missed the thread. Actually, I've seen some performance improvement with the CLogControlLock patch. But, then it turned out all the improvements were because of the CLogControlLock patch al

Re: [HACKERS] An attempt to reduce WALWriteLock contention

2017-06-21 Thread Kuntal Ghosh
to re-submit the patch. Actually, I'm not sure what I should try next. I would love to get some advice/direction regarding this. -- Thanks & Regards, Kuntal Ghosh EnterpriseDB: http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Autovacuum launcher occurs error when cancelled by SIGINT

2017-06-21 Thread Kuntal Ghosh
On Thu, Jun 22, 2017 at 9:48 AM, Michael Paquier wrote: > On Thu, Jun 22, 2017 at 1:29 AM, Kuntal Ghosh > wrote: >> On Wed, Jun 21, 2017 at 7:52 PM, Dilip Kumar wrote: >>> On Wed, Jun 21, 2017 at 7:47 PM, Kuntal Ghosh >>> wrote: >>>>> IMHO, It&#x

Re: [HACKERS] Autovacuum launcher occurs error when cancelled by SIGINT

2017-06-23 Thread Kuntal Ghosh
On Fri, Jun 23, 2017 at 3:01 AM, Thomas Munro wrote: > On Thu, Jun 22, 2017 at 4:29 AM, Kuntal Ghosh > wrote: >> On Wed, Jun 21, 2017 at 7:52 PM, Dilip Kumar wrote: >>> On Wed, Jun 21, 2017 at 7:47 PM, Kuntal Ghosh >>> wrote: >>>>> IMHO, It's

[HACKERS] Error while copying a large file in pg_rewind

2017-07-03 Thread Kuntal Ghosh
COPY fetchchunks, line 2659, column begin: "214800" I guess we've to change the data type to bigint. Also, we need some implementation of ntohl() for 8-byte data types. I've attached a script to reproduce the error and a draft patch. -- Thanks & Regards, Kuntal G

  1   2   >