Re: range test for hash index?

2019-09-14 Thread Amit Kapila
backend/utils/adt/rangetypes.c.gcov.html -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2019-09-15 Thread Amit Kapila
On Sat, Sep 14, 2019 at 3:00 PM Michael Paquier wrote: > > On Sat, Sep 14, 2019 at 03:03:57PM +0900, Michael Paquier wrote: > > On Sat, Sep 14, 2019 at 11:18:37AM +0530, Amit Kapila wrote: > >> Won't 'Lateral' clause be helpful here as the patch contains it

Re: block-level incremental backup

2019-09-16 Thread Amit Kapila
information can be passed via incremental backup. I am not sure if this is a problem, but it would be good if we can somehow handle this. Isn't some operations where at the end we directly call heap_sync without writing WAL will have a similar problem as well? Similarly, it is not very clear if unlogged relations are handled in some way if not, the same could be documented. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: range test for hash index?

2019-09-16 Thread Amit Kapila
On Mon, Sep 16, 2019 at 7:23 AM Paul A Jungwirth wrote: > > On Sat, Sep 14, 2019 at 5:13 AM Amit Kapila wrote: > > In general, the hash_range is covered by some of the existing test, > > but I don't which test. See the code coverage report here: > > https://covera

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2019-09-16 Thread Amit Kapila
t it keeps the code + * simple. + */ + cnt = 0; + flags = (Datum *) palloc0(sizeof(Datum) * bitcnt); If you want to use the same size array, then you might want to just memset the previous array rather than first freeing it and then again allocating it. This is not a big point, so any which way is

Re: [DOC] Document auto vacuum interruption

2019-09-16 Thread Amit Kapila
On Fri, Sep 13, 2019 at 11:59 PM James Coleman wrote: > > On Sat, Aug 31, 2019 at 10:51 PM Amit Kapila wrote: > > > > Updated patch attached. I changed the wording to be about conflicting > locks rather than a single lock type, added a link to the conflicting > locks tab

Re: block-level incremental backup

2019-09-17 Thread Amit Kapila
of those included even > though the full backup likely also does > Yeah, this is quite unfortunate. One more thing to note is that the same is true for other operation like 'create index' (ex. nbtree bypasses buffer manager while creating the index, doesn't write wal for wal_level=minimal and then syncs at the end). -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: block-level incremental backup

2019-09-17 Thread Amit Kapila
On Mon, Sep 16, 2019 at 7:00 PM Robert Haas wrote: > > On Mon, Sep 16, 2019 at 4:31 AM Amit Kapila wrote: > > This seems to be a blocking problem for the LSN based design. > > Well, only the simplest version of it, I think. > > > Can we think of using creation time f

Re: pgbench - allow to create partitioned tables

2019-09-17 Thread Amit Kapila
t; Because I noticed that it had the same description as the previous one, so > I made the test name distinct and more precise, while I was adding options > on it. > Good observation, but better be done separately. I think in general the more unrelated changes are present in patch, the more time it takes to review. One more comment: +typedef enum { PART_NONE, PART_RANGE, PART_HASH, PART_UNKNOWN } + partition_method_t; See, if we can eliminate PART_UNKNOWN. I don't see much use of same. It is used at one place where we can set PART_NONE without much loss. Having lesser invalid values makes code easier to follow. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: pgbench - allow to create partitioned tables

2019-09-17 Thread Amit Kapila
On Tue, Sep 17, 2019 at 4:24 PM Amit Kapila wrote: > > On Sat, Sep 14, 2019 at 6:35 PM Fabien COELHO wrote: > > One more comment: > +typedef enum { PART_NONE, PART_RANGE, PART_HASH, PART_UNKNOWN } > + partition_method_t; > > See, if we can eliminate PART_UNKNOWN. I do

Re: pgbench - allow to create partitioned tables

2019-09-17 Thread Amit Kapila
s should not be a problem here. The part of query is as below: + res = PQexec(con, + "select p.partstrat, count(p.partrelid) " + "from pg_catalog.pg_class as c " This uses pg_catalog, so it should not have multiple entries for "pgbench_accounts". -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: range test for hash index?

2019-09-17 Thread Amit Kapila
On Mon, Sep 16, 2019 at 11:24 PM Paul A Jungwirth wrote: > > On Mon, Sep 16, 2019 at 5:28 AM Amit Kapila wrote: > > I don't see this function on the master branch. Is this function name > > correct? Are you looking at some different branch? > > Sorry about th

Re: [DOC] Document auto vacuum interruption

2019-09-17 Thread Amit Kapila
On Tue, Sep 17, 2019 at 5:48 PM James Coleman wrote: > > On Tue, Sep 17, 2019 at 2:21 AM Amit Kapila wrote: > > > > > > Let me know what you think of attached? I think we can back-patch > > this patch. What do you think? Does anyone else have an opinion on >

Re: POC: Cleaning up orphaned files using undo logs

2019-09-18 Thread Amit Kapila
won't know that while applying undo unless we check the state of a tuple on the page. So, with this idea, we will make one undo record part of multiple chains which might need some consideration at different places like above. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: pgbench - allow to create partitioned tables

2019-09-18 Thread Amit Kapila
not using pgbench with older server versions. I guess it didn't matter, but with this feature, it can matter. Do we need to document this? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] [PATCH] pageinspect function to decode infomasks

2019-09-18 Thread Amit Kapila
gt; > columns for the function, etc.? > > Okay, I have done an extra round of review, and committed it. > Thanks. I was about to have a look today, but anyway I checked the committed patch and it looks fine. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: pgbench - allow to create partitioned tables

2019-09-18 Thread Amit Kapila
not specifically required for the functionality of this patch, then I suggest keeping that behavior as it is. > Attached v11: > > - add quite a few comments on the pg_catalog query > > - reverts the partitions >= 1 test; If some new partition method is > added that pgbench does not know about, the failure mode will be that > nothing is printed rather than printing something strange like > "method none with 2 partitions". > but how will that new partition method will be associated with a table created via pgbench? I think the previous check was good because it makes partition checking consistent throughout the patch. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: pgbench - allow to create partitioned tables

2019-09-18 Thread Amit Kapila
partitioning was present. So, we shouldn't ignore this point. > One thing we could eventually do is just to check pgbench version against > the server version like psql does and output a generic warning if they > differ, but franckly I do not think it is worth the effo

Re: one line doc patch for v12

2019-09-19 Thread Amit Kapila
OIDS=TRUE is not supported anymore. > - OIDs. > > > > Looks good to me, will take care of pushing this change in some time unless someone else takes care of it before me. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: one line doc patch for v12

2019-09-19 Thread Amit Kapila
On Thu, Sep 19, 2019 at 2:15 PM Amit Kapila wrote: > > On Thu, Sep 19, 2019 at 1:40 PM Filip Rembiałkowski > wrote: > > > > There is a small but eye catching glitch in the v12 (and master) docs > > for "CREATE TABLE AS". > > > > https://

Re: [DOC] Document auto vacuum interruption

2019-09-19 Thread Amit Kapila
On Wed, Sep 18, 2019 at 10:25 AM Amit Kapila wrote: > > On Tue, Sep 17, 2019 at 5:48 PM James Coleman wrote: > > > > On Tue, Sep 17, 2019 at 2:21 AM Amit Kapila wrote: > > > > > > > > > Let me know what you think of attached? I think we can ba

Re: pgbench - allow to create partitioned tables

2019-09-19 Thread Amit Kapila
an assert > You seem to have attached some previous version (v2) of this patch. I could see old issues in the patch which we have sorted out in the review. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: pgbench - allow to create partitioned tables

2019-09-19 Thread Amit Kapila
>> nothing is printed rather than printing something strange like > >> "method none with 2 partitions". > > > > but how will that new partition method will be associated with a table > > created via pgbench? > > The user could do a -i with a version of pgbench and bench with another > one. I do that often while developing… > I am not following what you want to say here especially ("pgbench and bench with another one"). Can you explain with some example? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: pgbench - allow to create partitioned tables

2019-09-20 Thread Amit Kapila
ounts table, error out instead of continuing * ensure append_fillfactor doesn't assume that it has to append fillfactor and removed fillfactor < 100 check from it. * improve the comments around query to fetch partitions * improve the comments in the patch and make the code look like nearby co

Re: pgbench - allow to create partitioned tables

2019-09-20 Thread Amit Kapila
; I think we should try to add some note or comment that why we only > > choose to partition pgbench_accounts table when the user has given > > --partitions option. > > Added as a comment on the initPartition function. > I am not sure if something like that is required in the docs, but we can leave it for now. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-09-21 Thread Amit Kapila
); .. } Do we really need to call WaitForParallelWorkersToFinish here as it must have been called in lazy_parallel_vacuum_or_cleanup_indexes before this time? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: pgbench - allow to create partitioned tables

2019-09-21 Thread Amit Kapila
> I also don't agree with your position. My main concern here is that we can't implicitly assume that fillfactor need to be appended. At the very least we should have a comment saying why we are always appending the fillfactor for partitions, something like I had in my patch. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: pgbench - allow to create partitioned tables

2019-09-22 Thread Amit Kapila
why we are always > > appending the fillfactor for partitions > > The patch does not do that, the condition is just before the call, not > inside it with a boolean passed as an argument. AFAICS the behavior of v14 > is exactly the same as your version and as the initial code. > Here, I am talking about the call to append_fillfactor in createPartitions() function. See, in my version, there are some comments. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: pgbench - allow to create partitioned tables

2019-09-23 Thread Amit Kapila
t; [...] I am talking about the call to append_fillfactor in > > createPartitions() function. See, in my version, there are some > > comments. > > Ok, I understand that you want a comment. Patch v15 does that. > Thanks! -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: abort-time portal cleanup

2019-09-24 Thread Amit Kapila
rces, because they would disappear + * in the upcoming transaction-wide cleanup. */ if (portal->status == PORTAL_ACTIVE) I am not able to understand how we can reach with the portal state as 'active' for a multi-transaction command. It seems wherever we mark portal as active, we don't relinquish the control unless its state is changed. Can you share some example where this can happen? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: dropdb --force

2019-09-24 Thread Amit Kapila
ections do not terminate in 5 seconds. + Is there any implementation in the patch for the above note? [1] - https://www.postgresql.org/message-id/20190903164633.GA16408%40alvherre.pgsql -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: dropdb --force

2019-09-24 Thread Amit Kapila
On Tue, Sep 24, 2019 at 6:22 PM Amit Kapila wrote: > On Sat, Sep 21, 2019 at 10:09 PM Pavel Stehule > wrote: > > > > Thank you for check. I am sending updated patch > > > > Alvaro has up thread suggested some alternative syntax [1] for this > patch, but I don&

Re: range test for hash index?

2019-09-24 Thread Amit Kapila
On Wed, Sep 18, 2019 at 9:30 AM Amit Kapila wrote: > > On Mon, Sep 16, 2019 at 11:24 PM Paul A Jungwirth > wrote: > > > > On Mon, Sep 16, 2019 at 5:28 AM Amit Kapila wrote: > > > I don't see this function on the master branch. Is this function name >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-09-25 Thread Amit Kapila
ssage-id/CAMGcDxcBmN6jNeQkgWddfhX8HbSjQpW%3DUo70iBY3P_EPdp%2BLTQ%40mail.gmail.com [2] - https://www.postgresql.org/message-id/EEBD82AA-61EE-46F4-845E-05B94168E8F2%40postgrespro.ru [3] - https://www.postgresql.org/message-id/87a7py4iwl.fsf%40ars-thinkpad -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: pgbench - allow to create partitioned tables

2019-09-25 Thread Amit Kapila
ase \"%s\"\n", PQdb(con)); We don't recommend to start messages with a capital letter. See "Upper Case vs. Lower Case" section in docs [1]. It is not that we have not used it anywhere else, but I think we should try to avoid it. [1] - https://www.postgresql.org/docs/devel/error-style-guide.html -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-09-26 Thread Amit Kapila
ready, so ignore if that is the case. [1] - https://www.postgresql.org/message-id/76fc440e-91c3-afe2-b78a-987205b3c758%402ndquadrant.com -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: range test for hash index?

2019-09-26 Thread Amit Kapila
On Fri, Sep 27, 2019 at 4:03 AM Tomas Vondra wrote: > > On Wed, Sep 25, 2019 at 09:07:13AM +0530, Amit Kapila wrote: > >On Wed, Sep 18, 2019 at 9:30 AM Amit Kapila wrote: > >> > >> On Mon, Sep 16, 2019 at 11:24 PM Paul A Jungwirth > >> wrote: > >&g

Re: pgbench - allow to create partitioned tables

2019-09-26 Thread Amit Kapila
here is no big value addition with that modification as far as this patch is concerned. OTOH, as such there is no harm in testing with non-default tablespaces. The other thing is that the query used in patch to fetch partition information seems correct to me, but maybe there is a better way to get that information. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: dropdb --force

2019-09-26 Thread Amit Kapila
TH clause for a somewhat similar purpose. I think we use WITH clause in other cases while specifying multiple options. So to me, using WITH here doesn't sound to be a bad idea. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: range test for hash index?

2019-09-26 Thread Amit Kapila
On Fri, Sep 27, 2019 at 6:02 AM Amit Kapila wrote: > > On Fri, Sep 27, 2019 at 4:03 AM Tomas Vondra > wrote: > > > > > > By "inconsistent" you mean that pre-10 versions will have different > > expected output than versions with WAL-logged hash indexes?

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-09-26 Thread Amit Kapila
On Fri, Sep 27, 2019 at 12:06 AM Tomas Vondra wrote: > > On Thu, Sep 26, 2019 at 06:58:17PM +0530, Amit Kapila wrote: > > >3. > >+ * Find the largest transaction (toplevel or subxact) to evict (spill to > >disk). > >+ * > >+ * XXX With many subtransactions

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-09-27 Thread Amit Kapila
y use work_mem for this case? If we can't use work_mem, then I think the name proposed by you (logical_decoding_work_mem) sounds good to me. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-09-27 Thread Amit Kapila
On Thu, Sep 26, 2019 at 11:37 PM Tomas Vondra wrote: > > On Wed, Sep 25, 2019 at 06:55:01PM +0530, Amit Kapila wrote: > >On Tue, Sep 3, 2019 at 4:30 AM Alvaro Herrera > >wrote: > >> > >> In the interest of moving things forward, how far are we from making >

Re: pgbench - allow to create partitioned tables

2019-09-27 Thread Amit Kapila
On Fri, Sep 27, 2019 at 7:05 PM Alvaro Herrera wrote: > > On 2019-Sep-27, Amit Kapila wrote: > > > The other thing is that the query used in patch to fetch partition > > information seems correct to me, but maybe there is a better way to > > get that information. >

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-09-28 Thread Amit Kapila
On Fri, Sep 27, 2019 at 4:55 PM Tomas Vondra wrote: > > On Fri, Sep 27, 2019 at 02:33:32PM +0530, Amit Kapila wrote: > >On Tue, Jan 9, 2018 at 7:55 AM Peter Eisentraut > > wrote: > >> > >> On 1/3/18 14:53, Tomas Vondra wrote: > >> &

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-09-28 Thread Amit Kapila
r can ever use this. 4. Can we some way expose the memory consumed by ReorderBuffer? If so, we might be able to write some tests covering new functionality. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-09-28 Thread Amit Kapila
On Sun, Sep 29, 2019 at 12:39 AM Tomas Vondra wrote: > > On Sat, Sep 28, 2019 at 01:36:46PM +0530, Amit Kapila wrote: > >On Fri, Sep 27, 2019 at 4:55 PM Tomas Vondra > > wrote: > > I do think having a separate GUC is a must, irrespectedly of what other > GUC (if any) i

Re: pgbench - allow to create partitioned tables

2019-09-29 Thread Amit Kapila
ory $ts"; +my $ets = TestLib::perl2host($ts); +# add needed escaping! +$ets =~ s/'/''/; I am not sure if we really need this quote skipping stuff. Why can't we write the test as below: # tablespace for testing my $basedir = $node->basedir; my $ts = "$basedir/re

Re: pgbench - allow to create partitioned tables

2019-09-30 Thread Amit Kapila
e other places in tests like src/bin/pg_checksums/t/002_actions.pl which uses the way I have mentioned. I don't think there is any need for escaping single-quotes here and I am not seeing the use of same. I don't want to introduce a new pattern in tests which people can then tomorrow copy at other places even though such code is not required. OTOH, if there is a genuine need for the same, then I am fine. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: Commit fest 2019-09

2019-09-30 Thread Amit Kapila
11 │11 │12 > > This means we got 28 patches committed during this month ... which is > a rather concerningly low figure. > Yes, I agree that we can improve these numbers. However, like others, I would also like to thank you for sincere and dedicated efforts in this CF. It real

Re: pgbench - allow to create partitioned tables

2019-09-30 Thread Amit Kapila
it(1); + } Can anyone else think of a better error message either in wording or style for above case? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com pgbench-init-partitioned-21.patch Description: Binary data

Re: Proposal: Make use of C99 designated initialisers for nulls/values arrays

2019-10-01 Thread Amit Kapila
can review and take this forward unless there are objections from others. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-01 Thread Amit Kapila
On Sun, Sep 29, 2019 at 11:24 AM Amit Kapila wrote: > On Sun, Sep 29, 2019 at 12:39 AM Tomas Vondra > wrote: > > > > Yeah, it is better to deal it separately as I am also not entirely > convinced at this stage about this parameter. I have mentioned the > same in th

Re: [HACKERS] Block level parallel vacuum

2019-10-01 Thread Amit Kapila
On Sat, Sep 21, 2019 at 6:01 PM Amit Kapila wrote: > On Fri, Jun 7, 2019 at 12:03 PM Masahiko Sawada > wrote: > > > > Since the previous version patch conflicts with current HEAD, I've > > attached the updated version patches. > > > > Review comments: &

Re: pgbench - allow to create partitioned tables

2019-10-01 Thread Amit Kapila
ench_accounts is missing, you need to do initialization (\"pgbench -i\") in database \"%s\"\n"? Feel free to propose something else on similar lines? If possible, I want to convey this information in a single sentence. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-01 Thread Amit Kapila
On Tue, Oct 1, 2019 at 7:21 PM Tomas Vondra wrote: > On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote: > > > >On further testing, I found that the patch seems to have problems with > >toast. Consider below scenario: > >Session-1 > >Create table la

Re: pgbench - allow to create partitioned tables

2019-10-01 Thread Amit Kapila
two spaces after the period if another sentence follows (for English text; might be inappropriate in other languages)." Also, the similar style is used in other places in code, see contrib/oid2name/oid2name.c, contrib/pg_standby/pg_standby.c for similar usage. I shall modify this before commit unless you disagree. [1] - https://www.postgresql.org/docs/devel/error-style-guide.html -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: Proposal: Make use of C99 designated initialisers for nulls/values arrays

2019-10-01 Thread Amit Kapila
nion on this, but I would mildly prefer to initialize boolean array with false just for the sake of readability (we generally initializing booleans with false). -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: dropdb --force

2019-10-01 Thread Amit Kapila
On Thu, Sep 26, 2019 at 7:18 PM Pavel Stehule wrote: > > út 24. 9. 2019 v 14:52 odesílatel Amit Kapila > napsal: > >> >> One other minor comment: >> + >> + This will also fail, if the connections do not terminate in 5 >> seconds. >> + &g

Re: pgbench - allow to create partitioned tables

2019-10-02 Thread Amit Kapila
On Tue, Oct 1, 2019 at 10:20 AM Amit Kapila wrote: > On Mon, Sep 30, 2019 at 5:17 PM Fabien COELHO wrote: > > > > > I don't want to introduce a new pattern in tests which people can then > > > tomorrow copy at other places even though such code is not required. &

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-03 Thread Amit Kapila
On Thu, Oct 3, 2019 at 4:03 AM Tomas Vondra wrote: > On Wed, Oct 02, 2019 at 04:27:30AM +0530, Amit Kapila wrote: > >On Tue, Oct 1, 2019 at 7:21 PM Tomas Vondra > > >wrote: > > > >> On Tue, Oct 01, 2019 at 06:55:52PM +0530, Amit Kapila wrote: > >> &g

Re: Proposal: Make use of C99 designated initialisers for nulls/values arrays

2019-10-03 Thread Amit Kapila
m values[N] = INIT_ALL_ZEROES; > This is a good idea, but by reading the thread it is not completely clear if we want to pursue this or want to explore something else or leave the current code as it is. Also, if we want to pursue, do we want to use INIT_ALL_ZEROES for bool arrays as well? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-10-03 Thread Amit Kapila
On Wed, Oct 2, 2019 at 7:29 PM Masahiko Sawada wrote: > On Sat, Sep 21, 2019 at 9:31 PM Amit Kapila > wrote: > > * > > In function compute_parallel_workers, don't we want to cap the number > > of workers based on maintenance_work_mem as we do in > > plan_crea

Re: [HACKERS] Block level parallel vacuum

2019-10-03 Thread Amit Kapila
otherwise complete code is the same > > for > > both the cases. Can't we create some string and based on the value of > > the for_cleanup and append it in the error message that way we can > > avoid duplicating this at many places? > > I think it's necessary for translation. IIUC if we construct the > message it cannot be translated. > > Do we really need to log all those messages? The other places where we launch parallel workers doesn't seem to be using such messages. Why do you think it is important to log the messages here when other cases don't use it? -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: Proposal: Make use of C99 designated initialisers for nulls/values arrays

2019-10-03 Thread Amit Kapila
ed > INIT_ALL_ELEMS_ZERO to a bool array. > > How about I just define them both the same? > #define INIT_ALL_ELEMS_ZERO {0} > #define INIT_ALL_ELEMS_FALSE{0} > > I think using one define would be preferred, but you can wait and see if others prefer defining different macros for the same thing. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-10-04 Thread Amit Kapila
On Wed, Oct 2, 2019 at 7:29 PM Masahiko Sawada wrote: > On Sat, Sep 21, 2019 at 9:31 PM Amit Kapila > wrote: > > > > * > > +end_parallel_vacuum(LVParallelState *lps, Relation *Irel, int nindexes) > > { > > .. > > + /* Shutdown worker

Re: [HACKERS] Block level parallel vacuum

2019-10-05 Thread Amit Kapila
On Fri, Oct 4, 2019 at 7:57 PM Masahiko Sawada wrote: > On Fri, Oct 4, 2019 at 2:31 PM Amit Kapila > wrote: > >> > > > > Do we really need to log all those messages? The other places where we > launch parallel workers doesn't seem to be using such messages.

maintenance_work_mem used by Vacuum

2019-10-06 Thread Amit Kapila
current memory usage model is right. [1] - https://www.postgresql.org/docs/devel/runtime-config-resource.html [2] - https://www.postgresql.org/message-id/CAD21AoARj%3De%3D6_KOZnaR66jRkDmGaVdLcrt33Ua-zMUugKU3mQ%40mail.gmail.com -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-10-06 Thread Amit Kapila
On Fri, Oct 4, 2019 at 7:34 PM Masahiko Sawada wrote: > On Fri, Oct 4, 2019 at 2:02 PM Amit Kapila > wrote: > >> > >> I'd also prefer to use maintenance_work_mem at max during parallel > >> vacuum regardless of the number of parallel workers. This is curre

Re: [HACKERS] Block level parallel vacuum

2019-10-06 Thread Amit Kapila
On Mon, Oct 7, 2019 at 10:00 AM Masahiko Sawada wrote: > On Sat, Oct 5, 2019 at 8:22 PM Amit Kapila > wrote: > > > > On Fri, Oct 4, 2019 at 7:57 PM Masahiko Sawada > wrote: > >> > >> On Fri, Oct 4, 2019 at 2:31 PM Amit Kapila > wrote: > >>

Re: maintenance_work_mem used by Vacuum

2019-10-07 Thread Amit Kapila
for GIN indexes and 3/4 (75%) of maintenance_work_mem for collection dead tuples. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: maintenance_work_mem used by Vacuum

2019-10-07 Thread Amit Kapila
On Tue, Oct 8, 2019 at 12:57 AM Robert Haas wrote: > > On Sun, Oct 6, 2019 at 6:55 AM Amit Kapila wrote: > > We wanted to decide how a parallel vacuum should use memory? Can each > > worker consume maintenance_work_mem to clean up the gin Index or all > > workers

Re: Ordering of header file inclusion

2019-10-07 Thread Amit Kapila
es.h" - #include "common/logging.h" - +#include "variables.h" Same as above. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: Proposal: Make use of C99 designated initialisers for nulls/values arrays

2019-10-08 Thread Amit Kapila
e same patch, we can even do it as a separate patch. I am personally still in the camp of people advocating the use of macro for this purpose. It is quite possible after reading your points, some people might change their opinion or some others also share their opinion against using a macro in which case we can drop the idea of using a macro. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: Ordering of header file inclusion

2019-10-08 Thread Amit Kapila
On Tue, Oct 8, 2019 at 8:19 PM Tom Lane wrote: > > Amit Kapila writes: > > On Wed, Oct 2, 2019 at 2:57 PM vignesh C wrote: > >> I noticed that some of the header files inclusion is not ordered as > >> per the usual standard that is followed. > >> The

Re: maintenance_work_mem used by Vacuum

2019-10-09 Thread Amit Kapila
On Wed, Oct 9, 2019 at 2:00 PM Dilip Kumar wrote: > > On Wed, Oct 9, 2019 at 10:22 AM Masahiko Sawada wrote: > > > > On Tue, Oct 8, 2019 at 2:45 PM Amit Kapila wrote: > > > > > > On Tue, Oct 8, 2019 at 1:48 AM Peter Geoghegan wrote: > > > > &

Re: [HACKERS] Block level parallel vacuum

2019-10-09 Thread Amit Kapila
rate a lot of I/O which might choke other operations. I think if we want we can provide Guc(s) to control such behavior, but initially providing it via command should be a good start so that users can knowingly use it in appropriate cases. We can later extend it for autovacuum if required. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-10-09 Thread Amit Kapila
On Fri, Oct 4, 2019 at 4:18 PM Amit Kapila wrote: > > On Wed, Oct 2, 2019 at 7:29 PM Masahiko Sawada wrote: >> Few more comments: - 1. Caurrently parallel vacuum is allowed for temporary relations which is wrong. It leads to below error: postg

Re: maintenance_work_mem used by Vacuum

2019-10-09 Thread Amit Kapila
indexes, in which we try to accumulate multiple index entries in a holding area before adding them to the main index structure. This helps because bulk insert is (usually) significantly faster than retail insert for GIN. .. .. Teodor Sigaev -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: maintenance_work_mem used by Vacuum

2019-10-10 Thread Amit Kapila
On Thu, Oct 10, 2019 at 2:10 PM Masahiko Sawada wrote: > > On Thu, Oct 10, 2019 at 3:36 PM Amit Kapila wrote: > > > > On Thu, Oct 10, 2019 at 9:58 AM Masahiko Sawada > > wrote: > > > > > > On Wed, Oct 9, 2019 at 7:12 PM Dilip Kumar wrote: > >

Re: abort-time portal cleanup

2019-10-10 Thread Amit Kapila
_explain is active, but it's really rather surprising that nobody has seen an issue with this before. I'm back-patching this to 8.4, since that's the first version that contains auto_explain or an ExecutorEnd hook, but I wonder whether we shouldn't back-patch further. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: maintenance_work_mem used by Vacuum

2019-10-11 Thread Amit Kapila
On Fri, Oct 11, 2019 at 7:36 AM Masahiko Sawada wrote: > > On Thu, Oct 10, 2019 at 6:38 PM Amit Kapila wrote: > > > > > > It seems you want to say about commit id > > a1b395b6a26ae80cde17fdfd2def8d351872f399. > > Yeah thanks. > > >

Re: [HACKERS] Block level parallel vacuum

2019-10-11 Thread Amit Kapila
ion in end_parallel_vacuum, see if this fixes the problem for you. You need to apply the attached patch atop v28-0001-Add-parallel-option-to-VACUUM-command posted above by Sawada-San. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com Fix-memory-allocation-for-copying-the-stats.patch Description: Binary data

Re: [HACKERS] Block level parallel vacuum

2019-10-12 Thread Amit Kapila
On Sat, Oct 12, 2019 at 11:29 AM Masahiko Sawada wrote: > > On Sat, Oct 12, 2019 at 12:33 PM Amit Kapila wrote: > > > > On Fri, Oct 11, 2019 at 4:47 PM Mahendra Singh wrote: > > > > > Thank you for reviewing and creating the patch! > > I think the patch f

Re: maintenance_work_mem used by Vacuum

2019-10-12 Thread Amit Kapila
On Sat, Oct 12, 2019 at 10:49 AM Masahiko Sawada wrote: > > On Fri, Oct 11, 2019 at 5:13 PM Amit Kapila wrote: > > > > That's right, but OTOH, if the user specifies gin_pending_list_limit > > as an option during Create Index with a value greater than GUC > > g

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-13 Thread Amit Kapila
On Sun, Oct 13, 2019 at 12:25 PM Dilip Kumar wrote: > > On Thu, Oct 3, 2019 at 2:43 PM Amit Kapila wrote: > > > > 3. > > + * > > + * While updating the existing change with detoasted tuple data, we need to > > + * update the memory accounting info,

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2019-10-13 Thread Amit Kapila
On Mon, Oct 14, 2019 at 6:51 AM Craig Ringer wrote: > > On Sun, 13 Oct 2019 at 19:50, Amit Kapila wrote: >> > > > Does anyone object if we add the reorder buffer total size & in-memory size > to struct WalSnd too, so we can report it in pg_stat_replication? >

Re: [HACKERS] Block level parallel vacuum

2019-10-14 Thread Amit Kapila
On Sat, Oct 12, 2019 at 4:50 PM Amit Kapila wrote: > On Sat, Oct 12, 2019 at 11:29 AM Masahiko Sawada > wrote: > > > > I see a much bigger problem with the way this patch collects the index > stats in shared memory. IIUC, it allocates the shared memory (DSM) > for all

Questions/Observations related to Gist vacuum

2019-10-14 Thread Amit Kapila
sql.org/message-id/CAA4eK1JEQ2y3uNucNopDjK8pse6xSe5%3D_oknoWfRQvAF%3DVqsBA%40mail.gmail.com -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-10-14 Thread Amit Kapila
On Tue, Oct 15, 2019 at 10:34 AM Masahiko Sawada wrote: > > On Mon, Oct 14, 2019 at 6:37 PM Amit Kapila wrote: > > > > > 3. Do we really need to give the responsibility of deleting empty > > pages (gistvacuum_delete_empty_pages) to gistvacuumcleanup. Can't we &g

Re: ProcArrayGroupClearXid() compare-exchange style

2019-10-15 Thread Amit Kapila
ge it in HEAD. > +1. I am not sure if it would improve performance as this whole optimization was to reduce the number of attempts to acquire LWLock, but definitely, it makes the code consistent. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-10-15 Thread Amit Kapila
m patch so that it checks the participation of indexes > > and then vacuums on un-participated indexes after parallel vacuum. > > amcanparallelvacuum is not necessary to be a callback, it can be a > boolean field of IndexAmRoutine. > Yes, it will be a boolean. Note that for parallel-index scans, we already have amcanparallel. -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: Ordering of header file inclusion

2019-10-15 Thread Amit Kapila
On Tue, Oct 15, 2019 at 10:57 PM vignesh C wrote: > > On Wed, Oct 9, 2019 at 11:37 AM Amit Kapila wrote: > > > > On Tue, Oct 8, 2019 at 8:19 PM Tom Lane wrote: > > > > > > Amit Kapila writes: > > > > On Wed, Oct 2, 2019 at 2:57 PM vignesh C w

Re: [HACKERS] Block level parallel vacuum

2019-10-15 Thread Amit Kapila
On Wed, Oct 16, 2019 at 6:50 AM Masahiko Sawada wrote: > > On Tue, Oct 15, 2019 at 6:33 PM Amit Kapila wrote: > > > > Attached updated patch set. 0001 patch introduces new index AM field > amcanparallelvacuum. All index AMs except for gist sets true for now. > 0002 pa

Re: maintenance_work_mem used by Vacuum

2019-10-15 Thread Amit Kapila
On Wed, Oct 16, 2019 at 7:20 AM Masahiko Sawada wrote: > > On Sat, Oct 12, 2019 at 8:45 PM Amit Kapila wrote: > > > > On Sat, Oct 12, 2019 at 10:49 AM Masahiko Sawada > > wrote: > > > > > > On Fri, Oct 11, 2019 at 5:13 PM Amit Kapila > > >

Re: Questions/Observations related to Gist vacuum

2019-10-16 Thread Amit Kapila
On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas wrote: > > On 15/10/2019 09:37, Amit Kapila wrote: > > 2. Right now, in gistbulkdelete we make a note of empty leaf pages and > > internals pages and then in the second pass during gistvacuumcleanup, > > we delete all the

Re: Questions/Observations related to Gist vacuum

2019-10-16 Thread Amit Kapila
On Wed, Oct 16, 2019 at 11:20 AM Dilip Kumar wrote: > > On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas wrote: > > > > On 15/10/2019 09:37, Amit Kapila wrote: > > > While reviewing a parallel vacuum patch [1], we noticed a few things > > >

Re: Questions/Observations related to Gist vacuum

2019-10-16 Thread Amit Kapila
On Wed, Oct 16, 2019 at 7:21 PM Heikki Linnakangas wrote: > > On 16 October 2019 12:57:03 CEST, Amit Kapila wrote: > >On Tue, Oct 15, 2019 at 7:13 PM Heikki Linnakangas > >wrote: > >> All things > >> considered, I'm not sure which is better. > > &

Re: maintenance_work_mem used by Vacuum

2019-10-16 Thread Amit Kapila
gt; with tuples in the pending queue not getting vacuumed?) Yeah, I also think so. BTW, good point! -- With Regards, Amit Kapila. EnterpriseDB: http://www.enterprisedb.com

Re: [HACKERS] Block level parallel vacuum

2019-10-16 Thread Amit Kapila
On Thu, Oct 17, 2019 at 10:56 AM Masahiko Sawada wrote: > > On Wed, Oct 16, 2019 at 3:02 PM Amit Kapila wrote: > > > > On Wed, Oct 16, 2019 at 6:50 AM Masahiko Sawada > > wrote: > > > > > > On Tue, Oct 15, 2019 at 6:33 PM Amit Kapila > > >

<    8   9   10   11   12   13   14   15   16   17   >