Re: psql - improve test coverage from 41% to 88%

2020-08-01 Thread Fabien COELHO
Hello, This patch no longer applies: http://cfbot.cputube.org/patch_27_2262.log CF entry has been updated to Waiting on Author. This patch hasn't been updated and still doesn't apply, do you intend to rebase it during this commitfest or should we move it to returned with feedback? It can a

Re: psql - improve test coverage from 41% to 88%

2020-08-03 Thread Fabien COELHO
Re-reading this thread, I see no complaints about introducing a dependency on Expect. Indeed, Tom's complaint was on another thread, possibly when interactive tests "src/bin/psql/t/010_tab_completion.pl" were added. ISTM that one of the issue was that some farm animal would be broken. I'm

Re: Allow continuations in "pg_hba.conf" files

2020-09-04 Thread Fabien COELHO
Hello Tom, Accordingly, I borrowed some code from that thread and present the attached revision. I also added some test coverage, since that was lacking before, and wordsmithed docs and comments slightly. Hearing no comments, pushed that way. Thanks for the fixes and improvements! I noti

Re: psql - add SHOW_ALL_RESULTS option

2021-11-24 Thread Fabien COELHO
Hello Daniel, This patch still has a few of the problems reported earlier this year. The patch fails to apply and the thread seems to have taken a nap. I'm not napping:-) I just do not have enough time available this month. I intend to work on the patch in the next CF (January). AFAICR th

Re: rand48 replacement

2021-11-27 Thread Fabien COELHO
Hello Tom, Thanks for the feedback. +/* use Donald Knuth's LCG constants for default state */ How did Knuth get into this? This algorithm is certainly not his, so why are those constants at all relevant? They are not more nor less relevant than any other "random" constant. The state need

Re: rand48 replacement

2021-11-28 Thread Fabien COELHO
Hello, They are not more nor less relevant than any other "random" constant. The state needs a default initialization. The point of using DK's is that it is somehow cannot be some specially crafted value which would have some special property only know to the purveyor of the constant and could

Re: rand48 replacement

2021-11-29 Thread Fabien COELHO
Pushed with that change and some others, notably: Thanks for the improvements and the push! -- Fabien.

Re: Re: csv format for psql

2018-03-22 Thread Fabien COELHO
Hello Pavel, Using \pset format csv means overwriting field sep every time - nobody uses | Yep. The alternative is to have a csv-specific separator variable, which does not seem very useful, must be remembered, but this is indeed debatable. I think so dependency on order of psql argument

Re: Re: csv format for psql

2018-03-23 Thread Fabien COELHO
Hello Daniel, Do you know when you'll have an updated patch that addresses the minor issues brought up in review and the concern above? Here's an update addressing the issues discussed: - fieldsep and recordsep are used, no more fieldsep_csv - the command line option is --csv without short o

Re: Re: csv format for psql

2018-03-24 Thread Fabien COELHO
Hello Pavel, The patch adds a simple way to generate csv output from "psql" queries, much simpler than playing around with COPY or \copy. It allows to generate a clean CSV dump from something as short as: sh> psql --csv -c 'TABLE foo' > foo.csv Documentation is clear. Test cover a signific

Re: Re: csv format for psql

2018-03-24 Thread Fabien COELHO
Hello Pavel, I'm suggesting to add \csv which would behave like \H to toggle CSV mode so as to improve this situation, with a caveat which is that toggling back \csv would have forgotted the previous settings (just like \H does, though, so would for instance reset to aligned wit

Re: pgbench randomness initialization

2018-03-24 Thread Fabien COELHO
Patch isn't applyed cleanly anymore. Indeed. Here is a rebase. All pgbench patches conflict about test cases. Patch v12, yet another rebase. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index d52d324..41d9030 100644 --- a/doc/src/sgml/ref/pgbench.sgml

Re: pgbench - test whether a variable exists

2018-03-24 Thread Fabien COELHO
While investigating moving pgbench expressions to fe_utils so that they can be shared with psql (\if ..., \let ?), I figure out that psql's \if has a syntax to test whether a variable exists, which is not yet available to pgbench. This patch adds the same syntax to pgbench expression so that

Re: [HACKERS] pgbench - allow to store select results into variables

2018-03-24 Thread Fabien COELHO
Here is a v14, after yet another rebase, and some comments added to answer your new comments. Attached v15 is a simple rebase after Teodor push of new functions & operators in pgbench. Patch v16 is a rebase. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sg

Re: PATCH: pgbench - option to build using ppoll() for larger connection counts

2018-03-25 Thread Fabien COELHO
Hello Doug, Updated the patch to not do the #undef pgbench11-ppoll-v11.patch attached. Patch applies. Do not forget to regenerate configure to test... I've compiled and run with both ppoll & select options without issues. Two quite minor style comment (at least 2 instances): if (cond) re

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2018-03-25 Thread Fabien COELHO
Hm, I took a look on both thread about patch and it seems to me now it's overcomplicated. With recently committed enhancements of pgbench (\if, \when) it becomes close to impossible to retry transation in case of failure. So, initial approach just to rollback such transaction looks more attra

Re: Re: csv format for psql

2018-03-25 Thread Fabien COELHO
Hello Pavel, [...] it is correct. Default format is aligned, that doesn't use fieldsep. My comment is that currently fieldsep is kind of a variable, the value of which is displayed and reliable wrt commands executed afterwards, and the proposed approach changes that by adding a new "defau

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2018-03-26 Thread Fabien COELHO
Hello Marina, Many thanks to both of you! I'm working on a patch in this direction.. I think that the best approach for now is simply to reset (command zero, random generator) and start over the whole script, without attempting to be more intelligent. The limitations should be clearly documen

Re: PATCH: pgbench - option to build using ppoll() for larger connection counts

2018-03-27 Thread Fabien COELHO
Hello Doug, I've compiled and run with both ppoll & select options without issues. Two quite minor style comment (at least 2 instances): Fixed. Fixed. Also fixed issue with 'timeout' not being passed as NULL when no timeout time. v12 compiled and tested with both ppoll & select (by forcing

Re: Re: csv format for psql

2018-03-27 Thread Fabien COELHO
Hello Pavel, I'd like to convince you to compromise, because otherwise I'm afraid we will not get this feature. 1. use special default string for formats that doesn't field sep - like "not used" or some 2. I can implemet the option fieldsep_default - very similary to fieldsep_zero to reset

Re: pgbench - test whether a variable exists

2018-03-27 Thread Fabien COELHO
Patch v2 is a rebase. Patch v3 is also a rebase. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index 41d9030..7b3badf 100644 --- a/doc/src/sgml/ref/pgbench.sgml +++ b/doc/src/sgml/ref/pgbench.sgml @@ -971,6 +971,8 @@ pgbench options d integer c

pgbench doc typos

2018-03-27 Thread Fabien COELHO
Attached patch fixes two very minor typos in pgbench recently added documentations: - one about the mod() function - two about the hash() function -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index 41d9030..e4b37dd 100644 --- a/doc/src/sgml/ref/pgben

new buildfarm with gcc & clang "trunk" -Werror?

2018-03-28 Thread Fabien COELHO
Dear devs, I've been running buildfarm members moonjelly and seawasp which use gcc & clang weekly recompiled trunk versions to build postgres branch HEAD for about 6 months These helped uncover bugs in both gcc & clang, which were reported and fixed, so there is a benefit on this side. Th

Re: Re: csv format for psql

2018-03-28 Thread Fabien COELHO
Hello Pavel, I'd like to convince you to compromise, because otherwise I'm afraid we will not get this feature. [...] The only no-surprise, no-behavioral-change, alternative I see is to have a csv-specific fieldsep. I'm not keen on that one because it is yet another variable, one has to rem

Re: new buildfarm with gcc & clang "trunk" -Werror?

2018-03-29 Thread Fabien COELHO
Hello Peter, - fix these warnings (other than putting -Wno-format-truncation or similar workarounds...). I've been tracking gcc-8, and I have a patch ready, but these things change a bit with every compiler snapshot, so I'm waiting until things settle down. Ok, so I will not submit an

Re: [HACKERS] WIP Patch: Pgbench Serialization and deadlock errors

2018-03-29 Thread Fabien COELHO
Conception of max-retry option seems strange for me. if number of retries reaches max-retry option, then we just increment counter of failed transaction and try again (possibly, with different random numbers). At the end we should distinguish number of error transaction and failed transaction,

Re: pgbench doc typos

2018-03-29 Thread Fabien COELHO
Hello Edmund, Thanks for the check. You might consider turning the patch as ready in the cf app. Fixing the abs/hash bracketing seems clear. The wasn't sure about rewriting "mod(i, bj)" as "mod(i, j)", because there could be some convention about parameter names, but I can't think of anyth

Re: 2018-03 Commitfest Summary (Andres #1)

2018-03-31 Thread Fabien COELHO
Hello Bruce, Has anyone considered moving pgbench out of our git tree and into a separate project where a separate team could maintain and improve it? The movements has been the exact reverse: it was initially in contrib where it had some independence, and has been promoted to the main sourc

Re: csv format for psql

2018-03-31 Thread Fabien COELHO
Bonjour Daniel, For csv, Fabien and Peter expressed the opinion that we shouldn't create another fieldsep-like variable specifically for it, but instead reuse fieldsep. That's what my latest patch does. Now it turns out that sharing fieldsep comes with some problems. Personnaly I do not have

Re: csv format for psql

2018-03-31 Thread Fabien COELHO
Hello Isaac, Personnaly I do not have any problem with CSV defaulting to '|' separator, given that anyway people often use anything but a comma for the purpose, including '|'. However Pavel wants to block the patch on this point. Too bad. OK, mostly trying to avoid commenting because I doubt

Re: [HACKERS] pgbench - allow to store select results into variables

2018-04-07 Thread Fabien COELHO
Hello Stephen, Here's that review. Thanks for the review. + + + \cset [prefix] or + \gset [prefix] + Seems like this should really be moved down below the section for '\set'. Dunno. I put them there because it is in alphabetical order (for cset at least) and bec

Re: [HACKERS] pgbench - allow to store select results into variables

2018-04-07 Thread Fabien COELHO
Hello Stephen, Might be interesting to support mutli-row (or no row?) returns in the future, but not something this patch needs to do, so this looks fine to me. It could match PL/pgSQL's INTO, that is first row or NULL if none. Yeah, but that's not really something that needs to go into thi

Re: Double-writes, take two?

2018-04-18 Thread Fabien COELHO
Bonjour Michaël, - double-write buffers use a pre-decided numbers of pages (32 for the checkpointer, 128 divided into 4 buckets for the backends), which are synced into disk once each batch is full. - The double-write file of the checkpointer uses ordering of pages using blocks number and fi

Re: refactoring - share str2*int64 functions

2019-09-04 Thread Fabien COELHO
Bonjour Michaël, Attached a rebased version which implements the int64/uint64 stuff. It is basically the previous patch without the overflow inlined functions. - if (!strtoint64(yytext, true, &yylval->ival)) + if (unlikely(pg_strtoint64(yytext, &yylval->ival) != PG_STRTOINT_OK)) It

Re: BUG #15977: Inconsistent behavior in chained transactions

2019-09-06 Thread Fabien COELHO
Hello, I do think the fact that COMMIT in multi-statement implicit transaction has some usecase, is an argument for just implementing it properly... Like Peter, I would also keep an ERROR for now, as we could always relax that later on. I can agree with both warning and error, but for me t

Re: BUG #15977: Inconsistent behavior in chained transactions

2019-09-07 Thread Fabien COELHO
I made another patch for that. I don't have much confidence with my English spelling so further improvements may be needed. If it is too much a change and potential regression, then I think that the "and chain" variants should be consistent and just raise warnings. We don't have an exact a

Re: BUG #15977: Inconsistent behavior in chained transactions

2019-09-07 Thread Fabien COELHO
Now, I'd prefer error in all cases, no doubt about that, which might be considered a regression. A way around that could be to have a GUC decide between a strict behavior (error) and the old behavior (warning). I think it's more better to have a GUC to disable implicit transaction 'block' feat

Re: moonjelly vs tsearch

2019-09-07 Thread Fabien COELHO
News from Fabien's bleeding edge compiler zoo: apparently GCC 10 20190831 thinks the tsearch test should produce different output than 20190824 did. It looks like the parsing of question marks change... Indeed, that looks pretty strange. I can't wait for next week's installment. I'll rel

Re: pgbench - allow to create partitioned tables

2019-09-11 Thread Fabien COELHO
Hello Amit, Thanks for the feedback. + case 11: /* partitions */ + initialization_option_set = true; + partitions = atoi(optarg); + if (partitions < 0) + { + fprintf(stderr, "invalid number of partitions: \"%s\"\n", + optarg); + exit(1); + } + break; Is there a reason why we treat "partitions

Re: [patch]socket_timeout in interfaces/libpq

2019-09-11 Thread Fabien COELHO
By the way, Fabien, you are marked as a reviewer of this patch since the end of June. Are you planning to review it? Not this round. -- Fabien.

Re: psql - improve test coverage from 41% to 88%

2019-09-11 Thread Fabien COELHO
Bonjour Michaël, +=item $node->icommand_checks(cmd, ...) + +=cut + +sub icommand_checks Surely this can have a better description, like say PostgresNode::command_checks_all. Ok. Is Expect compatible down to perl 5.8.0 which is the minimum required for the TAP tests (see src/test/perl/READM

Re: psql - improve test coverage from 41% to 88%

2019-09-11 Thread Fabien COELHO
On Wed, Sep 11, 2019 at 10:52:01PM +0200, Fabien COELHO wrote: AFAICR this is because the coverage was not the same:-) Some backslash commands just skip silently to the end of the line, so that intermediate \commands on the same line are not recognized/processed the same, so I moved everything

Re: psql - improve test coverage from 41% to 88%

2019-09-12 Thread Fabien COELHO
removed, some more are present below these lines also. Thanks for this review. The lines were really tests I did that had some issues because of the way the Expect module works, and are not useful for inclusion in the code base. Here is a v5. -- Fabien Coelho - CRI, MINES ParisTechdiff --git

Re: psql - improve test coverage from 41% to 88%

2019-09-12 Thread Fabien COELHO
Here is a v5. Few more in icommand_checks subroutine: Few unwanted code can be removed. Indeed, more debug and test code. Attached v6 fixes these, and I checked for remaining scrubs without finding any. -- Fabien.diff --git a/src/bin/pg_basebackup/t/010_pg_basebackup.pl b/src/bin/pg_bas

Re: pgbench - allow to create partitioned tables

2019-09-13 Thread Fabien COELHO
Hello Dilip, Generally, we give one blank line between the variable declaration and the first statement of the block. Ok. (p-1) -> (p - 1) Ok. I am just wondering will it be a good idea to expand it to support multi-level partitioning? ISTM that how the user could specify multi-level

Re: pgbench - allow to create partitioned tables

2019-09-13 Thread Fabien COELHO
Hello Amit, Is there a reason why we treat "partitions = 0" as a valid value? Yes. It is an explicit "do not create partitioned tables", which differ from 1 which says "create a partitionned table with just one partition". Why would anyone want to use --partitions option in the first case

Re: pgbench - allow to create partitioned tables

2019-09-13 Thread Fabien COELHO
Hello Dilip, + /* For RANGE, we use open-ended partitions at the beginning and end */ + if (p == 1) + sprintf(minvalue, "minvalue"); + else + sprintf(minvalue, INT64_FORMAT, (p-1) * part_size + 1); + + if (p < partitions) + sprintf(maxvalue, INT64_FORMAT, p * part_size + 1); + else + sprintf(m

Re: pgbench - allow to create partitioned tables

2019-09-13 Thread Fabien COELHO
Hello Amit, + res = PQexec(con, + "select p.partstrat, count(*) " + "from pg_catalog.pg_class as c " + "left join pg_catalog.pg_partitioned_table as p on (p.partrelid = c.oid) " + "left join pg_catalog.pg_inherits as i on (c.oid = i.inhparent) " + "where c.relname = 'pgbench_accounts' " + "grou

Re: psql - improve test coverage from 41% to 88%

2019-09-13 Thread Fabien COELHO
Hello Alvaro, I think the TestLib.pm changes should be done separately, not together with the rest of the hacking in this patch. Mostly, because I think they're going to cause trouble. Adding a parameter in the middle of the list may cause trouble for third-party users of TestLib. That is

Re: refactoring - share str2*int64 functions

2019-09-14 Thread Fabien COELHO
Bonjour Michaël, - Switching INT to use pg_strtoint32() causes a set of warnings as for example with AttrNumber: 72 | (void) pg_strtoint32(token, &local_node->fldname) | ^ | | | Att

Re: pgbench - allow to create partitioned tables

2019-09-14 Thread Fabien COELHO
Hello Amit, I'm ensuring that there is always a one line answer, whether it is partitioned or not. Maybe the count(*) should be count(something in p) to get 0 instead of 1 on non partitioned tables, though, but this is hidden in the display anyway. Sure, but I feel the code will be simplified

Re: refactoring - share str2*int64 functions

2019-09-16 Thread Fabien COELHO
Bonjour Michaël, Otherwise, this batch of changes looks ok to me. Thanks. About v15: applies cleanly, compiles, "make check" ok. While re-reading the patch, there are bit of repetitions on pg_strtou?int* comments. I'm wondering whether it would make sense to write a global comments befor

Re: pgbench - allow to create partitioned tables

2019-09-17 Thread Fabien COELHO
Hello Amit, One more comment: +typedef enum { PART_NONE, PART_RANGE, PART_HASH, PART_UNKNOWN } + partition_method_t; See, if we can eliminate PART_UNKNOWN. I'm not very happy with this one, but I wanted to differentiate "we do know that it is not partitioned" from "we do not know if it is

Re: pgbench - allow to create partitioned tables

2019-09-17 Thread Fabien COELHO
Hello Amit, Why can't we change it as attached? I think that your version works, but I do not like much the condition for the normal case which is implicitely assumed. The solution I took has 3 clear-cut cases: 1 error against a server without partition support, detect multiple pgbench_accoun

Re: pgbench - allow to create partitioned tables

2019-09-17 Thread Fabien COELHO
Attached v9: - remove the PART_UNKNOWN and use partitions = -1 to tell that there is an error, and partitions >= 1 to print info - use search_path to find at most one pgbench_accounts It still uses left join because I still think that it is appropriate. I added a lateral to avoid repe

Re: pgbench - allow to create partitioned tables

2019-09-17 Thread Fabien COELHO
Hello Erikjan, [pgbench-init-partitioned-9.patch] Turns out this patch needed a dos2unix treatment. It's easy to do but it takes time to figure it out (I'm dumb). I for one would be happy to receive patches not so encumbered :) AFAICR this is usually because your mailer does not confor

Re: pgbench - allow to create partitioned tables

2019-09-18 Thread Fabien COELHO
Hello Amit, +fprintf(stderr, "invalid partition type, expecting \"range\" or \"hash\"," How about "partitioning method" instead of "partition type"? Indeed, this is a left over from a previous version. +fprintf(stderr, "--partition-method requires actual par

Re: pgbench - allow to create partitioned tables

2019-09-18 Thread Fabien COELHO
+ "group by 1, 2 " I have a question, wouldn't it be sufficient to just group by 1? Conceptually yes, it is what is happening in practice, but SQL requires that non aggregated columns must appear explicitely in the GROUP BY clause, so I have to put it even if it will not change groups. -

Re: pgbench - allow to create partitioned tables

2019-09-18 Thread Fabien COELHO
Hello Amit, - use search_path to find at most one pgbench_accounts It still uses left join because I still think that it is appropriate. I added a lateral to avoid repeating the array_position call to manage the search_path, and use explicit pg_catalog everywhere. It would be go

Re: pgbench - allow to create partitioned tables

2019-09-18 Thread Fabien COELHO
Hello Amit, Yes, on -i it will fail because the syntax will not be recognized. Maybe we should be checking the server version, which would allow to produce more useful error messages when these options are used against older servers, like if (sversion < 1) fprintf(stderr, "cannot use

Re: pgbench - allow to create partitioned tables

2019-09-19 Thread Fabien COELHO
Hello Amit, [...] 'ps' itself won't be NULL in that case, the value it contains is NULL. I have debugged this case as well. 'ps' itself can be NULL only when you pass wrong column number or something like that to PQgetvalue. Argh, you are right! I mixed up C NULL and SQL NULL:-( If we don

Re: pgbench - allow to create partitioned tables

2019-09-19 Thread Fabien COELHO
v12: - fixes NULL vs NULL - works correctly with a partitioned table without partitions attached - generates an error if the partition method is unknown - adds an assert You seem to have attached some previous version (v2) of this patch. I could see old issues in the patch which we ha

Re: pgbench - allow to create partitioned tables

2019-09-19 Thread Fabien COELHO
The behavior is not actually changed, but I had to move fillfactor away because it cannot be declared on partitioned tables, it must be declared on partitions only. Once there is a function to handle that it is pretty easy to add the test. I can remove it but franckly there are only benefits: t

Re: psql - add SHOW_ALL_RESULTS option

2019-09-20 Thread Fabien COELHO
Should we add function header for the below function to maintain the common standard of this file: Yes. Attached v6 does that. -- Fabien.diff --git a/contrib/pg_stat_statements/expected/pg_stat_statements.out b/contrib/pg_stat_statements/expected/pg_stat_statements.out index 6787ec1efd..de59a

Re: psql - add SHOW_ALL_RESULTS option

2019-09-20 Thread Fabien COELHO
The below addition can be removed, it seems to be duplicate: Indeed. I'm unsure how this got into the patch, probably some rebase mix-up. Attached v7 removes the duplicates. -- Fabien.diff --git a/contrib/pg_stat_statements/expected/pg_stat_statements.out b/contrib/pg_stat_statements/expect

Re: pgbench - allow to create partitioned tables

2019-09-20 Thread Fabien COELHO
Hello Amit, I would not bother to create a patch for so small an improvement. This makes sense in passing because the created function makes it very easy, but otherwise I'll just drop it. I would prefer to drop for now. Attached v13 does that, I added a comment instead. I do not think that

Re: pgbench - allow to create partitioned tables

2019-09-21 Thread Fabien COELHO
Hello Amit, Yes, this code is correct. I am not sure if you understood the point, so let me try again. I am bothered about below code in the patch: + /* only print partitioning information if some partitioning was detected */ + if (partition_method != PART_NONE) This is the only place now whe

Re: pgbench - allow to create partitioned tables

2019-09-21 Thread Fabien COELHO
Hello Amit, sh> pgbench -T 10 ... partitions: 0 I am not sure how many users would be able to make out that it is a run where actual partitions are not present unless they beforehand know and detect such a condition in their scripts. What is the use of such a run which completes

Re: pgbench - allow to create partitioned tables

2019-09-22 Thread Fabien COELHO
Hello Amit, It is better for a user to write a custom script for such cases. I kind-of agree, but IMHO this is not for pgbench to decide what is better for the user and to fail on a script that would not fail. Because after that "select-only" or "simple-update" script doesn't make any sen

Re: pgbench - allow to create partitioned tables

2019-09-24 Thread Fabien COELHO
Hello Amit, {...] If you agree with this, then why haven't you changed below check in patch: + if (partition_method != PART_NONE) + printf("partition method: %s\npartitions: %d\n", +PARTITION_METHOD[partition_method], partitions); This is exactly the thing bothering me. It won't be easy

Re: Proposal for syntax to support creation of partition tables when creating parent table

2019-09-25 Thread Fabien COELHO
a link to one such discussion: https://www.postgresql.org/message-id/alpine.DEB.2.21.1907150711080.22273%40lancre Please feel free to share your thoughts. Best Regards ... Muhammad Usama Highgo Software Canada URL : http://www.highgo.ca ADDR: 10318 WHALLEY BLVD, Surrey, BC -- Fabien Coelho - CRI, MINES ParisTech

Re: pgbench - allow to create partitioned tables

2019-09-25 Thread Fabien COELHO
Okay, I think making the check consistent is a step forward. The latest patch is not compiling for me. Argh, shame on me! [...] We don't recommend to start messages with a capital letter. See "Upper Case vs. Lower Case" section in docs [1]. It is not that we have not used it anywhere els

Re: pgbench - allow to create partitioned tables

2019-09-26 Thread Fabien COELHO
Hello Alvaro, pgbench's main() is overly long already, and the new code being added seems to pollute it even more. Can we split it out into a static function that gets placed, say, just below disconnect_all() or maybe after runInitSteps? I agree that refactoring is a good idea, but I do not

Re: pgbench - allow to create partitioned tables

2019-09-27 Thread Fabien COELHO
Hello Amit, I think we might also need to use pg_get_partkeydef along with pg_partition_tree to fetch the partition method information. However, I think to find reloid of pgbench_accounts in the current search path, we might need to use some part of query constructed by Fabien. Fabien, what d

Re: pgbench - allow to create partitioned tables

2019-09-30 Thread Fabien COELHO
Hello Amit, Attached v18: - remove the test tablespace I had to work around a strange issue around partitioned tables and the default tablespace. - if (tablespace != NULL) + if (tablespace != NULL && strcmp(tablespace, "pg_default") != 0) [...] I don't think we need any such check

Re: pgbench - allow to create partitioned tables

2019-09-30 Thread Fabien COELHO
Hello Amit, $node->safe_psql('postgres', "CREATE TABLESPACE regress_pgbench_tap_1_ts LOCATION '$ets';" I think that this last command fails if the path contains a "'", so the '-escaping is necessary. I had to make changes in TAP tests before because it was not working when the path was

Re: pgbench - allow to create partitioned tables

2019-09-30 Thread Fabien COELHO
Hello Amit, 1. ran pgindent 2. As per Alvaro's suggestions move few function definitions. 3. Changed one or two comments and fixed spelling at one place. Thanks for the improvements. Not sure why you put "XXX - " in front of "append_fillfactor" comment, though. + fprintf(stderr, + "no p

Re: pgbench - allow to create partitioned tables

2019-10-01 Thread Fabien COELHO
Yeah, I know that, but this doesn't look quite right. I mean to say whatever we want to say via this message is correct, but I am not completely happy with the display part. How about something like: "pgbench_accounts is missing, you need to do initialization (\"pgbench -i\") in database \"%s

Re: pgbench - allow to create partitioned tables

2019-10-02 Thread Fabien COELHO
Thanks, attached is a patch with minor modifications which I am planning to push after one more round of review on Thursday morning IST unless there are more comments by anyone else. Pushed. Thanks! -- Fabien.

Re: pgbench - allow to create partitioned tables

2019-10-03 Thread Fabien COELHO
Hello, As partitions is an integer type variable, the maximum value it can hold is "2147483647". But if I specify partitions as "3147483647", atoi function returns a value lesser than zero and pgbench terminates with an error. However, if the value for number of partitions specified is somethi

Re: pgbench - extend initialization phase control

2019-10-10 Thread Fabien COELHO
Attached v2 is a rebase after ce8f9467. Here is rebase v3. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index e3a0abb4c7..e9f43f3b26 100644 --- a/doc/src/sgml/ref/pgbench.sgml +++ b/doc/src/sgml/ref/pgbench.sgml @@ -167,7 +167,7 @@ pgbench options d

Re: pgbench - extend initialization phase control

2019-10-17 Thread Fabien COELHO
Hello, Failed regression test. It's necessary to change the first a in “allowed step characters are” to uppercase A in the regression test of 002_pgbench_no_server.pl. Argh. I think I ran the test, then stupidly updated the message afterwards to better match best practices, without rechecki

pgbench - refactor init functions with buffers

2019-10-21 Thread Fabien COELHO
Hello, While developing pgbench to allow partitioned tabled, I reproduced the string management style used in the corresponding functions, but was pretty unhappy with that kind of pattern: snprintf(buf + strlen(buf), sizeof(buf) - strlen(buf), ...) However adding a feature is not th

Re: pgbench - refactor init functions with buffers

2019-10-22 Thread Fabien COELHO
Hello Dilip, - for (i = 0; i < nbranches * scale; i++) + for (int i = 0; i < nbranches * scale; i++) ... - for (i = 0; i < ntellers * scale; i++) + for (int i = 0; i < ntellers * scale; i++) { I haven't read the complete patch. But, I have noticed that many places you changed the variable d

Re: pgbench - refactor init functions with buffers

2019-10-22 Thread Fabien COELHO
Hello Jeevan, I haven't read the complete patch. But, I have noticed that many places you changed the variable declaration from c to c++ style (i.e moved the declaration in the for loop). IMHO, generally in PG, we don't follow this convention. Is there any specific reason to do this? +1.

Re: pgbench - refactor init functions with buffers

2019-10-22 Thread Fabien COELHO
The patch does not apply on master, needs rebase. Hmmm. "git apply pgbench-buffer-1.patch" works for me on current master. Also, I got some whitespace errors. It possible, but I cannot see any. Could you be more specific? For me it failing, see below: $ git log -1 commit ad4b7aeb84434c9

Re: pgbench - refactor init functions with buffers

2019-10-23 Thread Fabien COELHO
Hello Jeevan, +static void +executeStatementExpect(PGconn *con, const char *sql, const ExecStatusType expected, bool errorOK) +{ I think some instances like this need 80 column alignment? Yep. Applying the pgindent is kind-of a pain, so I tend to do a reasonable job by hand and rely on the

Re: pgbench - extend initialization phase control

2019-10-24 Thread Fabien COELHO
Hello Masao-san, The benefit of controlling where begin/end actually occur is that it may have an impact on performance, and it allows to check that. I still fail to understand the benefit of addition of () settings. Could you clarify what case () settings are useful for? You are thinking to

Re: pgbench - extend initialization phase control

2019-10-24 Thread Fabien COELHO
Hello, Yep. Or anything else, including without (), to allow checking the performance impact or non impact of transactions on the initialization phase. Is there actually such performance impact? AFAIR most time-consuming part in initialization phase is the generation of pgbench_accounts data

Re: get_controlfile() can leak fds in the backend

2019-02-28 Thread Fabien COELHO
Hello Andres, Note that my concern is not about the page size, but rather that as more commands may change the cluster status by editing the control file, it would be better that a postmaster does not start while a pg_rewind or enable checksum or whatever is in progress, and currently there is

Re: Online verification of checksums

2019-02-28 Thread Fabien COELHO
Hallo Mickael, So I have now changed behaviour so that short writes count as skipped files and pg_verify_checksums no longer bails out on them. When this occors a warning is written to stderr and their overall count is also reported at the end. However, unless there are other blocks with bad c

Re: get_controlfile() can leak fds in the backend

2019-02-28 Thread Fabien COELHO
Hello Andres, I think putting this into the control file is a seriously bad idea. Postmaster interlocks against other postmasters running via postmaster.pid. Having a second interlock mechanism, in a different file, doesn't make any sort of sense. Nor does it seem sane to have external tool

Re: pg_dump multi VALUES INSERT

2019-03-01 Thread Fabien COELHO
Hello David & Surafel, I think this can be marked as ready for committer now, but I'll defer to Fabien to see if he's any other comments. Patch v16 applies and compiles cleanly, local and global "make check" are ok. Doc build is ok. I did some manual testing with limit cases which did wor

RE: Timeout parameters

2019-03-02 Thread Fabien COELHO
Hello Ryohei-san, There are three independent patches in this thread. About the socket timeout patch v7: Patch applies & compiles cleanly. "make check" is ok, although there are no specific tests, which is probably ok. Doc build is ok. I'd simplify the doc first sentence to: """ Number of

RE: Timeout parameters

2019-03-02 Thread Fabien COELHO
Hello again Ryohei-san, About the tcp_user_timeout libpq parameter v8. Patch applies & compiles cleanly. Global check is ok, although there are no specific tests. Documentation English can be improved. Could a native speaker help, please? ISTM that the documentation both states that it w

Re: Online verification of checksums

2019-03-02 Thread Fabien COELHO
Bonjour Michaël, I gotta say, my conclusion from this debate is that it's simply a mistake to do this without involvement of the server that can use locking to prevent these kind of issues. It seems pretty absurd to me to have hacky workarounds around partial writes of a live server, around tr

RE: Timeout parameters

2019-03-02 Thread Fabien COELHO
About the tcp_user_timeout libpq parameter v8. Basically same thing about the tcp_user_timeout guc v8, especially: do you have any advice about how I can test the feature, i.e. trigger a timeout? Patch applies & compiles cleanly. Global check is ok, although there are no specific tests.

Re: pgbench - add pseudo-random permutation function

2019-03-03 Thread Fabien COELHO
Indeed, the patch needs a rebase & conflit resolution. I'll do it. Later. Here is an update: - take advantage of pg_bitutils (although I noted that the "slow" popcount there could be speeded-up and shorten with a bitwise operator implementation that I just removed from pgbench). - ad

Re: Online verification of checksums

2019-03-03 Thread Fabien COELHO
Bonjour Michaël, I agree that having a server function (extension?) to do a full checksum verification, possibly bandwidth-controlled, would be a good thing. However it would have side effects, such as interfering deeply with the server page cache, which may or may not be desirable. In what i

RE: pgbench - doCustom cleanup

2019-03-04 Thread Fabien COELHO
Hello Kirk, so I tried to apply the patch, but part of the chunk failed, because of the unused line below which was already removed in the recent related commit. PGResult*res; I removed the line and fixed the other trailing whitespaces. Indeed. Thanks for the fix. See t

<    1   2   3   4   5   6   7   8   9   10   >