Re: refactoring - share str2*int64 functions

2019-08-01 Thread Fabien COELHO
Bonjour Michaël, Yep, I will try for this week. Please note that for now I have marked the patch as returned with feedback as the CF is ending. Ok. I have looked quickly at it, but I'm not sure that there is an agreement about what should be done precisely, so the feedback is not clearly

Re: refactoring - share str2*int64 functions

2019-08-01 Thread Fabien COELHO
Michaël-san, I have looked quickly at it, but I'm not sure that there is an agreement about what should be done precisely, so the feedback is not clearly actionable. Per the latest trends, it seems that the input of Andres was kind of the most interesting pieces. Yes, definitely. I understo

Re: refactoring - share str2*int64 functions

2019-08-01 Thread Fabien COELHO
extern uint64 pg_strtouint64(const char *str, char **endptr, int base); called 3 times, always with base == 10. We have a similar name but a totally different interface, so basically it would have to be replaced by something like the first interface. My understanding on this one was to nuke

Re: pgbench - implement strict TPC-B benchmark

2019-08-01 Thread Fabien COELHO
Hello Robert, All in all, pgbench overheads are small compared to postgres processing times and representative of a reasonably optimized client application. It's pretty easy to devise tests where pgbench is client-limited -- just try running it with threads = clients/4, sometimes even clients

Re: pgbench - implement strict TPC-B benchmark

2019-08-02 Thread Fabien COELHO
Hello Andres, Thanks a lot for these feedbacks and comments. Using pgbench -Mprepared -n -c 8 -j 8 -S pgbench_100 -T 10 -r -P1 e.g. shows pgbench to use 189% CPU in my 4/8 core/thread laptop. That's a pretty significant share. Fine, but what is the corresponding server load? 211%? 611%? And

Re: Commitfest 2019-07, the first of five* for PostgreSQL 13

2019-08-02 Thread Fabien COELHO
I guess the CF app could show those kind of metrics, but having a written report from a human seems to be a good idea (I got it from Alvaro's blog[1]). The CF is now closed, and here are the final numbers: status | w1 | w2 | w3 | w4 | final +++--

Re: pgbench - implement strict TPC-B benchmark

2019-08-03 Thread Fabien COELHO
Hello Andres, Using pgbench -Mprepared -n -c 8 -j 8 -S pgbench_100 -T 10 -r -P1 e.g. shows pgbench to use 189% CPU in my 4/8 core/thread laptop. That's a pretty significant share. Fine, but what is the corresponding server load? 211%? 611%? And what actual time is spent in pgbench itself, vs

Re: Undocumented PQdisplayTuples and PQprintTuples in libpq

2019-08-04 Thread Fabien COELHO
Bonjour Michaël, I have noticed today that the two functions in $subject are part of libpq and remain around undocumented since they are around (see 6ef5846 and 2b84cbb). Isn't it past time to get rid of them? We have PQprint as well, which was used in the past by psql but not today, still th

Re: pgbench - implement strict TPC-B benchmark

2019-08-05 Thread Fabien COELHO
Hello Andres, If not, do you think advisable to spend time improving the evaluator & variable stuff and possibly other places for an overall 15% gain? Also, what would be the likelyhood of such optimization patch to pass? I could do a limited variable management improvement patch, eventually

Re: pgbench - implement strict TPC-B benchmark

2019-08-05 Thread Fabien COELHO
Hello Andres, Which is a (somehow disappointing) * 3.3 speedup. The impact on the 3 complex expressions tests is not measurable, though. I don't know why that could be disappointing. We put in much more work for much smaller gains in other places. Probably, but I thought I would have a bett

Re: pgbench - implement strict TPC-B benchmark

2019-08-05 Thread Fabien COELHO
Hello Robert, Ok, one thread cannot feed an N core server if enough client are executed per thread and the server has few things to do. Right ... where N is, uh, TWO. Yes, two indeed… For low-work cpu-bound load, given libpq & system overheads, you cannot really hope for a better deal. I

pgbench - rework variable management

2019-08-13 Thread Fabien COELHO
Hello pgdevs, The attached patch improves pgbench variable management as discussed in: https://www.postgresql.org/message-id/flat/alpine.DEB.2.21.1904081752210.5867@lancre As discussed there as well, the overall effect is small compared to libpq & system costs when pgbench is talking to a pos

Re: Creating partitions automatically at least on HASH?

2019-08-18 Thread Fabien COELHO
Hello Robert & Robert, - no partitions are created immediately (current case) but will have to be created manually later - static partitions are created automatically, based on provided parameters - dynamic partitions will be created later, when needed, based on provided parameters agai

Re: pg_checksums --help synopsis is quite long

2019-08-22 Thread Fabien COELHO
Hello Peter, With the additional functionality, the --help synopsis of pg_checksums has gotten quite long: pg_checksums enables, disables, or verifies data checksums in a PostgreSQL database cluster. Can we try to shorten this a bit? Maybe pg_checksums manages data checksums in a Post

Re: pg_checksums --help synopsis is quite long

2019-08-23 Thread Fabien COELHO
Bonjour Michaël, It would like to keep "data checksums" in the output, You can do as you feel. I have decided that PostgreSQL is a mouthful, thus I'm rather using "Postgres". Changing that in one tool and not everything would of course be really silly. And if you want to bring up the ren

Re: BUG #15977: Inconsistent behavior in chained transactions

2019-08-25 Thread Fabien COELHO
The following bug has been logged on the website: Bug reference: 15977 Logged by: mtlh kdvt Email address: emuser20140...@gmail.com PostgreSQL version: 12beta3 Operating system: Windows Description: When a ROLLBACK AND CHAIN command is executed in the implicit transaction

Re: refactoring - share str2*int64 functions

2019-08-26 Thread Fabien COELHO
ning stuff is simple enough, but I'm unclear about what to do with pg_uint64, which has a totally different signature. Should it be aligned? -- Fabien Coelho - CRI, MINES ParisTech

Re: pgbench - allow to create partitioned tables

2019-08-26 Thread Fabien COELHO
Just one suggestion --partition-method option should be made dependent on --partitions, because it has no use unless used with --partitions. What do you think? Why not. V4 attached. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index 816f9cc4c7..38f4ac1

Re: Creating partitions automatically at least on HASH?

2019-08-26 Thread Fabien COELHO
Hello Rafia, CREATE TABLE Stuff (...) PARTITION BY [HASH | RANGE | LIST] (…) DO NONE -- this is the default DO [IMMEDIATE|DEFERRED] USING (…) Where the USING part would be generic keword value pairs, eg: For HASH: (MODULUS 8) and/or (NPARTS 10) For RANGE: (START '1970-0

Re: refactoring - share str2*int64 functions

2019-08-26 Thread Fabien COELHO
Bonjour Michaël, The error returning stuff is simple enough, but I'm unclear about what to do with pg_uint64, which has a totally different signature. Should it be aligned? I am not sure what you mean with aligned here. I meant same signature. If you mean consistent, getting into a state w

Re: pgbench - implement strict TPC-B benchmark

2019-08-27 Thread Fabien COELHO
Hello Dmitry, Well, it could be added? While doing benchmarking using different tools, including pgbench, I found it useful as a temporary hack to add copy freeze and maintenance_work_mem options (the last one not as an env variable, just as a set before, although not sure if it's a best idea

Re: refactoring - share str2*int64 functions

2019-08-27 Thread Fabien COELHO
Bonjour Michaël, So I'll set out to write and use "pg_strtou?int64", i.e. 2 functions, and then possibly generalize to lower sizes, 32, 16, depending on what is actually needed. I am interested in this patch, and the next commit fest is close by. Are you working on an updated version? If not

Re: refactoring - share str2*int64 functions

2019-08-28 Thread Fabien COELHO
Michaël, I have taken the liberty to optimize the existing int64 function by removing spurious tests. Which are? - *ptr && WHATEVER(*ptr) *ptr is redundant, WHATEVER yields false on '\0', and it costs on each char but at the end. It might be debatable in some places, e.g. it is likel

psql - improve test coverage from 41% to 88%

2019-08-28 Thread Fabien COELHO
Hello devs, The attached patch improves psql code coverage by adding a specific TAP test. The 1709 tests take 4 seconds CPU (6.3 elapsed time) on my laptop. The infrastructure is updated to require perl module "Expect", allowing to test interactive features such as tab completion and prompt

Re: refactoring - share str2*int64 functions

2019-08-28 Thread Fabien COELHO
Bonjour Michaël, - *ptr && WHATEVER(*ptr) *ptr is redundant, WHATEVER yields false on '\0', and it costs on each char but at the end. It might be debatable in some places, e.g. it is likely that there are no spaces in the string, but likely that there are more than one digit. Still t

Re: BUG #15977: Inconsistent behavior in chained transactions

2019-08-28 Thread Fabien COELHO
Hello, COMMIT AND CHAIN in implicit block leaves blockState as TBLOCK_STARTED, which doesn't trigger the chaining. but ROLLBACK AND CHAIN sets the blockState into TBLOCK_ABORT_PENDING, so the chaining is triggered. I think disabling s->chain beforehand should do the desired behavior. Patc

Re: BUG #15977: Inconsistent behavior in chained transactions

2019-08-29 Thread Fabien COELHO
Patch works for me, and solution seems appropriate. It should be committed for pg 12.0. I have listed this as an open issue of the upcoming pg 12: https://wiki.postgresql.org/wiki/PostgreSQL_12_Open_Items#Open_Issues -- Fabien.

Re: BUG #15977: Inconsistent behavior in chained transactions

2019-08-29 Thread Fabien COELHO
Hello, Added two kinds of test for the implicit transaction: in single query and in implicit block. Ok. The patch file is now created with Unix-style line ending (LF). Thanks. Patch applies and compiles cleanly. However, "make check" is not ok on the added tests. SHOW transaction_re

Re: BUG #15977: Inconsistent behavior in chained transactions

2019-08-29 Thread Fabien COELHO
Hello, transaction_read_only must be 'on' because AND CHAIN test sets the default_transaction_read_only to 'on'. Failure of this test means that the transaction was chained from an implicit transaction, which is not our desired behavior. Perhaps you are using a wrong binary? Nope, I blind

Re: BUG #15977: Inconsistent behavior in chained transactions

2019-08-29 Thread Fabien COELHO
Thanks, I got it. I have never made a patch before so I'll keep it in my mind. Self-contained patch is now attached. v3 applies, compiles, "make check" ok. I turned it ready on the app. -- Fabien

Re: pg_upgrade: Error out on too many command-line arguments

2019-08-29 Thread Fabien COELHO
Hello Peter, On Sun, Aug 25, 2019 at 05:10:47PM +0200, Julien Rouhaud wrote: > I did some searching, and oid2name.c is also missing this. And pgbench, no?   Yes, the patch is slightly different. Thanks, pushed all that together. Great! Could we maintain coverage by adding a TAP

Re: pg_upgrade: Error out on too many command-line arguments

2019-08-30 Thread Fabien COELHO
Bonjour Michaël, I don't see why not. Perhaps this could be done for pgbench and oid2name as well? This is for pgbench. I did not found a TAP test in pg_upgrade, I do not think that it is worth creating one for that purpose. The "test.sh" script does not seem appropriate for this kind of

Re: refactoring - share str2*int64 functions

2019-08-30 Thread Fabien COELHO
Michaël, attached is a first cut that I would like to commit separately which adds all the compatibility overflow routines to int.h for uint16, uint32 and uint64 with all the fallback implementations (int128-based method added as well if available). I have also grouped at the top of the file

Re: WIP: Generic functions for Node types using generated metadata

2019-08-30 Thread Fabien COELHO
Hello Andres, Just my 0.02 €: There's been a lot of complaints over the years about how annoying it is to keep the out/read/copy/equalfuncs.c functions in sync with the actual underlying structs. There've been various calls for automating their generation, but no actual patches that I am awar

Re: WIP: Generic functions for Node types using generated metadata

2019-08-30 Thread Fabien COELHO
Hallo Andres, There've been various calls for automating their generation, but no actual patches that I am aware of. I started something a while back I have found this thread: https://www.postgresql.org/message-id/flat/E1cq93r-0004ey-Mp%40gemulon.postgresql.org It seems that comments from

Re: pg_upgrade: Error out on too many command-line arguments

2019-08-30 Thread Fabien COELHO
Hello Tom, Could we maintain coverage by adding a TAP test? See 1 liner attached. Is this issue *really* worth expending test cycles on forevermore? With this argument consistently applied, postgres code coverage is consistently weak, with 25% of the code never executed, and 15% of functi

Re: refactoring - share str2*int64 functions

2019-08-30 Thread Fabien COELHO
Michaël, For symmetry and efficiency, ISTM that uint16 mul overflow could use uint32 and uint32 could use uint64, and the division-based method be dropped in these cases. Yes, the division would be worse than the other. What do you think about using the previous module I sent and measure how

Re: pg_upgrade: Error out on too many command-line arguments

2019-08-30 Thread Fabien COELHO
Is this issue *really* worth expending test cycles on forevermore? With this argument consistently applied, postgres code coverage is consistently weak, with 25% of the code never executed, and 15% of functions never called. "psql" is abysmal, "libpq" is really weak. It's all a question of

Re: refactoring - share str2*int64 functions

2019-09-01 Thread Fabien COELHO
Bonjour Michaël, You are right as well that having symmetry with the signed methods is much better. In order to see the difference, you can just do that with the extension attached, after of course hijacking int.h with some undefs and recompiling the backend and the module: select pg_overfl

Re: refactoring - share str2*int64 functions

2019-09-01 Thread Fabien COELHO
Michaël, I would put back unlikely() on overflow tests, as there are indeed unlikely to occur and it may help some compilers, and cannot be harmful. It also helps the code reader to know that these path are not expected to be taken often. Hm. I don't agree about that part, and the original s

Re: moonjelly vs tsearch

2019-09-01 Thread Fabien COELHO
News from Fabien's bleeding edge compiler zoo: apparently GCC 10 20190831 thinks the tsearch test should produce different output than 20190824 did. It looks like the parsing of question marks change... Indeed, that looks pretty strange. I can't wait for next week's installment. I'll re

Re: pg_dump --exclude-* options documentation

2019-09-01 Thread Fabien COELHO
We have the following options in pg_dump: --exclude-schema=schema --exclude-table=table --exclude-table-data=table and new in pg_dumpall: --exclude-database=pattern I was momentarily confused that the latter is documented as taking a "pattern" but the others are not. Of course they all take

Re: pgbench - rework variable management

2019-09-02 Thread Fabien COELHO
Some windows-specific hacks are note tested. Somehow this macro hackery has upset the Windows socket headers: https://ci.appveyor.com/project/postgresql-cfbot/postgresql/build/1.0.55019 I noticed, but I do not have any windows host so I cannot test locally. The issue is how to do a mutex

Re: psql - improve test coverage from 41% to 88%

2019-09-02 Thread Fabien COELHO
Attached is a rebase after TestLib.pm got a documentation in 6fcc40b1. The attached patch improves psql code coverage by adding a specific TAP test. The 1709 tests take 4 seconds CPU (6.3 elapsed time) on my laptop. The infrastructure is updated to require perl module "Expect", allowing to t

Re: BUG #15977: Inconsistent behavior in chained transactions

2019-09-03 Thread Fabien COELHO
v3 applies, compiles, "make check" ok. I turned it ready on the app. Should we make it an error instead of a warning? ISTM that it made sense to have the same behavior as out of transaction COMMIT or ROLLBACK. -- Fabien.

Re: pgbench - rework variable management

2019-09-03 Thread Fabien COELHO
Hello Thomas, I noticed, but I do not have any windows host so I cannot test locally. The issue is how to do a mutex on Windows, which does not have pthread so it has to be emulated. I'll try again by sending a blind update to the patch and see how it goes. If you have the patience and a git

Re: refactoring - share str2*int64 functions

2019-09-03 Thread Fabien COELHO
This part has been committed, now let's move to the next parts of your patch. Attached a rebased version which implements the int64/uint64 stuff. It is basically the previous patch without the overflow inlined functions. -- Fabien Coelho - CRI, MINES ParisTechdiff --git a/co

Re: [HACKERS] pgbench randomness initialization

2018-01-02 Thread Fabien COELHO
Hello Alvaro, I revive this patch because controlling the seed is useful for tap testing pgbench. The output should include the random seed used, whether it was passed with --random-seed, environment variable or randomly determined. That way, the user that later wants to verify why a parti

Re: [HACKERS] pgbench randomness initialization

2018-01-02 Thread Fabien COELHO
Here is a new version which output use used seed when a seed is explicitely set with an option or from the environment. It is even better without xml typos, with simpler coding and the doc in the right place... Sorry for the noise. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/

Re: WIP Patch: Pgbench Serialization and deadlock errors

2018-01-03 Thread Fabien COELHO
Hello Marina, Some comment about WIP pgbench error handling v4. Patch applies, compiles, global & local "make check" are ok. doc compiles. I'm generally okay with having such a feature, but I'd like it to be *MUCH* simpler, otherwise it is going to make pgbench unmaintainable:-( Also, ISTM

Re: [HACKERS] pgbench more operators & functions

2018-01-03 Thread Fabien COELHO
SQL doesn't evaluate unneeded arguments: Here is a version with some lazy evaluation for and, or & case. v23 is a rebase. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index 1519fe7..3dd492c 100644 --- a/doc/src/sgml/ref/pgbench.sgml +++ b/doc/src/sgml

Re: pgbench - add \if support

2018-01-03 Thread Fabien COELHO
Another rebase to try to please the patch tester. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index 1519fe7..7068063 100644 --- a/doc/src/sgml/ref/pgbench.sgml +++ b/doc/src/sgml/ref/pgbench.sgml @@ -895,6 +895,21 @@ pgbench options d + +

Re: pgbench - add \if support

2018-01-08 Thread Fabien COELHO
Hello Dmitry, Thanks for working on this. I had just a quick look at it, but I hope I'll have time to post a proper review. In the meantime I'm wondering what am I doing wrong here (I see a similar example in your first message)? ``` -- test.sql \if random(0, 99) < 85 \set test 1 \else

Re: Re: [HACKERS] pgbench randomness initialization

2018-01-09 Thread Fabien COELHO
Hello Chapman, Thanks for the review, The tests assume that stdlib random/srandom behavior is standard thus deterministic between platform. Is the behavior of srandom() and the system generator really so precisely specified that seed 5432 will produce the same values hardcoded in the tests o

Re: pgbench - add \if support

2018-01-09 Thread Fabien COELHO
Hello Dmitry, What you are trying to do is the result of combining the pgbench-if patch and the pgbench-more-ops-and-funcs patch. Oh, I see. I missed the first message about this patch, sorry. But for some reason I can't apply both patches (pgbench-more-ops-funcs-23.patch and pgbench-if-4.pat

Re: pgbench - add \if support

2018-01-09 Thread Fabien COELHO
Hello Vik, This patch adds a \if construct so that an expression on variables, possibly with data coming from the database, can change the behavior of a script. I have given this patch a pretty good shake and I'm happy with it. I did not test it with the other two patches, only on its own.

Re: [HACKERS] pgbench more operators & functions

2018-01-09 Thread Fabien COELHO
Thank you, pushed Thanks! -- Fabien.

Re: General purpose hashing func in pgbench

2018-01-09 Thread Fabien COELHO
Hello Ildar, Sorry for a long delay. I've added hash() function which is just an alias for murmur2. I've also utilized variable arguments feature from least()/greates() functions to make optional seed parameter, but I consider this as a hack. Patch needs a rebase after Teodor push for a set o

Re: [HACKERS] pgbench more operators & functions

2018-01-10 Thread Fabien COELHO
Some of the Windows buildfarm members aren't too happy with this. Indeed. Windows prettyprinting of double inserts a spurious "0" at the beginning of the exponent. Makes it look like an octal. Here is a patch to fix it, which I cannot test on Windows. -- Fabien.diff --git a/src/bin/pgbenc

Re: pgbench - add \if support

2018-01-10 Thread Fabien COELHO
A new automaton state is added to quickly step over false branches. This one took me a little while to understand while reading the patch, but mostly because of how diff doesn't handle moving things around. "git diff -w --patience" may help. Marking as ready for committer. Here is a reba

Re: pgbench - add \if support

2018-01-10 Thread Fabien COELHO
Here is a rebase. I made some tests use actual expressions instead of just 0 and 1. No other changes. Sigh. Better with the attachment. Sorry for the noise. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index 3dd492c..c203c41 100644 --- a/doc/src/sgml/re

Re: [HACKERS] pgbench - allow to store select results into variables

2018-01-10 Thread Fabien COELHO
Here is a v14, after yet another rebase, and some comments added to answer your new comments. Attached v15 is a simple rebase after Teodor push of new functions & operators in pgbench. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index 3dd492c..b5139

Re: Re: [HACKERS] pgbench randomness initialization

2018-01-10 Thread Fabien COELHO
This is a simple patch that does what it says on the tin. I ran into trouble with the pgbench TAP test *even before applying the patch*, but only because I was doing a VPATH build as a user without 'write' on the source tree (001_pgbench_with_server.pl tried to make pgbench create log files ther

Re: Re: [HACKERS] pgbench randomness initialization

2018-01-10 Thread Fabien COELHO
Here is a rebase, plus some more changes: I have improved the error message to tell from where the value was provided. I have removed the test to the exact values produced from the expression test run. I have added a test which run from the same seed value several times and checks that the

Re: pgsql: Improve scripting language in pgbench

2018-01-10 Thread Fabien COELHO
Hello Teodor, I just noticed while rebasing stuff that there is some crust in "pgbench/t/001_pgbench_with_server.pl" coming from this patch: +=head + +} }); + +=cut I cannot find any use for these lines which are ignored by perl execution anyway. It may be some leftovers from debugging

Re: General purpose hashing func in pgbench

2018-01-10 Thread Fabien COELHO
Hello Ildar, Patch needs a rebase after Teodor push for a set of pgbench functions. Done. Congratulations on your patch finally being committed : ) Over 21 months... I hope that pgbench will have hash functions sooner:-) Should we probably add some infrastructure for optional arguments?

Re: General purpose hashing func in pgbench

2018-01-10 Thread Fabien COELHO
Patch needs a rebase after Teodor push for a set of pgbench functions. Done. Congratulations on your patch finally being committed : ) I forgot: please provide a doc & some coverage tests as well! -- Fabien.

Re: bytea bitwise logical operations implementation (xor / and / or / not)

2018-01-11 Thread Fabien COELHO
Hello Christian, Currently, `bytea` does not have any bitwise logical operations yet. This issue came up in an old thread from 2006 [1], but nobody seemed to have picked this issue so far. I remember this one because I needed them for checksuming set of rows. There is a whole set of missing

Re: WIP Patch: Pgbench Serialization and deadlock errors

2018-01-12 Thread Fabien COELHO
Hello Marina, A partial answer, to focus on how the feature may be simplified. I apologise as it seems that I overlooked one of your mail. Changing the thread has not helped. I'm wondering whether the whole feature could be simplified by considering that one script is one "transaction" (it

Re: General purpose hashing func in pgbench

2018-01-12 Thread Fabien COELHO
Hello Ildar, Hmm. I do not think that we should want a shared seed value. The seed should be different for each call so as to avoid undesired correlations. If wanted, correlation could be obtained by using an explicit identical seed. Probably I'm missing something but I cannot see the point.

Re: General purpose hashing func in pgbench

2018-01-12 Thread Fabien COELHO
Hmm. I do not think that we should want a shared seed value. The seed should be different for each call so as to avoid undesired correlations. If wanted, correlation could be obtained by using an explicit identical seed. ISTM that the best way to add the seed is to call random() when the second

Re: WIP Patch: Pgbench Serialization and deadlock errors

2018-01-12 Thread Fabien COELHO
Hello Marina, If you want 2 transactions, then you have to put them in two scripts, which looks fine with me. Different transactions are expected to be independent, otherwise they should be merged into one transaction. Therefore if the script consists of several single statements (= several

Re: pgbench - add \if support

2018-01-12 Thread Fabien COELHO
Hm, isn't already commited when/case/then/else syntax do the same? No, not strictly. The "CASE WHEN" is an if *within* an expression: \set i CASE WHEN condition THEN val1 ELSE val2 END The \if is at the script level, like psql already available version, which can change what SQL is sent.

Re: General purpose hashing func in pgbench

2018-01-13 Thread Fabien COELHO
Hello Ildar, so that different instances of hash function within one script would have different seeds. Yes, that is a good idea, I can do that. Added this feature in attached patch. But on a second thought this could be something that user won't expect. For example, they may want to run pgbe

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-01-13 Thread Fabien COELHO
Hello Tom, Thanks for having a look at this attempt. Attached is an attempt at improving the situation: I took a quick look at this and can't really convince myself that it improves our situation. The fact that it prints nothing if the runtime is outside of a tightly constrained range seems

Re: Minor fix for pgbench documentation

2018-01-13 Thread Fabien COELHO
Here is a patch that adds missing random_zipfian func to the paragraph in pgbench documentation about random functions parameterization. Indeed. Patch applies cleanly, doc build ok. Marked as "ready". I have added it to the next commitfest. -- Fabien.

Re: Implementing SQL ASSERTION

2018-01-15 Thread Fabien COELHO
Hello Joe, Just a reaction to the example, which is maybe addressed in the patch which I have not investigated. * certain combinations of aggregates with comparison operations cannot be invalidating. As an example of the last point, the expression "CHECK (10 > (SELECT COUNT(*) FROM t))" c

Re: Implementing SQL ASSERTION

2018-01-15 Thread Fabien COELHO
I'm wondering about the effect of MVVC on this: if the check is performed when the INSERT is done, concurrent inserting transactions would count the current status which would be ok, but on commit all concurrent inserts would be there and the count could not be ok anymore? The patch doesn’t

Re: pgbench - add \if support

2018-01-15 Thread Fabien COELHO
Here is a rebase. I made some tests use actual expressions instead of just 0 and 1. No other changes. Sigh. Better with the attachment. Sorry for the noise. Here is a very minor rebase. -- Fabien.diff --git a/doc/src/sgml/ref/pgbench.sgml b/doc/src/sgml/ref/pgbench.sgml index 3dd492c..c203c

Re: General purpose hashing func in pgbench

2018-01-16 Thread Fabien COELHO
Hello Ildar, Here is a new version of patch. I've splitted it into two parts. The first one is almost the same as v4 from [1] with some refactoring. The second part introduces random_seed variable as you proposed. Patch 1 applies. Compilations fails, there are two "hash_seed" declared in "p

Re: Setting BLCKSZ 4kB

2018-01-17 Thread Fabien COELHO
Hello, What are the cons of setting BLCKSZ as 4kB? When saw the results published on [...]. There were other posts and publications which points to the same direction consistently. This matches my deep belief is that postgres default block size is a reasonable compromise for HDD, but is l

Re: General purpose hashing func in pgbench

2018-01-18 Thread Fabien COELHO
Hello Ildar, Patch v8 applies cleanly and compiles. Global and local "make check ok". Doc build ok. For me "random seed" is what is passed to srandom, so the variable should rather be named hash_seed because there could also be a random seed (actually, there is in another parallel patch:-). Mo

Re: [HACKERS] Re: [COMMITTERS] pgsql: Remove pgbench "progress" test pending solution of its timing is (fwd)

2018-01-18 Thread Fabien COELHO
Hello Tom, From my previous answer, to motivate these tests: The compromise I'm proposing is to skip time-related stuff if too slow. The value I see is that it provides coverage for all time-related features. I can also add a check that the run is *at least* 2 seconds when two seconds are

Re: Re: [HACKERS] pgbench randomness initialization

2018-01-18 Thread Fabien COELHO
Here is a rebase, plus some more changes: I have improved the error message to tell from where the value was provided. I have removed the test to the exact values produced from the expression test run. I have added a test which run from the same seed value several times and checks that the

Re: pgbench - add \if support

2018-01-21 Thread Fabien COELHO
Helo Craig, I'm a smidge worried about this. It seems like psql is growing a scripting language. The patch is about aligning pgbench with psql, which already has \if. Do we want to go our own way with a kind of organically grown scripting system? Or should we be looking at embedding client-

Re: pgbench - add \if support

2018-01-22 Thread Fabien COELHO
few scripting features doesn't mean scripting language. \if in psql is nice feature that reduce duplicate code, unreadable code, and helps with deployment and test scripts. pgbench and psql should to have similar environment - and I am thinking so \if should be there. Using Lua is not bad idea

Re: PATCH: pgbench - option to build using ppoll() for larger connection counts

2018-01-24 Thread Fabien COELHO
Hello Doug, This version of the patch attempts to address the feedback for the previous submission on 28-Nov-2017 Please avoid recreating a thread, but rather respond to the previous one, I missed this post. The overall function-based implementation with limited ifdefs seems readable and

Re: Re: [HACKERS] pgbench randomness initialization

2018-01-24 Thread Fabien COELHO
Here is a rebase, plus some more changes: I have improved the error message to tell from where the value was provided. I have removed the test to the exact values produced from the expression test run. I have added a test which run from the same seed value several times and checks that th

Re: PATCH: pgbench - option to build using ppoll() for larger connection counts

2018-01-25 Thread Fabien COELHO
ISTM that the v7 patch version you sent is identical to v6. -- Fabien.

Re: PATCH: pgbench - option to build using ppoll() for larger connection counts

2018-01-25 Thread Fabien COELHO
Hello Doug, This time with the revised patch file: pgbench11-ppoll-v8.patch Patch applies cleanly. Compiles cleanly and runs fine in both ppoll & select cases. I'm okay with having a preferred ppoll implementation because of its improved capability. A few minor additional comments/sugges

Re: General purpose hashing func in pgbench

2018-01-25 Thread Fabien COELHO
Hello Ildar, Applies, compiles, runs. I still have a few very minor comments, sorry for this (hopefully) last iteration from my part. I'm kind of iterative... The XML documentation source should avoid a paragraph on one very long line, but rather be indented like other sections. I'd propo

Re: PATCH: pgbench - option to build using ppoll() for larger connection counts

2018-01-26 Thread Fabien COELHO
Hello Doug, Patch applies, compiles, tests ok. > [...] Replaced USE_PPOLL with HAVE_PPOLL as having both seems redundant. I'm okay with that. I'm wondering whether there should be a way to force using one or the other when both are available. Not sure. Added option to force use of s

Re: General purpose hashing func in pgbench

2018-01-28 Thread Fabien COELHO
Hello Ildar, I did everything you mention here and attached a new version on the patch. Patch applies, compiles, runs ok. Alas, I still have a few more very minor comments about the doc, sorry again: +default_seed + random seed used in hash functions by default s/rando

Re: Best way to select a random row from a derived table

2018-01-28 Thread Fabien COELHO
I am aware of 2 ways to select a random row from a table: 1) select * from table_name order by random() limit 1; -- terribly inefficient 2) select * from table_name tablesample system_rows(1) limit 1; -- only works on tables, not views or subqueries Is there an option that

Re: PATCH: pgbench - option to build using ppoll() for larger connection counts

2018-01-29 Thread Fabien COELHO
Hello Doug, I'm not sure why you do the following trick, could you explain? +#undef USE_SELECT +#define USE_SELECT This was due to compiler complaint about USE_SELECT being redefined. Have replaced that "trick" with a new #define POLL_USING_SELECT which is used elsewhere in pgb

Re: General purpose hashing func in pgbench

2018-01-29 Thread Fabien COELHO
Hello Ildar, Fixed the doc, attached the patch. Thanks! Patch applies, compiles, pgbench & global "make check" ok, doc built ok. Ok for me, switched to "Ready". -- Fabien.

Re: PATCH: pgbench - break out timing data for initialization phases

2018-01-29 Thread Fabien COELHO
Hello Doug, With patch and ‘-I dtgvpf’ options: pgrun pgbench -i -s 2000 -F 90 -q -I dtgvpf dropping old tables... creating tables... generating data... … 2 of 2 tuples (100%) done (elapsed 168.76 s, remaining 0.00 s) vacuuming... creating primary keys... creating foreign keys..

Re: csv format for psql

2018-01-30 Thread Fabien COELHO
Hello Daniel, This patch implements csv as an output format in psql (\pset format csv). Would you consider registering it in the next CF? -- Fabien.

Re: [PATCH] pgbench - refactor some connection finish/null into common function

2018-01-30 Thread Fabien COELHO
Hello Doug, This patch refactors all of the connection state PQfinish() and NULL’ing into a single function. Excludes PQfinish() in doConnect(). My 0.02€: The argument could be "PGconn **" instead of a "CState *"? If so, it may be used in a few more places. What is your opinion? I'm fine w

Re: pgbench - extend initialization phase control

2019-10-28 Thread Fabien COELHO
Hello Masao-san, Maybe. If you cannot check, you can only guess. Probably it should be small, but the current version does not allow to check whether it is so. Could you elaborate what you actually want to measure the performance impact by adding explicit begin and commit? Currently pgbench -

<    1   2   3   4   5   6   7   8   9   10   >