Re: [HACKERS] Doubt in pgbench TPS number

2015-09-29 Thread Fabien COELHO
Hello Tatsuo, So on second thought the formula should rather be: ... / (is_connect? nclients: nthreads) I don't think this is quite correct. If is_connect is false, then following loop is executed in threadRun(): /* make connections to the database */ for

Re: [HACKERS] Doubt in pgbench TPS number

2015-09-29 Thread Tatsuo Ishii
>> FWIW, I vote with Tatsuo-san. Such a change will break comparability of >> results with all previous versions, which means it's not something to do >> in minor releases, even if we now believe the previous results were >> somewhat bogus. Arguing that it's "not a behavioral change" seems >> qui

Re: [HACKERS] Doubt in pgbench TPS number

2015-09-29 Thread Tatsuo Ishii
>> Here conn_total_time is the sum of time to establish connection to >> PostgreSQL. Since establishing connections to PostgreSQL is done in >> serial rather than in parallel, conn_total_time should have been >> divided by nclients. > > After some more thinking and looking again at the connection

Re: [HACKERS] Doubt in pgbench TPS number

2015-09-28 Thread Fabien COELHO
(Note: I've not read the patch, so this is not an opinion about its correctness.) As Fabien anayzed the problem was that pgbench simply uses wrong variable: nthreads (number of threads, specified by "-j" option) vs. nclients (number of clients, specified by "-c" option). @@ -2616,7 +2616,7 @@

Re: [HACKERS] Doubt in pgbench TPS number

2015-09-28 Thread Tatsuo Ishii
> FWIW, I vote with Tatsuo-san. Such a change will break comparability of > results with all previous versions, which means it's not something to do > in minor releases, even if we now believe the previous results were > somewhat bogus. Arguing that it's "not a behavioral change" seems > quite lo

Re: [HACKERS] Doubt in pgbench TPS number

2015-09-28 Thread Fabien COELHO
Hello Tom, FWIW, I vote with Tatsuo-san. Such a change will break comparability of results I would not classify a performance measure as a "result compatibility" issue. What matters is the *correction* of the results. When a bug is fixed anywhere in pg it may change performance significan

Re: [HACKERS] Doubt in pgbench TPS number

2015-09-28 Thread Tom Lane
Fabien COELHO writes: >> Yeah, that's definitely a bug but I'm afraid the fix will change the >> TPS number and may break the backward compatibility. Since we have >> lived with bug for years, I hesitate to back port to older stable >> branches... > My 2¥: I do not think of a good argument to kee

Re: [HACKERS] Doubt in pgbench TPS number

2015-09-28 Thread Tatsuo Ishii
>> Yeah, that's definitely a bug but I'm afraid the fix will change the >> TPS number and may break the backward compatibility. Since we have >> lived with bug for years, I hesitate to back port to older stable >> branches... > > My 2¥: I do not think of a good argument to keep wrong tps numbers >

Re: [HACKERS] Doubt in pgbench TPS number

2015-09-28 Thread Fabien COELHO
#5 0x00402b2b in doConnect () at pgbench.c:650 #6 0x00404591 in doCustom (thread=0x25c2f40, st=0x25c2770, conn_time=0x25c2f90, logfile=0x0, agg=0x7fffba224260) at pgbench.c:1353 #7 0x0040a1d5 in threadRun (arg=0x25c2f40) at pgbench.c:3581 #8 0x00409ab4 in m

Re: [HACKERS] Doubt in pgbench TPS number

2015-09-28 Thread Tatsuo Ishii
>> I'm not sure about this. I think pgbench will not start transactions >> until all clients establish connections to PostgreSQL. > > I think that is true if "!is_connect", all client connections are > performed at the beginning of threadRun, but under -C each client has > its own connect/deconnec

Re: [HACKERS] Doubt in pgbench TPS number

2015-09-28 Thread Fabien COELHO
Hello Tatsuo-san, I think that the degree of parallelism to consider is nclients, not nthreads: while connection time is accumulated in conn_time, other clients are possibly doing their transactions, in parallel, I'm not sure about this. I think pgbench will not start transactions until all c

Re: [HACKERS] Doubt in pgbench TPS number

2015-09-28 Thread Tatsuo Ishii
> I have tested your patch. It seems the tolerance is much better than > before with your patch. [snip] > I'm going to commit your patch if there's no objection. I think we should commit this to master and 9.5 stable branch only because the fix significantly changes the benchmark result of pgben

Re: [HACKERS] Doubt in pgbench TPS number

2015-09-27 Thread Tatsuo Ishii
> I think that the degree of parallelism to consider is nclients, not > nthreads: while connection time is accumulated in conn_time, other > clients are possibly doing their transactions, in parallel, I'm not sure about this. I think pgbench will not start transactions until all clients establish

Re: [HACKERS] Doubt in pgbench TPS number

2015-09-25 Thread Fabien COELHO
Hello Tatsuo, Hmmm... I never use -C. The formula seems ok: tps_exclude = normal_xacts / (time_include - (INSTR_TIME_GET_DOUBLE(conn_total_time) / nthreads)); Hmmm... it is not:-) I think that the degree of parallelism to consider is nclients, not nthreads: while

Re: [HACKERS] Doubt in pgbench TPS number

2015-09-24 Thread Fabien COELHO
Hello Tatsuo-san, $ pgbench -C -n -p 11002 -c 10 -T 30 -f test.sql test tps = 95.799114 (including connections establishing) tps = 124.487149 (excluding connections establishing) Interesting thing is, the number from "excluding connections establishing". 124.487149 tps means 0.008032 seconds

[HACKERS] Doubt in pgbench TPS number

2015-09-24 Thread Tatsuo Ishii
Today I got an interesting output from pgbench. The scenario is executing SELECT pg_sleep(0.1). $ cat test.sql select pg_sleep(0.1); $ pgbench -C -n -p 11002 -c 10 -T 30 -f test.sql test transaction type: Custom query scaling factor: 1 query mode: simple number of clients: 10 number of threads: