Re: benchmark results comparing versions 15.2 and 16

2023-07-07 Thread Alexander Lakhin
Hello David, 11.05.2023 16:00, Alexander Lakhin wrote: Yeah, I see. It's also interesting to me, which tests perform better after that commit. It takes several hours to run all tests, so I can't present results quickly, but I'll try to collect this information next week. To my regret, I could

Re: benchmark results comparing versions 15.2 and 16

2023-05-31 Thread Peter Geoghegan
Hi Mark, On Tue, May 30, 2023 at 1:03 PM MARK CALLAGHAN wrote: > Do you want me to try PG 16 without ICU or PG 15 with ICU? > I can do that, but it will take a few days before the server is available. Sorry for the late reply. Most of the Postgres developers (myself included) are attending pgCon

Re: benchmark results comparing versions 15.2 and 16

2023-05-30 Thread MARK CALLAGHAN
Do you want me to try PG 16 without ICU or PG 15 with ICU? I can do that, but it will take a few days before the server is available. On Mon, May 29, 2023 at 9:55 AM Peter Geoghegan wrote: > On Sun, May 28, 2023 at 2:42 PM David Rowley wrote: > > c6e0fe1f2 might have helped improve some of that

Re: benchmark results comparing versions 15.2 and 16

2023-05-29 Thread Peter Geoghegan
On Sun, May 28, 2023 at 2:42 PM David Rowley wrote: > c6e0fe1f2 might have helped improve some of that performance, but I > suspect there must be something else as ~3x seems much more than I'd > expect from reducing the memory overheads. Testing versions before > and after that commit might give

Re: benchmark results comparing versions 15.2 and 16

2023-05-28 Thread David Rowley
On Tue, 23 May 2023 at 07:40, MARK CALLAGHAN wrote: (pg15) > --- Q2.10k : explain analyze SELECT c FROM sbtest1 WHERE id BETWEEN 1000 > AND 1001 order by c; > QUERY PLAN > -

Re: benchmark results comparing versions 15.2 and 16

2023-05-27 Thread MARK CALLAGHAN
Results for 16 beta1 are similar to what I shared above: * no regressions * a few queries are >= 1.5 times faster which make the read-only transaction >= 1.5 times faster http://smalldatum.blogspot.com/2023/05/postgres-16beta1-looks-good-vs-sysbench.html -- Mark Callaghan mdcal...@gmail.com

Re: benchmark results comparing versions 15.2 and 16

2023-05-22 Thread MARK CALLAGHAN
I ran sysbench on Postgres 15.2, 15.3 and 16 prebeta at git sha 1c006c0 (built on May19). The workload was in-memory on a small server (8 cores, 16G RAM) and the workload had 1 connection (no concurrency). For some details on past benchmarks like this see: http://smalldatum.blogspot.com/2023/03/sea

Re: benchmark results comparing versions 15.2 and 16

2023-05-20 Thread MARK CALLAGHAN
On Fri, May 19, 2023 at 4:04 PM Andres Freund wrote: > With "yet to see any significant changes" do you mean that the runs are > comparable with earlier runs, showing the same regression? Or that the > regression vanished? Or ...? > I mean that I might be chasing noise and the mean+stddev for th

Re: benchmark results comparing versions 15.2 and 16

2023-05-19 Thread Andres Freund
Hi, On 2023-05-19 15:44:09 -0700, MARK CALLAGHAN wrote: > On Tue, May 9, 2023 at 10:36 AM MARK CALLAGHAN wrote: > > > > > > > On Fri, May 5, 2023 at 10:01 PM MARK CALLAGHAN wrote: > > > >> I have two more runs of the benchmark in progress so we will have 3 > >> results for each of the test case

Re: benchmark results comparing versions 15.2 and 16

2023-05-19 Thread MARK CALLAGHAN
On Tue, May 9, 2023 at 10:36 AM MARK CALLAGHAN wrote: > > > On Fri, May 5, 2023 at 10:01 PM MARK CALLAGHAN wrote: > >> I have two more runs of the benchmark in progress so we will have 3 >> results for each of the test cases to confirm that the small regressions >> are repeatable. >> > I repeate

Re: benchmark results comparing versions 15.2 and 16

2023-05-17 Thread Alexander Lakhin
17.05.2023 04:25, Michael Paquier wrote: The numbers between f193883fc~1 and HEAD+patch are close to each other. It does not seem to make the whole difference with 15.3, but most of it. The difference can also be explained with some noise, based on the number patterns of the third runs? I have

Re: benchmark results comparing versions 15.2 and 16

2023-05-16 Thread Michael Paquier
On Tue, May 16, 2023 at 06:00:00PM +0300, Alexander Lakhin wrote: > I can confirm that the patches improve (restore) performance for my test: > pgbench -i benchdb > pgbench -c 10 -T 300 benchdb Thanks for running these! > tps (over three runs): > HEAD (08c45ae23): > 10238.441580, 10697.202119, 10

Re: benchmark results comparing versions 15.2 and 16

2023-05-16 Thread Alexander Lakhin
15.05.2023 08:20, Michael Paquier wrote: I am not going to let that hang in the air with beta1 getting released next week, though, so attached are two patches to revert the change (these would be combined, just posted this way for clarity). I can confirm that the patches improve (restore) perfo

Re: benchmark results comparing versions 15.2 and 16

2023-05-15 Thread Michael Paquier
On Mon, May 15, 2023 at 05:54:53PM -0700, Andres Freund wrote: > Yes. numactl --physcpubind ... in my case. Linux has an optimization where it > does not need to send an IPI when the client and server are scheduled on the > same core. For single threaded ping-pong tasks like pgbench -c1, that can

Re: benchmark results comparing versions 15.2 and 16

2023-05-15 Thread Andres Freund
Hi, On 2023-05-16 09:42:31 +0900, Michael Paquier wrote: > > I get quite variable performance if I don't pin client / server to the same > > core, but even the slow performance is faster than 45k. > > Okay. You mean with something like taskset or similar, I guess? Yes. numactl --physcpubind ...

Re: benchmark results comparing versions 15.2 and 16

2023-05-15 Thread Michael Paquier
On Mon, May 15, 2023 at 05:14:47PM -0700, Andres Freund wrote: > On 2023-05-15 14:20:24 +0900, Michael Paquier wrote: >> On Thu, May 11, 2023 at 01:28:40PM +0900, Michael Paquier wrote: >> Extreme is adapted for a worst-case scenario. Looking at my notes >> from a few months back, that's kind of w

Re: benchmark results comparing versions 15.2 and 16

2023-05-15 Thread Andres Freund
Hi, On 2023-05-15 14:20:24 +0900, Michael Paquier wrote: > On Thu, May 11, 2023 at 01:28:40PM +0900, Michael Paquier wrote: > > On Tue, May 09, 2023 at 09:48:24AM -0700, Andres Freund wrote: > >> On 2023-05-08 12:11:17 -0700, Andres Freund wrote: > >>> I can reproduce a significant regression due

Re: benchmark results comparing versions 15.2 and 16

2023-05-14 Thread Michael Paquier
On Thu, May 11, 2023 at 01:28:40PM +0900, Michael Paquier wrote: > On Tue, May 09, 2023 at 09:48:24AM -0700, Andres Freund wrote: >> On 2023-05-08 12:11:17 -0700, Andres Freund wrote: >>> I can reproduce a significant regression due to f193883fc of a workload just >>> running >>> SELECT CURRENT_TIM

Re: benchmark results comparing versions 15.2 and 16

2023-05-11 Thread Alexander Lakhin
11.05.2023 01:27, David Rowley wrote: On Thu, 11 May 2023 at 01:00, Alexander Lakhin wrote: This time `git bisect` pointed at 3c6fc5820. Having compared execution plans (both attached), I see the following differences (3c6fc5820~1 vs 3c6fc5820): Based on what you've sent, I'm uninspired to wan

Re: benchmark results comparing versions 15.2 and 16

2023-05-10 Thread Michael Paquier
On Tue, May 09, 2023 at 09:48:24AM -0700, Andres Freund wrote: > On 2023-05-08 12:11:17 -0700, Andres Freund wrote: >> I can reproduce a significant regression due to f193883fc of a workload just >> running >> SELECT CURRENT_TIMESTAMP; >> >> A single session running it on my workstation via pgbenc

Re: benchmark results comparing versions 15.2 and 16

2023-05-10 Thread David Rowley
On Thu, 11 May 2023 at 01:00, Alexander Lakhin wrote: > This time `git bisect` pointed at 3c6fc5820. Having compared execution plans > (both attached), I see the following differences (3c6fc5820~1 vs 3c6fc5820): Based on what you've sent, I'm uninspired to want to try to do anything about it. Th

Re: benchmark results comparing versions 15.2 and 16

2023-05-10 Thread Alexander Lakhin
08.05.2023 16:00, Alexander Lakhin wrote: ... Having compared 15.3 (56e869a09) with master (58f5edf84) I haven't seen significant regressions except a few minor ones. First regression observed with a simple pgbench test: Another noticeable, but not critical performance degradation is revealed b

Re: benchmark results comparing versions 15.2 and 16

2023-05-09 Thread MARK CALLAGHAN
On Fri, May 5, 2023 at 10:01 PM MARK CALLAGHAN wrote: > I have two more runs of the benchmark in progress so we will have 3 > results for each of the test cases to confirm that the small regressions > are repeatable. > They get similar results. Then I tried Linux perf but the hierarchical call s

Re: benchmark results comparing versions 15.2 and 16

2023-05-09 Thread Andres Freund
Hi, On 2023-05-08 12:11:17 -0700, Andres Freund wrote: > Hi, > > On 2023-05-08 16:00:01 +0300, Alexander Lakhin wrote: > > This difference is confirmed by multiple test runs. `git bisect` for this > > regression pointed at f193883fc. > > I can reproduce a significant regression due to f193883fc

Re: benchmark results comparing versions 15.2 and 16

2023-05-08 Thread Andres Freund
Hi, On 2023-05-08 16:00:01 +0300, Alexander Lakhin wrote: > This difference is confirmed by multiple test runs. `git bisect` for this > regression pointed at f193883fc. I can reproduce a significant regression due to f193883fc of a workload just running SELECT CURRENT_TIMESTAMP; A single session

Re: benchmark results comparing versions 15.2 and 16

2023-05-08 Thread Alexander Lakhin
Hello Mark, 05.05.2023 20:45, MARK CALLAGHAN wrote: This is mostly a hobby project for me - my other hobby is removing invasive weeds. I am happy to answer questions and run more tests, but turnaround for answers won't be instant. Getting results from Linux perf for these tests is on my TODO li

Re: benchmark results comparing versions 15.2 and 16

2023-05-05 Thread MARK CALLAGHAN
I have two more runs of the benchmark in progress so we will have 3 results for each of the test cases to confirm that the small regressions are repeatable. On Fri, May 5, 2023 at 1:34 PM Andres Freund wrote: > One thing that's somewhat odd is that there's very marked changes in l.i0's > p99 la

Re: benchmark results comparing versions 15.2 and 16

2023-05-05 Thread Andres Freund
Hi, On 2023-05-05 10:45:12 -0700, MARK CALLAGHAN wrote: > This is mostly a hobby project for me - my other hobby is removing > invasive weeds. Hah :) > Summary of the results: > * from r1 - insert-heavy (l.i0, l.i1) and create indexes (l.x) steps are > ~2% slower in PG 16 > * from r2 - create i

benchmark results comparing versions 15.2 and 16

2023-05-05 Thread MARK CALLAGHAN
Let me know if results like this shouldn't be posted here. This is mostly a hobby project for me - my other hobby is removing invasive weeds. I am happy to answer questions and run more tests, but turnaround for answers won't be instant. Getting results from Linux perf for these tests is on my TOD