Skip to site navigation (Press enter)

Re: v17 vs v16 performance comparison

Alexander Lakhin Mon, 02 Sep 2024 22:00:24 -0700

Hello Thomas,

02.08.2024 12:00, Alexander Lakhin wrote:


I had payed attention to:
Best pg-src-17--.* worse than pg-src-16--.* by 57.9 percents (225.11 > 142.52): 
pg_tpcds.query15
Average pg-src-17--.* worse than pg-src-16--.* by 55.5 percents (230.57 > 
148.29): pg_tpcds.query15
in May, performed `git bisect` for this degradation, that led me to commit
b7b0f3f27 [1].

This time I bisected the following anomaly:
Best pg-src-17--.* worse than pg-src-16--.* by 23.6 percents (192.25 > 155.58): 
pg_tpcds.query21
Average pg-src-17--.* worse than pg-src-16--.* by 25.1 percents (196.19 > 
156.85): pg_tpcds.query21
and to my surprise I got "b7b0f3f27 is the first bad commit".

Moreover, bisecting of another anomaly:
Best pg-src-17--.* worse than pg-src-16--.* by 24.2 percents (24269.21 > 
19539.89): pg_tpcds.query72
Average pg-src-17--.* worse than pg-src-16--.* by 24.2 percents (24517.66 > 
19740.12): pg_tpcds.query72
pointed at the same commit again.

...

But beside that, I've found a separate regression. Bisecting for this 
degradation:
Best pg-src-17--.* worse than pg-src-16--.* by 105.0 percents (356.63 > 
173.96): s64da_tpcds.query95
Average pg-src-17--.* worse than pg-src-16--.* by 105.2 percents (357.79 > 
174.38): s64da_tpcds.query95
pointed at f7816aec2.


Meanwhile I've bisected another degradation:
Best pg-src-17--.* worse than pg-src-16--.* by 11.3 percents (7.17 > 6.44): 
job.query6f
and came to the commit b7b0f3f27 again.


Now that the unfairness in all-cached parallel seq scan eliminated (with
3ed3683618),  I've re-run the same performance tests and got new results
(please see attached). As we can see, the aforementioned pg_tpcds.query72
got better:
              2024-05-15  2024-07-30 2024-09-03
pg-src-16--1   20492.58    19669.34    19913.32
pg-src-17--1   25286.10    24269.21    20654.95
pg-src-16--2   20769.88    19539.89    20429.72
pg-src-17--2   25771.90    24530.51    21244.92
pg-src-17--3   25978.55    24753.25    20904.09
pg-src-16--3   20943.10    20011.13    20086.61

We can also see the improvement of pg_tpcds.query16, but not on all runs:
              2024-05-15  2024-07-30 2024-09-03
pg-src-16--1     105.36       94.31       97.74
pg-src-17--1     145.74      145.53      145.51
pg-src-16--2     101.82       98.36       96.63
pg-src-17--2     140.07      146.90       96.93
pg-src-17--3     154.89      148.11      106.18
pg-src-16--3     101.03      100.94       93.44

So it looks like now we see the same instability, that we observed before
([1]).

Unfortunately, the troublesome tpcds.query15 hasn't produced good numbers
this time too:
              2024-05-15  2024-07-30 2024-09-03
pg-src-16--1     153.41      142.52      142.54
pg-src-17--1     229.84      225.11      212.51
pg-src-16--2     153.47      150.13      149.37
pg-src-17--2     236.34      227.15      232.73
pg-src-17--3     236.43      239.46      233.77
pg-src-16--3     151.03      152.23      144.90

From a bird's eye view, new v17-vs-v16 comparison has only 87 "worse",
while the previous one had 115 (it requires deeper analysis, of course, but
still...).

[1] 
https://www.postgresql.org/message-id/d1fb5c09-dd03-2540-9ec2-86dbfdfa2c65%40gmail.com

Best regards,
Alexander

Title: Postgres benchmarking results

Benchmarking run started at 2024-09-02T09:38:29, ended at 2024-09-03T03:22:21

Benchmark version
Instance	version
pg-src-16--1	PostgreSQL 16.4-REL_16_STABLE/2015dd5c90 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
pg-src-17--1	PostgreSQL 17beta3-REL_17_STABLE/3ed3683618 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
pg-src-16--2	PostgreSQL 16.4-REL_16_STABLE/2015dd5c90 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
pg-src-17--2	PostgreSQL 17beta3-REL_17_STABLE/3ed3683618 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
pg-src-17--3	PostgreSQL 17beta3-REL_17_STABLE/3ed3683618 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit
pg-src-16--3	PostgreSQL 16.4-REL_16_STABLE/2015dd5c90 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, 64-bit

Benchmark pgbench_native
Instance	tps
pg-src-16--1	10555.04
pg-src-17--1	10713.05
pg-src-16--2	10539.33
pg-src-17--2	10689.77
pg-src-17--3	10713.91
pg-src-16--3	10562.90

Benchmark pgbench_reference
Instance	latency_avg	tps1	tps2
pg-src-16--1	8.57	7471.28	7471.31
pg-src-17--1	8.31	7700.45	7700.49
pg-src-16--2	8.58	7463.04	7463.07
pg-src-17--2	8.31	7701.52	7701.55
pg-src-17--3	8.36	7655.43	7655.46
pg-src-16--3	8.57	7464.59	7464.63

Benchmark pg_tpch
Instance	query1	query2	query3	query4	query5	query6	query7	query8	query9	query10	query11	query12	query13	query14	query15	query16	query17	query18	query19	query20	query21	query22
pg-src-16--1	9.63	1.45	2.97	0.70	0.90	1.44	2.89	4.63	0.84	2.45	0.77	2.18	2.86	0.85	3.16	0.98	8.64	14.30	0.19	5.79	2.36	0.23
pg-src-17--1	9.30	1.48	2.82	0.69	0.88	1.35	2.79	4.55	0.95	2.33	0.78	2.12	2.76	0.86	2.99	0.99	8.20	14.54	0.19	5.46	2.24	0.23
pg-src-16--2	9.73	1.43	3.17	0.70	0.90	1.45	2.88	4.44	0.94	2.41	0.76	2.17	2.84	0.85	3.16	0.98	8.75	14.32	0.19	5.79	2.33	0.23
pg-src-17--2	9.15	1.46	3.06	0.69	0.88	1.35	2.78	4.48	0.93	2.36	0.77	2.10	2.82	0.86	3.00	0.96	8.57	13.83	0.19	5.48	2.25	0.22
pg-src-17--3	9.06	1.48	2.82	0.69	0.88	1.35	2.79	4.61	0.94	2.32	0.78	2.08	2.80	0.85	2.99	1.00	8.23	14.41	0.19	5.56	2.27	0.22
pg-src-16--3	9.63	1.46	3.00	0.68	0.88	1.44	2.87	4.49	0.93	2.41	0.78	2.16	2.78	0.87	3.16	0.98	8.61	14.40	0.19	5.77	2.33	0.23

Benchmark pg_tpcds
Instance	load_size	indexes_size	vacuum_full_size	vacuum_freeze_size	load_time	indexes_time	vacuum_full_time	vacuum_freeze_time	analyze_time	query92	query91	query13	query9	query21	query90	query93	query16	query68	query48	query76	query7	query37	query43	query72	query82	query79	query55	query66	query71	query69	query34	query45	query10	query28	query3	query15	query50	query85	query84	query42	query73	query94	query46	query88	query19	query40	query26	query96	query6	query52
pg-src-16--1	5069452.00	7250172.00	6939848.00	6942520.00	77.89	72.52	94.32	2.17	3.47	80.15	45.44	639.11	1440.39	156.46	148.06	4.89	97.74	267.27	776.70	385.43	1191.35	8.05	711.56	19913.32	10.04	345.33	276.69	293.88	237.71	110.77	245.37	129.32	31.24	673.80	28.70	142.54	76.02	113.66	15.49	280.83	200.49	74.22	491.49	1671.62	305.71	71.62	578.10	203.47	22641.96	279.38
pg-src-17--1	5069696.00	7250136.00	6939804.00	6942476.00	77.29	70.62	91.97	2.07	3.38	76.00	42.77	602.86	1379.45	192.72	137.82	5.30	145.51	248.11	747.61	375.89	1196.38	7.40	687.38	20654.95	8.95	321.39	273.67	302.34	222.26	115.06	228.31	133.03	32.03	648.93	30.22	212.51	77.60	115.46	14.82	275.94	186.77	71.32	483.36	1516.78	300.41	77.48	580.21	183.54	17055.95	272.62
pg-src-16--2	5069432.00	7250160.00	6939852.00	6942524.00	77.86	72.61	95.60	2.17	3.43	78.74	44.00	638.01	1445.93	159.91	146.34	4.54	96.63	268.90	775.20	382.70	1190.93	7.96	706.85	20429.72	9.60	342.47	280.45	296.88	235.95	112.20	243.51	130.54	31.02	670.15	28.73	149.37	55.97	113.53	14.95	281.02	200.28	74.44	729.02	1661.96	311.59	73.31	575.33	201.95	22805.14	281.19
pg-src-17--2	5069576.00	7250012.00	6939812.00	6942484.00	78.25	70.72	91.84	2.06	3.37	73.18	43.18	607.48	1351.40	199.49	136.08	4.95	96.93	251.86	745.81	381.36	1207.63	7.37	688.86	21244.92	8.68	322.98	268.75	290.70	223.16	112.85	227.21	129.29	31.40	651.05	29.03	232.73	80.41	113.95	14.50	268.75	181.10	69.33	483.30	1512.00	299.27	73.39	578.25	184.24	16965.94	268.77
pg-src-17--3	5069464.00	7249888.00	6939840.00	6942480.00	77.62	73.23	91.89	3.03	3.40	74.77	43.56	601.38	1355.91	192.86	134.39	5.02	106.18	250.13	739.19	370.71	1204.34	7.39	690.72	20904.09	8.75	321.06	275.54	300.61	221.62	115.71	226.54	1317.50	31.17	645.85	30.15	233.77	56.32	114.61	14.45	274.20	183.04	69.90	479.30	1505.33	310.84	98.38	578.89	184.36	16931.08	272.48
pg-src-16--3	5069444.00	7250236.00	6939824.00	6942520.00	79.83	71.54	95.13	2.21	3.45	80.95	46.05	644.11	1444.63	159.84	148.25	4.82	93.44	271.00	778.54	386.46	1198.84	8.09	712.96	20086.61	10.02	344.74	282.13	310.14	241.05	112.59	244.63	132.10	31.46	676.07	31.18	144.90	78.14	115.94	16.40	291.00	207.17	76.39	496.92	1660.93	310.52	72.97	569.19	202.79	22667.43	284.54

Benchmark ycsb
Instance	load_a_size	load_b_size	load_c_size	load_d_size	load_f_size	load_a_time	load_b_time	load_c_time	load_d_time	load_f_time	load_a_10_runtime	load_a_25_runtime	load_a_50_runtime	load_b_10_runtime	load_b_25_runtime	load_b_50_runtime	load_c_10_runtime	load_c_25_runtime	load_c_50_runtime	load_d_10_runtime	load_d_25_runtime	load_d_50_runtime	load_f_10_runtime	load_f_25_runtime	load_f_50_runtime	load_a_10_ops	load_a_25_ops	load_a_50_ops	load_b_10_ops	load_b_25_ops	load_b_50_ops	load_c_10_ops	load_c_25_ops	load_c_50_ops	load_d_10_ops	load_d_25_ops	load_d_50_ops	load_f_10_ops	load_f_25_ops	load_f_50_ops
pg-src-16--1	2259520.00	2277852.00	2277512.00	2277160.00	2277956.00	41.17	42.57	41.88	41.44	41.59	9474.00	5122.00	5482.00	4890.00	3361.00	3372.00	4505.00	2969.00	3032.00	4483.00	3068.00	3141.00	9105.00	6606.00	6648.00	105552.04	195236.24	182415.18	204498.98	297530.50	296559.91	221975.58	336813.74	329815.30	223064.91	325945.24	318369.95	109829.76	151377.54	150421.18
pg-src-17--1	2259492.00	2277576.00	2277476.00	2277148.00	2279224.00	41.76	40.75	40.89	40.52	41.39	9859.00	5318.00	6029.00	5410.00	3295.00	3324.00	4608.00	3042.00	3017.00	4478.00	3064.00	2957.00	9167.00	6769.00	7037.00	101430.17	188040.62	165864.99	184842.88	303490.14	300842.36	217013.89	328731.10	331455.09	223313.98	326370.76	338180.59	109086.94	147732.31	142106.01
pg-src-16--2	2259520.00	2277604.00	2277512.00	2277168.00	2277972.00	41.29	40.81	42.44	43.61	43.36	9960.00	5266.00	5478.00	5509.00	3467.00	3430.00	4502.00	3012.00	3061.00	4490.00	3072.00	3117.00	9293.00	7031.00	6684.00	100401.61	189897.46	182548.38	181521.15	288433.80	291545.19	222123.50	332005.31	326690.62	222717.15	325520.83	320821.30	107607.88	142227.28	149611.01
pg-src-17--2	2259488.00	2277584.00	2277228.00	2277148.00	2278592.00	40.28	41.68	40.54	40.00	40.45	9453.00	5489.00	5785.00	5169.00	3322.00	3274.00	4455.00	2938.00	2996.00	4447.00	3153.00	3060.00	10325.00	7248.00	7066.00	105786.52	182182.55	172860.85	193461.02	301023.48	305436.77	224466.89	340367.60	333778.37	224870.70	317158.26	326797.39	96852.30	137969.09	141522.79
pg-src-17--3	2259488.00	2277840.00	2277484.00	2277156.00	2279112.00	40.82	41.65	40.92	40.47	40.25	10754.00	5422.00	5861.00	4864.00	3300.00	3191.00	4620.00	2978.00	2996.00	4497.00	3133.00	2988.00	9171.00	8964.00	6921.00	92988.66	184433.79	170619.35	205592.11	303030.30	313381.39	216450.22	335795.84	333778.37	222370.47	319182.89	334672.02	109039.36	111557.34	144487.79
pg-src-16--3	2259520.00	2277604.00	2277248.00	2277168.00	2279132.00	40.31	41.67	41.38	40.69	40.56	9679.00	5088.00	5373.00	5689.00	3326.00	3490.00	4678.00	3102.00	3000.00	4439.00	3101.00	3049.00	10008.00	6589.00	6642.00	103316.46	196540.88	186115.76	175777.82	300661.46	286532.95	213766.57	322372.66	333333.33	225275.96	322476.62	327976.39	99920.06	151768.10	150557.06

Benchmark benchmarksql_tpcc
Instance	tpm_total	xacts	data_size
pg-src-16--1	211328.00	634002.00	2806768.00
pg-src-17--1	213625.00	640914.00	2762864.00
pg-src-16--2	199387.00	598174.00	2749036.00
pg-src-17--2	213212.00	639665.00	2766324.00
pg-src-17--3	214646.00	643956.00	2770352.00
pg-src-16--3	205839.00	617536.00	2773592.00

Benchmark benchbase
Instance	auctionmark_rps	epinions_rps	hyadapt_rps	otmetrics_rps	resourcestresser_rps	seats_rps	sibench_rps	smallbank_rps	tatp_rps	tpcc_rps	tpch_rps	twitter_rps	voter_rps	ycsb_rps
pg-src-16--1	154.38	78243.50	14336.08	7274.82	0.92	10744.29	51430.20	36134.25	97194.15	6939.04	0.29	74403.02	45462.27	124689.01
pg-src-17--1	162.06	87788.38	13956.05	7454.92	0.95	10660.26	73194.55	34851.55	98576.71	7019.19	0.29	75914.13	45944.85	126591.48
pg-src-16--2	156.93	78554.15	14178.05	7024.97	1.21	10851.55	52952.04	36077.11	95605.61	7067.59	0.29	75978.10	45845.00	123972.37
pg-src-17--2	165.00	88362.54	13864.50	7963.87	0.83	10858.99	69443.05	34107.73	96731.71	6984.12	0.29	77016.32	46236.50	124870.76
pg-src-17--3	151.35	87351.93	14294.03	7747.94	1.02	10525.72	64885.66	34135.57	100915.55	6508.56	0.29	75961.52	45795.22	124918.29
pg-src-16--3	162.39	79741.88	14220.14	6982.66	1.00	10446.83	52647.37	34715.17	97599.14	6933.27	0.29	73328.21	43990.38	122215.06

Benchmark gdprbench
Instance	load_controller_ops	run_controller_ops	load_customer_ops	run_customer_ops	load_processor_ops	run_processor_ops
pg-src-16--1	19723.87	1733.10	16666.67	2016.13	17953.32	1295.34
pg-src-17--1	16920.47	1779.36	17391.30	2016.13	17636.68	1204.82
pg-src-16--2	16806.72	1821.49	19801.98	1883.24	18621.97	1293.66
pg-src-17--2	16863.41	1814.88	17513.13	1980.20	17123.29	1331.56
pg-src-17--3	17241.38	1805.05	16750.42	1988.07	17730.50	1344.09
pg-src-16--3	17211.70	1766.78	17421.60	1923.08	16863.41	1314.06

Benchmark s64da_tpch
Instance	total	query1	query2	query3	query4	query5	query6	query7	query8	query9	query10	query11	query12	query13	query14	query15	query16	query17	query18	query19	query20	query21	query22
pg-src-16--1	67.42	9.26	1.37	2.93	0.89	1.62	1.35	1.91	1.09	3.98	2.25	0.73	2.49	2.89	0.86	1.54	1.02	4.31	20.67	0.18	1.05	4.81	0.21
pg-src-17--1	65.89	8.71	1.42	2.74	0.88	1.52	1.27	1.78	1.04	4.59	2.10	0.73	2.27	2.78	0.85	1.40	1.01	4.09	20.66	0.18	1.07	4.58	0.20
pg-src-16--2	67.12	9.40	1.37	2.91	0.88	1.59	1.35	1.89	1.10	3.85	2.26	0.72	2.49	2.85	0.85	1.51	1.01	4.29	20.61	0.18	1.05	4.75	0.21
pg-src-17--2	65.72	8.93	1.40	2.73	0.89	1.46	1.24	1.76	1.02	4.55	2.09	0.71	2.26	2.80	0.85	1.40	1.01	4.12	20.49	0.18	1.07	4.56	0.20
pg-src-17--3	65.09	8.87	1.37	2.73	0.89	1.57	1.24	1.77	1.08	4.60	2.13	0.71	2.26	2.78	0.86	1.41	1.02	4.18	19.47	0.18	1.05	4.71	0.20
pg-src-16--3	67.50	9.27	1.38	2.93	0.89	1.66	1.35	1.90	1.13	3.99	2.27	0.73	2.48	2.86	0.86	1.53	1.01	4.37	20.54	0.19	1.06	4.89	0.21

Benchmark s64da_tpcds
Instance	total	query1	query2	query3	query5	query6	query7	query8	query9	query12	query13	query14a	query14b	query15	query16	query17	query18	query19	query20	query21	query22	query23a	query23b	query24a	query24b	query25	query26	query27	query28	query29	query30	query31	query32	query33	query34	query35	query36	query37	query38	query39a	query39b	query40	query41	query42	query43	query44	query45	query46	query47	query48	query49	query50	query51	query52	query53	query54	query55	query56	query57	query58	query59	query60	query61	query62	query63	query64	query65	query66	query67	query68	query69	query70	query71	query72	query73	query75	query76	query78	query79	query80	query81	query82	query83	query84	query85	query86	query87	query88	query89	query90	query91	query92	query93	query94	query95	query96	query97	query98	query99
pg-src-16--1	2714.29	0.00	0.98	0.02	0.58	205.22	1.16	0.06	2.55	0.05	0.64	30.63	67.84	0.14	0.09	0.92	0.69	0.31	0.07	0.12	4.67	7.01	6.89	0.01	0.01	0.64	0.57	0.99	1.94	0.41	55.75	17.29	0.05	0.21	0.25	2.32	0.83	0.01	2.17	4.31	4.21	0.08	1.16	0.30	0.01	0.00	1.27	0.50	2.13	0.77	0.16	0.08	2.66	0.29	0.50	0.11	0.29	1.05	1.07	0.12	1.58	2.27	0.01	0.22	0.49	0.58	1.48	0.30	2.10	0.23	0.14	1.13	0.25	20.52	0.20	2.06	0.37	2.77	0.39	0.53	259.11	0.01	0.10	0.03	0.17	0.27	2.20	1.67	0.35	0.15	0.05	0.07	0.10	0.09	175.36	0.22	1.06	0.16	0.38
pg-src-17--1	3800.07	934.27	0.94	0.02	0.57	195.18	1.13	0.05	2.38	0.05	0.61	31.39	67.01	0.14	0.09	0.92	0.75	0.30	0.07	0.12	4.52	6.89	6.86	0.01	0.01	0.65	0.73	0.99	1.82	0.42	54.28	16.64	0.05	0.22	0.23	2.30	0.80	0.01	1.92	3.88	3.94	0.11	1.10	0.28	0.01	0.00	1.28	0.48	2.30	0.74	0.16	0.08	2.41	0.28	0.49	0.11	0.28	1.10	1.18	0.12	1.35	2.34	0.01	0.21	0.50	0.56	1.35	0.29	1.87	0.21	0.14	1.07	0.23	21.58	0.19	1.86	0.38	2.77	0.37	0.54	241.63	0.01	0.10	0.03	0.17	0.27	1.96	1.53	0.33	0.14	0.05	0.06	0.09	0.08	357.94	0.20	1.04	0.24	0.67
pg-src-16--2	2734.45	0.00	0.98	0.01	0.58	204.12	1.16	0.06	2.56	0.05	0.65	30.68	68.44	0.14	0.09	0.92	0.71	0.23	0.07	0.12	4.74	7.02	6.97	0.01	0.01	0.63	0.58	1.00	1.95	0.40	56.14	17.09	0.05	0.21	0.25	2.33	0.82	0.01	2.18	4.21	4.18	0.08	1.14	0.31	0.01	0.00	0.14	0.50	2.15	0.77	0.16	0.08	2.61	0.30	0.49	0.11	0.21	1.04	1.11	0.12	1.60	2.34	0.01	0.22	0.49	0.59	1.47	0.30	1.92	0.23	0.14	1.13	0.15	21.36	0.21	2.06	0.38	2.89	0.39	0.54	252.18	0.01	0.10	0.03	0.17	0.27	2.20	1.69	0.36	0.15	0.05	0.08	0.09	0.09	202.93	0.22	1.07	0.25	0.38
pg-src-17--2	3685.00	977.37	0.93	0.01	0.56	197.09	1.14	0.06	2.38	0.05	0.61	30.98	66.77	0.15	0.09	0.90	0.75	0.36	0.07	0.12	4.50	6.90	6.85	0.01	0.01	0.64	0.71	0.99	1.82	0.41	56.37	16.66	0.05	0.21	0.24	2.31	0.81	0.01	1.92	3.92	3.94	0.11	1.11	0.32	0.01	0.00	1.30	0.48	2.34	0.73	0.16	0.06	2.43	0.33	0.49	0.11	0.22	1.06	1.24	0.12	1.35	2.27	0.01	0.21	0.49	0.55	1.35	0.28	1.86	0.21	0.14	1.07	0.15	20.62	0.19	1.84	0.37	2.75	0.37	0.53	242.69	0.01	0.10	0.03	0.17	0.27	1.95	1.55	0.34	0.14	0.05	0.06	0.09	0.08	196.51	0.21	1.04	0.23	0.55
pg-src-17--3	3805.86	934.80	0.94	0.01	0.57	195.71	1.13	0.06	2.39	0.05	0.61	31.28	66.89	0.14	0.12	0.91	0.75	0.24	0.07	0.12	4.50	6.86	6.82	0.01	0.01	0.65	0.72	0.98	1.82	0.42	54.24	16.67	0.05	0.21	0.23	2.34	0.81	0.01	2.15	3.95	4.21	0.12	1.10	0.31	0.01	0.00	1.31	0.48	2.32	0.74	0.16	0.06	2.46	0.32	0.50	0.11	0.22	1.11	1.18	0.12	1.34	2.35	0.01	0.21	0.51	0.56	1.38	0.30	1.85	0.21	0.14	1.07	0.15	21.63	0.19	1.83	0.38	2.79	0.37	0.53	241.65	0.01	0.10	0.03	0.17	0.27	2.19	1.55	0.34	0.14	0.05	0.06	0.09	0.08	362.16	0.21	1.04	0.24	0.60
pg-src-16--3	2702.45	0.00	0.97	0.01	0.58	205.35	1.14	0.07	2.55	0.05	0.64	30.71	68.00	0.14	0.09	0.91	0.69	0.23	0.07	0.12	4.67	7.06	6.92	0.01	0.01	0.64	0.57	0.99	1.94	0.41	55.35	17.18	0.05	0.20	0.25	2.30	0.84	0.01	2.16	4.20	4.17	0.08	1.14	0.31	0.01	0.00	0.14	0.50	2.14	0.77	0.15	0.08	2.60	0.30	0.50	0.11	0.21	1.04	1.08	0.12	1.58	2.22	0.01	0.22	0.49	0.59	1.47	0.30	1.90	0.23	0.14	1.13	0.15	20.67	0.21	2.07	0.38	2.85	0.39	0.53	248.87	0.01	0.10	0.03	0.17	0.27	2.17	1.68	0.35	0.15	0.05	0.07	0.09	0.08	175.52	0.21	1.06	0.16	0.37

Benchmark s64da_ssb
Instance	total	queryQ1.1	queryQ1.2	queryQ1.3	queryQ2.1	queryQ2.2	queryQ2.3	queryQ3.1	queryQ3.2	queryQ3.3	queryQ3.4	queryQ4.1	queryQ4.2	queryQ4.3
pg-src-16--1	33.09	2.53	2.44	2.45	2.79	2.38	2.12	3.49	2.12	1.87	1.87	3.24	3.07	2.71
pg-src-17--1	30.60	2.38	2.29	2.27	2.62	2.23	1.96	2.92	1.93	1.75	1.74	3.07	2.90	2.55
pg-src-16--2	32.57	2.54	2.43	2.41	2.49	2.37	2.11	3.49	2.32	1.86	1.87	3.27	3.08	2.33
pg-src-17--2	31.03	2.36	2.32	2.25	2.61	2.26	1.97	3.31	1.91	1.75	1.75	3.08	2.90	2.57
pg-src-17--3	30.93	2.37	2.29	2.25	2.61	2.23	1.95	3.31	1.93	1.74	1.74	3.04	2.91	2.55
pg-src-16--3	33.12	2.53	2.44	2.42	2.51	2.39	2.15	3.49	2.30	1.87	1.87	3.31	3.12	2.73

Benchmark job
Instance	query10a	query10b	query10c	query11a	query11b	query11c	query11d	query12a	query12b	query12c	query13a	query13b	query13c	query13d	query14a	query14b	query14c	query15a	query15b	query15c	query15d	query16a	query16b	query16c	query16d	query17a	query17b	query17c	query17d	query17e	query17f	query18a	query18b	query18c	query19a	query19b	query19c	query19d	query1a	query1b	query1c	query1d	query20a	query20b	query20c	query21a	query21b	query21c	query22a	query22b	query22c	query22d	query23a	query23b	query23c	query24a	query24b	query25a	query25b	query25c	query26a	query26b	query26c	query27a	query27b	query27c	query28a	query28b	query28c	query29a	query29b	query29c	query2a	query2b	query2c	query2d	query30a	query30b	query30c	query31a	query31b	query31c	query32a	query32b	query33a	query33b	query33c	query3a	query3b	query3c	query4a	query4b	query4c	query5a	query5b	query5c	query6a	query6b	query6c	query6d	query6e	query6f	query7a	query7b	query7c	query8a	query8b	query8c	query8d	query9a	query9b	query9c	query9d
pg-src-16--1	0.55	0.38	3.27	0.26	0.09	1.12	4.54	0.38	0.06	1.06	1.29	0.65	0.63	1.96	0.65	0.17	0.95	0.32	0.04	0.85	0.95	0.21	16.67	1.59	1.26	9.34	7.67	7.45	7.61	11.07	10.02	4.88	1.33	6.28	0.30	0.19	6.63	7.13	0.07	0.06	0.06	0.06	2.59	1.14	0.64	0.34	0.25	0.48	0.59	0.44	2.16	1.35	0.59	0.07	0.36	1.75	0.22	9.08	0.48	12.20	0.67	0.17	1.86	0.34	0.21	0.73	0.93	0.74	2.32	0.22	0.12	0.22	0.50	0.46	0.44	0.59	2.72	0.68	6.45	3.12	0.68	3.15	0.00	1.22	0.86	0.98	0.90	0.17	0.12	0.26	0.10	0.12	0.10	0.17	0.16	0.22	0.07	1.22	0.03	6.44	0.07	6.47	0.22	0.20	26.40	2.26	0.19	4.77	3.13	0.28	0.27	2.42	3.95
pg-src-17--1	0.55	0.38	3.15	0.28	0.09	1.13	4.65	0.40	0.06	1.08	1.34	0.69	0.64	2.03	0.70	0.17	0.97	0.33	0.04	0.72	0.91	0.25	17.41	1.66	1.36	9.98	7.88	7.57	7.66	11.56	10.15	0.36	1.39	6.43	3.90	0.18	6.61	7.08	0.06	0.05	0.06	0.05	2.63	1.17	0.73	0.35	0.26	0.51	0.61	0.45	2.15	1.39	0.56	0.07	0.35	1.75	0.08	4.00	0.49	11.73	0.82	0.42	2.24	0.34	0.21	0.74	0.97	0.77	2.37	0.19	0.07	3.22	0.51	0.48	0.47	0.61	1.67	0.69	5.45	3.22	0.66	3.24	0.00	1.24	0.77	0.81	2.53	0.19	0.12	0.27	0.10	0.12	0.11	0.16	0.15	0.21	0.07	1.40	0.04	7.41	0.07	7.52	0.24	0.19	15.16	2.36	0.19	4.60	2.94	2.46	1.47	2.27	4.27
pg-src-16--2	0.55	0.39	3.31	0.28	0.09	1.13	4.59	0.40	0.06	1.09	1.31	0.66	0.64	2.01	0.66	0.17	0.95	0.33	0.04	0.86	0.97	0.23	17.10	1.63	1.29	9.65	7.78	7.56	7.61	11.46	10.04	0.39	1.34	6.35	0.30	0.20	6.64	7.17	0.07	0.06	0.06	0.06	2.61	1.15	0.68	0.33	0.25	0.48	0.59	0.44	2.16	1.39	0.56	0.07	0.82	1.80	0.06	3.99	0.48	12.36	0.36	0.16	2.05	0.53	0.15	0.74	0.94	1.42	0.46	0.08	0.07	0.85	0.52	0.47	0.45	0.60	5.35	0.67	3.47	3.19	0.65	3.22	0.00	1.21	0.84	5.36	2.54	0.32	0.12	0.26	0.10	0.12	0.10	0.17	0.16	0.22	0.06	1.27	0.03	6.63	0.07	6.79	0.26	0.20	17.83	2.28	0.20	4.79	3.13	0.29	0.27	2.43	4.45
pg-src-17--2	0.56	0.38	3.17	0.29	0.09	1.40	4.69	0.40	0.06	1.06	1.34	0.69	0.65	2.02	0.68	0.17	0.95	0.33	0.04	0.72	0.91	0.24	17.37	1.63	1.28	9.49	7.90	7.51	7.55	11.24	10.01	0.51	1.34	6.26	3.81	0.18	6.39	6.99	0.06	0.05	0.06	0.05	2.61	1.15	0.67	0.33	0.25	0.48	0.59	0.44	2.14	1.37	0.55	0.07	0.79	0.85	0.06	3.94	0.48	11.51	0.80	0.16	2.16	0.34	0.20	0.74	0.83	0.60	2.37	0.08	0.10	0.21	0.51	0.47	0.45	0.61	5.34	0.66	3.50	3.19	0.67	3.25	0.00	1.21	0.80	2.03	2.51	0.18	0.12	0.26	0.10	0.12	0.10	0.16	0.15	0.21	0.06	1.36	0.04	7.22	0.07	7.36	0.24	0.18	26.52	2.29	0.19	4.71	2.97	2.44	1.44	2.28	4.26
pg-src-17--3	0.58	0.38	3.17	0.28	0.09	1.44	5.13	0.40	0.06	1.11	1.33	0.68	0.65	2.04	0.72	0.18	1.01	0.33	0.04	0.72	0.91	0.24	17.81	1.68	1.31	9.58	7.75	7.49	7.62	11.44	10.06	4.81	1.36	6.31	3.82	0.18	6.47	6.99	0.06	0.05	0.06	0.05	2.59	1.17	0.70	0.34	0.25	0.49	0.59	0.43	2.16	1.36	0.55	0.07	0.34	1.75	0.08	3.95	0.49	11.48	0.80	0.21	2.16	0.34	0.23	0.74	5.85	0.69	2.38	0.08	0.07	0.21	0.50	0.47	0.44	0.60	10.82	0.70	5.19	3.22	0.65	3.22	0.00	1.25	0.84	0.81	2.51	0.17	0.12	0.26	0.10	0.12	0.10	0.16	0.15	0.21	0.06	1.35	0.04	7.15	0.07	7.20	0.25	0.18	20.04	2.29	0.19	4.65	2.99	2.44	1.44	2.26	4.32
pg-src-16--3	0.60	0.39	3.28	0.27	0.11	1.13	4.57	0.41	0.06	1.09	1.31	0.69	0.64	2.00	0.67	0.17	1.04	0.33	0.07	0.86	0.96	0.23	17.27	1.62	1.29	9.39	7.60	7.32	7.44	11.24	9.87	0.38	1.39	6.54	0.30	0.20	6.77	7.12	0.06	0.06	0.06	0.06	2.59	1.15	0.68	0.33	0.24	0.47	0.64	0.47	2.24	1.44	0.57	0.07	0.82	0.84	0.06	3.98	0.48	11.71	0.37	0.21	0.75	0.34	0.15	0.51	0.94	0.67	2.40	17.64	1.05	0.51	0.50	0.46	0.45	0.59	11.43	0.65	3.63	3.12	0.64	3.18	0.00	1.22	1.02	1.71	2.50	0.17	0.12	0.26	0.10	0.12	0.10	0.17	0.16	0.22	0.07	1.22	0.03	6.59	0.07	6.64	0.27	0.20	22.53	2.37	0.20	4.86	3.16	0.28	0.27	2.45	4.53

Benchmark sysbench
Instance	load_bulk_insert_tps	load_oltp_delete_tps	load_oltp_insert_tps	load_oltp_point_select_tps	load_oltp_read_only_tps	load_oltp_read_write_tps	load_oltp_update_index_tps	load_oltp_update_non_index_tps	load_oltp_write_only_tps	load_select_random_points_tps	load_select_random_ranges_tps	load_bulk_insert_ops	load_oltp_delete_ops	load_oltp_insert_ops	load_oltp_point_select_ops	load_oltp_read_only_ops	load_oltp_read_write_ops	load_oltp_update_index_ops	load_oltp_update_non_index_ops	load_oltp_write_only_ops	load_select_random_points_ops	load_select_random_ranges_ops
pg-src-16--1	2121442.62	367043.92	127736.19	356984.19	15044.88	9301.14	154214.33	181749.42	29968.05	221029.48	147619.65	74.02	367043.92	127736.19	356984.19	240718.14	186106.27	154214.33	181749.42	180117.54	221029.48	147619.65
pg-src-17--1	2223417.57	364937.72	130976.71	355356.29	14958.78	8947.69	151461.57	182404.03	25055.27	260191.45	148597.15	77.92	364937.72	130976.71	355356.29	239340.54	179039.43	151461.57	182404.03	150592.32	260191.45	148597.15
pg-src-16--2	2166684.64	364913.57	127671.80	351179.43	14852.25	8934.82	153759.63	180060.04	26420.27	218552.77	143672.73	75.77	364913.57	127671.80	351179.43	237635.93	178784.64	153759.63	180060.04	158791.37	218552.77	143672.73
pg-src-17--2	2255472.85	371336.90	128456.03	361004.71	15040.88	8997.72	149289.12	182014.73	29327.65	264282.82	147883.95	79.15	371336.90	128456.03	361004.71	240654.14	180036.51	149289.12	182014.73	176267.01	264282.82	147883.95
pg-src-17--3	2279598.26	370515.75	130290.53	356668.45	15010.62	9393.94	152135.93	184423.29	27282.70	264570.35	148655.94	80.09	370515.75	130290.53	356668.45	240169.95	187978.70	152135.93	184423.29	163984.94	264570.35	148655.94
pg-src-16--3	2174533.66	362506.75	126130.59	352806.94	14923.84	8883.89	152955.32	179857.60	26151.47	219180.57	145376.44	76.06	362506.75	126130.59	352806.94	238781.38	177763.58	152955.32	179857.60	157177.05	219180.57	145376.44

Benchmark insert-1m
Instance	time
pg-src-16--1	34.90
pg-src-17--1	35.02
pg-src-16--2	35.15
pg-src-17--2	33.63
pg-src-17--3	34.89
pg-src-16--3	34.28

./analyze-benchmarks.py -i 'pg-src-17--.*'  'pg-src-16--.*'
Best pg-src-17--.* better than pg-src-16--.* by 5.9 percents (9.06 < 9.63): 
pg_tpch.query1
Average pg-src-17--.* better than pg-src-16--.* by 5.1 percents (9.17 < 9.66): 
pg_tpch.query1
Best pg-src-17--.* better than pg-src-16--.* by 5.1 percents (2.82 < 2.97): 
pg_tpch.query3
Best pg-src-17--.* better than pg-src-16--.* by 6.2 percents (1.35 < 1.44): 
pg_tpch.query6
Average pg-src-17--.* better than pg-src-16--.* by 6.5 percents (1.35 < 1.44): 
pg_tpch.query6
Best pg-src-17--.* worse than pg-src-16--.* by 10.7 percents (0.93 > 0.84): 
pg_tpch.query9
Best pg-src-17--.* better than pg-src-16--.* by 5.4 percents (2.99 < 3.16): 
pg_tpch.query15
Average pg-src-17--.* better than pg-src-16--.* by 5.3 percents (2.99 < 3.16): 
pg_tpch.query15
Best pg-src-17--.* better than pg-src-16--.* by 5.4 percents (5.46 < 5.77): 
pg_tpch.query20
Best pg-src-17--.* better than pg-src-16--.* by 5.1 percents (2.06 < 2.17): 
pg_tpcds.vacuum_freeze_time
Average pg-src-17--.* worse than pg-src-16--.* by 9.3 percents (2.39 > 2.18): 
pg_tpcds.vacuum_freeze_time
Best pg-src-17--.* better than pg-src-16--.* by 7.1 percents (73.18 < 78.73): 
pg_tpcds.query92
Average pg-src-17--.* better than pg-src-16--.* by 6.6 percents (74.65 < 
79.94): pg_tpcds.query92
Best pg-src-17--.* better than pg-src-16--.* by 5.7 percents (601.38 < 638.01): 
pg_tpcds.query13
Average pg-src-17--.* better than pg-src-16--.* by 5.7 percents (603.90 < 
640.41): pg_tpcds.query13
Best pg-src-17--.* better than pg-src-16--.* by 6.2 percents (1351.40 < 
1440.39): pg_tpcds.query9
Average pg-src-17--.* better than pg-src-16--.* by 5.6 percents (1362.25 < 
1443.65): pg_tpcds.query9
Best pg-src-17--.* worse than pg-src-16--.* by 23.2 percents (192.72 > 156.46): 
pg_tpcds.query21
Average pg-src-17--.* worse than pg-src-16--.* by 22.9 percents (195.02 > 
158.73): pg_tpcds.query21
Best pg-src-17--.* better than pg-src-16--.* by 8.2 percents (134.39 < 146.34): 
pg_tpcds.query90
Average pg-src-17--.* better than pg-src-16--.* by 7.8 percents (136.10 < 
147.55): pg_tpcds.query90
Best pg-src-17--.* worse than pg-src-16--.* by 9.1 percents (4.95 > 4.54): 
pg_tpcds.query93
Average pg-src-17--.* worse than pg-src-16--.* by 7.1 percents (5.09 > 4.75): 
pg_tpcds.query93
Average pg-src-17--.* worse than pg-src-16--.* by 21.1 percents (116.21 > 
95.94): pg_tpcds.query16
Best pg-src-17--.* better than pg-src-16--.* by 7.2 percents (248.11 < 267.27): 
pg_tpcds.query68
Average pg-src-17--.* better than pg-src-16--.* by 7.1 percents (250.03 < 
269.06): pg_tpcds.query68
Best pg-src-17--.* better than pg-src-16--.* by 7.4 percents (7.37 < 7.96): 
pg_tpcds.query37
Average pg-src-17--.* better than pg-src-16--.* by 8.0 percents (7.39 < 8.03): 
pg_tpcds.query37
Best pg-src-17--.* better than pg-src-16--.* by 9.6 percents (8.68 < 9.60): 
pg_tpcds.query82
Average pg-src-17--.* better than pg-src-16--.* by 11.1 percents (8.79 < 9.88): 
pg_tpcds.query82
Best pg-src-17--.* better than pg-src-16--.* by 6.3 percents (321.06 < 342.47): 
pg_tpcds.query79
Average pg-src-17--.* better than pg-src-16--.* by 6.5 percents (321.81 < 
344.18): pg_tpcds.query79
Best pg-src-17--.* better than pg-src-16--.* by 6.1 percents (221.62 < 235.94): 
pg_tpcds.query71
Average pg-src-17--.* better than pg-src-16--.* by 6.7 percents (222.34 < 
238.23): pg_tpcds.query71
Best pg-src-17--.* better than pg-src-16--.* by 7.0 percents (226.54 < 243.51): 
pg_tpcds.query34
Average pg-src-17--.* better than pg-src-16--.* by 7.0 percents (227.35 < 
244.50): pg_tpcds.query34
Average pg-src-17--.* worse than pg-src-16--.* by 303.0 percents (526.61 > 
130.66): pg_tpcds.query45
Best pg-src-17--.* worse than pg-src-16--.* by 49.1 percents (212.51 > 142.54): 
pg_tpcds.query15
Average pg-src-17--.* worse than pg-src-16--.* by 55.5 percents (226.34 > 
145.60): pg_tpcds.query15
Average pg-src-17--.* better than pg-src-16--.* by 6.6 percents (14.59 < 
15.61): pg_tpcds.query84
Best pg-src-17--.* better than pg-src-16--.* by 9.6 percents (181.10 < 200.28): 
pg_tpcds.query73
Average pg-src-17--.* better than pg-src-16--.* by 9.4 percents (183.64 < 
202.65): pg_tpcds.query73
Best pg-src-17--.* better than pg-src-16--.* by 6.6 percents (69.33 < 74.22): 
pg_tpcds.query94
Average pg-src-17--.* better than pg-src-16--.* by 6.4 percents (70.18 < 
75.02): pg_tpcds.query94
Average pg-src-17--.* better than pg-src-16--.* by 15.8 percents (481.98 < 
572.48): pg_tpcds.query46
Best pg-src-17--.* better than pg-src-16--.* by 9.4 percents (1505.33 < 
1660.93): pg_tpcds.query88
Average pg-src-17--.* better than pg-src-16--.* by 9.2 percents (1511.37 < 
1664.83): pg_tpcds.query88
Average pg-src-17--.* worse than pg-src-16--.* by 14.4 percents (83.08 > 
72.63): pg_tpcds.query40
Best pg-src-17--.* better than pg-src-16--.* by 9.1 percents (183.54 < 201.95): 
pg_tpcds.query96
Average pg-src-17--.* better than pg-src-16--.* by 9.2 percents (184.05 < 
202.74): pg_tpcds.query96
Best pg-src-17--.* better than pg-src-16--.* by 25.2 percents (16931.08 < 
22641.96): pg_tpcds.query6
Average pg-src-17--.* better than pg-src-16--.* by 25.2 percents (16984.32 < 
22704.84): pg_tpcds.query6
Best pg-src-17--.* worse than pg-src-16--.* by 7.7 percents (5785.00 > 
5373.00): ycsb.load_a_50_runtime
Average pg-src-17--.* worse than pg-src-16--.* by 8.2 percents (5891.67 > 
5444.33): ycsb.load_a_50_runtime
Best pg-src-17--.* better than pg-src-16--.* by 5.4 percents (3191.00 < 
3372.00): ycsb.load_b_50_runtime
Average pg-src-17--.* worse than pg-src-16--.* by 13.6 percents (7660.33 > 
6742.00): ycsb.load_f_25_runtime
Average pg-src-17--.* worse than pg-src-16--.* by 5.3 percents (7008.00 > 
6658.00): ycsb.load_f_50_runtime
Best pg-src-17--.* worse than pg-src-16--.* by 7.1 percents (172860.85 < 
186115.76): ycsb.load_a_50_ops
Average pg-src-17--.* worse than pg-src-16--.* by 7.6 percents (169781.73 < 
183693.11): ycsb.load_a_50_ops
Best pg-src-17--.* better than pg-src-16--.* by 5.7 percents (313381.39 > 
296559.91): ycsb.load_b_50_ops
Average pg-src-17--.* better than pg-src-16--.* by 5.1 percents (306553.51 > 
291546.02): ycsb.load_b_50_ops
Average pg-src-17--.* worse than pg-src-16--.* by 10.8 percents (132419.58 < 
148457.64): ycsb.load_f_25_ops
Best pg-src-17--.* better than pg-src-16--.* by 10.8 percents (88362.54 > 
79741.88): benchbase.epinions_rps
Average pg-src-17--.* better than pg-src-16--.* by 11.4 percents (87834.28 > 
78846.51): benchbase.epinions_rps
Best pg-src-17--.* better than pg-src-16--.* by 9.5 percents (7963.87 > 
7274.82): benchbase.otmetrics_rps
Average pg-src-17--.* better than pg-src-16--.* by 8.9 percents (7722.24 > 
7094.15): benchbase.otmetrics_rps
Best pg-src-17--.* worse than pg-src-16--.* by 15.6 percents (1.02 < 1.21): 
benchbase.resourcestresser_rps
Average pg-src-17--.* worse than pg-src-16--.* by 10.3 percents (0.94 < 1.04): 
benchbase.resourcestresser_rps
Best pg-src-17--.* better than pg-src-16--.* by 38.2 percents (73194.55 > 
52952.04): benchbase.sibench_rps
Average pg-src-17--.* better than pg-src-16--.* by 32.2 percents (69174.42 > 
52343.20): benchbase.sibench_rps
Best pg-src-17--.* worse than pg-src-16--.* by 12.6 percents (17241.38 < 
19723.87): gdprbench.load_controller_ops
Average pg-src-17--.* worse than pg-src-16--.* by 5.1 percents (17008.42 < 
17914.10): gdprbench.load_controller_ops
Best pg-src-17--.* worse than pg-src-16--.* by 11.6 percents (17513.13 < 
19801.98): gdprbench.load_customer_ops
Best pg-src-17--.* better than pg-src-16--.* by 5.9 percents (8.71 < 9.26): 
s64da_tpch.query1
Average pg-src-17--.* better than pg-src-16--.* by 5.1 percents (8.84 < 9.31): 
s64da_tpch.query1
Best pg-src-17--.* better than pg-src-16--.* by 6.2 percents (2.73 < 2.91): 
s64da_tpch.query3
Average pg-src-17--.* better than pg-src-16--.* by 6.5 percents (2.73 < 2.92): 
s64da_tpch.query3
Best pg-src-17--.* better than pg-src-16--.* by 8.2 percents (1.46 < 1.59): 
s64da_tpch.query5
Average pg-src-17--.* better than pg-src-16--.* by 6.6 percents (1.52 < 1.62): 
s64da_tpch.query5
Best pg-src-17--.* better than pg-src-16--.* by 8.1 percents (1.24 < 1.35): 
s64da_tpch.query6
Average pg-src-17--.* better than pg-src-16--.* by 7.4 percents (1.25 < 1.35): 
s64da_tpch.query6
Best pg-src-17--.* better than pg-src-16--.* by 6.9 percents (1.76 < 1.89): 
s64da_tpch.query7
Average pg-src-17--.* better than pg-src-16--.* by 6.8 percents (1.77 < 1.90): 
s64da_tpch.query7
Best pg-src-17--.* better than pg-src-16--.* by 6.4 percents (1.02 < 1.09): 
s64da_tpch.query8
Average pg-src-17--.* better than pg-src-16--.* by 5.4 percents (1.05 < 1.11): 
s64da_tpch.query8
Best pg-src-17--.* worse than pg-src-16--.* by 18.2 percents (4.55 > 3.85): 
s64da_tpch.query9
Average pg-src-17--.* worse than pg-src-16--.* by 16.2 percents (4.58 > 3.94): 
s64da_tpch.query9
Best pg-src-17--.* better than pg-src-16--.* by 7.1 percents (2.09 < 2.25): 
s64da_tpch.query10
Average pg-src-17--.* better than pg-src-16--.* by 6.8 percents (2.11 < 2.26): 
s64da_tpch.query10
Best pg-src-17--.* better than pg-src-16--.* by 8.9 percents (2.26 < 2.48): 
s64da_tpch.query12
Average pg-src-17--.* better than pg-src-16--.* by 9.0 percents (2.26 < 2.49): 
s64da_tpch.query12
Best pg-src-17--.* better than pg-src-16--.* by 7.3 percents (1.40 < 1.51): 
s64da_tpch.query15
Average pg-src-17--.* better than pg-src-16--.* by 8.1 percents (1.40 < 1.53): 
s64da_tpch.query15
Best pg-src-17--.* better than pg-src-16--.* by 5.2 percents (19.47 < 20.54): 
s64da_tpch.query18
Best pg-src-17--.* worse than pg-src-16--.* by 36.4 percents (3685.00 > 
2702.45): s64da_tpcds.total
Average pg-src-17--.* worse than pg-src-16--.* by 38.5 percents (3763.64 > 
2717.06): s64da_tpcds.total
Best pg-src-17--.* better than pg-src-16--.* by 16.7 percents (0.05 < 0.06): 
s64da_tpcds.query8
Average pg-src-17--.* better than pg-src-16--.* by 10.5 percents (0.06 < 0.06): 
s64da_tpcds.query8
Best pg-src-17--.* better than pg-src-16--.* by 6.7 percents (2.38 < 2.55): 
s64da_tpcds.query9
Average pg-src-17--.* better than pg-src-16--.* by 6.7 percents (2.38 < 2.55): 
s64da_tpcds.query9
Average pg-src-17--.* better than pg-src-16--.* by 5.2 percents (0.61 < 0.64): 
s64da_tpcds.query13
Average pg-src-17--.* worse than pg-src-16--.* by 11.1 percents (0.10 > 0.09): 
s64da_tpcds.query16
Best pg-src-17--.* worse than pg-src-16--.* by 8.7 percents (0.75 > 0.69): 
s64da_tpcds.query18
Average pg-src-17--.* worse than pg-src-16--.* by 7.7 percents (0.75 > 0.70): 
s64da_tpcds.query18
Average pg-src-17--.* worse than pg-src-16--.* by 16.9 percents (0.30 > 0.26): 
s64da_tpcds.query19
Best pg-src-17--.* worse than pg-src-16--.* by 24.6 percents (0.71 > 0.57): 
s64da_tpcds.query26
Average pg-src-17--.* worse than pg-src-16--.* by 25.6 percents (0.72 > 0.57): 
s64da_tpcds.query26
Best pg-src-17--.* better than pg-src-16--.* by 6.2 percents (1.82 < 1.94): 
s64da_tpcds.query28
Average pg-src-17--.* better than pg-src-16--.* by 6.3 percents (1.82 < 1.94): 
s64da_tpcds.query28
Best pg-src-17--.* better than pg-src-16--.* by 8.0 percents (0.23 < 0.25): 
s64da_tpcds.query34
Average pg-src-17--.* better than pg-src-16--.* by 6.7 percents (0.23 < 0.25): 
s64da_tpcds.query34
Best pg-src-17--.* better than pg-src-16--.* by 11.1 percents (1.92 < 2.16): 
s64da_tpcds.query38
Average pg-src-17--.* better than pg-src-16--.* by 8.0 percents (2.00 < 2.17): 
s64da_tpcds.query38
Best pg-src-17--.* better than pg-src-16--.* by 7.6 percents (3.88 < 4.20): 
s64da_tpcds.query39a
Average pg-src-17--.* better than pg-src-16--.* by 7.6 percents (3.92 < 4.24): 
s64da_tpcds.query39a
Best pg-src-17--.* better than pg-src-16--.* by 5.5 percents (3.94 < 4.17): 
s64da_tpcds.query39b
Best pg-src-17--.* worse than pg-src-16--.* by 37.5 percents (0.11 > 0.08): 
s64da_tpcds.query40
Average pg-src-17--.* worse than pg-src-16--.* by 41.7 percents (0.11 > 0.08): 
s64da_tpcds.query40
Best pg-src-17--.* better than pg-src-16--.* by 6.7 percents (0.28 < 0.30): 
s64da_tpcds.query42
Best pg-src-17--.* worse than pg-src-16--.* by 814.3 percents (1.28 > 0.14): 
s64da_tpcds.query45
Average pg-src-17--.* worse than pg-src-16--.* by 151.0 percents (1.30 > 0.52): 
s64da_tpcds.query45
Best pg-src-17--.* worse than pg-src-16--.* by 8.0 percents (2.30 > 2.13): 
s64da_tpcds.query47
Average pg-src-17--.* worse than pg-src-16--.* by 8.4 percents (2.32 > 2.14): 
s64da_tpcds.query47
Best pg-src-17--.* better than pg-src-16--.* by 5.2 percents (0.73 < 0.77): 
s64da_tpcds.query48
Best pg-src-17--.* worse than pg-src-16--.* by 6.7 percents (0.16 > 0.15): 
s64da_tpcds.query49
Best pg-src-17--.* better than pg-src-16--.* by 25.0 percents (0.06 < 0.08): 
s64da_tpcds.query50
Average pg-src-17--.* better than pg-src-16--.* by 16.7 percents (0.07 < 0.08): 
s64da_tpcds.query50
Best pg-src-17--.* better than pg-src-16--.* by 7.3 percents (2.41 < 2.60): 
s64da_tpcds.query51
Average pg-src-17--.* better than pg-src-16--.* by 7.2 percents (2.43 < 2.62): 
s64da_tpcds.query51
Best pg-src-17--.* worse than pg-src-16--.* by 10.3 percents (1.18 > 1.07): 
s64da_tpcds.query57
Average pg-src-17--.* worse than pg-src-16--.* by 10.4 percents (1.20 > 1.09): 
s64da_tpcds.query57
Best pg-src-17--.* better than pg-src-16--.* by 15.2 percents (1.34 < 1.58): 
s64da_tpcds.query59
Average pg-src-17--.* better than pg-src-16--.* by 15.1 percents (1.35 < 1.59): 
s64da_tpcds.query59
Best pg-src-17--.* better than pg-src-16--.* by 5.2 percents (0.55 < 0.58): 
s64da_tpcds.query64
Average pg-src-17--.* better than pg-src-16--.* by 5.1 percents (0.56 < 0.59): 
s64da_tpcds.query64
Best pg-src-17--.* better than pg-src-16--.* by 8.2 percents (1.35 < 1.47): 
s64da_tpcds.query65
Average pg-src-17--.* better than pg-src-16--.* by 7.7 percents (1.36 < 1.47): 
s64da_tpcds.query65
Best pg-src-17--.* better than pg-src-16--.* by 6.7 percents (0.28 < 0.30): 
s64da_tpcds.query66
Average pg-src-17--.* better than pg-src-16--.* by 5.7 percents (1.86 < 1.97): 
s64da_tpcds.query67
Best pg-src-17--.* better than pg-src-16--.* by 8.7 percents (0.21 < 0.23): 
s64da_tpcds.query68
Average pg-src-17--.* better than pg-src-16--.* by 8.7 percents (0.21 < 0.23): 
s64da_tpcds.query68
Best pg-src-17--.* better than pg-src-16--.* by 5.3 percents (1.07 < 1.13): 
s64da_tpcds.query70
Average pg-src-17--.* better than pg-src-16--.* by 5.3 percents (1.07 < 1.13): 
s64da_tpcds.query70
Best pg-src-17--.* better than pg-src-16--.* by 5.0 percents (0.19 < 0.20): 
s64da_tpcds.query73
Average pg-src-17--.* better than pg-src-16--.* by 8.1 percents (0.19 < 0.21): 
s64da_tpcds.query73
Best pg-src-17--.* better than pg-src-16--.* by 11.2 percents (1.83 < 2.06): 
s64da_tpcds.query75
Average pg-src-17--.* better than pg-src-16--.* by 10.7 percents (1.84 < 2.06): 
s64da_tpcds.query75
Best pg-src-17--.* better than pg-src-16--.* by 5.1 percents (0.37 < 0.39): 
s64da_tpcds.query79
Average pg-src-17--.* better than pg-src-16--.* by 5.1 percents (0.37 < 0.39): 
s64da_tpcds.query79
Best pg-src-17--.* better than pg-src-16--.* by 10.1 percents (1.95 < 2.17): 
s64da_tpcds.query87
Average pg-src-17--.* better than pg-src-16--.* by 7.2 percents (2.03 < 2.19): 
s64da_tpcds.query87
Best pg-src-17--.* better than pg-src-16--.* by 8.4 percents (1.53 < 1.67): 
s64da_tpcds.query88
Average pg-src-17--.* better than pg-src-16--.* by 8.1 percents (1.54 < 1.68): 
s64da_tpcds.query88
Best pg-src-17--.* better than pg-src-16--.* by 5.7 percents (0.33 < 0.35): 
s64da_tpcds.query89
Best pg-src-17--.* better than pg-src-16--.* by 6.7 percents (0.14 < 0.15): 
s64da_tpcds.query90
Average pg-src-17--.* better than pg-src-16--.* by 6.7 percents (0.14 < 0.15): 
s64da_tpcds.query90
Best pg-src-17--.* better than pg-src-16--.* by 14.3 percents (0.06 < 0.07): 
s64da_tpcds.query92
Average pg-src-17--.* better than pg-src-16--.* by 18.2 percents (0.06 < 0.07): 
s64da_tpcds.query92
Average pg-src-17--.* better than pg-src-16--.* by 7.7 percents (0.08 < 0.09): 
s64da_tpcds.query94
Best pg-src-17--.* worse than pg-src-16--.* by 12.1 percents (196.51 > 175.36): 
s64da_tpcds.query95
Average pg-src-17--.* worse than pg-src-16--.* by 65.5 percents (305.54 > 
184.60): s64da_tpcds.query95
Best pg-src-17--.* worse than pg-src-16--.* by 43.8 percents (0.23 > 0.16): 
s64da_tpcds.query98
Average pg-src-17--.* worse than pg-src-16--.* by 24.6 percents (0.24 > 0.19): 
s64da_tpcds.query98
Best pg-src-17--.* worse than pg-src-16--.* by 48.6 percents (0.55 > 0.37): 
s64da_tpcds.query99
Average pg-src-17--.* worse than pg-src-16--.* by 61.1 percents (0.61 > 0.38): 
s64da_tpcds.query99
Best pg-src-17--.* better than pg-src-16--.* by 6.0 percents (30.60 < 32.57): 
s64da_ssb.total
Average pg-src-17--.* better than pg-src-16--.* by 6.3 percents (30.85 < 
32.93): s64da_ssb.total
Best pg-src-17--.* better than pg-src-16--.* by 6.7 percents (2.36 < 2.53): 
s64da_ssb.queryQ1.1
Average pg-src-17--.* better than pg-src-16--.* by 6.4 percents (2.37 < 2.53): 
s64da_ssb.queryQ1.1
Best pg-src-17--.* better than pg-src-16--.* by 5.8 percents (2.29 < 2.43): 
s64da_ssb.queryQ1.2
Average pg-src-17--.* better than pg-src-16--.* by 5.6 percents (2.30 < 2.44): 
s64da_ssb.queryQ1.2
Best pg-src-17--.* better than pg-src-16--.* by 6.6 percents (2.25 < 2.41): 
s64da_ssb.queryQ1.3
Average pg-src-17--.* better than pg-src-16--.* by 7.0 percents (2.26 < 2.43): 
s64da_ssb.queryQ1.3
Best pg-src-17--.* better than pg-src-16--.* by 5.9 percents (2.23 < 2.37): 
s64da_ssb.queryQ2.2
Average pg-src-17--.* better than pg-src-16--.* by 5.9 percents (2.24 < 2.38): 
s64da_ssb.queryQ2.2
Best pg-src-17--.* better than pg-src-16--.* by 7.6 percents (1.95 < 2.11): 
s64da_ssb.queryQ2.3
Average pg-src-17--.* better than pg-src-16--.* by 7.8 percents (1.96 < 2.13): 
s64da_ssb.queryQ2.3
Best pg-src-17--.* better than pg-src-16--.* by 16.3 percents (2.92 < 3.49): 
s64da_ssb.queryQ3.1
Average pg-src-17--.* better than pg-src-16--.* by 8.9 percents (3.18 < 3.49): 
s64da_ssb.queryQ3.1
Best pg-src-17--.* better than pg-src-16--.* by 9.9 percents (1.91 < 2.12): 
s64da_ssb.queryQ3.2
Average pg-src-17--.* better than pg-src-16--.* by 14.4 percents (1.92 < 2.25): 
s64da_ssb.queryQ3.2
Best pg-src-17--.* better than pg-src-16--.* by 6.5 percents (1.74 < 1.86): 
s64da_ssb.queryQ3.3
Average pg-src-17--.* better than pg-src-16--.* by 6.4 percents (1.75 < 1.87): 
s64da_ssb.queryQ3.3
Best pg-src-17--.* better than pg-src-16--.* by 7.0 percents (1.74 < 1.87): 
s64da_ssb.queryQ3.4
Average pg-src-17--.* better than pg-src-16--.* by 6.8 percents (1.74 < 1.87): 
s64da_ssb.queryQ3.4
Best pg-src-17--.* better than pg-src-16--.* by 6.2 percents (3.04 < 3.24): 
s64da_ssb.queryQ4.1
Average pg-src-17--.* better than pg-src-16--.* by 6.4 percents (3.06 < 3.27): 
s64da_ssb.queryQ4.1
Best pg-src-17--.* better than pg-src-16--.* by 5.5 percents (2.90 < 3.07): 
s64da_ssb.queryQ4.2
Average pg-src-17--.* better than pg-src-16--.* by 6.0 percents (2.90 < 3.09): 
s64da_ssb.queryQ4.2
Best pg-src-17--.* worse than pg-src-16--.* by 9.4 percents (2.55 > 2.33): 
s64da_ssb.queryQ4.3
Best pg-src-17--.* worse than pg-src-16--.* by 7.7 percents (0.28 > 0.26): 
job.query11a
Average pg-src-17--.* better than pg-src-16--.* by 6.9 percents (0.09 < 0.10): 
job.query11b
Average pg-src-17--.* worse than pg-src-16--.* by 17.5 percents (1.32 > 1.13): 
job.query11c
Average pg-src-17--.* worse than pg-src-16--.* by 5.6 percents (4.82 > 4.57): 
job.query11d
Best pg-src-17--.* worse than pg-src-16--.* by 5.3 percents (0.40 > 0.38): 
job.query12a
Average pg-src-17--.* worse than pg-src-16--.* by 6.1 percents (0.70 > 0.66): 
job.query14a
Average pg-src-17--.* better than pg-src-16--.* by 20.0 percents (0.04 < 0.05): 
job.query15b
Best pg-src-17--.* better than pg-src-16--.* by 15.3 percents (0.72 < 0.85): 
job.query15c
Average pg-src-17--.* better than pg-src-16--.* by 16.0 percents (0.72 < 0.86): 
job.query15c
Average pg-src-17--.* better than pg-src-16--.* by 5.2 percents (0.91 < 0.96): 
job.query15d
Best pg-src-17--.* worse than pg-src-16--.* by 14.3 percents (0.24 > 0.21): 
job.query16a
Average pg-src-17--.* worse than pg-src-16--.* by 9.0 percents (0.24 > 0.22): 
job.query16a
Best pg-src-17--.* better than pg-src-16--.* by 5.3 percents (0.36 < 0.38): 
job.query18a
Best pg-src-17--.* worse than pg-src-16--.* by 1170.0 percents (3.81 > 0.30): 
job.query19a
Average pg-src-17--.* worse than pg-src-16--.* by 1181.1 percents (3.84 > 
0.30): job.query19a
Best pg-src-17--.* better than pg-src-16--.* by 5.3 percents (0.18 < 0.19): 
job.query19b
Average pg-src-17--.* better than pg-src-16--.* by 8.5 percents (0.18 < 0.20): 
job.query19b
Average pg-src-17--.* better than pg-src-16--.* by 10.0 percents (0.06 < 0.07): 
job.query1a
Best pg-src-17--.* better than pg-src-16--.* by 16.7 percents (0.05 < 0.06): 
job.query1b
Average pg-src-17--.* better than pg-src-16--.* by 16.7 percents (0.05 < 0.06): 
job.query1b
Best pg-src-17--.* better than pg-src-16--.* by 16.7 percents (0.05 < 0.06): 
job.query1d
Average pg-src-17--.* better than pg-src-16--.* by 16.7 percents (0.05 < 0.06): 
job.query1d
Best pg-src-17--.* better than pg-src-16--.* by 5.6 percents (0.34 < 0.36): 
job.query23c
Average pg-src-17--.* better than pg-src-16--.* by 26.0 percents (0.49 < 0.67): 
job.query23c
Average pg-src-17--.* better than pg-src-16--.* by 35.3 percents (0.07 < 0.11): 
job.query24b
Average pg-src-17--.* better than pg-src-16--.* by 30.3 percents (3.96 < 5.68): 
job.query25a
Best pg-src-17--.* worse than pg-src-16--.* by 122.2 percents (0.80 > 0.36): 
job.query26a
Average pg-src-17--.* worse than pg-src-16--.* by 72.9 percents (0.81 > 0.47): 
job.query26a
Average pg-src-17--.* worse than pg-src-16--.* by 46.3 percents (0.26 > 0.18): 
job.query26b
Best pg-src-17--.* worse than pg-src-16--.* by 188.0 percents (2.16 > 0.75): 
job.query26c
Average pg-src-17--.* worse than pg-src-16--.* by 40.8 percents (2.19 > 1.55): 
job.query26c
Average pg-src-17--.* better than pg-src-16--.* by 15.7 percents (0.34 < 0.40): 
job.query27a
Best pg-src-17--.* worse than pg-src-16--.* by 33.3 percents (0.20 > 0.15): 
job.query27b
Average pg-src-17--.* worse than pg-src-16--.* by 25.5 percents (0.21 > 0.17): 
job.query27b
Best pg-src-17--.* worse than pg-src-16--.* by 45.1 percents (0.74 > 0.51): 
job.query27c
Average pg-src-17--.* worse than pg-src-16--.* by 12.1 percents (0.74 > 0.66): 
job.query27c
Best pg-src-17--.* better than pg-src-16--.* by 10.8 percents (0.83 < 0.93): 
job.query28a
Average pg-src-17--.* worse than pg-src-16--.* by 172.2 percents (2.55 > 0.94): 
job.query28a
Best pg-src-17--.* better than pg-src-16--.* by 10.4 percents (0.60 < 0.67): 
job.query28b
Average pg-src-17--.* better than pg-src-16--.* by 27.2 percents (0.69 < 0.94): 
job.query28b
Best pg-src-17--.* worse than pg-src-16--.* by 415.2 percents (2.37 > 0.46): 
job.query28c
Average pg-src-17--.* worse than pg-src-16--.* by 37.5 percents (2.37 > 1.73): 
job.query28c
Average pg-src-17--.* better than pg-src-16--.* by 98.0 percents (0.12 < 5.98): 
job.query29a
Average pg-src-17--.* better than pg-src-16--.* by 80.6 percents (0.08 < 0.41): 
job.query29b
Average pg-src-17--.* worse than pg-src-16--.* by 130.4 percents (1.21 > 0.53): 
job.query29c
Best pg-src-17--.* better than pg-src-16--.* by 38.6 percents (1.67 < 2.72): 
job.query30a
Average pg-src-17--.* better than pg-src-16--.* by 8.6 percents (5.94 < 6.50): 
job.query30a
Best pg-src-17--.* better than pg-src-16--.* by 8.3 percents (0.77 < 0.84): 
job.query33a
Average pg-src-17--.* better than pg-src-16--.* by 11.4 percents (0.80 < 0.91): 
job.query33a
Best pg-src-17--.* better than pg-src-16--.* by 17.3 percents (0.81 < 0.98): 
job.query33b
Average pg-src-17--.* better than pg-src-16--.* by 54.7 percents (1.22 < 2.68): 
job.query33b
Best pg-src-17--.* worse than pg-src-16--.* by 178.9 percents (2.51 > 0.90): 
job.query33c
Average pg-src-17--.* worse than pg-src-16--.* by 27.1 percents (2.52 > 1.98): 
job.query33c
Average pg-src-17--.* better than pg-src-16--.* by 18.2 percents (0.18 < 0.22): 
job.query3a
Best pg-src-17--.* better than pg-src-16--.* by 5.9 percents (0.16 < 0.17): 
job.query5a
Average pg-src-17--.* better than pg-src-16--.* by 5.9 percents (0.16 < 0.17): 
job.query5a
Best pg-src-17--.* better than pg-src-16--.* by 6.3 percents (0.15 < 0.16): 
job.query5b
Average pg-src-17--.* better than pg-src-16--.* by 6.3 percents (0.15 < 0.16): 
job.query5b
Best pg-src-17--.* worse than pg-src-16--.* by 10.7 percents (1.35 > 1.22): 
job.query6b
Average pg-src-17--.* worse than pg-src-16--.* by 10.8 percents (1.37 > 1.24): 
job.query6b
Best pg-src-17--.* worse than pg-src-16--.* by 33.3 percents (0.04 > 0.03): 
job.query6c
Average pg-src-17--.* worse than pg-src-16--.* by 33.3 percents (0.04 > 0.03): 
job.query6c
Best pg-src-17--.* worse than pg-src-16--.* by 11.0 percents (7.15 > 6.44): 
job.query6d
Average pg-src-17--.* worse than pg-src-16--.* by 10.8 percents (7.26 > 6.55): 
job.query6d
Best pg-src-17--.* worse than pg-src-16--.* by 11.3 percents (7.20 > 6.47): 
job.query6f
Average pg-src-17--.* worse than pg-src-16--.* by 11.0 percents (7.36 > 6.63): 
job.query6f
Best pg-src-17--.* worse than pg-src-16--.* by 9.1 percents (0.24 > 0.22): 
job.query7a
Best pg-src-17--.* better than pg-src-16--.* by 10.0 percents (0.18 < 0.20): 
job.query7b
Average pg-src-17--.* better than pg-src-16--.* by 8.3 percents (0.18 < 0.20): 
job.query7b
Best pg-src-17--.* better than pg-src-16--.* by 15.0 percents (15.16 < 17.83): 
job.query7c
Average pg-src-17--.* better than pg-src-16--.* by 7.5 percents (20.57 < 
22.25): job.query7c
Best pg-src-17--.* better than pg-src-16--.* by 6.1 percents (2.94 < 3.13): 
job.query8d
Average pg-src-17--.* better than pg-src-16--.* by 5.5 percents (2.97 < 3.14): 
job.query8d
Best pg-src-17--.* worse than pg-src-16--.* by 771.4 percents (2.44 > 0.28): 
job.query9a
Average pg-src-17--.* worse than pg-src-16--.* by 763.5 percents (2.45 > 0.28): 
job.query9a
Best pg-src-17--.* worse than pg-src-16--.* by 433.3 percents (1.44 > 0.27): 
job.query9b
Average pg-src-17--.* worse than pg-src-16--.* by 437.0 percents (1.45 > 0.27): 
job.query9b
Best pg-src-17--.* better than pg-src-16--.* by 6.6 percents (2.26 < 2.42): 
job.query9c
Average pg-src-17--.* better than pg-src-16--.* by 6.7 percents (2.27 < 2.43): 
job.query9c
Best pg-src-17--.* worse than pg-src-16--.* by 7.8 percents (4.26 > 3.95): 
job.query9d
Best pg-src-17--.* better than pg-src-16--.* by 19.7 percents (264570.35 > 
221029.48): sysbench.load_select_random_points_tps
Average pg-src-17--.* better than pg-src-16--.* by 19.8 percents (263014.87 > 
219587.61): sysbench.load_select_random_points_tps
Best pg-src-17--.* better than pg-src-16--.* by 5.3 percents (80.09 > 76.06): 
sysbench.load_bulk_insert_ops
Average pg-src-17--.* better than pg-src-16--.* by 5.0 percents (79.05 > 
75.28): sysbench.load_bulk_insert_ops
Best pg-src-17--.* better than pg-src-16--.* by 19.7 percents (264570.35 > 
221029.48): sysbench.load_select_random_points_ops
Average pg-src-17--.* better than pg-src-16--.* by 19.8 percents (263014.87 > 
219587.61): sysbench.load_select_random_points_ops