On 2017-02-16 00:43, Petr Jelinek wrote:
On 13/02/17 14:51, Erik Rijkers wrote:
On 2017-02-11 11:16, Erik Rijkers wrote:
On 2017-02-08 23:25, Petr Jelinek wrote:

0001-Use-asynchronous-connect-API-in-libpqwalreceiver-v2.patch
0002-Always-initialize-stringinfo-buffers-in-walsender-v2.patch
0003-Fix-after-trigger-execution-in-logical-replication-v2.patch
0004-Add-RENAME-support-for-PUBLICATIONs-and-SUBSCRIPTION-v2.patch
0001-Logical-replication-support-for-initial-data-copy-v4.patch

This often works but it also fails far too often (in my hands).  I

Could you periodically dump contents of the pg_subscription_rel on
subscriber (ideally when dumping the md5 of the data) and attach that as
well?

I attach a bash script (and its output) that polls the 4 pgbench table's md5s and the pg_subscription_rel table, each second, while I run the pgbench_derail2.sh (for that see my earlier mail).

pgbench_derail2.sh writes a 'header' into the same output stream (search for '^===' ).

The .out file reflects a session where I started pgbench_derail2.sh twice (it removes the publication and subscription at startup). So there are 2 headers in the attached cb_20170216_10_04_47.out. The first run ended in a succesful replication (=all 4 pgbench tables md5-identical). The second run does not end correctly: it has (one of) the typical faulty end-states: pgbench_accounts, the copy, has a few less rows than the master table.

Other typical endstates are:
same number of rows but content not identical (for some, typically < 20 rows).
mostly pgbench_accounts and pgbench_history are affected.

(I see now that I made some mistakes in generating the timestamps in the .out file but I suppose it doesn't matter too much)

I hope it helps; let me know if I can do any other test(s).

20170216_10_04_49_1487  6972 a,b,t,h: 100000      1     10    776   24be8c7be  
cf860f1f2  aed87334f  f2bfaa587   master
20170216_10_04_50_1487  6973 a,b,t,h:  99996      1     10    776   74cd7528c  
cf860f1f2  aed87334f  f2bfaa587   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:04:50.242616+01 |   25398 |   25375 | r          | 
 2017-02-16 10:04:50.242616+01 |   25398 |   25378 | r          | 
 2017-02-16 10:04:50.242616+01 |   25398 |   25381 | r          | 
 2017-02-16 10:04:50.242616+01 |   25398 |   25386 | r          | 
(4 rows)

20170216_10_04_51_1487  6972 a,b,t,h: 100000      1     10    776   24be8c7be  
cf860f1f2  aed87334f  f2bfaa587   master
20170216_10_04_51_1487  6973 a,b,t,h:  99996      1     10    776   74cd7528c  
cf860f1f2  aed87334f  f2bfaa587   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:04:51.945931+01 |   25398 |   25375 | r          | 
 2017-02-16 10:04:51.945931+01 |   25398 |   25378 | r          | 
 2017-02-16 10:04:51.945931+01 |   25398 |   25381 | r          | 
 2017-02-16 10:04:51.945931+01 |   25398 |   25386 | r          | 
(4 rows)

============================================================================
--     20170216 10:04:S
--     scale  1     clients  1   INIT_WAIT  0
-- 
/home/aardvark/pg_stuff/pg_installations/pgsql.logical_replication/bin.fast/postgres
20170216_10_04_53_1487  6972 a,b,t,h: 100000      1     10    776   24be8c7be  
cf860f1f2  aed87334f  f2bfaa587   master
20170216_10_04_53_1487  6973 a,b,t,h:  99996      1     10    776   74cd7528c  
cf860f1f2  aed87334f  f2bfaa587   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:04:53.635163+01 |   25398 |   25375 | r          | 
 2017-02-16 10:04:53.635163+01 |   25398 |   25378 | r          | 
 2017-02-16 10:04:53.635163+01 |   25398 |   25381 | r          | 
 2017-02-16 10:04:53.635163+01 |   25398 |   25386 | r          | 
(4 rows)

20170216_10_04_54_1487  6972 a,b,t,h:      0      0      0      0   24be8c7be  
d41d8cd98  d41d8cd98  d41d8cd98   master
20170216_10_04_55_1487  6973 a,b,t,h:      0      0      0      0   d41d8cd98  
d41d8cd98  d41d8cd98  d41d8cd98   replica NOK
 now | srsubid | srrelid | srsubstate | srsublsn 
-----+---------+---------+------------+----------
(0 rows)

20170216_10_04_56_1487  6972 a,b,t,h: 100000      1     10      0   68d91d95b  
6c4f8b9aa  92162c9b8  d41d8cd98   master
20170216_10_04_56_1487  6973 a,b,t,h:      0      0      0      0   d41d8cd98  
d41d8cd98  d41d8cd98  d41d8cd98   replica NOK
 now | srsubid | srrelid | srsubstate | srsublsn 
-----+---------+---------+------------+----------
(0 rows)

20170216_10_04_57_1487  6972 a,b,t,h: 100000      1     10      1   68d91d95b  
6c4f8b9aa  92162c9b8  d41d8cd98   master
20170216_10_04_58_1487  6973 a,b,t,h:      0      0      0      0   d41d8cd98  
d41d8cd98  d41d8cd98  d41d8cd98   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:04:58.103852+01 |   25426 |   25403 | d          | 
 2017-02-16 10:04:58.103852+01 |   25426 |   25406 | i          | 
 2017-02-16 10:04:58.103852+01 |   25426 |   25409 | i          | 
 2017-02-16 10:04:58.103852+01 |   25426 |   25414 | i          | 
(4 rows)

20170216_10_04_59_1487  6972 a,b,t,h: 100000      1     10     28   ec6d45aa9  
dd01a2d39  9b3c4011c  2b6c34c8f   master
20170216_10_04_59_1487  6973 a,b,t,h: 100000      0      0      0   39948b525  
d41d8cd98  d41d8cd98  d41d8cd98   replica NOK
              now              | srsubid | srrelid | srsubstate |  srsublsn  
-------------------------------+---------+---------+------------+------------
 2017-02-16 10:04:59.780581+01 |   25426 |   25403 | w          | 0/26518170
 2017-02-16 10:04:59.780581+01 |   25426 |   25406 | d          | 
 2017-02-16 10:04:59.780581+01 |   25426 |   25409 | i          | 
 2017-02-16 10:04:59.780581+01 |   25426 |   25414 | i          | 
(4 rows)

20170216_10_05_01_1487  6972 a,b,t,h: 100000      1     10     58   79a3ec4d1  
003871581  6a6435bb5  8099f4d73   master
20170216_10_05_01_1487  6973 a,b,t,h: 100000      1      0     49   39948b525  
e0d08a373  d41d8cd98  d009b674f   replica NOK
              now              | srsubid | srrelid | srsubstate |  srsublsn  
-------------------------------+---------+---------+------------+------------
 2017-02-16 10:05:01.468313+01 |   25426 |   25403 | s          | 0/26518170
 2017-02-16 10:05:01.468313+01 |   25426 |   25406 | s          | 0/265248F8
 2017-02-16 10:05:01.468313+01 |   25426 |   25409 | w          | 0/26527320
 2017-02-16 10:05:01.468313+01 |   25426 |   25414 | d          | 
(4 rows)

20170216_10_05_02_1487  6972 a,b,t,h: 100000      1     10    107   834e625eb  
65620887b  c7224e34f  b5aee4011   master
20170216_10_05_03_1487  6973 a,b,t,h: 100000      1     10     61   c55f9fe26  
003871581  f796399c4  e2a3315dd   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:03.168927+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:03.168927+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:03.168927+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:03.168927+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_04_1487  6972 a,b,t,h: 100000      1     10    176   3e16b6065  
305b019bf  007d3e684  235c2b1e8   master
20170216_10_05_04_1487  6973 a,b,t,h: 100000      1     10    123   d6ba506a4  
f9e6557a4  65b047e3e  366710007   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:04.833204+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:04.833204+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:04.833204+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:04.833204+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_06_1487  6972 a,b,t,h: 100000      1     10    227   c205cb75f  
bc8ab7461  354a64875  ee4cede80   master
20170216_10_05_06_1487  6973 a,b,t,h: 100000      1     10    183   b1257fd54  
80a59e609  4d4e1b8b1  901f5091f   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:06.519971+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:06.519971+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:06.519971+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:06.519971+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_07_1487  6972 a,b,t,h: 100000      1     10    286   7b5da721f  
0b90966de  d8d82f07f  ded5226c1   master
20170216_10_05_08_1487  6973 a,b,t,h: 100000      1     10    245   af8e7b2db  
26935856d  f56d8ded6  3f626d9e1   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:08.196214+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:08.196214+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:08.196214+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:08.196214+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_09_1487  6972 a,b,t,h: 100000      1     10    343   af5764fdb  
b1101edba  4208841bb  794a5c9ec   master
20170216_10_05_09_1487  6973 a,b,t,h: 100000      1     10    308   065f12159  
296715e4d  24c0cc958  0fc085140   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:09.931395+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:09.931395+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:09.931395+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:09.931395+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_11_1487  6972 a,b,t,h: 100000      1     10    396   a75d7f157  
9aa8764b8  76f7947e0  2a8ff0fe2   master
20170216_10_05_11_1487  6973 a,b,t,h: 100000      1     10    368   dc221616c  
ff9ae1f6a  617b253f8  0e7434098   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:11.610951+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:11.610951+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:11.610951+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:11.610951+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_12_1487  6972 a,b,t,h: 100000      1     10    453   da52ac409  
64f780651  854df56cd  8cad593c8   master
20170216_10_05_13_1487  6973 a,b,t,h: 100000      1     10    427   dfdf59e69  
59d864864  83455e5a8  7935150b6   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:13.268571+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:13.268571+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:13.268571+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:13.268571+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_14_1487  6972 a,b,t,h: 100000      1     10    512   48aa40ca0  
6b18fe4d3  28467fc6a  e64f6fe53   master
20170216_10_05_14_1487  6973 a,b,t,h: 100000      1     10    492   0fcff56fa  
d3dd1957d  3d6313610  b6b5eb8a5   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:14.976636+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:14.976636+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:14.976636+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:14.976636+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_16_1487  6972 a,b,t,h: 100000      1     10    568   c3707ce2f  
1b33b2fa2  e4acf88de  12592c7e7   master
20170216_10_05_16_1487  6973 a,b,t,h: 100000      1     10    543   6c306c810  
a16430907  887867b04  03c57041d   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:16.656157+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:16.656157+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:16.656157+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:16.656157+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_18_1487  6972 a,b,t,h: 100000      1     10    617   30609390b  
f144cedbb  c268012bd  4778a0383   master
20170216_10_05_18_1487  6973 a,b,t,h: 100000      1     10    617   7c73d5bbf  
f2c98c45a  c268012bd  4778a0383   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:18.366128+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:18.366128+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:18.366128+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:18.366128+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_19_1487  6972 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   master
20170216_10_05_20_1487  6973 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   replica ok
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:20.068776+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:20.068776+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:20.068776+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:20.068776+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_21_1487  6972 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   master
20170216_10_05_21_1487  6973 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   replica ok
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:21.754966+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:21.754966+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:21.754966+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:21.754966+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_23_1487  6972 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   master
20170216_10_05_23_1487  6973 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   replica ok
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:23.421029+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:23.421029+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:23.421029+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:23.421029+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_24_1487  6972 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   master
20170216_10_05_25_1487  6973 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   replica ok
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:25.067658+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:25.067658+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:25.067658+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:25.067658+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_26_1487  6972 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   master
20170216_10_05_26_1487  6973 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   replica ok
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:26.737333+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:26.737333+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:26.737333+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:26.737333+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_28_1487  6972 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   master
20170216_10_05_28_1487  6973 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   replica ok
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:28.424525+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:28.424525+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:28.424525+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:28.424525+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_29_1487  6972 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   master
20170216_10_05_30_1487  6973 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   replica ok
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:30.124997+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:30.124997+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:30.124997+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:30.124997+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_31_1487  6972 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   master
20170216_10_05_31_1487  6973 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   replica ok
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:31.806628+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:31.806628+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:31.806628+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:31.806628+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_33_1487  6972 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   master
20170216_10_05_33_1487  6973 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   replica ok
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:33.476968+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:33.476968+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:33.476968+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:33.476968+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_34_1487  6972 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   master
20170216_10_05_35_1487  6973 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   replica ok
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:35.185978+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:35.185978+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:35.185978+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:35.185978+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_36_1487  6972 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   master
20170216_10_05_36_1487  6973 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   replica ok
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:36.868243+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:36.868243+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:36.868243+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:36.868243+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_38_1487  6972 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   master
20170216_10_05_38_1487  6973 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   replica ok
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:38.525211+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:38.525211+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:38.525211+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:38.525211+01 |   25426 |   25414 | r          | 
(4 rows)

============================================================================
--     20170216 10:05:S
--     scale  1     clients  1   INIT_WAIT  0
-- 
/home/aardvark/pg_stuff/pg_installations/pgsql.logical_replication/bin.fast/postgres
20170216_10_05_39_1487  6972 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   master
20170216_10_05_40_1487  6973 a,b,t,h: 100000      1     10    617   9edc2a8f6  
f144cedbb  c268012bd  4778a0383   replica ok
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:40.209912+01 |   25426 |   25403 | r          | 
 2017-02-16 10:05:40.209912+01 |   25426 |   25406 | r          | 
 2017-02-16 10:05:40.209912+01 |   25426 |   25409 | r          | 
 2017-02-16 10:05:40.209912+01 |   25426 |   25414 | r          | 
(4 rows)

20170216_10_05_42_1487  6972 a,b,t,h: 100000      1     10      0   68d91d95b  
6c4f8b9aa  92162c9b8  d41d8cd98   master
20170216_10_05_42_1487  6973 a,b,t,h:      0      0      0      0   d41d8cd98  
d41d8cd98  d41d8cd98  d41d8cd98   replica NOK
 now | srsubid | srrelid | srsubstate | srsublsn 
-----+---------+---------+------------+----------
(0 rows)

20170216_10_05_43_1487  6972 a,b,t,h: 100000      1     10      0   68d91d95b  
6c4f8b9aa  92162c9b8  d41d8cd98   master
20170216_10_05_43_1487  6973 a,b,t,h:      0      0      0      0   d41d8cd98  
d41d8cd98  d41d8cd98  d41d8cd98   replica NOK
 now | srsubid | srrelid | srsubstate | srsublsn 
-----+---------+---------+------------+----------
(0 rows)

20170216_10_05_44_1487  6972 a,b,t,h: 100000      1     10     16   1f5276ceb  
769b5e247  f5b6279d9  7878d43f0   master
20170216_10_05_45_1487  6973 a,b,t,h:      0      0      0      0   d41d8cd98  
d41d8cd98  d41d8cd98  d41d8cd98   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:45.152976+01 |   25454 |   25431 | d          | 
 2017-02-16 10:05:45.152976+01 |   25454 |   25434 | i          | 
 2017-02-16 10:05:45.152976+01 |   25454 |   25437 | i          | 
 2017-02-16 10:05:45.152976+01 |   25454 |   25442 | i          | 
(4 rows)

20170216_10_05_46_1487  6972 a,b,t,h: 100000      1     10     47   cd9b785da  
73c97454b  6a583aab6  4eb307a9e   master
20170216_10_05_46_1487  6973 a,b,t,h:  99994      1      0      0   1200b6523  
7acb6b66b  d41d8cd98  d41d8cd98   replica NOK
              now              | srsubid | srrelid | srsubstate |  srsublsn  
-------------------------------+---------+---------+------------+------------
 2017-02-16 10:05:46.816859+01 |   25454 |   25431 | w          | 0/2737F820
 2017-02-16 10:05:46.816859+01 |   25454 |   25434 | w          | 0/27383EC8
 2017-02-16 10:05:46.816859+01 |   25454 |   25437 | d          | 
 2017-02-16 10:05:46.816859+01 |   25454 |   25442 | i          | 
(4 rows)

20170216_10_05_48_1487  6972 a,b,t,h: 100000      1     10     88   eb2b2ed4c  
b569a6f6a  4cb97000d  98f4ca1f3   master
20170216_10_05_48_1487  6973 a,b,t,h:  99994      1     10     47   50e5ea392  
7acb6b66b  f9ae8a633  490b76eb3   replica NOK
              now              | srsubid | srrelid | srsubstate |  srsublsn  
-------------------------------+---------+---------+------------+------------
 2017-02-16 10:05:48.500204+01 |   25454 |   25431 | r          | 
 2017-02-16 10:05:48.500204+01 |   25454 |   25434 | s          | 0/27383EC8
 2017-02-16 10:05:48.500204+01 |   25454 |   25437 | s          | 0/273867A8
 2017-02-16 10:05:48.500204+01 |   25454 |   25442 | s          | 0/27389B00
(4 rows)

20170216_10_05_49_1487  6972 a,b,t,h: 100000      1     10    140   79bcd81c7  
fda632daa  9f913e044  e33366ef9   master
20170216_10_05_50_1487  6973 a,b,t,h:  99994      1     10     89   c6d4ab30f  
b569a6f6a  4cb97000d  98f4ca1f3   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:50.177364+01 |   25454 |   25431 | r          | 
 2017-02-16 10:05:50.177364+01 |   25454 |   25434 | r          | 
 2017-02-16 10:05:50.177364+01 |   25454 |   25437 | r          | 
 2017-02-16 10:05:50.177364+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_05_51_1487  6972 a,b,t,h: 100000      1     10    203   b42acfb83  
df46a249c  4f7f4c7f0  403287a49   master
20170216_10_05_51_1487  6973 a,b,t,h:  99994      1     10    155   dadd350d7  
20d70cac0  23ac08826  6f1a2a3d0   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:51.880396+01 |   25454 |   25431 | r          | 
 2017-02-16 10:05:51.880396+01 |   25454 |   25434 | r          | 
 2017-02-16 10:05:51.880396+01 |   25454 |   25437 | r          | 
 2017-02-16 10:05:51.880396+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_05_53_1487  6972 a,b,t,h: 100000      1     10    266   3941d00b6  
b75f6fe77  b2dbfd6b3  94e575779   master
20170216_10_05_53_1487  6973 a,b,t,h:  99994      1     10    218   caf05e592  
21fe0b540  08841fd45  aab521584   replica NOK
             now              | srsubid | srrelid | srsubstate | srsublsn 
------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:53.57113+01 |   25454 |   25431 | r          | 
 2017-02-16 10:05:53.57113+01 |   25454 |   25434 | r          | 
 2017-02-16 10:05:53.57113+01 |   25454 |   25437 | r          | 
 2017-02-16 10:05:53.57113+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_05_54_1487  6972 a,b,t,h: 100000      1     10    328   d70964c0d  
d6b0a4409  be1f6268e  7985ebfae   master
20170216_10_05_55_1487  6973 a,b,t,h:  99994      1     10    285   b1bc6eb68  
0c95fcea5  986a299ea  4a6b23646   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:55.252785+01 |   25454 |   25431 | r          | 
 2017-02-16 10:05:55.252785+01 |   25454 |   25434 | r          | 
 2017-02-16 10:05:55.252785+01 |   25454 |   25437 | r          | 
 2017-02-16 10:05:55.252785+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_05_56_1487  6972 a,b,t,h: 100000      1     10    388   185116b14  
b3e476238  ed98808b3  bdf3daaf3   master
20170216_10_05_56_1487  6973 a,b,t,h:  99994      1     10    340   61cbbf6a1  
13d3d8e5e  050718581  f9dfd78ae   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:56.938827+01 |   25454 |   25431 | r          | 
 2017-02-16 10:05:56.938827+01 |   25454 |   25434 | r          | 
 2017-02-16 10:05:56.938827+01 |   25454 |   25437 | r          | 
 2017-02-16 10:05:56.938827+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_05_58_1487  6972 a,b,t,h: 100000      1     10    438   87560adf9  
ee144c33b  6977fd77a  84a7add1a   master
20170216_10_05_58_1487  6973 a,b,t,h:  99994      1     10    400   b3a5a5089  
f1c434247  72ed3534c  1227305b2   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:05:58.632431+01 |   25454 |   25431 | r          | 
 2017-02-16 10:05:58.632431+01 |   25454 |   25434 | r          | 
 2017-02-16 10:05:58.632431+01 |   25454 |   25437 | r          | 
 2017-02-16 10:05:58.632431+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_05_59_1487  6972 a,b,t,h: 100000      1     10    482   6fc5d5b9a  
c037ede93  2baeac00a  5f727749d   master
20170216_10_06_00_1487  6973 a,b,t,h:  99994      1     10    444   c24701cae  
3ada4e55c  488f7b4de  eff199816   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:00.336906+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:00.336906+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:00.336906+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:00.336906+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_01_1487  6972 a,b,t,h: 100000      1     10    573   c59fdfdac  
c64404239  e0da40285  045982840   master
20170216_10_06_02_1487  6973 a,b,t,h:  99994      1     10    549   e6a6888fd  
2b759e10b  bb9e0ed28  519888cba   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:02.048506+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:02.048506+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:02.048506+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:02.048506+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_03_1487  6972 a,b,t,h: 100000      1     10    683   31bcac7a9  
d73346d36  8102a796a  8c3b582b7   master
20170216_10_06_03_1487  6973 a,b,t,h:  99994      1     10    666   263f932a8  
15e91e4e2  44039d9ca  bbc538273   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:03.768224+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:03.768224+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:03.768224+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:03.768224+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_05_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_05_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
             now              | srsubid | srrelid | srsubstate | srsublsn 
------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:05.43718+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:05.43718+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:05.43718+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:05.43718+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_06_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_07_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
             now              | srsubid | srrelid | srsubstate | srsublsn 
------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:07.10223+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:07.10223+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:07.10223+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:07.10223+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_08_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_08_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:08.796768+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:08.796768+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:08.796768+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:08.796768+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_10_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_10_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:10.498424+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:10.498424+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:10.498424+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:10.498424+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_11_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_12_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
             now              | srsubid | srrelid | srsubstate | srsublsn 
------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:12.15206+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:12.15206+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:12.15206+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:12.15206+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_13_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_13_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:13.847929+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:13.847929+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:13.847929+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:13.847929+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_15_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_15_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:15.512172+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:15.512172+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:15.512172+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:15.512172+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_16_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_17_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:17.192081+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:17.192081+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:17.192081+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:17.192081+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_18_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_18_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:18.896788+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:18.896788+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:18.896788+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:18.896788+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_20_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_20_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:20.606603+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:20.606603+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:20.606603+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:20.606603+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_21_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_22_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:22.285976+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:22.285976+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:22.285976+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:22.285976+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_23_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_23_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:23.962149+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:23.962149+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:23.962149+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:23.962149+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_25_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_25_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:25.656771+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:25.656771+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:25.656771+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:25.656771+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_27_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_27_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:27.357104+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:27.357104+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:27.357104+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:27.357104+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_28_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_29_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:29.048487+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:29.048487+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:29.048487+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:29.048487+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_30_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_30_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:30.737921+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:30.737921+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:30.737921+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:30.737921+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_32_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_32_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:32.418458+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:32.418458+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:32.418458+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:32.418458+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_33_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_34_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:34.090288+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:34.090288+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:34.090288+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:34.090288+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_35_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_35_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:35.763712+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:35.763712+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:35.763712+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:35.763712+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_37_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_37_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:37.445038+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:37.445038+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:37.445038+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:37.445038+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_38_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_39_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:39.151441+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:39.151441+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:39.151441+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:39.151441+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_40_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_40_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:40.817596+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:40.817596+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:40.817596+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:40.817596+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_42_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_42_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:42.507035+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:42.507035+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:42.507035+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:42.507035+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_43_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_44_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:44.157016+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:44.157016+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:44.157016+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:44.157016+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_45_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_45_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
              now              | srsubid | srrelid | srsubstate | srsublsn 
-------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:45.791053+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:45.791053+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:45.791053+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:45.791053+01 |   25454 |   25442 | r          | 
(4 rows)

20170216_10_06_47_1487  6972 a,b,t,h: 100000      1     10    732   f0574e6e2  
c914d51a9  31f041438  d80263699   master
20170216_10_06_47_1487  6973 a,b,t,h:  99994      1     10    732   b87eb9b40  
c914d51a9  31f041438  d80263699   replica NOK
             now              | srsubid | srrelid | srsubstate | srsublsn 
------------------------------+---------+---------+------------+----------
 2017-02-16 10:06:47.48427+01 |   25454 |   25431 | r          | 
 2017-02-16 10:06:47.48427+01 |   25454 |   25434 | r          | 
 2017-02-16 10:06:47.48427+01 |   25454 |   25437 | r          | 
 2017-02-16 10:06:47.48427+01 |   25454 |   25442 | r          | 
(4 rows)

#!/bin/sh

#
#  cb.sh - display pgbench tables content-md5's
#

# outf=cb_$( date +"%Y%m%d %H:%M:%S %s%N" | cut -b1-22 | perl -ne 's{[: ]}{_}g;print ' ).out
  outf=cb_$( date +"%Y%m%d %H:%M:%S"                   | perl -ne 's{[: ]}{_}g;print ' ).out

SLEEP=1
echo >$outf
echo
echo "tail -F $outf"
echo 
rc=0
while  [[ $rc -eq 0 ]]
do
   # num_tables=$( echo "select count(*) from information_schema.tables where table_schema = 'public' and table_name ~ '^pgbench_'" | psql -qtAX )
   num_tables=$( echo "select count(*) from pg_class where relkind = 'r' and relname ~ '^pgbench_'" | psql -qtAX )
   if [[ $num_tables -ne 4 ]] 
   then
      echo "pgbench tables not 4 - bailing out" >> $outf
      sleep $SLEEP
      continue
   fi
   for port in 6972 6973
   do
     md5_a=$( echo "select * from pgbench_accounts /* where aid % 100 = 0 */ order by aid"|psql -qtAXp$port | md5sum | cut -b 1-9 )
     md5_b=$( echo "select * from pgbench_branches /* where bid % 100 = 0 */ order by bid"|psql -qtAXp$port | md5sum | cut -b 1-9 )
     md5_t=$( echo "select * from pgbench_tellers  /* where tid % 100 = 0 */ order by tid"|psql -qtAXp$port | md5sum | cut -b 1-9 )
     md5_h=$( echo "select * from pgbench_history  /* where hid % 100 = 0 */ order by hid"|psql -qtAXp$port | md5sum | cut -b 1-9 )
     md5_total[$port]=$( echo "${md5_a} ${md5_b} ${md5_t} ${md5_h}" | md5sum )
     cnt_a=$(echo "select count(*) from pgbench_accounts"|psql -qtAXp $port)
     cnt_b=$(echo "select count(*) from pgbench_branches"|psql -qtAXp $port)
     cnt_t=$(echo "select count(*) from pgbench_tellers" |psql -qtAXp $port)
     cnt_h=$(echo "select count(*) from pgbench_history" |psql -qtAXp $port)
     d=$( date +"%Y%m%d %H:%M:%S %s%N " | cut -b1-22 | perl -ne 'chomp; s{[: ]}{_}g; print ' )
     printf "$d  $port a,b,t,h: %6d %6d %6d %6d" $cnt_a  $cnt_b  $cnt_t  $cnt_h  >> $outf
     echo -n "   $md5_a  $md5_b  $md5_t  $md5_h"                                 >> $outf
     if   [[ $port -eq 6972 ]]; then echo    "   master"                         >> $outf
     elif [[ $port -eq 6973 ]]; then echo -n "   replica"                        >> $outf
     else                            echo "        ERROR"                        >> $outf
     fi
   done
   if [[ "${md5_total[6972]}" == "${md5_total[6973]}" ]]
   then
     echo " ok"  >> $outf
   else
     echo " NOK" >> $outf
   fi
   echo 'select now(), * from pg_subscription_rel order by srrelid; 
      -- select now(), * from pg_stat_subscription' | psql -qXp6973 >> $outf
   sleep $SLEEP
   rc=$?
done

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers

Reply via email to