It shows how the data is distributed. Looking at the size of the persistence files isn’t necessarily a good metric, as Ilya notes.
> On 9 Nov 2021, at 10:21, Ibrahim Altun <ibrahim.al...@segmentify.com> wrote: > > @ilya We did not force system to remove any keys, just add a new node to the > cluster, so my expectation was system to distribute ~120GB data across the > nodes, and expect to see ~80GB of data on each node (have total ~60GB(without > backup size) primary data, divided into 3 nodes ~40GB primary ~40GB backup) > > @Stephen what will distribution indicate? is there a way force cluster to > redistribute all data? > > On Tue, 9 Nov 2021 at 12:31, Stephen Darlington > <stephen.darling...@gridgain.com <mailto:stephen.darling...@gridgain.com>> > wrote: > You can check the distribution with the control script: > > ./control.sh —cache distribution null > > This displays the number of records per partition for all caches and nodes. > >> On 8 Nov 2021, at 20:24, Shishkov Ilya <shishkovi...@gmail.com >> <mailto:shishkovi...@gmail.com>> wrote: >> >> Hi! >> As I know, if you remove some cache entries (keys), their corresponding >> segments in the data pages remain empty in the persistent store until you >> insert these keys back into Ignite. So, it looks like voids in data pages. >> But the rebalanced data is put into persistent storage without any kind of >> "voids", i.e. it is already compressed during the rebalance routine. >> >> пн, 8 нояб. 2021 г. в 17:36, Ibrahim Altun <ibrahim.al...@segmentify.com >> <mailto:ibrahim.al...@segmentify.com>>: >> Hi, >> >> We had a 2 node cluster with persistence enabled with backup(1) enabled, >> this morning we've added a new node to the cluster. >> Although rebalancing is finished Ignite Persistence values are not evenly >> distributed; >> >> server-1: >> [2021-11-08T14:31:29,371][INFO ][grid-timeout-worker-#13][IgniteKernal] >> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >> ^-- Node [id=d9a3fb2f, uptime=01:35:00.491] >> ^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36, >> minorTopVer=3] >> ^-- Network [addrs=[10.156.0.112, 127.0.0.1], discoPort=47500, >> commPort=47100] >> ^-- CPU [CPUs=4, curLoad=0.33%, avgLoad=3.48%, GC=0%] >> ^-- Heap [used=5418MB, free=33.85%, comm=8192MB] >> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >> ^-- Page memory [pages=1524682] >> ^-- sysMemPlc region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, >> allocRam=100MB, allocTotal=0MB] >> ^-- default region [type=default, persistence=true, lazyAlloc=true, >> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >> allocRam=6144MB, allocTotal=80891MB] >> ^-- metastoreMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.81%, >> allocRam=0MB, allocTotal=0MB] >> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=100MB, allocTotal=0MB] >> ^-- volatileDsMemPlc region [type=user, persistence=false, >> lazyAlloc=true, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=0MB] >> ^-- Ignite persistence [used=80891MB] >> ^-- Outbound messages queue [size=0] >> ^-- Public thread pool [active=0, idle=0, qSize=0] >> ^-- System thread pool [active=0, idle=4, qSize=0] >> >> server-2: >> [2021-11-08T14:31:20,475][INFO ][grid-timeout-worker-#13][IgniteKernal] >> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >> ^-- Node [id=d001436d, uptime=00:46:00.231] >> ^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36, >> minorTopVer=3] >> ^-- Network [addrs=[10.156.0.113, 127.0.0.1], discoPort=47500, >> commPort=47100] >> ^-- CPU [CPUs=4, curLoad=4.43%, avgLoad=5.11%, GC=0%] >> ^-- Heap [used=6468MB, free=21.04%, comm=8192MB] >> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >> ^-- Page memory [pages=1524684] >> ^-- sysMemPlc region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, >> allocRam=100MB, allocTotal=0MB] >> ^-- default region [type=default, persistence=true, lazyAlloc=true, >> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >> allocRam=6144MB, allocTotal=82852MB] >> ^-- metastoreMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.8%, >> allocRam=0MB, allocTotal=0MB] >> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=100MB, allocTotal=0MB] >> ^-- volatileDsMemPlc region [type=user, persistence=false, >> lazyAlloc=true, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=0MB] >> ^-- Ignite persistence [used=82852MB] >> ^-- Outbound messages queue [size=0] >> ^-- Public thread pool [active=0, idle=0, qSize=0] >> ^-- System thread pool [active=0, idle=4, qSize=0] >> >> server-3: >> [2021-11-08T14:31:21,364][INFO ][grid-timeout-worker-#13][IgniteKernal] >> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >> ^-- Node [id=186395d1, uptime=03:36:01.279] >> ^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36, >> minorTopVer=3] >> ^-- Network [addrs=[10.156.0.10, 127.0.0.1], discoPort=47500, >> commPort=47100] >> ^-- CPU [CPUs=4, curLoad=0.4%, avgLoad=8.55%, GC=0%] >> ^-- Heap [used=6153MB, free=24.89%, comm=8192MB] >> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >> ^-- Page memory [pages=1524749] >> ^-- sysMemPlc region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.98%, >> allocRam=100MB, allocTotal=0MB] >> ^-- default region [type=default, persistence=true, lazyAlloc=true, >> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >> allocRam=6144MB, allocTotal=16164MB] >> ^-- metastoreMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.56%, >> allocRam=0MB, allocTotal=0MB] >> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=100MB, allocTotal=0MB] >> ^-- volatileDsMemPlc region [type=user, persistence=false, >> lazyAlloc=true, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=0MB] >> ^-- Ignite persistence [used=16165MB] >> ^-- Outbound messages queue [size=0] >> ^-- Public thread pool [active=0, idle=0, qSize=0] >> ^-- System thread pool [active=0, idle=4, qSize=0] >> >> Before new node added to cluster data was evenly distributed; >> >> server-1: >> [2021-11-08T06:00:52,779][INFO ][grid-timeout-worker-#13][IgniteKernal] >> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >> ^-- Node [id=cde7abcb, uptime=4 days, 03:38:05.557] >> ^-- Cluster [hosts=7, CPUs=28, servers=2, clients=7, topVer=17, >> minorTopVer=0] >> ^-- Network [addrs=[10.156.0.112, 127.0.0.1], discoPort=47500, >> commPort=47100] >> ^-- CPU [CPUs=4, curLoad=100%, avgLoad=5.78%, GC=117%] >> ^-- Heap [used=7669MB, free=6.37%, comm=8192MB] >> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >> ^-- Page memory [pages=1524669] >> ^-- sysMemPlc region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, >> allocRam=100MB, allocTotal=0MB] >> ^-- default region [type=default, persistence=true, lazyAlloc=true, >> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >> allocRam=6144MB, allocTotal=121061MB] >> ^-- metastoreMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.86%, >> allocRam=0MB, allocTotal=0MB] >> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=100MB, allocTotal=0MB] >> ^-- volatileDsMemPlc region [type=user, persistence=false, >> lazyAlloc=true, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=0MB] >> ^-- Ignite persistence [used=121061MB] >> ^-- Outbound messages queue [size=1] >> ^-- Public thread pool [active=0, idle=0, qSize=0] >> ^-- System thread pool [active=1, idle=3, qSize=1] >> >> server-2: >> [2021-11-08T06:00:37,491][INFO ][grid-timeout-worker-#13][IgniteKernal] >> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >> ^-- Node [id=90f15a32, uptime=4 days, 03:37:54.391] >> ^-- Cluster [hosts=7, CPUs=28, servers=2, clients=7, topVer=17, >> minorTopVer=0] >> ^-- Network [addrs=[10.156.0.113, 127.0.0.1], discoPort=47500, >> commPort=47100] >> ^-- CPU [CPUs=4, curLoad=2.6%, avgLoad=7.38%, GC=0%] >> ^-- Heap [used=7877MB, free=3.83%, comm=8192MB] >> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >> ^-- Page memory [pages=1524670] >> ^-- sysMemPlc region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, >> allocRam=100MB, allocTotal=0MB] >> ^-- default region [type=default, persistence=true, lazyAlloc=true, >> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >> allocRam=6144MB, allocTotal=121310MB] >> ^-- metastoreMemPlc region [type=internal, persistence=true, >> lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.86%, >> allocRam=0MB, allocTotal=0MB] >> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=100MB, allocTotal=0MB] >> ^-- volatileDsMemPlc region [type=user, persistence=false, >> lazyAlloc=true, >> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >> allocRam=0MB] >> ^-- Ignite persistence [used=121310MB] >> ^-- Outbound messages queue [size=0] >> ^-- Public thread pool [active=0, idle=0, qSize=0] >> ^-- System thread pool [active=0, idle=4, qSize=0] >> >> >> My expectation is data to be distributed evenly. >> >> What am I missing? >> >> Regardds. >> >> -- >> <https://www.segmentify.com/> >> İbrahim Halil Altun >> Senior Software Engineer >> >> +90 536 3327510 • segmentify.com → <https://www.segmentify.com/> >> UK • Germany • Turkey >> >> <https://www.segmentify.com/ecommerce-growth-show> >> <https://www.g2.com/products/segmentify/reviews> > > > > -- > <https://www.segmentify.com/> > İbrahim Halil Altun > Senior Software Engineer > > +90 536 3327510 • segmentify.com → <https://www.segmentify.com/> > UK • Germany • Turkey > > <https://www.segmentify.com/ecommerce-growth-show> > <https://www.g2.com/products/segmentify/reviews>
