Ignite exports lots of other metrics: https://ignite.apache.org/docs/latest/monitoring-metrics/intro
You need to look at the number of pages used and the fill rate (how full each page is on average) to figure out how much space you’re actually using. > On 9 Nov 2021, at 11:22, Ibrahim Altun <ibrahim.al...@segmentify.com> wrote: > > Looking at the size of the persistence files isn’t necessarily a good metric, > as Ilya notes. -> but still files physically occupy disk space, in this case > i don't know if i need to increase disk space or not. > > On Tue, 9 Nov 2021 at 14:17, Stephen Darlington > <stephen.darling...@gridgain.com <mailto:stephen.darling...@gridgain.com>> > wrote: > It shows how the data is distributed. > > Looking at the size of the persistence files isn’t necessarily a good metric, > as Ilya notes. > >> On 9 Nov 2021, at 10:21, Ibrahim Altun <ibrahim.al...@segmentify.com >> <mailto:ibrahim.al...@segmentify.com>> wrote: >> >> @ilya We did not force system to remove any keys, just add a new node to the >> cluster, so my expectation was system to distribute ~120GB data across the >> nodes, and expect to see ~80GB of data on each node (have total >> ~60GB(without backup size) primary data, divided into 3 nodes ~40GB primary >> ~40GB backup) >> >> @Stephen what will distribution indicate? is there a way force cluster to >> redistribute all data? >> >> On Tue, 9 Nov 2021 at 12:31, Stephen Darlington >> <stephen.darling...@gridgain.com <mailto:stephen.darling...@gridgain.com>> >> wrote: >> You can check the distribution with the control script: >> >> ./control.sh —cache distribution null >> >> This displays the number of records per partition for all caches and nodes. >> >>> On 8 Nov 2021, at 20:24, Shishkov Ilya <shishkovi...@gmail.com >>> <mailto:shishkovi...@gmail.com>> wrote: >>> >>> Hi! >>> As I know, if you remove some cache entries (keys), their corresponding >>> segments in the data pages remain empty in the persistent store until you >>> insert these keys back into Ignite. So, it looks like voids in data pages. >>> But the rebalanced data is put into persistent storage without any kind of >>> "voids", i.e. it is already compressed during the rebalance routine. >>> >>> пн, 8 нояб. 2021 г. в 17:36, Ibrahim Altun <ibrahim.al...@segmentify.com >>> <mailto:ibrahim.al...@segmentify.com>>: >>> Hi, >>> >>> We had a 2 node cluster with persistence enabled with backup(1) enabled, >>> this morning we've added a new node to the cluster. >>> Although rebalancing is finished Ignite Persistence values are not evenly >>> distributed; >>> >>> server-1: >>> [2021-11-08T14:31:29,371][INFO ][grid-timeout-worker-#13][IgniteKernal] >>> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >>> ^-- Node [id=d9a3fb2f, uptime=01:35:00.491] >>> ^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36, >>> minorTopVer=3] >>> ^-- Network [addrs=[10.156.0.112, 127.0.0.1], discoPort=47500, >>> commPort=47100] >>> ^-- CPU [CPUs=4, curLoad=0.33%, avgLoad=3.48%, GC=0%] >>> ^-- Heap [used=5418MB, free=33.85%, comm=8192MB] >>> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >>> ^-- Page memory [pages=1524682] >>> ^-- sysMemPlc region [type=internal, persistence=true, >>> lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, >>> allocRam=100MB, allocTotal=0MB] >>> ^-- default region [type=default, persistence=true, lazyAlloc=true, >>> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >>> allocRam=6144MB, allocTotal=80891MB] >>> ^-- metastoreMemPlc region [type=internal, persistence=true, >>> lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.81%, >>> allocRam=0MB, allocTotal=0MB] >>> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >>> allocRam=100MB, allocTotal=0MB] >>> ^-- volatileDsMemPlc region [type=user, persistence=false, >>> lazyAlloc=true, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >>> allocRam=0MB] >>> ^-- Ignite persistence [used=80891MB] >>> ^-- Outbound messages queue [size=0] >>> ^-- Public thread pool [active=0, idle=0, qSize=0] >>> ^-- System thread pool [active=0, idle=4, qSize=0] >>> >>> server-2: >>> [2021-11-08T14:31:20,475][INFO ][grid-timeout-worker-#13][IgniteKernal] >>> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >>> ^-- Node [id=d001436d, uptime=00:46:00.231] >>> ^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36, >>> minorTopVer=3] >>> ^-- Network [addrs=[10.156.0.113, 127.0.0.1], discoPort=47500, >>> commPort=47100] >>> ^-- CPU [CPUs=4, curLoad=4.43%, avgLoad=5.11%, GC=0%] >>> ^-- Heap [used=6468MB, free=21.04%, comm=8192MB] >>> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >>> ^-- Page memory [pages=1524684] >>> ^-- sysMemPlc region [type=internal, persistence=true, >>> lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, >>> allocRam=100MB, allocTotal=0MB] >>> ^-- default region [type=default, persistence=true, lazyAlloc=true, >>> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >>> allocRam=6144MB, allocTotal=82852MB] >>> ^-- metastoreMemPlc region [type=internal, persistence=true, >>> lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.8%, >>> allocRam=0MB, allocTotal=0MB] >>> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >>> allocRam=100MB, allocTotal=0MB] >>> ^-- volatileDsMemPlc region [type=user, persistence=false, >>> lazyAlloc=true, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >>> allocRam=0MB] >>> ^-- Ignite persistence [used=82852MB] >>> ^-- Outbound messages queue [size=0] >>> ^-- Public thread pool [active=0, idle=0, qSize=0] >>> ^-- System thread pool [active=0, idle=4, qSize=0] >>> >>> server-3: >>> [2021-11-08T14:31:21,364][INFO ][grid-timeout-worker-#13][IgniteKernal] >>> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >>> ^-- Node [id=186395d1, uptime=03:36:01.279] >>> ^-- Cluster [hosts=8, CPUs=32, servers=3, clients=7, topVer=36, >>> minorTopVer=3] >>> ^-- Network [addrs=[10.156.0.10, 127.0.0.1], discoPort=47500, >>> commPort=47100] >>> ^-- CPU [CPUs=4, curLoad=0.4%, avgLoad=8.55%, GC=0%] >>> ^-- Heap [used=6153MB, free=24.89%, comm=8192MB] >>> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >>> ^-- Page memory [pages=1524749] >>> ^-- sysMemPlc region [type=internal, persistence=true, >>> lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.98%, >>> allocRam=100MB, allocTotal=0MB] >>> ^-- default region [type=default, persistence=true, lazyAlloc=true, >>> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >>> allocRam=6144MB, allocTotal=16164MB] >>> ^-- metastoreMemPlc region [type=internal, persistence=true, >>> lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.56%, >>> allocRam=0MB, allocTotal=0MB] >>> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >>> allocRam=100MB, allocTotal=0MB] >>> ^-- volatileDsMemPlc region [type=user, persistence=false, >>> lazyAlloc=true, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >>> allocRam=0MB] >>> ^-- Ignite persistence [used=16165MB] >>> ^-- Outbound messages queue [size=0] >>> ^-- Public thread pool [active=0, idle=0, qSize=0] >>> ^-- System thread pool [active=0, idle=4, qSize=0] >>> >>> Before new node added to cluster data was evenly distributed; >>> >>> server-1: >>> [2021-11-08T06:00:52,779][INFO ][grid-timeout-worker-#13][IgniteKernal] >>> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >>> ^-- Node [id=cde7abcb, uptime=4 days, 03:38:05.557] >>> ^-- Cluster [hosts=7, CPUs=28, servers=2, clients=7, topVer=17, >>> minorTopVer=0] >>> ^-- Network [addrs=[10.156.0.112, 127.0.0.1], discoPort=47500, >>> commPort=47100] >>> ^-- CPU [CPUs=4, curLoad=100%, avgLoad=5.78%, GC=117%] >>> ^-- Heap [used=7669MB, free=6.37%, comm=8192MB] >>> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >>> ^-- Page memory [pages=1524669] >>> ^-- sysMemPlc region [type=internal, persistence=true, >>> lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, >>> allocRam=100MB, allocTotal=0MB] >>> ^-- default region [type=default, persistence=true, lazyAlloc=true, >>> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >>> allocRam=6144MB, allocTotal=121061MB] >>> ^-- metastoreMemPlc region [type=internal, persistence=true, >>> lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.86%, >>> allocRam=0MB, allocTotal=0MB] >>> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >>> allocRam=100MB, allocTotal=0MB] >>> ^-- volatileDsMemPlc region [type=user, persistence=false, >>> lazyAlloc=true, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >>> allocRam=0MB] >>> ^-- Ignite persistence [used=121061MB] >>> ^-- Outbound messages queue [size=1] >>> ^-- Public thread pool [active=0, idle=0, qSize=0] >>> ^-- System thread pool [active=1, idle=3, qSize=1] >>> >>> server-2: >>> [2021-11-08T06:00:37,491][INFO ][grid-timeout-worker-#13][IgniteKernal] >>> Metrics for local node (to disable set 'metricsLogFrequency' to 0) >>> ^-- Node [id=90f15a32, uptime=4 days, 03:37:54.391] >>> ^-- Cluster [hosts=7, CPUs=28, servers=2, clients=7, topVer=17, >>> minorTopVer=0] >>> ^-- Network [addrs=[10.156.0.113, 127.0.0.1], discoPort=47500, >>> commPort=47100] >>> ^-- CPU [CPUs=4, curLoad=2.6%, avgLoad=7.38%, GC=0%] >>> ^-- Heap [used=7877MB, free=3.83%, comm=8192MB] >>> ^-- Off-heap memory [used=6025MB, free=7.92%, allocated=6344MB] >>> ^-- Page memory [pages=1524670] >>> ^-- sysMemPlc region [type=internal, persistence=true, >>> lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.99%, >>> allocRam=100MB, allocTotal=0MB] >>> ^-- default region [type=default, persistence=true, lazyAlloc=true, >>> ... initCfg=256MB, maxCfg=6144MB, usedRam=6025MB, freeRam=1.93%, >>> allocRam=6144MB, allocTotal=121310MB] >>> ^-- metastoreMemPlc region [type=internal, persistence=true, >>> lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=99.86%, >>> allocRam=0MB, allocTotal=0MB] >>> ^-- TxLog region [type=internal, persistence=true, lazyAlloc=false, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >>> allocRam=100MB, allocTotal=0MB] >>> ^-- volatileDsMemPlc region [type=user, persistence=false, >>> lazyAlloc=true, >>> ... initCfg=40MB, maxCfg=100MB, usedRam=0MB, freeRam=100%, >>> allocRam=0MB] >>> ^-- Ignite persistence [used=121310MB] >>> ^-- Outbound messages queue [size=0] >>> ^-- Public thread pool [active=0, idle=0, qSize=0] >>> ^-- System thread pool [active=0, idle=4, qSize=0] >>> >>> >>> My expectation is data to be distributed evenly. >>> >>> What am I missing? >>> >>> Regardds. >>> >>> -- >>> <https://www.segmentify.com/> >>> İbrahim Halil Altun >>> Senior Software Engineer >>> >>> +90 536 3327510 • segmentify.com → <https://www.segmentify.com/> >>> UK • Germany • Turkey >>> >>> <https://www.segmentify.com/ecommerce-growth-show> >>> <https://www.g2.com/products/segmentify/reviews> >> >> >> >> -- >> <https://www.segmentify.com/> >> İbrahim Halil Altun >> Senior Software Engineer >> >> +90 536 3327510 • segmentify.com → <https://www.segmentify.com/> >> UK • Germany • Turkey >> >> <https://www.segmentify.com/ecommerce-growth-show> >> <https://www.g2.com/products/segmentify/reviews> > > > > -- > <https://www.segmentify.com/> > İbrahim Halil Altun > Senior Software Engineer > > +90 536 3327510 • segmentify.com → <https://www.segmentify.com/> > UK • Germany • Turkey > > <https://www.segmentify.com/ecommerce-growth-show> > <https://www.g2.com/products/segmentify/reviews>
