Node permanently in maintenance mode

2021-07-30 Thread Piotr Jagielski
Hi, We recently switched our cluster to 2.10. We had problems with one of the caches before the upgrade, and deleted some data. After the upgrade Ignite has run for 1-2 day and today one of the nodes stated in "Maintenance mode" after the restart: 2021-07-30 12:59:41 WARN Maintenance task found

Re: Node permanently in maintenance mode

2021-07-30 Thread Piotr Jagielski
Ok I managed to start the node by deleting maintenance_tasks.mntc file and corrupted cache from node storage directory On 2021/07/30 11:09:01, Piotr Jagielski wrote: > Hi, > We recently switched our cluster to 2.10. > We had problems with one of the caches before the upgrade, and del

Re: apache ignite 2.10.0 heap starvation

2021-09-07 Thread Piotr Jagielski
Hi, Can you provide some information on how you use the cluster? How many reads/writes/updates per second? Also CPU / RAM spec of cluster nodes? We observed full GC / CPU load / OOM killer when loading big amount of data (15 mln records, data streamer + allowOverwrite=true). We've seen 200-400k

Ignite cluster stability problems under heavy load

2021-10-06 Thread Piotr Jagielski
Hi, We experience stability problems on our Ignite cluster (2.10) under heavy load. Our cluster nodes are 3x 8 CPU, 32GB RAM. We mainly use 2 persistent caches: - aggregates - only updates, around 6K records / sec, ~70 mln records total, stored mostly on disk (dataRegion maxSize = 4GB) - custo

Re: Ignite cluster stability problems under heavy load

2021-10-07 Thread Piotr Jagielski
;? Is there any growing metric exposed via JMX to check? Regarding HDD I managed to run hdparm: /dev/sda1:  Timing cached reads:   15036 MB in  2.00 seconds = 7525.21 MB/sec  Timing buffered disk reads: 2664 MB in  3.00 seconds = 887.36 MB/sec Regards, Piotr On 2021/10/06 12:26:07, Piotr Jagielsk

Re: Ignite cluster stability problems under heavy load

2021-10-18 Thread Piotr Jagielski
der 20 millis. So, is the throttling the thing that helped? Is our disk too slow? W dniu 08.10.2021 o 08:44, Piotr Jagielski pisze: Hi again, Any advice? We're really struggling with our cluster stability - we had to turn on throttling before sending data to DataStreamer, but proble