[
https://issues.apache.org/jira/browse/NIFI-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18117717#comment-18117717
]
WojciechWitos commented on NIFI-16358:
--------------------------------------
After removing around 500 of processors from one process group NiFi started and
operates on the same values as mentioned in the main post.
Tomorrow I'll check if some specific flow causes the issues and will try to
import those to version 2.12 and restart the cluster.
First which comes to my mind is issue with too many "round robin" queues - even
if those are empty.
> Unusual RAM consumption after the upgrade
> -----------------------------------------
>
> Key: NIFI-16358
> URL: https://issues.apache.org/jira/browse/NIFI-16358
> Project: Apache NiFi
> Issue Type: Bug
> Components: NiFi Stateless
> Affects Versions: 2.12.0
> Environment: 4x nodes,
> 6x CPU,
> 25 GB RAM on each node, 20 configured for NiFi in Boostrap
> OS Rhel 9
> Reporter: WojciechWitos
> Priority: Major
>
> After migration from 2.11 to 2.12 coordinator starts to consume huge amount
> of RAM during startup and crashes down and restarts the application.
>
> On version 2.11 NiFi starts normally and operates on the setup and on normal
> load:
> | |20 GB|20 GB|8.23 GB|41.0%|405.88 MB|390.99 MB| |00:05:37.279|
> | |20 GB|20 GB|6.14 GB|31.0%|378 MB|361.22 MB| |00:05:36.980|
> | |20 GB|20 GB|5.98 GB|30.0%|378 MB|361.36 MB| |00:05:37.553|
> | |20 GB|20 GB|6.96 GB|35.0%|380.94 MB|363.52 MB| |00:05:37.503|
>
> Coordinator works on the Core load avarage: 2.0-2.7, rest of the nodes
> operates on the 1.0-1.8 core load avarage.
>
> After loading flows on coordinator it crashes down with an error:
> {code:java}
> 2026-09-17 10:34:34,601 ERROR [main] org.apache.nifi.web.server.JettyServer
> Failed to start Serverjava.lang.OutOfMemoryError: unable to create native
> thread: possibly out of memory or process/resource limits reached at
> java.base/java.lang.Thread.start0(Native Method) at
> java.base/java.lang.Thread.start(Thread.java:1553) at
> java.base/java.lang.System$2.start(System.java:2578) at
> java.base/jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:152)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
> at
> org.apache.nifi.controller.queue.clustered.partition.StandardRebalancingPartition.submitRebalanceTask(StandardRebalancingPartition.java:225)
> at
> org.apache.nifi.controller.queue.clustered.partition.StandardRebalancingPartition.start(StandardRebalancingPartition.java:189)
> at
> org.apache.nifi.controller.queue.clustered.SocketLoadBalancedFlowFileQueue.startLoadBalancing(SocketLoadBalancedFlowFileQueue.java:320)
> at
> org.apache.nifi.controller.FlowController.onFlowInitialized(FlowController.java:1615)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:967)
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:491)
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:946)
> at
> org.apache.nifi.runtime.Application.startServer(Application.java:131)
> at org.apache.nifi.runtime.Application.run(Application.java:78) at
> org.apache.nifi.runtime.Application.run(Application.java:60) at
> org.apache.nifi.NiFi.main(NiFi.java:42)
> {code}
> Starts the application again, but other nodes do not change the coordinator -
> and it changes coordinator after some time, but situation is being repeat on
> other nodes also.
> On the cluster there are 6500 processors, where 2250 are disabled, 3200 are
> stopped.
> On the other cluster there are 3 nodes, 1050 processors active and 350
> processors in disabled state and the migration went fine and upgrade was
> successful.
>
> If something more is needed, let me know I will put requested details as fast
> as it's possible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
