[
https://issues.apache.org/jira/browse/NIFI-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18116655#comment-18116655
]
WojciechWitos commented on NIFI-16358:
--------------------------------------
What I've noticed during the startup - even those stopped and disabled
processors are trying to be started in version 2.12 - which didn't occur on
version 2.11.
Also I'll have to review the flows - OOME might not be casued by the version
itself - a lot of concurrent tasks on the processors and autostart during
startup.
> Unusual RAM consumption after the upgrade
> -----------------------------------------
>
> Key: NIFI-16358
> URL: https://issues.apache.org/jira/browse/NIFI-16358
> Project: Apache NiFi
> Issue Type: Bug
> Components: NiFi Stateless
> Affects Versions: 2.12.0
> Environment: 4x nodes,
> 6x CPU,
> 25 GB RAM on each node, 20 configured for NiFi in Boostrap
> OS Rhel 9
> Reporter: WojciechWitos
> Priority: Major
>
> After migration from 2.11 to 2.12 coordinator starts to consume huge amount
> of RAM during startup and crashes down and restarts the application.
>
> On version 2.11 NiFi starts normally and operates on the setup and on normal
> load:
> | |20 GB|20 GB|8.23 GB|41.0%|405.88 MB|390.99 MB| |00:05:37.279|
> | |20 GB|20 GB|6.14 GB|31.0%|378 MB|361.22 MB| |00:05:36.980|
> | |20 GB|20 GB|5.98 GB|30.0%|378 MB|361.36 MB| |00:05:37.553|
> | |20 GB|20 GB|6.96 GB|35.0%|380.94 MB|363.52 MB| |00:05:37.503|
>
> Coordinator works on the Core load avarage: 2.0-2.7, rest of the nodes
> operates on the 1.0-1.8 core load avarage.
>
> After loading flows on coordinator it crashes down with an error:
> {code:java}
> 2026-09-17 10:34:34,601 ERROR [main] org.apache.nifi.web.server.JettyServer
> Failed to start Serverjava.lang.OutOfMemoryError: unable to create native
> thread: possibly out of memory or process/resource limits reached at
> java.base/java.lang.Thread.start0(Native Method) at
> java.base/java.lang.Thread.start(Thread.java:1553) at
> java.base/java.lang.System$2.start(System.java:2578) at
> java.base/jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:152)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
> at
> org.apache.nifi.controller.queue.clustered.partition.StandardRebalancingPartition.submitRebalanceTask(StandardRebalancingPartition.java:225)
> at
> org.apache.nifi.controller.queue.clustered.partition.StandardRebalancingPartition.start(StandardRebalancingPartition.java:189)
> at
> org.apache.nifi.controller.queue.clustered.SocketLoadBalancedFlowFileQueue.startLoadBalancing(SocketLoadBalancedFlowFileQueue.java:320)
> at
> org.apache.nifi.controller.FlowController.onFlowInitialized(FlowController.java:1615)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:967)
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:491)
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:946)
> at
> org.apache.nifi.runtime.Application.startServer(Application.java:131)
> at org.apache.nifi.runtime.Application.run(Application.java:78) at
> org.apache.nifi.runtime.Application.run(Application.java:60) at
> org.apache.nifi.NiFi.main(NiFi.java:42)
> {code}
> Starts the application again, but other nodes do not change the coordinator -
> and it changes coordinator after some time, but situation is being repeat on
> other nodes also.
> On the cluster there are 6500 processors, where 2250 are disabled, 3200 are
> stopped.
> On the other cluster there are 3 nodes, 1050 processors active and 350
> processors in disabled state and the migration went fine and upgrade was
> successful.
>
> If something more is needed, let me know I will put requested details as fast
> as it's possible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
