[
https://issues.apache.org/jira/browse/NIFI-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18118000#comment-18118000
]
WojciechWitos commented on NIFI-16358:
--------------------------------------
Attempt number 2 I've removed the processors which were inside process group
and it wasn't sufficient. After removing ~400 processors NiFi could load in
version 2.12.
Attempt number 3 I've removed other ~500 processors from other process group
and NiFi started. Cannot say which specific processor or the queues are
impacting how the cluster is working.
I've limited also RAM in bootstrap to 15 GB - even the limits were set, when
tried to migrate it to 2.12 it was going over 15 GB consuming all of the RAM to
last seen 1.20 GB free RAM - then the restart of NiFi started.
Here are some screenshots from the Heap usage from NiFi where all of the
processors were stopped, not processing anything, and ammount of the data on
the cluster was like that:
!flowfiles.png!
NiFi 2-11:
!Nifi_2-11.png!
NiFi 2-12:
!NiFi_2-12.png!
NiFi cluster heap from the 2-12:
!Nifi_2-12_cluster.png!
NiFi cluster heap from the 2-11:
!NiFi-2-11_clust.png!
Empty NiFi without running any services:
!Empty_Nifi.png!
> Unusual RAM consumption after the upgrade
> -----------------------------------------
>
> Key: NIFI-16358
> URL: https://issues.apache.org/jira/browse/NIFI-16358
> Project: Apache NiFi
> Issue Type: Bug
> Components: NiFi Stateless
> Affects Versions: 2.12.0
> Environment: 4x nodes,
> 6x CPU,
> 25 GB RAM on each node, 20 configured for NiFi in Boostrap
> OS Rhel 9
> Reporter: WojciechWitos
> Priority: Major
> Attachments: Empty_Nifi.png, NiFi-2-11_clust.png, NiFi_2-12.png,
> Nifi_2-11.png, Nifi_2-12_cluster.png, flowfiles.png
>
>
> After migration from 2.11 to 2.12 coordinator starts to consume huge amount
> of RAM during startup and crashes down and restarts the application.
>
> On version 2.11 NiFi starts normally and operates on the setup and on normal
> load:
> | |20 GB|20 GB|8.23 GB|41.0%|405.88 MB|390.99 MB| |00:05:37.279|
> | |20 GB|20 GB|6.14 GB|31.0%|378 MB|361.22 MB| |00:05:36.980|
> | |20 GB|20 GB|5.98 GB|30.0%|378 MB|361.36 MB| |00:05:37.553|
> | |20 GB|20 GB|6.96 GB|35.0%|380.94 MB|363.52 MB| |00:05:37.503|
>
> Coordinator works on the Core load avarage: 2.0-2.7, rest of the nodes
> operates on the 1.0-1.8 core load avarage.
>
> After loading flows on coordinator it crashes down with an error:
> {code:java}
> 2026-09-17 10:34:34,601 ERROR [main] org.apache.nifi.web.server.JettyServer
> Failed to start Serverjava.lang.OutOfMemoryError: unable to create native
> thread: possibly out of memory or process/resource limits reached at
> java.base/java.lang.Thread.start0(Native Method) at
> java.base/java.lang.Thread.start(Thread.java:1553) at
> java.base/java.lang.System$2.start(System.java:2578) at
> java.base/jdk.internal.vm.SharedThreadContainer.start(SharedThreadContainer.java:152)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:953)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1364)
> at
> org.apache.nifi.controller.queue.clustered.partition.StandardRebalancingPartition.submitRebalanceTask(StandardRebalancingPartition.java:225)
> at
> org.apache.nifi.controller.queue.clustered.partition.StandardRebalancingPartition.start(StandardRebalancingPartition.java:189)
> at
> org.apache.nifi.controller.queue.clustered.SocketLoadBalancedFlowFileQueue.startLoadBalancing(SocketLoadBalancedFlowFileQueue.java:320)
> at
> org.apache.nifi.controller.FlowController.onFlowInitialized(FlowController.java:1615)
> at
> org.apache.nifi.controller.StandardFlowService.loadFromConnectionResponse(StandardFlowService.java:967)
> at
> org.apache.nifi.controller.StandardFlowService.load(StandardFlowService.java:491)
> at org.apache.nifi.web.server.JettyServer.start(JettyServer.java:946)
> at
> org.apache.nifi.runtime.Application.startServer(Application.java:131)
> at org.apache.nifi.runtime.Application.run(Application.java:78) at
> org.apache.nifi.runtime.Application.run(Application.java:60) at
> org.apache.nifi.NiFi.main(NiFi.java:42)
> {code}
> Starts the application again, but other nodes do not change the coordinator -
> and it changes coordinator after some time, but situation is being repeat on
> other nodes also.
> On the cluster there are 6500 processors, where 2250 are disabled, 3200 are
> stopped.
> On the other cluster there are 3 nodes, 1050 processors active and 350
> processors in disabled state and the migration went fine and upgrade was
> successful.
>
> If something more is needed, let me know I will put requested details as fast
> as it's possible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
