[
https://issues.apache.org/jira/browse/NIFI-4837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joseph Percivall updated NIFI-4837:
-----------------------------------
Description:
When you have multiple HandleHTTPRequest processors trying to listen on the
same port, for every Listen attempt NiFi builds a new thread and never recycles
the old thread which eventually leads to NiFi shutting down when reaching the
OS limit of the number of threads (default is 10.000).
The following error can be seen in nifi-app.log:
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
This has happened before with version 1.2 and probably even with older
versions. but I could also replicate the issue with the latest 1.5 version.
Steps to replicate the issue:
1) build a simple flow with 2 HandleHTTPRequest processors listening on the
same port.
!image-2018-02-02-11-14-51-964.png!
2) Start the processors.
— The second HandleHTTPRequest processor starts logging following as
expected:
2018-02-02 16:18:29,518 ERROR [Timer-Driven Process Thread-3]
o.a.n.p.standard.HandleHttpRequest
HandleHttpRequest[id=af013c62-b26f-1eeb-ae81-8423c70bdc7f] Failed to process
session due to org.apache.nifi.processor.exception.ProcessException: Failed to
initialize the server: {}
org.apache.nifi.processor.exception.ProcessException: Failed to initialize the
server
....
Caused by: java.net.BindException: Address already in use
...
... 12 common frames omitted
3) Go to the Summary section in NiFi and watch the number of threads going up
to 9959.
!image-2018-02-02-11-16-52-389.png!
With above, I had processors scheduled on primary node only as to not affect
every node.
If you stop the second HandleHTTPRequest processor the threads stop climbing,
but are not released.
After this, NiFi will soon stop.
A restart of NIFi is required to release these threads.
was:
When you have multiple HandleHTTPRequest processors trying to listen on the
same port, for every Listen attempt NiFi builds a new thread and never recycles
the old thread which eventually leads to NiFi shutting down when reaching the
OS limit of the number of threads (default is 10.000).
The following error can be seen in nifi-app.log:
Caused by: java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
This has happened before with HDF 3.0.1 and probably even with older versions.
but I could also replicate the issue with the latest HDF 3.1.
Steps to replicate the issue:
1) build a simple flow with 2 HandleHTTPRequest processors listening on the
same port.
!image-2018-02-02-11-14-51-964.png!
2) Start the processors.
— The second HandleHTTPRequest processor starts logging following as
expected:
2018-02-02 16:18:29,518 ERROR [Timer-Driven Process Thread-3]
o.a.n.p.standard.HandleHttpRequest
HandleHttpRequest[id=af013c62-b26f-1eeb-ae81-8423c70bdc7f] Failed to process
session due to org.apache.nifi.processor.exception.ProcessException: Failed to
initialize the server: {}
org.apache.nifi.processor.exception.ProcessException: Failed to initialize the
server
....
Caused by: java.net.BindException: Address already in use
...
... 12 common frames omitted
3) Go to the Summary section in NiFi and watch the number of threads going up
to 9959.
!image-2018-02-02-11-16-52-389.png!
With above, I had processors scheduled on primary node only as to not affect
every node.
If you stop the second HandleHTTPRequest processor the threads stop climbing,
but are not released.
After this, NiFi will soon stop.
A restart of NIFi is required to release these threads.
> Thread leak on HandleHTTPRequest processor
> ------------------------------------------
>
> Key: NIFI-4837
> URL: https://issues.apache.org/jira/browse/NIFI-4837
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0
> Environment: CENTOS 7
> Reporter: Matthew Clarke
> Priority: Blocker
> Attachments: image-2018-02-02-11-14-51-964.png,
> image-2018-02-02-11-16-52-389.png
>
>
> When you have multiple HandleHTTPRequest processors trying to listen on the
> same port, for every Listen attempt NiFi builds a new thread and never
> recycles the old thread which eventually leads to NiFi shutting down when
> reaching the OS limit of the number of threads (default is 10.000).
> The following error can be seen in nifi-app.log:
> Caused by: java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:714)
> This has happened before with version 1.2 and probably even with older
> versions. but I could also replicate the issue with the latest 1.5 version.
> Steps to replicate the issue:
> 1) build a simple flow with 2 HandleHTTPRequest processors listening on the
> same port.
> !image-2018-02-02-11-14-51-964.png!
> 2) Start the processors.
> — The second HandleHTTPRequest processor starts logging following as
> expected:
> 2018-02-02 16:18:29,518 ERROR [Timer-Driven Process Thread-3]
> o.a.n.p.standard.HandleHttpRequest
> HandleHttpRequest[id=af013c62-b26f-1eeb-ae81-8423c70bdc7f] Failed to process
> session due to org.apache.nifi.processor.exception.ProcessException: Failed
> to initialize the server: {}
> org.apache.nifi.processor.exception.ProcessException: Failed to initialize
> the server
> ....
> Caused by: java.net.BindException: Address already in use
> ...
> ... 12 common frames omitted
>
> 3) Go to the Summary section in NiFi and watch the number of threads going up
> to 9959.
> !image-2018-02-02-11-16-52-389.png!
>
> With above, I had processors scheduled on primary node only as to not affect
> every node.
> If you stop the second HandleHTTPRequest processor the threads stop climbing,
> but are not released.
>
> After this, NiFi will soon stop.
>
> A restart of NIFi is required to release these threads.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
