Thank you, Yangze and Yang.
Turns out the high-availability.cluster-id parameter on the TM and JM were
different. After updating it, the issue went away.
On 17/01/20, 3:14 PM, "Yangze Guo" <karma...@gmail.com> wrote:
Hi, Harshith
As a supplementary note to Yang, the issue seems to be that something
went wrong when trying to connect to the ResourceManager.
There may be two possibilities, the leader of ResourceManager does not
write the znode or the TaskExecutor fails to connect to it. If you
turn on the DEBUG log, it will help a lot. Also, you could watch the
content znode "/leader/resource_manager_lock" of ZooKeeper.
Best,
Yangze Guo
On Fri, Jan 17, 2020 at 5:11 PM Yang Wang <danrtsey...@gmail.com> wrote:
>
> Hi Kumar Bolar, Harshith,
>
> Could you please check the jobmanager log to find out what address the
akka is listening?
> Also the address could be used to connected to the jobmanager on the
taskmanger machine.
>
> BTW, if you could share the debug level logs of jobmanger and taskmanger.
It will help a lot to find
> the root cause.
>
>
> Best,
> Yang
>
> Kumar Bolar, Harshith <hk...@arity.com> 于2020年1月16日周四 下午7:10写道：
>>
>> Hi all,
>>
>>
>>
>> We were previously using RHEL for our Flink machines. I'm currently
working on moving them over to Ubuntu. When I start the task manager, it fails
to connect to the job manager with the following message -
>>
>>
>>
>> 2020-01-16 10:54:42,777 INFO
org.apache.flink.runtime.util.LeaderRetrievalUtils - Trying to
select the network interface and address to use by connecting to the leading
JobManager.
>>
>> 2020-01-16 10:54:42,778 INFO
org.apache.flink.runtime.util.LeaderRetrievalUtils - TaskManager
will try to connect for 10000 milliseconds before falling back to heuristics
>>
>> 2020-01-16 10:54:52,780 WARN
org.apache.flink.runtime.net.ConnectionUtils - Could not find
any IPv4 address that is not loopback or link-local. Using localhost address.
>>
>>
>>
>> The network interface on the machine looks like this -
>>
>>
>>
>>
>>
>> ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 9001
>>
>> inet
https://urldefense.proofpoint.com/v2/url?u=http-3A__10.16.75.30&d=DwIFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=-xUFfpbE8LMFd_Z2bLTC60iRhyX6kRY17t4_2KSy_xs&s=53OA1njQf1iaaB9btpV8bgi7qYya9rwWK9DUUr9A580&e=
netmask
https://urldefense.proofpoint.com/v2/url?u=http-3A__255.255.255.128&d=DwIFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=-xUFfpbE8LMFd_Z2bLTC60iRhyX6kRY17t4_2KSy_xs&s=VAkjHsvEbPDirc3_I_ZhOOrimAfIYYsGUUEVaC1rhH8&e=
broadcast
https://urldefense.proofpoint.com/v2/url?u=http-3A__10.16.75.127&d=DwIFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=-xUFfpbE8LMFd_Z2bLTC60iRhyX6kRY17t4_2KSy_xs&s=hXAq7oqzjzyE4mGfBDaQQhn-LhoxC7tjLRTzYvNufxE&e=
>>
>> ether 02:f1:8b:34:75:51 txqueuelen 1000 (Ethernet)
>>
>> RX packets 69370 bytes 80369110 (80.3 MB)
>>
>> RX errors 0 dropped 0 overruns 0 frame 0
>>
>> TX packets 28787 bytes 2898540 (2.8 MB)
>>
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>>
>>
>> lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
>>
>> inet
https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1&d=DwIFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=-xUFfpbE8LMFd_Z2bLTC60iRhyX6kRY17t4_2KSy_xs&s=qzQ1Wkjhm6A1Y5WVp7oWupNN6xESCglqAofLiJnyXXg&e=
netmask
https://urldefense.proofpoint.com/v2/url?u=http-3A__255.0.0.0&d=DwIFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=-xUFfpbE8LMFd_Z2bLTC60iRhyX6kRY17t4_2KSy_xs&s=bdn272PCxu0YYSd6dH2BPxITorcdno5flP1nOB379ns&e=
>>
>> loop txqueuelen 1000 (Local Loopback)
>>
>> RX packets 9562 bytes 1596138 (1.5 MB)
>>
>> RX errors 0 dropped 0 overruns 0 frame 0
>>
>> TX packets 9562 bytes 1596138 (1.5 MB)
>>
>> TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
>>
>>
>>
>>
>>
>> Note: On RHEL, the primary network interface was eth0. Could this be the
issue?
>>
>>
>>
>> Here's the full task manager log -
https://urldefense.proofpoint.com/v2/url?u=https-3A__paste.ubuntu.com_p_vgh96FHzRq_&d=DwIFaQ&c=gtIjdLs6LnStUpy9cTOW9w&r=61bFb6zUNKZxlAQDRo_jKA&m=-xUFfpbE8LMFd_Z2bLTC60iRhyX6kRY17t4_2KSy_xs&s=xslfIsmeNZvi6en0rEcsuZ-0ODtAxVYaNgAJViglZDY&e=
>>
>>
>>
>> Thanks
>>
>> Harshith
