Hi Patrick "unique-dns-address"  is an alias for private IP. If XXX-XX-XX-XXX
is the private IP, then ip-XXX-XX-XX-XXX.aws-region-X is the
"unique-dns-address".
We were using auto scaling groups.

What worked out with the givens above was setting docker --net=host , and
then we saw "unique-dns-address" was set for the taskmanagers akka address.
Given we are ok with 1 taskmanager container per host, this all worked out.

Thanks,

Colin Williams



On Tue, Nov 21, 2017 at 6:14 AM, Patrick Lucas <[email protected]>
wrote:

> Hi Colin,
>
> Is each instance's "unique-dns-address" equal to the hostname of the
> instance or is the hostname something else? If it's different from the
> hostname, you're correct in assuming you need to configure each node to
> advertise its unique-dns-address intead.
>
> Are the unique-dns-addresses aliases for public or private IPs? I.e. in
> your example of a unique-dns-address do the X's map to the private IP of
> the instance or some public IP? If I recall correctly, in AWS (at least
> within a VPC), instance's public IPs are not actually bound to the instance
> itself and are more like a NAT/DMZ address, meaning you can't actually bind
> a port to them. This might work differently in EC2-Classic.
>
> If you ensure that each node advertises a bindable, resolvable name or IP
> address—with jobmanager.rpc.address on the jobmanager and
> taskmanager.hostname on the taskmanager—then they should all be able to
> discover, address, and communicate with each other with no problems.
>
> --
> Patrick Lucas
>
> On Tue, Nov 21, 2017 at 6:44 AM, Colin Williams <
> [email protected]> wrote:
>
>> Hi,
>>
>> We noticed that we couldn't parallelize our flink docker containers and
>> this looks like an issue that other have experienced. In our environment we
>> were not setting any hostname in the flink configuration. This worked for
>> the single node, but it looks like the taskmanagers would have the
>> exception also similar to others:
>>
>> org.apache.flink.runtime.io.network.partition.PartitionNotFoundException: 
>> Partition 2ec10eeac8969a16945b3713b63c0f4f@11052a989c7921423e44653285481e23 
>> not found.
>>>     at 
>>> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.failPartitionRequest(RemoteInputChannel.java:203)
>>>     at 
>>> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannel.retriggerSubpartitionRequest(RemoteInputChannel.java:128)
>>>     at 
>>> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGate.retriggerPartitionRequest(SingleInputGate.java:345)
>>>     at 
>>> org.apache.flink.runtime.taskmanager.Task.onPartitionStateUpdate(Task.java:1286)
>>>     at org.apache.flink.runtime.taskmanager.Task$2.apply(Task.java:1123)
>>>     at org.apache.flink.runtime.taskmanager.Task$2.apply(Task.java:1118)
>>>     at 
>>> org.apache.flink.runtime.concurrent.impl.FlinkFuture$5.onComplete(FlinkFuture.java:272)
>>>     at akka.dispatch.OnComplete.internal(Future.scala:248)
>>>     at akka.dispatch.OnComplete.internal(Future.scala:245)
>>>     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:175)
>>>     at akka.dispatch.japi$CallbackBridge.apply(Future.scala:172)
>>>     at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
>>>     at 
>>> akka.dispatch.BatchingExecutor$AbstractBatch.processBatch(BatchingExecutor.scala:55)
>>>     at 
>>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:91)
>>>     at 
>>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
>>>     at 
>>> akka.dispatch.BatchingExecutor$BlockableBatch$$anonfun$run$1.apply(BatchingExecutor.scala:91)
>>>     at 
>>> scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
>>>     at 
>>> akka.dispatch.BatchingExecutor$BlockableBatch.run(BatchingExecutor.scala:90)
>>>     at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
>>>     at 
>>> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
>>>     at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>>>     at 
>>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>>>     at 
>>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>>>     at 
>>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>>>
>>>
>> In our AWS environment we are only running one container per EC2
>> instance, and each instance has a "unique-dns-address" associated with it.
>> uniquie-dns-address is similar to ip-XXX-XX-XX-XXX.aws-region-X .
>>
>> Then so that we don't have to do any additional DNS configuration, it
>> would be convenient to exploit this dns address for each taskmanager to
>> talk to each other.
>>
>> I tested that I could reach each taskmanager from the unique-dns-address
>> via telnet to one of the taskmanager ports and I was able to connect. This
>> made me think that setting taskmanager.hostname to the address would solve
>> my issue.
>>
>> However when I tried to set taskmanager.hostname :
>> unique-dns-address in flink-conf.yaml I ended up with a 
>> java.net.BindException:
>> Cannot assign requested address.I'm not entirely sure why this happened.
>>
>> But I looked around and found some other list message that mentioned
>> https://doc.akka.io/docs/akka/2.4.1/additional/faq.html / RE: remote
>> actor.
>>
>> So I set *akka.remote.netty.tcp.hostname*: unique-dns-address again for
>> each instance. However I was not certain which taskmanager port to set for 
>> akka.remote.netty.tcp.port
>> . Then I left it unset but tried again I had the same
>> PartitionNotFoundException
>>
>> I realize this is a complicated issue which varies for each environment.
>> But I am asking for advice regarding other things I should try to tackle
>> the issue.
>>
>> Furhtermore, if I'm on the right track, what taskmanager service port
>> should correspond to akka.remote.netty.tcp.port ?
>>
>>
>>
>>
>

Reply via email to