Hi Averell,
Flink compares the number of user selected vcores to the vcores configured
in
the yarn-site.xml of the submitting node, i.e., in your case the master
node.
If there are not enough configured vcores, the client throws an exception.
This behavior is not ideal and I found an old JIRA ticket for it [1]. We
could
either remove this check, or – as the original ticket suggests – reuse the
logic from "yarn-session.sh -q" to determine if there is enough capacity in
the cluster.
As a workaround, you can set in the yarn-site.xml
yarn.nodemanager.resource.cpu-vcores
to 16, or alternatively run multiple smaller TaskManagers on each node [2].
Best,
Gary
[1] https://issues.apache.org/jira/browse/FLINK-5542
[2]
https://github.com/apache/flink/blob/09abba37c7d760236c2ba002fa4a3aac11c2641b/flink-yarn/src/main/java/org/apache/flink/yarn/AbstractYarnClusterDescriptor.java#L288
On Tue, Sep 18, 2018 at 4:43 AM, Averell <lvhu...@gmail.com> wrote:
> Thank you Gary.
>
> Regarding the option to use a smaller server for the master node, when
> starting a flink job, I would get an error like the following;
>
> /Caused by: org.apache.flink.configuration.IllegalConfigurationException:
> *The number of virtual cores per node were configured with 16 but Yarn only
> has 4 virtual cores available*. Please note that the number of virtual
> cores
> is set to the number of task slots by default unless configured in the
> Flink
> config with 'yarn.containers.vcores.'/
>
> To get around that error, I need to start the job from one of the core
> node.
> Should that be an expected behaviour?
>
> Thanks and regards,
> Averell
>
>
>
> --
> Sent from: http://apache-flink-user-mailing-list-archive.2336050.
> n4.nabble.com/
>
