Aljoscha,
You're correct that I'm using Beam. Here's the shell script I'm using to start
the job:
hadoop jar **path to jar file** org.apache.beam.examples.complete.TfIdf
\--runner=FlinkRunner \--input= **path to directory** \--output=tfIdf
There are two lines before that to set the Flink config path, as well as the
Hadoop classpath.
Thank you,Will.
On Tuesday, June 6, 2017 5:45 AM, Aljoscha Krettek <aljos...@apache.org>
wrote:
Hi Will,
How are you starting your cluster/executing your program? From what I can
gather you are using Beam, is that right? The line about the FlinkMiniCluster
seems strange because this would hint at the fact that the Runner is trying to
execute in local mode.
Best,Aljoscha
On 6. Jun 2017, at 02:13, Will Walters <wwalt...@yahoo-inc.com> wrote:
Hello,
I'm having issues editing the default Flink memory settings. I'm attempting to
run a Flink task on a cluster at scale. The log shows my edited config settings
having been read into the program, but they're having no effect. Here's the
trace:
17/06/05 23:45:41 INFO flink.FlinkRunner: Starting execution of Flink
program.17/06/05 23:45:41 INFO java.ExecutionEnvironment: The job has 0
registered types and 0 default Kryo serializers17/06/05 23:45:41 INFO
configuration.GlobalConfiguration: Loading configuration property:
taskmanager.memory.size, 10017/06/05 23:45:41 INFO
configuration.GlobalConfiguration: Loading configuration property:
taskmanager.heap.mb, 25617/06/05 23:45:41 INFO
configuration.GlobalConfiguration: Loading configuration property:
taskmanager.debug.memory.startLogThread, true17/06/05 23:45:41 INFO
minicluster.FlinkMiniCluster: Disabled queryable state server17/06/05 23:45:41
INFO minicluster.FlinkMiniCluster: Starting FlinkMiniCluster....17/06/05
23:45:41 INFO network.NetworkEnvironment: Starting the network environment and
its components.17/06/05 23:45:41 INFO taskmanager.TaskManager: Limiting managed
memory to 17592186044406 MB, memory will be allocated lazily.17/06/05 23:45:41
ERROR flink.FlinkRunner: Pipeline execution
failedjava.lang.IllegalArgumentException: Size of total memory must be positive.
And here's the config file I'm using:
taskmanager.memory.size: 100taskmanager.heap.mb:
256taskmanager.debug.memory.startLogThread: true
In the shell script I'm using to run the task, I've edited the FLINK_CONF_DIR
to direct to the config file that I created.
If anyone has any advice or inputs it would be much appreciated.
Thanks!Will Walters.
