Hi Flavio!
ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment()
by default picks up the number of cores as the parallelism, while the
manual environments do not do that.
You can still set it manually set the parallelism
"env.setParallelism(Runtime.getRuntime().availableProcessors());"
I would not configure the slots for the local execution, they should be
automatically configured based on the max parallelism.
Greetings,
Stephan
On Wed, Oct 14, 2015 at 3:36 PM, Flavio Pompermaier <pomperma...@okkam.it>
wrote:
> Hi Fabian and Stephan, back to work :)
>
> I finally managed to find the problem of the parallelism encountered by my
> colleague!
> Basically that was introduced by this API change. Before I was using
> env.setConfiguration() to merge the default params with some custom ones.
> Now, after the API change I was using the following code:
>
> ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
> if (env instanceof LocalEnvironment) {
> Configuration c = new Configuration();
> c.setString(ConfigConstants.TASK_MANAGER_TMP_DIR_KEY, FLINK_TEST_TMP_DIR);
> c.setString(ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY,FLINK_TEST_TMP_DIR);
> c.setLong(ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY, 2048 * 2);
> c.setLong(ConfigConstants.TASK_MANAGER_NUM_TASK_SLOTS, 4);
> env = ExecutionEnvironment.createLocalEnvironment(c);
> }
>
> However, the first env and the reassigned one doesn't behave in the same
> manner.
> If I don't reassign env I have parallelism=8, otherwise it's 1 :(
> Am I using the wrong APIs or the execution environment doesn't allow now
> to configure such parameters anymore?
>
> Thanks in advance,
> Flavio
>
>
> On Tue, Oct 6, 2015 at 11:31 AM, Flavio Pompermaier <pomperma...@okkam.it>
> wrote:
>
>> That makes sense: what can be configured should be differentiated between
>> local and remote envs (obviously this is a minor issue/improvement)
>>
>> Thanks again,
>> Flavio
>>
>> On Tue, Oct 6, 2015 at 11:25 AM, Stephan Ewen <se...@apache.org> wrote:
>>
>>> We can think about that, but I think it may be quite confusing. The
>>> configurations actually mean something different for local and remote
>>> environments:
>>>
>>> - For the local environment, the config basically describes the entire
>>> Flink cluster setup (for the local execution cluster in the background)
>>> - For the remote environment, the config describes the parameters for
>>> the client that connects to the cluster (akka paramters, optimizer
>>> parameters, ...), but not parameters of the cluster itself (like
>>> taskmanager slots and memory).
>>>
>>> Greetings,
>>> Stephan
>>>
>>>
>>> On Tue, Oct 6, 2015 at 10:56 AM, Flavio Pompermaier <
>>> pomperma...@okkam.it> wrote:
>>>
>>>> However it could be a good idea to overload also
>>>> the getExecutionEnvironment() to be able to pass a custom
>>>> configuration..what do you think?
>>>> Otherwise I have to know a priori if I'm working in a local deployment
>>>> or in a remote one, or check if getExecutionEnvironment() returned an
>>>> instance of LocalEnvironment/RemoteEnvironment..
>>>>
>>>>
>>>>
>>>> On Tue, Oct 6, 2015 at 10:53 AM, Flavio Pompermaier <
>>>> pomperma...@okkam.it> wrote:
>>>>
>>>>> Yes Stephan!
>>>>> I usually work with the master version, at least in development ;)
>>>>> Thanks for the quick support!
>>>>>
>>>>> Best,
>>>>> Flavio
>>>>>
>>>>> On Tue, Oct 6, 2015 at 10:48 AM, Stephan Ewen <se...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Hi!
>>>>>>
>>>>>> Are you on the SNAPSHOT master version?
>>>>>>
>>>>>> You can pass the configuration to the constructor of the execution
>>>>>> environment, or create one via
>>>>>> ExecutionEnvironment.createLocalEnvironment(config) or via
>>>>>> createRemoteEnvironment(host, port, configuration, jarFiles);
>>>>>>
>>>>>> The change of the signature was part of an API cleanup for the next
>>>>>> release. Sorry for the inconvenience...
>>>>>>
>>>>>> Stephan
>>>>>>
>>>>>>
>>>>>> On Tue, Oct 6, 2015 at 10:36 AM, Flavio Pompermaier <
>>>>>> pomperma...@okkam.it> wrote:
>>>>>>
>>>>>>> Hi to all,
>>>>>>>
>>>>>>> today my code doesn't compile anymore because ExecutionEnvironment
>>>>>>> doesn't have setConfiguration() anymore..how can I set the following
>>>>>>> parameters in my unit tests?
>>>>>>>
>>>>>>> - ConfigConstants.TASK_MANAGER_TMP_DIR_KEY
>>>>>>> - ConfigConstants.BLOB_STORAGE_DIRECTORY_KEY
>>>>>>> - ConfigConstants.TASK_MANAGER_NETWORK_NUM_BUFFERS_KEY
>>>>>>>
>>>>>>> Best,
>>>>>>> Flavio
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
