Hey Stefano, this should work by setting the parallelism on the environment, e.g.
env.setParallelism(32) Is this what you are doing? The task threads are not part of a pool, but each submitted task creates its own Thread. – Ufuk On Fri, Mar 25, 2016 at 9:10 PM, Flavio Pompermaier <[email protected]> wrote: > Any help here? I think that the problem is that the JobManager creates the > executionContext of the scheduler with > > val executionContext = ExecutionContext.fromExecutor(new > ForkJoinPool()) > > and thus the number of concurrently running threads is limited to the number > of cores (using the default constructor of the ForkJoinPool). > What do you think? > > > On Wed, Mar 23, 2016 at 6:55 PM, Stefano Bortoli <[email protected]> > wrote: >> >> Hi guys, >> >> I am trying to test a job that should run a number of tasks to read from a >> RDBMS using an improved JDBC connector. The connection and the reading run >> smoothly, but I cannot seem to be able to move above the limit of 8 >> concurrent threads running. 8 is of course the number of cores of my >> machine. >> >> I have tried working around configurations and settings, but the Executor >> within the ExecutionContext keeps on having a parallelism of 8. Although, of >> course, the parallelism of the execution environment is much higher (in fact >> I have many more tasks to be allocated). >> >> I feel it may be an issue of the LocalMiniCluster configuration that may >> just override/neglect my wish for higher degree of parallelism. Is there a >> way for me to work around this issue? >> >> please let me know. Thanks a lot for you help! :-) >> >> saluti, >> Stefano > > >
