Well, in theory yes. Each task has a thread, but only a number is run in parallel (the job of the scheduler). Parallelism is set in the environment. However, whereas the parallelism parameter is set and read correctly, when it comes to actual starting of the threads, the number is fix to 8. We run a debugger to get to the point where the thread was started. As Flavio mentioned, the ExecutionContext has the parallelims set to 8. We have a pool of connections to a RDBS and il logs the creation of just 8 connections although parallelism is much higher.
My question is whether this is a bug (or a feature) of the LocalMiniCluster. :-) I am not scala expert, but I see some variable assignment in setting up of the MiniCluster, involving parallelism and 'default values'. Default values in terms of parallelism are based on the number of cores. thanks a lot for the support! saluti, Stefano 2016-03-29 11:51 GMT+02:00 Ufuk Celebi <u...@apache.org>: > Hey Stefano, > > this should work by setting the parallelism on the environment, e.g. > > env.setParallelism(32) > > Is this what you are doing? > > The task threads are not part of a pool, but each submitted task > creates its own Thread. > > – Ufuk > > > On Fri, Mar 25, 2016 at 9:10 PM, Flavio Pompermaier > <pomperma...@okkam.it> wrote: > > Any help here? I think that the problem is that the JobManager creates > the > > executionContext of the scheduler with > > > > val executionContext = ExecutionContext.fromExecutor(new > > ForkJoinPool()) > > > > and thus the number of concurrently running threads is limited to the > number > > of cores (using the default constructor of the ForkJoinPool). > > What do you think? > > > > > > On Wed, Mar 23, 2016 at 6:55 PM, Stefano Bortoli <s.bort...@gmail.com> > > wrote: > >> > >> Hi guys, > >> > >> I am trying to test a job that should run a number of tasks to read > from a > >> RDBMS using an improved JDBC connector. The connection and the reading > run > >> smoothly, but I cannot seem to be able to move above the limit of 8 > >> concurrent threads running. 8 is of course the number of cores of my > >> machine. > >> > >> I have tried working around configurations and settings, but the > Executor > >> within the ExecutionContext keeps on having a parallelism of 8. > Although, of > >> course, the parallelism of the execution environment is much higher (in > fact > >> I have many more tasks to be allocated). > >> > >> I feel it may be an issue of the LocalMiniCluster configuration that may > >> just override/neglect my wish for higher degree of parallelism. Is > there a > >> way for me to work around this issue? > >> > >> please let me know. Thanks a lot for you help! :-) > >> > >> saluti, > >> Stefano > > > > > > >
