No problem ;-)
On Wed, Apr 13, 2016 at 11:38 AM, Stefano Bortoli
wrote:
> Sounds you are damn right! thanks for the insight, dumb on us for not
> checking this before.
>
> saluti,
> Stefano
>
> 2016-04-13 11:05 GMT+02:00 Stephan Ewen :
>
>> Sounds actually not like a Flink issue. I would look in
Sounds you are damn right! thanks for the insight, dumb on us for not
checking this before.
saluti,
Stefano
2016-04-13 11:05 GMT+02:00 Stephan Ewen :
> Sounds actually not like a Flink issue. I would look into the commons pool
> docs.
> Maybe they size their pools by default with the number of c
Sounds actually not like a Flink issue. I would look into the commons pool
docs.
Maybe they size their pools by default with the number of cores, so the
pool has only 8 threads, and other requests are queues?
On Wed, Apr 13, 2016 at 10:29 AM, Flavio Pompermaier
wrote:
> Any feedback about our JD
Any feedback about our JDBC InputFormat issue..?
On Thu, Apr 7, 2016 at 12:37 PM, Flavio Pompermaier
wrote:
> We've finally created a running example (For Flink 0.10.2) of our improved
> JDBC imputformat that you can run from an IDE (it creates an in-memory
> derby database with 1000 rows and ba
We've finally created a running example (For Flink 0.10.2) of our improved
JDBC imputformat that you can run from an IDE (it creates an in-memory
derby database with 1000 rows and batch of 10) at
https://gist.github.com/fpompermaier/bcd704abc93b25b6744ac76ac17ed351.
The first time you run the progr
Hi Ufuk,
here is our preliminary input formar implementation:
https://gist.github.com/anonymous/dbf05cad2a6cc07b8aa88e74a2c23119
if you need a running project, I will have to create a test one cause I
cannot share the current configuration.
thanks a lot in advance!
2016-03-30 10:13 GMT+02:00
Do you have the code somewhere online? Maybe someone can have a quick
look over it later. I'm pretty sure that is indeed a problem with the
custom input format.
– Ufuk
On Tue, Mar 29, 2016 at 3:50 PM, Stefano Bortoli wrote:
> Perhaps there is a misunderstanding on my side over the parallelism an
Perhaps there is a misunderstanding on my side over the parallelism and
split management given a data source.
We started from the current JDBCInputFormat to make it multi-thread. Then,
given a space of keys, we create the splits based on a fetchsize set as a
parameter. In the open, we get a connec
That is exactly my point. I should have 32 threads running, but I have only
8. 32 Task are created, but only only 8 are run concurrently. Flavio and I
will try to make a simple program to produce the problem. If we solve our
issues on the way, we'll let you know.
thanks a lot anyway.
saluti,
Stef
Then it shouldn’t be a problem. The ExeuctionContetxt is used to run
futures and their callbacks. But as Ufuk said, each task will spawn it’s
own thread and if you set the parallelism to 32 then you should have 32
threads running.
On Tue, Mar 29, 2016 at 12:29 PM, Stefano Bortoli
wrote:
> In f
In fact, I don't use it. I just had to crawl back the runtime
implementation to get to the point where parallelism was switching from 32
to 8.
saluti,
Stefano
2016-03-29 12:24 GMT+02:00 Till Rohrmann :
> Hi,
>
> for what do you use the ExecutionContext? That should actually be
> something which
Hi,
for what do you use the ExecutionContext? That should actually be something
which you shouldn’t be concerned with since it is only used internally by
the runtime.
Cheers,
Till
On Tue, Mar 29, 2016 at 12:09 PM, Stefano Bortoli
wrote:
> Well, in theory yes. Each task has a thread, but only
Well, in theory yes. Each task has a thread, but only a number is run in
parallel (the job of the scheduler). Parallelism is set in the
environment. However, whereas the parallelism parameter is set and read
correctly, when it comes to actual starting of the threads, the number is
fix to 8. We run
Hey Stefano,
this should work by setting the parallelism on the environment, e.g.
env.setParallelism(32)
Is this what you are doing?
The task threads are not part of a pool, but each submitted task
creates its own Thread.
– Ufuk
On Fri, Mar 25, 2016 at 9:10 PM, Flavio Pompermaier
wrote:
> A
Any help here? I think that the problem is that the JobManager creates the
executionContext of the scheduler with
val executionContext = ExecutionContext.fromExecutor(new
ForkJoinPool())
and thus the number of concurrently running threads is limited to the
number of cores (using the defaul
Hi guys,
I am trying to test a job that should run a number of tasks to read from a
RDBMS using an improved JDBC connector. The connection and the reading run
smoothly, but I cannot seem to be able to move above the limit of 8
concurrent threads running. 8 is of course the number of cores of my
ma
16 matches
Mail list logo
