Re: Programmatic way to determine required number of slots for a job?

I found in the “Flip6LocalStreamEnvironment” the following code, but with the caveat that it doesn’t include slot reuse? int slotsCount = 0; for (org.apache.flink.runtime.jobgraph.JobVertex jobVertex : flink.getStreamGraph().getJobGraph().getVertices()) { slotsCount += jobVertex.getParal

Programmatic way to determine required number of slots for a job?

Is there a way to determine via an API the required number of task manager slots for a particular job graph? I’ve looked at "flink.getStreamGraph().getJobGraph().getNumberOfVertices()” but it doesn’t seem to correspond with what the job actually ends up using. -- Jared Stehler Chief Architect -

Re: Data sources and slices

Hi David, in case of a streaming program with a degree of parallelism of 128, Flink would create 128 splits. One split for each parallel sub task. The logic is that one split will have the size of one block unless this would not give you enough splits such that every task receives at least one spl

Re: PrometheusReporter error

Hi, Two Jira issues are mentioned in this thread: - [FLINK-7069] Catch exceptions for each reporter separately : Fixed - [FLINK-7100] TaskManager metrics are registered twice : Unresolved I guess your error message is just an warning message. You can safely ignore it if you are using 1.3.2

Re: Checkpoint was declined (tasks not ready)

Yes please open the PR against Flink's master branch. You can also ping me once you've opened the PR. Then we can hopefully quickly merge it :-) Cheers, Till On Thu, Oct 26, 2017 at 12:44 PM, bartektartanus wrote: > I think we could try with option number one, as it seems to be easier to > impl

Re: PrometheusReporter error

Hi, there is PR open for fixing the multiple TM metrics registration [1]. We hope to get it merged in the next days such that it will be part of the upcoming 1.4 release. [1] https://github.com/apache/flink/pull/4872 Cheers, Till On Thu, Oct 26, 2017 at 5:21 PM, cslotterback wrote: > Hello 김동

Re: Execute multiple jobs in parallel (threading): java.io.OptionalDataException

Hi David, I cannot exactly tell how you ended up seeing an OptionalDataException without seeing your code. Flink supports to run multiple jobs on the same cluster. That’s what we call the session mode. You should not reuse the ExecutionEnvironment because then, you will create a single job which

Re: Capacity Planning For Large State in YARN Cluster

Hi Ashish, what you are describing should be a good use case for Flink and it should be able to run your program. When you are seeing a GC overhead limit exceeded error, then it means that Flink or your program are creating too many/too large objects filling up the memory in a short time. I would

Re: StreamTransformation object

Hi Andrea, I don't think you need to touch `StreamTransformation`. If you can get the result from build(), you can do the same thing I mentioned above: casting it as SingleOutputStreamOperator. Then, you can pass it to select function to add the next operator, and get the result to add another slo

Re: StreamTransformation object

I'm trying to create an API to get results from build() but anyway I'm stuck on the StreamTransformation which is essential to implement slotSharingGroup(...). I have to provide it from the main class. Tony Wei wrote > Hi Andrea, > > The `learn` operator is defined in this method [1]. If you nee

Re: Not enough free slots to run the job

Hi David, that's correct. A TM is a single process. A slot is just a virtual concept in the TM process and runs its program slice in multiple threads. Besides managed memory (which is split into chunks add assigned to slots) all other resources (CPU, heap, network, disk) are not isolated and free