Hi to all,
I had a very long but useful chat with Fabian and I understood a lot of
concepts that was not clear at all to me. We started from the Flink runtime
documentation page (
https://ci.apache.org/projects/flink/flink-docs-release-1.2/concepts/runtime.html)
but
I discovered that the terminology is very inconsistent and misleading along
the page...
For example, one of the very first sentences is :
"Flink chains operator subtasks together into tasks. Each task is executed
by one thread."
What I first understood was that every operator can be executed only by a
single thread in all the cluster....probably it should be better "one
thread per task slot" (at least).
Moreover, if I'm not wrong, a Task Slot can execute only 1 subtask (aka
parallel instance) of each task and there's no limit to the number of
subtasks per slot (and this is not highlighted at all in that document).
The only constraint is that they should belong to different tasks (right?).
If there's a google doc version of that page I could try to rewrite it down
in order to make it easier to understand some parts...however I still have
some more questions:
1. Is it correct that a single Task Slot can execute only a single
subtask of each task and that this task is executed by a single thread
within the slot)?
2. If it so:
1. why at that page there's written "By default, Flink allows
subtasks to share slots even if they are subtasks of different tasks, so
long as they are from the same job"? It seems that it is more
common to run
multiple subtasks of the same task (in a slot) than executing different
substasks of different tasks, although this is still
permitted...from what
I understood a slot cannot run multiple subtask of the same task at all!
2. and why this constraint? Is there any good reason for that? A
subtask is mapped to 1 thread in the TaskManager, so why a TM
with 2 slots
can run 2 subtasks of the same task (in the same JVM) while a TM with 1
slot cannot (while it can execute an arbitrary number of subtasks of
different tasks)?
3. It it is not so, there's no images representing such a situation in
that page...
4. Isn't dangerous to allow (potentially) an unlimited number of threads
per TM slot??
Cheers,
Flavio
