Yes, it is a JobClient.
Cheers,
Kostas
On Thu, Oct 10, 2019 at 11:26 AM Zili Chen wrote:
>
> Thanks for your explanation Kostas.
>
> I agree that Clients are independent from the Executors. From
> your text I wonder one thing that whether Executor#execute
> returns a cluster client or a job clie
Thanks for your explanation Kostas.
I agree that Clients are independent from the Executors. From
your text I wonder one thing that whether Executor#execute
returns a cluster client or a job client? As discussed previously
I think conceptually it is a job client?
Best,
tison.
Kostas Kloudas 于2
Hi Tison,
I would say that as a first step, and until we see that the interfaces
we introduce cover all intended purposes, we keep the Executors
non-public.
>From the previous discussion, I think that in general the Clients are
independent from the Executors, as the Executors simply use the
client
Hi Kostas & Aljoscha,
I'm drafting a plan exposing multi-layered clients. It is mainly about
how we distinguish different layers and what clients we're going to
expose.
In FLIP-73 scope I'd like to ask a question that whether or not Executor
becomes a public interface that can be made use of by d
It might be useful to mention on FLIP-73 that the intention for
Executor.execute is to be an asynchronous API once it becomes public and
also refer to FLIP-74 as such.
On Fri, Oct 4, 2019 at 2:52 AM Aljoscha Krettek wrote:
> Hi Tison,
>
> I agree, for now the async Executor.execute() is an inte
Hi Tison,
I agree, for now the async Executor.execute() is an internal detail but during
your work for FLIP-74 it will probably also reach the public API.
Best,
Aljoscha
> On 4. Oct 2019, at 11:39, Zili Chen wrote:
>
> Hi Aljoscha,
>
> After clearly narrow the scope of this FLIP it looks goo
Hi Aljoscha,
After clearly narrow the scope of this FLIP it looks good to me the
interface
Executor and its discovery so that I'm glad to see the vote thread.
As you said, we should still discuss on implementation details but I don't
think
it should be a blocker of the vote thread because a vote
Do you all think we could agree on the basic executor primitives and start
voting on this FLIP? There are still some implementation details but I think we
can discuss/tackle them when we get to them and the various people implementing
this should be in close collaboration.
Best,
Aljoscha
> On
Hi,
I think the end goal is to have only one environment per API, but I think we
won’t be able to achieve that in the short-term because of backwards
compatibility. This is most notable with the context environment, preview
environments etc.
To keep this FLIP very slim we can make this only ab
Thanks for your explanation Kostas to make it clear subtasks under FLIP-73.
As you described, changes of Environment are included in this FLIP. For
"each
API to have a single Environment", it could be helpful to describe which
APIs we'd
like to have after FLIP-73. And if we keep multiple Environme
Hi Tison,
The changes that this FLIP propose are:
- the introduction of the Executor interface
- the fact that everything in the current state of job submission in
Flink can be defined through configuration parameters
- implementation of Executors that do not change any of the semantics
of the cur
- for Preview/OptimizedPlanEnv: I think they are orthogonal to the
Executors work, as they are using the exexute() method because this is
the only "entry" to the user program. To this regard, I believe we
should just see the fact that they have their dedicated environment as
an "implementation det
BTW, correct me if I misunderstand, now I learn more about our community
way. Since FLIP-73 aimed at introducing an interface with community
consensus the discussion is more about the interface in order to properly
define a useful and extensible API. The integration story could be a follow
up
since
Hi Kostas,
It seems does no harm we have a configuration parameter of Executor#execute
since we can merge this one with the one configured on Executor created and
let this one overwhelm that one.
I can see it is useful that conceptually we can create an Executor for a
series jobs
to the same clus
Hi again,
I did not include this to my previous email, as this is related to the
proposal on the FLIP itself.
In the existing proposal, the Executor interface is the following.
public interface Executor {
JobExecutionResult execute(Pipeline pipeline) throws Exception;
}
This implies that al
Hi all,
I agree with Tison that we should disentangle threads so that people
can work independently.
For FLIP-73:
- for Preview/OptimizedPlanEnv: I think they are orthogonal to the
Executors work, as they are using the exexute() method because this is
the only "entry" to the user program. To thi
Thanks for your thoughts Aljoscha.
Another question since FLIP-73 might contains refactors on Environemnt:
shall we support
something like PreviewPlanEnvironment? If so, how? From a user perspective
preview plan
is useful, by give visual view, to modify topos and configure without
submit it.
Best
I agree with Till that we should not change the semantics of per-job mode. In
my opinion per-job mode means that the cluster (JobManager) is brought up with
one job and it only executes that one job. There should be no open
ports/anything that would allow submitting further jobs. This is very im
Thanks for your clarification Till.
I agree with the current semantics of the per-job mode, one should deploy a
new cluster for each part of the job. Apart from the performance concern
it also means that PerJobExecutor knows how to deploy a cluster actually,
which is different from the description
I'm not saying that one cannot change the semantics of the per-job mode,
I'm mainly wondering whether this needs to happen as part of this FLIP. In
my understanding this has lot of overlap with the driver mode effort and
maybe not so much the executors. However, I agree that it is helpful to
have a
Hi Till,
The purpose to post thoughts above here is because FLIP-73 is unclear on
how to
achieve PerJobExecutor. In order to discuss this topic it is necessary to
clarify how
per-job mode runs regardless what it is now.
With PerJobExecutor called in Environment I don't think we still keep
current
To be honest I formerly want to firstly start a thread discuss about
what per-job mode means because things gets quite different whether
or not per-job mode contains exactly one JobGraph or allow to have
multiple part. Plus the complexity that whether or not we support
post-execution logic it becom
I'm not sure whether removing the current per-job mode semantics all
together is a good idea. It has some nice properties, for example the
JobGraph stays constant. With your proposal which I would coin the
driver mode, the JobGraph would be regenerated in case of a failover.
Depending on the user c
Thomas,
For your requirement on jar run REST endpoint we can follow a similar way
as job cluster way described above. That is, we submit the jar and runs user
program on cluster side.
Also cc Aljoscha,
There is no JobManager.runJar in mind. All logic that handles jar run is in
WebMonitor,
we pre
Thanks for your thoughts Kostas!
I agree Executor to be a concept on clients now. And sincerely second the
description
Now the Executor simply uses a client, e.g. a ClusterClient, to submit
the job (JobGraph) that it will create from the user program.
In that sense, the Executor is one level of a
Hi Zili,
I think we are more or less on the same page with most of the stuff
you mentioned.
The only slight difference, at least in my opinion, is that I do not
see the Executors as being a "Client".
As you mentioned, I can see having the following:
1). ClusterClientFactory: responsible for depl
Thanks for your reply Kostas.
As mentioned in FLIP-74 thread[1] there are two questions on Executor design
(1) Where Executor is in a multi-layered clients view.
(2) A bit more details about PerJobExecutor implementation.
For (1) Where Executor is in a multi-layered clients view,
As described i
Hi,
@Aljoscha, I believe that it is better to be done like this so that we
do not step on each-other's feet. If the executor already "knew" about
the JobClient, then we should also know about how we expect the
JobExecutionResult is retrieved (which is part of FLIP-74). I think it
is nice to have e
Hi,
I’m fine with either signature for the new execute() method but I think we
should focus on the executor discovery and executor configuration part in this
FLIP while FLIP-74 is about the evolution of the method signature to return a
future.
I understand that it’s a bit weird, that this FLIP
Hi Thomas and Zili,
As you both said the Executor is a new addition so there are no
compatibility concerns.
Backwards compatibility comes into play on the
(Stream)ExecutionEnvironment#execute().
This method has to stay and keep having the same (blocking) semantics,
but we can
add a new one, sth a
>Since Exceutor is a new interface, why is backward compatibility a concern?
For backward compatibility, it is on (Stream)ExecutionEnvironment#execute.
You're right that we don't stick to blocking to return a JobExecutionResult
in
Executor aspect but implementing env.execute with a unique
Executo
Since Exceutor is a new interface, why is backward compatibility a concern?
I could see that become an issue later when replacing Executor execute with
executeAsync. Or are both targeted for 1.10?
On Tue, Sep 24, 2019 at 10:24 AM Zili Chen wrote:
> Hi Thomas,
>
> >Should the new Executor execu
Hi Thomas,
>Should the new Executor execute method be defined as asynchronous? It could
>return a job handle to interact with the job and the legacy environments
>can still block to retain their semantics.
During our discussion there will be a method
executeAsync(...): CompletableFuture
where J
Thanks for the proposal. These changes will make it significantly easier to
programmatically use Flink in downstream frameworks.
Should the new Executor execute method be defined as asynchronous? It could
return a job handle to interact with the job and the legacy environments
can still block to r
Hi all,
In the context of the discussion about introducing the Job Client API [1],
there was a side-discussion about refactoring the way users submit jobs in
Flink. There were many different interesting ideas on the topic and 3
design documents that were trying to tackle both the issue about code
35 matches
Mail list logo
