Hi.
> Let me give you a concrete example. The Gateway may use a Generic catalog
> implementation backed by a Database (where table properties would be
stored
> in a DB). This would be a common setup in different Flink services.
> In this model the catalog itself needs DB credentials while the runn
Hey!
> Actually, a job requires same credentials to read the data or write data.
> When compiling job in the job manager, we don't require extra credentials
You are making a lot of assumptions here regarding the catalog
implementations that are not generally true.
Let me give you a concrete examp
Hi.
> My point regarding the credentials wasn't about lazy/non-lazy catalog
> initialization. The problem is that JM may run in an env where you don't
> necessarily want to have the same credentials at all.
Actually, a job requires same credentials to read the data or write data.
When compiling j
Hey!
My point regarding the credentials wasn't about lazy/non-lazy catalog
initialization. The problem is that JM may run in an env where you don't
necessarily want to have the same credentials at all.
> I agree we need 2 different runners or designs here. But I don't think we
> should only suppo
Hi, Gyula.
Thanks a lot for your response.
> In a production environment the gateway would hold the credentials to the
different catalogs and may even contain temporary tables etc.
In the FLIP-295[1], SQL Gateway has already supported using catalog stores
to initialize the catalog lazily. I thin
Hey!
I am a bit concerned about the design for gateway based submission model.
I think it's not a good model to access catalog information on the
JobManager side. In a production environment the gateway would hold the
credentials to the different catalogs and may even contain temporary tables
etc
Hi, eveyone.
The discussion doesn't receive any response for a while. I will close the
discussion and start the vote tomorrow if the discussion doesn't receive
any response today. Thanks all yours response!
Best,
Shengkai
Shengkai Fang 于2024年11月5日周二 14:00写道：
> Hi, Lincoln. Thanks for your resp
Hi, Lincoln. Thanks for your response.
> since both scriptPath and script(statements) can be null, we need to
clarify the behavior when both are empty, such as throwing an error
Yes, you are correct. I have updated the FLIP about this. When these fields
are both empty, the server throws an excep
Thanks Shengkai for driving this! Overall, looks good!\
I have two minor questions:
1. Regarding the interface parameters (including REST API
& Java interfaces), since both scriptPath and script(statements)
can be null, we need to clarify the behavior when both are
empty, such as throwing an error
Hi, Ferenc.
Thanks for your clarification. We can hard code these different options in
the sql-gateway module. I have updated the FLIP and PoC branch about this
part. But I think we should provide a unified API to ship artifacts to
different deployment.
Best,
Shengkai
Ferenc Csaky 于2024年11月4日
Hi Shengkai,
Thank you for driving this FLIP! I think this is a good way to
close this gap on the short-term until FLIP-316 can be finished.
I would only like to add one thing: YARN has a `yarn.ship-files`
config option that ships local or DFS files/directories to the
YARN cluster [1].
Best,
Fer
Hi, Shegnkai.
Thank you for your answer. I have no further questions.
--
Best！
Xuyang
At 2024-11-04 10:00:32, "Shengkai Fang" wrote:
>Hi, Xuyang. Thanks a lot for your response!
>
>> Does that means we will support multi DMLs, multi DQLs, mixed DMLs & DQLs
>in one sql script?
>
Hi, Xuyang. Thanks a lot for your response!
> Does that means we will support multi DMLs, multi DQLs, mixed DMLs & DQLs
in one sql script?
According to the doc[1], application mode only supports one job in ha
mode[2]. If users submit multiple jobs, dispatcher throws a
DuplicateJobSubmissionExcept
Hi, Shengkai.
Thanks for driving this great work. LGTM overall, I just have one question.
IIUC, application mode supports to run multi-execute in a single `main`
function[1]. Does that means
we will support multi DMLs, multi DQLs, mixed DMLs & DQLs in one sql script? If
yes, can you explain
Thanks Shengakai for driving this proposal. Supporting the deployment of
sql scripts in application mode is important for improving SQL job
submission. This FLIP design looks good to me.
Best,
Junrui
Ron Liu 于2024年10月31日周四 10:20写道：
> Hi, Shengkai
>
> Thanks for your quick response. It looks goo
Hi, Ron!
> I noticed that you say this FLIP focuses on supporting deploy sql
scripts to the application cluster, does it mean that it only supports
non-interactive gateway mode?
Yes. This FLIP only supports to deploy a script in non-interactive mode.
> Whether all SQL commands such as DDL & DML
Hi, Shengkai
Thanks for your quick response. It looks good to me.
Best
Ron
Shengkai Fang 于2024年10月31日周四 10:08写道：
> Hi, Ron!
>
> > I noticed that you say this FLIP focuses on supporting deploy sql
> scripts to the application cluster, does it mean that it only supports
> non-interactive gatewa
Hi, Feng!
> if only clusterID is available, it may not be very convenient to connect
to this application later on.
If FLIP-479 is accepted, I think we can just adapt the sql-gateway
behaviour to the behaviour that FLIP-479 mentioned.
Best,
Shengkai
Hi Shengkai,
Thank you for the timely updates and replies:
1. I have been studying your POC code, introducing OperationExecutor in
SqlRunner is a great idea, which can maximize compatibility with SqlClient.
Especially some SQL statements cannot be executed directly in the Table
Environmen
I also have some questions:
1. Whether all SQL commands such as DDL & DML & SELECT are supported.
2. How to determine JobID and return JobID & ClusterId from the application
cluster
3. How to dynamically download the JAR specified by the user when
submitting the sql script, and whether it is possi
Hi, Shengkai
Thanks for initializing this FLIP, supports application mode for SQL
Gateway is a great job. The FLIP design looks good to me.
I've read the FLIP-316 which mentions supporting deploying SQL job to
application clusters for interactive or non-interactive gateway mode.
But I noticed t
Hi, HongShun. Thanks a lot for your response!
> I wonder what is the scope of this FLIP, only aim for k8s, not including
yarn?
This FLIP also works for the yarn-application mode. But the yarn deployment
doesn't support to ship the artifacts into the remote side. Please
correct me if I'm wrong.
>
Hi, Feng.
Thanks for your response.
> Will FLIP-316 merge into Flink 2.0 too ?
I don't have time to finish the FLIP-316. So it depends on whether anyone
else can help to continue the discussion.
> Will SqlDriver use the same one?
Yes. We should reuse the same driver. I think the driver is the
Hi, ShengKai
Thanks for pushing for this meaningful job. I have some problems:
Firstly, the outline design show the overall design in k8s. I wonder what
is the scope of this FLIP, only aim for k8s, not including yarn?
Secondly,
> Flink offers great flexibility by permitting users to pass custom
Hi, Shenkai
Thank you for initiating this FLIP, I understand that supporting
application mode for SQL gateway is very important. There are two small
issues.
> FLIP-480 is different from FLIP-316
1. Will FLIP-316 merge into Flink 2.0 too ?
2. Will SqlDriver use the same one?
The detail
Hi, everyone.
I'd like to initiate a discussion about FLIP-480: Support to deploy script
in application mode[1].
FLIP-480 supports to solve the problem that table program can not run in
application mode. Comparing to FLIP-316[2], FLIP-480 tries to compile the
script in the JM side, which is free
26 matches
Mail list logo
