Hi Mason, I get your point. I'm increasingly feeling the need to introduce a built-in file distribution mechanism for flink-kubernetes module, just like Spark does with `spark.kubernetes.file.upload.path` [1].
I’m assuming the workflow is as follows: - KubernetesClusterDescripter uploads all local resources to a remote storage via Flink filesystem (skips if the resources are already remote). - KubernetesApplicationClusterEntrypoint downloads the resources and put them in the classpath during startup. I wouldn't mind splitting it into another FLIP to ensure that everything is done correctly. cc'ed @Yang to gather more opinions. [1] https://spark.apache.org/docs/latest/running-on-kubernetes.html#dependency-management Best, Paul Lam > 2023年6月8日 12:15，Mason Chen <mas.chen6...@gmail.com> 写道： > > Hi Paul, > > Thanks for your response! > > I agree that utilizing SQL Drivers in Java applications is equally important >> as employing them in SQL Gateway. WRT init containers, I think most >> users use them just as a workaround. For example, wget a jar from the >> maven repo. >> >> We could implement the functionality in SQL Driver in a more graceful >> way and the flink-supported filesystem approach seems to be a >> good choice. >> > > My main point is: can we solve the problem with a design agnostic of SQL > and Stream API? I mentioned a use case where this ability is useful for > Java or Stream API applications. Maybe this is even a non-goal to your FLIP > since you are focusing on the driver entrypoint. > > Jark mentioned some optimizations: > >> This allows SQLGateway to leverage some metadata caching and UDF JAR >> caching for better compiling performance. >> > It would be great to see this even outside the SQLGateway (i.e. UDF JAR > caching). > > Best, > Mason > > On Wed, Jun 7, 2023 at 2:26 AM Shengkai Fang <fskm...@gmail.com> wrote: > >> Hi. Paul. Thanks for your update and the update makes me understand the >> design much better. >> >> But I still have some questions about the FLIP. >> >>> For SQL Gateway, only DMLs need to be delegated to the SQL server >>> Driver. I would think about the details and update the FLIP. Do you have >> some >>> ideas already? >> >> If the applicaiton mode can not support library mode, I think we should >> only execute INSERT INTO and UPDATE/ DELETE statement in the application >> mode. AFAIK, we can not support ANALYZE TABLE and CALL PROCEDURE >> statements. The ANALYZE TABLE syntax need to register the statistic to the >> catalog after job finishes and the CALL PROCEDURE statement doesn't >> generate the ExecNodeGraph. >> >> * Introduce storage via option `sql-gateway.application.storage-dir` >> >> If we can not support to submit the jars through web submission, +1 to >> introduce the options to upload the files. While I think the uploader >> should be responsible to remove the uploaded jars. Can we remove the jars >> if the job is running or gateway exits? >> >> * JobID is not avaliable >> >> Can we use the returned rest client by ApplicationDeployer to query the job >> id? I am concerned that users don't know which job is related to the >> submitted SQL. >> >> * Do we need to introduce a new module named flink-table-sql-runner? >> >> It seems we need to introduce a new module. Will the new module is >> available in the distribution package? I agree with Jark that we don't need >> to introduce this for table-API users and these users have their main >> class. If we want to make users write the k8s operator more easily, I think >> we should modify the k8s operator repo. If we don't need to support SQL >> files, can we make this jar only visible in the sql-gateway like we do in >> the planner loader?[1] >> >> [1] >> >> https://github.com/apache/flink/blob/master/flink-table/flink-table-planner-loader/src/main/java/org/apache/flink/table/planner/loader/PlannerModule.java#L95 >> >> Best, >> Shengkai >> >> >> >> >> >> >> >> >> Weihua Hu <huweihua....@gmail.com> 于2023年6月7日周三 10:52写道： >> >>> Hi, >>> >>> Thanks for updating the FLIP. >>> >>> I have two cents on the distribution of SQLs and resources. >>> 1. Should we support a common file distribution mechanism for k8s >>> application mode? >>> I have seen some issues and requirements on the mailing list. >>> In our production environment, we implement the download command in the >>> CliFrontend. >>> And automatically add an init container to the POD for file >> downloading. >>> The advantage of this >>> is that we can use all Flink-supported file systems to store files. >>> >>> This need more discussion. I would appreciate hearing more opinions. >>> >>> 2. In this FLIP, we distribute files in two different ways in YARN and >>> Kubernetes. Can we combine it in one way? >>> If we don't want to implement a common file distribution for k8s >>> application mode. Could we use the SQLDriver >>> to download the files both in YARN and K8S? IMO, this can reduce the >> cost >>> of code maintenance. >>> >>> Best, >>> Weihua >>> >>> >>> On Wed, Jun 7, 2023 at 10:18 AM Paul Lam <paullin3...@gmail.com> wrote: >>> >>>> Hi Mason, >>>> >>>> Thanks for your input! >>>> >>>>> +1 for init containers or a more generalized way of obtaining >> arbitrary >>>>> files. File fetching isn't specific to just SQL--it also matters for >>> Java >>>>> applications if the user doesn't want to rebuild a Flink image and >> just >>>>> wants to modify the user application fat jar. >>>> >>>> I agree that utilizing SQL Drivers in Java applications is equally >>>> important >>>> as employing them in SQL Gateway. WRT init containers, I think most >>>> users use them just as a workaround. For example, wget a jar from the >>>> maven repo. >>>> >>>> We could implement the functionality in SQL Driver in a more graceful >>>> way and the flink-supported filesystem approach seems to be a >>>> good choice. >>>> >>>>> Also, what do you think about prefixing the config options with >>>>> `sql-driver` instead of just `sql` to be more specific? >>>> >>>> LGTM, since SQL Driver is a public interface and the options are >>>> specific to it. >>>> >>>> Best, >>>> Paul Lam >>>> >>>>> 2023年6月6日 06:30，Mason Chen <mas.chen6...@gmail.com> 写道： >>>>> >>>>> Hi Paul, >>>>> >>>>> +1 for this feature and supporting SQL file + JSON plans. We get a >> lot >>> of >>>>> requests to just be able to submit a SQL file, but the JSON plan >>>>> optimizations make sense. >>>>> >>>>> +1 for init containers or a more generalized way of obtaining >> arbitrary >>>>> files. File fetching isn't specific to just SQL--it also matters for >>> Java >>>>> applications if the user doesn't want to rebuild a Flink image and >> just >>>>> wants to modify the user application fat jar. >>>>> >>>>> Please note that we could reuse the checkpoint storage like S3/HDFS, >>>> which >>>>>> should >>>>> >>>>> be required to run Flink in production, so I guess that would be >>>> acceptable >>>>>> for most >>>>> >>>>> users. WDYT? >>>>> >>>>> >>>>> If you do go this route, it would be nice to support writing these >>> files >>>> to >>>>> S3/HDFS via Flink. This makes access control and policy management >>>> simpler. >>>>> >>>>> Also, what do you think about prefixing the config options with >>>>> `sql-driver` instead of just `sql` to be more specific? >>>>> >>>>> Best, >>>>> Mason >>>>> >>>>> On Mon, Jun 5, 2023 at 2:28 AM Paul Lam <paullin3...@gmail.com >>> <mailto: >>>> paullin3...@gmail.com>> wrote: >>>>> >>>>>> Hi Jark, >>>>>> >>>>>> Thanks for your input! Please see my comments inline. >>>>>> >>>>>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL? >>>>>>> DataStream API also doesn't provide a default main class for users, >>>>>>> why do we need to provide such one for SQL? >>>>>> >>>>>> Sorry for the confusion I caused. By DataStream jobs, I mean jobs >>>> submitted >>>>>> via Flink CLI which actually could be DataStream/Table jobs. >>>>>> >>>>>> I think a default main class would be user-friendly which eliminates >>> the >>>>>> need >>>>>> for users to write a main class as SQLRunner in Flink K8s operator >>> [1]. >>>>>> >>>>>>> I thought the proposed SqlDriver was a dedicated main class >> accepting >>>>>> SQL files, is >>>>>>> that correct? >>>>>> >>>>>> Both JSON plans and SQL files are accepted. SQL Gateway should use >>> JSON >>>>>> plans, >>>>>> while CLI users may use either JSON plans or SQL files. >>>>>> >>>>>> Please see the updated FLIP[2] for more details. >>>>>> >>>>>>> Personally, I prefer the way of init containers which doesn't >> depend >>> on >>>>>>> additional components. >>>>>>> This can reduce the moving parts of a production environment. >>>>>>> Depending on a distributed file system makes the testing, demo, and >>>> local >>>>>>> setup harder than init containers. >>>>>> >>>>>> Please note that we could reuse the checkpoint storage like S3/HDFS, >>>> which >>>>>> should >>>>>> be required to run Flink in production, so I guess that would be >>>>>> acceptable for most >>>>>> users. WDYT? >>>>>> >>>>>> WRT testing, demo, and local setups, I think we could support the >>> local >>>>>> filesystem >>>>>> scheme i.e. file://** as the state backends do. It works as long as >>> SQL >>>>>> Gateway >>>>>> and JobManager(or SQL Driver) can access the resource directory >>>> (specified >>>>>> via >>>>>> `sql-gateway.application.storage-dir`). >>>>>> >>>>>> Thanks! >>>>>> >>>>>> [1] >>>>>> >>>> >>> >> https://github.com/apache/flink-kubernetes-operator/blob/main/examples/flink-sql-runner-example/src/main/java/org/apache/flink/examples/SqlRunner.java >>>>>> [2] >>>>>> >>>> >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver >>>>>> [3] >>>>>> >>>> >>> >> https://github.com/apache/flink/blob/3245e0443b2a4663552a5b707c5c8c46876c1f6d/flink-runtime/src/test/java/org/apache/flink/runtime/state/filesystem/AbstractFileCheckpointStorageAccessTestBase.java#L161 >>>>>> >>>>>> Best, >>>>>> Paul Lam >>>>>> >>>>>>> 2023年6月3日 12:21，Jark Wu <imj...@gmail.com> 写道： >>>>>>> >>>>>>> Hi Paul, >>>>>>> >>>>>>> Thanks for your reply. I left my comments inline. >>>>>>> >>>>>>>> As the FLIP said, it’s good to have a default main class for Flink >>>> SQLs, >>>>>>>> which allows users to submit Flink SQLs in the same way as >>> DataStream >>>>>>>> jobs, or else users need to write their own main class. >>>>>>> >>>>>>> Isn't Table API the same way as DataSream jobs to submit Flink SQL? >>>>>>> DataStream API also doesn't provide a default main class for users, >>>>>>> why do we need to provide such one for SQL? >>>>>>> >>>>>>>> With the help of ExecNodeGraph, do we still need the serialized >>>>>>>> SessionState? If not, we could make SQL Driver accepts two >>> serialized >>>>>>>> formats: >>>>>>> >>>>>>> No, ExecNodeGraph doesn't need to serialize SessionState. I thought >>> the >>>>>>> proposed SqlDriver was a dedicated main class accepting SQL files, >> is >>>>>>> that correct? >>>>>>> If true, we have to ship the SessionState for this case which is a >>>> large >>>>>>> work. >>>>>>> I think we just need a JsonPlanDriver which is a main class that >>>> accepts >>>>>>> JsonPlan as the parameter. >>>>>>> >>>>>>> >>>>>>>> The common solutions I know is to use distributed file systems or >>> use >>>>>>>> init containers to localize the resources. >>>>>>> >>>>>>> Personally, I prefer the way of init containers which doesn't >> depend >>> on >>>>>>> additional components. >>>>>>> This can reduce the moving parts of a production environment. >>>>>>> Depending on a distributed file system makes the testing, demo, and >>>> local >>>>>>> setup harder than init containers. >>>>>>> >>>>>>> Best, >>>>>>> Jark >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Fri, 2 Jun 2023 at 18:10, Paul Lam <paullin3...@gmail.com >>> <mailto: >>>> paullin3...@gmail.com> <mailto: >>>>>> paullin3...@gmail.com <mailto:paullin3...@gmail.com>>> wrote: >>>>>>> >>>>>>>> The FLIP is in the early phase and some details are not included, >>> but >>>>>>>> fortunately, we got lots of valuable ideas from the discussion. >>>>>>>> >>>>>>>> Thanks to everyone who joined the dissuasion! >>>>>>>> @Weihua @Shanmon @Shengkai @Biao @Jark >>>>>>>> >>>>>>>> This weekend I’m gonna revisit and update the FLIP, adding more >>>>>>>> details. Hopefully, we can further align our opinions. >>>>>>>> >>>>>>>> Best, >>>>>>>> Paul Lam >>>>>>>> >>>>>>>>> 2023年6月2日 18:02，Paul Lam <paullin3...@gmail.com <mailto: >>>> paullin3...@gmail.com>> 写道： >>>>>>>>> >>>>>>>>> Hi Jark, >>>>>>>>> >>>>>>>>> Thanks a lot for your input! >>>>>>>>> >>>>>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it >>>> still >>>>>>>>>> necessary to support SQL Driver? >>>>>>>>> >>>>>>>>> I think so. Apart from usage in SQL Gateway, SQL Driver could >>>> simplify >>>>>>>>> Flink SQL execution with Flink CLI. >>>>>>>>> >>>>>>>>> As the FLIP said, it’s good to have a default main class for >> Flink >>>>>> SQLs, >>>>>>>>> which allows users to submit Flink SQLs in the same way as >>> DataStream >>>>>>>>> jobs, or else users need to write their own main class. >>>>>>>>> >>>>>>>>>> SQL Driver needs to serialize SessionState which is very >>> challenging >>>>>>>>>> but not detailed covered in the FLIP. >>>>>>>>> >>>>>>>>> With the help of ExecNodeGraph, do we still need the serialized >>>>>>>>> SessionState? If not, we could make SQL Driver accepts two >>> serialized >>>>>>>>> formats: >>>>>>>>> >>>>>>>>> - SQL files for user-facing public usage >>>>>>>>> - ExecNodeGraph for internal usage >>>>>>>>> >>>>>>>>> It’s kind of similar to the relationship between job jars and >>>>>> jobgraphs. >>>>>>>>> >>>>>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that >>>> true? >>>>>>>> Is it >>>>>>>>>> possible to support it? >>>>>>>>> >>>>>>>>> Yes, K8s doesn’t distribute any files. It’s the users’ >>> responsibility >>>>>> to >>>>>>>> make >>>>>>>>> sure the resources are accessible in the containers. The common >>>>>> solutions >>>>>>>>> I know is to use distributed file systems or use init containers >> to >>>>>>>> localize the >>>>>>>>> resources. >>>>>>>>> >>>>>>>>> Now I lean toward introducing a fs to do the distribution job. >>> WDYT? >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Paul Lam >>>>>>>>> >>>>>>>>>> 2023年6月1日 20:33，Jark Wu <imj...@gmail.com <mailto: >>> imj...@gmail.com> >>>> <mailto:imj...@gmail.com <mailto:imj...@gmail.com>> >>>>>> <mailto:imj...@gmail.com <mailto:imj...@gmail.com> <mailto: >>>> imj...@gmail.com <mailto:imj...@gmail.com>>>> >>>>>>>> 写道： >>>>>>>>>> >>>>>>>>>> Hi Paul, >>>>>>>>>> >>>>>>>>>> Thanks for starting this discussion. I like the proposal! This >> is >>> a >>>>>>>>>> frequently requested feature! >>>>>>>>>> >>>>>>>>>> I agree with Shengkai that ExecNodeGraph as the submission >> object >>>> is a >>>>>>>>>> better idea than SQL file. To be more specific, it should be >>>>>>>> JsonPlanGraph >>>>>>>>>> or CompiledPlan which is the serializable representation. >>>> CompiledPlan >>>>>>>> is a >>>>>>>>>> clear separation between compiling/optimization/validation and >>>>>>>> execution. >>>>>>>>>> This can keep the validation and metadata accessing still on the >>>>>>>> SQLGateway >>>>>>>>>> side. This allows SQLGateway to leverage some metadata caching >> and >>>> UDF >>>>>>>> JAR >>>>>>>>>> caching for better compiling performance. >>>>>>>>>> >>>>>>>>>> If we decide to submit ExecNodeGraph instead of SQL file, is it >>>> still >>>>>>>>>> necessary to support SQL Driver? Regarding non-interactive SQL >>> jobs, >>>>>>>> users >>>>>>>>>> can use the Table API program for application mode. SQL Driver >>> needs >>>>>> to >>>>>>>>>> serialize SessionState which is very challenging but not >> detailed >>>>>>>> covered >>>>>>>>>> in the FLIP. >>>>>>>>>> >>>>>>>>>> Regarding "K8S doesn't support shipping multiple jars", is that >>>> true? >>>>>>>> Is it >>>>>>>>>> possible to support it? >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Jark >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, 1 Jun 2023 at 16:58, Paul Lam <paullin3...@gmail.com >>>> <mailto:paullin3...@gmail.com> <mailto: >>>>>> paullin3...@gmail.com <mailto:paullin3...@gmail.com>> <mailto: >>>>>>>> paullin3...@gmail.com <mailto:paullin3...@gmail.com> <mailto: >>>> paullin3...@gmail.com <mailto:paullin3...@gmail.com>>>> wrote: >>>>>>>>>> >>>>>>>>>>> Hi Weihua, >>>>>>>>>>> >>>>>>>>>>> You’re right. Distributing the SQLs to the TMs is one of the >>>>>>>> challenging >>>>>>>>>>> parts of this FLIP. >>>>>>>>>>> >>>>>>>>>>> Web submission is not enabled in application mode currently as >>> you >>>>>>>> said, >>>>>>>>>>> but it could be changed if we have good reasons. >>>>>>>>>>> >>>>>>>>>>> What do you think about introducing a distributed storage for >> SQL >>>>>>>> Gateway? >>>>>>>>>>> >>>>>>>>>>> We could make use of Flink file systems [1] to distribute the >> SQL >>>>>>>> Gateway >>>>>>>>>>> generated resources, that should solve the problem at its root >>>> cause. >>>>>>>>>>> >>>>>>>>>>> Users could specify Flink-supported file systems to ship files. >>>> It’s >>>>>>>> only >>>>>>>>>>> required when using SQL Gateway with K8s application mode. >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>>>> >>>>>>>> >>>>>> >>>> >>> >> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ >>>> < >>>> >>> >> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ >>>>> >>>>>> < >>>>>> >>>> >>> >> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ >>>> < >>>> >>> >> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ >>>>> >>>>>>> >>>>>>>> < >>>>>>>> >>>>>> >>>> >>> >> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ >>>> < >>>> >>> >> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ >>>>> >>>>>> < >>>>>> >>>> >>> >> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ >>>> < >>>> >>> >> https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/filesystems/overview/ >>>>> >>>>>>> >>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Paul Lam >>>>>>>>>>> >>>>>>>>>>>> 2023年6月1日 13:55，Weihua Hu <huweihua....@gmail.com <mailto: >>>> huweihua....@gmail.com> <mailto: >>>>>> huweihua....@gmail.com <mailto:huweihua....@gmail.com>>> 写道： >>>>>>>>>>>> >>>>>>>>>>>> Thanks Paul for your reply. >>>>>>>>>>>> >>>>>>>>>>>> SQLDriver looks good to me. >>>>>>>>>>>> >>>>>>>>>>>> 2. Do you mean a pass the SQL string a configuration or a >>> program >>>>>>>>>>> argument? >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I brought this up because we were unable to pass the SQL file >> to >>>>>> Flink >>>>>>>>>>>> using Kubernetes mode. >>>>>>>>>>>> For DataStream/Python users, they need to prepare their images >>> for >>>>>> the >>>>>>>>>>> jars >>>>>>>>>>>> and dependencies. >>>>>>>>>>>> But for SQL users, they can use a common image to run >> different >>>> SQL >>>>>>>>>>> queries >>>>>>>>>>>> if there are no other udf requirements. >>>>>>>>>>>> It would be great if the SQL query and image were not bound. >>>>>>>>>>>> >>>>>>>>>>>> Using strings is a way to decouple these, but just as you >>>> mentioned, >>>>>>>> it's >>>>>>>>>>>> not easy to pass complex SQL. >>>>>>>>>>>> >>>>>>>>>>>>> use web submission >>>>>>>>>>>> AFAIK, we can not use web submission in the Application mode. >>>> Please >>>>>>>>>>>> correct me if I'm wrong. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Best, >>>>>>>>>>>> Weihua >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, May 31, 2023 at 9:37 PM Paul Lam < >> paullin3...@gmail.com >>>> <mailto:paullin3...@gmail.com> >>>>>> <mailto:paullin3...@gmail.com <mailto:paullin3...@gmail.com>>> >>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hi Biao, >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for your comments! >>>>>>>>>>>>> >>>>>>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive >> Flink >>>> SQL >>>>>>>> jobs >>>>>>>>>>>>> in >>>>>>>>>>>>>> Application mode? More specifically, if we use SQL >>>> client/gateway >>>>>> to >>>>>>>>>>>>>> execute some interactive SQLs like a SELECT query, can we >> ask >>>>>> flink >>>>>>>> to >>>>>>>>>>>>> use >>>>>>>>>>>>>> Application mode to execute those queries after this FLIP? >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for pointing it out. I think only DMLs would be >> executed >>>> via >>>>>>>> SQL >>>>>>>>>>>>> Driver. >>>>>>>>>>>>> I'll add the scope to the FLIP. >>>>>>>>>>>>> >>>>>>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is >>>>>>>> trivial as >>>>>>>>>>>>> we >>>>>>>>>>>>>> can ship files via YARN's tool easily but for K8s, things >> can >>> be >>>>>>>> more >>>>>>>>>>>>>> complicated as Shengkai said. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Your input is very informative. I’m thinking about using web >>>>>>>> submission, >>>>>>>>>>>>> but it requires exposing the JobManager port which could also >>> be >>>> a >>>>>>>>>>> problem >>>>>>>>>>>>> on K8s. >>>>>>>>>>>>> >>>>>>>>>>>>> Another approach is to explicitly require a distributed >> storage >>>> to >>>>>>>> ship >>>>>>>>>>>>> files, >>>>>>>>>>>>> but we may need a new deployment executor for that. >>>>>>>>>>>>> >>>>>>>>>>>>> What do you think of these two approaches? >>>>>>>>>>>>> >>>>>>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are >>>> some >>>>>>>>>>>>>> unserializable fields >>>>>>>>>>>>>> like >>>>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader. >>>>>>>>>>> It >>>>>>>>>>>>>> may be worthwhile to add more details about the >> serialization >>>>>> part. >>>>>>>>>>>>> >>>>>>>>>>>>> I agree. That’s a missing part. But if we use ExecNodeGraph >> as >>>>>>>> Shengkai >>>>>>>>>>>>> mentioned, do we eliminate the need for serialization of >>>>>>>> SessionState? >>>>>>>>>>>>> >>>>>>>>>>>>> Best, >>>>>>>>>>>>> Paul Lam >>>>>>>>>>>>> >>>>>>>>>>>>>> 2023年5月31日 13:07，Biao Geng <biaoge...@gmail.com <mailto: >>>> biaoge...@gmail.com> <mailto: >>>>>> biaoge...@gmail.com <mailto:biaoge...@gmail.com>>> 写道： >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks Paul for the proposal!I believe it would be very >> useful >>>> for >>>>>>>>>>> flink >>>>>>>>>>>>>> users. >>>>>>>>>>>>>> After reading the FLIP, I have some questions: >>>>>>>>>>>>>> 1. Scope: is this FLIP only targeted for non-interactive >> Flink >>>> SQL >>>>>>>> jobs >>>>>>>>>>>>> in >>>>>>>>>>>>>> Application mode? More specifically, if we use SQL >>>> client/gateway >>>>>> to >>>>>>>>>>>>>> execute some interactive SQLs like a SELECT query, can we >> ask >>>>>> flink >>>>>>>> to >>>>>>>>>>>>> use >>>>>>>>>>>>>> Application mode to execute those queries after this FLIP? >>>>>>>>>>>>>> 2. Deployment: I believe in YARN mode, the implementation is >>>>>>>> trivial as >>>>>>>>>>>>> we >>>>>>>>>>>>>> can ship files via YARN's tool easily but for K8s, things >> can >>> be >>>>>>>> more >>>>>>>>>>>>>> complicated as Shengkai said. I have implemented a simple >> POC >>>>>>>>>>>>>> < >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>> >>>> >>> >> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 >>>> < >>>> >>> >> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 >>>>> >>>>>> < >>>>>> >>>> >>> >> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 >>>> < >>>> >>> >> https://github.com/bgeng777/flink/commit/5b4338fe52ec343326927f0fc12f015dd22b1133 >>>>> >>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> based on SQL client before(i.e. consider the SQL client >> which >>>>>>>> supports >>>>>>>>>>>>>> executing a SQL file as the SQL driver in this FLIP). One >>>> problem >>>>>> I >>>>>>>>>>> have >>>>>>>>>>>>>> met is how do we ship SQL files ( or Job Graph) to the k8s >>> side. >>>>>>>>>>> Without >>>>>>>>>>>>>> such support, users have to modify the initContainer or >>> rebuild >>>> a >>>>>>>> new >>>>>>>>>>> K8s >>>>>>>>>>>>>> image every time to fetch the SQL file. Like the flink k8s >>>>>> operator, >>>>>>>>>>> one >>>>>>>>>>>>>> workaround is to utilize the flink config(transforming the >> SQL >>>>>> file >>>>>>>> to >>>>>>>>>>> a >>>>>>>>>>>>>> escaped string like Weihua mentioned) which will be >> converted >>>> to a >>>>>>>>>>>>>> ConfigMap but K8s has size limit of ConfigMaps(no larger >> than >>>> 1MB >>>>>>>>>>>>>> < >> https://kubernetes.io/docs/concepts/configuration/configmap/ >>> < >>>> https://kubernetes.io/docs/concepts/configuration/configmap/> < >>>>>> https://kubernetes.io/docs/concepts/configuration/configmap/ < >>>> https://kubernetes.io/docs/concepts/configuration/configmap/>>>). >>>>>>>> Not >>>>>>>>>>>>> sure >>>>>>>>>>>>>> if we have better solutions. >>>>>>>>>>>>>> 3. Serialization of SessionState: in SessionState, there are >>>> some >>>>>>>>>>>>>> unserializable fields >>>>>>>>>>>>>> like >>>>>>>> org.apache.flink.table.resource.ResourceManager#userClassLoader. >>>>>>>>>>> It >>>>>>>>>>>>>> may be worthwhile to add more details about the >> serialization >>>>>> part. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, >>>>>>>>>>>>>> Biao Geng >>>>>>>>>>>>>> >>>>>>>>>>>>>> Paul Lam <paullin3...@gmail.com <mailto: >> paullin3...@gmail.com >>>> >>>> <mailto:paullin3...@gmail.com <mailto:paullin3...@gmail.com>>> >>>>>> 于2023年5月31日周三 11:49写道： >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Weihua, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks a lot for your input! Please see my comments inline. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL >>> Job. >>>>>> (Not >>>>>>>>>>>>>>> strong, >>>>>>>>>>>>>>>> the SQLDriver is fine for me) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I’ve thought about SQL Runner but picked SQL Driver for the >>>>>>>> following >>>>>>>>>>>>>>> reasons FYI: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 1. I have a PythonDriver doing the same job for PyFlink [1] >>>>>>>>>>>>>>> 2. Flink program's main class is sort of like Driver in >> JDBC >>>>>> which >>>>>>>>>>>>>>> translates SQLs into >>>>>>>>>>>>>>> databases specific languages. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> In general, I’m +1 for SQL Driver and +0 for SQL Runner. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, >> we >>>> need >>>>>>>> to >>>>>>>>>>>>>>> prepare >>>>>>>>>>>>>>>> a SQL file in an image for Kubernetes application mode, >>> which >>>>>> may >>>>>>>> be >>>>>>>>>>> a >>>>>>>>>>>>>>> bit >>>>>>>>>>>>>>>> cumbersome. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Do you mean a pass the SQL string a configuration or a >>> program >>>>>>>>>>> argument? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I thought it might be convenient for testing propose, but >> not >>>>>>>>>>>>> recommended >>>>>>>>>>>>>>> for production, >>>>>>>>>>>>>>> cause Flink SQLs could be complicated and involves lots of >>>>>>>> characters >>>>>>>>>>>>> that >>>>>>>>>>>>>>> need to escape. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> WDYT? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the >>>>>>>>>>>>>>> "run-application" >>>>>>>>>>>>>>>> command. Does that mean we need to perform automatic >>> detection >>>>>> in >>>>>>>>>>>>> Flink? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Yes! It’s like running a PyFlink job with the following >>>> command: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> ``` >>>>>>>>>>>>>>> ./bin/flink run \ >>>>>>>>>>>>>>> --pyModule table.word_count \ >>>>>>>>>>>>>>> --pyFiles examples/python/table >>>>>>>>>>>>>>> ``` >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The CLI determines if it’s a SQL job, if yes apply the SQL >>>> Driver >>>>>>>>>>>>>>> automatically. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>> >>>> >>> >> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java >>>> < >>>> >>> >> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java >>>>> >>>>>> < >>>>>> >>>> >>> >> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java >>>> < >>>> >>> >> https://github.com/apache/flink/blob/master/flink-python/src/main/java/org/apache/flink/client/python/PythonDriver.java >>>>> >>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>> Paul Lam >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 2023年5月30日 21:56，Weihua Hu <huweihua....@gmail.com >> <mailto: >>>> huweihua....@gmail.com> <mailto: >>>>>> huweihua....@gmail.com>> 写道： >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks Paul for the proposal. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> +1 for this. It is valuable in improving ease of use. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> I have a few questions. >>>>>>>>>>>>>>>> - Is SQLRunner the better name? We use this to run a SQL >>> Job. >>>>>> (Not >>>>>>>>>>>>>>> strong, >>>>>>>>>>>>>>>> the SQLDriver is fine for me) >>>>>>>>>>>>>>>> - Could we run SQL jobs using SQL in strings? Otherwise, >> we >>>> need >>>>>>>> to >>>>>>>>>>>>>>> prepare >>>>>>>>>>>>>>>> a SQL file in an image for Kubernetes application mode, >>> which >>>>>> may >>>>>>>> be >>>>>>>>>>> a >>>>>>>>>>>>>>> bit >>>>>>>>>>>>>>>> cumbersome. >>>>>>>>>>>>>>>> - I noticed that we don't specify the SQLDriver jar in the >>>>>>>>>>>>>>> "run-application" >>>>>>>>>>>>>>>> command. Does that mean we need to perform automatic >>> detection >>>>>> in >>>>>>>>>>>>> Flink? >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>> Weihua >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, May 29, 2023 at 7:24 PM Paul Lam < >>>> paullin3...@gmail.com >>>>>> <mailto:paullin3...@gmail.com <mailto:paullin3...@gmail.com>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi team, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I’d like to start a discussion about FLIP-316 [1], which >>>>>>>> introduces >>>>>>>>>>> a >>>>>>>>>>>>>>> SQL >>>>>>>>>>>>>>>>> driver as the >>>>>>>>>>>>>>>>> default main class for Flink SQL jobs. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Currently, Flink SQL could be executed out of the box >>> either >>>>>> via >>>>>>>> SQL >>>>>>>>>>>>>>>>> Client/Gateway >>>>>>>>>>>>>>>>> or embedded in a Flink Java/Python program. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> However, each one has its drawback: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> - SQL Client/Gateway doesn’t support the application >>>> deployment >>>>>>>> mode >>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>> - Flink Java/Python program requires extra work to write >> a >>>>>>>> non-SQL >>>>>>>>>>>>>>> program >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Therefore, I propose adding a SQL driver to act as the >>>> default >>>>>>>> main >>>>>>>>>>>>>>> class >>>>>>>>>>>>>>>>> for SQL jobs. >>>>>>>>>>>>>>>>> Please see the FLIP docs for details and feel free to >>>> comment. >>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>> >>>> >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316%3A+Introduce+SQL+Driver >>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>> >>>>>>>> >>>>>> >>>> >>> >> https://cwiki.apache.org/confluence/display/FLINK/FLIP-316:+Introduce+SQL+Driver >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [2] https://issues.apache.org/jira/browse/FLINK-26541 < >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/FLINK-26541> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>> Paul Lam >>>> >>>> >>> >>
