Re: [VOTE] Flink minor version support policy for old releases

[Jingsong Li]

+1 (binding)

On Thu, Mar 2, 2023 at 1:30 PM Yu Li  wrote:
>
> +1 (binding)
>
> Best Regards,
> Yu
>
>
> On Thu, 2 Mar 2023 at 09:53, Jark Wu  wrote:
>
> > +1 (binding)
> >
> > Best,
> > Jark
> >
> > > 2023年3月2日 05:03，Gyula Fóra  写道：
> > >
> > > +1 (binding)
> > >
> > > Gyula
> > >
> > > On Wed, Mar 1, 2023 at 9:57 PM Thomas Weise  wrote:
> > >
> > >> +1 (binding)
> > >>
> > >> Thanks,
> > >> Thomas
> > >>
> > >> On Tue, Feb 28, 2023 at 6:53 AM Sergey Nuyanzin 
> > >> wrote:
> > >>
> > >>> +1 (non-binding)
> > >>>
> > >>> Thanks for driving this Danny.
> > >>>
> > >>> On Tue, Feb 28, 2023 at 9:41 AM Samrat Deb 
> > >> wrote:
> > >>>
> >  +1 (non binding)
> > 
> >  Thanks for driving it
> > 
> >  Bests,
> >  Samrat
> > 
> >  On Tue, 28 Feb 2023 at 1:36 PM, Junrui Lee 
> > >> wrote:
> > 
> > > Thanks Danny for driving it.
> > >
> > > +1 (non-binding)
> > >
> > > Best regards,
> > > Junrui
> > >
> > > yuxia  于2023年2月28日周二 14:04写道：
> > >
> > >> Thanks Danny for driving it.
> > >>
> > >> +1 (non-binding)
> > >>
> > >> Best regards,
> > >> Yuxia
> > >>
> > >> - 原始邮件 -
> > >> 发件人: "Weihua Hu" 
> > >> 收件人: "dev" 
> > >> 发送时间: 星期二, 2023年 2 月 28日 下午 12:48:09
> > >> 主题: Re: [VOTE] Flink minor version support policy for old releases
> > >>
> > >> Thanks, Danny.
> > >>
> > >> +1 (non-binding)
> > >>
> > >> Best,
> > >> Weihua
> > >>
> > >>
> > >> On Tue, Feb 28, 2023 at 12:38 PM weijie guo <
> > >>> guoweijieres...@gmail.com
> > >
> > >> wrote:
> > >>
> > >>> Thanks Danny for bring this.
> > >>>
> > >>> +1 (non-binding)
> > >>>
> > >>> Best regards,
> > >>>
> > >>> Weijie
> > >>>
> > >>>
> > >>> Jing Ge  于2023年2月27日周一 20:23写道：
> > >>>
> >  +1 (non-binding)
> > 
> >  BTW, should we follow the content style [1] to describe the new
> >  rule
> > >>> using
> >  1.2.x, 1.1.y, 1.1.z?
> > 
> >  [1]
> > > https://flink.apache.org/downloads/#update-policy-for-old-releases
> > 
> >  Best regards,
> >  Jing
> > 
> >  On Mon, Feb 27, 2023 at 1:06 PM Matthias Pohl
> >   wrote:
> > 
> > > Thanks, Danny. Sounds good to me.
> > >
> > > +1 (non-binding)
> > >
> > > On Wed, Feb 22, 2023 at 10:11 AM Danny Cranmer <
> > >>> dannycran...@apache.org>
> > > wrote:
> > >
> > >> I am starting a vote to update the "Update Policy for old
> > > releases"
> > >>> [1]
> > > to
> > >> include additional bugfix support for end of life versions.
> > >>
> > >> As per the discussion thread [2], the change we are voting
> > >> on
> >  is:
> > >> - Support policy: updated to include: "Upon release of a
> > >> new
> > > Flink
> >  minor
> > >> version, the community will perform one final bugfix
> > >> release
> >  for
> >  resolved
> > >> critical/blocker issues in the Flink minor version losing
> > > support."
> > >> - Release process: add a step to start the discussion
> > >> thread
> >  for
> > >> the
> > > final
> > >> patch version, if there are resolved critical/blocking
> > >> issues
> >  to
> > >>> flush.
> > >>
> > >> Voting schema: since our bylaws [3] do not cover this
> >  particular
> > > scenario,
> > >> and releases require PMC involvement, we will use a
> > >> consensus
> > > vote
> > >>> with
> > > PMC
> > >> binding votes.
> > >>
> > >> Thanks,
> > >> Danny
> > >>
> > >> [1]
> > >
> > >>
> > >>> https://flink.apache.org/downloads.html#update-policy-for-old-releases
> > >> [2]
> > >> https://lists.apache.org/thread/szq23kr3rlkm80rw7k9n95js5vqpsnbv
> > >> [3]
> > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
> > >>
> > >
> > 
> > >>>
> > >>
> > >
> > 
> > >>>
> > >>>
> > >>> --
> > >>> Best regards,
> > >>> Sergey
> > >>>
> > >>
> >
> >

[DISCUSS] PRs in flink-ci/flink-mirror

[Matthias Pohl]

Hi everyone,
Weihua Hu [1] notified me of a section in Flink's Azure Pipeline
documentation [2] where it's suggested to create PRs against
flink-ci/mirror as a workaround if you're not having a private Azure
Pipeline account and want to run CI with your code changes. Even though
it's a viable solution it does generate noise in the Slack channel for
build failure (#builds). Additionally, I don't see any extra value in
comparison to using your own fork and creating a PR on your branch against
the apache/flink repo. CI will be picked up by Flink's CiBot. Or am I
missing something?

I couldn't find any ML discussion on that matter. I suggest removing this
section in the docs.

Best,
Matthias

[1] https://github.com/flink-ci/flink-mirror/pull/16#issuecomment-1451183958
[2]
https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-AzurePipelineUsageRestrictions

-- 

[image: Aiven] 

*Matthias Pohl*
Opensource Software Engineer, *Aiven*
matthias.p...@aiven.io|  +49 170 9869525
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [DISCUSS] FLIP-299 Pub/Sub Lite Connector

[Ryan Skraba]

Hello Daniel!  Quite a while ago, I started porting the Pub/Sub connector
(from an existing PR) to the new source API in the new
flink-connector-gcp-pubsub repository [PR2].  As Martijn mentioned, there
hasn't been a lot of attention on this connector; any community involvement
would be appreciated!

Instead of considering this a new connector, is there an opportunity here
to offer the two variants (Pub/Sub and Pub/Sub Lite) as different artifacts
in that same repo?  Is there much common logic that can be shared between
the two?  I'm not as familiar as I should be with Lite, but I do recall
that they share many concepts and _some_ dependencies.

All my best, Ryan


On Wed, Mar 1, 2023 at 11:21 PM Daniel Collins 
wrote:

> Hello all,
>
> I'd like to start an official discuss thread for adding a Pub/Sub Lite
> Connector to Flink. We've had requests from our users to add flink support,
> and are willing to maintain and support this connector long term from the
> product team.
>
> The proposal is https://cwiki.apache.org/confluence/x/P51bDg, what would
> be
> people's thoughts on adding this connector?
>
> -Daniel
>

Re: [VOTE] Release 1.15.4, release candidate #1

[Yu Li]

+1 (binding)


- Checked the diff between 1.15.3 and 1.15.4-rc1: *OK* (
https://github.com/apache/flink/compare/release-1.15.3...release-1.15.4-rc1)

  - AWS SDKv2 version has been bumped to 2.19.14 through FLINK-30633 and
all NOTICE files updated correctly

- Checked release notes: *OK*

- Checked sums and signatures: *OK*

- Maven clean install from source: *OK* (8u181)

- Checked the jars in the staging repo: *OK*

- Checked the website updates: *OK*


Thanks for driving this release, Danny!


Best Regards,
Yu


On Wed, 1 Mar 2023 at 02:01, Ahmed Hamdy  wrote:

> Thanks Danny,
>
> +1 (non-binding)
>
> - Verified hashes and signatures
> - Built Source archive using maven
> - Web PR looks good.
> - Started WordCount Example
>
> On Tue, 28 Feb 2023 at 16:37, Jing Ge  wrote:
>
> > Thanks Danny,
> >
> > +1 (non-binding)
> >
> >  - GPG signatures looks good
> >  - checked dist and maven repo
> >  - maven clean install from source
> >  - checked version consistency in pom files
> >  - went through the web release notes and found one task is still open:
> > FLINK-31133 [1]
> >  - download artifacts
> >  - started/stopped local cluster and ran WordCount job in streaming and
> > batch
> >
> > Best regards,
> > Jing
> >
> > [1] https://issues.apache.org/jira/browse/FLINK-31133
> >
> > On Tue, Feb 28, 2023 at 3:12 PM Matthias Pohl
> >  wrote:
> >
> > > Thanks Danny.
> > >
> > > +1 (non-binding)
> > >
> > > * Downloaded artifacts
> > > * Built Flink from sources
> > > * Verified SHA512 checksums GPG signatures
> > > * Compared checkout with provided sources
> > > * Verified pom file versions
> > > * Went over NOTICE file/pom files changes without finding anything
> > > suspicious
> > > * Deployed standalone session cluster and ran WordCount example in
> batch
> > > and streaming: Nothing suspicious in log files found
> > >
> > > On Tue, Feb 28, 2023 at 9:50 AM Teoh, Hong
>  > >
> > > wrote:
> > >
> > > > Thanks Danny for driving this
> > > >
> > > > +1 (non-binding)
> > > >
> > > > * Hashes and Signatures look good
> > > > * All required files on dist.apache.org
> > > > * Source archive builds using maven
> > > > * Started packaged example WordCountSQLExample job
> > > > * Web PR looks good.
> > > >
> > > > Cheers,
> > > > Hong
> > > >
> > > >
> > > >
> > > > > On 24 Feb 2023, at 05:36, Weihua Hu 
> wrote:
> > > > >
> > > > > CAUTION: This email originated from outside of the organization. Do
> > not
> > > > click links or open attachments unless you can confirm the sender and
> > > know
> > > > the content is safe.
> > > > >
> > > > >
> > > > >
> > > > > Thanks Danny.
> > > > >
> > > > > +1(non-binding)
> > > > >
> > > > > Tested the following:
> > > > > - Download the artifacts and build image
> > > > > - Ran WordCount on Kubernetes(session mode and application mode)
> > > > >
> > > > >
> > > > > Best,
> > > > > Weihua
> > > > >
> > > > >
> > > > > On Fri, Feb 24, 2023 at 12:29 PM Yanfei Lei 
> > > wrote:
> > > > >
> > > > >> Thanks Danny.
> > > > >> +1 (non-binding)
> > > > >>
> > > > >> - Downloaded artifacts & built Flink from sources
> > > > >> - Verified GPG signatures of bin and source.
> > > > >> - Verified version in poms
> > > > >> - Ran WordCount example in streaming and batch mode(standalone
> > > cluster)
> > > > >> - Went over flink-web PR, looks good except for Sergey's remark.
> > > > >>
> > > > >> Danny Cranmer  于2023年2月24日周五 02:08写道：
> > > > >>>
> > > > >>> Hi everyone,
> > > > >>> Please review and vote on the release candidate #1 for the
> version
> > > > >> 1.15.4,
> > > > >>> as follows:
> > > > >>> [ ] +1, Approve the release
> > > > >>> [ ] -1, Do not approve the release (please provide specific
> > comments)
> > > > >>>
> > > > >>>
> > > > >>> The complete staging area is available for your review, which
> > > includes:
> > > > >>> * JIRA release notes [1],
> > > > >>> * the official Apache source release and binary convenience
> > releases
> > > to
> > > > >> be
> > > > >>> deployed to dist.apache.org [2], which are signed with the key
> > with
> > > > >>> fingerprint 125FD8DB [3],
> > > > >>> * all artifacts to be deployed to the Maven Central Repository
> [4],
> > > > >>> * source code tag "release-1.15.4-rc1" [5],
> > > > >>> * website pull request listing the new release and adding
> > > announcement
> > > > >> blog
> > > > >>> post [6].
> > > > >>>
> > > > >>> The vote will be open for at least 72 hours (excluding weekends
> > > > >> 2023-02-28
> > > > >>> 19:00). It is adopted by majority approval, with at least 3 PMC
> > > > >> affirmative
> > > > >>> votes.
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Danny
> > > > >>>
> > > > >>> [1]
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12352526
> > > > >>> [2]
> https://dist.apache.org/repos/dist/dev/flink/flink-1.15.4-rc1/
> > > > >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS
> > > > >>> [4]
> > > > >>>
> > > > >>
> > > >
> > >
> >
> https://re

[jira] [Created] (FLINK-31297) FineGrainedSlotManagerTest.testTaskManagerRegistrationDeductPendingTaskManager untable when running it a single time

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-31297:
-

 Summary: 
FineGrainedSlotManagerTest.testTaskManagerRegistrationDeductPendingTaskManager 
untable when running it a single time
 Key: FLINK-31297
 URL: https://issues.apache.org/jira/browse/FLINK-31297
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.17.0
Reporter: Matthias Pohl


We noticed a weird test-instability in 
{{FineGrainedSlotManagerTest.testTaskManagerRegistrationDeductPendingTaskManager}}
 when switching to sequential test execution (see FLINK-31278). I couldn't 
reproduce it in 1.16, therefore, marking it as a blocker for now. But it feels 
to be more of a test code issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31298) ConnectionUtilsTest.testFindConnectingAddressWhenGetLocalHostThrows swallows IllegalArgumentException

[Matthias Pohl (Jira)]

Matthias Pohl created FLINK-31298:
-

 Summary: 
ConnectionUtilsTest.testFindConnectingAddressWhenGetLocalHostThrows swallows 
IllegalArgumentException
 Key: FLINK-31298
 URL: https://issues.apache.org/jira/browse/FLINK-31298
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Network
Affects Versions: 1.16.1, 1.15.3, 1.17.0
Reporter: Matthias Pohl


FLINK-24156 introduced {{NetUtils.acceptWithoutTimeout}} which caused the test 
to print a the stacktrace of an {{IllegalArgumentException}}:
{code}
Exception in thread "Thread-0" java.lang.IllegalArgumentException: serverSocket 
SO_TIMEOUT option must be 0
at 
org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138)
at 
org.apache.flink.util.NetUtils.acceptWithoutTimeout(NetUtils.java:139)
at 
org.apache.flink.runtime.net.ConnectionUtilsTest$1.run(ConnectionUtilsTest.java:83)
at java.lang.Thread.run(Thread.java:750)
{code}

This is also shown in the Maven output of CI runs and might cause confusion. 
The test should be fixed.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

[Shammon FY]

Hi

Thanks for the feedback from Jingsong and Benchao.

For @Jingsong
> If the user does not cast into a FlinkResultSet, will there be
serious consequences here (RowKind is ignored)?

I agree with you that it's indeed a big deal if users ignore the row kind
when they must know it. One idea that comes to my mind is we can add an
option such as `table.result.changelog-mode`, users can set it through
connection properties or set dynamic parameters. The option value can be
`insert-only`, `upset` or `all` and the default value is `insert-only`.

If the result does not conform to the changelog mode, the jdbc driver
throws an exception. What do you think?


For @Benchao
> Besides `java.sql.Driver`, have you considered also adding support for
`javax.sql.DataSource` interface?

I missed the `javax.sql.DataSource` and I have added it to the FLIP, thanks
Benchao


Best,
Shammon

On Wed, Mar 1, 2023 at 7:57 PM Benchao Li  wrote:

> +1 for the FLIP, thanks Shammon for driving this.
>
> JDBC is quite useful in OLAP scenarios, supporting JDBC would enable Flink
> to be used with existing tools, such as Tableau.
>
> Regarding the JDBC interfaces listed in the FLIP, I think they looks good
> already. Besides `java.sql.Driver`, have you considered also adding support
> for `javax.sql.DataSource` interface?
>
> Jingsong Li  于2023年3月1日周三 17:53写道：
>
> > Thanks Shammon for driving.
> >
> > Big +1 for this.
> >
> > I heard that many users want to use FlinkGateway + JDBC to do some
> > queries, but at present, only Hive JDBC can be used. It is Hive
> > dialect by default, and the experience is also different from
> > FlinkSQL. We need to have our own JDBC.
> >
> > I took a look at your `Public Interface` part, only
> > `FlinkResultSet.getRowKind` is a true new interface, others are just
> > implementations.
> >
> > If the user does not cast into a FlinkResultSet, will there be serious
> > consequences here (RowKind is ignored)?
> >
> > Best,
> > Jingsong
> >
> > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY  wrote:
> > >
> > > Hi devs,
> > >
> > > I'd like to start a discussion about FLIP-293: Introduce Flink Jdbc
> > Driver
> > > For Sql Gateway[1].
> > >
> > > FLIP-275[2] supports remote sql client based on gateway, users can
> > interact
> > > with gateway by flink console. However, for users who create session
> > > clusters with Flink, they'd like to use Jdbc Driver to interact with
> the
> > > gateway in their applications, such as olap queries..
> > >
> > > I have discussed this proposal with @shengkaifang and @jinsonglee. In
> > this
> > > FLIP, we'd like to introduce Jdbc Driver for gateway. Users can use
> Jdbc
> > > Driver to submit their queries and get results like a database in their
> > > applications.
> > >
> > > Looking forward to your feedback, thanks.
> > >
> > >
> > > [1]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway
> > > [2]
> > >
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway
> > >
> > >
> > > Best,
> > > Shammon
> >
>
>
> --
>
> Best,
> Benchao Li
>

[jira] [Created] (FLINK-31299) PendingRecords metric might not be available

[Maximilian Michels (Jira)]

Maximilian Michels created FLINK-31299:
--

 Summary: PendingRecords metric might not be available
 Key: FLINK-31299
 URL: https://issues.apache.org/jira/browse/FLINK-31299
 Project: Flink
  Issue Type: Bug
  Components: Autoscaler, Kubernetes Operator
Reporter: Maximilian Michels
Assignee: Maximilian Michels
 Fix For: kubernetes-operator-1.5.0


The Kafka pendingRecords metric is only initialized on receiving the first 
record. For empty topics or checkpointed topics without any incoming data, the 
metric won't appear.

We need to handle this case in the autoscaler and allow downscaling.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-299 Pub/Sub Lite Connector

[Daniel Collins]

Hello Ryan,

Unfortunately there's not much shared logic between the two- the clients
have to look fundamentally different since the Pub/Sub Lite client exposes
partitions to the split level for repeatable reads.

I have no objection to this living in the same repo as the Pub/Sub
connector, if this is an easier way forward than setting up a new repo,
sounds good to me. The Pub/Sub team is organizationally close to us, and is
looking into providing more support for the flink connector in the near
future.

-Daniel

On Thu, Mar 2, 2023 at 3:26 AM Ryan Skraba 
wrote:

> Hello Daniel!  Quite a while ago, I started porting the Pub/Sub connector
> (from an existing PR) to the new source API in the new
> flink-connector-gcp-pubsub repository [PR2].  As Martijn mentioned, there
> hasn't been a lot of attention on this connector; any community involvement
> would be appreciated!
>
> Instead of considering this a new connector, is there an opportunity here
> to offer the two variants (Pub/Sub and Pub/Sub Lite) as different artifacts
> in that same repo?  Is there much common logic that can be shared between
> the two?  I'm not as familiar as I should be with Lite, but I do recall
> that they share many concepts and _some_ dependencies.
>
> All my best, Ryan
>
>
> On Wed, Mar 1, 2023 at 11:21 PM Daniel Collins
> 
> wrote:
>
> > Hello all,
> >
> > I'd like to start an official discuss thread for adding a Pub/Sub Lite
> > Connector to Flink. We've had requests from our users to add flink
> support,
> > and are willing to maintain and support this connector long term from the
> > product team.
> >
> > The proposal is https://cwiki.apache.org/confluence/x/P51bDg, what would
> > be
> > people's thoughts on adding this connector?
> >
> > -Daniel
> >
>

[jira] [Created] (FLINK-31300) TRY_CAST fails for constructed types

[Sergey Nuyanzin (Jira)]

Sergey Nuyanzin created FLINK-31300:
---

 Summary: TRY_CAST fails for constructed types
 Key: FLINK-31300
 URL: https://issues.apache.org/jira/browse/FLINK-31300
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.16.1, 1.17.0
Reporter: Sergey Nuyanzin


In case of problems with cast it is expected to return {{null}}

however for arrays, maps it fails

example of failing queries
{code:sql}
select try_cast(array['a'] as array);
select try_cast(map['a', '1'] as map);
{code}

 {noformat}
[ERROR] Could not execute SQL statement. Reason:
java.lang.NumberFormatException: For input string: 'a'. Invalid character found.
at 
org.apache.flink.table.data.binary.BinaryStringDataUtil.numberFormatExceptionFor(BinaryStringDataUtil.java:585)
at 
org.apache.flink.table.data.binary.BinaryStringDataUtil.toInt(BinaryStringDataUtil.java:518)
at StreamExecCalc$15.processElement(Unknown Source)
at 
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:82)
at 
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
at 
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
at 
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
at 
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
at 
org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:418)
at 
org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:513)
at 
org.apache.flink.streaming.api.operators.StreamSourceContexts$SwitchingOnClose.collect(StreamSourceContexts.java:103)
at 
org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:92)
at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
at 
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:67)
at 
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:333)
{noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31301) Unsupported nested columns in column list of insert statement

[lincoln lee (Jira)]

lincoln lee created FLINK-31301:
---

 Summary: Unsupported nested columns in column list of insert 
statement
 Key: FLINK-31301
 URL: https://issues.apache.org/jira/browse/FLINK-31301
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / API
Affects Versions: 1.16.1, 1.17.0
Reporter: lincoln lee


Currently an error will be raised when use nested columns in column list of 
insert statement, e.g.,

{code}
INSERT INTO nested_type_sink (a,b.b1,c.c2,f)
SELECT a,b.b1,c.c2,f FROM complex_type_src
{code}

 

{code}

java.lang.AssertionError
    at org.apache.calcite.sql.SqlIdentifier.getSimple(SqlIdentifier.java:333)
    at 
org.apache.calcite.sql.validate.SqlValidatorUtil.getTargetField(SqlValidatorUtil.java:612)
    at 
org.apache.flink.table.planner.calcite.PreValidateReWriter$.$anonfun$appendPartitionAndNullsProjects$3(PreValidateReWriter.scala:171)
    at 
scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:233)
    at scala.collection.Iterator.foreach(Iterator.scala:937)
    at scala.collection.Iterator.foreach$(Iterator.scala:937)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1425)
    at scala.collection.IterableLike.foreach(IterableLike.scala:70)
    at scala.collection.IterableLike.foreach$(IterableLike.scala:69)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike.map(TraversableLike.scala:233)
    at scala.collection.TraversableLike.map$(TraversableLike.scala:226)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at 
org.apache.flink.table.planner.calcite.PreValidateReWriter$.appendPartitionAndNullsProjects(PreValidateReWriter.scala:164)
    at 
org.apache.flink.table.planner.calcite.PreValidateReWriter.rewriteInsert(PreValidateReWriter.scala:71)
    at 
org.apache.flink.table.planner.calcite.PreValidateReWriter.visit(PreValidateReWriter.scala:61)
    at 
org.apache.flink.table.planner.calcite.PreValidateReWriter.visit(PreValidateReWriter.scala:50)
    at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:161)
    at 
org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:118)
    at 
org.apache.flink.table.planner.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:113)
    at 
org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:281)
    at 
org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:106)
    at 
org.apache.flink.table.api.internal.StatementSetImpl.addInsertSql(StatementSetImpl.java:63)

{code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31302) Split spark modules according to version

[yuzelin (Jira)]

yuzelin created FLINK-31302:
---

 Summary: Split spark modules according to version
 Key: FLINK-31302
 URL: https://issues.apache.org/jira/browse/FLINK-31302
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Affects Versions: table-store-0.4.0
Reporter: yuzelin






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31303) k8s operator should gather job cpu and memory utilization metrics

[Jira]

Márton Balassi created FLINK-31303:
--

 Summary: k8s operator should gather job cpu and memory utilization 
metrics
 Key: FLINK-31303
 URL: https://issues.apache.org/jira/browse/FLINK-31303
 Project: Flink
  Issue Type: New Feature
  Components: Kubernetes Operator, Runtime / Metrics
Affects Versions: kubernetes-operator-1.5.0
Reporter: Márton Balassi
Assignee: Márton Balassi


We should extend the operator metrics system to gather this additional 
information and be able to expose it.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31304) Very slow job start if topic has been used before

[Yordan Pavlov (Jira)]

Yordan Pavlov created FLINK-31304:
-

 Summary: Very slow job start if topic has been used before
 Key: FLINK-31304
 URL: https://issues.apache.org/jira/browse/FLINK-31304
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Affects Versions: 1.15.2
Reporter: Yordan Pavlov


We have the following use case. We use KafkaSink with Exactly once semantic, 
from time to time we would re-start the job clean, in doing so we delete and 
re-create the output topic and also any Flink checkpoints. In such situation it 
would take close to an hour for Flink to start. In the the time the job is 
idling we would see the following log in the Taskmanager:


{code:java}
2023-03-02 16:33:42.004 [Source: Kafka source blocks -> Deduplicate blocks -> 
Map -> Parse blocks -> Map -> Kafka sink volume: Writer -> Kafka sink volume: 
Committer (2/5)#0] INFO  
o.apache.kafka.clients.producer.internals.TransactionManager  - [Producer 
clientId=producer-state.clickhouse-0-1-1, 
transactionalId=state.clickhouse-0-1-1] Invoking InitProducerId for the first 
time in order to acquire a producer ID
2023-03-02 16:33:42.005 [kafka-producer-network-thread | 
producer-state.clickhouse-0-2-1] INFO  
o.apache.kafka.clients.producer.internals.TransactionManager  - [Producer 
clientId=producer-state.clickhouse-0-2-1, 
transactionalId=state.clickhouse-0-2-1] ProducerId set to 31719488 with epoch 
8{code}

If we use a brand new output topic name, the job would start straight away. 
Could you advise if this can be improved?

Such logs would go on and on in what seems forever.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Flink minor version support policy for old releases

[Danny Cranmer]

Hi everyone,

This vote thread is now closed. We have reached a consensus with 6
binding votes, 8 non-binding votes and no vetoes, I will follow up and
apply the changes.

+1s
- Matthias
- Jing
- Weijie
- Weihua
- Yuxia
- Junrui
- Samrat
- Sergey
- Thomas (binding)
- Gyula (binding)
- Jark (binding)
- Yu (binding)
- Jingsong (binding)
- Danny (binding)

Thanks,
Danny

On Thu, Mar 2, 2023 at 8:06 AM Jingsong Li  wrote:

> +1 (binding)
>
> On Thu, Mar 2, 2023 at 1:30 PM Yu Li  wrote:
> >
> > +1 (binding)
> >
> > Best Regards,
> > Yu
> >
> >
> > On Thu, 2 Mar 2023 at 09:53, Jark Wu  wrote:
> >
> > > +1 (binding)
> > >
> > > Best,
> > > Jark
> > >
> > > > 2023年3月2日 05:03，Gyula Fóra  写道：
> > > >
> > > > +1 (binding)
> > > >
> > > > Gyula
> > > >
> > > > On Wed, Mar 1, 2023 at 9:57 PM Thomas Weise  wrote:
> > > >
> > > >> +1 (binding)
> > > >>
> > > >> Thanks,
> > > >> Thomas
> > > >>
> > > >> On Tue, Feb 28, 2023 at 6:53 AM Sergey Nuyanzin <
> snuyan...@gmail.com>
> > > >> wrote:
> > > >>
> > > >>> +1 (non-binding)
> > > >>>
> > > >>> Thanks for driving this Danny.
> > > >>>
> > > >>> On Tue, Feb 28, 2023 at 9:41 AM Samrat Deb 
> > > >> wrote:
> > > >>>
> > >  +1 (non binding)
> > > 
> > >  Thanks for driving it
> > > 
> > >  Bests,
> > >  Samrat
> > > 
> > >  On Tue, 28 Feb 2023 at 1:36 PM, Junrui Lee 
> > > >> wrote:
> > > 
> > > > Thanks Danny for driving it.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > Best regards,
> > > > Junrui
> > > >
> > > > yuxia  于2023年2月28日周二 14:04写道：
> > > >
> > > >> Thanks Danny for driving it.
> > > >>
> > > >> +1 (non-binding)
> > > >>
> > > >> Best regards,
> > > >> Yuxia
> > > >>
> > > >> - 原始邮件 -
> > > >> 发件人: "Weihua Hu" 
> > > >> 收件人: "dev" 
> > > >> 发送时间: 星期二, 2023年 2 月 28日 下午 12:48:09
> > > >> 主题: Re: [VOTE] Flink minor version support policy for old
> releases
> > > >>
> > > >> Thanks, Danny.
> > > >>
> > > >> +1 (non-binding)
> > > >>
> > > >> Best,
> > > >> Weihua
> > > >>
> > > >>
> > > >> On Tue, Feb 28, 2023 at 12:38 PM weijie guo <
> > > >>> guoweijieres...@gmail.com
> > > >
> > > >> wrote:
> > > >>
> > > >>> Thanks Danny for bring this.
> > > >>>
> > > >>> +1 (non-binding)
> > > >>>
> > > >>> Best regards,
> > > >>>
> > > >>> Weijie
> > > >>>
> > > >>>
> > > >>> Jing Ge  于2023年2月27日周一 20:23写道：
> > > >>>
> > >  +1 (non-binding)
> > > 
> > >  BTW, should we follow the content style [1] to describe the
> new
> > >  rule
> > > >>> using
> > >  1.2.x, 1.1.y, 1.1.z?
> > > 
> > >  [1]
> > > >
> https://flink.apache.org/downloads/#update-policy-for-old-releases
> > > 
> > >  Best regards,
> > >  Jing
> > > 
> > >  On Mon, Feb 27, 2023 at 1:06 PM Matthias Pohl
> > >   wrote:
> > > 
> > > > Thanks, Danny. Sounds good to me.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > On Wed, Feb 22, 2023 at 10:11 AM Danny Cranmer <
> > > >>> dannycran...@apache.org>
> > > > wrote:
> > > >
> > > >> I am starting a vote to update the "Update Policy for old
> > > > releases"
> > > >>> [1]
> > > > to
> > > >> include additional bugfix support for end of life versions.
> > > >>
> > > >> As per the discussion thread [2], the change we are voting
> > > >> on
> > >  is:
> > > >> - Support policy: updated to include: "Upon release of a
> > > >> new
> > > > Flink
> > >  minor
> > > >> version, the community will perform one final bugfix
> > > >> release
> > >  for
> > >  resolved
> > > >> critical/blocker issues in the Flink minor version losing
> > > > support."
> > > >> - Release process: add a step to start the discussion
> > > >> thread
> > >  for
> > > >> the
> > > > final
> > > >> patch version, if there are resolved critical/blocking
> > > >> issues
> > >  to
> > > >>> flush.
> > > >>
> > > >> Voting schema: since our bylaws [3] do not cover this
> > >  particular
> > > > scenario,
> > > >> and releases require PMC involvement, we will use a
> > > >> consensus
> > > > vote
> > > >>> with
> > > > PMC
> > > >> binding votes.
> > > >>
> > > >> Thanks,
> > > >> Danny
> > > >>
> > > >> [1]
> > > >
> > > >>
> > > >>>
> https://flink.apache.org/downloads.html#update-policy-for-old-releases
> > > >> [2]
> > > >>
> https://lists.apache.org/thread/szq23kr3rlkm80rw7k9n95js5vqpsnbv
> > > >> [3]
> > > > https://cwiki.apache.org/confluence/display/FLINK/Flink+Bylaws
> > > >>
> > > >
> > > 
> > > >>>
> > 

[jira] [Created] (FLINK-31305) KafkaWriter doesn't wait for errors for in-flight records before completing flush

[Mason Chen (Jira)]

Mason Chen created FLINK-31305:
--

 Summary: KafkaWriter doesn't wait for errors for in-flight records 
before completing flush
 Key: FLINK-31305
 URL: https://issues.apache.org/jira/browse/FLINK-31305
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Kafka
Affects Versions: 1.16.1, 1.17.0
Reporter: Mason Chen
 Fix For: 1.17.0


The KafkaWriter flushing needs to wait for all in-flight records to send 
successfully. This can be achieved by tracking requests and returning a 
response from the registered callback from the producer#send() logic.

There is potential for data loss since the checkpoint does not accurately 
reflect that all records have been sent successfully, to preserve at least once 
semantics.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL

[Jing Ge]

Hi,

Thanks Kui for driving this Flip and thanks all for the informative
discussion.

@Timo

Your suggestion about the naming convention is excellent. Thanks! I was
wondering why you, exceptionally, suggested 'scan.idle-timeout' instead of
'scan.watermark.idle-timeout'. I must miss something here.

There is one more NIT. I am just aware that "drift" is used for the
watermark alignment. It seems to be fine while using DataStream API,
because we will not really see it. But with the OPTIONS in SQL, a much
bigger group of users (including SRE, tech support, etc) will see the word
"drift". Given that "drift" wasn't used widely yet and with all training
materials, Flink doc [1][2][3] (search with "lag"), "lag" has been used to
describe timestamp difference between watermark and its
corresponding event. Do we really need to introduce another term for the
same thing? How about using 'scan.watermark.alignment.max-lag'='1min' and
change the parameter name from maxAllowedWatermarkDrift to
maxAllowedWatermarkLag [4] because of naming consistency? Just my two cents
worth.

@Kui

After reading the most up-to-date Flip, I didn't find any information if
this solution will support aligning splits/partitions/shards [1]. Did I
miss anything?

+1 for the concern about Table API. We'd be better keep Table API and SQL
synced for new features.

Best regards,
Jing


[1]
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/datastream/event-time/generating_watermarks/#watermark-alignment-_beta_
[2]
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/datastream/event-time/built_in/#fixed-amount-of-lateness

[3]
https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/kafka/
[4]
https://github.com/apache/flink/blob/4aacff572a9e3996c5dee9273638831e4040c767/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarkStrategy.java#L169



On Wed, Mar 1, 2023 at 3:54 PM Timo Walther  wrote:

> Reg. 2:
>  > event gap emit strategy [...] no matter in DataStream or SQL
>
> Jark raised a very good point. I thought we only expose what is
> contained in DataStream API already. If this strategy is not part of
> DataStream API, would like to exclude it from the FLIP. We need to be
> careful which strategies we offer by default.
>
> Reg 1:
> This already has a JIRA ticket with additional thoughts on this topic:
> https://issues.apache.org/jira/browse/FLINK-25221
>
> Regards,
> Timo
>
>
>
> On 01.03.23 12:31, Jark Wu wrote:
> > Sorry, I forgot to remind you that Timo's concern about the changes to
> the
> > CompiledPlan looks like is still not covered in the FLIP.
> >
> > Best,
> > Jark
> >
> > On Wed, 1 Mar 2023 at 19:28, Jark Wu  wrote:
> >
> >> Hi Kui,
> >>
> >> Thank you for the great proposal, I think this is already in a good
> shape.
> >>
> >> Just a kind reminder, according to the community guidelines[1],
> >> if there are unresponsive reviewers, a typical reasonable time
> >> to wait for responses is one week, but be pragmatic about it.
> >>
> >> Regarding the FLIP, I have some comments below:
> >>
> >> 1. IIRC, this is the first time we introduce the framework-level
> connector
> >> options that the option is not recognized and handled by connectors.
> >> The FLIP should cover how framework filters the watermark related
> options
> >> to avoid discover connector factory failed, and what happens if the
> >> connector
> >> already supported the conflict options.
> >>
> >> 2. I'm not sure about the usage scenarios of event gap emit strategy. Do
> >> you have any specific use case of this strategy? I'm confused why no one
> >> requested this strategy before no matter in DataStream or SQL, but maybe
> >> I missed something. I'm not against to add this option, but just want to
> >> be
> >> careful when adding new API because it's hard to remove in the future.
> >>
> >>
> >> 3. Adding a "Public Interface"[2] section to summarize the
> >> proposed APIs and options would be better for developers to
> >> know the impact. Currently, the APIs are scattered in the long
> >> design sections.
> >>
> >> Best,
> >> Jark
> >>
> >>
> >> [1]:
> >>
> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals
> >> [2]: https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template
> >>
> >> On Wed, 1 Mar 2023 at 16:56, Kui Yuan  wrote:
> >>
> >>> Hi all,
> >>>
> >>> Thanks for all discussions!
> >>>
> >>> Anyone else have questions or suggestions? if not, I will start a vote
> >>> thread later.
> >>>
> >>> Best
> >>> Kui Yuan
> >>>
> >>> kui yuan  于2023年2月27日周一 20:21写道：
> >>>
>  Hi Timo,
> 
>  Thanks for your advice. I totally agree with your suggestion of naming
>  convention, I will rename these options and update the flip later,
> >>> thanks
>  very much.
> 
>  In our internal implementation we had put these options inside the
>  `FactoryUtil`, just as you expect.  We have also taken into account
> the
>  changes to the CompiledPl

[jira] [Created] (FLINK-31306) Add Servable for PipelineModel

[Jiang Xin (Jira)]

Jiang Xin created FLINK-31306:
-

 Summary: Add Servable for PipelineModel
 Key: FLINK-31306
 URL: https://issues.apache.org/jira/browse/FLINK-31306
 Project: Flink
  Issue Type: Improvement
  Components: Library / Machine Learning
Reporter: Jiang Xin
 Fix For: ml-2.2.0


Add servable for PipelineModel based on flip-289.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] Release 1.15.4, release candidate #1

[Dian Fu]

Hi Danny,

I'm sorry that I'm coming to this thread a little late. It seems that this
will be the last bugfix release of Flink 1.15? If so, I'd like to also
include https://issues.apache.org/jira/browse/FLINK-31272 into this release
which fixes a serious issue of PyFlink.

Regards,
Dian



On Thu, Mar 2, 2023 at 5:51 PM Yu Li  wrote:

> +1 (binding)
>
>
> - Checked the diff between 1.15.3 and 1.15.4-rc1: *OK* (
> https://github.com/apache/flink/compare/release-1.15.3...release-1.15.4-rc1
> )
>
>   - AWS SDKv2 version has been bumped to 2.19.14 through FLINK-30633 and
> all NOTICE files updated correctly
>
> - Checked release notes: *OK*
>
> - Checked sums and signatures: *OK*
>
> - Maven clean install from source: *OK* (8u181)
>
> - Checked the jars in the staging repo: *OK*
>
> - Checked the website updates: *OK*
>
>
> Thanks for driving this release, Danny!
>
>
> Best Regards,
> Yu
>
>
> On Wed, 1 Mar 2023 at 02:01, Ahmed Hamdy  wrote:
>
> > Thanks Danny,
> >
> > +1 (non-binding)
> >
> > - Verified hashes and signatures
> > - Built Source archive using maven
> > - Web PR looks good.
> > - Started WordCount Example
> >
> > On Tue, 28 Feb 2023 at 16:37, Jing Ge 
> wrote:
> >
> > > Thanks Danny,
> > >
> > > +1 (non-binding)
> > >
> > >  - GPG signatures looks good
> > >  - checked dist and maven repo
> > >  - maven clean install from source
> > >  - checked version consistency in pom files
> > >  - went through the web release notes and found one task is still open:
> > > FLINK-31133 [1]
> > >  - download artifacts
> > >  - started/stopped local cluster and ran WordCount job in streaming and
> > > batch
> > >
> > > Best regards,
> > > Jing
> > >
> > > [1] https://issues.apache.org/jira/browse/FLINK-31133
> > >
> > > On Tue, Feb 28, 2023 at 3:12 PM Matthias Pohl
> > >  wrote:
> > >
> > > > Thanks Danny.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > * Downloaded artifacts
> > > > * Built Flink from sources
> > > > * Verified SHA512 checksums GPG signatures
> > > > * Compared checkout with provided sources
> > > > * Verified pom file versions
> > > > * Went over NOTICE file/pom files changes without finding anything
> > > > suspicious
> > > > * Deployed standalone session cluster and ran WordCount example in
> > batch
> > > > and streaming: Nothing suspicious in log files found
> > > >
> > > > On Tue, Feb 28, 2023 at 9:50 AM Teoh, Hong
> >  > > >
> > > > wrote:
> > > >
> > > > > Thanks Danny for driving this
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > * Hashes and Signatures look good
> > > > > * All required files on dist.apache.org
> > > > > * Source archive builds using maven
> > > > > * Started packaged example WordCountSQLExample job
> > > > > * Web PR looks good.
> > > > >
> > > > > Cheers,
> > > > > Hong
> > > > >
> > > > >
> > > > >
> > > > > > On 24 Feb 2023, at 05:36, Weihua Hu 
> > wrote:
> > > > > >
> > > > > > CAUTION: This email originated from outside of the organization.
> Do
> > > not
> > > > > click links or open attachments unless you can confirm the sender
> and
> > > > know
> > > > > the content is safe.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Thanks Danny.
> > > > > >
> > > > > > +1(non-binding)
> > > > > >
> > > > > > Tested the following:
> > > > > > - Download the artifacts and build image
> > > > > > - Ran WordCount on Kubernetes(session mode and application mode)
> > > > > >
> > > > > >
> > > > > > Best,
> > > > > > Weihua
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 24, 2023 at 12:29 PM Yanfei Lei  >
> > > > wrote:
> > > > > >
> > > > > >> Thanks Danny.
> > > > > >> +1 (non-binding)
> > > > > >>
> > > > > >> - Downloaded artifacts & built Flink from sources
> > > > > >> - Verified GPG signatures of bin and source.
> > > > > >> - Verified version in poms
> > > > > >> - Ran WordCount example in streaming and batch mode(standalone
> > > > cluster)
> > > > > >> - Went over flink-web PR, looks good except for Sergey's remark.
> > > > > >>
> > > > > >> Danny Cranmer  于2023年2月24日周五 02:08写道：
> > > > > >>>
> > > > > >>> Hi everyone,
> > > > > >>> Please review and vote on the release candidate #1 for the
> > version
> > > > > >> 1.15.4,
> > > > > >>> as follows:
> > > > > >>> [ ] +1, Approve the release
> > > > > >>> [ ] -1, Do not approve the release (please provide specific
> > > comments)
> > > > > >>>
> > > > > >>>
> > > > > >>> The complete staging area is available for your review, which
> > > > includes:
> > > > > >>> * JIRA release notes [1],
> > > > > >>> * the official Apache source release and binary convenience
> > > releases
> > > > to
> > > > > >> be
> > > > > >>> deployed to dist.apache.org [2], which are signed with the key
> > > with
> > > > > >>> fingerprint 125FD8DB [3],
> > > > > >>> * all artifacts to be deployed to the Maven Central Repository
> > [4],
> > > > > >>> * source code tag "release-1.15.4-rc1" [5],
> > > > > >>> * website pull request listing the new release and adding
> > > > announcement
> > > > > >> blog
> > > > > >>> post [6].
>

[jira] [Created] (FLINK-31307) RocksDB：java.lang.UnsatisfiedLinkError

[Wujunzhe (Jira)]

Wujunzhe created FLINK-31307:


 Summary: RocksDB：java.lang.UnsatisfiedLinkError
 Key: FLINK-31307
 URL: https://issues.apache.org/jira/browse/FLINK-31307
 Project: Flink
  Issue Type: Bug
Affects Versions: 1.14.5
Reporter: Wujunzhe
 Attachments: image-2023-03-03-10-27-04-810.png, 
image-2023-03-03-10-29-27-477.png

when i use rocksdb like 

!image-2023-03-03-10-27-04-810.png!

 

 I got an unsolvable exception. 

!image-2023-03-03-10-29-27-477.png!

 What can I do to troubleshoot or solve this problem? 

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31308) JobManager's metaspace out-of-memory when submit a flinksessionjobs

[tanjialiang (Jira)]

tanjialiang created FLINK-31308:
---

 Summary: JobManager's metaspace out-of-memory when submit a 
flinksessionjobs
 Key: FLINK-31308
 URL: https://issues.apache.org/jira/browse/FLINK-31308
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator, Table SQL / API
Affects Versions: kubernetes-operator-1.4.0, 1.16.1
Reporter: tanjialiang
 Attachments: image-2023-03-03-10-34-46-681.png

Hello teams, when i try to recurring submit a flinksessionjobs by flink 
operator, it will be make JobManager's metaspace OOM. My Job having some 
flink-sql logic, it is the userclassloader didn't closed? Or may be beacuase of 
flink-sql's codegen? By the way, it not appear when i using flink-sql-gateway 
to submit.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31309) Rollback DFS schema if hive sync fail in HiveCatalog.createTable

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-31309:


 Summary: Rollback DFS schema if hive sync fail in 
HiveCatalog.createTable
 Key: FLINK-31309
 URL: https://issues.apache.org/jira/browse/FLINK-31309
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Reporter: Jingsong Lee
 Fix For: table-store-0.4.0


Avoid schema residue on DFS.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31310) Force clear directory no matter what situation in HiveCatalog.dropTable

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-31310:


 Summary: Force clear directory no matter what situation in 
HiveCatalog.dropTable
 Key: FLINK-31310
 URL: https://issues.apache.org/jira/browse/FLINK-31310
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Reporter: Jingsong Lee
 Fix For: table-store-0.4.0


Currently, if no table in hive, will not clear the table.

We should clear table directory in any situation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] FLIP-300: Add targetColumns to DynamicTableSink#Context to solve the null overwrite problem of partial-insert

[Lincoln Lee]

Hi everyone,

This FLIP[1] aims to support connectors in avoiding overwriting non-target
columns with null values when processing partial column updates, we propose
adding information on the target column list to DynamicTableSink#Context.

FLINK-18726[2] supports inserting statements with specified column list, it
fills null values (or potentially declared default values in the future)
for columns not appearing in the column list of insert statement to the
target table.
But this behavior does not satisfy some partial column update requirements
of some storage systems which allow storing null values. The problem is
that connectors cannot distinguish whether the null value of a column is
really from the user's data or whether it is a null value populated because
of partial insert behavior.

Looking forward to your comments or feedback.

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240885081
[2] https://issues.apache.org/jira/browse/FLINK-18726

Best,
Lincoln Lee

Re: Re: [DISCUSS] Extract core autoscaling algorithm as new SubModule in flink-kubernetes-operator

[ramkrishna vasudevan]

Hi Samrat/Prabhu

My preliminary question here would be that though YARN is the platform for
Flink, but the yarn also runs over K8s ? Is that the reason why you wanted
the autoscale logic to be generic but inside the operator itself?

So if the case is that Yarn is the resource management, then the autoscale
logic it makes sense to be in the flink repo or some other repo for
autoscale specifically?

Regards
Ram



On Tue, Feb 21, 2023 at 7:49 AM ConradJam  wrote:

> +1
>
> Maximilian Michels  于2023年2月21日周二 00:21写道：
>
> > Great to see the interest here! I think the next step would be to
> > write a FLIP which explains how the autoscaler implementation would be
> > made agnostic to the resource management framework (k8s / yarn / etc).
> > There will have to be platform-agnostic abstractions and interfaces
> > for the implementation to work across multiple frameworks. It is
> > important that none of the existing features are compromised in this
> > process and continue to function in a k8s environment.
> >
> > -Max
> >
> > On Mon, Feb 20, 2023 at 11:37 AM zhangjiao  wrote:
> > >
> > > Hi,
> > > Glad to hear that, we’re very interested in that too.
> > >
> > > Currently, all of our jobs are running on yarn and our team have
> > implemented autoscaler  in our production.
> > > We prepare to upgrade it base on flip-271.It’ll be very nice that have
> a
> > version compatible with yarn and k8s.
> > > Hope to see it in the near future. We can also join and do our bit.
> > >
> > > Best,
> > > zlzhang0122
> > >
> > >
> > > On 2023/02/20 08:14:36 Matt Wang wrote:
> > > > Hi,
> > > > Thank you gays for bringing this up, we're very interested in that as
> > well.
> > > >
> > > > We are currently migrating from yarn to kubernetes, but this will
> last
> > for a long time, so the support of yarn is also more important. We have
> now
> > started to promote Autoscaling in our internal business. The model we use
> > is the DS2 model similar to flip-271. In the near future, we will also
> > communicate with you about the problems we encounter online.
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Best,
> > > > Matt Wang
> > > >
> > > >
> > > >  Replied Message 
> > > > | From | Rui Fan<19...@gmail.com> |
> > > > | Date | 02/20/2023 10:35 |
> > > > | To |  |
> > > > | Subject | Re: [DISCUSS] Extract core autoscaling algorithm as new
> > SubModule in flink-kubernetes-operator |
> > > > Hi Gyula, Samrat and Shammon,
> > > >
> > > > My team is also looking forward to autoscaler is compatible with
> yarn.
> > > >
> > > > Currently, all of our flink jobs are running on yarn. And autoscaler
> is
> > > > a great feature for flink users, it can greatly simplify the process
> of
> > > > tuning parallelism.
> > > >
> > > > If the autoscaler supports yarn, I propose to divide it into two
> > stages:
> > > > 1. It only collects and evaluates scaling related performance metrics
> > > > but does not trigger any job upgrades.
> > > > 2. Support for automatic upgrades of yarn jobs.
> > > >
> > > > Also, I also hope to join it, and improve it together.
> > > >
> > > > And very happy Gyula can help with the review.
> > > >
> > > > Best,
> > > > Rui Fan
> > > >
> > > > On Mon, Feb 20, 2023 at 8:56 AM Shammon FY  wrote:
> > > >
> > > > Hi Samrat
> > > >
> > > > My team is also looking at this piece. After you give your proposal,
> we
> > > > also hope to join it with you if possible. I hope we can improve this
> > > > together for use in our production too, thanks :)
> > > >
> > > > Best,
> > > > Shammon
> > > >
> > > > On Fri, Feb 17, 2023 at 9:27 PM Samrat Deb  wrote:
> > > >
> > > > @Gyula
> > > > Thank you
> > > > We will work on this and try to come up with an approach.
> > > >
> > > >
> > > >
> > > >
> > > > On Fri, Feb 17, 2023 at 6:12 PM Gyula Fóra  wrote:
> > > >
> > > > In case you guys feel strongly about this I suggest you try to fork
> the
> > > > autoscaler implementation and make a version that works with both the
> > > > Kubernetes operator and YARN.
> > > > If your solution is generic and works well, we can discuss the way
> > > > forward.
> > > >
> > > > Unfortunately me or my team don't really have the resources to assist
> > > > you
> > > > with the YARN effort as we are mostly invested in Kubernetes but of
> > > > course
> > > > we are happy to review your work.
> > > >
> > > > Gyula
> > > >
> > > >
> > > > On Fri, Feb 17, 2023 at 1:09 PM Prabhu Joseph <
> > > > prabhujose.ga...@gmail.com>
> > > > wrote:
> > > >
> > > > @Gyula
> > > >
> > > > It is easier to make the operator work with jobs running in
> > > > different
> > > > types of clusters than to take the
> > > > autoscaler module itself and plug that in somewhere else.
> > > >
> > > > Our (part of Samrat's team) main problem is to leverage the
> > > > AutoScaler
> > > > Recommendation Engine part of Flink-Kubernetes-Operator for our Flink
> > > > jobs
> > > > running on YARN.
> > > > Currently, it is not feasible as the autoscaler module is tightly
> > > >

[jira] [Created] (FLINK-31311) Supports Bounded Watermark streaming read

[Jingsong Lee (Jira)]

Jingsong Lee created FLINK-31311:


 Summary: Supports Bounded Watermark streaming read
 Key: FLINK-31311
 URL: https://issues.apache.org/jira/browse/FLINK-31311
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Reporter: Jingsong Lee
Assignee: Jingsong Lee
 Fix For: table-store-0.4.0


There are some bound stream scenarios that require that stream reading can be 
ended. Generally speaking, the end event time is the better.

So in this ticket, supports writing the watermark to the snapshot and can 
specify the ending watermark when reading the stream.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-31312) EnableObjectReuse cause different behaviors

[Jiang Xin (Jira)]

Jiang Xin created FLINK-31312:
-

 Summary: EnableObjectReuse cause different behaviors
 Key: FLINK-31312
 URL: https://issues.apache.org/jira/browse/FLINK-31312
 Project: Flink
  Issue Type: Bug
  Components: API / DataStream
Reporter: Jiang Xin


I have the following test code which works well, however, if I remove the 
`enableObjectReuse`, the test case would fail with the exception `Accessing a 
field by name is not supported in position-based field mode`.
{code:java}
public static void main(String[] args) throws Exception {
StreamExecutionEnvironment env = 
StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);

// The test fails with enableObjectReuse
env.getConfig().enableObjectReuse();

final SourceFunction rowGenerator =
new SourceFunction() {
@Override
public final void run(SourceContext ctx) throws Exception {
Row row = new Row(1);
row.setField(0, "a");
ctx.collect(row);
}

@Override
public void cancel() {}
};

final RowTypeInfo typeInfo =
new RowTypeInfo(new TypeInformation[] {Types.STRING}, new String[] 
{"col1"});

DataStream dataStream = env.addSource(rowGenerator, typeInfo);

DataStream transformedDataStream =
dataStream.map(
(MapFunction) value -> 
Row.of(value.getField("col1")), typeInfo);

transformedDataStream.addSink(new PrintSinkFunction<>());
env.execute("Mini Test");
} {code}
The `SourceFunction` generates rows without field names, but the return type 
info is assigned by `env.addSource(rowGenerator, typeInfo)`.

With object-reuse enabled, rows would be passed to the mapFunction directly, so 
the exception raises. While if the object-reuse is disabled,  rows would be 
reconstructed and given field names when passing to the next operator, so the 
test case works well.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

[Jingsong Li]

Hi, Shammon,

I took a look at JDBC `ResultSet` and `Statement`.  They are
complicated and have many interfaces. Some of the interfaces may not
be very suitable for streaming.

I think maybe we can just implement JDBC for batch/olap only. It is
hard to have an integration for JDBC and streaming...

Do you need to use JDBC in streaming mode? Or do we just implement
JDBC for batch only first?

Best,
Jingsong


On Thu, Mar 2, 2023 at 6:22 PM Shammon FY  wrote:
>
> Hi
>
> Thanks for the feedback from Jingsong and Benchao.
>
> For @Jingsong
> > If the user does not cast into a FlinkResultSet, will there be
> serious consequences here (RowKind is ignored)?
>
> I agree with you that it's indeed a big deal if users ignore the row kind
> when they must know it. One idea that comes to my mind is we can add an
> option such as `table.result.changelog-mode`, users can set it through
> connection properties or set dynamic parameters. The option value can be
> `insert-only`, `upset` or `all` and the default value is `insert-only`.
>
> If the result does not conform to the changelog mode, the jdbc driver
> throws an exception. What do you think?
>
>
> For @Benchao
> > Besides `java.sql.Driver`, have you considered also adding support for
> `javax.sql.DataSource` interface?
>
> I missed the `javax.sql.DataSource` and I have added it to the FLIP, thanks
> Benchao
>
>
> Best,
> Shammon
>
> On Wed, Mar 1, 2023 at 7:57 PM Benchao Li  wrote:
>
> > +1 for the FLIP, thanks Shammon for driving this.
> >
> > JDBC is quite useful in OLAP scenarios, supporting JDBC would enable Flink
> > to be used with existing tools, such as Tableau.
> >
> > Regarding the JDBC interfaces listed in the FLIP, I think they looks good
> > already. Besides `java.sql.Driver`, have you considered also adding support
> > for `javax.sql.DataSource` interface?
> >
> > Jingsong Li  于2023年3月1日周三 17:53写道：
> >
> > > Thanks Shammon for driving.
> > >
> > > Big +1 for this.
> > >
> > > I heard that many users want to use FlinkGateway + JDBC to do some
> > > queries, but at present, only Hive JDBC can be used. It is Hive
> > > dialect by default, and the experience is also different from
> > > FlinkSQL. We need to have our own JDBC.
> > >
> > > I took a look at your `Public Interface` part, only
> > > `FlinkResultSet.getRowKind` is a true new interface, others are just
> > > implementations.
> > >
> > > If the user does not cast into a FlinkResultSet, will there be serious
> > > consequences here (RowKind is ignored)?
> > >
> > > Best,
> > > Jingsong
> > >
> > > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY  wrote:
> > > >
> > > > Hi devs,
> > > >
> > > > I'd like to start a discussion about FLIP-293: Introduce Flink Jdbc
> > > Driver
> > > > For Sql Gateway[1].
> > > >
> > > > FLIP-275[2] supports remote sql client based on gateway, users can
> > > interact
> > > > with gateway by flink console. However, for users who create session
> > > > clusters with Flink, they'd like to use Jdbc Driver to interact with
> > the
> > > > gateway in their applications, such as olap queries..
> > > >
> > > > I have discussed this proposal with @shengkaifang and @jinsonglee. In
> > > this
> > > > FLIP, we'd like to introduce Jdbc Driver for gateway. Users can use
> > Jdbc
> > > > Driver to submit their queries and get results like a database in their
> > > > applications.
> > > >
> > > > Looking forward to your feedback, thanks.
> > > >
> > > >
> > > > [1]
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway
> > > > [2]
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway
> > > >
> > > >
> > > > Best,
> > > > Shammon
> > >
> >
> >
> > --
> >
> > Best,
> > Benchao Li
> >

Flink k8s native support - pod deployments and upgrades

[ramkrishna vasudevan]

Hi All,

The native implementation of the App mode and session mode - does not have
any replica set .
Instead it just allows the JM to create TM pods on demand.

This is simple and easy in terms of creation of resources, but for an
upgrade story, how is this managed? Leaving K8s to manage a replica set
based upgrade might be easier right?

Just wanted to understand how upgrades are handled in native K8s mode.

Regards
Ram

Re: [DISCUSS] PRs in flink-ci/flink-mirror

[Weihua Hu]

Hi, Matthias

Thanks for bringing this discussion.

When I wanted to trigger a CI pipeline, my first thought was to submit a PR
to flink repo. But considering that the PR was not intended to be merged
in,
it might interfere with others. So I tried to retrieve how to run the CI
pipeline
without PR, Then I found this documentation and flows it's suggestion to
submit a PR to flink-mirror. Sorry for making noise in the slack channel.

+1 for removing this section in the docs.


Best,
Weihua


On Thu, Mar 2, 2023 at 4:17 PM Matthias Pohl 
wrote:

> Hi everyone,
> Weihua Hu [1] notified me of a section in Flink's Azure Pipeline
> documentation [2] where it's suggested to create PRs against
> flink-ci/mirror as a workaround if you're not having a private Azure
> Pipeline account and want to run CI with your code changes. Even though
> it's a viable solution it does generate noise in the Slack channel for
> build failure (#builds). Additionally, I don't see any extra value in
> comparison to using your own fork and creating a PR on your branch against
> the apache/flink repo. CI will be picked up by Flink's CiBot. Or am I
> missing something?
>
> I couldn't find any ML discussion on that matter. I suggest removing this
> section in the docs.
>
> Best,
> Matthias
>
> [1]
> https://github.com/flink-ci/flink-mirror/pull/16#issuecomment-1451183958
> [2]
>
> https://cwiki.apache.org/confluence/display/FLINK/Azure+Pipelines#AzurePipelines-AzurePipelineUsageRestrictions
>
> --
>
> [image: Aiven] 
>
> *Matthias Pohl*
> Opensource Software Engineer, *Aiven*
> matthias.p...@aiven.io|  +49 170 9869525
> aiven.io    |    >
>      <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>

Re: [DISCUSS] FLIP-296: Watermark options for table API & SQL

[Kui Yuan]

Hi all,

Thanks for all. There are more questions and I will answer one by one.

@Jark Thanks for your tips. For the first question, I will add more details
in the flip, and give a POC[1] so that pepole can know how I'm currently
implementing these features.

> IIRC, this is the first time we introduce the framework-level connector
> options that the option is not recognized and handled by connectors.
> The FLIP should cover how framework filters the watermark related options
> to avoid discover connector factory failed, and what happens if the
> connector already supported the conflict options

For the second question, We know that the default strategy is 'on-periodic'
in SQL layer, and the default interval is 200ms. The reason for emiting
watermark periodically is that the time advancement of consecutive events
may be very small, we don't need to calculate watermark for each event.
Same for 'on-event' strategy, so my idea is that we can set a fixed gap for
'on-event' strategy.

> I'm not sure about the usage scenarios of event gap emit strategy. Do
> you have any specific use case of this strategy? I'm confused why no one
> requested this strategy before no matter in DataStream or SQL, but maybe
> I missed something. I'm not against to add this option, but just want to
be
> careful when adding new API because it's hard to remove in the future.

As @Timo said, There is no default features like 'on-event-gap' in
DataStream API, but the users can achieve the 'on-event-gap' feature by
using `WatermarkGenerator` interface, just like the implemention in my
POC[1]. However, If we don't provide it  in SQL layer, there is no way for
users to use similar features.

> Jark raised a very good point. I thought we only expose what is
> contained in DataStream API already. If this strategy is not part of
> DataStream API, would like to exclude it from the FLIP. We need to be
> careful which strategies we offer by default.

@Jark @Timo I'm sorry, perhaps I don't understand what are your concerns
about CompiledPlan, maybe I missed something else, maybe you can look at my
POC first to see if there is somewhere to worry about.

> Sorry, I forgot to remind you that Timo's concern about the changes to the
> CompiledPlan looks like is still not covered in the FLIP.

@Jing We could have more discussion about naming, but I prefer that the
naming should be consistent with the DataStream API.
About aligning splits/partitions/shards, maybe you missed FLIP-217[2] which
aims to support watermark alignment of source splits.

> After reading the most up-to-date Flip, I didn't find any information if
> this solution will support aligning splits/partitions/shards [1]. Did I
> miss anything?

Best
Kui Yuan

[1] the POC:
https://github.com/yuchengxin/flink/tree/yuankui/watermark_params
[2] FLIP-217:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits


Jing Ge  于2023年3月3日周五 08:03写道：

> Hi,
>
> Thanks Kui for driving this Flip and thanks all for the informative
> discussion.
>
> @Timo
>
> Your suggestion about the naming convention is excellent. Thanks! I was
> wondering why you, exceptionally, suggested 'scan.idle-timeout' instead of
> 'scan.watermark.idle-timeout'. I must miss something here.
>
> There is one more NIT. I am just aware that "drift" is used for the
> watermark alignment. It seems to be fine while using DataStream API,
> because we will not really see it. But with the OPTIONS in SQL, a much
> bigger group of users (including SRE, tech support, etc) will see the word
> "drift". Given that "drift" wasn't used widely yet and with all training
> materials, Flink doc [1][2][3] (search with "lag"), "lag" has been used to
> describe timestamp difference between watermark and its
> corresponding event. Do we really need to introduce another term for the
> same thing? How about using 'scan.watermark.alignment.max-lag'='1min' and
> change the parameter name from maxAllowedWatermarkDrift to
> maxAllowedWatermarkLag [4] because of naming consistency? Just my two cents
> worth.
>
> @Kui
>
> After reading the most up-to-date Flip, I didn't find any information if
> this solution will support aligning splits/partitions/shards [1]. Did I
> miss anything?
>
> +1 for the concern about Table API. We'd be better keep Table API and SQL
> synced for new features.
>
> Best regards,
> Jing
>
>
> [1]
>
> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/datastream/event-time/generating_watermarks/#watermark-alignment-_beta_
> [2]
>
> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/datastream/event-time/built_in/#fixed-amount-of-lateness
>
> [3]
>
> https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/kafka/
> [4]
>
> https://github.com/apache/flink/blob/4aacff572a9e3996c5dee9273638831e4040c767/flink-core/src/main/java/org/apache/flink/api/common/eventtime/WatermarkStrategy.java#L169
>
>
>
> On Wed, Mar 1, 2023 at 3:54 PM Tim

Re: [DISCUSS] FLIP-293: Introduce Flink Jdbc Driver For Sql Gateway

[Shammon FY]

Hi,

Thanks jinsong. I think implementing JDBC for batch mode first sounds good.
This will simplify the implementation and we can also remove the row kind
first. We can claim this in the FLIP and docs, I will update the FLIP.

Best,
Shammon


On Fri, Mar 3, 2023 at 2:36 PM Jingsong Li  wrote:

> Hi, Shammon,
>
> I took a look at JDBC `ResultSet` and `Statement`.  They are
> complicated and have many interfaces. Some of the interfaces may not
> be very suitable for streaming.
>
> I think maybe we can just implement JDBC for batch/olap only. It is
> hard to have an integration for JDBC and streaming...
>
> Do you need to use JDBC in streaming mode? Or do we just implement
> JDBC for batch only first?
>
> Best,
> Jingsong
>
>
> On Thu, Mar 2, 2023 at 6:22 PM Shammon FY  wrote:
> >
> > Hi
> >
> > Thanks for the feedback from Jingsong and Benchao.
> >
> > For @Jingsong
> > > If the user does not cast into a FlinkResultSet, will there be
> > serious consequences here (RowKind is ignored)?
> >
> > I agree with you that it's indeed a big deal if users ignore the row kind
> > when they must know it. One idea that comes to my mind is we can add an
> > option such as `table.result.changelog-mode`, users can set it through
> > connection properties or set dynamic parameters. The option value can be
> > `insert-only`, `upset` or `all` and the default value is `insert-only`.
> >
> > If the result does not conform to the changelog mode, the jdbc driver
> > throws an exception. What do you think?
> >
> >
> > For @Benchao
> > > Besides `java.sql.Driver`, have you considered also adding support for
> > `javax.sql.DataSource` interface?
> >
> > I missed the `javax.sql.DataSource` and I have added it to the FLIP,
> thanks
> > Benchao
> >
> >
> > Best,
> > Shammon
> >
> > On Wed, Mar 1, 2023 at 7:57 PM Benchao Li  wrote:
> >
> > > +1 for the FLIP, thanks Shammon for driving this.
> > >
> > > JDBC is quite useful in OLAP scenarios, supporting JDBC would enable
> Flink
> > > to be used with existing tools, such as Tableau.
> > >
> > > Regarding the JDBC interfaces listed in the FLIP, I think they looks
> good
> > > already. Besides `java.sql.Driver`, have you considered also adding
> support
> > > for `javax.sql.DataSource` interface?
> > >
> > > Jingsong Li  于2023年3月1日周三 17:53写道：
> > >
> > > > Thanks Shammon for driving.
> > > >
> > > > Big +1 for this.
> > > >
> > > > I heard that many users want to use FlinkGateway + JDBC to do some
> > > > queries, but at present, only Hive JDBC can be used. It is Hive
> > > > dialect by default, and the experience is also different from
> > > > FlinkSQL. We need to have our own JDBC.
> > > >
> > > > I took a look at your `Public Interface` part, only
> > > > `FlinkResultSet.getRowKind` is a true new interface, others are just
> > > > implementations.
> > > >
> > > > If the user does not cast into a FlinkResultSet, will there be
> serious
> > > > consequences here (RowKind is ignored)?
> > > >
> > > > Best,
> > > > Jingsong
> > > >
> > > > On Wed, Mar 1, 2023 at 4:59 PM Shammon FY  wrote:
> > > > >
> > > > > Hi devs,
> > > > >
> > > > > I'd like to start a discussion about FLIP-293: Introduce Flink Jdbc
> > > > Driver
> > > > > For Sql Gateway[1].
> > > > >
> > > > > FLIP-275[2] supports remote sql client based on gateway, users can
> > > > interact
> > > > > with gateway by flink console. However, for users who create
> session
> > > > > clusters with Flink, they'd like to use Jdbc Driver to interact
> with
> > > the
> > > > > gateway in their applications, such as olap queries..
> > > > >
> > > > > I have discussed this proposal with @shengkaifang and @jinsonglee.
> In
> > > > this
> > > > > FLIP, we'd like to introduce Jdbc Driver for gateway. Users can use
> > > Jdbc
> > > > > Driver to submit their queries and get results like a database in
> their
> > > > > applications.
> > > > >
> > > > > Looking forward to your feedback, thanks.
> > > > >
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-293%3A+Introduce+Flink+Jdbc+Driver+For+Sql+Gateway
> > > > > [2]
> > > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway
> > > > >
> > > > >
> > > > > Best,
> > > > > Shammon
> > > >
> > >
> > >
> > > --
> > >
> > > Best,
> > > Benchao Li
> > >
>