[jira] [Created] (FLINK-26535) Introduce StoreTableSource and StoreTableSink

[Jane Chan (Jira)]

Jane Chan created FLINK-26535:
-

 Summary: Introduce StoreTableSource and StoreTableSink
 Key: FLINK-26535
 URL: https://issues.apache.org/jira/browse/FLINK-26535
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Affects Versions: 0.1.0
Reporter: Jane Chan






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: Re: [DISCUSS] Flink's supported APIs and Hive query syntax

[Jark Wu]

Hi Martijn,

Thanks for starting this discussion. I think it's great
for the community to to reach a consensus on the roadmap
of Hive query syntax.

I agree that the Hive project is not actively developed nowadays.
However, Hive still occupies the majority of the batch market
and the Hive ecosystem is even more active now. For example,
the Apache Kyuubi[1] is a new project that is a JDBC server
which is compatible with HiveServer2. And the Apache Iceberg
and Apache Hudi are mainly using Hive Metastore as the table catalog.
The Spark SQL is 99% compatible with Hive SQL. We have to admit
that Hive is the open-source de facto standard for batch processing.

As far as I can see, almost all the companies (including ByteDance,
Kuaishou, NetEase, etc..) in China are using Hive SQL for batch
processing, even the underlying is using Spark as the engine.
I don't know how the batch users can migrate to Flink if Flink
doesn't provide the Hive compatibility. IMO, in the short term,
Hive syntax compatibility is the ticket for us to have a seat
in the batch processing. In the long term, we can drop it and
focus on Flink SQL itself both for batch and stream processing.

Regarding the maintenance concern you raised, I think that's a good
point and they are in the plan. The Hive dialect has already been
a plugin and option now, and the implementation is located in
hive-connector module. We still need some work to make the Hive
dialect purely rely on public APIs, and the Hive connector should be
decopule with table planner. At that time, we can move the whole Hive
connector into a separate repository (I guess this is also in the
externalize connectors plan).

What do you think?

Best,
Jark

[1]:
https://kyuubi.apache.org/docs/latest/overview/kyuubi_vs_thriftserver.html
[2]: https://iceberg.apache.org/docs/latest/spark-configuration/
[3]: https://hudi.apache.org/docs/next/syncing_metastore/

On Tue, 8 Mar 2022 at 11:46, Mang Zhang  wrote:

> Hi Martijn,
>
> Thanks for driving this discussion.
>
> +1 on efforts on more hive/spark syntax compatibility.The hive/spark
> syntax is the most popular in batch computing.Within our company, many
> users have the desire to use Flink to realize the integration of streaming
> and batching,and some users have been running in production for months.And
> we have integrated Flink with our internal remote shuffle service, flink
> save user a lot of development and maintenance costs,user feedback is very
> good.Enrich flink's ecology and provide users with more choices, so I think
> pluggable support for hive/spark dialects is very necessary.We need better
> designs for future multi-source fusion.
>
>
>
>
>
>
>
> Best regards,
>
> Mang Zhang
>
>
>
>
>
> At 2022-03-07 20:52:42, "Jing Zhang"  wrote:
> >Hi Martijn,
> >
> >Thanks for driving this discussion.
> >
> >+1 on efforts on more hive syntax compatibility.
> >
> >With the efforts on batch processing in recent versions(1.10~1.15), many
> >users have run batch processing jobs based on Flink.
> >In our team, we are trying to migrate most of the existing online batch
> >jobs from Hive/Spark to Flink. We hope this migration does not require
> >users to modify their sql.
> >Although Hive is not as popular as it used to be, Hive SQL is still alive
> >because many users still use Hive SQL to run spark jobs.
> >Therefore, compatibility with more HIVE syntax is critical to this
> >migration work.
> >
> >Best,
> >Jing Zhang
> >
> >
> >
> >Martijn Visser  于2022年3月7日周一 19:23写道：
> >
> >> Hi everyone,
> >>
> >> Flink currently has 4 APIs with multiple language support which can be
> used
> >> to develop applications:
> >>
> >> * DataStream API, both Java and Scala
> >> * Table API, both Java and Scala
> >> * Flink SQL, both in Flink query syntax and Hive query syntax
> (partially)
> >> * Python API
> >>
> >> Since FLIP-152 [1] the Flink SQL support has been extended to also
> support
> >> the Hive query syntax. There is now a follow-up FLINK-26360 [2] to
> address
> >> more syntax compatibility issues.
> >>
> >> I would like to open a discussion on Flink directly supporting the Hive
> >> query syntax. I have some concerns if having a 100% Hive query syntax is
> >> indeed something that we should aim for in Flink.
> >>
> >> I can understand that having Hive query syntax support in Flink could
> help
> >> users due to interoperability and being able to migrate. However:
> >>
> >> - Adding full Hive query syntax support will mean that we go from 6
> fully
> >> supported API/language combinations to 7. I think we are currently
> already
> >> struggling with maintaining the existing combinations, let another one
> >> more.
> >> - Apache Hive is/appears to be a project that's not that actively
> developed
> >> anymore. The last release was made in January 2021. It's popularity is
> >> rapidly declining in Europe and the United State, also due Hadoop
> becoming
> >> less popular.
> >> - Related to the previous topic, other software like Snowflake,
> >> Trino/Pres

Re: [ANNOUNCE] New Apache Flink Committer - David Morávek

[Johannes Moser]

Very well deserved.

> On 08.03.2022, at 05:43, Lincoln Lee  wrote:
> 
> Congratulations David!
> 
> Best,
> Lincoln Lee
> 
> 
> Yun Gao  于2022年3月8日周二 11:24写道：
> 
>> Congratulations David!
>> 
>> Best,
>> Yun Gao
>> 
>> 
>> --
>> From:Jing Zhang 
>> Send Time:2022 Mar. 8 (Tue.) 11:10
>> To:dev 
>> Subject:Re: [ANNOUNCE] New Apache Flink Committer - David Morávek
>> 
>> Congratulations David!
>> 
>> Ryan Skraba  于2022年3月7日周一 22:18写道：
>> 
>>> Congratulations David!
>>> 
>>> On Mon, Mar 7, 2022 at 9:54 AM Jan Lukavský  wrote:
>>> 
 Congratulations David!
 
  Jan
 
 On 3/7/22 09:44, Etienne Chauchot wrote:
> Congrats David !
> 
> Well deserved !
> 
> Etienne
> 
> Le 07/03/2022 à 08:47, David Morávek a écrit :
>> Thanks everyone!
>> 
>> Best,
>> D.
>> 
>> On Sun 6. 3. 2022 at 9:07, Yuan Mei  wrote:
>> 
>>> Congratulations, David!
>>> 
>>> Best Regards,
>>> Yuan
>>> 
>>> On Sat, Mar 5, 2022 at 8:13 PM Roman Khachatryan >> 
>>> wrote:
>>> 
 Congratulations, David!
 
 Regards,
 Roman
 
 On Fri, Mar 4, 2022 at 7:54 PM Austin Cawley-Edwards
  wrote:
> Congrats David!
> 
> On Fri, Mar 4, 2022 at 12:18 PM Zhilong Hong <
>> zhlongh...@gmail.com
 
 wrote:
>> Congratulations, David!
>> 
>> Best,
>> Zhilong
>> 
>> On Sat, Mar 5, 2022 at 1:09 AM Piotr Nowojski <
>>> pnowoj...@apache.org
> 
>> wrote:
>> 
>>> Congratulations :)
>>> 
>>> pt., 4 mar 2022 o 16:04 Aitozi 
>>> napisał(a):
>>> 
 Congratulations David!
 
 Ingo Bürk  于2022年3月4日周五 22:56写道：
 
> Congrats, David!
> 
> On 04.03.22 12:34, Robert Metzger wrote:
>> Hi everyone,
>> 
>> On behalf of the PMC, I'm very happy to announce David
>>> Morávek
 as a
>>> new
>> Flink committer.
>> 
>> His first contributions to Flink date back to 2019. He has
>>> been
>> increasingly active with reviews and driving major
>>> initiatives
 in
>> the
>> community. David brings valuable experience from being a
 committer
>> in
 the
>> Apache Beam project to Flink.
>> 
>> 
>> Please join me in congratulating David for becoming a Flink
>>> committer!
>> Cheers,
>> Robert
>> 
 
>>> 
>> 

status of Apple Silicon (M1) as Flink dev platform?

[David Anderson]

What's the current status of using the Apple Silicon (M1) platform for
Flink development? Have we reached the point where everything "just works",
or do there remain lingering annoyances (or worse)?

In the past, I've seen reports of issues involving, e.g., RocksDB, nodejs,
protobuf, and pyflink. Looking in Jira, I see these issues haven't been
resolved yet, but I'm not sure what to read into that:

https://issues.apache.org/jira/browse/FLINK-24932 (Frocksdb cannot run on
Apple M1)
https://issues.apache.org/jira/browse/FLINK-25188 (Cannot install PyFlink
on MacOS with M1 chip)

Best,
David

Re: [DISCUSS] Flink's supported APIs and Hive query syntax

[Zou Dan]

Hi Martijn,
Thanks for bringing this up.
Hive SQL (using in Hive & Spark) plays an important role in batch processing, 
it has almost become de facto standard in batch processing. In our company, 
there are hundreds of thousands of spark jobs each day.
IMO, if we want to promote Flink batch, Hive syntax compatibility is a crucial 
point of it.
Thanks to this feature, we have migrated 800+ Spark jobs to Flink smoothly.

So, I quite agree with putting more effort into Hive syntax compatibility.

Best,
Dan Zou

> 2022年3月7日 19:23，Martijn Visser  写道：
> 
> query

Re: Flink 1.15 Stabilisation Sync skipped tomorrow

[Johannes Moser]

Thanks so much Gabor and Galen.

Very much appreciated.

> On 07.03.2022, at 19:00, Galen Warren  wrote:
> 
> I just ran through the testing steps on [FLINK-26462] Release Testing:
> Running Python UDF in different Execution Mode - ASF JIRA (apache.org)
> .
> 
> Everything worked but I think there might be a typo in the steps to set up
> the tests, I left a comment on the issue.
> 
> On Mon, Mar 7, 2022 at 12:01 PM Gabor Somogyi 
> wrote:
> 
>> Since I know this area I help to test FLINK-26468
>> .
>> 
>> G
>> 
>> On Mon, Mar 7, 2022 at 5:15 PM Johannes Moser  wrote:
>> 
>>> Hello,
>>> 
>>> We will skip tomorrows weekly stabilisation sync.
>>> It looks like all blockers are assigned and worked on. A significant
>>> number of contributors are on holiday tomorrow.
>>> 
>>> Currently we plan to cut off the 1.15 branch on Wednesday of Thursday
>> this
>>> week.
>>> 
>>> In case you want to contribute something to the release there are some
>>> features to perform manual tests:
>>> * https://issues.apache.org/jira/browse/FLINK-26451 <
>>> https://issues.apache.org/jira/browse/FLINK-26451>
>>> * https://issues.apache.org/jira/browse/FLINK-26462 <
>>> https://issues.apache.org/jira/browse/FLINK-26462>
>>> * https://issues.apache.org/jira/browse/FLINK-26468 <
>>> https://issues.apache.org/jira/browse/FLINK-26468>
>>> 
>>> 
>>> Best, Yun Gao and Joe
>> 

[jira] [Created] (FLINK-26536) PyFlink RemoteKeyedStateBackend#merge_namespaces bug

[Juntao Hu (Jira)]

Juntao Hu created FLINK-26536:
-

 Summary: PyFlink RemoteKeyedStateBackend#merge_namespaces bug
 Key: FLINK-26536
 URL: https://issues.apache.org/jira/browse/FLINK-26536
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.14.3
Reporter: Juntao Hu
 Fix For: 1.15.0


There's two bugs in RemoteKeyedStateBackend#merge_namespaces:
 * data in source namespaces are not commited before merging
 * target namespace is added at the head of sources_bytes, which causes java 
side SimpleStateRequestHandler to leave one source namespace unmerged



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Enable scala formatting check

[David Anderson]

+1

For flink-training we initially tried cloning the scalastyle setup from
flink, but we decided to use spotless + scalafmt instead.

David

On Mon, Mar 7, 2022 at 1:12 PM Timo Walther  wrote:

> Big +1
>
> This will improve the contribution experience. Even though we stopped
> adding more Scala code, it is still necessary from time to time.
>
> Regards,
> Timo
>
> Am 02.03.22 um 09:29 schrieb 刘首维:
> > +1
> >
> >
> > I still remember my first pr. Lack of experience, I had to pay attention
> to Scala code format and corrected the format manually, which made me a
> little embarrassed(though I'm a big fan of Scala). I think this
> proposal will lighten the burden of writing Scala code.
> >
> >
> > Shouwei Liu
> >
> >
> > -- 原始邮件 --
> > 发件人:
> "dev"
>   <
> kna...@apache.org>;
> > 发送时间: 2022年3月2日(星期三) 下午3:01
> > 收件人: "dev" >
> > 主题: Re: [DISCUSS] Enable scala formatting check
> >
> >
> >
> > +1 I've never written any Scala in Flink, but this makes a lot of sense
> to
> > me. Converging on a smaller set of tools and simplifying the build is
> > always a good idea and the Community already concluded before that
> spotless
> > is generally a good approach.
> >
> > On Tue, Mar 1, 2022 at 5:52 PM Francesco Guardiani <
> france...@ververica.com>
> > wrote:
> >
> > > Hi all,
> > >
> > > I want to propose to enable the spotless scalafmt integration and
> remove
> > > the scalastyle plugin.
> > >
> > > From an initial analysis, scalafmt can do everything scalastyle can
> do, and
> > > the integration with spotless looks easy to enable:
> > > https://github.com/diffplug/spotless/tree/main/plugin-maven#scala.
> The
> > > scalafmt conf file gets picked up automatically from every IDE, and
> it can
> > > be heavily tuned.
> > >
> > > This way we can unify the formatting and integrate with our CI
> without any
> > > additional configurations. And we won't need scalastyle anymore, as
> > > scalafmt will take care of the checks:
> > >
> > > * mvn spotless:check will check both java and scala
> > > * mvn spotless:apply will format both java and scala
> > >
> > > WDYT?
> > >
> > > FG
> > >
> > >
> > >
> > > --
> > >
> > > Francesco Guardiani | Software Engineer
> > >
> > > france...@ververica.com
> > >
> > >
> > > ;
> > >
> > > Follow us @VervericaData
> > >
> > > --
> > >
> > > Join Flink Forward ; - The Apache
> Flink
> > > Conference
> > >
> > > Stream Processing | Event Driven | Real Time
> > >
> > > --
> > >
> > > Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany
> > >
> > > --
> > >
> > > Ververica GmbH
> > >
> > > Registered at Amtsgericht Charlottenburg: HRB 158244 B
> > >
> > > Managing Directors: Karl Anton Wehner, Holger Temme, Yip Park Tung
> Jason,
> > > Jinwei (Kevin) Zhang
> > >
> >
> >
>
>

Re: [DISCUSS] Flink's supported APIs and Hive query syntax

[Jingsong Li]

Thanks all for your discussions.

I'll share my opinion here:

1. Hive SQL and Hive-like SQL are the absolute mainstay of current
Batch ETL in China. Hive+Spark (HiveSQL-like)+Databricks also occupies
a large market worldwide.

- Unlike OLAP SQL (such as presto, which is ansi-sql rather than hive
sql), Batch ETL is run periodically, which means that a large number
of Batch Pipelines have already been built, and if they need to be
migrated to a new system, it will be extremely costly to migrate the
SQLs.

2. Our current Hive dialect is immature and we need to put more effort
to decouple it from the flink planner.

Best,
Jingsong

On Tue, Mar 8, 2022 at 4:27 PM Zou Dan  wrote:
>
> Hi Martijn,
> Thanks for bringing this up.
> Hive SQL (using in Hive & Spark) plays an important role in batch processing, 
> it has almost become de facto standard in batch processing. In our company, 
> there are hundreds of thousands of spark jobs each day.
> IMO, if we want to promote Flink batch, Hive syntax compatibility is a 
> crucial point of it.
> Thanks to this feature, we have migrated 800+ Spark jobs to Flink smoothly.
>
> So, I quite agree with putting more effort into Hive syntax compatibility.
>
> Best,
> Dan Zou
>
> 2022年3月7日 19:23，Martijn Visser  写道：
>
> query
>
>

回复：[DISCUSS] Flink's supported APIs and Hive query syntax

[罗宇侠(莫辞)]

Hi Martijn,
Thanks for driving this discussion. 

About your concerns, I would like to share my opinion.
Actually, more exactly, FLIP-152 [1] is not to extend Flink SQL to support Hive 
query synax, it provides a Hive dialect option to enable users to switch to 
Hive dialect. From the commits about the corresponding FLINK-21529, it doesn't 
involve much about Flink itself. 

- About the struggling with maintaining. The current implementation is just to 
provide an option for user to use Hive dialect. I think there won't be much 
bother.

- Although Apache Hive is less popular, it's widely used as an open source 
database over the years. There still exists many Hive SQL jobs in many 
companies.

- As I said, the current implementation for Hive SQL synax is more like 
pluggable, we can also support for Snowflake and the others as long as it's 
necessary.

- As for the know security vulnerabilities of Hive, maybe it's not a critical 
problem in this discuss.

- For current implementation for Hive SQL syntax, it uses a pluggable 
HiveParser[3] to parse the SQL statement. I think there won't be much 
complexity brought to Flink to support Hive syntax. 

From my perspective, Hive is still widely used and there exists many running 
Hive SQL jobs, so why not to provide users a better experience to help them 
migrate Hive jobs to Flink? Also, it doesn't conflict with Flink SQL as it's 
just a dialect option. 

[1] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=165227316
[2] https://issues.apache.org/jira/browse/FLINK-21529
[3] https://issues.apache.org/jira/browse/FLINK-21531
Best regards,
Yuxia.

​

 --原始邮件 --
发件人:Martijn Visser 
发送时间:Mon Mar 7 19:23:15 2022
收件人:dev , User 
主题:[DISCUSS] Flink's supported APIs and Hive query syntax
Hi everyone,

Flink currently has 4 APIs with multiple language support which can be used
to develop applications:

* DataStream API, both Java and Scala
* Table API, both Java and Scala
* Flink SQL, both in Flink query syntax and Hive query syntax (partially)
* Python API

Since FLIP-152 [1] the Flink SQL support has been extended to also support
the Hive query syntax. There is now a follow-up FLINK-26360 [2] to address
more syntax compatibility issues.

I would like to open a discussion on Flink directly supporting the Hive
query syntax. I have some concerns if having a 100% Hive query syntax is
indeed something that we should aim for in Flink.

I can understand that having Hive query syntax support in Flink could help
users due to interoperability and being able to migrate. However:

- Adding full Hive query syntax support will mean that we go from 6 fully
supported API/language combinations to 7. I think we are currently already
struggling with maintaining the existing combinations, let another one
more.
- Apache Hive is/appears to be a project that's not that actively developed
anymore. The last release was made in January 2021. It's popularity is
rapidly declining in Europe and the United State, also due Hadoop becoming
less popular.
- Related to the previous topic, other software like Snowflake,
Trino/Presto, Databricks are becoming more and more popular. If we add full
support for the Hive query syntax, then why not add support for Snowflake
and the others?
- We are supporting Hive versions that are no longer supported by the Hive
community with known security vulnerabilities. This makes Flink also
vulnerable for those type of vulnerabilities.
- The currently Hive implementation is done by using a lot of internals of
Flink, making Flink hard to maintain, with lots of tech debt and making
things overly complex.

From my perspective, I think it would be better to not have Hive query
syntax compatibility directly in Flink itself. Of course we should have a
proper Hive connector and a proper Hive catalog to make connectivity with
Hive (the versions that are still supported by the Hive community) itself
possible. Alternatively, if Hive query syntax is so important, it should
not rely on internals but be available as a dialect/pluggable option. That
could also open up the possibility to add more syntax support for others in
the future, but I really think we should just focus on Flink SQL itself.
That's already hard enough to maintain and improve on.

I'm looking forward to the thoughts of both Developers and Users, so I'm
cross-posting to both mailing lists.

Best regards,

Martijn Visser
https://twitter.com/MartijnVisser82

[1]
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=165227316
[2] https://issues.apache.org/jira/browse/FLINK-21529

Re:Re: [DISCUSS] The abstraction of cache lookupFunction and cache metric

[zst...@163.com]

Hi Qingsheng,
Sorry for my wrong format.

> If I understand correctly these are specified in DDL table options by users.




It's inconvenient for user to checkout the options when they in front of a 
running task. And they don't know the real underlying options in effect if 
there are some bugs or other incorrect configurations lead to invalid.




> I don’t think there's a rule that all metric names should be in MetricNames 
> class, but it would be great to aggregate these constants into a unified 
> place. 


It's a good choice to aggregate the constants together.


Best regards,
Yuan

At 2022-03-08 09:57:30, "Qingsheng Ren"  wrote:
>Hi Yuan, 
>
>> how can we tell the real  “identifier” and “type” options in effect to users?
>
>If I understand correctly these are specified in DDL table options by users. 
>For example: 
>
>CREATE TABLE DimTable (…) WITH (
>...
>“cache.identifier” = “guava”, 
>“cache.type” = “LRU”
>);
>
>> Does MetricNames.java contain all metric names?
>
>
>I don’t think there's a rule that all metric names should be in MetricNames 
>class, but it would be great to aggregate these constants into a unified 
>place. 
>
>Cheers, 
>
>Qingsheng
>
>
>> On Mar 8, 2022, at 10:22, zst...@163.com wrote:
>> 
>> Hi Qingsheng Ren,
>> Thanks for your feedback.
>> 
>> 
>>> 1. It looks like “identifier” and “type” are options of cache instead of 
>>> metrics. I think they are finalized once the cache is created so maybe it’s 
>>> not quite helpful to report them to the metric system.
>> 
>> 
>> Maybe it's not quite appropriate to report them to the metric system, but 
>> how can we tell the real  “identifier” and “type” options in effect to users?
>> 
>> 
>> 
>> 
>>> 2. Some metrics can be aggregated simply in metric systems, like loadCount 
>>> = loadSuccessCount + loadExceptionCount, so maybe we can just keep 
>>> fundamental metrics (like loadSuccessCount and loadExceptionCount) to avoid 
>>> redundancy.
>> 
>> 
>> I agree with you. I have removed these redundant metrics.
>> 
>> 
>>> 3. About the interface of CacheMetricGroup I think it would be easier for 
>>> cache implementers to use if we expose wrapped function instead of let 
>>> users provide gauges directly. 
>> 
>> 
>> Thanks for your advice, they are helpful and I have adjusted it. I have a 
>> question about it. Does MetricNames.java 
>> 
>>  contain all metric names? Should I put the cache metric names here?
>> 
>> 
>> 
>> 
>> 
>> 
>> Best regards,
>> Yuan
>> 
>> 
>> 
>> 
>> 
>> At 2022-03-07 16:55:18, "Qingsheng Ren"  wrote:
>>> Hi Yuan, 
>>> 
>>> Thanks for raising this discussion! I believe this will be very helpful for 
>>> lookup table developers, and standardizing metrics would be essential  for 
>>> users to tuning their systems. 
>>> 
>>> Here’s some thoughts in my mind:
>>> 
>>> 1. It looks like “identifier” and “type” are options of cache instead of 
>>> metrics. I think they are finalized once the cache is created so maybe it’s 
>>> not quite helpful to report them to the metric system.
>>> 
>>> 2. Some metrics can be aggregated simply in metric systems, like loadCount 
>>> = loadSuccessCount + loadExceptionCount, so maybe we can just keep 
>>> fundamental metrics (like loadSuccessCount and loadExceptionCount) to avoid 
>>> redundancy.
>>> 
>>> 3. About the interface of CacheMetricGroup I think it would be easier for 
>>> cache implementers to use if we expose wrapped function instead of let 
>>> users provide gauges directly. For example:
>>> 
>>> public interface CacheMetricGroup extends MetricGroup {
>>>   // Mark a cache hit
>>>   public void markCacheHit();
>>>   // Mark a cache miss
>>>   public void recordCacheMiss();
>>>   ...
>>> } 
>>> 
>>> You can check SourceReaderMetricGroup[1] and its implementation[2] as a 
>>> reference.
>>> 
>>> Hope these would be helpful!
>>> 
>>> Best regards, 
>>> 
>>> Qingsheng Ren
>>> 
>>> [1] 
>>> https://github.com/apache/flink/blob/master/flink-metrics/flink-metrics-core/src/main/java/org/apache/flink/metrics/groups/SourceReaderMetricGroup.java
>>> [2] 
>>> https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/InternalSourceReaderMetricGroup.java
>>> 
>>> 
 On Mar 7, 2022, at 16:00, zst...@163.com wrote:
 
 Hi devs,
 
 
 I would like to propose a discussion thread about abstraction of Cache 
 LookupFunction with metrics for cache in connectors to make cache out of 
 box for connector developers. There are multiple LookupFunction 
 implementations in individual connectors [1][2][3][4] so far. 
 At the same time, users can monitor cache in LookupFunction by adding 
 uniform cache metrics to optimize tasks or troubleshoot.
 
 
 I have posted an issue about this, see 
 , a

Why does Flink Table API not wait for job to complete?

[Adesh Dsilva]

Hi,

I wrote a simple program using Flink Table API. 
There is a Table source reading from an avro file, after doing some operations 
the result is stored using a sink csv table using `executeInsert()`

The program works fine and creates a csv file however the flink command does 
not wait for job to complete.

```
adsilva@C02XH3ZSJGH6 flink-1.14.3 % ./bin/flink run 
~/IdeaProjects/quickstart/target/app-0.1.jar --input 
part-v001-o000-r-00330.avro --output 
~/Downloads/dev-things/flink-1.14.3/output-8
Job has been submitted with JobID ff6f2e89a11e135d4978f7e4d2b1e2bc
adsilva@C02XH3ZSJGH6 flink-1.14.3 % 
```

However if I write the same program as a mixture of DataStream and Table API by 
taking input from DataStream but operating on it using Table API and writing it 
using Table sink then it does wait for job completion.


```
adsilva@C02XH3ZSJGH6 flink-1.14.3 % ./bin/flink run 
~/IdeaProjects/quickstart/target/domains_stats_processor-0.1.jar --input 
part-v001-o000-r-00330.avro --output 
~/Downloads/dev-things/flink-1.14.3/output-7
Job has been submitted with JobID a00ffab083747ec0e421c1d6ab0822ea
Job has been submitted with JobID 84ccd320f8015a53c8807380b829e529
Program execution finished
Job with JobID 84ccd320f8015a53c8807380b829e529 has finished.
Job Runtime: 31521 ms
```

It nicely prints the runtime as well (Although not sure why its starting two 
jobs here).

Why does this happen? How can I make Flink wait for job completion with pure 
Table API?

Please help, Thank you!

Re: Why does Flink Table API not wait for job to complete?

[Shuo Cheng]

I think what you need is something like:

table.executeInsert("MySink").*await()*


On Tue, Mar 8, 2022 at 8:24 PM Adesh Dsilva  wrote:

> Hi,
>
> I wrote a simple program using Flink Table API.
> There is a Table source reading from an avro file, after doing some
> operations the result is stored using a sink csv table using
> `executeInsert()`
>
> The program works fine and creates a csv file however the flink command
> does not wait for job to complete.
>
> ```
> adsilva@C02XH3ZSJGH6 flink-1.14.3 % ./bin/flink run
> ~/IdeaProjects/quickstart/target/app-0.1.jar --input
> part-v001-o000-r-00330.avro --output
> ~/Downloads/dev-things/flink-1.14.3/output-8
> Job has been submitted with JobID ff6f2e89a11e135d4978f7e4d2b1e2bc
> adsilva@C02XH3ZSJGH6 flink-1.14.3 %
> ```
>
> However if I write the same program as a mixture of DataStream and Table
> API by taking input from DataStream but operating on it using Table API and
> writing it using Table sink then it does wait for job completion.
>
>
> ```
> adsilva@C02XH3ZSJGH6 flink-1.14.3 % ./bin/flink run
> ~/IdeaProjects/quickstart/target/domains_stats_processor-0.1.jar --input
> part-v001-o000-r-00330.avro --output
> ~/Downloads/dev-things/flink-1.14.3/output-7
> Job has been submitted with JobID a00ffab083747ec0e421c1d6ab0822ea
> Job has been submitted with JobID 84ccd320f8015a53c8807380b829e529
> Program execution finished
> Job with JobID 84ccd320f8015a53c8807380b829e529 has finished.
> Job Runtime: 31521 ms
> ```
>
> It nicely prints the runtime as well (Although not sure why its starting
> two jobs here).
>
> Why does this happen? How can I make Flink wait for job completion with
> pure Table API?
>
> Please help, Thank you!

Re: Flink 1.15 Stabilisation Sync skipped tomorrow

[Galen Warren]

You're welcome. Fwiw, I've also run through testing steps on my
contribution -- [FLINK-26451] Test GCS FileSystem w/ RecoverableWriter
manually - ASF JIRA (apache.org)
 -- but I don't think
I'm supposed to be officially doing manually testing on my own stuff.

On Tue, Mar 8, 2022 at 3:46 AM Johannes Moser  wrote:

> Thanks so much Gabor and Galen.
>
> Very much appreciated.
>
> > On 07.03.2022, at 19:00, Galen Warren  wrote:
> >
> > I just ran through the testing steps on [FLINK-26462] Release Testing:
> > Running Python UDF in different Execution Mode - ASF JIRA (apache.org)
> > .
> >
> > Everything worked but I think there might be a typo in the steps to set
> up
> > the tests, I left a comment on the issue.
> >
> > On Mon, Mar 7, 2022 at 12:01 PM Gabor Somogyi  >
> > wrote:
> >
> >> Since I know this area I help to test FLINK-26468
> >> .
> >>
> >> G
> >>
> >> On Mon, Mar 7, 2022 at 5:15 PM Johannes Moser 
> wrote:
> >>
> >>> Hello,
> >>>
> >>> We will skip tomorrows weekly stabilisation sync.
> >>> It looks like all blockers are assigned and worked on. A significant
> >>> number of contributors are on holiday tomorrow.
> >>>
> >>> Currently we plan to cut off the 1.15 branch on Wednesday of Thursday
> >> this
> >>> week.
> >>>
> >>> In case you want to contribute something to the release there are some
> >>> features to perform manual tests:
> >>> * https://issues.apache.org/jira/browse/FLINK-26451 <
> >>> https://issues.apache.org/jira/browse/FLINK-26451>
> >>> * https://issues.apache.org/jira/browse/FLINK-26462 <
> >>> https://issues.apache.org/jira/browse/FLINK-26462>
> >>> * https://issues.apache.org/jira/browse/FLINK-26468 <
> >>> https://issues.apache.org/jira/browse/FLINK-26468>
> >>>
> >>>
> >>> Best, Yun Gao and Joe
> >>
>
>

[jira] [Created] (FLINK-26537) Allow disabling StatefulFunctionsConfigValidator validation for classloader.parent-first-patterns.additional

[Fil Karnicki (Jira)]

Fil Karnicki created FLINK-26537:


 Summary: Allow disabling StatefulFunctionsConfigValidator 
validation for classloader.parent-first-patterns.additional
 Key: FLINK-26537
 URL: https://issues.apache.org/jira/browse/FLINK-26537
 Project: Flink
  Issue Type: Improvement
  Components: Stateful Functions
Reporter: Fil Karnicki


For some deployments of stateful functions onto existing, shared clusters, it 
is impossible to tailor which classes exist on the classpath. An example would 
be a Cloudera Flink cluster, which adds protobuf-java classes that clash with 
statefun ones.

Stateful functions require the following flink-config.yaml setting:

{{classloader.parent-first-patterns.additional: 
org.apache.flink.statefun;org.apache.kafka;{+}*com.google.protobuf*{+}}} 

In the case of the cloudera flink cluster, this will cause old, 2.5.0 protobuf 
classes to be loaded by statefun, which causes MethodNotFound exceptions. 

The idea is to allow disabling of the validation below, if some config setting 
is present in the global flink configuration, for example: 
statefun.validation.parent-first-classloader.disable=true

 
{code:java}
private static void validateParentFirstClassloaderPatterns(Configuration 
configuration) {
  final Set parentFirstClassloaderPatterns =
  parentFirstClassloaderPatterns(configuration);
  if 
(!parentFirstClassloaderPatterns.containsAll(PARENT_FIRST_CLASSLOADER_PATTERNS))
 {
throw new StatefulFunctionsInvalidConfigException(
CoreOptions.ALWAYS_PARENT_FIRST_LOADER_PATTERNS_ADDITIONAL,
"Must contain all of " + String.join(", ", 
PARENT_FIRST_CLASSLOADER_PATTERNS));
  }
} {code}
 

Then, we wouldn't need to contain com.google.protobuf in 
{{classloader.parent-first-patterns.additional:}} and it would be up to the 
statefun user to use protobuf classes which are compatible with their version 
of statefun.

 

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[DISCUSS] (FLINK-26537) Allow disabling StatefulFunctionsConfigValidator validation for classloader.parent-first-patterns.additional

[Filip Karnicki]

Hi, we're having trouble running statefun jobs on a shared cloudera flink
cluster due to the presence of old protobuf-java classes on the classpath,
which don't work with statefun, causing MethodNotFound exceptions.

https://issues.apache.org/jira/browse/FLINK-26537
is to allow users to disable the verification, allowing the creators of
uber jars to include a version of protobuf-java that would be compatible
with the version of statefun that they're using.

I was thinking of adding a flink setting like "
statefun.validation.parent-first-classloader.disable=true", which would
suppress the validation in StatefulFunctionsConfigValidator#
validateParentFirstClassloaderPatterns

Please let me know if I can proceed with an implementation

[jira] [Created] (FLINK-26538) Ability to restart deployment w/o spec change

[Thomas Weise (Jira)]

Thomas Weise created FLINK-26538:


 Summary: Ability to restart deployment w/o spec change
 Key: FLINK-26538
 URL: https://issues.apache.org/jira/browse/FLINK-26538
 Project: Flink
  Issue Type: Sub-task
  Components: Kubernetes Operator
Reporter: Thomas Weise


Operator should allow restart of the Flink deployment w/o any other spec 
change. This provides the escape hatch for an operator to recover a deployment 
that has gone into a bad state (for whatever reason including memory leaks, 
hung JVM etc.) without direct access to the k8s cluster. This can be addressed 
by adding a restartNonce to the CRD.

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] New Apache Flink Committer - Martijn Visser

[Jing Zhang]

Congratulations, Martin!

Best,
Jing Zhang

Zhilong Hong  于2022年3月5日周六 01:17写道：

> Congratulations, Martin! Well deserved!
>
> Best,
> Zhilong
>
> On Sat, Mar 5, 2022 at 1:09 AM Piotr Nowojski 
> wrote:
>
> > Congratulations!
> >
> > Piotrek
> >
> > pt., 4 mar 2022 o 16:05 Aitozi  napisał(a):
> >
> > > Congratulations Martjin!
> > >
> > > Best,
> > > Aitozi
> > >
> > > Martijn Visser  于2022年3月4日周五 17:10写道：
> > >
> > > > Thank you all!
> > > >
> > > > On Fri, 4 Mar 2022 at 10:00, Niels Basjes  wrote:
> > > >
> > > > > Congratulations Martjin!
> > > > >
> > > > > On Fri, Mar 4, 2022 at 9:43 AM Johannes Moser 
> > > wrote:
> > > > >
> > > > > > Congratulations Martijn,
> > > > > >
> > > > > > Well deserved.
> > > > > >
> > > > > > > On 03.03.2022, at 16:49, Robert Metzger 
> > > wrote:
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > On behalf of the PMC, I'm very happy to announce Martijn Visser
> > as
> > > a
> > > > > new
> > > > > > > Flink committer.
> > > > > > >
> > > > > > > Martijn is a very active Flink community member, driving a lot
> of
> > > > > efforts
> > > > > > > on the dev@flink mailing list. He also pushes projects such as
> > > > > replacing
> > > > > > > Google Analytics with Matomo, so that we can generate our web
> > > > analytics
> > > > > > > within the Apache Software Foundation.
> > > > > > >
> > > > > > > Please join me in congratulating Martijn for becoming a Flink
> > > > > committer!
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Robert
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > Best regards / Met vriendelijke groeten,
> > > > >
> > > > > Niels Basjes
> > > > >
> > > >
> > >
> >
>

[jira] [Created] (FLINK-26539) Support dynamic partial features update in Cassandra connector

[Zhenqiu Huang (Jira)]

Zhenqiu Huang created FLINK-26539:
-

 Summary: Support dynamic partial features update in Cassandra 
connector
 Key: FLINK-26539
 URL: https://issues.apache.org/jira/browse/FLINK-26539
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Cassandra
Affects Versions: 1.14.3
Reporter: Zhenqiu Huang






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26540) Support handle join involving complex types in on condition

[luoyuxia (Jira)]

luoyuxia created FLINK-26540:


 Summary: Support handle join involving complex types in on 
condition
 Key: FLINK-26540
 URL: https://issues.apache.org/jira/browse/FLINK-26540
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive
Reporter: luoyuxia
 Fix For: 1.16.0


Now, the Hive dialect can't handle join involving complex types in on 
condition, which can be reproduced using the following code in 
HiveDialectITCase:

{code:java}
tableEnv.executeSql("CREATE TABLE test2a (a ARRAY)");
tableEnv.executeSql("CREATE TABLE test2b (a INT)");
List results =
CollectionUtil.iteratorToList(
tableEnv.executeSql(
"select *  from test2b join test2a on 
test2b.a = test2a.a[1]")
.collect());
{code}




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26541) SQL Client should support submitting SQL jobs in application mode

[Jark Wu (Jira)]

Jark Wu created FLINK-26541:
---

 Summary: SQL Client should support submitting SQL jobs in 
application mode
 Key: FLINK-26541
 URL: https://issues.apache.org/jira/browse/FLINK-26541
 Project: Flink
  Issue Type: Improvement
  Components: Deployment / Kubernetes, Deployment / YARN, Table SQL / 
Client
Reporter: Jark Wu


Currently, the SQL Client only supports submitting jobs in session mode and 
per-job mode. As the community going to drop the per-job mode (FLINK-26000), 
SQL Client should support application mode as well. Otherwise, SQL Client can 
only submit SQL in session mode then, but streaming jobs should be submitted in 
per-job or application mode to have bettter resource isolation.

Disucssions: https://lists.apache.org/thread/2yq351nb721x23rz1q8qlyf2tqrk147r



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26542) Fix the issue that schema of both sides of union won't match when one side of union select fields with same names

[luoyuxia (Jira)]

luoyuxia created FLINK-26542:


 Summary: Fix the issue that  schema of both sides of union won't 
match when one side of union select fields with same names
 Key: FLINK-26542
 URL: https://issues.apache.org/jira/browse/FLINK-26542
 Project: Flink
  Issue Type: Sub-task
  Components: Connectors / Hive
Reporter: luoyuxia
 Fix For: 1.16.0


With Hive dialect, for union SQL statement, when one side of union selects 
multiple fields with same name, the fields will be overwritten and then only 
retain one field. Then, it will throw the exception " Schema of both sides of 
union should match".
The exception can be reproduced using following code:

{code:java}
tableEnv.executeSql("create table src(key string, value string)");
tableEnv.executeSql("create table src2(key string, value string)");
List results =
queryResult(
tableEnv.sqlQuery(
"select key, value from src union all select 
key, key from src2"));
{code}




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [ANNOUNCE] New Apache Flink Committer - David Morávek

[Yu Li]

Congrats, David!

Best Regards,
Yu


On Tue, 8 Mar 2022 at 16:11, Johannes Moser  wrote:

> Very well deserved.
>
> > On 08.03.2022, at 05:43, Lincoln Lee  wrote:
> >
> > Congratulations David!
> >
> > Best,
> > Lincoln Lee
> >
> >
> > Yun Gao  于2022年3月8日周二 11:24写道：
> >
> >> Congratulations David!
> >>
> >> Best,
> >> Yun Gao
> >>
> >>
> >> --
> >> From:Jing Zhang 
> >> Send Time:2022 Mar. 8 (Tue.) 11:10
> >> To:dev 
> >> Subject:Re: [ANNOUNCE] New Apache Flink Committer - David Morávek
> >>
> >> Congratulations David!
> >>
> >> Ryan Skraba  于2022年3月7日周一 22:18写道：
> >>
> >>> Congratulations David!
> >>>
> >>> On Mon, Mar 7, 2022 at 9:54 AM Jan Lukavský  wrote:
> >>>
>  Congratulations David!
> 
>   Jan
> 
>  On 3/7/22 09:44, Etienne Chauchot wrote:
> > Congrats David !
> >
> > Well deserved !
> >
> > Etienne
> >
> > Le 07/03/2022 à 08:47, David Morávek a écrit :
> >> Thanks everyone!
> >>
> >> Best,
> >> D.
> >>
> >> On Sun 6. 3. 2022 at 9:07, Yuan Mei  wrote:
> >>
> >>> Congratulations, David!
> >>>
> >>> Best Regards,
> >>> Yuan
> >>>
> >>> On Sat, Mar 5, 2022 at 8:13 PM Roman Khachatryan  >>>
> >>> wrote:
> >>>
>  Congratulations, David!
> 
>  Regards,
>  Roman
> 
>  On Fri, Mar 4, 2022 at 7:54 PM Austin Cawley-Edwards
>   wrote:
> > Congrats David!
> >
> > On Fri, Mar 4, 2022 at 12:18 PM Zhilong Hong <
> >> zhlongh...@gmail.com
> 
>  wrote:
> >> Congratulations, David!
> >>
> >> Best,
> >> Zhilong
> >>
> >> On Sat, Mar 5, 2022 at 1:09 AM Piotr Nowojski <
> >>> pnowoj...@apache.org
> >
> >> wrote:
> >>
> >>> Congratulations :)
> >>>
> >>> pt., 4 mar 2022 o 16:04 Aitozi 
> >>> napisał(a):
> >>>
>  Congratulations David!
> 
>  Ingo Bürk  于2022年3月4日周五 22:56写道：
> 
> > Congrats, David!
> >
> > On 04.03.22 12:34, Robert Metzger wrote:
> >> Hi everyone,
> >>
> >> On behalf of the PMC, I'm very happy to announce David
> >>> Morávek
>  as a
> >>> new
> >> Flink committer.
> >>
> >> His first contributions to Flink date back to 2019. He has
> >>> been
> >> increasingly active with reviews and driving major
> >>> initiatives
>  in
> >> the
> >> community. David brings valuable experience from being a
>  committer
> >> in
>  the
> >> Apache Beam project to Flink.
> >>
> >>
> >> Please join me in congratulating David for becoming a Flink
> >>> committer!
> >> Cheers,
> >> Robert
> >>
> 
> >>>
> >>
>
>

Re: [ANNOUNCE] New Apache Flink Committer - David Morávek

[Jiangang Liu]

Congratulations, David.

Best
Liu Jiangang

Yu Li  于2022年3月9日周三 14:23写道：

> Congrats, David!
>
> Best Regards,
> Yu
>
>
> On Tue, 8 Mar 2022 at 16:11, Johannes Moser  wrote:
>
> > Very well deserved.
> >
> > > On 08.03.2022, at 05:43, Lincoln Lee  wrote:
> > >
> > > Congratulations David!
> > >
> > > Best,
> > > Lincoln Lee
> > >
> > >
> > > Yun Gao  于2022年3月8日周二 11:24写道：
> > >
> > >> Congratulations David!
> > >>
> > >> Best,
> > >> Yun Gao
> > >>
> > >>
> > >> --
> > >> From:Jing Zhang 
> > >> Send Time:2022 Mar. 8 (Tue.) 11:10
> > >> To:dev 
> > >> Subject:Re: [ANNOUNCE] New Apache Flink Committer - David Morávek
> > >>
> > >> Congratulations David!
> > >>
> > >> Ryan Skraba  于2022年3月7日周一 22:18写道：
> > >>
> > >>> Congratulations David!
> > >>>
> > >>> On Mon, Mar 7, 2022 at 9:54 AM Jan Lukavský  wrote:
> > >>>
> >  Congratulations David!
> > 
> >   Jan
> > 
> >  On 3/7/22 09:44, Etienne Chauchot wrote:
> > > Congrats David !
> > >
> > > Well deserved !
> > >
> > > Etienne
> > >
> > > Le 07/03/2022 à 08:47, David Morávek a écrit :
> > >> Thanks everyone!
> > >>
> > >> Best,
> > >> D.
> > >>
> > >> On Sun 6. 3. 2022 at 9:07, Yuan Mei 
> wrote:
> > >>
> > >>> Congratulations, David!
> > >>>
> > >>> Best Regards,
> > >>> Yuan
> > >>>
> > >>> On Sat, Mar 5, 2022 at 8:13 PM Roman Khachatryan <
> ro...@apache.org
> > >>>
> > >>> wrote:
> > >>>
> >  Congratulations, David!
> > 
> >  Regards,
> >  Roman
> > 
> >  On Fri, Mar 4, 2022 at 7:54 PM Austin Cawley-Edwards
> >   wrote:
> > > Congrats David!
> > >
> > > On Fri, Mar 4, 2022 at 12:18 PM Zhilong Hong <
> > >> zhlongh...@gmail.com
> > 
> >  wrote:
> > >> Congratulations, David!
> > >>
> > >> Best,
> > >> Zhilong
> > >>
> > >> On Sat, Mar 5, 2022 at 1:09 AM Piotr Nowojski <
> > >>> pnowoj...@apache.org
> > >
> > >> wrote:
> > >>
> > >>> Congratulations :)
> > >>>
> > >>> pt., 4 mar 2022 o 16:04 Aitozi 
> > >>> napisał(a):
> > >>>
> >  Congratulations David!
> > 
> >  Ingo Bürk  于2022年3月4日周五 22:56写道：
> > 
> > > Congrats, David!
> > >
> > > On 04.03.22 12:34, Robert Metzger wrote:
> > >> Hi everyone,
> > >>
> > >> On behalf of the PMC, I'm very happy to announce David
> > >>> Morávek
> >  as a
> > >>> new
> > >> Flink committer.
> > >>
> > >> His first contributions to Flink date back to 2019. He has
> > >>> been
> > >> increasingly active with reviews and driving major
> > >>> initiatives
> >  in
> > >> the
> > >> community. David brings valuable experience from being a
> >  committer
> > >> in
> >  the
> > >> Apache Beam project to Flink.
> > >>
> > >>
> > >> Please join me in congratulating David for becoming a
> Flink
> > >>> committer!
> > >> Cheers,
> > >> Robert
> > >>
> > 
> > >>>
> > >>
> >
> >
>

[jira] [Created] (FLINK-26543) Fix the issue that exceptions generated in startup is missed in Python loopback mode

[Huang Xingbo (Jira)]

Huang Xingbo created FLINK-26543:


 Summary: Fix the issue that exceptions generated in startup is 
missed in Python loopback mode
 Key: FLINK-26543
 URL: https://issues.apache.org/jira/browse/FLINK-26543
 Project: Flink
  Issue Type: Bug
  Components: API / Python
Affects Versions: 1.14.3, 1.15.0
Reporter: Huang Xingbo
Assignee: Huang Xingbo
 Fix For: 1.15.0, 1.14.4






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: status of Apple Silicon (M1) as Flink dev platform?

[Yun Tang]

Hi David,

For the problem of RocksDB running on M1 machines, RocksDB community has 
recently resolved this in RocksDB-6.29 [1], maybe we could involve this change 
in next Flink-1.16.


[1] https://github.com/facebook/rocksdb/pull/9662

Best
Yun Tang

From: David Anderson 
Sent: Tuesday, March 8, 2022 16:25
To: dev 
Subject: status of Apple Silicon (M1) as Flink dev platform?

What's the current status of using the Apple Silicon (M1) platform for
Flink development? Have we reached the point where everything "just works",
or do there remain lingering annoyances (or worse)?

In the past, I've seen reports of issues involving, e.g., RocksDB, nodejs,
protobuf, and pyflink. Looking in Jira, I see these issues haven't been
resolved yet, but I'm not sure what to read into that:

https://issues.apache.org/jira/browse/FLINK-24932 (Frocksdb cannot run on
Apple M1)
https://issues.apache.org/jira/browse/FLINK-25188 (Cannot install PyFlink
on MacOS with M1 chip)

Best,
David

[jira] [Created] (FLINK-26544) Apply key & value filter in FileStoreScanImpl

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-26544:
---

 Summary: Apply key & value filter in FileStoreScanImpl
 Key: FLINK-26544
 URL: https://issues.apache.org/jira/browse/FLINK-26544
 Project: Flink
  Issue Type: Sub-task
  Components: Table Store
Affects Versions: 0.1.0
Reporter: Caizhi Weng


Now that we've implemented sst file statistics in FLINK-26346, we can apply key 
& value filter in {{FileStoreScanImpl}}.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (FLINK-26545) Ingress rules should be created in the same namespace with FlinkDeployment CR

[Yang Wang (Jira)]

Yang Wang created FLINK-26545:
-

 Summary: Ingress rules should be created in the same namespace 
with FlinkDeployment CR
 Key: FLINK-26545
 URL: https://issues.apache.org/jira/browse/FLINK-26545
 Project: Flink
  Issue Type: Bug
  Components: Kubernetes Operator
Reporter: Yang Wang


Currently, the ingress rules are always created in the operator namespace(e.g. 
default). It could not work when the FlinkDeloyment CR is submitted in a 
different namespace(e.g. flink-test).

Refer to here[1] for why it could not work.

 

[1]. 
https://stackoverflow.com/questions/59844622/ingress-configuration-for-k8s-in-different-namespaces

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)