[jira] [Created] (FLINK-28605) Throw exception intentionally when new snapshots are committed during restore

[Caizhi Weng (Jira)]

Caizhi Weng created FLINK-28605:
---

 Summary: Throw exception intentionally when new snapshots are 
committed during restore
 Key: FLINK-28605
 URL: https://issues.apache.org/jira/browse/FLINK-28605
 Project: Flink
  Issue Type: Improvement
  Components: Table Store
Affects Versions: table-store-0.2.0
Reporter: Caizhi Weng
 Fix For: table-store-0.2.0


Currently snapshots are committed in {{notifyCheckpointComplete}}. If the job 
fails between a successful checkpoint and the call of 
{{notifyCheckpointComplete}}, these snapshots will be committed after job 
restarts.

However when the writer starts they also need to read from the latest snapshot 
(to build the latest structure of LSM tree). These two steps may happen 
concurrently and what the writers see may not be the latest snapshot.

To fix this problem, we can throw exception intentionally after new snapshots 
are committed during restore. In this way the job will be forcefully restarted 
and it is very likely that the writers can see the latest snapshot.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-28606) Preserve distributed consistency of OperatorEvents from OperatorCoordinator to subtasks

[Yunfeng Zhou (Jira)]

Yunfeng Zhou created FLINK-28606:


 Summary: Preserve distributed consistency of OperatorEvents from 
OperatorCoordinator to subtasks
 Key: FLINK-28606
 URL: https://issues.apache.org/jira/browse/FLINK-28606
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Checkpointing
Affects Versions: 1.14.3
Reporter: Yunfeng Zhou
 Fix For: 1.16.0


This is a component of our solution to the consistency issue in the operator 
coordinator mechanism. In this step, we would guarantee the consistency of all 
communications in one direction, from OC to subtasks. This would need less 
workload and should unblock the implementation of the CEP coordinator in 
FLIP-200.

Roughly, we would need to implement the following process in this step.
 # 
Let the OC finish processing all the incoming OperatorEvents before the 
snapshot.
 # 
Closes the gateway that sends operator events to its subtasks when the OC 
completes snapshot.
 # 
Wait until all the outgoing OperatorEvents before the snapshot are sent and 
acked.
 # 
Send checkpoint barriers to the Source operators.
 # 
Open the corresponding gateway of a subtask when the OC learned that the 
subtask has completed the checkpoint.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-28607) Table-API print connector encoding issue

[Sarah Cho (Jira)]

Sarah Cho created FLINK-28607:
-

 Summary: Table-API print connector encoding issue
 Key: FLINK-28607
 URL: https://issues.apache.org/jira/browse/FLINK-28607
 Project: Flink
  Issue Type: Bug
Reporter: Sarah Cho


I found out Table API's print connector has an encoding issue.

 

[https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/common/functions/util/PrintSinkOutputWriter.java#L54]

You can set PrintStream with encoding like this.

```java

PrintStream stream = new PrintStream(System.out, true, StandardCharsets.UTF_8);

```

 

Can I make a PR for this?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-28608) Make Hadoop FS token renewer configurable

[Gabor Somogyi (Jira)]

Gabor Somogyi created FLINK-28608:
-

 Summary: Make Hadoop FS token renewer configurable
 Key: FLINK-28608
 URL: https://issues.apache.org/jira/browse/FLINK-28608
 Project: Flink
  Issue Type: Sub-task
Affects Versions: 1.16.0
Reporter: Gabor Somogyi


Please see issue in gist: 
https://gist.github.com/JackWangCS/0b1ec2c1137c686ab874124569063234



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] FLIP-251: Support collecting arbitrary number of streams

[Chesnay Schepler]

I'd like to proceed with the vote for FLIP-251 [1], as no objections or issues 
were raised in [2].

The vote will last for at least 72 hours unless there is an objection or
insufficient votes.

 [1] 
https://cwiki.apache.org/confluence/display/FLINK/FLIP-251%3A+Support+collecting+arbitrary+number+of+streams
 [2] https://lists.apache.org/thread/ksv71m7rvcwslonw07h2qnw77zpqozvh

[SUMMARY] Flink 1.16 release sync of 2022-07-19

[Xingbo Huang]

Hi everyone,

I would like to give you a brief update of the Flink 1.16 release status
till 2022-07-19.

Currently, we have 12 features that have been completed for this release
and 41 features are still expected to make it. We only have 3 weeks
remaining until the feature freeze (at 2022-08-09).

We currently have one blocker ticket that is being worked on. Many thanks
to these contributors and reviewers.

Next, we also have some critical test stability tickets[1] that are not
picked up. We need to guarantee the CI stable before the release branch
cut. You can either directly assign it to yourself (don't forget to mark it
as In Progress) or ping me (@Huang Xingbo) to get it assigned to you. Your
help is much appreciated.

For more information about Flink release 1.16, you can refer to
https://cwiki.apache.org/confluence/display/FLINK/1.16+Release

The next Flink release sync will be on Tuesday the 26th of July at 9am
CEST/ 3pm China Standard Time / 7am UTC. The link can also be found on the
first page.

On behalf of all the release managers, best regards,

Xingbo

[1]
https://issues.apache.org/jira/issues/?jql=project%20%3D%20FLINK%20AND%20issuetype%20%3D%20Bug%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened)%20AND%20priority%20in%20(Blocker%2C%20Critical)%20AND%20labels%20%3D%20test-stability%20AND%20assignee%20in%20(EMPTY)%20ORDER%20BY%20updated%20DESC%2C%20summary%20DESC

[jira] [Created] (FLINK-28609) Flink-Pulsar connector fails on larger schemas

[Jacek Wislicki (Jira)]

Jacek Wislicki created FLINK-28609:
--

 Summary: Flink-Pulsar connector fails on larger schemas
 Key: FLINK-28609
 URL: https://issues.apache.org/jira/browse/FLINK-28609
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Pulsar
Affects Versions: 1.15.1, 1.14.5, 1.14.4, 1.14.3
Reporter: Jacek Wislicki
 Attachments: exception.txt

When a model results in a larger schema (this seems to be related to its byte 
array representation), the number of expected bytes to read is different than 
the number of actually read bytes: [^exception.txt]. The "read" is such a case 
is always 1018 while the expected "byteLen" gives a greater value. For smaller 
schemata, the numbers are equal (less than 1018) and no issue occurs.

The problem reproduction is on 
[GitHub|https://github.com/JacekWislicki/vp-test2]. There are 2 simple jobs 
(SimpleJob1 and SimpleJob2) using basic models for the Pulsar source definition 
(PulsarMessage1 and PulsarMessage2, respectively). Each of the corresponding 
schemata is properly serialised and deserialised, unless an effective byte 
array length becomes excessive (marked with "the problem begins" in model 
classes). The fail condition can be achieved by a number of fields 
(PulsarMessage1) or just longer field names (PulsarMessage2). The problem 
occurs on either Avro or a JSON schema set in the Pulsar source definition.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-28610) Enable speculative execution of sources

[Zhu Zhu (Jira)]

Zhu Zhu created FLINK-28610:
---

 Summary: Enable speculative execution of sources
 Key: FLINK-28610
 URL: https://issues.apache.org/jira/browse/FLINK-28610
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Zhu Zhu
 Fix For: 1.16.0


Currently speculative execution of sources is disabled. It can be enabled with 
the improvement done to support InputFormat sources and new sources to work 
correctly with speculative execution.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (FLINK-28611) Support ElementwiseProduct in FlinkML

[weibo zhao (Jira)]

weibo zhao created FLINK-28611:
--

 Summary: Support ElementwiseProduct in FlinkML
 Key: FLINK-28611
 URL: https://issues.apache.org/jira/browse/FLINK-28611
 Project: Flink
  Issue Type: New Feature
  Components: Library / Machine Learning
Reporter: weibo zhao


Support ElementwiseProduct in FlinkML.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-251: Support collecting arbitrary number of streams

[Martijn Visser]

Thanks for creating the FLIP and opening the vote Chesnay.

+1 (binding)

Op di 19 jul. 2022 om 10:26 schreef Chesnay Schepler :

> I'd like to proceed with the vote for FLIP-251 [1], as no objections or
> issues were raised in [2].
>
> The vote will last for at least 72 hours unless there is an objection or
> insufficient votes.
>
>   [1]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-251%3A+Support+collecting+arbitrary+number+of+streams
>   [2] https://lists.apache.org/thread/ksv71m7rvcwslonw07h2qnw77zpqozvh
>
>

[jira] [Created] (FLINK-28612) Cancel pending slot allocation after canceling executions

[Zhu Zhu (Jira)]

Zhu Zhu created FLINK-28612:
---

 Summary: Cancel pending slot allocation after canceling executions
 Key: FLINK-28612
 URL: https://issues.apache.org/jira/browse/FLINK-28612
 Project: Flink
  Issue Type: Sub-task
  Components: Runtime / Coordination
Reporter: Zhu Zhu
 Fix For: 1.16.0


Canceling pending slot allocation before canceling executions will result in 
execution failures  and pollute the logs. It will also result in an execution 
to be FAILED even if the execution vertex has FINISHED, which breaks the 
assumption of SpeculativeScheduler#isExecutionVertexPossibleToFinish().



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] FLIP-247 Bulk fetch of table and column statistics for given partitions

[Jing Ge]

Thanks Jingsong for the suggestion.

Do you mean using a different naming convention? There is a thought and
description in the FLIP about using "list" or "bulkGet":

   - bulkGetPartitionStatistics(...) has been chosen over
   listPartitionStatistics(...), because, comparing to database and partition
   that are static and can be listed, statistics are more dynamic and will
   need more computation logic to create, therefore using "get" is
   semantically more feasible than list. The "bulk" gives users the hint that
   this method will work in the bulk mode and return a collection of instances.


As a reference, we can see that no method in MetaStoreClient, that
calculates statistics, uses the "list" naming convention.

Best regards,
Jing

On Fri, Jul 15, 2022 at 5:38 AM Jingsong Li  wrote:

> Thanks for starting this discussion.
>
> Have we considered introducing a listPartitionWithStats() in Catalog?
>
> Best,
> Jingsong
>
> On Fri, Jul 15, 2022 at 10:08 AM Jark Wu  wrote:
> >
> > Hi Jing,
> >
> > Thanks for starting this discussion. The bulk fetch is a great
> improvement
> > for the optimizer.
> > The FLIP looks good to me.
> >
> > Best,
> > Jark
> >
> > On Fri, 8 Jul 2022 at 17:36, Jing Ge  wrote:
> >
> > > Hi devs,
> > >
> > > After having multiple discussions with Jark and Goldfrey, I'd like to
> start
> > > a discussion on the mailing list w.r.t. FLIP-247[1], which will
> > > significantly improve the performance by providing the bulk fetch
> > > capability for table and column statistics.
> > >
> > > Currently the statistics information about tables can only be fetched
> from
> > > the catalog by each given partition iteratively. Since getting
> statistics
> > > information from catalogs is a very heavy operation, in order to
> improve
> > > the query performance, we’d better provide functionality to fetch the
> > > statistics information of a table for all given partitions in one shot.
> > >
> > > Based on the manual performance test, for 2000 partitions, the cost
> will be
> > > improved from 10s to 2s. The improvement result is 500%.
> > >
> > > [1]
> > >
> > >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-247%3A+Bulk+fetch+of+table+and+column+statistics+for+given+partitions
> > >
> > > Best regards,
> > > Jing
> > >
>

[jira] [Created] (FLINK-28613) PyFlink 1.15 unable to start in Application Mode in k8s

[Clive Wong (Jira)]

Clive Wong created FLINK-28613:
--

 Summary: PyFlink 1.15 unable to start in Application Mode in k8s
 Key: FLINK-28613
 URL: https://issues.apache.org/jira/browse/FLINK-28613
 Project: Flink
  Issue Type: Bug
  Components: Client / Job Submission
Affects Versions: 1.15.1
Reporter: Clive Wong


I recently bumped my PyFlink job from 1.14 to 1.15, and the job is failing with 
build 1.15 in k8s.

The error is due to NetUtils not able to getAvailablePort. I suspect this is 
related to the version bump of py4j from 0.10.8.1 to 0.10.9.3 in required by 
apache-flink 1.15 in python.

The error stack is:
{code:java}
2022-07-19 11:17:06,225 INFO  
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - 
Start SessionDispatcherLeaderProcess.
2022-07-19 11:17:06,226 INFO  
org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl [] - 
Starting resource manager service.
2022-07-19 11:17:06,227 INFO  
org.apache.flink.runtime.resourcemanager.ResourceManagerServiceImpl [] - 
Resource manager service is granted leadership with session id 
----.
2022-07-19 11:17:06,229 INFO  
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - 
Recover all persisted job graphs that are not finished, yet.
2022-07-19 11:17:06,229 INFO  
org.apache.flink.runtime.dispatcher.runner.SessionDispatcherLeaderProcess [] - 
Successfully recovered 0 persisted job graphs.
2022-07-19 11:17:06,306 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService  
           [] - Starting RPC endpoint for 
org.apache.flink.runtime.dispatcher.StandaloneDispatcher at 
akka://flink/user/rpc/dispatcher_0 .
2022-07-19 11:17:06,309 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService  
           [] - Starting RPC endpoint for 
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager at 
akka://flink/user/rpc/resourcemanager_1 .
2022-07-19 11:17:06,317 INFO  
org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - 
Starting the resource manager.
2022-07-19 11:17:06,401 INFO  org.apache.flink.client.ClientUtils               
           [] - Starting program (detached: true)
2022-07-19 11:17:06,500 WARN  
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap 
[] - Application failed unexpectedly: 
java.util.concurrent.CompletionException: 
org.apache.flink.client.deployment.application.ApplicationExecutionException: 
Could not execute application.
    at 
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
 ~[?:?]
    at 
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
 ~[?:?]
    at 
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063)
 ~[?:?]
    at 
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) 
~[?:?]
    at 
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088)
 ~[?:?]
    at 
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:323)
 ~[flink-dist-1.15.0-stream1.jar:1.15.0-stream1]
    at 
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAsync$2(ApplicationDispatcherBootstrap.java:244)
 ~[flink-dist-1.15.0-stream1.jar:1.15.0-stream1]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) 
~[?:?]
    at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
    at 
org.apache.flink.runtime.concurrent.akka.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:171)
 ~[flink-rpc-akka_73d9230b-9d22-4143-8bbc-2ab5d539166f.jar:1.15.0-stream1]
    at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
 ~[flink-rpc-akka_73d9230b-9d22-4143-8bbc-2ab5d539166f.jar:1.15.0-stream1]
    at 
org.apache.flink.runtime.concurrent.akka.ClassLoadingUtils.lambda$withContextClassLoader$0(ClassLoadingUtils.java:41)
 ~[flink-rpc-akka_73d9230b-9d22-4143-8bbc-2ab5d539166f.jar:1.15.0-stream1]
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49) 
[flink-rpc-akka_73d9230b-9d22-4143-8bbc-2ab5d539166f.jar:1.15.0-stream1]
    at 
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:48)
 [flink-rpc-akka_73d9230b-9d22-4143-8bbc-2ab5d539166f.jar:1.15.0-stream1]
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
    at 
java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
 [?:?]
    at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
    at 
java.util.concurrent.ForkJoinWorkerThread.run(For

[DISCUSS] TLS issues

[Jean-Damien Hatzenbuhler]

Hello, 

I created two JIRA tickets regarding TLS issues in flink:
- https://issues.apache.org/jira/browse/FLINK-28520 

- https://issues.apache.org/jira/browse/FLINK-28521 


I didn’t receive any reaction except affected version and priority.
I’m following our guidelines and reach dev mailing list to get news.

I got the following question for you: 
- Is there an issue in my proposed solutions ?
- If not can I open PRs to solve those issues ?
- For those PRs, I would like to backport correction in version 1.13.x, is this 
something you will allow ?

Thank you in advance for your time

Regards,

Jean-Damien HATZENBUHLER

Re: [DISCUSS] TLS issues

[Martijn Visser]

Hi Jean-Damien,

First of all, thanks for filing those tickets. We would very much
appreciate getting PRs to improve on this. I've assigned them to you.

With regards to backports, this depends on the compatibility of the PRs. If
they would not break anything that we can't break in a patch release,
backports can be possible. However, the Flink 1.13 release is officially
not supported anymore. That doesn't hurt with regards to backporting, but
that most likely means that there won't be another official patch release
for that.

Best regards,

Martijn

Op di 19 jul. 2022 om 15:55 schreef Jean-Damien Hatzenbuhler
:

> Hello,
>
> I created two JIRA tickets regarding TLS issues in flink:
> - https://issues.apache.org/jira/browse/FLINK-28520 <
> https://issues.apache.org/jira/browse/FLINK-28520>
> - https://issues.apache.org/jira/browse/FLINK-28521 <
> https://issues.apache.org/jira/browse/FLINK-28521>
>
> I didn’t receive any reaction except affected version and priority.
> I’m following our guidelines and reach dev mailing list to get news.
>
> I got the following question for you:
> - Is there an issue in my proposed solutions ?
> - If not can I open PRs to solve those issues ?
> - For those PRs, I would like to backport correction in version 1.13.x, is
> this something you will allow ?
>
> Thank you in advance for your time
>
> Regards,
>
> Jean-Damien HATZENBUHLER

Re: [DISCUSS] TLS issues

[Jean-Damien Hatzenbuhler]

Hello Martijn,

Thank you I will propose PRs to correct those issues.

Regards,

Jean-Damien HATZENBUHLER

On Tue, Jul 19, 2022 at 5:04 PM Martijn Visser 
wrote:

> Hi Jean-Damien,
>
> First of all, thanks for filing those tickets. We would very much
> appreciate getting PRs to improve on this. I've assigned them to you.
>
> With regards to backports, this depends on the compatibility of the PRs. If
> they would not break anything that we can't break in a patch release,
> backports can be possible. However, the Flink 1.13 release is officially
> not supported anymore. That doesn't hurt with regards to backporting, but
> that most likely means that there won't be another official patch release
> for that.
>
> Best regards,
>
> Martijn
>
> Op di 19 jul. 2022 om 15:55 schreef Jean-Damien Hatzenbuhler
> :
>
> > Hello,
> >
> > I created two JIRA tickets regarding TLS issues in flink:
> > - https://issues.apache.org/jira/browse/FLINK-28520 <
> > https://issues.apache.org/jira/browse/FLINK-28520>
> > - https://issues.apache.org/jira/browse/FLINK-28521 <
> > https://issues.apache.org/jira/browse/FLINK-28521>
> >
> > I didn’t receive any reaction except affected version and priority.
> > I’m following our guidelines and reach dev mailing list to get news.
> >
> > I got the following question for you:
> > - Is there an issue in my proposed solutions ?
> > - If not can I open PRs to solve those issues ?
> > - For those PRs, I would like to backport correction in version 1.13.x,
> is
> > this something you will allow ?
> >
> > Thank you in advance for your time
> >
> > Regards,
> >
> > Jean-Damien HATZENBUHLER
>

Re: [VOTE] FLIP-251: Support collecting arbitrary number of streams

[Alexander Fedulov]

+1
Looking forward to using the API to simplify tests setups.

Best,
Alexander Fedulov

On Tue, Jul 19, 2022 at 2:31 PM Martijn Visser 
wrote:

> Thanks for creating the FLIP and opening the vote Chesnay.
>
> +1 (binding)
>
> Op di 19 jul. 2022 om 10:26 schreef Chesnay Schepler :
>
> > I'd like to proceed with the vote for FLIP-251 [1], as no objections or
> > issues were raised in [2].
> >
> > The vote will last for at least 72 hours unless there is an objection or
> > insufficient votes.
> >
> >   [1]
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-251%3A+Support+collecting+arbitrary+number+of+streams
> >   [2] https://lists.apache.org/thread/ksv71m7rvcwslonw07h2qnw77zpqozvh
> >
> >
>

[VOTE] FLIP-238: Introduce FLIP-27-based Data Generator Source

[Alexander Fedulov]

Hi everyone,

following the discussion in [1], I would like to open up a vote for
adding a FLIP-27-based Data Generator Source [2].

The addition of this source also unblocks the currently pending
efforts for deprecating the Source Function API [3].

The poll will be open until July 25 (72h + weekend), unless there is
an objection or not enough votes.

[1] https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt
[2] https://cwiki.apache.org/confluence/x/9Av1D
[3] https://github.com/apache/flink/pull/20049#issuecomment-1170948767

Best,
Alexander Fedulov

Re: [VOTE] FLIP-238: Introduce FLIP-27-based Data Generator Source

[Martijn Visser]

+1 (binding)

Thanks for the efforts Alex!

Op di 19 jul. 2022 om 21:31 schreef Alexander Fedulov <
alexan...@ververica.com>:

> Hi everyone,
>
> following the discussion in [1], I would like to open up a vote for
> adding a FLIP-27-based Data Generator Source [2].
>
> The addition of this source also unblocks the currently pending
> efforts for deprecating the Source Function API [3].
>
> The poll will be open until July 25 (72h + weekend), unless there is
> an objection or not enough votes.
>
> [1] https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt
> [2] https://cwiki.apache.org/confluence/x/9Av1D
> [3] https://github.com/apache/flink/pull/20049#issuecomment-1170948767
>
> Best,
> Alexander Fedulov
>

[DISCUSS] Handling of removed splits for FLIP-27

[Xinbin Huang]

Hi everyone,

I would like to start a discussion about state recovery of removed splits
for the source API (FLIP-27). By looking at the implementation and behavior
we observed, it seems that the source reader doesn't correctly handle a
split being discovered as "removed" from the enumerator. Instead, the
reader would still read from these removed splits because the reader is
getting those splits from the state[1].

For example, if Kafka source is subscribing to a list of topics, and later
on remove one topic from the subscribed list. This would lead to some
unexpected behavior. And I think this can be considered as a bug. To fix
this, one solution would be adding a method (i.e. removeSplit)
SplitEnumeratorContext to signal these removed splits to the reader.

Any thoughts?

[1]:
https://github.com/apache/flink/blob/180774e93902862cf3bfa03de00437ae49d743eb/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/operators/SourceOperator.java#L313-L316

Best
Bin Huang

Re: [DISCUSS] FLIP-247 Bulk fetch of table and column statistics for given partitions

[Jingsong Li]

Hi Jing,

I understand that the statistics for partitions are currently only
used by Hive, so we can look at the Hive implementation:

See HiveCatalog.getPartitionStatistics.
To get the statistics, we actually get them from the
org.apache.hadoop.hive.metastore.api.Partition object.

According to HiveMetastore's API, partition-related operations
actually get the partition as well as the statistics information.

So if the current partition statistics are just for Hive, can we
consider unifying it with Hive?

For example, in PushPartitionIntoTableSourceScanRule, just use
`listPartitionWithStats`, and adjust table statistics from partitions.

Best,
Jingsong

On Tue, Jul 19, 2022 at 8:44 PM Jing Ge  wrote:
>
> Thanks Jingsong for the suggestion.
>
> Do you mean using a different naming convention? There is a thought and
> description in the FLIP about using "list" or "bulkGet":
>
>- bulkGetPartitionStatistics(...) has been chosen over
>listPartitionStatistics(...), because, comparing to database and partition
>that are static and can be listed, statistics are more dynamic and will
>need more computation logic to create, therefore using "get" is
>semantically more feasible than list. The "bulk" gives users the hint that
>this method will work in the bulk mode and return a collection of 
> instances.
>
>
> As a reference, we can see that no method in MetaStoreClient, that
> calculates statistics, uses the "list" naming convention.
>
> Best regards,
> Jing
>
> On Fri, Jul 15, 2022 at 5:38 AM Jingsong Li  wrote:
>
> > Thanks for starting this discussion.
> >
> > Have we considered introducing a listPartitionWithStats() in Catalog?
> >
> > Best,
> > Jingsong
> >
> > On Fri, Jul 15, 2022 at 10:08 AM Jark Wu  wrote:
> > >
> > > Hi Jing,
> > >
> > > Thanks for starting this discussion. The bulk fetch is a great
> > improvement
> > > for the optimizer.
> > > The FLIP looks good to me.
> > >
> > > Best,
> > > Jark
> > >
> > > On Fri, 8 Jul 2022 at 17:36, Jing Ge  wrote:
> > >
> > > > Hi devs,
> > > >
> > > > After having multiple discussions with Jark and Goldfrey, I'd like to
> > start
> > > > a discussion on the mailing list w.r.t. FLIP-247[1], which will
> > > > significantly improve the performance by providing the bulk fetch
> > > > capability for table and column statistics.
> > > >
> > > > Currently the statistics information about tables can only be fetched
> > from
> > > > the catalog by each given partition iteratively. Since getting
> > statistics
> > > > information from catalogs is a very heavy operation, in order to
> > improve
> > > > the query performance, we’d better provide functionality to fetch the
> > > > statistics information of a table for all given partitions in one shot.
> > > >
> > > > Based on the manual performance test, for 2000 partitions, the cost
> > will be
> > > > improved from 10s to 2s. The improvement result is 500%.
> > > >
> > > > [1]
> > > >
> > > >
> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-247%3A+Bulk+fetch+of+table+and+column+statistics+for+given+partitions
> > > >
> > > > Best regards,
> > > > Jing
> > > >
> >

Re: [VOTE] FLIP-238: Introduce FLIP-27-based Data Generator Source

[Rui Fan]

+1(non-binding)

New Source can better support some features, such as
Unaligned Checkpoint, Watermark alignment, etc.
The data generator based on the new Source is very helpful
for daily testing.

Very much looking forward to using it.

Best wishes
Rui Fan

On Wed, Jul 20, 2022 at 4:22 AM Martijn Visser 
wrote:

> +1 (binding)
>
> Thanks for the efforts Alex!
>
> Op di 19 jul. 2022 om 21:31 schreef Alexander Fedulov <
> alexan...@ververica.com>:
>
> > Hi everyone,
> >
> > following the discussion in [1], I would like to open up a vote for
> > adding a FLIP-27-based Data Generator Source [2].
> >
> > The addition of this source also unblocks the currently pending
> > efforts for deprecating the Source Function API [3].
> >
> > The poll will be open until July 25 (72h + weekend), unless there is
> > an objection or not enough votes.
> >
> > [1] https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt
> > [2] https://cwiki.apache.org/confluence/x/9Av1D
> > [3] https://github.com/apache/flink/pull/20049#issuecomment-1170948767
> >
> > Best,
> > Alexander Fedulov
> >
>

邮件退订

[cason0126]

邮件退订


| |
cason0126
|
|
cason0...@163.com
|

[jira] [Created] (FLINK-28614) Empty local state folders not cleanup on retrieving local state

[Yanfei Lei (Jira)]

Yanfei Lei created FLINK-28614:
--

 Summary: Empty local state folders not cleanup on retrieving local 
state
 Key: FLINK-28614
 URL: https://issues.apache.org/jira/browse/FLINK-28614
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.15.1, 1.15.0, 1.16.0
Reporter: Yanfei Lei
 Fix For: 1.16.0


It would create a checkpoint directory when trying to load 
{{TaskStateSnapshot}} from the disk. The local checkpoint directory is not 
deleted on exit {{tryLoadTaskStateSnapshotFromDisk() }}even though 
{{TaskStateSnapshot}} doesn't exist. 

 
{code:java}
File getTaskStateSnapshotFile(long checkpointId) {
final File checkpointDirectory =
localRecoveryConfig
.getLocalStateDirectoryProvider()
.orElseThrow(
() -> new IllegalStateException("Local recovery 
must be enabled."))
.subtaskSpecificCheckpointDirectory(checkpointId);

if (!checkpointDirectory.exists() && !checkpointDirectory.mkdirs()) {
throw new FlinkRuntimeException(
String.format(
"Could not create the checkpoint directory '%s'", 
checkpointDirectory));
}

return new File(checkpointDirectory, TASK_STATE_SNAPSHOT_FILENAME);
} {code}
 

 

This will cause the folder in /{{{}localState{}}} to remain after failover. 
Here is an example: 
{code:java}
41854 [flink-akka.actor.default-dispatcher-8] INFO  
org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Restoring job 
35644df535ca04613d6a6116dcfcfd59 from Checkpoint 2 @ 1658292943408 for 
35644df535ca04613d6a6116dcfcfd59 located at 
file:/var/folders/4n/q3r37vws2f910rt_f469kwg0gn/T/junit1426665332205293555/junit63847204117629783/35644df535ca04613d6a6116dcfcfd59/chk-2.

___
directory of localState
___ 
tm_2
    │   ├── blobStorage
    │   ├── localState
    │   │   └── aid_6df21e53ca06ea69ee0643d25d27dbee
    │   │       └── jid_35644df535ca04613d6a6116dcfcfd59
    │   │           └── vtx_0a448493b4782967b150582570326227_sti_1
    │   │               ├── chk_2
    │   │               └── chk_5
    │   │                   ├── _task_state_snapshot
    │   │                   ├── edab98058083464a9ca29b6d7a950c68
    │   │                   │   ├── 14.sst
    │   │                   │   ├── 15.sst
    │   │                   │   ├── 22.sst
    │   │                   │   ├── 23.sst
    │   │                   │   ├── CURRENT
    │   │                   │   ├── MANIFEST-18
    │   │                   │   └── OPTIONS-21
    │   │                   └── f3724ae6-fd24-4e9a-80a8-02aa34bca0f0 {code}
cc: [~trohrmann] , [~masteryhx] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] FLIP-238: Introduce FLIP-27-based Data Generator Source

[Robert Metzger]

+1

On Wed, Jul 20, 2022 at 4:41 AM Rui Fan <1996fan...@gmail.com> wrote:

> +1(non-binding)
>
> New Source can better support some features, such as
> Unaligned Checkpoint, Watermark alignment, etc.
> The data generator based on the new Source is very helpful
> for daily testing.
>
> Very much looking forward to using it.
>
> Best wishes
> Rui Fan
>
> On Wed, Jul 20, 2022 at 4:22 AM Martijn Visser 
> wrote:
>
> > +1 (binding)
> >
> > Thanks for the efforts Alex!
> >
> > Op di 19 jul. 2022 om 21:31 schreef Alexander Fedulov <
> > alexan...@ververica.com>:
> >
> > > Hi everyone,
> > >
> > > following the discussion in [1], I would like to open up a vote for
> > > adding a FLIP-27-based Data Generator Source [2].
> > >
> > > The addition of this source also unblocks the currently pending
> > > efforts for deprecating the Source Function API [3].
> > >
> > > The poll will be open until July 25 (72h + weekend), unless there is
> > > an objection or not enough votes.
> > >
> > > [1] https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt
> > > [2] https://cwiki.apache.org/confluence/x/9Av1D
> > > [3] https://github.com/apache/flink/pull/20049#issuecomment-1170948767
> > >
> > > Best,
> > > Alexander Fedulov
> > >
> >
>

Re: 邮件退订

[yuxia]

To unsubscribe, you can send any email to dev-unsubscr...@flink.apache.org

Best regards,
Yuxia

- 原始邮件 -
发件人: "cason0126" 
收件人: "dev" 
发送时间: 星期三, 2022年 7 月 20日 上午 11:49:33
主题: 邮件退订

邮件退订


| |
cason0126
|
|
cason0...@163.com
|

Re: [DISCUSS] FLIP-252: Amazon DynamoDB Sink Connector

[Robert Metzger]

Thanks a lot for this nice proposal!

DynamoDB seems to be a connector that Flink is still lacking, and with the
Async Sink interface, it seems that we can implement this fairly easily.

+1 to proceed to the formal vote for this FLIP!

On Fri, Jul 15, 2022 at 7:51 PM Danny Cranmer 
wrote:

> Hello all,
>
> We would like to start a discussion thread on FLIP-252: Amazon DynamoDB
> Sink Connector [1] where we propose to provide a sink connector for Amazon
> DynamoDB [2] based on the Async Sink [3]. Looking forward to comments and
> feedback. Thank you.
>
> [1]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-252%3A+Amazon+DynamoDB+Sink+Connector
> [2] https://aws.amazon.com/dynamodb
> [3]
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink
>

Re: [VOTE] FLIP-238: Introduce FLIP-27-based Data Generator Source

[Jing Ge]

+1(non-binding)

Thanks for driving this!

Best regards,
Jing

On Wed, Jul 20, 2022 at 7:48 AM Robert Metzger  wrote:

> +1
>
> On Wed, Jul 20, 2022 at 4:41 AM Rui Fan <1996fan...@gmail.com> wrote:
>
> > +1(non-binding)
> >
> > New Source can better support some features, such as
> > Unaligned Checkpoint, Watermark alignment, etc.
> > The data generator based on the new Source is very helpful
> > for daily testing.
> >
> > Very much looking forward to using it.
> >
> > Best wishes
> > Rui Fan
> >
> > On Wed, Jul 20, 2022 at 4:22 AM Martijn Visser  >
> > wrote:
> >
> > > +1 (binding)
> > >
> > > Thanks for the efforts Alex!
> > >
> > > Op di 19 jul. 2022 om 21:31 schreef Alexander Fedulov <
> > > alexan...@ververica.com>:
> > >
> > > > Hi everyone,
> > > >
> > > > following the discussion in [1], I would like to open up a vote for
> > > > adding a FLIP-27-based Data Generator Source [2].
> > > >
> > > > The addition of this source also unblocks the currently pending
> > > > efforts for deprecating the Source Function API [3].
> > > >
> > > > The poll will be open until July 25 (72h + weekend), unless there is
> > > > an objection or not enough votes.
> > > >
> > > > [1] https://lists.apache.org/thread/7gjxto1rmkpff4kl54j8nlg5db2rqhkt
> > > > [2] https://cwiki.apache.org/confluence/x/9Av1D
> > > > [3]
> https://github.com/apache/flink/pull/20049#issuecomment-1170948767
> > > >
> > > > Best,
> > > > Alexander Fedulov
> > > >
> > >
> >
>

[ANNOUNCE] Table Store release-0.2 branch cut

[Jingsong Li]

Hi Flink devs!

The version on the main branch has been updated to 0.3-SNAPSHOT.

The release-0.2 branch has been forked from main:
https://github.com/apache/flink-table-store/tree/release-0.2

Documentation:
https://nightlies.apache.org/flink/flink-table-store-docs-release-0.2/

The most important thing about Table Store 0.2 is that it enriches the
Table Store ecosystem. Now not only Flink 1.15, but also Flink 1.14,
Hive 2.3, Spark 2.4, Spark 3+, and Trino are all supported.

More features are: Catalog (Including Hive Metastore), Rescale bucket,
Append-only mode, Improved documentation etc..

You are very welcome to test it!

We will try to prepare the first RC in the next week.

Best,
Jingsong