[jira] [Created] (FLINK-37108) Source/Sink test suites test invalid recovery

2025-01-13 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-37108: --- Summary: Source/Sink test suites test invalid recovery Key: FLINK-37108 URL: https://issues.apache.org/jira/browse/FLINK-37108 Project: Flink Issue Type

[jira] [Created] (FLINK-36807) Add coverage for pre-writer and pre-commit topologies for SinkV2

2024-11-27 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-36807: --- Summary: Add coverage for pre-writer and pre-commit topologies for SinkV2 Key: FLINK-36807 URL: https://issues.apache.org/jira/browse/FLINK-36807 Project: Flink

[ANNOUNCE] Apache flink-connector-kafka 3.4.0 released

2024-11-26 Thread Arvid Heise
ards, Arvid Heise

[RESULT][VOTE] flink-connector-kafka 3.4.0, release candidate #1

2024-11-25 Thread Arvid Heise
I'm happy to announce that we have unanimously approved this release. There are 5 approving votes, 3 of which are binding: * Yanquan Lv * Leonard Xu (binding) * Martijn Visser (binding) * Ahmed Hamdy * Danny Cranmer (binding) There are no disapproving votes. Thanks everyone! Arvid

Re: Subject: [VOTE] Release flink-connector-kafka v3.4.0, release candidate #1

2024-11-25 Thread Arvid Heise
gt; > > > > > > > > On Nov 15, 2024, at 10:00 AM, Yanquan Lv > > wrote: > > > > > > > > > > Thanks Arvid for driving it. > > > > > > > > > > +1 (non-binding) > > > > > I checked: >

[jira] [Created] (FLINK-36788) Add coverage for GlobalCommitter for SinkV2

2024-11-25 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-36788: --- Summary: Add coverage for GlobalCommitter for SinkV2 Key: FLINK-36788 URL: https://issues.apache.org/jira/browse/FLINK-36788 Project: Flink Issue Type: Bug

Re: [VOTE] Release flink-connector-hbase v4.0.0, release candidate #1

2024-11-22 Thread Arvid Heise
+1 (binding) - Built from sources with JDK 11 and executed all tests successfully, also against 1.19.1 distribution - Verified checksum and signature on sources - No binaries in sources - Double-check flink-web PR - Verified artifacts - Checked JIRA release notes On Wed, Nov 20, 2024 at 3:22 PM G

Re: Kafka connector releases

2024-11-14 Thread Arvid Heise
e of flink-2.0-preview of Flink Kafka > connector, and I’d like assist it too. > > Best, > Leonard > > > > 2024年11月6日 下午6:58,Arvid Heise 写道: > > > > Hi Yanquan, > > > > the current state of the 3.4.0 release is that it's still pending on the

Subject: [VOTE] Release flink-connector-kafka v3.4.0, release candidate #1

2024-11-14 Thread Arvid Heise
Hi everyone, Please review and vote on release candidate #1 for flink-connector-kafka v3.4.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release

Re: Kafka connector releases

2024-11-06 Thread Arvid Heise
to know if there is already work in the community to bump to > 2.0-preview1. If not, I can help complete this task (but some suggestions > may be needed for testing the adaptation in the code). > > > > > > > 2024年9月27日 16:23,Arvid Heise 写道: > > > > Dear Flink de

Re: Micro batching with flink

2024-11-05 Thread Arvid Heise
Hey, could you please check if a bucket assigner is already enough? If not, what's missing? FileSink orcSink = FileSink .forBulkFormat(new Path("s3a://mybucket/flink_file_sink_orc_test"), factory) .withBucketAssigner(new DateTimeBucketAssigner<>("'dt='MMdd/'hour='HH",

[ANNOUNCE] Apache flink-connector-kafka 3.3.0 released

2024-10-17 Thread Arvid Heise
The Apache Flink community is very happy to announce the release of Apache flink-connector-kafka 3.3.0. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming applications. The release is available for downlo

[RESULT][VOTE] flink-connector-kafka 3.3.0, release candidate #1

2024-10-16 Thread Arvid Heise
I'm happy to announce that we have unanimously approved this release. There are 3 approving votes, 3 of which are binding: * Matthias Pohl (binding) * Rui Fan (binding) * Leonard Xu (binding) We also had approving votes for the version with incorrect snapshot names: * Yanquan Lv * Ahmed Hamdy * D

Re: [VOTE] Release flink-connector-kafka v3.3.0, release candidate #1

2024-10-16 Thread Arvid Heise
> * Diff of git tag checkout with downloaded sources > > * Verified SHA512 checksums & GPG certification > > * Checked that all POMs have the right expected version > > * Generated diffs to compare pom file changes with NOTICE files > > > > Thanks Arvid. Looks good

Re: [VOTE] Release flink-connector-kafka v3.3.0, release candidate #1

2024-10-14 Thread Arvid Heise
tent/repositories/staging/org/apache/flink/flink-connector-kafka/3.3.0-1.19/ [2] https://repository.apache.org/content/repositories/staging/org/apache/flink/flink-connector-kafka/3.3.0-1.20/ On Tue, Oct 15, 2024 at 8:21 AM Arvid Heise wrote: > That's a good catch Leonard. I'll ch

Re: [VOTE] Release flink-connector-kafka v3.3.0, release candidate #1

2024-10-14 Thread Arvid Heise
s and checksums are correct > >> - There are no binaries in the source archive > >> - Contents of mvn dist look good > >> - Binary signatures and checksums are correct > >> - CI build of tag successful [1] > >> - NOTICE and LICENSE files look correct &

Re: [VOTE] Release flink-connector-kafka v3.3.0, release candidate #1

2024-10-14 Thread Arvid Heise
gt; > > Best Regards > Ahmed Hamdy > > > On Sat, 12 Oct 2024 at 05:23, Yanquan Lv wrote: > > > +1 (non-binding) > > I checked: > > - Review JIRA release notes > > - Verify hashes > > - Verify signatures > > - Build from source with JDK 8/11/17

[VOTE] Release flink-connector-kafka v3.3.0, release candidate #1

2024-10-11 Thread Arvid Heise
new release [6]. * CI build of the tag [7]. The vote will be open for at least 72 hours. It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Arvid Heise [1] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12354606 [2] h

[jira] [Created] (FLINK-36455) Sink should commit everything on notifyCheckpointCompleted

2024-10-09 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-36455: --- Summary: Sink should commit everything on notifyCheckpointCompleted Key: FLINK-36455 URL: https://issues.apache.org/jira/browse/FLINK-36455 Project: Flink

[jira] [Created] (FLINK-36454) FlinkContainer always overwrites default config

2024-10-09 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-36454: --- Summary: FlinkContainer always overwrites default config Key: FLINK-36454 URL: https://issues.apache.org/jira/browse/FLINK-36454 Project: Flink Issue Type

[jira] [Created] (FLINK-36441) Resource leaks in Kafka Tests

2024-10-08 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-36441: --- Summary: Resource leaks in Kafka Tests Key: FLINK-36441 URL: https://issues.apache.org/jira/browse/FLINK-36441 Project: Flink Issue Type: Bug Affects

[jira] [Created] (FLINK-36434) Revise threading model of (KafkaPartition)SplitReader

2024-10-07 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-36434: --- Summary: Revise threading model of (KafkaPartition)SplitReader Key: FLINK-36434 URL: https://issues.apache.org/jira/browse/FLINK-36434 Project: Flink Issue

Re: Kafka connector releases

2024-09-30 Thread Arvid Heise
not dropping support for 1.18 in 3.2.1, > and we release one more version 3.2.1-1.20? Then we can use 3.3.0 for the > new lineage feature in 1.20 and drop support for 1.18 and 1.19. > > This is a possibility, but still results in 3 releases, so only worth it if > it is simpl

Re: Kafka connector releases

2024-09-30 Thread Arvid Heise
n use > > 3.3.0 for the new lineage feature in 1.20 and drop support for 1.18 > > and 1.19. > > > > And for the 4.0.0-preview version I'd like to help with it :-) > > > > Best, > > Qingsheng > > > > On Fri, Sep 27, 2024 at 6:13 PM Arvid He

Re: Kafka connector releases

2024-09-27 Thread Arvid Heise
ould it be worth have a config switch to enable the lineage in the > connector so that we could use it with 1.19? We could maybe do a 3.3 if > this was the case. > > WDYT? >Kind regards, David. > > > > From: Arvid Heise > Date: Friday, 27 September 2024 at 09:24

Kafka connector releases

2024-09-27 Thread Arvid Heise
Dear Flink devs, I'd like to initiate three(!) Kafka connector releases. The main reason for having three releases is that we have been slacking a bit in keeping up with the latest changes. Here is the summary: 1. Release kafka-3.3.0 targeting 1.19 and 1.20 (asap) - Incorporates lots of deprecati

Re: [VOTE] FLIP-477: Amazon SQS Source Connector

2024-09-27 Thread Arvid Heise
+1 (binding), Best, Arvid On Tue, Sep 3, 2024 at 12:35 PM Saurabh Singh wrote: > Hi Flink Devs, > > Gentle Reminder for voting on FLIP-477: Amazon SQS Source Connector [1]. > [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-477 > +Amazon+SQS+Source+Connector > > Regards > Saurabh & A

Re: [DISCUSS] FLIP-XXX Amazon SQS Source Connector

2024-09-27 Thread Arvid Heise
. > Updated Google Doc Link - > https://docs.google.com/document/d/1lreo27jNh0LkRs1Mj9B3wj3itrzMa38D4_XGryOIFks/edit?usp=sharing > > Thanks > Saurabh & Abhi > > > ----- > *Slack Conversation Details **s

[jira] [Created] (FLINK-36379) Improve (Global)Committer with UC disabled

2024-09-26 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-36379: --- Summary: Improve (Global)Committer with UC disabled Key: FLINK-36379 URL: https://issues.apache.org/jira/browse/FLINK-36379 Project: Flink Issue Type: Bug

[jira] [Created] (FLINK-36368) Fix subtask management in CommittableCollector

2024-09-25 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-36368: --- Summary: Fix subtask management in CommittableCollector Key: FLINK-36368 URL: https://issues.apache.org/jira/browse/FLINK-36368 Project: Flink Issue Type

Re: [DISCUSS] FLIP-XXX Apicurio-avro format

2024-09-24 Thread Arvid Heise
Hi Robert, thanks for looping me in. I have looked at the branch and the FLIP. Apicurio looks like a promising alternative to Confluent SR and I'm certain that it's a good addition to Flink. However, at the current form it looks heavily overengineered. I'm suspecting that comes from the attempt t

Re: [DISCUSS] FLIP-XXX Amazon SQS Source Connector

2024-09-16 Thread Arvid Heise
we will support this functionality in > the future. However, in the meantime, this can still be achieved by > creating multiple sources with specific queues.* > > > Please review and let us know your feedback on this. > > [1] > https://docs.aws.amazon.com/AWSSimpleQueue

[jira] [Created] (FLINK-36287) Sink with topologies should not participate in UC

2024-09-16 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-36287: --- Summary: Sink with topologies should not participate in UC Key: FLINK-36287 URL: https://issues.apache.org/jira/browse/FLINK-36287 Project: Flink Issue Type

[jira] [Created] (FLINK-36278) Fix Kafka connector logs getting too big

2024-09-13 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-36278: --- Summary: Fix Kafka connector logs getting too big Key: FLINK-36278 URL: https://issues.apache.org/jira/browse/FLINK-36278 Project: Flink Issue Type: Technical

Re: [DISCUSS] FLIP-XXX Amazon SQS Source Connector

2024-09-12 Thread Arvid Heise
che/flink/api/common/state/CheckpointListener.html > *[2] * > https://github.com/apache/flink/blob/master//flink-core/src/main/java/org/apache/flink/api/connector/source/DynamicParallelismInference.java#L29 > *[3]* > https://github.com/aws/aws-sdk-java/blob/61d73631fac8535ad70666bbce9e70

Re: [DISCUSS] FLIP-XXX Amazon SQS Source Connector

2024-09-03 Thread Arvid Heise
Sorry for being late to the party. I saw your call to vote and looked at the FLIP. First, most of the design is looking really good and it will be good to have another connector integrated into the AWS ecosystem. A couple of questions/remarks: 1) Since we only have 1 split, we should also limit th

[jira] [Created] (FLINK-36177) Deprecating KafkaShuffle

2024-08-29 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-36177: --- Summary: Deprecating KafkaShuffle Key: FLINK-36177 URL: https://issues.apache.org/jira/browse/FLINK-36177 Project: Flink Issue Type: Technical Debt

[jira] [Created] (FLINK-36176) Remove support for ancient Kafka versions

2024-08-29 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-36176: --- Summary: Remove support for ancient Kafka versions Key: FLINK-36176 URL: https://issues.apache.org/jira/browse/FLINK-36176 Project: Flink Issue Type

Re: [DISCUSS] FLIP-471: Fixing watermark idleness timeout accounting

2024-07-26 Thread Arvid Heise
Maybe the java doc should be clarified? > > > > > > > 4. I now realized that I changed semantics compared to the proposal: > > this > > > > idle clock would already calculate the time difference (now - last > > > event). > > > >

Re: [DISCUSS] FLIP-471: Fixing watermark idleness timeout accounting

2024-07-24 Thread Arvid Heise
Hi Piotr, thank you very much for addressing this issue. I'm convinced that the approach is the right solution also in contrast to the alternatives. Ultimately, only WatermarkGenratorWithIdleness needs to be adjusted with this change. My only concerns are regarding the actual code. 1. `RelativeCl

[jira] [Created] (FLINK-35796) Ensure that MailboxExecutor.submit is used correctly

2024-07-09 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-35796: --- Summary: Ensure that MailboxExecutor.submit is used correctly Key: FLINK-35796 URL: https://issues.apache.org/jira/browse/FLINK-35796 Project: Flink Issue

Re: [DISCUSS] FLIP-221 Abstraction for lookup source cache and metric

2022-04-25 Thread Arvid Heise
Hi Qingsheng, Thanks for driving this; the inconsistency was not satisfying for me. I second Alexander's idea though but could also live with an easier solution as the first step: Instead of making caching an implementation detail of TableFunction X, rather devise a caching layer around X. So the

Re: [DISCUSS] FLIP-217 Support watermark alignment of source splits

2022-04-25 Thread Arvid Heise
watermark. I > thought the problem is that we may have late data for this slow > > partition. > > I have another question about the restart. Say split alignment is > triggered. checkpoint is completed. job failed and restored from the last > checkpoint. because alignment deci

Re: [ANNOUNCE] New Apache Flink Committer - David Morávek

2022-03-14 Thread Arvid Heise
Gratz! On Mon, Mar 14, 2022 at 9:24 AM Anton Kalashnikov wrote: > Congratulations, David! > > -- > > Best regards, > Anton Kalashnikov > > 14.03.2022 09:18, Matthias Pohl пишет: > > Congratulations, David! > >

Re: [ANNOUNCE] New PMC member: Yuan Mei

2022-03-14 Thread Arvid Heise
Congratulations and well deserved! On Mon, Mar 14, 2022 at 9:30 AM Matthias Pohl wrote: > Congratulations, Yuan. > > On Mon, Mar 14, 2022 at 9:25 AM Shuo Cheng wrote: > > > Congratulations, Yuan! > > > > On Mon, Mar 14, 2022 at 4:22 PM Anton Kalashnikov > > wrote: > > > > > Congratulations, Yu

Re: [VOTE] FLIP-191: Extend unified Sink interface to support small file compaction

2022-01-04 Thread Arvid Heise
+1 (binding). Thanks for driving! On Tue, Jan 4, 2022 at 10:31 AM Yun Gao wrote: > +1 (binding). > > Very thanks for proposing the FLIP! > > Best, > Yun > > > -- > From:Martijn Visser > Send Time:2022 Jan. 4 (Tue.) 17:22 > To:dev

Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

2021-12-16 Thread Arvid Heise
s well, so the FLIP is not making anything worse. On Thu, Dec 16, 2021 at 1:53 PM Arvid Heise wrote: > Hi Yun, > > the basic idea is to use different regions for the different stages. > > So you have your stages: > Stage 1: -> pre-writer topology -> writer -> co

Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

2021-12-16 Thread Arvid Heise
t do it without breaking the Sink > > > >> > > > >> interface since the GlobalCommittables are part of the parameterized > > > >> > > > >> Sink interface. Thus, we propose building a new Sink V2 interface > > > >> > > > >> consisting

Re: Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

2021-12-16 Thread Arvid Heise
> > > >> consisting of composable interfaces that do not offer the > > > >> > > > >> GlobalCommitter anymore. We will implement a utility to extend a > Sink > > > >> > > > >> with post topology that mimics the behavior of t

Re: [DISCUSS] Creating an external connector repository

2021-12-09 Thread Arvid Heise
hat utilities and infrastructure code do you intend to share? Using git > > submodules can definitely be one option to share code. However, it might > > also be ok to depend on flink-connector-common artifacts which could make > > things easier. Where I am unsure is whether git submodu

Re: [DISCUSS] Drop Zookeeper 3.4

2021-12-06 Thread Arvid Heise
Could someone please help me understand the implications of the upgrade? As far as I understood this upgrade would only affect users that have a zookeeper shared across multiple services, some of which require ZK 3.4-? A workaround for those users would be to run two ZKs with different versions, e

Re: [VOTE] Deprecate Java 8 support

2021-12-06 Thread Arvid Heise
+1 (binding) On Mon, Dec 6, 2021 at 5:43 PM Timo Walther wrote: > +1 (binding) > > Thanks, > Timo > > On 06.12.21 17:28, David Morávek wrote: > > +1 (non-binding) > > > > On Mon, Dec 6, 2021 at 4:55 PM Ingo Bürk wrote: > > > >> +1 (non-binding) > >> > >> > >> Ingo > >> > >> On Mon, Dec 6, 2021

Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

2021-11-29 Thread Arvid Heise
> > > One way to avoid write-read-merge is by wrapping SinkWriter with > > another one, which would buffer input elements in a temporary storage > > (e.g. local file) until a threshold is reached; after that, it would > > invoke the original SinkWriter. And if a checkpoint barrier comes in > > earl

Re: [DISCUSS] Creating an external connector repository

2021-11-25 Thread Arvid Heise
https://github.com/pravega/flink-connectors/releases/tag/v0.10.1 > [4] https://search.maven.org/search?q=pravega-connectors-flink > > Best Regards, > Brian > > > Internal Use - Confidential > > -Original Message- > From: Arvid Heise > Sent: Friday, November

Re: [DISCUSS] Deprecate Java 8 support

2021-11-25 Thread Arvid Heise
+1 to deprecate Java 8, so we can hopefully incorporate the module concept in Flink. On Thu, Nov 25, 2021 at 9:49 AM Chesnay Schepler wrote: > Users can already use APIs from Java 8/11. > > On 25/11/2021 09:35, Francesco Guardiani wrote: > > +1 with what both Ingo and Matthias sad, personally, I

Re: [DISCUSS] Creating an external connector repository

2021-11-19 Thread Arvid Heise
ink community. [1] https://github.com/apache/flink-connectors [2] https://github.com/ververica/flink-cdc-connectors/ On Fri, Nov 12, 2021 at 3:39 PM Arvid Heise wrote: > Hi everyone, > > I created the flink-connectors repo [1] to advance the topic. We would > create a proof-of-concept in the ne

[ANNOUNCE] New Apache Flink Committer - Fabian Paul

2021-11-15 Thread Arvid Heise
Hi everyone, On behalf of the PMC, I'm very happy to announce Fabian Paul as a new Flink committer. Fabian Paul has been actively improving the connector ecosystem by migrating Kafka and ElasticSearch to the Sink interface and is currently driving FLIP-191 [1] to tackle the sink compaction issue.

Re: [DISCUSS] Conventions on assertions to use in tests

2021-11-12 Thread Arvid Heise
y > started with Junit4, then we chose to use Hamcrest because of its better > expressiveness. Most recently, there was an effort started that aimed at > switching over to Junit5 [1, 2]. @Arvid Heise knows > more about the current status. > > Personally, I don't have a strong p

Re: [DISCUSS] Creating an external connector repository

2021-11-12 Thread Arvid Heise
g range that fits "more frequent than Flink" > (per-commit, > > > > daily, > > > > > > >> > weekly, bi-weekly, monthly, even bi-monthly). > > > > > > >> > > > > > > > >> > On 19/10/2021 14:15, Martij

Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

2021-11-04 Thread Arvid Heise
e no longer works after FLIP-147 when executing the > final checkpoint. The problem is that the final checkpoint happens after > the EOI and we would like to keep the property that you can terminate the > whole topology with a single checkpoint, if possible. > > Cheers, > Till

Re: [DISCUSS] FLIP-191: Extend unified Sink interface to support small file compaction

2021-11-04 Thread Arvid Heise
Hi folks, thanks for the lively discussion. Let me present my point of view on a couple of items: *Impact on checkpointing times* Currently, we send the committables of the writer downstream before the barrier is sent. That allows us to include all committables in the state of the committers, su

Re: [NOTICE] flink-streaming-java no longer depends on Scala and lost it's suffix

2021-10-26 Thread Arvid Heise
Awesome. Thank you very much for all the hard work! On Tue, Oct 26, 2021 at 1:06 AM Chesnay Schepler wrote: > This time with proper formatting... > > flink-batch-sql-test > flink-cep > flink-cli-test > flink-clients > flink-connector-elasticsearch-base > flink-connector-elasticsearch5 > flink-co

Re: [DISCUSS] Creating an external connector repository

2021-10-26 Thread Arvid Heise
o > > > >> a > > > >> > >> net improvement. > > > >> > >> > > > >> > >> It would be great if we can find a setup that allows for > > connectors > > > >> to > > > >> > >> be r

Re: [DISCUSS] Creating an external connector repository

2021-10-19 Thread Arvid Heise
back to a previous version of the snapshot. Which > > also means that checking out older commits can be problematic because > > you'd still work against the latest snapshots, and they not be > > compatible with each other. > > > > > > On 18/10/2021 15:22, Arvid Heise wrote: > > > I was actually betting on snapshots versions. What are the limits? > > > Obviously, we can only do a release of a 1.15 connector after 1.15 is > > > release. > > > > >

Re: [VOTE] Release 1.13.3, release candidate #1

2021-10-19 Thread Arvid Heise
+1 (binding) - build from source on scala 2_12 profile - ran standalone cluster with examples Best, Arvid On Tue, Oct 19, 2021 at 4:48 AM Dian Fu wrote: > +1 (binding) > > - verified the checksum and signature > - checked the dependency changes since 1.13.2. There is only one dependency > cha

Re: [DISCUSS] Creating an external connector repository

2021-10-18 Thread Arvid Heise
e new established >> flink-extended org might be another choice, but considering the amount of >> connectors, I prefer to use an individual org for connectors to avoid >> flushing other repos under flink-extended. >> >> In the meantime, we need to provide a well-esta

[jira] [Created] (FLINK-24574) Allow metrics to be removed in UDFs

2021-10-18 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-24574: --- Summary: Allow metrics to be removed in UDFs Key: FLINK-24574 URL: https://issues.apache.org/jira/browse/FLINK-24574 Project: Flink Issue Type: Improvement

Re: dataStream can not use multiple classloaders

2021-10-18 Thread Arvid Heise
You also must ensure that your SourceFunction is serializable, so it's not enough to just refer to some classloader, you must ensure that you have access to it also after deserialization on the task managers. On Mon, Oct 18, 2021 at 4:24 AM Caizhi Weng wrote: > Hi! > > There is only one classloa

[DISCUSS] Creating an external connector repository

2021-10-15 Thread Arvid Heise
Dear community, Today I would like to kickstart a series of discussions around creating an external connector repository. The main idea is to decouple the release cycle of Flink with the release cycles of the connectors. This is a common approach in other big data analytics projects and seems to s

Re: [NOTICE] CiBot improvements

2021-10-11 Thread Arvid Heise
Awesome! On Tue, Oct 12, 2021 at 3:11 AM Guowei Ma wrote: > Thanks for your effort! > > Best, > Guowei > > > On Mon, Oct 11, 2021 at 9:26 PM Stephan Ewen wrote: > > > Great initiative, thanks for doing this! > > > > On Mon, Oct 11, 2021 at 10:52 AM Till Rohrmann > > wrote: > > > > > Thanks a l

Re: The Apache Flink should pay more attention to ensuring API compatibility.

2021-10-04 Thread Arvid Heise
t; > If we change it back, then a specific connector would work for 1.14.1 > and > > 1.13.X but not for 1.14.0 and this would be even more confusing. > > I think this is fine. IMO, this is a blocker issue of 1.14.0 which breaks > > Source connectors. > > We should sugges

Re: The Apache Flink should pay more attention to ensuring API compatibility.

2021-10-01 Thread Arvid Heise
ny idea >> to >> fix it. We should >> fix it ASAP! Otherwise iceberg/hudi/cdc communities will get frustrated >> again when upgrading >> to 1.14. Maybe we still have time to release a minor version, because >> there is no >> connector upgraded to 1.14.0 yet.

Re: The Apache Flink should pay more attention to ensuring API compatibility.

2021-09-28 Thread Arvid Heise
Thanks for starting the discussion. I think both issues are valid concerns that we need to tackle. I guess the biggest issue is that now it's just not possible to write 1 connector that runs for Flink 1.13 and 1.14, so we make it much harder for devs in the ecosystem (and our goal is to make it eas

[jira] [Created] (FLINK-24252) HybridSource should return TypeInformation

2021-09-10 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-24252: --- Summary: HybridSource should return TypeInformation Key: FLINK-24252 URL: https://issues.apache.org/jira/browse/FLINK-24252 Project: Flink Issue Type

[jira] [Created] (FLINK-24250) Add De/Serialization API to tear-down user code

2021-09-10 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-24250: --- Summary: Add De/Serialization API to tear-down user code Key: FLINK-24250 URL: https://issues.apache.org/jira/browse/FLINK-24250 Project: Flink Issue Type

Re: [DISCUSS] FLIP-182: Watermark alignment

2021-09-07 Thread Arvid Heise
my opinion, this limitation is perfectly fine for the MVP. > > > > Watermark > > > > > > > alignment is a long-standing issue and this already moves the > > ball > > > so > > > > > far > > > > > > > forward. > > > &g

[jira] [Created] (FLINK-24182) Tasks canceler should not immediately interrupt

2021-09-07 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-24182: --- Summary: Tasks canceler should not immediately interrupt Key: FLINK-24182 URL: https://issues.apache.org/jira/browse/FLINK-24182 Project: Flink Issue Type

Re: [ANNOUNCE] Apache Flink Stateful Functions 3.1.0 released

2021-09-06 Thread Arvid Heise
Congratulations! New features look awesome. On Wed, Sep 1, 2021 at 9:10 AM Till Rohrmann wrote: > Great news! Thanks a lot for all your work on the new release :-) > > Cheers, > Till > > On Wed, Sep 1, 2021 at 9:07 AM Johannes Moser wrote: > >> Congratulations, great job. 🎉 >> >> On 31.08.2021,

[jira] [Created] (FLINK-24108) Translate KafkaSink documentation to Chinese

2021-09-01 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-24108: --- Summary: Translate KafkaSink documentation to Chinese Key: FLINK-24108 URL: https://issues.apache.org/jira/browse/FLINK-24108 Project: Flink Issue Type

Re: [VOTE] Apache Flink Stateful Functions 3.1.0, release candidate #1

2021-08-26 Thread Arvid Heise
+1 (binding) - Built from downloaded sources with Java 8 (mvn install -Prun-e2e-tests) - Verified signatures and hashes Best, Arvid On Thu, Aug 26, 2021 at 8:32 AM Tzu-Li (Gordon) Tai wrote: > +1 (binding) > > - Built from source with Java 11 and Java 8 (mvn clean install > -Prun-e2e-tests) >

[jira] [Created] (FLINK-23969) Test Pulsar source end 2 end

2021-08-25 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-23969: --- Summary: Test Pulsar source end 2 end Key: FLINK-23969 URL: https://issues.apache.org/jira/browse/FLINK-23969 Project: Flink Issue Type: Sub-task

[DISCUSS] Merge Kafka-related PRs after feature freeze

2021-08-18 Thread Arvid Heise
Dear devs, we would like to merge these PRs after features freeze: FLINK-23838: Add FLIP-33 metrics to new KafkaSink [1] FLINK-23801: Add FLIP-33 metrics to KafkaSource [2] FLINK-23640: Create a KafkaRecordSerializationSchemas builder [3] All of the 3 PRs are smaller quality of life improvements

[jira] [Created] (FLINK-23858) Clarify StreamRecord#timestamp.

2021-08-18 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-23858: --- Summary: Clarify StreamRecord#timestamp. Key: FLINK-23858 URL: https://issues.apache.org/jira/browse/FLINK-23858 Project: Flink Issue Type: Technical Debt

[jira] [Created] (FLINK-23817) Write documentation for standardized operator metrics

2021-08-16 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-23817: --- Summary: Write documentation for standardized operator metrics Key: FLINK-23817 URL: https://issues.apache.org/jira/browse/FLINK-23817 Project: Flink Issue

Re: [DISCUSS] Backport HybridSource to 1.13.x

2021-08-16 Thread Arvid Heise
cation jar. It does not require > any addition to the runtime and therefore would work on any 1.13.x dist. > > For reference, I use it internally on top of 1.12.4. > > Thanks, > Thomas > > > On Mon, Aug 16, 2021 at 10:13 AM Arvid Heise wrote: > > > Hi Thoma

[jira] [Created] (FLINK-23816) Test new (Kafka) metrics in cluster

2021-08-16 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-23816: --- Summary: Test new (Kafka) metrics in cluster Key: FLINK-23816 URL: https://issues.apache.org/jira/browse/FLINK-23816 Project: Flink Issue Type: Sub-task

Re: [DISCUSS] Backport HybridSource to 1.13.x

2021-08-16 Thread Arvid Heise
Hi Thomas, since the change didn't modify any existing classes, I'm weakly in favor of backporting. My reluctance mainly stems from possible disappointments from 1.13 users that use an earlier bugfix level. So we need to make documentation clear. In general, I'm seeing connectors as something ext

[jira] [Created] (FLINK-23807) Use metrics to detect restarts in MiniClusterTestEnvironment#triggerTaskManagerFailover

2021-08-16 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-23807: --- Summary: Use metrics to detect restarts in MiniClusterTestEnvironment#triggerTaskManagerFailover Key: FLINK-23807 URL: https://issues.apache.org/jira/browse/FLINK-23807

[jira] [Created] (FLINK-23803) Improve instantiation of InMemoryReporter

2021-08-16 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-23803: --- Summary: Improve instantiation of InMemoryReporter Key: FLINK-23803 URL: https://issues.apache.org/jira/browse/FLINK-23803 Project: Flink Issue Type

[jira] [Created] (FLINK-23767) FLIP-180: Adjust StreamStatus and Idleness definition

2021-08-13 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-23767: --- Summary: FLIP-180: Adjust StreamStatus and Idleness definition Key: FLINK-23767 URL: https://issues.apache.org/jira/browse/FLINK-23767 Project: Flink Issue

[RESULT][VOTE] FLIP-180: Adjust StreamStatus and Idleness definition

2021-08-13 Thread Arvid Heise
; > On Mon, Aug 9, 2021 at 12:08 PM Till Rohrmann > > > wrote: > > > > > > > +1 (binding) > > > > > > > > Cheers, > > > > Till > > > > > > > > On Thu, Aug 5, 2021 at 9:09 PM Arvid Heise wrote: > >

[jira] [Created] (FLINK-23759) notifyCheckpointComplete without corresponding snapshotState

2021-08-13 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-23759: --- Summary: notifyCheckpointComplete without corresponding snapshotState Key: FLINK-23759 URL: https://issues.apache.org/jira/browse/FLINK-23759 Project: Flink

[jira] [Created] (FLINK-23722) S3 Tests fail on AZP: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region.

2021-08-11 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-23722: --- Summary: S3 Tests fail on AZP: Unable to find a region via the region provider chain. Must provide an explicit region in the builder or setup environment to supply a region. Key: FLINK-23722

Re: [ANNOUNCE] Apache Flink 1.12.5 released

2021-08-10 Thread Arvid Heise
Thank you. On Tue, Aug 10, 2021 at 11:44 AM Till Rohrmann wrote: > This is great news. Thanks a lot for being our release manager Jingsong > and also to everyone who made this release possible. > > Cheers, > Till > > On Tue, Aug 10, 2021 at 10:57 AM Jingsong Lee > wrote: > >> The Apache Flink c

Re: [ANNOUNCE] Apache Flink 1.11.4 released

2021-08-10 Thread Arvid Heise
Awesome! Thank you for driving this. On Tue, Aug 10, 2021 at 11:45 AM Till Rohrmann wrote: > This is great news. Thanks a lot for being our release manager Godfrey and > also to everyone who made this release possible. > > Cheers, > Till > > On Tue, Aug 10, 2021 at 11:09 AM godfrey he wrote: >

Re: [ANNOUNCE] Apache Flink 1.13.2 released

2021-08-10 Thread Arvid Heise
Thank you! On Tue, Aug 10, 2021 at 11:04 AM Jingsong Li wrote: > Thanks Yun Tang and everyone! > > Best, > Jingsong > > On Tue, Aug 10, 2021 at 9:37 AM Xintong Song > wrote: > >> Thanks Yun and everyone~! >> >> Thank you~ >> >> Xintong Song >> >> >> >> On Mon, Aug 9, 2021 at 10:14 PM Till Rohrm

[jira] [Created] (FLINK-23652) Implement FLIP-179

2021-08-05 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-23652: --- Summary: Implement FLIP-179 Key: FLINK-23652 URL: https://issues.apache.org/jira/browse/FLINK-23652 Project: Flink Issue Type: Improvement

Re: [DISCUSS] FLIP-180: Adjust StreamStatus and Idleness definition

2021-08-05 Thread Arvid Heise
FLIP as small as possible. If there is a good reason > to expose the WatermarkStatus, then we can probably do it. > > Cheers, > Till > > On Fri, Jul 30, 2021 at 2:29 PM Arvid Heise > wrote: > > > Hi Martijn, > > 1. Good question. The watermarks and statuses o

[VOTE] FLIP-180: Adjust StreamStatus and Idleness definition

2021-08-05 Thread Arvid Heise
Dear devs, I'd like to open a vote on FLIP-180: Adjust StreamStatus and Idleness definition [1] which was discussed in this thread [2]. The vote will be open for at least 72 hours unless there is an objection or not enough votes. Best, Arvid [1] https://cwiki.apache.org/confluence/display/FLINK

[jira] [Created] (FLINK-23621) FLIP-177: Extend Sink API

2021-08-04 Thread Arvid Heise (Jira)
Arvid Heise created FLINK-23621: --- Summary: FLIP-177: Extend Sink API Key: FLINK-23621 URL: https://issues.apache.org/jira/browse/FLINK-23621 Project: Flink Issue Type: Improvement

  1   2   3   4   >