Re: KafkaSource vs FlinkKafkaConsumer010

The latter link you posted refers to a very old flink release. You shold use the first link, which refers to latest release FG On Tue, Feb 1, 2022 at 8:20 AM HG wrote: > Hello all > > I am confused. > What is the difference between KafkaSource as defined in : > https://nightlies.apache.org/flin

Re: MetricRegistryTestUtils java class (flink-runtime/metrics) not found in source code version 1.14.3

Hi Prasanna, Just a quick note that the Github links are all pointing to release candidate 1 for 1.14.3. The released version is in https://github.com/apache/flink/blob/release-1.14.3/flink-runtime/src/test/java/org/apache/flink/runtime/metrics/MetricRegistryTestUtils.java Best regards, Martijn

KafkaSource vs FlinkKafkaConsumer010

Hello all I am confused. What is the difference between KafkaSource as defined in : https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/datastream/kafka/ and FlinkKafkaConsumer010 as defined in https://nightlies.apache.org/flink/flink-docs-release- 1.2/api/java/org/apache/fl

Re: Flink 1.14 metrics : taskmanager host name missing

Hi, Can someone please take a look at this? It's a major blocker for us. Thanks, Mayur On Fri, Jan 21, 2022 at 2:11 PM Mayur Gubrele wrote: > Hello, > > We recently upgraded our Flink cluster to 1.14 and noticed that all the > taskmanager metrics we receive in our Prometheus data source get ho

Re: Tumbling window apply will not "fire"

ok it's working! Thanks. Just out of curiosity, why is the println of keyBy printing twice? On Mon, 31 Jan 2022 at 17:22, John Smith wrote: > Oh ok. I was reading here: > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/learn-flink/streaming_analytics/#latency-vs-completeness > an

Re: Tumbling window apply will not "fire"

Oh ok. I was reading here: https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/learn-flink/streaming_analytics/#latency-vs-completeness and Idid a cut and paste lol Ok let you know. On Mon, 31 Jan 2022 at 17:18, Dario Heinisch wrote: > Then you should be using a process based time w

Re: Tumbling window apply will not "fire"

Then you should be using a process based time window, in your case: TumblingProcessingTimeWindows See https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/ for more info On 31.01.22 23:13, John Smith wrote: Hi Dario, I don't care about event time I just

Re: Tumbling window apply will not "fire"

Hi Dario, I don't care about event time I just want to do tumbling window over the "processing time" I.e: count whatever I have in the last 5 minutes. On Mon, 31 Jan 2022 at 17:09, Dario Heinisch wrote: > Hi John > > This is because you are using event time (TumblingEventTimeWinodws) but > you d

Re: Tumbling window apply will not "fire"

Hi John This is because you are using event time (TumblingEventTimeWinodws) but you do not have a event time watermark strategy. It is also why I opened: https://issues.apache.org/jira/browse/FLINK-24623 because I feel like Flink should be throwing an exception in that case on startup. Take

Read Avro type records from kafka using Python - Datastream API

Hello, I am currently working on program that uses flink to read avro type records from kafka. I have the avro schema of the records I want to read in a file but I looked all over github, the documentation and stack Overflow for examples on how to use AvroRowDeserializationSchema to deserialize

Tumbling window apply will not "fire"

Hi I have the following job... I'm expecting the System.out .println(key.toString()); to at least print, but nothing prints. - .flatMap: Fires prints my debug message once as expected. - .keyBy: Also fires, but prints my debug message twice. - .apply: Doesn't seem to fire. The debug statement do

Re: Kafka Consumer Group Name not set if no checkpointing?

Hi yes, see below. So it only seems to show the consumer offsets if checkpointing is on... That's the only diff I can see between my two different jobs. And the moment I enabled it on the job. It started showing in Kafka Explorer here: https://www.kafkatool.com/ return KafkaSource.builder()

Re: Resolving a CatalogTable

Hi Timo, Thanks for the reply. I've thought a bit more about the problem, considering your points. This is not critical as of now, but for the sake of discussion, I think it could be interesting. The problem stems from the fact that we don't create the table via DDL, but the following custom meth

Re: [DISCUSS] Deprecate/remove Twitter connector

https://golb.hplar.ch/2018/02/Access-Server-Sent-Events-from-Java.html looks like a nice tutorial. On Mon, Jan 31, 2022 at 12:27 PM Andrew Otto wrote: > Any SSE/EventSource Java Client should work. I have not personally used > one. From a quick search, maybe > https://github.com/launchdarkly/o

Re: [DISCUSS] Deprecate/remove Twitter connector

Any SSE/EventSource Java Client should work. I have not personally used one. From a quick search, maybe https://github.com/launchdarkly/okhttp-eventsource or something like it? On Mon, Jan 31, 2022 at 11:45 AM Francesco Guardiani < france...@ververica.com> wrote: > > Shameless plug: Maybe th

Re: How to put external data in EmbeddedRocksDB

Hi Surendra, I do not think there is an out-of-the-box way to do look-ups to the local rocksdb instance. In general, I am a bit skeptical about whether it is a good idea to use the rocksdb instance for your state management, and the looks up in parallel. It may overload the rocksdb and cause unexp

Re: Flink 1.14.2/3 - KafkaSink vs deprecated FlinkKafkaProducer

Hi Daniel, Thanks for reaching out, we are constantly trying to improve the reliability of our connectors. I assume you are running the KafkaSink with exactly-once delivery guarantee. On startup, the KafkaSink tries to abort lingering transactions from previous executions. Unfortunately, nothing c

Re: Kafka Consumer Group Name not set if no checkpointing?

Hi John, First I would like to ask you to give us some more information about how you consume from Kafka with Flink. It is currently recommended to use the KafkaSource that subsumes the deprecated FlinkKafkaConsumer. One thing to already note is that by default Flink does not commit the Kafka to o

Re: [DISCUSS] Deprecate/remove Twitter connector

> Shameless plug: Maybe the Wikipedia EventStreams SSE API would make for a great connector example in Flink? Sounds like a great idea! Do you have a ready to use Java Client for that?

Re: [DISCUSS] Deprecate/remove Twitter connector

Thanks @Martijn for driving this! +1 for deprecating and removing it. All the concerns mentioned previously are valid. It is good to know that the upcoming connector template/archetype will help the user for the kickoff. Beyond that, speaking of using a real connector as a sample, since Flink is he

Kafka Consumer Group Name not set if no checkpointing?

Hi using flink 1.14.3, I have a job that doesn't use checkpointing. I have noticed that the Kafka Consumer Group is not set? I use Kafka Explorer to see all the consumers and when I run the job I don't see the consumer group. Finally I decided to enable checkpointing and restart the job and finall

Flink 1.14.2/3 - KafkaSink vs deprecated FlinkKafkaProducer

Hi everyone, Has anyone encountered any problem with the new KafkaSink that is used in Flink 1.14 ? When running our jobs, the *sinks* of some of our jobs are stuck in initializing for more than an hour. The only thing that helps is deleting the topic *__transaction_state*. After deleting this to

Re: [DISCUSS] Deprecate/remove Twitter connector

Shameless plug: Maybe the Wikipedia EventStreams SSE API would make for a great connector example in Flink? :D On Mon, Jan 31, 2022 at 5:41 AM Martijn Visser wrote: > Hi all, > > Than

Re: [DISCUSS] Deprecate/remove Twitter connector

Hi all, Thanks for your feedback. It's not about having this connector in the main repo, that has been voted on already. This is strictly about the connector itself, since it's not maintained and most probably also can't be used due to changes in Twitter's API that aren't reflected in our connecto

Re: [DISCUSS] Deprecate/remove Twitter connector

Hi, I agree with the concern about having this connector in the main repo. But I think in general it doesn't harm to have a sample connector to show how to develop a custom connector, and I think that the Twitter connector can be a good candidate for such a template. It needs rework for sure, as i