Re: Task manger isn’t initiating with defined values in Flink 1.11 version as part of EMR 6.1.0

Hi Deep, Flink has dropped support for specifying the number of TMs via -n since the introduction of Flip-6. Since then, Flink will automatically start TMs depending on the required resources. Hence, there is no need to specify the -n parameter anymore. Instead, you should specify the parallelism

Re: Tumbling Time Window

For straightforward tumbling windows, the regular DSL windowing performs noticeably better than a custom process function because it takes advantage of an internal API to avoid some serialization overhead. There's a simple example of a ProcessWindowFunction in [1], and an example of using a KeyedP

How to register TableSinks

Hi and Happy New Year, I’m currently trying to remove deprecations to prepare for the upgrade to Flink 1.12. currently running on 1.11. Specifically I need to update our code that registers table sinks into the StreamTableEnvironment. I’m maintaining jobs that use DataStreams with multiple sin

Re: Flink sink never executes

Hi Ben, Sorry for the late reply but I guess that your question was answered in StackOverflow, right? Did that answer solve your problem? Cheers, Kostas On Mon, Dec 21, 2020 at 9:09 AM Ben Beasley wrote: > > First off I want to thank the folks in this email list for their help thus > far. > >

Re: How to register TableSinks

Hi Patrick. Happy New Year to you too ;) The method you referring was deprecated along with the TableSink whatsoever in favour of a much improved and feature rich new Source & Sink API. You can find an extensive documentation on this new API here[1]. Therefore if you use the old TableSink interf

Re: Is chk-$id/_metadata created regardless of enabling externalized checkpoints?

Hi Dongwon, Happy new year! One meta file would be stored on top of HDFS even if external-checkpoint is not enabled. If external checkpoint is not enabled, flink would delete all the checkpoints on exit, and if external checkpoint is enabled, the checkpoints would be kept on cancel or fail c

Re: How to register TableSinks

Hey, Thanks Dawid, One more question: Does the StatementSet API supposed to work with the old sink interface? I get the following error when I’m using it with the deprecated registerTableSink method: The main method caused an error: requirement failed: operations should not be empty Thanks!

Re: Visualizing Flink Statefun's "Logical Co-location, Physical Separation" Properties

Hi Le, Let me try to answer to your multiple questions, one by one: > I'm trying to understand the internal mechanism used by Flink Statefun to > dispatch functions to Flink cluster. In particular, I was trying to find a > good example demonstrating Statefun's "Logical Co-location, Physical > Sep

Re: Flink Stateful Function: The program's entry point class not found in the jar file

Hi Le, Looking at your pom.xml, you are pointing to the wrong main method here: https://github.com/flint-stone/flink-statefun/blob/lx-base/statefun-examples/statefun-flink-datastream-example/pom.xml#L161 You need to change it to your Example class, this should work. On Tue, Dec 29, 2020 at 5:06

Fwd: How to gracefully avoid "Generic types have been disabled in the ExecutionConfig and type java.util.List is treated as a generic type"?

Any advice would be appreciated :) Thanks, Dongwon -- Forwarded message - From: Dongwon Kim Date: Mon, Dec 14, 2020 at 11:27 PM Subject: How to gracefully avoid "Generic types have been disabled in the ExecutionConfig and type java.util.List is treated as a generic type"? To: us

Batch with Flink Steraming API version 1.12.0

I have a Kafka source that I would like to run a batch job on. Since Version 1.12.0 is now soft deprecating the DataSet API in favor of the DataStream API, can someone show me an example of this? (Using DataStream) thanks -- Robert Cullen 240-475-4490

Re: Flink sink never executes

Yes, it did. Thanks for checking, Kostas. Also, thanks again for helping me with the other issue. What a great community Flink has. From: Kostas Kloudas Date: Monday, January 4, 2021 at 6:21 AM To: Ben Beasley Cc: user@flink.apache.org Subject: Re: Flink sink never executes Hi Ben, Sorry for

Kafka SQL Connector Behavior (v1.11)

Based on these docs, https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/connectors/kafka.html, the default partitioning behavior is not quite clear to me. If no value for sink-partitioner is given, is the default behavior just that of the native Kafka library? (with key use murm

Comparing Flink vs Materialize

Has anyone compared Flink with Materialize? A friend recommended me switch to Materialize. In one of their blog posts, it says that Flink splits operators across CPUs (instead of splitting partitions across CPUs). Is this true? Is it configurable? https://materialize.com/blog-rocksdb/

Replace (xx,'#','') has error

What can i do? 发自我的iPhone

Re: pause and resume flink stream job based on certain condition

Hi Robert, sorry for the late reply, I just did a quick test up, this seems working: 1. during the time checkpoints could expire, but once the thread is not blocked, it will continue checkpointing 2. this guarantees the message ordering Thanks a lot! Eleanore On Tue, Dec 15, 2020 at 10:42 PM Rob

Re: Replace (xx,'#','') has error

Hi, without seeing the error and an example, it's hard to help. Are you sure that xx is a string? You may need to convert it beforehand with CAST(xx AS VARCHAR) On Tue, Jan 5, 2021 at 3:12 AM wrote: > What can i do? > > 发自我的iPhone > -- Arvid Heise | Senior Java Developer

Re: numRecordsOutPerSecond metric and side outputs

Hi Alexey, side outputs should be counted in numRecordsOutPerSecond. However, there is a bug that this is not happening for side-outputs in the middle of the chain [1]. [1] https://issues.apache.org/jira/browse/FLINK-18808 On Tue, Dec 22, 2020 at 1:14 AM Alexey Trenikhun wrote: > Hello, > Does