Never terminating test ...

2020-12-13 Thread Avi Levi
I have the following test. the problem is it doesn't end ... meaning it doesn't reach the assertion point. What am I doing wrong? "kinesis consumer" should "consume message from kinesis stream" in { import ExecutionContext.Implicits.global val sampleData = Seq("a", "b", "c") val env: S

How to tell when flink is done restoring from a savepoint

2020-12-13 Thread Rex Fenley
Hi, Every time I restore from a savepoint it looks like it can take 20+ min to restore given the network i/o graphs I'm seeing. However, I can't find a way to see in the Flink UI if the savepoint is currently restoring or if it's finished to be sure. Is there a way to tell if flink is in the middl

Flink 1.12

2020-12-13 Thread Boris Lublinsky
It is great that Flink 1.12 is out. Several questions: 1. Is official Flink 1.12 distribution https://flink.apache.org/downloads.html specifies Scala versions, but not Java versions. Is it Java 8? 2. I do not see any 1.12 docker images here https://hub.d

flink-cdc-connector 使用场景和限制是什么?

2020-12-13 Thread 陈帅
传统CDC方式是通过 mysql -> debezium -> kafka, 这样便于DBA管控资源,因为像postgres库需要创建slot资源,但如果像 flink-cdc-connector 每(几)张表就创建一个CDC流的话对数据库的资源要求很高,而且不可控。所以我的理解flink-cdc-connector更适合少量的即席cdc处理,而不是大规模的cdc处理,不知我的理解对不对?

Re: Flink UDF registration from jar at runtime

2020-12-13 Thread Jakub N
Hi Dawid, thanks a lot for your help. Unfortunately our issue still persists but with your example we managed to reconstruct our problem in the following code: File folder = new File("custom"); URLClassLoader functionClassloader = ClassLoaderUtils.compileAndLoadJava( folder, "StringFunc.

How to Implement a simple boolean .trigger()

2020-12-13 Thread Simone Cavallarin
Hi All, I'm trying to understand how to create a sample trigger. Let's say that I have a stream like this one: Event: "YELLOW, BLUE, WHITE, RED, GREEN, RED, GREEN, RED, YELLOW, YELLOW" Event: "YELLOW, BLUE, BLACK, RED,BLUE, RED, PINK, RED, YELLOW, YELLOW" My stream is then mapped, and I produce