Re: How do I pass the aggregated results of an SQL TUMBLING Window as one set of data to a Process Function?

Hi Marco, sorry for the late reply. Have you looked into user-defined aggregate functions for SQL? I think your requirements can be easily implemented there. You can declare multiple aggregate functions per window. There is also the built-in function LISTAGG that might help for your use case.

Re: Is there any way to directly convert org.apache.flink.table.api.Table into my POJO.

Hi, first, we should clarify "continue to be put into the Flink table": A Flink Table object does not physically store the data. It is basically a view that contains a transformation pipeline. When you are calling `collect()` the pipeline is executed and all results from the cluster are stre

Re: How does Flink cache values that also do not exist in the database?

Hi Marco, when you say "database" are you refering to the JDBC connector or would you like to perform a JDBC query within some UDF? In the latter case, I would recommend to use Flink's ProcessFunction because you can store the cache hits in state (and thus keep them forever). SQL/Table API doe

Re: How to debug a Flink Exception that does not have a stack trace?

Thanks a lot for this information Steven. I learned again something :-) Cheers, Till On Sat, Dec 12, 2020 at 9:02 PM Dan Hill wrote: > Thanks! That makes sense. > > On Sat, Dec 12, 2020 at 11:13 AM Steven Wu wrote: > >> This is a performance optimization in JVM when the same exception is >> t

unsubscribe

unsubscribe

Re: flink-cdc-connector 使用场景和限制是什么？

Moving to chinese user mailing ist. On 12/14/2020 3:19 AM, 陈帅 wrote: 传统CDC方式是通过 mysql -> debezium -> kafka, 这样便于DBA管控资源，因为像postgres库需要创建slot资源，但如果像 flink-cdc-connector 每（几）张表就创建一个CDC流的话对数据库的资源要求很高，而且不可控。所以我的理解flink-cdc-connector更适合少量的即席cdc处理，而不是大规模的cdc处理，不知我的理解对不对？

Re: Never terminating test ...

My guess would be that the consumer does not stop running once it exhausted the kinesis stream. Which makes sense since this isn't a batch job. (Wouldn't want the source to shut down just because it happened to catch up with your input ;) ) On 12/14/2020 8:09 AM, Avi Levi wrote: I have the f

Trying to simulate the Split Distinct Aggregation optimizations from Table API

Hi, I am trying to simulate the "Split Distinct Aggregation" [1] with the data from Taxi Ride. I am using the following query: SELECT dayOfTheYear, COUNT(DISTINCT driverId) FROM TaxiRide GROUP BY dayOfTheYear and I am analyzing the different methods for optimizing. So I started using (1) no opti

Re: Flink 1.12

1) It is compiled with Java 8 but runs on Java 8 & 11. 2) Docker images are not yet published. 3) It is mentioned at the top of the Kubernetes HA Services documentation that it also works for the native Kubernetes integration. /Kubernetes high availability services can only be used when d

Re: unsubscribe

To unsubscribe from this mailing list, please send a mail to /user-unsubscr...@flink.apache.org/ . On 12/14/2020 10:25 AM, yuguangyuan wrote: unsubscribe

Re: How to tell when flink is done restoring from a savepoint

I do not believe there is anything in the UI, or the logs for that matter, that give a definite answer to that. I suppose if a new checkpoint was completed then you can be sure the state was restored. FLINK-19013 On 12/14/2020 6:40 AM, Rex Fe

upsert-kafka to do temporal table joins

I just try to follow the example list in the page https://flink.apache.org/news/2020/12/10/release-1.12.0.html#table-apisql-support-for-temporal-table-joins-in-sql Unfortunately when I try to upload the python file to flink cluster, the error occured: py4j.protocol.Py4JJavaError: An error occu

Re: Getting an exception while stopping Flink with savepoints on Kubernetes+Minio

Hi Piotrek, Sorry for late response. I have another problem with setting logs. I think the logging issue comes from using Flink on my host machine and running a job on a jobmanager in K8s. I'm managing the issue. But, this is what I got in /log folder of my host machine: 2020-12-14 15:04:51,329

How to gracefully avoid "Generic types have been disabled in the ExecutionConfig and type java.util.List is treated as a generic type"?

Hi, The following program compiles and runs w/o exceptions: > public class Test { > > public static class A { > private int n; > > public A() { } > public int getN() { return n; } > public void setN(int n) { this.n = n; } > } > > public static class B { > private Lis

Re: [DISCUSS] FLIP-147: Support Checkpoints After Tasks Finished

Hi all, I would like to resume this discussion for supporting checkpoints after tasks Finished :) Based on the previous discussion, we now implement a version of PoC [1] to try the idea. During the PoC we also met with some possible issues: 1. To include EndOfPartition into considerat

Fine-grained task recovery

Hi, I'm running a simple, "embarassingly parallel" ETL-type job. I noticed that a failure in one subtask causes the entire job to restart. Even with the region failover strategy, all subtasks of this task and connected ones would fail. Is there any way to limit restarting to only the single subtas

Re: Flink jobmanager TLS connectivity to Zookeeper

Hey Matthias, I have and it doesn't seem like there are any native properties that support this interaction. I did try enabling the rest/internal SSL properties to see if that would work but when my jobmanager tries to make a connection to zookeeper, zookeeper is rejecting the connection saying it

Re: How to tell when flink is done restoring from a savepoint

Ok, thank you. On Mon, Dec 14, 2020 at 2:07 AM Chesnay Schepler wrote: > I do not believe there is anything in the UI, or the logs for that matter, > that give a definite answer to that. > I suppose if a new checkpoint was completed then you can be sure the state > was restored. > > FLINK-19013

Re: pause and resume flink stream job based on certain condition

Hi Guowei and Arvid, Thanks for the suggestion. I wonder if it makes sense and possible that the operator will produce a side output message telling the source to 'pause', and the same side output as the side input to the source, based on which, the source would pause and resume? Thanks a lot! El

Direct Memory full

Hello, Our job consistently shows Outside JVM Type Count Used Capacity *Direct* 32,839 1.03 GB 1.03 GB for direct memory. Is it typical for it to be full? What are the consequences that we may not be noticing of direct memory being full? Thanks! -- Rex Fenley | Software Engineer - Mobile an