Re: How to write unit test for stateful operators in Pyflink apps

y appreciate it. Best, Long On Mon, Nov 8, 2021 at 8:25 AM Dian Fu wrote: > Hi Long, > > I agree with Fabian that currently you have to test it with a e2e job. > There are still no such test harnesses for PyFlink jobs. > > Regards, > Dian > > On Fri, Nov 5, 2021 at

Re: "sink.partition-commit.success-file.name" option in the FileSystem connector does not work

le which is written done to the > dir when the partition is done. It's content have nothing to do with your > bussiness. > > > > Long Nguyễn 于2021年11月5日周五 下午5:07写道： > >> > >> Thank you, Paul. > >> > >> The answer is so clear and helpful

Re: How to write unit test for stateful operators in Pyflink apps

Thanks, Fabian. I'll check it out. Hope that Dian can also give me some advice. Best, Long On Fri, Nov 5, 2021 at 3:48 PM Fabian Paul wrote: > Hi, > > Since you want to use Table API you probably can write a more high-level > test around executing the complete program. A good examples are the

Re: "sink.partition-commit.success-file.name" option in the FileSystem connector does not work

Thank you, Paul. The answer is so clear and helpful. But I'm still wondering what is the purpose of the sink.partition-commit.success-file.name option if the sin

"sink.partition-commit.success-file.name" option in the FileSystem connector does not work

Hi. I am trying to use the FileSystem connector to simply read data from a text file and then write that data to an output CSV file. I notice that Flink allows specifying the name of the output file by

How to write unit test for stateful operators in Pyflink apps

Hi. I'm using Pyflink and Table APIs to create a window with Python. I have just read the Unit Testing Stateful or Timely UDFs & Custom Operators secti

Why do the count windows in Flink Table APIs require processing time for sorting whereas in Flink Datastream APIs they do not

I have read about the Window operator in Flink documentation and know that it groups rows into finite groups based on time or row-count intervals. I saw an example of a sliding count window right

Re: Calculate summary of bits in a sliding count window

y the > value, thus only the records with the same value will be in the same group. > I guess this is no intended. > > Long Nguyễn 于2021年11月2日周二 上午3:05写道： > >> I have set up a program that takes bits 0 and 1 from a Kafka topic and >> then uses Flink to create a sliding co

Calculate summary of bits in a sliding count window

I have set up a program that takes bits 0 and 1 from a Kafka topic and then uses Flink to create a sliding count window of size 5. In that window, I'd like to output 1 if there are 3 or more of the bit 1, otherwise, output 0. Currently, I follow the way of calculating the sum of bits in the window.