Question on the SQL "GROUPING SETS" and "CUBE" syntax usability

Hi, >From the Flink 1.10 official document ( https://ci.apache.org/projects/flink/flink-docs-release-1.10/dev/table/sql/queries.html), we could see that GROUPING SETS is only supported in Batch mode. [image: image.png] However, we also found that in https://issues.apache.org/jira/browse/FLINK-1

Re: Backpressure and 99th percentile latency

Indeed, it is a bit tricky to understand the relation between floatingBuffersUsage, exclusiveBuffersUsage. I am reading again that table on (https://flink.apache.org/2019/07/23/flink-network-stack-2.html) but I guess I can rely on the latency metric that I implemented on my operator (not the defaul

How to change the flink web-ui jobServer?

Hi community, now I am moving the flink job to k8s,and I plan to use the ingress to show the flink web ui , the problem is that fink job server aren't correct, so I want to change the flink web-ui jobserver ,I don't find the any method to change it ,are there some method to do that?

Re: History server UI not working

Hey Robert, I just tried Flink 1.10 and the history server UI works for me too. Only Flink 1.9.2 is not loading. Since we were already looking into upgrading to 1.10, I might just do that now. Thanks, Peter -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Kafka sink only support append mode?

I wrote a simple program reading from kafka using sql and sink to kafka. But only 'update-mode' = 'append' is supported for sink table and the query sql must have no group statement. Only append mode is supported for kafka sink? Thanks, Lei

Re: Kafka sink only support append mode?

Hi Lei, Yes. Currently, Kafka sink only supports append mode. Other update mode (e.g. upsert mode / retract mode) is on the agenda. For now, you can customize a KafkaTableSink with implementing UpsertStreamTableSink interface, where you will get a Tuple2 records, and the Boolean represents insert

How does custom object/data structures get mapped into rocksdb underneath?

Hi All, I want to do stateful streaming and I was wondering how Custom objects get mapped into rocksdb? say I have the following class that represents my state public class MyState { private HashMap map1 ; // T can be any type private HashMap map2; // S can be any type } I wonder how th

Re: Use flink to calculate sum of the inventory under certain conditions

Hey Jiawei, I'm sorry that you haven't received an answer yet. So you basically have a stream of dynamodb table updates (let's call id CDC stream), and you would like to maintain the inventory of the last 15 days for each vendor. Whenever there's an update in the inventory data (a new event arriv

Re: How does custom object/data structures get mapped into rocksdb underneath?

Hi, when working with state you have to create a state descriptor ([1]). This will have a TypeInformation or TypeSerializer that Flink uses to (de)serialize your state. This will be used to serialize your state to bytes, in the case of RocksDB we serialize both the key (the key of the record

Re: Writing a DataSet to ElasticSearch

Hey Niels, For the OOM problem: Did you try RocksDB? I don't think there's an ES OutputFormat. I guess there's no way around implementing your own OutputFormat for ES, if you want to use the DataSet API. It should not be too hard to implement. On Sun, Mar 1, 2020 at 1:42 PM Niels Basjes wrote

Re: Very large _metadata file

Hi As Gordon said, the metadata will contain the ByteStreamStateHandle, when writing out the ByteStreamStateHandle, will write out the handle name -- which is a path(as you saw). The ByteStreamStateHandle will be created when state size is small than `state.backend.fs.memory-threshold`(default is

Re: Batch Flink Job S3 write performance vs Spark

Hey, I don't think there will be a big performance difference. Both systems have many users writing data to S3, so this will be optimized for both. On Tue, Feb 25, 2020 at 6:03 PM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi All, > > have a question did anyone compared the

Re: Understanding n LIST calls as part of checkpointing

Hi Yun, Thanks for getting back. We’re on a fork of Flink 1.9 (basically 1.9 with some backported fixes from 1.10 and a couple of minor patches) - https://github.com/criteo-forks/flink/tree/criteo-1.9 I’ll check the jira + fix and see if there’s something that was potentially missed. -- Piyush

Re: Alink and Flink ML

Hi, Flavio. Indeed, Becket is the best person to answer this question, but as far as I understand the idea is that Alink will be contributed back to Flink in the form of a refactored Flink ML library (sitting on top of the Table API) [1]. You can follow the progress of these efforts by tracking FL

Re: Flink Session Windows State TTL

Hey Karl, sorry for the late reply! Let me first quickly answer your questions: > >- are expired session windows automatically removed from state? if >not, what's the best way to do it? > > Yes, they should get removed automatically. > >- >- how can we query state size? > > You

Re: Alink and Flink ML

Thanks Marta for the clarification! On Mon, Mar 9, 2020 at 3:26 PM Marta Paes Moreira wrote: > Hi, Flavio. > > Indeed, Becket is the best person to answer this question, but as far as I > understand the idea is that Alink will be contributed back to Flink in the > form of a refactored Flink ML l

Re: Flink Session Windows State TTL

Sorry, I pressed the send button too fast. You also attached an exception to the email, which reads as follows: Caused by: org.apache.flink.runtime.resourcemanager.exceptions.UnfulfillableSlotRequestException: Could not fulfill slot request c28970b7cd4f68383e242703bdac81ca. Requested resource pr

Re: Flink Serialization as stable (kafka) output format?

Hi Theo, However, in most benchmarks, avro turns out to be rather slow in terms of > CPU cycles ( e.g. [1] ) Avro is slower compared to what? You should not only benchmark the CPU cycles for serializing the data. If you are sending JSON strings ac

RE: Flink Conf "yarn.flink-dist-jar" Question

Hi Yang, Yes, a combination of these two would be very helpful for us. We have a single shaded binary which we use to run all of the jobs on our YARN cluster. If we could designate a single location in HDFS for that as well, we could also greatly benefit from FLINK-13938. It sounds like a gene

RE: Flink Conf "yarn.flink-dist-jar" Question

Also may I ask what causes these application ID directories to be left behind? Is it a job failure, or can they persist even if the application succeeds? I’d like to know so that I can implement my own cleanup in the interim to prevent exceeding user disk space quotas. // ah From: Hailu, Andre

Re: Process parquet files in batch mode with blink planner

Hi olivier, Sorry for the late reply. In blink planner, - only hive parquet table can be read now. - If you want to support native parquet files, you can modify `ParquetTableSource` a little bit, extends StreamTableSource. Best, Jingsong Lee On Wed, Feb 26, 2020 at 7:50 PM wrote: > Hi communit

Do I need to set assignTimestampsAndWatermarks if I set my time characteristic to IngestionTime?

Hi All, Do I need to set assignTimestampsAndWatermarks if I set my time characteristic to IngestionTime? say I set my time characteristic of stream execution environment to Ingestion time as follows streamExecutionEnvironment.setStreamTimeCharacteristic(TimeCharacteristic.IngestionTime); do I n