Re: unsubscribe

Hi Please send an email to user-unsubscr...@flink.apache.org for unsubscribing. you can get more info here[1] [1] https://flink.apache.org/community.html#mailing-lists Best, Congxian Maatary Okouya 于2020年7月30日周四 上午12:09写道： > >

Re: Flink jobs getting finished because of "Could not allocate the required slot within slot request timeout"

Hi Sateesh, Would you check Flink jobmanager log to see whether it has sent container requests to YARN RM? If the request is sent but not fulfilled, you will need to check the YARN RM logs or the YARN cluster resources at that time to see whether that container request is fulfillable. The resource

Flink Kafka EXACTLY_ONCE causing KafkaException ByteArraySerializer is not an instance of Serializer

I'm using Flink 1.10 and Kafka (AWS MSK) 2.2 and trying to do a simple app that consumes from one kafka topic and produces events into another topic. I would like to utilize the exactly_once semantic, however, I am experiencing the following error: org.apache.kafka.common.KafkaException: org.apach

Re: [Compression] Flink DataStream[GenericRecord] Avrowriters

Hi Ravi, With CustomAvroWriters (SNAPPY) when I run on a cluster, it does create files, but files are not recognized as avro files by avro tools jar, when I try to deserialize it to json. Flink Logs shows: 2020-07-29 23:54:23,270 INFO com.hadoop.compression.lzo.LzoCodec - Successfully loaded & in

Re: [Compression] Flink DataStream[GenericRecord] Avrowriters

Hi Ravi, Thanks for details. CustomAvrowriter was working for now. Although its failing for Snappy codec in local IDE with "java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z" I think I will have to try running it in an EMR/Hadoop environment to get the

Re: [Compression] Flink DataStream[GenericRecord] Avrowriters

There is another alternative which you could try like this val stream:DataStream[GenericRecord] = _ val aof:AvroOutputFormat[GenericRecord] = new AvroOutputFormat(new org.apache.flink.core.fs.Path(""),classOf[GenericRecord]) aof.setSchema(schema) aof.setCodec(AvroOutputFormat.Codec.SNAPPY) str

Re: [Compression] Flink DataStream[GenericRecord] Avrowriters

Hi Vijayendra, Currently AvroWriters doesn't support compression. If you want to use compression then you need to have a custom implementation of AvroWriter where you can add features of compression. Please find a sample customization for AvroWriters where you could use compression. You can use th

Flink jobs getting finished because of "Could not allocate the required slot within slot request timeout"

Hi All, I have an EMR Cluster with one Master Node and 3 worker Nodes ( it has auto scaling enabled and the max no.of worker nodes can go up to 8). I have 3 Spark Jobs that are running currently on the Cluster. I submitted 3 Flink Jobs and all of them finished as the slots are not available er

[Compression] Flink DataStream[GenericRecord] Avrowriters

Hi Team, Could you please provide a sample for Enabling Compression (Snappy) of Avro: DataStream[GenericRecord] AvroWriters.forGenericRecord(schema) Regards, Vijay

Matching largest event pattern without duplicates

Hey all, I'm trying to complete a small POC to see if Flink is suitable for our needs and the first step is to evaluate a stream of events and continually output the largest active group that does not contain duplicates. I'm attempting to do this with the CEP pattern matching. For example, for t

Re: Compression Streamingfilesink ROW-encoded format

Thank You. On Wed, Jul 29, 2020 at 2:07 AM Ravi Bhushan Ratnakar < ravibhushanratna...@gmail.com> wrote: > Yes, flink-compress module is supported from 1.10.0 and onward. > > Regards, > Ravi > > On Tue 28 Jul, 2020, 23:11 Vijayendra Yadav, > wrote: > >> Thank You Ravi for Quick help. One Last

unsubscribe

Re: Making sense of Meter metrics graph on Grafana

Thanks. Got it. With regards On Wed, Jul 29, 2020 at 8:18 PM Chesnay Schepler wrote: > Yes; a rate of 1 means that 1 event occurred per second, which in your > case means one call to markEvent() per second. > > Note that the default Meter implementation calculates the rate per > second over the

Re: Making sense of Meter metrics graph on Grafana

Yes; a rate of 1 means that 1 event occurred per second, which in your case means one call to markEvent() per second. Note that the default Meter implementation calculates the rate per second over the last minute (basically, rate(T) = (count(T) - count(T-60)) / 60; so short spikes tend to be f

Making sense of Meter metrics graph on Grafana

I have added Meter metrics to my flink job code,and in grafana I can see the graph for the same. What I observe is that the graph initially rises, and then plateaus at 1, with occasional variations. As Meter calculates throughput, so does it mean that the map function, wherein I invoke markEvent(

Re: Flink state reconciliation

Hi Kostas Thanks for a possible help! пт, 24 июл. 2020 г., 19:08 Kostas Kloudas : > Hi Alex, > > Maybe Seth (cc'ed) may have an opinion on this. > > Cheers, > Kostas > > On Thu, Jul 23, 2020 at 12:08 PM Александр Сергеенко > wrote: > > > > Hi, > > > > We use so-called "control stream" pattern t

Re: Flink 1.11.0 UDAF fails with "No match found for function signature fun()"

Hi Timo, Thank you for your time and your help! Described approach works. On 29/07/2020, 10:22, "Timo Walther" wrote: Hi Dmytro, I would not recommend to use internal functions from `org.apache.flink.table.planner.functions.aggfunctions`. They are called by a slightly d

Colocating Compute

Hello, I am using Flink v1.10 in a distributed environment to run SQL queries on batch and streaming data. In my setup, data is sharded and distributed across the cluster. Each shard receives streaming updates from some external source. I wish to minimize data movement during query evaluation for

[DISCUSS] FLIP-131: Consolidate the user-facing Dataflow SDKs/APIs (and deprecate the DataSet API)

Hi Everyone, my colleagues (in cc) and I would like to propose this FLIP for discussion. In short, we want to reduce the number of APIs that we have by deprecating the DataSet API. This is a big step for Flink, that's why I'm also cross-posting this to the User Mailing List. FLIP-131: http:/

State Restoration issue with flink 1.10.1

Hi, Presently I have a flink application running on version 1.8.2 I have taken a savepoint on the running app which is stored in s3 , Now I have changed my flink version to 1.10.1 , Now when I running the new application on version flink-1.10.1 from the savepoint taken on flink 1.8.2 it is throwin

Re: PyFlink DDL UDTF join error

Hi Wei, Thank you for the clarification and workaround. Regards, Manas On Wed, Jul 29, 2020 at 12:55 PM Wei Zhong wrote: > Hi Manas, > > It seems a bug of the create view operation. I have created a JIRA for it: > https://issues.apache.org/jira/browse/FLINK-18750 > > Before repairing, please do

Re: Compression Streamingfilesink ROW-encoded format

Yes, flink-compress module is supported from 1.10.0 and onward. Regards, Ravi On Tue 28 Jul, 2020, 23:11 Vijayendra Yadav, wrote: > Thank You Ravi for Quick help. One Last Question is this compression > supported with Flink Version 1.10.0 ? > > Regards, > Vijay > > On Tue, Jul 28, 2020 at 1:2

Re: Count of records coming into the Flink App from KDS and at each step through Execution Graph

I'd recommend to do the aggregation over 5 seconds in graphite/prometheus etc., and expose a counter in Flink for each attribute/event_name. User variables are a good choice for encoding the attribute/event_name values. As for your remaining questions: Flink does not support aggregating ope

Re: Flink 1.11.0 UDAF fails with "No match found for function signature fun()"

Hi Dmytro, I would not recommend to use internal functions from `org.apache.flink.table.planner.functions.aggfunctions`. They are called by a slightly different stack that might cause this exception. Instead you can use the testing functions in `org.apache.flink.table.planner.runtime.utils.Ja

Re: PyFlink DDL UDTF join error

Hi Manas, It seems a bug of the create view operation. I have created a JIRA for it: https://issues.apache.org/jira/browse/FLINK-18750 Before repairing, please do not use create view operation for udtf call. Best, Wei > 在 2020年7月28日，21:19，W

Re: Flink Deployment on Kubernetes session Cluster

Hi Yang, Thank you for your reply. Yes, we have evaluated job specific clusters (as we used to deploy the same in YARN) , the main issue is Job monitoring of multiple jobs as we won't be having a single endpoint like YARN does . We will evaluate K8's operator you have suggested Thanks and Regar