Unsubscribe

Unsubscribe

Unsubscribe

Re: Handling late events with Table API / SQL

urse of the operator receiving the first event that belongs to this window until the time this window fires. Please help understanding these. Thanks From: Feng Jin Sent: 06 March 2024 07:08 To: Sunny S Cc: user@flink.apache.org Subject: Re: Handling

Handling late events with Table API / SQL

Hi, I am using Flink SQL to create a table something like this : CREATE TABLE some-table (   ...,   ...,   ...,   ...,   event_timestamp as TO_TIMESTAMP_LTZ(event_time*1000, 3),   WATERMARK FOR event_timestamp AS event_timestamp - INTERVAL '30' SECOND ) WITH (   'conne

ClassNotFoundException for External Dependencies in Flink Session Cluster on Kubernetes

Hello team, I'm currently deploying a Flink session cluster on Kubernetes using the Flink Kubernetes operator. My Flink job, which utilizes the DataStream API for its logic, requires several external dependencies. , so I've used an init container to copy all necessary jars to a /mnt/external-jars

Support for ConfigMap for Runtime Arguments in Flink Kubernetes Operator

Hi team, I am currently in the process of deploying Flink on Kubernetes using the Flink Kubernetes Operator and have encountered a scenario where I need to pass runtime arguments to my Flink application from a properties file. Given the dynamic nature of Kubernetes environments and the need for fl

RE: Apache Flink lifecycle and Java 17 support

supported by Flink community with Flink 1.18? Can we use this combination in our commercial product release? Regards, Deepti Sharma From: xiangyu feng Sent: 24 January 2024 18:11 To: Deepti Sharma S Cc: user@flink.apache.org Subject: Re: Apache Flink lifecycle and Java 17 support Hi Deepti, For

Apache Flink lifecycle and Java 17 support

Hello Team, Can you please let me know the lifecycle for Flink 1.x versions? Also does any version supports Java17? Regards, Deepti Sharma

Concerns and Anomalies in Flink Window Functions with TumblingProcessingTimeWindows

Hi team, I'm a newcomer to Flink's window functions, specifically utilizing TumblingProcessingTimeWindows with a configured window duration of 20 minutes. However, I've noticed an anomaly where the window output occurs within 16 to 18 minutes. This has left me uncertain about whether I overlooked a

Inquiry Regarding Flink Tumbling Window Persistence and Restart Handling for File Source

Hello team, I'm relatively new to Flink's window functions, and I've configured a tumbling window with a 10-minute duration. I'm wondering about the scenario where the Flink job is restarted or the Flink application goes down. Is there a mechanism to persist the aggregated values, allowing the pro

Seeking Guidance on Optimal Keying Strategy for Flink Use Case

Hello team, I'm currently working on a Flink use case where I need to calculate the sum of occurrences for each "customer_id" within a 10-minute duration and send the results to Kafka, associating each "customer_id" with its corresponding count (e.g., 101:5). In this scenario, my data source is a

Re: Exploring Data Storage Mechanisms in Flink with RocksDB: A Focus on Windowing and Aggregation

0 > > > -- > Best！ > Xuyang > > > At 2023-12-01 15:08:41, "arjun s" wrote: > > Hi team, > I'm new to Flink's window and aggregate functions, and I've configured my > state backend as RocksDB. Currently, I'm computing the c

Exploring Data Storage Mechanisms in Flink with RocksDB: A Focus on Windowing and Aggregation

shMap or Hap, are stored in RocksDB, and what type of data is being stored in RocksDB. Thanks in Advance, Arjun S

Re: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

s. > > Please let me know if there are any other problems. > > Best, > Yu Chen > > > 2023年11月7日 18:11，arjun s 写道： > > > > Hi Chen, > > I attempted to configure the 'source.path.regex-pattern' property in the > table settings as '^cu

Re: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

ill recursive all files > under the directory-- of `path` option > > > Best, > Yu Chen > > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/filesystem/ > > --

Re: Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

-- of `path` option > > > Best, > Yu Chen > > > [1] > https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/connectors/table/filesystem/ > > -- > *发件人:* arjun s > *发送时间:* 2023年11月6日 20:50 > *收件人:* user@flin

Re: Clear the State Backends in Flink

> But it cannot be done for Keyed state for users because every operation > for it is binded with a specific key currently. > BTW, Could you also share your business scenario ? It could help us to > rethink the interface. Thanks! > > On Tue, Oct 31, 2023 at 12:02 AM arjun s wrot

Handling Schema Variability and Applying Regex Patterns in Flink Job Configuration

Hi team, I'm currently utilizing the Table API function within my Flink job, with the objective of reading records from CSV files located in a source directory. To obtain the file names, I'm creating a table and specifying the schema using the Table API in Flink. Consequently, when the schema match

Clear the State Backends in Flink

Hi team, I'm interested in understanding if there is a method available for clearing the State Backends in Flink. If so, could you please provide guidance on how to accomplish this particular use case? Thanks and regards, Arjun S

Monitoring File Processing Progress in Flink Jobs

Hi team, I'm also interested in finding out if there is Java code available to determine the extent to which a Flink job has processed files within a directory. Additionally, I'm curious about where the details of the processed files are stored within Flink. Thanks and regards, Arjun S

Re: Enhancing File Processing and Kafka Integration with Flink Jobs

Hi team, I'm also interested in finding out if there is Java code available to determine the extent to which a Flink job has processed files within a directory. Additionally, I'm curious about where the details of the processed files are stored within Flink. Thanks and regards, Arjun

Re: Enhancing File Processing and Kafka Integration with Flink Jobs

gt;> to do this outside of the Flink job periodically (cron, whatever), because >>>> on restart it won't reprocess the files that have been committed in the >>>> checkpoints. >>>> >>>> >>>> https://nightlies.apache.org/flink/flink

Re: Enhancing File Processing and Kafka Integration with Flink Jobs

t; and STREAMING that reads or writes (par... > > <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/connectors/datastream/filesystem/> > > > > On Thursday, 26 October, 2023 at 06:53:23 pm IST, arjun s < > arjunjoice...@gmail.com> wrote: > > > H

Enhancing File Processing and Kafka Integration with Flink Jobs

Hello team, I'm currently in the process of configuring a Flink job. This job entails reading files from a specified directory and then transmitting the data to a Kafka sink. I've already successfully designed a Flink job that reads the file contents in a streaming manner and effectively sends them

Re: Urgent: Mitigating Slow Consumer Impact and Seeking Open-SourceSolutions in Apache Kafka Consumers

Hi All We are also seeking for a custom partitioning strategy, it will be helpful for us too. Thanks and regards, Gowtham S On Mon, 18 Sept 2023 at 12:13, Karthick wrote: > Thanks Liu Ron for the suggestion. > > Can you please give any pointers/Reference for the custom par

Re: Urgent: Mitigating Slow Consumer Impact and Seeking Open-Source Solutions in Apache Kafka Consumers

Hi Karthik This appears to be a common challenge related to a slow-consuming situation. Those with relevant experience in addressing such matters should be capable of providing assistance. Thanks and regards, Gowtham S On Fri, 15 Sept 2023 at 23:06, Giannis Polyzos wrote: > Hi Karth

Re: Question about serialization of java.util classes

Hello Alexis, Thank you for sharing the helper classes this but unfortunately I have no idea how to use these classes or how they might be able to help me. This is all very new to me and I honestly can't wrap my head around Flink's type information system. Best regards, Saleh. > On 14 Aug 202

Re: Question about serialization of java.util classes

Hi, Here's a minimal example using an ArrayList, a HashSet, and a TreeSet: ``` package com.example; import java.util.ArrayList; import java.util.HashSet; import java.util.TreeSet; import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; public class App { public static cla

Re: Question about serialization of java.util classes

Hi, According to this blog post https://flink.apache.org/2020/04/15/flink-serialization-tuning-vol.-1-choosing-your-serializer-if-you-can/#pojoserializer The "Must be proc

Question about serialization of java.util classes

Greetings, I am working on a project that needs to process around 100k events per second and I'm trying to improve performance. Most of the classes being used are POJOs but have a couple of fields using a `java.util` class, either `ArrayList`, `HashSet` or `SortedSet` etc. This forces Flink to

Integarting Apache Flink with Apache Ranger

Hi, I'm interested to know if there is an available feature for integrating Apache Flink with Apache Ranger. If so, could you kindly share the relevant documentation with me? Thanks and regards, Arjun S

Re: Apache Atlas - Flink Integration

Hi Shammon, Thank you for your prompt reply.Aslo I'm interested to know if there is an available feature for integrating Apache Flink with Apache Ranger. If so, could you kindly share the relevant documentation with me? Thanks & Regards, Arjun

Apache Atlas - Flink Integration

this link the feature is still open. https://issues.apache.org/jira/browse/FLINK-6757 I would like to know whether this feature on Apache Atlas with Apache Flink is released or not. If yes, anyone could share with me the references for integrating. Thanks and Regards, Arjun S

Running Into Error When Installing Apache Flink Using Python

(240.2 MB) ━━━╸ 240.2/240.2 MB 19.0 MB/s eta 0:00:01 ERROR: Exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/pip/_vendor/urllib3/response.py", line 438, in _error_catcher yield File "/usr/lib/python3/dist-pack

apache-flink java question

Hi, Does Apache flink table api work with java 17? Thanks, Chris Joseph

Re: Using TumblingEventTimeWindows on low traffic kafka topic

gt; > Best regards, > Yuxia > > ------ > *发件人: *"deepthi s" > *收件人: *"User" > *发送时间: *星期四, 2022年 12 月 22日 上午 9:46:00 > *主题: *Using TumblingEventTimeWindows on low traffic kafka topic > > (Adding subject) > > On Wed, Dec 21, 2022 at 5:41 PM d

Using TumblingEventTimeWindows on low traffic kafka topic

(Adding subject) On Wed, Dec 21, 2022 at 5:41 PM deepthi s wrote: > Hello, I am new to even-time processing and need some help. > > > > We have a kafka source with very low qps and multiple topic partitions > have no data for long periods of time. Additionally, data from the

[no subject]

Hello, I am new to even-time processing and need some help. We have a kafka source with very low qps and multiple topic partitions have no data for long periods of time. Additionally, data from the source can come out of order (within bounds) and the application needs to process the events in o

Flink falls back on to kryo serializer for GenericTypes

Hello, How to avoid flink's kryo serializer for GenericTypes ? Kryo is having some performance issues. Tried below but no luck. env.getConfig().disableForceKryo(); env.getConfig().enableForceAvro(); Tried this - env.getConfig().disableGenericTypes(); getting - Generic types have been disabled

Deserialize avro message without reader schema.

Hi, I'm trying to deserialize avro messages from the kafka topic as a consumer. As a kafka consumer, I do not want to hold the reader schema on my end. Is there a way to deserialize avro messages to GenericRecord without providing the reader schema ? Regards, Sucheth Shivakumar website : https:

RE: Flink support on Avro1.11

Hello Martijn, Thank you for the details. Just one more question when we have plan for Flink 1.16 release with Avro 1.11.0? Regards, Deepti Sharma PMP® & ITIL From: Martijn Visser Sent: Wednesday, August 3, 2022 6:50 PM To: Deepti Sharma S Cc: user Subject: Re: Flink support on Avro

Flink support on Avro1.11

Hello Team, Please confirm, if any Flink version supports Avro version 1.11? If not, what are the plans to support Avro v1.11? Regards, Deepti Sharma PMP(r) & ITIL

Re: Flink config driven tool ?

You are just spamming the inbox by sending these emails. You can just ignore sending those emails if it isn't adding any value. Regards, Sucheth Shivakumar website : https://sucheths.com mobile : +1(650)-576-8050 San Mateo, United States On Wed, Jun 15, 2022 at 1:41 PM sri hari kali charan Tumm

Re: Apache flink doesn't work with avro kafka topic with multiple event types

ster/docs/connectors/table/formats/raw/ > > Best regards, > > Qingsheng > > On Sun, Jun 12, 2022 at 12:53 PM Sucheth S wrote: > > > > Hi, > > > > Apache Flink doesn't work with Avro serialized kafka topic with multiple > event types ( > > To

Apache flink doesn't work with avro kafka topic with multiple event types

Hi, Apache Flink doesn't work with Avro serialized kafka topic with multiple event types ( TopicRecordNameStrategy for subject ) Is there a way to read a generic record from avro serialized kafka topic which can have messages with different schemas, basically TopicRecordNameStrategy for the subje

Re: filesink part files roll over

Thank you! On Mon, Jun 6, 2022 at 2:57 AM Qingsheng Ren wrote: > Hi Sucheth, > > Please see https://issues.apache.org/jira/browse/FLINK-27910 > > Best, > > Qingsheng > > > On Jun 5, 2022, at 23:21, Sucheth S wrote: > > > > Hi, > > > >

Re: Need help to join Apache Flink community on.Slack

, Jun 5, 2022 at 6:06 PM Sucheth S wrote: > >> Hello Jing, >> >> Can you please add me - suchet...@gmail.com >> >> >> On Sun, Jun 5, 2022 at 9:02 AM sri hari kali charan Tummala < >> kali.tumm...@gmail.com> wrote: >> >>> Hi Jing, >&g

Re: Need help to join Apache Flink community on.Slack

Hello Jing, Can you please add me - suchet...@gmail.com On Sun, Jun 5, 2022 at 9:02 AM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi Jing, > > Please add me kali.tumm...@gmail.com. > > Thanks > Sri > > On Sat, Jun 4, 2022 at 4:47 PM Jing Ge wrote: > >> Hi Santhosh, >> >> j

filesink part files roll over

Hi, Can someone please help me with this please - https://stackoverflow.com/q/72496963/9125940 ? Regards, Sucheth Shivakumar website : https://sucheths.com mobile : +1(650)-576-8050 San Mateo, United States

RE: Flink versions release 1.14.3

Hello Martijn, Can you please confirm if Flink version 1.14.3 is released? I can see the package on Github, under Tags. However not published on Flink website? Regards, Deepti Sharma PMP® & ITIL From: Martijn Visser Sent: Friday, January 7, 2022 1:21 PM To: Deepti Sharma S Cc:

RE: Flink upgrade and rollback query

Hello Team, Could you please help us in below question. Regards, Deepti Sharma PMPR & ITIL From: Deepti Sharma S Sent: Friday, January 7, 2022 11:58 AM To: user@flink.apache.org Subject: Flink upgrade and rollback query Importance: High Hello Team, We are using F

Flink versions release 1.14.3

Hello Team, Can you please confirm, when we have Flink version release 1.14.3 which has Log4J version 2.17? Regards, Deepti Sharma PMP(r) & ITIL

Flink upgrade and rollback query

down time. Regards, Deepti Sharma PMPR & ITIL smime.p7s Description: S/MIME cryptographic signature

Re: Unable to update logback configuration in Flink Native Kubernetes

Hi Sharon Thanks a lot. I just updated the files (flink-conf.yaml and logback-console.xml) in the local conf folder and it worked as expected. Thanks & Regards Raghavendar T S MERAS Plugins <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-em

Unable to update logback configuration in Flink Native Kubernetes

work? Thanks & Regards Raghavendar T S <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-emai

RE: FlinkJobNotFoundException

Hi Matthias, Do you have any update here? Thank you, Doug From: Gusick, Doug S [Engineering] Sent: Thursday, October 7, 2021 9:03 AM To: Hailu, Andreas [Engineering] ; Matthias Pohl Cc: user@flink.apache.org; Erai, Rahul [Engineering] Subject: RE: FlinkJobNotFoundException Hi Matthias, I

RE: FlinkJobNotFoundException

Hi Matthias, I just wanted to follow up here. Were you able to access the jobmanager log? If so, were you able to find anything around the issues we have been facing? Best, Doug From: Hailu, Andreas [Engineering] Sent: Thursday, September 30, 2021 8:56 AM To: Matthias Pohl ; Gusick, Doug S

RE: FlinkJobNotFoundException

logs there. Please let me know if you need any more information. Thank you, Doug From: Matthias Pohl Sent: Wednesday, September 29, 2021 12:00 PM To: Gusick, Doug S [Engineering] Cc: user@flink.apache.org; Erai, Rahul [Engineering] Subject: Re: FlinkJobNotFoundException Hi Doug, thanks for

FlinkJobNotFoundException

Hello, We are facing an issue with some of our applications that are submitting a high volume of jobs to Flink (we are using v1.9.2). We are observing that numerous jobs (in this case 44 out of 350+) fail with the same FlinkJobNotFoundException within a 45 second timeframe. >From our client lo

Re: Flink Checkpoint for Stateless Operators

o say > that we need to use a sink instead of a flat map? > I'm not sure I understand the use case. What do you do with the results of > Flat Map? > > [1 https://arxiv.org/pdf/1506.08603.pdf > [2] > https://flink.apache.org/features/2018/03/01/end-to-end-exactly-once-apa

Re: Flink Checkpoint for Stateless Operators

//ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#kafka-consumers-offset-committing-behaviour-configuration > > Regards, > Roman > > > On Thu, Apr 29, 2021 at 12:53 PM Raghavendar T S > wrote: > >> Hi Team >> >> Assume that we ha

Flink Checkpoint for Stateless Operators

ssful checkpoint, do we need to manually re-process (retry using queue or any other business logic) the event? Job: Kafka Source -> Flat Map Thank you -- Raghavendar T S www.teknosrc.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm

Re: Key by Kafka partition / Kinesis shard

S merasplugins.com teknosrc.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail> Virus-free. www.avast.com <https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm

Re: Approaches for external state for Flink

- Is Cassandra the right choice here? Note, the state size is very large > and I have to feed the state from batch flow firstly. Thus I can not use > the internal state like rocksdb. > > 3- Can i improve this logic? > > 4- May be any bottleneck in that flow? I think to use asyncMap functions > for state read/write operations. > -- Raghavendar T S www.teknosrc.com

Re: Producing to Kafka topics dynamically without redeployment

n demand as needed. > > > On Tuesday, April 20, 2021, 01:11:08 PM PDT, Ejaskhan S < > iamejask...@gmail.com> wrote: > > > Hi Ahmed, > > If you want to dynamically produce events to different topics and you have > the logic to identify the target

Re: Producing to Kafka topics dynamically without redeployment

Hi Ahmed, If you want to dynamically produce events to different topics and you have the logic to identify the target topics, you will be able to achieve this in the following way. - Suppose this is your event after the transformation logic(if any) : EVENT. - This is the target topic f

Re: Editing job graph at runtime

Kafka consumer C1* >>> secondInputStream = addSource(...) *//Kafka consumer C2* >>> >>> outputStream = firstInputStream,keyBy(a -> a.key) >>> .connect(secondInputStream.keyBy(b->b.key)) >>> .coProcessFunction() >>> * // logic determines : whether a

Re: Editing job graph at runtime

t;> outputStream = firstInputStream,keyBy(a -> a.key) >>> .connect(secondInputStream.keyBy(b->b.key)) >>> .coProcessFunction() >>> * // logic determines : whether a new sink should be added to the >>> application or not ?. If not: then the event wi

Re: Question on Flink and Rest API

Yes Gordon, it's obviously gave me a starting point to think about. On Wed, Feb 3, 2021, 12:02 PM Tzu-Li (Gordon) Tai wrote: > Hi, > > There is no out-of-box Flink source/sink connector for this, but it isn't > unheard of that users have implemented something to support what you > outlined. > >

Re: Question on Flink and Rest API

Hi Raghavendar, Yes , you are right. Your approach is correct ,and it is the most straightforward one.but I was just thinking about the possibilities of my question mentioned. Thanks EK On Wed, Feb 3, 2021, 12:02 PM Raghavendar T S wrote: > Hi Ejaskhan > > As per my understand

Re: Question on Flink and Rest API

)/RabbitMQ(queue) without persisting it in the data store. Let Flink do all the processing and finally write to the data store. Thank you Raghavendar T S https://www.linkedin.com/in/raghavendar-ts On Wed, Feb 3, 2021 at 11:29 AM Ejaskhan S wrote: > Team, > > It's just a random thought.

Question on Flink and Rest API

Team, It's just a random thought. Can I make the Flink application exposing a rest endpoint for the data source? So a client could send data to this endpoint. Subsequently, Flink processed this data and responded to the client application through the endpoint, like a client-server model. Thanks

Re: Flink CheckPoint/Savepoint Behavior Question

tion phase? Basically will the code >before the RichFunction' *open()* be run? If not, would the >operators.open() functions run, when Flink restore from >checkpoint/savepoint? > > Thanks, > Jason > -- Raghavendar T S www.teknosrc.com <https://www.avast.com/sig-email?

Running Flink job as a rest

Can Flink job be running as Rest Server, Where Apache Flink job is listening on a port (443). When a user calls this URL with payload, data directly goes to the Apache Flink windowing function. Right now Flink can ingest data from Kafka or Kinesis, but we have a use case where we would like to pus

Re: Is Apache Flink suitable for an application where messages are routed to different services

plicable laws. If you have received this > email by mistake, please (i) notify the sender immediately; (ii) delete it > from your database; and (iii) do not disclose the contents to anyone or > make copies thereof. Razorpay accepts no liability caused due to any > inadvertent/ unintentional data transmitted through this email. > > --- > -- Raghavendar T S www.teknosrc.com

Getting UnsupportedException in Kyro for proto maps

Hi, I have a proto which has a map field in it. When i try to use it in Flink using Kyro serializer, im getting following error when pushing the output to Kafka Caused by: java.lang.UnsupportedOperationException at java.util.Collections$UnmodifiableCollection.add(Collections.java:1057) at com.eso

Rocksdb state directory path in EMR

Hi, I have enabled rocksdb(State store) with s3 (external checkpoint) on EMR . I am using rocksdb as state store with ValueState and checkpoints are stored in s3. I am able to see checkpoints in s3 and functionality with respect to state store is working fine. But i am trying to dissect rocksdb

adding s3 object metadata while using StreamFileSink

We are creating files in S3 and we want to update the S3 object metadata with some security-related information for governance purposes. Right now Apache Flink totally abstracts how and when S3 object gets created in the system. Is there a way that we can pass the S3 object metadata and update it

Getting Window information from coGroup functin

Hi, I have a usecase where i want to join two streams. I am using coGroup for this KeyBuilder leftKey = new KeyBuilder(jobConfiguration.getConnectStream().getLeftKey()); KeyBuilder rightKey = new KeyBuilder(jobConfiguration.getConnectStream().getRightKey()); leftSource.coGroup(rightSource).where(

Re: Question on stream joins

rectly, and > in join function, the framework will iterate the elements for you and you > can only specify the logic for each (left, right) pair. > > Best, > Yun > > --Original Mail -- > *Sender:*Sudan S > *Send Date:*Fri May 29 01:40:59

Question on stream joins

Hi , I have two usecases 1. I have two streams which `leftSource` and `rightSource` which i want to join without partitioning over a window and find the difference of count of elements of leftSource and rightSource and emit the result of difference. Which is the appropriate join function ican use

Re: changing the output files names in Streamfilesink from part-00 to something else

ribe your needs? Why do you need to modify file names > flexibly? What kind of name do you want? > > Best, > Jingsong Lee > > On Thu, May 14, 2020 at 2:05 AM dhurandar S > wrote: > >> Yes we looked at it , >> The problem is the file name gets generated in a dy

Re: changing the output files names in Streamfilesink from part-00 to something else

ojects/flink/flink-docs-stable/dev/connectors/streamfile_sink.html#part-file-configuration > > > > ------ > 发件人：dhurandar S > 日 期：2020年05月13日 05:13:04 > 收件人：user; > 主 题：changing the output files names in Streamfilesink from part-00

changing the output files names in Streamfilesink from part-00 to something else

We want to change the name of the file being generated as the output of our StreamFileSink. , when files are generated they are named part-00*, is there a way that we can change the name. In Hadoop, we can change RecordWriters and MultipleOutputs. May I please some help in this regard. This is cau

Re: Flink error;

It is working as expected. If I'm right, the print operator will simply call the `.toString()` on the input element. If you want to visualize your payload in JSON format, override the toString() in `SensorData` class with the code to form your payload as a JSON representation using ObjectMapper or

Re: doing demultiplexing using Apache flink

many > > -- > > Ververica GmbH > Registered at Amtsgericht Charlottenburg: HRB 158244 B > Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji > (Tony) Cheng > > > > On Thu, Apr 30, 2020 at 5:21 AM dhurandar S > wrote: > >> Thank yo

Re: doing demultiplexing using Apache flink

Thank you Alexander for the response. This is very helpful. Can i apply the same pattern to S3 as well , as in read from Kafka or Kinesis and write multiple files in S3 or multiple topics in Kinesis ? regards, Rahul On Wed, Apr 29, 2020 at 2:32 PM Alexander Fedulov wrote: > Hi Dhurandar, > > it

doing demultiplexing using Apache flink

Hi , We have a use case where we have to demultiplex the incoming stream to multiple output streams. We read from 1 Kafka topic and as an output we generate multiple Kafka topics. The logic of generating each new Kafka topic is different and not known beforehand. Users of the system keep adding n

doing demultiplexing using Apache flink

> > Hi , > > We have a use case where we have to demultiplex the incoming stream to > multiple output streams. > > We read from 1 Kafka topic and as an output we generate multiple Kafka > topics. The logic of generating each new Kafka topic is different and not > known beforehand. Users of the syst

doing demultiplexing using Apache flink

Hi , We have a use case where we have to demultiplex the incoming stream to multiple output streams. We read from 1 Kafka topic and as an output we generate multiple Kafka topics. The logic of generating each new Kafka topic is different and not known beforehand. Users of the system keep adding n

doing demultiplexing using Apache flink

Hi , We have a use case where we have to demultiplex the incoming stream to multiple output streams. We read from 1 Kafka topic and as an output we generate multiple Kafka topics. The logic of generating each new Kafka topic is different and not known beforehand. Users of the system keep adding n

Problem getting watermark right with event time

Hi, I am having a problem getting watermark right. The setup is - I have a Flink Job which reads from a Kafka topic, uses Protobuf Deserialization, uses Sliding Window of (120seconds, 30 seconds), sums up the value and finally returns the result. The code is pasted below. The problem here is, I'

Commercial support for Apache Flink

Hello Team, Can you please let us know who all are the companies provides the commercial support for Apache Flink. One we know i.e. Data Artisans. Please let me know if any other company provides the same? [Ericsson] DEEPTI SHARMA Specialist ITIL 2011 Foundation Certi

Re: Does FlinkKafkaConsumer010 care about consumer group?

itly is > usually always recommended before moving to production. See [1]. > > If your job code hasn’t changed across the restores, then it should be > fine even if you didn’t set the UID. > > > [1] https://ci.apache.org/projects/flink/flink-docs- > release-1.3/ops/produ

Re: Does FlinkKafkaConsumer010 care about consumer group?

Does this mean I can use the same consumer group G1 for the newer version A'? And inspite of same consumer group, A' will receive messages from all partitions when its started from savepoint? I am using Flink 1.2.1. Does the above plan require setting uid on the Kafka source in the job? Thanks, M

Re: Does job restart resume from last known internal checkpoint?

//lists.apache.org/thread.html/ > a1a0d04e7707f4b0ac8b8b2f368110b898b2ba11463d32f9bba73968@ > %3Cuser.flink.apache.org%3E > > > Nico > > On Thursday, 1 June 2017 20:30:59 CEST Moiz S Jinia wrote: > > Bump.. > > > > On Tue, May 30, 2017 at 10:17 PM, Moiz S Jinia > wro

Re: Does job restart resume from last known internal checkpoint?

Bump.. On Tue, May 30, 2017 at 10:17 PM, Moiz S Jinia wrote: > In a checkpointed Flink job will doing a graceful restart make it resume > from last known internal checkpoint? Or are all checkpoints discarded when > the job is stopped? > > If discarded, what will be the resume point? > > Moiz >

Does job restart resume from last known internal checkpoint?

In a checkpointed Flink job will doing a graceful restart make it resume from last known internal checkpoint? Or are all checkpoints discarded when the job is stopped? If discarded, what will be the resume point? Moiz

Re: Kafka partitions -> task slots? (keyed stream)

All other operators could still benefit from a higher > parallelism. > > > Am 30.05.2017 um 09:49 schrieb Moiz S Jinia : > > > > For a keyed stream (where the key is also the message key in the source > kafka topic), is the parallelism of the job restricted to the number of

Kafka partitions -> task slots? (keyed stream)

For a keyed stream (where the key is also the message key in the source kafka topic), is the parallelism of the job restricted to the number of partitions in the topic? Source topic has 5 partitions, but available task slots are 12. (3 task managers each with 4 slots) Moiz

  1   2   >