Integrating LLMs into apache flink pipelines

timent analysis in apache flink pipeline runner Check it out and share your thoughts. Post - https://medium.com/@ganxesh/integrating-llms-into-apache-flink-pipelines-8fb433743761 Langchian-Beam - https://github.com/Ganeshsivakumar/langchain-beam Thanks, Ganesh.

Re: StreamGraph vs JobGraph vs ExecutionGraph

Hi Prabhu, For detail understanding check it out this link: https://www.alibabacloud.com/blog/in-depth-analysis-of-flink-job-execution-flink-advanced-tutorials_596633 Thanks, Sriram G On Tue, May 24, 2022, 07:53 Shengkai Fang wrote: > I think the question also confused me for a long time. > >

Issue in seeing job manager and task manager logs in history sever

Hi Everyone, I am Sriram Ganesh part of the Microsoft Flink platform team. I recently went through the history server setup in Flink. I found an issue in the history server that was not showing the job manager and task manager logs. We are running clusters in session mode and using native

Re: Issue in seeing job manager and task manager logs in history sever

d up with gargantuan logs being archived for every job. > Similarly, and form of log rotation would reduce the value of such a > mechanism. > > I would recommend to set up some logging storage instead, e.g., logstash. > > On 25/05/2022 07:29, Sriram Ganesh wrote: > > Hi

Performance benchmarking for flink.

Hi Everyone, I am looking for industry-standard performance benchmarking for flink for both stateless and stateful data processing for streaming. I came across the yahoo performance benchmarking tool. And flink-benchmarks repo (https://github.com/apache/flink-benchmarks). Can anyone suggest to m

Re: Performance benchmarking for flink.

at 2:04 PM Sriram Ganesh wrote: > > > Hi Everyone, > > > > I am looking for industry-standard performance benchmarking for flink for > > both stateless and stateful data processing for streaming. > > > > I came across the yahoo performance benchmarking

Need help to join Apache Flink community on.Slack

Hi Everyone, I want to join the Apache Flink community on Slack, and when I click the link, it displays "*This link is no longer effective"*, Could you invite me to go in? Thanks, -- *Sriram G* *Tech*

Re: Need help to join Apache Flink community on.Slack

> > > > Do you want to use srigns...@gmail.com to join slack? Someone who has > > already joined can invite you[1]. > > > > [1] https://lists.apache.org/list?dev@flink.apache.org:lte=1M:slack > > > > On Sat, Jun 4, 2022 at 5:41 PM Sriram Ganesh > wrote: >

Need an help in understanding Kafka connector

Hi Everyone, I am trying to understand how Flink works in realtime with Kafka. Since Kafka works on polling, what will be the minimal time for Flink to poll Kafka?. Any explanation or documentation will be helpful. Thanks, Sriram G

Re: Need an help in understanding Kafka connector

! > > [1] > https://github.com/apache/flink/blob/d1cb177c91b41a5387814ad60d1799c08caf3ad9/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/KafkaPartitionSplitReader.java#L101 > > Best, > Qingsheng > On Oct 8, 2022, 02:44 +0800, Sriram Ganesh , wrote: > > Hi Eve

Modify savepoints in Flink

Hi All, I am working on a scenario where I need to modify the existing savepoint operator state. Ex: Wanted to remove some offset of the savepoint. What is the better practice for these scenarios?. Could you please help me with any example as such? Thanks in advance. -- *Sriram G* *Tech*

Re: Modify savepoints in Flink

Sorry. Ignore. Moved this question to the user. On Fri, Oct 21, 2022 at 1:22 PM Sriram Ganesh wrote: > Hi All, > > I am working on a scenario where I need to modify the existing savepoint > operator state. Ex: Wanted to remove some offset of the savepoint. > > What is the b

Building flink kubernetes operator on Mac.

Hi Everyone, I am trying to build a flink-kubernetes-operator on Mac. I am getting errors. Java version: 1.16 Command: mvn clean install -DskipTests Error: Oct 22, 2022 4:51:34 PM com.diffplug.spotless.FormatExceptionPolicyLegacy error SEVERE: Step 'removeUnusedImports' found problem in 'src/tes

Re: Building flink kubernetes operator on Mac.

ps://github.com/diffplug/spotless/issues/834> of spotless, it looks > like the exception is caused by the incompatibility of spotless and high > JDK version. > > Best, > Biao Geng > > From: Sriram Ganesh > Date: Saturday, October 22, 2022 at 7:30 PM > To: dev@flink.apache

Re: [ANNOUNCE] New Apache Flink Committer - Matyas Orhidi

Congrats Matayas! On Mon, Nov 21, 2022, 20:15 Jing Ge wrote: > Congrats Matyas! > > Best Regards, > Jing > > On Mon, Nov 21, 2022 at 3:43 PM Ferenc Csaky > wrote: > > > Congrats Matyas! > > > > Best, > > F > > > > > > > > --- Original Message --- > > On Monday, November 21st, 2022 at 15

[jira] [Created] (FLINK-31924) [Flink operator] Flink Autoscale - Limit the max number of scale ups

Sriram Ganesh created FLINK-31924: - Summary: [Flink operator] Flink Autoscale - Limit the max number of scale ups Key: FLINK-31924 URL: https://issues.apache.org/jira/browse/FLINK-31924 Project

[jira] [Created] (FLINK-30543) Adding more examples for setting up jobs via operator.

Sriram Ganesh created FLINK-30543: - Summary: Adding more examples for setting up jobs via operator. Key: FLINK-30543 URL: https://issues.apache.org/jira/browse/FLINK-30543 Project: Flink

[jira] [Created] (FLINK-31135) ConfigMap DataSize went > 1 MB and cluster stopped working

Sriram Ganesh created FLINK-31135: - Summary: ConfigMap DataSize went > 1 MB and cluster stopped working Key: FLINK-31135 URL: https://issues.apache.org/jira/browse/FLINK-31135 Project: Fl

[jira] [Created] (FLINK-9068) Website documentation issue - html tag visible on screen

SHANKAR GANESH created FLINK-9068: - Summary: Website documentation issue - html tag visible on screen Key: FLINK-9068 URL: https://issues.apache.org/jira/browse/FLINK-9068 Project: Flink