Re: Need help in understanding PojoSerializer

Hi Sachin, Check the last few comments I wrote in this thread: https://lists.apache.org/thread/l71d1cqo9xv8rsw0gfjo19kb1pct2xj1 Regards, Alexis. On Wed, 20 Mar 2024, 18:51 Sachin Mittal, wrote: > Hi, > I saw the post but I did not understand how I would configure these fields > to use those s

Re: Need help in understanding PojoSerializer

Hi, I saw the post but I did not understand how I would configure these fields to use those serializers. (I can change the set type to a list type for now). As per the docs I see that we can annotate fields with @TypeInfo But what I did not get is how using this annotation I can use ListSerializer

Re: Need help in understanding PojoSerializer

Flink doesn’t have built-in support for serializing Sets. See this (stale) issue about the same: https://issues.apache.org/jira/browse/FLINK-16729 You could create a custom serializer for sets, see https://stackoverflow.com/questions/59800851/flink-serialization-of-java-util-list-and-java-util-

Need help in understanding PojoSerializer

Hi, I have a Pojo class like this public class A { public String str; public Set aSet; public Map dMap; } However when I start the flink program I get this message: org.apache.flink.api.java.typeutils.TypeExtractor[] - Field A# dMap will be processed as GenericType. Please read t

Flink 1.15.1 issue we need help on

Hi, we are trying to determine how fix the following exception. This is an issue that is repeatedly happening for us. We have tried looking online for some solutions. One thread suggested setting idleTimeout but this doesn’t seem supported in Flink Source code. https://github.com/netty/netty/

Need help to debug the issue 413 Request Entity Too Large

Hi Flink experts, I have been facing this issue for quite some days as I am not able to submit/run SQL Queries using REST CLIENT OR FLINK SQL CLIENT via SQL gateway. Here's my Setup Cluster Set up via k8s operator to deploy Session Cluster, Service gateway and ingress files. I am able to run ``

Re: Need help how to use Table API to join two Kafka streams

Moving the Dev mailing list to BCC and adding the User ML in this thread On Wed, Feb 8, 2023 at 8:08 AM Amir Hossein Sharifzadeh < amirsharifza...@gmail.com> wrote: > Thanks. If you look at the code, I am defining/creating the table as: > > create_kafka_source_ddl = """ > CREATE TABLE pay

Re: Need help to join Apache Flink community on.Slack

I got it, thanks! Regards, Sucheth Shivakumar website : https://sucheths.com mobile : +1(650)-576-8050 San Mateo, United States On Sun, Jun 5, 2022 at 6:48 PM Jing Ge wrote: > Hi Sucheth, > > Just invited you, please check. Thanks! > > Best Regards, > Jing > > On Sun, Jun 5, 2022 at 6:06 PM Suc

Re: Need help to join Apache Flink community on.Slack

Hi Sucheth, Just invited you, please check. Thanks! Best Regards, Jing On Sun, Jun 5, 2022 at 6:06 PM Sucheth S wrote: > Hello Jing, > > Can you please add me - suchet...@gmail.com > > > On Sun, Jun 5, 2022 at 9:02 AM sri hari kali charan Tummala < > kali.tumm...@gmail.com> wrote: > >> Hi Jing

Re: Need help to join Apache Flink community on.Slack

done, please check. Thanks Best regards, Jing On Sun, Jun 5, 2022 at 6:05 PM Deepak Sharma wrote: > I need the invite as well . > kdq...@gmail.com > > On Sun, 5 Jun 2022 at 9:32 PM, sri hari kali charan Tummala < > kali.tumm...@gmail.com> wrote: > >> Hi Jing, >> >> Please add me kali.tumm...@gm

Re: Need help to join Apache Flink community on.Slack

Hi Sri, I have invited you, please check. Thanks! Best regards, Jing On Sun, Jun 5, 2022 at 6:02 PM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi Jing, > > Please add me kali.tumm...@gmail.com. > > Thanks > Sri > > On Sat, Jun 4, 2022 at 4:47 PM Jing Ge wrote: > >> Hi Sant

Re: Need help to join Apache Flink community on.Slack

Hello Jing, Can you please add me - suchet...@gmail.com On Sun, Jun 5, 2022 at 9:02 AM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi Jing, > > Please add me kali.tumm...@gmail.com. > > Thanks > Sri > > On Sat, Jun 4, 2022 at 4:47 PM Jing Ge wrote: > >> Hi Santhosh, >> >> j

Re: Need help to join Apache Flink community on.Slack

I need the invite as well . kdq...@gmail.com On Sun, 5 Jun 2022 at 9:32 PM, sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi Jing, > > Please add me kali.tumm...@gmail.com. > > Thanks > Sri > > On Sat, Jun 4, 2022 at 4:47 PM Jing Ge wrote: > >> Hi Santhosh, >> >> just invited y

Re: Need help to join Apache Flink community on.Slack

Hi Jing, Please add me kali.tumm...@gmail.com. Thanks Sri On Sat, Jun 4, 2022 at 4:47 PM Jing Ge wrote: > Hi Santhosh, > > just invited you. Please check your email. Looking forward to knowing your > story! Thanks! > > To anyone else who wants to join, please send an email to > user@flink.apac

Re: Need help to join Apache Flink community on.Slack

Hi Santhosh, just invited you. Please check your email. Looking forward to knowing your story! Thanks! To anyone else who wants to join, please send an email to user@flink.apache.org, you might have a better chance to get the invite. Thanks. Regards, Jing On Sat, Jun 4, 2022 at 10:37 PM santhos

Need Help

Team, Is it possible for two independent flink-statefun applications can communicate via Http Rest API ? In other words does flink-statefun support HTTP REST as Ingress ? We are using python sdk for statefun application. Regards, Himanshu

Re: Need help on implementing custom`snapshotState` in KafkaSource & KafkaSourceReader

Hi Santosh, It’s best to avoid cross-posting. Let’s keep the discussion to SO. Best regards, Niklas > On 12. Feb 2022, at 16:39, santosh joshi wrote: > > We are migrating to KafkaSource from FlinkKafkaConsumer. We have disabled > auto commit of offset and instead committing them manually to s

Need help on implementing custom`snapshotState` in KafkaSource & KafkaSourceReader

We are migrating to KafkaSource from FlinkKafkaConsumer

Re: Need help with window TopN query

Sorry for late response, Martijn and Francesco have already give good advises to find out the problem. I only have one minor supplementary information, window rank/join/aggregate would emit results after the end of the window. Now the window size is 24 hour, is there any possible the first window i

Re: Need help with window TopN query

As a rule of thumb, I would first try to check that Flink ingests correctly your csv. Perhaps try to run just a select on your input and see if the input is parsed as expected and is ordered. On Thu, Nov 4, 2021 at 12:47 PM Martijn Visser wrote: > Hi Pavel, > > There's a Flink SQL recipe in the

Re: Need help with window TopN query

Hi Pavel, There's a Flink SQL recipe in the Flink SQL Cookbook for a Window TopN, see https://github.com/ververica/flink-sql-cookbook/blob/main/aggregations-and-analytics/11_window_top_n/11_window_top_n.md. I think that could help you with your use case too. Best regards, Martijn On Thu, 4 Nov

Re: Need help with window TopN query

When the query changed to SELECT user_id, ts, rownum FROM ( SELECT user_id, ts, ROW_NUMBER() OVER (PARTITION BY window_start, window_end, user_id ORDER BY ts ASC) as rownum FROM TABLE( TUMBLE(TABLE visits, DESCRIPTOR(ts), INTERVAL '24' HOURS)) ) WHERE rownum = 1 runs but doesn't produce a

Re: Need help with window TopN query

I think the issue here is that the nested select is selecting all the fields produced by the TVF, including window_time (which is implicitly added by the TVF as described here ). Because of

Need help with window TopN query

I'm trying to express a supposedly simple query with Flink SQL - log the first visit a day for each user. Source table is defined like CREATE TABLE visits (user_id int, ts timestamp(3), WATERMARK FOR ts AS ts) WITH ('connector' = 'filesystem', 'path' = 'file:///visits.csv', 'format' = 'csv') The

Re: Need help of deploying Flink HA on kubernetes cluster

Could you please check that the allocated load balancer could be accessed locally(on the Flink client side)? Best, Yang Fabian Paul 于2021年7月29日周四 下午7:45写道： > Hi Dhiru, > > Sorry for the late reply. Once the cluster is successfully started the web > UI should be reachable if you somehow forward

Re: Need help of deploying Flink HA on kubernetes cluster

Hi Dhiru, Sorry for the late reply. Once the cluster is successfully started the web UI should be reachable if you somehow forward the port of the running pod. Although with the exception you have shared I suspect the cluster never fully runs (or not long enough). Can you share the full stacktra

Re: Need help when using kubernetes.jobmanager.annotations

Could you please ensure that you are using the native Kubernetes mode[1]? For standalone on K8s[2], you need to manually set the annotation in the jobmanager yaml file. [1]. https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/ [2]. https://c

Need help when using kubernetes.jobmanager.annotations

Hi Team, I have set the "kubernetes.jobmanager.annotations". But I can't find these in the k8s deployment. As these can be found in the job manager pod. Is it by design or just be missed?

Re: Need help of deploying Flink HA on kubernetes cluster

Hi Dhiru, No worries I completely understand your point. Usually all the executable scripts from Flink can be found in the main repository [1]. We also provide a community edition of our commercial product [2] which manages the lifecycle of the cluster and you do not have to use these scripts an

Re: Need help of deploying Flink HA on kubernetes cluster

Hi Dhirendra, Thanks for reaching out. A good way to start is to have a look at [1] and [2]. Once you have everything setup it should be possible to delete the pod of the JobManager while an application is running and the job successfully recovers. You can use one of the example Flink applicati

Need help of deploying Flink HA on kubernetes cluster

hi ,    I am very new to flink , I am planning to install Flink HA setup on eks cluster with 5 worker nodes . Please can some one point me to right materials or direction how to install as well as any sample job which I can run only for testing and confirm all things are working as expected . --

Re: error message， need help

Hi, can you maybe share some details about the code you're running? Regards Ingo On Tue, May 18, 2021 at 5:10 AM 杨建春/00250041 wrote: > I'm using flink1.13.0, table Function, why report this error ? what > reason ? Thanks! > > > > Traceback (most recent call last): > File "D:/yjc/AIOPS/Flink

error message， need help

I'm using flink1.13.0, table Function, why report this error ? what reason ? Thanks! Traceback (most recent call last): File "D:/yjc/AIOPS/Flink/UDTFcallstack.py", line 149, in t_result.wait() File "D:\Program Files (x86)\python36\lib\site-packages\pyflink\table\table_result.py", line

Re: [External] : Re: Need help with executing Flink CLI for native Kubernetes deployment

Hi Yang, Thanks for the reply, those information is very helpful. Best, Fuyao From: Yang Wang Date: Tuesday, April 6, 2021 at 01:11 To: Fuyao Li Cc: user Subject: Re: [External] : Re: Need help with executing Flink CLI for native Kubernetes deployment Hi Fuyao, Sorry for the late reply

Re: [External] : Re: Need help with executing Flink CLI for native Kubernetes deployment

help of ZooKeeper. In terms > of this, native k8s is better. Any other ideas? Thanks for your help. > > > > Best, > > Fuyao > > > > *From: *Fuyao Li > *Date: *Thursday, April 1, 2021 at 12:22 > *To: *Yang Wang > *Cc: *user > *Subject: *Re: [External] : R

Re: [External] : Re: Need help with executing Flink CLI for native Kubernetes deployment

for your help. Best, Fuyao From: Fuyao Li Date: Thursday, April 1, 2021 at 12:22 To: Yang Wang Cc: user Subject: Re: [External] : Re: Need help with executing Flink CLI for native Kubernetes deployment Hi Yang, Thanks for sharing the insights. For problem 1: I think I can’t do telnet in the

Re: [External] : Re: Need help with executing Flink CLI for native Kubernetes deployment

://github.com/GoogleCloudPlatform/flink-on-k8s-operator [2] https://github.com/lyft/flinkk8soperator [3] https://youtu.be/pdFPr_VOWTU Best, Fuyao From: Yang Wang Date: Tuesday, March 30, 2021 at 19:15 To: Fuyao Li Cc: user Subject: Re: [External] : Re: Need help with executing Flink CLI for native

Re: [External] : Re: Need help with executing Flink CLI for native Kubernetes deployment

network configuration, but failed to find a reason, could you give me some >hints? >3. In production, what is the suggested approach to list and cancel >jobs? The current manual work of “kubectl exec” into pods is not very >reliable.. How to automate this process

Re: [External] : Re: Need help with executing Flink CLI for native Kubernetes deployment

thanks. Best, Fuyao From: Yang Wang Date: Monday, March 29, 2021 at 20:40 To: Fuyao Li Cc: user Subject: [External] : Re: Need help with executing Flink CLI for native Kubernetes deployment Hi Fuyao, Thanks for trying the native Kubernetes integration. Just like you know, the Flink rest serv

Re: Need help with executing Flink CLI for native Kubernetes deployment

Hi Fuyao, Thanks for trying the native Kubernetes integration. Just like you know, the Flink rest service could be exposed in following three types, configured via "kubernetes.rest-service.exposed.type". * ClusterIP, which means you could only access the Flink rest endpoint inside the K8s cluste

Need help with executing Flink CLI for native Kubernetes deployment

Hi Community, Yang, I am new to Flink on native Kubernetes and I am trying to do a POC for native Kubernetes application mode on Oracle Cloud Infrastructure. I was following the documentation here step by step: [1] I am using Flink 1.12.1, Scala 2.11, java 11. I was able to create a native Kube

Re: [External] : Re: Need help with JDBC Broken Pipeline Issue after some idle time

at 15:14 To: XU Qinghui Cc: user , Timo Walther Subject: Re: [External] : Re: Need help with JDBC Broken Pipeline Issue after some idle time Hi Qinghui, I agree. I am trying to found internal and resources on the internet to fix the issue. Idle Time Limits<https://docs.oracle.com/en/cl

Re: [External] : Re: Need help with JDBC Broken Pipeline Issue after some idle time

Fuyao From: XU Qinghui Date: Tuesday, March 2, 2021 at 13:40 To: Fuyao Li Cc: user , Timo Walther Subject: [External] : Re: Need help with JDBC Broken Pipeline Issue after some idle time It sounds like the jdbc driver's connection is closed somehow, and probably has nothing to do with fl

Re: Need help with JDBC Broken Pipeline Issue after some idle time

at java.base/java.lang.Thread.run(Thread.java:834) > > Caused by: java.io.IOException: Reestablish JDBC connection failed > > at > org.apache.flink.connector.jdbc.internal.JdbcBatchingOutputFormat.flush(JdbcBatchingOutputFormat.java:202) > > a

Re: Need help with JDBC Broken Pipeline Issue after some idle time

tFormat.java:195) ... 30 more Thanks, Best regards, Fuyao From: Fuyao Li Date: Tuesday, March 2, 2021 at 10:33 To: user , Timo Walther Subject: Need help with JDBC Broken Pipeline Issue after some idle time Hi Flink Community, I need some help with JDBC sink in Datastrea

Need help with JDBC Broken Pipeline Issue after some idle time

Hi Flink Community, I need some help with JDBC sink in Datastream API. I can produce some records and sink it to database correctly. However, if I wait for 5 minutes between insertions. I will run into broken pipeline issue. Ater that, the Flink application will restart and recover from checkpo

need help about executeinsert,thanks!

I'm learning executeinsert from the document My code is: https://paste.ubuntu.com/p/d2TDdcy7GB/ I guess the function createTemporaryView can create a table needed by the function executeinsert I got: No table was registered under the name `default_catalog`.`default_database`.`OutOrders`.

need help about executeinsert,thanks!

I'm learning executeinsert from the document My code is: https://paste.ubuntu.com/p/d2TDdcy7GB/ I guess the function createTemporaryView can create a table needed by the function executeinsert I got: No table was registered under the name `default_catalog`.`default_database`.`OutOrders`.

?????? need help about "incremental checkpoint",Thanks

I have solved it. I resumed from the wrong class in the jar. while the jar contains two different experiment. Thanks for your help. Please close my issue. MUCH MUCH THANKS! --  -- ??:

Re: 回复： need help about "incremental checkpoint",Thanks

I'm forwarding my comment from the Jira Issue [1]: In https://github.com/appleyuchi/wrongcheckpoint/blob/master/src/main/scala/wordcount_increstate.scala you set the RocksDBStateBackend, in https://github.com/appleyuchi/wrongcheckpoint/blob/master/src/main/scala/StateWordCount.scala you set t

?????? need help about "incremental checkpoint",Thanks

I don't  know where I did change the state backends. There are two meaning of "restarting": ??Restarting automatically(success in my experiment) ??Restarting manually(failure in my experiment) The whole experiment(just a wordcount case) and steps are listed in my github: https://github.com/appl

Re: need help about "incremental checkpoint",Thanks

. Best, David On Mon, Oct 5, 2020 at 2:38 PM 大森林 wrote: > Could you give more details? > Thanks > > > -- 原始邮件 -- > *发件人:* "大森林" ; > *发送时间:* 2020年10月3日(星期六) 上午9:30 > *收件人:* "David Anderson"; > *抄送:* "user"; &g

?????? need help about "incremental checkpoint",Thanks

Could you give more details? Thanks --  -- ??: "??"

Re: 回复： need help about "incremental checkpoint",Thanks

ink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/AbstractFsCheckpointStorage.java#L60 Best Yun Tang From: 大森林 Sent: Saturday, October 3, 2020 9:30 To: David Anderson Cc: user Subject: 回复： need help about "incremental checkpoint",T

?????? need help about "incremental checkpoint",Thanks

where's the actual path? I can only get one path from the WEB UI Is it possible that this error happened in step 5 is due to my code's  fault? -- -- ??:

Re: need help about "incremental checkpoint",Thanks

kEnd)? > > Much Thanks~! > > > -- 原始邮件 -- > 发件人: "大森林" ; > 发送时间: 2020年10月2日(星期五) 晚上11:41 > 收件人: "David Anderson"; > 抄送: "user"; > 主题: 回复： need help about "incremental checkpoint",Thanks > >

?????? need help about "incremental checkpoint",Thanks

Thanks for your replies~! Could you tell me what the right command is to recover from checkpoint  manually using Rocksdb file? I understand that checkpoint is for automatically recovery, but in this experiment I stop it by force(input 4 error in nc -lk ), Is there a way to recover from inc

Re: need help about "incremental checkpoint",Thanks

sStateBackend.* > *It's NOT a match.* > So I'm wrong in step 5? > Is my above understanding right? > > Thanks for your help. > > -- 原始邮件 -- > *发件人:* "David Anderson" ; > *发送时间:* 2020年10月2日(星期五) 晚上10:35 > *收件人:* "大森林&

?????? need help about "incremental checkpoint",Thanks

Thanks for your replies~! My English is poor ,I have an understanding of your replies: Write in RocksDbStateBackend. Read in FsStateBackend. It's NOT a match. So I'm wrong in step 5? Is my above understanding right? Thanks for your help. --  -- ??

Re: need help about "incremental checkpoint",Thanks

It looks like you were trying to resume from a checkpoint taken with the FsStateBackend into a revised version of the job that uses the RocksDbStateBackend. Switching state backends in this way is not supported: checkpoints and savepoints are written in a state-backend-specific format, and can only

need help about "incremental checkpoint",Thanks

I want to do an experiment of"incremental checkpoint" my code is: https://paste.ubuntu.com/p/DpTyQKq6Vk/   pom.xml is: http://maven.apache.org/POM/4.0.0"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.or

Re: Need Help on Flink suitability to our usecase

Thanks Robert for the reply. On Fri 29 May, 2020, 12:31 Robert Metzger, wrote: > Hey Prasanna, > > (Side note: there is not need to send this email to multiple mailing > lists. The user@ list is the right one) > > Let me quickly go through your questions: > > Is this usecase suited for flink ? >

Re: Need Help on Flink suitability to our usecase

Hey Prasanna, (Side note: there is not need to send this email to multiple mailing lists. The user@ list is the right one) Let me quickly go through your questions: Is this usecase suited for flink ? Based on the details you've provided: Yes What you also need to consider are the hardware requ

Need Help on Flink suitability to our usecase

Hi, I have the following usecase to implement in my organization. Say there is huge relational database(1000 tables for each of our 30k customers) in our monolith setup We want to reduce the load on the DB and prevent the applications from hitting it for latest events. So an extract is done from

RE: Need help on timestamp type conversion for Table API on Pravega Connector

Timo Walther Sent: Tuesday, March 24, 2020 16:40 To: Zhou, Brian; imj...@gmail.com Cc: user@flink.apache.org Subject: Re: Need help on timestamp type conversion for Table API on Pravega Connector [EXTERNAL EMAIL] This issue is tracked under: https://issues.apache.org/jira/browse/FLINK-1

Re: Need help on timestamp type conversion for Table API on Pravega Connector

o/pravega/connectors/flink/FlinkPravegaTableITCase.java#L310 Best Regards, Brian *From:* Jark Wu *Sent:* Thursday, March 19, 2020 20:25 *To:* Till Rohrmann *Cc:* Zhou, Brian; Timo Walther; Jingsong Li; user *Subject:* Re: Need help on timestamp type conversion for Table API on Pravega Connect

RE: Need help on timestamp type conversion for Table API on Pravega Connector

mann Cc: Zhou, Brian; Timo Walther; Jingsong Li; user Subject: Re: Need help on timestamp type conversion for Table API on Pravega Connector [EXTERNAL EMAIL] This maybe a similar issue to [1], we continue the discussion there. Best, Jark [1]: http://apache-flink-user-mailing-list-archive.23360

Re: Need help on timestamp type conversion for Table API on Pravega Connector

This maybe a similar issue to [1], we continue the discussion there. Best, Jark [1]: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/SQL-Timetamp-types-incompatible-after-migration-to-1-10-td33784.html#a33791 On Tue, 17 Mar 2020 at 18:05, Till Rohrmann wrote: > Thanks for

Re: Need help on timestamp type conversion for Table API on Pravega Connector

Thanks for reporting this issue Brian. I'm not a Table API expert but I know that there is some work on the type system ongoing. I've pulled Timo and Jingsong into the conversation who might be able to tell you what exactly changed and whether the timestamp issue might be caused by the changes. Ch

Need help on timestamp type conversion for Table API on Pravega Connector

Hi community, Pravega connector is a connector that provides both Batch and Streaming Table API implementation. We uses descriptor API to build Table source. When we plan to upgrade to Flink 1.10, we found the unit tests are not passing with our existing Batch Table API. There is a type convers

Re: Need help using AggregateFunction instead of FoldFunction

Hi Devin, for event-time based windows, you need to give Flink two types of information: - timestamp of records, which I assume is in your case already embedded into the Pulsar records - and a watermark assigner. The watermarks help Flink to determine when windows can be closed in respect to out-

Re: Need help using AggregateFunction instead of FoldFunction

I did confirm that I got no resulting output after 20 seconds and after sending additional data after waiting over a minute between batches of data. My code looks like this: PulsarSourceBuilder builder = PulsarSourceBuilder .builder(new SimpleStringSchema()) .serviceUrl(SERVICE_URL)

Re: Need help using AggregateFunction instead of FoldFunction

getResult will only be called when the window is triggered. For a fixed-time window, it triggers at the end of the window. However, for EventTimeSessionWindows you need to have gaps in the data. Can you verify that there is actually a 20sec pause inbetween data points for your keys? Additionally,

Re: Need help using AggregateFunction instead of FoldFunction

Hi dev, The time of the window may have different semantics. In the session window, it's only a time gap, the size of the window is driven via activity events. In the tumbling or sliding window, it means the size of the window. For more details, please see the official documentation.[1] Best, Vi

Re: Need help using AggregateFunction instead of FoldFunction

I think there might be a bug in `.window(EventTimeSessionWindows.withGap(Time.seconds(5)))` (unless I'm just not using it correctly) because I'm able to get output when I use the simpler window `.timeWindow(Time.seconds(5))` However, I don't get any output when I used the session-based window.

Re: Need help using AggregateFunction instead of FoldFunction

They released Pulsar 2.4.2, and I was able to pull its dependencies and successfully submit the Flink job. It's able to receive messages from the Pulsar topic successfully. However, I still don't think I'm using the AggregateFunction correctly. I added logging statements everywhere in my code, an

Re: Need help using AggregateFunction instead of FoldFunction

To: user@flink.apache.org Sent: 05/12/2019 04:35:05 Subject: Re: Need help using AggregateFunction instead of FoldFunction It turns out that the exception that I was getting is actually related to Pulsar since I'm using the Pulsar Flink connector. I found the exact issue reported here: https://github.co

Re: Need help using AggregateFunction instead of FoldFunction

It turns out that the exception that I was getting is actually related to Pulsar since I'm using the Pulsar Flink connector. I found the exact issue reported here: https://github.com/apache/pulsar/issues/4721 devinbost wrote > I was able to make more progress (based on the documentation you > pro

Re: Need help using AggregateFunction instead of FoldFunction

Thanks for the help. I was able to make more progress (based on the documentation you provided), but now I'm getting this exception: org.apache.pulsar.client.impl.DefaultBatcherBuilder@3b5fad2d is not serializable. The object probably contains or references non serializable fields. org.apa

Re: Need help using AggregateFunction instead of FoldFunction

Hi devinbost, Sharing two example links with you : - the example code of official documentation[1]; - a StackOverflow answer of a similar question[2]; [1]: https://ci.apache.org/projects/flink/flink-docs-release-1.9/dev/stream/operators/windows.html#aggregatefunction [2]: https://stackove

Need help using AggregateFunction instead of FoldFunction

Hi, In my use case, I am attempting to create a keyedStream (on a string) and then window that stream (which represents keyed JSON objects) with EventTimeSessionWindows (so that I have a separate window for each set of JSON messages, according to the key), and then concatenate the JSON objects by

Re: Need help to understand memory consumption

If yes, does that mean that I have to purge old state > backend in RocksDB ? > > Thanks a lot ! > > Regards, > Julien. > > - Mail original - > De: "Fabian Hueske" > À: "wangzhijiang999" > Cc: "Paul Lam" , jpreis...@fre

Re: Need help to understand memory consumption

bian Hueske" À: "wangzhijiang999" Cc: "Paul Lam" , jpreis...@free.fr, "user" Envoyé: Mercredi 17 Octobre 2018 10:03:35 Objet: Re: Need help to understand memory consumption Hi, As was said before, managed memory (as described in the blog post [1]) is only

Re: Need help to understand memory consumption

--- > 发件人：Paul Lam > 发送时间：2018年10月17日(星期三) 14:35 > 收件人：Zhijiang(wangzhijiang999) > 抄 送：jpreisner ; user > 主 题：Re: Need help to understand memory consumption > > Hi Zhijiang, > > Does the memory management apply to streaming jobs as well? A previous > post

回复：Need help to understand memory consumption

(wangzhijiang999) 抄　送：jpreisner ; user 主　题：Re: Need help to understand memory consumption Hi Zhijiang, Does the memory management apply to streaming jobs as well? A previous post[1] said that it can only be used in batch API, but I might miss some updates on that. Thank you! [1] https

Re: Need help to understand memory consumption

lazy allocating via taskmanager.memory.preallocate=false. > > Best, > Zhijiang > -- > 发件人：Paul Lam > 发送时间：2018年10月17日(星期三) 12:31 > 收件人：jpreisner > 抄　送：user > 主　题：Re: Need help to understand memory consumption

回复：Need help to understand memory consumption

- 发件人：Paul Lam 发送时间：2018年10月17日(星期三) 12:31 收件人：jpreisner 抄　送：user 主　题：Re: Need help to understand memory consumption Hi Julien, AFAIK, streaming jobs put data objects on heap, so the it depends on the JVM GC to release the memory. Best, Paul Lam > 在 2018年10月12日，14:29，jpreis...@free.fr 写

Re: Need help to understand memory consumption

Hi Julien, AFAIK, streaming jobs put data objects on heap, so the it depends on the JVM GC to release the memory. Best, Paul Lam > 在 2018年10月12日，14:29，jpreis...@free.fr 写道： > > Hi, > > My use case is : > - I use Flink 1.4.1 in standalone cluster with 5 VM (1 VM = 1 JobManager + 1 > TaskM

Re: Need help regarding Flink Batch Application

The code or the execution plan (ExecutionEnvironment.getExecutionPlan()) of the job would be interesting. 2018-08-08 10:26 GMT+02:00 Chesnay Schepler : > What have you tried so far to increase performance? (Did you try different > combinations of -yn and -ys?) > > Can you provide us with your app

Re: Need help regarding Flink Batch Application

What have you tried so far to increase performance? (Did you try different combinations of -yn and -ys?) Can you provide us with your application? What source/sink are you using? On 08.08.2018 07:59, Ravi Bhushan Ratnakar wrote: Hi Everybody, Currently I am working on a project where i need t

Need help regarding Flink Batch Application

Hi Everybody, Currently I am working on a project where i need to write a Flink Batch Application which has to process hourly data around 400GB of compressed sequence file. After processing, it has write it as compressed parquet format in S3. I have managed to write the application in Flink and a

Re: Need Help/Code Examples with reading/writing Parquet File with Flink ?

You can use the corresponding HadoopInputformat within Flink > On 18. Apr 2018, at 07:23, sohimankotia wrote: > > Hi .. > > I have file in hdfs in format file.snappy.parquet . Can someone please > point/help with code example of reading parquet files . > > > -Sohi > > > > -- > Sent from:

Re: Need Help/Code Examples with reading/writing Parquet File with Flink ?

AFA I remember, there is no ParquetInputFormat in Flink. But there is a JIRA logged and an attempt in this PR , but was never merged. We do have an internal implementation that is being used in our c

Need Help/Code Examples with reading/writing Parquet File with Flink ?

Hi .. I have file in hdfs in format file.snappy.parquet . Can someone please point/help with code example of reading parquet files . -Sohi -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Need help on Streaming API | Flink | GlobalWindow and Customized Trigger

Hi, I think the problem is that the Trigger only uses processing time to determine when to trigger. If the job shuts down (which happens when the sources shut down and the whole pipeline is flushed) pending processing-time triggers are not fired. You can use the fact that when sources shutdown

Re: Need help debugging back pressure job

Hi Fritz, you're right that back pressure should propagate upstream to the sources. Thus, the cause of the back pressure should be the operator following the last operator with back pressure. In order to debug it you could take a look at the stack trace of the TM. Simply go to the machine on whic

Need help debugging back pressure job

Hi All, Any tips on debugging back pressure ? I have a workload where it get stuck after it ran for a couple of hours. I assume the cause of the back pressure is the block next to the one showing as having the back pressure, is this right ? Any idea on how to get the backtrace ? (I’m using stan

Re: Need help on Streaming API | Flink | GlobalWindow and Customized Trigger

Hi, If your could give us a look at your custom Trigger we might be able to figure out what’s going on. Best, Aljoscha > On 22. May 2017, at 09:06, Samim Ahmed wrote: > > Hello All, > > Hope you are doing well.. > > Myself Samim and I am working of POC(proof of concept) for a project. In th

Need help on Streaming API | Flink | GlobalWindow and Customized Trigger

Hello All, Hope you are doing well.. Myself Samim and I am working of POC(proof of concept) for a project. In this project we are using Apache Flink to process the stream data and find the required pattern and finally dump those patterns in DB. So to implement this we have used the global w

Re: Flink Application on YARN failed on losing Job Manager | No recovery | Need help debug the cause from logs

On 4 November 2016 at 17:09:25, Josh (jof...@gmail.com) wrote: > Thanks, I didn't know about the -z flag! > > I haven't been able to get it to work though (using yarn-cluster, with a > zookeeper root configured to /flink in my flink-conf.yaml) > > I can see my job directory in ZK under > /fl

  1   2   >