Re: Stateful Functions: Routing to remote functions

Hi Igal, thanks for your reply. Initially I thought the same thing, but it turns out I am able to call the remote function from an embedded "wrapper" function using the exact same setup (Relevant Code below). So that's one kind of solution to that Problem. But to me it seems like it's a bit of

Re: Integrating prometheus

Yes, you do need to extend RichFunction; there's no way around that. As for the missing metric, the usual cause is that the job/task finishes so quickly that the metric is never reported. If this is not the cause I would recommend enabling DEBUG logging and searching for warnings from the metr

Re: Integrating prometheus

Also, it seems custom metrics can only be recorded if we extend RichFunction, as it allows us to override open wherein we can get hold of context and metrics constructs. Please let me know if there are other ways too. On Fri, Jul 3, 2020 at 10:05 PM Manish G wrote: > Hi, > > I am basically look

Re: Task recovery?

Here is one log https://www.dropbox.com/s/s8uom5uto708izf/flink-job-001.log?dl=0 If I understand correctly on June 23rd it suspended the jobs? So at that point they would no longer show in the UI or be restarted? On Fri, 3 Jul 2020 at 12:05, John Smith wrote: > I didn't restart the job man

Re: Integrating prometheus

Hi, I am basically looking for : throughput, success rate, error rate. For experimental purposes I could complete all configurations as explained in the official documentation. But somehow my custom metrics(a simple Counter) is still not shown on the prometheus board, though default metrics I can

Re: Flink AskTimeoutException killing the jobs

Hi Xintong/LakeShen: We have the following setting in flink-conf.yaml akka.ask.timeout: 180 s akka.tcp.timeout: 180 s But still see this exception.  Are there multiple akka.ask.timeout or additional settings required ? Thanks Mans On Friday, July 3, 2020, 01:08:05 AM EDT, Xintong Song

Re: Task recovery?

I didn't restart the job manager. Let me see if I can dig up the logs... Also I just realised it's possible that the retry attempts to recover may have been exhausted.

Re: Stateful Functions: Routing to remote functions

Hi Jan, Judging by the exception message it seems like the function type "demo/eventCounterPython" is not known to stateful functions. This could happen if the module.yaml (provided in your email) was accidentally excluded from the resulting artifact (Docker image or a jar-with-dependencies) Can yo

Re: Integrating prometheus

What metrics specifically are you interested in? On 03/07/2020 17:22, Robert Metzger wrote: Hi Manish, Currently, Flink's metric system does not support metrics via annotations. You need to go with the documented approach. But of course, you can try to build your own metrics abstraction based

Re: Integrating prometheus

Hi Manish, Currently, Flink's metric system does not support metrics via annotations. You need to go with the documented approach. But of course, you can try to build your own metrics abstraction based on Flink's metric system. On Fri, Jul 3, 2020 at 9:35 AM Manish G wrote: > Hi, > > I am follo

Re: Custom service configs in flink

(oops, I accidentally responded to you personally only. The emails are supposed to go onto the list. I added the thread back to the list) But is the config so big that memory usage is a concern here? Also note, that the stuff that runs in main() is just generating a streaming execution plan, whic

Re: Reading and updating rule-sets from a file

Thanks Till, I understand making my FileInputFormat "unsplittable" guarantees a file is always read by a single task. But how can I produce a single record for the entire file? As my file is a CSV with some idiosyncrasies, I am extending CsvInputFormat not to reinvent the wheel of the CSV parsing

Re: Dynamic source and sink.

Hi paul, Thanks for the response. Can you point out an example of how to create a dynamic client or wrapper operator. Thanks and Regards, Dinesh. On Thu, Jul 2, 2020 at 12:28 PM Paul Lam wrote: > Hi Doinesh, > > I think the problem you meet is quite common. > > But with the current Flink

Re: flink interval join后按窗口聚组问题

Hi Bob, This is Flink user mailing list. Please send to this mailing list using english. If you want to use Chinese, you can send it to user...@flink.apache.org 元始(Bob Hu) <657390...@qq.com> 于2020年7月3日周五 下午3:29写道： > 您好，我想请教一个问题： > flink双流表 interval join后再做window group是不是有问题呢，有些left join关联不上的数据会被

Re:

Hi, For the others checking the user@ mailing list: It seems this problem will get resolved through [1]. [1] https://issues.apache.org/jira/browse/FLINK-18478 On Mon, Jun 29, 2020 at 1:23 PM Georg Heiler wrote: > Hi, > > I try to use the confluent schema registry in an interactive Flink Scala >

Re: datadog failed to send report

Hi, could this be another symptom of this issue: https://issues.apache.org/jira/browse/FLINK-16611? I guess you'll have to ask DataDog to check at their end, maybe you are running into some rate limit there? On Fri, Jun 26, 2020 at 5:42 PM seeksst wrote: > > > 原始邮件 > *发件人:* seeksst > *收件人:* Fa

Re: Dynamic partitioner for Flink based on incoming load

> This will mean 2 shuffles, and 1 node might bottleneck if 1 topic has too much data? Yes > Is there a way to avoid shuffle at all (or do only 1) and avoid a situation when 1 node will become a hotspot? Do you know the amount of data per kafka topic beforehand, or does this have to be dynamic?

Re: Issue with job status

Hi Bhaskar, The definition of when a job is marked as RUNNING in Flink is debatable. For a streaming job, RUNNING is when all tasks are running, however for a batch job, if some tasks are running, it is RUNNING already. Since the scheduler does not distinguish between these types of jobs, currentl

Re: Checkpoint failed because of TimeWindow cannot be cast to VoidNamespace

Thanks for your help 1. I started the job from scratch, not a savepoint or externalized checkpoint 2. No job graph change 3. env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime) 4. My Flink version is 1.9.1 Khachatryan Roman 于2020年7月3日周五 下午4:49写道： > I still wasn't able to reproduce the

Custom service configs in flink

I have multiple flink jobs and have custom business configs which are shared between the job. Is it possible if one flink job loads configs in memory and all the flink jobs share the same configs? Basically, I am thinking to fetch configs in one flink job in memory via rest call which is one tim

Re: Avro from avrohugger still invalid

Hi, I don't think there's a workaround, except copying the code and manually fixing it. Did you check out my comment on the Jira issue and the new one I created? Best, Aljoscha On 03.07.20 07:19, Georg Heiler wrote: But would it be possible to somehow use AvroSerializer for now? Best, Geor

Re: Checkpoint failed because of TimeWindow cannot be cast to VoidNamespace

I still wasn't able to reproduce the issue. Can you also clarify: - Are you starting the job from a savepoint or externalized checkpoint? - If yes, was the job graph changed? - What StreamTimeCharacteristic is set, if any? - What exact version of Flink do you use? Regards, Roman On Fri, Jul 3,

Re: Dockerised Flink 1.8 with Hadoop S3 FS support

Hi Lorenzo, Since Flink 1.8 does not support plugin mechanism to load filesystem, you need to copy flink-s3-fs-hadoop-*.jar from opt to lib directory. The dockerfile could be like following. FROM flink:1.8-scala_2.11 RUN cp /opt/flink/opt/flink-s3-fs-hadoop-*.jar /opt/flink/lib Then build you d

Re: Task recovery?

Hi John, did you also restart the JobManager, or just the TaskManagers? In either case, they should recover. Do you still have the JobManager logs around, so that we can analyze them? On Thu, Jun 25, 2020 at 6:40 PM John Smith wrote: > Hi running 1.10.0 > > 3 Zookeepers > 3 Job Nodes > 3 Task

Integrating prometheus

Hi, I am following this link on how to integrate prometheus with flink. Going by the code sample, I would need to insert related metrics code in the main logic. Is it avoidable, like by using some annotations on met

flink interval join????????????????

flink?? interval join??window groupleft join ??select * from a,b where a.id=b.id and b.rowtime between a.rowtime and a.rowtime + INTERVAL '1' HOUR leftRelativeSize=1??rightRelativeSize=0??cle

Re: Flink Kafka connector in Python

Hi Xingbo, Thanks for the reply, I didn't know that a table schema also needs to be declared after the connect or but I understand now. I have another question: how do I write the parsing schemas for a field that itself is a valid JSON string? For example: { "monitorId": 865, "deviceId": "9

Re: Parquet data stream group converter error

Hi, > MessageType schema = reader.getFooter().getFileMetaData().getSchema(); The first thing I'd suggest is to verify that the file contains a valid schema and can be read by some other program, e.g. parquet-tools schema or cat [1]. Regards, Roman On Thu, Jul 2, 2020 at 11:36 PM Jesse Lord wro

Re: Checkpoint is disable, will history data in rocksdb be leak when job restart?

Hi If you do not enable checkpoint and have you ever restored checkpoint for the new job. As what I have said, the timer would also be restored and the event time would also be triggered so that following onEventTime() could also be triggered to clean history data. For the 2nd question, why yo

Re: Timeout when using RockDB to handle large state in a stream app

yes. I agree. because RocsDB will spill data to disk if there is not enough space in memory. Thanks -- -- Felipe Gutierrez -- skype: felipe.o.gutierrez -- https://felipeogutierrez.blogspot.com On Fri, Jul 3, 2020 at 8:27 AM Yun Tang wrote: > > Hi Felipe, > > I noticed my previous mail has a typo: