Re: Capping RocksDb memory usage

Hi Cam, AFAIK, that's not an easy thing. Actually it's more like a Rocksdb issue. There is a document explaining the memory usage of Rocksdb [1]. It might be helpful. You could define your own option to tune Rocksdb through "state.backend.rocksdb.options-factory" [2]. However I would suggest not

Re: Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations Hequn! Well deserved! Best Regards, Yu On Thu, 8 Aug 2019 at 03:53, Haibo Sun wrote: > Congratulations! > > Best, > Haibo > > At 2019-08-08 02:08:21, "Yun Tang" wrote: > >Congratulations Hequn. > > > >Best > >Yun Tang > > > >From: Rong Rong > >

Operator state

Hi All, Is there a way to share operator state among operators? For example, I have an operator which has union state and the same state data is required in a downstream operator. If not, is there a recommended way to share the state? Thanks

Re: Capping RocksDb memory usage

Yes, that is correct. Cam Mach Software Engineer E-mail: cammac...@gmail.com Tel: 206 972 2768 On Wed, Aug 7, 2019 at 8:33 PM Biao Liu wrote: > Hi Cam, > > Do you mean you want to limit the memory usage of RocksDB state backend? > > Thanks, > Biao /'bɪ.aʊ/ > > > > On Thu, Aug 8, 2019 at 2:23 A

Re: Does RocksDBStateBackend need a separate RocksDB service？

Hi wanglei, > Is there an embeded RocksDB service in the flink task? Yes, and "RocksDB is an embeddable persistent key-value store for fast storage". [1] 1. http://rocksdb.org/ Thanks, Biao /'bɪ.aʊ/ On Wed, Aug 7, 2019 at 7:27 PM miki haiat wrote: > There is no need to add an external R

Re: Capping RocksDb memory usage

Hi Cam, Do you mean you want to limit the memory usage of RocksDB state backend? Thanks, Biao /'bɪ.aʊ/ On Thu, Aug 8, 2019 at 2:23 AM miki haiat wrote: > I think using metrics exporter is the easiest way > > [1] > https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html

Re:Re: FlatMap returning Row<> based on ArrayList elements()

Hi AU, > The problem with this approach is that I'm looking for a standard FlatMap > anonymous function that could return every time: 1. different number of > elements within the Array and 2. the data type can be random likewise. I mean > is not fixed the whole time then my TypeInformation ret

Re: Can Flink help us solve the following use case

Hi Yoandy, Could you explain more of your requirements? Why do you want to split data into "time slices"? Do you want to do some aggregations or just give each record a tag or tags? Thanks, Biao /'bɪ.aʊ/ On Thu, Aug 8, 2019 at 4:52 AM Sameer Wadkar wrote: > You could do this using custom tri

Re: [EXTERNAL] Re: Delayed processing and Rate limiting

Hi Shakir, I'm not sure I have fully understand your requirements. I'll try to answer your questions. >From my understanding, there is no built-in feature of Flink to support "rate limit" directly. I guess you need to implement one yourself. Both of MapFunction or AsyncFunction could satisfy your

Re: Passing jvm options to flink

Hi Vishwas, For question #1, You can use *env.java.opts* to customize java opts for all flink JVMs, including JM and TM. Or you can use *env.java.opts.jobmanager* or *env.java.opts.taskmanager* accordingly for JM or TM. For question #2, You can use* -C* to specify user classpaths when invoking *

Re:Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations! Best, Haibo At 2019-08-08 02:08:21, "Yun Tang" wrote: >Congratulations Hequn. > >Best >Yun Tang > >From: Rong Rong >Sent: Thursday, August 8, 2019 0:41 >Cc: dev ; user >Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer > >Congratulations H

Re: Flink Table API schema doesn't support nested object in ObjectArray

thanks Fabian , I created a Jira ticket with a code sample . https://issues.apache.org/jira/projects/FLINK/issues/FLINK-13603?filter=allopenissues I think if the root cause I found is correct, fix this issue could be pretty simple . Thanks Jacky Du Fabian Hueske 于2019年8月2日周五 下午12:07写道： > Than

Re: Can Flink help us solve the following use case

You could do this using custom triggers and evictors in Flink. That way you can control when the windows fire and what elements are fired with it. And lastly the custom evictors know when to remove elements from the window. Yes Flink can support it. Sent from my iPhone > On Aug 7, 2019, at 4

Can Flink help us solve the following use case

Hello everybody, We have the following situation: 1) A data stream which collects all system events (near 1/2 a mil per day). 2) A database storing some aggregation of the data. We want to split the data into different "time slices" and be able to "tag it" accordingly. Example: the events in

FlinkDynamoDBStreamsConsumer Atleast Once

Hi Flink Experts, how to achieve at least once semantics with FlinkDynamoDBStreamsConsumer + DynamoDB Streams ? Flink checkpointing or save points do the job? My Scenario:- Flink application uses FlinkDynamoDBStreamsConsumer which reads latest changes from DynamoDB streams but if my software fail

Re: [EXTERNAL] Re: Delayed processing and Rate limiting

Hi Victor, Thanks for the reply and it helps. For the delayed processing, this is exactly what I was looking for. But for the Rate Limit, the one you suggested can only control the number of parallel requests. What I am looking is like limit the number of request per second or minute etc. Like

Re: Capping RocksDb memory usage

I think using metrics exporter is the easiest way [1] https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/metrics.html#rocksdb On Wed, Aug 7, 2019, 20:28 Cam Mach wrote: > Hello everyone, > > What is the most easy and efficiently way to cap RocksDb's memory usage? > > Thanks, > C

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations Hequn. Best Yun Tang From: Rong Rong Sent: Thursday, August 8, 2019 0:41 Cc: dev ; user Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer Congratulations Hequn, well deserved! -- Rong On Wed, Aug 7, 2019 at 8:30 AM mailto:xingc...@gmail.c

Capping RocksDb memory usage

Hello everyone, What is the most easy and efficiently way to cap RocksDb's memory usage? Thanks, Cam

Re: how to get the code produced by Flink Code Generator

+1. I think this would be a very nice way to provide more verbose feedback for debugging. -- Rong On Wed, Aug 7, 2019 at 9:28 AM Fabian Hueske wrote: > Hi Vincent, > > I don't think there is such a flag in Flink. > However, this sounds like a really good idea. > > Would you mind creating a Jira

Passing jvm options to flink

Hi , I am running flink on a standalone cluster without any resource manager like yarn or K8s. I am submitting my job using command line "*f**link run ..." . *I have a couple of questions: *1. *How can I pass JVM parameters to this job. I want to pass a parameter for a dylib like this -Djava.libr

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations Hequn, well deserved! -- Rong On Wed, Aug 7, 2019 at 8:30 AM wrote: > Congratulations, Hequn! > > > > *From:* Xintong Song > *Sent:* Wednesday, August 07, 2019 10:41 AM > *To:* d...@flink.apache.org > *Cc:* user > *Subject:* Re: [ANNOUNCE] Hequn becomes a Flink committer > > >

Re: how to get the code produced by Flink Code Generator

Hi Vincent, I don't think there is such a flag in Flink. However, this sounds like a really good idea. Would you mind creating a Jira ticket for this? Thank you, Fabian Am Di., 6. Aug. 2019 um 17:53 Uhr schrieb Vincent Cai < caidezhi...@foxmail.com>: > Hi Users, > In Spark, we can invoke Data

RE: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations, Hequn! From: Xintong Song Sent: Wednesday, August 07, 2019 10:41 AM To: d...@flink.apache.org Cc: user Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer Congratulations~! Thank you~ Xintong Song On Wed, Aug 7, 2019 at 4:00 PM vino yang mailto:yanghua1..

Re: FlatMap returning Row<> based on ArrayList elements()

Hello Victor , You are totally right , so now this turn into is Flink capable to handle these cases where would be required define the type info in the row and the Table will infer the columns separated by comma or something similar? thanks AU On Wed, Aug 7, 2019 at 10:33 AM Victor Wong wrote:

Re: Delayed processing and Rate limiting

Hi Shakir, > Delayed Processing Maybe you can make use of the function ‘org.apache.flink.streaming.api.TimerService#registerProcessingTimeTimer’, check this doc for more details: https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/process_function.html#example > Rate Lim

Greedy operator in match_recognize doesn't work with defined time interval as I expect it to work

Hi there, I built myself a small example in order to test MATCH_RECOGNIZE in flink. I have a pattern of format "A B* C", where B is a `1=1` event matching anything. Additionally, I have a time interval of '2' DAY. Now I create testdata matching event A, send some random events (B), followed

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations~! Thank you~ Xintong Song On Wed, Aug 7, 2019 at 4:00 PM vino yang wrote: > Congratulations! > > highfei2...@126.com 于2019年8月7日周三 下午7:09写道： > > > Congrats Hequn! > > > > Best, > > Jeff Yang > > > > > > Original Message > > Subject: Re: [ANNOUNCE] Hequn beco

Re: FlatMap returning Row<> based on ArrayList elements()

Hi Andres, I’d like to share my thoughts: When you register a “Table”, you need to specify its “schema”, so how can you register the table when the number of elements/columns and data types are both nondeterministic. Correct me if I misunderstood your meaning. Best, Victor From: Andres

Delayed processing and Rate limiting

Hi Flink Team, I am looking for some direction/recommendation for below tasks 1. Delayed Processing: Having a use case where we need to process events after a time-delay from event time. Let’s say, the event happened at time t1, and it reached the Flink immediately, but I have to wait t1+2m

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations! highfei2...@126.com 于2019年8月7日周三 下午7:09写道： > Congrats Hequn! > > Best, > Jeff Yang > > > Original Message > Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer > From: Piotr Nowojski > To: JingsongLee > CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam

Re: FlatMap returning Row<> based on ArrayList elements()

Hello everyone, let me be more precis on what I'm looking for at the end because your example is right and very accurate in the way about how to turn an array into a Row() object. I have done it seamlessly: out.collect(Row.of(pelements.toArray())); Then I printed and the outcome is as expected:

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations Hequn, well deserved! Best, Terry Wang > 在 2019年8月7日，下午9:16，Oytun Tez 写道： > > Congratulations Hequn! > > --- > Oytun Tez > > M O T A W O R D > The World's Fastest Human Translation Platform. > oy...@motaword.com — www.motaword.com >

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations Hequn! --- Oytun Tez *M O T A W O R D* The World's Fastest Human Translation Platform. oy...@motaword.com — www.motaword.com On Wed, Aug 7, 2019 at 9:03 AM Jark Wu wrote: > Congratulations Hequn! It's great to have you in the community! > > > > On Wed, 7 Aug 2019 at 21:00, Fab

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations Hequn! It's great to have you in the community! On Wed, 7 Aug 2019 at 21:00, Fabian Hueske wrote: > Congratulations Hequn! > > Am Mi., 7. Aug. 2019 um 14:50 Uhr schrieb Robert Metzger < > rmetz...@apache.org>: > >> Congratulations! >> >> On Wed, Aug 7, 2019 at 1:09 PM highfei2.

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations Hequn! Am Mi., 7. Aug. 2019 um 14:50 Uhr schrieb Robert Metzger < rmetz...@apache.org>: > Congratulations! > > On Wed, Aug 7, 2019 at 1:09 PM highfei2...@126.com > wrote: > > > Congrats Hequn! > > > > Best, > > Jeff Yang > > > > > > Original Message > > Subject:

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations! On Wed, Aug 7, 2019 at 1:09 PM highfei2...@126.com wrote: > Congrats Hequn! > > Best, > Jeff Yang > > > Original Message > Subject: Re: [ANNOUNCE] Hequn becomes a Flink committer > From: Piotr Nowojski > To: JingsongLee > CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff

Re: Configure Prometheus Exporter

Got it. I’ll look into this option. Thanks! > On Aug 7, 2019, at 08:46, Chesnay Schepler wrote: > > The only thing you can do at the moment, to limit which metrics are exposed, > is to implement your own MetricsReporter. You could extend the prometheus > one, and introduce any rules you want i

Re: Configure Prometheus Exporter

The only thing you can do at the moment, to limit which metrics are exposed, is to implement your own MetricsReporter. You could extend the prometheus one, and introduce any rules you want into notifyOfAddedMetric(). On 07/08/2019 14:42, Chaoran Yu wrote: Thanks for the reply. Yes. That’s what

Re: Configure Prometheus Exporter

Thanks for the reply. Yes. That’s what I’m trying to do. I think Flink by default exports all metrics. Is there anything else I can do to achieve this goal? > On Aug 7, 2019, at 03:58, Chesnay Schepler wrote: > > This is not possible. Are you trying to limit which metrics are exposed? > > On

XML equivalent to JsonRowDeserializationSchema

Hi everyone, I've had a good experience with JsonRowDeserializationSchema to deserialise nested json records according to Flink's TypeInformation. As I look to process XML records now, does anyone use an equivalent for this format please? Nested records may be a pain to process sometimes and sinc

Re: Does RocksDBStateBackend need a separate RocksDB service？

There is no need to add an external RocksDB instance . *The RocksDBStateBackend holds in-flight data in a RocksDB database that is (per default) stored in the TaskManager data directories. * [1] https://ci.apache.org/projects/flink/flink-docs-stable/ops/state/state_backen

Re:[ANNOUNCE] Hequn becomes a Flink committer

Congrats Hequn!Best,Jeff Yang Original Message Subject: Re: [ANNOUNCE] Hequn becomes a Flink committerFrom: Piotr Nowojski To: JingsongLee CC: Biao Liu ,Zhu Zhu ,Zili Chen ,Jeff Zhang ,Paul Lam ,jincheng sun ,dev ,user Congratulations :)On 7 Aug 2019, at 12:09, JingsongLee

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations :) > On 7 Aug 2019, at 12:09, JingsongLee wrote: > > Congrats Hequn! > > Best, > Jingsong Lee > > -- > From:Biao Liu > Send Time:2019年8月7日(星期三) 12:05 > To:Zhu Zhu > Cc:Zili Chen ; Jeff Zhang ; Paul Lam > ; jinch

Does RocksDBStateBackend need a separate RocksDB service？

In my code, I just setStateBackend with a hdfs direcoty. env.setStateBackend(new RocksDBStateBackend("hdfs://user/test/job")); Is there an embeded RocksDB service in the flink task? wangl...@geekplus.com.cn

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congrats Hequn! Best, Jingsong Lee -- From:Biao Liu Send Time:2019年8月7日(星期三) 12:05 To:Zhu Zhu Cc:Zili Chen ; Jeff Zhang ; Paul Lam ; jincheng sun ; dev ; user Subject:Re: [ANNOUNCE] Hequn becomes a Flink committer Congrats Heq

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congrats Hequn! Thanks, Biao /'bɪ.aʊ/ On Wed, Aug 7, 2019 at 6:00 PM Zhu Zhu wrote: > Congratulations to Hequn! > > Thanks, > Zhu Zhu > > Zili Chen 于2019年8月7日周三 下午5:16写道： > >> Congrats Hequn! >> >> Best, >> tison. >> >> >> Jeff Zhang 于2019年8月7日周三 下午5:14写道： >> >>> Congrats Hequn! >>> >>> Pau

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations to Hequn! Thanks, Zhu Zhu Zili Chen 于2019年8月7日周三 下午5:16写道： > Congrats Hequn! > > Best, > tison. > > > Jeff Zhang 于2019年8月7日周三 下午5:14写道： > >> Congrats Hequn! >> >> Paul Lam 于2019年8月7日周三 下午5:08写道： >> >>> Congrats Hequn! Well deserved! >>> >>> Best, >>> Paul Lam >>> >>> 在 2019年8月

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations Hequn! Glad to have you in the community! On 07/08/2019 10:28, jincheng sun wrote: > Hi everyone, > > I'm very happy to announce that Hequn accepted the offer of the Flink > PMC to become a committer of the Flink project. > > Hequn has been contributing to Flink for many years, mai

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congrats Hequn and welcome onboard as a committer :-) Cheers, Till On Wed, Aug 7, 2019 at 11:30 AM Becket Qin wrote: > Congrats, Hequn! Well deserved! > > On Wed, Aug 7, 2019 at 11:16 AM Zili Chen wrote: > >> Congrats Hequn! >> >> Best, >> tison. >> >> >> Jeff Zhang 于2019年8月7日周三 下午5:14写道： >>

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congrats, Hequn! Well deserved! On Wed, Aug 7, 2019 at 11:16 AM Zili Chen wrote: > Congrats Hequn! > > Best, > tison. > > > Jeff Zhang 于2019年8月7日周三 下午5:14写道： > >> Congrats Hequn! >> >> Paul Lam 于2019年8月7日周三 下午5:08写道： >> >>> Congrats Hequn! Well deserved! >>> >>> Best, >>> Paul Lam >>> >>> 在 20

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congrats Hequn! Best, tison. Jeff Zhang 于2019年8月7日周三 下午5:14写道： > Congrats Hequn! > > Paul Lam 于2019年8月7日周三 下午5:08写道： > >> Congrats Hequn! Well deserved! >> >> Best, >> Paul Lam >> >> 在 2019年8月7日，16:28，jincheng sun 写道： >> >> Hi everyone, >> >> I'm very happy to announce that Hequn accepted th

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congratulations, Hequn! Well deserved! > 在 2019年8月7日，下午5:13，Jeff Zhang 写道： > > Congrats Hequn! > > Paul Lam mailto:paullin3...@gmail.com>> 于2019年8月7日周三 > 下午5:08写道： > Congrats Hequn! Well deserved! > > Best, > Paul Lam > >> 在 2019年8月7日，16:28，jincheng sun > > 写

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congrats Hequn! Paul Lam 于2019年8月7日周三 下午5:08写道： > Congrats Hequn! Well deserved! > > Best, > Paul Lam > > 在 2019年8月7日，16:28，jincheng sun 写道： > > Hi everyone, > > I'm very happy to announce that Hequn accepted the offer of the Flink PMC > to become a committer of the Flink project. > > Hequn has

Re: [ANNOUNCE] Hequn becomes a Flink committer

Congrats Hequn! Well deserved! Best, Paul Lam > 在 2019年8月7日，16:28，jincheng sun 写道： > > Hi everyone, > > I'm very happy to announce that Hequn accepted the offer of the Flink PMC to > become a committer of the Flink project. > > Hequn has been contributing to Flink for many years, mainly work

Re: From Kafka Stream to Flink

Hi, LAST_VAL is not a built-in function, so you'd need to implement it as a user-defined aggregate function (UDAGG) and register it. The problem with joining an append only table with an updating table is the following. Consider two tables: users (uid, name, zip) and orders (oid, uid, product),

[ANNOUNCE] Hequn becomes a Flink committer

Hi everyone, I'm very happy to announce that Hequn accepted the offer of the Flink PMC to become a committer of the Flink project. Hequn has been contributing to Flink for many years, mainly working on SQL/Table API features. He's also frequently helping out on the user mailing lists and helping

Re: Consuming data from dynamoDB streams to flink

Hi Andrey, Thank you for your reply, I understand that the checkpoints are gone when the job is cancelled or killed, may be configuring external checkpoints will help here so that we can resume from there. My points was if the job is terminated, and the stream position is set to TRIM_HORIZON , th

Re: Configure Prometheus Exporter

This is not possible. Are you trying to limit which metrics are exposed? On 07/08/2019 06:52, Chaoran Yu wrote: Hello guys,    Does anyone know if the Prometheus metrics exported via the JMX reporter or the Prometheus reporter can be configured using a YAML file similar to this one

Re:FlatMap returning Row<> based on ArrayList elements()

Hi Andres Angel, I guess people don't understand your problem (including me). I don't know if the following sample code is what you want, if not, can you describe the problem more clearly? StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.fromElements

Re: Flink 1.8.1: Seeing {"errors":["Not found."]} when trying to access the Jobmanagers web interface

Yes, I did, but I am using a session cluster https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/deployment/docker.html#flink-session-cluster to run many jobs On Tue, Aug 6, 2019 at 2:05 PM Ufuk Celebi wrote: > Hey Tobias, > > out of curiosity: were you using the job/application clus