StackOverflow Error

Hi, I am trying to run a pipeline on Flink 1.8.1 ,getting the following exception: *java.lang.StackOverflowError at java.lang.Exception.(Exception.java:66) at java.lang.ReflectiveOperationException.(ReflectiveOperationException.java:56) at java.lang.NoSuchMethodException.(NoSuchM

Re: [DISCUSS] Create a Flink ecosystem website

[Sorry for the incomplete message. Clicked send by mistake...] I agree with Marta that it might be good to have multi-language support as a mid-term goal. Jiangjie (Becket) Qin On Sat, Jul 20, 2019 at 11:22 AM Becket Qin wrote: > The website is awesome! I really like its conciseness and yet fa

Re: [DISCUSS] Create a Flink ecosystem website

The website is awesome! I really like its conciseness and yet fairly useful information and functionalities. I cannot think of much to improve at the moment. Just one thought, do we need an "others" category, just in case a package does not fit into any of the current given categories? Thanks Robe

Re: [Table API] ClassCastException when converting a table to DataStream

Hi Rong, I have to dig deeper into the code to reproduce this error. This seems to > be a bug to me and will update once I find anything. Thanks a lot for spending your time on this. However from what you explained, if I understand correctly you can do all > of your processing within the TableAP

Re: Jython support for Flink

Hi Dante, Both Jython and Jython support for Flink are out of development and maintain. As pointed out by Jeff, Flink 1.9 supports Python api via py4j[1] and the document page as posted. I guess your algorithms are written in CPython instead of Jython and want Jython only for interoperate, and th

Re: Job submission timeout with no error info.

Hi Andrey, Flink version: 1.4.2 Please find the client log attached and job manager log is at: job manager log. Thanks, -Fakrudeen (define (sqrte n xn eph) (if (> eph (abs (- n (* xn xn xn (sqrte n (/ (+ xn

From Kafka Stream to Flink

Hi, I am a user of Kafka Stream so far. However, because i have been face with several limitation in particular in performing Join on KTable. I was wondering what is the appraoch in Flink to achieve (1) the concept of KTable, i.e. a Table that represent a changeLog, i.e. only the latest version

Re: Extending REST API with new endpoints

Yep, I scanned all of the issues in Jira and the codebase, I couldn't find a way to plug my new endpoint in. I am basically trying to open up an endpoint for queryable state client. I also read somewhere that this may cause some issues due to SSL communication within the cluster. Any pointers?

Extending REST API with new endpoints

Hi there, I am trying to add a new endpoint to the REST API, by extending AbstractRestHandler. But this new handler needs to be added in WebMonitorEndpoint, which has no interface for outside. Can I do this with 1.8? Any other way or plans to make this possible? --- Oytun Tez *M O T A W O R D*

Re: Apache Flink - Manipulating the state of an object in an evictor or trigger

Biao: I am asking this question, just to understand the impact and best practices around it.  The state I am referring to it the objects that are passed to evictor. https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/operators/windows.html#evictors void evictBefore(Iterable> eleme

Re: Apache Flink - Event time and process time timers with same timestamp

Hi Bioa/Andrey: Just to clarify, can we register two timers (one for processing time and one for event time) with the same timestamp and if so, which one will fire. Also, is it ok to register an event time time and then deregister processing time time (or vice versa) ?  Here is the example I am

Re: Apache Flink - Side output time semantics for DataStream

Hi Biao:  Thanks for your response. My question is that if i have streaming application and with timestamp assigned to the elements, and in one of the process functions I have side output, then there are two situations possible:1. The side output is the same type as main stream.2. The side outp

Re: Job submission timeout with no error info.

Hi Fakrudeen, which Flink version do you use? could you share full client and job manager logs? Best, Andrey On Fri, Jul 19, 2019 at 7:00 PM Fakrudeen Ali Ahmed wrote: > Hi, > > > > We are submitting a Flink topology [YARN] and it fails during upload of > the jar with no error info. > > > > [m

Re: [DISCUSS] Create a Flink ecosystem website

Hey, Robert. I will keep an eye on the overall progress and get started on the blog post to make the community announcement. Are there (mid-term) plans to translate/localize this website as well? It might be a point worth mentioning in the blogpost. Hats off to you and Daryl — this turned out ama

Re: Consuming data from dynamoDB streams to flink

Hi Vinay, 1. I would assume it works similar to kinesis connector (correct me if wrong, people who actually developed it) 2. If you have activated just checkpointing, the checkpoints are gone if you externally kill the job. You might be interested in savepoints [1] 3. See paragraph in [2] about ki

Job submission timeout with no error info.

Hi, We are submitting a Flink topology [YARN] and it fails during upload of the jar with no error info. [main] INFO org.apache.flink.runtime.client.JobClient - Checking and uploading JAR files [main] ERROR org.apache.flink.client.CliFrontend - Error while running the command. org.apache.fli

Re: Apache Flink - Event time and process time timers with same timestamp

Hi, Event and processing time timers have independent state storage. You can use both independently, so I would expect two firings with different domains. `TimeCharacteristic` is for operations where you do not explicitly tell the time type, like windowing. Best, Andrey On Fri, Jul 19, 2019 at 8

Re: [Table API] ClassCastException when converting a table to DataStream

Hi Dongwon, I have to dig deeper into the code to reproduce this error. This seems to be a bug to me and will update once I find anything. However from what you explained, if I understand correctly you can do all of your processing within the TableAPI scope without converting it back and forth to

Re: Checkpoints timing out for no apparent reason

Hi Sergei, If you want just to try increasing the timeouts, you could change the checkpoint timeout in env.getCheckpointConfig().setCheckpointTimeout(...) [1] or s3 client timeouts (see presto or hdfs for s3 configuration, there are some network timeouts) [2]. Otherwise it would be easier to inve

Re: Jython support for Flink

Hi Dante, Flink 1.9 support python api, which may be what you want. See https://ci.apache.org/projects/flink/flink-docs-master/tutorials/python_table_api.html Dante Van den Broeke 于2019年7月19日周五 下午10:40写道： > Dear, > > > I'm a student currently working on a project involving apache kafka and > f

Jython support for Flink

Dear, I'm a student currently working on a project involving apache kafka and flink. The project itself is revolved around path prediction and machine learning for websites. To test a prove of concept I setup a kafka server locally (goal is to expend this to a google cloud server or similar la

Re: apache flink: Why checkpoint coordinator takes long time to get completion

btw. it seems like this issue has been fixed in 1.8.1 On Fri, 19 Jul 2019 at 12:21, Xiangyu Su wrote: > Ok, thanks. > > and this time-consuming until now always happens after 3rd checkpointing, > and this unexpected time-consuming was always consistent (~ 4 min by under > 4G/min incoming traffi

Re: Consuming data from dynamoDB streams to flink

Hi, I am using this consumer for processing records from DynamoDb Streams , few questions on this : 1. How does checkpointing works with Dstreams, since this class is extending FlinkKinesisConsumer, I am assuming it will start from the last successful checkpoint in case of failure, right ? 2. Cur

Re: apache flink: Why checkpoint coordinator takes long time to get completion

Ok, thanks. and this time-consuming until now always happens after 3rd checkpointing, and this unexpected time-consuming was always consistent (~ 4 min by under 4G/min incoming traffic). On Fri, 19 Jul 2019 at 11:06, Biao Liu wrote: > Hi Xiangyu, > > Just took a glance at the relevant codes. T

Re: left join failing with FlinkLogicalJoinConverter NPE

Hi, I also found the similar issue here [1]. Best, Tony Wei [1] https://issues.apache.org/jira/browse/FLINK-11433 Tony Wei 於 2019年7月19日 週五 下午5:38寫道： > Hi, > > Is there any update for this issue? I have had the same problem just like > Karl's. > After I remove query like "select collect(data) .

Re: left join failing with FlinkLogicalJoinConverter NPE

Hi, Is there any update for this issue? I have had the same problem just like Karl's. After I remove query like "select collect(data) ..." from one of the joined tables, the sql can be executed correctly without throwing any NPE. Best regards, Tony Wei Xingcan Cui 於 2019年2月27日 週三 下午12:53寫道： >

Re: apache flink: Why checkpoint coordinator takes long time to get completion

Hi Xiangyu, Just took a glance at the relevant codes. There is a gap between calculating the duration and logging it out. I guess the checkpoint 4 is finished in 1 minute, but there is an unexpected time-consuming operation during that time. But I can't tell which part it is. Xiangyu Su 于2019年7

Parallelism issue

Hello. I wrote below codes. It works extraordinarily. Processing performs after SourceFunction generates all data and quit. If SourceFunction works infinitely, processing is never performed. But, it works well when parallelismForTimestamp is given other value (eg. 3), I want to know the mecha

Re:Re: Writing Flink logs into specific file

Hi, Soheil Placing the log configuration file in the resource directory of the job's jar will not be used by Flink, because the log configuration is explicitly specified by the script under the bin directory of Flink and the bootstrap code （for example the BootstrapTools class). If you want to

apache flink: Why checkpoint coordinator takes long time to get completion

Dear flink community, We are POC flink(1.8) to process data in real time, and using global checkpointing(S3) and local checkpointing(EBS), deploy cluster on EKS. Our application is consuming data from Kinesis. For my test e.g I am using checkpointing interval 5min. and minimum pause 2min. The is

Fwd: Issue running basic example locally

Just forward it to user mailing list. it's not a development issue. -- Forwarded message - 发件人： Caizhi Weng Date: 2019年7月19日周五 上午8:56 Subject: Re: Issue running basic example locally To: Hi Andres, `provided` of flink-streaming-java seems suspicious, can you remove it and see