Re: Savepoint failed with error "Checkpoint expired before completing"

Haha, actually externalized checkpoint also support parallelism changes, you could read my email posted in dev-mail-list.

Re: Flink CEP Watermark Exception

Hi Austin, Could you elaborate a bit more what do you mean by "after a checkpoint fails", what is the reason why checkpoint fails? Would it be possible for you to prepare some reproducible example for that problem? Finally, I would also recommend trying out Flink 1.6.x, as we reworked the underlyi

Re: 1.6 UI issues

Hi Juan >From our experience, you could check the jobmanager.log first to see whether >existing similar logs below: max allowed size 128000 bytes, actual size of encoded class akka.actor.Status$Success was xxx bytes If you see these logs, you should increase the akka.framesize to larger value

Re: Flink SQL string literal does not support double quotation?

Hi Henry, In most SQL conventions, single quotes are for Strings, while double quotes are for identifiers. Best, Xingcan > On Oct 31, 2018, at 7:53 PM, 徐涛 wrote: > > Hi Experts, > When I am running the following SQL in FLink 1.6.2, I got > org.apache.calcite.sql.parser.impl.ParseExcept

Re: Continue batch job with streaming job

Hi Sergei, Batch mode does not support state. At the moment there is no “out of the box” way to bootstrap the state for streaming job. You can have a look at Bravo tool [1] which is developed to address this problem. Best, Andrey [1] https://github.com/king/bravo > On 29 Oct 2018, at 22:29, S

Re: Flink CEP Watermark Exception

Hi Dawid, Thank you for your reply. I'm out for the next few days, so I hope you don't mind me cc'ing my team in here. We all really appreciate you and the rest of the people monitoring the mailing list. We've only seen this SharedBuffer problem in production, after sending around 20 GB of data

Re: Error with custom `SourceFunction` wrapping `InputFormatSourceFunction` (Flink 1.6)

Hey Friends! Last ping and I'll move this over to a ticket. If anyone can provide any insight or advice, that would be helpful! Thanks again. Best, Aaron Levin On Fri, Oct 26, 2018 at 9:55 AM, Aaron Levin wrote: > Hey, > > Not sure how convo threading works on this list, so in case the folks

Re: Error with custom `SourceFunction` wrapping `InputFormatSourceFunction` (Flink 1.6)

Hi Aaron, I'l like to take a step back and understand why you're trying to wrap an InputFormatSourceFunction? In my opinion, InputFormatSourceFunction should not be used because it has some shortcomings, the most prominent among them that it does not support checkpointing, i.e. in case of fail

Re: TaskManagers cannot contact JobManager in Kubernetes when JobManager HA is enabled

I've managed to resolve the issue. With HA enabled, you will see this message in the logs: 2018-11-01 13:38:52,467 INFO org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Actor system started at akka.tcp://flink@flink-jobmanager:40641 Without HA enabled, you will see this messag

Re: Error with custom `SourceFunction` wrapping `InputFormatSourceFunction` (Flink 1.6)

Hey, Thanks for reaching out! I'd love to take a step back and find a better solution, so I'll try to be succint in what I'm trying to accomplish: We're trying to write a SourceFunction which: * reads some Sequence files from S3 in a particular order (each task gets files in a specific order). *

FileNotFoundException on starting the job

Hi everyone, I'm having an issue when restarting a job in Flink. I'm doing a simple stop with savepoint and then start from the savepoint. Savepoints are stored in a separate folder, there is no configuration for "/tmp" folder in my setup. There is only 1 task manager and parallelism is 1. I'm ge

Re: Using a ProcessFunction as a "Source"

This is fairly stale, but getting back to this: We ended up going the route of using the Operator API and implementing something similar to the `readFile` API with one real source function that reads out splits and then a small abstraction over the AbstractStreamOperator, a `MessagableSourceFunct

Re: Parallelize an incoming stream into 5 streams with the same data

Thanks,Hequn. If I have to do a TumblingWindow operation like: .timeWindowAll(org.apache.flink.streaming.api.windowing.time.Time.of(FIVE, TimeUnit.SECONDS)) I am not able to do that on the output of keyBy(..) which is a KeyedStream. I was hoping to groupBy(key._1,key._2) etc and then do a tumbli

Starting a seperate Java process within a Flink cluster

Hello, I am currently working on my masters and I encountered a difficult problem. Background (for context): I am trying to connect different data stream processors. Therefore i am using Flink's internal mechanisms of creating custom sinks and sources to receive from and send to different dat

Flink SQL questions

I am trying to get a minimal Flink SQL program going to test it out. I have a StreamExecutionEnvironment and from that created a StreamTableEnvironment. The docs indicate there should be a fromDataStream method to create a Table, but none appears to exist according to Eclipse. The method regi

Job manager UI improvement

I would really like to see the job manager show metrics on state size, not just io per task. Is there a way to do that now, or is the metric there, and just needs some UI Ewok to show it? Michael Sent from my iPad

Re: Parallelize an incoming stream into 5 streams with the same data

Hi Vijay, > I was hoping to groupBy(key._1,key._2) etc and then do a tumblingWindow operation on the KeyedStream and then perform group operation on the resultant set to get total count etc. >From your description, I think you can perform a TumblingEventTimeWindow first, something looks like: >

Re: Flink SQL questions

Hi Michael, There are some test cases in Flink git, such as[1] which I think may help you. Best, Hequn [1] https://github.com/apache/flink/blob/master/flink-libraries/flink-table/src/test/java/org/apache/flink/table/runtime/stream/sql/JavaSqlITCase.java On Fri, Nov 2, 2018 at 7:46 AM TechnoMage

Re: Flink SQL questions

Thanks, I will check it out. Michael Sent from my iPad > On Nov 1, 2018, at 8:22 PM, Hequn Cheng wrote: > > Hi Michael, > > There are some test cases in Flink git, such as[1] which I think may help you. > > Best, Hequn > [1] > https://github.com/apache/flink/blob/master/flink-libraries/fli

Re: Starting a seperate Java process within a Flink cluster

Hi Since you use the message-buffer-process as a dependency and the error tells you class not found, have you ever check your application jar package whether containing the wanted MessageBufferProcess.class? If not existed, try to use assembly-maven

Re: Job manager UI improvement

Hi Michael You could view state size metrics in 'Checkpoints' UI tab[1], I think the state size shown here could meet your needs in most cases. [1] https://ci.apache.org/projects/flink/flink-docs-stable/monitoring/checkpoint_monitoring.html#history-tab Apache Flink 1.6 Documentation: Monitoring

java.io.IOException: NSS is already initialized

I am on Flink 1.6.2 (no Hadoop, in docker + K8S), using rocksdb and S3 (presto) I got this error when flink creating a checking point === 2018-11-02 04:00:55,011 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Job ConnectedStreams maxwell.accounts (00

Re: Flink SQL questions

Usually, the problem occurs when users import the wrong classes. The current class naming is a bit confusing as there are 3 StreamTableEnvironment classes. You need to choose the one that matches your programming language. E.g. org.apache.flink.table.api.java.StreamTableEnvironment. Regards,