Flink Vs Google Cloud Dataflow?

Did anyone study Cloud DataFlow as an alternative to Flink? If yes, can someone summarize their analysis of Dataflow as against Flink?

Re: AW: Is watermark used by joining two streams

Hi Fabian, How do I order by the merge time. Let's say I merge the stream at T1. I wanted to drop T2 merge if T2 < T1. Now depending on the arrival of data from individual stream and the time at which the merge happens, they become out of order. Any thoughts will be really appreciated. Regards, V

Re: a lot of connections in state "CLOSE_WAIT"

1. yes and yes. 2. Yes,it was shown correctly. 3.I wasn’t modify this setting. > 在 2017年7月26日，18:06，Chesnay Schepler [via Apache Flink User Mailing List > archive.] 写道： > > So this only happens when you select a metric? Without a selected metric > everything works fine? > > Are the metrics yo

re: How can I submit a flink job to YARN/Cluster from java code?

You can see the detail information at https://ci.apache.org/projects/flink/flink-docs-release-1.3/monitoring/rest_api.html. In this link, it is not said the restful API can not used in yarn cluster. I think it is ok, and you can try. :) z...@zjdex.com ?? 2017-07-28

Re: Invalid path exception

This is what I tired and it doesn't work. Is this a bug? format.setFilePath("file:///c:/proj/test/a.txt.txt"); On Sun, Jul 30, 2017 at 2:10 PM, Chesnay Schepler wrote: > Did the path by chance start with file://C:/... ? > > If so, please try file:///C: ... > > > On 30.07.2017 22:28, Mohit Anchl

Re: Invalid path exception

Did the path by chance start with file://C:/... ? If so, please try file:///C: ... On 30.07.2017 22:28, Mohit Anchlia wrote: I am using flink 1.3.1 and getting this exception. Is there a workaround? Caused by: _java.nio.file.InvalidPathException_: Illegal char <:> at index 2: /C:/Users/m/defa

Invalid path exception

I am using flink 1.3.1 and getting this exception. Is there a workaround? Caused by: *java.nio.file.InvalidPathException*: Illegal char <:> at index 2: /C:/Users/m/default/flink-example/pom.xml at sun.nio.fs.WindowsPathParser.normalize(Unknown Source) at sun.nio.fs.WindowsPathParser.parse(Unknow

AW: Is watermark used by joining two streams

Hello Fabian, thank you for your answer! Does it mean that the operator will wait until get two watermarks from the input streams and emits then the “slower” watermark? Best regards Wei Von: Fabian Hueske [mailto:fhue...@gmail.com] Gesendet: Sunday, July 30, 2017 11:17 AM An: xie

Flink -mesos-app master hang

Hi All, I'm trying to run a flink docker from the marathon with mesos app master; I could see it goes on a continuous loop and failed to launch the task manger. If I go to mesos master UI I could see job manager web UI with task manager zero . I have pretty much checked every possible log starting

Re: Customer inputformat

For #1, you can find quite a few classes which extend FileInputFormat. e.g. flink-connectors/flink-avro/src/main/java/org/apache/flink/api/java/io/AvroInputFormat.java:public class AvroInputFormat extends FileInputFormat implements ResultTypeQuer flink-core/src/main/java/org/apache/flink/api/commo

Re: Customer inputformat

Thanks. Few more questions: - Is there an example for FileInputFormat? - how to make it read all the files in a directory? - how to make an inputformat a streaming input instead of batch? Eg: read as new files come to a dir. Thanks again. On Sun, Jul 30, 2017 at 12:53 AM, Fabian Hueske wrote:

Re: Is watermark used by joining two streams

Periodic and punctuated watermarks only differ in the way that they are generated. Afterwards they are treated the same. An operator with two input streams will always sync its own watermarks to the watermarks of both input streams, i.e., to the "slower" watermark of both inputs. So if the left inp

Re: Reading byte[] from socket

Hi Paolo, have a look at SocketTextStreamFunction [1] which is internally used when StreamExecutionEnviornment.socketTextStream() is called. Best, Fabian [1] https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functions/source/SocketTextS

Re: Customer inputformat

Hi, Flink calls the reachedEnd() method before it calls nextRecord() and closes the IF when reachedEnd() returns true. So, it should not return true until nextRecord() was called and the first and last record was emitted. You might also want to built your PDFFileInputFormat on FileInputFormat and