Use cases for Flink

I have been trying out various scenarios with Flink with various sources and sinks.  I want to know how folks have been using Flink in production.  What are the common use cases.  Essentially,  I have implemented similar use cases in Spark and now I find it fairly straightforward to convert thes

Re: Re: part files written to HDFS with .pending extension

s, Urs On 02.09.2017 02:46, Krishnanand Khambadkone wrote: >  BTW, I am using a BucketingSink and a DateTimeBucketer.  Do I need to set >any other property to move the files from .pending state. > BucketingSink sink = new > BucketingSink("hdfs://localhost:8020/flinktwit

Re: part files written to HDFS with .pending extension

On Friday, September 1, 2017, 5:03:46 PM PDT, Krishnanand Khambadkone wrote: This message is eligible for Automatic Cleanup! (kkhambadk...@yahoo.com) Add cleanup rule | More info Hi,  I have written a small program that uses a Twitter input stream and a HDFS output sink.   When the files

part files written to HDFS with .pending extension

Hi,  I have written a small program that uses a Twitter input stream and a HDFS output sink.   When the files are written to HDFS each part file in the directory has a .pending extension.  I am able to cat the file and see the tweet text.  Is this normal for the part files to have .pending exten

Re: Error submitting flink job

t; > fs.overwrite-files: true in your flink-conf.yaml > > >> On 1 Sep 2017, at 23:36, Krishnanand Khambadkone >> wrote: >> >> I am trying to submit a flink job from the command line and seeing this >> error. Any idea what could be happening >> >

Re: Re: Error submitting flink job

-conf.yaml On 1 Sep 2017, at 23:36, Krishnanand Khambadkone wrote: I am trying to submit a flink job from the command line and seeing this error.   Any idea what could be happening java.io.IOException: File or directory already exists. Existing files and directories are not overwritten in

Error submitting flink job

I am trying to submit a flink job from the command line and seeing this error.   Any idea what could be happening java.io.IOException: File or directory already exists. Existing files and directories are not overwritten in NO_OVERWRITE mode. Use OVERWRITE mode to overwrite existing files and dir

Re: Twitter example

s. > The TM stdout is usually redirected to an out file in the ./log folder. > > 2017-08-30 17:20 GMT+02:00 Krishnanand Khambadkone : >> I am running this standalone, not under yarn, on a single instance setup. >> I believe you are referring to the flink log files >>

Re: Twitter example

> sender | Approve domain > Hi, > > print() writes the data to the out files of the TaskManagers. So you need to > go to the machine that runs the TM and check its out file which is located in > the log folder. > > Best, Fabian > > 2017-08-29 23:53 GMT+02:00 Krishnan

Twitter example

I am trying to run the basic twitter example,  it does deploy correctly and show as RUNNING both in the command line and in the UI.  However I see no output.  I am able to verify that my credentials are correct with another small client I have built that does connect to twitter and prints out tw

Re: Exception when trying to run flink twitter example

.    On Tuesday, August 22, 2017, 9:27:07 AM PDT, Till Rohrmann wrote: Hi Krishnanand, could you check that you have the build.properties file in you fat jar containing the field version=? Cheers,Till On Tue, Aug 22, 2017 at 6:19 PM, Krishnanand Khambadkone wrote: Hi,  I have created a fat jar

Exception when trying to run flink twitter example

Hi,  I have created a fat jar with my twitterexample classes and am running it like this,  ~/flink-1.3.2/build-target/bin/flink run -c TwitterExample ./flinktwitter.jar  --twitter-source.consumerKey --twitter-source.consumerSecret --twitter-source.token --twitter-source.tokenSecret I am pro