Re: Problem to show logs in task managers

Hi Till, Thanks for that! I can see the "Logger in LineSplitter.flatMap” output if I retrieve the task manager logs manually (under /var/log/hadoop-yarn/containers/application_X/…). However that solution is not ideal when for instance I am using 32 machines for my mapReduce operations. I would

Re: Problem to show logs in task managers

Hi Ana, good to hear that you found the logging statements. You can check in Yarn’s web interface whether there are still occupied containers. Alternatively you can go to the different machines and run jps which lists you the running Java processes. If you see an ApplicationMaster or YarnTaskManag

Re: Machine Learning on Apache Fink

Hi Ashutosh, Flink’s ML library flinkML is maybe not as extensive as Spark’s MLlib. However, Flink has native support for iterations which makes them blazingly fast. Iterations in Flink are a distinct operator so that they don’t have to communicate after each iteration with the client. Furthermore

Flink streaming Python

Hi Is streaming supported in Flink-Python API? If so, can you point me to the documentation?

Re: Problem to show logs in task managers

Hi Till, Thanks for your help. I have checked both in Yarn’s web interface and through command line and it seems that there are not occupied containers. Additionally, I have checked the configuration values in the web interface and even though I have changed the log.aggregation property in the

Re: Flink streaming Python

> On 11 Jan 2016, at 13:03, Madhukar Thota wrote: > > Hi > > Is streaming supported in Flink-Python API? If so, can you point me to the > documentation? No. Only the DataSet API has Python support at the moment. I expect this to change at some point in time, but I’m not aware of any concret

Re: Problem to show logs in task managers

You have to restart the yarn cluster to let your changes take effect. You can do that via HADOOP_HOME/sbin/stop-yarn.sh; HADOOP_HOME/sbin/start-yarn.sh. The commands yarn-session.sh ... and bin/flink run -m yarn cluster start a new yarn application within the yarn cluster. Cheers, Till ​ On Mon,

Re: Security in Flink

Thanks Steven for your details response. Things are more clear to me now. A follow up Qs - Looks like most of the security support depends on Hadoop ? What happens if anyone wants to use Flink with Hadoop (in a cluster where Hadoop is not there) ? Regards, Sourav On Sun, Jan 10, 2016 at 12:41 PM

Flink with Yarn

I am going through the documentation of integrating Flink with YARN. However not sure whether Flink can be run on YARN in two modes (like Spark). In one mode the driver/client program of Flink is also managed by YARN. In the second mode where the client program is outside the control of YARN. Is t

Re: Flink with Yarn

Hi! Flink is different than Spark in that respect. The driver program in Flink can submit a program to the master (in YARN Application Master) and disconnect then. It is not a part of the distributed execution - that is coordinated only by the master (JobManager). The driver can stay connected to

Re: Flink with Yarn

Hi Stephan, Thanks for the explanation. >From your explanation it looks like Flink runs in a mode similar to Spark's YARN-Client mode. Regards, Sourav On Mon, Jan 11, 2016 at 8:27 AM, Stephan Ewen wrote: > Hi! > > Flink is different than Spark in that respect. The driver program in Flink > ca

Re: Flink with Yarn

Hi Sourav, A little help with more clarification on your last comment. In sense of "where" the driver program is executed, then yes the Flink driver program runs in a mode similar to Spark's YARN-client. However, the "role" of the driver program and the work that it is responsible of is quite di

Re: Flink with Yarn

Hi Gordon, Thanks for the explanation. It is much clear now. Looks like a much cleaner approach. In that way the driver program can run in a machine which does not need connectivity to all worker nodes. Regards, Sourav On Mon, Jan 11, 2016 at 9:22 AM, Tzu-Li (Gordon) Tai wrote: > Hi Sourav, >

How to sort tuples in DataStream

Hi, I'm trying to do a simple application in Flink Stream to count the top N words on a window-basis, but I don't know how to sort the words by their frequency in Flink. In spark streaming, I would do something like this: « val isAscending = true stream.reduceByKeyAndWindow(reduceFunc, Seconds(10

DeserializationSchema isEndOfStream usage?

Hello all, I saw that DeserializationSchema has an API "isEndOfStream()". https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/util/serialization/DeserializationSchema.java Can *isEndOfStream* be utilized to somehow terminate a streaming flink

Re: DeserializationSchema isEndOfStream usage?

Hi David, In theory isEndOfStream() is absolutely the right way to go for stopping data sources in Flink. That its not working as expected is a bug. I have a pending pull request for adding a Kafka 0.9 connector, which fixes this issue as well (for all supported Kafka versions). Sorry for the

SQL query support in Flink

Hi, Just wanted to check whether one can directly run SQL queries on Flink. For example whether one can define a table on a dataset and then run queries like dataset.sql ("select column1, column2 from mytable"). I used to think that this is possible right now in 0.10.1. But when I checked the bl

Re: Security in Flink

Hi Stephen, Do you have any plan on which encryption method and mechanism will be used on Flink ? Could you share about the detail on this ? We have very strict requirement from client that every communication need to be encryption. So any detail would be really appreciated for answering their se