Re: Udf Performance and Object Creation

Thanks Timo That is a good interview question Best regards Hawin On Thu, Aug 13, 2015 at 1:11 AM, Michael Huelfenhaus < m.huelfenh...@davengo.com> wrote: > Hey Timo, > > yes that is what I needed to know. > > Thanks > - Michael > > Am 12.08.2015 um 12:44 schrieb Timo Walther : > > > Hello Mic

Re: FYI: Blog Post on Flink's Streaming Performance and Fault Tolerance

Great job, Guys Let me read it carefully. On Wed, Aug 5, 2015 at 7:25 AM, Stephan Ewen wrote: > I forgot the link ;-) > > > http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream-processing-with-apache-flink/ > > On Wed, Aug 5, 2015 at 4:11 PM, Stephan Ewen wrote: >

RE: Flink Kafka cannot find org/I0Itec/zkclient/serialize/ZkSerializer

Hi Wendong Please make sure you have dependencies as below. Good luck * org.apache.flink flink-java 0.9.0

Re: Flink Kafka cannot find org/I0Itec/zkclient/serialize/ZkSerializer

Maybe you can posted your pom.xml file to identify your issue. Best regards Hawin On Tue, Jul 21, 2015 at 2:57 PM, Wendong wrote: > also tried using zkclient-0.3.jar in lib/, updated build.sbt and rebuild. > It > doesn't help. Still got the same error of NoClassDefFoundError: > ZkSerializer >

Re: Sort Benchmark infrastructure

> Good luck on your attempts. > -George > > > George Porter > Assistant Professor, Dept. of Computer Science and Engineering > Associate Director, UCSD Center for Networked Systems > UC San Diego, La Jolla CA > http://www.cs.ucsd.edu/~gmporter/ > > > > On W

Re: Sort Benchmark infrastructure

ironment was pretty stock--we didn't get any special support or >> features from AWS. >> >> Best of luck with your profiling and benchmarking. Do let us know how >> you perform. Flink looks like a pretty interesting project, and so let us >> know if w

Re: Building Big Data Benchmarking suite for Apache Flink

Hi Slim I will follow this and keep you posted. Thanks. Best regards Hawin On Mon, Jul 13, 2015 at 7:04 PM, Slim Baltagi wrote: > Hi > > BigDataBench is an open source Big Data Benchmarking suite from both > industry and academia. As a subset of BigDataBench, BigDataBench-DCA is > China’

Sort Benchmark infrastructure

Hi Michael and George First of all, congratulation you guys have won the sort game again. We are coming from Flink community. I am not sure if it is possible to get your test environment to test our Flink for free. we saw that Apache spark did a good job as well. We want to challenge yo

Re: TeraSort on Flink and Spark

Hi Kim and Stephan Kim's report is sorting 3360GB per 1427 seconds by Flink 0.9.0. 3360 = 80*42 ((80GB/per node and 42 nodes) Based on Kim's report. The TPS is 2.35GB/sec for Flink 0.9.0 Kim was using 42 nodes for testing purposes. I found that the best Spark performance result was using

Re: Benchmark results between Flink and Spark

Hi Stephan Yes. You are correct. It looks like the TPCx-HS is an industry standard for big data. But how to get a Flink number on that. I think it is also difficult to get a Spark performance number based on TPCx-HS. if you know someone can provide servers for performance testing. I would like t

Re: Benchmark results between Flink and Spark

Hi Slim and Fabian Here is the Spark benchmark. https://amplab.cs.berkeley.edu/benchmark/ Do we have s similar report or comparison like that. Thanks. Best regards Hawin On Mon, Jul 6, 2015 at 6:32 AM, Slim Baltagi wrote: > Hi Fabian > > > I could not find which versions of Flink and Spar

RE: Best way to write data to HDFS by Flink

:09 PM To: user@flink.apache.org Subject: Re: Best way to write data to HDFS by Flink Dear Hawin, As for your issues with running the Flink Kafka examples: are those resolved with Aljoscha's comment in the other thread? :) Best, Marton On Fri, Jun 26, 2015 at 8:40 AM, Hawin

Re: Kafka0.8.2.1 + Flink0.9.0 issue

one where you get this > exception: > org.apache.commons.lang3.SerializationException: > java.io.StreamCorruptedException: invalid stream header: 68617769 > > Cheers, > Aljoscha > > On Fri, 26 Jun 2015 at 08:21 Hawin Jiang wrote: > >> Dear Marto

Re: Best way to write data to HDFS by Flink

n, >> >> We do not have out of the box support for that, it is something you would >> need to implement yourself in a custom SinkFunction. >> >> Best, >> >> Marton >> >> On Mon, Jun 22, 2015 at 11:51 PM, Hawin Jiang >> wrote: >> >&g

Re: Kafka0.8.2.1 + Flink0.9.0 issue

$producer$SyncProducer$$doSend(SyncProducer.scala:72) at kafka.producer.SyncProducer.send(SyncProducer.scala:113) at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:58) On Thu, Jun 25, 2015 at 11:06 PM, Hawin Jiang wrote: > Dear Marton > > I have upgraded my Flink to 0.9.

Re: Kafka0.8.2.1 + Flink0.9.0 issue

owing dependencies and it works fine . >> >> >> org.apache.flink >> flink-java >> 0.9-SNAPSHOT >> >> >> org.apache.flink >> flink-clients >> 0.9-SNAPSHOT >> >> >> org.apache.flink >> flink-streaming-core >>

Re: writeAsCsv on HDFS

HI Flavio Here is the example from Marton: You can used env.writeAsText method directly. StreamExecutionEnvironment env = StreamExecutionEnvironment. getExecutionEnvironment(); env.addSource(PerisitentKafkaSource(..)) .map(/* do you operations*/) .wirteAsText("hdfs://:/path/to/your/file

Re: Best way to write data to HDFS by Flink

provided a similar partition API or configuration for this. Thanks. Best regards Hawin On Wed, Jun 10, 2015 at 10:31 AM, Hawin Jiang wrote: > Thanks Marton > I will use this code to implement my testing. > > > > Best regards > Hawin > > On Wed, Jun 10, 2015 at 1:30

Re: Kafka0.8.2.1 + Flink0.9.0 issue

-dataflow/blob/master/pom.xml#L286-296 > > On Thu, Jun 11, 2015 at 10:32 AM, Hawin Jiang > wrote: > >> Dear Marton >> >> What do you meaning for locally Eclipse with 'Run'. >> Do you want to me to run it on Namenode? >> But my namenode didn't in

Re: Kafka0.8.2.1 + Flink0.9.0 issue

jar for your project containing all the dependencies with 'mvn > assembly:assembly'. Neither of these are beautiful but would help tracking > down the root cause. > > On Thu, Jun 11, 2015 at 10:04 AM, Hawin Jiang > wrote: > >> Dear Marton >> >> >> >> Than

RE: Kafka0.8.2.1 + Flink0.9.0 issue

find the kafka dependency. How are you trying to run this example? Is it from an IDE or submitting it to a flink cluster with bin/flink run? How do you define your dependencies, do you use maven or sbt for instance? Best, Marton On Thu, Jun 11, 2015 at 9:43 AM, Hawin Jiang wrote:

Kafka0.8.2.1 + Flink0.9.0 issue

Hi All I am preparing Kafka and Flink performance test now. In order to avoid my mistakes, I have downloaded Kafka example from http://kafka.apache.org/ and Flink streaming Kafka example from http://flink.apache.org I have run both producer examples on the same cluster. No any issues from ka

Re: Best wishes for Kostas Tzoumas and Robert Metzger

ink-deepdive-hadoop-summit-2015-in-san-jose-ca > Thank you for the wishes. The talk was very well received. > > On Wed, Jun 10, 2015 at 10:41 AM, Hawin Jiang > wrote: > >> Hi Michels >> >> I don't think you can watch them online now. >> >> Can

Re: Best wishes for Kostas Tzoumas and Robert Metzger

wondering, is it possible to stream the talks or watch them > later on? > > On Mon, Jun 8, 2015 at 2:54 AM, Hawin Jiang wrote: > >> Hi All >> >> >> >> As you know that Kostas Tzoumas and Robert Metzger will give us two Flink >> talks on 2015 Hado

Re: Best way to write data to HDFS by Flink

) > > .wirteAsText("hdfs://:/path/to/your/file"); > > Check out the relevant section of the streaming docs for more info. [1] > > [1] > http://ci.apache.org/projects/flink/flink-docs-master/apis/streaming_guide.html#connecting-to-the-outside-world > > Best,

Best way to write data to HDFS by Flink

Hi All Can someone tell me what is the best way to write data to HDFS when Flink received data from Kafka? Big thanks for your example. Best regards Hawin

Re: Apache Flink transactions

t some days ago ( > http://mail-archives.apache.org/mod_mbox/flink-user/201506.mbox/%3cd1972778.64426%25jspa...@cray.com%3e). > He seems to be running some tests to compare Flink, Spark and MapReduce. > > Regards, > Aljoscha > > On Mon, Jun 8, 2015 at 9:09 PM, Hawin Jiang wrot

Re: Apache Flink transactions

t; > Regards, > Aljoscha > > On Mon, Jun 8, 2015 at 9:09 PM, Hawin Jiang wrote: > >> Hi Aljoscha >> >> I want to know what is the apache flink performance if I run the same SQL >> as below. >> Do you have any apache flink benchmark information? >

Re: Apache Flink transactions

:03 AM, Aljoscha Krettek wrote: > Hi, > actually, what do you want to know about Flink SQL? > > Aljoscha > > On Sat, Jun 6, 2015 at 2:22 AM, Hawin Jiang wrote: > > Thanks all > > > > Actually, I want to know more info about Flink SQL and Flink performance >

Best wishes for Kostas Tzoumas and Robert Metzger

Hi All As you know that Kostas Tzoumas and Robert Metzger will give us two Flink talks on 2015 Hadoop summit. That is an excellent opportunity to introduce Apache Flink to the world. Best wishes for Kostas Tzoumas and Robert Metzger. Here is the details info: Topic: Apache Flink

Re: Apache Flink transactions

Thanks all Actually, I want to know more info about Flink SQL and Flink performance Here is the Spark benchmark. Maybe you already saw it before. https://amplab.cs.berkeley.edu/benchmark/ Thanks. Best regards Hawin On Fri, Jun 5, 2015 at 1:35 AM, Fabian Hueske wrote: > If you want to appe

Re: WELCOME to user@flink.apache.org

at 10:44 AM, Robert Metzger wrote: > Yes, I've got this message. > > On Thu, Jun 4, 2015 at 7:42 PM, Hawin Jiang wrote: > >> Hi Admin >> >> Please let me know if you are received my email or not. >> Thanks. >> >> >> >> Best reg

Re: WELCOME to user@flink.apache.org

Hi Admin Please let me know if you are received my email or not. Thanks. Best regards Hawin Jiang On Thu, Jun 4, 2015 at 10:26 AM, wrote: > Hi! This is the ezmlm program. I'm managing the > user@flink.apache.org mailing list. > > Acknowledgment: I have added the addres