look for help about task hooks of storm-compatibility

2015-09-08 Thread Fangfengbin
Hi @mjsax @StephanEwen @rmetzger Can you give me some suggestion about my idea of how to transfer user-defined class code to task? Thank you very much! Regards Fengbin Fang # Dear all, I am work on task hooks of storm-compatibility. Storm support add hook through the Storm

Re: Releasing 0.10.0-milestone1

2015-09-08 Thread Stephan Ewen
Great! I'd like to push one more commit later today. A fix for https://issues.apache.org/jira/browse/FLINK-2632 would also be highly appreciated by some users. Anyone volunteering as release manager (for creating release candidates and uploading them)? On Tue, Sep 8, 2015 at 6:11 PM, Kostas Tzo

[jira] [Created] (FLINK-2641) Integrate the off-heap memory configuration with the TaskManager start script

2015-09-08 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-2641: --- Summary: Integrate the off-heap memory configuration with the TaskManager start script Key: FLINK-2641 URL: https://issues.apache.org/jira/browse/FLINK-2641 Project: Fl

[jira] [Created] (FLINK-2640) Integrate the off-heap configurations with YARN runner

2015-09-08 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-2640: --- Summary: Integrate the off-heap configurations with YARN runner Key: FLINK-2640 URL: https://issues.apache.org/jira/browse/FLINK-2640 Project: Flink Issue Type

[jira] [Created] (FLINK-2639) Building Flink for specific HDP versions fails

2015-09-08 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-2639: - Summary: Building Flink for specific HDP versions fails Key: FLINK-2639 URL: https://issues.apache.org/jira/browse/FLINK-2639 Project: Flink Issue Type: Bu

Re: Releasing 0.10.0-milestone1

2015-09-08 Thread Kostas Tzoumas
+1 for a milestone release On Tue, Sep 8, 2015 at 5:43 PM, Robert Metzger wrote: > +1 for a "milestone1" release. We have a lot of good features in master > that people can benefit from. > > On Tue, Sep 8, 2015 at 5:10 PM, Maximilian Michels wrote: > > > +1 for releasing a milestone release soo

Re: Releasing 0.10.0-milestone1

2015-09-08 Thread Fabian Hueske
+1 for a milestone 0.10 release. The TypeInformation issues (FLINK-2633, FLINK-2637) might be worth to be fixed in the release. 2015-09-08 17:10 GMT+02:00 Maximilian Michels : > +1 for releasing a milestone release soon to encourage people to try > out the new features. > > There is this bug: htt

Re: Releasing 0.10.0-milestone1

2015-09-08 Thread Robert Metzger
+1 for a "milestone1" release. We have a lot of good features in master that people can benefit from. On Tue, Sep 8, 2015 at 5:10 PM, Maximilian Michels wrote: > +1 for releasing a milestone release soon to encourage people to try > out the new features. > > There is this bug: https://issues.apa

Re: Releasing 0.10.0-milestone1

2015-09-08 Thread Stephan Ewen
Okay, can the authors of the respective fixes make sure they get merged into both master and the milestone branch? On Tue, Sep 8, 2015 at 5:38 PM, Fabian Hueske wrote: > +1 for a milestone 0.10 release. > The TypeInformation issues (FLINK-2633, FLINK-2637) might be worth to be > fixed in the rel

[jira] [Created] (FLINK-2638) Add @SafeVarargs to the environment.fromElements(...) method

2015-09-08 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-2638: --- Summary: Add @SafeVarargs to the environment.fromElements(...) method Key: FLINK-2638 URL: https://issues.apache.org/jira/browse/FLINK-2638 Project: Flink Iss

[jira] [Created] (FLINK-2637) Add abstract equals, hashCode and toString methods to TypeInformation

2015-09-08 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-2637: Summary: Add abstract equals, hashCode and toString methods to TypeInformation Key: FLINK-2637 URL: https://issues.apache.org/jira/browse/FLINK-2637 Project: Flink

Re: Releasing 0.10.0-milestone1

2015-09-08 Thread Maximilian Michels
+1 for releasing a milestone release soon to encourage people to try out the new features. There is this bug: https://issues.apache.org/jira/browse/FLINK-2632 which affects the Web Client's error and results display for jobs. Would be nice to fix it but IMHO it is not critical for the milestone re

Re: Streaming KV store abstraction

2015-09-08 Thread Aljoscha Krettek
That's a very nice application of the Stream API and partitioned state. :D I think we should run some tests on a cluster based on this to see what kind of throughput the partitioned state system can handle and also how it behaves with larger numbers of keys. The KVStore is just an interface and t

答复: Some questions about storm metrics

2015-09-08 Thread huangwei (G)
@Matthias Thanks for advice :-) == I personally never used Metrics in Storm. It would be best, if you design an own example. I guess you are familiar with https://storm.apache.org/documentation/Metrics.html As a first step, I would try to include the predefi

Re: Streaming KV store abstraction

2015-09-08 Thread Gyula Fóra
@Stephan: Technically speaking this is really just a partitioned key-value state and a fancy operator executing special operations on this state. >From the user's perspective though this is something hard to implement. If you want to share state between two stream for instance this way (getting u

Re: Streaming KV store abstraction

2015-09-08 Thread Stephan Ewen
@Gyula Can you explain a bit what this KeyValue store would do more then the partitioned key/value state? On Tue, Sep 8, 2015 at 2:49 PM, Gábor Gévay wrote: > Hello, > > As for use cases, in my old job at Ericsson we were building a > streaming system that was processing data from telephone net

Re: Streaming KV store abstraction

2015-09-08 Thread Gábor Gévay
Hello, As for use cases, in my old job at Ericsson we were building a streaming system that was processing data from telephone networks, and it was using key-value stores a LOT. For example, keeping track of various state info of the users (which cell are they currently connected to, what bearers

[jira] [Created] (FLINK-2636) Clean up StreamRecord and Watermark types

2015-09-08 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-2636: --- Summary: Clean up StreamRecord and Watermark types Key: FLINK-2636 URL: https://issues.apache.org/jira/browse/FLINK-2636 Project: Flink Issue Type: Improvement

[jira] [Created] (FLINK-2635) Unify StreamInputProcessor and OneInputStreamTask

2015-09-08 Thread Stephan Ewen (JIRA)
Stephan Ewen created FLINK-2635: --- Summary: Unify StreamInputProcessor and OneInputStreamTask Key: FLINK-2635 URL: https://issues.apache.org/jira/browse/FLINK-2635 Project: Flink Issue Type: Imp

Streaming KV store abstraction

2015-09-08 Thread Gyula Fóra
Hey All, The last couple of days I have been playing around with the idea of building a streaming key-value store abstraction using stateful streaming operators that can be used within Flink Streaming programs seamlessly. Operations executed on this KV store would be fault tolerant as it integrat

[jira] [Created] (FLINK-2634) Add a Vertex-centric Version of the Tringle Count Library Method

2015-09-08 Thread Andra Lungu (JIRA)
Andra Lungu created FLINK-2634: -- Summary: Add a Vertex-centric Version of the Tringle Count Library Method Key: FLINK-2634 URL: https://issues.apache.org/jira/browse/FLINK-2634 Project: Flink I

Re: Releasing 0.10.0-milestone1

2015-09-08 Thread Ufuk Celebi
> On 08 Sep 2015, at 12:01, Stephan Ewen wrote: > > Hi all! > > Some day back we talked about releasing an 0.10.0-milestone1 release. The > master has advanced quite a bit (especially due to high-availability code). > > I cherry picked the important additions to the release-0.10.0-milestone1 >

Releasing 0.10.0-milestone1

2015-09-08 Thread Stephan Ewen
Hi all! Some day back we talked about releasing an 0.10.0-milestone1 release. The master has advanced quite a bit (especially due to high-availability code). I cherry picked the important additions to the release-0.10.0-milestone1 branch (fixes and Kafka consumer/producer rework). How about rele

look for help about task hooks of storm-compatibility

2015-09-08 Thread Fangfengbin
Dear all, I am work on task hooks of storm-compatibility. Storm support add hook through the Storm configuration using the "topology.auto.task.hooks" config. Users can use user-defined hooks class names as value of "topology.auto.task.hooks" of Configuration Map. These hooks are automatically r