Re: Tests for the Steaming classes

2015-03-23 Thread Stephan Ewen
I am currently travelling, I'll open the JIRAs as soon as I find a bit of time to sit down... On Mon, Mar 23, 2015 at 5:20 PM, Henry Saputra wrote: > +1 for more tests and plus more documentation both internal and javadoc. > Some stuff like memory usafe and data flows are bit different from batc

Re: [DISCUSS] Distributed TPC-H DataGenerator for flink-contrib

2015-03-23 Thread Fabian Hueske
+1 On Mar 23, 2015 7:39 PM, "Robert Metzger" wrote: > Great, thanks for taking some time looking at this! > > If nobody objects in the next 48 hours, I'll open a PR for the TPC-H data > generator. > > On Mon, Mar 23, 2015 at 7:34 PM, Henry Saputra > wrote: > > > Yeah, I believe it should be ok s

Re: [DISCUSS] Distributed TPC-H DataGenerator for flink-contrib

2015-03-23 Thread Robert Metzger
Great, thanks for taking some time looking at this! If nobody objects in the next 48 hours, I'll open a PR for the TPC-H data generator. On Mon, Mar 23, 2015 at 7:34 PM, Henry Saputra wrote: > Yeah, I believe it should be ok since we do not actually package any > code bits from TPC-H. > > I thi

Re: [DISCUSS] Distributed TPC-H DataGenerator for flink-contrib

2015-03-23 Thread Henry Saputra
Yeah, I believe it should be ok since we do not actually package any code bits from TPC-H. I think giving trademark nudge to TPC-H in our NOTICE file should be good. - Henry On Mon, Mar 23, 2015 at 11:23 AM, Robert Metzger wrote: > I've send a message to admin-i...@tpc.org and never got an answ

Re: [DISCUSS] Distributed TPC-H DataGenerator for flink-contrib

2015-03-23 Thread Robert Metzger
I've send a message to admin-i...@tpc.org and never got an answer. (on http://www.tpc.org/trademarks/ they list ad...@tpc.org as the right address, but sending a message to admin@ redirects to admin-info@.) My code doesn't contain any TPC data or code. Its a Java re-implementation of the C data ge

Re: [DISCUSS] Distributed TPC-H DataGenerator for flink-contrib

2015-03-23 Thread Henry Saputra
Robert, Just curious if you did try to send email to tpc.org to ask about fair usage of example data? - Henry On Sat, Feb 28, 2015 at 12:07 PM, Robert Metzger wrote: > I tried twice writing them but I didn't receive an answer. > But given that Apache Calcite is also using airlift/tpch in its >

[jira] [Created] (FLINK-1776) APIs provide invalid Semantic Properties for Operators with SelectorFunction keys

2015-03-23 Thread Fabian Hueske (JIRA)
Fabian Hueske created FLINK-1776: Summary: APIs provide invalid Semantic Properties for Operators with SelectorFunction keys Key: FLINK-1776 URL: https://issues.apache.org/jira/browse/FLINK-1776 Proje

[jira] [Created] (FLINK-1775) BarrierBuffers don't correctly handle end of stream events

2015-03-23 Thread Gyula Fora (JIRA)
Gyula Fora created FLINK-1775: - Summary: BarrierBuffers don't correctly handle end of stream events Key: FLINK-1775 URL: https://issues.apache.org/jira/browse/FLINK-1775 Project: Flink Issue Type

Re: Compilation error after guava shading

2015-03-23 Thread Vasiliki Kalavri
Awesome, thanks Robert! On Mar 23, 2015 5:15 PM, "Robert Metzger" wrote: > Hi, > > you have to add guava as a dependency to your project. > With the shading, users won't see Flink depending on Guava anymore. This > allows them (users) to use any guava version they want. > > > > On Mon, Mar 23, 20

[jira] [Created] (FLINK-1774) Remove the redundant code in try{} block

2015-03-23 Thread Sibao Hong (JIRA)
Sibao Hong created FLINK-1774: - Summary: Remove the redundant code in try{} block Key: FLINK-1774 URL: https://issues.apache.org/jira/browse/FLINK-1774 Project: Flink Issue Type: Improvement

Re: Tests for the Steaming classes

2015-03-23 Thread Henry Saputra
+1 for more tests and plus more documentation both internal and javadoc. Some stuff like memory usafe and data flows are bit different from batch which have not been documented much. Looks like more will be coming to Flink to check out streaming solution in Flink as alternative to others. - Henry

[jira] [Created] (FLINK-1773) Add interface for grouped policies to broadcast information to other groups

2015-03-23 Thread Gyula Fora (JIRA)
Gyula Fora created FLINK-1773: - Summary: Add interface for grouped policies to broadcast information to other groups Key: FLINK-1773 URL: https://issues.apache.org/jira/browse/FLINK-1773 Project: Flink

Re: Compilation error after guava shading

2015-03-23 Thread Robert Metzger
Hi, you have to add guava as a dependency to your project. With the shading, users won't see Flink depending on Guava anymore. This allows them (users) to use any guava version they want. On Mon, Mar 23, 2015 at 5:11 PM, Vasiliki Kalavri wrote: > Hi squirrels, > > I have a maven project that

Compilation error after guava shading

2015-03-23 Thread Vasiliki Kalavri
Hi squirrels, I have a maven project that only depends on flink-gelly, but after the guava shading change I get a compilation error in one of the classes that is using the primitives library: "package com.google.common does not exist". Is this to be expected? Do I need to add anything to my pom?

[jira] [Created] (FLINK-1772) Chain of split/select does not work

2015-03-23 Thread JIRA
Gábor Hermann created FLINK-1772: Summary: Chain of split/select does not work Key: FLINK-1772 URL: https://issues.apache.org/jira/browse/FLINK-1772 Project: Flink Issue Type: Bug C

[jira] [Created] (FLINK-1771) Add tests for setting the processing slots for the YARN client

2015-03-23 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-1771: - Summary: Add tests for setting the processing slots for the YARN client Key: FLINK-1771 URL: https://issues.apache.org/jira/browse/FLINK-1771 Project: Flink

[jira] [Created] (FLINK-1770) Rename the variable 'contentAdressable' to 'contentAddressable'

2015-03-23 Thread Sibao Hong (JIRA)
Sibao Hong created FLINK-1770: - Summary: Rename the variable 'contentAdressable' to 'contentAddressable' Key: FLINK-1770 URL: https://issues.apache.org/jira/browse/FLINK-1770 Project: Flink Issu

Re: Improve the documentation of the Flink Architecture and internals

2015-03-23 Thread Ufuk Celebi
I couldn't have a look at it earlier, because the Wiki was down. Very nice overview of the flow of things. I like the text and pictures a lot. I will add content about: 1) The way that we do the network transfers with Netty 2) A more detailed message flow for pipelined vs. blocking results. I

Re: Tests for the Steaming classes

2015-03-23 Thread Gyula Fóra
+1 for lots of streaming tests On Mon, Mar 23, 2015 at 11:23 AM, Márton Balassi wrote: > Thanks for looking into this, Stephan. +1 for the JIRAs. > > On Mon, Mar 23, 2015 at 10:55 AM, Ufuk Celebi wrote: > > > On 23 Mar 2015, at 10:44, Stephan Ewen wrote: > > > > > Hi everyone! > > > > > > With

Re: Tests for the Steaming classes

2015-03-23 Thread Márton Balassi
Thanks for looking into this, Stephan. +1 for the JIRAs. On Mon, Mar 23, 2015 at 10:55 AM, Ufuk Celebi wrote: > On 23 Mar 2015, at 10:44, Stephan Ewen wrote: > > > Hi everyone! > > > > With the streaming stuff getting heavier exposure, I think it needs a few > > more tests. With so many changes

Re: Improve the documentation of the Flink Architecture and internals

2015-03-23 Thread Maximilian Michels
Very insightful post. Thank you! On Sat, Mar 21, 2015 at 5:30 PM, Till Rohrmann wrote: > I wrote some internal documentation for Akka and the distributed > communication [1]. > > Cheers, > > Till > > [1] https://cwiki.apache.org/confluence/display/FLINK/Akka+and+Actors > > On Fri, Mar 20, 2015 a

[jira] [Created] (FLINK-1769) Maven deploy is broken (build artifacts are cleaned in docs-and-sources profile)

2015-03-23 Thread Robert Metzger (JIRA)
Robert Metzger created FLINK-1769: - Summary: Maven deploy is broken (build artifacts are cleaned in docs-and-sources profile) Key: FLINK-1769 URL: https://issues.apache.org/jira/browse/FLINK-1769 Proj

Re: Tests for the Steaming classes

2015-03-23 Thread Ufuk Celebi
On 23 Mar 2015, at 10:44, Stephan Ewen wrote: > Hi everyone! > > With the streaming stuff getting heavier exposure, I think it needs a few > more tests. With so many changes, untested features are running a high risk > of being "patched away" by accident. > > For the runtime and batch API part,

Tests for the Steaming classes

2015-03-23 Thread Stephan Ewen
Hi everyone! With the streaming stuff getting heavier exposure, I think it needs a few more tests. With so many changes, untested features are running a high risk of being "patched away" by accident. For the runtime and batch API part, we go with the policy that every new feature can only be merg

Re: Queries regarding RDFs with Flink

2015-03-23 Thread Flavio Pompermaier
Thanks Vasiliki, when I'll find the time I'll try to make a quick prototype using the pointers you suggested! Thanks for the support, Flavio On Mon, Mar 23, 2015 at 10:31 AM, Vasiliki Kalavri < vasilikikala...@gmail.com> wrote: > Hi Flavio, > > I'm not familiar with JSON-LD, but as far as I unde

Re: Queries regarding RDFs with Flink

2015-03-23 Thread Vasiliki Kalavri
Hi Flavio, I'm not familiar with JSON-LD, but as far as I understand, you want to generate some trees from selected root nodes. Once you have created the Graph as Andra describes above, you can first filter out the edges that are of no interest to you, using filterOnEdges. There is a description