Advice - Drools in Flink

Hello Firstly, I am an absolute Flink newbie. I am interested in using Drools in Flink - in a similar case to what is described in this blog, where Drools is used in Spark. http://blog.cloudera.com/blog/2015/11/how-to-build-a-complex-event-processing-app-on-apache-spark-and-drools/ The basic ide

Re: kafka partition assignment

Hi Michal, I see, thanks for the description. I think you’ve definitely raised an interesting point. Yes, there may be unnecessary shuffle in this case (the partitionCustom on consumed DataStream doesn’t override the assignPartitions in the Kafka connector; custom partitioners are applied after th

[jira] [Created] (FLINK-4096) Nested FileOutputStream is not closed in JarFileCreator

Ted Yu created FLINK-4096: - Summary: Nested FileOutputStream is not closed in JarFileCreator Key: FLINK-4096 URL: https://issues.apache.org/jira/browse/FLINK-4096 Project: Flink Issue Type: Bug

Re: kafka partition assignment

Hi Tai, I was referring to co-partitioning, not co-location of leaders, i.e. multiple topics that share the same partitioning scheme. By example, say I have 2 topics which share the same keyspace and which are produced by something other than Flink using identical partitioner. The data in these 2 t

Shading Plugin + Hadoop Dependency

Hello, I was trying to include "hadoop-minikdc" component to Yarn test framework by adding the dependency in "flink-shaded-include-yarn-tests" pom.xml file. org.apache.hadoop hadoop-minikdc ${hadoop.version} The dependency inclusion seems to be working from IDE. IntelliJ picked up the

Re: offheap memory allocation and memory leak bug

That would be great. Best you directly post the link to the JIRA issue. Cheers, Till On Mon, Jun 20, 2016 at 12:55 PM, CPC wrote: > Hi Till, > > I saw jira issue. Do you want me to upload input dataset as well? If you > want i can prepare a github repo if it would be more easier. > On Jun 20, 2

[jira] [Created] (FLINK-4095) Add configDir argument to shell scripts

Ufuk Celebi created FLINK-4095: -- Summary: Add configDir argument to shell scripts Key: FLINK-4095 URL: https://issues.apache.org/jira/browse/FLINK-4095 Project: Flink Issue Type: Improvement

Re: offheap memory allocation and memory leak bug

Hi Till, I saw jira issue. Do you want me to upload input dataset as well? If you want i can prepare a github repo if it would be more easier. On Jun 20, 2016 1:10 PM, "Till Rohrmann" wrote: > Hi, > > your observation sounds like a bug to me and we have to further investigate > it. I assume that

[jira] [Created] (FLINK-4094) Off heap memory deallocation might not work properly

Till Rohrmann created FLINK-4094: Summary: Off heap memory deallocation might not work properly Key: FLINK-4094 URL: https://issues.apache.org/jira/browse/FLINK-4094 Project: Flink Issue Type

Re: offheap memory allocation and memory leak bug

Hi, your observation sounds like a bug to me and we have to further investigate it. I assume that you’re running a batch job, right? Could you maybe share your complete configuration and the job to reproduce the problem with us? I think that your investigation that direct buffers are not properly

Re: kafka partition assignment

Hi Michal, Whether or not the external system's partitioning scheme is referenced when assigning partitions to the consumer parallel subtasks depends on the implementation of each connector / source. First, clarification on “co-partitioning": from your context I’m assuming you’re referring to co-

[jira] [Created] (FLINK-4093) Expose metric interfaces

Chesnay Schepler created FLINK-4093: --- Summary: Expose metric interfaces Key: FLINK-4093 URL: https://issues.apache.org/jira/browse/FLINK-4093 Project: Flink Issue Type: Improvement