Question about the process order in stream aggregate

2017-04-10 Thread Xingcan Cui
Hi all, I run some tests for stream aggregation on rows. The data stream is simply registered as val orderA: DataStream[Order] = env.fromCollection(Seq( Order(1L, "beer", 1), Order(2L, "diaper", 2), Order(3L, "diaper", 3), Order(4L, "rubber", 4))) tEnv.registerDataStream("

Re: [VOTE] Release Apache Flink 1.2.1 (RC1)

2017-04-10 Thread Robert Metzger
I've now started building the next release candidate. On Sun, Apr 9, 2017 at 12:37 PM, Robert Metzger wrote: > Hi Gyula, > > I'm trying to push Stefan R. to get the RocksDB fixes in asap. > > On Sat, Apr 8, 2017 at 5:17 PM, Gyula Fóra wrote: > >> Hi All, >> >> Any updates on this? >> >> It woul

[jira] [Created] (FLINK-6290) SharedBuffer is improperly released when multiple edges between entries

2017-04-10 Thread Dawid Wysakowicz (JIRA)
Dawid Wysakowicz created FLINK-6290: --- Summary: SharedBuffer is improperly released when multiple edges between entries Key: FLINK-6290 URL: https://issues.apache.org/jira/browse/FLINK-6290 Project:

Re: Possible bug in Kafka producer partitioning logic

2017-04-10 Thread Gyula Fóra
In the worst case scenario we will have a custom build that will just cache the different partition numbers in a map. (But still call partitioner.open only once) I think this simple intermediate fix would actually be good enough for most people who get blocked by this in the short run. Gyula Gyul

[jira] [Created] (FLINK-6289) ExecutionEnvironment.readTextFile() can read gzip files & directories but not both

2017-04-10 Thread Arnaud Linz (JIRA)
Arnaud Linz created FLINK-6289: -- Summary: ExecutionEnvironment.readTextFile() can read gzip files & directories but not both Key: FLINK-6289 URL: https://issues.apache.org/jira/browse/FLINK-6289 Project:

Re: Possible bug in Kafka producer partitioning logic

2017-04-10 Thread Gyula Fóra
I understand the reasoning, on the other hand this creates a problem that is very hard to work around. :/ Do you have any suggestions how to get around this? Gyula Tzu-Li (Gordon) Tai ezt írta (időpont: 2017. ápr. 10., H, 15:57): > I would prefer to make this a blocker for a future bugfix actu

Re: Possible bug in Kafka producer partitioning logic

2017-04-10 Thread Tzu-Li (Gordon) Tai
I would prefer to make this a blocker for a future bugfix actually, and not 1.2.1. The reason is that to fix this properly we might need to look again into (and possibly change) how partitioners are provided. The main problem is that the `open` method can only possibly be called once with the p

Re: Possible bug in Kafka producer partitioning logic

2017-04-10 Thread Gyula Fóra
Thanks for checking this out. I would say this is definitely a blocking issue for the bugfix release, what do you think? Gyula Tzu-Li (Gordon) Tai ezt írta (időpont: 2017. ápr. 10., H, 15:39): Hi Gyula, Yes, I think the semantics of the Partitioner interface is a bit off. The `numPartitions`

Re: Possible bug in Kafka producer partitioning logic

2017-04-10 Thread Tzu-Li (Gordon) Tai
Hi Gyula, Yes, I think the semantics of the Partitioner interface is a bit off. The `numPartitions` value ideally should be the number of partitions of the `targetTopic`. Here’s a JIRA I just filed to track the issue:  https://issues.apache.org/jira/browse/FLINK-6288. Cheers, Gordon On April 1

[jira] [Created] (FLINK-6288) FlinkKafkaProducer's custom Partitioner is always invoked with number of partitions of default topic

2017-04-10 Thread Tzu-Li (Gordon) Tai (JIRA)
Tzu-Li (Gordon) Tai created FLINK-6288: -- Summary: FlinkKafkaProducer's custom Partitioner is always invoked with number of partitions of default topic Key: FLINK-6288 URL: https://issues.apache.org/jira/brows

[jira] [Created] (FLINK-6287) Flakey JobManagerRegistrationTest

2017-04-10 Thread Nico Kruber (JIRA)
Nico Kruber created FLINK-6287: -- Summary: Flakey JobManagerRegistrationTest Key: FLINK-6287 URL: https://issues.apache.org/jira/browse/FLINK-6287 Project: Flink Issue Type: Bug Compone

[jira] [Created] (FLINK-6286) hbase command not found error

2017-04-10 Thread Jinjiang Ling (JIRA)
Jinjiang Ling created FLINK-6286: Summary: hbase command not found error Key: FLINK-6286 URL: https://issues.apache.org/jira/browse/FLINK-6286 Project: Flink Issue Type: Bug Repor

Possible bug in Kafka producer partitioning logic

2017-04-10 Thread Gyula Fóra
Hi all, We had some problems with custom partitioning for the 0.8 Kafka producer and now that I checked the code it seems there might be a problem with the logic. The producer determines the number of partitions in the open method and seems to be using that as a value passed to the custom partiti