Re: KafkaSystemProducer partitioning

2015-12-21 Thread Yi Pan
Hi, Michal, Yeah, unfortunately, based on the current code base, the only short term solution would be your proposal.The issue is the following: 1. Kafka producer only applies the specified partition.class to the ProducerRecord key. Samza allows the user to specify a partition key that is differen

Re: KafkaSystemProducer partitioning

2015-12-18 Thread Michal Haris
Ah, that's unfortunate. Basically we have an existing stream-processing pipeline which relies on different partitioning schemes and we are writing some upstream Samza jobs. The only way to get it write in a particular partitioning scheme is then to write a different KafkaSystemFactory right ? Or pe

Re: KafkaSystemProducer partitioning

2015-12-17 Thread Yi Pan
Hi, Michal, Sorry to reply late. Actually, you are right that the "partition.class" configuration is not used in Samza to determine the outgoing partition. In Samza, partition is defined by the following code sections: {code} val topicName = envelope.getSystemStream.getStream val partitions: java