RdKafka & setting offsets

2016-12-07 Thread Doyle, Keith
I'm attempting to set the offset for a RdKafka-dotnet consumer in order to re-read the topic, but I've not seen any documentation or examples that do this. I saw a reference for librdkafka that seems to show a set_offset method off the TopicPartition class and an Offset property on the TopicP

Some general questions...

2016-12-05 Thread Doyle, Keith
We're beginning to make use of Kafka, and it is encouraging. But there are a couple of questions I've had a hard time finding answers for. We're using the rdkafka-dotnet client on the consumer side and it's straightforward as far as it goes. However, documentation seems to be scant-the Wiki

Topics vs Partitions

2016-11-16 Thread Doyle, Keith
Trying to wrap my head around some of the concepts... For semantic partitioning, is there any rationale for choosing to divide a = single topic into multiple partitions as opposed to just creating multiple = topics each with a single partition? -- [Greenway_Health_PNG_large_NO_tag] Keith

Co-locating brokers

2016-08-11 Thread Doyle, Keith
Hi, We're considering Kafka to provide a queued transport mechanism for an ETL process with near-real-time capability. Kafka is looking pretty good but I'm wondering about a couple of things. It's not clear- my first inclination is to co-locate a broker on the client servers in order to pro