Consumer position not returning correct offset - 0.10.0.1

2018-09-05 Thread Henry Thacker
Hi, In testing - i’ve got a 5 node Kafka cluster with min.insync.replicas set to 4. The cluster is currently running version 0.10.0.1. There’s an application that publishes to a topic - and on restart, it attempts to read the full contents of the topic up until the high watermark before then

Re: Kafka on Windows

2018-09-05 Thread jan
Hi Liam, as a DB guy that does MSSQL (on windows, obviously) I literally have no idea what a .sh file is,or what that would imply. I guess it's bash so yeah. But what's so terrifying about actually stating in the FAQ that you should not run it as production on windows? Why should I conclude that k

Re: Kafka on Windows

2018-09-05 Thread M. Manna
I believe this needs to be articulated a bit more, which is not a big issue. The common practice for a large-scale distributed system is to use Linux servers, and it's always been a popular choice. That doesn't mean the Windows is ignored. The issue at hand is that file deletion/renaming seems to h

Re: Kafka on Windows

2018-09-05 Thread Liam Clarke
Hi Jan, .sh is not .bat and easily googleable. Nothing is terrifying about updating the documentation, I just took umbrage at you "resenting" your 3 days you expended on learning the hard way that Windows is the minority market share in servers - you're resenting a free open source project? Becau

Re: Kafka on Windows

2018-09-05 Thread jan
@M.Manna - thanks for your reply. I did read that document but clearly didn't digest that bit properly. @Liam Clarke I don't expect code's users (either maintainers or end-users) to need to deduce *anything*. If it needs knowing, I documented it. I'll send you an actual example so you know it's n

Re: Controlled topics creation in Kafka cluster, how does that affect a Kafka Streams App that uses join?

2018-09-05 Thread John Roesler
Hi Meeiling, It sounds like "formal process" is more like "fill out a form beforehand" than just "access controlled". In that case, although it's somewhat of an internal detail, I think the easiest thing would be to figure out what the name of the internal topics will be and request their creatio

Re: Best way to manage topics using Java API

2018-09-05 Thread John Roesler
Hi Robin, The AdminClient is what you want. "Evolving" is just a heads-up that the API is relatively new and hasn't stood the test of time, so you shouldn't be *too* surprised to see it change in the future. That said, even APIs marked "Evolving" need to go through the KIP process to be changed,

Re: Controlled topics creation in Kafka cluster, how does that affect a Kafka Streams App that uses join?

2018-09-05 Thread Matthias J. Sax
You can figure out the names via builder.build().describe() The naming pattern is --[repartition|changelog] You will also need to create those topic with the correct number of partitions (usually the number of input topic partitions). With this information, you can pre-create the topics and Kaf

Re: GlobalKTable/KTable initialization differences

2018-09-05 Thread Matthias J. Sax
I create https://issues.apache.org/jira/browse/KAFKA-7380 to track this. -Matthias On 8/27/18 12:07 PM, Guozhang Wang wrote: > Hello Patrik, > > Thanks for the email and this is indeed a good question! :) > > There are some historic reasons that we did the global state restoration in > a differ

Re: Kafka stream only consume n messages

2018-09-05 Thread Matthias J. Sax
1. There is not API for this 2. I guess it might be possible, but might not be the best way to do it. 3. That is also not possible. I would recommend something like this: > final AtomicBoolean shutdown = new AtomicBoolean(false); > > StreamsBuilder builder = ... > > KStream stream = builder