I have some unit tests in which I create an embedded single broker kafka
cluster, using :
EmbeddedSingleNodeKafkaCluster.java from
https://github.com/confluentinc/examples/blob/master/kafka-streams/src/test/java/io/confluent/examples/streams/kafka/EmbeddedSingleNodeKafkaCluster.java
That class al
Hi list,
I've added a kafka node on our Hortonworks cluster and while executing the
reassign partition procedure something went wrong and I'm kind of stuck.
ZK nodes: RHTPINEC001, RHTPINEC004, RHTPINEC005
Kafka nodes: RHTPINEC008 (broker id = 1001), RHTPINEC007 (broker id = 1002,
this is the new
Hi Ali,
You're right, shutting down the broker and ZK is expensive. We kept the number
of integration tests relatively small (and pushed some more tests as system
tests, while doing as many as possible as unit tests). It's not just the
shutdown that's expensive, it's also the starting up unfort
Hey Eno,
Thanks for the quick reply.
In the meantime, is it possible to just send a sigterm / kill -9 which just
kills the zookeeper + kafka? I can figure out how to do it if you can point
out which class / method creates the processes / threads.
Thanks.
On Fri, Nov 11, 2016 at 9:24 PM, Eno The
Unless I'm missing anything, there's no reason why these throwaway
processes should be shutdown gracefully. Just kill them as soon as the test
finishes.
On Fri, Nov 11, 2016 at 9:26 PM, Ali Akhtar wrote:
> Hey Eno,
>
> Thanks for the quick reply.
>
> In the meantime, is it possible to just send
For me, the startup doesn't take anywhere near as long as shutdown does.
On Fri, Nov 11, 2016 at 9:37 PM, Ali Akhtar wrote:
> Unless I'm missing anything, there's no reason why these throwaway
> processes should be shutdown gracefully. Just kill them as soon as the test
> finishes.
>
> On Fri, N
It's the org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster (start
ZK and calls org.apache.kafka.streams.integration.utils.KafkaEmbedded to start
Kafka).
So these are embedded in the sense that it's not another process, just threads
within the main streams test process.
Thanks
Eno
Oh, so it seems like there's no easy way to just Thread.stop() without
changing the internal kafka / zk code? :(
Perhaps its possible to start kafka / zk within another thread, and then
kill the wrapper thread. Will that stop the children threads, if the
wrapper thread is killed?
Hmm, or may be a
Eno,
I tried the following, but i haven't seen any improvement. In fact it seems
to run longer. Are you sure Kafka / ZK are run as threads and not as
process:
Set threadSet = Thread.getAllStackTraces().keySet();
threadSet.forEach(t ->
{
if (t == Thread.current
Hi,
I am not an expert but from what read and understood from HDFS
documentation, If you want to upgrade zookeeper, you can not avoid
downtime.
Thank you,
Amit
On Thursday, November 10, 2016, ZHU Hua B
wrote:
> Hi,
>
>
> For a rolling upgrade, Kafka suggest upgrade the brokers one at a time
>
Becket/Jason,
We deployed a jar with the base 0.10.1.0 release plus the KAFKA-3994 patch,
but we're seeing the same exact issue. It doesnt' seem like the patch
fixes the problem we're seeing.
At this point, we're considering downgrading our prod clusters back to
0.10.0.1. Is there any concern/i
Hi Marcos,
Thanks for the update. It looks the deadlock you saw was another one. Do
you mind sending us a full stack trace after this happens?
Regarding the downgrade, the steps would be the following:
1. change the inter.broker.protocol to 0.10.0
2. rolling bounce the cluster
3. deploy the 0.10.
Thanks Becket,
We should get a full thread dump the next time, so I'll send it as soon
that happens.
Marcos
On Fri, Nov 11, 2016 at 11:27 AM, Becket Qin wrote:
> Hi Marcos,
>
> Thanks for the update. It looks the deadlock you saw was another one. Do
> you mind sending us a full stack trace aft
Thanks again for the replies, and the time to suggest alternate solution. I
know your time is valuable.
The reason for the List in the aggregate, which I missed trying to keep it
simple, is that the order of records added to topic is not guaranteed to be
chronological. Keeping the list allow
14 matches
Mail list logo
