Re: [Streams] Threading Frustration

2016-12-12 Thread Avi Flax
> On Dec 12, 2016, at 12:29, Damian Guy wrote: > > Yep - that looks correct Fantastic! Thanks again! Software Architect @ Park Assist We’re hiring! http://tech.parkassist.com/jobs/

Re: [Streams] Threading Frustration

2016-12-12 Thread Damian Guy
It is the same. The ruby code in this example is just calling into the Java API. On Mon, 12 Dec 2016 at 17:40 Ali Akhtar wrote: > @Damian, > > In the Java equivalent of this, does each KStream / KStreamBuilder.stream() > invocation create its own topic group, i.e its own thread? > > On Mon, Dec

Re: [Streams] Threading Frustration

2016-12-12 Thread Ali Akhtar
@Damian, In the Java equivalent of this, does each KStream / KStreamBuilder.stream() invocation create its own topic group, i.e its own thread? On Mon, Dec 12, 2016 at 10:29 PM, Damian Guy wrote: > Yep - that looks correct > > On Mon, 12 Dec 2016 at 17:18 Avi Flax wrote: > > > > > > On Dec 12,

Re: [Streams] Threading Frustration

2016-12-12 Thread Damian Guy
Yep - that looks correct On Mon, 12 Dec 2016 at 17:18 Avi Flax wrote: > > > On Dec 12, 2016, at 11:42, Damian Guy wrote: > > > > If you want to split these out so that they can run in parallel, then you > > will need to create a new stream for each topic. > > > Just to make sure I’m understandi

Re: [Streams] Threading Frustration

2016-12-12 Thread Avi Flax
> On Dec 12, 2016, at 11:42, Damian Guy wrote: > > If you want to split these out so that they can run in parallel, then you > will need to create a new stream for each topic. Just to make sure I’m understanding this, does this code change look right? From this: ```ruby topic_names = SDP::Co

Re: [Streams] Threading Frustration

2016-12-12 Thread Avi Flax
> On Dec 12, 2016, at 11:42, Damian Guy wrote: > > If you want to split these out so that they can run in parallel, then you > will need to create a new stream for each topic. Super, super helpful — thanks so much! Avi

Re: [Streams] Threading Frustration

2016-12-12 Thread Damian Guy
Hi Avi, Thanks for sharing your code. I believe the reason you are only seeing 20 threads used is due to a couple of things: KafkaStreams uses a concept of topic groups to group related topics into StreamTasks. A StreamTask is a task an independent part of the topology that can be executed on its

Re: [Streams] Threading Frustration

2016-12-12 Thread Avi Flax
> On Dec 12, 2016, at 10:24, Damian Guy wrote: > > The code for your Streams Application. Doesn't have to be the actual code, > but an example of how you are using Kafka Streams. OK, I’ve prepared a gist with (I hope) the relevant code, and also some log records just in case they might help:

Re: [Streams] Threading Frustration

2016-12-12 Thread Damian Guy
Hi Avi, The code for your Streams Application. Doesn't have to be the actual code, but an example of how you are using Kafka Streams. On Mon, 12 Dec 2016 at 15:13 Avi Flax wrote: > > > On Dec 12, 2016, at 10:08, Damian Guy wrote: > > > > Can you provide an example of your topology? > > I’d be

Re: [Streams] Threading Frustration

2016-12-12 Thread Avi Flax
> On Dec 12, 2016, at 10:08, Damian Guy wrote: > > Can you provide an example of your topology? I’d be happy to, but I’m not sure what you mean? Are you interested in the implementation at the code level? You want a better understanding of the work being done? The infrastructure?

Re: [Streams] Threading Frustration

2016-12-12 Thread Damian Guy
Hi Avi, Can you provide an example of your topology? Thanks, Damian On Mon, 12 Dec 2016 at 15:02 Avi Flax wrote: Hi all, I’m running Kafka 0.10.0.1 on Java 8 on Linux — same for brokers and streams nodes. I’m attempting to scale up a Streams app that does a lot of I/O — in particular, I’m ho