Re: Kafka Streams: Inconsistent Results with KStream-KTable Join Under Load

2025-01-10 Thread Matthias J. Sax
Hi, thanks for reporting this issue. I did look into it, and it is indeed a bug: https://issues.apache.org/jira/browse/KAFKA-18478 The workaround should be to set the StreamsConfig value serde to the stream-side value serde to make it work. -Matthias On 1/10/25 5:36 AM, Hagiu Alexandru wr

Re: Kafka Streams: Inconsistent Results with KStream-KTable Join Under Load

2025-01-10 Thread Hagiu Alexandru
Hello, Kind reminder :) Also, I want to add that I am trying to add a grace period for my kstream-ktable join, but I encounter an exception. It seems the Serde that I am providing in the join description is not taken into consideration when creating the *RocksDBTimeOrderedKeyValueBuffer*. *https

Re: Kafka Streams: Inconsistent Results with KStream-KTable Join Under Load

2024-12-30 Thread Matthias J. Sax
Seems this question was cross-posted on SO: https://stackoverflow.com/questions/79299571/inconsistent-results-with-kstream-ktable-join-under-load Left an answer there. On 12/28/24 11:45 PM, Hagiu Alexandru wrote: Hello, I'm developing a Quarkus microservice that utilizes Kafka Streams to proc

Re: Kafka-Streams-Scala for Scala 3

2024-02-09 Thread Josep Prat
Hi Matthias, Yes, it's just a matter of adding the [DISCUSS] prefix in the subject. By the way, I didn't say this won't need a KIP, just that I won't be pushing for it, but other maintainers might think it's needed. For the discuss thread, you should write down what changes in the build and what

Re: Kafka-Streams-Scala for Scala 3

2024-02-09 Thread Matthias Berndt
Hey Josep, I'm glad you agree that a KIP is not needed here, and I agree with you that how to publish these artifacts should be discussed with the Kafka team. In fact, this is what I created this thread for 😁 This is my first time contributing to Kafka, so I'm going to have to ask what a DISCUSS t

Re: Kafka-Streams-Scala for Scala 3

2024-02-09 Thread Josep Prat
Hi Matthias, It's not adding a new functionality but it's changing the way to generate artifacts. In the end we are talking about generating a new binary. I could live with not having a KIP, but a DISCUSS thread I think it's necessary. This signals the community members and maintainers that their

Re: Kafka-Streams-Scala for Scala 3

2024-02-09 Thread Matthias Berndt
Hi Matthias, Hi Josep, I'm afraid I can't do the KIP thing as the signup process for Apache Confluence requires sending me a password reset link via E-Mail and said E-Mail doesn't seem to reach me for some reason. I've contacted the Apache infrastructure team but haven't yet heard back from them.

Re: Kafka-Streams-Scala for Scala 3

2024-02-08 Thread Matthias J. Sax
Josep, thanks for helping with this. I was also thinking if we might need a KIP for this change. Since you had the same though, I would say, yes, let's do a KIP. @Matthias: can you prepare a KIP? You can read up on the details on the wiki page: https://cwiki.apache.org/confluence/display/KA

Re: Kafka-Streams-Scala for Scala 3

2024-02-08 Thread Matthias Berndt
Hey Josep et al, I've created a ticket regarding this. https://issues.apache.org/jira/browse/KAFKA-16237 All the best, Matthias Am Do., 8. Feb. 2024 um 11:42 Uhr schrieb Josep Prat : > > Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know when > your accounts are created and w

Re: Kafka-Streams-Scala for Scala 3

2024-02-08 Thread Josep Prat
Go ahead and ask for a JIRA and Wiki account (Confluence). Let us know when your accounts are created and we'll properly set them up so you can create and assign tickets to you. Best, On Thu, Feb 8, 2024 at 11:32β€―AM Matthias Berndt wrote: > Thanks Josep, I've applied for a JIRA account and addr

Re: Kafka-Streams-Scala for Scala 3

2024-02-08 Thread Matthias Berndt
Thanks Josep, I've applied for a JIRA account and addressed your review comments. Am Do., 8. Feb. 2024 um 09:19 Uhr schrieb Josep Prat : > > Hi Matthias, > > I think for this particular case it would be worth creating a JIRA ticket > for this as it's a new "feature". > Regarding the change itself,

Re: Kafka-Streams-Scala for Scala 3

2024-02-08 Thread Josep Prat
Hi Matthias, I think for this particular case it would be worth creating a JIRA ticket for this as it's a new "feature". Regarding the change itself, I think we need to clarify how the release process would work. Right now, the script `gradlewAll` is used (which basically runs the build with Scala

Re: Kafka-Streams-Scala for Scala 3

2024-02-07 Thread Matthias Berndt
Hi Matthias J., Hi Lucas, Hi Josep, Thank you for your encouraging responses regarding a Scala 3 port of Kafka-Streams-Scala, and apologies for the late response from my side. I have now created a PR to port Kafka-Streams-Scala to Scala 3 (while retaining support for 2.13 and 2.12). Almost no chan

Re: Kafka-Streams-Scala for Scala 3

2024-02-01 Thread Josep Prat
Hi, For reference, prior work on this: https://github.com/apache/kafka/pull/11350 https://github.com/apache/kafka/pull/11432 Best, On Thu, Feb 1, 2024, 15:55 Lucas Brutschy wrote: > Hi Matthiases, > > I know Scala 2 fairly well, so I'd be happy to review changes that add > Scala 3 support. How

Re: Kafka-Streams-Scala for Scala 3

2024-02-01 Thread Lucas Brutschy
Hi Matthiases, I know Scala 2 fairly well, so I'd be happy to review changes that add Scala 3 support. However, as Matthias S. said, it has to be driven by people who use Scala day-to-day, since I believe most Kafka Streams committers are working with Java. Rewriting the tests to not use Embedded

Re: Kafka-Streams-Scala for Scala 3

2024-01-31 Thread Matthias J. Sax
Thanks for raising this. The `kafka-streams-scala` module seems to be an important feature for Kafka Streams and I am generally in favor of your proposal to add Scala 3 support. However, I am personally no Scala person and it sounds like quite some overhead. If you are willing to drive and own

Re: [Kafka Streams], Processor API for KTable and KGroupedStream

2024-01-12 Thread Matthias J. Sax
`KGroupedStream` is just an "intermediate representation" to get a better flow in the DSL. It's not a "top level" abstraction like KStream/KTable. For `KTable` there is `transformValue()` -- there is no `transform()` because keying must be preserved -- if you want to change the keying you ne

Re: Kafka Streams Issue

2022-04-01 Thread John Roesler
Hi Daan, First of all, it does sound like that is a correct implementation of QueryableStoreProvider. Kudos for taking that on; the complexity of that API was one of my top motivations for replacing it with IQv2! (https://cwiki.apache.org/confluence/display/KAFKA/KIP-796%3A+Interactive+Query+v2 )

Re: Kafka Streams: Dynamic Topic Routing & Nonexistent Topics

2020-07-08 Thread Rhys Anthony McCaig
Guozhang - thank you for your thoughts. You are right - this is more about the producer client than the streams client. caching the metadata outside producer, e.g. in an admin client would not > be a perfect solution since in either way your metadata cache inside the > producer or inside the admi

Re: Kafka Streams: Dynamic Topic Routing & Nonexistent Topics

2020-07-04 Thread Guozhang Wang
Hello, Thanks for reaching out to the community for this. I think (maybe you've also suggested) it is rather an observation on producer client than on streams client. Generally speaking we want to know if we can fail fast if the metadata cannot be found in producer.send() call. And here are my tho

Re: kafka streams application design on state stores

2019-10-07 Thread Matthias J. Sax
I am not sure if I understand your scenario. What do you exactly mean by "back ground job" and by "sweeps the records in the store using range calls" ? As you noted, only the `Processor` that have access to the store should write/delete anything from it. You should not try to modify the store from

Re: Kafka streams rebalancing issue

2019-06-17 Thread Matthias J. Sax
> This leads to lockexception in >> consumer2 and consumer2 remains in livelock to create state directories for >> those two partitons. There is a fix for a `LockException` during rebalance in 0.11.0.1: https://issues.apache.org/jira/browse/KAFKA-5167 Maybe upgrading helps? Note, that you can upg

Re: Kafka streams issue

2019-06-14 Thread Matthias J. Sax
It is correct, that Kafka Streams does not create input/output/through() topics. You should create those topics upfront. > We've *set auto.create.topics.enable=true *in our kafka >> server properties file and we'd expect that the topics get created, but the >> consumer goes into SHUTDOWN mode and

Re: Kafka streams issue

2019-06-12 Thread Brian Putt
After adding more logging details, we found that we hadn't created topics yet for the consumer & producer for the streams application. We've added a check when starting up to verify that the topics exist, otherwise we exit our app. We're not dynamically creating topics and we want to create them up

Re: Kafka streams issue

2019-06-12 Thread Matthias J. Sax
Does the application transit to RUNNING state? Also check `KafkaStreams#localThreadsMetadata()` what tasks are assigned? You might also enable DEBUG logs for `org.apache.kafka.clients.consumer.**` classes to see if the consumer sends fetch request to the broker. -Matthias On 6/11/19 7:03 PM, Br

Re: Kafka streams issue

2019-06-11 Thread Brian Putt
The application just hangs (we let it sit for ~1 hour, small dataset as we're testing), we can restart it listening to 1 of the 3 topics we start it with and it chugs along, no problem. The same code is executed as separate application.ids listening to other topics without any issues. We'll try to

Re: Kafka streams issue

2019-06-11 Thread Matthias J. Sax
What do you exactly observe? - Does the application rebalance correctly? - Does it start processing? - Anything in the logs about the status of the application? -Matthias On 6/10/19 6:19 PM, Brian Putt wrote: > Hello, > > I'm working with the kafka streams api and am running into issues whe

Re: Kafka Streams Library for Go

2019-05-22 Thread Alex Woolford
Hi Jay, I'm not aware of a Go implementation of Kafka Streams in Apache Kafka. If anyone here knows something I don't, feel free to correct me. goka (https://github.com/lovoo/goka), while not part of Apache Kafka, seems to be pretty popular. You may also want to check out KSQL which, although it'

Re: Kafka Streams Join does not work

2019-03-15 Thread Ryanne Dolan
If your producer is using transactions, it's possible that the producer was killed in the middle of a transaction, in which case any un-committed records would be logged on the broker but skipped by downstream consumers. Otherwise, it's likely that the leader for the partition crashed before the r

Re: Kafka Streams Join does not work

2019-03-15 Thread Π€Π΅Π΄ΠΎΡ€ Π§Π΅Ρ€Π½ΠΈΠ»ΠΈΠ½
Thanks, I understand how consumer works. The main question is related to why the join did not work and how it happened that only one message remained uncommitted. ΠΏΡ‚, 15 ΠΌΠ°Ρ€. 2019 Π³. Π² 16:29, Ryanne Dolan : > Hello! When using exactly-once semantics, uncommitted or aborted records > are skipped b

Re: Kafka Streams Join does not work

2019-03-15 Thread Ryanne Dolan
Hello! When using exactly-once semantics, uncommitted or aborted records are skipped by the consumer as if they don't exist. When inspecting the topic manually, use isolation.level=read_committed to get the same behavior. Ryanne On Fri, Mar 15, 2019, 6:08 AM Π€Π΅Π΄ΠΎΡ€ Π§Π΅Ρ€Π½ΠΈΠ»ΠΈΠ½ wrote: > I also noti

Re: Kafka Streams Join does not work

2019-03-15 Thread Π€Π΅Π΄ΠΎΡ€ Π§Π΅Ρ€Π½ΠΈΠ»ΠΈΠ½
I also noticed another important thing now. Message which used for join is uncommitted. I understood it with the help of consumer's setting isolation level - read_committed. The message got into the topic using the same stream app. Remind that stream app has processing guarantee = exactly_once. How

Re: Kafka Streams new release

2018-07-10 Thread Ofir Manor
>From the release plan: " While the target release date is fixed at ~2w after code freeze, RCs will roll out as needed until the release vote passes" https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80448820 You can follow the voting threads on this mailing list - the current t

Re: Kafka Streams debugging with "no fluent" API choice

2017-08-01 Thread Damian Guy
e Advisor > > Twitter : @ppatierno<http://twitter.com/ppatierno> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> > Blog : DevExperience<http://paolopatierno.wordpress.com/> > > > ________ > From: Damian Guy > Sent: Tuesd

Re: Kafka Streams debugging with "no fluent" API choice

2017-08-01 Thread Paolo Patierno
From: Damian Guy Sent: Tuesday, August 1, 2017 2:11 PM To: dev@kafka.apache.org Subject: Re: Kafka Streams debugging with "no fluent" API choice Hi Paolo, The change would require a KIP as it is a public API change. I don't see any harm in making the

Re: Kafka Streams debugging with "no fluent" API choice

2017-08-01 Thread Damian Guy
<http://paolopatierno.wordpress.com/> > > > > From: Damian Guy > Sent: Tuesday, August 1, 2017 12:11 PM > To: dev@kafka.apache.org > Subject: Re: Kafka Streams debugging with "no fluent" API choice > > I don't know s

Re: Kafka Streams debugging with "no fluent" API choice

2017-08-01 Thread Paolo Patierno
ev@kafka.apache.org Subject: Re: Kafka Streams debugging with "no fluent" API choice I don't know specifically why this is removed, however if you want to get the same functionality you can use peek, i.e: stream.map(...).peek(...).filter(..) You can log the key values out in t

Re: Kafka Streams debugging with "no fluent" API choice

2017-08-01 Thread Damian Guy
I don't know specifically why this is removed, however if you want to get the same functionality you can use peek, i.e: stream.map(...).peek(...).filter(..) You can log the key values out in the peek call. On Tue, 1 Aug 2017 at 11:48 Paolo Patierno wrote: > Hi guys, > > > I was thinking about

Re: Kafka streams KStream and ktable join issue

2017-06-24 Thread Matthias J. Sax
I thought, we drop records with null key? No? -Matthias On 6/23/17 12:25 AM, Damian Guy wrote: > My guess is it is because the record doesn't have a key, i.e., the key is > null. We have a fix for this in 0.11, in that we will skip records with a > null key during restore. > > On Fri, 23 Jun 201

Re: Kafka streams KStream and ktable join issue

2017-06-23 Thread Damian Guy
My guess is it is because the record doesn't have a key, i.e., the key is null. We have a fix for this in 0.11, in that we will skip records with a null key during restore. On Fri, 23 Jun 2017 at 03:57 Matthias J. Sax wrote: > Hi, > > can you reproduce the error reliably? Are use using 0.10.2.0

Re: Kafka streams KStream and ktable join issue

2017-06-22 Thread Matthias J. Sax
Hi, can you reproduce the error reliably? Are use using 0.10.2.0 or 0.10.2.1? It's unclear to me, how an NPE can occur. It seems to happen within Streams library. Might be a bug. Not sure atm. -Matthias On 6/22/17 9:43 AM, Shekar Tippur wrote: > Hello, > > I am trying to perform a simple join

Re: Kafka Streams : parallelism related to repartition topic partitions as well

2017-06-20 Thread Matthias J. Sax
Your observation is correct. The paragraph you quote is not very precise but also not necessarily wrong. The example is simplified and assumes that there is no re-partitioning even if it is not mentioned explicitly. -Matthias On 6/20/17 9:32 AM, Paolo Patierno wrote: > Hi devs, > > > at foll

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-18 Thread Paolo Patierno
p://it.linkedin.com/in/paolopatierno> Blog : DevExperience<http://paolopatierno.wordpress.com/> From: Michal Borowiecki Sent: Sunday, June 18, 2017 9:34 AM To: dev@kafka.apache.org; Jay Kreps Cc: us...@kafka.apache.org; Matthias J. Sax Subject: Re: Kafka Streams vs Spark

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-18 Thread Michal Borowiecki
DevExperience<http://paolopatierno.wordpress.com/> <http://paolopatierno.wordpress.com/> From: Eno Thereska <mailto:eno.there...@gmail.com> Sent: Thursday, June 15, 2017 3:57 PM To:us...@kafka.apache.org <mailto:us..

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-16 Thread Matthias J. Sax
> your article ! Thanks ! >>> >>> >>> Paolo >>> >>> >>> Paolo Patierno >>> Senior Software Engineer (IoT) @ Red Hat >>> Microsoft MVP on Windows Embedded & IoT >>> Microsoft Azure Advisor >>> >>> Twitter

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-16 Thread Jay Kreps
o even because the > usage is pretty simple but this scenario could have a lack against Spark. > > > Thanks, > > Paolo. > > > Paolo Patierno > Senior Software Engineer (IoT) @ Red Hat > Microsoft MVP on Windows Embedded & IoT > Microsoft Azure Advisor > > T

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-16 Thread Jay Kreps
o even because the > usage is pretty simple but this scenario could have a lack against Spark. > > > Thanks, > > Paolo. > > > Paolo Patierno > Senior Software Engineer (IoT) @ Red Hat > Microsoft MVP on Windows Embedded & IoT > Microsoft Azure Advisor > > T

Re: Kafka Streams vs Spark Streaming : reduce by window

2017-06-16 Thread Michal Borowiecki
! Paolo Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows Embedded & IoT Microsoft Azure Advisor Twitter : @ppatierno<http://twitter.com/ppatierno> Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> Blog : DevExperience<http://paolopati

Re: Kafka-Streams: Cogroup

2017-05-01 Thread Guozhang Wang
Kyle, What's your apache id? I can grant you the permission. Guozhang On Sat, Apr 29, 2017 at 7:33 AM, Kyle Winkelman wrote: > I don't seem to have permission. When logged in I can neither edit the > main page nor create an additional KIP. > > Thanks, > Kyle > > On Thu, Apr 27, 2017 at 12:35

Re: Kafka-Streams: Cogroup

2017-04-29 Thread Kyle Winkelman
I don't seem to have permission. When logged in I can neither edit the main page nor create an additional KIP. Thanks, Kyle On Thu, Apr 27, 2017 at 12:35 PM, Eno Thereska wrote: > Hi Kyle, > > I believe Guozhang has now given you permission to edit the KIP wiki at > https://cwiki.apache.org/con

Re: Kafka-Streams: Cogroup

2017-04-27 Thread Eno Thereska
Hi Kyle, I believe Guozhang has now given you permission to edit the KIP wiki at https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals. Could you see if you can add this there? Many thanks Eno On Wed, Apr 26, 2017 at 6:00 PM, Kyle Winkelman wrote: > Thank you for your r

Re: Kafka-Streams: Cogroup

2017-04-26 Thread Kyle Winkelman
Thank you for your reply. I have attached my first attempt at writing a KIP and I was wondering if you could review it and share your thoughts. Going forward I would like to create this KIP. I was wondering whom I should ask to get the necessary permissions on the wiki. Username: winkelman.kyle

Re: Kafka-Streams: Cogroup

2017-04-21 Thread Eno Thereska
Hi Kyle, Sorry for the delay in replying. I think it's worth doing a KIP for this one. One super helpful thing with KIPs is to list a few more scenarios that would benefit from this approach. In particular it seems the main benefit is from reducing the number of state stores. Does this necessarily

Re: Kafka-Streams: Cogroup

2017-04-14 Thread Kyle Winkelman
Eno, Thanks for the response. The figure was just a restatement of my questions. I have made an attempt at a low level processor and it appears to work but it isn't very pretty and was hoping for something at the streams api level. I have written some code to show an example of how I see the Cogro

Re: Kafka-Streams: Cogroup

2017-04-13 Thread Eno Thereska
Hi Kyle, (cc-ing user list as well) This could be an interesting scenario. Two things to help us think through it some more: 1) it seems you attached a figure, but I cannot seem to open it. 2) what about using the low level processor API instead of the DSL as approach 3? Do you have any thought

Re: Kafka Streams for Remote Server

2016-08-03 Thread Guozhang Wang
Hi Misha, If it works locally, then I would still suspect that it is due to a transient timing issue: note that the create-topic script is non-blocking, i.e. even when it returns it does not necessarily guarantee that the leader metadata information has been completed propagating to brokers. Gu

Re: Kafka Streams for Remote Server

2016-07-28 Thread mishadoff
Thanks Guozhang, Yes, I rely on auto-create, and it works locally. Maybe I need to tweak some timeout conf for that? Also I identified, that even I manually create a topic, it lists but I can not produce messages to this topic with the same exception. Producing to other topics works well, so it

Re: Kafka Streams for Remote Server

2016-07-27 Thread Guozhang Wang
Misha, Did you pre-create the sink topic before starting your application or you are relying on the broker-side auto-create for that topic? If you are relying on auto-create, then there is a transient period where the topic is created but the metadata has not been propagated to the brokers so the

Re: Kafka Streams question

2016-07-14 Thread Michael Noll
Also, in the next version of Kafka / Kafka Streams such "intermediate" topics will automatically be created for you when you do joins or aggregations: https://issues.apache.org/jira/browse/KAFKA-3561 So my previous message explained your options today when using the current release of Kafka Strea

Re: Kafka Streams question

2016-07-14 Thread Michael Noll
Poul, to add to what Matthias said: If you are wondering how to manually create a topic, you have basically two options. A. Use Kafka's CLI tools to create the topic "from the outside". # Example $ kafka-topics.sh --create --topic my-custom-toipc --zookeeper localhost:2181 --partitions

Re: Kafka Streams question

2016-07-14 Thread Matthias J. Sax
Hi, you can manually create a topic with the number of partitions you want to have and use this topic via through() KStream input = ... input.map().through("manually-created-topic").join(...) However, both KStream and KTable need to have the same number of partitions for perform the join. Thus,

Re: Kafka Streams - production use

2016-07-06 Thread Michael Noll
Dirk, we included the note "be careful when using Kafka Streams in production" because Kafka Streams as shipped in Kafka 0.10.0.0 is the first-ever release of Kafka Streams. In practice, users are running Streams applications in a variety of stages -- some are doing pilots or evaluations, some ar

Re: [Kafka Streams] Source Nodes

2016-06-29 Thread mishadoff
Thanks Matthias! Got it working with through, still use 0.10.0.0 version. > On Jun 29, 2016, at 22:42, Matthias J. Sax wrote: > > Hi, > > for joins, data of both inputs must be co-located, ie, partitioned on > the same key and have the same number of partitions: > > See "Note" box at: > http:

Re: [Kafka Streams] Source Nodes

2016-06-29 Thread Matthias J. Sax
Hi, for joins, data of both inputs must be co-located, ie, partitioned on the same key and have the same number of partitions: See "Note" box at: http://docs.confluent.io/3.0.0/streams/developer-guide.html?highlight=join#joining-streams From an older email thread about the same issue: >>> If yo

Re: Kafka Streams aggregation store

2016-06-08 Thread Eno Thereska
Hi Alexander, I haven't tried Kafka Streams on Windows but did notice that Microsoft has merged code into github to make RocksDB available on Windows. Perhaps this is useful: https://blogs.msdn.microsoft.com/bingdevcenter/2015/07/22/open-source-contribution-from-bing-rocksdb-is-now-available-in-

Re: Kafka Streams file handle leak

2016-04-25 Thread Greg Fodor
Great, thanks: https://issues.apache.org/jira/browse/KAFKA-3619 On Mon, Apr 25, 2016 at 11:13 AM, Guozhang Wang wrote: > Thanks for reporting that Greg. > > I have just added you as a Kafka contributor so you should be able to > create JIRAs now. Could you create a new JIRA for this and change

Re: Kafka Streams file handle leak

2016-04-25 Thread Guozhang Wang
Thanks for reporting that Greg. I have just added you as a Kafka contributor so you should be able to create JIRAs now. Could you create a new JIRA for this and change the RB title with the JIRA number? Thanks! Guozhang On Mon, Apr 25, 2016 at 12:13 AM, Greg Fodor wrote: > Apologies for a sec

Re: Kafka Streams - read topic from beginning

2016-04-21 Thread Guozhang Wang
Hello Maria, We have some thoughts about supporting finer grained flow controls in Kafka Streams https://issues.apache.org/jira/browse/KAFKA-3478 as part of a big effort to improve re-processing user experience, which covers this use case. We are shooting to have this post 0.10.0.0. As for now, o

Re: kafka-streams: Using custom partitioner with higher level DSL constructs?

2016-03-21 Thread Guozhang Wang
Also, have you looked at Kafka Connect released in 0.9? It has a MySQL binlog connector implementation in progress, just thinking maybe you would be interested to check it out and see if there are any feedbacks that you want to give. https://github.com/wushujames/kafka-mysql-connector Guozhang O

Re: kafka-streams: Using custom partitioner with higher level DSL constructs?

2016-03-21 Thread Guozhang Wang
Hello Ben, 1. Currently Kafka Streams high-level DSL does not take the StreamPartitioner yet, please feel free to file a JIRA so that we can keep track and discuss of whether / how to incorporate it into Kafka Streams DSL. 2. As for now, you can do two work arounds: 1) use `process()` function t

Re: kafka streams

2016-03-05 Thread Guozhang Wang
Thanks Bill, The example code looks great! I will read through it, and am expecting your high-level examples already! Guozhang On Sat, Mar 5, 2016 at 12:39 PM, Bill Bejeck wrote: > Oops, forgot that little bit of information - > http://codingjunkie.net/kafka-processor-part1/ > > A review wou

Re: kafka streams

2016-03-05 Thread Bill Bejeck
Oops, forgot that little bit of information - http://codingjunkie.net/kafka-processor-part1/ A review would be greatly appreciated to make sure I'm on track. Thanks On Sat, Mar 5, 2016 at 3:12 PM, Neha Narkhede wrote: > Great. Mind sharing the draft or linking to the post if already publishe

Re: kafka streams

2016-03-05 Thread Neha Narkhede
Great. Mind sharing the draft or linking to the post if already published? Happy to review if that is useful. On Sat, Mar 5, 2016 at 12:07 PM, Bill Bejeck wrote: > All, > > I'm pretty excited about kafka-streams and I've written a blog (first of > 2-3) on the subject, and I'd thought I'd share.