Hey thanks for the help everyone, I’m gonna use the new scala 2.0 libraries. Im getting the craziest errorwhen building this though but I’m not a maven expert. I have to use maven right now (not sbt) because I don’t own this project at work. Anyway whenever I add the maven dependency - <dependency> <groupId>org.apache.org</groupId> <artifactId>kafka-streams-scala_2.11</artifactId> <version>2.0.0</version> </dependency>
I then get all kinds of errors in this code that already runs. Like if I leave the old Scala wrapper Lifht code in there and just bring in this new dependency into the Pom file, I can’t even do some import statements. Like when I do “import scala.collection.JavaConverters._” then I get the following error - “object ‘language’ is not a member of package org.apache.kafka.streams.scala”. So what can I be checking to make sure that won’t do this? Thanks! Sent from my iPhone > On Sep 10, 2018, at 9:48 PM, Liam Clarke <liam.cla...@adscale.co.nz> wrote: > > Hi Michael Eugene, > > You're correct - you only need to upgrade your Kafka Streams dependencies > in your build file. Looking at MVN Repository, the streams lib will > implicitly bring in its dependency on kafka-clients, but you can always > include your own explicit dependency on it. > https://mvnrepository.com/artifact/org.apache.kafka/kafka-streams/2.0.0 > > Kind regards, > > Liam Clarke > >> On Tue, Sep 11, 2018 at 1:43 PM, Michael Eugene <far...@hotmail.com> wrote: >> >> Well to bring up that kafka to 2.0, do I just need for sbt kafka clients >> and kafka streams 2.0 for sbt? And it doesn't matter if the system is not >> Kafka 2.0? Upgrading Kafka itself is probably not an option or me right now. >> >> >> >> >> >> ________________________________ >> From: John Roesler <j...@confluent.io> >> Sent: Monday, September 10, 2018 4:18:24 PM >> To: users@kafka.apache.org >> Subject: Re: SAM Scala aggregate >> >> In addition to the other suggestions, if you're having too much trouble >> with the interface, you can always fall back to creating anonymous >> Initializer/Aggregator instances the way you would if programming in Java. >> This way, you wouldn't need SAM conversion at all (all it's doing is >> turning your functions into an Initializer and Aggregator). >> >> On Mon, Sep 10, 2018 at 8:32 AM Sean Glover <sean.glo...@lightbend.com> >> wrote: >> >>> Hi, >>> >>> SAM conversions can be enabled in 2.11 with the -Xexperimental to scalac. >>> However, this version of SAM conversions isn't recommended for production >>> and was significantly refactored for 2.12. When we contributed this API >> to >>> Kafka we explicitly removed the SAM conversions because we didn't want to >>> require the Kafka build (or end user builds) to enable this experimental >>> feature. >>> >>> Echoing what Debasish has already said, the lightbend/kafka-streams-scala >>> project is deprecated. If the app your building is destined for >> production >>> you should use the Kafka 2.0+ client libraries which are backwards >>> compatible with older versions of the brokers; you don't need to get >>> everyone else to upgrade their clients or your cluster. >>> >>> Regards >>> Sean >>> >>> On Sun, Sep 9, 2018 at 3:59 PM Michael Eugene <far...@hotmail.com> >> wrote: >>> >>>> I’m integrating with lot of other applications and it would be a little >>>> cowboy to chose my own version. I can commend that we all went to 2.0 >>> but >>>> there are already working application me so it won’t be possible to >>> upgrade >>>> everything that everyone else is doing. That will have to happen >>> sometime >>>> though. It’s just not gonna happen this week and my goal is to get >>>> something working by tomorrow and I’ll stay up all night if I have to. >>>> >>>> Sent from my iPhone >>>> >>>>> On Sep 9, 2018, at 2:11 PM, Matthias J. Sax <matth...@confluent.io> >>>> wrote: >>>>> >>>>> Why can't you use Kafka Streams 2.0? >>>>> >>>>> Note: Kafka Streams is backward compatible and it can connect to >> older >>>>> brokers -- it's not required to upgrade your cluster to use Kafka >>>>> Streams 2.0 -- updating you maven/gradle dependency is sufficient. >>>>> >>>>> Also, AFAIK SAM conversions are only available in Scala 2.12. >>>>> >>>>> >>>>> -Matthias >>>>> >>>>>> On 9/9/18 11:37 AM, Michael Eugene wrote: >>>>>> I can’t do Kafka 2.0. I am limited to this version right now. If I >>>> continue to struggle with it this much, I can eventually do that. >>> However, >>>> I know other people in the organization have things kafka working with >>>> Scala. Probably not a good idea to say it’s a necessity when it’s not >>>> completely necessary. Your point is well taken though, I am considering >>> it. >>>>>> >>>>>> Sent from my iPhone >>>>>> >>>>>>> On Sep 9, 2018, at 1:10 PM, Debasish Ghosh < >> ghosh.debas...@gmail.com >>>> >>>> wrote: >>>>>>> >>>>>>> I don't have the environment to run the Scala code right now. Will >> be >>>>>>> tomorrow until I have one .. >>>>>>> >>>>>>> Now that Scala API is part of the official Kafka distribution. Can >> u >>>> please >>>>>>> try that out instead of kafka-streams-scala ? The library is now >>>> deprecated >>>>>>> and I remember we ran into some SAM related issues with Scala 2.11 >>>> (which >>>>>>> worked fine with 2.12). They were finally fixed in the Kafka >>>> distribution - >>>>>>> there are some differences in the APIs as well .. >>>>>>> >>>>>>> regards. >>>>>>> >>>>>>>> On Sun, Sep 9, 2018 at 11:32 PM Michael Eugene < >> far...@hotmail.com> >>>> wrote: >>>>>>>> >>>>>>>> I’m using 2.11.11 >>>>>>>> >>>>>>>> Sent from my iPhone >>>>>>>> >>>>>>>>> On Sep 9, 2018, at 12:13 PM, Debasish Ghosh < >>>> ghosh.debas...@gmail.com> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Which version of Scala are u using ? >>>>>>>>> >>>>>>>>>> On Sun, 9 Sep 2018 at 10:44 AM, Michael Eugene < >>> far...@hotmail.com> >>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I am using kafak-sreams-scala >>>>>>>>>> https://github.com/lightbend/kafka-streams-scala, and I am >> trying >>>> to >>>>>>>>>> implement something very simple and I am getting a compilation >>>> error by >>>>>>>> the >>>>>>>>>> "aggregate" method. The error is "Cannot resolve overload method >>>>>>>>>> 'aggregate'" and "Unspecified value parameters: materialized: >>>>>>>>>> Materialized[String, NotInferedVR, KeyValueStore[Bytes, >>>> Array[Byte]]]" >>>>>>>>>> [https://avatars0.githubusercontent.com/u/16247783?s=400&v=4]< >>>>>>>>>> https://github.com/lightbend/kafka-streams-scala> >>>>>>>>>> >>>>>>>>>> GitHub >>>>>>>>>> - lightbend/kafka-streams-scala: Thin Scala wrapper ...< >>>>>>>>>> https://github.com/lightbend/kafka-streams-scala> >>>>>>>>>> github.com >>>>>>>>>> Note: >>>>>>>>>> Scala API for Kafka Streams have been accepted for inclusion in >>>> Apache >>>>>>>>>> Kafka. We have been working with the Kafka team since the last >>>> couple of >>>>>>>>>> months working towards meeting the standards and guidelines for >>> this >>>>>>>>>> activity. Lightbend and Alexis Seigneurin have >>>>>>>>>> contributed this library (with ... >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> However when I add a third argument for a Materialized, I get >> the >>>>>>>>>> compilation error "Too may arguments for method aggregate(() >> =>VR, >>>>>>>> (K,V,VR) >>>>>>>>>> => VR)" >>>>>>>>>> >>>>>>>>>> It doesn't make sense anymore what could be breaking this. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> val myStream = builder >>>>>>>>>> .stream(inputTopic) >>>>>>>>>> .map{ (key: String, value: Array[Byte]) => >>>>>>>>>> println(s"key = ${key}") >>>>>>>>>> val newKey = GroupByAction.getGroupByKeyFromByteAray(value) >>>>>>>>>> >>>>>>>>>> val newValue = GroupByAction.getGroupByValueFromByteAray( >> value) >>>>>>>>>> >>>>>>>>>> println(s"newKey = ${newKey}") >>>>>>>>>> (newKey, serialise(newValue))} >>>>>>>>>> >>>>>>>>>> .groupByKey >>>>>>>>>> .aggregate(()=> 0L, (k,v,vr) => vr + 1) >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> Sent from my iPhone >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Debasish Ghosh >>>>>>> http://manning.com/ghosh2 >>>>>>> http://manning.com/ghosh >>>>>>> >>>>>>> Twttr: @debasishg >>>>>>> Blog: http://debasishg.blogspot.com >>>>>>> Code: http://github.com/debasishg >>>>> >>>> >>> >>> >>> -- >>> Senior Software Engineer, Lightbend, Inc. >>> >>> <http://lightbend.com> >>> >>> @seg1o <https://twitter.com/seg1o>, in/seanaglover >>> <https://www.linkedin.com/in/seanaglover/> >>> >>
