Forgot to add. On the naming issues, Kafka Metamorphosis is a clear winner :)
-- Gianmarco On 7 July 2015 at 13:26, Gianmarco De Francisci Morales <g...@apache.org> wrote: > Hi, > > @Martin, thanks for you comments. > Maybe I'm missing some important point, but I think coupling the releases > is actually a *good* thing. > To make an example, would it be better if the MR and HDFS components of > Hadoop had different release schedules? > > Actually, keeping the discussion in a single place would make agreeing on > releases (and backwards compatibility) much easier, as everybody would be > responsible for the whole codebase. > > That said, I like the idea of absorbing samza-core as a sub-project, and > leave the fancy stuff separate. > It probably gives 90% of the benefits we have been discussing here. > > Cheers, > > -- > Gianmarco > > On 7 July 2015 at 02:30, Jay Kreps <jay.kr...@gmail.com> wrote: > >> Hey Martin, >> >> I agree coupling release schedules is a downside. >> >> Definitely we can try to solve some of the integration problems in >> Confluent Platform or in other distributions. But I think this ends up >> being really shallow. I guess I feel to really get a good user experience >> the two systems have to kind of feel like part of the same thing and you >> can't really add that in later--you can put both in the same downloadable >> tar file but it doesn't really give a very cohesive feeling. I agree that >> ultimately any of the project stuff is as much social and naming as >> anything else--theoretically two totally independent projects could work >> to >> tightly align. In practice this seems to be quite difficult though. >> >> For the frameworks--totally agree it would be good to maintain the >> framework support with the project. In some cases there may not be too >> much >> there since the integration gets lighter but I think whatever stubs you >> need should be included. So no I definitely wasn't trying to imply >> dropping >> support for these frameworks, just making the integration lighter by >> separating process management from partition management. >> >> You raise two good points we would have to figure out if we went down the >> alignment path: >> 1. With respect to the name, yeah I think the first question is whether >> some "re-branding" would be worth it. If so then I think we can have a big >> thread on the name. I'm definitely not set on Kafka Streaming or Kafka >> Streams I was just using them to be kind of illustrative. I agree with >> your >> critique of these names, though I think people would get the idea. >> 2. Yeah you also raise a good point about how to "factor" it. Here are the >> options I see (I could get enthusiastic about any of them): >> a. One repo for both Kafka and Samza >> b. Two repos, retaining the current seperation >> c. Two repos, the equivalent of samza-api and samza-core is absorbed >> almost like a third client >> >> Cheers, >> >> -Jay >> >> On Mon, Jul 6, 2015 at 1:18 PM, Martin Kleppmann <mar...@kleppmann.com> >> wrote: >> >> > Ok, thanks for the clarifications. Just a few follow-up comments. >> > >> > - I see the appeal of merging with Kafka or becoming a subproject: the >> > reasons you mention are good. The risk I see is that release schedules >> > become coupled to each other, which can slow everyone down, and large >> > projects with many contributors are harder to manage. (Jakob, can you >> speak >> > from experience, having seen a wider range of Hadoop ecosystem >> projects?) >> > >> > Some of the goals of a better unified developer experience could also be >> > solved by integrating Samza nicely into a Kafka distribution (such as >> > Confluent's). I'm not against merging projects if we decide that's the >> way >> > to go, just pointing out the same goals can perhaps also be achieved in >> > other ways. >> > >> > - With regard to dropping the YARN dependency: are you proposing that >> > Samza doesn't give any help to people wanting to run on >> YARN/Mesos/AWS/etc? >> > So the docs would basically have a link to Slider and nothing else? Or >> > would we maintain integrations with a bunch of popular deployment >> methods >> > (e.g. the necessary glue and shell scripts to make Samza work with >> Slider)? >> > >> > I absolutely think it's a good idea to have the "as a library" and "as a >> > process" (using Yi's taxonomy) options for people who want them, but I >> > think there should also be a low-friction path for common "as a service" >> > deployment methods, for which we probably need to maintain integrations. >> > >> > - Project naming: "Kafka Streams" seems odd to me, because Kafka is all >> > about streams already. Perhaps "Kafka Transformers" or "Kafka Filters" >> > would be more apt? >> > >> > One suggestion: perhaps the core of Samza (stream transformation with >> > state management -- i.e. the "Samza as a library" bit) could become >> part of >> > Kafka, while higher-level tools such as streaming SQL and integrations >> with >> > deployment frameworks remain in a separate project? In other words, >> Kafka >> > would absorb the proven, stable core of Samza, which would become the >> > "third Kafka client" mentioned early in this thread. The Samza project >> > would then target that third Kafka client as its base API, and the >> project >> > would be freed up to explore more experimental new horizons. >> > >> > Martin >> > >> > On 6 Jul 2015, at 18:51, Jay Kreps <jay.kr...@gmail.com> wrote: >> > >> > > Hey Martin, >> > > >> > > For the YARN/Mesos/etc decoupling I actually don't think it ties our >> > hands >> > > at all, all it does is refactor things. The division of >> responsibility is >> > > that Samza core is responsible for task lifecycle, state, and >> partition >> > > management (using the Kafka co-ordinator) but it is NOT responsible >> for >> > > packaging, configuration deployment or execution of processes. The >> > problem >> > > of packaging and starting these processes is >> > > framework/environment-specific. This leaves individual frameworks to >> be >> > as >> > > fancy or vanilla as they like. So you can get simple stateless >> support in >> > > YARN, Mesos, etc using their off-the-shelf app framework (Slider, >> > Marathon, >> > > etc). These are well known by people and have nice UIs and a lot of >> > > flexibility. I don't think they have node affinity as a built in >> option >> > > (though I could be wrong). So if we want that we can either wait for >> them >> > > to add it or do a custom framework to add that feature (as now). >> > Obviously >> > > if you manage things with old-school ops tools (puppet/chef/etc) you >> get >> > > locality easily. The nice thing, though, is that all the samza >> "business >> > > logic" around partition management and fault tolerance is in Samza >> core >> > so >> > > it is shared across frameworks and the framework specific bit is just >> > > whether it is smart enough to try to get the same host when a job is >> > > restarted. >> > > >> > > With respect to the Kafka-alignment, yeah I think the goal would be >> (a) >> > > actually get better alignment in user experience, and (b) express >> this in >> > > the naming and project branding. Specifically: >> > > 1. Website/docs, it would be nice for the "transformation" api to be >> > > discoverable in the main Kafka docs--i.e. be able to explain when to >> use >> > > the consumer and when to use the stream processing functionality and >> lead >> > > people into that experience. >> > > 2. Align releases so if you get Kafkza 1.4.2 (or whatever) that has >> both >> > > Kafka and the stream processing part and they actually work together. >> > > 3. Unify the programming experience so the client and Samza api share >> > > config/monitoring/naming/packaging/etc. >> > > >> > > I think sub-projects keep separate committers and can have a separate >> > repo, >> > > but I'm actually not really sure (I can't find a definition of a >> > subproject >> > > in Apache). >> > > >> > > Basically at a high-level you want the experience to "feel" like a >> single >> > > system, not to relatively independent things that are kind of >> awkwardly >> > > glued together. >> > > >> > > I think if we did that they having naming or branding like "kafka >> > > streaming" or "kafka streams" or something like that would actually >> do a >> > > good job of conveying what it is. I do that this would help adoption >> > quite >> > > a lot as it would correctly convey that using Kafka Streaming with >> Kafka >> > is >> > > a fairly seamless experience and Kafka is pretty heavily adopted at >> this >> > > point. >> > > >> > > Fwiw we actually considered this model originally when open sourcing >> > Samza, >> > > however at that time Kafka was relatively unknown and we decided not >> to >> > do >> > > it since we felt it would be limiting. From my point of view the three >> > > things have changed (1) Kafka is now really heavily used for stream >> > > processing, (2) we learned that abstracting out the stream well is >> > > basically impossible, (3) we learned it is really hard to keep the two >> > > things feeling like a single product. >> > > >> > > -Jay >> > > >> > > >> > > On Mon, Jul 6, 2015 at 3:37 AM, Martin Kleppmann < >> mar...@kleppmann.com> >> > > wrote: >> > > >> > >> Hi all, >> > >> >> > >> Lots of good thoughts here. >> > >> >> > >> I agree with the general philosophy of tying Samza more firmly to >> Kafka. >> > >> After I spent a while looking at integrating other message brokers >> (e.g. >> > >> Kinesis) with SystemConsumer, I came to the conclusion that >> > SystemConsumer >> > >> tacitly assumes a model so much like Kafka's that pretty much nobody >> but >> > >> Kafka actually implements it. (Databus is perhaps an exception, but >> it >> > >> isn't widely used outside of LinkedIn.) Thus, making Samza fully >> > dependent >> > >> on Kafka acknowledges that the system-independence was never as real >> as >> > we >> > >> perhaps made it out to be. The gains of code reuse are real. >> > >> >> > >> The idea of decoupling Samza from YARN has also always been >> appealing to >> > >> me, for various reasons already mentioned in this thread. Although >> > making >> > >> Samza jobs deployable on anything (YARN/Mesos/AWS/etc) seems >> laudable, >> > I am >> > >> a little concerned that it will restrict us to a lowest common >> > denominator. >> > >> For example, would host affinity (SAMZA-617) still be possible? For >> jobs >> > >> with large amounts of state, I think SAMZA-617 would be a big boon, >> > since >> > >> restoring state off the changelog on every single restart is painful, >> > due >> > >> to long recovery times. It would be a shame if the decoupling from >> YARN >> > >> made host affinity impossible. >> > >> >> > >> Jay, a question about the proposed API for instantiating a job in >> code >> > >> (rather than a properties file): when submitting a job to a cluster, >> is >> > the >> > >> idea that the instantiation code runs on a client somewhere, which >> then >> > >> pokes the necessary endpoints on YARN/Mesos/AWS/etc? Or does that >> code >> > run >> > >> on each container that is part of the job (in which case, how does >> the >> > job >> > >> submission to the cluster work)? >> > >> >> > >> I agree with Garry that it doesn't feel right to make a 1.0 release >> > with a >> > >> plan for it to be immediately obsolete. So if this is going to >> happen, I >> > >> think it would be more honest to stick with 0.* version numbers until >> > the >> > >> library-ified Samza has been implemented, is stable and widely used. >> > >> >> > >> Should the new Samza be a subproject of Kafka? There is precedent for >> > >> tight coupling between different Apache projects (e.g. Curator and >> > >> Zookeeper, or Slider and YARN), so I think remaining separate would >> be >> > ok. >> > >> Even if Samza is fully dependent on Kafka, there is enough substance >> in >> > >> Samza that it warrants being a separate project. An argument in >> favour >> > of >> > >> merging would be if we think Kafka has a much stronger "brand >> presence" >> > >> than Samza; I'm ambivalent on that one. If the Kafka project is >> willing >> > to >> > >> endorse Samza as the "official" way of doing stateful stream >> > >> transformations, that would probably have much the same effect as >> > >> re-branding Samza as "Kafka Stream Processors" or suchlike. Close >> > >> collaboration between the two projects will be needed in any case. >> > >> >> > >> From a project management perspective, I guess the "new Samza" would >> > have >> > >> to be developed on a branch alongside ongoing maintenance of the >> current >> > >> line of development? I think it would be important to continue >> > supporting >> > >> existing users, and provide a graceful migration path to the new >> > version. >> > >> Leaving the current versions unsupported and forcing people to >> rewrite >> > >> their jobs would send a bad signal. >> > >> >> > >> Best, >> > >> Martin >> > >> >> > >> On 2 Jul 2015, at 16:59, Jay Kreps <j...@confluent.io> wrote: >> > >> >> > >>> Hey Garry, >> > >>> >> > >>> Yeah that's super frustrating. I'd be happy to chat more about this >> if >> > >>> you'd be interested. I think Chris and I started with the idea of >> "what >> > >>> would it take to make Samza a kick-ass ingestion tool" but >> ultimately >> > we >> > >>> kind of came around to the idea that ingestion and transformation >> had >> > >>> pretty different needs and coupling the two made things hard. >> > >>> >> > >>> For what it's worth I think copycat (KIP-26) actually will do what >> you >> > >> are >> > >>> looking for. >> > >>> >> > >>> With regard to your point about slider, I don't necessarily >> disagree. >> > >> But I >> > >>> think getting good YARN support is quite doable and I think we can >> make >> > >>> that work well. I think the issue this proposal solves is that >> > >> technically >> > >>> it is pretty hard to support multiple cluster management systems the >> > way >> > >>> things are now, you need to write an "app master" or "framework" for >> > each >> > >>> and they are all a little different so testing is really hard. In >> the >> > >>> absence of this we have been stuck with just YARN which has >> fantastic >> > >>> penetration in the Hadoopy part of the org, but zero penetration >> > >> elsewhere. >> > >>> Given the huge amount of work being put in to slider, marathon, aws >> > >>> tooling, not to mention the umpteen related packaging technologies >> > people >> > >>> want to use (Docker, Kubernetes, various cloud-specific deploy >> tools, >> > >> etc) >> > >>> I really think it is important to get this right. >> > >>> >> > >>> -Jay >> > >>> >> > >>> On Thu, Jul 2, 2015 at 4:17 AM, Garry Turkington < >> > >>> g.turking...@improvedigital.com> wrote: >> > >>> >> > >>>> Hi all, >> > >>>> >> > >>>> I think the question below re does Samza become a sub-project of >> Kafka >> > >>>> highlights the broader point around migration. Chris mentions >> Samza's >> > >>>> maturity is heading towards a v1 release but I'm not sure it feels >> > >> right to >> > >>>> launch a v1 then immediately plan to deprecate most of it. >> > >>>> >> > >>>> From a selfish perspective I have some guys who have started >> working >> > >> with >> > >>>> Samza and building some new consumers/producers was next up. Sounds >> > like >> > >>>> that is absolutely not the direction to go. I need to look into the >> > KIP >> > >> in >> > >>>> more detail but for me the attractiveness of adding new Samza >> > >>>> consumer/producers -- even if yes all they were doing was really >> > getting >> > >>>> data into and out of Kafka -- was to avoid having to worry about >> the >> > >>>> lifecycle management of external clients. If there is a generic >> Kafka >> > >>>> ingress/egress layer that I can plug a new connector into and have >> a >> > >> lot of >> > >>>> the heavy lifting re scale and reliability done for me then it >> gives >> > me >> > >> all >> > >>>> the pushing new consumers/producers would. If not then it >> complicates >> > my >> > >>>> operational deployments. >> > >>>> >> > >>>> Which is similar to my other question with the proposal -- if we >> > build a >> > >>>> fully available/stand-alone Samza plus the requisite shims to >> > integrate >> > >>>> with Slider etc I suspect the former may be a lot more work than we >> > >> think. >> > >>>> We may make it much easier for a newcomer to get something running >> but >> > >>>> having them step up and get a reliable production deployment may >> still >> > >>>> dominate mailing list traffic, if for different reasons than >> today. >> > >>>> >> > >>>> Don't get me wrong -- I'm comfortable with making the Samza >> dependency >> > >> on >> > >>>> Kafka much more explicit and I absolutely see the benefits in the >> > >>>> reduction of duplication and clashing terminologies/abstractions >> that >> > >>>> Chris/Jay describe. Samza as a library would likely be a very nice >> > tool >> > >> to >> > >>>> add to the Kafka ecosystem. I just have the concerns above re the >> > >>>> operational side. >> > >>>> >> > >>>> Garry >> > >>>> >> > >>>> -----Original Message----- >> > >>>> From: Gianmarco De Francisci Morales [mailto:g...@apache.org] >> > >>>> Sent: 02 July 2015 12:56 >> > >>>> To: dev@samza.apache.org >> > >>>> Subject: Re: Thoughts and obesrvations on Samza >> > >>>> >> > >>>> Very interesting thoughts. >> > >>>> From outside, I have always perceived Samza as a computing layer >> over >> > >>>> Kafka. >> > >>>> >> > >>>> The question, maybe a bit provocative, is "should Samza be a >> > sub-project >> > >>>> of Kafka then?" >> > >>>> Or does it make sense to keep it as a separate project with a >> separate >> > >>>> governance? >> > >>>> >> > >>>> Cheers, >> > >>>> >> > >>>> -- >> > >>>> Gianmarco >> > >>>> >> > >>>> On 2 July 2015 at 08:59, Yan Fang <yanfang...@gmail.com> wrote: >> > >>>> >> > >>>>> Overall, I agree to couple with Kafka more tightly. Because Samza >> de >> > >>>>> facto is based on Kafka, and it should leverage what Kafka has. At >> > the >> > >>>>> same time, Kafka does not need to reinvent what Samza already >> has. I >> > >>>>> also like the idea of separating the ingestion and transformation. >> > >>>>> >> > >>>>> But it is a little difficult for me to image how the Samza will >> look >> > >>>> like. >> > >>>>> And I feel Chris and Jay have a little difference in terms of how >> > >>>>> Samza should look like. >> > >>>>> >> > >>>>> *** Will it look like what Jay's code shows (A client of Kakfa) ? >> And >> > >>>>> user's application code calls this client? >> > >>>>> >> > >>>>> 1. If we make Samza be a library of Kafka (like what the code >> shows), >> > >>>>> how do we implement auto-balance and fault-tolerance? Are they >> taken >> > >>>>> care by the Kafka broker or other mechanism, such as "Samza >> worker" >> > >>>>> (just make up the name) ? >> > >>>>> >> > >>>>> 2. What about other features, such as auto-scaling, shared state, >> > >>>>> monitoring? >> > >>>>> >> > >>>>> >> > >>>>> *** If we have Samza standalone, (is this what Chris suggests?) >> > >>>>> >> > >>>>> 1. we still need to ingest data from Kakfa and produce to it. >> Then it >> > >>>>> becomes the same as what Samza looks like now, except it does not >> > rely >> > >>>>> on Yarn anymore. >> > >>>>> >> > >>>>> 2. if it is standalone, how can it leverage Kafka's metrics, logs, >> > >>>>> etc? Use Kafka code as the dependency? >> > >>>>> >> > >>>>> >> > >>>>> Thanks, >> > >>>>> >> > >>>>> Fang, Yan >> > >>>>> yanfang...@gmail.com >> > >>>>> >> > >>>>> On Wed, Jul 1, 2015 at 5:46 PM, Guozhang Wang <wangg...@gmail.com >> > >> > >>>> wrote: >> > >>>>> >> > >>>>>> Read through the code example and it looks good to me. A few >> > >>>>>> thoughts regarding deployment: >> > >>>>>> >> > >>>>>> Today Samza deploys as executable runnable like: >> > >>>>>> >> > >>>>>> deploy/samza/bin/run-job.sh --config-factory=... >> > >>>> --config-path=file://... >> > >>>>>> >> > >>>>>> And this proposal advocate for deploying Samza more as embedded >> > >>>>>> libraries in user application code (ignoring the terminology >> since >> > >>>>>> it is not the >> > >>>>> same >> > >>>>>> as the prototype code): >> > >>>>>> >> > >>>>>> StreamTask task = new MyStreamTask(configs); Thread thread = new >> > >>>>>> Thread(task); thread.start(); >> > >>>>>> >> > >>>>>> I think both of these deployment modes are important for >> different >> > >>>>>> types >> > >>>>> of >> > >>>>>> users. That said, I think making Samza purely standalone is still >> > >>>>>> sufficient for either runnable or library modes. >> > >>>>>> >> > >>>>>> Guozhang >> > >>>>>> >> > >>>>>> On Tue, Jun 30, 2015 at 11:33 PM, Jay Kreps <j...@confluent.io> >> > wrote: >> > >>>>>> >> > >>>>>>> Looks like gmail mangled the code example, it was supposed to >> look >> > >>>>>>> like >> > >>>>>>> this: >> > >>>>>>> >> > >>>>>>> Properties props = new Properties(); >> > >>>>>>> props.put("bootstrap.servers", "localhost:4242"); >> StreamingConfig >> > >>>>>>> config = new StreamingConfig(props); >> > >>>>>>> config.subscribe("test-topic-1", "test-topic-2"); >> > >>>>>>> config.processor(ExampleStreamProcessor.class); >> > >>>>>>> config.serialization(new StringSerializer(), new >> > >>>>>>> StringDeserializer()); KafkaStreaming container = new >> > >>>>>>> KafkaStreaming(config); container.run(); >> > >>>>>>> >> > >>>>>>> -Jay >> > >>>>>>> >> > >>>>>>> On Tue, Jun 30, 2015 at 11:32 PM, Jay Kreps <j...@confluent.io> >> > >>>> wrote: >> > >>>>>>> >> > >>>>>>>> Hey guys, >> > >>>>>>>> >> > >>>>>>>> This came out of some conversations Chris and I were having >> > >>>>>>>> around >> > >>>>>>> whether >> > >>>>>>>> it would make sense to use Samza as a kind of data ingestion >> > >>>>> framework >> > >>>>>>> for >> > >>>>>>>> Kafka (which ultimately lead to KIP-26 "copycat"). This kind of >> > >>>>>> combined >> > >>>>>>>> with complaints around config and YARN and the discussion >> around >> > >>>>>>>> how >> > >>>>> to >> > >>>>>>>> best do a standalone mode. >> > >>>>>>>> >> > >>>>>>>> So the thought experiment was, given that Samza was basically >> > >>>>>>>> already totally Kafka specific, what if you just embraced that >> > >>>>>>>> and turned it >> > >>>>>> into >> > >>>>>>>> something less like a heavyweight framework and more like a >> > >>>>>>>> third >> > >>>>> Kafka >> > >>>>>>>> client--a kind of "producing consumer" with state management >> > >>>>>> facilities. >> > >>>>>>>> Basically a library. Instead of a complex stream processing >> > >>>>>>>> framework >> > >>>>>>> this >> > >>>>>>>> would actually be a very simple thing, not much more >> complicated >> > >>>>>>>> to >> > >>>>> use >> > >>>>>>> or >> > >>>>>>>> operate than a Kafka consumer. As Chris said we thought about >> it >> > >>>>>>>> a >> > >>>>> lot >> > >>>>>> of >> > >>>>>>>> what Samza (and the other stream processing systems were doing) >> > >>>>> seemed >> > >>>>>>> like >> > >>>>>>>> kind of a hangover from MapReduce. >> > >>>>>>>> >> > >>>>>>>> Of course you need to ingest/output data to and from the stream >> > >>>>>>>> processing. But when we actually looked into how that would >> > >>>>>>>> work, >> > >>>>> Samza >> > >>>>>>>> isn't really an ideal data ingestion framework for a bunch of >> > >>>>> reasons. >> > >>>>>> To >> > >>>>>>>> really do that right you need a pretty different internal data >> > >>>>>>>> model >> > >>>>>> and >> > >>>>>>>> set of apis. So what if you split them and had an api for Kafka >> > >>>>>>>> ingress/egress (copycat AKA KIP-26) and a separate api for >> Kafka >> > >>>>>>>> transformation (Samza). >> > >>>>>>>> >> > >>>>>>>> This would also allow really embracing the same terminology and >> > >>>>>>>> conventions. One complaint about the current state is that the >> > >>>>>>>> two >> > >>>>>>> systems >> > >>>>>>>> kind of feel bolted on. Terminology like "stream" vs "topic" >> and >> > >>>>>>> different >> > >>>>>>>> config and monitoring systems means you kind of have to learn >> > >>>>>>>> Kafka's >> > >>>>>>> way, >> > >>>>>>>> then learn Samza's slightly different way, then kind of >> > >>>>>>>> understand >> > >>>>> how >> > >>>>>>> they >> > >>>>>>>> map to each other, which having walked a few people through >> this >> > >>>>>>>> is surprisingly tricky for folks to get. >> > >>>>>>>> >> > >>>>>>>> Since I have been spending a lot of time on airplanes I hacked >> > >>>>>>>> up an ernest but still somewhat incomplete prototype of what >> > >>>>>>>> this would >> > >>>>> look >> > >>>>>>>> like. This is just unceremoniously dumped into Kafka as it >> > >>>>>>>> required a >> > >>>>>> few >> > >>>>>>>> changes to the new consumer. Here is the code: >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>> >> > >>>>>> >> > >>>>> >> > https://github.com/jkreps/kafka/tree/streams/clients/src/main/java/org >> > >>>>> /apache/kafka/clients/streaming >> > >>>>>>>> >> > >>>>>>>> For the purpose of the prototype I just liberally renamed >> > >>>>>>>> everything >> > >>>>> to >> > >>>>>>>> try to align it with Kafka with no regard for compatibility. >> > >>>>>>>> >> > >>>>>>>> To use this would be something like this: >> > >>>>>>>> Properties props = new Properties(); >> > >>>>>>>> props.put("bootstrap.servers", "localhost:4242"); >> > >>>>>>>> StreamingConfig config = new >> > >>>>> StreamingConfig(props); >> > >>>>>>> config.subscribe("test-topic-1", >> > >>>>>>>> "test-topic-2"); >> config.processor(ExampleStreamProcessor.class); >> > >>>>>>> config.serialization(new >> > >>>>>>>> StringSerializer(), new StringDeserializer()); KafkaStreaming >> > >>>>>> container = >> > >>>>>>>> new KafkaStreaming(config); container.run(); >> > >>>>>>>> >> > >>>>>>>> KafkaStreaming is basically the SamzaContainer; StreamProcessor >> > >>>>>>>> is basically StreamTask. >> > >>>>>>>> >> > >>>>>>>> So rather than putting all the class names in a file and then >> > >>>>>>>> having >> > >>>>>> the >> > >>>>>>>> job assembled by reflection, you just instantiate the container >> > >>>>>>>> programmatically. Work is balanced over however many instances >> > >>>>>>>> of >> > >>>>> this >> > >>>>>>> are >> > >>>>>>>> alive at any time (i.e. if an instance dies, new tasks are >> added >> > >>>>>>>> to >> > >>>>> the >> > >>>>>>>> existing containers without shutting them down). >> > >>>>>>>> >> > >>>>>>>> We would provide some glue for running this stuff in YARN via >> > >>>>>>>> Slider, Mesos via Marathon, and AWS using some of their tools >> > >>>>>>>> but from the >> > >>>>>> point >> > >>>>>>> of >> > >>>>>>>> view of these frameworks these stream processing jobs are just >> > >>>>>> stateless >> > >>>>>>>> services that can come and go and expand and contract at will. >> > >>>>>>>> There >> > >>>>> is >> > >>>>>>> no >> > >>>>>>>> more custom scheduler. >> > >>>>>>>> >> > >>>>>>>> Here are some relevant details: >> > >>>>>>>> >> > >>>>>>>> 1. It is only ~1300 lines of code, it would get larger if we >> > >>>>>>>> productionized but not vastly larger. We really do get a ton >> > >>>>>>>> of >> > >>>>>>> leverage >> > >>>>>>>> out of Kafka. >> > >>>>>>>> 2. Partition management is fully delegated to the new >> consumer. >> > >>>>> This >> > >>>>>>>> is nice since now any partition management strategy available >> > >>>>>>>> to >> > >>>>>> Kafka >> > >>>>>>>> consumer is also available to Samza (and vice versa) and with >> > >>>>>>>> the >> > >>>>>>> exact >> > >>>>>>>> same configs. >> > >>>>>>>> 3. It supports state as well as state reuse >> > >>>>>>>> >> > >>>>>>>> Anyhow take a look, hopefully it is thought provoking. >> > >>>>>>>> >> > >>>>>>>> -Jay >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>>> On Tue, Jun 30, 2015 at 6:55 PM, Chris Riccomini < >> > >>>>>> criccom...@apache.org> >> > >>>>>>>> wrote: >> > >>>>>>>> >> > >>>>>>>>> Hey all, >> > >>>>>>>>> >> > >>>>>>>>> I have had some discussions with Samza engineers at LinkedIn >> > >>>>>>>>> and >> > >>>>>>> Confluent >> > >>>>>>>>> and we came up with a few observations and would like to >> > >>>>>>>>> propose >> > >>>>> some >> > >>>>>>>>> changes. >> > >>>>>>>>> >> > >>>>>>>>> We've observed some things that I want to call out about >> > >>>>>>>>> Samza's >> > >>>>>> design, >> > >>>>>>>>> and I'd like to propose some changes. >> > >>>>>>>>> >> > >>>>>>>>> * Samza is dependent upon a dynamic deployment system. >> > >>>>>>>>> * Samza is too pluggable. >> > >>>>>>>>> * Samza's SystemConsumer/SystemProducer and Kafka's consumer >> > >>>>>>>>> APIs >> > >>>>> are >> > >>>>>>>>> trying to solve a lot of the same problems. >> > >>>>>>>>> >> > >>>>>>>>> All three of these issues are related, but I'll address them >> in >> > >>>>> order. >> > >>>>>>>>> >> > >>>>>>>>> Deployment >> > >>>>>>>>> >> > >>>>>>>>> Samza strongly depends on the use of a dynamic deployment >> > >>>>>>>>> scheduler >> > >>>>>> such >> > >>>>>>>>> as >> > >>>>>>>>> YARN, Mesos, etc. When we initially built Samza, we bet that >> > >>>>>>>>> there >> > >>>>>> would >> > >>>>>>>>> be >> > >>>>>>>>> one or two winners in this area, and we could support them, >> and >> > >>>>>>>>> the >> > >>>>>> rest >> > >>>>>>>>> would go away. In reality, there are many variations. >> > >>>>>>>>> Furthermore, >> > >>>>>> many >> > >>>>>>>>> people still prefer to just start their processors like normal >> > >>>>>>>>> Java processes, and use traditional deployment scripts such as >> > >>>>>>>>> Fabric, >> > >>>>>> Chef, >> > >>>>>>>>> Ansible, etc. Forcing a deployment system on users makes the >> > >>>>>>>>> Samza start-up process really painful for first time users. >> > >>>>>>>>> >> > >>>>>>>>> Dynamic deployment as a requirement was also a bit of a >> > >>>>>>>>> mis-fire >> > >>>>>> because >> > >>>>>>>>> of >> > >>>>>>>>> a fundamental misunderstanding between the nature of batch >> jobs >> > >>>>>>>>> and >> > >>>>>>> stream >> > >>>>>>>>> processing jobs. Early on, we made conscious effort to favor >> > >>>>>>>>> the >> > >>>>>> Hadoop >> > >>>>>>>>> (Map/Reduce) way of doing things, since it worked and was well >> > >>>>>>> understood. >> > >>>>>>>>> One thing that we missed was that batch jobs have a definite >> > >>>>>> beginning, >> > >>>>>>>>> and >> > >>>>>>>>> end, and stream processing jobs don't (usually). This leads to >> > >>>>>>>>> a >> > >>>>> much >> > >>>>>>>>> simpler scheduling problem for stream processors. You >> basically >> > >>>>>>>>> just >> > >>>>>>> need >> > >>>>>>>>> to find a place to start the processor, and start it. The way >> > >>>>>>>>> we run grids, at LinkedIn, there's no concept of a cluster >> > >>>>>>>>> being "full". We always >> > >>>>>> add >> > >>>>>>>>> more machines. The problem with coupling Samza with a >> scheduler >> > >>>>>>>>> is >> > >>>>>> that >> > >>>>>>>>> Samza (as a framework) now has to handle deployment. This >> pulls >> > >>>>>>>>> in a >> > >>>>>>> bunch >> > >>>>>>>>> of things such as configuration distribution (config stream), >> > >>>>>>>>> shell >> > >>>>>>> scrips >> > >>>>>>>>> (bin/run-job.sh, JobRunner), packaging (all the .tgz stuff), >> etc. >> > >>>>>>>>> >> > >>>>>>>>> Another reason for requiring dynamic deployment was to support >> > >>>>>>>>> data locality. If you want to have locality, you need to put >> > >>>>>>>>> your >> > >>>>>> processors >> > >>>>>>>>> close to the data they're processing. Upon further >> > >>>>>>>>> investigation, >> > >>>>>>> though, >> > >>>>>>>>> this feature is not that beneficial. There is some good >> > >>>>>>>>> discussion >> > >>>>>> about >> > >>>>>>>>> some problems with it on SAMZA-335. Again, we took the >> > >>>>>>>>> Map/Reduce >> > >>>>>> path, >> > >>>>>>>>> but >> > >>>>>>>>> there are some fundamental differences between HDFS and Kafka. >> > >>>>>>>>> HDFS >> > >>>>>> has >> > >>>>>>>>> blocks, while Kafka has partitions. This leads to less >> > >>>>>>>>> optimization potential with stream processors on top of Kafka. >> > >>>>>>>>> >> > >>>>>>>>> This feature is also used as a crutch. Samza doesn't have any >> > >>>>>>>>> built >> > >>>>> in >> > >>>>>>>>> fault-tolerance logic. Instead, it depends on the dynamic >> > >>>>>>>>> deployment scheduling system to handle restarts when a >> > >>>>>>>>> processor dies. This has >> > >>>>>>> made >> > >>>>>>>>> it very difficult to write a standalone Samza container >> > >>>> (SAMZA-516). >> > >>>>>>>>> >> > >>>>>>>>> Pluggability >> > >>>>>>>>> >> > >>>>>>>>> In some cases pluggability is good, but I think that we've >> gone >> > >>>>>>>>> too >> > >>>>>> far >> > >>>>>>>>> with it. Currently, Samza has: >> > >>>>>>>>> >> > >>>>>>>>> * Pluggable config. >> > >>>>>>>>> * Pluggable metrics. >> > >>>>>>>>> * Pluggable deployment systems. >> > >>>>>>>>> * Pluggable streaming systems (SystemConsumer, SystemProducer, >> > >>>> etc). >> > >>>>>>>>> * Pluggable serdes. >> > >>>>>>>>> * Pluggable storage engines. >> > >>>>>>>>> * Pluggable strategies for just about every component >> > >>>>> (MessageChooser, >> > >>>>>>>>> SystemStreamPartitionGrouper, ConfigRewriter, etc). >> > >>>>>>>>> >> > >>>>>>>>> There's probably more that I've forgotten, as well. Some of >> > >>>>>>>>> these >> > >>>>> are >> > >>>>>>>>> useful, but some have proven not to be. This all comes at a >> cost: >> > >>>>>>>>> complexity. This complexity is making it harder for our users >> > >>>>>>>>> to >> > >>>>> pick >> > >>>>>> up >> > >>>>>>>>> and use Samza out of the box. It also makes it difficult for >> > >>>>>>>>> Samza developers to reason about what the characteristics of >> > >>>>>>>>> the container (since the characteristics change depending on >> > >>>>>>>>> which plugins are use). >> > >>>>>>>>> >> > >>>>>>>>> The issues with pluggability are most visible in the System >> APIs. >> > >>>>> What >> > >>>>>>>>> Samza really requires to be functional is Kafka as its >> > >>>>>>>>> transport >> > >>>>>> layer. >> > >>>>>>>>> But >> > >>>>>>>>> we've conflated two unrelated use cases into one API: >> > >>>>>>>>> >> > >>>>>>>>> 1. Get data into/out of Kafka. >> > >>>>>>>>> 2. Process the data in Kafka. >> > >>>>>>>>> >> > >>>>>>>>> The current System API supports both of these use cases. The >> > >>>>>>>>> problem >> > >>>>>> is, >> > >>>>>>>>> we >> > >>>>>>>>> actually want different features for each use case. By >> papering >> > >>>>>>>>> over >> > >>>>>>> these >> > >>>>>>>>> two use cases, and providing a single API, we've introduced a >> > >>>>>>>>> ton of >> > >>>>>>> leaky >> > >>>>>>>>> abstractions. >> > >>>>>>>>> >> > >>>>>>>>> For example, what we'd really like in (2) is to have >> > >>>>>>>>> monotonically increasing longs for offsets (like Kafka). This >> > >>>>>>>>> would be at odds >> > >>>>> with >> > >>>>>>> (1), >> > >>>>>>>>> though, since different systems have different >> > >>>>>>> SCNs/Offsets/UUIDs/vectors. >> > >>>>>>>>> There was discussion both on the mailing list and the SQL >> JIRAs >> > >>>>> about >> > >>>>>>> the >> > >>>>>>>>> need for this. >> > >>>>>>>>> >> > >>>>>>>>> The same thing holds true for replayability. Kafka allows us >> to >> > >>>>> rewind >> > >>>>>>>>> when >> > >>>>>>>>> we have a failure. Many other systems don't. In some cases, >> > >>>>>>>>> systems >> > >>>>>>> return >> > >>>>>>>>> null for their offsets (e.g. WikipediaSystemConsumer) because >> > >>>>>>>>> they >> > >>>>>> have >> > >>>>>>> no >> > >>>>>>>>> offsets. >> > >>>>>>>>> >> > >>>>>>>>> Partitioning is another example. Kafka supports partitioning, >> > >>>>>>>>> but >> > >>>>> many >> > >>>>>>>>> systems don't. We model this by having a single partition for >> > >>>>>>>>> those systems. Still, other systems model partitioning >> > >>>> differently (e.g. >> > >>>>>>>>> Kinesis). >> > >>>>>>>>> >> > >>>>>>>>> The SystemAdmin interface is also a mess. Creating streams in >> a >> > >>>>>>>>> system-agnostic way is almost impossible. As is modeling >> > >>>>>>>>> metadata >> > >>>>> for >> > >>>>>>> the >> > >>>>>>>>> system (replication factor, partitions, location, etc). The >> > >>>>>>>>> list >> > >>>>> goes >> > >>>>>>> on. >> > >>>>>>>>> >> > >>>>>>>>> Duplicate work >> > >>>>>>>>> >> > >>>>>>>>> At the time that we began writing Samza, Kafka's consumer and >> > >>>>> producer >> > >>>>>>>>> APIs >> > >>>>>>>>> had a relatively weak feature set. On the consumer-side, you >> > >>>>>>>>> had two >> > >>>>>>>>> options: use the high level consumer, or the simple consumer. >> > >>>>>>>>> The >> > >>>>>>> problem >> > >>>>>>>>> with the high-level consumer was that it controlled your >> > >>>>>>>>> offsets, partition assignments, and the order in which you >> > >>>>>>>>> received messages. The >> > >>>>> problem >> > >>>>>>>>> with >> > >>>>>>>>> the simple consumer is that it's not simple. It's basic. You >> > >>>>>>>>> end up >> > >>>>>>> having >> > >>>>>>>>> to handle a lot of really low-level stuff that you shouldn't. >> > >>>>>>>>> We >> > >>>>>> spent a >> > >>>>>>>>> lot of time to make Samza's KafkaSystemConsumer very robust. >> It >> > >>>>>>>>> also allows us to support some cool features: >> > >>>>>>>>> >> > >>>>>>>>> * Per-partition message ordering and prioritization. >> > >>>>>>>>> * Tight control over partition assignment to support joins, >> > >>>>>>>>> global >> > >>>>>> state >> > >>>>>>>>> (if we want to implement it :)), etc. >> > >>>>>>>>> * Tight control over offset checkpointing. >> > >>>>>>>>> >> > >>>>>>>>> What we didn't realize at the time is that these features >> > >>>>>>>>> should >> > >>>>>>> actually >> > >>>>>>>>> be in Kafka. A lot of Kafka consumers (not just Samza stream >> > >>>>>> processors) >> > >>>>>>>>> end up wanting to do things like joins and partition >> > >>>>>>>>> assignment. The >> > >>>>>>> Kafka >> > >>>>>>>>> community has come to the same conclusion. They're adding a >> ton >> > >>>>>>>>> of upgrades into their new Kafka consumer implementation. To a >> > >>>>>>>>> large extent, >> > >>>>> it's >> > >>>>>>>>> duplicate work to what we've already done in Samza. >> > >>>>>>>>> >> > >>>>>>>>> On top of this, Kafka ended up taking a very similar approach >> > >>>>>>>>> to >> > >>>>>> Samza's >> > >>>>>>>>> KafkaCheckpointManager implementation for handling offset >> > >>>>>> checkpointing. >> > >>>>>>>>> Like Samza, Kafka's new offset management feature stores >> offset >> > >>>>>>>>> checkpoints in a topic, and allows you to fetch them from the >> > >>>>>>>>> broker. >> > >>>>>>>>> >> > >>>>>>>>> A lot of this seems like a waste, since we could have shared >> > >>>>>>>>> the >> > >>>>> work >> > >>>>>> if >> > >>>>>>>>> it >> > >>>>>>>>> had been done in Kafka from the get-go. >> > >>>>>>>>> >> > >>>>>>>>> Vision >> > >>>>>>>>> >> > >>>>>>>>> All of this leads me to a rather radical proposal. Samza is >> > >>>>> relatively >> > >>>>>>>>> stable at this point. I'd venture to say that we're near a 1.0 >> > >>>>>> release. >> > >>>>>>>>> I'd >> > >>>>>>>>> like to propose that we take what we've learned, and begin >> > >>>>>>>>> thinking >> > >>>>>>> about >> > >>>>>>>>> Samza beyond 1.0. What would we change if we were starting >> from >> > >>>>>> scratch? >> > >>>>>>>>> My >> > >>>>>>>>> proposal is to: >> > >>>>>>>>> >> > >>>>>>>>> 1. Make Samza standalone the *only* way to run Samza >> > >>>>>>>>> processors, and eliminate all direct dependences on YARN, >> Mesos, >> > >>>> etc. >> > >>>>>>>>> 2. Make a definitive call to support only Kafka as the stream >> > >>>>>> processing >> > >>>>>>>>> layer. >> > >>>>>>>>> 3. Eliminate Samza's metrics, logging, serialization, and >> > >>>>>>>>> config >> > >>>>>>> systems, >> > >>>>>>>>> and simply use Kafka's instead. >> > >>>>>>>>> >> > >>>>>>>>> This would fix all of the issues that I outlined above. It >> > >>>>>>>>> should >> > >>>>> also >> > >>>>>>>>> shrink the Samza code base pretty dramatically. Supporting >> only >> > >>>>>>>>> a standalone container will allow Samza to be executed on YARN >> > >>>>>>>>> (using Slider), Mesos (using Marathon/Aurora), or most other >> > >>>>>>>>> in-house >> > >>>>>>> deployment >> > >>>>>>>>> systems. This should make life a lot easier for new users. >> > >>>>>>>>> Imagine >> > >>>>>>> having >> > >>>>>>>>> the hello-samza tutorial without YARN. The drop in mailing >> list >> > >>>>>> traffic >> > >>>>>>>>> will be pretty dramatic. >> > >>>>>>>>> >> > >>>>>>>>> Coupling with Kafka seems long overdue to me. The reality is, >> > >>>>> everyone >> > >>>>>>>>> that >> > >>>>>>>>> I'm aware of is using Samza with Kafka. We basically require >> it >> > >>>>>> already >> > >>>>>>> in >> > >>>>>>>>> order for most features to work. Those that are using other >> > >>>>>>>>> systems >> > >>>>>> are >> > >>>>>>>>> generally using it for ingest into Kafka (1), and then they do >> > >>>>>>>>> the processing on top. There is already discussion ( >> > >>>>>>>>> >> > >>>>>>> >> > >>>>>> >> > >>>>> >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851 >> > >>>>> 767 >> > >>>>>>>>> ) >> > >>>>>>>>> in Kafka to make ingesting into Kafka extremely easy. >> > >>>>>>>>> >> > >>>>>>>>> Once we make the call to couple with Kafka, we can leverage a >> > >>>>>>>>> ton of >> > >>>>>>> their >> > >>>>>>>>> ecosystem. We no longer have to maintain our own config, >> > >>>>>>>>> metrics, >> > >>>>> etc. >> > >>>>>>> We >> > >>>>>>>>> can all share the same libraries, and make them better. This >> > >>>>>>>>> will >> > >>>>> also >> > >>>>>>>>> allow us to share the consumer/producer APIs, and will let us >> > >>>>> leverage >> > >>>>>>>>> their offset management and partition management, rather than >> > >>>>>>>>> having >> > >>>>>> our >> > >>>>>>>>> own. All of the coordinator stream code would go away, as >> would >> > >>>>>>>>> most >> > >>>>>> of >> > >>>>>>>>> the >> > >>>>>>>>> YARN AppMaster code. We'd probably have to push some partition >> > >>>>>>> management >> > >>>>>>>>> features into the Kafka broker, but they're already moving in >> > >>>>>>>>> that direction with the new consumer API. The features we have >> > >>>>>>>>> for >> > >>>>>> partition >> > >>>>>>>>> assignment aren't unique to Samza, and seem like they should >> be >> > >>>>>>>>> in >> > >>>>>> Kafka >> > >>>>>>>>> anyway. There will always be some niche usages which will >> > >>>>>>>>> require >> > >>>>>> extra >> > >>>>>>>>> care and hence full control over partition assignments much >> > >>>>>>>>> like the >> > >>>>>>> Kafka >> > >>>>>>>>> low level consumer api. These would continue to be supported. >> > >>>>>>>>> >> > >>>>>>>>> These items will be good for the Samza community. They'll make >> > >>>>>>>>> Samza easier to use, and make it easier for developers to add >> > >>>>>>>>> new features. >> > >>>>>>>>> >> > >>>>>>>>> Obviously this is a fairly large (and somewhat backwards >> > >>>>> incompatible >> > >>>>>>>>> change). If we choose to go this route, it's important that we >> > >>>>> openly >> > >>>>>>>>> communicate how we're going to provide a migration path from >> > >>>>>>>>> the >> > >>>>>>> existing >> > >>>>>>>>> APIs to the new ones (if we make incompatible changes). I >> think >> > >>>>>>>>> at a minimum, we'd probably need to provide a wrapper to allow >> > >>>>>>>>> existing StreamTask implementations to continue running on the >> > >>>> new container. >> > >>>>>>> It's >> > >>>>>>>>> also important that we openly communicate about timing, and >> > >>>>>>>>> stages >> > >>>>> of >> > >>>>>>> the >> > >>>>>>>>> migration. >> > >>>>>>>>> >> > >>>>>>>>> If you made it this far, I'm sure you have opinions. :) Please >> > >>>>>>>>> send >> > >>>>>> your >> > >>>>>>>>> thoughts and feedback. >> > >>>>>>>>> >> > >>>>>>>>> Cheers, >> > >>>>>>>>> Chris >> > >>>>>>>>> >> > >>>>>>>> >> > >>>>>>>> >> > >>>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> >> > >>>>>> -- >> > >>>>>> -- Guozhang >> > >>>>>> >> > >>>>> >> > >>>> >> > >> >> > >> >> > >> > >> > >> > >
