That was meant to be "thread" not "threat". lol. :) On Sun, Jul 12, 2015 at 5:54 PM, Chris Riccomini <criccom...@apache.org> wrote:
> Hey all, > > I want to start by saying that I'm absolutely thrilled to be a part of > this community. The amount of level-headed, thoughtful, educated discussion > that's gone on over the past ~10 days is overwhelming. Wonderful. > > It seems like discussion is waning a bit, and we've reached some > conclusions. There are several key emails in this threat, which I want to > call out: > > 1. Jakob's summary of the three potential ways forward. > > http://mail-archives.apache.org/mod_mbox/samza-dev/201507.mbox/%3CCADiKvVu-hxdBfyQ4qm3LDC55cUQbPdmbe4zGzTOOatYF1Pz43A%40mail.gmail.com%3E > 2. Julian's call out that we should be focusing on community over code. > > http://mail-archives.apache.org/mod_mbox/samza-dev/201507.mbox/%3CCAPSgeESZ_7bVFbwN%2Bzqi5MH%3D4CWu9MZUSanKg0-1woMqt55Fvg%40mail.gmail.com%3E > 3. Martin's summary about the benefits of merging communities. > > http://mail-archives.apache.org/mod_mbox/samza-dev/201507.mbox/%3CBFB866B6-D9D8-4578-93C0-FFAEB1DF00FC%40kleppmann.com%3E > 4. Jakob's comments about the distinction between community and code paths. > > http://mail-archives.apache.org/mod_mbox/samza-dev/201507.mbox/%3CCADiKvVtWPjHLLDsmxvz9KggVA5DfBi-nUvfqB6QdA-du%2B_a9Ng%40mail.gmail.com%3E > > I agree with the comments on all of these emails. I think Martin's summary > of his position aligns very closely with my own. To that end, I think we > should get concrete about what the proposal is, and call a vote on it. > Given that Jay, Martin, and I seem to be aligning fairly closely, I think > we should start with: > > 1. [community] Make Samza a subproject of Kafka. > 2. [community] Make all Samza PMC/committers committers of the subproject. > 3. [community] Migrate Samza's website/documentation into Kafka's. > 4. [code] Have the Samza community and the Kafka community start a > from-scratch reboot together in the new Kafka subproject. We can > borrow/copy & paste significant chunks of code from Samza's code base. > 5. [code] The subproject would intentionally eliminate support for both > other streaming systems and all deployment systems. > 6. [code] Attempt to provide a bridge from our SystemConsumer to KIP-26 > (copy cat) > 7. [code] Attempt to provide a bridge from the new subproject's processor > interface to our legacy StreamTask interface. > 8. [code/community] Sunset Samza as a TLP when we have a working Kafka > subproject that has a fault-tolerant container with state management. > > It's likely that (6) and (7) won't be fully drop-in. Still, the closer we > can get, the better it's going to be for our existing community. > > One thing that I didn't touch on with (2) is whether any Samza PMC members > should be rolled into Kafka PMC membership as well (though, Jay and Jakob > are already PMC members on both). I think that Samza's community deserves a > voice on the PMC, so I'd propose that we roll at least a few PMC members > into the Kafka PMC, but I don't have a strong framework for which people to > pick. > > Before (8), I think that Samza's TLP can continue to commit bug fixes and > patches as it sees fit, provided that we openly communicate that we won't > necessarily migrate new features to the new subproject, and that the TLP > will be shut down after the migration to the Kafka subproject occurs. > > Jakob, I could use your guidance here about about how to achieve this from > an Apache process perspective (sorry). > > * Should I just call a vote on this proposal? > * Should it happen on dev or private? > * Do committers have binding votes, or just PMC? > > Having trouble finding much detail on the Apache wikis. :( > > Cheers, > Chris > > On Fri, Jul 10, 2015 at 2:38 PM, Yan Fang <yanfang...@gmail.com> wrote: > >> Thanks, Jay. This argument persuaded me actually. :) >> >> Fang, Yan >> yanfang...@gmail.com >> >> On Fri, Jul 10, 2015 at 2:33 PM, Jay Kreps <j...@confluent.io> wrote: >> >> > Hey Yan, >> > >> > Yeah philosophically I think the argument is that you should capture the >> > stream in Kafka independent of the transformation. This is obviously a >> > Kafka-centric view point. >> > >> > Advantages of this: >> > - In practice I think this is what e.g. Storm people often end up doing >> > anyway. You usually need to throttle any access to a live serving >> database. >> > - Can have multiple subscribers and they get the same thing without >> > additional load on the source system. >> > - Applications can tap into the stream if need be by subscribing. >> > - You can debug your transformation by tailing the Kafka topic with the >> > console consumer >> > - Can tee off the same data stream for batch analysis or Lambda arch >> style >> > re-processing >> > >> > The disadvantage is that it will use Kafka resources. But the idea is >> > eventually you will have multiple subscribers to any data source (at >> least >> > for monitoring) so you will end up there soon enough anyway. >> > >> > Down the road the technical benefit is that I think it gives us a good >> path >> > towards end-to-end exactly once semantics from source to destination. >> > Basically the connectors need to support idempotence when talking to >> Kafka >> > and we need the transactional write feature in Kafka to make the >> > transformation atomic. This is actually pretty doable if you separate >> > connector=>kafka problem from the generic transformations which are >> always >> > kafka=>kafka. However I think it is quite impossible to do in a >> all_things >> > => all_things environment. Today you can say "well the semantics of the >> > Samza APIs depend on the connectors you use" but it is actually worse >> then >> > that because the semantics actually depend on the pairing of >> connectors--so >> > not only can you probably not get a usable "exactly once" guarantee >> > end-to-end it can actually be quite hard to reverse engineer what >> property >> > (if any) your end-to-end flow has if you have heterogenous systems. >> > >> > -Jay >> > >> > On Fri, Jul 10, 2015 at 2:00 PM, Yan Fang <yanfang...@gmail.com> wrote: >> > >> > > {quote} >> > > maintained in a separate repository and retaining the existing >> > > committership but sharing as much else as possible (website, etc) >> > > {quote} >> > > >> > > Overall, I agree on this idea. Now the question is more about "how to >> do >> > > it". >> > > >> > > On the other hand, one thing I want to point out is that, if we >> decide to >> > > go this way, how do we want to support >> > > otherSystem-transformation-otherSystem use case? >> > > >> > > Basically, there are four user groups here: >> > > >> > > 1. Kafka-transformation-Kafka >> > > 2. Kafka-transformation-otherSystem >> > > 3. otherSystem-transformation-Kafka >> > > 4. otherSystem-transformation-otherSystem >> > > >> > > For group 1, they can easily use the new Samza library to achieve. For >> > > group 2 and 3, they can use copyCat -> transformation -> Kafka or >> Kafka-> >> > > transformation -> copyCat. >> > > >> > > The problem is for group 4. Do we want to abandon this or still >> support >> > it? >> > > Of course, this use case can be achieved by using copyCat -> >> > transformation >> > > -> Kafka -> transformation -> copyCat, the thing is how we persuade >> them >> > to >> > > do this long chain. If yes, it will also be a win for Kafka too. Or if >> > > there is no one in this community actually doing this so far, maybe >> ok to >> > > not support the group 4 directly. >> > > >> > > Thanks, >> > > >> > > Fang, Yan >> > > yanfang...@gmail.com >> > > >> > > On Fri, Jul 10, 2015 at 12:58 PM, Jay Kreps <j...@confluent.io> wrote: >> > > >> > > > Yeah I agree with this summary. I think there are kind of two >> questions >> > > > here: >> > > > 1. Technically does alignment/reliance on Kafka make sense >> > > > 2. Branding wise (naming, website, concepts, etc) does alignment >> with >> > > Kafka >> > > > make sense >> > > > >> > > > Personally I do think both of these things would be really valuable, >> > and >> > > > would dramatically alter the trajectory of the project. >> > > > >> > > > My preference would be to see if people can mostly agree on a >> direction >> > > > rather than splintering things off. From my point of view the ideal >> > > outcome >> > > > of all the options discussed would be to make Samza a closely >> aligned >> > > > subproject, maintained in a separate repository and retaining the >> > > existing >> > > > committership but sharing as much else as possible (website, etc). >> No >> > > idea >> > > > about how these things work, Jacob, you probably know more. >> > > > >> > > > No discussion amongst the Kafka folks has happened on this, but >> likely >> > we >> > > > should figure out what the Samza community actually wants first. >> > > > >> > > > I admit that this is a fairly radical departure from how things are. >> > > > >> > > > If that doesn't fly, I think, yeah we could leave Samza as it is >> and do >> > > the >> > > > more radical reboot inside Kafka. From my point of view that does >> leave >> > > > things in a somewhat confusing state since now there are two stream >> > > > processing systems more or less coupled to Kafka in large part made >> by >> > > the >> > > > same people. But, arguably that might be a cleaner way to make the >> > > cut-over >> > > > and perhaps less risky for Samza community since if it works people >> can >> > > > switch and if it doesn't nothing will have changed. Dunno, how do >> > people >> > > > feel about this? >> > > > >> > > > -Jay >> > > > >> > > > On Fri, Jul 10, 2015 at 11:49 AM, Jakob Homan <jgho...@gmail.com> >> > wrote: >> > > > >> > > > > > This leads me to thinking that merging projects and communities >> > > might >> > > > > be a good idea: with the union of experience from both >> communities, >> > we >> > > > will >> > > > > probably build a better system that is better for users. >> > > > > Is this what's being proposed though? Merging the projects seems >> like >> > > > > a consequence of at most one of the three directions under >> > discussion: >> > > > > 1) Samza 2.0: The Samza community relies more heavily on Kafka for >> > > > > configuration, etc. (to a greater or lesser extent to be >> determined) >> > > > > but the Samza community would not automatically merge withe Kafka >> > > > > community (the Phoenix/HBase example is a good one here). >> > > > > 2) Samza Reboot: The Samza community continues to exist with a >> > limited >> > > > > project scope, but similarly would not need to be part of the >> Kafka >> > > > > community (ie given committership) to progress. Here, maybe the >> > Samza >> > > > > team would become a subproject of Kafka (the Board frowns on >> > > > > subprojects at the moment, so I'm not sure if that's even >> feasible), >> > > > > but that would not be required. >> > > > > 3) Hey Samza! FYI, Kafka does streaming now: In this option the >> Kafka >> > > > > team builds its own streaming library, possibly off of Jay's >> > > > > prototype, which has not direct lineage to the Samza team. >> There's >> > no >> > > > > reason for the Kafka team to bring in the Samza team. >> > > > > >> > > > > Is the Kafka community on board with this? >> > > > > >> > > > > To be clear, all three options under discussion are interesting, >> > > > > technically valid and likely healthy directions for the project. >> > > > > Also, they are not mutually exclusive. The Samza community could >> > > > > decide to pursue, say, 'Samza 2.0', while the Kafka community went >> > > > > forward with 'Hey Samza!' My points above are directed entirely >> at >> > > > > the community aspect of these choices. >> > > > > -Jakob >> > > > > >> > > > > On 10 July 2015 at 09:10, Roger Hoover <roger.hoo...@gmail.com> >> > wrote: >> > > > > > That's great. Thanks, Jay. >> > > > > > >> > > > > > On Fri, Jul 10, 2015 at 8:46 AM, Jay Kreps <j...@confluent.io> >> > wrote: >> > > > > > >> > > > > >> Yeah totally agree. I think you have this issue even today, >> right? >> > > > I.e. >> > > > > if >> > > > > >> you need to make a simple config change and you're running in >> YARN >> > > > today >> > > > > >> you end up bouncing the job which then rebuilds state. I think >> the >> > > fix >> > > > > is >> > > > > >> exactly what you described which is to have a long timeout on >> > > > partition >> > > > > >> movement for stateful jobs so that if a job is just getting >> > bounced, >> > > > and >> > > > > >> the cluster manager (or admin) is smart enough to restart it on >> > the >> > > > same >> > > > > >> host when possible, it can optimistically reuse any existing >> state >> > > it >> > > > > finds >> > > > > >> on disk (if it is valid). >> > > > > >> >> > > > > >> So in this model the charter of the CM is to place processes as >> > > > > stickily as >> > > > > >> possible and to restart or re-place failed processes. The >> charter >> > of >> > > > the >> > > > > >> partition management system is to control the assignment of >> work >> > to >> > > > > these >> > > > > >> processes. The nice thing about this is that the work >> assignment, >> > > > > timeouts, >> > > > > >> behavior, configs, and code will all be the same across all >> > cluster >> > > > > >> managers. >> > > > > >> >> > > > > >> So I think that prototype would actually give you exactly what >> you >> > > > want >> > > > > >> today for any cluster manager (or manual placement + restart >> > script) >> > > > > that >> > > > > >> was sticky in terms of host placement since there is already a >> > > > > configurable >> > > > > >> partition movement timeout and task-by-task state reuse with a >> > check >> > > > on >> > > > > >> state validity. >> > > > > >> >> > > > > >> -Jay >> > > > > >> >> > > > > >> On Fri, Jul 10, 2015 at 8:34 AM, Roger Hoover < >> > > roger.hoo...@gmail.com >> > > > > >> > > > > >> wrote: >> > > > > >> >> > > > > >> > That would be great to let Kafka do as much heavy lifting as >> > > > possible >> > > > > and >> > > > > >> > make it easier for other languages to implement Samza apis. >> > > > > >> > >> > > > > >> > One thing to watch out for is the interplay between Kafka's >> > group >> > > > > >> > management and the external scheduler/process manager's fault >> > > > > tolerance. >> > > > > >> > If a container dies, the Kafka group membership protocol will >> > try >> > > to >> > > > > >> assign >> > > > > >> > it's tasks to other containers while at the same time the >> > process >> > > > > manager >> > > > > >> > is trying to relaunch the container. Without some >> consideration >> > > for >> > > > > this >> > > > > >> > (like a configurable amount of time to wait before Kafka >> alters >> > > the >> > > > > group >> > > > > >> > membership), there may be thrashing going on which is >> especially >> > > bad >> > > > > for >> > > > > >> > containers with large amounts of local state. >> > > > > >> > >> > > > > >> > Someone else pointed this out already but I thought it might >> be >> > > > worth >> > > > > >> > calling out again. >> > > > > >> > >> > > > > >> > Cheers, >> > > > > >> > >> > > > > >> > Roger >> > > > > >> > >> > > > > >> > >> > > > > >> > On Tue, Jul 7, 2015 at 11:35 AM, Jay Kreps <j...@confluent.io >> > >> > > > wrote: >> > > > > >> > >> > > > > >> > > Hey Roger, >> > > > > >> > > >> > > > > >> > > I couldn't agree more. We spent a bunch of time talking to >> > > people >> > > > > and >> > > > > >> > that >> > > > > >> > > is exactly the stuff we heard time and again. What makes it >> > > hard, >> > > > of >> > > > > >> > > course, is that there is some tension between compatibility >> > with >> > > > > what's >> > > > > >> > > there now and making things better for new users. >> > > > > >> > > >> > > > > >> > > I also strongly agree with the importance of multi-language >> > > > > support. We >> > > > > >> > are >> > > > > >> > > talking now about Java, but for application development use >> > > cases >> > > > > >> people >> > > > > >> > > want to work in whatever language they are using >> elsewhere. I >> > > > think >> > > > > >> > moving >> > > > > >> > > to a model where Kafka itself does the group membership, >> > > lifecycle >> > > > > >> > control, >> > > > > >> > > and partition assignment has the advantage of putting all >> that >> > > > > complex >> > > > > >> > > stuff behind a clean api that the clients are already >> going to >> > > be >> > > > > >> > > implementing for their consumer, so the added functionality >> > for >> > > > > stream >> > > > > >> > > processing beyond a consumer becomes very minor. >> > > > > >> > > >> > > > > >> > > -Jay >> > > > > >> > > >> > > > > >> > > On Tue, Jul 7, 2015 at 10:49 AM, Roger Hoover < >> > > > > roger.hoo...@gmail.com> >> > > > > >> > > wrote: >> > > > > >> > > >> > > > > >> > > > Metamorphosis...nice. :) >> > > > > >> > > > >> > > > > >> > > > This has been a great discussion. As a user of Samza >> who's >> > > > > recently >> > > > > >> > > > integrated it into a relatively large organization, I >> just >> > > want >> > > > to >> > > > > >> add >> > > > > >> > > > support to a few points already made. >> > > > > >> > > > >> > > > > >> > > > The biggest hurdles to adoption of Samza as it currently >> > > exists >> > > > > that >> > > > > >> > I've >> > > > > >> > > > experienced are: >> > > > > >> > > > 1) YARN - YARN is overly complex in many environments >> where >> > > > Puppet >> > > > > >> > would >> > > > > >> > > do >> > > > > >> > > > just fine but it was the only mechanism to get fault >> > > tolerance. >> > > > > >> > > > 2) Configuration - I think I like the idea of configuring >> > most >> > > > of >> > > > > the >> > > > > >> > job >> > > > > >> > > > in code rather than config files. In general, I think >> the >> > > goal >> > > > > >> should >> > > > > >> > be >> > > > > >> > > > to make it harder to make mistakes, especially of the >> kind >> > > where >> > > > > the >> > > > > >> > code >> > > > > >> > > > expects something and the config doesn't match. The >> current >> > > > > config >> > > > > >> is >> > > > > >> > > > quite intricate and error-prone. For example, the >> > application >> > > > > logic >> > > > > >> > may >> > > > > >> > > > depend on bootstrapping a topic but rather than asserting >> > that >> > > > in >> > > > > the >> > > > > >> > > code, >> > > > > >> > > > you have to rely on getting the config right. Likewise >> with >> > > > > serdes, >> > > > > >> > the >> > > > > >> > > > Java representations produced by various serdes (JSON, >> Avro, >> > > > etc.) >> > > > > >> are >> > > > > >> > > not >> > > > > >> > > > equivalent so you cannot just reconfigure a serde without >> > > > changing >> > > > > >> the >> > > > > >> > > > code. It would be nice for jobs to be able to assert >> what >> > > they >> > > > > >> expect >> > > > > >> > > > from their input topics in terms of partitioning. This >> is >> > > > > getting a >> > > > > >> > > little >> > > > > >> > > > off topic but I was even thinking about creating a "Samza >> > > config >> > > > > >> > linter" >> > > > > >> > > > that would sanity check a set of configs. Especially in >> > > > > >> organizations >> > > > > >> > > > where config is managed by a different team than the >> > > application >> > > > > >> > > developer, >> > > > > >> > > > it's very hard to get avoid config mistakes. >> > > > > >> > > > 3) Java/Scala centric - for many teams (especially >> > DevOps-type >> > > > > >> folks), >> > > > > >> > > the >> > > > > >> > > > pain of the Java toolchain (maven, slow builds, weak >> command >> > > > line >> > > > > >> > > support, >> > > > > >> > > > configuration over convention) really inhibits >> productivity. >> > > As >> > > > > more >> > > > > >> > and >> > > > > >> > > > more high-quality clients become available for Kafka, I >> hope >> > > > > they'll >> > > > > >> > > follow >> > > > > >> > > > Samza's model. Not sure how much it affects the >> proposals >> > in >> > > > this >> > > > > >> > thread >> > > > > >> > > > but please consider other languages in the ecosystem as >> > well. >> > > > > From >> > > > > >> > what >> > > > > >> > > > I've heard, Spark has more Python users than Java/Scala. >> > > > > >> > > > (FYI, we added a Jython wrapper for the Samza API >> > > > > >> > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> https://github.com/Quantiply/rico/tree/master/jython/src/main/java/com/quantiply/samza >> > > > > >> > > > and are working on a Yeoman generator >> > > > > >> > > > https://github.com/Quantiply/generator-rico for >> > Jython/Samza >> > > > > >> projects >> > > > > >> > to >> > > > > >> > > > alleviate some of the pain) >> > > > > >> > > > >> > > > > >> > > > I also want to underscore Jay's point about improving the >> > user >> > > > > >> > > experience. >> > > > > >> > > > That's a very important factor for adoption. I think the >> > goal >> > > > > should >> > > > > >> > be >> > > > > >> > > to >> > > > > >> > > > make Samza as easy to get started with as something like >> > > > Logstash. >> > > > > >> > > > Logstash is vastly inferior in terms of capabilities to >> > Samza >> > > > but >> > > > > >> it's >> > > > > >> > > easy >> > > > > >> > > > to get started and that makes a big difference. >> > > > > >> > > > >> > > > > >> > > > Cheers, >> > > > > >> > > > >> > > > > >> > > > Roger >> > > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > >> > > > > >> > > > On Tue, Jul 7, 2015 at 3:29 AM, Gianmarco De Francisci >> > > Morales < >> > > > > >> > > > g...@apache.org> wrote: >> > > > > >> > > > >> > > > > >> > > > > Forgot to add. On the naming issues, Kafka >> Metamorphosis >> > is >> > > a >> > > > > clear >> > > > > >> > > > winner >> > > > > >> > > > > :) >> > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > >> > > > > Gianmarco >> > > > > >> > > > > >> > > > > >> > > > > On 7 July 2015 at 13:26, Gianmarco De Francisci >> Morales < >> > > > > >> > > g...@apache.org >> > > > > >> > > > > >> > > > > >> > > > > wrote: >> > > > > >> > > > > >> > > > > >> > > > > > Hi, >> > > > > >> > > > > > >> > > > > >> > > > > > @Martin, thanks for you comments. >> > > > > >> > > > > > Maybe I'm missing some important point, but I think >> > > coupling >> > > > > the >> > > > > >> > > > releases >> > > > > >> > > > > > is actually a *good* thing. >> > > > > >> > > > > > To make an example, would it be better if the MR and >> > HDFS >> > > > > >> > components >> > > > > >> > > of >> > > > > >> > > > > > Hadoop had different release schedules? >> > > > > >> > > > > > >> > > > > >> > > > > > Actually, keeping the discussion in a single place >> would >> > > > make >> > > > > >> > > agreeing >> > > > > >> > > > on >> > > > > >> > > > > > releases (and backwards compatibility) much easier, >> as >> > > > > everybody >> > > > > >> > > would >> > > > > >> > > > be >> > > > > >> > > > > > responsible for the whole codebase. >> > > > > >> > > > > > >> > > > > >> > > > > > That said, I like the idea of absorbing samza-core >> as a >> > > > > >> > sub-project, >> > > > > >> > > > and >> > > > > >> > > > > > leave the fancy stuff separate. >> > > > > >> > > > > > It probably gives 90% of the benefits we have been >> > > > discussing >> > > > > >> here. >> > > > > >> > > > > > >> > > > > >> > > > > > Cheers, >> > > > > >> > > > > > >> > > > > >> > > > > > -- >> > > > > >> > > > > > Gianmarco >> > > > > >> > > > > > >> > > > > >> > > > > > On 7 July 2015 at 02:30, Jay Kreps < >> jay.kr...@gmail.com >> > > >> > > > > wrote: >> > > > > >> > > > > > >> > > > > >> > > > > >> Hey Martin, >> > > > > >> > > > > >> >> > > > > >> > > > > >> I agree coupling release schedules is a downside. >> > > > > >> > > > > >> >> > > > > >> > > > > >> Definitely we can try to solve some of the >> integration >> > > > > problems >> > > > > >> in >> > > > > >> > > > > >> Confluent Platform or in other distributions. But I >> > think >> > > > > this >> > > > > >> > ends >> > > > > >> > > up >> > > > > >> > > > > >> being really shallow. I guess I feel to really get a >> > good >> > > > > user >> > > > > >> > > > > experience >> > > > > >> > > > > >> the two systems have to kind of feel like part of >> the >> > > same >> > > > > thing >> > > > > >> > and >> > > > > >> > > > you >> > > > > >> > > > > >> can't really add that in later--you can put both in >> the >> > > > same >> > > > > >> > > > > downloadable >> > > > > >> > > > > >> tar file but it doesn't really give a very cohesive >> > > > feeling. >> > > > > I >> > > > > >> > agree >> > > > > >> > > > > that >> > > > > >> > > > > >> ultimately any of the project stuff is as much >> social >> > and >> > > > > naming >> > > > > >> > as >> > > > > >> > > > > >> anything else--theoretically two totally independent >> > > > projects >> > > > > >> > could >> > > > > >> > > > work >> > > > > >> > > > > >> to >> > > > > >> > > > > >> tightly align. In practice this seems to be quite >> > > difficult >> > > > > >> > though. >> > > > > >> > > > > >> >> > > > > >> > > > > >> For the frameworks--totally agree it would be good >> to >> > > > > maintain >> > > > > >> the >> > > > > >> > > > > >> framework support with the project. In some cases >> there >> > > may >> > > > > not >> > > > > >> be >> > > > > >> > > too >> > > > > >> > > > > >> much >> > > > > >> > > > > >> there since the integration gets lighter but I think >> > > > whatever >> > > > > >> > stubs >> > > > > >> > > > you >> > > > > >> > > > > >> need should be included. So no I definitely wasn't >> > trying >> > > > to >> > > > > >> imply >> > > > > >> > > > > >> dropping >> > > > > >> > > > > >> support for these frameworks, just making the >> > integration >> > > > > >> lighter >> > > > > >> > by >> > > > > >> > > > > >> separating process management from partition >> > management. >> > > > > >> > > > > >> >> > > > > >> > > > > >> You raise two good points we would have to figure >> out >> > if >> > > we >> > > > > went >> > > > > >> > > down >> > > > > >> > > > > the >> > > > > >> > > > > >> alignment path: >> > > > > >> > > > > >> 1. With respect to the name, yeah I think the first >> > > > question >> > > > > is >> > > > > >> > > > whether >> > > > > >> > > > > >> some "re-branding" would be worth it. If so then I >> > think >> > > we >> > > > > can >> > > > > >> > > have a >> > > > > >> > > > > big >> > > > > >> > > > > >> thread on the name. I'm definitely not set on Kafka >> > > > > Streaming or >> > > > > >> > > Kafka >> > > > > >> > > > > >> Streams I was just using them to be kind of >> > > illustrative. I >> > > > > >> agree >> > > > > >> > > with >> > > > > >> > > > > >> your >> > > > > >> > > > > >> critique of these names, though I think people would >> > get >> > > > the >> > > > > >> idea. >> > > > > >> > > > > >> 2. Yeah you also raise a good point about how to >> > "factor" >> > > > it. >> > > > > >> Here >> > > > > >> > > are >> > > > > >> > > > > the >> > > > > >> > > > > >> options I see (I could get enthusiastic about any of >> > > them): >> > > > > >> > > > > >> a. One repo for both Kafka and Samza >> > > > > >> > > > > >> b. Two repos, retaining the current seperation >> > > > > >> > > > > >> c. Two repos, the equivalent of samza-api and >> > > samza-core >> > > > > is >> > > > > >> > > > absorbed >> > > > > >> > > > > >> almost like a third client >> > > > > >> > > > > >> >> > > > > >> > > > > >> Cheers, >> > > > > >> > > > > >> >> > > > > >> > > > > >> -Jay >> > > > > >> > > > > >> >> > > > > >> > > > > >> On Mon, Jul 6, 2015 at 1:18 PM, Martin Kleppmann < >> > > > > >> > > > mar...@kleppmann.com> >> > > > > >> > > > > >> wrote: >> > > > > >> > > > > >> >> > > > > >> > > > > >> > Ok, thanks for the clarifications. Just a few >> > follow-up >> > > > > >> > comments. >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > - I see the appeal of merging with Kafka or >> becoming >> > a >> > > > > >> > subproject: >> > > > > >> > > > the >> > > > > >> > > > > >> > reasons you mention are good. The risk I see is >> that >> > > > > release >> > > > > >> > > > schedules >> > > > > >> > > > > >> > become coupled to each other, which can slow >> everyone >> > > > down, >> > > > > >> and >> > > > > >> > > > large >> > > > > >> > > > > >> > projects with many contributors are harder to >> manage. >> > > > > (Jakob, >> > > > > >> > can >> > > > > >> > > > you >> > > > > >> > > > > >> speak >> > > > > >> > > > > >> > from experience, having seen a wider range of >> Hadoop >> > > > > ecosystem >> > > > > >> > > > > >> projects?) >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > Some of the goals of a better unified developer >> > > > experience >> > > > > >> could >> > > > > >> > > > also >> > > > > >> > > > > be >> > > > > >> > > > > >> > solved by integrating Samza nicely into a Kafka >> > > > > distribution >> > > > > >> > (such >> > > > > >> > > > as >> > > > > >> > > > > >> > Confluent's). I'm not against merging projects if >> we >> > > > decide >> > > > > >> > that's >> > > > > >> > > > the >> > > > > >> > > > > >> way >> > > > > >> > > > > >> > to go, just pointing out the same goals can >> perhaps >> > > also >> > > > be >> > > > > >> > > achieved >> > > > > >> > > > > in >> > > > > >> > > > > >> > other ways. >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > - With regard to dropping the YARN dependency: are >> > you >> > > > > >> proposing >> > > > > >> > > > that >> > > > > >> > > > > >> > Samza doesn't give any help to people wanting to >> run >> > on >> > > > > >> > > > > >> YARN/Mesos/AWS/etc? >> > > > > >> > > > > >> > So the docs would basically have a link to Slider >> and >> > > > > nothing >> > > > > >> > > else? >> > > > > >> > > > Or >> > > > > >> > > > > >> > would we maintain integrations with a bunch of >> > popular >> > > > > >> > deployment >> > > > > >> > > > > >> methods >> > > > > >> > > > > >> > (e.g. the necessary glue and shell scripts to make >> > > Samza >> > > > > work >> > > > > >> > with >> > > > > >> > > > > >> Slider)? >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > I absolutely think it's a good idea to have the >> "as a >> > > > > library" >> > > > > >> > and >> > > > > >> > > > > "as a >> > > > > >> > > > > >> > process" (using Yi's taxonomy) options for people >> who >> > > > want >> > > > > >> them, >> > > > > >> > > > but I >> > > > > >> > > > > >> > think there should also be a low-friction path for >> > > common >> > > > > "as >> > > > > >> a >> > > > > >> > > > > service" >> > > > > >> > > > > >> > deployment methods, for which we probably need to >> > > > maintain >> > > > > >> > > > > integrations. >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > - Project naming: "Kafka Streams" seems odd to me, >> > > > because >> > > > > >> Kafka >> > > > > >> > > is >> > > > > >> > > > > all >> > > > > >> > > > > >> > about streams already. Perhaps "Kafka >> Transformers" >> > or >> > > > > "Kafka >> > > > > >> > > > Filters" >> > > > > >> > > > > >> > would be more apt? >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > One suggestion: perhaps the core of Samza (stream >> > > > > >> transformation >> > > > > >> > > > with >> > > > > >> > > > > >> > state management -- i.e. the "Samza as a library" >> > bit) >> > > > > could >> > > > > >> > > become >> > > > > >> > > > > >> part of >> > > > > >> > > > > >> > Kafka, while higher-level tools such as streaming >> SQL >> > > and >> > > > > >> > > > integrations >> > > > > >> > > > > >> with >> > > > > >> > > > > >> > deployment frameworks remain in a separate >> project? >> > In >> > > > > other >> > > > > >> > > words, >> > > > > >> > > > > >> Kafka >> > > > > >> > > > > >> > would absorb the proven, stable core of Samza, >> which >> > > > would >> > > > > >> > become >> > > > > >> > > > the >> > > > > >> > > > > >> > "third Kafka client" mentioned early in this >> thread. >> > > The >> > > > > Samza >> > > > > >> > > > project >> > > > > >> > > > > >> > would then target that third Kafka client as its >> base >> > > > API, >> > > > > and >> > > > > >> > the >> > > > > >> > > > > >> project >> > > > > >> > > > > >> > would be freed up to explore more experimental new >> > > > > horizons. >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > Martin >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > On 6 Jul 2015, at 18:51, Jay Kreps < >> > > jay.kr...@gmail.com> >> > > > > >> wrote: >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > > Hey Martin, >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > For the YARN/Mesos/etc decoupling I actually >> don't >> > > > think >> > > > > it >> > > > > >> > ties >> > > > > >> > > > our >> > > > > >> > > > > >> > hands >> > > > > >> > > > > >> > > at all, all it does is refactor things. The >> > division >> > > of >> > > > > >> > > > > >> responsibility is >> > > > > >> > > > > >> > > that Samza core is responsible for task >> lifecycle, >> > > > state, >> > > > > >> and >> > > > > >> > > > > >> partition >> > > > > >> > > > > >> > > management (using the Kafka co-ordinator) but >> it is >> > > NOT >> > > > > >> > > > responsible >> > > > > >> > > > > >> for >> > > > > >> > > > > >> > > packaging, configuration deployment or >> execution of >> > > > > >> processes. >> > > > > >> > > The >> > > > > >> > > > > >> > problem >> > > > > >> > > > > >> > > of packaging and starting these processes is >> > > > > >> > > > > >> > > framework/environment-specific. This leaves >> > > individual >> > > > > >> > > frameworks >> > > > > >> > > > to >> > > > > >> > > > > >> be >> > > > > >> > > > > >> > as >> > > > > >> > > > > >> > > fancy or vanilla as they like. So you can get >> > simple >> > > > > >> stateless >> > > > > >> > > > > >> support in >> > > > > >> > > > > >> > > YARN, Mesos, etc using their off-the-shelf app >> > > > framework >> > > > > >> > > (Slider, >> > > > > >> > > > > >> > Marathon, >> > > > > >> > > > > >> > > etc). These are well known by people and have >> nice >> > > UIs >> > > > > and a >> > > > > >> > lot >> > > > > >> > > > of >> > > > > >> > > > > >> > > flexibility. I don't think they have node >> affinity >> > > as a >> > > > > >> built >> > > > > >> > in >> > > > > >> > > > > >> option >> > > > > >> > > > > >> > > (though I could be wrong). So if we want that we >> > can >> > > > > either >> > > > > >> > wait >> > > > > >> > > > for >> > > > > >> > > > > >> them >> > > > > >> > > > > >> > > to add it or do a custom framework to add that >> > > feature >> > > > > (as >> > > > > >> > now). >> > > > > >> > > > > >> > Obviously >> > > > > >> > > > > >> > > if you manage things with old-school ops tools >> > > > > >> > (puppet/chef/etc) >> > > > > >> > > > you >> > > > > >> > > > > >> get >> > > > > >> > > > > >> > > locality easily. The nice thing, though, is that >> > all >> > > > the >> > > > > >> samza >> > > > > >> > > > > >> "business >> > > > > >> > > > > >> > > logic" around partition management and fault >> > > tolerance >> > > > > is in >> > > > > >> > > Samza >> > > > > >> > > > > >> core >> > > > > >> > > > > >> > so >> > > > > >> > > > > >> > > it is shared across frameworks and the framework >> > > > specific >> > > > > >> bit >> > > > > >> > is >> > > > > >> > > > > just >> > > > > >> > > > > >> > > whether it is smart enough to try to get the >> same >> > > host >> > > > > when >> > > > > >> a >> > > > > >> > > job >> > > > > >> > > > is >> > > > > >> > > > > >> > > restarted. >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > With respect to the Kafka-alignment, yeah I >> think >> > the >> > > > > goal >> > > > > >> > would >> > > > > >> > > > be >> > > > > >> > > > > >> (a) >> > > > > >> > > > > >> > > actually get better alignment in user >> experience, >> > and >> > > > (b) >> > > > > >> > > express >> > > > > >> > > > > >> this in >> > > > > >> > > > > >> > > the naming and project branding. Specifically: >> > > > > >> > > > > >> > > 1. Website/docs, it would be nice for the >> > > > > "transformation" >> > > > > >> api >> > > > > >> > > to >> > > > > >> > > > be >> > > > > >> > > > > >> > > discoverable in the main Kafka docs--i.e. be >> able >> > to >> > > > > explain >> > > > > >> > > when >> > > > > >> > > > to >> > > > > >> > > > > >> use >> > > > > >> > > > > >> > > the consumer and when to use the stream >> processing >> > > > > >> > functionality >> > > > > >> > > > and >> > > > > >> > > > > >> lead >> > > > > >> > > > > >> > > people into that experience. >> > > > > >> > > > > >> > > 2. Align releases so if you get Kafkza 1.4.2 (or >> > > > > whatever) >> > > > > >> > that >> > > > > >> > > > has >> > > > > >> > > > > >> both >> > > > > >> > > > > >> > > Kafka and the stream processing part and they >> > > actually >> > > > > work >> > > > > >> > > > > together. >> > > > > >> > > > > >> > > 3. Unify the programming experience so the >> client >> > and >> > > > > Samza >> > > > > >> > api >> > > > > >> > > > > share >> > > > > >> > > > > >> > > config/monitoring/naming/packaging/etc. >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > I think sub-projects keep separate committers >> and >> > can >> > > > > have a >> > > > > >> > > > > separate >> > > > > >> > > > > >> > repo, >> > > > > >> > > > > >> > > but I'm actually not really sure (I can't find a >> > > > > definition >> > > > > >> > of a >> > > > > >> > > > > >> > subproject >> > > > > >> > > > > >> > > in Apache). >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > Basically at a high-level you want the >> experience >> > to >> > > > > "feel" >> > > > > >> > > like a >> > > > > >> > > > > >> single >> > > > > >> > > > > >> > > system, not to relatively independent things >> that >> > are >> > > > > kind >> > > > > >> of >> > > > > >> > > > > >> awkwardly >> > > > > >> > > > > >> > > glued together. >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > I think if we did that they having naming or >> > branding >> > > > > like >> > > > > >> > > "kafka >> > > > > >> > > > > >> > > streaming" or "kafka streams" or something like >> > that >> > > > > would >> > > > > >> > > > actually >> > > > > >> > > > > >> do a >> > > > > >> > > > > >> > > good job of conveying what it is. I do that this >> > > would >> > > > > help >> > > > > >> > > > adoption >> > > > > >> > > > > >> > quite >> > > > > >> > > > > >> > > a lot as it would correctly convey that using >> Kafka >> > > > > >> Streaming >> > > > > >> > > with >> > > > > >> > > > > >> Kafka >> > > > > >> > > > > >> > is >> > > > > >> > > > > >> > > a fairly seamless experience and Kafka is pretty >> > > > heavily >> > > > > >> > adopted >> > > > > >> > > > at >> > > > > >> > > > > >> this >> > > > > >> > > > > >> > > point. >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > Fwiw we actually considered this model >> originally >> > > when >> > > > > open >> > > > > >> > > > sourcing >> > > > > >> > > > > >> > Samza, >> > > > > >> > > > > >> > > however at that time Kafka was relatively >> unknown >> > and >> > > > we >> > > > > >> > decided >> > > > > >> > > > not >> > > > > >> > > > > >> to >> > > > > >> > > > > >> > do >> > > > > >> > > > > >> > > it since we felt it would be limiting. From my >> > point >> > > of >> > > > > view >> > > > > >> > the >> > > > > >> > > > > three >> > > > > >> > > > > >> > > things have changed (1) Kafka is now really >> heavily >> > > > used >> > > > > for >> > > > > >> > > > stream >> > > > > >> > > > > >> > > processing, (2) we learned that abstracting out >> the >> > > > > stream >> > > > > >> > well >> > > > > >> > > is >> > > > > >> > > > > >> > > basically impossible, (3) we learned it is >> really >> > > hard >> > > > to >> > > > > >> keep >> > > > > >> > > the >> > > > > >> > > > > two >> > > > > >> > > > > >> > > things feeling like a single product. >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > -Jay >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > > On Mon, Jul 6, 2015 at 3:37 AM, Martin >> Kleppmann < >> > > > > >> > > > > >> mar...@kleppmann.com> >> > > > > >> > > > > >> > > wrote: >> > > > > >> > > > > >> > > >> > > > > >> > > > > >> > >> Hi all, >> > > > > >> > > > > >> > >> >> > > > > >> > > > > >> > >> Lots of good thoughts here. >> > > > > >> > > > > >> > >> >> > > > > >> > > > > >> > >> I agree with the general philosophy of tying >> Samza >> > > > more >> > > > > >> > firmly >> > > > > >> > > to >> > > > > >> > > > > >> Kafka. >> > > > > >> > > > > >> > >> After I spent a while looking at integrating >> other >> > > > > message >> > > > > >> > > > brokers >> > > > > >> > > > > >> (e.g. >> > > > > >> > > > > >> > >> Kinesis) with SystemConsumer, I came to the >> > > conclusion >> > > > > that >> > > > > >> > > > > >> > SystemConsumer >> > > > > >> > > > > >> > >> tacitly assumes a model so much like Kafka's >> that >> > > > pretty >> > > > > >> much >> > > > > >> > > > > nobody >> > > > > >> > > > > >> but >> > > > > >> > > > > >> > >> Kafka actually implements it. (Databus is >> perhaps >> > an >> > > > > >> > exception, >> > > > > >> > > > but >> > > > > >> > > > > >> it >> > > > > >> > > > > >> > >> isn't widely used outside of LinkedIn.) Thus, >> > making >> > > > > Samza >> > > > > >> > > fully >> > > > > >> > > > > >> > dependent >> > > > > >> > > > > >> > >> on Kafka acknowledges that the >> system-independence >> > > was >> > > > > >> never >> > > > > >> > as >> > > > > >> > > > > real >> > > > > >> > > > > >> as >> > > > > >> > > > > >> > we >> > > > > >> > > > > >> > >> perhaps made it out to be. The gains of code >> reuse >> > > are >> > > > > >> real. >> > > > > >> > > > > >> > >> >> > > > > >> > > > > >> > >> The idea of decoupling Samza from YARN has also >> > > always >> > > > > been >> > > > > >> > > > > >> appealing to >> > > > > >> > > > > >> > >> me, for various reasons already mentioned in >> this >> > > > > thread. >> > > > > >> > > > Although >> > > > > >> > > > > >> > making >> > > > > >> > > > > >> > >> Samza jobs deployable on anything >> > > (YARN/Mesos/AWS/etc) >> > > > > >> seems >> > > > > >> > > > > >> laudable, >> > > > > >> > > > > >> > I am >> > > > > >> > > > > >> > >> a little concerned that it will restrict us to >> a >> > > > lowest >> > > > > >> > common >> > > > > >> > > > > >> > denominator. >> > > > > >> > > > > >> > >> For example, would host affinity (SAMZA-617) >> still >> > > be >> > > > > >> > possible? >> > > > > >> > > > For >> > > > > >> > > > > >> jobs >> > > > > >> > > > > >> > >> with large amounts of state, I think SAMZA-617 >> > would >> > > > be >> > > > > a >> > > > > >> big >> > > > > >> > > > boon, >> > > > > >> > > > > >> > since >> > > > > >> > > > > >> > >> restoring state off the changelog on every >> single >> > > > > restart >> > > > > >> is >> > > > > >> > > > > painful, >> > > > > >> > > > > >> > due >> > > > > >> > > > > >> > >> to long recovery times. It would be a shame if >> the >> > > > > >> decoupling >> > > > > >> > > > from >> > > > > >> > > > > >> YARN >> > > > > >> > > > > >> > >> made host affinity impossible. >> > > > > >> > > > > >> > >> >> > > > > >> > > > > >> > >> Jay, a question about the proposed API for >> > > > > instantiating a >> > > > > >> > job >> > > > > >> > > in >> > > > > >> > > > > >> code >> > > > > >> > > > > >> > >> (rather than a properties file): when >> submitting a >> > > job >> > > > > to a >> > > > > >> > > > > cluster, >> > > > > >> > > > > >> is >> > > > > >> > > > > >> > the >> > > > > >> > > > > >> > >> idea that the instantiation code runs on a >> client >> > > > > >> somewhere, >> > > > > >> > > > which >> > > > > >> > > > > >> then >> > > > > >> > > > > >> > >> pokes the necessary endpoints on >> > YARN/Mesos/AWS/etc? >> > > > Or >> > > > > >> does >> > > > > >> > > that >> > > > > >> > > > > >> code >> > > > > >> > > > > >> > run >> > > > > >> > > > > >> > >> on each container that is part of the job (in >> > which >> > > > > case, >> > > > > >> how >> > > > > >> > > > does >> > > > > >> > > > > >> the >> > > > > >> > > > > >> > job >> > > > > >> > > > > >> > >> submission to the cluster work)? >> > > > > >> > > > > >> > >> >> > > > > >> > > > > >> > >> I agree with Garry that it doesn't feel right >> to >> > > make >> > > > a >> > > > > 1.0 >> > > > > >> > > > release >> > > > > >> > > > > >> > with a >> > > > > >> > > > > >> > >> plan for it to be immediately obsolete. So if >> this >> > > is >> > > > > going >> > > > > >> > to >> > > > > >> > > > > >> happen, I >> > > > > >> > > > > >> > >> think it would be more honest to stick with 0.* >> > > > version >> > > > > >> > numbers >> > > > > >> > > > > until >> > > > > >> > > > > >> > the >> > > > > >> > > > > >> > >> library-ified Samza has been implemented, is >> > stable >> > > > and >> > > > > >> > widely >> > > > > >> > > > > used. >> > > > > >> > > > > >> > >> >> > > > > >> > > > > >> > >> Should the new Samza be a subproject of Kafka? >> > There >> > > > is >> > > > > >> > > precedent >> > > > > >> > > > > for >> > > > > >> > > > > >> > >> tight coupling between different Apache >> projects >> > > (e.g. >> > > > > >> > Curator >> > > > > >> > > > and >> > > > > >> > > > > >> > >> Zookeeper, or Slider and YARN), so I think >> > remaining >> > > > > >> separate >> > > > > >> > > > would >> > > > > >> > > > > >> be >> > > > > >> > > > > >> > ok. >> > > > > >> > > > > >> > >> Even if Samza is fully dependent on Kafka, >> there >> > is >> > > > > enough >> > > > > >> > > > > substance >> > > > > >> > > > > >> in >> > > > > >> > > > > >> > >> Samza that it warrants being a separate >> project. >> > An >> > > > > >> argument >> > > > > >> > in >> > > > > >> > > > > >> favour >> > > > > >> > > > > >> > of >> > > > > >> > > > > >> > >> merging would be if we think Kafka has a much >> > > stronger >> > > > > >> "brand >> > > > > >> > > > > >> presence" >> > > > > >> > > > > >> > >> than Samza; I'm ambivalent on that one. If the >> > Kafka >> > > > > >> project >> > > > > >> > is >> > > > > >> > > > > >> willing >> > > > > >> > > > > >> > to >> > > > > >> > > > > >> > >> endorse Samza as the "official" way of doing >> > > stateful >> > > > > >> stream >> > > > > >> > > > > >> > >> transformations, that would probably have much >> the >> > > > same >> > > > > >> > effect >> > > > > >> > > as >> > > > > >> > > > > >> > >> re-branding Samza as "Kafka Stream Processors" >> or >> > > > > suchlike. >> > > > > >> > > Close >> > > > > >> > > > > >> > >> collaboration between the two projects will be >> > > needed >> > > > in >> > > > > >> any >> > > > > >> > > > case. >> > > > > >> > > > > >> > >> >> > > > > >> > > > > >> > >> From a project management perspective, I guess >> the >> > > > "new >> > > > > >> > Samza" >> > > > > >> > > > > would >> > > > > >> > > > > >> > have >> > > > > >> > > > > >> > >> to be developed on a branch alongside ongoing >> > > > > maintenance >> > > > > >> of >> > > > > >> > > the >> > > > > >> > > > > >> current >> > > > > >> > > > > >> > >> line of development? I think it would be >> important >> > > to >> > > > > >> > continue >> > > > > >> > > > > >> > supporting >> > > > > >> > > > > >> > >> existing users, and provide a graceful >> migration >> > > path >> > > > to >> > > > > >> the >> > > > > >> > > new >> > > > > >> > > > > >> > version. >> > > > > >> > > > > >> > >> Leaving the current versions unsupported and >> > forcing >> > > > > people >> > > > > >> > to >> > > > > >> > > > > >> rewrite >> > > > > >> > > > > >> > >> their jobs would send a bad signal. >> > > > > >> > > > > >> > >> >> > > > > >> > > > > >> > >> Best, >> > > > > >> > > > > >> > >> Martin >> > > > > >> > > > > >> > >> >> > > > > >> > > > > >> > >> On 2 Jul 2015, at 16:59, Jay Kreps < >> > > j...@confluent.io> >> > > > > >> wrote: >> > > > > >> > > > > >> > >> >> > > > > >> > > > > >> > >>> Hey Garry, >> > > > > >> > > > > >> > >>> >> > > > > >> > > > > >> > >>> Yeah that's super frustrating. I'd be happy to >> > chat >> > > > > more >> > > > > >> > about >> > > > > >> > > > > this >> > > > > >> > > > > >> if >> > > > > >> > > > > >> > >>> you'd be interested. I think Chris and I >> started >> > > with >> > > > > the >> > > > > >> > idea >> > > > > >> > > > of >> > > > > >> > > > > >> "what >> > > > > >> > > > > >> > >>> would it take to make Samza a kick-ass >> ingestion >> > > > tool" >> > > > > but >> > > > > >> > > > > >> ultimately >> > > > > >> > > > > >> > we >> > > > > >> > > > > >> > >>> kind of came around to the idea that ingestion >> > and >> > > > > >> > > > transformation >> > > > > >> > > > > >> had >> > > > > >> > > > > >> > >>> pretty different needs and coupling the two >> made >> > > > things >> > > > > >> > hard. >> > > > > >> > > > > >> > >>> >> > > > > >> > > > > >> > >>> For what it's worth I think copycat (KIP-26) >> > > actually >> > > > > will >> > > > > >> > do >> > > > > >> > > > what >> > > > > >> > > > > >> you >> > > > > >> > > > > >> > >> are >> > > > > >> > > > > >> > >>> looking for. >> > > > > >> > > > > >> > >>> >> > > > > >> > > > > >> > >>> With regard to your point about slider, I >> don't >> > > > > >> necessarily >> > > > > >> > > > > >> disagree. >> > > > > >> > > > > >> > >> But I >> > > > > >> > > > > >> > >>> think getting good YARN support is quite >> doable >> > > and I >> > > > > >> think >> > > > > >> > we >> > > > > >> > > > can >> > > > > >> > > > > >> make >> > > > > >> > > > > >> > >>> that work well. I think the issue this >> proposal >> > > > solves >> > > > > is >> > > > > >> > that >> > > > > >> > > > > >> > >> technically >> > > > > >> > > > > >> > >>> it is pretty hard to support multiple cluster >> > > > > management >> > > > > >> > > systems >> > > > > >> > > > > the >> > > > > >> > > > > >> > way >> > > > > >> > > > > >> > >>> things are now, you need to write an "app >> master" >> > > or >> > > > > >> > > "framework" >> > > > > >> > > > > for >> > > > > >> > > > > >> > each >> > > > > >> > > > > >> > >>> and they are all a little different so >> testing is >> > > > > really >> > > > > >> > hard. >> > > > > >> > > > In >> > > > > >> > > > > >> the >> > > > > >> > > > > >> > >>> absence of this we have been stuck with just >> YARN >> > > > which >> > > > > >> has >> > > > > >> > > > > >> fantastic >> > > > > >> > > > > >> > >>> penetration in the Hadoopy part of the org, >> but >> > > zero >> > > > > >> > > penetration >> > > > > >> > > > > >> > >> elsewhere. >> > > > > >> > > > > >> > >>> Given the huge amount of work being put in to >> > > slider, >> > > > > >> > > marathon, >> > > > > >> > > > > aws >> > > > > >> > > > > >> > >>> tooling, not to mention the umpteen related >> > > packaging >> > > > > >> > > > technologies >> > > > > >> > > > > >> > people >> > > > > >> > > > > >> > >>> want to use (Docker, Kubernetes, various >> > > > cloud-specific >> > > > > >> > deploy >> > > > > >> > > > > >> tools, >> > > > > >> > > > > >> > >> etc) >> > > > > >> > > > > >> > >>> I really think it is important to get this >> right. >> > > > > >> > > > > >> > >>> >> > > > > >> > > > > >> > >>> -Jay >> > > > > >> > > > > >> > >>> >> > > > > >> > > > > >> > >>> On Thu, Jul 2, 2015 at 4:17 AM, Garry >> Turkington >> > < >> > > > > >> > > > > >> > >>> g.turking...@improvedigital.com> wrote: >> > > > > >> > > > > >> > >>> >> > > > > >> > > > > >> > >>>> Hi all, >> > > > > >> > > > > >> > >>>> >> > > > > >> > > > > >> > >>>> I think the question below re does Samza >> become >> > a >> > > > > >> > sub-project >> > > > > >> > > > of >> > > > > >> > > > > >> Kafka >> > > > > >> > > > > >> > >>>> highlights the broader point around >> migration. >> > > Chris >> > > > > >> > mentions >> > > > > >> > > > > >> Samza's >> > > > > >> > > > > >> > >>>> maturity is heading towards a v1 release but >> I'm >> > > not >> > > > > sure >> > > > > >> > it >> > > > > >> > > > > feels >> > > > > >> > > > > >> > >> right to >> > > > > >> > > > > >> > >>>> launch a v1 then immediately plan to >> deprecate >> > > most >> > > > of >> > > > > >> it. >> > > > > >> > > > > >> > >>>> >> > > > > >> > > > > >> > >>>> From a selfish perspective I have some guys >> who >> > > have >> > > > > >> > started >> > > > > >> > > > > >> working >> > > > > >> > > > > >> > >> with >> > > > > >> > > > > >> > >>>> Samza and building some new >> consumers/producers >> > > was >> > > > > next >> > > > > >> > up. >> > > > > >> > > > > Sounds >> > > > > >> > > > > >> > like >> > > > > >> > > > > >> > >>>> that is absolutely not the direction to go. I >> > need >> > > > to >> > > > > >> look >> > > > > >> > > into >> > > > > >> > > > > the >> > > > > >> > > > > >> > KIP >> > > > > >> > > > > >> > >> in >> > > > > >> > > > > >> > >>>> more detail but for me the attractiveness of >> > > adding >> > > > > new >> > > > > >> > Samza >> > > > > >> > > > > >> > >>>> consumer/producers -- even if yes all they >> were >> > > > doing >> > > > > was >> > > > > >> > > > really >> > > > > >> > > > > >> > getting >> > > > > >> > > > > >> > >>>> data into and out of Kafka -- was to avoid >> > > having >> > > > to >> > > > > >> > worry >> > > > > >> > > > > about >> > > > > >> > > > > >> the >> > > > > >> > > > > >> > >>>> lifecycle management of external clients. If >> > there >> > > > is >> > > > > a >> > > > > >> > > generic >> > > > > >> > > > > >> Kafka >> > > > > >> > > > > >> > >>>> ingress/egress layer that I can plug a new >> > > connector >> > > > > into >> > > > > >> > and >> > > > > >> > > > > have >> > > > > >> > > > > >> a >> > > > > >> > > > > >> > >> lot of >> > > > > >> > > > > >> > >>>> the heavy lifting re scale and reliability >> done >> > > for >> > > > me >> > > > > >> then >> > > > > >> > > it >> > > > > >> > > > > >> gives >> > > > > >> > > > > >> > me >> > > > > >> > > > > >> > >> all >> > > > > >> > > > > >> > >>>> the pushing new consumers/producers would. If >> > not >> > > > > then it >> > > > > >> > > > > >> complicates >> > > > > >> > > > > >> > my >> > > > > >> > > > > >> > >>>> operational deployments. >> > > > > >> > > > > >> > >>>> >> > > > > >> > > > > >> > >>>> Which is similar to my other question with >> the >> > > > > proposal >> > > > > >> -- >> > > > > >> > if >> > > > > >> > > > we >> > > > > >> > > > > >> > build a >> > > > > >> > > > > >> > >>>> fully available/stand-alone Samza plus the >> > > requisite >> > > > > >> shims >> > > > > >> > to >> > > > > >> > > > > >> > integrate >> > > > > >> > > > > >> > >>>> with Slider etc I suspect the former may be a >> > lot >> > > > more >> > > > > >> work >> > > > > >> > > > than >> > > > > >> > > > > we >> > > > > >> > > > > >> > >> think. >> > > > > >> > > > > >> > >>>> We may make it much easier for a newcomer to >> get >> > > > > >> something >> > > > > >> > > > > running >> > > > > >> > > > > >> but >> > > > > >> > > > > >> > >>>> having them step up and get a reliable >> > production >> > > > > >> > deployment >> > > > > >> > > > may >> > > > > >> > > > > >> still >> > > > > >> > > > > >> > >>>> dominate mailing list traffic, if for >> different >> > > > > reasons >> > > > > >> > than >> > > > > >> > > > > >> today. >> > > > > >> > > > > >> > >>>> >> > > > > >> > > > > >> > >>>> Don't get me wrong -- I'm comfortable with >> > making >> > > > the >> > > > > >> Samza >> > > > > >> > > > > >> dependency >> > > > > >> > > > > >> > >> on >> > > > > >> > > > > >> > >>>> Kafka much more explicit and I absolutely see >> > the >> > > > > >> benefits >> > > > > >> > > in >> > > > > >> > > > > the >> > > > > >> > > > > >> > >>>> reduction of duplication and clashing >> > > > > >> > > > terminologies/abstractions >> > > > > >> > > > > >> that >> > > > > >> > > > > >> > >>>> Chris/Jay describe. Samza as a library would >> > > likely >> > > > > be a >> > > > > >> > very >> > > > > >> > > > > nice >> > > > > >> > > > > >> > tool >> > > > > >> > > > > >> > >> to >> > > > > >> > > > > >> > >>>> add to the Kafka ecosystem. I just have the >> > > concerns >> > > > > >> above >> > > > > >> > re >> > > > > >> > > > the >> > > > > >> > > > > >> > >>>> operational side. >> > > > > >> > > > > >> > >>>> >> > > > > >> > > > > >> > >>>> Garry >> > > > > >> > > > > >> > >>>> >> > > > > >> > > > > >> > >>>> -----Original Message----- >> > > > > >> > > > > >> > >>>> From: Gianmarco De Francisci Morales [mailto: >> > > > > >> > g...@apache.org >> > > > > >> > > ] >> > > > > >> > > > > >> > >>>> Sent: 02 July 2015 12:56 >> > > > > >> > > > > >> > >>>> To: dev@samza.apache.org >> > > > > >> > > > > >> > >>>> Subject: Re: Thoughts and obesrvations on >> Samza >> > > > > >> > > > > >> > >>>> >> > > > > >> > > > > >> > >>>> Very interesting thoughts. >> > > > > >> > > > > >> > >>>> From outside, I have always perceived Samza >> as a >> > > > > >> computing >> > > > > >> > > > layer >> > > > > >> > > > > >> over >> > > > > >> > > > > >> > >>>> Kafka. >> > > > > >> > > > > >> > >>>> >> > > > > >> > > > > >> > >>>> The question, maybe a bit provocative, is >> > "should >> > > > > Samza >> > > > > >> be >> > > > > >> > a >> > > > > >> > > > > >> > sub-project >> > > > > >> > > > > >> > >>>> of Kafka then?" >> > > > > >> > > > > >> > >>>> Or does it make sense to keep it as a >> separate >> > > > project >> > > > > >> > with a >> > > > > >> > > > > >> separate >> > > > > >> > > > > >> > >>>> governance? >> > > > > >> > > > > >> > >>>> >> > > > > >> > > > > >> > >>>> Cheers, >> > > > > >> > > > > >> > >>>> >> > > > > >> > > > > >> > >>>> -- >> > > > > >> > > > > >> > >>>> Gianmarco >> > > > > >> > > > > >> > >>>> >> > > > > >> > > > > >> > >>>> On 2 July 2015 at 08:59, Yan Fang < >> > > > > yanfang...@gmail.com> >> > > > > >> > > > wrote: >> > > > > >> > > > > >> > >>>> >> > > > > >> > > > > >> > >>>>> Overall, I agree to couple with Kafka more >> > > tightly. >> > > > > >> > Because >> > > > > >> > > > > Samza >> > > > > >> > > > > >> de >> > > > > >> > > > > >> > >>>>> facto is based on Kafka, and it should >> leverage >> > > > what >> > > > > >> Kafka >> > > > > >> > > > has. >> > > > > >> > > > > At >> > > > > >> > > > > >> > the >> > > > > >> > > > > >> > >>>>> same time, Kafka does not need to reinvent >> what >> > > > Samza >> > > > > >> > > already >> > > > > >> > > > > >> has. I >> > > > > >> > > > > >> > >>>>> also like the idea of separating the >> ingestion >> > > and >> > > > > >> > > > > transformation. >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >>>>> But it is a little difficult for me to image >> > how >> > > > the >> > > > > >> Samza >> > > > > >> > > > will >> > > > > >> > > > > >> look >> > > > > >> > > > > >> > >>>> like. >> > > > > >> > > > > >> > >>>>> And I feel Chris and Jay have a little >> > difference >> > > > in >> > > > > >> terms >> > > > > >> > > of >> > > > > >> > > > > how >> > > > > >> > > > > >> > >>>>> Samza should look like. >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >>>>> *** Will it look like what Jay's code shows >> (A >> > > > > client of >> > > > > >> > > > Kakfa) >> > > > > >> > > > > ? >> > > > > >> > > > > >> And >> > > > > >> > > > > >> > >>>>> user's application code calls this client? >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >>>>> 1. If we make Samza be a library of Kafka >> (like >> > > > what >> > > > > the >> > > > > >> > > code >> > > > > >> > > > > >> shows), >> > > > > >> > > > > >> > >>>>> how do we implement auto-balance and >> > > > fault-tolerance? >> > > > > >> Are >> > > > > >> > > they >> > > > > >> > > > > >> taken >> > > > > >> > > > > >> > >>>>> care by the Kafka broker or other mechanism, >> > such >> > > > as >> > > > > >> > "Samza >> > > > > >> > > > > >> worker" >> > > > > >> > > > > >> > >>>>> (just make up the name) ? >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >>>>> 2. What about other features, such as >> > > auto-scaling, >> > > > > >> shared >> > > > > >> > > > > state, >> > > > > >> > > > > >> > >>>>> monitoring? >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >>>>> *** If we have Samza standalone, (is this >> what >> > > > Chris >> > > > > >> > > > suggests?) >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >>>>> 1. we still need to ingest data from Kakfa >> and >> > > > > produce >> > > > > >> to >> > > > > >> > > it. >> > > > > >> > > > > >> Then it >> > > > > >> > > > > >> > >>>>> becomes the same as what Samza looks like >> now, >> > > > > except it >> > > > > >> > > does >> > > > > >> > > > > not >> > > > > >> > > > > >> > rely >> > > > > >> > > > > >> > >>>>> on Yarn anymore. >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >>>>> 2. if it is standalone, how can it leverage >> > > Kafka's >> > > > > >> > metrics, >> > > > > >> > > > > logs, >> > > > > >> > > > > >> > >>>>> etc? Use Kafka code as the dependency? >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >>>>> Thanks, >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >>>>> Fang, Yan >> > > > > >> > > > > >> > >>>>> yanfang...@gmail.com >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >>>>> On Wed, Jul 1, 2015 at 5:46 PM, Guozhang >> Wang < >> > > > > >> > > > > wangg...@gmail.com >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > >>>> wrote: >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >>>>>> Read through the code example and it looks >> > good >> > > to >> > > > > me. >> > > > > >> A >> > > > > >> > > few >> > > > > >> > > > > >> > >>>>>> thoughts regarding deployment: >> > > > > >> > > > > >> > >>>>>> >> > > > > >> > > > > >> > >>>>>> Today Samza deploys as executable runnable >> > like: >> > > > > >> > > > > >> > >>>>>> >> > > > > >> > > > > >> > >>>>>> deploy/samza/bin/run-job.sh >> > --config-factory=... >> > > > > >> > > > > >> > >>>> --config-path=file://... >> > > > > >> > > > > >> > >>>>>> >> > > > > >> > > > > >> > >>>>>> And this proposal advocate for deploying >> Samza >> > > > more >> > > > > as >> > > > > >> > > > embedded >> > > > > >> > > > > >> > >>>>>> libraries in user application code >> (ignoring >> > the >> > > > > >> > > terminology >> > > > > >> > > > > >> since >> > > > > >> > > > > >> > >>>>>> it is not the >> > > > > >> > > > > >> > >>>>> same >> > > > > >> > > > > >> > >>>>>> as the prototype code): >> > > > > >> > > > > >> > >>>>>> >> > > > > >> > > > > >> > >>>>>> StreamTask task = new >> MyStreamTask(configs); >> > > > Thread >> > > > > >> > thread >> > > > > >> > > = >> > > > > >> > > > > new >> > > > > >> > > > > >> > >>>>>> Thread(task); thread.start(); >> > > > > >> > > > > >> > >>>>>> >> > > > > >> > > > > >> > >>>>>> I think both of these deployment modes are >> > > > important >> > > > > >> for >> > > > > >> > > > > >> different >> > > > > >> > > > > >> > >>>>>> types >> > > > > >> > > > > >> > >>>>> of >> > > > > >> > > > > >> > >>>>>> users. That said, I think making Samza >> purely >> > > > > >> standalone >> > > > > >> > is >> > > > > >> > > > > still >> > > > > >> > > > > >> > >>>>>> sufficient for either runnable or library >> > modes. >> > > > > >> > > > > >> > >>>>>> >> > > > > >> > > > > >> > >>>>>> Guozhang >> > > > > >> > > > > >> > >>>>>> >> > > > > >> > > > > >> > >>>>>> On Tue, Jun 30, 2015 at 11:33 PM, Jay >> Kreps < >> > > > > >> > > > j...@confluent.io> >> > > > > >> > > > > >> > wrote: >> > > > > >> > > > > >> > >>>>>> >> > > > > >> > > > > >> > >>>>>>> Looks like gmail mangled the code >> example, it >> > > was >> > > > > >> > supposed >> > > > > >> > > > to >> > > > > >> > > > > >> look >> > > > > >> > > > > >> > >>>>>>> like >> > > > > >> > > > > >> > >>>>>>> this: >> > > > > >> > > > > >> > >>>>>>> >> > > > > >> > > > > >> > >>>>>>> Properties props = new Properties(); >> > > > > >> > > > > >> > >>>>>>> props.put("bootstrap.servers", >> > > "localhost:4242"); >> > > > > >> > > > > >> StreamingConfig >> > > > > >> > > > > >> > >>>>>>> config = new StreamingConfig(props); >> > > > > >> > > > > >> > >>>>>>> config.subscribe("test-topic-1", >> > > "test-topic-2"); >> > > > > >> > > > > >> > >>>>>>> >> > config.processor(ExampleStreamProcessor.class); >> > > > > >> > > > > >> > >>>>>>> config.serialization(new >> StringSerializer(), >> > > new >> > > > > >> > > > > >> > >>>>>>> StringDeserializer()); KafkaStreaming >> > > container = >> > > > > new >> > > > > >> > > > > >> > >>>>>>> KafkaStreaming(config); container.run(); >> > > > > >> > > > > >> > >>>>>>> >> > > > > >> > > > > >> > >>>>>>> -Jay >> > > > > >> > > > > >> > >>>>>>> >> > > > > >> > > > > >> > >>>>>>> On Tue, Jun 30, 2015 at 11:32 PM, Jay >> Kreps < >> > > > > >> > > > j...@confluent.io >> > > > > >> > > > > > >> > > > > >> > > > > >> > >>>> wrote: >> > > > > >> > > > > >> > >>>>>>> >> > > > > >> > > > > >> > >>>>>>>> Hey guys, >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> This came out of some conversations Chris >> > and >> > > I >> > > > > were >> > > > > >> > > having >> > > > > >> > > > > >> > >>>>>>>> around >> > > > > >> > > > > >> > >>>>>>> whether >> > > > > >> > > > > >> > >>>>>>>> it would make sense to use Samza as a >> kind >> > of >> > > > data >> > > > > >> > > > ingestion >> > > > > >> > > > > >> > >>>>> framework >> > > > > >> > > > > >> > >>>>>>> for >> > > > > >> > > > > >> > >>>>>>>> Kafka (which ultimately lead to KIP-26 >> > > > "copycat"). >> > > > > >> This >> > > > > >> > > > kind >> > > > > >> > > > > of >> > > > > >> > > > > >> > >>>>>> combined >> > > > > >> > > > > >> > >>>>>>>> with complaints around config and YARN >> and >> > the >> > > > > >> > discussion >> > > > > >> > > > > >> around >> > > > > >> > > > > >> > >>>>>>>> how >> > > > > >> > > > > >> > >>>>> to >> > > > > >> > > > > >> > >>>>>>>> best do a standalone mode. >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> So the thought experiment was, given that >> > > Samza >> > > > > was >> > > > > >> > > > basically >> > > > > >> > > > > >> > >>>>>>>> already totally Kafka specific, what if >> you >> > > just >> > > > > >> > embraced >> > > > > >> > > > > that >> > > > > >> > > > > >> > >>>>>>>> and turned it >> > > > > >> > > > > >> > >>>>>> into >> > > > > >> > > > > >> > >>>>>>>> something less like a heavyweight >> framework >> > > and >> > > > > more >> > > > > >> > > like a >> > > > > >> > > > > >> > >>>>>>>> third >> > > > > >> > > > > >> > >>>>> Kafka >> > > > > >> > > > > >> > >>>>>>>> client--a kind of "producing consumer" >> with >> > > > state >> > > > > >> > > > management >> > > > > >> > > > > >> > >>>>>> facilities. >> > > > > >> > > > > >> > >>>>>>>> Basically a library. Instead of a complex >> > > stream >> > > > > >> > > processing >> > > > > >> > > > > >> > >>>>>>>> framework >> > > > > >> > > > > >> > >>>>>>> this >> > > > > >> > > > > >> > >>>>>>>> would actually be a very simple thing, >> not >> > > much >> > > > > more >> > > > > >> > > > > >> complicated >> > > > > >> > > > > >> > >>>>>>>> to >> > > > > >> > > > > >> > >>>>> use >> > > > > >> > > > > >> > >>>>>>> or >> > > > > >> > > > > >> > >>>>>>>> operate than a Kafka consumer. As Chris >> said >> > > we >> > > > > >> thought >> > > > > >> > > > about >> > > > > >> > > > > >> it >> > > > > >> > > > > >> > >>>>>>>> a >> > > > > >> > > > > >> > >>>>> lot >> > > > > >> > > > > >> > >>>>>> of >> > > > > >> > > > > >> > >>>>>>>> what Samza (and the other stream >> processing >> > > > > systems >> > > > > >> > were >> > > > > >> > > > > doing) >> > > > > >> > > > > >> > >>>>> seemed >> > > > > >> > > > > >> > >>>>>>> like >> > > > > >> > > > > >> > >>>>>>>> kind of a hangover from MapReduce. >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> Of course you need to ingest/output data >> to >> > > and >> > > > > from >> > > > > >> > the >> > > > > >> > > > > stream >> > > > > >> > > > > >> > >>>>>>>> processing. But when we actually looked >> into >> > > how >> > > > > that >> > > > > >> > > would >> > > > > >> > > > > >> > >>>>>>>> work, >> > > > > >> > > > > >> > >>>>> Samza >> > > > > >> > > > > >> > >>>>>>>> isn't really an ideal data ingestion >> > framework >> > > > > for a >> > > > > >> > > bunch >> > > > > >> > > > of >> > > > > >> > > > > >> > >>>>> reasons. >> > > > > >> > > > > >> > >>>>>> To >> > > > > >> > > > > >> > >>>>>>>> really do that right you need a pretty >> > > different >> > > > > >> > internal >> > > > > >> > > > > data >> > > > > >> > > > > >> > >>>>>>>> model >> > > > > >> > > > > >> > >>>>>> and >> > > > > >> > > > > >> > >>>>>>>> set of apis. So what if you split them >> and >> > had >> > > > an >> > > > > api >> > > > > >> > for >> > > > > >> > > > > Kafka >> > > > > >> > > > > >> > >>>>>>>> ingress/egress (copycat AKA KIP-26) and a >> > > > separate >> > > > > >> api >> > > > > >> > > for >> > > > > >> > > > > >> Kafka >> > > > > >> > > > > >> > >>>>>>>> transformation (Samza). >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> This would also allow really embracing >> the >> > > same >> > > > > >> > > terminology >> > > > > >> > > > > and >> > > > > >> > > > > >> > >>>>>>>> conventions. One complaint about the >> current >> > > > > state is >> > > > > >> > > that >> > > > > >> > > > > the >> > > > > >> > > > > >> > >>>>>>>> two >> > > > > >> > > > > >> > >>>>>>> systems >> > > > > >> > > > > >> > >>>>>>>> kind of feel bolted on. Terminology like >> > > > "stream" >> > > > > vs >> > > > > >> > > > "topic" >> > > > > >> > > > > >> and >> > > > > >> > > > > >> > >>>>>>> different >> > > > > >> > > > > >> > >>>>>>>> config and monitoring systems means you >> kind >> > > of >> > > > > have >> > > > > >> to >> > > > > >> > > > learn >> > > > > >> > > > > >> > >>>>>>>> Kafka's >> > > > > >> > > > > >> > >>>>>>> way, >> > > > > >> > > > > >> > >>>>>>>> then learn Samza's slightly different >> way, >> > > then >> > > > > kind >> > > > > >> of >> > > > > >> > > > > >> > >>>>>>>> understand >> > > > > >> > > > > >> > >>>>> how >> > > > > >> > > > > >> > >>>>>>> they >> > > > > >> > > > > >> > >>>>>>>> map to each other, which having walked a >> few >> > > > > people >> > > > > >> > > through >> > > > > >> > > > > >> this >> > > > > >> > > > > >> > >>>>>>>> is surprisingly tricky for folks to get. >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> Since I have been spending a lot of time >> on >> > > > > >> airplanes I >> > > > > >> > > > > hacked >> > > > > >> > > > > >> > >>>>>>>> up an ernest but still somewhat >> incomplete >> > > > > prototype >> > > > > >> of >> > > > > >> > > > what >> > > > > >> > > > > >> > >>>>>>>> this would >> > > > > >> > > > > >> > >>>>> look >> > > > > >> > > > > >> > >>>>>>>> like. This is just unceremoniously dumped >> > into >> > > > > Kafka >> > > > > >> as >> > > > > >> > > it >> > > > > >> > > > > >> > >>>>>>>> required a >> > > > > >> > > > > >> > >>>>>> few >> > > > > >> > > > > >> > >>>>>>>> changes to the new consumer. Here is the >> > code: >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>> >> > > > > >> > > > > >> > >>>>>> >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > > > > >> > >> > > > > >> > https://github.com/jkreps/kafka/tree/streams/clients/src/main/java/org >> > > > > >> > > > > >> > >>>>> /apache/kafka/clients/streaming >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> For the purpose of the prototype I just >> > > > liberally >> > > > > >> > renamed >> > > > > >> > > > > >> > >>>>>>>> everything >> > > > > >> > > > > >> > >>>>> to >> > > > > >> > > > > >> > >>>>>>>> try to align it with Kafka with no regard >> > for >> > > > > >> > > > compatibility. >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> To use this would be something like this: >> > > > > >> > > > > >> > >>>>>>>> Properties props = new Properties(); >> > > > > >> > > > > >> > >>>>>>>> props.put("bootstrap.servers", >> > > > "localhost:4242"); >> > > > > >> > > > > >> > >>>>>>>> StreamingConfig config = new >> > > > > >> > > > > >> > >>>>> StreamingConfig(props); >> > > > > >> > > > > >> > >>>>>>> config.subscribe("test-topic-1", >> > > > > >> > > > > >> > >>>>>>>> "test-topic-2"); >> > > > > >> > > > > >> config.processor(ExampleStreamProcessor.class); >> > > > > >> > > > > >> > >>>>>>> config.serialization(new >> > > > > >> > > > > >> > >>>>>>>> StringSerializer(), new >> > StringDeserializer()); >> > > > > >> > > > KafkaStreaming >> > > > > >> > > > > >> > >>>>>> container = >> > > > > >> > > > > >> > >>>>>>>> new KafkaStreaming(config); >> container.run(); >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> KafkaStreaming is basically the >> > > SamzaContainer; >> > > > > >> > > > > StreamProcessor >> > > > > >> > > > > >> > >>>>>>>> is basically StreamTask. >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> So rather than putting all the class >> names >> > in >> > > a >> > > > > file >> > > > > >> > and >> > > > > >> > > > then >> > > > > >> > > > > >> > >>>>>>>> having >> > > > > >> > > > > >> > >>>>>> the >> > > > > >> > > > > >> > >>>>>>>> job assembled by reflection, you just >> > > > instantiate >> > > > > the >> > > > > >> > > > > container >> > > > > >> > > > > >> > >>>>>>>> programmatically. Work is balanced over >> > > however >> > > > > many >> > > > > >> > > > > instances >> > > > > >> > > > > >> > >>>>>>>> of >> > > > > >> > > > > >> > >>>>> this >> > > > > >> > > > > >> > >>>>>>> are >> > > > > >> > > > > >> > >>>>>>>> alive at any time (i.e. if an instance >> dies, >> > > new >> > > > > >> tasks >> > > > > >> > > are >> > > > > >> > > > > >> added >> > > > > >> > > > > >> > >>>>>>>> to >> > > > > >> > > > > >> > >>>>> the >> > > > > >> > > > > >> > >>>>>>>> existing containers without shutting them >> > > down). >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> We would provide some glue for running >> this >> > > > stuff >> > > > > in >> > > > > >> > YARN >> > > > > >> > > > via >> > > > > >> > > > > >> > >>>>>>>> Slider, Mesos via Marathon, and AWS using >> > some >> > > > of >> > > > > >> their >> > > > > >> > > > tools >> > > > > >> > > > > >> > >>>>>>>> but from the >> > > > > >> > > > > >> > >>>>>> point >> > > > > >> > > > > >> > >>>>>>> of >> > > > > >> > > > > >> > >>>>>>>> view of these frameworks these stream >> > > processing >> > > > > jobs >> > > > > >> > are >> > > > > >> > > > > just >> > > > > >> > > > > >> > >>>>>> stateless >> > > > > >> > > > > >> > >>>>>>>> services that can come and go and expand >> and >> > > > > contract >> > > > > >> > at >> > > > > >> > > > > will. >> > > > > >> > > > > >> > >>>>>>>> There >> > > > > >> > > > > >> > >>>>> is >> > > > > >> > > > > >> > >>>>>>> no >> > > > > >> > > > > >> > >>>>>>>> more custom scheduler. >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> Here are some relevant details: >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> 1. It is only ~1300 lines of code, it >> would >> > > get >> > > > > >> larger >> > > > > >> > > if >> > > > > >> > > > we >> > > > > >> > > > > >> > >>>>>>>> productionized but not vastly larger. We >> > > really >> > > > > do >> > > > > >> > get a >> > > > > >> > > > ton >> > > > > >> > > > > >> > >>>>>>>> of >> > > > > >> > > > > >> > >>>>>>> leverage >> > > > > >> > > > > >> > >>>>>>>> out of Kafka. >> > > > > >> > > > > >> > >>>>>>>> 2. Partition management is fully >> delegated >> > to >> > > > the >> > > > > >> new >> > > > > >> > > > > >> consumer. >> > > > > >> > > > > >> > >>>>> This >> > > > > >> > > > > >> > >>>>>>>> is nice since now any partition >> management >> > > > > strategy >> > > > > >> > > > > available >> > > > > >> > > > > >> > >>>>>>>> to >> > > > > >> > > > > >> > >>>>>> Kafka >> > > > > >> > > > > >> > >>>>>>>> consumer is also available to Samza (and >> > vice >> > > > > versa) >> > > > > >> > and >> > > > > >> > > > > with >> > > > > >> > > > > >> > >>>>>>>> the >> > > > > >> > > > > >> > >>>>>>> exact >> > > > > >> > > > > >> > >>>>>>>> same configs. >> > > > > >> > > > > >> > >>>>>>>> 3. It supports state as well as state >> reuse >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> Anyhow take a look, hopefully it is >> thought >> > > > > >> provoking. >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> -Jay >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> On Tue, Jun 30, 2015 at 6:55 PM, Chris >> > > > Riccomini < >> > > > > >> > > > > >> > >>>>>> criccom...@apache.org> >> > > > > >> > > > > >> > >>>>>>>> wrote: >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> Hey all, >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> I have had some discussions with Samza >> > > > engineers >> > > > > at >> > > > > >> > > > LinkedIn >> > > > > >> > > > > >> > >>>>>>>>> and >> > > > > >> > > > > >> > >>>>>>> Confluent >> > > > > >> > > > > >> > >>>>>>>>> and we came up with a few observations >> and >> > > > would >> > > > > >> like >> > > > > >> > to >> > > > > >> > > > > >> > >>>>>>>>> propose >> > > > > >> > > > > >> > >>>>> some >> > > > > >> > > > > >> > >>>>>>>>> changes. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> We've observed some things that I want >> to >> > > call >> > > > > out >> > > > > >> > about >> > > > > >> > > > > >> > >>>>>>>>> Samza's >> > > > > >> > > > > >> > >>>>>> design, >> > > > > >> > > > > >> > >>>>>>>>> and I'd like to propose some changes. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> * Samza is dependent upon a dynamic >> > > deployment >> > > > > >> system. >> > > > > >> > > > > >> > >>>>>>>>> * Samza is too pluggable. >> > > > > >> > > > > >> > >>>>>>>>> * Samza's SystemConsumer/SystemProducer >> and >> > > > > Kafka's >> > > > > >> > > > consumer >> > > > > >> > > > > >> > >>>>>>>>> APIs >> > > > > >> > > > > >> > >>>>> are >> > > > > >> > > > > >> > >>>>>>>>> trying to solve a lot of the same >> problems. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> All three of these issues are related, >> but >> > > I'll >> > > > > >> > address >> > > > > >> > > > them >> > > > > >> > > > > >> in >> > > > > >> > > > > >> > >>>>> order. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> Deployment >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> Samza strongly depends on the use of a >> > > dynamic >> > > > > >> > > deployment >> > > > > >> > > > > >> > >>>>>>>>> scheduler >> > > > > >> > > > > >> > >>>>>> such >> > > > > >> > > > > >> > >>>>>>>>> as >> > > > > >> > > > > >> > >>>>>>>>> YARN, Mesos, etc. When we initially >> built >> > > > Samza, >> > > > > we >> > > > > >> > bet >> > > > > >> > > > that >> > > > > >> > > > > >> > >>>>>>>>> there >> > > > > >> > > > > >> > >>>>>> would >> > > > > >> > > > > >> > >>>>>>>>> be >> > > > > >> > > > > >> > >>>>>>>>> one or two winners in this area, and we >> > could >> > > > > >> support >> > > > > >> > > > them, >> > > > > >> > > > > >> and >> > > > > >> > > > > >> > >>>>>>>>> the >> > > > > >> > > > > >> > >>>>>> rest >> > > > > >> > > > > >> > >>>>>>>>> would go away. In reality, there are >> many >> > > > > >> variations. >> > > > > >> > > > > >> > >>>>>>>>> Furthermore, >> > > > > >> > > > > >> > >>>>>> many >> > > > > >> > > > > >> > >>>>>>>>> people still prefer to just start their >> > > > > processors >> > > > > >> > like >> > > > > >> > > > > normal >> > > > > >> > > > > >> > >>>>>>>>> Java processes, and use traditional >> > > deployment >> > > > > >> scripts >> > > > > >> > > > such >> > > > > >> > > > > as >> > > > > >> > > > > >> > >>>>>>>>> Fabric, >> > > > > >> > > > > >> > >>>>>> Chef, >> > > > > >> > > > > >> > >>>>>>>>> Ansible, etc. Forcing a deployment >> system >> > on >> > > > > users >> > > > > >> > makes >> > > > > >> > > > the >> > > > > >> > > > > >> > >>>>>>>>> Samza start-up process really painful >> for >> > > first >> > > > > time >> > > > > >> > > > users. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> Dynamic deployment as a requirement was >> > also >> > > a >> > > > > bit >> > > > > >> of >> > > > > >> > a >> > > > > >> > > > > >> > >>>>>>>>> mis-fire >> > > > > >> > > > > >> > >>>>>> because >> > > > > >> > > > > >> > >>>>>>>>> of >> > > > > >> > > > > >> > >>>>>>>>> a fundamental misunderstanding between >> the >> > > > > nature of >> > > > > >> > > batch >> > > > > >> > > > > >> jobs >> > > > > >> > > > > >> > >>>>>>>>> and >> > > > > >> > > > > >> > >>>>>>> stream >> > > > > >> > > > > >> > >>>>>>>>> processing jobs. Early on, we made >> > conscious >> > > > > effort >> > > > > >> to >> > > > > >> > > > favor >> > > > > >> > > > > >> > >>>>>>>>> the >> > > > > >> > > > > >> > >>>>>> Hadoop >> > > > > >> > > > > >> > >>>>>>>>> (Map/Reduce) way of doing things, since >> it >> > > > worked >> > > > > >> and >> > > > > >> > > was >> > > > > >> > > > > well >> > > > > >> > > > > >> > >>>>>>> understood. >> > > > > >> > > > > >> > >>>>>>>>> One thing that we missed was that batch >> > jobs >> > > > > have a >> > > > > >> > > > definite >> > > > > >> > > > > >> > >>>>>> beginning, >> > > > > >> > > > > >> > >>>>>>>>> and >> > > > > >> > > > > >> > >>>>>>>>> end, and stream processing jobs don't >> > > > (usually). >> > > > > >> This >> > > > > >> > > > leads >> > > > > >> > > > > to >> > > > > >> > > > > >> > >>>>>>>>> a >> > > > > >> > > > > >> > >>>>> much >> > > > > >> > > > > >> > >>>>>>>>> simpler scheduling problem for stream >> > > > processors. >> > > > > >> You >> > > > > >> > > > > >> basically >> > > > > >> > > > > >> > >>>>>>>>> just >> > > > > >> > > > > >> > >>>>>>> need >> > > > > >> > > > > >> > >>>>>>>>> to find a place to start the processor, >> and >> > > > start >> > > > > >> it. >> > > > > >> > > The >> > > > > >> > > > > way >> > > > > >> > > > > >> > >>>>>>>>> we run grids, at LinkedIn, there's no >> > concept >> > > > of >> > > > > a >> > > > > >> > > cluster >> > > > > >> > > > > >> > >>>>>>>>> being "full". We always >> > > > > >> > > > > >> > >>>>>> add >> > > > > >> > > > > >> > >>>>>>>>> more machines. The problem with coupling >> > > Samza >> > > > > with >> > > > > >> a >> > > > > >> > > > > >> scheduler >> > > > > >> > > > > >> > >>>>>>>>> is >> > > > > >> > > > > >> > >>>>>> that >> > > > > >> > > > > >> > >>>>>>>>> Samza (as a framework) now has to handle >> > > > > deployment. >> > > > > >> > > This >> > > > > >> > > > > >> pulls >> > > > > >> > > > > >> > >>>>>>>>> in a >> > > > > >> > > > > >> > >>>>>>> bunch >> > > > > >> > > > > >> > >>>>>>>>> of things such as configuration >> > distribution >> > > > > (config >> > > > > >> > > > > stream), >> > > > > >> > > > > >> > >>>>>>>>> shell >> > > > > >> > > > > >> > >>>>>>> scrips >> > > > > >> > > > > >> > >>>>>>>>> (bin/run-job.sh, JobRunner), packaging >> (all >> > > the >> > > > > .tgz >> > > > > >> > > > stuff), >> > > > > >> > > > > >> etc. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> Another reason for requiring dynamic >> > > deployment >> > > > > was >> > > > > >> to >> > > > > >> > > > > support >> > > > > >> > > > > >> > >>>>>>>>> data locality. If you want to have >> > locality, >> > > > you >> > > > > >> need >> > > > > >> > to >> > > > > >> > > > put >> > > > > >> > > > > >> > >>>>>>>>> your >> > > > > >> > > > > >> > >>>>>> processors >> > > > > >> > > > > >> > >>>>>>>>> close to the data they're processing. >> Upon >> > > > > further >> > > > > >> > > > > >> > >>>>>>>>> investigation, >> > > > > >> > > > > >> > >>>>>>> though, >> > > > > >> > > > > >> > >>>>>>>>> this feature is not that beneficial. >> There >> > is >> > > > > some >> > > > > >> > good >> > > > > >> > > > > >> > >>>>>>>>> discussion >> > > > > >> > > > > >> > >>>>>> about >> > > > > >> > > > > >> > >>>>>>>>> some problems with it on SAMZA-335. >> Again, >> > we >> > > > > took >> > > > > >> the >> > > > > >> > > > > >> > >>>>>>>>> Map/Reduce >> > > > > >> > > > > >> > >>>>>> path, >> > > > > >> > > > > >> > >>>>>>>>> but >> > > > > >> > > > > >> > >>>>>>>>> there are some fundamental differences >> > > between >> > > > > HDFS >> > > > > >> > and >> > > > > >> > > > > Kafka. >> > > > > >> > > > > >> > >>>>>>>>> HDFS >> > > > > >> > > > > >> > >>>>>> has >> > > > > >> > > > > >> > >>>>>>>>> blocks, while Kafka has partitions. This >> > > leads >> > > > to >> > > > > >> less >> > > > > >> > > > > >> > >>>>>>>>> optimization potential with stream >> > processors >> > > > on >> > > > > top >> > > > > >> > of >> > > > > >> > > > > Kafka. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> This feature is also used as a crutch. >> > Samza >> > > > > doesn't >> > > > > >> > > have >> > > > > >> > > > > any >> > > > > >> > > > > >> > >>>>>>>>> built >> > > > > >> > > > > >> > >>>>> in >> > > > > >> > > > > >> > >>>>>>>>> fault-tolerance logic. Instead, it >> depends >> > on >> > > > the >> > > > > >> > > dynamic >> > > > > >> > > > > >> > >>>>>>>>> deployment scheduling system to handle >> > > restarts >> > > > > >> when a >> > > > > >> > > > > >> > >>>>>>>>> processor dies. This has >> > > > > >> > > > > >> > >>>>>>> made >> > > > > >> > > > > >> > >>>>>>>>> it very difficult to write a standalone >> > Samza >> > > > > >> > container >> > > > > >> > > > > >> > >>>> (SAMZA-516). >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> Pluggability >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> In some cases pluggability is good, but >> I >> > > think >> > > > > that >> > > > > >> > > we've >> > > > > >> > > > > >> gone >> > > > > >> > > > > >> > >>>>>>>>> too >> > > > > >> > > > > >> > >>>>>> far >> > > > > >> > > > > >> > >>>>>>>>> with it. Currently, Samza has: >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> * Pluggable config. >> > > > > >> > > > > >> > >>>>>>>>> * Pluggable metrics. >> > > > > >> > > > > >> > >>>>>>>>> * Pluggable deployment systems. >> > > > > >> > > > > >> > >>>>>>>>> * Pluggable streaming systems >> > > (SystemConsumer, >> > > > > >> > > > > SystemProducer, >> > > > > >> > > > > >> > >>>> etc). >> > > > > >> > > > > >> > >>>>>>>>> * Pluggable serdes. >> > > > > >> > > > > >> > >>>>>>>>> * Pluggable storage engines. >> > > > > >> > > > > >> > >>>>>>>>> * Pluggable strategies for just about >> every >> > > > > >> component >> > > > > >> > > > > >> > >>>>> (MessageChooser, >> > > > > >> > > > > >> > >>>>>>>>> SystemStreamPartitionGrouper, >> > ConfigRewriter, >> > > > > etc). >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> There's probably more that I've >> forgotten, >> > as >> > > > > well. >> > > > > >> > Some >> > > > > >> > > > of >> > > > > >> > > > > >> > >>>>>>>>> these >> > > > > >> > > > > >> > >>>>> are >> > > > > >> > > > > >> > >>>>>>>>> useful, but some have proven not to be. >> > This >> > > > all >> > > > > >> comes >> > > > > >> > > at >> > > > > >> > > > a >> > > > > >> > > > > >> cost: >> > > > > >> > > > > >> > >>>>>>>>> complexity. This complexity is making it >> > > harder >> > > > > for >> > > > > >> > our >> > > > > >> > > > > users >> > > > > >> > > > > >> > >>>>>>>>> to >> > > > > >> > > > > >> > >>>>> pick >> > > > > >> > > > > >> > >>>>>> up >> > > > > >> > > > > >> > >>>>>>>>> and use Samza out of the box. It also >> makes >> > > it >> > > > > >> > difficult >> > > > > >> > > > for >> > > > > >> > > > > >> > >>>>>>>>> Samza developers to reason about what >> the >> > > > > >> > > characteristics >> > > > > >> > > > of >> > > > > >> > > > > >> > >>>>>>>>> the container (since the characteristics >> > > change >> > > > > >> > > depending >> > > > > >> > > > on >> > > > > >> > > > > >> > >>>>>>>>> which plugins are use). >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> The issues with pluggability are most >> > visible >> > > > in >> > > > > the >> > > > > >> > > > System >> > > > > >> > > > > >> APIs. >> > > > > >> > > > > >> > >>>>> What >> > > > > >> > > > > >> > >>>>>>>>> Samza really requires to be functional >> is >> > > Kafka >> > > > > as >> > > > > >> its >> > > > > >> > > > > >> > >>>>>>>>> transport >> > > > > >> > > > > >> > >>>>>> layer. >> > > > > >> > > > > >> > >>>>>>>>> But >> > > > > >> > > > > >> > >>>>>>>>> we've conflated two unrelated use cases >> > into >> > > > one >> > > > > >> API: >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> 1. Get data into/out of Kafka. >> > > > > >> > > > > >> > >>>>>>>>> 2. Process the data in Kafka. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> The current System API supports both of >> > these >> > > > use >> > > > > >> > cases. >> > > > > >> > > > The >> > > > > >> > > > > >> > >>>>>>>>> problem >> > > > > >> > > > > >> > >>>>>> is, >> > > > > >> > > > > >> > >>>>>>>>> we >> > > > > >> > > > > >> > >>>>>>>>> actually want different features for >> each >> > use >> > > > > case. >> > > > > >> By >> > > > > >> > > > > >> papering >> > > > > >> > > > > >> > >>>>>>>>> over >> > > > > >> > > > > >> > >>>>>>> these >> > > > > >> > > > > >> > >>>>>>>>> two use cases, and providing a single >> API, >> > > > we've >> > > > > >> > > > introduced >> > > > > >> > > > > a >> > > > > >> > > > > >> > >>>>>>>>> ton of >> > > > > >> > > > > >> > >>>>>>> leaky >> > > > > >> > > > > >> > >>>>>>>>> abstractions. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> For example, what we'd really like in >> (2) >> > is >> > > to >> > > > > have >> > > > > >> > > > > >> > >>>>>>>>> monotonically increasing longs for >> offsets >> > > > (like >> > > > > >> > Kafka). >> > > > > >> > > > > This >> > > > > >> > > > > >> > >>>>>>>>> would be at odds >> > > > > >> > > > > >> > >>>>> with >> > > > > >> > > > > >> > >>>>>>> (1), >> > > > > >> > > > > >> > >>>>>>>>> though, since different systems have >> > > different >> > > > > >> > > > > >> > >>>>>>> SCNs/Offsets/UUIDs/vectors. >> > > > > >> > > > > >> > >>>>>>>>> There was discussion both on the mailing >> > list >> > > > and >> > > > > >> the >> > > > > >> > > SQL >> > > > > >> > > > > >> JIRAs >> > > > > >> > > > > >> > >>>>> about >> > > > > >> > > > > >> > >>>>>>> the >> > > > > >> > > > > >> > >>>>>>>>> need for this. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> The same thing holds true for >> > replayability. >> > > > > Kafka >> > > > > >> > > allows >> > > > > >> > > > us >> > > > > >> > > > > >> to >> > > > > >> > > > > >> > >>>>> rewind >> > > > > >> > > > > >> > >>>>>>>>> when >> > > > > >> > > > > >> > >>>>>>>>> we have a failure. Many other systems >> > don't. >> > > In >> > > > > some >> > > > > >> > > > cases, >> > > > > >> > > > > >> > >>>>>>>>> systems >> > > > > >> > > > > >> > >>>>>>> return >> > > > > >> > > > > >> > >>>>>>>>> null for their offsets (e.g. >> > > > > >> WikipediaSystemConsumer) >> > > > > >> > > > > because >> > > > > >> > > > > >> > >>>>>>>>> they >> > > > > >> > > > > >> > >>>>>> have >> > > > > >> > > > > >> > >>>>>>> no >> > > > > >> > > > > >> > >>>>>>>>> offsets. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> Partitioning is another example. Kafka >> > > supports >> > > > > >> > > > > partitioning, >> > > > > >> > > > > >> > >>>>>>>>> but >> > > > > >> > > > > >> > >>>>> many >> > > > > >> > > > > >> > >>>>>>>>> systems don't. We model this by having a >> > > single >> > > > > >> > > partition >> > > > > >> > > > > for >> > > > > >> > > > > >> > >>>>>>>>> those systems. Still, other systems >> model >> > > > > >> partitioning >> > > > > >> > > > > >> > >>>> differently (e.g. >> > > > > >> > > > > >> > >>>>>>>>> Kinesis). >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> The SystemAdmin interface is also a >> mess. >> > > > > Creating >> > > > > >> > > streams >> > > > > >> > > > > in >> > > > > >> > > > > >> a >> > > > > >> > > > > >> > >>>>>>>>> system-agnostic way is almost >> impossible. >> > As >> > > is >> > > > > >> > modeling >> > > > > >> > > > > >> > >>>>>>>>> metadata >> > > > > >> > > > > >> > >>>>> for >> > > > > >> > > > > >> > >>>>>>> the >> > > > > >> > > > > >> > >>>>>>>>> system (replication factor, partitions, >> > > > location, >> > > > > >> > etc). >> > > > > >> > > > The >> > > > > >> > > > > >> > >>>>>>>>> list >> > > > > >> > > > > >> > >>>>> goes >> > > > > >> > > > > >> > >>>>>>> on. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> Duplicate work >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> At the time that we began writing Samza, >> > > > Kafka's >> > > > > >> > > consumer >> > > > > >> > > > > and >> > > > > >> > > > > >> > >>>>> producer >> > > > > >> > > > > >> > >>>>>>>>> APIs >> > > > > >> > > > > >> > >>>>>>>>> had a relatively weak feature set. On >> the >> > > > > >> > consumer-side, >> > > > > >> > > > you >> > > > > >> > > > > >> > >>>>>>>>> had two >> > > > > >> > > > > >> > >>>>>>>>> options: use the high level consumer, or >> > the >> > > > > simple >> > > > > >> > > > > consumer. >> > > > > >> > > > > >> > >>>>>>>>> The >> > > > > >> > > > > >> > >>>>>>> problem >> > > > > >> > > > > >> > >>>>>>>>> with the high-level consumer was that it >> > > > > controlled >> > > > > >> > your >> > > > > >> > > > > >> > >>>>>>>>> offsets, partition assignments, and the >> > order >> > > > in >> > > > > >> which >> > > > > >> > > you >> > > > > >> > > > > >> > >>>>>>>>> received messages. The >> > > > > >> > > > > >> > >>>>> problem >> > > > > >> > > > > >> > >>>>>>>>> with >> > > > > >> > > > > >> > >>>>>>>>> the simple consumer is that it's not >> > simple. >> > > > It's >> > > > > >> > basic. >> > > > > >> > > > You >> > > > > >> > > > > >> > >>>>>>>>> end up >> > > > > >> > > > > >> > >>>>>>> having >> > > > > >> > > > > >> > >>>>>>>>> to handle a lot of really low-level >> stuff >> > > that >> > > > > you >> > > > > >> > > > > shouldn't. >> > > > > >> > > > > >> > >>>>>>>>> We >> > > > > >> > > > > >> > >>>>>> spent a >> > > > > >> > > > > >> > >>>>>>>>> lot of time to make Samza's >> > > KafkaSystemConsumer >> > > > > very >> > > > > >> > > > robust. >> > > > > >> > > > > >> It >> > > > > >> > > > > >> > >>>>>>>>> also allows us to support some cool >> > features: >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> * Per-partition message ordering and >> > > > > prioritization. >> > > > > >> > > > > >> > >>>>>>>>> * Tight control over partition >> assignment >> > to >> > > > > support >> > > > > >> > > > joins, >> > > > > >> > > > > >> > >>>>>>>>> global >> > > > > >> > > > > >> > >>>>>> state >> > > > > >> > > > > >> > >>>>>>>>> (if we want to implement it :)), etc. >> > > > > >> > > > > >> > >>>>>>>>> * Tight control over offset >> checkpointing. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> What we didn't realize at the time is >> that >> > > > these >> > > > > >> > > features >> > > > > >> > > > > >> > >>>>>>>>> should >> > > > > >> > > > > >> > >>>>>>> actually >> > > > > >> > > > > >> > >>>>>>>>> be in Kafka. A lot of Kafka consumers >> (not >> > > just >> > > > > >> Samza >> > > > > >> > > > stream >> > > > > >> > > > > >> > >>>>>> processors) >> > > > > >> > > > > >> > >>>>>>>>> end up wanting to do things like joins >> and >> > > > > partition >> > > > > >> > > > > >> > >>>>>>>>> assignment. The >> > > > > >> > > > > >> > >>>>>>> Kafka >> > > > > >> > > > > >> > >>>>>>>>> community has come to the same >> conclusion. >> > > > > They're >> > > > > >> > > adding >> > > > > >> > > > a >> > > > > >> > > > > >> ton >> > > > > >> > > > > >> > >>>>>>>>> of upgrades into their new Kafka >> consumer >> > > > > >> > > implementation. >> > > > > >> > > > > To a >> > > > > >> > > > > >> > >>>>>>>>> large extent, >> > > > > >> > > > > >> > >>>>> it's >> > > > > >> > > > > >> > >>>>>>>>> duplicate work to what we've already >> done >> > in >> > > > > Samza. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> On top of this, Kafka ended up taking a >> > very >> > > > > similar >> > > > > >> > > > > approach >> > > > > >> > > > > >> > >>>>>>>>> to >> > > > > >> > > > > >> > >>>>>> Samza's >> > > > > >> > > > > >> > >>>>>>>>> KafkaCheckpointManager implementation >> for >> > > > > handling >> > > > > >> > > offset >> > > > > >> > > > > >> > >>>>>> checkpointing. >> > > > > >> > > > > >> > >>>>>>>>> Like Samza, Kafka's new offset >> management >> > > > feature >> > > > > >> > stores >> > > > > >> > > > > >> offset >> > > > > >> > > > > >> > >>>>>>>>> checkpoints in a topic, and allows you >> to >> > > fetch >> > > > > them >> > > > > >> > > from >> > > > > >> > > > > the >> > > > > >> > > > > >> > >>>>>>>>> broker. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> A lot of this seems like a waste, since >> we >> > > > could >> > > > > >> have >> > > > > >> > > > shared >> > > > > >> > > > > >> > >>>>>>>>> the >> > > > > >> > > > > >> > >>>>> work >> > > > > >> > > > > >> > >>>>>> if >> > > > > >> > > > > >> > >>>>>>>>> it >> > > > > >> > > > > >> > >>>>>>>>> had been done in Kafka from the get-go. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> Vision >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> All of this leads me to a rather radical >> > > > > proposal. >> > > > > >> > Samza >> > > > > >> > > > is >> > > > > >> > > > > >> > >>>>> relatively >> > > > > >> > > > > >> > >>>>>>>>> stable at this point. I'd venture to say >> > that >> > > > > we're >> > > > > >> > > near a >> > > > > >> > > > > 1.0 >> > > > > >> > > > > >> > >>>>>> release. >> > > > > >> > > > > >> > >>>>>>>>> I'd >> > > > > >> > > > > >> > >>>>>>>>> like to propose that we take what we've >> > > > learned, >> > > > > and >> > > > > >> > > begin >> > > > > >> > > > > >> > >>>>>>>>> thinking >> > > > > >> > > > > >> > >>>>>>> about >> > > > > >> > > > > >> > >>>>>>>>> Samza beyond 1.0. What would we change >> if >> > we >> > > > were >> > > > > >> > > starting >> > > > > >> > > > > >> from >> > > > > >> > > > > >> > >>>>>> scratch? >> > > > > >> > > > > >> > >>>>>>>>> My >> > > > > >> > > > > >> > >>>>>>>>> proposal is to: >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> 1. Make Samza standalone the *only* way >> to >> > > run >> > > > > Samza >> > > > > >> > > > > >> > >>>>>>>>> processors, and eliminate all direct >> > > > dependences >> > > > > on >> > > > > >> > > YARN, >> > > > > >> > > > > >> Mesos, >> > > > > >> > > > > >> > >>>> etc. >> > > > > >> > > > > >> > >>>>>>>>> 2. Make a definitive call to support >> only >> > > Kafka >> > > > > as >> > > > > >> the >> > > > > >> > > > > stream >> > > > > >> > > > > >> > >>>>>> processing >> > > > > >> > > > > >> > >>>>>>>>> layer. >> > > > > >> > > > > >> > >>>>>>>>> 3. Eliminate Samza's metrics, logging, >> > > > > >> serialization, >> > > > > >> > > and >> > > > > >> > > > > >> > >>>>>>>>> config >> > > > > >> > > > > >> > >>>>>>> systems, >> > > > > >> > > > > >> > >>>>>>>>> and simply use Kafka's instead. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> This would fix all of the issues that I >> > > > outlined >> > > > > >> > above. >> > > > > >> > > It >> > > > > >> > > > > >> > >>>>>>>>> should >> > > > > >> > > > > >> > >>>>> also >> > > > > >> > > > > >> > >>>>>>>>> shrink the Samza code base pretty >> > > dramatically. >> > > > > >> > > Supporting >> > > > > >> > > > > >> only >> > > > > >> > > > > >> > >>>>>>>>> a standalone container will allow Samza >> to >> > be >> > > > > >> executed >> > > > > >> > > on >> > > > > >> > > > > YARN >> > > > > >> > > > > >> > >>>>>>>>> (using Slider), Mesos (using >> > > Marathon/Aurora), >> > > > or >> > > > > >> most >> > > > > >> > > > other >> > > > > >> > > > > >> > >>>>>>>>> in-house >> > > > > >> > > > > >> > >>>>>>> deployment >> > > > > >> > > > > >> > >>>>>>>>> systems. This should make life a lot >> easier >> > > for >> > > > > new >> > > > > >> > > users. >> > > > > >> > > > > >> > >>>>>>>>> Imagine >> > > > > >> > > > > >> > >>>>>>> having >> > > > > >> > > > > >> > >>>>>>>>> the hello-samza tutorial without YARN. >> The >> > > drop >> > > > > in >> > > > > >> > > mailing >> > > > > >> > > > > >> list >> > > > > >> > > > > >> > >>>>>> traffic >> > > > > >> > > > > >> > >>>>>>>>> will be pretty dramatic. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> Coupling with Kafka seems long overdue >> to >> > me. >> > > > The >> > > > > >> > > reality >> > > > > >> > > > > is, >> > > > > >> > > > > >> > >>>>> everyone >> > > > > >> > > > > >> > >>>>>>>>> that >> > > > > >> > > > > >> > >>>>>>>>> I'm aware of is using Samza with Kafka. >> We >> > > > > basically >> > > > > >> > > > require >> > > > > >> > > > > >> it >> > > > > >> > > > > >> > >>>>>> already >> > > > > >> > > > > >> > >>>>>>> in >> > > > > >> > > > > >> > >>>>>>>>> order for most features to work. Those >> that >> > > are >> > > > > >> using >> > > > > >> > > > other >> > > > > >> > > > > >> > >>>>>>>>> systems >> > > > > >> > > > > >> > >>>>>> are >> > > > > >> > > > > >> > >>>>>>>>> generally using it for ingest into Kafka >> > (1), >> > > > and >> > > > > >> then >> > > > > >> > > > they >> > > > > >> > > > > do >> > > > > >> > > > > >> > >>>>>>>>> the processing on top. There is already >> > > > > discussion ( >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>> >> > > > > >> > > > > >> > >>>>>> >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > > > > >> > >> > > > > >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=58851 >> > > > > >> > > > > >> > >>>>> 767 >> > > > > >> > > > > >> > >>>>>>>>> ) >> > > > > >> > > > > >> > >>>>>>>>> in Kafka to make ingesting into Kafka >> > > extremely >> > > > > >> easy. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> Once we make the call to couple with >> Kafka, >> > > we >> > > > > can >> > > > > >> > > > leverage >> > > > > >> > > > > a >> > > > > >> > > > > >> > >>>>>>>>> ton of >> > > > > >> > > > > >> > >>>>>>> their >> > > > > >> > > > > >> > >>>>>>>>> ecosystem. We no longer have to maintain >> > our >> > > > own >> > > > > >> > config, >> > > > > >> > > > > >> > >>>>>>>>> metrics, >> > > > > >> > > > > >> > >>>>> etc. >> > > > > >> > > > > >> > >>>>>>> We >> > > > > >> > > > > >> > >>>>>>>>> can all share the same libraries, and >> make >> > > them >> > > > > >> > better. >> > > > > >> > > > This >> > > > > >> > > > > >> > >>>>>>>>> will >> > > > > >> > > > > >> > >>>>> also >> > > > > >> > > > > >> > >>>>>>>>> allow us to share the consumer/producer >> > APIs, >> > > > and >> > > > > >> will >> > > > > >> > > let >> > > > > >> > > > > us >> > > > > >> > > > > >> > >>>>> leverage >> > > > > >> > > > > >> > >>>>>>>>> their offset management and partition >> > > > management, >> > > > > >> > rather >> > > > > >> > > > > than >> > > > > >> > > > > >> > >>>>>>>>> having >> > > > > >> > > > > >> > >>>>>> our >> > > > > >> > > > > >> > >>>>>>>>> own. All of the coordinator stream code >> > would >> > > > go >> > > > > >> away, >> > > > > >> > > as >> > > > > >> > > > > >> would >> > > > > >> > > > > >> > >>>>>>>>> most >> > > > > >> > > > > >> > >>>>>> of >> > > > > >> > > > > >> > >>>>>>>>> the >> > > > > >> > > > > >> > >>>>>>>>> YARN AppMaster code. We'd probably have >> to >> > > push >> > > > > some >> > > > > >> > > > > partition >> > > > > >> > > > > >> > >>>>>>> management >> > > > > >> > > > > >> > >>>>>>>>> features into the Kafka broker, but >> they're >> > > > > already >> > > > > >> > > moving >> > > > > >> > > > > in >> > > > > >> > > > > >> > >>>>>>>>> that direction with the new consumer >> API. >> > The >> > > > > >> features >> > > > > >> > > we >> > > > > >> > > > > have >> > > > > >> > > > > >> > >>>>>>>>> for >> > > > > >> > > > > >> > >>>>>> partition >> > > > > >> > > > > >> > >>>>>>>>> assignment aren't unique to Samza, and >> seem >> > > > like >> > > > > >> they >> > > > > >> > > > should >> > > > > >> > > > > >> be >> > > > > >> > > > > >> > >>>>>>>>> in >> > > > > >> > > > > >> > >>>>>> Kafka >> > > > > >> > > > > >> > >>>>>>>>> anyway. There will always be some niche >> > > usages >> > > > > which >> > > > > >> > > will >> > > > > >> > > > > >> > >>>>>>>>> require >> > > > > >> > > > > >> > >>>>>> extra >> > > > > >> > > > > >> > >>>>>>>>> care and hence full control over >> partition >> > > > > >> assignments >> > > > > >> > > > much >> > > > > >> > > > > >> > >>>>>>>>> like the >> > > > > >> > > > > >> > >>>>>>> Kafka >> > > > > >> > > > > >> > >>>>>>>>> low level consumer api. These would >> > continue >> > > to >> > > > > be >> > > > > >> > > > > supported. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> These items will be good for the Samza >> > > > community. >> > > > > >> > > They'll >> > > > > >> > > > > make >> > > > > >> > > > > >> > >>>>>>>>> Samza easier to use, and make it easier >> for >> > > > > >> developers >> > > > > >> > > to >> > > > > >> > > > > add >> > > > > >> > > > > >> > >>>>>>>>> new features. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> Obviously this is a fairly large (and >> > > somewhat >> > > > > >> > backwards >> > > > > >> > > > > >> > >>>>> incompatible >> > > > > >> > > > > >> > >>>>>>>>> change). If we choose to go this route, >> > it's >> > > > > >> important >> > > > > >> > > > that >> > > > > >> > > > > we >> > > > > >> > > > > >> > >>>>> openly >> > > > > >> > > > > >> > >>>>>>>>> communicate how we're going to provide a >> > > > > migration >> > > > > >> > path >> > > > > >> > > > from >> > > > > >> > > > > >> > >>>>>>>>> the >> > > > > >> > > > > >> > >>>>>>> existing >> > > > > >> > > > > >> > >>>>>>>>> APIs to the new ones (if we make >> > incompatible >> > > > > >> > changes). >> > > > > >> > > I >> > > > > >> > > > > >> think >> > > > > >> > > > > >> > >>>>>>>>> at a minimum, we'd probably need to >> > provide a >> > > > > >> wrapper >> > > > > >> > to >> > > > > >> > > > > allow >> > > > > >> > > > > >> > >>>>>>>>> existing StreamTask implementations to >> > > continue >> > > > > >> > running >> > > > > >> > > on >> > > > > >> > > > > the >> > > > > >> > > > > >> > >>>> new container. >> > > > > >> > > > > >> > >>>>>>> It's >> > > > > >> > > > > >> > >>>>>>>>> also important that we openly >> communicate >> > > about >> > > > > >> > timing, >> > > > > >> > > > and >> > > > > >> > > > > >> > >>>>>>>>> stages >> > > > > >> > > > > >> > >>>>> of >> > > > > >> > > > > >> > >>>>>>> the >> > > > > >> > > > > >> > >>>>>>>>> migration. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> If you made it this far, I'm sure you >> have >> > > > > opinions. >> > > > > >> > :) >> > > > > >> > > > > Please >> > > > > >> > > > > >> > >>>>>>>>> send >> > > > > >> > > > > >> > >>>>>> your >> > > > > >> > > > > >> > >>>>>>>>> thoughts and feedback. >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>>> Cheers, >> > > > > >> > > > > >> > >>>>>>>>> Chris >> > > > > >> > > > > >> > >>>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>>> >> > > > > >> > > > > >> > >>>>>>> >> > > > > >> > > > > >> > >>>>>> >> > > > > >> > > > > >> > >>>>>> >> > > > > >> > > > > >> > >>>>>> >> > > > > >> > > > > >> > >>>>>> -- >> > > > > >> > > > > >> > >>>>>> -- Guozhang >> > > > > >> > > > > >> > >>>>>> >> > > > > >> > > > > >> > >>>>> >> > > > > >> > > > > >> > >>>> >> > > > > >> > > > > >> > >> >> > > > > >> > > > > >> > >> >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > >> > > > > >> > > > > >> > >> > > > > >> > > > > >> >> > > > > >> > > > > > >> > > > > >> > > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > > > >> > > >> > > > > >> > >> > > > > >> >> > > > > >> > > > >> > > >> > >> > >
