During the summit, I also had a lot of discussions over similar topics with multiple Committers and active users. I heard many fantastic ideas. I believe Spark improvement proposals are good channels to collect the requirements/designs.
IMO, we also need to consider the priority when working on these items. Even if the proposal is accepted, it does not mean it will be implemented and merged immediately. It is not a FIFO queue. Even if some PRs are merged, sometimes, we still have to revert them back, if the design and implementation are not reviewed carefully. We have to ensure our quality. Spark is not an application software. It is an infrastructure software that is being used by many many companies. We have to be very careful in the design and implementation, especially adding/changing the external APIs. When I developed the Mainframe infrastructure/middleware software in the past 6 years, I were involved in the discussions with external/internal customers. The to-do feature list was always above 100. Sometimes, the customers are feeling frustrated when we are unable to deliver them on time due to the resource limits and others. Even if they paid us billions, we still need to do it phase by phase or sometimes they have to accept the workarounds. That is the reality everyone has to face, I think. Thanks, Xiao Li 2017-02-11 7:57 GMT-08:00 Cody Koeninger <c...@koeninger.org>: > At the spark summit this week, everyone from PMC members to users I had > never met before were asking me about the Spark improvement proposals > idea. It's clear that it's a real community need. > > But it's been almost half a year, and nothing visible has been done. > > Reynold, are you going to do this? > > If so, when? > > If not, why? > > You already did the right thing by including long-deserved committers. > Please keep doing the right thing for the community. > > On Wed, Jan 11, 2017 at 4:13 AM, Reynold Xin <r...@databricks.com> wrote: > >> +1 on all counts (consensus, time bound, define roles) >> >> I can update the doc in the next few days and share back. Then maybe we >> can just officially vote on this. As Tim suggested, we might not get it >> 100% right the first time and would need to re-iterate. But that's fine. >> >> >> On Thu, Jan 5, 2017 at 3:29 PM, Tim Hunter <timhun...@databricks.com> >> wrote: >> >>> Hi Cody, >>> thank you for bringing up this topic, I agree it is very important to >>> keep a cohesive community around some common, fluid goals. Here are a few >>> comments about the current document: >>> >>> 1. name: it should not overlap with an existing one such as SIP. Can you >>> imagine someone trying to discuss a scala spore proposal for spark? >>> "[Spark] SIP-3 is intended to evolve in tandem with [Scala] SIP-21". SPIP >>> sounds great. >>> >>> 2. roles: at a high level, SPIPs are meant to reach consensus for >>> technical decisions with a lasting impact. As such, the template should >>> emphasize the role of the various parties during this process: >>> >>> - the SPIP author is responsible for building consensus. She is the >>> champion driving the process forward and is responsible for ensuring that >>> the SPIP follows the general guidelines. The author should be identified in >>> the SPIP. The authorship of a SPIP can be transferred if the current author >>> is not interested and someone else wants to move the SPIP forward. There >>> should probably be 2-3 authors at most for each SPIP. >>> >>> - someone with voting power should probably shepherd the SPIP (and be >>> recorded as such): ensuring that the final decision over the SPIP is >>> recorded (rejected, accepted, etc.), and advising about the technical >>> quality of the SPIP: this person need not be a champion for the SPIP or >>> contribute to it, but rather makes sure it stands a chance of being >>> approved when the vote happens. Also, if the author cannot find anyone who >>> would want to take this role, this proposal is likely to be rejected anyway. >>> >>> - users, committers, contributors have the roles already outlined in >>> the document >>> >>> 3. timeline: ideally, once a SPIP has been offered for voting, it should >>> move swiftly into either being accepted or rejected, so that we do not end >>> up with a distracting long tail of half-hearted proposals. >>> >>> These rules are meant to be flexible, but the current document should be >>> clear about who is in charge of a SPIP, and the state it is currently in. >>> >>> We have had long discussions over some very important questions such as >>> approval. I do not have an opinion on these, but why not make a pick and >>> reevaluate this decision later? This is not a binding process at this point. >>> >>> Tim >>> >>> >>> On Tue, Jan 3, 2017 at 3:16 PM, Cody Koeninger <c...@koeninger.org> >>> wrote: >>> >>>> I don't have a concern about voting vs consensus. >>>> >>>> I have a concern that whatever the decision making process is, it is >>>> explicitly announced on the ticket for the given proposal, with an explicit >>>> deadline, and an explicit outcome. >>>> >>>> >>>> On Tue, Jan 3, 2017 at 4:08 PM, Imran Rashid <iras...@cloudera.com> >>>> wrote: >>>> >>>>> I'm also in favor of this. Thanks for your persistence Cody. >>>>> >>>>> My take on the specific issues Joseph mentioned: >>>>> >>>>> 1) voting vs. consensus -- I agree with the argument Ryan Blue made >>>>> earlier for consensus: >>>>> >>>>> > Majority vs consensus: My rationale is that I don't think we want to >>>>> consider a proposal approved if it had objections serious enough that >>>>> committers down-voted (or PMC depending on who gets a vote). If these >>>>> proposals are like PEPs, then they represent a significant amount of >>>>> community effort and I wouldn't want to move forward if up to half of the >>>>> community thinks it's an untenable idea. >>>>> >>>>> 2) Design doc template -- agree this would be useful, but also seems >>>>> totally orthogonal to moving forward on the SIP proposal. >>>>> >>>>> 3) agree w/ Joseph's proposal for updating the template. >>>>> >>>>> One small addition: >>>>> >>>>> 4) Deciding on a name -- minor, but I think its wroth disambiguating >>>>> from Scala's SIPs, and the best proposal I've heard is "SPIP". At least, >>>>> no one has objected. (don't care enough that I'd object to anything else, >>>>> though.) >>>>> >>>>> >>>>> On Tue, Jan 3, 2017 at 3:30 PM, Joseph Bradley <jos...@databricks.com> >>>>> wrote: >>>>> >>>>>> Hi Cody, >>>>>> >>>>>> Thanks for being persistent about this. I too would like to see this >>>>>> happen. Reviewing the thread, it sounds like the main things remaining >>>>>> are: >>>>>> * Decide about a few issues >>>>>> * Finalize the doc(s) >>>>>> * Vote on this proposal >>>>>> >>>>>> Issues & TODOs: >>>>>> >>>>>> (1) The main issue I see above is voting vs. consensus. I have >>>>>> little preference here. It sounds like something which could be tailored >>>>>> based on whether we see too many or too few SIPs being approved. >>>>>> >>>>>> (2) Design doc template (This would be great to have for Spark >>>>>> regardless of this SIP discussion.) >>>>>> * Reynold, are you still putting this together? >>>>>> >>>>>> (3) Template cleanups. Listing some items mentioned above + a new >>>>>> one w.r.t. Reynold's draft >>>>>> <https://docs.google.com/document/d/1-Zdi_W-wtuxS9hTK0P9qb2x-nRanvXmnZ7SUi4qMljg/edit#> >>>>>> : >>>>>> * Reinstate the "Where" section with links to current and past SIPs >>>>>> * Add field for stating explicit deadlines for approval >>>>>> * Add field for stating Author & Committer shepherd >>>>>> >>>>>> Thanks all! >>>>>> Joseph >>>>>> >>>>>> On Mon, Jan 2, 2017 at 7:45 AM, Cody Koeninger <c...@koeninger.org> >>>>>> wrote: >>>>>> >>>>>>> I'm bumping this one more time for the new year, and then I'm giving >>>>>>> up. >>>>>>> >>>>>>> Please, fix your process, even if it isn't exactly the way I >>>>>>> suggested. >>>>>>> >>>>>>> On Tue, Nov 8, 2016 at 11:14 AM, Ryan Blue <rb...@netflix.com> >>>>>>> wrote: >>>>>>> > On lazy consensus as opposed to voting: >>>>>>> > >>>>>>> > First, why lazy consensus? The proposal was for consensus, which >>>>>>> is at least >>>>>>> > three +1 votes and no vetos. Consensus has no losing side, it >>>>>>> requires >>>>>>> > getting to a point where there is agreement. Isn't that agreement >>>>>>> what we >>>>>>> > want to achieve with these proposals? >>>>>>> > >>>>>>> > Second, lazy consensus only removes the requirement for three +1 >>>>>>> votes. Why >>>>>>> > would we not want at least three committers to think something is >>>>>>> a good >>>>>>> > idea before adopting the proposal? >>>>>>> > >>>>>>> > rb >>>>>>> > >>>>>>> > On Tue, Nov 8, 2016 at 8:13 AM, Cody Koeninger <c...@koeninger.org> >>>>>>> wrote: >>>>>>> >> >>>>>>> >> So there are some minor things (the Where section heading appears >>>>>>> to >>>>>>> >> be dropped; wherever this document is posted it needs to actually >>>>>>> link >>>>>>> >> to a jira filter showing current / past SIPs) but it doesn't look >>>>>>> like >>>>>>> >> I can comment on the google doc. >>>>>>> >> >>>>>>> >> The major substantive issue that I have is that this version is >>>>>>> >> significantly less clear as to the outcome of an SIP. >>>>>>> >> >>>>>>> >> The apache example of lazy consensus at >>>>>>> >> http://apache.org/foundation/voting.html#LazyConsensus involves >>>>>>> an >>>>>>> >> explicit announcement of an explicit deadline, which I think are >>>>>>> >> necessary for clarity. >>>>>>> >> >>>>>>> >> >>>>>>> >> >>>>>>> >> On Mon, Nov 7, 2016 at 1:55 PM, Reynold Xin <r...@databricks.com> >>>>>>> wrote: >>>>>>> >> > It turned out suggested edits (trackable) don't show up for >>>>>>> non-owners, >>>>>>> >> > so >>>>>>> >> > I've just merged all the edits in place. It should be visible >>>>>>> now. >>>>>>> >> > >>>>>>> >> > On Mon, Nov 7, 2016 at 10:10 AM, Reynold Xin < >>>>>>> r...@databricks.com> >>>>>>> >> > wrote: >>>>>>> >> >> >>>>>>> >> >> Oops. Let me try figure that out. >>>>>>> >> >> >>>>>>> >> >> >>>>>>> >> >> On Monday, November 7, 2016, Cody Koeninger < >>>>>>> c...@koeninger.org> wrote: >>>>>>> >> >>> >>>>>>> >> >>> Thanks for picking up on this. >>>>>>> >> >>> >>>>>>> >> >>> Maybe I fail at google docs, but I can't see any edits on the >>>>>>> document >>>>>>> >> >>> you linked. >>>>>>> >> >>> >>>>>>> >> >>> Regarding lazy consensus, if the board in general has less of >>>>>>> an issue >>>>>>> >> >>> with that, sure. As long as it is clearly announced, lasts >>>>>>> at least >>>>>>> >> >>> 72 hours, and has a clear outcome. >>>>>>> >> >>> >>>>>>> >> >>> The other points are hard to comment on without being able to >>>>>>> see the >>>>>>> >> >>> text in question. >>>>>>> >> >>> >>>>>>> >> >>> >>>>>>> >> >>> On Mon, Nov 7, 2016 at 3:11 AM, Reynold Xin < >>>>>>> r...@databricks.com> >>>>>>> >> >>> wrote: >>>>>>> >> >>> > I just looked through the entire thread again tonight - >>>>>>> there are a >>>>>>> >> >>> > lot >>>>>>> >> >>> > of >>>>>>> >> >>> > great ideas being discussed. Thanks Cody for taking the >>>>>>> first crack >>>>>>> >> >>> > at >>>>>>> >> >>> > the >>>>>>> >> >>> > proposal. >>>>>>> >> >>> > >>>>>>> >> >>> > I want to first comment on the context. Spark is one of the >>>>>>> most >>>>>>> >> >>> > innovative >>>>>>> >> >>> > and important projects in (big) data -- overall technical >>>>>>> decisions >>>>>>> >> >>> > made in >>>>>>> >> >>> > Apache Spark are sound. But of course, a project as large >>>>>>> and active >>>>>>> >> >>> > as >>>>>>> >> >>> > Spark always have room for improvement, and we as a >>>>>>> community should >>>>>>> >> >>> > strive >>>>>>> >> >>> > to take it to the next level. >>>>>>> >> >>> > >>>>>>> >> >>> > To that end, the two biggest areas for improvements in my >>>>>>> opinion >>>>>>> >> >>> > are: >>>>>>> >> >>> > >>>>>>> >> >>> > 1. Visibility: There are so much happening that it is >>>>>>> difficult to >>>>>>> >> >>> > know >>>>>>> >> >>> > what >>>>>>> >> >>> > really is going on. For people that don't follow closely, >>>>>>> it is >>>>>>> >> >>> > difficult to >>>>>>> >> >>> > know what the important initiatives are. Even for people >>>>>>> that do >>>>>>> >> >>> > follow, it >>>>>>> >> >>> > is difficult to know what specific things require their >>>>>>> attention, >>>>>>> >> >>> > since the >>>>>>> >> >>> > number of pull requests and JIRA tickets are high and it's >>>>>>> difficult >>>>>>> >> >>> > to >>>>>>> >> >>> > extract signal from noise. >>>>>>> >> >>> > >>>>>>> >> >>> > 2. Solicit user (broadly defined, including developers >>>>>>> themselves) >>>>>>> >> >>> > input >>>>>>> >> >>> > more proactively: At the end of the day the project >>>>>>> provides value >>>>>>> >> >>> > because >>>>>>> >> >>> > users use it. Users can't tell us exactly what to build, >>>>>>> but it is >>>>>>> >> >>> > important >>>>>>> >> >>> > to get their inputs. >>>>>>> >> >>> > >>>>>>> >> >>> > >>>>>>> >> >>> > I've taken Cody's doc and edited it: >>>>>>> >> >>> > >>>>>>> >> >>> > >>>>>>> >> >>> > https://docs.google.com/docume >>>>>>> nt/d/1-Zdi_W-wtuxS9hTK0P9qb2x-nRanvXmnZ7SUi4qMljg/edit#headi >>>>>>> ng=h.36ut37zh7w2b >>>>>>> >> >>> > (I've made all my modifications trackable) >>>>>>> >> >>> > >>>>>>> >> >>> > There are couple high level changes I made: >>>>>>> >> >>> > >>>>>>> >> >>> > 1. I've consulted a board member and he recommended lazy >>>>>>> consensus >>>>>>> >> >>> > as >>>>>>> >> >>> > opposed to voting. The reason being in voting there can >>>>>>> easily be a >>>>>>> >> >>> > "loser' >>>>>>> >> >>> > that gets outvoted. >>>>>>> >> >>> > >>>>>>> >> >>> > 2. I made it lighter weight, and renamed "strategy" to >>>>>>> "optional >>>>>>> >> >>> > design >>>>>>> >> >>> > sketch". Echoing one of the earlier email: "IMHO so far >>>>>>> aside from >>>>>>> >> >>> > tagging >>>>>>> >> >>> > things and linking them elsewhere simply having design docs >>>>>>> and >>>>>>> >> >>> > prototypes >>>>>>> >> >>> > implementations in PRs is not something that has not worked >>>>>>> so far". >>>>>>> >> >>> > >>>>>>> >> >>> > 3. I made some the language tweaks to focus more on >>>>>>> visibility. For >>>>>>> >> >>> > example, >>>>>>> >> >>> > "The purpose of an SIP is to inform and involve", rather >>>>>>> than just >>>>>>> >> >>> > "involve". SIPs should also have at least two emails that >>>>>>> go to >>>>>>> >> >>> > dev@. >>>>>>> >> >>> > >>>>>>> >> >>> > >>>>>>> >> >>> > While I was editing this, I thought we really needed a >>>>>>> suggested >>>>>>> >> >>> > template >>>>>>> >> >>> > for design doc too. I will get to that too ... >>>>>>> >> >>> > >>>>>>> >> >>> > >>>>>>> >> >>> > On Tue, Nov 1, 2016 at 12:09 AM, Reynold Xin < >>>>>>> r...@databricks.com> >>>>>>> >> >>> > wrote: >>>>>>> >> >>> >> >>>>>>> >> >>> >> Most things looked OK to me too, although I do plan to >>>>>>> take a >>>>>>> >> >>> >> closer >>>>>>> >> >>> >> look >>>>>>> >> >>> >> after Nov 1st when we cut the release branch for 2.1. >>>>>>> >> >>> >> >>>>>>> >> >>> >> >>>>>>> >> >>> >> On Mon, Oct 31, 2016 at 3:12 PM, Marcelo Vanzin >>>>>>> >> >>> >> <van...@cloudera.com> >>>>>>> >> >>> >> wrote: >>>>>>> >> >>> >>> >>>>>>> >> >>> >>> The proposal looks OK to me. I assume, even though it's >>>>>>> not >>>>>>> >> >>> >>> explicitly >>>>>>> >> >>> >>> called, that voting would happen by e-mail? A template >>>>>>> for the >>>>>>> >> >>> >>> proposal document (instead of just a bullet nice) would >>>>>>> also be >>>>>>> >> >>> >>> nice, >>>>>>> >> >>> >>> but that can be done at any time. >>>>>>> >> >>> >>> >>>>>>> >> >>> >>> BTW, shameless plug: I filed SPARK-18085 which I consider >>>>>>> a >>>>>>> >> >>> >>> candidate >>>>>>> >> >>> >>> for a SIP, given the scope of the work. The document >>>>>>> attached even >>>>>>> >> >>> >>> somewhat matches the proposed format. So if anyone wants >>>>>>> to try >>>>>>> >> >>> >>> out >>>>>>> >> >>> >>> the process... >>>>>>> >> >>> >>> >>>>>>> >> >>> >>> On Mon, Oct 31, 2016 at 10:34 AM, Cody Koeninger >>>>>>> >> >>> >>> <c...@koeninger.org> >>>>>>> >> >>> >>> wrote: >>>>>>> >> >>> >>> > Now that spark summit europe is over, are any committers >>>>>>> >> >>> >>> > interested >>>>>>> >> >>> >>> > in >>>>>>> >> >>> >>> > moving forward with this? >>>>>>> >> >>> >>> > >>>>>>> >> >>> >>> > >>>>>>> >> >>> >>> > >>>>>>> >> >>> >>> > >>>>>>> >> >>> >>> > https://github.com/koeninger/s >>>>>>> park-1/blob/SIP-0/docs/spark-improvement-proposals.md >>>>>>> >> >>> >>> > >>>>>>> >> >>> >>> > Or are we going to let this discussion die on the vine? >>>>>>> >> >>> >>> > >>>>>>> >> >>> >>> > On Mon, Oct 17, 2016 at 10:05 AM, Tomasz Gawęda >>>>>>> >> >>> >>> > <tomasz.gaw...@outlook.com> wrote: >>>>>>> >> >>> >>> >> Maybe my mail was not clear enough. >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> I didn't want to write "lets focus on Flink" or any >>>>>>> other >>>>>>> >> >>> >>> >> framework. >>>>>>> >> >>> >>> >> The >>>>>>> >> >>> >>> >> idea with benchmarks was to show two things: >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> - why some people are doing bad PR for Spark >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> - how - in easy way - we can change it and show that >>>>>>> Spark is >>>>>>> >> >>> >>> >> still on >>>>>>> >> >>> >>> >> the >>>>>>> >> >>> >>> >> top >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> No more, no less. Benchmarks will be helpful, but I >>>>>>> don't think >>>>>>> >> >>> >>> >> they're the >>>>>>> >> >>> >>> >> most important thing in Spark :) On the Spark main >>>>>>> page there >>>>>>> >> >>> >>> >> is >>>>>>> >> >>> >>> >> still >>>>>>> >> >>> >>> >> chart >>>>>>> >> >>> >>> >> "Spark vs Hadoop". It is important to show that >>>>>>> framework is >>>>>>> >> >>> >>> >> not >>>>>>> >> >>> >>> >> the >>>>>>> >> >>> >>> >> same >>>>>>> >> >>> >>> >> Spark with other API, but much faster and optimized, >>>>>>> comparable >>>>>>> >> >>> >>> >> or >>>>>>> >> >>> >>> >> even >>>>>>> >> >>> >>> >> faster than other frameworks. >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> About real-time streaming, I think it would be just >>>>>>> good to see >>>>>>> >> >>> >>> >> it >>>>>>> >> >>> >>> >> in >>>>>>> >> >>> >>> >> Spark. >>>>>>> >> >>> >>> >> I very like current Spark model, but many voices that >>>>>>> says "we >>>>>>> >> >>> >>> >> need >>>>>>> >> >>> >>> >> more" - >>>>>>> >> >>> >>> >> community should listen also them and try to help >>>>>>> them. With >>>>>>> >> >>> >>> >> SIPs >>>>>>> >> >>> >>> >> it >>>>>>> >> >>> >>> >> would >>>>>>> >> >>> >>> >> be easier, I've just posted this example as "thing >>>>>>> that may be >>>>>>> >> >>> >>> >> changed >>>>>>> >> >>> >>> >> with >>>>>>> >> >>> >>> >> SIP". >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> I very like unification via Datasets, but there is a >>>>>>> lot of >>>>>>> >> >>> >>> >> algorithms >>>>>>> >> >>> >>> >> inside - let's make easy API, but with strong >>>>>>> background >>>>>>> >> >>> >>> >> (articles, >>>>>>> >> >>> >>> >> benchmarks, descriptions, etc) that shows that Spark >>>>>>> is still >>>>>>> >> >>> >>> >> modern >>>>>>> >> >>> >>> >> framework. >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> Maybe now my intention will be clearer :) As I said >>>>>>> >> >>> >>> >> organizational >>>>>>> >> >>> >>> >> ideas >>>>>>> >> >>> >>> >> were already mentioned and I agree with them, my mail >>>>>>> was just >>>>>>> >> >>> >>> >> to >>>>>>> >> >>> >>> >> show >>>>>>> >> >>> >>> >> some >>>>>>> >> >>> >>> >> aspects from my side, so from theside of developer and >>>>>>> person >>>>>>> >> >>> >>> >> who >>>>>>> >> >>> >>> >> is >>>>>>> >> >>> >>> >> trying >>>>>>> >> >>> >>> >> to help others with Spark (via StackOverflow or other >>>>>>> ways) >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> Pozdrawiam / Best regards, >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> Tomasz >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> ________________________________ >>>>>>> >> >>> >>> >> Od: Cody Koeninger <c...@koeninger.org> >>>>>>> >> >>> >>> >> Wysłane: 17 października 2016 16:46 >>>>>>> >> >>> >>> >> Do: Debasish Das >>>>>>> >> >>> >>> >> DW: Tomasz Gawęda; dev@spark.apache.org >>>>>>> >> >>> >>> >> Temat: Re: Spark Improvement Proposals >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> I think narrowly focusing on Flink or benchmarks is >>>>>>> missing my >>>>>>> >> >>> >>> >> point. >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> My point is evolve or die. Spark's governance and >>>>>>> organization >>>>>>> >> >>> >>> >> is >>>>>>> >> >>> >>> >> hampering its ability to evolve technologically, and >>>>>>> it needs >>>>>>> >> >>> >>> >> to >>>>>>> >> >>> >>> >> change. >>>>>>> >> >>> >>> >> >>>>>>> >> >>> >>> >> On Sun, Oct 16, 2016 at 9:21 PM, Debasish Das >>>>>>> >> >>> >>> >> <debasish.da...@gmail.com> >>>>>>> >> >>> >>> >> wrote: >>>>>>> >> >>> >>> >>> Thanks Cody for bringing up a valid point...I picked >>>>>>> up Spark >>>>>>> >> >>> >>> >>> in >>>>>>> >> >>> >>> >>> 2014 >>>>>>> >> >>> >>> >>> as >>>>>>> >> >>> >>> >>> soon as I looked into it since compared to writing >>>>>>> Java >>>>>>> >> >>> >>> >>> map-reduce >>>>>>> >> >>> >>> >>> and >>>>>>> >> >>> >>> >>> Cascading code, Spark made writing distributed code >>>>>>> fun...But >>>>>>> >> >>> >>> >>> now >>>>>>> >> >>> >>> >>> as >>>>>>> >> >>> >>> >>> we >>>>>>> >> >>> >>> >>> went >>>>>>> >> >>> >>> >>> deeper with Spark and real-time streaming use-case >>>>>>> gets more >>>>>>> >> >>> >>> >>> prominent, I >>>>>>> >> >>> >>> >>> think it is time to bring a messaging model in >>>>>>> conjunction >>>>>>> >> >>> >>> >>> with >>>>>>> >> >>> >>> >>> the >>>>>>> >> >>> >>> >>> batch/micro-batch API that Spark is good >>>>>>> at....akka-streams >>>>>>> >> >>> >>> >>> close >>>>>>> >> >>> >>> >>> integration with spark micro-batching APIs looks like >>>>>>> a great >>>>>>> >> >>> >>> >>> direction to >>>>>>> >> >>> >>> >>> stay in the game with Apache Flink...Spark 2.0 >>>>>>> integrated >>>>>>> >> >>> >>> >>> streaming >>>>>>> >> >>> >>> >>> with >>>>>>> >> >>> >>> >>> batch with the assumption is that micro-batching is >>>>>>> sufficient >>>>>>> >> >>> >>> >>> to >>>>>>> >> >>> >>> >>> run >>>>>>> >> >>> >>> >>> SQL >>>>>>> >> >>> >>> >>> commands on stream but do we really have time to do >>>>>>> SQL >>>>>>> >> >>> >>> >>> processing at >>>>>>> >> >>> >>> >>> streaming data within 1-2 seconds ? >>>>>>> >> >>> >>> >>> >>>>>>> >> >>> >>> >>> After reading the email chain, I started to look into >>>>>>> Flink >>>>>>> >> >>> >>> >>> documentation >>>>>>> >> >>> >>> >>> and if you compare it with Spark documentation, I >>>>>>> think we >>>>>>> >> >>> >>> >>> have >>>>>>> >> >>> >>> >>> major >>>>>>> >> >>> >>> >>> work >>>>>>> >> >>> >>> >>> to do detailing out Spark internals so that more >>>>>>> people from >>>>>>> >> >>> >>> >>> community >>>>>>> >> >>> >>> >>> start >>>>>>> >> >>> >>> >>> to take active role in improving the issues so that >>>>>>> Spark >>>>>>> >> >>> >>> >>> stays >>>>>>> >> >>> >>> >>> strong >>>>>>> >> >>> >>> >>> compared to Flink. >>>>>>> >> >>> >>> >>> >>>>>>> >> >>> >>> >>> >>>>>>> >> >>> >>> >>> https://cwiki.apache.org/confl >>>>>>> uence/display/SPARK/Spark+Internals >>>>>>> >> >>> >>> >>> >>>>>>> >> >>> >>> >>> >>>>>>> >> >>> >>> >>> https://cwiki.apache.org/confl >>>>>>> uence/display/FLINK/Flink+Internals >>>>>>> >> >>> >>> >>> >>>>>>> >> >>> >>> >>> Spark is no longer an engine that works for >>>>>>> micro-batch and >>>>>>> >> >>> >>> >>> batch...We >>>>>>> >> >>> >>> >>> (and >>>>>>> >> >>> >>> >>> I am sure many others) are pushing spark as an engine >>>>>>> for >>>>>>> >> >>> >>> >>> stream >>>>>>> >> >>> >>> >>> and >>>>>>> >> >>> >>> >>> query >>>>>>> >> >>> >>> >>> processing.....we need to make it a state-of-the-art >>>>>>> engine >>>>>>> >> >>> >>> >>> for >>>>>>> >> >>> >>> >>> high >>>>>>> >> >>> >>> >>> speed >>>>>>> >> >>> >>> >>> streaming data and user queries as well ! >>>>>>> >> >>> >>> >>> >>>>>>> >> >>> >>> >>> On Sun, Oct 16, 2016 at 1:30 PM, Tomasz Gawęda >>>>>>> >> >>> >>> >>> <tomasz.gaw...@outlook.com> >>>>>>> >> >>> >>> >>> wrote: >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> Hi everyone, >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> I'm quite late with my answer, but I think my >>>>>>> suggestions may >>>>>>> >> >>> >>> >>>> help a >>>>>>> >> >>> >>> >>>> little bit. :) Many technical and organizational >>>>>>> topics were >>>>>>> >> >>> >>> >>>> mentioned, >>>>>>> >> >>> >>> >>>> but I want to focus on these negative posts about >>>>>>> Spark and >>>>>>> >> >>> >>> >>>> about >>>>>>> >> >>> >>> >>>> "haters" >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> I really like Spark. Easy of use, speed, very good >>>>>>> community >>>>>>> >> >>> >>> >>>> - >>>>>>> >> >>> >>> >>>> it's >>>>>>> >> >>> >>> >>>> everything here. But Every project has to "flight" on >>>>>>> >> >>> >>> >>>> "framework >>>>>>> >> >>> >>> >>>> market" >>>>>>> >> >>> >>> >>>> to be still no 1. I'm following many Spark and Big >>>>>>> Data >>>>>>> >> >>> >>> >>>> communities, >>>>>>> >> >>> >>> >>>> maybe my mail will inspire someone :) >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> You (every Spark developer; so far I didn't have >>>>>>> enough time >>>>>>> >> >>> >>> >>>> to >>>>>>> >> >>> >>> >>>> join >>>>>>> >> >>> >>> >>>> contributing to Spark) has done excellent job. So >>>>>>> why are >>>>>>> >> >>> >>> >>>> some >>>>>>> >> >>> >>> >>>> people >>>>>>> >> >>> >>> >>>> saying that Flink (or other framework) is better, >>>>>>> like it was >>>>>>> >> >>> >>> >>>> posted >>>>>>> >> >>> >>> >>>> in >>>>>>> >> >>> >>> >>>> this mailing list? No, not because that framework is >>>>>>> better >>>>>>> >> >>> >>> >>>> in >>>>>>> >> >>> >>> >>>> all >>>>>>> >> >>> >>> >>>> cases.. In my opinion, many of these discussions >>>>>>> where >>>>>>> >> >>> >>> >>>> started >>>>>>> >> >>> >>> >>>> after >>>>>>> >> >>> >>> >>>> Flink marketing-like posts. Please look at >>>>>>> StackOverflow >>>>>>> >> >>> >>> >>>> "Flink >>>>>>> >> >>> >>> >>>> vs >>>>>>> >> >>> >>> >>>> ...." >>>>>>> >> >>> >>> >>>> posts, almost every post in "winned" by Flink. >>>>>>> Answers are >>>>>>> >> >>> >>> >>>> sometimes >>>>>>> >> >>> >>> >>>> saying nothing about other frameworks, Flink's users >>>>>>> (often >>>>>>> >> >>> >>> >>>> PMC's) >>>>>>> >> >>> >>> >>>> are >>>>>>> >> >>> >>> >>>> just posting same information about real-time >>>>>>> streaming, >>>>>>> >> >>> >>> >>>> about >>>>>>> >> >>> >>> >>>> delta >>>>>>> >> >>> >>> >>>> iterations, etc. It look smart and very often it is >>>>>>> marked as >>>>>>> >> >>> >>> >>>> an >>>>>>> >> >>> >>> >>>> aswer, >>>>>>> >> >>> >>> >>>> even if - in my opinion - there wasn't told all the >>>>>>> truth. >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> My suggestion: I don't have enough money and >>>>>>> knowledgle to >>>>>>> >> >>> >>> >>>> perform >>>>>>> >> >>> >>> >>>> huge >>>>>>> >> >>> >>> >>>> performance test. Maybe some company, that supports >>>>>>> Spark >>>>>>> >> >>> >>> >>>> (Databricks, >>>>>>> >> >>> >>> >>>> Cloudera? - just saying you're most visible in >>>>>>> community :) ) >>>>>>> >> >>> >>> >>>> could >>>>>>> >> >>> >>> >>>> perform performance test of: >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> - streaming engine - probably Spark will loose >>>>>>> because of >>>>>>> >> >>> >>> >>>> mini-batch >>>>>>> >> >>> >>> >>>> model, however currently the difference should be >>>>>>> much lower >>>>>>> >> >>> >>> >>>> that in >>>>>>> >> >>> >>> >>>> previous versions >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> - Machine Learning models >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> - batch jobs >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> - Graph jobs >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> - SQL queries >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> People will see that Spark is envolving and is also >>>>>>> a modern >>>>>>> >> >>> >>> >>>> framework, >>>>>>> >> >>> >>> >>>> because after reading posts mentioned above people >>>>>>> may think >>>>>>> >> >>> >>> >>>> "it >>>>>>> >> >>> >>> >>>> is >>>>>>> >> >>> >>> >>>> outdated, future is in framework X". >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> Matei Zaharia posted excellent blog post about how >>>>>>> Spark >>>>>>> >> >>> >>> >>>> Structured >>>>>>> >> >>> >>> >>>> Streaming beats every other framework in terms of >>>>>>> easy-of-use >>>>>>> >> >>> >>> >>>> and >>>>>>> >> >>> >>> >>>> reliability. Performance tests, done in various >>>>>>> environments >>>>>>> >> >>> >>> >>>> (in >>>>>>> >> >>> >>> >>>> example: laptop, small 2 node cluster, 10-node >>>>>>> cluster, >>>>>>> >> >>> >>> >>>> 20-node >>>>>>> >> >>> >>> >>>> cluster), could be also very good marketing stuff to >>>>>>> say >>>>>>> >> >>> >>> >>>> "hey, >>>>>>> >> >>> >>> >>>> you're >>>>>>> >> >>> >>> >>>> telling that you're better, but Spark is still >>>>>>> faster and is >>>>>>> >> >>> >>> >>>> still >>>>>>> >> >>> >>> >>>> getting even more fast!". This would be based on >>>>>>> facts (just >>>>>>> >> >>> >>> >>>> numbers), >>>>>>> >> >>> >>> >>>> not opinions. It would be good for companies, for >>>>>>> marketing >>>>>>> >> >>> >>> >>>> puproses >>>>>>> >> >>> >>> >>>> and >>>>>>> >> >>> >>> >>>> for every Spark developer >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> Second: real-time streaming. I've written some time >>>>>>> ago about >>>>>>> >> >>> >>> >>>> real-time >>>>>>> >> >>> >>> >>>> streaming support in Spark Structured Streaming. >>>>>>> Some work >>>>>>> >> >>> >>> >>>> should be >>>>>>> >> >>> >>> >>>> done to make SSS more low-latency, but I think it's >>>>>>> possible. >>>>>>> >> >>> >>> >>>> Maybe >>>>>>> >> >>> >>> >>>> Spark may look at Gearpump, which is also built on >>>>>>> top of >>>>>>> >> >>> >>> >>>> Akka? >>>>>>> >> >>> >>> >>>> I >>>>>>> >> >>> >>> >>>> don't >>>>>>> >> >>> >>> >>>> know yet, it is good topic for SIP. However I think >>>>>>> that >>>>>>> >> >>> >>> >>>> Spark >>>>>>> >> >>> >>> >>>> should >>>>>>> >> >>> >>> >>>> have real-time streaming support. Currently I see >>>>>>> many >>>>>>> >> >>> >>> >>>> posts/comments >>>>>>> >> >>> >>> >>>> that "Spark has too big latency". Spark Streaming is >>>>>>> doing >>>>>>> >> >>> >>> >>>> very >>>>>>> >> >>> >>> >>>> good >>>>>>> >> >>> >>> >>>> jobs with micro-batches, however I think it is >>>>>>> possible to >>>>>>> >> >>> >>> >>>> add >>>>>>> >> >>> >>> >>>> also >>>>>>> >> >>> >>> >>>> more >>>>>>> >> >>> >>> >>>> real-time processing. >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> Other people said much more and I agree with >>>>>>> proposal of SIP. >>>>>>> >> >>> >>> >>>> I'm >>>>>>> >> >>> >>> >>>> also >>>>>>> >> >>> >>> >>>> happy that PMC's are not saying that they will not >>>>>>> listen to >>>>>>> >> >>> >>> >>>> users, >>>>>>> >> >>> >>> >>>> but >>>>>>> >> >>> >>> >>>> they really want to make Spark better for every user. >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> What do you think about these two topics? Especially >>>>>>> I'm >>>>>>> >> >>> >>> >>>> looking >>>>>>> >> >>> >>> >>>> at >>>>>>> >> >>> >>> >>>> Cody >>>>>>> >> >>> >>> >>>> (who has started this topic) and PMCs :) >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> Pozdrawiam / Best regards, >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> Tomasz >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>> >>>>>>> >> >>> >>> >>>>>>> >> >>> >> >>>>>>> >> >>> > >>>>>>> >> >>> > >>>>>>> >> > >>>>>>> >> > >>>>>>> >> >>>>>>> >> ------------------------------------------------------------ >>>>>>> --------- >>>>>>> >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>> >> >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > -- >>>>>>> > Ryan Blue >>>>>>> > Software Engineer >>>>>>> > Netflix >>>>>>> >>>>>>> ------------------------------------------------------------ >>>>>>> --------- >>>>>>> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Joseph Bradley >>>>>> >>>>>> Software Engineer - Machine Learning >>>>>> >>>>>> Databricks, Inc. >>>>>> >>>>>> [image: http://databricks.com] <http://databricks.com/> >>>>>> >>>>> >>>>> >>>> >>> >> >
