Thanks for going through it Gyula. I've made the necessary amends to the timeline and submitted the proposal.
Regards, Akshay Dixit On Thu, Mar 26, 2015 at 8:53 PM, Gyula Fóra <gyf...@apache.org> wrote: > I think it looks good for a start, we will have to work on the API a little > bit together to make it fit smoothly with what we currently have. > > There is a few gaps in the timeline but that you have probably noticed :) > > Otherwise +1 from me. > > On Wed, Mar 25, 2015 at 11:35 PM, Akshay Dixit <akshayd...@gmail.com> > wrote: > > > Hi, > > The link to the draft proposal that I've prepared is > > https://gist.github.com/akshaydixi/88f3fbcebab0119a6a31 > > It would be great if I could get some feedback on it. > > Regards, > > Akshay Dixit > > > > On Wed, Mar 25, 2015 at 2:03 AM, Akshay Dixit <akshayd...@gmail.com> > > wrote: > > > > > Thanks Gyula. > > > > > > I agree too that simple and working implementations are preferrable > over > > > hacky complex solutions. I'll start sketching out an initial > > straighforward > > > API with only basic pattern matching features > > > and base it on the existing windowing API. I'll post a draft of the > > > proposal, keeping the points you've said in mind, tomorrow, so you can > > > look it over to see if its all right. > > > Regards, > > > Akshay Dixit > > > > > > On Tue, Mar 24, 2015 at 6:30 PM, Gyula Fóra <gyf...@apache.org> wrote: > > > > > >> Hey Dixit, > > >> > > >> Sorry for the delay, I had to discuss this in more detail with some of > > our > > >> other core developers. > > >> > > >> The consensus seems to be that we would like push this project in a > > >> direction where the changes can be quickly included in the next > > releases. > > >> For this it is essential that we implement features that are complete > > (and > > >> clean) from the users perspective. This does not necessarily mean that > > we > > >> would like to have everything at once but rather that it is preferable > > to > > >> start with something clean and simple (for instance the naive chained > > >> filter approach) and progressively build more complex logic. > > >> > > >> This also mean that we would like to avoid "researchy" code in the > > >> codebase > > >> as much as possible. Of course once we have a stable api for this > > >> functionality we can work towards making the optimizations that you > have > > >> mentioned like operator sharing and so on. > > >> > > >> The ideal proposal would give a clear sketch of the pattern matching > API > > >> that you would like to implement, which might be some added operators > at > > >> first to the current API and possible a DSL later with more advanced > > >> functionality (this would probably go in a separate library until it > is > > >> very stable). > > >> > > >> So please in the proposal include a preview of what the pattern > matching > > >> syntax would look like integrated with the current operators, how it > > would > > >> interact with other parts of the system etc. > > >> > > >> These are the thing we need to figure out before we consider the > > >> optimizations I think, because it usually turns out, that the API > > >> semantics > > >> you would like to provide can hugely affect (probably limit) the > > >> possibilities that you have afterwards in terms of optimizations. > > >> > > >> Let me know if you have further questions regarding this :) > > >> > > >> Gyula > > >> > > >> On Tue, Mar 24, 2015 at 12:01 PM, Gyula Fóra <gyf...@apache.org> > wrote: > > >> > > >> > Hey, > > >> > > > >> > Give me an hour or so as I am in a meeting currently, but I will get > > >> back > > >> > to you afterwards. > > >> > > > >> > Regards, > > >> > Gyula > > >> > > > >> > On Tue, Mar 24, 2015 at 11:03 AM, Akshay Dixit < > akshayd...@gmail.com> > > >> > wrote: > > >> > > > >> >> Hi, > > >> >> It'd really help if I got a reply soon. It'll be helpful in writing > > the > > >> >> proposal since the deadline is on 27th. Thanks > > >> >> Regards, > > >> >> Akshay Dixit > > >> >> > > >> >> On Sun, Mar 22, 2015 at 1:17 AM, Akshay Dixit < > akshayd...@gmail.com> > > >> >> wrote: > > >> >> > > >> >> > Thanks for the explanation Marton. I've decided to try out for > > >> >> FLINK-1534. > > >> >> > > > >> >> > After reading through the thesis[4] and a few other > > papers[1][2][3], > > >> I > > >> >> > believe I've gathered a little context to ask more questions. But > > I'm > > >> >> still > > >> >> > not sure how Flink's internals work > > >> >> > so please bear with me. Although the ongoing effort to document > the > > >> >> > architecture and internal is really helpful for newbies like me > and > > >> >> would > > >> >> > greatly decrease the ramping up time. > > >> >> > > > >> >> > Detecting a pattern of events would comprise of a pipeline that > > >> accepts > > >> >> > the pattern query and > > >> >> > sources of DataStreams, and outputs detected matches of that > > pattern > > >> to > > >> >> a > > >> >> > sink or forwards it > > >> >> > along to another stream for further computation. > > >> >> > > > >> >> > As you said, a simple filter-join-aggregate query system could be > > >> >> > developed implementing using the existing Streaming windowing > API. > > >> >> > But matching over complex events and decoding their pattern > queries > > >> >> would > > >> >> > require implementing a DSL that transforms queries into an > > evaluation > > >> >> > model. For e.g, > > >> >> > in [1], the authors have implemented an NFA automaton with a > shared > > >> >> > versioned buffer that models the queries. In [4], the authors > > >> >> > propose a new language that is much more expressive and compiles > > >> into a > > >> >> > topology graph for Storm. > > >> >> > > > >> >> > So in Flink's case, I believe the proposed DSL would generate > > >> operator > > >> >> > graphs for the Flink compiler to schedule Jobgraphs over > > >> TaskManagers. > > >> >> > If we don't depend on the Windowing API, would we need to create > > new > > >> >> > operators such as the Projection, Conjunction and Union operators > > >> >> defined > > >> >> > in [4] ? > > >> >> > Also I would like to hear your thoughts on how to approach > scaling > > >> the > > >> >> > pattern matching query. Note all these techniques talk about > > scaling > > >> a > > >> >> > single query. > > >> >> > I've read various ways such as > > >> >> > > > >> >> > 1. Merging equivalent runs[1] -: This seems a good way to squash > > >> >> multiple > > >> >> > instances of pattern matching forks into a single one if they > have > > >> the > > >> >> same > > >> >> > state. > > >> >> > But I'm not sure how we would implement this in Flink since this > > is a > > >> >> > runtime optimization. > > >> >> > > > >> >> > 2. Implementing a matched version buffer[1] -: This would > involve > > >> >> sharing > > >> >> > state of a buffer datastructure across multiple candidate match > > >> >> instances > > >> >> > for the pattern. > > >> >> > > > >> >> > 3. Splitting complex composite patterns into simpler > > sub-patterns[4] > > >> >> and > > >> >> > executing separate queries to detect those sub-patterns. This > might > > >> >> > translate into different > > >> >> > tasks and duplicating the source datastreams to all the new > > generated > > >> >> > tasks. > > >> >> > > > >> >> > Also since I don't know how the Flink compiler behaves, would > some > > of > > >> >> the > > >> >> > optimizations involve making changes to it too? > > >> >> > > > >> >> > Regards, > > >> >> > Akshay Dixit > > >> >> > > > >> >> > [1] : Efficient Pattern Matching over Event Streams > > >> >> > < > > >> > http://people.cs.umass.edu/~yanlei/publications/sase-sigmod08-long.pdf > > >> >> > > > >> >> > [2] : On Supporting Kleene Closure over Event Streams > > >> >> > <http://people.cs.umass.edu/~yanlei/publications/sase-icde08.pdf > > > > >> >> > [3] : Processing Flows of Information: From Data Stream to > Complex > > >> Event > > >> >> > Processing > > >> >> > < > > >> >> > > >> > > > http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.396.1785&rep=rep1&type=pdf > > >> >> > > > >> >> > [4] : Distributing Complex Event Detection > > >> >> > < > > >> >> > > >> > http://www.doc.ic.ac.uk/teaching/distinguished-projects/2012/k.nagy.pdf > > > > > >> >> > > > >> >> > On Mon, Mar 16, 2015 at 3:22 PM, Márton Balassi < > > >> >> balassi.mar...@gmail.com> > > >> >> > wrote: > > >> >> > > > >> >> >> Dear Akshay, > > >> >> >> > > >> >> >> Thanks again for your interest and for the recent contribution > to > > >> >> >> streaming. > > >> >> >> > > >> >> >> Both of the projects mentioned wold be largely appreciated by > the > > >> >> >> community, and you can also propose other project suggestions > here > > >> for > > >> >> >> discussion. > > >> >> >> > > >> >> >> Regarding FLINK-1534, the thesis I mentioned serves as a > starting > > >> point > > >> >> >> and > > >> >> >> indeed the basic solution can be implemented with filtering and > > >> >> >> windowing/mapping with some state storing whether the cause of > an > > >> event > > >> >> >> has > > >> >> >> been already seen. Solely relying on the now existing windowing > > API > > >> >> this > > >> >> >> however might cause performance issues if the events also have > an > > >> >> >> expiration timeout - some optimization there would be included. > > The > > >> >> >> further > > >> >> >> challenge is to try to further exploit the parallel job > execution > > of > > >> >> Flink > > >> >> >> to possibly scale a pattern matching query. > > >> >> >> > > >> >> >> Best, > > >> >> >> > > >> >> >> Marton > > >> >> >> > > >> >> >> On Sun, Mar 15, 2015 at 3:22 PM, Akshay Dixit < > > akshayd...@gmail.com > > >> > > > >> >> >> wrote: > > >> >> >> > > >> >> >> > Hi, > > >> >> >> > I'm Akshay Dixit[1], a 4th year undergrad at VIT Vellore, > India. > > >> I'm > > >> >> >> > currently interested in distributed systems and stream > > processing > > >> >> and am > > >> >> >> > looking to delve deeper into the subject, and hope to get some > > >> >> insight > > >> >> >> by > > >> >> >> > contributing to Apache Flink. I've gathered some idea of the > > >> >> >> > flink-streaming codebase by recently working on a PR for > > >> >> FLINK-1450[2]. > > >> >> >> > > > >> >> >> > Both FLINK-1617[3] and FLINK-1534[4] are interesting projects > > >> that I > > >> >> >> would > > >> >> >> > love to work on over the summer. I was wondering which amongst > > >> these > > >> >> >> would > > >> >> >> > be more appreciated by the community, so I can start working > > >> towards > > >> >> a > > >> >> >> > proposal for either one. > > >> >> >> > > > >> >> >> > Regarding FLINK-1534, I was wondering why would simply merging > > and > > >> >> >> > filtering the existing streams for events we want to detect > not > > >> work? > > >> >> >> Also > > >> >> >> > on going through the document mentioned by @mbalassi in the > JIRA > > >> >> >> > comment[5], the authors specify some Runtime Event Detection > > >> >> concepts in > > >> >> >> > Section 5.2. I'm assuming the project entails on building a > > >> similar > > >> >> >> analogy > > >> >> >> > using Flink and the deliverables would include working pattern > > >> >> matching > > >> >> >> > operators over Flink DataStreams as described in the report. > If > > >> so, > > >> >> then > > >> >> >> > shouldn't it be trivial to implement the described the Binary > > >> >> operator > > >> >> >> > using a WindowedStream and a Filter? > > >> >> >> > I hope my questions don't seem misplaced here and I would > > >> appreciate > > >> >> >> links > > >> >> >> > to literature where I can learn more on the topic. > > >> >> >> > > > >> >> >> > Regards, > > >> >> >> > Akshay Dixit > > >> >> >> > > > >> >> >> > [1] : http://akshaydixi.me > > >> >> >> > [2] : https://github.com/apache/flink/pull/481 > > >> >> >> > [3] : https://issues.apache.org/jira/browse/FLINK-1617 > > >> >> >> > [4] : https://issues.apache.org/jira/browse/FLINK-1534 > > >> >> >> > [5] : > > >> >> >> > > > >> >> > > >> > http://www.doc.ic.ac.uk/teaching/distinguished-projects/2012/k.nagy.pdf > > >> >> >> > > > >> >> >> > > >> >> > > > >> >> > > > >> >> > > >> > > > >> > > > >> > > > > > > > > >
