Re: Hello World / CRUNCH Framework

2018-12-16 Thread Julian Feinauer
Hi Julian, thanks for your answer and your insights. I agree with you on many points (especially our last discussion on the Calcite ML made me think a lot). So I agree with your "layered" approach, and in fact this is what we currently do (without stating it explicit enough, I think). Basically

Re: Hello World / CRUNCH Framework

2018-12-15 Thread Julian Hyde
Hi Julian, Regarding whether to do this as a streaming engine (with its own query language) or as a framework above a streaming engine, I’d say that’s a false choice. If there is relational algebra inside your system, you can provide a high-level query language that can be translated to a lower

Re: Hello World / CRUNCH Framework

2018-12-14 Thread Julian Feinauer
Hi Brian, thanks for your email! Regarding your hint, we know about the "real" CRUNCH project and are willing to change the projects name if we would enter the incubator (sadly we are massively lacking creativity...). Regarding Zipkin, I could imagine that some of the things we are doing fits

Re: Hello World / CRUNCH Framework

2018-12-14 Thread Julian Feinauer
Hi Brian, thanks for your email. Regarding your hint, we know about the "real" CRUNCH project and are willing to change the projects name if we would enter the incubator (sadly we are massively lacking creativity...). Regarding Zipkin, I could imagine that some of the things we are doing fits

Re: Hello World / CRUNCH Framework

2018-12-14 Thread Brian Devins-Suresh
Hi Julian, This seems like a cool project. I'd like to point out one thing with your name, there is already an Apache Crunch project: https://crunch.apache.org On a couple collaborative notes, as a (incubating) Zipkin developer I could see some use for your project in our community. We've had tro

Re: Hello World / CRUNCH Framework

2018-12-14 Thread Julian Feinauer
Hi Chris, yes, you got it right. We do not care about "how does this message get from this processing node to that". We "transpile" the higher level input into a DAG which can then run on basically every streaming engine (I agree, we do NOT need yet another one), in that sense it is a bit like

Re: Hello World / CRUNCH Framework

2018-12-14 Thread Christofer Dutz
Hi Julian, For me it always felt like crunch can't directly be compared to the other "streaming engines" as I see it as a bundle of a streaming engine and a higher level framework for doing typical industry operations on top of that. I think the higher level library should actually be able to r