I think it’s better to provide lower-level APIs. Though high-level APIs are
preferred by many users, but lower-level APIs are still needed to enhance the
expressiveness and ease the optimization.
Flink APIs now provide good abstraction for many operations, but higher
abstraction always means p
I do actually think that all levels of abstraction have their value.
If you wish, we have (top to bottom):
(1) SQL
(2) Table API with simplified Stream/Table duality
(3) DataStream API / windows
(4) DataStream API with custom windows and triggers
(5) Custom Operators
The Data Scientist may n
I agree with Vasia that for data scientist it's likely easier to learn the
high-level api. I like the material from
http://dataartisans.github.io/flink-training/ but all of them focus on the
high level api.
Maybe we could have a guide (blog post, lecture, whatever) on how to get
into Flink as a St
Jamie,
I think you raise a valid concern but I would hesitate to accept the suggestion
that the low-level API be promoted to app developers.
Higher-level abstractions tend to be more constrained and more optimized,
whereas lower-level abstractions tend to be more powerful, be more laborious to
Hi Jamie,
thanks for sharing your thoughts on this! You're raising some interesting
points.
Whether users find the lower-level primitives more intuitive depends on
their background I believe. From what I've seen, if users are coming from
the S4/Storm world and are used to the "compositional" way
You lost me at lattice, Aljoscha ;)
I do think something like the more powerful N-way FlatMap w/ Timers
Aljoscha is describing here would probably solve most of the problem.
Often Flink's higher level primitives work well for people and that's
great. It's just that I also spend a fair amount of t
Hi,
I'm also not sure whether we should start teaching Flink by demonstrating
the low-level APIs. According to my experience, people new to Flink should
first learn a very basic set of primitive operations. Usually this is map,
flatmap, join, windows, etc. The semantics of these operations is well
Hi All,
I also thought about this recently. A good think would be to add a good
user facing operator that behaves more or less like an enhanced FlatMap
with multiple inputs, multiple outputs, state access and keyed timers. I'm
a bit hesitant, though, since users rarely think about the implications
It really depends on the skill level of the developer. Using low-level
API requires to think about many details (eg. state handling etc.) that
could be done wrong.
As Flink gets a broader community, more people will use it who might not
have the required skill level to deal with low-level API. For
Hi Jamie,
I agree that it is often much easier to work on the lower level APIs if you
know what you are doing.
I think it would be nice to have very clean abstractions on that level so
we could teach this to the users first but currently I thinm its not easy
enough to be good starting point.
The
10 matches
Mail list logo
