Hi Dan, I have not touched Materialize yet, but the picture of them is too simplifying. When you run Flink in parallel, then each source shard is assigned to one Flink source operator. Similarly, filter and map would run in parallel. Flink chains simple operators that have the same degree of parallelism by default, making them run on the same core. [1]
That means that their example looks exactly the same in Flink if you run Flink with parallelism 4. You need to exchange data only for aggregations with different keys (count in their example). The same is true for Kafka Streams btw. So while there may be differences in how Materialize and Flink works, this example is not suitable to depict it. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/concepts/flink-architecture.html#tasks-and-operator-chains On Tue, Jan 5, 2021 at 2:03 AM Dan Hill <quietgol...@gmail.com> wrote: > Has anyone compared Flink with Materialize? A friend recommended me > switch to Materialize. > > In one of their blog posts, it says that Flink splits operators across > CPUs (instead of splitting partitions across CPUs). Is this true? Is it > configurable? > > https://materialize.com/blog-rocksdb/ > > -- Arvid Heise | Senior Java Developer <https://www.ververica.com/> Follow us @VervericaData -- Join Flink Forward <https://flink-forward.org/> - The Apache Flink Conference Stream Processing | Event Driven | Real Time -- Ververica GmbH | Invalidenstrasse 115, 10115 Berlin, Germany -- Ververica GmbH Registered at Amtsgericht Charlottenburg: HRB 158244 B Managing Directors: Timothy Alexander Steinert, Yip Park Tung Jason, Ji (Toni) Cheng
