Hi Kanchi, One can already directly query Kafka topics using Flink SQL. The Flink Kafka connector allows one to use Kafka to back a Flink table.
As a concrete example, this blog post should explain things: https://flink.apache.org/2020/07/28/flink-sql-demo-building-an-end-to-end-streaming-application/#creating-a-kafka-table-using-ddl Is there some capability which is missing from the existing set of features? Cheers, Jim On Fri, May 10, 2024 at 2:21 AM Kanchi Masalia <kmasa...@pinterest.com.invalid> wrote: > Dear Flink Community, > > I hope this message finds you well. I am considering the idea of extending > FlinkSQL's capabilities to include direct queries from Kafka topics, > specifically through the use of "SELECT * FROM kafka_topic". Currently, as > we are aware, FlinkSQL permits queries like "SELECT * FROM > flink_sql_table". > > I am writing to seek your valued opinions and insights on whether extending > this capability to include direct Kafka topic queries would align with the > guidelines and architectural philosophy of FlinkSQL. This potential feature > could offer a more streamlined approach for users to access and analyze > data directly from Kafka potentially simplifying workflow processes. > > Here are a few points I would like your input on: > > 1. Does this idea align with the current and future usage philosophy of > FlinkSQL? > 2. Are there potential challenges or conflicts this feature might pose > against existing FlinkSQL guidelines? > 3. Would this enhancement align with how you currently use or plan to > use Flink in your data processing tasks? > > Your feedback will be instrumental in guiding the direction of this > potential feature. > > Thank you very much for considering my inquiry. I am looking forward to > your insights and hope this can spark a constructive discussion within our > community. > > Thanks, > Kanchi Masalia >
