Fabian, could you please clarify the following statement:
However joining an append-only table with this view without adding temporal join condition, means that the stream is fully materialized as state. This is because previously emitted results must be updated when the view changes. It really depends on the semantics of the join and query that you need, how much state the query will need to maintain. I am not sure to understand the problem. If i have to append-only table and perform some join on it, what's the issue ? On Tue, Aug 6, 2019 at 8:03 PM Maatary Okouya <maatarioko...@gmail.com> wrote: > Thank you for the clarification. Really appreciated. > > Is Last_val part of the API ? > > On Fri, Aug 2, 2019 at 10:49 AM Fabian Hueske <fhue...@gmail.com> wrote: > >> Hi, >> >> Flink does not distinguish between streams and tables. For the Table API >> / SQL, there are only tables that are changing over time, i.e., dynamic >> tables. >> A Stream in the Kafka Streams or KSQL sense, is in Flink a Table with >> append-only changes, i.e., records are only inserted and never deleted or >> modified. >> A Table in the Kafka Streams or KSQL sense, is in Flink a Table that has >> upsert and delete changes, i.e., the table has a unique key and records are >> inserted, deleted, or updated per key. >> >> In the current version, Flink does not have native support to ingest an >> upsert stream as a dynamic table (right now only append-only tables can be >> ingested, native support for upsert tables will be added soon.). >> However, you can create a view with the following SQL query on an >> append-only table that creates an upsert table: >> >> SELECT key, LAST_VAL(v1), LAST_VAL(v2), ... >> FROM appendOnlyTable >> GROUP BY key >> >> Given, this view, you can run all kinds of SQL queries on it. >> However joining an append-only table with this view without adding >> temporal join condition, means that the stream is fully materialized as >> state. >> This is because previously emitted results must be updated when the view >> changes. >> It really depends on the semantics of the join and query that you need, >> how much state the query will need to maintain. >> >> An alternative to using Table API / SQL and it's dynamic table >> abstraction is to use Flink's DataStream API and ProcessFunctions. >> These APIs are more low level and expose access to state and timers, >> which are the core ingredients for stream processing. >> You can implement pretty much all logic of KStreams and more in these >> APIs. >> >> Best, Fabian >> >> >> Am Di., 23. Juli 2019 um 13:06 Uhr schrieb Maatary Okouya < >> maatarioko...@gmail.com>: >> >>> I would like to have a KTable, or maybe in Flink term a dynamic Table, >>> that only contains the latest value for each keyed record. This would allow >>> me to perform aggregation and join, based on the latest state of every >>> record, as opposed to every record over time, or a period of time. >>> >>> On Sun, Jul 21, 2019 at 8:21 AM miki haiat <miko5...@gmail.com> wrote: >>> >>>> Can you elaborate more about your use case . >>>> >>>> >>>> On Sat, Jul 20, 2019 at 1:04 AM Maatary Okouya <maatarioko...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I am a user of Kafka Stream so far. However, because i have been face >>>>> with several limitation in particular in performing Join on KTable. >>>>> >>>>> I was wondering what is the appraoch in Flink to achieve (1) the >>>>> concept of KTable, i.e. a Table that represent a changeLog, i.e. only the >>>>> latest version of all keyed records, and (2) joining those. >>>>> >>>>> There are currently a lot of limitation around that on Kafka Stream, >>>>> and i need that for performing some ETL process, where i need to mirror >>>>> entire databases in Kafka, and then do some join on the table to emit the >>>>> logical entity in Kafka Topics. I was hoping that somehow i could acheive >>>>> that by using FLink as intermediary. >>>>> >>>>> I can see that you support any kind of join, but i just don't see the >>>>> notion of Ktable. >>>>> >>>>> >>>>>
