ok, but is it? As I said, both sources have different data types. In the example here:
https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/hybridsource/ We are using both sources as returning string but in our case, one source would return a protobuf event while the other would return a pojo. How can we make the 2 sources share the same datatype so that we can successfully use hybrid source? Regards, Oscar On Tue, 4 Jul 2023 at 12:04, Alexey Novakov <ale...@ververica.com> wrote: > Hi Oscar, > > You could use connected streams and put your file into a special Kafka > topic before starting such a job: > https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/operators/overview/#connect > But this may require more work and the event ordering (which is shuffled) > in the connected streams is probably not what you are looking for. > > I think HybridSource is the right solution. > > Best regards, > Alexey > > On Mon, Jul 3, 2023 at 3:44 PM Oscar Perez via user <user@flink.apache.org> > wrote: > >> Hei, We want to bootstrap some data from a CSV file before reading from a >> kafka topic that has a retention period of 7 days. >> >> We believe the best tool for that would be the HybridSource but the >> problem we are facing is that both datasources are of different nature. The >> KafkaSource returns a protobuf event while the CSV is a POJO with just 3 >> fields. >> >> We could hack the kafkasource implementation and then in the >> valuedeserializer do the mapping from protobuf to the CSV POJO but that >> seems rather hackish. Is there a way more elegant to unify both datatypes >> from both sources using Hybrid Source? >> >> thanks >> Oscar >> >
