Hi Oscar,

Couldn’t you have both the Kafka and File sources return an Either<POJO from 
CSV File, Protobuf from Kafka>, and then (after the HybridSource) use a 
MapFunction to convert to the unified/correct type?

— Ken


> On Jul 4, 2023, at 12:13 PM, Oscar Perez via user <user@flink.apache.org> 
> wrote:
> 
> Hei,
> 1) We populate state based on this CSV data and do business logic based on 
> this state and events coming from other unrelated streams.
> 2) We are using low level process function in order to process this future 
> hybrid source
> 
> Regardless of the aforementioned points, please note that the main challenge 
> is to combine in a hybridsource CSV and kafka topic that return different 
> datatypes so I dont know how my answers relate to the original problem tbh. 
> Regards,
> Oscar
> 
> On Tue, 4 Jul 2023 at 20:53, Alexander Fedulov <alexander.fedu...@gmail.com 
> <mailto:alexander.fedu...@gmail.com>> wrote:
> @Oscar <>
> 1. How do you plan to use that CSV data? Is it needed for lookup from the 
> "main" stream?
> 2. Which API are you using? DataStream/SQL/Table or low level ProcessFunction?
> 
> Best,
> Alex
> 
>  <>
> On Tue, 4 Jul 2023 at 11:14, Oscar Perez via user <user@flink.apache.org 
> <mailto:user@flink.apache.org>> wrote:
> ok, but is it? As I said, both sources have different data types. In the 
> example here:
> 
> https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/hybridsource/
>  
> <https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/datastream/hybridsource/>
> 
> We are using both sources as returning string but in our case, one source 
> would return a protobuf event while the other would return a pojo. How can we 
> make the 2 sources share the same datatype so that we can successfully use 
> hybrid source?
> 
> Regards,
> Oscar
> 
> On Tue, 4 Jul 2023 at 12:04, Alexey Novakov <ale...@ververica.com 
> <mailto:ale...@ververica.com>> wrote:
> Hi Oscar,
> 
> You could use connected streams and put your file into a special Kafka topic 
> before starting such a job: 
> https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/operators/overview/#connect
>  
> <https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/operators/overview/#connect>
> But this may require more work and the event ordering (which is shuffled) in 
> the connected streams is probably not what you are looking for.
> 
> I think HybridSource is the right solution.
> 
> Best regards,
> Alexey
> 
> On Mon, Jul 3, 2023 at 3:44 PM Oscar Perez via user <user@flink.apache.org 
> <mailto:user@flink.apache.org>> wrote:
> Hei, We want to bootstrap some data from a CSV file before reading from a 
> kafka topic that has a retention period of 7 days.
> 
> We believe the best tool for that would be the HybridSource but the problem 
> we are facing is that both datasources are of different nature. The 
> KafkaSource returns a protobuf event while the CSV is a POJO with just 3 
> fields.
> 
> We could hack the kafkasource implementation and then in the 
> valuedeserializer do the mapping from protobuf to the CSV POJO but that seems 
> rather hackish. Is there a way more elegant to unify both datatypes from both 
> sources using Hybrid Source?
> 
> thanks
> Oscar

--------------------------
Ken Krugler
http://www.scaleunlimited.com
Custom big data solutions
Flink, Pinot, Solr, Elasticsearch



Reply via email to