Re: Loading a Hive lookup table data into TM memory takes so long.

2022-02-07 Thread Jason Yi
is not only slow but also very > memory-consuming. > > Have you tried joining your main stream with the hive table directly (that > is, using streaming joins instead of lookup joins)? Does that meet your > need or why do you have to use lookup joins? > > Jason Yi <93t...@gmail.

Loading a Hive lookup table data into TM memory takes so long.

2022-02-04 Thread Jason Yi
Hello, I created external tables on Hive with data in s3 and wanted to use those tables as a lookup table in Flink. When I used an external table containing a small size of data as a lookup table, Flink quickly loaded the data into TM memory and did a Temporal join to an event stream. But, when I

Re: Question - Filesystem connector for lookup table

2022-01-20 Thread Jason Yi
wrote: > Hi Jason, > > It's not (properly) supported and we should update the documentation. > > There is no out of the box possibility to use a file from filesystem as a > lookup table as far as I know. > > Best regards, > > Martijn > > Op do 20 jan. 20

Question - Filesystem connector for lookup table

2022-01-20 Thread Jason Yi
Hello, I have data sets in s3 and want to use them as lookup tables in Flink. I defined tables with the filesystem connector and joined the tables to a table, defined with the Kinesis connector, in my Flink application. I expected its output to be written to s3, but no data was written to a sink t