Re: Streaming kafka data sink to hive

2020-03-19 Thread Jingsong Li
Hi wanglei, > 1 Is there any flink-hive-connector that i can use to write to hive streamingly? "Streaming kafka data sink to hive" is under discussion.[1] And POC work is ongoing.[2] We want to support it in release-1.11. > 2 Since HDFS is not friendly to frequently append an

Streaming kafka data sink to hive

2020-03-19 Thread wangl...@geekplus.com.cn
We have many app logs on our app server and want to parse the logs to structed table format and then sink to hive. Seems it is good to use batch mode. The app log is hourly compressed and it is convenience to do partitioning. We want to use streaming mode. Tail the app logs to Kafka, then use