Yun Gao created FLINK-17590:
-------------------------------

             Summary: Add Bucket lifecycle listener to support acquiring bucket 
state
                 Key: FLINK-17590
                 URL: https://issues.apache.org/jira/browse/FLINK-17590
             Project: Flink
          Issue Type: New Feature
          Components: Connectors / FileSystem
            Reporter: Yun Gao
             Fix For: 1.11.0


Hive sink will reuse the Buckets class of StreamingFileSink, which encapsulate 
most of the logic of StreamingFileSink. Hive sink requires to writing one-piece 
of meta-info into Hive meta store after a partition (namely Bucket in 
StreamingFileSink) has been terminated. Currently the termination is judged by 
event-time/processing time 
([FLIP-115|[https://cwiki.apache.org/confluence/display/FLINK/FLIP-115%3A+Filesystem+connector+in+Table]]).

 

To support the requirement of the Hive Sink, we would add listener for 
acquiring the event bucket creation and getting inactive. A bucket get inactive 
if all the previous records have been committed. Then Hive Sink could safely 
writing meta-info if the time has exceeded the bucket's boundary and it has 
been inactive. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to