Thanks Deepak. Does this mean Streaming from HBase is not possible using current Streaming API? Also request you to shred some light on HBase checkpointing. I referred the below URL to implement checkpointing however in the example I see count is passed in the SourceFunction ( SourceFunction<Long>) Is it possible to checkpoint based on the data we read from HBase
https://ci.apache.org/projects/flink/flink-docs-release-1.12/api/java/org/apache/flink/streaming/connectors/twitter/TwitterSource.html Regards,Sunitha. On Monday, December 28, 2020, 10:51:45 AM GMT+5:30, Deepak Sharma <deepakmc...@gmail.com> wrote: I would suggest another approach here.1.Write a job that reads from hbase , checkpoints and pushes the data to broker such as Kafka.2.Flink streaming job would be the second job to read for kafka and process data. With the separation of the concern as above , maintaining it would be simpler. ThanksDeepak On Mon, Dec 28, 2020 at 10:42 AM s_penakalap...@yahoo.com <s_penakalap...@yahoo.com> wrote: Hi Team, Kindly help me with some inputs.. I am using Flink 1.12. Regards,Sunitha. On Thursday, December 24, 2020, 08:34:00 PM GMT+5:30, s_penakalap...@yahoo.com <s_penakalap...@yahoo.com> wrote: Hi Team, I recently encountered one usecase in my project as described below: My data source is HBaseWe receive huge volume of data at very high speed to HBase tables from source system.Need to read from HBase, perform computation and insert to postgreSQL. I would like few inputs on the below points: - Using Flink streaming API, is continuous streaming possible from HBase Database? As I tried using RichSourceFunction ,StreamExecutionEnvironment and was able to read data but Job stops once all data is read from HBase. My requirement is Job should be continuously executing and read data as and when data arrives to HBase table. - If continuous streaming from HBase is supported, How can Checkpointing be done on HBase so that Job can be restarted from the pointed where Job aborted. I tried googling but no luck. Request to help with any simple example or approach. - If continuous streaming from HBase is not supported then what should be alternative approach - Batch Job?(Our requirement is to process the realtime data from HBase and not to launch multiple ETL Job) Happy Christmas to all :) Regards,Sunitha. -- Thanks Deepak www.bigdatabig.com www.keosha.net
