Thanks for suggestion Netanel,
Sorry for less information, I am specifically looking for something inside
Hadoop ecosystem.
-
Manjunath
From: Netanel Malka
Sent: Wednesday, March 18, 2020 5:26 PM
To: Manjunath Shetty H
Subject: Re: Saving Spark run stats and
Hi All,
Want to save each spark batch run stats (start, end, ID etc) and watermark (
Last processed timestamp from external data source).
We have tried Hive JDBC, but it is very slow due MR jobs it will trigger. Cant
save to normal Hive tables as it will create lots of small files in HDFS.
Ple
