Hi, I was wondering if anyone has done some work around measuring the cluster resource utilization of a "typical" spark streaming job.
We are trying to build a message ingestion system which will read from Kafka and do some processing. We have had some concerns raised in the team that a 24*7 streaming job might not be the best use of cluster resources especially when our use cases are to process data in a micro batch fashion and are not truly streaming. We wanted to measure as to how much resource does a spark streaming process take. Any pointers on where one would start? We are on Yarn and plan to use spark 2.1 Thanks in advance, Nadeem
