Hi,
I'm trying to work with sliding window example given by databricks.
https://databricks.gitbooks.io/databricks-spark-reference-applications/content/logs_analyzer/chapter1/windows.html
It works fine as expected.
My question is how do I determine when the last phase of of slider has
reached. I want to perform final operation and notify other system when end
of the slider has reched to the window duarions. e.g. in below example
from databricks,
JavaDStream<ApacheAccessLog> windowDStream =
accessLogDStream.window(WINDOW_LENGTH, SLIDE_INTERVAL);
windowDStream.foreachRDD(accessLogs -> {
if (accessLogs.count() == 0) {
System.out.println("No access logs in this time interval");
return null;
}
// Insert code verbatim from LogAnalyzer.java or LogAnalyzerSQL.java here.
// Calculate statistics based on the content size.
JavaRDD<Long> contentSizes =
accessLogs.map(ApacheAccessLog::getContentSize).cache();
System.out.println(String.format("Content Size Avg: %s, Min: %s, Max: %s",
contentSizes.reduce(SUM_REDUCER) / contentSizes.count(),
contentSizes.min(Comparator.naturalOrder()),
contentSizes.max(Comparator.naturalOrder())));
//.....
}
I want to check if entire average at the end of window falls below
certain value and send alert. How do I get this?
Thanks,
LCassa